2023-06-10T18:32:00.727 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-10T18:32:00.729 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --description fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} --name pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167 -- /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/orig.config.yaml 2023-06-10T18:32:00.772 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-10T18:32:00.867 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167 branch: main description: fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} email: pdonnell@redhat.com first_in_suite: false job_id: '7301167' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: main ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - but it is still running - slow request - MDS_CLIENT_LATE_RELEASE - t responding to mclientcaps - file system flag refuse_client_session is set sha1: 219d315c3a0b2075e570fb1964be2ad33fe7a9c8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 219d315c3a0b2075e570fb1964be2ad33fe7a9c8 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i61409 sha1: 3486dd872ffd89042ca079d680841ce3ef6f4b6f owner: scheduled_pdonnell@teuthology priority: 70 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5151 sha1: 219d315c3a0b2075e570fb1964be2ad33fe7a9c8 sleep_before_teardown: 0 suite: fs:functional suite_branch: i61409 suite_path: /home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 3486dd872ffd89042ca079d680841ce3ef6f4b6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_client_recovery teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61 timestamp: 2023-06-10_18:20:34 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-10T18:32:00.867 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa; will attempt to use it 2023-06-10T18:32:00.869 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa/tasks 2023-06-10T18:32:00.869 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-10T18:32:00.873 INFO:teuthology.task.internal:Checking packages... 2023-06-10T18:32:00.921 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '219d315c3a0b2075e570fb1964be2ad33fe7a9c8' 2023-06-10T18:32:00.921 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-10T18:32:00.922 INFO:teuthology.packaging:ref: None 2023-06-10T18:32:00.922 INFO:teuthology.packaging:tag: None 2023-06-10T18:32:00.922 INFO:teuthology.packaging:branch: main 2023-06-10T18:32:00.923 INFO:teuthology.packaging:sha1: 219d315c3a0b2075e570fb1964be2ad33fe7a9c8 2023-06-10T18:32:00.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=main 2023-06-10T18:32:01.136 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4359.g219d315c 2023-06-10T18:32:01.137 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-10T18:32:01.143 INFO:teuthology.task.internal:no buildpackages task found 2023-06-10T18:32:01.144 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-10T18:32:01.150 INFO:teuthology.task.internal:Saving configuration 2023-06-10T18:32:01.175 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-10T18:32:01.185 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-10T18:32:01.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-10 18:26:56.703761', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-10T18:32:01.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-10 18:26:56.701891', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-10T18:32:01.247 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-10T18:32:01.254 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-10T18:32:01.254 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-10T18:32:01.254 INFO:teuthology.run_tasks:Running task console_log... 2023-06-10T18:32:01.345 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1eb8ec8f70>, signals=[15]) 2023-06-10T18:32:01.348 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-10T18:32:01.356 INFO:teuthology.task.internal:Opening connections... 2023-06-10T18:32:01.357 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:32:01.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-10T18:32:01.436 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:32:01.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-10T18:32:01.506 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-10T18:32:01.512 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2023-06-10T18:32:01.532 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2023-06-10T18:32:01.533 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2023-06-10T18:32:01.592 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Red Hat Enterprise Linux" 2023-06-10T18:32:01.592 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8.6 (Ootpa)" 2023-06-10T18:32:01.592 INFO:teuthology.orchestra.run.smithi104.stdout:ID="rhel" 2023-06-10T18:32:01.593 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="fedora" 2023-06-10T18:32:01.593 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8.6" 2023-06-10T18:32:01.593 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2023-06-10T18:32:01.593 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.redhat.com/" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:32:01.594 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-10T18:32:01.595 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-06-10T18:32:01.595 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-10T18:32:01.595 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-06-10T18:32:01.595 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2023-06-10T18:32:01.620 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2023-06-10T18:32:01.641 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2023-06-10T18:32:01.641 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2023-06-10T18:32:01.701 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Red Hat Enterprise Linux" 2023-06-10T18:32:01.701 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8.6 (Ootpa)" 2023-06-10T18:32:01.702 INFO:teuthology.orchestra.run.smithi114.stdout:ID="rhel" 2023-06-10T18:32:01.702 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="fedora" 2023-06-10T18:32:01.705 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8.6" 2023-06-10T18:32:01.705 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.redhat.com/" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-10T18:32:01.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:32:01.707 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-10T18:32:01.707 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-06-10T18:32:01.707 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-10T18:32:01.707 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-06-10T18:32:01.707 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2023-06-10T18:32:01.728 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-10T18:32:01.743 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-10T18:32:01.749 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-10T18:32:01.750 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2023-06-10T18:32:01.752 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2023-06-10T18:32:01.770 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-10T18:32:01.777 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-10T18:32:01.777 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2023-06-10T18:32:01.811 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2023-06-10T18:32:01.844 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-10T18:32:01.916 INFO:teuthology.run_tasks:Running task kernel... 2023-06-10T18:32:01.937 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-06-10T18:32:01.939 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-06-10T18:32:01.939 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-10T18:32:01.941 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-10T18:32:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:32:01.960 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-06-10T18:32:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:32:01.962 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2023-06-10T18:32:02.019 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:02.019 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:02.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2023-06-10T18:32:02.021 INFO:teuthology.orchestra.run.smithi104.stdout:4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:02.021 INFO:teuthology.task.kernel:Running kernel on smithi104: 4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:02.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2023-06-10T18:32:03.124 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:03.125 INFO:teuthology.orchestra.run.smithi104.stdout:Unable to read consumer identity 2023-06-10T18:32:03.225 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:03.226 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-06-10T18:32:06.788 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-10T18:32:06.889 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-10T18:32:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-06-10T18:32:08.456 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-06-10T18:32:11.801 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 459 kB/s | 24 kB 00:00 2023-06-10T18:32:11.923 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 303 kB/s | 24 kB 00:00 2023-06-10T18:32:12.435 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-06-10T18:32:12.485 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-06-10T18:32:12.486 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2023-06-10T18:32:12.487 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-06-10T18:32:12.613 DEBUG:teuthology.orchestra.run.smithi114:> echo no | sudo yum reinstall kernel || true 2023-06-10T18:32:12.625 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-06-10T18:32:12.679 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2023-06-10T18:32:12.680 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2023-06-10T18:32:12.681 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2023-06-10T18:32:12.819 DEBUG:teuthology.orchestra.run.smithi104:> echo no | sudo yum reinstall kernel || true 2023-06-10T18:32:12.946 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:12.947 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-06-10T18:32:13.152 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:13.153 INFO:teuthology.orchestra.run.smithi104.stdout:Unable to read consumer identity 2023-06-10T18:32:13.208 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:02 ago on Sat 10 Jun 2023 06:32:11 PM UTC. 2023-06-10T18:32:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-06-10T18:32:13.279 INFO:teuthology.orchestra.run.smithi114.stderr:Error: No packages marked for reinstall. 2023-06-10T18:32:13.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum reinstall -y kernel || true 2023-06-10T18:32:13.409 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:02 ago on Sat 10 Jun 2023 06:32:11 PM UTC. 2023-06-10T18:32:13.449 INFO:teuthology.orchestra.run.smithi104.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-06-10T18:32:13.478 INFO:teuthology.orchestra.run.smithi104.stderr:Error: No packages marked for reinstall. 2023-06-10T18:32:13.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum reinstall -y kernel || true 2023-06-10T18:32:13.656 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:13.656 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-06-10T18:32:13.851 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-06-10T18:32:13.852 INFO:teuthology.orchestra.run.smithi104.stdout:Unable to read consumer identity 2023-06-10T18:32:13.909 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:02 ago on Sat 10 Jun 2023 06:32:11 PM UTC. 2023-06-10T18:32:13.949 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-06-10T18:32:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:Error: No packages marked for reinstall. 2023-06-10T18:32:14.014 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-06-10T18:32:14.095 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:14.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:14.096 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-06-10T18:32:14.096 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-06-10T18:32:14.096 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-10T18:32:14.103 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:03 ago on Sat 10 Jun 2023 06:32:11 PM UTC. 2023-06-10T18:32:14.123 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2023-06-10T18:32:14.134 DEBUG:teuthology.parallel:result is None 2023-06-10T18:32:14.143 INFO:teuthology.orchestra.run.smithi104.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-06-10T18:32:14.166 INFO:teuthology.orchestra.run.smithi104.stderr:Error: No packages marked for reinstall. 2023-06-10T18:32:14.214 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2023-06-10T18:32:14.296 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:14.296 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-06-10T18:32:14.296 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-06-10T18:32:14.297 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-06-10T18:32:14.297 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-10T18:32:14.341 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2023-06-10T18:32:14.355 DEBUG:teuthology.parallel:result is None 2023-06-10T18:32:14.355 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-10T18:32:14.363 INFO:teuthology.task.internal:Creating test directory... 2023-06-10T18:32:14.363 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-10T18:32:14.365 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-10T18:32:14.383 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-10T18:32:14.428 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-10T18:32:14.438 INFO:teuthology.task.internal:Creating archive directory... 2023-06-10T18:32:14.439 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-10T18:32:14.441 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-10T18:32:14.465 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-10T18:32:14.473 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-10T18:32:14.473 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-10T18:32:14.504 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-10T18:32:14.530 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-10T18:32:14.532 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-10T18:32:14.540 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-10T18:32:14.541 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-10T18:32:14.543 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-10T18:32:14.555 INFO:teuthology.task.internal:Configuring sudo... 2023-06-10T18:32:14.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-10T18:32:14.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-10T18:32:14.610 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-10T18:32:14.621 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-10T18:32:14.622 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-10T18:32:14.650 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-10T18:32:14.666 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-10T18:32:14.750 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-10T18:32:14.808 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-10T18:32:14.855 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-10T18:32:14.893 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:32:14.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-10T18:32:14.960 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-10T18:32:15.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-10T18:32:15.070 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-10T18:32:15.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-10T18:32:15.155 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:32:15.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-10T18:32:15.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2023-06-10T18:32:15.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2023-06-10T18:32:15.277 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-10T18:32:15.319 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-10T18:32:15.653 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-10T18:32:15.661 INFO:teuthology.task.internal:Starting timer... 2023-06-10T18:32:15.662 INFO:teuthology.run_tasks:Running task pcp... 2023-06-10T18:32:15.675 INFO:teuthology.run_tasks:Running task selinux... 2023-06-10T18:32:15.688 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-10T18:32:15.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2023-06-10T18:32:15.753 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2023-06-10T18:32:15.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2023-06-10T18:32:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2023-06-10T18:32:15.817 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-10T18:32:15.817 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2023-06-10T18:32:15.843 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2023-06-10T18:32:15.844 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2023-06-10T18:32:15.861 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2023-06-10T18:32:15.862 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2023-06-10T18:32:15.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-10T18:32:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:32:15.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-10T18:32:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:32:15.939 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-10T18:32:15.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2023-06-10T18:32:15.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2023-06-10T18:32:16.005 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-10T18:32:16.014 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-10T18:32:16.015 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-10T18:32:16.037 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-10T18:32:16.039 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-10T18:40:42.777 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2023-06-10T18:40:42.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2023-06-10T18:40:42.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-10T18:40:42.860 DEBUG:teuthology.orchestra.run.smithi104:> true 2023-06-10T18:40:42.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-06-10T18:40:42.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-10T18:40:42.952 DEBUG:teuthology.orchestra.run.smithi114:> true 2023-06-10T18:40:43.016 INFO:teuthology.run_tasks:Running task clock... 2023-06-10T18:40:43.027 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-10T18:40:43.027 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-10T18:40:43.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-10T18:40:43.030 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-10T18:40:43.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-10T18:40:43.058 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-10T18:40:43.072 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-10T18:40:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-10T18:40:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-10T18:40:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2023-06-10T18:40:43.153 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2023-06-10T18:40:43.167 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-10T18:40:43.181 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-10T18:40:43.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2023-06-10T18:40:43.195 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2023-06-10T18:40:43.209 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-10T18:40:43.222 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-10T18:40:43.254 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2023-06-10T18:40:43.257 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-10T18:40:43.257 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2023-06-10T18:40:43.257 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.258 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.258 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.258 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.298 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2023-06-10T18:40:43.301 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-10T18:40:43.302 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2023-06-10T18:40:43.302 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.302 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.302 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.302 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T18:40:43.303 INFO:teuthology.run_tasks:Running task install... 2023-06-10T18:40:43.317 DEBUG:teuthology.task.install:project ceph 2023-06-10T18:40:43.318 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8'}} 2023-06-10T18:40:43.319 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8'} 2023-06-10T18:40:43.319 INFO:teuthology.task.install:Using flavor: default 2023-06-10T18:40:43.333 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-10T18:40:43.334 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-10T18:40:43.334 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8', 'tag': None, 'wait_for_package': False} 2023-06-10T18:40:43.335 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=219d315c3a0b2075e570fb1964be2ad33fe7a9c8 2023-06-10T18:40:43.337 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8', 'tag': None, 'wait_for_package': False} 2023-06-10T18:40:43.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=219d315c3a0b2075e570fb1964be2ad33fe7a9c8 2023-06-10T18:40:43.481 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/ 2023-06-10T18:40:43.481 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4359.g219d315c 2023-06-10T18:40:43.552 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/ 2023-06-10T18:40:43.553 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4359.g219d315c 2023-06-10T18:40:43.587 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-10T18:40:43.587 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:40:43.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-10T18:40:43.615 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-06-10T18:40:43.615 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-10T18:40:43.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-10T18:40:43.722 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-10T18:40:43.723 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:40:43.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-10T18:40:43.751 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-06-10T18:40:43.751 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/219d315c3a0b2075e570fb1964be2ad33fe7a9c8/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-10T18:40:43.758 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-10T18:40:43.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-10T18:40:43.823 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2023-06-10T18:40:43.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2023-06-10T18:40:43.894 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-10T18:40:43.959 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2023-06-10T18:40:43.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2023-06-10T18:40:44.196 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-06-10T18:40:44.331 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-06-10T18:40:44.471 INFO:teuthology.orchestra.run.smithi104.stdout:44 files removed 2023-06-10T18:40:44.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-06-10T18:40:44.570 INFO:teuthology.orchestra.run.smithi114.stdout:44 files removed 2023-06-10T18:40:44.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-06-10T18:40:44.845 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-06-10T18:40:44.947 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-06-10T18:40:45.972 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 103 kB/s | 81 kB 00:00 2023-06-10T18:40:46.030 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 110 kB/s | 81 kB 00:00 2023-06-10T18:40:46.682 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-06-10T18:40:46.781 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-06-10T18:40:47.358 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2023-06-10T18:40:47.532 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-06-10T18:40:49.249 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-06-10T18:40:49.783 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-10T18:40:50.210 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-06-10T18:40:50.960 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-06-10T18:40:53.691 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 392 kB/s | 24 kB 00:00 2023-06-10T18:40:54.392 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 396 kB/s | 24 kB 00:00 2023-06-10T18:40:54.543 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-06-10T18:40:55.293 INFO:teuthology.orchestra.run.smithi114.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-06-10T18:41:02.427 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-06-10T18:41:03.447 INFO:teuthology.orchestra.run.smithi114.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 47 MB 00:00 2023-06-10T18:41:13.532 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-10T18:41:13.533 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-10T18:41:13.548 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-06-10T18:41:13.549 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-06-10T18:41:13.550 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-06-10T18:41:13.551 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-10T18:41:13.553 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-10T18:41:13.556 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-10T18:41:13.557 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-10T18:41:13.693 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2023-06-10T18:41:13.705 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-06-10T18:41:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2023-06-10T18:41:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-06-10T18:41:13.706 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2023-06-10T18:41:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-06-10T18:41:13.706 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:18.0.0-4359.g219d315c.el8 ceph 6.4 k 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:18.0.0-4359.g219d315c.el8 ceph 5.4 M 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 847 k 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4359.g219d315c.el8 ceph 165 k 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:18.0.0-4359.g219d315c.el8 ceph 1.5 M 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 128 k 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 1.7 M 2023-06-10T18:41:13.707 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 7.4 M 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 47 k 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:18.0.0-4359.g219d315c.el8 ceph 13 M 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:18.0.0-4359.g219d315c.el8 ceph 48 M 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 258 k 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 160 k 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:18.0.0-4359.g219d315c.el8 ceph 258 k 2023-06-10T18:41:13.708 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 18 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:18.0.0-4359.g219d315c.el8 ceph 31 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 771 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:18.0.0-4359.g219d315c.el8 ceph 139 k 2023-06-10T18:41:13.709 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:18.0.0-4359.g219d315c.el8 ceph 197 k 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:18.0.0-4359.g219d315c.el8 ceph 370 k 2023-06-10T18:41:13.710 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 365 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:18.0.0-4359.g219d315c.el8 ceph 106 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 94 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.4 M 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 179 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-06-10T18:41:13.711 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.7 M 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.6 M 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:18.0.0-4359.g219d315c.el8 ceph 22 M 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 23 k 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:18.0.0-4359.g219d315c.el8 ceph 2.3 M 2023-06-10T18:41:13.712 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 240 k 2023-06-10T18:41:13.713 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:18.0.0-4359.g219d315c.el8 ceph 4.6 M 2023-06-10T18:41:13.713 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 18 M 2023-06-10T18:41:13.713 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 15 k 2023-06-10T18:41:13.713 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:18.0.0-4359.g219d315c.el8 ceph 24 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:18.0.0-4359.g219d315c.el8 ceph 137 k 2023-06-10T18:41:13.714 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-10T18:41:13.715 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 473 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 6.7 M 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-10T18:41:13.716 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-10T18:41:13.717 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 45 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:18.0.0-4359.g219d315c.el8 ceph 119 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-10T18:41:13.718 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-10T18:41:13.719 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-10T18:41:13.720 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-10T18:41:13.721 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-10T18:41:13.722 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-10T18:41:13.723 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-10T18:41:13.724 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2023-06-10T18:41:13.725 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql 10 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout:Install 128 Packages 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 3 Packages 2023-06-10T18:41:13.726 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:41:13.727 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 205 M 2023-06-10T18:41:13.727 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2023-06-10T18:41:14.302 INFO:teuthology.orchestra.run.smithi104.stdout:(1/131): ceph-18.0.0-4359.g219d315c.el8.x86_64. 14 kB/s | 6.4 kB 00:00 2023-06-10T18:41:14.325 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-10T18:41:14.326 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-10T18:41:14.340 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-06-10T18:41:14.340 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-06-10T18:41:14.342 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-06-10T18:41:14.343 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-10T18:41:14.345 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-10T18:41:14.348 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-10T18:41:14.349 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-10T18:41:14.479 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-06-10T18:41:14.492 INFO:teuthology.orchestra.run.smithi114.stdout:=========================================================================================================================== 2023-06-10T18:41:14.492 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2023-06-10T18:41:14.492 INFO:teuthology.orchestra.run.smithi114.stdout:=========================================================================================================================== 2023-06-10T18:41:14.492 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2023-06-10T18:41:14.492 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:18.0.0-4359.g219d315c.el8 ceph 6.4 k 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:18.0.0-4359.g219d315c.el8 ceph 5.4 M 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 847 k 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4359.g219d315c.el8 ceph 165 k 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:18.0.0-4359.g219d315c.el8 ceph 1.5 M 2023-06-10T18:41:14.493 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 128 k 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 1.7 M 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 7.4 M 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 47 k 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:18.0.0-4359.g219d315c.el8 ceph 13 M 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:18.0.0-4359.g219d315c.el8 ceph 48 M 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 258 k 2023-06-10T18:41:14.494 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 160 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:18.0.0-4359.g219d315c.el8 ceph 258 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 18 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-10T18:41:14.495 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:18.0.0-4359.g219d315c.el8 ceph 31 k 2023-06-10T18:41:14.496 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 771 k 2023-06-10T18:41:14.496 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:18.0.0-4359.g219d315c.el8 ceph 139 k 2023-06-10T18:41:14.496 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:18.0.0-4359.g219d315c.el8 ceph 197 k 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:18.0.0-4359.g219d315c.el8 ceph 370 k 2023-06-10T18:41:14.497 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 365 k 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:18.0.0-4359.g219d315c.el8 ceph 106 k 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 94 k 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.4 M 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 179 k 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2023-06-10T18:41:14.498 INFO:teuthology.orchestra.run.smithi114.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.7 M 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 3.6 M 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:18.0.0-4359.g219d315c.el8 ceph 22 M 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 23 k 2023-06-10T18:41:14.499 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:18.0.0-4359.g219d315c.el8 ceph 2.3 M 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 240 k 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:18.0.0-4359.g219d315c.el8 ceph 4.6 M 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:18.0.0-4359.g219d315c.el8 ceph 18 M 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4359.g219d315c.el8 ceph-noarch 15 k 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:18.0.0-4359.g219d315c.el8 ceph 24 k 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-10T18:41:14.500 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:18.0.0-4359.g219d315c.el8 ceph 137 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-06-10T18:41:14.501 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 473 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:18.0.0-4359.g219d315c.el8 ceph 6.7 M 2023-06-10T18:41:14.502 INFO:teuthology.orchestra.run.smithi114.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-06-10T18:41:14.503 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-10T18:41:14.504 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-10T18:41:14.504 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-10T18:41:14.504 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-10T18:41:14.505 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-10T18:41:14.505 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-10T18:41:14.505 INFO:teuthology.orchestra.run.smithi114.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:18.0.0-4359.g219d315c.el8 ceph 45 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:18.0.0-4359.g219d315c.el8 ceph 119 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-10T18:41:14.506 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-10T18:41:14.507 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-10T18:41:14.508 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-06-10T18:41:14.509 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-10T18:41:14.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-10T18:41:14.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-06-10T18:41:14.512 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling module streams: 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql 10 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout:=========================================================================================================================== 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout:Install 128 Packages 2023-06-10T18:41:14.513 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 3 Packages 2023-06-10T18:41:14.514 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:41:14.514 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 205 M 2023-06-10T18:41:14.514 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2023-06-10T18:41:15.178 INFO:teuthology.orchestra.run.smithi114.stdout:(1/131): ceph-18.0.0-4359.g219d315c.el8.x86_64. 12 kB/s | 6.4 kB 00:00 2023-06-10T18:41:15.293 INFO:teuthology.orchestra.run.smithi104.stdout:(2/131): ceph-fuse-18.0.0-4359.g219d315c.el8.x8 855 kB/s | 847 kB 00:00 2023-06-10T18:41:15.493 INFO:teuthology.orchestra.run.smithi104.stdout:(3/131): ceph-immutable-object-cache-18.0.0-435 828 kB/s | 165 kB 00:00 2023-06-10T18:41:16.760 INFO:teuthology.orchestra.run.smithi104.stdout:(4/131): ceph-base-18.0.0-4359.g219d315c.el8.x8 1.9 MB/s | 5.4 MB 00:02 2023-06-10T18:41:16.803 INFO:teuthology.orchestra.run.smithi104.stdout:(5/131): ceph-mds-18.0.0-4359.g219d315c.el8.x86 1.7 MB/s | 2.3 MB 00:01 2023-06-10T18:41:17.245 INFO:teuthology.orchestra.run.smithi114.stdout:(2/131): ceph-base-18.0.0-4359.g219d315c.el8.x8 2.1 MB/s | 5.4 MB 00:02 2023-06-10T18:41:17.345 INFO:teuthology.orchestra.run.smithi114.stdout:(3/131): ceph-immutable-object-cache-18.0.0-435 1.6 MB/s | 165 kB 00:00 2023-06-10T18:41:17.469 INFO:teuthology.orchestra.run.smithi104.stdout:(6/131): ceph-mgr-18.0.0-4359.g219d315c.el8.x86 2.2 MB/s | 1.5 MB 00:00 2023-06-10T18:41:18.145 INFO:teuthology.orchestra.run.smithi114.stdout:(4/131): ceph-fuse-18.0.0-4359.g219d315c.el8.x8 286 kB/s | 847 kB 00:02 2023-06-10T18:41:18.329 INFO:teuthology.orchestra.run.smithi114.stdout:(5/131): ceph-mds-18.0.0-4359.g219d315c.el8.x86 2.3 MB/s | 2.3 MB 00:00 2023-06-10T18:41:18.786 INFO:teuthology.orchestra.run.smithi104.stdout:(7/131): ceph-mon-18.0.0-4359.g219d315c.el8.x86 2.3 MB/s | 4.6 MB 00:01 2023-06-10T18:41:20.254 INFO:teuthology.orchestra.run.smithi114.stdout:(6/131): ceph-mon-18.0.0-4359.g219d315c.el8.x86 2.4 MB/s | 4.6 MB 00:01 2023-06-10T18:41:21.213 INFO:teuthology.orchestra.run.smithi114.stdout:(7/131): ceph-mgr-18.0.0-4359.g219d315c.el8.x86 517 kB/s | 1.5 MB 00:03 2023-06-10T18:41:24.246 INFO:teuthology.orchestra.run.smithi114.stdout:(8/131): ceph-common-18.0.0-4359.g219d315c.el8. 2.3 MB/s | 22 MB 00:09 2023-06-10T18:41:24.338 INFO:teuthology.orchestra.run.smithi114.stdout:(9/131): ceph-selinux-18.0.0-4359.g219d315c.el8 266 kB/s | 24 kB 00:00 2023-06-10T18:41:24.535 INFO:teuthology.orchestra.run.smithi104.stdout:(8/131): ceph-osd-18.0.0-4359.g219d315c.el8.x86 2.5 MB/s | 18 MB 00:07 2023-06-10T18:41:24.628 INFO:teuthology.orchestra.run.smithi104.stdout:(9/131): ceph-selinux-18.0.0-4359.g219d315c.el8 266 kB/s | 24 kB 00:00 2023-06-10T18:41:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:(10/131): ceph-osd-18.0.0-4359.g219d315c.el8.x8 2.9 MB/s | 18 MB 00:06 2023-06-10T18:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:(11/131): cephfs-mirror-18.0.0-4359.g219d315c.e 1.4 MB/s | 258 kB 00:00 2023-06-10T18:41:26.630 INFO:teuthology.orchestra.run.smithi114.stdout:(12/131): libcephfs-devel-18.0.0-4359.g219d315c 338 kB/s | 31 kB 00:00 2023-06-10T18:41:26.972 INFO:teuthology.orchestra.run.smithi114.stdout:(13/131): libcephfs2-18.0.0-4359.g219d315c.el8. 2.2 MB/s | 771 kB 00:00 2023-06-10T18:41:27.073 INFO:teuthology.orchestra.run.smithi114.stdout:(14/131): libcephsqlite-18.0.0-4359.g219d315c.e 1.3 MB/s | 137 kB 00:00 2023-06-10T18:41:27.181 INFO:teuthology.orchestra.run.smithi114.stdout:(15/131): librados-devel-18.0.0-4359.g219d315c. 1.3 MB/s | 139 kB 00:00 2023-06-10T18:41:27.365 INFO:teuthology.orchestra.run.smithi114.stdout:(16/131): libradosstriper1-18.0.0-4359.g219d315 2.5 MB/s | 473 kB 00:00 2023-06-10T18:41:28.152 INFO:teuthology.orchestra.run.smithi104.stdout:(10/131): ceph-radosgw-18.0.0-4359.g219d315c.el 1.4 MB/s | 13 MB 00:09 2023-06-10T18:41:28.386 INFO:teuthology.orchestra.run.smithi104.stdout:(11/131): cephfs-mirror-18.0.0-4359.g219d315c.e 1.1 MB/s | 258 kB 00:00 2023-06-10T18:41:28.528 INFO:teuthology.orchestra.run.smithi104.stdout:(12/131): libcephfs-devel-18.0.0-4359.g219d315c 219 kB/s | 31 kB 00:00 2023-06-10T18:41:29.103 INFO:teuthology.orchestra.run.smithi104.stdout:(13/131): libcephfs2-18.0.0-4359.g219d315c.el8. 1.3 MB/s | 771 kB 00:00 2023-06-10T18:41:29.203 INFO:teuthology.orchestra.run.smithi104.stdout:(14/131): libcephsqlite-18.0.0-4359.g219d315c.e 1.3 MB/s | 137 kB 00:00 2023-06-10T18:41:29.295 INFO:teuthology.orchestra.run.smithi104.stdout:(15/131): librados-devel-18.0.0-4359.g219d315c. 1.5 MB/s | 139 kB 00:00 2023-06-10T18:41:29.356 INFO:teuthology.orchestra.run.smithi114.stdout:(17/131): librgw2-18.0.0-4359.g219d315c.el8.x86 3.4 MB/s | 6.7 MB 00:01 2023-06-10T18:41:29.423 INFO:teuthology.orchestra.run.smithi114.stdout:(18/131): python3-ceph-argparse-18.0.0-4359.g21 675 kB/s | 45 kB 00:00 2023-06-10T18:41:29.509 INFO:teuthology.orchestra.run.smithi114.stdout:(19/131): python3-ceph-common-18.0.0-4359.g219d 1.4 MB/s | 119 kB 00:00 2023-06-10T18:41:29.512 INFO:teuthology.orchestra.run.smithi104.stdout:(16/131): libradosstriper1-18.0.0-4359.g219d315 2.1 MB/s | 473 kB 00:00 2023-06-10T18:41:29.566 INFO:teuthology.orchestra.run.smithi114.stdout:(20/131): python3-cephfs-18.0.0-4359.g219d315c. 3.4 MB/s | 197 kB 00:00 2023-06-10T18:41:29.691 INFO:teuthology.orchestra.run.smithi114.stdout:(21/131): python3-rados-18.0.0-4359.g219d315c.e 2.9 MB/s | 370 kB 00:00 2023-06-10T18:41:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:(22/131): python3-rbd-18.0.0-4359.g219d315c.el8 3.1 MB/s | 365 kB 00:00 2023-06-10T18:41:29.867 INFO:teuthology.orchestra.run.smithi114.stdout:(23/131): python3-rgw-18.0.0-4359.g219d315c.el8 1.8 MB/s | 106 kB 00:00 2023-06-10T18:41:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:(24/131): rbd-fuse-18.0.0-4359.g219d315c.el8.x8 1.1 MB/s | 94 kB 00:00 2023-06-10T18:41:30.342 INFO:teuthology.orchestra.run.smithi114.stdout:(25/131): ceph-radosgw-18.0.0-4359.g219d315c.el 1.5 MB/s | 13 MB 00:09 2023-06-10T18:41:30.501 INFO:teuthology.orchestra.run.smithi114.stdout:(26/131): rbd-nbd-18.0.0-4359.g219d315c.el8.x86 1.1 MB/s | 179 kB 00:00 2023-06-10T18:41:30.576 INFO:teuthology.orchestra.run.smithi114.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4359.g 308 kB/s | 23 kB 00:00 2023-06-10T18:41:30.651 INFO:teuthology.orchestra.run.smithi114.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4359.g219d315 1.7 MB/s | 128 kB 00:00 2023-06-10T18:41:30.710 INFO:teuthology.orchestra.run.smithi114.stdout:(29/131): rbd-mirror-18.0.0-4359.g219d315c.el8. 4.5 MB/s | 3.4 MB 00:00 2023-06-10T18:41:31.102 INFO:teuthology.orchestra.run.smithi114.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4359.g219d3 3.8 MB/s | 1.7 MB 00:00 2023-06-10T18:41:31.186 INFO:teuthology.orchestra.run.smithi114.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4359.g21 2.8 MB/s | 240 kB 00:00 2023-06-10T18:41:31.411 INFO:teuthology.orchestra.run.smithi114.stdout:(32/131): ceph-mgr-rook-18.0.0-4359.g219d315c.e 211 kB/s | 47 kB 00:00 2023-06-10T18:41:31.486 INFO:teuthology.orchestra.run.smithi114.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4359.g2 194 kB/s | 15 kB 00:00 2023-06-10T18:41:31.570 INFO:teuthology.orchestra.run.smithi114.stdout:(34/131): ceph-volume-18.0.0-4359.g219d315c.el8 3.0 MB/s | 258 kB 00:00 2023-06-10T18:41:31.653 INFO:teuthology.orchestra.run.smithi114.stdout:(35/131): cephadm-18.0.0-4359.g219d315c.el8.noa 1.9 MB/s | 160 kB 00:00 2023-06-10T18:41:31.729 INFO:teuthology.orchestra.run.smithi114.stdout:(36/131): cephfs-top-18.0.0-4359.g219d315c.el8. 239 kB/s | 18 kB 00:00 2023-06-10T18:41:32.013 INFO:teuthology.orchestra.run.smithi114.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 5.7 MB/s | 7.4 MB 00:01 2023-06-10T18:41:32.113 INFO:teuthology.orchestra.run.smithi114.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 798 kB/s | 306 kB 00:00 2023-06-10T18:41:32.188 INFO:teuthology.orchestra.run.smithi114.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-06-10T18:41:32.254 INFO:teuthology.orchestra.run.smithi104.stdout:(17/131): librgw2-18.0.0-4359.g219d315c.el8.x86 2.5 MB/s | 6.7 MB 00:02 2023-06-10T18:41:32.255 INFO:teuthology.orchestra.run.smithi114.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 243 kB/s | 59 kB 00:00 2023-06-10T18:41:32.321 INFO:teuthology.orchestra.run.smithi104.stdout:(18/131): python3-ceph-argparse-18.0.0-4359.g21 675 kB/s | 45 kB 00:00 2023-06-10T18:41:32.335 INFO:teuthology.orchestra.run.smithi114.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.2 MB/s | 476 kB 00:00 2023-06-10T18:41:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 435 kB/s | 44 kB 00:00 2023-06-10T18:41:32.379 INFO:teuthology.orchestra.run.smithi104.stdout:(19/131): python3-ceph-common-18.0.0-4359.g219d 2.0 MB/s | 119 kB 00:00 2023-06-10T18:41:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.6 MB/s | 185 kB 00:00 2023-06-10T18:41:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 494 kB/s | 33 kB 00:00 2023-06-10T18:41:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 279 kB/s | 16 kB 00:00 2023-06-10T18:41:32.513 INFO:teuthology.orchestra.run.smithi104.stdout:(20/131): python3-cephfs-18.0.0-4359.g219d315c. 1.4 MB/s | 197 kB 00:00 2023-06-10T18:41:32.532 INFO:teuthology.orchestra.run.smithi114.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.6 MB/s | 173 kB 00:00 2023-06-10T18:41:32.574 INFO:teuthology.orchestra.run.smithi114.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2023-06-10T18:41:32.599 INFO:teuthology.orchestra.run.smithi114.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2023-06-10T18:41:32.641 INFO:teuthology.orchestra.run.smithi114.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-06-10T18:41:32.666 INFO:teuthology.orchestra.run.smithi114.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2023-06-10T18:41:32.688 INFO:teuthology.orchestra.run.smithi104.stdout:(21/131): python3-rados-18.0.0-4359.g219d315c.e 2.1 MB/s | 370 kB 00:00 2023-06-10T18:41:32.733 INFO:teuthology.orchestra.run.smithi114.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 733 kB/s | 49 kB 00:00 2023-06-10T18:41:32.792 INFO:teuthology.orchestra.run.smithi114.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-06-10T18:41:32.830 INFO:teuthology.orchestra.run.smithi104.stdout:(22/131): python3-rbd-18.0.0-4359.g219d315c.el8 2.5 MB/s | 365 kB 00:00 2023-06-10T18:41:32.850 INFO:teuthology.orchestra.run.smithi114.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2023-06-10T18:41:32.884 INFO:teuthology.orchestra.run.smithi114.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 4.8 MB/s | 1.2 MB 00:00 2023-06-10T18:41:32.914 INFO:teuthology.orchestra.run.smithi104.stdout:(23/131): python3-rgw-18.0.0-4359.g219d315c.el8 1.2 MB/s | 106 kB 00:00 2023-06-10T18:41:32.918 INFO:teuthology.orchestra.run.smithi114.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2023-06-10T18:41:32.951 INFO:teuthology.orchestra.run.smithi114.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2023-06-10T18:41:33.006 INFO:teuthology.orchestra.run.smithi104.stdout:(24/131): rbd-fuse-18.0.0-4359.g219d315c.el8.x8 1.0 MB/s | 94 kB 00:00 2023-06-10T18:41:33.043 INFO:teuthology.orchestra.run.smithi114.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-06-10T18:41:33.069 INFO:teuthology.orchestra.run.smithi114.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 223 kB/s | 34 kB 00:00 2023-06-10T18:41:33.111 INFO:teuthology.orchestra.run.smithi114.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 920 kB/s | 61 kB 00:00 2023-06-10T18:41:33.136 INFO:teuthology.orchestra.run.smithi114.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 280 kB/s | 19 kB 00:00 2023-06-10T18:41:33.178 INFO:teuthology.orchestra.run.smithi114.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 358 kB/s | 24 kB 00:00 2023-06-10T18:41:33.212 INFO:teuthology.orchestra.run.smithi114.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2023-06-10T18:41:33.245 INFO:teuthology.orchestra.run.smithi114.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 409 kB/s | 27 kB 00:00 2023-06-10T18:41:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2023-06-10T18:41:33.312 INFO:teuthology.orchestra.run.smithi114.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2023-06-10T18:41:33.338 INFO:teuthology.orchestra.run.smithi114.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-06-10T18:41:33.380 INFO:teuthology.orchestra.run.smithi114.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2023-06-10T18:41:33.405 INFO:teuthology.orchestra.run.smithi114.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2023-06-10T18:41:33.422 INFO:teuthology.orchestra.run.smithi114.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-06-10T18:41:33.472 INFO:teuthology.orchestra.run.smithi114.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 939 kB/s | 47 kB 00:00 2023-06-10T18:41:33.514 INFO:teuthology.orchestra.run.smithi114.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-06-10T18:41:33.781 INFO:teuthology.orchestra.run.smithi104.stdout:(25/131): ceph-test-18.0.0-4359.g219d315c.el8.x 5.2 MB/s | 48 MB 00:09 2023-06-10T18:41:33.831 INFO:teuthology.orchestra.run.smithi104.stdout:(26/131): rbd-nbd-18.0.0-4359.g219d315c.el8.x86 3.5 MB/s | 179 kB 00:00 2023-06-10T18:41:33.873 INFO:teuthology.orchestra.run.smithi104.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4359.g 556 kB/s | 23 kB 00:00 2023-06-10T18:41:33.924 INFO:teuthology.orchestra.run.smithi104.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4359.g219d315 2.5 MB/s | 128 kB 00:00 2023-06-10T18:41:33.981 INFO:teuthology.orchestra.run.smithi114.stdout:(72/131): ceph-test-18.0.0-4359.g219d315c.el8.x 4.9 MB/s | 48 MB 00:09 2023-06-10T18:41:34.007 INFO:teuthology.orchestra.run.smithi114.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 87 kB/s | 43 kB 00:00 2023-06-10T18:41:34.057 INFO:teuthology.orchestra.run.smithi114.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 2.6 MB/s | 1.7 MB 00:00 2023-06-10T18:41:34.058 INFO:teuthology.orchestra.run.smithi104.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4359.g219d3 13 MB/s | 1.7 MB 00:00 2023-06-10T18:41:34.082 INFO:teuthology.orchestra.run.smithi114.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 690 kB/s | 48 kB 00:00 2023-06-10T18:41:34.116 INFO:teuthology.orchestra.run.smithi114.stdout:(76/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-06-10T18:41:34.124 INFO:teuthology.orchestra.run.smithi104.stdout:(30/131): rbd-mirror-18.0.0-4359.g219d315c.el8. 3.1 MB/s | 3.4 MB 00:01 2023-06-10T18:41:34.141 INFO:teuthology.orchestra.run.smithi114.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 803 kB/s | 43 kB 00:00 2023-06-10T18:41:34.166 INFO:teuthology.orchestra.run.smithi114.stdout:(78/131): libsepol-devel-2.9-3.el8.x86_64.rpm 799 kB/s | 87 kB 00:00 2023-06-10T18:41:34.192 INFO:teuthology.orchestra.run.smithi114.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 1.4 MB/s | 109 kB 00:00 2023-06-10T18:41:34.217 INFO:teuthology.orchestra.run.smithi114.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-06-10T18:41:34.233 INFO:teuthology.orchestra.run.smithi104.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4359.g21 2.2 MB/s | 240 kB 00:00 2023-06-10T18:41:34.233 INFO:teuthology.orchestra.run.smithi114.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 448 kB/s | 19 kB 00:00 2023-06-10T18:41:34.259 INFO:teuthology.orchestra.run.smithi114.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-06-10T18:41:34.293 INFO:teuthology.orchestra.run.smithi114.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.3 MB/s | 560 kB 00:00 2023-06-10T18:41:34.308 INFO:teuthology.orchestra.run.smithi104.stdout:(32/131): ceph-mgr-rook-18.0.0-4359.g219d315c.e 633 kB/s | 47 kB 00:00 2023-06-10T18:41:34.309 INFO:teuthology.orchestra.run.smithi114.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 357 kB/s | 18 kB 00:00 2023-06-10T18:41:34.343 INFO:teuthology.orchestra.run.smithi114.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 5.0 MB/s | 559 kB 00:00 2023-06-10T18:41:34.369 INFO:teuthology.orchestra.run.smithi114.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-06-10T18:41:34.383 INFO:teuthology.orchestra.run.smithi104.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4359.g2 194 kB/s | 15 kB 00:00 2023-06-10T18:41:34.402 INFO:teuthology.orchestra.run.smithi114.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.4 MB/s | 605 kB 00:00 2023-06-10T18:41:34.427 INFO:teuthology.orchestra.run.smithi114.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-06-10T18:41:34.444 INFO:teuthology.orchestra.run.smithi114.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-06-10T18:41:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.7 MB/s | 60 kB 00:00 2023-06-10T18:41:34.495 INFO:teuthology.orchestra.run.smithi114.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-06-10T18:41:34.520 INFO:teuthology.orchestra.run.smithi114.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 981 kB/s | 58 kB 00:00 2023-06-10T18:41:34.546 INFO:teuthology.orchestra.run.smithi114.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 768 kB/s | 39 kB 00:00 2023-06-10T18:41:34.567 INFO:teuthology.orchestra.run.smithi104.stdout:(34/131): ceph-mgr-diskprediction-local-18.0.0- 15 MB/s | 7.4 MB 00:00 2023-06-10T18:41:34.579 INFO:teuthology.orchestra.run.smithi114.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 11 MB/s | 644 kB 00:00 2023-06-10T18:41:34.592 INFO:teuthology.orchestra.run.smithi104.stdout:(35/131): ceph-volume-18.0.0-4359.g219d315c.el8 1.2 MB/s | 258 kB 00:00 2023-06-10T18:41:34.618 INFO:teuthology.orchestra.run.smithi104.stdout:(36/131): cephadm-18.0.0-4359.g219d315c.el8.noa 3.1 MB/s | 160 kB 00:00 2023-06-10T18:41:34.630 INFO:teuthology.orchestra.run.smithi114.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.0 MB/s | 171 kB 00:00 2023-06-10T18:41:34.651 INFO:teuthology.orchestra.run.smithi104.stdout:(37/131): cephfs-top-18.0.0-4359.g219d315c.el8. 308 kB/s | 18 kB 00:00 2023-06-10T18:41:34.663 INFO:teuthology.orchestra.run.smithi114.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.6 MB/s | 138 kB 00:00 2023-06-10T18:41:34.722 INFO:teuthology.orchestra.run.smithi114.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 6.1 MB/s | 2.3 MB 00:00 2023-06-10T18:41:34.747 INFO:teuthology.orchestra.run.smithi114.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2023-06-10T18:41:34.773 INFO:teuthology.orchestra.run.smithi114.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-06-10T18:41:34.798 INFO:teuthology.orchestra.run.smithi114.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-06-10T18:41:34.823 INFO:teuthology.orchestra.run.smithi114.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-06-10T18:41:34.848 INFO:teuthology.orchestra.run.smithi114.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 773 kB/s | 39 kB 00:00 2023-06-10T18:41:34.890 INFO:teuthology.orchestra.run.smithi114.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-06-10T18:41:34.932 INFO:teuthology.orchestra.run.smithi114.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 6.3 MB/s | 538 kB 00:00 2023-06-10T18:41:34.957 INFO:teuthology.orchestra.run.smithi114.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-06-10T18:41:34.982 INFO:teuthology.orchestra.run.smithi114.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-06-10T18:41:35.016 INFO:teuthology.orchestra.run.smithi114.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-06-10T18:41:35.058 INFO:teuthology.orchestra.run.smithi114.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 636 kB/s | 27 kB 00:00 2023-06-10T18:41:35.093 INFO:teuthology.orchestra.run.smithi104.stdout:(38/131): ceph-common-18.0.0-4359.g219d315c.el8 1.0 MB/s | 22 MB 00:21 2023-06-10T18:41:35.100 INFO:teuthology.orchestra.run.smithi114.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-06-10T18:41:35.118 INFO:teuthology.orchestra.run.smithi104.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 126 kB/s | 59 kB 00:00 2023-06-10T18:41:35.134 INFO:teuthology.orchestra.run.smithi114.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-06-10T18:41:35.219 INFO:teuthology.orchestra.run.smithi104.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2023-06-10T18:41:35.225 INFO:teuthology.orchestra.run.smithi114.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-06-10T18:41:35.276 INFO:teuthology.orchestra.run.smithi114.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-06-10T18:41:35.319 INFO:teuthology.orchestra.run.smithi104.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 436 kB/s | 44 kB 00:00 2023-06-10T18:41:35.361 INFO:teuthology.orchestra.run.smithi104.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.9 MB/s | 476 kB 00:00 2023-06-10T18:41:35.394 INFO:teuthology.orchestra.run.smithi104.stdout:(43/131): libunwind-1.3.1-3.el8.x86_64.rpm 250 kB/s | 75 kB 00:00 2023-06-10T18:41:35.419 INFO:teuthology.orchestra.run.smithi104.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-06-10T18:41:35.453 INFO:teuthology.orchestra.run.smithi104.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 358 kB/s | 33 kB 00:00 2023-06-10T18:41:35.478 INFO:teuthology.orchestra.run.smithi104.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 194 kB/s | 16 kB 00:00 2023-06-10T18:41:35.520 INFO:teuthology.orchestra.run.smithi104.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2023-06-10T18:41:35.554 INFO:teuthology.orchestra.run.smithi104.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.7 MB/s | 384 kB 00:00 2023-06-10T18:41:35.584 INFO:teuthology.orchestra.run.smithi114.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 730 kB/s | 225 kB 00:00 2023-06-10T18:41:35.629 INFO:teuthology.orchestra.run.smithi104.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-06-10T18:41:35.646 INFO:teuthology.orchestra.run.smithi104.stdout:(50/131): python3-google-auth-1.1.1-10.el8.noar 485 kB/s | 81 kB 00:00 2023-06-10T18:41:35.651 INFO:teuthology.orchestra.run.smithi114.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 7.2 MB/s | 3.7 MB 00:00 2023-06-10T18:41:35.672 INFO:teuthology.orchestra.run.smithi104.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 150 kB/s | 18 kB 00:00 2023-06-10T18:41:35.677 INFO:teuthology.orchestra.run.smithi114.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 697 kB/s | 64 kB 00:00 2023-06-10T18:41:35.702 INFO:teuthology.orchestra.run.smithi114.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-06-10T18:41:35.727 INFO:teuthology.orchestra.run.smithi114.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-06-10T18:41:35.730 INFO:teuthology.orchestra.run.smithi104.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 581 kB/s | 49 kB 00:00 2023-06-10T18:41:35.753 INFO:teuthology.orchestra.run.smithi114.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-06-10T18:41:35.755 INFO:teuthology.orchestra.run.smithi104.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 703 kB/s | 59 kB 00:00 2023-06-10T18:41:35.794 INFO:teuthology.orchestra.run.smithi114.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-06-10T18:41:35.823 INFO:teuthology.orchestra.run.smithi104.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 655 kB/s | 60 kB 00:00 2023-06-10T18:41:35.836 INFO:teuthology.orchestra.run.smithi114.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-06-10T18:41:35.881 INFO:teuthology.orchestra.run.smithi104.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 4.6 MB/s | 1.2 MB 00:00 2023-06-10T18:41:35.915 INFO:teuthology.orchestra.run.smithi104.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-06-10T18:41:35.940 INFO:teuthology.orchestra.run.smithi104.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-06-10T18:41:35.982 INFO:teuthology.orchestra.run.smithi104.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 335 kB/s | 34 kB 00:00 2023-06-10T18:41:35.987 INFO:teuthology.orchestra.run.smithi114.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 10 MB/s | 14 MB 00:01 2023-06-10T18:41:36.016 INFO:teuthology.orchestra.run.smithi104.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-06-10T18:41:36.041 INFO:teuthology.orchestra.run.smithi104.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 612 kB/s | 61 kB 00:00 2023-06-10T18:41:36.054 INFO:teuthology.orchestra.run.smithi114.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2023-06-10T18:41:36.092 INFO:teuthology.orchestra.run.smithi104.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-06-10T18:41:36.096 INFO:teuthology.orchestra.run.smithi114.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-06-10T18:41:36.116 INFO:teuthology.orchestra.run.smithi104.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2023-06-10T18:41:36.141 INFO:teuthology.orchestra.run.smithi104.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-06-10T18:41:36.188 INFO:teuthology.orchestra.run.smithi114.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 11 MB/s | 4.9 MB 00:00 2023-06-10T18:41:36.192 INFO:teuthology.orchestra.run.smithi104.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-06-10T18:41:36.217 INFO:teuthology.orchestra.run.smithi104.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.5 MB/s | 255 kB 00:00 2023-06-10T18:41:36.242 INFO:teuthology.orchestra.run.smithi104.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-06-10T18:41:36.263 INFO:teuthology.orchestra.run.smithi114.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-06-10T18:41:36.293 INFO:teuthology.orchestra.run.smithi104.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 609 kB/s | 61 kB 00:00 2023-06-10T18:41:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-06-10T18:41:36.318 INFO:teuthology.orchestra.run.smithi104.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 849 kB/s | 85 kB 00:00 2023-06-10T18:41:36.343 INFO:teuthology.orchestra.run.smithi104.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2023-06-10T18:41:36.355 INFO:teuthology.orchestra.run.smithi114.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 16 MB/s | 1.5 MB 00:00 2023-06-10T18:41:36.400 INFO:teuthology.orchestra.run.smithi104.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 848 kB/s | 70 kB 00:00 2023-06-10T18:41:36.431 INFO:teuthology.orchestra.run.smithi114.stdout:(128/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-06-10T18:41:36.461 INFO:teuthology.orchestra.run.smithi104.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.0 MB/s | 110 kB 00:00 2023-06-10T18:41:36.489 INFO:teuthology.orchestra.run.smithi114.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-06-10T18:41:36.494 INFO:teuthology.orchestra.run.smithi104.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-06-10T18:41:36.536 INFO:teuthology.orchestra.run.smithi104.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-06-10T18:41:36.578 INFO:teuthology.orchestra.run.smithi104.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-06-10T18:41:36.620 INFO:teuthology.orchestra.run.smithi104.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 932 kB/s | 39 kB 00:00 2023-06-10T18:41:36.662 INFO:teuthology.orchestra.run.smithi104.stdout:(76/131): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2023-06-10T18:41:36.704 INFO:teuthology.orchestra.run.smithi104.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.0 MB/s | 688 kB 00:00 2023-06-10T18:41:36.737 INFO:teuthology.orchestra.run.smithi104.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2023-06-10T18:41:36.762 INFO:teuthology.orchestra.run.smithi104.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-06-10T18:41:36.764 INFO:teuthology.orchestra.run.smithi114.stdout:(130/131): librbd1-18.0.0-4359.g219d315c.el8.x8 8.7 MB/s | 3.6 MB 00:00 2023-06-10T18:41:36.788 INFO:teuthology.orchestra.run.smithi104.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-06-10T18:41:36.830 INFO:teuthology.orchestra.run.smithi104.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-06-10T18:41:36.863 INFO:teuthology.orchestra.run.smithi104.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 797 kB/s | 27 kB 00:00 2023-06-10T18:41:36.873 INFO:teuthology.orchestra.run.smithi114.stdout:(131/131): librados2-18.0.0-4359.g219d315c.el8. 6.4 MB/s | 3.7 MB 00:00 2023-06-10T18:41:36.876 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2023-06-10T18:41:36.876 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.2 MB/s | 205 MB 00:22 2023-06-10T18:41:36.905 INFO:teuthology.orchestra.run.smithi104.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-06-10T18:41:36.939 INFO:teuthology.orchestra.run.smithi104.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2023-06-10T18:41:37.081 INFO:teuthology.orchestra.run.smithi104.stdout:(85/131): python3-scipy-1.0.0-21.module+el8.5.0 19 MB/s | 14 MB 00:00 2023-06-10T18:41:37.190 INFO:teuthology.orchestra.run.smithi104.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2023-06-10T18:41:37.273 INFO:teuthology.orchestra.run.smithi104.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.3 MB/s | 457 kB 00:00 2023-06-10T18:41:37.298 INFO:teuthology.orchestra.run.smithi104.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.0 MB/s | 225 kB 00:00 2023-06-10T18:41:37.315 INFO:teuthology.orchestra.run.smithi104.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-06-10T18:41:37.341 INFO:teuthology.orchestra.run.smithi104.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.9 MB/s | 126 kB 00:00 2023-06-10T18:41:37.366 INFO:teuthology.orchestra.run.smithi104.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 399 kB/s | 20 kB 00:00 2023-06-10T18:41:37.391 INFO:teuthology.orchestra.run.smithi104.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-06-10T18:41:37.425 INFO:teuthology.orchestra.run.smithi104.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-06-10T18:41:37.501 INFO:teuthology.orchestra.run.smithi104.stdout:(94/131): python3-numpy-1.14.3-10.el8.x86_64.rp 6.7 MB/s | 3.7 MB 00:00 2023-06-10T18:41:37.525 INFO:teuthology.orchestra.run.smithi104.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2023-06-10T18:41:37.567 INFO:teuthology.orchestra.run.smithi104.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-06-10T18:41:37.609 INFO:teuthology.orchestra.run.smithi104.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-06-10T18:41:37.696 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2023-06-10T18:41:37.701 INFO:teuthology.orchestra.run.smithi104.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-06-10T18:41:37.777 INFO:teuthology.orchestra.run.smithi104.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-06-10T18:41:37.810 INFO:teuthology.orchestra.run.smithi104.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-06-10T18:41:37.815 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2023-06-10T18:41:37.816 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2023-06-10T18:41:37.852 INFO:teuthology.orchestra.run.smithi104.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-06-10T18:41:37.894 INFO:teuthology.orchestra.run.smithi104.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-06-10T18:41:37.952 INFO:teuthology.orchestra.run.smithi104.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.9 MB/s | 200 kB 00:00 2023-06-10T18:41:37.978 INFO:teuthology.orchestra.run.smithi104.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-06-10T18:41:38.003 INFO:teuthology.orchestra.run.smithi104.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-06-10T18:41:38.028 INFO:teuthology.orchestra.run.smithi104.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 956 kB/s | 48 kB 00:00 2023-06-10T18:41:38.054 INFO:teuthology.orchestra.run.smithi104.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-06-10T18:41:38.079 INFO:teuthology.orchestra.run.smithi104.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 846 kB/s | 43 kB 00:00 2023-06-10T18:41:38.104 INFO:teuthology.orchestra.run.smithi104.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-06-10T18:41:38.130 INFO:teuthology.orchestra.run.smithi104.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-06-10T18:41:38.163 INFO:teuthology.orchestra.run.smithi104.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-06-10T18:41:38.230 INFO:teuthology.orchestra.run.smithi104.stdout:(112/131): postgresql-server-10.21-2.module+el8 8.2 MB/s | 5.1 MB 00:00 2023-06-10T18:41:38.255 INFO:teuthology.orchestra.run.smithi104.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 149 kB/s | 19 kB 00:00 2023-06-10T18:41:38.281 INFO:teuthology.orchestra.run.smithi104.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.3 MB/s | 155 kB 00:00 2023-06-10T18:41:38.306 INFO:teuthology.orchestra.run.smithi104.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 358 kB/s | 18 kB 00:00 2023-06-10T18:41:38.339 INFO:teuthology.orchestra.run.smithi104.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 6.5 MB/s | 559 kB 00:00 2023-06-10T18:41:38.365 INFO:teuthology.orchestra.run.smithi104.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.0 MB/s | 605 kB 00:00 2023-06-10T18:41:38.390 INFO:teuthology.orchestra.run.smithi104.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-06-10T18:41:38.415 INFO:teuthology.orchestra.run.smithi104.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-06-10T18:41:38.441 INFO:teuthology.orchestra.run.smithi104.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-06-10T18:41:38.466 INFO:teuthology.orchestra.run.smithi104.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-06-10T18:41:38.483 INFO:teuthology.orchestra.run.smithi104.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 2.3 MB/s | 98 kB 00:00 2023-06-10T18:41:38.500 INFO:teuthology.orchestra.run.smithi104.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.7 MB/s | 58 kB 00:00 2023-06-10T18:41:38.517 INFO:teuthology.orchestra.run.smithi104.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-06-10T18:41:38.550 INFO:teuthology.orchestra.run.smithi104.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 13 MB/s | 644 kB 00:00 2023-06-10T18:41:38.576 INFO:teuthology.orchestra.run.smithi104.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.8 MB/s | 171 kB 00:00 2023-06-10T18:41:38.601 INFO:teuthology.orchestra.run.smithi104.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-06-10T18:41:38.651 INFO:teuthology.orchestra.run.smithi104.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 7.5 MB/s | 2.3 MB 00:00 2023-06-10T18:41:38.718 INFO:teuthology.orchestra.run.smithi104.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-06-10T18:41:38.927 INFO:teuthology.orchestra.run.smithi104.stdout:(130/131): librbd1-18.0.0-4359.g219d315c.el8.x8 11 MB/s | 3.6 MB 00:00 2023-06-10T18:41:38.927 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2023-06-10T18:41:38.931 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2023-06-10T18:41:39.418 INFO:teuthology.orchestra.run.smithi104.stdout:(131/131): librados2-18.0.0-4359.g219d315c.el8. 4.3 MB/s | 3.7 MB 00:00 2023-06-10T18:41:39.421 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2023-06-10T18:41:39.422 INFO:teuthology.orchestra.run.smithi104.stdout:Total 8.0 MB/s | 205 MB 00:25 2023-06-10T18:41:40.223 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2023-06-10T18:41:40.343 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2023-06-10T18:41:40.343 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2023-06-10T18:41:41.459 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2023-06-10T18:41:41.462 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2023-06-10T18:41:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2023-06-10T18:41:44.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-06-10T18:41:44.534 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-06-10T18:41:44.703 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-10T18:41:44.826 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-10T18:41:45.028 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-10T18:41:45.426 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-10T18:41:45.645 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:18.0.0-4359.g219d315c.el8.x86_64 5/134 2023-06-10T18:41:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:18.0.0-4359.g219d315c.el8.x86_64 5/134 2023-06-10T18:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:41:46.355 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:41:46.521 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2023-06-10T18:41:46.544 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:41:46.669 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:41:46.690 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-06-10T18:41:46.853 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-06-10T18:41:46.866 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 8/134 2023-06-10T18:41:47.062 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-06-10T18:41:47.082 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-06-10T18:41:47.292 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-06-10T18:41:47.307 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-06-10T18:41:47.414 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-06-10T18:41:47.530 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-06-10T18:41:47.617 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-06-10T18:41:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-06-10T18:41:48.016 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-06-10T18:41:48.217 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:41:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-06-10T18:41:48.427 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 14/134 2023-06-10T18:41:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 14/134 2023-06-10T18:41:48.610 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:41:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 15/134 2023-06-10T18:41:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:41:48.878 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 15/134 2023-06-10T18:41:48.928 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:41:49.056 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-06-10T18:41:49.108 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 8/134 2023-06-10T18:41:49.191 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 8/134 2023-06-10T18:41:49.225 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-06-10T18:41:49.392 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 9/134 2023-06-10T18:41:49.448 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-06-10T18:41:49.594 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-10T18:41:49.654 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-06-10T18:41:49.735 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-10T18:41:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-06-10T18:41:49.894 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-10T18:41:50.248 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-06-10T18:41:50.458 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-06-10T18:41:50.508 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-10T18:41:50.600 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 14/134 2023-06-10T18:41:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-10T18:41:50.711 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 14/134 2023-06-10T18:41:50.866 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 15/134 2023-06-10T18:41:50.946 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 15/134 2023-06-10T18:41:51.082 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-06-10T18:41:51.214 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-06-10T18:41:51.282 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-10T18:41:51.358 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-06-10T18:41:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-06-10T18:41:51.954 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-06-10T18:41:52.124 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-06-10T18:41:52.216 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-06-10T18:41:52.511 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-06-10T18:41:52.716 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-06-10T18:41:52.719 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-06-10T18:41:52.899 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-06-10T18:41:52.978 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-06-10T18:41:53.166 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:18.0.0-4359.g219d315c.el8. 26/134 2023-06-10T18:41:53.313 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:18.0.0-4359.g219d315c.el 27/134 2023-06-10T18:41:53.589 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_6 28/134 2023-06-10T18:41:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-06-10T18:41:53.818 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-10T18:41:53.855 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-06-10T18:41:54.067 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-06-10T18:41:54.261 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-06-10T18:41:54.413 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-06-10T18:41:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-06-10T18:41:54.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-06-10T18:41:54.988 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:18.0.0-4359.g219d315c.el8. 26/134 2023-06-10T18:41:55.118 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:18.0.0-4359.g219d315c.el 27/134 2023-06-10T18:41:55.567 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-10T18:41:56.004 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_6 28/134 2023-06-10T18:41:56.295 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-06-10T18:41:56.567 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-06-10T18:41:56.609 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-06-10T18:41:56.792 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-06-10T18:41:56.965 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-06-10T18:41:57.110 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-06-10T18:41:57.411 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-06-10T18:41:57.608 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-06-10T18:41:57.821 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-06-10T18:41:58.036 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-06-10T18:41:58.215 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 36/134 2023-06-10T18:41:58.374 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:18.0.0-4359.g219d315c.el8.x86_6 37/134 2023-06-10T18:41:58.525 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-06-10T18:41:58.702 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-06-10T18:41:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-06-10T18:41:58.884 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-06-10T18:41:58.952 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-06-10T18:41:58.999 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-06-10T18:41:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-06-10T18:41:59.209 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-06-10T18:41:59.375 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-06-10T18:41:59.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-06-10T18:41:59.561 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 36/134 2023-06-10T18:41:59.567 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-10T18:41:59.647 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-10T18:41:59.710 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:18.0.0-4359.g219d315c.el8.x86_6 37/134 2023-06-10T18:41:59.853 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-06-10T18:41:59.888 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-06-10T18:42:00.016 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-06-10T18:42:00.088 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-06-10T18:42:00.225 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-06-10T18:42:00.226 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-06-10T18:42:00.359 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-06-10T18:42:00.386 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-06-10T18:42:00.553 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-06-10T18:42:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-06-10T18:42:00.712 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-06-10T18:42:00.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-06-10T18:42:00.890 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-06-10T18:42:01.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-06-10T18:42:01.169 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-06-10T18:42:01.442 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-06-10T18:42:01.462 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-06-10T18:42:01.632 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-06-10T18:42:01.752 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-06-10T18:42:01.842 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-06-10T18:42:01.970 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-06-10T18:42:02.094 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-06-10T18:42:02.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-06-10T18:42:02.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-06-10T18:42:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-06-10T18:42:02.647 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-06-10T18:42:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-06-10T18:42:02.824 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-06-10T18:42:02.983 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-06-10T18:42:03.035 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-06-10T18:42:03.205 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-06-10T18:42:03.215 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-06-10T18:42:03.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-06-10T18:42:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-06-10T18:42:03.751 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-06-10T18:42:03.811 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-06-10T18:42:04.077 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-06-10T18:42:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-06-10T18:42:04.327 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-06-10T18:42:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-06-10T18:42:04.542 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-06-10T18:42:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-06-10T18:42:04.736 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-06-10T18:42:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-06-10T18:42:04.920 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-06-10T18:42:05.066 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-06-10T18:42:05.072 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-06-10T18:42:05.263 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-06-10T18:42:05.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-06-10T18:42:05.323 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-06-10T18:42:05.541 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-06-10T18:42:05.543 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-06-10T18:42:05.741 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-06-10T18:42:05.790 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-06-10T18:42:05.880 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-06-10T18:42:05.959 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-06-10T18:42:05.971 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-06-10T18:42:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-06-10T18:42:06.166 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-06-10T18:42:06.322 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-06-10T18:42:06.358 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-06-10T18:42:06.504 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-06-10T18:42:06.569 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-06-10T18:42:06.665 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-06-10T18:42:06.760 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-06-10T18:42:06.912 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-06-10T18:42:06.928 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-06-10T18:42:07.098 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-06-10T18:42:07.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-06-10T18:42:07.284 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-06-10T18:42:07.339 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-06-10T18:42:07.431 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-06-10T18:42:07.595 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-06-10T18:42:07.612 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-06-10T18:42:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-06-10T18:42:07.839 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-06-10T18:42:07.955 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-06-10T18:42:08.090 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-06-10T18:42:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-06-10T18:42:08.209 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-06-10T18:42:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-06-10T18:42:08.387 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-06-10T18:42:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-06-10T18:42:08.565 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-06-10T18:42:08.767 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-06-10T18:42:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-06-10T18:42:09.022 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-06-10T18:42:09.228 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-06-10T18:42:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-06-10T18:42:09.600 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-06-10T18:42:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-06-10T18:42:10.103 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-06-10T18:42:10.340 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-06-10T18:42:10.640 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-06-10T18:42:10.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-06-10T18:42:10.857 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-06-10T18:42:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-06-10T18:42:11.120 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-06-10T18:42:11.179 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-06-10T18:42:11.394 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-06-10T18:42:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-06-10T18:42:11.880 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-06-10T18:42:11.996 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-06-10T18:42:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 92/134 2023-06-10T18:42:12.255 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 92/134 2023-06-10T18:42:12.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 93/134 2023-06-10T18:42:12.676 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-06-10T18:42:12.870 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 92/134 2023-06-10T18:42:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 92/134 2023-06-10T18:42:13.095 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 93/134 2023-06-10T18:42:16.031 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:16.274 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:16.733 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:16.853 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:16.919 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:17.129 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 95/134 2023-06-10T18:42:17.215 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 95/134 2023-06-10T18:42:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 96/134 2023-06-10T18:42:17.503 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 94/134 2023-06-10T18:42:17.760 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 95/134 2023-06-10T18:42:17.845 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 95/134 2023-06-10T18:42:18.020 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 96/134 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 96/134 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /dev 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /run 2023-06-10T18:42:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2023-06-10T18:42:35.850 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2023-06-10T18:42:35.850 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2023-06-10T18:42:35.850 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2023-06-10T18:42:35.850 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /dev 2023-06-10T18:42:35.850 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 97/134 2023-06-10T18:42:36.811 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 96/134 2023-06-10T18:42:36.812 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2023-06-10T18:42:36.812 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2023-06-10T18:42:36.812 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2023-06-10T18:42:36.812 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /run 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2023-06-10T18:42:36.813 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:42:37.131 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 97/134 2023-06-10T18:42:37.778 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 97/134 2023-06-10T18:42:37.943 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 98/134 2023-06-10T18:42:38.160 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 98/134 2023-06-10T18:42:38.321 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 99/134 2023-06-10T18:42:38.711 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 97/134 2023-06-10T18:42:38.772 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 99/134 2023-06-10T18:42:38.889 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 98/134 2023-06-10T18:42:39.004 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 100/134 2023-06-10T18:42:39.021 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 100/134 2023-06-10T18:42:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 98/134 2023-06-10T18:42:39.273 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 99/134 2023-06-10T18:42:39.725 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 99/134 2023-06-10T18:42:39.948 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 100/134 2023-06-10T18:42:39.964 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 100/134 2023-06-10T18:42:40.636 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:18.0.0-4359.g219d315c.el8.noarch 101/134 2023-06-10T18:42:40.837 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:18.0.0-4359.g219d315c.el8.noarch 101/134 2023-06-10T18:42:40.959 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.e 102/134 2023-06-10T18:42:41.143 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4359.g219d315c. 103/134 2023-06-10T18:42:41.354 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 104/134 2023-06-10T18:42:41.569 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 104/134 2023-06-10T18:42:41.780 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:18.0.0-4359.g219d315c.el8.noarch 101/134 2023-06-10T18:42:41.892 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 105/134 2023-06-10T18:42:41.957 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:18.0.0-4359.g219d315c.el8.noarch 101/134 2023-06-10T18:42:42.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.e 102/134 2023-06-10T18:42:42.304 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4359.g219d315c. 103/134 2023-06-10T18:42:42.481 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 104/134 2023-06-10T18:42:42.693 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 104/134 2023-06-10T18:42:42.848 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 105/134 2023-06-10T18:42:42.986 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 106/134 2023-06-10T18:42:43.054 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 105/134 2023-06-10T18:42:43.076 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 106/134 2023-06-10T18:42:43.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el 107/134 2023-06-10T18:42:43.651 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 108/134 2023-06-10T18:42:43.686 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 108/134 2023-06-10T18:42:43.862 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 109/134 2023-06-10T18:42:43.873 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 109/134 2023-06-10T18:42:44.012 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 105/134 2023-06-10T18:42:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 106/134 2023-06-10T18:42:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 106/134 2023-06-10T18:42:44.629 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el 107/134 2023-06-10T18:42:44.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 108/134 2023-06-10T18:42:44.814 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 108/134 2023-06-10T18:42:44.963 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 109/134 2023-06-10T18:42:44.974 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 109/134 2023-06-10T18:42:45.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:18.0.0-4359.g219d315c.el8.x86_64 110/134 2023-06-10T18:42:45.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 111/134 2023-06-10T18:42:45.467 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 111/134 2023-06-10T18:42:45.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4359.g219d3 112/134 2023-06-10T18:42:45.661 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4359.g219d3 112/134 2023-06-10T18:42:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 113/134 2023-06-10T18:42:46.155 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 113/134 2023-06-10T18:42:46.244 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:18.0.0-4359.g219d315c.el8.x86_64 110/134 2023-06-10T18:42:46.320 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 114/134 2023-06-10T18:42:46.433 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 111/134 2023-06-10T18:42:46.467 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 111/134 2023-06-10T18:42:46.621 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4359.g219d3 112/134 2023-06-10T18:42:46.663 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4359.g219d3 112/134 2023-06-10T18:42:46.804 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 113/134 2023-06-10T18:42:47.149 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 113/134 2023-06-10T18:42:47.288 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 114/134 2023-06-10T18:42:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 114/134 2023-06-10T18:42:51.741 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 115/134 2023-06-10T18:42:51.786 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 114/134 2023-06-10T18:42:52.053 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 115/134 2023-06-10T18:42:52.058 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-06-10T18:42:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-06-10T18:42:52.257 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 117/134 2023-06-10T18:42:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-06-10T18:42:52.495 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-06-10T18:42:52.652 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-06-10T18:42:52.863 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-06-10T18:42:53.106 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-06-10T18:42:53.268 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-06-10T18:42:53.298 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_ 121/134 2023-06-10T18:42:53.436 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-06-10T18:42:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-06-10T18:42:53.754 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 119/134 2023-06-10T18:42:53.948 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-06-10T18:42:54.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_ 121/134 2023-06-10T18:42:54.262 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-06-10T18:42:55.536 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:55.985 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 124/134 2023-06-10T18:42:56.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 125/134 2023-06-10T18:42:56.443 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:56.571 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 126/134 2023-06-10T18:42:56.710 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:56.727 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-10T18:42:56.877 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 124/134 2023-06-10T18:42:56.957 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-06-10T18:42:57.143 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-06-10T18:42:57.185 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 125/134 2023-06-10T18:42:57.223 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-06-10T18:42:57.426 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-06-10T18:42:57.439 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-06-10T18:42:57.483 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 126/134 2023-06-10T18:42:57.593 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-10T18:42:57.601 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-10T18:42:57.737 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-06-10T18:42:57.737 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:57.770 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-06-10T18:42:57.918 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-06-10T18:42:57.927 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-06-10T18:42:57.938 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:57.938 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-10T18:42:58.101 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-06-10T18:42:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-10T18:42:58.310 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-06-10T18:42:58.375 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:42:58.394 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-06-10T18:42:58.593 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-06-10T18:42:58.606 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-06-10T18:42:58.606 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:58.654 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:42:58.655 INFO:teuthology.orchestra.run.smithi114.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-10T18:42:58.655 INFO:teuthology.orchestra.run.smithi114.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-10T18:42:58.655 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:42:58.788 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:58.788 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-10T18:42:58.871 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:18.0.0-4359.g219d315c.el8.x86_64 1/134 2023-06-10T18:42:58.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 2/134 2023-06-10T18:42:58.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 3/134 2023-06-10T18:42:58.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 4/134 2023-06-10T18:42:58.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4359.g219d3 5/134 2023-06-10T18:42:58.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 8/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 9/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 10/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 11/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 12/134 2023-06-10T18:42:58.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 13/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_ 14/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 15/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 16/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:18.0.0-4359.g219d315c.el8.x86_6 17/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 18/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 19/134 2023-06-10T18:42:58.887 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4359.g219d315c.el 20/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:18.0.0-4359.g219d315c.el8. 21/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_6 22/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 23/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 24/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 25/134 2023-06-10T18:42:58.888 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 26/134 2023-06-10T18:42:58.895 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 27/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 28/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4359.g219d315c. 29/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 30/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 31/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 32/134 2023-06-10T18:42:58.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el 33/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 34/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.e 35/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 36/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:18.0.0-4359.g219d315c.el8.noarch 37/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 38/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-06-10T18:42:58.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-06-10T18:42:58.898 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-06-10T18:42:58.899 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-06-10T18:42:58.900 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-06-10T18:42:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-06-10T18:42:58.902 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-06-10T18:42:58.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-06-10T18:42:58.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-06-10T18:42:58.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-06-10T18:42:58.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-06-10T18:42:58.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-06-10T18:42:58.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-06-10T18:42:58.905 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-06-10T18:42:58.906 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-06-10T18:42:58.907 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-06-10T18:42:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-06-10T18:42:58.909 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:18.0.0-4359.g219d315c.el8.x86_64 129/134 2023-06-10T18:42:58.910 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-06-10T18:42:58.911 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 131/134 2023-06-10T18:42:58.911 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:58.911 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-06-10T18:42:59.024 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-10T18:42:59.192 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:42:59.456 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:42:59.457 INFO:teuthology.orchestra.run.smithi104.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-10T18:42:59.457 INFO:teuthology.orchestra.run.smithi104.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-10T18:42:59.457 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:42:59.677 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:18.0.0-4359.g219d315c.el8.x86_64 1/134 2023-06-10T18:42:59.677 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 2/134 2023-06-10T18:42:59.677 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 3/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 4/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4359.g219d3 5/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 6/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 7/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 8/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 9/134 2023-06-10T18:42:59.678 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 10/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 11/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 12/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 13/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_ 14/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 15/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 16/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:18.0.0-4359.g219d315c.el8.x86_6 17/134 2023-06-10T18:42:59.679 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86 18/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 19/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4359.g219d315c.el 20/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:18.0.0-4359.g219d315c.el8. 21/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_6 22/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 23/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 24/134 2023-06-10T18:42:59.680 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 25/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 26/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 27/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 28/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4359.g219d315c. 29/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noa 30/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.n 31/134 2023-06-10T18:42:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4359.g219 32/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el 33/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 34/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.e 35/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 36/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:18.0.0-4359.g219d315c.el8.noarch 37/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 38/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-06-10T18:42:59.682 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-06-10T18:42:59.683 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-06-10T18:42:59.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-06-10T18:42:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-06-10T18:42:59.686 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-06-10T18:42:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-06-10T18:42:59.688 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-06-10T18:42:59.689 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-06-10T18:42:59.690 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-06-10T18:42:59.691 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-06-10T18:42:59.692 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-06-10T18:42:59.693 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:18.0.0-4359.g219d315c.el8.x86_64 129/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 131/134 2023-06-10T18:42:59.694 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-10T18:42:59.695 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-06-10T18:43:01.569 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:43:01.569 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:59 ago on Sat 10 Jun 2023 06:41:02 PM UTC. 2023-06-10T18:43:02.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-10T18:43:02.392 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:00 ago on Sat 10 Jun 2023 06:41:02 PM UTC. 2023-06-10T18:43:04.247 INFO:teuthology.orchestra.run.smithi114.stdout:Installed products updated. 2023-06-10T18:43:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2023-06-10T18:43:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.1-7.el8.noarch 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.0.4-10.el8.x86_64 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.1-9.el8.x86_64 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.7-15.el8.x86_64 2023-06-10T18:43:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-10T18:43:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-3.el8.x86_64 2023-06-10T18:43:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-10T18:43:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-06-10T18:43:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-10T18:43:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-06-10T18:43:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-06-10T18:43:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-10T18:43:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: libpq-13.5-1.el8.x86_64 2023-06-10T18:43:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-06-10T18:43:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-10T18:43:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-10T18:43:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-10T18:43:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-10T18:43:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-10T18:43:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-10T18:43:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-06-10T18:43:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-10T18:43:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.15-3.el8.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-10T18:43:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-10T18:43:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-10T18:43:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-06-10T18:43:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-10T18:43:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-10T18:43:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-06-10T18:43:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-10T18:43:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-10T18:43:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-10T18:43:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-2.6-1.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-10T18:43:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-10T18:43:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-10T18:43:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-10T18:43:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-10T18:43:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-43.el8.x86_64 2023-06-10T18:43:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-06-10T18:43:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-10T18:43:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-06-10T18:43:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:04.494 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-06-10T18:43:04.667 DEBUG:teuthology.parallel:result is None 2023-06-10T18:43:05.143 INFO:teuthology.orchestra.run.smithi104.stdout:Installed products updated. 2023-06-10T18:43:05.367 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2023-06-10T18:43:05.368 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-06-10T18:43:05.369 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.1-7.el8.noarch 2023-06-10T18:43:05.369 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.0.4-10.el8.x86_64 2023-06-10T18:43:05.369 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.369 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.369 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.370 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.371 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.372 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.372 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.372 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.372 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:18.0.0-4359.g219d315c.el8.noarch 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.1-9.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.7-15.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-3.el8.x86_64 2023-06-10T18:43:05.373 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-06-10T18:43:05.374 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: libpq-13.5-1.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-10T18:43:05.375 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-06-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.15-3.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-10T18:43:05.377 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-10T18:43:05.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-06-10T18:43:05.379 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-10T18:43:05.380 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-10T18:43:05.380 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-06-10T18:43:05.380 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-10T18:43:05.381 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-10T18:43:05.381 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-10T18:43:05.381 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-10T18:43:05.381 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-10T18:43:05.382 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-2.6-1.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-10T18:43:05.383 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.384 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-10T18:43:05.385 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-10T18:43:05.386 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-10T18:43:05.387 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-10T18:43:05.387 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.387 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.387 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:18.0.0-4359.g219d315c.el8.x86_64 2023-06-10T18:43:05.387 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-10T18:43:05.388 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-10T18:43:05.388 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-43.el8.x86_64 2023-06-10T18:43:05.388 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-06-10T18:43:05.388 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-10T18:43:05.388 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-06-10T18:43:05.389 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:05.389 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2023-06-10T18:43:05.538 DEBUG:teuthology.parallel:result is None 2023-06-10T18:43:05.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=219d315c3a0b2075e570fb1964be2ad33fe7a9c8 2023-06-10T18:43:05.762 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:43:05.785 INFO:teuthology.orchestra.run.smithi104.stdout:18.0.0-4359.g219d315c.el8 2023-06-10T18:43:05.786 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4359.g219d315c.el8 2023-06-10T18:43:05.786 INFO:teuthology.task.install:The correct ceph version 18.0.0-4359.g219d315c is installed. 2023-06-10T18:43:05.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=219d315c3a0b2075e570fb1964be2ad33fe7a9c8 2023-06-10T18:43:06.014 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:43:06.042 INFO:teuthology.orchestra.run.smithi114.stdout:18.0.0-4359.g219d315c.el8 2023-06-10T18:43:06.042 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4359.g219d315c.el8 2023-06-10T18:43:06.043 INFO:teuthology.task.install:The correct ceph version 18.0.0-4359.g219d315c is installed. 2023-06-10T18:43:06.043 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-10T18:43:06.043 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:06.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-10T18:43:06.084 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:06.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-10T18:43:06.129 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-10T18:43:06.129 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:06.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2023-06-10T18:43:06.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-10T18:43:06.220 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:06.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2023-06-10T18:43:06.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-10T18:43:06.317 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-10T18:43:06.317 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:06.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-10T18:43:06.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-10T18:43:06.409 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:06.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-10T18:43:06.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-10T18:43:06.505 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-10T18:43:06.506 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:06.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2023-06-10T18:43:06.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-10T18:43:06.597 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:06.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2023-06-10T18:43:06.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-10T18:43:06.692 INFO:teuthology.run_tasks:Running task ceph... 2023-06-10T18:43:06.759 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-10T18:43:06.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2023-06-10T18:43:06.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2023-06-10T18:43:06.788 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-10T18:43:06.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-10T18:43:06.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-10T18:43:06.855 INFO:tasks.ceph:Creating extra log directories... 2023-06-10T18:43:06.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-10T18:43:06.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-10T18:43:06.926 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-10T18:43:06.927 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-10T18:43:06.927 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-10T18:43:06.927 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:06.928 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-10T18:43:06.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:07.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:07.131 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:07.131 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-10T18:43:07.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:07.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:07.321 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-10T18:43:07.322 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'but it is still running', 'slow request', 'MDS_CLIENT_LATE_RELEASE', 't responding to mclientcaps', 'file system flag refuse_client_session is set'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-10T18:43:07.322 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167', 'branch': 'main', 'description': 'fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7301167', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'but it is still running', 'slow request', 'MDS_CLIENT_LATE_RELEASE', 't responding to mclientcaps', 'file system flag refuse_client_session is set'], 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'i61409', 'sha1': '3486dd872ffd89042ca079d680841ce3ef6f4b6f'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5151, 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8', 'sleep_before_teardown': 0, 'suite': 'fs:functional', 'suite_branch': 'i61409', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph.git', 'suite_sha1': '3486dd872ffd89042ca079d680841ce3ef6f4b6f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi104.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '219d315c3a0b2075e570fb1964be2ad33fe7a9c8'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_client_recovery']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61', 'timestamp': '2023-06-10_18:20:34', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-10T18:43:07.323 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-10T18:43:07.325 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-10T18:43:07.371 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2023-06-10T18:43:07.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2023-06-10T18:43:07.438 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:07.439 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2023-06-10T18:43:07.463 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-10T18:43:07.464 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2023-06-10T18:43:07.520 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-10T18:43:07.520 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:07.520 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 371464 Links: 1 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-06-10 18:42:59.372763895 +0000 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-06-10 18:40:06.290532752 +0000 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-06-10 18:40:06.290532752 +0000 2023-06-10T18:43:07.521 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-06-10T18:43:07.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-10T18:43:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-06-10T18:43:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-06-10T18:43:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000249577 s, 2.1 MB/s 2023-06-10T18:43:07.586 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-10T18:43:07.643 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2023-06-10T18:43:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-10T18:43:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 377991 Links: 1 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-06-10 18:42:59.372763895 +0000 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-06-10 18:40:06.811526767 +0000 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-06-10 18:40:06.811526767 +0000 2023-06-10T18:43:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-06-10T18:43:07.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-10T18:43:07.765 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-06-10T18:43:07.765 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-06-10T18:43:07.765 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000216965 s, 2.4 MB/s 2023-06-10T18:43:07.766 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-10T18:43:07.823 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2023-06-10T18:43:07.880 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-10T18:43:07.880 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:07.880 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 371685 Links: 1 2023-06-10T18:43:07.880 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:07.881 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:07.881 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-06-10 18:42:59.372763895 +0000 2023-06-10T18:43:07.881 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-06-10 18:40:07.330520804 +0000 2023-06-10T18:43:07.881 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-06-10 18:40:07.330520804 +0000 2023-06-10T18:43:07.881 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-06-10T18:43:07.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-10T18:43:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-06-10T18:43:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-06-10T18:43:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000225423 s, 2.3 MB/s 2023-06-10T18:43:07.946 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-10T18:43:08.004 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2023-06-10T18:43:08.060 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-10T18:43:08.061 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:08.061 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 379950 Links: 1 2023-06-10T18:43:08.061 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:08.061 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:08.061 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-06-10 18:42:59.372763895 +0000 2023-06-10T18:43:08.062 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-06-10 18:40:07.837514979 +0000 2023-06-10T18:43:08.062 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-06-10 18:40:07.837514979 +0000 2023-06-10T18:43:08.062 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-06-10T18:43:08.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-10T18:43:08.126 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-06-10T18:43:08.127 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-06-10T18:43:08.127 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000190882 s, 2.7 MB/s 2023-06-10T18:43:08.128 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-10T18:43:08.185 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:08.186 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:08.186 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2023-06-10T18:43:08.204 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-10T18:43:08.205 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2023-06-10T18:43:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-10T18:43:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:08.263 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 375178 Links: 1 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:42:58.572854414 +0000 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:40:14.023541424 +0000 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:40:14.023541424 +0000 2023-06-10T18:43:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:43:08.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-10T18:43:08.332 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-06-10T18:43:08.332 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-06-10T18:43:08.332 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000334924 s, 1.5 MB/s 2023-06-10T18:43:08.333 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-10T18:43:08.396 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2023-06-10T18:43:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-10T18:43:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 372554 Links: 1 2023-06-10T18:43:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:42:58.572854414 +0000 2023-06-10T18:43:08.457 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:40:14.539529926 +0000 2023-06-10T18:43:08.457 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:40:14.539529926 +0000 2023-06-10T18:43:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:43:08.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-10T18:43:08.523 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-06-10T18:43:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-06-10T18:43:08.524 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000234681 s, 2.2 MB/s 2023-06-10T18:43:08.525 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-10T18:43:08.584 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2023-06-10T18:43:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 379995 Links: 1 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:42:58.573854392 +0000 2023-06-10T18:43:08.643 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:40:15.044518673 +0000 2023-06-10T18:43:08.644 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:40:15.044518673 +0000 2023-06-10T18:43:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:43:08.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-10T18:43:08.709 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-06-10T18:43:08.710 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-06-10T18:43:08.710 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000221357 s, 2.3 MB/s 2023-06-10T18:43:08.711 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-10T18:43:08.770 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2023-06-10T18:43:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 375391 Links: 1 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:42:58.573854392 +0000 2023-06-10T18:43:08.828 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:40:15.547507464 +0000 2023-06-10T18:43:08.829 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:40:15.547507464 +0000 2023-06-10T18:43:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:43:08.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-10T18:43:08.895 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-06-10T18:43:08.895 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-06-10T18:43:08.895 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000186453 s, 2.7 MB/s 2023-06-10T18:43:08.896 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-10T18:43:08.954 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:08.955 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-10T18:43:08.955 INFO:tasks.ceph:Generating config... 2023-06-10T18:43:08.956 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-10T18:43:08.956 INFO:tasks.ceph:[client] debug client = 20 2023-06-10T18:43:08.957 INFO:tasks.ceph:[client] debug ms = 1 2023-06-10T18:43:08.957 INFO:tasks.ceph:[client] pid file = 2023-06-10T18:43:08.957 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-10T18:43:08.957 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-10T18:43:08.957 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-10T18:43:08.958 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-06-10T18:43:08.958 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-10T18:43:08.958 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-10T18:43:08.958 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-10T18:43:08.958 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-10T18:43:08.959 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-10T18:43:08.959 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-10T18:43:08.959 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-10T18:43:08.959 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-10T18:43:08.959 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-10T18:43:08.960 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-10T18:43:08.960 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-10T18:43:08.960 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-10T18:43:08.960 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-10T18:43:08.960 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-10T18:43:08.961 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-10T18:43:08.961 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-10T18:43:08.961 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-06-10T18:43:08.961 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-06-10T18:43:08.961 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-10T18:43:08.962 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-10T18:43:08.963 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-10T18:43:08.963 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-10T18:43:08.963 INFO:tasks.ceph:Setting up mon.a... 2023-06-10T18:43:08.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-10T18:43:09.034 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.keyring 2023-06-10T18:43:09.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-10T18:43:09.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-10T18:43:09.109 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.104'), ('mon.b', '172.21.15.114'), ('mon.c', '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]')] 2023-06-10T18:43:09.109 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.104,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-10T18:43:09.110 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:09.111 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-10T18:43:09.166 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.104 --add b 172.21.15.114 --addv c '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:09.254 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: generated fsid f08e2e30-012f-4260-8ced-2a652c595f11 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:setting min_mon_release = pacific 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 0 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:fsid f08e2e30-012f-4260-8ced-2a652c595f11 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:last_changed 2023-06-10T18:43:09.250650+0000 2023-06-10T18:43:09.255 INFO:teuthology.orchestra.run.smithi104.stdout:created 2023-06-10T18:43:09.250650+0000 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:min_mon_release 16 (pacific) 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:election_strategy: 1 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.a 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2023-06-10T18:43:09.256 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-10T18:43:09.257 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-10T18:43:09.273 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f08e2e30-012f-4260-8ced-2a652c595f11... 2023-06-10T18:43:09.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-10T18:43:09.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-10T18:43:09.360 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2023-06-10T18:43:09.360 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2023-06-10T18:43:09.360 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-10T18:43:09.360 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2023-06-10T18:43:09.360 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2023-06-10T18:43:09.361 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2023-06-10T18:43:09.362 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2023-06-10T18:43:09.363 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2023-06-10T18:43:09.364 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2023-06-10T18:43:09.364 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2023-06-10T18:43:09.364 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2023-06-10T18:43:09.364 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2023-06-10T18:43:09.364 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-10T18:43:09.365 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-10T18:43:09.365 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2023-06-10T18:43:09.365 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.365 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-10T18:43:09.365 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.366 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2023-06-10T18:43:09.366 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.366 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2023-06-10T18:43:09.366 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.367 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2023-06-10T18:43:09.367 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2023-06-10T18:43:09.367 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2023-06-10T18:43:09.367 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.104,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2023-06-10T18:43:09.367 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = f08e2e30-012f-4260-8ced-2a652c595f11 2023-06-10T18:43:09.368 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.368 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2023-06-10T18:43:09.368 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2023-06-10T18:43:09.368 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.368 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2023-06-10T18:43:09.369 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.370 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2023-06-10T18:43:09.370 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2023-06-10T18:43:09.370 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2023-06-10T18:43:09.370 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.371 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2023-06-10T18:43:09.371 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2023-06-10T18:43:09.371 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.371 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2023-06-10T18:43:09.371 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.372 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2023-06-10T18:43:09.372 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.372 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2023-06-10T18:43:09.372 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.373 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2023-06-10T18:43:09.373 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2023-06-10T18:43:09.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.373 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2023-06-10T18:43:09.373 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2023-06-10T18:43:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2023-06-10T18:43:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2023-06-10T18:43:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2023-06-10T18:43:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2023-06-10T18:43:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2023-06-10T18:43:09.375 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2023-06-10T18:43:09.375 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2023-06-10T18:43:09.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2023-06-10T18:43:09.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2023-06-10T18:43:09.376 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2023-06-10T18:43:09.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2023-06-10T18:43:09.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2023-06-10T18:43:09.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd op complaint time = 180 2023-06-10T18:43:09.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.377 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2023-06-10T18:43:09.377 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2023-06-10T18:43:09.377 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2023-06-10T18:43:09.377 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2023-06-10T18:43:09.378 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2023-06-10T18:43:09.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2023-06-10T18:43:09.378 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2023-06-10T18:43:09.378 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2023-06-10T18:43:09.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.379 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2023-06-10T18:43:09.379 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2023-06-10T18:43:09.379 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2023-06-10T18:43:09.379 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2023-06-10T18:43:09.379 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2023-06-10T18:43:09.380 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon op complaint time = 120 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2023-06-10T18:43:09.383 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2023-06-10T18:43:09.384 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-10T18:43:09.384 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: client mount timeout = 600 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: debug client = 20 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = "" 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: rados mon op timeout = 900 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout: rados osd op timeout = 900 2023-06-10T18:43:09.385 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.a] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.c] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.b] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.d] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mds] 2023-06-10T18:43:09.386 INFO:teuthology.orchestra.run.smithi104.stdout: debug mds = 20 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: debug mds balancer = 20 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: mds debug frag = True 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: mds debug scatterstat = True 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: mds op complaint time = 180 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: mds verify scatter = True 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: osd op complaint time = 180 2023-06-10T18:43:09.387 INFO:teuthology.orchestra.run.smithi104.stdout: rados mon op timeout = 900 2023-06-10T18:43:09.388 INFO:teuthology.orchestra.run.smithi104.stdout: rados osd op timeout = 900 2023-06-10T18:43:09.389 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2023-06-10T18:43:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2023-06-10T18:43:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2023-06-10T18:43:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2023-06-10T18:43:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-10T18:43:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.104,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = f08e2e30-012f-4260-8ced-2a652c595f11 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2023-06-10T18:43:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2023-06-10T18:43:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2023-06-10T18:43:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2023-06-10T18:43:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2023-06-10T18:43:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2023-06-10T18:43:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2023-06-10T18:43:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2023-06-10T18:43:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2023-06-10T18:43:09.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2023-06-10T18:43:09.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2023-06-10T18:43:09.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2023-06-10T18:43:09.404 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.405 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-10T18:43:09.405 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2023-06-10T18:43:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2023-06-10T18:43:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2023-06-10T18:43:09.409 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2023-06-10T18:43:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2023-06-10T18:43:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2023-06-10T18:43:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2023-06-10T18:43:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2023-06-10T18:43:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2023-06-10T18:43:09.411 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-10T18:43:09.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-10T18:43:09.485 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-10T18:43:09.486 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:09.486 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-10T18:43:09.502 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:09.502 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-10T18:43:09.558 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:43:09.559 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:09.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-10T18:43:09.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-10T18:43:09.634 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:09.634 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:09.690 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:43:09.691 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:09.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-10T18:43:09.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-10T18:43:09.734 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:09.734 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:09.792 INFO:tasks.ceph:Setting up mon nodes... 2023-06-10T18:43:09.793 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-10T18:43:09.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-10T18:43:09.846 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-10T18:43:09.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-10T18:43:09.914 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-10T18:43:09.917 INFO:tasks.ceph:Setting up mds nodes... 2023-06-10T18:43:09.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-10T18:43:09.970 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-10T18:43:09.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-10T18:43:09.998 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-10T18:43:10.091 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-10T18:43:10.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-10T18:43:10.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-10T18:43:10.175 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-10T18:43:10.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-10T18:43:10.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-10T18:43:10.298 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-10T18:43:10.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-10T18:43:10.326 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-10T18:43:10.326 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-10T18:43:10.369 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-10T18:43:10.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-10T18:43:10.465 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-10T18:43:10.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-10T18:43:10.520 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-10T18:43:10.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-10T18:43:10.616 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-10T18:43:10.628 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-10T18:43:10.629 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-10T18:43:10.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-10T18:43:10.655 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:10.656 INFO:tasks.ceph:role: osd.0 2023-06-10T18:43:10.656 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:43:10.656 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-10T18:43:10.729 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:10.730 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:10.731 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:10.731 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:11.171 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2023-06-10T18:43:11.173 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2023-06-10T18:43:11.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-10T18:43:11.353 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-10T18:43:11.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-10T18:43:11.454 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:11.455 INFO:tasks.ceph:role: osd.1 2023-06-10T18:43:11.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:43:11.455 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-10T18:43:11.527 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:11.528 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:11.528 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:11.528 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:11.528 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:11.529 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:11.530 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:11.530 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:11.530 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:11.531 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:11.951 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2023-06-10T18:43:11.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2023-06-10T18:43:11.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-10T18:43:12.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-10T18:43:12.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-10T18:43:12.166 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:12.166 INFO:tasks.ceph:role: osd.2 2023-06-10T18:43:12.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:43:12.167 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-10T18:43:12.240 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:12.240 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:12.241 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:12.688 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2023-06-10T18:43:12.690 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2023-06-10T18:43:12.691 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-10T18:43:12.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-10T18:43:12.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-10T18:43:12.935 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:12.935 INFO:tasks.ceph:role: osd.3 2023-06-10T18:43:12.936 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:43:12.936 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-10T18:43:13.010 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:13.011 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:13.012 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:13.012 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:13.456 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2023-06-10T18:43:13.458 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2023-06-10T18:43:13.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-10T18:43:13.647 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-10T18:43:13.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:13.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.793+0000 7f58221bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-10T18:43:13.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.793+0000 7f58221bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-10T18:43:13.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.793+0000 7f58221bd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-10T18:43:13.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.793+0000 7f58221bd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-10T18:43:13.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.793+0000 7f58221bd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-10T18:43:13.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:13.794+0000 7f58221bd640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-10T18:43:16.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-10T18:43:16.693 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:16.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-10T18:43:16.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-10T18:43:16.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-10T18:43:16.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-10T18:43:16.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-10T18:43:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:16.777+0000 7f64627c9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-10T18:43:19.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-10T18:43:19.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.768+0000 7fa4e3331640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-10T18:43:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.768+0000 7fa4e3331640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-10T18:43:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.768+0000 7fa4e3331640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-10T18:43:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.768+0000 7fa4e3331640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-10T18:43:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.768+0000 7fa4e3331640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-10T18:43:19.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:19.769+0000 7fa4e3331640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-10T18:43:22.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-10T18:43:22.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:22.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.762+0000 7f5249c18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-10T18:43:22.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.763+0000 7f5249c18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-10T18:43:22.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.763+0000 7f5249c18640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-10T18:43:22.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.763+0000 7f5249c18640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-10T18:43:22.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.763+0000 7f5249c18640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-10T18:43:22.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:22.763+0000 7f5249c18640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-10T18:43:25.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-10T18:43:25.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-10T18:43:25.693 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:25.694 INFO:tasks.ceph:role: osd.4 2023-06-10T18:43:25.694 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:43:25.694 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-10T18:43:25.770 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:25.771 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:25.772 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:25.772 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:25.773 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:26.199 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2023-06-10T18:43:26.201 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2023-06-10T18:43:26.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-10T18:43:26.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-10T18:43:26.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-10T18:43:26.499 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:26.500 INFO:tasks.ceph:role: osd.5 2023-06-10T18:43:26.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:43:26.500 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-10T18:43:26.574 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:26.575 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:26.576 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:26.576 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:26.576 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:26.993 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2023-06-10T18:43:26.995 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2023-06-10T18:43:26.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-10T18:43:27.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-10T18:43:27.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-10T18:43:27.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:27.213 INFO:tasks.ceph:role: osd.6 2023-06-10T18:43:27.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:43:27.213 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:27.288 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:27.289 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:27.289 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:27.289 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:27.289 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:27.735 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2023-06-10T18:43:27.737 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2023-06-10T18:43:27.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-10T18:43:27.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-10T18:43:27.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-10T18:43:27.996 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-10T18:43:27.997 INFO:tasks.ceph:role: osd.7 2023-06-10T18:43:27.997 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:43:27.997 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-10T18:43:28.074 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-10T18:43:28.074 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-10T18:43:28.074 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-10T18:43:28.075 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-10T18:43:28.512 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2023-06-10T18:43:28.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2023-06-10T18:43:28.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-10T18:43:28.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-10T18:43:28.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.844+0000 7fa37204e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-10T18:43:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.844+0000 7fa37204e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-10T18:43:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.844+0000 7fa37204e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-10T18:43:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.844+0000 7fa37204e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-10T18:43:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.844+0000 7fa37204e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-10T18:43:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:28.845+0000 7fa37204e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-10T18:43:31.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-10T18:43:31.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:31.832 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.827+0000 7f23a3d7e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-10T18:43:31.832 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.828+0000 7f23a3d7e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-10T18:43:31.833 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.828+0000 7f23a3d7e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-10T18:43:31.833 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.828+0000 7f23a3d7e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-10T18:43:31.833 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.828+0000 7f23a3d7e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-10T18:43:31.834 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:31.828+0000 7f23a3d7e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-10T18:43:34.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-10T18:43:34.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:34.818 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.813+0000 7fd21a76b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-10T18:43:34.818 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.813+0000 7fd21a76b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-10T18:43:34.818 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.813+0000 7fd21a76b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-10T18:43:34.818 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.813+0000 7fd21a76b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-10T18:43:34.818 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.813+0000 7fd21a76b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-10T18:43:34.819 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:34.814+0000 7fd21a76b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-10T18:43:37.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:37.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:43:37.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-10T18:43:37.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:37.807 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.802+0000 7fcdd2fb6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-10T18:43:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.802+0000 7fcdd2fb6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-10T18:43:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.802+0000 7fcdd2fb6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-10T18:43:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.802+0000 7fcdd2fb6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-10T18:43:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.802+0000 7fcdd2fb6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-10T18:43:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:37.803+0000 7fcdd2fb6640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-10T18:43:40.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-10T18:43:40.699 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-10T18:43:40.699 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:40.700 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-10T18:43:40.731 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:40.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-10T18:43:40.800 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:40.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-10T18:43:40.868 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:40.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-10T18:43:40.936 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:40.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-10T18:43:41.004 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:41.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-10T18:43:41.071 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:41.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-10T18:43:41.138 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-10T18:43:41.170 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-10T18:43:41.238 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-10T18:43:41.306 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-10T18:43:41.373 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-10T18:43:41.439 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-10T18:43:41.505 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-10T18:43:41.571 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:41.571 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-10T18:43:41.590 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:41.590 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-10T18:43:41.648 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.649 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-10T18:43:41.665 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:41.665 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-10T18:43:41.720 INFO:tasks.ceph:Adding keys to all mons... 2023-06-10T18:43:41.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-10T18:43:41.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-10T18:43:41.765 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.y] 2023-06-10T18:43:41.765 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw== 2023-06-10T18:43:41.765 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.a] 2023-06-10T18:43:41.766 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA== 2023-06-10T18:43:41.766 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.c] 2023-06-10T18:43:41.766 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ== 2023-06-10T18:43:41.766 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2023-06-10T18:43:41.766 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q== 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw== 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg== 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2023-06-10T18:43:41.767 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q== 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg== 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.b] 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w== 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout:[mds.d] 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw== 2023-06-10T18:43:41.768 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2023-06-10T18:43:41.769 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA== 2023-06-10T18:43:41.769 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2023-06-10T18:43:41.769 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA== 2023-06-10T18:43:41.769 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.6] 2023-06-10T18:43:41.769 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw== 2023-06-10T18:43:41.770 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.7] 2023-06-10T18:43:41.770 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA== 2023-06-10T18:43:41.770 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2023-06-10T18:43:41.770 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA== 2023-06-10T18:43:41.770 INFO:teuthology.orchestra.run.smithi104.stdout:[client.2] 2023-06-10T18:43:41.771 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ== 2023-06-10T18:43:41.771 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2023-06-10T18:43:41.771 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw== 2023-06-10T18:43:41.771 INFO:teuthology.orchestra.run.smithi104.stdout:[client.3] 2023-06-10T18:43:41.771 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA== 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw== 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA== 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2023-06-10T18:43:41.786 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ== 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q== 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw== 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg== 2023-06-10T18:43:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2023-06-10T18:43:41.788 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q== 2023-06-10T18:43:41.788 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2023-06-10T18:43:41.788 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg== 2023-06-10T18:43:41.788 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2023-06-10T18:43:41.788 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w== 2023-06-10T18:43:41.789 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2023-06-10T18:43:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw== 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA== 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA== 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2023-06-10T18:43:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw== 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA== 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA== 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout:[client.2] 2023-06-10T18:43:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ== 2023-06-10T18:43:41.792 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2023-06-10T18:43:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw== 2023-06-10T18:43:41.792 INFO:teuthology.orchestra.run.smithi114.stdout:[client.3] 2023-06-10T18:43:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA== 2023-06-10T18:43:41.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-10T18:43:41.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-10T18:43:41.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:41.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:41.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:41.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:42.005 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.224 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-10T18:43:42.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-10T18:43:42.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:42.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:42.336 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:42.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-10T18:43:42.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-10T18:43:42.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-10T18:43:42.848 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-10T18:43:42.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-10T18:43:42.874 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-10T18:43:42.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-10T18:43:42.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-10T18:43:43.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-10T18:43:43.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-10T18:43:43.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-10T18:43:43.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-10T18:43:43.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-10T18:43:43.336 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:43.338 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-10T18:43:43.392 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-10T18:43:43.393 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-10T18:43:43.393 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-10T18:43:43.395 INFO:tasks.ceph.mon.a:Started 2023-06-10T18:43:43.396 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-10T18:43:43.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-10T18:43:43.434 INFO:tasks.ceph.mon.b:Started 2023-06-10T18:43:43.435 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-10T18:43:43.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-10T18:43:43.437 INFO:tasks.ceph.mon.c:Started 2023-06-10T18:43:43.437 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-10T18:43:43.437 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-10T18:43:43.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-10T18:43:43.440 INFO:tasks.ceph.mgr.y:Started 2023-06-10T18:43:43.440 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-10T18:43:43.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-10T18:43:43.442 INFO:tasks.ceph.mgr.x:Started 2023-06-10T18:43:43.443 INFO:tasks.ceph:Setting crush tunables to default 2023-06-10T18:43:43.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd crush tunables default 2023-06-10T18:43:43.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.560+0000 7f95404e1700 1 Processor -- start 2023-06-10T18:43:43.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.560+0000 7f95404e1700 1 -- start start 2023-06-10T18:43:43.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f95404e1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f95404e1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f95380d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f95404e1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380ad3c0 con 0x7f95380d0830 2023-06-10T18:43:43.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f95404e1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95380cecf0 con 0x7f9538157b30 2023-06-10T18:43:43.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f95404e1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9538157f00 con 0x7f95380ce880 2023-06-10T18:43:43.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953da7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f95380d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41144/0 (socket says 172.21.15.104:41144) 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 -- 172.21.15.104:0/2607491353 learned_addr learned my addr 172.21.15.104:0/2607491353 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953ea7e700 1 --1- 172.21.15.104:0/2607491353 >> v1:172.21.15.114:6789/0 conn(0x7f95380d0830 0x7f953815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:57320/0 (socket says 172.21.15.104:57320) 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 -- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 msgr2=0x7f95380ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953da7c700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 msgr2=0x7f95380d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 -- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 msgr2=0x7f95380ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953da7c700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 msgr2=0x7f95380d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-10T18:43:43.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 --2- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-10T18:43:43.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953da7c700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f95380d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-10T18:43:43.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953e27d700 1 --2- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-10T18:43:43.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.561+0000 7f953da7c700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f95380d02f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-10T18:43:43.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.757+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646224288 0 0) 0x7f95380ad3c0 con 0x7f95380d0830 2023-06-10T18:43:43.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.757+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95240035b0 con 0x7f95380d0830 2023-06-10T18:43:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471758176 0 0) 0x7f95240035b0 con 0x7f95380d0830 2023-06-10T18:43:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 msgr2=0x7f95380d02f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f95380d02f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 msgr2=0x7f95380ce340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 --2- 172.21.15.104:0/2607491353 >> v2:172.21.15.114:3300/0 conn(0x7f9538157b30 0x7f95380ce340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380b0170 con 0x7f95380d0830 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f95300019f0 con 0x7f95380d0830 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 >> v1:172.21.15.114:6789/0 conn(0x7f95380d0830 legacy=0x7f953815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524003a00 0x7f9524003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f9524004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9524004d90 0x7f9524005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95380cecf0 con 0x7f9524004d90 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9538157f00 con 0x7f9524003a00 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952400bb10 con 0x7f95380ce880 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953da7c700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f9524004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9524004d90 0x7f9524005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953e27d700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524003a00 0x7f9524003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 msgr2=0x7f9524004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380ce880 0x7f9524004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524003a00 msgr2=0x7f9524003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524003a00 0x7f9524003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.761+0000 7f953ea7e700 1 -- 172.21.15.104:0/2607491353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240035b0 con 0x7f9524004d90 2023-06-10T18:43:43.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f953ea7e700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9524004d90 0x7f9524005200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f953000a760 tx=0x7f9524005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681e0f43fa81e8b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:43.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000a250 con 0x7f9524004d90 2023-06-10T18:43:43.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95300072f0 con 0x7f9524004d90 2023-06-10T18:43:43.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f953d27b700 1 -- 172.21.15.104:0/2607491353 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95300076a0 con 0x7f9524004d90 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 -- 172.21.15.104:0/2607491353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9524004d90 msgr2=0x7f9524005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 --2- 172.21.15.104:0/2607491353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9524004d90 0x7f9524005200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f953000a760 tx=0x7f9524005880 comp rx=0 tx=0).stop 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f953ea7e700 1 -- 172.21.15.104:0/2607491353 reap_dead start 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 -- 172.21.15.104:0/2607491353 shutdown_connections 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 -- 172.21.15.104:0/2607491353 >> 172.21.15.104:0/2607491353 conn(0x7f9538089fd0 msgr2=0x7f953808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:43.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 -- 172.21.15.104:0/2607491353 shutdown_connections 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.762+0000 7f95404e1700 1 -- 172.21.15.104:0/2607491353 wait complete. 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.764+0000 7f95404e1700 1 Processor -- start 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.764+0000 7f95404e1700 1 -- start start 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 0x7f95380b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380b26f0 0x7f95380b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95380b30b0 con 0x7f9538154e40 2023-06-10T18:43:43.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9538153100 con 0x7f9524004d90 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95380b45b0 con 0x7f95380b26f0 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953e27d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 0x7f95380b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53664/0 (socket says 172.21.15.104:53664) 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953e27d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 0x7f95380b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41164/0 (socket says 172.21.15.104:41164) 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 -- 172.21.15.104:0/699765298 learned_addr learned my addr 172.21.15.104:0/699765298 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:43.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953ea7e700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380b26f0 0x7f95380b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 -- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380b26f0 msgr2=0x7f95380b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380b26f0 0x7f95380b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 -- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 msgr2=0x7f95380b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 0x7f95380b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 -- 172.21.15.104:0/699765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380b4890 con 0x7f9538154e40 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f953da7c700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f95280109f0 tx=0x7f9528010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528006ea0 con 0x7f9538154e40 2023-06-10T18:43:43.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952800ea70 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95380b4b70 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528009730 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.765+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95380b5130 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.766+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9528015470 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.766+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f95280164b0 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.766+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95380b5350 con 0x7f9538154e40 2023-06-10T18:43:43.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.769+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f952801a020 con 0x7f9538154e40 2023-06-10T18:43:43.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.842+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9538074050 con 0x7f9538154e40 2023-06-10T18:43:43.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.844+0000 7f952effd700 1 -- 172.21.15.104:0/699765298 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f952802e090 con 0x7f9538154e40 2023-06-10T18:43:43.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 msgr2=0x7f9538152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:43.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f95280109f0 tx=0x7f9528010d00 comp rx=0 tx=0).stop 2023-06-10T18:43:43.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 shutdown_connections 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524004d90 0x7f95380b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95380b26f0 0x7f95380b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 --2- 172.21.15.104:0/699765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9538154e40 0x7f9538152bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 >> 172.21.15.104:0/699765298 conn(0x7f9538089fd0 msgr2=0x7f95380aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 shutdown_connections 2023-06-10T18:43:43.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:43.845+0000 7f95404e1700 1 -- 172.21.15.104:0/699765298 wait complete. 2023-06-10T18:43:43.851 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2023-06-10T18:43:43.861 INFO:tasks.ceph:check_enable_crimson: False 2023-06-10T18:43:43.861 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-10T18:43:43.861 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:43.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-10T18:43:43.877 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:43.872+0000 7f7890154200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-10T18:43:43.887 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:43.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-10T18:43:43.942 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:43.937+0000 7f7890154200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-10T18:43:43.952 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:43.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-10T18:43:44.022 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.012+0000 7f7890154200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-10T18:43:44.023 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:43:44.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-10T18:43:44.089 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:44.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-10T18:43:44.095 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.090+0000 7f7890154200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-10T18:43:44.115 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:44.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-10T18:43:44.180 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:44.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-10T18:43:44.233 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.229+0000 7f55327f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-10T18:43:44.245 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:43:44.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-10T18:43:44.247 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.241+0000 7f7890154200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-10T18:43:44.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 909ccd07-b679-4767-9aeb-dded1012650a 0 2023-06-10T18:43:44.299 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.294+0000 7f55327f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-10T18:43:44.374 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.369+0000 7f55327f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-10T18:43:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.427+0000 7f3c4c66c700 1 Processor -- start 2023-06-10T18:43:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.428+0000 7f3c4c66c700 1 -- start start 2023-06-10T18:43:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4c66c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c440d0a00 0x7f3c44164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4c66c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440ad3c0 con 0x7f3c440cea50 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4c66c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44157f00 con 0x7f3c44157b30 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4c66c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440d0e60 con 0x7f3c440d0a00 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4a408700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3c44157b30 0x7f3c440ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38958/0 (socket says 172.21.15.114:38958) 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4a408700 1 -- 172.21.15.114:0/118602486 learned_addr learned my addr 172.21.15.114:0/118602486 (peer_addr_for_me v1:172.21.15.114:0/0) 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.429+0000 7f3c4ac09700 1 --2- 172.21.15.114:0/118602486 >> v2:172.21.15.104:3300/0 conn(0x7f3c440d0a00 0x7f3c44164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562550212 0 0) 0x7f3c44157f00 con 0x7f3c44157b30 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c300035b0 con 0x7f3c44157b30 2023-06-10T18:43:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c4ac09700 1 -- 172.21.15.114:0/118602486 >> v1:172.21.15.114:6789/0 conn(0x7f3c44157b30 legacy=0x7f3c440ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528671690 0 0) 0x7f3c440ad3c0 con 0x7f3c440cea50 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c4ac09700 1 -- 172.21.15.114:0/118602486 >> v1:172.21.15.104:6789/0 conn(0x7f3c440cea50 legacy=0x7f3c440d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c4ac09700 1 -- 172.21.15.114:0/118602486 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440b0170 con 0x7f3c440d0a00 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c4ac09700 1 --2- 172.21.15.114:0/118602486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440d0a00 0x7f3c44164e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b23754b1d45f83a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c00e070 con 0x7f3c440d0a00 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c00bb20 con 0x7f3c440d0a00 2023-06-10T18:43:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.430+0000 7f3c49406700 1 -- 172.21.15.114:0/118602486 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c00bed0 con 0x7f3c440d0a00 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 -- 172.21.15.114:0/118602486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440d0a00 msgr2=0x7f3c44164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/118602486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440d0a00 0x7f3c44164e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c00b330 comp rx=0 tx=0).stop 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 -- 172.21.15.114:0/118602486 shutdown_connections 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/118602486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440d0a00 0x7f3c44164e60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 -- 172.21.15.114:0/118602486 >> 172.21.15.114:0/118602486 conn(0x7f3c44089fd0 msgr2=0x7f3c4408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 -- 172.21.15.114:0/118602486 shutdown_connections 2023-06-10T18:43:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.431+0000 7f3c4c66c700 1 -- 172.21.15.114:0/118602486 wait complete. 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.432+0000 7f3c4c66c700 1 Processor -- start 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.432+0000 7f3c4c66c700 1 -- start start 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 0x7f3c4406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c440d0a00 0x7f3c44152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4406bc40 con 0x7f3c440cea50 2023-06-10T18:43:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4406bf20 con 0x7f3c44157b30 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4406c200 con 0x7f3c440d0a00 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60858/0 (socket says 172.21.15.114:60858) 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c49c07700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c440d0a00 0x7f3c44152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4a408700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 0x7f3c4406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 -- 172.21.15.114:0/147505084 learned_addr learned my addr 172.21.15.114:0/147505084 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 -- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c440d0a00 msgr2=0x7f3c44152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c440d0a00 0x7f3c44152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 -- 172.21.15.114:0/147505084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 msgr2=0x7f3c4406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 0x7f3c4406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 -- 172.21.15.114:0/147505084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009a70 con 0x7f3c44157b30 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4a408700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 0x7f3c4406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4ac09700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c44066690 tx=0x7f3c3c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c01b810 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c020070 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4406c4e0 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.433+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c01bbd0 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.434+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c44155bf0 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.434+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3c3c027780 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.434+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3c3c022070 con 0x7f3c44157b30 2023-06-10T18:43:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.435+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c440cc2c0 con 0x7f3c44157b30 2023-06-10T18:43:44.448 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.437+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f3c3c045b00 con 0x7f3c44157b30 2023-06-10T18:43:44.451 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.447+0000 7f55327f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-10T18:43:44.526 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.521+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "909ccd07-b679-4767-9aeb-dded1012650a", "id": 0} v 0) v1 -- 0x7f3c440cc6a0 con 0x7f3c44157b30 2023-06-10T18:43:44.530 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.524+0000 7f3c3affd700 1 -- 172.21.15.114:0/147505084 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "909ccd07-b679-4767-9aeb-dded1012650a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c440cc6a0 con 0x7f3c44157b30 2023-06-10T18:43:44.530 INFO:teuthology.orchestra.run.smithi114.stdout:0 2023-06-10T18:43:44.531 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 msgr2=0x7f3c4406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c44066690 tx=0x7f3c3c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 shutdown_connections 2023-06-10T18:43:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c44157b30 0x7f3c4406ee40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c440cea50 0x7f3c4406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.533 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 --2- 172.21.15.114:0/147505084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c440d0a00 0x7f3c44152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.533 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 >> 172.21.15.114:0/147505084 conn(0x7f3c44089fd0 msgr2=0x7f3c440b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.533 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 shutdown_connections 2023-06-10T18:43:44.534 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.526+0000 7f3c4c66c700 1 -- 172.21.15.114:0/147505084 wait complete. 2023-06-10T18:43:44.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 32d5a91b-dc25-4bb1-97ec-855dc9e01601 1 2023-06-10T18:43:44.594 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.590+0000 7f55327f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-10T18:43:44.609 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.604+0000 7f7890154200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-10T18:43:44.667 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.662+0000 7f6ca30cc700 1 Processor -- start 2023-06-10T18:43:44.667 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.662+0000 7f6ca30cc700 1 -- start start 2023-06-10T18:43:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca30cc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.669 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca30cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0b1780 0x7f6c9c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca30cc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c15a700 con 0x7f6c9c0cb420 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca30cc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c06d7b0 con 0x7f6c9c15a330 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca30cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0cbe00 con 0x7f6c9c0b1780 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca1669700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0b1780 0x7f6c9c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.670 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60860/0 (socket says 172.21.15.114:60860) 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 -- 172.21.15.114:0/637076980 learned_addr learned my addr 172.21.15.114:0/637076980 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 -- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0b1780 msgr2=0x7f6c9c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 --2- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0b1780 0x7f6c9c06dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 -- 172.21.15.114:0/637076980 >> v1:172.21.15.104:6789/0 conn(0x7f6c9c0cb420 legacy=0x7f6c9c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.663+0000 7f6ca0e68700 1 -- 172.21.15.114:0/637076980 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c06ce80 con 0x7f6c9c15a330 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.664+0000 7f6ca0e68700 1 --2- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6c90006fd0 tx=0x7f6c90009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f20fe22c196753c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.671 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.664+0000 7f6c9b7fe700 1 -- 172.21.15.114:0/637076980 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90005e30 con 0x7f6c9c15a330 2023-06-10T18:43:44.672 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.664+0000 7f6c9b7fe700 1 -- 172.21.15.114:0/637076980 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90004030 con 0x7f6c9c15a330 2023-06-10T18:43:44.672 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.664+0000 7f6c9b7fe700 1 -- 172.21.15.114:0/637076980 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900043e0 con 0x7f6c9c15a330 2023-06-10T18:43:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 -- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 msgr2=0x7f6c9c0ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6c90006fd0 tx=0x7f6c90009bd0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 -- 172.21.15.114:0/637076980 shutdown_connections 2023-06-10T18:43:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0b1780 0x7f6c9c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/637076980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c0ac1c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 -- 172.21.15.114:0/637076980 >> 172.21.15.114:0/637076980 conn(0x7f6c9c06b8a0 msgr2=0x7f6c9c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 -- 172.21.15.114:0/637076980 shutdown_connections 2023-06-10T18:43:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.665+0000 7f6ca30cc700 1 -- 172.21.15.114:0/637076980 wait complete. 2023-06-10T18:43:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.667+0000 7f6ca30cc700 1 Processor -- start 2023-06-10T18:43:44.674 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.667+0000 7f6ca30cc700 1 -- start start 2023-06-10T18:43:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c9c0b1780 0x7f6c9c0c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 0x7f6c9c0c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c152110 con 0x7f6c9c0b1780 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0c6b70 con 0x7f6c9c15a330 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca30cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c0c6da0 con 0x7f6c9c0cb420 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6c9bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 0x7f6c9c0c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6c9bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 0x7f6c9c0c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39542/0 (socket says 172.21.15.114:39542) 2023-06-10T18:43:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6c9bfff700 1 -- 172.21.15.114:0/2533784857 learned_addr learned my addr 172.21.15.114:0/2533784857 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca0e68700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c9c0b1780 0x7f6c9c0c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 -- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 msgr2=0x7f6c9c0c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 0x7f6c9c0c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 -- 172.21.15.114:0/2533784857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c9c0b1780 msgr2=0x7f6c9c0c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c9c0b1780 0x7f6c9c0c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.668+0000 7f6ca1669700 1 -- 172.21.15.114:0/2533784857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009500 con 0x7f6c9c15a330 2023-06-10T18:43:44.678 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6ca1669700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c150d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400eb70 tx=0x7f6c9400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94009a80 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9400f040 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c0c7080 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400cc80 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c0c7920 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6c94014780 con 0x7f6c9c15a330 2023-06-10T18:43:44.679 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.669+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f6c9401dc30 con 0x7f6c9c15a330 2023-06-10T18:43:44.680 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.670+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c074050 con 0x7f6c9c15a330 2023-06-10T18:43:44.680 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.672+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f6c94018020 con 0x7f6c9c15a330 2023-06-10T18:43:44.690 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.685+0000 7f7890154200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-10T18:43:44.761 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.755+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "32d5a91b-dc25-4bb1-97ec-855dc9e01601", "id": 1} v 0) v1 -- 0x7f6c9c06a700 con 0x7f6c9c15a330 2023-06-10T18:43:44.766 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.760+0000 7f6c997fa700 1 -- 172.21.15.114:0/2533784857 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32d5a91b-dc25-4bb1-97ec-855dc9e01601", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6c94014c60 con 0x7f6c9c15a330 2023-06-10T18:43:44.766 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:43:44.767 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.761+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 msgr2=0x7f6c9c150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.767 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.761+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c150d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400eb70 tx=0x7f6c9400ee80 comp rx=0 tx=0).stop 2023-06-10T18:43:44.767 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.761+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 shutdown_connections 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.761+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c9c15a330 0x7f6c9c150d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.761+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c9c0b1780 0x7f6c9c0c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.762+0000 7f6ca30cc700 1 --2- 172.21.15.114:0/2533784857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c9c0cb420 0x7f6c9c0c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.762+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 >> 172.21.15.114:0/2533784857 conn(0x7f6c9c06b8a0 msgr2=0x7f6c9c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.762+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 shutdown_connections 2023-06-10T18:43:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.762+0000 7f6ca30cc700 1 -- 172.21.15.114:0/2533784857 wait complete. 2023-06-10T18:43:44.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new f7f5934b-8789-436d-b4d5-95dd9334b240 2 2023-06-10T18:43:44.815 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.810+0000 7f7890154200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-10T18:43:44.899 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.894+0000 7f9135b2d700 1 Processor -- start 2023-06-10T18:43:44.899 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:44.894+0000 7f7890154200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-10T18:43:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.894+0000 7f9135b2d700 1 -- start start 2023-06-10T18:43:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.894+0000 7f9135b2d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91300d0d20 0x7f91300ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f9135b2d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91300ce340 con 0x7f91300ce7d0 2023-06-10T18:43:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f9135b2d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91300d00d0 con 0x7f913006af80 2023-06-10T18:43:44.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f9135b2d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91300cd680 con 0x7f91300d0d20 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912effd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91300d0d20 0x7f91300ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f913006af80 0x7f913006b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38982/0 (socket says 172.21.15.114:38982) 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912f7fe700 1 -- 172.21.15.114:0/251754667 learned_addr learned my addr 172.21.15.114:0/251754667 (peer_addr_for_me v1:172.21.15.114:0/0) 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912effd700 1 -- 172.21.15.114:0/251754667 >> v1:172.21.15.114:6789/0 conn(0x7f913006af80 legacy=0x7f913006b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912effd700 1 -- 172.21.15.114:0/251754667 >> v1:172.21.15.104:6789/0 conn(0x7f91300ce7d0 legacy=0x7f91300cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912e7fc700 1 -- 172.21.15.114:0/251754667 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101522226 0 0) 0x7f91300d00d0 con 0x7f913006af80 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912effd700 1 -- 172.21.15.114:0/251754667 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91300cd960 con 0x7f91300d0d20 2023-06-10T18:43:44.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912effd700 1 --2- 172.21.15.114:0/251754667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91300d0d20 0x7f91300ccff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9124009840 tx=0x7f9124009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d03d448d8bf24b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912e7fc700 1 -- 172.21.15.114:0/251754667 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124011830 con 0x7f91300d0d20 2023-06-10T18:43:44.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912e7fc700 1 -- 172.21.15.114:0/251754667 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124011990 con 0x7f91300d0d20 2023-06-10T18:43:44.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.895+0000 7f912e7fc700 1 -- 172.21.15.114:0/251754667 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124011d20 con 0x7f91300d0d20 2023-06-10T18:43:44.905 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 -- 172.21.15.114:0/251754667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91300d0d20 msgr2=0x7f91300ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 --2- 172.21.15.114:0/251754667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91300d0d20 0x7f91300ccff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9124009840 tx=0x7f9124009b50 comp rx=0 tx=0).stop 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 -- 172.21.15.114:0/251754667 shutdown_connections 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 --2- 172.21.15.114:0/251754667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91300d0d20 0x7f91300ccff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 -- 172.21.15.114:0/251754667 >> 172.21.15.114:0/251754667 conn(0x7f9130089fd0 msgr2=0x7f913008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 -- 172.21.15.114:0/251754667 shutdown_connections 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.896+0000 7f9135b2d700 1 -- 172.21.15.114:0/251754667 wait complete. 2023-06-10T18:43:44.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.897+0000 7f9135b2d700 1 Processor -- start 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.897+0000 7f9135b2d700 1 -- start start 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 0x7f91301565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91300d0d20 0x7f91300b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c5730 con 0x7f91300d0d20 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c5940 con 0x7f913006af80 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f9135b2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c5c20 con 0x7f91300ce7d0 2023-06-10T18:43:44.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912effd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 0x7f91301565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60878/0 (socket says 172.21.15.114:60878) 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 -- 172.21.15.114:0/3113355465 learned_addr learned my addr 172.21.15.114:0/3113355465 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912ffff700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91300d0d20 0x7f91300b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 -- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 msgr2=0x7f91301565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 0x7f91301565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 -- 172.21.15.114:0/3113355465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91300d0d20 msgr2=0x7f91300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.898+0000 7f912f7fe700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91300d0d20 0x7f91300b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f912f7fe700 1 -- 172.21.15.114:0/3113355465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240094a0 con 0x7f913006af80 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f912effd700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 0x7f91301565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f912f7fe700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f912000c520 tx=0x7f912000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120004c80 con 0x7f913006af80 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120004e00 con 0x7f913006af80 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91300c5f00 con 0x7f913006af80 2023-06-10T18:43:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120007310 con 0x7f913006af80 2023-06-10T18:43:44.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91300b2b60 con 0x7f913006af80 2023-06-10T18:43:44.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f912001d3f0 con 0x7f913006af80 2023-06-10T18:43:44.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.899+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f912001dac0 con 0x7f913006af80 2023-06-10T18:43:44.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.900+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9130074050 con 0x7f913006af80 2023-06-10T18:43:44.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.904+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f912001dd20 con 0x7f913006af80 2023-06-10T18:43:44.927 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.923+0000 7f55327f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-10T18:43:44.991 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.986+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f7f5934b-8789-436d-b4d5-95dd9334b240", "id": 2} v 0) v1 -- 0x7f91300afc60 con 0x7f913006af80 2023-06-10T18:43:44.997 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.992+0000 7f9117fff700 1 -- 172.21.15.114:0/3113355465 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f7f5934b-8789-436d-b4d5-95dd9334b240", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f912001d770 con 0x7f913006af80 2023-06-10T18:43:44.997 INFO:teuthology.orchestra.run.smithi114.stdout:2 2023-06-10T18:43:44.998 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.992+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 msgr2=0x7f91300c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:44.998 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.992+0000 7f9135b2d700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f912000c520 tx=0x7f912000c830 comp rx=0 tx=0).stop 2023-06-10T18:43:44.998 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 shutdown_connections 2023-06-10T18:43:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913006af80 0x7f91300c5190 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91300d0d20 0x7f91300b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 --2- 172.21.15.114:0/3113355465 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91300ce7d0 0x7f91301565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 >> 172.21.15.114:0/3113355465 conn(0x7f9130089fd0 msgr2=0x7f913008a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:44.999 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 shutdown_connections 2023-06-10T18:43:45.000 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:44.993+0000 7f9135b2d700 1 -- 172.21.15.114:0/3113355465 wait complete. 2023-06-10T18:43:45.002 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:44.998+0000 7f55327f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-10T18:43:45.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 45f300ab-4919-4d85-aca4-c1234f7bed6a 3 2023-06-10T18:43:45.129 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:45.124+0000 7f55327f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-10T18:43:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.166+0000 7fb0c33e4700 1 Processor -- start 2023-06-10T18:43:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 -- start start 2023-06-10T18:43:45.175 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc0ad3c0 con 0x7fb0bc0d0830 2023-06-10T18:43:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0cecf0 con 0x7fb0bc0ce880 2023-06-10T18:43:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c33e4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc157f00 con 0x7fb0bc157b30 2023-06-10T18:43:45.176 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c097f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c1180700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.177 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c097f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60886/0 (socket says 172.21.15.114:60886) 2023-06-10T18:43:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c1180700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39558/0 (socket says 172.21.15.114:39558) 2023-06-10T18:43:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.167+0000 7fb0c097f700 1 -- 172.21.15.114:0/2649243000 learned_addr learned my addr 172.21.15.114:0/2649243000 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c1180700 1 -- 172.21.15.114:0/2649243000 >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 msgr2=0x7fb0bc0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c1180700 1 --2- 172.21.15.114:0/2649243000 >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c1180700 1 -- 172.21.15.114:0/2649243000 >> v1:172.21.15.104:6789/0 conn(0x7fb0bc0d0830 legacy=0x7fb0bc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c1180700 1 -- 172.21.15.114:0/2649243000 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc0b0170 con 0x7fb0bc157b30 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c097f700 1 --2- 172.21.15.114:0/2649243000 >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0c1180700 1 --2- 172.21.15.114:0/2649243000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009d90 tx=0x7fb0ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576672203004ff25 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0bbfff700 1 -- 172.21.15.114:0/2649243000 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac00e070 con 0x7fb0bc157b30 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0bbfff700 1 -- 172.21.15.114:0/2649243000 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac00bb20 con 0x7fb0bc157b30 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.168+0000 7fb0bbfff700 1 -- 172.21.15.114:0/2649243000 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac012430 con 0x7fb0bc157b30 2023-06-10T18:43:45.179 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 -- 172.21.15.114:0/2649243000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 msgr2=0x7fb0bc0ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/2649243000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009d90 tx=0x7fb0ac00b330 comp rx=0 tx=0).stop 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 -- 172.21.15.114:0/2649243000 shutdown_connections 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/2649243000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc157b30 0x7fb0bc0ce340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/2649243000 >> v2:172.21.15.114:3300/0 conn(0x7fb0bc0ce880 0x7fb0bc0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 -- 172.21.15.114:0/2649243000 >> 172.21.15.114:0/2649243000 conn(0x7fb0bc089fd0 msgr2=0x7fb0bc08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.180 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 -- 172.21.15.114:0/2649243000 shutdown_connections 2023-06-10T18:43:45.181 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.169+0000 7fb0c33e4700 1 -- 172.21.15.114:0/2649243000 wait complete. 2023-06-10T18:43:45.181 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.170+0000 7fb0c33e4700 1 Processor -- start 2023-06-10T18:43:45.181 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.170+0000 7fb0c33e4700 1 -- start start 2023-06-10T18:43:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0bc0d0830 0x7fb0bc0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 0x7fb0bc14a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.183 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc1598b0 con 0x7fb0bc157b30 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c47b0 con 0x7fb0bc0d0830 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c33e4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c4a90 con 0x7fb0bc0ce880 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39562/0 (socket says 172.21.15.114:39562) 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 -- 172.21.15.114:0/689600906 learned_addr learned my addr 172.21.15.114:0/689600906 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.184 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c097f700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0bc0d0830 0x7fb0bc0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1981700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 0x7fb0bc14a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 -- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0bc0d0830 msgr2=0x7fb0bc0b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0bc0d0830 0x7fb0bc0b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 -- 172.21.15.114:0/689600906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 msgr2=0x7fb0bc14a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.171+0000 7fb0c1180700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 0x7fb0bc14a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.185 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0c1180700 1 -- 172.21.15.114:0/689600906 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0009550 con 0x7fb0bc0ce880 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0c1981700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 0x7fb0bc14a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0c1180700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac014040 tx=0x7fb0ac004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac012430 con 0x7fb0bc0ce880 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac020070 con 0x7fb0bc0ce880 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac009a70 con 0x7fb0bc0ce880 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac004280 con 0x7fb0bc0ce880 2023-06-10T18:43:45.186 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc14aa10 con 0x7fb0bc0ce880 2023-06-10T18:43:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb0ac0046e0 con 0x7fb0bc0ce880 2023-06-10T18:43:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.172+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fb0ac022070 con 0x7fb0bc0ce880 2023-06-10T18:43:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.173+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc0b03c0 con 0x7fb0bc0ce880 2023-06-10T18:43:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.175+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fb0ac027480 con 0x7fb0bc0ce880 2023-06-10T18:43:45.209 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:45.205+0000 7f55327f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-10T18:43:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.258+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "45f300ab-4919-4d85-aca4-c1234f7bed6a", "id": 3} v 0) v1 -- 0x7fb0bc074050 con 0x7fb0bc0ce880 2023-06-10T18:43:45.271 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.266+0000 7fb0b9ffb700 1 -- 172.21.15.114:0/689600906 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45f300ab-4919-4d85-aca4-c1234f7bed6a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb0ac027930 con 0x7fb0bc0ce880 2023-06-10T18:43:45.272 INFO:teuthology.orchestra.run.smithi114.stdout:3 2023-06-10T18:43:45.273 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 msgr2=0x7fb0bc0b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.273 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac014040 tx=0x7fb0ac004ad0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 shutdown_connections 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0bc157b30 0x7fb0bc14a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0bc0ce880 0x7fb0bc0b35a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 --2- 172.21.15.114:0/689600906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0bc0d0830 0x7fb0bc0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 >> 172.21.15.114:0/689600906 conn(0x7fb0bc089fd0 msgr2=0x7fb0bc0cdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 shutdown_connections 2023-06-10T18:43:45.274 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.267+0000 7fb0c33e4700 1 -- 172.21.15.114:0/689600906 wait complete. 2023-06-10T18:43:45.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 5e64e93e-2f14-433f-b034-016069e85071 4 2023-06-10T18:43:45.358 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:45.353+0000 7f7890154200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-10T18:43:45.409 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.404+0000 7f8033adb700 1 Processor -- start 2023-06-10T18:43:45.409 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.404+0000 7f8033adb700 1 -- start start 2023-06-10T18:43:45.412 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8033adb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f802c158790 0x7f802c0ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.412 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8033adb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c071580 con 0x7f802c0caf70 2023-06-10T18:43:45.413 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8033adb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c0c9020 con 0x7f802c0ccfe0 2023-06-10T18:43:45.413 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8033adb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c070f30 con 0x7f802c158790 2023-06-10T18:43:45.413 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031877700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f802c158790 0x7f802c0ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.413 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031076700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f802c0ccfe0 0x7f802c0c95c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39004/0 (socket says 172.21.15.114:39004) 2023-06-10T18:43:45.413 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031076700 1 -- 172.21.15.114:0/3653432533 learned_addr learned my addr 172.21.15.114:0/3653432533 (peer_addr_for_me v1:172.21.15.114:0/0) 2023-06-10T18:43:45.414 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031877700 1 -- 172.21.15.114:0/3653432533 >> v1:172.21.15.114:6789/0 conn(0x7f802c0ccfe0 legacy=0x7f802c0c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.414 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031877700 1 -- 172.21.15.114:0/3653432533 >> v1:172.21.15.104:6789/0 conn(0x7f802c0caf70 legacy=0x7f802c071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.414 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031877700 1 -- 172.21.15.114:0/3653432533 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c071210 con 0x7f802c158790 2023-06-10T18:43:45.414 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8031877700 1 --2- 172.21.15.114:0/3653432533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c158790 0x7f802c0ccaa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8028009840 tx=0x7f8028009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d9f7e0d4d9d1118 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8030875700 1 -- 172.21.15.114:0/3653432533 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028010800 con 0x7f802c158790 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8030875700 1 -- 172.21.15.114:0/3653432533 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8028010960 con 0x7f802c158790 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.405+0000 7f8030875700 1 -- 172.21.15.114:0/3653432533 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028010cf0 con 0x7f802c158790 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 -- 172.21.15.114:0/3653432533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c158790 msgr2=0x7f802c0ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 --2- 172.21.15.114:0/3653432533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c158790 0x7f802c0ccaa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8028009840 tx=0x7f8028009b50 comp rx=0 tx=0).stop 2023-06-10T18:43:45.415 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 -- 172.21.15.114:0/3653432533 shutdown_connections 2023-06-10T18:43:45.416 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 --2- 172.21.15.114:0/3653432533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c158790 0x7f802c0ccaa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.416 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 -- 172.21.15.114:0/3653432533 >> 172.21.15.114:0/3653432533 conn(0x7f802c06aac0 msgr2=0x7f802c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.416 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 -- 172.21.15.114:0/3653432533 shutdown_connections 2023-06-10T18:43:45.416 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.406+0000 7f8033adb700 1 -- 172.21.15.114:0/3653432533 wait complete. 2023-06-10T18:43:45.418 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.407+0000 7f8033adb700 1 Processor -- start 2023-06-10T18:43:45.418 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.407+0000 7f8033adb700 1 -- start start 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 0x7f802c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0ad970 con 0x7f802c0caf70 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c157430 con 0x7f802c0ccfe0 2023-06-10T18:43:45.419 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8033adb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c3e60 con 0x7f802c158790 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8031076700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8032078700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8031076700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60902/0 (socket says 172.21.15.114:60902) 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8031076700 1 -- 172.21.15.114:0/2036813931 learned_addr learned my addr 172.21.15.114:0/2036813931 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8032078700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39578/0 (socket says 172.21.15.114:39578) 2023-06-10T18:43:45.420 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.408+0000 7f8031877700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 0x7f802c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 -- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 msgr2=0x7f802c0c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 -- 172.21.15.114:0/2036813931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 msgr2=0x7f802c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 0x7f802c156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 -- 172.21.15.114:0/2036813931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280094a0 con 0x7f802c0ccfe0 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8032078700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.421 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031877700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 0x7f802c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8031076700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c520 tx=0x7f801c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c005d70 con 0x7f802c0ccfe0 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c0070d0 con 0x7f802c0ccfe0 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c0c32e0 con 0x7f802c0ccfe0 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c007440 con 0x7f802c0ccfe0 2023-06-10T18:43:45.422 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.409+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c0c38a0 con 0x7f802c0ccfe0 2023-06-10T18:43:45.423 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.410+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f801c01c440 con 0x7f802c0ccfe0 2023-06-10T18:43:45.423 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.410+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f801c012910 con 0x7f802c0ccfe0 2023-06-10T18:43:45.423 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.410+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c074050 con 0x7f802c0ccfe0 2023-06-10T18:43:45.423 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.412+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f801c021060 con 0x7f802c0ccfe0 2023-06-10T18:43:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.496+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5e64e93e-2f14-433f-b034-016069e85071", "id": 4} v 0) v1 -- 0x7f802c0ccb40 con 0x7f802c0ccfe0 2023-06-10T18:43:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.504+0000 7f80227fc700 1 -- 172.21.15.114:0/2036813931 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e64e93e-2f14-433f-b034-016069e85071", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f801c017070 con 0x7f802c0ccfe0 2023-06-10T18:43:45.510 INFO:teuthology.orchestra.run.smithi114.stdout:4 2023-06-10T18:43:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 msgr2=0x7f802c0c2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c520 tx=0x7f801c00c830 comp rx=0 tx=0).stop 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 shutdown_connections 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f802c158790 0x7f802c0c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f802c0caf70 0x7f802c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 --2- 172.21.15.114:0/2036813931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f802c0ccfe0 0x7f802c0c2560 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 >> 172.21.15.114:0/2036813931 conn(0x7f802c06aac0 msgr2=0x7f802c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 shutdown_connections 2023-06-10T18:43:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.505+0000 7f8033adb700 1 -- 172.21.15.114:0/2036813931 wait complete. 2023-06-10T18:43:45.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 5f09f9c5-d82b-4c8c-8fd7-1273dd788097 5 2023-06-10T18:43:45.646 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.640+0000 7fe3665db700 1 Processor -- start 2023-06-10T18:43:45.646 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.640+0000 7fe3665db700 1 -- start start 2023-06-10T18:43:45.648 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe3665db700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe3601545a0 0x7fe36015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.649 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe3665db700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.649 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe3665db700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36015aa00 con 0x7fe36015e720 2023-06-10T18:43:45.649 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe3665db700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36015dab0 con 0x7fe3601545a0 2023-06-10T18:43:45.649 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe3665db700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360154970 con 0x7fe3600af940 2023-06-10T18:43:45.649 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39582/0 (socket says 172.21.15.114:39582) 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 -- 172.21.15.114:0/544629820 learned_addr learned my addr 172.21.15.114:0/544629820 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35ffff700 1 --2- 172.21.15.114:0/544629820 >> v2:172.21.15.104:3300/0 conn(0x7fe3601545a0 0x7fe36015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 -- 172.21.15.114:0/544629820 >> v2:172.21.15.104:3300/0 conn(0x7fe3601545a0 msgr2=0x7fe36015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 --2- 172.21.15.114:0/544629820 >> v2:172.21.15.104:3300/0 conn(0x7fe3601545a0 0x7fe36015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 -- 172.21.15.114:0/544629820 >> v1:172.21.15.104:6789/0 conn(0x7fe36015e720 legacy=0x7fe3600cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.641+0000 7fe35f7fe700 1 -- 172.21.15.114:0/544629820 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3600cb0b0 con 0x7fe3600af940 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe35f7fe700 1 --2- 172.21.15.114:0/544629820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3540098f0 tx=0x7fe354009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd60dbc68eb447f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe35effd700 1 -- 172.21.15.114:0/544629820 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354011830 con 0x7fe3600af940 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe35effd700 1 -- 172.21.15.114:0/544629820 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354011990 con 0x7fe3600af940 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe35effd700 1 -- 172.21.15.114:0/544629820 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354011d20 con 0x7fe3600af940 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 -- 172.21.15.114:0/544629820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 msgr2=0x7fe36015e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 --2- 172.21.15.114:0/544629820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3540098f0 tx=0x7fe354009c00 comp rx=0 tx=0).stop 2023-06-10T18:43:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 -- 172.21.15.114:0/544629820 shutdown_connections 2023-06-10T18:43:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 --2- 172.21.15.114:0/544629820 >> v2:172.21.15.104:3300/0 conn(0x7fe3601545a0 0x7fe36015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 --2- 172.21.15.114:0/544629820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3600af940 0x7fe36015e1e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 -- 172.21.15.114:0/544629820 >> 172.21.15.114:0/544629820 conn(0x7fe36006e370 msgr2=0x7fe36006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 -- 172.21.15.114:0/544629820 shutdown_connections 2023-06-10T18:43:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.642+0000 7fe3665db700 1 -- 172.21.15.114:0/544629820 wait complete. 2023-06-10T18:43:45.657 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.643+0000 7fe3665db700 1 Processor -- start 2023-06-10T18:43:45.657 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 -- start start 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 0x7fe360151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 0x7fe360156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36006c3d0 con 0x7fe3600af940 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36006c0f0 con 0x7fe36015e720 2023-06-10T18:43:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe3665db700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360155660 con 0x7fe3601545a0 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe35f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.644+0000 7fe364b78700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 0x7fe360156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39588/0 (socket says 172.21.15.114:39588) 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 -- 172.21.15.114:0/1852242515 learned_addr learned my addr 172.21.15.114:0/1852242515 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35ffff700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 0x7fe360151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 -- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 msgr2=0x7fe360156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 0x7fe360156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 -- 172.21.15.114:0/1852242515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 msgr2=0x7fe360151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 0x7fe360151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 -- 172.21.15.114:0/1852242515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009550 con 0x7fe3601545a0 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe364b78700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 0x7fe360156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35ffff700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 0x7fe360151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35f7fe700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe354009c40 tx=0x7fe354004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354004510 con 0x7fe3601545a0 2023-06-10T18:43:45.661 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35401e070 con 0x7fe3601545a0 2023-06-10T18:43:45.661 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe360155940 con 0x7fe3601545a0 2023-06-10T18:43:45.661 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3540038e0 con 0x7fe3601545a0 2023-06-10T18:43:45.661 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.645+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe360151990 con 0x7fe3601545a0 2023-06-10T18:43:45.662 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.646+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe354003d40 con 0x7fe3601545a0 2023-06-10T18:43:45.662 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.646+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fe35401a9a0 con 0x7fe3601545a0 2023-06-10T18:43:45.662 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.646+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3601554c0 con 0x7fe3601545a0 2023-06-10T18:43:45.662 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.648+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fe3601554c0 con 0x7fe3601545a0 2023-06-10T18:43:45.663 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:45.656+0000 7f55327f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-10T18:43:45.739 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.734+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5f09f9c5-d82b-4c8c-8fd7-1273dd788097", "id": 5} v 0) v1 -- 0x7fe360074050 con 0x7fe3601545a0 2023-06-10T18:43:45.745 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.740+0000 7fe35cff9700 1 -- 172.21.15.114:0/1852242515 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f09f9c5-d82b-4c8c-8fd7-1273dd788097", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe354020070 con 0x7fe3601545a0 2023-06-10T18:43:45.745 INFO:teuthology.orchestra.run.smithi114.stdout:5 2023-06-10T18:43:45.746 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 msgr2=0x7fe360150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.746 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe354009c40 tx=0x7fe354004750 comp rx=0 tx=0).stop 2023-06-10T18:43:45.746 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 shutdown_connections 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe36015e720 0x7fe360156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3600af940 0x7fe360151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 --2- 172.21.15.114:0/1852242515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3601545a0 0x7fe360150a00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 >> 172.21.15.114:0/1852242515 conn(0x7fe36006e370 msgr2=0x7fe360158030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 shutdown_connections 2023-06-10T18:43:45.747 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.741+0000 7fe3665db700 1 -- 172.21.15.114:0/1852242515 wait complete. 2023-06-10T18:43:45.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 940b7625-3cf0-4851-8942-24160a1cfd7e 6 2023-06-10T18:43:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.893+0000 7f64c63ae700 1 Processor -- start 2023-06-10T18:43:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.893+0000 7f64c63ae700 1 -- start start 2023-06-10T18:43:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c63ae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f64c0157b30 0x7f64c00ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c63ae700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.901 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c63ae700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00ad3c0 con 0x7f64c00d0830 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c63ae700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c00cecf0 con 0x7f64c0157b30 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c63ae700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0157f00 con 0x7f64c00ce880 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64bf7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c494b700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f64c00d0830 0x7f64c015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39032/0 (socket says 172.21.15.114:39032) 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64bf7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39594/0 (socket says 172.21.15.114:39594) 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.894+0000 7f64c494b700 1 -- 172.21.15.114:0/1712792162 learned_addr learned my addr 172.21.15.114:0/1712792162 (peer_addr_for_me v1:172.21.15.114:0/0) 2023-06-10T18:43:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bffff700 1 --2- 172.21.15.114:0/1712792162 >> v2:172.21.15.104:3300/0 conn(0x7f64c0157b30 0x7f64c00ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bf7fe700 1 -- 172.21.15.114:0/1712792162 >> v2:172.21.15.104:3300/0 conn(0x7f64c0157b30 msgr2=0x7f64c00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64beffd700 1 -- 172.21.15.114:0/1712792162 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604786989 0 0) 0x7f64c00ad3c0 con 0x7f64c00d0830 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bf7fe700 1 --2- 172.21.15.114:0/1712792162 >> v2:172.21.15.104:3300/0 conn(0x7f64c0157b30 0x7f64c00ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bf7fe700 1 -- 172.21.15.114:0/1712792162 >> v1:172.21.15.114:6789/0 conn(0x7f64c00d0830 legacy=0x7f64c015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bf7fe700 1 -- 172.21.15.114:0/1712792162 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c00b0170 con 0x7f64c00ce880 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64bf7fe700 1 --2- 172.21.15.114:0/1712792162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009840 tx=0x7f64b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c18d04171be67f7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64beffd700 1 -- 172.21.15.114:0/1712792162 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4011830 con 0x7f64c00ce880 2023-06-10T18:43:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64beffd700 1 -- 172.21.15.114:0/1712792162 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b4011990 con 0x7f64c00ce880 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.895+0000 7f64beffd700 1 -- 172.21.15.114:0/1712792162 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4011d20 con 0x7f64c00ce880 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 -- 172.21.15.114:0/1712792162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 msgr2=0x7f64c00d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 --2- 172.21.15.114:0/1712792162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009840 tx=0x7f64b4009b50 comp rx=0 tx=0).stop 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 -- 172.21.15.114:0/1712792162 shutdown_connections 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 --2- 172.21.15.114:0/1712792162 >> v2:172.21.15.104:3300/0 conn(0x7f64c0157b30 0x7f64c00ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 --2- 172.21.15.114:0/1712792162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c00ce880 0x7f64c00d02f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 -- 172.21.15.114:0/1712792162 >> 172.21.15.114:0/1712792162 conn(0x7f64c0089fd0 msgr2=0x7f64c008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:45.905 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 -- 172.21.15.114:0/1712792162 shutdown_connections 2023-06-10T18:43:45.905 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.896+0000 7f64c63ae700 1 -- 172.21.15.114:0/1712792162 wait complete. 2023-06-10T18:43:45.906 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.897+0000 7f64c63ae700 1 Processor -- start 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.897+0000 7f64c63ae700 1 -- start start 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 0x7f64c00b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c0157b30 0x7f64c00b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c009e430 con 0x7f64c00d0830 2023-06-10T18:43:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c00b3ab0 con 0x7f64c00ce880 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c63ae700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c00b3d90 con 0x7f64c0157b30 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64c494b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c0157b30 0x7f64c00b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60924/0 (socket says 172.21.15.114:60924) 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 -- 172.21.15.114:0/2334438667 learned_addr learned my addr 172.21.15.114:0/2334438667 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bf7fe700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 0x7f64c00b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 -- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c0157b30 msgr2=0x7f64c00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c0157b30 0x7f64c00b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 -- 172.21.15.114:0/2334438667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 msgr2=0x7f64c00b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 0x7f64c00b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bffff700 1 -- 172.21.15.114:0/2334438667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40094a0 con 0x7f64c00ce880 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.898+0000 7f64bf7fe700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 0x7f64c00b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bffff700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64b000e870 tx=0x7f64b000eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b0009590 con 0x7f64c00ce880 2023-06-10T18:43:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b0013680 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c00b4070 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b00139f0 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c00b63f0 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f64b0005a20 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.899+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f64b00066b0 con 0x7f64c00ce880 2023-06-10T18:43:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.900+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c009e5a0 con 0x7f64c00ce880 2023-06-10T18:43:45.911 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.903+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f64c009e5a0 con 0x7f64c00ce880 2023-06-10T18:43:45.952 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:45.947+0000 7f7890154200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-10T18:43:45.992 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.987+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "940b7625-3cf0-4851-8942-24160a1cfd7e", "id": 6} v 0) v1 -- 0x7f64c0074050 con 0x7f64c00ce880 2023-06-10T18:43:46.001 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.995+0000 7f64bcff9700 1 -- 172.21.15.114:0/2334438667 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "940b7625-3cf0-4851-8942-24160a1cfd7e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f64b0021060 con 0x7f64c00ce880 2023-06-10T18:43:46.001 INFO:teuthology.orchestra.run.smithi114.stdout:6 2023-06-10T18:43:46.002 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 msgr2=0x7f64c009def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.002 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64b000e870 tx=0x7f64b000eb80 comp rx=0 tx=0).stop 2023-06-10T18:43:46.002 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 shutdown_connections 2023-06-10T18:43:46.002 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64c00ce880 0x7f64c009def0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.003 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64c0157b30 0x7f64c00b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.003 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.996+0000 7f64c63ae700 1 --2- 172.21.15.114:0/2334438667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64c00d0830 0x7f64c00b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.003 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.997+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 >> 172.21.15.114:0/2334438667 conn(0x7f64c0089fd0 msgr2=0x7f64c0156c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:46.003 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.997+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 shutdown_connections 2023-06-10T18:43:46.003 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:45.997+0000 7f64c63ae700 1 -- 172.21.15.114:0/2334438667 wait complete. 2023-06-10T18:43:46.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new aa52f88f-f755-4d94-9418-8c557c565c82 7 2023-06-10T18:43:46.114 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:46.109+0000 7f7890154200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-10T18:43:46.137 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.131+0000 7f7316024700 1 Processor -- start 2023-06-10T18:43:46.137 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.131+0000 7f7316024700 1 -- start start 2023-06-10T18:43:46.139 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f7316024700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 0x7f731006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.139 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f7316024700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100ae7e0 con 0x7f731015c6f0 2023-06-10T18:43:46.139 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f7316024700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731006df90 con 0x7f731006ea70 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f7316024700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731006e270 con 0x7f73100b1780 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730effd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 0x7f731006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730ffff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f731006ea70 0x7f731006d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39044/0 (socket says 172.21.15.114:39044) 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730ffff700 1 -- 172.21.15.114:0/3751084679 learned_addr learned my addr 172.21.15.114:0/3751084679 (peer_addr_for_me v1:172.21.15.114:0/0) 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730e7fc700 1 -- 172.21.15.114:0/3751084679 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195720798 0 0) 0x7f731006df90 con 0x7f731006ea70 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730e7fc700 1 -- 172.21.15.114:0/3751084679 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f80035b0 con 0x7f731006ea70 2023-06-10T18:43:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730effd700 1 -- 172.21.15.114:0/3751084679 >> v1:172.21.15.114:6789/0 conn(0x7f731006ea70 legacy=0x7f731006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730effd700 1 -- 172.21.15.114:0/3751084679 >> v1:172.21.15.104:6789/0 conn(0x7f731015c6f0 legacy=0x7f73100b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730effd700 1 -- 172.21.15.114:0/3751084679 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731006e550 con 0x7f73100b1780 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.132+0000 7f730effd700 1 --2- 172.21.15.114:0/3751084679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 0x7f731006cde0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f730400bb10 tx=0x7f730400be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf8a545267526a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f730e7fc700 1 -- 172.21.15.114:0/3751084679 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730400dc70 con 0x7f73100b1780 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f730e7fc700 1 -- 172.21.15.114:0/3751084679 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400ddd0 con 0x7f73100b1780 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f730e7fc700 1 -- 172.21.15.114:0/3751084679 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304010690 con 0x7f73100b1780 2023-06-10T18:43:46.141 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 -- 172.21.15.114:0/3751084679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 msgr2=0x7f731006cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 --2- 172.21.15.114:0/3751084679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 0x7f731006cde0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f730400bb10 tx=0x7f730400be20 comp rx=0 tx=0).stop 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 -- 172.21.15.114:0/3751084679 shutdown_connections 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 --2- 172.21.15.114:0/3751084679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73100b1780 0x7f731006cde0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 -- 172.21.15.114:0/3751084679 >> 172.21.15.114:0/3751084679 conn(0x7f731006b8a0 msgr2=0x7f73100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 -- 172.21.15.114:0/3751084679 shutdown_connections 2023-06-10T18:43:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.133+0000 7f7316024700 1 -- 172.21.15.114:0/3751084679 wait complete. 2023-06-10T18:43:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.135+0000 7f7316024700 1 Processor -- start 2023-06-10T18:43:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.135+0000 7f7316024700 1 -- start start 2023-06-10T18:43:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f731006ea70 0x7f731014b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f731015c6f0 0x7f73101f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.146 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100af4d0 con 0x7f731006ea70 2023-06-10T18:43:46.146 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100c79d0 con 0x7f73100b1780 2023-06-10T18:43:46.146 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f7316024700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100af6c0 con 0x7f731015c6f0 2023-06-10T18:43:46.146 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60938/0 (socket says 172.21.15.114:60938) 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 -- 172.21.15.114:0/836195615 learned_addr learned my addr 172.21.15.114:0/836195615 (peer_addr_for_me v2:172.21.15.114:0/0) 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730ffff700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f731015c6f0 0x7f73101f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730f7fe700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f731006ea70 0x7f731014b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 -- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f731015c6f0 msgr2=0x7f73101f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f731015c6f0 0x7f73101f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.147 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 -- 172.21.15.114:0/836195615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f731006ea70 msgr2=0x7f731014b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f731006ea70 0x7f731014b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 -- 172.21.15.114:0/836195615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730400b740 con 0x7f73100b1780 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.136+0000 7f730effd700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f730400ba60 tx=0x7f7304004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730401f070 con 0x7f73100b1780 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304003970 con 0x7f73100b1780 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304003ce0 con 0x7f73100b1780 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73100af9a0 con 0x7f73100b1780 2023-06-10T18:43:46.148 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73101f2140 con 0x7f73100b1780 2023-06-10T18:43:46.149 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7304010470 con 0x7f73100b1780 2023-06-10T18:43:46.149 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.137+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7304010d80 con 0x7f73100b1780 2023-06-10T18:43:46.149 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.138+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7310066f10 con 0x7f73100b1780 2023-06-10T18:43:46.149 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.140+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f7304029070 con 0x7f73100b1780 2023-06-10T18:43:46.213 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:46.209+0000 7f55327f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-10T18:43:46.228 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.222+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "aa52f88f-f755-4d94-9418-8c557c565c82", "id": 7} v 0) v1 -- 0x7f73100afec0 con 0x7f73100b1780 2023-06-10T18:43:46.235 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.230+0000 7f72f7fff700 1 -- 172.21.15.114:0/836195615 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aa52f88f-f755-4d94-9418-8c557c565c82", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f73040107f0 con 0x7f73100b1780 2023-06-10T18:43:46.236 INFO:teuthology.orchestra.run.smithi114.stdout:7 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 msgr2=0x7f73100aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f730400ba60 tx=0x7f7304004750 comp rx=0 tx=0).stop 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 shutdown_connections 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f731015c6f0 0x7f73101f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f731006ea70 0x7f731014b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.237 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 --2- 172.21.15.114:0/836195615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73100b1780 0x7f73100aef90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.238 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 >> 172.21.15.114:0/836195615 conn(0x7f731006b8a0 msgr2=0x7f7310155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:46.238 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 shutdown_connections 2023-06-10T18:43:46.238 INFO:teuthology.orchestra.run.smithi114.stderr:2023-06-10T18:43:46.231+0000 7f7316024700 1 -- 172.21.15.114:0/836195615 wait complete. 2023-06-10T18:43:46.249 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-10T18:43:46.249 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-10T18:43:46.251 INFO:tasks.ceph.osd.0:Started 2023-06-10T18:43:46.251 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-10T18:43:46.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-10T18:43:46.254 INFO:tasks.ceph.osd.1:Started 2023-06-10T18:43:46.254 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-10T18:43:46.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-10T18:43:46.256 INFO:tasks.ceph.osd.2:Started 2023-06-10T18:43:46.256 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-10T18:43:46.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-10T18:43:46.258 INFO:tasks.ceph.osd.3:Started 2023-06-10T18:43:46.259 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-10T18:43:46.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-10T18:43:46.261 INFO:tasks.ceph.osd.4:Started 2023-06-10T18:43:46.261 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-10T18:43:46.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-10T18:43:46.263 INFO:tasks.ceph.osd.5:Started 2023-06-10T18:43:46.263 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-10T18:43:46.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-10T18:43:46.265 INFO:tasks.ceph.osd.6:Started 2023-06-10T18:43:46.266 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-10T18:43:46.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-10T18:43:46.268 INFO:tasks.ceph.osd.7:Started 2023-06-10T18:43:46.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:43:46.409 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:46.405+0000 7f55327f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-10T18:43:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.440+0000 7fa1e78f7700 1 Processor -- start 2023-06-10T18:43:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.440+0000 7fa1e78f7700 1 -- start start 2023-06-10T18:43:46.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e78f7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1e00cfa00 0x7fa1e00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e78f7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa1e00cad90 0x7fa1e00b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e78f7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e00ccca0 con 0x7fa1e015a250 2023-06-10T18:43:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e78f7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e00cee50 con 0x7fa1e00cad90 2023-06-10T18:43:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e78f7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e00cf130 con 0x7fa1e00cfa00 2023-06-10T18:43:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.441+0000 7fa1e5e94700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa1e00cad90 0x7fa1e00b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa1e00cad90 0x7fa1e00b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53704/0 (socket says 172.21.15.104:53704) 2023-06-10T18:43:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 -- 172.21.15.104:0/3987832560 learned_addr learned my addr 172.21.15.104:0/3987832560 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5693700 1 --2- 172.21.15.104:0/3987832560 >> v2:172.21.15.114:3300/0 conn(0x7fa1e00cfa00 0x7fa1e00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 -- 172.21.15.104:0/3987832560 >> v2:172.21.15.114:3300/0 conn(0x7fa1e00cfa00 msgr2=0x7fa1e00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 --2- 172.21.15.104:0/3987832560 >> v2:172.21.15.114:3300/0 conn(0x7fa1e00cfa00 0x7fa1e00701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 -- 172.21.15.104:0/3987832560 >> v1:172.21.15.114:6789/0 conn(0x7fa1e015a250 legacy=0x7fa1e00cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 -- 172.21.15.104:0/3987832560 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00cc930 con 0x7fa1e00cad90 2023-06-10T18:43:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1e5e94700 1 --2- 172.21.15.104:0/3987832560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e00cad90 0x7fa1e00b3590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00b3300 tx=0x7fa1d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef863e85eed06c90 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1d7fff700 1 -- 172.21.15.104:0/3987832560 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800e070 con 0x7fa1e00cad90 2023-06-10T18:43:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1d7fff700 1 -- 172.21.15.104:0/3987832560 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d800b910 con 0x7fa1e00cad90 2023-06-10T18:43:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.442+0000 7fa1d7fff700 1 -- 172.21.15.104:0/3987832560 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800bcc0 con 0x7fa1e00cad90 2023-06-10T18:43:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 -- 172.21.15.104:0/3987832560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e00cad90 msgr2=0x7fa1e00b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/3987832560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e00cad90 0x7fa1e00b3590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00b3300 tx=0x7fa1d800b330 comp rx=0 tx=0).stop 2023-06-10T18:43:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 -- 172.21.15.104:0/3987832560 shutdown_connections 2023-06-10T18:43:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/3987832560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e00cad90 0x7fa1e00b3590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/3987832560 >> v2:172.21.15.114:3300/0 conn(0x7fa1e00cfa00 0x7fa1e00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 -- 172.21.15.104:0/3987832560 >> 172.21.15.104:0/3987832560 conn(0x7fa1e006e370 msgr2=0x7fa1e006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 -- 172.21.15.104:0/3987832560 shutdown_connections 2023-06-10T18:43:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.443+0000 7fa1e78f7700 1 -- 172.21.15.104:0/3987832560 wait complete. 2023-06-10T18:43:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.444+0000 7fa1e78f7700 1 Processor -- start 2023-06-10T18:43:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.444+0000 7fa1e78f7700 1 -- start start 2023-06-10T18:43:46.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1e00cad90 0x7fa1e01589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1e00cfa00 0x7fa1e01f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0158510 con 0x7fa1e015a250 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01f3e30 con 0x7fa1e00cfa00 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0151290 con 0x7fa1e00cad90 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53710/0 (socket says 172.21.15.104:53710) 2023-06-10T18:43:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 -- 172.21.15.104:0/2821367459 learned_addr learned my addr 172.21.15.104:0/2821367459 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e4e92700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1e00cfa00 0x7fa1e01f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5693700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1e00cad90 0x7fa1e01589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 -- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1e00cad90 msgr2=0x7fa1e01589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1e00cad90 0x7fa1e01589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 -- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1e00cfa00 msgr2=0x7fa1e01f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1e00cfa00 0x7fa1e01f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 -- 172.21.15.104:0/2821367459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8009b30 con 0x7fa1e015a250 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e5e94700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800b370 tx=0x7fa1d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8004880 con 0x7fa1e015a250 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d80099b0 con 0x7fa1e015a250 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e0157900 con 0x7fa1e015a250 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.445+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d801c680 con 0x7fa1e015a250 2023-06-10T18:43:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.446+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e0157ec0 con 0x7fa1e015a250 2023-06-10T18:43:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.447+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa1d801cae0 con 0x7fa1e015a250 2023-06-10T18:43:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.447+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e0070bb0 con 0x7fa1e015a250 2023-06-10T18:43:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.447+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa1d80123f0 con 0x7fa1e015a250 2023-06-10T18:43:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.449+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fa1d8043b00 con 0x7fa1e015a250 2023-06-10T18:43:46.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.517+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1e0074050 con 0x7fa1e015a250 2023-06-10T18:43:46.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.518+0000 7fa1d5ffb700 1 -- 172.21.15.104:0/2821367459 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa1d8032070 con 0x7fa1e015a250 2023-06-10T18:43:46.523 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:43:46.524 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":10,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:43:46.224581+0000","last_up_change":"0.000000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:43:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 msgr2=0x7fa1e0150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800b370 tx=0x7fa1d8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 shutdown_connections 2023-06-10T18:43:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1e015a250 0x7fa1e0150d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1e00cad90 0x7fa1e01589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 --2- 172.21.15.104:0/2821367459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1e00cfa00 0x7fa1e01f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 >> 172.21.15.104:0/2821367459 conn(0x7fa1e006e370 msgr2=0x7fa1e00b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:46.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 shutdown_connections 2023-06-10T18:43:46.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:43:46.519+0000 7fa1e78f7700 1 -- 172.21.15.104:0/2821367459 wait complete. 2023-06-10T18:43:46.535 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-10T18:43:46.536 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-10T18:43:46.842 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:46.837+0000 7f7890154200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-10T18:43:46.866 INFO:tasks.ceph.osd.2.smithi104.stderr:2023-06-10T18:43:46.861+0000 7f2b575f7640 -1 Falling back to public interface 2023-06-10T18:43:46.875 INFO:tasks.ceph.osd.4.smithi114.stderr:2023-06-10T18:43:46.870+0000 7fd86e7e1640 -1 Falling back to public interface 2023-06-10T18:43:46.877 INFO:tasks.ceph.osd.3.smithi104.stderr:2023-06-10T18:43:46.873+0000 7fe833a18640 -1 Falling back to public interface 2023-06-10T18:43:46.887 INFO:tasks.ceph.osd.1.smithi104.stderr:2023-06-10T18:43:46.882+0000 7f1533953640 -1 Falling back to public interface 2023-06-10T18:43:46.888 INFO:tasks.ceph.osd.7.smithi114.stderr:2023-06-10T18:43:46.882+0000 7f12003b3640 -1 Falling back to public interface 2023-06-10T18:43:46.895 INFO:tasks.ceph.osd.0.smithi104.stderr:2023-06-10T18:43:46.891+0000 7fb5926d1640 -1 Falling back to public interface 2023-06-10T18:43:46.896 INFO:tasks.ceph.osd.5.smithi114.stderr:2023-06-10T18:43:46.891+0000 7f0b64df6640 -1 Falling back to public interface 2023-06-10T18:43:46.897 INFO:tasks.ceph.osd.6.smithi114.stderr:2023-06-10T18:43:46.891+0000 7f9834a7b640 -1 Falling back to public interface 2023-06-10T18:43:46.897 INFO:tasks.ceph.osd.5.smithi114.stderr:2023-06-10T18:43:46.892+0000 7f0b5c1b4700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-06-10T18:43:46.922 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:46.917+0000 7f7890154200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-10T18:43:47.062 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:47.056+0000 7f7890154200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-10T18:43:47.105 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.100+0000 7f55327f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-10T18:43:47.183 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.179+0000 7f55327f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-10T18:43:47.298 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:47.292+0000 7f7890154200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-10T18:43:47.323 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.319+0000 7f55327f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-10T18:43:47.341 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-10T18:43:47.373 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:47.367+0000 7f7890154200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-10T18:43:47.453 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.449+0000 7f094c07c700 1 Processor -- start 2023-06-10T18:43:47.453 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.449+0000 7f094c07c700 1 -- start start 2023-06-10T18:43:47.455 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094c07c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 0x7f09440b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:47.455 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094c07c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440cf390 con 0x7f09440b15b0 2023-06-10T18:43:47.455 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094c07c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440ae3f0 con 0x7f094415f9d0 2023-06-10T18:43:47.455 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094c07c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09440b1a50 con 0x7f09440b1c30 2023-06-10T18:43:47.455 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094a619700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f09440b15b0 0x7f094406d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51544/0 (socket says 172.21.15.104:51544) 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f094a619700 1 -- 172.21.15.104:0/3287854556 learned_addr learned my addr 172.21.15.104:0/3287854556 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0949617700 1 --2- 172.21.15.104:0/3287854556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 0x7f09440b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084941583 0 0) 0x7f09440cf390 con 0x7f09440b15b0 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09300035b0 con 0x7f09440b15b0 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832599878 0 0) 0x7f09440ae3f0 con 0x7f094415f9d0 2023-06-10T18:43:47.456 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09440cf390 con 0x7f094415f9d0 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0949617700 1 -- 172.21.15.104:0/3287854556 >> v1:172.21.15.114:6789/0 conn(0x7f094415f9d0 legacy=0x7f09440cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0949617700 1 -- 172.21.15.104:0/3287854556 >> v1:172.21.15.104:6789/0 conn(0x7f09440b15b0 legacy=0x7f094406d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0949617700 1 -- 172.21.15.104:0/3287854556 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440ae6d0 con 0x7f09440b1c30 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.450+0000 7f0949617700 1 --2- 172.21.15.104:0/3287854556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 0x7f09440b1070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0934009870 tx=0x7f0934009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73fee8a878bf396f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934011830 con 0x7f09440b1c30 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934011990 con 0x7f09440b1c30 2023-06-10T18:43:47.457 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f0948e16700 1 -- 172.21.15.104:0/3287854556 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934011d20 con 0x7f09440b1c30 2023-06-10T18:43:47.459 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 -- 172.21.15.104:0/3287854556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 msgr2=0x7f09440b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 --2- 172.21.15.104:0/3287854556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 0x7f09440b1070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0934009870 tx=0x7f0934009b80 comp rx=0 tx=0).stop 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 -- 172.21.15.104:0/3287854556 shutdown_connections 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 --2- 172.21.15.104:0/3287854556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b1c30 0x7f09440b1070 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 -- 172.21.15.104:0/3287854556 >> 172.21.15.104:0/3287854556 conn(0x7f094406b8a0 msgr2=0x7f09440cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 -- 172.21.15.104:0/3287854556 shutdown_connections 2023-06-10T18:43:47.460 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.451+0000 7f094c07c700 1 -- 172.21.15.104:0/3287854556 wait complete. 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.452+0000 7f094c07c700 1 Processor -- start 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.452+0000 7f094c07c700 1 -- start start 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b15b0 0x7f09441f1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09440b1c30 0x7f09440c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094415cdc0 con 0x7f094415f9d0 2023-06-10T18:43:47.461 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09441f1ee0 con 0x7f09440b1c30 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094c07c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09441f21c0 con 0x7f09440b15b0 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094a619700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094a619700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53722/0 (socket says 172.21.15.104:53722) 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f094a619700 1 -- 172.21.15.104:0/3924635167 learned_addr learned my addr 172.21.15.104:0/3924635167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f0949617700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09440b1c30 0x7f09440c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.453+0000 7f0949e18700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b15b0 0x7f09441f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:47.462 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 -- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b15b0 msgr2=0x7f09441f1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b15b0 0x7f09441f1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 -- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09440b1c30 msgr2=0x7f09440c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09440b1c30 0x7f09440c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 -- 172.21.15.104:0/3924635167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340094a0 con 0x7f094415f9d0 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094a619700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f093c00cb40 tx=0x7f093c00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c00a9e0 con 0x7f094415f9d0 2023-06-10T18:43:47.463 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00ab40 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09441f24a0 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c012400 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.454+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09441f2780 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.455+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f093c012560 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.455+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f093c01ba80 con 0x7f094415f9d0 2023-06-10T18:43:47.464 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.455+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944073720 con 0x7f094415f9d0 2023-06-10T18:43:47.465 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.457+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f093c017070 con 0x7f094415f9d0 2023-06-10T18:43:47.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.525+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0944067220 con 0x7f094415f9d0 2023-06-10T18:43:47.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.526+0000 7f093a7fc700 1 -- 172.21.15.104:0/3924635167 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0944067220 con 0x7f094415f9d0 2023-06-10T18:43:47.530 INFO:teuthology.misc.health.smithi104.stdout: 2023-06-10T18:43:47.531 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:43:46.224581+0000","last_up_change":"0.000000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 msgr2=0x7f09441538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f093c00cb40 tx=0x7f093c00ce50 comp rx=0 tx=0).stop 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 shutdown_connections 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094415f9d0 0x7f09441538a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09440b15b0 0x7f09441f1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 --2- 172.21.15.104:0/3924635167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09440b1c30 0x7f09440c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:47.533 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 >> 172.21.15.104:0/3924635167 conn(0x7f094406b8a0 msgr2=0x7f094406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:47.533 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 shutdown_connections 2023-06-10T18:43:47.533 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:47.527+0000 7f094c07c700 1 -- 172.21.15.104:0/3924635167 wait complete. 2023-06-10T18:43:47.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-10T18:43:47.566 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.561+0000 7f55327f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-10T18:43:47.639 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.635+0000 7f55327f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-10T18:43:47.643 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:47.638+0000 7f7890154200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-10T18:43:47.862 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:47.856+0000 7f7890154200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-10T18:43:47.913 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:47.908+0000 7f55327f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-10T18:43:48.019 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:48.014+0000 7f7890154200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-10T18:43:48.130 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:48.125+0000 7f55327f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-10T18:43:48.135 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:48.129+0000 7f7890154200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-10T18:43:48.283 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:48.277+0000 7f7890154200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-10T18:43:48.286 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:48.282+0000 7f55327f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-10T18:43:48.401 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:48.396+0000 7f55327f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-10T18:43:48.550 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:48.546+0000 7f55327f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-10T18:43:48.557 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:48.552+0000 7f7890154200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-10T18:43:48.742 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:48.736+0000 7f7890154200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-10T18:43:48.822 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:48.818+0000 7f55327f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-10T18:43:49.006 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:49.001+0000 7f55327f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-10T18:43:49.329 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:49.324+0000 7f7890154200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-10T18:43:49.603 INFO:tasks.ceph.mgr.y.smithi104.stderr:2023-06-10T18:43:49.599+0000 7f55327f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-10T18:43:50.354 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:50.349+0000 7f7848380700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-10T18:43:50.355 INFO:tasks.ceph.mgr.x.smithi114.stderr:2023-06-10T18:43:50.349+0000 7f7848380700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-10T18:43:50.389 INFO:tasks.ceph.osd.2.smithi104.stderr:2023-06-10T18:43:50.384+0000 7f2b575f7640 -1 osd.2 0 log_to_monitors true 2023-06-10T18:43:50.401 INFO:tasks.ceph.osd.3.smithi104.stderr:2023-06-10T18:43:50.397+0000 7fe833a18640 -1 osd.3 0 log_to_monitors true 2023-06-10T18:43:50.413 INFO:tasks.ceph.osd.0.smithi104.stderr:2023-06-10T18:43:50.409+0000 7fb5926d1640 -1 osd.0 0 log_to_monitors true 2023-06-10T18:43:50.413 INFO:tasks.ceph.osd.4.smithi114.stderr:2023-06-10T18:43:50.408+0000 7fd86e7e1640 -1 osd.4 0 log_to_monitors true 2023-06-10T18:43:50.419 INFO:tasks.ceph.osd.1.smithi104.stderr:2023-06-10T18:43:50.415+0000 7f1533953640 -1 osd.1 0 log_to_monitors true 2023-06-10T18:43:50.446 INFO:tasks.ceph.osd.5.smithi114.stderr:2023-06-10T18:43:50.440+0000 7f0b64df6640 -1 osd.5 0 log_to_monitors true 2023-06-10T18:43:50.454 INFO:tasks.ceph.osd.7.smithi114.stderr:2023-06-10T18:43:50.448+0000 7f12003b3640 -1 osd.7 0 log_to_monitors true 2023-06-10T18:43:50.463 INFO:tasks.ceph.osd.6.smithi114.stderr:2023-06-10T18:43:50.458+0000 7f9834a7b640 -1 osd.6 0 log_to_monitors true 2023-06-10T18:43:54.340 INFO:tasks.ceph.osd.4.smithi114.stderr:2023-06-10T18:43:54.335+0000 7fd862b99700 -1 osd.4 0 waiting for initial osdmap 2023-06-10T18:43:54.348 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-10T18:43:54.354 INFO:tasks.ceph.osd.4.smithi114.stderr:2023-06-10T18:43:54.349+0000 7fd85a773700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.379 INFO:tasks.ceph.osd.5.smithi114.stderr:2023-06-10T18:43:54.373+0000 7f0b599af700 -1 osd.5 0 waiting for initial osdmap 2023-06-10T18:43:54.386 INFO:tasks.ceph.osd.6.smithi114.stderr:2023-06-10T18:43:54.381+0000 7f9828e33700 -1 osd.6 0 waiting for initial osdmap 2023-06-10T18:43:54.389 INFO:tasks.ceph.osd.5.smithi114.stderr:2023-06-10T18:43:54.384+0000 7f0b50d88700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.396 INFO:tasks.ceph.osd.6.smithi114.stderr:2023-06-10T18:43:54.391+0000 7f9820a0d700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.412 INFO:tasks.ceph.osd.1.smithi104.stderr:2023-06-10T18:43:54.408+0000 7f1527d0b700 -1 osd.1 0 waiting for initial osdmap 2023-06-10T18:43:54.419 INFO:tasks.ceph.osd.7.smithi114.stderr:2023-06-10T18:43:54.413+0000 7f11f476b700 -1 osd.7 0 waiting for initial osdmap 2023-06-10T18:43:54.421 INFO:tasks.ceph.osd.2.smithi104.stderr:2023-06-10T18:43:54.417+0000 7f2b4b9af700 -1 osd.2 0 waiting for initial osdmap 2023-06-10T18:43:54.429 INFO:tasks.ceph.osd.1.smithi104.stderr:2023-06-10T18:43:54.424+0000 7f151f8e5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.430 INFO:tasks.ceph.osd.0.smithi104.stderr:2023-06-10T18:43:54.425+0000 7fb586a89700 -1 osd.0 0 waiting for initial osdmap 2023-06-10T18:43:54.433 INFO:tasks.ceph.osd.3.smithi104.stderr:2023-06-10T18:43:54.429+0000 7fe827dd0700 -1 osd.3 0 waiting for initial osdmap 2023-06-10T18:43:54.433 INFO:tasks.ceph.osd.7.smithi114.stderr:2023-06-10T18:43:54.428+0000 7f11ec345700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.437 INFO:tasks.ceph.osd.2.smithi104.stderr:2023-06-10T18:43:54.433+0000 7f2b43589700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.443 INFO:tasks.ceph.osd.0.smithi104.stderr:2023-06-10T18:43:54.439+0000 7fb57e663700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.447 INFO:tasks.ceph.osd.3.smithi104.stderr:2023-06-10T18:43:54.443+0000 7fe81f9aa700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-10T18:43:54.528 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.523+0000 7f3da377b700 1 Processor -- start 2023-06-10T18:43:54.528 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.523+0000 7f3da377b700 1 -- start start 2023-06-10T18:43:54.529 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da377b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.529 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da377b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da377b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0cccf0 con 0x7f3d9c159ff0 2023-06-10T18:43:54.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da377b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0ccfd0 con 0x7f3d9c15a3c0 2023-06-10T18:43:54.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da377b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0cd2b0 con 0x7f3d9c0cc830 2023-06-10T18:43:54.530 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da1d18700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da0d16700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da1d18700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41282/0 (socket says 172.21.15.104:41282) 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da1517700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3d9c159ff0 0x7f3d9c0cae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:57450/0 (socket says 172.21.15.104:57450) 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da1d18700 1 -- 172.21.15.104:0/2874958951 learned_addr learned my addr 172.21.15.104:0/2874958951 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.524+0000 7f3da0d16700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:44930/0 (socket says 172.21.15.104:44930) 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da0d16700 1 -- 172.21.15.104:0/2874958951 >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 msgr2=0x7f3d9c0b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:43:54.531 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da0d16700 1 --2- 172.21.15.104:0/2874958951 >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da0d16700 1 -- 172.21.15.104:0/2874958951 >> v1:172.21.15.114:6789/0 conn(0x7f3d9c159ff0 legacy=0x7f3d9c0cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da0d16700 1 -- 172.21.15.104:0/2874958951 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0cd590 con 0x7f3d9c0cc830 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3d93fff700 1 -- 172.21.15.104:0/2874958951 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269465388 0 0) 0x7f3d9c0cccf0 con 0x7f3d9c159ff0 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da1d18700 1 --2- 172.21.15.104:0/2874958951 >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3da0d16700 1 --2- 172.21.15.104:0/2874958951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009840 tx=0x7f3d98009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a18862bd7fe5c3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3d93fff700 1 -- 172.21.15.104:0/2874958951 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98011830 con 0x7f3d9c0cc830 2023-06-10T18:43:54.532 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3d93fff700 1 -- 172.21.15.104:0/2874958951 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d98011990 con 0x7f3d9c0cc830 2023-06-10T18:43:54.533 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.525+0000 7f3d93fff700 1 -- 172.21.15.104:0/2874958951 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98011d20 con 0x7f3d9c0cc830 2023-06-10T18:43:54.534 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 -- 172.21.15.104:0/2874958951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 msgr2=0x7f3d9c0b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 --2- 172.21.15.104:0/2874958951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009840 tx=0x7f3d98009b50 comp rx=0 tx=0).stop 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 -- 172.21.15.104:0/2874958951 shutdown_connections 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 --2- 172.21.15.104:0/2874958951 >> v2:172.21.15.114:3300/0 conn(0x7f3d9c15a3c0 0x7f3d9c0b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 --2- 172.21.15.104:0/2874958951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c0cc830 0x7f3d9c0b2290 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 -- 172.21.15.104:0/2874958951 >> 172.21.15.104:0/2874958951 conn(0x7f3d9c06e4a0 msgr2=0x7f3d9c06e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:54.535 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 -- 172.21.15.104:0/2874958951 shutdown_connections 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.526+0000 7f3da377b700 1 -- 172.21.15.104:0/2874958951 wait complete. 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.527+0000 7f3da377b700 1 Processor -- start 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.527+0000 7f3da377b700 1 -- start start 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d9c0cc830 0x7f3d9c14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c15a3c0 0x7f3d9c15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0c5040 con 0x7f3d9c159ff0 2023-06-10T18:43:54.536 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c14d140 con 0x7f3d9c0cc830 2023-06-10T18:43:54.537 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da377b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0c43c0 con 0x7f3d9c15a3c0 2023-06-10T18:43:54.537 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da0d16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.537 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da0d16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53784/0 (socket says 172.21.15.104:53784) 2023-06-10T18:43:54.537 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.528+0000 7f3da0d16700 1 -- 172.21.15.104:0/383567414 learned_addr learned my addr 172.21.15.104:0/383567414 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:43:54.538 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da1d18700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c15a3c0 0x7f3d9c15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.538 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da1517700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d9c0cc830 0x7f3d9c14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.538 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 -- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c15a3c0 msgr2=0x7f3d9c15e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.538 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c15a3c0 0x7f3d9c15e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.539 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 -- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d9c0cc830 msgr2=0x7f3d9c14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.539 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d9c0cc830 0x7f3d9c14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.539 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 -- 172.21.15.104:0/383567414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d94009b30 con 0x7f3d9c159ff0 2023-06-10T18:43:54.539 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da0d16700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009fd0 tx=0x7f3d98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:54.540 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d980036a0 con 0x7f3d9c159ff0 2023-06-10T18:43:54.540 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9801f070 con 0x7f3d9c159ff0 2023-06-10T18:43:54.540 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d980094a0 con 0x7f3d9c159ff0 2023-06-10T18:43:54.540 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98003a60 con 0x7f3d9c159ff0 2023-06-10T18:43:54.541 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.529+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c0c4980 con 0x7f3d9c159ff0 2023-06-10T18:43:54.541 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.530+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c0c8180 con 0x7f3d9c159ff0 2023-06-10T18:43:54.542 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.531+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) v1 ==== 88981+0+0 (secure 0 0 0) 0x7f3d98003bc0 con 0x7f3d9c159ff0 2023-06-10T18:43:54.542 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.531+0000 7f3d91ffb700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 0x7f3d8006d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:43:54.542 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.531+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f3d98021070 con 0x7f3d9c159ff0 2023-06-10T18:43:54.542 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.531+0000 7f3da1517700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 0x7f3d8006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:43:54.543 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.531+0000 7f3da1517700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 0x7f3d8006d100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c007fd0 tx=0x7f3d8c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:43:54.543 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.536+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d98055d10 con 0x7f3d9c159ff0 2023-06-10T18:43:54.726 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.722+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d9c0b2f90 con 0x7f3d9c159ff0 2023-06-10T18:43:54.727 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.722+0000 7f3d91ffb700 1 -- 172.21.15.104:0/383567414 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3d98015090 con 0x7f3d9c159ff0 2023-06-10T18:43:54.727 INFO:teuthology.misc.health.smithi104.stdout: 2023-06-10T18:43:54.727 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":12,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:43:52.345759+0000","last_up_change":"0.000000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:43:54.729 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.724+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 msgr2=0x7f3d8006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.729 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.724+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 0x7f3d8006d100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c007fd0 tx=0x7f3d8c007de0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.729 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.724+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 msgr2=0x7f3d9c15dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.724+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009fd0 tx=0x7f3d98004750 comp rx=0 tx=0).stop 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 shutdown_connections 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d8006ac50 0x7f3d8006d100 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d9c0cc830 0x7f3d9c14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d9c15a3c0 0x7f3d9c15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 --2- 172.21.15.104:0/383567414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d9c159ff0 0x7f3d9c15dc60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:43:54.730 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 >> 172.21.15.104:0/383567414 conn(0x7f3d9c06e4a0 msgr2=0x7f3d9c0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:43:54.731 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 shutdown_connections 2023-06-10T18:43:54.731 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:43:54.725+0000 7f3da377b700 1 -- 172.21.15.104:0/383567414 wait complete. 2023-06-10T18:43:54.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-10T18:44:01.545 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:01.667 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.662+0000 7fda2ce3e700 1 Processor -- start 2023-06-10T18:44:01.667 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.662+0000 7fda2ce3e700 1 -- start start 2023-06-10T18:44:01.669 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda2815d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.670 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda280b2290 0x7fda280b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.670 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda2815fea0 0x7fda28160320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.670 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2815d870 con 0x7fda280b2290 2023-06-10T18:44:01.670 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda280b2c40 con 0x7fda2815fea0 2023-06-10T18:44:01.670 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda2ce3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda280cebb0 con 0x7fda280cda00 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda25d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda280b2290 0x7fda280b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda25d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda280b2290 0x7fda280b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53974/0 (socket says 172.21.15.104:53974) 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.663+0000 7fda25d9b700 1 -- 172.21.15.104:0/2149461374 learned_addr learned my addr 172.21.15.104:0/2149461374 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda2659c700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda2815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda26d9d700 1 --2- 172.21.15.104:0/2149461374 >> v2:172.21.15.114:3300/0 conn(0x7fda2815fea0 0x7fda28160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 -- 172.21.15.104:0/2149461374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 msgr2=0x7fda2815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:01.671 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda2815d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 -- 172.21.15.104:0/2149461374 >> v2:172.21.15.114:3300/0 conn(0x7fda2815fea0 msgr2=0x7fda28160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 --2- 172.21.15.104:0/2149461374 >> v2:172.21.15.114:3300/0 conn(0x7fda2815fea0 0x7fda28160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 -- 172.21.15.104:0/2149461374 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280d0410 con 0x7fda280b2290 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda2659c700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda2815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda25d9b700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda280b2290 0x7fda280b2700 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fda2806adc0 tx=0x7fda18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882f9dbd3b7da6de server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:01.672 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda2559a700 1 -- 172.21.15.104:0/2149461374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1800b870 con 0x7fda280b2290 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda2559a700 1 -- 172.21.15.104:0/2149461374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1800b9d0 con 0x7fda280b2290 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.664+0000 7fda2559a700 1 -- 172.21.15.104:0/2149461374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1800bd60 con 0x7fda280b2290 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 -- 172.21.15.104:0/2149461374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda280b2290 msgr2=0x7fda280b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda280b2290 0x7fda280b2700 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fda2806adc0 tx=0x7fda18009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 -- 172.21.15.104:0/2149461374 shutdown_connections 2023-06-10T18:44:01.673 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/2149461374 >> v2:172.21.15.114:3300/0 conn(0x7fda2815fea0 0x7fda28160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.674 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda2815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.674 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/2149461374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda280b2290 0x7fda280b2700 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.674 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 -- 172.21.15.104:0/2149461374 >> 172.21.15.104:0/2149461374 conn(0x7fda2806e4a0 msgr2=0x7fda2806e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:01.674 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 -- 172.21.15.104:0/2149461374 shutdown_connections 2023-06-10T18:44:01.674 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.665+0000 7fda2ce3e700 1 -- 172.21.15.104:0/2149461374 wait complete. 2023-06-10T18:44:01.676 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.666+0000 7fda2ce3e700 1 Processor -- start 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.666+0000 7fda2ce3e700 1 -- start start 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 0x7fda280b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda280c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda281f1d50 con 0x7fda2815fea0 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda280c5f20 con 0x7fda280b2290 2023-06-10T18:44:01.677 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2ce3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda280c6200 con 0x7fda280cda00 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53982/0 (socket says 172.21.15.104:53982) 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 -- 172.21.15.104:0/3201888673 learned_addr learned my addr 172.21.15.104:0/3201888673 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda25d9b700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda280c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2659c700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 0x7fda280b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 -- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 msgr2=0x7fda280c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:01.678 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda280c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 -- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 msgr2=0x7fda280b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 0x7fda280b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 -- 172.21.15.104:0/3201888673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009550 con 0x7fda2815fea0 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda2659c700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 0x7fda280b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.667+0000 7fda26d9d700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda20007e40 tx=0x7fda2000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.668+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2000eea0 con 0x7fda2815fea0 2023-06-10T18:44:01.679 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.668+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20014420 con 0x7fda2815fea0 2023-06-10T18:44:01.680 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.668+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda280c64e0 con 0x7fda2815fea0 2023-06-10T18:44:01.680 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.668+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200147b0 con 0x7fda2815fea0 2023-06-10T18:44:01.680 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.668+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda280b16c0 con 0x7fda2815fea0 2023-06-10T18:44:01.680 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.669+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda281f1ea0 con 0x7fda2815fea0 2023-06-10T18:44:01.680 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.669+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fda20014910 con 0x7fda2815fea0 2023-06-10T18:44:01.681 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.670+0000 7fda16ffd700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 0x7fda0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:01.681 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.670+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda200898b0 con 0x7fda2815fea0 2023-06-10T18:44:01.681 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.670+0000 7fda2659c700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 0x7fda0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:01.681 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.670+0000 7fda2659c700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 0x7fda0806d190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:01.682 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.674+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fda281f1ea0 con 0x7fda2815fea0 2023-06-10T18:44:01.868 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.863+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda281f1ea0 con 0x7fda2815fea0 2023-06-10T18:44:01.868 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.864+0000 7fda16ffd700 1 -- 172.21.15.104:0/3201888673 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12898 (secure 0 0 0) 0x7fda281f1ea0 con 0x7fda2815fea0 2023-06-10T18:44:01.869 INFO:teuthology.misc.health.smithi104.stdout: 2023-06-10T18:44:01.869 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":16,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:43:58.385085+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:01.871 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 msgr2=0x7fda0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:01.871 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 0x7fda0806d190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:01.871 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 msgr2=0x7fda280c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:01.871 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda20007e40 tx=0x7fda2000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.871 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 shutdown_connections 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda0806ace0 0x7fda0806d190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda280b2290 0x7fda280b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda2815fea0 0x7fda280c9050 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 --2- 172.21.15.104:0/3201888673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda280cda00 0x7fda280c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 >> 172.21.15.104:0/3201888673 conn(0x7fda2806e4a0 msgr2=0x7fda28072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 shutdown_connections 2023-06-10T18:44:01.872 INFO:teuthology.misc.health.smithi104.stderr:2023-06-10T18:44:01.866+0000 7fda2ce3e700 1 -- 172.21.15.104:0/3201888673 wait complete. 2023-06-10T18:44:01.881 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-10T18:44:01.882 INFO:tasks.ceph:Creating RBD pool 2023-06-10T18:44:01.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-10T18:44:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.996+0000 7f26e3655700 1 Processor -- start 2023-06-10T18:44:02.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.996+0000 7f26e3655700 1 -- start start 2023-06-10T18:44:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e3655700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f26dc1588f0 0x7f26dc0af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e3655700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26dc071230 0x7f26dc069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e3655700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc158cc0 con 0x7f26dc06a3b0 2023-06-10T18:44:02.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e3655700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26dc06a7b0 con 0x7f26dc071230 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e3655700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26dc0b0120 con 0x7f26dc1588f0 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e0bf0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26dc071230 0x7f26dc069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e1bf2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f26dc06a3b0 0x7f26dc06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51818/0 (socket says 172.21.15.104:51818) 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.997+0000 7f26e1bf2700 1 -- 172.21.15.104:0/53095322 learned_addr learned my addr 172.21.15.104:0/53095322 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e13f1700 1 --2- 172.21.15.104:0/53095322 >> v2:172.21.15.114:3300/0 conn(0x7f26dc1588f0 0x7f26dc0af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e0bf0700 1 -- 172.21.15.104:0/53095322 >> v2:172.21.15.114:3300/0 conn(0x7f26dc1588f0 msgr2=0x7f26dc0af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:02.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e0bf0700 1 --2- 172.21.15.104:0/53095322 >> v2:172.21.15.114:3300/0 conn(0x7f26dc1588f0 0x7f26dc0af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e0bf0700 1 -- 172.21.15.104:0/53095322 >> v1:172.21.15.104:6789/0 conn(0x7f26dc06a3b0 legacy=0x7f26dc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e0bf0700 1 -- 172.21.15.104:0/53095322 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc070470 con 0x7f26dc071230 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26d3fff700 1 -- 172.21.15.104:0/53095322 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191926372 0 0) 0x7f26dc158cc0 con 0x7f26dc06a3b0 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e0bf0700 1 --2- 172.21.15.104:0/53095322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc069e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f26dc070260 tx=0x7f26d800aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9686e6890af9248c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26d3fff700 1 -- 172.21.15.104:0/53095322 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8005e30 con 0x7f26dc071230 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26d3fff700 1 -- 172.21.15.104:0/53095322 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d800c760 con 0x7f26dc071230 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26d3fff700 1 -- 172.21.15.104:0/53095322 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800caf0 con 0x7f26dc071230 2023-06-10T18:44:02.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e3655700 1 -- 172.21.15.104:0/53095322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 msgr2=0x7f26dc069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.998+0000 7f26e3655700 1 --2- 172.21.15.104:0/53095322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc069e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f26dc070260 tx=0x7f26d800aca0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 -- 172.21.15.104:0/53095322 shutdown_connections 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 --2- 172.21.15.104:0/53095322 >> v2:172.21.15.114:3300/0 conn(0x7f26dc1588f0 0x7f26dc0af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 --2- 172.21.15.104:0/53095322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc069e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 -- 172.21.15.104:0/53095322 >> 172.21.15.104:0/53095322 conn(0x7f26dc0ce940 msgr2=0x7f26dc0ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 -- 172.21.15.104:0/53095322 shutdown_connections 2023-06-10T18:44:02.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:01.999+0000 7f26e3655700 1 -- 172.21.15.104:0/53095322 wait complete. 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.000+0000 7f26e3655700 1 Processor -- start 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.000+0000 7f26e3655700 1 -- start start 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26dc06a3b0 0x7f26dc157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26dc1588f0 0x7f26dc0b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0b35c0 con 0x7f26dc071230 2023-06-10T18:44:02.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc1ed400 con 0x7f26dc06a3b0 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc1f0e40 con 0x7f26dc1588f0 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:53994/0 (socket says 172.21.15.104:53994) 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 -- 172.21.15.104:0/2857883738 learned_addr learned my addr 172.21.15.104:0/2857883738 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e1bf2700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26dc1588f0 0x7f26dc0b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e13f1700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26dc06a3b0 0x7f26dc157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 -- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26dc1588f0 msgr2=0x7f26dc0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26dc1588f0 0x7f26dc0b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 -- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26dc06a3b0 msgr2=0x7f26dc157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26dc06a3b0 0x7f26dc157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 -- 172.21.15.104:0/2857883738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d800a5d0 con 0x7f26dc071230 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e0bf0700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f26d800ace0 tx=0x7f26d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800cea0 con 0x7f26dc071230 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d800cae0 con 0x7f26dc071230 2023-06-10T18:44:02.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc1f1120 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8004240 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.001+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc1f16e0 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.002+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc074050 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.003+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f26d80043a0 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.003+0000 7f26d1ffb700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 0x7f26c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.003+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26d808b790 con 0x7f26dc071230 2023-06-10T18:44:02.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.003+0000 7f26e13f1700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 0x7f26c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.004+0000 7f26e13f1700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 0x7f26c006d190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00b9b0 tx=0x7f26cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.008+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f26d8056d80 con 0x7f26dc071230 2023-06-10T18:44:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.195+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f26dc06c420 con 0x7f26dc071230 2023-06-10T18:44:02.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.532+0000 7f26d1ffb700 1 -- 172.21.15.104:0/2857883738 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f26d805a3a0 con 0x7f26dc071230 2023-06-10T18:44:02.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 msgr2=0x7f26c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 0x7f26c006d190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00b9b0 tx=0x7f26cc008040 comp rx=0 tx=0).stop 2023-06-10T18:44:02.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 msgr2=0x7f26dc0b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f26d800ace0 tx=0x7f26d8004750 comp rx=0 tx=0).stop 2023-06-10T18:44:02.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 shutdown_connections 2023-06-10T18:44:02.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f26c006ace0 0x7f26c006d190 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26dc06a3b0 0x7f26dc157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26dc1588f0 0x7f26dc0b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 --2- 172.21.15.104:0/2857883738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26dc071230 0x7f26dc0b3ca0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 >> 172.21.15.104:0/2857883738 conn(0x7f26dc0ce940 msgr2=0x7f26dc154420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:02.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 shutdown_connections 2023-06-10T18:44:02.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.535+0000 7f26e3655700 1 -- 172.21.15.104:0/2857883738 wait complete. 2023-06-10T18:44:02.544 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'rbd' created 2023-06-10T18:44:02.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-10T18:44:02.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.668+0000 7fec11478700 1 Processor -- start 2023-06-10T18:44:02.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.668+0000 7fec11478700 1 -- start start 2023-06-10T18:44:02.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec11478700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec11478700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec0c06ea70 0x7fec0c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec11478700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0ae7e0 con 0x7fec0c0aec00 2023-06-10T18:44:02.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec11478700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0c06df90 con 0x7fec0c06ea70 2023-06-10T18:44:02.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec11478700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c06e270 con 0x7fec0c15c6f0 2023-06-10T18:44:02.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec0a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec0c06ea70 0x7fec0c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec0a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fec0c06ea70 0x7fec0c0b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54000/0 (socket says 172.21.15.104:54000) 2023-06-10T18:44:02.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1304537402 learned_addr learned my addr 172.21.15.104:0/1304537402 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:02.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.669+0000 7fec0affd700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1304537402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 msgr2=0x7fec0c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:02.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0a7fc700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1304537402 >> v1:172.21.15.114:6789/0 conn(0x7fec0c0aec00 legacy=0x7fec0c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1304537402 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c06e550 con 0x7fec0c06ea70 2023-06-10T18:44:02.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0affd700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:02.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec0a7fc700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c06ea70 0x7fec0c0b15b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febfc0098f0 tx=0x7febfc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdde40d51d6ece2a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec09ffb700 1 -- 172.21.15.104:0/1304537402 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00b8a0 con 0x7fec0c06ea70 2023-06-10T18:44:02.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec09ffb700 1 -- 172.21.15.104:0/1304537402 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00ba00 con 0x7fec0c06ea70 2023-06-10T18:44:02.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec09ffb700 1 -- 172.21.15.104:0/1304537402 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00bd90 con 0x7fec0c06ea70 2023-06-10T18:44:02.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 -- 172.21.15.104:0/1304537402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c06ea70 msgr2=0x7fec0c0b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c06ea70 0x7fec0c0b15b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febfc0098f0 tx=0x7febfc009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:02.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 -- 172.21.15.104:0/1304537402 shutdown_connections 2023-06-10T18:44:02.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 --2- 172.21.15.104:0/1304537402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c06ea70 0x7fec0c0b15b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.670+0000 7fec11478700 1 -- 172.21.15.104:0/1304537402 >> 172.21.15.104:0/1304537402 conn(0x7fec0c06b8a0 msgr2=0x7fec0c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:02.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.671+0000 7fec11478700 1 -- 172.21.15.104:0/1304537402 shutdown_connections 2023-06-10T18:44:02.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.671+0000 7fec11478700 1 -- 172.21.15.104:0/1304537402 wait complete. 2023-06-10T18:44:02.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.672+0000 7fec11478700 1 Processor -- start 2023-06-10T18:44:02.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.672+0000 7fec11478700 1 -- start start 2023-06-10T18:44:02.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec0c06ea70 0x7fec0c152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0c3090 con 0x7fec0c0aec00 2023-06-10T18:44:02.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0c5bd0 con 0x7fec0c06ea70 2023-06-10T18:44:02.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec11478700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0c5eb0 con 0x7fec0c15c6f0 2023-06-10T18:44:02.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54006/0 (socket says 172.21.15.104:54006) 2023-06-10T18:44:02.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1475032694 learned_addr learned my addr 172.21.15.104:0/1475032694 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:02.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec0b7fe700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.673+0000 7fec0affd700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec0c06ea70 0x7fec0c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 msgr2=0x7fec0c0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec0c06ea70 msgr2=0x7fec0c152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:02.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec0c06ea70 0x7fec0c152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:02.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 -- 172.21.15.104:0/1475032694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc009550 con 0x7fec0c0aec00 2023-06-10T18:44:02.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec0a7fc700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febfc009c40 tx=0x7febfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc004400 con 0x7fec0c0aec00 2023-06-10T18:44:02.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc01f070 con 0x7fec0c0aec00 2023-06-10T18:44:02.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0038a0 con 0x7fec0c0aec00 2023-06-10T18:44:02.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c0c32e0 con 0x7fec0c0aec00 2023-06-10T18:44:02.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.674+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c0c5860 con 0x7fec0c0aec00 2023-06-10T18:44:02.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.675+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c0c3800 con 0x7fec0c0aec00 2023-06-10T18:44:02.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.675+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7febfc021030 con 0x7fec0c0aec00 2023-06-10T18:44:02.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.676+0000 7febf37fe700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 0x7febf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:02.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.676+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7febfc08a710 con 0x7fec0c0aec00 2023-06-10T18:44:02.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.676+0000 7fec0affd700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 0x7febf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:02.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.676+0000 7fec0affd700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 0x7febf806d190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c920 tx=0x7fec00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:02.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.680+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fec0c0c3800 con 0x7fec0c0aec00 2023-06-10T18:44:02.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:02.868+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fec0c0c3800 con 0x7fec0c0aec00 2023-06-10T18:44:03.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.536+0000 7febf37fe700 1 -- 172.21.15.104:0/1475032694 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7fec0c0c3800 con 0x7fec0c0aec00 2023-06-10T18:44:03.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 msgr2=0x7febf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 0x7febf806d190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec0000c920 tx=0x7fec00008040 comp rx=0 tx=0).stop 2023-06-10T18:44:03.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 msgr2=0x7fec0c0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febfc009c40 tx=0x7febfc004750 comp rx=0 tx=0).stop 2023-06-10T18:44:03.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 shutdown_connections 2023-06-10T18:44:03.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7febf806ace0 0x7febf806d190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec0c06ea70 0x7fec0c152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec0c15c6f0 0x7fec0c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 --2- 172.21.15.104:0/1475032694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec0c0aec00 0x7fec0c0b4b00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 >> 172.21.15.104:0/1475032694 conn(0x7fec0c06b8a0 msgr2=0x7fec0c0b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:03.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 shutdown_connections 2023-06-10T18:44:03.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.539+0000 7fec11478700 1 -- 172.21.15.104:0/1475032694 wait complete. 2023-06-10T18:44:03.548 INFO:teuthology.orchestra.run.smithi104.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-10T18:44:03.557 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-10T18:44:03.558 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:44:03.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:44:03.560 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:44:03.561 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:44:03.561 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:44:03.563 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:44:03.564 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:44:03.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:44:03.566 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:44:03.566 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:44:03.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:44:03.568 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:44:03.569 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-10T18:44:03.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:03.640 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:44:03.642 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:44:03.642 INFO:tasks.ceph.mds.a.smithi104.stderr:2023-06-10T18:44:03.637+0000 7fed4e924700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6833/0: (98) Address already in use 2023-06-10T18:44:03.654 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:44:03.655 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:44:03.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.693+0000 7f2639bfb700 1 Processor -- start 2023-06-10T18:44:03.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.693+0000 7f2639bfb700 1 -- start start 2023-06-10T18:44:03.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2639bfb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26340ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2639bfb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26340ade00 0x7f26340ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2639bfb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26340cec20 con 0x7f26340ce0b0 2023-06-10T18:44:03.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2639bfb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26340cf430 con 0x7f26340ade00 2023-06-10T18:44:03.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2639bfb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340cee70 con 0x7f26340ada30 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26340ade00 0x7f26340ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f26340ade00 0x7f26340ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54040/0 (socket says 172.21.15.104:54040) 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2633fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f26340ce0b0 0x7f26340ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51870/0 (socket says 172.21.15.104:51870) 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 -- 172.21.15.104:0/348681000 learned_addr learned my addr 172.21.15.104:0/348681000 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f26337fe700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26340ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 -- 172.21.15.104:0/348681000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 msgr2=0x7f26340ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:03.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26340ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 -- 172.21.15.104:0/348681000 >> v1:172.21.15.104:6789/0 conn(0x7f26340ce0b0 legacy=0x7f26340ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f2632ffd700 1 -- 172.21.15.104:0/348681000 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340cf150 con 0x7f26340ade00 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f26327fc700 1 -- 172.21.15.104:0/348681000 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691913631 0 0) 0x7f26340cec20 con 0x7f26340ce0b0 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.694+0000 7f26337fe700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26340ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2632ffd700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26340ade00 0x7f26340ae930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26240098f0 tx=0x7f2624009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c3e7bf8a3853d5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:03.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f26327fc700 1 -- 172.21.15.104:0/348681000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400b8a0 con 0x7f26340ade00 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f26327fc700 1 -- 172.21.15.104:0/348681000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262400ba00 con 0x7f26340ade00 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f26327fc700 1 -- 172.21.15.104:0/348681000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400bd90 con 0x7f26340ade00 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 -- 172.21.15.104:0/348681000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26340ade00 msgr2=0x7f26340ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26340ade00 0x7f26340ae930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f26240098f0 tx=0x7f2624009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 -- 172.21.15.104:0/348681000 shutdown_connections 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26340ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 --2- 172.21.15.104:0/348681000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26340ade00 0x7f26340ae930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 -- 172.21.15.104:0/348681000 >> 172.21.15.104:0/348681000 conn(0x7f263406b8a0 msgr2=0x7f26340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:03.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 -- 172.21.15.104:0/348681000 shutdown_connections 2023-06-10T18:44:03.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.695+0000 7f2639bfb700 1 -- 172.21.15.104:0/348681000 wait complete. 2023-06-10T18:44:03.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.697+0000 7f2639bfb700 1 Processor -- start 2023-06-10T18:44:03.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.697+0000 7f2639bfb700 1 -- start start 2023-06-10T18:44:03.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26341e5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26340ce0b0 0x7f2634099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634099d20 con 0x7f2634099880 2023-06-10T18:44:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263409cf70 con 0x7f26340ce0b0 2023-06-10T18:44:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2639bfb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263409c360 con 0x7f26340ada30 2023-06-10T18:44:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54048/0 (socket says 172.21.15.104:54048) 2023-06-10T18:44:03.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 -- 172.21.15.104:0/1176474186 learned_addr learned my addr 172.21.15.104:0/1176474186 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:03.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f26337fe700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26341e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2632ffd700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26340ce0b0 0x7f2634099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 -- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 msgr2=0x7f26341e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26341e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 -- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26340ce0b0 msgr2=0x7f2634099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26340ce0b0 0x7f2634099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.698+0000 7f2633fff700 1 -- 172.21.15.104:0/1176474186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624009550 con 0x7f2634099880 2023-06-10T18:44:03.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f2633fff700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f262c00cc70 tx=0x7f262c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:03.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00aac0 con 0x7f2634099880 2023-06-10T18:44:03.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c00ac40 con 0x7f2634099880 2023-06-10T18:44:03.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263409c640 con 0x7f2634099880 2023-06-10T18:44:03.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c0070d0 con 0x7f2634099880 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.699+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263409cc00 con 0x7f2634099880 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.700+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2634074050 con 0x7f2634099880 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.701+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f262c007250 con 0x7f2634099880 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.701+0000 7f261bfff700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 0x7f262006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.701+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f262c088fa0 con 0x7f2634099880 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.701+0000 7f26337fe700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 0x7f262006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:03.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.701+0000 7f26337fe700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 0x7f262006d1c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2628009930 tx=0x7f2628008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:03.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.705+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f262c0bb310 con 0x7f2634099880 2023-06-10T18:44:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.895+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26340cd6d0 con 0x7f2634099880 2023-06-10T18:44:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.896+0000 7f261bfff700 1 -- 172.21.15.104:0/1176474186 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14697 (secure 0 0 0) 0x7f262c018070 con 0x7f2634099880 2023-06-10T18:44:03.900 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:03.901 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":18,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:03.529241+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 msgr2=0x7f262006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 0x7f262006d1c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2628009930 tx=0x7f2628008040 comp rx=0 tx=0).stop 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 msgr2=0x7f263409b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f262c00cc70 tx=0x7f262c00cf80 comp rx=0 tx=0).stop 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 shutdown_connections 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2634099880 0x7f263409b5f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f26340ada30 0x7f26341e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f262006ad10 0x7f262006d1c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 --2- 172.21.15.104:0/1176474186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26340ce0b0 0x7f2634099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 >> 172.21.15.104:0/1176474186 conn(0x7f263406b8a0 msgr2=0x7f26340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 shutdown_connections 2023-06-10T18:44:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:03.898+0000 7f2639bfb700 1 -- 172.21.15.104:0/1176474186 wait complete. 2023-06-10T18:44:03.914 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:44:03.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:04.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.033+0000 7f6872265700 1 Processor -- start 2023-06-10T18:44:04.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.033+0000 7f6872265700 1 -- start start 2023-06-10T18:44:04.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f6872265700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f686c06cde0 0x7f686c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f6872265700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c06e250 con 0x7f686c15c580 2023-06-10T18:44:04.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f6872265700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c15c950 con 0x7f686c06e6e0 2023-06-10T18:44:04.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f6872265700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686c0b1e80 con 0x7f686c06cde0 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f686bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f686c15c580 0x7f686c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51880/0 (socket says 172.21.15.104:51880) 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f686bfff700 1 -- 172.21.15.104:0/3122790744 learned_addr learned my addr 172.21.15.104:0/3122790744 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.034+0000 7f6870802700 1 --2- 172.21.15.104:0/3122790744 >> v2:172.21.15.104:3300/0 conn(0x7f686c06cde0 0x7f686c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f686affd700 1 -- 172.21.15.104:0/3122790744 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852654103 0 0) 0x7f686c06e250 con 0x7f686c15c580 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f686affd700 1 -- 172.21.15.104:0/3122790744 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68540035b0 con 0x7f686c15c580 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6870802700 1 -- 172.21.15.104:0/3122790744 >> v1:172.21.15.114:6789/0 conn(0x7f686c06e6e0 legacy=0x7f686c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6870802700 1 -- 172.21.15.104:0/3122790744 >> v1:172.21.15.104:6789/0 conn(0x7f686c15c580 legacy=0x7f686c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6870802700 1 -- 172.21.15.104:0/3122790744 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c06d920 con 0x7f686c06cde0 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6870802700 1 --2- 172.21.15.104:0/3122790744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c0b1780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6864009d90 tx=0x7f686400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73105a49a0ef7404 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f686affd700 1 -- 172.21.15.104:0/3122790744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686400e070 con 0x7f686c06cde0 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f686affd700 1 -- 172.21.15.104:0/3122790744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686400bb20 con 0x7f686c06cde0 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f686affd700 1 -- 172.21.15.104:0/3122790744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6864012430 con 0x7f686c06cde0 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6872265700 1 -- 172.21.15.104:0/3122790744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 msgr2=0x7f686c0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6872265700 1 --2- 172.21.15.104:0/3122790744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c0b1780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6864009d90 tx=0x7f686400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:04.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6872265700 1 -- 172.21.15.104:0/3122790744 shutdown_connections 2023-06-10T18:44:04.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6872265700 1 --2- 172.21.15.104:0/3122790744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c0b1780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.035+0000 7f6872265700 1 -- 172.21.15.104:0/3122790744 >> 172.21.15.104:0/3122790744 conn(0x7f686c06b8a0 msgr2=0x7f686c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.036+0000 7f6872265700 1 -- 172.21.15.104:0/3122790744 shutdown_connections 2023-06-10T18:44:04.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.036+0000 7f6872265700 1 -- 172.21.15.104:0/3122790744 wait complete. 2023-06-10T18:44:04.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.037+0000 7f6872265700 1 Processor -- start 2023-06-10T18:44:04.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.037+0000 7f6872265700 1 -- start start 2023-06-10T18:44:04.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f686c06e6e0 0x7f686c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f686c15c580 0x7f686c0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c1487c0 con 0x7f686c06cde0 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c1e5bb0 con 0x7f686c06e6e0 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c1e5e90 con 0x7f686c15c580 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54056/0 (socket says 172.21.15.104:54056) 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 -- 172.21.15.104:0/3363181885 learned_addr learned my addr 172.21.15.104:0/3363181885 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:04.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6870802700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f686c15c580 0x7f686c0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686b7fe700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f686c06e6e0 0x7f686c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 -- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f686c15c580 msgr2=0x7f686c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f686c15c580 0x7f686c0b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 -- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f686c06e6e0 msgr2=0x7f686c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f686c06e6e0 0x7f686c0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 -- 172.21.15.104:0/3363181885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864009a70 con 0x7f686c06cde0 2023-06-10T18:44:04.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f686bfff700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6860001260 tx=0x7f686000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68600074e0 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6860007640 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c1e6170 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860005a50 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.038+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c1e6730 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.039+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c074050 con 0x7f686c06cde0 2023-06-10T18:44:04.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.040+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6860005bb0 con 0x7f686c06cde0 2023-06-10T18:44:04.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.040+0000 7f6868ff9700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 0x7f685806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.040+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f68600884a0 con 0x7f686c06cde0 2023-06-10T18:44:04.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.040+0000 7f686b7fe700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 0x7f685806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.041+0000 7f686b7fe700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 0x7f685806d190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f686c1e4500 tx=0x7f685c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.044+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6860053930 con 0x7f686c06cde0 2023-06-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.227+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f686c14a5a0 con 0x7f686c06cde0 2023-06-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.228+0000 7f6868ff9700 1 -- 172.21.15.104:0/3363181885 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6860007e60 con 0x7f686c06cde0 2023-06-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 msgr2=0x7f685806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 0x7f685806d190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f686c1e4500 tx=0x7f685c009040 comp rx=0 tx=0).stop 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 msgr2=0x7f686c14a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6860001260 tx=0x7f686000c7b0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 shutdown_connections 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f686c15c580 0x7f686c0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f686c06cde0 0x7f686c14a060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f685806ace0 0x7f685806d190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 --2- 172.21.15.104:0/3363181885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f686c06e6e0 0x7f686c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 >> 172.21.15.104:0/3363181885 conn(0x7f686c06b8a0 msgr2=0x7f686c0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 shutdown_connections 2023-06-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.230+0000 7f6872265700 1 -- 172.21.15.104:0/3363181885 wait complete. 2023-06-10T18:44:04.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:04.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.365+0000 7f22ef5bc700 1 Processor -- start 2023-06-10T18:44:04.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.365+0000 7f22ef5bc700 1 -- start start 2023-06-10T18:44:04.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ef5bc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 0x7f22e806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ef5bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 0x7f22e806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ef5bc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e80ae7e0 con 0x7f22e815c6f0 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ef5bc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e806df90 con 0x7f22e80b1780 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ef5bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e806e270 con 0x7f22e806ea70 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ed358700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f22e815c6f0 0x7f22e80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51892/0 (socket says 172.21.15.104:51892) 2023-06-10T18:44:04.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ed358700 1 -- 172.21.15.104:0/2535528859 learned_addr learned my addr 172.21.15.104:0/2535528859 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 --2- 172.21.15.104:0/2535528859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 0x7f22e806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ecb57700 1 --2- 172.21.15.104:0/2535528859 >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 0x7f22e806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368638257 0 0) 0x7f22e80ae7e0 con 0x7f22e815c6f0 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d40035b0 con 0x7f22e815c6f0 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 -- 172.21.15.104:0/2535528859 >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 msgr2=0x7f22e806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 --2- 172.21.15.104:0/2535528859 >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 0x7f22e806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 -- 172.21.15.104:0/2535528859 >> v1:172.21.15.104:6789/0 conn(0x7f22e815c6f0 legacy=0x7f22e80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131317372 0 0) 0x7f22d40035b0 con 0x7f22e815c6f0 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 -- 172.21.15.104:0/2535528859 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e806e550 con 0x7f22e806ea70 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22ecb57700 1 --2- 172.21.15.104:0/2535528859 >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 0x7f22e806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22edb59700 1 --2- 172.21.15.104:0/2535528859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 0x7f22e806d320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009d90 tx=0x7f22e000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6db0fc2d4a359b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.366+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e000e070 con 0x7f22e806ea70 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e0003a20 con 0x7f22e806ea70 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22dffff700 1 -- 172.21.15.104:0/2535528859 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d80017f0 con 0x7f22e806ea70 2023-06-10T18:44:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 -- 172.21.15.104:0/2535528859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 msgr2=0x7f22e806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/2535528859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 0x7f22e806d320 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009d90 tx=0x7f22e000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 -- 172.21.15.104:0/2535528859 shutdown_connections 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/2535528859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e806ea70 0x7f22e806d320 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/2535528859 >> v2:172.21.15.114:3300/0 conn(0x7f22e80b1780 0x7f22e806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 -- 172.21.15.104:0/2535528859 >> 172.21.15.104:0/2535528859 conn(0x7f22e806b8a0 msgr2=0x7f22e80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 -- 172.21.15.104:0/2535528859 shutdown_connections 2023-06-10T18:44:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.367+0000 7f22ef5bc700 1 -- 172.21.15.104:0/2535528859 wait complete. 2023-06-10T18:44:04.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.369+0000 7f22ef5bc700 1 Processor -- start 2023-06-10T18:44:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.369+0000 7f22ef5bc700 1 -- start start 2023-06-10T18:44:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 0x7f22e81f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22e815c6f0 0x7f22e81f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80ad490 con 0x7f22e806ea70 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80acc90 con 0x7f22e815c6f0 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e81f3400 con 0x7f22e80b1780 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54070/0 (socket says 172.21.15.104:54070) 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 -- 172.21.15.104:0/1897187459 learned_addr learned my addr 172.21.15.104:0/1897187459 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ecb57700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 0x7f22e81f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22edb59700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22e815c6f0 0x7f22e81f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 -- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 msgr2=0x7f22e81f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 0x7f22e81f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 -- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22e815c6f0 msgr2=0x7f22e81f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22e815c6f0 0x7f22e81f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 -- 172.21.15.104:0/1897187459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e40094a0 con 0x7f22e806ea70 2023-06-10T18:44:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ecb57700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 0x7f22e81f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ed358700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c750 tx=0x7f22d800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d80074e0 con 0x7f22e806ea70 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d8007660 con 0x7f22e806ea70 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e0009a70 con 0x7f22e806ea70 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8005a70 con 0x7f22e806ea70 2023-06-10T18:44:04.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.370+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e81f3dd0 con 0x7f22e806ea70 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.371+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e81f36e0 con 0x7f22e806ea70 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.372+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f22d8005bd0 con 0x7f22e806ea70 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.372+0000 7f22ddffb700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 0x7f22cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.372+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f22d8088750 con 0x7f22e806ea70 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.372+0000 7f22ecb57700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 0x7f22cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.373+0000 7f22ecb57700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 0x7f22cc06d190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22e815b9b0 tx=0x7f22e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.376+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22d8053ad0 con 0x7f22e806ea70 2023-06-10T18:44:04.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.563+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f22e8074050 con 0x7f22e806ea70 2023-06-10T18:44:04.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.563+0000 7f22ddffb700 1 -- 172.21.15.104:0/1897187459 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f22d80570f0 con 0x7f22e806ea70 2023-06-10T18:44:04.569 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:04.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 msgr2=0x7f22cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 0x7f22cc06d190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22e815b9b0 tx=0x7f22e4008040 comp rx=0 tx=0).stop 2023-06-10T18:44:04.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 msgr2=0x7f22e8153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c750 tx=0x7f22d800cb10 comp rx=0 tx=0).stop 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 shutdown_connections 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22cc06ace0 0x7f22cc06d190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22e815c6f0 0x7f22e81f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22e806ea70 0x7f22e8153cf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 --2- 172.21.15.104:0/1897187459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22e80b1780 0x7f22e81f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 >> 172.21.15.104:0/1897187459 conn(0x7f22e806b8a0 msgr2=0x7f22e8158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 shutdown_connections 2023-06-10T18:44:04.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.565+0000 7f22ef5bc700 1 -- 172.21.15.104:0/1897187459 wait complete. 2023-06-10T18:44:04.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.696+0000 7f21b932e700 1 Processor -- start 2023-06-10T18:44:04.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.696+0000 7f21b932e700 1 -- start start 2023-06-10T18:44:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b932e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 0x7f21b40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b932e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f21b408a3d0 0x7f21b40cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b932e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b40cadd0 con 0x7f21b406b1f0 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b932e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b40cfeb0 con 0x7f21b408a3d0 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b932e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40cb0b0 con 0x7f21b40ce350 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f21b408a3d0 0x7f21b40cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f21b408a3d0 0x7f21b40cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54084/0 (socket says 172.21.15.104:54084) 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b37fe700 1 -- 172.21.15.104:0/4097017788 learned_addr learned my addr 172.21.15.104:0/4097017788 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.697+0000 7f21b27fc700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 0x7f21b40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b37fe700 1 -- 172.21.15.104:0/4097017788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 msgr2=0x7f21b40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b37fe700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 0x7f21b40cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b37fe700 1 -- 172.21.15.104:0/4097017788 >> v1:172.21.15.104:6789/0 conn(0x7f21b406b1f0 legacy=0x7f21b40cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b37fe700 1 -- 172.21.15.104:0/4097017788 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40cf550 con 0x7f21b408a3d0 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b1ffb700 1 -- 172.21.15.104:0/4097017788 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453856709 0 0) 0x7f21b40cadd0 con 0x7f21b406b1f0 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b27fc700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 0x7f21b40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b37fe700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b408a3d0 0x7f21b40cb540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009e50 tx=0x7f21ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6387e183510dd3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b1ffb700 1 -- 172.21.15.104:0/4097017788 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00e070 con 0x7f21b408a3d0 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b1ffb700 1 -- 172.21.15.104:0/4097017788 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac00b9c0 con 0x7f21b408a3d0 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b1ffb700 1 -- 172.21.15.104:0/4097017788 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00bd70 con 0x7f21b408a3d0 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 -- 172.21.15.104:0/4097017788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b408a3d0 msgr2=0x7f21b40cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b408a3d0 0x7f21b40cb540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009e50 tx=0x7f21ac00b330 comp rx=0 tx=0).stop 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 -- 172.21.15.104:0/4097017788 shutdown_connections 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b408a3d0 0x7f21b40cb540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 --2- 172.21.15.104:0/4097017788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b40ce350 0x7f21b40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 -- 172.21.15.104:0/4097017788 >> 172.21.15.104:0/4097017788 conn(0x7f21b4089fd0 msgr2=0x7f21b40d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 -- 172.21.15.104:0/4097017788 shutdown_connections 2023-06-10T18:44:04.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.698+0000 7f21b932e700 1 -- 172.21.15.104:0/4097017788 wait complete. 2023-06-10T18:44:04.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 Processor -- start 2023-06-10T18:44:04.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 -- start start 2023-06-10T18:44:04.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 0x7f21b4151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 0x7f21b40c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40c4c20 con 0x7f21b40ce350 2023-06-10T18:44:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40c4ec0 con 0x7f21b408a3d0 2023-06-10T18:44:04.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.700+0000 7f21b932e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40c51a0 con 0x7f21b406b1f0 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54090/0 (socket says 172.21.15.104:54090) 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 -- 172.21.15.104:0/872828632 learned_addr learned my addr 172.21.15.104:0/872828632 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b2ffd700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 0x7f21b4151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b27fc700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 0x7f21b40c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 -- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 msgr2=0x7f21b4151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 0x7f21b4151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 -- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 msgr2=0x7f21b40c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 0x7f21b40c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 -- 172.21.15.104:0/872828632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac009b30 con 0x7f21b40ce350 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b2ffd700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 0x7f21b4151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b27fc700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 0x7f21b40c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:04.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b37fe700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00b730 tx=0x7f21ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0099b0 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac0092e0 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b40c5480 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac01c640 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.701+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b4151fb0 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.702+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b4074050 con 0x7f21b40ce350 2023-06-10T18:44:04.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.703+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f21ac01c7a0 con 0x7f21b40ce350 2023-06-10T18:44:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.703+0000 7f219b7fe700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 0x7f21a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.703+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f21ac08aaa0 con 0x7f21b40ce350 2023-06-10T18:44:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.703+0000 7f21b2ffd700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 0x7f21a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.704+0000 7f21b2ffd700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 0x7f21a006d190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000f60 tx=0x7f21a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.707+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f21ac055f30 con 0x7f21b40ce350 2023-06-10T18:44:04.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.934+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21b40b55c0 con 0x7f21b40ce350 2023-06-10T18:44:04.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.934+0000 7f219b7fe700 1 -- 172.21.15.104:0/872828632 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3533 (secure 0 0 0) 0x7f21ac059550 con 0x7f21b40ce350 2023-06-10T18:44:04.940 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:04.940 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4296,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-10T18:44:04.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 msgr2=0x7f21a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 0x7f21a006d190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21a8000f60 tx=0x7f21a8009040 comp rx=0 tx=0).stop 2023-06-10T18:44:04.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 msgr2=0x7f21b415dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00b730 tx=0x7f21ac004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 shutdown_connections 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40ce350 0x7f21b415dd90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21b406b1f0 0x7f21b4151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f21a006ace0 0x7f21a006d190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 --2- 172.21.15.104:0/872828632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21b408a3d0 0x7f21b40c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:04.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 >> 172.21.15.104:0/872828632 conn(0x7f21b4089fd0 msgr2=0x7f21b406f430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:04.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 shutdown_connections 2023-06-10T18:44:04.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:04.936+0000 7f21b932e700 1 -- 172.21.15.104:0/872828632 wait complete. 2023-06-10T18:44:04.944 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 3 2023-06-10T18:44:04.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:05.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.068+0000 7fb054821700 1 Processor -- start 2023-06-10T18:44:05.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.068+0000 7fb054821700 1 -- start start 2023-06-10T18:44:05.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb054821700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb0500aeaa0 0x7fb0500cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb054821700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 0x7fb0500cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb054821700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500c9e60 con 0x7fb0500ca2f0 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb054821700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0500c9bc0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb054821700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0500aee70 con 0x7fb0500b1080 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb04e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb0500aeaa0 0x7fb0500cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb04e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb0500aeaa0 0x7fb0500cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54094/0 (socket says 172.21.15.104:54094) 2023-06-10T18:44:05.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.069+0000 7fb04e59c700 1 -- 172.21.15.104:0/4153595361 learned_addr learned my addr 172.21.15.104:0/4153595361 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04dd9b700 1 --2- 172.21.15.104:0/4153595361 >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 0x7fb0500cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04e59c700 1 -- 172.21.15.104:0/4153595361 >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 msgr2=0x7fb0500cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04e59c700 1 --2- 172.21.15.104:0/4153595361 >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 0x7fb0500cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04e59c700 1 -- 172.21.15.104:0/4153595361 >> v1:172.21.15.114:6789/0 conn(0x7fb0500ca2f0 legacy=0x7fb0500ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04e59c700 1 -- 172.21.15.104:0/4153595361 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05006e540 con 0x7fb0500aeaa0 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04dd9b700 1 --2- 172.21.15.104:0/4153595361 >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 0x7fb0500cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:05.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04e59c700 1 --2- 172.21.15.104:0/4153595361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500cd140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb050070110 tx=0x7fb04400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71d103badb7dc4e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04d59a700 1 -- 172.21.15.104:0/4153595361 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044005e30 con 0x7fb0500aeaa0 2023-06-10T18:44:05.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04d59a700 1 -- 172.21.15.104:0/4153595361 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04400d8b0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb04d59a700 1 -- 172.21.15.104:0/4153595361 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04400dc40 con 0x7fb0500aeaa0 2023-06-10T18:44:05.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb054821700 1 -- 172.21.15.104:0/4153595361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 msgr2=0x7fb0500cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.070+0000 7fb054821700 1 --2- 172.21.15.104:0/4153595361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500cd140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb050070110 tx=0x7fb04400be70 comp rx=0 tx=0).stop 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 -- 172.21.15.104:0/4153595361 shutdown_connections 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 --2- 172.21.15.104:0/4153595361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500cd140 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 --2- 172.21.15.104:0/4153595361 >> v2:172.21.15.114:3300/0 conn(0x7fb0500b1080 0x7fb0500cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 -- 172.21.15.104:0/4153595361 >> 172.21.15.104:0/4153595361 conn(0x7fb05006b8a0 msgr2=0x7fb0500cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 -- 172.21.15.104:0/4153595361 shutdown_connections 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.071+0000 7fb054821700 1 -- 172.21.15.104:0/4153595361 wait complete. 2023-06-10T18:44:05.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.072+0000 7fb054821700 1 Processor -- start 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.072+0000 7fb054821700 1 -- start start 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0500b1080 0x7fb05015ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0500ca2f0 0x7fb05015b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500b2d90 con 0x7fb0500aeaa0 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015b8a0 con 0x7fb0500b1080 2023-06-10T18:44:05.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500adb00 con 0x7fb0500ca2f0 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54100/0 (socket says 172.21.15.104:54100) 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 -- 172.21.15.104:0/3756774363 learned_addr learned my addr 172.21.15.104:0/3756774363 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04dd9b700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0500b1080 0x7fb05015ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04ed9d700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0500ca2f0 0x7fb05015b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 -- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0500ca2f0 msgr2=0x7fb05015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0500ca2f0 0x7fb05015b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 -- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0500b1080 msgr2=0x7fb05015ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0500b1080 0x7fb05015ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 -- 172.21.15.104:0/3756774363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04400b7a0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb04e59c700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb05006cc20 tx=0x7fb044004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04401f070 con 0x7fb0500aeaa0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04400dc30 con 0x7fb0500aeaa0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500adde0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04401a690 con 0x7fb0500aeaa0 2023-06-10T18:44:05.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.073+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500b29b0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.074+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb050074050 con 0x7fb0500aeaa0 2023-06-10T18:44:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.075+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb04401a7f0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.075+0000 7fb03effd700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 0x7fb03006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.075+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb04408aa60 con 0x7fb0500aeaa0 2023-06-10T18:44:05.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.075+0000 7fb04dd9b700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 0x7fb03006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.076+0000 7fb04dd9b700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 0x7fb03006d190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0500c9b20 tx=0x7fb040008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.079+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb044055e60 con 0x7fb0500aeaa0 2023-06-10T18:44:05.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.267+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0501574b0 con 0x7fb0500aeaa0 2023-06-10T18:44:05.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.268+0000 7fb03effd700 1 -- 172.21.15.104:0/3756774363 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14684 (secure 0 0 0) 0x7fb044015020 con 0x7fb0500aeaa0 2023-06-10T18:44:05.273 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:05.273 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":19,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:04.533680+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:05.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 msgr2=0x7fb03006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 0x7fb03006d190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0500c9b20 tx=0x7fb040008040 comp rx=0 tx=0).stop 2023-06-10T18:44:05.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 msgr2=0x7fb0500b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb05006cc20 tx=0x7fb044004750 comp rx=0 tx=0).stop 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 shutdown_connections 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0500ca2f0 0x7fb05015b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0500aeaa0 0x7fb0500b4e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb03006ace0 0x7fb03006d190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 --2- 172.21.15.104:0/3756774363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0500b1080 0x7fb05015ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 >> 172.21.15.104:0/3756774363 conn(0x7fb05006b8a0 msgr2=0x7fb0500cac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 shutdown_connections 2023-06-10T18:44:05.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.270+0000 7fb054821700 1 -- 172.21.15.104:0/3756774363 wait complete. 2023-06-10T18:44:05.287 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:44:05.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:05.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.405+0000 7fd21f319700 1 Processor -- start 2023-06-10T18:44:05.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.405+0000 7fd21f319700 1 -- start start 2023-06-10T18:44:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21f319700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd21815c580 0x7fd2180b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21f319700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd21806e6e0 0x7fd2180b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21f319700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21806e250 con 0x7fd21806cde0 2023-06-10T18:44:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21f319700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21815c950 con 0x7fd21806e6e0 2023-06-10T18:44:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21f319700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180cf0e0 con 0x7fd21815c580 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd21806e6e0 0x7fd2180b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd21806e6e0 0x7fd2180b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54108/0 (socket says 172.21.15.104:54108) 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21d8b6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd21806cde0 0x7fd218069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51940/0 (socket says 172.21.15.104:51940) 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 -- 172.21.15.104:0/2080953962 learned_addr learned my addr 172.21.15.104:0/2080953962 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21d0b5700 1 --2- 172.21.15.104:0/2080953962 >> v2:172.21.15.114:3300/0 conn(0x7fd21815c580 0x7fd2180b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 -- 172.21.15.104:0/2080953962 >> v2:172.21.15.114:3300/0 conn(0x7fd21815c580 msgr2=0x7fd2180b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 --2- 172.21.15.104:0/2080953962 >> v2:172.21.15.114:3300/0 conn(0x7fd21815c580 0x7fd2180b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 -- 172.21.15.104:0/2080953962 >> v1:172.21.15.104:6789/0 conn(0x7fd21806cde0 legacy=0x7fd218069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 -- 172.21.15.104:0/2080953962 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b1e80 con 0x7fd21806e6e0 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd21c8b4700 1 --2- 172.21.15.104:0/2080953962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21806e6e0 0x7fd2180b15b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd218070300 tx=0x7fd208009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1fde5539ff7886 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd20ffff700 1 -- 172.21.15.104:0/2080953962 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20800b870 con 0x7fd21806e6e0 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd20ffff700 1 -- 172.21.15.104:0/2080953962 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20800b9d0 con 0x7fd21806e6e0 2023-06-10T18:44:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.406+0000 7fd20ffff700 1 -- 172.21.15.104:0/2080953962 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20800bd60 con 0x7fd21806e6e0 2023-06-10T18:44:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 -- 172.21.15.104:0/2080953962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21806e6e0 msgr2=0x7fd2180b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 --2- 172.21.15.104:0/2080953962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21806e6e0 0x7fd2180b15b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd218070300 tx=0x7fd208009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 -- 172.21.15.104:0/2080953962 shutdown_connections 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 --2- 172.21.15.104:0/2080953962 >> v2:172.21.15.114:3300/0 conn(0x7fd21815c580 0x7fd2180b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 --2- 172.21.15.104:0/2080953962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21806e6e0 0x7fd2180b15b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 -- 172.21.15.104:0/2080953962 >> 172.21.15.104:0/2080953962 conn(0x7fd21806b8a0 msgr2=0x7fd2180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 -- 172.21.15.104:0/2080953962 shutdown_connections 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.407+0000 7fd21f319700 1 -- 172.21.15.104:0/2080953962 wait complete. 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.408+0000 7fd21f319700 1 Processor -- start 2023-06-10T18:44:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.408+0000 7fd21f319700 1 -- start start 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd21806cde0 0x7fd2180cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd21806e6e0 0x7fd218149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180af180 con 0x7fd21815c580 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180af460 con 0x7fd21806cde0 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21f319700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180af740 con 0x7fd21806e6e0 2023-06-10T18:44:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54118/0 (socket says 172.21.15.104:54118) 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 -- 172.21.15.104:0/3475449714 learned_addr learned my addr 172.21.15.104:0/3475449714 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21c8b4700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd21806e6e0 0x7fd218149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d0b5700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd21806cde0 0x7fd2180cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 -- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd21806e6e0 msgr2=0x7fd218149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd21806e6e0 0x7fd218149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 -- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd21806cde0 msgr2=0x7fd2180cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd21806cde0 0x7fd2180cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.409+0000 7fd21d8b6700 1 -- 172.21.15.104:0/3475449714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd208009550 con 0x7fd21815c580 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd21d8b6700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd21000cbc0 tx=0x7fd21000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21000aaa0 con 0x7fd21815c580 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21000ac00 con 0x7fd21815c580 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2180afa20 con 0x7fd21815c580 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210007070 con 0x7fd21815c580 2023-06-10T18:44:05.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.410+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2180cf9d0 con 0x7fd21815c580 2023-06-10T18:44:05.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.411+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2180aff40 con 0x7fd21815c580 2023-06-10T18:44:05.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.411+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd2100071d0 con 0x7fd21815c580 2023-06-10T18:44:05.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.411+0000 7fd20dffb700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 0x7fd1fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.411+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd210088cf0 con 0x7fd21815c580 2023-06-10T18:44:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.412+0000 7fd21d0b5700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 0x7fd1fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.412+0000 7fd21d0b5700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 0x7fd1fc06d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd214006fd0 tx=0x7fd214008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.416+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd2180aff40 con 0x7fd21815c580 2023-06-10T18:44:05.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.602+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2180aff40 con 0x7fd21815c580 2023-06-10T18:44:05.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.602+0000 7fd20dffb700 1 -- 172.21.15.104:0/3475449714 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2180aff40 con 0x7fd21815c580 2023-06-10T18:44:05.608 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 msgr2=0x7fd1fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 0x7fd1fc06d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd214006fd0 tx=0x7fd214008040 comp rx=0 tx=0).stop 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 msgr2=0x7fd2180c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd21000cbc0 tx=0x7fd21000ced0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 shutdown_connections 2023-06-10T18:44:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1fc06ace0 0x7fd1fc06d190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd21806cde0 0x7fd2180cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd21815c580 0x7fd2180c89a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 --2- 172.21.15.104:0/3475449714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd21806e6e0 0x7fd218149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 >> 172.21.15.104:0/3475449714 conn(0x7fd21806b8a0 msgr2=0x7fd2180cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 shutdown_connections 2023-06-10T18:44:05.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.604+0000 7fd21f319700 1 -- 172.21.15.104:0/3475449714 wait complete. 2023-06-10T18:44:05.621 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:05.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 Processor -- start 2023-06-10T18:44:05.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 -- start start 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa24c06cde0 0x7fa24c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c06e250 con 0x7fa24c15c580 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24c15c950 con 0x7fa24c06cde0 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.739+0000 7fa2523a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0b1e80 con 0x7fa24c06e6e0 2023-06-10T18:44:05.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa24c06cde0 0x7fa24c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45276/0 (socket says 172.21.15.104:45276) 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24b7fe700 1 -- 172.21.15.104:0/376884061 learned_addr learned my addr 172.21.15.104:0/376884061 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24bfff700 1 --1- 172.21.15.104:0/376884061 >> v1:172.21.15.114:6789/0 conn(0x7fa24c15c580 0x7fa24c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:57796/0 (socket says 172.21.15.104:57796) 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 -- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 msgr2=0x7fa24c0b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 -- 172.21.15.104:0/376884061 >> v1:172.21.15.114:6789/0 conn(0x7fa24c15c580 legacy=0x7fa24c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 -- 172.21.15.104:0/376884061 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c06d920 con 0x7fa24c06cde0 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24b7fe700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa250941700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0ce0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa24400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eac063ba4eb44ee server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24affd700 1 -- 172.21.15.104:0/376884061 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24400e070 con 0x7fa24c06cde0 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24affd700 1 -- 172.21.15.104:0/376884061 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400bb20 con 0x7fa24c06cde0 2023-06-10T18:44:05.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.740+0000 7fa24affd700 1 -- 172.21.15.104:0/376884061 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244012430 con 0x7fa24c06cde0 2023-06-10T18:44:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 -- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 msgr2=0x7fa24c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0ce0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa24400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:05.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 -- 172.21.15.104:0/376884061 shutdown_connections 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0ce0d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 --2- 172.21.15.104:0/376884061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c0b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 -- 172.21.15.104:0/376884061 >> 172.21.15.104:0/376884061 conn(0x7fa24c06b8a0 msgr2=0x7fa24c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 -- 172.21.15.104:0/376884061 shutdown_connections 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.741+0000 7fa2523a4700 1 -- 172.21.15.104:0/376884061 wait complete. 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.742+0000 7fa2523a4700 1 Processor -- start 2023-06-10T18:44:05.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.742+0000 7fa2523a4700 1 -- start start 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c57b0 con 0x7fa24c15c580 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c4b30 con 0x7fa24c06cde0 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa2523a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c4e10 con 0x7fa24c06e6e0 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa24b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa24b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c15abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45284/0 (socket says 172.21.15.104:45284) 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54130/0 (socket says 172.21.15.104:54130) 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa24b7fe700 1 -- 172.21.15.104:0/1784273112 learned_addr learned my addr 172.21.15.104:0/1784273112 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa24bfff700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 -- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 msgr2=0x7fa24c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 -- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 msgr2=0x7fa24c0c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.743+0000 7fa250941700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa250941700 1 -- 172.21.15.104:0/1784273112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240009550 con 0x7fa24c15c580 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa250941700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa244016000 tx=0x7fa244004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244012430 con 0x7fa24c15c580 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244020070 con 0x7fa24c15c580 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa244009a70 con 0x7fa24c15c580 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2440098f0 con 0x7fa24c15c580 2023-06-10T18:44:05.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.744+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c0c53d0 con 0x7fa24c15c580 2023-06-10T18:44:05.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.745+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c0c4f50 con 0x7fa24c15c580 2023-06-10T18:44:05.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.745+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa244004040 con 0x7fa24c15c580 2023-06-10T18:44:05.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.745+0000 7fa248ff9700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 0x7fa23806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:05.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.746+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa244022070 con 0x7fa24c15c580 2023-06-10T18:44:05.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.746+0000 7fa24bfff700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 0x7fa23806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:05.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.746+0000 7fa24bfff700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 0x7fa23806d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000fb0 tx=0x7fa23c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:05.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.750+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa24c0c4f50 con 0x7fa24c15c580 2023-06-10T18:44:05.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.934+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa24c0c58f0 con 0x7fa24c15c580 2023-06-10T18:44:05.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.935+0000 7fa248ff9700 1 -- 172.21.15.104:0/1784273112 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa24c0c58f0 con 0x7fa24c15c580 2023-06-10T18:44:05.940 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:05.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 msgr2=0x7fa23806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 0x7fa23806d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000fb0 tx=0x7fa23c007ec0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 msgr2=0x7fa24c14e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa244016000 tx=0x7fa244004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 shutdown_connections 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa23806ace0 0x7fa23806d190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa24c06cde0 0x7fa24c0c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa24c15c580 0x7fa24c14e670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 --2- 172.21.15.104:0/1784273112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa24c06e6e0 0x7fa24c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:05.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 >> 172.21.15.104:0/1784273112 conn(0x7fa24c06b8a0 msgr2=0x7fa24c0ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:05.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 shutdown_connections 2023-06-10T18:44:05.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:05.937+0000 7fa2523a4700 1 -- 172.21.15.104:0/1784273112 wait complete. 2023-06-10T18:44:05.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:44:06.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.068+0000 7f3a74ec6700 1 Processor -- start 2023-06-10T18:44:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.068+0000 7f3a74ec6700 1 -- start start 2023-06-10T18:44:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a700aeaa0 0x7f3a700cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700b1080 0x7f3a700cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a700ca2f0 0x7f3a700ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700c9bc0 con 0x7f3a700ca2f0 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700c9e60 con 0x7f3a700aeaa0 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a74ec6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700aee70 con 0x7f3a700b1080 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a6ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a700ca2f0 0x7f3a700ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a6ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a700ca2f0 0x7f3a700ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54140/0 (socket says 172.21.15.104:54140) 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a6ed9d700 1 -- 172.21.15.104:0/2192619797 learned_addr learned my addr 172.21.15.104:0/2192619797 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a6dd9b700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700b1080 0x7f3a700cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.069+0000 7f3a6e59c700 1 --2- 172.21.15.104:0/2192619797 >> v2:172.21.15.114:3300/0 conn(0x7f3a700aeaa0 0x7f3a700cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 -- 172.21.15.104:0/2192619797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700b1080 msgr2=0x7f3a700cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700b1080 0x7f3a700cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 -- 172.21.15.104:0/2192619797 >> v2:172.21.15.114:3300/0 conn(0x7f3a700aeaa0 msgr2=0x7f3a700cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:06.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 --2- 172.21.15.104:0/2192619797 >> v2:172.21.15.114:3300/0 conn(0x7f3a700aeaa0 0x7f3a700cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 -- 172.21.15.104:0/2192619797 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7006e540 con 0x7f3a700ca2f0 2023-06-10T18:44:06.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6ed9d700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700ca2f0 0x7f3a700ca750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a700cf5d0 tx=0x7f3a6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c235616f744d4ec8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6d59a700 1 -- 172.21.15.104:0/2192619797 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800e070 con 0x7f3a700ca2f0 2023-06-10T18:44:06.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6d59a700 1 -- 172.21.15.104:0/2192619797 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800b910 con 0x7f3a700ca2f0 2023-06-10T18:44:06.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a6d59a700 1 -- 172.21.15.104:0/2192619797 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800bcc0 con 0x7f3a700ca2f0 2023-06-10T18:44:06.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 -- 172.21.15.104:0/2192619797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700ca2f0 msgr2=0x7f3a700ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700ca2f0 0x7f3a700ca750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a700cf5d0 tx=0x7f3a6800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:06.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 -- 172.21.15.104:0/2192619797 shutdown_connections 2023-06-10T18:44:06.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700ca2f0 0x7f3a700ca750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/2192619797 >> v2:172.21.15.114:3300/0 conn(0x7f3a700aeaa0 0x7f3a700cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/2192619797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700b1080 0x7f3a700cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.070+0000 7f3a74ec6700 1 -- 172.21.15.104:0/2192619797 >> 172.21.15.104:0/2192619797 conn(0x7f3a7006b8a0 msgr2=0x7f3a700cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.071+0000 7f3a74ec6700 1 -- 172.21.15.104:0/2192619797 shutdown_connections 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.071+0000 7f3a74ec6700 1 -- 172.21.15.104:0/2192619797 wait complete. 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.072+0000 7f3a74ec6700 1 Processor -- start 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.072+0000 7f3a74ec6700 1 -- start start 2023-06-10T18:44:06.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a700aeaa0 0x7f3a70157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700ca2f0 0x7f3a70143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700c2ed0 con 0x7f3a700b1080 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70154ee0 con 0x7f3a700aeaa0 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a74ec6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700b4f00 con 0x7f3a700ca2f0 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a6dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54144/0 (socket says 172.21.15.104:54144) 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a6dd9b700 1 -- 172.21.15.104:0/4072868922 learned_addr learned my addr 172.21.15.104:0/4072868922 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a6ed9d700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700ca2f0 0x7f3a70143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.073+0000 7f3a6e59c700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a700aeaa0 0x7f3a70157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 -- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700ca2f0 msgr2=0x7f3a70143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700ca2f0 0x7f3a70143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 -- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a700aeaa0 msgr2=0x7f3a70157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a700aeaa0 0x7f3a70157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 -- 172.21.15.104:0/4072868922 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68009b30 con 0x7f3a700b1080 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a6dd9b700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a60010bf0 tx=0x7f3a60010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60009710 con 0x7f3a700b1080 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6000ecc0 con 0x7f3a700b1080 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a700b4bd0 con 0x7f3a700b1080 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60011450 con 0x7f3a700b1080 2023-06-10T18:44:06.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.074+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a700b27b0 con 0x7f3a700b1080 2023-06-10T18:44:06.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.075+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a70074050 con 0x7f3a700b1080 2023-06-10T18:44:06.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.075+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3a600115b0 con 0x7f3a700b1080 2023-06-10T18:44:06.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.076+0000 7f3a5effd700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 0x7f3a5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.076+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3a6008b360 con 0x7f3a700b1080 2023-06-10T18:44:06.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.076+0000 7f3a6e59c700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 0x7f3a5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.076+0000 7f3a6e59c700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 0x7f3a5006d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009b40 tx=0x7f3a640091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.080+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3a6005a1f0 con 0x7f3a700b1080 2023-06-10T18:44:06.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.268+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3a70072f90 con 0x7f3a700b1080 2023-06-10T18:44:06.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.268+0000 7f3a5effd700 1 -- 172.21.15.104:0/4072868922 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7f3a60059d80 con 0x7f3a700b1080 2023-06-10T18:44:06.274 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:06.275 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:44:06.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:44:06.275 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:44:06.275 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:44:06.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:06.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:44:06.276 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:44:06.276 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:44:06.276 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:44:06.276 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:44:06.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 msgr2=0x7f3a5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 0x7f3a5006d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009b40 tx=0x7f3a640091c0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 msgr2=0x7f3a70156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a60010bf0 tx=0x7f3a60010f00 comp rx=0 tx=0).stop 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 shutdown_connections 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a5006ace0 0x7f3a5006d190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a700aeaa0 0x7f3a70157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a700ca2f0 0x7f3a70143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 --2- 172.21.15.104:0/4072868922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a700b1080 0x7f3a70156660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 >> 172.21.15.104:0/4072868922 conn(0x7f3a7006b8a0 msgr2=0x7f3a700cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.270+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 shutdown_connections 2023-06-10T18:44:06.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.271+0000 7f3a74ec6700 1 -- 172.21.15.104:0/4072868922 wait complete. 2023-06-10T18:44:06.288 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:44:06.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:44:06.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.406+0000 7f91f31cc700 1 Processor -- start 2023-06-10T18:44:06.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.406+0000 7f91f31cc700 1 -- start start 2023-06-10T18:44:06.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f31cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f91ec15c580 0x7f91ec0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f31cc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91ec06e6e0 0x7f91ec0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f31cc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec06e250 con 0x7f91ec06cde0 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f31cc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec15c950 con 0x7f91ec15c580 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f31cc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0cf0e0 con 0x7f91ec06e6e0 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f91ec15c580 0x7f91ec0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f1769700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f91ec06cde0 0x7f91ec069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51980/0 (socket says 172.21.15.104:51980) 2023-06-10T18:44:06.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f1769700 1 -- 172.21.15.104:0/1801517594 learned_addr learned my addr 172.21.15.104:0/1801517594 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91ebfff700 1 --2- 172.21.15.104:0/1801517594 >> v2:172.21.15.114:3300/0 conn(0x7f91ec06e6e0 0x7f91ec0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 -- 172.21.15.104:0/1801517594 >> v2:172.21.15.114:3300/0 conn(0x7f91ec06e6e0 msgr2=0x7f91ec0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 --2- 172.21.15.104:0/1801517594 >> v2:172.21.15.114:3300/0 conn(0x7f91ec06e6e0 0x7f91ec0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 -- 172.21.15.104:0/1801517594 >> v1:172.21.15.104:6789/0 conn(0x7f91ec06cde0 legacy=0x7f91ec069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91eb7fe700 1 -- 172.21.15.104:0/1801517594 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579724701 0 0) 0x7f91ec06e250 con 0x7f91ec06cde0 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 -- 172.21.15.104:0/1801517594 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0b1e80 con 0x7f91ec15c580 2023-06-10T18:44:06.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91f0f68700 1 --2- 172.21.15.104:0/1801517594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec15c580 0x7f91ec0b1070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91ec06af30 tx=0x7f91e0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65c333dc4e306b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.407+0000 7f91eb7fe700 1 -- 172.21.15.104:0/1801517594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0005e30 con 0x7f91ec15c580 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91eb7fe700 1 -- 172.21.15.104:0/1801517594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e0006030 con 0x7f91ec15c580 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91eb7fe700 1 -- 172.21.15.104:0/1801517594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00063c0 con 0x7f91ec15c580 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 -- 172.21.15.104:0/1801517594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec15c580 msgr2=0x7f91ec0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 --2- 172.21.15.104:0/1801517594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec15c580 0x7f91ec0b1070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91ec06af30 tx=0x7f91e0009b00 comp rx=0 tx=0).stop 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 -- 172.21.15.104:0/1801517594 shutdown_connections 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 --2- 172.21.15.104:0/1801517594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec15c580 0x7f91ec0b1070 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 --2- 172.21.15.104:0/1801517594 >> v2:172.21.15.114:3300/0 conn(0x7f91ec06e6e0 0x7f91ec0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 -- 172.21.15.104:0/1801517594 >> 172.21.15.104:0/1801517594 conn(0x7f91ec06b8a0 msgr2=0x7f91ec0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:06.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 -- 172.21.15.104:0/1801517594 shutdown_connections 2023-06-10T18:44:06.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.408+0000 7f91f31cc700 1 -- 172.21.15.104:0/1801517594 wait complete. 2023-06-10T18:44:06.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.409+0000 7f91f31cc700 1 Processor -- start 2023-06-10T18:44:06.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.409+0000 7f91f31cc700 1 -- start start 2023-06-10T18:44:06.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91ec06e6e0 0x7f91ec0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91ec15c580 0x7f91ec0c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0ad0f0 con 0x7f91ec06cde0 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0acb30 con 0x7f91ec15c580 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f31cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0ace10 con 0x7f91ec06e6e0 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54154/0 (socket says 172.21.15.104:54154) 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 -- 172.21.15.104:0/3265805359 learned_addr learned my addr 172.21.15.104:0/3265805359 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91ebfff700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91ec06e6e0 0x7f91ec0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f1769700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91ec15c580 0x7f91ec0c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 -- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91ec06e6e0 msgr2=0x7f91ec0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91ec06e6e0 0x7f91ec0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 -- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91ec15c580 msgr2=0x7f91ec0c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.410+0000 7f91f0f68700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91ec15c580 0x7f91ec0c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91f0f68700 1 -- 172.21.15.104:0/3265805359 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0009430 con 0x7f91ec06cde0 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91f0f68700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91e0000c00 tx=0x7f91e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0004040 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e0020070 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec0af0f0 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00043c0 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.411+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0ac700 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.412+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec0ac890 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.412+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f91e0022030 con 0x7f91ec06cde0 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.412+0000 7f91e97fa700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 0x7f91d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:06.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.413+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f91e008a660 con 0x7f91ec06cde0 2023-06-10T18:44:06.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.413+0000 7f91ebfff700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 0x7f91d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:06.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.413+0000 7f91ebfff700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 0x7f91d006d190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0b0890 tx=0x7f91d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:06.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.417+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f91e0055af0 con 0x7f91ec06cde0 2023-06-10T18:44:06.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:06.603+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f91ec066f10 con 0x7f91ec06cde0 2023-06-10T18:44:07.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:44:07.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:44:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.558+0000 7f91e97fa700 1 -- 172.21.15.104:0/3265805359 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f91e0059110 con 0x7f91ec06cde0 2023-06-10T18:44:07.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 msgr2=0x7f91d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:07.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 0x7f91d006d190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0b0890 tx=0x7f91d8008040 comp rx=0 tx=0).stop 2023-06-10T18:44:07.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 msgr2=0x7f91ec0ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91e0000c00 tx=0x7f91e0004750 comp rx=0 tx=0).stop 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 shutdown_connections 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91d006ace0 0x7f91d006d190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91ec15c580 0x7f91ec0c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91ec06cde0 0x7f91ec0ac1c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 --2- 172.21.15.104:0/3265805359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91ec06e6e0 0x7f91ec0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.562+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 >> 172.21.15.104:0/3265805359 conn(0x7f91ec06b8a0 msgr2=0x7f91ec15afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:07.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.563+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 shutdown_connections 2023-06-10T18:44:07.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.563+0000 7f91f31cc700 1 -- 172.21.15.104:0/3265805359 wait complete. 2023-06-10T18:44:07.570 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:44:07.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-06-10T18:44:07.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.708+0000 7f3e02697700 1 Processor -- start 2023-06-10T18:44:07.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.708+0000 7f3e02697700 1 -- start start 2023-06-10T18:44:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e02697700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 0x7f3dfc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e02697700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e02697700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc06e250 con 0x7f3dfc06e6e0 2023-06-10T18:44:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e02697700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc15c950 con 0x7f3dfc06cde0 2023-06-10T18:44:07.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e02697700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b1e80 con 0x7f3dfc15c580 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54164/0 (socket says 172.21.15.104:54164) 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 -- 172.21.15.104:0/1588692486 learned_addr learned my addr 172.21.15.104:0/1588692486 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 -- 172.21.15.104:0/1588692486 >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 msgr2=0x7f3dfc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3dfbfff700 1 --2- 172.21.15.104:0/1588692486 >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 0x7f3dfc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 --2- 172.21.15.104:0/1588692486 >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 0x7f3dfc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 -- 172.21.15.104:0/1588692486 >> v1:172.21.15.114:6789/0 conn(0x7f3dfc06e6e0 legacy=0x7f3dfc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3e00c34700 1 -- 172.21.15.104:0/1588692486 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc06d920 con 0x7f3dfc06cde0 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.709+0000 7f3dfbfff700 1 --2- 172.21.15.104:0/1588692486 >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 0x7f3dfc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e00c34700 1 --2- 172.21.15.104:0/1588692486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc070300 tx=0x7f3df400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f884507d8b874e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3dfaffd700 1 -- 172.21.15.104:0/1588692486 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df400e070 con 0x7f3dfc06cde0 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3dfaffd700 1 -- 172.21.15.104:0/1588692486 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4003a20 con 0x7f3dfc06cde0 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3dfaffd700 1 -- 172.21.15.104:0/1588692486 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4003dd0 con 0x7f3dfc06cde0 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 -- 172.21.15.104:0/1588692486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 msgr2=0x7f3dfc0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:07.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 --2- 172.21.15.104:0/1588692486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc070300 tx=0x7f3df400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 -- 172.21.15.104:0/1588692486 shutdown_connections 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 --2- 172.21.15.104:0/1588692486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc0ce0d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 --2- 172.21.15.104:0/1588692486 >> v2:172.21.15.114:3300/0 conn(0x7f3dfc15c580 0x7f3dfc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 -- 172.21.15.104:0/1588692486 >> 172.21.15.104:0/1588692486 conn(0x7f3dfc06b8a0 msgr2=0x7f3dfc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 -- 172.21.15.104:0/1588692486 shutdown_connections 2023-06-10T18:44:07.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.710+0000 7f3e02697700 1 -- 172.21.15.104:0/1588692486 wait complete. 2023-06-10T18:44:07.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.711+0000 7f3e02697700 1 Processor -- start 2023-06-10T18:44:07.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 -- start start 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dfc06e6e0 0x7f3dfc0b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dfc15c580 0x7f3dfc0c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c9000 con 0x7f3dfc06cde0 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c3590 con 0x7f3dfc15c580 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3e02697700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c3870 con 0x7f3dfc06e6e0 2023-06-10T18:44:07.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.712+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54166/0 (socket says 172.21.15.104:54166) 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 -- 172.21.15.104:0/470785071 learned_addr learned my addr 172.21.15.104:0/470785071 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfb7fe700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dfc06e6e0 0x7f3dfc0b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3e00c34700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dfc15c580 0x7f3dfc0c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 -- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dfc06e6e0 msgr2=0x7f3dfc0b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dfc06e6e0 0x7f3dfc0b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 -- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dfc15c580 msgr2=0x7f3dfc0c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dfc15c580 0x7f3dfc0c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 -- 172.21.15.104:0/470785071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4009b30 con 0x7f3dfc06cde0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3dfbfff700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c7f0 tx=0x7f3dec00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec005c20 con 0x7f3dfc06cde0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec005d80 con 0x7f3dfc06cde0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc0c3b50 con 0x7f3dfc06cde0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec011820 con 0x7f3dfc06cde0 2023-06-10T18:44:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.713+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc0c4110 con 0x7f3dfc06cde0 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.714+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc074050 con 0x7f3dfc06cde0 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.715+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3dec011980 con 0x7f3dfc06cde0 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.715+0000 7f3df8ff9700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 0x7f3de806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.715+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f3dec088f50 con 0x7f3dfc06cde0 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.715+0000 7f3dfb7fe700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 0x7f3de806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.716+0000 7f3dfb7fe700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 0x7f3de806d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc158c10 tx=0x7f3df0007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.720+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3dec057c70 con 0x7f3dfc06cde0 2023-06-10T18:44:07.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:07.909+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f3dfc0ca3f0 con 0x7f3dfc06cde0 2023-06-10T18:44:08.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.561+0000 7f3df8ff9700 1 -- 172.21.15.104:0/470785071 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7f3dec01e0d0 con 0x7f3dfc06cde0 2023-06-10T18:44:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.567+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 msgr2=0x7f3de806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.567+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 0x7f3de806d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc158c10 tx=0x7f3df0007ec0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.567+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 msgr2=0x7f3dfc1586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.567+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c7f0 tx=0x7f3dec00cb00 comp rx=0 tx=0).stop 2023-06-10T18:44:08.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 shutdown_connections 2023-06-10T18:44:08.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3de806ace0 0x7f3de806d190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dfc15c580 0x7f3dfc0c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3dfc06cde0 0x7f3dfc1586d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 --2- 172.21.15.104:0/470785071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dfc06e6e0 0x7f3dfc0b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 >> 172.21.15.104:0/470785071 conn(0x7f3dfc06b8a0 msgr2=0x7f3dfc0cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:08.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.570+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 shutdown_connections 2023-06-10T18:44:08.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.571+0000 7f3e02697700 1 -- 172.21.15.104:0/470785071 wait complete. 2023-06-10T18:44:08.579 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:44:08.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:44:08.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.717+0000 7f2090d26700 1 Processor -- start 2023-06-10T18:44:08.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.717+0000 7f2090d26700 1 -- start start 2023-06-10T18:44:08.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f208c15c580 0x7f208c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f208c06cde0 0x7f208c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c06e250 con 0x7f208c06cde0 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c15c950 con 0x7f208c15c580 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f2090d26700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0b1e80 con 0x7f208c06e6e0 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.718+0000 7f208ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f208c06cde0 0x7f208c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f208c06cde0 0x7f208c069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54178/0 (socket says 172.21.15.104:54178) 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 -- 172.21.15.104:0/4205420811 learned_addr learned my addr 172.21.15.104:0/4205420811 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f2089d9b700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208a59c700 1 --2- 172.21.15.104:0/4205420811 >> v2:172.21.15.114:3300/0 conn(0x7f208c15c580 0x7f208c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 -- 172.21.15.104:0/4205420811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 msgr2=0x7f208c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 -- 172.21.15.104:0/4205420811 >> v2:172.21.15.114:3300/0 conn(0x7f208c15c580 msgr2=0x7f208c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 --2- 172.21.15.104:0/4205420811 >> v2:172.21.15.114:3300/0 conn(0x7f208c15c580 0x7f208c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 -- 172.21.15.104:0/4205420811 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c06d920 con 0x7f208c06cde0 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f2089d9b700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:08.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208ad9d700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c069e70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f208c070300 tx=0x7f208400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d83b7562f0b3a9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208959a700 1 -- 172.21.15.104:0/4205420811 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208400e070 con 0x7f208c06cde0 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208959a700 1 -- 172.21.15.104:0/4205420811 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208400b9c0 con 0x7f208c06cde0 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.719+0000 7f208959a700 1 -- 172.21.15.104:0/4205420811 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208400bd70 con 0x7f208c06cde0 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 -- 172.21.15.104:0/4205420811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 msgr2=0x7f208c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c069e70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f208c070300 tx=0x7f208400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 -- 172.21.15.104:0/4205420811 shutdown_connections 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c069e70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 --2- 172.21.15.104:0/4205420811 >> v2:172.21.15.114:3300/0 conn(0x7f208c15c580 0x7f208c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 --2- 172.21.15.104:0/4205420811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 -- 172.21.15.104:0/4205420811 >> 172.21.15.104:0/4205420811 conn(0x7f208c06b8a0 msgr2=0x7f208c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:08.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 -- 172.21.15.104:0/4205420811 shutdown_connections 2023-06-10T18:44:08.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.720+0000 7f2090d26700 1 -- 172.21.15.104:0/4205420811 wait complete. 2023-06-10T18:44:08.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.721+0000 7f2090d26700 1 Processor -- start 2023-06-10T18:44:08.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.721+0000 7f2090d26700 1 -- start start 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c15d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 0x7f208c0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0c36e0 con 0x7f208c06cde0 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c15eb80 con 0x7f208c15c580 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c15ee60 con 0x7f208c06e6e0 2023-06-10T18:44:08.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54180/0 (socket says 172.21.15.104:54180) 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 -- 172.21.15.104:0/3949075645 learned_addr learned my addr 172.21.15.104:0/3949075645 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208ad9d700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 0x7f208c0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2089d9b700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c15d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 -- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 msgr2=0x7f208c15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c15d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 -- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 msgr2=0x7f208c0ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 0x7f208c0ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 -- 172.21.15.104:0/3949075645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009b30 con 0x7f208c06cde0 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2089d9b700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208ad9d700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 0x7f208c0ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f208a59c700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f208000c7f0 tx=0x7f208000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080011800 con 0x7f208c06cde0 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2080011960 con 0x7f208c06cde0 2023-06-10T18:44:08.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c0c3090 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.722+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080011d10 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.723+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c157c90 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.723+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c074050 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.724+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2080018360 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.724+0000 7f207affd700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 0x7f206c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.724+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f20800893f0 con 0x7f208c06cde0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.725+0000 7f2089d9b700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 0x7f206c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:08.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.725+0000 7f2089d9b700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 0x7f206c06d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f208c15be60 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:08.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.729+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2080057fa0 con 0x7f208c06cde0 2023-06-10T18:44:08.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.957+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f208c1e67d0 con 0x7f208c06cde0 2023-06-10T18:44:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.966+0000 7f207affd700 1 -- 172.21.15.104:0/3949075645 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2080057b30 con 0x7f208c06cde0 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 msgr2=0x7f206c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 0x7f206c06d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f208c15be60 tx=0x7f207c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 msgr2=0x7f208c157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f208000c7f0 tx=0x7f208000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 shutdown_connections 2023-06-10T18:44:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f206c06ace0 0x7f206c06d190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f208c15c580 0x7f208c0ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f208c06cde0 0x7f208c157750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 --2- 172.21.15.104:0/3949075645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f208c06e6e0 0x7f208c15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 >> 172.21.15.104:0/3949075645 conn(0x7f208c06b8a0 msgr2=0x7f208c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 shutdown_connections 2023-06-10T18:44:08.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:08.969+0000 7f2090d26700 1 -- 172.21.15.104:0/3949075645 wait complete. 2023-06-10T18:44:08.977 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 3 and data pool 4 2023-06-10T18:44:08.994 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-06-10T18:44:08.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-06-10T18:44:09.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.115+0000 7fdc0020f700 1 Processor -- start 2023-06-10T18:44:09.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.115+0000 7fdc0020f700 1 -- start start 2023-06-10T18:44:09.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdc0020f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbf806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdc0020f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf80ae7e0 con 0x7fdbf815c6f0 2023-06-10T18:44:09.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdc0020f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf806df90 con 0x7fdbf80b1780 2023-06-10T18:44:09.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdc0020f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf806e270 con 0x7fdbf806ea70 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfdfab700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdbf815c6f0 0x7fdbf80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52016/0 (socket says 172.21.15.104:52016) 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfdfab700 1 -- 172.21.15.104:0/726639565 learned_addr learned my addr 172.21.15.104:0/726639565 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfe7ac700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbf806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802899396 0 0) 0x7fdbf80ae7e0 con 0x7fdbf815c6f0 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe40035b0 con 0x7fdbf815c6f0 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203430078 0 0) 0x7fdbe40035b0 con 0x7fdbf815c6f0 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 msgr2=0x7fdbf806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbf806d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 >> v1:172.21.15.114:6789/0 conn(0x7fdbf80b1780 legacy=0x7fdbf806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf806e550 con 0x7fdbf815c6f0 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdbf40017f0 con 0x7fdbf815c6f0 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 >> v1:172.21.15.104:6789/0 conn(0x7fdbf815c6f0 legacy=0x7fdbf80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbe4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe4003fb0 0x7fdbe4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe4004d90 0x7fdbe4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.116+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf806e270 con 0x7fdbe4003fb0 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400bba0 con 0x7fdbe4004d90 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400df70 con 0x7fdbf806ea70 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe4003fb0 0x7fdbe4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfdfab700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbe4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfe7ac700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe4004d90 0x7fdbe4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 -- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 msgr2=0x7fdbe4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:09.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf806ea70 0x7fdbe4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 -- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe4004d90 msgr2=0x7fdbe4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbe4004d90 0x7fdbe4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 -- 172.21.15.104:0/726639565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40035b0 con 0x7fdbe4003fb0 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe4003fb0 0x7fdbe4004850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8007fd0 tx=0x7fdbe4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c7bc66c128f905 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80012f0 con 0x7fdbe4003fb0 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe8005220 con 0x7fdbe4003fb0 2023-06-10T18:44:09.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.117+0000 7fdbfcfa9700 1 -- 172.21.15.104:0/726639565 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8006860 con 0x7fdbe4003fb0 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 -- 172.21.15.104:0/726639565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe4003fb0 msgr2=0x7fdbe4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 --2- 172.21.15.104:0/726639565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe4003fb0 0x7fdbe4004850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8007fd0 tx=0x7fdbe4005880 comp rx=0 tx=0).stop 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdbfe7ac700 1 -- 172.21.15.104:0/726639565 reap_dead start 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 -- 172.21.15.104:0/726639565 shutdown_connections 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 -- 172.21.15.104:0/726639565 >> 172.21.15.104:0/726639565 conn(0x7fdbf806b8a0 msgr2=0x7fdbf80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:09.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 -- 172.21.15.104:0/726639565 shutdown_connections 2023-06-10T18:44:09.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.118+0000 7fdc0020f700 1 -- 172.21.15.104:0/726639565 wait complete. 2023-06-10T18:44:09.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.119+0000 7fdc0020f700 1 Processor -- start 2023-06-10T18:44:09.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.119+0000 7fdc0020f700 1 -- start start 2023-06-10T18:44:09.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 0x7fdbf81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 0x7fdbf81511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b39f0 con 0x7fdbf80b3040 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b02e0 con 0x7fdbf80afa90 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80b05c0 con 0x7fdbe4003fb0 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54204/0 (socket says 172.21.15.104:54204) 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 -- 172.21.15.104:0/2357618918 learned_addr learned my addr 172.21.15.104:0/2357618918 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:09.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfdfab700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 0x7fdbf81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 0x7fdbf81511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 -- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 msgr2=0x7fdbf81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 0x7fdbf81519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 -- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 msgr2=0x7fdbf81511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 0x7fdbf81511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 -- 172.21.15.104:0/2357618918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80b08a0 con 0x7fdbf80b3040 2023-06-10T18:44:09.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfdfab700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 0x7fdbf81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfd7aa700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 0x7fdbf81511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbfe7ac700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8072560 tx=0x7fdbf0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf000e040 con 0x7fdbf80b3040 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf000f040 con 0x7fdbf80b3040 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf000fdb0 con 0x7fdbf80b3040 2023-06-10T18:44:09.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf80b0b80 con 0x7fdbf80b3040 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.120+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf8151700 con 0x7fdbf80b3040 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.121+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf8074050 con 0x7fdbf80b3040 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.122+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdbf0022460 con 0x7fdbf80b3040 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.122+0000 7fdbee7fc700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 0x7fdbdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.122+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdbf0089a20 con 0x7fdbf80b3040 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.122+0000 7fdbfdfab700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 0x7fdbdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:09.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.123+0000 7fdbfdfab700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 0x7fdbdc06d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40011d0 tx=0x7fdbf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:09.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.126+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdbf0054ad0 con 0x7fdbf80b3040 2023-06-10T18:44:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.322+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fdbf80d0030 con 0x7fdbf80b3040 2023-06-10T18:44:09.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.968+0000 7fdbee7fc700 1 -- 172.21.15.104:0/2357618918 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v23) v1 ==== 157+0+0 (secure 0 0 0) 0x7fdbf00580f0 con 0x7fdbf80b3040 2023-06-10T18:44:09.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 msgr2=0x7fdbdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 0x7fdbdc06d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40011d0 tx=0x7fdbf4009040 comp rx=0 tx=0).stop 2023-06-10T18:44:09.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 msgr2=0x7fdbf80b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:09.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8072560 tx=0x7fdbf0011f80 comp rx=0 tx=0).stop 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 shutdown_connections 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbf80b3040 0x7fdbf80b34b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe4003fb0 0x7fdbf81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbdc06ace0 0x7fdbdc06d190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 --2- 172.21.15.104:0/2357618918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbf80afa90 0x7fdbf81511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 >> 172.21.15.104:0/2357618918 conn(0x7fdbf806b8a0 msgr2=0x7fdbf80ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:09.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 shutdown_connections 2023-06-10T18:44:09.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:09.970+0000 7fdc0020f700 1 -- 172.21.15.104:0/2357618918 wait complete. 2023-06-10T18:44:09.989 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:44:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.113+0000 7fb6e0784700 1 Processor -- start 2023-06-10T18:44:10.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.113+0000 7fb6e0784700 1 -- start start 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6e0784700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6d80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6e0784700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d806e250 con 0x7fb6d806e6e0 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6e0784700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d815c950 con 0x7fb6d806cde0 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6e0784700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80b1e80 con 0x7fb6d815c580 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6ddd1f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb6d806e6e0 0x7fb6d80b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52038/0 (socket says 172.21.15.104:52038) 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.114+0000 7fb6ddd1f700 1 -- 172.21.15.104:0/2760703284 learned_addr learned my addr 172.21.15.104:0/2760703284 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:10.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268226870 0 0) 0x7fb6d806e250 con 0x7fb6d806e6e0 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c40035b0 con 0x7fb6d806e6e0 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6de520700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6d80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751634233 0 0) 0x7fb6c40035b0 con 0x7fb6d806e6e0 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 msgr2=0x7fb6d80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6d80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 >> v1:172.21.15.114:6789/0 conn(0x7fb6d806cde0 legacy=0x7fb6d80ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:10.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d806d920 con 0x7fb6d806e6e0 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb6d4001670 con 0x7fb6d806e6e0 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 >> v1:172.21.15.104:6789/0 conn(0x7fb6d806e6e0 legacy=0x7fb6d80b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6c4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6c4003fb0 0x7fb6c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6c4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80b1e80 con 0x7fb6c4004d90 2023-06-10T18:44:10.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c400baf0 con 0x7fb6c4003fb0 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c400bcb0 con 0x7fb6d815c580 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6de520700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6c4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6c4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ddd1f700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6c4003fb0 0x7fb6c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 -- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 msgr2=0x7fb6c4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d815c580 0x7fb6c4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:10.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 -- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6c4003fb0 msgr2=0x7fb6c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6c4003fb0 0x7fb6c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.115+0000 7fb6ded21700 1 -- 172.21.15.104:0/2760703284 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40035b0 con 0x7fb6c4004d90 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6ded21700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6c4005200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000c160 tx=0x7fb6c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec10de7c5ebdc9c5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0001db0 con 0x7fb6c4004d90 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d000cca0 con 0x7fb6c4004d90 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6dd51e700 1 -- 172.21.15.104:0/2760703284 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0003c40 con 0x7fb6c4004d90 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 -- 172.21.15.104:0/2760703284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 msgr2=0x7fb6c4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:10.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 --2- 172.21.15.104:0/2760703284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6c4005200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000c160 tx=0x7fb6c4005880 comp rx=0 tx=0).stop 2023-06-10T18:44:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6ded21700 1 -- 172.21.15.104:0/2760703284 reap_dead start 2023-06-10T18:44:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 -- 172.21.15.104:0/2760703284 shutdown_connections 2023-06-10T18:44:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 -- 172.21.15.104:0/2760703284 >> 172.21.15.104:0/2760703284 conn(0x7fb6d806b8a0 msgr2=0x7fb6d80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 -- 172.21.15.104:0/2760703284 shutdown_connections 2023-06-10T18:44:10.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.116+0000 7fb6e0784700 1 -- 172.21.15.104:0/2760703284 wait complete. 2023-06-10T18:44:10.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.117+0000 7fb6e0784700 1 Processor -- start 2023-06-10T18:44:10.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 -- start start 2023-06-10T18:44:10.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d8155c80 0x7fb6d80ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d81f31a0 0x7fb6d81f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d81560f0 con 0x7fb6c4004d90 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80b3f00 con 0x7fb6d81f31a0 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6e0784700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d81f2d90 con 0x7fb6d8155c80 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6de520700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6de520700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54220/0 (socket says 172.21.15.104:54220) 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6de520700 1 -- 172.21.15.104:0/3942035015 learned_addr learned my addr 172.21.15.104:0/3942035015 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6ddd1f700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d8155c80 0x7fb6d80ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.118+0000 7fb6ded21700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d81f31a0 0x7fb6d81f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 -- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d8155c80 msgr2=0x7fb6d80ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d8155c80 0x7fb6d80ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 -- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d81f31a0 msgr2=0x7fb6d81f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:10.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d81f31a0 0x7fb6d81f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 -- 172.21.15.104:0/3942035015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80b38b0 con 0x7fb6c4004d90 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6de520700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8011a20 tx=0x7fb6c8011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8011eb0 con 0x7fb6c4004d90 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c8005c30 con 0x7fb6c4004d90 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c800f6d0 con 0x7fb6c4004d90 2023-06-10T18:44:10.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d80b3b90 con 0x7fb6c4004d90 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.119+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d80b34d0 con 0x7fb6c4004d90 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.120+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d80b41e0 con 0x7fb6c4004d90 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.120+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb6c800f830 con 0x7fb6c4004d90 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.120+0000 7fb6ceffd700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 0x7fb6bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.121+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fb6c8089b60 con 0x7fb6c4004d90 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.121+0000 7fb6ddd1f700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 0x7fb6bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.121+0000 7fb6ddd1f700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 0x7fb6bc06d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80c9aa0 tx=0x7fb6d4001170 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:10.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.125+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb6c8058680 con 0x7fb6c4004d90 2023-06-10T18:44:10.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:10.314+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb6d806ad00 con 0x7fb6c4004d90 2023-06-10T18:44:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.980+0000 7fb6ceffd700 1 -- 172.21.15.104:0/3942035015 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v25) v1 ==== 228+0+0 (secure 0 0 0) 0x7fb6c8021100 con 0x7fb6c4004d90 2023-06-10T18:44:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.994+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 msgr2=0x7fb6bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.994+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 0x7fb6bc06d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80c9aa0 tx=0x7fb6d4001170 comp rx=0 tx=0).stop 2023-06-10T18:44:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.994+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 msgr2=0x7fb6d81f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.994+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8011a20 tx=0x7fb6c8011d30 comp rx=0 tx=0).stop 2023-06-10T18:44:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 shutdown_connections 2023-06-10T18:44:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6bc06ace0 0x7fb6bc06d190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d81f31a0 0x7fb6d81f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6c4004d90 0x7fb6d81f27b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 --2- 172.21.15.104:0/3942035015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d8155c80 0x7fb6d80ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.995+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 >> 172.21.15.104:0/3942035015 conn(0x7fb6d806b8a0 msgr2=0x7fb6d8159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.998+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 shutdown_connections 2023-06-10T18:44:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:11.998+0000 7fb6e0784700 1 -- 172.21.15.104:0/3942035015 wait complete. 2023-06-10T18:44:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data_ec' created 2023-06-10T18:44:12.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-06-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.142+0000 7f22feefd700 1 Processor -- start 2023-06-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.142+0000 7f22feefd700 1 -- start start 2023-06-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22feefd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22feefd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22feefd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80ad3c0 con 0x7f22f8157b30 2023-06-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22feefd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f8157f00 con 0x7f22f80d0830 2023-06-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22feefd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80b0170 con 0x7f22f80cea50 2023-06-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22fd49a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45382/0 (socket says 172.21.15.104:45382) 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22fd49a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41734/0 (socket says 172.21.15.104:41734) 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 -- 172.21.15.104:0/2332082399 learned_addr learned my addr 172.21.15.104:0/2332082399 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22fcc99700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f22f8157b30 0x7f22f80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:57902/0 (socket says 172.21.15.104:57902) 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 -- 172.21.15.104:0/2332082399 >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 msgr2=0x7f22f80b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 --2- 172.21.15.104:0/2332082399 >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 -- 172.21.15.104:0/2332082399 >> v1:172.21.15.114:6789/0 conn(0x7f22f8157b30 legacy=0x7f22f80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22f7fff700 1 -- 172.21.15.104:0/2332082399 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80b0410 con 0x7f22f80cea50 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.143+0000 7f22fd49a700 1 --2- 172.21.15.104:0/2332082399 >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22f7fff700 1 --2- 172.21.15.104:0/2332082399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22e40098f0 tx=0x7f22e4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fcfdb8e6f70799 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22f77fe700 1 -- 172.21.15.104:0/2332082399 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4011830 con 0x7f22f80cea50 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22f77fe700 1 -- 172.21.15.104:0/2332082399 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e4011990 con 0x7f22f80cea50 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22f77fe700 1 -- 172.21.15.104:0/2332082399 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4011d20 con 0x7f22f80cea50 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 -- 172.21.15.104:0/2332082399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 msgr2=0x7f22f80d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 --2- 172.21.15.104:0/2332082399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22e40098f0 tx=0x7f22e4009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 -- 172.21.15.104:0/2332082399 shutdown_connections 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 --2- 172.21.15.104:0/2332082399 >> v2:172.21.15.114:3300/0 conn(0x7f22f80d0830 0x7f22f80b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 --2- 172.21.15.104:0/2332082399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f80d02f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 -- 172.21.15.104:0/2332082399 >> 172.21.15.104:0/2332082399 conn(0x7f22f8089fd0 msgr2=0x7f22f808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 -- 172.21.15.104:0/2332082399 shutdown_connections 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.144+0000 7f22feefd700 1 -- 172.21.15.104:0/2332082399 wait complete. 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.145+0000 7f22feefd700 1 Processor -- start 2023-06-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 -- start start 2023-06-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f80d0830 0x7f22f8154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80c4140 con 0x7f22f8157b30 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80b1970 con 0x7f22f80d0830 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.146+0000 7f22feefd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80b0df0 con 0x7f22f80cea50 2023-06-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54238/0 (socket says 172.21.15.104:54238) 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 -- 172.21.15.104:0/2464626627 learned_addr learned my addr 172.21.15.104:0/2464626627 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fcc99700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22f7fff700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f80d0830 0x7f22f8154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 -- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 msgr2=0x7f22f814ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f814ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 -- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f80d0830 msgr2=0x7f22f8154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f80d0830 0x7f22f8154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 -- 172.21.15.104:0/2464626627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0009a70 con 0x7f22f8157b30 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22fd49a700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f22f000efc0 tx=0x7f22f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0009d70 con 0x7f22f8157b30 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f000f040 con 0x7f22f8157b30 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e4009550 con 0x7f22f8157b30 2023-06-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f000ce00 con 0x7f22f8157b30 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.147+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f80b1320 con 0x7f22f8157b30 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.148+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f80b1b20 con 0x7f22f8157b30 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.149+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f22f0014480 con 0x7f22f8157b30 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.149+0000 7f22f57fa700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 0x7f22dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.149+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f22f0089f40 con 0x7f22f8157b30 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.149+0000 7f22fcc99700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 0x7f22dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.150+0000 7f22fcc99700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 0x7f22dc06d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000f60 tx=0x7f22ec007d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.154+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22f80b1b20 con 0x7f22f8157b30 2023-06-10T18:44:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.338+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f22f80b1b20 con 0x7f22f8157b30 2023-06-10T18:44:12.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.982+0000 7f22f57fa700 1 -- 172.21.15.104:0/2464626627 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f22f80b1b20 con 0x7f22f8157b30 2023-06-10T18:44:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.986+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 msgr2=0x7f22dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.986+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 0x7f22dc06d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000f60 tx=0x7f22ec007d10 comp rx=0 tx=0).stop 2023-06-10T18:44:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.986+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 msgr2=0x7f22f8154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.986+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f22f000efc0 tx=0x7f22f000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 shutdown_connections 2023-06-10T18:44:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8157b30 0x7f22f8154fa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22f80cea50 0x7f22f814ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22dc06ace0 0x7f22dc06d190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 --2- 172.21.15.104:0/2464626627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f80d0830 0x7f22f8154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:12.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.988+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 >> 172.21.15.104:0/2464626627 conn(0x7f22f8089fd0 msgr2=0x7f22f814c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:12.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.990+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 shutdown_connections 2023-06-10T18:44:12.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:12.991+0000 7f22feefd700 1 -- 172.21.15.104:0/2464626627 wait complete. 2023-06-10T18:44:13.006 INFO:teuthology.orchestra.run.smithi104.stderr:set pool 5 allow_ec_overwrites to true 2023-06-10T18:44:13.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-06-10T18:44:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.139+0000 7f4c2b1e7700 1 Processor -- start 2023-06-10T18:44:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.139+0000 7f4c2b1e7700 1 -- start start 2023-06-10T18:44:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c2b1e7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c2415c500 0x7f4c240ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c2b1e7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c2406a850 0x7f4c2406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c2b1e7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240ce970 con 0x7f4c2406e420 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c2b1e7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2406e8a0 con 0x7f4c2406a850 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c2b1e7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c240cec50 con 0x7f4c2415c500 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c2406a850 0x7f4c2406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c2406a850 0x7f4c2406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54244/0 (socket says 172.21.15.104:54244) 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 -- 172.21.15.104:0/2442652977 learned_addr learned my addr 172.21.15.104:0/2442652977 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c28f83700 1 --2- 172.21.15.104:0/2442652977 >> v2:172.21.15.114:3300/0 conn(0x7f4c2415c500 0x7f4c240ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 -- 172.21.15.104:0/2442652977 >> v2:172.21.15.114:3300/0 conn(0x7f4c2415c500 msgr2=0x7f4c240ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 --2- 172.21.15.104:0/2442652977 >> v2:172.21.15.114:3300/0 conn(0x7f4c2415c500 0x7f4c240ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 -- 172.21.15.104:0/2442652977 >> v1:172.21.15.114:6789/0 conn(0x7f4c2406e420 legacy=0x7f4c2406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.140+0000 7f4c23fff700 1 -- 172.21.15.104:0/2442652977 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240cef30 con 0x7f4c2406a850 2023-06-10T18:44:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c23fff700 1 --2- 172.21.15.104:0/2442652977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c2406dee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c240717c0 tx=0x7f4c10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d422ba559767074d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c237fe700 1 -- 172.21.15.104:0/2442652977 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10011830 con 0x7f4c2406a850 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c237fe700 1 -- 172.21.15.104:0/2442652977 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c10011990 con 0x7f4c2406a850 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c237fe700 1 -- 172.21.15.104:0/2442652977 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10011d20 con 0x7f4c2406a850 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/2442652977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 msgr2=0x7f4c2406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/2442652977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c2406dee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c240717c0 tx=0x7f4c10009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/2442652977 shutdown_connections 2023-06-10T18:44:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/2442652977 >> v2:172.21.15.114:3300/0 conn(0x7f4c2415c500 0x7f4c240ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/2442652977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c2406dee0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/2442652977 >> 172.21.15.104:0/2442652977 conn(0x7f4c2406b8a0 msgr2=0x7f4c240cf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/2442652977 shutdown_connections 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.141+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/2442652977 wait complete. 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.142+0000 7f4c2b1e7700 1 Processor -- start 2023-06-10T18:44:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.142+0000 7f4c2b1e7700 1 -- start start 2023-06-10T18:44:13.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c2406e420 0x7f4c240b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2415c500 0x7f4c24145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c240afd40 con 0x7f4c2406a850 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c240b02f0 con 0x7f4c2406e420 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c2b1e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2415a270 con 0x7f4c2415c500 2023-06-10T18:44:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c28f83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c28f83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54248/0 (socket says 172.21.15.104:54248) 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.143+0000 7f4c28f83700 1 -- 172.21.15.104:0/350276401 learned_addr learned my addr 172.21.15.104:0/350276401 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c29784700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2415c500 0x7f4c24145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c23fff700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c2406e420 0x7f4c240b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 -- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2415c500 msgr2=0x7f4c24145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2415c500 0x7f4c24145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 -- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c2406e420 msgr2=0x7f4c240b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c2406e420 0x7f4c240b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 -- 172.21.15.104:0/350276401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10009550 con 0x7f4c2406a850 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c28f83700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e900 tx=0x7f4c1800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18013680 con 0x7f4c2406a850 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c180137e0 con 0x7f4c2406a850 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2415a550 con 0x7f4c2406a850 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18013b50 con 0x7f4c2406a850 2023-06-10T18:44:13.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.144+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c240aff80 con 0x7f4c2406a850 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.145+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c240cd6a0 con 0x7f4c2406a850 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.145+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4c18013cb0 con 0x7f4c2406a850 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.146+0000 7f4c217fa700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 0x7f4c0806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.146+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f4c18089aa0 con 0x7f4c2406a850 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.146+0000 7f4c23fff700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 0x7f4c0806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.146+0000 7f4c23fff700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 0x7f4c0806d1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009c10 tx=0x7f4c10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.150+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4c180547f0 con 0x7f4c2406a850 2023-06-10T18:44:13.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.385+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f4c240b05d0 con 0x7f4c2406a850 2023-06-10T18:44:13.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.392+0000 7f4c217fa700 1 -- 172.21.15.104:0/350276401 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4c18021020 con 0x7f4c2406a850 2023-06-10T18:44:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 msgr2=0x7f4c0806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 0x7f4c0806d1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009c10 tx=0x7f4c10008030 comp rx=0 tx=0).stop 2023-06-10T18:44:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 msgr2=0x7f4c24157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e900 tx=0x7f4c1800ec10 comp rx=0 tx=0).stop 2023-06-10T18:44:13.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 shutdown_connections 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2415c500 0x7f4c24145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c2406a850 0x7f4c24157560 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c0806ad10 0x7f4c0806d1c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 --2- 172.21.15.104:0/350276401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c2406e420 0x7f4c240b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 >> 172.21.15.104:0/350276401 conn(0x7f4c2406b8a0 msgr2=0x7f4c240c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 shutdown_connections 2023-06-10T18:44:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.394+0000 7f4c2b1e7700 1 -- 172.21.15.104:0/350276401 wait complete. 2023-06-10T18:44:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:added data pool 5 to fsmap 2023-06-10T18:44:13.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:13.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.527+0000 7f8d90d0b700 1 Processor -- start 2023-06-10T18:44:13.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.527+0000 7f8d90d0b700 1 -- start start 2023-06-10T18:44:13.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d90d0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c06e0e0 0x7f8d8c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d90d0b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d90d0b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c06d290 con 0x7f8d8c06e4b0 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d90d0b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c071640 con 0x7f8d8c069e70 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d90d0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0aef00 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54256/0 (socket says 172.21.15.104:54256) 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 -- 172.21.15.104:0/2776551503 learned_addr learned my addr 172.21.15.104:0/2776551503 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d89d9b700 1 --1- 172.21.15.104:0/2776551503 >> v1:172.21.15.104:6789/0 conn(0x7f8d8c06e4b0 0x7f8d8c0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52082/0 (socket says 172.21.15.104:52082) 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8a59c700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c06e0e0 0x7f8d8c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 -- 172.21.15.104:0/2776551503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c06e0e0 msgr2=0x7f8d8c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c06e0e0 0x7f8d8c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 -- 172.21.15.104:0/2776551503 >> v1:172.21.15.104:6789/0 conn(0x7f8d8c06e4b0 legacy=0x7f8d8c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 -- 172.21.15.104:0/2776551503 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0ca2d0 con 0x7f8d8c069e70 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8959a700 1 -- 172.21.15.104:0/2776551503 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866132271 0 0) 0x7f8d8c06d290 con 0x7f8d8c06e4b0 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8ad9d700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009e50 tx=0x7f8d8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7af5d6f15a823a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8959a700 1 -- 172.21.15.104:0/2776551503 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8400e070 con 0x7f8d8c069e70 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8959a700 1 -- 172.21.15.104:0/2776551503 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8400b9c0 con 0x7f8d8c069e70 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.528+0000 7f8d8959a700 1 -- 172.21.15.104:0/2776551503 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8400bd70 con 0x7f8d8c069e70 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 -- 172.21.15.104:0/2776551503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c069e70 msgr2=0x7f8d8c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009e50 tx=0x7f8d8400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 -- 172.21.15.104:0/2776551503 shutdown_connections 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c069e70 0x7f8d8c0ce9f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/2776551503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c06e0e0 0x7f8d8c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 -- 172.21.15.104:0/2776551503 >> 172.21.15.104:0/2776551503 conn(0x7f8d8c06b8a0 msgr2=0x7f8d8c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 -- 172.21.15.104:0/2776551503 shutdown_connections 2023-06-10T18:44:13.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.529+0000 7f8d90d0b700 1 -- 172.21.15.104:0/2776551503 wait complete. 2023-06-10T18:44:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.530+0000 7f8d90d0b700 1 Processor -- start 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.530+0000 7f8d90d0b700 1 -- start start 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 0x7f8d8c157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c06e4b0 0x7f8d8c0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c09d1c0 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0af740 con 0x7f8d8c06e4b0 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d90d0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0afa20 con 0x7f8d8c069e70 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54260/0 (socket says 172.21.15.104:54260) 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 -- 172.21.15.104:0/3610647870 learned_addr learned my addr 172.21.15.104:0/3610647870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d8a59c700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 0x7f8d8c157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d8ad9d700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c06e4b0 0x7f8d8c0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 -- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 msgr2=0x7f8d8c157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 0x7f8d8c157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 -- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c06e4b0 msgr2=0x7f8d8c0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c06e4b0 0x7f8d8c0c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d89d9b700 1 -- 172.21.15.104:0/3610647870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d84009b30 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.531+0000 7f8d8a59c700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 0x7f8d8c157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d89d9b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d800011e0 tx=0x7f8d8000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80004c80 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c0afd00 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d80004e00 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80007310 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.532+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c0b02c0 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.533+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c074050 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.533+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d80007470 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.533+0000 7f8d7affd700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 0x7f8d6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.534+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8d800896c0 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.534+0000 7f8d8a59c700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 0x7f8d6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.535+0000 7f8d8a59c700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 0x7f8d6c06d190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006fd0 tx=0x7f8d7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.538+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d80054370 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.765+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d8c0ae2b0 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.766+0000 7f8d7affd700 1 -- 172.21.15.104:0/3610647870 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f8d80057990 con 0x7f8d8c06e0e0 2023-06-10T18:44:13.771 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:13.772 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:13.387388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-06-10T18:44:13.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 msgr2=0x7f8d6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 0x7f8d6c06d190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006fd0 tx=0x7f8d7c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 msgr2=0x7f8d8c0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d800011e0 tx=0x7f8d8000cae0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 shutdown_connections 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d6c06ace0 0x7f8d6c06d190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c06e4b0 0x7f8d8c0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d8c069e70 0x7f8d8c157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 --2- 172.21.15.104:0/3610647870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d8c06e0e0 0x7f8d8c0af040 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 >> 172.21.15.104:0/3610647870 conn(0x7f8d8c06b8a0 msgr2=0x7f8d8c0cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 shutdown_connections 2023-06-10T18:44:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.768+0000 7f8d90d0b700 1 -- 172.21.15.104:0/3610647870 wait complete. 2023-06-10T18:44:13.775 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2023-06-10T18:44:13.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:13.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.899+0000 7f1732aa2700 1 Processor -- start 2023-06-10T18:44:13.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.899+0000 7f1732aa2700 1 -- start start 2023-06-10T18:44:13.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f1732aa2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 0x7f172c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f1732aa2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f172c069e70 0x7f172c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f1732aa2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172c06d290 con 0x7f172c06e4b0 2023-06-10T18:44:13.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f1732aa2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172c071640 con 0x7f172c069e70 2023-06-10T18:44:13.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f1732aa2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172c0aef00 con 0x7f172c06e0e0 2023-06-10T18:44:13.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f173103f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f172c069e70 0x7f172c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f173083e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 0x7f172c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f173103f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f172c069e70 0x7f172c0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41768/0 (socket says 172.21.15.104:41768) 2023-06-10T18:44:13.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.900+0000 7f173103f700 1 -- 172.21.15.104:0/2484473461 learned_addr learned my addr 172.21.15.104:0/2484473461 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173103f700 1 -- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 msgr2=0x7f172c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173103f700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 0x7f172c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173103f700 1 -- 172.21.15.104:0/2484473461 >> v1:172.21.15.114:6789/0 conn(0x7f172c06e4b0 legacy=0x7f172c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173103f700 1 -- 172.21.15.104:0/2484473461 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c0ca2d0 con 0x7f172c069e70 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173083e700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 0x7f172c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f173103f700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c0ce9f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1724009e50 tx=0x7f172400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33986cb3e9b04041 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f172b7fe700 1 -- 172.21.15.104:0/2484473461 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400e070 con 0x7f172c069e70 2023-06-10T18:44:13.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f172b7fe700 1 -- 172.21.15.104:0/2484473461 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400b910 con 0x7f172c069e70 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.901+0000 7f172b7fe700 1 -- 172.21.15.104:0/2484473461 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400bcc0 con 0x7f172c069e70 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 -- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 msgr2=0x7f172c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c0ce9f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1724009e50 tx=0x7f172400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 -- 172.21.15.104:0/2484473461 shutdown_connections 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c0ce9f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 --2- 172.21.15.104:0/2484473461 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e0e0 0x7f172c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 -- 172.21.15.104:0/2484473461 >> 172.21.15.104:0/2484473461 conn(0x7f172c06b8a0 msgr2=0x7f172c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 -- 172.21.15.104:0/2484473461 shutdown_connections 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.902+0000 7f1732aa2700 1 -- 172.21.15.104:0/2484473461 wait complete. 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.903+0000 7f1732aa2700 1 Processor -- start 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.903+0000 7f1732aa2700 1 -- start start 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e4b0 0x7f172c0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c0b2430 con 0x7f172c06e0e0 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c0b1550 con 0x7f172c069e70 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172c0b1830 con 0x7f172c06e4b0 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54274/0 (socket says 172.21.15.104:54274) 2023-06-10T18:44:13.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 -- 172.21.15.104:0/3591141941 learned_addr learned my addr 172.21.15.104:0/3591141941 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f173103f700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e4b0 0x7f172c0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f173083e700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 -- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e4b0 msgr2=0x7f172c0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e4b0 0x7f172c0c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 -- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 msgr2=0x7f172c154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:13.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:13.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 -- 172.21.15.104:0/3591141941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720009430 con 0x7f172c06e0e0 2023-06-10T18:44:13.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f172bfff700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f171800c490 tx=0x7f171800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171800cea0 con 0x7f172c06e0e0 2023-06-10T18:44:13.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17180052f0 con 0x7f172c06e0e0 2023-06-10T18:44:13.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718006860 con 0x7f172c06e0e0 2023-06-10T18:44:13.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.904+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1724009b30 con 0x7f172c06e0e0 2023-06-10T18:44:13.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.905+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172c0b1df0 con 0x7f172c06e0e0 2023-06-10T18:44:13.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.906+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172c074050 con 0x7f172c06e0e0 2023-06-10T18:44:13.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.906+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1718003680 con 0x7f172c06e0e0 2023-06-10T18:44:13.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.906+0000 7f17297fa700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 0x7f171006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:13.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.906+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f17180888b0 con 0x7f172c06e0e0 2023-06-10T18:44:13.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.906+0000 7f173083e700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 0x7f171006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:13.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.907+0000 7f173083e700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 0x7f171006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1720009cb0 tx=0x7f17200091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:13.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:13.911+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1718053700 con 0x7f172c06e0e0 2023-06-10T18:44:14.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.102+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f172c0b2570 con 0x7f172c06e0e0 2023-06-10T18:44:14.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.104+0000 7f17297fa700 1 -- 172.21.15.104:0/3591141941 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f1718056d20 con 0x7f172c06e0e0 2023-06-10T18:44:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:14.110 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 msgr2=0x7f171006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 0x7f171006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1720009cb0 tx=0x7f17200091c0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 msgr2=0x7f172c0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f171800c490 tx=0x7f171800c7a0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 shutdown_connections 2023-06-10T18:44:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171006ace0 0x7f171006d190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f172c069e70 0x7f172c154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172c06e4b0 0x7f172c0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 --2- 172.21.15.104:0/3591141941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f172c06e0e0 0x7f172c0c8810 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 >> 172.21.15.104:0/3591141941 conn(0x7f172c06b8a0 msgr2=0x7f172c0ae240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 shutdown_connections 2023-06-10T18:44:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.106+0000 7f1732aa2700 1 -- 172.21.15.104:0/3591141941 wait complete. 2023-06-10T18:44:14.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:14.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 Processor -- start 2023-06-10T18:44:14.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 -- start start 2023-06-10T18:44:14.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f341815a420 0x7f34180b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 0x7f34180ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 0x7f34180b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180ced10 con 0x7f341815a420 2023-06-10T18:44:14.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180b6070 con 0x7f34180ce890 2023-06-10T18:44:14.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f341d17d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b5460 con 0x7f34180b4ff0 2023-06-10T18:44:14.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.240+0000 7f3416d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f341815a420 0x7f34180b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f341815a420 0x7f34180b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54280/0 (socket says 172.21.15.104:54280) 2023-06-10T18:44:14.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 -- 172.21.15.104:0/3901961321 learned_addr learned my addr 172.21.15.104:0/3901961321 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:14.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f341659c700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 0x7f34180ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f341759e700 1 --2- 172.21.15.104:0/3901961321 >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 0x7f34180b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 -- 172.21.15.104:0/3901961321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 msgr2=0x7f34180ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:14.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 0x7f34180ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 -- 172.21.15.104:0/3901961321 >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 msgr2=0x7f34180b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 --2- 172.21.15.104:0/3901961321 >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 0x7f34180b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 -- 172.21.15.104:0/3901961321 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b5740 con 0x7f341815a420 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f341659c700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 0x7f34180ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f341759e700 1 --2- 172.21.15.104:0/3901961321 >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 0x7f34180b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3416d9d700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f341815a420 0x7f34180b1660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f341808abf0 tx=0x7f3400009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661a9defa284e6b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3415d9b700 1 -- 172.21.15.104:0/3901961321 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340000b870 con 0x7f341815a420 2023-06-10T18:44:14.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3415d9b700 1 -- 172.21.15.104:0/3901961321 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340000b9d0 con 0x7f341815a420 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.241+0000 7f3415d9b700 1 -- 172.21.15.104:0/3901961321 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340000bd60 con 0x7f341815a420 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 -- 172.21.15.104:0/3901961321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f341815a420 msgr2=0x7f34180b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f341815a420 0x7f34180b1660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f341808abf0 tx=0x7f3400009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 -- 172.21.15.104:0/3901961321 shutdown_connections 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f341815a420 0x7f34180b1660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 --2- 172.21.15.104:0/3901961321 >> v2:172.21.15.114:3300/0 conn(0x7f34180ce890 0x7f34180b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 --2- 172.21.15.104:0/3901961321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34180b4ff0 0x7f34180ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 -- 172.21.15.104:0/3901961321 >> 172.21.15.104:0/3901961321 conn(0x7f3418089fd0 msgr2=0x7f34180d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 -- 172.21.15.104:0/3901961321 shutdown_connections 2023-06-10T18:44:14.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.242+0000 7f341d17d700 1 -- 172.21.15.104:0/3901961321 wait complete. 2023-06-10T18:44:14.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.243+0000 7f341d17d700 1 Processor -- start 2023-06-10T18:44:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.243+0000 7f341d17d700 1 -- start start 2023-06-10T18:44:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 0x7f34180c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f341815a420 0x7f34181e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180c7980 con 0x7f34180ce890 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180c47f0 con 0x7f34180b4ff0 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341d17d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180c4a90 con 0x7f341815a420 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54288/0 (socket says 172.21.15.104:54288) 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 -- 172.21.15.104:0/1745222685 learned_addr learned my addr 172.21.15.104:0/1745222685 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341759e700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f341815a420 0x7f34181e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f3416d9d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 0x7f34180c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 -- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f341815a420 msgr2=0x7f34181e0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f341815a420 0x7f34181e0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 -- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 msgr2=0x7f34180c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 0x7f34180c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 -- 172.21.15.104:0/1745222685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400009550 con 0x7f34180ce890 2023-06-10T18:44:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f3416d9d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 0x7f34180c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.244+0000 7f341659c700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f340800c7f0 tx=0x7f340800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.245+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408011800 con 0x7f34180ce890 2023-06-10T18:44:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.245+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408011960 con 0x7f34180ce890 2023-06-10T18:44:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.245+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34180c4d70 con 0x7f34180ce890 2023-06-10T18:44:14.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.245+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408011d10 con 0x7f34180ce890 2023-06-10T18:44:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.245+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34180c5330 con 0x7f34180ce890 2023-06-10T18:44:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.246+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34180b64e0 con 0x7f34180ce890 2023-06-10T18:44:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.246+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3408018360 con 0x7f34180ce890 2023-06-10T18:44:14.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.246+0000 7f340f7fe700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 0x7f33f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.246+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3408089760 con 0x7f34180ce890 2023-06-10T18:44:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.247+0000 7f3416d9d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 0x7f33f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.247+0000 7f3416d9d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 0x7f33f806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3400000c00 tx=0x7f3400008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.251+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f34180b64e0 con 0x7f34180ce890 2023-06-10T18:44:14.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.440+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34180b64e0 con 0x7f34180ce890 2023-06-10T18:44:14.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.441+0000 7f340f7fe700 1 -- 172.21.15.104:0/1745222685 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f34180b64e0 con 0x7f34180ce890 2023-06-10T18:44:14.447 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:14.447 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:14.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 msgr2=0x7f33f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 0x7f33f806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3400000c00 tx=0x7f3400008030 comp rx=0 tx=0).stop 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 msgr2=0x7f34181e0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f340800c7f0 tx=0x7f340800cbb0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 shutdown_connections 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33f806ace0 0x7f33f806d190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180b4ff0 0x7f34180c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f341815a420 0x7f34181e0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 --2- 172.21.15.104:0/1745222685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34180ce890 0x7f34181e0aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 >> 172.21.15.104:0/1745222685 conn(0x7f3418089fd0 msgr2=0x7f341806ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 shutdown_connections 2023-06-10T18:44:14.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.443+0000 7f341d17d700 1 -- 172.21.15.104:0/1745222685 wait complete. 2023-06-10T18:44:14.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:14.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.591+0000 7f9bccd51700 1 Processor -- start 2023-06-10T18:44:14.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.591+0000 7f9bccd51700 1 -- start start 2023-06-10T18:44:14.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bccd51700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9bc806e0e0 0x7f9bc806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bccd51700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bccd51700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc806d290 con 0x7f9bc806e4b0 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bccd51700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc8071640 con 0x7f9bc806e0e0 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bccd51700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80aef00 con 0x7f9bc8069e70 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc5d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9bc806e4b0 0x7f9bc80c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52124/0 (socket says 172.21.15.104:52124) 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc5d9b700 1 -- 172.21.15.104:0/3546376697 learned_addr learned my addr 172.21.15.104:0/3546376697 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc659c700 1 --2- 172.21.15.104:0/3546376697 >> v2:172.21.15.114:3300/0 conn(0x7f9bc806e0e0 0x7f9bc806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc6d9d700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc559a700 1 -- 172.21.15.104:0/3546376697 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088939908 0 0) 0x7f9bc806d290 con 0x7f9bc806e4b0 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc559a700 1 -- 172.21.15.104:0/3546376697 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb00035b0 con 0x7f9bc806e4b0 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc659c700 1 -- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 msgr2=0x7f9bc80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc659c700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.592+0000 7f9bc659c700 1 -- 172.21.15.104:0/3546376697 >> v1:172.21.15.104:6789/0 conn(0x7f9bc806e4b0 legacy=0x7f9bc80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bc659c700 1 -- 172.21.15.104:0/3546376697 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80ca2d0 con 0x7f9bc806e0e0 2023-06-10T18:44:14.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bc659c700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc806cd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009820 tx=0x7f9bbc009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5443f83923ac4882 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bc559a700 1 -- 172.21.15.104:0/3546376697 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc005e30 con 0x7f9bc806e0e0 2023-06-10T18:44:14.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bc559a700 1 -- 172.21.15.104:0/3546376697 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc006030 con 0x7f9bc806e0e0 2023-06-10T18:44:14.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bc559a700 1 -- 172.21.15.104:0/3546376697 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0063c0 con 0x7f9bc806e0e0 2023-06-10T18:44:14.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bccd51700 1 -- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 msgr2=0x7f9bc806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bccd51700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc806cd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009820 tx=0x7f9bbc009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:14.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bccd51700 1 -- 172.21.15.104:0/3546376697 shutdown_connections 2023-06-10T18:44:14.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.593+0000 7f9bccd51700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.594+0000 7f9bccd51700 1 --2- 172.21.15.104:0/3546376697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc806cd50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.594+0000 7f9bccd51700 1 -- 172.21.15.104:0/3546376697 >> 172.21.15.104:0/3546376697 conn(0x7f9bc806b8a0 msgr2=0x7f9bc80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.594+0000 7f9bccd51700 1 -- 172.21.15.104:0/3546376697 shutdown_connections 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.594+0000 7f9bccd51700 1 -- 172.21.15.104:0/3546376697 wait complete. 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.595+0000 7f9bccd51700 1 Processor -- start 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.595+0000 7f9bccd51700 1 -- start start 2023-06-10T18:44:14.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc81e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81e1570 con 0x7f9bc806e4b0 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81e1850 con 0x7f9bc806e0e0 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81e1b30 con 0x7f9bc8069e70 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54304/0 (socket says 172.21.15.104:54304) 2023-06-10T18:44:14.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 -- 172.21.15.104:0/1062632389 learned_addr learned my addr 172.21.15.104:0/1062632389 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc5d9b700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc81e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc659c700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 -- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 msgr2=0x7f9bc80b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 -- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 msgr2=0x7f9bc81e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc81e2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 -- 172.21.15.104:0/1062632389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc009430 con 0x7f9bc806e4b0 2023-06-10T18:44:14.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bc6d9d700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000efc0 tx=0x7f9bc000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000cad0 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc000f040 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc81e1e10 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000ce70 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.596+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc80b1480 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.597+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc8074050 con 0x7f9bc806e4b0 2023-06-10T18:44:14.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.598+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9bc0014480 con 0x7f9bc806e4b0 2023-06-10T18:44:14.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.598+0000 7f9bb6ffd700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 0x7f9ba806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.598+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9bc008a220 con 0x7f9bc806e4b0 2023-06-10T18:44:14.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.598+0000 7f9bc659c700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 0x7f9ba806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.599+0000 7f9bc659c700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 0x7f9ba806d190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000c00 tx=0x7f9bbc008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.602+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9bc0054f60 con 0x7f9bc806e4b0 2023-06-10T18:44:14.667 INFO:tasks.ceph.mds.d.smithi114.stderr:2023-06-10T18:44:14.662+0000 7fbb094e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:44:14.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.830+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bc8072d10 con 0x7f9bc806e4b0 2023-06-10T18:44:14.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.830+0000 7f9bb6ffd700 1 -- 172.21.15.104:0/1062632389 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f9bc0058580 con 0x7f9bc806e4b0 2023-06-10T18:44:14.836 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:14.836 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:13.387388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-06-10T18:44:14.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.832+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 msgr2=0x7f9ba806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.832+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 0x7f9ba806d190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000c00 tx=0x7f9bbc008070 comp rx=0 tx=0).stop 2023-06-10T18:44:14.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.832+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 msgr2=0x7f9bc81f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.832+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000efc0 tx=0x7f9bc000c0c0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 shutdown_connections 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bc806e4b0 0x7f9bc81f2a10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bc8069e70 0x7f9bc80b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9ba806ace0 0x7f9ba806d190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 --2- 172.21.15.104:0/1062632389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bc806e0e0 0x7f9bc81e2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 >> 172.21.15.104:0/1062632389 conn(0x7f9bc806b8a0 msgr2=0x7f9bc80ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 shutdown_connections 2023-06-10T18:44:14.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.833+0000 7f9bccd51700 1 -- 172.21.15.104:0/1062632389 wait complete. 2023-06-10T18:44:14.840 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 7 2023-06-10T18:44:14.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:14.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.964+0000 7fb4dfd35700 1 Processor -- start 2023-06-10T18:44:14.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.964+0000 7fb4dfd35700 1 -- start start 2023-06-10T18:44:14.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dfd35700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4d80cd850 0x7fb4d806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dfd35700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4d80cb300 0x7fb4d806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dfd35700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d80c9e60 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dfd35700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80c9bc0 con 0x7fb4d80cd850 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dfd35700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80aee70 con 0x7fb4d80cb300 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4d80cd850 0x7fb4d806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4d80cd850 0x7fb4d806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54308/0 (socket says 172.21.15.104:54308) 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 -- 172.21.15.104:0/3223455685 learned_addr learned my addr 172.21.15.104:0/3223455685 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:14.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 -- 172.21.15.104:0/3223455685 >> v2:172.21.15.114:3300/0 conn(0x7fb4d80cb300 msgr2=0x7fb4d806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 --2- 172.21.15.104:0/3223455685 >> v2:172.21.15.114:3300/0 conn(0x7fb4d80cb300 0x7fb4d806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.965+0000 7fb4dd2d0700 1 -- 172.21.15.104:0/3223455685 >> v1:172.21.15.114:6789/0 conn(0x7fb4d80aeaa0 legacy=0x7fb4d80cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dd2d0700 1 -- 172.21.15.104:0/3223455685 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d806e540 con 0x7fb4d80cd850 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dd2d0700 1 --2- 172.21.15.104:0/3223455685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80cd850 0x7fb4d806cd50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80722a0 tx=0x7fb4d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6b0f2e1db7f34b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dcacf700 1 -- 172.21.15.104:0/3223455685 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4004230 con 0x7fb4d80cd850 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dcacf700 1 -- 172.21.15.104:0/3223455685 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d4004390 con 0x7fb4d80cd850 2023-06-10T18:44:14.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dcacf700 1 -- 172.21.15.104:0/3223455685 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40036a0 con 0x7fb4d80cd850 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3223455685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80cd850 msgr2=0x7fb4d806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3223455685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80cd850 0x7fb4d806cd50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80722a0 tx=0x7fb4d4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3223455685 shutdown_connections 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3223455685 >> v2:172.21.15.114:3300/0 conn(0x7fb4d80cb300 0x7fb4d806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3223455685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80cd850 0x7fb4d806cd50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3223455685 >> 172.21.15.104:0/3223455685 conn(0x7fb4d806b8a0 msgr2=0x7fb4d80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:14.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3223455685 shutdown_connections 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.966+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3223455685 wait complete. 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 Processor -- start 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 -- start start 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d80cb300 0x7fb4d80b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4d80cd850 0x7fb4d80c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80b2c90 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d815b100 con 0x7fb4d80cd850 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4dfd35700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d815b3e0 con 0x7fb4d80cb300 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4ddad1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4ddad1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54312/0 (socket says 172.21.15.104:54312) 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.968+0000 7fb4ddad1700 1 -- 172.21.15.104:0/3056760967 learned_addr learned my addr 172.21.15.104:0/3056760967 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4dd2d0700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d80cb300 0x7fb4d80b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4de2d2700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4d80cd850 0x7fb4d80c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 -- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d80cb300 msgr2=0x7fb4d80b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d80cb300 0x7fb4d80b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 -- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4d80cd850 msgr2=0x7fb4d80c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4d80cd850 0x7fb4d80c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 -- 172.21.15.104:0/3056760967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d4009550 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ddad1700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c520 tx=0x7fb4c800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80054e0 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80036d0 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8003a80 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d815b6c0 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.969+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d815bc80 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.970+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d8074050 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.970+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb4c8003be0 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.971+0000 7fb4ce7fc700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 0x7fb4bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.971+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb4c80887b0 con 0x7fb4d80aeaa0 2023-06-10T18:44:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.971+0000 7fb4dd2d0700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 0x7fb4bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.971+0000 7fb4dd2d0700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 0x7fb4bc06d190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009130 tx=0x7fb4d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:14.975+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4c8053570 con 0x7fb4d80aeaa0 2023-06-10T18:44:15.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.161+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4d80c6400 con 0x7fb4d80aeaa0 2023-06-10T18:44:15.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.162+0000 7fb4ce7fc700 1 -- 172.21.15.104:0/3056760967 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7fb4c8056b90 con 0x7fb4d80aeaa0 2023-06-10T18:44:15.168 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:15.168 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 msgr2=0x7fb4bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 0x7fb4bc06d190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009130 tx=0x7fb4d4008030 comp rx=0 tx=0).stop 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 msgr2=0x7fb4d815c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c520 tx=0x7fb4c800c830 comp rx=0 tx=0).stop 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 shutdown_connections 2023-06-10T18:44:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4bc06ace0 0x7fb4bc06d190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4d80cd850 0x7fb4d80c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4d80aeaa0 0x7fb4d815c1a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 --2- 172.21.15.104:0/3056760967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d80cb300 0x7fb4d80b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 >> 172.21.15.104:0/3056760967 conn(0x7fb4d806b8a0 msgr2=0x7fb4d80af530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 shutdown_connections 2023-06-10T18:44:15.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.164+0000 7fb4dfd35700 1 -- 172.21.15.104:0/3056760967 wait complete. 2023-06-10T18:44:15.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:44:08.959124+0000', 'modified': '2023-06-10T18:44:13.387388+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4296': {'gid': 4296, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/3495904966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3495904966}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 3495904966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-06-10T18:44:15.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:44:15.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:44:15.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.700+0000 7f4f8e327700 1 Processor -- start 2023-06-10T18:44:15.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.700+0000 7f4f8e327700 1 -- start start 2023-06-10T18:44:15.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f8e327700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4f880cc710 0x7f4f8806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f8e327700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8815cb00 con 0x7f4f8815c730 2023-06-10T18:44:15.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f8e327700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8806ecd0 con 0x7f4f880ca0b0 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f8e327700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f880aed80 con 0x7f4f880cc710 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f87fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4f8815c730 0x7f4f880cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52148/0 (socket says 172.21.15.104:52148) 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f87fff700 1 -- 172.21.15.104:0/2498900643 learned_addr learned my addr 172.21.15.104:0/2498900643 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796159376 0 0) 0x7f4f7c001290 con 0x7f4f8815c730 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f700035b0 con 0x7f4f8815c730 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2498900643 >> v2:172.21.15.114:3300/0 conn(0x7f4f880cc710 0x7f4f8806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.701+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71093160 0 0) 0x7f4f700035b0 con 0x7f4f8815c730 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 >> v2:172.21.15.114:3300/0 conn(0x7f4f880cc710 msgr2=0x7f4f8806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 --2- 172.21.15.104:0/2498900643 >> v2:172.21.15.114:3300/0 conn(0x7f4f880cc710 0x7f4f8806e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 >> v1:172.21.15.114:6789/0 conn(0x7f4f880ca0b0 legacy=0x7f4f880ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880cd280 con 0x7f4f8815c730 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4f7c001980 con 0x7f4f8815c730 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 >> v1:172.21.15.104:6789/0 conn(0x7f4f8815c730 legacy=0x7f4f880cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f70003a00 0x7f4f70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70004850 0x7f4f70004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 0x7f4f7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880aed80 con 0x7f4f70005320 2023-06-10T18:44:15.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7000c1e0 con 0x7f4f70003a00 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7000e5b0 con 0x7f4f70004850 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 0x7f4f7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f70003a00 0x7f4f70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70004850 0x7f4f70004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70004850 msgr2=0x7f4f70004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70004850 0x7f4f70004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f70003a00 msgr2=0x7f4f70003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f70003a00 0x7f4f70003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2498900643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f700035b0 con 0x7f4f70005320 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f70003a00 0x7f4f70003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2498900643 reap_dead start 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 0x7f4f7000ba10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c190 tx=0x7f4f70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3667b03549250b12 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80001db0 con 0x7f4f70005320 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8000ccf0 con 0x7f4f70005320 2023-06-10T18:44:15.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.702+0000 7f4f86ffd700 1 -- 172.21.15.104:0/2498900643 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80003c40 con 0x7f4f70005320 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 -- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 msgr2=0x7f4f7000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 0x7f4f7000ba10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c190 tx=0x7f4f70005880 comp rx=0 tx=0).stop 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 -- 172.21.15.104:0/2498900643 shutdown_connections 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 --2- 172.21.15.104:0/2498900643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f70005320 0x7f4f7000ba10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 -- 172.21.15.104:0/2498900643 >> 172.21.15.104:0/2498900643 conn(0x7f4f8806b8a0 msgr2=0x7f4f880cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 -- 172.21.15.104:0/2498900643 shutdown_connections 2023-06-10T18:44:15.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.703+0000 7f4f8e327700 1 -- 172.21.15.104:0/2498900643 wait complete. 2023-06-10T18:44:15.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.704+0000 7f4f8e327700 1 Processor -- start 2023-06-10T18:44:15.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.704+0000 7f4f8e327700 1 -- start start 2023-06-10T18:44:15.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70005320 0x7f4f881ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 0x7f4f8806d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8806daa0 con 0x7f4f8806d620 2023-06-10T18:44:15.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f881eec40 con 0x7f4f881f0370 2023-06-10T18:44:15.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8e327700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f880b4150 con 0x7f4f70005320 2023-06-10T18:44:15.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54334/0 (socket says 172.21.15.104:54334) 2023-06-10T18:44:15.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/4243814370 learned_addr learned my addr 172.21.15.104:0/4243814370 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f87fff700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70005320 0x7f4f881ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f877fe700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 0x7f4f8806d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70005320 msgr2=0x7f4f881ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70005320 0x7f4f881ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 msgr2=0x7f4f8806d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 0x7f4f8806d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/4243814370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880b4430 con 0x7f4f8806d620 2023-06-10T18:44:15.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.705+0000 7f4f877fe700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 0x7f4f8806d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:15.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005040 tx=0x7f4f800014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f84ff9700 1 -- 172.21.15.104:0/4243814370 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f800052d0 con 0x7f4f8806d620 2023-06-10T18:44:15.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f84ff9700 1 -- 172.21.15.104:0/4243814370 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80005430 con 0x7f4f8806d620 2023-06-10T18:44:15.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f880b08a0 con 0x7f4f8806d620 2023-06-10T18:44:15.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f84ff9700 1 -- 172.21.15.104:0/4243814370 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f800057a0 con 0x7f4f8806d620 2023-06-10T18:44:15.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.706+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f880b04e0 con 0x7f4f8806d620 2023-06-10T18:44:15.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.707+0000 7f4f84ff9700 1 -- 172.21.15.104:0/4243814370 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4f8001d480 con 0x7f4f8806d620 2023-06-10T18:44:15.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.707+0000 7f4f84ff9700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 0x7f4f7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.708+0000 7f4f84ff9700 1 -- 172.21.15.104:0/4243814370 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4f8008a150 con 0x7f4f8806d620 2023-06-10T18:44:15.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.708+0000 7f4f87fff700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 0x7f4f7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.708+0000 7f4f87fff700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 0x7f4f7406d190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c000c00 tx=0x7f4f7c001290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 Processor -- start 2023-06-10T18:44:15.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 -- start start 2023-06-10T18:44:15.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d8619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8651a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d73fc60 con 0x55915d83b270 2023-06-10T18:44:15.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d7929f0 con 0x55915d838740 2023-06-10T18:44:15.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55915d78be80 con 0x55915d861f00 2023-06-10T18:44:15.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54338/0 (socket says 172.21.15.104:54338) 2023-06-10T18:44:15.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f87fff700 1 -- 172.21.15.104:0/389870913 learned_addr learned my addr 172.21.15.104:0/389870913 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.710+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f877fe700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d8619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 -- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 msgr2=0x55915d8651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8651a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 -- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 msgr2=0x55915d8619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d8619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 -- 172.21.15.104:0/389870913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55915d60d5e0 con 0x55915d83b270 2023-06-10T18:44:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8651a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f877fe700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d8619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f87fff700 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c007200 tx=0x7f4f7c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed64e0a93e49ea7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f6dffb700 1 -- 172.21.15.104:0/389870913 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c015750 con 0x55915d83b270 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f6dffb700 1 -- 172.21.15.104:0/389870913 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0158b0 con 0x55915d83b270 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f6dffb700 1 -- 172.21.15.104:0/389870913 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c015c20 con 0x55915d83b270 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/389870913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 msgr2=0x55915d83b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c007200 tx=0x7f4f7c015000 comp rx=0 tx=0).stop 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/389870913 shutdown_connections 2023-06-10T18:44:15.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8651a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d83b640 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/389870913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d8619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/389870913 >> 172.21.15.104:0/389870913 conn(0x55915d82ffc0 msgr2=0x55915d8801a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/389870913 shutdown_connections 2023-06-10T18:44:15.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.711+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/389870913 wait complete. 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.712+0000 7f4f9a4acb80 1 -- start start 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.712+0000 7f4f9a4acb80 1 Processor -- start 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d838740 0x55915d85dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d882890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d72ba70 con 0x55915d861f00 2023-06-10T18:44:15.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d72b770 con 0x55915d83b270 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55915d60f8d0 con 0x55915d838740 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f9a4acb80 10 client.0 _ll_register_callbacks cb 0x55915d731210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54348/0 (socket says 172.21.15.104:54348) 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2878478685 learned_addr learned my addr 172.21.15.104:0/2878478685 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d838740 0x55915d85dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d882890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d838740 msgr2=0x55915d85dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d838740 0x55915d85dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 msgr2=0x55915d882890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d882890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f8c8c4700 1 -- 172.21.15.104:0/2878478685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f880cd280 con 0x55915d861f00 2023-06-10T18:44:15.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.713+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d882890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f80055210 tx=0x7f4f8005cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63d4878a9cff82a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:44:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80060b20 con 0x55915d861f00 2023-06-10T18:44:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4f80060c80 con 0x55915d861f00 2023-06-10T18:44:15.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55915d82d780 con 0x55915d861f00 2023-06-10T18:44:15.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f80070100 con 0x55915d861f00 2023-06-10T18:44:15.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 10 client.4437 fetch_fsmap learned FSMap version 7 2023-06-10T18:44:15.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55915d5b9d90 con 0x55915d861f00 2023-06-10T18:44:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80087080 con 0x55915d861f00 2023-06-10T18:44:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f4f80070260 con 0x55915d861f00 2023-06-10T18:44:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 10 client.4437 fetch_fsmap finished waiting for FSMap version 7 2023-06-10T18:44:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 10 client.4437 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:44:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 20 client.4437 populate_metadata read hostname 'smithi104' 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8675a0 0x55915d867970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 4 client.4437 mds_command: new command op to 4296 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 --> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55915d784e60 con 0x55915d8675a0 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.714+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8675a0 0x55915d867970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.715+0000 7f4f66ffd700 10 client.4437 ms_handle_connect on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.715+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8675a0 0x55915d867970 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2ca9219d760dee server_cookie=d1f56cbcc7998d7d in_seq=0 out_seq=0 2023-06-10T18:44:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f66ffd700 1 -- 172.21.15.104:0/2878478685 <== mds.0 v2:172.21.15.114:6834/3495904966 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55915d784e60 con 0x55915d8675a0 2023-06-10T18:44:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f66ffd700 10 client.4437 handle_command_reply: tid=0 2023-06-10T18:44:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 client.4437 shutdown 2023-06-10T18:44:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 msgr2=0x55915d882dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f80055210 tx=0x7f4f8005cd10 comp rx=0 tx=0).stop 2023-06-10T18:44:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 shutdown_connections 2023-06-10T18:44:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8675a0 0x55915d867970 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d861f00 0x55915d882dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d838740 0x55915d85dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f66ffd700 0 client.4437 ms_handle_reset on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2878478685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d882890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 >> 172.21.15.104:0/2878478685 conn(0x55915d82ffc0 msgr2=0x55915d83a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 shutdown_connections 2023-06-10T18:44:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2878478685 wait complete. 2023-06-10T18:44:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.716+0000 7f4f9a4acb80 20 client.4437 trim_cache size 0 max 16384 2023-06-10T18:44:15.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 Processor -- start 2023-06-10T18:44:15.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 -- start start 2023-06-10T18:44:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d867970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8622d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d863c80 con 0x55915d83b270 2023-06-10T18:44:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d865460 con 0x55915d838740 2023-06-10T18:44:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55915d8693f0 con 0x55915d861f00 2023-06-10T18:44:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f877fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f877fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54352/0 (socket says 172.21.15.104:54352) 2023-06-10T18:44:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.734+0000 7f4f877fe700 1 -- 172.21.15.104:0/2456069948 learned_addr learned my addr 172.21.15.104:0/2456069948 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 -- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 msgr2=0x55915d8622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8622d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f87fff700 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d867970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8622d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 -- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 msgr2=0x55915d867970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d867970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 -- 172.21.15.104:0/2456069948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55915d5a2c10 con 0x55915d83b270 2023-06-10T18:44:15.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f877fe700 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4f780099d0 tx=0x7f4f78009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bfbcd92511f98a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f6e7fc700 1 -- 172.21.15.104:0/2456069948 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004220 con 0x55915d83b270 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f6e7fc700 1 -- 172.21.15.104:0/2456069948 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7800c040 con 0x55915d83b270 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f6e7fc700 1 -- 172.21.15.104:0/2456069948 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78016860 con 0x55915d83b270 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2456069948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 msgr2=0x55915d867eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.735+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4f780099d0 tx=0x7f4f78009ce0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2456069948 shutdown_connections 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d838740 0x55915d867970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d8622d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/2456069948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d83b270 0x55915d867eb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2456069948 >> 172.21.15.104:0/2456069948 conn(0x55915d82ffc0 msgr2=0x55915d87f290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2456069948 shutdown_connections 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/2456069948 wait complete. 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 -- start start 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.736+0000 7f4f9a4acb80 1 Processor -- start 2023-06-10T18:44:15.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d89d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d89d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d860850 con 0x55915d838740 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55915d860640 con 0x55915d83b270 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55915d8682b0 con 0x55915d861f00 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f9a4acb80 10 client.0 _ll_register_callbacks cb 0x55915d731210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:44:15.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54358/0 (socket says 172.21.15.104:54358) 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 -- 172.21.15.104:0/220849905 learned_addr learned my addr 172.21.15.104:0/220849905 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f8c8c4700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d89d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f877fe700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d89d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 -- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 msgr2=0x55915d89d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d89d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 -- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 msgr2=0x55915d89d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d89d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 -- 172.21.15.104:0/220849905 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7800e040 con 0x55915d838740 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f87fff700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c001510 tx=0x7f4f7c001d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b931866fe0389c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.737+0000 7f4f66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c03a070 con 0x55915d838740 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4f7c036550 con 0x55915d838740 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f7c0158b0 con 0x55915d838740 2023-06-10T18:44:15.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7c032040 con 0x55915d838740 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 10 client.4443 fetch_fsmap learned FSMap version 7 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55915d5e0200 con 0x55915d838740 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0158b0 con 0x55915d838740 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f4f7c0368e0 con 0x55915d838740 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 10 client.4443 fetch_fsmap finished waiting for FSMap version 7 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 10 client.4443 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:44:15.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 20 client.4443 populate_metadata read hostname 'smithi104' 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8eea90 0x55915d8eee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 4 client.4443 mds_command: new command op to 4296 tid=0[{"prefix": "status"}] 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 --> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55915d747000 con 0x55915d8eea90 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.738+0000 7f4f877fe700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8eea90 0x55915d8eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f877fe700 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8eea90 0x55915d8eee60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d65ec55176612a70 server_cookie=fb6ee2646beab298 in_seq=0 out_seq=0 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f66ffd700 10 client.4443 ms_handle_connect on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:15.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f66ffd700 1 -- 172.21.15.104:0/220849905 <== mds.0 v2:172.21.15.114:6834/3495904966 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x55915d747000 con 0x55915d8eea90 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f66ffd700 10 client.4443 handle_command_reply: tid=0 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 client.4443 shutdown 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 msgr2=0x55915d867d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c001510 tx=0x7f4f7c001d10 comp rx=0 tx=0).stop 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 shutdown_connections 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55915d8eea90 0x55915d8eee60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55915d861f00 0x55915d89d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f66ffd700 0 client.4443 ms_handle_reset on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55915d838740 0x55915d867d70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 --2- 172.21.15.104:0/220849905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55915d83b270 0x55915d89d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 >> 172.21.15.104:0/220849905 conn(0x55915d82ffc0 msgr2=0x55915d83a760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 shutdown_connections 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 1 -- 172.21.15.104:0/220849905 wait complete. 2023-06-10T18:44:15.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.739+0000 7f4f9a4acb80 20 client.4443 trim_cache size 0 max 16384 2023-06-10T18:44:15.781 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:44:15.781 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:44:15.781 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:44:15.781 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4296, 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 6.777513398, 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 7, 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 27, 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 26, 2023-06-10T18:44:15.782 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 12.088475212000001 2023-06-10T18:44:15.783 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:44:15.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 msgr2=0x7f4f7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 0x7f4f7406d190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c000c00 tx=0x7f4f7c001290 comp rx=0 tx=0).stop 2023-06-10T18:44:15.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 msgr2=0x7f4f880b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005040 tx=0x7f4f800014a0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 shutdown_connections 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f8806d620 0x7f4f880b3490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f70005320 0x7f4f881ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f7406ace0 0x7f4f7406d190 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 --2- 172.21.15.104:0/4243814370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f881f0370 0x7f4f8806d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 >> 172.21.15.104:0/4243814370 conn(0x7f4f8806b8a0 msgr2=0x7f4f880ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 shutdown_connections 2023-06-10T18:44:15.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.740+0000 7f4f8e327700 1 -- 172.21.15.104:0/4243814370 wait complete. 2023-06-10T18:44:15.786 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:44:15.787 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:15.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:44:15.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.902+0000 7f6b1b597700 1 Processor -- start 2023-06-10T18:44:15.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.902+0000 7f6b1b597700 1 -- start start 2023-06-10T18:44:15.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.902+0000 7f6b1b597700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b140aeaa0 0x7f6b140cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b1b597700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b140b1080 0x7f6b140cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b1b597700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b140c9e60 con 0x7f6b140ca2f0 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b1b597700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b140c9bc0 con 0x7f6b140b1080 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b1b597700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b140aee70 con 0x7f6b140aeaa0 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b140b1080 0x7f6b140cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b19b34700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6b140ca2f0 0x7f6b140ca710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52198/0 (socket says 172.21.15.104:52198) 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b19b34700 1 -- 172.21.15.104:0/4156916759 learned_addr learned my addr 172.21.15.104:0/4156916759 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b19333700 1 --2- 172.21.15.104:0/4156916759 >> v2:172.21.15.114:3300/0 conn(0x7f6b140aeaa0 0x7f6b140cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 -- 172.21.15.104:0/4156916759 >> v2:172.21.15.114:3300/0 conn(0x7f6b140aeaa0 msgr2=0x7f6b140cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 --2- 172.21.15.104:0/4156916759 >> v2:172.21.15.114:3300/0 conn(0x7f6b140aeaa0 0x7f6b140cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 -- 172.21.15.104:0/4156916759 >> v1:172.21.15.104:6789/0 conn(0x7f6b140ca2f0 legacy=0x7f6b140ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b0bfff700 1 -- 172.21.15.104:0/4156916759 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876501946 0 0) 0x7f6b140c9e60 con 0x7f6b140ca2f0 2023-06-10T18:44:15.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 -- 172.21.15.104:0/4156916759 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1406e540 con 0x7f6b140b1080 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b18b32700 1 --2- 172.21.15.104:0/4156916759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140b1080 0x7f6b140cd680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cf5d0 tx=0x7f6b1000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa03617bffe0746e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b0bfff700 1 -- 172.21.15.104:0/4156916759 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000dc70 con 0x7f6b140b1080 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b0bfff700 1 -- 172.21.15.104:0/4156916759 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000ddd0 con 0x7f6b140b1080 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.903+0000 7f6b0bfff700 1 -- 172.21.15.104:0/4156916759 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10010690 con 0x7f6b140b1080 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 -- 172.21.15.104:0/4156916759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140b1080 msgr2=0x7f6b140cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 --2- 172.21.15.104:0/4156916759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140b1080 0x7f6b140cd680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cf5d0 tx=0x7f6b1000bdc0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 -- 172.21.15.104:0/4156916759 shutdown_connections 2023-06-10T18:44:15.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 --2- 172.21.15.104:0/4156916759 >> v2:172.21.15.114:3300/0 conn(0x7f6b140aeaa0 0x7f6b140cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 --2- 172.21.15.104:0/4156916759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140b1080 0x7f6b140cd680 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 -- 172.21.15.104:0/4156916759 >> 172.21.15.104:0/4156916759 conn(0x7f6b1406b8a0 msgr2=0x7f6b140cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:15.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 -- 172.21.15.104:0/4156916759 shutdown_connections 2023-06-10T18:44:15.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.904+0000 7f6b1b597700 1 -- 172.21.15.104:0/4156916759 wait complete. 2023-06-10T18:44:15.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.905+0000 7f6b1b597700 1 Processor -- start 2023-06-10T18:44:15.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.905+0000 7f6b1b597700 1 -- start start 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b140aeaa0 0x7f6b14151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b140b1080 0x7f6b140ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14152150 con 0x7f6b140ca2f0 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1415b010 con 0x7f6b140aeaa0 2023-06-10T18:44:15.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1415a610 con 0x7f6b140b1080 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54374/0 (socket says 172.21.15.104:54374) 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 -- 172.21.15.104:0/2835756941 learned_addr learned my addr 172.21.15.104:0/2835756941 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b18b32700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b140b1080 0x7f6b140ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19333700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b140aeaa0 0x7f6b14151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 -- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b140b1080 msgr2=0x7f6b140ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b140b1080 0x7f6b140ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 -- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b140aeaa0 msgr2=0x7f6b14151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b140aeaa0 0x7f6b14151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 -- 172.21.15.104:0/2835756941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1000b740 con 0x7f6b140ca2f0 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b19b34700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00cba0 tx=0x7f6b0c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00ab20 con 0x7f6b140ca2f0 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c00aca0 con 0x7f6b140ca2f0 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1415a050 con 0x7f6b140ca2f0 2023-06-10T18:44:15.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.906+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c007310 con 0x7f6b140ca2f0 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.907+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b140b2560 con 0x7f6b140ca2f0 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.908+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14074050 con 0x7f6b140ca2f0 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.908+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b0c007470 con 0x7f6b140ca2f0 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.908+0000 7f6b09ffb700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 0x7f6af806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.908+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6b0c0894f0 con 0x7f6b140ca2f0 2023-06-10T18:44:15.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.908+0000 7f6b19333700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 0x7f6af806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:15.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.909+0000 7f6b19333700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 0x7f6af806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b04006fd0 tx=0x7f6b04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:15.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:15.912+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6b0c054340 con 0x7f6b140ca2f0 2023-06-10T18:44:16.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.144+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6b140af4c0 con 0x7f6b140ca2f0 2023-06-10T18:44:16.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.144+0000 7f6b09ffb700 1 -- 172.21.15.104:0/2835756941 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6b0c057960 con 0x7f6b140ca2f0 2023-06-10T18:44:16.150 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-10T18:44:16.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 msgr2=0x7f6af806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:16.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 0x7f6af806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b04006fd0 tx=0x7f6b04008040 comp rx=0 tx=0).stop 2023-06-10T18:44:16.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 msgr2=0x7f6b140ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:16.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00cba0 tx=0x7f6b0c00cf60 comp rx=0 tx=0).stop 2023-06-10T18:44:16.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 shutdown_connections 2023-06-10T18:44:16.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6af806ace0 0x7f6af806d190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:16.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b140aeaa0 0x7f6b14151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:16.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b140ca2f0 0x7f6b140ad7c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:16.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 --2- 172.21.15.104:0/2835756941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b140b1080 0x7f6b140ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:16.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 >> 172.21.15.104:0/2835756941 conn(0x7f6b1406b8a0 msgr2=0x7f6b14158700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:16.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 shutdown_connections 2023-06-10T18:44:16.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:16.146+0000 7f6b1b597700 1 -- 172.21.15.104:0/2835756941 wait complete. 2023-06-10T18:44:16.164 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-06-10T18:44:16.164 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:44:16.164 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:44:16.165 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-06-10T18:44:16.165 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:16.165 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-06-10T18:44:16.165 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:44:16.165 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:44:16.166 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-06-10T18:44:16.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.167 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:44:16.184 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:16.184 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:16.184 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:16.185 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:16.185 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:16.185 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:16.185 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:16.186 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:16.186 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:16.186 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:16.186 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42428sec preferred_lft 42428sec 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:44:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:16.188 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-10T18:44:16.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.189 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:16.189 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:16.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2023-06-10T18:44:16.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2023-06-10T18:44:16.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2023-06-10T18:44:16.190 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-10T18:44:16.190 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:16.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.349 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:44:16.377 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:44:16.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.379 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:44:16.458 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:44:16.458 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:44:16.459 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:44:16.459 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:44:16.459 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:44:16.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.462 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:16.463 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:16.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:44:16.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:44:16.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:44:16.466 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:16.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.742 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:44:16.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.760 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2023-06-10T18:44:16.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.816 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:16.816 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:16.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-06-10T18:44:16.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-06-10T18:44:16.817 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:16.908 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-06-10T18:44:16.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:16.909 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:16.909 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:16.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-06-10T18:44:16.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-10T18:44:16.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-06-10T18:44:16.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-06-10T18:44:16.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-06-10T18:44:16.911 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:17.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:17.126 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:17.126 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:17.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:44:17.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2023-06-10T18:44:17.127 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:17.231 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-06-10T18:44:17.231 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:17.232 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:17.250 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-06-10T18:44:17.250 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:17.251 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:17.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:17.375 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:17.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:17.441 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:17.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:17.446 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:17.503 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:44:17.503 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-06-10T18:44:17.545 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:17.571 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:17.571 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:17.637 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:17.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:17.639 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:17.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:2023-06-10T18:44:17.635+0000 7faa098f35c0 -1 init, newargv = 0x56090fa05290 newargc=15 2023-06-10T18:44:17.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[107831]: starting ceph client 2023-06-10T18:44:17.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[107831]: starting fuse 2023-06-10T18:44:17.657 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:44:17.657 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:44:17.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:44:17.658 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:44:17.658 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:44:17.658 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:44:17.659 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:44:17.659 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:17.659 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:44:17.660 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-06-10T18:44:17.660 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:44:17.661 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-06-10T18:44:17.661 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:17.662 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:44:17.662 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:44:17.663 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:44:17.663 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:44:17.663 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:17.663 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:44:17.664 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:44:17.664 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:44:17.664 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:44:17.664 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:44:17.665 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:44:17.665 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:44:17.665 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:44:17.666 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:17.666 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.admin")) 2023-06-10T18:44:17.666 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:44:17.780 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.admin.107831.asok 2023-06-10T18:44:17.963 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.107831.asok 2023-06-10T18:44:17.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:17.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.107831.asok status 2023-06-10T18:44:18.119 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:44:18.119 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "admin", 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-06-10T18:44:18.120 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "107831", 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4455, 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:44:18.121 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4455 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2226763707 2023-06-10T18:44:18.122 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2226763707 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.4455 192.168.0.1:0/2226763707", 2023-06-10T18:44:18.123 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2226763707", 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 7, 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 27, 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:44:18.124 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:44:18.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:18.130 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:44:18.130 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:18.147 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:44:18.147 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:18.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:18.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:18.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:18.215 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-06-10T18:44:18.395 INFO:teuthology.orchestra.run.smithi104.stdout:# file: . 2023-06-10T18:44:18.395 INFO:teuthology.orchestra.run.smithi104.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-06-10T18:44:18.395 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:18.396 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:44:18.396 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:44:18.414 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:18.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.569+0000 7f3d2006d700 1 Processor -- start 2023-06-10T18:44:18.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 -- start start 2023-06-10T18:44:18.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3d181567f0 0x7f3d180cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 0x7f3d180cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d180cb2d0 con 0x7f3d180701a0 2023-06-10T18:44:18.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1815cca0 con 0x7f3d181567f0 2023-06-10T18:44:18.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.570+0000 7f3d2006d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1815cf80 con 0x7f3d1815c7a0 2023-06-10T18:44:18.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3d181567f0 0x7f3d180cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1e60a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3d180701a0 0x7f3d180705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52234/0 (socket says 172.21.15.104:52234) 2023-06-10T18:44:18.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1e60a700 1 -- 172.21.15.104:0/854952041 learned_addr learned my addr 172.21.15.104:0/854952041 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:18.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1d608700 1 --2- 172.21.15.104:0/854952041 >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 0x7f3d180cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 -- 172.21.15.104:0/854952041 >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 msgr2=0x7f3d180cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:18.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 --2- 172.21.15.104:0/854952041 >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 0x7f3d180cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 -- 172.21.15.104:0/854952041 >> v1:172.21.15.104:6789/0 conn(0x7f3d180701a0 legacy=0x7f3d180705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1ce07700 1 -- 172.21.15.104:0/854952041 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286950443 0 0) 0x7f3d180cb2d0 con 0x7f3d180701a0 2023-06-10T18:44:18.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 -- 172.21.15.104:0/854952041 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1815d260 con 0x7f3d181567f0 2023-06-10T18:44:18.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1d608700 1 --2- 172.21.15.104:0/854952041 >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 0x7f3d180cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:18.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1de09700 1 --2- 172.21.15.104:0/854952041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d181567f0 0x7f3d180cad90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3d180cacc0 tx=0x7f3d08009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfba4481bd05de1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1ce07700 1 -- 172.21.15.104:0/854952041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011830 con 0x7f3d181567f0 2023-06-10T18:44:18.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1ce07700 1 -- 172.21.15.104:0/854952041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08011990 con 0x7f3d181567f0 2023-06-10T18:44:18.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.571+0000 7f3d1ce07700 1 -- 172.21.15.104:0/854952041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011d20 con 0x7f3d181567f0 2023-06-10T18:44:18.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 -- 172.21.15.104:0/854952041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d181567f0 msgr2=0x7f3d180cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 --2- 172.21.15.104:0/854952041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d181567f0 0x7f3d180cad90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3d180cacc0 tx=0x7f3d08009b20 comp rx=0 tx=0).stop 2023-06-10T18:44:18.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 -- 172.21.15.104:0/854952041 shutdown_connections 2023-06-10T18:44:18.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 --2- 172.21.15.104:0/854952041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d181567f0 0x7f3d180cad90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 --2- 172.21.15.104:0/854952041 >> v2:172.21.15.114:3300/0 conn(0x7f3d1815c7a0 0x7f3d180cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 -- 172.21.15.104:0/854952041 >> 172.21.15.104:0/854952041 conn(0x7f3d1806e370 msgr2=0x7f3d1806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:18.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 -- 172.21.15.104:0/854952041 shutdown_connections 2023-06-10T18:44:18.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.572+0000 7f3d2006d700 1 -- 172.21.15.104:0/854952041 wait complete. 2023-06-10T18:44:18.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.573+0000 7f3d2006d700 1 Processor -- start 2023-06-10T18:44:18.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.573+0000 7f3d2006d700 1 -- start start 2023-06-10T18:44:18.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d180701a0 0x7f3d18146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d181567f0 0x7f3d180b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181470f0 con 0x7f3d1815c7a0 2023-06-10T18:44:18.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c7bf0 con 0x7f3d181567f0 2023-06-10T18:44:18.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c7ed0 con 0x7f3d180701a0 2023-06-10T18:44:18.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54414/0 (socket says 172.21.15.104:54414) 2023-06-10T18:44:18.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 -- 172.21.15.104:0/4010370580 learned_addr learned my addr 172.21.15.104:0/4010370580 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:18.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1de09700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d180701a0 0x7f3d18146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1d608700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d181567f0 0x7f3d180b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 -- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d180701a0 msgr2=0x7f3d18146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d180701a0 0x7f3d18146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 -- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d181567f0 msgr2=0x7f3d180b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d181567f0 0x7f3d180b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 -- 172.21.15.104:0/4010370580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080094a0 con 0x7f3d1815c7a0 2023-06-10T18:44:18.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d1e60a700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d10002230 tx=0x7f3d1000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000aac0 con 0x7f3d1815c7a0 2023-06-10T18:44:18.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1000ac40 con 0x7f3d1815c7a0 2023-06-10T18:44:18.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d180c81b0 con 0x7f3d1815c7a0 2023-06-10T18:44:18.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100070d0 con 0x7f3d1815c7a0 2023-06-10T18:44:18.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.574+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d180c8770 con 0x7f3d1815c7a0 2023-06-10T18:44:18.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.575+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d180cda20 con 0x7f3d1815c7a0 2023-06-10T18:44:18.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.576+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3d10007250 con 0x7f3d1815c7a0 2023-06-10T18:44:18.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.576+0000 7f3d0e7fc700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 0x7f3cfc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.576+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3d10089550 con 0x7f3d1815c7a0 2023-06-10T18:44:18.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.576+0000 7f3d1de09700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 0x7f3cfc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.577+0000 7f3d1de09700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 0x7f3cfc06d190 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d080090b0 tx=0x7f3d08008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.580+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d10054310 con 0x7f3d1815c7a0 2023-06-10T18:44:18.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.769+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d18074050 con 0x7f3d1815c7a0 2023-06-10T18:44:18.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.771+0000 7f3d0e7fc700 1 -- 172.21.15.104:0/4010370580 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f3d10057930 con 0x7f3d1815c7a0 2023-06-10T18:44:18.777 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:18.777 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 msgr2=0x7f3cfc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 0x7f3cfc06d190 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d080090b0 tx=0x7f3d08008030 comp rx=0 tx=0).stop 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 msgr2=0x7f3d1806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d10002230 tx=0x7f3d1000cf50 comp rx=0 tx=0).stop 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 shutdown_connections 2023-06-10T18:44:18.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1815c7a0 0x7f3d1806c130 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d180701a0 0x7f3d18146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cfc06ace0 0x7f3cfc06d190 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 --2- 172.21.15.104:0/4010370580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d181567f0 0x7f3d180b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 >> 172.21.15.104:0/4010370580 conn(0x7f3d1806e370 msgr2=0x7f3d1806b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 shutdown_connections 2023-06-10T18:44:18.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.773+0000 7f3d2006d700 1 -- 172.21.15.104:0/4010370580 wait complete. 2023-06-10T18:44:18.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:18.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.910+0000 7fd136717700 1 Processor -- start 2023-06-10T18:44:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.910+0000 7fd136717700 1 -- start start 2023-06-10T18:44:18.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd136717700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 0x7fd1300b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd136717700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd136717700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13006e250 con 0x7fd13006e6e0 2023-06-10T18:44:18.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd136717700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13015c950 con 0x7fd13015c580 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd136717700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300b1e80 con 0x7fd13006cde0 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd12ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 0x7fd1300b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45574/0 (socket says 172.21.15.104:45574) 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd12ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 0x7fd1300b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:41920/0 (socket says 172.21.15.104:41920) 2023-06-10T18:44:18.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd12f7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd13006e6e0 0x7fd1300b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58092/0 (socket says 172.21.15.104:58092) 2023-06-10T18:44:18.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 -- 172.21.15.104:0/2776102173 learned_addr learned my addr 172.21.15.104:0/2776102173 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:18.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 -- 172.21.15.104:0/2776102173 >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 msgr2=0x7fd1300b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:18.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 --2- 172.21.15.104:0/2776102173 >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 0x7fd1300b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 -- 172.21.15.104:0/2776102173 >> v1:172.21.15.114:6789/0 conn(0x7fd13006e6e0 legacy=0x7fd1300b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 -- 172.21.15.104:0/2776102173 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13006d920 con 0x7fd13006cde0 2023-06-10T18:44:18.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.911+0000 7fd134cb4700 1 --2- 172.21.15.104:0/2776102173 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd128009e50 tx=0x7fd12800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73fa399b91d0f8a8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd12effd700 1 -- 172.21.15.104:0/2776102173 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12800e070 con 0x7fd13006cde0 2023-06-10T18:44:18.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd12effd700 1 -- 172.21.15.104:0/2776102173 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12800b910 con 0x7fd13006cde0 2023-06-10T18:44:18.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd12effd700 1 -- 172.21.15.104:0/2776102173 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12800bcc0 con 0x7fd13006cde0 2023-06-10T18:44:18.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 -- 172.21.15.104:0/2776102173 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 msgr2=0x7fd1300ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 --2- 172.21.15.104:0/2776102173 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd128009e50 tx=0x7fd12800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:18.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 -- 172.21.15.104:0/2776102173 shutdown_connections 2023-06-10T18:44:18.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 --2- 172.21.15.104:0/2776102173 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd1300ce0d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 --2- 172.21.15.104:0/2776102173 >> v2:172.21.15.114:3300/0 conn(0x7fd13015c580 0x7fd1300b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 -- 172.21.15.104:0/2776102173 >> 172.21.15.104:0/2776102173 conn(0x7fd13006b8a0 msgr2=0x7fd1300cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:18.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 -- 172.21.15.104:0/2776102173 shutdown_connections 2023-06-10T18:44:18.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.912+0000 7fd136717700 1 -- 172.21.15.104:0/2776102173 wait complete. 2023-06-10T18:44:18.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.913+0000 7fd136717700 1 Processor -- start 2023-06-10T18:44:18.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.913+0000 7fd136717700 1 -- start start 2023-06-10T18:44:18.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd130158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13006e6e0 0x7fd1300cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd130158b60 con 0x7fd13015c580 2023-06-10T18:44:18.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300ce270 con 0x7fd13006e6e0 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd136717700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300ce550 con 0x7fd13006cde0 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd134cb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd134cb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54428/0 (socket says 172.21.15.104:54428) 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.914+0000 7fd134cb4700 1 -- 172.21.15.104:0/3311241629 learned_addr learned my addr 172.21.15.104:0/3311241629 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12f7fe700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13006e6e0 0x7fd1300cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12ffff700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd130158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 -- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 msgr2=0x7fd130158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd130158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 -- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13006e6e0 msgr2=0x7fd1300cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13006e6e0 0x7fd1300cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 -- 172.21.15.104:0/3311241629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd128009b30 con 0x7fd13015c580 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12ffff700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd130158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd134cb4700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b1fc0 tx=0x7fd128004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1280048b0 con 0x7fd13015c580 2023-06-10T18:44:18.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1280092e0 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1300ce830 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12801c680 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.915+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1300cedf0 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.916+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd130074050 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.916+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd12801c7e0 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.917+0000 7fd12cff9700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 0x7fd11c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.917+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd12808afe0 con 0x7fd13015c580 2023-06-10T18:44:18.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.917+0000 7fd12ffff700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 0x7fd11c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:18.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.917+0000 7fd12ffff700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 0x7fd11c06d190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b26d0 tx=0x7fd120008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:18.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:18.921+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd128055e30 con 0x7fd13015c580 2023-06-10T18:44:19.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.106+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1300b2f20 con 0x7fd13015c580 2023-06-10T18:44:19.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.107+0000 7fd12cff9700 1 -- 172.21.15.104:0/3311241629 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd128059450 con 0x7fd13015c580 2023-06-10T18:44:19.113 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:19.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 msgr2=0x7fd11c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 0x7fd11c06d190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b26d0 tx=0x7fd120008040 comp rx=0 tx=0).stop 2023-06-10T18:44:19.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 msgr2=0x7fd130147690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b1fc0 tx=0x7fd128004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 shutdown_connections 2023-06-10T18:44:19.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd13015c580 0x7fd130147690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13006cde0 0x7fd130158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd11c06ace0 0x7fd11c06d190 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 --2- 172.21.15.104:0/3311241629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13006e6e0 0x7fd1300cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 >> 172.21.15.104:0/3311241629 conn(0x7fd13006b8a0 msgr2=0x7fd1300cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 shutdown_connections 2023-06-10T18:44:19.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.109+0000 7fd136717700 1 -- 172.21.15.104:0/3311241629 wait complete. 2023-06-10T18:44:19.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:19.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.245+0000 7ff8959a0700 1 Processor -- start 2023-06-10T18:44:19.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.245+0000 7ff8959a0700 1 -- start start 2023-06-10T18:44:19.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.245+0000 7ff8959a0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff8900d04f0 0x7ff8900d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff8959a0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8900d25c0 0x7ff8900d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff8959a0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900d0e00 con 0x7ff89006b510 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff8959a0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8900af530 con 0x7ff8900d04f0 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff8959a0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8900af810 con 0x7ff8900d25c0 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff8900d04f0 0x7ff8900d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff8900d04f0 0x7ff8900d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54432/0 (socket says 172.21.15.104:54432) 2023-06-10T18:44:19.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 -- 172.21.15.104:0/2695678454 learned_addr learned my addr 172.21.15.104:0/2695678454 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88e7fc700 1 --2- 172.21.15.104:0/2695678454 >> v2:172.21.15.114:3300/0 conn(0x7ff8900d25c0 0x7ff8900d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 -- 172.21.15.104:0/2695678454 >> v2:172.21.15.114:3300/0 conn(0x7ff8900d25c0 msgr2=0x7ff8900d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 --2- 172.21.15.104:0/2695678454 >> v2:172.21.15.114:3300/0 conn(0x7ff8900d25c0 0x7ff8900d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 -- 172.21.15.104:0/2695678454 >> v1:172.21.15.114:6789/0 conn(0x7ff89006b510 legacy=0x7ff89006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 -- 172.21.15.104:0/2695678454 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900afaf0 con 0x7ff8900d04f0 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88effd700 1 --2- 172.21.15.104:0/2695678454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900d34d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8900b5490 tx=0x7ff880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0449ebc7113ce85 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88dffb700 1 -- 172.21.15.104:0/2695678454 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880005e30 con 0x7ff8900d04f0 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88dffb700 1 -- 172.21.15.104:0/2695678454 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff880004030 con 0x7ff8900d04f0 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.246+0000 7ff88dffb700 1 -- 172.21.15.104:0/2695678454 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800043e0 con 0x7ff8900d04f0 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 -- 172.21.15.104:0/2695678454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 msgr2=0x7ff8900d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 --2- 172.21.15.104:0/2695678454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900d34d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8900b5490 tx=0x7ff880009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 -- 172.21.15.104:0/2695678454 shutdown_connections 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 --2- 172.21.15.104:0/2695678454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900d34d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 --2- 172.21.15.104:0/2695678454 >> v2:172.21.15.114:3300/0 conn(0x7ff8900d25c0 0x7ff8900d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 -- 172.21.15.104:0/2695678454 >> 172.21.15.104:0/2695678454 conn(0x7ff890071ec0 msgr2=0x7ff89009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 -- 172.21.15.104:0/2695678454 shutdown_connections 2023-06-10T18:44:19.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.247+0000 7ff8959a0700 1 -- 172.21.15.104:0/2695678454 wait complete. 2023-06-10T18:44:19.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.248+0000 7ff8959a0700 1 Processor -- start 2023-06-10T18:44:19.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.248+0000 7ff8959a0700 1 -- start start 2023-06-10T18:44:19.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 0x7ff89009eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 0x7ff8900cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890157e50 con 0x7ff8900d04f0 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901f2790 con 0x7ff89006b510 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff8959a0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901f2a70 con 0x7ff8900d25c0 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54438/0 (socket says 172.21.15.104:54438) 2023-06-10T18:44:19.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 -- 172.21.15.104:0/3225655934 learned_addr learned my addr 172.21.15.104:0/3225655934 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88effd700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 0x7ff89009eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88f7fe700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 0x7ff8900cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 -- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 msgr2=0x7ff8900cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 0x7ff8900cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 -- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 msgr2=0x7ff89009eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 0x7ff89009eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 -- 172.21.15.104:0/3225655934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009500 con 0x7ff8900d04f0 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88effd700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 0x7ff89009eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88f7fe700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 0x7ff8900cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.249+0000 7ff88e7fc700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e9b0 tx=0x7ff88400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.250+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884009710 con 0x7ff8900d04f0 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.250+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88400c850 con 0x7ff8900d04f0 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.250+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901f2d50 con 0x7ff8900d04f0 2023-06-10T18:44:19.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.250+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88400cbc0 con 0x7ff8900d04f0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.250+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89009f530 con 0x7ff8900d04f0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.251+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890067010 con 0x7ff8900d04f0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.251+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff88400cd20 con 0x7ff8900d04f0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.251+0000 7ff877fff700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 0x7ff87c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.252+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff8840898e0 con 0x7ff8900d04f0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.252+0000 7ff88effd700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 0x7ff87c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.252+0000 7ff88effd700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 0x7ff87c06d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff880014040 tx=0x7ff880008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.256+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff884058130 con 0x7ff8900d04f0 2023-06-10T18:44:19.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.442+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff8900d2120 con 0x7ff8900d04f0 2023-06-10T18:44:19.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.442+0000 7ff877fff700 1 -- 172.21.15.104:0/3225655934 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff884057cc0 con 0x7ff8900d04f0 2023-06-10T18:44:19.448 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 msgr2=0x7ff87c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 0x7ff87c06d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff880014040 tx=0x7ff880008070 comp rx=0 tx=0).stop 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 msgr2=0x7ff8900cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e9b0 tx=0x7ff88400ed70 comp rx=0 tx=0).stop 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 shutdown_connections 2023-06-10T18:44:19.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff87c06ace0 0x7ff87c06d190 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff89006b510 0x7ff89009eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8900d25c0 0x7ff8900cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 --2- 172.21.15.104:0/3225655934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8900d04f0 0x7ff8900cb2d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 >> 172.21.15.104:0/3225655934 conn(0x7ff890071ec0 msgr2=0x7ff8900b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 shutdown_connections 2023-06-10T18:44:19.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.444+0000 7ff8959a0700 1 -- 172.21.15.104:0/3225655934 wait complete. 2023-06-10T18:44:19.461 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:19.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.581+0000 7f094ef5c700 1 Processor -- start 2023-06-10T18:44:19.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.581+0000 7f094ef5c700 1 -- start start 2023-06-10T18:44:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ef5c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f094806ea70 0x7f094806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ef5c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09480ae7e0 con 0x7f094815c6f0 2023-06-10T18:44:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ef5c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f094806df90 con 0x7f09480b1780 2023-06-10T18:44:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ef5c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094806e270 con 0x7f094806ea70 2023-06-10T18:44:19.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f094806ea70 0x7f094806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ccf8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f094815c6f0 0x7f09480b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52272/0 (socket says 172.21.15.104:52272) 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094ccf8700 1 -- 172.21.15.104:0/2102213391 learned_addr learned my addr 172.21.15.104:0/2102213391 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 --2- 172.21.15.104:0/2102213391 >> v2:172.21.15.104:3300/0 conn(0x7f094806ea70 0x7f094806d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54448/0 (socket says 172.21.15.104:54448) 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 -- 172.21.15.104:0/2102213391 >> v1:172.21.15.114:6789/0 conn(0x7f09480b1780 legacy=0x7f094806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 -- 172.21.15.104:0/2102213391 >> v1:172.21.15.104:6789/0 conn(0x7f094815c6f0 legacy=0x7f09480b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 -- 172.21.15.104:0/2102213391 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094806e550 con 0x7f094806ea70 2023-06-10T18:44:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f094d4f9700 1 --2- 172.21.15.104:0/2102213391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094806ea70 0x7f094806d4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d90 tx=0x7f094000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d01b592e2a3ded server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f09477fe700 1 -- 172.21.15.104:0/2102213391 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000e070 con 0x7f094806ea70 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f09477fe700 1 -- 172.21.15.104:0/2102213391 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094000bb90 con 0x7f094806ea70 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.582+0000 7f09477fe700 1 -- 172.21.15.104:0/2102213391 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940003a20 con 0x7f094806ea70 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 -- 172.21.15.104:0/2102213391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094806ea70 msgr2=0x7f094806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2102213391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094806ea70 0x7f094806d4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0940009d90 tx=0x7f094000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 -- 172.21.15.104:0/2102213391 shutdown_connections 2023-06-10T18:44:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2102213391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f094806ea70 0x7f094806d4f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 -- 172.21.15.104:0/2102213391 >> 172.21.15.104:0/2102213391 conn(0x7f094806b8a0 msgr2=0x7f09480cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 -- 172.21.15.104:0/2102213391 shutdown_connections 2023-06-10T18:44:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.583+0000 7f094ef5c700 1 -- 172.21.15.104:0/2102213391 wait complete. 2023-06-10T18:44:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.584+0000 7f094ef5c700 1 Processor -- start 2023-06-10T18:44:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.584+0000 7f094ef5c700 1 -- start start 2023-06-10T18:44:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f094806ea70 0x7f0948148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 0x7f094809c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094809a1f0 con 0x7f09480b1780 2023-06-10T18:44:19.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0948099340 con 0x7f094806ea70 2023-06-10T18:44:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ef5c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480995e0 con 0x7f094815c6f0 2023-06-10T18:44:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54450/0 (socket says 172.21.15.104:54450) 2023-06-10T18:44:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 -- 172.21.15.104:0/2829545556 learned_addr learned my addr 172.21.15.104:0/2829545556 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094ccf8700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f094806ea70 0x7f0948148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094d4f9700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 0x7f094809c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 -- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 msgr2=0x7f094809c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 0x7f094809c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 -- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f094806ea70 msgr2=0x7f0948148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f094806ea70 0x7f0948148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f0947fff700 1 -- 172.21.15.104:0/2829545556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940009a70 con 0x7f09480b1780 2023-06-10T18:44:19.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.585+0000 7f094d4f9700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 0x7f094809c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f0947fff700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f093400c4a0 tx=0x7f093400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934011790 con 0x7f09480b1780 2023-06-10T18:44:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09340118f0 con 0x7f09480b1780 2023-06-10T18:44:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934011ca0 con 0x7f09480b1780 2023-06-10T18:44:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09480998c0 con 0x7f09480b1780 2023-06-10T18:44:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.586+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09480c5970 con 0x7f09480b1780 2023-06-10T18:44:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.587+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09480aebb0 con 0x7f09480b1780 2023-06-10T18:44:19.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.587+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f093401c460 con 0x7f09480b1780 2023-06-10T18:44:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.587+0000 7f09457fa700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 0x7f092c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.588+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f09340895a0 con 0x7f09480b1780 2023-06-10T18:44:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.588+0000 7f094ccf8700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 0x7f092c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.588+0000 7f094ccf8700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 0x7f092c06d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f093c0010e0 tx=0x7f093c007d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.592+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f09340542d0 con 0x7f09480b1780 2023-06-10T18:44:19.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.777+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f09480c5b00 con 0x7f09480b1780 2023-06-10T18:44:19.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.778+0000 7f09457fa700 1 -- 172.21.15.104:0/2829545556 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f09340578f0 con 0x7f09480b1780 2023-06-10T18:44:19.784 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:19.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 msgr2=0x7f092c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 0x7f092c06d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f093c0010e0 tx=0x7f093c007d90 comp rx=0 tx=0).stop 2023-06-10T18:44:19.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 msgr2=0x7f09480c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f093400c4a0 tx=0x7f093400c7b0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 shutdown_connections 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f092c06ace0 0x7f092c06d190 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f094806ea70 0x7f0948148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f094815c6f0 0x7f094809c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 --2- 172.21.15.104:0/2829545556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09480b1780 0x7f09480c5430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 >> 172.21.15.104:0/2829545556 conn(0x7f094806b8a0 msgr2=0x7f094815b550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 shutdown_connections 2023-06-10T18:44:19.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.780+0000 7f094ef5c700 1 -- 172.21.15.104:0/2829545556 wait complete. 2023-06-10T18:44:19.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:19.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.914+0000 7f89fb3fb700 1 Processor -- start 2023-06-10T18:44:19.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.914+0000 7f89fb3fb700 1 -- start start 2023-06-10T18:44:19.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89fb3fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f89f406cde0 0x7f89f40b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89fb3fb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f406e250 con 0x7f89f415c580 2023-06-10T18:44:19.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89fb3fb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f415c950 con 0x7f89f406e6e0 2023-06-10T18:44:19.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89fb3fb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f40b1e80 con 0x7f89f406cde0 2023-06-10T18:44:19.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f89f406cde0 0x7f89f40b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9197700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f89f415c580 0x7f89f40b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52286/0 (socket says 172.21.15.104:52286) 2023-06-10T18:44:19.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f89f406cde0 0x7f89f40b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54462/0 (socket says 172.21.15.104:54462) 2023-06-10T18:44:19.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9197700 1 -- 172.21.15.104:0/49891551 learned_addr learned my addr 172.21.15.104:0/49891551 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:19.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 -- 172.21.15.104:0/49891551 >> v1:172.21.15.114:6789/0 conn(0x7f89f406e6e0 legacy=0x7f89f40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 -- 172.21.15.104:0/49891551 >> v1:172.21.15.104:6789/0 conn(0x7f89f415c580 legacy=0x7f89f40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89ebfff700 1 -- 172.21.15.104:0/49891551 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169151578 0 0) 0x7f89f406e250 con 0x7f89f415c580 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 -- 172.21.15.104:0/49891551 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f406d920 con 0x7f89f406cde0 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89f9998700 1 --2- 172.21.15.104:0/49891551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406cde0 0x7f89f40b1780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009d90 tx=0x7f89ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe46b598af61a030 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.915+0000 7f89ebfff700 1 -- 172.21.15.104:0/49891551 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00e070 con 0x7f89f406cde0 2023-06-10T18:44:19.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89ebfff700 1 -- 172.21.15.104:0/49891551 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec00bbd0 con 0x7f89f406cde0 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89ebfff700 1 -- 172.21.15.104:0/49891551 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec012430 con 0x7f89f406cde0 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 -- 172.21.15.104:0/49891551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406cde0 msgr2=0x7f89f40b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/49891551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406cde0 0x7f89f40b1780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009d90 tx=0x7f89ec00b330 comp rx=0 tx=0).stop 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 -- 172.21.15.104:0/49891551 shutdown_connections 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/49891551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406cde0 0x7f89f40b1780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 -- 172.21.15.104:0/49891551 >> 172.21.15.104:0/49891551 conn(0x7f89f406b8a0 msgr2=0x7f89f40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:19.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 -- 172.21.15.104:0/49891551 shutdown_connections 2023-06-10T18:44:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.916+0000 7f89fb3fb700 1 -- 172.21.15.104:0/49891551 wait complete. 2023-06-10T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.917+0000 7f89fb3fb700 1 Processor -- start 2023-06-10T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.917+0000 7f89fb3fb700 1 -- start start 2023-06-10T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 0x7f89f40cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 0x7f89f41f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40af590 con 0x7f89f406e6e0 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f41f1c20 con 0x7f89f406cde0 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89fb3fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40af7c0 con 0x7f89f415c580 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54464/0 (socket says 172.21.15.104:54464) 2023-06-10T18:44:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 -- 172.21.15.104:0/1288221596 learned_addr learned my addr 172.21.15.104:0/1288221596 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f9197700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 0x7f89f40cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f9998700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 0x7f89f41f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 -- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 msgr2=0x7f89f41f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 0x7f89f41f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 -- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 msgr2=0x7f89f40cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 0x7f89f40cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f8996700 1 -- 172.21.15.104:0/1288221596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec009a70 con 0x7f89f406e6e0 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f9197700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 0x7f89f40cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.918+0000 7f89f9998700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 0x7f89f41f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89f8996700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89e400c4b0 tx=0x7f89e400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40052d0 con 0x7f89f406e6e0 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4005430 con 0x7f89f406e6e0 2023-06-10T18:44:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f40afaa0 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40038c0 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.919+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f40b0060 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.920+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f4074050 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.920+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f89e4003a20 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.920+0000 7f89e9ffb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 0x7f89d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.921+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f89e4088c20 con 0x7f89f406e6e0 2023-06-10T18:44:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.921+0000 7f89f9197700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 0x7f89d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.921+0000 7f89f9197700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 0x7f89d806d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:19.925+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f89e40539e0 con 0x7f89f406e6e0 2023-06-10T18:44:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.111+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89f40b5450 con 0x7f89f406e6e0 2023-06-10T18:44:20.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.112+0000 7f89e9ffb700 1 -- 172.21.15.104:0/1288221596 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f89e4017020 con 0x7f89f406e6e0 2023-06-10T18:44:20.118 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 msgr2=0x7f89d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 0x7f89d806d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0009040 comp rx=0 tx=0).stop 2023-06-10T18:44:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 msgr2=0x7f89f40af050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f89e400c4b0 tx=0x7f89e400c870 comp rx=0 tx=0).stop 2023-06-10T18:44:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 shutdown_connections 2023-06-10T18:44:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89d806ace0 0x7f89d806d190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89f406cde0 0x7f89f40cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89f415c580 0x7f89f41f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 --2- 172.21.15.104:0/1288221596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89f406e6e0 0x7f89f40af050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 >> 172.21.15.104:0/1288221596 conn(0x7f89f406b8a0 msgr2=0x7f89f40cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 shutdown_connections 2023-06-10T18:44:20.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.114+0000 7f89fb3fb700 1 -- 172.21.15.104:0/1288221596 wait complete. 2023-06-10T18:44:20.131 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:20.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.247+0000 7fb51fef6700 1 Processor -- start 2023-06-10T18:44:20.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.248+0000 7fb51fef6700 1 -- start start 2023-06-10T18:44:20.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.248+0000 7fb51fef6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb51806ea70 0x7fb51806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.248+0000 7fb51fef6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5180ae7e0 con 0x7fb5180b1780 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.248+0000 7fb51fef6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51806df90 con 0x7fb51815c6f0 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.248+0000 7fb51fef6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51806e270 con 0x7fb51806ea70 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51d491700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb5180b1780 0x7fb51806cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52300/0 (socket says 172.21.15.104:52300) 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51e493700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb51806ea70 0x7fb51806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51d491700 1 -- 172.21.15.104:0/730619902 learned_addr learned my addr 172.21.15.104:0/730619902 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51cc90700 1 -- 172.21.15.104:0/730619902 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434489999 0 0) 0x7fb5180ae7e0 con 0x7fb5180b1780 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51cc90700 1 -- 172.21.15.104:0/730619902 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5040035b0 con 0x7fb5180b1780 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51e493700 1 -- 172.21.15.104:0/730619902 >> v1:172.21.15.114:6789/0 conn(0x7fb51815c6f0 legacy=0x7fb5180b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51e493700 1 -- 172.21.15.104:0/730619902 >> v1:172.21.15.104:6789/0 conn(0x7fb5180b1780 legacy=0x7fb51806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51e493700 1 -- 172.21.15.104:0/730619902 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51806e550 con 0x7fb51806ea70 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51e493700 1 --2- 172.21.15.104:0/730619902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51806ea70 0x7fb51806d4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb51000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff163ccf9d8fdbeb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51cc90700 1 -- 172.21.15.104:0/730619902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51000e070 con 0x7fb51806ea70 2023-06-10T18:44:20.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51cc90700 1 -- 172.21.15.104:0/730619902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51000bb20 con 0x7fb51806ea70 2023-06-10T18:44:20.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.249+0000 7fb51cc90700 1 -- 172.21.15.104:0/730619902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510012430 con 0x7fb51806ea70 2023-06-10T18:44:20.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 -- 172.21.15.104:0/730619902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51806ea70 msgr2=0x7fb51806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 --2- 172.21.15.104:0/730619902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51806ea70 0x7fb51806d4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb510009d90 tx=0x7fb51000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:20.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 -- 172.21.15.104:0/730619902 shutdown_connections 2023-06-10T18:44:20.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 --2- 172.21.15.104:0/730619902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51806ea70 0x7fb51806d4f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 -- 172.21.15.104:0/730619902 >> 172.21.15.104:0/730619902 conn(0x7fb51806b8a0 msgr2=0x7fb5180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 -- 172.21.15.104:0/730619902 shutdown_connections 2023-06-10T18:44:20.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.250+0000 7fb51fef6700 1 -- 172.21.15.104:0/730619902 wait complete. 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.251+0000 7fb51fef6700 1 Processor -- start 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.251+0000 7fb51fef6700 1 -- start start 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 0x7fb518150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51815c6f0 0x7fb518153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb518158710 con 0x7fb5180b1780 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5180b35b0 con 0x7fb51806ea70 2023-06-10T18:44:20.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51fef6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb518150960 con 0x7fb51815c6f0 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54480/0 (socket says 172.21.15.104:54480) 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 -- 172.21.15.104:0/4169560615 learned_addr learned my addr 172.21.15.104:0/4169560615 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51e493700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51815c6f0 0x7fb518153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51dc92700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 0x7fb518150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 -- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51815c6f0 msgr2=0x7fb518153af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51815c6f0 0x7fb518153af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 -- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 msgr2=0x7fb518150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 0x7fb518150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 -- 172.21.15.104:0/4169560615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb510009a70 con 0x7fb5180b1780 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51dc92700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 0x7fb518150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb51d491700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c700 tx=0x7fb51400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.252+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514005d70 con 0x7fb5180b1780 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.253+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5140070d0 con 0x7fb5180b1780 2023-06-10T18:44:20.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.253+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514007460 con 0x7fb5180b1780 2023-06-10T18:44:20.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.253+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb518157e70 con 0x7fb5180b1780 2023-06-10T18:44:20.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.253+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb518158430 con 0x7fb5180b1780 2023-06-10T18:44:20.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.253+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5180aebc0 con 0x7fb5180b1780 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.254+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb5140075c0 con 0x7fb5180b1780 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.254+0000 7fb50e7fc700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 0x7fb4fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.254+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb514088cc0 con 0x7fb5180b1780 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.255+0000 7fb51dc92700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 0x7fb4fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.255+0000 7fb51dc92700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 0x7fb4fc06d190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb508000fb0 tx=0x7fb508009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.259+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb51408f080 con 0x7fb5180b1780 2023-06-10T18:44:20.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.442+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb5180b3720 con 0x7fb5180b1780 2023-06-10T18:44:20.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.442+0000 7fb50e7fc700 1 -- 172.21.15.104:0/4169560615 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb514053a70 con 0x7fb5180b1780 2023-06-10T18:44:20.449 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 msgr2=0x7fb4fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 0x7fb4fc06d190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb508000fb0 tx=0x7fb508009040 comp rx=0 tx=0).stop 2023-06-10T18:44:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 msgr2=0x7fb5180c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c700 tx=0x7fb51400ca10 comp rx=0 tx=0).stop 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 shutdown_connections 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4fc06ace0 0x7fb4fc06d190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb51806ea70 0x7fb518150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51815c6f0 0x7fb518153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 --2- 172.21.15.104:0/4169560615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5180b1780 0x7fb5180c5de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 >> 172.21.15.104:0/4169560615 conn(0x7fb51806b8a0 msgr2=0x7fb5181592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 shutdown_connections 2023-06-10T18:44:20.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.444+0000 7fb51fef6700 1 -- 172.21.15.104:0/4169560615 wait complete. 2023-06-10T18:44:20.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:44:20.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 Processor -- start 2023-06-10T18:44:20.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 -- start start 2023-06-10T18:44:20.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90dc0b1080 0x7f90dc0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc0c9e60 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc0c9bc0 con 0x7f90dc0b1080 2023-06-10T18:44:20.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.578+0000 7f90e24c2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0aee70 con 0x7f90dc0aeaa0 2023-06-10T18:44:20.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90e0a5f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f90dc0ca2f0 0x7f90dc0ca750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52316/0 (socket says 172.21.15.104:52316) 2023-06-10T18:44:20.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90e0a5f700 1 -- 172.21.15.104:0/1154110973 learned_addr learned my addr 172.21.15.104:0/1154110973 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:20.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 --2- 172.21.15.104:0/1154110973 >> v2:172.21.15.114:3300/0 conn(0x7f90dc0b1080 0x7f90dc0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90dbfff700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90daffd700 1 -- 172.21.15.104:0/1154110973 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823529597 0 0) 0x7f90dc0c9e60 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90daffd700 1 -- 172.21.15.104:0/1154110973 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c40035b0 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 -- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 msgr2=0x7f90dc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 -- 172.21.15.104:0/1154110973 >> v1:172.21.15.104:6789/0 conn(0x7f90dc0ca2f0 legacy=0x7f90dc0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 -- 172.21.15.104:0/1154110973 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc06e540 con 0x7f90dc0b1080 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90dbfff700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:20.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90db7fe700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc0cd680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009840 tx=0x7f90d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de960ec2a0f1822d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90daffd700 1 -- 172.21.15.104:0/1154110973 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0011830 con 0x7f90dc0b1080 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90daffd700 1 -- 172.21.15.104:0/1154110973 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0011990 con 0x7f90dc0b1080 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.579+0000 7f90daffd700 1 -- 172.21.15.104:0/1154110973 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0011d20 con 0x7f90dc0b1080 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 -- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 msgr2=0x7f90dc0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc0cd680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009840 tx=0x7f90d0009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:20.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 -- 172.21.15.104:0/1154110973 shutdown_connections 2023-06-10T18:44:20.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 --2- 172.21.15.104:0/1154110973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc0cd680 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 -- 172.21.15.104:0/1154110973 >> 172.21.15.104:0/1154110973 conn(0x7f90dc06b8a0 msgr2=0x7f90dc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 -- 172.21.15.104:0/1154110973 shutdown_connections 2023-06-10T18:44:20.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.580+0000 7f90e24c2700 1 -- 172.21.15.104:0/1154110973 wait complete. 2023-06-10T18:44:20.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.581+0000 7f90e24c2700 1 Processor -- start 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.581+0000 7f90e24c2700 1 -- start start 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc1f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc14c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc14c340 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc14b6c0 con 0x7f90dc0b1080 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e24c2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc14b9a0 con 0x7f90dc0aeaa0 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54496/0 (socket says 172.21.15.104:54496) 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 -- 172.21.15.104:0/2199052855 learned_addr learned my addr 172.21.15.104:0/2199052855 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 -- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 msgr2=0x7f90dc1f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90dbfff700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc1f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90db7fe700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc14c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc1f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 -- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 msgr2=0x7f90dc14c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc14c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.582+0000 7f90e0a5f700 1 -- 172.21.15.104:0/2199052855 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc009430 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90e0a5f700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90d400caa0 tx=0x7f90d400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d400aa80 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d400abe0 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00094a0 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4012400 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.583+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc14bf60 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.584+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc14c480 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.584+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90d401b3b0 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.585+0000 7f90d8ff9700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 0x7f90c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.585+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f90d40894b0 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.585+0000 7f90dbfff700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 0x7f90c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.585+0000 7f90dbfff700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 0x7f90c806d190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009c00 tx=0x7f90cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.589+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90d4054300 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.789+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90dc074050 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.789+0000 7f90d8ff9700 1 -- 172.21.15.104:0/2199052855 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90d4057920 con 0x7f90dc0ca2f0 2023-06-10T18:44:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 msgr2=0x7f90c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 0x7f90c806d190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009c00 tx=0x7f90cc008040 comp rx=0 tx=0).stop 2023-06-10T18:44:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 msgr2=0x7f90dc14cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90d400caa0 tx=0x7f90d400ce60 comp rx=0 tx=0).stop 2023-06-10T18:44:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 shutdown_connections 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90dc0ca2f0 0x7f90dc14cd30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90dc0aeaa0 0x7f90dc1f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c806ace0 0x7f90c806d190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 --2- 172.21.15.104:0/2199052855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90dc0b1080 0x7f90dc14c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 >> 172.21.15.104:0/2199052855 conn(0x7f90dc06b8a0 msgr2=0x7f90dc0cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 shutdown_connections 2023-06-10T18:44:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.791+0000 7f90e24c2700 1 -- 172.21.15.104:0/2199052855 wait complete. 2023-06-10T18:44:20.801 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:44:20.808 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:44:20.809 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:44:20.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:20.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.981+0000 7f9f77a65700 1 Processor -- start 2023-06-10T18:44:20.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.982+0000 7f9f77a65700 1 -- start start 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f77a65700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f700cd850 0x7f9f7006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f77a65700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700c9e60 con 0x7f9f700cb300 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f77a65700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700aee70 con 0x7f9f700aeaa0 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f77a65700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700c9bc0 con 0x7f9f700cd850 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f700cd850 0x7f9f7006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f700cd850 0x7f9f7006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54502/0 (socket says 172.21.15.104:54502) 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 -- 172.21.15.104:0/4156399352 learned_addr learned my addr 172.21.15.104:0/4156399352 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 -- 172.21.15.104:0/4156399352 >> v1:172.21.15.114:6789/0 conn(0x7f9f700aeaa0 legacy=0x7f9f700cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 -- 172.21.15.104:0/4156399352 >> v1:172.21.15.104:6789/0 conn(0x7f9f700cb300 legacy=0x7f9f7006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 -- 172.21.15.104:0/4156399352 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7006e540 con 0x7f9f700cd850 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f75000700 1 --2- 172.21.15.104:0/4156399352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cd850 0x7f9f7006cd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098f0 tx=0x7f9f60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374ab881380455f3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f67fff700 1 -- 172.21.15.104:0/4156399352 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000b8a0 con 0x7f9f700cd850 2023-06-10T18:44:20.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f67fff700 1 -- 172.21.15.104:0/4156399352 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6000ba00 con 0x7f9f700cd850 2023-06-10T18:44:20.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.983+0000 7f9f67fff700 1 -- 172.21.15.104:0/4156399352 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000bd90 con 0x7f9f700cd850 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 -- 172.21.15.104:0/4156399352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cd850 msgr2=0x7f9f7006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 --2- 172.21.15.104:0/4156399352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cd850 0x7f9f7006cd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098f0 tx=0x7f9f60009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 -- 172.21.15.104:0/4156399352 shutdown_connections 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 --2- 172.21.15.104:0/4156399352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cd850 0x7f9f7006cd50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 -- 172.21.15.104:0/4156399352 >> 172.21.15.104:0/4156399352 conn(0x7f9f7006b8a0 msgr2=0x7f9f700cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:20.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 -- 172.21.15.104:0/4156399352 shutdown_connections 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.984+0000 7f9f77a65700 1 -- 172.21.15.104:0/4156399352 wait complete. 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.985+0000 7f9f77a65700 1 Processor -- start 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.985+0000 7f9f77a65700 1 -- start start 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700aeaa0 0x7f9f700adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f700cd850 0x7f9f701e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700ad410 con 0x7f9f700cb300 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700c9410 con 0x7f9f700aeaa0 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f77a65700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700ac800 con 0x7f9f700cd850 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f75000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f75000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54508/0 (socket says 172.21.15.104:54508) 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f75000700 1 -- 172.21.15.104:0/1061597259 learned_addr learned my addr 172.21.15.104:0/1061597259 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f75801700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700aeaa0 0x7f9f700adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.986+0000 7f9f76002700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f700cd850 0x7f9f701e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 -- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f700cd850 msgr2=0x7f9f701e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f700cd850 0x7f9f701e5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 -- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700aeaa0 msgr2=0x7f9f700adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700aeaa0 0x7f9f700adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 -- 172.21.15.104:0/1061597259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60009550 con 0x7f9f700cb300 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f75000700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009130 tx=0x7f9f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f600036a0 con 0x7f9f700cb300 2023-06-10T18:44:20.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6001e070 con 0x7f9f700cb300 2023-06-10T18:44:20.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60003a60 con 0x7f9f700cb300 2023-06-10T18:44:20.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f700acae0 con 0x7f9f700cb300 2023-06-10T18:44:20.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.987+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f700ad0a0 con 0x7f9f700cb300 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.988+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f700ad5c0 con 0x7f9f700cb300 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.988+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9f60003bc0 con 0x7f9f700cb300 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.989+0000 7f9f65ffb700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 0x7f9f5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.989+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9f60020070 con 0x7f9f700cb300 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.989+0000 7f9f75801700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 0x7f9f5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:20.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.990+0000 7f9f75801700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 0x7f9f5406d190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000fb0 tx=0x7f9f6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:20.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:20.993+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9f700ad5c0 con 0x7f9f700cb300 2023-06-10T18:44:21.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.185+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f70074050 con 0x7f9f700cb300 2023-06-10T18:44:21.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.186+0000 7f9f65ffb700 1 -- 172.21.15.104:0/1061597259 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f9f60015090 con 0x7f9f700cb300 2023-06-10T18:44:21.192 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:21.192 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 msgr2=0x7f9f5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 0x7f9f5406d190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000fb0 tx=0x7f9f6c009040 comp rx=0 tx=0).stop 2023-06-10T18:44:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 msgr2=0x7f9f700c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009130 tx=0x7f9f60004750 comp rx=0 tx=0).stop 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 shutdown_connections 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f5406ace0 0x7f9f5406d190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700aeaa0 0x7f9f700adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f700cd850 0x7f9f701e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 --2- 172.21.15.104:0/1061597259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f700cb300 0x7f9f700c7590 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 >> 172.21.15.104:0/1061597259 conn(0x7f9f7006b8a0 msgr2=0x7f9f700af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 shutdown_connections 2023-06-10T18:44:21.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.188+0000 7f9f77a65700 1 -- 172.21.15.104:0/1061597259 wait complete. 2023-06-10T18:44:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:21.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:21.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.323+0000 7f0a89079700 1 Processor -- start 2023-06-10T18:44:21.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.323+0000 7f0a89079700 1 -- start start 2023-06-10T18:44:21.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a89079700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0a84070a10 0x7f0a84070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a89079700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a8406aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a89079700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a840c4770 con 0x7f0a84145cc0 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a89079700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a840713d0 con 0x7f0a84070a10 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a89079700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84069e70 con 0x7f0a8406a5e0 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0a84070a10 0x7f0a84070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a82d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0a84145cc0 0x7f0a8406c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52342/0 (socket says 172.21.15.104:52342) 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a82d9d700 1 -- 172.21.15.104:0/1730368113 learned_addr learned my addr 172.21.15.104:0/1730368113 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 --2- 172.21.15.104:0/1730368113 >> v2:172.21.15.104:3300/0 conn(0x7f0a84070a10 0x7f0a84070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54516/0 (socket says 172.21.15.104:54516) 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 -- 172.21.15.104:0/1730368113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 msgr2=0x7f0a8406aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8359e700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a8406aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a81d9b700 1 -- 172.21.15.104:0/1730368113 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549885211 0 0) 0x7f0a840c4770 con 0x7f0a84145cc0 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a8406aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 -- 172.21.15.104:0/1730368113 >> v1:172.21.15.104:6789/0 conn(0x7f0a84145cc0 legacy=0x7f0a8406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 -- 172.21.15.104:0/1730368113 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8406cf10 con 0x7f0a84070a10 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.324+0000 7f0a8259c700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84070a10 0x7f0a84070e90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009840 tx=0x7f0a74009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be35c9bf81cc2027 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a81d9b700 1 -- 172.21.15.104:0/1730368113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74011830 con 0x7f0a84070a10 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a81d9b700 1 -- 172.21.15.104:0/1730368113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74011990 con 0x7f0a84070a10 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a81d9b700 1 -- 172.21.15.104:0/1730368113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74011d20 con 0x7f0a84070a10 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 -- 172.21.15.104:0/1730368113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84070a10 msgr2=0x7f0a84070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84070a10 0x7f0a84070e90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009840 tx=0x7f0a74009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:21.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 -- 172.21.15.104:0/1730368113 shutdown_connections 2023-06-10T18:44:21.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a8406aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 --2- 172.21.15.104:0/1730368113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84070a10 0x7f0a84070e90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 -- 172.21.15.104:0/1730368113 >> 172.21.15.104:0/1730368113 conn(0x7f0a840ceec0 msgr2=0x7f0a840cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:21.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 -- 172.21.15.104:0/1730368113 shutdown_connections 2023-06-10T18:44:21.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.325+0000 7f0a89079700 1 -- 172.21.15.104:0/1730368113 wait complete. 2023-06-10T18:44:21.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 Processor -- start 2023-06-10T18:44:21.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 -- start start 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a840c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a84070a10 0x7f0a84144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8415e260 con 0x7f0a84145cc0 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8415d650 con 0x7f0a84070a10 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.327+0000 7f0a89079700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8415d930 con 0x7f0a8406a5e0 2023-06-10T18:44:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54522/0 (socket says 172.21.15.104:54522) 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 -- 172.21.15.104:0/2636504598 learned_addr learned my addr 172.21.15.104:0/2636504598 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a82d9d700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a840c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8259c700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a84070a10 0x7f0a84144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 -- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 msgr2=0x7f0a840c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a840c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 -- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a84070a10 msgr2=0x7f0a84144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a84070a10 0x7f0a84144430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 -- 172.21.15.104:0/2636504598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a740094a0 con 0x7f0a84145cc0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a8359e700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00cd70 tx=0x7f0a7c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c003a20 con 0x7f0a84145cc0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7c003b80 con 0x7f0a84145cc0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8415dc10 con 0x7f0a84145cc0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c004020 con 0x7f0a84145cc0 2023-06-10T18:44:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.328+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a840c65b0 con 0x7f0a84145cc0 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.329+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8406d6d0 con 0x7f0a84145cc0 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.330+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0a7c004180 con 0x7f0a84145cc0 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.330+0000 7f0a6b7fe700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 0x7f0a7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.330+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0a7c012070 con 0x7f0a84145cc0 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.330+0000 7f0a82d9d700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 0x7f0a7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.331+0000 7f0a82d9d700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 0x7f0a7006d190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000f60 tx=0x7f0a78007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.334+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0a7c054c90 con 0x7f0a84145cc0 2023-06-10T18:44:21.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.523+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a84074050 con 0x7f0a84145cc0 2023-06-10T18:44:21.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.523+0000 7f0a6b7fe700 1 -- 172.21.15.104:0/2636504598 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a7c0582b0 con 0x7f0a84145cc0 2023-06-10T18:44:21.529 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:21.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 msgr2=0x7f0a7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 0x7f0a7006d190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000f60 tx=0x7f0a78007e70 comp rx=0 tx=0).stop 2023-06-10T18:44:21.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 msgr2=0x7f0a84144970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00cd70 tx=0x7f0a7c00a570 comp rx=0 tx=0).stop 2023-06-10T18:44:21.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 shutdown_connections 2023-06-10T18:44:21.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a84145cc0 0x7f0a84144970 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a8406a5e0 0x7f0a840c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a7006ace0 0x7f0a7006d190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 --2- 172.21.15.104:0/2636504598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a84070a10 0x7f0a84144430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 >> 172.21.15.104:0/2636504598 conn(0x7f0a840ceec0 msgr2=0x7f0a840cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:21.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 shutdown_connections 2023-06-10T18:44:21.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.525+0000 7f0a89079700 1 -- 172.21.15.104:0/2636504598 wait complete. 2023-06-10T18:44:21.542 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:21.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.659+0000 7fadbdedc700 1 Processor -- start 2023-06-10T18:44:21.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.659+0000 7fadbdedc700 1 -- start start 2023-06-10T18:44:21.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadbdedc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fadb80ce390 0x7fadb8070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadbdedc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fadb8070f50 0x7fadb80c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadbdedc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb806d7d0 con 0x7fadb81586a0 2023-06-10T18:44:21.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadbdedc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb80cd6c0 con 0x7fadb8070f50 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadbdedc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb80c95d0 con 0x7fadb80ce390 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fadb8070f50 0x7fadb80c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fadb81586a0 0x7fadb806cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52356/0 (socket says 172.21.15.104:52356) 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb77fe700 1 -- 172.21.15.104:0/3642906756 learned_addr learned my addr 172.21.15.104:0/3642906756 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb6ffd700 1 --2- 172.21.15.104:0/3642906756 >> v2:172.21.15.114:3300/0 conn(0x7fadb80ce390 0x7fadb8070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 -- 172.21.15.104:0/3642906756 >> v2:172.21.15.114:3300/0 conn(0x7fadb80ce390 msgr2=0x7fadb8070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 --2- 172.21.15.104:0/3642906756 >> v2:172.21.15.114:3300/0 conn(0x7fadb80ce390 0x7fadb8070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 -- 172.21.15.104:0/3642906756 >> v1:172.21.15.104:6789/0 conn(0x7fadb81586a0 legacy=0x7fadb806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb67fc700 1 -- 172.21.15.104:0/3642906756 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821860598 0 0) 0x7fadb806d7d0 con 0x7fadb81586a0 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 -- 172.21.15.104:0/3642906756 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8071390 con 0x7fadb8070f50 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.660+0000 7fadb7fff700 1 --2- 172.21.15.104:0/3642906756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb8070f50 0x7fadb80c9a60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fadb80cb910 tx=0x7fadb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f4e1f6326b306d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadb67fc700 1 -- 172.21.15.104:0/3642906756 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb000e070 con 0x7fadb8070f50 2023-06-10T18:44:21.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadb67fc700 1 -- 172.21.15.104:0/3642906756 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb000bbd0 con 0x7fadb8070f50 2023-06-10T18:44:21.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadb67fc700 1 -- 172.21.15.104:0/3642906756 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0012430 con 0x7fadb8070f50 2023-06-10T18:44:21.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 -- 172.21.15.104:0/3642906756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb8070f50 msgr2=0x7fadb80c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 --2- 172.21.15.104:0/3642906756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb8070f50 0x7fadb80c9a60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fadb80cb910 tx=0x7fadb000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 -- 172.21.15.104:0/3642906756 shutdown_connections 2023-06-10T18:44:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 --2- 172.21.15.104:0/3642906756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb8070f50 0x7fadb80c9a60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 --2- 172.21.15.104:0/3642906756 >> v2:172.21.15.114:3300/0 conn(0x7fadb80ce390 0x7fadb8070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 -- 172.21.15.104:0/3642906756 >> 172.21.15.104:0/3642906756 conn(0x7fadb80ceec0 msgr2=0x7fadb80cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 -- 172.21.15.104:0/3642906756 shutdown_connections 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.661+0000 7fadbdedc700 1 -- 172.21.15.104:0/3642906756 wait complete. 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.662+0000 7fadbdedc700 1 Processor -- start 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 -- start start 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb8070f50 0x7fadb814a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 0x7fadb80b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb814d1c0 con 0x7fadb81586a0 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb814c5b0 con 0x7fadb8070f50 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadbdedc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb814c890 con 0x7fadb80ce390 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54538/0 (socket says 172.21.15.104:54538) 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadb7fff700 1 -- 172.21.15.104:0/1749934063 learned_addr learned my addr 172.21.15.104:0/1749934063 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadb77fe700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb8070f50 0x7fadb814a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.663+0000 7fadb6ffd700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 0x7fadb80b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 -- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 msgr2=0x7fadb80b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 0x7fadb80b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 -- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb8070f50 msgr2=0x7fadb814a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb8070f50 0x7fadb814a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 -- 172.21.15.104:0/1749934063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb0009a70 con 0x7fadb81586a0 2023-06-10T18:44:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb6ffd700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 0x7fadb80b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:21.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadb7fff700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadb0014040 tx=0x7fadb0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0012430 con 0x7fadb81586a0 2023-06-10T18:44:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0020070 con 0x7fadb81586a0 2023-06-10T18:44:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb814cb70 con 0x7fadb81586a0 2023-06-10T18:44:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0004280 con 0x7fadb81586a0 2023-06-10T18:44:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.664+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb814d650 con 0x7fadb81586a0 2023-06-10T18:44:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.665+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb814d330 con 0x7fadb81586a0 2023-06-10T18:44:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.665+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fadb00043e0 con 0x7fadb81586a0 2023-06-10T18:44:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.666+0000 7fadaffff700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 0x7fad9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.666+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fadb0022070 con 0x7fadb81586a0 2023-06-10T18:44:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.666+0000 7fadb77fe700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 0x7fad9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.666+0000 7fadb77fe700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 0x7fad9806d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fada0000f60 tx=0x7fada0007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:21.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.670+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fadb00563b0 con 0x7fadb81586a0 2023-06-10T18:44:21.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.855+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fadb8074050 con 0x7fadb81586a0 2023-06-10T18:44:21.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.855+0000 7fadaffff700 1 -- 172.21.15.104:0/1749934063 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7fadb00599d0 con 0x7fadb81586a0 2023-06-10T18:44:21.862 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:21.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 msgr2=0x7fad9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 0x7fad9806d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fada0000f60 tx=0x7fada0007e70 comp rx=0 tx=0).stop 2023-06-10T18:44:21.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 msgr2=0x7fadb80c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:21.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadb0014040 tx=0x7fadb0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 shutdown_connections 2023-06-10T18:44:21.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad9806ace0 0x7fad9806d190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb8070f50 0x7fadb814a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadb81586a0 0x7fadb80c5f90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 --2- 172.21.15.104:0/1749934063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadb80ce390 0x7fadb80b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 >> 172.21.15.104:0/1749934063 conn(0x7fadb80ceec0 msgr2=0x7fadb8150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 shutdown_connections 2023-06-10T18:44:21.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.857+0000 7fadbdedc700 1 -- 172.21.15.104:0/1749934063 wait complete. 2023-06-10T18:44:21.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:21.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.991+0000 7f9272f54700 1 Processor -- start 2023-06-10T18:44:21.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.991+0000 7f9272f54700 1 -- start start 2023-06-10T18:44:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9272f54700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f926c070a10 0x7f926c06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9272f54700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c06c800 0x7f926c06cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9272f54700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0cec40 con 0x7f926c143a40 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9272f54700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0cca90 con 0x7f926c070a10 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9272f54700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0ce960 con 0x7f926c06c800 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f926c070a10 0x7f926c06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f926bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f926c143a40 0x7f926c070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52370/0 (socket says 172.21.15.104:52370) 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f926bfff700 1 -- 172.21.15.104:0/3917844996 learned_addr learned my addr 172.21.15.104:0/3917844996 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 -- 172.21.15.104:0/3917844996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c06c800 msgr2=0x7f926c06cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f92714f1700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c06c800 0x7f926c06cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f926b7fe700 1 -- 172.21.15.104:0/3917844996 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525347964 0 0) 0x7f926c0cec40 con 0x7f926c143a40 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c06c800 0x7f926c06cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 -- 172.21.15.104:0/3917844996 >> v1:172.21.15.104:6789/0 conn(0x7f926c143a40 legacy=0x7f926c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 -- 172.21.15.104:0/3917844996 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c06a9b0 con 0x7f926c070a10 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f9270cf0700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c06e560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9260009840 tx=0x7f9260009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de10d3dccd5f4d1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.992+0000 7f926b7fe700 1 -- 172.21.15.104:0/3917844996 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260011830 con 0x7f926c070a10 2023-06-10T18:44:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f926b7fe700 1 -- 172.21.15.104:0/3917844996 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9260011990 con 0x7f926c070a10 2023-06-10T18:44:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f926b7fe700 1 -- 172.21.15.104:0/3917844996 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260011d20 con 0x7f926c070a10 2023-06-10T18:44:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 -- 172.21.15.104:0/3917844996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 msgr2=0x7f926c06e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c06e560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9260009840 tx=0x7f9260009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 -- 172.21.15.104:0/3917844996 shutdown_connections 2023-06-10T18:44:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c06e560 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 --2- 172.21.15.104:0/3917844996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c06c800 0x7f926c06cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 -- 172.21.15.104:0/3917844996 >> 172.21.15.104:0/3917844996 conn(0x7f926c0ceec0 msgr2=0x7f926c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 -- 172.21.15.104:0/3917844996 shutdown_connections 2023-06-10T18:44:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.993+0000 7f9272f54700 1 -- 172.21.15.104:0/3917844996 wait complete. 2023-06-10T18:44:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.994+0000 7f9272f54700 1 Processor -- start 2023-06-10T18:44:22.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.994+0000 7f9272f54700 1 -- start start 2023-06-10T18:44:22.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f926c06c800 0x7f926c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c143a40 0x7f926c13dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c8210 con 0x7f926c070a10 2023-06-10T18:44:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c6290 con 0x7f926c06c800 2023-06-10T18:44:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9272f54700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c139160 con 0x7f926c143a40 2023-06-10T18:44:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f926bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f926bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54550/0 (socket says 172.21.15.104:54550) 2023-06-10T18:44:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f926bfff700 1 -- 172.21.15.104:0/383752822 learned_addr learned my addr 172.21.15.104:0/383752822 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f9270cf0700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f926c06c800 0x7f926c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.995+0000 7f92714f1700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c143a40 0x7f926c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 -- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c143a40 msgr2=0x7f926c13dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c143a40 0x7f926c13dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 -- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f926c06c800 msgr2=0x7f926c159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f926c06c800 0x7f926c159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 -- 172.21.15.104:0/383752822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92600094a0 con 0x7f926c070a10 2023-06-10T18:44:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f926bfff700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f92580011e0 tx=0x7f925800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258004c80 con 0x7f926c070a10 2023-06-10T18:44:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9258004e00 con 0x7f926c070a10 2023-06-10T18:44:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258007310 con 0x7f926c070a10 2023-06-10T18:44:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c0cd0a0 con 0x7f926c070a10 2023-06-10T18:44:22.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.996+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c139400 con 0x7f926c070a10 2023-06-10T18:44:22.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.997+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c074050 con 0x7f926c070a10 2023-06-10T18:44:22.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.997+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9258007470 con 0x7f926c070a10 2023-06-10T18:44:22.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.998+0000 7f92697fa700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 0x7f925006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.998+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f92580894b0 con 0x7f926c070a10 2023-06-10T18:44:22.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.998+0000 7f9270cf0700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 0x7f925006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:21.998+0000 7f9270cf0700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 0x7f925006d190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92600090b0 tx=0x7f9260008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.002+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9258054300 con 0x7f926c070a10 2023-06-10T18:44:22.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.188+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f926c141e80 con 0x7f926c070a10 2023-06-10T18:44:22.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.188+0000 7f92697fa700 1 -- 172.21.15.104:0/383752822 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9258057920 con 0x7f926c070a10 2023-06-10T18:44:22.194 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:22.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 msgr2=0x7f925006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 0x7f925006d190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92600090b0 tx=0x7f9260008030 comp rx=0 tx=0).stop 2023-06-10T18:44:22.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 msgr2=0x7f926c13d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f92580011e0 tx=0x7f925800cae0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 shutdown_connections 2023-06-10T18:44:22.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f925006ace0 0x7f925006d190 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f926c06c800 0x7f926c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f926c143a40 0x7f926c13dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 --2- 172.21.15.104:0/383752822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f926c070a10 0x7f926c13d690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 >> 172.21.15.104:0/383752822 conn(0x7f926c0ceec0 msgr2=0x7f926c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 shutdown_connections 2023-06-10T18:44:22.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.190+0000 7f9272f54700 1 -- 172.21.15.104:0/383752822 wait complete. 2023-06-10T18:44:22.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 Processor -- start 2023-06-10T18:44:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 -- start start 2023-06-10T18:44:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 0x7fc9180ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc9180ce880 0x7fc9180d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9180ad3c0 con 0x7fc9180d0830 2023-06-10T18:44:22.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9180cecf0 con 0x7fc9180ce880 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.325+0000 7fc91e471700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc918157f00 con 0x7fc918157b30 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc9180ce880 0x7fc9180d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc91ca0e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc9180d0830 0x7fc91815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52386/0 (socket says 172.21.15.104:52386) 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc9180ce880 0x7fc9180d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54556/0 (socket says 172.21.15.104:54556) 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc91ca0e700 1 -- 172.21.15.104:0/1486522499 learned_addr learned my addr 172.21.15.104:0/1486522499 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc917fff700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 0x7fc9180ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 -- 172.21.15.104:0/1486522499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 msgr2=0x7fc9180ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 0x7fc9180ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 -- 172.21.15.104:0/1486522499 >> v1:172.21.15.104:6789/0 conn(0x7fc9180d0830 legacy=0x7fc91815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc916ffd700 1 -- 172.21.15.104:0/1486522499 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139568794 0 0) 0x7fc9180ad3c0 con 0x7fc9180d0830 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 -- 172.21.15.104:0/1486522499 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9180b0170 con 0x7fc9180ce880 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc917fff700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 0x7fc9180ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:22.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc9177fe700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9180d02f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc900009840 tx=0x7fc900009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae49818ffec95b7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc916ffd700 1 -- 172.21.15.104:0/1486522499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900011830 con 0x7fc9180ce880 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc916ffd700 1 -- 172.21.15.104:0/1486522499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc900011990 con 0x7fc9180ce880 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.326+0000 7fc916ffd700 1 -- 172.21.15.104:0/1486522499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900011d20 con 0x7fc9180ce880 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 -- 172.21.15.104:0/1486522499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 msgr2=0x7fc9180d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9180d02f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc900009840 tx=0x7fc900009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 -- 172.21.15.104:0/1486522499 shutdown_connections 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc918157b30 0x7fc9180ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 --2- 172.21.15.104:0/1486522499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9180d02f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 -- 172.21.15.104:0/1486522499 >> 172.21.15.104:0/1486522499 conn(0x7fc918089fd0 msgr2=0x7fc91808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 -- 172.21.15.104:0/1486522499 shutdown_connections 2023-06-10T18:44:22.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.327+0000 7fc91e471700 1 -- 172.21.15.104:0/1486522499 wait complete. 2023-06-10T18:44:22.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.328+0000 7fc91e471700 1 Processor -- start 2023-06-10T18:44:22.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.328+0000 7fc91e471700 1 -- start start 2023-06-10T18:44:22.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9180d0830 0x7fc9180b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc918157b30 0x7fc9180b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180c6f50 con 0x7fc9180ce880 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180cbc50 con 0x7fc918157b30 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91e471700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180b6360 con 0x7fc9180d0830 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54560/0 (socket says 172.21.15.104:54560) 2023-06-10T18:44:22.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 -- 172.21.15.104:0/635333720 learned_addr learned my addr 172.21.15.104:0/635333720 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc91ca0e700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc918157b30 0x7fc9180b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 -- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9180d0830 msgr2=0x7fc9180b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc9177fe700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9180d0830 0x7fc9180b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9180d0830 0x7fc9180b58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 -- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc918157b30 msgr2=0x7fc9180b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc918157b30 0x7fc9180b5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 -- 172.21.15.104:0/635333720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000094a0 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.329+0000 7fc917fff700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c820 tx=0x7fc90800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.330+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011800 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.330+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc908011960 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.330+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9180cbf00 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.330+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011d10 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.330+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9180cc740 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.331+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9180cc960 con 0x7fc9180ce880 2023-06-10T18:44:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.331+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc908018360 con 0x7fc9180ce880 2023-06-10T18:44:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.331+0000 7fc914ff9700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 0x7fc8f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.331+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc908089740 con 0x7fc9180ce880 2023-06-10T18:44:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.332+0000 7fc9177fe700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 0x7fc8f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.332+0000 7fc9177fe700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 0x7fc8f806d190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc9180b5600 tx=0x7fc900008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.336+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc908057fa0 con 0x7fc9180ce880 2023-06-10T18:44:22.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.520+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc918074050 con 0x7fc9180ce880 2023-06-10T18:44:22.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.524+0000 7fc914ff9700 1 -- 172.21.15.104:0/635333720 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc908057b30 con 0x7fc9180ce880 2023-06-10T18:44:22.529 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:22.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 msgr2=0x7fc8f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 0x7fc8f806d190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc9180b5600 tx=0x7fc900008030 comp rx=0 tx=0).stop 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 msgr2=0x7fc9181f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c820 tx=0x7fc90800cbe0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 shutdown_connections 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8f806ace0 0x7fc8f806d190 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc918157b30 0x7fc9180b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9180ce880 0x7fc9181f4a20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 --2- 172.21.15.104:0/635333720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9180d0830 0x7fc9180b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 >> 172.21.15.104:0/635333720 conn(0x7fc918089fd0 msgr2=0x7fc91815b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 shutdown_connections 2023-06-10T18:44:22.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.526+0000 7fc91e471700 1 -- 172.21.15.104:0/635333720 wait complete. 2023-06-10T18:44:22.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:22.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.659+0000 7f003e500700 1 Processor -- start 2023-06-10T18:44:22.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.660+0000 7f003e500700 1 -- start start 2023-06-10T18:44:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003e500700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003e500700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003e500700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00380c54f0 con 0x7f0038158d30 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003e500700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00380ced10 con 0x7f00380c5980 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003e500700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00380c48e0 con 0x7f003806db70 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003ca9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003ca9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42074/0 (socket says 172.21.15.104:42074) 2023-06-10T18:44:22.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45722/0 (socket says 172.21.15.104:45722) 2023-06-10T18:44:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f0037fff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0038158d30 0x7f003806d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58242/0 (socket says 172.21.15.104:58242) 2023-06-10T18:44:22.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003ca9d700 1 -- 172.21.15.104:0/3823538117 learned_addr learned my addr 172.21.15.104:0/3823538117 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 -- 172.21.15.104:0/3823538117 >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 msgr2=0x7f00380ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 --2- 172.21.15.104:0/3823538117 >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 -- 172.21.15.104:0/3823538117 >> v1:172.21.15.114:6789/0 conn(0x7f0038158d30 legacy=0x7f003806d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f00377fe700 1 -- 172.21.15.104:0/3823538117 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00380c4bc0 con 0x7f003806db70 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.661+0000 7f003ca9d700 1 --2- 172.21.15.104:0/3823538117 >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f00377fe700 1 --2- 172.21.15.104:0/3823538117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0020009840 tx=0x7f0020009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a753745012ca283 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f0036ffd700 1 -- 172.21.15.104:0/3823538117 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020011830 con 0x7f003806db70 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f0036ffd700 1 -- 172.21.15.104:0/3823538117 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020011990 con 0x7f003806db70 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f0036ffd700 1 -- 172.21.15.104:0/3823538117 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020011d20 con 0x7f003806db70 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 -- 172.21.15.104:0/3823538117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 msgr2=0x7f00380c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 --2- 172.21.15.104:0/3823538117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0020009840 tx=0x7f0020009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 -- 172.21.15.104:0/3823538117 shutdown_connections 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 --2- 172.21.15.104:0/3823538117 >> v2:172.21.15.114:3300/0 conn(0x7f00380c5980 0x7f00380ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 --2- 172.21.15.104:0/3823538117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f003806db70 0x7f00380c4250 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 -- 172.21.15.104:0/3823538117 >> 172.21.15.104:0/3823538117 conn(0x7f00380ceec0 msgr2=0x7f00380cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 -- 172.21.15.104:0/3823538117 shutdown_connections 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.662+0000 7f003e500700 1 -- 172.21.15.104:0/3823538117 wait complete. 2023-06-10T18:44:22.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.664+0000 7f003e500700 1 Processor -- start 2023-06-10T18:44:22.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.664+0000 7f003e500700 1 -- start start 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 0x7f0038153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0038158d30 0x7f0038157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00381eec40 con 0x7f00380c5980 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0038153b90 con 0x7f003806db70 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003e500700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00381eeeb0 con 0x7f0038158d30 2023-06-10T18:44:22.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54576/0 (socket says 172.21.15.104:54576) 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 -- 172.21.15.104:0/3717699121 learned_addr learned my addr 172.21.15.104:0/3717699121 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f0037fff700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 0x7f0038153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f003ca9d700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0038158d30 0x7f0038157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 -- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0038158d30 msgr2=0x7f0038157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0038158d30 0x7f0038157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 -- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 msgr2=0x7f0038153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 0x7f0038153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f00377fe700 1 -- 172.21.15.104:0/3717699121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0030009b30 con 0x7f00380c5980 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.665+0000 7f0037fff700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 0x7f0038153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f00377fe700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00200090b0 tx=0x7f0020004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020004530 con 0x7f00380c5980 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200036a0 con 0x7f00380c5980 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00200094a0 con 0x7f00380c5980 2023-06-10T18:44:22.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020003a50 con 0x7f00380c5980 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.666+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00381ef470 con 0x7f00380c5980 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.667+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00380ccb90 con 0x7f00380c5980 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.667+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0020023070 con 0x7f00380c5980 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.668+0000 7f0034ff9700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 0x7f001806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.668+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f002008c690 con 0x7f00380c5980 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.668+0000 7f0037fff700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 0x7f001806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:22.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.668+0000 7f0037fff700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 0x7f001806d190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f60 tx=0x7f0028009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:22.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.672+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0020057600 con 0x7f00380c5980 2023-06-10T18:44:22.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.859+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0038074050 con 0x7f00380c5980 2023-06-10T18:44:22.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.860+0000 7f0034ff9700 1 -- 172.21.15.104:0/3717699121 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f002005ac20 con 0x7f00380c5980 2023-06-10T18:44:22.866 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:22.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 msgr2=0x7f001806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 0x7f001806d190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f60 tx=0x7f0028009040 comp rx=0 tx=0).stop 2023-06-10T18:44:22.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 msgr2=0x7f0038156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:22.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00200090b0 tx=0x7f0020004750 comp rx=0 tx=0).stop 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 shutdown_connections 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f001806ace0 0x7f001806d190 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f003806db70 0x7f0038153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0038158d30 0x7f0038157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 --2- 172.21.15.104:0/3717699121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00380c5980 0x7f0038156dc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 >> 172.21.15.104:0/3717699121 conn(0x7f00380ceec0 msgr2=0x7f00380ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:22.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 shutdown_connections 2023-06-10T18:44:22.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.862+0000 7f003e500700 1 -- 172.21.15.104:0/3717699121 wait complete. 2023-06-10T18:44:22.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:44:23.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.995+0000 7fa6ca8b8700 1 Processor -- start 2023-06-10T18:44:23.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.995+0000 7fa6ca8b8700 1 -- start start 2023-06-10T18:44:23.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6ca8b8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa6c406e0e0 0x7fa6c406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:23.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6ca8b8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c406d290 con 0x7fa6c406e4b0 2023-06-10T18:44:23.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6ca8b8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40ca2d0 con 0x7fa6c4069e70 2023-06-10T18:44:23.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6ca8b8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c4071640 con 0x7fa6c406e0e0 2023-06-10T18:44:23.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa6c406e0e0 0x7fa6c406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:23.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c37fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa6c406e4b0 0x7fa6c40c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52412/0 (socket says 172.21.15.104:52412) 2023-06-10T18:44:23.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c37fe700 1 -- 172.21.15.104:0/329366403 learned_addr learned my addr 172.21.15.104:0/329366403 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:23.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c3fff700 1 -- 172.21.15.104:0/329366403 >> v1:172.21.15.114:6789/0 conn(0x7fa6c4069e70 legacy=0x7fa6c40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c3fff700 1 -- 172.21.15.104:0/329366403 >> v1:172.21.15.104:6789/0 conn(0x7fa6c406e4b0 legacy=0x7fa6c40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c3fff700 1 -- 172.21.15.104:0/329366403 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c406d4c0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/329366403 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745972514 0 0) 0x7fa6c406d290 con 0x7fa6c406e4b0 2023-06-10T18:44:23.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c3fff700 1 --2- 172.21.15.104:0/329366403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c406cd50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009820 tx=0x7fa6b8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6eefcdb2c81957 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:23.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.996+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/329366403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8005e30 con 0x7fa6c406e0e0 2023-06-10T18:44:23.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/329366403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b8006030 con 0x7fa6c406e0e0 2023-06-10T18:44:23.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/329366403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80063e0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/329366403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 msgr2=0x7fa6c406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/329366403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c406cd50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009820 tx=0x7fa6b8009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:23.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/329366403 shutdown_connections 2023-06-10T18:44:23.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/329366403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c406cd50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/329366403 >> 172.21.15.104:0/329366403 conn(0x7fa6c406b8a0 msgr2=0x7fa6c40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/329366403 shutdown_connections 2023-06-10T18:44:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.997+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/329366403 wait complete. 2023-06-10T18:44:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.998+0000 7fa6ca8b8700 1 Processor -- start 2023-06-10T18:44:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 -- start start 2023-06-10T18:44:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c4069e70 0x7fa6c40b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c406e4b0 0x7fa6c40b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c414e2c0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b4350 con 0x7fa6c406e4b0 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6ca8b8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40b4630 con 0x7fa6c4069e70 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54590/0 (socket says 172.21.15.104:54590) 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:22.999+0000 7fa6c37fe700 1 -- 172.21.15.104:0/1080143349 learned_addr learned my addr 172.21.15.104:0/1080143349 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c3fff700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c4069e70 0x7fa6c40b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c8e55700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c406e4b0 0x7fa6c40b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 -- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c4069e70 msgr2=0x7fa6c40b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c4069e70 0x7fa6c40b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 -- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c406e4b0 msgr2=0x7fa6c40b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c406e4b0 0x7fa6c40b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 -- 172.21.15.104:0/1080143349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8009430 con 0x7fa6c406e0e0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c37fe700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c520 tx=0x7fa6b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4004c80 con 0x7fa6c406e0e0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4004e00 con 0x7fa6c406e0e0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c40b4910 con 0x7fa6c406e0e0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4007310 con 0x7fa6c406e0e0 2023-06-10T18:44:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.000+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c40b4ed0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.001+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c40ce4b0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.001+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa6b4007470 con 0x7fa6c406e0e0 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.002+0000 7fa6c0ff9700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 0x7fa6b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.002+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa6b4089530 con 0x7fa6c406e0e0 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.002+0000 7fa6c3fff700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 0x7fa6b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.003+0000 7fa6c3fff700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 0x7fa6b006d190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8000c00 tx=0x7fa6b8008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:23.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.006+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa6c40ce4b0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.207+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa6c40ce4b0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.208+0000 7fa6c0ff9700 1 -- 172.21.15.104:0/1080143349 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa6c40ce4b0 con 0x7fa6c406e0e0 2023-06-10T18:44:23.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.209+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 msgr2=0x7fa6b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.209+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 0x7fa6b006d190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8000c00 tx=0x7fa6b8008070 comp rx=0 tx=0).stop 2023-06-10T18:44:23.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.209+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 msgr2=0x7fa6c41f1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c520 tx=0x7fa6b400c8e0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 shutdown_connections 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b006ace0 0x7fa6b006d190 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c406e4b0 0x7fa6c40b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c4069e70 0x7fa6c40b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 --2- 172.21.15.104:0/1080143349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c406e0e0 0x7fa6c41f1c90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:23.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 >> 172.21.15.104:0/1080143349 conn(0x7fa6c406b8a0 msgr2=0x7fa6c40c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 shutdown_connections 2023-06-10T18:44:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:23.210+0000 7fa6ca8b8700 1 -- 172.21.15.104:0/1080143349 wait complete. 2023-06-10T18:44:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:44:23.227 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:44:23.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:23.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-06-10T18:44:23.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi104.stderr:ceph-fuse[107831]: fuse finished with error 0 and tester_r 0 2023-06-10T18:44:23.265 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:44:29.367 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:44:29.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:29.367 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-06-10T18:44:29.398 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:29.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.566+0000 7fd18db64700 1 Processor -- start 2023-06-10T18:44:29.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.566+0000 7fd18db64700 1 -- start start 2023-06-10T18:44:29.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.566+0000 7fd18db64700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd18db64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd18db64700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880cee20 con 0x7fd188070580 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd18db64700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1880caa90 con 0x7fd1880ce510 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd18db64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd188071fe0 con 0x7fd188070130 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd1877fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd1877fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42098/0 (socket says 172.21.15.104:42098) 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45750/0 (socket says 172.21.15.104:45750) 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd1877fe700 1 -- 172.21.15.104:0/3854422526 learned_addr learned my addr 172.21.15.104:0/3854422526 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd187fff700 1 --1- 172.21.15.104:0/3854422526 >> v1:172.21.15.114:6789/0 conn(0x7fd188070580 0x7fd1880cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58274/0 (socket says 172.21.15.104:58274) 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 -- 172.21.15.104:0/3854422526 >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 msgr2=0x7fd1880ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 --2- 172.21.15.104:0/3854422526 >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 -- 172.21.15.104:0/3854422526 >> v1:172.21.15.114:6789/0 conn(0x7fd188070580 legacy=0x7fd1880cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd1867fc700 1 -- 172.21.15.104:0/3854422526 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944567083 0 0) 0x7fd1880cee20 con 0x7fd188070580 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd186ffd700 1 -- 172.21.15.104:0/3854422526 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1880722c0 con 0x7fd188070130 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.567+0000 7fd1877fe700 1 --2- 172.21.15.104:0/3854422526 >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd186ffd700 1 --2- 172.21.15.104:0/3854422526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009840 tx=0x7fd17c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=964ed181c8007b4d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd1867fc700 1 -- 172.21.15.104:0/3854422526 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011830 con 0x7fd188070130 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd1867fc700 1 -- 172.21.15.104:0/3854422526 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c011990 con 0x7fd188070130 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd1867fc700 1 -- 172.21.15.104:0/3854422526 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011d20 con 0x7fd188070130 2023-06-10T18:44:29.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 -- 172.21.15.104:0/3854422526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 msgr2=0x7fd1880c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 --2- 172.21.15.104:0/3854422526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009840 tx=0x7fd17c009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 -- 172.21.15.104:0/3854422526 shutdown_connections 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 --2- 172.21.15.104:0/3854422526 >> v2:172.21.15.114:3300/0 conn(0x7fd1880ce510 0x7fd1880ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 --2- 172.21.15.104:0/3854422526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd188070130 0x7fd1880c8bd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 -- 172.21.15.104:0/3854422526 >> 172.21.15.104:0/3854422526 conn(0x7fd18806ab00 msgr2=0x7fd1880b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 -- 172.21.15.104:0/3854422526 shutdown_connections 2023-06-10T18:44:29.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.568+0000 7fd18db64700 1 -- 172.21.15.104:0/3854422526 wait complete. 2023-06-10T18:44:29.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.569+0000 7fd18db64700 1 Processor -- start 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 -- start start 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd188070130 0x7fd1881e6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1880ce510 0x7fd1881efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1881e6740 con 0x7fd188070580 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188145e40 con 0x7fd188070130 2023-06-10T18:44:29.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.570+0000 7fd18db64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd188153da0 con 0x7fd1880ce510 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54604/0 (socket says 172.21.15.104:54604) 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 -- 172.21.15.104:0/2839757843 learned_addr learned my addr 172.21.15.104:0/2839757843 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd1877fe700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd188070130 0x7fd1881e6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd187fff700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1880ce510 0x7fd1881efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 -- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1880ce510 msgr2=0x7fd1881efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1880ce510 0x7fd1881efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 -- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd188070130 msgr2=0x7fd1881e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd188070130 0x7fd1881e6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 -- 172.21.15.104:0/2839757843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd178009550 con 0x7fd188070580 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd186ffd700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd17c014040 tx=0x7fd17c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c0036a0 con 0x7fd188070580 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c01e070 con 0x7fd188070580 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c0094a0 con 0x7fd188070580 2023-06-10T18:44:29.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c003a40 con 0x7fd188070580 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.571+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd188154360 con 0x7fd188070580 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.572+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188153ee0 con 0x7fd188070580 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.573+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd17c003ba0 con 0x7fd188070580 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.573+0000 7fd16ffff700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 0x7fd17406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.573+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd17c08a6a0 con 0x7fd188070580 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.573+0000 7fd1877fe700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 0x7fd17406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.574+0000 7fd1877fe700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 0x7fd17406d190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd178006d30 tx=0x7fd1780092c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.578+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd188153ee0 con 0x7fd188070580 2023-06-10T18:44:29.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.766+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd188153ee0 con 0x7fd188070580 2023-06-10T18:44:29.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.767+0000 7fd16ffff700 1 -- 172.21.15.104:0/2839757843 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7fd188153ee0 con 0x7fd188070580 2023-06-10T18:44:29.773 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:29.773 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 msgr2=0x7fd17406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 0x7fd17406d190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd178006d30 tx=0x7fd1780092c0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 msgr2=0x7fd188145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd17c014040 tx=0x7fd17c004750 comp rx=0 tx=0).stop 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 shutdown_connections 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd17406ace0 0x7fd17406d190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd188070130 0x7fd1881e6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1880ce510 0x7fd1881efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 --2- 172.21.15.104:0/2839757843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd188070580 0x7fd188145900 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 >> 172.21.15.104:0/2839757843 conn(0x7fd18806ab00 msgr2=0x7fd1880b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 shutdown_connections 2023-06-10T18:44:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.769+0000 7fd18db64700 1 -- 172.21.15.104:0/2839757843 wait complete. 2023-06-10T18:44:29.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:29.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.901+0000 7f3325b6a700 1 Processor -- start 2023-06-10T18:44:29.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.901+0000 7f3325b6a700 1 -- start start 2023-06-10T18:44:29.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3325b6a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3320070a20 0x7f332006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3325b6a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3320070df0 0x7f33201587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3325b6a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332006ae00 con 0x7f3320158ce0 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3325b6a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3320069e70 con 0x7f3320070df0 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3325b6a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33201595f0 con 0x7f3320070a20 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3317fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3320070df0 0x7f33201587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3317fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3320070df0 0x7f33201587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54612/0 (socket says 172.21.15.104:54612) 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.902+0000 7f3317fff700 1 -- 172.21.15.104:0/477180978 learned_addr learned my addr 172.21.15.104:0/477180978 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f331f7fe700 1 --2- 172.21.15.104:0/477180978 >> v2:172.21.15.114:3300/0 conn(0x7f3320070a20 0x7f332006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3317fff700 1 -- 172.21.15.104:0/477180978 >> v2:172.21.15.114:3300/0 conn(0x7f3320070a20 msgr2=0x7f332006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:29.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3317fff700 1 --2- 172.21.15.104:0/477180978 >> v2:172.21.15.114:3300/0 conn(0x7f3320070a20 0x7f332006a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3317fff700 1 -- 172.21.15.104:0/477180978 >> v1:172.21.15.114:6789/0 conn(0x7f3320158ce0 legacy=0x7f3320159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:29.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3317fff700 1 -- 172.21.15.104:0/477180978 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33201598d0 con 0x7f3320070df0 2023-06-10T18:44:29.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3317fff700 1 --2- 172.21.15.104:0/477180978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320070df0 0x7f33201587a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3320072330 tx=0x7f3308009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2119817b1ea203ce server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f331effd700 1 -- 172.21.15.104:0/477180978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308004230 con 0x7f3320070df0 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f331effd700 1 -- 172.21.15.104:0/477180978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308004390 con 0x7f3320070df0 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f331effd700 1 -- 172.21.15.104:0/477180978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080036a0 con 0x7f3320070df0 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 -- 172.21.15.104:0/477180978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320070df0 msgr2=0x7f33201587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 --2- 172.21.15.104:0/477180978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320070df0 0x7f33201587a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3320072330 tx=0x7f3308009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 -- 172.21.15.104:0/477180978 shutdown_connections 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 --2- 172.21.15.104:0/477180978 >> v2:172.21.15.114:3300/0 conn(0x7f3320070a20 0x7f332006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 --2- 172.21.15.104:0/477180978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320070df0 0x7f33201587a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 -- 172.21.15.104:0/477180978 >> 172.21.15.104:0/477180978 conn(0x7f33200b1610 msgr2=0x7f33200ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:29.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 -- 172.21.15.104:0/477180978 shutdown_connections 2023-06-10T18:44:29.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.903+0000 7f3325b6a700 1 -- 172.21.15.104:0/477180978 wait complete. 2023-06-10T18:44:29.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 Processor -- start 2023-06-10T18:44:29.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 -- start start 2023-06-10T18:44:29.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3320070a20 0x7f33200c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33200cb810 0x7f33200ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200c8d30 con 0x7f3320158ce0 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320154a30 con 0x7f33200cb810 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3325b6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200c83d0 con 0x7f3320070a20 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3317fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3317fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54618/0 (socket says 172.21.15.104:54618) 2023-06-10T18:44:29.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f3317fff700 1 -- 172.21.15.104:0/706871363 learned_addr learned my addr 172.21.15.104:0/706871363 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f331f7fe700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3320070a20 0x7f33200c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.905+0000 7f331ffff700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33200cb810 0x7f33200ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 -- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3320070a20 msgr2=0x7f33200c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3320070a20 0x7f33200c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 -- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33200cb810 msgr2=0x7f33200ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33200cb810 0x7f33200ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:29.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 -- 172.21.15.104:0/706871363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308009550 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3317fff700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3308009130 tx=0x7f3308004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308003bd0 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330801f070 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308011630 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33200c2ef0 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.906+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33200c31b0 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.907+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33200c3340 con 0x7f3320158ce0 2023-06-10T18:44:29.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.907+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3308021030 con 0x7f3320158ce0 2023-06-10T18:44:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.908+0000 7f331cff9700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 0x7f330006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.908+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f330808aa00 con 0x7f3320158ce0 2023-06-10T18:44:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.908+0000 7f331f7fe700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 0x7f330006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.908+0000 7f331f7fe700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 0x7f330006d190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33200c2ce0 tx=0x7f3310008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:29.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:29.912+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3308055850 con 0x7f3320158ce0 2023-06-10T18:44:30.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.098+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f332006ae00 con 0x7f3320158ce0 2023-06-10T18:44:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.099+0000 7f331cff9700 1 -- 172.21.15.104:0/706871363 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f332006ae00 con 0x7f3320158ce0 2023-06-10T18:44:30.106 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:30.106 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 msgr2=0x7f330006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 0x7f330006d190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33200c2ce0 tx=0x7f3310008040 comp rx=0 tx=0).stop 2023-06-10T18:44:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 msgr2=0x7f33201544f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3308009130 tx=0x7f3308004750 comp rx=0 tx=0).stop 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 shutdown_connections 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330006ace0 0x7f330006d190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33200cb810 0x7f33200ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3320070a20 0x7f33200c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 --2- 172.21.15.104:0/706871363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3320158ce0 0x7f33201544f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 >> 172.21.15.104:0/706871363 conn(0x7f33200b1610 msgr2=0x7f33200aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:30.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 shutdown_connections 2023-06-10T18:44:30.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.101+0000 7f3325b6a700 1 -- 172.21.15.104:0/706871363 wait complete. 2023-06-10T18:44:30.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:44:30.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.236+0000 7f0ebfc14700 1 Processor -- start 2023-06-10T18:44:30.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.237+0000 7f0ebfc14700 1 -- start start 2023-06-10T18:44:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.237+0000 7f0ebfc14700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0eb815c580 0x7f0eb80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebfc14700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb806cde0 0x7f0eb80ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebfc14700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb806e250 con 0x7f0eb806e6e0 2023-06-10T18:44:30.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebfc14700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb815c950 con 0x7f0eb815c580 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebfc14700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b1e80 con 0x7f0eb806cde0 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0eb815c580 0x7f0eb80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd1af700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0eb806e6e0 0x7f0eb80b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52452/0 (socket says 172.21.15.104:52452) 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd1af700 1 -- 172.21.15.104:0/1791373086 learned_addr learned my addr 172.21.15.104:0/1791373086 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebe1b1700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb806cde0 0x7f0eb80ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/1791373086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb806cde0 msgr2=0x7f0eb80ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:30.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb806cde0 0x7f0eb80ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/1791373086 >> v1:172.21.15.104:6789/0 conn(0x7f0eb806e6e0 legacy=0x7f0eb80b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/1791373086 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb806d920 con 0x7f0eb815c580 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebc9ae700 1 -- 172.21.15.104:0/1791373086 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221434064 0 0) 0x7f0eb806e250 con 0x7f0eb806e6e0 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebd9b0700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb815c580 0x7f0eb80b1070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009820 tx=0x7f0ea8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dcd5c271390f59 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebc9ae700 1 -- 172.21.15.104:0/1791373086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8005e30 con 0x7f0eb815c580 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebc9ae700 1 -- 172.21.15.104:0/1791373086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea8006030 con 0x7f0eb815c580 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.238+0000 7f0ebc9ae700 1 -- 172.21.15.104:0/1791373086 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80063e0 con 0x7f0eb815c580 2023-06-10T18:44:30.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 -- 172.21.15.104:0/1791373086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb815c580 msgr2=0x7f0eb80b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb815c580 0x7f0eb80b1070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009820 tx=0x7f0ea8009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 -- 172.21.15.104:0/1791373086 shutdown_connections 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb815c580 0x7f0eb80b1070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/1791373086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb806cde0 0x7f0eb80ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 -- 172.21.15.104:0/1791373086 >> 172.21.15.104:0/1791373086 conn(0x7f0eb806b8a0 msgr2=0x7f0eb80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:30.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 -- 172.21.15.104:0/1791373086 shutdown_connections 2023-06-10T18:44:30.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.239+0000 7f0ebfc14700 1 -- 172.21.15.104:0/1791373086 wait complete. 2023-06-10T18:44:30.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.240+0000 7f0ebfc14700 1 Processor -- start 2023-06-10T18:44:30.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.240+0000 7f0ebfc14700 1 -- start start 2023-06-10T18:44:30.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb806e6e0 0x7f0eb80cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb815c580 0x7f0eb80cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c4440 con 0x7f0eb806cde0 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80cf610 con 0x7f0eb806e6e0 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebfc14700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8158a70 con 0x7f0eb815c580 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54630/0 (socket says 172.21.15.104:54630) 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/2442113587 learned_addr learned my addr 172.21.15.104:0/2442113587 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb815c580 msgr2=0x7f0eb80cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb815c580 0x7f0eb80cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb806e6e0 msgr2=0x7f0eb80cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb806e6e0 0x7f0eb80cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 -- 172.21.15.104:0/2442113587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8009430 con 0x7f0eb806cde0 2023-06-10T18:44:30.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.241+0000 7f0ebd9b0700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009b70 tx=0x7f0ea8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8025430 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea8020070 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80257d0 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb8158d50 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb8157a00 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.242+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb8074050 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.243+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0ea8022030 con 0x7f0eb806cde0 2023-06-10T18:44:30.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.243+0000 7f0eae7fc700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 0x7f0e9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.243+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0ea808aeb0 con 0x7f0eb806cde0 2023-06-10T18:44:30.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.244+0000 7f0ebd1af700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 0x7f0e9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.244+0000 7f0ebd1af700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 0x7f0e9c06d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40011e0 tx=0x7f0eb40079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.248+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ea8055d00 con 0x7f0eb806cde0 2023-06-10T18:44:30.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.477+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0eb806dc00 con 0x7f0eb806cde0 2023-06-10T18:44:30.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.542+0000 7f0eae7fc700 1 -- 172.21.15.104:0/2442113587 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0ea8059320 con 0x7f0eb806cde0 2023-06-10T18:44:30.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 msgr2=0x7f0e9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 0x7f0e9c06d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40011e0 tx=0x7f0eb40079b0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 msgr2=0x7f0eb8156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009b70 tx=0x7f0ea8004750 comp rx=0 tx=0).stop 2023-06-10T18:44:30.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 shutdown_connections 2023-06-10T18:44:30.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eb815c580 0x7f0eb80cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eb806cde0 0x7f0eb8156c70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e9c06ace0 0x7f0e9c06d190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 --2- 172.21.15.104:0/2442113587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb806e6e0 0x7f0eb80cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 >> 172.21.15.104:0/2442113587 conn(0x7f0eb806b8a0 msgr2=0x7f0eb8159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:30.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.544+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 shutdown_connections 2023-06-10T18:44:30.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.545+0000 7f0ebfc14700 1 -- 172.21.15.104:0/2442113587 wait complete. 2023-06-10T18:44:30.562 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-06-10T18:44:30.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:44:30.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.678+0000 7f70e83c8700 1 Processor -- start 2023-06-10T18:44:30.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.678+0000 7f70e83c8700 1 -- start start 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.679+0000 7f70e83c8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70e00cea50 0x7f70e00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.679+0000 7f70e83c8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00ad3c0 con 0x7f70e0157b30 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.679+0000 7f70e83c8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e0157f00 con 0x7f70e00d0830 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.679+0000 7f70e83c8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e00b0170 con 0x7f70e00cea50 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70e00cea50 0x7f70e00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70e00cea50 0x7f70e00d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54640/0 (socket says 172.21.15.104:54640) 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 -- 172.21.15.104:0/454564593 learned_addr learned my addr 172.21.15.104:0/454564593 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e6164700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f70e0157b30 0x7f70e00ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52466/0 (socket says 172.21.15.104:52466) 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 -- 172.21.15.104:0/454564593 >> v1:172.21.15.114:6789/0 conn(0x7f70e00d0830 legacy=0x7f70e00b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 -- 172.21.15.104:0/454564593 >> v1:172.21.15.104:6789/0 conn(0x7f70e0157b30 legacy=0x7f70e00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 -- 172.21.15.104:0/454564593 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00b0410 con 0x7f70e00cea50 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5963700 1 --2- 172.21.15.104:0/454564593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e00cea50 0x7f70e00d02f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0098f0 tx=0x7f70dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f768d4c12930545a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5162700 1 -- 172.21.15.104:0/454564593 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc004230 con 0x7f70e00cea50 2023-06-10T18:44:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5162700 1 -- 172.21.15.104:0/454564593 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc004390 con 0x7f70e00cea50 2023-06-10T18:44:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.680+0000 7f70e5162700 1 -- 172.21.15.104:0/454564593 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0036a0 con 0x7f70e00cea50 2023-06-10T18:44:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 -- 172.21.15.104:0/454564593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e00cea50 msgr2=0x7f70e00d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 --2- 172.21.15.104:0/454564593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e00cea50 0x7f70e00d02f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0098f0 tx=0x7f70dc009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 -- 172.21.15.104:0/454564593 shutdown_connections 2023-06-10T18:44:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 --2- 172.21.15.104:0/454564593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e00cea50 0x7f70e00d02f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 -- 172.21.15.104:0/454564593 >> 172.21.15.104:0/454564593 conn(0x7f70e0089fd0 msgr2=0x7f70e008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 -- 172.21.15.104:0/454564593 shutdown_connections 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.681+0000 7f70e83c8700 1 -- 172.21.15.104:0/454564593 wait complete. 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.682+0000 7f70e83c8700 1 Processor -- start 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.682+0000 7f70e83c8700 1 -- start start 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 0x7f70e0146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 0x7f70e01596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e0159c20 con 0x7f70e0157b30 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00b2cf0 con 0x7f70e00cea50 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00b2fd0 con 0x7f70e00d0830 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54648/0 (socket says 172.21.15.104:54648) 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 -- 172.21.15.104:0/2616896127 learned_addr learned my addr 172.21.15.104:0/2616896127 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e5963700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 0x7f70e01596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6164700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 0x7f70e0146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 -- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 msgr2=0x7f70e01596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 0x7f70e01596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 -- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 msgr2=0x7f70e0146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 0x7f70e0146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 -- 172.21.15.104:0/2616896127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc009550 con 0x7f70e0157b30 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6164700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 0x7f70e0146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:30.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e5963700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 0x7f70e01596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e6965700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70d800cc10 tx=0x7f70d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d800abb0 con 0x7f70e0157b30 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d800ad10 con 0x7f70e0157b30 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e00b18a0 con 0x7f70e0157b30 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8012660 con 0x7f70e0157b30 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.683+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e00b1e60 con 0x7f70e0157b30 2023-06-10T18:44:30.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.684+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e00b2050 con 0x7f70e0157b30 2023-06-10T18:44:30.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.685+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f70d80127c0 con 0x7f70e0157b30 2023-06-10T18:44:30.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.685+0000 7f70d6ffd700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 0x7f70c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:30.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.685+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f70d8089ef0 con 0x7f70e0157b30 2023-06-10T18:44:30.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.685+0000 7f70e6164700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 0x7f70c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:30.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.686+0000 7f70e6164700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 0x7f70c406d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000f60 tx=0x7f70d0007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:30.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.690+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f70d8054cb0 con 0x7f70e0157b30 2023-06-10T18:44:30.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:30.920+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f70e0074050 con 0x7f70e0157b30 2023-06-10T18:44:31.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.547+0000 7f70d6ffd700 1 -- 172.21.15.104:0/2616896127 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f70d8016020 con 0x7f70e0157b30 2023-06-10T18:44:31.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 msgr2=0x7f70c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 0x7f70c406d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000f60 tx=0x7f70d0007df0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 msgr2=0x7f70e015b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70d800cc10 tx=0x7f70d800a060 comp rx=0 tx=0).stop 2023-06-10T18:44:31.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 shutdown_connections 2023-06-10T18:44:31.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c406ace0 0x7f70c406d190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70e00cea50 0x7f70e0146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70e0157b30 0x7f70e015b510 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 --2- 172.21.15.104:0/2616896127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70e00d0830 0x7f70e01596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 >> 172.21.15.104:0/2616896127 conn(0x7f70e0089fd0 msgr2=0x7f70e0153000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:31.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 shutdown_connections 2023-06-10T18:44:31.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.550+0000 7f70e83c8700 1 -- 172.21.15.104:0/2616896127 wait complete. 2023-06-10T18:44:31.570 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:44:31.571 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:31.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.691+0000 7fe46c3bf700 1 Processor -- start 2023-06-10T18:44:31.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.691+0000 7fe46c3bf700 1 -- start start 2023-06-10T18:44:31.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe46415a420 0x7fe4640b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4640b4ff0 0x7fe4640ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4640ce890 0x7fe4640b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640ced10 con 0x7fe46415a420 2023-06-10T18:44:31.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640b6070 con 0x7fe4640ce890 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46c3bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640b5460 con 0x7fe4640b4ff0 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46a15b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe46415a420 0x7fe4640b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46a15b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe46415a420 0x7fe4640b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54652/0 (socket says 172.21.15.104:54652) 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.692+0000 7fe46a15b700 1 -- 172.21.15.104:0/2049080028 learned_addr learned my addr 172.21.15.104:0/2049080028 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a95c700 1 --2- 172.21.15.104:0/2049080028 >> v2:172.21.15.114:3300/0 conn(0x7fe4640ce890 0x7fe4640b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46995a700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4640b4ff0 0x7fe4640ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 -- 172.21.15.104:0/2049080028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4640b4ff0 msgr2=0x7fe4640ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:31.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4640b4ff0 0x7fe4640ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 -- 172.21.15.104:0/2049080028 >> v2:172.21.15.114:3300/0 conn(0x7fe4640ce890 msgr2=0x7fe4640b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 --2- 172.21.15.104:0/2049080028 >> v2:172.21.15.114:3300/0 conn(0x7fe4640ce890 0x7fe4640b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 -- 172.21.15.104:0/2049080028 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640b5740 con 0x7fe46415a420 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46a15b700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe46415a420 0x7fe4640b1660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe46408abf0 tx=0x7fe454009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab6249eed6763cb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe469159700 1 -- 172.21.15.104:0/2049080028 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45400b870 con 0x7fe46415a420 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe469159700 1 -- 172.21.15.104:0/2049080028 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45400b9d0 con 0x7fe46415a420 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe469159700 1 -- 172.21.15.104:0/2049080028 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45400bd60 con 0x7fe46415a420 2023-06-10T18:44:31.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46c3bf700 1 -- 172.21.15.104:0/2049080028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe46415a420 msgr2=0x7fe4640b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.693+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe46415a420 0x7fe4640b1660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe46408abf0 tx=0x7fe454009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 -- 172.21.15.104:0/2049080028 shutdown_connections 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe46415a420 0x7fe4640b1660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/2049080028 >> v2:172.21.15.114:3300/0 conn(0x7fe4640ce890 0x7fe4640b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/2049080028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4640b4ff0 0x7fe4640ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 -- 172.21.15.104:0/2049080028 >> 172.21.15.104:0/2049080028 conn(0x7fe464089fd0 msgr2=0x7fe4640d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:31.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 -- 172.21.15.104:0/2049080028 shutdown_connections 2023-06-10T18:44:31.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.694+0000 7fe46c3bf700 1 -- 172.21.15.104:0/2049080028 wait complete. 2023-06-10T18:44:31.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.695+0000 7fe46c3bf700 1 Processor -- start 2023-06-10T18:44:31.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.695+0000 7fe46c3bf700 1 -- start start 2023-06-10T18:44:31.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.695+0000 7fe46c3bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.695+0000 7fe46c3bf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 0x7fe4640c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.695+0000 7fe46c3bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe46415a420 0x7fe4641f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46c3bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640ca7c0 con 0x7fe4640b4ff0 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46c3bf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640c4510 con 0x7fe4640ce890 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46c3bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640c47f0 con 0x7fe46415a420 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54658/0 (socket says 172.21.15.104:54658) 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 -- 172.21.15.104:0/3062886951 learned_addr learned my addr 172.21.15.104:0/3062886951 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:31.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a95c700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe46415a420 0x7fe4641f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46995a700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 0x7fe4640c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 -- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe46415a420 msgr2=0x7fe4641f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe46415a420 0x7fe4641f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 -- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 msgr2=0x7fe4640c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 0x7fe4640c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 -- 172.21.15.104:0/3062886951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe454009550 con 0x7fe4640b4ff0 2023-06-10T18:44:31.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46995a700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 0x7fe4640c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46a15b700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe454000c00 tx=0x7fe454004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454004540 con 0x7fe4640b4ff0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540036a0 con 0x7fe4640b4ff0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4640c4ad0 con 0x7fe4640b4ff0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454003a50 con 0x7fe4640b4ff0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.696+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe464155ef0 con 0x7fe4640b4ff0 2023-06-10T18:44:31.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.697+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe464074050 con 0x7fe4640b4ff0 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.698+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe454021070 con 0x7fe4640b4ff0 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.698+0000 7fe45affd700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 0x7fe44806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.698+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe45408ae10 con 0x7fe4640b4ff0 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.698+0000 7fe46995a700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 0x7fe44806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.699+0000 7fe46995a700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 0x7fe44806d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe460006fd0 tx=0x7fe460008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:31.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.702+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe454055c60 con 0x7fe4640b4ff0 2023-06-10T18:44:31.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.931+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4640cdff0 con 0x7fe4640b4ff0 2023-06-10T18:44:31.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.932+0000 7fe45affd700 1 -- 172.21.15.104:0/3062886951 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7fe454015090 con 0x7fe4640b4ff0 2023-06-10T18:44:31.938 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:31.938 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:44:31.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 msgr2=0x7fe44806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 0x7fe44806d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe460006fd0 tx=0x7fe460008040 comp rx=0 tx=0).stop 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 msgr2=0x7fe4640caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe454000c00 tx=0x7fe454004750 comp rx=0 tx=0).stop 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 shutdown_connections 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe46415a420 0x7fe4641f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4640b4ff0 0x7fe4640caeb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe44806ace0 0x7fe44806d190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 --2- 172.21.15.104:0/3062886951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4640ce890 0x7fe4640c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:31.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 >> 172.21.15.104:0/3062886951 conn(0x7fe464089fd0 msgr2=0x7fe464158490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:31.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 shutdown_connections 2023-06-10T18:44:31.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:31.934+0000 7fe46c3bf700 1 -- 172.21.15.104:0/3062886951 wait complete. 2023-06-10T18:44:31.942 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:44:31.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.065+0000 7fd9b06d6700 1 Processor -- start 2023-06-10T18:44:32.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.065+0000 7fd9b06d6700 1 -- start start 2023-06-10T18:44:32.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd9a815a0b0 0x7fd9a80717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd9a8070a20 0x7fd9a8070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a80713e0 con 0x7fd9a8070a20 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a806b2e0 con 0x7fd9a815a0b0 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9b06d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a815e150 con 0x7fd9a80c93e0 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd9a8070a20 0x7fd9a8070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd9a8070a20 0x7fd9a8070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54668/0 (socket says 172.21.15.104:54668) 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 -- 172.21.15.104:0/3213758518 learned_addr learned my addr 172.21.15.104:0/3213758518 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9ae472700 1 --2- 172.21.15.104:0/3213758518 >> v2:172.21.15.114:3300/0 conn(0x7fd9a815a0b0 0x7fd9a80717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9adc71700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 -- 172.21.15.104:0/3213758518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 msgr2=0x7fd9a815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a815e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 -- 172.21.15.104:0/3213758518 >> v2:172.21.15.114:3300/0 conn(0x7fd9a815a0b0 msgr2=0x7fd9a80717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 --2- 172.21.15.104:0/3213758518 >> v2:172.21.15.114:3300/0 conn(0x7fd9a815a0b0 0x7fd9a80717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 -- 172.21.15.104:0/3213758518 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a815db40 con 0x7fd9a8070a20 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9aec73700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a8070ea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8066150 tx=0x7fd9a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a23c4e39c1a207 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.066+0000 7fd9ad470700 1 -- 172.21.15.104:0/3213758518 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000e070 con 0x7fd9a8070a20 2023-06-10T18:44:32.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9ad470700 1 -- 172.21.15.104:0/3213758518 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000b9c0 con 0x7fd9a8070a20 2023-06-10T18:44:32.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9ad470700 1 -- 172.21.15.104:0/3213758518 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000bd70 con 0x7fd9a8070a20 2023-06-10T18:44:32.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 -- 172.21.15.104:0/3213758518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 msgr2=0x7fd9a8070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a8070ea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8066150 tx=0x7fd9a000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:32.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 -- 172.21.15.104:0/3213758518 shutdown_connections 2023-06-10T18:44:32.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a8070ea0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/3213758518 >> v2:172.21.15.114:3300/0 conn(0x7fd9a815a0b0 0x7fd9a80717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/3213758518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a815e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 -- 172.21.15.104:0/3213758518 >> 172.21.15.104:0/3213758518 conn(0x7fd9a80b1610 msgr2=0x7fd9a80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 -- 172.21.15.104:0/3213758518 shutdown_connections 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.067+0000 7fd9b06d6700 1 -- 172.21.15.104:0/3213758518 wait complete. 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.068+0000 7fd9b06d6700 1 Processor -- start 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.068+0000 7fd9b06d6700 1 -- start start 2023-06-10T18:44:32.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a8158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 0x7fd9a81e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a81e47e0 con 0x7fd9a8070a20 2023-06-10T18:44:32.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8156ce0 con 0x7fd9a815a0b0 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9b06d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8156f80 con 0x7fd9a80c93e0 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54672/0 (socket says 172.21.15.104:54672) 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 -- 172.21.15.104:0/2493626312 learned_addr learned my addr 172.21.15.104:0/2493626312 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9adc71700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a8158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9aec73700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 0x7fd9a81e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 -- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 msgr2=0x7fd9a8158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a8158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 -- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 msgr2=0x7fd9a81e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 0x7fd9a81e42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9ae472700 1 -- 172.21.15.104:0/2493626312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0009b30 con 0x7fd9a8070a20 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9adc71700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a8158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.069+0000 7fd9aec73700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 0x7fd9a81e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:32.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd9ae472700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d800 tx=0x7fd9a400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40125f0 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4012750 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4012b00 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a8157260 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.070+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a81f3d10 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.071+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a81574a0 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.071+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd9a4012c60 con 0x7fd9a8070a20 2023-06-10T18:44:32.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.071+0000 7fd99effd700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 0x7fd98c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.072+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd9a4089a00 con 0x7fd9a8070a20 2023-06-10T18:44:32.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.072+0000 7fd9adc71700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 0x7fd98c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.072+0000 7fd9adc71700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 0x7fd98c06d190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80c5900 tx=0x7fd998008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.076+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd9a4054850 con 0x7fd9a8070a20 2023-06-10T18:44:32.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.261+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9a8074050 con 0x7fd9a8070a20 2023-06-10T18:44:32.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.263+0000 7fd99effd700 1 -- 172.21.15.104:0/2493626312 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7fd9a4057e70 con 0x7fd9a8070a20 2023-06-10T18:44:32.268 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:32.269 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 msgr2=0x7fd98c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 0x7fd98c06d190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80c5900 tx=0x7fd998008040 comp rx=0 tx=0).stop 2023-06-10T18:44:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 msgr2=0x7fd9a81e3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d800 tx=0x7fd9a400dbc0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 shutdown_connections 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd98c06ace0 0x7fd98c06d190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9a815a0b0 0x7fd9a81e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9a8070a20 0x7fd9a81e3930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 --2- 172.21.15.104:0/2493626312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9a80c93e0 0x7fd9a8158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 >> 172.21.15.104:0/2493626312 conn(0x7fd9a80b1610 msgr2=0x7fd9a80cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:32.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 shutdown_connections 2023-06-10T18:44:32.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.265+0000 7fd9b06d6700 1 -- 172.21.15.104:0/2493626312 wait complete. 2023-06-10T18:44:32.283 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:44:32.284 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:44:32.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.398+0000 7f6635448700 1 Processor -- start 2023-06-10T18:44:32.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.398+0000 7f6635448700 1 -- start start 2023-06-10T18:44:32.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f6635448700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f66300d0c00 0x7f663006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f6635448700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663006bfd0 con 0x7f663006b820 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f6635448700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663006c2b0 con 0x7f66300b1e30 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f6635448700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66300ccf70 con 0x7f66300d0c00 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f662f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f663006b820 0x7f66300cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52510/0 (socket says 172.21.15.104:52510) 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f662f7fe700 1 -- 172.21.15.104:0/3688691503 learned_addr learned my addr 172.21.15.104:0/3688691503 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f662e7fc700 1 --2- 172.21.15.104:0/3688691503 >> v2:172.21.15.114:3300/0 conn(0x7f66300d0c00 0x7f663006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f662dffb700 1 -- 172.21.15.104:0/3688691503 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725472375 0 0) 0x7f663006bfd0 con 0x7f663006b820 2023-06-10T18:44:32.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.399+0000 7f662dffb700 1 -- 172.21.15.104:0/3688691503 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66180035b0 con 0x7f663006b820 2023-06-10T18:44:32.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662e7fc700 1 -- 172.21.15.104:0/3688691503 >> v1:172.21.15.114:6789/0 conn(0x7f66300b1e30 legacy=0x7f66300d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662e7fc700 1 -- 172.21.15.104:0/3688691503 >> v1:172.21.15.104:6789/0 conn(0x7f663006b820 legacy=0x7f66300cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662e7fc700 1 -- 172.21.15.104:0/3688691503 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300cd250 con 0x7f66300d0c00 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662e7fc700 1 --2- 172.21.15.104:0/3688691503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66300d0c00 0x7f663006b2e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6620009840 tx=0x7f6620009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90ffd34cd315db89 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662dffb700 1 -- 172.21.15.104:0/3688691503 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620011830 con 0x7f66300d0c00 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662dffb700 1 -- 172.21.15.104:0/3688691503 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6620011990 con 0x7f66300d0c00 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f662dffb700 1 -- 172.21.15.104:0/3688691503 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620011d20 con 0x7f66300d0c00 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f6635448700 1 -- 172.21.15.104:0/3688691503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66300d0c00 msgr2=0x7f663006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.400+0000 7f6635448700 1 --2- 172.21.15.104:0/3688691503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66300d0c00 0x7f663006b2e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6620009840 tx=0x7f6620009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:32.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.401+0000 7f6635448700 1 -- 172.21.15.104:0/3688691503 shutdown_connections 2023-06-10T18:44:32.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.401+0000 7f6635448700 1 --2- 172.21.15.104:0/3688691503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66300d0c00 0x7f663006b2e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.401+0000 7f6635448700 1 -- 172.21.15.104:0/3688691503 >> 172.21.15.104:0/3688691503 conn(0x7f6630071ec0 msgr2=0x7f663009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:32.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.401+0000 7f6635448700 1 -- 172.21.15.104:0/3688691503 shutdown_connections 2023-06-10T18:44:32.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.401+0000 7f6635448700 1 -- 172.21.15.104:0/3688691503 wait complete. 2023-06-10T18:44:32.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 Processor -- start 2023-06-10T18:44:32.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 -- start start 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f663006b820 0x7f663014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66300b1e30 0x7f66300c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b7c40 con 0x7f66300d0c00 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b7f20 con 0x7f663006b820 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.402+0000 7f6635448700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b8200 con 0x7f66300b1e30 2023-06-10T18:44:32.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54688/0 (socket says 172.21.15.104:54688) 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 -- 172.21.15.104:0/1101038960 learned_addr learned my addr 172.21.15.104:0/1101038960 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662e7fc700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66300b1e30 0x7f66300c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662effd700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f663006b820 0x7f663014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 -- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66300b1e30 msgr2=0x7f66300c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66300b1e30 0x7f66300c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 -- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f663006b820 msgr2=0x7f663014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f663006b820 0x7f663014fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 -- 172.21.15.104:0/1101038960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200094a0 con 0x7f66300d0c00 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f662f7fe700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f662800cb40 tx=0x7f662800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662800a9e0 con 0x7f66300d0c00 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662800ab40 con 0x7f66300d0c00 2023-06-10T18:44:32.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66300b84e0 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628012400 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.403+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66300c8e90 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.404+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66300d1860 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.405+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f662801b3b0 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.405+0000 7f66177fe700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 0x7f661c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.405+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6628089250 con 0x7f66300d0c00 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.405+0000 7f662effd700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 0x7f661c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.406+0000 7f662effd700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 0x7f661c06d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6624007fd0 tx=0x7f6624009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.409+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6628054150 con 0x7f66300d0c00 2023-06-10T18:44:32.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.596+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66300b8090 con 0x7f661c06ace0 2023-06-10T18:44:32.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.607+0000 7f66177fe700 1 -- 172.21.15.104:0/1101038960 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367290 (secure 0 0 0) 0x7f66300b8090 con 0x7f661c06ace0 2023-06-10T18:44:32.617 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 msgr2=0x7f661c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 0x7f661c06d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6624007fd0 tx=0x7f6624009040 comp rx=0 tx=0).stop 2023-06-10T18:44:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 msgr2=0x7f66300c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f662800cb40 tx=0x7f662800ce50 comp rx=0 tx=0).stop 2023-06-10T18:44:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 shutdown_connections 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f661c06ace0 0x7f661c06d190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f663006b820 0x7f663014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66300d0c00 0x7f66300c77e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 --2- 172.21.15.104:0/1101038960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66300b1e30 0x7f66300c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 >> 172.21.15.104:0/1101038960 conn(0x7f6630071ec0 msgr2=0x7f6630072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 shutdown_connections 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.611+0000 7f6635448700 1 -- 172.21.15.104:0/1101038960 wait complete. 2023-06-10T18:44:32.620 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:44:32.628 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-06-10T18:44:31.353446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":3480,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":3563520,"data_stored":2145052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423396+0000","last_change":"2023-06-10T18:44:08.974320+0000","last_active":"2023-06-10T18:44:13.423396+0000","last_peered":"2023-06-10T18:44:13.423396+0000","last_clean":"2023-06-10T18:44:13.423396+0000","last_became_active":"2023-06-10T18:44:08.973998+0000","last_became_peered":"2023-06-10T18:44:08.973998+0000","last_unstale":"2023-06-10T18:44:13.423396+0000","last_undegraded":"2023-06-10T18:44:13.423396+0000","last_fullsized":"2023-06-10T18:44:13.423396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:29.774977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393150+0000","last_change":"2023-06-10T18:44:08.986524+0000","last_active":"2023-06-10T18:44:13.393150+0000","last_peered":"2023-06-10T18:44:13.393150+0000","last_clean":"2023-06-10T18:44:13.393150+0000","last_became_active":"2023-06-10T18:44:08.986238+0000","last_became_peered":"2023-06-10T18:44:08.986238+0000","last_unstale":"2023-06-10T18:44:13.393150+0000","last_undegraded":"2023-06-10T18:44:13.393150+0000","last_fullsized":"2023-06-10T18:44:13.393150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423472+0000","last_change":"2023-06-10T18:44:08.991803+0000","last_active":"2023-06-10T18:44:13.423472+0000","last_peered":"2023-06-10T18:44:13.423472+0000","last_clean":"2023-06-10T18:44:13.423472+0000","last_became_active":"2023-06-10T18:44:08.986777+0000","last_became_peered":"2023-06-10T18:44:08.986777+0000","last_unstale":"2023-06-10T18:44:13.423472+0000","last_undegraded":"2023-06-10T18:44:13.423472+0000","last_fullsized":"2023-06-10T18:44:13.423472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423103+0000","last_change":"2023-06-10T18:44:08.984677+0000","last_active":"2023-06-10T18:44:13.423103+0000","last_peered":"2023-06-10T18:44:13.423103+0000","last_clean":"2023-06-10T18:44:13.423103+0000","last_became_active":"2023-06-10T18:44:08.984368+0000","last_became_peered":"2023-06-10T18:44:08.984368+0000","last_unstale":"2023-06-10T18:44:13.423103+0000","last_undegraded":"2023-06-10T18:44:13.423103+0000","last_fullsized":"2023-06-10T18:44:13.423103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:21.561425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423951+0000","last_change":"2023-06-10T18:44:08.988022+0000","last_active":"2023-06-10T18:44:13.423951+0000","last_peered":"2023-06-10T18:44:13.423951+0000","last_clean":"2023-06-10T18:44:13.423951+0000","last_became_active":"2023-06-10T18:44:08.983186+0000","last_became_peered":"2023-06-10T18:44:08.983186+0000","last_unstale":"2023-06-10T18:44:13.423951+0000","last_undegraded":"2023-06-10T18:44:13.423951+0000","last_fullsized":"2023-06-10T18:44:13.423951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:24.256200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423368+0000","last_change":"2023-06-10T18:44:08.986749+0000","last_active":"2023-06-10T18:44:13.423368+0000","last_peered":"2023-06-10T18:44:13.423368+0000","last_clean":"2023-06-10T18:44:13.423368+0000","last_became_active":"2023-06-10T18:44:08.986456+0000","last_became_peered":"2023-06-10T18:44:08.986456+0000","last_unstale":"2023-06-10T18:44:13.423368+0000","last_undegraded":"2023-06-10T18:44:13.423368+0000","last_fullsized":"2023-06-10T18:44:13.423368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394293+0000","last_change":"2023-06-10T18:44:08.992850+0000","last_active":"2023-06-10T18:44:13.394293+0000","last_peered":"2023-06-10T18:44:13.394293+0000","last_clean":"2023-06-10T18:44:13.394293+0000","last_became_active":"2023-06-10T18:44:08.992572+0000","last_became_peered":"2023-06-10T18:44:08.992572+0000","last_unstale":"2023-06-10T18:44:13.394293+0000","last_undegraded":"2023-06-10T18:44:13.394293+0000","last_fullsized":"2023-06-10T18:44:13.394293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392634+0000","last_change":"2023-06-10T18:44:08.979428+0000","last_active":"2023-06-10T18:44:13.392634+0000","last_peered":"2023-06-10T18:44:13.392634+0000","last_clean":"2023-06-10T18:44:13.392634+0000","last_became_active":"2023-06-10T18:44:08.979157+0000","last_became_peered":"2023-06-10T18:44:08.979157+0000","last_unstale":"2023-06-10T18:44:13.392634+0000","last_undegraded":"2023-06-10T18:44:13.392634+0000","last_fullsized":"2023-06-10T18:44:13.392634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:24.114600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423251+0000","last_change":"2023-06-10T18:44:08.984436+0000","last_active":"2023-06-10T18:44:13.423251+0000","last_peered":"2023-06-10T18:44:13.423251+0000","last_clean":"2023-06-10T18:44:13.423251+0000","last_became_active":"2023-06-10T18:44:08.984021+0000","last_became_peered":"2023-06-10T18:44:08.984021+0000","last_unstale":"2023-06-10T18:44:13.423251+0000","last_undegraded":"2023-06-10T18:44:13.423251+0000","last_fullsized":"2023-06-10T18:44:13.423251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:23:54.181578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393491+0000","last_change":"2023-06-10T18:44:08.984325+0000","last_active":"2023-06-10T18:44:13.393491+0000","last_peered":"2023-06-10T18:44:13.393491+0000","last_clean":"2023-06-10T18:44:13.393491+0000","last_became_active":"2023-06-10T18:44:08.983999+0000","last_became_peered":"2023-06-10T18:44:08.983999+0000","last_unstale":"2023-06-10T18:44:13.393491+0000","last_undegraded":"2023-06-10T18:44:13.393491+0000","last_fullsized":"2023-06-10T18:44:13.393491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423627+0000","last_change":"2023-06-10T18:44:08.991961+0000","last_active":"2023-06-10T18:44:13.423627+0000","last_peered":"2023-06-10T18:44:13.423627+0000","last_clean":"2023-06-10T18:44:13.423627+0000","last_became_active":"2023-06-10T18:44:08.991527+0000","last_became_peered":"2023-06-10T18:44:08.991527+0000","last_unstale":"2023-06-10T18:44:13.423627+0000","last_undegraded":"2023-06-10T18:44:13.423627+0000","last_fullsized":"2023-06-10T18:44:13.423627+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423011+0000","last_change":"2023-06-10T18:44:08.995250+0000","last_active":"2023-06-10T18:44:13.423011+0000","last_peered":"2023-06-10T18:44:13.423011+0000","last_clean":"2023-06-10T18:44:13.423011+0000","last_became_active":"2023-06-10T18:44:08.994854+0000","last_became_peered":"2023-06-10T18:44:08.994854+0000","last_unstale":"2023-06-10T18:44:13.423011+0000","last_undegraded":"2023-06-10T18:44:13.423011+0000","last_fullsized":"2023-06-10T18:44:13.423011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:13:54.715925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422801+0000","last_change":"2023-06-10T18:44:08.973692+0000","last_active":"2023-06-10T18:44:13.422801+0000","last_peered":"2023-06-10T18:44:13.422801+0000","last_clean":"2023-06-10T18:44:13.422801+0000","last_became_active":"2023-06-10T18:44:08.973417+0000","last_became_peered":"2023-06-10T18:44:08.973417+0000","last_unstale":"2023-06-10T18:44:13.422801+0000","last_undegraded":"2023-06-10T18:44:13.422801+0000","last_fullsized":"2023-06-10T18:44:13.422801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:27:20.138188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392896+0000","last_change":"2023-06-10T18:44:08.976711+0000","last_active":"2023-06-10T18:44:13.392896+0000","last_peered":"2023-06-10T18:44:13.392896+0000","last_clean":"2023-06-10T18:44:13.392896+0000","last_became_active":"2023-06-10T18:44:08.976415+0000","last_became_peered":"2023-06-10T18:44:08.976415+0000","last_unstale":"2023-06-10T18:44:13.392896+0000","last_undegraded":"2023-06-10T18:44:13.392896+0000","last_fullsized":"2023-06-10T18:44:13.392896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394066+0000","last_change":"2023-06-10T18:44:08.992788+0000","last_active":"2023-06-10T18:44:13.394066+0000","last_peered":"2023-06-10T18:44:13.394066+0000","last_clean":"2023-06-10T18:44:13.394066+0000","last_became_active":"2023-06-10T18:44:08.992335+0000","last_became_peered":"2023-06-10T18:44:08.992335+0000","last_unstale":"2023-06-10T18:44:13.394066+0000","last_undegraded":"2023-06-10T18:44:13.394066+0000","last_fullsized":"2023-06-10T18:44:13.394066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423080+0000","last_change":"2023-06-10T18:44:08.993080+0000","last_active":"2023-06-10T18:44:13.423080+0000","last_peered":"2023-06-10T18:44:13.423080+0000","last_clean":"2023-06-10T18:44:13.423080+0000","last_became_active":"2023-06-10T18:44:08.992794+0000","last_became_peered":"2023-06-10T18:44:08.992794+0000","last_unstale":"2023-06-10T18:44:13.423080+0000","last_undegraded":"2023-06-10T18:44:13.423080+0000","last_fullsized":"2023-06-10T18:44:13.423080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423378+0000","last_change":"2023-06-10T18:44:08.984915+0000","last_active":"2023-06-10T18:44:13.423378+0000","last_peered":"2023-06-10T18:44:13.423378+0000","last_clean":"2023-06-10T18:44:13.423378+0000","last_became_active":"2023-06-10T18:44:08.984639+0000","last_became_peered":"2023-06-10T18:44:08.984639+0000","last_unstale":"2023-06-10T18:44:13.423378+0000","last_undegraded":"2023-06-10T18:44:13.423378+0000","last_fullsized":"2023-06-10T18:44:13.423378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:53.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424069+0000","last_change":"2023-06-10T18:44:08.989447+0000","last_active":"2023-06-10T18:44:13.424069+0000","last_peered":"2023-06-10T18:44:13.424069+0000","last_clean":"2023-06-10T18:44:13.424069+0000","last_became_active":"2023-06-10T18:44:08.987688+0000","last_became_peered":"2023-06-10T18:44:08.987688+0000","last_unstale":"2023-06-10T18:44:13.424069+0000","last_undegraded":"2023-06-10T18:44:13.424069+0000","last_fullsized":"2023-06-10T18:44:13.424069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422601+0000","last_change":"2023-06-10T18:44:08.984545+0000","last_active":"2023-06-10T18:44:13.422601+0000","last_peered":"2023-06-10T18:44:13.422601+0000","last_clean":"2023-06-10T18:44:13.422601+0000","last_became_active":"2023-06-10T18:44:08.983857+0000","last_became_peered":"2023-06-10T18:44:08.983857+0000","last_unstale":"2023-06-10T18:44:13.422601+0000","last_undegraded":"2023-06-10T18:44:13.422601+0000","last_fullsized":"2023-06-10T18:44:13.422601+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:46.973658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423818+0000","last_change":"2023-06-10T18:44:08.993840+0000","last_active":"2023-06-10T18:44:13.423818+0000","last_peered":"2023-06-10T18:44:13.423818+0000","last_clean":"2023-06-10T18:44:13.423818+0000","last_became_active":"2023-06-10T18:44:08.993524+0000","last_became_peered":"2023-06-10T18:44:08.993524+0000","last_unstale":"2023-06-10T18:44:13.423818+0000","last_undegraded":"2023-06-10T18:44:13.423818+0000","last_fullsized":"2023-06-10T18:44:13.423818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393980+0000","last_change":"2023-06-10T18:44:08.994356+0000","last_active":"2023-06-10T18:44:13.393980+0000","last_peered":"2023-06-10T18:44:13.393980+0000","last_clean":"2023-06-10T18:44:13.393980+0000","last_became_active":"2023-06-10T18:44:08.994055+0000","last_became_peered":"2023-06-10T18:44:08.994055+0000","last_unstale":"2023-06-10T18:44:13.393980+0000","last_undegraded":"2023-06-10T18:44:13.393980+0000","last_fullsized":"2023-06-10T18:44:13.393980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423446+0000","last_change":"2023-06-10T18:44:08.988865+0000","last_active":"2023-06-10T18:44:13.423446+0000","last_peered":"2023-06-10T18:44:13.423446+0000","last_clean":"2023-06-10T18:44:13.423446+0000","last_became_active":"2023-06-10T18:44:08.988631+0000","last_became_peered":"2023-06-10T18:44:08.988631+0000","last_unstale":"2023-06-10T18:44:13.423446+0000","last_undegraded":"2023-06-10T18:44:13.423446+0000","last_fullsized":"2023-06-10T18:44:13.423446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:18.367120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424783+0000","last_change":"2023-06-10T18:44:08.985010+0000","last_active":"2023-06-10T18:44:13.424783+0000","last_peered":"2023-06-10T18:44:13.424783+0000","last_clean":"2023-06-10T18:44:13.424783+0000","last_became_active":"2023-06-10T18:44:08.984770+0000","last_became_peered":"2023-06-10T18:44:08.984770+0000","last_unstale":"2023-06-10T18:44:13.424783+0000","last_undegraded":"2023-06-10T18:44:13.424783+0000","last_fullsized":"2023-06-10T18:44:13.424783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394283+0000","last_change":"2023-06-10T18:44:08.994280+0000","last_active":"2023-06-10T18:44:13.394283+0000","last_peered":"2023-06-10T18:44:13.394283+0000","last_clean":"2023-06-10T18:44:13.394283+0000","last_became_active":"2023-06-10T18:44:08.993996+0000","last_became_peered":"2023-06-10T18:44:08.993996+0000","last_unstale":"2023-06-10T18:44:13.394283+0000","last_undegraded":"2023-06-10T18:44:13.394283+0000","last_fullsized":"2023-06-10T18:44:13.394283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394507+0000","last_change":"2023-06-10T18:44:08.988231+0000","last_active":"2023-06-10T18:44:13.394507+0000","last_peered":"2023-06-10T18:44:13.394507+0000","last_clean":"2023-06-10T18:44:13.394507+0000","last_became_active":"2023-06-10T18:44:08.987866+0000","last_became_peered":"2023-06-10T18:44:08.987866+0000","last_unstale":"2023-06-10T18:44:13.394507+0000","last_undegraded":"2023-06-10T18:44:13.394507+0000","last_fullsized":"2023-06-10T18:44:13.394507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394552+0000","last_change":"2023-06-10T18:44:08.984357+0000","last_active":"2023-06-10T18:44:13.394552+0000","last_peered":"2023-06-10T18:44:13.394552+0000","last_clean":"2023-06-10T18:44:13.394552+0000","last_became_active":"2023-06-10T18:44:08.984034+0000","last_became_peered":"2023-06-10T18:44:08.984034+0000","last_unstale":"2023-06-10T18:44:13.394552+0000","last_undegraded":"2023-06-10T18:44:13.394552+0000","last_fullsized":"2023-06-10T18:44:13.394552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393451+0000","last_change":"2023-06-10T18:44:08.987489+0000","last_active":"2023-06-10T18:44:13.393451+0000","last_peered":"2023-06-10T18:44:13.393451+0000","last_clean":"2023-06-10T18:44:13.393451+0000","last_became_active":"2023-06-10T18:44:08.985804+0000","last_became_peered":"2023-06-10T18:44:08.985804+0000","last_unstale":"2023-06-10T18:44:13.393451+0000","last_undegraded":"2023-06-10T18:44:13.393451+0000","last_fullsized":"2023-06-10T18:44:13.393451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424186+0000","last_change":"2023-06-10T18:44:13.001999+0000","last_active":"2023-06-10T18:44:13.424186+0000","last_peered":"2023-06-10T18:44:13.424186+0000","last_clean":"2023-06-10T18:44:13.424186+0000","last_became_active":"2023-06-10T18:44:13.001637+0000","last_became_peered":"2023-06-10T18:44:13.001637+0000","last_unstale":"2023-06-10T18:44:13.424186+0000","last_undegraded":"2023-06-10T18:44:13.424186+0000","last_fullsized":"2023-06-10T18:44:13.424186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423197+0000","last_change":"2023-06-10T18:44:08.575638+0000","last_active":"2023-06-10T18:44:13.423197+0000","last_peered":"2023-06-10T18:44:13.423197+0000","last_clean":"2023-06-10T18:44:13.423197+0000","last_became_active":"2023-06-10T18:44:08.575345+0000","last_became_peered":"2023-06-10T18:44:08.575345+0000","last_unstale":"2023-06-10T18:44:13.423197+0000","last_undegraded":"2023-06-10T18:44:13.423197+0000","last_fullsized":"2023-06-10T18:44:13.423197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:11.022238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425003+0000","last_change":"2023-06-10T18:44:08.986591+0000","last_active":"2023-06-10T18:44:13.425003+0000","last_peered":"2023-06-10T18:44:13.425003+0000","last_clean":"2023-06-10T18:44:13.425003+0000","last_became_active":"2023-06-10T18:44:08.986320+0000","last_became_peered":"2023-06-10T18:44:08.986320+0000","last_unstale":"2023-06-10T18:44:13.425003+0000","last_undegraded":"2023-06-10T18:44:13.425003+0000","last_fullsized":"2023-06-10T18:44:13.425003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424063+0000","last_change":"2023-06-10T18:44:13.002454+0000","last_active":"2023-06-10T18:44:13.424063+0000","last_peered":"2023-06-10T18:44:13.424063+0000","last_clean":"2023-06-10T18:44:13.424063+0000","last_became_active":"2023-06-10T18:44:13.002101+0000","last_became_peered":"2023-06-10T18:44:13.002101+0000","last_unstale":"2023-06-10T18:44:13.424063+0000","last_undegraded":"2023-06-10T18:44:13.424063+0000","last_fullsized":"2023-06-10T18:44:13.424063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:38.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393702+0000","last_change":"2023-06-10T18:44:08.581341+0000","last_active":"2023-06-10T18:44:13.393702+0000","last_peered":"2023-06-10T18:44:13.393702+0000","last_clean":"2023-06-10T18:44:13.393702+0000","last_became_active":"2023-06-10T18:44:08.580977+0000","last_became_peered":"2023-06-10T18:44:08.580977+0000","last_unstale":"2023-06-10T18:44:13.393702+0000","last_undegraded":"2023-06-10T18:44:13.393702+0000","last_fullsized":"2023-06-10T18:44:13.393702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423101+0000","last_change":"2023-06-10T18:44:08.990386+0000","last_active":"2023-06-10T18:44:13.423101+0000","last_peered":"2023-06-10T18:44:13.423101+0000","last_clean":"2023-06-10T18:44:13.423101+0000","last_became_active":"2023-06-10T18:44:08.989816+0000","last_became_peered":"2023-06-10T18:44:08.989816+0000","last_unstale":"2023-06-10T18:44:13.423101+0000","last_undegraded":"2023-06-10T18:44:13.423101+0000","last_fullsized":"2023-06-10T18:44:13.423101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:27.419215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422737+0000","last_change":"2023-06-10T18:44:13.011429+0000","last_active":"2023-06-10T18:44:13.422737+0000","last_peered":"2023-06-10T18:44:13.422737+0000","last_clean":"2023-06-10T18:44:13.422737+0000","last_became_active":"2023-06-10T18:44:13.010928+0000","last_became_peered":"2023-06-10T18:44:13.010928+0000","last_unstale":"2023-06-10T18:44:13.422737+0000","last_undegraded":"2023-06-10T18:44:13.422737+0000","last_fullsized":"2023-06-10T18:44:13.422737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:25:46.759866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425044+0000","last_change":"2023-06-10T18:44:08.583484+0000","last_active":"2023-06-10T18:44:13.425044+0000","last_peered":"2023-06-10T18:44:13.425044+0000","last_clean":"2023-06-10T18:44:13.425044+0000","last_became_active":"2023-06-10T18:44:08.583197+0000","last_became_peered":"2023-06-10T18:44:08.583197+0000","last_unstale":"2023-06-10T18:44:13.425044+0000","last_undegraded":"2023-06-10T18:44:13.425044+0000","last_fullsized":"2023-06-10T18:44:13.425044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:33.286044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424162+0000","last_change":"2023-06-10T18:44:08.983580+0000","last_active":"2023-06-10T18:44:13.424162+0000","last_peered":"2023-06-10T18:44:13.424162+0000","last_clean":"2023-06-10T18:44:13.424162+0000","last_became_active":"2023-06-10T18:44:08.983254+0000","last_became_peered":"2023-06-10T18:44:08.983254+0000","last_unstale":"2023-06-10T18:44:13.424162+0000","last_undegraded":"2023-06-10T18:44:13.424162+0000","last_fullsized":"2023-06-10T18:44:13.424162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:53:36.396900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423826+0000","last_change":"2023-06-10T18:44:12.997904+0000","last_active":"2023-06-10T18:44:13.423826+0000","last_peered":"2023-06-10T18:44:13.423826+0000","last_clean":"2023-06-10T18:44:13.423826+0000","last_became_active":"2023-06-10T18:44:12.997502+0000","last_became_peered":"2023-06-10T18:44:12.997502+0000","last_unstale":"2023-06-10T18:44:13.423826+0000","last_undegraded":"2023-06-10T18:44:13.423826+0000","last_fullsized":"2023-06-10T18:44:13.423826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:17.892337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423483+0000","last_change":"2023-06-10T18:44:08.575792+0000","last_active":"2023-06-10T18:44:13.423483+0000","last_peered":"2023-06-10T18:44:13.423483+0000","last_clean":"2023-06-10T18:44:13.423483+0000","last_became_active":"2023-06-10T18:44:08.573780+0000","last_became_peered":"2023-06-10T18:44:08.573780+0000","last_unstale":"2023-06-10T18:44:13.423483+0000","last_undegraded":"2023-06-10T18:44:13.423483+0000","last_fullsized":"2023-06-10T18:44:13.423483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:19.709371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424910+0000","last_change":"2023-06-10T18:44:08.986631+0000","last_active":"2023-06-10T18:44:13.424910+0000","last_peered":"2023-06-10T18:44:13.424910+0000","last_clean":"2023-06-10T18:44:13.424910+0000","last_became_active":"2023-06-10T18:44:08.986357+0000","last_became_peered":"2023-06-10T18:44:08.986357+0000","last_unstale":"2023-06-10T18:44:13.424910+0000","last_undegraded":"2023-06-10T18:44:13.424910+0000","last_fullsized":"2023-06-10T18:44:13.424910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393507+0000","last_change":"2023-06-10T18:44:13.002734+0000","last_active":"2023-06-10T18:44:13.393507+0000","last_peered":"2023-06-10T18:44:13.393507+0000","last_clean":"2023-06-10T18:44:13.393507+0000","last_became_active":"2023-06-10T18:44:13.002226+0000","last_became_peered":"2023-06-10T18:44:13.002226+0000","last_unstale":"2023-06-10T18:44:13.393507+0000","last_undegraded":"2023-06-10T18:44:13.393507+0000","last_fullsized":"2023-06-10T18:44:13.393507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:11.264615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393372+0000","last_change":"2023-06-10T18:44:08.571167+0000","last_active":"2023-06-10T18:44:13.393372+0000","last_peered":"2023-06-10T18:44:13.393372+0000","last_clean":"2023-06-10T18:44:13.393372+0000","last_became_active":"2023-06-10T18:44:08.570885+0000","last_became_peered":"2023-06-10T18:44:08.570885+0000","last_unstale":"2023-06-10T18:44:13.393372+0000","last_undegraded":"2023-06-10T18:44:13.393372+0000","last_fullsized":"2023-06-10T18:44:13.393372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.984523+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.984265+0000","last_became_peered":"2023-06-10T18:44:08.984265+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425522+0000","last_change":"2023-06-10T18:44:13.005254+0000","last_active":"2023-06-10T18:44:13.425522+0000","last_peered":"2023-06-10T18:44:13.425522+0000","last_clean":"2023-06-10T18:44:13.425522+0000","last_became_active":"2023-06-10T18:44:13.004908+0000","last_became_peered":"2023-06-10T18:44:13.004908+0000","last_unstale":"2023-06-10T18:44:13.425522+0000","last_undegraded":"2023-06-10T18:44:13.425522+0000","last_fullsized":"2023-06-10T18:44:13.425522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:23.228638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394120+0000","last_change":"2023-06-10T18:44:08.577421+0000","last_active":"2023-06-10T18:44:13.394120+0000","last_peered":"2023-06-10T18:44:13.394120+0000","last_clean":"2023-06-10T18:44:13.394120+0000","last_became_active":"2023-06-10T18:44:08.577127+0000","last_became_peered":"2023-06-10T18:44:08.577127+0000","last_unstale":"2023-06-10T18:44:13.394120+0000","last_undegraded":"2023-06-10T18:44:13.394120+0000","last_fullsized":"2023-06-10T18:44:13.394120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:14.393845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393966+0000","last_change":"2023-06-10T18:44:08.986118+0000","last_active":"2023-06-10T18:44:13.393966+0000","last_peered":"2023-06-10T18:44:13.393966+0000","last_clean":"2023-06-10T18:44:13.393966+0000","last_became_active":"2023-06-10T18:44:08.985845+0000","last_became_peered":"2023-06-10T18:44:08.985845+0000","last_unstale":"2023-06-10T18:44:13.393966+0000","last_undegraded":"2023-06-10T18:44:13.393966+0000","last_fullsized":"2023-06-10T18:44:13.393966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393763+0000","last_change":"2023-06-10T18:44:13.009330+0000","last_active":"2023-06-10T18:44:13.393763+0000","last_peered":"2023-06-10T18:44:13.393763+0000","last_clean":"2023-06-10T18:44:13.393763+0000","last_became_active":"2023-06-10T18:44:13.008968+0000","last_became_peered":"2023-06-10T18:44:13.008968+0000","last_unstale":"2023-06-10T18:44:13.393763+0000","last_undegraded":"2023-06-10T18:44:13.393763+0000","last_fullsized":"2023-06-10T18:44:13.393763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:01.454996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:18.388487+0000","last_change":"2023-06-10T18:44:08.577256+0000","last_active":"2023-06-10T18:44:18.388487+0000","last_peered":"2023-06-10T18:44:18.388487+0000","last_clean":"2023-06-10T18:44:18.388487+0000","last_became_active":"2023-06-10T18:44:08.576987+0000","last_became_peered":"2023-06-10T18:44:08.576987+0000","last_unstale":"2023-06-10T18:44:18.388487+0000","last_undegraded":"2023-06-10T18:44:18.388487+0000","last_fullsized":"2023-06-10T18:44:18.388487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424232+0000","last_change":"2023-06-10T18:44:08.989046+0000","last_active":"2023-06-10T18:44:13.424232+0000","last_peered":"2023-06-10T18:44:13.424232+0000","last_clean":"2023-06-10T18:44:13.424232+0000","last_became_active":"2023-06-10T18:44:08.988763+0000","last_became_peered":"2023-06-10T18:44:08.988763+0000","last_unstale":"2023-06-10T18:44:13.424232+0000","last_undegraded":"2023-06-10T18:44:13.424232+0000","last_fullsized":"2023-06-10T18:44:13.424232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393726+0000","last_change":"2023-06-10T18:44:13.009976+0000","last_active":"2023-06-10T18:44:13.393726+0000","last_peered":"2023-06-10T18:44:13.393726+0000","last_clean":"2023-06-10T18:44:13.393726+0000","last_became_active":"2023-06-10T18:44:13.009599+0000","last_became_peered":"2023-06-10T18:44:13.009599+0000","last_unstale":"2023-06-10T18:44:13.393726+0000","last_undegraded":"2023-06-10T18:44:13.393726+0000","last_fullsized":"2023-06-10T18:44:13.393726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393682+0000","last_change":"2023-06-10T18:44:08.578965+0000","last_active":"2023-06-10T18:44:13.393682+0000","last_peered":"2023-06-10T18:44:13.393682+0000","last_clean":"2023-06-10T18:44:13.393682+0000","last_became_active":"2023-06-10T18:44:08.578666+0000","last_became_peered":"2023-06-10T18:44:08.578666+0000","last_unstale":"2023-06-10T18:44:13.393682+0000","last_undegraded":"2023-06-10T18:44:13.393682+0000","last_fullsized":"2023-06-10T18:44:13.393682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425838+0000","last_change":"2023-06-10T18:44:08.986550+0000","last_active":"2023-06-10T18:44:13.425838+0000","last_peered":"2023-06-10T18:44:13.425838+0000","last_clean":"2023-06-10T18:44:13.425838+0000","last_became_active":"2023-06-10T18:44:08.986267+0000","last_became_peered":"2023-06-10T18:44:08.986267+0000","last_unstale":"2023-06-10T18:44:13.425838+0000","last_undegraded":"2023-06-10T18:44:13.425838+0000","last_fullsized":"2023-06-10T18:44:13.425838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425693+0000","last_change":"2023-06-10T18:44:13.007347+0000","last_active":"2023-06-10T18:44:13.425693+0000","last_peered":"2023-06-10T18:44:13.425693+0000","last_clean":"2023-06-10T18:44:13.425693+0000","last_became_active":"2023-06-10T18:44:13.006948+0000","last_became_peered":"2023-06-10T18:44:13.006948+0000","last_unstale":"2023-06-10T18:44:13.425693+0000","last_undegraded":"2023-06-10T18:44:13.425693+0000","last_fullsized":"2023-06-10T18:44:13.425693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423561+0000","last_change":"2023-06-10T18:44:08.575263+0000","last_active":"2023-06-10T18:44:13.423561+0000","last_peered":"2023-06-10T18:44:13.423561+0000","last_clean":"2023-06-10T18:44:13.423561+0000","last_became_active":"2023-06-10T18:44:08.574716+0000","last_became_peered":"2023-06-10T18:44:08.574716+0000","last_unstale":"2023-06-10T18:44:13.423561+0000","last_undegraded":"2023-06-10T18:44:13.423561+0000","last_fullsized":"2023-06-10T18:44:13.423561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393754+0000","last_change":"2023-06-10T18:44:08.986065+0000","last_active":"2023-06-10T18:44:13.393754+0000","last_peered":"2023-06-10T18:44:13.393754+0000","last_clean":"2023-06-10T18:44:13.393754+0000","last_became_active":"2023-06-10T18:44:08.984998+0000","last_became_peered":"2023-06-10T18:44:08.984998+0000","last_unstale":"2023-06-10T18:44:13.393754+0000","last_undegraded":"2023-06-10T18:44:13.393754+0000","last_fullsized":"2023-06-10T18:44:13.393754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425425+0000","last_change":"2023-06-10T18:44:13.004286+0000","last_active":"2023-06-10T18:44:13.425425+0000","last_peered":"2023-06-10T18:44:13.425425+0000","last_clean":"2023-06-10T18:44:13.425425+0000","last_became_active":"2023-06-10T18:44:13.003671+0000","last_became_peered":"2023-06-10T18:44:13.003671+0000","last_unstale":"2023-06-10T18:44:13.425425+0000","last_undegraded":"2023-06-10T18:44:13.425425+0000","last_fullsized":"2023-06-10T18:44:13.425425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394170+0000","last_change":"2023-06-10T18:44:08.578231+0000","last_active":"2023-06-10T18:44:13.394170+0000","last_peered":"2023-06-10T18:44:13.394170+0000","last_clean":"2023-06-10T18:44:13.394170+0000","last_became_active":"2023-06-10T18:44:08.577966+0000","last_became_peered":"2023-06-10T18:44:08.577966+0000","last_unstale":"2023-06-10T18:44:13.394170+0000","last_undegraded":"2023-06-10T18:44:13.394170+0000","last_fullsized":"2023-06-10T18:44:13.394170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423359+0000","last_change":"2023-06-10T18:44:08.990234+0000","last_active":"2023-06-10T18:44:13.423359+0000","last_peered":"2023-06-10T18:44:13.423359+0000","last_clean":"2023-06-10T18:44:13.423359+0000","last_became_active":"2023-06-10T18:44:08.989505+0000","last_became_peered":"2023-06-10T18:44:08.989505+0000","last_unstale":"2023-06-10T18:44:13.423359+0000","last_undegraded":"2023-06-10T18:44:13.423359+0000","last_fullsized":"2023-06-10T18:44:13.423359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:31.544241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393091+0000","last_change":"2023-06-10T18:44:13.009889+0000","last_active":"2023-06-10T18:44:13.393091+0000","last_peered":"2023-06-10T18:44:13.393091+0000","last_clean":"2023-06-10T18:44:13.393091+0000","last_became_active":"2023-06-10T18:44:13.009528+0000","last_became_peered":"2023-06-10T18:44:13.009528+0000","last_unstale":"2023-06-10T18:44:13.393091+0000","last_undegraded":"2023-06-10T18:44:13.393091+0000","last_fullsized":"2023-06-10T18:44:13.393091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:15.449838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":29,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422838+0000","last_change":"2023-06-10T18:44:08.573929+0000","last_active":"2023-06-10T18:44:13.422838+0000","last_peered":"2023-06-10T18:44:13.422838+0000","last_clean":"2023-06-10T18:44:13.422838+0000","last_became_active":"2023-06-10T18:44:08.573652+0000","last_became_peered":"2023-06-10T18:44:08.573652+0000","last_unstale":"2023-06-10T18:44:13.422838+0000","last_undegraded":"2023-06-10T18:44:13.422838+0000","last_fullsized":"2023-06-10T18:44:13.422838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:08.763880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425617+0000","last_change":"2023-06-10T18:44:08.984556+0000","last_active":"2023-06-10T18:44:13.425617+0000","last_peered":"2023-06-10T18:44:13.425617+0000","last_clean":"2023-06-10T18:44:13.425617+0000","last_became_active":"2023-06-10T18:44:08.984321+0000","last_became_peered":"2023-06-10T18:44:08.984321+0000","last_unstale":"2023-06-10T18:44:13.425617+0000","last_undegraded":"2023-06-10T18:44:13.425617+0000","last_fullsized":"2023-06-10T18:44:13.425617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394464+0000","last_change":"2023-06-10T18:44:13.006271+0000","last_active":"2023-06-10T18:44:13.394464+0000","last_peered":"2023-06-10T18:44:13.394464+0000","last_clean":"2023-06-10T18:44:13.394464+0000","last_became_active":"2023-06-10T18:44:13.005878+0000","last_became_peered":"2023-06-10T18:44:13.005878+0000","last_unstale":"2023-06-10T18:44:13.394464+0000","last_undegraded":"2023-06-10T18:44:13.394464+0000","last_fullsized":"2023-06-10T18:44:13.394464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:21.046063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394016+0000","last_change":"2023-06-10T18:44:08.578331+0000","last_active":"2023-06-10T18:44:13.394016+0000","last_peered":"2023-06-10T18:44:13.394016+0000","last_clean":"2023-06-10T18:44:13.394016+0000","last_became_active":"2023-06-10T18:44:08.577846+0000","last_became_peered":"2023-06-10T18:44:08.577846+0000","last_unstale":"2023-06-10T18:44:13.394016+0000","last_undegraded":"2023-06-10T18:44:13.394016+0000","last_fullsized":"2023-06-10T18:44:13.394016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392530+0000","last_change":"2023-06-10T18:44:08.984476+0000","last_active":"2023-06-10T18:44:13.392530+0000","last_peered":"2023-06-10T18:44:13.392530+0000","last_clean":"2023-06-10T18:44:13.392530+0000","last_became_active":"2023-06-10T18:44:08.984095+0000","last_became_peered":"2023-06-10T18:44:08.984095+0000","last_unstale":"2023-06-10T18:44:13.392530+0000","last_undegraded":"2023-06-10T18:44:13.392530+0000","last_fullsized":"2023-06-10T18:44:13.392530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422417+0000","last_change":"2023-06-10T18:44:13.006928+0000","last_active":"2023-06-10T18:44:13.422417+0000","last_peered":"2023-06-10T18:44:13.422417+0000","last_clean":"2023-06-10T18:44:13.422417+0000","last_became_active":"2023-06-10T18:44:13.006526+0000","last_became_peered":"2023-06-10T18:44:13.006526+0000","last_unstale":"2023-06-10T18:44:13.422417+0000","last_undegraded":"2023-06-10T18:44:13.422417+0000","last_fullsized":"2023-06-10T18:44:13.422417+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:42:15.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393898+0000","last_change":"2023-06-10T18:44:08.580704+0000","last_active":"2023-06-10T18:44:13.393898+0000","last_peered":"2023-06-10T18:44:13.393898+0000","last_clean":"2023-06-10T18:44:13.393898+0000","last_became_active":"2023-06-10T18:44:08.580413+0000","last_became_peered":"2023-06-10T18:44:08.580413+0000","last_unstale":"2023-06-10T18:44:13.393898+0000","last_undegraded":"2023-06-10T18:44:13.393898+0000","last_fullsized":"2023-06-10T18:44:13.393898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:29.052625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423333+0000","last_change":"2023-06-10T18:44:08.988350+0000","last_active":"2023-06-10T18:44:13.423333+0000","last_peered":"2023-06-10T18:44:13.423333+0000","last_clean":"2023-06-10T18:44:13.423333+0000","last_became_active":"2023-06-10T18:44:08.988073+0000","last_became_peered":"2023-06-10T18:44:08.988073+0000","last_unstale":"2023-06-10T18:44:13.423333+0000","last_undegraded":"2023-06-10T18:44:13.423333+0000","last_fullsized":"2023-06-10T18:44:13.423333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422684+0000","last_change":"2023-06-10T18:44:13.001742+0000","last_active":"2023-06-10T18:44:13.422684+0000","last_peered":"2023-06-10T18:44:13.422684+0000","last_clean":"2023-06-10T18:44:13.422684+0000","last_became_active":"2023-06-10T18:44:13.001373+0000","last_became_peered":"2023-06-10T18:44:13.001373+0000","last_unstale":"2023-06-10T18:44:13.422684+0000","last_undegraded":"2023-06-10T18:44:13.422684+0000","last_fullsized":"2023-06-10T18:44:13.422684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:11.616196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394281+0000","last_change":"2023-06-10T18:44:08.574389+0000","last_active":"2023-06-10T18:44:13.394281+0000","last_peered":"2023-06-10T18:44:13.394281+0000","last_clean":"2023-06-10T18:44:13.394281+0000","last_became_active":"2023-06-10T18:44:08.574056+0000","last_became_peered":"2023-06-10T18:44:08.574056+0000","last_unstale":"2023-06-10T18:44:13.394281+0000","last_undegraded":"2023-06-10T18:44:13.394281+0000","last_fullsized":"2023-06-10T18:44:13.394281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423143+0000","last_change":"2023-06-10T18:44:08.987503+0000","last_active":"2023-06-10T18:44:13.423143+0000","last_peered":"2023-06-10T18:44:13.423143+0000","last_clean":"2023-06-10T18:44:13.423143+0000","last_became_active":"2023-06-10T18:44:08.987204+0000","last_became_peered":"2023-06-10T18:44:08.987204+0000","last_unstale":"2023-06-10T18:44:13.423143+0000","last_undegraded":"2023-06-10T18:44:13.423143+0000","last_fullsized":"2023-06-10T18:44:13.423143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:45.870218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422702+0000","last_change":"2023-06-10T18:44:13.003895+0000","last_active":"2023-06-10T18:44:13.422702+0000","last_peered":"2023-06-10T18:44:13.422702+0000","last_clean":"2023-06-10T18:44:13.422702+0000","last_became_active":"2023-06-10T18:44:13.001542+0000","last_became_peered":"2023-06-10T18:44:13.001542+0000","last_unstale":"2023-06-10T18:44:13.422702+0000","last_undegraded":"2023-06-10T18:44:13.422702+0000","last_fullsized":"2023-06-10T18:44:13.422702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:54:25.091621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393042+0000","last_change":"2023-06-10T18:44:08.580607+0000","last_active":"2023-06-10T18:44:13.393042+0000","last_peered":"2023-06-10T18:44:13.393042+0000","last_clean":"2023-06-10T18:44:13.393042+0000","last_became_active":"2023-06-10T18:44:08.580289+0000","last_became_peered":"2023-06-10T18:44:08.580289+0000","last_unstale":"2023-06-10T18:44:13.393042+0000","last_undegraded":"2023-06-10T18:44:13.393042+0000","last_fullsized":"2023-06-10T18:44:13.393042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:09.158752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424116+0000","last_change":"2023-06-10T18:44:08.987624+0000","last_active":"2023-06-10T18:44:13.424116+0000","last_peered":"2023-06-10T18:44:13.424116+0000","last_clean":"2023-06-10T18:44:13.424116+0000","last_became_active":"2023-06-10T18:44:08.987375+0000","last_became_peered":"2023-06-10T18:44:08.987375+0000","last_unstale":"2023-06-10T18:44:13.424116+0000","last_undegraded":"2023-06-10T18:44:13.424116+0000","last_fullsized":"2023-06-10T18:44:13.424116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422769+0000","last_change":"2023-06-10T18:44:13.002722+0000","last_active":"2023-06-10T18:44:13.422769+0000","last_peered":"2023-06-10T18:44:13.422769+0000","last_clean":"2023-06-10T18:44:13.422769+0000","last_became_active":"2023-06-10T18:44:13.002214+0000","last_became_peered":"2023-06-10T18:44:13.002214+0000","last_unstale":"2023-06-10T18:44:13.422769+0000","last_undegraded":"2023-06-10T18:44:13.422769+0000","last_fullsized":"2023-06-10T18:44:13.422769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:04:35.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425192+0000","last_change":"2023-06-10T18:44:08.578128+0000","last_active":"2023-06-10T18:44:13.425192+0000","last_peered":"2023-06-10T18:44:13.425192+0000","last_clean":"2023-06-10T18:44:13.425192+0000","last_became_active":"2023-06-10T18:44:08.577863+0000","last_became_peered":"2023-06-10T18:44:08.577863+0000","last_unstale":"2023-06-10T18:44:13.425192+0000","last_undegraded":"2023-06-10T18:44:13.425192+0000","last_fullsized":"2023-06-10T18:44:13.425192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423537+0000","last_change":"2023-06-10T18:44:08.995303+0000","last_active":"2023-06-10T18:44:13.423537+0000","last_peered":"2023-06-10T18:44:13.423537+0000","last_clean":"2023-06-10T18:44:13.423537+0000","last_became_active":"2023-06-10T18:44:08.995025+0000","last_became_peered":"2023-06-10T18:44:08.995025+0000","last_unstale":"2023-06-10T18:44:13.423537+0000","last_undegraded":"2023-06-10T18:44:13.423537+0000","last_fullsized":"2023-06-10T18:44:13.423537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394147+0000","last_change":"2023-06-10T18:44:13.008926+0000","last_active":"2023-06-10T18:44:13.394147+0000","last_peered":"2023-06-10T18:44:13.394147+0000","last_clean":"2023-06-10T18:44:13.394147+0000","last_became_active":"2023-06-10T18:44:13.008249+0000","last_became_peered":"2023-06-10T18:44:13.008249+0000","last_unstale":"2023-06-10T18:44:13.394147+0000","last_undegraded":"2023-06-10T18:44:13.394147+0000","last_fullsized":"2023-06-10T18:44:13.394147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423270+0000","last_change":"2023-06-10T18:44:08.575472+0000","last_active":"2023-06-10T18:44:13.423270+0000","last_peered":"2023-06-10T18:44:13.423270+0000","last_clean":"2023-06-10T18:44:13.423270+0000","last_became_active":"2023-06-10T18:44:08.575186+0000","last_became_peered":"2023-06-10T18:44:08.575186+0000","last_unstale":"2023-06-10T18:44:13.423270+0000","last_undegraded":"2023-06-10T18:44:13.423270+0000","last_fullsized":"2023-06-10T18:44:13.423270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:33.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423243+0000","last_change":"2023-06-10T18:44:08.987175+0000","last_active":"2023-06-10T18:44:13.423243+0000","last_peered":"2023-06-10T18:44:13.423243+0000","last_clean":"2023-06-10T18:44:13.423243+0000","last_became_active":"2023-06-10T18:44:08.986395+0000","last_became_peered":"2023-06-10T18:44:08.986395+0000","last_unstale":"2023-06-10T18:44:13.423243+0000","last_undegraded":"2023-06-10T18:44:13.423243+0000","last_fullsized":"2023-06-10T18:44:13.423243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393572+0000","last_change":"2023-06-10T18:44:13.002535+0000","last_active":"2023-06-10T18:44:13.393572+0000","last_peered":"2023-06-10T18:44:13.393572+0000","last_clean":"2023-06-10T18:44:13.393572+0000","last_became_active":"2023-06-10T18:44:13.002065+0000","last_became_peered":"2023-06-10T18:44:13.002065+0000","last_unstale":"2023-06-10T18:44:13.393572+0000","last_undegraded":"2023-06-10T18:44:13.393572+0000","last_fullsized":"2023-06-10T18:44:13.393572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:56.164577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423083+0000","last_change":"2023-06-10T18:44:08.572499+0000","last_active":"2023-06-10T18:44:13.423083+0000","last_peered":"2023-06-10T18:44:13.423083+0000","last_clean":"2023-06-10T18:44:13.423083+0000","last_became_active":"2023-06-10T18:44:08.572196+0000","last_became_peered":"2023-06-10T18:44:08.572196+0000","last_unstale":"2023-06-10T18:44:13.423083+0000","last_undegraded":"2023-06-10T18:44:13.423083+0000","last_fullsized":"2023-06-10T18:44:13.423083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423986+0000","last_change":"2023-06-10T18:44:08.989104+0000","last_active":"2023-06-10T18:44:13.423986+0000","last_peered":"2023-06-10T18:44:13.423986+0000","last_clean":"2023-06-10T18:44:13.423986+0000","last_became_active":"2023-06-10T18:44:08.988824+0000","last_became_peered":"2023-06-10T18:44:08.988824+0000","last_unstale":"2023-06-10T18:44:13.423986+0000","last_undegraded":"2023-06-10T18:44:13.423986+0000","last_fullsized":"2023-06-10T18:44:13.423986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:19.807436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423861+0000","last_change":"2023-06-10T18:44:13.006826+0000","last_active":"2023-06-10T18:44:13.423861+0000","last_peered":"2023-06-10T18:44:13.423861+0000","last_clean":"2023-06-10T18:44:13.423861+0000","last_became_active":"2023-06-10T18:44:13.006458+0000","last_became_peered":"2023-06-10T18:44:13.006458+0000","last_unstale":"2023-06-10T18:44:13.423861+0000","last_undegraded":"2023-06-10T18:44:13.423861+0000","last_fullsized":"2023-06-10T18:44:13.423861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.577680+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.577419+0000","last_became_peered":"2023-06-10T18:44:08.577419+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394577+0000","last_change":"2023-06-10T18:44:08.991268+0000","last_active":"2023-06-10T18:44:13.394577+0000","last_peered":"2023-06-10T18:44:13.394577+0000","last_clean":"2023-06-10T18:44:13.394577+0000","last_became_active":"2023-06-10T18:44:08.990999+0000","last_became_peered":"2023-06-10T18:44:08.990999+0000","last_unstale":"2023-06-10T18:44:13.394577+0000","last_undegraded":"2023-06-10T18:44:13.394577+0000","last_fullsized":"2023-06-10T18:44:13.394577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423567+0000","last_change":"2023-06-10T18:44:13.010490+0000","last_active":"2023-06-10T18:44:13.423567+0000","last_peered":"2023-06-10T18:44:13.423567+0000","last_clean":"2023-06-10T18:44:13.423567+0000","last_became_active":"2023-06-10T18:44:13.010098+0000","last_became_peered":"2023-06-10T18:44:13.010098+0000","last_unstale":"2023-06-10T18:44:13.423567+0000","last_undegraded":"2023-06-10T18:44:13.423567+0000","last_fullsized":"2023-06-10T18:44:13.423567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:58.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394227+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:44:13.394227+0000","last_peered":"2023-06-10T18:44:13.394227+0000","last_clean":"2023-06-10T18:44:13.394227+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:44:13.394227+0000","last_undegraded":"2023-06-10T18:44:13.394227+0000","last_fullsized":"2023-06-10T18:44:13.394227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393285+0000","last_change":"2023-06-10T18:44:08.577289+0000","last_active":"2023-06-10T18:44:13.393285+0000","last_peered":"2023-06-10T18:44:13.393285+0000","last_clean":"2023-06-10T18:44:13.393285+0000","last_became_active":"2023-06-10T18:44:08.577021+0000","last_became_peered":"2023-06-10T18:44:08.577021+0000","last_unstale":"2023-06-10T18:44:13.393285+0000","last_undegraded":"2023-06-10T18:44:13.393285+0000","last_fullsized":"2023-06-10T18:44:13.393285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393847+0000","last_change":"2023-06-10T18:44:08.983928+0000","last_active":"2023-06-10T18:44:13.393847+0000","last_peered":"2023-06-10T18:44:13.393847+0000","last_clean":"2023-06-10T18:44:13.393847+0000","last_became_active":"2023-06-10T18:44:08.983562+0000","last_became_peered":"2023-06-10T18:44:08.983562+0000","last_unstale":"2023-06-10T18:44:13.393847+0000","last_undegraded":"2023-06-10T18:44:13.393847+0000","last_fullsized":"2023-06-10T18:44:13.393847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423202+0000","last_change":"2023-06-10T18:44:13.006596+0000","last_active":"2023-06-10T18:44:13.423202+0000","last_peered":"2023-06-10T18:44:13.423202+0000","last_clean":"2023-06-10T18:44:13.423202+0000","last_became_active":"2023-06-10T18:44:13.006258+0000","last_became_peered":"2023-06-10T18:44:13.006258+0000","last_unstale":"2023-06-10T18:44:13.423202+0000","last_undegraded":"2023-06-10T18:44:13.423202+0000","last_fullsized":"2023-06-10T18:44:13.423202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:31.503221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393245+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:44:13.393245+0000","last_peered":"2023-06-10T18:44:13.393245+0000","last_clean":"2023-06-10T18:44:13.393245+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:44:13.393245+0000","last_undegraded":"2023-06-10T18:44:13.393245+0000","last_fullsized":"2023-06-10T18:44:13.393245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:28.260256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424001+0000","last_change":"2023-06-10T18:44:08.578515+0000","last_active":"2023-06-10T18:44:13.424001+0000","last_peered":"2023-06-10T18:44:13.424001+0000","last_clean":"2023-06-10T18:44:13.424001+0000","last_became_active":"2023-06-10T18:44:08.578184+0000","last_became_peered":"2023-06-10T18:44:08.578184+0000","last_unstale":"2023-06-10T18:44:13.424001+0000","last_undegraded":"2023-06-10T18:44:13.424001+0000","last_fullsized":"2023-06-10T18:44:13.424001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394190+0000","last_change":"2023-06-10T18:44:08.985054+0000","last_active":"2023-06-10T18:44:13.394190+0000","last_peered":"2023-06-10T18:44:13.394190+0000","last_clean":"2023-06-10T18:44:13.394190+0000","last_became_active":"2023-06-10T18:44:08.984698+0000","last_became_peered":"2023-06-10T18:44:08.984698+0000","last_unstale":"2023-06-10T18:44:13.394190+0000","last_undegraded":"2023-06-10T18:44:13.394190+0000","last_fullsized":"2023-06-10T18:44:13.394190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423677+0000","last_change":"2023-06-10T18:44:13.002621+0000","last_active":"2023-06-10T18:44:13.423677+0000","last_peered":"2023-06-10T18:44:13.423677+0000","last_clean":"2023-06-10T18:44:13.423677+0000","last_became_active":"2023-06-10T18:44:13.002229+0000","last_became_peered":"2023-06-10T18:44:13.002229+0000","last_unstale":"2023-06-10T18:44:13.423677+0000","last_undegraded":"2023-06-10T18:44:13.423677+0000","last_fullsized":"2023-06-10T18:44:13.423677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:44:56.053966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422499+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:44:13.422499+0000","last_peered":"2023-06-10T18:44:13.422499+0000","last_clean":"2023-06-10T18:44:13.422499+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:44:13.422499+0000","last_undegraded":"2023-06-10T18:44:13.422499+0000","last_fullsized":"2023-06-10T18:44:13.422499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:49:35.702822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393742+0000","last_change":"2023-06-10T18:44:08.571468+0000","last_active":"2023-06-10T18:44:13.393742+0000","last_peered":"2023-06-10T18:44:13.393742+0000","last_clean":"2023-06-10T18:44:13.393742+0000","last_became_active":"2023-06-10T18:44:08.570651+0000","last_became_peered":"2023-06-10T18:44:08.570651+0000","last_unstale":"2023-06-10T18:44:13.393742+0000","last_undegraded":"2023-06-10T18:44:13.393742+0000","last_fullsized":"2023-06-10T18:44:13.393742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394594+0000","last_change":"2023-06-10T18:44:08.983687+0000","last_active":"2023-06-10T18:44:13.394594+0000","last_peered":"2023-06-10T18:44:13.394594+0000","last_clean":"2023-06-10T18:44:13.394594+0000","last_became_active":"2023-06-10T18:44:08.983381+0000","last_became_peered":"2023-06-10T18:44:08.983381+0000","last_unstale":"2023-06-10T18:44:13.394594+0000","last_undegraded":"2023-06-10T18:44:13.394594+0000","last_fullsized":"2023-06-10T18:44:13.394594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393971+0000","last_change":"2023-06-10T18:44:13.009660+0000","last_active":"2023-06-10T18:44:13.393971+0000","last_peered":"2023-06-10T18:44:13.393971+0000","last_clean":"2023-06-10T18:44:13.393971+0000","last_became_active":"2023-06-10T18:44:13.009208+0000","last_became_peered":"2023-06-10T18:44:13.009208+0000","last_unstale":"2023-06-10T18:44:13.393971+0000","last_undegraded":"2023-06-10T18:44:13.393971+0000","last_fullsized":"2023-06-10T18:44:13.393971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392956+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:44:13.392956+0000","last_peered":"2023-06-10T18:44:13.392956+0000","last_clean":"2023-06-10T18:44:13.392956+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:44:13.392956+0000","last_undegraded":"2023-06-10T18:44:13.392956+0000","last_fullsized":"2023-06-10T18:44:13.392956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":183,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394153+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:44:13.394153+0000","last_peered":"2023-06-10T18:44:13.394153+0000","last_clean":"2023-06-10T18:44:13.394153+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:44:13.394153+0000","last_undegraded":"2023-06-10T18:44:13.394153+0000","last_fullsized":"2023-06-10T18:44:13.394153+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:22.105411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425014+0000","last_change":"2023-06-10T18:44:08.576254+0000","last_active":"2023-06-10T18:44:13.425014+0000","last_peered":"2023-06-10T18:44:13.425014+0000","last_clean":"2023-06-10T18:44:13.425014+0000","last_became_active":"2023-06-10T18:44:08.575980+0000","last_became_peered":"2023-06-10T18:44:08.575980+0000","last_unstale":"2023-06-10T18:44:13.425014+0000","last_undegraded":"2023-06-10T18:44:13.425014+0000","last_fullsized":"2023-06-10T18:44:13.425014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423295+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.423295+0000","last_peered":"2023-06-10T18:44:13.423295+0000","last_clean":"2023-06-10T18:44:13.423295+0000","last_became_active":"2023-06-10T18:44:08.988159+0000","last_became_peered":"2023-06-10T18:44:08.988159+0000","last_unstale":"2023-06-10T18:44:13.423295+0000","last_undegraded":"2023-06-10T18:44:13.423295+0000","last_fullsized":"2023-06-10T18:44:13.423295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:55.371558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393218+0000","last_change":"2023-06-10T18:44:13.006561+0000","last_active":"2023-06-10T18:44:13.393218+0000","last_peered":"2023-06-10T18:44:13.393218+0000","last_clean":"2023-06-10T18:44:13.393218+0000","last_became_active":"2023-06-10T18:44:13.006171+0000","last_became_peered":"2023-06-10T18:44:13.006171+0000","last_unstale":"2023-06-10T18:44:13.393218+0000","last_undegraded":"2023-06-10T18:44:13.393218+0000","last_fullsized":"2023-06-10T18:44:13.393218+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:18.821445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394377+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:44:13.394377+0000","last_peered":"2023-06-10T18:44:13.394377+0000","last_clean":"2023-06-10T18:44:13.394377+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:44:13.394377+0000","last_undegraded":"2023-06-10T18:44:13.394377+0000","last_fullsized":"2023-06-10T18:44:13.394377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:21.913689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423028+0000","last_change":"2023-06-10T18:44:08.578010+0000","last_active":"2023-06-10T18:44:13.423028+0000","last_peered":"2023-06-10T18:44:13.423028+0000","last_clean":"2023-06-10T18:44:13.423028+0000","last_became_active":"2023-06-10T18:44:08.577727+0000","last_became_peered":"2023-06-10T18:44:08.577727+0000","last_unstale":"2023-06-10T18:44:13.423028+0000","last_undegraded":"2023-06-10T18:44:13.423028+0000","last_fullsized":"2023-06-10T18:44:13.423028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:00:49.837648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423225+0000","last_change":"2023-06-10T18:44:08.990076+0000","last_active":"2023-06-10T18:44:13.423225+0000","last_peered":"2023-06-10T18:44:13.423225+0000","last_clean":"2023-06-10T18:44:13.423225+0000","last_became_active":"2023-06-10T18:44:08.989191+0000","last_became_peered":"2023-06-10T18:44:08.989191+0000","last_unstale":"2023-06-10T18:44:13.423225+0000","last_undegraded":"2023-06-10T18:44:13.423225+0000","last_fullsized":"2023-06-10T18:44:13.423225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:35.616295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423712+0000","last_change":"2023-06-10T18:44:12.999386+0000","last_active":"2023-06-10T18:44:13.423712+0000","last_peered":"2023-06-10T18:44:13.423712+0000","last_clean":"2023-06-10T18:44:13.423712+0000","last_became_active":"2023-06-10T18:44:12.998986+0000","last_became_peered":"2023-06-10T18:44:12.998986+0000","last_unstale":"2023-06-10T18:44:13.423712+0000","last_undegraded":"2023-06-10T18:44:13.423712+0000","last_fullsized":"2023-06-10T18:44:13.423712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:10.294554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394138+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:44:13.394138+0000","last_peered":"2023-06-10T18:44:13.394138+0000","last_clean":"2023-06-10T18:44:13.394138+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:44:13.394138+0000","last_undegraded":"2023-06-10T18:44:13.394138+0000","last_fullsized":"2023-06-10T18:44:13.394138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:43.350286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394357+0000","last_change":"2023-06-10T18:44:08.572021+0000","last_active":"2023-06-10T18:44:13.394357+0000","last_peered":"2023-06-10T18:44:13.394357+0000","last_clean":"2023-06-10T18:44:13.394357+0000","last_became_active":"2023-06-10T18:44:08.571716+0000","last_became_peered":"2023-06-10T18:44:08.571716+0000","last_unstale":"2023-06-10T18:44:13.394357+0000","last_undegraded":"2023-06-10T18:44:13.394357+0000","last_fullsized":"2023-06-10T18:44:13.394357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423849+0000","last_change":"2023-06-10T18:44:08.983810+0000","last_active":"2023-06-10T18:44:13.423849+0000","last_peered":"2023-06-10T18:44:13.423849+0000","last_clean":"2023-06-10T18:44:13.423849+0000","last_became_active":"2023-06-10T18:44:08.983539+0000","last_became_peered":"2023-06-10T18:44:08.983539+0000","last_unstale":"2023-06-10T18:44:13.423849+0000","last_undegraded":"2023-06-10T18:44:13.423849+0000","last_fullsized":"2023-06-10T18:44:13.423849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423587+0000","last_change":"2023-06-10T18:44:13.000564+0000","last_active":"2023-06-10T18:44:13.423587+0000","last_peered":"2023-06-10T18:44:13.423587+0000","last_clean":"2023-06-10T18:44:13.423587+0000","last_became_active":"2023-06-10T18:44:13.000141+0000","last_became_peered":"2023-06-10T18:44:13.000141+0000","last_unstale":"2023-06-10T18:44:13.423587+0000","last_undegraded":"2023-06-10T18:44:13.423587+0000","last_fullsized":"2023-06-10T18:44:13.423587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:45.731880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425478+0000","last_change":"2023-06-10T18:44:08.579728+0000","last_active":"2023-06-10T18:44:13.425478+0000","last_peered":"2023-06-10T18:44:13.425478+0000","last_clean":"2023-06-10T18:44:13.425478+0000","last_became_active":"2023-06-10T18:44:08.579431+0000","last_became_peered":"2023-06-10T18:44:08.579431+0000","last_unstale":"2023-06-10T18:44:13.425478+0000","last_undegraded":"2023-06-10T18:44:13.425478+0000","last_fullsized":"2023-06-10T18:44:13.425478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394345+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:44:13.394345+0000","last_peered":"2023-06-10T18:44:13.394345+0000","last_clean":"2023-06-10T18:44:13.394345+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:44:13.394345+0000","last_undegraded":"2023-06-10T18:44:13.394345+0000","last_fullsized":"2023-06-10T18:44:13.394345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:05.996450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424774+0000","last_change":"2023-06-10T18:44:08.983770+0000","last_active":"2023-06-10T18:44:13.424774+0000","last_peered":"2023-06-10T18:44:13.424774+0000","last_clean":"2023-06-10T18:44:13.424774+0000","last_became_active":"2023-06-10T18:44:08.983526+0000","last_became_peered":"2023-06-10T18:44:08.983526+0000","last_unstale":"2023-06-10T18:44:13.424774+0000","last_undegraded":"2023-06-10T18:44:13.424774+0000","last_fullsized":"2023-06-10T18:44:13.424774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424718+0000","last_change":"2023-06-10T18:44:13.003992+0000","last_active":"2023-06-10T18:44:13.424718+0000","last_peered":"2023-06-10T18:44:13.424718+0000","last_clean":"2023-06-10T18:44:13.424718+0000","last_became_active":"2023-06-10T18:44:13.003565+0000","last_became_peered":"2023-06-10T18:44:13.003565+0000","last_unstale":"2023-06-10T18:44:13.424718+0000","last_undegraded":"2023-06-10T18:44:13.424718+0000","last_fullsized":"2023-06-10T18:44:13.424718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423787+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:44:13.423787+0000","last_peered":"2023-06-10T18:44:13.423787+0000","last_clean":"2023-06-10T18:44:13.423787+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:44:13.423787+0000","last_undegraded":"2023-06-10T18:44:13.423787+0000","last_fullsized":"2023-06-10T18:44:13.423787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:03:46.107219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423014+0000","last_change":"2023-06-10T18:44:08.576079+0000","last_active":"2023-06-10T18:44:13.423014+0000","last_peered":"2023-06-10T18:44:13.423014+0000","last_clean":"2023-06-10T18:44:13.423014+0000","last_became_active":"2023-06-10T18:44:08.575346+0000","last_became_peered":"2023-06-10T18:44:08.575346+0000","last_unstale":"2023-06-10T18:44:13.423014+0000","last_undegraded":"2023-06-10T18:44:13.423014+0000","last_fullsized":"2023-06-10T18:44:13.423014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:54:23.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423207+0000","last_change":"2023-06-10T18:44:08.990722+0000","last_active":"2023-06-10T18:44:13.423207+0000","last_peered":"2023-06-10T18:44:13.423207+0000","last_clean":"2023-06-10T18:44:13.423207+0000","last_became_active":"2023-06-10T18:44:08.988467+0000","last_became_peered":"2023-06-10T18:44:08.988467+0000","last_unstale":"2023-06-10T18:44:13.423207+0000","last_undegraded":"2023-06-10T18:44:13.423207+0000","last_fullsized":"2023-06-10T18:44:13.423207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394392+0000","last_change":"2023-06-10T18:44:13.008834+0000","last_active":"2023-06-10T18:44:13.394392+0000","last_peered":"2023-06-10T18:44:13.394392+0000","last_clean":"2023-06-10T18:44:13.394392+0000","last_became_active":"2023-06-10T18:44:13.008433+0000","last_became_peered":"2023-06-10T18:44:13.008433+0000","last_unstale":"2023-06-10T18:44:13.394392+0000","last_undegraded":"2023-06-10T18:44:13.394392+0000","last_fullsized":"2023-06-10T18:44:13.394392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423733+0000","last_change":"2023-06-10T18:44:08.576600+0000","last_active":"2023-06-10T18:44:13.423733+0000","last_peered":"2023-06-10T18:44:13.423733+0000","last_clean":"2023-06-10T18:44:13.423733+0000","last_became_active":"2023-06-10T18:44:08.575635+0000","last_became_peered":"2023-06-10T18:44:08.575635+0000","last_unstale":"2023-06-10T18:44:13.423733+0000","last_undegraded":"2023-06-10T18:44:13.423733+0000","last_fullsized":"2023-06-10T18:44:13.423733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423305+0000","last_change":"2023-06-10T18:44:08.985629+0000","last_active":"2023-06-10T18:44:13.423305+0000","last_peered":"2023-06-10T18:44:13.423305+0000","last_clean":"2023-06-10T18:44:13.423305+0000","last_became_active":"2023-06-10T18:44:08.985367+0000","last_became_peered":"2023-06-10T18:44:08.985367+0000","last_unstale":"2023-06-10T18:44:13.423305+0000","last_undegraded":"2023-06-10T18:44:13.423305+0000","last_fullsized":"2023-06-10T18:44:13.423305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:57.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394246+0000","last_change":"2023-06-10T18:44:13.010558+0000","last_active":"2023-06-10T18:44:13.394246+0000","last_peered":"2023-06-10T18:44:13.394246+0000","last_clean":"2023-06-10T18:44:13.394246+0000","last_became_active":"2023-06-10T18:44:13.010123+0000","last_became_peered":"2023-06-10T18:44:13.010123+0000","last_unstale":"2023-06-10T18:44:13.394246+0000","last_undegraded":"2023-06-10T18:44:13.394246+0000","last_fullsized":"2023-06-10T18:44:13.394246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:13.872743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393194+0000","last_change":"2023-06-10T18:44:08.571654+0000","last_active":"2023-06-10T18:44:13.393194+0000","last_peered":"2023-06-10T18:44:13.393194+0000","last_clean":"2023-06-10T18:44:13.393194+0000","last_became_active":"2023-06-10T18:44:08.571363+0000","last_became_peered":"2023-06-10T18:44:08.571363+0000","last_unstale":"2023-06-10T18:44:13.393194+0000","last_undegraded":"2023-06-10T18:44:13.393194+0000","last_fullsized":"2023-06-10T18:44:13.393194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:48.376074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423819+0000","last_change":"2023-06-10T18:44:08.996056+0000","last_active":"2023-06-10T18:44:13.423819+0000","last_peered":"2023-06-10T18:44:13.423819+0000","last_clean":"2023-06-10T18:44:13.423819+0000","last_became_active":"2023-06-10T18:44:08.995902+0000","last_became_peered":"2023-06-10T18:44:08.995902+0000","last_unstale":"2023-06-10T18:44:13.423819+0000","last_undegraded":"2023-06-10T18:44:13.423819+0000","last_fullsized":"2023-06-10T18:44:13.423819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393057+0000","last_change":"2023-06-10T18:44:13.004134+0000","last_active":"2023-06-10T18:44:13.393057+0000","last_peered":"2023-06-10T18:44:13.393057+0000","last_clean":"2023-06-10T18:44:13.393057+0000","last_became_active":"2023-06-10T18:44:13.003596+0000","last_became_peered":"2023-06-10T18:44:13.003596+0000","last_unstale":"2023-06-10T18:44:13.393057+0000","last_undegraded":"2023-06-10T18:44:13.393057+0000","last_fullsized":"2023-06-10T18:44:13.393057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393105+0000","last_change":"2023-06-10T18:44:08.575332+0000","last_active":"2023-06-10T18:44:13.393105+0000","last_peered":"2023-06-10T18:44:13.393105+0000","last_clean":"2023-06-10T18:44:13.393105+0000","last_became_active":"2023-06-10T18:44:08.575065+0000","last_became_peered":"2023-06-10T18:44:08.575065+0000","last_unstale":"2023-06-10T18:44:13.393105+0000","last_undegraded":"2023-06-10T18:44:13.393105+0000","last_fullsized":"2023-06-10T18:44:13.393105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424331+0000","last_change":"2023-06-10T18:44:08.995435+0000","last_active":"2023-06-10T18:44:13.424331+0000","last_peered":"2023-06-10T18:44:13.424331+0000","last_clean":"2023-06-10T18:44:13.424331+0000","last_became_active":"2023-06-10T18:44:08.995141+0000","last_became_peered":"2023-06-10T18:44:08.995141+0000","last_unstale":"2023-06-10T18:44:13.424331+0000","last_undegraded":"2023-06-10T18:44:13.424331+0000","last_fullsized":"2023-06-10T18:44:13.424331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:52.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422884+0000","last_change":"2023-06-10T18:44:12.998566+0000","last_active":"2023-06-10T18:44:13.422884+0000","last_peered":"2023-06-10T18:44:13.422884+0000","last_clean":"2023-06-10T18:44:13.422884+0000","last_became_active":"2023-06-10T18:44:12.998185+0000","last_became_peered":"2023-06-10T18:44:12.998185+0000","last_unstale":"2023-06-10T18:44:13.422884+0000","last_undegraded":"2023-06-10T18:44:13.422884+0000","last_fullsized":"2023-06-10T18:44:13.422884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425429+0000","last_change":"2023-06-10T18:44:08.576599+0000","last_active":"2023-06-10T18:44:13.425429+0000","last_peered":"2023-06-10T18:44:13.425429+0000","last_clean":"2023-06-10T18:44:13.425429+0000","last_became_active":"2023-06-10T18:44:08.576329+0000","last_became_peered":"2023-06-10T18:44:08.576329+0000","last_unstale":"2023-06-10T18:44:13.425429+0000","last_undegraded":"2023-06-10T18:44:13.425429+0000","last_fullsized":"2023-06-10T18:44:13.425429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:16.618462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424932+0000","last_change":"2023-06-10T18:44:08.975414+0000","last_active":"2023-06-10T18:44:13.424932+0000","last_peered":"2023-06-10T18:44:13.424932+0000","last_clean":"2023-06-10T18:44:13.424932+0000","last_became_active":"2023-06-10T18:44:08.975156+0000","last_became_peered":"2023-06-10T18:44:08.975156+0000","last_unstale":"2023-06-10T18:44:13.424932+0000","last_undegraded":"2023-06-10T18:44:13.424932+0000","last_fullsized":"2023-06-10T18:44:13.424932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393972+0000","last_change":"2023-06-10T18:44:13.007025+0000","last_active":"2023-06-10T18:44:13.393972+0000","last_peered":"2023-06-10T18:44:13.393972+0000","last_clean":"2023-06-10T18:44:13.393972+0000","last_became_active":"2023-06-10T18:44:13.006654+0000","last_became_peered":"2023-06-10T18:44:13.006654+0000","last_unstale":"2023-06-10T18:44:13.393972+0000","last_undegraded":"2023-06-10T18:44:13.393972+0000","last_fullsized":"2023-06-10T18:44:13.393972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:27.677403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423673+0000","last_change":"2023-06-10T18:44:08.575515+0000","last_active":"2023-06-10T18:44:13.423673+0000","last_peered":"2023-06-10T18:44:13.423673+0000","last_clean":"2023-06-10T18:44:13.423673+0000","last_became_active":"2023-06-10T18:44:08.573539+0000","last_became_peered":"2023-06-10T18:44:08.573539+0000","last_unstale":"2023-06-10T18:44:13.423673+0000","last_undegraded":"2023-06-10T18:44:13.423673+0000","last_fullsized":"2023-06-10T18:44:13.423673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:57.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422973+0000","last_change":"2023-06-10T18:44:08.993519+0000","last_active":"2023-06-10T18:44:13.422973+0000","last_peered":"2023-06-10T18:44:13.422973+0000","last_clean":"2023-06-10T18:44:13.422973+0000","last_became_active":"2023-06-10T18:44:08.993146+0000","last_became_peered":"2023-06-10T18:44:08.993146+0000","last_unstale":"2023-06-10T18:44:13.422973+0000","last_undegraded":"2023-06-10T18:44:13.422973+0000","last_fullsized":"2023-06-10T18:44:13.422973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422934+0000","last_change":"2023-06-10T18:44:08.989855+0000","last_active":"2023-06-10T18:44:13.422934+0000","last_peered":"2023-06-10T18:44:13.422934+0000","last_clean":"2023-06-10T18:44:13.422934+0000","last_became_active":"2023-06-10T18:44:08.989140+0000","last_became_peered":"2023-06-10T18:44:08.989140+0000","last_unstale":"2023-06-10T18:44:13.422934+0000","last_undegraded":"2023-06-10T18:44:13.422934+0000","last_fullsized":"2023-06-10T18:44:13.422934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:14.496637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425109+0000","last_change":"2023-06-10T18:44:08.992461+0000","last_active":"2023-06-10T18:44:13.425109+0000","last_peered":"2023-06-10T18:44:13.425109+0000","last_clean":"2023-06-10T18:44:13.425109+0000","last_became_active":"2023-06-10T18:44:08.992160+0000","last_became_peered":"2023-06-10T18:44:08.992160+0000","last_unstale":"2023-06-10T18:44:13.425109+0000","last_undegraded":"2023-06-10T18:44:13.425109+0000","last_fullsized":"2023-06-10T18:44:13.425109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:42.310678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423600+0000","last_change":"2023-06-10T18:44:08.991382+0000","last_active":"2023-06-10T18:44:13.423600+0000","last_peered":"2023-06-10T18:44:13.423600+0000","last_clean":"2023-06-10T18:44:13.423600+0000","last_became_active":"2023-06-10T18:44:08.991122+0000","last_became_peered":"2023-06-10T18:44:08.991122+0000","last_unstale":"2023-06-10T18:44:13.423600+0000","last_undegraded":"2023-06-10T18:44:13.423600+0000","last_fullsized":"2023-06-10T18:44:13.423600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:10.451541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425330+0000","last_change":"2023-06-10T18:44:08.987112+0000","last_active":"2023-06-10T18:44:13.425330+0000","last_peered":"2023-06-10T18:44:13.425330+0000","last_clean":"2023-06-10T18:44:13.425330+0000","last_became_active":"2023-06-10T18:44:08.986804+0000","last_became_peered":"2023-06-10T18:44:08.986804+0000","last_unstale":"2023-06-10T18:44:13.425330+0000","last_undegraded":"2023-06-10T18:44:13.425330+0000","last_fullsized":"2023-06-10T18:44:13.425330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394328+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.394328+0000","last_peered":"2023-06-10T18:44:13.394328+0000","last_clean":"2023-06-10T18:44:13.394328+0000","last_became_active":"2023-06-10T18:44:08.987510+0000","last_became_peered":"2023-06-10T18:44:08.987510+0000","last_unstale":"2023-06-10T18:44:13.394328+0000","last_undegraded":"2023-06-10T18:44:13.394328+0000","last_fullsized":"2023-06-10T18:44:13.394328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":913408,"data_stored":714279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":884736,"data_stored":709832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":122732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:44:32.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:44:32.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 Processor -- start 2023-06-10T18:44:32.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 -- start start 2023-06-10T18:44:32.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc15a250 0x7f4fbc0cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fbc0ccca0 con 0x7f4fbc0cad90 2023-06-10T18:44:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fbc0d0310 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fc1bff700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0cee50 con 0x7f4fbc15a250 2023-06-10T18:44:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.783+0000 7f4fbb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54692/0 (socket says 172.21.15.104:54692) 2023-06-10T18:44:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 -- 172.21.15.104:0/1104655399 learned_addr learned my addr 172.21.15.104:0/1104655399 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbaffd700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc15a250 0x7f4fbc0cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 -- 172.21.15.104:0/1104655399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc15a250 msgr2=0x7f4fbc0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc15a250 0x7f4fbc0cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 -- 172.21.15.104:0/1104655399 >> v1:172.21.15.114:6789/0 conn(0x7f4fbc0cad90 legacy=0x7f4fbc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 -- 172.21.15.104:0/1104655399 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0cf130 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fbb7fe700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0098f0 tx=0x7f4fac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7cf028b5042fd1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fba7fc700 1 -- 172.21.15.104:0/1104655399 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00b8a0 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fba7fc700 1 -- 172.21.15.104:0/1104655399 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac00ba00 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.784+0000 7f4fba7fc700 1 -- 172.21.15.104:0/1104655399 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00bd90 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 -- 172.21.15.104:0/1104655399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc0cfa00 msgr2=0x7f4fbc0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0098f0 tx=0x7f4fac009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 -- 172.21.15.104:0/1104655399 shutdown_connections 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc0701e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/1104655399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc15a250 0x7f4fbc0cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 -- 172.21.15.104:0/1104655399 >> 172.21.15.104:0/1104655399 conn(0x7f4fbc06e370 msgr2=0x7f4fbc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:32.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 -- 172.21.15.104:0/1104655399 shutdown_connections 2023-06-10T18:44:32.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.785+0000 7f4fc1bff700 1 -- 172.21.15.104:0/1104655399 wait complete. 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.786+0000 7f4fc1bff700 1 Processor -- start 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.786+0000 7f4fc1bff700 1 -- start start 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc0cad90 0x7f4fbc0b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc15b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc06b2d0 con 0x7f4fbc15a250 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc15b730 con 0x7f4fbc0cfa00 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc06af80 con 0x7f4fbc0cad90 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54702/0 (socket says 172.21.15.104:54702) 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 -- 172.21.15.104:0/2452954733 learned_addr learned my addr 172.21.15.104:0/2452954733 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbb7fe700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc0cad90 0x7f4fbc0b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbaffd700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc15b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 -- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc0cad90 msgr2=0x7f4fbc0b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc0cad90 0x7f4fbc0b0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 -- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fbc0cfa00 msgr2=0x7f4fbc15b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc15b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 -- 172.21.15.104:0/2452954733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009550 con 0x7f4fbc15a250 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fbbfff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400cb50 tx=0x7f4fb400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4003c00 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb4003d60 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.787+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fbc15b960 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.788+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400aa20 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.788+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc15bf20 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.788+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fbc074050 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.789+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4fb400ab80 con 0x7f4fbc15a250 2023-06-10T18:44:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.789+0000 7f4fb3fff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 0x7f4f9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:32.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.790+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4fb4089690 con 0x7f4fbc15a250 2023-06-10T18:44:32.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.790+0000 7f4fbb7fe700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 0x7f4f9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:32.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.790+0000 7f4fbb7fe700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 0x7f4f9c06d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009c40 tx=0x7f4fac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:32.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.794+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4fb40544e0 con 0x7f4fbc15a250 2023-06-10T18:44:32.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.974+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4fbc0cf760 con 0x7f4f9c06ace0 2023-06-10T18:44:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.986+0000 7f4fb3fff700 1 -- 172.21.15.104:0/2452954733 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367290 (secure 0 0 0) 0x7f4fac00bb70 con 0x7f4f9c06ace0 2023-06-10T18:44:32.995 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:32.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 msgr2=0x7f4f9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 0x7f4f9c06d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009c40 tx=0x7f4fac008030 comp rx=0 tx=0).stop 2023-06-10T18:44:32.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 msgr2=0x7f4fbc14eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:32.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400cb50 tx=0x7f4fb400ce60 comp rx=0 tx=0).stop 2023-06-10T18:44:32.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 shutdown_connections 2023-06-10T18:44:32.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fbc15a250 0x7f4fbc14eca0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fbc0cad90 0x7f4fbc0b0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f9c06ace0 0x7f4f9c06d190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:32.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 --2- 172.21.15.104:0/2452954733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fbc0cfa00 0x7f4fbc15b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.989+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 >> 172.21.15.104:0/2452954733 conn(0x7f4fbc06e370 msgr2=0x7f4fbc159290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.990+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 shutdown_connections 2023-06-10T18:44:33.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:32.990+0000 7f4fc1bff700 1 -- 172.21.15.104:0/2452954733 wait complete. 2023-06-10T18:44:33.000 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:44:33.007 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-06-10T18:44:31.353446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":3480,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":3563520,"data_stored":2145052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423396+0000","last_change":"2023-06-10T18:44:08.974320+0000","last_active":"2023-06-10T18:44:13.423396+0000","last_peered":"2023-06-10T18:44:13.423396+0000","last_clean":"2023-06-10T18:44:13.423396+0000","last_became_active":"2023-06-10T18:44:08.973998+0000","last_became_peered":"2023-06-10T18:44:08.973998+0000","last_unstale":"2023-06-10T18:44:13.423396+0000","last_undegraded":"2023-06-10T18:44:13.423396+0000","last_fullsized":"2023-06-10T18:44:13.423396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:29.774977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393150+0000","last_change":"2023-06-10T18:44:08.986524+0000","last_active":"2023-06-10T18:44:13.393150+0000","last_peered":"2023-06-10T18:44:13.393150+0000","last_clean":"2023-06-10T18:44:13.393150+0000","last_became_active":"2023-06-10T18:44:08.986238+0000","last_became_peered":"2023-06-10T18:44:08.986238+0000","last_unstale":"2023-06-10T18:44:13.393150+0000","last_undegraded":"2023-06-10T18:44:13.393150+0000","last_fullsized":"2023-06-10T18:44:13.393150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423472+0000","last_change":"2023-06-10T18:44:08.991803+0000","last_active":"2023-06-10T18:44:13.423472+0000","last_peered":"2023-06-10T18:44:13.423472+0000","last_clean":"2023-06-10T18:44:13.423472+0000","last_became_active":"2023-06-10T18:44:08.986777+0000","last_became_peered":"2023-06-10T18:44:08.986777+0000","last_unstale":"2023-06-10T18:44:13.423472+0000","last_undegraded":"2023-06-10T18:44:13.423472+0000","last_fullsized":"2023-06-10T18:44:13.423472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423103+0000","last_change":"2023-06-10T18:44:08.984677+0000","last_active":"2023-06-10T18:44:13.423103+0000","last_peered":"2023-06-10T18:44:13.423103+0000","last_clean":"2023-06-10T18:44:13.423103+0000","last_became_active":"2023-06-10T18:44:08.984368+0000","last_became_peered":"2023-06-10T18:44:08.984368+0000","last_unstale":"2023-06-10T18:44:13.423103+0000","last_undegraded":"2023-06-10T18:44:13.423103+0000","last_fullsized":"2023-06-10T18:44:13.423103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:21.561425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423951+0000","last_change":"2023-06-10T18:44:08.988022+0000","last_active":"2023-06-10T18:44:13.423951+0000","last_peered":"2023-06-10T18:44:13.423951+0000","last_clean":"2023-06-10T18:44:13.423951+0000","last_became_active":"2023-06-10T18:44:08.983186+0000","last_became_peered":"2023-06-10T18:44:08.983186+0000","last_unstale":"2023-06-10T18:44:13.423951+0000","last_undegraded":"2023-06-10T18:44:13.423951+0000","last_fullsized":"2023-06-10T18:44:13.423951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:24.256200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423368+0000","last_change":"2023-06-10T18:44:08.986749+0000","last_active":"2023-06-10T18:44:13.423368+0000","last_peered":"2023-06-10T18:44:13.423368+0000","last_clean":"2023-06-10T18:44:13.423368+0000","last_became_active":"2023-06-10T18:44:08.986456+0000","last_became_peered":"2023-06-10T18:44:08.986456+0000","last_unstale":"2023-06-10T18:44:13.423368+0000","last_undegraded":"2023-06-10T18:44:13.423368+0000","last_fullsized":"2023-06-10T18:44:13.423368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394293+0000","last_change":"2023-06-10T18:44:08.992850+0000","last_active":"2023-06-10T18:44:13.394293+0000","last_peered":"2023-06-10T18:44:13.394293+0000","last_clean":"2023-06-10T18:44:13.394293+0000","last_became_active":"2023-06-10T18:44:08.992572+0000","last_became_peered":"2023-06-10T18:44:08.992572+0000","last_unstale":"2023-06-10T18:44:13.394293+0000","last_undegraded":"2023-06-10T18:44:13.394293+0000","last_fullsized":"2023-06-10T18:44:13.394293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392634+0000","last_change":"2023-06-10T18:44:08.979428+0000","last_active":"2023-06-10T18:44:13.392634+0000","last_peered":"2023-06-10T18:44:13.392634+0000","last_clean":"2023-06-10T18:44:13.392634+0000","last_became_active":"2023-06-10T18:44:08.979157+0000","last_became_peered":"2023-06-10T18:44:08.979157+0000","last_unstale":"2023-06-10T18:44:13.392634+0000","last_undegraded":"2023-06-10T18:44:13.392634+0000","last_fullsized":"2023-06-10T18:44:13.392634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:24.114600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423251+0000","last_change":"2023-06-10T18:44:08.984436+0000","last_active":"2023-06-10T18:44:13.423251+0000","last_peered":"2023-06-10T18:44:13.423251+0000","last_clean":"2023-06-10T18:44:13.423251+0000","last_became_active":"2023-06-10T18:44:08.984021+0000","last_became_peered":"2023-06-10T18:44:08.984021+0000","last_unstale":"2023-06-10T18:44:13.423251+0000","last_undegraded":"2023-06-10T18:44:13.423251+0000","last_fullsized":"2023-06-10T18:44:13.423251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:23:54.181578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393491+0000","last_change":"2023-06-10T18:44:08.984325+0000","last_active":"2023-06-10T18:44:13.393491+0000","last_peered":"2023-06-10T18:44:13.393491+0000","last_clean":"2023-06-10T18:44:13.393491+0000","last_became_active":"2023-06-10T18:44:08.983999+0000","last_became_peered":"2023-06-10T18:44:08.983999+0000","last_unstale":"2023-06-10T18:44:13.393491+0000","last_undegraded":"2023-06-10T18:44:13.393491+0000","last_fullsized":"2023-06-10T18:44:13.393491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423627+0000","last_change":"2023-06-10T18:44:08.991961+0000","last_active":"2023-06-10T18:44:13.423627+0000","last_peered":"2023-06-10T18:44:13.423627+0000","last_clean":"2023-06-10T18:44:13.423627+0000","last_became_active":"2023-06-10T18:44:08.991527+0000","last_became_peered":"2023-06-10T18:44:08.991527+0000","last_unstale":"2023-06-10T18:44:13.423627+0000","last_undegraded":"2023-06-10T18:44:13.423627+0000","last_fullsized":"2023-06-10T18:44:13.423627+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423011+0000","last_change":"2023-06-10T18:44:08.995250+0000","last_active":"2023-06-10T18:44:13.423011+0000","last_peered":"2023-06-10T18:44:13.423011+0000","last_clean":"2023-06-10T18:44:13.423011+0000","last_became_active":"2023-06-10T18:44:08.994854+0000","last_became_peered":"2023-06-10T18:44:08.994854+0000","last_unstale":"2023-06-10T18:44:13.423011+0000","last_undegraded":"2023-06-10T18:44:13.423011+0000","last_fullsized":"2023-06-10T18:44:13.423011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:13:54.715925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422801+0000","last_change":"2023-06-10T18:44:08.973692+0000","last_active":"2023-06-10T18:44:13.422801+0000","last_peered":"2023-06-10T18:44:13.422801+0000","last_clean":"2023-06-10T18:44:13.422801+0000","last_became_active":"2023-06-10T18:44:08.973417+0000","last_became_peered":"2023-06-10T18:44:08.973417+0000","last_unstale":"2023-06-10T18:44:13.422801+0000","last_undegraded":"2023-06-10T18:44:13.422801+0000","last_fullsized":"2023-06-10T18:44:13.422801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:27:20.138188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392896+0000","last_change":"2023-06-10T18:44:08.976711+0000","last_active":"2023-06-10T18:44:13.392896+0000","last_peered":"2023-06-10T18:44:13.392896+0000","last_clean":"2023-06-10T18:44:13.392896+0000","last_became_active":"2023-06-10T18:44:08.976415+0000","last_became_peered":"2023-06-10T18:44:08.976415+0000","last_unstale":"2023-06-10T18:44:13.392896+0000","last_undegraded":"2023-06-10T18:44:13.392896+0000","last_fullsized":"2023-06-10T18:44:13.392896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394066+0000","last_change":"2023-06-10T18:44:08.992788+0000","last_active":"2023-06-10T18:44:13.394066+0000","last_peered":"2023-06-10T18:44:13.394066+0000","last_clean":"2023-06-10T18:44:13.394066+0000","last_became_active":"2023-06-10T18:44:08.992335+0000","last_became_peered":"2023-06-10T18:44:08.992335+0000","last_unstale":"2023-06-10T18:44:13.394066+0000","last_undegraded":"2023-06-10T18:44:13.394066+0000","last_fullsized":"2023-06-10T18:44:13.394066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423080+0000","last_change":"2023-06-10T18:44:08.993080+0000","last_active":"2023-06-10T18:44:13.423080+0000","last_peered":"2023-06-10T18:44:13.423080+0000","last_clean":"2023-06-10T18:44:13.423080+0000","last_became_active":"2023-06-10T18:44:08.992794+0000","last_became_peered":"2023-06-10T18:44:08.992794+0000","last_unstale":"2023-06-10T18:44:13.423080+0000","last_undegraded":"2023-06-10T18:44:13.423080+0000","last_fullsized":"2023-06-10T18:44:13.423080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423378+0000","last_change":"2023-06-10T18:44:08.984915+0000","last_active":"2023-06-10T18:44:13.423378+0000","last_peered":"2023-06-10T18:44:13.423378+0000","last_clean":"2023-06-10T18:44:13.423378+0000","last_became_active":"2023-06-10T18:44:08.984639+0000","last_became_peered":"2023-06-10T18:44:08.984639+0000","last_unstale":"2023-06-10T18:44:13.423378+0000","last_undegraded":"2023-06-10T18:44:13.423378+0000","last_fullsized":"2023-06-10T18:44:13.423378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:53.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424069+0000","last_change":"2023-06-10T18:44:08.989447+0000","last_active":"2023-06-10T18:44:13.424069+0000","last_peered":"2023-06-10T18:44:13.424069+0000","last_clean":"2023-06-10T18:44:13.424069+0000","last_became_active":"2023-06-10T18:44:08.987688+0000","last_became_peered":"2023-06-10T18:44:08.987688+0000","last_unstale":"2023-06-10T18:44:13.424069+0000","last_undegraded":"2023-06-10T18:44:13.424069+0000","last_fullsized":"2023-06-10T18:44:13.424069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422601+0000","last_change":"2023-06-10T18:44:08.984545+0000","last_active":"2023-06-10T18:44:13.422601+0000","last_peered":"2023-06-10T18:44:13.422601+0000","last_clean":"2023-06-10T18:44:13.422601+0000","last_became_active":"2023-06-10T18:44:08.983857+0000","last_became_peered":"2023-06-10T18:44:08.983857+0000","last_unstale":"2023-06-10T18:44:13.422601+0000","last_undegraded":"2023-06-10T18:44:13.422601+0000","last_fullsized":"2023-06-10T18:44:13.422601+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:46.973658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423818+0000","last_change":"2023-06-10T18:44:08.993840+0000","last_active":"2023-06-10T18:44:13.423818+0000","last_peered":"2023-06-10T18:44:13.423818+0000","last_clean":"2023-06-10T18:44:13.423818+0000","last_became_active":"2023-06-10T18:44:08.993524+0000","last_became_peered":"2023-06-10T18:44:08.993524+0000","last_unstale":"2023-06-10T18:44:13.423818+0000","last_undegraded":"2023-06-10T18:44:13.423818+0000","last_fullsized":"2023-06-10T18:44:13.423818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393980+0000","last_change":"2023-06-10T18:44:08.994356+0000","last_active":"2023-06-10T18:44:13.393980+0000","last_peered":"2023-06-10T18:44:13.393980+0000","last_clean":"2023-06-10T18:44:13.393980+0000","last_became_active":"2023-06-10T18:44:08.994055+0000","last_became_peered":"2023-06-10T18:44:08.994055+0000","last_unstale":"2023-06-10T18:44:13.393980+0000","last_undegraded":"2023-06-10T18:44:13.393980+0000","last_fullsized":"2023-06-10T18:44:13.393980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423446+0000","last_change":"2023-06-10T18:44:08.988865+0000","last_active":"2023-06-10T18:44:13.423446+0000","last_peered":"2023-06-10T18:44:13.423446+0000","last_clean":"2023-06-10T18:44:13.423446+0000","last_became_active":"2023-06-10T18:44:08.988631+0000","last_became_peered":"2023-06-10T18:44:08.988631+0000","last_unstale":"2023-06-10T18:44:13.423446+0000","last_undegraded":"2023-06-10T18:44:13.423446+0000","last_fullsized":"2023-06-10T18:44:13.423446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:18.367120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424783+0000","last_change":"2023-06-10T18:44:08.985010+0000","last_active":"2023-06-10T18:44:13.424783+0000","last_peered":"2023-06-10T18:44:13.424783+0000","last_clean":"2023-06-10T18:44:13.424783+0000","last_became_active":"2023-06-10T18:44:08.984770+0000","last_became_peered":"2023-06-10T18:44:08.984770+0000","last_unstale":"2023-06-10T18:44:13.424783+0000","last_undegraded":"2023-06-10T18:44:13.424783+0000","last_fullsized":"2023-06-10T18:44:13.424783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394283+0000","last_change":"2023-06-10T18:44:08.994280+0000","last_active":"2023-06-10T18:44:13.394283+0000","last_peered":"2023-06-10T18:44:13.394283+0000","last_clean":"2023-06-10T18:44:13.394283+0000","last_became_active":"2023-06-10T18:44:08.993996+0000","last_became_peered":"2023-06-10T18:44:08.993996+0000","last_unstale":"2023-06-10T18:44:13.394283+0000","last_undegraded":"2023-06-10T18:44:13.394283+0000","last_fullsized":"2023-06-10T18:44:13.394283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394507+0000","last_change":"2023-06-10T18:44:08.988231+0000","last_active":"2023-06-10T18:44:13.394507+0000","last_peered":"2023-06-10T18:44:13.394507+0000","last_clean":"2023-06-10T18:44:13.394507+0000","last_became_active":"2023-06-10T18:44:08.987866+0000","last_became_peered":"2023-06-10T18:44:08.987866+0000","last_unstale":"2023-06-10T18:44:13.394507+0000","last_undegraded":"2023-06-10T18:44:13.394507+0000","last_fullsized":"2023-06-10T18:44:13.394507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394552+0000","last_change":"2023-06-10T18:44:08.984357+0000","last_active":"2023-06-10T18:44:13.394552+0000","last_peered":"2023-06-10T18:44:13.394552+0000","last_clean":"2023-06-10T18:44:13.394552+0000","last_became_active":"2023-06-10T18:44:08.984034+0000","last_became_peered":"2023-06-10T18:44:08.984034+0000","last_unstale":"2023-06-10T18:44:13.394552+0000","last_undegraded":"2023-06-10T18:44:13.394552+0000","last_fullsized":"2023-06-10T18:44:13.394552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393451+0000","last_change":"2023-06-10T18:44:08.987489+0000","last_active":"2023-06-10T18:44:13.393451+0000","last_peered":"2023-06-10T18:44:13.393451+0000","last_clean":"2023-06-10T18:44:13.393451+0000","last_became_active":"2023-06-10T18:44:08.985804+0000","last_became_peered":"2023-06-10T18:44:08.985804+0000","last_unstale":"2023-06-10T18:44:13.393451+0000","last_undegraded":"2023-06-10T18:44:13.393451+0000","last_fullsized":"2023-06-10T18:44:13.393451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424186+0000","last_change":"2023-06-10T18:44:13.001999+0000","last_active":"2023-06-10T18:44:13.424186+0000","last_peered":"2023-06-10T18:44:13.424186+0000","last_clean":"2023-06-10T18:44:13.424186+0000","last_became_active":"2023-06-10T18:44:13.001637+0000","last_became_peered":"2023-06-10T18:44:13.001637+0000","last_unstale":"2023-06-10T18:44:13.424186+0000","last_undegraded":"2023-06-10T18:44:13.424186+0000","last_fullsized":"2023-06-10T18:44:13.424186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423197+0000","last_change":"2023-06-10T18:44:08.575638+0000","last_active":"2023-06-10T18:44:13.423197+0000","last_peered":"2023-06-10T18:44:13.423197+0000","last_clean":"2023-06-10T18:44:13.423197+0000","last_became_active":"2023-06-10T18:44:08.575345+0000","last_became_peered":"2023-06-10T18:44:08.575345+0000","last_unstale":"2023-06-10T18:44:13.423197+0000","last_undegraded":"2023-06-10T18:44:13.423197+0000","last_fullsized":"2023-06-10T18:44:13.423197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:11.022238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425003+0000","last_change":"2023-06-10T18:44:08.986591+0000","last_active":"2023-06-10T18:44:13.425003+0000","last_peered":"2023-06-10T18:44:13.425003+0000","last_clean":"2023-06-10T18:44:13.425003+0000","last_became_active":"2023-06-10T18:44:08.986320+0000","last_became_peered":"2023-06-10T18:44:08.986320+0000","last_unstale":"2023-06-10T18:44:13.425003+0000","last_undegraded":"2023-06-10T18:44:13.425003+0000","last_fullsized":"2023-06-10T18:44:13.425003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424063+0000","last_change":"2023-06-10T18:44:13.002454+0000","last_active":"2023-06-10T18:44:13.424063+0000","last_peered":"2023-06-10T18:44:13.424063+0000","last_clean":"2023-06-10T18:44:13.424063+0000","last_became_active":"2023-06-10T18:44:13.002101+0000","last_became_peered":"2023-06-10T18:44:13.002101+0000","last_unstale":"2023-06-10T18:44:13.424063+0000","last_undegraded":"2023-06-10T18:44:13.424063+0000","last_fullsized":"2023-06-10T18:44:13.424063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:38.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393702+0000","last_change":"2023-06-10T18:44:08.581341+0000","last_active":"2023-06-10T18:44:13.393702+0000","last_peered":"2023-06-10T18:44:13.393702+0000","last_clean":"2023-06-10T18:44:13.393702+0000","last_became_active":"2023-06-10T18:44:08.580977+0000","last_became_peered":"2023-06-10T18:44:08.580977+0000","last_unstale":"2023-06-10T18:44:13.393702+0000","last_undegraded":"2023-06-10T18:44:13.393702+0000","last_fullsized":"2023-06-10T18:44:13.393702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423101+0000","last_change":"2023-06-10T18:44:08.990386+0000","last_active":"2023-06-10T18:44:13.423101+0000","last_peered":"2023-06-10T18:44:13.423101+0000","last_clean":"2023-06-10T18:44:13.423101+0000","last_became_active":"2023-06-10T18:44:08.989816+0000","last_became_peered":"2023-06-10T18:44:08.989816+0000","last_unstale":"2023-06-10T18:44:13.423101+0000","last_undegraded":"2023-06-10T18:44:13.423101+0000","last_fullsized":"2023-06-10T18:44:13.423101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:27.419215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422737+0000","last_change":"2023-06-10T18:44:13.011429+0000","last_active":"2023-06-10T18:44:13.422737+0000","last_peered":"2023-06-10T18:44:13.422737+0000","last_clean":"2023-06-10T18:44:13.422737+0000","last_became_active":"2023-06-10T18:44:13.010928+0000","last_became_peered":"2023-06-10T18:44:13.010928+0000","last_unstale":"2023-06-10T18:44:13.422737+0000","last_undegraded":"2023-06-10T18:44:13.422737+0000","last_fullsized":"2023-06-10T18:44:13.422737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:25:46.759866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425044+0000","last_change":"2023-06-10T18:44:08.583484+0000","last_active":"2023-06-10T18:44:13.425044+0000","last_peered":"2023-06-10T18:44:13.425044+0000","last_clean":"2023-06-10T18:44:13.425044+0000","last_became_active":"2023-06-10T18:44:08.583197+0000","last_became_peered":"2023-06-10T18:44:08.583197+0000","last_unstale":"2023-06-10T18:44:13.425044+0000","last_undegraded":"2023-06-10T18:44:13.425044+0000","last_fullsized":"2023-06-10T18:44:13.425044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:33.286044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424162+0000","last_change":"2023-06-10T18:44:08.983580+0000","last_active":"2023-06-10T18:44:13.424162+0000","last_peered":"2023-06-10T18:44:13.424162+0000","last_clean":"2023-06-10T18:44:13.424162+0000","last_became_active":"2023-06-10T18:44:08.983254+0000","last_became_peered":"2023-06-10T18:44:08.983254+0000","last_unstale":"2023-06-10T18:44:13.424162+0000","last_undegraded":"2023-06-10T18:44:13.424162+0000","last_fullsized":"2023-06-10T18:44:13.424162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:53:36.396900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423826+0000","last_change":"2023-06-10T18:44:12.997904+0000","last_active":"2023-06-10T18:44:13.423826+0000","last_peered":"2023-06-10T18:44:13.423826+0000","last_clean":"2023-06-10T18:44:13.423826+0000","last_became_active":"2023-06-10T18:44:12.997502+0000","last_became_peered":"2023-06-10T18:44:12.997502+0000","last_unstale":"2023-06-10T18:44:13.423826+0000","last_undegraded":"2023-06-10T18:44:13.423826+0000","last_fullsized":"2023-06-10T18:44:13.423826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:17.892337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423483+0000","last_change":"2023-06-10T18:44:08.575792+0000","last_active":"2023-06-10T18:44:13.423483+0000","last_peered":"2023-06-10T18:44:13.423483+0000","last_clean":"2023-06-10T18:44:13.423483+0000","last_became_active":"2023-06-10T18:44:08.573780+0000","last_became_peered":"2023-06-10T18:44:08.573780+0000","last_unstale":"2023-06-10T18:44:13.423483+0000","last_undegraded":"2023-06-10T18:44:13.423483+0000","last_fullsized":"2023-06-10T18:44:13.423483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:19.709371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424910+0000","last_change":"2023-06-10T18:44:08.986631+0000","last_active":"2023-06-10T18:44:13.424910+0000","last_peered":"2023-06-10T18:44:13.424910+0000","last_clean":"2023-06-10T18:44:13.424910+0000","last_became_active":"2023-06-10T18:44:08.986357+0000","last_became_peered":"2023-06-10T18:44:08.986357+0000","last_unstale":"2023-06-10T18:44:13.424910+0000","last_undegraded":"2023-06-10T18:44:13.424910+0000","last_fullsized":"2023-06-10T18:44:13.424910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393507+0000","last_change":"2023-06-10T18:44:13.002734+0000","last_active":"2023-06-10T18:44:13.393507+0000","last_peered":"2023-06-10T18:44:13.393507+0000","last_clean":"2023-06-10T18:44:13.393507+0000","last_became_active":"2023-06-10T18:44:13.002226+0000","last_became_peered":"2023-06-10T18:44:13.002226+0000","last_unstale":"2023-06-10T18:44:13.393507+0000","last_undegraded":"2023-06-10T18:44:13.393507+0000","last_fullsized":"2023-06-10T18:44:13.393507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:11.264615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393372+0000","last_change":"2023-06-10T18:44:08.571167+0000","last_active":"2023-06-10T18:44:13.393372+0000","last_peered":"2023-06-10T18:44:13.393372+0000","last_clean":"2023-06-10T18:44:13.393372+0000","last_became_active":"2023-06-10T18:44:08.570885+0000","last_became_peered":"2023-06-10T18:44:08.570885+0000","last_unstale":"2023-06-10T18:44:13.393372+0000","last_undegraded":"2023-06-10T18:44:13.393372+0000","last_fullsized":"2023-06-10T18:44:13.393372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.984523+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.984265+0000","last_became_peered":"2023-06-10T18:44:08.984265+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425522+0000","last_change":"2023-06-10T18:44:13.005254+0000","last_active":"2023-06-10T18:44:13.425522+0000","last_peered":"2023-06-10T18:44:13.425522+0000","last_clean":"2023-06-10T18:44:13.425522+0000","last_became_active":"2023-06-10T18:44:13.004908+0000","last_became_peered":"2023-06-10T18:44:13.004908+0000","last_unstale":"2023-06-10T18:44:13.425522+0000","last_undegraded":"2023-06-10T18:44:13.425522+0000","last_fullsized":"2023-06-10T18:44:13.425522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:23.228638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394120+0000","last_change":"2023-06-10T18:44:08.577421+0000","last_active":"2023-06-10T18:44:13.394120+0000","last_peered":"2023-06-10T18:44:13.394120+0000","last_clean":"2023-06-10T18:44:13.394120+0000","last_became_active":"2023-06-10T18:44:08.577127+0000","last_became_peered":"2023-06-10T18:44:08.577127+0000","last_unstale":"2023-06-10T18:44:13.394120+0000","last_undegraded":"2023-06-10T18:44:13.394120+0000","last_fullsized":"2023-06-10T18:44:13.394120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:14.393845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393966+0000","last_change":"2023-06-10T18:44:08.986118+0000","last_active":"2023-06-10T18:44:13.393966+0000","last_peered":"2023-06-10T18:44:13.393966+0000","last_clean":"2023-06-10T18:44:13.393966+0000","last_became_active":"2023-06-10T18:44:08.985845+0000","last_became_peered":"2023-06-10T18:44:08.985845+0000","last_unstale":"2023-06-10T18:44:13.393966+0000","last_undegraded":"2023-06-10T18:44:13.393966+0000","last_fullsized":"2023-06-10T18:44:13.393966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393763+0000","last_change":"2023-06-10T18:44:13.009330+0000","last_active":"2023-06-10T18:44:13.393763+0000","last_peered":"2023-06-10T18:44:13.393763+0000","last_clean":"2023-06-10T18:44:13.393763+0000","last_became_active":"2023-06-10T18:44:13.008968+0000","last_became_peered":"2023-06-10T18:44:13.008968+0000","last_unstale":"2023-06-10T18:44:13.393763+0000","last_undegraded":"2023-06-10T18:44:13.393763+0000","last_fullsized":"2023-06-10T18:44:13.393763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:01.454996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:18.388487+0000","last_change":"2023-06-10T18:44:08.577256+0000","last_active":"2023-06-10T18:44:18.388487+0000","last_peered":"2023-06-10T18:44:18.388487+0000","last_clean":"2023-06-10T18:44:18.388487+0000","last_became_active":"2023-06-10T18:44:08.576987+0000","last_became_peered":"2023-06-10T18:44:08.576987+0000","last_unstale":"2023-06-10T18:44:18.388487+0000","last_undegraded":"2023-06-10T18:44:18.388487+0000","last_fullsized":"2023-06-10T18:44:18.388487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424232+0000","last_change":"2023-06-10T18:44:08.989046+0000","last_active":"2023-06-10T18:44:13.424232+0000","last_peered":"2023-06-10T18:44:13.424232+0000","last_clean":"2023-06-10T18:44:13.424232+0000","last_became_active":"2023-06-10T18:44:08.988763+0000","last_became_peered":"2023-06-10T18:44:08.988763+0000","last_unstale":"2023-06-10T18:44:13.424232+0000","last_undegraded":"2023-06-10T18:44:13.424232+0000","last_fullsized":"2023-06-10T18:44:13.424232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393726+0000","last_change":"2023-06-10T18:44:13.009976+0000","last_active":"2023-06-10T18:44:13.393726+0000","last_peered":"2023-06-10T18:44:13.393726+0000","last_clean":"2023-06-10T18:44:13.393726+0000","last_became_active":"2023-06-10T18:44:13.009599+0000","last_became_peered":"2023-06-10T18:44:13.009599+0000","last_unstale":"2023-06-10T18:44:13.393726+0000","last_undegraded":"2023-06-10T18:44:13.393726+0000","last_fullsized":"2023-06-10T18:44:13.393726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393682+0000","last_change":"2023-06-10T18:44:08.578965+0000","last_active":"2023-06-10T18:44:13.393682+0000","last_peered":"2023-06-10T18:44:13.393682+0000","last_clean":"2023-06-10T18:44:13.393682+0000","last_became_active":"2023-06-10T18:44:08.578666+0000","last_became_peered":"2023-06-10T18:44:08.578666+0000","last_unstale":"2023-06-10T18:44:13.393682+0000","last_undegraded":"2023-06-10T18:44:13.393682+0000","last_fullsized":"2023-06-10T18:44:13.393682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425838+0000","last_change":"2023-06-10T18:44:08.986550+0000","last_active":"2023-06-10T18:44:13.425838+0000","last_peered":"2023-06-10T18:44:13.425838+0000","last_clean":"2023-06-10T18:44:13.425838+0000","last_became_active":"2023-06-10T18:44:08.986267+0000","last_became_peered":"2023-06-10T18:44:08.986267+0000","last_unstale":"2023-06-10T18:44:13.425838+0000","last_undegraded":"2023-06-10T18:44:13.425838+0000","last_fullsized":"2023-06-10T18:44:13.425838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425693+0000","last_change":"2023-06-10T18:44:13.007347+0000","last_active":"2023-06-10T18:44:13.425693+0000","last_peered":"2023-06-10T18:44:13.425693+0000","last_clean":"2023-06-10T18:44:13.425693+0000","last_became_active":"2023-06-10T18:44:13.006948+0000","last_became_peered":"2023-06-10T18:44:13.006948+0000","last_unstale":"2023-06-10T18:44:13.425693+0000","last_undegraded":"2023-06-10T18:44:13.425693+0000","last_fullsized":"2023-06-10T18:44:13.425693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423561+0000","last_change":"2023-06-10T18:44:08.575263+0000","last_active":"2023-06-10T18:44:13.423561+0000","last_peered":"2023-06-10T18:44:13.423561+0000","last_clean":"2023-06-10T18:44:13.423561+0000","last_became_active":"2023-06-10T18:44:08.574716+0000","last_became_peered":"2023-06-10T18:44:08.574716+0000","last_unstale":"2023-06-10T18:44:13.423561+0000","last_undegraded":"2023-06-10T18:44:13.423561+0000","last_fullsized":"2023-06-10T18:44:13.423561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393754+0000","last_change":"2023-06-10T18:44:08.986065+0000","last_active":"2023-06-10T18:44:13.393754+0000","last_peered":"2023-06-10T18:44:13.393754+0000","last_clean":"2023-06-10T18:44:13.393754+0000","last_became_active":"2023-06-10T18:44:08.984998+0000","last_became_peered":"2023-06-10T18:44:08.984998+0000","last_unstale":"2023-06-10T18:44:13.393754+0000","last_undegraded":"2023-06-10T18:44:13.393754+0000","last_fullsized":"2023-06-10T18:44:13.393754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425425+0000","last_change":"2023-06-10T18:44:13.004286+0000","last_active":"2023-06-10T18:44:13.425425+0000","last_peered":"2023-06-10T18:44:13.425425+0000","last_clean":"2023-06-10T18:44:13.425425+0000","last_became_active":"2023-06-10T18:44:13.003671+0000","last_became_peered":"2023-06-10T18:44:13.003671+0000","last_unstale":"2023-06-10T18:44:13.425425+0000","last_undegraded":"2023-06-10T18:44:13.425425+0000","last_fullsized":"2023-06-10T18:44:13.425425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394170+0000","last_change":"2023-06-10T18:44:08.578231+0000","last_active":"2023-06-10T18:44:13.394170+0000","last_peered":"2023-06-10T18:44:13.394170+0000","last_clean":"2023-06-10T18:44:13.394170+0000","last_became_active":"2023-06-10T18:44:08.577966+0000","last_became_peered":"2023-06-10T18:44:08.577966+0000","last_unstale":"2023-06-10T18:44:13.394170+0000","last_undegraded":"2023-06-10T18:44:13.394170+0000","last_fullsized":"2023-06-10T18:44:13.394170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423359+0000","last_change":"2023-06-10T18:44:08.990234+0000","last_active":"2023-06-10T18:44:13.423359+0000","last_peered":"2023-06-10T18:44:13.423359+0000","last_clean":"2023-06-10T18:44:13.423359+0000","last_became_active":"2023-06-10T18:44:08.989505+0000","last_became_peered":"2023-06-10T18:44:08.989505+0000","last_unstale":"2023-06-10T18:44:13.423359+0000","last_undegraded":"2023-06-10T18:44:13.423359+0000","last_fullsized":"2023-06-10T18:44:13.423359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:31.544241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393091+0000","last_change":"2023-06-10T18:44:13.009889+0000","last_active":"2023-06-10T18:44:13.393091+0000","last_peered":"2023-06-10T18:44:13.393091+0000","last_clean":"2023-06-10T18:44:13.393091+0000","last_became_active":"2023-06-10T18:44:13.009528+0000","last_became_peered":"2023-06-10T18:44:13.009528+0000","last_unstale":"2023-06-10T18:44:13.393091+0000","last_undegraded":"2023-06-10T18:44:13.393091+0000","last_fullsized":"2023-06-10T18:44:13.393091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:15.449838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":29,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422838+0000","last_change":"2023-06-10T18:44:08.573929+0000","last_active":"2023-06-10T18:44:13.422838+0000","last_peered":"2023-06-10T18:44:13.422838+0000","last_clean":"2023-06-10T18:44:13.422838+0000","last_became_active":"2023-06-10T18:44:08.573652+0000","last_became_peered":"2023-06-10T18:44:08.573652+0000","last_unstale":"2023-06-10T18:44:13.422838+0000","last_undegraded":"2023-06-10T18:44:13.422838+0000","last_fullsized":"2023-06-10T18:44:13.422838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:08.763880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425617+0000","last_change":"2023-06-10T18:44:08.984556+0000","last_active":"2023-06-10T18:44:13.425617+0000","last_peered":"2023-06-10T18:44:13.425617+0000","last_clean":"2023-06-10T18:44:13.425617+0000","last_became_active":"2023-06-10T18:44:08.984321+0000","last_became_peered":"2023-06-10T18:44:08.984321+0000","last_unstale":"2023-06-10T18:44:13.425617+0000","last_undegraded":"2023-06-10T18:44:13.425617+0000","last_fullsized":"2023-06-10T18:44:13.425617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394464+0000","last_change":"2023-06-10T18:44:13.006271+0000","last_active":"2023-06-10T18:44:13.394464+0000","last_peered":"2023-06-10T18:44:13.394464+0000","last_clean":"2023-06-10T18:44:13.394464+0000","last_became_active":"2023-06-10T18:44:13.005878+0000","last_became_peered":"2023-06-10T18:44:13.005878+0000","last_unstale":"2023-06-10T18:44:13.394464+0000","last_undegraded":"2023-06-10T18:44:13.394464+0000","last_fullsized":"2023-06-10T18:44:13.394464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:21.046063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394016+0000","last_change":"2023-06-10T18:44:08.578331+0000","last_active":"2023-06-10T18:44:13.394016+0000","last_peered":"2023-06-10T18:44:13.394016+0000","last_clean":"2023-06-10T18:44:13.394016+0000","last_became_active":"2023-06-10T18:44:08.577846+0000","last_became_peered":"2023-06-10T18:44:08.577846+0000","last_unstale":"2023-06-10T18:44:13.394016+0000","last_undegraded":"2023-06-10T18:44:13.394016+0000","last_fullsized":"2023-06-10T18:44:13.394016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392530+0000","last_change":"2023-06-10T18:44:08.984476+0000","last_active":"2023-06-10T18:44:13.392530+0000","last_peered":"2023-06-10T18:44:13.392530+0000","last_clean":"2023-06-10T18:44:13.392530+0000","last_became_active":"2023-06-10T18:44:08.984095+0000","last_became_peered":"2023-06-10T18:44:08.984095+0000","last_unstale":"2023-06-10T18:44:13.392530+0000","last_undegraded":"2023-06-10T18:44:13.392530+0000","last_fullsized":"2023-06-10T18:44:13.392530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422417+0000","last_change":"2023-06-10T18:44:13.006928+0000","last_active":"2023-06-10T18:44:13.422417+0000","last_peered":"2023-06-10T18:44:13.422417+0000","last_clean":"2023-06-10T18:44:13.422417+0000","last_became_active":"2023-06-10T18:44:13.006526+0000","last_became_peered":"2023-06-10T18:44:13.006526+0000","last_unstale":"2023-06-10T18:44:13.422417+0000","last_undegraded":"2023-06-10T18:44:13.422417+0000","last_fullsized":"2023-06-10T18:44:13.422417+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:42:15.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393898+0000","last_change":"2023-06-10T18:44:08.580704+0000","last_active":"2023-06-10T18:44:13.393898+0000","last_peered":"2023-06-10T18:44:13.393898+0000","last_clean":"2023-06-10T18:44:13.393898+0000","last_became_active":"2023-06-10T18:44:08.580413+0000","last_became_peered":"2023-06-10T18:44:08.580413+0000","last_unstale":"2023-06-10T18:44:13.393898+0000","last_undegraded":"2023-06-10T18:44:13.393898+0000","last_fullsized":"2023-06-10T18:44:13.393898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:29.052625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423333+0000","last_change":"2023-06-10T18:44:08.988350+0000","last_active":"2023-06-10T18:44:13.423333+0000","last_peered":"2023-06-10T18:44:13.423333+0000","last_clean":"2023-06-10T18:44:13.423333+0000","last_became_active":"2023-06-10T18:44:08.988073+0000","last_became_peered":"2023-06-10T18:44:08.988073+0000","last_unstale":"2023-06-10T18:44:13.423333+0000","last_undegraded":"2023-06-10T18:44:13.423333+0000","last_fullsized":"2023-06-10T18:44:13.423333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422684+0000","last_change":"2023-06-10T18:44:13.001742+0000","last_active":"2023-06-10T18:44:13.422684+0000","last_peered":"2023-06-10T18:44:13.422684+0000","last_clean":"2023-06-10T18:44:13.422684+0000","last_became_active":"2023-06-10T18:44:13.001373+0000","last_became_peered":"2023-06-10T18:44:13.001373+0000","last_unstale":"2023-06-10T18:44:13.422684+0000","last_undegraded":"2023-06-10T18:44:13.422684+0000","last_fullsized":"2023-06-10T18:44:13.422684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:11.616196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394281+0000","last_change":"2023-06-10T18:44:08.574389+0000","last_active":"2023-06-10T18:44:13.394281+0000","last_peered":"2023-06-10T18:44:13.394281+0000","last_clean":"2023-06-10T18:44:13.394281+0000","last_became_active":"2023-06-10T18:44:08.574056+0000","last_became_peered":"2023-06-10T18:44:08.574056+0000","last_unstale":"2023-06-10T18:44:13.394281+0000","last_undegraded":"2023-06-10T18:44:13.394281+0000","last_fullsized":"2023-06-10T18:44:13.394281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423143+0000","last_change":"2023-06-10T18:44:08.987503+0000","last_active":"2023-06-10T18:44:13.423143+0000","last_peered":"2023-06-10T18:44:13.423143+0000","last_clean":"2023-06-10T18:44:13.423143+0000","last_became_active":"2023-06-10T18:44:08.987204+0000","last_became_peered":"2023-06-10T18:44:08.987204+0000","last_unstale":"2023-06-10T18:44:13.423143+0000","last_undegraded":"2023-06-10T18:44:13.423143+0000","last_fullsized":"2023-06-10T18:44:13.423143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:45.870218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422702+0000","last_change":"2023-06-10T18:44:13.003895+0000","last_active":"2023-06-10T18:44:13.422702+0000","last_peered":"2023-06-10T18:44:13.422702+0000","last_clean":"2023-06-10T18:44:13.422702+0000","last_became_active":"2023-06-10T18:44:13.001542+0000","last_became_peered":"2023-06-10T18:44:13.001542+0000","last_unstale":"2023-06-10T18:44:13.422702+0000","last_undegraded":"2023-06-10T18:44:13.422702+0000","last_fullsized":"2023-06-10T18:44:13.422702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:54:25.091621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393042+0000","last_change":"2023-06-10T18:44:08.580607+0000","last_active":"2023-06-10T18:44:13.393042+0000","last_peered":"2023-06-10T18:44:13.393042+0000","last_clean":"2023-06-10T18:44:13.393042+0000","last_became_active":"2023-06-10T18:44:08.580289+0000","last_became_peered":"2023-06-10T18:44:08.580289+0000","last_unstale":"2023-06-10T18:44:13.393042+0000","last_undegraded":"2023-06-10T18:44:13.393042+0000","last_fullsized":"2023-06-10T18:44:13.393042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:09.158752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424116+0000","last_change":"2023-06-10T18:44:08.987624+0000","last_active":"2023-06-10T18:44:13.424116+0000","last_peered":"2023-06-10T18:44:13.424116+0000","last_clean":"2023-06-10T18:44:13.424116+0000","last_became_active":"2023-06-10T18:44:08.987375+0000","last_became_peered":"2023-06-10T18:44:08.987375+0000","last_unstale":"2023-06-10T18:44:13.424116+0000","last_undegraded":"2023-06-10T18:44:13.424116+0000","last_fullsized":"2023-06-10T18:44:13.424116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422769+0000","last_change":"2023-06-10T18:44:13.002722+0000","last_active":"2023-06-10T18:44:13.422769+0000","last_peered":"2023-06-10T18:44:13.422769+0000","last_clean":"2023-06-10T18:44:13.422769+0000","last_became_active":"2023-06-10T18:44:13.002214+0000","last_became_peered":"2023-06-10T18:44:13.002214+0000","last_unstale":"2023-06-10T18:44:13.422769+0000","last_undegraded":"2023-06-10T18:44:13.422769+0000","last_fullsized":"2023-06-10T18:44:13.422769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:04:35.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425192+0000","last_change":"2023-06-10T18:44:08.578128+0000","last_active":"2023-06-10T18:44:13.425192+0000","last_peered":"2023-06-10T18:44:13.425192+0000","last_clean":"2023-06-10T18:44:13.425192+0000","last_became_active":"2023-06-10T18:44:08.577863+0000","last_became_peered":"2023-06-10T18:44:08.577863+0000","last_unstale":"2023-06-10T18:44:13.425192+0000","last_undegraded":"2023-06-10T18:44:13.425192+0000","last_fullsized":"2023-06-10T18:44:13.425192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423537+0000","last_change":"2023-06-10T18:44:08.995303+0000","last_active":"2023-06-10T18:44:13.423537+0000","last_peered":"2023-06-10T18:44:13.423537+0000","last_clean":"2023-06-10T18:44:13.423537+0000","last_became_active":"2023-06-10T18:44:08.995025+0000","last_became_peered":"2023-06-10T18:44:08.995025+0000","last_unstale":"2023-06-10T18:44:13.423537+0000","last_undegraded":"2023-06-10T18:44:13.423537+0000","last_fullsized":"2023-06-10T18:44:13.423537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394147+0000","last_change":"2023-06-10T18:44:13.008926+0000","last_active":"2023-06-10T18:44:13.394147+0000","last_peered":"2023-06-10T18:44:13.394147+0000","last_clean":"2023-06-10T18:44:13.394147+0000","last_became_active":"2023-06-10T18:44:13.008249+0000","last_became_peered":"2023-06-10T18:44:13.008249+0000","last_unstale":"2023-06-10T18:44:13.394147+0000","last_undegraded":"2023-06-10T18:44:13.394147+0000","last_fullsized":"2023-06-10T18:44:13.394147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423270+0000","last_change":"2023-06-10T18:44:08.575472+0000","last_active":"2023-06-10T18:44:13.423270+0000","last_peered":"2023-06-10T18:44:13.423270+0000","last_clean":"2023-06-10T18:44:13.423270+0000","last_became_active":"2023-06-10T18:44:08.575186+0000","last_became_peered":"2023-06-10T18:44:08.575186+0000","last_unstale":"2023-06-10T18:44:13.423270+0000","last_undegraded":"2023-06-10T18:44:13.423270+0000","last_fullsized":"2023-06-10T18:44:13.423270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:33.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423243+0000","last_change":"2023-06-10T18:44:08.987175+0000","last_active":"2023-06-10T18:44:13.423243+0000","last_peered":"2023-06-10T18:44:13.423243+0000","last_clean":"2023-06-10T18:44:13.423243+0000","last_became_active":"2023-06-10T18:44:08.986395+0000","last_became_peered":"2023-06-10T18:44:08.986395+0000","last_unstale":"2023-06-10T18:44:13.423243+0000","last_undegraded":"2023-06-10T18:44:13.423243+0000","last_fullsized":"2023-06-10T18:44:13.423243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393572+0000","last_change":"2023-06-10T18:44:13.002535+0000","last_active":"2023-06-10T18:44:13.393572+0000","last_peered":"2023-06-10T18:44:13.393572+0000","last_clean":"2023-06-10T18:44:13.393572+0000","last_became_active":"2023-06-10T18:44:13.002065+0000","last_became_peered":"2023-06-10T18:44:13.002065+0000","last_unstale":"2023-06-10T18:44:13.393572+0000","last_undegraded":"2023-06-10T18:44:13.393572+0000","last_fullsized":"2023-06-10T18:44:13.393572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:56.164577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423083+0000","last_change":"2023-06-10T18:44:08.572499+0000","last_active":"2023-06-10T18:44:13.423083+0000","last_peered":"2023-06-10T18:44:13.423083+0000","last_clean":"2023-06-10T18:44:13.423083+0000","last_became_active":"2023-06-10T18:44:08.572196+0000","last_became_peered":"2023-06-10T18:44:08.572196+0000","last_unstale":"2023-06-10T18:44:13.423083+0000","last_undegraded":"2023-06-10T18:44:13.423083+0000","last_fullsized":"2023-06-10T18:44:13.423083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423986+0000","last_change":"2023-06-10T18:44:08.989104+0000","last_active":"2023-06-10T18:44:13.423986+0000","last_peered":"2023-06-10T18:44:13.423986+0000","last_clean":"2023-06-10T18:44:13.423986+0000","last_became_active":"2023-06-10T18:44:08.988824+0000","last_became_peered":"2023-06-10T18:44:08.988824+0000","last_unstale":"2023-06-10T18:44:13.423986+0000","last_undegraded":"2023-06-10T18:44:13.423986+0000","last_fullsized":"2023-06-10T18:44:13.423986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:19.807436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423861+0000","last_change":"2023-06-10T18:44:13.006826+0000","last_active":"2023-06-10T18:44:13.423861+0000","last_peered":"2023-06-10T18:44:13.423861+0000","last_clean":"2023-06-10T18:44:13.423861+0000","last_became_active":"2023-06-10T18:44:13.006458+0000","last_became_peered":"2023-06-10T18:44:13.006458+0000","last_unstale":"2023-06-10T18:44:13.423861+0000","last_undegraded":"2023-06-10T18:44:13.423861+0000","last_fullsized":"2023-06-10T18:44:13.423861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.577680+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.577419+0000","last_became_peered":"2023-06-10T18:44:08.577419+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394577+0000","last_change":"2023-06-10T18:44:08.991268+0000","last_active":"2023-06-10T18:44:13.394577+0000","last_peered":"2023-06-10T18:44:13.394577+0000","last_clean":"2023-06-10T18:44:13.394577+0000","last_became_active":"2023-06-10T18:44:08.990999+0000","last_became_peered":"2023-06-10T18:44:08.990999+0000","last_unstale":"2023-06-10T18:44:13.394577+0000","last_undegraded":"2023-06-10T18:44:13.394577+0000","last_fullsized":"2023-06-10T18:44:13.394577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423567+0000","last_change":"2023-06-10T18:44:13.010490+0000","last_active":"2023-06-10T18:44:13.423567+0000","last_peered":"2023-06-10T18:44:13.423567+0000","last_clean":"2023-06-10T18:44:13.423567+0000","last_became_active":"2023-06-10T18:44:13.010098+0000","last_became_peered":"2023-06-10T18:44:13.010098+0000","last_unstale":"2023-06-10T18:44:13.423567+0000","last_undegraded":"2023-06-10T18:44:13.423567+0000","last_fullsized":"2023-06-10T18:44:13.423567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:58.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394227+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:44:13.394227+0000","last_peered":"2023-06-10T18:44:13.394227+0000","last_clean":"2023-06-10T18:44:13.394227+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:44:13.394227+0000","last_undegraded":"2023-06-10T18:44:13.394227+0000","last_fullsized":"2023-06-10T18:44:13.394227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393285+0000","last_change":"2023-06-10T18:44:08.577289+0000","last_active":"2023-06-10T18:44:13.393285+0000","last_peered":"2023-06-10T18:44:13.393285+0000","last_clean":"2023-06-10T18:44:13.393285+0000","last_became_active":"2023-06-10T18:44:08.577021+0000","last_became_peered":"2023-06-10T18:44:08.577021+0000","last_unstale":"2023-06-10T18:44:13.393285+0000","last_undegraded":"2023-06-10T18:44:13.393285+0000","last_fullsized":"2023-06-10T18:44:13.393285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393847+0000","last_change":"2023-06-10T18:44:08.983928+0000","last_active":"2023-06-10T18:44:13.393847+0000","last_peered":"2023-06-10T18:44:13.393847+0000","last_clean":"2023-06-10T18:44:13.393847+0000","last_became_active":"2023-06-10T18:44:08.983562+0000","last_became_peered":"2023-06-10T18:44:08.983562+0000","last_unstale":"2023-06-10T18:44:13.393847+0000","last_undegraded":"2023-06-10T18:44:13.393847+0000","last_fullsized":"2023-06-10T18:44:13.393847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423202+0000","last_change":"2023-06-10T18:44:13.006596+0000","last_active":"2023-06-10T18:44:13.423202+0000","last_peered":"2023-06-10T18:44:13.423202+0000","last_clean":"2023-06-10T18:44:13.423202+0000","last_became_active":"2023-06-10T18:44:13.006258+0000","last_became_peered":"2023-06-10T18:44:13.006258+0000","last_unstale":"2023-06-10T18:44:13.423202+0000","last_undegraded":"2023-06-10T18:44:13.423202+0000","last_fullsized":"2023-06-10T18:44:13.423202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:31.503221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393245+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:44:13.393245+0000","last_peered":"2023-06-10T18:44:13.393245+0000","last_clean":"2023-06-10T18:44:13.393245+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:44:13.393245+0000","last_undegraded":"2023-06-10T18:44:13.393245+0000","last_fullsized":"2023-06-10T18:44:13.393245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:28.260256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424001+0000","last_change":"2023-06-10T18:44:08.578515+0000","last_active":"2023-06-10T18:44:13.424001+0000","last_peered":"2023-06-10T18:44:13.424001+0000","last_clean":"2023-06-10T18:44:13.424001+0000","last_became_active":"2023-06-10T18:44:08.578184+0000","last_became_peered":"2023-06-10T18:44:08.578184+0000","last_unstale":"2023-06-10T18:44:13.424001+0000","last_undegraded":"2023-06-10T18:44:13.424001+0000","last_fullsized":"2023-06-10T18:44:13.424001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394190+0000","last_change":"2023-06-10T18:44:08.985054+0000","last_active":"2023-06-10T18:44:13.394190+0000","last_peered":"2023-06-10T18:44:13.394190+0000","last_clean":"2023-06-10T18:44:13.394190+0000","last_became_active":"2023-06-10T18:44:08.984698+0000","last_became_peered":"2023-06-10T18:44:08.984698+0000","last_unstale":"2023-06-10T18:44:13.394190+0000","last_undegraded":"2023-06-10T18:44:13.394190+0000","last_fullsized":"2023-06-10T18:44:13.394190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423677+0000","last_change":"2023-06-10T18:44:13.002621+0000","last_active":"2023-06-10T18:44:13.423677+0000","last_peered":"2023-06-10T18:44:13.423677+0000","last_clean":"2023-06-10T18:44:13.423677+0000","last_became_active":"2023-06-10T18:44:13.002229+0000","last_became_peered":"2023-06-10T18:44:13.002229+0000","last_unstale":"2023-06-10T18:44:13.423677+0000","last_undegraded":"2023-06-10T18:44:13.423677+0000","last_fullsized":"2023-06-10T18:44:13.423677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:44:56.053966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422499+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:44:13.422499+0000","last_peered":"2023-06-10T18:44:13.422499+0000","last_clean":"2023-06-10T18:44:13.422499+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:44:13.422499+0000","last_undegraded":"2023-06-10T18:44:13.422499+0000","last_fullsized":"2023-06-10T18:44:13.422499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:49:35.702822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393742+0000","last_change":"2023-06-10T18:44:08.571468+0000","last_active":"2023-06-10T18:44:13.393742+0000","last_peered":"2023-06-10T18:44:13.393742+0000","last_clean":"2023-06-10T18:44:13.393742+0000","last_became_active":"2023-06-10T18:44:08.570651+0000","last_became_peered":"2023-06-10T18:44:08.570651+0000","last_unstale":"2023-06-10T18:44:13.393742+0000","last_undegraded":"2023-06-10T18:44:13.393742+0000","last_fullsized":"2023-06-10T18:44:13.393742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394594+0000","last_change":"2023-06-10T18:44:08.983687+0000","last_active":"2023-06-10T18:44:13.394594+0000","last_peered":"2023-06-10T18:44:13.394594+0000","last_clean":"2023-06-10T18:44:13.394594+0000","last_became_active":"2023-06-10T18:44:08.983381+0000","last_became_peered":"2023-06-10T18:44:08.983381+0000","last_unstale":"2023-06-10T18:44:13.394594+0000","last_undegraded":"2023-06-10T18:44:13.394594+0000","last_fullsized":"2023-06-10T18:44:13.394594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393971+0000","last_change":"2023-06-10T18:44:13.009660+0000","last_active":"2023-06-10T18:44:13.393971+0000","last_peered":"2023-06-10T18:44:13.393971+0000","last_clean":"2023-06-10T18:44:13.393971+0000","last_became_active":"2023-06-10T18:44:13.009208+0000","last_became_peered":"2023-06-10T18:44:13.009208+0000","last_unstale":"2023-06-10T18:44:13.393971+0000","last_undegraded":"2023-06-10T18:44:13.393971+0000","last_fullsized":"2023-06-10T18:44:13.393971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392956+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:44:13.392956+0000","last_peered":"2023-06-10T18:44:13.392956+0000","last_clean":"2023-06-10T18:44:13.392956+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:44:13.392956+0000","last_undegraded":"2023-06-10T18:44:13.392956+0000","last_fullsized":"2023-06-10T18:44:13.392956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":183,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394153+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:44:13.394153+0000","last_peered":"2023-06-10T18:44:13.394153+0000","last_clean":"2023-06-10T18:44:13.394153+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:44:13.394153+0000","last_undegraded":"2023-06-10T18:44:13.394153+0000","last_fullsized":"2023-06-10T18:44:13.394153+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:22.105411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425014+0000","last_change":"2023-06-10T18:44:08.576254+0000","last_active":"2023-06-10T18:44:13.425014+0000","last_peered":"2023-06-10T18:44:13.425014+0000","last_clean":"2023-06-10T18:44:13.425014+0000","last_became_active":"2023-06-10T18:44:08.575980+0000","last_became_peered":"2023-06-10T18:44:08.575980+0000","last_unstale":"2023-06-10T18:44:13.425014+0000","last_undegraded":"2023-06-10T18:44:13.425014+0000","last_fullsized":"2023-06-10T18:44:13.425014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423295+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.423295+0000","last_peered":"2023-06-10T18:44:13.423295+0000","last_clean":"2023-06-10T18:44:13.423295+0000","last_became_active":"2023-06-10T18:44:08.988159+0000","last_became_peered":"2023-06-10T18:44:08.988159+0000","last_unstale":"2023-06-10T18:44:13.423295+0000","last_undegraded":"2023-06-10T18:44:13.423295+0000","last_fullsized":"2023-06-10T18:44:13.423295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:55.371558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393218+0000","last_change":"2023-06-10T18:44:13.006561+0000","last_active":"2023-06-10T18:44:13.393218+0000","last_peered":"2023-06-10T18:44:13.393218+0000","last_clean":"2023-06-10T18:44:13.393218+0000","last_became_active":"2023-06-10T18:44:13.006171+0000","last_became_peered":"2023-06-10T18:44:13.006171+0000","last_unstale":"2023-06-10T18:44:13.393218+0000","last_undegraded":"2023-06-10T18:44:13.393218+0000","last_fullsized":"2023-06-10T18:44:13.393218+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:18.821445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394377+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:44:13.394377+0000","last_peered":"2023-06-10T18:44:13.394377+0000","last_clean":"2023-06-10T18:44:13.394377+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:44:13.394377+0000","last_undegraded":"2023-06-10T18:44:13.394377+0000","last_fullsized":"2023-06-10T18:44:13.394377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:21.913689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423028+0000","last_change":"2023-06-10T18:44:08.578010+0000","last_active":"2023-06-10T18:44:13.423028+0000","last_peered":"2023-06-10T18:44:13.423028+0000","last_clean":"2023-06-10T18:44:13.423028+0000","last_became_active":"2023-06-10T18:44:08.577727+0000","last_became_peered":"2023-06-10T18:44:08.577727+0000","last_unstale":"2023-06-10T18:44:13.423028+0000","last_undegraded":"2023-06-10T18:44:13.423028+0000","last_fullsized":"2023-06-10T18:44:13.423028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:00:49.837648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423225+0000","last_change":"2023-06-10T18:44:08.990076+0000","last_active":"2023-06-10T18:44:13.423225+0000","last_peered":"2023-06-10T18:44:13.423225+0000","last_clean":"2023-06-10T18:44:13.423225+0000","last_became_active":"2023-06-10T18:44:08.989191+0000","last_became_peered":"2023-06-10T18:44:08.989191+0000","last_unstale":"2023-06-10T18:44:13.423225+0000","last_undegraded":"2023-06-10T18:44:13.423225+0000","last_fullsized":"2023-06-10T18:44:13.423225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:35.616295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423712+0000","last_change":"2023-06-10T18:44:12.999386+0000","last_active":"2023-06-10T18:44:13.423712+0000","last_peered":"2023-06-10T18:44:13.423712+0000","last_clean":"2023-06-10T18:44:13.423712+0000","last_became_active":"2023-06-10T18:44:12.998986+0000","last_became_peered":"2023-06-10T18:44:12.998986+0000","last_unstale":"2023-06-10T18:44:13.423712+0000","last_undegraded":"2023-06-10T18:44:13.423712+0000","last_fullsized":"2023-06-10T18:44:13.423712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:10.294554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394138+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:44:13.394138+0000","last_peered":"2023-06-10T18:44:13.394138+0000","last_clean":"2023-06-10T18:44:13.394138+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:44:13.394138+0000","last_undegraded":"2023-06-10T18:44:13.394138+0000","last_fullsized":"2023-06-10T18:44:13.394138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:43.350286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394357+0000","last_change":"2023-06-10T18:44:08.572021+0000","last_active":"2023-06-10T18:44:13.394357+0000","last_peered":"2023-06-10T18:44:13.394357+0000","last_clean":"2023-06-10T18:44:13.394357+0000","last_became_active":"2023-06-10T18:44:08.571716+0000","last_became_peered":"2023-06-10T18:44:08.571716+0000","last_unstale":"2023-06-10T18:44:13.394357+0000","last_undegraded":"2023-06-10T18:44:13.394357+0000","last_fullsized":"2023-06-10T18:44:13.394357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423849+0000","last_change":"2023-06-10T18:44:08.983810+0000","last_active":"2023-06-10T18:44:13.423849+0000","last_peered":"2023-06-10T18:44:13.423849+0000","last_clean":"2023-06-10T18:44:13.423849+0000","last_became_active":"2023-06-10T18:44:08.983539+0000","last_became_peered":"2023-06-10T18:44:08.983539+0000","last_unstale":"2023-06-10T18:44:13.423849+0000","last_undegraded":"2023-06-10T18:44:13.423849+0000","last_fullsized":"2023-06-10T18:44:13.423849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423587+0000","last_change":"2023-06-10T18:44:13.000564+0000","last_active":"2023-06-10T18:44:13.423587+0000","last_peered":"2023-06-10T18:44:13.423587+0000","last_clean":"2023-06-10T18:44:13.423587+0000","last_became_active":"2023-06-10T18:44:13.000141+0000","last_became_peered":"2023-06-10T18:44:13.000141+0000","last_unstale":"2023-06-10T18:44:13.423587+0000","last_undegraded":"2023-06-10T18:44:13.423587+0000","last_fullsized":"2023-06-10T18:44:13.423587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:45.731880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425478+0000","last_change":"2023-06-10T18:44:08.579728+0000","last_active":"2023-06-10T18:44:13.425478+0000","last_peered":"2023-06-10T18:44:13.425478+0000","last_clean":"2023-06-10T18:44:13.425478+0000","last_became_active":"2023-06-10T18:44:08.579431+0000","last_became_peered":"2023-06-10T18:44:08.579431+0000","last_unstale":"2023-06-10T18:44:13.425478+0000","last_undegraded":"2023-06-10T18:44:13.425478+0000","last_fullsized":"2023-06-10T18:44:13.425478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394345+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:44:13.394345+0000","last_peered":"2023-06-10T18:44:13.394345+0000","last_clean":"2023-06-10T18:44:13.394345+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:44:13.394345+0000","last_undegraded":"2023-06-10T18:44:13.394345+0000","last_fullsized":"2023-06-10T18:44:13.394345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:05.996450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424774+0000","last_change":"2023-06-10T18:44:08.983770+0000","last_active":"2023-06-10T18:44:13.424774+0000","last_peered":"2023-06-10T18:44:13.424774+0000","last_clean":"2023-06-10T18:44:13.424774+0000","last_became_active":"2023-06-10T18:44:08.983526+0000","last_became_peered":"2023-06-10T18:44:08.983526+0000","last_unstale":"2023-06-10T18:44:13.424774+0000","last_undegraded":"2023-06-10T18:44:13.424774+0000","last_fullsized":"2023-06-10T18:44:13.424774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424718+0000","last_change":"2023-06-10T18:44:13.003992+0000","last_active":"2023-06-10T18:44:13.424718+0000","last_peered":"2023-06-10T18:44:13.424718+0000","last_clean":"2023-06-10T18:44:13.424718+0000","last_became_active":"2023-06-10T18:44:13.003565+0000","last_became_peered":"2023-06-10T18:44:13.003565+0000","last_unstale":"2023-06-10T18:44:13.424718+0000","last_undegraded":"2023-06-10T18:44:13.424718+0000","last_fullsized":"2023-06-10T18:44:13.424718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423787+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:44:13.423787+0000","last_peered":"2023-06-10T18:44:13.423787+0000","last_clean":"2023-06-10T18:44:13.423787+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:44:13.423787+0000","last_undegraded":"2023-06-10T18:44:13.423787+0000","last_fullsized":"2023-06-10T18:44:13.423787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:03:46.107219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423014+0000","last_change":"2023-06-10T18:44:08.576079+0000","last_active":"2023-06-10T18:44:13.423014+0000","last_peered":"2023-06-10T18:44:13.423014+0000","last_clean":"2023-06-10T18:44:13.423014+0000","last_became_active":"2023-06-10T18:44:08.575346+0000","last_became_peered":"2023-06-10T18:44:08.575346+0000","last_unstale":"2023-06-10T18:44:13.423014+0000","last_undegraded":"2023-06-10T18:44:13.423014+0000","last_fullsized":"2023-06-10T18:44:13.423014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:54:23.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423207+0000","last_change":"2023-06-10T18:44:08.990722+0000","last_active":"2023-06-10T18:44:13.423207+0000","last_peered":"2023-06-10T18:44:13.423207+0000","last_clean":"2023-06-10T18:44:13.423207+0000","last_became_active":"2023-06-10T18:44:08.988467+0000","last_became_peered":"2023-06-10T18:44:08.988467+0000","last_unstale":"2023-06-10T18:44:13.423207+0000","last_undegraded":"2023-06-10T18:44:13.423207+0000","last_fullsized":"2023-06-10T18:44:13.423207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394392+0000","last_change":"2023-06-10T18:44:13.008834+0000","last_active":"2023-06-10T18:44:13.394392+0000","last_peered":"2023-06-10T18:44:13.394392+0000","last_clean":"2023-06-10T18:44:13.394392+0000","last_became_active":"2023-06-10T18:44:13.008433+0000","last_became_peered":"2023-06-10T18:44:13.008433+0000","last_unstale":"2023-06-10T18:44:13.394392+0000","last_undegraded":"2023-06-10T18:44:13.394392+0000","last_fullsized":"2023-06-10T18:44:13.394392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423733+0000","last_change":"2023-06-10T18:44:08.576600+0000","last_active":"2023-06-10T18:44:13.423733+0000","last_peered":"2023-06-10T18:44:13.423733+0000","last_clean":"2023-06-10T18:44:13.423733+0000","last_became_active":"2023-06-10T18:44:08.575635+0000","last_became_peered":"2023-06-10T18:44:08.575635+0000","last_unstale":"2023-06-10T18:44:13.423733+0000","last_undegraded":"2023-06-10T18:44:13.423733+0000","last_fullsized":"2023-06-10T18:44:13.423733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423305+0000","last_change":"2023-06-10T18:44:08.985629+0000","last_active":"2023-06-10T18:44:13.423305+0000","last_peered":"2023-06-10T18:44:13.423305+0000","last_clean":"2023-06-10T18:44:13.423305+0000","last_became_active":"2023-06-10T18:44:08.985367+0000","last_became_peered":"2023-06-10T18:44:08.985367+0000","last_unstale":"2023-06-10T18:44:13.423305+0000","last_undegraded":"2023-06-10T18:44:13.423305+0000","last_fullsized":"2023-06-10T18:44:13.423305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:57.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394246+0000","last_change":"2023-06-10T18:44:13.010558+0000","last_active":"2023-06-10T18:44:13.394246+0000","last_peered":"2023-06-10T18:44:13.394246+0000","last_clean":"2023-06-10T18:44:13.394246+0000","last_became_active":"2023-06-10T18:44:13.010123+0000","last_became_peered":"2023-06-10T18:44:13.010123+0000","last_unstale":"2023-06-10T18:44:13.394246+0000","last_undegraded":"2023-06-10T18:44:13.394246+0000","last_fullsized":"2023-06-10T18:44:13.394246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:13.872743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393194+0000","last_change":"2023-06-10T18:44:08.571654+0000","last_active":"2023-06-10T18:44:13.393194+0000","last_peered":"2023-06-10T18:44:13.393194+0000","last_clean":"2023-06-10T18:44:13.393194+0000","last_became_active":"2023-06-10T18:44:08.571363+0000","last_became_peered":"2023-06-10T18:44:08.571363+0000","last_unstale":"2023-06-10T18:44:13.393194+0000","last_undegraded":"2023-06-10T18:44:13.393194+0000","last_fullsized":"2023-06-10T18:44:13.393194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:48.376074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423819+0000","last_change":"2023-06-10T18:44:08.996056+0000","last_active":"2023-06-10T18:44:13.423819+0000","last_peered":"2023-06-10T18:44:13.423819+0000","last_clean":"2023-06-10T18:44:13.423819+0000","last_became_active":"2023-06-10T18:44:08.995902+0000","last_became_peered":"2023-06-10T18:44:08.995902+0000","last_unstale":"2023-06-10T18:44:13.423819+0000","last_undegraded":"2023-06-10T18:44:13.423819+0000","last_fullsized":"2023-06-10T18:44:13.423819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393057+0000","last_change":"2023-06-10T18:44:13.004134+0000","last_active":"2023-06-10T18:44:13.393057+0000","last_peered":"2023-06-10T18:44:13.393057+0000","last_clean":"2023-06-10T18:44:13.393057+0000","last_became_active":"2023-06-10T18:44:13.003596+0000","last_became_peered":"2023-06-10T18:44:13.003596+0000","last_unstale":"2023-06-10T18:44:13.393057+0000","last_undegraded":"2023-06-10T18:44:13.393057+0000","last_fullsized":"2023-06-10T18:44:13.393057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393105+0000","last_change":"2023-06-10T18:44:08.575332+0000","last_active":"2023-06-10T18:44:13.393105+0000","last_peered":"2023-06-10T18:44:13.393105+0000","last_clean":"2023-06-10T18:44:13.393105+0000","last_became_active":"2023-06-10T18:44:08.575065+0000","last_became_peered":"2023-06-10T18:44:08.575065+0000","last_unstale":"2023-06-10T18:44:13.393105+0000","last_undegraded":"2023-06-10T18:44:13.393105+0000","last_fullsized":"2023-06-10T18:44:13.393105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424331+0000","last_change":"2023-06-10T18:44:08.995435+0000","last_active":"2023-06-10T18:44:13.424331+0000","last_peered":"2023-06-10T18:44:13.424331+0000","last_clean":"2023-06-10T18:44:13.424331+0000","last_became_active":"2023-06-10T18:44:08.995141+0000","last_became_peered":"2023-06-10T18:44:08.995141+0000","last_unstale":"2023-06-10T18:44:13.424331+0000","last_undegraded":"2023-06-10T18:44:13.424331+0000","last_fullsized":"2023-06-10T18:44:13.424331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:52.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422884+0000","last_change":"2023-06-10T18:44:12.998566+0000","last_active":"2023-06-10T18:44:13.422884+0000","last_peered":"2023-06-10T18:44:13.422884+0000","last_clean":"2023-06-10T18:44:13.422884+0000","last_became_active":"2023-06-10T18:44:12.998185+0000","last_became_peered":"2023-06-10T18:44:12.998185+0000","last_unstale":"2023-06-10T18:44:13.422884+0000","last_undegraded":"2023-06-10T18:44:13.422884+0000","last_fullsized":"2023-06-10T18:44:13.422884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425429+0000","last_change":"2023-06-10T18:44:08.576599+0000","last_active":"2023-06-10T18:44:13.425429+0000","last_peered":"2023-06-10T18:44:13.425429+0000","last_clean":"2023-06-10T18:44:13.425429+0000","last_became_active":"2023-06-10T18:44:08.576329+0000","last_became_peered":"2023-06-10T18:44:08.576329+0000","last_unstale":"2023-06-10T18:44:13.425429+0000","last_undegraded":"2023-06-10T18:44:13.425429+0000","last_fullsized":"2023-06-10T18:44:13.425429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:16.618462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424932+0000","last_change":"2023-06-10T18:44:08.975414+0000","last_active":"2023-06-10T18:44:13.424932+0000","last_peered":"2023-06-10T18:44:13.424932+0000","last_clean":"2023-06-10T18:44:13.424932+0000","last_became_active":"2023-06-10T18:44:08.975156+0000","last_became_peered":"2023-06-10T18:44:08.975156+0000","last_unstale":"2023-06-10T18:44:13.424932+0000","last_undegraded":"2023-06-10T18:44:13.424932+0000","last_fullsized":"2023-06-10T18:44:13.424932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393972+0000","last_change":"2023-06-10T18:44:13.007025+0000","last_active":"2023-06-10T18:44:13.393972+0000","last_peered":"2023-06-10T18:44:13.393972+0000","last_clean":"2023-06-10T18:44:13.393972+0000","last_became_active":"2023-06-10T18:44:13.006654+0000","last_became_peered":"2023-06-10T18:44:13.006654+0000","last_unstale":"2023-06-10T18:44:13.393972+0000","last_undegraded":"2023-06-10T18:44:13.393972+0000","last_fullsized":"2023-06-10T18:44:13.393972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:27.677403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423673+0000","last_change":"2023-06-10T18:44:08.575515+0000","last_active":"2023-06-10T18:44:13.423673+0000","last_peered":"2023-06-10T18:44:13.423673+0000","last_clean":"2023-06-10T18:44:13.423673+0000","last_became_active":"2023-06-10T18:44:08.573539+0000","last_became_peered":"2023-06-10T18:44:08.573539+0000","last_unstale":"2023-06-10T18:44:13.423673+0000","last_undegraded":"2023-06-10T18:44:13.423673+0000","last_fullsized":"2023-06-10T18:44:13.423673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:57.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422973+0000","last_change":"2023-06-10T18:44:08.993519+0000","last_active":"2023-06-10T18:44:13.422973+0000","last_peered":"2023-06-10T18:44:13.422973+0000","last_clean":"2023-06-10T18:44:13.422973+0000","last_became_active":"2023-06-10T18:44:08.993146+0000","last_became_peered":"2023-06-10T18:44:08.993146+0000","last_unstale":"2023-06-10T18:44:13.422973+0000","last_undegraded":"2023-06-10T18:44:13.422973+0000","last_fullsized":"2023-06-10T18:44:13.422973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422934+0000","last_change":"2023-06-10T18:44:08.989855+0000","last_active":"2023-06-10T18:44:13.422934+0000","last_peered":"2023-06-10T18:44:13.422934+0000","last_clean":"2023-06-10T18:44:13.422934+0000","last_became_active":"2023-06-10T18:44:08.989140+0000","last_became_peered":"2023-06-10T18:44:08.989140+0000","last_unstale":"2023-06-10T18:44:13.422934+0000","last_undegraded":"2023-06-10T18:44:13.422934+0000","last_fullsized":"2023-06-10T18:44:13.422934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:14.496637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425109+0000","last_change":"2023-06-10T18:44:08.992461+0000","last_active":"2023-06-10T18:44:13.425109+0000","last_peered":"2023-06-10T18:44:13.425109+0000","last_clean":"2023-06-10T18:44:13.425109+0000","last_became_active":"2023-06-10T18:44:08.992160+0000","last_became_peered":"2023-06-10T18:44:08.992160+0000","last_unstale":"2023-06-10T18:44:13.425109+0000","last_undegraded":"2023-06-10T18:44:13.425109+0000","last_fullsized":"2023-06-10T18:44:13.425109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:42.310678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423600+0000","last_change":"2023-06-10T18:44:08.991382+0000","last_active":"2023-06-10T18:44:13.423600+0000","last_peered":"2023-06-10T18:44:13.423600+0000","last_clean":"2023-06-10T18:44:13.423600+0000","last_became_active":"2023-06-10T18:44:08.991122+0000","last_became_peered":"2023-06-10T18:44:08.991122+0000","last_unstale":"2023-06-10T18:44:13.423600+0000","last_undegraded":"2023-06-10T18:44:13.423600+0000","last_fullsized":"2023-06-10T18:44:13.423600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:10.451541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425330+0000","last_change":"2023-06-10T18:44:08.987112+0000","last_active":"2023-06-10T18:44:13.425330+0000","last_peered":"2023-06-10T18:44:13.425330+0000","last_clean":"2023-06-10T18:44:13.425330+0000","last_became_active":"2023-06-10T18:44:08.986804+0000","last_became_peered":"2023-06-10T18:44:08.986804+0000","last_unstale":"2023-06-10T18:44:13.425330+0000","last_undegraded":"2023-06-10T18:44:13.425330+0000","last_fullsized":"2023-06-10T18:44:13.425330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394328+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.394328+0000","last_peered":"2023-06-10T18:44:13.394328+0000","last_clean":"2023-06-10T18:44:13.394328+0000","last_became_active":"2023-06-10T18:44:08.987510+0000","last_became_peered":"2023-06-10T18:44:08.987510+0000","last_unstale":"2023-06-10T18:44:13.394328+0000","last_undegraded":"2023-06-10T18:44:13.394328+0000","last_fullsized":"2023-06-10T18:44:13.394328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":913408,"data_stored":714279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":884736,"data_stored":709832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":122732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:44:33.016 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:44:33.017 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-06-10T18:44:33.017 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-06-10T18:44:33.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-06-10T18:44:33.018 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-06-10T18:44:33.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.161+0000 7f1322df2700 1 Processor -- start 2023-06-10T18:44:33.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.161+0000 7f1322df2700 1 -- start start 2023-06-10T18:44:33.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1322df2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f131c15c730 0x7f131c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1322df2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f131c0cc000 0x7f131c0cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1322df2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c15cb00 con 0x7f131c06cd70 2023-06-10T18:44:33.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1322df2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131c0aed80 con 0x7f131c15c730 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1322df2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131c0c9c60 con 0x7f131c0cc000 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1320b8e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f131c15c730 0x7f131c06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1320b8e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f131c15c730 0x7f131c06e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54706/0 (socket says 172.21.15.104:54706) 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f1320b8e700 1 -- 172.21.15.104:0/1907386992 learned_addr learned my addr 172.21.15.104:0/1907386992 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.162+0000 7f132138f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f131c06cd70 0x7f131c06d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52538/0 (socket says 172.21.15.104:52538) 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1320b8e700 1 -- 172.21.15.104:0/1907386992 >> v2:172.21.15.114:3300/0 conn(0x7f131c0cc000 msgr2=0x7f131c0cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:33.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f131bfff700 1 --2- 172.21.15.104:0/1907386992 >> v2:172.21.15.114:3300/0 conn(0x7f131c0cc000 0x7f131c0cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1320b8e700 1 --2- 172.21.15.104:0/1907386992 >> v2:172.21.15.114:3300/0 conn(0x7f131c0cc000 0x7f131c0cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1320b8e700 1 -- 172.21.15.104:0/1907386992 >> v1:172.21.15.104:6789/0 conn(0x7f131c06cd70 legacy=0x7f131c06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f131b7fe700 1 -- 172.21.15.104:0/1907386992 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448420798 0 0) 0x7f1314001490 con 0x7f131c06cd70 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1320b8e700 1 -- 172.21.15.104:0/1907386992 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c06ecd0 con 0x7f131c15c730 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1320b8e700 1 --2- 172.21.15.104:0/1907386992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c15c730 0x7f131c06e790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f131c06fd10 tx=0x7f1310009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fdbd2b33295d11 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f131b7fe700 1 -- 172.21.15.104:0/1907386992 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310005e30 con 0x7f131c15c730 2023-06-10T18:44:33.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f131b7fe700 1 -- 172.21.15.104:0/1907386992 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310004030 con 0x7f131c15c730 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f131b7fe700 1 -- 172.21.15.104:0/1907386992 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100043e0 con 0x7f131c15c730 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 -- 172.21.15.104:0/1907386992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c15c730 msgr2=0x7f131c06e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 --2- 172.21.15.104:0/1907386992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c15c730 0x7f131c06e790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f131c06fd10 tx=0x7f1310009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 -- 172.21.15.104:0/1907386992 shutdown_connections 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 --2- 172.21.15.104:0/1907386992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c15c730 0x7f131c06e790 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 --2- 172.21.15.104:0/1907386992 >> v2:172.21.15.114:3300/0 conn(0x7f131c0cc000 0x7f131c0cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 -- 172.21.15.104:0/1907386992 >> 172.21.15.104:0/1907386992 conn(0x7f131c06b8a0 msgr2=0x7f131c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 -- 172.21.15.104:0/1907386992 shutdown_connections 2023-06-10T18:44:33.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.163+0000 7f1322df2700 1 -- 172.21.15.104:0/1907386992 wait complete. 2023-06-10T18:44:33.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 Processor -- start 2023-06-10T18:44:33.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 -- start start 2023-06-10T18:44:33.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0cc000 0x7f131c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f131c15c730 0x7f131c0af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c14adf0 con 0x7f131c06cd70 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0b4110 con 0x7f131c0cc000 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.165+0000 7f1322df2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0b43f0 con 0x7f131c15c730 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54712/0 (socket says 172.21.15.104:54712) 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 -- 172.21.15.104:0/133789859 learned_addr learned my addr 172.21.15.104:0/133789859 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:33.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f131bfff700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0cc000 0x7f131c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 -- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f131c15c730 msgr2=0x7f131c0af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f131c15c730 0x7f131c0af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 -- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0cc000 msgr2=0x7f131c0af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0cc000 0x7f131c0af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 -- 172.21.15.104:0/133789859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310009500 con 0x7f131c06cd70 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1320b8e700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1310009c10 tx=0x7f1310004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100045f0 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131000b660 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131000ba10 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c0b46d0 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.166+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c0b4c90 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.167+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c066f10 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.168+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1310018030 con 0x7f131c06cd70 2023-06-10T18:44:33.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.169+0000 7f13197fa700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 0x7f130006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.169+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f131008b180 con 0x7f131c06cd70 2023-06-10T18:44:33.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.169+0000 7f131bfff700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 0x7f130006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.169+0000 7f131bfff700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 0x7f130006d1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f131c0b5130 tx=0x7f1308008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.172+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1310055fb0 con 0x7f131c06cd70 2023-06-10T18:44:33.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.410+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f131c074050 con 0x7f131c06cd70 2023-06-10T18:44:33.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.413+0000 7f13197fa700 1 -- 172.21.15.104:0/133789859 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170849 (secure 0 0 0) 0x7f13100595d0 con 0x7f131c06cd70 2023-06-10T18:44:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 msgr2=0x7f130006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 0x7f130006d1e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f131c0b5130 tx=0x7f1308008040 comp rx=0 tx=0).stop 2023-06-10T18:44:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 msgr2=0x7f131c14a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1310009c10 tx=0x7f1310004750 comp rx=0 tx=0).stop 2023-06-10T18:44:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 shutdown_connections 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f131c15c730 0x7f131c0af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f131c06cd70 0x7f131c14a8b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f130006ad30 0x7f130006d1e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 --2- 172.21.15.104:0/133789859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0cc000 0x7f131c0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 >> 172.21.15.104:0/133789859 conn(0x7f131c06b8a0 msgr2=0x7f131c06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 shutdown_connections 2023-06-10T18:44:33.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.416+0000 7f1322df2700 1 -- 172.21.15.104:0/133789859 wait complete. 2023-06-10T18:44:33.433 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2421772364},{"type":"v1","addr":"172.21.15.114:6833","nonce":2421772364}]},"active_addr":"172.21.15.114:6833/2421772364","active_change":"2023-06-10T18:43:49.338788+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4111,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2346444483}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1366697862}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3704548413}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2327125739}]}]} 2023-06-10T18:44:33.438 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-06-10T18:44:33.438 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-06-10T18:44:33.438 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:33.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.553+0000 7fcc6070b700 1 Processor -- start 2023-06-10T18:44:33.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.553+0000 7fcc6070b700 1 -- start start 2023-06-10T18:44:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc6070b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc580cb300 0x7fcc58163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc6070b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc580c9e60 con 0x7fcc580aeaa0 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc6070b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc580aee70 con 0x7fcc580cd850 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc6070b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc580c9bc0 con 0x7fcc580cb300 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc5e4a7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcc580aeaa0 0x7fcc580cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52548/0 (socket says 172.21.15.104:52548) 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc5e4a7700 1 -- 172.21.15.104:0/754015539 learned_addr learned my addr 172.21.15.104:0/754015539 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.554+0000 7fcc5eca8700 1 --2- 172.21.15.104:0/754015539 >> v2:172.21.15.114:3300/0 conn(0x7fcc580cb300 0x7fcc58163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337116240 0 0) 0x7fcc580c9e60 con 0x7fcc580aeaa0 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc440035b0 con 0x7fcc580aeaa0 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 891957585 0 0) 0x7fcc440035b0 con 0x7fcc580aeaa0 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5eca8700 1 -- 172.21.15.104:0/754015539 >> v1:172.21.15.114:6789/0 conn(0x7fcc580cd850 legacy=0x7fcc5806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5eca8700 1 -- 172.21.15.104:0/754015539 >> v1:172.21.15.104:6789/0 conn(0x7fcc580aeaa0 legacy=0x7fcc580cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5eca8700 1 -- 172.21.15.104:0/754015539 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5806e540 con 0x7fcc580cb300 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5eca8700 1 --2- 172.21.15.104:0/754015539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580cb300 0x7fcc58163c30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009d90 tx=0x7fcc5000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24a7711a4747a869 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000e070 con 0x7fcc580cb300 2023-06-10T18:44:33.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50003a20 con 0x7fcc580cb300 2023-06-10T18:44:33.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.555+0000 7fcc5d4a5700 1 -- 172.21.15.104:0/754015539 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480017f0 con 0x7fcc580cb300 2023-06-10T18:44:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 -- 172.21.15.104:0/754015539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580cb300 msgr2=0x7fcc58163c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 --2- 172.21.15.104:0/754015539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580cb300 0x7fcc58163c30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009d90 tx=0x7fcc5000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 -- 172.21.15.104:0/754015539 shutdown_connections 2023-06-10T18:44:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 --2- 172.21.15.104:0/754015539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580cb300 0x7fcc58163c30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 -- 172.21.15.104:0/754015539 >> 172.21.15.104:0/754015539 conn(0x7fcc5806b8a0 msgr2=0x7fcc580cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 -- 172.21.15.104:0/754015539 shutdown_connections 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.556+0000 7fcc6070b700 1 -- 172.21.15.104:0/754015539 wait complete. 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.557+0000 7fcc6070b700 1 Processor -- start 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 -- start start 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580cb300 0x7fcc580ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580ac770 con 0x7fcc580cd850 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580c7470 con 0x7fcc580aeaa0 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.558+0000 7fcc6070b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580c7750 con 0x7fcc580cb300 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54730/0 (socket says 172.21.15.104:54730) 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 -- 172.21.15.104:0/1091200938 learned_addr learned my addr 172.21.15.104:0/1091200938 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5dca6700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580cb300 0x7fcc580ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5e4a7700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 -- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580cb300 msgr2=0x7fcc580ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580cb300 0x7fcc580ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 -- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 msgr2=0x7fcc58152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 -- 172.21.15.104:0/1091200938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc50009a70 con 0x7fcc580cd850 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc5eca8700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b370 tx=0x7fcc50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50004540 con 0x7fcc580cd850 2023-06-10T18:44:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc500046a0 con 0x7fcc580cd850 2023-06-10T18:44:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc580c7a30 con 0x7fcc580cd850 2023-06-10T18:44:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5001d470 con 0x7fcc580cd850 2023-06-10T18:44:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.559+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc580c7ff0 con 0x7fcc580cd850 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.560+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc580cd140 con 0x7fcc580cd850 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.561+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcc5001d5d0 con 0x7fcc580cd850 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.561+0000 7fcc4effd700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 0x7fcc3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.561+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fcc5008b010 con 0x7fcc580cd850 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.561+0000 7fcc5e4a7700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 0x7fcc3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.562+0000 7fcc5e4a7700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 0x7fcc3c06d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcc48000ef0 tx=0x7fcc480012f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.565+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcc500598f0 con 0x7fcc580cd850 2023-06-10T18:44:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.752+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc581efc80 con 0x7fcc580cd850 2023-06-10T18:44:33.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.753+0000 7fcc4effd700 1 -- 172.21.15.104:0/1091200938 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7fcc50059480 con 0x7fcc580cd850 2023-06-10T18:44:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:33.759 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:33.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 msgr2=0x7fcc3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 0x7fcc3c06d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcc48000ef0 tx=0x7fcc480012f0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 msgr2=0x7fcc580b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b370 tx=0x7fcc50004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 shutdown_connections 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ace0 0x7fcc3c06d190 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580cd850 0x7fcc580b4640 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 --2- 172.21.15.104:0/1091200938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580cb300 0x7fcc580ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 >> 172.21.15.104:0/1091200938 conn(0x7fcc5806b8a0 msgr2=0x7fcc580b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 shutdown_connections 2023-06-10T18:44:33.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.755+0000 7fcc6070b700 1 -- 172.21.15.104:0/1091200938 wait complete. 2023-06-10T18:44:33.773 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-06-10T18:44:33.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.888+0000 7faca7b67700 1 Processor -- start 2023-06-10T18:44:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.888+0000 7faca7b67700 1 -- start start 2023-06-10T18:44:33.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca7b67700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faca00b1070 0x7faca00b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca7b67700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faca00aec00 0x7faca00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca7b67700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca006e5e0 con 0x7faca006ea70 2023-06-10T18:44:33.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca7b67700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca00b1980 con 0x7faca00aec00 2023-06-10T18:44:33.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca7b67700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca00ae7e0 con 0x7faca00b1070 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faca00aec00 0x7faca00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca5102700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faca006ea70 0x7faca00ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52562/0 (socket says 172.21.15.104:52562) 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca5102700 1 -- 172.21.15.104:0/3506263780 learned_addr learned my addr 172.21.15.104:0/3506263780 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca5903700 1 --2- 172.21.15.104:0/3506263780 >> v2:172.21.15.114:3300/0 conn(0x7faca00b1070 0x7faca00b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 -- 172.21.15.104:0/3506263780 >> v2:172.21.15.114:3300/0 conn(0x7faca00b1070 msgr2=0x7faca00b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 --2- 172.21.15.104:0/3506263780 >> v2:172.21.15.114:3300/0 conn(0x7faca00b1070 0x7faca00b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 -- 172.21.15.104:0/3506263780 >> v1:172.21.15.104:6789/0 conn(0x7faca006ea70 legacy=0x7faca00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 -- 172.21.15.104:0/3506263780 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca006df90 con 0x7faca00aec00 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca4901700 1 -- 172.21.15.104:0/3506263780 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662185993 0 0) 0x7faca006e5e0 con 0x7faca006ea70 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca6104700 1 --2- 172.21.15.104:0/3506263780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca00aec00 0x7faca00ac8d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faca00d0660 tx=0x7fac9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc153a75db29cb8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca4901700 1 -- 172.21.15.104:0/3506263780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9800e070 con 0x7faca00aec00 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca4901700 1 -- 172.21.15.104:0/3506263780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9800bbd0 con 0x7faca00aec00 2023-06-10T18:44:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.889+0000 7faca4901700 1 -- 172.21.15.104:0/3506263780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac98012430 con 0x7faca00aec00 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 -- 172.21.15.104:0/3506263780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca00aec00 msgr2=0x7faca00ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 --2- 172.21.15.104:0/3506263780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca00aec00 0x7faca00ac8d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faca00d0660 tx=0x7fac9800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 -- 172.21.15.104:0/3506263780 shutdown_connections 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 --2- 172.21.15.104:0/3506263780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca00aec00 0x7faca00ac8d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 --2- 172.21.15.104:0/3506263780 >> v2:172.21.15.114:3300/0 conn(0x7faca00b1070 0x7faca00b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 -- 172.21.15.104:0/3506263780 >> 172.21.15.104:0/3506263780 conn(0x7faca006b8a0 msgr2=0x7faca00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 -- 172.21.15.104:0/3506263780 shutdown_connections 2023-06-10T18:44:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.890+0000 7faca7b67700 1 -- 172.21.15.104:0/3506263780 wait complete. 2023-06-10T18:44:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.891+0000 7faca7b67700 1 Processor -- start 2023-06-10T18:44:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.891+0000 7faca7b67700 1 -- start start 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00aec00 0x7faca00c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00b1070 0x7faca00ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca0148f20 con 0x7faca006ea70 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca00c6480 con 0x7faca00b1070 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca7b67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca00c61a0 con 0x7faca00aec00 2023-06-10T18:44:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54738/0 (socket says 172.21.15.104:54738) 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 -- 172.21.15.104:0/1977529942 learned_addr learned my addr 172.21.15.104:0/1977529942 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 -- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00aec00 msgr2=0x7faca00c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00aec00 0x7faca00c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 -- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00b1070 msgr2=0x7faca00ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00b1070 0x7faca00ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 -- 172.21.15.104:0/1977529942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac98009a70 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.892+0000 7faca5903700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00d7d0 tx=0x7fac9c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0125f0 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c012750 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca00acd70 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c012b00 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca00ad330 con 0x7faca006ea70 2023-06-10T18:44:33.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.893+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca00ad6a0 con 0x7faca006ea70 2023-06-10T18:44:33.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.894+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac9c012c60 con 0x7faca006ea70 2023-06-10T18:44:33.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.894+0000 7fac967fc700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 0x7fac8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:33.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.895+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fac9c089900 con 0x7faca006ea70 2023-06-10T18:44:33.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.895+0000 7faca5102700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 0x7fac8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:33.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.895+0000 7faca5102700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 0x7fac8406d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fac900011e0 tx=0x7fac900079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:33.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:33.899+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac9c0546c0 con 0x7faca006ea70 2023-06-10T18:44:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.087+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faca00acbd0 con 0x7faca006ea70 2023-06-10T18:44:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.089+0000 7fac967fc700 1 -- 172.21.15.104:0/1977529942 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7faca00acbd0 con 0x7faca006ea70 2023-06-10T18:44:34.095 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:34.095 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 msgr2=0x7fac8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 0x7fac8406d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fac900011e0 tx=0x7fac900079b0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 msgr2=0x7faca00c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00d7d0 tx=0x7fac9c00db90 comp rx=0 tx=0).stop 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 shutdown_connections 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac8406ace0 0x7fac8406d190 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00b1070 0x7faca00ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faca006ea70 0x7faca00c3ac0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 --2- 172.21.15.104:0/1977529942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00aec00 0x7faca00c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 >> 172.21.15.104:0/1977529942 conn(0x7faca006b8a0 msgr2=0x7faca006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 shutdown_connections 2023-06-10T18:44:34.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.091+0000 7faca7b67700 1 -- 172.21.15.104:0/1977529942 wait complete. 2023-06-10T18:44:34.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-06-10T18:44:34.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-06-10T18:44:34.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-06-10T18:44:34.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-06-10T18:44:34.111 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-06-10T18:44:34.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-06-10T18:44:34.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-06-10T18:44:34.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-06-10T18:44:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.310+0000 7f6b3459f700 1 Processor -- start 2023-06-10T18:44:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.310+0000 7f6b3459f700 1 -- start start 2023-06-10T18:44:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b3459f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b3459f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c0ccc20 con 0x7f6b2c0c92e0 2023-06-10T18:44:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b3459f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c0ce0c0 con 0x7f6b2c158310 2023-06-10T18:44:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b3459f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c15cc50 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b32b3c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b32b3c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45902/0 (socket says 172.21.15.104:45902) 2023-06-10T18:44:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.312+0000 7f6b32b3c700 1 -- 172.21.15.104:0/1615490149 learned_addr learned my addr 172.21.15.104:0/1615490149 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.313+0000 7f6b32b3c700 1 -- 172.21.15.104:0/1615490149 >> v1:172.21.15.114:6789/0 conn(0x7f6b2c158310 legacy=0x7f6b2c0c8bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.313+0000 7f6b32b3c700 1 -- 172.21.15.104:0/1615490149 >> v1:172.21.15.104:6789/0 conn(0x7f6b2c0c92e0 legacy=0x7f6b2c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.314+0000 7f6b32b3c700 1 -- 172.21.15.104:0/1615490149 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0cc2f0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.314+0000 7f6b32b3c700 1 --2- 172.21.15.104:0/1615490149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00a9c0 tx=0x7f6b1c009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7128038daf642bb7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.314+0000 7fe2d7ca2700 1 Processor -- start 2023-06-10T18:44:34.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f6b31339700 1 -- 172.21.15.104:0/1615490149 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004030 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f6b31339700 1 -- 172.21.15.104:0/1615490149 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c004190 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f6b31339700 1 -- 172.21.15.104:0/1615490149 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004520 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f16b94a1700 1 Processor -- start 2023-06-10T18:44:34.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f6b3459f700 1 -- 172.21.15.104:0/1615490149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 msgr2=0x7f6b2c163080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7f6b3459f700 1 --2- 172.21.15.104:0/1615490149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00a9c0 tx=0x7f6b1c009520 comp rx=0 tx=0).stop 2023-06-10T18:44:34.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7f16b94a1700 1 -- start start 2023-06-10T18:44:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.315+0000 7fe2d7ca2700 1 -- start start 2023-06-10T18:44:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d7ca2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 0x7fe2d00cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d7ca2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00cdae0 con 0x7fe2d00b1080 2023-06-10T18:44:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d7ca2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00c9e60 con 0x7fe2d00cd680 2023-06-10T18:44:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d7ca2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d00c9bc0 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d5a3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 0x7fe2d00cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d623f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe2d00cd680 0x7fe2d006dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58426/0 (socket says 172.21.15.104:58426) 2023-06-10T18:44:34.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.316+0000 7fe2d623f700 1 -- 172.21.15.104:0/3676855038 learned_addr learned my addr 172.21.15.104:0/3676855038 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f6b3459f700 1 -- 172.21.15.104:0/1615490149 shutdown_connections 2023-06-10T18:44:34.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f6b3459f700 1 --2- 172.21.15.104:0/1615490149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c0cebb0 0x7f6b2c163080 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f6b3459f700 1 -- 172.21.15.104:0/1615490149 >> 172.21.15.104:0/1615490149 conn(0x7f6b2c0b0090 msgr2=0x7f6b2c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b94a1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16b4142540 0x7f16b40c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b94a1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b41429a0 0x7f16b40ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b94a1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40cd470 con 0x7f16b40ce9d0 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b94a1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b40ce070 con 0x7f16b4142540 2023-06-10T18:44:34.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b94a1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40c5df0 con 0x7f16b41429a0 2023-06-10T18:44:34.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f6b3459f700 1 -- 172.21.15.104:0/1615490149 shutdown_connections 2023-06-10T18:44:34.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16b4142540 0x7f16b40c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16b4142540 0x7f16b40c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54760/0 (socket says 172.21.15.104:54760) 2023-06-10T18:44:34.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b2ffd700 1 -- 172.21.15.104:0/1613840146 learned_addr learned my addr 172.21.15.104:0/1613840146 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f16b27fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b41429a0 0x7f16b40ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.317+0000 7f6b3459f700 1 -- 172.21.15.104:0/1615490149 wait complete. 2023-06-10T18:44:34.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7fc8917a7700 1 Processor -- start 2023-06-10T18:44:34.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b2ffd700 1 -- 172.21.15.104:0/1613840146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b41429a0 msgr2=0x7f16b40ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b2ffd700 1 --2- 172.21.15.104:0/1613840146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b41429a0 0x7f16b40ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b2ffd700 1 -- 172.21.15.104:0/1613840146 >> v1:172.21.15.104:6789/0 conn(0x7f16b40ce9d0 legacy=0x7f16b4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b2ffd700 1 -- 172.21.15.104:0/1613840146 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40cd710 con 0x7f16b4142540 2023-06-10T18:44:34.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b2ffd700 1 --2- 172.21.15.104:0/1613840146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b4142540 0x7f16b40c55a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009920 tx=0x7f16ac009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb46aec94dff6a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b1ffb700 1 -- 172.21.15.104:0/1613840146 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00ba20 con 0x7f16b4142540 2023-06-10T18:44:34.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b1ffb700 1 -- 172.21.15.104:0/1613840146 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac00bb80 con 0x7f16b4142540 2023-06-10T18:44:34.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7f16b1ffb700 1 -- 172.21.15.104:0/1613840146 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac009e70 con 0x7f16b4142540 2023-06-10T18:44:34.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.318+0000 7fc8917a7700 1 -- start start 2023-06-10T18:44:34.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc88c15ac20 0x7fc88c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc88c06eae0 0x7fc88c160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c15aff0 con 0x7fc88c06eae0 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c06ef30 con 0x7fc88c15ac20 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc8917a7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0b3d90 con 0x7fc88c160f70 2023-06-10T18:44:34.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7f6b3459f700 1 Processor -- start 2023-06-10T18:44:34.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7f6b3459f700 1 -- start start 2023-06-10T18:44:34.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45918/0 (socket says 172.21.15.104:45918) 2023-06-10T18:44:34.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 -- 172.21.15.104:0/2250806563 learned_addr learned my addr 172.21.15.104:0/2250806563 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7f16b94a1700 1 -- 172.21.15.104:0/1613840146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b4142540 msgr2=0x7f16b40c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7f16b94a1700 1 --2- 172.21.15.104:0/1613840146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b4142540 0x7f16b40c55a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009920 tx=0x7f16ac009c30 comp rx=0 tx=0).stop 2023-06-10T18:44:34.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 -- 172.21.15.104:0/2250806563 >> v2:172.21.15.114:3300/0 conn(0x7fc88c15ac20 msgr2=0x7fc88c06d030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 --2- 172.21.15.104:0/2250806563 >> v2:172.21.15.114:3300/0 conn(0x7fc88c15ac20 0x7fc88c06d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 -- 172.21.15.104:0/2250806563 >> v2:172.21.15.104:3300/0 conn(0x7fc88c06eae0 msgr2=0x7fc88c160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 --2- 172.21.15.104:0/2250806563 >> v2:172.21.15.104:3300/0 conn(0x7fc88c06eae0 0x7fc88c160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.319+0000 7fc88b7fe700 1 -- 172.21.15.104:0/2250806563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c06c750 con 0x7fc88c160f70 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f16b94a1700 1 -- 172.21.15.104:0/1613840146 shutdown_connections 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f16b94a1700 1 --2- 172.21.15.104:0/1613840146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b4142540 0x7f16b40c55a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f16b94a1700 1 --2- 172.21.15.104:0/1613840146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b41429a0 0x7f16b40ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f16b94a1700 1 -- 172.21.15.104:0/1613840146 >> 172.21.15.104:0/1613840146 conn(0x7f16b40ceec0 msgr2=0x7f16b40cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fc88b7fe700 1 --2- 172.21.15.104:0/2250806563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc88c067240 tx=0x7fc87c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc17805ee5a959e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fccd151c700 1 Processor -- start 2023-06-10T18:44:34.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fccd151c700 1 -- start start 2023-06-10T18:44:34.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fc889ffb700 1 -- 172.21.15.104:0/2250806563 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00b870 con 0x7fc88c160f70 2023-06-10T18:44:34.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fc889ffb700 1 -- 172.21.15.104:0/2250806563 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c00b9d0 con 0x7fc88c160f70 2023-06-10T18:44:34.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fc889ffb700 1 -- 172.21.15.104:0/2250806563 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00bd60 con 0x7fc88c160f70 2023-06-10T18:44:34.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fe2d5a3e700 1 -- 172.21.15.104:0/3676855038 >> v1:172.21.15.114:6789/0 conn(0x7fe2d00cd680 legacy=0x7fe2d006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fe2d5a3e700 1 -- 172.21.15.104:0/3676855038 >> v1:172.21.15.104:6789/0 conn(0x7fe2d00b1080 legacy=0x7fe2d00ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fe2d5a3e700 1 -- 172.21.15.104:0/3676855038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00aee70 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b2c0c92e0 0x7f6b2c143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 0x7f6b2c0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c158310 0x7f6b2c0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c154f80 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0c8430 con 0x7f6b2c0c92e0 2023-06-10T18:44:34.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b3459f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c154650 con 0x7f6b2c158310 2023-06-10T18:44:34.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7fe2d5a3e700 1 --2- 172.21.15.104:0/3676855038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 0x7fe2d00cd140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d080 tx=0x7fe2c800d390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e252c50ec6c71bfa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b31b3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 0x7f6b2c0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b31b3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 0x7f6b2c0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54770/0 (socket says 172.21.15.104:54770) 2023-06-10T18:44:34.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.320+0000 7f6b31b3a700 1 -- 172.21.15.104:0/2108816096 learned_addr learned my addr 172.21.15.104:0/2108816096 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b32b3c700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c158310 0x7f6b2c0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b3233b700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b2c0c92e0 0x7f6b2c143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fe2d4a3c700 1 -- 172.21.15.104:0/3676855038 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8010070 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fe2d4a3c700 1 -- 172.21.15.104:0/3676855038 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c800b2e0 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fe2d4a3c700 1 -- 172.21.15.104:0/3676855038 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8003c40 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f16b94a1700 1 -- 172.21.15.104:0/1613840146 shutdown_connections 2023-06-10T18:44:34.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 -- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c158310 msgr2=0x7f6b2c0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b2c158310 0x7f6b2c0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 -- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b2c0c92e0 msgr2=0x7f6b2c143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b2c0c92e0 0x7f6b2c143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c00a5f0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b31b3a700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 0x7f6b2c0c7a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800c4a0 tx=0x7f6b2800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b280054e0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b28003680 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28003a30 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c154930 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c0b13b0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 -- 172.21.15.104:0/2250806563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 msgr2=0x7fc88c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 --2- 172.21.15.104:0/2250806563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc88c067240 tx=0x7fc87c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 -- 172.21.15.104:0/2250806563 shutdown_connections 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 --2- 172.21.15.104:0/2250806563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c069e70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 --2- 172.21.15.104:0/2250806563 >> v2:172.21.15.114:3300/0 conn(0x7fc88c15ac20 0x7fc88c06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 --2- 172.21.15.104:0/2250806563 >> v2:172.21.15.104:3300/0 conn(0x7fc88c06eae0 0x7fc88c160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 -- 172.21.15.104:0/2250806563 >> 172.21.15.104:0/2250806563 conn(0x7fc88c0b05b0 msgr2=0x7fc88c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 -- 172.21.15.104:0/2250806563 shutdown_connections 2023-06-10T18:44:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7f16b94a1700 1 -- 172.21.15.104:0/1613840146 wait complete. 2023-06-10T18:44:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/3676855038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 msgr2=0x7fe2d00cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fe2d7ca2700 1 --2- 172.21.15.104:0/3676855038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 0x7fe2d00cd140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d080 tx=0x7fe2c800d390 comp rx=0 tx=0).stop 2023-06-10T18:44:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7f16b94a1700 1 Processor -- start 2023-06-10T18:44:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7f16b94a1700 1 -- start start 2023-06-10T18:44:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/3676855038 shutdown_connections 2023-06-10T18:44:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fe2d7ca2700 1 --2- 172.21.15.104:0/3676855038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00aeaa0 0x7fe2d00cd140 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/3676855038 >> 172.21.15.104:0/3676855038 conn(0x7fe2d006b8a0 msgr2=0x7fe2d00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/3676855038 shutdown_connections 2023-06-10T18:44:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/3676855038 wait complete. 2023-06-10T18:44:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b2c0b15d0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fc8917a7700 1 -- 172.21.15.104:0/2250806563 wait complete. 2023-06-10T18:44:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fc8917a7700 1 Processor -- start 2023-06-10T18:44:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccd151c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcccc1606f0 0x7fcccc06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccd151c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0cb930 con 0x7fcccc0cc520 2023-06-10T18:44:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccd151c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0cd4a0 con 0x7fcccc0cc150 2023-06-10T18:44:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccd151c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0c1e50 con 0x7fcccc1606f0 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccca7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcccc1606f0 0x7fcccc06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccca7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcccc1606f0 0x7fcccc06e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54778/0 (socket says 172.21.15.104:54778) 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.321+0000 7fccca7fc700 1 -- 172.21.15.104:0/2867554947 learned_addr learned my addr 172.21.15.104:0/2867554947 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccca7fc700 1 -- 172.21.15.104:0/2867554947 >> v1:172.21.15.114:6789/0 conn(0x7fcccc0cc150 legacy=0x7fcccc0cb140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccca7fc700 1 -- 172.21.15.104:0/2867554947 >> v1:172.21.15.104:6789/0 conn(0x7fcccc0cc520 legacy=0x7fcccc0cd980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccca7fc700 1 -- 172.21.15.104:0/2867554947 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0c8d00 con 0x7fcccc1606f0 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccca7fc700 1 --2- 172.21.15.104:0/2867554947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc1606f0 0x7fcccc06e780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009e50 tx=0x7fccc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d84269f87f81eb7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccc9ffb700 1 -- 172.21.15.104:0/2867554947 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400e070 con 0x7fcccc1606f0 2023-06-10T18:44:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16b40ce9d0 0x7f16b415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 0x7f16b4159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b41429a0 0x7f16b41500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b41597a0 con 0x7f16b41429a0 2023-06-10T18:44:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4159a40 con 0x7f16b40ce9d0 2023-06-10T18:44:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7f16b94a1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4159d20 con 0x7f16b4142540 2023-06-10T18:44:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b28003b90 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f6b22ffd700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b1006ade0 0x7f6b1006d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccc9ffb700 1 -- 172.21.15.104:0/2867554947 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc4003a20 con 0x7fcccc1606f0 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.322+0000 7fccc9ffb700 1 -- 172.21.15.104:0/2867554947 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4003dd0 con 0x7fcccc1606f0 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fccd151c700 1 -- 172.21.15.104:0/2867554947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc1606f0 msgr2=0x7fcccc06e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.323+0000 7fccd151c700 1 --2- 172.21.15.104:0/2867554947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc1606f0 0x7fcccc06e780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009e50 tx=0x7fccc400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fccd151c700 1 -- 172.21.15.104:0/2867554947 shutdown_connections 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fccd151c700 1 --2- 172.21.15.104:0/2867554947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc1606f0 0x7fcccc06e780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fccd151c700 1 -- 172.21.15.104:0/2867554947 >> 172.21.15.104:0/2867554947 conn(0x7fcccc0b0090 msgr2=0x7fcccc0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fccd151c700 1 -- 172.21.15.104:0/2867554947 shutdown_connections 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.324+0000 7fccd151c700 1 -- 172.21.15.104:0/2867554947 wait complete. 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fccd151c700 1 Processor -- start 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7fc8917a7700 1 -- start start 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 0x7fc88c0cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c15ac20 0x7fc88c161e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0cc370 con 0x7fc88c15ac20 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0cb760 con 0x7fc88c06eae0 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc8917a7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0cba40 con 0x7fc88c160f70 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f16b27fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 0x7f16b4159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f16b27fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 0x7f16b4159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:45932/0 (socket says 172.21.15.104:45932) 2023-06-10T18:44:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f16b27fc700 1 -- 172.21.15.104:0/2577779695 learned_addr learned my addr 172.21.15.104:0/2577779695 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f16b37fe700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b41429a0 0x7f16b41500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7fe2d7ca2700 1 Processor -- start 2023-06-10T18:44:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7fe2d7ca2700 1 -- start start 2023-06-10T18:44:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f27c9f70700 1 Processor -- start 2023-06-10T18:44:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.325+0000 7f27c9f70700 1 -- start start 2023-06-10T18:44:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6b28088be0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b22ffd700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f6b10074a30 0x7f6b10076e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b10077530 con 0x7f6b10074a30 2023-06-10T18:44:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b3233b700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b1006ade0 0x7f6b1006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b32b3c700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f6b10074a30 0x7f6b10076e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc88affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 0x7fc88c0cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc88b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc88affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 0x7fc88c0cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42288/0 (socket says 172.21.15.104:42288) 2023-06-10T18:44:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fc88affd700 1 -- 172.21.15.104:0/581446151 learned_addr learned my addr 172.21.15.104:0/581446151 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b280011e0 con 0x7f6b2c0cebb0 2023-06-10T18:44:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 -- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 msgr2=0x7fc88c139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc88c160f70 0x7fc88c139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 -- 172.21.15.104:0/581446151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c15ac20 msgr2=0x7fc88c161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88a7fc700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c15ac20 0x7fc88c161e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7f27c9f70700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f27c4159f40 0x7f27c40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c9f70700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c40cd160 con 0x7f27c406a580 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c9f70700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c406d980 con 0x7f27c40ce110 2023-06-10T18:44:34.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c9f70700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c40b39d0 con 0x7f27c4159f40 2023-06-10T18:44:34.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2d00aeaa0 0x7fe2d014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d00b1080 0x7fe2d00c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00cd680 0x7fe2d01f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d00b2a80 con 0x7fe2d00aeaa0 2023-06-10T18:44:34.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d00b4670 con 0x7fe2d00b1080 2023-06-10T18:44:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fe2d7ca2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d01fa3b0 con 0x7fe2d00cd680 2023-06-10T18:44:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fe2d523d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d00b1080 0x7fe2d00c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fe2d523d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d00b1080 0x7fe2d00c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42300/0 (socket says 172.21.15.104:42300) 2023-06-10T18:44:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fe2d523d700 1 -- 172.21.15.104:0/779931467 learned_addr learned my addr 172.21.15.104:0/779931467 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c15ac20 0x7fc88c161e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c009550 con 0x7fc88c06eae0 2023-06-10T18:44:34.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88a7fc700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c15ac20 0x7fc88c161e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:34.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fe2d623f700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00cd680 0x7fe2d01f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fe2d5a3e700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2d00aeaa0 0x7fe2d014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f27c4159f40 0x7f27c40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f27c4159f40 0x7f27c40cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54804/0 (socket says 172.21.15.104:54804) 2023-06-10T18:44:34.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f27c37fe700 1 -- 172.21.15.104:0/3925565515 learned_addr learned my addr 172.21.15.104:0/3925565515 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7fc88affd700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 0x7fc88c0cb180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc88000b550 tx=0x7fc88000b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fe2d623f700 1 -- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d00b1080 msgr2=0x7fe2d00c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fe2d623f700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d00b1080 0x7fe2d00c7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fe2d623f700 1 -- 172.21.15.104:0/779931467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2d00aeaa0 msgr2=0x7fe2d014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fe2d623f700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2d00aeaa0 0x7fe2d014da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fe2d623f700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c800bc80 con 0x7fe2d00cd680 2023-06-10T18:44:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f27c37fe700 1 -- 172.21.15.104:0/3925565515 >> v1:172.21.15.114:6789/0 conn(0x7f27c40ce110 legacy=0x7f27c4069e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f27c37fe700 1 -- 172.21.15.104:0/3925565515 >> v1:172.21.15.104:6789/0 conn(0x7f27c406a580 legacy=0x7f27c415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f27c37fe700 1 -- 172.21.15.104:0/3925565515 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c40cd3d0 con 0x7f27c4159f40 2023-06-10T18:44:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f6b32b3c700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f6b10074a30 0x7f6b10076e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.326+0000 7fccd151c700 1 -- start start 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc0cc150 0x7fcccc200910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc0cc520 0x7fcccc0c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcccc1606f0 0x7fcccc203b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0c6f60 con 0x7fcccc0cc520 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0c7240 con 0x7fcccc0cc150 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccd151c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0c7520 con 0x7fcccc1606f0 2023-06-10T18:44:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fcccaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc0cc150 0x7fcccc200910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fcccaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc0cc150 0x7fcccc200910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42310/0 (socket says 172.21.15.104:42310) 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fcccaffd700 1 -- 172.21.15.104:0/2852369141 learned_addr learned my addr 172.21.15.104:0/2852369141 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc0cc520 0x7fcccc0c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fcccb7fe700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcccc1606f0 0x7fcccc203b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 -- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcccc1606f0 msgr2=0x7fcccc203b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcccc1606f0 0x7fcccc203b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 -- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc0cc150 msgr2=0x7fcccc200910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc0cc150 0x7fcccc200910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4009b30 con 0x7fcccc0cc520 2023-06-10T18:44:34.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880011830 con 0x7fc88c06eae0 2023-06-10T18:44:34.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8800119b0 con 0x7fc88c06eae0 2023-06-10T18:44:34.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880011d60 con 0x7fc88c06eae0 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.327+0000 7f16b2ffd700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16b40ce9d0 0x7f16b415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f16b27fc700 1 -- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16b40ce9d0 msgr2=0x7f16b415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f16b27fc700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16b40ce9d0 0x7f16b415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f16b27fc700 1 -- 172.21.15.104:0/2577779695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b41429a0 msgr2=0x7f16b41500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f16b27fc700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16b41429a0 0x7f16b41500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f16b27fc700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac009550 con 0x7f16b4142540 2023-06-10T18:44:34.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c0cbd20 con 0x7fc88c06eae0 2023-06-10T18:44:34.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f27c37fe700 1 --2- 172.21.15.104:0/3925565515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4159f40 0x7f27c40cdbd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27b80098f0 tx=0x7f27b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40add32ea5511fe2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c1629c0 con 0x7fc88c06eae0 2023-06-10T18:44:34.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7f6b3233b700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b1006ade0 0x7f6b1006d290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009fd0 tx=0x7f6b2400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.328+0000 7fccca7fc700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc0cc520 0x7fcccc0c66a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fccc4014000 tx=0x7fccc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16b27fc700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 0x7f16b4159260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16a4007ec0 tx=0x7f16a400e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4004440 con 0x7fcccc0cc520 2023-06-10T18:44:34.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc40045c0 con 0x7fcccc0cc520 2023-06-10T18:44:34.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4004970 con 0x7fcccc0cc520 2023-06-10T18:44:34.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a400eea0 con 0x7f16b4142540 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a4003a20 con 0x7f16b4142540 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4003d90 con 0x7f16b4142540 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b4157470 con 0x7f16b4142540 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b4157a30 con 0x7f16b4142540 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f27c27fc700 1 -- 172.21.15.104:0/3925565515 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8004230 con 0x7f27c4159f40 2023-06-10T18:44:34.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f27c27fc700 1 -- 172.21.15.104:0/3925565515 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b8004390 con 0x7f27c4159f40 2023-06-10T18:44:34.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f27c27fc700 1 -- 172.21.15.104:0/3925565515 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b80036a0 con 0x7f27c4159f40 2023-06-10T18:44:34.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== osd.1 v2:172.21.15.104:6816/2177374922 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6b10077530 con 0x7f6b10074a30 2023-06-10T18:44:34.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.329+0000 7fe2d623f700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00cd680 0x7fe2d01f9840 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c470 tx=0x7fe2cc00c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc004c80 con 0x7fe2d00cd680 2023-06-10T18:44:34.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc004de0 con 0x7fe2d00cd680 2023-06-10T18:44:34.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0072f0 con 0x7fe2d00cd680 2023-06-10T18:44:34.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7f27c9f70700 1 -- 172.21.15.104:0/3925565515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4159f40 msgr2=0x7f27c40cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7f27c9f70700 1 --2- 172.21.15.104:0/3925565515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4159f40 0x7f27c40cdbd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27b80098f0 tx=0x7f27b8009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:34.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7f27c9f70700 1 -- 172.21.15.104:0/3925565515 shutdown_connections 2023-06-10T18:44:34.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7f27c9f70700 1 --2- 172.21.15.104:0/3925565515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c4159f40 0x7f27c40cdbd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7f27c9f70700 1 -- 172.21.15.104:0/3925565515 >> 172.21.15.104:0/3925565515 conn(0x7f27c40b05b0 msgr2=0x7f27c40b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0c7800 con 0x7fcccc0cc520 2023-06-10T18:44:34.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0c6be0 con 0x7fcccc0cc520 2023-06-10T18:44:34.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d01fa690 con 0x7fe2d00cd680 2023-06-10T18:44:34.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.330+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d00b2710 con 0x7fe2d00cd680 2023-06-10T18:44:34.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.331+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc878000ff0 con 0x7fc88c06eae0 2023-06-10T18:44:34.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.332+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc88001a400 con 0x7fc88c06eae0 2023-06-10T18:44:34.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.332+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2d006e820 con 0x7fe2d00cd680 2023-06-10T18:44:34.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1698000fc0 con 0x7f16b4142540 2023-06-10T18:44:34.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.332+0000 7fc8737fe700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc87406ae90 0x7fc87406d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.332+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc880089680 con 0x7fc88c06eae0 2023-06-10T18:44:34.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc8737fe700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fc874074b60 0x7fc874076f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc874077660 con 0x7fc874074b60 2023-06-10T18:44:34.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc88008bd90 con 0x7fc88c06eae0 2023-06-10T18:44:34.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc88a7fc700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc87406ae90 0x7fc87406d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe2cc007450 con 0x7fe2d00cd680 2023-06-10T18:44:34.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc88b7fe700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fc874074b60 0x7fc874076f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.332+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcccc0c73b0 con 0x7fcccc0cc520 2023-06-10T18:44:34.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7f27c9f70700 1 -- 172.21.15.104:0/3925565515 shutdown_connections 2023-06-10T18:44:34.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7f27c9f70700 1 -- 172.21.15.104:0/3925565515 wait complete. 2023-06-10T18:44:34.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc88b7fe700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fc874074b60 0x7fc874076f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fc88a7fc700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc87406ae90 0x7fc87406d340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0cdf30 tx=0x7fc8840073c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fccc4012430 con 0x7fcccc0cc520 2023-06-10T18:44:34.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.333+0000 7fe2c67fc700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2b406ad30 0x7fe2b406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe2cc089150 con 0x7fe2d00cd680 2023-06-10T18:44:34.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2d5a3e700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2b406ad30 0x7fe2b406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f16a4004030 con 0x7f16b4142540 2023-06-10T18:44:34.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2c67fc700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fe2b4074a00 0x7fe2b4076e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2b4077500 con 0x7fe2b4074a00 2023-06-10T18:44:34.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc08b860 con 0x7fe2d00cd680 2023-06-10T18:44:34.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fe2d523d700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fe2b4074a00 0x7fe2b4076e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fccb37fe700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fccb806adb0 0x7fccb806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fdcaf206700 1 Processor -- start 2023-06-10T18:44:34.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fccc408b320 con 0x7fcccc0cc520 2023-06-10T18:44:34.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fdcaf206700 1 -- start start 2023-06-10T18:44:34.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fccb37fe700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fccb8074a00 0x7fccb8076e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccb8077500 con 0x7fccb8074a00 2023-06-10T18:44:34.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccc40560a0 con 0x7fcccc0cc520 2023-06-10T18:44:34.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.334+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== osd.4 v2:172.21.15.114:6800/3316008371 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc874077660 con 0x7fc874074b60 2023-06-10T18:44:34.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7f27c9f70700 1 Processor -- start 2023-06-10T18:44:34.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fcccb7fe700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fccb8074a00 0x7fccb8076e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fe2d523d700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fe2b4074a00 0x7fe2b4076e30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fcccb7fe700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fccb8074a00 0x7fccb8076e30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fdcaf206700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdca80b07a0 0x7fdca80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fdcaf206700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdca80b0c20 0x7fdca80cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fdcaf206700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca815bfc0 con 0x7fdca815bbf0 2023-06-10T18:44:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fdcaf206700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca80c9970 con 0x7fdca80b0c20 2023-06-10T18:44:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7fdcaf206700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca80ccb70 con 0x7fdca80b07a0 2023-06-10T18:44:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fe2d5a3e700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2b406ad30 0x7fe2b406d1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d050 tx=0x7fe2c800b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fcccaffd700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fccb806adb0 0x7fccb806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdca80b07a0 0x7fdca80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdca80b07a0 0x7fdca80ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42328/0 (socket says 172.21.15.104:42328) 2023-06-10T18:44:34.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 -- 172.21.15.104:0/2572132362 learned_addr learned my addr 172.21.15.104:0/2572132362 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== osd.6 v2:172.21.15.114:6816/4101853965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe2b4077500 con 0x7fe2b4074a00 2023-06-10T18:44:34.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 -- 172.21.15.104:0/2572132362 >> v2:172.21.15.104:3300/0 conn(0x7fdca80b0c20 msgr2=0x7fdca80cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 --2- 172.21.15.104:0/2572132362 >> v2:172.21.15.104:3300/0 conn(0x7fdca80b0c20 0x7fdca80cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 -- 172.21.15.104:0/2572132362 >> v1:172.21.15.114:6789/0 conn(0x7fdca815bbf0 legacy=0x7fdca8070310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fdca7fff700 1 -- 172.21.15.104:0/2572132362 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80cce50 con 0x7fdca80b07a0 2023-06-10T18:44:34.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== osd.2 v2:172.21.15.104:6800/128700238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fccb8077500 con 0x7fccb8074a00 2023-06-10T18:44:34.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.335+0000 7f27c9f70700 1 -- start start 2023-06-10T18:44:34.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 0x7f27c414b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27c4159f40 0x7f27c41fa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c4143a20 0x7f27c4143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c41431d0 con 0x7f27c406a580 2023-06-10T18:44:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4144400 con 0x7f27c4143a20 2023-06-10T18:44:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f27c9f70700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c414bc00 con 0x7f27c4159f40 2023-06-10T18:44:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.336+0000 7f16ab7fe700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f169406adb0 0x7f169406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f16a4089870 con 0x7f16b4142540 2023-06-10T18:44:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16ab7fe700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f1694074a80 0x7f1694076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1694077580 con 0x7f1694074a80 2023-06-10T18:44:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a408bdd0 con 0x7f16b4142540 2023-06-10T18:44:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7fdca7fff700 1 --2- 172.21.15.104:0/2572132362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80ce510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdca80cc100 tx=0x7fdc9c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e12a1b49a55b58 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7fdca77fe700 1 -- 172.21.15.104:0/2572132362 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c004230 con 0x7fdca80b07a0 2023-06-10T18:44:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16b2ffd700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f169406adb0 0x7f169406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7f16b37fe700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f1694074a80 0x7f1694076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7f16b37fe700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f1694074a80 0x7f1694076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7f16b2ffd700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f169406adb0 0x7f169406d260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009840 tx=0x7f16ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7fdca77fe700 1 -- 172.21.15.104:0/2572132362 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c004390 con 0x7fdca80b07a0 2023-06-10T18:44:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7fdca77fe700 1 -- 172.21.15.104:0/2572132362 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c0036a0 con 0x7fdca80b07a0 2023-06-10T18:44:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 -- 172.21.15.104:0/2572132362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 msgr2=0x7fdca80ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 --2- 172.21.15.104:0/2572132362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80ce510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdca80cc100 tx=0x7fdc9c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 -- 172.21.15.104:0/2572132362 shutdown_connections 2023-06-10T18:44:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 --2- 172.21.15.104:0/2572132362 >> v2:172.21.15.104:3300/0 conn(0x7fdca80b0c20 0x7fdca80cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 --2- 172.21.15.104:0/2572132362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80ce510 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 -- 172.21.15.104:0/2572132362 >> 172.21.15.104:0/2572132362 conn(0x7fdca806ab00 msgr2=0x7fdca80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.337+0000 7fcccaffd700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fccb806adb0 0x7fccb806d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009ad0 tx=0x7fccbc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 -- 172.21.15.104:0/2572132362 shutdown_connections 2023-06-10T18:44:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7fdcaf206700 1 -- 172.21.15.104:0/2572132362 wait complete. 2023-06-10T18:44:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.338+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== osd.5 v2:172.21.15.114:6817/1721425129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1694077580 con 0x7f1694074a80 2023-06-10T18:44:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 0x7f27c414b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 0x7f27c414b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54836/0 (socket says 172.21.15.104:54836) 2023-06-10T18:44:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 -- 172.21.15.104:0/1582406082 learned_addr learned my addr 172.21.15.104:0/1582406082 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c3fff700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c4143a20 0x7f27c4143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c2ffd700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27c4159f40 0x7f27c41fa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7fdcaf206700 1 Processor -- start 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7fdcaf206700 1 -- start start 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdca80b0c20 0x7fdca80af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdca815bbf0 0x7fdca80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca816a500 con 0x7fdca80b0c20 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca814a020 con 0x7fdca80b07a0 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcaf206700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80c56f0 con 0x7fdca815bbf0 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdca7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdca80b0c20 0x7fdca80af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdca7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdca80b0c20 0x7fdca80af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54844/0 (socket says 172.21.15.104:54844) 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdca7fff700 1 -- 172.21.15.104:0/1301527986 learned_addr learned my addr 172.21.15.104:0/1301527986 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcacfa2700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcad7a3700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdca815bbf0 0x7fdca80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcacfa2700 1 -- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdca815bbf0 msgr2=0x7fdca80af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcacfa2700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdca815bbf0 0x7fdca80af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcacfa2700 1 -- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdca80b0c20 msgr2=0x7fdca80af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.342+0000 7fdcacfa2700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdca80b0c20 0x7fdca80af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.343+0000 7fdcacfa2700 1 -- 172.21.15.104:0/1301527986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c009550 con 0x7fdca80b07a0 2023-06-10T18:44:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.343+0000 7fdcacfa2700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80b3b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc94006fd0 tx=0x7fdc9400b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 -- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27c4159f40 msgr2=0x7f27c41fa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27c4159f40 0x7f27c41fa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 -- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c4143a20 msgr2=0x7f27c4143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27c4143a20 0x7f27c4143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.340+0000 7f27c37fe700 1 -- 172.21.15.104:0/1582406082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8009550 con 0x7f27c406a580 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27c37fe700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 0x7f27c414b6c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b3c20 tx=0x7f27b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8011830 con 0x7f27c406a580 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b8011990 con 0x7f27c406a580 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8011d20 con 0x7f27c406a580 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c41422b0 con 0x7f27c406a580 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.341+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c4142830 con 0x7f27c406a580 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.343+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94011830 con 0x7fdca80b07a0 2023-06-10T18:44:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94011990 con 0x7fdca80b07a0 2023-06-10T18:44:34.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94011d40 con 0x7fdca80b07a0 2023-06-10T18:44:34.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f27b8021070 con 0x7f27c406a580 2023-06-10T18:44:34.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7f27abfff700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f27b006ace0 0x7f27b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f27b808afa0 con 0x7f27c406a580 2023-06-10T18:44:34.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7f27c2ffd700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f27b006ace0 0x7f27b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.345+0000 7f27c2ffd700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f27b006ace0 0x7f27b006d190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27c415d410 tx=0x7f27bc0094f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca816a750 con 0x7fdca80b07a0 2023-06-10T18:44:34.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.344+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80c3420 con 0x7fdca80b07a0 2023-06-10T18:44:34.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.346+0000 7f27c9f70700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f27c40b31f0 0x7f27c4145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.346+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 --> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27c40cd6b0 con 0x7f27c40b31f0 2023-06-10T18:44:34.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.347+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdc9401a400 con 0x7fdca80b07a0 2023-06-10T18:44:34.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.347+0000 7fdca57fa700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc8c06ad30 0x7fdc8c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.347+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdc94089550 con 0x7fdca80b07a0 2023-06-10T18:44:34.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.347+0000 7fdca7fff700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc8c06ad30 0x7fdc8c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.348+0000 7feeeb8d6700 1 Processor -- start 2023-06-10T18:44:34.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.348+0000 7feeeb8d6700 1 -- start start 2023-06-10T18:44:34.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feeeb8d6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feee4158eb0 0x7feee4159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feeeb8d6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee4069e70 con 0x7feee4070a20 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feeeb8d6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee41595f0 con 0x7feee406a8c0 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feeeb8d6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee41598d0 con 0x7feee4158eb0 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feee9672700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feee4070a20 0x7feee4070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52680/0 (socket says 172.21.15.104:52680) 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.349+0000 7feee9672700 1 -- 172.21.15.104:0/3958153329 learned_addr learned my addr 172.21.15.104:0/3958153329 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feee9e73700 1 --2- 172.21.15.104:0/3958153329 >> v2:172.21.15.114:3300/0 conn(0x7feee4158eb0 0x7feee4159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533004590 0 0) 0x7feee4069e70 con 0x7feee4070a20 2023-06-10T18:44:34.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed00035b0 con 0x7feee4070a20 2023-06-10T18:44:34.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7fdca7fff700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc8c06ad30 0x7fdc8c06d1e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009c10 tx=0x7fdc9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461912142 0 0) 0x7feed00035b0 con 0x7feee4070a20 2023-06-10T18:44:34.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 >> v2:172.21.15.114:3300/0 conn(0x7feee4158eb0 msgr2=0x7feee4159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 --2- 172.21.15.104:0/3958153329 >> v2:172.21.15.114:3300/0 conn(0x7feee4158eb0 0x7feee4159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 >> v1:172.21.15.114:6789/0 conn(0x7feee406a8c0 legacy=0x7feee41587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee406a0c0 con 0x7feee4070a20 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7feee0001670 con 0x7feee4070a20 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 >> v1:172.21.15.104:6789/0 conn(0x7feee4070a20 legacy=0x7feee4070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 0x7feed0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0004850 0x7feed0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feed0005200 0x7feed000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41598d0 con 0x7feed0004850 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed000c110 con 0x7feed0003a00 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed000c2d0 con 0x7feed0005200 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7fdcaf206700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7fdc880015e0 0x7fdc88003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.350+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 --> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc88006bc0 con 0x7fdc880015e0 2023-06-10T18:44:34.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7fdcad7a3700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7fdc880015e0 0x7fdc88003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 0x7feed0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9e73700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feed0005200 0x7feed000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee8e71700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0004850 0x7feed0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 -- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feed0005200 msgr2=0x7feed000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feed0005200 0x7feed000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 -- 172.21.15.104:0/3958153329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0004850 msgr2=0x7feed0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0004850 0x7feed0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9672700 1 -- 172.21.15.104:0/3958153329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee406a0c0 con 0x7feed0003a00 2023-06-10T18:44:34.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7feee9e73700 1 -- 172.21.15.104:0/3958153329 reap_dead start 2023-06-10T18:44:34.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.351+0000 7fdcad7a3700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7fdc880015e0 0x7fdc88003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.352+0000 7feee9672700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 0x7feed0003df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feee0007df0 tx=0x7feed0005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0dc98f1e20b8318 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.352+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0007c50 con 0x7feed0003a00 2023-06-10T18:44:34.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.352+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee0005960 con 0x7feed0003a00 2023-06-10T18:44:34.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.352+0000 7feedbfff700 1 -- 172.21.15.104:0/3958153329 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0005d10 con 0x7feed0003a00 2023-06-10T18:44:34.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.352+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== osd.0 v2:172.21.15.104:6824/1623254903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdc88006bc0 con 0x7fdc880015e0 2023-06-10T18:44:34.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 msgr2=0x7feed0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 0x7feed0003df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feee0007df0 tx=0x7feed0005880 comp rx=0 tx=0).stop 2023-06-10T18:44:34.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3958153329 shutdown_connections 2023-06-10T18:44:34.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3958153329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feed0003a00 0x7feed0003df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3958153329 >> 172.21.15.104:0/3958153329 conn(0x7feee40b1610 msgr2=0x7feee40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3958153329 shutdown_connections 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.353+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3958153329 wait complete. 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.355+0000 7feeeb8d6700 1 Processor -- start 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.355+0000 7feeeb8d6700 1 -- start start 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0003a00 0x7feee41e0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feee41e0dd0 0x7feee41f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feee41f61b0 0x7feee41f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41e21a0 con 0x7feed0003a00 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee41e0a60 con 0x7feee41e0dd0 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feeeb8d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee41f6b50 con 0x7feee41f61b0 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feee9672700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0003a00 0x7feee41e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feee9672700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0003a00 0x7feee41e0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54868/0 (socket says 172.21.15.104:54868) 2023-06-10T18:44:34.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.356+0000 7feee9672700 1 -- 172.21.15.104:0/3031271477 learned_addr learned my addr 172.21.15.104:0/3031271477 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feee41f61b0 0x7feee41f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 -- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feee41e0dd0 msgr2=0x7feee41f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feee41e0dd0 0x7feee41f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 -- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0003a00 msgr2=0x7feee41e0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feed0003a00 0x7feee41e0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.357+0000 7feee9e73700 1 -- 172.21.15.104:0/3031271477 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee41e1520 con 0x7feee41f61b0 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feee9e73700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feee41f61b0 0x7feee41f6610 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feee40c9370 tx=0x7feed400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4004440 con 0x7feee41f61b0 2023-06-10T18:44:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed40045a0 con 0x7feee41f61b0 2023-06-10T18:44:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40038e0 con 0x7feee41f61b0 2023-06-10T18:44:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee41e1800 con 0x7feee41f61b0 2023-06-10T18:44:34.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.358+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee41e1dc0 con 0x7feee41f61b0 2023-06-10T18:44:34.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.361+0000 7f27c3fff700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f27c40b31f0 0x7f27c4145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.361+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feed4003a40 con 0x7feee41f61b0 2023-06-10T18:44:34.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.361+0000 7f27c3fff700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f27c40b31f0 0x7f27c4145a10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.362+0000 7feed9ffb700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feec806ad30 0x7feec806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.362+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7feed40885a0 con 0x7feee41f61b0 2023-06-10T18:44:34.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.362+0000 7feee9672700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feec806ad30 0x7feec806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.362+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== osd.7 v2:172.21.15.114:6808/3766591503 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f27b4003a10 con 0x7f27c40b31f0 2023-06-10T18:44:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.362+0000 7feee9672700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feec806ad30 0x7feec806d1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feee0009010 tx=0x7feee0007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.364+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 --> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b2c0cc8b0 con 0x7f6b10074a30 2023-06-10T18:44:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.364+0000 7f6b22ffd700 1 -- 172.21.15.104:0/2108816096 <== osd.1 v2:172.21.15.104:6816/2177374922 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b1c028040 con 0x7f6b10074a30 2023-06-10T18:44:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.364+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 --> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc878002d70 con 0x7fc874074b60 2023-06-10T18:44:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.365+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f6b10074a30 msgr2=0x7f6b10076e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.365+0000 7f6b3459f700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f6b10074a30 0x7f6b10076e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.365+0000 7fc8737fe700 1 -- 172.21.15.104:0/581446151 <== osd.4 v2:172.21.15.114:6800/3316008371 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc878002d70 con 0x7fc874074b60 2023-06-10T18:44:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.365+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fc874074b60 msgr2=0x7fc874076f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.365+0000 7fc8917a7700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fc874074b60 0x7fc874076f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc87406ae90 msgr2=0x7fc87406d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7fc8917a7700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc87406ae90 0x7fc87406d340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0cdf30 tx=0x7fc8840073c0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 msgr2=0x7fc88c0cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7fc8917a7700 1 --2- 172.21.15.104:0/581446151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc88c06eae0 0x7fc88c0cb180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc88000b550 tx=0x7fc88000b860 comp rx=0 tx=0).stop 2023-06-10T18:44:34.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7fc88b7fe700 1 -- 172.21.15.104:0/581446151 reap_dead start 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b1006ade0 msgr2=0x7f6b1006d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b1006ade0 0x7f6b1006d290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009fd0 tx=0x7f6b2400d040 comp rx=0 tx=0).stop 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 msgr2=0x7f6b2c0c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 --2- 172.21.15.104:0/2108816096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b2c0cebb0 0x7f6b2c0c7a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800c4a0 tx=0x7f6b2800c7b0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b32b3c700 1 -- 172.21.15.104:0/2108816096 reap_dead start 2023-06-10T18:44:34.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 shutdown_connections 2023-06-10T18:44:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 >> 172.21.15.104:0/2108816096 conn(0x7f6b2c0b0090 msgr2=0x7f6b2c071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 shutdown_connections 2023-06-10T18:44:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.366+0000 7f6b3459f700 1 -- 172.21.15.104:0/2108816096 wait complete. 2023-06-10T18:44:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.367+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 --> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1698002d40 con 0x7f1694074a80 2023-06-10T18:44:34.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16ab7fe700 1 -- 172.21.15.104:0/2577779695 <== osd.5 v2:172.21.15.114:6817/1721425129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1698002d40 con 0x7f1694074a80 2023-06-10T18:44:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 shutdown_connections 2023-06-10T18:44:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 >> 172.21.15.104:0/581446151 conn(0x7fc88c0b05b0 msgr2=0x7fc88c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 --> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcccc0c92c0 con 0x7fccb8074a00 2023-06-10T18:44:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f1694074a80 msgr2=0x7f1694076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f1694074a80 0x7f1694076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7feec40015e0 0x7feec4003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 --> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feec4006bc0 con 0x7feec40015e0 2023-06-10T18:44:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 shutdown_connections 2023-06-10T18:44:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f169406adb0 msgr2=0x7f169406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f169406adb0 0x7f169406d260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009840 tx=0x7f16ac008030 comp rx=0 tx=0).stop 2023-06-10T18:44:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 msgr2=0x7f16b4159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 --2- 172.21.15.104:0/2577779695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16b4142540 0x7f16b4159260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16a4007ec0 tx=0x7f16a400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b37fe700 1 -- 172.21.15.104:0/2577779695 reap_dead start 2023-06-10T18:44:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 shutdown_connections 2023-06-10T18:44:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.368+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 >> 172.21.15.104:0/2577779695 conn(0x7f16b40ceec0 msgr2=0x7f16b40c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7feee8e71700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7feec40015e0 0x7feec4003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fc8917a7700 1 -- 172.21.15.104:0/581446151 wait complete. 2023-06-10T18:44:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 --> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2d006e540 con 0x7fe2b4074a00 2023-06-10T18:44:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fccb37fe700 1 -- 172.21.15.104:0/2852369141 <== osd.2 v2:172.21.15.104:6800/128700238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fccc00095d0 con 0x7fccb8074a00 2023-06-10T18:44:34.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7feee8e71700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7feec40015e0 0x7feec4003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fccb8074a00 msgr2=0x7fccb8076e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.369+0000 7fccd151c700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fccb8074a00 0x7fccb8076e30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fccb806adb0 msgr2=0x7fccb806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fccb806adb0 0x7fccb806d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009ad0 tx=0x7fccbc009210 comp rx=0 tx=0).stop 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc0cc520 msgr2=0x7fcccc0c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 --2- 172.21.15.104:0/2852369141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcccc0cc520 0x7fcccc0c66a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fccc4014000 tx=0x7fccc4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fcccb7fe700 1 -- 172.21.15.104:0/2852369141 reap_dead start 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 shutdown_connections 2023-06-10T18:44:34.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 >> 172.21.15.104:0/2852369141 conn(0x7fcccc0b0090 msgr2=0x7fcccc070960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 shutdown_connections 2023-06-10T18:44:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7fccd151c700 1 -- 172.21.15.104:0/2852369141 wait complete. 2023-06-10T18:44:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.370+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== osd.3 v2:172.21.15.104:6808/3217431778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7feec4006bc0 con 0x7feec40015e0 2023-06-10T18:44:34.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.371+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 shutdown_connections 2023-06-10T18:44:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.371+0000 7f16b94a1700 1 -- 172.21.15.104:0/2577779695 wait complete. 2023-06-10T18:44:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.371+0000 7fe2c67fc700 1 -- 172.21.15.104:0/779931467 <== osd.6 v2:172.21.15.114:6816/4101853965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2c00015e0 con 0x7fe2b4074a00 2023-06-10T18:44:34.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 --> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdc88005ca0 con 0x7fdc880015e0 2023-06-10T18:44:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fe2b4074a00 msgr2=0x7fe2b4076e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fe2b4074a00 0x7fe2b4076e30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2b406ad30 msgr2=0x7fe2b406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2b406ad30 0x7fe2b406d1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d050 tx=0x7fe2c800b000 comp rx=0 tx=0).stop 2023-06-10T18:44:34.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00cd680 msgr2=0x7fe2d01f9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d7ca2700 1 --2- 172.21.15.104:0/779931467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d00cd680 0x7fe2d01f9840 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c470 tx=0x7fe2cc00c780 comp rx=0 tx=0).stop 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.380+0000 7fe2d623f700 1 -- 172.21.15.104:0/779931467 reap_dead start 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.381+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 shutdown_connections 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.381+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 >> 172.21.15.104:0/779931467 conn(0x7fe2d006b8a0 msgr2=0x7fe2d0157d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.381+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 shutdown_connections 2023-06-10T18:44:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.381+0000 7fe2d7ca2700 1 -- 172.21.15.104:0/779931467 wait complete. 2023-06-10T18:44:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.385+0000 7fdca57fa700 1 -- 172.21.15.104:0/1301527986 <== osd.0 v2:172.21.15.104:6824/1623254903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdc88005ca0 con 0x7fdc880015e0 2023-06-10T18:44:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7fdc880015e0 msgr2=0x7fdc88003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7fdc880015e0 0x7fdc88003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc8c06ad30 msgr2=0x7fdc8c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc8c06ad30 0x7fdc8c06d1e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009c10 tx=0x7fdc9c008030 comp rx=0 tx=0).stop 2023-06-10T18:44:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 msgr2=0x7fdca80b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcaf206700 1 --2- 172.21.15.104:0/1301527986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdca80b07a0 0x7fdca80b3b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc94006fd0 tx=0x7fdc9400b730 comp rx=0 tx=0).stop 2023-06-10T18:44:34.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.386+0000 7fdcad7a3700 1 -- 172.21.15.104:0/1301527986 reap_dead start 2023-06-10T18:44:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.389+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 --> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feec4005ca0 con 0x7feec40015e0 2023-06-10T18:44:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.389+0000 7feed9ffb700 1 -- 172.21.15.104:0/3031271477 <== osd.3 v2:172.21.15.104:6808/3217431778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feec4005ca0 con 0x7feec40015e0 2023-06-10T18:44:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.390+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7feec40015e0 msgr2=0x7feec4003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.390+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7feec40015e0 0x7feec4003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feec806ad30 msgr2=0x7feec806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feec806ad30 0x7feec806d1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feee0009010 tx=0x7feee0007000 comp rx=0 tx=0).stop 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feee41f61b0 msgr2=0x7feee41f6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 --2- 172.21.15.104:0/3031271477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feee41f61b0 0x7feee41f6610 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feee40c9370 tx=0x7feed400f7f0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feee9e73700 1 -- 172.21.15.104:0/3031271477 reap_dead start 2023-06-10T18:44:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 shutdown_connections 2023-06-10T18:44:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 >> 172.21.15.104:0/3031271477 conn(0x7feee40b1610 msgr2=0x7feee4155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 shutdown_connections 2023-06-10T18:44:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.391+0000 7feeeb8d6700 1 -- 172.21.15.104:0/3031271477 wait complete. 2023-06-10T18:44:34.446 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:34.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-06-10T18:44:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.392+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 --> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f27c4142f60 con 0x7f27c40b31f0 2023-06-10T18:44:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 shutdown_connections 2023-06-10T18:44:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 >> 172.21.15.104:0/1301527986 conn(0x7fdca806ab00 msgr2=0x7fdca8070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 shutdown_connections 2023-06-10T18:44:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7fdcaf206700 1 -- 172.21.15.104:0/1301527986 wait complete. 2023-06-10T18:44:34.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7f27abfff700 1 -- 172.21.15.104:0/1582406082 <== osd.7 v2:172.21.15.114:6808/3766591503 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f27c4142f60 con 0x7f27c40b31f0 2023-06-10T18:44:34.448 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:34.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-06-10T18:44:34.450 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f27c40b31f0 msgr2=0x7f27c4145a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.393+0000 7f27c9f70700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f27c40b31f0 0x7f27c4145a10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f27b006ace0 msgr2=0x7f27b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f27b006ace0 0x7f27b006d190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27c415d410 tx=0x7f27bc0094f0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 msgr2=0x7f27c414b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 --2- 172.21.15.104:0/1582406082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27c406a580 0x7f27c414b6c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b3c20 tx=0x7f27b8004750 comp rx=0 tx=0).stop 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c3fff700 1 -- 172.21.15.104:0/1582406082 reap_dead start 2023-06-10T18:44:34.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 shutdown_connections 2023-06-10T18:44:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 >> 172.21.15.104:0/1582406082 conn(0x7f27c40b05b0 msgr2=0x7f27c4071280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 shutdown_connections 2023-06-10T18:44:34.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.394+0000 7f27c9f70700 1 -- 172.21.15.104:0/1582406082 wait complete. 2023-06-10T18:44:34.454 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:34.454 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-06-10T18:44:34.455 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:34.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-06-10T18:44:34.456 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:34.456 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-06-10T18:44:34.457 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:34.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-06-10T18:44:34.458 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:34.458 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-06-10T18:44:34.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.676+0000 7fd453331700 1 Processor -- start 2023-06-10T18:44:34.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.676+0000 7fd453331700 1 -- start start 2023-06-10T18:44:34.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd453331700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd44c158310 0x7fd44c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd453331700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c0c9560 con 0x7fd44c0c9110 2023-06-10T18:44:34.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd453331700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c0ccc20 con 0x7fd44c0cebb0 2023-06-10T18:44:34.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd453331700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd44c15cc50 con 0x7fd44c158310 2023-06-10T18:44:34.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd4508cc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd44c0c9110 0x7fd44c0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52710/0 (socket says 172.21.15.104:52710) 2023-06-10T18:44:34.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.678+0000 7fd4508cc700 1 -- 172.21.15.104:0/1126143904 learned_addr learned my addr 172.21.15.104:0/1126143904 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd4510cd700 1 --2- 172.21.15.104:0/1126143904 >> v2:172.21.15.104:3300/0 conn(0x7fd44c158310 0x7fd44c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722465425 0 0) 0x7fd44c0c9560 con 0x7fd44c0c9110 2023-06-10T18:44:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4380034d0 con 0x7fd44c0c9110 2023-06-10T18:44:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232634542 0 0) 0x7fd4380034d0 con 0x7fd44c0c9110 2023-06-10T18:44:34.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 >> v2:172.21.15.104:3300/0 conn(0x7fd44c158310 msgr2=0x7fd44c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 --2- 172.21.15.104:0/1126143904 >> v2:172.21.15.104:3300/0 conn(0x7fd44c158310 0x7fd44c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 >> v1:172.21.15.114:6789/0 conn(0x7fd44c0cebb0 legacy=0x7fd44c0cd0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0cc2f0 con 0x7fd44c0c9110 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd43c004d60 con 0x7fd44c0c9110 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 >> v1:172.21.15.104:6789/0 conn(0x7fd44c0c9110 legacy=0x7fd44c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd438003af0 0x7fd438003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd43800ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd43800b4e0 0x7fd43800d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c15cc50 con 0x7fd438003af0 2023-06-10T18:44:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43800e270 con 0x7fd43800b4e0 2023-06-10T18:44:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.679+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43800e430 con 0x7fd4380088e0 2023-06-10T18:44:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4510cd700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd438003af0 0x7fd438003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd43800ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 -- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd43800b4e0 msgr2=0x7fd43800d990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd43800b4e0 0x7fd43800d990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 -- 172.21.15.104:0/1126143904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd438003af0 msgr2=0x7fd438003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd438003af0 0x7fd438003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 -- 172.21.15.104:0/1126143904 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0cc2f0 con 0x7fd4380088e0 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.680+0000 7fd4508cc700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd43800ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd43c001ac0 tx=0x7fd438003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13f5d68a921434d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.681+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0127a0 con 0x7fd4380088e0 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.681+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c012900 con 0x7fd4380088e0 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.681+0000 7fd443fff700 1 -- 172.21.15.104:0/1126143904 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c012cb0 con 0x7fd4380088e0 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.682+0000 7fd453331700 1 -- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 msgr2=0x7fd43800ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.682+0000 7fd453331700 1 --2- 172.21.15.104:0/1126143904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd43800ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd43c001ac0 tx=0x7fd438003780 comp rx=0 tx=0).stop 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.682+0000 7fd4518ce700 1 -- 172.21.15.104:0/1126143904 reap_dead start 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.682+0000 7fd453331700 1 -- 172.21.15.104:0/1126143904 shutdown_connections 2023-06-10T18:44:34.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.682+0000 7fd453331700 1 -- 172.21.15.104:0/1126143904 >> 172.21.15.104:0/1126143904 conn(0x7fd44c0b0090 msgr2=0x7fd44c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.690+0000 7fd453331700 1 -- 172.21.15.104:0/1126143904 shutdown_connections 2023-06-10T18:44:34.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.691+0000 7fd453331700 1 -- 172.21.15.104:0/1126143904 wait complete. 2023-06-10T18:44:34.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.692+0000 7fd453331700 1 Processor -- start 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.693+0000 7fd453331700 1 -- start start 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd44c0c2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd44c143520 0x7fd44c1439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c071dd0 con 0x7fd44c071420 2023-06-10T18:44:34.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c0c37c0 con 0x7fd44c143520 2023-06-10T18:44:34.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd453331700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c0c2700 con 0x7fd4380088e0 2023-06-10T18:44:34.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd4508cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd4508cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54888/0 (socket says 172.21.15.104:54888) 2023-06-10T18:44:34.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7fd4508cc700 1 -- 172.21.15.104:0/278889128 learned_addr learned my addr 172.21.15.104:0/278889128 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4518ce700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd44c143520 0x7fd44c1439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4510cd700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd44c0c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.693+0000 7f7440e2d700 1 Processor -- start 2023-06-10T18:44:34.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.693+0000 7f7440e2d700 1 -- start start 2023-06-10T18:44:34.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7f7440e2d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f743c0cea80 0x7f743c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7f7440e2d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743c0c8e00 con 0x7f743c089510 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7f7440e2d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743c0c9290 con 0x7f743c16d0d0 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.694+0000 7f7440e2d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743c0cbea0 con 0x7f743c0cea80 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f7439d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f743c089510 0x7f743c16c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52726/0 (socket says 172.21.15.104:52726) 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f7439d9b700 1 -- 172.21.15.104:0/1075965055 learned_addr learned my addr 172.21.15.104:0/1075965055 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f743a59c700 1 --2- 172.21.15.104:0/1075965055 >> v2:172.21.15.104:3300/0 conn(0x7f743c0cea80 0x7f743c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f51eec4f700 1 Processor -- start 2023-06-10T18:44:34.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4508cc700 1 -- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 msgr2=0x7fd44c0c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4508cc700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd44c0c2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4508cc700 1 -- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd44c143520 msgr2=0x7fd44c1439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4508cc700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd44c143520 0x7fd44c1439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7fd4508cc700 1 -- 172.21.15.104:0/278889128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0c29e0 con 0x7fd44c071420 2023-06-10T18:44:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f743a59c700 1 -- 172.21.15.104:0/1075965055 >> v1:172.21.15.114:6789/0 conn(0x7f743c16d0d0 legacy=0x7f743c0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7f743a59c700 1 -- 172.21.15.104:0/1075965055 >> v1:172.21.15.104:6789/0 conn(0x7f743c089510 legacy=0x7f743c16c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7f743a59c700 1 -- 172.21.15.104:0/1075965055 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c16d530 con 0x7f743c0cea80 2023-06-10T18:44:34.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7fd4508cc700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00d990 tx=0x7fd43c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.695+0000 7f51eec4f700 1 -- start start 2023-06-10T18:44:34.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7f743a59c700 1 --2- 172.21.15.104:0/1075965055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c06e780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f742c0096c0 tx=0x7f742c009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f068a0c308a69b8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51eec4f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51eec4f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f51e80ca8b0 0x7f51e80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51eec4f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80ceeb0 con 0x7f51e80cea50 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51eec4f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e80c1c30 con 0x7f51e80ca8b0 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51eec4f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e8164b90 con 0x7f51e8160880 2023-06-10T18:44:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51ec9eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51ec9eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46052/0 (socket says 172.21.15.104:46052) 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7f51ec9eb700 1 -- 172.21.15.104:0/2877487937 learned_addr learned my addr 172.21.15.104:0/2877487937 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.698+0000 7f51e7fff700 1 --2- 172.21.15.104:0/2877487937 >> v2:172.21.15.114:3300/0 conn(0x7f51e80ca8b0 0x7f51e80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.698+0000 7f51ec9eb700 1 -- 172.21.15.104:0/2877487937 >> v2:172.21.15.114:3300/0 conn(0x7f51e80ca8b0 msgr2=0x7f51e80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.698+0000 7f51ec9eb700 1 --2- 172.21.15.104:0/2877487937 >> v2:172.21.15.114:3300/0 conn(0x7f51e80ca8b0 0x7f51e80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.698+0000 7f51ec9eb700 1 -- 172.21.15.104:0/2877487937 >> v1:172.21.15.104:6789/0 conn(0x7f51e80cea50 legacy=0x7f51e80caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.698+0000 7f51ec9eb700 1 -- 172.21.15.104:0/2877487937 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8161c60 con 0x7f51e8160880 2023-06-10T18:44:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c012240 con 0x7fd44c071420 2023-06-10T18:44:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c0123a0 con 0x7fd44c071420 2023-06-10T18:44:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c012730 con 0x7fd44c071420 2023-06-10T18:44:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.696+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c0c2cc0 con 0x7fd44c071420 2023-06-10T18:44:34.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.697+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c143f00 con 0x7fd44c071420 2023-06-10T18:44:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7f51ec9eb700 1 --2- 172.21.15.104:0/2877487937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098f0 tx=0x7f51dc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7c545b191b44d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7f51e77fe700 1 -- 172.21.15.104:0/2877487937 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc004230 con 0x7f51e8160880 2023-06-10T18:44:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7f51e77fe700 1 -- 172.21.15.104:0/2877487937 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc004390 con 0x7f51e8160880 2023-06-10T18:44:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7f51e77fe700 1 -- 172.21.15.104:0/2877487937 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0036a0 con 0x7f51e8160880 2023-06-10T18:44:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7faf87fb6700 1 Processor -- start 2023-06-10T18:44:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7f9440f7e700 1 Processor -- start 2023-06-10T18:44:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 -- 172.21.15.104:0/2877487937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 msgr2=0x7f51e80c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 --2- 172.21.15.104:0/2877487937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098f0 tx=0x7f51dc009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42c0052f0 con 0x7fd44c071420 2023-06-10T18:44:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd43c018020 con 0x7fd44c071420 2023-06-10T18:44:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7fd441ffb700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 0x7fd43006d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.699+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd43c08ae10 con 0x7fd44c071420 2023-06-10T18:44:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7fd4510cd700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 0x7fd43006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 -- 172.21.15.104:0/2877487937 shutdown_connections 2023-06-10T18:44:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 --2- 172.21.15.104:0/2877487937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c8bd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 --2- 172.21.15.104:0/2877487937 >> v2:172.21.15.114:3300/0 conn(0x7f51e80ca8b0 0x7f51e80ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 -- 172.21.15.104:0/2877487937 >> 172.21.15.104:0/2877487937 conn(0x7f51e806ab00 msgr2=0x7f51e80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f9440f7e700 1 -- start start 2023-06-10T18:44:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 -- 172.21.15.104:0/2877487937 shutdown_connections 2023-06-10T18:44:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7fd4510cd700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 0x7fd43006d230 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd44c152ea0 tx=0x7fd448008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f743959a700 1 -- 172.21.15.104:0/1075965055 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c011610 con 0x7f743c0cea80 2023-06-10T18:44:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f743959a700 1 -- 172.21.15.104:0/1075965055 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c011770 con 0x7f743c0cea80 2023-06-10T18:44:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f743959a700 1 -- 172.21.15.104:0/1075965055 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c011b20 con 0x7f743c0cea80 2023-06-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7faf87fb6700 1 -- start start 2023-06-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f9440f7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c0cec20 0x7f943c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f9440f7e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c164b90 con 0x7f943c0ca8b0 2023-06-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f9440f7e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943c0c9260 con 0x7f943c160880 2023-06-10T18:44:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f9440f7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0c1c30 con 0x7f943c0cec20 2023-06-10T18:44:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f943a59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f943c160880 0x7f943c0ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58582/0 (socket says 172.21.15.104:58582) 2023-06-10T18:44:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7f943a59c700 1 -- 172.21.15.104:0/974107981 learned_addr learned my addr 172.21.15.104:0/974107981 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.701+0000 7faf87fb6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf87fb6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800ae720 con 0x7faf8015c040 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf87fb6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800ce2c0 con 0x7faf800ae280 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf87fb6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf800cf750 con 0x7faf80071130 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46066/0 (socket says 172.21.15.104:46066) 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 -- 172.21.15.104:0/251062829 learned_addr learned my addr 172.21.15.104:0/251062829 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482556755 0 0) 0x7f943c0c9260 con 0x7f943c160880 2023-06-10T18:44:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94240035b0 con 0x7f943c160880 2023-06-10T18:44:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 -- 172.21.15.104:0/251062829 >> v1:172.21.15.114:6789/0 conn(0x7faf800ae280 legacy=0x7faf800ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 -- 172.21.15.104:0/251062829 >> v1:172.21.15.104:6789/0 conn(0x7faf8015c040 legacy=0x7faf80070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7faf85551700 1 -- 172.21.15.104:0/251062829 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800cfa30 con 0x7faf80071130 2023-06-10T18:44:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.700+0000 7f51eec4f700 1 -- 172.21.15.104:0/2877487937 wait complete. 2023-06-10T18:44:34.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f9439d9b700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c0cec20 0x7f943c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777042725 0 0) 0x7f94240035b0 con 0x7f943c160880 2023-06-10T18:44:34.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c0cec20 msgr2=0x7f943c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f51eec4f700 1 Processor -- start 2023-06-10T18:44:34.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f51eec4f700 1 -- start start 2023-06-10T18:44:34.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c0cec20 0x7f943c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 >> v1:172.21.15.104:6789/0 conn(0x7f943c0ca8b0 legacy=0x7f943c160c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c164260 con 0x7f943c160880 2023-06-10T18:44:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9430001670 con 0x7f943c160880 2023-06-10T18:44:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.702+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 >> v1:172.21.15.114:6789/0 conn(0x7f943c160880 legacy=0x7f943c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f96d0e09700 1 Processor -- start 2023-06-10T18:44:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9424003a00 0x7f9424003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9424004850 0x7f9424004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424005200 0x7f942400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0c1c30 con 0x7f9424003a00 2023-06-10T18:44:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942400c040 con 0x7f9424005200 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942400c200 con 0x7f9424004850 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9424003a00 0x7f9424003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943ad9d700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424005200 0x7f942400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 -- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9424004850 msgr2=0x7f9424004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9424004850 0x7f9424004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 -- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424005200 msgr2=0x7f942400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424005200 0x7f942400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 -- 172.21.15.104:0/974107981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c164260 con 0x7f9424003a00 2023-06-10T18:44:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f943a59c700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9424003a00 0x7f9424003e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9430001eb0 tx=0x7f9424005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b681954d125128 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7faf85551700 1 --2- 172.21.15.104:0/251062829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0098a0 tx=0x7faf7c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c11f8becf7bbce72 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7faf84d50700 1 -- 172.21.15.104:0/251062829 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c004230 con 0x7faf80071130 2023-06-10T18:44:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 0x7f51e815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e81fef10 con 0x7f51e80ca8b0 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e80c74c0 con 0x7f51e80cea50 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51eec4f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e80c6c20 con 0x7f51e8160880 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ed1ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51e7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 0x7f51e815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51e7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42426/0 (socket says 172.21.15.104:42426) 2023-06-10T18:44:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51e7fff700 1 -- 172.21.15.104:0/414793449 learned_addr learned my addr 172.21.15.104:0/414793449 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.703+0000 7f96d0e09700 1 -- start start 2023-06-10T18:44:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96d0e09700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0c92e0 0x7f96cc0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96d0e09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96d0e09700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc0cee30 con 0x7f96cc158310 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96d0e09700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc0ccc20 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96d0e09700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15cc50 con 0x7f96cc0c92e0 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96cad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96cad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54930/0 (socket says 172.21.15.104:54930) 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96cad9d700 1 -- 172.21.15.104:0/995970185 learned_addr learned my addr 172.21.15.104:0/995970185 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 -- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 msgr2=0x7f51e80c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 -- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 msgr2=0x7f51e80c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f51ec9eb700 1 -- 172.21.15.104:0/414793449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc009550 con 0x7f51e80ca8b0 2023-06-10T18:44:34.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f51e7fff700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.704+0000 7f96c9d9b700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0c92e0 0x7f96cc0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f96cad9d700 1 -- 172.21.15.104:0/995970185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0c92e0 msgr2=0x7f96cc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f96cad9d700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0c92e0 0x7f96cc0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f96cad9d700 1 -- 172.21.15.104:0/995970185 >> v1:172.21.15.114:6789/0 conn(0x7f96cc158310 legacy=0x7f96cc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f96cad9d700 1 -- 172.21.15.104:0/995970185 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0cc2f0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f51ec9eb700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 0x7f51e815fb30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f51dc015000 tx=0x7f51dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f96cad9d700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96c00088f0 tx=0x7f96c000d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6362465c05cec8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f7440e2d700 1 -- 172.21.15.104:0/1075965055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 msgr2=0x7f743c06e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.705+0000 7f7440e2d700 1 --2- 172.21.15.104:0/1075965055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c06e780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f742c0096c0 tx=0x7f742c009a80 comp rx=0 tx=0).stop 2023-06-10T18:44:34.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7faf84d50700 1 -- 172.21.15.104:0/251062829 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c004390 con 0x7faf80071130 2023-06-10T18:44:34.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7faf84d50700 1 -- 172.21.15.104:0/251062829 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c0036a0 con 0x7faf80071130 2023-06-10T18:44:34.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7f7440e2d700 1 -- 172.21.15.104:0/1075965055 shutdown_connections 2023-06-10T18:44:34.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7f7440e2d700 1 --2- 172.21.15.104:0/1075965055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c06e780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7f7440e2d700 1 -- 172.21.15.104:0/1075965055 >> 172.21.15.104:0/1075965055 conn(0x7f743c0b0090 msgr2=0x7f743c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7f7440e2d700 1 -- 172.21.15.104:0/1075965055 shutdown_connections 2023-06-10T18:44:34.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.706+0000 7f7440e2d700 1 -- 172.21.15.104:0/1075965055 wait complete. 2023-06-10T18:44:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430005960 con 0x7f9424003a00 2023-06-10T18:44:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430005ac0 con 0x7f9424003a00 2023-06-10T18:44:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7f943959a700 1 -- 172.21.15.104:0/974107981 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430005e30 con 0x7f9424003a00 2023-06-10T18:44:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd43c055bc0 con 0x7fd44c071420 2023-06-10T18:44:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7faf87fb6700 1 -- 172.21.15.104:0/251062829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 msgr2=0x7faf800ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7faf87fb6700 1 --2- 172.21.15.104:0/251062829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0098a0 tx=0x7faf7c009bb0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7faf87fb6700 1 -- 172.21.15.104:0/251062829 shutdown_connections 2023-06-10T18:44:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7faf87fb6700 1 --2- 172.21.15.104:0/251062829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf80071130 0x7faf800ce780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7faf87fb6700 1 -- 172.21.15.104:0/251062829 >> 172.21.15.104:0/251062829 conn(0x7faf800b1610 msgr2=0x7faf800ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7f7440e2d700 1 Processor -- start 2023-06-10T18:44:34.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.707+0000 7f7440e2d700 1 -- start start 2023-06-10T18:44:34.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f0ba9a32700 1 Processor -- start 2023-06-10T18:44:34.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f96c959a700 1 -- 172.21.15.104:0/995970185 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0010070 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f96c959a700 1 -- 172.21.15.104:0/995970185 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c000da80 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f96c959a700 1 -- 172.21.15.104:0/995970185 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c000de10 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 -- 172.21.15.104:0/974107981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9424003a00 msgr2=0x7f9424003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 --2- 172.21.15.104:0/974107981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9424003a00 0x7f9424003e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9430001eb0 tx=0x7f9424005880 comp rx=0 tx=0).stop 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f943ad9d700 1 -- 172.21.15.104:0/974107981 reap_dead start 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 -- 172.21.15.104:0/974107981 shutdown_connections 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 -- 172.21.15.104:0/974107981 >> 172.21.15.104:0/974107981 conn(0x7f943c06ab00 msgr2=0x7f943c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 -- 172.21.15.104:0/974107981 shutdown_connections 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f9440f7e700 1 -- 172.21.15.104:0/974107981 wait complete. 2023-06-10T18:44:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7faf87fb6700 1 -- 172.21.15.104:0/251062829 shutdown_connections 2023-06-10T18:44:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c089510 0x7f743c0c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f743c16d0d0 0x7f743c16b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c0b3190 con 0x7f743c0cea80 2023-06-10T18:44:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c0b3470 con 0x7f743c089510 2023-06-10T18:44:34.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f7440e2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743c0b3750 con 0x7f743c16d0d0 2023-06-10T18:44:34.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f96d0e09700 1 -- 172.21.15.104:0/995970185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 msgr2=0x7f96cc0cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f96d0e09700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96c00088f0 tx=0x7f96c000d330 comp rx=0 tx=0).stop 2023-06-10T18:44:34.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54938/0 (socket says 172.21.15.104:54938) 2023-06-10T18:44:34.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 -- 172.21.15.104:0/3789523510 learned_addr learned my addr 172.21.15.104:0/3789523510 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.708+0000 7f0ba9a32700 1 -- start start 2023-06-10T18:44:34.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba9a32700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0ba40aeaa0 0x7f0ba40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba9a32700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40c9e60 con 0x7f0ba40b1080 2023-06-10T18:44:34.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba9a32700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40cdaa0 con 0x7f0ba40cd680 2023-06-10T18:44:34.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba9a32700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba40c9bc0 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 -- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f743c16d0d0 msgr2=0x7f743c16b690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-10T18:44:34.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f743c16d0d0 0x7f743c16b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 -- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c089510 msgr2=0x7f743c0c4510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-10T18:44:34.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c089510 0x7f743c0c4510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 -- 172.21.15.104:0/3789523510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c00d040 con 0x7f743c0cea80 2023-06-10T18:44:34.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f7439d9b700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f743000c780 tx=0x7f743000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba37fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0ba40cd680 0x7f0ba406dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58614/0 (socket says 172.21.15.104:58614) 2023-06-10T18:44:34.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba37fe700 1 -- 172.21.15.104:0/2742325296 learned_addr learned my addr 172.21.15.104:0/2742325296 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7f0ba27fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0ba40b1080 0x7f0ba40ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52770/0 (socket says 172.21.15.104:52770) 2023-06-10T18:44:34.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430004c80 con 0x7f743c0cea80 2023-06-10T18:44:34.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/2742325296 >> v2:172.21.15.104:3300/0 conn(0x7f0ba40aeaa0 0x7f0ba40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f0ba2ffd700 1 -- 172.21.15.104:0/2742325296 >> v1:172.21.15.114:6789/0 conn(0x7f0ba40cd680 legacy=0x7f0ba406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f0ba2ffd700 1 -- 172.21.15.104:0/2742325296 >> v1:172.21.15.104:6789/0 conn(0x7f0ba40b1080 legacy=0x7f0ba40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f0ba2ffd700 1 -- 172.21.15.104:0/2742325296 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40aee70 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f9440f7e700 1 Processor -- start 2023-06-10T18:44:34.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.709+0000 7faf87fb6700 1 -- 172.21.15.104:0/251062829 wait complete. 2023-06-10T18:44:34.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7faf87fb6700 1 Processor -- start 2023-06-10T18:44:34.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f9440f7e700 1 -- start start 2023-06-10T18:44:34.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7faf87fb6700 1 -- start start 2023-06-10T18:44:34.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7430004e00 con 0x7f743c0cea80 2023-06-10T18:44:34.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.710+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430007310 con 0x7f743c0cea80 2023-06-10T18:44:34.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743c0b3a30 con 0x7f743c0cea80 2023-06-10T18:44:34.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743c0c4a50 con 0x7f743c0cea80 2023-06-10T18:44:34.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 0x7f943c15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 0x7f943c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c162a10 0x7f943c162e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c0b1c30 con 0x7f943c0c3c70 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c1633d0 con 0x7f9424003a00 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9440f7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c15d540 con 0x7f943c162a10 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.711+0000 7f9439d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 0x7f943c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f9439d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 0x7f943c15c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54948/0 (socket says 172.21.15.104:54948) 2023-06-10T18:44:34.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f9439d9b700 1 -- 172.21.15.104:0/2749219488 learned_addr learned my addr 172.21.15.104:0/2749219488 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 0x7f943c15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf80071130 0x7faf801fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 0x7faf80158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 0x7faf80158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf800b20a0 con 0x7faf80071130 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf801fef10 con 0x7faf8015c040 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf87fb6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf80203390 con 0x7faf800ae280 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf86553700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 0x7faf80158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf86553700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 0x7faf80158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42456/0 (socket says 172.21.15.104:42456) 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf86553700 1 -- 172.21.15.104:0/2093522418 learned_addr learned my addr 172.21.15.104:0/2093522418 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 0x7faf80158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 -- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c162a10 msgr2=0x7f943c162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c162a10 0x7f943c162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 -- 172.21.15.104:0/2749219488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 msgr2=0x7f943c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 0x7f943c15c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 -- 172.21.15.104:0/2749219488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c15cca0 con 0x7f9424003a00 2023-06-10T18:44:34.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f943a59c700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 0x7f943c15ba60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9430014000 tx=0x7f9430001a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743c0b3f50 con 0x7f743c0cea80 2023-06-10T18:44:34.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 -- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 msgr2=0x7faf80158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 0x7faf80158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 -- 172.21.15.104:0/2093522418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf80071130 msgr2=0x7faf801fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf80071130 0x7faf801fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7faf85551700 1 -- 172.21.15.104:0/2093522418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c009550 con 0x7faf800ae280 2023-06-10T18:44:34.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430001c80 con 0x7f9424003a00 2023-06-10T18:44:34.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f96d0e09700 1 -- 172.21.15.104:0/995970185 shutdown_connections 2023-06-10T18:44:34.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f96d0e09700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc0cd0b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f96d0e09700 1 --2- 172.21.15.104:0/995970185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0c92e0 0x7f96cc0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.712+0000 7f96d0e09700 1 -- 172.21.15.104:0/995970185 >> 172.21.15.104:0/995970185 conn(0x7f96cc0b0090 msgr2=0x7f96cc0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94300072d0 con 0x7f9424003a00 2023-06-10T18:44:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f9440f7e700 1 -- 172.21.15.104:0/2749219488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c15cf80 con 0x7f9424003a00 2023-06-10T18:44:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430007640 con 0x7f9424003a00 2023-06-10T18:44:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc003e20 con 0x7f51e80ca8b0 2023-06-10T18:44:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc011630 con 0x7f51e80ca8b0 2023-06-10T18:44:34.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf85551700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 0x7faf80158740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf7c015000 tx=0x7faf7c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0119c0 con 0x7f51e80ca8b0 2023-06-10T18:44:34.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f9440f7e700 1 -- 172.21.15.104:0/2749219488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c15bfa0 con 0x7f9424003a00 2023-06-10T18:44:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f96d0e09700 1 -- 172.21.15.104:0/995970185 shutdown_connections 2023-06-10T18:44:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7f96d0e09700 1 -- 172.21.15.104:0/995970185 wait complete. 2023-06-10T18:44:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c011630 con 0x7faf800ae280 2023-06-10T18:44:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c011790 con 0x7faf800ae280 2023-06-10T18:44:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c011b00 con 0x7faf800ae280 2023-06-10T18:44:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf800c5fe0 con 0x7faf800ae280 2023-06-10T18:44:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.713+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf800c65a0 con 0x7faf800ae280 2023-06-10T18:44:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.714+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f943001e430 con 0x7f9424003a00 2023-06-10T18:44:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.714+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e80c6f00 con 0x7f51e80ca8b0 2023-06-10T18:44:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.714+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e81ff100 con 0x7f51e80ca8b0 2023-06-10T18:44:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.714+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7430007470 con 0x7f743c0cea80 2023-06-10T18:44:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f96d0e09700 1 Processor -- start 2023-06-10T18:44:34.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f96d0e09700 1 -- start start 2023-06-10T18:44:34.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf7c011c60 con 0x7faf800ae280 2023-06-10T18:44:34.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f942affd700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 0x7f941c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f943008a600 con 0x7f9424003a00 2023-06-10T18:44:34.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f742affd700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 0x7f741c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f74300899d0 con 0x7f743c0cea80 2023-06-10T18:44:34.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7faf767fc700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 0x7faf6c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.715+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7faf7c08aba0 con 0x7faf800ae280 2023-06-10T18:44:34.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f9439d9b700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 0x7f941c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0c92e0 0x7f96cc0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f743a59c700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 0x7f741c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc158310 0x7f96cc071a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc1586e0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0c3940 con 0x7f96cc0c92e0 2023-06-10T18:44:34.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96d0e09700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0c2d30 con 0x7f96cc158310 2023-06-10T18:44:34.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf680052f0 con 0x7faf800ae280 2023-06-10T18:44:34.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7faf85d52700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 0x7faf6c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c15c130 con 0x7f9424003a00 2023-06-10T18:44:34.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e81ff320 con 0x7f51e80ca8b0 2023-06-10T18:44:34.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96c9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96c9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54964/0 (socket says 172.21.15.104:54964) 2023-06-10T18:44:34.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96c9d9b700 1 -- 172.21.15.104:0/4169989398 learned_addr learned my addr 172.21.15.104:0/4169989398 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96cad9d700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc158310 0x7f96cc071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.716+0000 7f96ca59c700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0c92e0 0x7f96cc0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 -- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc158310 msgr2=0x7f96cc071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc158310 0x7f96cc071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 -- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0c92e0 msgr2=0x7f96cc0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0c92e0 0x7f96cc0b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 -- 172.21.15.104:0/4169989398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00085d0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f96c9d9b700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96c4007e40 tx=0x7f96c400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f743a59c700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 0x7f741c06d260 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f742c000c00 tx=0x7f742c008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7f9439d9b700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 0x7f941c06d190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f943c0c4680 tx=0x7f942c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.717+0000 7faf85d52700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 0x7faf6c06d2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faf70003df0 tx=0x7faf70007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.718+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/2742325296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40cd140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009820 tx=0x7f0b98009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b00ebc24713197 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.721+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7430054800 con 0x7f743c0cea80 2023-06-10T18:44:34.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.721+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf7c055990 con 0x7faf800ae280 2023-06-10T18:44:34.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9430055500 con 0x7f9424003a00 2023-06-10T18:44:34.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c400ed30 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c400ee90 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4014660 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f0ba1ffb700 1 -- 172.21.15.104:0/2742325296 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98005e30 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f0ba1ffb700 1 -- 172.21.15.104:0/2742325296 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98006030 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.722+0000 7f0ba1ffb700 1 -- 172.21.15.104:0/2742325296 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b980063e0 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.723+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc0c3010 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.723+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc0c35d0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.724+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc0c3ab0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.726+0000 7f0ba9a32700 1 -- 172.21.15.104:0/2742325296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40aeaa0 msgr2=0x7f0ba40cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.726+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/2742325296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40cd140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009820 tx=0x7f0b98009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.726+0000 7f0ba9a32700 1 -- 172.21.15.104:0/2742325296 shutdown_connections 2023-06-10T18:44:34.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.726+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/2742325296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40cd140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.726+0000 7f0ba9a32700 1 -- 172.21.15.104:0/2742325296 >> 172.21.15.104:0/2742325296 conn(0x7f0ba406b8a0 msgr2=0x7f0ba40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.727+0000 7f6fb8c2a700 1 Processor -- start 2023-06-10T18:44:34.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.727+0000 7f6fb8c2a700 1 -- start start 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb8c2a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6fb4070130 0x7f6fb415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb8c2a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb415eb50 con 0x7f6fb40cb660 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb8c2a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb4071bb0 con 0x7f6fb415a240 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb8c2a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb4071e30 con 0x7f6fb4070130 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb259c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6fb415a240 0x7f6fb40caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58644/0 (socket says 172.21.15.104:58644) 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb259c700 1 -- 172.21.15.104:0/473772860 learned_addr learned my addr 172.21.15.104:0/473772860 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:34.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342659614 0 0) 0x7f6fb4071bb0 con 0x7f6fb415a240 2023-06-10T18:44:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.728+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f9c0035b0 con 0x7f6fb415a240 2023-06-10T18:44:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.723+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f51dc01b440 con 0x7f51e80ca8b0 2023-06-10T18:44:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.723+0000 7f51e57fa700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 0x7f51cc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.723+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f51dc08afa0 con 0x7f51e80ca8b0 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145696597 0 0) 0x7f6f9c0035b0 con 0x7f6fb415a240 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 >> v2:172.21.15.104:3300/0 conn(0x7f6fb4070130 msgr2=0x7f6fb415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 --2- 172.21.15.104:0/473772860 >> v2:172.21.15.104:3300/0 conn(0x7f6fb4070130 0x7f6fb415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 >> v1:172.21.15.104:6789/0 conn(0x7f6fb40cb660 legacy=0x7f6fb40c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb4072110 con 0x7f6fb415a240 2023-06-10T18:44:34.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6fa8001670 con 0x7f6fb415a240 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 >> v1:172.21.15.114:6789/0 conn(0x7f6fb415a240 legacy=0x7f6fb40caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 0x7f6f9c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c004850 0x7f6f9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9c005300 0x7f6f9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb4071e30 con 0x7f6f9c005300 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c00c230 con 0x7f6f9c004850 2023-06-10T18:44:34.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c00c3f0 con 0x7f6f9c003a50 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 0x7f6f9c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb1d9b700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c004850 0x7f6f9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 -- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c004850 msgr2=0x7f6f9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c004850 0x7f6f9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 -- 172.21.15.104:0/473772860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9c005300 msgr2=0x7f6f9c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9c005300 0x7f6f9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb259c700 1 -- 172.21.15.104:0/473772860 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c0035b0 con 0x7f6f9c003a50 2023-06-10T18:44:34.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.729+0000 7f6fb2d9d700 1 -- 172.21.15.104:0/473772860 reap_dead start 2023-06-10T18:44:34.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f6fb1d9b700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c004850 0x7f6f9c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:34.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f6fb259c700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 0x7f6f9c003e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c210 tx=0x7f6f9c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b27cc7a57247a7a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8007b70 con 0x7f6f9c003a50 2023-06-10T18:44:34.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa800cdb0 con 0x7f6f9c003a50 2023-06-10T18:44:34.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f6fb159a700 1 -- 172.21.15.104:0/473772860 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8005ba0 con 0x7f6f9c003a50 2023-06-10T18:44:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.730+0000 7f51e7fff700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 0x7f51cc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f51e7fff700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 0x7f51cc06d1e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f51e81635c0 tx=0x7f51d4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 msgr2=0x7f6f9c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 0x7f6f9c003e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800c210 tx=0x7f6f9c005880 comp rx=0 tx=0).stop 2023-06-10T18:44:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/473772860 shutdown_connections 2023-06-10T18:44:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/473772860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f9c003a50 0x7f6f9c003e40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/473772860 >> 172.21.15.104:0/473772860 conn(0x7f6fb40b0090 msgr2=0x7f6fb40b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/473772860 shutdown_connections 2023-06-10T18:44:34.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.733+0000 7f0ba9a32700 1 -- 172.21.15.104:0/2742325296 shutdown_connections 2023-06-10T18:44:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.731+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/473772860 wait complete. 2023-06-10T18:44:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.733+0000 7f6fb8c2a700 1 Processor -- start 2023-06-10T18:44:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.733+0000 7f6fb8c2a700 1 -- start start 2023-06-10T18:44:34.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.734+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f51dc055e40 con 0x7f51e80ca8b0 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c003a50 0x7f6fb415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb414c100 0x7f6fb40c2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb414c580 con 0x7f6fb4148de0 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb4148ab0 con 0x7f6f9c003a50 2023-06-10T18:44:34.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb8c2a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb415b890 con 0x7f6fb414c100 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54984/0 (socket says 172.21.15.104:54984) 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 -- 172.21.15.104:0/2134449362 learned_addr learned my addr 172.21.15.104:0/2134449362 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 -- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb414c100 msgr2=0x7f6fb40c2070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb414c100 0x7f6fb40c2070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 -- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c003a50 msgr2=0x7f6fb415b350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c003a50 0x7f6fb415b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 -- 172.21.15.104:0/2134449362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb415bae0 con 0x7f6fb4148de0 2023-06-10T18:44:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f6fb1d9b700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400f770 tx=0x7f6fa400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96c40147c0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f96baffd700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 0x7f96ac06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.735+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f96c408ad10 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.736+0000 7f96ca59c700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 0x7f96ac06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.736+0000 7f96ca59c700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 0x7f96ac06d260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96bc000f60 tx=0x7f96bc00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.737+0000 7f0ba9a32700 1 -- 172.21.15.104:0/2742325296 wait complete. 2023-06-10T18:44:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.738+0000 7f0ba9a32700 1 Processor -- start 2023-06-10T18:44:34.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.739+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96c40595f0 con 0x7f96cc0ce9e0 2023-06-10T18:44:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.739+0000 7f0ba9a32700 1 -- start start 2023-06-10T18:44:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.738+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40052e0 con 0x7f6fb4148de0 2023-06-10T18:44:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.738+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa4005440 con 0x7f6fb4148de0 2023-06-10T18:44:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.738+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40038c0 con 0x7f6fb4148de0 2023-06-10T18:44:34.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.739+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb415bdc0 con 0x7f6fb4148de0 2023-06-10T18:44:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.739+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb40c3a60 con 0x7f6fb4148de0 2023-06-10T18:44:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 0x7f0ba40ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40c9600 con 0x7f0ba40b1080 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40c9170 con 0x7f0ba40aeaa0 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba9a32700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40c8560 con 0x7f0ba40cd680 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54988/0 (socket says 172.21.15.104:54988) 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba27fc700 1 -- 172.21.15.104:0/916916343 learned_addr learned my addr 172.21.15.104:0/916916343 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:34.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.740+0000 7f0ba37fe700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 0x7f0ba40ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 -- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 msgr2=0x7f0ba40adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 -- 172.21.15.104:0/916916343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 msgr2=0x7f0ba40c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:34.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6fa4003a20 con 0x7f6fb4148de0 2023-06-10T18:44:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 -- 172.21.15.104:0/916916343 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009430 con 0x7f0ba40cd680 2023-06-10T18:44:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba27fc700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:34.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f0ba37fe700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 0x7f0ba40ac230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00cb70 tx=0x7f0b9c00ce80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.741+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb4074050 con 0x7f6fb4148de0 2023-06-10T18:44:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00aaa0 con 0x7f0ba40cd680 2023-06-10T18:44:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f6fa2ffd700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 0x7f6f9406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6fa4089350 con 0x7f6fb4148de0 2023-06-10T18:44:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c00ac00 con 0x7f0ba40cd680 2023-06-10T18:44:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0070d0 con 0x7f0ba40cd680 2023-06-10T18:44:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba40c8840 con 0x7f0ba40cd680 2023-06-10T18:44:34.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.742+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba40c8e00 con 0x7f0ba40cd680 2023-06-10T18:44:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.743+0000 7f6fb259c700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 0x7f6f9406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.743+0000 7f6fb259c700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 0x7f6f9406d1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40cf1c0 tx=0x7f6fa8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.748+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0b9c007230 con 0x7f0ba40cd680 2023-06-10T18:44:34.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.748+0000 7f0b8bfff700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.748+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0b9c089250 con 0x7f0ba40cd680 2023-06-10T18:44:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.749+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.749+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6fa4054170 con 0x7f6fb4148de0 2023-06-10T18:44:34.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.756+0000 7f0ba2ffd700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0b98000c00 tx=0x7f0b98007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:34.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.756+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b840052f0 con 0x7f0ba40cd680 2023-06-10T18:44:34.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:34.761+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0b9c053f70 con 0x7f0ba40cd680 2023-06-10T18:44:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.101+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd42c0055a0 con 0x7fd44c071420 2023-06-10T18:44:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.102+0000 7fd441ffb700 1 -- 172.21.15.104:0/278889128 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd43c02a090 con 0x7fd44c071420 2023-06-10T18:44:35.110 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:35.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 msgr2=0x7fd43006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 0x7fd43006d230 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd44c152ea0 tx=0x7fd448008040 comp rx=0 tx=0).stop 2023-06-10T18:44:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 msgr2=0x7fd44c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00d990 tx=0x7fd43c005020 comp rx=0 tx=0).stop 2023-06-10T18:44:35.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faf680055a0 con 0x7faf800ae280 2023-06-10T18:44:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 shutdown_connections 2023-06-10T18:44:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd43006ad80 0x7fd43006d230 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd44c143520 0x7fd44c1439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4380088e0 0x7fd44c0c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 --2- 172.21.15.104:0/278889128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd44c071420 0x7fd44c071890 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 >> 172.21.15.104:0/278889128 conn(0x7fd44c0b0090 msgr2=0x7fd44c0cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.112+0000 7faf767fc700 1 -- 172.21.15.104:0/2093522418 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf7c028030 con 0x7faf800ae280 2023-06-10T18:44:35.121 INFO:teuthology.orchestra.run.smithi104.stdout:55834574858 2023-06-10T18:44:35.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 shutdown_connections 2023-06-10T18:44:35.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7fd453331700 1 -- 172.21.15.104:0/278889128 wait complete. 2023-06-10T18:44:35.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 msgr2=0x7faf6c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 0x7faf6c06d2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faf70003df0 tx=0x7faf70007a40 comp rx=0 tx=0).stop 2023-06-10T18:44:35.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 msgr2=0x7faf80158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.116+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 0x7faf80158740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf7c015000 tx=0x7faf7c004750 comp rx=0 tx=0).stop 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 shutdown_connections 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf6c06adf0 0x7faf6c06d2a0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faf70003df0 tx=0x7faf70007a40 comp rx=0 tx=0).stop 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8015c040 0x7faf80158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf80071130 0x7faf801fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 --2- 172.21.15.104:0/2093522418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf800ae280 0x7faf80158740 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 >> 172.21.15.104:0/2093522418 conn(0x7faf800b1610 msgr2=0x7faf800afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f943c074050 con 0x7f9424003a00 2023-06-10T18:44:35.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 shutdown_connections 2023-06-10T18:44:35.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.117+0000 7faf87fb6700 1 -- 172.21.15.104:0/2093522418 wait complete. 2023-06-10T18:44:35.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.120+0000 7f942affd700 1 -- 172.21.15.104:0/2749219488 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9430017030 con 0x7f9424003a00 2023-06-10T18:44:35.127 INFO:teuthology.orchestra.run.smithi104.stdout:55834574858 2023-06-10T18:44:35.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.124+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f96cc074050 con 0x7f96cc0ce9e0 2023-06-10T18:44:35.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 msgr2=0x7f941c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 0x7f941c06d190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f943c0c4680 tx=0x7f942c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:35.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 msgr2=0x7f943c15ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 0x7f943c15ba60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9430014000 tx=0x7f9430001a00 comp rx=0 tx=0).stop 2023-06-10T18:44:35.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f96baffd700 1 -- 172.21.15.104:0/4169989398 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96c4059180 con 0x7f96cc0ce9e0 2023-06-10T18:44:35.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.124+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f743c0b3f50 con 0x7f743c0cea80 2023-06-10T18:44:35.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f742affd700 1 -- 172.21.15.104:0/3789523510 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f743c0b3f50 con 0x7f743c0cea80 2023-06-10T18:44:35.132 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:35.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 shutdown_connections 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f941c06ace0 0x7f941c06d190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9424003a00 0x7f943c15ba60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f943c162a10 0x7f943c162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 --2- 172.21.15.104:0/2749219488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f943c0c3c70 0x7f943c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 >> 172.21.15.104:0/2749219488 conn(0x7f943c06ab00 msgr2=0x7f943c0ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 shutdown_connections 2023-06-10T18:44:35.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.125+0000 7f9428ff9700 1 -- 172.21.15.104:0/2749219488 wait complete. 2023-06-10T18:44:35.134 INFO:teuthology.orchestra.run.smithi104.stdout:55834574858 2023-06-10T18:44:35.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.130+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 msgr2=0x7f741c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.130+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 0x7f741c06d260 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f742c000c00 tx=0x7f742c008000 comp rx=0 tx=0).stop 2023-06-10T18:44:35.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.130+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 msgr2=0x7f743c0b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.130+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f743000c780 tx=0x7f743000cb40 comp rx=0 tx=0).stop 2023-06-10T18:44:35.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.1 2023-06-10T18:44:35.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 msgr2=0x7f96ac06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 0x7f96ac06d260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96bc000f60 tx=0x7f96bc00d040 comp rx=0 tx=0).stop 2023-06-10T18:44:35.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 msgr2=0x7f96cc071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96c4007e40 tx=0x7f96c400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 shutdown_connections 2023-06-10T18:44:35.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06adb0 0x7f741c06d260 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c089510 0x7f743c0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f743c16d0d0 0x7f743c16b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 --2- 172.21.15.104:0/3789523510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743c0cea80 0x7f743c0b2820 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.134+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 >> 172.21.15.104:0/3789523510 conn(0x7f743c0b0090 msgr2=0x7f743c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 shutdown_connections 2023-06-10T18:44:35.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f7440e2d700 1 -- 172.21.15.104:0/3789523510 wait complete. 2023-06-10T18:44:35.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 shutdown_connections 2023-06-10T18:44:35.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06adb0 0x7f96ac06d260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0c92e0 0x7f96cc0b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc158310 0x7f96cc071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 --2- 172.21.15.104:0/4169989398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0ce9e0 0x7f96cc071520 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.137+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 >> 172.21.15.104:0/4169989398 conn(0x7f96cc0b0090 msgr2=0x7f96cc0cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.138+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 shutdown_connections 2023-06-10T18:44:35.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.138+0000 7f96d0e09700 1 -- 172.21.15.104:0/4169989398 wait complete. 2023-06-10T18:44:35.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.2 2023-06-10T18:44:35.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.143+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f51e8074050 con 0x7f51e80ca8b0 2023-06-10T18:44:35.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.143+0000 7f51e57fa700 1 -- 172.21.15.104:0/414793449 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51dc059460 con 0x7f51e80ca8b0 2023-06-10T18:44:35.149 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:35.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.4 2023-06-10T18:44:35.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.146+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 msgr2=0x7f51cc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.146+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 0x7f51cc06d1e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f51e81635c0 tx=0x7f51d4009210 comp rx=0 tx=0).stop 2023-06-10T18:44:35.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.146+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 msgr2=0x7f51e815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.146+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 0x7f51e815fb30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f51dc015000 tx=0x7f51dc004750 comp rx=0 tx=0).stop 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 shutdown_connections 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51e8160880 0x7f51e80c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51e80ca8b0 0x7f51e815fb30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51cc06ad30 0x7f51cc06d1e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 --2- 172.21.15.104:0/414793449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51e80cea50 0x7f51e80c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 >> 172.21.15.104:0/414793449 conn(0x7f51e806ab00 msgr2=0x7f51e81637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 shutdown_connections 2023-06-10T18:44:35.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.147+0000 7f51eec4f700 1 -- 172.21.15.104:0/414793449 wait complete. 2023-06-10T18:44:35.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.150+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6fb4159a60 con 0x7f6fb4148de0 2023-06-10T18:44:35.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.150+0000 7f6fa2ffd700 1 -- 172.21.15.104:0/2134449362 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6fa4057790 con 0x7f6fb4148de0 2023-06-10T18:44:35.156 INFO:teuthology.orchestra.run.smithi104.stdout:55834574858 2023-06-10T18:44:35.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.152+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 msgr2=0x7f6f9406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.152+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 0x7f6f9406d1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40cf1c0 tx=0x7f6fa8007000 comp rx=0 tx=0).stop 2023-06-10T18:44:35.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.152+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 msgr2=0x7f6fb414bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.152+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400f770 tx=0x7f6fa400fb30 comp rx=0 tx=0).stop 2023-06-10T18:44:35.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 shutdown_connections 2023-06-10T18:44:35.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f9406ad30 0x7f6f9406d1e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9c003a50 0x7f6fb415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb414c100 0x7f6fb40c2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 --2- 172.21.15.104:0/2134449362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6fb4148de0 0x7f6fb414bbc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 >> 172.21.15.104:0/2134449362 conn(0x7f6fb40b0090 msgr2=0x7f6fb40af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 shutdown_connections 2023-06-10T18:44:35.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.153+0000 7f6fb8c2a700 1 -- 172.21.15.104:0/2134449362 wait complete. 2023-06-10T18:44:35.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.6 2023-06-10T18:44:35.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.160+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0b840055a0 con 0x7f0ba40cd680 2023-06-10T18:44:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.161+0000 7f0b8bfff700 1 -- 172.21.15.104:0/916916343 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0b9c018070 con 0x7f0ba40cd680 2023-06-10T18:44:35.166 INFO:teuthology.orchestra.run.smithi104.stdout:55834574858 2023-06-10T18:44:35.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.3 2023-06-10T18:44:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 msgr2=0x7f0b9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0b98000c00 tx=0x7f0b98007000 comp rx=0 tx=0).stop 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 msgr2=0x7f0ba40ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 0x7f0ba40ac230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00cb70 tx=0x7f0b9c00ce80 comp rx=0 tx=0).stop 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 shutdown_connections 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba40aeaa0 0x7f0ba40adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0ba40cd680 0x7f0ba40ac230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 --2- 172.21.15.104:0/916916343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba40b1080 0x7f0ba40c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 >> 172.21.15.104:0/916916343 conn(0x7f0ba406b8a0 msgr2=0x7f0ba40cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 shutdown_connections 2023-06-10T18:44:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:35.164+0000 7f0ba9a32700 1 -- 172.21.15.104:0/916916343 wait complete. 2023-06-10T18:44:35.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2023-06-10T18:44:35.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.5 2023-06-10T18:44:35.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.0 2023-06-10T18:44:36.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-06-10T18:44:36.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-06-10T18:44:36.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-06-10T18:44:36.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-06-10T18:44:36.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-06-10T18:44:36.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-06-10T18:44:36.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-06-10T18:44:36.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-06-10T18:44:36.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.330+0000 7f8f47319700 1 Processor -- start 2023-06-10T18:44:36.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.330+0000 7f8f47319700 1 -- start start 2023-06-10T18:44:36.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.331+0000 7f8f47319700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8f401611b0 0x7f8f40161580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.331+0000 7f8f47319700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 0x7f8f400c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.331+0000 7f8f47319700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f40161ac0 con 0x7f8f400c18d0 2023-06-10T18:44:36.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.331+0000 7f8f47319700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400c93e0 con 0x7f8f401611b0 2023-06-10T18:44:36.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.331+0000 7f8f47319700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400c1d50 con 0x7f8f400c8bd0 2023-06-10T18:44:36.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.332+0000 7f8f458b6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8f400c18d0 0x7f8f401640b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52828/0 (socket says 172.21.15.104:52828) 2023-06-10T18:44:36.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.332+0000 7f8f458b6700 1 -- 172.21.15.104:0/2078914923 learned_addr learned my addr 172.21.15.104:0/2078914923 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:36.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.332+0000 7f8f448b4700 1 --2- 172.21.15.104:0/2078914923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 0x7f8f400c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.333+0000 7f8f450b5700 1 --2- 172.21.15.104:0/2078914923 >> v2:172.21.15.104:3300/0 conn(0x7f8f401611b0 0x7f8f40161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.334+0000 7f8f448b4700 1 -- 172.21.15.104:0/2078914923 >> v2:172.21.15.104:3300/0 conn(0x7f8f401611b0 msgr2=0x7f8f40161580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.334+0000 7f8f448b4700 1 --2- 172.21.15.104:0/2078914923 >> v2:172.21.15.104:3300/0 conn(0x7f8f401611b0 0x7f8f40161580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.334+0000 7f8f448b4700 1 -- 172.21.15.104:0/2078914923 >> v1:172.21.15.104:6789/0 conn(0x7f8f400c18d0 legacy=0x7f8f401640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.334+0000 7f8f448b4700 1 -- 172.21.15.104:0/2078914923 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400cb410 con 0x7f8f400c8bd0 2023-06-10T18:44:36.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.335+0000 7f8f448b4700 1 --2- 172.21.15.104:0/2078914923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 0x7f8f400c1390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009840 tx=0x7f8f30009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68032163ab5c9d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.335+0000 7f8f37fff700 1 -- 172.21.15.104:0/2078914923 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30011830 con 0x7f8f400c8bd0 2023-06-10T18:44:36.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.335+0000 7f8f37fff700 1 -- 172.21.15.104:0/2078914923 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30011990 con 0x7f8f400c8bd0 2023-06-10T18:44:36.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.335+0000 7f8f37fff700 1 -- 172.21.15.104:0/2078914923 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30011d20 con 0x7f8f400c8bd0 2023-06-10T18:44:36.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 -- 172.21.15.104:0/2078914923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 msgr2=0x7f8f400c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 --2- 172.21.15.104:0/2078914923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 0x7f8f400c1390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009840 tx=0x7f8f30009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:36.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 -- 172.21.15.104:0/2078914923 shutdown_connections 2023-06-10T18:44:36.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 --2- 172.21.15.104:0/2078914923 >> v2:172.21.15.104:3300/0 conn(0x7f8f401611b0 0x7f8f40161580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 --2- 172.21.15.104:0/2078914923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f400c8bd0 0x7f8f400c1390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.336+0000 7f8f47319700 1 -- 172.21.15.104:0/2078914923 >> 172.21.15.104:0/2078914923 conn(0x7f8f400b0090 msgr2=0x7f8f400b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.337+0000 7f8f47319700 1 -- 172.21.15.104:0/2078914923 shutdown_connections 2023-06-10T18:44:36.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.337+0000 7f8f47319700 1 -- 172.21.15.104:0/2078914923 wait complete. 2023-06-10T18:44:36.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.339+0000 7f8f47319700 1 Processor -- start 2023-06-10T18:44:36.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.339+0000 7f8f47319700 1 -- start start 2023-06-10T18:44:36.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f400c8bd0 0x7f8f400c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f401611b0 0x7f8f400b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400c76b0 con 0x7f8f400c18d0 2023-06-10T18:44:36.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400b1450 con 0x7f8f400c8bd0 2023-06-10T18:44:36.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.340+0000 7f8f47319700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400c78e0 con 0x7f8f401611b0 2023-06-10T18:44:36.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.341+0000 7f8f450b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.341+0000 7f8f450b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55004/0 (socket says 172.21.15.104:55004) 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.341+0000 7f8f450b5700 1 -- 172.21.15.104:0/4027386242 learned_addr learned my addr 172.21.15.104:0/4027386242 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 -- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f401611b0 msgr2=0x7f8f400b37b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f401611b0 0x7f8f400b37b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 -- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f400c8bd0 msgr2=0x7f8f400c7170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f400c8bd0 0x7f8f400c7170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 -- 172.21.15.104:0/4027386242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300094a0 con 0x7f8f400c18d0 2023-06-10T18:44:36.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.342+0000 7f8f450b5700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8f38007e40 tx=0x7f8f3800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.344+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800eea0 con 0x7f8f400c18d0 2023-06-10T18:44:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.344+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38014420 con 0x7f8f400c18d0 2023-06-10T18:44:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.344+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f380147d0 con 0x7f8f400c18d0 2023-06-10T18:44:36.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.344+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f400b16f0 con 0x7f8f400c18d0 2023-06-10T18:44:36.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.344+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400c7bc0 con 0x7f8f400c18d0 2023-06-10T18:44:36.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.350+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8f38014930 con 0x7f8f400c18d0 2023-06-10T18:44:36.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.350+0000 7f8f35ffb700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 0x7f8f2406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.350+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8f38089ec0 con 0x7f8f400c18d0 2023-06-10T18:44:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.351+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f200052f0 con 0x7f8f400c18d0 2023-06-10T18:44:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.352+0000 7f8f448b4700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 0x7f8f2406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.360+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8f38054cf0 con 0x7f8f400c18d0 2023-06-10T18:44:36.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.365+0000 7f8f448b4700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 0x7f8f2406d260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f40071f00 tx=0x7f8f30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.398+0000 7fa31b775700 1 Processor -- start 2023-06-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.399+0000 7fa31b775700 1 -- start start 2023-06-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa31b775700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3140c92e0 0x7fa3140ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa31b775700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140ccc20 con 0x7fa3140ce9e0 2023-06-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa31b775700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140cee20 con 0x7fa314158310 2023-06-10T18:44:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa31b775700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31415cc50 con 0x7fa3140c92e0 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa319d12700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa3140ce9e0 0x7fa3140cd0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52842/0 (socket says 172.21.15.104:52842) 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa319d12700 1 -- 172.21.15.104:0/2538644855 learned_addr learned my addr 172.21.15.104:0/2538644855 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.400+0000 7fa318d10700 1 --2- 172.21.15.104:0/2538644855 >> v2:172.21.15.104:3300/0 conn(0x7fa3140c92e0 0x7fa3140ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa318d10700 1 -- 172.21.15.104:0/2538644855 >> v1:172.21.15.114:6789/0 conn(0x7fa314158310 legacy=0x7fa3140c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa318d10700 1 -- 172.21.15.104:0/2538644855 >> v1:172.21.15.104:6789/0 conn(0x7fa3140ce9e0 legacy=0x7fa3140cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa318d10700 1 -- 172.21.15.104:0/2538644855 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140cc2f0 con 0x7fa3140c92e0 2023-06-10T18:44:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa318d10700 1 --2- 172.21.15.104:0/2538644855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140c92e0 0x7fa3140ce4a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00ad90 tx=0x7fa30c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4861f59b0d398ea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa30bfff700 1 -- 172.21.15.104:0/2538644855 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00cad0 con 0x7fa3140c92e0 2023-06-10T18:44:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa30bfff700 1 -- 172.21.15.104:0/2538644855 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00cc30 con 0x7fa3140c92e0 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.401+0000 7fa30bfff700 1 -- 172.21.15.104:0/2538644855 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c003a20 con 0x7fa3140c92e0 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.402+0000 7fa31b775700 1 -- 172.21.15.104:0/2538644855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140c92e0 msgr2=0x7fa3140ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.402+0000 7fa31b775700 1 --2- 172.21.15.104:0/2538644855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140c92e0 0x7fa3140ce4a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00ad90 tx=0x7fa30c00c330 comp rx=0 tx=0).stop 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.403+0000 7fa31b775700 1 -- 172.21.15.104:0/2538644855 shutdown_connections 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.403+0000 7fa31b775700 1 --2- 172.21.15.104:0/2538644855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140c92e0 0x7fa3140ce4a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.403+0000 7fa31b775700 1 -- 172.21.15.104:0/2538644855 >> 172.21.15.104:0/2538644855 conn(0x7fa3140b0090 msgr2=0x7fa3140b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.404+0000 7fa31b775700 1 -- 172.21.15.104:0/2538644855 shutdown_connections 2023-06-10T18:44:36.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.404+0000 7fa31b775700 1 -- 172.21.15.104:0/2538644855 wait complete. 2023-06-10T18:44:36.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.405+0000 7fd76ab66700 1 Processor -- start 2023-06-10T18:44:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.405+0000 7f30cee8d700 1 Processor -- start 2023-06-10T18:44:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fa31b775700 1 Processor -- start 2023-06-10T18:44:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.405+0000 7fd76ab66700 1 -- start start 2023-06-10T18:44:36.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fd76ab66700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd764071240 0x7fd764158f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fd76ab66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640cc6e0 0x7fd76415cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fd76ab66700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640cd720 con 0x7fd76406f3b0 2023-06-10T18:44:36.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fd76ab66700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640cda00 con 0x7fd764071240 2023-06-10T18:44:36.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fd76ab66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7640cdce0 con 0x7fd7640cc6e0 2023-06-10T18:44:36.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.405+0000 7f3c83075700 1 Processor -- start 2023-06-10T18:44:36.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.405+0000 7f3c83075700 1 -- start start 2023-06-10T18:44:36.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3c7c0ce470 0x7f3c7c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c7c0cbb40 0x7f3c7c089510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0ce840 0x7f3c7c0cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0cb6b0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0cb380 con 0x7f3c7c0ce470 2023-06-10T18:44:36.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f3c83075700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0c8c70 con 0x7f3c7c0ce840 2023-06-10T18:44:36.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fd763fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd76406f3b0 0x7fd7640cbfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52848/0 (socket says 172.21.15.104:52848) 2023-06-10T18:44:36.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fd763fff700 1 -- 172.21.15.104:0/1501991268 learned_addr learned my addr 172.21.15.104:0/1501991268 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:36.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f3c7bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c7c0cbb40 0x7f3c7c089510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f3c7bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3c7c0cbb40 0x7f3c7c089510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55018/0 (socket says 172.21.15.104:55018) 2023-06-10T18:44:36.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f3c7bfff700 1 -- 172.21.15.104:0/3534498555 learned_addr learned my addr 172.21.15.104:0/3534498555 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223166362 0 0) 0x7fd7640cd720 con 0x7fd76406f3b0 2023-06-10T18:44:36.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c0035b0 con 0x7fd76406f3b0 2023-06-10T18:44:36.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f3c81612700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0ce840 0x7f3c7c0cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7f30cee8d700 1 -- start start 2023-06-10T18:44:36.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f30cee8d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c806b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f30cee8d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30c80cca60 0x7f30c8164300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f30cee8d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c806b900 con 0x7f30c80cace0 2023-06-10T18:44:36.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f30cee8d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80cb0b0 con 0x7f30c80cca60 2023-06-10T18:44:36.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7f30cee8d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8161650 con 0x7f30c8071130 2023-06-10T18:44:36.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.406+0000 7fa31b775700 1 -- start start 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3140c92e0 0x7fa314156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3141536c0 0x7fa3140b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa314156b70 con 0x7fa3140ce9e0 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3141f0740 con 0x7fa3141536c0 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.407+0000 7fa31b775700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa314155660 con 0x7fa3140c92e0 2023-06-10T18:44:36.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55026/0 (socket says 172.21.15.104:55026) 2023-06-10T18:44:36.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 -- 172.21.15.104:0/2088134687 learned_addr learned my addr 172.21.15.104:0/2088134687 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa319d12700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3141536c0 0x7fa3140b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa319511700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3140c92e0 0x7fa314156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30c80cca60 0x7f30c8164300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f30c80cca60 0x7f30c8164300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55036/0 (socket says 172.21.15.104:55036) 2023-06-10T18:44:36.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 -- 172.21.15.104:0/4037555141 learned_addr learned my addr 172.21.15.104:0/4037555141 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f68ee897700 1 Processor -- start 2023-06-10T18:44:36.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 -- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3140c92e0 msgr2=0x7fa314156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3140c92e0 0x7fa314156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 -- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3141536c0 msgr2=0x7fa3140b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3141536c0 0x7fa3140b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 -- 172.21.15.104:0/2088134687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c00aa70 con 0x7fa3140ce9e0 2023-06-10T18:44:36.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f68ee897700 1 -- start start 2023-06-10T18:44:36.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 -- 172.21.15.104:0/4037555141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 msgr2=0x7f30c806b3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:36.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 --2- 172.21.15.104:0/4037555141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c806b3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 -- 172.21.15.104:0/4037555141 >> v1:172.21.15.114:6789/0 conn(0x7f30c80cace0 legacy=0x7f30c8070a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:36.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7f30cd42a700 1 -- 172.21.15.104:0/4037555141 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80ce460 con 0x7f30c80cca60 2023-06-10T18:44:36.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fa318d10700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3140cf1f0 tx=0x7fa30c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673506050 0 0) 0x7fd74c0035b0 con 0x7fd76406f3b0 2023-06-10T18:44:36.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640cc6e0 msgr2=0x7fd76415cb80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:36.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640cc6e0 0x7fd76415cb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 >> v2:172.21.15.114:3300/0 conn(0x7fd764071240 msgr2=0x7fd764158f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:36.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 --2- 172.21.15.104:0/1501991268 >> v2:172.21.15.114:3300/0 conn(0x7fd764071240 0x7fd764158f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640cdfc0 con 0x7fd76406f3b0 2023-06-10T18:44:36.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd75c0019f0 con 0x7fd76406f3b0 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.408+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 >> v1:172.21.15.104:6789/0 conn(0x7fd76406f3b0 legacy=0x7fd7640cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd74c003a00 0x7fd74c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd74c004850 0x7fd74c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd74c005320 0x7fd74c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7640cda00 con 0x7fd74c003a00 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7640cdce0 con 0x7fd74c005320 2023-06-10T18:44:36.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c00c2b0 con 0x7fd74c004850 2023-06-10T18:44:36.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f30cd42a700 1 --2- 172.21.15.104:0/4037555141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c8164300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0098f0 tx=0x7f30bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d499b4569bb9cf11 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c80e11700 1 --2- 172.21.15.104:0/3534498555 >> v2:172.21.15.114:3300/0 conn(0x7f3c7c0ce470 0x7f3c7c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd768902700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd74c003a00 0x7fd74c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c7bfff700 1 -- 172.21.15.104:0/3534498555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0ce840 msgr2=0x7f3c7c0cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c7bfff700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0ce840 0x7f3c7c0cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c7bfff700 1 -- 172.21.15.104:0/3534498555 >> v2:172.21.15.114:3300/0 conn(0x7f3c7c0ce470 msgr2=0x7f3c7c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c7bfff700 1 --2- 172.21.15.104:0/3534498555 >> v2:172.21.15.114:3300/0 conn(0x7f3c7c0ce470 0x7f3c7c06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f3c7bfff700 1 -- 172.21.15.104:0/3534498555 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0c8f50 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd763fff700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd74c004850 0x7fd74c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7fd769103700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd74c005320 0x7fd74c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f68ee897700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f68e8156650 0x7f68e815d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f68ee897700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68e80c8bd0 0x7f68e80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f68ee897700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e8070b90 con 0x7f68e815fcc0 2023-06-10T18:44:36.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f68ee897700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80cb9d0 con 0x7f68e8156650 2023-06-10T18:44:36.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.409+0000 7f68ee897700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80c9070 con 0x7f68e80c8bd0 2023-06-10T18:44:36.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f30c77fe700 1 -- 172.21.15.104:0/4037555141 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc004230 con 0x7f30c80cca60 2023-06-10T18:44:36.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f30c77fe700 1 -- 172.21.15.104:0/4037555141 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc004390 con 0x7f30c80cca60 2023-06-10T18:44:36.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f30c77fe700 1 -- 172.21.15.104:0/4037555141 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0036a0 con 0x7f30c80cca60 2023-06-10T18:44:36.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7f30cee8d700 1 -- 172.21.15.104:0/4037555141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 msgr2=0x7f30c8164300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7f30cee8d700 1 --2- 172.21.15.104:0/4037555141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c8164300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0098f0 tx=0x7f30bc009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f68e8156650 0x7f68e815d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f68e8156650 0x7f68e815d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55044/0 (socket says 172.21.15.104:55044) 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 -- 172.21.15.104:0/2548940308 learned_addr learned my addr 172.21.15.104:0/2548940308 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 -- 172.21.15.104:0/2548940308 >> v2:172.21.15.114:3300/0 conn(0x7f68e80c8bd0 msgr2=0x7f68e80cb490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 --2- 172.21.15.104:0/2548940308 >> v2:172.21.15.114:3300/0 conn(0x7f68e80c8bd0 0x7f68e80cb490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 -- 172.21.15.104:0/2548940308 >> v1:172.21.15.104:6789/0 conn(0x7f68e815fcc0 legacy=0x7f68e81600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 -- 172.21.15.104:0/2548940308 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e8070260 con 0x7f68e8156650 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f68e7fff700 1 --2- 172.21.15.104:0/2548940308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e8156650 0x7f68e815d890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68e806ff80 tx=0x7f68e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bded3fb14e2f0dd1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f68e6ffd700 1 -- 172.21.15.104:0/2548940308 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000e070 con 0x7f68e8156650 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f68e6ffd700 1 -- 172.21.15.104:0/2548940308 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e0003a20 con 0x7f68e8156650 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f68e6ffd700 1 -- 172.21.15.104:0/2548940308 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0003dd0 con 0x7f68e8156650 2023-06-10T18:44:36.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c004570 con 0x7fa3140ce9e0 2023-06-10T18:44:36.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c020070 con 0x7fa3140ce9e0 2023-06-10T18:44:36.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c004910 con 0x7fa3140ce9e0 2023-06-10T18:44:36.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa314155380 con 0x7fa3140ce9e0 2023-06-10T18:44:36.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa314155e30 con 0x7fa3140ce9e0 2023-06-10T18:44:36.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa314074050 con 0x7fa3140ce9e0 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 -- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd74c005320 msgr2=0x7fd74c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd74c005320 0x7fd74c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 -- 172.21.15.104:0/1501991268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd74c003a00 msgr2=0x7fd74c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd74c003a00 0x7fd74c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 -- 172.21.15.104:0/1501991268 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0035b0 con 0x7fd74c004850 2023-06-10T18:44:36.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7fd763fff700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd74c004850 0x7fd74c004cc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00a7b0 tx=0x7fd74c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae82486e1cbbf9fd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c012600 con 0x7fd74c004850 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c0127a0 con 0x7fd74c004850 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd7637fe700 1 -- 172.21.15.104:0/1501991268 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c012b50 con 0x7fd74c004850 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd76ab66700 1 -- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd74c004850 msgr2=0x7fd74c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd76ab66700 1 --2- 172.21.15.104:0/1501991268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd74c004850 0x7fd74c004cc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00a7b0 tx=0x7fd74c005880 comp rx=0 tx=0).stop 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7fd769103700 1 -- 172.21.15.104:0/1501991268 reap_dead start 2023-06-10T18:44:36.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7fd76ab66700 1 -- 172.21.15.104:0/1501991268 shutdown_connections 2023-06-10T18:44:36.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7fd76ab66700 1 -- 172.21.15.104:0/1501991268 >> 172.21.15.104:0/1501991268 conn(0x7fd76406d240 msgr2=0x7fd76406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.410+0000 7f3c7bfff700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0cbb40 0x7f3c7c089510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0cf170 tx=0x7f3c700095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9266379aa0400f8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7f3c7b7fe700 1 -- 172.21.15.104:0/3534498555 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000c900 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7f3c7b7fe700 1 -- 172.21.15.104:0/3534498555 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7000ca60 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.411+0000 7f3c7b7fe700 1 -- 172.21.15.104:0/3534498555 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000cdf0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 -- 172.21.15.104:0/3534498555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0cbb40 msgr2=0x7f3c7c089510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0cbb40 0x7f3c7c089510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0cf170 tx=0x7f3c700095c0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 -- 172.21.15.104:0/3534498555 shutdown_connections 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0ce840 0x7f3c7c0cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 --2- 172.21.15.104:0/3534498555 >> v2:172.21.15.114:3300/0 conn(0x7f3c7c0ce470 0x7f3c7c06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 --2- 172.21.15.104:0/3534498555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0cbb40 0x7f3c7c089510 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 -- 172.21.15.104:0/3534498555 >> 172.21.15.104:0/3534498555 conn(0x7f3c7c0b0090 msgr2=0x7f3c7c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 -- 172.21.15.104:0/3534498555 shutdown_connections 2023-06-10T18:44:36.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.412+0000 7f3c83075700 1 -- 172.21.15.104:0/3534498555 wait complete. 2023-06-10T18:44:36.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 -- 172.21.15.104:0/2548940308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e8156650 msgr2=0x7f68e815d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 --2- 172.21.15.104:0/2548940308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e8156650 0x7f68e815d890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68e806ff80 tx=0x7f68e000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:36.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 -- 172.21.15.104:0/2548940308 shutdown_connections 2023-06-10T18:44:36.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 --2- 172.21.15.104:0/2548940308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e8156650 0x7f68e815d890 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 --2- 172.21.15.104:0/2548940308 >> v2:172.21.15.114:3300/0 conn(0x7f68e80c8bd0 0x7f68e80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f68ee897700 1 -- 172.21.15.104:0/2548940308 >> 172.21.15.104:0/2548940308 conn(0x7f68e80b0090 msgr2=0x7f68e80b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 -- 172.21.15.104:0/4037555141 shutdown_connections 2023-06-10T18:44:36.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 --2- 172.21.15.104:0/4037555141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c8164300 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 --2- 172.21.15.104:0/4037555141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c806b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 -- 172.21.15.104:0/4037555141 >> 172.21.15.104:0/4037555141 conn(0x7f30c80b1610 msgr2=0x7f30c80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 -- 172.21.15.104:0/4037555141 shutdown_connections 2023-06-10T18:44:36.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7fc1b5a5e700 1 Processor -- start 2023-06-10T18:44:36.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7fc1b5a5e700 1 -- start start 2023-06-10T18:44:36.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f377eac4700 1 Processor -- start 2023-06-10T18:44:36.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f30cee8d700 1 -- 172.21.15.104:0/4037555141 wait complete. 2023-06-10T18:44:36.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f30cee8d700 1 Processor -- start 2023-06-10T18:44:36.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f30cee8d700 1 -- start start 2023-06-10T18:44:36.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f68ee897700 1 -- 172.21.15.104:0/2548940308 shutdown_connections 2023-06-10T18:44:36.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa30c01c450 con 0x7fa3140ce9e0 2023-06-10T18:44:36.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fa309ffb700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 0x7fa2f806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa30c08b210 con 0x7fa3140ce9e0 2023-06-10T18:44:36.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fa319511700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 0x7fa2f806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fd76ab66700 1 -- 172.21.15.104:0/1501991268 shutdown_connections 2023-06-10T18:44:36.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.413+0000 7f377eac4700 1 -- start start 2023-06-10T18:44:36.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f377eac4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f377eac4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3778164620 0x7f37780cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f377eac4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3778164f40 con 0x7f3778160400 2023-06-10T18:44:36.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f377eac4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3778164a70 con 0x7f3778164620 2023-06-10T18:44:36.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f377eac4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37780cdb00 con 0x7f37781607d0 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7f3c83075700 1 Processor -- start 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7f3c83075700 1 -- start start 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c7c0ce470 0x7f3c7c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0ce840 0x7f3c7c087350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c085770 con 0x7f3c7c0ce840 2023-06-10T18:44:36.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0862d0 con 0x7f3c7c0ce470 2023-06-10T18:44:36.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f3c83075700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0865b0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fa319511700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 0x7fa2f806d260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3141fa8e0 tx=0x7fa310009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f3c80e11700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46214/0 (socket says 172.21.15.104:46214) 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 -- 172.21.15.104:0/3691781180 learned_addr learned my addr 172.21.15.104:0/3691781180 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 -- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c7c0ce470 msgr2=0x7f3c7c0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c7c0ce470 0x7f3c7c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 -- 172.21.15.104:0/3691781180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0ce840 msgr2=0x7f3c7c087350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0ce840 0x7f3c7c087350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 -- 172.21.15.104:0/3691781180 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70009de0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7f68ee897700 1 -- 172.21.15.104:0/2548940308 wait complete. 2023-06-10T18:44:36.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f68ee897700 1 Processor -- start 2023-06-10T18:44:36.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c80e11700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9b0 tx=0x7f3c6c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46212/0 (socket says 172.21.15.104:46212) 2023-06-10T18:44:36.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 -- 172.21.15.104:0/3670181417 learned_addr learned my addr 172.21.15.104:0/3670181417 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30c80cace0 0x7f30c814f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c815f490 con 0x7f30c80cca60 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81603a0 con 0x7f30c80cace0 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30cee8d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c815fbc0 con 0x7f30c8071130 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30ccc29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30ccc29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46220/0 (socket says 172.21.15.104:46220) 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7f30ccc29700 1 -- 172.21.15.104:0/2471056258 learned_addr learned my addr 172.21.15.104:0/2471056258 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f30ccc29700 1 -- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30c80cace0 msgr2=0x7f30c814f280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-10T18:44:36.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f30ccc29700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30c80cace0 0x7f30c814f280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f30ccc29700 1 -- 172.21.15.104:0/2471056258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 msgr2=0x7f30c814f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f30ccc29700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f30ccc29700 1 -- 172.21.15.104:0/2471056258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc009550 con 0x7f30c8071130 2023-06-10T18:44:36.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00c850 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fc1b5a5e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc1b0089510 0x7fc1b006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fc1b5a5e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00cb6b0 con 0x7fc1b00cbb40 2023-06-10T18:44:36.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fc1b5a5e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00cb380 con 0x7fc1b00cc860 2023-06-10T18:44:36.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.414+0000 7fc1b5a5e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00c8c70 con 0x7fc1b0089510 2023-06-10T18:44:36.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1af7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc1b00cc860 0x7fc1b016c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:58724/0 (socket says 172.21.15.104:58724) 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1af7fe700 1 -- 172.21.15.104:0/3884932484 learned_addr learned my addr 172.21.15.104:0/3884932484 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1ae7fc700 1 --2- 172.21.15.104:0/3884932484 >> v2:172.21.15.114:3300/0 conn(0x7fc1b0089510 0x7fc1b006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226060323 0 0) 0x7fc1b00cb380 con 0x7fc1b00cc860 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1980035b0 con 0x7fc1b00cc860 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653735285 0 0) 0x7fc1980035b0 con 0x7fc1b00cc860 2023-06-10T18:44:36.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 >> v2:172.21.15.114:3300/0 conn(0x7fc1b0089510 msgr2=0x7fc1b006e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 --2- 172.21.15.104:0/3884932484 >> v2:172.21.15.114:3300/0 conn(0x7fc1b0089510 0x7fc1b006e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 >> v1:172.21.15.104:6789/0 conn(0x7fc1b00cbb40 legacy=0x7fc1b00cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00c8f50 con 0x7fc1b00cc860 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc1a80019f0 con 0x7fc1b00cc860 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 >> v1:172.21.15.114:6789/0 conn(0x7fc1b00cc860 legacy=0x7fc1b016c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198003a00 0x7fc198003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc198004850 0x7fc198004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 0x7fc19801ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fd76ab66700 1 -- 172.21.15.104:0/1501991268 wait complete. 2023-06-10T18:44:36.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c00c9b0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7fd76ab66700 1 Processor -- start 2023-06-10T18:44:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00cd20 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00c8c70 con 0x7fc198003a00 2023-06-10T18:44:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19801d290 con 0x7fc198004850 2023-06-10T18:44:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.415+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19801d450 con 0x7fc198005350 2023-06-10T18:44:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 0x7fc19801ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 -- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc198004850 msgr2=0x7fc198004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc198004850 0x7fc198004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 -- 172.21.15.104:0/3884932484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198003a00 msgr2=0x7fc198003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198003a00 0x7fc198003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 -- 172.21.15.104:0/3884932484 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00c8f50 con 0x7fc198005350 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 -- 172.21.15.104:0/3884932484 reap_dead start 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 0x7fc19801ca20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80098d0 tx=0x7fc198005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd0c02fa4b1972b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.416+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8030700 con 0x7fc198005350 2023-06-10T18:44:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 -- 172.21.15.104:0/3670181417 >> v2:172.21.15.114:3300/0 conn(0x7f3778164620 msgr2=0x7f37780cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 --2- 172.21.15.104:0/3670181417 >> v2:172.21.15.114:3300/0 conn(0x7f3778164620 0x7f37780cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7f3777fff700 1 -- 172.21.15.104:0/3670181417 >> v1:172.21.15.104:6789/0 conn(0x7f3778160400 legacy=0x7f37780c8b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:44:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8030860 con 0x7fc198005350 2023-06-10T18:44:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7fc1adffb700 1 -- 172.21.15.104:0/3884932484 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8030c10 con 0x7fc198005350 2023-06-10T18:44:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 msgr2=0x7fc19801ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.417+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 0x7fc19801ca20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80098d0 tx=0x7fc198005880 comp rx=0 tx=0).stop 2023-06-10T18:44:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7f3777fff700 1 -- 172.21.15.104:0/3670181417 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780cb530 con 0x7f37781607d0 2023-06-10T18:44:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/3884932484 shutdown_connections 2023-06-10T18:44:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/3884932484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc198005350 0x7fc19801ca20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/3884932484 >> 172.21.15.104:0/3884932484 conn(0x7fc1b00b0090 msgr2=0x7fc1b00b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7fd76ab66700 1 -- start start 2023-06-10T18:44:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c086890 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c086e50 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f3777fff700 1 --2- 172.21.15.104:0/3670181417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3770009e50 tx=0x7f377000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2682838dec689b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.418+0000 7f68ee897700 1 -- start start 2023-06-10T18:44:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8156650 0x7f68e81556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e815fcc0 0x7f68e80b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80ccd10 con 0x7f68e80cdd20 2023-06-10T18:44:36.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80ccf60 con 0x7f68e815fcc0 2023-06-10T18:44:36.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640c89f0 0x7fd7640c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7640c9310 0x7fd7640c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641563c0 con 0x7fd7640c8620 2023-06-10T18:44:36.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7640c79b0 con 0x7fd7640c9310 2023-06-10T18:44:36.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd76ab66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7640c7c90 con 0x7fd7640c89f0 2023-06-10T18:44:36.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd768902700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f37777fe700 1 -- 172.21.15.104:0/3670181417 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000e070 con 0x7f37781607d0 2023-06-10T18:44:36.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f37777fe700 1 -- 172.21.15.104:0/3670181417 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770003a20 con 0x7f37781607d0 2023-06-10T18:44:36.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f30ccc29700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f30c000b840 tx=0x7f30c000bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ee897700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80cd240 con 0x7f68e8156650 2023-06-10T18:44:36.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ece34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ece34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55084/0 (socket says 172.21.15.104:55084) 2023-06-10T18:44:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68ece34700 1 -- 172.21.15.104:0/2418483227 learned_addr learned my addr 172.21.15.104:0/2418483227 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7f68e7fff700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8156650 0x7f68e81556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd768902700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55078/0 (socket says 172.21.15.104:55078) 2023-06-10T18:44:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.419+0000 7fd768902700 1 -- 172.21.15.104:0/3606930626 learned_addr learned my addr 172.21.15.104:0/3606930626 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000f840 con 0x7f30c8071130 2023-06-10T18:44:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 -- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640c89f0 msgr2=0x7fd7640c6750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-10T18:44:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640c89f0 0x7fd7640c6750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 -- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7640c9310 msgr2=0x7fd7640c6c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7640c9310 0x7fd7640c6c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 -- 172.21.15.104:0/3606930626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640c7f70 con 0x7fd7640c8620 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd768902700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd75800d800 tx=0x7fd75800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758018070 con 0x7fd7640c8620 2023-06-10T18:44:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7640af260 con 0x7fd7640c8620 2023-06-10T18:44:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7640c73b0 con 0x7fd7640c8620 2023-06-10T18:44:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f68e77fe700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e815fcc0 0x7f68e80b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa30c056040 con 0x7fa3140ce9e0 2023-06-10T18:44:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7580125f0 con 0x7fd7640c8620 2023-06-10T18:44:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758012760 con 0x7fd7640c8620 2023-06-10T18:44:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3c6c013470 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7f3c797fa700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 0x7f3c6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3c6c089840 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/3884932484 shutdown_connections 2023-06-10T18:44:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/3884932484 wait complete. 2023-06-10T18:44:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.421+0000 7fc1b5a5e700 1 Processor -- start 2023-06-10T18:44:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f3c7bfff700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 0x7f3c6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f3c7bfff700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 0x7f3c6806d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c082670 tx=0x7f3c70011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7f377eac4700 1 -- 172.21.15.104:0/3670181417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 msgr2=0x7f37781640e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7f37777fe700 1 -- 172.21.15.104:0/3670181417 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770003db0 con 0x7f37781607d0 2023-06-10T18:44:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7f377eac4700 1 --2- 172.21.15.104:0/3670181417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3770009e50 tx=0x7f377000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 -- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8156650 msgr2=0x7f68e81556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8156650 0x7f68e81556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 -- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e815fcc0 msgr2=0x7f68e80b13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e815fcc0 0x7f68e80b13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 -- 172.21.15.104:0/2418483227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009b30 con 0x7f68e80cdd20 2023-06-10T18:44:36.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7f68ece34700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c780 tx=0x7f68d800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7fc1b5a5e700 1 -- start start 2023-06-10T18:44:36.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c000f9a0 con 0x7f30c8071130 2023-06-10T18:44:36.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000fd50 con 0x7f30c8071130 2023-06-10T18:44:36.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.422+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7640c81b0 con 0x7fd7640c8620 2023-06-10T18:44:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80b2240 con 0x7f30c8071130 2023-06-10T18:44:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.420+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c814e950 con 0x7f30c8071130 2023-06-10T18:44:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f30c000d400 con 0x7f30c8071130 2023-06-10T18:44:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a80052f0 con 0x7f30c8071130 2023-06-10T18:44:36.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f30c57fa700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 0x7f30b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd7580128c0 con 0x7fd7640c8620 2023-06-10T18:44:36.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.423+0000 7fd7617fa700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 0x7fd75006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd758089440 con 0x7fd7640c8620 2023-06-10T18:44:36.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fd763fff700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 0x7fd75006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80054e0 con 0x7f68e80cdd20 2023-06-10T18:44:36.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f30c0089bc0 con 0x7f30c8071130 2023-06-10T18:44:36.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f30c7fff700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 0x7f30b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 0x7fc1b0081160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b00b20a0 0x7fc1b0167aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fd763fff700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 0x7fd75006d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00a7b0 tx=0x7fd75c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b1d60 con 0x7fc198005350 2023-06-10T18:44:36.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00816a0 con 0x7fc1b00b20a0 2023-06-10T18:44:36.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7fc1b5a5e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b3bd0 con 0x7fc1b00b13b0 2023-06-10T18:44:36.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f377eac4700 1 -- 172.21.15.104:0/3670181417 shutdown_connections 2023-06-10T18:44:36.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f377eac4700 1 --2- 172.21.15.104:0/3670181417 >> v2:172.21.15.114:3300/0 conn(0x7f3778164620 0x7f37780cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f377eac4700 1 --2- 172.21.15.104:0/3670181417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37781607d0 0x7f37781640e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f377eac4700 1 -- 172.21.15.104:0/3670181417 >> 172.21.15.104:0/3670181417 conn(0x7f3778069e70 msgr2=0x7f377806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e80cd520 con 0x7f68e80cdd20 2023-06-10T18:44:36.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.424+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e80ce700 con 0x7f68e80cdd20 2023-06-10T18:44:36.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7f377eac4700 1 -- 172.21.15.104:0/3670181417 shutdown_connections 2023-06-10T18:44:36.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7f377eac4700 1 -- 172.21.15.104:0/3670181417 wait complete. 2023-06-10T18:44:36.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7f30c7fff700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 0x7f30b406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30c806b4e0 tx=0x7f30b8006c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c0871c0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7fc1af7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7fc1af7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46248/0 (socket says 172.21.15.104:46248) 2023-06-10T18:44:36.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7fc1af7fe700 1 -- 172.21.15.104:0/757231371 learned_addr learned my addr 172.21.15.104:0/757231371 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.425+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e80cda40 con 0x7f68e80cdd20 2023-06-10T18:44:36.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1ae7fc700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b00b20a0 0x7fc1b0167aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1aeffd700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 0x7fc1b0081160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 -- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b00b20a0 msgr2=0x7fc1b0167aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b00b20a0 0x7fc1b0167aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 -- 172.21.15.104:0/757231371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 msgr2=0x7fc1b0081160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 0x7fc1b0081160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 -- 172.21.15.104:0/757231371 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00b3eb0 con 0x7fc1b00b13b0 2023-06-10T18:44:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1aeffd700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 0x7fc1b0081160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:36.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7fc1af7fe700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009ec0 tx=0x7fc1a8001cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.426+0000 7f377eac4700 1 Processor -- start 2023-06-10T18:44:36.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 -- start start 2023-06-10T18:44:36.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 0x7f377815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3778164620 0x7f37780c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3778150440 con 0x7f3778160400 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37780c7530 con 0x7f37781607d0 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.427+0000 7f377eac4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3778151600 con 0x7f3778164620 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55100/0 (socket says 172.21.15.104:55100) 2023-06-10T18:44:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 -- 172.21.15.104:0/1658778149 learned_addr learned my addr 172.21.15.104:0/1658778149 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f3777fff700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 0x7f377815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377d061700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3778164620 0x7f37780c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 -- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3778164620 msgr2=0x7f37780c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3778164620 0x7f37780c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 -- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 msgr2=0x7f377815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 0x7f377815ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 -- 172.21.15.104:0/1658778149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770009b30 con 0x7f3778160400 2023-06-10T18:44:36.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f3777fff700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 0x7f377815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:36.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.428+0000 7f377c860700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f376c00b450 tx=0x7f376c00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.429+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f30c00548e0 con 0x7f30c8071130 2023-06-10T18:44:36.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.430+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c011830 con 0x7f3778160400 2023-06-10T18:44:36.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.431+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd758054180 con 0x7fd7640c8620 2023-06-10T18:44:36.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.431+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8003680 con 0x7f68e80cdd20 2023-06-10T18:44:36.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.431+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37781518e0 con 0x7f3778160400 2023-06-10T18:44:36.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.431+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d80039f0 con 0x7f68e80cdd20 2023-06-10T18:44:36.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.431+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3778151ea0 con 0x7f3778160400 2023-06-10T18:44:36.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.432+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3778074050 con 0x7f3778160400 2023-06-10T18:44:36.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.432+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c011990 con 0x7f3778160400 2023-06-10T18:44:36.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.432+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c011d40 con 0x7f3778160400 2023-06-10T18:44:36.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.433+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f68d8003b50 con 0x7f68e80cdd20 2023-06-10T18:44:36.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.433+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3c6c0545f0 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.434+0000 7f68e4ff9700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 0x7f68d006d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.434+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f68d8089240 con 0x7f68e80cdd20 2023-06-10T18:44:36.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.434+0000 7f68e7fff700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 0x7f68d006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.434+0000 7f68e7fff700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 0x7f68d006d310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68e000b370 tx=0x7f68e0009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.436+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800e360 con 0x7fc1b00b13b0 2023-06-10T18:44:36.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.436+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8044410 con 0x7fc1b00b13b0 2023-06-10T18:44:36.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.436+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b00b4190 con 0x7fc1b00b13b0 2023-06-10T18:44:36.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.436+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80447a0 con 0x7fc1b00b13b0 2023-06-10T18:44:36.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.436+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b00b2510 con 0x7fc1b00b13b0 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.437+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc1a8044900 con 0x7fc1b00b13b0 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.438+0000 7fc197fff700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 0x7fc19c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.438+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc1a80ad160 con 0x7fc1b00b13b0 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.438+0000 7fc1aeffd700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 0x7fc19c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.441+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1900052f0 con 0x7fc1b00b13b0 2023-06-10T18:44:36.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.443+0000 7fc1aeffd700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 0x7fc19c06d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4001740 tx=0x7fc1a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.443+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f68d8054020 con 0x7f68e80cdd20 2023-06-10T18:44:36.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.445+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f376c01a400 con 0x7f3778160400 2023-06-10T18:44:36.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.445+0000 7f37757fa700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 0x7f376006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:36.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.445+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f376c089df0 con 0x7f3778160400 2023-06-10T18:44:36.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.446+0000 7f3777fff700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 0x7f376006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:36.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.446+0000 7f3777fff700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 0x7f376006d2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3770015040 tx=0x7f3770009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:36.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.450+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc1a8078070 con 0x7fc1b00b13b0 2023-06-10T18:44:36.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.452+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f376c054bc0 con 0x7f3778160400 2023-06-10T18:44:36.750 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:36.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.740+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8f200055a0 con 0x7f8f400c18d0 2023-06-10T18:44:36.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.740+0000 7f8f35ffb700 1 -- 172.21.15.104:0/4027386242 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f38058310 con 0x7f8f400c18d0 2023-06-10T18:44:36.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 msgr2=0x7f8f2406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 0x7f8f2406d260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f40071f00 tx=0x7f8f30008030 comp rx=0 tx=0).stop 2023-06-10T18:44:36.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 msgr2=0x7f8f4014eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8f38007e40 tx=0x7f8f3800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 shutdown_connections 2023-06-10T18:44:36.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f401611b0 0x7f8f400b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f400c18d0 0x7f8f4014eaf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f2406adb0 0x7f8f2406d260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 --2- 172.21.15.104:0/4027386242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f400c8bd0 0x7f8f400c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.744+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 >> 172.21.15.104:0/4027386242 conn(0x7f8f400b0090 msgr2=0x7f8f40070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.749+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 shutdown_connections 2023-06-10T18:44:36.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.749+0000 7f8f47319700 1 -- 172.21.15.104:0/4027386242 wait complete. 2023-06-10T18:44:36.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2023-06-10T18:44:36.773 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.802+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3c7c074050 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.802+0000 7f3c797fa700 1 -- 172.21.15.104:0/3691781180 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c6c021090 con 0x7f3c7c0cbb40 2023-06-10T18:44:36.808 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:36.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 msgr2=0x7f3c6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 0x7f3c6806d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c082670 tx=0x7f3c70011000 comp rx=0 tx=0).stop 2023-06-10T18:44:36.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 msgr2=0x7f3c7c085c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9b0 tx=0x7f3c6c00ed70 comp rx=0 tx=0).stop 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 shutdown_connections 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c7c0ce840 0x7f3c7c087350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c7c0cbb40 0x7f3c7c085c40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c6806ace0 0x7f3c6806d190 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 --2- 172.21.15.104:0/3691781180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c7c0ce470 0x7f3c7c0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 >> 172.21.15.104:0/3691781180 conn(0x7f3c7c0b0090 msgr2=0x7f3c7c071550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 shutdown_connections 2023-06-10T18:44:36.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.805+0000 7f3c83075700 1 -- 172.21.15.104:0/3691781180 wait complete. 2023-06-10T18:44:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.813+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa314074050 con 0x7fa3140ce9e0 2023-06-10T18:44:36.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.814+0000 7fa309ffb700 1 -- 172.21.15.104:0/2088134687 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa30c059660 con 0x7fa3140ce9e0 2023-06-10T18:44:36.820 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:36.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2023-06-10T18:44:36.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.818+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 msgr2=0x7fa2f806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.818+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 0x7fa2f806d260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3141fa8e0 tx=0x7fa310009040 comp rx=0 tx=0).stop 2023-06-10T18:44:36.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.818+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 msgr2=0x7fa314153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.818+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3140cf1f0 tx=0x7fa30c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.824 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 shutdown_connections 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f806adb0 0x7fa2f806d260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3141536c0 0x7fa3140b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3140c92e0 0x7fa314156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 --2- 172.21.15.104:0/2088134687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3140ce9e0 0x7fa314153180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 >> 172.21.15.104:0/2088134687 conn(0x7fa3140b0090 msgr2=0x7fa314070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 shutdown_connections 2023-06-10T18:44:36.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.819+0000 7fa31b775700 1 -- 172.21.15.104:0/2088134687 wait complete. 2023-06-10T18:44:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.827+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f30a80055a0 con 0x7f30c8071130 2023-06-10T18:44:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.828+0000 7f30c57fa700 1 -- 172.21.15.104:0/2471056258 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f30c001c020 con 0x7f30c8071130 2023-06-10T18:44:36.835 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd7640af450 con 0x7fd7640c8620 2023-06-10T18:44:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 msgr2=0x7f30b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 0x7f30b406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30c806b4e0 tx=0x7f30b8006c60 comp rx=0 tx=0).stop 2023-06-10T18:44:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 msgr2=0x7f30c81605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f30c000b840 tx=0x7f30c000bc00 comp rx=0 tx=0).stop 2023-06-10T18:44:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 shutdown_connections 2023-06-10T18:44:36.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c80cca60 0x7f30c814f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30c8071130 0x7f30c81605c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30b406ace0 0x7f30b406d190 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30c806b4e0 tx=0x7f30b8006c60 comp rx=0 tx=0).stop 2023-06-10T18:44:36.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 --2- 172.21.15.104:0/2471056258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30c80cace0 0x7f30c814f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.831+0000 7fd7617fa700 1 -- 172.21.15.104:0/3606930626 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7580577a0 con 0x7fd7640c8620 2023-06-10T18:44:36.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.832+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 >> 172.21.15.104:0/2471056258 conn(0x7f30c80b1610 msgr2=0x7f30c806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.839 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:36.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.833+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 shutdown_connections 2023-06-10T18:44:36.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.833+0000 7f30cee8d700 1 -- 172.21.15.104:0/2471056258 wait complete. 2023-06-10T18:44:36.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.835+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 msgr2=0x7fd75006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.835+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 0x7fd75006d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00a7b0 tx=0x7fd75c009030 comp rx=0 tx=0).stop 2023-06-10T18:44:36.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.835+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 msgr2=0x7fd7640c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.835+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd75800d800 tx=0x7fd75800dbc0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 shutdown_connections 2023-06-10T18:44:36.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd75006ace0 0x7fd75006d190 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00a7b0 tx=0x7fd75c009030 comp rx=0 tx=0).stop 2023-06-10T18:44:36.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7640c9310 0x7fd7640c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7640c8620 0x7fd7640c2b50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 --2- 172.21.15.104:0/3606930626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7640c89f0 0x7fd7640c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.836+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 >> 172.21.15.104:0/3606930626 conn(0x7fd76406d240 msgr2=0x7fd7641580c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.6 2023-06-10T18:44:36.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.837+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 shutdown_connections 2023-06-10T18:44:36.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.837+0000 7fd76ab66700 1 -- 172.21.15.104:0/3606930626 wait complete. 2023-06-10T18:44:36.845 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.840+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f68e80cda40 con 0x7f68e80cdd20 2023-06-10T18:44:36.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.840+0000 7f68e4ff9700 1 -- 172.21.15.104:0/2418483227 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f68e80cda40 con 0x7f68e80cdd20 2023-06-10T18:44:36.846 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:36.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.842+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc190005870 con 0x7fc1b00b13b0 2023-06-10T18:44:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.842+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 msgr2=0x7f68d006d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.842+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 0x7f68d006d310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68e000b370 tx=0x7f68e0009000 comp rx=0 tx=0).stop 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 msgr2=0x7f68e80ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f68d800c780 tx=0x7f68d800ca90 comp rx=0 tx=0).stop 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 shutdown_connections 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68e80cdd20 0x7f68e80ce1c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8156650 0x7f68e81556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68d006ae60 0x7f68d006d310 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68e000b370 tx=0x7f68e0009000 comp rx=0 tx=0).stop 2023-06-10T18:44:36.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 --2- 172.21.15.104:0/2418483227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e815fcc0 0x7f68e80b13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 >> 172.21.15.104:0/2418483227 conn(0x7f68e80b0090 msgr2=0x7f68e80718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.843+0000 7fc197fff700 1 -- 172.21.15.104:0/757231371 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1a807b690 con 0x7fc1b00b13b0 2023-06-10T18:44:36.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.844+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 shutdown_connections 2023-06-10T18:44:36.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.844+0000 7f68ee897700 1 -- 172.21.15.104:0/2418483227 wait complete. 2023-06-10T18:44:36.851 INFO:teuthology.orchestra.run.smithi104.stdout:55834574859 2023-06-10T18:44:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.844+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f377806a3e0 con 0x7f3778160400 2023-06-10T18:44:36.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.845+0000 7f37757fa700 1 -- 172.21.15.104:0/1658778149 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f376c0581e0 con 0x7f3778160400 2023-06-10T18:44:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 msgr2=0x7fc19c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 0x7fc19c06d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4001740 tx=0x7fc1a4009040 comp rx=0 tx=0).stop 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 msgr2=0x7fc1b00b1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009ec0 tx=0x7fc1a8001cb0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 shutdown_connections 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b00b13b0 0x7fc1b00b1820 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc198005350 0x7fc1b0081160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc19c06ace0 0x7fc19c06d190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 --2- 172.21.15.104:0/757231371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b00b20a0 0x7fc1b0167aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 >> 172.21.15.104:0/757231371 conn(0x7fc1b00b0090 msgr2=0x7fc1b0070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 shutdown_connections 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.847+0000 7fc1b5a5e700 1 -- 172.21.15.104:0/757231371 wait complete. 2023-06-10T18:44:36.854 INFO:teuthology.orchestra.run.smithi104.stdout:55834574860 2023-06-10T18:44:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.850+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 msgr2=0x7f376006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.850+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 0x7f376006d2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3770015040 tx=0x7f3770009000 comp rx=0 tx=0).stop 2023-06-10T18:44:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.850+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 msgr2=0x7f377814ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.850+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f376c00b450 tx=0x7f376c00b760 comp rx=0 tx=0).stop 2023-06-10T18:44:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 shutdown_connections 2023-06-10T18:44:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3778164620 0x7f37780c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3778160400 0x7f377814ff00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f376006ae40 0x7f376006d2f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 --2- 172.21.15.104:0/1658778149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37781607d0 0x7f377815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 >> 172.21.15.104:0/1658778149 conn(0x7f3778069e70 msgr2=0x7f37780716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 shutdown_connections 2023-06-10T18:44:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:36.854+0000 7f377eac4700 1 -- 172.21.15.104:0/1658778149 wait complete. 2023-06-10T18:44:36.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.2 2023-06-10T18:44:36.863 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2023-06-10T18:44:36.865 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2023-06-10T18:44:36.866 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2023-06-10T18:44:36.869 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2023-06-10T18:44:36.874 DEBUG:teuthology.parallel:result is None 2023-06-10T18:44:36.875 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-06-10T18:44:36.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:44:37.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.031+0000 7faa9fd69700 1 Processor -- start 2023-06-10T18:44:37.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.031+0000 7faa9fd69700 1 -- start start 2023-06-10T18:44:37.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9fd69700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa9815a0b0 0x7faa980717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9fd69700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faa98071130 0x7faa9815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9fd69700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9806b2e0 con 0x7faa980c93e0 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9fd69700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9815e150 con 0x7faa9815a0b0 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9fd69700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9815db40 con 0x7faa98071130 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa9815a0b0 0x7faa980717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa9815a0b0 0x7faa980717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55112/0 (socket says 172.21.15.104:55112) 2023-06-10T18:44:37.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 -- 172.21.15.104:0/2051576308 learned_addr learned my addr 172.21.15.104:0/2051576308 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 -- 172.21.15.104:0/2051576308 >> v2:172.21.15.114:3300/0 conn(0x7faa98071130 msgr2=0x7faa9815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 --2- 172.21.15.104:0/2051576308 >> v2:172.21.15.114:3300/0 conn(0x7faa98071130 0x7faa9815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 -- 172.21.15.104:0/2051576308 >> v1:172.21.15.114:6789/0 conn(0x7faa980c93e0 legacy=0x7faa98070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 -- 172.21.15.104:0/2051576308 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9806b530 con 0x7faa9815a0b0 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9db05700 1 --2- 172.21.15.104:0/2051576308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980717d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faa980cfd20 tx=0x7faa94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe120cba2dbef429 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9cb03700 1 -- 172.21.15.104:0/2051576308 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94004230 con 0x7faa9815a0b0 2023-06-10T18:44:37.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9cb03700 1 -- 172.21.15.104:0/2051576308 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa94004390 con 0x7faa9815a0b0 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.032+0000 7faa9cb03700 1 -- 172.21.15.104:0/2051576308 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa940036a0 con 0x7faa9815a0b0 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 -- 172.21.15.104:0/2051576308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 msgr2=0x7faa980717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2051576308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980717d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faa980cfd20 tx=0x7faa94009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 -- 172.21.15.104:0/2051576308 shutdown_connections 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2051576308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980717d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2051576308 >> v2:172.21.15.114:3300/0 conn(0x7faa98071130 0x7faa9815e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 -- 172.21.15.104:0/2051576308 >> 172.21.15.104:0/2051576308 conn(0x7faa980b1610 msgr2=0x7faa980ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:37.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 -- 172.21.15.104:0/2051576308 shutdown_connections 2023-06-10T18:44:37.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.033+0000 7faa9fd69700 1 -- 172.21.15.104:0/2051576308 wait complete. 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.035+0000 7faa9fd69700 1 Processor -- start 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.035+0000 7faa9fd69700 1 -- start start 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa98071130 0x7faa981f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa980c93e0 0x7faa980c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa981f4a00 con 0x7faa9815a0b0 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa981f3a20 con 0x7faa98071130 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa980c52f0 con 0x7faa980c93e0 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55122/0 (socket says 172.21.15.104:55122) 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 -- 172.21.15.104:0/2628620801 learned_addr learned my addr 172.21.15.104:0/2628620801 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9db05700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa98071130 0x7faa981f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9d304700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa980c93e0 0x7faa980c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 -- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa980c93e0 msgr2=0x7faa980c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa980c93e0 0x7faa980c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 -- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa98071130 msgr2=0x7faa981f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa98071130 0x7faa981f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 -- 172.21.15.104:0/2628620801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa94009550 con 0x7faa9815a0b0 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9e306700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa9000b970 tx=0x7faa9000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9000c9b0 con 0x7faa9815a0b0 2023-06-10T18:44:37.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9000cb30 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa980c55d0 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90014440 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.036+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa981f3fa0 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.037+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa980cf4d0 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.038+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faa9001d3b0 con 0x7faa9815a0b0 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.038+0000 7faa8e7fc700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 0x7faa7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.038+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7faa90006e40 con 0x7faa9815a0b0 2023-06-10T18:44:37.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.038+0000 7faa9db05700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 0x7faa7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.039+0000 7faa9db05700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 0x7faa7c06d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa94009130 tx=0x7faa94008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.042+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faa90054290 con 0x7faa9815a0b0 2023-06-10T18:44:37.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.223+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa9806b810 con 0x7faa7c06ace0 2023-06-10T18:44:37.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.234+0000 7faa8e7fc700 1 -- 172.21.15.104:0/2628620801 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367305 (secure 0 0 0) 0x7faa94020070 con 0x7faa7c06ace0 2023-06-10T18:44:37.243 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:37.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 msgr2=0x7faa7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 0x7faa7c06d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa94009130 tx=0x7faa94008030 comp rx=0 tx=0).stop 2023-06-10T18:44:37.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 msgr2=0x7faa980c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa9000b970 tx=0x7faa9000bd30 comp rx=0 tx=0).stop 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 shutdown_connections 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa7c06ace0 0x7faa7c06d190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa98071130 0x7faa981f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa9815a0b0 0x7faa980c4c70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 --2- 172.21.15.104:0/2628620801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa980c93e0 0x7faa980c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 >> 172.21.15.104:0/2628620801 conn(0x7faa980b1610 msgr2=0x7faa9815a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:37.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 shutdown_connections 2023-06-10T18:44:37.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.237+0000 7faa9fd69700 1 -- 172.21.15.104:0/2628620801 wait complete. 2023-06-10T18:44:37.247 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:44:37.255 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-06-10T18:44:35.354532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":3480,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":3563520,"data_stored":2145052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003345"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423396+0000","last_change":"2023-06-10T18:44:08.974320+0000","last_active":"2023-06-10T18:44:13.423396+0000","last_peered":"2023-06-10T18:44:13.423396+0000","last_clean":"2023-06-10T18:44:13.423396+0000","last_became_active":"2023-06-10T18:44:08.973998+0000","last_became_peered":"2023-06-10T18:44:08.973998+0000","last_unstale":"2023-06-10T18:44:13.423396+0000","last_undegraded":"2023-06-10T18:44:13.423396+0000","last_fullsized":"2023-06-10T18:44:13.423396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:29.774977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393150+0000","last_change":"2023-06-10T18:44:08.986524+0000","last_active":"2023-06-10T18:44:13.393150+0000","last_peered":"2023-06-10T18:44:13.393150+0000","last_clean":"2023-06-10T18:44:13.393150+0000","last_became_active":"2023-06-10T18:44:08.986238+0000","last_became_peered":"2023-06-10T18:44:08.986238+0000","last_unstale":"2023-06-10T18:44:13.393150+0000","last_undegraded":"2023-06-10T18:44:13.393150+0000","last_fullsized":"2023-06-10T18:44:13.393150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423472+0000","last_change":"2023-06-10T18:44:08.991803+0000","last_active":"2023-06-10T18:44:13.423472+0000","last_peered":"2023-06-10T18:44:13.423472+0000","last_clean":"2023-06-10T18:44:13.423472+0000","last_became_active":"2023-06-10T18:44:08.986777+0000","last_became_peered":"2023-06-10T18:44:08.986777+0000","last_unstale":"2023-06-10T18:44:13.423472+0000","last_undegraded":"2023-06-10T18:44:13.423472+0000","last_fullsized":"2023-06-10T18:44:13.423472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423103+0000","last_change":"2023-06-10T18:44:08.984677+0000","last_active":"2023-06-10T18:44:13.423103+0000","last_peered":"2023-06-10T18:44:13.423103+0000","last_clean":"2023-06-10T18:44:13.423103+0000","last_became_active":"2023-06-10T18:44:08.984368+0000","last_became_peered":"2023-06-10T18:44:08.984368+0000","last_unstale":"2023-06-10T18:44:13.423103+0000","last_undegraded":"2023-06-10T18:44:13.423103+0000","last_fullsized":"2023-06-10T18:44:13.423103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:21.561425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423951+0000","last_change":"2023-06-10T18:44:08.988022+0000","last_active":"2023-06-10T18:44:13.423951+0000","last_peered":"2023-06-10T18:44:13.423951+0000","last_clean":"2023-06-10T18:44:13.423951+0000","last_became_active":"2023-06-10T18:44:08.983186+0000","last_became_peered":"2023-06-10T18:44:08.983186+0000","last_unstale":"2023-06-10T18:44:13.423951+0000","last_undegraded":"2023-06-10T18:44:13.423951+0000","last_fullsized":"2023-06-10T18:44:13.423951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:24.256200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423368+0000","last_change":"2023-06-10T18:44:08.986749+0000","last_active":"2023-06-10T18:44:13.423368+0000","last_peered":"2023-06-10T18:44:13.423368+0000","last_clean":"2023-06-10T18:44:13.423368+0000","last_became_active":"2023-06-10T18:44:08.986456+0000","last_became_peered":"2023-06-10T18:44:08.986456+0000","last_unstale":"2023-06-10T18:44:13.423368+0000","last_undegraded":"2023-06-10T18:44:13.423368+0000","last_fullsized":"2023-06-10T18:44:13.423368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394293+0000","last_change":"2023-06-10T18:44:08.992850+0000","last_active":"2023-06-10T18:44:13.394293+0000","last_peered":"2023-06-10T18:44:13.394293+0000","last_clean":"2023-06-10T18:44:13.394293+0000","last_became_active":"2023-06-10T18:44:08.992572+0000","last_became_peered":"2023-06-10T18:44:08.992572+0000","last_unstale":"2023-06-10T18:44:13.394293+0000","last_undegraded":"2023-06-10T18:44:13.394293+0000","last_fullsized":"2023-06-10T18:44:13.394293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392634+0000","last_change":"2023-06-10T18:44:08.979428+0000","last_active":"2023-06-10T18:44:13.392634+0000","last_peered":"2023-06-10T18:44:13.392634+0000","last_clean":"2023-06-10T18:44:13.392634+0000","last_became_active":"2023-06-10T18:44:08.979157+0000","last_became_peered":"2023-06-10T18:44:08.979157+0000","last_unstale":"2023-06-10T18:44:13.392634+0000","last_undegraded":"2023-06-10T18:44:13.392634+0000","last_fullsized":"2023-06-10T18:44:13.392634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:24.114600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423251+0000","last_change":"2023-06-10T18:44:08.984436+0000","last_active":"2023-06-10T18:44:13.423251+0000","last_peered":"2023-06-10T18:44:13.423251+0000","last_clean":"2023-06-10T18:44:13.423251+0000","last_became_active":"2023-06-10T18:44:08.984021+0000","last_became_peered":"2023-06-10T18:44:08.984021+0000","last_unstale":"2023-06-10T18:44:13.423251+0000","last_undegraded":"2023-06-10T18:44:13.423251+0000","last_fullsized":"2023-06-10T18:44:13.423251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:23:54.181578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393491+0000","last_change":"2023-06-10T18:44:08.984325+0000","last_active":"2023-06-10T18:44:13.393491+0000","last_peered":"2023-06-10T18:44:13.393491+0000","last_clean":"2023-06-10T18:44:13.393491+0000","last_became_active":"2023-06-10T18:44:08.983999+0000","last_became_peered":"2023-06-10T18:44:08.983999+0000","last_unstale":"2023-06-10T18:44:13.393491+0000","last_undegraded":"2023-06-10T18:44:13.393491+0000","last_fullsized":"2023-06-10T18:44:13.393491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423627+0000","last_change":"2023-06-10T18:44:08.991961+0000","last_active":"2023-06-10T18:44:13.423627+0000","last_peered":"2023-06-10T18:44:13.423627+0000","last_clean":"2023-06-10T18:44:13.423627+0000","last_became_active":"2023-06-10T18:44:08.991527+0000","last_became_peered":"2023-06-10T18:44:08.991527+0000","last_unstale":"2023-06-10T18:44:13.423627+0000","last_undegraded":"2023-06-10T18:44:13.423627+0000","last_fullsized":"2023-06-10T18:44:13.423627+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423011+0000","last_change":"2023-06-10T18:44:08.995250+0000","last_active":"2023-06-10T18:44:13.423011+0000","last_peered":"2023-06-10T18:44:13.423011+0000","last_clean":"2023-06-10T18:44:13.423011+0000","last_became_active":"2023-06-10T18:44:08.994854+0000","last_became_peered":"2023-06-10T18:44:08.994854+0000","last_unstale":"2023-06-10T18:44:13.423011+0000","last_undegraded":"2023-06-10T18:44:13.423011+0000","last_fullsized":"2023-06-10T18:44:13.423011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:13:54.715925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422801+0000","last_change":"2023-06-10T18:44:08.973692+0000","last_active":"2023-06-10T18:44:13.422801+0000","last_peered":"2023-06-10T18:44:13.422801+0000","last_clean":"2023-06-10T18:44:13.422801+0000","last_became_active":"2023-06-10T18:44:08.973417+0000","last_became_peered":"2023-06-10T18:44:08.973417+0000","last_unstale":"2023-06-10T18:44:13.422801+0000","last_undegraded":"2023-06-10T18:44:13.422801+0000","last_fullsized":"2023-06-10T18:44:13.422801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:27:20.138188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392896+0000","last_change":"2023-06-10T18:44:08.976711+0000","last_active":"2023-06-10T18:44:13.392896+0000","last_peered":"2023-06-10T18:44:13.392896+0000","last_clean":"2023-06-10T18:44:13.392896+0000","last_became_active":"2023-06-10T18:44:08.976415+0000","last_became_peered":"2023-06-10T18:44:08.976415+0000","last_unstale":"2023-06-10T18:44:13.392896+0000","last_undegraded":"2023-06-10T18:44:13.392896+0000","last_fullsized":"2023-06-10T18:44:13.392896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394066+0000","last_change":"2023-06-10T18:44:08.992788+0000","last_active":"2023-06-10T18:44:13.394066+0000","last_peered":"2023-06-10T18:44:13.394066+0000","last_clean":"2023-06-10T18:44:13.394066+0000","last_became_active":"2023-06-10T18:44:08.992335+0000","last_became_peered":"2023-06-10T18:44:08.992335+0000","last_unstale":"2023-06-10T18:44:13.394066+0000","last_undegraded":"2023-06-10T18:44:13.394066+0000","last_fullsized":"2023-06-10T18:44:13.394066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423080+0000","last_change":"2023-06-10T18:44:08.993080+0000","last_active":"2023-06-10T18:44:13.423080+0000","last_peered":"2023-06-10T18:44:13.423080+0000","last_clean":"2023-06-10T18:44:13.423080+0000","last_became_active":"2023-06-10T18:44:08.992794+0000","last_became_peered":"2023-06-10T18:44:08.992794+0000","last_unstale":"2023-06-10T18:44:13.423080+0000","last_undegraded":"2023-06-10T18:44:13.423080+0000","last_fullsized":"2023-06-10T18:44:13.423080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423378+0000","last_change":"2023-06-10T18:44:08.984915+0000","last_active":"2023-06-10T18:44:13.423378+0000","last_peered":"2023-06-10T18:44:13.423378+0000","last_clean":"2023-06-10T18:44:13.423378+0000","last_became_active":"2023-06-10T18:44:08.984639+0000","last_became_peered":"2023-06-10T18:44:08.984639+0000","last_unstale":"2023-06-10T18:44:13.423378+0000","last_undegraded":"2023-06-10T18:44:13.423378+0000","last_fullsized":"2023-06-10T18:44:13.423378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:53.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424069+0000","last_change":"2023-06-10T18:44:08.989447+0000","last_active":"2023-06-10T18:44:13.424069+0000","last_peered":"2023-06-10T18:44:13.424069+0000","last_clean":"2023-06-10T18:44:13.424069+0000","last_became_active":"2023-06-10T18:44:08.987688+0000","last_became_peered":"2023-06-10T18:44:08.987688+0000","last_unstale":"2023-06-10T18:44:13.424069+0000","last_undegraded":"2023-06-10T18:44:13.424069+0000","last_fullsized":"2023-06-10T18:44:13.424069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422601+0000","last_change":"2023-06-10T18:44:08.984545+0000","last_active":"2023-06-10T18:44:13.422601+0000","last_peered":"2023-06-10T18:44:13.422601+0000","last_clean":"2023-06-10T18:44:13.422601+0000","last_became_active":"2023-06-10T18:44:08.983857+0000","last_became_peered":"2023-06-10T18:44:08.983857+0000","last_unstale":"2023-06-10T18:44:13.422601+0000","last_undegraded":"2023-06-10T18:44:13.422601+0000","last_fullsized":"2023-06-10T18:44:13.422601+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:46.973658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423818+0000","last_change":"2023-06-10T18:44:08.993840+0000","last_active":"2023-06-10T18:44:13.423818+0000","last_peered":"2023-06-10T18:44:13.423818+0000","last_clean":"2023-06-10T18:44:13.423818+0000","last_became_active":"2023-06-10T18:44:08.993524+0000","last_became_peered":"2023-06-10T18:44:08.993524+0000","last_unstale":"2023-06-10T18:44:13.423818+0000","last_undegraded":"2023-06-10T18:44:13.423818+0000","last_fullsized":"2023-06-10T18:44:13.423818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393980+0000","last_change":"2023-06-10T18:44:08.994356+0000","last_active":"2023-06-10T18:44:13.393980+0000","last_peered":"2023-06-10T18:44:13.393980+0000","last_clean":"2023-06-10T18:44:13.393980+0000","last_became_active":"2023-06-10T18:44:08.994055+0000","last_became_peered":"2023-06-10T18:44:08.994055+0000","last_unstale":"2023-06-10T18:44:13.393980+0000","last_undegraded":"2023-06-10T18:44:13.393980+0000","last_fullsized":"2023-06-10T18:44:13.393980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423446+0000","last_change":"2023-06-10T18:44:08.988865+0000","last_active":"2023-06-10T18:44:13.423446+0000","last_peered":"2023-06-10T18:44:13.423446+0000","last_clean":"2023-06-10T18:44:13.423446+0000","last_became_active":"2023-06-10T18:44:08.988631+0000","last_became_peered":"2023-06-10T18:44:08.988631+0000","last_unstale":"2023-06-10T18:44:13.423446+0000","last_undegraded":"2023-06-10T18:44:13.423446+0000","last_fullsized":"2023-06-10T18:44:13.423446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:18.367120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424783+0000","last_change":"2023-06-10T18:44:08.985010+0000","last_active":"2023-06-10T18:44:13.424783+0000","last_peered":"2023-06-10T18:44:13.424783+0000","last_clean":"2023-06-10T18:44:13.424783+0000","last_became_active":"2023-06-10T18:44:08.984770+0000","last_became_peered":"2023-06-10T18:44:08.984770+0000","last_unstale":"2023-06-10T18:44:13.424783+0000","last_undegraded":"2023-06-10T18:44:13.424783+0000","last_fullsized":"2023-06-10T18:44:13.424783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394283+0000","last_change":"2023-06-10T18:44:08.994280+0000","last_active":"2023-06-10T18:44:13.394283+0000","last_peered":"2023-06-10T18:44:13.394283+0000","last_clean":"2023-06-10T18:44:13.394283+0000","last_became_active":"2023-06-10T18:44:08.993996+0000","last_became_peered":"2023-06-10T18:44:08.993996+0000","last_unstale":"2023-06-10T18:44:13.394283+0000","last_undegraded":"2023-06-10T18:44:13.394283+0000","last_fullsized":"2023-06-10T18:44:13.394283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394507+0000","last_change":"2023-06-10T18:44:08.988231+0000","last_active":"2023-06-10T18:44:13.394507+0000","last_peered":"2023-06-10T18:44:13.394507+0000","last_clean":"2023-06-10T18:44:13.394507+0000","last_became_active":"2023-06-10T18:44:08.987866+0000","last_became_peered":"2023-06-10T18:44:08.987866+0000","last_unstale":"2023-06-10T18:44:13.394507+0000","last_undegraded":"2023-06-10T18:44:13.394507+0000","last_fullsized":"2023-06-10T18:44:13.394507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394552+0000","last_change":"2023-06-10T18:44:08.984357+0000","last_active":"2023-06-10T18:44:13.394552+0000","last_peered":"2023-06-10T18:44:13.394552+0000","last_clean":"2023-06-10T18:44:13.394552+0000","last_became_active":"2023-06-10T18:44:08.984034+0000","last_became_peered":"2023-06-10T18:44:08.984034+0000","last_unstale":"2023-06-10T18:44:13.394552+0000","last_undegraded":"2023-06-10T18:44:13.394552+0000","last_fullsized":"2023-06-10T18:44:13.394552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393451+0000","last_change":"2023-06-10T18:44:08.987489+0000","last_active":"2023-06-10T18:44:13.393451+0000","last_peered":"2023-06-10T18:44:13.393451+0000","last_clean":"2023-06-10T18:44:13.393451+0000","last_became_active":"2023-06-10T18:44:08.985804+0000","last_became_peered":"2023-06-10T18:44:08.985804+0000","last_unstale":"2023-06-10T18:44:13.393451+0000","last_undegraded":"2023-06-10T18:44:13.393451+0000","last_fullsized":"2023-06-10T18:44:13.393451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424186+0000","last_change":"2023-06-10T18:44:13.001999+0000","last_active":"2023-06-10T18:44:13.424186+0000","last_peered":"2023-06-10T18:44:13.424186+0000","last_clean":"2023-06-10T18:44:13.424186+0000","last_became_active":"2023-06-10T18:44:13.001637+0000","last_became_peered":"2023-06-10T18:44:13.001637+0000","last_unstale":"2023-06-10T18:44:13.424186+0000","last_undegraded":"2023-06-10T18:44:13.424186+0000","last_fullsized":"2023-06-10T18:44:13.424186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423197+0000","last_change":"2023-06-10T18:44:08.575638+0000","last_active":"2023-06-10T18:44:13.423197+0000","last_peered":"2023-06-10T18:44:13.423197+0000","last_clean":"2023-06-10T18:44:13.423197+0000","last_became_active":"2023-06-10T18:44:08.575345+0000","last_became_peered":"2023-06-10T18:44:08.575345+0000","last_unstale":"2023-06-10T18:44:13.423197+0000","last_undegraded":"2023-06-10T18:44:13.423197+0000","last_fullsized":"2023-06-10T18:44:13.423197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:11.022238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425003+0000","last_change":"2023-06-10T18:44:08.986591+0000","last_active":"2023-06-10T18:44:13.425003+0000","last_peered":"2023-06-10T18:44:13.425003+0000","last_clean":"2023-06-10T18:44:13.425003+0000","last_became_active":"2023-06-10T18:44:08.986320+0000","last_became_peered":"2023-06-10T18:44:08.986320+0000","last_unstale":"2023-06-10T18:44:13.425003+0000","last_undegraded":"2023-06-10T18:44:13.425003+0000","last_fullsized":"2023-06-10T18:44:13.425003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424063+0000","last_change":"2023-06-10T18:44:13.002454+0000","last_active":"2023-06-10T18:44:13.424063+0000","last_peered":"2023-06-10T18:44:13.424063+0000","last_clean":"2023-06-10T18:44:13.424063+0000","last_became_active":"2023-06-10T18:44:13.002101+0000","last_became_peered":"2023-06-10T18:44:13.002101+0000","last_unstale":"2023-06-10T18:44:13.424063+0000","last_undegraded":"2023-06-10T18:44:13.424063+0000","last_fullsized":"2023-06-10T18:44:13.424063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:38.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393702+0000","last_change":"2023-06-10T18:44:08.581341+0000","last_active":"2023-06-10T18:44:13.393702+0000","last_peered":"2023-06-10T18:44:13.393702+0000","last_clean":"2023-06-10T18:44:13.393702+0000","last_became_active":"2023-06-10T18:44:08.580977+0000","last_became_peered":"2023-06-10T18:44:08.580977+0000","last_unstale":"2023-06-10T18:44:13.393702+0000","last_undegraded":"2023-06-10T18:44:13.393702+0000","last_fullsized":"2023-06-10T18:44:13.393702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423101+0000","last_change":"2023-06-10T18:44:08.990386+0000","last_active":"2023-06-10T18:44:13.423101+0000","last_peered":"2023-06-10T18:44:13.423101+0000","last_clean":"2023-06-10T18:44:13.423101+0000","last_became_active":"2023-06-10T18:44:08.989816+0000","last_became_peered":"2023-06-10T18:44:08.989816+0000","last_unstale":"2023-06-10T18:44:13.423101+0000","last_undegraded":"2023-06-10T18:44:13.423101+0000","last_fullsized":"2023-06-10T18:44:13.423101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:27.419215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422737+0000","last_change":"2023-06-10T18:44:13.011429+0000","last_active":"2023-06-10T18:44:13.422737+0000","last_peered":"2023-06-10T18:44:13.422737+0000","last_clean":"2023-06-10T18:44:13.422737+0000","last_became_active":"2023-06-10T18:44:13.010928+0000","last_became_peered":"2023-06-10T18:44:13.010928+0000","last_unstale":"2023-06-10T18:44:13.422737+0000","last_undegraded":"2023-06-10T18:44:13.422737+0000","last_fullsized":"2023-06-10T18:44:13.422737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:25:46.759866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425044+0000","last_change":"2023-06-10T18:44:08.583484+0000","last_active":"2023-06-10T18:44:13.425044+0000","last_peered":"2023-06-10T18:44:13.425044+0000","last_clean":"2023-06-10T18:44:13.425044+0000","last_became_active":"2023-06-10T18:44:08.583197+0000","last_became_peered":"2023-06-10T18:44:08.583197+0000","last_unstale":"2023-06-10T18:44:13.425044+0000","last_undegraded":"2023-06-10T18:44:13.425044+0000","last_fullsized":"2023-06-10T18:44:13.425044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:33.286044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424162+0000","last_change":"2023-06-10T18:44:08.983580+0000","last_active":"2023-06-10T18:44:13.424162+0000","last_peered":"2023-06-10T18:44:13.424162+0000","last_clean":"2023-06-10T18:44:13.424162+0000","last_became_active":"2023-06-10T18:44:08.983254+0000","last_became_peered":"2023-06-10T18:44:08.983254+0000","last_unstale":"2023-06-10T18:44:13.424162+0000","last_undegraded":"2023-06-10T18:44:13.424162+0000","last_fullsized":"2023-06-10T18:44:13.424162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:53:36.396900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423826+0000","last_change":"2023-06-10T18:44:12.997904+0000","last_active":"2023-06-10T18:44:13.423826+0000","last_peered":"2023-06-10T18:44:13.423826+0000","last_clean":"2023-06-10T18:44:13.423826+0000","last_became_active":"2023-06-10T18:44:12.997502+0000","last_became_peered":"2023-06-10T18:44:12.997502+0000","last_unstale":"2023-06-10T18:44:13.423826+0000","last_undegraded":"2023-06-10T18:44:13.423826+0000","last_fullsized":"2023-06-10T18:44:13.423826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:17.892337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423483+0000","last_change":"2023-06-10T18:44:08.575792+0000","last_active":"2023-06-10T18:44:13.423483+0000","last_peered":"2023-06-10T18:44:13.423483+0000","last_clean":"2023-06-10T18:44:13.423483+0000","last_became_active":"2023-06-10T18:44:08.573780+0000","last_became_peered":"2023-06-10T18:44:08.573780+0000","last_unstale":"2023-06-10T18:44:13.423483+0000","last_undegraded":"2023-06-10T18:44:13.423483+0000","last_fullsized":"2023-06-10T18:44:13.423483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:19.709371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424910+0000","last_change":"2023-06-10T18:44:08.986631+0000","last_active":"2023-06-10T18:44:13.424910+0000","last_peered":"2023-06-10T18:44:13.424910+0000","last_clean":"2023-06-10T18:44:13.424910+0000","last_became_active":"2023-06-10T18:44:08.986357+0000","last_became_peered":"2023-06-10T18:44:08.986357+0000","last_unstale":"2023-06-10T18:44:13.424910+0000","last_undegraded":"2023-06-10T18:44:13.424910+0000","last_fullsized":"2023-06-10T18:44:13.424910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393507+0000","last_change":"2023-06-10T18:44:13.002734+0000","last_active":"2023-06-10T18:44:13.393507+0000","last_peered":"2023-06-10T18:44:13.393507+0000","last_clean":"2023-06-10T18:44:13.393507+0000","last_became_active":"2023-06-10T18:44:13.002226+0000","last_became_peered":"2023-06-10T18:44:13.002226+0000","last_unstale":"2023-06-10T18:44:13.393507+0000","last_undegraded":"2023-06-10T18:44:13.393507+0000","last_fullsized":"2023-06-10T18:44:13.393507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:11.264615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393372+0000","last_change":"2023-06-10T18:44:08.571167+0000","last_active":"2023-06-10T18:44:13.393372+0000","last_peered":"2023-06-10T18:44:13.393372+0000","last_clean":"2023-06-10T18:44:13.393372+0000","last_became_active":"2023-06-10T18:44:08.570885+0000","last_became_peered":"2023-06-10T18:44:08.570885+0000","last_unstale":"2023-06-10T18:44:13.393372+0000","last_undegraded":"2023-06-10T18:44:13.393372+0000","last_fullsized":"2023-06-10T18:44:13.393372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.984523+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.984265+0000","last_became_peered":"2023-06-10T18:44:08.984265+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425522+0000","last_change":"2023-06-10T18:44:13.005254+0000","last_active":"2023-06-10T18:44:13.425522+0000","last_peered":"2023-06-10T18:44:13.425522+0000","last_clean":"2023-06-10T18:44:13.425522+0000","last_became_active":"2023-06-10T18:44:13.004908+0000","last_became_peered":"2023-06-10T18:44:13.004908+0000","last_unstale":"2023-06-10T18:44:13.425522+0000","last_undegraded":"2023-06-10T18:44:13.425522+0000","last_fullsized":"2023-06-10T18:44:13.425522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:23.228638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394120+0000","last_change":"2023-06-10T18:44:08.577421+0000","last_active":"2023-06-10T18:44:13.394120+0000","last_peered":"2023-06-10T18:44:13.394120+0000","last_clean":"2023-06-10T18:44:13.394120+0000","last_became_active":"2023-06-10T18:44:08.577127+0000","last_became_peered":"2023-06-10T18:44:08.577127+0000","last_unstale":"2023-06-10T18:44:13.394120+0000","last_undegraded":"2023-06-10T18:44:13.394120+0000","last_fullsized":"2023-06-10T18:44:13.394120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:14.393845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393966+0000","last_change":"2023-06-10T18:44:08.986118+0000","last_active":"2023-06-10T18:44:13.393966+0000","last_peered":"2023-06-10T18:44:13.393966+0000","last_clean":"2023-06-10T18:44:13.393966+0000","last_became_active":"2023-06-10T18:44:08.985845+0000","last_became_peered":"2023-06-10T18:44:08.985845+0000","last_unstale":"2023-06-10T18:44:13.393966+0000","last_undegraded":"2023-06-10T18:44:13.393966+0000","last_fullsized":"2023-06-10T18:44:13.393966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393763+0000","last_change":"2023-06-10T18:44:13.009330+0000","last_active":"2023-06-10T18:44:13.393763+0000","last_peered":"2023-06-10T18:44:13.393763+0000","last_clean":"2023-06-10T18:44:13.393763+0000","last_became_active":"2023-06-10T18:44:13.008968+0000","last_became_peered":"2023-06-10T18:44:13.008968+0000","last_unstale":"2023-06-10T18:44:13.393763+0000","last_undegraded":"2023-06-10T18:44:13.393763+0000","last_fullsized":"2023-06-10T18:44:13.393763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:01.454996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:18.388487+0000","last_change":"2023-06-10T18:44:08.577256+0000","last_active":"2023-06-10T18:44:18.388487+0000","last_peered":"2023-06-10T18:44:18.388487+0000","last_clean":"2023-06-10T18:44:18.388487+0000","last_became_active":"2023-06-10T18:44:08.576987+0000","last_became_peered":"2023-06-10T18:44:08.576987+0000","last_unstale":"2023-06-10T18:44:18.388487+0000","last_undegraded":"2023-06-10T18:44:18.388487+0000","last_fullsized":"2023-06-10T18:44:18.388487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424232+0000","last_change":"2023-06-10T18:44:08.989046+0000","last_active":"2023-06-10T18:44:13.424232+0000","last_peered":"2023-06-10T18:44:13.424232+0000","last_clean":"2023-06-10T18:44:13.424232+0000","last_became_active":"2023-06-10T18:44:08.988763+0000","last_became_peered":"2023-06-10T18:44:08.988763+0000","last_unstale":"2023-06-10T18:44:13.424232+0000","last_undegraded":"2023-06-10T18:44:13.424232+0000","last_fullsized":"2023-06-10T18:44:13.424232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393726+0000","last_change":"2023-06-10T18:44:13.009976+0000","last_active":"2023-06-10T18:44:13.393726+0000","last_peered":"2023-06-10T18:44:13.393726+0000","last_clean":"2023-06-10T18:44:13.393726+0000","last_became_active":"2023-06-10T18:44:13.009599+0000","last_became_peered":"2023-06-10T18:44:13.009599+0000","last_unstale":"2023-06-10T18:44:13.393726+0000","last_undegraded":"2023-06-10T18:44:13.393726+0000","last_fullsized":"2023-06-10T18:44:13.393726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393682+0000","last_change":"2023-06-10T18:44:08.578965+0000","last_active":"2023-06-10T18:44:13.393682+0000","last_peered":"2023-06-10T18:44:13.393682+0000","last_clean":"2023-06-10T18:44:13.393682+0000","last_became_active":"2023-06-10T18:44:08.578666+0000","last_became_peered":"2023-06-10T18:44:08.578666+0000","last_unstale":"2023-06-10T18:44:13.393682+0000","last_undegraded":"2023-06-10T18:44:13.393682+0000","last_fullsized":"2023-06-10T18:44:13.393682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425838+0000","last_change":"2023-06-10T18:44:08.986550+0000","last_active":"2023-06-10T18:44:13.425838+0000","last_peered":"2023-06-10T18:44:13.425838+0000","last_clean":"2023-06-10T18:44:13.425838+0000","last_became_active":"2023-06-10T18:44:08.986267+0000","last_became_peered":"2023-06-10T18:44:08.986267+0000","last_unstale":"2023-06-10T18:44:13.425838+0000","last_undegraded":"2023-06-10T18:44:13.425838+0000","last_fullsized":"2023-06-10T18:44:13.425838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425693+0000","last_change":"2023-06-10T18:44:13.007347+0000","last_active":"2023-06-10T18:44:13.425693+0000","last_peered":"2023-06-10T18:44:13.425693+0000","last_clean":"2023-06-10T18:44:13.425693+0000","last_became_active":"2023-06-10T18:44:13.006948+0000","last_became_peered":"2023-06-10T18:44:13.006948+0000","last_unstale":"2023-06-10T18:44:13.425693+0000","last_undegraded":"2023-06-10T18:44:13.425693+0000","last_fullsized":"2023-06-10T18:44:13.425693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423561+0000","last_change":"2023-06-10T18:44:08.575263+0000","last_active":"2023-06-10T18:44:13.423561+0000","last_peered":"2023-06-10T18:44:13.423561+0000","last_clean":"2023-06-10T18:44:13.423561+0000","last_became_active":"2023-06-10T18:44:08.574716+0000","last_became_peered":"2023-06-10T18:44:08.574716+0000","last_unstale":"2023-06-10T18:44:13.423561+0000","last_undegraded":"2023-06-10T18:44:13.423561+0000","last_fullsized":"2023-06-10T18:44:13.423561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393754+0000","last_change":"2023-06-10T18:44:08.986065+0000","last_active":"2023-06-10T18:44:13.393754+0000","last_peered":"2023-06-10T18:44:13.393754+0000","last_clean":"2023-06-10T18:44:13.393754+0000","last_became_active":"2023-06-10T18:44:08.984998+0000","last_became_peered":"2023-06-10T18:44:08.984998+0000","last_unstale":"2023-06-10T18:44:13.393754+0000","last_undegraded":"2023-06-10T18:44:13.393754+0000","last_fullsized":"2023-06-10T18:44:13.393754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425425+0000","last_change":"2023-06-10T18:44:13.004286+0000","last_active":"2023-06-10T18:44:13.425425+0000","last_peered":"2023-06-10T18:44:13.425425+0000","last_clean":"2023-06-10T18:44:13.425425+0000","last_became_active":"2023-06-10T18:44:13.003671+0000","last_became_peered":"2023-06-10T18:44:13.003671+0000","last_unstale":"2023-06-10T18:44:13.425425+0000","last_undegraded":"2023-06-10T18:44:13.425425+0000","last_fullsized":"2023-06-10T18:44:13.425425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394170+0000","last_change":"2023-06-10T18:44:08.578231+0000","last_active":"2023-06-10T18:44:13.394170+0000","last_peered":"2023-06-10T18:44:13.394170+0000","last_clean":"2023-06-10T18:44:13.394170+0000","last_became_active":"2023-06-10T18:44:08.577966+0000","last_became_peered":"2023-06-10T18:44:08.577966+0000","last_unstale":"2023-06-10T18:44:13.394170+0000","last_undegraded":"2023-06-10T18:44:13.394170+0000","last_fullsized":"2023-06-10T18:44:13.394170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423359+0000","last_change":"2023-06-10T18:44:08.990234+0000","last_active":"2023-06-10T18:44:13.423359+0000","last_peered":"2023-06-10T18:44:13.423359+0000","last_clean":"2023-06-10T18:44:13.423359+0000","last_became_active":"2023-06-10T18:44:08.989505+0000","last_became_peered":"2023-06-10T18:44:08.989505+0000","last_unstale":"2023-06-10T18:44:13.423359+0000","last_undegraded":"2023-06-10T18:44:13.423359+0000","last_fullsized":"2023-06-10T18:44:13.423359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:31.544241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393091+0000","last_change":"2023-06-10T18:44:13.009889+0000","last_active":"2023-06-10T18:44:13.393091+0000","last_peered":"2023-06-10T18:44:13.393091+0000","last_clean":"2023-06-10T18:44:13.393091+0000","last_became_active":"2023-06-10T18:44:13.009528+0000","last_became_peered":"2023-06-10T18:44:13.009528+0000","last_unstale":"2023-06-10T18:44:13.393091+0000","last_undegraded":"2023-06-10T18:44:13.393091+0000","last_fullsized":"2023-06-10T18:44:13.393091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:15.449838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":29,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422838+0000","last_change":"2023-06-10T18:44:08.573929+0000","last_active":"2023-06-10T18:44:13.422838+0000","last_peered":"2023-06-10T18:44:13.422838+0000","last_clean":"2023-06-10T18:44:13.422838+0000","last_became_active":"2023-06-10T18:44:08.573652+0000","last_became_peered":"2023-06-10T18:44:08.573652+0000","last_unstale":"2023-06-10T18:44:13.422838+0000","last_undegraded":"2023-06-10T18:44:13.422838+0000","last_fullsized":"2023-06-10T18:44:13.422838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:08.763880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425617+0000","last_change":"2023-06-10T18:44:08.984556+0000","last_active":"2023-06-10T18:44:13.425617+0000","last_peered":"2023-06-10T18:44:13.425617+0000","last_clean":"2023-06-10T18:44:13.425617+0000","last_became_active":"2023-06-10T18:44:08.984321+0000","last_became_peered":"2023-06-10T18:44:08.984321+0000","last_unstale":"2023-06-10T18:44:13.425617+0000","last_undegraded":"2023-06-10T18:44:13.425617+0000","last_fullsized":"2023-06-10T18:44:13.425617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394464+0000","last_change":"2023-06-10T18:44:13.006271+0000","last_active":"2023-06-10T18:44:13.394464+0000","last_peered":"2023-06-10T18:44:13.394464+0000","last_clean":"2023-06-10T18:44:13.394464+0000","last_became_active":"2023-06-10T18:44:13.005878+0000","last_became_peered":"2023-06-10T18:44:13.005878+0000","last_unstale":"2023-06-10T18:44:13.394464+0000","last_undegraded":"2023-06-10T18:44:13.394464+0000","last_fullsized":"2023-06-10T18:44:13.394464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:21.046063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394016+0000","last_change":"2023-06-10T18:44:08.578331+0000","last_active":"2023-06-10T18:44:13.394016+0000","last_peered":"2023-06-10T18:44:13.394016+0000","last_clean":"2023-06-10T18:44:13.394016+0000","last_became_active":"2023-06-10T18:44:08.577846+0000","last_became_peered":"2023-06-10T18:44:08.577846+0000","last_unstale":"2023-06-10T18:44:13.394016+0000","last_undegraded":"2023-06-10T18:44:13.394016+0000","last_fullsized":"2023-06-10T18:44:13.394016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392530+0000","last_change":"2023-06-10T18:44:08.984476+0000","last_active":"2023-06-10T18:44:13.392530+0000","last_peered":"2023-06-10T18:44:13.392530+0000","last_clean":"2023-06-10T18:44:13.392530+0000","last_became_active":"2023-06-10T18:44:08.984095+0000","last_became_peered":"2023-06-10T18:44:08.984095+0000","last_unstale":"2023-06-10T18:44:13.392530+0000","last_undegraded":"2023-06-10T18:44:13.392530+0000","last_fullsized":"2023-06-10T18:44:13.392530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422417+0000","last_change":"2023-06-10T18:44:13.006928+0000","last_active":"2023-06-10T18:44:13.422417+0000","last_peered":"2023-06-10T18:44:13.422417+0000","last_clean":"2023-06-10T18:44:13.422417+0000","last_became_active":"2023-06-10T18:44:13.006526+0000","last_became_peered":"2023-06-10T18:44:13.006526+0000","last_unstale":"2023-06-10T18:44:13.422417+0000","last_undegraded":"2023-06-10T18:44:13.422417+0000","last_fullsized":"2023-06-10T18:44:13.422417+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:42:15.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393898+0000","last_change":"2023-06-10T18:44:08.580704+0000","last_active":"2023-06-10T18:44:13.393898+0000","last_peered":"2023-06-10T18:44:13.393898+0000","last_clean":"2023-06-10T18:44:13.393898+0000","last_became_active":"2023-06-10T18:44:08.580413+0000","last_became_peered":"2023-06-10T18:44:08.580413+0000","last_unstale":"2023-06-10T18:44:13.393898+0000","last_undegraded":"2023-06-10T18:44:13.393898+0000","last_fullsized":"2023-06-10T18:44:13.393898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:29.052625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423333+0000","last_change":"2023-06-10T18:44:08.988350+0000","last_active":"2023-06-10T18:44:13.423333+0000","last_peered":"2023-06-10T18:44:13.423333+0000","last_clean":"2023-06-10T18:44:13.423333+0000","last_became_active":"2023-06-10T18:44:08.988073+0000","last_became_peered":"2023-06-10T18:44:08.988073+0000","last_unstale":"2023-06-10T18:44:13.423333+0000","last_undegraded":"2023-06-10T18:44:13.423333+0000","last_fullsized":"2023-06-10T18:44:13.423333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422684+0000","last_change":"2023-06-10T18:44:13.001742+0000","last_active":"2023-06-10T18:44:13.422684+0000","last_peered":"2023-06-10T18:44:13.422684+0000","last_clean":"2023-06-10T18:44:13.422684+0000","last_became_active":"2023-06-10T18:44:13.001373+0000","last_became_peered":"2023-06-10T18:44:13.001373+0000","last_unstale":"2023-06-10T18:44:13.422684+0000","last_undegraded":"2023-06-10T18:44:13.422684+0000","last_fullsized":"2023-06-10T18:44:13.422684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:11.616196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394281+0000","last_change":"2023-06-10T18:44:08.574389+0000","last_active":"2023-06-10T18:44:13.394281+0000","last_peered":"2023-06-10T18:44:13.394281+0000","last_clean":"2023-06-10T18:44:13.394281+0000","last_became_active":"2023-06-10T18:44:08.574056+0000","last_became_peered":"2023-06-10T18:44:08.574056+0000","last_unstale":"2023-06-10T18:44:13.394281+0000","last_undegraded":"2023-06-10T18:44:13.394281+0000","last_fullsized":"2023-06-10T18:44:13.394281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423143+0000","last_change":"2023-06-10T18:44:08.987503+0000","last_active":"2023-06-10T18:44:13.423143+0000","last_peered":"2023-06-10T18:44:13.423143+0000","last_clean":"2023-06-10T18:44:13.423143+0000","last_became_active":"2023-06-10T18:44:08.987204+0000","last_became_peered":"2023-06-10T18:44:08.987204+0000","last_unstale":"2023-06-10T18:44:13.423143+0000","last_undegraded":"2023-06-10T18:44:13.423143+0000","last_fullsized":"2023-06-10T18:44:13.423143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:45.870218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422702+0000","last_change":"2023-06-10T18:44:13.003895+0000","last_active":"2023-06-10T18:44:13.422702+0000","last_peered":"2023-06-10T18:44:13.422702+0000","last_clean":"2023-06-10T18:44:13.422702+0000","last_became_active":"2023-06-10T18:44:13.001542+0000","last_became_peered":"2023-06-10T18:44:13.001542+0000","last_unstale":"2023-06-10T18:44:13.422702+0000","last_undegraded":"2023-06-10T18:44:13.422702+0000","last_fullsized":"2023-06-10T18:44:13.422702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:54:25.091621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393042+0000","last_change":"2023-06-10T18:44:08.580607+0000","last_active":"2023-06-10T18:44:13.393042+0000","last_peered":"2023-06-10T18:44:13.393042+0000","last_clean":"2023-06-10T18:44:13.393042+0000","last_became_active":"2023-06-10T18:44:08.580289+0000","last_became_peered":"2023-06-10T18:44:08.580289+0000","last_unstale":"2023-06-10T18:44:13.393042+0000","last_undegraded":"2023-06-10T18:44:13.393042+0000","last_fullsized":"2023-06-10T18:44:13.393042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:09.158752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424116+0000","last_change":"2023-06-10T18:44:08.987624+0000","last_active":"2023-06-10T18:44:13.424116+0000","last_peered":"2023-06-10T18:44:13.424116+0000","last_clean":"2023-06-10T18:44:13.424116+0000","last_became_active":"2023-06-10T18:44:08.987375+0000","last_became_peered":"2023-06-10T18:44:08.987375+0000","last_unstale":"2023-06-10T18:44:13.424116+0000","last_undegraded":"2023-06-10T18:44:13.424116+0000","last_fullsized":"2023-06-10T18:44:13.424116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422769+0000","last_change":"2023-06-10T18:44:13.002722+0000","last_active":"2023-06-10T18:44:13.422769+0000","last_peered":"2023-06-10T18:44:13.422769+0000","last_clean":"2023-06-10T18:44:13.422769+0000","last_became_active":"2023-06-10T18:44:13.002214+0000","last_became_peered":"2023-06-10T18:44:13.002214+0000","last_unstale":"2023-06-10T18:44:13.422769+0000","last_undegraded":"2023-06-10T18:44:13.422769+0000","last_fullsized":"2023-06-10T18:44:13.422769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:04:35.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425192+0000","last_change":"2023-06-10T18:44:08.578128+0000","last_active":"2023-06-10T18:44:13.425192+0000","last_peered":"2023-06-10T18:44:13.425192+0000","last_clean":"2023-06-10T18:44:13.425192+0000","last_became_active":"2023-06-10T18:44:08.577863+0000","last_became_peered":"2023-06-10T18:44:08.577863+0000","last_unstale":"2023-06-10T18:44:13.425192+0000","last_undegraded":"2023-06-10T18:44:13.425192+0000","last_fullsized":"2023-06-10T18:44:13.425192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423537+0000","last_change":"2023-06-10T18:44:08.995303+0000","last_active":"2023-06-10T18:44:13.423537+0000","last_peered":"2023-06-10T18:44:13.423537+0000","last_clean":"2023-06-10T18:44:13.423537+0000","last_became_active":"2023-06-10T18:44:08.995025+0000","last_became_peered":"2023-06-10T18:44:08.995025+0000","last_unstale":"2023-06-10T18:44:13.423537+0000","last_undegraded":"2023-06-10T18:44:13.423537+0000","last_fullsized":"2023-06-10T18:44:13.423537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394147+0000","last_change":"2023-06-10T18:44:13.008926+0000","last_active":"2023-06-10T18:44:13.394147+0000","last_peered":"2023-06-10T18:44:13.394147+0000","last_clean":"2023-06-10T18:44:13.394147+0000","last_became_active":"2023-06-10T18:44:13.008249+0000","last_became_peered":"2023-06-10T18:44:13.008249+0000","last_unstale":"2023-06-10T18:44:13.394147+0000","last_undegraded":"2023-06-10T18:44:13.394147+0000","last_fullsized":"2023-06-10T18:44:13.394147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423270+0000","last_change":"2023-06-10T18:44:08.575472+0000","last_active":"2023-06-10T18:44:13.423270+0000","last_peered":"2023-06-10T18:44:13.423270+0000","last_clean":"2023-06-10T18:44:13.423270+0000","last_became_active":"2023-06-10T18:44:08.575186+0000","last_became_peered":"2023-06-10T18:44:08.575186+0000","last_unstale":"2023-06-10T18:44:13.423270+0000","last_undegraded":"2023-06-10T18:44:13.423270+0000","last_fullsized":"2023-06-10T18:44:13.423270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:33.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423243+0000","last_change":"2023-06-10T18:44:08.987175+0000","last_active":"2023-06-10T18:44:13.423243+0000","last_peered":"2023-06-10T18:44:13.423243+0000","last_clean":"2023-06-10T18:44:13.423243+0000","last_became_active":"2023-06-10T18:44:08.986395+0000","last_became_peered":"2023-06-10T18:44:08.986395+0000","last_unstale":"2023-06-10T18:44:13.423243+0000","last_undegraded":"2023-06-10T18:44:13.423243+0000","last_fullsized":"2023-06-10T18:44:13.423243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393572+0000","last_change":"2023-06-10T18:44:13.002535+0000","last_active":"2023-06-10T18:44:13.393572+0000","last_peered":"2023-06-10T18:44:13.393572+0000","last_clean":"2023-06-10T18:44:13.393572+0000","last_became_active":"2023-06-10T18:44:13.002065+0000","last_became_peered":"2023-06-10T18:44:13.002065+0000","last_unstale":"2023-06-10T18:44:13.393572+0000","last_undegraded":"2023-06-10T18:44:13.393572+0000","last_fullsized":"2023-06-10T18:44:13.393572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:56.164577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423083+0000","last_change":"2023-06-10T18:44:08.572499+0000","last_active":"2023-06-10T18:44:13.423083+0000","last_peered":"2023-06-10T18:44:13.423083+0000","last_clean":"2023-06-10T18:44:13.423083+0000","last_became_active":"2023-06-10T18:44:08.572196+0000","last_became_peered":"2023-06-10T18:44:08.572196+0000","last_unstale":"2023-06-10T18:44:13.423083+0000","last_undegraded":"2023-06-10T18:44:13.423083+0000","last_fullsized":"2023-06-10T18:44:13.423083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423986+0000","last_change":"2023-06-10T18:44:08.989104+0000","last_active":"2023-06-10T18:44:13.423986+0000","last_peered":"2023-06-10T18:44:13.423986+0000","last_clean":"2023-06-10T18:44:13.423986+0000","last_became_active":"2023-06-10T18:44:08.988824+0000","last_became_peered":"2023-06-10T18:44:08.988824+0000","last_unstale":"2023-06-10T18:44:13.423986+0000","last_undegraded":"2023-06-10T18:44:13.423986+0000","last_fullsized":"2023-06-10T18:44:13.423986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:19.807436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423861+0000","last_change":"2023-06-10T18:44:13.006826+0000","last_active":"2023-06-10T18:44:13.423861+0000","last_peered":"2023-06-10T18:44:13.423861+0000","last_clean":"2023-06-10T18:44:13.423861+0000","last_became_active":"2023-06-10T18:44:13.006458+0000","last_became_peered":"2023-06-10T18:44:13.006458+0000","last_unstale":"2023-06-10T18:44:13.423861+0000","last_undegraded":"2023-06-10T18:44:13.423861+0000","last_fullsized":"2023-06-10T18:44:13.423861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.577680+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.577419+0000","last_became_peered":"2023-06-10T18:44:08.577419+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394577+0000","last_change":"2023-06-10T18:44:08.991268+0000","last_active":"2023-06-10T18:44:13.394577+0000","last_peered":"2023-06-10T18:44:13.394577+0000","last_clean":"2023-06-10T18:44:13.394577+0000","last_became_active":"2023-06-10T18:44:08.990999+0000","last_became_peered":"2023-06-10T18:44:08.990999+0000","last_unstale":"2023-06-10T18:44:13.394577+0000","last_undegraded":"2023-06-10T18:44:13.394577+0000","last_fullsized":"2023-06-10T18:44:13.394577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423567+0000","last_change":"2023-06-10T18:44:13.010490+0000","last_active":"2023-06-10T18:44:13.423567+0000","last_peered":"2023-06-10T18:44:13.423567+0000","last_clean":"2023-06-10T18:44:13.423567+0000","last_became_active":"2023-06-10T18:44:13.010098+0000","last_became_peered":"2023-06-10T18:44:13.010098+0000","last_unstale":"2023-06-10T18:44:13.423567+0000","last_undegraded":"2023-06-10T18:44:13.423567+0000","last_fullsized":"2023-06-10T18:44:13.423567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:58.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394227+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:44:13.394227+0000","last_peered":"2023-06-10T18:44:13.394227+0000","last_clean":"2023-06-10T18:44:13.394227+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:44:13.394227+0000","last_undegraded":"2023-06-10T18:44:13.394227+0000","last_fullsized":"2023-06-10T18:44:13.394227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393285+0000","last_change":"2023-06-10T18:44:08.577289+0000","last_active":"2023-06-10T18:44:13.393285+0000","last_peered":"2023-06-10T18:44:13.393285+0000","last_clean":"2023-06-10T18:44:13.393285+0000","last_became_active":"2023-06-10T18:44:08.577021+0000","last_became_peered":"2023-06-10T18:44:08.577021+0000","last_unstale":"2023-06-10T18:44:13.393285+0000","last_undegraded":"2023-06-10T18:44:13.393285+0000","last_fullsized":"2023-06-10T18:44:13.393285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393847+0000","last_change":"2023-06-10T18:44:08.983928+0000","last_active":"2023-06-10T18:44:13.393847+0000","last_peered":"2023-06-10T18:44:13.393847+0000","last_clean":"2023-06-10T18:44:13.393847+0000","last_became_active":"2023-06-10T18:44:08.983562+0000","last_became_peered":"2023-06-10T18:44:08.983562+0000","last_unstale":"2023-06-10T18:44:13.393847+0000","last_undegraded":"2023-06-10T18:44:13.393847+0000","last_fullsized":"2023-06-10T18:44:13.393847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423202+0000","last_change":"2023-06-10T18:44:13.006596+0000","last_active":"2023-06-10T18:44:13.423202+0000","last_peered":"2023-06-10T18:44:13.423202+0000","last_clean":"2023-06-10T18:44:13.423202+0000","last_became_active":"2023-06-10T18:44:13.006258+0000","last_became_peered":"2023-06-10T18:44:13.006258+0000","last_unstale":"2023-06-10T18:44:13.423202+0000","last_undegraded":"2023-06-10T18:44:13.423202+0000","last_fullsized":"2023-06-10T18:44:13.423202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:31.503221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393245+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:44:13.393245+0000","last_peered":"2023-06-10T18:44:13.393245+0000","last_clean":"2023-06-10T18:44:13.393245+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:44:13.393245+0000","last_undegraded":"2023-06-10T18:44:13.393245+0000","last_fullsized":"2023-06-10T18:44:13.393245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:28.260256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424001+0000","last_change":"2023-06-10T18:44:08.578515+0000","last_active":"2023-06-10T18:44:13.424001+0000","last_peered":"2023-06-10T18:44:13.424001+0000","last_clean":"2023-06-10T18:44:13.424001+0000","last_became_active":"2023-06-10T18:44:08.578184+0000","last_became_peered":"2023-06-10T18:44:08.578184+0000","last_unstale":"2023-06-10T18:44:13.424001+0000","last_undegraded":"2023-06-10T18:44:13.424001+0000","last_fullsized":"2023-06-10T18:44:13.424001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394190+0000","last_change":"2023-06-10T18:44:08.985054+0000","last_active":"2023-06-10T18:44:13.394190+0000","last_peered":"2023-06-10T18:44:13.394190+0000","last_clean":"2023-06-10T18:44:13.394190+0000","last_became_active":"2023-06-10T18:44:08.984698+0000","last_became_peered":"2023-06-10T18:44:08.984698+0000","last_unstale":"2023-06-10T18:44:13.394190+0000","last_undegraded":"2023-06-10T18:44:13.394190+0000","last_fullsized":"2023-06-10T18:44:13.394190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423677+0000","last_change":"2023-06-10T18:44:13.002621+0000","last_active":"2023-06-10T18:44:13.423677+0000","last_peered":"2023-06-10T18:44:13.423677+0000","last_clean":"2023-06-10T18:44:13.423677+0000","last_became_active":"2023-06-10T18:44:13.002229+0000","last_became_peered":"2023-06-10T18:44:13.002229+0000","last_unstale":"2023-06-10T18:44:13.423677+0000","last_undegraded":"2023-06-10T18:44:13.423677+0000","last_fullsized":"2023-06-10T18:44:13.423677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:44:56.053966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422499+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:44:13.422499+0000","last_peered":"2023-06-10T18:44:13.422499+0000","last_clean":"2023-06-10T18:44:13.422499+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:44:13.422499+0000","last_undegraded":"2023-06-10T18:44:13.422499+0000","last_fullsized":"2023-06-10T18:44:13.422499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:49:35.702822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393742+0000","last_change":"2023-06-10T18:44:08.571468+0000","last_active":"2023-06-10T18:44:13.393742+0000","last_peered":"2023-06-10T18:44:13.393742+0000","last_clean":"2023-06-10T18:44:13.393742+0000","last_became_active":"2023-06-10T18:44:08.570651+0000","last_became_peered":"2023-06-10T18:44:08.570651+0000","last_unstale":"2023-06-10T18:44:13.393742+0000","last_undegraded":"2023-06-10T18:44:13.393742+0000","last_fullsized":"2023-06-10T18:44:13.393742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394594+0000","last_change":"2023-06-10T18:44:08.983687+0000","last_active":"2023-06-10T18:44:13.394594+0000","last_peered":"2023-06-10T18:44:13.394594+0000","last_clean":"2023-06-10T18:44:13.394594+0000","last_became_active":"2023-06-10T18:44:08.983381+0000","last_became_peered":"2023-06-10T18:44:08.983381+0000","last_unstale":"2023-06-10T18:44:13.394594+0000","last_undegraded":"2023-06-10T18:44:13.394594+0000","last_fullsized":"2023-06-10T18:44:13.394594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393971+0000","last_change":"2023-06-10T18:44:13.009660+0000","last_active":"2023-06-10T18:44:13.393971+0000","last_peered":"2023-06-10T18:44:13.393971+0000","last_clean":"2023-06-10T18:44:13.393971+0000","last_became_active":"2023-06-10T18:44:13.009208+0000","last_became_peered":"2023-06-10T18:44:13.009208+0000","last_unstale":"2023-06-10T18:44:13.393971+0000","last_undegraded":"2023-06-10T18:44:13.393971+0000","last_fullsized":"2023-06-10T18:44:13.393971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392956+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:44:13.392956+0000","last_peered":"2023-06-10T18:44:13.392956+0000","last_clean":"2023-06-10T18:44:13.392956+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:44:13.392956+0000","last_undegraded":"2023-06-10T18:44:13.392956+0000","last_fullsized":"2023-06-10T18:44:13.392956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":183,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394153+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:44:13.394153+0000","last_peered":"2023-06-10T18:44:13.394153+0000","last_clean":"2023-06-10T18:44:13.394153+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:44:13.394153+0000","last_undegraded":"2023-06-10T18:44:13.394153+0000","last_fullsized":"2023-06-10T18:44:13.394153+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:22.105411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425014+0000","last_change":"2023-06-10T18:44:08.576254+0000","last_active":"2023-06-10T18:44:13.425014+0000","last_peered":"2023-06-10T18:44:13.425014+0000","last_clean":"2023-06-10T18:44:13.425014+0000","last_became_active":"2023-06-10T18:44:08.575980+0000","last_became_peered":"2023-06-10T18:44:08.575980+0000","last_unstale":"2023-06-10T18:44:13.425014+0000","last_undegraded":"2023-06-10T18:44:13.425014+0000","last_fullsized":"2023-06-10T18:44:13.425014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423295+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.423295+0000","last_peered":"2023-06-10T18:44:13.423295+0000","last_clean":"2023-06-10T18:44:13.423295+0000","last_became_active":"2023-06-10T18:44:08.988159+0000","last_became_peered":"2023-06-10T18:44:08.988159+0000","last_unstale":"2023-06-10T18:44:13.423295+0000","last_undegraded":"2023-06-10T18:44:13.423295+0000","last_fullsized":"2023-06-10T18:44:13.423295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:55.371558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393218+0000","last_change":"2023-06-10T18:44:13.006561+0000","last_active":"2023-06-10T18:44:13.393218+0000","last_peered":"2023-06-10T18:44:13.393218+0000","last_clean":"2023-06-10T18:44:13.393218+0000","last_became_active":"2023-06-10T18:44:13.006171+0000","last_became_peered":"2023-06-10T18:44:13.006171+0000","last_unstale":"2023-06-10T18:44:13.393218+0000","last_undegraded":"2023-06-10T18:44:13.393218+0000","last_fullsized":"2023-06-10T18:44:13.393218+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:18.821445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394377+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:44:13.394377+0000","last_peered":"2023-06-10T18:44:13.394377+0000","last_clean":"2023-06-10T18:44:13.394377+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:44:13.394377+0000","last_undegraded":"2023-06-10T18:44:13.394377+0000","last_fullsized":"2023-06-10T18:44:13.394377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:21.913689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423028+0000","last_change":"2023-06-10T18:44:08.578010+0000","last_active":"2023-06-10T18:44:13.423028+0000","last_peered":"2023-06-10T18:44:13.423028+0000","last_clean":"2023-06-10T18:44:13.423028+0000","last_became_active":"2023-06-10T18:44:08.577727+0000","last_became_peered":"2023-06-10T18:44:08.577727+0000","last_unstale":"2023-06-10T18:44:13.423028+0000","last_undegraded":"2023-06-10T18:44:13.423028+0000","last_fullsized":"2023-06-10T18:44:13.423028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:00:49.837648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423225+0000","last_change":"2023-06-10T18:44:08.990076+0000","last_active":"2023-06-10T18:44:13.423225+0000","last_peered":"2023-06-10T18:44:13.423225+0000","last_clean":"2023-06-10T18:44:13.423225+0000","last_became_active":"2023-06-10T18:44:08.989191+0000","last_became_peered":"2023-06-10T18:44:08.989191+0000","last_unstale":"2023-06-10T18:44:13.423225+0000","last_undegraded":"2023-06-10T18:44:13.423225+0000","last_fullsized":"2023-06-10T18:44:13.423225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:35.616295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423712+0000","last_change":"2023-06-10T18:44:12.999386+0000","last_active":"2023-06-10T18:44:13.423712+0000","last_peered":"2023-06-10T18:44:13.423712+0000","last_clean":"2023-06-10T18:44:13.423712+0000","last_became_active":"2023-06-10T18:44:12.998986+0000","last_became_peered":"2023-06-10T18:44:12.998986+0000","last_unstale":"2023-06-10T18:44:13.423712+0000","last_undegraded":"2023-06-10T18:44:13.423712+0000","last_fullsized":"2023-06-10T18:44:13.423712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:10.294554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394138+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:44:13.394138+0000","last_peered":"2023-06-10T18:44:13.394138+0000","last_clean":"2023-06-10T18:44:13.394138+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:44:13.394138+0000","last_undegraded":"2023-06-10T18:44:13.394138+0000","last_fullsized":"2023-06-10T18:44:13.394138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:43.350286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394357+0000","last_change":"2023-06-10T18:44:08.572021+0000","last_active":"2023-06-10T18:44:13.394357+0000","last_peered":"2023-06-10T18:44:13.394357+0000","last_clean":"2023-06-10T18:44:13.394357+0000","last_became_active":"2023-06-10T18:44:08.571716+0000","last_became_peered":"2023-06-10T18:44:08.571716+0000","last_unstale":"2023-06-10T18:44:13.394357+0000","last_undegraded":"2023-06-10T18:44:13.394357+0000","last_fullsized":"2023-06-10T18:44:13.394357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423849+0000","last_change":"2023-06-10T18:44:08.983810+0000","last_active":"2023-06-10T18:44:13.423849+0000","last_peered":"2023-06-10T18:44:13.423849+0000","last_clean":"2023-06-10T18:44:13.423849+0000","last_became_active":"2023-06-10T18:44:08.983539+0000","last_became_peered":"2023-06-10T18:44:08.983539+0000","last_unstale":"2023-06-10T18:44:13.423849+0000","last_undegraded":"2023-06-10T18:44:13.423849+0000","last_fullsized":"2023-06-10T18:44:13.423849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423587+0000","last_change":"2023-06-10T18:44:13.000564+0000","last_active":"2023-06-10T18:44:13.423587+0000","last_peered":"2023-06-10T18:44:13.423587+0000","last_clean":"2023-06-10T18:44:13.423587+0000","last_became_active":"2023-06-10T18:44:13.000141+0000","last_became_peered":"2023-06-10T18:44:13.000141+0000","last_unstale":"2023-06-10T18:44:13.423587+0000","last_undegraded":"2023-06-10T18:44:13.423587+0000","last_fullsized":"2023-06-10T18:44:13.423587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:45.731880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425478+0000","last_change":"2023-06-10T18:44:08.579728+0000","last_active":"2023-06-10T18:44:13.425478+0000","last_peered":"2023-06-10T18:44:13.425478+0000","last_clean":"2023-06-10T18:44:13.425478+0000","last_became_active":"2023-06-10T18:44:08.579431+0000","last_became_peered":"2023-06-10T18:44:08.579431+0000","last_unstale":"2023-06-10T18:44:13.425478+0000","last_undegraded":"2023-06-10T18:44:13.425478+0000","last_fullsized":"2023-06-10T18:44:13.425478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394345+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:44:13.394345+0000","last_peered":"2023-06-10T18:44:13.394345+0000","last_clean":"2023-06-10T18:44:13.394345+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:44:13.394345+0000","last_undegraded":"2023-06-10T18:44:13.394345+0000","last_fullsized":"2023-06-10T18:44:13.394345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:05.996450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424774+0000","last_change":"2023-06-10T18:44:08.983770+0000","last_active":"2023-06-10T18:44:13.424774+0000","last_peered":"2023-06-10T18:44:13.424774+0000","last_clean":"2023-06-10T18:44:13.424774+0000","last_became_active":"2023-06-10T18:44:08.983526+0000","last_became_peered":"2023-06-10T18:44:08.983526+0000","last_unstale":"2023-06-10T18:44:13.424774+0000","last_undegraded":"2023-06-10T18:44:13.424774+0000","last_fullsized":"2023-06-10T18:44:13.424774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424718+0000","last_change":"2023-06-10T18:44:13.003992+0000","last_active":"2023-06-10T18:44:13.424718+0000","last_peered":"2023-06-10T18:44:13.424718+0000","last_clean":"2023-06-10T18:44:13.424718+0000","last_became_active":"2023-06-10T18:44:13.003565+0000","last_became_peered":"2023-06-10T18:44:13.003565+0000","last_unstale":"2023-06-10T18:44:13.424718+0000","last_undegraded":"2023-06-10T18:44:13.424718+0000","last_fullsized":"2023-06-10T18:44:13.424718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423787+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:44:13.423787+0000","last_peered":"2023-06-10T18:44:13.423787+0000","last_clean":"2023-06-10T18:44:13.423787+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:44:13.423787+0000","last_undegraded":"2023-06-10T18:44:13.423787+0000","last_fullsized":"2023-06-10T18:44:13.423787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:03:46.107219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423014+0000","last_change":"2023-06-10T18:44:08.576079+0000","last_active":"2023-06-10T18:44:13.423014+0000","last_peered":"2023-06-10T18:44:13.423014+0000","last_clean":"2023-06-10T18:44:13.423014+0000","last_became_active":"2023-06-10T18:44:08.575346+0000","last_became_peered":"2023-06-10T18:44:08.575346+0000","last_unstale":"2023-06-10T18:44:13.423014+0000","last_undegraded":"2023-06-10T18:44:13.423014+0000","last_fullsized":"2023-06-10T18:44:13.423014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:54:23.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423207+0000","last_change":"2023-06-10T18:44:08.990722+0000","last_active":"2023-06-10T18:44:13.423207+0000","last_peered":"2023-06-10T18:44:13.423207+0000","last_clean":"2023-06-10T18:44:13.423207+0000","last_became_active":"2023-06-10T18:44:08.988467+0000","last_became_peered":"2023-06-10T18:44:08.988467+0000","last_unstale":"2023-06-10T18:44:13.423207+0000","last_undegraded":"2023-06-10T18:44:13.423207+0000","last_fullsized":"2023-06-10T18:44:13.423207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394392+0000","last_change":"2023-06-10T18:44:13.008834+0000","last_active":"2023-06-10T18:44:13.394392+0000","last_peered":"2023-06-10T18:44:13.394392+0000","last_clean":"2023-06-10T18:44:13.394392+0000","last_became_active":"2023-06-10T18:44:13.008433+0000","last_became_peered":"2023-06-10T18:44:13.008433+0000","last_unstale":"2023-06-10T18:44:13.394392+0000","last_undegraded":"2023-06-10T18:44:13.394392+0000","last_fullsized":"2023-06-10T18:44:13.394392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423733+0000","last_change":"2023-06-10T18:44:08.576600+0000","last_active":"2023-06-10T18:44:13.423733+0000","last_peered":"2023-06-10T18:44:13.423733+0000","last_clean":"2023-06-10T18:44:13.423733+0000","last_became_active":"2023-06-10T18:44:08.575635+0000","last_became_peered":"2023-06-10T18:44:08.575635+0000","last_unstale":"2023-06-10T18:44:13.423733+0000","last_undegraded":"2023-06-10T18:44:13.423733+0000","last_fullsized":"2023-06-10T18:44:13.423733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423305+0000","last_change":"2023-06-10T18:44:08.985629+0000","last_active":"2023-06-10T18:44:13.423305+0000","last_peered":"2023-06-10T18:44:13.423305+0000","last_clean":"2023-06-10T18:44:13.423305+0000","last_became_active":"2023-06-10T18:44:08.985367+0000","last_became_peered":"2023-06-10T18:44:08.985367+0000","last_unstale":"2023-06-10T18:44:13.423305+0000","last_undegraded":"2023-06-10T18:44:13.423305+0000","last_fullsized":"2023-06-10T18:44:13.423305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:57.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394246+0000","last_change":"2023-06-10T18:44:13.010558+0000","last_active":"2023-06-10T18:44:13.394246+0000","last_peered":"2023-06-10T18:44:13.394246+0000","last_clean":"2023-06-10T18:44:13.394246+0000","last_became_active":"2023-06-10T18:44:13.010123+0000","last_became_peered":"2023-06-10T18:44:13.010123+0000","last_unstale":"2023-06-10T18:44:13.394246+0000","last_undegraded":"2023-06-10T18:44:13.394246+0000","last_fullsized":"2023-06-10T18:44:13.394246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:13.872743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393194+0000","last_change":"2023-06-10T18:44:08.571654+0000","last_active":"2023-06-10T18:44:13.393194+0000","last_peered":"2023-06-10T18:44:13.393194+0000","last_clean":"2023-06-10T18:44:13.393194+0000","last_became_active":"2023-06-10T18:44:08.571363+0000","last_became_peered":"2023-06-10T18:44:08.571363+0000","last_unstale":"2023-06-10T18:44:13.393194+0000","last_undegraded":"2023-06-10T18:44:13.393194+0000","last_fullsized":"2023-06-10T18:44:13.393194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:48.376074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423819+0000","last_change":"2023-06-10T18:44:08.996056+0000","last_active":"2023-06-10T18:44:13.423819+0000","last_peered":"2023-06-10T18:44:13.423819+0000","last_clean":"2023-06-10T18:44:13.423819+0000","last_became_active":"2023-06-10T18:44:08.995902+0000","last_became_peered":"2023-06-10T18:44:08.995902+0000","last_unstale":"2023-06-10T18:44:13.423819+0000","last_undegraded":"2023-06-10T18:44:13.423819+0000","last_fullsized":"2023-06-10T18:44:13.423819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393057+0000","last_change":"2023-06-10T18:44:13.004134+0000","last_active":"2023-06-10T18:44:13.393057+0000","last_peered":"2023-06-10T18:44:13.393057+0000","last_clean":"2023-06-10T18:44:13.393057+0000","last_became_active":"2023-06-10T18:44:13.003596+0000","last_became_peered":"2023-06-10T18:44:13.003596+0000","last_unstale":"2023-06-10T18:44:13.393057+0000","last_undegraded":"2023-06-10T18:44:13.393057+0000","last_fullsized":"2023-06-10T18:44:13.393057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393105+0000","last_change":"2023-06-10T18:44:08.575332+0000","last_active":"2023-06-10T18:44:13.393105+0000","last_peered":"2023-06-10T18:44:13.393105+0000","last_clean":"2023-06-10T18:44:13.393105+0000","last_became_active":"2023-06-10T18:44:08.575065+0000","last_became_peered":"2023-06-10T18:44:08.575065+0000","last_unstale":"2023-06-10T18:44:13.393105+0000","last_undegraded":"2023-06-10T18:44:13.393105+0000","last_fullsized":"2023-06-10T18:44:13.393105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424331+0000","last_change":"2023-06-10T18:44:08.995435+0000","last_active":"2023-06-10T18:44:13.424331+0000","last_peered":"2023-06-10T18:44:13.424331+0000","last_clean":"2023-06-10T18:44:13.424331+0000","last_became_active":"2023-06-10T18:44:08.995141+0000","last_became_peered":"2023-06-10T18:44:08.995141+0000","last_unstale":"2023-06-10T18:44:13.424331+0000","last_undegraded":"2023-06-10T18:44:13.424331+0000","last_fullsized":"2023-06-10T18:44:13.424331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:52.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422884+0000","last_change":"2023-06-10T18:44:12.998566+0000","last_active":"2023-06-10T18:44:13.422884+0000","last_peered":"2023-06-10T18:44:13.422884+0000","last_clean":"2023-06-10T18:44:13.422884+0000","last_became_active":"2023-06-10T18:44:12.998185+0000","last_became_peered":"2023-06-10T18:44:12.998185+0000","last_unstale":"2023-06-10T18:44:13.422884+0000","last_undegraded":"2023-06-10T18:44:13.422884+0000","last_fullsized":"2023-06-10T18:44:13.422884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425429+0000","last_change":"2023-06-10T18:44:08.576599+0000","last_active":"2023-06-10T18:44:13.425429+0000","last_peered":"2023-06-10T18:44:13.425429+0000","last_clean":"2023-06-10T18:44:13.425429+0000","last_became_active":"2023-06-10T18:44:08.576329+0000","last_became_peered":"2023-06-10T18:44:08.576329+0000","last_unstale":"2023-06-10T18:44:13.425429+0000","last_undegraded":"2023-06-10T18:44:13.425429+0000","last_fullsized":"2023-06-10T18:44:13.425429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:16.618462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424932+0000","last_change":"2023-06-10T18:44:08.975414+0000","last_active":"2023-06-10T18:44:13.424932+0000","last_peered":"2023-06-10T18:44:13.424932+0000","last_clean":"2023-06-10T18:44:13.424932+0000","last_became_active":"2023-06-10T18:44:08.975156+0000","last_became_peered":"2023-06-10T18:44:08.975156+0000","last_unstale":"2023-06-10T18:44:13.424932+0000","last_undegraded":"2023-06-10T18:44:13.424932+0000","last_fullsized":"2023-06-10T18:44:13.424932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393972+0000","last_change":"2023-06-10T18:44:13.007025+0000","last_active":"2023-06-10T18:44:13.393972+0000","last_peered":"2023-06-10T18:44:13.393972+0000","last_clean":"2023-06-10T18:44:13.393972+0000","last_became_active":"2023-06-10T18:44:13.006654+0000","last_became_peered":"2023-06-10T18:44:13.006654+0000","last_unstale":"2023-06-10T18:44:13.393972+0000","last_undegraded":"2023-06-10T18:44:13.393972+0000","last_fullsized":"2023-06-10T18:44:13.393972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:27.677403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423673+0000","last_change":"2023-06-10T18:44:08.575515+0000","last_active":"2023-06-10T18:44:13.423673+0000","last_peered":"2023-06-10T18:44:13.423673+0000","last_clean":"2023-06-10T18:44:13.423673+0000","last_became_active":"2023-06-10T18:44:08.573539+0000","last_became_peered":"2023-06-10T18:44:08.573539+0000","last_unstale":"2023-06-10T18:44:13.423673+0000","last_undegraded":"2023-06-10T18:44:13.423673+0000","last_fullsized":"2023-06-10T18:44:13.423673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:57.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422973+0000","last_change":"2023-06-10T18:44:08.993519+0000","last_active":"2023-06-10T18:44:13.422973+0000","last_peered":"2023-06-10T18:44:13.422973+0000","last_clean":"2023-06-10T18:44:13.422973+0000","last_became_active":"2023-06-10T18:44:08.993146+0000","last_became_peered":"2023-06-10T18:44:08.993146+0000","last_unstale":"2023-06-10T18:44:13.422973+0000","last_undegraded":"2023-06-10T18:44:13.422973+0000","last_fullsized":"2023-06-10T18:44:13.422973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422934+0000","last_change":"2023-06-10T18:44:08.989855+0000","last_active":"2023-06-10T18:44:13.422934+0000","last_peered":"2023-06-10T18:44:13.422934+0000","last_clean":"2023-06-10T18:44:13.422934+0000","last_became_active":"2023-06-10T18:44:08.989140+0000","last_became_peered":"2023-06-10T18:44:08.989140+0000","last_unstale":"2023-06-10T18:44:13.422934+0000","last_undegraded":"2023-06-10T18:44:13.422934+0000","last_fullsized":"2023-06-10T18:44:13.422934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:14.496637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425109+0000","last_change":"2023-06-10T18:44:08.992461+0000","last_active":"2023-06-10T18:44:13.425109+0000","last_peered":"2023-06-10T18:44:13.425109+0000","last_clean":"2023-06-10T18:44:13.425109+0000","last_became_active":"2023-06-10T18:44:08.992160+0000","last_became_peered":"2023-06-10T18:44:08.992160+0000","last_unstale":"2023-06-10T18:44:13.425109+0000","last_undegraded":"2023-06-10T18:44:13.425109+0000","last_fullsized":"2023-06-10T18:44:13.425109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:42.310678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423600+0000","last_change":"2023-06-10T18:44:08.991382+0000","last_active":"2023-06-10T18:44:13.423600+0000","last_peered":"2023-06-10T18:44:13.423600+0000","last_clean":"2023-06-10T18:44:13.423600+0000","last_became_active":"2023-06-10T18:44:08.991122+0000","last_became_peered":"2023-06-10T18:44:08.991122+0000","last_unstale":"2023-06-10T18:44:13.423600+0000","last_undegraded":"2023-06-10T18:44:13.423600+0000","last_fullsized":"2023-06-10T18:44:13.423600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:10.451541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425330+0000","last_change":"2023-06-10T18:44:08.987112+0000","last_active":"2023-06-10T18:44:13.425330+0000","last_peered":"2023-06-10T18:44:13.425330+0000","last_clean":"2023-06-10T18:44:13.425330+0000","last_became_active":"2023-06-10T18:44:08.986804+0000","last_became_peered":"2023-06-10T18:44:08.986804+0000","last_unstale":"2023-06-10T18:44:13.425330+0000","last_undegraded":"2023-06-10T18:44:13.425330+0000","last_fullsized":"2023-06-10T18:44:13.425330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394328+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.394328+0000","last_peered":"2023-06-10T18:44:13.394328+0000","last_clean":"2023-06-10T18:44:13.394328+0000","last_became_active":"2023-06-10T18:44:08.987510+0000","last_became_peered":"2023-06-10T18:44:08.987510+0000","last_unstale":"2023-06-10T18:44:13.394328+0000","last_undegraded":"2023-06-10T18:44:13.394328+0000","last_fullsized":"2023-06-10T18:44:13.394328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":913408,"data_stored":714279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":884736,"data_stored":709832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":122732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:44:37.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:44:37.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.409+0000 7f11b32b2700 1 Processor -- start 2023-06-10T18:44:37.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.409+0000 7f11b32b2700 1 -- start start 2023-06-10T18:44:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f11ac06e0e0 0x7f11ac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e4b0 0x7f11ac0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11ac0ca100 0x7f11ac0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac071640 con 0x7f11ac0ca100 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac0aef00 con 0x7f11ac06e0e0 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b32b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac06d290 con 0x7f11ac06e4b0 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b184f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11ac0ca100 0x7f11ac0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b184f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11ac0ca100 0x7f11ac0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55130/0 (socket says 172.21.15.104:55130) 2023-06-10T18:44:37.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b184f700 1 -- 172.21.15.104:0/2609949660 learned_addr learned my addr 172.21.15.104:0/2609949660 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b084d700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e4b0 0x7f11ac0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.410+0000 7f11b104e700 1 --2- 172.21.15.104:0/2609949660 >> v2:172.21.15.114:3300/0 conn(0x7f11ac06e0e0 0x7f11ac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 -- 172.21.15.104:0/2609949660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e4b0 msgr2=0x7f11ac0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e4b0 0x7f11ac0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 -- 172.21.15.104:0/2609949660 >> v2:172.21.15.114:3300/0 conn(0x7f11ac06e0e0 msgr2=0x7f11ac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 --2- 172.21.15.104:0/2609949660 >> v2:172.21.15.114:3300/0 conn(0x7f11ac06e0e0 0x7f11ac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 -- 172.21.15.104:0/2609949660 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac06d4c0 con 0x7f11ac0ca100 2023-06-10T18:44:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b184f700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0ce9f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11ac06af50 tx=0x7f11a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0079c57674fa9e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11a3fff700 1 -- 172.21.15.104:0/2609949660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a400e070 con 0x7f11ac0ca100 2023-06-10T18:44:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11a3fff700 1 -- 172.21.15.104:0/2609949660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a400b8d0 con 0x7f11ac0ca100 2023-06-10T18:44:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11a3fff700 1 -- 172.21.15.104:0/2609949660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a400bc80 con 0x7f11ac0ca100 2023-06-10T18:44:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 -- 172.21.15.104:0/2609949660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 msgr2=0x7f11ac0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0ce9f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11ac06af50 tx=0x7f11a400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 -- 172.21.15.104:0/2609949660 shutdown_connections 2023-06-10T18:44:37.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0ce9f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 --2- 172.21.15.104:0/2609949660 >> v2:172.21.15.114:3300/0 conn(0x7f11ac06e0e0 0x7f11ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 --2- 172.21.15.104:0/2609949660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e4b0 0x7f11ac0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.411+0000 7f11b32b2700 1 -- 172.21.15.104:0/2609949660 >> 172.21.15.104:0/2609949660 conn(0x7f11ac06b8a0 msgr2=0x7f11ac0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.412+0000 7f11b32b2700 1 -- 172.21.15.104:0/2609949660 shutdown_connections 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.412+0000 7f11b32b2700 1 -- 172.21.15.104:0/2609949660 wait complete. 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.413+0000 7f11b32b2700 1 Processor -- start 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.413+0000 7f11b32b2700 1 -- start start 2023-06-10T18:44:37.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 0x7f11ac0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ac06e4b0 0x7f11ac1f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0af870 con 0x7f11ac0ca100 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0afb50 con 0x7f11ac06e4b0 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b32b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0afe30 con 0x7f11ac06e0e0 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55136/0 (socket says 172.21.15.104:55136) 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 -- 172.21.15.104:0/908466313 learned_addr learned my addr 172.21.15.104:0/908466313 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b084d700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ac06e4b0 0x7f11ac1f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b104e700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 0x7f11ac0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 -- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 msgr2=0x7f11ac0af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 0x7f11ac0af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 -- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ac06e4b0 msgr2=0x7f11ac1f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ac06e4b0 0x7f11ac1f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b184f700 1 -- 172.21.15.104:0/908466313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a4009b30 con 0x7f11ac0ca100 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.414+0000 7f11b104e700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 0x7f11ac0af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11b184f700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f11ac06cc20 tx=0x7f11a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4003e10 con 0x7f11ac0ca100 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a4004250 con 0x7f11ac0ca100 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac0b24d0 con 0x7f11ac0ca100 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4004600 con 0x7f11ac0ca100 2023-06-10T18:44:37.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.415+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac0af580 con 0x7f11ac0ca100 2023-06-10T18:44:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.416+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac0b29f0 con 0x7f11ac0ca100 2023-06-10T18:44:37.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.416+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f11a4004760 con 0x7f11ac0ca100 2023-06-10T18:44:37.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.417+0000 7f11a1ffb700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.417+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f11a408ac00 con 0x7f11ac0ca100 2023-06-10T18:44:37.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.417+0000 7f11b104e700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.417+0000 7f11b104e700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f11ac155970 tx=0x7f119c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.421+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11ac0b29f0 con 0x7f11ac0ca100 2023-06-10T18:44:37.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:44:37.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:44:37.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.609+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11ac06da80 con 0x7f119006ace0 2023-06-10T18:44:37.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.619+0000 7f11a1ffb700 1 -- 172.21.15.104:0/908466313 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367286 (secure 0 0 0) 0x7f119c0096b0 con 0x7f119006ace0 2023-06-10T18:44:37.628 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:37.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 msgr2=0x7f119006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f11ac155970 tx=0x7f119c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:37.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 msgr2=0x7f11ac0b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f11ac06cc20 tx=0x7f11a4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 shutdown_connections 2023-06-10T18:44:37.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11ac0ca100 0x7f11ac0b2cd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11ac06e0e0 0x7f11ac0af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 --2- 172.21.15.104:0/908466313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11ac06e4b0 0x7f11ac1f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 >> 172.21.15.104:0/908466313 conn(0x7f11ac06b8a0 msgr2=0x7f11ac0ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 shutdown_connections 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.622+0000 7f11b32b2700 1 -- 172.21.15.104:0/908466313 wait complete. 2023-06-10T18:44:37.632 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:44:37.640 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-06-10T18:44:37.355262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":3480,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":3563520,"data_stored":2145052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003412"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423396+0000","last_change":"2023-06-10T18:44:08.974320+0000","last_active":"2023-06-10T18:44:13.423396+0000","last_peered":"2023-06-10T18:44:13.423396+0000","last_clean":"2023-06-10T18:44:13.423396+0000","last_became_active":"2023-06-10T18:44:08.973998+0000","last_became_peered":"2023-06-10T18:44:08.973998+0000","last_unstale":"2023-06-10T18:44:13.423396+0000","last_undegraded":"2023-06-10T18:44:13.423396+0000","last_fullsized":"2023-06-10T18:44:13.423396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:29.774977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393150+0000","last_change":"2023-06-10T18:44:08.986524+0000","last_active":"2023-06-10T18:44:13.393150+0000","last_peered":"2023-06-10T18:44:13.393150+0000","last_clean":"2023-06-10T18:44:13.393150+0000","last_became_active":"2023-06-10T18:44:08.986238+0000","last_became_peered":"2023-06-10T18:44:08.986238+0000","last_unstale":"2023-06-10T18:44:13.393150+0000","last_undegraded":"2023-06-10T18:44:13.393150+0000","last_fullsized":"2023-06-10T18:44:13.393150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423472+0000","last_change":"2023-06-10T18:44:08.991803+0000","last_active":"2023-06-10T18:44:13.423472+0000","last_peered":"2023-06-10T18:44:13.423472+0000","last_clean":"2023-06-10T18:44:13.423472+0000","last_became_active":"2023-06-10T18:44:08.986777+0000","last_became_peered":"2023-06-10T18:44:08.986777+0000","last_unstale":"2023-06-10T18:44:13.423472+0000","last_undegraded":"2023-06-10T18:44:13.423472+0000","last_fullsized":"2023-06-10T18:44:13.423472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423103+0000","last_change":"2023-06-10T18:44:08.984677+0000","last_active":"2023-06-10T18:44:13.423103+0000","last_peered":"2023-06-10T18:44:13.423103+0000","last_clean":"2023-06-10T18:44:13.423103+0000","last_became_active":"2023-06-10T18:44:08.984368+0000","last_became_peered":"2023-06-10T18:44:08.984368+0000","last_unstale":"2023-06-10T18:44:13.423103+0000","last_undegraded":"2023-06-10T18:44:13.423103+0000","last_fullsized":"2023-06-10T18:44:13.423103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:21.561425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423951+0000","last_change":"2023-06-10T18:44:08.988022+0000","last_active":"2023-06-10T18:44:13.423951+0000","last_peered":"2023-06-10T18:44:13.423951+0000","last_clean":"2023-06-10T18:44:13.423951+0000","last_became_active":"2023-06-10T18:44:08.983186+0000","last_became_peered":"2023-06-10T18:44:08.983186+0000","last_unstale":"2023-06-10T18:44:13.423951+0000","last_undegraded":"2023-06-10T18:44:13.423951+0000","last_fullsized":"2023-06-10T18:44:13.423951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:24.256200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423368+0000","last_change":"2023-06-10T18:44:08.986749+0000","last_active":"2023-06-10T18:44:13.423368+0000","last_peered":"2023-06-10T18:44:13.423368+0000","last_clean":"2023-06-10T18:44:13.423368+0000","last_became_active":"2023-06-10T18:44:08.986456+0000","last_became_peered":"2023-06-10T18:44:08.986456+0000","last_unstale":"2023-06-10T18:44:13.423368+0000","last_undegraded":"2023-06-10T18:44:13.423368+0000","last_fullsized":"2023-06-10T18:44:13.423368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394293+0000","last_change":"2023-06-10T18:44:08.992850+0000","last_active":"2023-06-10T18:44:13.394293+0000","last_peered":"2023-06-10T18:44:13.394293+0000","last_clean":"2023-06-10T18:44:13.394293+0000","last_became_active":"2023-06-10T18:44:08.992572+0000","last_became_peered":"2023-06-10T18:44:08.992572+0000","last_unstale":"2023-06-10T18:44:13.394293+0000","last_undegraded":"2023-06-10T18:44:13.394293+0000","last_fullsized":"2023-06-10T18:44:13.394293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392634+0000","last_change":"2023-06-10T18:44:08.979428+0000","last_active":"2023-06-10T18:44:13.392634+0000","last_peered":"2023-06-10T18:44:13.392634+0000","last_clean":"2023-06-10T18:44:13.392634+0000","last_became_active":"2023-06-10T18:44:08.979157+0000","last_became_peered":"2023-06-10T18:44:08.979157+0000","last_unstale":"2023-06-10T18:44:13.392634+0000","last_undegraded":"2023-06-10T18:44:13.392634+0000","last_fullsized":"2023-06-10T18:44:13.392634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:24.114600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423251+0000","last_change":"2023-06-10T18:44:08.984436+0000","last_active":"2023-06-10T18:44:13.423251+0000","last_peered":"2023-06-10T18:44:13.423251+0000","last_clean":"2023-06-10T18:44:13.423251+0000","last_became_active":"2023-06-10T18:44:08.984021+0000","last_became_peered":"2023-06-10T18:44:08.984021+0000","last_unstale":"2023-06-10T18:44:13.423251+0000","last_undegraded":"2023-06-10T18:44:13.423251+0000","last_fullsized":"2023-06-10T18:44:13.423251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:23:54.181578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393491+0000","last_change":"2023-06-10T18:44:08.984325+0000","last_active":"2023-06-10T18:44:13.393491+0000","last_peered":"2023-06-10T18:44:13.393491+0000","last_clean":"2023-06-10T18:44:13.393491+0000","last_became_active":"2023-06-10T18:44:08.983999+0000","last_became_peered":"2023-06-10T18:44:08.983999+0000","last_unstale":"2023-06-10T18:44:13.393491+0000","last_undegraded":"2023-06-10T18:44:13.393491+0000","last_fullsized":"2023-06-10T18:44:13.393491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:58.095055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423627+0000","last_change":"2023-06-10T18:44:08.991961+0000","last_active":"2023-06-10T18:44:13.423627+0000","last_peered":"2023-06-10T18:44:13.423627+0000","last_clean":"2023-06-10T18:44:13.423627+0000","last_became_active":"2023-06-10T18:44:08.991527+0000","last_became_peered":"2023-06-10T18:44:08.991527+0000","last_unstale":"2023-06-10T18:44:13.423627+0000","last_undegraded":"2023-06-10T18:44:13.423627+0000","last_fullsized":"2023-06-10T18:44:13.423627+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423011+0000","last_change":"2023-06-10T18:44:08.995250+0000","last_active":"2023-06-10T18:44:13.423011+0000","last_peered":"2023-06-10T18:44:13.423011+0000","last_clean":"2023-06-10T18:44:13.423011+0000","last_became_active":"2023-06-10T18:44:08.994854+0000","last_became_peered":"2023-06-10T18:44:08.994854+0000","last_unstale":"2023-06-10T18:44:13.423011+0000","last_undegraded":"2023-06-10T18:44:13.423011+0000","last_fullsized":"2023-06-10T18:44:13.423011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:13:54.715925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422801+0000","last_change":"2023-06-10T18:44:08.973692+0000","last_active":"2023-06-10T18:44:13.422801+0000","last_peered":"2023-06-10T18:44:13.422801+0000","last_clean":"2023-06-10T18:44:13.422801+0000","last_became_active":"2023-06-10T18:44:08.973417+0000","last_became_peered":"2023-06-10T18:44:08.973417+0000","last_unstale":"2023-06-10T18:44:13.422801+0000","last_undegraded":"2023-06-10T18:44:13.422801+0000","last_fullsized":"2023-06-10T18:44:13.422801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:27:20.138188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392896+0000","last_change":"2023-06-10T18:44:08.976711+0000","last_active":"2023-06-10T18:44:13.392896+0000","last_peered":"2023-06-10T18:44:13.392896+0000","last_clean":"2023-06-10T18:44:13.392896+0000","last_became_active":"2023-06-10T18:44:08.976415+0000","last_became_peered":"2023-06-10T18:44:08.976415+0000","last_unstale":"2023-06-10T18:44:13.392896+0000","last_undegraded":"2023-06-10T18:44:13.392896+0000","last_fullsized":"2023-06-10T18:44:13.392896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394066+0000","last_change":"2023-06-10T18:44:08.992788+0000","last_active":"2023-06-10T18:44:13.394066+0000","last_peered":"2023-06-10T18:44:13.394066+0000","last_clean":"2023-06-10T18:44:13.394066+0000","last_became_active":"2023-06-10T18:44:08.992335+0000","last_became_peered":"2023-06-10T18:44:08.992335+0000","last_unstale":"2023-06-10T18:44:13.394066+0000","last_undegraded":"2023-06-10T18:44:13.394066+0000","last_fullsized":"2023-06-10T18:44:13.394066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423080+0000","last_change":"2023-06-10T18:44:08.993080+0000","last_active":"2023-06-10T18:44:13.423080+0000","last_peered":"2023-06-10T18:44:13.423080+0000","last_clean":"2023-06-10T18:44:13.423080+0000","last_became_active":"2023-06-10T18:44:08.992794+0000","last_became_peered":"2023-06-10T18:44:08.992794+0000","last_unstale":"2023-06-10T18:44:13.423080+0000","last_undegraded":"2023-06-10T18:44:13.423080+0000","last_fullsized":"2023-06-10T18:44:13.423080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423378+0000","last_change":"2023-06-10T18:44:08.984915+0000","last_active":"2023-06-10T18:44:13.423378+0000","last_peered":"2023-06-10T18:44:13.423378+0000","last_clean":"2023-06-10T18:44:13.423378+0000","last_became_active":"2023-06-10T18:44:08.984639+0000","last_became_peered":"2023-06-10T18:44:08.984639+0000","last_unstale":"2023-06-10T18:44:13.423378+0000","last_undegraded":"2023-06-10T18:44:13.423378+0000","last_fullsized":"2023-06-10T18:44:13.423378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:53.794032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424069+0000","last_change":"2023-06-10T18:44:08.989447+0000","last_active":"2023-06-10T18:44:13.424069+0000","last_peered":"2023-06-10T18:44:13.424069+0000","last_clean":"2023-06-10T18:44:13.424069+0000","last_became_active":"2023-06-10T18:44:08.987688+0000","last_became_peered":"2023-06-10T18:44:08.987688+0000","last_unstale":"2023-06-10T18:44:13.424069+0000","last_undegraded":"2023-06-10T18:44:13.424069+0000","last_fullsized":"2023-06-10T18:44:13.424069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422601+0000","last_change":"2023-06-10T18:44:08.984545+0000","last_active":"2023-06-10T18:44:13.422601+0000","last_peered":"2023-06-10T18:44:13.422601+0000","last_clean":"2023-06-10T18:44:13.422601+0000","last_became_active":"2023-06-10T18:44:08.983857+0000","last_became_peered":"2023-06-10T18:44:08.983857+0000","last_unstale":"2023-06-10T18:44:13.422601+0000","last_undegraded":"2023-06-10T18:44:13.422601+0000","last_fullsized":"2023-06-10T18:44:13.422601+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:46.973658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423818+0000","last_change":"2023-06-10T18:44:08.993840+0000","last_active":"2023-06-10T18:44:13.423818+0000","last_peered":"2023-06-10T18:44:13.423818+0000","last_clean":"2023-06-10T18:44:13.423818+0000","last_became_active":"2023-06-10T18:44:08.993524+0000","last_became_peered":"2023-06-10T18:44:08.993524+0000","last_unstale":"2023-06-10T18:44:13.423818+0000","last_undegraded":"2023-06-10T18:44:13.423818+0000","last_fullsized":"2023-06-10T18:44:13.423818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393980+0000","last_change":"2023-06-10T18:44:08.994356+0000","last_active":"2023-06-10T18:44:13.393980+0000","last_peered":"2023-06-10T18:44:13.393980+0000","last_clean":"2023-06-10T18:44:13.393980+0000","last_became_active":"2023-06-10T18:44:08.994055+0000","last_became_peered":"2023-06-10T18:44:08.994055+0000","last_unstale":"2023-06-10T18:44:13.393980+0000","last_undegraded":"2023-06-10T18:44:13.393980+0000","last_fullsized":"2023-06-10T18:44:13.393980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423446+0000","last_change":"2023-06-10T18:44:08.988865+0000","last_active":"2023-06-10T18:44:13.423446+0000","last_peered":"2023-06-10T18:44:13.423446+0000","last_clean":"2023-06-10T18:44:13.423446+0000","last_became_active":"2023-06-10T18:44:08.988631+0000","last_became_peered":"2023-06-10T18:44:08.988631+0000","last_unstale":"2023-06-10T18:44:13.423446+0000","last_undegraded":"2023-06-10T18:44:13.423446+0000","last_fullsized":"2023-06-10T18:44:13.423446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:18.367120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424783+0000","last_change":"2023-06-10T18:44:08.985010+0000","last_active":"2023-06-10T18:44:13.424783+0000","last_peered":"2023-06-10T18:44:13.424783+0000","last_clean":"2023-06-10T18:44:13.424783+0000","last_became_active":"2023-06-10T18:44:08.984770+0000","last_became_peered":"2023-06-10T18:44:08.984770+0000","last_unstale":"2023-06-10T18:44:13.424783+0000","last_undegraded":"2023-06-10T18:44:13.424783+0000","last_fullsized":"2023-06-10T18:44:13.424783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:12.028636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394283+0000","last_change":"2023-06-10T18:44:08.994280+0000","last_active":"2023-06-10T18:44:13.394283+0000","last_peered":"2023-06-10T18:44:13.394283+0000","last_clean":"2023-06-10T18:44:13.394283+0000","last_became_active":"2023-06-10T18:44:08.993996+0000","last_became_peered":"2023-06-10T18:44:08.993996+0000","last_unstale":"2023-06-10T18:44:13.394283+0000","last_undegraded":"2023-06-10T18:44:13.394283+0000","last_fullsized":"2023-06-10T18:44:13.394283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394507+0000","last_change":"2023-06-10T18:44:08.988231+0000","last_active":"2023-06-10T18:44:13.394507+0000","last_peered":"2023-06-10T18:44:13.394507+0000","last_clean":"2023-06-10T18:44:13.394507+0000","last_became_active":"2023-06-10T18:44:08.987866+0000","last_became_peered":"2023-06-10T18:44:08.987866+0000","last_unstale":"2023-06-10T18:44:13.394507+0000","last_undegraded":"2023-06-10T18:44:13.394507+0000","last_fullsized":"2023-06-10T18:44:13.394507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394552+0000","last_change":"2023-06-10T18:44:08.984357+0000","last_active":"2023-06-10T18:44:13.394552+0000","last_peered":"2023-06-10T18:44:13.394552+0000","last_clean":"2023-06-10T18:44:13.394552+0000","last_became_active":"2023-06-10T18:44:08.984034+0000","last_became_peered":"2023-06-10T18:44:08.984034+0000","last_unstale":"2023-06-10T18:44:13.394552+0000","last_undegraded":"2023-06-10T18:44:13.394552+0000","last_fullsized":"2023-06-10T18:44:13.394552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393451+0000","last_change":"2023-06-10T18:44:08.987489+0000","last_active":"2023-06-10T18:44:13.393451+0000","last_peered":"2023-06-10T18:44:13.393451+0000","last_clean":"2023-06-10T18:44:13.393451+0000","last_became_active":"2023-06-10T18:44:08.985804+0000","last_became_peered":"2023-06-10T18:44:08.985804+0000","last_unstale":"2023-06-10T18:44:13.393451+0000","last_undegraded":"2023-06-10T18:44:13.393451+0000","last_fullsized":"2023-06-10T18:44:13.393451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:09.078873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424186+0000","last_change":"2023-06-10T18:44:13.001999+0000","last_active":"2023-06-10T18:44:13.424186+0000","last_peered":"2023-06-10T18:44:13.424186+0000","last_clean":"2023-06-10T18:44:13.424186+0000","last_became_active":"2023-06-10T18:44:13.001637+0000","last_became_peered":"2023-06-10T18:44:13.001637+0000","last_unstale":"2023-06-10T18:44:13.424186+0000","last_undegraded":"2023-06-10T18:44:13.424186+0000","last_fullsized":"2023-06-10T18:44:13.424186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423197+0000","last_change":"2023-06-10T18:44:08.575638+0000","last_active":"2023-06-10T18:44:13.423197+0000","last_peered":"2023-06-10T18:44:13.423197+0000","last_clean":"2023-06-10T18:44:13.423197+0000","last_became_active":"2023-06-10T18:44:08.575345+0000","last_became_peered":"2023-06-10T18:44:08.575345+0000","last_unstale":"2023-06-10T18:44:13.423197+0000","last_undegraded":"2023-06-10T18:44:13.423197+0000","last_fullsized":"2023-06-10T18:44:13.423197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:11.022238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425003+0000","last_change":"2023-06-10T18:44:08.986591+0000","last_active":"2023-06-10T18:44:13.425003+0000","last_peered":"2023-06-10T18:44:13.425003+0000","last_clean":"2023-06-10T18:44:13.425003+0000","last_became_active":"2023-06-10T18:44:08.986320+0000","last_became_peered":"2023-06-10T18:44:08.986320+0000","last_unstale":"2023-06-10T18:44:13.425003+0000","last_undegraded":"2023-06-10T18:44:13.425003+0000","last_fullsized":"2023-06-10T18:44:13.425003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424063+0000","last_change":"2023-06-10T18:44:13.002454+0000","last_active":"2023-06-10T18:44:13.424063+0000","last_peered":"2023-06-10T18:44:13.424063+0000","last_clean":"2023-06-10T18:44:13.424063+0000","last_became_active":"2023-06-10T18:44:13.002101+0000","last_became_peered":"2023-06-10T18:44:13.002101+0000","last_unstale":"2023-06-10T18:44:13.424063+0000","last_undegraded":"2023-06-10T18:44:13.424063+0000","last_fullsized":"2023-06-10T18:44:13.424063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:38.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393702+0000","last_change":"2023-06-10T18:44:08.581341+0000","last_active":"2023-06-10T18:44:13.393702+0000","last_peered":"2023-06-10T18:44:13.393702+0000","last_clean":"2023-06-10T18:44:13.393702+0000","last_became_active":"2023-06-10T18:44:08.580977+0000","last_became_peered":"2023-06-10T18:44:08.580977+0000","last_unstale":"2023-06-10T18:44:13.393702+0000","last_undegraded":"2023-06-10T18:44:13.393702+0000","last_fullsized":"2023-06-10T18:44:13.393702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423101+0000","last_change":"2023-06-10T18:44:08.990386+0000","last_active":"2023-06-10T18:44:13.423101+0000","last_peered":"2023-06-10T18:44:13.423101+0000","last_clean":"2023-06-10T18:44:13.423101+0000","last_became_active":"2023-06-10T18:44:08.989816+0000","last_became_peered":"2023-06-10T18:44:08.989816+0000","last_unstale":"2023-06-10T18:44:13.423101+0000","last_undegraded":"2023-06-10T18:44:13.423101+0000","last_fullsized":"2023-06-10T18:44:13.423101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:27.419215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422737+0000","last_change":"2023-06-10T18:44:13.011429+0000","last_active":"2023-06-10T18:44:13.422737+0000","last_peered":"2023-06-10T18:44:13.422737+0000","last_clean":"2023-06-10T18:44:13.422737+0000","last_became_active":"2023-06-10T18:44:13.010928+0000","last_became_peered":"2023-06-10T18:44:13.010928+0000","last_unstale":"2023-06-10T18:44:13.422737+0000","last_undegraded":"2023-06-10T18:44:13.422737+0000","last_fullsized":"2023-06-10T18:44:13.422737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:25:46.759866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425044+0000","last_change":"2023-06-10T18:44:08.583484+0000","last_active":"2023-06-10T18:44:13.425044+0000","last_peered":"2023-06-10T18:44:13.425044+0000","last_clean":"2023-06-10T18:44:13.425044+0000","last_became_active":"2023-06-10T18:44:08.583197+0000","last_became_peered":"2023-06-10T18:44:08.583197+0000","last_unstale":"2023-06-10T18:44:13.425044+0000","last_undegraded":"2023-06-10T18:44:13.425044+0000","last_fullsized":"2023-06-10T18:44:13.425044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:33.286044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424162+0000","last_change":"2023-06-10T18:44:08.983580+0000","last_active":"2023-06-10T18:44:13.424162+0000","last_peered":"2023-06-10T18:44:13.424162+0000","last_clean":"2023-06-10T18:44:13.424162+0000","last_became_active":"2023-06-10T18:44:08.983254+0000","last_became_peered":"2023-06-10T18:44:08.983254+0000","last_unstale":"2023-06-10T18:44:13.424162+0000","last_undegraded":"2023-06-10T18:44:13.424162+0000","last_fullsized":"2023-06-10T18:44:13.424162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:53:36.396900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423826+0000","last_change":"2023-06-10T18:44:12.997904+0000","last_active":"2023-06-10T18:44:13.423826+0000","last_peered":"2023-06-10T18:44:13.423826+0000","last_clean":"2023-06-10T18:44:13.423826+0000","last_became_active":"2023-06-10T18:44:12.997502+0000","last_became_peered":"2023-06-10T18:44:12.997502+0000","last_unstale":"2023-06-10T18:44:13.423826+0000","last_undegraded":"2023-06-10T18:44:13.423826+0000","last_fullsized":"2023-06-10T18:44:13.423826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:17.892337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423483+0000","last_change":"2023-06-10T18:44:08.575792+0000","last_active":"2023-06-10T18:44:13.423483+0000","last_peered":"2023-06-10T18:44:13.423483+0000","last_clean":"2023-06-10T18:44:13.423483+0000","last_became_active":"2023-06-10T18:44:08.573780+0000","last_became_peered":"2023-06-10T18:44:08.573780+0000","last_unstale":"2023-06-10T18:44:13.423483+0000","last_undegraded":"2023-06-10T18:44:13.423483+0000","last_fullsized":"2023-06-10T18:44:13.423483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:19.709371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424910+0000","last_change":"2023-06-10T18:44:08.986631+0000","last_active":"2023-06-10T18:44:13.424910+0000","last_peered":"2023-06-10T18:44:13.424910+0000","last_clean":"2023-06-10T18:44:13.424910+0000","last_became_active":"2023-06-10T18:44:08.986357+0000","last_became_peered":"2023-06-10T18:44:08.986357+0000","last_unstale":"2023-06-10T18:44:13.424910+0000","last_undegraded":"2023-06-10T18:44:13.424910+0000","last_fullsized":"2023-06-10T18:44:13.424910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393507+0000","last_change":"2023-06-10T18:44:13.002734+0000","last_active":"2023-06-10T18:44:13.393507+0000","last_peered":"2023-06-10T18:44:13.393507+0000","last_clean":"2023-06-10T18:44:13.393507+0000","last_became_active":"2023-06-10T18:44:13.002226+0000","last_became_peered":"2023-06-10T18:44:13.002226+0000","last_unstale":"2023-06-10T18:44:13.393507+0000","last_undegraded":"2023-06-10T18:44:13.393507+0000","last_fullsized":"2023-06-10T18:44:13.393507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:11.264615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393372+0000","last_change":"2023-06-10T18:44:08.571167+0000","last_active":"2023-06-10T18:44:13.393372+0000","last_peered":"2023-06-10T18:44:13.393372+0000","last_clean":"2023-06-10T18:44:13.393372+0000","last_became_active":"2023-06-10T18:44:08.570885+0000","last_became_peered":"2023-06-10T18:44:08.570885+0000","last_unstale":"2023-06-10T18:44:13.393372+0000","last_undegraded":"2023-06-10T18:44:13.393372+0000","last_fullsized":"2023-06-10T18:44:13.393372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.984523+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.984265+0000","last_became_peered":"2023-06-10T18:44:08.984265+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425522+0000","last_change":"2023-06-10T18:44:13.005254+0000","last_active":"2023-06-10T18:44:13.425522+0000","last_peered":"2023-06-10T18:44:13.425522+0000","last_clean":"2023-06-10T18:44:13.425522+0000","last_became_active":"2023-06-10T18:44:13.004908+0000","last_became_peered":"2023-06-10T18:44:13.004908+0000","last_unstale":"2023-06-10T18:44:13.425522+0000","last_undegraded":"2023-06-10T18:44:13.425522+0000","last_fullsized":"2023-06-10T18:44:13.425522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:23.228638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394120+0000","last_change":"2023-06-10T18:44:08.577421+0000","last_active":"2023-06-10T18:44:13.394120+0000","last_peered":"2023-06-10T18:44:13.394120+0000","last_clean":"2023-06-10T18:44:13.394120+0000","last_became_active":"2023-06-10T18:44:08.577127+0000","last_became_peered":"2023-06-10T18:44:08.577127+0000","last_unstale":"2023-06-10T18:44:13.394120+0000","last_undegraded":"2023-06-10T18:44:13.394120+0000","last_fullsized":"2023-06-10T18:44:13.394120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:14.393845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393966+0000","last_change":"2023-06-10T18:44:08.986118+0000","last_active":"2023-06-10T18:44:13.393966+0000","last_peered":"2023-06-10T18:44:13.393966+0000","last_clean":"2023-06-10T18:44:13.393966+0000","last_became_active":"2023-06-10T18:44:08.985845+0000","last_became_peered":"2023-06-10T18:44:08.985845+0000","last_unstale":"2023-06-10T18:44:13.393966+0000","last_undegraded":"2023-06-10T18:44:13.393966+0000","last_fullsized":"2023-06-10T18:44:13.393966+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393763+0000","last_change":"2023-06-10T18:44:13.009330+0000","last_active":"2023-06-10T18:44:13.393763+0000","last_peered":"2023-06-10T18:44:13.393763+0000","last_clean":"2023-06-10T18:44:13.393763+0000","last_became_active":"2023-06-10T18:44:13.008968+0000","last_became_peered":"2023-06-10T18:44:13.008968+0000","last_unstale":"2023-06-10T18:44:13.393763+0000","last_undegraded":"2023-06-10T18:44:13.393763+0000","last_fullsized":"2023-06-10T18:44:13.393763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:01.454996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:18.388487+0000","last_change":"2023-06-10T18:44:08.577256+0000","last_active":"2023-06-10T18:44:18.388487+0000","last_peered":"2023-06-10T18:44:18.388487+0000","last_clean":"2023-06-10T18:44:18.388487+0000","last_became_active":"2023-06-10T18:44:08.576987+0000","last_became_peered":"2023-06-10T18:44:08.576987+0000","last_unstale":"2023-06-10T18:44:18.388487+0000","last_undegraded":"2023-06-10T18:44:18.388487+0000","last_fullsized":"2023-06-10T18:44:18.388487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424232+0000","last_change":"2023-06-10T18:44:08.989046+0000","last_active":"2023-06-10T18:44:13.424232+0000","last_peered":"2023-06-10T18:44:13.424232+0000","last_clean":"2023-06-10T18:44:13.424232+0000","last_became_active":"2023-06-10T18:44:08.988763+0000","last_became_peered":"2023-06-10T18:44:08.988763+0000","last_unstale":"2023-06-10T18:44:13.424232+0000","last_undegraded":"2023-06-10T18:44:13.424232+0000","last_fullsized":"2023-06-10T18:44:13.424232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393726+0000","last_change":"2023-06-10T18:44:13.009976+0000","last_active":"2023-06-10T18:44:13.393726+0000","last_peered":"2023-06-10T18:44:13.393726+0000","last_clean":"2023-06-10T18:44:13.393726+0000","last_became_active":"2023-06-10T18:44:13.009599+0000","last_became_peered":"2023-06-10T18:44:13.009599+0000","last_unstale":"2023-06-10T18:44:13.393726+0000","last_undegraded":"2023-06-10T18:44:13.393726+0000","last_fullsized":"2023-06-10T18:44:13.393726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393682+0000","last_change":"2023-06-10T18:44:08.578965+0000","last_active":"2023-06-10T18:44:13.393682+0000","last_peered":"2023-06-10T18:44:13.393682+0000","last_clean":"2023-06-10T18:44:13.393682+0000","last_became_active":"2023-06-10T18:44:08.578666+0000","last_became_peered":"2023-06-10T18:44:08.578666+0000","last_unstale":"2023-06-10T18:44:13.393682+0000","last_undegraded":"2023-06-10T18:44:13.393682+0000","last_fullsized":"2023-06-10T18:44:13.393682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425838+0000","last_change":"2023-06-10T18:44:08.986550+0000","last_active":"2023-06-10T18:44:13.425838+0000","last_peered":"2023-06-10T18:44:13.425838+0000","last_clean":"2023-06-10T18:44:13.425838+0000","last_became_active":"2023-06-10T18:44:08.986267+0000","last_became_peered":"2023-06-10T18:44:08.986267+0000","last_unstale":"2023-06-10T18:44:13.425838+0000","last_undegraded":"2023-06-10T18:44:13.425838+0000","last_fullsized":"2023-06-10T18:44:13.425838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:23.390127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425693+0000","last_change":"2023-06-10T18:44:13.007347+0000","last_active":"2023-06-10T18:44:13.425693+0000","last_peered":"2023-06-10T18:44:13.425693+0000","last_clean":"2023-06-10T18:44:13.425693+0000","last_became_active":"2023-06-10T18:44:13.006948+0000","last_became_peered":"2023-06-10T18:44:13.006948+0000","last_unstale":"2023-06-10T18:44:13.425693+0000","last_undegraded":"2023-06-10T18:44:13.425693+0000","last_fullsized":"2023-06-10T18:44:13.425693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:08.470458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423561+0000","last_change":"2023-06-10T18:44:08.575263+0000","last_active":"2023-06-10T18:44:13.423561+0000","last_peered":"2023-06-10T18:44:13.423561+0000","last_clean":"2023-06-10T18:44:13.423561+0000","last_became_active":"2023-06-10T18:44:08.574716+0000","last_became_peered":"2023-06-10T18:44:08.574716+0000","last_unstale":"2023-06-10T18:44:13.423561+0000","last_undegraded":"2023-06-10T18:44:13.423561+0000","last_fullsized":"2023-06-10T18:44:13.423561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:30.369934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393754+0000","last_change":"2023-06-10T18:44:08.986065+0000","last_active":"2023-06-10T18:44:13.393754+0000","last_peered":"2023-06-10T18:44:13.393754+0000","last_clean":"2023-06-10T18:44:13.393754+0000","last_became_active":"2023-06-10T18:44:08.984998+0000","last_became_peered":"2023-06-10T18:44:08.984998+0000","last_unstale":"2023-06-10T18:44:13.393754+0000","last_undegraded":"2023-06-10T18:44:13.393754+0000","last_fullsized":"2023-06-10T18:44:13.393754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425425+0000","last_change":"2023-06-10T18:44:13.004286+0000","last_active":"2023-06-10T18:44:13.425425+0000","last_peered":"2023-06-10T18:44:13.425425+0000","last_clean":"2023-06-10T18:44:13.425425+0000","last_became_active":"2023-06-10T18:44:13.003671+0000","last_became_peered":"2023-06-10T18:44:13.003671+0000","last_unstale":"2023-06-10T18:44:13.425425+0000","last_undegraded":"2023-06-10T18:44:13.425425+0000","last_fullsized":"2023-06-10T18:44:13.425425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394170+0000","last_change":"2023-06-10T18:44:08.578231+0000","last_active":"2023-06-10T18:44:13.394170+0000","last_peered":"2023-06-10T18:44:13.394170+0000","last_clean":"2023-06-10T18:44:13.394170+0000","last_became_active":"2023-06-10T18:44:08.577966+0000","last_became_peered":"2023-06-10T18:44:08.577966+0000","last_unstale":"2023-06-10T18:44:13.394170+0000","last_undegraded":"2023-06-10T18:44:13.394170+0000","last_fullsized":"2023-06-10T18:44:13.394170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423359+0000","last_change":"2023-06-10T18:44:08.990234+0000","last_active":"2023-06-10T18:44:13.423359+0000","last_peered":"2023-06-10T18:44:13.423359+0000","last_clean":"2023-06-10T18:44:13.423359+0000","last_became_active":"2023-06-10T18:44:08.989505+0000","last_became_peered":"2023-06-10T18:44:08.989505+0000","last_unstale":"2023-06-10T18:44:13.423359+0000","last_undegraded":"2023-06-10T18:44:13.423359+0000","last_fullsized":"2023-06-10T18:44:13.423359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:31.544241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393091+0000","last_change":"2023-06-10T18:44:13.009889+0000","last_active":"2023-06-10T18:44:13.393091+0000","last_peered":"2023-06-10T18:44:13.393091+0000","last_clean":"2023-06-10T18:44:13.393091+0000","last_became_active":"2023-06-10T18:44:13.009528+0000","last_became_peered":"2023-06-10T18:44:13.009528+0000","last_unstale":"2023-06-10T18:44:13.393091+0000","last_undegraded":"2023-06-10T18:44:13.393091+0000","last_fullsized":"2023-06-10T18:44:13.393091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:15.449838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":29,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422838+0000","last_change":"2023-06-10T18:44:08.573929+0000","last_active":"2023-06-10T18:44:13.422838+0000","last_peered":"2023-06-10T18:44:13.422838+0000","last_clean":"2023-06-10T18:44:13.422838+0000","last_became_active":"2023-06-10T18:44:08.573652+0000","last_became_peered":"2023-06-10T18:44:08.573652+0000","last_unstale":"2023-06-10T18:44:13.422838+0000","last_undegraded":"2023-06-10T18:44:13.422838+0000","last_fullsized":"2023-06-10T18:44:13.422838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:08.763880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425617+0000","last_change":"2023-06-10T18:44:08.984556+0000","last_active":"2023-06-10T18:44:13.425617+0000","last_peered":"2023-06-10T18:44:13.425617+0000","last_clean":"2023-06-10T18:44:13.425617+0000","last_became_active":"2023-06-10T18:44:08.984321+0000","last_became_peered":"2023-06-10T18:44:08.984321+0000","last_unstale":"2023-06-10T18:44:13.425617+0000","last_undegraded":"2023-06-10T18:44:13.425617+0000","last_fullsized":"2023-06-10T18:44:13.425617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:28:40.616868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394464+0000","last_change":"2023-06-10T18:44:13.006271+0000","last_active":"2023-06-10T18:44:13.394464+0000","last_peered":"2023-06-10T18:44:13.394464+0000","last_clean":"2023-06-10T18:44:13.394464+0000","last_became_active":"2023-06-10T18:44:13.005878+0000","last_became_peered":"2023-06-10T18:44:13.005878+0000","last_unstale":"2023-06-10T18:44:13.394464+0000","last_undegraded":"2023-06-10T18:44:13.394464+0000","last_fullsized":"2023-06-10T18:44:13.394464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:21.046063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394016+0000","last_change":"2023-06-10T18:44:08.578331+0000","last_active":"2023-06-10T18:44:13.394016+0000","last_peered":"2023-06-10T18:44:13.394016+0000","last_clean":"2023-06-10T18:44:13.394016+0000","last_became_active":"2023-06-10T18:44:08.577846+0000","last_became_peered":"2023-06-10T18:44:08.577846+0000","last_unstale":"2023-06-10T18:44:13.394016+0000","last_undegraded":"2023-06-10T18:44:13.394016+0000","last_fullsized":"2023-06-10T18:44:13.394016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392530+0000","last_change":"2023-06-10T18:44:08.984476+0000","last_active":"2023-06-10T18:44:13.392530+0000","last_peered":"2023-06-10T18:44:13.392530+0000","last_clean":"2023-06-10T18:44:13.392530+0000","last_became_active":"2023-06-10T18:44:08.984095+0000","last_became_peered":"2023-06-10T18:44:08.984095+0000","last_unstale":"2023-06-10T18:44:13.392530+0000","last_undegraded":"2023-06-10T18:44:13.392530+0000","last_fullsized":"2023-06-10T18:44:13.392530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422417+0000","last_change":"2023-06-10T18:44:13.006928+0000","last_active":"2023-06-10T18:44:13.422417+0000","last_peered":"2023-06-10T18:44:13.422417+0000","last_clean":"2023-06-10T18:44:13.422417+0000","last_became_active":"2023-06-10T18:44:13.006526+0000","last_became_peered":"2023-06-10T18:44:13.006526+0000","last_unstale":"2023-06-10T18:44:13.422417+0000","last_undegraded":"2023-06-10T18:44:13.422417+0000","last_fullsized":"2023-06-10T18:44:13.422417+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:42:15.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393898+0000","last_change":"2023-06-10T18:44:08.580704+0000","last_active":"2023-06-10T18:44:13.393898+0000","last_peered":"2023-06-10T18:44:13.393898+0000","last_clean":"2023-06-10T18:44:13.393898+0000","last_became_active":"2023-06-10T18:44:08.580413+0000","last_became_peered":"2023-06-10T18:44:08.580413+0000","last_unstale":"2023-06-10T18:44:13.393898+0000","last_undegraded":"2023-06-10T18:44:13.393898+0000","last_fullsized":"2023-06-10T18:44:13.393898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:29.052625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423333+0000","last_change":"2023-06-10T18:44:08.988350+0000","last_active":"2023-06-10T18:44:13.423333+0000","last_peered":"2023-06-10T18:44:13.423333+0000","last_clean":"2023-06-10T18:44:13.423333+0000","last_became_active":"2023-06-10T18:44:08.988073+0000","last_became_peered":"2023-06-10T18:44:08.988073+0000","last_unstale":"2023-06-10T18:44:13.423333+0000","last_undegraded":"2023-06-10T18:44:13.423333+0000","last_fullsized":"2023-06-10T18:44:13.423333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:45.669703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422684+0000","last_change":"2023-06-10T18:44:13.001742+0000","last_active":"2023-06-10T18:44:13.422684+0000","last_peered":"2023-06-10T18:44:13.422684+0000","last_clean":"2023-06-10T18:44:13.422684+0000","last_became_active":"2023-06-10T18:44:13.001373+0000","last_became_peered":"2023-06-10T18:44:13.001373+0000","last_unstale":"2023-06-10T18:44:13.422684+0000","last_undegraded":"2023-06-10T18:44:13.422684+0000","last_fullsized":"2023-06-10T18:44:13.422684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:11.616196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394281+0000","last_change":"2023-06-10T18:44:08.574389+0000","last_active":"2023-06-10T18:44:13.394281+0000","last_peered":"2023-06-10T18:44:13.394281+0000","last_clean":"2023-06-10T18:44:13.394281+0000","last_became_active":"2023-06-10T18:44:08.574056+0000","last_became_peered":"2023-06-10T18:44:08.574056+0000","last_unstale":"2023-06-10T18:44:13.394281+0000","last_undegraded":"2023-06-10T18:44:13.394281+0000","last_fullsized":"2023-06-10T18:44:13.394281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423143+0000","last_change":"2023-06-10T18:44:08.987503+0000","last_active":"2023-06-10T18:44:13.423143+0000","last_peered":"2023-06-10T18:44:13.423143+0000","last_clean":"2023-06-10T18:44:13.423143+0000","last_became_active":"2023-06-10T18:44:08.987204+0000","last_became_peered":"2023-06-10T18:44:08.987204+0000","last_unstale":"2023-06-10T18:44:13.423143+0000","last_undegraded":"2023-06-10T18:44:13.423143+0000","last_fullsized":"2023-06-10T18:44:13.423143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:45.870218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422702+0000","last_change":"2023-06-10T18:44:13.003895+0000","last_active":"2023-06-10T18:44:13.422702+0000","last_peered":"2023-06-10T18:44:13.422702+0000","last_clean":"2023-06-10T18:44:13.422702+0000","last_became_active":"2023-06-10T18:44:13.001542+0000","last_became_peered":"2023-06-10T18:44:13.001542+0000","last_unstale":"2023-06-10T18:44:13.422702+0000","last_undegraded":"2023-06-10T18:44:13.422702+0000","last_fullsized":"2023-06-10T18:44:13.422702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:54:25.091621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393042+0000","last_change":"2023-06-10T18:44:08.580607+0000","last_active":"2023-06-10T18:44:13.393042+0000","last_peered":"2023-06-10T18:44:13.393042+0000","last_clean":"2023-06-10T18:44:13.393042+0000","last_became_active":"2023-06-10T18:44:08.580289+0000","last_became_peered":"2023-06-10T18:44:08.580289+0000","last_unstale":"2023-06-10T18:44:13.393042+0000","last_undegraded":"2023-06-10T18:44:13.393042+0000","last_fullsized":"2023-06-10T18:44:13.393042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:09.158752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424116+0000","last_change":"2023-06-10T18:44:08.987624+0000","last_active":"2023-06-10T18:44:13.424116+0000","last_peered":"2023-06-10T18:44:13.424116+0000","last_clean":"2023-06-10T18:44:13.424116+0000","last_became_active":"2023-06-10T18:44:08.987375+0000","last_became_peered":"2023-06-10T18:44:08.987375+0000","last_unstale":"2023-06-10T18:44:13.424116+0000","last_undegraded":"2023-06-10T18:44:13.424116+0000","last_fullsized":"2023-06-10T18:44:13.424116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:20:19.233767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422769+0000","last_change":"2023-06-10T18:44:13.002722+0000","last_active":"2023-06-10T18:44:13.422769+0000","last_peered":"2023-06-10T18:44:13.422769+0000","last_clean":"2023-06-10T18:44:13.422769+0000","last_became_active":"2023-06-10T18:44:13.002214+0000","last_became_peered":"2023-06-10T18:44:13.002214+0000","last_unstale":"2023-06-10T18:44:13.422769+0000","last_undegraded":"2023-06-10T18:44:13.422769+0000","last_fullsized":"2023-06-10T18:44:13.422769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:04:35.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425192+0000","last_change":"2023-06-10T18:44:08.578128+0000","last_active":"2023-06-10T18:44:13.425192+0000","last_peered":"2023-06-10T18:44:13.425192+0000","last_clean":"2023-06-10T18:44:13.425192+0000","last_became_active":"2023-06-10T18:44:08.577863+0000","last_became_peered":"2023-06-10T18:44:08.577863+0000","last_unstale":"2023-06-10T18:44:13.425192+0000","last_undegraded":"2023-06-10T18:44:13.425192+0000","last_fullsized":"2023-06-10T18:44:13.425192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:08.072475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423537+0000","last_change":"2023-06-10T18:44:08.995303+0000","last_active":"2023-06-10T18:44:13.423537+0000","last_peered":"2023-06-10T18:44:13.423537+0000","last_clean":"2023-06-10T18:44:13.423537+0000","last_became_active":"2023-06-10T18:44:08.995025+0000","last_became_peered":"2023-06-10T18:44:08.995025+0000","last_unstale":"2023-06-10T18:44:13.423537+0000","last_undegraded":"2023-06-10T18:44:13.423537+0000","last_fullsized":"2023-06-10T18:44:13.423537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394147+0000","last_change":"2023-06-10T18:44:13.008926+0000","last_active":"2023-06-10T18:44:13.394147+0000","last_peered":"2023-06-10T18:44:13.394147+0000","last_clean":"2023-06-10T18:44:13.394147+0000","last_became_active":"2023-06-10T18:44:13.008249+0000","last_became_peered":"2023-06-10T18:44:13.008249+0000","last_unstale":"2023-06-10T18:44:13.394147+0000","last_undegraded":"2023-06-10T18:44:13.394147+0000","last_fullsized":"2023-06-10T18:44:13.394147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423270+0000","last_change":"2023-06-10T18:44:08.575472+0000","last_active":"2023-06-10T18:44:13.423270+0000","last_peered":"2023-06-10T18:44:13.423270+0000","last_clean":"2023-06-10T18:44:13.423270+0000","last_became_active":"2023-06-10T18:44:08.575186+0000","last_became_peered":"2023-06-10T18:44:08.575186+0000","last_unstale":"2023-06-10T18:44:13.423270+0000","last_undegraded":"2023-06-10T18:44:13.423270+0000","last_fullsized":"2023-06-10T18:44:13.423270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:33.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423243+0000","last_change":"2023-06-10T18:44:08.987175+0000","last_active":"2023-06-10T18:44:13.423243+0000","last_peered":"2023-06-10T18:44:13.423243+0000","last_clean":"2023-06-10T18:44:13.423243+0000","last_became_active":"2023-06-10T18:44:08.986395+0000","last_became_peered":"2023-06-10T18:44:08.986395+0000","last_unstale":"2023-06-10T18:44:13.423243+0000","last_undegraded":"2023-06-10T18:44:13.423243+0000","last_fullsized":"2023-06-10T18:44:13.423243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:39.899358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393572+0000","last_change":"2023-06-10T18:44:13.002535+0000","last_active":"2023-06-10T18:44:13.393572+0000","last_peered":"2023-06-10T18:44:13.393572+0000","last_clean":"2023-06-10T18:44:13.393572+0000","last_became_active":"2023-06-10T18:44:13.002065+0000","last_became_peered":"2023-06-10T18:44:13.002065+0000","last_unstale":"2023-06-10T18:44:13.393572+0000","last_undegraded":"2023-06-10T18:44:13.393572+0000","last_fullsized":"2023-06-10T18:44:13.393572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:56.164577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423083+0000","last_change":"2023-06-10T18:44:08.572499+0000","last_active":"2023-06-10T18:44:13.423083+0000","last_peered":"2023-06-10T18:44:13.423083+0000","last_clean":"2023-06-10T18:44:13.423083+0000","last_became_active":"2023-06-10T18:44:08.572196+0000","last_became_peered":"2023-06-10T18:44:08.572196+0000","last_unstale":"2023-06-10T18:44:13.423083+0000","last_undegraded":"2023-06-10T18:44:13.423083+0000","last_fullsized":"2023-06-10T18:44:13.423083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:46.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423986+0000","last_change":"2023-06-10T18:44:08.989104+0000","last_active":"2023-06-10T18:44:13.423986+0000","last_peered":"2023-06-10T18:44:13.423986+0000","last_clean":"2023-06-10T18:44:13.423986+0000","last_became_active":"2023-06-10T18:44:08.988824+0000","last_became_peered":"2023-06-10T18:44:08.988824+0000","last_unstale":"2023-06-10T18:44:13.423986+0000","last_undegraded":"2023-06-10T18:44:13.423986+0000","last_fullsized":"2023-06-10T18:44:13.423986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:19.807436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423861+0000","last_change":"2023-06-10T18:44:13.006826+0000","last_active":"2023-06-10T18:44:13.423861+0000","last_peered":"2023-06-10T18:44:13.423861+0000","last_clean":"2023-06-10T18:44:13.423861+0000","last_became_active":"2023-06-10T18:44:13.006458+0000","last_became_peered":"2023-06-10T18:44:13.006458+0000","last_unstale":"2023-06-10T18:44:13.423861+0000","last_undegraded":"2023-06-10T18:44:13.423861+0000","last_fullsized":"2023-06-10T18:44:13.423861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:31.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424805+0000","last_change":"2023-06-10T18:44:08.577680+0000","last_active":"2023-06-10T18:44:13.424805+0000","last_peered":"2023-06-10T18:44:13.424805+0000","last_clean":"2023-06-10T18:44:13.424805+0000","last_became_active":"2023-06-10T18:44:08.577419+0000","last_became_peered":"2023-06-10T18:44:08.577419+0000","last_unstale":"2023-06-10T18:44:13.424805+0000","last_undegraded":"2023-06-10T18:44:13.424805+0000","last_fullsized":"2023-06-10T18:44:13.424805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:57.088657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394577+0000","last_change":"2023-06-10T18:44:08.991268+0000","last_active":"2023-06-10T18:44:13.394577+0000","last_peered":"2023-06-10T18:44:13.394577+0000","last_clean":"2023-06-10T18:44:13.394577+0000","last_became_active":"2023-06-10T18:44:08.990999+0000","last_became_peered":"2023-06-10T18:44:08.990999+0000","last_unstale":"2023-06-10T18:44:13.394577+0000","last_undegraded":"2023-06-10T18:44:13.394577+0000","last_fullsized":"2023-06-10T18:44:13.394577+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:49.382472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423567+0000","last_change":"2023-06-10T18:44:13.010490+0000","last_active":"2023-06-10T18:44:13.423567+0000","last_peered":"2023-06-10T18:44:13.423567+0000","last_clean":"2023-06-10T18:44:13.423567+0000","last_became_active":"2023-06-10T18:44:13.010098+0000","last_became_peered":"2023-06-10T18:44:13.010098+0000","last_unstale":"2023-06-10T18:44:13.423567+0000","last_undegraded":"2023-06-10T18:44:13.423567+0000","last_fullsized":"2023-06-10T18:44:13.423567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:58.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394227+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:44:13.394227+0000","last_peered":"2023-06-10T18:44:13.394227+0000","last_clean":"2023-06-10T18:44:13.394227+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:44:13.394227+0000","last_undegraded":"2023-06-10T18:44:13.394227+0000","last_fullsized":"2023-06-10T18:44:13.394227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393285+0000","last_change":"2023-06-10T18:44:08.577289+0000","last_active":"2023-06-10T18:44:13.393285+0000","last_peered":"2023-06-10T18:44:13.393285+0000","last_clean":"2023-06-10T18:44:13.393285+0000","last_became_active":"2023-06-10T18:44:08.577021+0000","last_became_peered":"2023-06-10T18:44:08.577021+0000","last_unstale":"2023-06-10T18:44:13.393285+0000","last_undegraded":"2023-06-10T18:44:13.393285+0000","last_fullsized":"2023-06-10T18:44:13.393285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:41.304279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393847+0000","last_change":"2023-06-10T18:44:08.983928+0000","last_active":"2023-06-10T18:44:13.393847+0000","last_peered":"2023-06-10T18:44:13.393847+0000","last_clean":"2023-06-10T18:44:13.393847+0000","last_became_active":"2023-06-10T18:44:08.983562+0000","last_became_peered":"2023-06-10T18:44:08.983562+0000","last_unstale":"2023-06-10T18:44:13.393847+0000","last_undegraded":"2023-06-10T18:44:13.393847+0000","last_fullsized":"2023-06-10T18:44:13.393847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423202+0000","last_change":"2023-06-10T18:44:13.006596+0000","last_active":"2023-06-10T18:44:13.423202+0000","last_peered":"2023-06-10T18:44:13.423202+0000","last_clean":"2023-06-10T18:44:13.423202+0000","last_became_active":"2023-06-10T18:44:13.006258+0000","last_became_peered":"2023-06-10T18:44:13.006258+0000","last_unstale":"2023-06-10T18:44:13.423202+0000","last_undegraded":"2023-06-10T18:44:13.423202+0000","last_fullsized":"2023-06-10T18:44:13.423202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:31.503221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393245+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:44:13.393245+0000","last_peered":"2023-06-10T18:44:13.393245+0000","last_clean":"2023-06-10T18:44:13.393245+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:44:13.393245+0000","last_undegraded":"2023-06-10T18:44:13.393245+0000","last_fullsized":"2023-06-10T18:44:13.393245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:28.260256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424001+0000","last_change":"2023-06-10T18:44:08.578515+0000","last_active":"2023-06-10T18:44:13.424001+0000","last_peered":"2023-06-10T18:44:13.424001+0000","last_clean":"2023-06-10T18:44:13.424001+0000","last_became_active":"2023-06-10T18:44:08.578184+0000","last_became_peered":"2023-06-10T18:44:08.578184+0000","last_unstale":"2023-06-10T18:44:13.424001+0000","last_undegraded":"2023-06-10T18:44:13.424001+0000","last_fullsized":"2023-06-10T18:44:13.424001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394190+0000","last_change":"2023-06-10T18:44:08.985054+0000","last_active":"2023-06-10T18:44:13.394190+0000","last_peered":"2023-06-10T18:44:13.394190+0000","last_clean":"2023-06-10T18:44:13.394190+0000","last_became_active":"2023-06-10T18:44:08.984698+0000","last_became_peered":"2023-06-10T18:44:08.984698+0000","last_unstale":"2023-06-10T18:44:13.394190+0000","last_undegraded":"2023-06-10T18:44:13.394190+0000","last_fullsized":"2023-06-10T18:44:13.394190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423677+0000","last_change":"2023-06-10T18:44:13.002621+0000","last_active":"2023-06-10T18:44:13.423677+0000","last_peered":"2023-06-10T18:44:13.423677+0000","last_clean":"2023-06-10T18:44:13.423677+0000","last_became_active":"2023-06-10T18:44:13.002229+0000","last_became_peered":"2023-06-10T18:44:13.002229+0000","last_unstale":"2023-06-10T18:44:13.423677+0000","last_undegraded":"2023-06-10T18:44:13.423677+0000","last_fullsized":"2023-06-10T18:44:13.423677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:44:56.053966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422499+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:44:13.422499+0000","last_peered":"2023-06-10T18:44:13.422499+0000","last_clean":"2023-06-10T18:44:13.422499+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:44:13.422499+0000","last_undegraded":"2023-06-10T18:44:13.422499+0000","last_fullsized":"2023-06-10T18:44:13.422499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:49:35.702822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393742+0000","last_change":"2023-06-10T18:44:08.571468+0000","last_active":"2023-06-10T18:44:13.393742+0000","last_peered":"2023-06-10T18:44:13.393742+0000","last_clean":"2023-06-10T18:44:13.393742+0000","last_became_active":"2023-06-10T18:44:08.570651+0000","last_became_peered":"2023-06-10T18:44:08.570651+0000","last_unstale":"2023-06-10T18:44:13.393742+0000","last_undegraded":"2023-06-10T18:44:13.393742+0000","last_fullsized":"2023-06-10T18:44:13.393742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394594+0000","last_change":"2023-06-10T18:44:08.983687+0000","last_active":"2023-06-10T18:44:13.394594+0000","last_peered":"2023-06-10T18:44:13.394594+0000","last_clean":"2023-06-10T18:44:13.394594+0000","last_became_active":"2023-06-10T18:44:08.983381+0000","last_became_peered":"2023-06-10T18:44:08.983381+0000","last_unstale":"2023-06-10T18:44:13.394594+0000","last_undegraded":"2023-06-10T18:44:13.394594+0000","last_fullsized":"2023-06-10T18:44:13.394594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:34.292443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393971+0000","last_change":"2023-06-10T18:44:13.009660+0000","last_active":"2023-06-10T18:44:13.393971+0000","last_peered":"2023-06-10T18:44:13.393971+0000","last_clean":"2023-06-10T18:44:13.393971+0000","last_became_active":"2023-06-10T18:44:13.009208+0000","last_became_peered":"2023-06-10T18:44:13.009208+0000","last_unstale":"2023-06-10T18:44:13.393971+0000","last_undegraded":"2023-06-10T18:44:13.393971+0000","last_fullsized":"2023-06-10T18:44:13.393971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.392956+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:44:13.392956+0000","last_peered":"2023-06-10T18:44:13.392956+0000","last_clean":"2023-06-10T18:44:13.392956+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:44:13.392956+0000","last_undegraded":"2023-06-10T18:44:13.392956+0000","last_fullsized":"2023-06-10T18:44:13.392956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:41.046288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":183,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394153+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:44:13.394153+0000","last_peered":"2023-06-10T18:44:13.394153+0000","last_clean":"2023-06-10T18:44:13.394153+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:44:13.394153+0000","last_undegraded":"2023-06-10T18:44:13.394153+0000","last_fullsized":"2023-06-10T18:44:13.394153+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:22.105411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425014+0000","last_change":"2023-06-10T18:44:08.576254+0000","last_active":"2023-06-10T18:44:13.425014+0000","last_peered":"2023-06-10T18:44:13.425014+0000","last_clean":"2023-06-10T18:44:13.425014+0000","last_became_active":"2023-06-10T18:44:08.575980+0000","last_became_peered":"2023-06-10T18:44:08.575980+0000","last_unstale":"2023-06-10T18:44:13.425014+0000","last_undegraded":"2023-06-10T18:44:13.425014+0000","last_fullsized":"2023-06-10T18:44:13.425014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:09.445143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423295+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.423295+0000","last_peered":"2023-06-10T18:44:13.423295+0000","last_clean":"2023-06-10T18:44:13.423295+0000","last_became_active":"2023-06-10T18:44:08.988159+0000","last_became_peered":"2023-06-10T18:44:08.988159+0000","last_unstale":"2023-06-10T18:44:13.423295+0000","last_undegraded":"2023-06-10T18:44:13.423295+0000","last_fullsized":"2023-06-10T18:44:13.423295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:55.371558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393218+0000","last_change":"2023-06-10T18:44:13.006561+0000","last_active":"2023-06-10T18:44:13.393218+0000","last_peered":"2023-06-10T18:44:13.393218+0000","last_clean":"2023-06-10T18:44:13.393218+0000","last_became_active":"2023-06-10T18:44:13.006171+0000","last_became_peered":"2023-06-10T18:44:13.006171+0000","last_unstale":"2023-06-10T18:44:13.393218+0000","last_undegraded":"2023-06-10T18:44:13.393218+0000","last_fullsized":"2023-06-10T18:44:13.393218+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:18.821445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394377+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:44:13.394377+0000","last_peered":"2023-06-10T18:44:13.394377+0000","last_clean":"2023-06-10T18:44:13.394377+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:44:13.394377+0000","last_undegraded":"2023-06-10T18:44:13.394377+0000","last_fullsized":"2023-06-10T18:44:13.394377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:21.913689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423028+0000","last_change":"2023-06-10T18:44:08.578010+0000","last_active":"2023-06-10T18:44:13.423028+0000","last_peered":"2023-06-10T18:44:13.423028+0000","last_clean":"2023-06-10T18:44:13.423028+0000","last_became_active":"2023-06-10T18:44:08.577727+0000","last_became_peered":"2023-06-10T18:44:08.577727+0000","last_unstale":"2023-06-10T18:44:13.423028+0000","last_undegraded":"2023-06-10T18:44:13.423028+0000","last_fullsized":"2023-06-10T18:44:13.423028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:00:49.837648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423225+0000","last_change":"2023-06-10T18:44:08.990076+0000","last_active":"2023-06-10T18:44:13.423225+0000","last_peered":"2023-06-10T18:44:13.423225+0000","last_clean":"2023-06-10T18:44:13.423225+0000","last_became_active":"2023-06-10T18:44:08.989191+0000","last_became_peered":"2023-06-10T18:44:08.989191+0000","last_unstale":"2023-06-10T18:44:13.423225+0000","last_undegraded":"2023-06-10T18:44:13.423225+0000","last_fullsized":"2023-06-10T18:44:13.423225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:29:35.616295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423712+0000","last_change":"2023-06-10T18:44:12.999386+0000","last_active":"2023-06-10T18:44:13.423712+0000","last_peered":"2023-06-10T18:44:13.423712+0000","last_clean":"2023-06-10T18:44:13.423712+0000","last_became_active":"2023-06-10T18:44:12.998986+0000","last_became_peered":"2023-06-10T18:44:12.998986+0000","last_unstale":"2023-06-10T18:44:13.423712+0000","last_undegraded":"2023-06-10T18:44:13.423712+0000","last_fullsized":"2023-06-10T18:44:13.423712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:10.294554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394138+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:44:13.394138+0000","last_peered":"2023-06-10T18:44:13.394138+0000","last_clean":"2023-06-10T18:44:13.394138+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:44:13.394138+0000","last_undegraded":"2023-06-10T18:44:13.394138+0000","last_fullsized":"2023-06-10T18:44:13.394138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:43.350286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394357+0000","last_change":"2023-06-10T18:44:08.572021+0000","last_active":"2023-06-10T18:44:13.394357+0000","last_peered":"2023-06-10T18:44:13.394357+0000","last_clean":"2023-06-10T18:44:13.394357+0000","last_became_active":"2023-06-10T18:44:08.571716+0000","last_became_peered":"2023-06-10T18:44:08.571716+0000","last_unstale":"2023-06-10T18:44:13.394357+0000","last_undegraded":"2023-06-10T18:44:13.394357+0000","last_fullsized":"2023-06-10T18:44:13.394357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423849+0000","last_change":"2023-06-10T18:44:08.983810+0000","last_active":"2023-06-10T18:44:13.423849+0000","last_peered":"2023-06-10T18:44:13.423849+0000","last_clean":"2023-06-10T18:44:13.423849+0000","last_became_active":"2023-06-10T18:44:08.983539+0000","last_became_peered":"2023-06-10T18:44:08.983539+0000","last_unstale":"2023-06-10T18:44:13.423849+0000","last_undegraded":"2023-06-10T18:44:13.423849+0000","last_fullsized":"2023-06-10T18:44:13.423849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:18.195658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423587+0000","last_change":"2023-06-10T18:44:13.000564+0000","last_active":"2023-06-10T18:44:13.423587+0000","last_peered":"2023-06-10T18:44:13.423587+0000","last_clean":"2023-06-10T18:44:13.423587+0000","last_became_active":"2023-06-10T18:44:13.000141+0000","last_became_peered":"2023-06-10T18:44:13.000141+0000","last_unstale":"2023-06-10T18:44:13.423587+0000","last_undegraded":"2023-06-10T18:44:13.423587+0000","last_fullsized":"2023-06-10T18:44:13.423587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:45.731880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425478+0000","last_change":"2023-06-10T18:44:08.579728+0000","last_active":"2023-06-10T18:44:13.425478+0000","last_peered":"2023-06-10T18:44:13.425478+0000","last_clean":"2023-06-10T18:44:13.425478+0000","last_became_active":"2023-06-10T18:44:08.579431+0000","last_became_peered":"2023-06-10T18:44:08.579431+0000","last_unstale":"2023-06-10T18:44:13.425478+0000","last_undegraded":"2023-06-10T18:44:13.425478+0000","last_fullsized":"2023-06-10T18:44:13.425478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:12.568842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394345+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:44:13.394345+0000","last_peered":"2023-06-10T18:44:13.394345+0000","last_clean":"2023-06-10T18:44:13.394345+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:44:13.394345+0000","last_undegraded":"2023-06-10T18:44:13.394345+0000","last_fullsized":"2023-06-10T18:44:13.394345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:43:05.996450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424774+0000","last_change":"2023-06-10T18:44:08.983770+0000","last_active":"2023-06-10T18:44:13.424774+0000","last_peered":"2023-06-10T18:44:13.424774+0000","last_clean":"2023-06-10T18:44:13.424774+0000","last_became_active":"2023-06-10T18:44:08.983526+0000","last_became_peered":"2023-06-10T18:44:08.983526+0000","last_unstale":"2023-06-10T18:44:13.424774+0000","last_undegraded":"2023-06-10T18:44:13.424774+0000","last_fullsized":"2023-06-10T18:44:13.424774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424718+0000","last_change":"2023-06-10T18:44:13.003992+0000","last_active":"2023-06-10T18:44:13.424718+0000","last_peered":"2023-06-10T18:44:13.424718+0000","last_clean":"2023-06-10T18:44:13.424718+0000","last_became_active":"2023-06-10T18:44:13.003565+0000","last_became_peered":"2023-06-10T18:44:13.003565+0000","last_unstale":"2023-06-10T18:44:13.424718+0000","last_undegraded":"2023-06-10T18:44:13.424718+0000","last_fullsized":"2023-06-10T18:44:13.424718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423787+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:44:13.423787+0000","last_peered":"2023-06-10T18:44:13.423787+0000","last_clean":"2023-06-10T18:44:13.423787+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:44:13.423787+0000","last_undegraded":"2023-06-10T18:44:13.423787+0000","last_fullsized":"2023-06-10T18:44:13.423787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:03:46.107219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423014+0000","last_change":"2023-06-10T18:44:08.576079+0000","last_active":"2023-06-10T18:44:13.423014+0000","last_peered":"2023-06-10T18:44:13.423014+0000","last_clean":"2023-06-10T18:44:13.423014+0000","last_became_active":"2023-06-10T18:44:08.575346+0000","last_became_peered":"2023-06-10T18:44:08.575346+0000","last_unstale":"2023-06-10T18:44:13.423014+0000","last_undegraded":"2023-06-10T18:44:13.423014+0000","last_fullsized":"2023-06-10T18:44:13.423014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:54:23.330172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423207+0000","last_change":"2023-06-10T18:44:08.990722+0000","last_active":"2023-06-10T18:44:13.423207+0000","last_peered":"2023-06-10T18:44:13.423207+0000","last_clean":"2023-06-10T18:44:13.423207+0000","last_became_active":"2023-06-10T18:44:08.988467+0000","last_became_peered":"2023-06-10T18:44:08.988467+0000","last_unstale":"2023-06-10T18:44:13.423207+0000","last_undegraded":"2023-06-10T18:44:13.423207+0000","last_fullsized":"2023-06-10T18:44:13.423207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:10.165150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394392+0000","last_change":"2023-06-10T18:44:13.008834+0000","last_active":"2023-06-10T18:44:13.394392+0000","last_peered":"2023-06-10T18:44:13.394392+0000","last_clean":"2023-06-10T18:44:13.394392+0000","last_became_active":"2023-06-10T18:44:13.008433+0000","last_became_peered":"2023-06-10T18:44:13.008433+0000","last_unstale":"2023-06-10T18:44:13.394392+0000","last_undegraded":"2023-06-10T18:44:13.394392+0000","last_fullsized":"2023-06-10T18:44:13.394392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:40:13.586352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423733+0000","last_change":"2023-06-10T18:44:08.576600+0000","last_active":"2023-06-10T18:44:13.423733+0000","last_peered":"2023-06-10T18:44:13.423733+0000","last_clean":"2023-06-10T18:44:13.423733+0000","last_became_active":"2023-06-10T18:44:08.575635+0000","last_became_peered":"2023-06-10T18:44:08.575635+0000","last_unstale":"2023-06-10T18:44:13.423733+0000","last_undegraded":"2023-06-10T18:44:13.423733+0000","last_fullsized":"2023-06-10T18:44:13.423733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:16:19.906249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423305+0000","last_change":"2023-06-10T18:44:08.985629+0000","last_active":"2023-06-10T18:44:13.423305+0000","last_peered":"2023-06-10T18:44:13.423305+0000","last_clean":"2023-06-10T18:44:13.423305+0000","last_became_active":"2023-06-10T18:44:08.985367+0000","last_became_peered":"2023-06-10T18:44:08.985367+0000","last_unstale":"2023-06-10T18:44:13.423305+0000","last_undegraded":"2023-06-10T18:44:13.423305+0000","last_fullsized":"2023-06-10T18:44:13.423305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:57.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394246+0000","last_change":"2023-06-10T18:44:13.010558+0000","last_active":"2023-06-10T18:44:13.394246+0000","last_peered":"2023-06-10T18:44:13.394246+0000","last_clean":"2023-06-10T18:44:13.394246+0000","last_became_active":"2023-06-10T18:44:13.010123+0000","last_became_peered":"2023-06-10T18:44:13.010123+0000","last_unstale":"2023-06-10T18:44:13.394246+0000","last_undegraded":"2023-06-10T18:44:13.394246+0000","last_fullsized":"2023-06-10T18:44:13.394246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:13.872743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393194+0000","last_change":"2023-06-10T18:44:08.571654+0000","last_active":"2023-06-10T18:44:13.393194+0000","last_peered":"2023-06-10T18:44:13.393194+0000","last_clean":"2023-06-10T18:44:13.393194+0000","last_became_active":"2023-06-10T18:44:08.571363+0000","last_became_peered":"2023-06-10T18:44:08.571363+0000","last_unstale":"2023-06-10T18:44:13.393194+0000","last_undegraded":"2023-06-10T18:44:13.393194+0000","last_fullsized":"2023-06-10T18:44:13.393194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:48.376074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423819+0000","last_change":"2023-06-10T18:44:08.996056+0000","last_active":"2023-06-10T18:44:13.423819+0000","last_peered":"2023-06-10T18:44:13.423819+0000","last_clean":"2023-06-10T18:44:13.423819+0000","last_became_active":"2023-06-10T18:44:08.995902+0000","last_became_peered":"2023-06-10T18:44:08.995902+0000","last_unstale":"2023-06-10T18:44:13.423819+0000","last_undegraded":"2023-06-10T18:44:13.423819+0000","last_fullsized":"2023-06-10T18:44:13.423819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:17.624861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393057+0000","last_change":"2023-06-10T18:44:13.004134+0000","last_active":"2023-06-10T18:44:13.393057+0000","last_peered":"2023-06-10T18:44:13.393057+0000","last_clean":"2023-06-10T18:44:13.393057+0000","last_became_active":"2023-06-10T18:44:13.003596+0000","last_became_peered":"2023-06-10T18:44:13.003596+0000","last_unstale":"2023-06-10T18:44:13.393057+0000","last_undegraded":"2023-06-10T18:44:13.393057+0000","last_fullsized":"2023-06-10T18:44:13.393057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:33.480226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393105+0000","last_change":"2023-06-10T18:44:08.575332+0000","last_active":"2023-06-10T18:44:13.393105+0000","last_peered":"2023-06-10T18:44:13.393105+0000","last_clean":"2023-06-10T18:44:13.393105+0000","last_became_active":"2023-06-10T18:44:08.575065+0000","last_became_peered":"2023-06-10T18:44:08.575065+0000","last_unstale":"2023-06-10T18:44:13.393105+0000","last_undegraded":"2023-06-10T18:44:13.393105+0000","last_fullsized":"2023-06-10T18:44:13.393105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:38.892959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424331+0000","last_change":"2023-06-10T18:44:08.995435+0000","last_active":"2023-06-10T18:44:13.424331+0000","last_peered":"2023-06-10T18:44:13.424331+0000","last_clean":"2023-06-10T18:44:13.424331+0000","last_became_active":"2023-06-10T18:44:08.995141+0000","last_became_peered":"2023-06-10T18:44:08.995141+0000","last_unstale":"2023-06-10T18:44:13.424331+0000","last_undegraded":"2023-06-10T18:44:13.424331+0000","last_fullsized":"2023-06-10T18:44:13.424331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:52.780444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422884+0000","last_change":"2023-06-10T18:44:12.998566+0000","last_active":"2023-06-10T18:44:13.422884+0000","last_peered":"2023-06-10T18:44:13.422884+0000","last_clean":"2023-06-10T18:44:13.422884+0000","last_became_active":"2023-06-10T18:44:12.998185+0000","last_became_peered":"2023-06-10T18:44:12.998185+0000","last_unstale":"2023-06-10T18:44:13.422884+0000","last_undegraded":"2023-06-10T18:44:13.422884+0000","last_fullsized":"2023-06-10T18:44:13.422884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:48:27.535802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425429+0000","last_change":"2023-06-10T18:44:08.576599+0000","last_active":"2023-06-10T18:44:13.425429+0000","last_peered":"2023-06-10T18:44:13.425429+0000","last_clean":"2023-06-10T18:44:13.425429+0000","last_became_active":"2023-06-10T18:44:08.576329+0000","last_became_peered":"2023-06-10T18:44:08.576329+0000","last_unstale":"2023-06-10T18:44:13.425429+0000","last_undegraded":"2023-06-10T18:44:13.425429+0000","last_fullsized":"2023-06-10T18:44:13.425429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:16.618462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.424932+0000","last_change":"2023-06-10T18:44:08.975414+0000","last_active":"2023-06-10T18:44:13.424932+0000","last_peered":"2023-06-10T18:44:13.424932+0000","last_clean":"2023-06-10T18:44:13.424932+0000","last_became_active":"2023-06-10T18:44:08.975156+0000","last_became_peered":"2023-06-10T18:44:08.975156+0000","last_unstale":"2023-06-10T18:44:13.424932+0000","last_undegraded":"2023-06-10T18:44:13.424932+0000","last_fullsized":"2023-06-10T18:44:13.424932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:27:15.400243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.393972+0000","last_change":"2023-06-10T18:44:13.007025+0000","last_active":"2023-06-10T18:44:13.393972+0000","last_peered":"2023-06-10T18:44:13.393972+0000","last_clean":"2023-06-10T18:44:13.393972+0000","last_became_active":"2023-06-10T18:44:13.006654+0000","last_became_peered":"2023-06-10T18:44:13.006654+0000","last_unstale":"2023-06-10T18:44:13.393972+0000","last_undegraded":"2023-06-10T18:44:13.393972+0000","last_fullsized":"2023-06-10T18:44:13.393972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:11.973415+0000","last_clean_scrub_stamp":"2023-06-10T18:44:11.973415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:27.677403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423673+0000","last_change":"2023-06-10T18:44:08.575515+0000","last_active":"2023-06-10T18:44:13.423673+0000","last_peered":"2023-06-10T18:44:13.423673+0000","last_clean":"2023-06-10T18:44:13.423673+0000","last_became_active":"2023-06-10T18:44:08.573539+0000","last_became_peered":"2023-06-10T18:44:08.573539+0000","last_unstale":"2023-06-10T18:44:13.423673+0000","last_undegraded":"2023-06-10T18:44:13.423673+0000","last_fullsized":"2023-06-10T18:44:13.423673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:07.545815+0000","last_clean_scrub_stamp":"2023-06-10T18:44:07.545815+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:57.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422973+0000","last_change":"2023-06-10T18:44:08.993519+0000","last_active":"2023-06-10T18:44:13.422973+0000","last_peered":"2023-06-10T18:44:13.422973+0000","last_clean":"2023-06-10T18:44:13.422973+0000","last_became_active":"2023-06-10T18:44:08.993146+0000","last_became_peered":"2023-06-10T18:44:08.993146+0000","last_unstale":"2023-06-10T18:44:13.422973+0000","last_undegraded":"2023-06-10T18:44:13.422973+0000","last_fullsized":"2023-06-10T18:44:13.422973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:07.843413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.422934+0000","last_change":"2023-06-10T18:44:08.989855+0000","last_active":"2023-06-10T18:44:13.422934+0000","last_peered":"2023-06-10T18:44:13.422934+0000","last_clean":"2023-06-10T18:44:13.422934+0000","last_became_active":"2023-06-10T18:44:08.989140+0000","last_became_peered":"2023-06-10T18:44:08.989140+0000","last_unstale":"2023-06-10T18:44:13.422934+0000","last_undegraded":"2023-06-10T18:44:13.422934+0000","last_fullsized":"2023-06-10T18:44:13.422934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:14.496637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425109+0000","last_change":"2023-06-10T18:44:08.992461+0000","last_active":"2023-06-10T18:44:13.425109+0000","last_peered":"2023-06-10T18:44:13.425109+0000","last_clean":"2023-06-10T18:44:13.425109+0000","last_became_active":"2023-06-10T18:44:08.992160+0000","last_became_peered":"2023-06-10T18:44:08.992160+0000","last_unstale":"2023-06-10T18:44:13.425109+0000","last_undegraded":"2023-06-10T18:44:13.425109+0000","last_fullsized":"2023-06-10T18:44:13.425109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:42.310678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.423600+0000","last_change":"2023-06-10T18:44:08.991382+0000","last_active":"2023-06-10T18:44:13.423600+0000","last_peered":"2023-06-10T18:44:13.423600+0000","last_clean":"2023-06-10T18:44:13.423600+0000","last_became_active":"2023-06-10T18:44:08.991122+0000","last_became_peered":"2023-06-10T18:44:08.991122+0000","last_unstale":"2023-06-10T18:44:13.423600+0000","last_undegraded":"2023-06-10T18:44:13.423600+0000","last_fullsized":"2023-06-10T18:44:13.423600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:40:10.451541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.425330+0000","last_change":"2023-06-10T18:44:08.987112+0000","last_active":"2023-06-10T18:44:13.425330+0000","last_peered":"2023-06-10T18:44:13.425330+0000","last_clean":"2023-06-10T18:44:13.425330+0000","last_became_active":"2023-06-10T18:44:08.986804+0000","last_became_peered":"2023-06-10T18:44:08.986804+0000","last_unstale":"2023-06-10T18:44:13.425330+0000","last_undegraded":"2023-06-10T18:44:13.425330+0000","last_fullsized":"2023-06-10T18:44:13.425330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:52.743375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-06-10T18:44:13.394328+0000","last_change":"2023-06-10T18:44:08.988415+0000","last_active":"2023-06-10T18:44:13.394328+0000","last_peered":"2023-06-10T18:44:13.394328+0000","last_clean":"2023-06-10T18:44:13.394328+0000","last_became_active":"2023-06-10T18:44:08.987510+0000","last_became_peered":"2023-06-10T18:44:08.987510+0000","last_unstale":"2023-06-10T18:44:13.394328+0000","last_undegraded":"2023-06-10T18:44:13.394328+0000","last_fullsized":"2023-06-10T18:44:13.394328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:08.552213+0000","last_clean_scrub_stamp":"2023-06-10T18:44:08.552213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:30.059024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":913408,"data_stored":714279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":286720,"data_stored":118882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":884736,"data_stored":709832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":290816,"data_stored":119901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.925}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9490000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":122732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":120553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:44:37.648 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-06-10T18:44:37.648 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-06-10T18:44:37.649 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-06-10T18:44:37.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-06-10T18:44:37.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.793+0000 7f133148a700 1 Processor -- start 2023-06-10T18:44:37.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.793+0000 7f133148a700 1 -- start start 2023-06-10T18:44:37.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f133148a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f132c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f133148a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c06bfd0 con 0x7f132c06b820 2023-06-10T18:44:37.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f133148a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c06c2b0 con 0x7f132c0b1e30 2023-06-10T18:44:37.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f133148a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0ccf70 con 0x7f132c0d0c00 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f132b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f132c06b820 0x7f132c0cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52972/0 (socket says 172.21.15.104:52972) 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f132b7fe700 1 -- 172.21.15.104:0/3505106433 learned_addr learned my addr 172.21.15.104:0/3505106433 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f132a7fc700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f132c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349716855 0 0) 0x7f132c06bfd0 con 0x7f132c06b820 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13140035b0 con 0x7f132c06b820 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1561875422 0 0) 0x7f13140035b0 con 0x7f132c06b820 2023-06-10T18:44:37.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 msgr2=0x7f132c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.794+0000 7f1329ffb700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f132c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 >> v1:172.21.15.114:6789/0 conn(0x7f132c0b1e30 legacy=0x7f132c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0cd250 con 0x7f132c06b820 2023-06-10T18:44:37.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f13240019f0 con 0x7f132c06b820 2023-06-10T18:44:37.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 >> v1:172.21.15.104:6789/0 conn(0x7f132c06b820 legacy=0x7f132c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f1314003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1314003fb0 0x7f1314004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314004d90 0x7f1314005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0ccf70 con 0x7f1314003fb0 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131400bba0 con 0x7f1314004d90 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131400bd60 con 0x7f132c0d0c00 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1314003fb0 0x7f1314004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132affd700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f1314003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132b7fe700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314004d90 0x7f1314005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 -- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 msgr2=0x7f1314003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c0d0c00 0x7f1314003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 -- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314004d90 msgr2=0x7f1314005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314004d90 0x7f1314005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 -- 172.21.15.104:0/3505106433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13140035b0 con 0x7f1314003fb0 2023-06-10T18:44:37.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f132a7fc700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1314003fb0 0x7f1314004850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f132c0d0ff0 tx=0x7f1314005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d3fde0d435971 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00e6c0 con 0x7f1314003fb0 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c00e820 con 0x7f1314003fb0 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.795+0000 7f1329ffb700 1 -- 172.21.15.104:0/3505106433 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00ebb0 con 0x7f1314003fb0 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 -- 172.21.15.104:0/3505106433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1314003fb0 msgr2=0x7f1314004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 --2- 172.21.15.104:0/3505106433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1314003fb0 0x7f1314004850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f132c0d0ff0 tx=0x7f1314005880 comp rx=0 tx=0).stop 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f132b7fe700 1 -- 172.21.15.104:0/3505106433 reap_dead start 2023-06-10T18:44:37.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 -- 172.21.15.104:0/3505106433 shutdown_connections 2023-06-10T18:44:37.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 -- 172.21.15.104:0/3505106433 >> 172.21.15.104:0/3505106433 conn(0x7f132c071ec0 msgr2=0x7f132c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:37.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 -- 172.21.15.104:0/3505106433 shutdown_connections 2023-06-10T18:44:37.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.796+0000 7f133148a700 1 -- 172.21.15.104:0/3505106433 wait complete. 2023-06-10T18:44:37.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.797+0000 7f133148a700 1 Processor -- start 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.797+0000 7f133148a700 1 -- start start 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 0x7f132c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 0x7f132c0cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0cbd70 con 0x7f132c0cc740 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c156980 con 0x7f1314003fb0 2023-06-10T18:44:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f133148a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0cb160 con 0x7f132c156b70 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55154/0 (socket says 172.21.15.104:55154) 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 -- 172.21.15.104:0/790344170 learned_addr learned my addr 172.21.15.104:0/790344170 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132a7fc700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 0x7f132c0cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132affd700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 0x7f132c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 -- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 msgr2=0x7f132c0cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 0x7f132c0cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 -- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 msgr2=0x7f132c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 0x7f132c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 -- 172.21.15.104:0/790344170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0cb440 con 0x7f132c0cc740 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132a7fc700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 0x7f132c0cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132affd700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 0x7f132c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.798+0000 7f132b7fe700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1324001330 tx=0x7f132400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.799+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132400ae50 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.799+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13240133f0 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.799+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132c0cb720 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.799+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13240137a0 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.799+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132c0cadf0 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.800+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132c0d1860 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.800+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1324013900 con 0x7f132c0cc740 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.800+0000 7f13137fe700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 0x7f131806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.800+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f132408a300 con 0x7f132c0cc740 2023-06-10T18:44:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.801+0000 7f132affd700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 0x7f131806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.801+0000 7f132affd700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 0x7f131806d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:37.805+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1324055150 con 0x7f132c0cc740 2023-06-10T18:44:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.035+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f132c074050 con 0x7f132c0cc740 2023-06-10T18:44:38.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.035+0000 7f13137fe700 1 -- 172.21.15.104:0/790344170 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1324017040 con 0x7f132c0cc740 2023-06-10T18:44:38.041 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:38.041 INFO:teuthology.orchestra.run.smithi104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-06-10T18:44:38.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 msgr2=0x7f131806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 0x7f131806d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320009040 comp rx=0 tx=0).stop 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 msgr2=0x7f132c0af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1324001330 tx=0x7f132400a620 comp rx=0 tx=0).stop 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 shutdown_connections 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f131806ace0 0x7f131806d190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1314003fb0 0x7f132c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f132c0cc740 0x7f132c0af6b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 --2- 172.21.15.104:0/790344170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132c156b70 0x7f132c0cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 >> 172.21.15.104:0/790344170 conn(0x7f132c071ec0 msgr2=0x7f132c0b8290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 shutdown_connections 2023-06-10T18:44:38.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.037+0000 7f133148a700 1 -- 172.21.15.104:0/790344170 wait complete. 2023-06-10T18:44:38.054 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-06-10T18:44:38.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:38.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.167+0000 7fb245c88700 1 Processor -- start 2023-06-10T18:44:38.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 -- start start 2023-06-10T18:44:38.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400aeaa0 0x7fb2400cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2400b1080 0x7fb2400cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400c9e60 con 0x7fb2400ca2f0 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400c9bc0 con 0x7fb2400b1080 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb245c88700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400aee70 con 0x7fb2400aeaa0 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb23effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2400b1080 0x7fb2400cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb23ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb2400ca2f0 0x7fb2400ca750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52990/0 (socket says 172.21.15.104:52990) 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb23effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2400b1080 0x7fb2400cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55160/0 (socket says 172.21.15.104:55160) 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.168+0000 7fb23ffff700 1 -- 172.21.15.104:0/3955648438 learned_addr learned my addr 172.21.15.104:0/3955648438 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:38.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23f7fe700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400aeaa0 0x7fb2400cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23effd700 1 -- 172.21.15.104:0/3955648438 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400aeaa0 msgr2=0x7fb2400cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23effd700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400aeaa0 0x7fb2400cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23effd700 1 -- 172.21.15.104:0/3955648438 >> v1:172.21.15.104:6789/0 conn(0x7fb2400ca2f0 legacy=0x7fb2400ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23effd700 1 -- 172.21.15.104:0/3955648438 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24006e540 con 0x7fb2400b1080 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23e7fc700 1 -- 172.21.15.104:0/3955648438 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470571670 0 0) 0x7fb2400c9e60 con 0x7fb2400ca2f0 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23effd700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400cd680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb22800bae0 tx=0x7fb22800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4279593b504d5f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23e7fc700 1 -- 172.21.15.104:0/3955648438 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22800dc70 con 0x7fb2400b1080 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23e7fc700 1 -- 172.21.15.104:0/3955648438 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22800ddd0 con 0x7fb2400b1080 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.169+0000 7fb23e7fc700 1 -- 172.21.15.104:0/3955648438 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228010690 con 0x7fb2400b1080 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 -- 172.21.15.104:0/3955648438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 msgr2=0x7fb2400cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400cd680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb22800bae0 tx=0x7fb22800bdf0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 -- 172.21.15.104:0/3955648438 shutdown_connections 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400aeaa0 0x7fb2400cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 --2- 172.21.15.104:0/3955648438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400cd680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 -- 172.21.15.104:0/3955648438 >> 172.21.15.104:0/3955648438 conn(0x7fb24006b8a0 msgr2=0x7fb2400cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 -- 172.21.15.104:0/3955648438 shutdown_connections 2023-06-10T18:44:38.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.170+0000 7fb245c88700 1 -- 172.21.15.104:0/3955648438 wait complete. 2023-06-10T18:44:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.171+0000 7fb245c88700 1 Processor -- start 2023-06-10T18:44:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.171+0000 7fb245c88700 1 -- start start 2023-06-10T18:44:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2400aeaa0 0x7fb2400ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 0x7fb2400c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400c5950 con 0x7fb2400b1080 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400c5bd0 con 0x7fb2400aeaa0 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb245c88700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400c5eb0 con 0x7fb2400ca2f0 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb23effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb23effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55166/0 (socket says 172.21.15.104:55166) 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.172+0000 7fb23effd700 1 -- 172.21.15.104:0/4142961310 learned_addr learned my addr 172.21.15.104:0/4142961310 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:38.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23f7fe700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2400aeaa0 0x7fb2400ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23ffff700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 0x7fb2400c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 -- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 msgr2=0x7fb2400c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 0x7fb2400c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 -- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2400aeaa0 msgr2=0x7fb2400ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2400aeaa0 0x7fb2400ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 -- 172.21.15.104:0/4142961310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22800b740 con 0x7fb2400b1080 2023-06-10T18:44:38.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23ffff700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 0x7fb2400c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb23effd700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b2c0 tx=0x7fb228004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22801f070 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2280039f0 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2400c6640 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228003d60 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.173+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2400c6c00 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.174+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2400714b0 con 0x7fb2400b1080 2023-06-10T18:44:38.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.174+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb228019400 con 0x7fb2400b1080 2023-06-10T18:44:38.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.175+0000 7fb237fff700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 0x7fb22006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.175+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb22808cba0 con 0x7fb2400b1080 2023-06-10T18:44:38.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.175+0000 7fb23f7fe700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 0x7fb22006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.175+0000 7fb23f7fe700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 0x7fb22006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb230009a90 tx=0x7fb230008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.179+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb228057aa0 con 0x7fb2400b1080 2023-06-10T18:44:38.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.366+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2400c6df0 con 0x7fb2400b1080 2023-06-10T18:44:38.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.367+0000 7fb237fff700 1 -- 172.21.15.104:0/4142961310 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7fb22805b0c0 con 0x7fb2400b1080 2023-06-10T18:44:38.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:38.373 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:38.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 msgr2=0x7fb22006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 0x7fb22006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb230009a90 tx=0x7fb230008040 comp rx=0 tx=0).stop 2023-06-10T18:44:38.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 msgr2=0x7fb2400c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b2c0 tx=0x7fb228004750 comp rx=0 tx=0).stop 2023-06-10T18:44:38.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 shutdown_connections 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb22006ace0 0x7fb22006d190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2400aeaa0 0x7fb2400ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2400ca2f0 0x7fb2400c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 --2- 172.21.15.104:0/4142961310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2400b1080 0x7fb2400c3060 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 >> 172.21.15.104:0/4142961310 conn(0x7fb24006b8a0 msgr2=0x7fb2400cb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 shutdown_connections 2023-06-10T18:44:38.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.369+0000 7fb245c88700 1 -- 172.21.15.104:0/4142961310 wait complete. 2023-06-10T18:44:38.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:38.388 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:38.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.505+0000 7f292b67a700 1 Processor -- start 2023-06-10T18:44:38.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.505+0000 7f292b67a700 1 -- start start 2023-06-10T18:44:38.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f292b67a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29241607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f292b67a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f292406b280 0x7f292406a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f292b67a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292406b800 con 0x7f292406ab20 2023-06-10T18:44:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f292b67a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2924160cf0 con 0x7f292406b280 2023-06-10T18:44:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f292b67a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2924069e70 con 0x7f2924145c70 2023-06-10T18:44:38.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f292406b280 0x7f292406a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f292406b280 0x7f292406a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55172/0 (socket says 172.21.15.104:55172) 2023-06-10T18:44:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 -- 172.21.15.104:0/3581208950 learned_addr learned my addr 172.21.15.104:0/3581208950 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2929416700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29241607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 -- 172.21.15.104:0/3581208950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 msgr2=0x7f29241607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29241607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 -- 172.21.15.104:0/3581208950 >> v1:172.21.15.114:6789/0 conn(0x7f292406ab20 legacy=0x7f29240c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 -- 172.21.15.104:0/3581208950 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292406c9a0 con 0x7f292406b280 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2929416700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29241607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.506+0000 7f2928c15700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406b280 0x7f292406a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f29200098f0 tx=0x7f2920009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a41d8a148e39fc5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f291bfff700 1 -- 172.21.15.104:0/3581208950 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920004230 con 0x7f292406b280 2023-06-10T18:44:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f291bfff700 1 -- 172.21.15.104:0/3581208950 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920004390 con 0x7f292406b280 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f291bfff700 1 -- 172.21.15.104:0/3581208950 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29200036a0 con 0x7f292406b280 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 -- 172.21.15.104:0/3581208950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406b280 msgr2=0x7f292406a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406b280 0x7f292406a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f29200098f0 tx=0x7f2920009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 -- 172.21.15.104:0/3581208950 shutdown_connections 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29241607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 --2- 172.21.15.104:0/3581208950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406b280 0x7f292406a5e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 -- 172.21.15.104:0/3581208950 >> 172.21.15.104:0/3581208950 conn(0x7f29240ceec0 msgr2=0x7f29240cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 -- 172.21.15.104:0/3581208950 shutdown_connections 2023-06-10T18:44:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.507+0000 7f292b67a700 1 -- 172.21.15.104:0/3581208950 wait complete. 2023-06-10T18:44:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.508+0000 7f292b67a700 1 Processor -- start 2023-06-10T18:44:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.508+0000 7f292b67a700 1 -- start start 2023-06-10T18:44:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f292406b280 0x7f292415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29240c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292415c050 con 0x7f292406ab20 2023-06-10T18:44:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29240c6d50 con 0x7f292406b280 2023-06-10T18:44:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f292b67a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29240c6a50 con 0x7f2924145c70 2023-06-10T18:44:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f2929416700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f2929416700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55176/0 (socket says 172.21.15.104:55176) 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f2929416700 1 -- 172.21.15.104:0/2740183 learned_addr learned my addr 172.21.15.104:0/2740183 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f2929c17700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29240c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.509+0000 7f2928c15700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f292406b280 0x7f292415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 -- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 msgr2=0x7f29240c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29240c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 -- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f292406b280 msgr2=0x7f292415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f292406b280 0x7f292415bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 -- 172.21.15.104:0/2740183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920009550 con 0x7f292406ab20 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2929416700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f291400c7b0 tx=0x7f291400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914011800 con 0x7f292406ab20 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2914011960 con 0x7f292406ab20 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29240c98a0 con 0x7f292406ab20 2023-06-10T18:44:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914011cd0 con 0x7f292406ab20 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.510+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29240c9e20 con 0x7f292406ab20 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.511+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2924074050 con 0x7f292406ab20 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.511+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2914021350 con 0x7f292406ab20 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.512+0000 7f2919ffb700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 0x7f290806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.512+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f29140896c0 con 0x7f292406ab20 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.512+0000 7f2928c15700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 0x7f290806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.512+0000 7f2928c15700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 0x7f290806d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2920009130 tx=0x7f2920008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.516+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2914057fa0 con 0x7f292406ab20 2023-06-10T18:44:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.703+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f292415c330 con 0x7f292406ab20 2023-06-10T18:44:38.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.703+0000 7f2919ffb700 1 -- 172.21.15.104:0/2740183 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2914057b30 con 0x7f292406ab20 2023-06-10T18:44:38.709 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:38.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 msgr2=0x7f290806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 0x7f290806d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2920009130 tx=0x7f2920008030 comp rx=0 tx=0).stop 2023-06-10T18:44:38.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 msgr2=0x7f29240ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f291400c7b0 tx=0x7f291400cb70 comp rx=0 tx=0).stop 2023-06-10T18:44:38.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 shutdown_connections 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2924145c70 0x7f29240c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f292406ab20 0x7f29240ad7c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f290806ace0 0x7f290806d190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 --2- 172.21.15.104:0/2740183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f292406b280 0x7f292415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 >> 172.21.15.104:0/2740183 conn(0x7f29240ceec0 msgr2=0x7f29240cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 shutdown_connections 2023-06-10T18:44:38.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.705+0000 7f292b67a700 1 -- 172.21.15.104:0/2740183 wait complete. 2023-06-10T18:44:38.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:38.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.839+0000 7f9a81e2b700 1 Processor -- start 2023-06-10T18:44:38.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.839+0000 7f9a81e2b700 1 -- start start 2023-06-10T18:44:38.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a81e2b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a7c06e6e0 0x7f9a7c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a81e2b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a81e2b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c06e250 con 0x7f9a7c15c580 2023-06-10T18:44:38.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a81e2b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c15c950 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a81e2b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0b1e80 con 0x7f9a7c06cde0 2023-06-10T18:44:38.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a7affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a7c06e6e0 0x7f9a7c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a7b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9a7c15c580 0x7f9a7c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53012/0 (socket says 172.21.15.104:53012) 2023-06-10T18:44:38.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.840+0000 7f9a7b7fe700 1 -- 172.21.15.104:0/186150189 learned_addr learned my addr 172.21.15.104:0/186150189 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:38.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7bfff700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7affd700 1 -- 172.21.15.104:0/186150189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 msgr2=0x7f9a7c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:38.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7affd700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7affd700 1 -- 172.21.15.104:0/186150189 >> v1:172.21.15.104:6789/0 conn(0x7f9a7c15c580 legacy=0x7f9a7c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7a7fc700 1 -- 172.21.15.104:0/186150189 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738455046 0 0) 0x7f9a7c06e250 con 0x7f9a7c15c580 2023-06-10T18:44:38.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7affd700 1 -- 172.21.15.104:0/186150189 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c06d920 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7affd700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0b1780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009840 tx=0x7f9a6c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19988a8774fbbd93 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7a7fc700 1 -- 172.21.15.104:0/186150189 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c011830 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7a7fc700 1 -- 172.21.15.104:0/186150189 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c011990 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.841+0000 7f9a7a7fc700 1 -- 172.21.15.104:0/186150189 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c011d20 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 -- 172.21.15.104:0/186150189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c06e6e0 msgr2=0x7f9a7c0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0b1780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009840 tx=0x7f9a6c009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:38.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 -- 172.21.15.104:0/186150189 shutdown_connections 2023-06-10T18:44:38.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/186150189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0b1780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 -- 172.21.15.104:0/186150189 >> 172.21.15.104:0/186150189 conn(0x7f9a7c06b8a0 msgr2=0x7f9a7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:38.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 -- 172.21.15.104:0/186150189 shutdown_connections 2023-06-10T18:44:38.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.842+0000 7f9a81e2b700 1 -- 172.21.15.104:0/186150189 wait complete. 2023-06-10T18:44:38.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.843+0000 7f9a81e2b700 1 Processor -- start 2023-06-10T18:44:38.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.843+0000 7f9a81e2b700 1 -- start start 2023-06-10T18:44:38.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0b3740 con 0x7f9a7c15c580 2023-06-10T18:44:38.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0af180 con 0x7f9a7c06e6e0 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0af420 con 0x7f9a7c06cde0 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55194/0 (socket says 172.21.15.104:55194) 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 -- 172.21.15.104:0/3678944745 learned_addr learned my addr 172.21.15.104:0/3678944745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7affd700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7b7fe700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 -- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 msgr2=0x7f9a7c150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 -- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 msgr2=0x7f9a7c0adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:38.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0adf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:38.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 -- 172.21.15.104:0/3678944745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0094a0 con 0x7f9a7c15c580 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7b7fe700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7affd700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a7bfff700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a74007e40 tx=0x7f9a7400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400eea0 con 0x7f9a7c15c580 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74014420 con 0x7f9a7c15c580 2023-06-10T18:44:38.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c0af700 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a740147b0 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.844+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c0afcc0 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.845+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c074050 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.846+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a74014910 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.846+0000 7f9a63fff700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 0x7f9a6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.846+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9a74089f20 con 0x7f9a7c15c580 2023-06-10T18:44:38.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.846+0000 7f9a7b7fe700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 0x7f9a6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:38.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.847+0000 7f9a7b7fe700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 0x7f9a6806d190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a700016e0 tx=0x7f9a70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:38.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:38.850+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a74054d70 con 0x7f9a7c15c580 2023-06-10T18:44:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.034+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9a7c15b460 con 0x7f9a7c15c580 2023-06-10T18:44:39.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.035+0000 7f9a63fff700 1 -- 172.21.15.104:0/3678944745 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9a74058390 con 0x7f9a7c15c580 2023-06-10T18:44:39.041 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 msgr2=0x7f9a6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 0x7f9a6806d190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a700016e0 tx=0x7f9a70009040 comp rx=0 tx=0).stop 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 msgr2=0x7f9a7c1e50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a74007e40 tx=0x7f9a7400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 shutdown_connections 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a7c15c580 0x7f9a7c1e50f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a7c06cde0 0x7f9a7c150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6806ace0 0x7f9a6806d190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 --2- 172.21.15.104:0/3678944745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c06e6e0 0x7f9a7c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 >> 172.21.15.104:0/3678944745 conn(0x7f9a7c06b8a0 msgr2=0x7f9a7c159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 shutdown_connections 2023-06-10T18:44:39.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.037+0000 7f9a81e2b700 1 -- 172.21.15.104:0/3678944745 wait complete. 2023-06-10T18:44:39.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.171+0000 7fc417b35700 1 Processor -- start 2023-06-10T18:44:39.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.171+0000 7fc417b35700 1 -- start start 2023-06-10T18:44:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc417b35700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4100d0a00 0x7fc410164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc417b35700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4100ad3c0 con 0x7fc4100cea50 2023-06-10T18:44:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc417b35700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc410157f00 con 0x7fc410157b30 2023-06-10T18:44:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc417b35700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4100d0e60 con 0x7fc4100d0a00 2023-06-10T18:44:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc4160d2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4100d0a00 0x7fc410164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc4150d0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc4100cea50 0x7fc4100d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53026/0 (socket says 172.21.15.104:53026) 2023-06-10T18:44:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc4160d2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4100d0a00 0x7fc410164e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55202/0 (socket says 172.21.15.104:55202) 2023-06-10T18:44:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.172+0000 7fc4150d0700 1 -- 172.21.15.104:0/1231401985 learned_addr learned my addr 172.21.15.104:0/1231401985 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:39.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4160d2700 1 -- 172.21.15.104:0/1231401985 >> v1:172.21.15.114:6789/0 conn(0x7fc410157b30 legacy=0x7fc4100ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4160d2700 1 -- 172.21.15.104:0/1231401985 >> v1:172.21.15.104:6789/0 conn(0x7fc4100cea50 legacy=0x7fc4100d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4160d2700 1 -- 172.21.15.104:0/1231401985 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4100b0170 con 0x7fc4100d0a00 2023-06-10T18:44:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4160d2700 1 --2- 172.21.15.104:0/1231401985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4100d0a00 0x7fc410164e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc408009d90 tx=0x7fc40800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acaf3447928b936 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4148cf700 1 -- 172.21.15.104:0/1231401985 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40800e070 con 0x7fc4100d0a00 2023-06-10T18:44:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4148cf700 1 -- 172.21.15.104:0/1231401985 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40800bb90 con 0x7fc4100d0a00 2023-06-10T18:44:39.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc4148cf700 1 -- 172.21.15.104:0/1231401985 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408003a20 con 0x7fc4100d0a00 2023-06-10T18:44:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc417b35700 1 -- 172.21.15.104:0/1231401985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4100d0a00 msgr2=0x7fc410164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc417b35700 1 --2- 172.21.15.104:0/1231401985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4100d0a00 0x7fc410164e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc408009d90 tx=0x7fc40800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc417b35700 1 -- 172.21.15.104:0/1231401985 shutdown_connections 2023-06-10T18:44:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc417b35700 1 --2- 172.21.15.104:0/1231401985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4100d0a00 0x7fc410164e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.173+0000 7fc417b35700 1 -- 172.21.15.104:0/1231401985 >> 172.21.15.104:0/1231401985 conn(0x7fc410089fd0 msgr2=0x7fc41008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.174+0000 7fc417b35700 1 -- 172.21.15.104:0/1231401985 shutdown_connections 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.174+0000 7fc417b35700 1 -- 172.21.15.104:0/1231401985 wait complete. 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 Processor -- start 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 -- start start 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4100cea50 0x7fc4101f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100d0a00 0x7fc410154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410149700 con 0x7fc410157b30 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410154920 con 0x7fc4100cea50 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.175+0000 7fc417b35700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc410153910 con 0x7fc4100d0a00 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55208/0 (socket says 172.21.15.104:55208) 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 -- 172.21.15.104:0/2873958516 learned_addr learned my addr 172.21.15.104:0/2873958516 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4158d1700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4100cea50 0x7fc4101f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4150d0700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100d0a00 0x7fc410154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 -- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100d0a00 msgr2=0x7fc410154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100d0a00 0x7fc410154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 -- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4100cea50 msgr2=0x7fc4101f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4100cea50 0x7fc4101f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 -- 172.21.15.104:0/2873958516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc408009a70 con 0x7fc410157b30 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4160d2700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc410066690 tx=0x7fc408004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408012630 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4080127b0 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc410153b60 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408012b60 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.176+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc410154120 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.177+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc410074050 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.177+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc408012cc0 con 0x7fc410157b30 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.178+0000 7fc4067fc700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.178+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc40808be90 con 0x7fc410157b30 2023-06-10T18:44:39.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.178+0000 7fc4158d1700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.178+0000 7fc4158d1700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc400009870 tx=0x7fc400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.182+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc408056ce0 con 0x7fc410157b30 2023-06-10T18:44:39.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.367+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4100afad0 con 0x7fc410157b30 2023-06-10T18:44:39.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.367+0000 7fc4067fc700 1 -- 172.21.15.104:0/2873958516 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc40805a300 con 0x7fc410157b30 2023-06-10T18:44:39.373 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:39.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 msgr2=0x7fc3f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc400009870 tx=0x7fc400008040 comp rx=0 tx=0).stop 2023-06-10T18:44:39.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 msgr2=0x7fc4100b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc410066690 tx=0x7fc408004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 shutdown_connections 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4100cea50 0x7fc4101f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc410157b30 0x7fc4100b1f40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 --2- 172.21.15.104:0/2873958516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100d0a00 0x7fc410154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 >> 172.21.15.104:0/2873958516 conn(0x7fc410089fd0 msgr2=0x7fc41014c600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 shutdown_connections 2023-06-10T18:44:39.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.369+0000 7fc417b35700 1 -- 172.21.15.104:0/2873958516 wait complete. 2023-06-10T18:44:39.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.503+0000 7fb478092700 1 Processor -- start 2023-06-10T18:44:39.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.503+0000 7fb478092700 1 -- start start 2023-06-10T18:44:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb478092700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb47015c580 0x7fb4700b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb478092700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 0x7fb4700ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb478092700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47006e250 con 0x7fb47006e6e0 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb478092700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47015c950 con 0x7fb47015c580 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb478092700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4700b1e80 con 0x7fb47006cde0 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb47015c580 0x7fb4700b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb47015c580 0x7fb4700b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55212/0 (socket says 172.21.15.104:55212) 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 -- 172.21.15.104:0/256282668 learned_addr learned my addr 172.21.15.104:0/256282668 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:39.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb47662f700 1 --2- 172.21.15.104:0/256282668 >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 0x7fb4700ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 -- 172.21.15.104:0/256282668 >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 msgr2=0x7fb4700ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 --2- 172.21.15.104:0/256282668 >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 0x7fb4700ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 -- 172.21.15.104:0/256282668 >> v1:172.21.15.114:6789/0 conn(0x7fb47006e6e0 legacy=0x7fb4700b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.504+0000 7fb475e2e700 1 -- 172.21.15.104:0/256282668 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47006d920 con 0x7fb47015c580 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb47662f700 1 --2- 172.21.15.104:0/256282668 >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 0x7fb4700ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb475e2e700 1 --2- 172.21.15.104:0/256282668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700b1070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4700cbc60 tx=0x7fb460009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5792026507a7a80 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb474e2c700 1 -- 172.21.15.104:0/256282668 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460005e30 con 0x7fb47015c580 2023-06-10T18:44:39.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb474e2c700 1 -- 172.21.15.104:0/256282668 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460006030 con 0x7fb47015c580 2023-06-10T18:44:39.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb474e2c700 1 -- 172.21.15.104:0/256282668 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4600063e0 con 0x7fb47015c580 2023-06-10T18:44:39.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 -- 172.21.15.104:0/256282668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 msgr2=0x7fb4700b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 --2- 172.21.15.104:0/256282668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700b1070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4700cbc60 tx=0x7fb460009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 -- 172.21.15.104:0/256282668 shutdown_connections 2023-06-10T18:44:39.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 --2- 172.21.15.104:0/256282668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700b1070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 --2- 172.21.15.104:0/256282668 >> v2:172.21.15.114:3300/0 conn(0x7fb47006cde0 0x7fb4700ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 -- 172.21.15.104:0/256282668 >> 172.21.15.104:0/256282668 conn(0x7fb47006b8a0 msgr2=0x7fb4700cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 -- 172.21.15.104:0/256282668 shutdown_connections 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.505+0000 7fb478092700 1 -- 172.21.15.104:0/256282668 wait complete. 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 Processor -- start 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 -- start start 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 0x7fb4700afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 0x7fb4700b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700c4f40 con 0x7fb47015c580 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700c4370 con 0x7fb47006cde0 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb478092700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700b0020 con 0x7fb47006e6e0 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb47662f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.507+0000 7fb47662f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55220/0 (socket says 172.21.15.104:55220) 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 -- 172.21.15.104:0/2992541667 learned_addr learned my addr 172.21.15.104:0/2992541667 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:39.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb475e2e700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 0x7fb4700afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47562d700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 0x7fb4700b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 -- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 msgr2=0x7fb4700b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 0x7fb4700b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 -- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 msgr2=0x7fb4700afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 0x7fb4700afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 -- 172.21.15.104:0/2992541667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460009500 con 0x7fb47015c580 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47562d700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 0x7fb4700b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb475e2e700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 0x7fb4700afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb47662f700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb46800cf20 tx=0x7fb46800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800ae80 con 0x7fb47015c580 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb468015440 con 0x7fb47015c580 2023-06-10T18:44:39.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4700c6220 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4680157d0 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.508+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4700c4880 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.509+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4700c5080 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.509+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb46800d430 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.510+0000 7fb4667fc700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 0x7fb45406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.510+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb468089d80 con 0x7fb47015c580 2023-06-10T18:44:39.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.510+0000 7fb475e2e700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 0x7fb45406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.511+0000 7fb475e2e700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 0x7fb45406d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb4700cbc60 tx=0x7fb460008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.514+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4700c5080 con 0x7fb47015c580 2023-06-10T18:44:39.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.702+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4700c5080 con 0x7fb47015c580 2023-06-10T18:44:39.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.702+0000 7fb4667fc700 1 -- 172.21.15.104:0/2992541667 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb4700c5080 con 0x7fb47015c580 2023-06-10T18:44:39.708 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:39.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 msgr2=0x7fb45406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 0x7fb45406d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb4700cbc60 tx=0x7fb460008070 comp rx=0 tx=0).stop 2023-06-10T18:44:39.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 msgr2=0x7fb4700c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb46800cf20 tx=0x7fb46800a340 comp rx=0 tx=0).stop 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 shutdown_connections 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb45406ace0 0x7fb45406d190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb47006cde0 0x7fb4700afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb47015c580 0x7fb4700c5ce0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.704+0000 7fb478092700 1 --2- 172.21.15.104:0/2992541667 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb47006e6e0 0x7fb4700b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.705+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 >> 172.21.15.104:0/2992541667 conn(0x7fb47006b8a0 msgr2=0x7fb4700ae8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.705+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 shutdown_connections 2023-06-10T18:44:39.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.705+0000 7fb478092700 1 -- 172.21.15.104:0/2992541667 wait complete. 2023-06-10T18:44:39.722 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.838+0000 7fe5ead29700 1 Processor -- start 2023-06-10T18:44:39.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.838+0000 7fe5ead29700 1 -- start start 2023-06-10T18:44:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5ead29700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5e415c580 0x7fe5e40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5ead29700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe5e406cde0 0x7fe5e40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5ead29700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e406e250 con 0x7fe5e406e6e0 2023-06-10T18:44:39.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5ead29700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e415c950 con 0x7fe5e415c580 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5ead29700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e40b1e80 con 0x7fe5e406cde0 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e8ac5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5e415c580 0x7fe5e40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe5e406e6e0 0x7fe5e40b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53054/0 (socket says 172.21.15.104:53054) 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e3fff700 1 -- 172.21.15.104:0/975940712 learned_addr learned my addr 172.21.15.104:0/975940712 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e92c6700 1 --2- 172.21.15.104:0/975940712 >> v2:172.21.15.114:3300/0 conn(0x7fe5e406cde0 0x7fe5e40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/975940712 >> v2:172.21.15.114:3300/0 conn(0x7fe5e406cde0 msgr2=0x7fe5e40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:39.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e8ac5700 1 --2- 172.21.15.104:0/975940712 >> v2:172.21.15.114:3300/0 conn(0x7fe5e406cde0 0x7fe5e40ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/975940712 >> v1:172.21.15.104:6789/0 conn(0x7fe5e406e6e0 legacy=0x7fe5e40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/975940712 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e406d920 con 0x7fe5e415c580 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.839+0000 7fe5e37fe700 1 -- 172.21.15.104:0/975940712 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860295988 0 0) 0x7fe5e406e250 con 0x7fe5e406e6e0 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5e8ac5700 1 --2- 172.21.15.104:0/975940712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e415c580 0x7fe5e40b1070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40cbc60 tx=0x7fe5d8009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7017dd009cfedede server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5e37fe700 1 -- 172.21.15.104:0/975940712 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8005e30 con 0x7fe5e415c580 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5e37fe700 1 -- 172.21.15.104:0/975940712 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8006030 con 0x7fe5e415c580 2023-06-10T18:44:39.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5e37fe700 1 -- 172.21.15.104:0/975940712 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80063c0 con 0x7fe5e415c580 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 -- 172.21.15.104:0/975940712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e415c580 msgr2=0x7fe5e40b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 --2- 172.21.15.104:0/975940712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e415c580 0x7fe5e40b1070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40cbc60 tx=0x7fe5d8009b00 comp rx=0 tx=0).stop 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 -- 172.21.15.104:0/975940712 shutdown_connections 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 --2- 172.21.15.104:0/975940712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e415c580 0x7fe5e40b1070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 --2- 172.21.15.104:0/975940712 >> v2:172.21.15.114:3300/0 conn(0x7fe5e406cde0 0x7fe5e40ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 -- 172.21.15.104:0/975940712 >> 172.21.15.104:0/975940712 conn(0x7fe5e406b8a0 msgr2=0x7fe5e40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 -- 172.21.15.104:0/975940712 shutdown_connections 2023-06-10T18:44:39.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.840+0000 7fe5ead29700 1 -- 172.21.15.104:0/975940712 wait complete. 2023-06-10T18:44:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.842+0000 7fe5ead29700 1 Processor -- start 2023-06-10T18:44:39.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.842+0000 7fe5ead29700 1 -- start start 2023-06-10T18:44:39.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e406e6e0 0x7fe5e4147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 0x7fe5e4147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40b2960 con 0x7fe5e406cde0 2023-06-10T18:44:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40ad680 con 0x7fe5e415c580 2023-06-10T18:44:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5ead29700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41472b0 con 0x7fe5e406e6e0 2023-06-10T18:44:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55230/0 (socket says 172.21.15.104:55230) 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/1193003660 learned_addr learned my addr 172.21.15.104:0/1193003660 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e92c6700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 0x7fe5e4147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e3fff700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e406e6e0 0x7fe5e4147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e406e6e0 msgr2=0x7fe5e4147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e406e6e0 0x7fe5e4147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 msgr2=0x7fe5e4147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 0x7fe5e4147ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e8ac5700 1 -- 172.21.15.104:0/1193003660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8009430 con 0x7fe5e406cde0 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.843+0000 7fe5e92c6700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 0x7fe5e4147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5e8ac5700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009fd0 tx=0x7fe5d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8004040 con 0x7fe5e406cde0 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8020070 con 0x7fe5e406cde0 2023-06-10T18:44:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e40adb10 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80043e0 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.844+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e40ad030 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.845+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e40ad960 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.845+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe5d8022030 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.845+0000 7fe5e17fa700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 0x7fe5c806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.845+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe5d808ae70 con 0x7fe5e406cde0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.846+0000 7fe5e3fff700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 0x7fe5c806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.846+0000 7fe5e3fff700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 0x7fe5c806d1c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5e41e6d90 tx=0x7fe5d0007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:39.850+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe5d8055c30 con 0x7fe5e406cde0 2023-06-10T18:44:40.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.035+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe5e4066f10 con 0x7fe5e406cde0 2023-06-10T18:44:40.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.035+0000 7fe5e17fa700 1 -- 172.21.15.104:0/1193003660 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe5d8059250 con 0x7fe5e406cde0 2023-06-10T18:44:40.041 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 msgr2=0x7fe5c806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 0x7fe5c806d1c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5e41e6d90 tx=0x7fe5d0007c00 comp rx=0 tx=0).stop 2023-06-10T18:44:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 msgr2=0x7fe5e40b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009fd0 tx=0x7fe5d8004750 comp rx=0 tx=0).stop 2023-06-10T18:44:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 shutdown_connections 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c806ad10 0x7fe5c806d1c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e415c580 0x7fe5e4147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5e406cde0 0x7fe5e40b2420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 --2- 172.21.15.104:0/1193003660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e406e6e0 0x7fe5e4147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 >> 172.21.15.104:0/1193003660 conn(0x7fe5e406b8a0 msgr2=0x7fe5e40c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 shutdown_connections 2023-06-10T18:44:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.037+0000 7fe5ead29700 1 -- 172.21.15.104:0/1193003660 wait complete. 2023-06-10T18:44:40.055 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:40.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.168+0000 7f787e38a700 1 Processor -- start 2023-06-10T18:44:40.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.168+0000 7f787e38a700 1 -- start start 2023-06-10T18:44:40.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787e38a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78780cf100 0x7f787806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787e38a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787806d290 con 0x7f787806e0e0 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787e38a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78780ca2d0 con 0x7f787806e4b0 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787e38a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7878071640 con 0x7f78780cf100 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787c927700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78780cf100 0x7f787806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f7877fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f787806e0e0 0x7f78780c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53066/0 (socket says 172.21.15.104:53066) 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f7877fff700 1 -- 172.21.15.104:0/3528740325 learned_addr learned my addr 172.21.15.104:0/3528740325 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:40.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787c927700 1 -- 172.21.15.104:0/3528740325 >> v1:172.21.15.114:6789/0 conn(0x7f787806e4b0 legacy=0x7f78780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787c927700 1 -- 172.21.15.104:0/3528740325 >> v1:172.21.15.104:6789/0 conn(0x7f787806e0e0 legacy=0x7f78780c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787c927700 1 -- 172.21.15.104:0/3528740325 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787806d4c0 con 0x7f78780cf100 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f7876ffd700 1 -- 172.21.15.104:0/3528740325 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827349655 0 0) 0x7f787806d290 con 0x7f787806e0e0 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f787c927700 1 --2- 172.21.15.104:0/3528740325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78780cf100 0x7f787806cd50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7870009d90 tx=0x7f787000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77157662e7caa403 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.169+0000 7f7876ffd700 1 -- 172.21.15.104:0/3528740325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000e070 con 0x7f78780cf100 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f7876ffd700 1 -- 172.21.15.104:0/3528740325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000bbd0 con 0x7f78780cf100 2023-06-10T18:44:40.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f7876ffd700 1 -- 172.21.15.104:0/3528740325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870012430 con 0x7f78780cf100 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 -- 172.21.15.104:0/3528740325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78780cf100 msgr2=0x7f787806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 --2- 172.21.15.104:0/3528740325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78780cf100 0x7f787806cd50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7870009d90 tx=0x7f787000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 -- 172.21.15.104:0/3528740325 shutdown_connections 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 --2- 172.21.15.104:0/3528740325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78780cf100 0x7f787806cd50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 -- 172.21.15.104:0/3528740325 >> 172.21.15.104:0/3528740325 conn(0x7f787806b8a0 msgr2=0x7f78780cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 -- 172.21.15.104:0/3528740325 shutdown_connections 2023-06-10T18:44:40.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.170+0000 7f787e38a700 1 -- 172.21.15.104:0/3528740325 wait complete. 2023-06-10T18:44:40.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.171+0000 7f787e38a700 1 Processor -- start 2023-06-10T18:44:40.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.171+0000 7f787e38a700 1 -- start start 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f787806e4b0 0x7f78780c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78780cf100 0x7f787815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c8a30 con 0x7f787806e0e0 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c6b80 con 0x7f787806e4b0 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787e38a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c6e50 con 0x7f78780cf100 2023-06-10T18:44:40.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55244/0 (socket says 172.21.15.104:55244) 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 -- 172.21.15.104:0/1986409521 learned_addr learned my addr 172.21.15.104:0/1986409521 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f787c927700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78780cf100 0x7f787815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f78777fe700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f787806e4b0 0x7f78780c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 -- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78780cf100 msgr2=0x7f787815a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78780cf100 0x7f787815a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 -- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f787806e4b0 msgr2=0x7f78780c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f787806e4b0 0x7f78780c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.172+0000 7f7877fff700 1 -- 172.21.15.104:0/1986409521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870009a70 con 0x7f787806e0e0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f7877fff700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c730 tx=0x7f786c00caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c0074e0 con 0x7f787806e0e0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c007660 con 0x7f787806e0e0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78780c7130 con 0x7f787806e0e0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c005a70 con 0x7f787806e0e0 2023-06-10T18:44:40.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.173+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78780c7410 con 0x7f787806e0e0 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.174+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7878074050 con 0x7f787806e0e0 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.174+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f786c005bd0 con 0x7f787806e0e0 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.174+0000 7f7874ff9700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 0x7f786406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.175+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f786c088cd0 con 0x7f787806e0e0 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.175+0000 7f78777fe700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 0x7f786406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.175+0000 7f78777fe700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 0x7f786406d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f78781f4fe0 tx=0x7f7868009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.179+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f786c053a90 con 0x7f787806e0e0 2023-06-10T18:44:40.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.406+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78780cd7a0 con 0x7f787806e0e0 2023-06-10T18:44:40.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.407+0000 7f7874ff9700 1 -- 172.21.15.104:0/1986409521 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f786c017090 con 0x7f787806e0e0 2023-06-10T18:44:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:40.413 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:44:40.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 msgr2=0x7f786406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 0x7f786406d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f78781f4fe0 tx=0x7f7868009040 comp rx=0 tx=0).stop 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 msgr2=0x7f78780c6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c730 tx=0x7f786c00caf0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 shutdown_connections 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78780cf100 0x7f787815a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f787806e0e0 0x7f78780c6640 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f786406ace0 0x7f786406d190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 --2- 172.21.15.104:0/1986409521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f787806e4b0 0x7f78780c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 >> 172.21.15.104:0/1986409521 conn(0x7f787806b8a0 msgr2=0x7f78780add80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 shutdown_connections 2023-06-10T18:44:40.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.409+0000 7f787e38a700 1 -- 172.21.15.104:0/1986409521 wait complete. 2023-06-10T18:44:40.416 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:44:40.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:40.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.540+0000 7f82f061f700 1 Processor -- start 2023-06-10T18:44:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.540+0000 7f82f061f700 1 -- start start 2023-06-10T18:44:40.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82f061f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82e806e4b0 0x7f82e806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82f061f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82f061f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e806d290 con 0x7f82e806e0e0 2023-06-10T18:44:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82f061f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e8071640 con 0x7f82e806e4b0 2023-06-10T18:44:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82f061f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80aef00 con 0x7f82e8069e70 2023-06-10T18:44:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82e806e4b0 0x7f82e806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82e806e4b0 0x7f82e806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55252/0 (socket says 172.21.15.104:55252) 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 -- 172.21.15.104:0/1360347669 learned_addr learned my addr 172.21.15.104:0/1360347669 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82eebbc700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 -- 172.21.15.104:0/1360347669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 msgr2=0x7f82e80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e80ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 -- 172.21.15.104:0/1360347669 >> v1:172.21.15.114:6789/0 conn(0x7f82e806e0e0 legacy=0x7f82e80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 -- 172.21.15.104:0/1360347669 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80ca2d0 con 0x7f82e806e4b0 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82eebbc700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e80ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82edbba700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e806cd50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098f0 tx=0x7f82d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26111921c0ce1287 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82ed3b9700 1 -- 172.21.15.104:0/1360347669 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8004230 con 0x7f82e806e4b0 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82ed3b9700 1 -- 172.21.15.104:0/1360347669 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d8004390 con 0x7f82e806e4b0 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.541+0000 7f82ed3b9700 1 -- 172.21.15.104:0/1360347669 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d80036a0 con 0x7f82e806e4b0 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 -- 172.21.15.104:0/1360347669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 msgr2=0x7f82e806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e806cd50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f82d80098f0 tx=0x7f82d8009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:40.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 -- 172.21.15.104:0/1360347669 shutdown_connections 2023-06-10T18:44:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e80ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 --2- 172.21.15.104:0/1360347669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e806cd50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 -- 172.21.15.104:0/1360347669 >> 172.21.15.104:0/1360347669 conn(0x7f82e806b8a0 msgr2=0x7f82e80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 -- 172.21.15.104:0/1360347669 shutdown_connections 2023-06-10T18:44:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.542+0000 7f82f061f700 1 -- 172.21.15.104:0/1360347669 wait complete. 2023-06-10T18:44:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.543+0000 7f82f061f700 1 Processor -- start 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.543+0000 7f82f061f700 1 -- start start 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e8149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e806e0e0 0x7f82e81f1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e815af00 con 0x7f82e806e4b0 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80b0090 con 0x7f82e806e0e0 2023-06-10T18:44:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82f061f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e814a4d0 con 0x7f82e8069e70 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55262/0 (socket says 172.21.15.104:55262) 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 -- 172.21.15.104:0/2000508828 learned_addr learned my addr 172.21.15.104:0/2000508828 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82edbba700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e806e0e0 0x7f82e81f1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82ee3bb700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e8149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 -- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 msgr2=0x7f82e8149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e8149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 -- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e806e0e0 msgr2=0x7f82e81f1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e806e0e0 0x7f82e81f1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82eebbc700 1 -- 172.21.15.104:0/2000508828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8009550 con 0x7f82e806e4b0 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.544+0000 7f82ee3bb700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e8149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82eebbc700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f82e000cd50 tx=0x7f82e000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e000f480 con 0x7f82e806e4b0 2023-06-10T18:44:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e000f600 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e80b30b0 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e000f9b0 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.545+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e80b05c0 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.546+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e8074050 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.546+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f82e000b040 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.546+0000 7f82deffd700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 0x7f82cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.547+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f82e0089f70 con 0x7f82e806e4b0 2023-06-10T18:44:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.547+0000 7f82ee3bb700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 0x7f82cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.547+0000 7f82ee3bb700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 0x7f82cc06d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f82e80b1cb0 tx=0x7f82e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.551+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f82e00bc310 con 0x7f82e806e4b0 2023-06-10T18:44:40.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.741+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82e80c8d60 con 0x7f82e806e4b0 2023-06-10T18:44:40.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.742+0000 7f82deffd700 1 -- 172.21.15.104:0/2000508828 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f82e0054dc0 con 0x7f82e806e4b0 2023-06-10T18:44:40.748 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:40.748 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:40.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 msgr2=0x7f82cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 0x7f82cc06d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f82e80b1cb0 tx=0x7f82e4009040 comp rx=0 tx=0).stop 2023-06-10T18:44:40.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 msgr2=0x7f82e809b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f82e000cd50 tx=0x7f82e000a560 comp rx=0 tx=0).stop 2023-06-10T18:44:40.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 shutdown_connections 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82e806e4b0 0x7f82e809b280 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82e8069e70 0x7f82e8149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82cc06ace0 0x7f82cc06d190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 --2- 172.21.15.104:0/2000508828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82e806e0e0 0x7f82e81f1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 >> 172.21.15.104:0/2000508828 conn(0x7f82e806b8a0 msgr2=0x7f82e80c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 shutdown_connections 2023-06-10T18:44:40.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.744+0000 7f82f061f700 1 -- 172.21.15.104:0/2000508828 wait complete. 2023-06-10T18:44:40.762 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:40.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:40.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.879+0000 7f0937d76700 1 Processor -- start 2023-06-10T18:44:40.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.879+0000 7f0937d76700 1 -- start start 2023-06-10T18:44:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0937d76700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0930070a10 0x7f093006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0937d76700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0930143a40 0x7f0930070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0937d76700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300cec40 con 0x7f093015db40 2023-06-10T18:44:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0937d76700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300cca90 con 0x7f0930070a10 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0937d76700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093006d810 con 0x7f0930143a40 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0930070a10 0x7f093006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0930070a10 0x7f093006e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55266/0 (socket says 172.21.15.104:55266) 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 -- 172.21.15.104:0/2842829416 learned_addr learned my addr 172.21.15.104:0/2842829416 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935311700 1 --2- 172.21.15.104:0/2842829416 >> v2:172.21.15.114:3300/0 conn(0x7f0930143a40 0x7f0930070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 -- 172.21.15.104:0/2842829416 >> v2:172.21.15.114:3300/0 conn(0x7f0930143a40 msgr2=0x7f0930070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 --2- 172.21.15.104:0/2842829416 >> v2:172.21.15.114:3300/0 conn(0x7f0930143a40 0x7f0930070de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 -- 172.21.15.104:0/2842829416 >> v1:172.21.15.114:6789/0 conn(0x7f093015db40 legacy=0x7f093015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.880+0000 7f0935b12700 1 -- 172.21.15.104:0/2842829416 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930071320 con 0x7f0930070a10 2023-06-10T18:44:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0935b12700 1 --2- 172.21.15.104:0/2842829416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0930070a10 0x7f093006e560 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09300cda50 tx=0x7f0920009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32beedae5ed1b74b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0934b10700 1 -- 172.21.15.104:0/2842829416 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920004230 con 0x7f0930070a10 2023-06-10T18:44:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0934b10700 1 -- 172.21.15.104:0/2842829416 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920004390 con 0x7f0930070a10 2023-06-10T18:44:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0934b10700 1 -- 172.21.15.104:0/2842829416 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200036a0 con 0x7f0930070a10 2023-06-10T18:44:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 -- 172.21.15.104:0/2842829416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0930070a10 msgr2=0x7f093006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 --2- 172.21.15.104:0/2842829416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0930070a10 0x7f093006e560 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09300cda50 tx=0x7f0920009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 -- 172.21.15.104:0/2842829416 shutdown_connections 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 --2- 172.21.15.104:0/2842829416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0930070a10 0x7f093006e560 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 --2- 172.21.15.104:0/2842829416 >> v2:172.21.15.114:3300/0 conn(0x7f0930143a40 0x7f0930070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 -- 172.21.15.104:0/2842829416 >> 172.21.15.104:0/2842829416 conn(0x7f09300ceec0 msgr2=0x7f09300cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 -- 172.21.15.104:0/2842829416 shutdown_connections 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.881+0000 7f0937d76700 1 -- 172.21.15.104:0/2842829416 wait complete. 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.882+0000 7f0937d76700 1 Processor -- start 2023-06-10T18:44:40.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.882+0000 7f0937d76700 1 -- start start 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0930070a10 0x7f0930158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0930143a40 0x7f093013e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093013d600 con 0x7f093015db40 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093013d8e0 con 0x7f0930070a10 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0937d76700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093013dbc0 con 0x7f0930143a40 2023-06-10T18:44:40.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0936313700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0936313700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55274/0 (socket says 172.21.15.104:55274) 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.883+0000 7f0936313700 1 -- 172.21.15.104:0/3864991379 learned_addr learned my addr 172.21.15.104:0/3864991379 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0935b12700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0930070a10 0x7f0930158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0935311700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0930143a40 0x7f093013e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 -- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0930143a40 msgr2=0x7f093013e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0930143a40 0x7f093013e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 -- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0930070a10 msgr2=0x7f0930158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0930070a10 0x7f0930158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 -- 172.21.15.104:0/3864991379 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920009550 con 0x7f093015db40 2023-06-10T18:44:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0936313700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f092800b990 tx=0x7f092800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092800c9d0 con 0x7f093015db40 2023-06-10T18:44:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092800cb30 con 0x7f093015db40 2023-06-10T18:44:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09300ac2c0 con 0x7f093015db40 2023-06-10T18:44:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928014440 con 0x7f093015db40 2023-06-10T18:44:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.884+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0930159520 con 0x7f093015db40 2023-06-10T18:44:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.885+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09300ac7e0 con 0x7f093015db40 2023-06-10T18:44:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.885+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0928006d50 con 0x7f093015db40 2023-06-10T18:44:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.886+0000 7f09267fc700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 0x7f091406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.886+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0928089220 con 0x7f093015db40 2023-06-10T18:44:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.886+0000 7f0935b12700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 0x7f091406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.887+0000 7f0935b12700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 0x7f091406d190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0920000c00 tx=0x7f0920008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:40.890+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f09300ac7e0 con 0x7f093015db40 2023-06-10T18:44:41.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.076+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09300ac7e0 con 0x7f093015db40 2023-06-10T18:44:41.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.077+0000 7f09267fc700 1 -- 172.21.15.104:0/3864991379 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09300ac7e0 con 0x7f093015db40 2023-06-10T18:44:41.082 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 msgr2=0x7f091406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 0x7f091406d190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0920000c00 tx=0x7f0920008030 comp rx=0 tx=0).stop 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 msgr2=0x7f09300c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f092800b990 tx=0x7f092800bd50 comp rx=0 tx=0).stop 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 shutdown_connections 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f091406ace0 0x7f091406d190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0930070a10 0x7f0930158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f093015db40 0x7f09300c6450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 --2- 172.21.15.104:0/3864991379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0930143a40 0x7f093013e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 >> 172.21.15.104:0/3864991379 conn(0x7f09300ceec0 msgr2=0x7f0930140080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 shutdown_connections 2023-06-10T18:44:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.079+0000 7f0937d76700 1 -- 172.21.15.104:0/3864991379 wait complete. 2023-06-10T18:44:41.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.214+0000 7fb455071700 1 Processor -- start 2023-06-10T18:44:41.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.214+0000 7fb455071700 1 -- start start 2023-06-10T18:44:41.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb455071700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb4500b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb455071700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45006e250 con 0x7fb45006e6e0 2023-06-10T18:44:41.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb455071700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45015c950 con 0x7fb45006cde0 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb455071700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500b1e80 con 0x7fb45015c580 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44e59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb45006e6e0 0x7fb4500b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53108/0 (socket says 172.21.15.104:53108) 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44e59c700 1 -- 172.21.15.104:0/298959771 learned_addr learned my addr 172.21.15.104:0/298959771 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb4500b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233828980 0 0) 0x7fb45006e250 con 0x7fb45006e6e0 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4380035b0 con 0x7fb45006e6e0 2023-06-10T18:44:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212217932 0 0) 0x7fb45015c950 con 0x7fb45006cde0 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45006e250 con 0x7fb45006cde0 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3950354337 0 0) 0x7fb4380035b0 con 0x7fb45006e6e0 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 msgr2=0x7fb4500b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb4500b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 >> v1:172.21.15.114:6789/0 conn(0x7fb45006cde0 legacy=0x7fb4500ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45006d920 con 0x7fb45006e6e0 2023-06-10T18:44:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb444001670 con 0x7fb45006e6e0 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 >> v1:172.21.15.104:6789/0 conn(0x7fb45006e6e0 legacy=0x7fb4500b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb438003bc0 0x7fb438003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438004850 0x7fb438004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb438005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500b1e80 con 0x7fb438003bc0 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43800bac0 con 0x7fb438004850 2023-06-10T18:44:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43800bc80 con 0x7fb45015c580 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.215+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb438003bc0 0x7fb438003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44e59c700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438004850 0x7fb438004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44f59e700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb438005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 -- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 msgr2=0x7fb438005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb45015c580 0x7fb438005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 -- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438004850 msgr2=0x7fb438004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438004850 0x7fb438004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 -- 172.21.15.104:0/298959771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380035b0 con 0x7fb438003bc0 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44e59c700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438004850 0x7fb438004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb438003bc0 0x7fb438003fb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb440009b10 tx=0x7fb438005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1afd7185827d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440005a20 con 0x7fb438003bc0 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb440005b80 con 0x7fb438003bc0 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44dd9b700 1 -- 172.21.15.104:0/298959771 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e6c0 con 0x7fb438003bc0 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb455071700 1 -- 172.21.15.104:0/298959771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb438003bc0 msgr2=0x7fb438003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb455071700 1 --2- 172.21.15.104:0/298959771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb438003bc0 0x7fb438003fb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb440009b10 tx=0x7fb438005880 comp rx=0 tx=0).stop 2023-06-10T18:44:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.216+0000 7fb44f59e700 1 -- 172.21.15.104:0/298959771 reap_dead start 2023-06-10T18:44:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.217+0000 7fb455071700 1 -- 172.21.15.104:0/298959771 shutdown_connections 2023-06-10T18:44:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.217+0000 7fb455071700 1 -- 172.21.15.104:0/298959771 >> 172.21.15.104:0/298959771 conn(0x7fb45006b8a0 msgr2=0x7fb4500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.217+0000 7fb455071700 1 -- 172.21.15.104:0/298959771 shutdown_connections 2023-06-10T18:44:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.217+0000 7fb455071700 1 -- 172.21.15.104:0/298959771 wait complete. 2023-06-10T18:44:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.218+0000 7fb455071700 1 Processor -- start 2023-06-10T18:44:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.218+0000 7fb455071700 1 -- start start 2023-06-10T18:44:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438003bc0 0x7fb4500b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 0x7fb4500c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500c6630 con 0x7fb450159de0 2023-06-10T18:44:41.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500c6840 con 0x7fb438003bc0 2023-06-10T18:44:41.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb455071700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500c6b20 con 0x7fb450159970 2023-06-10T18:44:41.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55294/0 (socket says 172.21.15.104:55294) 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 -- 172.21.15.104:0/2720731139 learned_addr learned my addr 172.21.15.104:0/2720731139 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438003bc0 0x7fb4500b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44e59c700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 0x7fb4500c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 -- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 msgr2=0x7fb4500c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 0x7fb4500c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 -- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438003bc0 msgr2=0x7fb4500b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438003bc0 0x7fb4500b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 -- 172.21.15.104:0/2720731139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500c6e00 con 0x7fb450159de0 2023-06-10T18:44:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44e59c700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 0x7fb4500c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.219+0000 7fb44f59e700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb43800be10 tx=0x7fb44800d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.220+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800dbc0 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.220+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44800dd20 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.220+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4500c70e0 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.220+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448003c40 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.220+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501f56f0 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.221+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4501f5410 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.221+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb4480050d0 con 0x7fb450159de0 2023-06-10T18:44:41.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.221+0000 7fb4377fe700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 0x7fb43c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.221+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb448089ad0 con 0x7fb450159de0 2023-06-10T18:44:41.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.222+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 0x7fb43c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.222+0000 7fb44ed9d700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 0x7fb43c06d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb440000c00 tx=0x7fb440008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.226+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4501f5410 con 0x7fb450159de0 2023-06-10T18:44:41.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.410+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb4501f5410 con 0x7fb450159de0 2023-06-10T18:44:41.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.410+0000 7fb4377fe700 1 -- 172.21.15.104:0/2720731139 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb4501f5410 con 0x7fb450159de0 2023-06-10T18:44:41.416 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:41.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 msgr2=0x7fb43c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 0x7fb43c06d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb440000c00 tx=0x7fb440008030 comp rx=0 tx=0).stop 2023-06-10T18:44:41.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 msgr2=0x7fb4500c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb43800be10 tx=0x7fb44800d660 comp rx=0 tx=0).stop 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 shutdown_connections 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb43c06ace0 0x7fb43c06d190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb438003bc0 0x7fb4500b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb450159de0 0x7fb4500c9060 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.412+0000 7fb455071700 1 --2- 172.21.15.104:0/2720731139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb450159970 0x7fb4500c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.413+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 >> 172.21.15.104:0/2720731139 conn(0x7fb45006b8a0 msgr2=0x7fb4500cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.413+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 shutdown_connections 2023-06-10T18:44:41.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.413+0000 7fb455071700 1 -- 172.21.15.104:0/2720731139 wait complete. 2023-06-10T18:44:41.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:41.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.547+0000 7f6ed70fc700 1 Processor -- start 2023-06-10T18:44:41.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.547+0000 7f6ed70fc700 1 -- start start 2023-06-10T18:44:41.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed70fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6ed006af80 0x7f6ed00d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed70fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 0x7f6ed00cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed70fc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed00ce340 con 0x7f6ed00ccff0 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed70fc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed00cd490 con 0x7f6ed006af80 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed70fc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00d00d0 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ecffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6ed00ccff0 0x7f6ed006b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53128/0 (socket says 172.21.15.104:53128) 2023-06-10T18:44:41.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ecffff700 1 -- 172.21.15.104:0/543624091 learned_addr learned my addr 172.21.15.104:0/543624091 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed5699700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 0x7f6ed00cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.548+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/543624091 >> v2:172.21.15.114:3300/0 conn(0x7f6ed006af80 0x7f6ed00d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770260564 0 0) 0x7f6ed00ce340 con 0x7f6ed00ccff0 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec00035b0 con 0x7f6ed00ccff0 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1181738356 0 0) 0x7f6ec00035b0 con 0x7f6ed00ccff0 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed4e98700 1 -- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 msgr2=0x7f6ed00cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 0x7f6ed00cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed4e98700 1 -- 172.21.15.104:0/543624091 >> v1:172.21.15.104:6789/0 conn(0x7f6ed00ccff0 legacy=0x7f6ed006b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed4e98700 1 -- 172.21.15.104:0/543624091 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed00cd680 con 0x7f6ed006af80 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed5699700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 0x7f6ed00cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed00d0d20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009840 tx=0x7f6ec4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80fa17eb428a7ac5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc001670 con 0x7f6ed006af80 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4004030 con 0x7f6ed006af80 2023-06-10T18:44:41.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.549+0000 7f6ecf7fe700 1 -- 172.21.15.104:0/543624091 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40043a0 con 0x7f6ed006af80 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 -- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 msgr2=0x7f6ed00d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed00d0d20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009840 tx=0x7f6ec4009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 -- 172.21.15.104:0/543624091 shutdown_connections 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ce7d0 0x7f6ed00cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/543624091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed00d0d20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 -- 172.21.15.104:0/543624091 >> 172.21.15.104:0/543624091 conn(0x7f6ed0089fd0 msgr2=0x7f6ed008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 -- 172.21.15.104:0/543624091 shutdown_connections 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.550+0000 7f6ed70fc700 1 -- 172.21.15.104:0/543624091 wait complete. 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.551+0000 7f6ed70fc700 1 Processor -- start 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.551+0000 7f6ed70fc700 1 -- start start 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed015e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 0x7f6ed00c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed00c47e0 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0153ec0 con 0x7f6ed006af80 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed70fc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed015eca0 con 0x7f6ed00ccff0 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55308/0 (socket says 172.21.15.104:55308) 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 -- 172.21.15.104:0/1406624247 learned_addr learned my addr 172.21.15.104:0/1406624247 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ecffff700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 0x7f6ed00c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 -- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 msgr2=0x7f6ed00c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 0x7f6ed00c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 -- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 msgr2=0x7f6ed015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed015e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 -- 172.21.15.104:0/1406624247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8009a70 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ecffff700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 0x7f6ed00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ed5699700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800efc0 tx=0x7f6ec800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.552+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec800cb40 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.553+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec800f040 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.553+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec40094a0 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.553+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8003a20 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.553+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed015f260 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.554+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed006c600 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.554+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6ec8014480 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.554+0000 7f6ecd7fa700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 0x7f6eb406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.555+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6ec808a0b0 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.555+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 0x7f6eb406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.555+0000 7f6ed4e98700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 0x7f6eb406d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4000c00 tx=0x7f6ec4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.559+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6ec8054f00 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.747+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ed015f480 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.747+0000 7f6ecd7fa700 1 -- 172.21.15.104:0/1406624247 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ec8009cd0 con 0x7f6ed00ce7d0 2023-06-10T18:44:41.754 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 msgr2=0x7f6eb406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 0x7f6eb406d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4000c00 tx=0x7f6ec4008030 comp rx=0 tx=0).stop 2023-06-10T18:44:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 msgr2=0x7f6ed0153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800efc0 tx=0x7f6ec800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 shutdown_connections 2023-06-10T18:44:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6eb406ace0 0x7f6eb406d190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ed006af80 0x7f6ed015e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ed00ce7d0 0x7f6ed0153980 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 --2- 172.21.15.104:0/1406624247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ed00ccff0 0x7f6ed00c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 >> 172.21.15.104:0/1406624247 conn(0x7f6ed0089fd0 msgr2=0x7f6ed006efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 shutdown_connections 2023-06-10T18:44:41.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.749+0000 7f6ed70fc700 1 -- 172.21.15.104:0/1406624247 wait complete. 2023-06-10T18:44:41.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:41.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.886+0000 7f80e4c4d700 1 Processor -- start 2023-06-10T18:44:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.886+0000 7f80e4c4d700 1 -- start start 2023-06-10T18:44:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.887+0000 7f80e4c4d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 0x7f80e00cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.887+0000 7f80e4c4d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e00b21c0 0x7f80e00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.887+0000 7f80e4c4d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00b2b40 con 0x7f80e0156760 2023-06-10T18:44:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.887+0000 7f80e4c4d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e00b30a0 con 0x7f80e00b21c0 2023-06-10T18:44:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.887+0000 7f80e4c4d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e015d740 con 0x7f80e015d1a0 2023-06-10T18:44:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e00b21c0 0x7f80e00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e00b21c0 0x7f80e00b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55316/0 (socket says 172.21.15.104:55316) 2023-06-10T18:44:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 -- 172.21.15.104:0/1517556124 learned_addr learned my addr 172.21.15.104:0/1517556124 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 0x7f80e00cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 -- 172.21.15.104:0/1517556124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 msgr2=0x7f80e00cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 0x7f80e00cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 -- 172.21.15.104:0/1517556124 >> v1:172.21.15.114:6789/0 conn(0x7f80e0156760 legacy=0x7f80e00ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 -- 172.21.15.104:0/1517556124 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00cfca0 con 0x7f80e00b21c0 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 0x7f80e00cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80ded9d700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00b2600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009e80 tx=0x7f80d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc7e00a996ed08b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80dd59a700 1 -- 172.21.15.104:0/1517556124 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d800e070 con 0x7f80e00b21c0 2023-06-10T18:44:41.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80dd59a700 1 -- 172.21.15.104:0/1517556124 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d800b980 con 0x7f80e00b21c0 2023-06-10T18:44:41.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.888+0000 7f80dd59a700 1 -- 172.21.15.104:0/1517556124 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d800bd30 con 0x7f80e00b21c0 2023-06-10T18:44:41.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 -- 172.21.15.104:0/1517556124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 msgr2=0x7f80e00b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00b2600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009e80 tx=0x7f80d800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:41.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 -- 172.21.15.104:0/1517556124 shutdown_connections 2023-06-10T18:44:41.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00b2600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/1517556124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e015d1a0 0x7f80e00cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 -- 172.21.15.104:0/1517556124 >> 172.21.15.104:0/1517556124 conn(0x7f80e006e370 msgr2=0x7f80e006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 -- 172.21.15.104:0/1517556124 shutdown_connections 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.889+0000 7f80e4c4d700 1 -- 172.21.15.104:0/1517556124 wait complete. 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.890+0000 7f80e4c4d700 1 Processor -- start 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.890+0000 7f80e4c4d700 1 -- start start 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 0x7f80e00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e015d1a0 0x7f80e01f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e01544c0 con 0x7f80e00b21c0 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e00c53d0 con 0x7f80e015d1a0 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80e4c4d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e01f2e40 con 0x7f80e0156760 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55318/0 (socket says 172.21.15.104:55318) 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 -- 172.21.15.104:0/3725515937 learned_addr learned my addr 172.21.15.104:0/3725515937 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80ded9d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e015d1a0 0x7f80e01f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 0x7f80e00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 -- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 msgr2=0x7f80e00ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 0x7f80e00ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 -- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e015d1a0 msgr2=0x7f80e01f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e015d1a0 0x7f80e01f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 -- 172.21.15.104:0/3725515937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d8009b30 con 0x7f80e00b21c0 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 0x7f80e00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:41.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80de59c700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c4e0 tx=0x7f80d000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.891+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d00052d0 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.892+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0005430 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.892+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d00038c0 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.892+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e00c4850 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.892+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e00c4e10 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.892+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e0074050 con 0x7f80e00b21c0 2023-06-10T18:44:41.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.893+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f80d0003a20 con 0x7f80e00b21c0 2023-06-10T18:44:41.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.893+0000 7f80d6ffd700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 0x7f80c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:41.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.893+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f80d0088b80 con 0x7f80e00b21c0 2023-06-10T18:44:41.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.893+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 0x7f80c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:41.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.894+0000 7f80ddd9b700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 0x7f80c006d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f80e015f1b0 tx=0x7f80c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:41.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:41.898+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f80d00539d0 con 0x7f80e00b21c0 2023-06-10T18:44:42.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.084+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80e0070340 con 0x7f80e00b21c0 2023-06-10T18:44:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.085+0000 7f80d6ffd700 1 -- 172.21.15.104:0/3725515937 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f80d0017020 con 0x7f80e00b21c0 2023-06-10T18:44:42.090 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.086+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 msgr2=0x7f80c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.086+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 0x7f80c006d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f80e015f1b0 tx=0x7f80c8008040 comp rx=0 tx=0).stop 2023-06-10T18:44:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.086+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 msgr2=0x7f80e00c4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.086+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c4e0 tx=0x7f80d000c7f0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 shutdown_connections 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80c006ace0 0x7f80c006d190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e015d1a0 0x7f80e01f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e00b21c0 0x7f80e00c4270 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 --2- 172.21.15.104:0/3725515937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e0156760 0x7f80e00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 >> 172.21.15.104:0/3725515937 conn(0x7f80e006e370 msgr2=0x7f80e006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 shutdown_connections 2023-06-10T18:44:42.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.087+0000 7f80e4c4d700 1 -- 172.21.15.104:0/3725515937 wait complete. 2023-06-10T18:44:42.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:42.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.225+0000 7f0bb5a67700 1 Processor -- start 2023-06-10T18:44:42.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.225+0000 7f0bb5a67700 1 -- start start 2023-06-10T18:44:42.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0bb5a67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0bb5a67700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bb0070a40 0x7f0bb0070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0bb5a67700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb00ceb10 con 0x7f0bb0159f70 2023-06-10T18:44:42.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0bb5a67700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015a340 con 0x7f0bb0070a40 2023-06-10T18:44:42.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0bb5a67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0071400 con 0x7f0bb006e5b0 2023-06-10T18:44:42.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0baeffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0baffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bb0070a40 0x7f0bb0070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0baeffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb015d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46478/0 (socket says 172.21.15.104:46478) 2023-06-10T18:44:42.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.226+0000 7f0baeffd700 1 -- 172.21.15.104:0/554480874 learned_addr learned my addr 172.21.15.104:0/554480874 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0baffff700 1 -- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 msgr2=0x7f0bb015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:42.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0baffff700 1 --2- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0baffff700 1 -- 172.21.15.104:0/554480874 >> v1:172.21.15.114:6789/0 conn(0x7f0bb0159f70 legacy=0x7f0bb00cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0baffff700 1 -- 172.21.15.104:0/554480874 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00ce500 con 0x7f0bb0070a40 2023-06-10T18:44:42.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0baffff700 1 --2- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0070ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009d90 tx=0x7f0ba800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e3582d31d6b72df server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bae7fc700 1 -- 172.21.15.104:0/554480874 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba800e070 con 0x7f0bb0070a40 2023-06-10T18:44:42.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bae7fc700 1 -- 172.21.15.104:0/554480874 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba800bb20 con 0x7f0bb0070a40 2023-06-10T18:44:42.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bae7fc700 1 -- 172.21.15.104:0/554480874 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8012430 con 0x7f0bb0070a40 2023-06-10T18:44:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 -- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 msgr2=0x7f0bb0070ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0070ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009d90 tx=0x7f0ba800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 -- 172.21.15.104:0/554480874 shutdown_connections 2023-06-10T18:44:42.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0070ec0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/554480874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.227+0000 7f0bb5a67700 1 -- 172.21.15.104:0/554480874 >> 172.21.15.104:0/554480874 conn(0x7f0bb00cedc0 msgr2=0x7f0bb00cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.228+0000 7f0bb5a67700 1 -- 172.21.15.104:0/554480874 shutdown_connections 2023-06-10T18:44:42.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.228+0000 7f0bb5a67700 1 -- 172.21.15.104:0/554480874 wait complete. 2023-06-10T18:44:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.229+0000 7f0bb5a67700 1 Processor -- start 2023-06-10T18:44:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.229+0000 7f0bb5a67700 1 -- start start 2023-06-10T18:44:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb0142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014be90 con 0x7f0bb0159f70 2023-06-10T18:44:42.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0142dd0 con 0x7f0bb0070a40 2023-06-10T18:44:42.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0150690 con 0x7f0bb006e5b0 2023-06-10T18:44:42.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baf7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb0142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42834/0 (socket says 172.21.15.104:42834) 2023-06-10T18:44:42.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55334/0 (socket says 172.21.15.104:55334) 2023-06-10T18:44:42.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baeffd700 1 -- 172.21.15.104:0/1871057647 learned_addr learned my addr 172.21.15.104:0/1871057647 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 -- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 msgr2=0x7f0bb0142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb0142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 -- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 msgr2=0x7f0bb0152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 -- 172.21.15.104:0/1871057647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009a70 con 0x7f0bb0159f70 2023-06-10T18:44:42.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baeffd700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:42.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0baffff700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8016000 tx=0x7f0ba8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8012430 con 0x7f0bb0159f70 2023-06-10T18:44:42.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8020070 con 0x7f0bb0159f70 2023-06-10T18:44:42.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80098f0 con 0x7f0bb0159f70 2023-06-10T18:44:42.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb0150970 con 0x7f0bb0159f70 2023-06-10T18:44:42.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.230+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb0150f30 con 0x7f0bb0159f70 2023-06-10T18:44:42.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.231+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb0074050 con 0x7f0bb0159f70 2023-06-10T18:44:42.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.232+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0ba8004040 con 0x7f0bb0159f70 2023-06-10T18:44:42.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.232+0000 7f0ba7fff700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.232+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0ba8022070 con 0x7f0bb0159f70 2023-06-10T18:44:42.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.233+0000 7f0baf7fe700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.233+0000 7f0baf7fe700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0007fd0 tx=0x7f0ba0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.237+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ba8056160 con 0x7f0bb0159f70 2023-06-10T18:44:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.422+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0bb0073020 con 0x7f0bb0159f70 2023-06-10T18:44:42.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.422+0000 7f0ba7fff700 1 -- 172.21.15.104:0/1871057647 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0ba8059780 con 0x7f0bb0159f70 2023-06-10T18:44:42.428 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:42.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 msgr2=0x7f0b9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0007fd0 tx=0x7f0ba0009040 comp rx=0 tx=0).stop 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 msgr2=0x7f0bb01526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8016000 tx=0x7f0ba8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 shutdown_connections 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bb0159f70 0x7f0bb01526a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bb006e5b0 0x7f0bb0142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b9006ace0 0x7f0b9006d190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 --2- 172.21.15.104:0/1871057647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb0070a40 0x7f0bb0152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 >> 172.21.15.104:0/1871057647 conn(0x7f0bb00cedc0 msgr2=0x7f0bb00b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 shutdown_connections 2023-06-10T18:44:42.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.425+0000 7f0bb5a67700 1 -- 172.21.15.104:0/1871057647 wait complete. 2023-06-10T18:44:42.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.557+0000 7f0964e49700 1 Processor -- start 2023-06-10T18:44:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.557+0000 7f0964e49700 1 -- start start 2023-06-10T18:44:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.558+0000 7f0964e49700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f09600ac230 0x7f0960069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.558+0000 7f0964e49700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09600cb860 0x7f096006d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.558+0000 7f0964e49700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096015a680 con 0x7f096015a2b0 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.558+0000 7f0964e49700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600d0510 con 0x7f09600cb860 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.558+0000 7f0964e49700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600ac680 con 0x7f09600ac230 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09600cb860 0x7f096006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09600cb860 0x7f096006d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55340/0 (socket says 172.21.15.104:55340) 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 -- 172.21.15.104:0/1455262826 learned_addr learned my addr 172.21.15.104:0/1455262826 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095dd9b700 1 --2- 172.21.15.104:0/1455262826 >> v2:172.21.15.114:3300/0 conn(0x7f09600ac230 0x7f0960069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 -- 172.21.15.104:0/1455262826 >> v2:172.21.15.114:3300/0 conn(0x7f09600ac230 msgr2=0x7f0960069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 --2- 172.21.15.104:0/1455262826 >> v2:172.21.15.114:3300/0 conn(0x7f09600ac230 0x7f0960069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 -- 172.21.15.104:0/1455262826 >> v1:172.21.15.114:6789/0 conn(0x7f096015a2b0 legacy=0x7f09600cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 -- 172.21.15.104:0/1455262826 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096006a3b0 con 0x7f09600cb860 2023-06-10T18:44:42.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095ed9d700 1 --2- 172.21.15.104:0/1455262826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f096006d860 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f09600700a0 tx=0x7f095800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59617192b34a6fc6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095d59a700 1 -- 172.21.15.104:0/1455262826 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095800e070 con 0x7f09600cb860 2023-06-10T18:44:42.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095d59a700 1 -- 172.21.15.104:0/1455262826 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095800b910 con 0x7f09600cb860 2023-06-10T18:44:42.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.559+0000 7f095d59a700 1 -- 172.21.15.104:0/1455262826 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095800bcc0 con 0x7f09600cb860 2023-06-10T18:44:42.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 -- 172.21.15.104:0/1455262826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 msgr2=0x7f096006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 --2- 172.21.15.104:0/1455262826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f096006d860 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f09600700a0 tx=0x7f095800b330 comp rx=0 tx=0).stop 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 -- 172.21.15.104:0/1455262826 shutdown_connections 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 --2- 172.21.15.104:0/1455262826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f096006d860 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 --2- 172.21.15.104:0/1455262826 >> v2:172.21.15.114:3300/0 conn(0x7f09600ac230 0x7f0960069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 -- 172.21.15.104:0/1455262826 >> 172.21.15.104:0/1455262826 conn(0x7f096006b8a0 msgr2=0x7f09600cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 -- 172.21.15.104:0/1455262826 shutdown_connections 2023-06-10T18:44:42.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.560+0000 7f0964e49700 1 -- 172.21.15.104:0/1455262826 wait complete. 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.561+0000 7f0964e49700 1 Processor -- start 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.561+0000 7f0964e49700 1 -- start start 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09600ac230 0x7f0960151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f096015a2b0 0x7f09600c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600ae540 con 0x7f09600cb860 2023-06-10T18:44:42.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600c3000 con 0x7f096015a2b0 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f0964e49700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09600ae2a0 con 0x7f09600ac230 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55346/0 (socket says 172.21.15.104:55346) 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 -- 172.21.15.104:0/1127724730 learned_addr learned my addr 172.21.15.104:0/1127724730 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095e59c700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09600ac230 0x7f0960151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095ed9d700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f096015a2b0 0x7f09600c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 -- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09600ac230 msgr2=0x7f0960151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09600ac230 0x7f0960151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 -- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f096015a2b0 msgr2=0x7f09600c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f096015a2b0 0x7f09600c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.562+0000 7f095dd9b700 1 -- 172.21.15.104:0/1127724730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958009b30 con 0x7f09600cb860 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f095dd9b700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f095000e9b0 tx=0x7f095000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950009710 con 0x7f09600cb860 2023-06-10T18:44:42.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000ee50 con 0x7f09600cb860 2023-06-10T18:44:42.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09600c32a0 con 0x7f09600cb860 2023-06-10T18:44:42.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000ca50 con 0x7f09600cb860 2023-06-10T18:44:42.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.563+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09600c3860 con 0x7f09600cb860 2023-06-10T18:44:42.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.564+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0960074050 con 0x7f09600cb860 2023-06-10T18:44:42.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.564+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f095000cbb0 con 0x7f09600cb860 2023-06-10T18:44:42.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.565+0000 7f094effd700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.565+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0950089750 con 0x7f09600cb860 2023-06-10T18:44:42.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.565+0000 7f095e59c700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.565+0000 7f095e59c700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0954000fb0 tx=0x7f0954009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.569+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0950057fa0 con 0x7f09600cb860 2023-06-10T18:44:42.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.799+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f096015d620 con 0x7f09600cb860 2023-06-10T18:44:42.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.799+0000 7f094effd700 1 -- 172.21.15.104:0/1127724730 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f095000ce60 con 0x7f09600cb860 2023-06-10T18:44:42.805 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:42.805 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:44:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 msgr2=0x7f094006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0954000fb0 tx=0x7f0954009040 comp rx=0 tx=0).stop 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 msgr2=0x7f09600add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f095000e9b0 tx=0x7f095000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 shutdown_connections 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f096015a2b0 0x7f09600c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f09600ac230 0x7f0960151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 --2- 172.21.15.104:0/1127724730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09600cb860 0x7f09600add60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 >> 172.21.15.104:0/1127724730 conn(0x7f096006b8a0 msgr2=0x7f09600cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 shutdown_connections 2023-06-10T18:44:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.802+0000 7f0964e49700 1 -- 172.21.15.104:0/1127724730 wait complete. 2023-06-10T18:44:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:44:42.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:42.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.933+0000 7f7e80c56700 1 Processor -- start 2023-06-10T18:44:42.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.933+0000 7f7e80c56700 1 -- start start 2023-06-10T18:44:42.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e80c56700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e7c15a420 0x7f7e7c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e80c56700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e80c56700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7c0b6070 con 0x7f7e7c0b4ff0 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e80c56700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b5460 con 0x7f7e7c0cea60 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e80c56700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b5740 con 0x7f7e7c15a420 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e7ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e7ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55352/0 (socket says 172.21.15.104:55352) 2023-06-10T18:44:42.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.934+0000 7f7e7ad9d700 1 -- 172.21.15.104:0/2394194284 learned_addr learned my addr 172.21.15.104:0/2394194284 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/2394194284 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c15a420 0x7f7e7c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7ad9d700 1 -- 172.21.15.104:0/2394194284 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c15a420 msgr2=0x7f7e7c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/2394194284 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c15a420 0x7f7e7c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7ad9d700 1 -- 172.21.15.104:0/2394194284 >> v1:172.21.15.114:6789/0 conn(0x7f7e7c0b4ff0 legacy=0x7f7e7c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7ad9d700 1 -- 172.21.15.104:0/2394194284 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0b5a20 con 0x7f7e7c0cea60 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/2394194284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0b32e0 tx=0x7f7e7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172b7a297214e5ea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7959a700 1 -- 172.21.15.104:0/2394194284 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400e070 con 0x7f7e7c0cea60 2023-06-10T18:44:42.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7959a700 1 -- 172.21.15.104:0/2394194284 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7400b910 con 0x7f7e7c0cea60 2023-06-10T18:44:42.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.935+0000 7f7e7959a700 1 -- 172.21.15.104:0/2394194284 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400bcc0 con 0x7f7e7c0cea60 2023-06-10T18:44:42.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 -- 172.21.15.104:0/2394194284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 msgr2=0x7f7e7c15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 --2- 172.21.15.104:0/2394194284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0b32e0 tx=0x7f7e7400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:42.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 -- 172.21.15.104:0/2394194284 shutdown_connections 2023-06-10T18:44:42.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 --2- 172.21.15.104:0/2394194284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c15a7f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 --2- 172.21.15.104:0/2394194284 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c15a420 0x7f7e7c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 -- 172.21.15.104:0/2394194284 >> 172.21.15.104:0/2394194284 conn(0x7f7e7c089fd0 msgr2=0x7f7e7c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 -- 172.21.15.104:0/2394194284 shutdown_connections 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.936+0000 7f7e80c56700 1 -- 172.21.15.104:0/2394194284 wait complete. 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.937+0000 7f7e80c56700 1 Processor -- start 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.937+0000 7f7e80c56700 1 -- start start 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0b4ff0 0x7f7e7c1e2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 0x7f7e7c06bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1f6330 con 0x7f7e7c0cea60 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1e32a0 con 0x7f7e7c15a420 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e80c56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1e3580 con 0x7f7e7c0b4ff0 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55356/0 (socket says 172.21.15.104:55356) 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 -- 172.21.15.104:0/150516340 learned_addr learned my addr 172.21.15.104:0/150516340 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:42.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 0x7f7e7c06bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0b4ff0 0x7f7e7c1e2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 -- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0b4ff0 msgr2=0x7f7e7c1e2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0b4ff0 0x7f7e7c1e2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 -- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 msgr2=0x7f7e7c06bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 0x7f7e7c06bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 -- 172.21.15.104:0/150516340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74009b30 con 0x7f7e7c0cea60 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 0x7f7e7c06bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:42.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.938+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00c4e0 tx=0x7f7e6c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.939+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0052d0 con 0x7f7e7c0cea60 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.939+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c005430 con 0x7f7e7c0cea60 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.939+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c1e3860 con 0x7f7e7c0cea60 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.939+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0038c0 con 0x7f7e7c0cea60 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.939+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c1e3e20 con 0x7f7e7c0cea60 2023-06-10T18:44:42.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.940+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c074050 con 0x7f7e7c0cea60 2023-06-10T18:44:42.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.940+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7e6c003a20 con 0x7f7e7c0cea60 2023-06-10T18:44:42.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.940+0000 7f7e6affd700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 0x7f7e5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:42.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.940+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7e6c088d10 con 0x7f7e7c0cea60 2023-06-10T18:44:42.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.941+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 0x7f7e5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:42.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.941+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 0x7f7e5c06d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009a10 tx=0x7f7e70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:42.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:42.945+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7e6c053a50 con 0x7f7e7c0cea60 2023-06-10T18:44:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.133+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e7c071920 con 0x7f7e7c0cea60 2023-06-10T18:44:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.134+0000 7f7e6affd700 1 -- 172.21.15.104:0/150516340 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+20168 (secure 0 0 0) 0x7f7e6c057070 con 0x7f7e7c0cea60 2023-06-10T18:44:43.140 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:43.140 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:13.387169+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 msgr2=0x7f7e5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 0x7f7e5c06d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009a10 tx=0x7f7e70008040 comp rx=0 tx=0).stop 2023-06-10T18:44:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 msgr2=0x7f7e7c1f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00c4e0 tx=0x7f7e6c00c8a0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 shutdown_connections 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e5c06ace0 0x7f7e5c06d190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c15a420 0x7f7e7c06bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0b4ff0 0x7f7e7c1e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 --2- 172.21.15.104:0/150516340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0cea60 0x7f7e7c1f5df0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 >> 172.21.15.104:0/150516340 conn(0x7f7e7c089fd0 msgr2=0x7f7e7c0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 shutdown_connections 2023-06-10T18:44:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.136+0000 7f7e80c56700 1 -- 172.21.15.104:0/150516340 wait complete. 2023-06-10T18:44:43.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:44:43.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.268+0000 7fe43766f700 1 Processor -- start 2023-06-10T18:44:43.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.268+0000 7fe43766f700 1 -- start start 2023-06-10T18:44:43.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe43766f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe43008a3d0 0x7fe430167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe43766f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4300cadd0 con 0x7fe4300ce350 2023-06-10T18:44:43.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe43766f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4300cb0b0 con 0x7fe43006b1f0 2023-06-10T18:44:43.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe43766f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300cfeb0 con 0x7fe43008a3d0 2023-06-10T18:44:43.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe434c0a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe4300ce350 0x7fe4300cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53194/0 (socket says 172.21.15.104:53194) 2023-06-10T18:44:43.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe434c0a700 1 -- 172.21.15.104:0/3531451062 learned_addr learned my addr 172.21.15.104:0/3531451062 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:43.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3531451062 >> v2:172.21.15.114:3300/0 conn(0x7fe43008a3d0 0x7fe430167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34714437 0 0) 0x7fe4300cadd0 con 0x7fe4300ce350 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c0035b0 con 0x7fe4300ce350 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655121123 0 0) 0x7fe41c0035b0 con 0x7fe4300ce350 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 >> v2:172.21.15.114:3300/0 conn(0x7fe43008a3d0 msgr2=0x7fe430167940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 --2- 172.21.15.104:0/3531451062 >> v2:172.21.15.114:3300/0 conn(0x7fe43008a3d0 0x7fe430167940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 >> v1:172.21.15.114:6789/0 conn(0x7fe43006b1f0 legacy=0x7fe4300cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300cdeb0 con 0x7fe4300ce350 2023-06-10T18:44:43.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.269+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fe420001670 con 0x7fe4300ce350 2023-06-10T18:44:43.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 >> v1:172.21.15.104:6789/0 conn(0x7fe4300ce350 legacy=0x7fe4300cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c003a00 0x7fe41c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe41c004850 0x7fe41c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 0x7fe41c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cfeb0 con 0x7fe41c005200 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c00c110 con 0x7fe41c004850 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c00c2d0 con 0x7fe41c003a00 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 0x7fe41c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe43540b700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c003a00 0x7fe41c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe434c0a700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe41c004850 0x7fe41c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 -- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c003a00 msgr2=0x7fe41c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:43.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c003a00 0x7fe41c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 -- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe41c004850 msgr2=0x7fe41c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:43.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe41c004850 0x7fe41c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 -- 172.21.15.104:0/3531451062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0035b0 con 0x7fe41c005200 2023-06-10T18:44:43.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe434c0a700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe41c004850 0x7fe41c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:43.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 -- 172.21.15.104:0/3531451062 reap_dead start 2023-06-10T18:44:43.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 0x7fe41c00ba10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe428009fc0 tx=0x7fe41c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad8c2766e2ead32 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:43.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428003a20 con 0x7fe41c005200 2023-06-10T18:44:43.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42800f040 con 0x7fe41c005200 2023-06-10T18:44:43.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.270+0000 7fe427fff700 1 -- 172.21.15.104:0/3531451062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe428003de0 con 0x7fe41c005200 2023-06-10T18:44:43.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 -- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 msgr2=0x7fe41c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 0x7fe41c00ba10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe428009fc0 tx=0x7fe41c005880 comp rx=0 tx=0).stop 2023-06-10T18:44:43.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 -- 172.21.15.104:0/3531451062 shutdown_connections 2023-06-10T18:44:43.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 --2- 172.21.15.104:0/3531451062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe41c005200 0x7fe41c00ba10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 -- 172.21.15.104:0/3531451062 >> 172.21.15.104:0/3531451062 conn(0x7fe430089fd0 msgr2=0x7fe4300d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:43.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 -- 172.21.15.104:0/3531451062 shutdown_connections 2023-06-10T18:44:43.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.271+0000 7fe43766f700 1 -- 172.21.15.104:0/3531451062 wait complete. 2023-06-10T18:44:43.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.272+0000 7fe43766f700 1 Processor -- start 2023-06-10T18:44:43.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.272+0000 7fe43766f700 1 -- start start 2023-06-10T18:44:43.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c005200 0x7fe430152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4300c61f0 0x7fe4300c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c6bd0 con 0x7fe4301e5320 2023-06-10T18:44:43.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe430160200 con 0x7fe4300c61f0 2023-06-10T18:44:43.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300c9fe0 con 0x7fe41c005200 2023-06-10T18:44:43.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55378/0 (socket says 172.21.15.104:55378) 2023-06-10T18:44:43.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 -- 172.21.15.104:0/3164660076 learned_addr learned my addr 172.21.15.104:0/3164660076 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:43.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43540b700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c005200 0x7fe430152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe435c0c700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4300c61f0 0x7fe4300c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 -- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c005200 msgr2=0x7fe430152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c005200 0x7fe430152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 -- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4300c61f0 msgr2=0x7fe4300c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4300c61f0 0x7fe4300c6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 -- 172.21.15.104:0/3164660076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4301e5790 con 0x7fe4301e5320 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe434c0a700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe42000e7b0 tx=0x7fe42000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420010510 con 0x7fe4301e5320 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe420010670 con 0x7fe4301e5320 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4301e5a70 con 0x7fe4301e5320 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.273+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420010a00 con 0x7fe4301e5320 2023-06-10T18:44:43.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.274+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4301e6030 con 0x7fe4301e5320 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.275+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4300cc760 con 0x7fe4301e5320 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.275+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe420010b60 con 0x7fe4301e5320 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.275+0000 7fe425ffb700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 0x7fe41406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.275+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe420088e20 con 0x7fe4301e5320 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.275+0000 7fe43540b700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 0x7fe41406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:43.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.276+0000 7fe43540b700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 0x7fe41406d190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe42c007fd0 tx=0x7fe42c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:43.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.280+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe420053c90 con 0x7fe4301e5320 2023-06-10T18:44:43.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.509+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe430074050 con 0x7fe4301e5320 2023-06-10T18:44:43.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.509+0000 7fe425ffb700 1 -- 172.21.15.104:0/3164660076 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7fe420019020 con 0x7fe4301e5320 2023-06-10T18:44:43.515 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:43.516 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:44:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 msgr2=0x7fe41406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 0x7fe41406d190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe42c007fd0 tx=0x7fe42c009040 comp rx=0 tx=0).stop 2023-06-10T18:44:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 msgr2=0x7fe4300c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:43.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe42000e7b0 tx=0x7fe42000eac0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 shutdown_connections 2023-06-10T18:44:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe41406ace0 0x7fe41406d190 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4300c61f0 0x7fe4300c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe41c005200 0x7fe430152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 --2- 172.21.15.104:0/3164660076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4301e5320 0x7fe4300c9aa0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 >> 172.21.15.104:0/3164660076 conn(0x7fe430089fd0 msgr2=0x7fe43006e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:43.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 shutdown_connections 2023-06-10T18:44:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:43.511+0000 7fe43766f700 1 -- 172.21.15.104:0/3164660076 wait complete. 2023-06-10T18:44:43.521 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:44:43.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:44:08.959124+0000', 'modified': '2023-06-10T18:44:31.543091+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4296': {'gid': 4296, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/3495904966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3495904966}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 3495904966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:44:43.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:44:43.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:44:44.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.048+0000 7f99a95b0700 1 Processor -- start 2023-06-10T18:44:44.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.048+0000 7f99a95b0700 1 -- start start 2023-06-10T18:44:44.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a95b0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 0x7f99a40cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a95b0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f99a40cd870 0x7f99a40ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a95b0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a40b3610 con 0x7f99a415e0f0 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a95b0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40d0000 con 0x7f99a40cd870 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a95b0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40b38d0 con 0x7f99a415a410 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f99a40cd870 0x7f99a40ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a27fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f99a415e0f0 0x7f99a40cd160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53214/0 (socket says 172.21.15.104:53214) 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a27fc700 1 -- 172.21.15.104:0/3687970090 learned_addr learned my addr 172.21.15.104:0/3687970090 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/3687970090 >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 0x7f99a40cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 -- 172.21.15.104:0/3687970090 >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 msgr2=0x7f99a40cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 --2- 172.21.15.104:0/3687970090 >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 0x7f99a40cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 -- 172.21.15.104:0/3687970090 >> v1:172.21.15.104:6789/0 conn(0x7f99a415e0f0 legacy=0x7f99a40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 -- 172.21.15.104:0/3687970090 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a415e6b0 con 0x7f99a40cd870 2023-06-10T18:44:44.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a1ffb700 1 -- 172.21.15.104:0/3687970090 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59273111 0 0) 0x7f99a40b3610 con 0x7f99a415e0f0 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/3687970090 >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 0x7f99a40cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.049+0000 7f99a37fe700 1 --2- 172.21.15.104:0/3687970090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a40ce8b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99a40b2360 tx=0x7f999c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fec053ba10a48c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a1ffb700 1 -- 172.21.15.104:0/3687970090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c00e070 con 0x7f99a40cd870 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a1ffb700 1 -- 172.21.15.104:0/3687970090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00bbd0 con 0x7f99a40cd870 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a1ffb700 1 -- 172.21.15.104:0/3687970090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c012430 con 0x7f99a40cd870 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 -- 172.21.15.104:0/3687970090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 msgr2=0x7f99a40ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 --2- 172.21.15.104:0/3687970090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a40ce8b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99a40b2360 tx=0x7f999c00b330 comp rx=0 tx=0).stop 2023-06-10T18:44:44.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 -- 172.21.15.104:0/3687970090 shutdown_connections 2023-06-10T18:44:44.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 --2- 172.21.15.104:0/3687970090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a40ce8b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 --2- 172.21.15.104:0/3687970090 >> v2:172.21.15.114:3300/0 conn(0x7f99a415a410 0x7f99a40cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 -- 172.21.15.104:0/3687970090 >> 172.21.15.104:0/3687970090 conn(0x7f99a4069e70 msgr2=0x7f99a406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 -- 172.21.15.104:0/3687970090 shutdown_connections 2023-06-10T18:44:44.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.050+0000 7f99a95b0700 1 -- 172.21.15.104:0/3687970090 wait complete. 2023-06-10T18:44:44.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 Processor -- start 2023-06-10T18:44:44.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 -- start start 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a415a410 0x7f99a40b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 0x7f99a40aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41f2200 con 0x7f99a40cd870 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41f24e0 con 0x7f99a415a410 2023-06-10T18:44:44.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a95b0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41f27c0 con 0x7f99a415e0f0 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55390/0 (socket says 172.21.15.104:55390) 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.052+0000 7f99a2ffd700 1 -- 172.21.15.104:0/166804512 learned_addr learned my addr 172.21.15.104:0/166804512 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a27fc700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a415a410 0x7f99a40b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a37fe700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 0x7f99a40aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 -- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 msgr2=0x7f99a40aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 0x7f99a40aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 -- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a415a410 msgr2=0x7f99a40b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a415a410 0x7f99a40b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 -- 172.21.15.104:0/166804512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c009a70 con 0x7f99a40cd870 2023-06-10T18:44:44.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a37fe700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 0x7f99a40aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:44.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f999800e760 tx=0x7f999800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f998b7fe700 1 -- 172.21.15.104:0/166804512 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998013610 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f998b7fe700 1 -- 172.21.15.104:0/166804512 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998013770 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a41f2aa0 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f998b7fe700 1 -- 172.21.15.104:0/166804512 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998013b20 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.053+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a41f1de0 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.054+0000 7f998b7fe700 1 -- 172.21.15.104:0/166804512 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9998013c80 con 0x7f99a40cd870 2023-06-10T18:44:44.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.055+0000 7f998b7fe700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 0x7f999006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.055+0000 7f998b7fe700 1 -- 172.21.15.104:0/166804512 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f99980892c0 con 0x7f99a40cd870 2023-06-10T18:44:44.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.055+0000 7f99a27fc700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 0x7f999006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.055+0000 7f99a27fc700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 0x7f999006d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9994000f60 tx=0x7f9994007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.057+0000 7f99b5735b80 1 Processor -- start 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.057+0000 7f99b5735b80 1 -- start start 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f9f50 0x55785b2fa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f7740 0x55785b324440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b1fec60 con 0x55785b321d80 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b2519f0 con 0x55785b2f9f50 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55785b24ae80 con 0x55785b2f7740 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55400/0 (socket says 172.21.15.104:55400) 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 -- 172.21.15.104:0/878050056 learned_addr learned my addr 172.21.15.104:0/878050056 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a37fe700 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f7740 0x55785b324440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f9f50 0x55785b2fa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 -- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f7740 msgr2=0x55785b324440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f7740 0x55785b324440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 -- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f9f50 msgr2=0x55785b2fa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f9f50 0x55785b2fa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 -- 172.21.15.104:0/878050056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55785b0cc5e0 con 0x55785b321d80 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f99a27fc700 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9994004960 tx=0x7f9994013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13299b26d46c2395 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f9988ff9700 1 -- 172.21.15.104:0/878050056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9994001290 con 0x55785b321d80 2023-06-10T18:44:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f9988ff9700 1 -- 172.21.15.104:0/878050056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99940013f0 con 0x55785b321d80 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.058+0000 7f9988ff9700 1 -- 172.21.15.104:0/878050056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9994001760 con 0x55785b321d80 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 -- 172.21.15.104:0/878050056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 msgr2=0x55785b320820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9994004960 tx=0x7f9994013000 comp rx=0 tx=0).stop 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 -- 172.21.15.104:0/878050056 shutdown_connections 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f9f50 0x55785b2fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f7740 0x55785b324440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 --2- 172.21.15.104:0/878050056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b320820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 -- 172.21.15.104:0/878050056 >> 172.21.15.104:0/878050056 conn(0x55785b2eefc0 msgr2=0x55785b33f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 -- 172.21.15.104:0/878050056 shutdown_connections 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.059+0000 7f99b5735b80 1 -- 172.21.15.104:0/878050056 wait complete. 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 -- start start 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 Processor -- start 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b337ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b33ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b1eaa70 con 0x55785b321d80 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b1ea770 con 0x55785b2f7740 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55785b0ce8d0 con 0x55785b2f9f50 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99b5735b80 10 client.0 _ll_register_callbacks cb 0x55785b1f0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.060+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55408/0 (socket says 172.21.15.104:55408) 2023-06-10T18:44:44.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 -- 172.21.15.104:0/1908624333 learned_addr learned my addr 172.21.15.104:0/1908624333 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b337ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a27fc700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b33ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 -- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 msgr2=0x55785b33ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b33ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 -- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 msgr2=0x55785b337ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b337ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 -- 172.21.15.104:0/1908624333 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994013550 con 0x55785b321d80 2023-06-10T18:44:44.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99a37fe700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f999c009fd0 tx=0x7f999c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738d58fbbb666b9e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c020070 con 0x55785b321d80 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f999c012c80 con 0x55785b321d80 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55785b2ec780 con 0x55785b321d80 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999c0098f0 con 0x55785b321d80 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 10 client.4899 fetch_fsmap learned FSMap version 9 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55785b078d90 con 0x55785b321d80 2023-06-10T18:44:44.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55785b2ec780 con 0x55785b321d80 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f999c004040 con 0x55785b321d80 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 10 client.4899 fetch_fsmap finished waiting for FSMap version 9 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 10 client.4899 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 20 client.4899 populate_metadata read hostname 'smithi104' 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b341cf0 0x55785b339c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 4 client.4899 mds_command: new command op to 4296 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:44:44.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.061+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 --> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55785b243e60 con 0x55785b341cf0 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.062+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b341cf0 0x55785b339c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.062+0000 7f9981ffb700 10 client.4899 ms_handle_connect on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.062+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b341cf0 0x55785b339c30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cf8df741a4a4915 server_cookie=ad7cc749000eb53f in_seq=0 out_seq=0 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f9981ffb700 1 -- 172.21.15.104:0/1908624333 <== mds.0 v2:172.21.15.114:6834/3495904966 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55785b243e60 con 0x55785b341cf0 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f9981ffb700 10 client.4899 handle_command_reply: tid=0 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 client.4899 shutdown 2023-06-10T18:44:44.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 msgr2=0x55785b3417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f999c009fd0 tx=0x7f999c004b40 comp rx=0 tx=0).stop 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 shutdown_connections 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b341cf0 0x55785b339c30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b337ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b3417b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f9981ffb700 0 client.4899 ms_handle_reset on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 --2- 172.21.15.104:0/1908624333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b33ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 >> 172.21.15.104:0/1908624333 conn(0x55785b2eefc0 msgr2=0x55785b320df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 shutdown_connections 2023-06-10T18:44:44.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 1 -- 172.21.15.104:0/1908624333 wait complete. 2023-06-10T18:44:44.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.063+0000 7f99b5735b80 20 client.4899 trim_cache size 0 max 16384 2023-06-10T18:44:44.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.080+0000 7f99b5735b80 1 Processor -- start 2023-06-10T18:44:44.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.080+0000 7f99b5735b80 1 -- start start 2023-06-10T18:44:44.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b34e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b34ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b342b60 con 0x55785b321d80 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b2faf60 con 0x55785b2f7740 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55785b2d63a0 con 0x55785b2f9f50 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55414/0 (socket says 172.21.15.104:55414) 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 -- 172.21.15.104:0/340474006 learned_addr learned my addr 172.21.15.104:0/340474006 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b34e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a27fc700 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b34ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 -- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 msgr2=0x55785b34ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b34ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 -- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 msgr2=0x55785b34e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b34e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.081+0000 7f99a37fe700 1 -- 172.21.15.104:0/340474006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55785b061c10 con 0x55785b321d80 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99a37fe700 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f999c002230 tx=0x7f999c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697a5b6a1a275f14 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99897fa700 1 -- 172.21.15.104:0/340474006 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c012430 con 0x55785b321d80 2023-06-10T18:44:44.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99897fa700 1 -- 172.21.15.104:0/340474006 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00b730 con 0x55785b321d80 2023-06-10T18:44:44.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99897fa700 1 -- 172.21.15.104:0/340474006 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c004550 con 0x55785b321d80 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 -- 172.21.15.104:0/340474006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 msgr2=0x55785b33e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f999c002230 tx=0x7f999c020000 comp rx=0 tx=0).stop 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 -- 172.21.15.104:0/340474006 shutdown_connections 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b34e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b33e9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 --2- 172.21.15.104:0/340474006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b34ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 -- 172.21.15.104:0/340474006 >> 172.21.15.104:0/340474006 conn(0x55785b2eefc0 msgr2=0x55785b2f0490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 -- 172.21.15.104:0/340474006 shutdown_connections 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.082+0000 7f99b5735b80 1 -- 172.21.15.104:0/340474006 wait complete. 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.083+0000 7f99b5735b80 1 -- start start 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.083+0000 7f99b5735b80 1 Processor -- start 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b358100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b39b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b3363e0 con 0x55785b321d80 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55785b320b70 con 0x55785b2f7740 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55785b358640 con 0x55785b2f9f50 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 10 client.0 _ll_register_callbacks cb 0x55785b1f0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b39b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42920/0 (socket says 172.21.15.104:42920) 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a2ffd700 1 -- 172.21.15.104:0/530553780 learned_addr learned my addr 172.21.15.104:0/530553780 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a27fc700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b358100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 -- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 msgr2=0x55785b358100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b358100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 -- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 msgr2=0x55785b357bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 -- 172.21.15.104:0/530553780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55785b078d90 con 0x55785b321d80 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99a37fe700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b39b9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f999c002230 tx=0x7f999c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73153bc621bb5786 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f9981ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c020050 con 0x55785b321d80 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f999c02a250 con 0x55785b321d80 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99940013f0 con 0x55785b321d80 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999c00e040 con 0x55785b321d80 2023-06-10T18:44:44.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.084+0000 7f99b5735b80 10 client.4905 fetch_fsmap learned FSMap version 9 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55785b09f200 con 0x55785b321d80 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99940013f0 con 0x55785b321d80 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f999c02a430 con 0x55785b321d80 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 10 client.4905 fetch_fsmap finished waiting for FSMap version 9 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 10 client.4905 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 20 client.4905 populate_metadata read hostname 'smithi104' 2023-06-10T18:44:44.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b33e9c0 0x55785b33ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 4 client.4905 mds_command: new command op to 4296 tid=0[{"prefix": "status"}] 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 --> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55785b206000 con 0x55785b33e9c0 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b33e9c0 0x55785b33ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f9981ffb700 10 client.4905 ms_handle_connect on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99a2ffd700 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b33e9c0 0x55785b33ed90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7498e4f9f3bbc9 server_cookie=7fce3466f66da61d in_seq=0 out_seq=0 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f9981ffb700 1 -- 172.21.15.104:0/530553780 <== mds.0 v2:172.21.15.114:6834/3495904966 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55785b206000 con 0x55785b33e9c0 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f9981ffb700 10 client.4905 handle_command_reply: tid=0 2023-06-10T18:44:44.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.085+0000 7f99b5735b80 1 client.4905 shutdown 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 msgr2=0x55785b39b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b39b9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f999c002230 tx=0x7f999c009750 comp rx=0 tx=0).stop 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 shutdown_connections 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:6834/3495904966,v1:172.21.15.114:6836/3495904966] conn(0x55785b33e9c0 0x55785b33ed90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55785b2f7740 0x55785b357bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55785b321d80 0x55785b39b9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f9981ffb700 0 client.4905 ms_handle_reset on v2:172.21.15.114:6834/3495904966 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 --2- 172.21.15.104:0/530553780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55785b2f9f50 0x55785b358100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 >> 172.21.15.104:0/530553780 conn(0x55785b2eefc0 msgr2=0x55785b350da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 shutdown_connections 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 1 -- 172.21.15.104:0/530553780 wait complete. 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99b5735b80 20 client.4905 trim_cache size 0 max 16384 2023-06-10T18:44:44.101 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4296, 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:44:44.102 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 35.124198448000001, 2023-06-10T18:44:44.103 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 9, 2023-06-10T18:44:44.103 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 27, 2023-06-10T18:44:44.103 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 27, 2023-06-10T18:44:44.103 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 40.435161180999998 2023-06-10T18:44:44.103 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:44:44.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 msgr2=0x7f999006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 0x7f999006d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9994000f60 tx=0x7f9994007e70 comp rx=0 tx=0).stop 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 msgr2=0x7f99a41598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f999800e760 tx=0x7f999800ea70 comp rx=0 tx=0).stop 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 shutdown_connections 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99a415e0f0 0x7f99a40aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99a40cd870 0x7f99a41598a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f999006acb0 0x7f999006d160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 --2- 172.21.15.104:0/166804512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a415a410 0x7f99a40b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 >> 172.21.15.104:0/166804512 conn(0x7f99a4069e70 msgr2=0x7f99a406fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 shutdown_connections 2023-06-10T18:44:44.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.086+0000 7f99a95b0700 1 -- 172.21.15.104:0/166804512 wait complete. 2023-06-10T18:44:44.106 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-06-10T18:44:44.138 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-06-10T18:44:44.138 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-06-10T18:44:44.138 INFO:tasks.ceph_fuse:client.0 config is {} 2023-06-10T18:44:44.139 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:44:44.139 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:44.139 INFO:tasks.ceph_fuse:client.2 config is {} 2023-06-10T18:44:44.139 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:44:44.139 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:44.140 INFO:tasks.ceph_fuse:client.1 config is {} 2023-06-10T18:44:44.140 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:44:44.140 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:44.140 INFO:tasks.ceph_fuse:client.3 config is {} 2023-06-10T18:44:44.140 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:44:44.141 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:44.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.141 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:44:44.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2023-06-10T18:44:44.236 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2023-06-10T18:44:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:44:44.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.238 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:44:44.256 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-06-10T18:44:44.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-06-10T18:44:44.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.338 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2023-06-10T18:44:44.443 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-06-10T18:44:44.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:44:44.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.559+0000 7f9f56006700 1 Processor -- start 2023-06-10T18:44:44.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.559+0000 7f9f56006700 1 -- start start 2023-06-10T18:44:44.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f56006700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f5015c730 0x7f9f5006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f56006700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 0x7f9f500cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f56006700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5015cb00 con 0x7f9f500cc000 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f56006700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f500aed80 con 0x7f9f5015c730 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f56006700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f500c9c60 con 0x7f9f500cb0c0 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f4f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f5015c730 0x7f9f5006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f4f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9f5015c730 0x7f9f5006e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55426/0 (socket says 172.21.15.104:55426) 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.560+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1435630828 learned_addr learned my addr 172.21.15.104:0/1435630828 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4ffff700 1 --2- 172.21.15.104:0/1435630828 >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 0x7f9f500cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1435630828 >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 msgr2=0x7f9f500cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4f7fe700 1 --2- 172.21.15.104:0/1435630828 >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 0x7f9f500cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1435630828 >> v1:172.21.15.114:6789/0 conn(0x7f9f500cc000 legacy=0x7f9f5006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1435630828 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5006ecd0 con 0x7f9f5015c730 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4ffff700 1 --2- 172.21.15.104:0/1435630828 >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 0x7f9f500cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4f7fe700 1 --2- 172.21.15.104:0/1435630828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f5015c730 0x7f9f5006e790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9f500c9bf0 tx=0x7f9f4400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efe94847b977fb7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4e7fc700 1 -- 172.21.15.104:0/1435630828 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44005e30 con 0x7f9f5015c730 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4e7fc700 1 -- 172.21.15.104:0/1435630828 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4400d8b0 con 0x7f9f5015c730 2023-06-10T18:44:44.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f4e7fc700 1 -- 172.21.15.104:0/1435630828 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400dc40 con 0x7f9f5015c730 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 -- 172.21.15.104:0/1435630828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f5015c730 msgr2=0x7f9f5006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 --2- 172.21.15.104:0/1435630828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f5015c730 0x7f9f5006e790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9f500c9bf0 tx=0x7f9f4400be70 comp rx=0 tx=0).stop 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 -- 172.21.15.104:0/1435630828 shutdown_connections 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 --2- 172.21.15.104:0/1435630828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f5015c730 0x7f9f5006e790 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 --2- 172.21.15.104:0/1435630828 >> v2:172.21.15.114:3300/0 conn(0x7f9f500cb0c0 0x7f9f500cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.561+0000 7f9f56006700 1 -- 172.21.15.104:0/1435630828 >> 172.21.15.104:0/1435630828 conn(0x7f9f5006b8a0 msgr2=0x7f9f500cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.562+0000 7f9f56006700 1 -- 172.21.15.104:0/1435630828 shutdown_connections 2023-06-10T18:44:44.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.562+0000 7f9f56006700 1 -- 172.21.15.104:0/1435630828 wait complete. 2023-06-10T18:44:44.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.563+0000 7f9f56006700 1 Processor -- start 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.563+0000 7f9f56006700 1 -- start start 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500cc000 0x7f9f500b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 0x7f9f5014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500c4ba0 con 0x7f9f500cb0c0 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500b0d40 con 0x7f9f500cc000 2023-06-10T18:44:44.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f50150410 con 0x7f9f5015c730 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55432/0 (socket says 172.21.15.104:55432) 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1692236674 learned_addr learned my addr 172.21.15.104:0/1692236674 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4effd700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500cc000 0x7f9f500b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4ffff700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 0x7f9f5014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 msgr2=0x7f9f5014fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 0x7f9f5014fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500cc000 msgr2=0x7f9f500b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500cc000 0x7f9f500b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 -- 172.21.15.104:0/1692236674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4400b7a0 con 0x7f9f500cb0c0 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4ffff700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 0x7f9f5014fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f4f7fe700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f500b2200 tx=0x7f9f44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4401a650 con 0x7f9f500cb0c0 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f500b12a0 con 0x7f9f500cb0c0 2023-06-10T18:44:44.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4401a7d0 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4401ab80 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.564+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f500c4550 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.565+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f50066f10 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.566+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9f4401ace0 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.566+0000 7f9f37fff700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 0x7f9f3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.566+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9f4408b090 con 0x7f9f500cb0c0 2023-06-10T18:44:44.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.566+0000 7f9f4effd700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 0x7f9f3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:44.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.567+0000 7f9f4effd700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 0x7f9f3c06d190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f500cdea0 tx=0x7f9f40007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:44.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.571+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9f44055e60 con 0x7f9f500cb0c0 2023-06-10T18:44:44.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.800+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9f5015cb00 con 0x7f9f500cb0c0 2023-06-10T18:44:44.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.801+0000 7f9f37fff700 1 -- 172.21.15.104:0/1692236674 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f9f5015cb00 con 0x7f9f500cb0c0 2023-06-10T18:44:44.807 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-10T18:44:44.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 msgr2=0x7f9f3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 0x7f9f3c06d190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f500cdea0 tx=0x7f9f40007ec0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 msgr2=0x7f9f500affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f500b2200 tx=0x7f9f44004750 comp rx=0 tx=0).stop 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 shutdown_connections 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5015c730 0x7f9f5014fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f500cb0c0 0x7f9f500affb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f3c06ace0 0x7f9f3c06d190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 --2- 172.21.15.104:0/1692236674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500cc000 0x7f9f500b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 >> 172.21.15.104:0/1692236674 conn(0x7f9f5006b8a0 msgr2=0x7f9f500ac350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 shutdown_connections 2023-06-10T18:44:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:44.803+0000 7f9f56006700 1 -- 172.21.15.104:0/1692236674 wait complete. 2023-06-10T18:44:44.820 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:44:44.820 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:44:44.820 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:44:44.821 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:44:44.821 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:44.821 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:44:44.821 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:44:44.821 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:44:44.822 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:44:44.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.822 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:44:44.839 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:44.839 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:44.839 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:44.840 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:44:44.841 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42399sec preferred_lft 42399sec 2023-06-10T18:44:44.842 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:44:44.842 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:44.842 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-10T18:44:44.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.843 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:44.843 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:44.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2023-06-10T18:44:44.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2023-06-10T18:44:44.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2023-06-10T18:44:44.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-10T18:44:44.844 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:44.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.954 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:44:44.979 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:44:44.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:44.981 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:44:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:44:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:44:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:44:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:44:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:44:45.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.040 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:45.040 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:45.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:44:45.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:44:45.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:44:45.041 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:45.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.144 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:44:45.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.161 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2023-06-10T18:44:45.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.218 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:45.218 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:45.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:44:45.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-06-10T18:44:45.219 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:45.311 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-06-10T18:44:45.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.311 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:45.312 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:45.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-06-10T18:44:45.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-10T18:44:45.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-06-10T18:44:45.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-06-10T18:44:45.313 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-06-10T18:44:45.313 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:45.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.510 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:45.511 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:45.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:44:45.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2023-06-10T18:44:45.511 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:45.621 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:44:45.622 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:45.622 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:45.641 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:44:45.641 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:45.642 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:45.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:45.766 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:45.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:45.832 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:45.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:45.834 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:45.891 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:44:45.891 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:44:45.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:45.959 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:45.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:46.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:44:46.014+0000 7f61095215c0 -1 init, newargv = 0x5627cd649ca0 newargc=15 2023-06-10T18:44:46.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[111951]: starting ceph client 2023-06-10T18:44:46.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[111951]: starting fuse 2023-06-10T18:44:46.026 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:46.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:46.028 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:46.084 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:44:46.085 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:44:46.085 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:44:46.086 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:44:46.086 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:44:46.086 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:44:46.087 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:44:46.087 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:46.087 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:44:46.088 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:44:46.088 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:44:46.089 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:44:46.089 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:46.089 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:44:46.089 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:44:46.090 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:44:46.090 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:44:46.091 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:46.091 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:44:46.091 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:44:46.092 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:44:46.092 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:44:46.093 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:44:46.093 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:44:46.094 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:44:46.095 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:44:46.095 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:46.098 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:44:46.098 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:44:46.205 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.111951.asok 2023-06-10T18:44:46.388 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.111951.asok 2023-06-10T18:44:46.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:46.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.111951.asok status 2023-06-10T18:44:46.544 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:44:46.544 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:44:46.544 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:44:46.544 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:44:46.544 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "111951", 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:44:46.545 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4917, 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4917 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:46.546 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2400238732 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:46.547 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2400238732 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.4917 192.168.0.1:0/2400238732", 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2400238732", 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 9, 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 27, 2023-06-10T18:44:46.548 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:44:46.549 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:44:46.549 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:44:46.549 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:44:46.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:44:46.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.668+0000 7feab5caa700 1 Processor -- start 2023-06-10T18:44:46.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.668+0000 7feab5caa700 1 -- start start 2023-06-10T18:44:46.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feab5caa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feab00b4ff0 0x7feab00ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feab5caa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feab5caa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00cecf0 con 0x7feab015a420 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feab5caa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab00b6070 con 0x7feab00b4ff0 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feab5caa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00b1d70 con 0x7feab00ce890 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feab00b4ff0 0x7feab00ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feab00b4ff0 0x7feab00ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:42964/0 (socket says 172.21.15.104:42964) 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 -- 172.21.15.104:0/2740622488 learned_addr learned my addr 172.21.15.104:0/2740622488 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:46.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaf7fe700 1 --1- 172.21.15.104:0/2740622488 >> v1:172.21.15.114:6789/0 conn(0x7feab015a420 0x7feab00b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59134/0 (socket says 172.21.15.104:59134) 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 -- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 msgr2=0x7feab015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 --2- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 -- 172.21.15.104:0/2740622488 >> v1:172.21.15.114:6789/0 conn(0x7feab015a420 legacy=0x7feab00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.669+0000 7feaaeffd700 1 -- 172.21.15.104:0/2740622488 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00b5460 con 0x7feab00b4ff0 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feaaeffd700 1 --2- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab00b4ff0 0x7feab00ce350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feaa00098f0 tx=0x7feaa0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65f094e393f58fc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feaae7fc700 1 -- 172.21.15.104:0/2740622488 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0011830 con 0x7feab00b4ff0 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feaae7fc700 1 -- 172.21.15.104:0/2740622488 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0011990 con 0x7feab00b4ff0 2023-06-10T18:44:46.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feaae7fc700 1 -- 172.21.15.104:0/2740622488 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0011d20 con 0x7feab00b4ff0 2023-06-10T18:44:46.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 -- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab00b4ff0 msgr2=0x7feab00ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 --2- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab00b4ff0 0x7feab00ce350 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feaa00098f0 tx=0x7feaa0009c00 comp rx=0 tx=0).stop 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 -- 172.21.15.104:0/2740622488 shutdown_connections 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 --2- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 --2- 172.21.15.104:0/2740622488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab00b4ff0 0x7feab00ce350 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 -- 172.21.15.104:0/2740622488 >> 172.21.15.104:0/2740622488 conn(0x7feab0089fd0 msgr2=0x7feab00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 -- 172.21.15.104:0/2740622488 shutdown_connections 2023-06-10T18:44:46.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.670+0000 7feab5caa700 1 -- 172.21.15.104:0/2740622488 wait complete. 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.672+0000 7feab5caa700 1 Processor -- start 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.672+0000 7feab5caa700 1 -- start start 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab00c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab015a420 0x7feab00c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab014c080 con 0x7feab00b4ff0 2023-06-10T18:44:46.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab006bd30 con 0x7feab015a420 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00c4ee0 con 0x7feab00ce890 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55466/0 (socket says 172.21.15.104:55466) 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 -- 172.21.15.104:0/3181967495 learned_addr learned my addr 172.21.15.104:0/3181967495 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaeffd700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab00c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaffff700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab015a420 0x7feab00c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 -- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 msgr2=0x7feab00c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab00c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 -- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab015a420 msgr2=0x7feab00c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab015a420 0x7feab00c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 -- 172.21.15.104:0/3181967495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa0009550 con 0x7feab00b4ff0 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feaaf7fe700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c530 tx=0x7feaa400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa400cea0 con 0x7feab00b4ff0 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa4005cb0 con 0x7feab00b4ff0 2023-06-10T18:44:46.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab006b120 con 0x7feab00b4ff0 2023-06-10T18:44:46.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa40072d0 con 0x7feab00b4ff0 2023-06-10T18:44:46.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.673+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab006b6e0 con 0x7feab00b4ff0 2023-06-10T18:44:46.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.674+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab0074050 con 0x7feab00b4ff0 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.675+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feaa4007430 con 0x7feab00b4ff0 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.675+0000 7fea97fff700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 0x7fea9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.675+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7feaa4088970 con 0x7feab00b4ff0 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.675+0000 7feaaeffd700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 0x7fea9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.676+0000 7feaaeffd700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 0x7fea9c06d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009c40 tx=0x7feaa0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:46.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.679+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feaa40537e0 con 0x7feab00b4ff0 2023-06-10T18:44:46.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.909+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feab0071640 con 0x7feab00b4ff0 2023-06-10T18:44:46.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.909+0000 7fea97fff700 1 -- 172.21.15.104:0/3181967495 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7feaa4017070 con 0x7feab00b4ff0 2023-06-10T18:44:46.915 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-10T18:44:46.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 msgr2=0x7fea9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 0x7fea9c06d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009c40 tx=0x7feaa0008030 comp rx=0 tx=0).stop 2023-06-10T18:44:46.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 msgr2=0x7feab014bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:46.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c530 tx=0x7feaa400c840 comp rx=0 tx=0).stop 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 shutdown_connections 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea9c06ace0 0x7fea9c06d190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab015a420 0x7feab00c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab00b4ff0 0x7feab014bb40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 --2- 172.21.15.104:0/3181967495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feab00ce890 0x7feab00c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 >> 172.21.15.104:0/3181967495 conn(0x7feab0089fd0 msgr2=0x7feab0072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:46.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 shutdown_connections 2023-06-10T18:44:46.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:46.911+0000 7feab5caa700 1 -- 172.21.15.104:0/3181967495 wait complete. 2023-06-10T18:44:46.928 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:44:46.929 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:44:46.929 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:44:46.929 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-06-10T18:44:46.930 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:46.930 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:44:46.930 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:44:46.930 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:44:46.930 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-06-10T18:44:46.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:46.931 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:44:46.948 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.949 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:46.950 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:46.950 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:46.950 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:46.950 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:46.950 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42397sec preferred_lft 42397sec 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.951 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:46.952 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:46.952 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:44:46.952 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.952 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:44:46.952 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.953 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:44:46.953 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:44:46.953 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link tentative 2023-06-10T18:44:46.953 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:46.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:46.954 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:44:47.013 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:44:47.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.014 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:44:47.072 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:44:47.072 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:44:47.072 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:44:47.072 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:44:47.072 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:44:47.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.073 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:47.074 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:47.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:44:47.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:44:47.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:44:47.075 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:47.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.177 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:44:47.194 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:44:47.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.196 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2023-06-10T18:44:47.250 INFO:teuthology.orchestra.run.smithi104.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-06-10T18:44:47.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.252 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:47.252 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:47.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:44:47.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-06-10T18:44:47.253 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:47.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-06-10T18:44:47.380 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:47.380 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:47.380 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 96:1f:bf:1c:a2:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:44:47.381 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.382 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::941f:bfff:fe1c:a2d0/64 scope link 2023-06-10T18:44:47.382 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-06-10T18:44:47.481 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:47.481 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:47.481 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 96:1f:bf:1c:a2:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.482 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::941f:bfff:fe1c:a2d0/64 scope link 2023-06-10T18:44:47.483 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:47.495 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-06-10T18:44:47.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.496 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:47.496 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:47.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-06-10T18:44:47.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-06-10T18:44:47.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-06-10T18:44:47.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-06-10T18:44:47.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-06-10T18:44:47.498 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:47.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:47.752 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:44:47.753 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:44:47.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.1 up 2023-06-10T18:44:47.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.1 master ceph-brx 2023-06-10T18:44:47.754 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:44:47.866 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-06-10T18:44:47.866 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:47.867 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:47.896 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-06-10T18:44:47.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:47.897 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:47.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:48.019 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:48.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:48.085 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:48.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:48.087 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:48.143 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:44:48.143 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2023-06-10T18:44:48.144 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-06-10T18:44:48.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:44:48.212 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:48.213 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:48.271 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi104.stderr:2023-06-10T18:44:48.266+0000 7f818491c5c0 -1 init, newargv = 0x562039dbfc30 newargc=15 2023-06-10T18:44:48.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi104.stderr:ceph-fuse[112605]: starting ceph client 2023-06-10T18:44:48.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi104.stderr:ceph-fuse[112605]: starting fuse 2023-06-10T18:44:48.279 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:48.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:48.281 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:44:48.338 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:44:48.338 INFO:teuthology.orchestra.run.smithi104.stdout:48 2023-06-10T18:44:48.338 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2023-06-10T18:44:48.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:44:48.339 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:44:48.339 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:44:48.339 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:44:48.340 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:44:48.340 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:48.340 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:44:48.340 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-06-10T18:44:48.341 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:44:48.341 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-06-10T18:44:48.341 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:48.341 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:44:48.342 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:44:48.343 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:44:48.343 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:44:48.344 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:48.344 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:44:48.344 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:44:48.344 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:44:48.345 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:44:48.345 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:44:48.345 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:44:48.345 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:44:48.346 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:44:48.346 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:44:48.346 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.2")) 2023-06-10T18:44:48.346 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:44:48.460 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.2.112605.asok 2023-06-10T18:44:48.643 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.112605.asok 2023-06-10T18:44:48.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:48.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.112605.asok status 2023-06-10T18:44:48.798 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:44:48.798 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:44:48.798 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:44:48.798 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "2", 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "112605", 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:48.799 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 4926, 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 4926 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:48.800 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.2:0", 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2731727970 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:44:48.801 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.2:0", 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2731727970 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.4926 192.168.0.2:0/2731727970", 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.2:0/2731727970", 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 9, 2023-06-10T18:44:48.802 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 27, 2023-06-10T18:44:48.803 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:44:48.803 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:44:48.803 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:44:48.803 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:44:48.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:44:48.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 Processor -- start 2023-06-10T18:44:48.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 -- start start 2023-06-10T18:44:48.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3500d2b20 0x7fa3500d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3500cd430 con 0x7fa3500ccfa0 2023-06-10T18:44:48.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3500d0ee0 con 0x7fa3500d2b20 2023-06-10T18:44:48.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.922+0000 7fa35605b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500cd710 con 0x7fa3500d0520 2023-06-10T18:44:48.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3500d2b20 0x7fa3500d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa3500ccfa0 0x7fa3500b2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53316/0 (socket says 172.21.15.104:53316) 2023-06-10T18:44:48.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34effd700 1 -- 172.21.15.104:0/876319195 learned_addr learned my addr 172.21.15.104:0/876319195 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:48.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34ffff700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 -- 172.21.15.104:0/876319195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 msgr2=0x7fa3500d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:48.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:48.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 -- 172.21.15.104:0/876319195 >> v1:172.21.15.104:6789/0 conn(0x7fa3500ccfa0 legacy=0x7fa3500b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 -- 172.21.15.104:0/876319195 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3500cd9f0 con 0x7fa3500d2b20 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34e7fc700 1 -- 172.21.15.104:0/876319195 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101281475 0 0) 0x7fa3500cd430 con 0x7fa3500ccfa0 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34f7fe700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3500d2b20 0x7fa3500d2f00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa340009840 tx=0x7fa340009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118534203a2e0685 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34e7fc700 1 -- 172.21.15.104:0/876319195 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340011830 con 0x7fa3500d2b20 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34e7fc700 1 -- 172.21.15.104:0/876319195 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340011990 con 0x7fa3500d2b20 2023-06-10T18:44:48.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.923+0000 7fa34e7fc700 1 -- 172.21.15.104:0/876319195 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340011d20 con 0x7fa3500d2b20 2023-06-10T18:44:48.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 -- 172.21.15.104:0/876319195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3500d2b20 msgr2=0x7fa3500d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:48.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3500d2b20 0x7fa3500d2f00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa340009840 tx=0x7fa340009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:48.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 -- 172.21.15.104:0/876319195 shutdown_connections 2023-06-10T18:44:48.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3500d2b20 0x7fa3500d2f00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:48.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 --2- 172.21.15.104:0/876319195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:48.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 -- 172.21.15.104:0/876319195 >> 172.21.15.104:0/876319195 conn(0x7fa350071ec0 msgr2=0x7fa35009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:48.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 -- 172.21.15.104:0/876319195 shutdown_connections 2023-06-10T18:44:48.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.924+0000 7fa35605b700 1 -- 172.21.15.104:0/876319195 wait complete. 2023-06-10T18:44:48.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.925+0000 7fa35605b700 1 Processor -- start 2023-06-10T18:44:48.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.925+0000 7fa35605b700 1 -- start start 2023-06-10T18:44:48.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3500ccfa0 0x7fa350158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa350158250 con 0x7fa350157dd0 2023-06-10T18:44:48.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500c80f0 con 0x7fa3500ccfa0 2023-06-10T18:44:48.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500c83d0 con 0x7fa3500d0520 2023-06-10T18:44:48.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55498/0 (socket says 172.21.15.104:55498) 2023-06-10T18:44:48.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 -- 172.21.15.104:0/1894156506 learned_addr learned my addr 172.21.15.104:0/1894156506 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:48.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 -- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 msgr2=0x7fa3500c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:48.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34f7fe700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3500ccfa0 0x7fa350158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34effd700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:48.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 -- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3500ccfa0 msgr2=0x7fa350158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:48.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3500ccfa0 0x7fa350158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:48.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 -- 172.21.15.104:0/1894156506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400094a0 con 0x7fa350157dd0 2023-06-10T18:44:48.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa34ffff700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa34800efc0 tx=0x7fa34800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:48.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34800cd40 con 0x7fa350157dd0 2023-06-10T18:44:48.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34800f040 con 0x7fa350157dd0 2023-06-10T18:44:48.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3480146c0 con 0x7fa350157dd0 2023-06-10T18:44:48.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.926+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3500c86b0 con 0x7fa350157dd0 2023-06-10T18:44:48.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.927+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35015d560 con 0x7fa350157dd0 2023-06-10T18:44:48.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.927+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3500c7750 con 0x7fa350157dd0 2023-06-10T18:44:48.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.928+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa348014820 con 0x7fa350157dd0 2023-06-10T18:44:48.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.928+0000 7fa347fff700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 0x7fa33006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:48.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.928+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa34808a3b0 con 0x7fa350157dd0 2023-06-10T18:44:48.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.928+0000 7fa34f7fe700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 0x7fa33006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:48.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.929+0000 7fa34f7fe700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 0x7fa33006d190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3500b0b30 tx=0x7fa340008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:48.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:48.933+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3500c7750 con 0x7fa350157dd0 2023-06-10T18:44:49.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.167+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa350074050 con 0x7fa350157dd0 2023-06-10T18:44:49.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.167+0000 7fa347fff700 1 -- 172.21.15.104:0/1894156506 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fa348058c90 con 0x7fa350157dd0 2023-06-10T18:44:49.173 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-10T18:44:49.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 msgr2=0x7fa33006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 0x7fa33006d190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3500b0b30 tx=0x7fa340008030 comp rx=0 tx=0).stop 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 msgr2=0x7fa35015d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa34800efc0 tx=0x7fa34800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 shutdown_connections 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa33006ace0 0x7fa33006d190 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:49.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3500ccfa0 0x7fa350158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:49.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa350157dd0 0x7fa35015d020 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:49.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 --2- 172.21.15.104:0/1894156506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3500d0520 0x7fa3500c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:49.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 >> 172.21.15.104:0/1894156506 conn(0x7fa350071ec0 msgr2=0x7fa35006c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:49.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 shutdown_connections 2023-06-10T18:44:49.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:49.169+0000 7fa35605b700 1 -- 172.21.15.104:0/1894156506 wait complete. 2023-06-10T18:44:49.186 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:44:49.187 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:44:49.187 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:44:49.187 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:44:49.188 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:49.188 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:44:49.188 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:44:49.189 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:44:49.189 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:44:49.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.190 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:44:49.211 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:49.212 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:49.212 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:49.212 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:49.212 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:49.213 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:49.213 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:49.213 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:49.213 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:49.214 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:49.214 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:49.214 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:49.214 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:44:49.214 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:49.215 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:44:49.215 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42391sec preferred_lft 42391sec 2023-06-10T18:44:49.215 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:44:49.215 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:49.216 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-10T18:44:49.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.217 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:49.217 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:49.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2023-06-10T18:44:49.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2023-06-10T18:44:49.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2023-06-10T18:44:49.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-10T18:44:49.219 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:49.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.383 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:44:49.411 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:44:49.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.413 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:44:49.492 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:44:49.492 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:44:49.493 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:44:49.493 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:44:49.493 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:44:49.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.494 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:49.494 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:49.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:44:49.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:44:49.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:44:49.495 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:49.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.758 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:44:49.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.777 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2023-06-10T18:44:49.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.833 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:49.833 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:49.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:44:49.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-06-10T18:44:49.834 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:49.926 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-06-10T18:44:49.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:49.927 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:49.927 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:49.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-06-10T18:44:49.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-10T18:44:49.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-06-10T18:44:49.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-06-10T18:44:49.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-06-10T18:44:49.929 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:50.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:50.121 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:50.121 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:50.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:44:50.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2023-06-10T18:44:50.122 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:50.222 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:44:50.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:50.223 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:50.244 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:44:50.244 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:50.244 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:50.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:44:50.367 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:50.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:50.433 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:50.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:50.435 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:44:50.492 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:44:50.492 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:44:50.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:44:50.559 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:50.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:50.625 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:50.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:50.627 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:44:50.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:44:50.624+0000 7faf2d9335c0 -1 init, newargv = 0x561d42f9e020 newargc=15 2023-06-10T18:44:50.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[106897]: starting ceph client 2023-06-10T18:44:50.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[106897]: starting fuse 2023-06-10T18:44:50.646 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:44:50.646 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:44:50.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:44:50.647 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:44:50.647 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:44:50.648 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:44:50.648 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:44:50.648 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:50.648 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:44:50.649 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:44:50.649 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:44:50.649 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:44:50.649 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:50.650 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:44:50.650 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:44:50.650 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:44:50.650 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:44:50.650 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:50.651 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:44:50.651 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:44:50.651 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:44:50.651 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:44:50.652 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:44:50.652 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:44:50.652 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:44:50.652 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:44:50.653 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:50.653 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:44:50.653 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:44:50.773 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.106897.asok 2023-06-10T18:44:50.954 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.106897.asok 2023-06-10T18:44:50.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:50.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.106897.asok status 2023-06-10T18:44:51.109 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:44:51.110 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:44:51.110 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:44:51.110 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:44:51.110 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:44:51.110 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "106897", 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:44:51.111 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 4619, 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4619 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:51.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4226481204 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:44:51.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4226481204 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.4619 192.168.0.1:0/4226481204", 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/4226481204", 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 9, 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 28, 2023-06-10T18:44:51.114 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:44:51.115 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:44:51.115 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:44:51.115 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:44:51.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:44:51.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.246+0000 7ff54ea22700 1 Processor -- start 2023-06-10T18:44:51.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.246+0000 7ff54ea22700 1 -- start start 2023-06-10T18:44:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54ea22700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5480ce510 0x7ff5480ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54ea22700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff5480c8c40 0x7ff5480c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54ea22700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54815c050 con 0x7ff54815bc80 2023-06-10T18:44:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54ea22700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5480ceed0 con 0x7ff5480c8c40 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54ea22700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5480c9970 con 0x7ff5480ce510 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54cfbf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff5480c8c40 0x7ff5480c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54cfbf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff5480c8c40 0x7ff5480c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55512/0 (socket says 172.21.15.104:55512) 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff54cfbf700 1 -- 172.21.15.104:0/4063076936 learned_addr learned my addr 172.21.15.104:0/4063076936 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.247+0000 7ff5477fe700 1 --2- 172.21.15.104:0/4063076936 >> v2:172.21.15.114:3300/0 conn(0x7ff5480ce510 0x7ff5480ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54cfbf700 1 -- 172.21.15.104:0/4063076936 >> v2:172.21.15.114:3300/0 conn(0x7ff5480ce510 msgr2=0x7ff5480ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54cfbf700 1 --2- 172.21.15.104:0/4063076936 >> v2:172.21.15.114:3300/0 conn(0x7ff5480ce510 0x7ff5480ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54cfbf700 1 -- 172.21.15.104:0/4063076936 >> v1:172.21.15.114:6789/0 conn(0x7ff54815bc80 legacy=0x7ff548070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54cfbf700 1 -- 172.21.15.104:0/4063076936 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480ccb70 con 0x7ff5480c8c40 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff546ffd700 1 -- 172.21.15.104:0/4063076936 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870628238 0 0) 0x7ff54815c050 con 0x7ff54815bc80 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54cfbf700 1 --2- 172.21.15.104:0/4063076936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480c8c40 0x7ff5480c90a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff5480cc100 tx=0x7ff54000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52163c681b5808dc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff546ffd700 1 -- 172.21.15.104:0/4063076936 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540010070 con 0x7ff5480c8c40 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff546ffd700 1 -- 172.21.15.104:0/4063076936 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000dad0 con 0x7ff5480c8c40 2023-06-10T18:44:51.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff546ffd700 1 -- 172.21.15.104:0/4063076936 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000de80 con 0x7ff5480c8c40 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 -- 172.21.15.104:0/4063076936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480c8c40 msgr2=0x7ff5480c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 --2- 172.21.15.104:0/4063076936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480c8c40 0x7ff5480c90a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff5480cc100 tx=0x7ff54000d430 comp rx=0 tx=0).stop 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 -- 172.21.15.104:0/4063076936 shutdown_connections 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 --2- 172.21.15.104:0/4063076936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480c8c40 0x7ff5480c90a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 --2- 172.21.15.104:0/4063076936 >> v2:172.21.15.114:3300/0 conn(0x7ff5480ce510 0x7ff5480ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 -- 172.21.15.104:0/4063076936 >> 172.21.15.104:0/4063076936 conn(0x7ff54806ab00 msgr2=0x7ff5480b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:51.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.248+0000 7ff54ea22700 1 -- 172.21.15.104:0/4063076936 shutdown_connections 2023-06-10T18:44:51.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.249+0000 7ff54ea22700 1 -- 172.21.15.104:0/4063076936 wait complete. 2023-06-10T18:44:51.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.250+0000 7ff54ea22700 1 Processor -- start 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.250+0000 7ff54ea22700 1 -- start start 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5480c8c40 0x7ff5480c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff54815bc80 0x7ff5481f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480ac300 con 0x7ff5480ce510 2023-06-10T18:44:51.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5481f6670 con 0x7ff5480c8c40 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5481f6950 con 0x7ff54815bc80 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff547fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5480c8c40 0x7ff5480c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55516/0 (socket says 172.21.15.104:55516) 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 -- 172.21.15.104:0/413635931 learned_addr learned my addr 172.21.15.104:0/413635931 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54cfbf700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff54815bc80 0x7ff5481f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 -- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff54815bc80 msgr2=0x7ff5481f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff54815bc80 0x7ff5481f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 -- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5480c8c40 msgr2=0x7ff5480c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5480c8c40 0x7ff5480c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 -- 172.21.15.104:0/413635931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54000bd40 con 0x7ff5480ce510 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff5477fe700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff53000e9b0 tx=0x7ff53000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff530009710 con 0x7ff5480ce510 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53000ee50 con 0x7ff5480ce510 2023-06-10T18:44:51.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5480b2400 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53000ca50 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.251+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5480b29c0 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.252+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548074050 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.253+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff53000cbb0 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.253+0000 7ff544ff9700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 0x7ff52806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.253+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff5300897a0 con 0x7ff5480ce510 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.253+0000 7ff547fff700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 0x7ff52806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:51.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.254+0000 7ff547fff700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 0x7ff52806d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff538007fd0 tx=0x7ff538009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:51.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.258+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff530057fb0 con 0x7ff5480ce510 2023-06-10T18:44:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.487+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff5480cc330 con 0x7ff5480ce510 2023-06-10T18:44:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.487+0000 7ff544ff9700 1 -- 172.21.15.104:0/413635931 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff53000ce60 con 0x7ff5480ce510 2023-06-10T18:44:51.493 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-10T18:44:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 msgr2=0x7ff52806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 0x7ff52806d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff538007fd0 tx=0x7ff538009040 comp rx=0 tx=0).stop 2023-06-10T18:44:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 msgr2=0x7ff5481f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff53000e9b0 tx=0x7ff53000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 shutdown_connections 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff52806ace0 0x7ff52806d190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5480c8c40 0x7ff5480c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff54815bc80 0x7ff5481f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 --2- 172.21.15.104:0/413635931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5480ce510 0x7ff5481f5b00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 >> 172.21.15.104:0/413635931 conn(0x7ff54806ab00 msgr2=0x7ff5480719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 shutdown_connections 2023-06-10T18:44:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:51.489+0000 7ff54ea22700 1 -- 172.21.15.104:0/413635931 wait complete. 2023-06-10T18:44:51.506 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:44:51.507 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:44:51.508 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:44:51.508 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-06-10T18:44:51.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.508 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:44:51.526 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:51.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:51.526 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:51.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:44:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42388sec preferred_lft 42388sec 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.528 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:44:51.529 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link tentative 2023-06-10T18:44:51.530 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.530 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:44:51.591 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:44:51.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.593 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:44:51.650 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:44:51.651 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:44:51.651 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:44:51.651 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:44:51.651 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:44:51.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.652 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:51.652 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:51.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:44:51.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:44:51.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:44:51.653 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:51.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.756 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:44:51.772 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:44:51.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.774 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2023-06-10T18:44:51.828 INFO:teuthology.orchestra.run.smithi114.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-06-10T18:44:51.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.830 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:51.830 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:51.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:44:51.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-06-10T18:44:51.831 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:51.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-06-10T18:44:51.954 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:51.954 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:51.954 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 3e:51:01:4a:f2:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:44:51.955 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.956 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::3c51:1ff:fe4a:f2d8/64 scope link 2023-06-10T18:44:51.956 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:51.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:51.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-06-10T18:44:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:44:52.053 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:44:52.053 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:44:52.053 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:44:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:44:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 3e:51:01:4a:f2:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:44:52.055 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:44:52.055 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:52.055 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::3c51:1ff:fe4a:f2d8/64 scope link 2023-06-10T18:44:52.056 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:44:52.064 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-06-10T18:44:52.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:52.065 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:52.066 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:52.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-06-10T18:44:52.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-06-10T18:44:52.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-06-10T18:44:52.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-06-10T18:44:52.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-06-10T18:44:52.068 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:52.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:52.318 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:44:52.318 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:44:52.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.1 up 2023-06-10T18:44:52.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.1 master ceph-brx 2023-06-10T18:44:52.319 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:44:52.443 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-06-10T18:44:52.444 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:52.444 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:52.472 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-06-10T18:44:52.472 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:44:52.473 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:52.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:44:52.595 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:52.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:52.661 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:52.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:52.663 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:44:52.719 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:44:52.719 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2023-06-10T18:44:52.720 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-06-10T18:44:52.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:44:52.787 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:44:52.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:44:52.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi114.stderr:2023-06-10T18:44:52.842+0000 7f934d1f55c0 -1 init, newargv = 0x5647f0d95de0 newargc=15 2023-06-10T18:44:52.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi114.stderr:ceph-fuse[107494]: starting ceph client 2023-06-10T18:44:52.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi114.stderr:ceph-fuse[107494]: starting fuse 2023-06-10T18:44:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:44:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:44:52.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:52.858 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:44:52.914 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:44:52.914 INFO:teuthology.orchestra.run.smithi114.stdout:48 2023-06-10T18:44:52.914 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2023-06-10T18:44:52.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:44:52.915 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:44:52.915 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:44:52.916 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:44:52.916 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:44:52.916 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:52.916 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:44:52.917 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-06-10T18:44:52.917 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:44:52.917 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-06-10T18:44:52.917 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:52.918 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:44:52.918 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:44:52.918 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:44:52.918 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:44:52.918 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:52.919 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:44:52.919 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:44:52.919 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:44:52.919 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:44:52.920 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:44:52.920 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:44:52.920 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:44:52.921 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:44:52.921 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:44:52.921 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.3")) 2023-06-10T18:44:52.921 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:44:53.039 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.3.107494.asok 2023-06-10T18:44:53.221 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.107494.asok 2023-06-10T18:44:53.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.107494.asok status 2023-06-10T18:44:53.377 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:44:53.377 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:44:53.377 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:44:53.377 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "3", 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "107494", 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:44:53.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 4747, 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4747 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:53.379 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.2:0", 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2895061603 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:44:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.2:0", 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2895061603 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.4747 192.168.0.2:0/2895061603", 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.2:0/2895061603", 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 9, 2023-06-10T18:44:53.381 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 29, 2023-06-10T18:44:53.382 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:44:53.382 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:44:53.382 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:44:53.382 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:44:53.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.387 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:44:53.387 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:53.408 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:44:53.409 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:53.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.409 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:44:53.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.480 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:44:53.480 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:53.538 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:44:53.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:53.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-06-10T18:44:53.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.609 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:44:53.609 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:53.626 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:44:53.627 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:53.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:44:53.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.694 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:44:53.694 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:53.750 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:44:53.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:53.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:44:53.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-06-10T18:44:53.816 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-06-10T18:44:53.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.962+0000 7f0bc195c700 1 Processor -- start 2023-06-10T18:44:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.962+0000 7f0bc195c700 1 -- start start 2023-06-10T18:44:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bc195c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0bbc1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bc195c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0ce2f0 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bc195c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc160c20 con 0x7f0bbc06ce20 2023-06-10T18:44:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bc195c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0cfd30 con 0x7f0bbc15c040 2023-06-10T18:44:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bba7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0bbc06c9b0 0x7f0bbc0ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53352/0 (socket says 172.21.15.104:53352) 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bba7fc700 1 -- 172.21.15.104:0/2158194387 learned_addr learned my addr 172.21.15.104:0/2158194387 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165562903 0 0) 0x7f0bbc0ce2f0 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0bbc1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba40035b0 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 898068245 0 0) 0x7f0ba40035b0 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 msgr2=0x7f0bbc1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0bbc1606e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 >> v1:172.21.15.114:6789/0 conn(0x7f0bbc06ce20 legacy=0x7f0bbc0adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0cfa50 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0bb0001670 con 0x7f0bbc06c9b0 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 >> v1:172.21.15.104:6789/0 conn(0x7f0bbc06c9b0 legacy=0x7f0bbc0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0ba4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0ba4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba4004d90 0x7f0ba4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0cfd30 con 0x7f0ba4003a30 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba400ba10 con 0x7f0ba4004d90 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba400bba0 con 0x7f0bbc15c040 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.963+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0ba4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bba7fc700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0ba4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbb7fe700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba4004d90 0x7f0ba4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 -- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 msgr2=0x7f0ba4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc15c040 0x7f0ba4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 -- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba4004d90 msgr2=0x7f0ba4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ba4004d90 0x7f0ba4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 -- 172.21.15.104:0/2158194387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40035b0 con 0x7f0ba4003a30 2023-06-10T18:44:53.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0ba4003e00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009b60 tx=0x7f0ba4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac891d9c09dc8ad server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00e6c0 con 0x7f0ba4003a30 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00e820 con 0x7f0ba4003a30 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.964+0000 7f0bb9ffb700 1 -- 172.21.15.104:0/2158194387 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00eb90 con 0x7f0ba4003a30 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 -- 172.21.15.104:0/2158194387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 msgr2=0x7f0ba4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 --2- 172.21.15.104:0/2158194387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0ba4003e00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009b60 tx=0x7f0ba4005880 comp rx=0 tx=0).stop 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bbb7fe700 1 -- 172.21.15.104:0/2158194387 reap_dead start 2023-06-10T18:44:53.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 -- 172.21.15.104:0/2158194387 shutdown_connections 2023-06-10T18:44:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 -- 172.21.15.104:0/2158194387 >> 172.21.15.104:0/2158194387 conn(0x7f0bbc0b1570 msgr2=0x7f0bbc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 -- 172.21.15.104:0/2158194387 shutdown_connections 2023-06-10T18:44:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.965+0000 7f0bc195c700 1 -- 172.21.15.104:0/2158194387 wait complete. 2023-06-10T18:44:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.966+0000 7f0bc195c700 1 Processor -- start 2023-06-10T18:44:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.966+0000 7f0bc195c700 1 -- start start 2023-06-10T18:44:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc0c4df0 0x7f0bbc0c5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc0c2930 0x7f0bbc0c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c57a0 con 0x7f0ba4003a30 2023-06-10T18:44:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1548a0 con 0x7f0bbc0c2930 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c3660 con 0x7f0bbc0c4df0 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55534/0 (socket says 172.21.15.104:55534) 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 -- 172.21.15.104:0/3468218951 learned_addr learned my addr 172.21.15.104:0/3468218951 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbb7fe700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc0c2930 0x7f0bbc0c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bba7fc700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc0c4df0 0x7f0bbc0c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 -- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc0c4df0 msgr2=0x7f0bbc0c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc0c4df0 0x7f0bbc0c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 -- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc0c2930 msgr2=0x7f0bbc0c2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc0c2930 0x7f0bbc0c2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 -- 172.21.15.104:0/3468218951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0c3940 con 0x7f0ba4003a30 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bbaffd700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4005800 tx=0x7f0bac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac021070 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac004470 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.967+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0c3c20 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.968+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00e6c0 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.968+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc0c41e0 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.968+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc1557b0 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.969+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0bac025070 con 0x7f0ba4003a30 2023-06-10T18:44:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.969+0000 7f0ba3fff700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 0x7f0ba806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.969+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0bac08a850 con 0x7f0ba4003a30 2023-06-10T18:44:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.969+0000 7f0bba7fc700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 0x7f0ba806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.970+0000 7f0bba7fc700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 0x7f0ba806d1c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0aff00 tx=0x7f0bb0000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:53.974+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0bbc1557b0 con 0x7f0ba4003a30 2023-06-10T18:44:54.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.161+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bbc1557b0 con 0x7f0ba4003a30 2023-06-10T18:44:54.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.162+0000 7f0ba3fff700 1 -- 172.21.15.104:0/3468218951 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f0bbc1557b0 con 0x7f0ba4003a30 2023-06-10T18:44:54.169 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:54.170 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:54.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 msgr2=0x7f0ba806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 0x7f0ba806d1c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0aff00 tx=0x7f0bb0000f80 comp rx=0 tx=0).stop 2023-06-10T18:44:54.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 msgr2=0x7f0bbc154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4005800 tx=0x7f0bac0047c0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 shutdown_connections 2023-06-10T18:44:54.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ba806ad10 0x7f0ba806d1c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bbc0c2930 0x7f0bbc0c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ba4003a30 0x7f0bbc154360 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 --2- 172.21.15.104:0/3468218951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bbc0c4df0 0x7f0bbc0c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 >> 172.21.15.104:0/3468218951 conn(0x7f0bbc0b1570 msgr2=0x7f0bbc069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 shutdown_connections 2023-06-10T18:44:54.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.165+0000 7f0bc195c700 1 -- 172.21.15.104:0/3468218951 wait complete. 2023-06-10T18:44:54.183 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:54.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:54.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.301+0000 7f8afd264700 1 Processor -- start 2023-06-10T18:44:54.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.301+0000 7f8afd264700 1 -- start start 2023-06-10T18:44:54.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8afd264700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 0x7f8af80d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8afd264700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 0x7f8af80b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8afd264700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af80af530 con 0x7f8af80d25c0 2023-06-10T18:44:54.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8afd264700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af80af810 con 0x7f8af80d04f0 2023-06-10T18:44:54.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8afd264700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af80afaf0 con 0x7f8af80b05d0 2023-06-10T18:44:54.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af67fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8af80d25c0 0x7f8af80d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53372/0 (socket says 172.21.15.104:53372) 2023-06-10T18:44:54.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af67fc700 1 -- 172.21.15.104:0/1412695520 learned_addr learned my addr 172.21.15.104:0/1412695520 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:54.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af77fe700 1 --2- 172.21.15.104:0/1412695520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 0x7f8af80b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af6ffd700 1 --2- 172.21.15.104:0/1412695520 >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 0x7f8af80d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af5ffb700 1 -- 172.21.15.104:0/1412695520 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874696164 0 0) 0x7f8af80af530 con 0x7f8af80d25c0 2023-06-10T18:44:54.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.302+0000 7f8af5ffb700 1 -- 172.21.15.104:0/1412695520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae00035b0 con 0x7f8af80d25c0 2023-06-10T18:44:54.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af77fe700 1 -- 172.21.15.104:0/1412695520 >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 msgr2=0x7f8af80d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af77fe700 1 --2- 172.21.15.104:0/1412695520 >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 0x7f8af80d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af77fe700 1 -- 172.21.15.104:0/1412695520 >> v1:172.21.15.104:6789/0 conn(0x7f8af80d25c0 legacy=0x7f8af80d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af77fe700 1 -- 172.21.15.104:0/1412695520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af80afdd0 con 0x7f8af80b05d0 2023-06-10T18:44:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af6ffd700 1 --2- 172.21.15.104:0/1412695520 >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 0x7f8af80d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af77fe700 1 --2- 172.21.15.104:0/1412695520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 0x7f8af80b0a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009d90 tx=0x7f8af000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7bb0b6392e8a309 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af5ffb700 1 -- 172.21.15.104:0/1412695520 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af000e070 con 0x7f8af80b05d0 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af5ffb700 1 -- 172.21.15.104:0/1412695520 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af000bb20 con 0x7f8af80b05d0 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8af5ffb700 1 -- 172.21.15.104:0/1412695520 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af0012430 con 0x7f8af80b05d0 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8afd264700 1 -- 172.21.15.104:0/1412695520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 msgr2=0x7f8af80b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.303+0000 7f8afd264700 1 --2- 172.21.15.104:0/1412695520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 0x7f8af80b0a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009d90 tx=0x7f8af000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 -- 172.21.15.104:0/1412695520 shutdown_connections 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 --2- 172.21.15.104:0/1412695520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80b05d0 0x7f8af80b0a50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 --2- 172.21.15.104:0/1412695520 >> v2:172.21.15.114:3300/0 conn(0x7f8af80d04f0 0x7f8af80d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 -- 172.21.15.104:0/1412695520 >> 172.21.15.104:0/1412695520 conn(0x7f8af8071ec0 msgr2=0x7f8af809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 -- 172.21.15.104:0/1412695520 shutdown_connections 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.304+0000 7f8afd264700 1 -- 172.21.15.104:0/1412695520 wait complete. 2023-06-10T18:44:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.305+0000 7f8afd264700 1 Processor -- start 2023-06-10T18:44:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.305+0000 7f8afd264700 1 -- start start 2023-06-10T18:44:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8af80b05d0 0x7f8af81514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80d25c0 0x7f8af8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af81f5210 con 0x7f8af80d04f0 2023-06-10T18:44:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8151a00 con 0x7f8af80b05d0 2023-06-10T18:44:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af81f54b0 con 0x7f8af80d25c0 2023-06-10T18:44:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55550/0 (socket says 172.21.15.104:55550) 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 -- 172.21.15.104:0/4288582365 learned_addr learned my addr 172.21.15.104:0/4288582365 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af77fe700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80d25c0 0x7f8af8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af6ffd700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8af80b05d0 0x7f8af81514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 -- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80d25c0 msgr2=0x7f8af8157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80d25c0 0x7f8af8157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 -- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8af80b05d0 msgr2=0x7f8af81514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8af80b05d0 0x7f8af81514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 -- 172.21.15.104:0/4288582365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec009430 con 0x7f8af80d04f0 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8af67fc700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8001eb0 tx=0x7f8ae800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8005d70 con 0x7f8af80d04f0 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae80070d0 con 0x7f8af80d04f0 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0009a70 con 0x7f8af80d04f0 2023-06-10T18:44:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8007460 con 0x7f8af80d04f0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.306+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af81f5a70 con 0x7f8af80d04f0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.307+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af8067010 con 0x7f8af80d04f0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.308+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8ae80075c0 con 0x7f8af80d04f0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.309+0000 7f8adf7fe700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 0x7f8ae406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.309+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8ae8088ed0 con 0x7f8af80d04f0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.309+0000 7f8af6ffd700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 0x7f8ae406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.309+0000 7f8af6ffd700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 0x7f8ae406d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009d50 tx=0x7f8aec0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.312+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ae8053dc0 con 0x7f8af80d04f0 2023-06-10T18:44:54.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.497+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8af8074050 con 0x7f8af80d04f0 2023-06-10T18:44:54.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.498+0000 7f8adf7fe700 1 -- 172.21.15.104:0/4288582365 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ae80573e0 con 0x7f8af80d04f0 2023-06-10T18:44:54.504 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:54.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.499+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 msgr2=0x7f8ae406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.499+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 0x7f8ae406d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009d50 tx=0x7f8aec0091c0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.499+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 msgr2=0x7f8af81e4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.499+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8001eb0 tx=0x7f8ae800c7b0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 shutdown_connections 2023-06-10T18:44:54.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ae406ace0 0x7f8ae406d190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8af80b05d0 0x7f8af81514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8af80d25c0 0x7f8af8157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 --2- 172.21.15.104:0/4288582365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8af80d04f0 0x7f8af81e4800 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 >> 172.21.15.104:0/4288582365 conn(0x7f8af8071ec0 msgr2=0x7f8af80cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 shutdown_connections 2023-06-10T18:44:54.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.500+0000 7f8afd264700 1 -- 172.21.15.104:0/4288582365 wait complete. 2023-06-10T18:44:54.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:54.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.635+0000 7f030c4c5700 1 Processor -- start 2023-06-10T18:44:54.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.635+0000 7f030c4c5700 1 -- start start 2023-06-10T18:44:54.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030c4c5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f030415c6f0 0x7f03040b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030c4c5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040ae7e0 con 0x7f03040aec00 2023-06-10T18:44:54.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030c4c5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030406df90 con 0x7f030406ea70 2023-06-10T18:44:54.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030c4c5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030406e270 con 0x7f030415c6f0 2023-06-10T18:44:54.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030aa62700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f03040aec00 0x7f03040ac8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53388/0 (socket says 172.21.15.104:53388) 2023-06-10T18:44:54.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030aa62700 1 -- 172.21.15.104:0/35190505 learned_addr learned my addr 172.21.15.104:0/35190505 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030a261700 1 --2- 172.21.15.104:0/35190505 >> v2:172.21.15.114:3300/0 conn(0x7f030415c6f0 0x7f03040b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507276798 0 0) 0x7f03040ae7e0 con 0x7f03040aec00 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f00035b0 con 0x7f03040aec00 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030a261700 1 -- 172.21.15.104:0/35190505 >> v1:172.21.15.114:6789/0 conn(0x7f030406ea70 legacy=0x7f03040ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327592968 0 0) 0x7f030406df90 con 0x7f030406ea70 2023-06-10T18:44:54.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030a261700 1 -- 172.21.15.104:0/35190505 >> v1:172.21.15.104:6789/0 conn(0x7f03040aec00 legacy=0x7f03040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030a261700 1 -- 172.21.15.104:0/35190505 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030406e550 con 0x7f030415c6f0 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.636+0000 7f030a261700 1 --2- 172.21.15.104:0/35190505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030415c6f0 0x7f03040b1070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0300009820 tx=0x7f0300009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706adb4aad1c3e97 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0019f0 con 0x7f030415c6f0 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300005c30 con 0x7f030415c6f0 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030925f700 1 -- 172.21.15.104:0/35190505 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300006030 con 0x7f030415c6f0 2023-06-10T18:44:54.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 -- 172.21.15.104:0/35190505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030415c6f0 msgr2=0x7f03040b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 --2- 172.21.15.104:0/35190505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030415c6f0 0x7f03040b1070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0300009820 tx=0x7f0300009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 -- 172.21.15.104:0/35190505 shutdown_connections 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 --2- 172.21.15.104:0/35190505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030415c6f0 0x7f03040b1070 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 -- 172.21.15.104:0/35190505 >> 172.21.15.104:0/35190505 conn(0x7f030406b8a0 msgr2=0x7f03040cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 -- 172.21.15.104:0/35190505 shutdown_connections 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.637+0000 7f030c4c5700 1 -- 172.21.15.104:0/35190505 wait complete. 2023-06-10T18:44:54.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.638+0000 7f030c4c5700 1 Processor -- start 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 -- start start 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030406ea70 0x7f03040c2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03040aec00 0x7f03040c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030415b110 con 0x7f030415c6f0 2023-06-10T18:44:54.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b3510 con 0x7f030406ea70 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030c4c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b37f0 con 0x7f03040aec00 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030aa62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030aa62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55566/0 (socket says 172.21.15.104:55566) 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030aa62700 1 -- 172.21.15.104:0/2589897592 learned_addr learned my addr 172.21.15.104:0/2589897592 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f030a261700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030406ea70 0x7f03040c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.639+0000 7f0309a60700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03040aec00 0x7f03040c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 -- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03040aec00 msgr2=0x7f03040c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03040aec00 0x7f03040c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 -- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030406ea70 msgr2=0x7f03040c2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030406ea70 0x7f03040c2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 -- 172.21.15.104:0/2589897592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300009430 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030aa62700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02fc001f60 tx=0x7f02fc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00acf0 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00ae50 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03040b3ad0 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc007310 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.640+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03040b4090 con 0x7f030415c6f0 2023-06-10T18:44:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.641+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304074050 con 0x7f030415c6f0 2023-06-10T18:44:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.641+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f02fc007470 con 0x7f030415c6f0 2023-06-10T18:44:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.642+0000 7f02faffd700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.642+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f02fc089360 con 0x7f030415c6f0 2023-06-10T18:44:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.642+0000 7f030a261700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.642+0000 7f030a261700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0300015040 tx=0x7f0300007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.646+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f02fc054060 con 0x7f030415c6f0 2023-06-10T18:44:54.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.833+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f03040ad950 con 0x7f030415c6f0 2023-06-10T18:44:54.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.833+0000 7f02faffd700 1 -- 172.21.15.104:0/2589897592 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f02fc057680 con 0x7f030415c6f0 2023-06-10T18:44:54.839 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:54.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.835+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 msgr2=0x7f02e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.835+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0300015040 tx=0x7f0300007000 comp rx=0 tx=0).stop 2023-06-10T18:44:54.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.835+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 msgr2=0x7f030415abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.835+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02fc001f60 tx=0x7f02fc00a340 comp rx=0 tx=0).stop 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 shutdown_connections 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f030406ea70 0x7f03040c2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f030415c6f0 0x7f030415abd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 --2- 172.21.15.104:0/2589897592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03040aec00 0x7f03040c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 >> 172.21.15.104:0/2589897592 conn(0x7f030406b8a0 msgr2=0x7f030406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 shutdown_connections 2023-06-10T18:44:54.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.836+0000 7f030c4c5700 1 -- 172.21.15.104:0/2589897592 wait complete. 2023-06-10T18:44:54.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:54.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.969+0000 7fbfe3fcc700 1 Processor -- start 2023-06-10T18:44:54.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.969+0000 7fbfe3fcc700 1 -- start start 2023-06-10T18:44:54.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfdc15c580 0x7fbfdc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 0x7fbfdc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc06cde0 0x7fbfdc069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc06e250 con 0x7fbfdc15c580 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc15c950 con 0x7fbfdc06e6e0 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe3fcc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b1e80 con 0x7fbfdc06cde0 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfdc15c580 0x7fbfdc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfdc15c580 0x7fbfdc0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55570/0 (socket says 172.21.15.104:55570) 2023-06-10T18:44:54.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 -- 172.21.15.104:0/220753793 learned_addr learned my addr 172.21.15.104:0/220753793 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1567700 1 --2- 172.21.15.104:0/220753793 >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 0x7fbfdc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe2569700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc06cde0 0x7fbfdc069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 -- 172.21.15.104:0/220753793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc06cde0 msgr2=0x7fbfdc069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc06cde0 0x7fbfdc069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 -- 172.21.15.104:0/220753793 >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 msgr2=0x7fbfdc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/220753793 >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 0x7fbfdc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 -- 172.21.15.104:0/220753793 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc06d920 con 0x7fbfdc15c580 2023-06-10T18:44:54.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1567700 1 --2- 172.21.15.104:0/220753793 >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 0x7fbfdc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:54.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc15c580 0x7fbfdc0b1070 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc06af30 tx=0x7fbfcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73781dfbd8125d88 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.970+0000 7fbfe0d66700 1 -- 172.21.15.104:0/220753793 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc005e30 con 0x7fbfdc15c580 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe0d66700 1 -- 172.21.15.104:0/220753793 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc004030 con 0x7fbfdc15c580 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe0d66700 1 -- 172.21.15.104:0/220753793 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc0043e0 con 0x7fbfdc15c580 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/220753793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc15c580 msgr2=0x7fbfdc0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc15c580 0x7fbfdc0b1070 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc06af30 tx=0x7fbfcc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/220753793 shutdown_connections 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc15c580 0x7fbfdc0b1070 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/220753793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc06cde0 0x7fbfdc069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/220753793 >> v2:172.21.15.114:3300/0 conn(0x7fbfdc06e6e0 0x7fbfdc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/220753793 >> 172.21.15.104:0/220753793 conn(0x7fbfdc06b8a0 msgr2=0x7fbfdc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:54.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/220753793 shutdown_connections 2023-06-10T18:44:54.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.971+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/220753793 wait complete. 2023-06-10T18:44:54.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.972+0000 7fbfe3fcc700 1 Processor -- start 2023-06-10T18:44:54.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.972+0000 7fbfe3fcc700 1 -- start start 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 0x7fbfdc0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc15c580 0x7fbfdc0c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b3bc0 con 0x7fbfdc06cde0 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b3ea0 con 0x7fbfdc06e6e0 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe3fcc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b4180 con 0x7fbfdc15c580 2023-06-10T18:44:54.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe1d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe1d68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55576/0 (socket says 172.21.15.104:55576) 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.973+0000 7fbfe1d68700 1 -- 172.21.15.104:0/2806903211 learned_addr learned my addr 172.21.15.104:0/2806903211 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1567700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 0x7fbfdc0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe2569700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc15c580 0x7fbfdc0c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 -- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc15c580 msgr2=0x7fbfdc0c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc15c580 0x7fbfdc0c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 -- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 msgr2=0x7fbfdc0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 0x7fbfdc0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 -- 172.21.15.104:0/2806903211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc009500 con 0x7fbfdc06cde0 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1567700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 0x7fbfdc0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe1d68700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc000c00 tx=0x7fbfcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc003d90 con 0x7fbfdc06cde0 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc011420 con 0x7fbfdc06cde0 2023-06-10T18:44:54.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc0b4460 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc0117b0 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.974+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc0ce980 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.975+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc0b4980 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.976+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbfcc011910 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.976+0000 7fbfd27fc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 0x7fbfc006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.976+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbfcc08a960 con 0x7fbfdc06cde0 2023-06-10T18:44:54.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.976+0000 7fbfe1567700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 0x7fbfc006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:54.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.977+0000 7fbfe1567700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 0x7fbfc006d190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0cf0a0 tx=0x7fbfd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:54.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:54.981+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbfcc055770 con 0x7fbfdc06cde0 2023-06-10T18:44:55.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.165+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbfdc066f10 con 0x7fbfdc06cde0 2023-06-10T18:44:55.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.165+0000 7fbfd27fc700 1 -- 172.21.15.104:0/2806903211 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbfcc058d90 con 0x7fbfdc06cde0 2023-06-10T18:44:55.172 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:55.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.167+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 msgr2=0x7fbfc006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.167+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 0x7fbfc006d190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0cf0a0 tx=0x7fbfd8008040 comp rx=0 tx=0).stop 2023-06-10T18:44:55.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 msgr2=0x7fbfdc0ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc000c00 tx=0x7fbfcc004750 comp rx=0 tx=0).stop 2023-06-10T18:44:55.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 shutdown_connections 2023-06-10T18:44:55.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfdc15c580 0x7fbfdc0c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfdc06cde0 0x7fbfdc0ce440 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfc006ace0 0x7fbfc006d190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 --2- 172.21.15.104:0/2806903211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfdc06e6e0 0x7fbfdc0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 >> 172.21.15.104:0/2806903211 conn(0x7fbfdc06b8a0 msgr2=0x7fbfdc0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 shutdown_connections 2023-06-10T18:44:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.168+0000 7fbfe3fcc700 1 -- 172.21.15.104:0/2806903211 wait complete. 2023-06-10T18:44:55.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:55.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.302+0000 7f5a88de9700 1 Processor -- start 2023-06-10T18:44:55.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.302+0000 7f5a88de9700 1 -- start start 2023-06-10T18:44:55.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a88de9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5a8415c580 0x7f5a840b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a88de9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a840b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a88de9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8406e250 con 0x7f5a8406cde0 2023-06-10T18:44:55.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a88de9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8415c950 con 0x7f5a8415c580 2023-06-10T18:44:55.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a88de9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840cf0e0 con 0x7f5a8406e6e0 2023-06-10T18:44:55.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a82d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5a8406cde0 0x7f5a84069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53416/0 (socket says 172.21.15.104:53416) 2023-06-10T18:44:55.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.303+0000 7f5a82d9d700 1 -- 172.21.15.104:0/3066888783 learned_addr learned my addr 172.21.15.104:0/3066888783 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461682310 0 0) 0x7f5a8406e250 con 0x7f5a8406cde0 2023-06-10T18:44:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c0035b0 con 0x7f5a8406cde0 2023-06-10T18:44:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a840b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8259c700 1 --2- 172.21.15.104:0/3066888783 >> v2:172.21.15.114:3300/0 conn(0x7f5a8415c580 0x7f5a840b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 161858060 0 0) 0x7f5a6c0035b0 con 0x7f5a8406cde0 2023-06-10T18:44:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 msgr2=0x7f5a840b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a840b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 >> v2:172.21.15.114:3300/0 conn(0x7f5a8415c580 msgr2=0x7f5a840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 --2- 172.21.15.104:0/3066888783 >> v2:172.21.15.114:3300/0 conn(0x7f5a8415c580 0x7f5a840b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840b1e80 con 0x7f5a8406cde0 2023-06-10T18:44:55.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5a7c0019f0 con 0x7f5a8406cde0 2023-06-10T18:44:55.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 >> v1:172.21.15.104:6789/0 conn(0x7f5a8406cde0 legacy=0x7f5a84069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a6c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a6c003f80 0x7f5a6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c004d90 0x7f5a6c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8415c950 con 0x7f5a6c004d90 2023-06-10T18:44:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840cf0e0 con 0x7f5a6c003f80 2023-06-10T18:44:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c00ba10 con 0x7f5a8406e6e0 2023-06-10T18:44:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a82d9d700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c004d90 0x7f5a6c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a8259c700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a6c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a6c003f80 0x7f5a6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 -- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 msgr2=0x7f5a6c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8406e6e0 0x7f5a6c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 -- 172.21.15.104:0/3066888783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c004d90 msgr2=0x7f5a6c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c004d90 0x7f5a6c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a81d9b700 1 -- 172.21.15.104:0/3066888783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0035b0 con 0x7f5a6c003f80 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.304+0000 7f5a82d9d700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c004d90 0x7f5a6c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a6c003f80 0x7f5a6c004850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009b60 tx=0x7f5a6c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d35f68d6ec3fc9b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400e6c0 con 0x7f5a6c003f80 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7400e820 con 0x7f5a6c003f80 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a8159a700 1 -- 172.21.15.104:0/3066888783 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7400eb90 con 0x7f5a6c003f80 2023-06-10T18:44:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 -- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a6c003f80 msgr2=0x7f5a6c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3066888783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a6c003f80 0x7f5a6c004850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009b60 tx=0x7f5a6c005880 comp rx=0 tx=0).stop 2023-06-10T18:44:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a82d9d700 1 -- 172.21.15.104:0/3066888783 reap_dead start 2023-06-10T18:44:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 -- 172.21.15.104:0/3066888783 shutdown_connections 2023-06-10T18:44:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 -- 172.21.15.104:0/3066888783 >> 172.21.15.104:0/3066888783 conn(0x7f5a8406b8a0 msgr2=0x7f5a840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 -- 172.21.15.104:0/3066888783 shutdown_connections 2023-06-10T18:44:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.305+0000 7f5a88de9700 1 -- 172.21.15.104:0/3066888783 wait complete. 2023-06-10T18:44:55.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.307+0000 7f5a88de9700 1 Processor -- start 2023-06-10T18:44:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.307+0000 7f5a88de9700 1 -- start start 2023-06-10T18:44:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a840ae120 0x7f5a8415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a84158f00 0x7f5a84159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a84158150 con 0x7f5a6c003f80 2023-06-10T18:44:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a841598b0 con 0x7f5a840ae120 2023-06-10T18:44:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a88de9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8415d070 con 0x7f5a84158f00 2023-06-10T18:44:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55596/0 (socket says 172.21.15.104:55596) 2023-06-10T18:44:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 -- 172.21.15.104:0/3658001743 learned_addr learned my addr 172.21.15.104:0/3658001743 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a82d9d700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a84158f00 0x7f5a84159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a840ae120 0x7f5a8415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 -- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a84158f00 msgr2=0x7f5a84159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a84158f00 0x7f5a84159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 -- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a840ae120 msgr2=0x7f5a8415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a840ae120 0x7f5a8415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.308+0000 7f5a8259c700 1 -- 172.21.15.104:0/3658001743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840ae590 con 0x7f5a6c003f80 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a8259c700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a78011aa0 tx=0x7f5a78011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78005a20 con 0x7f5a6c003f80 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78005b80 con 0x7f5a6c003f80 2023-06-10T18:44:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8415d2a0 con 0x7f5a6c003f80 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7800f940 con 0x7f5a6c003f80 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.309+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8415d860 con 0x7f5a6c003f80 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.310+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a840ceca0 con 0x7f5a6c003f80 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.310+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a7800faa0 con 0x7f5a6c003f80 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.310+0000 7f5a72ffd700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 0x7f5a6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.310+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5a78089b30 con 0x7f5a6c003f80 2023-06-10T18:44:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.311+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 0x7f5a6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.311+0000 7f5a81d9b700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 0x7f5a6406d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a740090b0 tx=0x7f5a74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.315+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a780548b0 con 0x7f5a6c003f80 2023-06-10T18:44:55.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.500+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a840cfa20 con 0x7f5a6c003f80 2023-06-10T18:44:55.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.501+0000 7f5a72ffd700 1 -- 172.21.15.104:0/3658001743 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a78022020 con 0x7f5a6c003f80 2023-06-10T18:44:55.506 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:55.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 msgr2=0x7f5a6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 0x7f5a6406d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a740090b0 tx=0x7f5a74008030 comp rx=0 tx=0).stop 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 msgr2=0x7f5a84157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a78011aa0 tx=0x7f5a78011db0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 shutdown_connections 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a84158f00 0x7f5a84159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a6c003f80 0x7f5a84157c10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a6406ace0 0x7f5a6406d190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 --2- 172.21.15.104:0/3658001743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a840ae120 0x7f5a8415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 >> 172.21.15.104:0/3658001743 conn(0x7f5a8406b8a0 msgr2=0x7f5a840b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 shutdown_connections 2023-06-10T18:44:55.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.503+0000 7f5a88de9700 1 -- 172.21.15.104:0/3658001743 wait complete. 2023-06-10T18:44:55.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:55.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 Processor -- start 2023-06-10T18:44:55.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 -- start start 2023-06-10T18:44:55.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80fc0b21c0 0x7f80fc0b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0b2b40 con 0x7f80fc156760 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc0b30a0 con 0x7f80fc0b21c0 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.638+0000 7f8102237700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc15d740 con 0x7f80fc15d1a0 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80faffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80fc0b21c0 0x7f80fc0b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80faffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46758/0 (socket says 172.21.15.104:46758) 2023-06-10T18:44:55.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80faffd700 1 -- 172.21.15.104:0/434190427 learned_addr learned my addr 172.21.15.104:0/434190427 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:55.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 -- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 msgr2=0x7f80fc0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 -- 172.21.15.104:0/434190427 >> v1:172.21.15.114:6789/0 conn(0x7f80fc156760 legacy=0x7f80fc0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 -- 172.21.15.104:0/434190427 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0cfca0 con 0x7f80fc0b21c0 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80faffd700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.639+0000 7f80fbfff700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc0b21c0 0x7f80fc0b2600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009dc0 tx=0x7f80f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecb443402a88f56f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f80fa7fc700 1 -- 172.21.15.104:0/434190427 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400e070 con 0x7f80fc0b21c0 2023-06-10T18:44:55.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f80fa7fc700 1 -- 172.21.15.104:0/434190427 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f400bae0 con 0x7f80fc0b21c0 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f80fa7fc700 1 -- 172.21.15.104:0/434190427 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400be90 con 0x7f80fc0b21c0 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 -- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc0b21c0 msgr2=0x7f80fc0b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc0b21c0 0x7f80fc0b2600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009dc0 tx=0x7f80f400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 -- 172.21.15.104:0/434190427 shutdown_connections 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc0b21c0 0x7f80fc0b2600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 --2- 172.21.15.104:0/434190427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc15d1a0 0x7f80fc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 -- 172.21.15.104:0/434190427 >> 172.21.15.104:0/434190427 conn(0x7f80fc06e370 msgr2=0x7f80fc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 -- 172.21.15.104:0/434190427 shutdown_connections 2023-06-10T18:44:55.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.640+0000 7f8102237700 1 -- 172.21.15.104:0/434190427 wait complete. 2023-06-10T18:44:55.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.641+0000 7f8102237700 1 Processor -- start 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 -- start start 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc0b21c0 0x7f80fc14e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc15d1a0 0x7f80fc0b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc14eed0 con 0x7f80fc156760 2023-06-10T18:44:55.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0c4fb0 con 0x7f80fc15d1a0 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f8102237700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0afe50 con 0x7f80fc0b21c0 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f80faffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f80faffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55612/0 (socket says 172.21.15.104:55612) 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.642+0000 7f80faffd700 1 -- 172.21.15.104:0/1999910373 learned_addr learned my addr 172.21.15.104:0/1999910373 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80fb7fe700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc0b21c0 0x7f80fc14e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80fbfff700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc15d1a0 0x7f80fc0b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 -- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc0b21c0 msgr2=0x7f80fc14e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc0b21c0 0x7f80fc14e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 -- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc15d1a0 msgr2=0x7f80fc0b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc15d1a0 0x7f80fc0b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 -- 172.21.15.104:0/1999910373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0009550 con 0x7f80fc156760 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80faffd700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e9b0 tx=0x7f80f000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000ee50 con 0x7f80fc156760 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f000c850 con 0x7f80fc156760 2023-06-10T18:44:55.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f4009a70 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f000cbc0 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.643+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc0b0410 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.644+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc074050 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.644+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f80f000cd20 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.645+0000 7f80e3fff700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 0x7f80e806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.645+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f80f00896a0 con 0x7f80fc156760 2023-06-10T18:44:55.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.645+0000 7f80fb7fe700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 0x7f80e806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.645+0000 7f80fb7fe700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 0x7f80e806d1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0018a0 tx=0x7f80ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.649+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f80f00544b0 con 0x7f80fc156760 2023-06-10T18:44:55.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.838+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f80fc0d01b0 con 0x7f80fc156760 2023-06-10T18:44:55.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.838+0000 7f80e3fff700 1 -- 172.21.15.104:0/1999910373 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f80f0057ad0 con 0x7f80fc156760 2023-06-10T18:44:55.844 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:55.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 msgr2=0x7f80e806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 0x7f80e806d1c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0018a0 tx=0x7f80ec009040 comp rx=0 tx=0).stop 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 msgr2=0x7f80fc0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e9b0 tx=0x7f80f000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 shutdown_connections 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80e806ad10 0x7f80e806d1c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80fc15d1a0 0x7f80fc0b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80fc0b21c0 0x7f80fc14e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 --2- 172.21.15.104:0/1999910373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80fc156760 0x7f80fc0c4480 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 >> 172.21.15.104:0/1999910373 conn(0x7f80fc06e370 msgr2=0x7f80fc15d880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 shutdown_connections 2023-06-10T18:44:55.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.840+0000 7f8102237700 1 -- 172.21.15.104:0/1999910373 wait complete. 2023-06-10T18:44:55.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:55.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.971+0000 7fd9436e7700 1 Processor -- start 2023-06-10T18:44:55.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.971+0000 7fd9436e7700 1 -- start start 2023-06-10T18:44:55.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd9436e7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd93c0cf100 0x7fd93c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd9436e7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c06d290 con 0x7fd93c06e4b0 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd9436e7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c0ca2d0 con 0x7fd93c06e0e0 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd9436e7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c071640 con 0x7fd93c0cf100 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd940c82700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd93c06e4b0 0x7fd93c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53448/0 (socket says 172.21.15.104:53448) 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd940c82700 1 -- 172.21.15.104:0/2100735131 learned_addr learned my addr 172.21.15.104:0/2100735131 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd941c84700 1 --2- 172.21.15.104:0/2100735131 >> v2:172.21.15.114:3300/0 conn(0x7fd93c0cf100 0x7fd93c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892524940 0 0) 0x7fd93c06d290 con 0x7fd93c06e4b0 2023-06-10T18:44:55.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.972+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9280035b0 con 0x7fd93c06e4b0 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966612389 0 0) 0x7fd9280035b0 con 0x7fd93c06e4b0 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 >> v2:172.21.15.114:3300/0 conn(0x7fd93c0cf100 msgr2=0x7fd93c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 --2- 172.21.15.104:0/2100735131 >> v2:172.21.15.114:3300/0 conn(0x7fd93c0cf100 0x7fd93c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 >> v1:172.21.15.114:6789/0 conn(0x7fd93c06e0e0 legacy=0x7fd93c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c06d4c0 con 0x7fd93c06e4b0 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd938001670 con 0x7fd93c06e4b0 2023-06-10T18:44:55.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 >> v1:172.21.15.104:6789/0 conn(0x7fd93c06e4b0 legacy=0x7fd93c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd928003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd928004850 0x7fd928004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd928005200 0x7fd92800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c071640 con 0x7fd928003a00 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92800c110 con 0x7fd928004850 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92800c2d0 con 0x7fd928005200 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd928003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd940c82700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd928004850 0x7fd928004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941c84700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd928005200 0x7fd92800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 -- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd928005200 msgr2=0x7fd92800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd928005200 0x7fd92800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 -- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd928004850 msgr2=0x7fd928004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd928004850 0x7fd928004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 -- 172.21.15.104:0/2100735131 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9280035b0 con 0x7fd928003a00 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941c84700 1 -- 172.21.15.104:0/2100735131 reap_dead start 2023-06-10T18:44:55.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd940c82700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd928004850 0x7fd928004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd941483700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd928003e10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd92c001820 tx=0x7fd928005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf8ff216a8aa2cd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0012e0 con 0x7fd928003a00 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c0052d0 con 0x7fd928003a00 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.973+0000 7fd933fff700 1 -- 172.21.15.104:0/2100735131 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0036d0 con 0x7fd928003a00 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 -- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 msgr2=0x7fd928003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd928003e10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd92c001820 tx=0x7fd928005880 comp rx=0 tx=0).stop 2023-06-10T18:44:55.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 -- 172.21.15.104:0/2100735131 shutdown_connections 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 --2- 172.21.15.104:0/2100735131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd928003e10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 -- 172.21.15.104:0/2100735131 >> 172.21.15.104:0/2100735131 conn(0x7fd93c06b8a0 msgr2=0x7fd93c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 -- 172.21.15.104:0/2100735131 shutdown_connections 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.974+0000 7fd9436e7700 1 -- 172.21.15.104:0/2100735131 wait complete. 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.975+0000 7fd9436e7700 1 Processor -- start 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.975+0000 7fd9436e7700 1 -- start start 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93c1505d0 0x7fd93c0b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93c0b2970 0x7fd93c150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b41c0 con 0x7fd928003a00 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b2de0 con 0x7fd93c1505d0 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd9436e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b4c50 con 0x7fd93c0b2970 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55630/0 (socket says 172.21.15.104:55630) 2023-06-10T18:44:55.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 -- 172.21.15.104:0/512957332 learned_addr learned my addr 172.21.15.104:0/512957332 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 -- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93c0b2970 msgr2=0x7fd93c150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93c0b2970 0x7fd93c150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 -- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93c1505d0 msgr2=0x7fd93c0b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93c1505d0 0x7fd93c0b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 -- 172.21.15.104:0/512957332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0b43b0 con 0x7fd928003a00 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.976+0000 7fd941483700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd92c001e10 tx=0x7fd92c001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.977+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0012e0 con 0x7fd928003a00 2023-06-10T18:44:55.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.977+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c0050d0 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.977+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c0b4690 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.977+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c01d420 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.977+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c0c6770 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.978+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c074050 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.978+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd92c01d580 con 0x7fd928003a00 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.979+0000 7fd931ffb700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 0x7fd92006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:55.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.979+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd92c08a060 con 0x7fd928003a00 2023-06-10T18:44:55.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.979+0000 7fd940c82700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 0x7fd92006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:55.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.979+0000 7fd940c82700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 0x7fd92006d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd938001bb0 tx=0x7fd938001b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:55.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:55.983+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd92c054e70 con 0x7fd928003a00 2023-06-10T18:44:56.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.172+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd93c0ac5b0 con 0x7fd928003a00 2023-06-10T18:44:56.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.173+0000 7fd931ffb700 1 -- 172.21.15.104:0/512957332 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7fd92c058490 con 0x7fd928003a00 2023-06-10T18:44:56.180 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:56.180 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 msgr2=0x7fd92006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 0x7fd92006d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd938001bb0 tx=0x7fd938001b20 comp rx=0 tx=0).stop 2023-06-10T18:44:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 msgr2=0x7fd93c0c89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd92c001e10 tx=0x7fd92c001170 comp rx=0 tx=0).stop 2023-06-10T18:44:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 shutdown_connections 2023-06-10T18:44:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93c0b2970 0x7fd93c150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd928003a00 0x7fd93c0c89e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd92006ace0 0x7fd92006d190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 --2- 172.21.15.104:0/512957332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93c1505d0 0x7fd93c0b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.175+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 >> 172.21.15.104:0/512957332 conn(0x7fd93c06b8a0 msgr2=0x7fd93c0cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.176+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 shutdown_connections 2023-06-10T18:44:56.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.176+0000 7fd9436e7700 1 -- 172.21.15.104:0/512957332 wait complete. 2023-06-10T18:44:56.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:56.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:56.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.310+0000 7f93ed899700 1 Processor -- start 2023-06-10T18:44:56.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.310+0000 7f93ed899700 1 -- start start 2023-06-10T18:44:56.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f93e80d04f0 0x7f93e80d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93e80d25c0 0x7f93e80d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e806b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e80d0e00 con 0x7f93e80d04f0 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e80af530 con 0x7f93e80d25c0 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93ed899700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80af810 con 0x7f93e806b510 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93e6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f93e80d04f0 0x7f93e80d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93e6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f93e80d04f0 0x7f93e80d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55638/0 (socket says 172.21.15.104:55638) 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93e6ffd700 1 -- 172.21.15.104:0/898223621 learned_addr learned my addr 172.21.15.104:0/898223621 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93e77fe700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e806b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.311+0000 7f93e67fc700 1 --2- 172.21.15.104:0/898223621 >> v2:172.21.15.114:3300/0 conn(0x7f93e80d25c0 0x7f93e80d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 -- 172.21.15.104:0/898223621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 msgr2=0x7f93e806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e806b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 -- 172.21.15.104:0/898223621 >> v2:172.21.15.114:3300/0 conn(0x7f93e80d25c0 msgr2=0x7f93e80d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/898223621 >> v2:172.21.15.114:3300/0 conn(0x7f93e80d25c0 0x7f93e80d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 -- 172.21.15.104:0/898223621 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80afaf0 con 0x7f93e80d04f0 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e77fe700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e806b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e80d34d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f93e80b5490 tx=0x7f93dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98fa85146a865de3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e5ffb700 1 -- 172.21.15.104:0/898223621 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc005e30 con 0x7f93e80d04f0 2023-06-10T18:44:56.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e5ffb700 1 -- 172.21.15.104:0/898223621 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc004030 con 0x7f93e80d04f0 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93e5ffb700 1 -- 172.21.15.104:0/898223621 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0043e0 con 0x7f93e80d04f0 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 -- 172.21.15.104:0/898223621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 msgr2=0x7f93e80d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e80d34d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f93e80b5490 tx=0x7f93dc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 -- 172.21.15.104:0/898223621 shutdown_connections 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e80d34d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 --2- 172.21.15.104:0/898223621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e806b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 --2- 172.21.15.104:0/898223621 >> v2:172.21.15.114:3300/0 conn(0x7f93e80d25c0 0x7f93e80d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 -- 172.21.15.104:0/898223621 >> 172.21.15.104:0/898223621 conn(0x7f93e8071ec0 msgr2=0x7f93e809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.312+0000 7f93ed899700 1 -- 172.21.15.104:0/898223621 shutdown_connections 2023-06-10T18:44:56.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.313+0000 7f93ed899700 1 -- 172.21.15.104:0/898223621 wait complete. 2023-06-10T18:44:56.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.314+0000 7f93ed899700 1 Processor -- start 2023-06-10T18:44:56.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.314+0000 7f93ed899700 1 -- start start 2023-06-10T18:44:56.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e81599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93e80d25c0 0x7f93e80cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80a0110 con 0x7f93e80d04f0 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80cc830 con 0x7f93e80d25c0 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e809f770 con 0x7f93e806b510 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55646/0 (socket says 172.21.15.104:55646) 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 -- 172.21.15.104:0/1607276659 learned_addr learned my addr 172.21.15.104:0/1607276659 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:56.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e77fe700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93e80d25c0 0x7f93e80cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e81599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 -- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 msgr2=0x7f93e81599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e81599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 -- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93e80d25c0 msgr2=0x7f93e80cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93e80d25c0 0x7f93e80cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 -- 172.21.15.104:0/1607276659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc009500 con 0x7f93e80d04f0 2023-06-10T18:44:56.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93e67fc700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f93d800e810 tx=0x7f93d800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d800c070 con 0x7f93e80d04f0 2023-06-10T18:44:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8013610 con 0x7f93e80d04f0 2023-06-10T18:44:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e809fa50 con 0x7f93e80d04f0 2023-06-10T18:44:56.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80139c0 con 0x7f93e80d04f0 2023-06-10T18:44:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.315+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e80a0530 con 0x7f93e80d04f0 2023-06-10T18:44:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.316+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e8067010 con 0x7f93e80d04f0 2023-06-10T18:44:56.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.317+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f93d8013b20 con 0x7f93e80d04f0 2023-06-10T18:44:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.317+0000 7f93cffff700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 0x7f93d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.317+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f93d8089480 con 0x7f93e80d04f0 2023-06-10T18:44:56.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.317+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 0x7f93d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.318+0000 7f93e6ffd700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 0x7f93d406d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93e80b5490 tx=0x7f93dc008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.321+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f93d8057ca0 con 0x7f93e80d04f0 2023-06-10T18:44:56.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.509+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93e8074050 con 0x7f93e80d04f0 2023-06-10T18:44:56.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.509+0000 7f93cffff700 1 -- 172.21.15.104:0/1607276659 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f93d8057830 con 0x7f93e80d04f0 2023-06-10T18:44:56.515 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:56.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 msgr2=0x7f93d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 0x7f93d406d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93e80b5490 tx=0x7f93dc008070 comp rx=0 tx=0).stop 2023-06-10T18:44:56.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 msgr2=0x7f93e8159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f93d800e810 tx=0x7f93d800eb20 comp rx=0 tx=0).stop 2023-06-10T18:44:56.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 shutdown_connections 2023-06-10T18:44:56.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f93d406ace0 0x7f93d406d190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93e80d25c0 0x7f93e80cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93e806b510 0x7f93e81599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 --2- 172.21.15.104:0/1607276659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93e80d04f0 0x7f93e8159120 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 >> 172.21.15.104:0/1607276659 conn(0x7f93e8071ec0 msgr2=0x7f93e80b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 shutdown_connections 2023-06-10T18:44:56.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.511+0000 7f93ed899700 1 -- 172.21.15.104:0/1607276659 wait complete. 2023-06-10T18:44:56.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.647+0000 7f98cb5a7700 1 Processor -- start 2023-06-10T18:44:56.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.647+0000 7f98cb5a7700 1 -- start start 2023-06-10T18:44:56.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98cb5a7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98c40b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98cb5a7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98c406cde0 0x7f98c40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98cb5a7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c406e250 con 0x7f98c415c580 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98cb5a7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c415c950 con 0x7f98c406cde0 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98cb5a7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40b1e80 con 0x7f98c406e6e0 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98c9343700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f98c415c580 0x7f98c40b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53480/0 (socket says 172.21.15.104:53480) 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98c9343700 1 -- 172.21.15.104:0/2104103070 learned_addr learned my addr 172.21.15.104:0/2104103070 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98c9b44700 1 --2- 172.21.15.104:0/2104103070 >> v2:172.21.15.114:3300/0 conn(0x7f98c406cde0 0x7f98c40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98c8b42700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98c40b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381312961 0 0) 0x7f98c406e250 con 0x7f98c415c580 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.648+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b00035b0 con 0x7f98c415c580 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360959361 0 0) 0x7f98b00035b0 con 0x7f98c415c580 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 msgr2=0x7f98c40b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98c40b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 >> v2:172.21.15.114:3300/0 conn(0x7f98c406cde0 msgr2=0x7f98c40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 --2- 172.21.15.104:0/2104103070 >> v2:172.21.15.114:3300/0 conn(0x7f98c406cde0 0x7f98c40ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c406d920 con 0x7f98c415c580 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f98b40017f0 con 0x7f98c415c580 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 >> v1:172.21.15.104:6789/0 conn(0x7f98c415c580 legacy=0x7f98c40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0003a00 0x7f98b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b0004430 0x7f98b00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98b0004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c415c950 con 0x7f98b0004430 2023-06-10T18:44:56.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40b1e80 con 0x7f98b0003a00 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b000ba10 con 0x7f98c406e6e0 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b0004430 0x7f98b00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c9343700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0003a00 0x7f98b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c9b44700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98b0004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 -- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 msgr2=0x7f98b0004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c406e6e0 0x7f98b0004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 -- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0003a00 msgr2=0x7f98b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0003a00 0x7f98b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 -- 172.21.15.104:0/2104103070 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b00035b0 con 0x7f98b0004430 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98c8b42700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b0004430 0x7f98b00048f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009b60 tx=0x7f98b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59291a834e745691 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c000e6c0 con 0x7f98b0004430 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000e820 con 0x7f98b0004430 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.649+0000 7f98bbfff700 1 -- 172.21.15.104:0/2104103070 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c000eb90 con 0x7f98b0004430 2023-06-10T18:44:56.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 -- 172.21.15.104:0/2104103070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b0004430 msgr2=0x7f98b00048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/2104103070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b0004430 0x7f98b00048f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009b60 tx=0x7f98b0005880 comp rx=0 tx=0).stop 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98c9b44700 1 -- 172.21.15.104:0/2104103070 reap_dead start 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 -- 172.21.15.104:0/2104103070 shutdown_connections 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 -- 172.21.15.104:0/2104103070 >> 172.21.15.104:0/2104103070 conn(0x7f98c406b8a0 msgr2=0x7f98c40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 -- 172.21.15.104:0/2104103070 shutdown_connections 2023-06-10T18:44:56.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.650+0000 7f98cb5a7700 1 -- 172.21.15.104:0/2104103070 wait complete. 2023-06-10T18:44:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.652+0000 7f98cb5a7700 1 Processor -- start 2023-06-10T18:44:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.652+0000 7f98cb5a7700 1 -- start start 2023-06-10T18:44:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0004430 0x7f98c4155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 0x7f98c4157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40ad970 con 0x7f98c40acfc0 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4156440 con 0x7f98b0004430 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40b3130 con 0x7f98c41571c0 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55668/0 (socket says 172.21.15.104:55668) 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 -- 172.21.15.104:0/726838726 learned_addr learned my addr 172.21.15.104:0/726838726 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:56.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9343700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0004430 0x7f98c4155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c8b42700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 0x7f98c4157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 -- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 msgr2=0x7f98c4157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 0x7f98c4157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 -- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0004430 msgr2=0x7f98c4155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0004430 0x7f98c4155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 -- 172.21.15.104:0/726838726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c41566e0 con 0x7f98c40acfc0 2023-06-10T18:44:56.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c8b42700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 0x7f98c4157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98c9b44700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f98c40701c0 tx=0x7f98bc011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00e040 con 0x7f98c40acfc0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00f040 con 0x7f98c40acfc0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c41569c0 con 0x7f98c40acfc0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.653+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc015430 con 0x7f98c40acfc0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.654+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4155fc0 con 0x7f98c40acfc0 2023-06-10T18:44:56.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.654+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c40d0500 con 0x7f98c40acfc0 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.655+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f98bc015590 con 0x7f98c40acfc0 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.655+0000 7f98b9ffb700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 0x7f98a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.655+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f98bc089d50 con 0x7f98c40acfc0 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.655+0000 7f98c9343700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 0x7f98a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.656+0000 7f98c9343700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 0x7f98a806d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000ef0 tx=0x7f98b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.660+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f98bc08d030 con 0x7f98c40acfc0 2023-06-10T18:44:56.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.847+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f98c4074050 con 0x7f98c40acfc0 2023-06-10T18:44:56.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.847+0000 7f98b9ffb700 1 -- 172.21.15.104:0/726838726 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f98bc054ad0 con 0x7f98c40acfc0 2023-06-10T18:44:56.854 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:56.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 msgr2=0x7f98a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 0x7f98a806d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000ef0 tx=0x7f98b4009040 comp rx=0 tx=0).stop 2023-06-10T18:44:56.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 msgr2=0x7f98c40ad430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f98c40701c0 tx=0x7f98bc011f60 comp rx=0 tx=0).stop 2023-06-10T18:44:56.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 shutdown_connections 2023-06-10T18:44:56.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f98a806ace0 0x7f98a806d190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98b0004430 0x7f98c4155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98c40acfc0 0x7f98c40ad430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 --2- 172.21.15.104:0/726838726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98c41571c0 0x7f98c4157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 >> 172.21.15.104:0/726838726 conn(0x7f98c406b8a0 msgr2=0x7f98c40cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 shutdown_connections 2023-06-10T18:44:56.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.849+0000 7f98cb5a7700 1 -- 172.21.15.104:0/726838726 wait complete. 2023-06-10T18:44:56.866 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.984+0000 7f5f1fe7f700 1 Processor -- start 2023-06-10T18:44:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.984+0000 7f5f1fe7f700 1 -- start start 2023-06-10T18:44:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1fe7f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f18071120 0x7f5f18163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:56.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1fe7f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1806d7d0 con 0x7f5f181586a0 2023-06-10T18:44:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1fe7f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180cd6c0 con 0x7f5f180ce390 2023-06-10T18:44:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1fe7f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180c95d0 con 0x7f5f18071120 2023-06-10T18:44:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1dc1b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5f181586a0 0x7f5f1806cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53500/0 (socket says 172.21.15.104:53500) 2023-06-10T18:44:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1dc1b700 1 -- 172.21.15.104:0/478315544 learned_addr learned my addr 172.21.15.104:0/478315544 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.985+0000 7f5f1e41c700 1 --2- 172.21.15.104:0/478315544 >> v2:172.21.15.114:3300/0 conn(0x7f5f18071120 0x7f5f18163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438937939 0 0) 0x7f5f1806d7d0 con 0x7f5f181586a0 2023-06-10T18:44:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f040046c0 con 0x7f5f181586a0 2023-06-10T18:44:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1e41c700 1 -- 172.21.15.104:0/478315544 >> v1:172.21.15.114:6789/0 conn(0x7f5f180ce390 legacy=0x7f5f18070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774375898 0 0) 0x7f5f180cd6c0 con 0x7f5f180ce390 2023-06-10T18:44:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1e41c700 1 -- 172.21.15.104:0/478315544 >> v1:172.21.15.104:6789/0 conn(0x7f5f181586a0 legacy=0x7f5f1806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1e41c700 1 -- 172.21.15.104:0/478315544 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f180cd910 con 0x7f5f18071120 2023-06-10T18:44:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1e41c700 1 --2- 172.21.15.104:0/478315544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f18071120 0x7f5f18163160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009d90 tx=0x7f5f1000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327b95c8d55fde2f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000e070 con 0x7f5f18071120 2023-06-10T18:44:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1000bb20 con 0x7f5f18071120 2023-06-10T18:44:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.986+0000 7f5f1cc19700 1 -- 172.21.15.104:0/478315544 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000bed0 con 0x7f5f18071120 2023-06-10T18:44:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/478315544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f18071120 msgr2=0x7f5f18163160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/478315544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f18071120 0x7f5f18163160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009d90 tx=0x7f5f1000b330 comp rx=0 tx=0).stop 2023-06-10T18:44:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/478315544 shutdown_connections 2023-06-10T18:44:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/478315544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f18071120 0x7f5f18163160 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/478315544 >> 172.21.15.104:0/478315544 conn(0x7f5f180ceec0 msgr2=0x7f5f180cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/478315544 shutdown_connections 2023-06-10T18:44:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.987+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/478315544 wait complete. 2023-06-10T18:44:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.988+0000 7f5f1fe7f700 1 Processor -- start 2023-06-10T18:44:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.988+0000 7f5f1fe7f700 1 -- start start 2023-06-10T18:44:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f180ce390 0x7f5f18158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f181586a0 0x7f5f18143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f181417d0 con 0x7f5f18071120 2023-06-10T18:44:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180ac750 con 0x7f5f181586a0 2023-06-10T18:44:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1fe7f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180aca30 con 0x7f5f180ce390 2023-06-10T18:44:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55678/0 (socket says 172.21.15.104:55678) 2023-06-10T18:44:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 -- 172.21.15.104:0/2531795978 learned_addr learned my addr 172.21.15.104:0/2531795978 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1d41a700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f180ce390 0x7f5f18158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1e41c700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f181586a0 0x7f5f18143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 -- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f180ce390 msgr2=0x7f5f18158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f180ce390 0x7f5f18158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 -- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f181586a0 msgr2=0x7f5f18143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f181586a0 0x7f5f18143b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.989+0000 7f5f1dc1b700 1 -- 172.21.15.104:0/2531795978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10009a70 con 0x7f5f18071120 2023-06-10T18:44:57.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f1dc1b700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f14001eb0 tx=0x7f5f1400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14005d70 con 0x7f5f18071120 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f140070d0 con 0x7f5f18071120 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f180acd10 con 0x7f5f18071120 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14007460 con 0x7f5f18071120 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.990+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f180ad2d0 con 0x7f5f18071120 2023-06-10T18:44:57.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.991+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f180ad4c0 con 0x7f5f18071120 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.991+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5f140075c0 con 0x7f5f18071120 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.992+0000 7f5f0e7fc700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 0x7f5efc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.992+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5f14088b80 con 0x7f5f18071120 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.992+0000 7f5f1d41a700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 0x7f5efc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.993+0000 7f5f1d41a700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 0x7f5efc06d1c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f180c4a40 tx=0x7f5f08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:56.997+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5f14053900 con 0x7f5f18071120 2023-06-10T18:44:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.184+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f18074050 con 0x7f5f18071120 2023-06-10T18:44:57.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.185+0000 7f5f0e7fc700 1 -- 172.21.15.104:0/2531795978 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f14056f20 con 0x7f5f18071120 2023-06-10T18:44:57.191 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 msgr2=0x7f5efc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 0x7f5efc06d1c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f180c4a40 tx=0x7f5f08009040 comp rx=0 tx=0).stop 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 msgr2=0x7f5f18141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f14001eb0 tx=0x7f5f1400c7b0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 shutdown_connections 2023-06-10T18:44:57.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5efc06ad10 0x7f5efc06d1c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f181586a0 0x7f5f18143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f18071120 0x7f5f18141290 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 --2- 172.21.15.104:0/2531795978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f180ce390 0x7f5f18158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 >> 172.21.15.104:0/2531795978 conn(0x7f5f180ceec0 msgr2=0x7f5f180cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 shutdown_connections 2023-06-10T18:44:57.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.187+0000 7f5f1fe7f700 1 -- 172.21.15.104:0/2531795978 wait complete. 2023-06-10T18:44:57.205 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:57.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.322+0000 7f57c093f700 1 Processor -- start 2023-06-10T18:44:57.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.322+0000 7f57c093f700 1 -- start start 2023-06-10T18:44:57.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57c093f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57c093f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 0x7f57bc06e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57c093f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc15c8d0 con 0x7f57bc0cf200 2023-06-10T18:44:57.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57c093f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc0aef20 con 0x7f57bc06dcd0 2023-06-10T18:44:57.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57c093f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc06e6b0 con 0x7f57bc15c500 2023-06-10T18:44:57.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57ba59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.323+0000 7f57bad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 0x7f57bc06e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57bad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 0x7f57bc06e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:43188/0 (socket says 172.21.15.104:43188) 2023-06-10T18:44:57.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57b9d9b700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f57bc0cf200 0x7f57bc069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59362/0 (socket says 172.21.15.104:59362) 2023-06-10T18:44:57.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:46838/0 (socket says 172.21.15.104:46838) 2023-06-10T18:44:57.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57bad9d700 1 -- 172.21.15.104:0/2947684749 learned_addr learned my addr 172.21.15.104:0/2947684749 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:57.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 -- 172.21.15.104:0/2947684749 >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 msgr2=0x7f57bc06e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:57.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 --2- 172.21.15.104:0/2947684749 >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 0x7f57bc06e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 -- 172.21.15.104:0/2947684749 >> v1:172.21.15.114:6789/0 conn(0x7f57bc0cf200 legacy=0x7f57bc069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 -- 172.21.15.104:0/2947684749 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc06abd0 con 0x7f57bc15c500 2023-06-10T18:44:57.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57ba59c700 1 --2- 172.21.15.104:0/2947684749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009820 tx=0x7f57b0009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfbb314b7674c8b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57b959a700 1 -- 172.21.15.104:0/2947684749 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0005e30 con 0x7f57bc15c500 2023-06-10T18:44:57.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57b959a700 1 -- 172.21.15.104:0/2947684749 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b0006030 con 0x7f57bc15c500 2023-06-10T18:44:57.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.324+0000 7f57b959a700 1 -- 172.21.15.104:0/2947684749 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00063c0 con 0x7f57bc15c500 2023-06-10T18:44:57.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 -- 172.21.15.104:0/2947684749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 msgr2=0x7f57bc0cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 --2- 172.21.15.104:0/2947684749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009820 tx=0x7f57b0009b30 comp rx=0 tx=0).stop 2023-06-10T18:44:57.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 -- 172.21.15.104:0/2947684749 shutdown_connections 2023-06-10T18:44:57.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 --2- 172.21.15.104:0/2947684749 >> v2:172.21.15.114:3300/0 conn(0x7f57bc06dcd0 0x7f57bc06e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 --2- 172.21.15.104:0/2947684749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc15c500 0x7f57bc0cecc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 -- 172.21.15.104:0/2947684749 >> 172.21.15.104:0/2947684749 conn(0x7f57bc06b8a0 msgr2=0x7f57bc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:57.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 -- 172.21.15.104:0/2947684749 shutdown_connections 2023-06-10T18:44:57.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.325+0000 7f57c093f700 1 -- 172.21.15.104:0/2947684749 wait complete. 2023-06-10T18:44:57.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.326+0000 7f57c093f700 1 Processor -- start 2023-06-10T18:44:57.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.326+0000 7f57c093f700 1 -- start start 2023-06-10T18:44:57.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc06dcd0 0x7f57bc0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57bc0cf200 0x7f57bc0c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc09d470 con 0x7f57bc15c500 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc09c150 con 0x7f57bc0cf200 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57c093f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc09c3d0 con 0x7f57bc06dcd0 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55694/0 (socket says 172.21.15.104:55694) 2023-06-10T18:44:57.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 -- 172.21.15.104:0/3210928038 learned_addr learned my addr 172.21.15.104:0/3210928038 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:57.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57b9d9b700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57bc0cf200 0x7f57bc0c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57ba59c700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc06dcd0 0x7f57bc0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 -- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc06dcd0 msgr2=0x7f57bc0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc06dcd0 0x7f57bc0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 -- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57bc0cf200 msgr2=0x7f57bc0c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57bc0cf200 0x7f57bc0c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 -- 172.21.15.104:0/3210928038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0009430 con 0x7f57bc15c500 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.327+0000 7f57bad9d700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f57b400efc0 tx=0x7f57b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.328+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4003c20 con 0x7f57bc15c500 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.328+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b400f040 con 0x7f57bc15c500 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.328+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc09c6b0 con 0x7f57bc15c500 2023-06-10T18:44:57.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.328+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4009e90 con 0x7f57bc15c500 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.328+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc09cc70 con 0x7f57bc15c500 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.329+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc074050 con 0x7f57bc15c500 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.329+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f57b400c930 con 0x7f57bc15c500 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.329+0000 7f57aaffd700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 0x7f579c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.329+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f57b4089a60 con 0x7f57bc15c500 2023-06-10T18:44:57.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.330+0000 7f57ba59c700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 0x7f579c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.330+0000 7f57ba59c700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 0x7f579c06d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57b0000c00 tx=0x7f57b0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.334+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f57b4054870 con 0x7f57bc15c500 2023-06-10T18:44:57.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.521+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57bc0ad750 con 0x7f57bc15c500 2023-06-10T18:44:57.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.521+0000 7f57aaffd700 1 -- 172.21.15.104:0/3210928038 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57b4057e90 con 0x7f57bc15c500 2023-06-10T18:44:57.527 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:57.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 msgr2=0x7f579c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 0x7f579c06d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57b0000c00 tx=0x7f57b0008070 comp rx=0 tx=0).stop 2023-06-10T18:44:57.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 msgr2=0x7f57bc14d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f57b400efc0 tx=0x7f57b400c3a0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 shutdown_connections 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57bc15c500 0x7f57bc14d680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57bc06dcd0 0x7f57bc0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579c06ace0 0x7f579c06d190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 --2- 172.21.15.104:0/3210928038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57bc0cf200 0x7f57bc0c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 >> 172.21.15.104:0/3210928038 conn(0x7f57bc06b8a0 msgr2=0x7f57bc0af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:57.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 shutdown_connections 2023-06-10T18:44:57.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.523+0000 7f57c093f700 1 -- 172.21.15.104:0/3210928038 wait complete. 2023-06-10T18:44:57.541 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:57.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.659+0000 7f5fb704f700 1 Processor -- start 2023-06-10T18:44:57.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.659+0000 7f5fb704f700 1 -- start start 2023-06-10T18:44:57.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb704f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fb015a330 0x7f5fb00ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb704f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb00b1780 0x7f5fb006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb704f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb015a700 con 0x7f5fb00cb420 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb704f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb006d7b0 con 0x7f5fb015a330 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb704f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00cbe00 con 0x7f5fb00b1780 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fb015a330 0x7f5fb00ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5fb015a330 0x7f5fb00ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55698/0 (socket says 172.21.15.104:55698) 2023-06-10T18:44:57.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5faffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5fb00cb420 0x7f5fb00b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53530/0 (socket says 172.21.15.104:53530) 2023-06-10T18:44:57.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 -- 172.21.15.104:0/3644665290 learned_addr learned my addr 172.21.15.104:0/3644665290 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:57.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb55ec700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb00b1780 0x7f5fb006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 -- 172.21.15.104:0/3644665290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb00b1780 msgr2=0x7f5fb006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:57.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb00b1780 0x7f5fb006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 -- 172.21.15.104:0/3644665290 >> v1:172.21.15.104:6789/0 conn(0x7f5fb00cb420 legacy=0x7f5fb00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 -- 172.21.15.104:0/3644665290 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb006ce80 con 0x7f5fb015a330 2023-06-10T18:44:57.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.660+0000 7f5fb4deb700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb015a330 0x7f5fb00ac1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f75c8af3dbfee48 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5faf7fe700 1 -- 172.21.15.104:0/3644665290 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4005e30 con 0x7f5fb015a330 2023-06-10T18:44:57.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5faf7fe700 1 -- 172.21.15.104:0/3644665290 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa4004030 con 0x7f5fb015a330 2023-06-10T18:44:57.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5faf7fe700 1 -- 172.21.15.104:0/3644665290 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40043e0 con 0x7f5fb015a330 2023-06-10T18:44:57.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 -- 172.21.15.104:0/3644665290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb015a330 msgr2=0x7f5fb00ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb015a330 0x7f5fb00ac1c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 -- 172.21.15.104:0/3644665290 shutdown_connections 2023-06-10T18:44:57.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb015a330 0x7f5fb00ac1c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 --2- 172.21.15.104:0/3644665290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb00b1780 0x7f5fb006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 -- 172.21.15.104:0/3644665290 >> 172.21.15.104:0/3644665290 conn(0x7f5fb006b8a0 msgr2=0x7f5fb00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:57.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 -- 172.21.15.104:0/3644665290 shutdown_connections 2023-06-10T18:44:57.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.661+0000 7f5fb704f700 1 -- 172.21.15.104:0/3644665290 wait complete. 2023-06-10T18:44:57.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.662+0000 7f5fb704f700 1 Processor -- start 2023-06-10T18:44:57.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.662+0000 7f5fb704f700 1 -- start start 2023-06-10T18:44:57.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fb00cb420 0x7f5fb00c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb015a330 0x7f5fb00cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c3fe0 con 0x7f5fb00b1780 2023-06-10T18:44:57.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c41d0 con 0x7f5fb00cb420 2023-06-10T18:44:57.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb704f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c44b0 con 0x7f5fb015a330 2023-06-10T18:44:57.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb4deb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb4deb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55704/0 (socket says 172.21.15.104:55704) 2023-06-10T18:44:57.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.663+0000 7f5fb4deb700 1 -- 172.21.15.104:0/2598180939 learned_addr learned my addr 172.21.15.104:0/2598180939 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:57.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 -- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb015a330 msgr2=0x7f5fb00cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb015a330 0x7f5fb00cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 -- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fb00cb420 msgr2=0x7f5fb00c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fb00cb420 0x7f5fb00c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 -- 172.21.15.104:0/2598180939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa4009500 con 0x7f5fb00b1780 2023-06-10T18:44:57.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb4deb700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009c10 tx=0x7f5fa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4003e30 con 0x7f5fb00b1780 2023-06-10T18:44:57.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa4011420 con 0x7f5fb00b1780 2023-06-10T18:44:57.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40117d0 con 0x7f5fb00b1780 2023-06-10T18:44:57.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00c4790 con 0x7f5fb00b1780 2023-06-10T18:44:57.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.664+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb00cc6f0 con 0x7f5fb00b1780 2023-06-10T18:44:57.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.665+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb015c1c0 con 0x7f5fb00b1780 2023-06-10T18:44:57.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.665+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5fa4011930 con 0x7f5fb00b1780 2023-06-10T18:44:57.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.666+0000 7f5fad7fa700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 0x7f5f9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.666+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5fa408ab90 con 0x7f5fb00b1780 2023-06-10T18:44:57.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.666+0000 7f5faffff700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 0x7f5f9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.666+0000 7f5faffff700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 0x7f5f9406d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c001660 tx=0x7f5f9c007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:57.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.670+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5fb015c1c0 con 0x7f5fb00b1780 2023-06-10T18:44:57.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.857+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5fb015c1c0 con 0x7f5fb00b1780 2023-06-10T18:44:57.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.857+0000 7f5fad7fa700 1 -- 172.21.15.104:0/2598180939 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5fb015c1c0 con 0x7f5fb00b1780 2023-06-10T18:44:57.863 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:57.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 msgr2=0x7f5f9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 0x7f5f9406d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c001660 tx=0x7f5f9c007a50 comp rx=0 tx=0).stop 2023-06-10T18:44:57.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 msgr2=0x7f5fb00c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:57.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009c10 tx=0x7f5fa4004750 comp rx=0 tx=0).stop 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 shutdown_connections 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5fb015a330 0x7f5fb00cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5fb00b1780 0x7f5fb00c5a80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f9406ace0 0x7f5f9406d190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 --2- 172.21.15.104:0/2598180939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fb00cb420 0x7f5fb00c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.859+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 >> 172.21.15.104:0/2598180939 conn(0x7f5fb006b8a0 msgr2=0x7f5fb00b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.860+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 shutdown_connections 2023-06-10T18:44:57.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.860+0000 7f5fb704f700 1 -- 172.21.15.104:0/2598180939 wait complete. 2023-06-10T18:44:57.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:44:57.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.990+0000 7fc66b8a4700 1 Processor -- start 2023-06-10T18:44:57.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 -- start start 2023-06-10T18:44:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc66406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc66406e4b0 0x7fc6640c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66406d290 con 0x7fc6640ca100 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc664071640 con 0x7fc66406e4b0 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.991+0000 7fc66b8a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640aef00 con 0x7fc66406e0e0 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc669e41700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc6640ca100 0x7fc6640ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53544/0 (socket says 172.21.15.104:53544) 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc669e41700 1 -- 172.21.15.104:0/3182218213 learned_addr learned my addr 172.21.15.104:0/3182218213 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 --2- 172.21.15.104:0/3182218213 >> v2:172.21.15.114:3300/0 conn(0x7fc66406e4b0 0x7fc6640c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc669640700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc66406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252140714 0 0) 0x7fc66406d290 con 0x7fc6640ca100 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6500035b0 con 0x7fc6640ca100 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851128260 0 0) 0x7fc6500035b0 con 0x7fc6640ca100 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 -- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 msgr2=0x7fc66406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc66406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 -- 172.21.15.104:0/3182218213 >> v1:172.21.15.104:6789/0 conn(0x7fc6640ca100 legacy=0x7fc6640ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 -- 172.21.15.104:0/3182218213 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66406d4c0 con 0x7fc66406e4b0 2023-06-10T18:44:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc669640700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc66406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.992+0000 7fc668e3f700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6640c9bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc660009840 tx=0x7fc660009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aaad05cfe012512 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0019f0 con 0x7fc66406e4b0 2023-06-10T18:44:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc660004030 con 0x7fc66406e4b0 2023-06-10T18:44:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc65bfff700 1 -- 172.21.15.104:0/3182218213 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6600043c0 con 0x7fc66406e4b0 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 -- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 msgr2=0x7fc6640c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6640c9bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc660009840 tx=0x7fc660009b50 comp rx=0 tx=0).stop 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 -- 172.21.15.104:0/3182218213 shutdown_connections 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc66406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/3182218213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6640c9bc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 -- 172.21.15.104:0/3182218213 >> 172.21.15.104:0/3182218213 conn(0x7fc66406b8a0 msgr2=0x7fc6640cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 -- 172.21.15.104:0/3182218213 shutdown_connections 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.993+0000 7fc66b8a4700 1 -- 172.21.15.104:0/3182218213 wait complete. 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.994+0000 7fc66b8a4700 1 Processor -- start 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.994+0000 7fc66b8a4700 1 -- start start 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc6640b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6641478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640aff30 con 0x7fc6640ca100 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640b0210 con 0x7fc66406e4b0 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc66b8a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640b04f0 con 0x7fc66406e0e0 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55722/0 (socket says 172.21.15.104:55722) 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 -- 172.21.15.104:0/83929512 learned_addr learned my addr 172.21.15.104:0/83929512 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc668e3f700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6641478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669640700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc6640b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 -- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 msgr2=0x7fc6640b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc6640b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.995+0000 7fc669e41700 1 -- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 msgr2=0x7fc6641478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc669e41700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6641478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc669e41700 1 -- 172.21.15.104:0/83929512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc654009430 con 0x7fc6640ca100 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc669640700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc6640b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc669e41700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00cca0 tx=0x7fc65c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c00cea0 con 0x7fc6640ca100 2023-06-10T18:44:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600094a0 con 0x7fc6640ca100 2023-06-10T18:44:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c00ae00 con 0x7fc6640ca100 2023-06-10T18:44:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c007310 con 0x7fc6640ca100 2023-06-10T18:44:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.996+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6640b0ab0 con 0x7fc6640ca100 2023-06-10T18:44:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.997+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc664074050 con 0x7fc6640ca100 2023-06-10T18:44:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.997+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc65c007470 con 0x7fc6640ca100 2023-06-10T18:44:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.998+0000 7fc659ffb700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 0x7fc64806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.998+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc65c089400 con 0x7fc6640ca100 2023-06-10T18:44:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.998+0000 7fc669640700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 0x7fc64806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:57.998+0000 7fc669640700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 0x7fc64806d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc654009c30 tx=0x7fc654008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.002+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc65c054190 con 0x7fc6640ca100 2023-06-10T18:44:58.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.189+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6640723b0 con 0x7fc6640ca100 2023-06-10T18:44:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.190+0000 7fc659ffb700 1 -- 172.21.15.104:0/83929512 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7fc65c0577b0 con 0x7fc6640ca100 2023-06-10T18:44:58.196 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:44:58.196 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:44:58.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 msgr2=0x7fc64806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 0x7fc64806d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc654009c30 tx=0x7fc654008040 comp rx=0 tx=0).stop 2023-06-10T18:44:58.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 msgr2=0x7fc6640c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00cca0 tx=0x7fc65c00a340 comp rx=0 tx=0).stop 2023-06-10T18:44:58.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 shutdown_connections 2023-06-10T18:44:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6640ca100 0x7fc6640c8e30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc66406e0e0 0x7fc6640b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc64806ace0 0x7fc64806d190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 --2- 172.21.15.104:0/83929512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc66406e4b0 0x7fc6641478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 >> 172.21.15.104:0/83929512 conn(0x7fc66406b8a0 msgr2=0x7fc6640ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 shutdown_connections 2023-06-10T18:44:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.192+0000 7fc66b8a4700 1 -- 172.21.15.104:0/83929512 wait complete. 2023-06-10T18:44:58.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:44:58.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:44:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.329+0000 7f033ca67700 1 Processor -- start 2023-06-10T18:44:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.329+0000 7f033ca67700 1 -- start start 2023-06-10T18:44:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f033806e6e0 0x7f03380b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 0x7f0338069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033806e250 con 0x7f033806e6e0 2023-06-10T18:44:58.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033815c950 con 0x7f033806cde0 2023-06-10T18:44:58.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033ca67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03380b1e80 con 0x7f033815c580 2023-06-10T18:44:58.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f033806e6e0 0x7f03380b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f033806e6e0 0x7f03380b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55726/0 (socket says 172.21.15.104:55726) 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 -- 172.21.15.104:0/2530245660 learned_addr learned my addr 172.21.15.104:0/2530245660 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f033659c700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0336d9d700 1 --2- 172.21.15.104:0/2530245660 >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 0x7f0338069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 -- 172.21.15.104:0/2530245660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 msgr2=0x7f03380b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 -- 172.21.15.104:0/2530245660 >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 msgr2=0x7f0338069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:58.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2530245660 >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 0x7f0338069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 -- 172.21.15.104:0/2530245660 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033806d920 con 0x7f033806e6e0 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0336d9d700 1 --2- 172.21.15.104:0/2530245660 >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 0x7f0338069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.330+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f03380b15b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f03380701f0 tx=0x7f0328009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b189af7fc69dfe4d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033559a700 1 -- 172.21.15.104:0/2530245660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800b870 con 0x7f033806e6e0 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033559a700 1 -- 172.21.15.104:0/2530245660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032800b9d0 con 0x7f033806e6e0 2023-06-10T18:44:58.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033559a700 1 -- 172.21.15.104:0/2530245660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800bd60 con 0x7f033806e6e0 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 -- 172.21.15.104:0/2530245660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 msgr2=0x7f03380b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f03380b15b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f03380701f0 tx=0x7f0328009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 -- 172.21.15.104:0/2530245660 shutdown_connections 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 --2- 172.21.15.104:0/2530245660 >> v2:172.21.15.114:3300/0 conn(0x7f033806cde0 0x7f0338069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 --2- 172.21.15.104:0/2530245660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f03380b15b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 -- 172.21.15.104:0/2530245660 >> 172.21.15.104:0/2530245660 conn(0x7f033806b8a0 msgr2=0x7f03380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 -- 172.21.15.104:0/2530245660 shutdown_connections 2023-06-10T18:44:58.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.331+0000 7f033ca67700 1 -- 172.21.15.104:0/2530245660 wait complete. 2023-06-10T18:44:58.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 Processor -- start 2023-06-10T18:44:58.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 -- start start 2023-06-10T18:44:58.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 0x7f0338157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033815cb30 con 0x7f033806e6e0 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033815ce10 con 0x7f033806cde0 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f033ca67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033815d0f0 con 0x7f033815c580 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f0335d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f0335d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55730/0 (socket says 172.21.15.104:55730) 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.333+0000 7f0335d9b700 1 -- 172.21.15.104:0/2075783014 learned_addr learned my addr 172.21.15.104:0/2075783014 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f033659c700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 0x7f0338157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0336d9d700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 -- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 msgr2=0x7f03380b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 -- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 msgr2=0x7f0338157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 0x7f0338157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 -- 172.21.15.104:0/2075783014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328009550 con 0x7f033806e6e0 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f033659c700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 0x7f0338157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0336d9d700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:58.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0335d9b700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0328000c00 tx=0x7f0328004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328004540 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03280036a0 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033815d3d0 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328003a50 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.334+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03380b0500 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.335+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033815d610 con 0x7f033806e6e0 2023-06-10T18:44:58.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.335+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0328021070 con 0x7f033806e6e0 2023-06-10T18:44:58.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.336+0000 7f0326ffd700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 0x7f031806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.336+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f032808ae80 con 0x7f033806e6e0 2023-06-10T18:44:58.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.336+0000 7f033659c700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 0x7f031806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.336+0000 7f033659c700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 0x7f031806d190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f032c009a00 tx=0x7f032c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.340+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f033815d610 con 0x7f033806e6e0 2023-06-10T18:44:58.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.529+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f033815d610 con 0x7f033806e6e0 2023-06-10T18:44:58.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.529+0000 7f0326ffd700 1 -- 172.21.15.104:0/2075783014 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f033815d610 con 0x7f033806e6e0 2023-06-10T18:44:58.535 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 msgr2=0x7f031806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 0x7f031806d190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f032c009a00 tx=0x7f032c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 msgr2=0x7f033815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0328000c00 tx=0x7f0328004750 comp rx=0 tx=0).stop 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 shutdown_connections 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f031806ace0 0x7f031806d190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033806cde0 0x7f0338157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033815c580 0x7f03380b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 --2- 172.21.15.104:0/2075783014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f033806e6e0 0x7f033815d8f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 >> 172.21.15.104:0/2075783014 conn(0x7f033806b8a0 msgr2=0x7f03380ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 shutdown_connections 2023-06-10T18:44:58.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.531+0000 7f033ca67700 1 -- 172.21.15.104:0/2075783014 wait complete. 2023-06-10T18:44:58.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:44:58.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.665+0000 7f156ae9e700 1 Processor -- start 2023-06-10T18:44:58.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.665+0000 7f156ae9e700 1 -- start start 2023-06-10T18:44:58.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156ae9e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f156415a0a0 0x7f156415a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156ae9e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 0x7f1564069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156ae9e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640cc050 con 0x7f156406d350 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156ae9e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640ac9d0 con 0x7f156415a0a0 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156ae9e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156406d0e0 con 0x7f156406e3d0 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f1568c3a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f156415a0a0 0x7f156415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f156943b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 0x7f1564069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f1568c3a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f156415a0a0 0x7f156415a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:43238/0 (socket says 172.21.15.104:43238) 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f1568c3a700 1 -- 172.21.15.104:0/445501573 learned_addr learned my addr 172.21.15.104:0/445501573 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.666+0000 7f1563fff700 1 --1- 172.21.15.104:0/445501573 >> v1:172.21.15.114:6789/0 conn(0x7f156406d350 0x7f15640ac2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59410/0 (socket says 172.21.15.104:59410) 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f1568c3a700 1 -- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 msgr2=0x7f1564069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f1568c3a700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 0x7f1564069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f1568c3a700 1 -- 172.21.15.104:0/445501573 >> v1:172.21.15.114:6789/0 conn(0x7f156406d350 legacy=0x7f15640ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f15637fe700 1 -- 172.21.15.104:0/445501573 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016825184 0 0) 0x7f15640cc050 con 0x7f156406d350 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f1568c3a700 1 -- 172.21.15.104:0/445501573 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156406ce00 con 0x7f156415a0a0 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f156943b700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 0x7f1564069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f1568c3a700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156415a0a0 0x7f156415a470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f3665633bc598e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f15637fe700 1 -- 172.21.15.104:0/445501573 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558005e30 con 0x7f156415a0a0 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f15637fe700 1 -- 172.21.15.104:0/445501573 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1558004030 con 0x7f156415a0a0 2023-06-10T18:44:58.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.667+0000 7f15637fe700 1 -- 172.21.15.104:0/445501573 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15580043e0 con 0x7f156415a0a0 2023-06-10T18:44:58.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 -- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156415a0a0 msgr2=0x7f156415a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156415a0a0 0x7f156415a470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 -- 172.21.15.104:0/445501573 shutdown_connections 2023-06-10T18:44:58.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156406e3d0 0x7f1564069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 --2- 172.21.15.104:0/445501573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156415a0a0 0x7f156415a470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 -- 172.21.15.104:0/445501573 >> 172.21.15.104:0/445501573 conn(0x7f156406b8a0 msgr2=0x7f15640cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 -- 172.21.15.104:0/445501573 shutdown_connections 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.668+0000 7f156ae9e700 1 -- 172.21.15.104:0/445501573 wait complete. 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.669+0000 7f156ae9e700 1 Processor -- start 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.669+0000 7f156ae9e700 1 -- start start 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156406e3d0 0x7f15640c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 0x7f15640b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640cde50 con 0x7f156406d350 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b44d0 con 0x7f156406e3d0 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156ae9e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15640c4640 con 0x7f156415a0a0 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55740/0 (socket says 172.21.15.104:55740) 2023-06-10T18:44:58.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 -- 172.21.15.104:0/3923522433 learned_addr learned my addr 172.21.15.104:0/3923522433 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:58.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1563fff700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156406e3d0 0x7f15640c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156943b700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 0x7f15640b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 -- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 msgr2=0x7f15640b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 0x7f15640b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 -- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156406e3d0 msgr2=0x7f15640c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156406e3d0 0x7f15640c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 -- 172.21.15.104:0/3923522433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c009a70 con 0x7f156406d350 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f156943b700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 0x7f15640b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.670+0000 7f1568c3a700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1558009c10 tx=0x7f1558004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.671+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15580045d0 con 0x7f156406d350 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.671+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155801d430 con 0x7f156406d350 2023-06-10T18:44:58.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.671+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1558009500 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.671+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155801d7c0 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.671+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15640cdae0 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.672+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15640cbf20 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.672+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f155801d920 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.672+0000 7f15617fa700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 0x7f154806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.673+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f155808ab30 con 0x7f156406d350 2023-06-10T18:44:58.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.673+0000 7f1563fff700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 0x7f154806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:58.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.673+0000 7f1563fff700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 0x7f154806d190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f15641f55a0 tx=0x7f1550007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:58.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.677+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1558055a60 con 0x7f156406d350 2023-06-10T18:44:58.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.861+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f15640c2cc0 con 0x7f156406d350 2023-06-10T18:44:58.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.862+0000 7f15617fa700 1 -- 172.21.15.104:0/3923522433 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1558059080 con 0x7f156406d350 2023-06-10T18:44:58.867 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:44:58.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 msgr2=0x7f154806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 0x7f154806d190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f15641f55a0 tx=0x7f1550007de0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 msgr2=0x7f15640b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1558009c10 tx=0x7f1558004750 comp rx=0 tx=0).stop 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 shutdown_connections 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f156415a0a0 0x7f15640b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f156406d350 0x7f15640b3f90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f154806ace0 0x7f154806d190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 --2- 172.21.15.104:0/3923522433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f156406e3d0 0x7f15640c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:58.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 >> 172.21.15.104:0/3923522433 conn(0x7f156406b8a0 msgr2=0x7f1564158fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:58.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 shutdown_connections 2023-06-10T18:44:58.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.864+0000 7f156ae9e700 1 -- 172.21.15.104:0/3923522433 wait complete. 2023-06-10T18:44:58.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:44:59.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.998+0000 7f0693a86700 1 Processor -- start 2023-06-10T18:44:59.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.998+0000 7f0693a86700 1 -- start start 2023-06-10T18:44:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0693a86700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f068c06ea70 0x7f068c06d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0693a86700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c0ae7e0 con 0x7f068c0b1780 2023-06-10T18:44:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0693a86700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c06df90 con 0x7f068c15c6f0 2023-06-10T18:44:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0693a86700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068c06e270 con 0x7f068c06ea70 2023-06-10T18:44:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0691021700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f068c0b1780 0x7f068c06cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53576/0 (socket says 172.21.15.104:53576) 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0691021700 1 -- 172.21.15.104:0/2177772332 learned_addr learned my addr 172.21.15.104:0/2177772332 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0692023700 1 --2- 172.21.15.104:0/2177772332 >> v2:172.21.15.104:3300/0 conn(0x7f068c06ea70 0x7f068c06d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0692023700 1 -- 172.21.15.104:0/2177772332 >> v1:172.21.15.114:6789/0 conn(0x7f068c15c6f0 legacy=0x7f068c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0692023700 1 -- 172.21.15.104:0/2177772332 >> v1:172.21.15.104:6789/0 conn(0x7f068c0b1780 legacy=0x7f068c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0692023700 1 -- 172.21.15.104:0/2177772332 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c06e550 con 0x7f068c06ea70 2023-06-10T18:44:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0692023700 1 --2- 172.21.15.104:0/2177772332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c06ea70 0x7f068c06d4f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfeef4e6e268658d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:58.999+0000 7f0690820700 1 -- 172.21.15.104:0/2177772332 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400e070 con 0x7f068c06ea70 2023-06-10T18:44:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0690820700 1 -- 172.21.15.104:0/2177772332 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068400bb90 con 0x7f068c06ea70 2023-06-10T18:44:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0690820700 1 -- 172.21.15.104:0/2177772332 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684003a20 con 0x7f068c06ea70 2023-06-10T18:44:59.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 -- 172.21.15.104:0/2177772332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c06ea70 msgr2=0x7f068c06d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 --2- 172.21.15.104:0/2177772332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c06ea70 0x7f068c06d4f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).stop 2023-06-10T18:44:59.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 -- 172.21.15.104:0/2177772332 shutdown_connections 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 --2- 172.21.15.104:0/2177772332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c06ea70 0x7f068c06d4f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 -- 172.21.15.104:0/2177772332 >> 172.21.15.104:0/2177772332 conn(0x7f068c06b8a0 msgr2=0x7f068c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 -- 172.21.15.104:0/2177772332 shutdown_connections 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.000+0000 7f0693a86700 1 -- 172.21.15.104:0/2177772332 wait complete. 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.001+0000 7f0693a86700 1 Processor -- start 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.001+0000 7f0693a86700 1 -- start start 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 0x7f068c1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f068c15c6f0 0x7f068c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0c6380 con 0x7f068c0b1780 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0b5280 con 0x7f068c15c6f0 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0693a86700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0c5700 con 0x7f068c06ea70 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0691021700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0691021700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55754/0 (socket says 172.21.15.104:55754) 2023-06-10T18:44:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0691021700 1 -- 172.21.15.104:0/94437073 learned_addr learned my addr 172.21.15.104:0/94437073 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0692023700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f068c15c6f0 0x7f068c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.002+0000 7f0691822700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 0x7f068c1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 -- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 msgr2=0x7f068c1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 0x7f068c1455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 -- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f068c15c6f0 msgr2=0x7f068c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f068c15c6f0 0x7f068c069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 -- 172.21.15.104:0/94437073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684009a70 con 0x7f068c0b1780 2023-06-10T18:44:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691822700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 0x7f068c1455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0691021700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f068800c530 tx=0x7f068800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688005d70 con 0x7f068c0b1780 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06880070d0 con 0x7f068c0b1780 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c0c59e0 con 0x7f068c0b1780 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688007460 con 0x7f068c0b1780 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.003+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c0c5fa0 con 0x7f068c0b1780 2023-06-10T18:44:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.004+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c074050 con 0x7f068c0b1780 2023-06-10T18:44:59.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.004+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f06880075c0 con 0x7f068c0b1780 2023-06-10T18:44:59.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.005+0000 7f06827fc700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 0x7f067006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.005+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0688088bf0 con 0x7f068c0b1780 2023-06-10T18:44:59.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.005+0000 7f0691822700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 0x7f067006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.005+0000 7f0691822700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 0x7f067006d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f067c00a8b0 tx=0x7f067c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.009+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0688053a00 con 0x7f068c0b1780 2023-06-10T18:44:59.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.196+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f068c0d0030 con 0x7f068c0b1780 2023-06-10T18:44:59.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.196+0000 7f06827fc700 1 -- 172.21.15.104:0/94437073 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0688017020 con 0x7f068c0b1780 2023-06-10T18:44:59.202 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:59.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 msgr2=0x7f067006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 0x7f067006d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f067c00a8b0 tx=0x7f067c008040 comp rx=0 tx=0).stop 2023-06-10T18:44:59.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 msgr2=0x7f068c0c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f068800c530 tx=0x7f068800c8f0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 shutdown_connections 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f067006ace0 0x7f067006d190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f068c15c6f0 0x7f068c069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f068c06ea70 0x7f068c1455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 --2- 172.21.15.104:0/94437073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0b1780 0x7f068c0c4d00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 >> 172.21.15.104:0/94437073 conn(0x7f068c06b8a0 msgr2=0x7f068c0b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 shutdown_connections 2023-06-10T18:44:59.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.198+0000 7f0693a86700 1 -- 172.21.15.104:0/94437073 wait complete. 2023-06-10T18:44:59.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:44:59.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.332+0000 7fac59383700 1 Processor -- start 2023-06-10T18:44:59.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.332+0000 7fac59383700 1 -- start start 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac59383700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac540d04f0 0x7fac540d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac59383700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540af530 con 0x7fac540d25c0 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac59383700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540af810 con 0x7fac540b05d0 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac59383700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540afaf0 con 0x7fac540d04f0 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac540d04f0 0x7fac540d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac540d04f0 0x7fac540d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55760/0 (socket says 172.21.15.104:55760) 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 -- 172.21.15.104:0/858804849 learned_addr learned my addr 172.21.15.104:0/858804849 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac527fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fac540d25c0 0x7fac540d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53588/0 (socket says 172.21.15.104:53588) 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 -- 172.21.15.104:0/858804849 >> v1:172.21.15.114:6789/0 conn(0x7fac540b05d0 legacy=0x7fac540b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 -- 172.21.15.104:0/858804849 >> v1:172.21.15.104:6789/0 conn(0x7fac540d25c0 legacy=0x7fac540d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 -- 172.21.15.104:0/858804849 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540afdd0 con 0x7fac540d04f0 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac52ffd700 1 --2- 172.21.15.104:0/858804849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac540d34d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45471ca86a398e38 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac51ffb700 1 -- 172.21.15.104:0/858804849 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44005e30 con 0x7fac540d04f0 2023-06-10T18:44:59.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac51ffb700 1 -- 172.21.15.104:0/858804849 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44004030 con 0x7fac540d04f0 2023-06-10T18:44:59.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.333+0000 7fac51ffb700 1 -- 172.21.15.104:0/858804849 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440043e0 con 0x7fac540d04f0 2023-06-10T18:44:59.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 -- 172.21.15.104:0/858804849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 msgr2=0x7fac540d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 --2- 172.21.15.104:0/858804849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac540d34d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44009bd0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 -- 172.21.15.104:0/858804849 shutdown_connections 2023-06-10T18:44:59.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 --2- 172.21.15.104:0/858804849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac540d34d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 -- 172.21.15.104:0/858804849 >> 172.21.15.104:0/858804849 conn(0x7fac54071ec0 msgr2=0x7fac5409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 -- 172.21.15.104:0/858804849 shutdown_connections 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.334+0000 7fac59383700 1 -- 172.21.15.104:0/858804849 wait complete. 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.335+0000 7fac59383700 1 Processor -- start 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.335+0000 7fac59383700 1 -- start start 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 0x7fac54159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac540d25c0 0x7fac54142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54158780 con 0x7fac540d04f0 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54158a60 con 0x7fac540b05d0 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac59383700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac54158d40 con 0x7fac540d25c0 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55770/0 (socket says 172.21.15.104:55770) 2023-06-10T18:44:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 -- 172.21.15.104:0/3047631095 learned_addr learned my addr 172.21.15.104:0/3047631095 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac52ffd700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 0x7fac54159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac537fe700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac540d25c0 0x7fac54142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 -- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac540d25c0 msgr2=0x7fac54142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac540d25c0 0x7fac54142820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 -- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 msgr2=0x7fac54159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 0x7fac54159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac527fc700 1 -- 172.21.15.104:0/3047631095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44009500 con 0x7fac540d04f0 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.336+0000 7fac52ffd700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 0x7fac54159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac527fc700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c780 tx=0x7fac4800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48007560 con 0x7fac540d04f0 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac480076c0 con 0x7fac540d04f0 2023-06-10T18:44:59.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac54159020 con 0x7fac540d04f0 2023-06-10T18:44:59.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48004ab0 con 0x7fac540d04f0 2023-06-10T18:44:59.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.337+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac54148b50 con 0x7fac540d04f0 2023-06-10T18:44:59.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.338+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54159540 con 0x7fac540d04f0 2023-06-10T18:44:59.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.338+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac48004c10 con 0x7fac540d04f0 2023-06-10T18:44:59.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.339+0000 7fac3b7fe700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 0x7fac4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.339+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fac48088d30 con 0x7fac540d04f0 2023-06-10T18:44:59.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.339+0000 7fac52ffd700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 0x7fac4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.339+0000 7fac52ffd700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 0x7fac4006d190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac44014040 tx=0x7fac44008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.343+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac54159540 con 0x7fac540d04f0 2023-06-10T18:44:59.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.529+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac54159540 con 0x7fac540d04f0 2023-06-10T18:44:59.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.529+0000 7fac3b7fe700 1 -- 172.21.15.104:0/3047631095 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac54159540 con 0x7fac540d04f0 2023-06-10T18:44:59.535 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:44:59.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 msgr2=0x7fac4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 0x7fac4006d190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac44014040 tx=0x7fac44008070 comp rx=0 tx=0).stop 2023-06-10T18:44:59.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 msgr2=0x7fac541f11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c780 tx=0x7fac4800cb40 comp rx=0 tx=0).stop 2023-06-10T18:44:59.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 shutdown_connections 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac4006ace0 0x7fac4006d190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540b05d0 0x7fac54159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac540d25c0 0x7fac54142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 --2- 172.21.15.104:0/3047631095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac540d04f0 0x7fac541f11e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 >> 172.21.15.104:0/3047631095 conn(0x7fac54071ec0 msgr2=0x7fac540b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 shutdown_connections 2023-06-10T18:44:59.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.531+0000 7fac59383700 1 -- 172.21.15.104:0/3047631095 wait complete. 2023-06-10T18:44:59.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:44:59.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.665+0000 7f0645494700 1 Processor -- start 2023-06-10T18:44:59.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.665+0000 7f0645494700 1 -- start start 2023-06-10T18:44:59.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f0645494700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f06400d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f0645494700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06400af530 con 0x7f06400d25c0 2023-06-10T18:44:59.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f0645494700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06400af810 con 0x7f06400b05d0 2023-06-10T18:44:59.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f0645494700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400afaf0 con 0x7f06400d04f0 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f063e7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f06400d25c0 0x7f06400d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53602/0 (socket says 172.21.15.104:53602) 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f063e7fc700 1 -- 172.21.15.104:0/765103025 learned_addr learned my addr 172.21.15.104:0/765103025 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734630004 0 0) 0x7f06400af530 con 0x7f06400d25c0 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06280035b0 con 0x7f06400d25c0 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.666+0000 7f063effd700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f06400d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402896717 0 0) 0x7f06280035b0 con 0x7f06400d25c0 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 msgr2=0x7f06400d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:59.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f06400d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 >> v1:172.21.15.114:6789/0 conn(0x7f06400b05d0 legacy=0x7f06400b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06400afdd0 con 0x7f06400d25c0 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0630001670 con 0x7f06400d25c0 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 >> v1:172.21.15.104:6789/0 conn(0x7f06400d25c0 legacy=0x7f06400d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f0628003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0628004060 0x7f0628004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0628004d90 0x7f0628005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400afaf0 con 0x7f0628004d90 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062800bc20 con 0x7f0628004060 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062800bde0 con 0x7f06400d04f0 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0628004d90 0x7f0628005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063effd700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f0628003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063e7fc700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0628004060 0x7f0628004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 -- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 msgr2=0x7f0628003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400d04f0 0x7f0628003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 -- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0628004060 msgr2=0x7f0628004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0628004060 0x7f0628004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 -- 172.21.15.104:0/765103025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06280035b0 con 0x7f0628004d90 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063f7fe700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0628004d90 0x7f0628005200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f063800c240 tx=0x7f0628005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3f6742fa2c8aaa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638001df0 con 0x7f0628004d90 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063800cda0 con 0x7f0628004d90 2023-06-10T18:44:59.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.667+0000 7f063dffb700 1 -- 172.21.15.104:0/765103025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638003c40 con 0x7f0628004d90 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 -- 172.21.15.104:0/765103025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0628004d90 msgr2=0x7f0628005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 --2- 172.21.15.104:0/765103025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0628004d90 0x7f0628005200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f063800c240 tx=0x7f0628005880 comp rx=0 tx=0).stop 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f063f7fe700 1 -- 172.21.15.104:0/765103025 reap_dead start 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 -- 172.21.15.104:0/765103025 shutdown_connections 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 -- 172.21.15.104:0/765103025 >> 172.21.15.104:0/765103025 conn(0x7f0640071ec0 msgr2=0x7f064009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 -- 172.21.15.104:0/765103025 shutdown_connections 2023-06-10T18:44:59.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.668+0000 7f0645494700 1 -- 172.21.15.104:0/765103025 wait complete. 2023-06-10T18:44:59.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.669+0000 7f0645494700 1 Processor -- start 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.669+0000 7f0645494700 1 -- start start 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 0x7f06400b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 0x7f0640156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0640157110 con 0x7f06400b7e20 2023-06-10T18:44:59.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400b7990 con 0x7f0640156730 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f0645494700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0640159a80 con 0x7f06400c8dd0 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55788/0 (socket says 172.21.15.104:55788) 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 -- 172.21.15.104:0/3516013711 learned_addr learned my addr 172.21.15.104:0/3516013711 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063f7fe700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 0x7f0640156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063effd700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 0x7f06400b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 -- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 msgr2=0x7f06400b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 0x7f06400b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.670+0000 7f063e7fc700 1 -- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 msgr2=0x7f0640156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f063e7fc700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 0x7f0640156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f063e7fc700 1 -- 172.21.15.104:0/3516013711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06400b6d80 con 0x7f06400b7e20 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f063f7fe700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 0x7f0640156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:59.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f063effd700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 0x7f06400b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f063e7fc700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0630007290 tx=0x7f0630007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063000f8d0 con 0x7f06400b7e20 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063000fa70 con 0x7f06400b7e20 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06400b7060 con 0x7f06400b7e20 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063000fe20 con 0x7f06400b7e20 2023-06-10T18:44:59.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.671+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06400b7620 con 0x7f06400b7e20 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.672+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06400b7b20 con 0x7f06400b7e20 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.672+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f063001d460 con 0x7f06400b7e20 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.673+0000 7f06277fe700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 0x7f062c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.673+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0630089930 con 0x7f06400b7e20 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.673+0000 7f063effd700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 0x7f062c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:44:59.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.673+0000 7f063effd700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 0x7f062c06d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0634006fd0 tx=0x7f0634008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:44:59.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.677+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f06300546a0 con 0x7f06400b7e20 2023-06-10T18:44:59.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.865+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0640067010 con 0x7f06400b7e20 2023-06-10T18:44:59.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.866+0000 7f06277fe700 1 -- 172.21.15.104:0/3516013711 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0630057cc0 con 0x7f06400b7e20 2023-06-10T18:44:59.871 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:44:59.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 msgr2=0x7f062c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 0x7f062c06d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0634006fd0 tx=0x7f0634008040 comp rx=0 tx=0).stop 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 msgr2=0x7f0640159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0630007290 tx=0x7f0630007650 comp rx=0 tx=0).stop 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 shutdown_connections 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f062c06ace0 0x7f062c06d190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0640156730 0x7f0640156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06400c8dd0 0x7f06400b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 --2- 172.21.15.104:0/3516013711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06400b7e20 0x7f0640159540 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:44:59.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 >> 172.21.15.104:0/3516013711 conn(0x7f0640071ec0 msgr2=0x7f06400c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:44:59.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 shutdown_connections 2023-06-10T18:44:59.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.868+0000 7f0645494700 1 -- 172.21.15.104:0/3516013711 wait complete. 2023-06-10T18:44:59.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:44:59.999+0000 7f743f34d700 1 Processor -- start 2023-06-10T18:45:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 -- start start 2023-06-10T18:45:00.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f743815c7a0 0x7f74380cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74380701a0 0x7f74380705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7438070ae0 con 0x7f74381567f0 2023-06-10T18:45:00.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743815cca0 con 0x7f743815c7a0 2023-06-10T18:45:00.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743f34d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f743815cf80 con 0x7f74380701a0 2023-06-10T18:45:00.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.000+0000 7f743c8e8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f743815c7a0 0x7f74380cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f743815c7a0 0x7f74380cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55796/0 (socket says 172.21.15.104:55796) 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 -- 172.21.15.104:0/3440341436 learned_addr learned my addr 172.21.15.104:0/3440341436 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743d8ea700 1 --2- 172.21.15.104:0/3440341436 >> v2:172.21.15.114:3300/0 conn(0x7f74380701a0 0x7f74380705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 -- 172.21.15.104:0/3440341436 >> v2:172.21.15.114:3300/0 conn(0x7f74380701a0 msgr2=0x7f74380705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 --2- 172.21.15.104:0/3440341436 >> v2:172.21.15.114:3300/0 conn(0x7f74380701a0 0x7f74380705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 -- 172.21.15.104:0/3440341436 >> v1:172.21.15.104:6789/0 conn(0x7f74381567f0 legacy=0x7f74380cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 -- 172.21.15.104:0/3440341436 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743815d260 con 0x7f743815c7a0 2023-06-10T18:45:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f743c8e8700 1 --2- 172.21.15.104:0/3440341436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743815c7a0 0x7f74380cd560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f743815d820 tx=0x7f7434009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b102ee91ab17699 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f742ffff700 1 -- 172.21.15.104:0/3440341436 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434011830 con 0x7f743815c7a0 2023-06-10T18:45:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f742ffff700 1 -- 172.21.15.104:0/3440341436 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7434011990 con 0x7f743815c7a0 2023-06-10T18:45:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.001+0000 7f742ffff700 1 -- 172.21.15.104:0/3440341436 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434011d20 con 0x7f743815c7a0 2023-06-10T18:45:00.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 -- 172.21.15.104:0/3440341436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743815c7a0 msgr2=0x7f74380cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 --2- 172.21.15.104:0/3440341436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743815c7a0 0x7f74380cd560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f743815d820 tx=0x7f7434009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 -- 172.21.15.104:0/3440341436 shutdown_connections 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 --2- 172.21.15.104:0/3440341436 >> v2:172.21.15.114:3300/0 conn(0x7f74380701a0 0x7f74380705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 --2- 172.21.15.104:0/3440341436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f743815c7a0 0x7f74380cd560 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 -- 172.21.15.104:0/3440341436 >> 172.21.15.104:0/3440341436 conn(0x7f743806e370 msgr2=0x7f743806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 -- 172.21.15.104:0/3440341436 shutdown_connections 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.002+0000 7f743f34d700 1 -- 172.21.15.104:0/3440341436 wait complete. 2023-06-10T18:45:00.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.003+0000 7f743f34d700 1 Processor -- start 2023-06-10T18:45:00.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.003+0000 7f743f34d700 1 -- start start 2023-06-10T18:45:00.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74380701a0 0x7f74381598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743815c7a0 0x7f74380b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381594c0 con 0x7f74381567f0 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438158840 con 0x7f743815c7a0 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7438158b20 con 0x7f74380701a0 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55802/0 (socket says 172.21.15.104:55802) 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 -- 172.21.15.104:0/2076565802 learned_addr learned my addr 172.21.15.104:0/2076565802 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:00.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743d8ea700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743815c7a0 0x7f74380b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743d0e9700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74380701a0 0x7f74381598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 -- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74380701a0 msgr2=0x7f74381598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74380701a0 0x7f74381598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 -- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743815c7a0 msgr2=0x7f74380b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743815c7a0 0x7f74380b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 -- 172.21.15.104:0/2076565802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7434009550 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743c8e8700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7434009c10 tx=0x7f7434004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434004510 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743401e070 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7438158e00 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.004+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74340038e0 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.005+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743814bf30 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.005+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438074050 con 0x7f74381567f0 2023-06-10T18:45:00.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.006+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7434003a40 con 0x7f74381567f0 2023-06-10T18:45:00.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.006+0000 7f742dffb700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 0x7f741c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.006+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f743408a6f0 con 0x7f74381567f0 2023-06-10T18:45:00.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.007+0000 7f743d0e9700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 0x7f741c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.007+0000 7f743d0e9700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 0x7f741c06d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74280018a0 tx=0x7f7428009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.011+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7434055500 con 0x7f74381567f0 2023-06-10T18:45:00.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.200+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74380cf320 con 0x7f74381567f0 2023-06-10T18:45:00.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.202+0000 7f742dffb700 1 -- 172.21.15.104:0/2076565802 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f7434058b20 con 0x7f74381567f0 2023-06-10T18:45:00.207 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:00.208 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:00.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 msgr2=0x7f741c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 0x7f741c06d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74280018a0 tx=0x7f7428009040 comp rx=0 tx=0).stop 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 msgr2=0x7f743814b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7434009c10 tx=0x7f7434004750 comp rx=0 tx=0).stop 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 shutdown_connections 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f741c06ace0 0x7f741c06d190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743815c7a0 0x7f74380b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74380701a0 0x7f74381598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 --2- 172.21.15.104:0/2076565802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74381567f0 0x7f743814b9f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 >> 172.21.15.104:0/2076565802 conn(0x7f743806e370 msgr2=0x7f74380cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 shutdown_connections 2023-06-10T18:45:00.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.204+0000 7f743f34d700 1 -- 172.21.15.104:0/2076565802 wait complete. 2023-06-10T18:45:00.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:00.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:00.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.340+0000 7f8e54eab700 1 Processor -- start 2023-06-10T18:45:00.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.340+0000 7f8e54eab700 1 -- start start 2023-06-10T18:45:00.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e54eab700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8e500cd8f0 0x7f8e500ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e54eab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e500adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e54eab700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5015cb00 con 0x7f8e5015c730 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e54eab700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e500acd90 con 0x7f8e500cd8f0 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e54eab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500ac460 con 0x7f8e500ad760 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4e59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8e5015c730 0x7f8e500cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53636/0 (socket says 172.21.15.104:53636) 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4e59c700 1 -- 172.21.15.104:0/3143101335 learned_addr learned my addr 172.21.15.104:0/3143101335 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3143101335 >> v2:172.21.15.114:3300/0 conn(0x7f8e500cd8f0 0x7f8e500ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4ed9d700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e500adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318286356 0 0) 0x7f8e40001290 con 0x7f8e5015c730 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e380057c0 con 0x7f8e5015c730 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266736676 0 0) 0x7f8e380057c0 con 0x7f8e5015c730 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 msgr2=0x7f8e500adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e500adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 >> v2:172.21.15.114:3300/0 conn(0x7f8e500cd8f0 msgr2=0x7f8e500ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 --2- 172.21.15.104:0/3143101335 >> v2:172.21.15.114:3300/0 conn(0x7f8e500cd8f0 0x7f8e500ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e500ac740 con 0x7f8e5015c730 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8e40001980 con 0x7f8e5015c730 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 >> v1:172.21.15.104:6789/0 conn(0x7f8e5015c730 legacy=0x7f8e500cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.341+0000 7f8e4d59a700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e38005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e38006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e38006fa0 0x7f8e38007410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500acd90 con 0x7f8e38006190 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500ac460 con 0x7f8e38006fa0 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800dc50 con 0x7f8e500ad760 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e38006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4e59c700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e38005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4ed9d700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e38006fa0 0x7f8e38007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 -- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 msgr2=0x7f8e38005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500ad760 0x7f8e38005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 -- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e38006fa0 msgr2=0x7f8e38007410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e38006fa0 0x7f8e38007410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 -- 172.21.15.104:0/3143101335 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380057c0 con 0x7f8e38006190 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e38006a60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009b60 tx=0x7f8e38007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d64b8e0a9ba0ad9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400e660 con 0x7f8e38006190 2023-06-10T18:45:00.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4400e7c0 con 0x7f8e38006190 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.342+0000 7f8e4d59a700 1 -- 172.21.15.104:0/3143101335 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400eb30 con 0x7f8e38006190 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 -- 172.21.15.104:0/3143101335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 msgr2=0x7f8e38006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3143101335 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e38006a60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009b60 tx=0x7f8e38007a90 comp rx=0 tx=0).stop 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e4ed9d700 1 -- 172.21.15.104:0/3143101335 reap_dead start 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 -- 172.21.15.104:0/3143101335 shutdown_connections 2023-06-10T18:45:00.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 -- 172.21.15.104:0/3143101335 >> 172.21.15.104:0/3143101335 conn(0x7f8e5006b8a0 msgr2=0x7f8e500cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 -- 172.21.15.104:0/3143101335 shutdown_connections 2023-06-10T18:45:00.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.343+0000 7f8e54eab700 1 -- 172.21.15.104:0/3143101335 wait complete. 2023-06-10T18:45:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 Processor -- start 2023-06-10T18:45:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 -- start start 2023-06-10T18:45:00.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e500c78e0 0x7f8e500ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500c7d50 0x7f8e500ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500b0290 con 0x7f8e38006190 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5014c940 con 0x7f8e500c78e0 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.345+0000 7f8e54eab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e501f1900 con 0x7f8e500c7d50 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55820/0 (socket says 172.21.15.104:55820) 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 -- 172.21.15.104:0/3511300725 learned_addr learned my addr 172.21.15.104:0/3511300725 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e500c78e0 0x7f8e500ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 -- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500c7d50 msgr2=0x7f8e500ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500c7d50 0x7f8e500ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 -- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e500c78e0 msgr2=0x7f8e500ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e500c78e0 0x7f8e500ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 -- 172.21.15.104:0/3511300725 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5006f150 con 0x7f8e38006190 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e4e59c700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8e40007fd0 tx=0x7f8e40007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4000f6c0 con 0x7f8e38006190 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4000f820 con 0x7f8e38006190 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e500b0530 con 0x7f8e38006190 2023-06-10T18:45:00.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4000fbd0 con 0x7f8e38006190 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.346+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e500b0af0 con 0x7f8e38006190 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.347+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50074050 con 0x7f8e38006190 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.348+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e4000fd30 con 0x7f8e38006190 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.348+0000 7f8e3effd700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 0x7f8e3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.348+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8e40089860 con 0x7f8e38006190 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.348+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 0x7f8e3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.349+0000 7f8e4dd9b700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 0x7f8e3006d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009d610 tx=0x7f8e44008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.352+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e400545e0 con 0x7f8e38006190 2023-06-10T18:45:00.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.536+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e5015cb00 con 0x7f8e38006190 2023-06-10T18:45:00.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.537+0000 7f8e3effd700 1 -- 172.21.15.104:0/3511300725 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e5015cb00 con 0x7f8e38006190 2023-06-10T18:45:00.543 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.538+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 msgr2=0x7f8e3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.538+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 0x7f8e3006d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009d610 tx=0x7f8e44008030 comp rx=0 tx=0).stop 2023-06-10T18:45:00.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.538+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 msgr2=0x7f8e5014c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.538+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8e40007fd0 tx=0x7f8e40007600 comp rx=0 tx=0).stop 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 shutdown_connections 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e500c7d50 0x7f8e500ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e38006190 0x7f8e5014c400 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e3006ace0 0x7f8e3006d190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 --2- 172.21.15.104:0/3511300725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e500c78e0 0x7f8e500ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 >> 172.21.15.104:0/3511300725 conn(0x7f8e5006b8a0 msgr2=0x7f8e500cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 shutdown_connections 2023-06-10T18:45:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.539+0000 7f8e54eab700 1 -- 172.21.15.104:0/3511300725 wait complete. 2023-06-10T18:45:00.556 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:00.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.673+0000 7f1bca7ce700 1 Processor -- start 2023-06-10T18:45:00.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.673+0000 7f1bca7ce700 1 -- start start 2023-06-10T18:45:00.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.673+0000 7f1bca7ce700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1bc415c6f0 0x7f1bc40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bca7ce700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 0x7f1bc40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bca7ce700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40ae7e0 con 0x7f1bc406ea70 2023-06-10T18:45:00.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bca7ce700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc406df90 con 0x7f1bc415c6f0 2023-06-10T18:45:00.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bca7ce700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc406e270 con 0x7f1bc40aec00 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1bc415c6f0 0x7f1bc40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1bc415c6f0 0x7f1bc40b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55828/0 (socket says 172.21.15.104:55828) 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 -- 172.21.15.104:0/1990028836 learned_addr learned my addr 172.21.15.104:0/1990028836 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc8d6b700 1 --2- 172.21.15.104:0/1990028836 >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 0x7f1bc40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 -- 172.21.15.104:0/1990028836 >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 msgr2=0x7f1bc40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 --2- 172.21.15.104:0/1990028836 >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 0x7f1bc40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 -- 172.21.15.104:0/1990028836 >> v1:172.21.15.104:6789/0 conn(0x7f1bc406ea70 legacy=0x7f1bc40ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 -- 172.21.15.104:0/1990028836 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc406e550 con 0x7f1bc415c6f0 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc8d6b700 1 --2- 172.21.15.104:0/1990028836 >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 0x7f1bc40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc3fff700 1 --2- 172.21.15.104:0/1990028836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b1070 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1bc406ff70 tx=0x7f1bb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f908d09b7f55555 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc2ffd700 1 -- 172.21.15.104:0/1990028836 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4005e30 con 0x7f1bc415c6f0 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc2ffd700 1 -- 172.21.15.104:0/1990028836 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb4004030 con 0x7f1bc415c6f0 2023-06-10T18:45:00.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.674+0000 7f1bc2ffd700 1 -- 172.21.15.104:0/1990028836 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40043e0 con 0x7f1bc415c6f0 2023-06-10T18:45:00.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 -- 172.21.15.104:0/1990028836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc415c6f0 msgr2=0x7f1bc40b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/1990028836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b1070 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1bc406ff70 tx=0x7f1bb4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 -- 172.21.15.104:0/1990028836 shutdown_connections 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/1990028836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b1070 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/1990028836 >> v2:172.21.15.114:3300/0 conn(0x7f1bc40aec00 0x7f1bc40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 -- 172.21.15.104:0/1990028836 >> 172.21.15.104:0/1990028836 conn(0x7f1bc406b8a0 msgr2=0x7f1bc40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 -- 172.21.15.104:0/1990028836 shutdown_connections 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.675+0000 7f1bca7ce700 1 -- 172.21.15.104:0/1990028836 wait complete. 2023-06-10T18:45:00.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.676+0000 7f1bca7ce700 1 Processor -- start 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.676+0000 7f1bca7ce700 1 -- start start 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1bc406ea70 0x7f1bc4147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc41489b0 con 0x7f1bc40aec00 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40b35a0 con 0x7f1bc415c6f0 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bca7ce700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40b4f20 con 0x7f1bc406ea70 2023-06-10T18:45:00.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55836/0 (socket says 172.21.15.104:55836) 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 -- 172.21.15.104:0/2726348623 learned_addr learned my addr 172.21.15.104:0/2726348623 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc3fff700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1bc406ea70 0x7f1bc4147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc8d6b700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 -- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1bc406ea70 msgr2=0x7f1bc4147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1bc406ea70 0x7f1bc4147a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 -- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 msgr2=0x7f1bc40b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc37fe700 1 -- 172.21.15.104:0/2726348623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4009500 con 0x7f1bc40aec00 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.677+0000 7f1bc8d6b700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bc37fe700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800c4b0 tx=0x7f1bb800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80052d0 con 0x7f1bc40aec00 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb8005430 con 0x7f1bc40aec00 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80038c0 con 0x7f1bc40aec00 2023-06-10T18:45:00.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc40b5200 con 0x7f1bc40aec00 2023-06-10T18:45:00.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.678+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc41485d0 con 0x7f1bc40aec00 2023-06-10T18:45:00.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.679+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc4074050 con 0x7f1bc40aec00 2023-06-10T18:45:00.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.679+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1bb8003a20 con 0x7f1bc40aec00 2023-06-10T18:45:00.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.679+0000 7f1bc0ff9700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 0x7f1bb006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:00.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.680+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1bb8088c40 con 0x7f1bc40aec00 2023-06-10T18:45:00.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.680+0000 7f1bc3fff700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 0x7f1bb006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:00.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.680+0000 7f1bc3fff700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 0x7f1bb006d190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bc406ff70 tx=0x7f1bb4008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:00.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.684+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1bb80539c0 con 0x7f1bc40aec00 2023-06-10T18:45:00.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.871+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1bc40cdb20 con 0x7f1bc40aec00 2023-06-10T18:45:00.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.871+0000 7f1bc0ff9700 1 -- 172.21.15.104:0/2726348623 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1bb8017020 con 0x7f1bc40aec00 2023-06-10T18:45:00.877 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:00.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 msgr2=0x7f1bb006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 0x7f1bb006d190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bc406ff70 tx=0x7f1bb4008070 comp rx=0 tx=0).stop 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 msgr2=0x7f1bc40b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800c4b0 tx=0x7f1bb800c870 comp rx=0 tx=0).stop 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 shutdown_connections 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1bb006ace0 0x7f1bb006d190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc415c6f0 0x7f1bc40b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1bc406ea70 0x7f1bc4147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 --2- 172.21.15.104:0/2726348623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1bc40aec00 0x7f1bc40b01d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:00.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 >> 172.21.15.104:0/2726348623 conn(0x7f1bc406b8a0 msgr2=0x7f1bc4151140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:00.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 shutdown_connections 2023-06-10T18:45:00.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:00.874+0000 7f1bca7ce700 1 -- 172.21.15.104:0/2726348623 wait complete. 2023-06-10T18:45:00.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:01.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.010+0000 7fbfa7c7e700 1 Processor -- start 2023-06-10T18:45:01.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.011+0000 7fbfa7c7e700 1 -- start start 2023-06-10T18:45:01.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.011+0000 7fbfa7c7e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfa015c580 0x7fbfa00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa7c7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa7c7e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa006e250 con 0x7fbfa006cde0 2023-06-10T18:45:01.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa7c7e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa015c950 con 0x7fbfa015c580 2023-06-10T18:45:01.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa7c7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00cf0e0 con 0x7fbfa006e6e0 2023-06-10T18:45:01.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfa015c580 0x7fbfa00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbfa015c580 0x7fbfa00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55842/0 (socket says 172.21.15.104:55842) 2023-06-10T18:45:01.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa621b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbfa006cde0 0x7fbfa0069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53674/0 (socket says 172.21.15.104:53674) 2023-06-10T18:45:01.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/745780304 learned_addr learned my addr 172.21.15.104:0/745780304 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5219700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/745780304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 msgr2=0x7fbfa00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:01.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/745780304 >> v1:172.21.15.104:6789/0 conn(0x7fbfa006cde0 legacy=0x7fbfa0069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/745780304 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa00b1e80 con 0x7fbfa015c580 2023-06-10T18:45:01.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa5a1a700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa015c580 0x7fbfa00b1070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c006fd0 tx=0x7fbf9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ec8c7122413376 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa4a18700 1 -- 172.21.15.104:0/745780304 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c005e30 con 0x7fbfa015c580 2023-06-10T18:45:01.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa4a18700 1 -- 172.21.15.104:0/745780304 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c004030 con 0x7fbfa015c580 2023-06-10T18:45:01.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.012+0000 7fbfa4a18700 1 -- 172.21.15.104:0/745780304 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0043e0 con 0x7fbfa015c580 2023-06-10T18:45:01.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/745780304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa015c580 msgr2=0x7fbfa00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa015c580 0x7fbfa00b1070 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c006fd0 tx=0x7fbf9c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/745780304 shutdown_connections 2023-06-10T18:45:01.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa015c580 0x7fbfa00b1070 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/745780304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/745780304 >> 172.21.15.104:0/745780304 conn(0x7fbfa006b8a0 msgr2=0x7fbfa00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/745780304 shutdown_connections 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.013+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/745780304 wait complete. 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.014+0000 7fbfa7c7e700 1 Processor -- start 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.014+0000 7fbfa7c7e700 1 -- start start 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa015c580 0x7fbfa0158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00c5140 con 0x7fbfa006cde0 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00c5cc0 con 0x7fbfa015c580 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa7c7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00c5390 con 0x7fbfa006e6e0 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55848/0 (socket says 172.21.15.104:55848) 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/2962280145 learned_addr learned my addr 172.21.15.104:0/2962280145 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5219700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa621b700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa015c580 0x7fbfa0158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 msgr2=0x7fbfa00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa015c580 msgr2=0x7fbfa0158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa015c580 0x7fbfa0158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 -- 172.21.15.104:0/2962280145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c009500 con 0x7fbfa006cde0 2023-06-10T18:45:01.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.015+0000 7fbfa5a1a700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009c10 tx=0x7fbf9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.016+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c003d90 con 0x7fbfa006cde0 2023-06-10T18:45:01.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.016+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c01d430 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.016+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa00c5670 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.016+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c01d7e0 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.016+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa0158a60 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.017+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa0074050 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.017+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbf9c01d940 con 0x7fbfa006cde0 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.017+0000 7fbf967fc700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.018+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbf9c08abd0 con 0x7fbfa006cde0 2023-06-10T18:45:01.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.018+0000 7fbfa5219700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.018+0000 7fbfa5219700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009ad0 tx=0x7fbf90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.022+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbf9c055940 con 0x7fbfa006cde0 2023-06-10T18:45:01.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.208+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbfa0066f10 con 0x7fbfa006cde0 2023-06-10T18:45:01.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.209+0000 7fbf967fc700 1 -- 172.21.15.104:0/2962280145 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf9c058f60 con 0x7fbfa006cde0 2023-06-10T18:45:01.215 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:01.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 msgr2=0x7fbf8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009ad0 tx=0x7fbf90008040 comp rx=0 tx=0).stop 2023-06-10T18:45:01.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 msgr2=0x7fbfa00b46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009c10 tx=0x7fbf9c004750 comp rx=0 tx=0).stop 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 shutdown_connections 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa015c580 0x7fbfa0158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfa006cde0 0x7fbfa00b46e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 --2- 172.21.15.104:0/2962280145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfa006e6e0 0x7fbfa00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 >> 172.21.15.104:0/2962280145 conn(0x7fbfa006b8a0 msgr2=0x7fbfa00ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 shutdown_connections 2023-06-10T18:45:01.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.211+0000 7fbfa7c7e700 1 -- 172.21.15.104:0/2962280145 wait complete. 2023-06-10T18:45:01.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:01.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.345+0000 7f1649eb9700 1 Processor -- start 2023-06-10T18:45:01.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.345+0000 7f1649eb9700 1 -- start start 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1649eb9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1649eb9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f164406cde0 0x7f16440ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1649eb9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164406e250 con 0x7f164406e6e0 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1649eb9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164415c950 con 0x7f164406cde0 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1649eb9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440b1e80 con 0x7f164415c580 2023-06-10T18:45:01.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f164406cde0 0x7f16440ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f164406cde0 0x7f16440ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55860/0 (socket says 172.21.15.104:55860) 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 -- 172.21.15.104:0/3268225818 learned_addr learned my addr 172.21.15.104:0/3268225818 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1642ffd700 1 --1- 172.21.15.104:0/3268225818 >> v1:172.21.15.104:6789/0 conn(0x7f164406e6e0 0x7f16440b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53686/0 (socket says 172.21.15.104:53686) 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f16437fe700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 -- 172.21.15.104:0/3268225818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 msgr2=0x7f16440b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 -- 172.21.15.104:0/3268225818 >> v1:172.21.15.104:6789/0 conn(0x7f164406e6e0 legacy=0x7f16440b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 -- 172.21.15.104:0/3268225818 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164406d920 con 0x7f164406cde0 2023-06-10T18:45:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f1643fff700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f16440ce0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f163c009e50 tx=0x7f163c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44438d01ccafd38d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f16427fc700 1 -- 172.21.15.104:0/3268225818 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c00e070 con 0x7f164406cde0 2023-06-10T18:45:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f16427fc700 1 -- 172.21.15.104:0/3268225818 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c00b9c0 con 0x7f164406cde0 2023-06-10T18:45:01.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.347+0000 7f16427fc700 1 -- 172.21.15.104:0/3268225818 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c00bd70 con 0x7f164406cde0 2023-06-10T18:45:01.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 -- 172.21.15.104:0/3268225818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 msgr2=0x7f16440ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f16440ce0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f163c009e50 tx=0x7f163c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 -- 172.21.15.104:0/3268225818 shutdown_connections 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f16440ce0d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 --2- 172.21.15.104:0/3268225818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 -- 172.21.15.104:0/3268225818 >> 172.21.15.104:0/3268225818 conn(0x7f164406b8a0 msgr2=0x7f16440cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 -- 172.21.15.104:0/3268225818 shutdown_connections 2023-06-10T18:45:01.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.348+0000 7f1649eb9700 1 -- 172.21.15.104:0/3268225818 wait complete. 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.349+0000 7f1649eb9700 1 Processor -- start 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.349+0000 7f1649eb9700 1 -- start start 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f164406e6e0 0x7f16441479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1644147750 con 0x7f164406cde0 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440ac1c0 con 0x7f164406e6e0 2023-06-10T18:45:01.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1649eb9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440ac3d0 con 0x7f164415c580 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55864/0 (socket says 172.21.15.104:55864) 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 -- 172.21.15.104:0/1369109985 learned_addr learned my addr 172.21.15.104:0/1369109985 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f1642ffd700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f164406e6e0 0x7f16441479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 -- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 msgr2=0x7f16440c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 -- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f164406e6e0 msgr2=0x7f16441479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f164406e6e0 0x7f16441479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.350+0000 7f16437fe700 1 -- 172.21.15.104:0/1369109985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c009b30 con 0x7f164406cde0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f16437fe700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f163800d7d0 tx=0x7f163800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380125f0 con 0x7f164406cde0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1638012750 con 0x7f164406cde0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16440ac6b0 con 0x7f164406cde0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638012b00 con 0x7f164406cde0 2023-06-10T18:45:01.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.351+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16440acc70 con 0x7f164406cde0 2023-06-10T18:45:01.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.352+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1644074050 con 0x7f164406cde0 2023-06-10T18:45:01.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.352+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1638012c60 con 0x7f164406cde0 2023-06-10T18:45:01.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.353+0000 7f162bfff700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 0x7f163006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.353+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1638089940 con 0x7f164406cde0 2023-06-10T18:45:01.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.353+0000 7f1642ffd700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 0x7f163006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.353+0000 7f1642ffd700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 0x7f163006d160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16440b0670 tx=0x7f1634009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.357+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f16380546c0 con 0x7f164406cde0 2023-06-10T18:45:01.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.542+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16440cb2c0 con 0x7f164406cde0 2023-06-10T18:45:01.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.543+0000 7f162bfff700 1 -- 172.21.15.104:0/1369109985 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1638057ce0 con 0x7f164406cde0 2023-06-10T18:45:01.549 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:01.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 msgr2=0x7f163006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 0x7f163006d160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16440b0670 tx=0x7f1634009040 comp rx=0 tx=0).stop 2023-06-10T18:45:01.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 msgr2=0x7f164414a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f163800d7d0 tx=0x7f163800db90 comp rx=0 tx=0).stop 2023-06-10T18:45:01.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 shutdown_connections 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f164415c580 0x7f16440c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f164406cde0 0x7f164414a280 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f163006acb0 0x7f163006d160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 --2- 172.21.15.104:0/1369109985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f164406e6e0 0x7f16441479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 >> 172.21.15.104:0/1369109985 conn(0x7f164406b8a0 msgr2=0x7f16440b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 shutdown_connections 2023-06-10T18:45:01.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.545+0000 7f1649eb9700 1 -- 172.21.15.104:0/1369109985 wait complete. 2023-06-10T18:45:01.563 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:01.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.680+0000 7f968cc47700 1 Processor -- start 2023-06-10T18:45:01.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.680+0000 7f968cc47700 1 -- start start 2023-06-10T18:45:01.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f968815a0a0 0x7f968815a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f968806e3d0 0x7f9688069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96880cc050 con 0x7f968815a0a0 2023-06-10T18:45:01.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968806d0e0 con 0x7f968806e3d0 2023-06-10T18:45:01.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968cc47700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968806ce00 con 0x7f968806d350 2023-06-10T18:45:01.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f968815a0a0 0x7f968815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f968815a0a0 0x7f968815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55870/0 (socket says 172.21.15.104:55870) 2023-06-10T18:45:01.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.681+0000 7f968659c700 1 -- 172.21.15.104:0/2633404641 learned_addr learned my addr 172.21.15.104:0/2633404641 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f9686d9d700 1 --2- 172.21.15.104:0/2633404641 >> v2:172.21.15.114:3300/0 conn(0x7f968806e3d0 0x7f9688069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f9685d9b700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 -- 172.21.15.104:0/2633404641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 msgr2=0x7f96880ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 -- 172.21.15.104:0/2633404641 >> v2:172.21.15.114:3300/0 conn(0x7f968806e3d0 msgr2=0x7f9688069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 --2- 172.21.15.104:0/2633404641 >> v2:172.21.15.114:3300/0 conn(0x7f968806e3d0 0x7f9688069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 -- 172.21.15.104:0/2633404641 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96880cba00 con 0x7f968815a0a0 2023-06-10T18:45:01.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f9685d9b700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:01.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968659c700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f968815a470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f96880722c0 tx=0x7f967c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a544e4ce07024443 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968559a700 1 -- 172.21.15.104:0/2633404641 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c005e30 con 0x7f968815a0a0 2023-06-10T18:45:01.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968559a700 1 -- 172.21.15.104:0/2633404641 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c004030 con 0x7f968815a0a0 2023-06-10T18:45:01.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968559a700 1 -- 172.21.15.104:0/2633404641 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0043e0 con 0x7f968815a0a0 2023-06-10T18:45:01.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968cc47700 1 -- 172.21.15.104:0/2633404641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 msgr2=0x7f968815a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.682+0000 7f968cc47700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f968815a470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f96880722c0 tx=0x7f967c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 -- 172.21.15.104:0/2633404641 shutdown_connections 2023-06-10T18:45:01.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f968815a470 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 --2- 172.21.15.104:0/2633404641 >> v2:172.21.15.114:3300/0 conn(0x7f968806e3d0 0x7f9688069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 --2- 172.21.15.104:0/2633404641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 -- 172.21.15.104:0/2633404641 >> 172.21.15.104:0/2633404641 conn(0x7f968806b8a0 msgr2=0x7f96880cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 -- 172.21.15.104:0/2633404641 shutdown_connections 2023-06-10T18:45:01.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.683+0000 7f968cc47700 1 -- 172.21.15.104:0/2633404641 wait complete. 2023-06-10T18:45:01.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.684+0000 7f968cc47700 1 Processor -- start 2023-06-10T18:45:01.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.684+0000 7f968cc47700 1 -- start start 2023-06-10T18:45:01.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 0x7f96881542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880af7d0 con 0x7f968815a0a0 2023-06-10T18:45:01.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9688154df0 con 0x7f968806e3d0 2023-06-10T18:45:01.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9688154b10 con 0x7f968806d350 2023-06-10T18:45:01.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55878/0 (socket says 172.21.15.104:55878) 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 -- 172.21.15.104:0/4125406853 learned_addr learned my addr 172.21.15.104:0/4125406853 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9685d9b700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 0x7f96881542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968659c700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 -- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 msgr2=0x7f96880b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 -- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 msgr2=0x7f96881542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 0x7f96881542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 -- 172.21.15.104:0/4125406853 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c009500 con 0x7f968815a0a0 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9685d9b700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 0x7f96881542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9686d9d700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f968000ef90 tx=0x7f968000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680009ca0 con 0x7f968815a0a0 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96880af110 con 0x7f968815a0a0 2023-06-10T18:45:01.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968000f040 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.685+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680014480 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.686+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96881547f0 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.686+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96880aef60 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.687+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96800145e0 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.687+0000 7f9676ffd700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 0x7f966806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.687+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f968008a1b0 con 0x7f968815a0a0 2023-06-10T18:45:01.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.687+0000 7f968659c700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 0x7f966806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:01.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.688+0000 7f968659c700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 0x7f966806d190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f967c009c10 tx=0x7f967c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:01.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.692+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96800589d0 con 0x7f968815a0a0 2023-06-10T18:45:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.877+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f96880af910 con 0x7f968815a0a0 2023-06-10T18:45:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.878+0000 7f9676ffd700 1 -- 172.21.15.104:0/4125406853 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9680058560 con 0x7f968815a0a0 2023-06-10T18:45:01.883 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:01.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.879+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 msgr2=0x7f966806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.879+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 0x7f966806d190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f967c009c10 tx=0x7f967c008070 comp rx=0 tx=0).stop 2023-06-10T18:45:01.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.879+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 msgr2=0x7f96881f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.879+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f968000ef90 tx=0x7f968000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 shutdown_connections 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f968815a0a0 0x7f96881f25c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f968806d350 0x7f96880b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f966806ace0 0x7f966806d190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 --2- 172.21.15.104:0/4125406853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f968806e3d0 0x7f96881542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:01.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 >> 172.21.15.104:0/4125406853 conn(0x7f968806b8a0 msgr2=0x7f9688157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:01.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 shutdown_connections 2023-06-10T18:45:01.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:01.880+0000 7f968cc47700 1 -- 172.21.15.104:0/4125406853 wait complete. 2023-06-10T18:45:01.906 INFO:tasks.cephfs_test_runner:Starting test: test_network_death (tasks.cephfs.test_client_recovery.TestClientNetworkRecovery) 2023-06-10T18:45:01.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientNetworkRecovery.test_network_death' 2023-06-10T18:45:02.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.022+0000 7fa90f553700 1 Processor -- start 2023-06-10T18:45:02.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.022+0000 7fa90f553700 1 -- start start 2023-06-10T18:45:02.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90f553700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9080aeaa0 0x7fa9080cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90f553700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9080b1080 0x7fa9080cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90f553700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080c9e60 con 0x7fa9080ca2f0 2023-06-10T18:45:02.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90f553700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9080c9bc0 con 0x7fa9080aeaa0 2023-06-10T18:45:02.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90f553700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9080aee70 con 0x7fa9080b1080 2023-06-10T18:45:02.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9080aeaa0 0x7fa9080cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9080aeaa0 0x7fa9080cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55882/0 (socket says 172.21.15.104:55882) 2023-06-10T18:45:02.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 -- 172.21.15.104:0/3322900507 learned_addr learned my addr 172.21.15.104:0/3322900507 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90daf0700 1 --1- 172.21.15.104:0/3322900507 >> v1:172.21.15.104:6789/0 conn(0x7fa9080ca2f0 0x7fa9080ca710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53714/0 (socket says 172.21.15.104:53714) 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90caee700 1 --2- 172.21.15.104:0/3322900507 >> v2:172.21.15.114:3300/0 conn(0x7fa9080b1080 0x7fa9080cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 -- 172.21.15.104:0/3322900507 >> v2:172.21.15.114:3300/0 conn(0x7fa9080b1080 msgr2=0x7fa9080cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/3322900507 >> v2:172.21.15.114:3300/0 conn(0x7fa9080b1080 0x7fa9080cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 -- 172.21.15.104:0/3322900507 >> v1:172.21.15.104:6789/0 conn(0x7fa9080ca2f0 legacy=0x7fa9080ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.023+0000 7fa90d2ef700 1 -- 172.21.15.104:0/3322900507 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90806e540 con 0x7fa9080aeaa0 2023-06-10T18:45:02.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/3322900507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080aeaa0 0x7fa9080cd140 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa908070110 tx=0x7fa8f800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1544e951e950bf2c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa8fffff700 1 -- 172.21.15.104:0/3322900507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8005e30 con 0x7fa9080aeaa0 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa8fffff700 1 -- 172.21.15.104:0/3322900507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f8004030 con 0x7fa9080aeaa0 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa8fffff700 1 -- 172.21.15.104:0/3322900507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80043e0 con 0x7fa9080aeaa0 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 -- 172.21.15.104:0/3322900507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080aeaa0 msgr2=0x7fa9080cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 --2- 172.21.15.104:0/3322900507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080aeaa0 0x7fa9080cd140 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa908070110 tx=0x7fa8f800be70 comp rx=0 tx=0).stop 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 -- 172.21.15.104:0/3322900507 shutdown_connections 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 --2- 172.21.15.104:0/3322900507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080aeaa0 0x7fa9080cd140 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 --2- 172.21.15.104:0/3322900507 >> v2:172.21.15.114:3300/0 conn(0x7fa9080b1080 0x7fa9080cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 -- 172.21.15.104:0/3322900507 >> 172.21.15.104:0/3322900507 conn(0x7fa90806b8a0 msgr2=0x7fa9080cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:02.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 -- 172.21.15.104:0/3322900507 shutdown_connections 2023-06-10T18:45:02.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.024+0000 7fa90f553700 1 -- 172.21.15.104:0/3322900507 wait complete. 2023-06-10T18:45:02.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 Processor -- start 2023-06-10T18:45:02.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 -- start start 2023-06-10T18:45:02.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 0x7fa9081f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 0x7fa908156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90815b060 con 0x7fa9080ca2f0 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90815a1a0 con 0x7fa9080aeaa0 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90f553700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90815a9a0 con 0x7fa9080b1080 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90daf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90daf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55892/0 (socket says 172.21.15.104:55892) 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.026+0000 7fa90daf0700 1 -- 172.21.15.104:0/1364295832 learned_addr learned my addr 172.21.15.104:0/1364295832 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:02.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 0x7fa9081f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90caee700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 0x7fa908156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 -- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 msgr2=0x7fa908156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 0x7fa908156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 -- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 msgr2=0x7fa9081f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 0x7fa9081f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 -- 172.21.15.104:0/1364295832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f800b7a0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 0x7fa9081f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:02.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90caee700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 0x7fa908156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:02.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90daf0700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa90000cbc0 tx=0x7fa90000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90000ab50 con 0x7fa9080ca2f0 2023-06-10T18:45:02.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90000acb0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90815ac80 con 0x7fa9080ca2f0 2023-06-10T18:45:02.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900007270 con 0x7fa9080ca2f0 2023-06-10T18:45:02.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.027+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9081ee970 con 0x7fa9080ca2f0 2023-06-10T18:45:02.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.028+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90815a800 con 0x7fa9080ca2f0 2023-06-10T18:45:02.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.028+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa9000073d0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.029+0000 7fa8fdffb700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 0x7fa8ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.029+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa9000894e0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.029+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 0x7fa8ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.029+0000 7fa90d2ef700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 0x7fa8ec06d190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa9080b0a00 tx=0x7fa8f801b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.033+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa90815a800 con 0x7fa9080ca2f0 2023-06-10T18:45:02.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.295+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientNetworkRecovery.test_network_death"]} v 0) v1 -- 0x7fa90815b1a0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.744+0000 7fa8fdffb700 1 -- 172.21.15.104:0/1364295832 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientNetworkRecovery.test_network_death"]}]=0 v72) v1 ==== 160+0+0 (secure 0 0 0) 0x7fa90815b1a0 con 0x7fa9080ca2f0 2023-06-10T18:45:02.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 msgr2=0x7fa8ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 0x7fa8ec06d190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa9080b0a00 tx=0x7fa8f801b040 comp rx=0 tx=0).stop 2023-06-10T18:45:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 msgr2=0x7fa908156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa90000cbc0 tx=0x7fa90000cf80 comp rx=0 tx=0).stop 2023-06-10T18:45:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 shutdown_connections 2023-06-10T18:45:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8ec06ace0 0x7fa8ec06d190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9080aeaa0 0x7fa9081f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9080ca2f0 0x7fa908156ea0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 --2- 172.21.15.104:0/1364295832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9080b1080 0x7fa908156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 >> 172.21.15.104:0/1364295832 conn(0x7fa90806b8a0 msgr2=0x7fa9080cb620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 shutdown_connections 2023-06-10T18:45:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.747+0000 7fa90f553700 1 -- 172.21.15.104:0/1364295832 wait complete. 2023-06-10T18:45:02.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:45:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.891+0000 7fa01d920700 1 Processor -- start 2023-06-10T18:45:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.892+0000 7fa01d920700 1 -- start start 2023-06-10T18:45:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa01d920700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0180d25c0 0x7fa0180d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa01d920700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180b05d0 0x7fa0180b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa01d920700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0180af530 con 0x7fa0180d04f0 2023-06-10T18:45:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa01d920700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0180af810 con 0x7fa0180d25c0 2023-06-10T18:45:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa01d920700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180afaf0 con 0x7fa0180b05d0 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0180d25c0 0x7fa0180d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa0180d25c0 0x7fa0180d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55898/0 (socket says 172.21.15.104:55898) 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 -- 172.21.15.104:0/214508747 learned_addr learned my addr 172.21.15.104:0/214508747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0177fe700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180b05d0 0x7fa0180b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 -- 172.21.15.104:0/214508747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180b05d0 msgr2=0x7fa0180b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180b05d0 0x7fa0180b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 -- 172.21.15.104:0/214508747 >> v1:172.21.15.104:6789/0 conn(0x7fa0180d04f0 legacy=0x7fa0180d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa015ffb700 1 -- 172.21.15.104:0/214508747 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608425147 0 0) 0x7fa0180af530 con 0x7fa0180d04f0 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 -- 172.21.15.104:0/214508747 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180afdd0 con 0x7fa0180d25c0 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa0167fc700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180d25c0 0x7fa0180d34d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa0080098f0 tx=0x7fa008009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30100d541bb499d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.893+0000 7fa015ffb700 1 -- 172.21.15.104:0/214508747 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00800b8a0 con 0x7fa0180d25c0 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa015ffb700 1 -- 172.21.15.104:0/214508747 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00800ba00 con 0x7fa0180d25c0 2023-06-10T18:45:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa015ffb700 1 -- 172.21.15.104:0/214508747 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00800bd90 con 0x7fa0180d25c0 2023-06-10T18:45:02.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 -- 172.21.15.104:0/214508747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180d25c0 msgr2=0x7fa0180d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180d25c0 0x7fa0180d34d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa0080098f0 tx=0x7fa008009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:02.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 -- 172.21.15.104:0/214508747 shutdown_connections 2023-06-10T18:45:02.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180b05d0 0x7fa0180b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 --2- 172.21.15.104:0/214508747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180d25c0 0x7fa0180d34d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 -- 172.21.15.104:0/214508747 >> 172.21.15.104:0/214508747 conn(0x7fa018071ec0 msgr2=0x7fa01809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 -- 172.21.15.104:0/214508747 shutdown_connections 2023-06-10T18:45:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.894+0000 7fa01d920700 1 -- 172.21.15.104:0/214508747 wait complete. 2023-06-10T18:45:02.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.895+0000 7fa01d920700 1 Processor -- start 2023-06-10T18:45:02.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.895+0000 7fa01d920700 1 -- start start 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180d04f0 0x7fa0180ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 0x7fa0181f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018159f50 con 0x7fa0180b05d0 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0181e3950 con 0x7fa0180d25c0 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa01d920700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0181e3c30 con 0x7fa0180d04f0 2023-06-10T18:45:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55906/0 (socket says 172.21.15.104:55906) 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 -- 172.21.15.104:0/3465956699 learned_addr learned my addr 172.21.15.104:0/3465956699 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa0177fe700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 0x7fa0181f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa0167fc700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180d04f0 0x7fa0180ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 -- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180d04f0 msgr2=0x7fa0180ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180d04f0 0x7fa0180ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 -- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 msgr2=0x7fa0181f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.896+0000 7fa016ffd700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 0x7fa0181f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7fa016ffd700 1 -- 172.21.15.104:0/3465956699 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa008009550 con 0x7fa0180b05d0 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7fa0177fe700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 0x7fa0181f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7fa016ffd700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c550 tx=0x7fa00c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c0054e0 con 0x7fa0180b05d0 2023-06-10T18:45:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c003680 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0181e3f10 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c003a30 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.897+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0181e44d0 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.898+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa018067010 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.898+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa00c003b90 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.899+0000 7f9ffffff700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 0x7fa00406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.899+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa00c088c90 con 0x7fa0180b05d0 2023-06-10T18:45:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.899+0000 7fa0167fc700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 0x7fa00406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:02.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.899+0000 7fa0167fc700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 0x7fa00406d190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa0180d01f0 tx=0x7fa008008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:02.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:02.903+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa00c053aa0 con 0x7fa0180b05d0 2023-06-10T18:45:03.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.089+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa018074050 con 0x7fa0180b05d0 2023-06-10T18:45:03.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.093+0000 7f9ffffff700 1 -- 172.21.15.104:0/3465956699 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa00c0170c0 con 0x7fa0180b05d0 2023-06-10T18:45:03.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 msgr2=0x7fa00406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 0x7fa00406d190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa0180d01f0 tx=0x7fa008008030 comp rx=0 tx=0).stop 2023-06-10T18:45:03.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 msgr2=0x7fa0181593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c550 tx=0x7fa00c00c910 comp rx=0 tx=0).stop 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 shutdown_connections 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa00406ace0 0x7fa00406d190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0180d25c0 0x7fa0181f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0180b05d0 0x7fa0181593e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 --2- 172.21.15.104:0/3465956699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0180d04f0 0x7fa0180ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 >> 172.21.15.104:0/3465956699 conn(0x7fa018071ec0 msgr2=0x7fa018072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 shutdown_connections 2023-06-10T18:45:03.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.095+0000 7fa01d920700 1 -- 172.21.15.104:0/3465956699 wait complete. 2023-06-10T18:45:03.113 INFO:tasks.cephfs_test_runner:test_network_death (tasks.cephfs.test_client_recovery.TestClientNetworkRecovery) ... skipped 'Require first client to be on separate server from MDSs' 2023-06-10T18:45:03.113 INFO:tasks.cephfs_test_runner:Starting test: test_abort_conn (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:45:03.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn' 2023-06-10T18:45:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.228+0000 7f3da0058700 1 Processor -- start 2023-06-10T18:45:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 -- start start 2023-06-10T18:45:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3d980d2b20 0x7f3d980d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 0x7f3d980b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d980cd430 con 0x7f3d980b2780 2023-06-10T18:45:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d980cd710 con 0x7f3d980d2b20 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.229+0000 7f3da0058700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980cd9f0 con 0x7f3d980ccfa0 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3d980d2b20 0x7f3d980d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3d980d2b20 0x7f3d980d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55910/0 (socket says 172.21.15.104:55910) 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 -- 172.21.15.104:0/826088789 learned_addr learned my addr 172.21.15.104:0/826088789 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9d5f3700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 0x7f3d980b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 -- 172.21.15.104:0/826088789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 msgr2=0x7f3d980b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 0x7f3d980b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 -- 172.21.15.104:0/826088789 >> v1:172.21.15.114:6789/0 conn(0x7f3d980b2780 legacy=0x7f3d980d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 -- 172.21.15.104:0/826088789 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d980cdcd0 con 0x7f3d980d2b20 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9d5f3700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 0x7f3d980b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9ddf4700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980d2b20 0x7f3d980d2f00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3d880098f0 tx=0x7f3d88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefadbd1130396be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9cdf2700 1 -- 172.21.15.104:0/826088789 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800b8a0 con 0x7f3d980d2b20 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9cdf2700 1 -- 172.21.15.104:0/826088789 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8800ba00 con 0x7f3d980d2b20 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.230+0000 7f3d9cdf2700 1 -- 172.21.15.104:0/826088789 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800bd90 con 0x7f3d980d2b20 2023-06-10T18:45:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 -- 172.21.15.104:0/826088789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980d2b20 msgr2=0x7f3d980d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980d2b20 0x7f3d980d2f00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3d880098f0 tx=0x7f3d88009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:03.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 -- 172.21.15.104:0/826088789 shutdown_connections 2023-06-10T18:45:03.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980d2b20 0x7f3d980d2f00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 --2- 172.21.15.104:0/826088789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d980ccfa0 0x7f3d980b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 -- 172.21.15.104:0/826088789 >> 172.21.15.104:0/826088789 conn(0x7f3d98071ec0 msgr2=0x7f3d9809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:03.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 -- 172.21.15.104:0/826088789 shutdown_connections 2023-06-10T18:45:03.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.231+0000 7f3da0058700 1 -- 172.21.15.104:0/826088789 wait complete. 2023-06-10T18:45:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.232+0000 7f3da0058700 1 Processor -- start 2023-06-10T18:45:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.232+0000 7f3da0058700 1 -- start start 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d980b2780 0x7f3d980b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 0x7f3d9815e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980b66d0 con 0x7f3d980ccfa0 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980b6930 con 0x7f3d980b2780 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980b6c10 con 0x7f3d98158dc0 2023-06-10T18:45:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55918/0 (socket says 172.21.15.104:55918) 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 -- 172.21.15.104:0/2137981707 learned_addr learned my addr 172.21.15.104:0/2137981707 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9ddf4700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d980b2780 0x7f3d980b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9e5f5700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 0x7f3d9815e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 -- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 msgr2=0x7f3d9815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 0x7f3d9815e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 -- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d980b2780 msgr2=0x7f3d980b8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d980b2780 0x7f3d980b8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 -- 172.21.15.104:0/2137981707 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88009550 con 0x7f3d980ccfa0 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9e5f5700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 0x7f3d9815e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d9d5f3700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c750 tx=0x7f3d9400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94011800 con 0x7f3d980ccfa0 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d94011960 con 0x7f3d980ccfa0 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d980b6ef0 con 0x7f3d980ccfa0 2023-06-10T18:45:03.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.233+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94011cd0 con 0x7f3d980ccfa0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.234+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d980ca780 con 0x7f3d980ccfa0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.234+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d98074050 con 0x7f3d980ccfa0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.235+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3d94021350 con 0x7f3d980ccfa0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.235+0000 7f3d8e7fc700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 0x7f3d7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.235+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3d94089760 con 0x7f3d980ccfa0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.235+0000 7f3d9ddf4700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 0x7f3d7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.236+0000 7f3d9ddf4700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 0x7f3d7c06d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d88014040 tx=0x7f3d88008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.239+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d94054570 con 0x7f3d980ccfa0 2023-06-10T18:45:03.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.501+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn"]} v 0) v1 -- 0x7f3d980caa00 con 0x7f3d980ccfa0 2023-06-10T18:45:03.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.745+0000 7f3d8e7fc700 1 -- 172.21.15.104:0/2137981707 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn"]}]=0 v73) v1 ==== 150+0+0 (secure 0 0 0) 0x7f3d94057b90 con 0x7f3d980ccfa0 2023-06-10T18:45:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 msgr2=0x7f3d7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 0x7f3d7c06d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d88014040 tx=0x7f3d88008030 comp rx=0 tx=0).stop 2023-06-10T18:45:03.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 msgr2=0x7f3d980b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c750 tx=0x7f3d9400cb10 comp rx=0 tx=0).stop 2023-06-10T18:45:03.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 shutdown_connections 2023-06-10T18:45:03.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d7c06ace0 0x7f3d7c06d190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d980b2780 0x7f3d980b8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d98158dc0 0x7f3d9815e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 --2- 172.21.15.104:0/2137981707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d980ccfa0 0x7f3d980b6190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 >> 172.21.15.104:0/2137981707 conn(0x7f3d98071ec0 msgr2=0x7f3d9806b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 shutdown_connections 2023-06-10T18:45:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.748+0000 7f3da0058700 1 -- 172.21.15.104:0/2137981707 wait complete. 2023-06-10T18:45:03.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:45:03.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 Processor -- start 2023-06-10T18:45:03.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 -- start start 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa050069e70 0x7fa05006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa05006e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05006e690 con 0x7fa05015c500 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0500cf4d0 con 0x7fa050069e70 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa056b3b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500ce040 con 0x7fa05006dcd0 2023-06-10T18:45:03.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa050069e70 0x7fa05006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa050069e70 0x7fa05006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55926/0 (socket says 172.21.15.104:55926) 2023-06-10T18:45:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 -- 172.21.15.104:0/3635088120 learned_addr learned my addr 172.21.15.104:0/3635088120 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 -- 172.21.15.104:0/3635088120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 msgr2=0x7fa05006e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa0550d8700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa05006e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa05006e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 -- 172.21.15.104:0/3635088120 >> v1:172.21.15.114:6789/0 conn(0x7fa05015c500 legacy=0x7fa0500cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 -- 172.21.15.104:0/3635088120 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05015c8d0 con 0x7fa050069e70 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa0550d8700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa05006e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04ffff700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa050069e70 0x7fa05006a2e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0098f0 tx=0x7fa03c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97e07165c11ff96 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04f7fe700 1 -- 172.21.15.104:0/3635088120 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00b8a0 con 0x7fa050069e70 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04f7fe700 1 -- 172.21.15.104:0/3635088120 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c00ba00 con 0x7fa050069e70 2023-06-10T18:45:03.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.885+0000 7fa04f7fe700 1 -- 172.21.15.104:0/3635088120 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00bd90 con 0x7fa050069e70 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 -- 172.21.15.104:0/3635088120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa050069e70 msgr2=0x7fa05006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa050069e70 0x7fa05006a2e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0098f0 tx=0x7fa03c009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 -- 172.21.15.104:0/3635088120 shutdown_connections 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa05006e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 --2- 172.21.15.104:0/3635088120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa050069e70 0x7fa05006a2e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 -- 172.21.15.104:0/3635088120 >> 172.21.15.104:0/3635088120 conn(0x7fa05006b8a0 msgr2=0x7fa0500cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:03.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 -- 172.21.15.104:0/3635088120 shutdown_connections 2023-06-10T18:45:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.886+0000 7fa056b3b700 1 -- 172.21.15.104:0/3635088120 wait complete. 2023-06-10T18:45:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.888+0000 7fa056b3b700 1 Processor -- start 2023-06-10T18:45:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.888+0000 7fa056b3b700 1 -- start start 2023-06-10T18:45:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa050069e70 0x7fa0500c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa0501f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa050159de0 con 0x7fa05015c500 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0501f5b40 con 0x7fa050069e70 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa056b3b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0501f5e20 con 0x7fa05006dcd0 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55934/0 (socket says 172.21.15.104:55934) 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 -- 172.21.15.104:0/2212450300 learned_addr learned my addr 172.21.15.104:0/2212450300 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa04ffff700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa0501f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0548d7700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa050069e70 0x7fa0500c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 -- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 msgr2=0x7fa0501f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa0501f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 -- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa050069e70 msgr2=0x7fa0500c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa050069e70 0x7fa0500c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 -- 172.21.15.104:0/2212450300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c009550 con 0x7fa05015c500 2023-06-10T18:45:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa0550d8700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa04800cef0 tx=0x7fa04800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04800ad00 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.889+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04800ae60 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.890+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0501f6100 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.890+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048015680 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.890+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa050158b30 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.890+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0501f6340 con 0x7fa05015c500 2023-06-10T18:45:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.891+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa04800d430 con 0x7fa05015c500 2023-06-10T18:45:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.891+0000 7fa04d7fa700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 0x7fa03406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.891+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa048089bd0 con 0x7fa05015c500 2023-06-10T18:45:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.892+0000 7fa0548d7700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 0x7fa03406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.892+0000 7fa0548d7700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 0x7fa03406d190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa044000fb0 tx=0x7fa044009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:03.896+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa0501f6340 con 0x7fa05015c500 2023-06-10T18:45:04.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.087+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa050158470 con 0x7fa05015c500 2023-06-10T18:45:04.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.088+0000 7fa04d7fa700 1 -- 172.21.15.104:0/2212450300 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa050158470 con 0x7fa05015c500 2023-06-10T18:45:04.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 msgr2=0x7fa03406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 0x7fa03406d190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa044000fb0 tx=0x7fa044009040 comp rx=0 tx=0).stop 2023-06-10T18:45:04.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 msgr2=0x7fa0501598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa04800cef0 tx=0x7fa04800cf20 comp rx=0 tx=0).stop 2023-06-10T18:45:04.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 shutdown_connections 2023-06-10T18:45:04.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa03406ace0 0x7fa03406d190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa050069e70 0x7fa0500c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa05015c500 0x7fa0501598a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 --2- 172.21.15.104:0/2212450300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa05006dcd0 0x7fa0501f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 >> 172.21.15.104:0/2212450300 conn(0x7fa05006b8a0 msgr2=0x7fa0500cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:04.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 shutdown_connections 2023-06-10T18:45:04.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.090+0000 7fa056b3b700 1 -- 172.21.15.104:0/2212450300 wait complete. 2023-06-10T18:45:04.108 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:45:04.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:45:04.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:45:04.206 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:45:04.206 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:04.207 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:04.224 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:04.224 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:04.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:04.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.434+0000 7f6f9b02c700 1 Processor -- start 2023-06-10T18:45:04.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f9b02c700 1 -- start start 2023-06-10T18:45:04.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f9b02c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6f9406a580 0x7f6f940ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f9b02c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9406dcb0 con 0x7f6f9415c500 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f9b02c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f940cf460 con 0x7f6f940cf030 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f9b02c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9406e850 con 0x7f6f9406a580 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f995c9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6f9406a580 0x7f6f940ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.435+0000 7f6f98dc8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6f9415c500 0x7f6f9406e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53766/0 (socket says 172.21.15.104:53766) 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f98dc8700 1 -- 172.21.15.104:0/2883925312 learned_addr learned my addr 172.21.15.104:0/2883925312 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:04.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f995c9700 1 -- 172.21.15.104:0/2883925312 >> v1:172.21.15.114:6789/0 conn(0x7f6f940cf030 legacy=0x7f6f94069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f995c9700 1 -- 172.21.15.104:0/2883925312 >> v1:172.21.15.104:6789/0 conn(0x7f6f9415c500 legacy=0x7f6f9406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f995c9700 1 -- 172.21.15.104:0/2883925312 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9406d0a0 con 0x7f6f9406a580 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f937fe700 1 -- 172.21.15.104:0/2883925312 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200545100 0 0) 0x7f6f9406dcb0 con 0x7f6f9415c500 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f995c9700 1 --2- 172.21.15.104:0/2883925312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940ce040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009d90 tx=0x7f6f8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee9eb6a2317fad6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f937fe700 1 -- 172.21.15.104:0/2883925312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00e070 con 0x7f6f9406a580 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f937fe700 1 -- 172.21.15.104:0/2883925312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c00bbd0 con 0x7f6f9406a580 2023-06-10T18:45:04.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.436+0000 7f6f937fe700 1 -- 172.21.15.104:0/2883925312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c012430 con 0x7f6f9406a580 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 -- 172.21.15.104:0/2883925312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 msgr2=0x7f6f940ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/2883925312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940ce040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009d90 tx=0x7f6f8c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 -- 172.21.15.104:0/2883925312 shutdown_connections 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/2883925312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940ce040 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 -- 172.21.15.104:0/2883925312 >> 172.21.15.104:0/2883925312 conn(0x7f6f9406b8a0 msgr2=0x7f6f940cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 -- 172.21.15.104:0/2883925312 shutdown_connections 2023-06-10T18:45:04.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.437+0000 7f6f9b02c700 1 -- 172.21.15.104:0/2883925312 wait complete. 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.438+0000 7f6f9b02c700 1 Processor -- start 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.438+0000 7f6f9b02c700 1 -- start start 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f940cf030 0x7f6f940ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9415c500 0x7f6f94155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940c6d80 con 0x7f6f9406a580 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940c7060 con 0x7f6f9415c500 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940c7340 con 0x7f6f940cf030 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55944/0 (socket says 172.21.15.104:55944) 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 -- 172.21.15.104:0/581827356 learned_addr learned my addr 172.21.15.104:0/581827356 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f995c9700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9415c500 0x7f6f94155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f93fff700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f940cf030 0x7f6f940ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 -- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f940cf030 msgr2=0x7f6f940ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f940cf030 0x7f6f940ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 -- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9415c500 msgr2=0x7f6f94155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9415c500 0x7f6f94155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 -- 172.21.15.104:0/581827356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c009a70 con 0x7f6f9406a580 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f98dc8700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c6d0 tx=0x7f6f8800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f880074e0 con 0x7f6f9406a580 2023-06-10T18:45:04.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88007640 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f940c7620 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88005a70 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.439+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f941f1700 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.440+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f94074050 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.441+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6f88005bd0 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.441+0000 7f6f917fa700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 0x7f6f7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.441+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6f88088cc0 con 0x7f6f9406a580 2023-06-10T18:45:04.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.441+0000 7f6f93fff700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 0x7f6f7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.442+0000 7f6f93fff700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 0x7f6f7806d190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f9409d5e0 tx=0x7f6f80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.446+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6f880539b0 con 0x7f6f9406a580 2023-06-10T18:45:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.636+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9406e1e0 con 0x7f6f9406a580 2023-06-10T18:45:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.637+0000 7f6f917fa700 1 -- 172.21.15.104:0/581827356 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f6f88017070 con 0x7f6f9406a580 2023-06-10T18:45:04.643 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:04.644 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.639+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 msgr2=0x7f6f7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.639+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 0x7f6f7806d190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f9409d5e0 tx=0x7f6f80009040 comp rx=0 tx=0).stop 2023-06-10T18:45:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.639+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 msgr2=0x7f6f940c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.639+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c6d0 tx=0x7f6f8800c9e0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 shutdown_connections 2023-06-10T18:45:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f7806ace0 0x7f6f7806d190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f9415c500 0x7f6f94155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f9406a580 0x7f6f940c66e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 --2- 172.21.15.104:0/581827356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f940cf030 0x7f6f940ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 >> 172.21.15.104:0/581827356 conn(0x7f6f9406b8a0 msgr2=0x7f6f940cb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 shutdown_connections 2023-06-10T18:45:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.640+0000 7f6f9b02c700 1 -- 172.21.15.104:0/581827356 wait complete. 2023-06-10T18:45:04.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:04.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:04.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 Processor -- start 2023-06-10T18:45:04.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 -- start start 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5bf0070a10 0x7f5bf006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bf0070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf00cec40 con 0x7f5bf015db40 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf00cca90 con 0x7f5bf0070a10 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bf60e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf006d810 con 0x7f5bf0143a40 2023-06-10T18:45:04.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5beffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5bf015db40 0x7f5bf015dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53784/0 (socket says 172.21.15.104:53784) 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5beffff700 1 -- 172.21.15.104:0/3226757556 learned_addr learned my addr 172.21.15.104:0/3226757556 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5beeffd700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bf0070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bef7fe700 1 --2- 172.21.15.104:0/3226757556 >> v2:172.21.15.114:3300/0 conn(0x7f5bf0070a10 0x7f5bf006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230625654 0 0) 0x7f5bf00cec40 con 0x7f5bf015db40 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd80035b0 con 0x7f5bf015db40 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1521040528 0 0) 0x7f5bd80035b0 con 0x7f5bf015db40 2023-06-10T18:45:04.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 msgr2=0x7f5bf0070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.779+0000 7f5bee7fc700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bf0070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 >> v2:172.21.15.114:3300/0 conn(0x7f5bf0070a10 msgr2=0x7f5bf006e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 --2- 172.21.15.104:0/3226757556 >> v2:172.21.15.114:3300/0 conn(0x7f5bf0070a10 0x7f5bf006e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0071320 con 0x7f5bf015db40 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5be80019f0 con 0x7f5bf015db40 2023-06-10T18:45:04.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 >> v1:172.21.15.104:6789/0 conn(0x7f5bf015db40 legacy=0x7f5bf015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bd8003a00 0x7f5bd8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bd8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bd8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00cca90 con 0x7f5bd8004850 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf006d810 con 0x7f5bd8003a00 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd800ba90 con 0x7f5bf0143a40 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bd8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bef7fe700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bd8003a00 0x7f5bd8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beffff700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bd8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 -- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 msgr2=0x7f5bd8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf0143a40 0x7f5bd8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 -- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bd8003a00 msgr2=0x7f5bd8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bd8003a00 0x7f5bd8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 -- 172.21.15.104:0/3226757556 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd80035b0 con 0x7f5bd8004850 2023-06-10T18:45:04.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5beeffd700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bd8004cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009b60 tx=0x7f5bd8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35617b95d06c356a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be000e6c0 con 0x7f5bd8004850 2023-06-10T18:45:04.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be000e820 con 0x7f5bd8004850 2023-06-10T18:45:04.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.780+0000 7f5bee7fc700 1 -- 172.21.15.104:0/3226757556 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be000eb90 con 0x7f5bd8004850 2023-06-10T18:45:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 -- 172.21.15.104:0/3226757556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 msgr2=0x7f5bd8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/3226757556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bd8004cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009b60 tx=0x7f5bd8005880 comp rx=0 tx=0).stop 2023-06-10T18:45:04.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5beffff700 1 -- 172.21.15.104:0/3226757556 reap_dead start 2023-06-10T18:45:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 -- 172.21.15.104:0/3226757556 shutdown_connections 2023-06-10T18:45:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 -- 172.21.15.104:0/3226757556 >> 172.21.15.104:0/3226757556 conn(0x7f5bf00ceec0 msgr2=0x7f5bf00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 -- 172.21.15.104:0/3226757556 shutdown_connections 2023-06-10T18:45:04.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.781+0000 7f5bf60e5700 1 -- 172.21.15.104:0/3226757556 wait complete. 2023-06-10T18:45:04.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.782+0000 7f5bf60e5700 1 Processor -- start 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.782+0000 7f5bf60e5700 1 -- start start 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 0x7f5bf00b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 0x7f5bf00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf015b830 con 0x7f5bd8004850 2023-06-10T18:45:04.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf015bb10 con 0x7f5bf00b2d50 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bf60e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf015bdf0 con 0x7f5bf00b4940 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55964/0 (socket says 172.21.15.104:55964) 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 -- 172.21.15.104:0/1410795433 learned_addr learned my addr 172.21.15.104:0/1410795433 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5beffff700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 0x7f5bf00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5beeffd700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 0x7f5bf00b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 -- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 msgr2=0x7f5bf00b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 0x7f5bf00b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 -- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 msgr2=0x7f5bf00b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 0x7f5bf00b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5bef7fe700 1 -- 172.21.15.104:0/1410795433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf015c0d0 con 0x7f5bd8004850 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5beffff700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 0x7f5bf00b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.783+0000 7f5beeffd700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 0x7f5bf00b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:04.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bef7fe700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5be4011aa0 tx=0x7f5be4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400f9e0 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be400fb40 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf015c3b0 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4014440 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.784+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf00b25d0 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.785+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf00b4d40 con 0x7f5bd8004850 2023-06-10T18:45:04.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.785+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5be40145a0 con 0x7f5bd8004850 2023-06-10T18:45:04.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.786+0000 7f5bd7fff700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 0x7f5bdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:04.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.786+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5be408a0b0 con 0x7f5bd8004850 2023-06-10T18:45:04.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.786+0000 7f5beeffd700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 0x7f5bdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:04.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.786+0000 7f5beeffd700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 0x7f5bdc06d190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5be00090b0 tx=0x7f5be0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:04.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.791+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5be4054e20 con 0x7f5bd8004850 2023-06-10T18:45:04.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.974+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5bf0074050 con 0x7f5bd8004850 2023-06-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.974+0000 7f5bd7fff700 1 -- 172.21.15.104:0/1410795433 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5be4058440 con 0x7f5bd8004850 2023-06-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 msgr2=0x7f5bdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 0x7f5bdc06d190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5be00090b0 tx=0x7f5be0008030 comp rx=0 tx=0).stop 2023-06-10T18:45:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 msgr2=0x7f5bf013c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5be4011aa0 tx=0x7f5be4011e60 comp rx=0 tx=0).stop 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 shutdown_connections 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bf00b4940 0x7f5bf00b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bd8004850 0x7f5bf013c240 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bdc06ace0 0x7f5bdc06d190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 --2- 172.21.15.104:0/1410795433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf00b2d50 0x7f5bf00b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 >> 172.21.15.104:0/1410795433 conn(0x7f5bf00ceec0 msgr2=0x7f5bf00aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 shutdown_connections 2023-06-10T18:45:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:04.976+0000 7f5bf60e5700 1 -- 172.21.15.104:0/1410795433 wait complete. 2023-06-10T18:45:04.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.112+0000 7f02308a5700 1 Processor -- start 2023-06-10T18:45:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.112+0000 7f02308a5700 1 -- start start 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f02308a5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f022c06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f02308a5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f022c06c800 0x7f022c06cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f02308a5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022c0cec40 con 0x7f022c070a10 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f02308a5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c0cca90 con 0x7f022c06c800 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f02308a5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0ce960 con 0x7f022c143a40 2023-06-10T18:45:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f022a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f022c070a10 0x7f022c070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53800/0 (socket says 172.21.15.104:53800) 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.113+0000 7f022a59c700 1 -- 172.21.15.104:0/3802877394 learned_addr learned my addr 172.21.15.104:0/3802877394 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> v2:172.21.15.114:3300/0 conn(0x7f022c06c800 0x7f022c06cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f0229d9b700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f022c06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084373730 0 0) 0x7f022c0cec40 con 0x7f022c070a10 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02140035b0 con 0x7f022c070a10 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203798237 0 0) 0x7f02140035b0 con 0x7f022c070a10 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 msgr2=0x7f022c06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f022c06e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 >> v2:172.21.15.114:3300/0 conn(0x7f022c06c800 msgr2=0x7f022c06cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 --2- 172.21.15.104:0/3802877394 >> v2:172.21.15.114:3300/0 conn(0x7f022c06c800 0x7f022c06cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c06a9b0 con 0x7f022c070a10 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> v2:172.21.15.114:3300/0 conn(0x7f022c06c800 0x7f022c06cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0220001670 con 0x7f022c070a10 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 >> v1:172.21.15.104:6789/0 conn(0x7f022c070a10 legacy=0x7f022c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f0214003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214003f80 0x7f0214004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0214004d90 0x7f0214005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0cca90 con 0x7f0214004d90 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0ce960 con 0x7f0214003f80 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021400baf0 con 0x7f022c143a40 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0214004d90 0x7f0214005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022a59c700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f0214003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f0229d9b700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214003f80 0x7f0214004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 -- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 msgr2=0x7f0214003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c143a40 0x7f0214003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 -- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214003f80 msgr2=0x7f0214004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214003f80 0x7f0214004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.114+0000 7f022ad9d700 1 -- 172.21.15.104:0/3802877394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140035b0 con 0x7f0214004d90 2023-06-10T18:45:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0214004d90 0x7f0214005200 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f022400e320 tx=0x7f0214005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a9f3c7f0e026a9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400edc0 con 0x7f0214004d90 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022400f040 con 0x7f0214004d90 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f022959a700 1 -- 172.21.15.104:0/3802877394 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02240146a0 con 0x7f0214004d90 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 -- 172.21.15.104:0/3802877394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0214004d90 msgr2=0x7f0214005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 --2- 172.21.15.104:0/3802877394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0214004d90 0x7f0214005200 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f022400e320 tx=0x7f0214005880 comp rx=0 tx=0).stop 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f022ad9d700 1 -- 172.21.15.104:0/3802877394 reap_dead start 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 -- 172.21.15.104:0/3802877394 shutdown_connections 2023-06-10T18:45:05.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 -- 172.21.15.104:0/3802877394 >> 172.21.15.104:0/3802877394 conn(0x7f022c0ceec0 msgr2=0x7f022c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 -- 172.21.15.104:0/3802877394 shutdown_connections 2023-06-10T18:45:05.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.115+0000 7f02308a5700 1 -- 172.21.15.104:0/3802877394 wait complete. 2023-06-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 Processor -- start 2023-06-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 -- start start 2023-06-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214004d90 0x7f022c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c070de0 0x7f022c071250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c159fc0 con 0x7f022c1f1d90 2023-06-10T18:45:05.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c071790 con 0x7f0214004d90 2023-06-10T18:45:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.117+0000 7f02308a5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c1f5c50 con 0x7f022c070de0 2023-06-10T18:45:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55988/0 (socket says 172.21.15.104:55988) 2023-06-10T18:45:05.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 -- 172.21.15.104:0/3160866503 learned_addr learned my addr 172.21.15.104:0/3160866503 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:05.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022a59c700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214004d90 0x7f022c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f0229d9b700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c070de0 0x7f022c071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 -- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c070de0 msgr2=0x7f022c071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c070de0 0x7f022c071250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 -- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214004d90 msgr2=0x7f022c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214004d90 0x7f022c159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 -- 172.21.15.104:0/3160866503 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c0cd0a0 con 0x7f022c1f1d90 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f022ad9d700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f021400ba40 tx=0x7f0224009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224007000 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02240043e0 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c1f60e0 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224004790 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.118+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c1f5600 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.119+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c1f5de0 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.119+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0224024020 con 0x7f022c1f1d90 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.120+0000 7f021affd700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 0x7f020c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.120+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f022408aef0 con 0x7f022c1f1d90 2023-06-10T18:45:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.120+0000 7f022a59c700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 0x7f020c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.120+0000 7f022a59c700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 0x7f020c06d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02200010e0 tx=0x7f0220007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.124+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0224055c80 con 0x7f022c1f1d90 2023-06-10T18:45:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.307+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f022c074050 con 0x7f022c1f1d90 2023-06-10T18:45:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.307+0000 7f021affd700 1 -- 172.21.15.104:0/3160866503 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f02240592a0 con 0x7f022c1f1d90 2023-06-10T18:45:05.313 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 msgr2=0x7f020c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 0x7f020c06d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02200010e0 tx=0x7f0220007980 comp rx=0 tx=0).stop 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 msgr2=0x7f022c1f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f021400ba40 tx=0x7f0224009750 comp rx=0 tx=0).stop 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 shutdown_connections 2023-06-10T18:45:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f020c06ace0 0x7f020c06d190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0214004d90 0x7f022c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f022c1f1d90 0x7f022c1f2200 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 --2- 172.21.15.104:0/3160866503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f022c070de0 0x7f022c071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 >> 172.21.15.104:0/3160866503 conn(0x7f022c0ceec0 msgr2=0x7f022c142990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 shutdown_connections 2023-06-10T18:45:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.309+0000 7f02308a5700 1 -- 172.21.15.104:0/3160866503 wait complete. 2023-06-10T18:45:05.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:05.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.444+0000 7f4e345b5700 1 Processor -- start 2023-06-10T18:45:05.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.444+0000 7f4e345b5700 1 -- start start 2023-06-10T18:45:05.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e345b5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e2c0cb420 0x7f4e2c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e345b5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 0x7f4e2c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e345b5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c15a700 con 0x7f4e2c15a330 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e345b5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c0ac8d0 con 0x7f4e2c0cb420 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e345b5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c06d7b0 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e31b50700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e2c0cb420 0x7f4e2c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e31b50700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e2c0cb420 0x7f4e2c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55994/0 (socket says 172.21.15.104:55994) 2023-06-10T18:45:05.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.445+0000 7f4e31b50700 1 -- 172.21.15.104:0/311924667 learned_addr learned my addr 172.21.15.104:0/311924667 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e32b52700 1 --2- 172.21.15.104:0/311924667 >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 0x7f4e2c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e31b50700 1 -- 172.21.15.104:0/311924667 >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 msgr2=0x7f4e2c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e31b50700 1 --2- 172.21.15.104:0/311924667 >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 0x7f4e2c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e31b50700 1 -- 172.21.15.104:0/311924667 >> v1:172.21.15.114:6789/0 conn(0x7f4e2c15a330 legacy=0x7f4e2c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e31b50700 1 -- 172.21.15.104:0/311924667 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0cbe00 con 0x7f4e2c0cb420 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e32b52700 1 --2- 172.21.15.104:0/311924667 >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 0x7f4e2c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e31b50700 1 --2- 172.21.15.104:0/311924667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb420 0x7f4e2c0b1070 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0670c0 tx=0x7f4e28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc25149f7c48f67f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e3134f700 1 -- 172.21.15.104:0/311924667 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28004230 con 0x7f4e2c0cb420 2023-06-10T18:45:05.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e3134f700 1 -- 172.21.15.104:0/311924667 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e28004390 con 0x7f4e2c0cb420 2023-06-10T18:45:05.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e3134f700 1 -- 172.21.15.104:0/311924667 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280036a0 con 0x7f4e2c0cb420 2023-06-10T18:45:05.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e345b5700 1 -- 172.21.15.104:0/311924667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb420 msgr2=0x7f4e2c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e345b5700 1 --2- 172.21.15.104:0/311924667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb420 0x7f4e2c0b1070 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0670c0 tx=0x7f4e28009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e345b5700 1 -- 172.21.15.104:0/311924667 shutdown_connections 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e345b5700 1 --2- 172.21.15.104:0/311924667 >> v2:172.21.15.114:3300/0 conn(0x7f4e2c0cb8c0 0x7f4e2c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.446+0000 7f4e345b5700 1 --2- 172.21.15.104:0/311924667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb420 0x7f4e2c0b1070 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.447+0000 7f4e345b5700 1 -- 172.21.15.104:0/311924667 >> 172.21.15.104:0/311924667 conn(0x7f4e2c06b8a0 msgr2=0x7f4e2c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.447+0000 7f4e345b5700 1 -- 172.21.15.104:0/311924667 shutdown_connections 2023-06-10T18:45:05.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.447+0000 7f4e345b5700 1 -- 172.21.15.104:0/311924667 wait complete. 2023-06-10T18:45:05.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 Processor -- start 2023-06-10T18:45:05.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 -- start start 2023-06-10T18:45:05.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e2c0cb420 0x7f4e2c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e2c15a330 0x7f4e2c151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1f0d00 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1f00f0 con 0x7f4e2c15a330 2023-06-10T18:45:05.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.448+0000 7f4e345b5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1f03d0 con 0x7f4e2c0cb420 2023-06-10T18:45:05.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56000/0 (socket says 172.21.15.104:56000) 2023-06-10T18:45:05.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 -- 172.21.15.104:0/1998531906 learned_addr learned my addr 172.21.15.104:0/1998531906 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:05.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e32b52700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e2c15a330 0x7f4e2c151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e32351700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e2c0cb420 0x7f4e2c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 -- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e2c0cb420 msgr2=0x7f4e2c154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e2c0cb420 0x7f4e2c154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 -- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e2c15a330 msgr2=0x7f4e2c151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e2c15a330 0x7f4e2c151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 -- 172.21.15.104:0/1998531906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009550 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e31b50700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009130 tx=0x7f4e28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28011830 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e28011990 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c1f06b0 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28011d40 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.449+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c1550a0 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.450+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c1f0e70 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.452+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4e2801b440 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.452+0000 7f4e22ffd700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 0x7f4e1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.452+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4e2808aea0 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.452+0000 7f4e32351700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 0x7f4e1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.452+0000 7f4e32351700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 0x7f4e1006d190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c003e90 tx=0x7f4e1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.456+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4e28055c30 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.642+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e2c074050 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.642+0000 7f4e22ffd700 1 -- 172.21.15.104:0/1998531906 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e28059250 con 0x7f4e2c0cb8c0 2023-06-10T18:45:05.647 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 msgr2=0x7f4e1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 0x7f4e1006d190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c003e90 tx=0x7f4e1c009040 comp rx=0 tx=0).stop 2023-06-10T18:45:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 msgr2=0x7f4e2c1ef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009130 tx=0x7f4e28004750 comp rx=0 tx=0).stop 2023-06-10T18:45:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 shutdown_connections 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e1006ace0 0x7f4e1006d190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e2c15a330 0x7f4e2c151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e2c0cb420 0x7f4e2c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 --2- 172.21.15.104:0/1998531906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e2c0cb8c0 0x7f4e2c1ef760 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 >> 172.21.15.104:0/1998531906 conn(0x7f4e2c06b8a0 msgr2=0x7f4e2c158bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.644+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 shutdown_connections 2023-06-10T18:45:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.645+0000 7f4e345b5700 1 -- 172.21.15.104:0/1998531906 wait complete. 2023-06-10T18:45:05.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.781+0000 7f40b7ba7700 1 Processor -- start 2023-06-10T18:45:05.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 -- start start 2023-06-10T18:45:05.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40b015d1a0 0x7f40b00cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b00b21c0 0x7f40b00b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00b2b20 con 0x7f40b0156760 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00b30a0 con 0x7f40b015d1a0 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.782+0000 7f40b7ba7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b015d740 con 0x7f40b00b21c0 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40b015d1a0 0x7f40b00cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5943700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f40b0156760 0x7f40b00ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53834/0 (socket says 172.21.15.104:53834) 2023-06-10T18:45:05.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5943700 1 -- 172.21.15.104:0/894113218 learned_addr learned my addr 172.21.15.104:0/894113218 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b6144700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b00b21c0 0x7f40b00b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 -- 172.21.15.104:0/894113218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b00b21c0 msgr2=0x7f40b00b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b00b21c0 0x7f40b00b25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 -- 172.21.15.104:0/894113218 >> v1:172.21.15.104:6789/0 conn(0x7f40b0156760 legacy=0x7f40b00ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 -- 172.21.15.104:0/894113218 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b00cfca0 con 0x7f40b015d1a0 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b5142700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b015d1a0 0x7f40b00cad90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009840 tx=0x7f40ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2db259dd7d5f7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b4941700 1 -- 172.21.15.104:0/894113218 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac011830 con 0x7f40b015d1a0 2023-06-10T18:45:05.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b4941700 1 -- 172.21.15.104:0/894113218 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac011990 con 0x7f40b015d1a0 2023-06-10T18:45:05.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.783+0000 7f40b4941700 1 -- 172.21.15.104:0/894113218 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac011d20 con 0x7f40b015d1a0 2023-06-10T18:45:05.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 -- 172.21.15.104:0/894113218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b015d1a0 msgr2=0x7f40b00cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b015d1a0 0x7f40b00cad90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009840 tx=0x7f40ac009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 -- 172.21.15.104:0/894113218 shutdown_connections 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b00b21c0 0x7f40b00b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/894113218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b015d1a0 0x7f40b00cad90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 -- 172.21.15.104:0/894113218 >> 172.21.15.104:0/894113218 conn(0x7f40b006e370 msgr2=0x7f40b006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 -- 172.21.15.104:0/894113218 shutdown_connections 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.784+0000 7f40b7ba7700 1 -- 172.21.15.104:0/894113218 wait complete. 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.785+0000 7f40b7ba7700 1 Processor -- start 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.785+0000 7f40b7ba7700 1 -- start start 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 0x7f40b014dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b015d1a0 0x7f40b00b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00ca310 con 0x7f40b00b21c0 2023-06-10T18:45:05.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b014ef80 con 0x7f40b0156760 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b014e6e0 con 0x7f40b015d1a0 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56012/0 (socket says 172.21.15.104:56012) 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 -- 172.21.15.104:0/1261117241 learned_addr learned my addr 172.21.15.104:0/1261117241 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5142700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 0x7f40b014dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b6144700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b015d1a0 0x7f40b00b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 -- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b015d1a0 msgr2=0x7f40b00b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b015d1a0 0x7f40b00b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 -- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 msgr2=0x7f40b014dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 0x7f40b014dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 -- 172.21.15.104:0/1261117241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0094a0 con 0x7f40b00b21c0 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5142700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 0x7f40b014dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b5943700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c490 tx=0x7f40a000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0005d70 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a00070d0 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b014e9c0 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0007460 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.786+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b0152dc0 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.787+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b0074050 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.788+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f40a00075c0 con 0x7f40b00b21c0 2023-06-10T18:45:05.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.788+0000 7f40a67fc700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 0x7f409406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:05.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.788+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f40a0088ce0 con 0x7f40b00b21c0 2023-06-10T18:45:05.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.788+0000 7f40b5142700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 0x7f409406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:05.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.789+0000 7f40b5142700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 0x7f409406d190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40b00b6690 tx=0x7f40ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:05.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.792+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40a0053a60 con 0x7f40b00b21c0 2023-06-10T18:45:05.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.976+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40b0070340 con 0x7f40b00b21c0 2023-06-10T18:45:05.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.977+0000 7f40a67fc700 1 -- 172.21.15.104:0/1261117241 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40a00170a0 con 0x7f40b00b21c0 2023-06-10T18:45:05.982 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 msgr2=0x7f409406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 0x7f409406d190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40b00b6690 tx=0x7f40ac008030 comp rx=0 tx=0).stop 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 msgr2=0x7f40b00c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c490 tx=0x7f40a000c850 comp rx=0 tx=0).stop 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 shutdown_connections 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40b015d1a0 0x7f40b00b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40b00b21c0 0x7f40b00c9dd0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f409406ace0 0x7f409406d190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 --2- 172.21.15.104:0/1261117241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b0156760 0x7f40b014dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:05.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 >> 172.21.15.104:0/1261117241 conn(0x7f40b006e370 msgr2=0x7f40b00cb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:05.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 shutdown_connections 2023-06-10T18:45:05.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:05.979+0000 7f40b7ba7700 1 -- 172.21.15.104:0/1261117241 wait complete. 2023-06-10T18:45:05.996 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:06.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.115+0000 7f1a634af700 1 Processor -- start 2023-06-10T18:45:06.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.115+0000 7f1a634af700 1 -- start start 2023-06-10T18:45:06.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a634af700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a5c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a634af700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c06e250 con 0x7f1a5c15c580 2023-06-10T18:45:06.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a634af700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c15c950 con 0x7f1a5c06cde0 2023-06-10T18:45:06.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a634af700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b1e80 con 0x7f1a5c06e6e0 2023-06-10T18:45:06.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a6124b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1a5c15c580 0x7f1a5c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53848/0 (socket says 172.21.15.104:53848) 2023-06-10T18:45:06.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a6124b700 1 -- 172.21.15.104:0/751804863 learned_addr learned my addr 172.21.15.104:0/751804863 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:06.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853764705 0 0) 0x7f1a5c06e250 con 0x7f1a5c15c580 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.116+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a480035b0 con 0x7f1a5c15c580 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a60a4a700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a5c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736645119 0 0) 0x7f1a480035b0 con 0x7f1a5c15c580 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 msgr2=0x7f1a5c0b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a5c0b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 >> v1:172.21.15.114:6789/0 conn(0x7f1a5c06cde0 legacy=0x7f1a5c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c06d920 con 0x7f1a5c15c580 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f1a4c0017f0 con 0x7f1a5c15c580 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 >> v1:172.21.15.104:6789/0 conn(0x7f1a5c15c580 legacy=0x7f1a5c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a48003a30 0x7f1a48003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a48004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a48004d90 0x7f1a48005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b1e80 con 0x7f1a48004d90 2023-06-10T18:45:06.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800bb20 con 0x7f1a48003a30 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4800bce0 con 0x7f1a5c06e6e0 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a48004d90 0x7f1a48005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a6124b700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a48003a30 0x7f1a48003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a60a4a700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a48004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 -- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 msgr2=0x7f1a48004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06e6e0 0x7f1a48004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 -- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a48003a30 msgr2=0x7f1a48003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a48003a30 0x7f1a48003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 -- 172.21.15.104:0/751804863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480035b0 con 0x7f1a48004d90 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.117+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a48004d90 0x7f1a48005200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c190 tx=0x7f1a48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01f7425418a2950 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54001db0 con 0x7f1a48004d90 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5400ccd0 con 0x7f1a48004d90 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a53fff700 1 -- 172.21.15.104:0/751804863 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54003c40 con 0x7f1a48004d90 2023-06-10T18:45:06.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 -- 172.21.15.104:0/751804863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a48004d90 msgr2=0x7f1a48005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 --2- 172.21.15.104:0/751804863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a48004d90 0x7f1a48005200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c190 tx=0x7f1a48005880 comp rx=0 tx=0).stop 2023-06-10T18:45:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a61a4c700 1 -- 172.21.15.104:0/751804863 reap_dead start 2023-06-10T18:45:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 -- 172.21.15.104:0/751804863 shutdown_connections 2023-06-10T18:45:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 -- 172.21.15.104:0/751804863 >> 172.21.15.104:0/751804863 conn(0x7f1a5c06b8a0 msgr2=0x7f1a5c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:06.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 -- 172.21.15.104:0/751804863 shutdown_connections 2023-06-10T18:45:06.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.118+0000 7f1a634af700 1 -- 172.21.15.104:0/751804863 wait complete. 2023-06-10T18:45:06.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.119+0000 7f1a634af700 1 Processor -- start 2023-06-10T18:45:06.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.119+0000 7f1a634af700 1 -- start start 2023-06-10T18:45:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c155af0 0x7f1a5c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c0ae530 0x7f1a5c157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c158100 con 0x7f1a5c157c60 2023-06-10T18:45:06.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b2d50 con 0x7f1a5c0ae530 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a634af700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b3030 con 0x7f1a5c155af0 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56034/0 (socket says 172.21.15.104:56034) 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 -- 172.21.15.104:0/2531570968 learned_addr learned my addr 172.21.15.104:0/2531570968 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a60a4a700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c0ae530 0x7f1a5c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 -- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c155af0 msgr2=0x7f1a5c0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c155af0 0x7f1a5c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 -- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c0ae530 msgr2=0x7f1a5c157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c0ae530 0x7f1a5c157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.120+0000 7f1a61a4c700 1 -- 172.21.15.104:0/2531570968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b3310 con 0x7f1a5c157c60 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a61a4c700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1a54015040 tx=0x7f1a540014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54001200 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a540050d0 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54005280 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0b35f0 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.121+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c155ec0 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.122+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c074050 con 0x7f1a5c157c60 2023-06-10T18:45:06.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.123+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a54019040 con 0x7f1a5c157c60 2023-06-10T18:45:06.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.123+0000 7f1a51ffb700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.123+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1a5408b090 con 0x7f1a5c157c60 2023-06-10T18:45:06.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.123+0000 7f1a6124b700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.124+0000 7f1a6124b700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0011d0 tx=0x7f1a4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.127+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1a540560a0 con 0x7f1a5c157c60 2023-06-10T18:45:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.314+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1a5c0cc480 con 0x7f1a5c157c60 2023-06-10T18:45:06.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.314+0000 7f1a51ffb700 1 -- 172.21.15.104:0/2531570968 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1a540596c0 con 0x7f1a5c157c60 2023-06-10T18:45:06.319 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:06.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 msgr2=0x7f1a4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0011d0 tx=0x7f1a4c009040 comp rx=0 tx=0).stop 2023-06-10T18:45:06.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 msgr2=0x7f1a5c0ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1a54015040 tx=0x7f1a540014a0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 shutdown_connections 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c157c60 0x7f1a5c0ac1c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c155af0 0x7f1a5c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 --2- 172.21.15.104:0/2531570968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c0ae530 0x7f1a5c157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 >> 172.21.15.104:0/2531570968 conn(0x7f1a5c06b8a0 msgr2=0x7f1a5c159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 shutdown_connections 2023-06-10T18:45:06.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.316+0000 7f1a634af700 1 -- 172.21.15.104:0/2531570968 wait complete. 2023-06-10T18:45:06.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:06.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.450+0000 7fc6d629a700 1 Processor -- start 2023-06-10T18:45:06.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.450+0000 7fc6d629a700 1 -- start start 2023-06-10T18:45:06.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6d629a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6d015a420 0x7fc6d00b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6d629a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d00b4ff0 0x7fc6d00ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6d629a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d00b6070 con 0x7fc6d00ce890 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6d629a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d00ced10 con 0x7fc6d015a420 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6d629a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00b5460 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6d015a420 0x7fc6d00b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6d015a420 0x7fc6d00b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56040/0 (socket says 172.21.15.104:56040) 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 -- 172.21.15.104:0/258153900 learned_addr learned my addr 172.21.15.104:0/258153900 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cf7fe700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d00b4ff0 0x7fc6d00ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 -- 172.21.15.104:0/258153900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d00b4ff0 msgr2=0x7fc6d00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d00b4ff0 0x7fc6d00ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 -- 172.21.15.104:0/258153900 >> v1:172.21.15.114:6789/0 conn(0x7fc6d00ce890 legacy=0x7fc6d00b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 -- 172.21.15.104:0/258153900 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00b5740 con 0x7fc6d015a420 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6cffff700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d015a420 0x7fc6d00b1660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40098f0 tx=0x7fc6c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb76bd1e732f912 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6ceffd700 1 -- 172.21.15.104:0/258153900 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400b8a0 con 0x7fc6d015a420 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6ceffd700 1 -- 172.21.15.104:0/258153900 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c400ba00 con 0x7fc6d015a420 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.451+0000 7fc6ceffd700 1 -- 172.21.15.104:0/258153900 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400bd90 con 0x7fc6d015a420 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 -- 172.21.15.104:0/258153900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d015a420 msgr2=0x7fc6d00b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d015a420 0x7fc6d00b1660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40098f0 tx=0x7fc6c4009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 -- 172.21.15.104:0/258153900 shutdown_connections 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d015a420 0x7fc6d00b1660 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 --2- 172.21.15.104:0/258153900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d00b4ff0 0x7fc6d00ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 -- 172.21.15.104:0/258153900 >> 172.21.15.104:0/258153900 conn(0x7fc6d0089fd0 msgr2=0x7fc6d00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:06.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 -- 172.21.15.104:0/258153900 shutdown_connections 2023-06-10T18:45:06.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.452+0000 7fc6d629a700 1 -- 172.21.15.104:0/258153900 wait complete. 2023-06-10T18:45:06.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.453+0000 7fc6d629a700 1 Processor -- start 2023-06-10T18:45:06.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.454+0000 7fc6d629a700 1 -- start start 2023-06-10T18:45:06.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d00ce890 0x7fc6d01e1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d015a420 0x7fc6d01e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c6580 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c5970 con 0x7fc6d00ce890 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c5c50 con 0x7fc6d015a420 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56044/0 (socket says 172.21.15.104:56044) 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 -- 172.21.15.104:0/540677235 learned_addr learned my addr 172.21.15.104:0/540677235 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:06.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cf7fe700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d00ce890 0x7fc6d01e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 -- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d015a420 msgr2=0x7fc6d01e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d4837700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d015a420 0x7fc6d01e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d015a420 0x7fc6d01e1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 -- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d00ce890 msgr2=0x7fc6d01e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d00ce890 0x7fc6d01e1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 -- 172.21.15.104:0/540677235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4009550 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6cffff700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4000c00 tx=0x7fc6c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4004480 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c401f070 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d00c5f30 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40038a0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.455+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d00c6a10 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.456+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d00b64e0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.457+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc6c4021030 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.457+0000 7fc6ccff9700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 0x7fc6bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.457+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc6c408ace0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.457+0000 7fc6cf7fe700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 0x7fc6bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.458+0000 7fc6cf7fe700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 0x7fc6bc06d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc6d01e3760 tx=0x7fc6c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.461+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc6d00b64e0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.666+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc6d00b64e0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.666+0000 7fc6ccff9700 1 -- 172.21.15.104:0/540677235 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc6d00b64e0 con 0x7fc6d00b4ff0 2023-06-10T18:45:06.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 msgr2=0x7fc6bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 0x7fc6bc06d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc6d01e3760 tx=0x7fc6c0008040 comp rx=0 tx=0).stop 2023-06-10T18:45:06.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 msgr2=0x7fc6d00c5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4000c00 tx=0x7fc6c4004750 comp rx=0 tx=0).stop 2023-06-10T18:45:06.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 shutdown_connections 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6d015a420 0x7fc6d01e1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6d00b4ff0 0x7fc6d00c5300 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6bc06ace0 0x7fc6bc06d190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 --2- 172.21.15.104:0/540677235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d00ce890 0x7fc6d01e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 >> 172.21.15.104:0/540677235 conn(0x7fc6d0089fd0 msgr2=0x7fc6d0072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 shutdown_connections 2023-06-10T18:45:06.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.668+0000 7fc6d629a700 1 -- 172.21.15.104:0/540677235 wait complete. 2023-06-10T18:45:06.678 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:06.685 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:06.685 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:06.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:06.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.857+0000 7f205ad0f700 1 Processor -- start 2023-06-10T18:45:06.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.857+0000 7f205ad0f700 1 -- start start 2023-06-10T18:45:06.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f205ad0f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f205406b1f0 0x7f20540cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f205ad0f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 0x7f20540cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f205ad0f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540cadd0 con 0x7f205408a3d0 2023-06-10T18:45:06.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f205ad0f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20540cfeb0 con 0x7f205406b1f0 2023-06-10T18:45:06.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f205ad0f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20540cb0b0 con 0x7f20540ce350 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f205406b1f0 0x7f20540cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f205406b1f0 0x7f20540cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56052/0 (socket says 172.21.15.104:56052) 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 -- 172.21.15.104:0/2329610724 learned_addr learned my addr 172.21.15.104:0/2329610724 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2053fff700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 0x7f20540cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 -- 172.21.15.104:0/2329610724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 msgr2=0x7f20540cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 0x7f20540cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 -- 172.21.15.104:0/2329610724 >> v1:172.21.15.114:6789/0 conn(0x7f205408a3d0 legacy=0x7f2054160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 -- 172.21.15.104:0/2329610724 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540cf550 con 0x7f205406b1f0 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2053fff700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 0x7f20540cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f2058aab700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20540cd090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20480098f0 tx=0x7f2048009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824e1783a09deb1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.858+0000 7f20537fe700 1 -- 172.21.15.104:0/2329610724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048004230 con 0x7f205406b1f0 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f20537fe700 1 -- 172.21.15.104:0/2329610724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2048004390 con 0x7f205406b1f0 2023-06-10T18:45:06.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f20537fe700 1 -- 172.21.15.104:0/2329610724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20480036a0 con 0x7f205406b1f0 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 -- 172.21.15.104:0/2329610724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 msgr2=0x7f20540cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20540cd090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20480098f0 tx=0x7f2048009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 -- 172.21.15.104:0/2329610724 shutdown_connections 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20540cd090 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 --2- 172.21.15.104:0/2329610724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20540ce350 0x7f20540cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 -- 172.21.15.104:0/2329610724 >> 172.21.15.104:0/2329610724 conn(0x7f2054089fd0 msgr2=0x7f20540d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:06.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 -- 172.21.15.104:0/2329610724 shutdown_connections 2023-06-10T18:45:06.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.859+0000 7f205ad0f700 1 -- 172.21.15.104:0/2329610724 wait complete. 2023-06-10T18:45:06.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.860+0000 7f205ad0f700 1 Processor -- start 2023-06-10T18:45:06.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.860+0000 7f205ad0f700 1 -- start start 2023-06-10T18:45:06.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f205408a3d0 0x7f20540c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20540ce350 0x7f205406f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2054160030 con 0x7f205406b1f0 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540c88d0 con 0x7f20540ce350 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f205ad0f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20540c7cc0 con 0x7f205408a3d0 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f2058aab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f2058aab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56058/0 (socket says 172.21.15.104:56058) 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.861+0000 7f2058aab700 1 -- 172.21.15.104:0/3023200368 learned_addr learned my addr 172.21.15.104:0/3023200368 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 -- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f205408a3d0 msgr2=0x7f20540c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f205408a3d0 0x7f20540c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 -- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20540ce350 msgr2=0x7f205406f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20540ce350 0x7f205406f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 -- 172.21.15.104:0/3023200368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2048009550 con 0x7f205406b1f0 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f2058aab700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f204800b0e0 tx=0x7f2048004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048003e30 con 0x7f205406b1f0 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204801f070 con 0x7f205406b1f0 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20540c7fa0 con 0x7f205406b1f0 2023-06-10T18:45:06.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800b8a0 con 0x7f205406b1f0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.862+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20540c8560 con 0x7f205406b1f0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.863+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2054074050 con 0x7f205406b1f0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.863+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2048021030 con 0x7f205406b1f0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.864+0000 7f20517fa700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 0x7f203806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.864+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f204808ae20 con 0x7f205406b1f0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.864+0000 7f2053fff700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 0x7f203806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:06.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.865+0000 7f2053fff700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 0x7f203806d190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20541e6850 tx=0x7f2040008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:06.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:06.868+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2048055d50 con 0x7f205406b1f0 2023-06-10T18:45:07.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.054+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2054074050 con 0x7f205406b1f0 2023-06-10T18:45:07.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.055+0000 7f20517fa700 1 -- 172.21.15.104:0/3023200368 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f2048059370 con 0x7f205406b1f0 2023-06-10T18:45:07.060 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:07.061 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:07.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 msgr2=0x7f203806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 0x7f203806d190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20541e6850 tx=0x7f2040008040 comp rx=0 tx=0).stop 2023-06-10T18:45:07.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 msgr2=0x7f20541e5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f204800b0e0 tx=0x7f2048004750 comp rx=0 tx=0).stop 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 shutdown_connections 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203806ace0 0x7f203806d190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20540ce350 0x7f205406f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f205406b1f0 0x7f20541e5d30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.057+0000 7f205ad0f700 1 --2- 172.21.15.104:0/3023200368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f205408a3d0 0x7f20540c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.058+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 >> 172.21.15.104:0/3023200368 conn(0x7f2054089fd0 msgr2=0x7f20540ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.058+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 shutdown_connections 2023-06-10T18:45:07.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.058+0000 7f205ad0f700 1 -- 172.21.15.104:0/3023200368 wait complete. 2023-06-10T18:45:07.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:07.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:07.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.194+0000 7f948393d700 1 Processor -- start 2023-06-10T18:45:07.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 -- start start 2023-06-10T18:45:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f947c1545a0 0x7f947c15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 0x7f947c15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c15dab0 con 0x7f947c1545a0 2023-06-10T18:45:07.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c15aa00 con 0x7f947c0af940 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f948393d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c154970 con 0x7f947c15e720 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.195+0000 7f94816d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f947c1545a0 0x7f947c15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f947c1545a0 0x7f947c15a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56066/0 (socket says 172.21.15.104:56066) 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 -- 172.21.15.104:0/1274114716 learned_addr learned my addr 172.21.15.104:0/1274114716 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9481eda700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9480ed8700 1 --2- 172.21.15.104:0/1274114716 >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 0x7f947c15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 -- 172.21.15.104:0/1274114716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 msgr2=0x7f947c0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 -- 172.21.15.104:0/1274114716 >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 msgr2=0x7f947c15e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 --2- 172.21.15.104:0/1274114716 >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 0x7f947c15e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 -- 172.21.15.104:0/1274114716 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0cb0b0 con 0x7f947c1545a0 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9480ed8700 1 --2- 172.21.15.104:0/1274114716 >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 0x7f947c15e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:07.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f94816d9700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c15a4c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f947c0ce2a0 tx=0x7f946c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817c0edae9786b05 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9473fff700 1 -- 172.21.15.104:0/1274114716 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c011830 con 0x7f947c1545a0 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9473fff700 1 -- 172.21.15.104:0/1274114716 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c011990 con 0x7f947c1545a0 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.196+0000 7f9473fff700 1 -- 172.21.15.104:0/1274114716 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c011d20 con 0x7f947c1545a0 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 -- 172.21.15.104:0/1274114716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 msgr2=0x7f947c15a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c15a4c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f947c0ce2a0 tx=0x7f946c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 -- 172.21.15.104:0/1274114716 shutdown_connections 2023-06-10T18:45:07.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c15a4c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 --2- 172.21.15.104:0/1274114716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 --2- 172.21.15.104:0/1274114716 >> v2:172.21.15.114:3300/0 conn(0x7f947c0af940 0x7f947c15e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 -- 172.21.15.104:0/1274114716 >> 172.21.15.104:0/1274114716 conn(0x7f947c06e370 msgr2=0x7f947c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 -- 172.21.15.104:0/1274114716 shutdown_connections 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.197+0000 7f948393d700 1 -- 172.21.15.104:0/1274114716 wait complete. 2023-06-10T18:45:07.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.198+0000 7f948393d700 1 Processor -- start 2023-06-10T18:45:07.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.198+0000 7f948393d700 1 -- start start 2023-06-10T18:45:07.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947c0af940 0x7f947c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0b0ff0 con 0x7f947c1545a0 2023-06-10T18:45:07.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c1efb20 con 0x7f947c0af940 2023-06-10T18:45:07.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c1efe00 con 0x7f947c15e720 2023-06-10T18:45:07.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56074/0 (socket says 172.21.15.104:56074) 2023-06-10T18:45:07.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 -- 172.21.15.104:0/4167320952 learned_addr learned my addr 172.21.15.104:0/4167320952 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f94816d9700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947c0af940 0x7f947c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9481eda700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 -- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 msgr2=0x7f947c0c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 -- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947c0af940 msgr2=0x7f947c0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947c0af940 0x7f947c0b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 -- 172.21.15.104:0/4167320952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c009550 con 0x7f947c1545a0 2023-06-10T18:45:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9481eda700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9480ed8700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f947800d800 tx=0x7f947800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478004030 con 0x7f947c1545a0 2023-06-10T18:45:07.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9478004190 con 0x7f947c1545a0 2023-06-10T18:45:07.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478004520 con 0x7f947c1545a0 2023-06-10T18:45:07.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.199+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c1f00e0 con 0x7f947c1545a0 2023-06-10T18:45:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.200+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c1f08e0 con 0x7f947c1545a0 2023-06-10T18:45:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.200+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c074050 con 0x7f947c1545a0 2023-06-10T18:45:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.201+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94780036a0 con 0x7f947c1545a0 2023-06-10T18:45:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.201+0000 7f9471ffb700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 0x7f946006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.201+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9478088d30 con 0x7f947c1545a0 2023-06-10T18:45:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.202+0000 7f94816d9700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 0x7f946006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.202+0000 7f94816d9700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 0x7f946006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f946c014040 tx=0x7f946c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.206+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9478057680 con 0x7f947c1545a0 2023-06-10T18:45:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.392+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f947c070340 con 0x7f947c1545a0 2023-06-10T18:45:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.392+0000 7f9471ffb700 1 -- 172.21.15.104:0/4167320952 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94780180a0 con 0x7f947c1545a0 2023-06-10T18:45:07.397 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:07.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 msgr2=0x7f946006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 0x7f946006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f946c014040 tx=0x7f946c008030 comp rx=0 tx=0).stop 2023-06-10T18:45:07.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 msgr2=0x7f947c0b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f947800d800 tx=0x7f947800db10 comp rx=0 tx=0).stop 2023-06-10T18:45:07.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 shutdown_connections 2023-06-10T18:45:07.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f946006ace0 0x7f946006d190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947c0af940 0x7f947c0b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947c15e720 0x7f947c0c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 --2- 172.21.15.104:0/4167320952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947c1545a0 0x7f947c0b0ab0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 >> 172.21.15.104:0/4167320952 conn(0x7f947c06e370 msgr2=0x7f947c0cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 shutdown_connections 2023-06-10T18:45:07.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.394+0000 7f948393d700 1 -- 172.21.15.104:0/4167320952 wait complete. 2023-06-10T18:45:07.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:07.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.529+0000 7fee76e4c700 1 Processor -- start 2023-06-10T18:45:07.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.529+0000 7fee76e4c700 1 -- start start 2023-06-10T18:45:07.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee76e4c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee700d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee76e4c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700ce340 con 0x7fee700ccff0 2023-06-10T18:45:07.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee76e4c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700d00d0 con 0x7fee700ce7d0 2023-06-10T18:45:07.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee76e4c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700cd460 con 0x7fee7006af80 2023-06-10T18:45:07.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fee700ccff0 0x7fee7006b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53912/0 (socket says 172.21.15.104:53912) 2023-06-10T18:45:07.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6ffff700 1 -- 172.21.15.104:0/2301207484 learned_addr learned my addr 172.21.15.104:0/2301207484 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee74be8700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee700d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825260405 0 0) 0x7fee700ce340 con 0x7fee700ccff0 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee600035b0 con 0x7fee700ccff0 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854324995 0 0) 0x7fee600035b0 con 0x7fee700ccff0 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 msgr2=0x7fee700d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee700d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 >> v1:172.21.15.114:6789/0 conn(0x7fee700ce7d0 legacy=0x7fee700cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700cd680 con 0x7fee700ccff0 2023-06-10T18:45:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.530+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fee5c001670 con 0x7fee700ccff0 2023-06-10T18:45:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 >> v1:172.21.15.104:6789/0 conn(0x7fee700ccff0 legacy=0x7fee7006b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60003a30 0x7fee60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee60004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee60004d90 0x7fee60005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700cd460 con 0x7fee60004d90 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6000ba10 con 0x7fee60003a30 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6000bbd0 con 0x7fee7006af80 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee60004d90 0x7fee60005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee74be8700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60003a30 0x7fee60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6ffff700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee60004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 -- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 msgr2=0x7fee60004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee7006af80 0x7fee60004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 -- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60003a30 msgr2=0x7fee60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60003a30 0x7fee60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 -- 172.21.15.104:0/2301207484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600035b0 con 0x7fee60004d90 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee753e9700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee60004d90 0x7fee60005200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c290 tx=0x7fee60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f4f713e13f0b3d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68001e70 con 0x7fee60004d90 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6800ce50 con 0x7fee60004d90 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.531+0000 7fee6f7fe700 1 -- 172.21.15.104:0/2301207484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68012660 con 0x7fee60004d90 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 -- 172.21.15.104:0/2301207484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee60004d90 msgr2=0x7fee60005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 --2- 172.21.15.104:0/2301207484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee60004d90 0x7fee60005200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c290 tx=0x7fee60005880 comp rx=0 tx=0).stop 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee753e9700 1 -- 172.21.15.104:0/2301207484 reap_dead start 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 -- 172.21.15.104:0/2301207484 shutdown_connections 2023-06-10T18:45:07.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 -- 172.21.15.104:0/2301207484 >> 172.21.15.104:0/2301207484 conn(0x7fee70089fd0 msgr2=0x7fee7008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 -- 172.21.15.104:0/2301207484 shutdown_connections 2023-06-10T18:45:07.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.532+0000 7fee76e4c700 1 -- 172.21.15.104:0/2301207484 wait complete. 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.533+0000 7fee76e4c700 1 Processor -- start 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.533+0000 7fee76e4c700 1 -- start start 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.534+0000 7fee76e4c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60004d90 0x7fee700c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.534+0000 7fee76e4c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.534+0000 7fee76e4c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee700b1480 0x7fee700b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee76e4c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7015a7b0 con 0x7fee70153780 2023-06-10T18:45:07.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee76e4c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700b1e30 con 0x7fee60004d90 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee76e4c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee70154000 con 0x7fee700b1480 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56092/0 (socket says 172.21.15.104:56092) 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 -- 172.21.15.104:0/3880432644 learned_addr learned my addr 172.21.15.104:0/3880432644 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee753e9700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee700b1480 0x7fee700b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 -- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee700b1480 msgr2=0x7fee700b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee74be8700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60004d90 0x7fee700c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee700b1480 0x7fee700b18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 -- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60004d90 msgr2=0x7fee700c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60004d90 0x7fee700c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 -- 172.21.15.104:0/3880432644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70158380 con 0x7fee70153780 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6ffff700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007320 tx=0x7fee5c007630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00f850 con 0x7fee70153780 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c00f9b0 con 0x7fee70153780 2023-06-10T18:45:07.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee700b0a30 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00fd60 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.535+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700b0fb0 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.536+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee70074050 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.537+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fee5c01e3b0 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.537+0000 7fee6d7fa700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 0x7fee5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.537+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fee5c089600 con 0x7fee70153780 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.538+0000 7fee74be8700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 0x7fee5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.538+0000 7fee74be8700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 0x7fee5406d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee64009aa0 tx=0x7fee64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.541+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fee5c054360 con 0x7fee70153780 2023-06-10T18:45:07.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:45:07.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:45:07.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.735+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fee700afc60 con 0x7fee70153780 2023-06-10T18:45:07.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.736+0000 7fee6d7fa700 1 -- 172.21.15.104:0/3880432644 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fee5c01a0a0 con 0x7fee70153780 2023-06-10T18:45:07.740 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:07.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.737+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 msgr2=0x7fee5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.737+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 0x7fee5406d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee64009aa0 tx=0x7fee64008040 comp rx=0 tx=0).stop 2023-06-10T18:45:07.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 msgr2=0x7fee7015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007320 tx=0x7fee5c007630 comp rx=0 tx=0).stop 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 shutdown_connections 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fee5406ace0 0x7fee5406d190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60004d90 0x7fee700c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee700b1480 0x7fee700b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 --2- 172.21.15.104:0/3880432644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee70153780 0x7fee7015a0b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 >> 172.21.15.104:0/3880432644 conn(0x7fee70089fd0 msgr2=0x7fee700ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 shutdown_connections 2023-06-10T18:45:07.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.738+0000 7fee76e4c700 1 -- 172.21.15.104:0/3880432644 wait complete. 2023-06-10T18:45:07.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:07.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.871+0000 7fe47ce2d700 1 Processor -- start 2023-06-10T18:45:07.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 -- start start 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 0x7fe4780cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4780cf200 0x7fe478069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47815c8d0 con 0x7fe47806a3b0 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4780aef20 con 0x7fe4780cf200 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.872+0000 7fe47ce2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47806abd0 con 0x7fe47815c500 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4780cf200 0x7fe478069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4780cf200 0x7fe478069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56104/0 (socket says 172.21.15.104:56104) 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 -- 172.21.15.104:0/501253936 learned_addr learned my addr 172.21.15.104:0/501253936 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe47659c700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 0x7fe4780cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 -- 172.21.15.104:0/501253936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 msgr2=0x7fe4780cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 0x7fe4780cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 -- 172.21.15.104:0/501253936 >> v1:172.21.15.114:6789/0 conn(0x7fe47806a3b0 legacy=0x7fe47806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 -- 172.21.15.104:0/501253936 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47806a8f0 con 0x7fe4780cf200 2023-06-10T18:45:07.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe47659c700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 0x7fe4780cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:07.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe475d9b700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4780cf200 0x7fe478069e70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098f0 tx=0x7fe468009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330eff006ed3af51 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe47559a700 1 -- 172.21.15.104:0/501253936 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800b8a0 con 0x7fe4780cf200 2023-06-10T18:45:07.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe47559a700 1 -- 172.21.15.104:0/501253936 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46800ba00 con 0x7fe4780cf200 2023-06-10T18:45:07.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.873+0000 7fe47559a700 1 -- 172.21.15.104:0/501253936 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800bd90 con 0x7fe4780cf200 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 -- 172.21.15.104:0/501253936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4780cf200 msgr2=0x7fe478069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4780cf200 0x7fe478069e70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098f0 tx=0x7fe468009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 -- 172.21.15.104:0/501253936 shutdown_connections 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47815c500 0x7fe4780cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/501253936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4780cf200 0x7fe478069e70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 -- 172.21.15.104:0/501253936 >> 172.21.15.104:0/501253936 conn(0x7fe47806b8a0 msgr2=0x7fe4780cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 -- 172.21.15.104:0/501253936 shutdown_connections 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.874+0000 7fe47ce2d700 1 -- 172.21.15.104:0/501253936 wait complete. 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.875+0000 7fe47ce2d700 1 Processor -- start 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.875+0000 7fe47ce2d700 1 -- start start 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 0x7fe4780b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4780cf200 0x7fe47809b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809cc40 con 0x7fe47815c500 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809bd60 con 0x7fe4780cf200 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47ce2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809c030 con 0x7fe47806a3b0 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56110/0 (socket says 172.21.15.104:56110) 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 -- 172.21.15.104:0/26544339 learned_addr learned my addr 172.21.15.104:0/26544339 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:07.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47659c700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 0x7fe4780b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe475d9b700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4780cf200 0x7fe47809b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 -- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 msgr2=0x7fe4780b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 0x7fe4780b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 -- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4780cf200 msgr2=0x7fe47809b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4780cf200 0x7fe47809b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 -- 172.21.15.104:0/26544339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009550 con 0x7fe47815c500 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe47659c700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 0x7fe4780b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:07.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.876+0000 7fe476d9d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe47000cb50 tx=0x7fe47000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.877+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470003c00 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.877+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470003d60 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.877+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47809c310 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.877+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47000ab70 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.877+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47809c8d0 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.878+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe478074050 con 0x7fe47815c500 2023-06-10T18:45:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.878+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe47000acd0 con 0x7fe47815c500 2023-06-10T18:45:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.878+0000 7fe466ffd700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 0x7fe45806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.878+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe470089850 con 0x7fe47815c500 2023-06-10T18:45:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.879+0000 7fe47659c700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 0x7fe45806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.879+0000 7fe47659c700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 0x7fe45806d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0099e0 tx=0x7fe46c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:07.883+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe4700545d0 con 0x7fe47815c500 2023-06-10T18:45:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.071+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe47806e740 con 0x7fe47815c500 2023-06-10T18:45:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.071+0000 7fe466ffd700 1 -- 172.21.15.104:0/26544339 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe470057bf0 con 0x7fe47815c500 2023-06-10T18:45:08.076 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 msgr2=0x7fe45806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 0x7fe45806d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0099e0 tx=0x7fe46c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 msgr2=0x7fe478156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe47000cb50 tx=0x7fe47000cf10 comp rx=0 tx=0).stop 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 shutdown_connections 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe47815c500 0x7fe478156480 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe47806a3b0 0x7fe4780b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe45806ace0 0x7fe45806d190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 --2- 172.21.15.104:0/26544339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4780cf200 0x7fe47809b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 >> 172.21.15.104:0/26544339 conn(0x7fe47806b8a0 msgr2=0x7fe4780af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 shutdown_connections 2023-06-10T18:45:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.073+0000 7fe47ce2d700 1 -- 172.21.15.104:0/26544339 wait complete. 2023-06-10T18:45:08.090 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:08.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 Processor -- start 2023-06-10T18:45:08.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 -- start start 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f12140b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12140ae7e0 con 0x7f12140aec00 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121406df90 con 0x7f121406ea70 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f121a773700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121406e270 con 0x7f121415c6f0 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f1218d10700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f12140aec00 0x7f12140ac8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53946/0 (socket says 172.21.15.104:53946) 2023-06-10T18:45:08.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.208+0000 7f1218d10700 1 -- 172.21.15.104:0/1687403605 learned_addr learned my addr 172.21.15.104:0/1687403605 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252023521 0 0) 0x7f12140ae7e0 con 0x7f12140aec00 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f12140b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12000035b0 con 0x7f12140aec00 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858024387 0 0) 0x7f12000035b0 con 0x7f12140aec00 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 msgr2=0x7f12140b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f12140b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 >> v1:172.21.15.114:6789/0 conn(0x7f121406ea70 legacy=0x7f12140ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121406e550 con 0x7f12140aec00 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f120c0019f0 con 0x7f12140aec00 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 >> v1:172.21.15.104:6789/0 conn(0x7f12140aec00 legacy=0x7f12140ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f1200003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1200004850 0x7f1200004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f1200005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121406e270 con 0x7f1200003a30 2023-06-10T18:45:08.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120000bb10 con 0x7f1200004850 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120000bcd0 con 0x7f121415c6f0 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f1200003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f12137fe700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1200004850 0x7f1200004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1218d10700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f1200005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 -- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 msgr2=0x7f1200005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f121415c6f0 0x7f1200005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 -- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1200004850 msgr2=0x7f1200004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1200004850 0x7f1200004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.209+0000 7f1213fff700 1 -- 172.21.15.104:0/1687403605 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000035b0 con 0x7f1200003a30 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f1213fff700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f1200003e40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1204009b40 tx=0x7f1200005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f7cbc4dee7be90 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204005a20 con 0x7f1200003a30 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1204005b80 con 0x7f1200003a30 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f1212ffd700 1 -- 172.21.15.104:0/1687403605 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120400e6c0 con 0x7f1200003a30 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 -- 172.21.15.104:0/1687403605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 msgr2=0x7f1200003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 --2- 172.21.15.104:0/1687403605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f1200003e40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1204009b40 tx=0x7f1200005880 comp rx=0 tx=0).stop 2023-06-10T18:45:08.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f1218d10700 1 -- 172.21.15.104:0/1687403605 reap_dead start 2023-06-10T18:45:08.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 -- 172.21.15.104:0/1687403605 shutdown_connections 2023-06-10T18:45:08.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 -- 172.21.15.104:0/1687403605 >> 172.21.15.104:0/1687403605 conn(0x7f121406b8a0 msgr2=0x7f12140cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 -- 172.21.15.104:0/1687403605 shutdown_connections 2023-06-10T18:45:08.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.210+0000 7f121a773700 1 -- 172.21.15.104:0/1687403605 wait complete. 2023-06-10T18:45:08.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.211+0000 7f121a773700 1 Processor -- start 2023-06-10T18:45:08.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 -- start start 2023-06-10T18:45:08.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 0x7f12140b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 0x7f12141521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140c5f40 con 0x7f1200003a30 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1214152700 con 0x7f12140c59f0 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.212+0000 7f121a773700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12140b2960 con 0x7f1214151d20 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56126/0 (socket says 172.21.15.104:56126) 2023-06-10T18:45:08.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 -- 172.21.15.104:0/3975118870 learned_addr learned my addr 172.21.15.104:0/3975118870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f12137fe700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 0x7f12140b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1218d10700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 0x7f12141521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 -- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 msgr2=0x7f12141521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 0x7f12141521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 -- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 msgr2=0x7f12140b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 0x7f12140b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 -- 172.21.15.104:0/3975118870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214153140 con 0x7f1200003a30 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1218d10700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 0x7f12141521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f12137fe700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 0x7f12140b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1213fff700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f120400a010 tx=0x7f12040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204021070 con 0x7f1200003a30 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1204005e30 con 0x7f1200003a30 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12140b0200 con 0x7f1200003a30 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204004480 con 0x7f1200003a30 2023-06-10T18:45:08.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.213+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12140b0780 con 0x7f1200003a30 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.214+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12140b0940 con 0x7f1200003a30 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.214+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12040036a0 con 0x7f1200003a30 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.215+0000 7f1210ff9700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 0x7f11f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.215+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f120408a710 con 0x7f1200003a30 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.215+0000 7f12137fe700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 0x7f11f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.215+0000 7f12137fe700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 0x7f11f406d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f12140ad590 tx=0x7f11fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.219+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1204055520 con 0x7f1200003a30 2023-06-10T18:45:08.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.404+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12140c30d0 con 0x7f1200003a30 2023-06-10T18:45:08.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.404+0000 7f1210ff9700 1 -- 172.21.15.104:0/3975118870 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1204058b40 con 0x7f1200003a30 2023-06-10T18:45:08.409 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 msgr2=0x7f11f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 0x7f11f406d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f12140ad590 tx=0x7f11fc009040 comp rx=0 tx=0).stop 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 msgr2=0x7f12140b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f120400a010 tx=0x7f12040047c0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 shutdown_connections 2023-06-10T18:45:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1214151d20 0x7f12141521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1200003a30 0x7f12140b2420 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f11f406ace0 0x7f11f406d190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 --2- 172.21.15.104:0/3975118870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12140c59f0 0x7f12140b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 >> 172.21.15.104:0/3975118870 conn(0x7f121406b8a0 msgr2=0x7f12140adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 shutdown_connections 2023-06-10T18:45:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.406+0000 7f121a773700 1 -- 172.21.15.104:0/3975118870 wait complete. 2023-06-10T18:45:08.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:08.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.542+0000 7f1fcd798700 1 Processor -- start 2023-06-10T18:45:08.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.542+0000 7f1fcd798700 1 -- start start 2023-06-10T18:45:08.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fcd798700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1fc80d25c0 0x7f1fc80d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fcd798700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1fc80b05d0 0x7f1fc80b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fcd798700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc80af530 con 0x7f1fc80d04f0 2023-06-10T18:45:08.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fcd798700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc80af810 con 0x7f1fc80d25c0 2023-06-10T18:45:08.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fcd798700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc80afaf0 con 0x7f1fc80b05d0 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1fc80d25c0 0x7f1fc80d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1fc80d25c0 0x7f1fc80d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56136/0 (socket says 172.21.15.104:56136) 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 -- 172.21.15.104:0/1058011540 learned_addr learned my addr 172.21.15.104:0/1058011540 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc77fe700 1 --2- 172.21.15.104:0/1058011540 >> v2:172.21.15.114:3300/0 conn(0x7f1fc80b05d0 0x7f1fc80b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 -- 172.21.15.104:0/1058011540 >> v2:172.21.15.114:3300/0 conn(0x7f1fc80b05d0 msgr2=0x7f1fc80b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 --2- 172.21.15.104:0/1058011540 >> v2:172.21.15.114:3300/0 conn(0x7f1fc80b05d0 0x7f1fc80b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 -- 172.21.15.104:0/1058011540 >> v1:172.21.15.114:6789/0 conn(0x7f1fc80d04f0 legacy=0x7f1fc80d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.543+0000 7f1fc67fc700 1 -- 172.21.15.104:0/1058011540 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80afdd0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fc67fc700 1 --2- 172.21.15.104:0/1058011540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80d34d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80c9970 tx=0x7f1fb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe5e9cdf71dd411 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fc5ffb700 1 -- 172.21.15.104:0/1058011540 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8004230 con 0x7f1fc80d25c0 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fc5ffb700 1 -- 172.21.15.104:0/1058011540 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb8004390 con 0x7f1fc80d25c0 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fc5ffb700 1 -- 172.21.15.104:0/1058011540 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb80036a0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 -- 172.21.15.104:0/1058011540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 msgr2=0x7f1fc80d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 --2- 172.21.15.104:0/1058011540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80d34d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80c9970 tx=0x7f1fb8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 -- 172.21.15.104:0/1058011540 shutdown_connections 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 --2- 172.21.15.104:0/1058011540 >> v2:172.21.15.114:3300/0 conn(0x7f1fc80b05d0 0x7f1fc80b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 --2- 172.21.15.104:0/1058011540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80d34d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 -- 172.21.15.104:0/1058011540 >> 172.21.15.104:0/1058011540 conn(0x7f1fc8071ec0 msgr2=0x7f1fc809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 -- 172.21.15.104:0/1058011540 shutdown_connections 2023-06-10T18:45:08.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.544+0000 7f1fcd798700 1 -- 172.21.15.104:0/1058011540 wait complete. 2023-06-10T18:45:08.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 Processor -- start 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 -- start start 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 0x7f1fc809d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1fc80d04f0 0x7f1fc81f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80b3060 con 0x7f1fc80d25c0 2023-06-10T18:45:08.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80b1bb0 con 0x7f1fc80b05d0 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.546+0000 7f1fcd798700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80b18d0 con 0x7f1fc80d04f0 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56144/0 (socket says 172.21.15.104:56144) 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 -- 172.21.15.104:0/814650668 learned_addr learned my addr 172.21.15.104:0/814650668 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc6ffd700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 0x7f1fc809d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc67fc700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1fc80d04f0 0x7f1fc81f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 -- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1fc80d04f0 msgr2=0x7f1fc81f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1fc80d04f0 0x7f1fc81f1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 -- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 msgr2=0x7f1fc809d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 0x7f1fc809d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 -- 172.21.15.104:0/814650668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb8009550 con 0x7f1fc80d25c0 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc6ffd700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 0x7f1fc809d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fc77fe700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007e40 tx=0x7f1fc000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000eea0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc0014420 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00147b0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc80b3300 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.547+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc80b38c0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.548+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc8067010 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.549+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1fc0014910 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.549+0000 7f1faf7fe700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 0x7f1fb406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.549+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1fc0089ff0 con 0x7f1fc80d25c0 2023-06-10T18:45:08.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.549+0000 7f1fc6ffd700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 0x7f1fb406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.550+0000 7f1fc6ffd700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 0x7f1fb406d190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc003f80 tx=0x7f1fbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.554+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1fc0054d70 con 0x7f1fc80d25c0 2023-06-10T18:45:08.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.739+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1fc8066000 con 0x7f1fc80d25c0 2023-06-10T18:45:08.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.739+0000 7f1faf7fe700 1 -- 172.21.15.104:0/814650668 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1fc0058390 con 0x7f1fc80d25c0 2023-06-10T18:45:08.744 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:08.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 msgr2=0x7f1fb406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 0x7f1fb406d190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc003f80 tx=0x7f1fbc009040 comp rx=0 tx=0).stop 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 msgr2=0x7f1fc80ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007e40 tx=0x7f1fc000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 shutdown_connections 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fb406ace0 0x7f1fb406d190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc80b05d0 0x7f1fc809d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc80d25c0 0x7f1fc80ce430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.741+0000 7f1fcd798700 1 --2- 172.21.15.104:0/814650668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1fc80d04f0 0x7f1fc81f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.742+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 >> 172.21.15.104:0/814650668 conn(0x7f1fc8071ec0 msgr2=0x7f1fc80b8290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.742+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 shutdown_connections 2023-06-10T18:45:08.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.742+0000 7f1fcd798700 1 -- 172.21.15.104:0/814650668 wait complete. 2023-06-10T18:45:08.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:08.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.874+0000 7fba5ac24700 1 Processor -- start 2023-06-10T18:45:08.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.874+0000 7fba5ac24700 1 -- start start 2023-06-10T18:45:08.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba5ac24700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba540cb300 0x7fba54163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba5ac24700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540c9e60 con 0x7fba540aeaa0 2023-06-10T18:45:08.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba5ac24700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540aee70 con 0x7fba540cd850 2023-06-10T18:45:08.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba5ac24700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba540c9bc0 con 0x7fba540cb300 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba591c1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba540cb300 0x7fba54163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba589c0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba540aeaa0 0x7fba540cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53976/0 (socket says 172.21.15.104:53976) 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba589c0700 1 -- 172.21.15.104:0/622223933 learned_addr learned my addr 172.21.15.104:0/622223933 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba591c1700 1 -- 172.21.15.104:0/622223933 >> v1:172.21.15.114:6789/0 conn(0x7fba540cd850 legacy=0x7fba5406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba591c1700 1 -- 172.21.15.104:0/622223933 >> v1:172.21.15.104:6789/0 conn(0x7fba540aeaa0 legacy=0x7fba540cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba591c1700 1 -- 172.21.15.104:0/622223933 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5406e540 con 0x7fba540cb300 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.875+0000 7fba537fe700 1 -- 172.21.15.104:0/622223933 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744319640 0 0) 0x7fba540c9e60 con 0x7fba540aeaa0 2023-06-10T18:45:08.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba591c1700 1 --2- 172.21.15.104:0/622223933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba54163c30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009d90 tx=0x7fba4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bfd9f223fd8170 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba537fe700 1 -- 172.21.15.104:0/622223933 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00e070 con 0x7fba540cb300 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba537fe700 1 -- 172.21.15.104:0/622223933 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c00bb90 con 0x7fba540cb300 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba537fe700 1 -- 172.21.15.104:0/622223933 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c003a20 con 0x7fba540cb300 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 -- 172.21.15.104:0/622223933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 msgr2=0x7fba54163c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 --2- 172.21.15.104:0/622223933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba54163c30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009d90 tx=0x7fba4c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 -- 172.21.15.104:0/622223933 shutdown_connections 2023-06-10T18:45:08.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 --2- 172.21.15.104:0/622223933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba54163c30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 -- 172.21.15.104:0/622223933 >> 172.21.15.104:0/622223933 conn(0x7fba5406b8a0 msgr2=0x7fba540cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:08.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 -- 172.21.15.104:0/622223933 shutdown_connections 2023-06-10T18:45:08.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.876+0000 7fba5ac24700 1 -- 172.21.15.104:0/622223933 wait complete. 2023-06-10T18:45:08.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.878+0000 7fba5ac24700 1 Processor -- start 2023-06-10T18:45:08.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.878+0000 7fba5ac24700 1 -- start start 2023-06-10T18:45:08.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 0x7fba540c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 0x7fba540c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba540b2970 con 0x7fba540cb300 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba540b2bf0 con 0x7fba540cd850 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba5ac24700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba540b2ed0 con 0x7fba540aeaa0 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56154/0 (socket says 172.21.15.104:56154) 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 -- 172.21.15.104:0/2373205229 learned_addr learned my addr 172.21.15.104:0/2373205229 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba591c1700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 0x7fba540c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba589c0700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 0x7fba540c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 -- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 msgr2=0x7fba540c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 0x7fba540c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 -- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 msgr2=0x7fba540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 0x7fba540c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba53fff700 1 -- 172.21.15.104:0/2373205229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c009a70 con 0x7fba540cb300 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba591c1700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 0x7fba540c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:08.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.879+0000 7fba589c0700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 0x7fba540c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba53fff700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba4000b4a0 tx=0x7fba4000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40011830 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40011990 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba540b31b0 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40011d40 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.880+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba540b3770 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.881+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba54074050 con 0x7fba540cb300 2023-06-10T18:45:08.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.881+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba4001a400 con 0x7fba540cb300 2023-06-10T18:45:08.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.881+0000 7fba517fa700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 0x7fba3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:08.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.881+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fba40089510 con 0x7fba540cb300 2023-06-10T18:45:08.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.882+0000 7fba589c0700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 0x7fba3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:08.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.882+0000 7fba589c0700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 0x7fba3806d190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fba48007fd0 tx=0x7fba48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:08.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:08.886+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba40054340 con 0x7fba540cb300 2023-06-10T18:45:09.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.090+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fba540b01a0 con 0x7fba540cb300 2023-06-10T18:45:09.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.091+0000 7fba517fa700 1 -- 172.21.15.104:0/2373205229 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fba40057960 con 0x7fba540cb300 2023-06-10T18:45:09.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 msgr2=0x7fba3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 0x7fba3806d190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fba48007fd0 tx=0x7fba48009040 comp rx=0 tx=0).stop 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 msgr2=0x7fba540b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba4000b4a0 tx=0x7fba4000b860 comp rx=0 tx=0).stop 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 shutdown_connections 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba3806ace0 0x7fba3806d190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cd850 0x7fba540c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540aeaa0 0x7fba540c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 --2- 172.21.15.104:0/2373205229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540cb300 0x7fba540b2430 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 >> 172.21.15.104:0/2373205229 conn(0x7fba5406b8a0 msgr2=0x7fba540af060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:09.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 shutdown_connections 2023-06-10T18:45:09.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.093+0000 7fba5ac24700 1 -- 172.21.15.104:0/2373205229 wait complete. 2023-06-10T18:45:09.100 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:09.109 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:45:09.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:09.110 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:09.149 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:09.149 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:45:09.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:09.150 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:45:09.206 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:09.207 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:09.227 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:09.228 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:09.288 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:09.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.403+0000 7f63fa103700 1 Processor -- start 2023-06-10T18:45:09.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.403+0000 7f63fa103700 1 -- start start 2023-06-10T18:45:09.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63fa103700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 0x7f63f40ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63fa103700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f63f4069e70 0x7f63f40cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63fa103700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f415a680 con 0x7f63f40cb690 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63fa103700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f40d0510 con 0x7f63f4069e70 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63fa103700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f406a5e0 con 0x7f63f415a2b0 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f63f4069e70 0x7f63f40cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f63f4069e70 0x7f63f40cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56162/0 (socket says 172.21.15.104:56162) 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 -- 172.21.15.104:0/3305402118 learned_addr learned my addr 172.21.15.104:0/3305402118 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f37fe700 1 --2- 172.21.15.104:0/3305402118 >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 0x7f63f40ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 -- 172.21.15.104:0/3305402118 >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 msgr2=0x7f63f40ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 --2- 172.21.15.104:0/3305402118 >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 0x7f63f40ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 -- 172.21.15.104:0/3305402118 >> v1:172.21.15.114:6789/0 conn(0x7f63f40cb690 legacy=0x7f63f40b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 -- 172.21.15.104:0/3305402118 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f406a8c0 con 0x7f63f4069e70 2023-06-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f37fe700 1 --2- 172.21.15.104:0/3305402118 >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 0x7f63f40ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f2ffd700 1 --2- 172.21.15.104:0/3305402118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f4069e70 0x7f63f40cb150 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63f4070110 tx=0x7f63e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1700f5c4243584 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f27fc700 1 -- 172.21.15.104:0/3305402118 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8004230 con 0x7f63f4069e70 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f27fc700 1 -- 172.21.15.104:0/3305402118 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8004390 con 0x7f63f4069e70 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.404+0000 7f63f27fc700 1 -- 172.21.15.104:0/3305402118 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80036a0 con 0x7f63f4069e70 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 -- 172.21.15.104:0/3305402118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f4069e70 msgr2=0x7f63f40cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 --2- 172.21.15.104:0/3305402118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f4069e70 0x7f63f40cb150 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63f4070110 tx=0x7f63e8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 -- 172.21.15.104:0/3305402118 shutdown_connections 2023-06-10T18:45:09.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 --2- 172.21.15.104:0/3305402118 >> v2:172.21.15.114:3300/0 conn(0x7f63f415a2b0 0x7f63f40ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 --2- 172.21.15.104:0/3305402118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f4069e70 0x7f63f40cb150 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 -- 172.21.15.104:0/3305402118 >> 172.21.15.104:0/3305402118 conn(0x7f63f406b8a0 msgr2=0x7f63f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:09.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 -- 172.21.15.104:0/3305402118 shutdown_connections 2023-06-10T18:45:09.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.405+0000 7f63fa103700 1 -- 172.21.15.104:0/3305402118 wait complete. 2023-06-10T18:45:09.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.406+0000 7f63fa103700 1 Processor -- start 2023-06-10T18:45:09.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.406+0000 7f63fa103700 1 -- start start 2023-06-10T18:45:09.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 0x7f63f40c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63f415a2b0 0x7f63f40af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f414ce40 con 0x7f63f40cb690 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f414d120 con 0x7f63f4069e70 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63fa103700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f414d400 con 0x7f63f415a2b0 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56168/0 (socket says 172.21.15.104:56168) 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 -- 172.21.15.104:0/570908720 learned_addr learned my addr 172.21.15.104:0/570908720 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f37fe700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 0x7f63f40c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f3fff700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63f415a2b0 0x7f63f40af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 -- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63f415a2b0 msgr2=0x7f63f40af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63f415a2b0 0x7f63f40af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 -- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 msgr2=0x7f63f40c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 0x7f63f40c4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f2ffd700 1 -- 172.21.15.104:0/570908720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e8009550 con 0x7f63f40cb690 2023-06-10T18:45:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.407+0000 7f63f37fe700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 0x7f63f40c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:09.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63f2ffd700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63f406cc20 tx=0x7f63e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8003d80 con 0x7f63f40cb690 2023-06-10T18:45:09.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e801f070 con 0x7f63f40cb690 2023-06-10T18:45:09.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8011870 con 0x7f63f40cb690 2023-06-10T18:45:09.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f414d6e0 con 0x7f63f40cb690 2023-06-10T18:45:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.408+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f40b0360 con 0x7f63f40cb690 2023-06-10T18:45:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.409+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f414dc00 con 0x7f63f40cb690 2023-06-10T18:45:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.409+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f63e8021030 con 0x7f63f40cb690 2023-06-10T18:45:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.409+0000 7f63dbfff700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 0x7f63e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.410+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f63e808ae20 con 0x7f63f40cb690 2023-06-10T18:45:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.410+0000 7f63f37fe700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 0x7f63e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.410+0000 7f63f37fe700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 0x7f63e006d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f63e400aa10 tx=0x7f63e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.414+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f63f414dc00 con 0x7f63f40cb690 2023-06-10T18:45:09.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.604+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63f414dc00 con 0x7f63f40cb690 2023-06-10T18:45:09.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.605+0000 7f63dbfff700 1 -- 172.21.15.104:0/570908720 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f63f414dc00 con 0x7f63f40cb690 2023-06-10T18:45:09.610 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:09.610 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:09.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 msgr2=0x7f63e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 0x7f63e006d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f63e400aa10 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 msgr2=0x7f63f40c4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63f406cc20 tx=0x7f63e8004750 comp rx=0 tx=0).stop 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 shutdown_connections 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63e006ace0 0x7f63e006d190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63f4069e70 0x7f63f40c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63f415a2b0 0x7f63f40af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 --2- 172.21.15.104:0/570908720 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63f40cb690 0x7f63f40c4bb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 >> 172.21.15.104:0/570908720 conn(0x7f63f406b8a0 msgr2=0x7f63f406e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:09.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 shutdown_connections 2023-06-10T18:45:09.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.608+0000 7f63fa103700 1 -- 172.21.15.104:0/570908720 wait complete. 2023-06-10T18:45:09.624 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:09.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:09.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.743+0000 7f7d869b8700 1 Processor -- start 2023-06-10T18:45:09.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.743+0000 7f7d869b8700 1 -- start start 2023-06-10T18:45:09.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d869b8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d800b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d869b8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d8006e6e0 0x7f7d800b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d869b8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8006e250 con 0x7f7d8006cde0 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d869b8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8015c950 con 0x7f7d8006e6e0 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d869b8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800cf0e0 con 0x7f7d8015c580 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d84f55700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7d8006cde0 0x7f7d80069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54004/0 (socket says 172.21.15.104:54004) 2023-06-10T18:45:09.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d84f55700 1 -- 172.21.15.104:0/1280657698 learned_addr learned my addr 172.21.15.104:0/1280657698 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/1280657698 >> v2:172.21.15.114:3300/0 conn(0x7f7d8006e6e0 0x7f7d800b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231652655 0 0) 0x7f7d8006e250 con 0x7f7d8006cde0 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d680035b0 con 0x7f7d8006cde0 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d800b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946197843 0 0) 0x7f7d680035b0 con 0x7f7d8006cde0 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 msgr2=0x7f7d800b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d800b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 >> v2:172.21.15.114:3300/0 conn(0x7f7d8006e6e0 msgr2=0x7f7d800b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 --2- 172.21.15.104:0/1280657698 >> v2:172.21.15.114:3300/0 conn(0x7f7d8006e6e0 0x7f7d800b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800b1e80 con 0x7f7d8006cde0 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7d780019f0 con 0x7f7d8006cde0 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 >> v1:172.21.15.104:6789/0 conn(0x7f7d8006cde0 legacy=0x7f7d80069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d68003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d68004030 0x7f7d68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004d90 0x7f7d68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.744+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015c950 con 0x7f7d68004030 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800cf0e0 con 0x7f7d68004d90 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6800bbd0 con 0x7f7d8015c580 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d68004030 0x7f7d68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d68003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d84f55700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004d90 0x7f7d68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 msgr2=0x7f7d68003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d68003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004d90 msgr2=0x7f7d68005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004d90 0x7f7d68005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/1280657698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d680035b0 con 0x7f7d68004030 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d8015c580 0x7f7d68003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d68004030 0x7f7d68004850 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009890 tx=0x7f7d68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5a8d76929398d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000c070 con 0x7f7d68004030 2023-06-10T18:45:09.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70013610 con 0x7f7d68004030 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.745+0000 7f7d7effd700 1 -- 172.21.15.104:0/1280657698 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700139a0 con 0x7f7d68004030 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 -- 172.21.15.104:0/1280657698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d68004030 msgr2=0x7f7d68004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 --2- 172.21.15.104:0/1280657698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d68004030 0x7f7d68004850 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009890 tx=0x7f7d68005880 comp rx=0 tx=0).stop 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d84f55700 1 -- 172.21.15.104:0/1280657698 reap_dead start 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 -- 172.21.15.104:0/1280657698 shutdown_connections 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 -- 172.21.15.104:0/1280657698 >> 172.21.15.104:0/1280657698 conn(0x7f7d8006b8a0 msgr2=0x7f7d800cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:09.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 -- 172.21.15.104:0/1280657698 shutdown_connections 2023-06-10T18:45:09.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.746+0000 7f7d869b8700 1 -- 172.21.15.104:0/1280657698 wait complete. 2023-06-10T18:45:09.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.747+0000 7f7d869b8700 1 Processor -- start 2023-06-10T18:45:09.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.747+0000 7f7d869b8700 1 -- start start 2023-06-10T18:45:09.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004030 0x7f7d8015f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 0x7f7d800b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015e630 con 0x7f7d801f2b40 2023-06-10T18:45:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015e910 con 0x7f7d68004030 2023-06-10T18:45:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8015ebf0 con 0x7f7d800b2420 2023-06-10T18:45:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56188/0 (socket says 172.21.15.104:56188) 2023-06-10T18:45:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/16382973 learned_addr learned my addr 172.21.15.104:0/16382973 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004030 0x7f7d8015f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d84f55700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 0x7f7d800b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 msgr2=0x7f7d800b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 0x7f7d800b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004030 msgr2=0x7f7d8015f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004030 0x7f7d8015f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 -- 172.21.15.104:0/16382973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8015eed0 con 0x7f7d801f2b40 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d84f55700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 0x7f7d800b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7f7fe700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009e80 tx=0x7f7d700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7001d840 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7001d9a0 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7001dd30 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.748+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8015f1b0 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.749+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800ada60 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.749+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80074050 con 0x7f7d801f2b40 2023-06-10T18:45:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.750+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7d70030440 con 0x7f7d801f2b40 2023-06-10T18:45:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.750+0000 7f7d7cff9700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 0x7f7d6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.750+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7d7008acd0 con 0x7f7d801f2b40 2023-06-10T18:45:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.750+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 0x7f7d6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.751+0000 7f7d7ffff700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 0x7f7d6c06d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.754+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7d70055a60 con 0x7f7d801f2b40 2023-06-10T18:45:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.941+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d800b5450 con 0x7f7d801f2b40 2023-06-10T18:45:09.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.942+0000 7f7d7cff9700 1 -- 172.21.15.104:0/16382973 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d70059080 con 0x7f7d801f2b40 2023-06-10T18:45:09.947 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 msgr2=0x7f7d6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 0x7f7d6c06d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74008040 comp rx=0 tx=0).stop 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 msgr2=0x7f7d800ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009e80 tx=0x7f7d700047c0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 shutdown_connections 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6c06ace0 0x7f7d6c06d190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d68004030 0x7f7d8015f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d800b2420 0x7f7d800b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 --2- 172.21.15.104:0/16382973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d801f2b40 0x7f7d800ad520 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 >> 172.21.15.104:0/16382973 conn(0x7f7d8006b8a0 msgr2=0x7f7d800cf250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 shutdown_connections 2023-06-10T18:45:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:09.944+0000 7f7d869b8700 1 -- 172.21.15.104:0/16382973 wait complete. 2023-06-10T18:45:09.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:10.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.078+0000 7f5befdc3700 1 Processor -- start 2023-06-10T18:45:10.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.078+0000 7f5befdc3700 1 -- start start 2023-06-10T18:45:10.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5be815c580 0x7f5be80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be806e6e0 0x7f5be80b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5be806cde0 0x7f5be8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be806e250 con 0x7f5be815c580 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be815c950 con 0x7f5be806cde0 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5befdc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80b1e80 con 0x7f5be806e6e0 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5bedb5f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5be815c580 0x7f5be80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5bedb5f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5be815c580 0x7f5be80b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56194/0 (socket says 172.21.15.104:56194) 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.079+0000 7f5bedb5f700 1 -- 172.21.15.104:0/2014141656 learned_addr learned my addr 172.21.15.104:0/2014141656 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bed35e700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be806e6e0 0x7f5be80b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bee360700 1 --2- 172.21.15.104:0/2014141656 >> v2:172.21.15.114:3300/0 conn(0x7f5be806cde0 0x7f5be8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 -- 172.21.15.104:0/2014141656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be806e6e0 msgr2=0x7f5be80b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be806e6e0 0x7f5be80b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 -- 172.21.15.104:0/2014141656 >> v2:172.21.15.114:3300/0 conn(0x7f5be806cde0 msgr2=0x7f5be8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 --2- 172.21.15.104:0/2014141656 >> v2:172.21.15.114:3300/0 conn(0x7f5be806cde0 0x7f5be8069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 -- 172.21.15.104:0/2014141656 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be806d920 con 0x7f5be815c580 2023-06-10T18:45:10.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5bedb5f700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be815c580 0x7f5be80b1070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5be80701f0 tx=0x7f5bd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c615cf566b6cdd1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5becb5d700 1 -- 172.21.15.104:0/2014141656 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8005e30 con 0x7f5be815c580 2023-06-10T18:45:10.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5becb5d700 1 -- 172.21.15.104:0/2014141656 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8004030 con 0x7f5be815c580 2023-06-10T18:45:10.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.080+0000 7f5becb5d700 1 -- 172.21.15.104:0/2014141656 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80043e0 con 0x7f5be815c580 2023-06-10T18:45:10.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 -- 172.21.15.104:0/2014141656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be815c580 msgr2=0x7f5be80b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be815c580 0x7f5be80b1070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5be80701f0 tx=0x7f5bd8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 -- 172.21.15.104:0/2014141656 shutdown_connections 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be815c580 0x7f5be80b1070 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 --2- 172.21.15.104:0/2014141656 >> v2:172.21.15.114:3300/0 conn(0x7f5be806cde0 0x7f5be8069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 --2- 172.21.15.104:0/2014141656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be806e6e0 0x7f5be80b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 -- 172.21.15.104:0/2014141656 >> 172.21.15.104:0/2014141656 conn(0x7f5be806b8a0 msgr2=0x7f5be80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 -- 172.21.15.104:0/2014141656 shutdown_connections 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.081+0000 7f5befdc3700 1 -- 172.21.15.104:0/2014141656 wait complete. 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.082+0000 7f5befdc3700 1 Processor -- start 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.082+0000 7f5befdc3700 1 -- start start 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be806cde0 0x7f5be8150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be815c580 0x7f5be80ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815e4c0 con 0x7f5be806e6e0 2023-06-10T18:45:10.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815d610 con 0x7f5be806cde0 2023-06-10T18:45:10.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be815e950 con 0x7f5be815c580 2023-06-10T18:45:10.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56200/0 (socket says 172.21.15.104:56200) 2023-06-10T18:45:10.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bedb5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be806cde0 0x7f5be8150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 -- 172.21.15.104:0/3494574899 learned_addr learned my addr 172.21.15.104:0/3494574899 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bee360700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be815c580 0x7f5be80ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 -- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be815c580 msgr2=0x7f5be80ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be815c580 0x7f5be80ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 -- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be806cde0 msgr2=0x7f5be8150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be806cde0 0x7f5be8150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 -- 172.21.15.104:0/3494574899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8009500 con 0x7f5be806e6e0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bed35e700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e810 tx=0x7f5be400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400c070 con 0x7f5be806e6e0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4013610 con 0x7f5be806e6e0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40139c0 con 0x7f5be806e6e0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be815d8b0 con 0x7f5be806e6e0 2023-06-10T18:45:10.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.083+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be815de70 con 0x7f5be806e6e0 2023-06-10T18:45:10.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.084+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be815e630 con 0x7f5be806e6e0 2023-06-10T18:45:10.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.085+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5be4013b20 con 0x7f5be806e6e0 2023-06-10T18:45:10.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.085+0000 7f5bde7fc700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 0x7f5bcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.085+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5be4089500 con 0x7f5be806e6e0 2023-06-10T18:45:10.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.085+0000 7f5bedb5f700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 0x7f5bcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.086+0000 7f5bedb5f700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 0x7f5bcc06d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5be80701f0 tx=0x7f5bd8008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.089+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5be4057d20 con 0x7f5be806e6e0 2023-06-10T18:45:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.276+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5be8074050 con 0x7f5be806e6e0 2023-06-10T18:45:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.276+0000 7f5bde7fc700 1 -- 172.21.15.104:0/3494574899 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5be40578b0 con 0x7f5be806e6e0 2023-06-10T18:45:10.281 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 msgr2=0x7f5bcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 0x7f5bcc06d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5be80701f0 tx=0x7f5bd8008070 comp rx=0 tx=0).stop 2023-06-10T18:45:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 msgr2=0x7f5be815f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e810 tx=0x7f5be400eb20 comp rx=0 tx=0).stop 2023-06-10T18:45:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 shutdown_connections 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5bcc06ace0 0x7f5bcc06d190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be806cde0 0x7f5be8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5be815c580 0x7f5be80ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 --2- 172.21.15.104:0/3494574899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5be806e6e0 0x7f5be815f1b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 >> 172.21.15.104:0/3494574899 conn(0x7f5be806b8a0 msgr2=0x7f5be80ce120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 shutdown_connections 2023-06-10T18:45:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.278+0000 7f5befdc3700 1 -- 172.21.15.104:0/3494574899 wait complete. 2023-06-10T18:45:10.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:10.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.412+0000 7f88cfbeb700 1 Processor -- start 2023-06-10T18:45:10.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.412+0000 7f88cfbeb700 1 -- start start 2023-06-10T18:45:10.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cfbeb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88c80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cfbeb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80ad3c0 con 0x7f88c8157b30 2023-06-10T18:45:10.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cfbeb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c8157f00 con 0x7f88c80d0830 2023-06-10T18:45:10.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cfbeb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b0170 con 0x7f88c80cea50 2023-06-10T18:45:10.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cd987700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f88c8157b30 0x7f88c80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54034/0 (socket says 172.21.15.104:54034) 2023-06-10T18:45:10.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.413+0000 7f88cd987700 1 -- 172.21.15.104:0/1550767949 learned_addr learned my addr 172.21.15.104:0/1550767949 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:10.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd186700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88c80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244675330 0 0) 0x7f88c80ad3c0 con 0x7f88c8157b30 2023-06-10T18:45:10.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b40035b0 con 0x7f88c8157b30 2023-06-10T18:45:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714203916 0 0) 0x7f88b40035b0 con 0x7f88c8157b30 2023-06-10T18:45:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 msgr2=0x7f88c80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88c80d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 >> v1:172.21.15.114:6789/0 conn(0x7f88c80d0830 legacy=0x7f88c80b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80b0410 con 0x7f88c8157b30 2023-06-10T18:45:10.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f88b8001670 con 0x7f88c8157b30 2023-06-10T18:45:10.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 >> v1:172.21.15.104:6789/0 conn(0x7f88c8157b30 legacy=0x7f88c80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88b4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b4004d90 0x7f88b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b0170 con 0x7f88b4003a30 2023-06-10T18:45:10.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400bac0 con 0x7f88b4004d90 2023-06-10T18:45:10.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400bc80 con 0x7f88c80cea50 2023-06-10T18:45:10.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88b4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd186700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88ce188700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b4004d90 0x7f88b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 -- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 msgr2=0x7f88b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80cea50 0x7f88b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 -- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b4004d90 msgr2=0x7f88b4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b4004d90 0x7f88b4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 -- 172.21.15.104:0/1550767949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40035b0 con 0x7f88b4003a30 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88ce188700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b4004d90 0x7f88b4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.414+0000 7f88cd987700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88b4003e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c2a0 tx=0x7f88b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e6ff6b677b395d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007c10 con 0x7f88b4003a30 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b800cdc0 con 0x7f88b4003a30 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cc985700 1 -- 172.21.15.104:0/1550767949 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8005ba0 con 0x7f88b4003a30 2023-06-10T18:45:10.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 -- 172.21.15.104:0/1550767949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 msgr2=0x7f88b4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/1550767949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88b4003e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c2a0 tx=0x7f88b4005880 comp rx=0 tx=0).stop 2023-06-10T18:45:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88ce188700 1 -- 172.21.15.104:0/1550767949 reap_dead start 2023-06-10T18:45:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 -- 172.21.15.104:0/1550767949 shutdown_connections 2023-06-10T18:45:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 -- 172.21.15.104:0/1550767949 >> 172.21.15.104:0/1550767949 conn(0x7f88c8089fd0 msgr2=0x7f88c808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 -- 172.21.15.104:0/1550767949 shutdown_connections 2023-06-10T18:45:10.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.415+0000 7f88cfbeb700 1 -- 172.21.15.104:0/1550767949 wait complete. 2023-06-10T18:45:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.416+0000 7f88cfbeb700 1 Processor -- start 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 -- start start 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c8153910 0x7f88c80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 0x7f88c806af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b2800 con 0x7f88b4003a30 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c814c110 con 0x7f88c80b2380 2023-06-10T18:45:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cfbeb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8154080 con 0x7f88c8153910 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cd987700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cd987700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56218/0 (socket says 172.21.15.104:56218) 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cd987700 1 -- 172.21.15.104:0/472970231 learned_addr learned my addr 172.21.15.104:0/472970231 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88ce188700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 0x7f88c806af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.417+0000 7f88cd186700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c8153910 0x7f88c80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 -- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c8153910 msgr2=0x7f88c80b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c8153910 0x7f88c80b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 -- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 msgr2=0x7f88c806af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 0x7f88c806af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 -- 172.21.15.104:0/472970231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8154360 con 0x7f88b4003a30 2023-06-10T18:45:10.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88ce188700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 0x7f88c806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:10.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cd987700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f88b40047d0 tx=0x7f88c8158090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8001640 con 0x7f88b4003a30 2023-06-10T18:45:10.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b80070d0 con 0x7f88b4003a30 2023-06-10T18:45:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c8154640 con 0x7f88b4003a30 2023-06-10T18:45:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80074a0 con 0x7f88b4003a30 2023-06-10T18:45:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.418+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8153d80 con 0x7f88b4003a30 2023-06-10T18:45:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.419+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c80b2780 con 0x7f88b4003a30 2023-06-10T18:45:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.419+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f88b8019030 con 0x7f88b4003a30 2023-06-10T18:45:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.420+0000 7f88be7fc700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 0x7f88ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.420+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f88b808a9e0 con 0x7f88b4003a30 2023-06-10T18:45:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.420+0000 7f88cd186700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 0x7f88ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.420+0000 7f88cd186700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 0x7f88ac06d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88c80cee40 tx=0x7f88c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.424+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f88b8055770 con 0x7f88b4003a30 2023-06-10T18:45:10.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.609+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88c80cd1e0 con 0x7f88b4003a30 2023-06-10T18:45:10.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.609+0000 7f88be7fc700 1 -- 172.21.15.104:0/472970231 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88b8007d70 con 0x7f88b4003a30 2023-06-10T18:45:10.614 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 msgr2=0x7f88ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 0x7f88ac06d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88c80cee40 tx=0x7f88c4008040 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 msgr2=0x7f88c806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f88b40047d0 tx=0x7f88c8158090 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 shutdown_connections 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88ac06ace0 0x7f88ac06d190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80b2380 0x7f88c806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88b4003a30 0x7f88c806cbe0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 --2- 172.21.15.104:0/472970231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c8153910 0x7f88c80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 >> 172.21.15.104:0/472970231 conn(0x7f88c8089fd0 msgr2=0x7f88c814c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 shutdown_connections 2023-06-10T18:45:10.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.611+0000 7f88cfbeb700 1 -- 172.21.15.104:0/472970231 wait complete. 2023-06-10T18:45:10.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:10.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.746+0000 7fd7372c9700 1 Processor -- start 2023-06-10T18:45:10.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 -- start start 2023-06-10T18:45:10.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd73015c6f0 0x7fd7300b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd7300aec00 0x7fd7300b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7300ae7e0 con 0x7fd73006ea70 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73006df90 con 0x7fd7300aec00 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.747+0000 7fd7372c9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73006e270 con 0x7fd73015c6f0 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd7300aec00 0x7fd7300b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd734864700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd73006ea70 0x7fd7300ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54054/0 (socket says 172.21.15.104:54054) 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd734864700 1 -- 172.21.15.104:0/4114355570 learned_addr learned my addr 172.21.15.104:0/4114355570 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735065700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd73015c6f0 0x7fd7300b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 -- 172.21.15.104:0/4114355570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd73015c6f0 msgr2=0x7fd7300b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd73015c6f0 0x7fd7300b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 -- 172.21.15.104:0/4114355570 >> v1:172.21.15.104:6789/0 conn(0x7fd73006ea70 legacy=0x7fd7300ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd727fff700 1 -- 172.21.15.104:0/4114355570 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655710941 0 0) 0x7fd7300ae7e0 con 0x7fd73006ea70 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 -- 172.21.15.104:0/4114355570 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73006e550 con 0x7fd7300aec00 2023-06-10T18:45:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd735866700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7300aec00 0x7fd7300b15b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd728009d90 tx=0x7fd72800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50f475bc7f1cd92 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd727fff700 1 -- 172.21.15.104:0/4114355570 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72800e070 con 0x7fd7300aec00 2023-06-10T18:45:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd727fff700 1 -- 172.21.15.104:0/4114355570 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72800bbd0 con 0x7fd7300aec00 2023-06-10T18:45:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.748+0000 7fd727fff700 1 -- 172.21.15.104:0/4114355570 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728012430 con 0x7fd7300aec00 2023-06-10T18:45:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 -- 172.21.15.104:0/4114355570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7300aec00 msgr2=0x7fd7300b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7300aec00 0x7fd7300b15b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd728009d90 tx=0x7fd72800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 -- 172.21.15.104:0/4114355570 shutdown_connections 2023-06-10T18:45:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7300aec00 0x7fd7300b15b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 --2- 172.21.15.104:0/4114355570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd73015c6f0 0x7fd7300b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 -- 172.21.15.104:0/4114355570 >> 172.21.15.104:0/4114355570 conn(0x7fd73006b8a0 msgr2=0x7fd7300cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 -- 172.21.15.104:0/4114355570 shutdown_connections 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.749+0000 7fd7372c9700 1 -- 172.21.15.104:0/4114355570 wait complete. 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.750+0000 7fd7372c9700 1 Processor -- start 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 -- start start 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7300aec00 0x7fd7300c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd73015c6f0 0x7fd7300cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c9920 con 0x7fd73006ea70 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c8880 con 0x7fd73015c6f0 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd7372c9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300c8b60 con 0x7fd7300aec00 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.751+0000 7fd735065700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56232/0 (socket says 172.21.15.104:56232) 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 -- 172.21.15.104:0/2495309657 learned_addr learned my addr 172.21.15.104:0/2495309657 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:10.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 -- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7300aec00 msgr2=0x7fd7300c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7300aec00 0x7fd7300c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735866700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd73015c6f0 0x7fd7300cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 -- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd73015c6f0 msgr2=0x7fd7300cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd73015c6f0 0x7fd7300cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 -- 172.21.15.104:0/2495309657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd728009a70 con 0x7fd73006ea70 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd735065700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d7d0 tx=0x7fd72000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7200125f0 con 0x7fd73006ea70 2023-06-10T18:45:10.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720012750 con 0x7fd73006ea70 2023-06-10T18:45:10.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720012b00 con 0x7fd73006ea70 2023-06-10T18:45:10.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7300c8e40 con 0x7fd73006ea70 2023-06-10T18:45:10.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.752+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7300c9400 con 0x7fd73006ea70 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.753+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7300c9620 con 0x7fd73006ea70 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.753+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd720012c60 con 0x7fd73006ea70 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.754+0000 7fd725ffb700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 0x7fd71406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.754+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd7200898e0 con 0x7fd73006ea70 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.754+0000 7fd734864700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 0x7fd71406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:10.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.755+0000 7fd734864700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 0x7fd71406d1c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd73006dde0 tx=0x7fd72c0079d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:10.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.758+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd720054660 con 0x7fd73006ea70 2023-06-10T18:45:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.943+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd730074050 con 0x7fd73006ea70 2023-06-10T18:45:10.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.944+0000 7fd725ffb700 1 -- 172.21.15.104:0/2495309657 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd720057c80 con 0x7fd73006ea70 2023-06-10T18:45:10.949 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:10.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 msgr2=0x7fd71406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 0x7fd71406d1c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd73006dde0 tx=0x7fd72c0079d0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 msgr2=0x7fd7300c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d7d0 tx=0x7fd72000db90 comp rx=0 tx=0).stop 2023-06-10T18:45:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 shutdown_connections 2023-06-10T18:45:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd71406ad10 0x7fd71406d1c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd73015c6f0 0x7fd7300cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd73006ea70 0x7fd7300c8280 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 --2- 172.21.15.104:0/2495309657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7300aec00 0x7fd7300c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 >> 172.21.15.104:0/2495309657 conn(0x7fd73006b8a0 msgr2=0x7fd7300ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 shutdown_connections 2023-06-10T18:45:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:10.946+0000 7fd7372c9700 1 -- 172.21.15.104:0/2495309657 wait complete. 2023-06-10T18:45:10.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.081+0000 7fb37fc0c700 1 Processor -- start 2023-06-10T18:45:11.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.081+0000 7fb37fc0c700 1 -- start start 2023-06-10T18:45:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37fc0c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 0x7fb37815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37fc0c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 0x7fb3780cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37fc0c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780cdab0 con 0x7fb3780af940 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37fc0c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37815dab0 con 0x7fb3780cd150 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37fc0c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37815aa00 con 0x7fb3781545a0 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37e1a9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 0x7fb3780cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37d9a8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 0x7fb37815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37d1a7700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb3780af940 0x7fb37815e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59914/0 (socket says 172.21.15.104:59914) 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37e1a9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 0x7fb3780cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:43746/0 (socket says 172.21.15.104:43746) 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.082+0000 7fb37d1a7700 1 -- 172.21.15.104:0/1567462086 learned_addr learned my addr 172.21.15.104:0/1567462086 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37d9a8700 1 -- 172.21.15.104:0/1567462086 >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 msgr2=0x7fb3780cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37d9a8700 1 --2- 172.21.15.104:0/1567462086 >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 0x7fb3780cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37d9a8700 1 -- 172.21.15.104:0/1567462086 >> v1:172.21.15.114:6789/0 conn(0x7fb3780af940 legacy=0x7fb37815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37d9a8700 1 -- 172.21.15.104:0/1567462086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378154970 con 0x7fb3781545a0 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37d9a8700 1 --2- 172.21.15.104:0/1567462086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 0x7fb37815a4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb368009840 tx=0x7fb368009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b9fe821d00b544 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37c9a6700 1 -- 172.21.15.104:0/1567462086 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011830 con 0x7fb3781545a0 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37c9a6700 1 -- 172.21.15.104:0/1567462086 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb368011990 con 0x7fb3781545a0 2023-06-10T18:45:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.083+0000 7fb37c9a6700 1 -- 172.21.15.104:0/1567462086 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011d20 con 0x7fb3781545a0 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 -- 172.21.15.104:0/1567462086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 msgr2=0x7fb37815a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/1567462086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 0x7fb37815a4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb368009840 tx=0x7fb368009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 -- 172.21.15.104:0/1567462086 shutdown_connections 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/1567462086 >> v2:172.21.15.114:3300/0 conn(0x7fb3780cd150 0x7fb3780cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/1567462086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3781545a0 0x7fb37815a4c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 -- 172.21.15.104:0/1567462086 >> 172.21.15.104:0/1567462086 conn(0x7fb37806e370 msgr2=0x7fb37806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 -- 172.21.15.104:0/1567462086 shutdown_connections 2023-06-10T18:45:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.084+0000 7fb37fc0c700 1 -- 172.21.15.104:0/1567462086 wait complete. 2023-06-10T18:45:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.085+0000 7fb37fc0c700 1 Processor -- start 2023-06-10T18:45:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.085+0000 7fb37fc0c700 1 -- start start 2023-06-10T18:45:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3780af940 0x7fb37814b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 0x7fb3780cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37806af80 con 0x7fb3780cd150 2023-06-10T18:45:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37814b7e0 con 0x7fb3781545a0 2023-06-10T18:45:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37806b1d0 con 0x7fb3780af940 2023-06-10T18:45:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56248/0 (socket says 172.21.15.104:56248) 2023-06-10T18:45:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 -- 172.21.15.104:0/2119345067 learned_addr learned my addr 172.21.15.104:0/2119345067 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d9a8700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3780af940 0x7fb37814b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37e1a9700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 0x7fb3780cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 -- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3780af940 msgr2=0x7fb37814b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3780af940 0x7fb37814b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 -- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 msgr2=0x7fb3780cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 0x7fb3780cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 -- 172.21.15.104:0/2119345067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680094a0 con 0x7fb3780cd150 2023-06-10T18:45:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37e1a9700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 0x7fb3780cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37d1a7700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c480 tx=0x7fb37400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374005d70 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37400ce90 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3740072d0 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.086+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37806b4b0 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.087+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37806ba70 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.087+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb378074050 con 0x7fb3780cd150 2023-06-10T18:45:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.088+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb374007430 con 0x7fb3780cd150 2023-06-10T18:45:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.088+0000 7fb36e7fc700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 0x7fb35c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.088+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb3740888d0 con 0x7fb3780cd150 2023-06-10T18:45:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.088+0000 7fb37d9a8700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 0x7fb35c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.089+0000 7fb37d9a8700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 0x7fb35c06d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb368000c00 tx=0x7fb36801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.092+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb374053790 con 0x7fb3780cd150 2023-06-10T18:45:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.277+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb37815b230 con 0x7fb3780cd150 2023-06-10T18:45:11.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.277+0000 7fb36e7fc700 1 -- 172.21.15.104:0/2119345067 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb374056db0 con 0x7fb3780cd150 2023-06-10T18:45:11.284 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:11.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 msgr2=0x7fb35c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 0x7fb35c06d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb368000c00 tx=0x7fb36801c040 comp rx=0 tx=0).stop 2023-06-10T18:45:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 msgr2=0x7fb3780cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c480 tx=0x7fb37400c790 comp rx=0 tx=0).stop 2023-06-10T18:45:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 shutdown_connections 2023-06-10T18:45:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb35c06ace0 0x7fb35c06d190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781545a0 0x7fb3780cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3780af940 0x7fb37814b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 --2- 172.21.15.104:0/2119345067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3780cd150 0x7fb3780cf400 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 >> 172.21.15.104:0/2119345067 conn(0x7fb37806e370 msgr2=0x7fb37815bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 shutdown_connections 2023-06-10T18:45:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.279+0000 7fb37fc0c700 1 -- 172.21.15.104:0/2119345067 wait complete. 2023-06-10T18:45:11.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:11.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.414+0000 7fb664361700 1 Processor -- start 2023-06-10T18:45:11.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.414+0000 7fb664361700 1 -- start start 2023-06-10T18:45:11.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 0x7fb65c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb65c0ca100 0x7fb65c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65c071640 con 0x7fb65c06e4b0 2023-06-10T18:45:11.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65c0aef00 con 0x7fb65c0ca100 2023-06-10T18:45:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb664361700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c06d290 con 0x7fb65c06e0e0 2023-06-10T18:45:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb6618fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb6618fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56254/0 (socket says 172.21.15.104:56254) 2023-06-10T18:45:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb6618fc700 1 -- 172.21.15.104:0/1248557125 learned_addr learned my addr 172.21.15.104:0/1248557125 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb6620fd700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 0x7fb65c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.415+0000 7fb6628fe700 1 --2- 172.21.15.104:0/1248557125 >> v2:172.21.15.114:3300/0 conn(0x7fb65c0ca100 0x7fb65c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 -- 172.21.15.104:0/1248557125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 msgr2=0x7fb65c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 0x7fb65c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 -- 172.21.15.104:0/1248557125 >> v2:172.21.15.114:3300/0 conn(0x7fb65c0ca100 msgr2=0x7fb65c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 --2- 172.21.15.104:0/1248557125 >> v2:172.21.15.114:3300/0 conn(0x7fb65c0ca100 0x7fb65c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 -- 172.21.15.104:0/1248557125 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c06d4c0 con 0x7fb65c06e4b0 2023-06-10T18:45:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6620fd700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 0x7fb65c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6618fc700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb65c071210 tx=0x7fb658009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a337ded9c5326bfb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6610fb700 1 -- 172.21.15.104:0/1248557125 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65800b860 con 0x7fb65c06e4b0 2023-06-10T18:45:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6610fb700 1 -- 172.21.15.104:0/1248557125 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65800b9c0 con 0x7fb65c06e4b0 2023-06-10T18:45:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb6610fb700 1 -- 172.21.15.104:0/1248557125 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65800bd50 con 0x7fb65c06e4b0 2023-06-10T18:45:11.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 -- 172.21.15.104:0/1248557125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e4b0 msgr2=0x7fb65c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb65c071210 tx=0x7fb658009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:11.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 -- 172.21.15.104:0/1248557125 shutdown_connections 2023-06-10T18:45:11.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 --2- 172.21.15.104:0/1248557125 >> v2:172.21.15.114:3300/0 conn(0x7fb65c0ca100 0x7fb65c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c06e0e0 0x7fb65c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 --2- 172.21.15.104:0/1248557125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e4b0 0x7fb65c0c9bc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.416+0000 7fb664361700 1 -- 172.21.15.104:0/1248557125 >> 172.21.15.104:0/1248557125 conn(0x7fb65c06b8a0 msgr2=0x7fb65c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:11.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.417+0000 7fb664361700 1 -- 172.21.15.104:0/1248557125 shutdown_connections 2023-06-10T18:45:11.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.417+0000 7fb664361700 1 -- 172.21.15.104:0/1248557125 wait complete. 2023-06-10T18:45:11.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 Processor -- start 2023-06-10T18:45:11.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 -- start start 2023-06-10T18:45:11.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c06e4b0 0x7fb65c099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c0ca100 0x7fb65c099880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c09b8d0 con 0x7fb65c06e0e0 2023-06-10T18:45:11.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c09bbb0 con 0x7fb65c06e4b0 2023-06-10T18:45:11.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.418+0000 7fb664361700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c09be90 con 0x7fb65c0ca100 2023-06-10T18:45:11.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56260/0 (socket says 172.21.15.104:56260) 2023-06-10T18:45:11.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 -- 172.21.15.104:0/2535957960 learned_addr learned my addr 172.21.15.104:0/2535957960 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:11.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6618fc700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c06e4b0 0x7fb65c099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6628fe700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c0ca100 0x7fb65c099880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 -- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c0ca100 msgr2=0x7fb65c099880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c0ca100 0x7fb65c099880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 -- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c06e4b0 msgr2=0x7fb65c099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c06e4b0 0x7fb65c099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 -- 172.21.15.104:0/2535957960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb658009550 con 0x7fb65c06e0e0 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb6620fd700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00e810 tx=0x7fb64c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c005c20 con 0x7fb65c06e0e0 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c005d80 con 0x7fb65c06e0e0 2023-06-10T18:45:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c09c170 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c004250 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.419+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c09b530 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.420+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65c09c4e0 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.420+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb64c0043b0 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.421+0000 7fb652ffd700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 0x7fb64006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.421+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb64c089350 con 0x7fb65c06e0e0 2023-06-10T18:45:11.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.421+0000 7fb6618fc700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 0x7fb64006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.421+0000 7fb6618fc700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 0x7fb64006d190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb658000c00 tx=0x7fb658008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.425+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb64c054160 con 0x7fb65c06e0e0 2023-06-10T18:45:11.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.627+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb65c09ba10 con 0x7fb65c06e0e0 2023-06-10T18:45:11.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.627+0000 7fb652ffd700 1 -- 172.21.15.104:0/2535957960 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb65c09ba10 con 0x7fb65c06e0e0 2023-06-10T18:45:11.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 msgr2=0x7fb64006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 0x7fb64006d190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb658000c00 tx=0x7fb658008030 comp rx=0 tx=0).stop 2023-06-10T18:45:11.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 msgr2=0x7fb65c1478d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00e810 tx=0x7fb64c00eb20 comp rx=0 tx=0).stop 2023-06-10T18:45:11.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 shutdown_connections 2023-06-10T18:45:11.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c0ca100 0x7fb65c099880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb65c06e0e0 0x7fb65c1478d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb64006ace0 0x7fb64006d190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 --2- 172.21.15.104:0/2535957960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c06e4b0 0x7fb65c099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 >> 172.21.15.104:0/2535957960 conn(0x7fb65c06b8a0 msgr2=0x7fb65c0ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:11.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 shutdown_connections 2023-06-10T18:45:11.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.630+0000 7fb664361700 1 -- 172.21.15.104:0/2535957960 wait complete. 2023-06-10T18:45:11.638 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:11.646 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:11.647 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:11.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:11.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.794+0000 7f37f9fd2700 1 Processor -- start 2023-06-10T18:45:11.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.794+0000 7f37f9fd2700 1 -- start start 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f9fd2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f9fd2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 0x7f37f40cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f9fd2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f40c9e60 con 0x7f37f40ca2f0 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f9fd2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f40c9bc0 con 0x7f37f40b1080 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f9fd2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40aee70 con 0x7f37f40aeaa0 2023-06-10T18:45:11.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f37fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f2ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 0x7f37f40cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f3fff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f37f40ca2f0 0x7f37f40ca750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59946/0 (socket says 172.21.15.104:59946) 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f3fff700 1 -- 172.21.15.104:0/463295935 learned_addr learned my addr 172.21.15.104:0/463295935 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.795+0000 7f37f37fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:47420/0 (socket says 172.21.15.104:47420) 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f27fc700 1 -- 172.21.15.104:0/463295935 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864252797 0 0) 0x7f37f40c9e60 con 0x7f37f40ca2f0 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f27fc700 1 -- 172.21.15.104:0/463295935 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e00035b0 con 0x7f37f40ca2f0 2023-06-10T18:45:11.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f37fe700 1 -- 172.21.15.104:0/463295935 >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 msgr2=0x7f37f40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f37fe700 1 --2- 172.21.15.104:0/463295935 >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 0x7f37f40cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f37fe700 1 -- 172.21.15.104:0/463295935 >> v1:172.21.15.114:6789/0 conn(0x7f37f40ca2f0 legacy=0x7f37f40ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f37fe700 1 -- 172.21.15.104:0/463295935 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f406e540 con 0x7f37f40aeaa0 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f2ffd700 1 --2- 172.21.15.104:0/463295935 >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 0x7f37f40cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f37fe700 1 --2- 172.21.15.104:0/463295935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bac0 tx=0x7f37dc00bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f3e9ee413760fb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f27fc700 1 -- 172.21.15.104:0/463295935 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc005e30 con 0x7f37f40aeaa0 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f27fc700 1 -- 172.21.15.104:0/463295935 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00d860 con 0x7f37f40aeaa0 2023-06-10T18:45:11.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.796+0000 7f37f27fc700 1 -- 172.21.15.104:0/463295935 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00dbd0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 -- 172.21.15.104:0/463295935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 msgr2=0x7f37f40cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/463295935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bac0 tx=0x7f37dc00bdd0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 -- 172.21.15.104:0/463295935 shutdown_connections 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/463295935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40aeaa0 0x7f37f40cd140 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/463295935 >> v2:172.21.15.114:3300/0 conn(0x7f37f40b1080 0x7f37f40cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 -- 172.21.15.104:0/463295935 >> 172.21.15.104:0/463295935 conn(0x7f37f406b8a0 msgr2=0x7f37f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 -- 172.21.15.104:0/463295935 shutdown_connections 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.797+0000 7f37f9fd2700 1 -- 172.21.15.104:0/463295935 wait complete. 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.798+0000 7f37f9fd2700 1 Processor -- start 2023-06-10T18:45:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.798+0000 7f37f9fd2700 1 -- start start 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40b1080 0x7f37f415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 0x7f37f40b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415bc40 con 0x7f37f40aeaa0 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415b030 con 0x7f37f40ca2f0 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f9fd2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415b310 con 0x7f37f40b1080 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56274/0 (socket says 172.21.15.104:56274) 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 -- 172.21.15.104:0/3610501878 learned_addr learned my addr 172.21.15.104:0/3610501878 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f3fff700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 0x7f37f40b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 -- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40b1080 msgr2=0x7f37f415aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f2ffd700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40b1080 0x7f37f415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40b1080 0x7f37f415aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 -- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 msgr2=0x7f37f40b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 0x7f37f40b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 -- 172.21.15.104:0/3610501878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40094a0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f3fff700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 0x7f37f40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37f37fe700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00be10 tx=0x7f37dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.799+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004050 con 0x7f37f40aeaa0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.800+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc0041b0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.800+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc00b6d0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.800+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004520 con 0x7f37f40aeaa0 2023-06-10T18:45:11.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.800+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f415b8d0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.800+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4074050 con 0x7f37f40aeaa0 2023-06-10T18:45:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.801+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f37dc0036a0 con 0x7f37f40aeaa0 2023-06-10T18:45:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.801+0000 7f37ebfff700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 0x7f37d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.801+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f37dc08d040 con 0x7f37f40aeaa0 2023-06-10T18:45:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.802+0000 7f37f2ffd700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 0x7f37d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.802+0000 7f37f2ffd700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 0x7f37d406d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37f40c2da0 tx=0x7f37e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.806+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f37dc058020 con 0x7f37f40aeaa0 2023-06-10T18:45:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.994+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f40c71f0 con 0x7f37f40aeaa0 2023-06-10T18:45:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.995+0000 7f37ebfff700 1 -- 172.21.15.104:0/3610501878 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f37dc05b640 con 0x7f37f40aeaa0 2023-06-10T18:45:12.001 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:12.001 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 msgr2=0x7f37d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 0x7f37d406d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37f40c2da0 tx=0x7f37e4008040 comp rx=0 tx=0).stop 2023-06-10T18:45:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 msgr2=0x7f37f415a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00be10 tx=0x7f37dc004750 comp rx=0 tx=0).stop 2023-06-10T18:45:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 shutdown_connections 2023-06-10T18:45:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37d406ace0 0x7f37d406d190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37f40ca2f0 0x7f37f40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37f40aeaa0 0x7f37f415a4c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 --2- 172.21.15.104:0/3610501878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37f40b1080 0x7f37f415aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 >> 172.21.15.104:0/3610501878 conn(0x7f37f406b8a0 msgr2=0x7f37f40ca900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 shutdown_connections 2023-06-10T18:45:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:11.997+0000 7f37f9fd2700 1 -- 172.21.15.104:0/3610501878 wait complete. 2023-06-10T18:45:12.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:12.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:12.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.134+0000 7f30149b7700 1 Processor -- start 2023-06-10T18:45:12.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.134+0000 7f30149b7700 1 -- start start 2023-06-10T18:45:12.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.135+0000 7f30149b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.135+0000 7f30149b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f30100ae230 0x7f301006d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.135+0000 7f30149b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301015cb00 con 0x7f30100cd720 2023-06-10T18:45:12.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.135+0000 7f30149b7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30100cdb90 con 0x7f30100ae230 2023-06-10T18:45:12.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.135+0000 7f30149b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30100acd90 con 0x7f301015c730 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300ed9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f30100ae230 0x7f301006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:47434/0 (socket says 172.21.15.104:47434) 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 -- 172.21.15.104:0/3713917178 learned_addr learned my addr 172.21.15.104:0/3713917178 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300dd9b700 1 --1- 172.21.15.104:0/3713917178 >> v1:172.21.15.114:6789/0 conn(0x7f30100cd720 0x7f301006cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59956/0 (socket says 172.21.15.104:59956) 2023-06-10T18:45:12.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 -- 172.21.15.104:0/3713917178 >> v2:172.21.15.114:3300/0 conn(0x7f30100ae230 msgr2=0x7f301006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 --2- 172.21.15.104:0/3713917178 >> v2:172.21.15.114:3300/0 conn(0x7f30100ae230 0x7f301006d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 -- 172.21.15.104:0/3713917178 >> v1:172.21.15.114:6789/0 conn(0x7f30100cd720 legacy=0x7f301006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 -- 172.21.15.104:0/3713917178 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100ac460 con 0x7f301015c730 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300e59c700 1 --2- 172.21.15.104:0/3713917178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3004006fd0 tx=0x7f3004009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb738a002f19597 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300d59a700 1 -- 172.21.15.104:0/3713917178 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004005e30 con 0x7f301015c730 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300d59a700 1 -- 172.21.15.104:0/3713917178 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3004004030 con 0x7f301015c730 2023-06-10T18:45:12.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.136+0000 7f300d59a700 1 -- 172.21.15.104:0/3713917178 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30040043e0 con 0x7f301015c730 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 -- 172.21.15.104:0/3713917178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 msgr2=0x7f30100cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 --2- 172.21.15.104:0/3713917178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3004006fd0 tx=0x7f3004009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 -- 172.21.15.104:0/3713917178 shutdown_connections 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 --2- 172.21.15.104:0/3713917178 >> v2:172.21.15.114:3300/0 conn(0x7f30100ae230 0x7f301006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 --2- 172.21.15.104:0/3713917178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f301015c730 0x7f30100cd1e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 -- 172.21.15.104:0/3713917178 >> 172.21.15.104:0/3713917178 conn(0x7f301006b8a0 msgr2=0x7f30100cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 -- 172.21.15.104:0/3713917178 shutdown_connections 2023-06-10T18:45:12.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.137+0000 7f30149b7700 1 -- 172.21.15.104:0/3713917178 wait complete. 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.138+0000 7f30149b7700 1 Processor -- start 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.138+0000 7f30149b7700 1 -- start start 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 0x7f30100c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 0x7f301006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30100b48b0 con 0x7f30100cd720 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30100b4010 con 0x7f301015c730 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f30149b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30100b42f0 con 0x7f30100ae230 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f300dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f300dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56290/0 (socket says 172.21.15.104:56290) 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f300dd9b700 1 -- 172.21.15.104:0/761769651 learned_addr learned my addr 172.21.15.104:0/761769651 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.139+0000 7f300ed9d700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 0x7f301006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300e59c700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 0x7f30100c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 -- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 msgr2=0x7f30100c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 0x7f30100c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 -- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 msgr2=0x7f301006e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 0x7f301006e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 -- 172.21.15.104:0/761769651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004009500 con 0x7f30100cd720 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300e59c700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 0x7f30100c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300ed9d700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 0x7f301006e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:12.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f300dd9b700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f300000e940 tx=0x7f300000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000005cb0 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000005e10 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30100b45d0 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000007290 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.140+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30100b3d20 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.141+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30100cd030 con 0x7f30100cd720 2023-06-10T18:45:12.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.141+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f30000073f0 con 0x7f30100cd720 2023-06-10T18:45:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.142+0000 7f2ffeffd700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 0x7f2ff006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.142+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f300008b7e0 con 0x7f30100cd720 2023-06-10T18:45:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.142+0000 7f300e59c700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 0x7f2ff006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.142+0000 7f300e59c700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 0x7f2ff006d190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3004009840 tx=0x7f3004008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.146+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f30000566b0 con 0x7f30100cd720 2023-06-10T18:45:12.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.329+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3010158500 con 0x7f30100cd720 2023-06-10T18:45:12.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.329+0000 7f2ffeffd700 1 -- 172.21.15.104:0/761769651 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f300001f020 con 0x7f30100cd720 2023-06-10T18:45:12.335 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:12.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 msgr2=0x7f2ff006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 0x7f2ff006d190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3004009840 tx=0x7f3004008070 comp rx=0 tx=0).stop 2023-06-10T18:45:12.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 msgr2=0x7f30100b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f300000e940 tx=0x7f300000ed00 comp rx=0 tx=0).stop 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 shutdown_connections 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ff006ace0 0x7f2ff006d190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f301015c730 0x7f301006e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30100ae230 0x7f30100c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 --2- 172.21.15.104:0/761769651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30100cd720 0x7f30100b37e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 >> 172.21.15.104:0/761769651 conn(0x7f301006b8a0 msgr2=0x7f30100ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 shutdown_connections 2023-06-10T18:45:12.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.331+0000 7f30149b7700 1 -- 172.21.15.104:0/761769651 wait complete. 2023-06-10T18:45:12.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:12.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.466+0000 7fd3f9679700 1 Processor -- start 2023-06-10T18:45:12.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.466+0000 7fd3f9679700 1 -- start start 2023-06-10T18:45:12.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f9679700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3f40d04f0 0x7fd3f40d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f9679700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f40af530 con 0x7fd3f40d25c0 2023-06-10T18:45:12.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f9679700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f40af810 con 0x7fd3f40b05d0 2023-06-10T18:45:12.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f9679700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40afaf0 con 0x7fd3f40d04f0 2023-06-10T18:45:12.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f27fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd3f40d25c0 0x7fd3f40d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54128/0 (socket says 172.21.15.104:54128) 2023-06-10T18:45:12.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f27fc700 1 -- 172.21.15.104:0/3595992819 learned_addr learned my addr 172.21.15.104:0/3595992819 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:12.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f2ffd700 1 --2- 172.21.15.104:0/3595992819 >> v2:172.21.15.114:3300/0 conn(0x7fd3f40d04f0 0x7fd3f40d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091194688 0 0) 0x7fd3f40af530 con 0x7fd3f40d25c0 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc0035b0 con 0x7fd3f40d25c0 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960164530 0 0) 0x7fd3dc0035b0 con 0x7fd3f40d25c0 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 >> v2:172.21.15.114:3300/0 conn(0x7fd3f40d04f0 msgr2=0x7fd3f40d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 --2- 172.21.15.104:0/3595992819 >> v2:172.21.15.114:3300/0 conn(0x7fd3f40d04f0 0x7fd3f40d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 >> v1:172.21.15.114:6789/0 conn(0x7fd3f40b05d0 legacy=0x7fd3f40b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40afdd0 con 0x7fd3f40d25c0 2023-06-10T18:45:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd3e4001670 con 0x7fd3f40d25c0 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.467+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 >> v1:172.21.15.104:6789/0 conn(0x7fd3f40d25c0 legacy=0x7fd3f40d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3dc003a00 0x7fd3dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3dc005200 0x7fd3dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40afaf0 con 0x7fd3dc004850 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00c110 con 0x7fd3dc005200 2023-06-10T18:45:12.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00c2d0 con 0x7fd3dc003a00 2023-06-10T18:45:12.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f2ffd700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3dc003a00 0x7fd3dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f37fe700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3dc005200 0x7fd3dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 -- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3dc003a00 msgr2=0x7fd3dc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:12.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3dc003a00 0x7fd3dc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 -- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3dc005200 msgr2=0x7fd3dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:12.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3dc005200 0x7fd3dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f37fe700 1 -- 172.21.15.104:0/3595992819 reap_dead start 2023-06-10T18:45:12.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 -- 172.21.15.104:0/3595992819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0035b0 con 0x7fd3dc004850 2023-06-10T18:45:12.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3dc004cc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c240 tx=0x7fd3dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d9b423901fa481 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4007c10 con 0x7fd3dc004850 2023-06-10T18:45:12.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e400cde0 con 0x7fd3dc004850 2023-06-10T18:45:12.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.468+0000 7fd3f1ffb700 1 -- 172.21.15.104:0/3595992819 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4005ba0 con 0x7fd3dc004850 2023-06-10T18:45:12.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 -- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 msgr2=0x7fd3dc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3dc004cc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c240 tx=0x7fd3dc005880 comp rx=0 tx=0).stop 2023-06-10T18:45:12.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 -- 172.21.15.104:0/3595992819 shutdown_connections 2023-06-10T18:45:12.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 --2- 172.21.15.104:0/3595992819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3dc004cc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 -- 172.21.15.104:0/3595992819 >> 172.21.15.104:0/3595992819 conn(0x7fd3f4071ec0 msgr2=0x7fd3f409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 -- 172.21.15.104:0/3595992819 shutdown_connections 2023-06-10T18:45:12.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.469+0000 7fd3f9679700 1 -- 172.21.15.104:0/3595992819 wait complete. 2023-06-10T18:45:12.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.470+0000 7fd3f9679700 1 Processor -- start 2023-06-10T18:45:12.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.470+0000 7fd3f9679700 1 -- start start 2023-06-10T18:45:12.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3f41e59f0 0x7fd3f40b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 0x7fd3f4154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4158bb0 con 0x7fd3dc004850 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40c9f70 con 0x7fd3f41e59f0 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f9679700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41e6240 con 0x7fd3f40ca390 2023-06-10T18:45:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56308/0 (socket says 172.21.15.104:56308) 2023-06-10T18:45:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 -- 172.21.15.104:0/2314748726 learned_addr learned my addr 172.21.15.104:0/2314748726 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3f41e59f0 0x7fd3f40b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f37fe700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 0x7fd3f4154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 -- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 msgr2=0x7fd3f4154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 0x7fd3f4154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 -- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3f41e59f0 msgr2=0x7fd3f40b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3f41e59f0 0x7fd3f40b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 -- 172.21.15.104:0/2314748726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f41e6520 con 0x7fd3dc004850 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f37fe700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 0x7fd3f4154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3f2ffd700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8011a70 tx=0x7fd3e8011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.471+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8005a20 con 0x7fd3dc004850 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.472+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e8005b80 con 0x7fd3dc004850 2023-06-10T18:45:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.472+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f40c9b90 con 0x7fd3dc004850 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.472+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e800f940 con 0x7fd3dc004850 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.472+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f40b84a0 con 0x7fd3dc004850 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.473+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f40b86f0 con 0x7fd3dc004850 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.473+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd3e800faa0 con 0x7fd3dc004850 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.473+0000 7fd3db7fe700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 0x7fd3e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.473+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd3e8089c20 con 0x7fd3dc004850 2023-06-10T18:45:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.473+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 0x7fd3e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.474+0000 7fd3f27fc700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 0x7fd3e006d190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4001930 tx=0x7fd3e4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.478+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd3e80549a0 con 0x7fd3dc004850 2023-06-10T18:45:12.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.666+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd3f40c99d0 con 0x7fd3dc004850 2023-06-10T18:45:12.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.667+0000 7fd3db7fe700 1 -- 172.21.15.104:0/2314748726 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd3e8022020 con 0x7fd3dc004850 2023-06-10T18:45:12.672 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 msgr2=0x7fd3e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 0x7fd3e006d190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4001930 tx=0x7fd3e4007000 comp rx=0 tx=0).stop 2023-06-10T18:45:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 msgr2=0x7fd3f4155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8011a70 tx=0x7fd3e8011d80 comp rx=0 tx=0).stop 2023-06-10T18:45:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 shutdown_connections 2023-06-10T18:45:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3f40ca390 0x7fd3f4154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3dc004850 0x7fd3f4155bd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd3e006ace0 0x7fd3e006d190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 --2- 172.21.15.104:0/2314748726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3f41e59f0 0x7fd3f40b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 >> 172.21.15.104:0/2314748726 conn(0x7fd3f4071ec0 msgr2=0x7fd3f4157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 shutdown_connections 2023-06-10T18:45:12.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.669+0000 7fd3f9679700 1 -- 172.21.15.104:0/2314748726 wait complete. 2023-06-10T18:45:12.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:12.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.804+0000 7f9473bb1700 1 Processor -- start 2023-06-10T18:45:12.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.804+0000 7f9473bb1700 1 -- start start 2023-06-10T18:45:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.805+0000 7f9473bb1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.805+0000 7f9473bb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.805+0000 7f9473bb1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c06e250 con 0x7f946c06e6e0 2023-06-10T18:45:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.805+0000 7f9473bb1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c15c950 con 0x7f946c15c580 2023-06-10T18:45:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.805+0000 7f9473bb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0b1e80 con 0x7f946c06cde0 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947194d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947194d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:43818/0 (socket says 172.21.15.104:43818) 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947114c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f946c06e6e0 0x7f946c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:59990/0 (socket says 172.21.15.104:59990) 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947194d700 1 -- 172.21.15.104:0/563694652 learned_addr learned my addr 172.21.15.104:0/563694652 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:12.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 -- 172.21.15.104:0/563694652 >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 msgr2=0x7f946c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 --2- 172.21.15.104:0/563694652 >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 -- 172.21.15.104:0/563694652 >> v1:172.21.15.114:6789/0 conn(0x7f946c06e6e0 legacy=0x7f946c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 -- 172.21.15.104:0/563694652 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c06d920 con 0x7f946c06cde0 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947194d700 1 --2- 172.21.15.104:0/563694652 >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947214e700 1 --2- 172.21.15.104:0/563694652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c0ce0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f946400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70e7e9fb4fef127 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947094b700 1 -- 172.21.15.104:0/563694652 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400e070 con 0x7f946c06cde0 2023-06-10T18:45:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947094b700 1 -- 172.21.15.104:0/563694652 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946400bb20 con 0x7f946c06cde0 2023-06-10T18:45:12.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.806+0000 7f947094b700 1 -- 172.21.15.104:0/563694652 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464012430 con 0x7f946c06cde0 2023-06-10T18:45:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 -- 172.21.15.104:0/563694652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 msgr2=0x7f946c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 --2- 172.21.15.104:0/563694652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c0ce0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f946400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 -- 172.21.15.104:0/563694652 shutdown_connections 2023-06-10T18:45:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 --2- 172.21.15.104:0/563694652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c0ce0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 --2- 172.21.15.104:0/563694652 >> v2:172.21.15.114:3300/0 conn(0x7f946c15c580 0x7f946c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 -- 172.21.15.104:0/563694652 >> 172.21.15.104:0/563694652 conn(0x7f946c06b8a0 msgr2=0x7f946c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 -- 172.21.15.104:0/563694652 shutdown_connections 2023-06-10T18:45:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.807+0000 7f9473bb1700 1 -- 172.21.15.104:0/563694652 wait complete. 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.808+0000 7f9473bb1700 1 Processor -- start 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.808+0000 7f9473bb1700 1 -- start start 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f946c06e6e0 0x7f946c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c158b60 con 0x7f946c15c580 2023-06-10T18:45:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0ce270 con 0x7f946c06e6e0 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f9473bb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0ce550 con 0x7f946c06cde0 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56324/0 (socket says 172.21.15.104:56324) 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 -- 172.21.15.104:0/1262771731 learned_addr learned my addr 172.21.15.104:0/1262771731 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947194d700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947114c700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f946c06e6e0 0x7f946c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 -- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 msgr2=0x7f946c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 -- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f946c06e6e0 msgr2=0x7f946c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f946c06e6e0 0x7f946c0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.809+0000 7f947214e700 1 -- 172.21.15.104:0/1262771731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c009500 con 0x7f946c15c580 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f947214e700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9464014040 tx=0x7f9464004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464012430 con 0x7f946c15c580 2023-06-10T18:45:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464020070 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464009a70 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640098f0 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.810+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c0ceb10 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.811+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c074050 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.811+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9464004040 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.812+0000 7f94627fc700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 0x7f945006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.812+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9464022070 con 0x7f946c15c580 2023-06-10T18:45:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.812+0000 7f947194d700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 0x7f945006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.812+0000 7f947194d700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 0x7f945006d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b26d0 tx=0x7f945c009290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:12.816+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9464056540 con 0x7f946c15c580 2023-06-10T18:45:13.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.005+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f946c066f10 con 0x7f946c15c580 2023-06-10T18:45:13.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.006+0000 7f94627fc700 1 -- 172.21.15.104:0/1262771731 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9464059b60 con 0x7f946c15c580 2023-06-10T18:45:13.011 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:13.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 msgr2=0x7f945006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 0x7f945006d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f946c0b26d0 tx=0x7f945c009290 comp rx=0 tx=0).stop 2023-06-10T18:45:13.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 msgr2=0x7f946c147690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9464014040 tx=0x7f9464004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 shutdown_connections 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f946c15c580 0x7f946c147690 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f946c06cde0 0x7f946c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945006ace0 0x7f945006d190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 --2- 172.21.15.104:0/1262771731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f946c06e6e0 0x7f946c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 >> 172.21.15.104:0/1262771731 conn(0x7f946c06b8a0 msgr2=0x7f946c0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 shutdown_connections 2023-06-10T18:45:13.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.008+0000 7f9473bb1700 1 -- 172.21.15.104:0/1262771731 wait complete. 2023-06-10T18:45:13.024 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.143+0000 7f7195dae700 1 Processor -- start 2023-06-10T18:45:13.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.143+0000 7f7195dae700 1 -- start start 2023-06-10T18:45:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f7195dae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f719015c500 0x7f71900cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f7195dae700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f71900cf200 0x7f7190069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f7195dae700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719015c8d0 con 0x7f719006a3b0 2023-06-10T18:45:13.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f7195dae700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71900aef20 con 0x7f719015c500 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f7195dae700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719006abd0 con 0x7f71900cf200 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f718f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f719015c500 0x7f71900cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f718f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f719015c500 0x7f71900cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56328/0 (socket says 172.21.15.104:56328) 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.144+0000 7f718f7fe700 1 -- 172.21.15.104:0/2249853452 learned_addr learned my addr 172.21.15.104:0/2249853452 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718effd700 1 --2- 172.21.15.104:0/2249853452 >> v2:172.21.15.114:3300/0 conn(0x7f71900cf200 0x7f7190069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718f7fe700 1 -- 172.21.15.104:0/2249853452 >> v2:172.21.15.114:3300/0 conn(0x7f71900cf200 msgr2=0x7f7190069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718f7fe700 1 --2- 172.21.15.104:0/2249853452 >> v2:172.21.15.114:3300/0 conn(0x7f71900cf200 0x7f7190069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718f7fe700 1 -- 172.21.15.104:0/2249853452 >> v1:172.21.15.114:6789/0 conn(0x7f719006a3b0 legacy=0x7f719006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718f7fe700 1 -- 172.21.15.104:0/2249853452 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719006a8f0 con 0x7f719015c500 2023-06-10T18:45:13.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718f7fe700 1 --2- 172.21.15.104:0/2249853452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719015c500 0x7f71900cecc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f71900cbca0 tx=0x7f7184009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2afa6b6691806ac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718e7fc700 1 -- 172.21.15.104:0/2249853452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184005e30 con 0x7f719015c500 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718e7fc700 1 -- 172.21.15.104:0/2249853452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7184004030 con 0x7f719015c500 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f718e7fc700 1 -- 172.21.15.104:0/2249853452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71840043e0 con 0x7f719015c500 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f7195dae700 1 -- 172.21.15.104:0/2249853452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719015c500 msgr2=0x7f71900cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.145+0000 7f7195dae700 1 --2- 172.21.15.104:0/2249853452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719015c500 0x7f71900cecc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f71900cbca0 tx=0x7f7184009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 -- 172.21.15.104:0/2249853452 shutdown_connections 2023-06-10T18:45:13.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 --2- 172.21.15.104:0/2249853452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719015c500 0x7f71900cecc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 --2- 172.21.15.104:0/2249853452 >> v2:172.21.15.114:3300/0 conn(0x7f71900cf200 0x7f7190069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 -- 172.21.15.104:0/2249853452 >> 172.21.15.104:0/2249853452 conn(0x7f719006b8a0 msgr2=0x7f71900cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 -- 172.21.15.104:0/2249853452 shutdown_connections 2023-06-10T18:45:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.146+0000 7f7195dae700 1 -- 172.21.15.104:0/2249853452 wait complete. 2023-06-10T18:45:13.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.147+0000 7f7195dae700 1 Processor -- start 2023-06-10T18:45:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.147+0000 7f7195dae700 1 -- start start 2023-06-10T18:45:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f719006a3b0 0x7f7190155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 0x7f71900c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71900c42c0 con 0x7f71900cf200 2023-06-10T18:45:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71900b3810 con 0x7f719006a3b0 2023-06-10T18:45:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71900c4e20 con 0x7f719015c500 2023-06-10T18:45:13.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56334/0 (socket says 172.21.15.104:56334) 2023-06-10T18:45:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 -- 172.21.15.104:0/4249563228 learned_addr learned my addr 172.21.15.104:0/4249563228 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718ffff700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 0x7f71900c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718f7fe700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f719006a3b0 0x7f7190155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 -- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 msgr2=0x7f71900c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 0x7f71900c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 -- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f719006a3b0 msgr2=0x7f7190155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f719006a3b0 0x7f7190155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 -- 172.21.15.104:0/4249563228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184009500 con 0x7f71900cf200 2023-06-10T18:45:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718ffff700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 0x7f71900c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f718effd700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f718000d800 tx=0x7f718000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800125f0 con 0x7f71900cf200 2023-06-10T18:45:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180012750 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180012b00 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71901560c0 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.148+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71900c47d0 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.149+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71900c2f10 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.150+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7180012c60 con 0x7f71900cf200 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.150+0000 7f7177fff700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 0x7f717c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.150+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7180089ba0 con 0x7f71900cf200 2023-06-10T18:45:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.150+0000 7f718f7fe700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 0x7f717c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.151+0000 7f718f7fe700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 0x7f717c06d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f71900cbca0 tx=0x7f7184008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.154+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7180054920 con 0x7f71900cf200 2023-06-10T18:45:13.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.342+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7190074050 con 0x7f71900cf200 2023-06-10T18:45:13.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.342+0000 7f7177fff700 1 -- 172.21.15.104:0/4249563228 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7180057f40 con 0x7f71900cf200 2023-06-10T18:45:13.347 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:13.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 msgr2=0x7f717c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 0x7f717c06d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f71900cbca0 tx=0x7f7184008070 comp rx=0 tx=0).stop 2023-06-10T18:45:13.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 msgr2=0x7f71900b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f718000d800 tx=0x7f718000dbc0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 shutdown_connections 2023-06-10T18:45:13.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717c06ace0 0x7f717c06d190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f719006a3b0 0x7f7190155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f719015c500 0x7f71900c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 --2- 172.21.15.104:0/4249563228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71900cf200 0x7f71900b2e50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 >> 172.21.15.104:0/4249563228 conn(0x7f719006b8a0 msgr2=0x7f7190072a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 shutdown_connections 2023-06-10T18:45:13.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.344+0000 7f7195dae700 1 -- 172.21.15.104:0/4249563228 wait complete. 2023-06-10T18:45:13.362 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:13.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.480+0000 7eff54c9a700 1 Processor -- start 2023-06-10T18:45:13.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.481+0000 7eff54c9a700 1 -- start start 2023-06-10T18:45:13.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff54c9a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff500cb210 0x7eff500cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff54c9a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff500cf0b0 0x7eff500cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff54c9a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff500cc9f0 con 0x7eff500cd720 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff54c9a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff500cbbd0 con 0x7eff500cf0b0 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff54c9a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff500cdb20 con 0x7eff500cb210 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff500cf0b0 0x7eff500cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7eff500cd720 0x7eff500c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54172/0 (socket says 172.21.15.104:54172) 2023-06-10T18:45:13.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4ed9d700 1 -- 172.21.15.104:0/2551084427 learned_addr learned my addr 172.21.15.104:0/2551084427 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4e59c700 1 --2- 172.21.15.104:0/2551084427 >> v2:172.21.15.114:3300/0 conn(0x7eff500cb210 0x7eff500cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 -- 172.21.15.104:0/2551084427 >> v2:172.21.15.114:3300/0 conn(0x7eff500cb210 msgr2=0x7eff500cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 --2- 172.21.15.104:0/2551084427 >> v2:172.21.15.114:3300/0 conn(0x7eff500cb210 0x7eff500cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 -- 172.21.15.104:0/2551084427 >> v1:172.21.15.104:6789/0 conn(0x7eff500cd720 legacy=0x7eff500c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4d59a700 1 -- 172.21.15.104:0/2551084427 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711085088 0 0) 0x7eff500cc9f0 con 0x7eff500cd720 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 -- 172.21.15.104:0/2551084427 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff500cc100 con 0x7eff500cf0b0 2023-06-10T18:45:13.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.482+0000 7eff4dd9b700 1 --2- 172.21.15.104:0/2551084427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff500cd1e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7eff500b21a0 tx=0x7eff40009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f23f0bec4b21c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff4d59a700 1 -- 172.21.15.104:0/2551084427 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4000f8a0 con 0x7eff500cf0b0 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff4d59a700 1 -- 172.21.15.104:0/2551084427 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4000fa00 con 0x7eff500cf0b0 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff4d59a700 1 -- 172.21.15.104:0/2551084427 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4000fd90 con 0x7eff500cf0b0 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 -- 172.21.15.104:0/2551084427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 msgr2=0x7eff500cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 --2- 172.21.15.104:0/2551084427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff500cd1e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7eff500b21a0 tx=0x7eff40009b20 comp rx=0 tx=0).stop 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 -- 172.21.15.104:0/2551084427 shutdown_connections 2023-06-10T18:45:13.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 --2- 172.21.15.104:0/2551084427 >> v2:172.21.15.114:3300/0 conn(0x7eff500cb210 0x7eff500cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 --2- 172.21.15.104:0/2551084427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff500cd1e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 -- 172.21.15.104:0/2551084427 >> 172.21.15.104:0/2551084427 conn(0x7eff5006b8a0 msgr2=0x7eff500cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 -- 172.21.15.104:0/2551084427 shutdown_connections 2023-06-10T18:45:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.483+0000 7eff54c9a700 1 -- 172.21.15.104:0/2551084427 wait complete. 2023-06-10T18:45:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 Processor -- start 2023-06-10T18:45:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 -- start start 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff500cb210 0x7eff500c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff500cd720 0x7eff50154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff500c5f90 con 0x7eff500cf0b0 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff501544b0 con 0x7eff500cd720 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff54c9a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff501538a0 con 0x7eff500cb210 2023-06-10T18:45:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff4ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff4ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56350/0 (socket says 172.21.15.104:56350) 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.485+0000 7eff4ed9d700 1 -- 172.21.15.104:0/259479311 learned_addr learned my addr 172.21.15.104:0/259479311 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4e59c700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff500cb210 0x7eff500c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4dd9b700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff500cd720 0x7eff50154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 -- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff500cb210 msgr2=0x7eff500c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff500cb210 0x7eff500c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 -- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff500cd720 msgr2=0x7eff50154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff500cd720 0x7eff50154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 -- 172.21.15.104:0/259479311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff400094a0 con 0x7eff500cf0b0 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff4ed9d700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7eff48002230 tx=0x7eff4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4800ac20 con 0x7eff500cf0b0 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4800ada0 con 0x7eff500cf0b0 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff50153b80 con 0x7eff500cf0b0 2023-06-10T18:45:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48007310 con 0x7eff500cf0b0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.486+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff50154140 con 0x7eff500cf0b0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.487+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff500cc8c0 con 0x7eff500cf0b0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.488+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff48007470 con 0x7eff500cf0b0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.488+0000 7eff46ffd700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 0x7eff3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.488+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7eff480896a0 con 0x7eff500cf0b0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.488+0000 7eff4e59c700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 0x7eff3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.489+0000 7eff4e59c700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 0x7eff3006d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff500cac50 tx=0x7eff38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.492+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff480544b0 con 0x7eff500cf0b0 2023-06-10T18:45:13.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.679+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7eff50074050 con 0x7eff500cf0b0 2023-06-10T18:45:13.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.679+0000 7eff46ffd700 1 -- 172.21.15.104:0/259479311 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7eff48018020 con 0x7eff500cf0b0 2023-06-10T18:45:13.684 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:13.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 msgr2=0x7eff3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 0x7eff3006d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7eff500cac50 tx=0x7eff38008040 comp rx=0 tx=0).stop 2023-06-10T18:45:13.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 msgr2=0x7eff501553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7eff48002230 tx=0x7eff4800a060 comp rx=0 tx=0).stop 2023-06-10T18:45:13.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 shutdown_connections 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff500cf0b0 0x7eff501553b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff500cb210 0x7eff500c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff3006ace0 0x7eff3006d190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 --2- 172.21.15.104:0/259479311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff500cd720 0x7eff50154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 >> 172.21.15.104:0/259479311 conn(0x7eff5006b8a0 msgr2=0x7eff500af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 shutdown_connections 2023-06-10T18:45:13.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.681+0000 7eff54c9a700 1 -- 172.21.15.104:0/259479311 wait complete. 2023-06-10T18:45:13.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:13.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.816+0000 7f2f55479700 1 Processor -- start 2023-06-10T18:45:13.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.816+0000 7f2f55479700 1 -- start start 2023-06-10T18:45:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f55479700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 0x7f2f500cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f55479700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f500b1c40 con 0x7f2f500b1e80 2023-06-10T18:45:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f55479700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f500b1680 con 0x7f2f500d0530 2023-06-10T18:45:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f55479700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500b1960 con 0x7f2f5006c260 2023-06-10T18:45:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2f500b1e80 0x7f2f500d2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54186/0 (socket says 172.21.15.104:54186) 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4f7fe700 1 -- 172.21.15.104:0/1740919721 learned_addr learned my addr 172.21.15.104:0/1740919721 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/1740919721 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 0x7f2f500cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381376495 0 0) 0x7f2f500b1c40 con 0x7f2f500b1e80 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f380035b0 con 0x7f2f500b1e80 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284986166 0 0) 0x7f2f500b1680 con 0x7f2f500d0530 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f500b1c40 con 0x7f2f500d0530 2023-06-10T18:45:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1541953667 0 0) 0x7f2f380035b0 con 0x7f2f500b1e80 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4e7fc700 1 -- 172.21.15.104:0/1740919721 >> v1:172.21.15.114:6789/0 conn(0x7f2f500d0530 legacy=0x7f2f500d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4e7fc700 1 -- 172.21.15.104:0/1740919721 >> v1:172.21.15.104:6789/0 conn(0x7f2f500b1e80 legacy=0x7f2f500d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4e7fc700 1 -- 172.21.15.104:0/1740919721 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5006b780 con 0x7f2f5006c260 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.817+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/1740919721 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 0x7f2f500cfd80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009870 tx=0x7f2f44009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51d9f47eb11ec29f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f480019f0 con 0x7f2f5006c260 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f44004030 con 0x7f2f5006c260 2023-06-10T18:45:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f4dffb700 1 -- 172.21.15.104:0/1740919721 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f440043c0 con 0x7f2f5006c260 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 -- 172.21.15.104:0/1740919721 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 msgr2=0x7f2f500cfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 --2- 172.21.15.104:0/1740919721 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 0x7f2f500cfd80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009870 tx=0x7f2f44009b80 comp rx=0 tx=0).stop 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 -- 172.21.15.104:0/1740919721 shutdown_connections 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 --2- 172.21.15.104:0/1740919721 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5006c260 0x7f2f500cfd80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 -- 172.21.15.104:0/1740919721 >> 172.21.15.104:0/1740919721 conn(0x7f2f50071ec0 msgr2=0x7f2f5009bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 -- 172.21.15.104:0/1740919721 shutdown_connections 2023-06-10T18:45:13.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.818+0000 7f2f55479700 1 -- 172.21.15.104:0/1740919721 wait complete. 2023-06-10T18:45:13.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.820+0000 7f2f55479700 1 Processor -- start 2023-06-10T18:45:13.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.820+0000 7f2f55479700 1 -- start start 2023-06-10T18:45:13.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 0x7f2f501f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 0x7f2f501f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f50156140 con 0x7f2f5006c260 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500b6440 con 0x7f2f500d0530 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500b6690 con 0x7f2f500b1e80 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56360/0 (socket says 172.21.15.104:56360) 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 -- 172.21.15.104:0/4035171534 learned_addr learned my addr 172.21.15.104:0/4035171534 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4f7fe700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 0x7f2f501f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 0x7f2f501f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 -- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 msgr2=0x7f2f501f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 0x7f2f501f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 -- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 msgr2=0x7f2f501f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 0x7f2f501f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 -- 172.21.15.104:0/4035171534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f440094a0 con 0x7f2f5006c260 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4f7fe700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 0x7f2f501f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 0x7f2f501f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f4effd700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c520 tx=0x7f2f4000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f400052d0 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40005430 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f400038c0 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.821+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f500b6970 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.822+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f500b6f30 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.822+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f50074050 con 0x7f2f5006c260 2023-06-10T18:45:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.823+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2f40003a20 con 0x7f2f5006c260 2023-06-10T18:45:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.823+0000 7f2f377fe700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 0x7f2f3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.823+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2f40088dd0 con 0x7f2f5006c260 2023-06-10T18:45:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.824+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 0x7f2f3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.824+0000 7f2f4e7fc700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 0x7f2f3c06d190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f5006c650 tx=0x7f2f44013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:13.828+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2f40053b50 con 0x7f2f5006c260 2023-06-10T18:45:14.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.029+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2f500d3870 con 0x7f2f5006c260 2023-06-10T18:45:14.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.029+0000 7f2f377fe700 1 -- 172.21.15.104:0/4035171534 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2f40057170 con 0x7f2f5006c260 2023-06-10T18:45:14.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 msgr2=0x7f2f3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 0x7f2f3c06d190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f5006c650 tx=0x7f2f44013040 comp rx=0 tx=0).stop 2023-06-10T18:45:14.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 msgr2=0x7f2f501e8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c520 tx=0x7f2f4000c8e0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 shutdown_connections 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f3c06ace0 0x7f2f3c06d190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f500d0530 0x7f2f501f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f5006c260 0x7f2f501e8070 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 --2- 172.21.15.104:0/4035171534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f500b1e80 0x7f2f501f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 >> 172.21.15.104:0/4035171534 conn(0x7f2f50071ec0 msgr2=0x7f2f5015a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:14.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 shutdown_connections 2023-06-10T18:45:14.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.031+0000 7f2f55479700 1 -- 172.21.15.104:0/4035171534 wait complete. 2023-06-10T18:45:14.040 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:14.048 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:45:14.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:14.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:14.091 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:14.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:45:14.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:14.092 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:45:14.173 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:14.173 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:14.191 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:14.191 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:14.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:14.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.405+0000 7f9a8ea13700 1 Processor -- start 2023-06-10T18:45:14.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.405+0000 7f9a8ea13700 1 -- start start 2023-06-10T18:45:14.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8ea13700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 0x7f9a880cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8ea13700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880c8de0 con 0x7f9a880c9270 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8ea13700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880ccc90 con 0x7f9a88158e50 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8ea13700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880cdd80 con 0x7f9a880caea0 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8cfb0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9a880c9270 0x7f9a880c96f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54200/0 (socket says 172.21.15.104:54200) 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a8cfb0700 1 -- 172.21.15.104:0/1186755887 learned_addr learned my addr 172.21.15.104:0/1186755887 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1186755887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 0x7f9a880cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644325093 0 0) 0x7f9a880c8de0 con 0x7f9a880c9270 2023-06-10T18:45:14.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a700035b0 con 0x7f9a880c9270 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703603093 0 0) 0x7f9a880ccc90 con 0x7f9a88158e50 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a880c8de0 con 0x7f9a88158e50 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378287168 0 0) 0x7f9a700035b0 con 0x7f9a880c9270 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a877fe700 1 -- 172.21.15.104:0/1186755887 >> v1:172.21.15.114:6789/0 conn(0x7f9a88158e50 legacy=0x7f9a880cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a877fe700 1 -- 172.21.15.104:0/1186755887 >> v1:172.21.15.104:6789/0 conn(0x7f9a880c9270 legacy=0x7f9a880c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.406+0000 7f9a877fe700 1 -- 172.21.15.104:0/1186755887 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880ccf30 con 0x7f9a880caea0 2023-06-10T18:45:14.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1186755887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 0x7f9a880cb2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a78009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d98470cda6057528 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80001970 con 0x7f9a880caea0 2023-06-10T18:45:14.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78004030 con 0x7f9a880caea0 2023-06-10T18:45:14.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a86ffd700 1 -- 172.21.15.104:0/1186755887 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780043a0 con 0x7f9a880caea0 2023-06-10T18:45:14.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1186755887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 msgr2=0x7f9a880cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1186755887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 0x7f9a880cb2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a78009b80 comp rx=0 tx=0).stop 2023-06-10T18:45:14.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1186755887 shutdown_connections 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1186755887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a880caea0 0x7f9a880cb2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1186755887 >> 172.21.15.104:0/1186755887 conn(0x7f9a8806ab00 msgr2=0x7f9a880b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1186755887 shutdown_connections 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.407+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1186755887 wait complete. 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 Processor -- start 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 -- start start 2023-06-10T18:45:14.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 0x7f9a880c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a88158e50 0x7f9a88156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88155880 con 0x7f9a880c9270 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a881583e0 con 0x7f9a880caea0 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.409+0000 7f9a8ea13700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88157ab0 con 0x7f9a88158e50 2023-06-10T18:45:14.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56374/0 (socket says 172.21.15.104:56374) 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 -- 172.21.15.104:0/1782101151 learned_addr learned my addr 172.21.15.104:0/1782101151 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 0x7f9a880c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a8cfb0700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a88158e50 0x7f9a88156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 -- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a88158e50 msgr2=0x7f9a88156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a88158e50 0x7f9a88156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 -- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 msgr2=0x7f9a880c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 0x7f9a880c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 -- 172.21.15.104:0/1782101151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094a0 con 0x7f9a880c9270 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 0x7f9a880c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a87fff700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c001eb0 tx=0x7f9a7c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c0052d0 con 0x7f9a880c9270 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c005450 con 0x7f9a880c9270 2023-06-10T18:45:14.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88157d90 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c0038c0 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.410+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a88157520 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.411+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a881586c0 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.411+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a7c003a20 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.412+0000 7f9a84ff9700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 0x7f9a7406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.412+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9a7c088d30 con 0x7f9a880c9270 2023-06-10T18:45:14.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.412+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 0x7f9a7406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.412+0000 7f9a877fe700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 0x7f9a7406d1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a780090b0 tx=0x7f9a78008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.416+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a7c053a20 con 0x7f9a880c9270 2023-06-10T18:45:14.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.606+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a88074050 con 0x7f9a880c9270 2023-06-10T18:45:14.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.607+0000 7f9a84ff9700 1 -- 172.21.15.104:0/1782101151 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f9a7c017020 con 0x7f9a880c9270 2023-06-10T18:45:14.612 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:14.613 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:14.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 msgr2=0x7f9a7406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 0x7f9a7406d1c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a780090b0 tx=0x7f9a78008030 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 msgr2=0x7f9a881552a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c001eb0 tx=0x7f9a7c00c860 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 shutdown_connections 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a88158e50 0x7f9a88156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a880c9270 0x7f9a881552a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a7406ad10 0x7f9a7406d1c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 --2- 172.21.15.104:0/1782101151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a880caea0 0x7f9a880c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 >> 172.21.15.104:0/1782101151 conn(0x7f9a8806ab00 msgr2=0x7f9a8806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:14.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 shutdown_connections 2023-06-10T18:45:14.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.610+0000 7f9a8ea13700 1 -- 172.21.15.104:0/1782101151 wait complete. 2023-06-10T18:45:14.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:14.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:14.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.747+0000 7f7c85836700 1 Processor -- start 2023-06-10T18:45:14.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 -- start start 2023-06-10T18:45:14.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c800d04f0 0x7f7c800d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c800d25c0 0x7f7c800d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c8006b510 0x7f7c8006b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c800d0e00 con 0x7f7c800d25c0 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c800af530 con 0x7f7c800d04f0 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c85836700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800af810 con 0x7f7c8006b510 2023-06-10T18:45:14.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.748+0000 7f7c7e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c800d25c0 0x7f7c800d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c800d25c0 0x7f7c800d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56384/0 (socket says 172.21.15.104:56384) 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 -- 172.21.15.104:0/3779696911 learned_addr learned my addr 172.21.15.104:0/3779696911 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7effd700 1 --2- 172.21.15.104:0/3779696911 >> v2:172.21.15.114:3300/0 conn(0x7f7c800d04f0 0x7f7c800d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7f7fe700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c8006b510 0x7f7c8006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 -- 172.21.15.104:0/3779696911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c8006b510 msgr2=0x7f7c8006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c8006b510 0x7f7c8006b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 -- 172.21.15.104:0/3779696911 >> v2:172.21.15.114:3300/0 conn(0x7f7c800d04f0 msgr2=0x7f7c800d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:14.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/3779696911 >> v2:172.21.15.114:3300/0 conn(0x7f7c800d04f0 0x7f7c800d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 -- 172.21.15.104:0/3779696911 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800afaf0 con 0x7f7c800d25c0 2023-06-10T18:45:14.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c800d25c0 0x7f7c800d08c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c80066690 tx=0x7f7c74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28570890328e6d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7dffb700 1 -- 172.21.15.104:0/3779696911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400b870 con 0x7f7c800d25c0 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7dffb700 1 -- 172.21.15.104:0/3779696911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7400b9d0 con 0x7f7c800d25c0 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.749+0000 7f7c7dffb700 1 -- 172.21.15.104:0/3779696911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400bd60 con 0x7f7c800d25c0 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 -- 172.21.15.104:0/3779696911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c800d25c0 msgr2=0x7f7c800d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c800d25c0 0x7f7c800d08c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c80066690 tx=0x7f7c74009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 -- 172.21.15.104:0/3779696911 shutdown_connections 2023-06-10T18:45:14.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c8006b510 0x7f7c8006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 --2- 172.21.15.104:0/3779696911 >> v2:172.21.15.114:3300/0 conn(0x7f7c800d04f0 0x7f7c800d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 --2- 172.21.15.104:0/3779696911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c800d25c0 0x7f7c800d08c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 -- 172.21.15.104:0/3779696911 >> 172.21.15.104:0/3779696911 conn(0x7f7c80071ec0 msgr2=0x7f7c8009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 -- 172.21.15.104:0/3779696911 shutdown_connections 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.750+0000 7f7c85836700 1 -- 172.21.15.104:0/3779696911 wait complete. 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.751+0000 7f7c85836700 1 Processor -- start 2023-06-10T18:45:14.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.751+0000 7f7c85836700 1 -- start start 2023-06-10T18:45:14.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c800d04f0 0x7f7c80156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c800d25c0 0x7f7c801574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800cee00 con 0x7f7c8006b510 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800cf0e0 con 0x7f7c800d25c0 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c85836700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800cf3c0 con 0x7f7c800d04f0 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56386/0 (socket says 172.21.15.104:56386) 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 -- 172.21.15.104:0/1507737312 learned_addr learned my addr 172.21.15.104:0/1507737312 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c800d04f0 0x7f7c80156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7f7fe700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c800d25c0 0x7f7c801574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 -- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c800d04f0 msgr2=0x7f7c80156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c800d04f0 0x7f7c80156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 -- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c800d25c0 msgr2=0x7f7c801574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c800d25c0 0x7f7c801574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 -- 172.21.15.104:0/1507737312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74009550 con 0x7f7c8006b510 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.752+0000 7f7c7effd700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e730 tx=0x7f7c7000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.753+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000c070 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.753+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c70013610 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.753+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c800cf6a0 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.753+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c700139c0 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.753+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c800cfa30 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.754+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c80067010 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.754+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7c70013b20 con 0x7f7c8006b510 2023-06-10T18:45:14.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.754+0000 7f7c67fff700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 0x7f7c6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.754+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7c70089150 con 0x7f7c8006b510 2023-06-10T18:45:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.755+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 0x7f7c6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.755+0000 7f7c7e7fc700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 0x7f7c6c06d190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c74000c00 tx=0x7f7c74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:14.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.759+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7c70053ed0 con 0x7f7c8006b510 2023-06-10T18:45:14.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.944+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c80067810 con 0x7f7c8006b510 2023-06-10T18:45:14.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.944+0000 7f7c67fff700 1 -- 172.21.15.104:0/1507737312 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c700574f0 con 0x7f7c8006b510 2023-06-10T18:45:14.949 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:14.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 msgr2=0x7f7c6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 0x7f7c6c06d190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7c74000c00 tx=0x7f7c74008030 comp rx=0 tx=0).stop 2023-06-10T18:45:14.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 msgr2=0x7f7c80158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:14.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e730 tx=0x7f7c7000ea40 comp rx=0 tx=0).stop 2023-06-10T18:45:14.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 shutdown_connections 2023-06-10T18:45:14.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c6c06ace0 0x7f7c6c06d190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c800d25c0 0x7f7c801574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c8006b510 0x7f7c80158330 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 --2- 172.21.15.104:0/1507737312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c800d04f0 0x7f7c80156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 >> 172.21.15.104:0/1507737312 conn(0x7f7c80071ec0 msgr2=0x7f7c8006b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 shutdown_connections 2023-06-10T18:45:14.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:14.946+0000 7f7c85836700 1 -- 172.21.15.104:0/1507737312 wait complete. 2023-06-10T18:45:14.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:15.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.081+0000 7fc104f4e700 1 Processor -- start 2023-06-10T18:45:15.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.081+0000 7fc104f4e700 1 -- start start 2023-06-10T18:45:15.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc104f4e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 0x7fc1000b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc104f4e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1000aec00 0x7fc1000b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc104f4e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1000ae7e0 con 0x7fc10006ea70 2023-06-10T18:45:15.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc104f4e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10006df90 con 0x7fc1000aec00 2023-06-10T18:45:15.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc104f4e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10006e270 con 0x7fc10015c6f0 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1000aec00 0x7fc1000b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1000aec00 0x7fc1000b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56396/0 (socket says 172.21.15.104:56396) 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 -- 172.21.15.104:0/2165857803 learned_addr learned my addr 172.21.15.104:0/2165857803 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fe59c700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 0x7fc1000b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 -- 172.21.15.104:0/2165857803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 msgr2=0x7fc1000b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 0x7fc1000b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 -- 172.21.15.104:0/2165857803 >> v1:172.21.15.114:6789/0 conn(0x7fc10006ea70 legacy=0x7fc1000ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 -- 172.21.15.104:0/2165857803 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10006e550 con 0x7fc1000aec00 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fe59c700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 0x7fc1000b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fed9d700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1000aec00 0x7fc1000b15b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800d150 tx=0x7fc0f800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8afd3a69c3eb6e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fd59a700 1 -- 172.21.15.104:0/2165857803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8010070 con 0x7fc1000aec00 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.082+0000 7fc0fd59a700 1 -- 172.21.15.104:0/2165857803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800dd80 con 0x7fc1000aec00 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc0fd59a700 1 -- 172.21.15.104:0/2165857803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800b310 con 0x7fc1000aec00 2023-06-10T18:45:15.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 -- 172.21.15.104:0/2165857803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1000aec00 msgr2=0x7fc1000b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1000aec00 0x7fc1000b15b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800d150 tx=0x7fc0f800d460 comp rx=0 tx=0).stop 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 -- 172.21.15.104:0/2165857803 shutdown_connections 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1000aec00 0x7fc1000b15b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 --2- 172.21.15.104:0/2165857803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc10015c6f0 0x7fc1000b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 -- 172.21.15.104:0/2165857803 >> 172.21.15.104:0/2165857803 conn(0x7fc10006b8a0 msgr2=0x7fc1000cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 -- 172.21.15.104:0/2165857803 shutdown_connections 2023-06-10T18:45:15.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.083+0000 7fc104f4e700 1 -- 172.21.15.104:0/2165857803 wait complete. 2023-06-10T18:45:15.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.084+0000 7fc104f4e700 1 Processor -- start 2023-06-10T18:45:15.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.084+0000 7fc104f4e700 1 -- start start 2023-06-10T18:45:15.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 0x7fc100151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 0x7fc1000b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1001f1ef0 con 0x7fc10006ea70 2023-06-10T18:45:15.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1001f5a00 con 0x7fc10015c6f0 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc104f4e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1001f5ce0 con 0x7fc1000aec00 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc0fe59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc0fe59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56398/0 (socket says 172.21.15.104:56398) 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc0fe59c700 1 -- 172.21.15.104:0/3239239822 learned_addr learned my addr 172.21.15.104:0/3239239822 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc0fdd9b700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 0x7fc100151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.085+0000 7fc0fed9d700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 0x7fc1000b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 -- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 msgr2=0x7fc100151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 0x7fc100151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 -- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 msgr2=0x7fc1000b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 0x7fc1000b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 -- 172.21.15.104:0/3239239822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f800bd40 con 0x7fc10006ea70 2023-06-10T18:45:15.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fdd9b700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 0x7fc100151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:15.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fed9d700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 0x7fc1000b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:15.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0fe59c700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c7f0 tx=0x7fc0f000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0005c20 con 0x7fc10006ea70 2023-06-10T18:45:15.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f0005d80 con 0x7fc10006ea70 2023-06-10T18:45:15.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1001f5fc0 con 0x7fc10006ea70 2023-06-10T18:45:15.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0011820 con 0x7fc10006ea70 2023-06-10T18:45:15.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.086+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1000b3ef0 con 0x7fc10006ea70 2023-06-10T18:45:15.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.087+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1001f2030 con 0x7fc10006ea70 2023-06-10T18:45:15.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.087+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc0f0011980 con 0x7fc10006ea70 2023-06-10T18:45:15.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.088+0000 7fc0eeffd700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 0x7fc0e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.088+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc0f0089680 con 0x7fc10006ea70 2023-06-10T18:45:15.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.088+0000 7fc0fdd9b700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 0x7fc0e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.088+0000 7fc0fdd9b700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 0x7fc0e006d190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc100157200 tx=0x7fc0f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.092+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc0f0057e10 con 0x7fc10006ea70 2023-06-10T18:45:15.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.278+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc100074050 con 0x7fc10006ea70 2023-06-10T18:45:15.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.278+0000 7fc0eeffd700 1 -- 172.21.15.104:0/3239239822 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc0f00579a0 con 0x7fc10006ea70 2023-06-10T18:45:15.283 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 msgr2=0x7fc0e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 0x7fc0e006d190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc100157200 tx=0x7fc0f4009040 comp rx=0 tx=0).stop 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 msgr2=0x7fc1000c3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c7f0 tx=0x7fc0f000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 shutdown_connections 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0e006ace0 0x7fc0e006d190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc10015c6f0 0x7fc1000b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc10006ea70 0x7fc1000c3930 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 --2- 172.21.15.104:0/3239239822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1000aec00 0x7fc100151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 >> 172.21.15.104:0/3239239822 conn(0x7fc10006b8a0 msgr2=0x7fc1001585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 shutdown_connections 2023-06-10T18:45:15.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.280+0000 7fc104f4e700 1 -- 172.21.15.104:0/3239239822 wait complete. 2023-06-10T18:45:15.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:15.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.415+0000 7f32632ef700 1 Processor -- start 2023-06-10T18:45:15.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.415+0000 7f32632ef700 1 -- start start 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f32632ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f325c15c580 0x7f325c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f32632ef700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f325c06e6e0 0x7f325c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f32632ef700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c06e250 con 0x7f325c06cde0 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f32632ef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c15c950 con 0x7f325c15c580 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f32632ef700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c0cf0e0 con 0x7f325c06e6e0 2023-06-10T18:45:15.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f325c15c580 0x7f325c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f325c15c580 0x7f325c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56406/0 (socket says 172.21.15.104:56406) 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 -- 172.21.15.104:0/803611377 learned_addr learned my addr 172.21.15.104:0/803611377 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326088a700 1 --2- 172.21.15.104:0/803611377 >> v2:172.21.15.114:3300/0 conn(0x7f325c06e6e0 0x7f325c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 -- 172.21.15.104:0/803611377 >> v2:172.21.15.114:3300/0 conn(0x7f325c06e6e0 msgr2=0x7f325c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 --2- 172.21.15.104:0/803611377 >> v2:172.21.15.114:3300/0 conn(0x7f325c06e6e0 0x7f325c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 -- 172.21.15.104:0/803611377 >> v1:172.21.15.114:6789/0 conn(0x7f325c06cde0 legacy=0x7f325c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 -- 172.21.15.104:0/803611377 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0b1e80 con 0x7f325c15c580 2023-06-10T18:45:15.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f326108b700 1 --2- 172.21.15.104:0/803611377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0b1070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f325c06af30 tx=0x7f3258009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e4aeacfb7f673a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f3253fff700 1 -- 172.21.15.104:0/803611377 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258005e30 con 0x7f325c15c580 2023-06-10T18:45:15.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f3253fff700 1 -- 172.21.15.104:0/803611377 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3258004030 con 0x7f325c15c580 2023-06-10T18:45:15.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.416+0000 7f3253fff700 1 -- 172.21.15.104:0/803611377 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580043e0 con 0x7f325c15c580 2023-06-10T18:45:15.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 -- 172.21.15.104:0/803611377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 msgr2=0x7f325c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 --2- 172.21.15.104:0/803611377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0b1070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f325c06af30 tx=0x7f3258009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 -- 172.21.15.104:0/803611377 shutdown_connections 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 --2- 172.21.15.104:0/803611377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0b1070 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 --2- 172.21.15.104:0/803611377 >> v2:172.21.15.114:3300/0 conn(0x7f325c06e6e0 0x7f325c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 -- 172.21.15.104:0/803611377 >> 172.21.15.104:0/803611377 conn(0x7f325c06b8a0 msgr2=0x7f325c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 -- 172.21.15.104:0/803611377 shutdown_connections 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.417+0000 7f32632ef700 1 -- 172.21.15.104:0/803611377 wait complete. 2023-06-10T18:45:15.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.418+0000 7f32632ef700 1 Processor -- start 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.418+0000 7f32632ef700 1 -- start start 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 0x7f325c0c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 0x7f325c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b4740 con 0x7f325c15c580 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b4a20 con 0x7f325c06e6e0 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f32632ef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b4d00 con 0x7f325c06cde0 2023-06-10T18:45:15.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56416/0 (socket says 172.21.15.104:56416) 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 -- 172.21.15.104:0/2514409903 learned_addr learned my addr 172.21.15.104:0/2514409903 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326088a700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 0x7f325c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 -- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 msgr2=0x7f325c0c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326108b700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 0x7f325c0c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 0x7f325c0c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 -- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 msgr2=0x7f325c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 0x7f325c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 -- 172.21.15.104:0/2514409903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3258009500 con 0x7f325c15c580 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326088a700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 0x7f325c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326108b700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 0x7f325c0c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.419+0000 7f326188c700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f325400cb50 tx=0x7f325400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.420+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254003c00 con 0x7f325c15c580 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.420+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254003d60 con 0x7f325c15c580 2023-06-10T18:45:15.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.420+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c0b4fe0 con 0x7f325c15c580 2023-06-10T18:45:15.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.420+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325400ab70 con 0x7f325c15c580 2023-06-10T18:45:15.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.420+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c0b2750 con 0x7f325c15c580 2023-06-10T18:45:15.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.421+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0b5350 con 0x7f325c15c580 2023-06-10T18:45:15.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.421+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f325400acd0 con 0x7f325c15c580 2023-06-10T18:45:15.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.421+0000 7f3251ffb700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 0x7f324006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.421+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f32540897c0 con 0x7f325c15c580 2023-06-10T18:45:15.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.422+0000 7f326108b700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 0x7f324006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.422+0000 7f326108b700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 0x7f324006d190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3258014040 tx=0x7f3258008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.426+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f32540545d0 con 0x7f325c15c580 2023-06-10T18:45:15.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.612+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f325c0af050 con 0x7f325c15c580 2023-06-10T18:45:15.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.612+0000 7f3251ffb700 1 -- 172.21.15.104:0/2514409903 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f325c0af050 con 0x7f325c15c580 2023-06-10T18:45:15.617 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:15.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 msgr2=0x7f324006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 0x7f324006d190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3258014040 tx=0x7f3258008070 comp rx=0 tx=0).stop 2023-06-10T18:45:15.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 msgr2=0x7f325c0c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f325400cb50 tx=0x7f325400cf10 comp rx=0 tx=0).stop 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 shutdown_connections 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f325c15c580 0x7f325c0c8e20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f325c06cde0 0x7f325c0c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f324006ace0 0x7f324006d190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 --2- 172.21.15.104:0/2514409903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f325c06e6e0 0x7f325c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.614+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 >> 172.21.15.104:0/2514409903 conn(0x7f325c06b8a0 msgr2=0x7f325c0c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.615+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 shutdown_connections 2023-06-10T18:45:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.615+0000 7f32632ef700 1 -- 172.21.15.104:0/2514409903 wait complete. 2023-06-10T18:45:15.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.750+0000 7fa5c3655700 1 Processor -- start 2023-06-10T18:45:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c3655700 1 -- start start 2023-06-10T18:45:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c3655700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa5bc071230 0x7fa5bc069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c3655700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc158cc0 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c3655700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0afee0 con 0x7fa5bc1588f0 2023-06-10T18:45:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c3655700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc06a7f0 con 0x7fa5bc071230 2023-06-10T18:45:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c1bf2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa5bc06a3b0 0x7fa5bc1623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54252/0 (socket says 172.21.15.104:54252) 2023-06-10T18:45:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.751+0000 7fa5c1bf2700 1 -- 172.21.15.104:0/2477549651 learned_addr learned my addr 172.21.15.104:0/2477549651 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2477549651 >> v2:172.21.15.114:3300/0 conn(0x7fa5bc071230 0x7fa5bc069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082677793 0 0) 0x7fa5bc158cc0 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a80035b0 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291315484 0 0) 0x7fa5a80035b0 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 >> v2:172.21.15.114:3300/0 conn(0x7fa5bc071230 msgr2=0x7fa5bc069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 --2- 172.21.15.104:0/2477549651 >> v2:172.21.15.114:3300/0 conn(0x7fa5bc071230 0x7fa5bc069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 >> v1:172.21.15.114:6789/0 conn(0x7fa5bc1588f0 legacy=0x7fa5bc0af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0b0120 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa5b40019f0 con 0x7fa5bc06a3b0 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 >> v1:172.21.15.104:6789/0 conn(0x7fa5bc06a3b0 legacy=0x7fa5bc1623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5a8003a00 0x7fa5a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5a8005200 0x7fa5a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc06a7f0 con 0x7fa5a8004850 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a800c080 con 0x7fa5a8005200 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a800c240 con 0x7fa5a8003a00 2023-06-10T18:45:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c1bf2700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5a8005200 0x7fa5a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c0bf0700 1 -- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5a8003a00 msgr2=0x7fa5a8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c13f1700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5a8003a00 0x7fa5a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.752+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5a8003a00 0x7fa5a8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c0bf0700 1 -- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5a8005200 msgr2=0x7fa5a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5a8005200 0x7fa5a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c1bf2700 1 -- 172.21.15.104:0/2477549651 reap_dead start 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c0bf0700 1 -- 172.21.15.104:0/2477549651 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80035b0 con 0x7fa5a8004850 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c13f1700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5a8003a00 0x7fa5a8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5a8004cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009b10 tx=0x7fa5a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6795ecc92982ff99 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8005a20 con 0x7fa5a8004850 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8005b80 con 0x7fa5a8004850 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5b3fff700 1 -- 172.21.15.104:0/2477549651 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800e6c0 con 0x7fa5a8004850 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 -- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 msgr2=0x7fa5a8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5a8004cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009b10 tx=0x7fa5a8005880 comp rx=0 tx=0).stop 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 -- 172.21.15.104:0/2477549651 shutdown_connections 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2477549651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5a8004cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 -- 172.21.15.104:0/2477549651 >> 172.21.15.104:0/2477549651 conn(0x7fa5bc0ce940 msgr2=0x7fa5bc0ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 -- 172.21.15.104:0/2477549651 shutdown_connections 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.753+0000 7fa5c3655700 1 -- 172.21.15.104:0/2477549651 wait complete. 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 Processor -- start 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 -- start start 2023-06-10T18:45:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 0x7fa5bc14c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5bc0b2da0 0x7fa5bc0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc144ba0 con 0x7fa5a8004850 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0b1d00 con 0x7fa5bc0b17b0 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c3655700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0b1fe0 con 0x7fa5bc0b2da0 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c13f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c13f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56432/0 (socket says 172.21.15.104:56432) 2023-06-10T18:45:15.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.755+0000 7fa5c13f1700 1 -- 172.21.15.104:0/2166402715 learned_addr learned my addr 172.21.15.104:0/2166402715 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 -- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5bc0b2da0 msgr2=0x7fa5bc0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5bc0b2da0 0x7fa5bc0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 -- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 msgr2=0x7fa5bc14c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 0x7fa5bc14c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 0x7fa5bc14c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 -- 172.21.15.104:0/2166402715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0b22c0 con 0x7fa5a8004850 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 0x7fa5bc14c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c13f1700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac001eb0 tx=0x7fa5ac00d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00b070 con 0x7fa5a8004850 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac0054e0 con 0x7fa5a8004850 2023-06-10T18:45:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc0b25a0 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac006aa0 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.756+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc14cf20 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.757+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc14d0e0 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.757+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa5ac006c00 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.758+0000 7fa5b1ffb700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 0x7fa5a006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.758+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa5ac088970 con 0x7fa5a8004850 2023-06-10T18:45:15.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.758+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 0x7fa5a006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.758+0000 7fa5c0bf0700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 0x7fa5a006d1c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0c9750 tx=0x7fa5b8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:15.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.762+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5bc14d0e0 con 0x7fa5a8004850 2023-06-10T18:45:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.949+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5bc0b2ac0 con 0x7fa5a8004850 2023-06-10T18:45:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.949+0000 7fa5b1ffb700 1 -- 172.21.15.104:0/2166402715 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa5bc0b2ac0 con 0x7fa5a8004850 2023-06-10T18:45:15.954 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:15.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 msgr2=0x7fa5a006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 0x7fa5a006d1c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0c9750 tx=0x7fa5b8008030 comp rx=0 tx=0).stop 2023-06-10T18:45:15.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 msgr2=0x7fa5bc144660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:15.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac001eb0 tx=0x7fa5ac00d850 comp rx=0 tx=0).stop 2023-06-10T18:45:15.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 shutdown_connections 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5bc0b2da0 0x7fa5bc0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5a8004850 0x7fa5bc144660 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5a006ad10 0x7fa5a006d1c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 --2- 172.21.15.104:0/2166402715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5bc0b17b0 0x7fa5bc14c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 >> 172.21.15.104:0/2166402715 conn(0x7fa5bc0ce940 msgr2=0x7fa5bc0caf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 shutdown_connections 2023-06-10T18:45:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:15.951+0000 7fa5c3655700 1 -- 172.21.15.104:0/2166402715 wait complete. 2023-06-10T18:45:15.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:16.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.086+0000 7f1841555700 1 Processor -- start 2023-06-10T18:45:16.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.086+0000 7f1841555700 1 -- start start 2023-06-10T18:45:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f1841555700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f183c06af80 0x7f183c0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f1841555700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f183c0ccff0 0x7f183c06b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f1841555700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183c0ce340 con 0x7f183c15fd00 2023-06-10T18:45:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f1841555700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0cd490 con 0x7f183c0ccff0 2023-06-10T18:45:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f1841555700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0d00d0 con 0x7f183c06af80 2023-06-10T18:45:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f183a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f183c0ccff0 0x7f183c06b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f183a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f183c0ccff0 0x7f183c06b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56442/0 (socket says 172.21.15.104:56442) 2023-06-10T18:45:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.087+0000 7f183a7fc700 1 -- 172.21.15.104:0/3907294870 learned_addr learned my addr 172.21.15.104:0/3907294870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f183a7fc700 1 -- 172.21.15.104:0/3907294870 >> v2:172.21.15.114:3300/0 conn(0x7f183c06af80 msgr2=0x7f183c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f183a7fc700 1 --2- 172.21.15.104:0/3907294870 >> v2:172.21.15.114:3300/0 conn(0x7f183c06af80 0x7f183c0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f183a7fc700 1 -- 172.21.15.104:0/3907294870 >> v1:172.21.15.114:6789/0 conn(0x7f183c15fd00 legacy=0x7f183c160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f183a7fc700 1 -- 172.21.15.104:0/3907294870 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c06b890 con 0x7f183c0ccff0 2023-06-10T18:45:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f183a7fc700 1 --2- 172.21.15.104:0/3907294870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c0ccff0 0x7f183c06b350 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f183c071e10 tx=0x7f182c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36ca002d34b43a3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1839ffb700 1 -- 172.21.15.104:0/3907294870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c004230 con 0x7f183c0ccff0 2023-06-10T18:45:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1839ffb700 1 -- 172.21.15.104:0/3907294870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c004390 con 0x7f183c0ccff0 2023-06-10T18:45:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1839ffb700 1 -- 172.21.15.104:0/3907294870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0036a0 con 0x7f183c0ccff0 2023-06-10T18:45:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 -- 172.21.15.104:0/3907294870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c0ccff0 msgr2=0x7f183c06b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 --2- 172.21.15.104:0/3907294870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c0ccff0 0x7f183c06b350 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f183c071e10 tx=0x7f182c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 -- 172.21.15.104:0/3907294870 shutdown_connections 2023-06-10T18:45:16.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 --2- 172.21.15.104:0/3907294870 >> v2:172.21.15.114:3300/0 conn(0x7f183c06af80 0x7f183c0d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 --2- 172.21.15.104:0/3907294870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c0ccff0 0x7f183c06b350 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.088+0000 7f1841555700 1 -- 172.21.15.104:0/3907294870 >> 172.21.15.104:0/3907294870 conn(0x7f183c089fd0 msgr2=0x7f183c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:16.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.089+0000 7f1841555700 1 -- 172.21.15.104:0/3907294870 shutdown_connections 2023-06-10T18:45:16.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.089+0000 7f1841555700 1 -- 172.21.15.104:0/3907294870 wait complete. 2023-06-10T18:45:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.090+0000 7f1841555700 1 Processor -- start 2023-06-10T18:45:16.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.090+0000 7f1841555700 1 -- start start 2023-06-10T18:45:16.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 0x7f183c0b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f183c15fd00 0x7f183c1f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0c5980 con 0x7f183c06af80 2023-06-10T18:45:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0c5c60 con 0x7f183c0ccff0 2023-06-10T18:45:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0c5f40 con 0x7f183c15fd00 2023-06-10T18:45:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56446/0 (socket says 172.21.15.104:56446) 2023-06-10T18:45:16.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 -- 172.21.15.104:0/4059891929 learned_addr learned my addr 172.21.15.104:0/4059891929 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183a7fc700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 0x7f183c0b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183b7fe700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f183c15fd00 0x7f183c1f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 -- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f183c15fd00 msgr2=0x7f183c1f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f183c15fd00 0x7f183c1f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 -- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 msgr2=0x7f183c0b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 0x7f183c0b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 -- 172.21.15.104:0/4059891929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c009550 con 0x7f183c06af80 2023-06-10T18:45:16.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183a7fc700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 0x7f183c0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f183affd700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f183000b530 tx=0x7f183000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830011830 con 0x7f183c06af80 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18300119b0 con 0x7f183c06af80 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c0c6220 con 0x7f183c06af80 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830011d60 con 0x7f183c06af80 2023-06-10T18:45:16.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.091+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c0b2f90 con 0x7f183c06af80 2023-06-10T18:45:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.092+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c0c6740 con 0x7f183c06af80 2023-06-10T18:45:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.093+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f183001a400 con 0x7f183c06af80 2023-06-10T18:45:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.093+0000 7f18237fe700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 0x7f182806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.093+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1830089870 con 0x7f183c06af80 2023-06-10T18:45:16.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.093+0000 7f183a7fc700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 0x7f182806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.094+0000 7f183a7fc700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 0x7f182806d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f182c009130 tx=0x7f182c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.097+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f183c0c6740 con 0x7f183c06af80 2023-06-10T18:45:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.284+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f183c0c6740 con 0x7f183c06af80 2023-06-10T18:45:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.284+0000 7f18237fe700 1 -- 172.21.15.104:0/4059891929 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f183c0c6740 con 0x7f183c06af80 2023-06-10T18:45:16.290 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:16.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 msgr2=0x7f182806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 0x7f182806d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f182c009130 tx=0x7f182c008030 comp rx=0 tx=0).stop 2023-06-10T18:45:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 msgr2=0x7f183c0c52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f183000b530 tx=0x7f183000b8f0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 shutdown_connections 2023-06-10T18:45:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f183c15fd00 0x7f183c1f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f183c06af80 0x7f183c0c52f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f182806ace0 0x7f182806d190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 --2- 172.21.15.104:0/4059891929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f183c0ccff0 0x7f183c0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.286+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 >> 172.21.15.104:0/4059891929 conn(0x7f183c089fd0 msgr2=0x7f183c150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.287+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 shutdown_connections 2023-06-10T18:45:16.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.287+0000 7f1841555700 1 -- 172.21.15.104:0/4059891929 wait complete. 2023-06-10T18:45:16.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:16.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.420+0000 7fc22af47700 1 Processor -- start 2023-06-10T18:45:16.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.420+0000 7fc22af47700 1 -- start start 2023-06-10T18:45:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc22af47700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc22415c730 0x7fc22406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc22af47700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2240cc000 0x7fc2240cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc22af47700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22415cb00 con 0x7fc22406cd70 2023-06-10T18:45:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc22af47700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240aed80 con 0x7fc2240cc000 2023-06-10T18:45:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc22af47700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240c9c60 con 0x7fc22415c730 2023-06-10T18:45:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc223fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2240cc000 0x7fc2240cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc2294e4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc22406cd70 0x7fc22406d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54286/0 (socket says 172.21.15.104:54286) 2023-06-10T18:45:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc2294e4700 1 -- 172.21.15.104:0/3172414863 learned_addr learned my addr 172.21.15.104:0/3172414863 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc228ce3700 1 --2- 172.21.15.104:0/3172414863 >> v2:172.21.15.114:3300/0 conn(0x7fc22415c730 0x7fc22406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc223fff700 1 -- 172.21.15.104:0/3172414863 >> v2:172.21.15.114:3300/0 conn(0x7fc22415c730 msgr2=0x7fc22406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc223fff700 1 --2- 172.21.15.104:0/3172414863 >> v2:172.21.15.114:3300/0 conn(0x7fc22415c730 0x7fc22406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc223fff700 1 -- 172.21.15.104:0/3172414863 >> v1:172.21.15.104:6789/0 conn(0x7fc22406cd70 legacy=0x7fc22406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc223fff700 1 -- 172.21.15.104:0/3172414863 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22406ecd0 con 0x7fc2240cc000 2023-06-10T18:45:16.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.421+0000 7fc2237fe700 1 -- 172.21.15.104:0/3172414863 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107798850 0 0) 0x7fc21c001490 con 0x7fc22406cd70 2023-06-10T18:45:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc223fff700 1 --2- 172.21.15.104:0/3172414863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2240cc000 0x7fc2240cc470 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cf570 tx=0x7fc210009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65f2dd06bf7b807 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc2237fe700 1 -- 172.21.15.104:0/3172414863 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210011830 con 0x7fc2240cc000 2023-06-10T18:45:16.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc2237fe700 1 -- 172.21.15.104:0/3172414863 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210011990 con 0x7fc2240cc000 2023-06-10T18:45:16.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc2237fe700 1 -- 172.21.15.104:0/3172414863 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210011d20 con 0x7fc2240cc000 2023-06-10T18:45:16.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 -- 172.21.15.104:0/3172414863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2240cc000 msgr2=0x7fc2240cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 --2- 172.21.15.104:0/3172414863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2240cc000 0x7fc2240cc470 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cf570 tx=0x7fc210009b20 comp rx=0 tx=0).stop 2023-06-10T18:45:16.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 -- 172.21.15.104:0/3172414863 shutdown_connections 2023-06-10T18:45:16.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 --2- 172.21.15.104:0/3172414863 >> v2:172.21.15.114:3300/0 conn(0x7fc22415c730 0x7fc22406e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 --2- 172.21.15.104:0/3172414863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2240cc000 0x7fc2240cc470 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 -- 172.21.15.104:0/3172414863 >> 172.21.15.104:0/3172414863 conn(0x7fc22406b8a0 msgr2=0x7fc2240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 -- 172.21.15.104:0/3172414863 shutdown_connections 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.422+0000 7fc22af47700 1 -- 172.21.15.104:0/3172414863 wait complete. 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.423+0000 7fc22af47700 1 Processor -- start 2023-06-10T18:45:16.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.423+0000 7fc22af47700 1 -- start start 2023-06-10T18:45:16.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 0x7fc2240afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 0x7fc2240b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240c6750 con 0x7fc22415c730 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240b1270 con 0x7fc22406cd70 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc22af47700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc224152230 con 0x7fc2240cc000 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc2294e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc2294e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56464/0 (socket says 172.21.15.104:56464) 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.424+0000 7fc2294e4700 1 -- 172.21.15.104:0/916140646 learned_addr learned my addr 172.21.15.104:0/916140646 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc228ce3700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 0x7fc2240afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc223fff700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 0x7fc2240b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 -- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 msgr2=0x7fc2240b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 0x7fc2240b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 -- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 msgr2=0x7fc2240afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 0x7fc2240afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 -- 172.21.15.104:0/916140646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2100094a0 con 0x7fc22415c730 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc228ce3700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 0x7fc2240afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc223fff700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 0x7fc2240b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2294e4700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00cca0 tx=0x7fc21c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c00cea0 con 0x7fc22415c730 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c00ada0 con 0x7fc22415c730 2023-06-10T18:45:16.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c007310 con 0x7fc22415c730 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2240b14d0 con 0x7fc22415c730 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.425+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2240b1a90 con 0x7fc22415c730 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.426+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc224074050 con 0x7fc22415c730 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.426+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc21c007470 con 0x7fc22415c730 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.427+0000 7fc2217fa700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 0x7fc20806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.427+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc21c089180 con 0x7fc22415c730 2023-06-10T18:45:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.427+0000 7fc228ce3700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 0x7fc20806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.427+0000 7fc228ce3700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 0x7fc20806d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc218006fd0 tx=0x7fc218008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.431+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc21c053f90 con 0x7fc22415c730 2023-06-10T18:45:16.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.631+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc22415cb00 con 0x7fc22415c730 2023-06-10T18:45:16.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.631+0000 7fc2217fa700 1 -- 172.21.15.104:0/916140646 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc22415cb00 con 0x7fc22415c730 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 msgr2=0x7fc20806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 0x7fc20806d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc218006fd0 tx=0x7fc218008040 comp rx=0 tx=0).stop 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 msgr2=0x7fc224151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00cca0 tx=0x7fc21c00a340 comp rx=0 tx=0).stop 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 shutdown_connections 2023-06-10T18:45:16.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc20806ace0 0x7fc20806d190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22406cd70 0x7fc2240afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22415c730 0x7fc224151cf0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 --2- 172.21.15.104:0/916140646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2240cc000 0x7fc2240b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 >> 172.21.15.104:0/916140646 conn(0x7fc22406b8a0 msgr2=0x7fc2240ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 shutdown_connections 2023-06-10T18:45:16.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.633+0000 7fc22af47700 1 -- 172.21.15.104:0/916140646 wait complete. 2023-06-10T18:45:16.640 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:16.649 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:16.650 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:16.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:16.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.822+0000 7f8d58d24700 1 Processor -- start 2023-06-10T18:45:16.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.822+0000 7f8d58d24700 1 -- start start 2023-06-10T18:45:16.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d540ada30 0x7f8d540ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ade00 0x7f8d540ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d540ce0b0 0x7f8d540ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d540cf430 con 0x7f8d540ce0b0 2023-06-10T18:45:16.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d540cee70 con 0x7f8d540ada30 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.823+0000 7f8d58d24700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540cf150 con 0x7f8d540ade00 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d540ce0b0 0x7f8d540ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d540ce0b0 0x7f8d540ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56472/0 (socket says 172.21.15.104:56472) 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 -- 172.21.15.104:0/215241289 learned_addr learned my addr 172.21.15.104:0/215241289 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d5259c700 1 --2- 172.21.15.104:0/215241289 >> v2:172.21.15.114:3300/0 conn(0x7f8d540ada30 0x7f8d540ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d51d9b700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ade00 0x7f8d540ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 -- 172.21.15.104:0/215241289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ade00 msgr2=0x7f8d540ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ade00 0x7f8d540ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 -- 172.21.15.104:0/215241289 >> v2:172.21.15.114:3300/0 conn(0x7f8d540ada30 msgr2=0x7f8d540ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 --2- 172.21.15.104:0/215241289 >> v2:172.21.15.114:3300/0 conn(0x7f8d540ada30 0x7f8d540ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 -- 172.21.15.104:0/215241289 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5406cf10 con 0x7f8d540ce0b0 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d52d9d700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d540ce510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8d5406ff20 tx=0x7f8d4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207189ef7a67d0a9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d5159a700 1 -- 172.21.15.104:0/215241289 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00e070 con 0x7f8d540ce0b0 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d5159a700 1 -- 172.21.15.104:0/215241289 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c00b9c0 con 0x7f8d540ce0b0 2023-06-10T18:45:16.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.824+0000 7f8d5159a700 1 -- 172.21.15.104:0/215241289 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00bd70 con 0x7f8d540ce0b0 2023-06-10T18:45:16.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 -- 172.21.15.104:0/215241289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 msgr2=0x7f8d540ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d540ce510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8d5406ff20 tx=0x7f8d4c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 -- 172.21.15.104:0/215241289 shutdown_connections 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d540ce510 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 --2- 172.21.15.104:0/215241289 >> v2:172.21.15.114:3300/0 conn(0x7f8d540ada30 0x7f8d540ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 --2- 172.21.15.104:0/215241289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ade00 0x7f8d540ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 -- 172.21.15.104:0/215241289 >> 172.21.15.104:0/215241289 conn(0x7f8d5406b8a0 msgr2=0x7f8d540cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 -- 172.21.15.104:0/215241289 shutdown_connections 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.825+0000 7f8d58d24700 1 -- 172.21.15.104:0/215241289 wait complete. 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.826+0000 7f8d58d24700 1 Processor -- start 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.826+0000 7f8d58d24700 1 -- start start 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ada30 0x7f8d5409b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d541e6000 0x7f8d541e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5409be60 con 0x7f8d540ce0b0 2023-06-10T18:45:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5409b8a0 con 0x7f8d541e6000 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d58d24700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5409bb80 con 0x7f8d540ada30 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56476/0 (socket says 172.21.15.104:56476) 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 -- 172.21.15.104:0/4227248633 learned_addr learned my addr 172.21.15.104:0/4227248633 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d5259c700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ada30 0x7f8d5409b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d52d9d700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d541e6000 0x7f8d541e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 -- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ada30 msgr2=0x7f8d5409b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ada30 0x7f8d5409b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 -- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d541e6000 msgr2=0x7f8d541e53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d541e6000 0x7f8d541e53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 -- 172.21.15.104:0/4227248633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c009b30 con 0x7f8d540ce0b0 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.827+0000 7f8d51d9b700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400e7e0 tx=0x7f8d4400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.828+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44013610 con 0x7f8d540ce0b0 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.828+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d44013770 con 0x7f8d540ce0b0 2023-06-10T18:45:16.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.828+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d541e4bc0 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.828+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44013ae0 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.828+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5415c090 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.829+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d541e4a20 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.829+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d44013c40 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.829+0000 7f8d4affd700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 0x7f8d3406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.829+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8d440892d0 con 0x7f8d540ce0b0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.830+0000 7f8d5259c700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 0x7f8d3406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:16.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.830+0000 7f8d5259c700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 0x7f8d3406d1c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c006fd0 tx=0x7f8d3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:16.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:16.834+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d541e4a20 con 0x7f8d540ce0b0 2023-06-10T18:45:17.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.023+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d541e50e0 con 0x7f8d540ce0b0 2023-06-10T18:45:17.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.024+0000 7f8d4affd700 1 -- 172.21.15.104:0/4227248633 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f8d541e50e0 con 0x7f8d540ce0b0 2023-06-10T18:45:17.029 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:17.029 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:17.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 msgr2=0x7f8d3406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 0x7f8d3406d1c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c006fd0 tx=0x7f8d3c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:17.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 msgr2=0x7f8d5415bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400e7e0 tx=0x7f8d4400eba0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 shutdown_connections 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d3406ad10 0x7f8d3406d1c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d541e6000 0x7f8d541e53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d540ada30 0x7f8d5409b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 --2- 172.21.15.104:0/4227248633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d540ce0b0 0x7f8d5415bb50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 >> 172.21.15.104:0/4227248633 conn(0x7f8d5406b8a0 msgr2=0x7f8d540af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 shutdown_connections 2023-06-10T18:45:17.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.026+0000 7f8d58d24700 1 -- 172.21.15.104:0/4227248633 wait complete. 2023-06-10T18:45:17.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:17.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:17.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.162+0000 7f199ae0b700 1 Processor -- start 2023-06-10T18:45:17.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.162+0000 7f199ae0b700 1 -- start start 2023-06-10T18:45:17.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f199ae0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f19940b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f199ae0b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406e250 con 0x7f199415c580 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f199ae0b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199415c950 con 0x7f199406cde0 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f199ae0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19940b1e80 con 0x7f199406e6e0 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f1998ba7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f199415c580 0x7f19940b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54310/0 (socket says 172.21.15.104:54310) 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f1998ba7700 1 -- 172.21.15.104:0/292749043 learned_addr learned my addr 172.21.15.104:0/292749043 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f1993fff700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f19940b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335709329 0 0) 0x7f199406e250 con 0x7f199415c580 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19840035b0 con 0x7f199415c580 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312751143 0 0) 0x7f199415c950 con 0x7f199406cde0 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199406e250 con 0x7f199406cde0 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1876818250 0 0) 0x7f19840035b0 con 0x7f199415c580 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 msgr2=0x7f19940b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f19940b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 >> v1:172.21.15.114:6789/0 conn(0x7f199406cde0 legacy=0x7f19940ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199406d920 con 0x7f199415c580 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.163+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f19880017f0 con 0x7f199415c580 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 >> v1:172.21.15.104:6789/0 conn(0x7f199415c580 legacy=0x7f19940b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f1984003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1984004170 0x7f1984004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1984004d90 0x7f1984005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19940b1e80 con 0x7f1984004d90 2023-06-10T18:45:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198400bb20 con 0x7f1984004170 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198400bce0 con 0x7f199406e6e0 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f1998ba7700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f1984003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f1993fff700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1984004170 0x7f1984004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1984004d90 0x7f1984005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 -- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 msgr2=0x7f1984003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199406e6e0 0x7f1984003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 -- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1984004170 msgr2=0x7f1984004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1984004170 0x7f1984004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 -- 172.21.15.104:0/292749043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19840035b0 con 0x7f1984004d90 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19993a8700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1984004d90 0x7f1984005200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c2f0 tx=0x7f1984005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720252f92523242a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c001f30 con 0x7f1984004d90 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c00ce90 con 0x7f1984004d90 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.164+0000 7f19937fe700 1 -- 172.21.15.104:0/292749043 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c003c40 con 0x7f1984004d90 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 -- 172.21.15.104:0/292749043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1984004d90 msgr2=0x7f1984005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 --2- 172.21.15.104:0/292749043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1984004d90 0x7f1984005200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c2f0 tx=0x7f1984005880 comp rx=0 tx=0).stop 2023-06-10T18:45:17.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f19993a8700 1 -- 172.21.15.104:0/292749043 reap_dead start 2023-06-10T18:45:17.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 -- 172.21.15.104:0/292749043 shutdown_connections 2023-06-10T18:45:17.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 -- 172.21.15.104:0/292749043 >> 172.21.15.104:0/292749043 conn(0x7f199406b8a0 msgr2=0x7f19940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 -- 172.21.15.104:0/292749043 shutdown_connections 2023-06-10T18:45:17.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.165+0000 7f199ae0b700 1 -- 172.21.15.104:0/292749043 wait complete. 2023-06-10T18:45:17.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.166+0000 7f199ae0b700 1 Processor -- start 2023-06-10T18:45:17.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.166+0000 7f199ae0b700 1 -- start start 2023-06-10T18:45:17.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 0x7f19941f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199415d260 0x7f199415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199415cb30 con 0x7f1994155d80 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941f54f0 con 0x7f1994156150 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f199ae0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19941f5700 con 0x7f199415d260 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56494/0 (socket says 172.21.15.104:56494) 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 -- 172.21.15.104:0/960122744 learned_addr learned my addr 172.21.15.104:0/960122744 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1993fff700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 0x7f19941f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f19993a8700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199415d260 0x7f199415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 -- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199415d260 msgr2=0x7f199415d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199415d260 0x7f199415d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 -- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 msgr2=0x7f19941f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 0x7f19941f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 -- 172.21.15.104:0/960122744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19941f59e0 con 0x7f1994155d80 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1993fff700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 0x7f19941f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.167+0000 7f1998ba7700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f198800f890 tx=0x7f198800fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.168+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800fe90 con 0x7f1994155d80 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.168+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19880072e0 con 0x7f1994155d80 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.168+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988007690 con 0x7f1994155d80 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.168+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19941f5cc0 con 0x7f1994155d80 2023-06-10T18:45:17.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.168+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19940c35c0 con 0x7f1994155d80 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.169+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199415cf80 con 0x7f1994155d80 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.169+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1988005830 con 0x7f1994155d80 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.169+0000 7f19917fa700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 0x7f197806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.169+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1988089880 con 0x7f1994155d80 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.170+0000 7f1993fff700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 0x7f197806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.170+0000 7f1993fff700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 0x7f197806d190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19940cc6a0 tx=0x7f1980008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.174+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f199415cf80 con 0x7f1994155d80 2023-06-10T18:45:17.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.360+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f199415cf80 con 0x7f1994155d80 2023-06-10T18:45:17.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.361+0000 7f19917fa700 1 -- 172.21.15.104:0/960122744 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f199415cf80 con 0x7f1994155d80 2023-06-10T18:45:17.366 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:17.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 msgr2=0x7f197806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 0x7f197806d190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19940cc6a0 tx=0x7f1980008040 comp rx=0 tx=0).stop 2023-06-10T18:45:17.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 msgr2=0x7f19940ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f198800f890 tx=0x7f198800fc50 comp rx=0 tx=0).stop 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 shutdown_connections 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f199415d260 0x7f199415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1994155d80 0x7f19940ada20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f197806ace0 0x7f197806d190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 --2- 172.21.15.104:0/960122744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1994156150 0x7f19941f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 >> 172.21.15.104:0/960122744 conn(0x7f199406b8a0 msgr2=0x7f1994159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 shutdown_connections 2023-06-10T18:45:17.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.363+0000 7f199ae0b700 1 -- 172.21.15.104:0/960122744 wait complete. 2023-06-10T18:45:17.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:17.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.497+0000 7f43fc093700 1 Processor -- start 2023-06-10T18:45:17.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.498+0000 7f43fc093700 1 -- start start 2023-06-10T18:45:17.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.498+0000 7f43fc093700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43f415c580 0x7f43f40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.498+0000 7f43fc093700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f406e250 con 0x7f43f406cde0 2023-06-10T18:45:17.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.498+0000 7f43fc093700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f415c950 con 0x7f43f406e6e0 2023-06-10T18:45:17.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.498+0000 7f43fc093700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f40b1e80 con 0x7f43f415c580 2023-06-10T18:45:17.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43fa630700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f43f406cde0 0x7f43f40ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54334/0 (socket says 172.21.15.104:54334) 2023-06-10T18:45:17.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43fa630700 1 -- 172.21.15.104:0/3422973109 learned_addr learned my addr 172.21.15.104:0/3422973109 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:17.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/3422973109 >> v2:172.21.15.114:3300/0 conn(0x7f43f415c580 0x7f43f40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952922212 0 0) 0x7f43f406e250 con 0x7f43f406cde0 2023-06-10T18:45:17.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e00035b0 con 0x7f43f406cde0 2023-06-10T18:45:17.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655152367 0 0) 0x7f43e00035b0 con 0x7f43f406cde0 2023-06-10T18:45:17.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 >> v2:172.21.15.114:3300/0 conn(0x7f43f415c580 msgr2=0x7f43f40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 --2- 172.21.15.104:0/3422973109 >> v2:172.21.15.114:3300/0 conn(0x7f43f415c580 0x7f43f40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 >> v1:172.21.15.114:6789/0 conn(0x7f43f406e6e0 legacy=0x7f43f40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f406d920 con 0x7f43f406cde0 2023-06-10T18:45:17.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f43ec0019f0 con 0x7f43f406cde0 2023-06-10T18:45:17.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 >> v1:172.21.15.104:6789/0 conn(0x7f43f406cde0 legacy=0x7f43f40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 0x7f43e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0004850 0x7f43e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0005200 0x7f43e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f40b1e80 con 0x7f43e0003a00 2023-06-10T18:45:17.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e000c080 con 0x7f43e0005200 2023-06-10T18:45:17.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e000c240 con 0x7f43e0004850 2023-06-10T18:45:17.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 0x7f43e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43fa630700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0005200 0x7f43e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.499+0000 7f43f962e700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0004850 0x7f43e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 -- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0004850 msgr2=0x7f43e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0004850 0x7f43e0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 -- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0005200 msgr2=0x7f43e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0005200 0x7f43e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fa630700 1 -- 172.21.15.104:0/3422973109 reap_dead start 2023-06-10T18:45:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 -- 172.21.15.104:0/3422973109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00035b0 con 0x7f43e0003a00 2023-06-10T18:45:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 0x7f43e0003e10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009b40 tx=0x7f43e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b023855f178a837 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4005a20 con 0x7f43e0003a00 2023-06-10T18:45:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4005b80 con 0x7f43e0003a00 2023-06-10T18:45:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43f8e2d700 1 -- 172.21.15.104:0/3422973109 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e400e6c0 con 0x7f43e0003a00 2023-06-10T18:45:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fc093700 1 -- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 msgr2=0x7f43e0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fc093700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 0x7f43e0003e10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009b40 tx=0x7f43e0005880 comp rx=0 tx=0).stop 2023-06-10T18:45:17.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fc093700 1 -- 172.21.15.104:0/3422973109 shutdown_connections 2023-06-10T18:45:17.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fc093700 1 --2- 172.21.15.104:0/3422973109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e0003a00 0x7f43e0003e10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.500+0000 7f43fc093700 1 -- 172.21.15.104:0/3422973109 >> 172.21.15.104:0/3422973109 conn(0x7f43f406b8a0 msgr2=0x7f43f40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.501+0000 7f43fc093700 1 -- 172.21.15.104:0/3422973109 shutdown_connections 2023-06-10T18:45:17.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.501+0000 7f43fc093700 1 -- 172.21.15.104:0/3422973109 wait complete. 2023-06-10T18:45:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.502+0000 7f43fc093700 1 Processor -- start 2023-06-10T18:45:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.502+0000 7f43fc093700 1 -- start start 2023-06-10T18:45:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0003a00 0x7f43f40ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f40cefa0 0x7f43f40ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f40cf410 con 0x7f43f414eaf0 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f41e6770 con 0x7f43f40cefa0 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fc093700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f41f54f0 con 0x7f43e0003a00 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56516/0 (socket says 172.21.15.104:56516) 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 -- 172.21.15.104:0/1904828831 learned_addr learned my addr 172.21.15.104:0/1904828831 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0003a00 0x7f43f40ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43fa630700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f40cefa0 0x7f43f40ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 -- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0003a00 msgr2=0x7f43f40ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0003a00 0x7f43f40ad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 -- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f40cefa0 msgr2=0x7f43f40ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f40cefa0 0x7f43f40ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.503+0000 7f43f962e700 1 -- 172.21.15.104:0/1904828831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f41f57d0 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43f962e700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f43f000e490 tx=0x7f43f000e7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000ee10 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00052e0 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f41f5ab0 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0003680 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.504+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f414ef60 con 0x7f43f414eaf0 2023-06-10T18:45:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.505+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f41f6100 con 0x7f43f414eaf0 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.505+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f43f0003800 con 0x7f43f414eaf0 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.506+0000 7f43ea7fc700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 0x7f43d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.506+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f43f0088c70 con 0x7f43f414eaf0 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.506+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 0x7f43d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.506+0000 7f43f9e2f700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 0x7f43d806d190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43e4000c00 tx=0x7f43e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.510+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43f0053a90 con 0x7f43f414eaf0 2023-06-10T18:45:17.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.695+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f43f40d0500 con 0x7f43f414eaf0 2023-06-10T18:45:17.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.696+0000 7f43ea7fc700 1 -- 172.21.15.104:0/1904828831 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f43f0019070 con 0x7f43f414eaf0 2023-06-10T18:45:17.701 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:17.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 msgr2=0x7f43d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 0x7f43d806d190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43e4000c00 tx=0x7f43e4008030 comp rx=0 tx=0).stop 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 msgr2=0x7f43f40cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f43f000e490 tx=0x7f43f000e7a0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 shutdown_connections 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43d806ace0 0x7f43d806d190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43f40cefa0 0x7f43f40ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e0003a00 0x7f43f40ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 --2- 172.21.15.104:0/1904828831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43f414eaf0 0x7f43f40cea60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 >> 172.21.15.104:0/1904828831 conn(0x7f43f406b8a0 msgr2=0x7f43f40cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 shutdown_connections 2023-06-10T18:45:17.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.698+0000 7f43fc093700 1 -- 172.21.15.104:0/1904828831 wait complete. 2023-06-10T18:45:17.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.832+0000 7f876ab82700 1 Processor -- start 2023-06-10T18:45:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.832+0000 7f876ab82700 1 -- start start 2023-06-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876ab82700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f87640b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876ab82700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876406e250 con 0x7f876415c580 2023-06-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876ab82700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876415c950 con 0x7f876406e6e0 2023-06-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876ab82700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b1e80 con 0x7f876406cde0 2023-06-10T18:45:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876891e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f876415c580 0x7f87640b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54350/0 (socket says 172.21.15.104:54350) 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.833+0000 7f876891e700 1 -- 172.21.15.104:0/3309191968 learned_addr learned my addr 172.21.15.104:0/3309191968 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660539588 0 0) 0x7f876406e250 con 0x7f876415c580 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87540035b0 con 0x7f876415c580 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f87640b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161536386 0 0) 0x7f87540035b0 con 0x7f876415c580 2023-06-10T18:45:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 msgr2=0x7f87640b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f87640b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 >> v1:172.21.15.114:6789/0 conn(0x7f876406e6e0 legacy=0x7f87640ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876406d920 con 0x7f876415c580 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f87580017f0 con 0x7f876415c580 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 >> v1:172.21.15.104:6789/0 conn(0x7f876415c580 legacy=0x7f87640b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f8754003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8754004060 0x7f8754004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8754004d90 0x7f8754005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b1e80 con 0x7f8754004d90 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875400bba0 con 0x7f8754004060 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875400bd60 con 0x7f876406cde0 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8754004d90 0x7f8754005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876891e700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f8754003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f8763fff700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8754004060 0x7f8754004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 -- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 msgr2=0x7f8754003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f876406cde0 0x7f8754003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 -- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8754004060 msgr2=0x7f8754004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8754004060 0x7f8754004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 -- 172.21.15.104:0/3309191968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87540035b0 con 0x7f8754004d90 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.834+0000 7f876911f700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8754004d90 0x7f8754005200 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f875c007360 tx=0x7f8754005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1458c8c75459cf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c007e80 con 0x7f8754004d90 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c014400 con 0x7f8754004d90 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f87637fe700 1 -- 172.21.15.104:0/3309191968 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c014790 con 0x7f8754004d90 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 -- 172.21.15.104:0/3309191968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8754004d90 msgr2=0x7f8754005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 --2- 172.21.15.104:0/3309191968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8754004d90 0x7f8754005200 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f875c007360 tx=0x7f8754005880 comp rx=0 tx=0).stop 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876911f700 1 -- 172.21.15.104:0/3309191968 reap_dead start 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 -- 172.21.15.104:0/3309191968 shutdown_connections 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 -- 172.21.15.104:0/3309191968 >> 172.21.15.104:0/3309191968 conn(0x7f876406b8a0 msgr2=0x7f87640cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:17.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 -- 172.21.15.104:0/3309191968 shutdown_connections 2023-06-10T18:45:17.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.835+0000 7f876ab82700 1 -- 172.21.15.104:0/3309191968 wait complete. 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.836+0000 7f876ab82700 1 Processor -- start 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.836+0000 7f876ab82700 1 -- start start 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8764155b60 0x7f87640c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8764155f30 0x7f87640c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640c7fa0 con 0x7f876415cf50 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640ac260 con 0x7f8764155f30 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876ab82700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640c81d0 con 0x7f8764155b60 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876911f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876911f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56538/0 (socket says 172.21.15.104:56538) 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876911f700 1 -- 172.21.15.104:0/4121107276 learned_addr learned my addr 172.21.15.104:0/4121107276 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f8763fff700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8764155f30 0x7f87640c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.837+0000 7f876891e700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8764155b60 0x7f87640c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 -- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8764155b60 msgr2=0x7f87640c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8764155b60 0x7f87640c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 -- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8764155f30 msgr2=0x7f87640c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8764155f30 0x7f87640c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 -- 172.21.15.104:0/4121107276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640ac500 con 0x7f876415cf50 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876911f700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f875c017040 tx=0x7f875c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c007000 con 0x7f876415cf50 2023-06-10T18:45:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c01e440 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c009410 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87640ac7e0 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.838+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876415d3c0 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.839+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876415d5b0 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.839+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f875c01e7f0 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.840+0000 7f87617fa700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 0x7f874806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.840+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f875c08abc0 con 0x7f876415cf50 2023-06-10T18:45:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.840+0000 7f876891e700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 0x7f874806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:17.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.840+0000 7f876891e700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 0x7f874806d190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8758000fd0 tx=0x7f8758009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:17.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:17.844+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f875c055af0 con 0x7f876415cf50 2023-06-10T18:45:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.028+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8764074050 con 0x7f876415cf50 2023-06-10T18:45:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.028+0000 7f87617fa700 1 -- 172.21.15.104:0/4121107276 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f875c059110 con 0x7f876415cf50 2023-06-10T18:45:18.033 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:18.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 msgr2=0x7f874806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 0x7f874806d190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8758000fd0 tx=0x7f8758009040 comp rx=0 tx=0).stop 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 msgr2=0x7f8764157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f875c017040 tx=0x7f875c004b40 comp rx=0 tx=0).stop 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 shutdown_connections 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415cf50 0x7f8764157580 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8764155b60 0x7f87640c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f874806ace0 0x7f874806d190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 --2- 172.21.15.104:0/4121107276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8764155f30 0x7f87640c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 >> 172.21.15.104:0/4121107276 conn(0x7f876406b8a0 msgr2=0x7f8764159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 shutdown_connections 2023-06-10T18:45:18.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.030+0000 7f876ab82700 1 -- 172.21.15.104:0/4121107276 wait complete. 2023-06-10T18:45:18.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.165+0000 7fc4902e8700 1 Processor -- start 2023-06-10T18:45:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.165+0000 7fc4902e8700 1 -- start start 2023-06-10T18:45:18.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc4902e8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc48815c6f0 0x7fc4880b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc4902e8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 0x7fc4880b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc4902e8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880ae7e0 con 0x7fc48806ea70 2023-06-10T18:45:18.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc4902e8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48806df90 con 0x7fc48815c6f0 2023-06-10T18:45:18.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc4902e8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48806e270 con 0x7fc4880aec00 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc48815c6f0 0x7fc4880b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d883700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc48806ea70 0x7fc4880ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54372/0 (socket says 172.21.15.104:54372) 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d883700 1 -- 172.21.15.104:0/1490997610 learned_addr learned my addr 172.21.15.104:0/1490997610 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e885700 1 --2- 172.21.15.104:0/1490997610 >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 0x7fc4880b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 -- 172.21.15.104:0/1490997610 >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 msgr2=0x7fc4880b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 --2- 172.21.15.104:0/1490997610 >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 0x7fc4880b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 -- 172.21.15.104:0/1490997610 >> v1:172.21.15.104:6789/0 conn(0x7fc48806ea70 legacy=0x7fc4880ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d082700 1 -- 172.21.15.104:0/1490997610 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897413365 0 0) 0x7fc4880ae7e0 con 0x7fc48806ea70 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 -- 172.21.15.104:0/1490997610 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48806e550 con 0x7fc48815c6f0 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e885700 1 --2- 172.21.15.104:0/1490997610 >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 0x7fc4880b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48e084700 1 --2- 172.21.15.104:0/1490997610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880b1070 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc48806ff70 tx=0x7fc484009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3153978408b4980 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d082700 1 -- 172.21.15.104:0/1490997610 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484005e30 con 0x7fc48815c6f0 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d082700 1 -- 172.21.15.104:0/1490997610 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484006030 con 0x7fc48815c6f0 2023-06-10T18:45:18.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.166+0000 7fc48d082700 1 -- 172.21.15.104:0/1490997610 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4840063c0 con 0x7fc48815c6f0 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 -- 172.21.15.104:0/1490997610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 msgr2=0x7fc4880b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 --2- 172.21.15.104:0/1490997610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880b1070 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc48806ff70 tx=0x7fc484009b00 comp rx=0 tx=0).stop 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 -- 172.21.15.104:0/1490997610 shutdown_connections 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 --2- 172.21.15.104:0/1490997610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880b1070 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 --2- 172.21.15.104:0/1490997610 >> v2:172.21.15.114:3300/0 conn(0x7fc4880aec00 0x7fc4880b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 -- 172.21.15.104:0/1490997610 >> 172.21.15.104:0/1490997610 conn(0x7fc48806b8a0 msgr2=0x7fc4880cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 -- 172.21.15.104:0/1490997610 shutdown_connections 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.167+0000 7fc4902e8700 1 -- 172.21.15.104:0/1490997610 wait complete. 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.168+0000 7fc4902e8700 1 Processor -- start 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.168+0000 7fc4902e8700 1 -- start start 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48806ea70 0x7fc48814fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4880aec00 0x7fc4880c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880c3780 con 0x7fc48815c6f0 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880c9210 con 0x7fc48806ea70 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc4902e8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880c94f0 con 0x7fc4880aec00 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56552/0 (socket says 172.21.15.104:56552) 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 -- 172.21.15.104:0/4214313314 learned_addr learned my addr 172.21.15.104:0/4214313314 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e084700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48806ea70 0x7fc48814fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 -- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4880aec00 msgr2=0x7fc4880c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4880aec00 0x7fc4880c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 -- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48806ea70 msgr2=0x7fc48814fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48806ea70 0x7fc48814fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.169+0000 7fc48e885700 1 -- 172.21.15.104:0/4214313314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484009430 con 0x7fc48815c6f0 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc48e885700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc48000cef0 tx=0x7fc480007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000ae20 con 0x7fc48815c6f0 2023-06-10T18:45:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc480015440 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4880c3130 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4800157f0 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.170+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4881500f0 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.171+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4880c38f0 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.171+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc48000d430 con 0x7fc48815c6f0 2023-06-10T18:45:18.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.172+0000 7fc47effd700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 0x7fc46c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.172+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc480089dd0 con 0x7fc48815c6f0 2023-06-10T18:45:18.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.172+0000 7fc48e084700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 0x7fc46c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.172+0000 7fc48e084700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 0x7fc46c06d190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc484013040 tx=0x7fc484008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.176+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc480054b50 con 0x7fc48815c6f0 2023-06-10T18:45:18.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.363+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4880cdb20 con 0x7fc48815c6f0 2023-06-10T18:45:18.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.364+0000 7fc47effd700 1 -- 172.21.15.104:0/4214313314 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc48008b020 con 0x7fc48815c6f0 2023-06-10T18:45:18.369 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:18.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 msgr2=0x7fc46c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 0x7fc46c06d190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc484013040 tx=0x7fc484008070 comp rx=0 tx=0).stop 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 msgr2=0x7fc4880c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc48000cef0 tx=0x7fc480007f60 comp rx=0 tx=0).stop 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 shutdown_connections 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc46c06ace0 0x7fc46c06d190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48806ea70 0x7fc48814fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48815c6f0 0x7fc4880c8790 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 --2- 172.21.15.104:0/4214313314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4880aec00 0x7fc4880c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 >> 172.21.15.104:0/4214313314 conn(0x7fc48806b8a0 msgr2=0x7fc4880af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 shutdown_connections 2023-06-10T18:45:18.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.366+0000 7fc4902e8700 1 -- 172.21.15.104:0/4214313314 wait complete. 2023-06-10T18:45:18.383 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:18.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.500+0000 7f623af61700 1 Processor -- start 2023-06-10T18:45:18.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 -- start start 2023-06-10T18:45:18.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f623415c580 0x7f62340b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 0x7f62340b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623406e250 con 0x7f623406cde0 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623415c950 con 0x7f623415c580 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f623af61700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62340cf0e0 con 0x7f623406e6e0 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f6238cfd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f623415c580 0x7f62340b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f6238cfd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f623415c580 0x7f62340b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56556/0 (socket says 172.21.15.104:56556) 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.501+0000 7f6238cfd700 1 -- 172.21.15.104:0/507003136 learned_addr learned my addr 172.21.15.104:0/507003136 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6233fff700 1 --2- 172.21.15.104:0/507003136 >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 0x7f62340b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6238cfd700 1 -- 172.21.15.104:0/507003136 >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 msgr2=0x7f62340b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6238cfd700 1 --2- 172.21.15.104:0/507003136 >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 0x7f62340b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6238cfd700 1 -- 172.21.15.104:0/507003136 >> v1:172.21.15.114:6789/0 conn(0x7f623406cde0 legacy=0x7f6234069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6238cfd700 1 -- 172.21.15.104:0/507003136 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62340b1e80 con 0x7f623415c580 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6233fff700 1 --2- 172.21.15.104:0/507003136 >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 0x7f62340b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f6238cfd700 1 --2- 172.21.15.104:0/507003136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623415c580 0x7f62340b1070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f623406af30 tx=0x7f6228009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aacd0f824e5fb89 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f62337fe700 1 -- 172.21.15.104:0/507003136 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228005e30 con 0x7f623415c580 2023-06-10T18:45:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f62337fe700 1 -- 172.21.15.104:0/507003136 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6228006030 con 0x7f623415c580 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f62337fe700 1 -- 172.21.15.104:0/507003136 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280063e0 con 0x7f623415c580 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f623af61700 1 -- 172.21.15.104:0/507003136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623415c580 msgr2=0x7f62340b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f623af61700 1 --2- 172.21.15.104:0/507003136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623415c580 0x7f62340b1070 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f623406af30 tx=0x7f6228009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.502+0000 7f623af61700 1 -- 172.21.15.104:0/507003136 shutdown_connections 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.503+0000 7f623af61700 1 --2- 172.21.15.104:0/507003136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623415c580 0x7f62340b1070 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.503+0000 7f623af61700 1 --2- 172.21.15.104:0/507003136 >> v2:172.21.15.114:3300/0 conn(0x7f623406e6e0 0x7f62340b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.503+0000 7f623af61700 1 -- 172.21.15.104:0/507003136 >> 172.21.15.104:0/507003136 conn(0x7f623406b8a0 msgr2=0x7f62340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.503+0000 7f623af61700 1 -- 172.21.15.104:0/507003136 shutdown_connections 2023-06-10T18:45:18.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.503+0000 7f623af61700 1 -- 172.21.15.104:0/507003136 wait complete. 2023-06-10T18:45:18.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.504+0000 7f623af61700 1 Processor -- start 2023-06-10T18:45:18.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.504+0000 7f623af61700 1 -- start start 2023-06-10T18:45:18.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f623406cde0 0x7f62341577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 0x7f62340c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62340c8210 con 0x7f623406e6e0 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234158db0 con 0x7f623415c580 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f623af61700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6234159090 con 0x7f623406cde0 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f6233fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f6233fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56564/0 (socket says 172.21.15.104:56564) 2023-06-10T18:45:18.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.505+0000 7f6233fff700 1 -- 172.21.15.104:0/757412634 learned_addr learned my addr 172.21.15.104:0/757412634 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f62394fe700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 0x7f62340c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6238cfd700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f623406cde0 0x7f62341577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 -- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f623406cde0 msgr2=0x7f62341577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f623406cde0 0x7f62341577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 -- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 msgr2=0x7f62340c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 0x7f62340c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 -- 172.21.15.104:0/757412634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6228009500 con 0x7f623406e6e0 2023-06-10T18:45:18.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f62394fe700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 0x7f62340c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f6233fff700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f622000c7f0 tx=0x7f622000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220011800 con 0x7f623406e6e0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6220011960 con 0x7f623406e6e0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62340c7ea0 con 0x7f623406e6e0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220011d10 con 0x7f623406e6e0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.506+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62340ae8b0 con 0x7f623406e6e0 2023-06-10T18:45:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.507+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62340c7ce0 con 0x7f623406e6e0 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.507+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6220018360 con 0x7f623406e6e0 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.508+0000 7f62317fa700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 0x7f621806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.508+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6220089690 con 0x7f623406e6e0 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.508+0000 7f6238cfd700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 0x7f621806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.508+0000 7f6238cfd700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 0x7f621806d190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f623406af30 tx=0x7f6228008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.512+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6220057f30 con 0x7f623406e6e0 2023-06-10T18:45:18.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.698+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f62340c62f0 con 0x7f623406e6e0 2023-06-10T18:45:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.698+0000 7f62317fa700 1 -- 172.21.15.104:0/757412634 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6220057ac0 con 0x7f623406e6e0 2023-06-10T18:45:18.703 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 msgr2=0x7f621806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 0x7f621806d190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f623406af30 tx=0x7f6228008070 comp rx=0 tx=0).stop 2023-06-10T18:45:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 msgr2=0x7f62340c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f622000c7f0 tx=0x7f622000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 shutdown_connections 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f621806ace0 0x7f621806d190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f623415c580 0x7f62340c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f623406cde0 0x7f62341577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 --2- 172.21.15.104:0/757412634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f623406e6e0 0x7f62340c5060 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 >> 172.21.15.104:0/757412634 conn(0x7f623406b8a0 msgr2=0x7f62340c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 shutdown_connections 2023-06-10T18:45:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.700+0000 7f623af61700 1 -- 172.21.15.104:0/757412634 wait complete. 2023-06-10T18:45:18.717 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.833+0000 7fc888f46700 1 Processor -- start 2023-06-10T18:45:18.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.833+0000 7fc888f46700 1 -- start start 2023-06-10T18:45:18.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc888f46700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc88415a420 0x7fc8840b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc888f46700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc8840cea60 0x7fc88415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc888f46700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840b6070 con 0x7fc8840b4ff0 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc888f46700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840b5460 con 0x7fc88415a420 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc888f46700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840b5740 con 0x7fc8840cea60 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc88415a420 0x7fc8840b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc88415a420 0x7fc8840b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56570/0 (socket says 172.21.15.104:56570) 2023-06-10T18:45:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 -- 172.21.15.104:0/1763731287 learned_addr learned my addr 172.21.15.104:0/1763731287 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 -- 172.21.15.104:0/1763731287 >> v2:172.21.15.114:3300/0 conn(0x7fc8840cea60 msgr2=0x7fc88415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc882d9d700 1 --2- 172.21.15.104:0/1763731287 >> v2:172.21.15.114:3300/0 conn(0x7fc8840cea60 0x7fc88415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 --2- 172.21.15.104:0/1763731287 >> v2:172.21.15.114:3300/0 conn(0x7fc8840cea60 0x7fc88415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 -- 172.21.15.104:0/1763731287 >> v1:172.21.15.114:6789/0 conn(0x7fc8840b4ff0 legacy=0x7fc8840ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 -- 172.21.15.104:0/1763731287 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840b5a20 con 0x7fc88415a420 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88259c700 1 --2- 172.21.15.104:0/1763731287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8840b1660 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc8840cad60 tx=0x7fc874009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4c104cee56eda5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88159a700 1 -- 172.21.15.104:0/1763731287 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874004230 con 0x7fc88415a420 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88159a700 1 -- 172.21.15.104:0/1763731287 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc874004390 con 0x7fc88415a420 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.834+0000 7fc88159a700 1 -- 172.21.15.104:0/1763731287 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8740036a0 con 0x7fc88415a420 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 -- 172.21.15.104:0/1763731287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 msgr2=0x7fc8840b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 --2- 172.21.15.104:0/1763731287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8840b1660 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc8840cad60 tx=0x7fc874009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 -- 172.21.15.104:0/1763731287 shutdown_connections 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 --2- 172.21.15.104:0/1763731287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8840b1660 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 --2- 172.21.15.104:0/1763731287 >> v2:172.21.15.114:3300/0 conn(0x7fc8840cea60 0x7fc88415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 -- 172.21.15.104:0/1763731287 >> 172.21.15.104:0/1763731287 conn(0x7fc884089fd0 msgr2=0x7fc8840d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 -- 172.21.15.104:0/1763731287 shutdown_connections 2023-06-10T18:45:18.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.835+0000 7fc888f46700 1 -- 172.21.15.104:0/1763731287 wait complete. 2023-06-10T18:45:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.836+0000 7fc888f46700 1 Processor -- start 2023-06-10T18:45:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.836+0000 7fc888f46700 1 -- start start 2023-06-10T18:45:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8840b4ff0 0x7fc8841539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 0x7fc88406bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88406c200 con 0x7fc88415a420 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840c43c0 con 0x7fc8840b4ff0 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc888f46700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840cbe00 con 0x7fc8840cea60 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56580/0 (socket says 172.21.15.104:56580) 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 -- 172.21.15.104:0/3486418871 learned_addr learned my addr 172.21.15.104:0/3486418871 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc88259c700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8840b4ff0 0x7fc8841539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc881d9b700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 0x7fc88406bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 -- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 msgr2=0x7fc88406bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 0x7fc88406bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 -- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8840b4ff0 msgr2=0x7fc8841539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:18.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8840b4ff0 0x7fc8841539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc882d9d700 1 -- 172.21.15.104:0/3486418871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874009550 con 0x7fc88415a420 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.837+0000 7fc881d9b700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 0x7fc88406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc882d9d700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00cda0 tx=0x7fc87c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c003a20 con 0x7fc88415a420 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c003b80 con 0x7fc88415a420 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c004020 con 0x7fc88415a420 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc884154090 con 0x7fc88415a420 2023-06-10T18:45:18.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.838+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8840cc330 con 0x7fc88415a420 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.839+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc884074050 con 0x7fc88415a420 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.839+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc87c004180 con 0x7fc88415a420 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.840+0000 7fc872ffd700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 0x7fc86406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.840+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc87c012070 con 0x7fc88415a420 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.840+0000 7fc88259c700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 0x7fc86406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:18.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.840+0000 7fc88259c700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 0x7fc86406d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc874009130 tx=0x7fc874008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:18.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:18.844+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc87c054df0 con 0x7fc88415a420 2023-06-10T18:45:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.045+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc88406f190 con 0x7fc88415a420 2023-06-10T18:45:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.045+0000 7fc872ffd700 1 -- 172.21.15.104:0/3486418871 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc87c058410 con 0x7fc88415a420 2023-06-10T18:45:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 msgr2=0x7fc86406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 0x7fc86406d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc874009130 tx=0x7fc874008030 comp rx=0 tx=0).stop 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 msgr2=0x7fc8841e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00cda0 tx=0x7fc87c00a570 comp rx=0 tx=0).stop 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 shutdown_connections 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc86406ace0 0x7fc86406d190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8840b4ff0 0x7fc8841539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88415a420 0x7fc8841e3910 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 --2- 172.21.15.104:0/3486418871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8840cea60 0x7fc88406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 >> 172.21.15.104:0/3486418871 conn(0x7fc884089fd0 msgr2=0x7fc88406f850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 shutdown_connections 2023-06-10T18:45:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.047+0000 7fc888f46700 1 -- 172.21.15.104:0/3486418871 wait complete. 2023-06-10T18:45:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:19.063 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:45:19.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:19.064 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-06-10T18:45:19.108 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:19.109 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:45:19.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:19.110 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:45:19.127 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:19.127 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:19.149 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:19.150 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:19.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.326+0000 7f3673ac9700 1 Processor -- start 2023-06-10T18:45:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.326+0000 7f3673ac9700 1 -- start start 2023-06-10T18:45:19.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 0x7f366c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f366c06e4b0 0x7f366c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c071640 con 0x7f366c06e4b0 2023-06-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c0aef00 con 0x7f366c06e0e0 2023-06-10T18:45:19.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3673ac9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c06d290 con 0x7f366c0ca100 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f366c06e4b0 0x7f366c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f366c06e4b0 0x7f366c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56586/0 (socket says 172.21.15.104:56586) 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 -- 172.21.15.104:0/2081244957 learned_addr learned my addr 172.21.15.104:0/2081244957 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671865700 1 --2- 172.21.15.104:0/2081244957 >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 0x7f366c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3672066700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 -- 172.21.15.104:0/2081244957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 msgr2=0x7f366c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:19.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 -- 172.21.15.104:0/2081244957 >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 msgr2=0x7f366c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 --2- 172.21.15.104:0/2081244957 >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 0x7f366c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 -- 172.21.15.104:0/2081244957 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c06d4c0 con 0x7f366c06e4b0 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3672066700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671865700 1 --2- 172.21.15.104:0/2081244957 >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 0x7f366c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:19.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3671064700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e4b0 0x7f366c0c9bc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f366c06af50 tx=0x7f365c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0cbdb776c57398 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.327+0000 7f3670863700 1 -- 172.21.15.104:0/2081244957 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c00b870 con 0x7f366c06e4b0 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3670863700 1 -- 172.21.15.104:0/2081244957 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c00b9d0 con 0x7f366c06e4b0 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3670863700 1 -- 172.21.15.104:0/2081244957 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c00bd60 con 0x7f366c06e4b0 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 -- 172.21.15.104:0/2081244957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e4b0 msgr2=0x7f366c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e4b0 0x7f366c0c9bc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f366c06af50 tx=0x7f365c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 -- 172.21.15.104:0/2081244957 shutdown_connections 2023-06-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 --2- 172.21.15.104:0/2081244957 >> v2:172.21.15.114:3300/0 conn(0x7f366c06e0e0 0x7f366c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 --2- 172.21.15.104:0/2081244957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e4b0 0x7f366c0c9bc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 -- 172.21.15.104:0/2081244957 >> 172.21.15.104:0/2081244957 conn(0x7f366c06b8a0 msgr2=0x7f366c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:19.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 -- 172.21.15.104:0/2081244957 shutdown_connections 2023-06-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.328+0000 7f3673ac9700 1 -- 172.21.15.104:0/2081244957 wait complete. 2023-06-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.329+0000 7f3673ac9700 1 Processor -- start 2023-06-10T18:45:19.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.329+0000 7f3673ac9700 1 -- start start 2023-06-10T18:45:19.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 0x7f366c0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0af3e0 con 0x7f366c06e0e0 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0af6c0 con 0x7f366c06e4b0 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3673ac9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0af9a0 con 0x7f366c0ca100 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56590/0 (socket says 172.21.15.104:56590) 2023-06-10T18:45:19.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 -- 172.21.15.104:0/1695572414 learned_addr learned my addr 172.21.15.104:0/1695572414 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671064700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 0x7f366c0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3672066700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 -- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 msgr2=0x7f366c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 -- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 msgr2=0x7f366c0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 0x7f366c0b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671865700 1 -- 172.21.15.104:0/1695572414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009550 con 0x7f366c06e0e0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.330+0000 7f3671064700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 0x7f366c0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f3671865700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f366800e810 tx=0x7f366800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668005c20 con 0x7f366c06e0e0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668005d80 con 0x7f366c06e0e0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c0afc80 con 0x7f366c06e0e0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668004250 con 0x7f366c06e0e0 2023-06-10T18:45:19.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.331+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c1f1b60 con 0x7f366c06e0e0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.332+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c074050 con 0x7f366c06e0e0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.332+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f36680043b0 con 0x7f366c06e0e0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.332+0000 7f36627fc700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 0x7f365006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.333+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f36680894e0 con 0x7f366c06e0e0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.333+0000 7f3671064700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 0x7f365006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.333+0000 7f3671064700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 0x7f365006d190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f365c000c00 tx=0x7f365c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.337+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f36680542f0 con 0x7f366c06e0e0 2023-06-10T18:45:19.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.523+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f366c0707a0 con 0x7f366c06e0e0 2023-06-10T18:45:19.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.525+0000 7f36627fc700 1 -- 172.21.15.104:0/1695572414 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f3668057910 con 0x7f366c06e0e0 2023-06-10T18:45:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:19.530 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:19.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 msgr2=0x7f365006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 0x7f365006d190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f365c000c00 tx=0x7f365c008030 comp rx=0 tx=0).stop 2023-06-10T18:45:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 msgr2=0x7f366c1f1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f366800e810 tx=0x7f366800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 shutdown_connections 2023-06-10T18:45:19.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f366c0ca100 0x7f366c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f366c06e0e0 0x7f366c1f1620 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f365006ace0 0x7f365006d190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 --2- 172.21.15.104:0/1695572414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f366c06e4b0 0x7f366c0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 >> 172.21.15.104:0/1695572414 conn(0x7f366c06b8a0 msgr2=0x7f366c0cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 shutdown_connections 2023-06-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.527+0000 7f3673ac9700 1 -- 172.21.15.104:0/1695572414 wait complete. 2023-06-10T18:45:19.543 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:19.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:19.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.662+0000 7fc910939700 1 Processor -- start 2023-06-10T18:45:19.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.662+0000 7fc910939700 1 -- start start 2023-06-10T18:45:19.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc910939700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc90c0b1780 0x7fc90c06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc910939700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c0ae7e0 con 0x7fc90c15c6f0 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc910939700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c06df90 con 0x7fc90c06ea70 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc910939700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c06e270 con 0x7fc90c0b1780 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc90c15c6f0 0x7fc90c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54426/0 (socket says 172.21.15.104:54426) 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90a59c700 1 -- 172.21.15.104:0/4205141383 learned_addr learned my addr 172.21.15.104:0/4205141383 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc909d9b700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc90c0b1780 0x7fc90c06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968736942 0 0) 0x7fc90c0ae7e0 con 0x7fc90c15c6f0 2023-06-10T18:45:19.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f80035b0 con 0x7fc90c15c6f0 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220298662 0 0) 0x7fc90c06df90 con 0x7fc90c06ea70 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90c0ae7e0 con 0x7fc90c06ea70 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3414576825 0 0) 0x7fc8f80035b0 con 0x7fc90c15c6f0 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc90c0b1780 msgr2=0x7fc90c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc90c0b1780 0x7fc90c06cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 >> v1:172.21.15.114:6789/0 conn(0x7fc90c06ea70 legacy=0x7fc90c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c06e550 con 0x7fc90c15c6f0 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc8fc001b70 con 0x7fc90c15c6f0 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 >> v1:172.21.15.104:6789/0 conn(0x7fc90c15c6f0 legacy=0x7fc90c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8003bc0 0x7fc8f8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8f8004850 0x7fc8f8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8005200 0x7fc8f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c06e270 con 0x7fc8f8005200 2023-06-10T18:45:19.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800c0e0 con 0x7fc8f8004850 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800c270 con 0x7fc8f8003bc0 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc909d9b700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc90c0b1780 0x7fc90c06cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8005200 0x7fc8f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.663+0000 7fc90a59c700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8003bc0 0x7fc8f8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc909d9b700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8f8004850 0x7fc8f8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 -- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8003bc0 msgr2=0x7fc8f8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:19.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8003bc0 0x7fc8f8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 -- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8f8004850 msgr2=0x7fc8f8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8f8004850 0x7fc8f8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 -- 172.21.15.104:0/4205141383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f80035b0 con 0x7fc8f8005200 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc909d9b700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8f8004850 0x7fc8f8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8005200 0x7fc8f800ba10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc904001930 tx=0x7fc8f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f41a38a1fac597 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040052d0 con 0x7fc8f8005200 2023-06-10T18:45:19.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904005430 con 0x7fc8f8005200 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90959a700 1 -- 172.21.15.104:0/4205141383 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040038c0 con 0x7fc8f8005200 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc910939700 1 -- 172.21.15.104:0/4205141383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8005200 msgr2=0x7fc8f800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc910939700 1 --2- 172.21.15.104:0/4205141383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8f8005200 0x7fc8f800ba10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc904001930 tx=0x7fc8f8005880 comp rx=0 tx=0).stop 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.664+0000 7fc90ad9d700 1 -- 172.21.15.104:0/4205141383 reap_dead start 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.665+0000 7fc910939700 1 -- 172.21.15.104:0/4205141383 shutdown_connections 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.665+0000 7fc910939700 1 -- 172.21.15.104:0/4205141383 >> 172.21.15.104:0/4205141383 conn(0x7fc90c06b8a0 msgr2=0x7fc90c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:19.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.665+0000 7fc910939700 1 -- 172.21.15.104:0/4205141383 shutdown_connections 2023-06-10T18:45:19.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.665+0000 7fc910939700 1 -- 172.21.15.104:0/4205141383 wait complete. 2023-06-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.666+0000 7fc910939700 1 Processor -- start 2023-06-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.666+0000 7fc910939700 1 -- start start 2023-06-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 0x7fc90c0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c0c8780 0x7fc90c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c152610 con 0x7fc90c151c60 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0b2800 con 0x7fc90c0c8780 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0b2ae0 con 0x7fc8f8005200 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56614/0 (socket says 172.21.15.104:56614) 2023-06-10T18:45:19.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 -- 172.21.15.104:0/522488763 learned_addr learned my addr 172.21.15.104:0/522488763 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90a59c700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 0x7fc90c0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc909d9b700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c0c8780 0x7fc90c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 -- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 msgr2=0x7fc90c0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 0x7fc90c0cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 -- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c0c8780 msgr2=0x7fc90c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c0c8780 0x7fc90c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 -- 172.21.15.104:0/522488763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0b2dc0 con 0x7fc90c151c60 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90a59c700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 0x7fc90c0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc90ad9d700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc904004000 tx=0x7fc9040012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904007240 con 0x7fc90c151c60 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9040073c0 con 0x7fc90c151c60 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c0b30a0 con 0x7fc90c151c60 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904001d60 con 0x7fc90c151c60 2023-06-10T18:45:19.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.667+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c0b24b0 con 0x7fc90c151c60 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.668+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c0b35c0 con 0x7fc90c151c60 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.669+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc904003680 con 0x7fc90c151c60 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.669+0000 7fc902ffd700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 0x7fc8ec06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.669+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc90408a470 con 0x7fc90c151c60 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.669+0000 7fc90a59c700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 0x7fc8ec06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.670+0000 7fc90a59c700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 0x7fc8ec06d1c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc007fd0 tx=0x7fc8fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:19.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.673+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc90c0b35c0 con 0x7fc90c151c60 2023-06-10T18:45:19.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.862+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc90c0b35c0 con 0x7fc90c151c60 2023-06-10T18:45:19.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.862+0000 7fc902ffd700 1 -- 172.21.15.104:0/522488763 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc90c0b35c0 con 0x7fc90c151c60 2023-06-10T18:45:19.867 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:19.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 msgr2=0x7fc8ec06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 0x7fc8ec06d1c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc007fd0 tx=0x7fc8fc009040 comp rx=0 tx=0).stop 2023-06-10T18:45:19.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 msgr2=0x7fc90c1520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:19.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc904004000 tx=0x7fc9040012d0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 shutdown_connections 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90c151c60 0x7fc90c1520d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8f8005200 0x7fc90c0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ec06ad10 0x7fc8ec06d1c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 --2- 172.21.15.104:0/522488763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c0c8780 0x7fc90c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 >> 172.21.15.104:0/522488763 conn(0x7fc90c06b8a0 msgr2=0x7fc90c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 shutdown_connections 2023-06-10T18:45:19.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.864+0000 7fc910939700 1 -- 172.21.15.104:0/522488763 wait complete. 2023-06-10T18:45:19.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:20.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.999+0000 7f4f4a1f0700 1 Processor -- start 2023-06-10T18:45:20.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:19.999+0000 7f4f4a1f0700 1 -- start start 2023-06-10T18:45:20.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f4a1f0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f440cea50 0x7f4f440d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f4a1f0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f440ad3c0 con 0x7f4f44157b30 2023-06-10T18:45:20.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f4a1f0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f44157f00 con 0x7f4f440d0830 2023-06-10T18:45:20.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f4a1f0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f440b0170 con 0x7f4f440cea50 2023-06-10T18:45:20.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f42ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f440cea50 0x7f4f440d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f437fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4f44157b30 0x7f4f440ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54446/0 (socket says 172.21.15.104:54446) 2023-06-10T18:45:20.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f437fe700 1 -- 172.21.15.104:0/3058456901 learned_addr learned my addr 172.21.15.104:0/3058456901 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:20.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.000+0000 7f4f42ffd700 1 -- 172.21.15.104:0/3058456901 >> v1:172.21.15.114:6789/0 conn(0x7f4f440d0830 legacy=0x7f4f440b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f42ffd700 1 -- 172.21.15.104:0/3058456901 >> v1:172.21.15.104:6789/0 conn(0x7f4f44157b30 legacy=0x7f4f440ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f42ffd700 1 -- 172.21.15.104:0/3058456901 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440b0410 con 0x7f4f440cea50 2023-06-10T18:45:20.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/3058456901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f440d02f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009840 tx=0x7f4f34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7de0671237d6fa3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f427fc700 1 -- 172.21.15.104:0/3058456901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f34004230 con 0x7f4f440cea50 2023-06-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f427fc700 1 -- 172.21.15.104:0/3058456901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f34004390 con 0x7f4f440cea50 2023-06-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f427fc700 1 -- 172.21.15.104:0/3058456901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340036a0 con 0x7f4f440cea50 2023-06-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3058456901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 msgr2=0x7f4f440d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3058456901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f440d02f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009840 tx=0x7f4f34009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:20.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3058456901 shutdown_connections 2023-06-10T18:45:20.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3058456901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f440d02f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3058456901 >> 172.21.15.104:0/3058456901 conn(0x7f4f44089fd0 msgr2=0x7f4f4408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3058456901 shutdown_connections 2023-06-10T18:45:20.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.001+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3058456901 wait complete. 2023-06-10T18:45:20.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.003+0000 7f4f4a1f0700 1 Processor -- start 2023-06-10T18:45:20.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.003+0000 7f4f4a1f0700 1 -- start start 2023-06-10T18:45:20.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.003+0000 7f4f4a1f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f440d0830 0x7f4f440c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44157b30 0x7f4f4406f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44152740 con 0x7f4f440cea50 2023-06-10T18:45:20.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440b3880 con 0x7f4f44157b30 2023-06-10T18:45:20.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c5fd0 con 0x7f4f440d0830 2023-06-10T18:45:20.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56624/0 (socket says 172.21.15.104:56624) 2023-06-10T18:45:20.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 -- 172.21.15.104:0/3411314715 learned_addr learned my addr 172.21.15.104:0/3411314715 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:20.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f440d0830 0x7f4f440c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f43fff700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44157b30 0x7f4f4406f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 -- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f440d0830 msgr2=0x7f4f440c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f440d0830 0x7f4f440c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 -- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44157b30 msgr2=0x7f4f4406f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44157b30 0x7f4f4406f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 -- 172.21.15.104:0/3411314715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340094a0 con 0x7f4f440cea50 2023-06-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f437fe700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c430 tx=0x7f4f3800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38005b70 con 0x7f4f440cea50 2023-06-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f38005cd0 con 0x7f4f440cea50 2023-06-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f440c5450 con 0x7f4f440cea50 2023-06-10T18:45:20.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f380072f0 con 0x7f4f440cea50 2023-06-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.004+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f440c5a10 con 0x7f4f440cea50 2023-06-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.005+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44074050 con 0x7f4f440cea50 2023-06-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.006+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4f38007450 con 0x7f4f440cea50 2023-06-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.006+0000 7f4f2bfff700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.006+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4f38088aa0 con 0x7f4f440cea50 2023-06-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.006+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.007+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009840 tx=0x7f4f3401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.010+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4f38053820 con 0x7f4f440cea50 2023-06-10T18:45:20.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.197+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4f440af260 con 0x7f4f440cea50 2023-06-10T18:45:20.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.197+0000 7f4f2bfff700 1 -- 172.21.15.104:0/3411314715 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4f38017020 con 0x7f4f440cea50 2023-06-10T18:45:20.202 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 msgr2=0x7f4f3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009840 tx=0x7f4f3401c040 comp rx=0 tx=0).stop 2023-06-10T18:45:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 msgr2=0x7f4f44152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c430 tx=0x7f4f3800c740 comp rx=0 tx=0).stop 2023-06-10T18:45:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 shutdown_connections 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44157b30 0x7f4f4406f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440cea50 0x7f4f44152200 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 --2- 172.21.15.104:0/3411314715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f440d0830 0x7f4f440c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 >> 172.21.15.104:0/3411314715 conn(0x7f4f44089fd0 msgr2=0x7f4f44156510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 shutdown_connections 2023-06-10T18:45:20.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.199+0000 7f4f4a1f0700 1 -- 172.21.15.104:0/3411314715 wait complete. 2023-06-10T18:45:20.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:20.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.332+0000 7ff18ce2a700 1 Processor -- start 2023-06-10T18:45:20.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.332+0000 7ff18ce2a700 1 -- start start 2023-06-10T18:45:20.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18ce2a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff1880b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18ce2a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff18806cde0 0x7ff1880ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18ce2a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18806e250 con 0x7ff18815c580 2023-06-10T18:45:20.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18ce2a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18815c950 con 0x7ff18806cde0 2023-06-10T18:45:20.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18ce2a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880b1e80 con 0x7ff18806e6e0 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff18815c580 0x7ff1880b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54460/0 (socket says 172.21.15.104:54460) 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff18659c700 1 -- 172.21.15.104:0/3993245631 learned_addr learned my addr 172.21.15.104:0/3993245631 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff185d9b700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff1880b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.333+0000 7ff186d9d700 1 --2- 172.21.15.104:0/3993245631 >> v2:172.21.15.114:3300/0 conn(0x7ff18806cde0 0x7ff1880ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048977431 0 0) 0x7ff18806e250 con 0x7ff18815c580 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1700035b0 con 0x7ff18815c580 2023-06-10T18:45:20.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035914062 0 0) 0x7ff1700035b0 con 0x7ff18815c580 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 msgr2=0x7ff1880b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff1880b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 >> v2:172.21.15.114:3300/0 conn(0x7ff18806cde0 msgr2=0x7ff1880ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 --2- 172.21.15.104:0/3993245631 >> v2:172.21.15.114:3300/0 conn(0x7ff18806cde0 0x7ff1880ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18806d920 con 0x7ff18815c580 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff17c0017f0 con 0x7ff18815c580 2023-06-10T18:45:20.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 >> v1:172.21.15.104:6789/0 conn(0x7ff18815c580 legacy=0x7ff1880b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff170003a00 0x7ff170003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff170004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff170005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18815c950 con 0x7ff170004d90 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880b1e80 con 0x7ff170003a00 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000ba10 con 0x7ff18806e6e0 2023-06-10T18:45:20.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff170005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18659c700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff170003a00 0x7ff170003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff185d9b700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff170004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 -- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 msgr2=0x7ff170004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff18806e6e0 0x7ff170004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 -- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff170003a00 msgr2=0x7ff170003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff170003a00 0x7ff170003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 -- 172.21.15.104:0/3993245631 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1700035b0 con 0x7ff170004d90 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff186d9d700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff170005200 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff180009fc0 tx=0x7ff170005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50131a986c231251 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.334+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18000ebd0 con 0x7ff170004d90 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000f040 con 0x7ff170004d90 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18559a700 1 -- 172.21.15.104:0/3993245631 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180014480 con 0x7ff170004d90 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 -- 172.21.15.104:0/3993245631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 msgr2=0x7ff170005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/3993245631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff170005200 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff180009fc0 tx=0x7ff170005880 comp rx=0 tx=0).stop 2023-06-10T18:45:20.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff186d9d700 1 -- 172.21.15.104:0/3993245631 reap_dead start 2023-06-10T18:45:20.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 -- 172.21.15.104:0/3993245631 shutdown_connections 2023-06-10T18:45:20.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 -- 172.21.15.104:0/3993245631 >> 172.21.15.104:0/3993245631 conn(0x7ff18806b8a0 msgr2=0x7ff1880cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 -- 172.21.15.104:0/3993245631 shutdown_connections 2023-06-10T18:45:20.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.335+0000 7ff18ce2a700 1 -- 172.21.15.104:0/3993245631 wait complete. 2023-06-10T18:45:20.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 Processor -- start 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 -- start start 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1881e5c10 0x7ff1881468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff188146df0 0x7ff1881498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1881e5780 con 0x7ff170004d90 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff188147270 con 0x7ff1881e5c10 2023-06-10T18:45:20.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18ce2a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1881e4e50 con 0x7ff188146df0 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56644/0 (socket says 172.21.15.104:56644) 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.337+0000 7ff18659c700 1 -- 172.21.15.104:0/1684081488 learned_addr learned my addr 172.21.15.104:0/1684081488 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 -- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff188146df0 msgr2=0x7ff1881498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff188146df0 0x7ff1881498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 -- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1881e5c10 msgr2=0x7ff1881468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1881e5c10 0x7ff1881468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 -- 172.21.15.104:0/1684081488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1881e5130 con 0x7ff170004d90 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18659c700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00f930 tx=0x7ff17c00fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0072e0 con 0x7ff170004d90 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c007440 con 0x7ff170004d90 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1881e5410 con 0x7ff170004d90 2023-06-10T18:45:20.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c005830 con 0x7ff170004d90 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.338+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1880cb570 con 0x7ff170004d90 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.339+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff188066f10 con 0x7ff170004d90 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.339+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff17c005990 con 0x7ff170004d90 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.340+0000 7ff176ffd700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 0x7ff16806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.340+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff17c089680 con 0x7ff170004d90 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.340+0000 7ff185d9b700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 0x7ff16806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.340+0000 7ff185d9b700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 0x7ff16806d1c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1881e6000 tx=0x7ff178009670 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.344+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff17c054400 con 0x7ff170004d90 2023-06-10T18:45:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.528+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1880cb570 con 0x7ff170004d90 2023-06-10T18:45:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.529+0000 7ff176ffd700 1 -- 172.21.15.104:0/1684081488 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff17c01a310 con 0x7ff170004d90 2023-06-10T18:45:20.534 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:20.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 msgr2=0x7ff16806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 0x7ff16806d1c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1881e6000 tx=0x7ff178009670 comp rx=0 tx=0).stop 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 msgr2=0x7ff1880cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00f930 tx=0x7ff17c00fcf0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 shutdown_connections 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff188146df0 0x7ff1881498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff170004d90 0x7ff1880cae50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ad10 0x7ff16806d1c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 --2- 172.21.15.104:0/1684081488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1881e5c10 0x7ff1881468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 >> 172.21.15.104:0/1684081488 conn(0x7ff18806b8a0 msgr2=0x7ff1880cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 shutdown_connections 2023-06-10T18:45:20.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.531+0000 7ff18ce2a700 1 -- 172.21.15.104:0/1684081488 wait complete. 2023-06-10T18:45:20.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.666+0000 7fb8dac28700 1 Processor -- start 2023-06-10T18:45:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.666+0000 7fb8dac28700 1 -- start start 2023-06-10T18:45:20.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8dac28700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8d415c730 0x7fb8d40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8dac28700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb8d40cd720 0x7fb8d406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8dac28700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d415cb00 con 0x7fb8d406d2b0 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8dac28700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40cdbc0 con 0x7fb8d415c730 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8dac28700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40acd90 con 0x7fb8d40cd720 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8d89c4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8d415c730 0x7fb8d40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8d91c5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb8d406d2b0 0x7fb8d406d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54482/0 (socket says 172.21.15.104:54482) 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.667+0000 7fb8d91c5700 1 -- 172.21.15.104:0/2185238914 learned_addr learned my addr 172.21.15.104:0/2185238914 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d3fff700 1 --2- 172.21.15.104:0/2185238914 >> v2:172.21.15.114:3300/0 conn(0x7fb8d40cd720 0x7fb8d406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d89c4700 1 -- 172.21.15.104:0/2185238914 >> v2:172.21.15.114:3300/0 conn(0x7fb8d40cd720 msgr2=0x7fb8d406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d89c4700 1 --2- 172.21.15.104:0/2185238914 >> v2:172.21.15.114:3300/0 conn(0x7fb8d40cd720 0x7fb8d406cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d89c4700 1 -- 172.21.15.104:0/2185238914 >> v1:172.21.15.104:6789/0 conn(0x7fb8d406d2b0 legacy=0x7fb8d406d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d89c4700 1 -- 172.21.15.104:0/2185238914 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40ac460 con 0x7fb8d415c730 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d89c4700 1 --2- 172.21.15.104:0/2185238914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d40cd1e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009820 tx=0x7fb8c8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdaa8a63981ffca6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d37fe700 1 -- 172.21.15.104:0/2185238914 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8005e30 con 0x7fb8d415c730 2023-06-10T18:45:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d37fe700 1 -- 172.21.15.104:0/2185238914 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8006030 con 0x7fb8d415c730 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8d37fe700 1 -- 172.21.15.104:0/2185238914 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80063c0 con 0x7fb8d415c730 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 -- 172.21.15.104:0/2185238914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 msgr2=0x7fb8d40cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 --2- 172.21.15.104:0/2185238914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d40cd1e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009820 tx=0x7fb8c8009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 -- 172.21.15.104:0/2185238914 shutdown_connections 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 --2- 172.21.15.104:0/2185238914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d40cd1e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 --2- 172.21.15.104:0/2185238914 >> v2:172.21.15.114:3300/0 conn(0x7fb8d40cd720 0x7fb8d406cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.668+0000 7fb8dac28700 1 -- 172.21.15.104:0/2185238914 >> 172.21.15.104:0/2185238914 conn(0x7fb8d406b8a0 msgr2=0x7fb8d40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.669+0000 7fb8dac28700 1 -- 172.21.15.104:0/2185238914 shutdown_connections 2023-06-10T18:45:20.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.669+0000 7fb8dac28700 1 -- 172.21.15.104:0/2185238914 wait complete. 2023-06-10T18:45:20.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.670+0000 7fb8dac28700 1 Processor -- start 2023-06-10T18:45:20.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.670+0000 7fb8dac28700 1 -- start start 2023-06-10T18:45:20.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.670+0000 7fb8dac28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8d406d2b0 0x7fb8d41f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.670+0000 7fb8dac28700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8d40cd720 0x7fb8d40c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d41f3830 con 0x7fb8d415c730 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d414c4d0 con 0x7fb8d406d2b0 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4148d30 con 0x7fb8d40cd720 2023-06-10T18:45:20.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56662/0 (socket says 172.21.15.104:56662) 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d89c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8d406d2b0 0x7fb8d41f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 -- 172.21.15.104:0/327410886 learned_addr learned my addr 172.21.15.104:0/327410886 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d3fff700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8d40cd720 0x7fb8d40c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 -- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8d40cd720 msgr2=0x7fb8d40c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8d40cd720 0x7fb8d40c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 -- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8d406d2b0 msgr2=0x7fb8d41f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8d406d2b0 0x7fb8d41f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 -- 172.21.15.104:0/327410886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8009430 con 0x7fb8d415c730 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d91c5700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00cbc0 tx=0x7fb8cc00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00a9e0 con 0x7fb8d415c730 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc00ab40 con 0x7fb8d415c730 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d4149010 con 0x7fb8d415c730 2023-06-10T18:45:20.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc012400 con 0x7fb8d415c730 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.671+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d41f3450 con 0x7fb8d415c730 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.672+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d41f3970 con 0x7fb8d415c730 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.673+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb8cc01b3b0 con 0x7fb8d415c730 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.673+0000 7fb8d17fa700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 0x7fb8b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.673+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb8cc0890c0 con 0x7fb8d415c730 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.673+0000 7fb8d89c4700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 0x7fb8b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:20.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.674+0000 7fb8d89c4700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 0x7fb8b806d190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8000c00 tx=0x7fb8c8008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:20.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.677+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb8d41f3970 con 0x7fb8d415c730 2023-06-10T18:45:20.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.865+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb8d415cb00 con 0x7fb8d415c730 2023-06-10T18:45:20.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.866+0000 7fb8d17fa700 1 -- 172.21.15.104:0/327410886 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb8d415cb00 con 0x7fb8d415c730 2023-06-10T18:45:20.870 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:20.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 msgr2=0x7fb8b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 0x7fb8b806d190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8000c00 tx=0x7fb8c8008070 comp rx=0 tx=0).stop 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 msgr2=0x7fb8d41481c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00cbc0 tx=0x7fb8cc00ced0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 shutdown_connections 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8b806ace0 0x7fb8b806d190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8d406d2b0 0x7fb8d41f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8d415c730 0x7fb8d41481c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 --2- 172.21.15.104:0/327410886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8d40cd720 0x7fb8d40c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:20.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 >> 172.21.15.104:0/327410886 conn(0x7fb8d406b8a0 msgr2=0x7fb8d40b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:20.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 shutdown_connections 2023-06-10T18:45:20.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:20.868+0000 7fb8dac28700 1 -- 172.21.15.104:0/327410886 wait complete. 2023-06-10T18:45:20.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:21.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.004+0000 7f0e8c71c700 1 Processor -- start 2023-06-10T18:45:21.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.004+0000 7f0e8c71c700 1 -- start start 2023-06-10T18:45:21.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8c71c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0e8406e6e0 0x7f0e840b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8c71c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8406e250 con 0x7f0e8415c580 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8c71c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8415c950 con 0x7f0e8406cde0 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8c71c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e840b1e80 con 0x7f0e8406e6e0 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e89cb7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0e8406e6e0 0x7f0e840b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8a4b8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0e8415c580 0x7f0e840b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54494/0 (socket says 172.21.15.104:54494) 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e8a4b8700 1 -- 172.21.15.104:0/2750690263 learned_addr learned my addr 172.21.15.104:0/2750690263 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:21.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e89cb7700 1 -- 172.21.15.104:0/2750690263 >> v1:172.21.15.114:6789/0 conn(0x7f0e8406cde0 legacy=0x7f0e840ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e89cb7700 1 -- 172.21.15.104:0/2750690263 >> v1:172.21.15.104:6789/0 conn(0x7f0e8415c580 legacy=0x7f0e840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e89cb7700 1 -- 172.21.15.104:0/2750690263 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8406d920 con 0x7f0e8406e6e0 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e894b6700 1 -- 172.21.15.104:0/2750690263 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440774665 0 0) 0x7f0e8406e250 con 0x7f0e8415c580 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e89cb7700 1 --2- 172.21.15.104:0/2750690263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e840b1780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009840 tx=0x7f0e74009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5eb88b0bd3db4d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e894b6700 1 -- 172.21.15.104:0/2750690263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004230 con 0x7f0e8406e6e0 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e894b6700 1 -- 172.21.15.104:0/2750690263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74004390 con 0x7f0e8406e6e0 2023-06-10T18:45:21.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.005+0000 7f0e894b6700 1 -- 172.21.15.104:0/2750690263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e740036a0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 -- 172.21.15.104:0/2750690263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 msgr2=0x7f0e840b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/2750690263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e840b1780 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009840 tx=0x7f0e74009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 -- 172.21.15.104:0/2750690263 shutdown_connections 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/2750690263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e840b1780 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 -- 172.21.15.104:0/2750690263 >> 172.21.15.104:0/2750690263 conn(0x7f0e8406b8a0 msgr2=0x7f0e840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 -- 172.21.15.104:0/2750690263 shutdown_connections 2023-06-10T18:45:21.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.006+0000 7f0e8c71c700 1 -- 172.21.15.104:0/2750690263 wait complete. 2023-06-10T18:45:21.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.007+0000 7f0e8c71c700 1 Processor -- start 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.007+0000 7f0e8c71c700 1 -- start start 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 0x7f0e840af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 0x7f0e8414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840afc50 con 0x7f0e8406e6e0 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840aff30 con 0x7f0e8406cde0 2023-06-10T18:45:21.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8c71c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840b0210 con 0x7f0e8415c580 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56674/0 (socket says 172.21.15.104:56674) 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 -- 172.21.15.104:0/4126701315 learned_addr learned my addr 172.21.15.104:0/4126701315 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8acb9700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 0x7f0e8414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8a4b8700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 0x7f0e840af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 -- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 msgr2=0x7f0e8414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 0x7f0e8414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 -- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 msgr2=0x7f0e840af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 0x7f0e840af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 -- 172.21.15.104:0/4126701315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e740094a0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8acb9700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 0x7f0e8414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e8a4b8700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 0x7f0e840af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e89cb7700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e740090b0 tx=0x7f0e74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74011830 con 0x7f0e8406e6e0 2023-06-10T18:45:21.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.008+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7401e070 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.009+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e840b04f0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.009+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74011bf0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.009+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e841576b0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.009+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e840b0a10 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.010+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0e74011d50 con 0x7f0e8406e6e0 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.010+0000 7f0e7affd700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 0x7f0e6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.010+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0e7408aac0 con 0x7f0e8406e6e0 2023-06-10T18:45:21.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.011+0000 7f0e8a4b8700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 0x7f0e6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.011+0000 7f0e8a4b8700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 0x7f0e6806d190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e800010e0 tx=0x7f0e80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.015+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0e840b0a10 con 0x7f0e8406e6e0 2023-06-10T18:45:21.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.199+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0e840b0a10 con 0x7f0e8406e6e0 2023-06-10T18:45:21.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.199+0000 7f0e7affd700 1 -- 172.21.15.104:0/4126701315 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0e74020020 con 0x7f0e8406e6e0 2023-06-10T18:45:21.204 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 msgr2=0x7f0e6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 0x7f0e6806d190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e800010e0 tx=0x7f0e80009040 comp rx=0 tx=0).stop 2023-06-10T18:45:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 msgr2=0x7f0e84157170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e740090b0 tx=0x7f0e74004750 comp rx=0 tx=0).stop 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 shutdown_connections 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e6806ace0 0x7f0e6806d190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e8406cde0 0x7f0e840af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e8415c580 0x7f0e8414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 --2- 172.21.15.104:0/4126701315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e8406e6e0 0x7f0e84157170 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 >> 172.21.15.104:0/4126701315 conn(0x7f0e8406b8a0 msgr2=0x7f0e84159890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 shutdown_connections 2023-06-10T18:45:21.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.201+0000 7f0e8c71c700 1 -- 172.21.15.104:0/4126701315 wait complete. 2023-06-10T18:45:21.218 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:21.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.333+0000 7f7dc1940700 1 Processor -- start 2023-06-10T18:45:21.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.333+0000 7f7dc1940700 1 -- start start 2023-06-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dc1940700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc0cd870 0x7f7dbc0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dc1940700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dc1940700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0b3610 con 0x7f7dbc15a410 2023-06-10T18:45:21.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dc1940700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0d0000 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dc1940700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0b38d0 con 0x7f7dbc0cd870 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56684/0 (socket says 172.21.15.104:56684) 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 -- 172.21.15.104:0/287246544 learned_addr learned my addr 172.21.15.104:0/287246544 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dba7fc700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc0cd870 0x7f7dbc0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 -- 172.21.15.104:0/287246544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc0cd870 msgr2=0x7f7dbc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:21.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc0cd870 0x7f7dbc0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 -- 172.21.15.104:0/287246544 >> v1:172.21.15.114:6789/0 conn(0x7f7dbc15a410 legacy=0x7f7dbc0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 -- 172.21.15.104:0/287246544 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc15e6b0 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.334+0000 7f7dbb7fe700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e532fe515b70d4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7db9ffb700 1 -- 172.21.15.104:0/287246544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400e070 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7db9ffb700 1 -- 172.21.15.104:0/287246544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400b910 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7db9ffb700 1 -- 172.21.15.104:0/287246544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400bcc0 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 -- 172.21.15.104:0/287246544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc15e0f0 msgr2=0x7f7dbc0ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 -- 172.21.15.104:0/287246544 shutdown_connections 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc15e0f0 0x7f7dbc0ce8b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 --2- 172.21.15.104:0/287246544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc0cd870 0x7f7dbc0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 -- 172.21.15.104:0/287246544 >> 172.21.15.104:0/287246544 conn(0x7f7dbc069e70 msgr2=0x7f7dbc06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 -- 172.21.15.104:0/287246544 shutdown_connections 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.335+0000 7f7dc1940700 1 -- 172.21.15.104:0/287246544 wait complete. 2023-06-10T18:45:21.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.336+0000 7f7dc1940700 1 Processor -- start 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 -- start start 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 0x7f7dbc152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc15e0f0 0x7f7dbc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc154f30 con 0x7f7dbc0cd870 2023-06-10T18:45:21.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc154660 con 0x7f7dbc15a410 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dc1940700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc1539d0 con 0x7f7dbc15e0f0 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dbaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dbaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56686/0 (socket says 172.21.15.104:56686) 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dbaffd700 1 -- 172.21.15.104:0/1969927875 learned_addr learned my addr 172.21.15.104:0/1969927875 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.337+0000 7f7dba7fc700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 0x7f7dbc152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbb7fe700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc15e0f0 0x7f7dbc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 -- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc15e0f0 msgr2=0x7f7dbc0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc15e0f0 0x7f7dbc0b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 -- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 msgr2=0x7f7dbc152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 0x7f7dbc152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 -- 172.21.15.104:0/1969927875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009b30 con 0x7f7dbc0cd870 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dba7fc700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 0x7f7dbc152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dbaffd700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c450 tx=0x7f7db000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db00052d0 con 0x7f7dbc0cd870 2023-06-10T18:45:21.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db0005430 con 0x7f7dbc0cd870 2023-06-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc154870 con 0x7f7dbc0cd870 2023-06-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db00038c0 con 0x7f7dbc0cd870 2023-06-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.338+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc154340 con 0x7f7dbc0cd870 2023-06-10T18:45:21.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.339+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc155070 con 0x7f7dbc0cd870 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.339+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7db0003a20 con 0x7f7dbc0cd870 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.340+0000 7f7da3fff700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 0x7f7da806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.340+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7db0088ac0 con 0x7f7dbc0cd870 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.340+0000 7f7dba7fc700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 0x7f7da806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.340+0000 7f7dba7fc700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 0x7f7da806d190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc1f36c0 tx=0x7f7dac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.344+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7dbc155070 con 0x7f7dbc0cd870 2023-06-10T18:45:21.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.546+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7dbc155070 con 0x7f7dbc0cd870 2023-06-10T18:45:21.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.547+0000 7f7da3fff700 1 -- 172.21.15.104:0/1969927875 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7dbc155070 con 0x7f7dbc0cd870 2023-06-10T18:45:21.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 msgr2=0x7f7da806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 0x7f7da806d190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc1f36c0 tx=0x7f7dac008040 comp rx=0 tx=0).stop 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 msgr2=0x7f7dbc153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c450 tx=0x7f7db000c810 comp rx=0 tx=0).stop 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 shutdown_connections 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7dbc15e0f0 0x7f7dbc0b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7dbc0cd870 0x7f7dbc153cb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7da806ace0 0x7f7da806d190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 --2- 172.21.15.104:0/1969927875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7dbc15a410 0x7f7dbc152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 >> 172.21.15.104:0/1969927875 conn(0x7f7dbc069e70 msgr2=0x7f7dbc0cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 shutdown_connections 2023-06-10T18:45:21.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.549+0000 7f7dc1940700 1 -- 172.21.15.104:0/1969927875 wait complete. 2023-06-10T18:45:21.555 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:21.565 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:45:21.565 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:21.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:21.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.701+0000 7f8ae1b39700 1 Processor -- start 2023-06-10T18:45:21.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.701+0000 7f8ae1b39700 1 -- start start 2023-06-10T18:45:21.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.702+0000 7f8ae1b39700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 0x7f8adc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.702+0000 7f8ae1b39700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8adc0cb300 0x7f8adc06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.702+0000 7f8ae1b39700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0c9e60 con 0x7f8adc0aeaa0 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.702+0000 7f8ae1b39700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0c9bc0 con 0x7f8adc0cb300 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.702+0000 7f8ae1b39700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0aee70 con 0x7f8adc0cd850 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8adc0cb300 0x7f8adc06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8adc0cb300 0x7f8adc06d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56696/0 (socket says 172.21.15.104:56696) 2023-06-10T18:45:21.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 -- 172.21.15.104:0/655081946 learned_addr learned my addr 172.21.15.104:0/655081946 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adaffd700 1 --2- 172.21.15.104:0/655081946 >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 0x7f8adc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 -- 172.21.15.104:0/655081946 >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 msgr2=0x7f8adc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 --2- 172.21.15.104:0/655081946 >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 0x7f8adc06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 -- 172.21.15.104:0/655081946 >> v1:172.21.15.114:6789/0 conn(0x7f8adc0aeaa0 legacy=0x7f8adc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 -- 172.21.15.104:0/655081946 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc06e540 con 0x7f8adc0cb300 2023-06-10T18:45:21.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adaffd700 1 --2- 172.21.15.104:0/655081946 >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 0x7f8adc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8adbfff700 1 --2- 172.21.15.104:0/655081946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc06d290 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8adc070110 tx=0x7f8ad400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191a0c4e15d3d3f2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8ada7fc700 1 -- 172.21.15.104:0/655081946 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400e070 con 0x7f8adc0cb300 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8ada7fc700 1 -- 172.21.15.104:0/655081946 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad4003a20 con 0x7f8adc0cb300 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.703+0000 7f8ada7fc700 1 -- 172.21.15.104:0/655081946 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4003dd0 con 0x7f8adc0cb300 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 -- 172.21.15.104:0/655081946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 msgr2=0x7f8adc06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/655081946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc06d290 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8adc070110 tx=0x7f8ad400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:21.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 -- 172.21.15.104:0/655081946 shutdown_connections 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/655081946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc06d290 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/655081946 >> v2:172.21.15.114:3300/0 conn(0x7f8adc0cd850 0x7f8adc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 -- 172.21.15.104:0/655081946 >> 172.21.15.104:0/655081946 conn(0x7f8adc06b8a0 msgr2=0x7f8adc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 -- 172.21.15.104:0/655081946 shutdown_connections 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.704+0000 7f8ae1b39700 1 -- 172.21.15.104:0/655081946 wait complete. 2023-06-10T18:45:21.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.705+0000 7f8ae1b39700 1 Processor -- start 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.705+0000 7f8ae1b39700 1 -- start start 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 0x7f8adc14b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8adc0cd850 0x7f8adc0c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc15a4c0 con 0x7f8adc0cb300 2023-06-10T18:45:21.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc14c290 con 0x7f8adc0cd850 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0c9700 con 0x7f8adc0aeaa0 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56702/0 (socket says 172.21.15.104:56702) 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 -- 172.21.15.104:0/2348171901 learned_addr learned my addr 172.21.15.104:0/2348171901 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adbfff700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8adc0cd850 0x7f8adc0c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adb7fe700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 0x7f8adc14b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 -- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 msgr2=0x7f8adc14b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 0x7f8adc14b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 -- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8adc0cd850 msgr2=0x7f8adc0c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8adc0cd850 0x7f8adc0c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 -- 172.21.15.104:0/2348171901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad4009b30 con 0x7f8adc0cb300 2023-06-10T18:45:21.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adb7fe700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 0x7f8adc14b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8adaffd700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00c820 tx=0x7f8acc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc011800 con 0x7f8adc0cb300 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc011960 con 0x7f8adc0cb300 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc15a710 con 0x7f8adc0cb300 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.706+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc011d10 con 0x7f8adc0cb300 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.707+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc15acd0 con 0x7f8adc0cb300 2023-06-10T18:45:21.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.707+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc15aef0 con 0x7f8adc0cb300 2023-06-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.708+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8acc018360 con 0x7f8adc0cb300 2023-06-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.708+0000 7f8ac3fff700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 0x7f8ac806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.708+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8acc089780 con 0x7f8adc0cb300 2023-06-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.708+0000 7f8adb7fe700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 0x7f8ac806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:21.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.709+0000 7f8adb7fe700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 0x7f8ac806d190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0003f80 tx=0x7f8ad0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:21.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.712+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8acc057fa0 con 0x7f8adc0cb300 2023-06-10T18:45:21.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.901+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8adc074050 con 0x7f8adc0cb300 2023-06-10T18:45:21.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.902+0000 7f8ac3fff700 1 -- 172.21.15.104:0/2348171901 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f8acc057b30 con 0x7f8adc0cb300 2023-06-10T18:45:21.907 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:21.908 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 msgr2=0x7f8ac806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 0x7f8ac806d190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0003f80 tx=0x7f8ad0009040 comp rx=0 tx=0).stop 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 msgr2=0x7f8adc14bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00c820 tx=0x7f8acc00cbe0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 shutdown_connections 2023-06-10T18:45:21.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ac806ace0 0x7f8ac806d190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8adc0cd850 0x7f8adc0c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8adc0aeaa0 0x7f8adc14b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 --2- 172.21.15.104:0/2348171901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8adc0cb300 0x7f8adc14bd50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 >> 172.21.15.104:0/2348171901 conn(0x7f8adc06b8a0 msgr2=0x7f8adc06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 shutdown_connections 2023-06-10T18:45:21.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:21.904+0000 7f8ae1b39700 1 -- 172.21.15.104:0/2348171901 wait complete. 2023-06-10T18:45:21.921 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:21.922 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:22.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.039+0000 7f35a64dc700 1 Processor -- start 2023-06-10T18:45:22.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.039+0000 7f35a64dc700 1 -- start start 2023-06-10T18:45:22.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a64dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a015c730 0x7f35a00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a64dc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35a00ae230 0x7f35a006d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a64dc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a015cb00 con 0x7f35a00cd720 2023-06-10T18:45:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a64dc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a00cdb90 con 0x7f35a00ae230 2023-06-10T18:45:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a64dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00acd90 con 0x7f35a015c730 2023-06-10T18:45:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35a00ae230 0x7f35a006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35a00ae230 0x7f35a006d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56712/0 (socket says 172.21.15.104:56712) 2023-06-10T18:45:22.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 -- 172.21.15.104:0/1154365267 learned_addr learned my addr 172.21.15.104:0/1154365267 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:22.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f359ffff700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a015c730 0x7f35a00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 -- 172.21.15.104:0/1154365267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a015c730 msgr2=0x7f35a00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:22.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a015c730 0x7f35a00cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 -- 172.21.15.104:0/1154365267 >> v1:172.21.15.114:6789/0 conn(0x7f35a00cd720 legacy=0x7f35a006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.040+0000 7f35a4a79700 1 -- 172.21.15.104:0/1154365267 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00ac460 con 0x7f35a00ae230 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a4a79700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00ae230 0x7f35a006d480 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3598009e50 tx=0x7f359800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6645f2375f2295 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f359effd700 1 -- 172.21.15.104:0/1154365267 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800e070 con 0x7f35a00ae230 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f359effd700 1 -- 172.21.15.104:0/1154365267 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800b9c0 con 0x7f35a00ae230 2023-06-10T18:45:22.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f359effd700 1 -- 172.21.15.104:0/1154365267 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800bd70 con 0x7f35a00ae230 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 -- 172.21.15.104:0/1154365267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00ae230 msgr2=0x7f35a006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00ae230 0x7f35a006d480 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3598009e50 tx=0x7f359800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 -- 172.21.15.104:0/1154365267 shutdown_connections 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00ae230 0x7f35a006d480 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 --2- 172.21.15.104:0/1154365267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a015c730 0x7f35a00cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 -- 172.21.15.104:0/1154365267 >> 172.21.15.104:0/1154365267 conn(0x7f35a006b8a0 msgr2=0x7f35a00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 -- 172.21.15.104:0/1154365267 shutdown_connections 2023-06-10T18:45:22.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.041+0000 7f35a64dc700 1 -- 172.21.15.104:0/1154365267 wait complete. 2023-06-10T18:45:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.042+0000 7f35a64dc700 1 Processor -- start 2023-06-10T18:45:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 -- start start 2023-06-10T18:45:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a00ae230 0x7f35a00c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a015c730 0x7f35a00b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c7500 con 0x7f35a00cd720 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00b0e60 con 0x7f35a015c730 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.043+0000 7f35a64dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c6880 con 0x7f35a00ae230 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56718/0 (socket says 172.21.15.104:56718) 2023-06-10T18:45:22.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 -- 172.21.15.104:0/540475604 learned_addr learned my addr 172.21.15.104:0/540475604 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359ffff700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a00ae230 0x7f35a00c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f35a4a79700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a015c730 0x7f35a00b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 -- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a00ae230 msgr2=0x7f35a00c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a00ae230 0x7f35a00c7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 -- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a015c730 msgr2=0x7f35a00b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a015c730 0x7f35a00b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 -- 172.21.15.104:0/540475604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598009b30 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359f7fe700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f359000c490 tx=0x7f359000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000cea0 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35900052f0 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a00c6b60 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590006860 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.044+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a00c7120 con 0x7f35a00cd720 2023-06-10T18:45:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.045+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00d05b0 con 0x7f35a00cd720 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.046+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3590003680 con 0x7f35a00cd720 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.046+0000 7f359cff9700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.046+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3590088740 con 0x7f35a00cd720 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.046+0000 7f359ffff700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.047+0000 7f359ffff700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3594006fd0 tx=0x7f3594008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.050+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3590053550 con 0x7f35a00cd720 2023-06-10T18:45:22.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.236+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35a015cb00 con 0x7f35a00cd720 2023-06-10T18:45:22.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.236+0000 7f359cff9700 1 -- 172.21.15.104:0/540475604 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35a015cb00 con 0x7f35a00cd720 2023-06-10T18:45:22.241 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 msgr2=0x7f358c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3594006fd0 tx=0x7f3594008040 comp rx=0 tx=0).stop 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 msgr2=0x7f35a0159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f359000c490 tx=0x7f359000c7a0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 shutdown_connections 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a015c730 0x7f35a00b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a00ae230 0x7f35a00c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 --2- 172.21.15.104:0/540475604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a00cd720 0x7f35a0159090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 >> 172.21.15.104:0/540475604 conn(0x7f35a006b8a0 msgr2=0x7f35a00ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 shutdown_connections 2023-06-10T18:45:22.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.238+0000 7f35a64dc700 1 -- 172.21.15.104:0/540475604 wait complete. 2023-06-10T18:45:22.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:22.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.372+0000 7f43e7233700 1 Processor -- start 2023-06-10T18:45:22.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.372+0000 7f43e7233700 1 -- start start 2023-06-10T18:45:22.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e7233700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43e00d0a00 0x7f43e0164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e7233700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00ad3c0 con 0x7f43e00cea50 2023-06-10T18:45:22.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e7233700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e0157f00 con 0x7f43e0157b30 2023-06-10T18:45:22.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e7233700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00d0e60 con 0x7f43e00d0a00 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43dffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f43e00cea50 0x7f43e00d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54550/0 (socket says 172.21.15.104:54550) 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43dffff700 1 -- 172.21.15.104:0/2635051199 learned_addr learned my addr 172.21.15.104:0/2635051199 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e57d0700 1 --2- 172.21.15.104:0/2635051199 >> v2:172.21.15.114:3300/0 conn(0x7f43e00d0a00 0x7f43e0164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187375948 0 0) 0x7f43e00ad3c0 con 0x7f43e00cea50 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d00035b0 con 0x7f43e00cea50 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993405288 0 0) 0x7f43d00035b0 con 0x7f43e00cea50 2023-06-10T18:45:22.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e57d0700 1 -- 172.21.15.104:0/2635051199 >> v1:172.21.15.114:6789/0 conn(0x7f43e0157b30 legacy=0x7f43e00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e57d0700 1 -- 172.21.15.104:0/2635051199 >> v1:172.21.15.104:6789/0 conn(0x7f43e00cea50 legacy=0x7f43e00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e57d0700 1 -- 172.21.15.104:0/2635051199 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00b0170 con 0x7f43e00d0a00 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43e57d0700 1 --2- 172.21.15.104:0/2635051199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e00d0a00 0x7f43e0164e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009d90 tx=0x7f43d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbafc06ed0498cdc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.373+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800e070 con 0x7f43e00d0a00 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8003a20 con 0x7f43e00d0a00 2023-06-10T18:45:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43df7fe700 1 -- 172.21.15.104:0/2635051199 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc001670 con 0x7f43e00d0a00 2023-06-10T18:45:22.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 -- 172.21.15.104:0/2635051199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e00d0a00 msgr2=0x7f43e0164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 --2- 172.21.15.104:0/2635051199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e00d0a00 0x7f43e0164e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009d90 tx=0x7f43d800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:22.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 -- 172.21.15.104:0/2635051199 shutdown_connections 2023-06-10T18:45:22.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 --2- 172.21.15.104:0/2635051199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e00d0a00 0x7f43e0164e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 -- 172.21.15.104:0/2635051199 >> 172.21.15.104:0/2635051199 conn(0x7f43e0089fd0 msgr2=0x7f43e008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 -- 172.21.15.104:0/2635051199 shutdown_connections 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.374+0000 7f43e7233700 1 -- 172.21.15.104:0/2635051199 wait complete. 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.375+0000 7f43e7233700 1 Processor -- start 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.375+0000 7f43e7233700 1 -- start start 2023-06-10T18:45:22.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e00d0a00 0x7f43e01599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 0x7f43e0159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00ce7e0 con 0x7f43e00cea50 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e0153270 con 0x7f43e0157b30 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e7233700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00cdb60 con 0x7f43e00d0a00 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e4fcf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e4fcf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56728/0 (socket says 172.21.15.104:56728) 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e4fcf700 1 -- 172.21.15.104:0/2477207754 learned_addr learned my addr 172.21.15.104:0/2477207754 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.376+0000 7f43e57d0700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 0x7f43e0159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 -- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e00d0a00 msgr2=0x7f43e01599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43dffff700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e00d0a00 0x7f43e01599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e00d0a00 0x7f43e01599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 -- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 msgr2=0x7f43e0159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 0x7f43e0159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 -- 172.21.15.104:0/2477207754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d8009a70 con 0x7f43e00cea50 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e57d0700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 0x7f43e0159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e4fcf700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43d400c450 tx=0x7f43d400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d40052d0 con 0x7f43e00cea50 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4005430 con 0x7f43e00cea50 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e00cde40 con 0x7f43e00cea50 2023-06-10T18:45:22.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d40038c0 con 0x7f43e00cea50 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.377+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e00ce400 con 0x7f43e00cea50 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.378+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e0074050 con 0x7f43e00cea50 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.378+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f43d4003a20 con 0x7f43e00cea50 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.379+0000 7f43dd7fa700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 0x7f43c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.379+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f43d4088b10 con 0x7f43e00cea50 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.379+0000 7f43dffff700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 0x7f43c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.379+0000 7f43dffff700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 0x7f43c406d190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43e00b6040 tx=0x7f43cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.383+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43d4053920 con 0x7f43e00cea50 2023-06-10T18:45:22.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.571+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f43e0157200 con 0x7f43e00cea50 2023-06-10T18:45:22.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.571+0000 7f43dd7fa700 1 -- 172.21.15.104:0/2477207754 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f43d4056f40 con 0x7f43e00cea50 2023-06-10T18:45:22.576 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 msgr2=0x7f43c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 0x7f43c406d190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43e00b6040 tx=0x7f43cc009040 comp rx=0 tx=0).stop 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 msgr2=0x7f43e01549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43d400c450 tx=0x7f43d400c810 comp rx=0 tx=0).stop 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 shutdown_connections 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43c406ace0 0x7f43c406d190 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43e0157b30 0x7f43e0159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43e00cea50 0x7f43e01549f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 --2- 172.21.15.104:0/2477207754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43e00d0a00 0x7f43e01599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 >> 172.21.15.104:0/2477207754 conn(0x7f43e0089fd0 msgr2=0x7f43e00b1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 shutdown_connections 2023-06-10T18:45:22.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.573+0000 7f43e7233700 1 -- 172.21.15.104:0/2477207754 wait complete. 2023-06-10T18:45:22.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:22.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.708+0000 7fd065ef6700 1 Processor -- start 2023-06-10T18:45:22.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.708+0000 7fd065ef6700 1 -- start start 2023-06-10T18:45:22.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd065ef6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 0x7fd0600b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd065ef6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd065ef6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06006e250 con 0x7fd06006e6e0 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd065ef6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06015c950 con 0x7fd06015c580 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd065ef6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0600b1e80 con 0x7fd06006cde0 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 0x7fd0600b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:47890/0 (socket says 172.21.15.104:47890) 2023-06-10T18:45:22.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 0x7fd0600b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:44238/0 (socket says 172.21.15.104:44238) 2023-06-10T18:45:22.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05ffff700 1 -- 172.21.15.104:0/1294305459 learned_addr learned my addr 172.21.15.104:0/1294305459 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:22.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.709+0000 7fd05effd700 1 --1- 172.21.15.104:0/1294305459 >> v1:172.21.15.114:6789/0 conn(0x7fd06006e6e0 0x7fd0600b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60414/0 (socket says 172.21.15.104:60414) 2023-06-10T18:45:22.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05ffff700 1 -- 172.21.15.104:0/1294305459 >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 msgr2=0x7fd0600b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:22.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05ffff700 1 --2- 172.21.15.104:0/1294305459 >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 0x7fd0600b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05ffff700 1 -- 172.21.15.104:0/1294305459 >> v1:172.21.15.114:6789/0 conn(0x7fd06006e6e0 legacy=0x7fd0600b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05ffff700 1 -- 172.21.15.104:0/1294305459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06006d920 con 0x7fd06006cde0 2023-06-10T18:45:22.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05ffff700 1 --2- 172.21.15.104:0/1294305459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd058009e50 tx=0x7fd05800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab11ab337deb124 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05e7fc700 1 -- 172.21.15.104:0/1294305459 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05800e070 con 0x7fd06006cde0 2023-06-10T18:45:22.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05e7fc700 1 -- 172.21.15.104:0/1294305459 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05800b910 con 0x7fd06006cde0 2023-06-10T18:45:22.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd05e7fc700 1 -- 172.21.15.104:0/1294305459 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05800bcc0 con 0x7fd06006cde0 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 -- 172.21.15.104:0/1294305459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 msgr2=0x7fd0600ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 --2- 172.21.15.104:0/1294305459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd058009e50 tx=0x7fd05800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 -- 172.21.15.104:0/1294305459 shutdown_connections 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 --2- 172.21.15.104:0/1294305459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd0600ce0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 --2- 172.21.15.104:0/1294305459 >> v2:172.21.15.114:3300/0 conn(0x7fd06015c580 0x7fd0600b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.710+0000 7fd065ef6700 1 -- 172.21.15.104:0/1294305459 >> 172.21.15.104:0/1294305459 conn(0x7fd06006b8a0 msgr2=0x7fd0600cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.711+0000 7fd065ef6700 1 -- 172.21.15.104:0/1294305459 shutdown_connections 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.711+0000 7fd065ef6700 1 -- 172.21.15.104:0/1294305459 wait complete. 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.712+0000 7fd065ef6700 1 Processor -- start 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.712+0000 7fd065ef6700 1 -- start start 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd060158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 0x7fd0600cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0601f2a50 con 0x7fd06006e6e0 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0601f1bd0 con 0x7fd06015c580 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd065ef6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0601f2df0 con 0x7fd06006cde0 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd05effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd05effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56744/0 (socket says 172.21.15.104:56744) 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.713+0000 7fd05effd700 1 -- 172.21.15.104:0/3411189217 learned_addr learned my addr 172.21.15.104:0/3411189217 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05ffff700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 0x7fd0600cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05f7fe700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd060158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 -- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 msgr2=0x7fd060158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd060158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 -- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 msgr2=0x7fd0600cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 0x7fd0600cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 -- 172.21.15.104:0/3411189217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058009b30 con 0x7fd06006e6e0 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05f7fe700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd060158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05ffff700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 0x7fd0600cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:22.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd05effd700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c4f0 tx=0x7fd05000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050005cb0 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0601f1e40 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050005e10 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500072d0 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.714+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0601f2400 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.715+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0601f2c00 con 0x7fd06006e6e0 2023-06-10T18:45:22.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.715+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd050007430 con 0x7fd06006e6e0 2023-06-10T18:45:22.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.716+0000 7fd047fff700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 0x7fd04c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:22.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.716+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd050088b40 con 0x7fd06006e6e0 2023-06-10T18:45:22.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.716+0000 7fd05f7fe700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 0x7fd04c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:22.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.716+0000 7fd05f7fe700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 0x7fd04c06d1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd054009aa0 tx=0x7fd054008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:22.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.720+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd0601f2c00 con 0x7fd06006e6e0 2023-06-10T18:45:22.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.908+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd060074050 con 0x7fd06006e6e0 2023-06-10T18:45:22.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.908+0000 7fd047fff700 1 -- 172.21.15.104:0/3411189217 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd050053980 con 0x7fd06006e6e0 2023-06-10T18:45:22.913 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:22.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 msgr2=0x7fd04c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 0x7fd04c06d1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd054009aa0 tx=0x7fd054008040 comp rx=0 tx=0).stop 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 msgr2=0x7fd0600b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c4f0 tx=0x7fd05000c8b0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 shutdown_connections 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ad10 0x7fd04c06d1c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06015c580 0x7fd0600cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06006cde0 0x7fd060158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 --2- 172.21.15.104:0/3411189217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06006e6e0 0x7fd0600b0280 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:22.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 >> 172.21.15.104:0/3411189217 conn(0x7fd06006b8a0 msgr2=0x7fd060159dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:22.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 shutdown_connections 2023-06-10T18:45:22.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:22.910+0000 7fd065ef6700 1 -- 172.21.15.104:0/3411189217 wait complete. 2023-06-10T18:45:22.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:23.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.044+0000 7fbd01d46700 1 Processor -- start 2023-06-10T18:45:23.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.044+0000 7fbd01d46700 1 -- start start 2023-06-10T18:45:23.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbd01d46700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbd01d46700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbcfc06cde0 0x7fbcfc0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbd01d46700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc06e250 con 0x7fbcfc15c580 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbd01d46700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcfc15c950 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbd01d46700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcfc0b1e80 con 0x7fbcfc06cde0 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56752/0 (socket says 172.21.15.104:56752) 2023-06-10T18:45:23.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 -- 172.21.15.104:0/1694317813 learned_addr learned my addr 172.21.15.104:0/1694317813 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfbfff700 1 --2- 172.21.15.104:0/1694317813 >> v2:172.21.15.114:3300/0 conn(0x7fbcfc06cde0 0x7fbcfc0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 -- 172.21.15.104:0/1694317813 >> v2:172.21.15.114:3300/0 conn(0x7fbcfc06cde0 msgr2=0x7fbcfc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 --2- 172.21.15.104:0/1694317813 >> v2:172.21.15.114:3300/0 conn(0x7fbcfc06cde0 0x7fbcfc0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 -- 172.21.15.104:0/1694317813 >> v1:172.21.15.114:6789/0 conn(0x7fbcfc15c580 legacy=0x7fbcfc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 -- 172.21.15.104:0/1694317813 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc06d920 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.045+0000 7fbcfaffd700 1 --2- 172.21.15.104:0/1694317813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0cbc60 tx=0x7fbcec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63c914c8bdedd52 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbcfa7fc700 1 -- 172.21.15.104:0/1694317813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00b870 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbcfa7fc700 1 -- 172.21.15.104:0/1694317813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec00b9d0 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbcfa7fc700 1 -- 172.21.15.104:0/1694317813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00bd60 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 -- 172.21.15.104:0/1694317813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06e6e0 msgr2=0x7fbcfc0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1694317813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0cbc60 tx=0x7fbcec009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 -- 172.21.15.104:0/1694317813 shutdown_connections 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1694317813 >> v2:172.21.15.114:3300/0 conn(0x7fbcfc06cde0 0x7fbcfc0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1694317813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b1780 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 -- 172.21.15.104:0/1694317813 >> 172.21.15.104:0/1694317813 conn(0x7fbcfc06b8a0 msgr2=0x7fbcfc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 -- 172.21.15.104:0/1694317813 shutdown_connections 2023-06-10T18:45:23.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.046+0000 7fbd01d46700 1 -- 172.21.15.104:0/1694317813 wait complete. 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.047+0000 7fbd01d46700 1 Processor -- start 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 -- start start 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcfc15c580 0x7fbcfc1f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0b4da0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0b4120 con 0x7fbcfc15c580 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbd01d46700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0b4400 con 0x7fbcfc06e6e0 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbcfb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbcfb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56756/0 (socket says 172.21.15.104:56756) 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.048+0000 7fbcfb7fe700 1 -- 172.21.15.104:0/1510565355 learned_addr learned my addr 172.21.15.104:0/1510565355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfbfff700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcfc15c580 0x7fbcfc1f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfaffd700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 -- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcfc06e6e0 msgr2=0x7fbcfc0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 -- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcfc15c580 msgr2=0x7fbcfc1f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcfc15c580 0x7fbcfc1f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 -- 172.21.15.104:0/1510565355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec009550 con 0x7fbcfc06cde0 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcfb7fe700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c4d0 tx=0x7fbce400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40052d0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce4005430 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc0b46e0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40038f0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.049+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc0b2960 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.050+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc0b2af0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.050+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbce4003a50 con 0x7fbcfc06cde0 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.051+0000 7fbcf3fff700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 0x7fbcdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.051+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbce4088c70 con 0x7fbcfc06cde0 2023-06-10T18:45:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.051+0000 7fbcfaffd700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 0x7fbcdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.052+0000 7fbcfaffd700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 0x7fbcdc06d190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc06eb20 tx=0x7fbcec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.055+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbce40539e0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.239+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbcfc06dee0 con 0x7fbcfc06cde0 2023-06-10T18:45:23.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.239+0000 7fbcf3fff700 1 -- 172.21.15.104:0/1510565355 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbce4017020 con 0x7fbcfc06cde0 2023-06-10T18:45:23.244 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:23.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 msgr2=0x7fbcdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 0x7fbcdc06d190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc06eb20 tx=0x7fbcec008030 comp rx=0 tx=0).stop 2023-06-10T18:45:23.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 msgr2=0x7fbcfc1f1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c4d0 tx=0x7fbce400c890 comp rx=0 tx=0).stop 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 shutdown_connections 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbcdc06ace0 0x7fbcdc06d190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcfc15c580 0x7fbcfc1f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbcfc06cde0 0x7fbcfc1f1920 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 --2- 172.21.15.104:0/1510565355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcfc06e6e0 0x7fbcfc0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.241+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 >> 172.21.15.104:0/1510565355 conn(0x7fbcfc06b8a0 msgr2=0x7fbcfc0cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.242+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 shutdown_connections 2023-06-10T18:45:23.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.242+0000 7fbd01d46700 1 -- 172.21.15.104:0/1510565355 wait complete. 2023-06-10T18:45:23.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:23.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.376+0000 7fd39610a700 1 Processor -- start 2023-06-10T18:45:23.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.376+0000 7fd39610a700 1 -- start start 2023-06-10T18:45:23.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd39610a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd390069e70 0x7fd39006a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd39610a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39015c8d0 con 0x7fd39015c500 2023-06-10T18:45:23.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd39610a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39006abd0 con 0x7fd39006dcd0 2023-06-10T18:45:23.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd39610a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3900aef20 con 0x7fd390069e70 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd390069e70 0x7fd39006a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd39015c500 0x7fd3900cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54592/0 (socket says 172.21.15.104:54592) 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38f7fe700 1 -- 172.21.15.104:0/1067985403 learned_addr learned my addr 172.21.15.104:0/1067985403 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38effd700 1 -- 172.21.15.104:0/1067985403 >> v1:172.21.15.114:6789/0 conn(0x7fd39006dcd0 legacy=0x7fd39006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38effd700 1 -- 172.21.15.104:0/1067985403 >> v1:172.21.15.104:6789/0 conn(0x7fd39015c500 legacy=0x7fd3900cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.377+0000 7fd38effd700 1 -- 172.21.15.104:0/1067985403 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39006a8f0 con 0x7fd390069e70 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd38effd700 1 --2- 172.21.15.104:0/1067985403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd39006a310 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd380009870 tx=0x7fd380009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809223e3f29e19af server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd38e7fc700 1 -- 172.21.15.104:0/1067985403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380004230 con 0x7fd390069e70 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd38e7fc700 1 -- 172.21.15.104:0/1067985403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd380004390 con 0x7fd390069e70 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd38e7fc700 1 -- 172.21.15.104:0/1067985403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3800036a0 con 0x7fd390069e70 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 -- 172.21.15.104:0/1067985403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 msgr2=0x7fd39006a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 --2- 172.21.15.104:0/1067985403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd39006a310 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd380009870 tx=0x7fd380009b80 comp rx=0 tx=0).stop 2023-06-10T18:45:23.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 -- 172.21.15.104:0/1067985403 shutdown_connections 2023-06-10T18:45:23.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 --2- 172.21.15.104:0/1067985403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd39006a310 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 -- 172.21.15.104:0/1067985403 >> 172.21.15.104:0/1067985403 conn(0x7fd39006b8a0 msgr2=0x7fd3900cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 -- 172.21.15.104:0/1067985403 shutdown_connections 2023-06-10T18:45:23.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.378+0000 7fd39610a700 1 -- 172.21.15.104:0/1067985403 wait complete. 2023-06-10T18:45:23.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 Processor -- start 2023-06-10T18:45:23.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 -- start start 2023-06-10T18:45:23.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd39006dcd0 0x7fd390155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd39015c500 0x7fd3900c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3900b3730 con 0x7fd390069e70 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3900c8b80 con 0x7fd39015c500 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.380+0000 7fd39610a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3900b2b20 con 0x7fd39006dcd0 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56770/0 (socket says 172.21.15.104:56770) 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 -- 172.21.15.104:0/2050702783 learned_addr learned my addr 172.21.15.104:0/2050702783 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38ffff700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd39015c500 0x7fd3900c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38effd700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd39006dcd0 0x7fd390155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 -- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd39006dcd0 msgr2=0x7fd390155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd39006dcd0 0x7fd390155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 -- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd39015c500 msgr2=0x7fd3900c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd39015c500 0x7fd3900c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 -- 172.21.15.104:0/2050702783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800094a0 con 0x7fd390069e70 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd38f7fe700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c700 tx=0x7fd37800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3780074e0 con 0x7fd390069e70 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd378007640 con 0x7fd390069e70 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378005a30 con 0x7fd390069e70 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3900b2e00 con 0x7fd390069e70 2023-06-10T18:45:23.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.381+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3900b33c0 con 0x7fd390069e70 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.382+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd390074050 con 0x7fd390069e70 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.383+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd378005b90 con 0x7fd390069e70 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.383+0000 7fd387fff700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 0x7fd37006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.383+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd378088a60 con 0x7fd390069e70 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.383+0000 7fd38effd700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 0x7fd37006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.384+0000 7fd38effd700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 0x7fd37006d1c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd3800097c0 tx=0x7fd380008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.388+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd378053890 con 0x7fd390069e70 2023-06-10T18:45:23.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.574+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd39006a550 con 0x7fd390069e70 2023-06-10T18:45:23.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.574+0000 7fd387fff700 1 -- 172.21.15.104:0/2050702783 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd3780170a0 con 0x7fd390069e70 2023-06-10T18:45:23.579 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:23.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 msgr2=0x7fd37006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 0x7fd37006d1c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd3800097c0 tx=0x7fd380008030 comp rx=0 tx=0).stop 2023-06-10T18:45:23.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 msgr2=0x7fd3900b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c700 tx=0x7fd37800ca10 comp rx=0 tx=0).stop 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 shutdown_connections 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd37006ad10 0x7fd37006d1c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd39015c500 0x7fd3900c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd390069e70 0x7fd3900b2440 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 --2- 172.21.15.104:0/2050702783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd39006dcd0 0x7fd390155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 >> 172.21.15.104:0/2050702783 conn(0x7fd39006b8a0 msgr2=0x7fd3900cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 shutdown_connections 2023-06-10T18:45:23.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.576+0000 7fd39610a700 1 -- 172.21.15.104:0/2050702783 wait complete. 2023-06-10T18:45:23.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:23.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.709+0000 7f4be6027700 1 Processor -- start 2023-06-10T18:45:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.709+0000 7f4be6027700 1 -- start start 2023-06-10T18:45:23.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4be6027700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4be00cb300 0x7f4be0163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4be6027700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00c9e60 con 0x7f4be00cd850 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4be6027700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00aee70 con 0x7f4be00aeaa0 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4be6027700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be00c9bc0 con 0x7f4be00cb300 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bdeffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4be00cd850 0x7f4be006cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54606/0 (socket says 172.21.15.104:54606) 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bdeffd700 1 -- 172.21.15.104:0/3791745399 learned_addr learned my addr 172.21.15.104:0/3791745399 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:23.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bdffff700 1 --2- 172.21.15.104:0/3791745399 >> v2:172.21.15.114:3300/0 conn(0x7f4be00cb300 0x7f4be0163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278068598 0 0) 0x7f4be00c9e60 con 0x7f4be00cd850 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc80035b0 con 0x7f4be00cd850 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582938660 0 0) 0x7f4bc80035b0 con 0x7f4be00cd850 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 >> v2:172.21.15.114:3300/0 conn(0x7f4be00cb300 msgr2=0x7f4be0163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 --2- 172.21.15.104:0/3791745399 >> v2:172.21.15.114:3300/0 conn(0x7f4be00cb300 0x7f4be0163c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 >> v1:172.21.15.114:6789/0 conn(0x7f4be00aeaa0 legacy=0x7f4be00cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be006e540 con 0x7f4be00cd850 2023-06-10T18:45:23.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4bd0001670 con 0x7f4be00cd850 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 >> v1:172.21.15.104:6789/0 conn(0x7f4be00cd850 legacy=0x7f4be006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.710+0000 7f4bde7fc700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4bc8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc8004850 0x7f4bc8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bc8005200 0x7f4bc800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c9bc0 con 0x7f4bc8003a00 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc800c080 con 0x7f4bc8004850 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc800e450 con 0x7f4bc8005200 2023-06-10T18:45:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4bc8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdeffd700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc8004850 0x7f4bc8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdffff700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bc8005200 0x7f4bc800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bc8005200 msgr2=0x7f4bc800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bc8005200 0x7f4bc800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc8004850 msgr2=0x7f4bc8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc8004850 0x7f4bc8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3791745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80035b0 con 0x7f4bc8003a00 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdffff700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bc8005200 0x7f4bc800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdffff700 1 -- 172.21.15.104:0/3791745399 reap_dead start 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4bc8003e10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4001820 tx=0x7f4bc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22532a527ab319db server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40012e0 con 0x7f4bc8003a00 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40052d0 con 0x7f4bc8003a00 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.711+0000 7f4bde7fc700 1 -- 172.21.15.104:0/3791745399 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40036d0 con 0x7f4bc8003a00 2023-06-10T18:45:23.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 -- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 msgr2=0x7f4bc8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4bc8003e10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4001820 tx=0x7f4bc8005880 comp rx=0 tx=0).stop 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 -- 172.21.15.104:0/3791745399 shutdown_connections 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 --2- 172.21.15.104:0/3791745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4bc8003e10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 -- 172.21.15.104:0/3791745399 >> 172.21.15.104:0/3791745399 conn(0x7f4be006b8a0 msgr2=0x7f4be00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 -- 172.21.15.104:0/3791745399 shutdown_connections 2023-06-10T18:45:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.712+0000 7f4be6027700 1 -- 172.21.15.104:0/3791745399 wait complete. 2023-06-10T18:45:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.713+0000 7f4be6027700 1 Processor -- start 2023-06-10T18:45:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.713+0000 7f4be6027700 1 -- start start 2023-06-10T18:45:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 0x7f4be006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00b44f0 0x7f4be00b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be006dd80 con 0x7f4bc8003a00 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be01ed260 con 0x7f4be00b44f0 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4be6027700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00b5260 con 0x7f4be006d3d0 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56788/0 (socket says 172.21.15.104:56788) 2023-06-10T18:45:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3443903565 learned_addr learned my addr 172.21.15.104:0/3443903565 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdeffd700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 0x7f4be006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdffff700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00b44f0 0x7f4be00b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 msgr2=0x7f4be006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 0x7f4be006d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00b44f0 msgr2=0x7f4be00b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00b44f0 0x7f4be00b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 -- 172.21.15.104:0/3443903565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be0159510 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdeffd700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 0x7f4be006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bdf7fe700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4005040 tx=0x7f4bd4001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.714+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4001280 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.715+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40050d0 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.715+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be01597f0 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.715+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd401d420 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.715+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be0159db0 con 0x7f4bc8003a00 2023-06-10T18:45:23.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.715+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be015a120 con 0x7f4bc8003a00 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.716+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4bd401d580 con 0x7f4bc8003a00 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.716+0000 7f4bc7fff700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 0x7f4bcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.716+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4bd4089dc0 con 0x7f4bc8003a00 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.717+0000 7f4bdeffd700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 0x7f4bcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.717+0000 7f4bdeffd700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 0x7f4bcc06d190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4be00b4020 tx=0x7f4bd0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:23.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.721+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4bd4054bd0 con 0x7f4bc8003a00 2023-06-10T18:45:23.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.923+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4be0159f40 con 0x7f4bc8003a00 2023-06-10T18:45:23.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.923+0000 7f4bc7fff700 1 -- 172.21.15.104:0/3443903565 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4bd4021080 con 0x7f4bc8003a00 2023-06-10T18:45:23.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 msgr2=0x7f4bcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 0x7f4bcc06d190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4be00b4020 tx=0x7f4bd0009040 comp rx=0 tx=0).stop 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 msgr2=0x7f4be01ecd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4005040 tx=0x7f4bd4001170 comp rx=0 tx=0).stop 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 shutdown_connections 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4bcc06ace0 0x7f4bcc06d190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4be00b44f0 0x7f4be00b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bc8003a00 0x7f4be01ecd20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 --2- 172.21.15.104:0/3443903565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4be006d3d0 0x7f4be006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 >> 172.21.15.104:0/3443903565 conn(0x7f4be006b8a0 msgr2=0x7f4be00b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 shutdown_connections 2023-06-10T18:45:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:23.925+0000 7f4be6027700 1 -- 172.21.15.104:0/3443903565 wait complete. 2023-06-10T18:45:23.931 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:23.941 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:45:23.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:23.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-06-10T18:45:23.985 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:23.986 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:45:23.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:23.987 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:45:24.044 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:45:24.045 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:24.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 Processor -- start 2023-06-10T18:45:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 -- start start 2023-06-10T18:45:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f914806e0e0 0x7f914806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914806d290 con 0x7f914806e4b0 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9148071640 con 0x7f914806e0e0 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.160+0000 7f914eb63700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91480aef00 con 0x7f9148069e70 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f9147fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f914806e4b0 0x7f91480c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54624/0 (socket says 172.21.15.104:54624) 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f9147fff700 1 -- 172.21.15.104:0/2950715385 learned_addr learned my addr 172.21.15.104:0/2950715385 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f914c8ff700 1 --2- 172.21.15.104:0/2950715385 >> v2:172.21.15.114:3300/0 conn(0x7f914806e0e0 0x7f914806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f914d100700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470043258 0 0) 0x7f914806d290 con 0x7f914806e4b0 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91380035b0 con 0x7f914806e4b0 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385282118 0 0) 0x7f91380035b0 con 0x7f914806e4b0 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 msgr2=0x7f91480ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 >> v2:172.21.15.114:3300/0 conn(0x7f914806e0e0 msgr2=0x7f914806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 --2- 172.21.15.104:0/2950715385 >> v2:172.21.15.114:3300/0 conn(0x7f914806e0e0 0x7f914806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91480ca2d0 con 0x7f914806e4b0 2023-06-10T18:45:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9134001670 con 0x7f914806e4b0 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 >> v1:172.21.15.104:6789/0 conn(0x7f914806e4b0 legacy=0x7f91480c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f9138003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9138003f80 0x7f9138004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9138004d90 0x7f9138005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9148071640 con 0x7f9138004d90 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480aef00 con 0x7f9138003f80 2023-06-10T18:45:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913800baf0 con 0x7f9148069e70 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f914d100700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9138004d90 0x7f9138005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f9147fff700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9138003f80 0x7f9138004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.161+0000 7f914c8ff700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f9138003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 -- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 msgr2=0x7f9138003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f9138003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 -- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9138003f80 msgr2=0x7f9138004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9138003f80 0x7f9138004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 -- 172.21.15.104:0/2950715385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91380035b0 con 0x7f9138004d90 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9138004d90 0x7f9138005200 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9140009f90 tx=0x7f9138005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf006b53ad4a794 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000ec30 con 0x7f9138004d90 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914000f040 con 0x7f9138004d90 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f91477fe700 1 -- 172.21.15.104:0/2950715385 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140014480 con 0x7f9138004d90 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 -- 172.21.15.104:0/2950715385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9138004d90 msgr2=0x7f9138005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 --2- 172.21.15.104:0/2950715385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9138004d90 0x7f9138005200 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9140009f90 tx=0x7f9138005880 comp rx=0 tx=0).stop 2023-06-10T18:45:24.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914d100700 1 -- 172.21.15.104:0/2950715385 reap_dead start 2023-06-10T18:45:24.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 -- 172.21.15.104:0/2950715385 shutdown_connections 2023-06-10T18:45:24.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 -- 172.21.15.104:0/2950715385 >> 172.21.15.104:0/2950715385 conn(0x7f914806b8a0 msgr2=0x7f91480cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:24.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 -- 172.21.15.104:0/2950715385 shutdown_connections 2023-06-10T18:45:24.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.162+0000 7f914eb63700 1 -- 172.21.15.104:0/2950715385 wait complete. 2023-06-10T18:45:24.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.164+0000 7f914eb63700 1 Processor -- start 2023-06-10T18:45:24.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.164+0000 7f914eb63700 1 -- start start 2023-06-10T18:45:24.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91480caac0 0x7f91481488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9148148e10 0x7f91480b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480caf30 con 0x7f91481f1800 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91481f53c0 con 0x7f9148148e10 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91481f5610 con 0x7f91480caac0 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56808/0 (socket says 172.21.15.104:56808) 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 -- 172.21.15.104:0/4079208028 learned_addr learned my addr 172.21.15.104:0/4079208028 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914d100700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9148148e10 0x7f91480b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f9147fff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91480caac0 0x7f91481488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 -- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91480caac0 msgr2=0x7f91481488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91480caac0 0x7f91481488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 -- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9148148e10 msgr2=0x7f91480b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9148148e10 0x7f91480b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 -- 172.21.15.104:0/4079208028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91481f58f0 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914c8ff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f913c011aa0 tx=0x7f913c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c005a20 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c005b80 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91481f5bd0 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c00f8e0 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.165+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91481f1bd0 con 0x7f91481f1800 2023-06-10T18:45:24.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.166+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9148074050 con 0x7f91481f1800 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.167+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f913c00fa40 con 0x7f91481f1800 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.167+0000 7f91457fa700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.167+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f913c089a70 con 0x7f91481f1800 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.167+0000 7f9147fff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.168+0000 7f9147fff700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9134001cf0 tx=0x7f9134007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.172+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f913c054880 con 0x7f91481f1800 2023-06-10T18:45:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.400+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91480ccc00 con 0x7f91481f1800 2023-06-10T18:45:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.400+0000 7f91457fa700 1 -- 172.21.15.104:0/4079208028 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f913c022090 con 0x7f91481f1800 2023-06-10T18:45:24.405 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:24.405 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 msgr2=0x7f912c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9134001cf0 tx=0x7f9134007980 comp rx=0 tx=0).stop 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 msgr2=0x7f91480ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f913c011aa0 tx=0x7f913c011db0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 shutdown_connections 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9148148e10 0x7f91480b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91481f1800 0x7f91480ca580 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 --2- 172.21.15.104:0/4079208028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91480caac0 0x7f91481488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 >> 172.21.15.104:0/4079208028 conn(0x7f914806b8a0 msgr2=0x7f914806a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 shutdown_connections 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.402+0000 7f914eb63700 1 -- 172.21.15.104:0/4079208028 wait complete. 2023-06-10T18:45:24.408 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:45:24.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.533+0000 7f756fb93700 1 Processor -- start 2023-06-10T18:45:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 -- start start 2023-06-10T18:45:24.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f756806e0e0 0x7f756806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f75680ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756806d290 con 0x7f756806e4b0 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7568071640 con 0x7f756806e0e0 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.534+0000 7f756fb93700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75680aef00 con 0x7f7568069e70 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f756806e0e0 0x7f756806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f756806e0e0 0x7f756806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56816/0 (socket says 172.21.15.104:56816) 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 -- 172.21.15.104:0/1763084576 learned_addr learned my addr 172.21.15.104:0/1763084576 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:24.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756e130700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f75680ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 -- 172.21.15.104:0/1763084576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 msgr2=0x7f75680ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f75680ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 -- 172.21.15.104:0/1763084576 >> v1:172.21.15.114:6789/0 conn(0x7f756806e4b0 legacy=0x7f75680c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 -- 172.21.15.104:0/1763084576 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75680ca2d0 con 0x7f756806e0e0 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756e130700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f75680ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756d92f700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e0e0 0x7f756806cd50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7558006fd0 tx=0x7f7558009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07b89953b1fc649 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756c92d700 1 -- 172.21.15.104:0/1763084576 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558005e30 con 0x7f756806e0e0 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756c92d700 1 -- 172.21.15.104:0/1763084576 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558004030 con 0x7f756806e0e0 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.535+0000 7f756c92d700 1 -- 172.21.15.104:0/1763084576 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580043e0 con 0x7f756806e0e0 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 -- 172.21.15.104:0/1763084576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e0e0 msgr2=0x7f756806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e0e0 0x7f756806cd50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7558006fd0 tx=0x7f7558009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 -- 172.21.15.104:0/1763084576 shutdown_connections 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e0e0 0x7f756806cd50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 --2- 172.21.15.104:0/1763084576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f75680ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 -- 172.21.15.104:0/1763084576 >> 172.21.15.104:0/1763084576 conn(0x7f756806b8a0 msgr2=0x7f75680cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:24.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 -- 172.21.15.104:0/1763084576 shutdown_connections 2023-06-10T18:45:24.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.536+0000 7f756fb93700 1 -- 172.21.15.104:0/1763084576 wait complete. 2023-06-10T18:45:24.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.537+0000 7f756fb93700 1 Processor -- start 2023-06-10T18:45:24.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.537+0000 7f756fb93700 1 -- start start 2023-06-10T18:45:24.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f756815a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 0x7f7568159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75680b4c40 con 0x7f756806e4b0 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75680b5450 con 0x7f756806e0e0 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756815a0f0 con 0x7f7568069e70 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56826/0 (socket says 172.21.15.104:56826) 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 -- 172.21.15.104:0/3327569573 learned_addr learned my addr 172.21.15.104:0/3327569573 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756d12e700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 0x7f7568159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756d92f700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f756815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 -- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 msgr2=0x7f756815a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f756815a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 -- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 msgr2=0x7f7568159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 0x7f7568159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 -- 172.21.15.104:0/3327569573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009500 con 0x7f756806e4b0 2023-06-10T18:45:24.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756d12e700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 0x7f7568159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756e130700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7560007df0 tx=0x7f756000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756000ddd0 con 0x7f756806e4b0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756000f040 con 0x7f756806e4b0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75680b4e90 con 0x7f756806e4b0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.538+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560003c60 con 0x7f756806e4b0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.539+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75680b34d0 con 0x7f756806e4b0 2023-06-10T18:45:24.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.539+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7568074050 con 0x7f756806e4b0 2023-06-10T18:45:24.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.540+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7560004030 con 0x7f756806e4b0 2023-06-10T18:45:24.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.541+0000 7f755e7fc700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.541+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7560089e90 con 0x7f756806e4b0 2023-06-10T18:45:24.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.541+0000 7f756d92f700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.541+0000 7f756d92f700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75680cb620 tx=0x7f7558008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.544+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7560054c10 con 0x7f756806e4b0 2023-06-10T18:45:24.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.733+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75680707a0 con 0x7f756806e4b0 2023-06-10T18:45:24.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.735+0000 7f755e7fc700 1 -- 172.21.15.104:0/3327569573 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f756001a090 con 0x7f756806e4b0 2023-06-10T18:45:24.740 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:24.740 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:24.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 msgr2=0x7f754c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75680cb620 tx=0x7f7558008070 comp rx=0 tx=0).stop 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 msgr2=0x7f7568099340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7560007df0 tx=0x7f756000d3b0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 shutdown_connections 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f756806e4b0 0x7f7568099340 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7568069e70 0x7f756815a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 --2- 172.21.15.104:0/3327569573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f756806e0e0 0x7f7568159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 >> 172.21.15.104:0/3327569573 conn(0x7f756806b8a0 msgr2=0x7f75680acbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:24.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 shutdown_connections 2023-06-10T18:45:24.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.737+0000 7f756fb93700 1 -- 172.21.15.104:0/3327569573 wait complete. 2023-06-10T18:45:24.753 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:44:06.622378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:44:07.927610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-10T18:44:10.992951+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-10T18:45:24.754 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.872+0000 7f9eb033b700 1 Processor -- start 2023-06-10T18:45:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.872+0000 7f9eb033b700 1 -- start start 2023-06-10T18:45:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eb033b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9ea8157b30 0x7f9ea80ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eb033b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea80ad3c0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eb033b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea8157f00 con 0x7f9ea80ce880 2023-06-10T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eb033b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea80ced20 con 0x7f9ea8157b30 2023-06-10T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eae0d7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9ea8157b30 0x7f9ea80ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eae8d8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9ea80d0a00 0x7f9ea80b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54662/0 (socket says 172.21.15.104:54662) 2023-06-10T18:45:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.873+0000 7f9eae8d8700 1 -- 172.21.15.104:0/3723438202 learned_addr learned my addr 172.21.15.104:0/3723438202 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eae0d7700 1 -- 172.21.15.104:0/3723438202 >> v1:172.21.15.114:6789/0 conn(0x7f9ea80ce880 legacy=0x7f9ea80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eae0d7700 1 -- 172.21.15.104:0/3723438202 >> v1:172.21.15.104:6789/0 conn(0x7f9ea80d0a00 legacy=0x7f9ea80b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9ead0d5700 1 -- 172.21.15.104:0/3723438202 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555372887 0 0) 0x7f9ea80ad3c0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eae0d7700 1 -- 172.21.15.104:0/3723438202 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea80b0170 con 0x7f9ea8157b30 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eae0d7700 1 --2- 172.21.15.104:0/3723438202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea8157b30 0x7f9ea80ce340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009840 tx=0x7f9e98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49d91ffb745aefe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9ead0d5700 1 -- 172.21.15.104:0/3723438202 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98004230 con 0x7f9ea8157b30 2023-06-10T18:45:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9ead0d5700 1 -- 172.21.15.104:0/3723438202 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e98004390 con 0x7f9ea8157b30 2023-06-10T18:45:24.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9ead0d5700 1 -- 172.21.15.104:0/3723438202 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e980036a0 con 0x7f9ea8157b30 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eb033b700 1 -- 172.21.15.104:0/3723438202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea8157b30 msgr2=0x7f9ea80ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.874+0000 7f9eb033b700 1 --2- 172.21.15.104:0/3723438202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea8157b30 0x7f9ea80ce340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009840 tx=0x7f9e98009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.875+0000 7f9eb033b700 1 -- 172.21.15.104:0/3723438202 shutdown_connections 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.875+0000 7f9eb033b700 1 --2- 172.21.15.104:0/3723438202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea8157b30 0x7f9ea80ce340 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.875+0000 7f9eb033b700 1 -- 172.21.15.104:0/3723438202 >> 172.21.15.104:0/3723438202 conn(0x7f9ea8089fd0 msgr2=0x7f9ea808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:24.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.875+0000 7f9eb033b700 1 -- 172.21.15.104:0/3723438202 shutdown_connections 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.875+0000 7f9eb033b700 1 -- 172.21.15.104:0/3723438202 wait complete. 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.876+0000 7f9eb033b700 1 Processor -- start 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.876+0000 7f9eb033b700 1 -- start start 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea80ce880 0x7f9ea8158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ea8157b30 0x7f9ea80b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea80cc2b0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea80cc550 con 0x7f9ea80ce880 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8158fe0 con 0x7f9ea8157b30 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56838/0 (socket says 172.21.15.104:56838) 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 -- 172.21.15.104:0/2197450874 learned_addr learned my addr 172.21.15.104:0/2197450874 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eae0d7700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea80ce880 0x7f9ea8158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eae8d8700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ea8157b30 0x7f9ea80b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 -- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ea8157b30 msgr2=0x7f9ea80b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ea8157b30 0x7f9ea80b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 -- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea80ce880 msgr2=0x7f9ea8158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea80ce880 0x7f9ea8158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 -- 172.21.15.104:0/2197450874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e980094a0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9ead8d6700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c500 tx=0x7f9ea400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011790 con 0x7f9ea80d0a00 2023-06-10T18:45:24.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea40118f0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea8159260 con 0x7f9ea80d0a00 2023-06-10T18:45:24.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011ca0 con 0x7f9ea80d0a00 2023-06-10T18:45:24.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.877+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea8159820 con 0x7f9ea80d0a00 2023-06-10T18:45:24.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.878+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea80cc700 con 0x7f9ea80d0a00 2023-06-10T18:45:24.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.879+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9ea401c460 con 0x7f9ea80d0a00 2023-06-10T18:45:24.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.879+0000 7f9e9effd700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 0x7f9e8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:24.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.879+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9ea4089450 con 0x7f9ea80d0a00 2023-06-10T18:45:24.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.879+0000 7f9eae0d7700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 0x7f9e8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:24.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.880+0000 7f9eae0d7700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 0x7f9e8c06d190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9e980090b0 tx=0x7f9e98008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:24.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:24.883+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9ea80cc700 con 0x7f9ea80d0a00 2023-06-10T18:45:25.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.071+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ea80cc6c0 con 0x7f9ea80d0a00 2023-06-10T18:45:25.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.071+0000 7f9e9effd700 1 -- 172.21.15.104:0/2197450874 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ea4054280 con 0x7f9ea80d0a00 2023-06-10T18:45:25.076 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 msgr2=0x7f9e8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 0x7f9e8c06d190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9e980090b0 tx=0x7f9e98008030 comp rx=0 tx=0).stop 2023-06-10T18:45:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 msgr2=0x7f9ea8155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c500 tx=0x7f9ea400c810 comp rx=0 tx=0).stop 2023-06-10T18:45:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 shutdown_connections 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e8c06ace0 0x7f9e8c06d190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea80ce880 0x7f9ea8158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ea8157b30 0x7f9ea80b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 --2- 172.21.15.104:0/2197450874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ea80d0a00 0x7f9ea8155810 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 >> 172.21.15.104:0/2197450874 conn(0x7f9ea8089fd0 msgr2=0x7f9ea808a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 shutdown_connections 2023-06-10T18:45:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.073+0000 7f9eb033b700 1 -- 172.21.15.104:0/2197450874 wait complete. 2023-06-10T18:45:25.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.207+0000 7f101baf4700 1 Processor -- start 2023-06-10T18:45:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.207+0000 7f101baf4700 1 -- start start 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101baf4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1014145c70 0x7f10141607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101baf4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101406b800 con 0x7f10140c3d60 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101baf4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1014069e70 con 0x7f101406b280 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101baf4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1014160cf0 con 0x7f1014145c70 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f1019890700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1014145c70 0x7f10141607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101a091700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f10140c3d60 0x7f1014070a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54672/0 (socket says 172.21.15.104:54672) 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101a091700 1 -- 172.21.15.104:0/3812820986 learned_addr learned my addr 172.21.15.104:0/3812820986 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f1019890700 1 -- 172.21.15.104:0/3812820986 >> v1:172.21.15.114:6789/0 conn(0x7f101406b280 legacy=0x7f101406a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101888e700 1 -- 172.21.15.104:0/3812820986 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839383745 0 0) 0x7f100c001490 con 0x7f10140c3d60 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f1019890700 1 -- 172.21.15.104:0/3812820986 >> v1:172.21.15.104:6789/0 conn(0x7f10140c3d60 legacy=0x7f1014070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f1019890700 1 -- 172.21.15.104:0/3812820986 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101406c9a0 con 0x7f1014145c70 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f1019890700 1 --2- 172.21.15.104:0/3812820986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1014145c70 0x7f10141607b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1010009840 tx=0x7f1010009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9abf55dca4724e1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.208+0000 7f101888e700 1 -- 172.21.15.104:0/3812820986 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010004230 con 0x7f1014145c70 2023-06-10T18:45:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101888e700 1 -- 172.21.15.104:0/3812820986 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010004390 con 0x7f1014145c70 2023-06-10T18:45:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101888e700 1 -- 172.21.15.104:0/3812820986 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10100036a0 con 0x7f1014145c70 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 -- 172.21.15.104:0/3812820986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1014145c70 msgr2=0x7f10141607b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 --2- 172.21.15.104:0/3812820986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1014145c70 0x7f10141607b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1010009840 tx=0x7f1010009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 -- 172.21.15.104:0/3812820986 shutdown_connections 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 --2- 172.21.15.104:0/3812820986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1014145c70 0x7f10141607b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 -- 172.21.15.104:0/3812820986 >> 172.21.15.104:0/3812820986 conn(0x7f10140ceec0 msgr2=0x7f10140cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 -- 172.21.15.104:0/3812820986 shutdown_connections 2023-06-10T18:45:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.209+0000 7f101baf4700 1 -- 172.21.15.104:0/3812820986 wait complete. 2023-06-10T18:45:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.211+0000 7f101baf4700 1 Processor -- start 2023-06-10T18:45:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.211+0000 7f101baf4700 1 -- start start 2023-06-10T18:45:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10140c3d60 0x7f10140ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1014145c70 0x7f101415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140ade40 con 0x7f101406b280 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140c6130 con 0x7f1014145c70 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140c6410 con 0x7f10140c3d60 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56850/0 (socket says 172.21.15.104:56850) 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 -- 172.21.15.104:0/3384633918 learned_addr learned my addr 172.21.15.104:0/3384633918 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101908f700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10140c3d60 0x7f10140ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101a091700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1014145c70 0x7f101415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 -- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10140c3d60 msgr2=0x7f10140ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10140c3d60 0x7f10140ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 -- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1014145c70 msgr2=0x7f101415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1014145c70 0x7f101415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 -- 172.21.15.104:0/3384633918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100094a0 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f1019890700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1014065cd0 tx=0x7f10100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010011630 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010011790 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010011b40 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10140c66f0 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.212+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10140adb50 con 0x7f101406b280 2023-06-10T18:45:25.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.213+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10140c6270 con 0x7f101406b280 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.214+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1010011ca0 con 0x7f101406b280 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.214+0000 7f100a7fc700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 0x7f0ff806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.214+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f101008b0c0 con 0x7f101406b280 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.214+0000 7f101908f700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 0x7f0ff806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.215+0000 7f101908f700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 0x7f0ff806d190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10140b50b0 tx=0x7f1004008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.218+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10140c6270 con 0x7f101406b280 2023-06-10T18:45:25.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.403+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10140c6c10 con 0x7f101406b280 2023-06-10T18:45:25.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.404+0000 7f100a7fc700 1 -- 172.21.15.104:0/3384633918 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10140c6c10 con 0x7f101406b280 2023-06-10T18:45:25.409 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:25.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 msgr2=0x7f0ff806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 0x7f0ff806d190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10140b50b0 tx=0x7f1004008040 comp rx=0 tx=0).stop 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 msgr2=0x7f10141f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1014065cd0 tx=0x7f10100047c0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 shutdown_connections 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ff806ace0 0x7f0ff806d190 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1014145c70 0x7f101415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f101406b280 0x7f10141f6440 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 --2- 172.21.15.104:0/3384633918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10140c3d60 0x7f10140ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 >> 172.21.15.104:0/3384633918 conn(0x7f10140ceec0 msgr2=0x7f10140ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 shutdown_connections 2023-06-10T18:45:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.406+0000 7f101baf4700 1 -- 172.21.15.104:0/3384633918 wait complete. 2023-06-10T18:45:25.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:25.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.541+0000 7f8994847700 1 Processor -- start 2023-06-10T18:45:25.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.541+0000 7f8994847700 1 -- start start 2023-06-10T18:45:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f8994847700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f899006cde0 0x7f899006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f8994847700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89900d0480 con 0x7f89900aec40 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f8994847700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899015dd00 con 0x7f89900ca850 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f8994847700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f899006d7a0 con 0x7f899006cde0 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898dd9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f899006cde0 0x7f899006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898e59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f89900aec40 0x7f8990069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54686/0 (socket says 172.21.15.104:54686) 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898e59c700 1 -- 172.21.15.104:0/3500279312 learned_addr learned my addr 172.21.15.104:0/3500279312 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898dd9b700 1 -- 172.21.15.104:0/3500279312 >> v1:172.21.15.114:6789/0 conn(0x7f89900ca850 legacy=0x7f89900cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898dd9b700 1 -- 172.21.15.104:0/3500279312 >> v1:172.21.15.104:6789/0 conn(0x7f89900aec40 legacy=0x7f8990069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898dd9b700 1 -- 172.21.15.104:0/3500279312 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899015a610 con 0x7f899006cde0 2023-06-10T18:45:25.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898dd9b700 1 --2- 172.21.15.104:0/3500279312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f899006d260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e25c9730d02b677 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898d59a700 1 -- 172.21.15.104:0/3500279312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984004230 con 0x7f899006cde0 2023-06-10T18:45:25.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898d59a700 1 -- 172.21.15.104:0/3500279312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984004390 con 0x7f899006cde0 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.542+0000 7f898d59a700 1 -- 172.21.15.104:0/3500279312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89840036a0 con 0x7f899006cde0 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 -- 172.21.15.104:0/3500279312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 msgr2=0x7f899006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 --2- 172.21.15.104:0/3500279312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f899006d260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 -- 172.21.15.104:0/3500279312 shutdown_connections 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 --2- 172.21.15.104:0/3500279312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f899006d260 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 -- 172.21.15.104:0/3500279312 >> 172.21.15.104:0/3500279312 conn(0x7f899006b8a0 msgr2=0x7f89900cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 -- 172.21.15.104:0/3500279312 shutdown_connections 2023-06-10T18:45:25.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.543+0000 7f8994847700 1 -- 172.21.15.104:0/3500279312 wait complete. 2023-06-10T18:45:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.544+0000 7f8994847700 1 Processor -- start 2023-06-10T18:45:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.544+0000 7f8994847700 1 -- start start 2023-06-10T18:45:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89900aec40 0x7f89900c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 0x7f8990155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89900c8e00 con 0x7f899006cde0 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89900c7700 con 0x7f89900aec40 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f8994847700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89900c79e0 con 0x7f89900ca850 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f898e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f898e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56864/0 (socket says 172.21.15.104:56864) 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.545+0000 7f898e59c700 1 -- 172.21.15.104:0/486800355 learned_addr learned my addr 172.21.15.104:0/486800355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898dd9b700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89900aec40 0x7f89900c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898ed9d700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 0x7f8990155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 -- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 msgr2=0x7f8990155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 0x7f8990155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 -- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89900aec40 msgr2=0x7f89900c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89900aec40 0x7f89900c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 -- 172.21.15.104:0/486800355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840094a0 con 0x7f899006cde0 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898ed9d700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 0x7f8990155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f898e59c700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f898000c4d0 tx=0x7f898000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800074e0 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980007640 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89900c7cc0 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980005a30 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.546+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89900c7fa0 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.547+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899015b250 con 0x7f899006cde0 2023-06-10T18:45:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.547+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8980005b90 con 0x7f899006cde0 2023-06-10T18:45:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.548+0000 7f897effd700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.548+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8980088a90 con 0x7f899006cde0 2023-06-10T18:45:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.548+0000 7f898dd9b700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.548+0000 7f898dd9b700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89840090b0 tx=0x7f8984008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.552+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8980053950 con 0x7f899006cde0 2023-06-10T18:45:25.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.736+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8990074050 con 0x7f899006cde0 2023-06-10T18:45:25.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.737+0000 7f897effd700 1 -- 172.21.15.104:0/486800355 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8980056f70 con 0x7f899006cde0 2023-06-10T18:45:25.741 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:25.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 msgr2=0x7f897006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89840090b0 tx=0x7f8984008030 comp rx=0 tx=0).stop 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 msgr2=0x7f89900c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f898000c4d0 tx=0x7f898000c890 comp rx=0 tx=0).stop 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 shutdown_connections 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89900ca850 0x7f8990155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f899006cde0 0x7f89900c7040 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 --2- 172.21.15.104:0/486800355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89900aec40 0x7f89900c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 >> 172.21.15.104:0/486800355 conn(0x7f899006b8a0 msgr2=0x7f89900cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 shutdown_connections 2023-06-10T18:45:25.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.739+0000 7f8994847700 1 -- 172.21.15.104:0/486800355 wait complete. 2023-06-10T18:45:25.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:25.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.876+0000 7ff32a50b700 1 Processor -- start 2023-06-10T18:45:25.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.876+0000 7ff32a50b700 1 -- start start 2023-06-10T18:45:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff32a50b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff32415c580 0x7ff3240b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff32a50b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 0x7ff3240ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff32a50b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32406e250 con 0x7ff32406e6e0 2023-06-10T18:45:25.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff32a50b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32415c950 con 0x7ff32415c580 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff32a50b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240b1e80 con 0x7ff32406cde0 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff328aa8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 0x7ff3240ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff3237fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff32406e6e0 0x7ff3240b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60550/0 (socket says 172.21.15.104:60550) 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff323fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff32415c580 0x7ff3240b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff3237fe700 1 -- 172.21.15.104:0/185434046 learned_addr learned my addr 172.21.15.104:0/185434046 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff322ffd700 1 -- 172.21.15.104:0/185434046 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736164691 0 0) 0x7ff32406e250 con 0x7ff32406e6e0 2023-06-10T18:45:25.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff322ffd700 1 -- 172.21.15.104:0/185434046 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30c0035b0 con 0x7ff32406e6e0 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff323fff700 1 -- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 msgr2=0x7ff3240ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff323fff700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 0x7ff3240ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff323fff700 1 -- 172.21.15.104:0/185434046 >> v1:172.21.15.114:6789/0 conn(0x7ff32406e6e0 legacy=0x7ff3240b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff323fff700 1 -- 172.21.15.104:0/185434046 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32406d920 con 0x7ff32415c580 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.877+0000 7ff328aa8700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 0x7ff3240ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff323fff700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32415c580 0x7ff3240b1070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff314009820 tx=0x7ff314009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6466812179b7276a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff322ffd700 1 -- 172.21.15.104:0/185434046 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314005e30 con 0x7ff32415c580 2023-06-10T18:45:25.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff322ffd700 1 -- 172.21.15.104:0/185434046 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314006030 con 0x7ff32415c580 2023-06-10T18:45:25.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff322ffd700 1 -- 172.21.15.104:0/185434046 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3140063c0 con 0x7ff32415c580 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 -- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32415c580 msgr2=0x7ff3240b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32415c580 0x7ff3240b1070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff314009820 tx=0x7ff314009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 -- 172.21.15.104:0/185434046 shutdown_connections 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32406cde0 0x7ff3240ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 --2- 172.21.15.104:0/185434046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32415c580 0x7ff3240b1070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 -- 172.21.15.104:0/185434046 >> 172.21.15.104:0/185434046 conn(0x7ff32406b8a0 msgr2=0x7ff3240cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:25.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 -- 172.21.15.104:0/185434046 shutdown_connections 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.878+0000 7ff32a50b700 1 -- 172.21.15.104:0/185434046 wait complete. 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.879+0000 7ff32a50b700 1 Processor -- start 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 -- start start 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32406cde0 0x7ff3240b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32415c580 0x7ff3241f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241f2ea0 con 0x7ff32406e6e0 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240adaa0 con 0x7ff32406cde0 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff32a50b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241f2bc0 con 0x7ff32415c580 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.880+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56882/0 (socket says 172.21.15.104:56882) 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 -- 172.21.15.104:0/3355546224 learned_addr learned my addr 172.21.15.104:0/3355546224 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff328aa8700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32415c580 0x7ff3241f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff323fff700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32406cde0 0x7ff3240b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 -- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32415c580 msgr2=0x7ff3241f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32415c580 0x7ff3241f2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 -- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32406cde0 msgr2=0x7ff3240b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32406cde0 0x7ff3240b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 -- 172.21.15.104:0/3355546224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c009a70 con 0x7ff32406e6e0 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff3237fe700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c700 tx=0x7ff31800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff318005d70 con 0x7ff32406e6e0 2023-06-10T18:45:25.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3180070d0 con 0x7ff32406e6e0 2023-06-10T18:45:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff314009430 con 0x7ff32406e6e0 2023-06-10T18:45:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff318007460 con 0x7ff32406e6e0 2023-06-10T18:45:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.881+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3240b2d90 con 0x7ff32406e6e0 2023-06-10T18:45:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.882+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3240d0500 con 0x7ff32406e6e0 2023-06-10T18:45:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.882+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff3180075c0 con 0x7ff32406e6e0 2023-06-10T18:45:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.883+0000 7ff320ff9700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 0x7ff31006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.883+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff318088c20 con 0x7ff32406e6e0 2023-06-10T18:45:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.883+0000 7ff323fff700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 0x7ff31006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.884+0000 7ff323fff700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 0x7ff31006d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff314009b70 tx=0x7ff314008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:25.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:25.887+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3180539a0 con 0x7ff32406e6e0 2023-06-10T18:45:26.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.072+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff32406dc00 con 0x7ff32406e6e0 2023-06-10T18:45:26.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.073+0000 7ff320ff9700 1 -- 172.21.15.104:0/3355546224 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff318056fc0 con 0x7ff32406e6e0 2023-06-10T18:45:26.078 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:26.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 msgr2=0x7ff31006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 0x7ff31006d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff314009b70 tx=0x7ff314008070 comp rx=0 tx=0).stop 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 msgr2=0x7ff3240ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c700 tx=0x7ff31800ca10 comp rx=0 tx=0).stop 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 shutdown_connections 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff31006ace0 0x7ff31006d190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff32406cde0 0x7ff3240b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff32415c580 0x7ff3241f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 --2- 172.21.15.104:0/3355546224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff32406e6e0 0x7ff3240ad560 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 >> 172.21.15.104:0/3355546224 conn(0x7ff32406b8a0 msgr2=0x7ff324159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 shutdown_connections 2023-06-10T18:45:26.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.075+0000 7ff32a50b700 1 -- 172.21.15.104:0/3355546224 wait complete. 2023-06-10T18:45:26.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-10T18:45:26.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.209+0000 7f47f7d8f700 1 Processor -- start 2023-06-10T18:45:26.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.209+0000 7f47f7d8f700 1 -- start start 2023-06-10T18:45:26.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f7d8f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47f015c580 0x7f47f00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f7d8f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f00ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f7d8f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f006e250 con 0x7f47f006e6e0 2023-06-10T18:45:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f7d8f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f015c950 con 0x7f47f015c580 2023-06-10T18:45:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f7d8f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00b1e80 con 0x7f47f006cde0 2023-06-10T18:45:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f632c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f00ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f5b2b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47f015c580 0x7f47f00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f532a700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f47f006e6e0 0x7f47f00b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60560/0 (socket says 172.21.15.104:60560) 2023-06-10T18:45:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f532a700 1 -- 172.21.15.104:0/297903253 learned_addr learned my addr 172.21.15.104:0/297903253 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f5b2b700 1 -- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 msgr2=0x7f47f00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f5b2b700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f00ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f5b2b700 1 -- 172.21.15.104:0/297903253 >> v1:172.21.15.114:6789/0 conn(0x7f47f006e6e0 legacy=0x7f47f00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f5b2b700 1 -- 172.21.15.104:0/297903253 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f006d920 con 0x7f47f015c580 2023-06-10T18:45:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f4b29700 1 -- 172.21.15.104:0/297903253 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075414033 0 0) 0x7f47f006e250 con 0x7f47f006e6e0 2023-06-10T18:45:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.210+0000 7f47f632c700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f5b2b700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f015c580 0x7f47f00b1070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009820 tx=0x7f47ec009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77138151e6d21a17 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f4b29700 1 -- 172.21.15.104:0/297903253 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec005e30 con 0x7f47f015c580 2023-06-10T18:45:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f4b29700 1 -- 172.21.15.104:0/297903253 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec006030 con 0x7f47f015c580 2023-06-10T18:45:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f4b29700 1 -- 172.21.15.104:0/297903253 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0063c0 con 0x7f47f015c580 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 -- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f015c580 msgr2=0x7f47f00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f015c580 0x7f47f00b1070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009820 tx=0x7f47ec009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 -- 172.21.15.104:0/297903253 shutdown_connections 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/297903253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f015c580 0x7f47f00b1070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 -- 172.21.15.104:0/297903253 >> 172.21.15.104:0/297903253 conn(0x7f47f006b8a0 msgr2=0x7f47f00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 -- 172.21.15.104:0/297903253 shutdown_connections 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.211+0000 7f47f7d8f700 1 -- 172.21.15.104:0/297903253 wait complete. 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 Processor -- start 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 -- start start 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f0153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f006e6e0 0x7f47f00c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01e2340 con 0x7f47f015c580 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01e31a0 con 0x7f47f006e6e0 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f7d8f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01e2590 con 0x7f47f006cde0 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.213+0000 7f47f632c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56896/0 (socket says 172.21.15.104:56896) 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 -- 172.21.15.104:0/2002274175 learned_addr learned my addr 172.21.15.104:0/2002274175 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f532a700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f006e6e0 0x7f47f00c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f5b2b700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f0153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 -- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 msgr2=0x7f47f0153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f0153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 -- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f006e6e0 msgr2=0x7f47f00c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f006e6e0 0x7f47f00c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 -- 172.21.15.104:0/2002274175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009a70 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f632c700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47e800efc0 tx=0x7f47e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e800ca90 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e800f040 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e800ce10 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec009430 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.214+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f01e2b50 con 0x7f47f015c580 2023-06-10T18:45:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.215+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0074050 con 0x7f47f015c580 2023-06-10T18:45:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.216+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f47e8014480 con 0x7f47f015c580 2023-06-10T18:45:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.216+0000 7f47e67fc700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 0x7f47d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.216+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f47e8089e00 con 0x7f47f015c580 2023-06-10T18:45:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.216+0000 7f47f5b2b700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 0x7f47d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.217+0000 7f47f5b2b700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 0x7f47d406d190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009b70 tx=0x7f47ec008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.220+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f47e8058610 con 0x7f47f015c580 2023-06-10T18:45:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.408+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f47f00cb140 con 0x7f47f015c580 2023-06-10T18:45:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.408+0000 7f47e67fc700 1 -- 172.21.15.104:0/2002274175 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f47e8009d70 con 0x7f47f015c580 2023-06-10T18:45:26.413 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:26.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 msgr2=0x7f47d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 0x7f47d406d190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009b70 tx=0x7f47ec008070 comp rx=0 tx=0).stop 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 msgr2=0x7f47f00c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47e800efc0 tx=0x7f47e800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 shutdown_connections 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f015c580 0x7f47f00c3aa0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f006cde0 0x7f47f0153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47d406ace0 0x7f47d406d190 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 --2- 172.21.15.104:0/2002274175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f006e6e0 0x7f47f00c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 >> 172.21.15.104:0/2002274175 conn(0x7f47f006b8a0 msgr2=0x7f47f0159c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 shutdown_connections 2023-06-10T18:45:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.410+0000 7f47f7d8f700 1 -- 172.21.15.104:0/2002274175 wait complete. 2023-06-10T18:45:26.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:26.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.540+0000 7f7f6f4dc700 1 Processor -- start 2023-06-10T18:45:26.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.541+0000 7f7f6f4dc700 1 -- start start 2023-06-10T18:45:26.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.541+0000 7f7f6f4dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f680b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6f4dc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7f680cea60 0x7f7f6815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6f4dc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f680b6070 con 0x7f7f680b4ff0 2023-06-10T18:45:26.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6f4dc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f680b5460 con 0x7f7f680cea60 2023-06-10T18:45:26.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6f4dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680b5740 con 0x7f7f6815a420 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7f680cea60 0x7f7f6815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7f680cea60 0x7f7f6815a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56904/0 (socket says 172.21.15.104:56904) 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 -- 172.21.15.104:0/471295960 learned_addr learned my addr 172.21.15.104:0/471295960 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6d278700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f680b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 -- 172.21.15.104:0/471295960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 msgr2=0x7f7f680b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f680b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 -- 172.21.15.104:0/471295960 >> v1:172.21.15.114:6789/0 conn(0x7f7f680b4ff0 legacy=0x7f7f680ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:26.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 -- 172.21.15.104:0/471295960 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f680b5a20 con 0x7f7f680cea60 2023-06-10T18:45:26.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f6da79700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f6815a7f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009e50 tx=0x7f7f6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95d9bb8fbc179ce server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f5ffff700 1 -- 172.21.15.104:0/471295960 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6000e070 con 0x7f7f680cea60 2023-06-10T18:45:26.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f5ffff700 1 -- 172.21.15.104:0/471295960 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f60003a20 con 0x7f7f680cea60 2023-06-10T18:45:26.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.542+0000 7f7f5ffff700 1 -- 172.21.15.104:0/471295960 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60003dd0 con 0x7f7f680cea60 2023-06-10T18:45:26.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/471295960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 msgr2=0x7f7f6815a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f6815a7f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009e50 tx=0x7f7f6000b330 comp rx=0 tx=0).stop 2023-06-10T18:45:26.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/471295960 shutdown_connections 2023-06-10T18:45:26.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f6815a7f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/471295960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f680b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/471295960 >> 172.21.15.104:0/471295960 conn(0x7f7f68089fd0 msgr2=0x7f7f680d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/471295960 shutdown_connections 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.543+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/471295960 wait complete. 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.544+0000 7f7f6f4dc700 1 Processor -- start 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.544+0000 7f7f6f4dc700 1 -- start start 2023-06-10T18:45:26.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f680b4ff0 0x7f7f681f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f681e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f681f1e50 con 0x7f7f680cea60 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f681f6460 con 0x7f7f680b4ff0 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6f4dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680c9bd0 con 0x7f7f6815a420 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56910/0 (socket says 172.21.15.104:56910) 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 -- 172.21.15.104:0/1894399059 learned_addr learned my addr 172.21.15.104:0/1894399059 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6d278700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f680b4ff0 0x7f7f681f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6da79700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f681e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 -- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 msgr2=0x7f7f681e3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f681e3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 -- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f680b4ff0 msgr2=0x7f7f681f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f680b4ff0 0x7f7f681f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 -- 172.21.15.104:0/1894399059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f60009b30 con 0x7f7f680cea60 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f6ca77700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c4d0 tx=0x7f7f6400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64011790 con 0x7f7f680cea60 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f640118f0 con 0x7f7f680cea60 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.545+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64011ca0 con 0x7f7f680cea60 2023-06-10T18:45:26.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.546+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f680c9eb0 con 0x7f7f680cea60 2023-06-10T18:45:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.546+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f680ca470 con 0x7f7f680cea60 2023-06-10T18:45:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.547+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f680cf6f0 con 0x7f7f680cea60 2023-06-10T18:45:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.547+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7f6401c460 con 0x7f7f680cea60 2023-06-10T18:45:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.547+0000 7f7f5dffb700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 0x7f7f4c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.547+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7f640894d0 con 0x7f7f680cea60 2023-06-10T18:45:26.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.547+0000 7f7f6d278700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 0x7f7f4c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.548+0000 7f7f6d278700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 0x7f7f4c06d190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7f6806cc90 tx=0x7f7f58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.551+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7f640542e0 con 0x7f7f680cea60 2023-06-10T18:45:26.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.783+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f68074050 con 0x7f7f680cea60 2023-06-10T18:45:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.784+0000 7f7f5dffb700 1 -- 172.21.15.104:0/1894399059 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f7f64057900 con 0x7f7f680cea60 2023-06-10T18:45:26.789 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:26.789 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 msgr2=0x7f7f4c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 0x7f7f4c06d190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7f6806cc90 tx=0x7f7f58008040 comp rx=0 tx=0).stop 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 msgr2=0x7f7f681e3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c4d0 tx=0x7f7f6400c7e0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 shutdown_connections 2023-06-10T18:45:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7f4c06ace0 0x7f7f4c06d190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f680b4ff0 0x7f7f681f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6815a420 0x7f7f681e3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 --2- 172.21.15.104:0/1894399059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f680cea60 0x7f7f681e3a70 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 >> 172.21.15.104:0/1894399059 conn(0x7f7f68089fd0 msgr2=0x7f7f6806f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 shutdown_connections 2023-06-10T18:45:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.786+0000 7f7f6f4dc700 1 -- 172.21.15.104:0/1894399059 wait complete. 2023-06-10T18:45:26.793 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:45:26.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:26.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.918+0000 7f4645cf3700 1 Processor -- start 2023-06-10T18:45:26.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.918+0000 7f4645cf3700 1 -- start start 2023-06-10T18:45:26.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f464006e0e0 0x7f464006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f464006e4b0 0x7f46400c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 0x7f46400ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4640071640 con 0x7f464006e0e0 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46400aef00 con 0x7f46400ca100 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f4645cf3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464006d290 con 0x7f464006e4b0 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f464006e0e0 0x7f464006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f464006e0e0 0x7f464006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56914/0 (socket says 172.21.15.104:56914) 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 -- 172.21.15.104:0/1309713870 learned_addr learned my addr 172.21.15.104:0/1309713870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463ffff700 1 --2- 172.21.15.104:0/1309713870 >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 0x7f46400ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463effd700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f464006e4b0 0x7f46400c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 -- 172.21.15.104:0/1309713870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f464006e4b0 msgr2=0x7f46400c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f464006e4b0 0x7f46400c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 -- 172.21.15.104:0/1309713870 >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 msgr2=0x7f46400ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:26.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 --2- 172.21.15.104:0/1309713870 >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 0x7f46400ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463f7fe700 1 -- 172.21.15.104:0/1309713870 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464006d4c0 con 0x7f464006e0e0 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.919+0000 7f463ffff700 1 --2- 172.21.15.104:0/1309713870 >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 0x7f46400ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f463f7fe700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464006cd50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4640071210 tx=0x7f4634009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62bdc267a580e45 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f463e7fc700 1 -- 172.21.15.104:0/1309713870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634005e30 con 0x7f464006e0e0 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f463e7fc700 1 -- 172.21.15.104:0/1309713870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634004030 con 0x7f464006e0e0 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f463e7fc700 1 -- 172.21.15.104:0/1309713870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46340043e0 con 0x7f464006e0e0 2023-06-10T18:45:26.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 -- 172.21.15.104:0/1309713870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 msgr2=0x7f464006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464006cd50 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4640071210 tx=0x7f4634009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 -- 172.21.15.104:0/1309713870 shutdown_connections 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464006cd50 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 --2- 172.21.15.104:0/1309713870 >> v2:172.21.15.114:3300/0 conn(0x7f46400ca100 0x7f46400ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 --2- 172.21.15.104:0/1309713870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f464006e4b0 0x7f46400c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 -- 172.21.15.104:0/1309713870 >> 172.21.15.104:0/1309713870 conn(0x7f464006b8a0 msgr2=0x7f46400cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:26.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 -- 172.21.15.104:0/1309713870 shutdown_connections 2023-06-10T18:45:26.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.920+0000 7f4645cf3700 1 -- 172.21.15.104:0/1309713870 wait complete. 2023-06-10T18:45:26.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 Processor -- start 2023-06-10T18:45:26.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 -- start start 2023-06-10T18:45:26.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 0x7f464014f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46400ca100 0x7f46400b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400b0480 con 0x7f464006e0e0 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400b0720 con 0x7f464006e4b0 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.922+0000 7f4645cf3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46400b0a00 con 0x7f46400ca100 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56920/0 (socket says 172.21.15.104:56920) 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 -- 172.21.15.104:0/410216475 learned_addr learned my addr 172.21.15.104:0/410216475 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:26.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463effd700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 0x7f464014f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463ffff700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46400ca100 0x7f46400b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 -- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46400ca100 msgr2=0x7f46400b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46400ca100 0x7f46400b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 -- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 msgr2=0x7f464014f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 0x7f464014f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 -- 172.21.15.104:0/410216475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634009500 con 0x7f464006e0e0 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463effd700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 0x7f464014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f463f7fe700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4634000c00 tx=0x7f4634004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634003ae0 con 0x7f464006e0e0 2023-06-10T18:45:26.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634003c40 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46400b0ce0 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634011420 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.923+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46400b12a0 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.924+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4640074050 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.925+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4634018070 con 0x7f464006e0e0 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.925+0000 7f4627fff700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 0x7f462c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:26.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.925+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4634089f70 con 0x7f464006e0e0 2023-06-10T18:45:26.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.925+0000 7f463effd700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 0x7f462c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:26.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.926+0000 7f463effd700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 0x7f462c06d190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46400c60a0 tx=0x7f4630008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:26.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:26.929+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f46340bc310 con 0x7f464006e0e0 2023-06-10T18:45:27.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.115+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46400707a0 con 0x7f464006e0e0 2023-06-10T18:45:27.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.117+0000 7f4627fff700 1 -- 172.21.15.104:0/410216475 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f4634054d80 con 0x7f464006e0e0 2023-06-10T18:45:27.121 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:27.122 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:27.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 msgr2=0x7f462c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 0x7f462c06d190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46400c60a0 tx=0x7f4630008040 comp rx=0 tx=0).stop 2023-06-10T18:45:27.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 msgr2=0x7f464014fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4634000c00 tx=0x7f4634004750 comp rx=0 tx=0).stop 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 shutdown_connections 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46400ca100 0x7f46400b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f464006e0e0 0x7f464014fe00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f462c06ace0 0x7f462c06d190 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 --2- 172.21.15.104:0/410216475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f464006e4b0 0x7f464014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 >> 172.21.15.104:0/410216475 conn(0x7f464006b8a0 msgr2=0x7f46400ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:27.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 shutdown_connections 2023-06-10T18:45:27.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.119+0000 7f4645cf3700 1 -- 172.21.15.104:0/410216475 wait complete. 2023-06-10T18:45:27.136 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:45:27.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:27.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.252+0000 7ff7966b7700 1 Processor -- start 2023-06-10T18:45:27.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.252+0000 7ff7966b7700 1 -- start start 2023-06-10T18:45:27.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff7966b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff79006b1f0 0x7ff7900cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff7966b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7900cadd0 con 0x7ff79008a3d0 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff7966b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7900cb0b0 con 0x7ff7900ce350 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff7966b7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7900cfeb0 con 0x7ff79006b1f0 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff794c54700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff79008a3d0 0x7ff7900cb540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54758/0 (socket says 172.21.15.104:54758) 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff794c54700 1 -- 172.21.15.104:0/809438109 learned_addr learned my addr 172.21.15.104:0/809438109 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.253+0000 7ff78ffff700 1 --2- 172.21.15.104:0/809438109 >> v2:172.21.15.114:3300/0 conn(0x7ff79006b1f0 0x7ff7900cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78effd700 1 -- 172.21.15.104:0/809438109 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519640588 0 0) 0x7ff7900cadd0 con 0x7ff79008a3d0 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78effd700 1 -- 172.21.15.104:0/809438109 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77c0035b0 con 0x7ff79008a3d0 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78ffff700 1 -- 172.21.15.104:0/809438109 >> v1:172.21.15.114:6789/0 conn(0x7ff7900ce350 legacy=0x7ff7900cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78ffff700 1 -- 172.21.15.104:0/809438109 >> v1:172.21.15.104:6789/0 conn(0x7ff79008a3d0 legacy=0x7ff7900cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78ffff700 1 -- 172.21.15.104:0/809438109 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900cf550 con 0x7ff79006b1f0 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78ffff700 1 --2- 172.21.15.104:0/809438109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79006b1f0 0x7ff7900cd090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff780009840 tx=0x7ff780009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9050e6561d42f66f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78effd700 1 -- 172.21.15.104:0/809438109 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780011830 con 0x7ff79006b1f0 2023-06-10T18:45:27.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78effd700 1 -- 172.21.15.104:0/809438109 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff780011990 con 0x7ff79006b1f0 2023-06-10T18:45:27.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.254+0000 7ff78effd700 1 -- 172.21.15.104:0/809438109 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780011d20 con 0x7ff79006b1f0 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 -- 172.21.15.104:0/809438109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79006b1f0 msgr2=0x7ff7900cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 --2- 172.21.15.104:0/809438109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79006b1f0 0x7ff7900cd090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff780009840 tx=0x7ff780009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 -- 172.21.15.104:0/809438109 shutdown_connections 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 --2- 172.21.15.104:0/809438109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79006b1f0 0x7ff7900cd090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 -- 172.21.15.104:0/809438109 >> 172.21.15.104:0/809438109 conn(0x7ff790089fd0 msgr2=0x7ff7900d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 -- 172.21.15.104:0/809438109 shutdown_connections 2023-06-10T18:45:27.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.255+0000 7ff7966b7700 1 -- 172.21.15.104:0/809438109 wait complete. 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.256+0000 7ff7966b7700 1 Processor -- start 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 -- start start 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff79006b1f0 0x7ff790153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 0x7ff7901e5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900ca0f0 con 0x7ff7900ce350 2023-06-10T18:45:27.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900c9e10 con 0x7ff79008a3d0 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff7966b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7901545a0 con 0x7ff79006b1f0 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.257+0000 7ff794c54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56938/0 (socket says 172.21.15.104:56938) 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 -- 172.21.15.104:0/1747629920 learned_addr learned my addr 172.21.15.104:0/1747629920 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78ffff700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff79006b1f0 0x7ff790153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78f7fe700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 0x7ff7901e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 -- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff79006b1f0 msgr2=0x7ff790153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff79006b1f0 0x7ff790153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 -- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 msgr2=0x7ff7901e5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 0x7ff7901e5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 -- 172.21.15.104:0/1747629920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7800094a0 con 0x7ff7900ce350 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78f7fe700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 0x7ff7901e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff794c54700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff78800cb40 tx=0x7ff78800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78800ab50 con 0x7ff7900ce350 2023-06-10T18:45:27.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78800acb0 con 0x7ff7900ce350 2023-06-10T18:45:27.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff790154880 con 0x7ff7900ce350 2023-06-10T18:45:27.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788012620 con 0x7ff7900ce350 2023-06-10T18:45:27.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.258+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7901570a0 con 0x7ff7900ce350 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.259+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790154da0 con 0x7ff7900ce350 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.260+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff78801b3b0 con 0x7ff7900ce350 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.260+0000 7ff78cff9700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 0x7ff77006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.260+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff788089630 con 0x7ff7900ce350 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.260+0000 7ff78ffff700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 0x7ff77006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.260+0000 7ff78ffff700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 0x7ff77006d190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff7800090b0 tx=0x7ff780008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.264+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff788054330 con 0x7ff7900ce350 2023-06-10T18:45:27.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.490+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7900cf830 con 0x7ff7900ce350 2023-06-10T18:45:27.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.490+0000 7ff78cff9700 1 -- 172.21.15.104:0/1747629920 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7ff788057950 con 0x7ff7900ce350 2023-06-10T18:45:27.495 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:27.495 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:45:27.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.492+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 msgr2=0x7ff77006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.492+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 0x7ff77006d190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff7800090b0 tx=0x7ff780008030 comp rx=0 tx=0).stop 2023-06-10T18:45:27.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.492+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 msgr2=0x7ff7901e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.492+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff78800cb40 tx=0x7ff78800cf00 comp rx=0 tx=0).stop 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 shutdown_connections 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7900ce350 0x7ff7901e5ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff79006b1f0 0x7ff790153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff77006ace0 0x7ff77006d190 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 --2- 172.21.15.104:0/1747629920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79008a3d0 0x7ff7901e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 >> 172.21.15.104:0/1747629920 conn(0x7ff790089fd0 msgr2=0x7ff7900ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:27.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 shutdown_connections 2023-06-10T18:45:27.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.493+0000 7ff7966b7700 1 -- 172.21.15.104:0/1747629920 wait complete. 2023-06-10T18:45:27.499 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:45:27.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.624+0000 7f65b26e9700 1 Processor -- start 2023-06-10T18:45:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.624+0000 7f65b26e9700 1 -- start start 2023-06-10T18:45:27.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65b26e9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 0x7f65ac0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65b26e9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65ac0b4ff0 0x7f65ac0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65b26e9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac0b6070 con 0x7f65ac0ce890 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65b26e9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ac0cecb0 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65b26e9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ac0b5460 con 0x7f65ac15a420 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65ab7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65ac0b4ff0 0x7f65ac0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65ab7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65ac0b4ff0 0x7f65ac0ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56942/0 (socket says 172.21.15.104:56942) 2023-06-10T18:45:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.625+0000 7f65ab7fe700 1 -- 172.21.15.104:0/2605063008 learned_addr learned my addr 172.21.15.104:0/2605063008 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65abfff700 1 --2- 172.21.15.104:0/2605063008 >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 0x7f65ac0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65ab7fe700 1 -- 172.21.15.104:0/2605063008 >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 msgr2=0x7f65ac0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65ab7fe700 1 --2- 172.21.15.104:0/2605063008 >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 0x7f65ac0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65ab7fe700 1 -- 172.21.15.104:0/2605063008 >> v1:172.21.15.114:6789/0 conn(0x7f65ac0ce890 legacy=0x7f65ac0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65ab7fe700 1 -- 172.21.15.104:0/2605063008 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac0b5740 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65abfff700 1 --2- 172.21.15.104:0/2605063008 >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 0x7f65ac0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65ab7fe700 1 --2- 172.21.15.104:0/2605063008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0b4ff0 0x7f65ac0ce350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0b32e0 tx=0x7f659c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22db623956ae3687 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65aaffd700 1 -- 172.21.15.104:0/2605063008 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c00b870 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65aaffd700 1 -- 172.21.15.104:0/2605063008 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c00b9d0 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65aaffd700 1 -- 172.21.15.104:0/2605063008 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c00bd60 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65b26e9700 1 -- 172.21.15.104:0/2605063008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0b4ff0 msgr2=0x7f65ac0ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.626+0000 7f65b26e9700 1 --2- 172.21.15.104:0/2605063008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0b4ff0 0x7f65ac0ce350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0b32e0 tx=0x7f659c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 -- 172.21.15.104:0/2605063008 shutdown_connections 2023-06-10T18:45:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 --2- 172.21.15.104:0/2605063008 >> v2:172.21.15.114:3300/0 conn(0x7f65ac15a420 0x7f65ac0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 --2- 172.21.15.104:0/2605063008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0b4ff0 0x7f65ac0ce350 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 -- 172.21.15.104:0/2605063008 >> 172.21.15.104:0/2605063008 conn(0x7f65ac089fd0 msgr2=0x7f65ac0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 -- 172.21.15.104:0/2605063008 shutdown_connections 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.627+0000 7f65b26e9700 1 -- 172.21.15.104:0/2605063008 wait complete. 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.628+0000 7f65b26e9700 1 Processor -- start 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.628+0000 7f65b26e9700 1 -- start start 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65ac0b4ff0 0x7f65ac14a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 0x7f65ac0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0c42b0 con 0x7f65ac0ce890 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0c8c40 con 0x7f65ac15a420 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0c8f20 con 0x7f65ac0b4ff0 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56950/0 (socket says 172.21.15.104:56950) 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 -- 172.21.15.104:0/1975100663 learned_addr learned my addr 172.21.15.104:0/1975100663 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65abfff700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65ac0b4ff0 0x7f65ac14a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b0c86700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 0x7f65ac0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 -- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65ac0b4ff0 msgr2=0x7f65ac14a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65ac0b4ff0 0x7f65ac14a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 -- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 msgr2=0x7f65ac0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 0x7f65ac0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 -- 172.21.15.104:0/1975100663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c009550 con 0x7f65ac0ce890 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b0c86700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 0x7f65ac0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65ab7fe700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f659c000c00 tx=0x7f659c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c004540 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c0036a0 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac0c9200 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c003a30 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.629+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac0c87a0 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.630+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac0c4740 con 0x7f65ac0ce890 2023-06-10T18:45:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.631+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f659c021070 con 0x7f65ac0ce890 2023-06-10T18:45:27.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.631+0000 7f65a8ff9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 0x7f659806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:27.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.631+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f659c08afc0 con 0x7f65ac0ce890 2023-06-10T18:45:27.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.631+0000 7f65abfff700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 0x7f659806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:27.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.632+0000 7f65abfff700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 0x7f659806d190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65a000c900 tx=0x7f65a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:27.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.635+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f65ac0c4740 con 0x7f65ac0ce890 2023-06-10T18:45:27.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.865+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65ac0c4740 con 0x7f65ac0ce890 2023-06-10T18:45:27.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.866+0000 7f65a8ff9700 1 -- 172.21.15.104:0/1975100663 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4676 (secure 0 0 0) 0x7f65ac0c4740 con 0x7f65ac0ce890 2023-06-10T18:45:27.871 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:27.871 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4268,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3321788728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3321788728},{"type":"v1","addr":"172.21.15.114:6837","nonce":3321788728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3274908199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3274908199},{"type":"v1","addr":"172.21.15.104:6834","nonce":3274908199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/602446782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":602446782},{"type":"v1","addr":"172.21.15.104:6835","nonce":602446782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:44:08.959124+0000","modified":"2023-06-10T18:44:31.543091+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3495904966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3495904966},{"type":"v1","addr":"172.21.15.114:6836","nonce":3495904966}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 msgr2=0x7f659806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 0x7f659806d190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65a000c900 tx=0x7f65a0008040 comp rx=0 tx=0).stop 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 msgr2=0x7f65ac0c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f659c000c00 tx=0x7f659c004750 comp rx=0 tx=0).stop 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 shutdown_connections 2023-06-10T18:45:27.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f659806ace0 0x7f659806d190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65ac15a420 0x7f65ac0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65ac0b4ff0 0x7f65ac14a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 --2- 172.21.15.104:0/1975100663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65ac0ce890 0x7f65ac0c8260 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 >> 172.21.15.104:0/1975100663 conn(0x7f65ac089fd0 msgr2=0x7f65ac157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 shutdown_connections 2023-06-10T18:45:27.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.868+0000 7f65b26e9700 1 -- 172.21.15.104:0/1975100663 wait complete. 2023-06-10T18:45:27.875 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 9 2023-06-10T18:45:27.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:28.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.999+0000 7f3c31ffa700 1 Processor -- start 2023-06-10T18:45:28.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:27.999+0000 7f3c31ffa700 1 -- start start 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c31ffa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c2c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c31ffa700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0ae720 con 0x7f3c2c0ae280 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c31ffa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0ce2c0 con 0x7f3c2c15c040 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c31ffa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0cf750 con 0x7f3c2c071130 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c2bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3c2c0ae280 0x7f3c2c0ca3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54788/0 (socket says 172.21.15.104:54788) 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c2bfff700 1 -- 172.21.15.104:0/2948597053 learned_addr learned my addr 172.21.15.104:0/2948597053 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:28.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c2affd700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c2c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627195771 0 0) 0x7f3c24001490 con 0x7f3c2c0ae280 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.000+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c140035b0 con 0x7f3c2c0ae280 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353232708 0 0) 0x7f3c140035b0 con 0x7f3c2c0ae280 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 msgr2=0x7f3c2c0ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c2c0ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 >> v1:172.21.15.114:6789/0 conn(0x7f3c2c15c040 legacy=0x7f3c2c070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c0cfa30 con 0x7f3c2c0ae280 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3c24001b80 con 0x7f3c2c0ae280 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 >> v1:172.21.15.104:6789/0 conn(0x7f3c2c0ae280 legacy=0x7f3c2c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c14003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14003f70 0x7f3c14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c14004d90 0x7f3c14005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0cf750 con 0x7f3c14004d90 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1400bb50 con 0x7f3c14003f70 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1400bd10 con 0x7f3c2c071130 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c14004d90 0x7f3c14005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2b7fe700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c14003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2affd700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14003f70 0x7f3c14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 -- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 msgr2=0x7f3c14003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c071130 0x7f3c14003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 -- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14003f70 msgr2=0x7f3c14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14003f70 0x7f3c14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 -- 172.21.15.104:0/2948597053 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c140035b0 con 0x7f3c14004d90 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c14004d90 0x7f3c14005200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400a750 tx=0x7f3c14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe3790136af8cda server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400a3e0 con 0x7f3c14004d90 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.001+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c240072f0 con 0x7f3c14004d90 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c2a7fc700 1 -- 172.21.15.104:0/2948597053 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240076a0 con 0x7f3c14004d90 2023-06-10T18:45:28.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 -- 172.21.15.104:0/2948597053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c14004d90 msgr2=0x7f3c14005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/2948597053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c14004d90 0x7f3c14005200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400a750 tx=0x7f3c14005880 comp rx=0 tx=0).stop 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c2bfff700 1 -- 172.21.15.104:0/2948597053 reap_dead start 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 -- 172.21.15.104:0/2948597053 shutdown_connections 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 -- 172.21.15.104:0/2948597053 >> 172.21.15.104:0/2948597053 conn(0x7f3c2c0b1610 msgr2=0x7f3c2c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 -- 172.21.15.104:0/2948597053 shutdown_connections 2023-06-10T18:45:28.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.002+0000 7f3c31ffa700 1 -- 172.21.15.104:0/2948597053 wait complete. 2023-06-10T18:45:28.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.003+0000 7f3c31ffa700 1 Processor -- start 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.003+0000 7f3c31ffa700 1 -- start start 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c0b4590 0x7f3c2c0c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 0x7f3c2c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c7550 con 0x7f3c2c0b4960 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0ad340 con 0x7f3c2c0ac980 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c31ffa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0b1c80 con 0x7f3c2c0b4590 2023-06-10T18:45:28.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56970/0 (socket says 172.21.15.104:56970) 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c2affd700 1 -- 172.21.15.104:0/380029336 learned_addr learned my addr 172.21.15.104:0/380029336 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 0x7f3c2c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.004+0000 7f3c2b7fe700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c0b4590 0x7f3c2c0c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 -- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c0b4590 msgr2=0x7f3c2c0c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c0b4590 0x7f3c2c0c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 -- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 msgr2=0x7f3c2c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 0x7f3c2c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 -- 172.21.15.104:0/380029336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c15e040 con 0x7f3c2c0b4960 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2bfff700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 0x7f3c2c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c2affd700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0119f0 tx=0x7f3c1c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c004300 con 0x7f3c2c0b4960 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c004460 con 0x7f3c2c0b4960 2023-06-10T18:45:28.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c00f950 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c0c83d0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.005+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c0c8990 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.006+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c15dea0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.006+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3c1c00fab0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.007+0000 7f3c13fff700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 0x7f3c1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.007+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3c1c089e10 con 0x7f3c2c0b4960 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.007+0000 7f3c2b7fe700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 0x7f3c1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.007+0000 7f3c2b7fe700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 0x7f3c1806d190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c1f5ad0 tx=0x7f3c20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.011+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3c2c15dea0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.198+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c2c15dea0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.199+0000 7f3c13fff700 1 -- 172.21.15.104:0/380029336 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20264 (secure 0 0 0) 0x7f3c2c15dea0 con 0x7f3c2c0b4960 2023-06-10T18:45:28.204 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:28.204 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":29,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:44:50.688129+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:44:06.622378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-10T18:44:07.927610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-10T18:44:10.992951+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.24","mappings":[{"from":1,"to":5}]},{"pgid":"4.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 msgr2=0x7f3c1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 0x7f3c1806d190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c1f5ad0 tx=0x7f3c20009040 comp rx=0 tx=0).stop 2023-06-10T18:45:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 msgr2=0x7f3c2c0c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0119f0 tx=0x7f3c1c011db0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 shutdown_connections 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3c1806ace0 0x7f3c1806d190 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c2c0ac980 0x7f3c2c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c2c0b4590 0x7f3c2c0c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 --2- 172.21.15.104:0/380029336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c2c0b4960 0x7f3c2c0c7010 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 >> 172.21.15.104:0/380029336 conn(0x7f3c2c0b1610 msgr2=0x7f3c2c0af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 shutdown_connections 2023-06-10T18:45:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.201+0000 7f3c31ffa700 1 -- 172.21.15.104:0/380029336 wait complete. 2023-06-10T18:45:28.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:45:28.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.333+0000 7f6a79ba9700 1 Processor -- start 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.333+0000 7f6a79ba9700 1 -- start start 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a7415c500 0x7f6a740cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a740ce040 0x7f6a7406e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 0x7f6a7406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7406a830 con 0x7f6a740ce040 2023-06-10T18:45:28.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7406dcb0 con 0x7f6a7415c500 2023-06-10T18:45:28.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a79ba9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740ce470 con 0x7f6a74069e70 2023-06-10T18:45:28.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a740ce040 0x7f6a7406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a740ce040 0x7f6a7406e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56976/0 (socket says 172.21.15.104:56976) 2023-06-10T18:45:28.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 -- 172.21.15.104:0/4089565402 learned_addr learned my addr 172.21.15.104:0/4089565402 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:28.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a737fe700 1 --2- 172.21.15.104:0/4089565402 >> v2:172.21.15.114:3300/0 conn(0x7f6a7415c500 0x7f6a740cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a73fff700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 0x7f6a7406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 -- 172.21.15.104:0/4089565402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 msgr2=0x7f6a7406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:28.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 0x7f6a7406a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 -- 172.21.15.104:0/4089565402 >> v2:172.21.15.114:3300/0 conn(0x7f6a7415c500 msgr2=0x7f6a740cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:28.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089565402 >> v2:172.21.15.114:3300/0 conn(0x7f6a7415c500 0x7f6a740cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.334+0000 7f6a72ffd700 1 -- 172.21.15.104:0/4089565402 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7406d0a0 con 0x7f6a740ce040 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a73fff700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 0x7f6a7406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a740ce040 0x7f6a7406e140 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a74067130 tx=0x7f6a64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58915beffc3bd33 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a727fc700 1 -- 172.21.15.104:0/4089565402 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400b870 con 0x7f6a740ce040 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a727fc700 1 -- 172.21.15.104:0/4089565402 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6400b9d0 con 0x7f6a740ce040 2023-06-10T18:45:28.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a727fc700 1 -- 172.21.15.104:0/4089565402 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400bd60 con 0x7f6a740ce040 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089565402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a740ce040 msgr2=0x7f6a7406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a740ce040 0x7f6a7406e140 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a74067130 tx=0x7f6a64009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089565402 shutdown_connections 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a74069e70 0x7f6a7406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089565402 >> v2:172.21.15.114:3300/0 conn(0x7f6a7415c500 0x7f6a740cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089565402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a740ce040 0x7f6a7406e140 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089565402 >> 172.21.15.104:0/4089565402 conn(0x7f6a7406b8a0 msgr2=0x7f6a740cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089565402 shutdown_connections 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.335+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089565402 wait complete. 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 Processor -- start 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 -- start start 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 0x7f6a740b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 0x7f6a740b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741500d0 con 0x7f6a74069e70 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7414cf00 con 0x7f6a7415c500 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.337+0000 7f6a79ba9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741f1fe0 con 0x7f6a740ce040 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56982/0 (socket says 172.21.15.104:56982) 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 -- 172.21.15.104:0/4089264228 learned_addr learned my addr 172.21.15.104:0/4089264228 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:28.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 0x7f6a740b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a73fff700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 0x7f6a740b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 -- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 msgr2=0x7f6a740b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 0x7f6a740b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 -- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 msgr2=0x7f6a740b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 0x7f6a740b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 -- 172.21.15.104:0/4089264228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009550 con 0x7f6a74069e70 2023-06-10T18:45:28.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a73fff700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 0x7f6a740b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 0x7f6a740b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a737fe700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00e730 tx=0x7f6a5c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c013610 con 0x7f6a74069e70 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c013770 con 0x7f6a74069e70 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7415ca70 con 0x7f6a74069e70 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c013b20 con 0x7f6a74069e70 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.338+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7415d030 con 0x7f6a74069e70 2023-06-10T18:45:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.339+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a74074050 con 0x7f6a74069e70 2023-06-10T18:45:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.339+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a5c013c80 con 0x7f6a74069e70 2023-06-10T18:45:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.340+0000 7f6a6bfff700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 0x7f6a5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.340+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6a5c089190 con 0x7f6a74069e70 2023-06-10T18:45:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.340+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 0x7f6a5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.341+0000 7f6a72ffd700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 0x7f6a5406d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009840 tx=0x7f6a64008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:28.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.344+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a5c053e80 con 0x7f6a74069e70 2023-06-10T18:45:28.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:28.569+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6a7406efd0 con 0x7f6a74069e70 2023-06-10T18:45:29.532 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:45:29.526+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:45:29.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.530+0000 7f6a6bfff700 1 -- 172.21.15.104:0/4089264228 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6a5c0574a0 con 0x7f6a74069e70 2023-06-10T18:45:29.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 msgr2=0x7f6a5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 0x7f6a5406d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009840 tx=0x7f6a64008030 comp rx=0 tx=0).stop 2023-06-10T18:45:29.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 msgr2=0x7f6a74143b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00e730 tx=0x7f6a5c00eaf0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 shutdown_connections 2023-06-10T18:45:29.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.532+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a5406ace0 0x7f6a5406d190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7415c500 0x7f6a740b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a74069e70 0x7f6a74143b80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 --2- 172.21.15.104:0/4089264228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a740ce040 0x7f6a740b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 >> 172.21.15.104:0/4089264228 conn(0x7f6a7406b8a0 msgr2=0x7f6a740cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:29.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 shutdown_connections 2023-06-10T18:45:29.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.533+0000 7f6a79ba9700 1 -- 172.21.15.104:0/4089264228 wait complete. 2023-06-10T18:45:29.541 INFO:tasks.ceph.mds.d.smithi114.stderr: -5493> 2023-06-10T18:44:14.662+0000 7fbb094e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:45:29.542 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:45:29.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:45:29.570 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:45:29.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.674+0000 7faee3124700 1 Processor -- start 2023-06-10T18:45:29.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.674+0000 7faee3124700 1 -- start start 2023-06-10T18:45:29.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee3124700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faedc0ad930 0x7faedc0add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:29.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee3124700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc15cb00 con 0x7faedc15c730 2023-06-10T18:45:29.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee3124700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc0acd90 con 0x7faedc0cd8f0 2023-06-10T18:45:29.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee3124700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0ac460 con 0x7faedc0ad930 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee0ec0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faedc15c730 0x7faedc0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54818/0 (socket says 172.21.15.104:54818) 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee0ec0700 1 -- 172.21.15.104:0/3855951415 learned_addr learned my addr 172.21.15.104:0/3855951415 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.675+0000 7faee16c1700 1 --2- 172.21.15.104:0/3855951415 >> v2:172.21.15.114:3300/0 conn(0x7faedc0ad930 0x7faedc0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faedb7fe700 1 -- 172.21.15.104:0/3855951415 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486971064 0 0) 0x7faed0001290 con 0x7faedc15c730 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faedb7fe700 1 -- 172.21.15.104:0/3855951415 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc0035b0 con 0x7faedc15c730 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee16c1700 1 -- 172.21.15.104:0/3855951415 >> v1:172.21.15.114:6789/0 conn(0x7faedc0cd8f0 legacy=0x7faedc0ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee16c1700 1 -- 172.21.15.104:0/3855951415 >> v1:172.21.15.104:6789/0 conn(0x7faedc15c730 legacy=0x7faedc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee16c1700 1 -- 172.21.15.104:0/3855951415 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0ac740 con 0x7faedc0ad930 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee16c1700 1 --2- 172.21.15.104:0/3855951415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0add50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faed4009d90 tx=0x7faed400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2a5084e172b655 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faedb7fe700 1 -- 172.21.15.104:0/3855951415 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400e070 con 0x7faedc0ad930 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faedb7fe700 1 -- 172.21.15.104:0/3855951415 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400bb20 con 0x7faedc0ad930 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faedb7fe700 1 -- 172.21.15.104:0/3855951415 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4012430 con 0x7faedc0ad930 2023-06-10T18:45:29.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee3124700 1 -- 172.21.15.104:0/3855951415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 msgr2=0x7faedc0add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.676+0000 7faee3124700 1 --2- 172.21.15.104:0/3855951415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0add50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faed4009d90 tx=0x7faed400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.677+0000 7faee3124700 1 -- 172.21.15.104:0/3855951415 shutdown_connections 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.677+0000 7faee3124700 1 --2- 172.21.15.104:0/3855951415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0add50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.677+0000 7faee3124700 1 -- 172.21.15.104:0/3855951415 >> 172.21.15.104:0/3855951415 conn(0x7faedc06b8a0 msgr2=0x7faedc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.677+0000 7faee3124700 1 -- 172.21.15.104:0/3855951415 shutdown_connections 2023-06-10T18:45:29.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.677+0000 7faee3124700 1 -- 172.21.15.104:0/3855951415 wait complete. 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.678+0000 7faee3124700 1 Processor -- start 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.678+0000 7faee3124700 1 -- start start 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0c3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 0x7faedc153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:29.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0c2f50 con 0x7faedc15c730 2023-06-10T18:45:29.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0c34d0 con 0x7faedc0ad930 2023-06-10T18:45:29.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b1aa0 con 0x7faedc0cd8f0 2023-06-10T18:45:29.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:29.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57000/0 (socket says 172.21.15.104:57000) 2023-06-10T18:45:29.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 -- 172.21.15.104:0/787682259 learned_addr learned my addr 172.21.15.104:0/787682259 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee0ec0700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faedbfff700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 0x7faedc153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 -- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 msgr2=0x7faedc153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 0x7faedc153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 -- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 msgr2=0x7faedc0c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 -- 172.21.15.104:0/787682259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4009a70 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faedbfff700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 0x7faedc153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee16c1700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faedc0b2200 tx=0x7faed4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4012430 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4020070 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4004280 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc06f150 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.679+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc0c3160 con 0x7faedc15c730 2023-06-10T18:45:29.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.680+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc0c3640 con 0x7faedc15c730 2023-06-10T18:45:29.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.681+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faed40043e0 con 0x7faedc15c730 2023-06-10T18:45:29.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.681+0000 7faed97fa700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 0x7faec006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:29.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.681+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6957+0+0 (secure 0 0 0) 0x7faed4022070 con 0x7faedc15c730 2023-06-10T18:45:29.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.681+0000 7faee0ec0700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 0x7faec006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:29.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.682+0000 7faee0ec0700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 0x7faec006d190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faed00010e0 tx=0x7faed0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:29.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.686+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faed4056250 con 0x7faedc15c730 2023-06-10T18:45:29.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:29.916+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faedc066f10 con 0x7faedc15c730 2023-06-10T18:45:30.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.531+0000 7faed97fa700 1 -- 172.21.15.104:0/787682259 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7faed4059870 con 0x7faedc15c730 2023-06-10T18:45:30.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 msgr2=0x7faec006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 0x7faec006d190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faed00010e0 tx=0x7faed0009040 comp rx=0 tx=0).stop 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 msgr2=0x7faedc0b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faedc0b2200 tx=0x7faed4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 shutdown_connections 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faec006ace0 0x7faec006d190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faedc0ad930 0x7faedc0c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faedc15c730 0x7faedc0b3a30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 --2- 172.21.15.104:0/787682259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faedc0cd8f0 0x7faedc153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 >> 172.21.15.104:0/787682259 conn(0x7faedc06b8a0 msgr2=0x7faedc0cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:30.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 shutdown_connections 2023-06-10T18:45:30.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.534+0000 7faee3124700 1 -- 172.21.15.104:0/787682259 wait complete. 2023-06-10T18:45:30.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:45:30.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.673+0000 7f335fe18700 1 Processor -- start 2023-06-10T18:45:30.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 -- start start 2023-06-10T18:45:30.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 0x7f33580b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f335806e6e0 0x7f33580b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335806e250 con 0x7f335806cde0 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335815c950 con 0x7f335806e6e0 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335fe18700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33580cf0e0 con 0x7f335815c580 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.674+0000 7f335d3b3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f335806e6e0 0x7f33580b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f335806e6e0 0x7f33580b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57006/0 (socket says 172.21.15.104:57006) 2023-06-10T18:45:30.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 -- 172.21.15.104:0/3887464664 learned_addr learned my addr 172.21.15.104:0/3887464664 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335dbb4700 1 --2- 172.21.15.104:0/3887464664 >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 0x7f33580b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 -- 172.21.15.104:0/3887464664 >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 msgr2=0x7f33580b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 --2- 172.21.15.104:0/3887464664 >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 0x7f33580b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 -- 172.21.15.104:0/3887464664 >> v1:172.21.15.114:6789/0 conn(0x7f335806cde0 legacy=0x7f3358069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 -- 172.21.15.104:0/3887464664 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580b1e80 con 0x7f335806e6e0 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335dbb4700 1 --2- 172.21.15.104:0/3887464664 >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 0x7f33580b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:30.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335d3b3700 1 --2- 172.21.15.104:0/3887464664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335806e6e0 0x7f33580b15b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3358070300 tx=0x7f3354009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ddc8c159d7c803 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:30.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335cbb2700 1 -- 172.21.15.104:0/3887464664 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335400b870 con 0x7f335806e6e0 2023-06-10T18:45:30.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335cbb2700 1 -- 172.21.15.104:0/3887464664 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335400b9d0 con 0x7f335806e6e0 2023-06-10T18:45:30.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.675+0000 7f335cbb2700 1 -- 172.21.15.104:0/3887464664 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335400bd60 con 0x7f335806e6e0 2023-06-10T18:45:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 -- 172.21.15.104:0/3887464664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335806e6e0 msgr2=0x7f33580b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 --2- 172.21.15.104:0/3887464664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335806e6e0 0x7f33580b15b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3358070300 tx=0x7f3354009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 -- 172.21.15.104:0/3887464664 shutdown_connections 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 --2- 172.21.15.104:0/3887464664 >> v2:172.21.15.114:3300/0 conn(0x7f335815c580 0x7f33580b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 --2- 172.21.15.104:0/3887464664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335806e6e0 0x7f33580b15b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 -- 172.21.15.104:0/3887464664 >> 172.21.15.104:0/3887464664 conn(0x7f335806b8a0 msgr2=0x7f33580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 -- 172.21.15.104:0/3887464664 shutdown_connections 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.676+0000 7f335fe18700 1 -- 172.21.15.104:0/3887464664 wait complete. 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.677+0000 7f335fe18700 1 Processor -- start 2023-06-10T18:45:30.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.677+0000 7f335fe18700 1 -- start start 2023-06-10T18:45:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f335806cde0 0x7f33580cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 0x7f3358149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580af180 con 0x7f335815c580 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580af460 con 0x7f335806e6e0 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33580af740 con 0x7f335806cde0 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57014/0 (socket says 172.21.15.104:57014) 2023-06-10T18:45:30.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 -- 172.21.15.104:0/956999681 learned_addr learned my addr 172.21.15.104:0/956999681 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335d3b3700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 0x7f3358149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335dbb4700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f335806cde0 0x7f33580cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 -- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f335806cde0 msgr2=0x7f33580cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f335806cde0 0x7f33580cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 -- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 msgr2=0x7f3358149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 0x7f3358149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:30.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 -- 172.21.15.104:0/956999681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3354009550 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335d3b3700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 0x7f3358149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335e3b5700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f335000cb50 tx=0x7f335000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350003c00 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3350003d60 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33580afa20 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335000ab70 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.678+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33580cf9d0 con 0x7f335815c580 2023-06-10T18:45:30.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.679+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33580aff40 con 0x7f335815c580 2023-06-10T18:45:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.680+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f335000acd0 con 0x7f335815c580 2023-06-10T18:45:30.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.680+0000 7f334e7fc700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 0x7f333c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.680+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3350089810 con 0x7f335815c580 2023-06-10T18:45:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.680+0000 7f335dbb4700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 0x7f333c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.681+0000 7f335dbb4700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 0x7f333c06d190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3348009920 tx=0x7f3348008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:30.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.685+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f33580aff40 con 0x7f335815c580 2023-06-10T18:45:30.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:30.872+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f33580aff40 con 0x7f335815c580 2023-06-10T18:45:31.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.843+0000 7f334e7fc700 1 -- 172.21.15.104:0/956999681 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v31) v1 ==== 181+0+0 (secure 0 0 0) 0x7f33580aff40 con 0x7f335815c580 2023-06-10T18:45:31.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 msgr2=0x7f333c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 0x7f333c06d190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3348009920 tx=0x7f3348008040 comp rx=0 tx=0).stop 2023-06-10T18:45:31.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 msgr2=0x7f33580c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f335000cb50 tx=0x7f335000cf10 comp rx=0 tx=0).stop 2023-06-10T18:45:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 shutdown_connections 2023-06-10T18:45:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f335815c580 0x7f33580c89a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f335806cde0 0x7f33580cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f333c06ace0 0x7f333c06d190 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 --2- 172.21.15.104:0/956999681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f335806e6e0 0x7f3358149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 >> 172.21.15.104:0/956999681 conn(0x7f335806b8a0 msgr2=0x7f33580cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:31.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 shutdown_connections 2023-06-10T18:45:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.849+0000 7f335fe18700 1 -- 172.21.15.104:0/956999681 wait complete. 2023-06-10T18:45:31.859 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:45:31.870 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:45:31.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.993+0000 7faa4fd56700 1 Processor -- start 2023-06-10T18:45:31.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 -- start start 2023-06-10T18:45:32.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faa4815c580 0x7faa480b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa480ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4806e250 con 0x7faa4806e6e0 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4815c950 con 0x7faa4815c580 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4fd56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480b1e80 con 0x7faa4806cde0 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4d2f1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faa4806e6e0 0x7faa480b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54846/0 (socket says 172.21.15.104:54846) 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.994+0000 7faa4d2f1700 1 -- 172.21.15.104:0/95143062 learned_addr learned my addr 172.21.15.104:0/95143062 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275471666 0 0) 0x7faa4806e250 con 0x7faa4806e6e0 2023-06-10T18:45:32.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa340035b0 con 0x7faa4806e6e0 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4daf2700 1 --2- 172.21.15.104:0/95143062 >> v2:172.21.15.114:3300/0 conn(0x7faa4815c580 0x7faa480b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4e2f3700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa480ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769902712 0 0) 0x7faa340035b0 con 0x7faa4806e6e0 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 msgr2=0x7faa480ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa480ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 >> v2:172.21.15.114:3300/0 conn(0x7faa4815c580 msgr2=0x7faa480b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:32.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 --2- 172.21.15.104:0/95143062 >> v2:172.21.15.114:3300/0 conn(0x7faa4815c580 0x7faa480b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4806d920 con 0x7faa4806e6e0 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7faa44001670 con 0x7faa4806e6e0 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4daf2700 1 --2- 172.21.15.104:0/95143062 >> v2:172.21.15.114:3300/0 conn(0x7faa4815c580 0x7faa480b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 >> v1:172.21.15.104:6789/0 conn(0x7faa4806e6e0 legacy=0x7faa480b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa34003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa34003f80 0x7faa34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa34004d90 0x7faa34005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4815c950 con 0x7faa34003f80 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa480b1e80 con 0x7faa34004d90 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3400ba40 con 0x7faa4806cde0 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa34003f80 0x7faa34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4daf2700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa34003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4e2f3700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa34004d90 0x7faa34005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4d2f1700 1 -- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 msgr2=0x7faa34003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa4806cde0 0x7faa34003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4d2f1700 1 -- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa34004d90 msgr2=0x7faa34005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.995+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa34004d90 0x7faa34005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4d2f1700 1 -- 172.21.15.104:0/95143062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa340035b0 con 0x7faa34003f80 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa34003f80 0x7faa34004850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa4806eb20 tx=0x7faa34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47f4f46d2620c74 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44007c60 con 0x7faa34003f80 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4400ce80 con 0x7faa34003f80 2023-06-10T18:45:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4caf0700 1 -- 172.21.15.104:0/95143062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44005ba0 con 0x7faa34003f80 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 -- 172.21.15.104:0/95143062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa34003f80 msgr2=0x7faa34004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 --2- 172.21.15.104:0/95143062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa34003f80 0x7faa34004850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa4806eb20 tx=0x7faa34005880 comp rx=0 tx=0).stop 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4e2f3700 1 -- 172.21.15.104:0/95143062 reap_dead start 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 -- 172.21.15.104:0/95143062 shutdown_connections 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 -- 172.21.15.104:0/95143062 >> 172.21.15.104:0/95143062 conn(0x7faa4806b8a0 msgr2=0x7faa480cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 -- 172.21.15.104:0/95143062 shutdown_connections 2023-06-10T18:45:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.996+0000 7faa4fd56700 1 -- 172.21.15.104:0/95143062 wait complete. 2023-06-10T18:45:32.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 Processor -- start 2023-06-10T18:45:32.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 -- start start 2023-06-10T18:45:32.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 0x7faa481e4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa48149860 0x7faa48149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa48147570 con 0x7faa481e6430 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4814a350 con 0x7faa48149860 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4fd56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4814a630 con 0x7faa34003f80 2023-06-10T18:45:32.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4d2f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4d2f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57032/0 (socket says 172.21.15.104:57032) 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.998+0000 7faa4d2f1700 1 -- 172.21.15.104:0/2612809269 learned_addr learned my addr 172.21.15.104:0/2612809269 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4e2f3700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa48149860 0x7faa48149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4daf2700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 0x7faa481e4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 -- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 msgr2=0x7faa481e4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 0x7faa481e4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 -- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa48149860 msgr2=0x7faa48149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa48149860 0x7faa48149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 -- 172.21.15.104:0/2612809269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa481e5c30 con 0x7faa481e6430 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4daf2700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 0x7faa481e4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4d2f1700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faa44011000 tx=0x7faa44000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440070d0 con 0x7faa481e6430 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44007250 con 0x7faa481e6430 2023-06-10T18:45:32.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa481e5f10 con 0x7faa481e6430 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44007600 con 0x7faa481e6430 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:31.999+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48158fe0 con 0x7faa481e6430 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.000+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48074050 con 0x7faa481e6430 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.000+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faa4401e430 con 0x7faa481e6430 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.001+0000 7faa3e7fc700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 0x7faa2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:32.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.001+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6519+0+0 (secure 0 0 0) 0x7faa4408a9a0 con 0x7faa481e6430 2023-06-10T18:45:32.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.001+0000 7faa4daf2700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 0x7faa2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:32.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.001+0000 7faa4daf2700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 0x7faa2c06d190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa480cb910 tx=0x7faa38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:32.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.005+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faa44055910 con 0x7faa481e6430 2023-06-10T18:45:32.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.190+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa4806dc00 con 0x7faa481e6430 2023-06-10T18:45:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.848+0000 7faa3e7fc700 1 -- 172.21.15.104:0/2612809269 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v32) v1 ==== 169+0+0 (secure 0 0 0) 0x7faa44058f30 con 0x7faa481e6430 2023-06-10T18:45:32.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.854+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 msgr2=0x7faa2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.854+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 0x7faa2c06d190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa480cb910 tx=0x7faa38008040 comp rx=0 tx=0).stop 2023-06-10T18:45:32.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.854+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 msgr2=0x7faa481e68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:32.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.854+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faa44011000 tx=0x7faa44000fe0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 shutdown_connections 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa2c06ace0 0x7faa2c06d190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa48149860 0x7faa48149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa34003f80 0x7faa481e4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 --2- 172.21.15.104:0/2612809269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa481e6430 0x7faa481e68a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 >> 172.21.15.104:0/2612809269 conn(0x7faa4806b8a0 msgr2=0x7faa4815a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 shutdown_connections 2023-06-10T18:45:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:32.858+0000 7faa4fd56700 1 -- 172.21.15.104:0/2612809269 wait complete. 2023-06-10T18:45:32.865 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:45:32.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-06-10T18:45:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.007+0000 7f88cd3cc700 1 Processor -- start 2023-06-10T18:45:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.007+0000 7f88cd3cc700 1 -- start start 2023-06-10T18:45:33.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88cd3cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88cd3cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88c80d25c0 0x7f88c80d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88cd3cc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80d0e00 con 0x7f88c806b510 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88cd3cc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80af530 con 0x7f88c80d25c0 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88cd3cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80af810 con 0x7f88c80d04f0 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88c80d25c0 0x7f88c80d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f88c80d25c0 0x7f88c80d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57040/0 (socket says 172.21.15.104:57040) 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 -- 172.21.15.104:0/2680292891 learned_addr learned my addr 172.21.15.104:0/2680292891 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c6ffd700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 -- 172.21.15.104:0/2680292891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 msgr2=0x7f88c80d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 -- 172.21.15.104:0/2680292891 >> v1:172.21.15.104:6789/0 conn(0x7f88c806b510 legacy=0x7f88c806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c5ffb700 1 -- 172.21.15.104:0/2680292891 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296885703 0 0) 0x7f88c80d0e00 con 0x7f88c806b510 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.008+0000 7f88c67fc700 1 -- 172.21.15.104:0/2680292891 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80afaf0 con 0x7f88c80d25c0 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88c67fc700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c80d25c0 0x7f88c80d08c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f88b00098f0 tx=0x7f88b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975f5c928df220a9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88c5ffb700 1 -- 172.21.15.104:0/2680292891 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0011830 con 0x7f88c80d25c0 2023-06-10T18:45:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88c5ffb700 1 -- 172.21.15.104:0/2680292891 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b0011990 con 0x7f88c80d25c0 2023-06-10T18:45:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88c5ffb700 1 -- 172.21.15.104:0/2680292891 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0011d20 con 0x7f88c80d25c0 2023-06-10T18:45:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 -- 172.21.15.104:0/2680292891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c80d25c0 msgr2=0x7f88c80d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c80d25c0 0x7f88c80d08c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f88b00098f0 tx=0x7f88b0009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 -- 172.21.15.104:0/2680292891 shutdown_connections 2023-06-10T18:45:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/2680292891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c80d25c0 0x7f88c80d08c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 -- 172.21.15.104:0/2680292891 >> 172.21.15.104:0/2680292891 conn(0x7f88c8071ec0 msgr2=0x7f88c809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 -- 172.21.15.104:0/2680292891 shutdown_connections 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.009+0000 7f88cd3cc700 1 -- 172.21.15.104:0/2680292891 wait complete. 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.011+0000 7f88cd3cc700 1 Processor -- start 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.011+0000 7f88cd3cc700 1 -- start start 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 0x7f88c8157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b5750 con 0x7f88c806b510 2023-06-10T18:45:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8156730 con 0x7f88c80d25c0 2023-06-10T18:45:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b39f0 con 0x7f88c80d04f0 2023-06-10T18:45:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57044/0 (socket says 172.21.15.104:57044) 2023-06-10T18:45:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 -- 172.21.15.104:0/315749339 learned_addr learned my addr 172.21.15.104:0/315749339 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c67fc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c77fe700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 0x7f88c8157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 -- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 msgr2=0x7f88c80b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 -- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 msgr2=0x7f88c8157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 0x7f88c8157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 -- 172.21.15.104:0/315749339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b0009550 con 0x7f88c806b510 2023-06-10T18:45:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c77fe700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 0x7f88c8157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88c6ffd700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f88c000e990 tx=0x7f88c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00096f0 con 0x7f88c806b510 2023-06-10T18:45:33.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c000c7f0 con 0x7f88c806b510 2023-06-10T18:45:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c80b36c0 con 0x7f88c806b510 2023-06-10T18:45:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000eee0 con 0x7f88c806b510 2023-06-10T18:45:33.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.012+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c80b3090 con 0x7f88c806b510 2023-06-10T18:45:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.013+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8067010 con 0x7f88c806b510 2023-06-10T18:45:33.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.014+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f88c000cb60 con 0x7f88c806b510 2023-06-10T18:45:33.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.014+0000 7f88bf7fe700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 0x7f88a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:33.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.014+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f88c00897b0 con 0x7f88c806b510 2023-06-10T18:45:33.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.014+0000 7f88c67fc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 0x7f88a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:33.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.015+0000 7f88c67fc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 0x7f88a806d190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88b0014040 tx=0x7f88b0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:33.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.018+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f88c0058390 con 0x7f88c806b510 2023-06-10T18:45:33.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.207+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88c8074050 con 0x7f88c806b510 2023-06-10T18:45:33.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.855+0000 7f88bf7fe700 1 -- 172.21.15.104:0/315749339 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v33) v1 ==== 178+0+0 (secure 0 0 0) 0x7f88c0057f20 con 0x7f88c806b510 2023-06-10T18:45:33.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 msgr2=0x7f88a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 0x7f88a806d190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88b0014040 tx=0x7f88b0008030 comp rx=0 tx=0).stop 2023-06-10T18:45:33.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 msgr2=0x7f88c81561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:33.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f88c000e990 tx=0x7f88c000eca0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 shutdown_connections 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f88a806ace0 0x7f88a806d190 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88c80d25c0 0x7f88c8157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88c806b510 0x7f88c81561f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 --2- 172.21.15.104:0/315749339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88c80d04f0 0x7f88c80b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 >> 172.21.15.104:0/315749339 conn(0x7f88c8071ec0 msgr2=0x7f88c80b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:33.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 shutdown_connections 2023-06-10T18:45:33.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:33.858+0000 7f88cd3cc700 1 -- 172.21.15.104:0/315749339 wait complete. 2023-06-10T18:45:33.866 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data_ec' removed 2023-06-10T18:45:33.877 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:45:33.878 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:45:33.878 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:45:33.878 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:33.878 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:45:33.879 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:45:33.879 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:45:33.879 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:33.879 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:45:33.880 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:45:33.880 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:45:33.880 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:33.880 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:45:33.880 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:45:33.881 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:45:33.881 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:45:33.888 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:45:33.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:45:33.892 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:45:33.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:45:33.894 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:45:33.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:45:33.895 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:45:33.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:45:33.931 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:45:33.932 DEBUG:teuthology.parallel:result is None 2023-06-10T18:45:33.933 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:45:33.934 DEBUG:teuthology.parallel:result is None 2023-06-10T18:45:33.936 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:45:33.937 DEBUG:teuthology.parallel:result is None 2023-06-10T18:45:33.937 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:45:33.937 DEBUG:teuthology.parallel:result is None 2023-06-10T18:45:33.938 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:45:34.010 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:45:34.010 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:45:34.023 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:45:34.026 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:45:34.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.068+0000 7fe7e2fa3700 1 Processor -- start 2023-06-10T18:45:34.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.068+0000 7fe7e2fa3700 1 -- start start 2023-06-10T18:45:34.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e2fa3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 0x7fe7dc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e2fa3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 0x7fe7dc06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e2fa3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc0c9e60 con 0x7fe7dc0aeaa0 2023-06-10T18:45:34.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e2fa3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7dc0c9bc0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e2fa3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc0aee70 con 0x7fe7dc0cd850 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e1540700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 0x7fe7dc06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 0x7fe7dc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e0d3f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe7dc0aeaa0 0x7fe7dc0cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60750/0 (socket says 172.21.15.104:60750) 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e0d3f700 1 -- 172.21.15.104:0/1233274695 learned_addr learned my addr 172.21.15.104:0/1233274695 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 -- 172.21.15.104:0/1233274695 >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 msgr2=0x7fe7dc06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 --2- 172.21.15.104:0/1233274695 >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 0x7fe7dc06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:34.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 -- 172.21.15.104:0/1233274695 >> v1:172.21.15.114:6789/0 conn(0x7fe7dc0aeaa0 legacy=0x7fe7dc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7db7fe700 1 -- 172.21.15.104:0/1233274695 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187593215 0 0) 0x7fe7dc0c9e60 con 0x7fe7dc0aeaa0 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 -- 172.21.15.104:0/1233274695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc06e540 con 0x7fe7dc0cd850 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7e1540700 1 --2- 172.21.15.104:0/1233274695 >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 0x7fe7dc06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7dbfff700 1 --2- 172.21.15.104:0/1233274695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 0x7fe7dc06cd50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009840 tx=0x7fe7c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f078db72db3f110b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7db7fe700 1 -- 172.21.15.104:0/1233274695 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4011830 con 0x7fe7dc0cd850 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7db7fe700 1 -- 172.21.15.104:0/1233274695 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c4011990 con 0x7fe7dc0cd850 2023-06-10T18:45:34.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.069+0000 7fe7db7fe700 1 -- 172.21.15.104:0/1233274695 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4011d20 con 0x7fe7dc0cd850 2023-06-10T18:45:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/1233274695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 msgr2=0x7fe7dc06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:34.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/1233274695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 0x7fe7dc06cd50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009840 tx=0x7fe7c4009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/1233274695 shutdown_connections 2023-06-10T18:45:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/1233274695 >> v2:172.21.15.114:3300/0 conn(0x7fe7dc0cb300 0x7fe7dc06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/1233274695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0cd850 0x7fe7dc06cd50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/1233274695 >> 172.21.15.104:0/1233274695 conn(0x7fe7dc06b8a0 msgr2=0x7fe7dc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:34.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/1233274695 shutdown_connections 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.070+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/1233274695 wait complete. 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.071+0000 7fe7e2fa3700 1 Processor -- start 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.071+0000 7fe7e2fa3700 1 -- start start 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0aeaa0 0x7fe7dc156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7dc0cd850 0x7fe7dc1e3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1572e0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc159510 con 0x7fe7dc0cd850 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e2fa3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc1597f0 con 0x7fe7dc0aeaa0 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57084/0 (socket says 172.21.15.104:57084) 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7dbfff700 1 -- 172.21.15.104:0/2264924827 learned_addr learned my addr 172.21.15.104:0/2264924827 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e1540700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7dc0cd850 0x7fe7dc1e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.072+0000 7fe7e0d3f700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0aeaa0 0x7fe7dc156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 -- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0aeaa0 msgr2=0x7fe7dc156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0aeaa0 0x7fe7dc156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 -- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7dc0cd850 msgr2=0x7fe7dc1e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7dc0cd850 0x7fe7dc1e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 -- 172.21.15.104:0/2264924827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d4009a70 con 0x7fe7dc0cb300 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7dbfff700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40090b0 tx=0x7fe7c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40045d0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c401e070 con 0x7fe7dc0cb300 2023-06-10T18:45:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c40094a0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40038e0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.073+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc159db0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.074+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7dc0ce890 con 0x7fe7dc0cb300 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.074+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe7c4003a40 con 0x7fe7dc0cb300 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.075+0000 7fe7d97fa700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 0x7fe7bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.075+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fe7c408a2c0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.075+0000 7fe7e0d3f700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 0x7fe7bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.075+0000 7fe7e0d3f700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 0x7fe7bc06d190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc000fb0 tx=0x7fe7cc007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:34.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.079+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe7c40555c0 con 0x7fe7dc0cb300 2023-06-10T18:45:34.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:34.281+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe7dc074050 con 0x7fe7dc0cb300 2023-06-10T18:45:35.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.059+0000 7fe7d97fa700 1 -- 172.21.15.104:0/2264924827 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe7c4058be0 con 0x7fe7dc0cb300 2023-06-10T18:45:35.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 msgr2=0x7fe7bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 0x7fe7bc06d190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc000fb0 tx=0x7fe7cc007d70 comp rx=0 tx=0).stop 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 msgr2=0x7fe7dc0c6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40090b0 tx=0x7fe7c4004750 comp rx=0 tx=0).stop 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 shutdown_connections 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe7bc06ace0 0x7fe7bc06d190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7dc0cd850 0x7fe7dc1e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7dc0aeaa0 0x7fe7dc156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 --2- 172.21.15.104:0/2264924827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7dc0cb300 0x7fe7dc0c6f20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 >> 172.21.15.104:0/2264924827 conn(0x7fe7dc06b8a0 msgr2=0x7fe7dc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:35.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 shutdown_connections 2023-06-10T18:45:35.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.062+0000 7fe7e2fa3700 1 -- 172.21.15.104:0/2264924827 wait complete. 2023-06-10T18:45:35.069 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:45:35.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:45:35.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.201+0000 7f173ec96700 1 Processor -- start 2023-06-10T18:45:35.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.201+0000 7f173ec96700 1 -- start start 2023-06-10T18:45:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ec96700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ec96700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f17380cd680 0x7f173806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ec96700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17380cdae0 con 0x7f17380b1080 2023-06-10T18:45:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ec96700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17380c9bc0 con 0x7f17380cd680 2023-06-10T18:45:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ec96700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17380c9e60 con 0x7f17380aeaa0 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173d233700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f17380cd680 0x7f173806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f1737fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f17380b1080 0x7f17380ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54922/0 (socket says 172.21.15.104:54922) 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173d233700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f17380cd680 0x7f173806dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57098/0 (socket says 172.21.15.104:57098) 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f1737fff700 1 -- 172.21.15.104:0/4046787533 learned_addr learned my addr 172.21.15.104:0/4046787533 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173ca32700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173d233700 1 -- 172.21.15.104:0/4046787533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 msgr2=0x7f17380cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173d233700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f17377fe700 1 -- 172.21.15.104:0/4046787533 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056577446 0 0) 0x7f17380cdae0 con 0x7f17380b1080 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.202+0000 7f173d233700 1 -- 172.21.15.104:0/4046787533 >> v1:172.21.15.104:6789/0 conn(0x7f17380b1080 legacy=0x7f17380ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173d233700 1 -- 172.21.15.104:0/4046787533 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17380aee70 con 0x7f17380cd680 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ca32700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173d233700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380cd680 0x7f173806dd60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1730009d90 tx=0x7f173000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b073fd965c614ee server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f17377fe700 1 -- 172.21.15.104:0/4046787533 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173000e070 con 0x7f17380cd680 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f17377fe700 1 -- 172.21.15.104:0/4046787533 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173000bbd0 con 0x7f17380cd680 2023-06-10T18:45:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f17377fe700 1 -- 172.21.15.104:0/4046787533 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730012430 con 0x7f17380cd680 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 -- 172.21.15.104:0/4046787533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380cd680 msgr2=0x7f173806dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380cd680 0x7f173806dd60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1730009d90 tx=0x7f173000b330 comp rx=0 tx=0).stop 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 -- 172.21.15.104:0/4046787533 shutdown_connections 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380cd680 0x7f173806dd60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 --2- 172.21.15.104:0/4046787533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 -- 172.21.15.104:0/4046787533 >> 172.21.15.104:0/4046787533 conn(0x7f173806b8a0 msgr2=0x7f17380cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 -- 172.21.15.104:0/4046787533 shutdown_connections 2023-06-10T18:45:35.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.203+0000 7f173ec96700 1 -- 172.21.15.104:0/4046787533 wait complete. 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 Processor -- start 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 -- start start 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 0x7f17381644f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380ad8c0 con 0x7f17380b1080 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1738165050 con 0x7f17380cd680 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.205+0000 7f173ec96700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17380acde0 con 0x7f17380aeaa0 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57102/0 (socket says 172.21.15.104:57102) 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 -- 172.21.15.104:0/3136384176 learned_addr learned my addr 172.21.15.104:0/3136384176 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f173d233700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 0x7f17381644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f173ca32700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 -- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 msgr2=0x7f17380b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 -- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 msgr2=0x7f17381644f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 0x7f17381644f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 -- 172.21.15.104:0/3136384176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009a70 con 0x7f17380b1080 2023-06-10T18:45:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f173d233700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 0x7f17381644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f1737fff700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17240011e0 tx=0x7f172400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724004c80 con 0x7f17380b1080 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724004e00 con 0x7f17380b1080 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17380ad0c0 con 0x7f17380b1080 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724007310 con 0x7f17380b1080 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.206+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1738149a80 con 0x7f17380b1080 2023-06-10T18:45:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.207+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17380acc10 con 0x7f17380b1080 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.207+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1724007470 con 0x7f17380b1080 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.208+0000 7f17357fa700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 0x7f171c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.208+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f17240890a0 con 0x7f17380b1080 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.208+0000 7f173ca32700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 0x7f171c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.208+0000 7f173ca32700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 0x7f171c06d190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f172c00a970 tx=0x7f172c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.212+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1724054360 con 0x7f17380b1080 2023-06-10T18:45:35.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.474+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1738074050 con 0x7f17380b1080 2023-06-10T18:45:35.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.475+0000 7f17357fa700 1 -- 172.21.15.104:0/3136384176 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f1724057980 con 0x7f17380b1080 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:45:35.481 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:45:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:45:35.483 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:45:35.485 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:45:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.487 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:45:35.488 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.489 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.493 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:45:35.494 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:45:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:45:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.497 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:45:35.499 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.500 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.501 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:45:35.502 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.503 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.504 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-mds", 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRkk+1cKxAAfzG9F9KVp5YIofhf4GPmtA==", 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-mds" 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-mgr", 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRk7vVcKxAAy/UTYg2rPuN/FZrZT3Kt2w==", 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-mgr" 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-osd", 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRkX/9cKxAAaJo+DJHzhaQSvbNtdKriew==", 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-osd" 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rbd", 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRkpgZdKxAAt73pE82E3viuQ4yGF28DFA==", 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rbd" 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRksQ5dKxAA+pYZjp/0y1kpwZI0iwXJ/g==", 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.510 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.bootstrap-rgw", 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBfxIRkrBZdKxAAl27ES1+hM0x3PnkKbCjnCw==", 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile bootstrap-rgw" 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.511 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:45:35.513 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:35.514 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:45:35.514 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:45:35.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 msgr2=0x7f171c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 0x7f171c06d190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f172c00a970 tx=0x7f172c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 msgr2=0x7f1738163fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17240011e0 tx=0x7f172400cae0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 shutdown_connections 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f171c06ace0 0x7f171c06d190 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17380cd680 0x7f17381644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17380aeaa0 0x7f17380b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 --2- 172.21.15.104:0/3136384176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17380b1080 0x7f1738163fb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 >> 172.21.15.104:0/3136384176 conn(0x7f173806b8a0 msgr2=0x7f1738150700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:35.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 shutdown_connections 2023-06-10T18:45:35.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.477+0000 7f173ec96700 1 -- 172.21.15.104:0/3136384176 wait complete. 2023-06-10T18:45:35.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-06-10T18:45:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 Processor -- start 2023-06-10T18:45:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 -- start start 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ff80c93e0 0x7f1ff80717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff806b2e0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff815e150 con 0x7f1ff8071130 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.632+0000 7f1ffdf06700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff815db40 con 0x7f1ff80c93e0 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1ff815a0b0 0x7f1ff8070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54936/0 (socket says 172.21.15.104:54936) 2023-06-10T18:45:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff77fe700 1 -- 172.21.15.104:0/419955128 learned_addr learned my addr 172.21.15.104:0/419955128 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff7fff700 1 --2- 172.21.15.104:0/419955128 >> v2:172.21.15.114:3300/0 conn(0x7f1ff80c93e0 0x7f1ff80717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482474828 0 0) 0x7f1ff806b2e0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe40035b0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999615314 0 0) 0x7f1ff815e150 con 0x7f1ff8071130 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff806b2e0 con 0x7f1ff8071130 2023-06-10T18:45:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff7fff700 1 -- 172.21.15.104:0/419955128 >> v1:172.21.15.114:6789/0 conn(0x7f1ff8071130 legacy=0x7f1ff815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff7fff700 1 -- 172.21.15.104:0/419955128 >> v1:172.21.15.104:6789/0 conn(0x7f1ff815a0b0 legacy=0x7f1ff8070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 317372393 0 0) 0x7f1fe40035b0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff7fff700 1 -- 172.21.15.104:0/419955128 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff806b530 con 0x7f1ff80c93e0 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff7fff700 1 --2- 172.21.15.104:0/419955128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff80717d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009d90 tx=0x7f1ff000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7006bf2ad5c7fe96 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff000e070 con 0x7f1ff80c93e0 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0003a20 con 0x7f1ff80c93e0 2023-06-10T18:45:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.633+0000 7f1ff67fc700 1 -- 172.21.15.104:0/419955128 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8001670 con 0x7f1ff80c93e0 2023-06-10T18:45:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 -- 172.21.15.104:0/419955128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 msgr2=0x7f1ff80717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/419955128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff80717d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009d90 tx=0x7f1ff000b330 comp rx=0 tx=0).stop 2023-06-10T18:45:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 -- 172.21.15.104:0/419955128 shutdown_connections 2023-06-10T18:45:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/419955128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff80717d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 -- 172.21.15.104:0/419955128 >> 172.21.15.104:0/419955128 conn(0x7f1ff80b1610 msgr2=0x7f1ff80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 -- 172.21.15.104:0/419955128 shutdown_connections 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.634+0000 7f1ffdf06700 1 -- 172.21.15.104:0/419955128 wait complete. 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.635+0000 7f1ffdf06700 1 Processor -- start 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.635+0000 7f1ffdf06700 1 -- start start 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ff8071130 0x7f1ff80c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff814c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80c9210 con 0x7f1ff815a0b0 2023-06-10T18:45:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80b1c80 con 0x7f1ff80c93e0 2023-06-10T18:45:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ffdf06700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff815cbd0 con 0x7f1ff8071130 2023-06-10T18:45:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57118/0 (socket says 172.21.15.104:57118) 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 -- 172.21.15.104:0/2194448714 learned_addr learned my addr 172.21.15.104:0/2194448714 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff77fe700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ff8071130 0x7f1ff80c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 -- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ff8071130 msgr2=0x7f1ff80c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ff8071130 0x7f1ff80c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 -- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 msgr2=0x7f1ff814c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff6ffd700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff814c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff814c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.636+0000 7f1ff7fff700 1 -- 172.21.15.104:0/2194448714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0009a70 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1ff7fff700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004000 tx=0x7f1ff0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00045c0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0004720 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff001d470 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff815ceb0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.637+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff815d470 con 0x7f1ff815a0b0 2023-06-10T18:45:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.638+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff80b1ea0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.638+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1ff001d5d0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.639+0000 7f1feffff700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 0x7f1fd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:35.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.639+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f1ff008af20 con 0x7f1ff815a0b0 2023-06-10T18:45:35.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.639+0000 7f1ff77fe700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 0x7f1fd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:35.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.639+0000 7f1ff77fe700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 0x7f1fd806d190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000fd0 tx=0x7f1fe8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:35.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.643+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1ff0059d00 con 0x7f1ff815a0b0 2023-06-10T18:45:35.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.905+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1ff815e5c0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.908+0000 7f1feffff700 1 -- 172.21.15.104:0/2194448714 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1ff815e5c0 con 0x7f1ff815a0b0 2023-06-10T18:45:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 msgr2=0x7f1fd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 0x7f1fd806d190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000fd0 tx=0x7f1fe8009040 comp rx=0 tx=0).stop 2023-06-10T18:45:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 msgr2=0x7f1ff80c6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004000 tx=0x7f1ff0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 shutdown_connections 2023-06-10T18:45:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ff815a0b0 0x7f1ff80c6990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ff8071130 0x7f1ff80c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1fd806ace0 0x7f1fd806d190 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 --2- 172.21.15.104:0/2194448714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff80c93e0 0x7f1ff814c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 >> 172.21.15.104:0/2194448714 conn(0x7f1ff80b1610 msgr2=0x7f1ff80ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 shutdown_connections 2023-06-10T18:45:35.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:35.910+0000 7f1ffdf06700 1 -- 172.21.15.104:0/2194448714 wait complete. 2023-06-10T18:45:35.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-06-10T18:45:36.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.043+0000 7feb4689e700 1 Processor -- start 2023-06-10T18:45:36.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.043+0000 7feb4689e700 1 -- start start 2023-06-10T18:45:36.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb4689e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb4015c730 0x7feb4006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb4689e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400cc000 0x7feb400cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb4689e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4015cb00 con 0x7feb4006cd70 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb4689e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb400aed80 con 0x7feb4015c730 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb4689e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400c9c60 con 0x7feb400cc000 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb44e3b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feb4006cd70 0x7feb4006d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54954/0 (socket says 172.21.15.104:54954) 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb44e3b700 1 -- 172.21.15.104:0/2475574886 learned_addr learned my addr 172.21.15.104:0/2475574886 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:36.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400cc000 0x7feb400cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> v2:172.21.15.114:3300/0 conn(0x7feb4015c730 0x7feb4006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310424819 0 0) 0x7feb38001490 con 0x7feb4006cd70 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c0035b0 con 0x7feb4006cd70 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848487692 0 0) 0x7feb2c0035b0 con 0x7feb4006cd70 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400cc000 msgr2=0x7feb400cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400cc000 0x7feb400cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 >> v2:172.21.15.114:3300/0 conn(0x7feb4015c730 msgr2=0x7feb4006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 --2- 172.21.15.104:0/2475574886 >> v2:172.21.15.114:3300/0 conn(0x7feb4015c730 0x7feb4006e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4006ecd0 con 0x7feb4006cd70 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7feb38001b80 con 0x7feb4006cd70 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.044+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 >> v1:172.21.15.104:6789/0 conn(0x7feb4006cd70 legacy=0x7feb4006d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb2c003a00 0x7feb2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c004430 0x7feb2c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2c005000 0x7feb2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400aed80 con 0x7feb2c003a00 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400c9c60 con 0x7feb2c004430 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c00bf50 con 0x7feb2c005000 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400cc000 0x7feb400cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> v2:172.21.15.114:3300/0 conn(0x7feb4015c730 0x7feb4006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:36.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb44e3b700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2c005000 0x7feb2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb2c003a00 0x7feb2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c004430 0x7feb2c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 -- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2c005000 msgr2=0x7feb2c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2c005000 0x7feb2c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 -- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c004430 msgr2=0x7feb2c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c004430 0x7feb2c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 -- 172.21.15.104:0/2475574886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0035b0 con 0x7feb2c003a00 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb2c003a00 0x7feb2c003e10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb28009cf0 tx=0x7feb2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5d5bca98f8ec9f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28004030 con 0x7feb2c003a00 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28004190 con 0x7feb2c003a00 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.045+0000 7feb3effd700 1 -- 172.21.15.104:0/2475574886 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28004500 con 0x7feb2c003a00 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 -- 172.21.15.104:0/2475574886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb2c003a00 msgr2=0x7feb2c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 --2- 172.21.15.104:0/2475574886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb2c003a00 0x7feb2c003e10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb28009cf0 tx=0x7feb2c005880 comp rx=0 tx=0).stop 2023-06-10T18:45:36.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb44e3b700 1 -- 172.21.15.104:0/2475574886 reap_dead start 2023-06-10T18:45:36.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 -- 172.21.15.104:0/2475574886 shutdown_connections 2023-06-10T18:45:36.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 -- 172.21.15.104:0/2475574886 >> 172.21.15.104:0/2475574886 conn(0x7feb4006b8a0 msgr2=0x7feb400cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:36.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 -- 172.21.15.104:0/2475574886 shutdown_connections 2023-06-10T18:45:36.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.046+0000 7feb4689e700 1 -- 172.21.15.104:0/2475574886 wait complete. 2023-06-10T18:45:36.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.047+0000 7feb4689e700 1 Processor -- start 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.047+0000 7feb4689e700 1 -- start start 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c003a00 0x7feb400af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 0x7feb400b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40148b90 con 0x7feb40148620 2023-06-10T18:45:36.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40148e70 con 0x7feb2c003a00 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb4689e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb40153400 con 0x7feb400b5000 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57136/0 (socket says 172.21.15.104:57136) 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 -- 172.21.15.104:0/2346722747 learned_addr learned my addr 172.21.15.104:0/2346722747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c003a00 0x7feb400af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb44e3b700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 0x7feb400b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 -- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 msgr2=0x7feb400b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 0x7feb400b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 -- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c003a00 msgr2=0x7feb400af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c003a00 0x7feb400af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 -- 172.21.15.104:0/2346722747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb401536e0 con 0x7feb40148620 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb44e3b700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 0x7feb400b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:36.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.048+0000 7feb3f7fe700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7feb30011c00 tx=0x7feb30011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3000fa40 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3000fba0 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb401539c0 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30014440 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb400bc090 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.049+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb40153d30 con 0x7feb40148620 2023-06-10T18:45:36.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.050+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feb300145a0 con 0x7feb40148620 2023-06-10T18:45:36.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.050+0000 7feb3cff9700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 0x7feb2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.050+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7feb30089b50 con 0x7feb40148620 2023-06-10T18:45:36.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.051+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 0x7feb2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.051+0000 7feb3ffff700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 0x7feb2006d190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb280095e0 tx=0x7feb28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.055+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feb30054d70 con 0x7feb40148620 2023-06-10T18:45:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.319+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7feb40070e00 con 0x7feb40148620 2023-06-10T18:45:36.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.322+0000 7feb3cff9700 1 -- 172.21.15.104:0/2346722747 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7feb300220a0 con 0x7feb40148620 2023-06-10T18:45:36.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 msgr2=0x7feb2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 0x7feb2006d190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb280095e0 tx=0x7feb28008030 comp rx=0 tx=0).stop 2023-06-10T18:45:36.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 msgr2=0x7feb400b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7feb30011c00 tx=0x7feb30011f10 comp rx=0 tx=0).stop 2023-06-10T18:45:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 shutdown_connections 2023-06-10T18:45:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feb2006ace0 0x7feb2006d190 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2c003a00 0x7feb400af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb400b5000 0x7feb400b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 --2- 172.21.15.104:0/2346722747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb40148620 0x7feb400b4ac0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.324+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 >> 172.21.15.104:0/2346722747 conn(0x7feb4006b8a0 msgr2=0x7feb400ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:36.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.325+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 shutdown_connections 2023-06-10T18:45:36.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.325+0000 7feb4689e700 1 -- 172.21.15.104:0/2346722747 wait complete. 2023-06-10T18:45:36.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-06-10T18:45:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.458+0000 7f08c1214700 1 Processor -- start 2023-06-10T18:45:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.458+0000 7f08c1214700 1 -- start start 2023-06-10T18:45:36.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08c1214700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 0x7f08bc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08c1214700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08bc069e70 0x7f08bc0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08c1214700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc06d290 con 0x7f08bc06e4b0 2023-06-10T18:45:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08c1214700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc071640 con 0x7f08bc069e70 2023-06-10T18:45:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08c1214700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0aef00 con 0x7f08bc06e0e0 2023-06-10T18:45:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08bc069e70 0x7f08bc0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08ba59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f08bc06e4b0 0x7f08bc0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54972/0 (socket says 172.21.15.104:54972) 2023-06-10T18:45:36.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08ba59c700 1 -- 172.21.15.104:0/733010384 learned_addr learned my addr 172.21.15.104:0/733010384 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bad9d700 1 --2- 172.21.15.104:0/733010384 >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 0x7f08bc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 -- 172.21.15.104:0/733010384 >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 msgr2=0x7f08bc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 --2- 172.21.15.104:0/733010384 >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 0x7f08bc06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 -- 172.21.15.104:0/733010384 >> v1:172.21.15.104:6789/0 conn(0x7f08bc06e4b0 legacy=0x7f08bc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08b9d9b700 1 -- 172.21.15.104:0/733010384 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740922437 0 0) 0x7f08bc06d290 con 0x7f08bc06e4b0 2023-06-10T18:45:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 -- 172.21.15.104:0/733010384 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0ca2d0 con 0x7f08bc069e70 2023-06-10T18:45:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bad9d700 1 --2- 172.21.15.104:0/733010384 >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 0x7f08bc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:36.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08bb59e700 1 --2- 172.21.15.104:0/733010384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc069e70 0x7f08bc0ce9f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08bc06af50 tx=0x7f08b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5a629686973938 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08b9d9b700 1 -- 172.21.15.104:0/733010384 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400e070 con 0x7f08bc069e70 2023-06-10T18:45:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08b9d9b700 1 -- 172.21.15.104:0/733010384 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b400bbd0 con 0x7f08bc069e70 2023-06-10T18:45:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.459+0000 7f08b9d9b700 1 -- 172.21.15.104:0/733010384 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4012430 con 0x7f08bc069e70 2023-06-10T18:45:36.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 -- 172.21.15.104:0/733010384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc069e70 msgr2=0x7f08bc0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 --2- 172.21.15.104:0/733010384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc069e70 0x7f08bc0ce9f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08bc06af50 tx=0x7f08b400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 -- 172.21.15.104:0/733010384 shutdown_connections 2023-06-10T18:45:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 --2- 172.21.15.104:0/733010384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc069e70 0x7f08bc0ce9f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 --2- 172.21.15.104:0/733010384 >> v2:172.21.15.114:3300/0 conn(0x7f08bc06e0e0 0x7f08bc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 -- 172.21.15.104:0/733010384 >> 172.21.15.104:0/733010384 conn(0x7f08bc06b8a0 msgr2=0x7f08bc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 -- 172.21.15.104:0/733010384 shutdown_connections 2023-06-10T18:45:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.460+0000 7f08c1214700 1 -- 172.21.15.104:0/733010384 wait complete. 2023-06-10T18:45:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.461+0000 7f08c1214700 1 Processor -- start 2023-06-10T18:45:36.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.461+0000 7f08c1214700 1 -- start start 2023-06-10T18:45:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08bc069e70 0x7f08bc1f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08bc06e0e0 0x7f08bc1f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0b3360 con 0x7f08bc06e4b0 2023-06-10T18:45:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0b3120 con 0x7f08bc069e70 2023-06-10T18:45:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08c1214700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc149510 con 0x7f08bc06e0e0 2023-06-10T18:45:36.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57152/0 (socket says 172.21.15.104:57152) 2023-06-10T18:45:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 -- 172.21.15.104:0/3905353232 learned_addr learned my addr 172.21.15.104:0/3905353232 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:36.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bad9d700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08bc069e70 0x7f08bc1f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08ba59c700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08bc06e0e0 0x7f08bc1f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 -- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08bc06e0e0 msgr2=0x7f08bc1f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08bc06e0e0 0x7f08bc1f5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 -- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08bc069e70 msgr2=0x7f08bc1f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.462+0000 7f08bb59e700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08bc069e70 0x7f08bc1f4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08bb59e700 1 -- 172.21.15.104:0/3905353232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4009a70 con 0x7f08bc06e4b0 2023-06-10T18:45:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08bb59e700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f08b4014040 tx=0x7f08b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4012430 con 0x7f08bc06e4b0 2023-06-10T18:45:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b4020070 con 0x7f08bc06e4b0 2023-06-10T18:45:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc0afb90 con 0x7f08bc06e4b0 2023-06-10T18:45:36.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4004280 con 0x7f08bc06e4b0 2023-06-10T18:45:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.463+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc0b0150 con 0x7f08bc06e4b0 2023-06-10T18:45:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.464+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc074050 con 0x7f08bc06e4b0 2023-06-10T18:45:36.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.464+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f08b40043e0 con 0x7f08bc06e4b0 2023-06-10T18:45:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.465+0000 7f08a37fe700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 0x7f08a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.465+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f08b4022070 con 0x7f08bc06e4b0 2023-06-10T18:45:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.465+0000 7f08bad9d700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 0x7f08a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.465+0000 7f08bad9d700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 0x7f08a806d190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00b840 tx=0x7f08ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.469+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f08b4056830 con 0x7f08bc06e4b0 2023-06-10T18:45:36.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.733+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f08bc0723b0 con 0x7f08bc06e4b0 2023-06-10T18:45:36.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.740+0000 7f08a37fe700 1 -- 172.21.15.104:0/3905353232 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f08b4059e50 con 0x7f08bc06e4b0 2023-06-10T18:45:36.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 msgr2=0x7f08a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 0x7f08a806d190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00b840 tx=0x7f08ac008040 comp rx=0 tx=0).stop 2023-06-10T18:45:36.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 msgr2=0x7f08bc0b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f08b4014040 tx=0x7f08b4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 shutdown_connections 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08a806ace0 0x7f08a806d190 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08bc069e70 0x7f08bc1f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08bc06e4b0 0x7f08bc0b2be0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 --2- 172.21.15.104:0/3905353232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08bc06e0e0 0x7f08bc1f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 >> 172.21.15.104:0/3905353232 conn(0x7f08bc06b8a0 msgr2=0x7f08bc0ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:36.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 shutdown_connections 2023-06-10T18:45:36.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.742+0000 7f08c1214700 1 -- 172.21.15.104:0/3905353232 wait complete. 2023-06-10T18:45:36.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-06-10T18:45:36.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.876+0000 7f088acb3700 1 Processor -- start 2023-06-10T18:45:36.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.876+0000 7f088acb3700 1 -- start start 2023-06-10T18:45:36.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f088acb3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08840cfa00 0x7f08840701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f088acb3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f088415a250 0x7f08840cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f088acb3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08840ccca0 con 0x7f08840cad90 2023-06-10T18:45:36.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f088acb3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840d0310 con 0x7f08840cfa00 2023-06-10T18:45:36.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f088acb3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840cee50 con 0x7f088415a250 2023-06-10T18:45:36.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08840cfa00 0x7f08840701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08840cfa00 0x7f08840701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57156/0 (socket says 172.21.15.104:57156) 2023-06-10T18:45:36.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 -- 172.21.15.104:0/811141066 learned_addr learned my addr 172.21.15.104:0/811141066 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:36.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0883fff700 1 --2- 172.21.15.104:0/811141066 >> v2:172.21.15.114:3300/0 conn(0x7f088415a250 0x7f08840cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 -- 172.21.15.104:0/811141066 >> v2:172.21.15.114:3300/0 conn(0x7f088415a250 msgr2=0x7f08840cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:36.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 --2- 172.21.15.104:0/811141066 >> v2:172.21.15.114:3300/0 conn(0x7f088415a250 0x7f08840cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 -- 172.21.15.104:0/811141066 >> v1:172.21.15.114:6789/0 conn(0x7f08840cad90 legacy=0x7f08840b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 -- 172.21.15.104:0/811141066 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840cf130 con 0x7f08840cfa00 2023-06-10T18:45:36.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f0888a4f700 1 --2- 172.21.15.104:0/811141066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08840cfa00 0x7f08840701e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0884071ed0 tx=0x7f0878009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9528afb6efb2a6a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.877+0000 7f08837fe700 1 -- 172.21.15.104:0/811141066 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800b870 con 0x7f08840cfa00 2023-06-10T18:45:36.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f08837fe700 1 -- 172.21.15.104:0/811141066 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087800b9d0 con 0x7f08840cfa00 2023-06-10T18:45:36.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f08837fe700 1 -- 172.21.15.104:0/811141066 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800bd60 con 0x7f08840cfa00 2023-06-10T18:45:36.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 -- 172.21.15.104:0/811141066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08840cfa00 msgr2=0x7f08840701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 --2- 172.21.15.104:0/811141066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08840cfa00 0x7f08840701e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0884071ed0 tx=0x7f0878009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 -- 172.21.15.104:0/811141066 shutdown_connections 2023-06-10T18:45:36.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 --2- 172.21.15.104:0/811141066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08840cfa00 0x7f08840701e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 --2- 172.21.15.104:0/811141066 >> v2:172.21.15.114:3300/0 conn(0x7f088415a250 0x7f08840cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 -- 172.21.15.104:0/811141066 >> 172.21.15.104:0/811141066 conn(0x7f088406e370 msgr2=0x7f088406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:36.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 -- 172.21.15.104:0/811141066 shutdown_connections 2023-06-10T18:45:36.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.878+0000 7f088acb3700 1 -- 172.21.15.104:0/811141066 wait complete. 2023-06-10T18:45:36.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.879+0000 7f088acb3700 1 Processor -- start 2023-06-10T18:45:36.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.879+0000 7f088acb3700 1 -- start start 2023-06-10T18:45:36.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08840cad90 0x7f08841f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 0x7f08840b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b1790 con 0x7f088415a250 2023-06-10T18:45:36.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b0b80 con 0x7f08840cfa00 2023-06-10T18:45:36.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f088acb3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08840b0e60 con 0x7f08840cad90 2023-06-10T18:45:36.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57166/0 (socket says 172.21.15.104:57166) 2023-06-10T18:45:36.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 -- 172.21.15.104:0/2119396244 learned_addr learned my addr 172.21.15.104:0/2119396244 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0883fff700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 0x7f08840b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 -- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08840cad90 msgr2=0x7f08841f2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08840cad90 0x7f08841f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.880+0000 7f0889250700 1 -- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 msgr2=0x7f08840b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f0889250700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 0x7f08840b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f0889250700 1 -- 172.21.15.104:0/2119396244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878009550 con 0x7f088415a250 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f0883fff700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 0x7f08840b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f0889250700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f087c00cb50 tx=0x7f087c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c003c00 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c003d60 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08840b1140 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c00ab70 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.881+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08841f3290 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.882+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0884074050 con 0x7f088415a250 2023-06-10T18:45:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.882+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f087c00acd0 con 0x7f088415a250 2023-06-10T18:45:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.883+0000 7f08817fa700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 0x7f086806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.883+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f087c0892d0 con 0x7f088415a250 2023-06-10T18:45:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.883+0000 7f0888a4f700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 0x7f086806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.883+0000 7f0888a4f700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 0x7f086806d190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0878009c10 tx=0x7f0878010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:36.887+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f087c054590 con 0x7f088415a250 2023-06-10T18:45:37.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.149+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f08840b18d0 con 0x7f088415a250 2023-06-10T18:45:37.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.155+0000 7f08817fa700 1 -- 172.21.15.104:0/2119396244 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f08840b18d0 con 0x7f088415a250 2023-06-10T18:45:37.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 msgr2=0x7f086806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 0x7f086806d190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0878009c10 tx=0x7f0878010000 comp rx=0 tx=0).stop 2023-06-10T18:45:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 msgr2=0x7f088415c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f087c00cb50 tx=0x7f087c00cf10 comp rx=0 tx=0).stop 2023-06-10T18:45:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 shutdown_connections 2023-06-10T18:45:37.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f088415a250 0x7f088415c730 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08840cad90 0x7f08841f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f086806ace0 0x7f086806d190 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 --2- 172.21.15.104:0/2119396244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08840cfa00 0x7f08840b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 >> 172.21.15.104:0/2119396244 conn(0x7f088406e370 msgr2=0x7f08840cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:37.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 shutdown_connections 2023-06-10T18:45:37.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.157+0000 7f088acb3700 1 -- 172.21.15.104:0/2119396244 wait complete. 2023-06-10T18:45:37.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-06-10T18:45:37.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.289+0000 7f7a99866700 1 Processor -- start 2023-06-10T18:45:37.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.290+0000 7f7a99866700 1 -- start start 2023-06-10T18:45:37.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.290+0000 7f7a99866700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9406c720 0x7f7a940ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.290+0000 7f7a99866700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a940d0340 con 0x7f7a940ce390 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.290+0000 7f7a99866700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9415cb60 con 0x7f7a940cae10 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.290+0000 7f7a99866700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a940d1660 con 0x7f7a9406c720 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9406c720 0x7f7a940ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a92ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7a940ce390 0x7f7a94160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54998/0 (socket says 172.21.15.104:54998) 2023-06-10T18:45:37.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9406c720 0x7f7a940ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57172/0 (socket says 172.21.15.104:57172) 2023-06-10T18:45:37.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a92ffd700 1 -- 172.21.15.104:0/2438812584 learned_addr learned my addr 172.21.15.104:0/2438812584 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:37.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 -- 172.21.15.104:0/2438812584 >> v1:172.21.15.114:6789/0 conn(0x7f7a940cae10 legacy=0x7f7a940cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:37.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 -- 172.21.15.104:0/2438812584 >> v1:172.21.15.104:6789/0 conn(0x7f7a940ce390 legacy=0x7f7a94160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a91ffb700 1 -- 172.21.15.104:0/2438812584 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316394026 0 0) 0x7f7a940d0340 con 0x7f7a940ce390 2023-06-10T18:45:37.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 -- 172.21.15.104:0/2438812584 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a940ceca0 con 0x7f7a9406c720 2023-06-10T18:45:37.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a927fc700 1 --2- 172.21.15.104:0/2438812584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a940ce760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009840 tx=0x7f7a84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6188878d0ed6020c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a91ffb700 1 -- 172.21.15.104:0/2438812584 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84004230 con 0x7f7a9406c720 2023-06-10T18:45:37.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a91ffb700 1 -- 172.21.15.104:0/2438812584 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84004390 con 0x7f7a9406c720 2023-06-10T18:45:37.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.291+0000 7f7a91ffb700 1 -- 172.21.15.104:0/2438812584 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840036a0 con 0x7f7a9406c720 2023-06-10T18:45:37.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 -- 172.21.15.104:0/2438812584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 msgr2=0x7f7a940ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 --2- 172.21.15.104:0/2438812584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a940ce760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009840 tx=0x7f7a84009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 -- 172.21.15.104:0/2438812584 shutdown_connections 2023-06-10T18:45:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 --2- 172.21.15.104:0/2438812584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a940ce760 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 -- 172.21.15.104:0/2438812584 >> 172.21.15.104:0/2438812584 conn(0x7f7a94069e70 msgr2=0x7f7a9406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 -- 172.21.15.104:0/2438812584 shutdown_connections 2023-06-10T18:45:37.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.292+0000 7f7a99866700 1 -- 172.21.15.104:0/2438812584 wait complete. 2023-06-10T18:45:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.293+0000 7f7a99866700 1 Processor -- start 2023-06-10T18:45:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.293+0000 7f7a99866700 1 -- start start 2023-06-10T18:45:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a940cae10 0x7f7a941478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a940ce390 0x7f7a940b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94147de0 con 0x7f7a9406c720 2023-06-10T18:45:37.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9414b050 con 0x7f7a940ce390 2023-06-10T18:45:37.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9414ad70 con 0x7f7a940cae10 2023-06-10T18:45:37.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57176/0 (socket says 172.21.15.104:57176) 2023-06-10T18:45:37.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 -- 172.21.15.104:0/1708131898 learned_addr learned my addr 172.21.15.104:0/1708131898 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:37.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 -- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a940cae10 msgr2=0x7f7a941478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a940cae10 0x7f7a941478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 -- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a940ce390 msgr2=0x7f7a940b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a940ce390 0x7f7a940b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 -- 172.21.15.104:0/1708131898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840094a0 con 0x7f7a9406c720 2023-06-10T18:45:37.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a92ffd700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7a880011e0 tx=0x7f7a8800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88004c80 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88004e00 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88007310 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a941e5900 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.294+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941e5ec0 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.295+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a941e60b0 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.296+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a88007470 con 0x7f7a9406c720 2023-06-10T18:45:37.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.296+0000 7f7a7bfff700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 0x7f7a8006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.296+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7a88088fa0 con 0x7f7a9406c720 2023-06-10T18:45:37.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.296+0000 7f7a927fc700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 0x7f7a8006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.297+0000 7f7a927fc700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 0x7f7a8006d190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a940ccbd0 tx=0x7f7a84013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.301+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a880bb310 con 0x7f7a9406c720 2023-06-10T18:45:37.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.558+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7a94074050 con 0x7f7a9406c720 2023-06-10T18:45:37.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.563+0000 7f7a7bfff700 1 -- 172.21.15.104:0/1708131898 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7a880542f0 con 0x7f7a9406c720 2023-06-10T18:45:37.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 msgr2=0x7f7a8006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 0x7f7a8006d190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a940ccbd0 tx=0x7f7a84013040 comp rx=0 tx=0).stop 2023-06-10T18:45:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 msgr2=0x7f7a94148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7a880011e0 tx=0x7f7a8800cae0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 shutdown_connections 2023-06-10T18:45:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a8006ace0 0x7f7a8006d190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a940ce390 0x7f7a940b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9406c720 0x7f7a94148040 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 --2- 172.21.15.104:0/1708131898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a940cae10 0x7f7a941478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.565+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 >> 172.21.15.104:0/1708131898 conn(0x7f7a94069e70 msgr2=0x7f7a9415bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:37.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.566+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 shutdown_connections 2023-06-10T18:45:37.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.566+0000 7f7a99866700 1 -- 172.21.15.104:0/1708131898 wait complete. 2023-06-10T18:45:37.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-06-10T18:45:37.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:45:37.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:45:37.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.733+0000 7fa65c666700 1 Processor -- start 2023-06-10T18:45:37.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.733+0000 7fa65c666700 1 -- start start 2023-06-10T18:45:37.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa65c666700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6540ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa65c666700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 0x7fa65406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa65c666700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65415a680 con 0x7fa654069e70 2023-06-10T18:45:37.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa65c666700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6540d0510 con 0x7fa6540cb860 2023-06-10T18:45:37.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa65c666700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65406a5e0 con 0x7fa65415a2b0 2023-06-10T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa659c01700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa654069e70 0x7fa6540cb150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55014/0 (socket says 172.21.15.104:55014) 2023-06-10T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.734+0000 7fa659c01700 1 -- 172.21.15.104:0/2412502500 learned_addr learned my addr 172.21.15.104:0/2412502500 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 --2- 172.21.15.104:0/2412502500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6540ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65ac03700 1 --2- 172.21.15.104:0/2412502500 >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 0x7fa65406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa659400700 1 -- 172.21.15.104:0/2412502500 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842500905 0 0) 0x7fa65415a680 con 0x7fa654069e70 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa659400700 1 -- 172.21.15.104:0/2412502500 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6400035b0 con 0x7fa654069e70 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 -- 172.21.15.104:0/2412502500 >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 msgr2=0x7fa65406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 --2- 172.21.15.104:0/2412502500 >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 0x7fa65406d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 -- 172.21.15.104:0/2412502500 >> v1:172.21.15.104:6789/0 conn(0x7fa654069e70 legacy=0x7fa6540cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 -- 172.21.15.104:0/2412502500 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65406a8c0 con 0x7fa65415a2b0 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65ac03700 1 --2- 172.21.15.104:0/2412502500 >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 0x7fa65406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:37.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa65a402700 1 --2- 172.21.15.104:0/2412502500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6540ac230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa650009820 tx=0x7fa650009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7384b8e04efa1122 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa659400700 1 -- 172.21.15.104:0/2412502500 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650005e30 con 0x7fa65415a2b0 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa659400700 1 -- 172.21.15.104:0/2412502500 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650006030 con 0x7fa65415a2b0 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.735+0000 7fa659400700 1 -- 172.21.15.104:0/2412502500 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6500063a0 con 0x7fa65415a2b0 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 -- 172.21.15.104:0/2412502500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 msgr2=0x7fa6540ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 --2- 172.21.15.104:0/2412502500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6540ac230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa650009820 tx=0x7fa650009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 -- 172.21.15.104:0/2412502500 shutdown_connections 2023-06-10T18:45:37.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 --2- 172.21.15.104:0/2412502500 >> v2:172.21.15.114:3300/0 conn(0x7fa6540cb860 0x7fa65406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 --2- 172.21.15.104:0/2412502500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6540ac230 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 -- 172.21.15.104:0/2412502500 >> 172.21.15.104:0/2412502500 conn(0x7fa65406b8a0 msgr2=0x7fa6540cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:37.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 -- 172.21.15.104:0/2412502500 shutdown_connections 2023-06-10T18:45:37.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.736+0000 7fa65c666700 1 -- 172.21.15.104:0/2412502500 wait complete. 2023-06-10T18:45:37.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.738+0000 7fa65c666700 1 Processor -- start 2023-06-10T18:45:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.738+0000 7fa65c666700 1 -- start start 2023-06-10T18:45:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6540cb860 0x7fa65414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6541eef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65414c720 con 0x7fa654069e70 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65415a7b0 con 0x7fa6540cb860 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65415aa90 con 0x7fa65415a2b0 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57190/0 (socket says 172.21.15.104:57190) 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 -- 172.21.15.104:0/639117987 learned_addr learned my addr 172.21.15.104:0/639117987 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa659c01700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6540cb860 0x7fa65414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65ac03700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6541eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 -- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 msgr2=0x7fa6541eef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6541eef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 -- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6540cb860 msgr2=0x7fa65414c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6540cb860 0x7fa65414c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 -- 172.21.15.104:0/639117987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c009a70 con 0x7fa654069e70 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65a402700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa650009b70 tx=0x7fa650004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6500063a0 con 0x7fa654069e70 2023-06-10T18:45:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650020070 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa650009430 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650006720 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.739+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65415b050 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.740+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa654065f00 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.741+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa650022030 con 0x7fa654069e70 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.741+0000 7fa64affd700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 0x7fa63806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:37.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.741+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fa65008a640 con 0x7fa654069e70 2023-06-10T18:45:37.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.741+0000 7fa659c01700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 0x7fa63806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:37.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.742+0000 7fa659c01700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 0x7fa63806d190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa6541f2cd0 tx=0x7fa644009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:37.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:37.745+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa650055990 con 0x7fa654069e70 2023-06-10T18:45:38.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.008+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fa654074050 con 0x7fa654069e70 2023-06-10T18:45:38.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.013+0000 7fa64affd700 1 -- 172.21.15.104:0/639117987 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa650058fb0 con 0x7fa654069e70 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 msgr2=0x7fa63806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 0x7fa63806d190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa6541f2cd0 tx=0x7fa644009040 comp rx=0 tx=0).stop 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 msgr2=0x7fa654148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa650009b70 tx=0x7fa650004750 comp rx=0 tx=0).stop 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 shutdown_connections 2023-06-10T18:45:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa65415a2b0 0x7fa6541eef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa654069e70 0x7fa654148d50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa63806ace0 0x7fa63806d190 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 --2- 172.21.15.104:0/639117987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6540cb860 0x7fa65414c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 >> 172.21.15.104:0/639117987 conn(0x7fa65406b8a0 msgr2=0x7fa6540ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 shutdown_connections 2023-06-10T18:45:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.015+0000 7fa65c666700 1 -- 172.21.15.104:0/639117987 wait complete. 2023-06-10T18:45:38.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:38.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.188+0000 7fc45b407700 1 Processor -- start 2023-06-10T18:45:38.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.188+0000 7fc45b407700 1 -- start start 2023-06-10T18:45:38.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc45b407700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc45415a140 0x7fc4540c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc45b407700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4540ca210 0x7fc454070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc45b407700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc454070810 con 0x7fc4540ce430 2023-06-10T18:45:38.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc45b407700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540b0ff0 con 0x7fc4540ca210 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc45b407700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540c97f0 con 0x7fc45415a140 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4540ca210 0x7fc454070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc4540ca210 0x7fc454070130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57202/0 (socket says 172.21.15.104:57202) 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 -- 172.21.15.104:0/3151127786 learned_addr learned my addr 172.21.15.104:0/3151127786 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4591a3700 1 --2- 172.21.15.104:0/3151127786 >> v2:172.21.15.114:3300/0 conn(0x7fc45415a140 0x7fc4540c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 -- 172.21.15.104:0/3151127786 >> v2:172.21.15.114:3300/0 conn(0x7fc45415a140 msgr2=0x7fc4540c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:38.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 --2- 172.21.15.104:0/3151127786 >> v2:172.21.15.114:3300/0 conn(0x7fc45415a140 0x7fc4540c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 -- 172.21.15.104:0/3151127786 >> v1:172.21.15.114:6789/0 conn(0x7fc4540ce430 legacy=0x7fc4540ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 -- 172.21.15.104:0/3151127786 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454071f70 con 0x7fc4540ca210 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc4589a2700 1 --2- 172.21.15.104:0/3151127786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ca210 0x7fc454070130 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc45406f9f0 tx=0x7fc444009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1cf3b9b11762ba server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.189+0000 7fc44bfff700 1 -- 172.21.15.104:0/3151127786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444004230 con 0x7fc4540ca210 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc44bfff700 1 -- 172.21.15.104:0/3151127786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444004390 con 0x7fc4540ca210 2023-06-10T18:45:38.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc44bfff700 1 -- 172.21.15.104:0/3151127786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4440036a0 con 0x7fc4540ca210 2023-06-10T18:45:38.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 -- 172.21.15.104:0/3151127786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ca210 msgr2=0x7fc454070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 --2- 172.21.15.104:0/3151127786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ca210 0x7fc454070130 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc45406f9f0 tx=0x7fc444009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 -- 172.21.15.104:0/3151127786 shutdown_connections 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 --2- 172.21.15.104:0/3151127786 >> v2:172.21.15.114:3300/0 conn(0x7fc45415a140 0x7fc4540c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 --2- 172.21.15.104:0/3151127786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ca210 0x7fc454070130 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 -- 172.21.15.104:0/3151127786 >> 172.21.15.104:0/3151127786 conn(0x7fc45406ab00 msgr2=0x7fc4540b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 -- 172.21.15.104:0/3151127786 shutdown_connections 2023-06-10T18:45:38.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.190+0000 7fc45b407700 1 -- 172.21.15.104:0/3151127786 wait complete. 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.191+0000 7fc45b407700 1 Processor -- start 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.191+0000 7fc45b407700 1 -- start start 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 0x7fc454159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 0x7fc4540c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b37f0 con 0x7fc4540ce430 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b2b70 con 0x7fc45415a140 2023-06-10T18:45:38.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc45b407700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540b2e50 con 0x7fc4540ca210 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57206/0 (socket says 172.21.15.104:57206) 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 -- 172.21.15.104:0/1565546637 learned_addr learned my addr 172.21.15.104:0/1565546637 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4591a3700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 0x7fc454159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 -- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 msgr2=0x7fc454159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4599a4700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 0x7fc4540c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 0x7fc454159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 -- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 msgr2=0x7fc4540c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 0x7fc4540c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4589a2700 1 -- 172.21.15.104:0/1565546637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009550 con 0x7fc4540ce430 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4591a3700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 0x7fc454159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.192+0000 7fc4599a4700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 0x7fc4540c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc4589a2700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc444009130 tx=0x7fc444004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444003d80 con 0x7fc4540ce430 2023-06-10T18:45:38.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44401f070 con 0x7fc4540ce430 2023-06-10T18:45:38.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4540b3130 con 0x7fc4540ce430 2023-06-10T18:45:38.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444011870 con 0x7fc4540ce430 2023-06-10T18:45:38.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.193+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4541595a0 con 0x7fc4540ce430 2023-06-10T18:45:38.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.194+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc454159730 con 0x7fc4540ce430 2023-06-10T18:45:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.194+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc444021030 con 0x7fc4540ce430 2023-06-10T18:45:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.195+0000 7fc449ffb700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 0x7fc43806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.195+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc44408a7f0 con 0x7fc4540ce430 2023-06-10T18:45:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.195+0000 7fc4591a3700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 0x7fc43806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.196+0000 7fc4591a3700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 0x7fc43806d190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc45415d010 tx=0x7fc450008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.199+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc444055b40 con 0x7fc4540ce430 2023-06-10T18:45:38.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.388+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc454074050 con 0x7fc4540ce430 2023-06-10T18:45:38.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.389+0000 7fc449ffb700 1 -- 172.21.15.104:0/1565546637 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+14892 (secure 0 0 0) 0x7fc444059160 con 0x7fc4540ce430 2023-06-10T18:45:38.394 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:38.395 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":38,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:35.056596+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:38.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 msgr2=0x7fc43806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 0x7fc43806d190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc45415d010 tx=0x7fc450008040 comp rx=0 tx=0).stop 2023-06-10T18:45:38.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 msgr2=0x7fc4540c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc444009130 tx=0x7fc444004750 comp rx=0 tx=0).stop 2023-06-10T18:45:38.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 shutdown_connections 2023-06-10T18:45:38.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc43806ace0 0x7fc43806d190 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45415a140 0x7fc4540c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4540ca210 0x7fc454159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 --2- 172.21.15.104:0/1565546637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4540ce430 0x7fc4540c5650 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 >> 172.21.15.104:0/1565546637 conn(0x7fc45406ab00 msgr2=0x7fc4540c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 shutdown_connections 2023-06-10T18:45:38.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.391+0000 7fc45b407700 1 -- 172.21.15.104:0/1565546637 wait complete. 2023-06-10T18:45:38.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:45:38.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:38.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.525+0000 7f0c1bc83700 1 Processor -- start 2023-06-10T18:45:38.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.525+0000 7f0c1bc83700 1 -- start start 2023-06-10T18:45:38.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1bc83700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 0x7f0c140d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1bc83700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0c140ce7d0 0x7f0c140cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1bc83700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140ce340 con 0x7f0c140ccff0 2023-06-10T18:45:38.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1bc83700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140cd490 con 0x7f0c140ce7d0 2023-06-10T18:45:38.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1bc83700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140d00d0 con 0x7f0c1406af80 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0c140ce7d0 0x7f0c140cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1921e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0c140ccff0 0x7f0c1406b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55040/0 (socket says 172.21.15.104:55040) 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0c140ce7d0 0x7f0c140cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57214/0 (socket says 172.21.15.104:57214) 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1921e700 1 -- 172.21.15.104:0/3510079738 learned_addr learned my addr 172.21.15.104:0/3510079738 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c19a1f700 1 --2- 172.21.15.104:0/3510079738 >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 0x7f0c140d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 -- 172.21.15.104:0/3510079738 >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 msgr2=0x7f0c140d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 --2- 172.21.15.104:0/3510079738 >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 0x7f0c140d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 -- 172.21.15.104:0/3510079738 >> v1:172.21.15.104:6789/0 conn(0x7f0c140ccff0 legacy=0x7f0c1406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c18a1d700 1 -- 172.21.15.104:0/3510079738 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964116448 0 0) 0x7f0c140ce340 con 0x7f0c140ccff0 2023-06-10T18:45:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 -- 172.21.15.104:0/3510079738 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140cd680 con 0x7f0c140ce7d0 2023-06-10T18:45:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c19a1f700 1 --2- 172.21.15.104:0/3510079738 >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 0x7f0c140d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.526+0000 7f0c1a220700 1 --2- 172.21.15.104:0/3510079738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c140ce7d0 0x7f0c140cec50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0c14071e10 tx=0x7f0c0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3994d2afccd94c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.527+0000 7f0c18a1d700 1 -- 172.21.15.104:0/3510079738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00e070 con 0x7f0c140ce7d0 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.527+0000 7f0c18a1d700 1 -- 172.21.15.104:0/3510079738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00bbd0 con 0x7f0c140ce7d0 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.527+0000 7f0c18a1d700 1 -- 172.21.15.104:0/3510079738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c012430 con 0x7f0c140ce7d0 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 -- 172.21.15.104:0/3510079738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c140ce7d0 msgr2=0x7f0c140cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/3510079738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c140ce7d0 0x7f0c140cec50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0c14071e10 tx=0x7f0c0c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 -- 172.21.15.104:0/3510079738 shutdown_connections 2023-06-10T18:45:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/3510079738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c140ce7d0 0x7f0c140cec50 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/3510079738 >> v2:172.21.15.114:3300/0 conn(0x7f0c1406af80 0x7f0c140d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 -- 172.21.15.104:0/3510079738 >> 172.21.15.104:0/3510079738 conn(0x7f0c14089fd0 msgr2=0x7f0c1408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 -- 172.21.15.104:0/3510079738 shutdown_connections 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.528+0000 7f0c1bc83700 1 -- 172.21.15.104:0/3510079738 wait complete. 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.529+0000 7f0c1bc83700 1 Processor -- start 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.529+0000 7f0c1bc83700 1 -- start start 2023-06-10T18:45:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c140ccff0 0x7f0c1415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c140ce7d0 0x7f0c140b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1415d540 con 0x7f0c1406af80 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1415c930 con 0x7f0c140ccff0 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c1bc83700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1415cc10 con 0x7f0c140ce7d0 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c19a1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c19a1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57216/0 (socket says 172.21.15.104:57216) 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.530+0000 7f0c19a1f700 1 -- 172.21.15.104:0/2731733771 learned_addr learned my addr 172.21.15.104:0/2731733771 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c1a220700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c140ce7d0 0x7f0c140b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c1921e700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c140ccff0 0x7f0c1415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 -- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c140ce7d0 msgr2=0x7f0c140b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c140ce7d0 0x7f0c140b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 -- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c140ccff0 msgr2=0x7f0c1415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c140ccff0 0x7f0c1415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 -- 172.21.15.104:0/2731733771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c009a70 con 0x7f0c1406af80 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.531+0000 7f0c19a1f700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000c750 tx=0x7f0c1000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.532+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011800 con 0x7f0c1406af80 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.532+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c10011960 con 0x7f0c1406af80 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.532+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1415cef0 con 0x7f0c1406af80 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.532+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011cd0 con 0x7f0c1406af80 2023-06-10T18:45:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.532+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c140b55a0 con 0x7f0c1406af80 2023-06-10T18:45:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.533+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1406c600 con 0x7f0c1406af80 2023-06-10T18:45:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.533+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0c10021350 con 0x7f0c1406af80 2023-06-10T18:45:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.533+0000 7f0c0a7fc700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 0x7f0bf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.533+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f0c100890e0 con 0x7f0c1406af80 2023-06-10T18:45:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.534+0000 7f0c1921e700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 0x7f0bf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.534+0000 7f0c1921e700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 0x7f0bf806d190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c14155ef0 tx=0x7f0c04007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.538+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0c10057e30 con 0x7f0c1406af80 2023-06-10T18:45:38.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.723+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c1415f650 con 0x7f0c1406af80 2023-06-10T18:45:38.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.724+0000 7f0c0a7fc700 1 -- 172.21.15.104:0/2731733771 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c100579c0 con 0x7f0c1406af80 2023-06-10T18:45:38.730 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 msgr2=0x7f0bf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 0x7f0bf806d190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c14155ef0 tx=0x7f0c04007de0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 msgr2=0x7f0c140b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000c750 tx=0x7f0c1000ca60 comp rx=0 tx=0).stop 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 shutdown_connections 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c140ce7d0 0x7f0c140b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c1406af80 0x7f0c140b5060 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bf806ace0 0x7f0bf806d190 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 --2- 172.21.15.104:0/2731733771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c140ccff0 0x7f0c1415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 >> 172.21.15.104:0/2731733771 conn(0x7f0c14089fd0 msgr2=0x7f0c14156060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 shutdown_connections 2023-06-10T18:45:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.726+0000 7f0c1bc83700 1 -- 172.21.15.104:0/2731733771 wait complete. 2023-06-10T18:45:38.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:38.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.861+0000 7f523872b700 1 Processor -- start 2023-06-10T18:45:38.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.861+0000 7f523872b700 1 -- start start 2023-06-10T18:45:38.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f523872b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f523006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f523872b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f523006ea70 0x7f523006d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f523872b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300ae7e0 con 0x7f523015c6f0 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f523872b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523006df90 con 0x7f523006ea70 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f523872b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523006e270 con 0x7f52300b1780 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f523006ea70 0x7f523006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f523006ea70 0x7f523006d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57228/0 (socket says 172.21.15.104:57228) 2023-06-10T18:45:38.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 -- 172.21.15.104:0/2448213591 learned_addr learned my addr 172.21.15.104:0/2448213591 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:38.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5235cc6700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f523006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 -- 172.21.15.104:0/2448213591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 msgr2=0x7f523006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:38.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f523006cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 -- 172.21.15.104:0/2448213591 >> v1:172.21.15.114:6789/0 conn(0x7f523015c6f0 legacy=0x7f52300b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.862+0000 7f5236cc8700 1 -- 172.21.15.104:0/2448213591 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523006e550 con 0x7f523006ea70 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f5236cc8700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f523006d320 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5228009e50 tx=0x7f522800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2b0fcfab8b7ac1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f52354c5700 1 -- 172.21.15.104:0/2448213591 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522800e070 con 0x7f523006ea70 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f52354c5700 1 -- 172.21.15.104:0/2448213591 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5228003a20 con 0x7f523006ea70 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f52354c5700 1 -- 172.21.15.104:0/2448213591 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228003dd0 con 0x7f523006ea70 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 -- 172.21.15.104:0/2448213591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 msgr2=0x7f523006d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f523006d320 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5228009e50 tx=0x7f522800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 -- 172.21.15.104:0/2448213591 shutdown_connections 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f523006d320 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 --2- 172.21.15.104:0/2448213591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f523006cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 -- 172.21.15.104:0/2448213591 >> 172.21.15.104:0/2448213591 conn(0x7f523006b8a0 msgr2=0x7f52300cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 -- 172.21.15.104:0/2448213591 shutdown_connections 2023-06-10T18:45:38.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.863+0000 7f523872b700 1 -- 172.21.15.104:0/2448213591 wait complete. 2023-06-10T18:45:38.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.864+0000 7f523872b700 1 Processor -- start 2023-06-10T18:45:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 -- start start 2023-06-10T18:45:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f52300c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 0x7f52300cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5230151530 con 0x7f523006ea70 2023-06-10T18:45:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52300c8900 con 0x7f523015c6f0 2023-06-10T18:45:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f523872b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52300c7040 con 0x7f52300b1780 2023-06-10T18:45:38.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f52364c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f52364c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57230/0 (socket says 172.21.15.104:57230) 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f52364c7700 1 -- 172.21.15.104:0/4071058770 learned_addr learned my addr 172.21.15.104:0/4071058770 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f5235cc6700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f52300c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.865+0000 7f5236cc8700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 0x7f52300cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 -- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 msgr2=0x7f52300c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f52300c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 -- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 msgr2=0x7f52300cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 0x7f52300cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 -- 172.21.15.104:0/4071058770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228009b30 con 0x7f523006ea70 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f5236cc8700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 0x7f52300cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f52364c7700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f522000b450 tx=0x7f522000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220004020 con 0x7f523006ea70 2023-06-10T18:45:38.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5220004180 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52300c7320 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220004530 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.866+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52300c78e0 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.867+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230074050 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.867+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f52200036d0 con 0x7f523006ea70 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.868+0000 7f5226ffd700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 0x7f521406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:38.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.868+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f52200889c0 con 0x7f523006ea70 2023-06-10T18:45:38.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.868+0000 7f5235cc6700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 0x7f521406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:38.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.868+0000 7f5235cc6700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 0x7f521406d190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52300cd570 tx=0x7f522c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:38.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:38.872+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5220053d10 con 0x7f523006ea70 2023-06-10T18:45:39.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.055+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f52301583d0 con 0x7f523006ea70 2023-06-10T18:45:39.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.056+0000 7f5226ffd700 1 -- 172.21.15.104:0/4071058770 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5220015020 con 0x7f523006ea70 2023-06-10T18:45:39.061 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.057+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 msgr2=0x7f521406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.057+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 0x7f521406d190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f52300cd570 tx=0x7f522c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.057+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 msgr2=0x7f5230150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.057+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f522000b450 tx=0x7f522000b760 comp rx=0 tx=0).stop 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 shutdown_connections 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f521406ace0 0x7f521406d190 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f523015c6f0 0x7f52300cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f523006ea70 0x7f5230150ff0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 --2- 172.21.15.104:0/4071058770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52300b1780 0x7f52300c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 >> 172.21.15.104:0/4071058770 conn(0x7f523006b8a0 msgr2=0x7f52300b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 shutdown_connections 2023-06-10T18:45:39.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.058+0000 7f523872b700 1 -- 172.21.15.104:0/4071058770 wait complete. 2023-06-10T18:45:39.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:45:39.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.189+0000 7fdf1be1c700 1 Processor -- start 2023-06-10T18:45:39.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.189+0000 7fdf1be1c700 1 -- start start 2023-06-10T18:45:39.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1be1c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 0x7fdf1406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1be1c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 0x7fdf1406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1be1c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf140c9e60 con 0x7fdf140aeaa0 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1be1c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf140c9bc0 con 0x7fdf140cb300 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1be1c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140aee70 con 0x7fdf140cd850 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1a3b9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 0x7fdf1406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf193b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 0x7fdf1406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf19bb8700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdf140aeaa0 0x7fdf140cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60912/0 (socket says 172.21.15.104:60912) 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf19bb8700 1 -- 172.21.15.104:0/970130987 learned_addr learned my addr 172.21.15.104:0/970130987 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf18bb6700 1 -- 172.21.15.104:0/970130987 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823886487 0 0) 0x7fdf140c9e60 con 0x7fdf140aeaa0 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf18bb6700 1 -- 172.21.15.104:0/970130987 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf000035b0 con 0x7fdf140aeaa0 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf193b7700 1 -- 172.21.15.104:0/970130987 >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 msgr2=0x7fdf1406d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf193b7700 1 --2- 172.21.15.104:0/970130987 >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 0x7fdf1406d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf193b7700 1 -- 172.21.15.104:0/970130987 >> v1:172.21.15.114:6789/0 conn(0x7fdf140aeaa0 legacy=0x7fdf140cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf193b7700 1 -- 172.21.15.104:0/970130987 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1406e540 con 0x7fdf140cd850 2023-06-10T18:45:39.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.190+0000 7fdf1a3b9700 1 --2- 172.21.15.104:0/970130987 >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 0x7fdf1406d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:39.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf193b7700 1 --2- 172.21.15.104:0/970130987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 0x7fdf1406cd50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009840 tx=0x7fdf04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18af6c20e70dd50 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf18bb6700 1 -- 172.21.15.104:0/970130987 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011830 con 0x7fdf140cd850 2023-06-10T18:45:39.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf18bb6700 1 -- 172.21.15.104:0/970130987 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf04011990 con 0x7fdf140cd850 2023-06-10T18:45:39.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf18bb6700 1 -- 172.21.15.104:0/970130987 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011d20 con 0x7fdf140cd850 2023-06-10T18:45:39.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 -- 172.21.15.104:0/970130987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 msgr2=0x7fdf1406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/970130987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 0x7fdf1406cd50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009840 tx=0x7fdf04009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:39.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 -- 172.21.15.104:0/970130987 shutdown_connections 2023-06-10T18:45:39.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/970130987 >> v2:172.21.15.114:3300/0 conn(0x7fdf140cb300 0x7fdf1406d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/970130987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140cd850 0x7fdf1406cd50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 -- 172.21.15.104:0/970130987 >> 172.21.15.104:0/970130987 conn(0x7fdf1406b8a0 msgr2=0x7fdf140cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:39.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 -- 172.21.15.104:0/970130987 shutdown_connections 2023-06-10T18:45:39.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.191+0000 7fdf1be1c700 1 -- 172.21.15.104:0/970130987 wait complete. 2023-06-10T18:45:39.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 Processor -- start 2023-06-10T18:45:39.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 -- start start 2023-06-10T18:45:39.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140aeaa0 0x7fdf140b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 0x7fdf140c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140b54f0 con 0x7fdf140cb300 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141551a0 con 0x7fdf140cd850 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf1be1c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141ee790 con 0x7fdf140aeaa0 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf193b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf193b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57246/0 (socket says 172.21.15.104:57246) 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf193b7700 1 -- 172.21.15.104:0/3204711214 learned_addr learned my addr 172.21.15.104:0/3204711214 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.193+0000 7fdf19bb8700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140aeaa0 0x7fdf140b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf1a3b9700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 0x7fdf140c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 -- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140aeaa0 msgr2=0x7fdf140b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140aeaa0 0x7fdf140b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 -- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 msgr2=0x7fdf140c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 0x7fdf140c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 -- 172.21.15.104:0/3204711214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c009a70 con 0x7fdf140cb300 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf1a3b9700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 0x7fdf140c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf193b7700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdf040090b0 tx=0x7fdf04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf040036a0 con 0x7fdf140cb300 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0401e070 con 0x7fdf140cb300 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04003a40 con 0x7fdf140cb300 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040094a0 con 0x7fdf140cb300 2023-06-10T18:45:39.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.194+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf140b3e00 con 0x7fdf140cb300 2023-06-10T18:45:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.195+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf14074050 con 0x7fdf140cb300 2023-06-10T18:45:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.195+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdf04003ba0 con 0x7fdf140cb300 2023-06-10T18:45:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.196+0000 7fdf0a7fc700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 0x7fdef806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.196+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fdf0408a0f0 con 0x7fdf140cb300 2023-06-10T18:45:39.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.196+0000 7fdf19bb8700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 0x7fdef806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.196+0000 7fdf19bb8700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 0x7fdef806d190 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf10001230 tx=0x7fdf10007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.200+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdf04055440 con 0x7fdf140cb300 2023-06-10T18:45:39.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.388+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdf14071540 con 0x7fdf140cb300 2023-06-10T18:45:39.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.388+0000 7fdf0a7fc700 1 -- 172.21.15.104:0/3204711214 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) v1 ==== 84+0+126 (secure 0 0 0) 0x7fdf04058a60 con 0x7fdf140cb300 2023-06-10T18:45:39.398 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:39.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:45:39.400 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:45:39.400 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:45:39.400 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:39.400 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:45:39.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 msgr2=0x7fdef806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 0x7fdef806d190 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf10001230 tx=0x7fdf10007df0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 msgr2=0x7fdf14154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdf040090b0 tx=0x7fdf04004750 comp rx=0 tx=0).stop 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 shutdown_connections 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdef806ace0 0x7fdef806d190 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf140cd850 0x7fdf140c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf140aeaa0 0x7fdf140b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 --2- 172.21.15.104:0/3204711214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf140cb300 0x7fdf14154c60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 >> 172.21.15.104:0/3204711214 conn(0x7fdf1406b8a0 msgr2=0x7fdf140b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:39.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 shutdown_connections 2023-06-10T18:45:39.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.390+0000 7fdf1be1c700 1 -- 172.21.15.104:0/3204711214 wait complete. 2023-06-10T18:45:39.408 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:45:39.408 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:45:39.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.526+0000 7f5d48124700 1 Processor -- start 2023-06-10T18:45:39.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 -- start start 2023-06-10T18:45:39.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 0x7f5d400cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5d40069e70 0x7f5d4006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d400ce040 con 0x7f5d4006dcd0 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4015c8d0 con 0x7f5d40069e70 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d48124700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4006da40 con 0x7f5d4015c500 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d456bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5d40069e70 0x7f5d4006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.527+0000 7f5d466c1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5d4006dcd0 0x7f5d400c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55084/0 (socket says 172.21.15.104:55084) 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5d40069e70 0x7f5d4006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57254/0 (socket says 172.21.15.104:57254) 2023-06-10T18:45:39.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d466c1700 1 -- 172.21.15.104:0/1806304561 learned_addr learned my addr 172.21.15.104:0/1806304561 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 0x7f5d400cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 -- 172.21.15.104:0/1806304561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 msgr2=0x7f5d400cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 0x7f5d400cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 -- 172.21.15.104:0/1806304561 >> v1:172.21.15.104:6789/0 conn(0x7f5d4006dcd0 legacy=0x7f5d400c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 -- 172.21.15.104:0/1806304561 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4006a9c0 con 0x7f5d40069e70 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d44ebe700 1 -- 172.21.15.104:0/1806304561 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530689672 0 0) 0x7f5d400ce040 con 0x7f5d4006dcd0 2023-06-10T18:45:39.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 0x7f5d400cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d456bf700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d40069e70 0x7f5d4006a2e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009840 tx=0x7f5d3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317f7e478c79cd3d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d44ebe700 1 -- 172.21.15.104:0/1806304561 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011830 con 0x7f5d40069e70 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d44ebe700 1 -- 172.21.15.104:0/1806304561 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c011990 con 0x7f5d40069e70 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.528+0000 7f5d44ebe700 1 -- 172.21.15.104:0/1806304561 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c011d20 con 0x7f5d40069e70 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 -- 172.21.15.104:0/1806304561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d40069e70 msgr2=0x7f5d4006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d40069e70 0x7f5d4006a2e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009840 tx=0x7f5d3c009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:39.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 -- 172.21.15.104:0/1806304561 shutdown_connections 2023-06-10T18:45:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4015c500 0x7f5d400cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 --2- 172.21.15.104:0/1806304561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d40069e70 0x7f5d4006a2e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 -- 172.21.15.104:0/1806304561 >> 172.21.15.104:0/1806304561 conn(0x7f5d4006b8a0 msgr2=0x7f5d400cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 -- 172.21.15.104:0/1806304561 shutdown_connections 2023-06-10T18:45:39.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.529+0000 7f5d48124700 1 -- 172.21.15.104:0/1806304561 wait complete. 2023-06-10T18:45:39.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.530+0000 7f5d48124700 1 Processor -- start 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.530+0000 7f5d48124700 1 -- start start 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 0x7f5d400c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4006dcd0 0x7f5d400b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d400b2440 con 0x7f5d4015c500 2023-06-10T18:45:39.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401f50d0 con 0x7f5d40069e70 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401f53b0 con 0x7f5d4006dcd0 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57262/0 (socket says 172.21.15.104:57262) 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 -- 172.21.15.104:0/1138052697 learned_addr learned my addr 172.21.15.104:0/1138052697 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d456bf700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4006dcd0 0x7f5d400b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 0x7f5d400c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 -- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4006dcd0 msgr2=0x7f5d400b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4006dcd0 0x7f5d400b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 -- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 msgr2=0x7f5d400c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 0x7f5d400c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 -- 172.21.15.104:0/1138052697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0094a0 con 0x7f5d4015c500 2023-06-10T18:45:39.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 0x7f5d400c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:39.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d466c1700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5d38002230 tx=0x7f5d3800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3800ac20 con 0x7f5d4015c500 2023-06-10T18:45:39.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3800ada0 con 0x7f5d4015c500 2023-06-10T18:45:39.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d401f5690 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38007310 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.531+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d400c6c20 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.532+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d40074050 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.533+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d38007470 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.533+0000 7f5d367fc700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 0x7f5d2406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.533+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f5d38089100 con 0x7f5d4015c500 2023-06-10T18:45:39.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.533+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 0x7f5d2406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:39.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.534+0000 7f5d45ec0700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 0x7f5d2406d160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5d30006fd0 tx=0x7f5d30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:39.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.537+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d38054450 con 0x7f5d4015c500 2023-06-10T18:45:39.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.722+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5d4006e7f0 con 0x7f5d4015c500 2023-06-10T18:45:39.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.887+0000 7f5d367fc700 1 -- 172.21.15.104:0/1138052697 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v39) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5d38057a70 con 0x7f5d4015c500 2023-06-10T18:45:39.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 msgr2=0x7f5d2406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 0x7f5d2406d160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5d30006fd0 tx=0x7f5d30008040 comp rx=0 tx=0).stop 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 msgr2=0x7f5d401f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5d38002230 tx=0x7f5d3800a060 comp rx=0 tx=0).stop 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 shutdown_connections 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d2406acb0 0x7f5d2406d160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d40069e70 0x7f5d400c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d4015c500 0x7f5d401f0bd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 --2- 172.21.15.104:0/1138052697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d4006dcd0 0x7f5d400b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:39.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 >> 172.21.15.104:0/1138052697 conn(0x7f5d4006b8a0 msgr2=0x7f5d400af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:39.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 shutdown_connections 2023-06-10T18:45:39.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:39.890+0000 7f5d48124700 1 -- 172.21.15.104:0/1138052697 wait complete. 2023-06-10T18:45:39.898 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:45:39.908 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:45:40.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.029+0000 7f7ce9aa1700 1 Processor -- start 2023-06-10T18:45:40.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.029+0000 7f7ce9aa1700 1 -- start start 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.030+0000 7f7ce9aa1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 0x7f7ce40cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.030+0000 7f7ce9aa1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 0x7f7ce406e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.030+0000 7f7ce9aa1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce415c8d0 con 0x7f7ce40cf200 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.030+0000 7f7ce9aa1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce40aef20 con 0x7f7ce406dcd0 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.030+0000 7f7ce9aa1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce406e6b0 con 0x7f7ce415c500 2023-06-10T18:45:40.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce3fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 0x7f7ce406e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 0x7f7ce40cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce2ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7ce40cf200 0x7f7ce4069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60940/0 (socket says 172.21.15.104:60940) 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce2ffd700 1 -- 172.21.15.104:0/3665649506 learned_addr learned my addr 172.21.15.104:0/3665649506 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 -- 172.21.15.104:0/3665649506 >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 msgr2=0x7f7ce406e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 --2- 172.21.15.104:0/3665649506 >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 0x7f7ce406e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 -- 172.21.15.104:0/3665649506 >> v1:172.21.15.114:6789/0 conn(0x7f7ce40cf200 legacy=0x7f7ce4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 -- 172.21.15.104:0/3665649506 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce406abd0 con 0x7f7ce415c500 2023-06-10T18:45:40.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce27fc700 1 -- 172.21.15.104:0/3665649506 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084319784 0 0) 0x7f7ce415c8d0 con 0x7f7ce40cf200 2023-06-10T18:45:40.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce3fff700 1 --2- 172.21.15.104:0/3665649506 >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 0x7f7ce406e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:40.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce37fe700 1 --2- 172.21.15.104:0/3665649506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 0x7f7ce40cecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009820 tx=0x7f7cd8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae64393bb244bdc5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:40.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce27fc700 1 -- 172.21.15.104:0/3665649506 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8005e30 con 0x7f7ce415c500 2023-06-10T18:45:40.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce27fc700 1 -- 172.21.15.104:0/3665649506 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8006030 con 0x7f7ce415c500 2023-06-10T18:45:40.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.031+0000 7f7ce27fc700 1 -- 172.21.15.104:0/3665649506 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80063c0 con 0x7f7ce415c500 2023-06-10T18:45:40.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/3665649506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 msgr2=0x7f7ce40cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:40.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/3665649506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 0x7f7ce40cecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009820 tx=0x7f7cd8009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/3665649506 shutdown_connections 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/3665649506 >> v2:172.21.15.114:3300/0 conn(0x7f7ce406dcd0 0x7f7ce406e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/3665649506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce415c500 0x7f7ce40cecc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/3665649506 >> 172.21.15.104:0/3665649506 conn(0x7f7ce406b8a0 msgr2=0x7f7ce40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/3665649506 shutdown_connections 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.032+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/3665649506 wait complete. 2023-06-10T18:45:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.033+0000 7f7ce9aa1700 1 Processor -- start 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.033+0000 7f7ce9aa1700 1 -- start start 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 0x7f7ce41f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 0x7f7ce40c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40c6d30 con 0x7f7ce406dcd0 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce415f1e0 con 0x7f7ce41f5650 2023-06-10T18:45:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce9aa1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40c66e0 con 0x7f7ce40cf200 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57272/0 (socket says 172.21.15.104:57272) 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 -- 172.21.15.104:0/1562247324 learned_addr learned my addr 172.21.15.104:0/1562247324 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce3fff700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 0x7f7ce40c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce2ffd700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 0x7f7ce41f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 -- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 msgr2=0x7f7ce41f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:40.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 0x7f7ce41f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 -- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 msgr2=0x7f7ce40c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 0x7f7ce40c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce37fe700 1 -- 172.21.15.104:0/1562247324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc009a70 con 0x7f7ce406dcd0 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce2ffd700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 0x7f7ce41f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.034+0000 7f7ce3fff700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 0x7f7ce40c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ce37fe700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8012000 tx=0x7f7cd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8003ab0 con 0x7f7ce406dcd0 2023-06-10T18:45:40.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8003c30 con 0x7f7ce406dcd0 2023-06-10T18:45:40.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd8009430 con 0x7f7ce406dcd0 2023-06-10T18:45:40.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80045d0 con 0x7f7ce406dcd0 2023-06-10T18:45:40.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.035+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41f5c10 con 0x7f7ce406dcd0 2023-06-10T18:45:40.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.036+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4074050 con 0x7f7ce406dcd0 2023-06-10T18:45:40.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.036+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7cd8022070 con 0x7f7ce406dcd0 2023-06-10T18:45:40.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.037+0000 7f7ccbfff700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 0x7f7cd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:40.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.037+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f7cd808a990 con 0x7f7ce406dcd0 2023-06-10T18:45:40.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.037+0000 7f7ce2ffd700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 0x7f7cd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:40.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.037+0000 7f7ce2ffd700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 0x7f7cd006d190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ce415fcc0 tx=0x7f7cd4007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:40.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.041+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7cd8055b70 con 0x7f7ce406dcd0 2023-06-10T18:45:40.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.226+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7ce4066f10 con 0x7f7ce406dcd0 2023-06-10T18:45:40.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.886+0000 7f7ccbfff700 1 -- 172.21.15.104:0/1562247324 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v40) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7cd8059190 con 0x7f7ce406dcd0 2023-06-10T18:45:41.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 msgr2=0x7f7cd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 0x7f7cd006d190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ce415fcc0 tx=0x7f7cd4007ca0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 msgr2=0x7f7ce415eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8012000 tx=0x7f7cd8004750 comp rx=0 tx=0).stop 2023-06-10T18:45:41.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 shutdown_connections 2023-06-10T18:45:41.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cd006ace0 0x7f7cd006d190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce41f5650 0x7f7ce40c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce406dcd0 0x7f7ce415eca0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 --2- 172.21.15.104:0/1562247324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce40cf200 0x7f7ce41f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.890+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 >> 172.21.15.104:0/1562247324 conn(0x7f7ce406b8a0 msgr2=0x7f7ce415b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.891+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 shutdown_connections 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:40.892+0000 7f7ce9aa1700 1 -- 172.21.15.104:0/1562247324 wait complete. 2023-06-10T18:45:41.069 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:45:41.070 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:45:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.193+0000 7f65b9884700 1 Processor -- start 2023-06-10T18:45:41.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.193+0000 7f65b9884700 1 -- start start 2023-06-10T18:45:41.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b9884700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65b415c580 0x7f65b40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b9884700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 0x7f65b40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b9884700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b406e250 con 0x7f65b406cde0 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b9884700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b415c950 con 0x7f65b415c580 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b9884700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40cf0e0 con 0x7f65b406e6e0 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b37fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f65b406cde0 0x7f65b4069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:60958/0 (socket says 172.21.15.104:60958) 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b2ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65b415c580 0x7f65b40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b37fe700 1 -- 172.21.15.104:0/1377375410 learned_addr learned my addr 172.21.15.104:0/1377375410 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.194+0000 7f65b27fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 0x7f65b40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b1ffb700 1 -- 172.21.15.104:0/1377375410 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374859712 0 0) 0x7f65b406e250 con 0x7f65b406cde0 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b1ffb700 1 -- 172.21.15.104:0/1377375410 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c0035b0 con 0x7f65b406cde0 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 msgr2=0x7f65b40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b2ffd700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 0x7f65b40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1377375410 >> v1:172.21.15.114:6789/0 conn(0x7f65b406cde0 legacy=0x7f65b4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1377375410 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40b1e80 con 0x7f65b415c580 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b27fc700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 0x7f65b40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:41.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b2ffd700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b415c580 0x7f65b40b1070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009820 tx=0x7f65a8009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a50ea8843686413 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b1ffb700 1 -- 172.21.15.104:0/1377375410 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8005e30 con 0x7f65b415c580 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b1ffb700 1 -- 172.21.15.104:0/1377375410 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8006030 con 0x7f65b415c580 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.195+0000 7f65b1ffb700 1 -- 172.21.15.104:0/1377375410 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80063c0 con 0x7f65b415c580 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 -- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b415c580 msgr2=0x7f65b40b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b415c580 0x7f65b40b1070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009820 tx=0x7f65a8009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 -- 172.21.15.104:0/1377375410 shutdown_connections 2023-06-10T18:45:41.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b415c580 0x7f65b40b1070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 --2- 172.21.15.104:0/1377375410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b406e6e0 0x7f65b40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 -- 172.21.15.104:0/1377375410 >> 172.21.15.104:0/1377375410 conn(0x7f65b406b8a0 msgr2=0x7f65b40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 -- 172.21.15.104:0/1377375410 shutdown_connections 2023-06-10T18:45:41.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.196+0000 7f65b9884700 1 -- 172.21.15.104:0/1377375410 wait complete. 2023-06-10T18:45:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.197+0000 7f65b9884700 1 Processor -- start 2023-06-10T18:45:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.197+0000 7f65b9884700 1 -- start start 2023-06-10T18:45:41.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b406e6e0 0x7f65b40c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b415c580 0x7f65b41584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40b2ee0 con 0x7f65b406cde0 2023-06-10T18:45:41.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40ade50 con 0x7f65b406e6e0 2023-06-10T18:45:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b9884700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40ac680 con 0x7f65b415c580 2023-06-10T18:45:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57286/0 (socket says 172.21.15.104:57286) 2023-06-10T18:45:41.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1981985799 learned_addr learned my addr 172.21.15.104:0/1981985799 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b37fe700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b415c580 0x7f65b41584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b27fc700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b406e6e0 0x7f65b40c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b415c580 msgr2=0x7f65b41584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b415c580 0x7f65b41584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b406e6e0 msgr2=0x7f65b40c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b406e6e0 0x7f65b40c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.198+0000 7f65b2ffd700 1 -- 172.21.15.104:0/1981985799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40094a0 con 0x7f65b406cde0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f65b2ffd700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80061a0 con 0x7f65b406cde0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8020070 con 0x7f65b406cde0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a8009430 con 0x7f65b406cde0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8006560 con 0x7f65b406cde0 2023-06-10T18:45:41.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.199+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b40acc40 con 0x7f65b406cde0 2023-06-10T18:45:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.200+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b40d0500 con 0x7f65b406cde0 2023-06-10T18:45:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.200+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f65a8022030 con 0x7f65b406cde0 2023-06-10T18:45:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.200+0000 7f659bfff700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 0x7f65a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.201+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f65a808aa50 con 0x7f65b406cde0 2023-06-10T18:45:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.201+0000 7f65b27fc700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 0x7f65a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.201+0000 7f65b27fc700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 0x7f65a006d190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009cd0 tx=0x7f65a4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.205+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f65a8055990 con 0x7f65b406cde0 2023-06-10T18:45:41.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.436+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f65b4066f10 con 0x7f65b406cde0 2023-06-10T18:45:41.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.440+0000 7f659bfff700 1 -- 172.21.15.104:0/1981985799 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f65a8058fb0 con 0x7f65b406cde0 2023-06-10T18:45:41.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 msgr2=0x7f65a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 0x7f65a006d190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009cd0 tx=0x7f65a4009210 comp rx=0 tx=0).stop 2023-06-10T18:45:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 msgr2=0x7f65b4157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a8004750 comp rx=0 tx=0).stop 2023-06-10T18:45:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 shutdown_connections 2023-06-10T18:45:41.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65b415c580 0x7f65b41584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65b406cde0 0x7f65b4157b10 secure :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a8004750 comp rx=0 tx=0).stop 2023-06-10T18:45:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65a006ace0 0x7f65a006d190 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 --2- 172.21.15.104:0/1981985799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65b406e6e0 0x7f65b40c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.445+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 >> 172.21.15.104:0/1981985799 conn(0x7f65b406b8a0 msgr2=0x7f65b40cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.451+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 shutdown_connections 2023-06-10T18:45:41.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.451+0000 7f65b9884700 1 -- 172.21.15.104:0/1981985799 wait complete. 2023-06-10T18:45:41.460 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 6 and data pool 7 2023-06-10T18:45:41.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.592+0000 7f76ce9e3700 1 Processor -- start 2023-06-10T18:45:41.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.592+0000 7f76ce9e3700 1 -- start start 2023-06-10T18:45:41.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76c806e4b0 0x7f76c80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76c80ca100 0x7f76c80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c8071640 con 0x7f76c80ca100 2023-06-10T18:45:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c80aef00 con 0x7f76c806e4b0 2023-06-10T18:45:41.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ce9e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c806d290 con 0x7f76c806e0e0 2023-06-10T18:45:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76c80ca100 0x7f76c80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f76c80ca100 0x7f76c80ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57298/0 (socket says 172.21.15.104:57298) 2023-06-10T18:45:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 -- 172.21.15.104:0/1587069605 learned_addr learned my addr 172.21.15.104:0/1587069605 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76c77fe700 1 --2- 172.21.15.104:0/1587069605 >> v2:172.21.15.114:3300/0 conn(0x7f76c806e4b0 0x7f76c80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76c7fff700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 -- 172.21.15.104:0/1587069605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 msgr2=0x7f76c806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:41.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 -- 172.21.15.104:0/1587069605 >> v2:172.21.15.114:3300/0 conn(0x7f76c806e4b0 msgr2=0x7f76c80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:41.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 --2- 172.21.15.104:0/1587069605 >> v2:172.21.15.114:3300/0 conn(0x7f76c806e4b0 0x7f76c80c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.593+0000 7f76ccf80700 1 -- 172.21.15.104:0/1587069605 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c806d4c0 con 0x7f76c80ca100 2023-06-10T18:45:41.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ccf80700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c80ca100 0x7f76c80ce9f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f76c80706d0 tx=0x7f76c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3565757ada72a0a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76c6ffd700 1 -- 172.21.15.104:0/1587069605 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c000e070 con 0x7f76c80ca100 2023-06-10T18:45:41.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76c6ffd700 1 -- 172.21.15.104:0/1587069605 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c000b980 con 0x7f76c80ca100 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76c6ffd700 1 -- 172.21.15.104:0/1587069605 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c000bd30 con 0x7f76c80ca100 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 -- 172.21.15.104:0/1587069605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c80ca100 msgr2=0x7f76c80ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c80ca100 0x7f76c80ce9f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f76c80706d0 tx=0x7f76c000b330 comp rx=0 tx=0).stop 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 -- 172.21.15.104:0/1587069605 shutdown_connections 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c80ca100 0x7f76c80ce9f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/1587069605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/1587069605 >> v2:172.21.15.114:3300/0 conn(0x7f76c806e4b0 0x7f76c80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 -- 172.21.15.104:0/1587069605 >> 172.21.15.104:0/1587069605 conn(0x7f76c806b8a0 msgr2=0x7f76c80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 -- 172.21.15.104:0/1587069605 shutdown_connections 2023-06-10T18:45:41.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.594+0000 7f76ce9e3700 1 -- 172.21.15.104:0/1587069605 wait complete. 2023-06-10T18:45:41.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.595+0000 7f76ce9e3700 1 Processor -- start 2023-06-10T18:45:41.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 -- start start 2023-06-10T18:45:41.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c80afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76c80ca100 0x7f76c80c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c80c7510 con 0x7f76c806e4b0 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c80c6640 con 0x7f76c80ca100 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.596+0000 7f76ce9e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c80c6890 con 0x7f76c806e0e0 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57302/0 (socket says 172.21.15.104:57302) 2023-06-10T18:45:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 -- 172.21.15.104:0/378457524 learned_addr learned my addr 172.21.15.104:0/378457524 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c7fff700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c80afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76ccf80700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76c80ca100 0x7f76c80c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 -- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 msgr2=0x7f76c80afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c80afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 -- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76c80ca100 msgr2=0x7f76c80c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76c80ca100 0x7f76c80c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 -- 172.21.15.104:0/378457524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c0009b30 con 0x7f76c806e4b0 2023-06-10T18:45:41.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c77fe700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f76b800e810 tx=0x7f76b800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8013610 con 0x7f76c806e4b0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8013770 con 0x7f76c806e4b0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c80c6b70 con 0x7f76c806e4b0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8013b20 con 0x7f76c806e4b0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.597+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c80c7130 con 0x7f76c806e4b0 2023-06-10T18:45:41.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.598+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c80c7650 con 0x7f76c806e4b0 2023-06-10T18:45:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.599+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f76b8013c80 con 0x7f76c806e4b0 2023-06-10T18:45:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.599+0000 7f76c4ff9700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 0x7f76b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.599+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f76b8089270 con 0x7f76c806e4b0 2023-06-10T18:45:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.599+0000 7f76c7fff700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 0x7f76b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.600+0000 7f76c7fff700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 0x7f76b406d190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f76c815c330 tx=0x7f76bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.603+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f76c80c7650 con 0x7f76c806e4b0 2023-06-10T18:45:41.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.790+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76c80c7650 con 0x7f76c806e4b0 2023-06-10T18:45:41.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.791+0000 7f76c4ff9700 1 -- 172.21.15.104:0/378457524 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18561 (secure 0 0 0) 0x7f76c80c7650 con 0x7f76c806e4b0 2023-06-10T18:45:41.797 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:41.798 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":41,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:41.438198+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:41.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 msgr2=0x7f76b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 0x7f76b406d190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f76c815c330 tx=0x7f76bc008040 comp rx=0 tx=0).stop 2023-06-10T18:45:41.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 msgr2=0x7f76c80b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f76b800e810 tx=0x7f76b800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 shutdown_connections 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76b406ace0 0x7f76b406d190 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76c80ca100 0x7f76c80c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76c806e0e0 0x7f76c80afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 --2- 172.21.15.104:0/378457524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76c806e4b0 0x7f76c80b0ab0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 >> 172.21.15.104:0/378457524 conn(0x7f76c806b8a0 msgr2=0x7f76c8072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 shutdown_connections 2023-06-10T18:45:41.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.793+0000 7f76ce9e3700 1 -- 172.21.15.104:0/378457524 wait complete. 2023-06-10T18:45:41.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.926+0000 7f4016aab700 1 Processor -- start 2023-06-10T18:45:41.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.926+0000 7f4016aab700 1 -- start start 2023-06-10T18:45:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 0x7f401006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f401006e4b0 0x7f40100c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 0x7f40100ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4010071640 con 0x7f401006e4b0 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40100aef00 con 0x7f401006e0e0 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4016aab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401006d290 con 0x7f40100ca100 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f401006e4b0 0x7f40100c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f401006e4b0 0x7f40100c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57308/0 (socket says 172.21.15.104:57308) 2023-06-10T18:45:41.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 -- 172.21.15.104:0/177884935 learned_addr learned my addr 172.21.15.104:0/177884935 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4014847700 1 --2- 172.21.15.104:0/177884935 >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 0x7f401006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4015048700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 0x7f40100ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 -- 172.21.15.104:0/177884935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 msgr2=0x7f40100ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 0x7f40100ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 -- 172.21.15.104:0/177884935 >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 msgr2=0x7f401006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 --2- 172.21.15.104:0/177884935 >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 0x7f401006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 -- 172.21.15.104:0/177884935 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401006d4c0 con 0x7f401006e4b0 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4014847700 1 --2- 172.21.15.104:0/177884935 >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 0x7f401006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f4015048700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 0x7f40100ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400ffff700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100c9bc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f401006af50 tx=0x7f3ffc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2c57d46bc681d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.927+0000 7f400f7fe700 1 -- 172.21.15.104:0/177884935 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00b870 con 0x7f401006e4b0 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f400f7fe700 1 -- 172.21.15.104:0/177884935 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc00b9d0 con 0x7f401006e4b0 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f400f7fe700 1 -- 172.21.15.104:0/177884935 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00bd60 con 0x7f401006e4b0 2023-06-10T18:45:41.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 -- 172.21.15.104:0/177884935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 msgr2=0x7f40100c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100c9bc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f401006af50 tx=0x7f3ffc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 -- 172.21.15.104:0/177884935 shutdown_connections 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40100ca100 0x7f40100ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 --2- 172.21.15.104:0/177884935 >> v2:172.21.15.114:3300/0 conn(0x7f401006e0e0 0x7f401006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 --2- 172.21.15.104:0/177884935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100c9bc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 -- 172.21.15.104:0/177884935 >> 172.21.15.104:0/177884935 conn(0x7f401006b8a0 msgr2=0x7f40100cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 -- 172.21.15.104:0/177884935 shutdown_connections 2023-06-10T18:45:41.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.928+0000 7f4016aab700 1 -- 172.21.15.104:0/177884935 wait complete. 2023-06-10T18:45:41.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.929+0000 7f4016aab700 1 Processor -- start 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.929+0000 7f4016aab700 1 -- start start 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f401006e0e0 0x7f401014aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40100ca100 0x7f40101f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100ac360 con 0x7f401006e4b0 2023-06-10T18:45:41.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100ac640 con 0x7f40100ca100 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4016aab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100ac920 con 0x7f401006e0e0 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57318/0 (socket says 172.21.15.104:57318) 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 -- 172.21.15.104:0/2294080174 learned_addr learned my addr 172.21.15.104:0/2294080174 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f4015048700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40100ca100 0x7f40101f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 -- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f401006e0e0 msgr2=0x7f401014aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f401006e0e0 0x7f401014aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 -- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40100ca100 msgr2=0x7f40101f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.930+0000 7f400ffff700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40100ca100 0x7f40101f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f400ffff700 1 -- 172.21.15.104:0/2294080174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc009550 con 0x7f401006e4b0 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f400ffff700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f401006cc20 tx=0x7f3ffc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0117a0 con 0x7f401006e4b0 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc01f070 con 0x7f401006e4b0 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40100acc00 con 0x7f401006e4b0 2023-06-10T18:45:41.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc011b00 con 0x7f401006e4b0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.931+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401014afb0 con 0x7f401006e4b0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.932+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40100ad120 con 0x7f401006e4b0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.932+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3ffc021030 con 0x7f401006e4b0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.933+0000 7f400d7fa700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 0x7f3ff406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.933+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3ffc08aab0 con 0x7f401006e4b0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.933+0000 7f4014847700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 0x7f3ff406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:41.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.933+0000 7f4014847700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 0x7f3ff406d190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f400400a890 tx=0x7f4004008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:41.937+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40100ad120 con 0x7f401006e4b0 2023-06-10T18:45:42.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.126+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40100ad120 con 0x7f401006e4b0 2023-06-10T18:45:42.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.127+0000 7f400d7fa700 1 -- 172.21.15.104:0/2294080174 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18561 (secure 0 0 0) 0x7f40100ad120 con 0x7f401006e4b0 2023-06-10T18:45:42.133 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:42.133 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":41,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:41.438198+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:42.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 msgr2=0x7f3ff406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:42.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 0x7f3ff406d190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f400400a890 tx=0x7f4004008040 comp rx=0 tx=0).stop 2023-06-10T18:45:42.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 msgr2=0x7f40100ad400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:42.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f401006cc20 tx=0x7f3ffc004750 comp rx=0 tx=0).stop 2023-06-10T18:45:42.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 shutdown_connections 2023-06-10T18:45:42.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3ff406ace0 0x7f3ff406d190 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40100ca100 0x7f40101f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f401006e0e0 0x7f401014aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 --2- 172.21.15.104:0/2294080174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f401006e4b0 0x7f40100ad400 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 >> 172.21.15.104:0/2294080174 conn(0x7f401006b8a0 msgr2=0x7f40100cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:42.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 shutdown_connections 2023-06-10T18:45:42.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.129+0000 7f4016aab700 1 -- 172.21.15.104:0/2294080174 wait complete. 2023-06-10T18:45:42.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:45:42.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.264+0000 7f5de8e7c700 1 Processor -- start 2023-06-10T18:45:42.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.264+0000 7f5de8e7c700 1 -- start start 2023-06-10T18:45:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5de40af940 0x7f5de415e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5de415e720 0x7f5de40cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de415dab0 con 0x7f5de40af940 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de415aa00 con 0x7f5de415e720 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de8e7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4154970 con 0x7f5de41545a0 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de1d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5de40af940 0x7f5de415e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de1d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5de40af940 0x7f5de415e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57322/0 (socket says 172.21.15.104:57322) 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de1d9b700 1 -- 172.21.15.104:0/42410793 learned_addr learned my addr 172.21.15.104:0/42410793 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:42.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de2d9d700 1 --2- 172.21.15.104:0/42410793 >> v2:172.21.15.114:3300/0 conn(0x7f5de415e720 0x7f5de40cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.265+0000 7f5de259c700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 -- 172.21.15.104:0/42410793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 msgr2=0x7f5de415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de415a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 -- 172.21.15.104:0/42410793 >> v2:172.21.15.114:3300/0 conn(0x7f5de415e720 msgr2=0x7f5de40cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/42410793 >> v2:172.21.15.114:3300/0 conn(0x7f5de415e720 0x7f5de40cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 -- 172.21.15.104:0/42410793 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de40cb0b0 con 0x7f5de40af940 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de415e1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5de4072150 tx=0x7f5dd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b6107c05129f01 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:42.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de159a700 1 -- 172.21.15.104:0/42410793 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400b870 con 0x7f5de40af940 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de159a700 1 -- 172.21.15.104:0/42410793 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd400b9d0 con 0x7f5de40af940 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de159a700 1 -- 172.21.15.104:0/42410793 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400bd60 con 0x7f5de40af940 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 -- 172.21.15.104:0/42410793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 msgr2=0x7f5de415e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de415e1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5de4072150 tx=0x7f5dd4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 -- 172.21.15.104:0/42410793 shutdown_connections 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/42410793 >> v2:172.21.15.114:3300/0 conn(0x7f5de415e720 0x7f5de40cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de415a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.266+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/42410793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de415e1e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.267+0000 7f5de8e7c700 1 -- 172.21.15.104:0/42410793 >> 172.21.15.104:0/42410793 conn(0x7f5de406e370 msgr2=0x7f5de406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.267+0000 7f5de8e7c700 1 -- 172.21.15.104:0/42410793 shutdown_connections 2023-06-10T18:45:42.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.267+0000 7f5de8e7c700 1 -- 172.21.15.104:0/42410793 wait complete. 2023-06-10T18:45:42.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.268+0000 7f5de8e7c700 1 Processor -- start 2023-06-10T18:45:42.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.268+0000 7f5de8e7c700 1 -- start start 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de41573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5de415e720 0x7f5de40c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4154d40 con 0x7f5de40af940 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4155020 con 0x7f5de415e720 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4155300 con 0x7f5de41545a0 2023-06-10T18:45:42.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57328/0 (socket says 172.21.15.104:57328) 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 -- 172.21.15.104:0/854915865 learned_addr learned my addr 172.21.15.104:0/854915865 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de2d9d700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5de415e720 0x7f5de40c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de41573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 -- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 msgr2=0x7f5de41573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de41573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 -- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5de415e720 msgr2=0x7f5de40c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5de415e720 0x7f5de40c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 -- 172.21.15.104:0/854915865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd4009550 con 0x7f5de40af940 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de259c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d800 tx=0x7f5dd800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80125f0 con 0x7f5de40af940 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd8012750 con 0x7f5de40af940 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de41578e0 con 0x7f5de40af940 2023-06-10T18:45:42.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8012ae0 con 0x7f5de40af940 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.269+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de40c5d60 con 0x7f5de40af940 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.270+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de406f280 con 0x7f5de40af940 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.271+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5dd8012c40 con 0x7f5de40af940 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.271+0000 7f5dd2ffd700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 0x7f5dc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.271+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5dd80894d0 con 0x7f5de40af940 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.271+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 0x7f5dc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:42.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.272+0000 7f5de1d9b700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 0x7f5dc406d190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4014040 tx=0x7f5dd4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:42.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.275+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5dd8054580 con 0x7f5de40af940 2023-06-10T18:45:42.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:42.507+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5de4154e80 con 0x7f5de40af940 2023-06-10T18:45:43.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.454+0000 7f5dd2ffd700 1 -- 172.21.15.104:0/854915865 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5de4154e80 con 0x7f5de40af940 2023-06-10T18:45:43.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 msgr2=0x7f5dc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 0x7f5dc406d190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4014040 tx=0x7f5dd4008030 comp rx=0 tx=0).stop 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 msgr2=0x7f5de406af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d800 tx=0x7f5dd800db10 comp rx=0 tx=0).stop 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 shutdown_connections 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc406ace0 0x7f5dc406d190 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5de415e720 0x7f5de40c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5de40af940 0x7f5de406af80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 --2- 172.21.15.104:0/854915865 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5de41545a0 0x7f5de41573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 >> 172.21.15.104:0/854915865 conn(0x7f5de406e370 msgr2=0x7f5de4072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 shutdown_connections 2023-06-10T18:45:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.457+0000 7f5de8e7c700 1 -- 172.21.15.104:0/854915865 wait complete. 2023-06-10T18:45:43.477 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:45:43.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:45:43.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.600+0000 7f997b1d8700 1 Processor -- start 2023-06-10T18:45:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.601+0000 7f997b1d8700 1 -- start start 2023-06-10T18:45:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f997b1d8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f997b1d8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 0x7f997406d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f997b1d8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997415cb00 con 0x7f99740cd720 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f997b1d8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99740cdb90 con 0x7f99740ae230 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f997b1d8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740acd90 con 0x7f997415c730 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9973fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f99740cd720 0x7f997406cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55166/0 (socket says 172.21.15.104:55166) 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9973fff700 1 -- 172.21.15.104:0/4104898458 learned_addr learned my addr 172.21.15.104:0/4104898458 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 --2- 172.21.15.104:0/4104898458 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9979775700 1 --2- 172.21.15.104:0/4104898458 >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 0x7f997406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f99737fe700 1 -- 172.21.15.104:0/4104898458 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813571537 0 0) 0x7f9960001110 con 0x7f99740cd720 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f99737fe700 1 -- 172.21.15.104:0/4104898458 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99640035b0 con 0x7f99740cd720 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 -- 172.21.15.104:0/4104898458 >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 msgr2=0x7f997406d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 --2- 172.21.15.104:0/4104898458 >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 0x7f997406d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 -- 172.21.15.104:0/4104898458 >> v1:172.21.15.104:6789/0 conn(0x7f99740cd720 legacy=0x7f997406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 -- 172.21.15.104:0/4104898458 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740ac460 con 0x7f997415c730 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9979775700 1 --2- 172.21.15.104:0/4104898458 >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 0x7f997406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:43.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.602+0000 7f9978f74700 1 --2- 172.21.15.104:0/4104898458 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740cd1e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9968009820 tx=0x7f9968009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5e5689ef91b79 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f99737fe700 1 -- 172.21.15.104:0/4104898458 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968005e30 con 0x7f997415c730 2023-06-10T18:45:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f99737fe700 1 -- 172.21.15.104:0/4104898458 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968006030 con 0x7f997415c730 2023-06-10T18:45:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f99737fe700 1 -- 172.21.15.104:0/4104898458 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680063a0 con 0x7f997415c730 2023-06-10T18:45:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 -- 172.21.15.104:0/4104898458 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 msgr2=0x7f99740cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 --2- 172.21.15.104:0/4104898458 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740cd1e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9968009820 tx=0x7f9968009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 -- 172.21.15.104:0/4104898458 shutdown_connections 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 --2- 172.21.15.104:0/4104898458 >> v2:172.21.15.114:3300/0 conn(0x7f99740ae230 0x7f997406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 --2- 172.21.15.104:0/4104898458 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740cd1e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 -- 172.21.15.104:0/4104898458 >> 172.21.15.104:0/4104898458 conn(0x7f997406b8a0 msgr2=0x7f99740cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 -- 172.21.15.104:0/4104898458 shutdown_connections 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.603+0000 7f997b1d8700 1 -- 172.21.15.104:0/4104898458 wait complete. 2023-06-10T18:45:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.604+0000 7f997b1d8700 1 Processor -- start 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.604+0000 7f997b1d8700 1 -- start start 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99740ae230 0x7f9974157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99741e39a0 con 0x7f99740cd720 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974157dd0 con 0x7f99740ae230 2023-06-10T18:45:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f997b1d8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740c9a00 con 0x7f997415c730 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f9973fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f9973fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57344/0 (socket says 172.21.15.104:57344) 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.605+0000 7f9973fff700 1 -- 172.21.15.104:0/1863417096 learned_addr learned my addr 172.21.15.104:0/1863417096 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9979775700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9978f74700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99740ae230 0x7f9974157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 -- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 msgr2=0x7f99740b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 -- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99740ae230 msgr2=0x7f9974157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99740ae230 0x7f9974157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 -- 172.21.15.104:0/1863417096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c009a70 con 0x7f99740cd720 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9979775700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f9973fff700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f996000c4d0 tx=0x7f996000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960005d70 con 0x7f99740cd720 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99600070d0 con 0x7f99740cd720 2023-06-10T18:45:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9968009430 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960007480 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.606+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99741e3bb0 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.607+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974074050 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.607+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f99600075e0 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.608+0000 7f99717fa700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 0x7f995806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.608+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9960088a70 con 0x7f99740cd720 2023-06-10T18:45:43.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.608+0000 7f9978f74700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 0x7f995806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:43.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.609+0000 7f9978f74700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 0x7f995806d190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9968013040 tx=0x7f9968008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:43.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.612+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9960053960 con 0x7f99740cd720 2023-06-10T18:45:43.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:43.843+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f997415cb00 con 0x7f99740cd720 2023-06-10T18:45:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.531+0000 7f99717fa700 1 -- 172.21.15.104:0/1863417096 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f997415cb00 con 0x7f99740cd720 2023-06-10T18:45:44.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 msgr2=0x7f995806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 0x7f995806d190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9968013040 tx=0x7f9968008070 comp rx=0 tx=0).stop 2023-06-10T18:45:44.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 msgr2=0x7f99740b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f996000c4d0 tx=0x7f996000c890 comp rx=0 tx=0).stop 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 shutdown_connections 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f995806ace0 0x7f995806d190 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99740ae230 0x7f9974157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f997415c730 0x7f99740b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 --2- 172.21.15.104:0/1863417096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99740cd720 0x7f99740b0d30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 >> 172.21.15.104:0/1863417096 conn(0x7f997406b8a0 msgr2=0x7f99740b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:44.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 shutdown_connections 2023-06-10T18:45:44.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.534+0000 7f997b1d8700 1 -- 172.21.15.104:0/1863417096 wait complete. 2023-06-10T18:45:44.553 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:45:44.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:44.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.675+0000 7f1a57fbf700 1 Processor -- start 2023-06-10T18:45:44.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.675+0000 7f1a57fbf700 1 -- start start 2023-06-10T18:45:44.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a57fbf700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 0x7f1a50070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a57fbf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a500ce510 0x7f1a500ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a57fbf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5015c050 con 0x7f1a500c8c40 2023-06-10T18:45:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a57fbf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a500ceed0 con 0x7f1a500ce510 2023-06-10T18:45:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a57fbf700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a500c9970 con 0x7f1a5015bc80 2023-06-10T18:45:44.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a500ce510 0x7f1a500ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5655c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1a500c8c40 0x7f1a500c90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55180/0 (socket says 172.21.15.104:55180) 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5655c700 1 -- 172.21.15.104:0/3546749484 learned_addr learned my addr 172.21.15.104:0/3546749484 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a55d5b700 1 --2- 172.21.15.104:0/3546749484 >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 0x7f1a50070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 -- 172.21.15.104:0/3546749484 >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 msgr2=0x7f1a50070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 --2- 172.21.15.104:0/3546749484 >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 0x7f1a50070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 -- 172.21.15.104:0/3546749484 >> v1:172.21.15.104:6789/0 conn(0x7f1a500c8c40 legacy=0x7f1a500c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 -- 172.21.15.104:0/3546749484 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500ccb70 con 0x7f1a500ce510 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a54d59700 1 -- 172.21.15.104:0/3546749484 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18853246 0 0) 0x7f1a5015c050 con 0x7f1a500c8c40 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a55d5b700 1 --2- 172.21.15.104:0/3546749484 >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 0x7f1a50070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a5555a700 1 --2- 172.21.15.104:0/3546749484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a500ce990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1a500cc710 tx=0x7f1a40009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f58a2866acc3d5f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.676+0000 7f1a54d59700 1 -- 172.21.15.104:0/3546749484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40011830 con 0x7f1a500ce510 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a54d59700 1 -- 172.21.15.104:0/3546749484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a40011990 con 0x7f1a500ce510 2023-06-10T18:45:44.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a54d59700 1 -- 172.21.15.104:0/3546749484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40011d20 con 0x7f1a500ce510 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 -- 172.21.15.104:0/3546749484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 msgr2=0x7f1a500ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/3546749484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a500ce990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1a500cc710 tx=0x7f1a40009b20 comp rx=0 tx=0).stop 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 -- 172.21.15.104:0/3546749484 shutdown_connections 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/3546749484 >> v2:172.21.15.114:3300/0 conn(0x7f1a5015bc80 0x7f1a50070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/3546749484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a500ce990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 -- 172.21.15.104:0/3546749484 >> 172.21.15.104:0/3546749484 conn(0x7f1a5006ab00 msgr2=0x7f1a500b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:44.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 -- 172.21.15.104:0/3546749484 shutdown_connections 2023-06-10T18:45:44.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.677+0000 7f1a57fbf700 1 -- 172.21.15.104:0/3546749484 wait complete. 2023-06-10T18:45:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.678+0000 7f1a57fbf700 1 Processor -- start 2023-06-10T18:45:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.678+0000 7f1a57fbf700 1 -- start start 2023-06-10T18:45:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a500c8c40 0x7f1a5014c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5015bc80 0x7f1a501637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5014ddc0 con 0x7f1a500ce510 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5014e0a0 con 0x7f1a500c8c40 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a57fbf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5014ba10 con 0x7f1a5015bc80 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57358/0 (socket says 172.21.15.104:57358) 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 -- 172.21.15.104:0/2869192516 learned_addr learned my addr 172.21.15.104:0/2869192516 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:44.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 -- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5015bc80 msgr2=0x7f1a501637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5015bc80 0x7f1a501637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 -- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a500c8c40 msgr2=0x7f1a5014c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a55d5b700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a500c8c40 0x7f1a5014c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a500c8c40 0x7f1a5014c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.679+0000 7f1a5555a700 1 -- 172.21.15.104:0/2869192516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a400094a0 con 0x7f1a500ce510 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a5555a700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1a400090b0 tx=0x7f1a40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:44.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a400038a0 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4001f070 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5014bcf0 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40003c00 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.680+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5014d5b0 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.681+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5014c210 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.681+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a4001a430 con 0x7f1a500ce510 2023-06-10T18:45:44.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.682+0000 7f1a467fc700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 0x7f1a3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.682+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1a40021070 con 0x7f1a500ce510 2023-06-10T18:45:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.682+0000 7f1a55d5b700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 0x7f1a3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.682+0000 7f1a55d5b700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 0x7f1a3406d190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a501f3870 tx=0x7f1a4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:44.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.686+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1a5014c210 con 0x7f1a500ce510 2023-06-10T18:45:44.868 INFO:tasks.ceph.mds.b.smithi114.stderr:2023-06-10T18:45:44.863+0000 7f647c7be700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:45:44.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.916+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5014c210 con 0x7f1a500ce510 2023-06-10T18:45:44.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.917+0000 7f1a467fc700 1 -- 172.21.15.104:0/2869192516 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f1a5014c210 con 0x7f1a500ce510 2023-06-10T18:45:44.924 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:44.925 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:45:44.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 msgr2=0x7f1a3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 0x7f1a3406d190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a501f3870 tx=0x7f1a4c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:44.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 msgr2=0x7f1a5014d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:44.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1a400090b0 tx=0x7f1a40004750 comp rx=0 tx=0).stop 2023-06-10T18:45:44.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 shutdown_connections 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a3406ace0 0x7f1a3406d190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a500c8c40 0x7f1a5014c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5015bc80 0x7f1a501637f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 --2- 172.21.15.104:0/2869192516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a500ce510 0x7f1a5014d070 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 >> 172.21.15.104:0/2869192516 conn(0x7f1a5006ab00 msgr2=0x7f1a50153d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:44.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 shutdown_connections 2023-06-10T18:45:44.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:44.919+0000 7f1a57fbf700 1 -- 172.21.15.104:0/2869192516 wait complete. 2023-06-10T18:45:44.929 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:45:44.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:45.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.057+0000 7fe210d75700 1 Processor -- start 2023-06-10T18:45:45.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.057+0000 7fe210d75700 1 -- start start 2023-06-10T18:45:45.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe210d75700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe210d75700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe210d75700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0c9e60 con 0x7fe20c0aeaa0 2023-06-10T18:45:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe210d75700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c0c9bc0 con 0x7fe20c0cb300 2023-06-10T18:45:45.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe210d75700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0aee70 con 0x7fe20c0cd850 2023-06-10T18:45:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe209d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe20ad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe20ad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:44868/0 (socket says 172.21.15.104:44868) 2023-06-10T18:45:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe209d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:48516/0 (socket says 172.21.15.104:48516) 2023-06-10T18:45:45.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.058+0000 7fe20ad9d700 1 -- 172.21.15.104:0/3539141389 learned_addr learned my addr 172.21.15.104:0/3539141389 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe209d9b700 1 -- 172.21.15.104:0/3539141389 >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 msgr2=0x7fe20c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe209d9b700 1 --2- 172.21.15.104:0/3539141389 >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe209d9b700 1 -- 172.21.15.104:0/3539141389 >> v1:172.21.15.114:6789/0 conn(0x7fe20c0aeaa0 legacy=0x7fe20c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe209d9b700 1 -- 172.21.15.104:0/3539141389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c06e540 con 0x7fe20c0cd850 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe20ad9d700 1 --2- 172.21.15.104:0/3539141389 >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:45.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe209d9b700 1 --2- 172.21.15.104:0/3539141389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009840 tx=0x7fe1fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1336b0888a253651 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe20959a700 1 -- 172.21.15.104:0/3539141389 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011830 con 0x7fe20c0cd850 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe20959a700 1 -- 172.21.15.104:0/3539141389 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc011990 con 0x7fe20c0cd850 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.059+0000 7fe20959a700 1 -- 172.21.15.104:0/3539141389 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc011d20 con 0x7fe20c0cd850 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 -- 172.21.15.104:0/3539141389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 msgr2=0x7fe20c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 --2- 172.21.15.104:0/3539141389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009840 tx=0x7fe1fc009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 -- 172.21.15.104:0/3539141389 shutdown_connections 2023-06-10T18:45:45.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 --2- 172.21.15.104:0/3539141389 >> v2:172.21.15.114:3300/0 conn(0x7fe20c0cb300 0x7fe20c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 --2- 172.21.15.104:0/3539141389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c06cd50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 -- 172.21.15.104:0/3539141389 >> 172.21.15.104:0/3539141389 conn(0x7fe20c06b8a0 msgr2=0x7fe20c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 -- 172.21.15.104:0/3539141389 shutdown_connections 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.060+0000 7fe210d75700 1 -- 172.21.15.104:0/3539141389 wait complete. 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.061+0000 7fe210d75700 1 Processor -- start 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.061+0000 7fe210d75700 1 -- start start 2023-06-10T18:45:45.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe20c0aeaa0 0x7fe20c0c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c0b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c1f5990 con 0x7fe20c0cb300 2023-06-10T18:45:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c1f4ac0 con 0x7fe20c0aeaa0 2023-06-10T18:45:45.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe210d75700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0c2950 con 0x7fe20c0cd850 2023-06-10T18:45:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57370/0 (socket says 172.21.15.104:57370) 2023-06-10T18:45:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 -- 172.21.15.104:0/3515621398 learned_addr learned my addr 172.21.15.104:0/3515621398 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe20a59c700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe20c0aeaa0 0x7fe20c0c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe20ad9d700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c0b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 -- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 msgr2=0x7fe20c0b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c0b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 -- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe20c0aeaa0 msgr2=0x7fe20c0c2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe20c0aeaa0 0x7fe20c0c2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.062+0000 7fe209d9b700 1 -- 172.21.15.104:0/3515621398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204009b30 con 0x7fe20c0cb300 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe209d9b700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0090b0 tx=0x7fe1fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0036a0 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc01e070 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc003a80 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc0094a0 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.063+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c1f4ff0 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.064+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c074050 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.064+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe1fc003be0 con 0x7fe20c0cb300 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.064+0000 7fe1faffd700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 0x7fe1ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.065+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe1fc08a5a0 con 0x7fe20c0cb300 2023-06-10T18:45:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.065+0000 7fe20a59c700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 0x7fe1ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.065+0000 7fe20a59c700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 0x7fe1ec06d190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe200000fb0 tx=0x7fe200009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.069+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe1fc0555a0 con 0x7fe20c0cb300 2023-06-10T18:45:45.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.256+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe20c06c320 con 0x7fe20c0cb300 2023-06-10T18:45:45.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.258+0000 7fe1faffd700 1 -- 172.21.15.104:0/3515621398 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18548 (secure 0 0 0) 0x7fe1fc058bc0 con 0x7fe20c0cb300 2023-06-10T18:45:45.264 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:45.264 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":42,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:42.440030+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:45.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 msgr2=0x7fe1ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 0x7fe1ec06d190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe200000fb0 tx=0x7fe200009040 comp rx=0 tx=0).stop 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 msgr2=0x7fe20c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0090b0 tx=0x7fe1fc004750 comp rx=0 tx=0).stop 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 shutdown_connections 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1ec06ace0 0x7fe1ec06d190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe20c0aeaa0 0x7fe20c0c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20c0cd850 0x7fe20c0b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 --2- 172.21.15.104:0/3515621398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20c0cb300 0x7fe20c0b01a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 >> 172.21.15.104:0/3515621398 conn(0x7fe20c06b8a0 msgr2=0x7fe20c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 shutdown_connections 2023-06-10T18:45:45.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.260+0000 7fe210d75700 1 -- 172.21.15.104:0/3515621398 wait complete. 2023-06-10T18:45:45.278 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:45:45.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:45:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.395+0000 7effb3cbc700 1 Processor -- start 2023-06-10T18:45:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.395+0000 7effb3cbc700 1 -- start start 2023-06-10T18:45:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb3cbc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7effac15a250 0x7effac0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb3cbc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7effac0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb3cbc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac0ccca0 con 0x7effac0cfa00 2023-06-10T18:45:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb3cbc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effac0cee50 con 0x7effac15a250 2023-06-10T18:45:45.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb3cbc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac0cf130 con 0x7effac0cad90 2023-06-10T18:45:45.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb1a58700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7effac0cfa00 0x7effac0cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55204/0 (socket says 172.21.15.104:55204) 2023-06-10T18:45:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.396+0000 7effb1a58700 1 -- 172.21.15.104:0/120204955 learned_addr learned my addr 172.21.15.104:0/120204955 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb1257700 1 --2- 172.21.15.104:0/120204955 >> v2:172.21.15.114:3300/0 conn(0x7effac15a250 0x7effac0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839017955 0 0) 0x7effac0ccca0 con 0x7effac0cfa00 2023-06-10T18:45:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7effac0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff980035b0 con 0x7effac0cfa00 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257066411 0 0) 0x7eff980035b0 con 0x7effac0cfa00 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 msgr2=0x7effac0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7effac0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 >> v2:172.21.15.114:3300/0 conn(0x7effac15a250 msgr2=0x7effac0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 --2- 172.21.15.104:0/120204955 >> v2:172.21.15.114:3300/0 conn(0x7effac15a250 0x7effac0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0cc930 con 0x7effac0cfa00 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7eff9c001670 con 0x7effac0cfa00 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 >> v1:172.21.15.104:6789/0 conn(0x7effac0cfa00 legacy=0x7effac0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff98003a00 0x7eff98003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7eff98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff98004d90 0x7eff98005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac0cee50 con 0x7eff98004d90 2023-06-10T18:45:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac0cf130 con 0x7eff98003a00 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9800ba10 con 0x7effac0cad90 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff98004d90 0x7eff98005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb1a58700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff98003a00 0x7eff98003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb1257700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7eff98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 -- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 msgr2=0x7eff98004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac0cad90 0x7eff98004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 -- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff98003a00 msgr2=0x7eff98003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff98003a00 0x7eff98003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.397+0000 7effb2259700 1 -- 172.21.15.104:0/120204955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980035b0 con 0x7eff98004d90 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb2259700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff98004d90 0x7eff98005200 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7effa4009fc0 tx=0x7eff98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f933050ba4586b39 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa400ec30 con 0x7eff98004d90 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400f040 con 0x7eff98004d90 2023-06-10T18:45:45.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb0a56700 1 -- 172.21.15.104:0/120204955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4014480 con 0x7eff98004d90 2023-06-10T18:45:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 -- 172.21.15.104:0/120204955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff98004d90 msgr2=0x7eff98005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 --2- 172.21.15.104:0/120204955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff98004d90 0x7eff98005200 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7effa4009fc0 tx=0x7eff98005880 comp rx=0 tx=0).stop 2023-06-10T18:45:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb2259700 1 -- 172.21.15.104:0/120204955 reap_dead start 2023-06-10T18:45:45.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 -- 172.21.15.104:0/120204955 shutdown_connections 2023-06-10T18:45:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 -- 172.21.15.104:0/120204955 >> 172.21.15.104:0/120204955 conn(0x7effac06e370 msgr2=0x7effac06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 -- 172.21.15.104:0/120204955 shutdown_connections 2023-06-10T18:45:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.398+0000 7effb3cbc700 1 -- 172.21.15.104:0/120204955 wait complete. 2023-06-10T18:45:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.399+0000 7effb3cbc700 1 Processor -- start 2023-06-10T18:45:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 -- start start 2023-06-10T18:45:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff98004d90 0x7effac0c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 0x7effac1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac1f3a40 con 0x7effac1ef730 2023-06-10T18:45:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac0b3030 con 0x7effac1f35c0 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb3cbc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac1570b0 con 0x7eff98004d90 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb1257700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb1257700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57390/0 (socket says 172.21.15.104:57390) 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.400+0000 7effb1257700 1 -- 172.21.15.104:0/2043903161 learned_addr learned my addr 172.21.15.104:0/2043903161 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 -- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff98004d90 msgr2=0x7effac0c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb2259700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 0x7effac1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1a58700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff98004d90 0x7effac0c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff98004d90 0x7effac0c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 -- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 msgr2=0x7effac1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 0x7effac1502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 -- 172.21.15.104:0/2043903161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac157390 con 0x7effac1ef730 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb2259700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 0x7effac1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb1257700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7effa8011a30 tx=0x7effa8011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8004240 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa80043a0 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac157670 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa800f7f0 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.401+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac0c7160 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.402+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac074050 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.402+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7effa800f950 con 0x7effac1ef730 2023-06-10T18:45:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.403+0000 7effa27fc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 0x7eff9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.403+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7effa8089cc0 con 0x7effac1ef730 2023-06-10T18:45:45.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.403+0000 7effb1a58700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 0x7eff9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.403+0000 7effb1a58700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 0x7eff9006d190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7eff9c001050 tx=0x7eff9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.407+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7effa8054c30 con 0x7effac1ef730 2023-06-10T18:45:45.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.593+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effac0cc5e0 con 0x7eff9006ace0 2023-06-10T18:45:45.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.602+0000 7effa27fc700 1 -- 172.21.15.104:0/2043903161 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+306040 (secure 0 0 0) 0x7eff9c000ed0 con 0x7eff9006ace0 2023-06-10T18:45:45.613 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 msgr2=0x7eff9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 0x7eff9006d190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7eff9c001050 tx=0x7eff9c009040 comp rx=0 tx=0).stop 2023-06-10T18:45:45.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 msgr2=0x7effac1f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7effa8011a30 tx=0x7effa8011df0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 shutdown_connections 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff9006ace0 0x7eff9006d190 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac1f35c0 0x7effac1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff98004d90 0x7effac0c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 --2- 172.21.15.104:0/2043903161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7effac1ef730 0x7effac1f3080 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 >> 172.21.15.104:0/2043903161 conn(0x7effac06e370 msgr2=0x7effac159100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 shutdown_connections 2023-06-10T18:45:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.606+0000 7effb3cbc700 1 -- 172.21.15.104:0/2043903161 wait complete. 2023-06-10T18:45:45.619 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:45:45.624 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-06-10T18:45:45.373915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220340,"kb_used_data":4724,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754380,"statfs":{"total":773094113280,"available":772868485120,"internally_reserved":0,"allocated":4837376,"data_stored":2790680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491539"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891546+0000","last_change":"2023-06-10T18:45:40.891546+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891546+0000","last_undegraded":"2023-06-10T18:45:40.891546+0000","last_fullsized":"2023-06-10T18:45:40.891546+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890299+0000","last_change":"2023-06-10T18:45:40.890299+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890299+0000","last_undegraded":"2023-06-10T18:45:40.890299+0000","last_fullsized":"2023-06-10T18:45:40.890299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893028+0000","last_change":"2023-06-10T18:45:40.893028+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893028+0000","last_undegraded":"2023-06-10T18:45:40.893028+0000","last_fullsized":"2023-06-10T18:45:40.893028+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898813+0000","last_change":"2023-06-10T18:45:40.898813+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898813+0000","last_undegraded":"2023-06-10T18:45:40.898813+0000","last_fullsized":"2023-06-10T18:45:40.898813+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890175+0000","last_change":"2023-06-10T18:45:40.890175+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890175+0000","last_undegraded":"2023-06-10T18:45:40.890175+0000","last_fullsized":"2023-06-10T18:45:40.890175+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892168+0000","last_change":"2023-06-10T18:45:40.892168+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892168+0000","last_undegraded":"2023-06-10T18:45:40.892168+0000","last_fullsized":"2023-06-10T18:45:40.892168+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891355+0000","last_change":"2023-06-10T18:45:40.891355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891355+0000","last_undegraded":"2023-06-10T18:45:40.891355+0000","last_fullsized":"2023-06-10T18:45:40.891355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898139+0000","last_change":"2023-06-10T18:45:40.898139+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898139+0000","last_undegraded":"2023-06-10T18:45:40.898139+0000","last_fullsized":"2023-06-10T18:45:40.898139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899051+0000","last_change":"2023-06-10T18:45:40.899051+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899051+0000","last_undegraded":"2023-06-10T18:45:40.899051+0000","last_fullsized":"2023-06-10T18:45:40.899051+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899223+0000","last_change":"2023-06-10T18:45:40.899223+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899223+0000","last_undegraded":"2023-06-10T18:45:40.899223+0000","last_fullsized":"2023-06-10T18:45:40.899223+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898294+0000","last_change":"2023-06-10T18:45:40.898294+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898294+0000","last_undegraded":"2023-06-10T18:45:40.898294+0000","last_fullsized":"2023-06-10T18:45:40.898294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890617+0000","last_change":"2023-06-10T18:45:40.890617+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890617+0000","last_undegraded":"2023-06-10T18:45:40.890617+0000","last_fullsized":"2023-06-10T18:45:40.890617+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889372+0000","last_change":"2023-06-10T18:45:40.889372+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889372+0000","last_undegraded":"2023-06-10T18:45:40.889372+0000","last_fullsized":"2023-06-10T18:45:40.889372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898070+0000","last_change":"2023-06-10T18:45:40.898070+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898070+0000","last_undegraded":"2023-06-10T18:45:40.898070+0000","last_fullsized":"2023-06-10T18:45:40.898070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891994+0000","last_change":"2023-06-10T18:45:40.891994+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891994+0000","last_undegraded":"2023-06-10T18:45:40.891994+0000","last_fullsized":"2023-06-10T18:45:40.891994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897556+0000","last_change":"2023-06-10T18:45:40.897556+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897556+0000","last_undegraded":"2023-06-10T18:45:40.897556+0000","last_fullsized":"2023-06-10T18:45:40.897556+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893903+0000","last_change":"2023-06-10T18:45:40.893903+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893903+0000","last_undegraded":"2023-06-10T18:45:40.893903+0000","last_fullsized":"2023-06-10T18:45:40.893903+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894160+0000","last_change":"2023-06-10T18:45:40.894160+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894160+0000","last_undegraded":"2023-06-10T18:45:40.894160+0000","last_fullsized":"2023-06-10T18:45:40.894160+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895909+0000","last_change":"2023-06-10T18:45:40.895909+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895909+0000","last_undegraded":"2023-06-10T18:45:40.895909+0000","last_fullsized":"2023-06-10T18:45:40.895909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888440+0000","last_change":"2023-06-10T18:45:40.888440+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888440+0000","last_undegraded":"2023-06-10T18:45:40.888440+0000","last_fullsized":"2023-06-10T18:45:40.888440+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891930+0000","last_change":"2023-06-10T18:45:40.891930+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891930+0000","last_undegraded":"2023-06-10T18:45:40.891930+0000","last_fullsized":"2023-06-10T18:45:40.891930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897471+0000","last_change":"2023-06-10T18:45:40.897471+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897471+0000","last_undegraded":"2023-06-10T18:45:40.897471+0000","last_fullsized":"2023-06-10T18:45:40.897471+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896446+0000","last_change":"2023-06-10T18:45:40.896446+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896446+0000","last_undegraded":"2023-06-10T18:45:40.896446+0000","last_fullsized":"2023-06-10T18:45:40.896446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894403+0000","last_change":"2023-06-10T18:45:40.894403+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894403+0000","last_undegraded":"2023-06-10T18:45:40.894403+0000","last_fullsized":"2023-06-10T18:45:40.894403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896595+0000","last_change":"2023-06-10T18:45:40.896595+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896595+0000","last_undegraded":"2023-06-10T18:45:40.896595+0000","last_fullsized":"2023-06-10T18:45:40.896595+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889402+0000","last_change":"2023-06-10T18:45:40.889402+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889402+0000","last_undegraded":"2023-06-10T18:45:40.889402+0000","last_fullsized":"2023-06-10T18:45:40.889402+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892579+0000","last_change":"2023-06-10T18:45:40.892579+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892579+0000","last_undegraded":"2023-06-10T18:45:40.892579+0000","last_fullsized":"2023-06-10T18:45:40.892579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891526+0000","last_change":"2023-06-10T18:45:40.891526+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891526+0000","last_undegraded":"2023-06-10T18:45:40.891526+0000","last_fullsized":"2023-06-10T18:45:40.891526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895022+0000","last_change":"2023-06-10T18:45:40.895022+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895022+0000","last_undegraded":"2023-06-10T18:45:40.895022+0000","last_fullsized":"2023-06-10T18:45:40.895022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896078+0000","last_change":"2023-06-10T18:45:40.896078+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896078+0000","last_undegraded":"2023-06-10T18:45:40.896078+0000","last_fullsized":"2023-06-10T18:45:40.896078+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891534+0000","last_change":"2023-06-10T18:45:40.891534+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891534+0000","last_undegraded":"2023-06-10T18:45:40.891534+0000","last_fullsized":"2023-06-10T18:45:40.891534+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888941+0000","last_change":"2023-06-10T18:45:40.888941+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888941+0000","last_undegraded":"2023-06-10T18:45:40.888941+0000","last_fullsized":"2023-06-10T18:45:40.888941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894627+0000","last_change":"2023-06-10T18:45:40.894627+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894627+0000","last_undegraded":"2023-06-10T18:45:40.894627+0000","last_fullsized":"2023-06-10T18:45:40.894627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907286+0000","last_change":"2023-06-10T18:45:40.904978+0000","last_active":"2023-06-10T18:45:40.907286+0000","last_peered":"2023-06-10T18:45:40.907286+0000","last_clean":"2023-06-10T18:45:40.907286+0000","last_became_active":"2023-06-10T18:45:40.904678+0000","last_became_peered":"2023-06-10T18:45:40.904678+0000","last_unstale":"2023-06-10T18:45:40.907286+0000","last_undegraded":"2023-06-10T18:45:40.907286+0000","last_fullsized":"2023-06-10T18:45:40.907286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:20.302658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897476+0000","last_change":"2023-06-10T18:45:40.897476+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897476+0000","last_undegraded":"2023-06-10T18:45:40.897476+0000","last_fullsized":"2023-06-10T18:45:40.897476+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906422+0000","last_change":"2023-06-10T18:45:40.904727+0000","last_active":"2023-06-10T18:45:40.906422+0000","last_peered":"2023-06-10T18:45:40.906422+0000","last_clean":"2023-06-10T18:45:40.906422+0000","last_became_active":"2023-06-10T18:45:40.904429+0000","last_became_peered":"2023-06-10T18:45:40.904429+0000","last_unstale":"2023-06-10T18:45:40.906422+0000","last_undegraded":"2023-06-10T18:45:40.906422+0000","last_fullsized":"2023-06-10T18:45:40.906422+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:41.516852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891076+0000","last_change":"2023-06-10T18:45:40.891076+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891076+0000","last_undegraded":"2023-06-10T18:45:40.891076+0000","last_fullsized":"2023-06-10T18:45:40.891076+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914200+0000","last_change":"2023-06-10T18:45:40.908799+0000","last_active":"2023-06-10T18:45:40.914200+0000","last_peered":"2023-06-10T18:45:40.914200+0000","last_clean":"2023-06-10T18:45:40.914200+0000","last_became_active":"2023-06-10T18:45:40.908494+0000","last_became_peered":"2023-06-10T18:45:40.908494+0000","last_unstale":"2023-06-10T18:45:40.914200+0000","last_undegraded":"2023-06-10T18:45:40.914200+0000","last_fullsized":"2023-06-10T18:45:40.914200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894922+0000","last_change":"2023-06-10T18:45:40.894922+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894922+0000","last_undegraded":"2023-06-10T18:45:40.894922+0000","last_fullsized":"2023-06-10T18:45:40.894922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909967+0000","last_change":"2023-06-10T18:45:40.904441+0000","last_active":"2023-06-10T18:45:40.909967+0000","last_peered":"2023-06-10T18:45:40.909967+0000","last_clean":"2023-06-10T18:45:40.909967+0000","last_became_active":"2023-06-10T18:45:40.904169+0000","last_became_peered":"2023-06-10T18:45:40.904169+0000","last_unstale":"2023-06-10T18:45:40.909967+0000","last_undegraded":"2023-06-10T18:45:40.909967+0000","last_fullsized":"2023-06-10T18:45:40.909967+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:33.234927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895645+0000","last_change":"2023-06-10T18:45:40.895645+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895645+0000","last_undegraded":"2023-06-10T18:45:40.895645+0000","last_fullsized":"2023-06-10T18:45:40.895645+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906515+0000","last_change":"2023-06-10T18:45:40.900293+0000","last_active":"2023-06-10T18:45:40.906515+0000","last_peered":"2023-06-10T18:45:40.906515+0000","last_clean":"2023-06-10T18:45:40.906515+0000","last_became_active":"2023-06-10T18:45:40.900103+0000","last_became_peered":"2023-06-10T18:45:40.900103+0000","last_unstale":"2023-06-10T18:45:40.906515+0000","last_undegraded":"2023-06-10T18:45:40.906515+0000","last_fullsized":"2023-06-10T18:45:40.906515+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897395+0000","last_change":"2023-06-10T18:45:40.897395+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897395+0000","last_undegraded":"2023-06-10T18:45:40.897395+0000","last_fullsized":"2023-06-10T18:45:40.897395+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905107+0000","last_change":"2023-06-10T18:45:40.903527+0000","last_active":"2023-06-10T18:45:40.905107+0000","last_peered":"2023-06-10T18:45:40.905107+0000","last_clean":"2023-06-10T18:45:40.905107+0000","last_became_active":"2023-06-10T18:45:40.903286+0000","last_became_peered":"2023-06-10T18:45:40.903286+0000","last_unstale":"2023-06-10T18:45:40.905107+0000","last_undegraded":"2023-06-10T18:45:40.905107+0000","last_fullsized":"2023-06-10T18:45:40.905107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:47.654330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895755+0000","last_change":"2023-06-10T18:45:40.895755+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895755+0000","last_undegraded":"2023-06-10T18:45:40.895755+0000","last_fullsized":"2023-06-10T18:45:40.895755+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912706+0000","last_change":"2023-06-10T18:45:40.909125+0000","last_active":"2023-06-10T18:45:40.912706+0000","last_peered":"2023-06-10T18:45:40.912706+0000","last_clean":"2023-06-10T18:45:40.912706+0000","last_became_active":"2023-06-10T18:45:40.908861+0000","last_became_peered":"2023-06-10T18:45:40.908861+0000","last_unstale":"2023-06-10T18:45:40.912706+0000","last_undegraded":"2023-06-10T18:45:40.912706+0000","last_fullsized":"2023-06-10T18:45:40.912706+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:35.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896005+0000","last_change":"2023-06-10T18:45:40.896005+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896005+0000","last_undegraded":"2023-06-10T18:45:40.896005+0000","last_fullsized":"2023-06-10T18:45:40.896005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.915588+0000","last_change":"2023-06-10T18:45:40.914575+0000","last_active":"2023-06-10T18:45:40.915588+0000","last_peered":"2023-06-10T18:45:40.915588+0000","last_clean":"2023-06-10T18:45:40.915588+0000","last_became_active":"2023-06-10T18:45:40.914270+0000","last_became_peered":"2023-06-10T18:45:40.914270+0000","last_unstale":"2023-06-10T18:45:40.915588+0000","last_undegraded":"2023-06-10T18:45:40.915588+0000","last_fullsized":"2023-06-10T18:45:40.915588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:12:15.904314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892405+0000","last_change":"2023-06-10T18:45:40.892405+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892405+0000","last_undegraded":"2023-06-10T18:45:40.892405+0000","last_fullsized":"2023-06-10T18:45:40.892405+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.904714+0000","last_change":"2023-06-10T18:45:40.902582+0000","last_active":"2023-06-10T18:45:40.904714+0000","last_peered":"2023-06-10T18:45:40.904714+0000","last_clean":"2023-06-10T18:45:40.904714+0000","last_became_active":"2023-06-10T18:45:40.902303+0000","last_became_peered":"2023-06-10T18:45:40.902303+0000","last_unstale":"2023-06-10T18:45:40.904714+0000","last_undegraded":"2023-06-10T18:45:40.904714+0000","last_fullsized":"2023-06-10T18:45:40.904714+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:54:45.433721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891086+0000","last_change":"2023-06-10T18:45:40.891086+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891086+0000","last_undegraded":"2023-06-10T18:45:40.891086+0000","last_fullsized":"2023-06-10T18:45:40.891086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906993+0000","last_change":"2023-06-10T18:45:40.905531+0000","last_active":"2023-06-10T18:45:40.906993+0000","last_peered":"2023-06-10T18:45:40.906993+0000","last_clean":"2023-06-10T18:45:40.906993+0000","last_became_active":"2023-06-10T18:45:40.905204+0000","last_became_peered":"2023-06-10T18:45:40.905204+0000","last_unstale":"2023-06-10T18:45:40.906993+0000","last_undegraded":"2023-06-10T18:45:40.906993+0000","last_fullsized":"2023-06-10T18:45:40.906993+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:20.447212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891468+0000","last_change":"2023-06-10T18:45:40.891468+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891468+0000","last_undegraded":"2023-06-10T18:45:40.891468+0000","last_fullsized":"2023-06-10T18:45:40.891468+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912627+0000","last_change":"2023-06-10T18:45:40.909552+0000","last_active":"2023-06-10T18:45:40.912627+0000","last_peered":"2023-06-10T18:45:40.912627+0000","last_clean":"2023-06-10T18:45:40.912627+0000","last_became_active":"2023-06-10T18:45:40.909075+0000","last_became_peered":"2023-06-10T18:45:40.909075+0000","last_unstale":"2023-06-10T18:45:40.912627+0000","last_undegraded":"2023-06-10T18:45:40.912627+0000","last_fullsized":"2023-06-10T18:45:40.912627+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:00.118012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891697+0000","last_change":"2023-06-10T18:45:40.891697+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891697+0000","last_undegraded":"2023-06-10T18:45:40.891697+0000","last_fullsized":"2023-06-10T18:45:40.891697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.902608+0000","last_change":"2023-06-10T18:45:40.899865+0000","last_active":"2023-06-10T18:45:40.902608+0000","last_peered":"2023-06-10T18:45:40.902608+0000","last_clean":"2023-06-10T18:45:40.902608+0000","last_became_active":"2023-06-10T18:45:40.899613+0000","last_became_peered":"2023-06-10T18:45:40.899613+0000","last_unstale":"2023-06-10T18:45:40.902608+0000","last_undegraded":"2023-06-10T18:45:40.902608+0000","last_fullsized":"2023-06-10T18:45:40.902608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:11.696529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897259+0000","last_change":"2023-06-10T18:45:40.897259+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897259+0000","last_undegraded":"2023-06-10T18:45:40.897259+0000","last_fullsized":"2023-06-10T18:45:40.897259+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908200+0000","last_change":"2023-06-10T18:45:40.906074+0000","last_active":"2023-06-10T18:45:40.908200+0000","last_peered":"2023-06-10T18:45:40.908200+0000","last_clean":"2023-06-10T18:45:40.908200+0000","last_became_active":"2023-06-10T18:45:40.905809+0000","last_became_peered":"2023-06-10T18:45:40.905809+0000","last_unstale":"2023-06-10T18:45:40.908200+0000","last_undegraded":"2023-06-10T18:45:40.908200+0000","last_fullsized":"2023-06-10T18:45:40.908200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:52:40.294776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895634+0000","last_change":"2023-06-10T18:45:40.895634+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895634+0000","last_undegraded":"2023-06-10T18:45:40.895634+0000","last_fullsized":"2023-06-10T18:45:40.895634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908195+0000","last_change":"2023-06-10T18:45:40.903305+0000","last_active":"2023-06-10T18:45:40.908195+0000","last_peered":"2023-06-10T18:45:40.908195+0000","last_clean":"2023-06-10T18:45:40.908195+0000","last_became_active":"2023-06-10T18:45:40.903030+0000","last_became_peered":"2023-06-10T18:45:40.903030+0000","last_unstale":"2023-06-10T18:45:40.908195+0000","last_undegraded":"2023-06-10T18:45:40.908195+0000","last_fullsized":"2023-06-10T18:45:40.908195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891503+0000","last_change":"2023-06-10T18:45:40.891503+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891503+0000","last_undegraded":"2023-06-10T18:45:40.891503+0000","last_fullsized":"2023-06-10T18:45:40.891503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908226+0000","last_change":"2023-06-10T18:45:40.906223+0000","last_active":"2023-06-10T18:45:40.908226+0000","last_peered":"2023-06-10T18:45:40.908226+0000","last_clean":"2023-06-10T18:45:40.908226+0000","last_became_active":"2023-06-10T18:45:40.905829+0000","last_became_peered":"2023-06-10T18:45:40.905829+0000","last_unstale":"2023-06-10T18:45:40.908226+0000","last_undegraded":"2023-06-10T18:45:40.908226+0000","last_fullsized":"2023-06-10T18:45:40.908226+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890820+0000","last_change":"2023-06-10T18:45:40.890820+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890820+0000","last_undegraded":"2023-06-10T18:45:40.890820+0000","last_fullsized":"2023-06-10T18:45:40.890820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906524+0000","last_change":"2023-06-10T18:45:40.904337+0000","last_active":"2023-06-10T18:45:40.906524+0000","last_peered":"2023-06-10T18:45:40.906524+0000","last_clean":"2023-06-10T18:45:40.906524+0000","last_became_active":"2023-06-10T18:45:40.904087+0000","last_became_peered":"2023-06-10T18:45:40.904087+0000","last_unstale":"2023-06-10T18:45:40.906524+0000","last_undegraded":"2023-06-10T18:45:40.906524+0000","last_fullsized":"2023-06-10T18:45:40.906524+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:51.173858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888868+0000","last_change":"2023-06-10T18:45:40.888868+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888868+0000","last_undegraded":"2023-06-10T18:45:40.888868+0000","last_fullsized":"2023-06-10T18:45:40.888868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.895387+0000","last_change":"2023-06-10T18:45:40.892521+0000","last_active":"2023-06-10T18:45:40.895387+0000","last_peered":"2023-06-10T18:45:40.895387+0000","last_clean":"2023-06-10T18:45:40.895387+0000","last_became_active":"2023-06-10T18:45:40.892250+0000","last_became_peered":"2023-06-10T18:45:40.892250+0000","last_unstale":"2023-06-10T18:45:40.895387+0000","last_undegraded":"2023-06-10T18:45:40.895387+0000","last_fullsized":"2023-06-10T18:45:40.895387+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:50.471888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897955+0000","last_change":"2023-06-10T18:45:40.897955+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897955+0000","last_undegraded":"2023-06-10T18:45:40.897955+0000","last_fullsized":"2023-06-10T18:45:40.897955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910052+0000","last_change":"2023-06-10T18:45:40.909132+0000","last_active":"2023-06-10T18:45:40.910052+0000","last_peered":"2023-06-10T18:45:40.910052+0000","last_clean":"2023-06-10T18:45:40.910052+0000","last_became_active":"2023-06-10T18:45:40.908770+0000","last_became_peered":"2023-06-10T18:45:40.908770+0000","last_unstale":"2023-06-10T18:45:40.910052+0000","last_undegraded":"2023-06-10T18:45:40.910052+0000","last_fullsized":"2023-06-10T18:45:40.910052+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:58:32.518563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895519+0000","last_change":"2023-06-10T18:45:40.895519+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895519+0000","last_undegraded":"2023-06-10T18:45:40.895519+0000","last_fullsized":"2023-06-10T18:45:40.895519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912749+0000","last_change":"2023-06-10T18:45:40.909792+0000","last_active":"2023-06-10T18:45:40.912749+0000","last_peered":"2023-06-10T18:45:40.912749+0000","last_clean":"2023-06-10T18:45:40.912749+0000","last_became_active":"2023-06-10T18:45:40.909529+0000","last_became_peered":"2023-06-10T18:45:40.909529+0000","last_unstale":"2023-06-10T18:45:40.912749+0000","last_undegraded":"2023-06-10T18:45:40.912749+0000","last_fullsized":"2023-06-10T18:45:40.912749+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:35.699893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895527+0000","last_change":"2023-06-10T18:45:40.895527+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895527+0000","last_undegraded":"2023-06-10T18:45:40.895527+0000","last_fullsized":"2023-06-10T18:45:40.895527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907177+0000","last_change":"2023-06-10T18:45:40.904862+0000","last_active":"2023-06-10T18:45:40.907177+0000","last_peered":"2023-06-10T18:45:40.907177+0000","last_clean":"2023-06-10T18:45:40.907177+0000","last_became_active":"2023-06-10T18:45:40.904590+0000","last_became_peered":"2023-06-10T18:45:40.904590+0000","last_unstale":"2023-06-10T18:45:40.907177+0000","last_undegraded":"2023-06-10T18:45:40.907177+0000","last_fullsized":"2023-06-10T18:45:40.907177+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:36:47.213889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891726+0000","last_change":"2023-06-10T18:45:40.891726+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891726+0000","last_undegraded":"2023-06-10T18:45:40.891726+0000","last_fullsized":"2023-06-10T18:45:40.891726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906313+0000","last_change":"2023-06-10T18:45:40.900179+0000","last_active":"2023-06-10T18:45:40.906313+0000","last_peered":"2023-06-10T18:45:40.906313+0000","last_clean":"2023-06-10T18:45:40.906313+0000","last_became_active":"2023-06-10T18:45:40.899762+0000","last_became_peered":"2023-06-10T18:45:40.899762+0000","last_unstale":"2023-06-10T18:45:40.906313+0000","last_undegraded":"2023-06-10T18:45:40.906313+0000","last_fullsized":"2023-06-10T18:45:40.906313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:25:43.791507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894081+0000","last_change":"2023-06-10T18:45:40.894081+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894081+0000","last_undegraded":"2023-06-10T18:45:40.894081+0000","last_fullsized":"2023-06-10T18:45:40.894081+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910760+0000","last_change":"2023-06-10T18:45:40.904961+0000","last_active":"2023-06-10T18:45:40.910760+0000","last_peered":"2023-06-10T18:45:40.910760+0000","last_clean":"2023-06-10T18:45:40.910760+0000","last_became_active":"2023-06-10T18:45:40.904660+0000","last_became_peered":"2023-06-10T18:45:40.904660+0000","last_unstale":"2023-06-10T18:45:40.910760+0000","last_undegraded":"2023-06-10T18:45:40.910760+0000","last_fullsized":"2023-06-10T18:45:40.910760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:40.145269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888355+0000","last_change":"2023-06-10T18:45:40.888355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888355+0000","last_undegraded":"2023-06-10T18:45:40.888355+0000","last_fullsized":"2023-06-10T18:45:40.888355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907024+0000","last_change":"2023-06-10T18:45:40.905644+0000","last_active":"2023-06-10T18:45:40.907024+0000","last_peered":"2023-06-10T18:45:40.907024+0000","last_clean":"2023-06-10T18:45:40.907024+0000","last_became_active":"2023-06-10T18:45:40.905375+0000","last_became_peered":"2023-06-10T18:45:40.905375+0000","last_unstale":"2023-06-10T18:45:40.907024+0000","last_undegraded":"2023-06-10T18:45:40.907024+0000","last_fullsized":"2023-06-10T18:45:40.907024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:28.816147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895956+0000","last_change":"2023-06-10T18:45:40.895956+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895956+0000","last_undegraded":"2023-06-10T18:45:40.895956+0000","last_fullsized":"2023-06-10T18:45:40.895956+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909939+0000","last_change":"2023-06-10T18:45:40.903928+0000","last_active":"2023-06-10T18:45:40.909939+0000","last_peered":"2023-06-10T18:45:40.909939+0000","last_clean":"2023-06-10T18:45:40.909939+0000","last_became_active":"2023-06-10T18:45:40.903597+0000","last_became_peered":"2023-06-10T18:45:40.903597+0000","last_unstale":"2023-06-10T18:45:40.909939+0000","last_undegraded":"2023-06-10T18:45:40.909939+0000","last_fullsized":"2023-06-10T18:45:40.909939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:04.405310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891920+0000","last_change":"2023-06-10T18:45:40.891920+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891920+0000","last_undegraded":"2023-06-10T18:45:40.891920+0000","last_fullsized":"2023-06-10T18:45:40.891920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.913712+0000","last_change":"2023-06-10T18:45:40.907369+0000","last_active":"2023-06-10T18:45:40.913712+0000","last_peered":"2023-06-10T18:45:40.913712+0000","last_clean":"2023-06-10T18:45:40.913712+0000","last_became_active":"2023-06-10T18:45:40.907099+0000","last_became_peered":"2023-06-10T18:45:40.907099+0000","last_unstale":"2023-06-10T18:45:40.913712+0000","last_undegraded":"2023-06-10T18:45:40.913712+0000","last_fullsized":"2023-06-10T18:45:40.913712+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:57.202185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896128+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:45:40.896128+0000","last_peered":"2023-06-10T18:45:40.896128+0000","last_clean":"2023-06-10T18:45:40.896128+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:45:40.896128+0000","last_undegraded":"2023-06-10T18:45:40.896128+0000","last_fullsized":"2023-06-10T18:45:40.896128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:35:09.859258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897049+0000","last_change":"2023-06-10T18:45:40.897049+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897049+0000","last_undegraded":"2023-06-10T18:45:40.897049+0000","last_fullsized":"2023-06-10T18:45:40.897049+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906525+0000","last_change":"2023-06-10T18:45:40.900220+0000","last_active":"2023-06-10T18:45:40.906525+0000","last_peered":"2023-06-10T18:45:40.906525+0000","last_clean":"2023-06-10T18:45:40.906525+0000","last_became_active":"2023-06-10T18:45:40.899948+0000","last_became_peered":"2023-06-10T18:45:40.899948+0000","last_unstale":"2023-06-10T18:45:40.906525+0000","last_undegraded":"2023-06-10T18:45:40.906525+0000","last_fullsized":"2023-06-10T18:45:40.906525+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:24.478651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.891471+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:45:40.891471+0000","last_peered":"2023-06-10T18:45:40.891471+0000","last_clean":"2023-06-10T18:45:40.891471+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:45:40.891471+0000","last_undegraded":"2023-06-10T18:45:40.891471+0000","last_fullsized":"2023-06-10T18:45:40.891471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:39.411023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897646+0000","last_change":"2023-06-10T18:45:40.897646+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897646+0000","last_undegraded":"2023-06-10T18:45:40.897646+0000","last_fullsized":"2023-06-10T18:45:40.897646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912608+0000","last_change":"2023-06-10T18:45:40.909238+0000","last_active":"2023-06-10T18:45:40.912608+0000","last_peered":"2023-06-10T18:45:40.912608+0000","last_clean":"2023-06-10T18:45:40.912608+0000","last_became_active":"2023-06-10T18:45:40.908968+0000","last_became_peered":"2023-06-10T18:45:40.908968+0000","last_unstale":"2023-06-10T18:45:40.912608+0000","last_undegraded":"2023-06-10T18:45:40.912608+0000","last_fullsized":"2023-06-10T18:45:40.912608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:32.937415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.888090+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:45:40.888090+0000","last_peered":"2023-06-10T18:45:40.888090+0000","last_clean":"2023-06-10T18:45:40.888090+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:45:40.888090+0000","last_undegraded":"2023-06-10T18:45:40.888090+0000","last_fullsized":"2023-06-10T18:45:40.888090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:52:00.598598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898298+0000","last_change":"2023-06-10T18:45:40.898298+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898298+0000","last_undegraded":"2023-06-10T18:45:40.898298+0000","last_fullsized":"2023-06-10T18:45:40.898298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910745+0000","last_change":"2023-06-10T18:45:40.904255+0000","last_active":"2023-06-10T18:45:40.910745+0000","last_peered":"2023-06-10T18:45:40.910745+0000","last_clean":"2023-06-10T18:45:40.910745+0000","last_became_active":"2023-06-10T18:45:40.903971+0000","last_became_peered":"2023-06-10T18:45:40.903971+0000","last_unstale":"2023-06-10T18:45:40.910745+0000","last_undegraded":"2023-06-10T18:45:40.910745+0000","last_fullsized":"2023-06-10T18:45:40.910745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:05.711201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.892795+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:45:40.892795+0000","last_peered":"2023-06-10T18:45:40.892795+0000","last_clean":"2023-06-10T18:45:40.892795+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:45:40.892795+0000","last_undegraded":"2023-06-10T18:45:40.892795+0000","last_fullsized":"2023-06-10T18:45:40.892795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:19.558248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":207,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894146+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:45:40.894146+0000","last_peered":"2023-06-10T18:45:40.894146+0000","last_clean":"2023-06-10T18:45:40.894146+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:45:40.894146+0000","last_undegraded":"2023-06-10T18:45:40.894146+0000","last_fullsized":"2023-06-10T18:45:40.894146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:15:52.844548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890750+0000","last_change":"2023-06-10T18:45:40.890750+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890750+0000","last_undegraded":"2023-06-10T18:45:40.890750+0000","last_fullsized":"2023-06-10T18:45:40.890750+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914162+0000","last_change":"2023-06-10T18:45:40.908782+0000","last_active":"2023-06-10T18:45:40.914162+0000","last_peered":"2023-06-10T18:45:40.914162+0000","last_clean":"2023-06-10T18:45:40.914162+0000","last_became_active":"2023-06-10T18:45:40.908452+0000","last_became_peered":"2023-06-10T18:45:40.908452+0000","last_unstale":"2023-06-10T18:45:40.914162+0000","last_undegraded":"2023-06-10T18:45:40.914162+0000","last_fullsized":"2023-06-10T18:45:40.914162+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:16.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894289+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:45:40.894289+0000","last_peered":"2023-06-10T18:45:40.894289+0000","last_clean":"2023-06-10T18:45:40.894289+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:45:40.894289+0000","last_undegraded":"2023-06-10T18:45:40.894289+0000","last_fullsized":"2023-06-10T18:45:40.894289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:21.044293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889714+0000","last_change":"2023-06-10T18:45:40.889714+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889714+0000","last_undegraded":"2023-06-10T18:45:40.889714+0000","last_fullsized":"2023-06-10T18:45:40.889714+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914181+0000","last_change":"2023-06-10T18:45:40.908831+0000","last_active":"2023-06-10T18:45:40.914181+0000","last_peered":"2023-06-10T18:45:40.914181+0000","last_clean":"2023-06-10T18:45:40.914181+0000","last_became_active":"2023-06-10T18:45:40.908610+0000","last_became_peered":"2023-06-10T18:45:40.908610+0000","last_unstale":"2023-06-10T18:45:40.914181+0000","last_undegraded":"2023-06-10T18:45:40.914181+0000","last_fullsized":"2023-06-10T18:45:40.914181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.898804+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:45:40.898804+0000","last_peered":"2023-06-10T18:45:40.898804+0000","last_clean":"2023-06-10T18:45:40.898804+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:45:40.898804+0000","last_undegraded":"2023-06-10T18:45:40.898804+0000","last_fullsized":"2023-06-10T18:45:40.898804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:29.019593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890847+0000","last_change":"2023-06-10T18:45:40.890847+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890847+0000","last_undegraded":"2023-06-10T18:45:40.890847+0000","last_fullsized":"2023-06-10T18:45:40.890847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905087+0000","last_change":"2023-06-10T18:45:40.903579+0000","last_active":"2023-06-10T18:45:40.905087+0000","last_peered":"2023-06-10T18:45:40.905087+0000","last_clean":"2023-06-10T18:45:40.905087+0000","last_became_active":"2023-06-10T18:45:40.903381+0000","last_became_peered":"2023-06-10T18:45:40.903381+0000","last_unstale":"2023-06-10T18:45:40.905087+0000","last_undegraded":"2023-06-10T18:45:40.905087+0000","last_fullsized":"2023-06-10T18:45:40.905087+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:24.524984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896517+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:45:40.896517+0000","last_peered":"2023-06-10T18:45:40.896517+0000","last_clean":"2023-06-10T18:45:40.896517+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:45:40.896517+0000","last_undegraded":"2023-06-10T18:45:40.896517+0000","last_fullsized":"2023-06-10T18:45:40.896517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:47.124020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890809+0000","last_change":"2023-06-10T18:45:40.890809+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890809+0000","last_undegraded":"2023-06-10T18:45:40.890809+0000","last_fullsized":"2023-06-10T18:45:40.890809+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912550+0000","last_change":"2023-06-10T18:45:40.908946+0000","last_active":"2023-06-10T18:45:40.912550+0000","last_peered":"2023-06-10T18:45:40.912550+0000","last_clean":"2023-06-10T18:45:40.912550+0000","last_became_active":"2023-06-10T18:45:40.908651+0000","last_became_peered":"2023-06-10T18:45:40.908651+0000","last_unstale":"2023-06-10T18:45:40.912550+0000","last_undegraded":"2023-06-10T18:45:40.912550+0000","last_fullsized":"2023-06-10T18:45:40.912550+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:23.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.897714+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:45:40.897714+0000","last_peered":"2023-06-10T18:45:40.897714+0000","last_clean":"2023-06-10T18:45:40.897714+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:45:40.897714+0000","last_undegraded":"2023-06-10T18:45:40.897714+0000","last_fullsized":"2023-06-10T18:45:40.897714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:14:14.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":1056768,"data_stored":794723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.625}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.6850000000000005,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.54400000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.315},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.28599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.497}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":1040384,"data_stored":786764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.245}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.501}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":462848,"data_stored":204355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.442}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.802,"5min":3.802,"15min":3.802},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.42899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.379}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.372},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.501}]},{"osd":4,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:45:45.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:45:45.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.780+0000 7f148f7d4700 1 Processor -- start 2023-06-10T18:45:45.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 -- start start 2023-06-10T18:45:45.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f148815a420 0x7f14880b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f14880ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 0x7f14880b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880ced10 con 0x7f148815a420 2023-06-10T18:45:45.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880b6070 con 0x7f14880ce890 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148f7d4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880b5460 con 0x7f14880b4ff0 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.781+0000 7f148d570700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f148815a420 0x7f14880b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f148815a420 0x7f14880b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57396/0 (socket says 172.21.15.104:57396) 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 -- 172.21.15.104:0/2847727554 learned_addr learned my addr 172.21.15.104:0/2847727554 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148cd6f700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f14880ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148dd71700 1 --2- 172.21.15.104:0/2847727554 >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 0x7f14880b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 -- 172.21.15.104:0/2847727554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 msgr2=0x7f14880ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f14880ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 -- 172.21.15.104:0/2847727554 >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 msgr2=0x7f14880b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:45.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 --2- 172.21.15.104:0/2847727554 >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 0x7f14880b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 -- 172.21.15.104:0/2847727554 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880b5740 con 0x7f148815a420 2023-06-10T18:45:45.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148cd6f700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f14880ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:45.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148dd71700 1 --2- 172.21.15.104:0/2847727554 >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 0x7f14880b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:45.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f148d570700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880b1660 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f148808abf0 tx=0x7f1478009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8874e5d9c36b14e2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f147ffff700 1 -- 172.21.15.104:0/2847727554 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800b870 con 0x7f148815a420 2023-06-10T18:45:45.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f147ffff700 1 -- 172.21.15.104:0/2847727554 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147800b9d0 con 0x7f148815a420 2023-06-10T18:45:45.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.782+0000 7f147ffff700 1 -- 172.21.15.104:0/2847727554 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800bd60 con 0x7f148815a420 2023-06-10T18:45:45.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 -- 172.21.15.104:0/2847727554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 msgr2=0x7f14880b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880b1660 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f148808abf0 tx=0x7f1478009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 -- 172.21.15.104:0/2847727554 shutdown_connections 2023-06-10T18:45:45.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880b1660 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 --2- 172.21.15.104:0/2847727554 >> v2:172.21.15.114:3300/0 conn(0x7f14880ce890 0x7f14880b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 --2- 172.21.15.104:0/2847727554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f14880ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 -- 172.21.15.104:0/2847727554 >> 172.21.15.104:0/2847727554 conn(0x7f1488089fd0 msgr2=0x7f14880d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 -- 172.21.15.104:0/2847727554 shutdown_connections 2023-06-10T18:45:45.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.783+0000 7f148f7d4700 1 -- 172.21.15.104:0/2847727554 wait complete. 2023-06-10T18:45:45.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.784+0000 7f148f7d4700 1 Processor -- start 2023-06-10T18:45:45.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.784+0000 7f148f7d4700 1 -- start start 2023-06-10T18:45:45.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f148814fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14880ce890 0x7f14880c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14881f4110 con 0x7f148815a420 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14881f3770 con 0x7f14880ce890 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14881f3a50 con 0x7f14880b4ff0 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57406/0 (socket says 172.21.15.104:57406) 2023-06-10T18:45:45.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 -- 172.21.15.104:0/1352408004 learned_addr learned my addr 172.21.15.104:0/1352408004 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148d570700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f148814fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148cd6f700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14880ce890 0x7f14880c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 -- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 msgr2=0x7f148814fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f148814fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 -- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14880ce890 msgr2=0x7f14880c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14880ce890 0x7f14880c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 -- 172.21.15.104:0/1352408004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478009550 con 0x7f148815a420 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148dd71700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1480007ef0 tx=0x7f148000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000dd80 con 0x7f148815a420 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000f040 con 0x7f148815a420 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14881f3d30 con 0x7f148815a420 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.785+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480003c60 con 0x7f148815a420 2023-06-10T18:45:45.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.786+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880c6ac0 con 0x7f148815a420 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.787+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14880b64e0 con 0x7f148815a420 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.787+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1480004030 con 0x7f148815a420 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.787+0000 7f147dffb700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 0x7f146c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.788+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1480089670 con 0x7f148815a420 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.788+0000 7f148d570700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 0x7f146c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.788+0000 7f148d570700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 0x7f146c06d190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1478000c00 tx=0x7f1478008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:45.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.792+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f14880b64e0 con 0x7f148815a420 2023-06-10T18:45:45.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.975+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14881f38e0 con 0x7f146c06ace0 2023-06-10T18:45:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.984+0000 7f147dffb700 1 -- 172.21.15.104:0/1352408004 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+306040 (secure 0 0 0) 0x7f14881f38e0 con 0x7f146c06ace0 2023-06-10T18:45:45.990 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 msgr2=0x7f146c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 0x7f146c06d190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1478000c00 tx=0x7f1478008030 comp rx=0 tx=0).stop 2023-06-10T18:45:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 msgr2=0x7f14880c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1480007ef0 tx=0x7f148000d3b0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 shutdown_connections 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f148815a420 0x7f14880c6490 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14880b4ff0 0x7f148814fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f146c06ace0 0x7f146c06d190 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 --2- 172.21.15.104:0/1352408004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14880ce890 0x7f14880c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 >> 172.21.15.104:0/1352408004 conn(0x7f1488089fd0 msgr2=0x7f148806e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 shutdown_connections 2023-06-10T18:45:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:45.987+0000 7f148f7d4700 1 -- 172.21.15.104:0/1352408004 wait complete. 2023-06-10T18:45:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:45:46.005 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-06-10T18:45:45.373915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220340,"kb_used_data":4724,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754380,"statfs":{"total":773094113280,"available":772868485120,"internally_reserved":0,"allocated":4837376,"data_stored":2790680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491539"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891546+0000","last_change":"2023-06-10T18:45:40.891546+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891546+0000","last_undegraded":"2023-06-10T18:45:40.891546+0000","last_fullsized":"2023-06-10T18:45:40.891546+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890299+0000","last_change":"2023-06-10T18:45:40.890299+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890299+0000","last_undegraded":"2023-06-10T18:45:40.890299+0000","last_fullsized":"2023-06-10T18:45:40.890299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893028+0000","last_change":"2023-06-10T18:45:40.893028+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893028+0000","last_undegraded":"2023-06-10T18:45:40.893028+0000","last_fullsized":"2023-06-10T18:45:40.893028+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898813+0000","last_change":"2023-06-10T18:45:40.898813+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898813+0000","last_undegraded":"2023-06-10T18:45:40.898813+0000","last_fullsized":"2023-06-10T18:45:40.898813+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890175+0000","last_change":"2023-06-10T18:45:40.890175+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890175+0000","last_undegraded":"2023-06-10T18:45:40.890175+0000","last_fullsized":"2023-06-10T18:45:40.890175+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892168+0000","last_change":"2023-06-10T18:45:40.892168+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892168+0000","last_undegraded":"2023-06-10T18:45:40.892168+0000","last_fullsized":"2023-06-10T18:45:40.892168+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891355+0000","last_change":"2023-06-10T18:45:40.891355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891355+0000","last_undegraded":"2023-06-10T18:45:40.891355+0000","last_fullsized":"2023-06-10T18:45:40.891355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898139+0000","last_change":"2023-06-10T18:45:40.898139+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898139+0000","last_undegraded":"2023-06-10T18:45:40.898139+0000","last_fullsized":"2023-06-10T18:45:40.898139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899051+0000","last_change":"2023-06-10T18:45:40.899051+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899051+0000","last_undegraded":"2023-06-10T18:45:40.899051+0000","last_fullsized":"2023-06-10T18:45:40.899051+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899223+0000","last_change":"2023-06-10T18:45:40.899223+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899223+0000","last_undegraded":"2023-06-10T18:45:40.899223+0000","last_fullsized":"2023-06-10T18:45:40.899223+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898294+0000","last_change":"2023-06-10T18:45:40.898294+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898294+0000","last_undegraded":"2023-06-10T18:45:40.898294+0000","last_fullsized":"2023-06-10T18:45:40.898294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890617+0000","last_change":"2023-06-10T18:45:40.890617+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890617+0000","last_undegraded":"2023-06-10T18:45:40.890617+0000","last_fullsized":"2023-06-10T18:45:40.890617+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889372+0000","last_change":"2023-06-10T18:45:40.889372+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889372+0000","last_undegraded":"2023-06-10T18:45:40.889372+0000","last_fullsized":"2023-06-10T18:45:40.889372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898070+0000","last_change":"2023-06-10T18:45:40.898070+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898070+0000","last_undegraded":"2023-06-10T18:45:40.898070+0000","last_fullsized":"2023-06-10T18:45:40.898070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891994+0000","last_change":"2023-06-10T18:45:40.891994+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891994+0000","last_undegraded":"2023-06-10T18:45:40.891994+0000","last_fullsized":"2023-06-10T18:45:40.891994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897556+0000","last_change":"2023-06-10T18:45:40.897556+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897556+0000","last_undegraded":"2023-06-10T18:45:40.897556+0000","last_fullsized":"2023-06-10T18:45:40.897556+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893903+0000","last_change":"2023-06-10T18:45:40.893903+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893903+0000","last_undegraded":"2023-06-10T18:45:40.893903+0000","last_fullsized":"2023-06-10T18:45:40.893903+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894160+0000","last_change":"2023-06-10T18:45:40.894160+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894160+0000","last_undegraded":"2023-06-10T18:45:40.894160+0000","last_fullsized":"2023-06-10T18:45:40.894160+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895909+0000","last_change":"2023-06-10T18:45:40.895909+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895909+0000","last_undegraded":"2023-06-10T18:45:40.895909+0000","last_fullsized":"2023-06-10T18:45:40.895909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888440+0000","last_change":"2023-06-10T18:45:40.888440+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888440+0000","last_undegraded":"2023-06-10T18:45:40.888440+0000","last_fullsized":"2023-06-10T18:45:40.888440+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891930+0000","last_change":"2023-06-10T18:45:40.891930+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891930+0000","last_undegraded":"2023-06-10T18:45:40.891930+0000","last_fullsized":"2023-06-10T18:45:40.891930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897471+0000","last_change":"2023-06-10T18:45:40.897471+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897471+0000","last_undegraded":"2023-06-10T18:45:40.897471+0000","last_fullsized":"2023-06-10T18:45:40.897471+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896446+0000","last_change":"2023-06-10T18:45:40.896446+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896446+0000","last_undegraded":"2023-06-10T18:45:40.896446+0000","last_fullsized":"2023-06-10T18:45:40.896446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894403+0000","last_change":"2023-06-10T18:45:40.894403+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894403+0000","last_undegraded":"2023-06-10T18:45:40.894403+0000","last_fullsized":"2023-06-10T18:45:40.894403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896595+0000","last_change":"2023-06-10T18:45:40.896595+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896595+0000","last_undegraded":"2023-06-10T18:45:40.896595+0000","last_fullsized":"2023-06-10T18:45:40.896595+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889402+0000","last_change":"2023-06-10T18:45:40.889402+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889402+0000","last_undegraded":"2023-06-10T18:45:40.889402+0000","last_fullsized":"2023-06-10T18:45:40.889402+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892579+0000","last_change":"2023-06-10T18:45:40.892579+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892579+0000","last_undegraded":"2023-06-10T18:45:40.892579+0000","last_fullsized":"2023-06-10T18:45:40.892579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891526+0000","last_change":"2023-06-10T18:45:40.891526+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891526+0000","last_undegraded":"2023-06-10T18:45:40.891526+0000","last_fullsized":"2023-06-10T18:45:40.891526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895022+0000","last_change":"2023-06-10T18:45:40.895022+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895022+0000","last_undegraded":"2023-06-10T18:45:40.895022+0000","last_fullsized":"2023-06-10T18:45:40.895022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896078+0000","last_change":"2023-06-10T18:45:40.896078+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896078+0000","last_undegraded":"2023-06-10T18:45:40.896078+0000","last_fullsized":"2023-06-10T18:45:40.896078+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891534+0000","last_change":"2023-06-10T18:45:40.891534+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891534+0000","last_undegraded":"2023-06-10T18:45:40.891534+0000","last_fullsized":"2023-06-10T18:45:40.891534+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888941+0000","last_change":"2023-06-10T18:45:40.888941+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888941+0000","last_undegraded":"2023-06-10T18:45:40.888941+0000","last_fullsized":"2023-06-10T18:45:40.888941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894627+0000","last_change":"2023-06-10T18:45:40.894627+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894627+0000","last_undegraded":"2023-06-10T18:45:40.894627+0000","last_fullsized":"2023-06-10T18:45:40.894627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907286+0000","last_change":"2023-06-10T18:45:40.904978+0000","last_active":"2023-06-10T18:45:40.907286+0000","last_peered":"2023-06-10T18:45:40.907286+0000","last_clean":"2023-06-10T18:45:40.907286+0000","last_became_active":"2023-06-10T18:45:40.904678+0000","last_became_peered":"2023-06-10T18:45:40.904678+0000","last_unstale":"2023-06-10T18:45:40.907286+0000","last_undegraded":"2023-06-10T18:45:40.907286+0000","last_fullsized":"2023-06-10T18:45:40.907286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:20.302658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897476+0000","last_change":"2023-06-10T18:45:40.897476+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897476+0000","last_undegraded":"2023-06-10T18:45:40.897476+0000","last_fullsized":"2023-06-10T18:45:40.897476+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906422+0000","last_change":"2023-06-10T18:45:40.904727+0000","last_active":"2023-06-10T18:45:40.906422+0000","last_peered":"2023-06-10T18:45:40.906422+0000","last_clean":"2023-06-10T18:45:40.906422+0000","last_became_active":"2023-06-10T18:45:40.904429+0000","last_became_peered":"2023-06-10T18:45:40.904429+0000","last_unstale":"2023-06-10T18:45:40.906422+0000","last_undegraded":"2023-06-10T18:45:40.906422+0000","last_fullsized":"2023-06-10T18:45:40.906422+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:41.516852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891076+0000","last_change":"2023-06-10T18:45:40.891076+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891076+0000","last_undegraded":"2023-06-10T18:45:40.891076+0000","last_fullsized":"2023-06-10T18:45:40.891076+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914200+0000","last_change":"2023-06-10T18:45:40.908799+0000","last_active":"2023-06-10T18:45:40.914200+0000","last_peered":"2023-06-10T18:45:40.914200+0000","last_clean":"2023-06-10T18:45:40.914200+0000","last_became_active":"2023-06-10T18:45:40.908494+0000","last_became_peered":"2023-06-10T18:45:40.908494+0000","last_unstale":"2023-06-10T18:45:40.914200+0000","last_undegraded":"2023-06-10T18:45:40.914200+0000","last_fullsized":"2023-06-10T18:45:40.914200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894922+0000","last_change":"2023-06-10T18:45:40.894922+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894922+0000","last_undegraded":"2023-06-10T18:45:40.894922+0000","last_fullsized":"2023-06-10T18:45:40.894922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909967+0000","last_change":"2023-06-10T18:45:40.904441+0000","last_active":"2023-06-10T18:45:40.909967+0000","last_peered":"2023-06-10T18:45:40.909967+0000","last_clean":"2023-06-10T18:45:40.909967+0000","last_became_active":"2023-06-10T18:45:40.904169+0000","last_became_peered":"2023-06-10T18:45:40.904169+0000","last_unstale":"2023-06-10T18:45:40.909967+0000","last_undegraded":"2023-06-10T18:45:40.909967+0000","last_fullsized":"2023-06-10T18:45:40.909967+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:33.234927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895645+0000","last_change":"2023-06-10T18:45:40.895645+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895645+0000","last_undegraded":"2023-06-10T18:45:40.895645+0000","last_fullsized":"2023-06-10T18:45:40.895645+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906515+0000","last_change":"2023-06-10T18:45:40.900293+0000","last_active":"2023-06-10T18:45:40.906515+0000","last_peered":"2023-06-10T18:45:40.906515+0000","last_clean":"2023-06-10T18:45:40.906515+0000","last_became_active":"2023-06-10T18:45:40.900103+0000","last_became_peered":"2023-06-10T18:45:40.900103+0000","last_unstale":"2023-06-10T18:45:40.906515+0000","last_undegraded":"2023-06-10T18:45:40.906515+0000","last_fullsized":"2023-06-10T18:45:40.906515+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897395+0000","last_change":"2023-06-10T18:45:40.897395+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897395+0000","last_undegraded":"2023-06-10T18:45:40.897395+0000","last_fullsized":"2023-06-10T18:45:40.897395+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905107+0000","last_change":"2023-06-10T18:45:40.903527+0000","last_active":"2023-06-10T18:45:40.905107+0000","last_peered":"2023-06-10T18:45:40.905107+0000","last_clean":"2023-06-10T18:45:40.905107+0000","last_became_active":"2023-06-10T18:45:40.903286+0000","last_became_peered":"2023-06-10T18:45:40.903286+0000","last_unstale":"2023-06-10T18:45:40.905107+0000","last_undegraded":"2023-06-10T18:45:40.905107+0000","last_fullsized":"2023-06-10T18:45:40.905107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:47.654330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895755+0000","last_change":"2023-06-10T18:45:40.895755+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895755+0000","last_undegraded":"2023-06-10T18:45:40.895755+0000","last_fullsized":"2023-06-10T18:45:40.895755+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912706+0000","last_change":"2023-06-10T18:45:40.909125+0000","last_active":"2023-06-10T18:45:40.912706+0000","last_peered":"2023-06-10T18:45:40.912706+0000","last_clean":"2023-06-10T18:45:40.912706+0000","last_became_active":"2023-06-10T18:45:40.908861+0000","last_became_peered":"2023-06-10T18:45:40.908861+0000","last_unstale":"2023-06-10T18:45:40.912706+0000","last_undegraded":"2023-06-10T18:45:40.912706+0000","last_fullsized":"2023-06-10T18:45:40.912706+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:35.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896005+0000","last_change":"2023-06-10T18:45:40.896005+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896005+0000","last_undegraded":"2023-06-10T18:45:40.896005+0000","last_fullsized":"2023-06-10T18:45:40.896005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.915588+0000","last_change":"2023-06-10T18:45:40.914575+0000","last_active":"2023-06-10T18:45:40.915588+0000","last_peered":"2023-06-10T18:45:40.915588+0000","last_clean":"2023-06-10T18:45:40.915588+0000","last_became_active":"2023-06-10T18:45:40.914270+0000","last_became_peered":"2023-06-10T18:45:40.914270+0000","last_unstale":"2023-06-10T18:45:40.915588+0000","last_undegraded":"2023-06-10T18:45:40.915588+0000","last_fullsized":"2023-06-10T18:45:40.915588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:12:15.904314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892405+0000","last_change":"2023-06-10T18:45:40.892405+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892405+0000","last_undegraded":"2023-06-10T18:45:40.892405+0000","last_fullsized":"2023-06-10T18:45:40.892405+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.904714+0000","last_change":"2023-06-10T18:45:40.902582+0000","last_active":"2023-06-10T18:45:40.904714+0000","last_peered":"2023-06-10T18:45:40.904714+0000","last_clean":"2023-06-10T18:45:40.904714+0000","last_became_active":"2023-06-10T18:45:40.902303+0000","last_became_peered":"2023-06-10T18:45:40.902303+0000","last_unstale":"2023-06-10T18:45:40.904714+0000","last_undegraded":"2023-06-10T18:45:40.904714+0000","last_fullsized":"2023-06-10T18:45:40.904714+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:54:45.433721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891086+0000","last_change":"2023-06-10T18:45:40.891086+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891086+0000","last_undegraded":"2023-06-10T18:45:40.891086+0000","last_fullsized":"2023-06-10T18:45:40.891086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906993+0000","last_change":"2023-06-10T18:45:40.905531+0000","last_active":"2023-06-10T18:45:40.906993+0000","last_peered":"2023-06-10T18:45:40.906993+0000","last_clean":"2023-06-10T18:45:40.906993+0000","last_became_active":"2023-06-10T18:45:40.905204+0000","last_became_peered":"2023-06-10T18:45:40.905204+0000","last_unstale":"2023-06-10T18:45:40.906993+0000","last_undegraded":"2023-06-10T18:45:40.906993+0000","last_fullsized":"2023-06-10T18:45:40.906993+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:20.447212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891468+0000","last_change":"2023-06-10T18:45:40.891468+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891468+0000","last_undegraded":"2023-06-10T18:45:40.891468+0000","last_fullsized":"2023-06-10T18:45:40.891468+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912627+0000","last_change":"2023-06-10T18:45:40.909552+0000","last_active":"2023-06-10T18:45:40.912627+0000","last_peered":"2023-06-10T18:45:40.912627+0000","last_clean":"2023-06-10T18:45:40.912627+0000","last_became_active":"2023-06-10T18:45:40.909075+0000","last_became_peered":"2023-06-10T18:45:40.909075+0000","last_unstale":"2023-06-10T18:45:40.912627+0000","last_undegraded":"2023-06-10T18:45:40.912627+0000","last_fullsized":"2023-06-10T18:45:40.912627+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:00.118012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891697+0000","last_change":"2023-06-10T18:45:40.891697+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891697+0000","last_undegraded":"2023-06-10T18:45:40.891697+0000","last_fullsized":"2023-06-10T18:45:40.891697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.902608+0000","last_change":"2023-06-10T18:45:40.899865+0000","last_active":"2023-06-10T18:45:40.902608+0000","last_peered":"2023-06-10T18:45:40.902608+0000","last_clean":"2023-06-10T18:45:40.902608+0000","last_became_active":"2023-06-10T18:45:40.899613+0000","last_became_peered":"2023-06-10T18:45:40.899613+0000","last_unstale":"2023-06-10T18:45:40.902608+0000","last_undegraded":"2023-06-10T18:45:40.902608+0000","last_fullsized":"2023-06-10T18:45:40.902608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:11.696529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897259+0000","last_change":"2023-06-10T18:45:40.897259+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897259+0000","last_undegraded":"2023-06-10T18:45:40.897259+0000","last_fullsized":"2023-06-10T18:45:40.897259+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908200+0000","last_change":"2023-06-10T18:45:40.906074+0000","last_active":"2023-06-10T18:45:40.908200+0000","last_peered":"2023-06-10T18:45:40.908200+0000","last_clean":"2023-06-10T18:45:40.908200+0000","last_became_active":"2023-06-10T18:45:40.905809+0000","last_became_peered":"2023-06-10T18:45:40.905809+0000","last_unstale":"2023-06-10T18:45:40.908200+0000","last_undegraded":"2023-06-10T18:45:40.908200+0000","last_fullsized":"2023-06-10T18:45:40.908200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:52:40.294776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895634+0000","last_change":"2023-06-10T18:45:40.895634+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895634+0000","last_undegraded":"2023-06-10T18:45:40.895634+0000","last_fullsized":"2023-06-10T18:45:40.895634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908195+0000","last_change":"2023-06-10T18:45:40.903305+0000","last_active":"2023-06-10T18:45:40.908195+0000","last_peered":"2023-06-10T18:45:40.908195+0000","last_clean":"2023-06-10T18:45:40.908195+0000","last_became_active":"2023-06-10T18:45:40.903030+0000","last_became_peered":"2023-06-10T18:45:40.903030+0000","last_unstale":"2023-06-10T18:45:40.908195+0000","last_undegraded":"2023-06-10T18:45:40.908195+0000","last_fullsized":"2023-06-10T18:45:40.908195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891503+0000","last_change":"2023-06-10T18:45:40.891503+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891503+0000","last_undegraded":"2023-06-10T18:45:40.891503+0000","last_fullsized":"2023-06-10T18:45:40.891503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908226+0000","last_change":"2023-06-10T18:45:40.906223+0000","last_active":"2023-06-10T18:45:40.908226+0000","last_peered":"2023-06-10T18:45:40.908226+0000","last_clean":"2023-06-10T18:45:40.908226+0000","last_became_active":"2023-06-10T18:45:40.905829+0000","last_became_peered":"2023-06-10T18:45:40.905829+0000","last_unstale":"2023-06-10T18:45:40.908226+0000","last_undegraded":"2023-06-10T18:45:40.908226+0000","last_fullsized":"2023-06-10T18:45:40.908226+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890820+0000","last_change":"2023-06-10T18:45:40.890820+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890820+0000","last_undegraded":"2023-06-10T18:45:40.890820+0000","last_fullsized":"2023-06-10T18:45:40.890820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906524+0000","last_change":"2023-06-10T18:45:40.904337+0000","last_active":"2023-06-10T18:45:40.906524+0000","last_peered":"2023-06-10T18:45:40.906524+0000","last_clean":"2023-06-10T18:45:40.906524+0000","last_became_active":"2023-06-10T18:45:40.904087+0000","last_became_peered":"2023-06-10T18:45:40.904087+0000","last_unstale":"2023-06-10T18:45:40.906524+0000","last_undegraded":"2023-06-10T18:45:40.906524+0000","last_fullsized":"2023-06-10T18:45:40.906524+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:51.173858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888868+0000","last_change":"2023-06-10T18:45:40.888868+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888868+0000","last_undegraded":"2023-06-10T18:45:40.888868+0000","last_fullsized":"2023-06-10T18:45:40.888868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.895387+0000","last_change":"2023-06-10T18:45:40.892521+0000","last_active":"2023-06-10T18:45:40.895387+0000","last_peered":"2023-06-10T18:45:40.895387+0000","last_clean":"2023-06-10T18:45:40.895387+0000","last_became_active":"2023-06-10T18:45:40.892250+0000","last_became_peered":"2023-06-10T18:45:40.892250+0000","last_unstale":"2023-06-10T18:45:40.895387+0000","last_undegraded":"2023-06-10T18:45:40.895387+0000","last_fullsized":"2023-06-10T18:45:40.895387+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:50.471888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897955+0000","last_change":"2023-06-10T18:45:40.897955+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897955+0000","last_undegraded":"2023-06-10T18:45:40.897955+0000","last_fullsized":"2023-06-10T18:45:40.897955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910052+0000","last_change":"2023-06-10T18:45:40.909132+0000","last_active":"2023-06-10T18:45:40.910052+0000","last_peered":"2023-06-10T18:45:40.910052+0000","last_clean":"2023-06-10T18:45:40.910052+0000","last_became_active":"2023-06-10T18:45:40.908770+0000","last_became_peered":"2023-06-10T18:45:40.908770+0000","last_unstale":"2023-06-10T18:45:40.910052+0000","last_undegraded":"2023-06-10T18:45:40.910052+0000","last_fullsized":"2023-06-10T18:45:40.910052+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:58:32.518563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895519+0000","last_change":"2023-06-10T18:45:40.895519+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895519+0000","last_undegraded":"2023-06-10T18:45:40.895519+0000","last_fullsized":"2023-06-10T18:45:40.895519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912749+0000","last_change":"2023-06-10T18:45:40.909792+0000","last_active":"2023-06-10T18:45:40.912749+0000","last_peered":"2023-06-10T18:45:40.912749+0000","last_clean":"2023-06-10T18:45:40.912749+0000","last_became_active":"2023-06-10T18:45:40.909529+0000","last_became_peered":"2023-06-10T18:45:40.909529+0000","last_unstale":"2023-06-10T18:45:40.912749+0000","last_undegraded":"2023-06-10T18:45:40.912749+0000","last_fullsized":"2023-06-10T18:45:40.912749+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:35.699893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895527+0000","last_change":"2023-06-10T18:45:40.895527+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895527+0000","last_undegraded":"2023-06-10T18:45:40.895527+0000","last_fullsized":"2023-06-10T18:45:40.895527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907177+0000","last_change":"2023-06-10T18:45:40.904862+0000","last_active":"2023-06-10T18:45:40.907177+0000","last_peered":"2023-06-10T18:45:40.907177+0000","last_clean":"2023-06-10T18:45:40.907177+0000","last_became_active":"2023-06-10T18:45:40.904590+0000","last_became_peered":"2023-06-10T18:45:40.904590+0000","last_unstale":"2023-06-10T18:45:40.907177+0000","last_undegraded":"2023-06-10T18:45:40.907177+0000","last_fullsized":"2023-06-10T18:45:40.907177+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:36:47.213889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891726+0000","last_change":"2023-06-10T18:45:40.891726+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891726+0000","last_undegraded":"2023-06-10T18:45:40.891726+0000","last_fullsized":"2023-06-10T18:45:40.891726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906313+0000","last_change":"2023-06-10T18:45:40.900179+0000","last_active":"2023-06-10T18:45:40.906313+0000","last_peered":"2023-06-10T18:45:40.906313+0000","last_clean":"2023-06-10T18:45:40.906313+0000","last_became_active":"2023-06-10T18:45:40.899762+0000","last_became_peered":"2023-06-10T18:45:40.899762+0000","last_unstale":"2023-06-10T18:45:40.906313+0000","last_undegraded":"2023-06-10T18:45:40.906313+0000","last_fullsized":"2023-06-10T18:45:40.906313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:25:43.791507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894081+0000","last_change":"2023-06-10T18:45:40.894081+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894081+0000","last_undegraded":"2023-06-10T18:45:40.894081+0000","last_fullsized":"2023-06-10T18:45:40.894081+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910760+0000","last_change":"2023-06-10T18:45:40.904961+0000","last_active":"2023-06-10T18:45:40.910760+0000","last_peered":"2023-06-10T18:45:40.910760+0000","last_clean":"2023-06-10T18:45:40.910760+0000","last_became_active":"2023-06-10T18:45:40.904660+0000","last_became_peered":"2023-06-10T18:45:40.904660+0000","last_unstale":"2023-06-10T18:45:40.910760+0000","last_undegraded":"2023-06-10T18:45:40.910760+0000","last_fullsized":"2023-06-10T18:45:40.910760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:40.145269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888355+0000","last_change":"2023-06-10T18:45:40.888355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888355+0000","last_undegraded":"2023-06-10T18:45:40.888355+0000","last_fullsized":"2023-06-10T18:45:40.888355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907024+0000","last_change":"2023-06-10T18:45:40.905644+0000","last_active":"2023-06-10T18:45:40.907024+0000","last_peered":"2023-06-10T18:45:40.907024+0000","last_clean":"2023-06-10T18:45:40.907024+0000","last_became_active":"2023-06-10T18:45:40.905375+0000","last_became_peered":"2023-06-10T18:45:40.905375+0000","last_unstale":"2023-06-10T18:45:40.907024+0000","last_undegraded":"2023-06-10T18:45:40.907024+0000","last_fullsized":"2023-06-10T18:45:40.907024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:28.816147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895956+0000","last_change":"2023-06-10T18:45:40.895956+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895956+0000","last_undegraded":"2023-06-10T18:45:40.895956+0000","last_fullsized":"2023-06-10T18:45:40.895956+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909939+0000","last_change":"2023-06-10T18:45:40.903928+0000","last_active":"2023-06-10T18:45:40.909939+0000","last_peered":"2023-06-10T18:45:40.909939+0000","last_clean":"2023-06-10T18:45:40.909939+0000","last_became_active":"2023-06-10T18:45:40.903597+0000","last_became_peered":"2023-06-10T18:45:40.903597+0000","last_unstale":"2023-06-10T18:45:40.909939+0000","last_undegraded":"2023-06-10T18:45:40.909939+0000","last_fullsized":"2023-06-10T18:45:40.909939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:04.405310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891920+0000","last_change":"2023-06-10T18:45:40.891920+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891920+0000","last_undegraded":"2023-06-10T18:45:40.891920+0000","last_fullsized":"2023-06-10T18:45:40.891920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.913712+0000","last_change":"2023-06-10T18:45:40.907369+0000","last_active":"2023-06-10T18:45:40.913712+0000","last_peered":"2023-06-10T18:45:40.913712+0000","last_clean":"2023-06-10T18:45:40.913712+0000","last_became_active":"2023-06-10T18:45:40.907099+0000","last_became_peered":"2023-06-10T18:45:40.907099+0000","last_unstale":"2023-06-10T18:45:40.913712+0000","last_undegraded":"2023-06-10T18:45:40.913712+0000","last_fullsized":"2023-06-10T18:45:40.913712+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:57.202185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896128+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:45:40.896128+0000","last_peered":"2023-06-10T18:45:40.896128+0000","last_clean":"2023-06-10T18:45:40.896128+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:45:40.896128+0000","last_undegraded":"2023-06-10T18:45:40.896128+0000","last_fullsized":"2023-06-10T18:45:40.896128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:35:09.859258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897049+0000","last_change":"2023-06-10T18:45:40.897049+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897049+0000","last_undegraded":"2023-06-10T18:45:40.897049+0000","last_fullsized":"2023-06-10T18:45:40.897049+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906525+0000","last_change":"2023-06-10T18:45:40.900220+0000","last_active":"2023-06-10T18:45:40.906525+0000","last_peered":"2023-06-10T18:45:40.906525+0000","last_clean":"2023-06-10T18:45:40.906525+0000","last_became_active":"2023-06-10T18:45:40.899948+0000","last_became_peered":"2023-06-10T18:45:40.899948+0000","last_unstale":"2023-06-10T18:45:40.906525+0000","last_undegraded":"2023-06-10T18:45:40.906525+0000","last_fullsized":"2023-06-10T18:45:40.906525+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:24.478651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.891471+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:45:40.891471+0000","last_peered":"2023-06-10T18:45:40.891471+0000","last_clean":"2023-06-10T18:45:40.891471+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:45:40.891471+0000","last_undegraded":"2023-06-10T18:45:40.891471+0000","last_fullsized":"2023-06-10T18:45:40.891471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:39.411023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897646+0000","last_change":"2023-06-10T18:45:40.897646+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897646+0000","last_undegraded":"2023-06-10T18:45:40.897646+0000","last_fullsized":"2023-06-10T18:45:40.897646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912608+0000","last_change":"2023-06-10T18:45:40.909238+0000","last_active":"2023-06-10T18:45:40.912608+0000","last_peered":"2023-06-10T18:45:40.912608+0000","last_clean":"2023-06-10T18:45:40.912608+0000","last_became_active":"2023-06-10T18:45:40.908968+0000","last_became_peered":"2023-06-10T18:45:40.908968+0000","last_unstale":"2023-06-10T18:45:40.912608+0000","last_undegraded":"2023-06-10T18:45:40.912608+0000","last_fullsized":"2023-06-10T18:45:40.912608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:32.937415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.888090+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:45:40.888090+0000","last_peered":"2023-06-10T18:45:40.888090+0000","last_clean":"2023-06-10T18:45:40.888090+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:45:40.888090+0000","last_undegraded":"2023-06-10T18:45:40.888090+0000","last_fullsized":"2023-06-10T18:45:40.888090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:52:00.598598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898298+0000","last_change":"2023-06-10T18:45:40.898298+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898298+0000","last_undegraded":"2023-06-10T18:45:40.898298+0000","last_fullsized":"2023-06-10T18:45:40.898298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910745+0000","last_change":"2023-06-10T18:45:40.904255+0000","last_active":"2023-06-10T18:45:40.910745+0000","last_peered":"2023-06-10T18:45:40.910745+0000","last_clean":"2023-06-10T18:45:40.910745+0000","last_became_active":"2023-06-10T18:45:40.903971+0000","last_became_peered":"2023-06-10T18:45:40.903971+0000","last_unstale":"2023-06-10T18:45:40.910745+0000","last_undegraded":"2023-06-10T18:45:40.910745+0000","last_fullsized":"2023-06-10T18:45:40.910745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:05.711201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.892795+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:45:40.892795+0000","last_peered":"2023-06-10T18:45:40.892795+0000","last_clean":"2023-06-10T18:45:40.892795+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:45:40.892795+0000","last_undegraded":"2023-06-10T18:45:40.892795+0000","last_fullsized":"2023-06-10T18:45:40.892795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:19.558248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":207,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894146+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:45:40.894146+0000","last_peered":"2023-06-10T18:45:40.894146+0000","last_clean":"2023-06-10T18:45:40.894146+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:45:40.894146+0000","last_undegraded":"2023-06-10T18:45:40.894146+0000","last_fullsized":"2023-06-10T18:45:40.894146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:15:52.844548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890750+0000","last_change":"2023-06-10T18:45:40.890750+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890750+0000","last_undegraded":"2023-06-10T18:45:40.890750+0000","last_fullsized":"2023-06-10T18:45:40.890750+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914162+0000","last_change":"2023-06-10T18:45:40.908782+0000","last_active":"2023-06-10T18:45:40.914162+0000","last_peered":"2023-06-10T18:45:40.914162+0000","last_clean":"2023-06-10T18:45:40.914162+0000","last_became_active":"2023-06-10T18:45:40.908452+0000","last_became_peered":"2023-06-10T18:45:40.908452+0000","last_unstale":"2023-06-10T18:45:40.914162+0000","last_undegraded":"2023-06-10T18:45:40.914162+0000","last_fullsized":"2023-06-10T18:45:40.914162+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:16.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894289+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:45:40.894289+0000","last_peered":"2023-06-10T18:45:40.894289+0000","last_clean":"2023-06-10T18:45:40.894289+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:45:40.894289+0000","last_undegraded":"2023-06-10T18:45:40.894289+0000","last_fullsized":"2023-06-10T18:45:40.894289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:21.044293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889714+0000","last_change":"2023-06-10T18:45:40.889714+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889714+0000","last_undegraded":"2023-06-10T18:45:40.889714+0000","last_fullsized":"2023-06-10T18:45:40.889714+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914181+0000","last_change":"2023-06-10T18:45:40.908831+0000","last_active":"2023-06-10T18:45:40.914181+0000","last_peered":"2023-06-10T18:45:40.914181+0000","last_clean":"2023-06-10T18:45:40.914181+0000","last_became_active":"2023-06-10T18:45:40.908610+0000","last_became_peered":"2023-06-10T18:45:40.908610+0000","last_unstale":"2023-06-10T18:45:40.914181+0000","last_undegraded":"2023-06-10T18:45:40.914181+0000","last_fullsized":"2023-06-10T18:45:40.914181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.898804+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:45:40.898804+0000","last_peered":"2023-06-10T18:45:40.898804+0000","last_clean":"2023-06-10T18:45:40.898804+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:45:40.898804+0000","last_undegraded":"2023-06-10T18:45:40.898804+0000","last_fullsized":"2023-06-10T18:45:40.898804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:29.019593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890847+0000","last_change":"2023-06-10T18:45:40.890847+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890847+0000","last_undegraded":"2023-06-10T18:45:40.890847+0000","last_fullsized":"2023-06-10T18:45:40.890847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905087+0000","last_change":"2023-06-10T18:45:40.903579+0000","last_active":"2023-06-10T18:45:40.905087+0000","last_peered":"2023-06-10T18:45:40.905087+0000","last_clean":"2023-06-10T18:45:40.905087+0000","last_became_active":"2023-06-10T18:45:40.903381+0000","last_became_peered":"2023-06-10T18:45:40.903381+0000","last_unstale":"2023-06-10T18:45:40.905087+0000","last_undegraded":"2023-06-10T18:45:40.905087+0000","last_fullsized":"2023-06-10T18:45:40.905087+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:24.524984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896517+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:45:40.896517+0000","last_peered":"2023-06-10T18:45:40.896517+0000","last_clean":"2023-06-10T18:45:40.896517+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:45:40.896517+0000","last_undegraded":"2023-06-10T18:45:40.896517+0000","last_fullsized":"2023-06-10T18:45:40.896517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:47.124020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890809+0000","last_change":"2023-06-10T18:45:40.890809+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890809+0000","last_undegraded":"2023-06-10T18:45:40.890809+0000","last_fullsized":"2023-06-10T18:45:40.890809+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912550+0000","last_change":"2023-06-10T18:45:40.908946+0000","last_active":"2023-06-10T18:45:40.912550+0000","last_peered":"2023-06-10T18:45:40.912550+0000","last_clean":"2023-06-10T18:45:40.912550+0000","last_became_active":"2023-06-10T18:45:40.908651+0000","last_became_peered":"2023-06-10T18:45:40.908651+0000","last_unstale":"2023-06-10T18:45:40.912550+0000","last_undegraded":"2023-06-10T18:45:40.912550+0000","last_fullsized":"2023-06-10T18:45:40.912550+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:23.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.897714+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:45:40.897714+0000","last_peered":"2023-06-10T18:45:40.897714+0000","last_clean":"2023-06-10T18:45:40.897714+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:45:40.897714+0000","last_undegraded":"2023-06-10T18:45:40.897714+0000","last_fullsized":"2023-06-10T18:45:40.897714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:14:14.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":1056768,"data_stored":794723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.625}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.6850000000000005,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.54400000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.315},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.28599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.497}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":1040384,"data_stored":786764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.245}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.501}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":462848,"data_stored":204355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.442}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.802,"5min":3.802,"15min":3.802},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.42899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.379}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.372},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.501}]},{"osd":4,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:45:46.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:45:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.162+0000 7fb693ca0700 1 Processor -- start 2023-06-10T18:45:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.162+0000 7fb693ca0700 1 -- start start 2023-06-10T18:45:46.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb693ca0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb68c071130 0x7fb68c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb693ca0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 0x7fb68c0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb693ca0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0ccea0 con 0x7fb68c15c040 2023-06-10T18:45:46.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb693ca0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c0ae700 con 0x7fb68c071130 2023-06-10T18:45:46.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb693ca0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0ce2c0 con 0x7fb68c0ae280 2023-06-10T18:45:46.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb69123b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb68c071130 0x7fb68c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb69223d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 0x7fb68c0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb691a3c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb68c15c040 0x7fb68c070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:32852/0 (socket says 172.21.15.104:32852) 2023-06-10T18:45:46.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.163+0000 7fb691a3c700 1 -- 172.21.15.104:0/1838647158 learned_addr learned my addr 172.21.15.104:0/1838647158 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:46.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69123b700 1 -- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 msgr2=0x7fb68c0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:46.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69123b700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 0x7fb68c0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69123b700 1 -- 172.21.15.104:0/1838647158 >> v1:172.21.15.114:6789/0 conn(0x7fb68c15c040 legacy=0x7fb68c070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69123b700 1 -- 172.21.15.104:0/1838647158 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0cf750 con 0x7fb68c071130 2023-06-10T18:45:46.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69223d700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 0x7fb68c0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:46.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb69123b700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c071130 0x7fb68c0ce780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb688009840 tx=0x7fb688009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88470cd2401b8208 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb690a3a700 1 -- 172.21.15.104:0/1838647158 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688011830 con 0x7fb68c071130 2023-06-10T18:45:46.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb690a3a700 1 -- 172.21.15.104:0/1838647158 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb688011990 con 0x7fb68c071130 2023-06-10T18:45:46.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.164+0000 7fb690a3a700 1 -- 172.21.15.104:0/1838647158 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688011d20 con 0x7fb68c071130 2023-06-10T18:45:46.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 -- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c071130 msgr2=0x7fb68c0ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c071130 0x7fb68c0ce780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb688009840 tx=0x7fb688009b50 comp rx=0 tx=0).stop 2023-06-10T18:45:46.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 -- 172.21.15.104:0/1838647158 shutdown_connections 2023-06-10T18:45:46.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c0ae280 0x7fb68c0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 --2- 172.21.15.104:0/1838647158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c071130 0x7fb68c0ce780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 -- 172.21.15.104:0/1838647158 >> 172.21.15.104:0/1838647158 conn(0x7fb68c0b1610 msgr2=0x7fb68c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 -- 172.21.15.104:0/1838647158 shutdown_connections 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.165+0000 7fb693ca0700 1 -- 172.21.15.104:0/1838647158 wait complete. 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 Processor -- start 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 -- start start 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c071130 0x7fb68c1ef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c15c040 0x7fb68c0af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0c6b10 con 0x7fb68c0ae280 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c1ef810 con 0x7fb68c15c040 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.166+0000 7fb693ca0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0c5f00 con 0x7fb68c071130 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57418/0 (socket says 172.21.15.104:57418) 2023-06-10T18:45:46.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 -- 172.21.15.104:0/2670054494 learned_addr learned my addr 172.21.15.104:0/2670054494 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:46.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69223d700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c15c040 0x7fb68c0af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb691a3c700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c071130 0x7fb68c1ef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 -- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c071130 msgr2=0x7fb68c1ef2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c071130 0x7fb68c1ef2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 -- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c15c040 msgr2=0x7fb68c0af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c15c040 0x7fb68c0af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 -- 172.21.15.104:0/2670054494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb684009a70 con 0x7fb68c0ae280 2023-06-10T18:45:46.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb69123b700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb6880090b0 tx=0x7fb688004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880045d0 con 0x7fb68c0ae280 2023-06-10T18:45:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6880094a0 con 0x7fb68c0ae280 2023-06-10T18:45:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68801e070 con 0x7fb68c0ae280 2023-06-10T18:45:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880038e0 con 0x7fb68c0ae280 2023-06-10T18:45:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.167+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c0c64c0 con 0x7fb68c0ae280 2023-06-10T18:45:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.168+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c074050 con 0x7fb68c0ae280 2023-06-10T18:45:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.169+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb688003a40 con 0x7fb68c0ae280 2023-06-10T18:45:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.169+0000 7fb6827fc700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 0x7fb67006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.169+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb68808a620 con 0x7fb68c0ae280 2023-06-10T18:45:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.169+0000 7fb691a3c700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 0x7fb67006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.170+0000 7fb691a3c700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 0x7fb67006d190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb67c000f60 tx=0x7fb67c007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.173+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb6880555a0 con 0x7fb68c0ae280 2023-06-10T18:45:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.355+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb68c1606b0 con 0x7fb67006ace0 2023-06-10T18:45:46.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.364+0000 7fb6827fc700 1 -- 172.21.15.104:0/2670054494 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+306040 (secure 0 0 0) 0x7fb67c004b70 con 0x7fb67006ace0 2023-06-10T18:45:46.371 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:46.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 msgr2=0x7fb67006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 0x7fb67006d190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb67c000f60 tx=0x7fb67c007f10 comp rx=0 tx=0).stop 2023-06-10T18:45:46.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 msgr2=0x7fb68c0c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb6880090b0 tx=0x7fb688004750 comp rx=0 tx=0).stop 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 shutdown_connections 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb67006ace0 0x7fb67006d190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb68c15c040 0x7fb68c0af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb68c071130 0x7fb68c1ef2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 --2- 172.21.15.104:0/2670054494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb68c0ae280 0x7fb68c0c7940 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 >> 172.21.15.104:0/2670054494 conn(0x7fb68c0b1610 msgr2=0x7fb68c15b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 shutdown_connections 2023-06-10T18:45:46.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.367+0000 7fb693ca0700 1 -- 172.21.15.104:0/2670054494 wait complete. 2023-06-10T18:45:46.379 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:45:46.385 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-06-10T18:45:45.373915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220340,"kb_used_data":4724,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754380,"statfs":{"total":773094113280,"available":772868485120,"internally_reserved":0,"allocated":4837376,"data_stored":2790680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491539"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891546+0000","last_change":"2023-06-10T18:45:40.891546+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891546+0000","last_undegraded":"2023-06-10T18:45:40.891546+0000","last_fullsized":"2023-06-10T18:45:40.891546+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890299+0000","last_change":"2023-06-10T18:45:40.890299+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890299+0000","last_undegraded":"2023-06-10T18:45:40.890299+0000","last_fullsized":"2023-06-10T18:45:40.890299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893028+0000","last_change":"2023-06-10T18:45:40.893028+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893028+0000","last_undegraded":"2023-06-10T18:45:40.893028+0000","last_fullsized":"2023-06-10T18:45:40.893028+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898813+0000","last_change":"2023-06-10T18:45:40.898813+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898813+0000","last_undegraded":"2023-06-10T18:45:40.898813+0000","last_fullsized":"2023-06-10T18:45:40.898813+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890175+0000","last_change":"2023-06-10T18:45:40.890175+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890175+0000","last_undegraded":"2023-06-10T18:45:40.890175+0000","last_fullsized":"2023-06-10T18:45:40.890175+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892168+0000","last_change":"2023-06-10T18:45:40.892168+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892168+0000","last_undegraded":"2023-06-10T18:45:40.892168+0000","last_fullsized":"2023-06-10T18:45:40.892168+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891355+0000","last_change":"2023-06-10T18:45:40.891355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891355+0000","last_undegraded":"2023-06-10T18:45:40.891355+0000","last_fullsized":"2023-06-10T18:45:40.891355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898139+0000","last_change":"2023-06-10T18:45:40.898139+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898139+0000","last_undegraded":"2023-06-10T18:45:40.898139+0000","last_fullsized":"2023-06-10T18:45:40.898139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899051+0000","last_change":"2023-06-10T18:45:40.899051+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899051+0000","last_undegraded":"2023-06-10T18:45:40.899051+0000","last_fullsized":"2023-06-10T18:45:40.899051+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899223+0000","last_change":"2023-06-10T18:45:40.899223+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899223+0000","last_undegraded":"2023-06-10T18:45:40.899223+0000","last_fullsized":"2023-06-10T18:45:40.899223+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898294+0000","last_change":"2023-06-10T18:45:40.898294+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898294+0000","last_undegraded":"2023-06-10T18:45:40.898294+0000","last_fullsized":"2023-06-10T18:45:40.898294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890617+0000","last_change":"2023-06-10T18:45:40.890617+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890617+0000","last_undegraded":"2023-06-10T18:45:40.890617+0000","last_fullsized":"2023-06-10T18:45:40.890617+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889372+0000","last_change":"2023-06-10T18:45:40.889372+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889372+0000","last_undegraded":"2023-06-10T18:45:40.889372+0000","last_fullsized":"2023-06-10T18:45:40.889372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898070+0000","last_change":"2023-06-10T18:45:40.898070+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898070+0000","last_undegraded":"2023-06-10T18:45:40.898070+0000","last_fullsized":"2023-06-10T18:45:40.898070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891994+0000","last_change":"2023-06-10T18:45:40.891994+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891994+0000","last_undegraded":"2023-06-10T18:45:40.891994+0000","last_fullsized":"2023-06-10T18:45:40.891994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897556+0000","last_change":"2023-06-10T18:45:40.897556+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897556+0000","last_undegraded":"2023-06-10T18:45:40.897556+0000","last_fullsized":"2023-06-10T18:45:40.897556+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893903+0000","last_change":"2023-06-10T18:45:40.893903+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893903+0000","last_undegraded":"2023-06-10T18:45:40.893903+0000","last_fullsized":"2023-06-10T18:45:40.893903+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894160+0000","last_change":"2023-06-10T18:45:40.894160+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894160+0000","last_undegraded":"2023-06-10T18:45:40.894160+0000","last_fullsized":"2023-06-10T18:45:40.894160+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895909+0000","last_change":"2023-06-10T18:45:40.895909+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895909+0000","last_undegraded":"2023-06-10T18:45:40.895909+0000","last_fullsized":"2023-06-10T18:45:40.895909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888440+0000","last_change":"2023-06-10T18:45:40.888440+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888440+0000","last_undegraded":"2023-06-10T18:45:40.888440+0000","last_fullsized":"2023-06-10T18:45:40.888440+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891930+0000","last_change":"2023-06-10T18:45:40.891930+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891930+0000","last_undegraded":"2023-06-10T18:45:40.891930+0000","last_fullsized":"2023-06-10T18:45:40.891930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897471+0000","last_change":"2023-06-10T18:45:40.897471+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897471+0000","last_undegraded":"2023-06-10T18:45:40.897471+0000","last_fullsized":"2023-06-10T18:45:40.897471+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896446+0000","last_change":"2023-06-10T18:45:40.896446+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896446+0000","last_undegraded":"2023-06-10T18:45:40.896446+0000","last_fullsized":"2023-06-10T18:45:40.896446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894403+0000","last_change":"2023-06-10T18:45:40.894403+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894403+0000","last_undegraded":"2023-06-10T18:45:40.894403+0000","last_fullsized":"2023-06-10T18:45:40.894403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896595+0000","last_change":"2023-06-10T18:45:40.896595+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896595+0000","last_undegraded":"2023-06-10T18:45:40.896595+0000","last_fullsized":"2023-06-10T18:45:40.896595+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889402+0000","last_change":"2023-06-10T18:45:40.889402+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889402+0000","last_undegraded":"2023-06-10T18:45:40.889402+0000","last_fullsized":"2023-06-10T18:45:40.889402+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892579+0000","last_change":"2023-06-10T18:45:40.892579+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892579+0000","last_undegraded":"2023-06-10T18:45:40.892579+0000","last_fullsized":"2023-06-10T18:45:40.892579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891526+0000","last_change":"2023-06-10T18:45:40.891526+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891526+0000","last_undegraded":"2023-06-10T18:45:40.891526+0000","last_fullsized":"2023-06-10T18:45:40.891526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895022+0000","last_change":"2023-06-10T18:45:40.895022+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895022+0000","last_undegraded":"2023-06-10T18:45:40.895022+0000","last_fullsized":"2023-06-10T18:45:40.895022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896078+0000","last_change":"2023-06-10T18:45:40.896078+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896078+0000","last_undegraded":"2023-06-10T18:45:40.896078+0000","last_fullsized":"2023-06-10T18:45:40.896078+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891534+0000","last_change":"2023-06-10T18:45:40.891534+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891534+0000","last_undegraded":"2023-06-10T18:45:40.891534+0000","last_fullsized":"2023-06-10T18:45:40.891534+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888941+0000","last_change":"2023-06-10T18:45:40.888941+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888941+0000","last_undegraded":"2023-06-10T18:45:40.888941+0000","last_fullsized":"2023-06-10T18:45:40.888941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894627+0000","last_change":"2023-06-10T18:45:40.894627+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894627+0000","last_undegraded":"2023-06-10T18:45:40.894627+0000","last_fullsized":"2023-06-10T18:45:40.894627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907286+0000","last_change":"2023-06-10T18:45:40.904978+0000","last_active":"2023-06-10T18:45:40.907286+0000","last_peered":"2023-06-10T18:45:40.907286+0000","last_clean":"2023-06-10T18:45:40.907286+0000","last_became_active":"2023-06-10T18:45:40.904678+0000","last_became_peered":"2023-06-10T18:45:40.904678+0000","last_unstale":"2023-06-10T18:45:40.907286+0000","last_undegraded":"2023-06-10T18:45:40.907286+0000","last_fullsized":"2023-06-10T18:45:40.907286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:20.302658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897476+0000","last_change":"2023-06-10T18:45:40.897476+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897476+0000","last_undegraded":"2023-06-10T18:45:40.897476+0000","last_fullsized":"2023-06-10T18:45:40.897476+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906422+0000","last_change":"2023-06-10T18:45:40.904727+0000","last_active":"2023-06-10T18:45:40.906422+0000","last_peered":"2023-06-10T18:45:40.906422+0000","last_clean":"2023-06-10T18:45:40.906422+0000","last_became_active":"2023-06-10T18:45:40.904429+0000","last_became_peered":"2023-06-10T18:45:40.904429+0000","last_unstale":"2023-06-10T18:45:40.906422+0000","last_undegraded":"2023-06-10T18:45:40.906422+0000","last_fullsized":"2023-06-10T18:45:40.906422+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:41.516852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891076+0000","last_change":"2023-06-10T18:45:40.891076+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891076+0000","last_undegraded":"2023-06-10T18:45:40.891076+0000","last_fullsized":"2023-06-10T18:45:40.891076+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914200+0000","last_change":"2023-06-10T18:45:40.908799+0000","last_active":"2023-06-10T18:45:40.914200+0000","last_peered":"2023-06-10T18:45:40.914200+0000","last_clean":"2023-06-10T18:45:40.914200+0000","last_became_active":"2023-06-10T18:45:40.908494+0000","last_became_peered":"2023-06-10T18:45:40.908494+0000","last_unstale":"2023-06-10T18:45:40.914200+0000","last_undegraded":"2023-06-10T18:45:40.914200+0000","last_fullsized":"2023-06-10T18:45:40.914200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894922+0000","last_change":"2023-06-10T18:45:40.894922+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894922+0000","last_undegraded":"2023-06-10T18:45:40.894922+0000","last_fullsized":"2023-06-10T18:45:40.894922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909967+0000","last_change":"2023-06-10T18:45:40.904441+0000","last_active":"2023-06-10T18:45:40.909967+0000","last_peered":"2023-06-10T18:45:40.909967+0000","last_clean":"2023-06-10T18:45:40.909967+0000","last_became_active":"2023-06-10T18:45:40.904169+0000","last_became_peered":"2023-06-10T18:45:40.904169+0000","last_unstale":"2023-06-10T18:45:40.909967+0000","last_undegraded":"2023-06-10T18:45:40.909967+0000","last_fullsized":"2023-06-10T18:45:40.909967+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:33.234927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895645+0000","last_change":"2023-06-10T18:45:40.895645+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895645+0000","last_undegraded":"2023-06-10T18:45:40.895645+0000","last_fullsized":"2023-06-10T18:45:40.895645+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906515+0000","last_change":"2023-06-10T18:45:40.900293+0000","last_active":"2023-06-10T18:45:40.906515+0000","last_peered":"2023-06-10T18:45:40.906515+0000","last_clean":"2023-06-10T18:45:40.906515+0000","last_became_active":"2023-06-10T18:45:40.900103+0000","last_became_peered":"2023-06-10T18:45:40.900103+0000","last_unstale":"2023-06-10T18:45:40.906515+0000","last_undegraded":"2023-06-10T18:45:40.906515+0000","last_fullsized":"2023-06-10T18:45:40.906515+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897395+0000","last_change":"2023-06-10T18:45:40.897395+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897395+0000","last_undegraded":"2023-06-10T18:45:40.897395+0000","last_fullsized":"2023-06-10T18:45:40.897395+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905107+0000","last_change":"2023-06-10T18:45:40.903527+0000","last_active":"2023-06-10T18:45:40.905107+0000","last_peered":"2023-06-10T18:45:40.905107+0000","last_clean":"2023-06-10T18:45:40.905107+0000","last_became_active":"2023-06-10T18:45:40.903286+0000","last_became_peered":"2023-06-10T18:45:40.903286+0000","last_unstale":"2023-06-10T18:45:40.905107+0000","last_undegraded":"2023-06-10T18:45:40.905107+0000","last_fullsized":"2023-06-10T18:45:40.905107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:47.654330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895755+0000","last_change":"2023-06-10T18:45:40.895755+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895755+0000","last_undegraded":"2023-06-10T18:45:40.895755+0000","last_fullsized":"2023-06-10T18:45:40.895755+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912706+0000","last_change":"2023-06-10T18:45:40.909125+0000","last_active":"2023-06-10T18:45:40.912706+0000","last_peered":"2023-06-10T18:45:40.912706+0000","last_clean":"2023-06-10T18:45:40.912706+0000","last_became_active":"2023-06-10T18:45:40.908861+0000","last_became_peered":"2023-06-10T18:45:40.908861+0000","last_unstale":"2023-06-10T18:45:40.912706+0000","last_undegraded":"2023-06-10T18:45:40.912706+0000","last_fullsized":"2023-06-10T18:45:40.912706+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:35.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896005+0000","last_change":"2023-06-10T18:45:40.896005+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896005+0000","last_undegraded":"2023-06-10T18:45:40.896005+0000","last_fullsized":"2023-06-10T18:45:40.896005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.915588+0000","last_change":"2023-06-10T18:45:40.914575+0000","last_active":"2023-06-10T18:45:40.915588+0000","last_peered":"2023-06-10T18:45:40.915588+0000","last_clean":"2023-06-10T18:45:40.915588+0000","last_became_active":"2023-06-10T18:45:40.914270+0000","last_became_peered":"2023-06-10T18:45:40.914270+0000","last_unstale":"2023-06-10T18:45:40.915588+0000","last_undegraded":"2023-06-10T18:45:40.915588+0000","last_fullsized":"2023-06-10T18:45:40.915588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:12:15.904314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892405+0000","last_change":"2023-06-10T18:45:40.892405+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892405+0000","last_undegraded":"2023-06-10T18:45:40.892405+0000","last_fullsized":"2023-06-10T18:45:40.892405+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.904714+0000","last_change":"2023-06-10T18:45:40.902582+0000","last_active":"2023-06-10T18:45:40.904714+0000","last_peered":"2023-06-10T18:45:40.904714+0000","last_clean":"2023-06-10T18:45:40.904714+0000","last_became_active":"2023-06-10T18:45:40.902303+0000","last_became_peered":"2023-06-10T18:45:40.902303+0000","last_unstale":"2023-06-10T18:45:40.904714+0000","last_undegraded":"2023-06-10T18:45:40.904714+0000","last_fullsized":"2023-06-10T18:45:40.904714+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:54:45.433721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891086+0000","last_change":"2023-06-10T18:45:40.891086+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891086+0000","last_undegraded":"2023-06-10T18:45:40.891086+0000","last_fullsized":"2023-06-10T18:45:40.891086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906993+0000","last_change":"2023-06-10T18:45:40.905531+0000","last_active":"2023-06-10T18:45:40.906993+0000","last_peered":"2023-06-10T18:45:40.906993+0000","last_clean":"2023-06-10T18:45:40.906993+0000","last_became_active":"2023-06-10T18:45:40.905204+0000","last_became_peered":"2023-06-10T18:45:40.905204+0000","last_unstale":"2023-06-10T18:45:40.906993+0000","last_undegraded":"2023-06-10T18:45:40.906993+0000","last_fullsized":"2023-06-10T18:45:40.906993+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:20.447212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891468+0000","last_change":"2023-06-10T18:45:40.891468+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891468+0000","last_undegraded":"2023-06-10T18:45:40.891468+0000","last_fullsized":"2023-06-10T18:45:40.891468+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912627+0000","last_change":"2023-06-10T18:45:40.909552+0000","last_active":"2023-06-10T18:45:40.912627+0000","last_peered":"2023-06-10T18:45:40.912627+0000","last_clean":"2023-06-10T18:45:40.912627+0000","last_became_active":"2023-06-10T18:45:40.909075+0000","last_became_peered":"2023-06-10T18:45:40.909075+0000","last_unstale":"2023-06-10T18:45:40.912627+0000","last_undegraded":"2023-06-10T18:45:40.912627+0000","last_fullsized":"2023-06-10T18:45:40.912627+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:00.118012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891697+0000","last_change":"2023-06-10T18:45:40.891697+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891697+0000","last_undegraded":"2023-06-10T18:45:40.891697+0000","last_fullsized":"2023-06-10T18:45:40.891697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.902608+0000","last_change":"2023-06-10T18:45:40.899865+0000","last_active":"2023-06-10T18:45:40.902608+0000","last_peered":"2023-06-10T18:45:40.902608+0000","last_clean":"2023-06-10T18:45:40.902608+0000","last_became_active":"2023-06-10T18:45:40.899613+0000","last_became_peered":"2023-06-10T18:45:40.899613+0000","last_unstale":"2023-06-10T18:45:40.902608+0000","last_undegraded":"2023-06-10T18:45:40.902608+0000","last_fullsized":"2023-06-10T18:45:40.902608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:11.696529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897259+0000","last_change":"2023-06-10T18:45:40.897259+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897259+0000","last_undegraded":"2023-06-10T18:45:40.897259+0000","last_fullsized":"2023-06-10T18:45:40.897259+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908200+0000","last_change":"2023-06-10T18:45:40.906074+0000","last_active":"2023-06-10T18:45:40.908200+0000","last_peered":"2023-06-10T18:45:40.908200+0000","last_clean":"2023-06-10T18:45:40.908200+0000","last_became_active":"2023-06-10T18:45:40.905809+0000","last_became_peered":"2023-06-10T18:45:40.905809+0000","last_unstale":"2023-06-10T18:45:40.908200+0000","last_undegraded":"2023-06-10T18:45:40.908200+0000","last_fullsized":"2023-06-10T18:45:40.908200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:52:40.294776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895634+0000","last_change":"2023-06-10T18:45:40.895634+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895634+0000","last_undegraded":"2023-06-10T18:45:40.895634+0000","last_fullsized":"2023-06-10T18:45:40.895634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908195+0000","last_change":"2023-06-10T18:45:40.903305+0000","last_active":"2023-06-10T18:45:40.908195+0000","last_peered":"2023-06-10T18:45:40.908195+0000","last_clean":"2023-06-10T18:45:40.908195+0000","last_became_active":"2023-06-10T18:45:40.903030+0000","last_became_peered":"2023-06-10T18:45:40.903030+0000","last_unstale":"2023-06-10T18:45:40.908195+0000","last_undegraded":"2023-06-10T18:45:40.908195+0000","last_fullsized":"2023-06-10T18:45:40.908195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891503+0000","last_change":"2023-06-10T18:45:40.891503+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891503+0000","last_undegraded":"2023-06-10T18:45:40.891503+0000","last_fullsized":"2023-06-10T18:45:40.891503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908226+0000","last_change":"2023-06-10T18:45:40.906223+0000","last_active":"2023-06-10T18:45:40.908226+0000","last_peered":"2023-06-10T18:45:40.908226+0000","last_clean":"2023-06-10T18:45:40.908226+0000","last_became_active":"2023-06-10T18:45:40.905829+0000","last_became_peered":"2023-06-10T18:45:40.905829+0000","last_unstale":"2023-06-10T18:45:40.908226+0000","last_undegraded":"2023-06-10T18:45:40.908226+0000","last_fullsized":"2023-06-10T18:45:40.908226+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890820+0000","last_change":"2023-06-10T18:45:40.890820+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890820+0000","last_undegraded":"2023-06-10T18:45:40.890820+0000","last_fullsized":"2023-06-10T18:45:40.890820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906524+0000","last_change":"2023-06-10T18:45:40.904337+0000","last_active":"2023-06-10T18:45:40.906524+0000","last_peered":"2023-06-10T18:45:40.906524+0000","last_clean":"2023-06-10T18:45:40.906524+0000","last_became_active":"2023-06-10T18:45:40.904087+0000","last_became_peered":"2023-06-10T18:45:40.904087+0000","last_unstale":"2023-06-10T18:45:40.906524+0000","last_undegraded":"2023-06-10T18:45:40.906524+0000","last_fullsized":"2023-06-10T18:45:40.906524+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:51.173858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888868+0000","last_change":"2023-06-10T18:45:40.888868+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888868+0000","last_undegraded":"2023-06-10T18:45:40.888868+0000","last_fullsized":"2023-06-10T18:45:40.888868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.895387+0000","last_change":"2023-06-10T18:45:40.892521+0000","last_active":"2023-06-10T18:45:40.895387+0000","last_peered":"2023-06-10T18:45:40.895387+0000","last_clean":"2023-06-10T18:45:40.895387+0000","last_became_active":"2023-06-10T18:45:40.892250+0000","last_became_peered":"2023-06-10T18:45:40.892250+0000","last_unstale":"2023-06-10T18:45:40.895387+0000","last_undegraded":"2023-06-10T18:45:40.895387+0000","last_fullsized":"2023-06-10T18:45:40.895387+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:50.471888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897955+0000","last_change":"2023-06-10T18:45:40.897955+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897955+0000","last_undegraded":"2023-06-10T18:45:40.897955+0000","last_fullsized":"2023-06-10T18:45:40.897955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910052+0000","last_change":"2023-06-10T18:45:40.909132+0000","last_active":"2023-06-10T18:45:40.910052+0000","last_peered":"2023-06-10T18:45:40.910052+0000","last_clean":"2023-06-10T18:45:40.910052+0000","last_became_active":"2023-06-10T18:45:40.908770+0000","last_became_peered":"2023-06-10T18:45:40.908770+0000","last_unstale":"2023-06-10T18:45:40.910052+0000","last_undegraded":"2023-06-10T18:45:40.910052+0000","last_fullsized":"2023-06-10T18:45:40.910052+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:58:32.518563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895519+0000","last_change":"2023-06-10T18:45:40.895519+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895519+0000","last_undegraded":"2023-06-10T18:45:40.895519+0000","last_fullsized":"2023-06-10T18:45:40.895519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912749+0000","last_change":"2023-06-10T18:45:40.909792+0000","last_active":"2023-06-10T18:45:40.912749+0000","last_peered":"2023-06-10T18:45:40.912749+0000","last_clean":"2023-06-10T18:45:40.912749+0000","last_became_active":"2023-06-10T18:45:40.909529+0000","last_became_peered":"2023-06-10T18:45:40.909529+0000","last_unstale":"2023-06-10T18:45:40.912749+0000","last_undegraded":"2023-06-10T18:45:40.912749+0000","last_fullsized":"2023-06-10T18:45:40.912749+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:35.699893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895527+0000","last_change":"2023-06-10T18:45:40.895527+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895527+0000","last_undegraded":"2023-06-10T18:45:40.895527+0000","last_fullsized":"2023-06-10T18:45:40.895527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907177+0000","last_change":"2023-06-10T18:45:40.904862+0000","last_active":"2023-06-10T18:45:40.907177+0000","last_peered":"2023-06-10T18:45:40.907177+0000","last_clean":"2023-06-10T18:45:40.907177+0000","last_became_active":"2023-06-10T18:45:40.904590+0000","last_became_peered":"2023-06-10T18:45:40.904590+0000","last_unstale":"2023-06-10T18:45:40.907177+0000","last_undegraded":"2023-06-10T18:45:40.907177+0000","last_fullsized":"2023-06-10T18:45:40.907177+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:36:47.213889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891726+0000","last_change":"2023-06-10T18:45:40.891726+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891726+0000","last_undegraded":"2023-06-10T18:45:40.891726+0000","last_fullsized":"2023-06-10T18:45:40.891726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906313+0000","last_change":"2023-06-10T18:45:40.900179+0000","last_active":"2023-06-10T18:45:40.906313+0000","last_peered":"2023-06-10T18:45:40.906313+0000","last_clean":"2023-06-10T18:45:40.906313+0000","last_became_active":"2023-06-10T18:45:40.899762+0000","last_became_peered":"2023-06-10T18:45:40.899762+0000","last_unstale":"2023-06-10T18:45:40.906313+0000","last_undegraded":"2023-06-10T18:45:40.906313+0000","last_fullsized":"2023-06-10T18:45:40.906313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:25:43.791507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894081+0000","last_change":"2023-06-10T18:45:40.894081+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894081+0000","last_undegraded":"2023-06-10T18:45:40.894081+0000","last_fullsized":"2023-06-10T18:45:40.894081+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910760+0000","last_change":"2023-06-10T18:45:40.904961+0000","last_active":"2023-06-10T18:45:40.910760+0000","last_peered":"2023-06-10T18:45:40.910760+0000","last_clean":"2023-06-10T18:45:40.910760+0000","last_became_active":"2023-06-10T18:45:40.904660+0000","last_became_peered":"2023-06-10T18:45:40.904660+0000","last_unstale":"2023-06-10T18:45:40.910760+0000","last_undegraded":"2023-06-10T18:45:40.910760+0000","last_fullsized":"2023-06-10T18:45:40.910760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:40.145269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888355+0000","last_change":"2023-06-10T18:45:40.888355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888355+0000","last_undegraded":"2023-06-10T18:45:40.888355+0000","last_fullsized":"2023-06-10T18:45:40.888355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907024+0000","last_change":"2023-06-10T18:45:40.905644+0000","last_active":"2023-06-10T18:45:40.907024+0000","last_peered":"2023-06-10T18:45:40.907024+0000","last_clean":"2023-06-10T18:45:40.907024+0000","last_became_active":"2023-06-10T18:45:40.905375+0000","last_became_peered":"2023-06-10T18:45:40.905375+0000","last_unstale":"2023-06-10T18:45:40.907024+0000","last_undegraded":"2023-06-10T18:45:40.907024+0000","last_fullsized":"2023-06-10T18:45:40.907024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:28.816147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895956+0000","last_change":"2023-06-10T18:45:40.895956+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895956+0000","last_undegraded":"2023-06-10T18:45:40.895956+0000","last_fullsized":"2023-06-10T18:45:40.895956+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909939+0000","last_change":"2023-06-10T18:45:40.903928+0000","last_active":"2023-06-10T18:45:40.909939+0000","last_peered":"2023-06-10T18:45:40.909939+0000","last_clean":"2023-06-10T18:45:40.909939+0000","last_became_active":"2023-06-10T18:45:40.903597+0000","last_became_peered":"2023-06-10T18:45:40.903597+0000","last_unstale":"2023-06-10T18:45:40.909939+0000","last_undegraded":"2023-06-10T18:45:40.909939+0000","last_fullsized":"2023-06-10T18:45:40.909939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:04.405310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891920+0000","last_change":"2023-06-10T18:45:40.891920+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891920+0000","last_undegraded":"2023-06-10T18:45:40.891920+0000","last_fullsized":"2023-06-10T18:45:40.891920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.913712+0000","last_change":"2023-06-10T18:45:40.907369+0000","last_active":"2023-06-10T18:45:40.913712+0000","last_peered":"2023-06-10T18:45:40.913712+0000","last_clean":"2023-06-10T18:45:40.913712+0000","last_became_active":"2023-06-10T18:45:40.907099+0000","last_became_peered":"2023-06-10T18:45:40.907099+0000","last_unstale":"2023-06-10T18:45:40.913712+0000","last_undegraded":"2023-06-10T18:45:40.913712+0000","last_fullsized":"2023-06-10T18:45:40.913712+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:57.202185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896128+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:45:40.896128+0000","last_peered":"2023-06-10T18:45:40.896128+0000","last_clean":"2023-06-10T18:45:40.896128+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:45:40.896128+0000","last_undegraded":"2023-06-10T18:45:40.896128+0000","last_fullsized":"2023-06-10T18:45:40.896128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:35:09.859258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897049+0000","last_change":"2023-06-10T18:45:40.897049+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897049+0000","last_undegraded":"2023-06-10T18:45:40.897049+0000","last_fullsized":"2023-06-10T18:45:40.897049+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906525+0000","last_change":"2023-06-10T18:45:40.900220+0000","last_active":"2023-06-10T18:45:40.906525+0000","last_peered":"2023-06-10T18:45:40.906525+0000","last_clean":"2023-06-10T18:45:40.906525+0000","last_became_active":"2023-06-10T18:45:40.899948+0000","last_became_peered":"2023-06-10T18:45:40.899948+0000","last_unstale":"2023-06-10T18:45:40.906525+0000","last_undegraded":"2023-06-10T18:45:40.906525+0000","last_fullsized":"2023-06-10T18:45:40.906525+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:24.478651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.891471+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:45:40.891471+0000","last_peered":"2023-06-10T18:45:40.891471+0000","last_clean":"2023-06-10T18:45:40.891471+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:45:40.891471+0000","last_undegraded":"2023-06-10T18:45:40.891471+0000","last_fullsized":"2023-06-10T18:45:40.891471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:39.411023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897646+0000","last_change":"2023-06-10T18:45:40.897646+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897646+0000","last_undegraded":"2023-06-10T18:45:40.897646+0000","last_fullsized":"2023-06-10T18:45:40.897646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912608+0000","last_change":"2023-06-10T18:45:40.909238+0000","last_active":"2023-06-10T18:45:40.912608+0000","last_peered":"2023-06-10T18:45:40.912608+0000","last_clean":"2023-06-10T18:45:40.912608+0000","last_became_active":"2023-06-10T18:45:40.908968+0000","last_became_peered":"2023-06-10T18:45:40.908968+0000","last_unstale":"2023-06-10T18:45:40.912608+0000","last_undegraded":"2023-06-10T18:45:40.912608+0000","last_fullsized":"2023-06-10T18:45:40.912608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:32.937415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.888090+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:45:40.888090+0000","last_peered":"2023-06-10T18:45:40.888090+0000","last_clean":"2023-06-10T18:45:40.888090+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:45:40.888090+0000","last_undegraded":"2023-06-10T18:45:40.888090+0000","last_fullsized":"2023-06-10T18:45:40.888090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:52:00.598598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898298+0000","last_change":"2023-06-10T18:45:40.898298+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898298+0000","last_undegraded":"2023-06-10T18:45:40.898298+0000","last_fullsized":"2023-06-10T18:45:40.898298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910745+0000","last_change":"2023-06-10T18:45:40.904255+0000","last_active":"2023-06-10T18:45:40.910745+0000","last_peered":"2023-06-10T18:45:40.910745+0000","last_clean":"2023-06-10T18:45:40.910745+0000","last_became_active":"2023-06-10T18:45:40.903971+0000","last_became_peered":"2023-06-10T18:45:40.903971+0000","last_unstale":"2023-06-10T18:45:40.910745+0000","last_undegraded":"2023-06-10T18:45:40.910745+0000","last_fullsized":"2023-06-10T18:45:40.910745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:05.711201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.892795+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:45:40.892795+0000","last_peered":"2023-06-10T18:45:40.892795+0000","last_clean":"2023-06-10T18:45:40.892795+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:45:40.892795+0000","last_undegraded":"2023-06-10T18:45:40.892795+0000","last_fullsized":"2023-06-10T18:45:40.892795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:19.558248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":207,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894146+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:45:40.894146+0000","last_peered":"2023-06-10T18:45:40.894146+0000","last_clean":"2023-06-10T18:45:40.894146+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:45:40.894146+0000","last_undegraded":"2023-06-10T18:45:40.894146+0000","last_fullsized":"2023-06-10T18:45:40.894146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:15:52.844548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890750+0000","last_change":"2023-06-10T18:45:40.890750+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890750+0000","last_undegraded":"2023-06-10T18:45:40.890750+0000","last_fullsized":"2023-06-10T18:45:40.890750+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914162+0000","last_change":"2023-06-10T18:45:40.908782+0000","last_active":"2023-06-10T18:45:40.914162+0000","last_peered":"2023-06-10T18:45:40.914162+0000","last_clean":"2023-06-10T18:45:40.914162+0000","last_became_active":"2023-06-10T18:45:40.908452+0000","last_became_peered":"2023-06-10T18:45:40.908452+0000","last_unstale":"2023-06-10T18:45:40.914162+0000","last_undegraded":"2023-06-10T18:45:40.914162+0000","last_fullsized":"2023-06-10T18:45:40.914162+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:16.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894289+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:45:40.894289+0000","last_peered":"2023-06-10T18:45:40.894289+0000","last_clean":"2023-06-10T18:45:40.894289+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:45:40.894289+0000","last_undegraded":"2023-06-10T18:45:40.894289+0000","last_fullsized":"2023-06-10T18:45:40.894289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:21.044293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889714+0000","last_change":"2023-06-10T18:45:40.889714+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889714+0000","last_undegraded":"2023-06-10T18:45:40.889714+0000","last_fullsized":"2023-06-10T18:45:40.889714+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914181+0000","last_change":"2023-06-10T18:45:40.908831+0000","last_active":"2023-06-10T18:45:40.914181+0000","last_peered":"2023-06-10T18:45:40.914181+0000","last_clean":"2023-06-10T18:45:40.914181+0000","last_became_active":"2023-06-10T18:45:40.908610+0000","last_became_peered":"2023-06-10T18:45:40.908610+0000","last_unstale":"2023-06-10T18:45:40.914181+0000","last_undegraded":"2023-06-10T18:45:40.914181+0000","last_fullsized":"2023-06-10T18:45:40.914181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.898804+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:45:40.898804+0000","last_peered":"2023-06-10T18:45:40.898804+0000","last_clean":"2023-06-10T18:45:40.898804+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:45:40.898804+0000","last_undegraded":"2023-06-10T18:45:40.898804+0000","last_fullsized":"2023-06-10T18:45:40.898804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:29.019593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890847+0000","last_change":"2023-06-10T18:45:40.890847+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890847+0000","last_undegraded":"2023-06-10T18:45:40.890847+0000","last_fullsized":"2023-06-10T18:45:40.890847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905087+0000","last_change":"2023-06-10T18:45:40.903579+0000","last_active":"2023-06-10T18:45:40.905087+0000","last_peered":"2023-06-10T18:45:40.905087+0000","last_clean":"2023-06-10T18:45:40.905087+0000","last_became_active":"2023-06-10T18:45:40.903381+0000","last_became_peered":"2023-06-10T18:45:40.903381+0000","last_unstale":"2023-06-10T18:45:40.905087+0000","last_undegraded":"2023-06-10T18:45:40.905087+0000","last_fullsized":"2023-06-10T18:45:40.905087+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:24.524984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896517+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:45:40.896517+0000","last_peered":"2023-06-10T18:45:40.896517+0000","last_clean":"2023-06-10T18:45:40.896517+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:45:40.896517+0000","last_undegraded":"2023-06-10T18:45:40.896517+0000","last_fullsized":"2023-06-10T18:45:40.896517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:47.124020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890809+0000","last_change":"2023-06-10T18:45:40.890809+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890809+0000","last_undegraded":"2023-06-10T18:45:40.890809+0000","last_fullsized":"2023-06-10T18:45:40.890809+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912550+0000","last_change":"2023-06-10T18:45:40.908946+0000","last_active":"2023-06-10T18:45:40.912550+0000","last_peered":"2023-06-10T18:45:40.912550+0000","last_clean":"2023-06-10T18:45:40.912550+0000","last_became_active":"2023-06-10T18:45:40.908651+0000","last_became_peered":"2023-06-10T18:45:40.908651+0000","last_unstale":"2023-06-10T18:45:40.912550+0000","last_undegraded":"2023-06-10T18:45:40.912550+0000","last_fullsized":"2023-06-10T18:45:40.912550+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:23.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.897714+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:45:40.897714+0000","last_peered":"2023-06-10T18:45:40.897714+0000","last_clean":"2023-06-10T18:45:40.897714+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:45:40.897714+0000","last_undegraded":"2023-06-10T18:45:40.897714+0000","last_fullsized":"2023-06-10T18:45:40.897714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:14:14.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":1056768,"data_stored":794723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.625}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.6850000000000005,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.54400000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.315},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.28599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.497}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":1040384,"data_stored":786764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.245}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.501}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":462848,"data_stored":204355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.442}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.802,"5min":3.802,"15min":3.802},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.42899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.379}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.372},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.501}]},{"osd":4,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:45:46.394 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2023-06-10T18:45:46.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891546+0000', 'last_change': '2023-06-10T18:45:40.891546+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891546+0000', 'last_undegraded': '2023-06-10T18:45:40.891546+0000', 'last_fullsized': '2023-06-10T18:45:40.891546+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2023-06-10T18:45:46.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890299+0000', 'last_change': '2023-06-10T18:45:40.890299+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890299+0000', 'last_undegraded': '2023-06-10T18:45:40.890299+0000', 'last_fullsized': '2023-06-10T18:45:40.890299+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2023-06-10T18:45:46.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.893028+0000', 'last_change': '2023-06-10T18:45:40.893028+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.893028+0000', 'last_undegraded': '2023-06-10T18:45:40.893028+0000', 'last_fullsized': '2023-06-10T18:45:40.893028+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2023-06-10T18:45:46.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.898813+0000', 'last_change': '2023-06-10T18:45:40.898813+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.898813+0000', 'last_undegraded': '2023-06-10T18:45:40.898813+0000', 'last_fullsized': '2023-06-10T18:45:40.898813+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2023-06-10T18:45:46.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890175+0000', 'last_change': '2023-06-10T18:45:40.890175+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890175+0000', 'last_undegraded': '2023-06-10T18:45:40.890175+0000', 'last_fullsized': '2023-06-10T18:45:40.890175+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2023-06-10T18:45:46.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.892168+0000', 'last_change': '2023-06-10T18:45:40.892168+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.892168+0000', 'last_undegraded': '2023-06-10T18:45:40.892168+0000', 'last_fullsized': '2023-06-10T18:45:40.892168+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2023-06-10T18:45:46.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891355+0000', 'last_change': '2023-06-10T18:45:40.891355+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891355+0000', 'last_undegraded': '2023-06-10T18:45:40.891355+0000', 'last_fullsized': '2023-06-10T18:45:40.891355+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2023-06-10T18:45:46.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.898139+0000', 'last_change': '2023-06-10T18:45:40.898139+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.898139+0000', 'last_undegraded': '2023-06-10T18:45:40.898139+0000', 'last_fullsized': '2023-06-10T18:45:40.898139+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2023-06-10T18:45:46.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.899051+0000', 'last_change': '2023-06-10T18:45:40.899051+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.899051+0000', 'last_undegraded': '2023-06-10T18:45:40.899051+0000', 'last_fullsized': '2023-06-10T18:45:40.899051+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2023-06-10T18:45:46.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.899223+0000', 'last_change': '2023-06-10T18:45:40.899223+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.899223+0000', 'last_undegraded': '2023-06-10T18:45:40.899223+0000', 'last_fullsized': '2023-06-10T18:45:40.899223+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2023-06-10T18:45:46.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.898294+0000', 'last_change': '2023-06-10T18:45:40.898294+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.898294+0000', 'last_undegraded': '2023-06-10T18:45:40.898294+0000', 'last_fullsized': '2023-06-10T18:45:40.898294+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2023-06-10T18:45:46.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890617+0000', 'last_change': '2023-06-10T18:45:40.890617+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890617+0000', 'last_undegraded': '2023-06-10T18:45:40.890617+0000', 'last_fullsized': '2023-06-10T18:45:40.890617+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2023-06-10T18:45:46.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.889372+0000', 'last_change': '2023-06-10T18:45:40.889372+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.889372+0000', 'last_undegraded': '2023-06-10T18:45:40.889372+0000', 'last_fullsized': '2023-06-10T18:45:40.889372+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2023-06-10T18:45:46.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.898070+0000', 'last_change': '2023-06-10T18:45:40.898070+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.898070+0000', 'last_undegraded': '2023-06-10T18:45:40.898070+0000', 'last_fullsized': '2023-06-10T18:45:40.898070+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2023-06-10T18:45:46.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891994+0000', 'last_change': '2023-06-10T18:45:40.891994+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891994+0000', 'last_undegraded': '2023-06-10T18:45:40.891994+0000', 'last_fullsized': '2023-06-10T18:45:40.891994+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2023-06-10T18:45:46.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897556+0000', 'last_change': '2023-06-10T18:45:40.897556+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897556+0000', 'last_undegraded': '2023-06-10T18:45:40.897556+0000', 'last_fullsized': '2023-06-10T18:45:40.897556+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2023-06-10T18:45:46.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.893903+0000', 'last_change': '2023-06-10T18:45:40.893903+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.893903+0000', 'last_undegraded': '2023-06-10T18:45:40.893903+0000', 'last_fullsized': '2023-06-10T18:45:40.893903+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2023-06-10T18:45:46.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.894160+0000', 'last_change': '2023-06-10T18:45:40.894160+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.894160+0000', 'last_undegraded': '2023-06-10T18:45:40.894160+0000', 'last_fullsized': '2023-06-10T18:45:40.894160+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2023-06-10T18:45:46.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895909+0000', 'last_change': '2023-06-10T18:45:40.895909+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895909+0000', 'last_undegraded': '2023-06-10T18:45:40.895909+0000', 'last_fullsized': '2023-06-10T18:45:40.895909+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2023-06-10T18:45:46.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.888440+0000', 'last_change': '2023-06-10T18:45:40.888440+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.888440+0000', 'last_undegraded': '2023-06-10T18:45:40.888440+0000', 'last_fullsized': '2023-06-10T18:45:40.888440+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2023-06-10T18:45:46.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891930+0000', 'last_change': '2023-06-10T18:45:40.891930+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891930+0000', 'last_undegraded': '2023-06-10T18:45:40.891930+0000', 'last_fullsized': '2023-06-10T18:45:40.891930+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2023-06-10T18:45:46.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897471+0000', 'last_change': '2023-06-10T18:45:40.897471+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897471+0000', 'last_undegraded': '2023-06-10T18:45:40.897471+0000', 'last_fullsized': '2023-06-10T18:45:40.897471+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.412 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2023-06-10T18:45:46.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.896446+0000', 'last_change': '2023-06-10T18:45:40.896446+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.896446+0000', 'last_undegraded': '2023-06-10T18:45:40.896446+0000', 'last_fullsized': '2023-06-10T18:45:40.896446+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.413 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2023-06-10T18:45:46.413 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.894403+0000', 'last_change': '2023-06-10T18:45:40.894403+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.894403+0000', 'last_undegraded': '2023-06-10T18:45:40.894403+0000', 'last_fullsized': '2023-06-10T18:45:40.894403+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.414 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2023-06-10T18:45:46.414 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.896595+0000', 'last_change': '2023-06-10T18:45:40.896595+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.896595+0000', 'last_undegraded': '2023-06-10T18:45:40.896595+0000', 'last_fullsized': '2023-06-10T18:45:40.896595+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.414 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2023-06-10T18:45:46.415 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.889402+0000', 'last_change': '2023-06-10T18:45:40.889402+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.889402+0000', 'last_undegraded': '2023-06-10T18:45:40.889402+0000', 'last_fullsized': '2023-06-10T18:45:40.889402+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.415 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2023-06-10T18:45:46.416 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.892579+0000', 'last_change': '2023-06-10T18:45:40.892579+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.892579+0000', 'last_undegraded': '2023-06-10T18:45:40.892579+0000', 'last_fullsized': '2023-06-10T18:45:40.892579+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.416 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2023-06-10T18:45:46.416 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891526+0000', 'last_change': '2023-06-10T18:45:40.891526+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891526+0000', 'last_undegraded': '2023-06-10T18:45:40.891526+0000', 'last_fullsized': '2023-06-10T18:45:40.891526+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.417 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2023-06-10T18:45:46.417 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895022+0000', 'last_change': '2023-06-10T18:45:40.895022+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895022+0000', 'last_undegraded': '2023-06-10T18:45:40.895022+0000', 'last_fullsized': '2023-06-10T18:45:40.895022+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.417 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2023-06-10T18:45:46.418 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.896078+0000', 'last_change': '2023-06-10T18:45:40.896078+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.896078+0000', 'last_undegraded': '2023-06-10T18:45:40.896078+0000', 'last_fullsized': '2023-06-10T18:45:40.896078+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.418 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2023-06-10T18:45:46.418 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891534+0000', 'last_change': '2023-06-10T18:45:40.891534+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891534+0000', 'last_undegraded': '2023-06-10T18:45:40.891534+0000', 'last_fullsized': '2023-06-10T18:45:40.891534+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.419 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2023-06-10T18:45:46.419 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.888941+0000', 'last_change': '2023-06-10T18:45:40.888941+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.888941+0000', 'last_undegraded': '2023-06-10T18:45:40.888941+0000', 'last_fullsized': '2023-06-10T18:45:40.888941+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.420 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2023-06-10T18:45:46.420 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.894627+0000', 'last_change': '2023-06-10T18:45:40.894627+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.894627+0000', 'last_undegraded': '2023-06-10T18:45:40.894627+0000', 'last_fullsized': '2023-06-10T18:45:40.894627+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-06-10T18:45:46.421 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2023-06-10T18:45:46.421 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897476+0000', 'last_change': '2023-06-10T18:45:40.897476+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897476+0000', 'last_undegraded': '2023-06-10T18:45:40.897476+0000', 'last_fullsized': '2023-06-10T18:45:40.897476+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.421 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2023-06-10T18:45:46.421 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891076+0000', 'last_change': '2023-06-10T18:45:40.891076+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891076+0000', 'last_undegraded': '2023-06-10T18:45:40.891076+0000', 'last_fullsized': '2023-06-10T18:45:40.891076+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.422 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2023-06-10T18:45:46.422 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.894922+0000', 'last_change': '2023-06-10T18:45:40.894922+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.894922+0000', 'last_undegraded': '2023-06-10T18:45:40.894922+0000', 'last_fullsized': '2023-06-10T18:45:40.894922+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-06-10T18:45:46.422 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2023-06-10T18:45:46.423 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895645+0000', 'last_change': '2023-06-10T18:45:40.895645+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895645+0000', 'last_undegraded': '2023-06-10T18:45:40.895645+0000', 'last_fullsized': '2023-06-10T18:45:40.895645+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.423 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2023-06-10T18:45:46.423 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897395+0000', 'last_change': '2023-06-10T18:45:40.897395+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897395+0000', 'last_undegraded': '2023-06-10T18:45:40.897395+0000', 'last_fullsized': '2023-06-10T18:45:40.897395+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2023-06-10T18:45:46.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895755+0000', 'last_change': '2023-06-10T18:45:40.895755+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895755+0000', 'last_undegraded': '2023-06-10T18:45:40.895755+0000', 'last_fullsized': '2023-06-10T18:45:40.895755+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2023-06-10T18:45:46.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.896005+0000', 'last_change': '2023-06-10T18:45:40.896005+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.896005+0000', 'last_undegraded': '2023-06-10T18:45:40.896005+0000', 'last_fullsized': '2023-06-10T18:45:40.896005+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2023-06-10T18:45:46.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.892405+0000', 'last_change': '2023-06-10T18:45:40.892405+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.892405+0000', 'last_undegraded': '2023-06-10T18:45:40.892405+0000', 'last_fullsized': '2023-06-10T18:45:40.892405+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2023-06-10T18:45:46.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891086+0000', 'last_change': '2023-06-10T18:45:40.891086+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891086+0000', 'last_undegraded': '2023-06-10T18:45:40.891086+0000', 'last_fullsized': '2023-06-10T18:45:40.891086+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2023-06-10T18:45:46.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891468+0000', 'last_change': '2023-06-10T18:45:40.891468+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891468+0000', 'last_undegraded': '2023-06-10T18:45:40.891468+0000', 'last_fullsized': '2023-06-10T18:45:40.891468+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2023-06-10T18:45:46.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891697+0000', 'last_change': '2023-06-10T18:45:40.891697+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891697+0000', 'last_undegraded': '2023-06-10T18:45:40.891697+0000', 'last_fullsized': '2023-06-10T18:45:40.891697+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2023-06-10T18:45:46.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897259+0000', 'last_change': '2023-06-10T18:45:40.897259+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897259+0000', 'last_undegraded': '2023-06-10T18:45:40.897259+0000', 'last_fullsized': '2023-06-10T18:45:40.897259+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2023-06-10T18:45:46.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895634+0000', 'last_change': '2023-06-10T18:45:40.895634+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895634+0000', 'last_undegraded': '2023-06-10T18:45:40.895634+0000', 'last_fullsized': '2023-06-10T18:45:40.895634+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2023-06-10T18:45:46.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891503+0000', 'last_change': '2023-06-10T18:45:40.891503+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891503+0000', 'last_undegraded': '2023-06-10T18:45:40.891503+0000', 'last_fullsized': '2023-06-10T18:45:40.891503+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2023-06-10T18:45:46.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890820+0000', 'last_change': '2023-06-10T18:45:40.890820+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890820+0000', 'last_undegraded': '2023-06-10T18:45:40.890820+0000', 'last_fullsized': '2023-06-10T18:45:40.890820+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2023-06-10T18:45:46.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.888868+0000', 'last_change': '2023-06-10T18:45:40.888868+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.888868+0000', 'last_undegraded': '2023-06-10T18:45:40.888868+0000', 'last_fullsized': '2023-06-10T18:45:40.888868+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2023-06-10T18:45:46.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897955+0000', 'last_change': '2023-06-10T18:45:40.897955+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897955+0000', 'last_undegraded': '2023-06-10T18:45:40.897955+0000', 'last_fullsized': '2023-06-10T18:45:40.897955+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.435 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2023-06-10T18:45:46.436 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895519+0000', 'last_change': '2023-06-10T18:45:40.895519+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895519+0000', 'last_undegraded': '2023-06-10T18:45:40.895519+0000', 'last_fullsized': '2023-06-10T18:45:40.895519+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.436 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2023-06-10T18:45:46.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895527+0000', 'last_change': '2023-06-10T18:45:40.895527+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895527+0000', 'last_undegraded': '2023-06-10T18:45:40.895527+0000', 'last_fullsized': '2023-06-10T18:45:40.895527+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.437 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2023-06-10T18:45:46.437 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891726+0000', 'last_change': '2023-06-10T18:45:40.891726+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891726+0000', 'last_undegraded': '2023-06-10T18:45:40.891726+0000', 'last_fullsized': '2023-06-10T18:45:40.891726+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.438 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2023-06-10T18:45:46.438 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.894081+0000', 'last_change': '2023-06-10T18:45:40.894081+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.894081+0000', 'last_undegraded': '2023-06-10T18:45:40.894081+0000', 'last_fullsized': '2023-06-10T18:45:40.894081+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-06-10T18:45:46.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2023-06-10T18:45:46.439 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.888355+0000', 'last_change': '2023-06-10T18:45:40.888355+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.888355+0000', 'last_undegraded': '2023-06-10T18:45:40.888355+0000', 'last_fullsized': '2023-06-10T18:45:40.888355+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-06-10T18:45:46.439 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2023-06-10T18:45:46.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.895956+0000', 'last_change': '2023-06-10T18:45:40.895956+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.895956+0000', 'last_undegraded': '2023-06-10T18:45:40.895956+0000', 'last_fullsized': '2023-06-10T18:45:40.895956+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.440 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2023-06-10T18:45:46.440 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.891920+0000', 'last_change': '2023-06-10T18:45:40.891920+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.891920+0000', 'last_undegraded': '2023-06-10T18:45:40.891920+0000', 'last_fullsized': '2023-06-10T18:45:40.891920+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-06-10T18:45:46.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2023-06-10T18:45:46.441 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897049+0000', 'last_change': '2023-06-10T18:45:40.897049+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897049+0000', 'last_undegraded': '2023-06-10T18:45:40.897049+0000', 'last_fullsized': '2023-06-10T18:45:40.897049+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-06-10T18:45:46.441 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2023-06-10T18:45:46.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.897646+0000', 'last_change': '2023-06-10T18:45:40.897646+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.897646+0000', 'last_undegraded': '2023-06-10T18:45:40.897646+0000', 'last_fullsized': '2023-06-10T18:45:40.897646+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.442 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2023-06-10T18:45:46.442 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.898298+0000', 'last_change': '2023-06-10T18:45:40.898298+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.898298+0000', 'last_undegraded': '2023-06-10T18:45:40.898298+0000', 'last_fullsized': '2023-06-10T18:45:40.898298+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-06-10T18:45:46.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2023-06-10T18:45:46.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890750+0000', 'last_change': '2023-06-10T18:45:40.890750+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890750+0000', 'last_undegraded': '2023-06-10T18:45:40.890750+0000', 'last_fullsized': '2023-06-10T18:45:40.890750+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-06-10T18:45:46.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2023-06-10T18:45:46.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.889714+0000', 'last_change': '2023-06-10T18:45:40.889714+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.889714+0000', 'last_undegraded': '2023-06-10T18:45:40.889714+0000', 'last_fullsized': '2023-06-10T18:45:40.889714+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-06-10T18:45:46.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2023-06-10T18:45:46.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890847+0000', 'last_change': '2023-06-10T18:45:40.890847+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890847+0000', 'last_undegraded': '2023-06-10T18:45:40.890847+0000', 'last_fullsized': '2023-06-10T18:45:40.890847+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2023-06-10T18:45:46.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 40, 'state': 'creating+peering', 'last_fresh': '2023-06-10T18:45:40.890809+0000', 'last_change': '2023-06-10T18:45:40.890809+0000', 'last_active': '2023-06-10T18:45:40.881331+0000', 'last_peered': '2023-06-10T18:45:40.881331+0000', 'last_clean': '2023-06-10T18:45:40.881331+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-06-10T18:45:40.890809+0000', 'last_undegraded': '2023-06-10T18:45:40.890809+0000', 'last_fullsized': '2023-06-10T18:45:40.890809+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 40, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'last_clean_scrub_stamp': '2023-06-10T18:45:40.881331+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-06-10T18:45:46.447 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-06-10T18:45:46.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.563+0000 7ff720d13700 1 Processor -- start 2023-06-10T18:45:46.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.563+0000 7ff720d13700 1 -- start start 2023-06-10T18:45:46.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff720d13700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 0x7ff71c167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff720d13700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0cadd0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff720d13700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0cb0b0 con 0x7ff71c0ce350 2023-06-10T18:45:46.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff720d13700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0cfeb0 con 0x7ff71c08a3d0 2023-06-10T18:45:46.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff71c06b1f0 0x7ff71c0cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55252/0 (socket says 172.21.15.104:55252) 2023-06-10T18:45:46.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71a59c700 1 -- 172.21.15.104:0/696444957 learned_addr learned my addr 172.21.15.104:0/696444957 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:46.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71ad9d700 1 --2- 172.21.15.104:0/696444957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 0x7ff71c167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099227900 0 0) 0x7ff71c0cadd0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7040035b0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71ad9d700 1 -- 172.21.15.104:0/696444957 >> v1:172.21.15.114:6789/0 conn(0x7ff71c0ce350 legacy=0x7ff71c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574192334 0 0) 0x7ff7040035b0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71ad9d700 1 -- 172.21.15.104:0/696444957 >> v1:172.21.15.104:6789/0 conn(0x7ff71c06b1f0 legacy=0x7ff71c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.564+0000 7ff71ad9d700 1 -- 172.21.15.104:0/696444957 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0cdeb0 con 0x7ff71c08a3d0 2023-06-10T18:45:46.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff71ad9d700 1 --2- 172.21.15.104:0/696444957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 0x7ff71c167940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff714009dc0 tx=0x7ff71400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ba5314d66019fc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71400e070 con 0x7ff71c08a3d0 2023-06-10T18:45:46.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff714003a20 con 0x7ff71c08a3d0 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff71959a700 1 -- 172.21.15.104:0/696444957 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c001670 con 0x7ff71c08a3d0 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 -- 172.21.15.104:0/696444957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 msgr2=0x7ff71c167940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 --2- 172.21.15.104:0/696444957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 0x7ff71c167940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff714009dc0 tx=0x7ff71400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 -- 172.21.15.104:0/696444957 shutdown_connections 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 --2- 172.21.15.104:0/696444957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c08a3d0 0x7ff71c167940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 -- 172.21.15.104:0/696444957 >> 172.21.15.104:0/696444957 conn(0x7ff71c089fd0 msgr2=0x7ff71c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:46.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 -- 172.21.15.104:0/696444957 shutdown_connections 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.565+0000 7ff720d13700 1 -- 172.21.15.104:0/696444957 wait complete. 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 Processor -- start 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 -- start start 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff71c08a3d0 0x7ff71c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c0ce350 0x7ff71c0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c1576a0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0c5360 con 0x7ff71c08a3d0 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff720d13700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0c8e80 con 0x7ff71c0ce350 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.567+0000 7ff71a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57430/0 (socket says 172.21.15.104:57430) 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 -- 172.21.15.104:0/1189541787 learned_addr learned my addr 172.21.15.104:0/1189541787 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71ad9d700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c0ce350 0x7ff71c0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff719d9b700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff71c08a3d0 0x7ff71c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 -- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c0ce350 msgr2=0x7ff71c0c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c0ce350 0x7ff71c0c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 -- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff71c08a3d0 msgr2=0x7ff71c0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff71c08a3d0 0x7ff71c0ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 -- 172.21.15.104:0/1189541787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714009a70 con 0x7ff71c06b1f0 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff71a59c700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00c520 tx=0x7ff70c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c004c80 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70c004de0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c007310 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c0c9160 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.568+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c0c9720 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.569+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c074050 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.569+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff70c007470 con 0x7ff71c06b1f0 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.570+0000 7ff70affd700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 0x7ff6fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.570+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff70c089440 con 0x7ff71c06b1f0 2023-06-10T18:45:46.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.570+0000 7ff719d9b700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 0x7ff6fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.570+0000 7ff719d9b700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 0x7ff6fc06d190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0b36b0 tx=0x7ff710009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.574+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff70c054330 con 0x7ff71c06b1f0 2023-06-10T18:45:46.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.837+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff71c0b63e0 con 0x7ff71c06b1f0 2023-06-10T18:45:46.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.837+0000 7ff70affd700 1 -- 172.21.15.104:0/1189541787 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1034 (secure 0 0 0) 0x7ff70c057950 con 0x7ff71c06b1f0 2023-06-10T18:45:46.843 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:46.843 INFO:teuthology.orchestra.run.smithi104.stdout:{"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":123,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":8,"num_up_osds":8,"osd_up_since":1686422635,"num_in_osds":8,"osd_in_since":1686422626,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":41}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":225628160,"bytes_avail":772868485120,"bytes_total":773094113280,"inactive_pgs_ratio":0.60952383279800415},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"b","status":"up:active","gid":5622}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-06-10T18:45:15.365959+0000","services":{}},"progress_events":{}} 2023-06-10T18:45:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 msgr2=0x7ff6fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 0x7ff6fc06d190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0b36b0 tx=0x7ff710009040 comp rx=0 tx=0).stop 2023-06-10T18:45:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 msgr2=0x7ff71c157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00c520 tx=0x7ff70c00c8e0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 shutdown_connections 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff71c0ce350 0x7ff71c0c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c06b1f0 0x7ff71c157160 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6fc06ace0 0x7ff6fc06d190 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 --2- 172.21.15.104:0/1189541787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff71c08a3d0 0x7ff71c0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.839+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 >> 172.21.15.104:0/1189541787 conn(0x7ff71c089fd0 msgr2=0x7ff71c0ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.840+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 shutdown_connections 2023-06-10T18:45:46.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.840+0000 7ff720d13700 1 -- 172.21.15.104:0/1189541787 wait complete. 2023-06-10T18:45:46.857 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-06-10T18:45:46.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:45:46.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.971+0000 7fdc5b76d700 1 Processor -- start 2023-06-10T18:45:46.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.971+0000 7fdc5b76d700 1 -- start start 2023-06-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc5b76d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc5415c7a0 0x7fdc540cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc5b76d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdc540701a0 0x7fdc540705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc5b76d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc54070b20 con 0x7fdc541567f0 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc5b76d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5415cca0 con 0x7fdc540701a0 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc5b76d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5415cf80 con 0x7fdc5415c7a0 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdc540701a0 0x7fdc540705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdc540701a0 0x7fdc540705e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57442/0 (socket says 172.21.15.104:57442) 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59509700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdc541567f0 0x7fdc540cad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55266/0 (socket says 172.21.15.104:55266) 2023-06-10T18:45:46.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 -- 172.21.15.104:0/531801700 learned_addr learned my addr 172.21.15.104:0/531801700 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 -- 172.21.15.104:0/531801700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc5415c7a0 msgr2=0x7fdc540cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc58d08700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc5415c7a0 0x7fdc540cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc5415c7a0 0x7fdc540cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc4bfff700 1 -- 172.21.15.104:0/531801700 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248363169 0 0) 0x7fdc54070b20 con 0x7fdc541567f0 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 -- 172.21.15.104:0/531801700 >> v1:172.21.15.104:6789/0 conn(0x7fdc541567f0 legacy=0x7fdc540cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.972+0000 7fdc59d0a700 1 -- 172.21.15.104:0/531801700 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5415d260 con 0x7fdc540701a0 2023-06-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc59d0a700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc540701a0 0x7fdc540705e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009e50 tx=0x7fdc4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1947958cf0e553 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc4bfff700 1 -- 172.21.15.104:0/531801700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00e070 con 0x7fdc540701a0 2023-06-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc4bfff700 1 -- 172.21.15.104:0/531801700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c00b9c0 con 0x7fdc540701a0 2023-06-10T18:45:46.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc4bfff700 1 -- 172.21.15.104:0/531801700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00bd70 con 0x7fdc540701a0 2023-06-10T18:45:46.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 -- 172.21.15.104:0/531801700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc540701a0 msgr2=0x7fdc540705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc540701a0 0x7fdc540705e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009e50 tx=0x7fdc4c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 -- 172.21.15.104:0/531801700 shutdown_connections 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc540701a0 0x7fdc540705e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/531801700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc5415c7a0 0x7fdc540cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 -- 172.21.15.104:0/531801700 >> 172.21.15.104:0/531801700 conn(0x7fdc5406e370 msgr2=0x7fdc5406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:46.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 -- 172.21.15.104:0/531801700 shutdown_connections 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.973+0000 7fdc5b76d700 1 -- 172.21.15.104:0/531801700 wait complete. 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.974+0000 7fdc5b76d700 1 Processor -- start 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 -- start start 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc540701a0 0x7fdc541f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 0x7fdc540b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540b0e40 con 0x7fdc5415c7a0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540b09b0 con 0x7fdc540701a0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc5b76d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540afda0 con 0x7fdc541567f0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc59d0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc59d0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57448/0 (socket says 172.21.15.104:57448) 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc59d0a700 1 -- 172.21.15.104:0/2928076277 learned_addr learned my addr 172.21.15.104:0/2928076277 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc59509700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc540701a0 0x7fdc541f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.975+0000 7fdc58d08700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 0x7fdc540b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 -- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 msgr2=0x7fdc540b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 0x7fdc540b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 -- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc540701a0 msgr2=0x7fdc541f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc540701a0 0x7fdc541f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 -- 172.21.15.104:0/2928076277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c009b30 con 0x7fdc5415c7a0 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc58d08700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 0x7fdc540b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc59d0a700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00b730 tx=0x7fdc4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c003ea0 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c004960 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc540b0080 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c01b670 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.976+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc540b0640 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.977+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc540b0800 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.978+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdc4c01b7d0 con 0x7fdc5415c7a0 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.978+0000 7fdc49ffb700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 0x7fdc3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:46.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.978+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdc4c08ae80 con 0x7fdc5415c7a0 2023-06-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.978+0000 7fdc59509700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 0x7fdc3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.979+0000 7fdc59509700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 0x7fdc3806d190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc44007fd0 tx=0x7fdc44007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:46.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:46.982+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdc4c055e80 con 0x7fdc5415c7a0 2023-06-10T18:45:47.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.165+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc540cdaa0 con 0x7fdc3806ace0 2023-06-10T18:45:47.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.174+0000 7fdc49ffb700 1 -- 172.21.15.104:0/2928076277 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+306040 (secure 0 0 0) 0x7fdc44001370 con 0x7fdc3806ace0 2023-06-10T18:45:47.183 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.177+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 msgr2=0x7fdc3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.177+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 0x7fdc3806d190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc44007fd0 tx=0x7fdc44007de0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.177+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 msgr2=0x7fdc54153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:47.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.177+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00b730 tx=0x7fdc4c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 shutdown_connections 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc3806ace0 0x7fdc3806d190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc540701a0 0x7fdc541f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc5415c7a0 0x7fdc54153a60 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 --2- 172.21.15.104:0/2928076277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc541567f0 0x7fdc540b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 >> 172.21.15.104:0/2928076277 conn(0x7fdc5406e370 msgr2=0x7fdc5406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:47.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 shutdown_connections 2023-06-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:47.178+0000 7fdc5b76d700 1 -- 172.21.15.104:0/2928076277 wait complete. 2023-06-10T18:45:47.187 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:45:47.195 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-06-10T18:45:45.373915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220340,"kb_used_data":4724,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754380,"statfs":{"total":773094113280,"available":772868485120,"internally_reserved":0,"allocated":4837376,"data_stored":2790680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.491539"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891546+0000","last_change":"2023-06-10T18:45:40.891546+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891546+0000","last_undegraded":"2023-06-10T18:45:40.891546+0000","last_fullsized":"2023-06-10T18:45:40.891546+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890299+0000","last_change":"2023-06-10T18:45:40.890299+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890299+0000","last_undegraded":"2023-06-10T18:45:40.890299+0000","last_fullsized":"2023-06-10T18:45:40.890299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893028+0000","last_change":"2023-06-10T18:45:40.893028+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893028+0000","last_undegraded":"2023-06-10T18:45:40.893028+0000","last_fullsized":"2023-06-10T18:45:40.893028+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898813+0000","last_change":"2023-06-10T18:45:40.898813+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898813+0000","last_undegraded":"2023-06-10T18:45:40.898813+0000","last_fullsized":"2023-06-10T18:45:40.898813+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890175+0000","last_change":"2023-06-10T18:45:40.890175+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890175+0000","last_undegraded":"2023-06-10T18:45:40.890175+0000","last_fullsized":"2023-06-10T18:45:40.890175+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892168+0000","last_change":"2023-06-10T18:45:40.892168+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892168+0000","last_undegraded":"2023-06-10T18:45:40.892168+0000","last_fullsized":"2023-06-10T18:45:40.892168+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891355+0000","last_change":"2023-06-10T18:45:40.891355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891355+0000","last_undegraded":"2023-06-10T18:45:40.891355+0000","last_fullsized":"2023-06-10T18:45:40.891355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898139+0000","last_change":"2023-06-10T18:45:40.898139+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898139+0000","last_undegraded":"2023-06-10T18:45:40.898139+0000","last_fullsized":"2023-06-10T18:45:40.898139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899051+0000","last_change":"2023-06-10T18:45:40.899051+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899051+0000","last_undegraded":"2023-06-10T18:45:40.899051+0000","last_fullsized":"2023-06-10T18:45:40.899051+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.899223+0000","last_change":"2023-06-10T18:45:40.899223+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.899223+0000","last_undegraded":"2023-06-10T18:45:40.899223+0000","last_fullsized":"2023-06-10T18:45:40.899223+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898294+0000","last_change":"2023-06-10T18:45:40.898294+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898294+0000","last_undegraded":"2023-06-10T18:45:40.898294+0000","last_fullsized":"2023-06-10T18:45:40.898294+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890617+0000","last_change":"2023-06-10T18:45:40.890617+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890617+0000","last_undegraded":"2023-06-10T18:45:40.890617+0000","last_fullsized":"2023-06-10T18:45:40.890617+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889372+0000","last_change":"2023-06-10T18:45:40.889372+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889372+0000","last_undegraded":"2023-06-10T18:45:40.889372+0000","last_fullsized":"2023-06-10T18:45:40.889372+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898070+0000","last_change":"2023-06-10T18:45:40.898070+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898070+0000","last_undegraded":"2023-06-10T18:45:40.898070+0000","last_fullsized":"2023-06-10T18:45:40.898070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891994+0000","last_change":"2023-06-10T18:45:40.891994+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891994+0000","last_undegraded":"2023-06-10T18:45:40.891994+0000","last_fullsized":"2023-06-10T18:45:40.891994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897556+0000","last_change":"2023-06-10T18:45:40.897556+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897556+0000","last_undegraded":"2023-06-10T18:45:40.897556+0000","last_fullsized":"2023-06-10T18:45:40.897556+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.893903+0000","last_change":"2023-06-10T18:45:40.893903+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.893903+0000","last_undegraded":"2023-06-10T18:45:40.893903+0000","last_fullsized":"2023-06-10T18:45:40.893903+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894160+0000","last_change":"2023-06-10T18:45:40.894160+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894160+0000","last_undegraded":"2023-06-10T18:45:40.894160+0000","last_fullsized":"2023-06-10T18:45:40.894160+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895909+0000","last_change":"2023-06-10T18:45:40.895909+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895909+0000","last_undegraded":"2023-06-10T18:45:40.895909+0000","last_fullsized":"2023-06-10T18:45:40.895909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888440+0000","last_change":"2023-06-10T18:45:40.888440+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888440+0000","last_undegraded":"2023-06-10T18:45:40.888440+0000","last_fullsized":"2023-06-10T18:45:40.888440+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891930+0000","last_change":"2023-06-10T18:45:40.891930+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891930+0000","last_undegraded":"2023-06-10T18:45:40.891930+0000","last_fullsized":"2023-06-10T18:45:40.891930+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897471+0000","last_change":"2023-06-10T18:45:40.897471+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897471+0000","last_undegraded":"2023-06-10T18:45:40.897471+0000","last_fullsized":"2023-06-10T18:45:40.897471+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896446+0000","last_change":"2023-06-10T18:45:40.896446+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896446+0000","last_undegraded":"2023-06-10T18:45:40.896446+0000","last_fullsized":"2023-06-10T18:45:40.896446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894403+0000","last_change":"2023-06-10T18:45:40.894403+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894403+0000","last_undegraded":"2023-06-10T18:45:40.894403+0000","last_fullsized":"2023-06-10T18:45:40.894403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896595+0000","last_change":"2023-06-10T18:45:40.896595+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896595+0000","last_undegraded":"2023-06-10T18:45:40.896595+0000","last_fullsized":"2023-06-10T18:45:40.896595+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889402+0000","last_change":"2023-06-10T18:45:40.889402+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889402+0000","last_undegraded":"2023-06-10T18:45:40.889402+0000","last_fullsized":"2023-06-10T18:45:40.889402+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892579+0000","last_change":"2023-06-10T18:45:40.892579+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892579+0000","last_undegraded":"2023-06-10T18:45:40.892579+0000","last_fullsized":"2023-06-10T18:45:40.892579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891526+0000","last_change":"2023-06-10T18:45:40.891526+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891526+0000","last_undegraded":"2023-06-10T18:45:40.891526+0000","last_fullsized":"2023-06-10T18:45:40.891526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895022+0000","last_change":"2023-06-10T18:45:40.895022+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895022+0000","last_undegraded":"2023-06-10T18:45:40.895022+0000","last_fullsized":"2023-06-10T18:45:40.895022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896078+0000","last_change":"2023-06-10T18:45:40.896078+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896078+0000","last_undegraded":"2023-06-10T18:45:40.896078+0000","last_fullsized":"2023-06-10T18:45:40.896078+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891534+0000","last_change":"2023-06-10T18:45:40.891534+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891534+0000","last_undegraded":"2023-06-10T18:45:40.891534+0000","last_fullsized":"2023-06-10T18:45:40.891534+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888941+0000","last_change":"2023-06-10T18:45:40.888941+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888941+0000","last_undegraded":"2023-06-10T18:45:40.888941+0000","last_fullsized":"2023-06-10T18:45:40.888941+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894627+0000","last_change":"2023-06-10T18:45:40.894627+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894627+0000","last_undegraded":"2023-06-10T18:45:40.894627+0000","last_fullsized":"2023-06-10T18:45:40.894627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907286+0000","last_change":"2023-06-10T18:45:40.904978+0000","last_active":"2023-06-10T18:45:40.907286+0000","last_peered":"2023-06-10T18:45:40.907286+0000","last_clean":"2023-06-10T18:45:40.907286+0000","last_became_active":"2023-06-10T18:45:40.904678+0000","last_became_peered":"2023-06-10T18:45:40.904678+0000","last_unstale":"2023-06-10T18:45:40.907286+0000","last_undegraded":"2023-06-10T18:45:40.907286+0000","last_fullsized":"2023-06-10T18:45:40.907286+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:20.302658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897476+0000","last_change":"2023-06-10T18:45:40.897476+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897476+0000","last_undegraded":"2023-06-10T18:45:40.897476+0000","last_fullsized":"2023-06-10T18:45:40.897476+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906422+0000","last_change":"2023-06-10T18:45:40.904727+0000","last_active":"2023-06-10T18:45:40.906422+0000","last_peered":"2023-06-10T18:45:40.906422+0000","last_clean":"2023-06-10T18:45:40.906422+0000","last_became_active":"2023-06-10T18:45:40.904429+0000","last_became_peered":"2023-06-10T18:45:40.904429+0000","last_unstale":"2023-06-10T18:45:40.906422+0000","last_undegraded":"2023-06-10T18:45:40.906422+0000","last_fullsized":"2023-06-10T18:45:40.906422+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:41.516852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891076+0000","last_change":"2023-06-10T18:45:40.891076+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891076+0000","last_undegraded":"2023-06-10T18:45:40.891076+0000","last_fullsized":"2023-06-10T18:45:40.891076+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914200+0000","last_change":"2023-06-10T18:45:40.908799+0000","last_active":"2023-06-10T18:45:40.914200+0000","last_peered":"2023-06-10T18:45:40.914200+0000","last_clean":"2023-06-10T18:45:40.914200+0000","last_became_active":"2023-06-10T18:45:40.908494+0000","last_became_peered":"2023-06-10T18:45:40.908494+0000","last_unstale":"2023-06-10T18:45:40.914200+0000","last_undegraded":"2023-06-10T18:45:40.914200+0000","last_fullsized":"2023-06-10T18:45:40.914200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894922+0000","last_change":"2023-06-10T18:45:40.894922+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894922+0000","last_undegraded":"2023-06-10T18:45:40.894922+0000","last_fullsized":"2023-06-10T18:45:40.894922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909967+0000","last_change":"2023-06-10T18:45:40.904441+0000","last_active":"2023-06-10T18:45:40.909967+0000","last_peered":"2023-06-10T18:45:40.909967+0000","last_clean":"2023-06-10T18:45:40.909967+0000","last_became_active":"2023-06-10T18:45:40.904169+0000","last_became_peered":"2023-06-10T18:45:40.904169+0000","last_unstale":"2023-06-10T18:45:40.909967+0000","last_undegraded":"2023-06-10T18:45:40.909967+0000","last_fullsized":"2023-06-10T18:45:40.909967+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:33.234927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895645+0000","last_change":"2023-06-10T18:45:40.895645+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895645+0000","last_undegraded":"2023-06-10T18:45:40.895645+0000","last_fullsized":"2023-06-10T18:45:40.895645+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906515+0000","last_change":"2023-06-10T18:45:40.900293+0000","last_active":"2023-06-10T18:45:40.906515+0000","last_peered":"2023-06-10T18:45:40.906515+0000","last_clean":"2023-06-10T18:45:40.906515+0000","last_became_active":"2023-06-10T18:45:40.900103+0000","last_became_peered":"2023-06-10T18:45:40.900103+0000","last_unstale":"2023-06-10T18:45:40.906515+0000","last_undegraded":"2023-06-10T18:45:40.906515+0000","last_fullsized":"2023-06-10T18:45:40.906515+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:01.087664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897395+0000","last_change":"2023-06-10T18:45:40.897395+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897395+0000","last_undegraded":"2023-06-10T18:45:40.897395+0000","last_fullsized":"2023-06-10T18:45:40.897395+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905107+0000","last_change":"2023-06-10T18:45:40.903527+0000","last_active":"2023-06-10T18:45:40.905107+0000","last_peered":"2023-06-10T18:45:40.905107+0000","last_clean":"2023-06-10T18:45:40.905107+0000","last_became_active":"2023-06-10T18:45:40.903286+0000","last_became_peered":"2023-06-10T18:45:40.903286+0000","last_unstale":"2023-06-10T18:45:40.905107+0000","last_undegraded":"2023-06-10T18:45:40.905107+0000","last_fullsized":"2023-06-10T18:45:40.905107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:47.654330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895755+0000","last_change":"2023-06-10T18:45:40.895755+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895755+0000","last_undegraded":"2023-06-10T18:45:40.895755+0000","last_fullsized":"2023-06-10T18:45:40.895755+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912706+0000","last_change":"2023-06-10T18:45:40.909125+0000","last_active":"2023-06-10T18:45:40.912706+0000","last_peered":"2023-06-10T18:45:40.912706+0000","last_clean":"2023-06-10T18:45:40.912706+0000","last_became_active":"2023-06-10T18:45:40.908861+0000","last_became_peered":"2023-06-10T18:45:40.908861+0000","last_unstale":"2023-06-10T18:45:40.912706+0000","last_undegraded":"2023-06-10T18:45:40.912706+0000","last_fullsized":"2023-06-10T18:45:40.912706+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:35.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.896005+0000","last_change":"2023-06-10T18:45:40.896005+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.896005+0000","last_undegraded":"2023-06-10T18:45:40.896005+0000","last_fullsized":"2023-06-10T18:45:40.896005+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.915588+0000","last_change":"2023-06-10T18:45:40.914575+0000","last_active":"2023-06-10T18:45:40.915588+0000","last_peered":"2023-06-10T18:45:40.915588+0000","last_clean":"2023-06-10T18:45:40.915588+0000","last_became_active":"2023-06-10T18:45:40.914270+0000","last_became_peered":"2023-06-10T18:45:40.914270+0000","last_unstale":"2023-06-10T18:45:40.915588+0000","last_undegraded":"2023-06-10T18:45:40.915588+0000","last_fullsized":"2023-06-10T18:45:40.915588+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:12:15.904314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.892405+0000","last_change":"2023-06-10T18:45:40.892405+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.892405+0000","last_undegraded":"2023-06-10T18:45:40.892405+0000","last_fullsized":"2023-06-10T18:45:40.892405+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.904714+0000","last_change":"2023-06-10T18:45:40.902582+0000","last_active":"2023-06-10T18:45:40.904714+0000","last_peered":"2023-06-10T18:45:40.904714+0000","last_clean":"2023-06-10T18:45:40.904714+0000","last_became_active":"2023-06-10T18:45:40.902303+0000","last_became_peered":"2023-06-10T18:45:40.902303+0000","last_unstale":"2023-06-10T18:45:40.904714+0000","last_undegraded":"2023-06-10T18:45:40.904714+0000","last_fullsized":"2023-06-10T18:45:40.904714+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:54:45.433721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891086+0000","last_change":"2023-06-10T18:45:40.891086+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891086+0000","last_undegraded":"2023-06-10T18:45:40.891086+0000","last_fullsized":"2023-06-10T18:45:40.891086+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906993+0000","last_change":"2023-06-10T18:45:40.905531+0000","last_active":"2023-06-10T18:45:40.906993+0000","last_peered":"2023-06-10T18:45:40.906993+0000","last_clean":"2023-06-10T18:45:40.906993+0000","last_became_active":"2023-06-10T18:45:40.905204+0000","last_became_peered":"2023-06-10T18:45:40.905204+0000","last_unstale":"2023-06-10T18:45:40.906993+0000","last_undegraded":"2023-06-10T18:45:40.906993+0000","last_fullsized":"2023-06-10T18:45:40.906993+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:20.447212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891468+0000","last_change":"2023-06-10T18:45:40.891468+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891468+0000","last_undegraded":"2023-06-10T18:45:40.891468+0000","last_fullsized":"2023-06-10T18:45:40.891468+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912627+0000","last_change":"2023-06-10T18:45:40.909552+0000","last_active":"2023-06-10T18:45:40.912627+0000","last_peered":"2023-06-10T18:45:40.912627+0000","last_clean":"2023-06-10T18:45:40.912627+0000","last_became_active":"2023-06-10T18:45:40.909075+0000","last_became_peered":"2023-06-10T18:45:40.909075+0000","last_unstale":"2023-06-10T18:45:40.912627+0000","last_undegraded":"2023-06-10T18:45:40.912627+0000","last_fullsized":"2023-06-10T18:45:40.912627+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:00.118012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891697+0000","last_change":"2023-06-10T18:45:40.891697+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891697+0000","last_undegraded":"2023-06-10T18:45:40.891697+0000","last_fullsized":"2023-06-10T18:45:40.891697+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.902608+0000","last_change":"2023-06-10T18:45:40.899865+0000","last_active":"2023-06-10T18:45:40.902608+0000","last_peered":"2023-06-10T18:45:40.902608+0000","last_clean":"2023-06-10T18:45:40.902608+0000","last_became_active":"2023-06-10T18:45:40.899613+0000","last_became_peered":"2023-06-10T18:45:40.899613+0000","last_unstale":"2023-06-10T18:45:40.902608+0000","last_undegraded":"2023-06-10T18:45:40.902608+0000","last_fullsized":"2023-06-10T18:45:40.902608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:11.696529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897259+0000","last_change":"2023-06-10T18:45:40.897259+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897259+0000","last_undegraded":"2023-06-10T18:45:40.897259+0000","last_fullsized":"2023-06-10T18:45:40.897259+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908200+0000","last_change":"2023-06-10T18:45:40.906074+0000","last_active":"2023-06-10T18:45:40.908200+0000","last_peered":"2023-06-10T18:45:40.908200+0000","last_clean":"2023-06-10T18:45:40.908200+0000","last_became_active":"2023-06-10T18:45:40.905809+0000","last_became_peered":"2023-06-10T18:45:40.905809+0000","last_unstale":"2023-06-10T18:45:40.908200+0000","last_undegraded":"2023-06-10T18:45:40.908200+0000","last_fullsized":"2023-06-10T18:45:40.908200+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:52:40.294776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895634+0000","last_change":"2023-06-10T18:45:40.895634+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895634+0000","last_undegraded":"2023-06-10T18:45:40.895634+0000","last_fullsized":"2023-06-10T18:45:40.895634+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908195+0000","last_change":"2023-06-10T18:45:40.903305+0000","last_active":"2023-06-10T18:45:40.908195+0000","last_peered":"2023-06-10T18:45:40.908195+0000","last_clean":"2023-06-10T18:45:40.908195+0000","last_became_active":"2023-06-10T18:45:40.903030+0000","last_became_peered":"2023-06-10T18:45:40.903030+0000","last_unstale":"2023-06-10T18:45:40.908195+0000","last_undegraded":"2023-06-10T18:45:40.908195+0000","last_fullsized":"2023-06-10T18:45:40.908195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891503+0000","last_change":"2023-06-10T18:45:40.891503+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891503+0000","last_undegraded":"2023-06-10T18:45:40.891503+0000","last_fullsized":"2023-06-10T18:45:40.891503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.908226+0000","last_change":"2023-06-10T18:45:40.906223+0000","last_active":"2023-06-10T18:45:40.908226+0000","last_peered":"2023-06-10T18:45:40.908226+0000","last_clean":"2023-06-10T18:45:40.908226+0000","last_became_active":"2023-06-10T18:45:40.905829+0000","last_became_peered":"2023-06-10T18:45:40.905829+0000","last_unstale":"2023-06-10T18:45:40.908226+0000","last_undegraded":"2023-06-10T18:45:40.908226+0000","last_fullsized":"2023-06-10T18:45:40.908226+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890820+0000","last_change":"2023-06-10T18:45:40.890820+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890820+0000","last_undegraded":"2023-06-10T18:45:40.890820+0000","last_fullsized":"2023-06-10T18:45:40.890820+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906524+0000","last_change":"2023-06-10T18:45:40.904337+0000","last_active":"2023-06-10T18:45:40.906524+0000","last_peered":"2023-06-10T18:45:40.906524+0000","last_clean":"2023-06-10T18:45:40.906524+0000","last_became_active":"2023-06-10T18:45:40.904087+0000","last_became_peered":"2023-06-10T18:45:40.904087+0000","last_unstale":"2023-06-10T18:45:40.906524+0000","last_undegraded":"2023-06-10T18:45:40.906524+0000","last_fullsized":"2023-06-10T18:45:40.906524+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:51.173858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888868+0000","last_change":"2023-06-10T18:45:40.888868+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888868+0000","last_undegraded":"2023-06-10T18:45:40.888868+0000","last_fullsized":"2023-06-10T18:45:40.888868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.895387+0000","last_change":"2023-06-10T18:45:40.892521+0000","last_active":"2023-06-10T18:45:40.895387+0000","last_peered":"2023-06-10T18:45:40.895387+0000","last_clean":"2023-06-10T18:45:40.895387+0000","last_became_active":"2023-06-10T18:45:40.892250+0000","last_became_peered":"2023-06-10T18:45:40.892250+0000","last_unstale":"2023-06-10T18:45:40.895387+0000","last_undegraded":"2023-06-10T18:45:40.895387+0000","last_fullsized":"2023-06-10T18:45:40.895387+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:50.471888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897955+0000","last_change":"2023-06-10T18:45:40.897955+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897955+0000","last_undegraded":"2023-06-10T18:45:40.897955+0000","last_fullsized":"2023-06-10T18:45:40.897955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910052+0000","last_change":"2023-06-10T18:45:40.909132+0000","last_active":"2023-06-10T18:45:40.910052+0000","last_peered":"2023-06-10T18:45:40.910052+0000","last_clean":"2023-06-10T18:45:40.910052+0000","last_became_active":"2023-06-10T18:45:40.908770+0000","last_became_peered":"2023-06-10T18:45:40.908770+0000","last_unstale":"2023-06-10T18:45:40.910052+0000","last_undegraded":"2023-06-10T18:45:40.910052+0000","last_fullsized":"2023-06-10T18:45:40.910052+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:58:32.518563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895519+0000","last_change":"2023-06-10T18:45:40.895519+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895519+0000","last_undegraded":"2023-06-10T18:45:40.895519+0000","last_fullsized":"2023-06-10T18:45:40.895519+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912749+0000","last_change":"2023-06-10T18:45:40.909792+0000","last_active":"2023-06-10T18:45:40.912749+0000","last_peered":"2023-06-10T18:45:40.912749+0000","last_clean":"2023-06-10T18:45:40.912749+0000","last_became_active":"2023-06-10T18:45:40.909529+0000","last_became_peered":"2023-06-10T18:45:40.909529+0000","last_unstale":"2023-06-10T18:45:40.912749+0000","last_undegraded":"2023-06-10T18:45:40.912749+0000","last_fullsized":"2023-06-10T18:45:40.912749+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:35.699893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895527+0000","last_change":"2023-06-10T18:45:40.895527+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895527+0000","last_undegraded":"2023-06-10T18:45:40.895527+0000","last_fullsized":"2023-06-10T18:45:40.895527+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907177+0000","last_change":"2023-06-10T18:45:40.904862+0000","last_active":"2023-06-10T18:45:40.907177+0000","last_peered":"2023-06-10T18:45:40.907177+0000","last_clean":"2023-06-10T18:45:40.907177+0000","last_became_active":"2023-06-10T18:45:40.904590+0000","last_became_peered":"2023-06-10T18:45:40.904590+0000","last_unstale":"2023-06-10T18:45:40.907177+0000","last_undegraded":"2023-06-10T18:45:40.907177+0000","last_fullsized":"2023-06-10T18:45:40.907177+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:36:47.213889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891726+0000","last_change":"2023-06-10T18:45:40.891726+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891726+0000","last_undegraded":"2023-06-10T18:45:40.891726+0000","last_fullsized":"2023-06-10T18:45:40.891726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906313+0000","last_change":"2023-06-10T18:45:40.900179+0000","last_active":"2023-06-10T18:45:40.906313+0000","last_peered":"2023-06-10T18:45:40.906313+0000","last_clean":"2023-06-10T18:45:40.906313+0000","last_became_active":"2023-06-10T18:45:40.899762+0000","last_became_peered":"2023-06-10T18:45:40.899762+0000","last_unstale":"2023-06-10T18:45:40.906313+0000","last_undegraded":"2023-06-10T18:45:40.906313+0000","last_fullsized":"2023-06-10T18:45:40.906313+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:25:43.791507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.894081+0000","last_change":"2023-06-10T18:45:40.894081+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.894081+0000","last_undegraded":"2023-06-10T18:45:40.894081+0000","last_fullsized":"2023-06-10T18:45:40.894081+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910760+0000","last_change":"2023-06-10T18:45:40.904961+0000","last_active":"2023-06-10T18:45:40.910760+0000","last_peered":"2023-06-10T18:45:40.910760+0000","last_clean":"2023-06-10T18:45:40.910760+0000","last_became_active":"2023-06-10T18:45:40.904660+0000","last_became_peered":"2023-06-10T18:45:40.904660+0000","last_unstale":"2023-06-10T18:45:40.910760+0000","last_undegraded":"2023-06-10T18:45:40.910760+0000","last_fullsized":"2023-06-10T18:45:40.910760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:40.145269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.888355+0000","last_change":"2023-06-10T18:45:40.888355+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.888355+0000","last_undegraded":"2023-06-10T18:45:40.888355+0000","last_fullsized":"2023-06-10T18:45:40.888355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.907024+0000","last_change":"2023-06-10T18:45:40.905644+0000","last_active":"2023-06-10T18:45:40.907024+0000","last_peered":"2023-06-10T18:45:40.907024+0000","last_clean":"2023-06-10T18:45:40.907024+0000","last_became_active":"2023-06-10T18:45:40.905375+0000","last_became_peered":"2023-06-10T18:45:40.905375+0000","last_unstale":"2023-06-10T18:45:40.907024+0000","last_undegraded":"2023-06-10T18:45:40.907024+0000","last_fullsized":"2023-06-10T18:45:40.907024+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:28.816147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.895956+0000","last_change":"2023-06-10T18:45:40.895956+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.895956+0000","last_undegraded":"2023-06-10T18:45:40.895956+0000","last_fullsized":"2023-06-10T18:45:40.895956+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.909939+0000","last_change":"2023-06-10T18:45:40.903928+0000","last_active":"2023-06-10T18:45:40.909939+0000","last_peered":"2023-06-10T18:45:40.909939+0000","last_clean":"2023-06-10T18:45:40.909939+0000","last_became_active":"2023-06-10T18:45:40.903597+0000","last_became_peered":"2023-06-10T18:45:40.903597+0000","last_unstale":"2023-06-10T18:45:40.909939+0000","last_undegraded":"2023-06-10T18:45:40.909939+0000","last_fullsized":"2023-06-10T18:45:40.909939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:04.405310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.891920+0000","last_change":"2023-06-10T18:45:40.891920+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.891920+0000","last_undegraded":"2023-06-10T18:45:40.891920+0000","last_fullsized":"2023-06-10T18:45:40.891920+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.913712+0000","last_change":"2023-06-10T18:45:40.907369+0000","last_active":"2023-06-10T18:45:40.913712+0000","last_peered":"2023-06-10T18:45:40.913712+0000","last_clean":"2023-06-10T18:45:40.913712+0000","last_became_active":"2023-06-10T18:45:40.907099+0000","last_became_peered":"2023-06-10T18:45:40.907099+0000","last_unstale":"2023-06-10T18:45:40.913712+0000","last_undegraded":"2023-06-10T18:45:40.913712+0000","last_fullsized":"2023-06-10T18:45:40.913712+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:57.202185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896128+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:45:40.896128+0000","last_peered":"2023-06-10T18:45:40.896128+0000","last_clean":"2023-06-10T18:45:40.896128+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:45:40.896128+0000","last_undegraded":"2023-06-10T18:45:40.896128+0000","last_fullsized":"2023-06-10T18:45:40.896128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:35:09.859258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897049+0000","last_change":"2023-06-10T18:45:40.897049+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897049+0000","last_undegraded":"2023-06-10T18:45:40.897049+0000","last_fullsized":"2023-06-10T18:45:40.897049+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.906525+0000","last_change":"2023-06-10T18:45:40.900220+0000","last_active":"2023-06-10T18:45:40.906525+0000","last_peered":"2023-06-10T18:45:40.906525+0000","last_clean":"2023-06-10T18:45:40.906525+0000","last_became_active":"2023-06-10T18:45:40.899948+0000","last_became_peered":"2023-06-10T18:45:40.899948+0000","last_unstale":"2023-06-10T18:45:40.906525+0000","last_undegraded":"2023-06-10T18:45:40.906525+0000","last_fullsized":"2023-06-10T18:45:40.906525+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:24.478651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.891471+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:45:40.891471+0000","last_peered":"2023-06-10T18:45:40.891471+0000","last_clean":"2023-06-10T18:45:40.891471+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:45:40.891471+0000","last_undegraded":"2023-06-10T18:45:40.891471+0000","last_fullsized":"2023-06-10T18:45:40.891471+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:39.411023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.897646+0000","last_change":"2023-06-10T18:45:40.897646+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.897646+0000","last_undegraded":"2023-06-10T18:45:40.897646+0000","last_fullsized":"2023-06-10T18:45:40.897646+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912608+0000","last_change":"2023-06-10T18:45:40.909238+0000","last_active":"2023-06-10T18:45:40.912608+0000","last_peered":"2023-06-10T18:45:40.912608+0000","last_clean":"2023-06-10T18:45:40.912608+0000","last_became_active":"2023-06-10T18:45:40.908968+0000","last_became_peered":"2023-06-10T18:45:40.908968+0000","last_unstale":"2023-06-10T18:45:40.912608+0000","last_undegraded":"2023-06-10T18:45:40.912608+0000","last_fullsized":"2023-06-10T18:45:40.912608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:32.937415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.888090+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:45:40.888090+0000","last_peered":"2023-06-10T18:45:40.888090+0000","last_clean":"2023-06-10T18:45:40.888090+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:45:40.888090+0000","last_undegraded":"2023-06-10T18:45:40.888090+0000","last_fullsized":"2023-06-10T18:45:40.888090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:52:00.598598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.898298+0000","last_change":"2023-06-10T18:45:40.898298+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.898298+0000","last_undegraded":"2023-06-10T18:45:40.898298+0000","last_fullsized":"2023-06-10T18:45:40.898298+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.910745+0000","last_change":"2023-06-10T18:45:40.904255+0000","last_active":"2023-06-10T18:45:40.910745+0000","last_peered":"2023-06-10T18:45:40.910745+0000","last_clean":"2023-06-10T18:45:40.910745+0000","last_became_active":"2023-06-10T18:45:40.903971+0000","last_became_peered":"2023-06-10T18:45:40.903971+0000","last_unstale":"2023-06-10T18:45:40.910745+0000","last_undegraded":"2023-06-10T18:45:40.910745+0000","last_fullsized":"2023-06-10T18:45:40.910745+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:05.711201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.892795+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:45:40.892795+0000","last_peered":"2023-06-10T18:45:40.892795+0000","last_clean":"2023-06-10T18:45:40.892795+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:45:40.892795+0000","last_undegraded":"2023-06-10T18:45:40.892795+0000","last_fullsized":"2023-06-10T18:45:40.892795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:19.558248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":207,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894146+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:45:40.894146+0000","last_peered":"2023-06-10T18:45:40.894146+0000","last_clean":"2023-06-10T18:45:40.894146+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:45:40.894146+0000","last_undegraded":"2023-06-10T18:45:40.894146+0000","last_fullsized":"2023-06-10T18:45:40.894146+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:15:52.844548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890750+0000","last_change":"2023-06-10T18:45:40.890750+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890750+0000","last_undegraded":"2023-06-10T18:45:40.890750+0000","last_fullsized":"2023-06-10T18:45:40.890750+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914162+0000","last_change":"2023-06-10T18:45:40.908782+0000","last_active":"2023-06-10T18:45:40.914162+0000","last_peered":"2023-06-10T18:45:40.914162+0000","last_clean":"2023-06-10T18:45:40.914162+0000","last_became_active":"2023-06-10T18:45:40.908452+0000","last_became_peered":"2023-06-10T18:45:40.908452+0000","last_unstale":"2023-06-10T18:45:40.914162+0000","last_undegraded":"2023-06-10T18:45:40.914162+0000","last_fullsized":"2023-06-10T18:45:40.914162+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:16.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.894289+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:45:40.894289+0000","last_peered":"2023-06-10T18:45:40.894289+0000","last_clean":"2023-06-10T18:45:40.894289+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:45:40.894289+0000","last_undegraded":"2023-06-10T18:45:40.894289+0000","last_fullsized":"2023-06-10T18:45:40.894289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:21.044293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.889714+0000","last_change":"2023-06-10T18:45:40.889714+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.889714+0000","last_undegraded":"2023-06-10T18:45:40.889714+0000","last_fullsized":"2023-06-10T18:45:40.889714+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.914181+0000","last_change":"2023-06-10T18:45:40.908831+0000","last_active":"2023-06-10T18:45:40.914181+0000","last_peered":"2023-06-10T18:45:40.914181+0000","last_clean":"2023-06-10T18:45:40.914181+0000","last_became_active":"2023-06-10T18:45:40.908610+0000","last_became_peered":"2023-06-10T18:45:40.908610+0000","last_unstale":"2023-06-10T18:45:40.914181+0000","last_undegraded":"2023-06-10T18:45:40.914181+0000","last_fullsized":"2023-06-10T18:45:40.914181+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:41.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.898804+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:45:40.898804+0000","last_peered":"2023-06-10T18:45:40.898804+0000","last_clean":"2023-06-10T18:45:40.898804+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:45:40.898804+0000","last_undegraded":"2023-06-10T18:45:40.898804+0000","last_fullsized":"2023-06-10T18:45:40.898804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:29.019593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890847+0000","last_change":"2023-06-10T18:45:40.890847+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890847+0000","last_undegraded":"2023-06-10T18:45:40.890847+0000","last_fullsized":"2023-06-10T18:45:40.890847+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.905087+0000","last_change":"2023-06-10T18:45:40.903579+0000","last_active":"2023-06-10T18:45:40.905087+0000","last_peered":"2023-06-10T18:45:40.905087+0000","last_clean":"2023-06-10T18:45:40.905087+0000","last_became_active":"2023-06-10T18:45:40.903381+0000","last_became_peered":"2023-06-10T18:45:40.903381+0000","last_unstale":"2023-06-10T18:45:40.905087+0000","last_undegraded":"2023-06-10T18:45:40.905087+0000","last_fullsized":"2023-06-10T18:45:40.905087+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:24.524984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.896517+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:45:40.896517+0000","last_peered":"2023-06-10T18:45:40.896517+0000","last_clean":"2023-06-10T18:45:40.896517+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:45:40.896517+0000","last_undegraded":"2023-06-10T18:45:40.896517+0000","last_fullsized":"2023-06-10T18:45:40.896517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:47.124020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":40,"state":"creating+peering","last_fresh":"2023-06-10T18:45:40.890809+0000","last_change":"2023-06-10T18:45:40.890809+0000","last_active":"2023-06-10T18:45:40.881331+0000","last_peered":"2023-06-10T18:45:40.881331+0000","last_clean":"2023-06-10T18:45:40.881331+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:45:40.890809+0000","last_undegraded":"2023-06-10T18:45:40.890809+0000","last_fullsized":"2023-06-10T18:45:40.890809+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.912550+0000","last_change":"2023-06-10T18:45:40.908946+0000","last_active":"2023-06-10T18:45:40.912550+0000","last_peered":"2023-06-10T18:45:40.912550+0000","last_clean":"2023-06-10T18:45:40.912550+0000","last_became_active":"2023-06-10T18:45:40.908651+0000","last_became_peered":"2023-06-10T18:45:40.908651+0000","last_unstale":"2023-06-10T18:45:40.912550+0000","last_undegraded":"2023-06-10T18:45:40.912550+0000","last_fullsized":"2023-06-10T18:45:40.912550+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:23.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-06-10T18:45:40.897714+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:45:40.897714+0000","last_peered":"2023-06-10T18:45:40.897714+0000","last_clean":"2023-06-10T18:45:40.897714+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:45:40.897714+0000","last_undegraded":"2023-06-10T18:45:40.897714+0000","last_fullsized":"2023-06-10T18:45:40.897714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:14:14.376360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":1056768,"data_stored":794723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.625}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.6850000000000005,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.54400000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.315},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.28599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.497}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":1040384,"data_stored":786764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.245}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.501}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":452,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":462848,"data_stored":204355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.442}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.802,"5min":3.802,"15min":3.802},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.42899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":458752,"data_stored":203336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.379}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":450560,"data_stored":197415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.372},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.501}]},{"osd":4,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:45:50.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:45:50.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.330+0000 7f90ddecb700 1 Processor -- start 2023-06-10T18:45:50.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.330+0000 7f90ddecb700 1 -- start start 2023-06-10T18:45:50.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90ddecb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90ddecb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90ddecb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d80b6070 con 0x7f90d80b4ff0 2023-06-10T18:45:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90ddecb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d80b5460 con 0x7f90d815a420 2023-06-10T18:45:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90ddecb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80b5740 con 0x7f90d80cea60 2023-06-10T18:45:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d77fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d7fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d77fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:44954/0 (socket says 172.21.15.104:44954) 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d6ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f90d80b4ff0 0x7f90d80ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:32894/0 (socket says 172.21.15.104:32894) 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d77fe700 1 -- 172.21.15.104:0/997755395 learned_addr learned my addr 172.21.15.104:0/997755395 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.331+0000 7f90d7fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:48606/0 (socket says 172.21.15.104:48606) 2023-06-10T18:45:50.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d7fff700 1 -- 172.21.15.104:0/997755395 >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 msgr2=0x7f90d80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d7fff700 1 --2- 172.21.15.104:0/997755395 >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d7fff700 1 -- 172.21.15.104:0/997755395 >> v1:172.21.15.114:6789/0 conn(0x7f90d80b4ff0 legacy=0x7f90d80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d7fff700 1 -- 172.21.15.104:0/997755395 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d80b5a20 con 0x7f90d80cea60 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d77fe700 1 --2- 172.21.15.104:0/997755395 >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d7fff700 1 --2- 172.21.15.104:0/997755395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5a831aa1440ba server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d67fc700 1 -- 172.21.15.104:0/997755395 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d000e070 con 0x7f90d80cea60 2023-06-10T18:45:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d67fc700 1 -- 172.21.15.104:0/997755395 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d000bb20 con 0x7f90d80cea60 2023-06-10T18:45:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.332+0000 7f90d67fc700 1 -- 172.21.15.104:0/997755395 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0012430 con 0x7f90d80cea60 2023-06-10T18:45:50.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 -- 172.21.15.104:0/997755395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 msgr2=0x7f90d815a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 --2- 172.21.15.104:0/997755395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009d90 tx=0x7f90d000b330 comp rx=0 tx=0).stop 2023-06-10T18:45:50.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 -- 172.21.15.104:0/997755395 shutdown_connections 2023-06-10T18:45:50.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 --2- 172.21.15.104:0/997755395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d80cea60 0x7f90d815a7f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 --2- 172.21.15.104:0/997755395 >> v2:172.21.15.114:3300/0 conn(0x7f90d815a420 0x7f90d80b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 -- 172.21.15.104:0/997755395 >> 172.21.15.104:0/997755395 conn(0x7f90d8089fd0 msgr2=0x7f90d80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 -- 172.21.15.104:0/997755395 shutdown_connections 2023-06-10T18:45:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.333+0000 7f90ddecb700 1 -- 172.21.15.104:0/997755395 wait complete. 2023-06-10T18:45:50.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.334+0000 7f90ddecb700 1 Processor -- start 2023-06-10T18:45:50.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.334+0000 7f90ddecb700 1 -- start start 2023-06-10T18:45:50.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90d80cea60 0x7f90d80c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 0x7f90d80cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d8156340 con 0x7f90d80b4ff0 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d806b4f0 con 0x7f90d80cea60 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90ddecb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d806b7d0 con 0x7f90d815a420 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57456/0 (socket says 172.21.15.104:57456) 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 -- 172.21.15.104:0/3040255142 learned_addr learned my addr 172.21.15.104:0/3040255142 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d6ffd700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90d80cea60 0x7f90d80c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d7fff700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 0x7f90d80cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 -- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 msgr2=0x7f90d80cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 0x7f90d80cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 -- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90d80cea60 msgr2=0x7f90d80c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90d80cea60 0x7f90d80c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 -- 172.21.15.104:0/3040255142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc009550 con 0x7f90d80b4ff0 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d7fff700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 0x7f90d80cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.335+0000 7f90d77fe700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e9b0 tx=0x7f90cc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.336+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00c850 con 0x7f90d80b4ff0 2023-06-10T18:45:50.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.336+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc00c9b0 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.336+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d0009a70 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.336+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00cd20 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.336+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d806bd90 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.337+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d80c8560 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.337+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90cc013470 con 0x7f90d80b4ff0 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.337+0000 7f90bffff700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 0x7f90c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.338+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f90cc089680 con 0x7f90d80b4ff0 2023-06-10T18:45:50.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.338+0000 7f90d6ffd700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 0x7f90c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.338+0000 7f90d6ffd700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 0x7f90c406d190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f90d815de50 tx=0x7f90c8007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.342+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90cc0545f0 con 0x7f90d80b4ff0 2023-06-10T18:45:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.526+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90d80c83d0 con 0x7f90c406ace0 2023-06-10T18:45:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.535+0000 7f90bffff700 1 -- 172.21.15.104:0/3040255142 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311388 (secure 0 0 0) 0x7f90d80c83d0 con 0x7f90c406ace0 2023-06-10T18:45:50.543 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 msgr2=0x7f90c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 0x7f90c406d190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f90d815de50 tx=0x7f90c8007ed0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 msgr2=0x7f90d8155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e9b0 tx=0x7f90cc00ecc0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 shutdown_connections 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90d815a420 0x7f90d80cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d80b4ff0 0x7f90d8155e00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90c406ace0 0x7f90c406d190 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 --2- 172.21.15.104:0/3040255142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90d80cea60 0x7f90d80c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 >> 172.21.15.104:0/3040255142 conn(0x7f90d8089fd0 msgr2=0x7f90d8157f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:50.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 shutdown_connections 2023-06-10T18:45:50.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.538+0000 7f90ddecb700 1 -- 172.21.15.104:0/3040255142 wait complete. 2023-06-10T18:45:50.547 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:45:50.556 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-06-10T18:45:49.374905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220632,"kb_used_data":5016,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754088,"statfs":{"total":773094113280,"available":772868186112,"internally_reserved":0,"allocated":5136384,"data_stored":2931592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002005"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229809+0000","last_change":"2023-06-10T18:45:41.454429+0000","last_active":"2023-06-10T18:45:43.229809+0000","last_peered":"2023-06-10T18:45:43.229809+0000","last_clean":"2023-06-10T18:45:43.229809+0000","last_became_active":"2023-06-10T18:45:41.454109+0000","last_became_peered":"2023-06-10T18:45:41.454109+0000","last_unstale":"2023-06-10T18:45:43.229809+0000","last_undegraded":"2023-06-10T18:45:43.229809+0000","last_fullsized":"2023-06-10T18:45:43.229809+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:00:54.060211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.447308+0000","last_change":"2023-06-10T18:45:41.459424+0000","last_active":"2023-06-10T18:45:42.447308+0000","last_peered":"2023-06-10T18:45:42.447308+0000","last_clean":"2023-06-10T18:45:42.447308+0000","last_became_active":"2023-06-10T18:45:41.459101+0000","last_became_peered":"2023-06-10T18:45:41.459101+0000","last_unstale":"2023-06-10T18:45:42.447308+0000","last_undegraded":"2023-06-10T18:45:42.447308+0000","last_fullsized":"2023-06-10T18:45:42.447308+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:43:27.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446979+0000","last_change":"2023-06-10T18:45:41.460764+0000","last_active":"2023-06-10T18:45:42.446979+0000","last_peered":"2023-06-10T18:45:42.446979+0000","last_clean":"2023-06-10T18:45:42.446979+0000","last_became_active":"2023-06-10T18:45:41.459717+0000","last_became_peered":"2023-06-10T18:45:41.459717+0000","last_unstale":"2023-06-10T18:45:42.446979+0000","last_undegraded":"2023-06-10T18:45:42.446979+0000","last_fullsized":"2023-06-10T18:45:42.446979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:00.475007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230052+0000","last_change":"2023-06-10T18:45:41.468690+0000","last_active":"2023-06-10T18:45:43.230052+0000","last_peered":"2023-06-10T18:45:43.230052+0000","last_clean":"2023-06-10T18:45:43.230052+0000","last_became_active":"2023-06-10T18:45:41.468406+0000","last_became_peered":"2023-06-10T18:45:41.468406+0000","last_unstale":"2023-06-10T18:45:43.230052+0000","last_undegraded":"2023-06-10T18:45:43.230052+0000","last_fullsized":"2023-06-10T18:45:43.230052+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:36.706565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.445960+0000","last_change":"2023-06-10T18:45:41.461128+0000","last_active":"2023-06-10T18:45:42.445960+0000","last_peered":"2023-06-10T18:45:42.445960+0000","last_clean":"2023-06-10T18:45:42.445960+0000","last_became_active":"2023-06-10T18:45:41.460879+0000","last_became_peered":"2023-06-10T18:45:41.460879+0000","last_unstale":"2023-06-10T18:45:42.445960+0000","last_undegraded":"2023-06-10T18:45:42.445960+0000","last_fullsized":"2023-06-10T18:45:42.445960+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:31:03.710676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443288+0000","last_change":"2023-06-10T18:45:41.452525+0000","last_active":"2023-06-10T18:45:42.443288+0000","last_peered":"2023-06-10T18:45:42.443288+0000","last_clean":"2023-06-10T18:45:42.443288+0000","last_became_active":"2023-06-10T18:45:41.452270+0000","last_became_peered":"2023-06-10T18:45:41.452270+0000","last_unstale":"2023-06-10T18:45:42.443288+0000","last_undegraded":"2023-06-10T18:45:42.443288+0000","last_fullsized":"2023-06-10T18:45:42.443288+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:36.706565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229428+0000","last_change":"2023-06-10T18:45:41.453437+0000","last_active":"2023-06-10T18:45:43.229428+0000","last_peered":"2023-06-10T18:45:43.229428+0000","last_clean":"2023-06-10T18:45:43.229428+0000","last_became_active":"2023-06-10T18:45:41.453098+0000","last_became_peered":"2023-06-10T18:45:41.453098+0000","last_unstale":"2023-06-10T18:45:43.229428+0000","last_undegraded":"2023-06-10T18:45:43.229428+0000","last_fullsized":"2023-06-10T18:45:43.229428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:31:03.710676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229275+0000","last_change":"2023-06-10T18:45:41.468850+0000","last_active":"2023-06-10T18:45:43.229275+0000","last_peered":"2023-06-10T18:45:43.229275+0000","last_clean":"2023-06-10T18:45:43.229275+0000","last_became_active":"2023-06-10T18:45:41.468584+0000","last_became_peered":"2023-06-10T18:45:41.468584+0000","last_unstale":"2023-06-10T18:45:43.229275+0000","last_undegraded":"2023-06-10T18:45:43.229275+0000","last_fullsized":"2023-06-10T18:45:43.229275+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:09.338601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230150+0000","last_change":"2023-06-10T18:45:41.466710+0000","last_active":"2023-06-10T18:45:43.230150+0000","last_peered":"2023-06-10T18:45:43.230150+0000","last_clean":"2023-06-10T18:45:43.230150+0000","last_became_active":"2023-06-10T18:45:41.466420+0000","last_became_peered":"2023-06-10T18:45:41.466420+0000","last_unstale":"2023-06-10T18:45:43.230150+0000","last_undegraded":"2023-06-10T18:45:43.230150+0000","last_fullsized":"2023-06-10T18:45:43.230150+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:33.944087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230348+0000","last_change":"2023-06-10T18:45:41.468314+0000","last_active":"2023-06-10T18:45:43.230348+0000","last_peered":"2023-06-10T18:45:43.230348+0000","last_clean":"2023-06-10T18:45:43.230348+0000","last_became_active":"2023-06-10T18:45:41.468016+0000","last_became_peered":"2023-06-10T18:45:41.468016+0000","last_unstale":"2023-06-10T18:45:43.230348+0000","last_undegraded":"2023-06-10T18:45:43.230348+0000","last_fullsized":"2023-06-10T18:45:43.230348+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:21:19.554845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230055+0000","last_change":"2023-06-10T18:45:41.466806+0000","last_active":"2023-06-10T18:45:43.230055+0000","last_peered":"2023-06-10T18:45:43.230055+0000","last_clean":"2023-06-10T18:45:43.230055+0000","last_became_active":"2023-06-10T18:45:41.466478+0000","last_became_peered":"2023-06-10T18:45:41.466478+0000","last_unstale":"2023-06-10T18:45:43.230055+0000","last_undegraded":"2023-06-10T18:45:43.230055+0000","last_fullsized":"2023-06-10T18:45:43.230055+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:01.124684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446888+0000","last_change":"2023-06-10T18:45:41.462157+0000","last_active":"2023-06-10T18:45:42.446888+0000","last_peered":"2023-06-10T18:45:42.446888+0000","last_clean":"2023-06-10T18:45:42.446888+0000","last_became_active":"2023-06-10T18:45:41.461586+0000","last_became_peered":"2023-06-10T18:45:41.461586+0000","last_unstale":"2023-06-10T18:45:42.446888+0000","last_undegraded":"2023-06-10T18:45:42.446888+0000","last_fullsized":"2023-06-10T18:45:42.446888+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:42:52.422815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229826+0000","last_change":"2023-06-10T18:45:41.457374+0000","last_active":"2023-06-10T18:45:43.229826+0000","last_peered":"2023-06-10T18:45:43.229826+0000","last_clean":"2023-06-10T18:45:43.229826+0000","last_became_active":"2023-06-10T18:45:41.456912+0000","last_became_peered":"2023-06-10T18:45:41.456912+0000","last_unstale":"2023-06-10T18:45:43.229826+0000","last_undegraded":"2023-06-10T18:45:43.229826+0000","last_fullsized":"2023-06-10T18:45:43.229826+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:06.404752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229489+0000","last_change":"2023-06-10T18:45:41.466706+0000","last_active":"2023-06-10T18:45:43.229489+0000","last_peered":"2023-06-10T18:45:43.229489+0000","last_clean":"2023-06-10T18:45:43.229489+0000","last_became_active":"2023-06-10T18:45:41.466436+0000","last_became_peered":"2023-06-10T18:45:41.466436+0000","last_unstale":"2023-06-10T18:45:43.229489+0000","last_undegraded":"2023-06-10T18:45:43.229489+0000","last_fullsized":"2023-06-10T18:45:43.229489+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:20:48.419228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443761+0000","last_change":"2023-06-10T18:45:41.458756+0000","last_active":"2023-06-10T18:45:42.443761+0000","last_peered":"2023-06-10T18:45:42.443761+0000","last_clean":"2023-06-10T18:45:42.443761+0000","last_became_active":"2023-06-10T18:45:41.458527+0000","last_became_peered":"2023-06-10T18:45:41.458527+0000","last_unstale":"2023-06-10T18:45:42.443761+0000","last_undegraded":"2023-06-10T18:45:42.443761+0000","last_fullsized":"2023-06-10T18:45:42.443761+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:01.982526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481009+0000","last_change":"2023-06-10T18:45:41.465841+0000","last_active":"2023-06-10T18:45:42.481009+0000","last_peered":"2023-06-10T18:45:42.481009+0000","last_clean":"2023-06-10T18:45:42.481009+0000","last_became_active":"2023-06-10T18:45:41.465590+0000","last_became_peered":"2023-06-10T18:45:41.465590+0000","last_unstale":"2023-06-10T18:45:42.481009+0000","last_undegraded":"2023-06-10T18:45:42.481009+0000","last_fullsized":"2023-06-10T18:45:42.481009+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:52:50.742193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446482+0000","last_change":"2023-06-10T18:45:41.458889+0000","last_active":"2023-06-10T18:45:42.446482+0000","last_peered":"2023-06-10T18:45:42.446482+0000","last_clean":"2023-06-10T18:45:42.446482+0000","last_became_active":"2023-06-10T18:45:41.458614+0000","last_became_peered":"2023-06-10T18:45:41.458614+0000","last_unstale":"2023-06-10T18:45:42.446482+0000","last_undegraded":"2023-06-10T18:45:42.446482+0000","last_fullsized":"2023-06-10T18:45:42.446482+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:20:41.794572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446521+0000","last_change":"2023-06-10T18:45:41.460564+0000","last_active":"2023-06-10T18:45:42.446521+0000","last_peered":"2023-06-10T18:45:42.446521+0000","last_clean":"2023-06-10T18:45:42.446521+0000","last_became_active":"2023-06-10T18:45:41.460291+0000","last_became_peered":"2023-06-10T18:45:41.460291+0000","last_unstale":"2023-06-10T18:45:42.446521+0000","last_undegraded":"2023-06-10T18:45:42.446521+0000","last_fullsized":"2023-06-10T18:45:42.446521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:15.183496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481410+0000","last_change":"2023-06-10T18:45:41.466402+0000","last_active":"2023-06-10T18:45:42.481410+0000","last_peered":"2023-06-10T18:45:42.481410+0000","last_clean":"2023-06-10T18:45:42.481410+0000","last_became_active":"2023-06-10T18:45:41.465924+0000","last_became_peered":"2023-06-10T18:45:41.465924+0000","last_unstale":"2023-06-10T18:45:42.481410+0000","last_undegraded":"2023-06-10T18:45:42.481410+0000","last_fullsized":"2023-06-10T18:45:42.481410+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:09:03.598790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229402+0000","last_change":"2023-06-10T18:45:41.469867+0000","last_active":"2023-06-10T18:45:43.229402+0000","last_peered":"2023-06-10T18:45:43.229402+0000","last_clean":"2023-06-10T18:45:43.229402+0000","last_became_active":"2023-06-10T18:45:41.469314+0000","last_became_peered":"2023-06-10T18:45:41.469314+0000","last_unstale":"2023-06-10T18:45:43.229402+0000","last_undegraded":"2023-06-10T18:45:43.229402+0000","last_fullsized":"2023-06-10T18:45:43.229402+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:46:03.243372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443253+0000","last_change":"2023-06-10T18:45:41.464630+0000","last_active":"2023-06-10T18:45:42.443253+0000","last_peered":"2023-06-10T18:45:42.443253+0000","last_clean":"2023-06-10T18:45:42.443253+0000","last_became_active":"2023-06-10T18:45:41.464373+0000","last_became_peered":"2023-06-10T18:45:41.464373+0000","last_unstale":"2023-06-10T18:45:42.443253+0000","last_undegraded":"2023-06-10T18:45:42.443253+0000","last_fullsized":"2023-06-10T18:45:42.443253+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:11.824175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229637+0000","last_change":"2023-06-10T18:45:41.463840+0000","last_active":"2023-06-10T18:45:43.229637+0000","last_peered":"2023-06-10T18:45:43.229637+0000","last_clean":"2023-06-10T18:45:43.229637+0000","last_became_active":"2023-06-10T18:45:41.463583+0000","last_became_peered":"2023-06-10T18:45:41.463583+0000","last_unstale":"2023-06-10T18:45:43.229637+0000","last_undegraded":"2023-06-10T18:45:43.229637+0000","last_fullsized":"2023-06-10T18:45:43.229637+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:37.360697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481064+0000","last_change":"2023-06-10T18:45:41.465581+0000","last_active":"2023-06-10T18:45:42.481064+0000","last_peered":"2023-06-10T18:45:42.481064+0000","last_clean":"2023-06-10T18:45:42.481064+0000","last_became_active":"2023-06-10T18:45:41.465140+0000","last_became_peered":"2023-06-10T18:45:41.465140+0000","last_unstale":"2023-06-10T18:45:42.481064+0000","last_undegraded":"2023-06-10T18:45:42.481064+0000","last_fullsized":"2023-06-10T18:45:42.481064+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:18:46.909678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481242+0000","last_change":"2023-06-10T18:45:41.467091+0000","last_active":"2023-06-10T18:45:42.481242+0000","last_peered":"2023-06-10T18:45:42.481242+0000","last_clean":"2023-06-10T18:45:42.481242+0000","last_became_active":"2023-06-10T18:45:41.466755+0000","last_became_peered":"2023-06-10T18:45:41.466755+0000","last_unstale":"2023-06-10T18:45:42.481242+0000","last_undegraded":"2023-06-10T18:45:42.481242+0000","last_fullsized":"2023-06-10T18:45:42.481242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:55:56.021498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481491+0000","last_change":"2023-06-10T18:45:41.463571+0000","last_active":"2023-06-10T18:45:42.481491+0000","last_peered":"2023-06-10T18:45:42.481491+0000","last_clean":"2023-06-10T18:45:42.481491+0000","last_became_active":"2023-06-10T18:45:41.463256+0000","last_became_peered":"2023-06-10T18:45:41.463256+0000","last_unstale":"2023-06-10T18:45:42.481491+0000","last_undegraded":"2023-06-10T18:45:42.481491+0000","last_fullsized":"2023-06-10T18:45:42.481491+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:48:44.048286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230488+0000","last_change":"2023-06-10T18:45:41.463951+0000","last_active":"2023-06-10T18:45:43.230488+0000","last_peered":"2023-06-10T18:45:43.230488+0000","last_clean":"2023-06-10T18:45:43.230488+0000","last_became_active":"2023-06-10T18:45:41.463492+0000","last_became_peered":"2023-06-10T18:45:41.463492+0000","last_unstale":"2023-06-10T18:45:43.230488+0000","last_undegraded":"2023-06-10T18:45:43.230488+0000","last_fullsized":"2023-06-10T18:45:43.230488+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:43.035428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.447118+0000","last_change":"2023-06-10T18:45:41.458638+0000","last_active":"2023-06-10T18:45:42.447118+0000","last_peered":"2023-06-10T18:45:42.447118+0000","last_clean":"2023-06-10T18:45:42.447118+0000","last_became_active":"2023-06-10T18:45:41.458364+0000","last_became_peered":"2023-06-10T18:45:41.458364+0000","last_unstale":"2023-06-10T18:45:42.447118+0000","last_undegraded":"2023-06-10T18:45:42.447118+0000","last_fullsized":"2023-06-10T18:45:42.447118+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:59:50.314860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446494+0000","last_change":"2023-06-10T18:45:41.462328+0000","last_active":"2023-06-10T18:45:42.446494+0000","last_peered":"2023-06-10T18:45:42.446494+0000","last_clean":"2023-06-10T18:45:42.446494+0000","last_became_active":"2023-06-10T18:45:41.461914+0000","last_became_peered":"2023-06-10T18:45:41.461914+0000","last_unstale":"2023-06-10T18:45:42.446494+0000","last_undegraded":"2023-06-10T18:45:42.446494+0000","last_fullsized":"2023-06-10T18:45:42.446494+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:57:27.560645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481248+0000","last_change":"2023-06-10T18:45:41.463531+0000","last_active":"2023-06-10T18:45:42.481248+0000","last_peered":"2023-06-10T18:45:42.481248+0000","last_clean":"2023-06-10T18:45:42.481248+0000","last_became_active":"2023-06-10T18:45:41.463209+0000","last_became_peered":"2023-06-10T18:45:41.463209+0000","last_unstale":"2023-06-10T18:45:42.481248+0000","last_undegraded":"2023-06-10T18:45:42.481248+0000","last_fullsized":"2023-06-10T18:45:42.481248+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:57:43.702937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.445798+0000","last_change":"2023-06-10T18:45:41.463747+0000","last_active":"2023-06-10T18:45:42.445798+0000","last_peered":"2023-06-10T18:45:42.445798+0000","last_clean":"2023-06-10T18:45:42.445798+0000","last_became_active":"2023-06-10T18:45:41.462206+0000","last_became_peered":"2023-06-10T18:45:41.462206+0000","last_unstale":"2023-06-10T18:45:42.445798+0000","last_undegraded":"2023-06-10T18:45:42.445798+0000","last_fullsized":"2023-06-10T18:45:42.445798+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:00:54.060211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443095+0000","last_change":"2023-06-10T18:45:41.457570+0000","last_active":"2023-06-10T18:45:42.443095+0000","last_peered":"2023-06-10T18:45:42.443095+0000","last_clean":"2023-06-10T18:45:42.443095+0000","last_became_active":"2023-06-10T18:45:41.457317+0000","last_became_peered":"2023-06-10T18:45:41.457317+0000","last_unstale":"2023-06-10T18:45:42.443095+0000","last_undegraded":"2023-06-10T18:45:42.443095+0000","last_fullsized":"2023-06-10T18:45:42.443095+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:06.717874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230607+0000","last_change":"2023-06-10T18:45:41.463629+0000","last_active":"2023-06-10T18:45:43.230607+0000","last_peered":"2023-06-10T18:45:43.230607+0000","last_clean":"2023-06-10T18:45:43.230607+0000","last_became_active":"2023-06-10T18:45:41.463373+0000","last_became_peered":"2023-06-10T18:45:41.463373+0000","last_unstale":"2023-06-10T18:45:43.230607+0000","last_undegraded":"2023-06-10T18:45:43.230607+0000","last_fullsized":"2023-06-10T18:45:43.230607+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:34:20.038115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230324+0000","last_change":"2023-06-10T18:45:41.453663+0000","last_active":"2023-06-10T18:45:43.230324+0000","last_peered":"2023-06-10T18:45:43.230324+0000","last_clean":"2023-06-10T18:45:43.230324+0000","last_became_active":"2023-06-10T18:45:41.453429+0000","last_became_peered":"2023-06-10T18:45:41.453429+0000","last_unstale":"2023-06-10T18:45:43.230324+0000","last_undegraded":"2023-06-10T18:45:43.230324+0000","last_fullsized":"2023-06-10T18:45:43.230324+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:53:33.944087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":17,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443472+0000","last_change":"2023-06-10T18:45:40.904978+0000","last_active":"2023-06-10T18:45:42.443472+0000","last_peered":"2023-06-10T18:45:42.443472+0000","last_clean":"2023-06-10T18:45:42.443472+0000","last_became_active":"2023-06-10T18:45:40.904678+0000","last_became_peered":"2023-06-10T18:45:40.904678+0000","last_unstale":"2023-06-10T18:45:42.443472+0000","last_undegraded":"2023-06-10T18:45:42.443472+0000","last_fullsized":"2023-06-10T18:45:42.443472+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:47.654330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481345+0000","last_change":"2023-06-10T18:45:41.464921+0000","last_active":"2023-06-10T18:45:42.481345+0000","last_peered":"2023-06-10T18:45:42.481345+0000","last_clean":"2023-06-10T18:45:42.481345+0000","last_became_active":"2023-06-10T18:45:41.464656+0000","last_became_peered":"2023-06-10T18:45:41.464656+0000","last_unstale":"2023-06-10T18:45:42.481345+0000","last_undegraded":"2023-06-10T18:45:42.481345+0000","last_fullsized":"2023-06-10T18:45:42.481345+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:09:58.501221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"41'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446835+0000","last_change":"2023-06-10T18:45:40.904727+0000","last_active":"2023-06-10T18:45:42.446835+0000","last_peered":"2023-06-10T18:45:42.446835+0000","last_clean":"2023-06-10T18:45:42.446835+0000","last_became_active":"2023-06-10T18:45:40.904429+0000","last_became_peered":"2023-06-10T18:45:40.904429+0000","last_unstale":"2023-06-10T18:45:42.446835+0000","last_undegraded":"2023-06-10T18:45:42.446835+0000","last_fullsized":"2023-06-10T18:45:42.446835+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:39:47.543338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446637+0000","last_change":"2023-06-10T18:45:41.457379+0000","last_active":"2023-06-10T18:45:42.446637+0000","last_peered":"2023-06-10T18:45:42.446637+0000","last_clean":"2023-06-10T18:45:42.446637+0000","last_became_active":"2023-06-10T18:45:41.457100+0000","last_became_peered":"2023-06-10T18:45:41.457100+0000","last_unstale":"2023-06-10T18:45:42.446637+0000","last_undegraded":"2023-06-10T18:45:42.446637+0000","last_fullsized":"2023-06-10T18:45:42.446637+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:48.484682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230173+0000","last_change":"2023-06-10T18:45:40.908799+0000","last_active":"2023-06-10T18:45:43.230173+0000","last_peered":"2023-06-10T18:45:43.230173+0000","last_clean":"2023-06-10T18:45:43.230173+0000","last_became_active":"2023-06-10T18:45:40.908494+0000","last_became_peered":"2023-06-10T18:45:40.908494+0000","last_unstale":"2023-06-10T18:45:43.230173+0000","last_undegraded":"2023-06-10T18:45:43.230173+0000","last_fullsized":"2023-06-10T18:45:43.230173+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:33.234927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230035+0000","last_change":"2023-06-10T18:45:41.468109+0000","last_active":"2023-06-10T18:45:43.230035+0000","last_peered":"2023-06-10T18:45:43.230035+0000","last_clean":"2023-06-10T18:45:43.230035+0000","last_became_active":"2023-06-10T18:45:41.467818+0000","last_became_peered":"2023-06-10T18:45:41.467818+0000","last_unstale":"2023-06-10T18:45:43.230035+0000","last_undegraded":"2023-06-10T18:45:43.230035+0000","last_fullsized":"2023-06-10T18:45:43.230035+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:59.405597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229420+0000","last_change":"2023-06-10T18:45:40.904441+0000","last_active":"2023-06-10T18:45:43.229420+0000","last_peered":"2023-06-10T18:45:43.229420+0000","last_clean":"2023-06-10T18:45:43.229420+0000","last_became_active":"2023-06-10T18:45:40.904169+0000","last_became_peered":"2023-06-10T18:45:40.904169+0000","last_unstale":"2023-06-10T18:45:43.229420+0000","last_undegraded":"2023-06-10T18:45:43.229420+0000","last_fullsized":"2023-06-10T18:45:43.229420+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:03:12.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.445995+0000","last_change":"2023-06-10T18:45:41.466176+0000","last_active":"2023-06-10T18:45:42.445995+0000","last_peered":"2023-06-10T18:45:42.445995+0000","last_clean":"2023-06-10T18:45:42.445995+0000","last_became_active":"2023-06-10T18:45:41.465918+0000","last_became_peered":"2023-06-10T18:45:41.465918+0000","last_unstale":"2023-06-10T18:45:42.445995+0000","last_undegraded":"2023-06-10T18:45:42.445995+0000","last_fullsized":"2023-06-10T18:45:42.445995+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:33:45.969358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443439+0000","last_change":"2023-06-10T18:45:40.900293+0000","last_active":"2023-06-10T18:45:42.443439+0000","last_peered":"2023-06-10T18:45:42.443439+0000","last_clean":"2023-06-10T18:45:42.443439+0000","last_became_active":"2023-06-10T18:45:40.900103+0000","last_became_peered":"2023-06-10T18:45:40.900103+0000","last_unstale":"2023-06-10T18:45:42.443439+0000","last_undegraded":"2023-06-10T18:45:42.443439+0000","last_fullsized":"2023-06-10T18:45:42.443439+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:50:24.104866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229755+0000","last_change":"2023-06-10T18:45:41.467626+0000","last_active":"2023-06-10T18:45:43.229755+0000","last_peered":"2023-06-10T18:45:43.229755+0000","last_clean":"2023-06-10T18:45:43.229755+0000","last_became_active":"2023-06-10T18:45:41.467369+0000","last_became_peered":"2023-06-10T18:45:41.467369+0000","last_unstale":"2023-06-10T18:45:43.229755+0000","last_undegraded":"2023-06-10T18:45:43.229755+0000","last_fullsized":"2023-06-10T18:45:43.229755+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:05.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"41'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446418+0000","last_change":"2023-06-10T18:45:40.903527+0000","last_active":"2023-06-10T18:45:42.446418+0000","last_peered":"2023-06-10T18:45:42.446418+0000","last_clean":"2023-06-10T18:45:42.446418+0000","last_became_active":"2023-06-10T18:45:40.903286+0000","last_became_peered":"2023-06-10T18:45:40.903286+0000","last_unstale":"2023-06-10T18:45:42.446418+0000","last_undegraded":"2023-06-10T18:45:42.446418+0000","last_fullsized":"2023-06-10T18:45:42.446418+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:26.422134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229424+0000","last_change":"2023-06-10T18:45:41.468997+0000","last_active":"2023-06-10T18:45:43.229424+0000","last_peered":"2023-06-10T18:45:43.229424+0000","last_clean":"2023-06-10T18:45:43.229424+0000","last_became_active":"2023-06-10T18:45:41.468691+0000","last_became_peered":"2023-06-10T18:45:41.468691+0000","last_unstale":"2023-06-10T18:45:43.229424+0000","last_undegraded":"2023-06-10T18:45:43.229424+0000","last_fullsized":"2023-06-10T18:45:43.229424+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:51:30.778890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.445943+0000","last_change":"2023-06-10T18:45:40.909125+0000","last_active":"2023-06-10T18:45:42.445943+0000","last_peered":"2023-06-10T18:45:42.445943+0000","last_clean":"2023-06-10T18:45:42.445943+0000","last_became_active":"2023-06-10T18:45:40.908861+0000","last_became_peered":"2023-06-10T18:45:40.908861+0000","last_unstale":"2023-06-10T18:45:42.445943+0000","last_undegraded":"2023-06-10T18:45:42.445943+0000","last_fullsized":"2023-06-10T18:45:42.445943+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:54.327736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481188+0000","last_change":"2023-06-10T18:45:41.462804+0000","last_active":"2023-06-10T18:45:42.481188+0000","last_peered":"2023-06-10T18:45:42.481188+0000","last_clean":"2023-06-10T18:45:42.481188+0000","last_became_active":"2023-06-10T18:45:41.462437+0000","last_became_peered":"2023-06-10T18:45:41.462437+0000","last_unstale":"2023-06-10T18:45:42.481188+0000","last_undegraded":"2023-06-10T18:45:42.481188+0000","last_fullsized":"2023-06-10T18:45:42.481188+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:18.731529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229545+0000","last_change":"2023-06-10T18:45:40.914575+0000","last_active":"2023-06-10T18:45:43.229545+0000","last_peered":"2023-06-10T18:45:43.229545+0000","last_clean":"2023-06-10T18:45:43.229545+0000","last_became_active":"2023-06-10T18:45:40.914270+0000","last_became_peered":"2023-06-10T18:45:40.914270+0000","last_unstale":"2023-06-10T18:45:43.229545+0000","last_undegraded":"2023-06-10T18:45:43.229545+0000","last_fullsized":"2023-06-10T18:45:43.229545+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:06:41.071417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446737+0000","last_change":"2023-06-10T18:45:41.461333+0000","last_active":"2023-06-10T18:45:42.446737+0000","last_peered":"2023-06-10T18:45:42.446737+0000","last_clean":"2023-06-10T18:45:42.446737+0000","last_became_active":"2023-06-10T18:45:41.461069+0000","last_became_peered":"2023-06-10T18:45:41.461069+0000","last_unstale":"2023-06-10T18:45:42.446737+0000","last_undegraded":"2023-06-10T18:45:42.446737+0000","last_fullsized":"2023-06-10T18:45:42.446737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:34:42.535188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446693+0000","last_change":"2023-06-10T18:45:40.902582+0000","last_active":"2023-06-10T18:45:42.446693+0000","last_peered":"2023-06-10T18:45:42.446693+0000","last_clean":"2023-06-10T18:45:42.446693+0000","last_became_active":"2023-06-10T18:45:40.902303+0000","last_became_peered":"2023-06-10T18:45:40.902303+0000","last_unstale":"2023-06-10T18:45:42.446693+0000","last_undegraded":"2023-06-10T18:45:42.446693+0000","last_fullsized":"2023-06-10T18:45:42.446693+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:23:57.826121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229806+0000","last_change":"2023-06-10T18:45:41.454877+0000","last_active":"2023-06-10T18:45:43.229806+0000","last_peered":"2023-06-10T18:45:43.229806+0000","last_clean":"2023-06-10T18:45:43.229806+0000","last_became_active":"2023-06-10T18:45:41.454664+0000","last_became_peered":"2023-06-10T18:45:41.454664+0000","last_unstale":"2023-06-10T18:45:43.229806+0000","last_undegraded":"2023-06-10T18:45:43.229806+0000","last_fullsized":"2023-06-10T18:45:43.229806+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:43:27.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230492+0000","last_change":"2023-06-10T18:45:40.905531+0000","last_active":"2023-06-10T18:45:43.230492+0000","last_peered":"2023-06-10T18:45:43.230492+0000","last_clean":"2023-06-10T18:45:43.230492+0000","last_became_active":"2023-06-10T18:45:40.905204+0000","last_became_peered":"2023-06-10T18:45:40.905204+0000","last_unstale":"2023-06-10T18:45:43.230492+0000","last_undegraded":"2023-06-10T18:45:43.230492+0000","last_fullsized":"2023-06-10T18:45:43.230492+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:40.145269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443393+0000","last_change":"2023-06-10T18:45:41.461719+0000","last_active":"2023-06-10T18:45:42.443393+0000","last_peered":"2023-06-10T18:45:42.443393+0000","last_clean":"2023-06-10T18:45:42.443393+0000","last_became_active":"2023-06-10T18:45:41.461459+0000","last_became_peered":"2023-06-10T18:45:41.461459+0000","last_unstale":"2023-06-10T18:45:42.443393+0000","last_undegraded":"2023-06-10T18:45:42.443393+0000","last_fullsized":"2023-06-10T18:45:42.443393+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:32:08.233022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229720+0000","last_change":"2023-06-10T18:45:40.909552+0000","last_active":"2023-06-10T18:45:43.229720+0000","last_peered":"2023-06-10T18:45:43.229720+0000","last_clean":"2023-06-10T18:45:43.229720+0000","last_became_active":"2023-06-10T18:45:40.909075+0000","last_became_peered":"2023-06-10T18:45:40.909075+0000","last_unstale":"2023-06-10T18:45:43.229720+0000","last_undegraded":"2023-06-10T18:45:43.229720+0000","last_fullsized":"2023-06-10T18:45:43.229720+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:47.478009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229656+0000","last_change":"2023-06-10T18:45:41.452905+0000","last_active":"2023-06-10T18:45:43.229656+0000","last_peered":"2023-06-10T18:45:43.229656+0000","last_clean":"2023-06-10T18:45:43.229656+0000","last_became_active":"2023-06-10T18:45:41.452612+0000","last_became_peered":"2023-06-10T18:45:41.452612+0000","last_unstale":"2023-06-10T18:45:43.229656+0000","last_undegraded":"2023-06-10T18:45:43.229656+0000","last_fullsized":"2023-06-10T18:45:43.229656+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:57:27.560645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":19,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229509+0000","last_change":"2023-06-10T18:45:40.899865+0000","last_active":"2023-06-10T18:45:43.229509+0000","last_peered":"2023-06-10T18:45:43.229509+0000","last_clean":"2023-06-10T18:45:43.229509+0000","last_became_active":"2023-06-10T18:45:40.899613+0000","last_became_peered":"2023-06-10T18:45:40.899613+0000","last_unstale":"2023-06-10T18:45:43.229509+0000","last_undegraded":"2023-06-10T18:45:43.229509+0000","last_fullsized":"2023-06-10T18:45:43.229509+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:35.796528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481068+0000","last_change":"2023-06-10T18:45:41.462808+0000","last_active":"2023-06-10T18:45:42.481068+0000","last_peered":"2023-06-10T18:45:42.481068+0000","last_clean":"2023-06-10T18:45:42.481068+0000","last_became_active":"2023-06-10T18:45:41.462357+0000","last_became_peered":"2023-06-10T18:45:41.462357+0000","last_unstale":"2023-06-10T18:45:42.481068+0000","last_undegraded":"2023-06-10T18:45:42.481068+0000","last_fullsized":"2023-06-10T18:45:42.481068+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:44:45.424249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443581+0000","last_change":"2023-06-10T18:45:40.906074+0000","last_active":"2023-06-10T18:45:42.443581+0000","last_peered":"2023-06-10T18:45:42.443581+0000","last_clean":"2023-06-10T18:45:42.443581+0000","last_became_active":"2023-06-10T18:45:40.905809+0000","last_became_peered":"2023-06-10T18:45:40.905809+0000","last_unstale":"2023-06-10T18:45:42.443581+0000","last_undegraded":"2023-06-10T18:45:42.443581+0000","last_fullsized":"2023-06-10T18:45:42.443581+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:58:32.518563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229894+0000","last_change":"2023-06-10T18:45:41.454805+0000","last_active":"2023-06-10T18:45:43.229894+0000","last_peered":"2023-06-10T18:45:43.229894+0000","last_clean":"2023-06-10T18:45:43.229894+0000","last_became_active":"2023-06-10T18:45:41.454550+0000","last_became_peered":"2023-06-10T18:45:41.454550+0000","last_unstale":"2023-06-10T18:45:43.229894+0000","last_undegraded":"2023-06-10T18:45:43.229894+0000","last_fullsized":"2023-06-10T18:45:43.229894+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:34.241599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229446+0000","last_change":"2023-06-10T18:45:40.903305+0000","last_active":"2023-06-10T18:45:43.229446+0000","last_peered":"2023-06-10T18:45:43.229446+0000","last_clean":"2023-06-10T18:45:43.229446+0000","last_became_active":"2023-06-10T18:45:40.903030+0000","last_became_peered":"2023-06-10T18:45:40.903030+0000","last_unstale":"2023-06-10T18:45:43.229446+0000","last_undegraded":"2023-06-10T18:45:43.229446+0000","last_fullsized":"2023-06-10T18:45:43.229446+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:51:04.993117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443083+0000","last_change":"2023-06-10T18:45:41.459249+0000","last_active":"2023-06-10T18:45:42.443083+0000","last_peered":"2023-06-10T18:45:42.443083+0000","last_clean":"2023-06-10T18:45:42.443083+0000","last_became_active":"2023-06-10T18:45:41.458999+0000","last_became_peered":"2023-06-10T18:45:41.458999+0000","last_unstale":"2023-06-10T18:45:42.443083+0000","last_undegraded":"2023-06-10T18:45:42.443083+0000","last_fullsized":"2023-06-10T18:45:42.443083+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:36.168114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443182+0000","last_change":"2023-06-10T18:45:40.906223+0000","last_active":"2023-06-10T18:45:42.443182+0000","last_peered":"2023-06-10T18:45:42.443182+0000","last_clean":"2023-06-10T18:45:42.443182+0000","last_became_active":"2023-06-10T18:45:40.905829+0000","last_became_peered":"2023-06-10T18:45:40.905829+0000","last_unstale":"2023-06-10T18:45:42.443182+0000","last_undegraded":"2023-06-10T18:45:42.443182+0000","last_fullsized":"2023-06-10T18:45:42.443182+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:59:16.765654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229393+0000","last_change":"2023-06-10T18:45:41.462810+0000","last_active":"2023-06-10T18:45:43.229393+0000","last_peered":"2023-06-10T18:45:43.229393+0000","last_clean":"2023-06-10T18:45:43.229393+0000","last_became_active":"2023-06-10T18:45:41.462556+0000","last_became_peered":"2023-06-10T18:45:41.462556+0000","last_unstale":"2023-06-10T18:45:43.229393+0000","last_undegraded":"2023-06-10T18:45:43.229393+0000","last_fullsized":"2023-06-10T18:45:43.229393+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:55.334409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":21,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229598+0000","last_change":"2023-06-10T18:45:40.904337+0000","last_active":"2023-06-10T18:45:43.229598+0000","last_peered":"2023-06-10T18:45:43.229598+0000","last_clean":"2023-06-10T18:45:43.229598+0000","last_became_active":"2023-06-10T18:45:40.904087+0000","last_became_peered":"2023-06-10T18:45:40.904087+0000","last_unstale":"2023-06-10T18:45:43.229598+0000","last_undegraded":"2023-06-10T18:45:43.229598+0000","last_fullsized":"2023-06-10T18:45:43.229598+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:23.832736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230627+0000","last_change":"2023-06-10T18:45:41.459111+0000","last_active":"2023-06-10T18:45:43.230627+0000","last_peered":"2023-06-10T18:45:43.230627+0000","last_clean":"2023-06-10T18:45:43.230627+0000","last_became_active":"2023-06-10T18:45:41.458830+0000","last_became_peered":"2023-06-10T18:45:41.458830+0000","last_unstale":"2023-06-10T18:45:43.230627+0000","last_undegraded":"2023-06-10T18:45:43.230627+0000","last_fullsized":"2023-06-10T18:45:43.230627+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:59:13.782139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229854+0000","last_change":"2023-06-10T18:45:40.892521+0000","last_active":"2023-06-10T18:45:43.229854+0000","last_peered":"2023-06-10T18:45:43.229854+0000","last_clean":"2023-06-10T18:45:43.229854+0000","last_became_active":"2023-06-10T18:45:40.892250+0000","last_became_peered":"2023-06-10T18:45:40.892250+0000","last_unstale":"2023-06-10T18:45:43.229854+0000","last_undegraded":"2023-06-10T18:45:43.229854+0000","last_fullsized":"2023-06-10T18:45:43.229854+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:52:25.082108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230410+0000","last_change":"2023-06-10T18:45:41.452576+0000","last_active":"2023-06-10T18:45:43.230410+0000","last_peered":"2023-06-10T18:45:43.230410+0000","last_clean":"2023-06-10T18:45:43.230410+0000","last_became_active":"2023-06-10T18:45:41.452326+0000","last_became_peered":"2023-06-10T18:45:41.452326+0000","last_unstale":"2023-06-10T18:45:43.230410+0000","last_undegraded":"2023-06-10T18:45:43.230410+0000","last_fullsized":"2023-06-10T18:45:43.230410+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:34.427894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446649+0000","last_change":"2023-06-10T18:45:40.909132+0000","last_active":"2023-06-10T18:45:42.446649+0000","last_peered":"2023-06-10T18:45:42.446649+0000","last_clean":"2023-06-10T18:45:42.446649+0000","last_became_active":"2023-06-10T18:45:40.908770+0000","last_became_peered":"2023-06-10T18:45:40.908770+0000","last_unstale":"2023-06-10T18:45:42.446649+0000","last_undegraded":"2023-06-10T18:45:42.446649+0000","last_fullsized":"2023-06-10T18:45:42.446649+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:03.313110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481133+0000","last_change":"2023-06-10T18:45:41.466082+0000","last_active":"2023-06-10T18:45:42.481133+0000","last_peered":"2023-06-10T18:45:42.481133+0000","last_clean":"2023-06-10T18:45:42.481133+0000","last_became_active":"2023-06-10T18:45:41.465825+0000","last_became_peered":"2023-06-10T18:45:41.465825+0000","last_unstale":"2023-06-10T18:45:42.481133+0000","last_undegraded":"2023-06-10T18:45:42.481133+0000","last_fullsized":"2023-06-10T18:45:42.481133+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:00:48.820744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.447234+0000","last_change":"2023-06-10T18:45:40.909792+0000","last_active":"2023-06-10T18:45:42.447234+0000","last_peered":"2023-06-10T18:45:42.447234+0000","last_clean":"2023-06-10T18:45:42.447234+0000","last_became_active":"2023-06-10T18:45:40.909529+0000","last_became_peered":"2023-06-10T18:45:40.909529+0000","last_unstale":"2023-06-10T18:45:42.447234+0000","last_undegraded":"2023-06-10T18:45:42.447234+0000","last_fullsized":"2023-06-10T18:45:42.447234+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:00:53.053538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481480+0000","last_change":"2023-06-10T18:45:41.466215+0000","last_active":"2023-06-10T18:45:42.481480+0000","last_peered":"2023-06-10T18:45:42.481480+0000","last_clean":"2023-06-10T18:45:42.481480+0000","last_became_active":"2023-06-10T18:45:41.465615+0000","last_became_peered":"2023-06-10T18:45:41.465615+0000","last_unstale":"2023-06-10T18:45:42.481480+0000","last_undegraded":"2023-06-10T18:45:42.481480+0000","last_fullsized":"2023-06-10T18:45:42.481480+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:37:56.870973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443489+0000","last_change":"2023-06-10T18:45:40.904862+0000","last_active":"2023-06-10T18:45:42.443489+0000","last_peered":"2023-06-10T18:45:42.443489+0000","last_clean":"2023-06-10T18:45:42.443489+0000","last_became_active":"2023-06-10T18:45:40.904590+0000","last_became_peered":"2023-06-10T18:45:40.904590+0000","last_unstale":"2023-06-10T18:45:42.443489+0000","last_undegraded":"2023-06-10T18:45:42.443489+0000","last_fullsized":"2023-06-10T18:45:42.443489+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:24.524984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443232+0000","last_change":"2023-06-10T18:45:41.459390+0000","last_active":"2023-06-10T18:45:42.443232+0000","last_peered":"2023-06-10T18:45:42.443232+0000","last_clean":"2023-06-10T18:45:42.443232+0000","last_became_active":"2023-06-10T18:45:41.459112+0000","last_became_peered":"2023-06-10T18:45:41.459112+0000","last_unstale":"2023-06-10T18:45:42.443232+0000","last_undegraded":"2023-06-10T18:45:42.443232+0000","last_fullsized":"2023-06-10T18:45:42.443232+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:41.151941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443643+0000","last_change":"2023-06-10T18:45:40.900179+0000","last_active":"2023-06-10T18:45:42.443643+0000","last_peered":"2023-06-10T18:45:42.443643+0000","last_clean":"2023-06-10T18:45:42.443643+0000","last_became_active":"2023-06-10T18:45:40.899762+0000","last_became_peered":"2023-06-10T18:45:40.899762+0000","last_unstale":"2023-06-10T18:45:42.443643+0000","last_undegraded":"2023-06-10T18:45:42.443643+0000","last_fullsized":"2023-06-10T18:45:42.443643+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:27:03.233438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230049+0000","last_change":"2023-06-10T18:45:41.459062+0000","last_active":"2023-06-10T18:45:43.230049+0000","last_peered":"2023-06-10T18:45:43.230049+0000","last_clean":"2023-06-10T18:45:43.230049+0000","last_became_active":"2023-06-10T18:45:41.458788+0000","last_became_peered":"2023-06-10T18:45:41.458788+0000","last_unstale":"2023-06-10T18:45:43.230049+0000","last_undegraded":"2023-06-10T18:45:43.230049+0000","last_fullsized":"2023-06-10T18:45:43.230049+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:46:05.411983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446102+0000","last_change":"2023-06-10T18:45:40.904961+0000","last_active":"2023-06-10T18:45:42.446102+0000","last_peered":"2023-06-10T18:45:42.446102+0000","last_clean":"2023-06-10T18:45:42.446102+0000","last_became_active":"2023-06-10T18:45:40.904660+0000","last_became_peered":"2023-06-10T18:45:40.904660+0000","last_unstale":"2023-06-10T18:45:42.446102+0000","last_undegraded":"2023-06-10T18:45:42.446102+0000","last_fullsized":"2023-06-10T18:45:42.446102+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:58.353386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229182+0000","last_change":"2023-06-10T18:45:41.463988+0000","last_active":"2023-06-10T18:45:43.229182+0000","last_peered":"2023-06-10T18:45:43.229182+0000","last_clean":"2023-06-10T18:45:43.229182+0000","last_became_active":"2023-06-10T18:45:41.463168+0000","last_became_peered":"2023-06-10T18:45:41.463168+0000","last_unstale":"2023-06-10T18:45:43.229182+0000","last_undegraded":"2023-06-10T18:45:43.229182+0000","last_fullsized":"2023-06-10T18:45:43.229182+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:34:45.337982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229826+0000","last_change":"2023-06-10T18:45:40.905644+0000","last_active":"2023-06-10T18:45:43.229826+0000","last_peered":"2023-06-10T18:45:43.229826+0000","last_clean":"2023-06-10T18:45:43.229826+0000","last_became_active":"2023-06-10T18:45:40.905375+0000","last_became_peered":"2023-06-10T18:45:40.905375+0000","last_unstale":"2023-06-10T18:45:43.229826+0000","last_undegraded":"2023-06-10T18:45:43.229826+0000","last_fullsized":"2023-06-10T18:45:43.229826+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229478+0000","last_change":"2023-06-10T18:45:41.467059+0000","last_active":"2023-06-10T18:45:43.229478+0000","last_peered":"2023-06-10T18:45:43.229478+0000","last_clean":"2023-06-10T18:45:43.229478+0000","last_became_active":"2023-06-10T18:45:41.466802+0000","last_became_peered":"2023-06-10T18:45:41.466802+0000","last_unstale":"2023-06-10T18:45:43.229478+0000","last_undegraded":"2023-06-10T18:45:43.229478+0000","last_fullsized":"2023-06-10T18:45:43.229478+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:53:48.809753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229136+0000","last_change":"2023-06-10T18:45:40.903928+0000","last_active":"2023-06-10T18:45:43.229136+0000","last_peered":"2023-06-10T18:45:43.229136+0000","last_clean":"2023-06-10T18:45:43.229136+0000","last_became_active":"2023-06-10T18:45:40.903597+0000","last_became_peered":"2023-06-10T18:45:40.903597+0000","last_unstale":"2023-06-10T18:45:43.229136+0000","last_undegraded":"2023-06-10T18:45:43.229136+0000","last_fullsized":"2023-06-10T18:45:43.229136+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:53.130998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.445883+0000","last_change":"2023-06-10T18:45:41.463907+0000","last_active":"2023-06-10T18:45:42.445883+0000","last_peered":"2023-06-10T18:45:42.445883+0000","last_clean":"2023-06-10T18:45:42.445883+0000","last_became_active":"2023-06-10T18:45:41.463635+0000","last_became_peered":"2023-06-10T18:45:41.463635+0000","last_unstale":"2023-06-10T18:45:42.445883+0000","last_undegraded":"2023-06-10T18:45:42.445883+0000","last_fullsized":"2023-06-10T18:45:42.445883+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:43:27.158196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229667+0000","last_change":"2023-06-10T18:45:40.907369+0000","last_active":"2023-06-10T18:45:43.229667+0000","last_peered":"2023-06-10T18:45:43.229667+0000","last_clean":"2023-06-10T18:45:43.229667+0000","last_became_active":"2023-06-10T18:45:40.907099+0000","last_became_peered":"2023-06-10T18:45:40.907099+0000","last_unstale":"2023-06-10T18:45:43.229667+0000","last_undegraded":"2023-06-10T18:45:43.229667+0000","last_fullsized":"2023-06-10T18:45:43.229667+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:58.398924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230309+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:45:43.230309+0000","last_peered":"2023-06-10T18:45:43.230309+0000","last_clean":"2023-06-10T18:45:43.230309+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:45:43.230309+0000","last_undegraded":"2023-06-10T18:45:43.230309+0000","last_fullsized":"2023-06-10T18:45:43.230309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:22.763380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481401+0000","last_change":"2023-06-10T18:45:41.463072+0000","last_active":"2023-06-10T18:45:42.481401+0000","last_peered":"2023-06-10T18:45:42.481401+0000","last_clean":"2023-06-10T18:45:42.481401+0000","last_became_active":"2023-06-10T18:45:41.462701+0000","last_became_peered":"2023-06-10T18:45:41.462701+0000","last_unstale":"2023-06-10T18:45:42.481401+0000","last_undegraded":"2023-06-10T18:45:42.481401+0000","last_fullsized":"2023-06-10T18:45:42.481401+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:04:41.662670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"42'1","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443429+0000","last_change":"2023-06-10T18:45:40.900220+0000","last_active":"2023-06-10T18:45:42.443429+0000","last_peered":"2023-06-10T18:45:42.443429+0000","last_clean":"2023-06-10T18:45:42.443429+0000","last_became_active":"2023-06-10T18:45:40.899948+0000","last_became_peered":"2023-06-10T18:45:40.899948+0000","last_unstale":"2023-06-10T18:45:42.443429+0000","last_undegraded":"2023-06-10T18:45:42.443429+0000","last_fullsized":"2023-06-10T18:45:42.443429+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:56.912879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446169+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:45:42.446169+0000","last_peered":"2023-06-10T18:45:42.446169+0000","last_clean":"2023-06-10T18:45:42.446169+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:45:42.446169+0000","last_undegraded":"2023-06-10T18:45:42.446169+0000","last_fullsized":"2023-06-10T18:45:42.446169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:49:27.638486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230436+0000","last_change":"2023-06-10T18:45:41.465571+0000","last_active":"2023-06-10T18:45:43.230436+0000","last_peered":"2023-06-10T18:45:43.230436+0000","last_clean":"2023-06-10T18:45:43.230436+0000","last_became_active":"2023-06-10T18:45:41.465336+0000","last_became_peered":"2023-06-10T18:45:41.465336+0000","last_unstale":"2023-06-10T18:45:43.230436+0000","last_undegraded":"2023-06-10T18:45:43.230436+0000","last_fullsized":"2023-06-10T18:45:43.230436+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:36.168114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229800+0000","last_change":"2023-06-10T18:45:40.909238+0000","last_active":"2023-06-10T18:45:43.229800+0000","last_peered":"2023-06-10T18:45:43.229800+0000","last_clean":"2023-06-10T18:45:43.229800+0000","last_became_active":"2023-06-10T18:45:40.908968+0000","last_became_peered":"2023-06-10T18:45:40.908968+0000","last_unstale":"2023-06-10T18:45:43.229800+0000","last_undegraded":"2023-06-10T18:45:43.229800+0000","last_fullsized":"2023-06-10T18:45:43.229800+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:42:51.416142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229302+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:45:43.229302+0000","last_peered":"2023-06-10T18:45:43.229302+0000","last_clean":"2023-06-10T18:45:43.229302+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:45:43.229302+0000","last_undegraded":"2023-06-10T18:45:43.229302+0000","last_fullsized":"2023-06-10T18:45:43.229302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:52:10.467509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230358+0000","last_change":"2023-06-10T18:45:41.454735+0000","last_active":"2023-06-10T18:45:43.230358+0000","last_peered":"2023-06-10T18:45:43.230358+0000","last_clean":"2023-06-10T18:45:43.230358+0000","last_became_active":"2023-06-10T18:45:41.454482+0000","last_became_peered":"2023-06-10T18:45:41.454482+0000","last_unstale":"2023-06-10T18:45:43.230358+0000","last_undegraded":"2023-06-10T18:45:43.230358+0000","last_fullsized":"2023-06-10T18:45:43.230358+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:49:07.380896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":15,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.447424+0000","last_change":"2023-06-10T18:45:40.904255+0000","last_active":"2023-06-10T18:45:42.447424+0000","last_peered":"2023-06-10T18:45:42.447424+0000","last_clean":"2023-06-10T18:45:42.447424+0000","last_became_active":"2023-06-10T18:45:40.903971+0000","last_became_peered":"2023-06-10T18:45:40.903971+0000","last_unstale":"2023-06-10T18:45:42.447424+0000","last_undegraded":"2023-06-10T18:45:42.447424+0000","last_fullsized":"2023-06-10T18:45:42.447424+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:33:44.962685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":57,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446219+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:45:42.446219+0000","last_peered":"2023-06-10T18:45:42.446219+0000","last_clean":"2023-06-10T18:45:42.446219+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:45:42.446219+0000","last_undegraded":"2023-06-10T18:45:42.446219+0000","last_fullsized":"2023-06-10T18:45:42.446219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:29:49.880024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":211,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229294+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:45:43.229294+0000","last_peered":"2023-06-10T18:45:43.229294+0000","last_clean":"2023-06-10T18:45:43.229294+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:45:43.229294+0000","last_undegraded":"2023-06-10T18:45:43.229294+0000","last_fullsized":"2023-06-10T18:45:43.229294+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:54:35.236468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229567+0000","last_change":"2023-06-10T18:45:41.460083+0000","last_active":"2023-06-10T18:45:43.229567+0000","last_peered":"2023-06-10T18:45:43.229567+0000","last_clean":"2023-06-10T18:45:43.229567+0000","last_became_active":"2023-06-10T18:45:41.459815+0000","last_became_peered":"2023-06-10T18:45:41.459815+0000","last_unstale":"2023-06-10T18:45:43.229567+0000","last_undegraded":"2023-06-10T18:45:43.229567+0000","last_fullsized":"2023-06-10T18:45:43.229567+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:31:28.241328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230189+0000","last_change":"2023-06-10T18:45:40.908782+0000","last_active":"2023-06-10T18:45:43.230189+0000","last_peered":"2023-06-10T18:45:43.230189+0000","last_clean":"2023-06-10T18:45:43.230189+0000","last_became_active":"2023-06-10T18:45:40.908452+0000","last_became_peered":"2023-06-10T18:45:40.908452+0000","last_unstale":"2023-06-10T18:45:43.230189+0000","last_undegraded":"2023-06-10T18:45:43.230189+0000","last_fullsized":"2023-06-10T18:45:43.230189+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:39:53.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229341+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:45:43.229341+0000","last_peered":"2023-06-10T18:45:43.229341+0000","last_clean":"2023-06-10T18:45:43.229341+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:45:43.229341+0000","last_undegraded":"2023-06-10T18:45:43.229341+0000","last_fullsized":"2023-06-10T18:45:43.229341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:17:31.159099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446215+0000","last_change":"2023-06-10T18:45:41.460028+0000","last_active":"2023-06-10T18:45:42.446215+0000","last_peered":"2023-06-10T18:45:42.446215+0000","last_clean":"2023-06-10T18:45:42.446215+0000","last_became_active":"2023-06-10T18:45:41.459759+0000","last_became_peered":"2023-06-10T18:45:41.459759+0000","last_unstale":"2023-06-10T18:45:42.446215+0000","last_undegraded":"2023-06-10T18:45:42.446215+0000","last_fullsized":"2023-06-10T18:45:42.446215+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:03:13.858675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229554+0000","last_change":"2023-06-10T18:45:40.908831+0000","last_active":"2023-06-10T18:45:43.229554+0000","last_peered":"2023-06-10T18:45:43.229554+0000","last_clean":"2023-06-10T18:45:43.229554+0000","last_became_active":"2023-06-10T18:45:40.908610+0000","last_became_peered":"2023-06-10T18:45:40.908610+0000","last_unstale":"2023-06-10T18:45:43.229554+0000","last_undegraded":"2023-06-10T18:45:43.229554+0000","last_fullsized":"2023-06-10T18:45:43.229554+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:36.354025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229825+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:45:43.229825+0000","last_peered":"2023-06-10T18:45:43.229825+0000","last_clean":"2023-06-10T18:45:43.229825+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:45:43.229825+0000","last_undegraded":"2023-06-10T18:45:43.229825+0000","last_fullsized":"2023-06-10T18:45:43.229825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:32:07.608054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443516+0000","last_change":"2023-06-10T18:45:41.459476+0000","last_active":"2023-06-10T18:45:42.443516+0000","last_peered":"2023-06-10T18:45:42.443516+0000","last_clean":"2023-06-10T18:45:42.443516+0000","last_became_active":"2023-06-10T18:45:41.459042+0000","last_became_peered":"2023-06-10T18:45:41.459042+0000","last_unstale":"2023-06-10T18:45:42.443516+0000","last_undegraded":"2023-06-10T18:45:42.443516+0000","last_fullsized":"2023-06-10T18:45:42.443516+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:54:46.440393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.446340+0000","last_change":"2023-06-10T18:45:40.903579+0000","last_active":"2023-06-10T18:45:42.446340+0000","last_peered":"2023-06-10T18:45:42.446340+0000","last_clean":"2023-06-10T18:45:42.446340+0000","last_became_active":"2023-06-10T18:45:40.903381+0000","last_became_peered":"2023-06-10T18:45:40.903381+0000","last_unstale":"2023-06-10T18:45:42.446340+0000","last_undegraded":"2023-06-10T18:45:42.446340+0000","last_fullsized":"2023-06-10T18:45:42.446340+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:05:48.340301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.230452+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:45:43.230452+0000","last_peered":"2023-06-10T18:45:43.230452+0000","last_clean":"2023-06-10T18:45:43.230452+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:45:43.230452+0000","last_undegraded":"2023-06-10T18:45:43.230452+0000","last_fullsized":"2023-06-10T18:45:43.230452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:47:29.019593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.443463+0000","last_change":"2023-06-10T18:45:41.457549+0000","last_active":"2023-06-10T18:45:42.443463+0000","last_peered":"2023-06-10T18:45:42.443463+0000","last_clean":"2023-06-10T18:45:42.443463+0000","last_became_active":"2023-06-10T18:45:41.457299+0000","last_became_peered":"2023-06-10T18:45:41.457299+0000","last_unstale":"2023-06-10T18:45:42.443463+0000","last_undegraded":"2023-06-10T18:45:42.443463+0000","last_fullsized":"2023-06-10T18:45:42.443463+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:40.881331+0000","last_clean_scrub_stamp":"2023-06-10T18:45:40.881331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:42.523524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":13,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:43.229565+0000","last_change":"2023-06-10T18:45:40.908946+0000","last_active":"2023-06-10T18:45:43.229565+0000","last_peered":"2023-06-10T18:45:43.229565+0000","last_clean":"2023-06-10T18:45:43.229565+0000","last_became_active":"2023-06-10T18:45:40.908651+0000","last_became_peered":"2023-06-10T18:45:40.908651+0000","last_unstale":"2023-06-10T18:45:43.229565+0000","last_undegraded":"2023-06-10T18:45:43.229565+0000","last_fullsized":"2023-06-10T18:45:43.229565+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:45:39.874658+0000","last_clean_scrub_stamp":"2023-06-10T18:45:39.874658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:58.353386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-06-10T18:45:42.481595+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:45:42.481595+0000","last_peered":"2023-06-10T18:45:42.481595+0000","last_clean":"2023-06-10T18:45:42.481595+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:45:42.481595+0000","last_undegraded":"2023-06-10T18:45:42.481595+0000","last_fullsized":"2023-06-10T18:45:42.481595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:04.314871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1085440,"data_stored":809234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.436},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.496},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":491520,"data_stored":218429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.504},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":9.6850000000000005,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.72699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.71299999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":487424,"data_stored":217827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.22},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.254}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.28899999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.621}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":1056,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1081344,"data_stored":808159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.29299999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.314},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.33400000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":503808,"data_stored":220362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.499,"15min":0.499},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":499712,"data_stored":219414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.437},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.48199999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.499},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.802,"5min":3.802,"15min":3.802},"last":0.76700000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78300000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.78900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":495616,"data_stored":219321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.69899999999999995}]},{"osd":1,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.59999999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.41799999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.442},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":491520,"data_stored":218846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.61099999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.439},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.52600000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:45:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:45:50.565 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:45:50.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:45:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.717+0000 7ff6fa9e0700 1 Processor -- start 2023-06-10T18:45:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.717+0000 7ff6fa9e0700 1 -- start start 2023-06-10T18:45:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6fa9e0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6f406e6e0 0x7ff6f40b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6fa9e0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6f40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6fa9e0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f406e250 con 0x7ff6f415c580 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6fa9e0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f415c950 con 0x7ff6f406e6e0 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6fa9e0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b1e80 con 0x7ff6f406cde0 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff6f415c580 0x7ff6f40b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55292/0 (socket says 172.21.15.104:55292) 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f3fff700 1 -- 172.21.15.104:0/1829165084 learned_addr learned my addr 172.21.15.104:0/1829165084 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/1829165084 >> v2:172.21.15.114:3300/0 conn(0x7ff6f406e6e0 0x7ff6f40b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f8f7d700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6f40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015961756 0 0) 0x7ff6f406e250 con 0x7ff6f415c580 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.718+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e00035b0 con 0x7ff6f415c580 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180536516 0 0) 0x7ff6e00035b0 con 0x7ff6f415c580 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 msgr2=0x7ff6f40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6f40ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 >> v2:172.21.15.114:3300/0 conn(0x7ff6f406e6e0 msgr2=0x7ff6f40b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 --2- 172.21.15.104:0/1829165084 >> v2:172.21.15.114:3300/0 conn(0x7ff6f406e6e0 0x7ff6f40b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f406d920 con 0x7ff6f415c580 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff6dc0017f0 con 0x7ff6f415c580 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 >> v1:172.21.15.104:6789/0 conn(0x7ff6f415c580 legacy=0x7ff6f40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6e0004430 0x7ff6e00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6e0004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f415c950 con 0x7ff6e0003a00 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b1e80 con 0x7ff6e0004430 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e000ba10 con 0x7ff6f406cde0 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/1829165084 >> v2:172.21.15.114:3300/0 conn(0x7ff6f406e6e0 0x7ff6f40b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f8f7d700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6e0004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6e0004430 0x7ff6e00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 -- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 msgr2=0x7ff6e0004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6e0004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 -- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6e0004430 msgr2=0x7ff6e00048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6e0004430 0x7ff6e00048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 -- 172.21.15.104:0/1829165084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e00035b0 con 0x7ff6e0003a00 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6e0004430 0x7ff6e00048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f8f7d700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f406cde0 0x7ff6e0004fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6e0003e10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00c320 tx=0x7ff6e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550873c0e7ae969c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00ce60 con 0x7ff6e0003a00 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc007bf0 con 0x7ff6e0003a00 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.719+0000 7ff6f2ffd700 1 -- 172.21.15.104:0/1829165084 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0072f0 con 0x7ff6e0003a00 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/1829165084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 msgr2=0x7ff6e0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/1829165084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6e0003e10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00c320 tx=0x7ff6e0005880 comp rx=0 tx=0).stop 2023-06-10T18:45:50.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6f8f7d700 1 -- 172.21.15.104:0/1829165084 reap_dead start 2023-06-10T18:45:50.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/1829165084 shutdown_connections 2023-06-10T18:45:50.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/1829165084 >> 172.21.15.104:0/1829165084 conn(0x7ff6f406b8a0 msgr2=0x7ff6f40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:50.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/1829165084 shutdown_connections 2023-06-10T18:45:50.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.720+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/1829165084 wait complete. 2023-06-10T18:45:50.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.721+0000 7ff6fa9e0700 1 Processor -- start 2023-06-10T18:45:50.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.721+0000 7ff6fa9e0700 1 -- start start 2023-06-10T18:45:50.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f41e17a0 0x7ff6f41e30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 0x7ff6f40afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40c3e80 con 0x7ff6e0003a00 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41e1c60 con 0x7ff6f41e3600 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6fa9e0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41e1f40 con 0x7ff6f41e17a0 2023-06-10T18:45:50.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57476/0 (socket says 172.21.15.104:57476) 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 -- 172.21.15.104:0/664957985 learned_addr learned my addr 172.21.15.104:0/664957985 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f8f7d700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 0x7ff6f40afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f41e17a0 0x7ff6f41e30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 -- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f41e17a0 msgr2=0x7ff6f41e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f41e17a0 0x7ff6f41e30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 -- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 msgr2=0x7ff6f40afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 0x7ff6f40afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 -- 172.21.15.104:0/664957985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f41e2220 con 0x7ff6e0003a00 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f8f7d700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 0x7ff6f40afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.722+0000 7ff6f3fff700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc015040 tx=0x7ff6dc001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.723+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0072d0 con 0x7ff6e0003a00 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.723+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc007430 con 0x7ff6e0003a00 2023-06-10T18:45:50.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.723+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f41e2500 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.723+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0077e0 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.723+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f40b2420 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.724+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f4074050 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.724+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff6dc030430 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.725+0000 7ff6f0ff9700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 0x7ff6d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.725+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff6dc08aef0 con 0x7ff6e0003a00 2023-06-10T18:45:50.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.725+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 0x7ff6d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:50.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.726+0000 7ff6f37fe700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 0x7ff6d406d190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff6f41e1b90 tx=0x7ff6e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:50.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.729+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff6dc055f10 con 0x7ff6e0003a00 2023-06-10T18:45:50.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.991+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff6f4066f10 con 0x7ff6e0003a00 2023-06-10T18:45:51.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.995+0000 7ff6f0ff9700 1 -- 172.21.15.104:0/664957985 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff6dc000f70 con 0x7ff6e0003a00 2023-06-10T18:45:51.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 msgr2=0x7ff6d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 0x7ff6d406d190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff6f41e1b90 tx=0x7ff6e4008040 comp rx=0 tx=0).stop 2023-06-10T18:45:51.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 msgr2=0x7ff6f40b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc015040 tx=0x7ff6dc001bf0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 shutdown_connections 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff6d406ace0 0x7ff6d406d190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f41e3600 0x7ff6f40afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6e0003a00 0x7ff6f40b4830 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 --2- 172.21.15.104:0/664957985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6f41e17a0 0x7ff6f41e30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 >> 172.21.15.104:0/664957985 conn(0x7ff6f406b8a0 msgr2=0x7ff6f40cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 shutdown_connections 2023-06-10T18:45:51.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:50.997+0000 7ff6fa9e0700 1 -- 172.21.15.104:0/664957985 wait complete. 2023-06-10T18:45:51.005 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:45:51.015 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:51.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 Processor -- start 2023-06-10T18:45:51.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 -- start start 2023-06-10T18:45:51.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b9006e0e0 0x7f6b9006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b90069e70 0x7f6b900ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9006d290 con 0x7f6b9006e4b0 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b90071640 con 0x7f6b90069e70 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.129+0000 7f6b9611a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900aef00 con 0x7f6b9006e0e0 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b90069e70 0x7f6b900ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b90069e70 0x7f6b900ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57488/0 (socket says 172.21.15.104:57488) 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 -- 172.21.15.104:0/329711455 learned_addr learned my addr 172.21.15.104:0/329711455 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8f7fe700 1 --2- 172.21.15.104:0/329711455 >> v2:172.21.15.114:3300/0 conn(0x7f6b9006e0e0 0x7f6b9006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 -- 172.21.15.104:0/329711455 >> v2:172.21.15.114:3300/0 conn(0x7f6b9006e0e0 msgr2=0x7f6b9006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 --2- 172.21.15.104:0/329711455 >> v2:172.21.15.114:3300/0 conn(0x7f6b9006e0e0 0x7f6b9006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 -- 172.21.15.104:0/329711455 >> v1:172.21.15.114:6789/0 conn(0x7f6b9006e4b0 legacy=0x7f6b900c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 -- 172.21.15.104:0/329711455 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900ca2d0 con 0x7f6b90069e70 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8ffff700 1 --2- 172.21.15.104:0/329711455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b90069e70 0x7f6b900ce9f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6b9006af50 tx=0x7f6b8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85631e4c4729fa55 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8e7fc700 1 -- 172.21.15.104:0/329711455 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8800e070 con 0x7f6b90069e70 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8e7fc700 1 -- 172.21.15.104:0/329711455 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b88003a20 con 0x7f6b90069e70 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.130+0000 7f6b8e7fc700 1 -- 172.21.15.104:0/329711455 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88003dd0 con 0x7f6b90069e70 2023-06-10T18:45:51.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 -- 172.21.15.104:0/329711455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b90069e70 msgr2=0x7f6b900ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 --2- 172.21.15.104:0/329711455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b90069e70 0x7f6b900ce9f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6b9006af50 tx=0x7f6b8800b330 comp rx=0 tx=0).stop 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 -- 172.21.15.104:0/329711455 shutdown_connections 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 --2- 172.21.15.104:0/329711455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b90069e70 0x7f6b900ce9f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 --2- 172.21.15.104:0/329711455 >> v2:172.21.15.114:3300/0 conn(0x7f6b9006e0e0 0x7f6b9006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 -- 172.21.15.104:0/329711455 >> 172.21.15.104:0/329711455 conn(0x7f6b9006b8a0 msgr2=0x7f6b900cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 -- 172.21.15.104:0/329711455 shutdown_connections 2023-06-10T18:45:51.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.131+0000 7f6b9611a700 1 -- 172.21.15.104:0/329711455 wait complete. 2023-06-10T18:45:51.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.132+0000 7f6b9611a700 1 Processor -- start 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.132+0000 7f6b9611a700 1 -- start start 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 0x7f6b900b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9006e0e0 0x7f6b9014fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901500e0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90156040 con 0x7f6b9006e0e0 2023-06-10T18:45:51.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b9611a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900b1e80 con 0x7f6b90069e70 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.133+0000 7f6b8ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57494/0 (socket says 172.21.15.104:57494) 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 -- 172.21.15.104:0/1354235358 learned_addr learned my addr 172.21.15.104:0/1354235358 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8effd700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9006e0e0 0x7f6b9014fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8f7fe700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 0x7f6b900b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 -- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 msgr2=0x7f6b900b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 0x7f6b900b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 -- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9006e0e0 msgr2=0x7f6b9014fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9006e0e0 0x7f6b9014fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 -- 172.21.15.104:0/1354235358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88009b30 con 0x7f6b9006e4b0 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8f7fe700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 0x7f6b900b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b8ffff700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6b88015040 tx=0x7f6b88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88004440 con 0x7f6b9006e4b0 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b880045a0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88004950 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b900b24d0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.134+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b900b2a90 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.135+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b90074050 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.136+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b88012430 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.136+0000 7f6b77fff700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 0x7f6b7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.136+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f6b8808aa60 con 0x7f6b9006e4b0 2023-06-10T18:45:51.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.136+0000 7f6b8f7fe700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 0x7f6b7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.137+0000 7f6b8f7fe700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 0x7f6b7c06d190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6b84006fd0 tx=0x7f6b84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.141+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6b880559a0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.367+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b900723b0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.368+0000 7f6b77fff700 1 -- 172.21.15.104:0/1354235358 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f6b88058fc0 con 0x7f6b9006e4b0 2023-06-10T18:45:51.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:51.374 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 msgr2=0x7f6b7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 0x7f6b7c06d190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6b84006fd0 tx=0x7f6b84008040 comp rx=0 tx=0).stop 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 msgr2=0x7f6b9015a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6b88015040 tx=0x7f6b88004ad0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 shutdown_connections 2023-06-10T18:45:51.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b9006e4b0 0x7f6b9015a3f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b90069e70 0x7f6b900b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b7c06ace0 0x7f6b7c06d190 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 --2- 172.21.15.104:0/1354235358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9006e0e0 0x7f6b9014fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 >> 172.21.15.104:0/1354235358 conn(0x7f6b9006b8a0 msgr2=0x7f6b900adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 shutdown_connections 2023-06-10T18:45:51.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.370+0000 7f6b9611a700 1 -- 172.21.15.104:0/1354235358 wait complete. 2023-06-10T18:45:51.378 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:45:51.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:45:41.438525+0000', 'modified': '2023-06-10T18:45:44.530451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5622}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5622': {'gid': 5622, 'name': 'b', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/698196467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 698196467}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 698196467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:45:51.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:45:51.792 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-06-10T18:45:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.909+0000 7f06d2fa5700 1 Processor -- start 2023-06-10T18:45:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.909+0000 7f06d2fa5700 1 -- start start 2023-06-10T18:45:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d2fa5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f06cc0b4ff0 0x7f06cc0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d2fa5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f06cc0ce890 0x7f06cc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d2fa5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0b6070 con 0x7f06cc15a420 2023-06-10T18:45:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d2fa5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06cc0b1d70 con 0x7f06cc0ce890 2023-06-10T18:45:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d2fa5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06cc0cecb0 con 0x7f06cc0b4ff0 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f06cc0ce890 0x7f06cc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f06cc0ce890 0x7f06cc15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57502/0 (socket says 172.21.15.104:57502) 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 -- 172.21.15.104:0/1540479438 learned_addr learned my addr 172.21.15.104:0/1540479438 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 -- 172.21.15.104:0/1540479438 >> v2:172.21.15.114:3300/0 conn(0x7f06cc0b4ff0 msgr2=0x7f06cc0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 --2- 172.21.15.104:0/1540479438 >> v2:172.21.15.114:3300/0 conn(0x7f06cc0b4ff0 0x7f06cc0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 -- 172.21.15.104:0/1540479438 >> v1:172.21.15.114:6789/0 conn(0x7f06cc15a420 legacy=0x7f06cc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 -- 172.21.15.104:0/1540479438 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0b5460 con 0x7f06cc0ce890 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06d1542700 1 --2- 172.21.15.104:0/1540479438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc15a7f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06cc066590 tx=0x7f06c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcb645f452a518f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06cb7fe700 1 -- 172.21.15.104:0/1540479438 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400e070 con 0x7f06cc0ce890 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06cb7fe700 1 -- 172.21.15.104:0/1540479438 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4003a20 con 0x7f06cc0ce890 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.910+0000 7f06cb7fe700 1 -- 172.21.15.104:0/1540479438 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4003dd0 con 0x7f06cc0ce890 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 -- 172.21.15.104:0/1540479438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 msgr2=0x7f06cc15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/1540479438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc15a7f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06cc066590 tx=0x7f06c400b330 comp rx=0 tx=0).stop 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 -- 172.21.15.104:0/1540479438 shutdown_connections 2023-06-10T18:45:51.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/1540479438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc15a7f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/1540479438 >> v2:172.21.15.114:3300/0 conn(0x7f06cc0b4ff0 0x7f06cc0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 -- 172.21.15.104:0/1540479438 >> 172.21.15.104:0/1540479438 conn(0x7f06cc089fd0 msgr2=0x7f06cc0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 -- 172.21.15.104:0/1540479438 shutdown_connections 2023-06-10T18:45:51.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.911+0000 7f06d2fa5700 1 -- 172.21.15.104:0/1540479438 wait complete. 2023-06-10T18:45:51.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.912+0000 7f06d2fa5700 1 Processor -- start 2023-06-10T18:45:51.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.912+0000 7f06d2fa5700 1 -- start start 2023-06-10T18:45:51.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06cc0b4ff0 0x7f06cc152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 0x7f06cc0c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc152fa0 con 0x7f06cc0ce890 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc153280 con 0x7f06cc0b4ff0 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d2fa5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc153560 con 0x7f06cc15a420 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57506/0 (socket says 172.21.15.104:57506) 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 -- 172.21.15.104:0/2854453285 learned_addr learned my addr 172.21.15.104:0/2854453285 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06cc0b4ff0 0x7f06cc152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d1542700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 0x7f06cc0c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 -- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 msgr2=0x7f06cc0c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 0x7f06cc0c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 -- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06cc0b4ff0 msgr2=0x7f06cc152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06cc0b4ff0 0x7f06cc152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06cbfff700 1 -- 172.21.15.104:0/2854453285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009b30 con 0x7f06cc0ce890 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.913+0000 7f06d1542700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 0x7f06cc0c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f06b800b4a0 tx=0x7f06b800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06c97fa700 1 -- 172.21.15.104:0/2854453285 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8011830 con 0x7f06cc0ce890 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06c97fa700 1 -- 172.21.15.104:0/2854453285 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b8011990 con 0x7f06cc0ce890 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc151280 con 0x7f06cc0ce890 2023-06-10T18:45:51.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06c97fa700 1 -- 172.21.15.104:0/2854453285 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8011d40 con 0x7f06cc0ce890 2023-06-10T18:45:51.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.914+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc152bb0 con 0x7f06cc0ce890 2023-06-10T18:45:51.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.915+0000 7f06c97fa700 1 -- 172.21.15.104:0/2854453285 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f06b801a400 con 0x7f06cc0ce890 2023-06-10T18:45:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.915+0000 7f06c97fa700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 0x7f06b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.916+0000 7f06c97fa700 1 -- 172.21.15.104:0/2854453285 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f06b80895f0 con 0x7f06cc0ce890 2023-06-10T18:45:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.916+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 0x7f06b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.916+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 0x7f06b006d190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06c0001730 tx=0x7f06c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 Processor -- start 2023-06-10T18:45:51.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 -- start start 2023-06-10T18:45:51.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e3fd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e2dbc60 con 0x55e86e3d7270 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e32e9f0 con 0x55e86e3fdf00 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e86e327e80 con 0x55e86e3d4740 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57512/0 (socket says 172.21.15.104:57512) 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.918+0000 7f06d0d41700 1 -- 172.21.15.104:0/421853125 learned_addr learned my addr 172.21.15.104:0/421853125 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 -- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 msgr2=0x55e86e3fd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e3fd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 -- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 msgr2=0x55e86e4011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 -- 172.21.15.104:0/421853125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e86e1a95e0 con 0x55e86e3d7270 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06d0d41700 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06c00053c0 tx=0x7f06c0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db161e2ba5d8adf0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06b67fc700 1 -- 172.21.15.104:0/421853125 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0003ea0 con 0x55e86e3d7270 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06b67fc700 1 -- 172.21.15.104:0/421853125 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0021450 con 0x55e86e3d7270 2023-06-10T18:45:51.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06b67fc700 1 -- 172.21.15.104:0/421853125 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c00217c0 con 0x55e86e3d7270 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06df12ab80 1 -- 172.21.15.104:0/421853125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 msgr2=0x55e86e3d7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.919+0000 7f06df12ab80 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06c00053c0 tx=0x7f06c0011000 comp rx=0 tx=0).stop 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 -- 172.21.15.104:0/421853125 shutdown_connections 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e3d7640 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 --2- 172.21.15.104:0/421853125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e3fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 -- 172.21.15.104:0/421853125 >> 172.21.15.104:0/421853125 conn(0x55e86e3cbfc0 msgr2=0x55e86e41c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 -- 172.21.15.104:0/421853125 shutdown_connections 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 -- 172.21.15.104:0/421853125 wait complete. 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 -- start start 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.920+0000 7f06df12ab80 1 Processor -- start 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 0x55e86e3d50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 0x55e86e3d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e2c7a70 con 0x55e86e3d4740 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e2c7770 con 0x55e86e3d7270 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e86e1ab8d0 con 0x55e86e3fdf00 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06df12ab80 10 client.0 _ll_register_callbacks cb 0x55e86e2cd210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57518/0 (socket says 172.21.15.104:57518) 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06d0d41700 1 -- 172.21.15.104:0/2223062899 learned_addr learned my addr 172.21.15.104:0/2223062899 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 0x55e86e3d50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.921+0000 7f06d1542700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 0x55e86e3d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 -- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 msgr2=0x55e86e3d60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 0x55e86e3d60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 -- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 msgr2=0x55e86e3d50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 0x55e86e3d50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 -- 172.21.15.104:0/2223062899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c000f040 con 0x55e86e3d4740 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 0x55e86e3d50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:51.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d1542700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 0x55e86e3d60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55e86e3d7b80 tx=0x7f06c0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401be2624ce8deaf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0028070 con 0x55e86e3d4740 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f06c0013420 con 0x55e86e3d4740 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e86e3c9780 con 0x55e86e3d4740 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06c0013620 con 0x55e86e3d4740 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 10 client.5811 fetch_fsmap learned FSMap version 17 2023-06-10T18:45:51.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55e86e155d90 con 0x55e86e3d4740 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e86e3c9780 con 0x55e86e3d4740 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f06c0013780 con 0x55e86e3d4740 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 10 client.5811 fetch_fsmap finished waiting for FSMap version 17 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 10 client.5811 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 20 client.5811 populate_metadata read hostname 'smithi104' 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e42fed0 0x55e86e4302a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 4 client.5811 mds_command: new command op to 5622 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:45:51.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.922+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 --> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e86e320e60 con 0x55e86e42fed0 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.923+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e42fed0 0x55e86e4302a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.923+0000 7f06ab7fe700 10 client.5811 ms_handle_connect on v2:172.21.15.114:6834/698196467 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.923+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e42fed0 0x55e86e4302a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf1f47d4c0cf93ce server_cookie=49f53d6a5f5292bf in_seq=0 out_seq=0 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06ab7fe700 1 -- 172.21.15.104:0/2223062899 <== mds.0 v2:172.21.15.114:6834/698196467 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e86e320e60 con 0x55e86e42fed0 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06ab7fe700 10 client.5811 handle_command_reply: tid=0 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 client.5811 shutdown 2023-06-10T18:45:51.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 msgr2=0x55e86e3fa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55e86e3d7b80 tx=0x7f06c0013000 comp rx=0 tx=0).stop 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 shutdown_connections 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e42fed0 0x55e86e4302a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3fdf00 0x55e86e3d60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06ab7fe700 0 client.5811 ms_handle_reset on v2:172.21.15.114:6834/698196467 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e3fa220 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2223062899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3d7270 0x55e86e3d50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 >> 172.21.15.104:0/2223062899 conn(0x55e86e3cbfc0 msgr2=0x55e86e41c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 shutdown_connections 2023-06-10T18:45:51.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 1 -- 172.21.15.104:0/2223062899 wait complete. 2023-06-10T18:45:51.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.924+0000 7f06df12ab80 20 client.5811 trim_cache size 0 max 16384 2023-06-10T18:45:51.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 Processor -- start 2023-06-10T18:45:51.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 -- start start 2023-06-10T18:45:51.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d7270 0x55e86e3d58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e3d5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e3cda70 con 0x55e86e3d4740 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e3d5610 con 0x55e86e3fdf00 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.942+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e86e4399e0 con 0x55e86e3d7270 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57528/0 (socket says 172.21.15.104:57528) 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 -- 172.21.15.104:0/2356035975 learned_addr learned my addr 172.21.15.104:0/2356035975 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06cbfff700 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d7270 0x55e86e3d58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d1542700 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e3d5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 -- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d7270 msgr2=0x55e86e3d58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d7270 0x55e86e3d58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 -- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 msgr2=0x55e86e3d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e3d5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 -- 172.21.15.104:0/2356035975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e86e13ec10 con 0x55e86e3d4740 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06d0d41700 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f06c0021950 tx=0x7f06c001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d999634a643f0ee server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06b6ffd700 1 -- 172.21.15.104:0/2356035975 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0013ea0 con 0x55e86e3d4740 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06b6ffd700 1 -- 172.21.15.104:0/2356035975 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0019040 con 0x55e86e3d4740 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.943+0000 7f06b6ffd700 1 -- 172.21.15.104:0/2356035975 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0013640 con 0x55e86e3d4740 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- 172.21.15.104:0/2356035975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 msgr2=0x55e86e4394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f06c0021950 tx=0x7f06c001d000 comp rx=0 tx=0).stop 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- 172.21.15.104:0/2356035975 shutdown_connections 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e3d5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d4740 0x55e86e4394a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 --2- 172.21.15.104:0/2356035975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d7270 0x55e86e3d58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- 172.21.15.104:0/2356035975 >> 172.21.15.104:0/2356035975 conn(0x55e86e3cbfc0 msgr2=0x55e86e3d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- 172.21.15.104:0/2356035975 shutdown_connections 2023-06-10T18:45:51.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- 172.21.15.104:0/2356035975 wait complete. 2023-06-10T18:45:51.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 -- start start 2023-06-10T18:45:51.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.944+0000 7f06df12ab80 1 Processor -- start 2023-06-10T18:45:51.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e48a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4008f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e3fd050 con 0x55e86e3d7270 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86e400f40 con 0x55e86e3fdf00 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e86e403670 con 0x55e86e3d4740 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06df12ab80 10 client.0 _ll_register_callbacks cb 0x55e86e2cd210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57534/0 (socket says 172.21.15.104:57534) 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 -- 172.21.15.104:0/3085742945 learned_addr learned my addr 172.21.15.104:0/3085742945 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06d0d41700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e48a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06d1542700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 -- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 msgr2=0x55e86e48a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e48a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 -- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 msgr2=0x55e86e4008f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.945+0000 7f06cbfff700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4008f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06cbfff700 1 -- 172.21.15.104:0/3085742945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e86e155d90 con 0x55e86e3d7270 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06d1542700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06cbfff700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f06b8057ea0 tx=0x7f06b8060560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a76beb719eafdc3b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80600b0 con 0x55e86e3d7270 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f06b8060210 con 0x55e86e3d7270 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06c0021450 con 0x55e86e3d7270 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06b806fbc0 con 0x55e86e3d7270 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 10 client.5817 fetch_fsmap learned FSMap version 17 2023-06-10T18:45:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55e86e17c200 con 0x55e86e3d7270 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0021450 con 0x55e86e3d7270 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f06b806fd20 con 0x55e86e3d7270 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 10 client.5817 fetch_fsmap finished waiting for FSMap version 17 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 10 client.5817 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 20 client.5817 populate_metadata read hostname 'smithi104' 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e427970 0x55e86e427d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 4 client.5817 mds_command: new command op to 5622 tid=0[{"prefix": "status"}] 2023-06-10T18:45:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 --> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e86e2e3000 con 0x55e86e427970 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.946+0000 7f06d0d41700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e427970 0x55e86e427d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06ab7fe700 10 client.5817 ms_handle_connect on v2:172.21.15.114:6834/698196467 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06d0d41700 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e427970 0x55e86e427d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd68a68d8d7e42a6 server_cookie=869785827120b4d3 in_seq=0 out_seq=0 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06ab7fe700 1 -- 172.21.15.104:0/3085742945 <== mds.0 v2:172.21.15.114:6834/698196467 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e86e2e3000 con 0x55e86e427970 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06ab7fe700 10 client.5817 handle_command_reply: tid=0 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 client.5817 shutdown 2023-06-10T18:45:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 msgr2=0x55e86e41d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f06b8057ea0 tx=0x7f06b8060560 comp rx=0 tx=0).stop 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 shutdown_connections 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:6834/698196467,v1:172.21.15.114:6836/698196467] conn(0x55e86e427970 0x55e86e427d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e86e3fdf00 0x55e86e4008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06ab7fe700 0 client.5817 ms_handle_reset on v2:172.21.15.114:6834/698196467 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55e86e3d4740 0x55e86e48a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 --2- 172.21.15.104:0/3085742945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e86e3d7270 0x55e86e41d940 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 >> 172.21.15.104:0/3085742945 conn(0x55e86e3cbfc0 msgr2=0x55e86e3fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 shutdown_connections 2023-06-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 1 -- 172.21.15.104:0/3085742945 wait complete. 2023-06-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.947+0000 7f06df12ab80 20 client.5817 trim_cache size 0 max 16384 2023-06-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:45:51.963 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 5622, 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 10.508066242, 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 17, 2023-06-10T18:45:51.964 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 44, 2023-06-10T18:45:51.965 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 42, 2023-06-10T18:45:51.965 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 17.925681955000002 2023-06-10T18:45:51.965 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:45:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 msgr2=0x7f06b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 0x7f06b006d190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06c0001730 tx=0x7f06c0009040 comp rx=0 tx=0).stop 2023-06-10T18:45:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 msgr2=0x7f06cc1ef780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f06b800b4a0 tx=0x7f06b800b860 comp rx=0 tx=0).stop 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 shutdown_connections 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f06b006ace0 0x7f06b006d190 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06cc0b4ff0 0x7f06cc152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06cc15a420 0x7f06cc0c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 --2- 172.21.15.104:0/2854453285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06cc0ce890 0x7f06cc1ef780 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 >> 172.21.15.104:0/2854453285 conn(0x7f06cc089fd0 msgr2=0x7f06cc0cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 shutdown_connections 2023-06-10T18:45:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:51.948+0000 7f06d2fa5700 1 -- 172.21.15.104:0/2854453285 wait complete. 2023-06-10T18:45:51.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:45:52.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.082+0000 7fc74b966700 1 Processor -- start 2023-06-10T18:45:52.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.082+0000 7fc74b966700 1 -- start start 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc74b966700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc7440aeaa0 0x7fc7440cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc74b966700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc7440b1080 0x7fc7440cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc74b966700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7440c9e60 con 0x7fc7440ca2f0 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc74b966700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440c9bc0 con 0x7fc7440b1080 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc74b966700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440aee70 con 0x7fc7440aeaa0 2023-06-10T18:45:52.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc748f01700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc7440b1080 0x7fc7440cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc748f01700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc7440b1080 0x7fc7440cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57542/0 (socket says 172.21.15.104:57542) 2023-06-10T18:45:52.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc748f01700 1 -- 172.21.15.104:0/3751471068 learned_addr learned my addr 172.21.15.104:0/3751471068 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:52.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.083+0000 7fc749702700 1 --2- 172.21.15.104:0/3751471068 >> v2:172.21.15.114:3300/0 conn(0x7fc7440aeaa0 0x7fc7440cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc748f01700 1 -- 172.21.15.104:0/3751471068 >> v2:172.21.15.114:3300/0 conn(0x7fc7440aeaa0 msgr2=0x7fc7440cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc748f01700 1 --2- 172.21.15.104:0/3751471068 >> v2:172.21.15.114:3300/0 conn(0x7fc7440aeaa0 0x7fc7440cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc748f01700 1 -- 172.21.15.104:0/3751471068 >> v1:172.21.15.114:6789/0 conn(0x7fc7440ca2f0 legacy=0x7fc7440ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc748f01700 1 -- 172.21.15.104:0/3751471068 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74406e540 con 0x7fc7440b1080 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc748f01700 1 --2- 172.21.15.104:0/3751471068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440b1080 0x7fc7440cd680 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7440cf5d0 tx=0x7fc740009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4081fe55aaba8dfb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc73bfff700 1 -- 172.21.15.104:0/3751471068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740011830 con 0x7fc7440b1080 2023-06-10T18:45:52.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc73bfff700 1 -- 172.21.15.104:0/3751471068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740011990 con 0x7fc7440b1080 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc73bfff700 1 -- 172.21.15.104:0/3751471068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740011d20 con 0x7fc7440b1080 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 -- 172.21.15.104:0/3751471068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440b1080 msgr2=0x7fc7440cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 --2- 172.21.15.104:0/3751471068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440b1080 0x7fc7440cd680 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7440cf5d0 tx=0x7fc740009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 -- 172.21.15.104:0/3751471068 shutdown_connections 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 --2- 172.21.15.104:0/3751471068 >> v2:172.21.15.114:3300/0 conn(0x7fc7440aeaa0 0x7fc7440cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 --2- 172.21.15.104:0/3751471068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440b1080 0x7fc7440cd680 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.084+0000 7fc74b966700 1 -- 172.21.15.104:0/3751471068 >> 172.21.15.104:0/3751471068 conn(0x7fc74406b8a0 msgr2=0x7fc7440cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.085+0000 7fc74b966700 1 -- 172.21.15.104:0/3751471068 shutdown_connections 2023-06-10T18:45:52.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.085+0000 7fc74b966700 1 -- 172.21.15.104:0/3751471068 wait complete. 2023-06-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 Processor -- start 2023-06-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 -- start start 2023-06-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7440aeaa0 0x7fc74414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7440b1080 0x7fc744069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74415a5a0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.086+0000 7fc74b966700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744152dc0 con 0x7fc7440b1080 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc74b966700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc744159f20 con 0x7fc7440aeaa0 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57550/0 (socket says 172.21.15.104:57550) 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 -- 172.21.15.104:0/3729820848 learned_addr learned my addr 172.21.15.104:0/3729820848 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:52.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749702700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7440aeaa0 0x7fc74414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc748f01700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7440b1080 0x7fc744069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 -- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7440aeaa0 msgr2=0x7fc74414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7440aeaa0 0x7fc74414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 -- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7440b1080 msgr2=0x7fc744069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7440b1080 0x7fc744069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 -- 172.21.15.104:0/3729820848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740009550 con 0x7fc7440ca2f0 2023-06-10T18:45:52.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc749f03700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00cb50 tx=0x7fc73c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c003c00 con 0x7fc7440ca2f0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c003d60 con 0x7fc7440ca2f0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7440adce0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00ab70 con 0x7fc7440ca2f0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.087+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74414af20 con 0x7fc7440ca2f0 2023-06-10T18:45:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.088+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74415a6e0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.089+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc73c00acd0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.089+0000 7fc739ffb700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 0x7fc72806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.089+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc73c089640 con 0x7fc7440ca2f0 2023-06-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.089+0000 7fc749702700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 0x7fc72806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.090+0000 7fc749702700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 0x7fc72806d190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc734009b50 tx=0x7fc734008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.093+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc74415a6e0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.321+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc74415a6e0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.321+0000 7fc739ffb700 1 -- 172.21.15.104:0/3729820848 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc74415a6e0 con 0x7fc7440ca2f0 2023-06-10T18:45:52.327 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:45:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 msgr2=0x7fc72806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 0x7fc72806d190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc734009b50 tx=0x7fc734008040 comp rx=0 tx=0).stop 2023-06-10T18:45:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 msgr2=0x7fc74415a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00cb50 tx=0x7fc73c00cf10 comp rx=0 tx=0).stop 2023-06-10T18:45:52.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 shutdown_connections 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7440ca2f0 0x7fc74415a9c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7440aeaa0 0x7fc74414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc72806ace0 0x7fc72806d190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 --2- 172.21.15.104:0/3729820848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7440b1080 0x7fc744069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 >> 172.21.15.104:0/3729820848 conn(0x7fc74406b8a0 msgr2=0x7fc7440af060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 shutdown_connections 2023-06-10T18:45:52.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:52.323+0000 7fc74b966700 1 -- 172.21.15.104:0/3729820848 wait complete. 2023-06-10T18:45:52.341 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:45:52.341 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:45:52.341 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:45:52.342 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:45:52.342 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:52.342 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:45:52.343 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:45:52.343 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:45:52.343 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:45:52.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.344 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:45:52.361 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:45:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:45:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:45:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:52.363 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42332sec preferred_lft 42332sec 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:45:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:45:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:45:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.366 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:45:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:45:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:45:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:52.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.367 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:45:52.426 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:45:52.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.428 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:45:52.485 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:45:52.486 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:45:52.486 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:45:52.486 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:45:52.486 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:45:52.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.487 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:45:52.487 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:45:52.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:45:52.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:45:52.488 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:45:52.488 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:45:52.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.590 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:45:52.606 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:45:52.607 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:45:52.607 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:45:52.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.608 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:45:52.674 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:45:52.674 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:45:52.675 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:52.730 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:45:52.731 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:45:52.731 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:52.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:45:52.853 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:45:52.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:45:52.918 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:45:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:45:52.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:52.920 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:45:52.977 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:45:52.977 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:45:53.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:45:53.046 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:45:53.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:45:53.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:45:53.100+0000 7f6b84e495c0 -1 init, newargv = 0x5585ddb338d0 newargc=15 2023-06-10T18:45:53.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[119644]: starting ceph client 2023-06-10T18:45:53.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[119644]: starting fuse 2023-06-10T18:45:53.114 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:45:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:45:53.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:53.116 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:45:53.173 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:45:53.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:45:53.174 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:45:53.174 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:45:53.174 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:45:53.174 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:45:53.175 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:45:53.175 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:45:53.175 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:45:53.176 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:45:53.176 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:45:53.176 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:45:53.177 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:45:53.177 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:45:53.177 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:45:53.178 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:45:53.178 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:45:53.178 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:45:53.178 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:45:53.179 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:45:53.179 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:45:53.179 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:45:53.180 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:45:53.180 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:45:53.180 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:45:53.181 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:45:53.181 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:45:53.181 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:45:53.294 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.119644.asok 2023-06-10T18:45:53.476 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.119644.asok 2023-06-10T18:45:53.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:53.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.119644.asok status 2023-06-10T18:45:53.632 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:45:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:45:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:45:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:45:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "119644", 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:45:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 5829, 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 5829 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:45:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2075233745 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:45:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2075233745 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.5829 192.168.0.1:0/2075233745", 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2075233745", 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 17, 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 44, 2023-06-10T18:45:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:45:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:45:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:45:53.638 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:45:53.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:53.643 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:45:53.643 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:53.660 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:45:53.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:53.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:53.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:45:53.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:45:53.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.882+0000 7f5af3d5e700 1 Processor -- start 2023-06-10T18:45:53.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.882+0000 7f5af3d5e700 1 -- start start 2023-06-10T18:45:53.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af3d5e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5aec165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af3d5e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0b6070 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af3d5e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0b1d70 con 0x7f5aec15a420 2023-06-10T18:45:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af3d5e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0b5460 con 0x7f5aec0cea60 2023-06-10T18:45:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af12f9700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5aec0b4ff0 0x7f5aec0ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55396/0 (socket says 172.21.15.104:55396) 2023-06-10T18:45:53.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.883+0000 7f5af12f9700 1 -- 172.21.15.104:0/4007318452 learned_addr learned my addr 172.21.15.104:0/4007318452 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5aec165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725211165 0 0) 0x7f5aec0b6070 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad80035b0 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462687951 0 0) 0x7f5aec0b1d70 con 0x7f5aec15a420 2023-06-10T18:45:53.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec0b6070 con 0x7f5aec15a420 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486265884 0 0) 0x7f5ad80035b0 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 msgr2=0x7f5aec165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5aec165060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 >> v1:172.21.15.114:6789/0 conn(0x7f5aec15a420 legacy=0x7f5aec0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0b5740 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5ae8001670 con 0x7f5aec0b4ff0 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 >> v1:172.21.15.104:6789/0 conn(0x7f5aec0b4ff0 legacy=0x7f5aec0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:53.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad8003bc0 0x7f5ad8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5ad8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ad8004d90 0x7f5ad8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0b5460 con 0x7f5ad8004d90 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad800bac0 con 0x7f5ad8003bc0 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad800bc30 con 0x7f5aec0cea60 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ad8004d90 0x7f5ad8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af1afa700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad8003bc0 0x7f5ad8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af12f9700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5ad8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 -- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 msgr2=0x7f5ad8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec0cea60 0x7f5ad8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 -- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad8003bc0 msgr2=0x7f5ad8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad8003bc0 0x7f5ad8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.884+0000 7f5af22fb700 1 -- 172.21.15.104:0/4007318452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad80035b0 con 0x7f5ad8004d90 2023-06-10T18:45:53.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af22fb700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ad8004d90 0x7f5ad8005200 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f90 tx=0x7f5ad8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe369cd76e0f73b0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400ebd0 con 0x7f5ad8004d90 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae400f040 con 0x7f5ad8004d90 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af0af8700 1 -- 172.21.15.104:0/4007318452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4014480 con 0x7f5ad8004d90 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 -- 172.21.15.104:0/4007318452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ad8004d90 msgr2=0x7f5ad8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/4007318452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ad8004d90 0x7f5ad8005200 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f90 tx=0x7f5ad8005880 comp rx=0 tx=0).stop 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af22fb700 1 -- 172.21.15.104:0/4007318452 reap_dead start 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 -- 172.21.15.104:0/4007318452 shutdown_connections 2023-06-10T18:45:53.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 -- 172.21.15.104:0/4007318452 >> 172.21.15.104:0/4007318452 conn(0x7f5aec089fd0 msgr2=0x7f5aec0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:53.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 -- 172.21.15.104:0/4007318452 shutdown_connections 2023-06-10T18:45:53.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.885+0000 7f5af3d5e700 1 -- 172.21.15.104:0/4007318452 wait complete. 2023-06-10T18:45:53.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 Processor -- start 2023-06-10T18:45:53.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 -- start start 2023-06-10T18:45:53.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec153be0 0x7f5aec1e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 0x7f5aec1e4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec06ff30 con 0x7f5aec06f590 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0c5880 con 0x7f5aec0c6920 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af3d5e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0c5b60 con 0x7f5aec153be0 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af22fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af22fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57582/0 (socket says 172.21.15.104:57582) 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.887+0000 7f5af22fb700 1 -- 172.21.15.104:0/3757888616 learned_addr learned my addr 172.21.15.104:0/3757888616 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af1afa700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec153be0 0x7f5aec1e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af12f9700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 0x7f5aec1e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 -- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec153be0 msgr2=0x7f5aec1e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec153be0 0x7f5aec1e4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 -- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 msgr2=0x7f5aec1e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 0x7f5aec1e4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 -- 172.21.15.104:0/3757888616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0c5e40 con 0x7f5aec06f590 2023-06-10T18:45:53.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af12f9700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 0x7f5aec1e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af22fb700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4016000 tx=0x7f5ae4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4007000 con 0x7f5aec06f590 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4003e50 con 0x7f5aec06f590 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4015b60 con 0x7f5aec06f590 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0c6120 con 0x7f5aec06f590 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.888+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec0c5530 con 0x7f5aec06f590 2023-06-10T18:45:53.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.889+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec0c6640 con 0x7f5aec06f590 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.889+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5ae4024070 con 0x7f5aec06f590 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.890+0000 7f5ae27fc700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 0x7f5ad006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.890+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f5ae408b440 con 0x7f5aec06f590 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.890+0000 7f5af1afa700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 0x7f5ad006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.890+0000 7f5af1afa700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 0x7f5ad006d190 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5adc007fd0 tx=0x7f5adc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:53.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:53.894+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5aec0c6640 con 0x7f5aec06f590 2023-06-10T18:45:54.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.126+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5aec0c6640 con 0x7f5aec06f590 2023-06-10T18:45:54.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.126+0000 7f5ae27fc700 1 -- 172.21.15.104:0/3757888616 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5aec0c6640 con 0x7f5aec06f590 2023-06-10T18:45:54.133 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:45:54.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 msgr2=0x7f5ad006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:54.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 0x7f5ad006d190 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5adc007fd0 tx=0x7f5adc009040 comp rx=0 tx=0).stop 2023-06-10T18:45:54.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 msgr2=0x7f5aec06f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:54.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4016000 tx=0x7f5ae4009750 comp rx=0 tx=0).stop 2023-06-10T18:45:54.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 shutdown_connections 2023-06-10T18:45:54.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5aec06f590 0x7f5aec06f9f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:54.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aec153be0 0x7f5aec1e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:54.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5ad006ace0 0x7f5ad006d190 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:54.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 --2- 172.21.15.104:0/3757888616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aec0c6920 0x7f5aec1e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:54.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 >> 172.21.15.104:0/3757888616 conn(0x7f5aec089fd0 msgr2=0x7f5aec0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:54.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 shutdown_connections 2023-06-10T18:45:54.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:54.129+0000 7f5af3d5e700 1 -- 172.21.15.104:0/3757888616 wait complete. 2023-06-10T18:45:54.146 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:45:54.146 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:45:54.146 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:45:54.147 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:45:54.147 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:54.147 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:45:54.147 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:45:54.148 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:45:54.148 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:45:54.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.149 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:45:54.170 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:45:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:45:54.170 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:54.171 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42326sec preferred_lft 42326sec 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:45:54.172 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.173 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:45:54.174 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:45:54.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.175 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:45:54.239 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:45:54.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.241 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:45:54.302 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:45:54.303 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:45:54.303 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:45:54.303 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:45:54.304 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:45:54.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.305 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:45:54.305 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:45:54.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:45:54.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:45:54.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:45:54.308 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:45:54.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.413 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:45:54.431 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:45:54.431 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:45:54.432 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:45:54.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:45:54.498 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:45:54.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:45:54.500 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:54.556 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:45:54.556 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:45:54.556 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:54.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:45:54.678 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:45:54.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:45:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:45:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:45:54.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.746 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:45:54.802 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:45:54.802 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:45:54.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:45:54.870 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:45:54.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:45:54.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:45:54.924+0000 7f01225335c0 -1 init, newargv = 0x5583ea00b900 newargc=15 2023-06-10T18:45:54.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[108371]: starting ceph client 2023-06-10T18:45:54.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[108371]: starting fuse 2023-06-10T18:45:54.936 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:45:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:45:54.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:54.939 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:45:54.994 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:45:54.995 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:45:54.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:45:54.995 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:45:54.996 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:45:54.996 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:45:54.996 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:45:54.997 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:45:54.997 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:45:54.997 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:45:54.998 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:45:54.998 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:45:54.998 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:45:54.998 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:45:54.999 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:45:54.999 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:45:54.999 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:45:55.000 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:45:55.000 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:45:55.000 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:45:55.001 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:45:55.001 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:45:55.001 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:45:55.001 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:45:55.002 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:45:55.002 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:45:55.002 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:45:55.003 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:45:55.003 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:45:55.116 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.108371.asok 2023-06-10T18:45:55.298 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.108371.asok 2023-06-10T18:45:55.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:55.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.108371.asok status 2023-06-10T18:45:55.454 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:45:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:45:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:45:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:45:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:45:55.455 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "108371", 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:45:55.456 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5437, 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5437 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:45:55.457 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4073369385 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:45:55.458 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4073369385 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.5437 192.168.0.1:0/4073369385", 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/4073369385", 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 17, 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 44, 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:45:55.459 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:45:55.460 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:45:55.460 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:45:55.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:55.465 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:45:55.465 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:55.482 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:45:55.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:55.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:45:55.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:45:55.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:45:55.697 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:45:55.698 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:45:55.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:45:55.832 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:45:55.833 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:45:55.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:55.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.946+0000 7f6381b5b700 1 Processor -- start 2023-06-10T18:45:55.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.946+0000 7f6381b5b700 1 -- start start 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f6381b5b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f637c06af80 0x7f637c06b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f6381b5b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c0cdec0 con 0x7f637c0b15b0 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f6381b5b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c06b890 con 0x7f637c0b1a20 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f6381b5b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c0b1ea0 con 0x7f637c06af80 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f637affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f637c0b15b0 0x7f637c0cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55414/0 (socket says 172.21.15.104:55414) 2023-06-10T18:45:55.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.947+0000 7f637affd700 1 -- 172.21.15.104:0/411375055 learned_addr learned my addr 172.21.15.104:0/411375055 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237310033 0 0) 0x7f637c0cdec0 con 0x7f637c0b15b0 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63640035b0 con 0x7f637c0b15b0 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 --2- 172.21.15.104:0/411375055 >> v2:172.21.15.114:3300/0 conn(0x7f637c06af80 0x7f637c06b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1661532424 0 0) 0x7f63640035b0 con 0x7f637c0b15b0 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 >> v2:172.21.15.114:3300/0 conn(0x7f637c06af80 msgr2=0x7f637c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 --2- 172.21.15.104:0/411375055 >> v2:172.21.15.114:3300/0 conn(0x7f637c06af80 0x7f637c06b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 >> v1:172.21.15.114:6789/0 conn(0x7f637c0b1a20 legacy=0x7f637c0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0cb880 con 0x7f637c0b15b0 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f636c001670 con 0x7f637c0b15b0 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 >> v1:172.21.15.104:6789/0 conn(0x7f637c0b15b0 legacy=0x7f637c0cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f6364003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6364004850 0x7f6364004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6364005200 0x7f636400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0b1ea0 con 0x7f6364004850 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636400c080 con 0x7f6364003a00 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636400c240 con 0x7f6364005200 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f6364003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637affd700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6364004850 0x7f6364004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637bfff700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6364005200 0x7f636400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 -- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6364005200 msgr2=0x7f636400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6364005200 0x7f636400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 -- 172.21.15.104:0/411375055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6364004850 msgr2=0x7f6364004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6364004850 0x7f6364004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 -- 172.21.15.104:0/411375055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640035b0 con 0x7f6364003a00 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637bfff700 1 -- 172.21.15.104:0/411375055 reap_dead start 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637affd700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6364004850 0x7f6364004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.948+0000 7f637b7fe700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f6364003e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6370009b60 tx=0x7f6364005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcabd2ee769ed9c1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000e6c0 con 0x7f6364003a00 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637000e820 con 0x7f6364003a00 2023-06-10T18:45:55.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f637a7fc700 1 -- 172.21.15.104:0/411375055 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000eb90 con 0x7f6364003a00 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 -- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 msgr2=0x7f6364003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f6364003e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6370009b60 tx=0x7f6364005880 comp rx=0 tx=0).stop 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 -- 172.21.15.104:0/411375055 shutdown_connections 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 --2- 172.21.15.104:0/411375055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f6364003e10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 -- 172.21.15.104:0/411375055 >> 172.21.15.104:0/411375055 conn(0x7f637c089fd0 msgr2=0x7f637c0d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 -- 172.21.15.104:0/411375055 shutdown_connections 2023-06-10T18:45:55.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.949+0000 7f6381b5b700 1 -- 172.21.15.104:0/411375055 wait complete. 2023-06-10T18:45:55.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.950+0000 7f6381b5b700 1 Processor -- start 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 -- start start 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f637c152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 0x7f637c0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c15b3f0 con 0x7f637c15aa10 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0b3650 con 0x7f6364003a00 2023-06-10T18:45:55.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f6381b5b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c152d90 con 0x7f637c15d320 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f637bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f637bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57602/0 (socket says 172.21.15.104:57602) 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.951+0000 7f637bfff700 1 -- 172.21.15.104:0/3279410895 learned_addr learned my addr 172.21.15.104:0/3279410895 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637b7fe700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f637c152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637affd700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 0x7f637c0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 -- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 msgr2=0x7f637c0c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:55.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 0x7f637c0c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 -- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 msgr2=0x7f637c152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f637c152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 -- 172.21.15.104:0/3279410895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0b38a0 con 0x7f637c15aa10 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637affd700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 0x7f637c0c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f637bfff700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f637c06c160 tx=0x7f637400d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6374003c10 con 0x7f637c15aa10 2023-06-10T18:45:55.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6374003d90 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400da10 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c0b3b80 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.952+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c0b4140 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.953+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c074050 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.953+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f637400db70 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.954+0000 7f6363fff700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 0x7f636806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.954+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f6374089710 con 0x7f637c15aa10 2023-06-10T18:45:55.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.954+0000 7f637b7fe700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 0x7f636806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:55.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.954+0000 7f637b7fe700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 0x7f636806d190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63700090b0 tx=0x7f637001f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:55.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:55.958+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f63740546d0 con 0x7f637c15aa10 2023-06-10T18:45:56.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.187+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f637c0d13f0 con 0x7f637c15aa10 2023-06-10T18:45:56.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.188+0000 7f6363fff700 1 -- 172.21.15.104:0/3279410895 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f6374057cf0 con 0x7f637c15aa10 2023-06-10T18:45:56.193 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:56.194 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:45:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 msgr2=0x7f636806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 0x7f636806d190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63700090b0 tx=0x7f637001f040 comp rx=0 tx=0).stop 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 msgr2=0x7f637c15aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f637c06c160 tx=0x7f637400d150 comp rx=0 tx=0).stop 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 shutdown_connections 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f636806ace0 0x7f636806d190 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6364003a00 0x7f637c152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f637c15aa10 0x7f637c15aeb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 --2- 172.21.15.104:0/3279410895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f637c15d320 0x7f637c0c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 >> 172.21.15.104:0/3279410895 conn(0x7f637c089fd0 msgr2=0x7f637c0cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 shutdown_connections 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.190+0000 7f6381b5b700 1 -- 172.21.15.104:0/3279410895 wait complete. 2023-06-10T18:45:56.197 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:45:56.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_defer_session_stale false 2023-06-10T18:45:56.346 INFO:teuthology.orchestra.run.smithi114.stdout:{"success":"mds_defer_session_stale = 'false' (not observed, change may require restart) "} 2023-06-10T18:45:56.346 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_defer_session_stale = 'false' (not observed, change may require restart) " } 2023-06-10T18:45:56.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:45:56.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.461+0000 7f2c2f496700 1 Processor -- start 2023-06-10T18:45:56.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.461+0000 7f2c2f496700 1 -- start start 2023-06-10T18:45:56.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.461+0000 7f2c2f496700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c280ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2f496700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c280b6070 con 0x7f2c280ce890 2023-06-10T18:45:56.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2f496700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c280b1d70 con 0x7f2c2815a420 2023-06-10T18:45:56.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2f496700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280ced10 con 0x7f2c280b4ff0 2023-06-10T18:45:56.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2da33700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2c280ce890 0x7f2c2815a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55438/0 (socket says 172.21.15.104:55438) 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2da33700 1 -- 172.21.15.104:0/1607568714 learned_addr learned my addr 172.21.15.104:0/1607568714 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2ca31700 1 --2- 172.21.15.104:0/1607568714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c280ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697731603 0 0) 0x7f2c280b6070 con 0x7f2c280ce890 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c140035b0 con 0x7f2c280ce890 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429218380 0 0) 0x7f2c280b1d70 con 0x7f2c2815a420 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c280b6070 con 0x7f2c2815a420 2023-06-10T18:45:56.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2ca31700 1 -- 172.21.15.104:0/1607568714 >> v1:172.21.15.114:6789/0 conn(0x7f2c2815a420 legacy=0x7f2c280b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2ca31700 1 -- 172.21.15.104:0/1607568714 >> v1:172.21.15.104:6789/0 conn(0x7f2c280ce890 legacy=0x7f2c2815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2ca31700 1 -- 172.21.15.104:0/1607568714 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c280b5460 con 0x7f2c280b4ff0 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c2ca31700 1 --2- 172.21.15.104:0/1607568714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c280ce350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43a57a8d3bb45da server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011830 con 0x7f2c280b4ff0 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c011990 con 0x7f2c280b4ff0 2023-06-10T18:45:56.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.462+0000 7f2c1bfff700 1 -- 172.21.15.104:0/1607568714 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c011d20 con 0x7f2c280b4ff0 2023-06-10T18:45:56.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 -- 172.21.15.104:0/1607568714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 msgr2=0x7f2c280ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 --2- 172.21.15.104:0/1607568714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c280ce350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c009b80 comp rx=0 tx=0).stop 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 -- 172.21.15.104:0/1607568714 shutdown_connections 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 --2- 172.21.15.104:0/1607568714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c280ce350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 -- 172.21.15.104:0/1607568714 >> 172.21.15.104:0/1607568714 conn(0x7f2c28089fd0 msgr2=0x7f2c280d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 -- 172.21.15.104:0/1607568714 shutdown_connections 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.463+0000 7f2c2f496700 1 -- 172.21.15.104:0/1607568714 wait complete. 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.464+0000 7f2c2f496700 1 Processor -- start 2023-06-10T18:45:56.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.464+0000 7f2c2f496700 1 -- start start 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c28150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2815a420 0x7f2c280cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280c65e0 con 0x7f2c280ce890 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280c51d0 con 0x7f2c2815a420 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2f496700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280c4450 con 0x7f2c280b4ff0 2023-06-10T18:45:56.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57614/0 (socket says 172.21.15.104:57614) 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 -- 172.21.15.104:0/3251613895 learned_addr learned my addr 172.21.15.104:0/3251613895 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2da33700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2815a420 0x7f2c280cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2d232700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c28150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 -- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 msgr2=0x7f2c28150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c28150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 -- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2815a420 msgr2=0x7f2c280cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2815a420 0x7f2c280cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2ca31700 1 -- 172.21.15.104:0/3251613895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0094a0 con 0x7f2c280ce890 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.465+0000 7f2c2d232700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c28150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c2ca31700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c014040 tx=0x7f2c1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c01a760 con 0x7f2c280ce890 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c01a8c0 con 0x7f2c280ce890 2023-06-10T18:45:56.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c280c46f0 con 0x7f2c280ce890 2023-06-10T18:45:56.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c01ac30 con 0x7f2c280ce890 2023-06-10T18:45:56.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.466+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c280c4cb0 con 0x7f2c280ce890 2023-06-10T18:45:56.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.467+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c280c4ed0 con 0x7f2c280ce890 2023-06-10T18:45:56.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.467+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2c1c022070 con 0x7f2c280ce890 2023-06-10T18:45:56.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.468+0000 7f2c19ffb700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 0x7f2c0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.468+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2c1c08a600 con 0x7f2c280ce890 2023-06-10T18:45:56.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.468+0000 7f2c2d232700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 0x7f2c0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.468+0000 7f2c2d232700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 0x7f2c0c06d190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c240011e0 tx=0x7f2c24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.472+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2c1c055640 con 0x7f2c280ce890 2023-06-10T18:45:56.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.700+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c28074050 con 0x7f2c280ce890 2023-06-10T18:45:56.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.700+0000 7f2c19ffb700 1 -- 172.21.15.104:0/3251613895 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f2c1c058c60 con 0x7f2c280ce890 2023-06-10T18:45:56.706 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:56.706 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:45:56.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 msgr2=0x7f2c0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 0x7f2c0c06d190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c240011e0 tx=0x7f2c24009040 comp rx=0 tx=0).stop 2023-06-10T18:45:56.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 msgr2=0x7f2c280c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c014040 tx=0x7f2c1c004750 comp rx=0 tx=0).stop 2023-06-10T18:45:56.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 shutdown_connections 2023-06-10T18:45:56.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c0c06ace0 0x7f2c0c06d190 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2815a420 0x7f2c280cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c280b4ff0 0x7f2c28150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.702+0000 7f2c2f496700 1 --2- 172.21.15.104:0/3251613895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c280ce890 0x7f2c280c5d90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.703+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 >> 172.21.15.104:0/3251613895 conn(0x7f2c28089fd0 msgr2=0x7f2c2806fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.703+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 shutdown_connections 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.703+0000 7f2c2f496700 1 -- 172.21.15.104:0/3251613895 wait complete. 2023-06-10T18:45:56.710 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:45:56.720 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:56.721 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:56.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:56.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.892+0000 7f4f16ef9700 1 Processor -- start 2023-06-10T18:45:56.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.892+0000 7f4f16ef9700 1 -- start start 2023-06-10T18:45:56.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f16ef9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f10158c50 0x7f4f1006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f16ef9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4f1006b180 0x7f4f100cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f16ef9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f100cccd0 con 0x7f4f1006ca40 2023-06-10T18:45:56.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f16ef9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f100ccfb0 con 0x7f4f10158c50 2023-06-10T18:45:56.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f16ef9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f100ce0b0 con 0x7f4f1006b180 2023-06-10T18:45:56.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f10158c50 0x7f4f1006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4f10158c50 0x7f4f1006aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57620/0 (socket says 172.21.15.104:57620) 2023-06-10T18:45:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 -- 172.21.15.104:0/4230779409 learned_addr learned my addr 172.21.15.104:0/4230779409 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f0ffff700 1 --2- 172.21.15.104:0/4230779409 >> v2:172.21.15.114:3300/0 conn(0x7f4f1006b180 0x7f4f100cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 -- 172.21.15.104:0/4230779409 >> v2:172.21.15.114:3300/0 conn(0x7f4f1006b180 msgr2=0x7f4f100cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 --2- 172.21.15.104:0/4230779409 >> v2:172.21.15.114:3300/0 conn(0x7f4f1006b180 0x7f4f100cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 -- 172.21.15.104:0/4230779409 >> v1:172.21.15.114:6789/0 conn(0x7f4f1006ca40 legacy=0x7f4f1015c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 -- 172.21.15.104:0/4230779409 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1006a010 con 0x7f4f10158c50 2023-06-10T18:45:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f14c95700 1 --2- 172.21.15.104:0/4230779409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f10158c50 0x7f4f1006aaf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4f1015d970 tx=0x7f4f04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886bb3e68c334b9b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f0f7fe700 1 -- 172.21.15.104:0/4230779409 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04011830 con 0x7f4f10158c50 2023-06-10T18:45:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f0f7fe700 1 -- 172.21.15.104:0/4230779409 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04011990 con 0x7f4f10158c50 2023-06-10T18:45:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.893+0000 7f4f0f7fe700 1 -- 172.21.15.104:0/4230779409 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04011d20 con 0x7f4f10158c50 2023-06-10T18:45:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 -- 172.21.15.104:0/4230779409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f10158c50 msgr2=0x7f4f1006aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/4230779409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f10158c50 0x7f4f1006aaf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4f1015d970 tx=0x7f4f04009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 -- 172.21.15.104:0/4230779409 shutdown_connections 2023-06-10T18:45:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/4230779409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f10158c50 0x7f4f1006aaf0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/4230779409 >> v2:172.21.15.114:3300/0 conn(0x7f4f1006b180 0x7f4f100cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 -- 172.21.15.104:0/4230779409 >> 172.21.15.104:0/4230779409 conn(0x7f4f100b1610 msgr2=0x7f4f100ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 -- 172.21.15.104:0/4230779409 shutdown_connections 2023-06-10T18:45:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.894+0000 7f4f16ef9700 1 -- 172.21.15.104:0/4230779409 wait complete. 2023-06-10T18:45:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.895+0000 7f4f16ef9700 1 Processor -- start 2023-06-10T18:45:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.895+0000 7f4f16ef9700 1 -- start start 2023-06-10T18:45:56.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 0x7f4f1014cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 0x7f4f10159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1014d700 con 0x7f4f1006ca40 2023-06-10T18:45:56.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1014d150 con 0x7f4f1006b180 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f16ef9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100b2ae0 con 0x7f4f10158c50 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57628/0 (socket says 172.21.15.104:57628) 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 -- 172.21.15.104:0/3068141419 learned_addr learned my addr 172.21.15.104:0/3068141419 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f14c95700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 0x7f4f1014cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f15496700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 0x7f4f10159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 -- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 msgr2=0x7f4f10159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 0x7f4f10159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 -- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 msgr2=0x7f4f1014cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 0x7f4f1014cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f0ffff700 1 -- 172.21.15.104:0/3068141419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04009550 con 0x7f4f1006ca40 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f14c95700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 0x7f4f1014cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.896+0000 7f4f15496700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 0x7f4f10159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f0ffff700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4efc010bf0 tx=0x7f4efc00e360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc009710 con 0x7f4f1006ca40 2023-06-10T18:45:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00eea0 con 0x7f4f1006ca40 2023-06-10T18:45:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f100b2dc0 con 0x7f4f1006ca40 2023-06-10T18:45:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc011650 con 0x7f4f1006ca40 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.897+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f100b3380 con 0x7f4f1006ca40 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.898+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f10074050 con 0x7f4f1006ca40 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.898+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4efc0117b0 con 0x7f4f1006ca40 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.899+0000 7f4f0d7fa700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 0x7f4ef406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.899+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4efc08b940 con 0x7f4f1006ca40 2023-06-10T18:45:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.899+0000 7f4f14c95700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 0x7f4ef406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.899+0000 7f4f14c95700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 0x7f4ef406d190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f04014040 tx=0x7f4f04008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:56.903+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4efc056900 con 0x7f4f1006ca40 2023-06-10T18:45:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.092+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f10072dd0 con 0x7f4f1006ca40 2023-06-10T18:45:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.093+0000 7f4f0d7fa700 1 -- 172.21.15.104:0/3068141419 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7f4efc059f20 con 0x7f4f1006ca40 2023-06-10T18:45:57.099 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:57.100 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 msgr2=0x7f4ef406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 0x7f4ef406d190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f04014040 tx=0x7f4f04008030 comp rx=0 tx=0).stop 2023-06-10T18:45:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 msgr2=0x7f4f10159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4efc010bf0 tx=0x7f4efc00e360 comp rx=0 tx=0).stop 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 shutdown_connections 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ef406ace0 0x7f4ef406d190 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f1006b180 0x7f4f1014cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f10158c50 0x7f4f10159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 --2- 172.21.15.104:0/3068141419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1006ca40 0x7f4f10159710 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 >> 172.21.15.104:0/3068141419 conn(0x7f4f100b1610 msgr2=0x7f4f101582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 shutdown_connections 2023-06-10T18:45:57.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.095+0000 7f4f16ef9700 1 -- 172.21.15.104:0/3068141419 wait complete. 2023-06-10T18:45:57.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:45:39.741633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:45:40.245539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:45:57.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:57.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.231+0000 7f5e9c814700 1 Processor -- start 2023-06-10T18:45:57.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.231+0000 7f5e9c814700 1 -- start start 2023-06-10T18:45:57.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9c814700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e980b1780 0x7f5e9806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9c814700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5e9806ea70 0x7f5e9806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9c814700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980ae7e0 con 0x7f5e9815c6f0 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9c814700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9806df90 con 0x7f5e980b1780 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9c814700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9806e270 con 0x7f5e9806ea70 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e980b1780 0x7f5e9806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e980b1780 0x7f5e9806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57638/0 (socket says 172.21.15.104:57638) 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 -- 172.21.15.104:0/3041854142 learned_addr learned my addr 172.21.15.104:0/3041854142 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e96d9d700 1 --2- 172.21.15.104:0/3041854142 >> v2:172.21.15.114:3300/0 conn(0x7f5e9806ea70 0x7f5e9806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 -- 172.21.15.104:0/3041854142 >> v2:172.21.15.114:3300/0 conn(0x7f5e9806ea70 msgr2=0x7f5e9806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 --2- 172.21.15.104:0/3041854142 >> v2:172.21.15.114:3300/0 conn(0x7f5e9806ea70 0x7f5e9806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 -- 172.21.15.104:0/3041854142 >> v1:172.21.15.114:6789/0 conn(0x7f5e9815c6f0 legacy=0x7f5e980b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 -- 172.21.15.104:0/3041854142 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9806e550 con 0x7f5e980b1780 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e95d9b700 1 --2- 172.21.15.104:0/3041854142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e9806cde0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c2fc0 tx=0x7f5e88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c36c19b8d1a216 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.232+0000 7f5e9559a700 1 -- 172.21.15.104:0/3041854142 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88004230 con 0x7f5e980b1780 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9559a700 1 -- 172.21.15.104:0/3041854142 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e88004390 con 0x7f5e980b1780 2023-06-10T18:45:57.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9559a700 1 -- 172.21.15.104:0/3041854142 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880036a0 con 0x7f5e980b1780 2023-06-10T18:45:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 -- 172.21.15.104:0/3041854142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 msgr2=0x7f5e9806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 --2- 172.21.15.104:0/3041854142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e9806cde0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c2fc0 tx=0x7f5e88009bd0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 -- 172.21.15.104:0/3041854142 shutdown_connections 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 --2- 172.21.15.104:0/3041854142 >> v2:172.21.15.114:3300/0 conn(0x7f5e9806ea70 0x7f5e9806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 --2- 172.21.15.104:0/3041854142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e9806cde0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 -- 172.21.15.104:0/3041854142 >> 172.21.15.104:0/3041854142 conn(0x7f5e9806b8a0 msgr2=0x7f5e980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 -- 172.21.15.104:0/3041854142 shutdown_connections 2023-06-10T18:45:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.233+0000 7f5e9c814700 1 -- 172.21.15.104:0/3041854142 wait complete. 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.234+0000 7f5e9c814700 1 Processor -- start 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.234+0000 7f5e9c814700 1 -- start start 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e9806ea70 0x7f5e980b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e9815c6f0 0x7f5e981ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980c6630 con 0x7f5e980b1780 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980b4820 con 0x7f5e9806ea70 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e9c814700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980b2a00 con 0x7f5e9815c6f0 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e95d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e95d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57642/0 (socket says 172.21.15.104:57642) 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.235+0000 7f5e95d9b700 1 -- 172.21.15.104:0/246388624 learned_addr learned my addr 172.21.15.104:0/246388624 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e9659c700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e9806ea70 0x7f5e980b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 -- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e9815c6f0 msgr2=0x7f5e981ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e9815c6f0 0x7f5e981ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 -- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e9806ea70 msgr2=0x7f5e980b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e9806ea70 0x7f5e980b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 -- 172.21.15.104:0/246388624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88009550 con 0x7f5e980b1780 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e95d9b700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009c10 tx=0x7f5e88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88003d80 con 0x7f5e980b1780 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8801f070 con 0x7f5e980b1780 2023-06-10T18:45:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e980b2ce0 con 0x7f5e980b1780 2023-06-10T18:45:57.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88011850 con 0x7f5e980b1780 2023-06-10T18:45:57.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.236+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e980b32a0 con 0x7f5e980b1780 2023-06-10T18:45:57.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.237+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e980c67a0 con 0x7f5e980b1780 2023-06-10T18:45:57.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.237+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5e88021030 con 0x7f5e980b1780 2023-06-10T18:45:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.238+0000 7f5e86ffd700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 0x7f5e7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.238+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f5e8808ab70 con 0x7f5e980b1780 2023-06-10T18:45:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.238+0000 7f5e9659c700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 0x7f5e7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.239+0000 7f5e9659c700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 0x7f5e7806d190 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c9cb0 tx=0x7f5e8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.242+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5e980c67a0 con 0x7f5e980b1780 2023-06-10T18:45:57.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.428+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e98066f10 con 0x7f5e980b1780 2023-06-10T18:45:57.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.428+0000 7f5e86ffd700 1 -- 172.21.15.104:0/246388624 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e88055b30 con 0x7f5e980b1780 2023-06-10T18:45:57.434 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:57.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 msgr2=0x7f5e7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 0x7f5e7806d190 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c9cb0 tx=0x7f5e8c008040 comp rx=0 tx=0).stop 2023-06-10T18:45:57.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 msgr2=0x7f5e980b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009c10 tx=0x7f5e88004750 comp rx=0 tx=0).stop 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 shutdown_connections 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e7806ace0 0x7f5e7806d190 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e9806ea70 0x7f5e980b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e9815c6f0 0x7f5e981ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 --2- 172.21.15.104:0/246388624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980b1780 0x7f5e980b42e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 >> 172.21.15.104:0/246388624 conn(0x7f5e9806b8a0 msgr2=0x7f5e98158e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 shutdown_connections 2023-06-10T18:45:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.430+0000 7f5e9c814700 1 -- 172.21.15.104:0/246388624 wait complete. 2023-06-10T18:45:57.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.566+0000 7fe2f9476700 1 Processor -- start 2023-06-10T18:45:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.566+0000 7fe2f9476700 1 -- start start 2023-06-10T18:45:57.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f9476700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe2f40d04f0 0x7fe2f40d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f9476700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40af530 con 0x7fe2f40d25c0 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f9476700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40af810 con 0x7fe2f40b05d0 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f9476700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f40afaf0 con 0x7fe2f40d04f0 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f27fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe2f40d25c0 0x7fe2f40d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55476/0 (socket says 172.21.15.104:55476) 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f27fc700 1 -- 172.21.15.104:0/1189772387 learned_addr learned my addr 172.21.15.104:0/1189772387 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/1189772387 >> v2:172.21.15.114:3300/0 conn(0x7fe2f40d04f0 0x7fe2f40d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293385761 0 0) 0x7fe2f40af530 con 0x7fe2f40d25c0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.567+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc0035b0 con 0x7fe2f40d25c0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894992078 0 0) 0x7fe2f40af810 con 0x7fe2f40b05d0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f40af530 con 0x7fe2f40b05d0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335545675 0 0) 0x7fe2dc0035b0 con 0x7fe2f40d25c0 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 >> v2:172.21.15.114:3300/0 conn(0x7fe2f40d04f0 msgr2=0x7fe2f40d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 --2- 172.21.15.104:0/1189772387 >> v2:172.21.15.114:3300/0 conn(0x7fe2f40d04f0 0x7fe2f40d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 >> v1:172.21.15.114:6789/0 conn(0x7fe2f40b05d0 legacy=0x7fe2f40b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40afdd0 con 0x7fe2f40d25c0 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fe2e8001670 con 0x7fe2f40d25c0 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/1189772387 >> v2:172.21.15.114:3300/0 conn(0x7fe2f40d04f0 0x7fe2f40d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 >> v1:172.21.15.104:6789/0 conn(0x7fe2f40d25c0 legacy=0x7fe2f40d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2dc003b90 0x7fe2dc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40afaf0 con 0x7fe2dc005200 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c110 con 0x7fe2dc004850 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c2d0 con 0x7fe2dc003b90 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f27fc700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2dc003b90 0x7fe2dc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 -- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2dc003b90 msgr2=0x7fe2dc003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2dc003b90 0x7fe2dc003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 -- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2dc004850 msgr2=0x7fe2dc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 -- 172.21.15.104:0/1189772387 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0035b0 con 0x7fe2dc005200 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 -- 172.21.15.104:0/1189772387 reap_dead start 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2dc00ba10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec002230 tx=0x7fe2dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aee0bca98d8a6f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec001f30 con 0x7fe2dc005200 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec00ce80 con 0x7fe2dc005200 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.568+0000 7fe2f1ffb700 1 -- 172.21.15.104:0/1189772387 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec003c40 con 0x7fe2dc005200 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 -- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 msgr2=0x7fe2dc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2dc00ba10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec002230 tx=0x7fe2dc005880 comp rx=0 tx=0).stop 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 -- 172.21.15.104:0/1189772387 shutdown_connections 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 --2- 172.21.15.104:0/1189772387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2dc00ba10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 -- 172.21.15.104:0/1189772387 >> 172.21.15.104:0/1189772387 conn(0x7fe2f4071ec0 msgr2=0x7fe2f409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 -- 172.21.15.104:0/1189772387 shutdown_connections 2023-06-10T18:45:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.569+0000 7fe2f9476700 1 -- 172.21.15.104:0/1189772387 wait complete. 2023-06-10T18:45:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.570+0000 7fe2f9476700 1 Processor -- start 2023-06-10T18:45:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.570+0000 7fe2f9476700 1 -- start start 2023-06-10T18:45:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 0x7fe2f40b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 0x7fe2f40b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40b87d0 con 0x7fe2dc005200 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40b7290 con 0x7fe2f40b5750 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f9476700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4153d30 con 0x7fe2f40b7e20 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57662/0 (socket says 172.21.15.104:57662) 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f2ffd700 1 -- 172.21.15.104:0/610436907 learned_addr learned my addr 172.21.15.104:0/610436907 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f27fc700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 0x7fe2f40b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.571+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 0x7fe2f40b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 -- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 msgr2=0x7fe2f40b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 0x7fe2f40b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 -- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 msgr2=0x7fe2f40b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 0x7fe2f40b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 -- 172.21.15.104:0/610436907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40b61f0 con 0x7fe2dc005200 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f27fc700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 0x7fe2f40b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f37fe700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 0x7fe2f40b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f2ffd700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4011ca0 tx=0x7fe2e400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400fb30 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e400fc90 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f40b64d0 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4004020 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.572+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f40b6a90 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.573+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f4067010 con 0x7fe2dc005200 2023-06-10T18:45:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.573+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe2e4004180 con 0x7fe2dc005200 2023-06-10T18:45:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.574+0000 7fe2db7fe700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 0x7fe2e006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.574+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fe2e4089ba0 con 0x7fe2dc005200 2023-06-10T18:45:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.574+0000 7fe2f27fc700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 0x7fe2e006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.575+0000 7fe2f27fc700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 0x7fe2e006d1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40cc100 tx=0x7fe2e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.578+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe2e4054a40 con 0x7fe2dc005200 2023-06-10T18:45:57.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.763+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe2f4074050 con 0x7fe2dc005200 2023-06-10T18:45:57.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.764+0000 7fe2db7fe700 1 -- 172.21.15.104:0/610436907 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe2e4022030 con 0x7fe2dc005200 2023-06-10T18:45:57.770 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:57.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 msgr2=0x7fe2e006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 0x7fe2e006d1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40cc100 tx=0x7fe2e8009040 comp rx=0 tx=0).stop 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 msgr2=0x7fe2f41537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4011ca0 tx=0x7fe2e400f520 comp rx=0 tx=0).stop 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 shutdown_connections 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2f40b7e20 0x7fe2f40b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2dc005200 0x7fe2f41537f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2e006ad10 0x7fe2e006d1c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 --2- 172.21.15.104:0/610436907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2f40b5750 0x7fe2f40b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 >> 172.21.15.104:0/610436907 conn(0x7fe2f4071ec0 msgr2=0x7fe2f40b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.765+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 shutdown_connections 2023-06-10T18:45:57.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.766+0000 7fe2f9476700 1 -- 172.21.15.104:0/610436907 wait complete. 2023-06-10T18:45:57.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:57.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.899+0000 7fd1dd6bd700 1 Processor -- start 2023-06-10T18:45:57.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.899+0000 7fd1dd6bd700 1 -- start start 2023-06-10T18:45:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1dd6bd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1d8069e70 0x7fd1d806a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1dd6bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1d806e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1dd6bd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d806e690 con 0x7fd1d815c500 2023-06-10T18:45:57.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1dd6bd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d80aef20 con 0x7fd1d8069e70 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1dd6bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d815c8d0 con 0x7fd1d806dcd0 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1d6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd1d815c500 0x7fd1d80cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55496/0 (socket says 172.21.15.104:55496) 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1d6ffd700 1 -- 172.21.15.104:0/3613354970 learned_addr learned my addr 172.21.15.104:0/3613354970 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1d67fc700 1 --2- 172.21.15.104:0/3613354970 >> v2:172.21.15.114:3300/0 conn(0x7fd1d8069e70 0x7fd1d806a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.900+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1d806e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394452304 0 0) 0x7fd1d806e690 con 0x7fd1d815c500 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c00035b0 con 0x7fd1d815c500 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1044986883 0 0) 0x7fd1c00035b0 con 0x7fd1d815c500 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 msgr2=0x7fd1d806e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1d806e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 >> v2:172.21.15.114:3300/0 conn(0x7fd1d8069e70 msgr2=0x7fd1d806a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 --2- 172.21.15.104:0/3613354970 >> v2:172.21.15.114:3300/0 conn(0x7fd1d8069e70 0x7fd1d806a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d806abd0 con 0x7fd1d815c500 2023-06-10T18:45:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd1cc0017f0 con 0x7fd1d815c500 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 >> v1:172.21.15.104:6789/0 conn(0x7fd1d815c500 legacy=0x7fd1d80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1c0004d90 0x7fd1c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d815c8d0 con 0x7fd1c0003a00 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c000ba10 con 0x7fd1c0004d90 2023-06-10T18:45:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c000bb50 con 0x7fd1d806dcd0 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d67fc700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1c0004d90 0x7fd1c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 -- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 msgr2=0x7fd1c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 -- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1c0004d90 msgr2=0x7fd1c0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1c0004d90 0x7fd1c0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 -- 172.21.15.104:0/3613354970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00035b0 con 0x7fd1c0003a00 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d67fc700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1d806dcd0 0x7fd1c0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1c0004d90 0x7fd1c0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c240 tx=0x7fd1c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35ec7f339ba9f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00cd80 con 0x7fd1c0003a00 2023-06-10T18:45:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc007b80 con 0x7fd1c0003a00 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.901+0000 7fd1d5ffb700 1 -- 172.21.15.104:0/3613354970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc0072d0 con 0x7fd1c0003a00 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/3613354970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1c0003a00 msgr2=0x7fd1c0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/3613354970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1c0003a00 0x7fd1c0003e10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c240 tx=0x7fd1c0005880 comp rx=0 tx=0).stop 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1d77fe700 1 -- 172.21.15.104:0/3613354970 reap_dead start 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/3613354970 shutdown_connections 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/3613354970 >> 172.21.15.104:0/3613354970 conn(0x7fd1d806b8a0 msgr2=0x7fd1d80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/3613354970 shutdown_connections 2023-06-10T18:45:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.902+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/3613354970 wait complete. 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.903+0000 7fd1dd6bd700 1 Processor -- start 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.903+0000 7fd1dd6bd700 1 -- start start 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 0x7fd1d80b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 0x7fd1d814d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d814aad0 con 0x7fd1d814a0f0 2023-06-10T18:45:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d81f18f0 con 0x7fd1d80c5af0 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1dd6bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80aee90 con 0x7fd1c0003a00 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57682/0 (socket says 172.21.15.104:57682) 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 -- 172.21.15.104:0/621596303 learned_addr learned my addr 172.21.15.104:0/621596303 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 0x7fd1d80b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d67fc700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 0x7fd1d814d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 -- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 msgr2=0x7fd1d80b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 0x7fd1d80b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 -- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 msgr2=0x7fd1d814d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 0x7fd1d814d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.904+0000 7fd1d77fe700 1 -- 172.21.15.104:0/621596303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d80b3f40 con 0x7fd1d814a0f0 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1d67fc700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 0x7fd1d814d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 0x7fd1d80b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1d77fe700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80b2200 tx=0x7fd1d00104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d001b070 con 0x7fd1d814a0f0 2023-06-10T18:45:57.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d000f040 con 0x7fd1d814a0f0 2023-06-10T18:45:57.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80b4220 con 0x7fd1d814a0f0 2023-06-10T18:45:57.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000e050 con 0x7fd1d814a0f0 2023-06-10T18:45:57.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.905+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d814e390 con 0x7fd1d814a0f0 2023-06-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.906+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d80b4410 con 0x7fd1d814a0f0 2023-06-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.906+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd1d0015440 con 0x7fd1d814a0f0 2023-06-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.907+0000 7fd1bf7fe700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.907+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd1d0089740 con 0x7fd1d814a0f0 2023-06-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.907+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.907+0000 7fd1d6ffd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 0x7fd1c406d190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc000ef0 tx=0x7fd1cc007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:57.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:57.911+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd1d0058240 con 0x7fd1d814a0f0 2023-06-10T18:45:58.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.096+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1d8066f70 con 0x7fd1d814a0f0 2023-06-10T18:45:58.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.096+0000 7fd1bf7fe700 1 -- 172.21.15.104:0/621596303 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1d8066f70 con 0x7fd1d814a0f0 2023-06-10T18:45:58.102 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:45:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 msgr2=0x7fd1c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 0x7fd1c406d190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc000ef0 tx=0x7fd1cc007000 comp rx=0 tx=0).stop 2023-06-10T18:45:58.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 msgr2=0x7fd1d814a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80b2200 tx=0x7fd1d00104a0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 shutdown_connections 2023-06-10T18:45:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d814a0f0 0x7fd1d814a590 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1c0003a00 0x7fd1d80b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 --2- 172.21.15.104:0/621596303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1d80c5af0 0x7fd1d814d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.098+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 >> 172.21.15.104:0/621596303 conn(0x7fd1d806b8a0 msgr2=0x7fd1d80ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.099+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 shutdown_connections 2023-06-10T18:45:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.099+0000 7fd1dd6bd700 1 -- 172.21.15.104:0/621596303 wait complete. 2023-06-10T18:45:58.118 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:45:58.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.236+0000 7f4b91328700 1 Processor -- start 2023-06-10T18:45:58.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.236+0000 7f4b91328700 1 -- start start 2023-06-10T18:45:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b91328700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4b8c0d04f0 0x7f4b8c0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b91328700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 0x7f4b8c0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b91328700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0af530 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b91328700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c0af810 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b91328700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0afaf0 con 0x7f4b8c0b05d0 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4b8c0d04f0 0x7f4b8c0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8a7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4b8c0d25c0 0x7f4b8c0d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55516/0 (socket says 172.21.15.104:55516) 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8a7fc700 1 -- 172.21.15.104:0/3800670570 learned_addr learned my addr 172.21.15.104:0/3800670570 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8b7fe700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 0x7f4b8c0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 -- 172.21.15.104:0/3800670570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 msgr2=0x7f4b8c0b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 0x7f4b8c0b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 -- 172.21.15.104:0/3800670570 >> v1:172.21.15.104:6789/0 conn(0x7f4b8c0d25c0 legacy=0x7f4b8c0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 -- 172.21.15.104:0/3800670570 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0afdd0 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8b7fe700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 0x7f4b8c0b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b89ffb700 1 -- 172.21.15.104:0/3800670570 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132483191 0 0) 0x7f4b8c0af530 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.237+0000 7f4b8affd700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d04f0 0x7f4b8c0d34d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009820 tx=0x7f4b80009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743d62bf83484955 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b89ffb700 1 -- 172.21.15.104:0/3800670570 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80005e30 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b89ffb700 1 -- 172.21.15.104:0/3800670570 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80006030 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b89ffb700 1 -- 172.21.15.104:0/3800670570 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b800063e0 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 -- 172.21.15.104:0/3800670570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d04f0 msgr2=0x7f4b8c0d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d04f0 0x7f4b8c0d34d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009820 tx=0x7f4b80009b30 comp rx=0 tx=0).stop 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 -- 172.21.15.104:0/3800670570 shutdown_connections 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d04f0 0x7f4b8c0d34d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 --2- 172.21.15.104:0/3800670570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0b05d0 0x7f4b8c0b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 -- 172.21.15.104:0/3800670570 >> 172.21.15.104:0/3800670570 conn(0x7f4b8c071ec0 msgr2=0x7f4b8c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 -- 172.21.15.104:0/3800670570 shutdown_connections 2023-06-10T18:45:58.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.238+0000 7f4b91328700 1 -- 172.21.15.104:0/3800670570 wait complete. 2023-06-10T18:45:58.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.239+0000 7f4b91328700 1 Processor -- start 2023-06-10T18:45:58.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 -- start start 2023-06-10T18:45:58.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c0b05d0 0x7f4b8c1e6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0d04f0 0x7f4b8c14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1e51e0 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1e6640 con 0x7f4b8c0b05d0 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b91328700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1e6870 con 0x7f4b8c0d04f0 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57698/0 (socket says 172.21.15.104:57698) 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b8b7fe700 1 -- 172.21.15.104:0/1244701499 learned_addr learned my addr 172.21.15.104:0/1244701499 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:58.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.240+0000 7f4b8affd700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c0b05d0 0x7f4b8c1e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8a7fc700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0d04f0 0x7f4b8c14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 -- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0d04f0 msgr2=0x7f4b8c14f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0d04f0 0x7f4b8c14f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 -- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c0b05d0 msgr2=0x7f4b8c1e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c0b05d0 0x7f4b8c1e6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 -- 172.21.15.104:0/1244701499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80009430 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b8b7fe700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009fd0 tx=0x7f4b8400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8400dde0 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8400f040 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c1e4420 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84003c60 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.241+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c1e49e0 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.242+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c067010 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.242+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4b84004030 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.243+0000 7f4b737fe700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 0x7f4b7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.243+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4b84089600 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.243+0000 7f4b8affd700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 0x7f4b7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.243+0000 7f4b8affd700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 0x7f4b7806d190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b80013040 tx=0x7f4b80008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.247+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4b840545c0 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.433+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4b8c074050 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.433+0000 7f4b737fe700 1 -- 172.21.15.104:0/1244701499 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4b84057be0 con 0x7f4b8c0d25c0 2023-06-10T18:45:58.439 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:45:58.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 msgr2=0x7f4b7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 0x7f4b7806d190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b80013040 tx=0x7f4b80008070 comp rx=0 tx=0).stop 2023-06-10T18:45:58.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 msgr2=0x7f4b8c1f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009fd0 tx=0x7f4b8400d3b0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 shutdown_connections 2023-06-10T18:45:58.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b7806ace0 0x7f4b7806d190 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c0b05d0 0x7f4b8c1e6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b8c0d25c0 0x7f4b8c1f52c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 --2- 172.21.15.104:0/1244701499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b8c0d04f0 0x7f4b8c14f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 >> 172.21.15.104:0/1244701499 conn(0x7f4b8c071ec0 msgr2=0x7f4b8c0cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 shutdown_connections 2023-06-10T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.435+0000 7f4b91328700 1 -- 172.21.15.104:0/1244701499 wait complete. 2023-06-10T18:45:58.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:45:58.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.568+0000 7fc981b78700 1 Processor -- start 2023-06-10T18:45:58.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.568+0000 7fc981b78700 1 -- start start 2023-06-10T18:45:58.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc981b78700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc97c06e0e0 0x7fc97c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc981b78700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc981b78700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c06d290 con 0x7fc97c0ca100 2023-06-10T18:45:58.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc981b78700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c071640 con 0x7fc97c06e4b0 2023-06-10T18:45:58.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc981b78700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97c0aef00 con 0x7fc97c06e0e0 2023-06-10T18:45:58.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57702/0 (socket says 172.21.15.104:57702) 2023-06-10T18:45:58.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 -- 172.21.15.104:0/577976831 learned_addr learned my addr 172.21.15.104:0/577976831 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:58.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/577976831 >> v2:172.21.15.114:3300/0 conn(0x7fc97c06e0e0 0x7fc97c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 -- 172.21.15.104:0/577976831 >> v2:172.21.15.114:3300/0 conn(0x7fc97c06e0e0 msgr2=0x7fc97c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:58.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 --2- 172.21.15.104:0/577976831 >> v2:172.21.15.114:3300/0 conn(0x7fc97c06e0e0 0x7fc97c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 -- 172.21.15.104:0/577976831 >> v1:172.21.15.114:6789/0 conn(0x7fc97c0ca100 legacy=0x7fc97c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:58.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 -- 172.21.15.104:0/577976831 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c06d4c0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97affd700 1 --2- 172.21.15.104:0/577976831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc97c06af50 tx=0x7fc96c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee506cb70ae4d87c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97a7fc700 1 -- 172.21.15.104:0/577976831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c004230 con 0x7fc97c06e4b0 2023-06-10T18:45:58.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97a7fc700 1 -- 172.21.15.104:0/577976831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c004390 con 0x7fc97c06e4b0 2023-06-10T18:45:58.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.569+0000 7fc97a7fc700 1 -- 172.21.15.104:0/577976831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0036a0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 -- 172.21.15.104:0/577976831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 msgr2=0x7fc97c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 --2- 172.21.15.104:0/577976831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc97c06af50 tx=0x7fc96c009c00 comp rx=0 tx=0).stop 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 -- 172.21.15.104:0/577976831 shutdown_connections 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 --2- 172.21.15.104:0/577976831 >> v2:172.21.15.114:3300/0 conn(0x7fc97c06e0e0 0x7fc97c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 --2- 172.21.15.104:0/577976831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0c9bc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 -- 172.21.15.104:0/577976831 >> 172.21.15.104:0/577976831 conn(0x7fc97c06b8a0 msgr2=0x7fc97c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 -- 172.21.15.104:0/577976831 shutdown_connections 2023-06-10T18:45:58.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.570+0000 7fc981b78700 1 -- 172.21.15.104:0/577976831 wait complete. 2023-06-10T18:45:58.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.571+0000 7fc981b78700 1 Processor -- start 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.571+0000 7fc981b78700 1 -- start start 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 0x7fc97c1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 0x7fc97c0b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0b3da0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1f13f0 con 0x7fc97c06e0e0 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc981b78700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0c8110 con 0x7fc97c0ca100 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57708/0 (socket says 172.21.15.104:57708) 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 -- 172.21.15.104:0/1736393793 learned_addr learned my addr 172.21.15.104:0/1736393793 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 0x7fc97c1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 0x7fc97c0b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 -- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 msgr2=0x7fc97c0b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 0x7fc97c0b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 -- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 msgr2=0x7fc97c1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 0x7fc97c1f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97affd700 1 -- 172.21.15.104:0/1736393793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c009550 con 0x7fc97c06e4b0 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 0x7fc97c1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.572+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 0x7fc97c0b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:58.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc97affd700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009870 tx=0x7fc96c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c003dc0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c01f070 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c0af0e0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c011850 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.573+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c0af6a0 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.574+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c074050 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.574+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc96c021030 con 0x7fc97c06e4b0 2023-06-10T18:45:58.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.574+0000 7fc963fff700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.575+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc96c08ac50 con 0x7fc97c06e4b0 2023-06-10T18:45:58.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.575+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.575+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc970009a00 tx=0x7fc970008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.579+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc96c090020 con 0x7fc97c06e4b0 2023-06-10T18:45:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.781+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc97c074050 con 0x7fc97c06e4b0 2023-06-10T18:45:58.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.782+0000 7fc963fff700 1 -- 172.21.15.104:0/1736393793 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc96c055d30 con 0x7fc97c06e4b0 2023-06-10T18:45:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 msgr2=0x7fc96806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc970009a00 tx=0x7fc970008040 comp rx=0 tx=0).stop 2023-06-10T18:45:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 msgr2=0x7fc97c0b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009870 tx=0x7fc96c004750 comp rx=0 tx=0).stop 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 shutdown_connections 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c06e0e0 0x7fc97c1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0ca100 0x7fc97c0b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 --2- 172.21.15.104:0/1736393793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c06e4b0 0x7fc97c0b3320 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 >> 172.21.15.104:0/1736393793 conn(0x7fc97c06b8a0 msgr2=0x7fc97c0ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 shutdown_connections 2023-06-10T18:45:58.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.784+0000 7fc981b78700 1 -- 172.21.15.104:0/1736393793 wait complete. 2023-06-10T18:45:58.793 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:45:58.801 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:45:58.802 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:45:58.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:45:58.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.972+0000 7fb97c6c7700 1 Processor -- start 2023-06-10T18:45:58.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.972+0000 7fb97c6c7700 1 -- start start 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97c6c7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb97415c7a0 0x7fb9740cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97c6c7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9740701a0 0x7fb9740705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97c6c7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb974070ae0 con 0x7fb9741567f0 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97c6c7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97415cca0 con 0x7fb9740701a0 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97c6c7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97415cf80 con 0x7fb97415c7a0 2023-06-10T18:45:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97ac64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9740701a0 0x7fb9740705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97ac64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb9740701a0 0x7fb9740705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57716/0 (socket says 172.21.15.104:57716) 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.973+0000 7fb97ac64700 1 -- 172.21.15.104:0/3883596367 learned_addr learned my addr 172.21.15.104:0/3883596367 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb979c62700 1 --2- 172.21.15.104:0/3883596367 >> v2:172.21.15.114:3300/0 conn(0x7fb97415c7a0 0x7fb9740cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97ac64700 1 -- 172.21.15.104:0/3883596367 >> v2:172.21.15.114:3300/0 conn(0x7fb97415c7a0 msgr2=0x7fb9740cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97ac64700 1 --2- 172.21.15.104:0/3883596367 >> v2:172.21.15.114:3300/0 conn(0x7fb97415c7a0 0x7fb9740cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97ac64700 1 -- 172.21.15.104:0/3883596367 >> v1:172.21.15.114:6789/0 conn(0x7fb9741567f0 legacy=0x7fb9740cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97ac64700 1 -- 172.21.15.104:0/3883596367 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97415d260 con 0x7fb9740701a0 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97ac64700 1 --2- 172.21.15.104:0/3883596367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9740701a0 0x7fb9740705a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb9740afb90 tx=0x7fb96c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec581b7767da62 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb979461700 1 -- 172.21.15.104:0/3883596367 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c00e070 con 0x7fb9740701a0 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb979461700 1 -- 172.21.15.104:0/3883596367 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c003a20 con 0x7fb9740701a0 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb979461700 1 -- 172.21.15.104:0/3883596367 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c003dd0 con 0x7fb9740701a0 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3883596367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9740701a0 msgr2=0x7fb9740705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.974+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3883596367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9740701a0 0x7fb9740705a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb9740afb90 tx=0x7fb96c00b330 comp rx=0 tx=0).stop 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3883596367 shutdown_connections 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3883596367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9740701a0 0x7fb9740705a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3883596367 >> v2:172.21.15.114:3300/0 conn(0x7fb97415c7a0 0x7fb9740cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3883596367 >> 172.21.15.104:0/3883596367 conn(0x7fb97406e370 msgr2=0x7fb97406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3883596367 shutdown_connections 2023-06-10T18:45:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.975+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3883596367 wait complete. 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.976+0000 7fb97c6c7700 1 Processor -- start 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.976+0000 7fb97c6c7700 1 -- start start 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.976+0000 7fb97c6c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 0x7fb974151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.976+0000 7fb97c6c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 0x7fb9740c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740b3570 con 0x7fb9741567f0 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740b43d0 con 0x7fb9740701a0 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740b37c0 con 0x7fb97415c7a0 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57722/0 (socket says 172.21.15.104:57722) 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 -- 172.21.15.104:0/3926492774 learned_addr learned my addr 172.21.15.104:0/3926492774 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97a463700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 0x7fb974151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97ac64700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 0x7fb9740c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 -- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 msgr2=0x7fb9740c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 0x7fb9740c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 -- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 msgr2=0x7fb974151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 0x7fb974151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 -- 172.21.15.104:0/3926492774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c009b30 con 0x7fb9741567f0 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97a463700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 0x7fb974151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97ac64700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 0x7fb9740c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb979c62700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb97000d830 tx=0x7fb97000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700125f0 con 0x7fb9741567f0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb970012750 con 0x7fb9741567f0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9740b3aa0 con 0x7fb9741567f0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970012ae0 con 0x7fb9741567f0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.977+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9740b4060 con 0x7fb9741567f0 2023-06-10T18:45:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.978+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb974074050 con 0x7fb9741567f0 2023-06-10T18:45:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.979+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb970012c40 con 0x7fb9741567f0 2023-06-10T18:45:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.979+0000 7fb96affd700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 0x7fb95806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.979+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fb970089910 con 0x7fb9741567f0 2023-06-10T18:45:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.979+0000 7fb97a463700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 0x7fb95806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.980+0000 7fb97a463700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 0x7fb95806d190 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb964003fc0 tx=0x7fb964009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:58.983+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb9700548d0 con 0x7fb9741567f0 2023-06-10T18:45:59.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.171+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9740b66c0 con 0x7fb9741567f0 2023-06-10T18:45:59.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.172+0000 7fb96affd700 1 -- 172.21.15.104:0/3926492774 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7fb970057ef0 con 0x7fb9741567f0 2023-06-10T18:45:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:45:59.179 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:45:59.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 msgr2=0x7fb95806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 0x7fb95806d190 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb964003fc0 tx=0x7fb964009040 comp rx=0 tx=0).stop 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 msgr2=0x7fb9740b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb97000d830 tx=0x7fb97000dbf0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 shutdown_connections 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95806ace0 0x7fb95806d190 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9740701a0 0x7fb974151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97415c7a0 0x7fb9740c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 --2- 172.21.15.104:0/3926492774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9741567f0 0x7fb9740b4860 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 >> 172.21.15.104:0/3926492774 conn(0x7fb97406e370 msgr2=0x7fb9740cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 shutdown_connections 2023-06-10T18:45:59.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.174+0000 7fb97c6c7700 1 -- 172.21.15.104:0/3926492774 wait complete. 2023-06-10T18:45:59.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:45:39.741633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:45:40.245539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:45:59.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:45:59.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.308+0000 7f1ad1c48700 1 Processor -- start 2023-06-10T18:45:59.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.308+0000 7f1ad1c48700 1 -- start start 2023-06-10T18:45:59.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1ad1c48700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1acc15c6f0 0x7f1acc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1ad1c48700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1acc06ea70 0x7f1acc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1ad1c48700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc0ae7e0 con 0x7f1acc0aec00 2023-06-10T18:45:59.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1ad1c48700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc06df90 con 0x7f1acc06ea70 2023-06-10T18:45:59.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1ad1c48700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc06e270 con 0x7f1acc15c6f0 2023-06-10T18:45:59.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1acc06ea70 0x7f1acc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acbfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1acc0aec00 0x7f1acc0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55558/0 (socket says 172.21.15.104:55558) 2023-06-10T18:45:59.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acbfff700 1 -- 172.21.15.104:0/2379158660 learned_addr learned my addr 172.21.15.104:0/2379158660 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:59.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acb7fe700 1 --2- 172.21.15.104:0/2379158660 >> v2:172.21.15.114:3300/0 conn(0x7f1acc15c6f0 0x7f1acc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acaffd700 1 -- 172.21.15.104:0/2379158660 >> v2:172.21.15.114:3300/0 conn(0x7f1acc15c6f0 msgr2=0x7f1acc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:59.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acaffd700 1 --2- 172.21.15.104:0/2379158660 >> v2:172.21.15.114:3300/0 conn(0x7f1acc15c6f0 0x7f1acc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1acaffd700 1 -- 172.21.15.104:0/2379158660 >> v1:172.21.15.104:6789/0 conn(0x7f1acc0aec00 legacy=0x7f1acc0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.309+0000 7f1aca7fc700 1 -- 172.21.15.104:0/2379158660 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508149516 0 0) 0x7f1acc0ae7e0 con 0x7f1acc0aec00 2023-06-10T18:45:59.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1acaffd700 1 -- 172.21.15.104:0/2379158660 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc06e550 con 0x7f1acc06ea70 2023-06-10T18:45:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1acaffd700 1 --2- 172.21.15.104:0/2379158660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc06ea70 0x7f1acc0b15b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0d0660 tx=0x7f1abc00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e31c59f328525d3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1aca7fc700 1 -- 172.21.15.104:0/2379158660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00dc70 con 0x7f1acc06ea70 2023-06-10T18:45:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1aca7fc700 1 -- 172.21.15.104:0/2379158660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc00ddd0 con 0x7f1acc06ea70 2023-06-10T18:45:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1aca7fc700 1 -- 172.21.15.104:0/2379158660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc010690 con 0x7f1acc06ea70 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 -- 172.21.15.104:0/2379158660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc06ea70 msgr2=0x7f1acc0b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/2379158660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc06ea70 0x7f1acc0b15b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0d0660 tx=0x7f1abc00bdc0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 -- 172.21.15.104:0/2379158660 shutdown_connections 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/2379158660 >> v2:172.21.15.114:3300/0 conn(0x7f1acc15c6f0 0x7f1acc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/2379158660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc06ea70 0x7f1acc0b15b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 -- 172.21.15.104:0/2379158660 >> 172.21.15.104:0/2379158660 conn(0x7f1acc06b8a0 msgr2=0x7f1acc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 -- 172.21.15.104:0/2379158660 shutdown_connections 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.310+0000 7f1ad1c48700 1 -- 172.21.15.104:0/2379158660 wait complete. 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 Processor -- start 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 -- start start 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1acc06ea70 0x7f1acc1f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1acc0aec00 0x7f1acc154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1f1a40 con 0x7f1acc15c6f0 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1f1c30 con 0x7f1acc06ea70 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.312+0000 7f1ad1c48700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1f5a80 con 0x7f1acc0aec00 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57738/0 (socket says 172.21.15.104:57738) 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 -- 172.21.15.104:0/663081929 learned_addr learned my addr 172.21.15.104:0/663081929 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acaffd700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1acc0aec00 0x7f1acc154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acb7fe700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1acc06ea70 0x7f1acc1f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 -- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1acc0aec00 msgr2=0x7f1acc154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1acc0aec00 0x7f1acc154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 -- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1acc06ea70 msgr2=0x7f1acc1f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1acc06ea70 0x7f1acc1f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 -- 172.21.15.104:0/663081929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc00b740 con 0x7f1acc15c6f0 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1acbfff700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400cbe0 tx=0x7f1ac400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac400aaa0 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac400ac20 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc1f5d60 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40070d0 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.313+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc1f2950 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.314+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc1f5fa0 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.315+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1ac4007230 con 0x7f1acc15c6f0 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.315+0000 7f1ab3fff700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 0x7f1ab806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.315+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1ac4089360 con 0x7f1acc15c6f0 2023-06-10T18:45:59.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.315+0000 7f1acb7fe700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 0x7f1ab806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.316+0000 7f1acb7fe700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 0x7f1ab806d190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0ce370 tx=0x7f1ac0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.319+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1ac4054290 con 0x7f1acc15c6f0 2023-06-10T18:45:59.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.505+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1acc074050 con 0x7f1acc15c6f0 2023-06-10T18:45:59.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.506+0000 7f1ab3fff700 1 -- 172.21.15.104:0/663081929 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ac405d0a0 con 0x7f1acc15c6f0 2023-06-10T18:45:59.512 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:45:59.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.507+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 msgr2=0x7f1ab806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 0x7f1ab806d190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0ce370 tx=0x7f1ac0008040 comp rx=0 tx=0).stop 2023-06-10T18:45:59.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 msgr2=0x7f1acc1f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400cbe0 tx=0x7f1ac400cef0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 shutdown_connections 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ab806ace0 0x7f1ab806d190 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1acc06ea70 0x7f1acc1f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1acc15c6f0 0x7f1acc1f5030 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 --2- 172.21.15.104:0/663081929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1acc0aec00 0x7f1acc154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 >> 172.21.15.104:0/663081929 conn(0x7f1acc06b8a0 msgr2=0x7f1acc158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 shutdown_connections 2023-06-10T18:45:59.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.508+0000 7f1ad1c48700 1 -- 172.21.15.104:0/663081929 wait complete. 2023-06-10T18:45:59.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:45:59.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.644+0000 7f68de907700 1 Processor -- start 2023-06-10T18:45:59.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.644+0000 7f68de907700 1 -- start start 2023-06-10T18:45:59.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68de907700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68d806e3e0 0x7f68d806a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68de907700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d815c8d0 con 0x7f68d8069e70 2023-06-10T18:45:59.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68de907700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806abd0 con 0x7f68d815c500 2023-06-10T18:45:59.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68de907700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d80aef20 con 0x7f68d806e3e0 2023-06-10T18:45:59.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f68d8069e70 0x7f68d806dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55572/0 (socket says 172.21.15.104:55572) 2023-06-10T18:45:59.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d77fe700 1 -- 172.21.15.104:0/3837181370 learned_addr learned my addr 172.21.15.104:0/3837181370 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107333477 0 0) 0x7f68d815c8d0 con 0x7f68d8069e70 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c00035b0 con 0x7f68d8069e70 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68dcea4700 1 --2- 172.21.15.104:0/3837181370 >> v2:172.21.15.114:3300/0 conn(0x7f68d806e3e0 0x7f68d806a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 827757813 0 0) 0x7f68c00035b0 con 0x7f68d8069e70 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 >> v2:172.21.15.114:3300/0 conn(0x7f68d806e3e0 msgr2=0x7f68d806a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 --2- 172.21.15.104:0/3837181370 >> v2:172.21.15.114:3300/0 conn(0x7f68d806e3e0 0x7f68d806a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 >> v1:172.21.15.114:6789/0 conn(0x7f68d815c500 legacy=0x7f68d80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d806a8f0 con 0x7f68d8069e70 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f68c8001670 con 0x7f68d8069e70 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 >> v1:172.21.15.104:6789/0 conn(0x7f68d8069e70 legacy=0x7f68d806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c0003a00 0x7f68c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68c0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c0005200 0x7f68c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c000c160 con 0x7f68c0004850 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c000c2d0 con 0x7f68c0003a00 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.645+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c000c490 con 0x7f68c0005200 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68c0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d7fff700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c0003a00 0x7f68c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68dcea4700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c0005200 0x7f68c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 -- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c0005200 msgr2=0x7f68c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c0005200 0x7f68c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 -- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c0003a00 msgr2=0x7f68c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c0003a00 0x7f68c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68dcea4700 1 -- 172.21.15.104:0/3837181370 reap_dead start 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 -- 172.21.15.104:0/3837181370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c00035b0 con 0x7f68c0004850 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d77fe700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68c0004cc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f68c800c210 tx=0x7f68c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d85b8df364770e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8007c10 con 0x7f68c0004850 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c800cd50 con 0x7f68c0004850 2023-06-10T18:45:59.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.646+0000 7f68d6ffd700 1 -- 172.21.15.104:0/3837181370 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8005ba0 con 0x7f68c0004850 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 -- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 msgr2=0x7f68c0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68c0004cc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f68c800c210 tx=0x7f68c0005880 comp rx=0 tx=0).stop 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 -- 172.21.15.104:0/3837181370 shutdown_connections 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 --2- 172.21.15.104:0/3837181370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68c0004cc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 -- 172.21.15.104:0/3837181370 >> 172.21.15.104:0/3837181370 conn(0x7f68d806b8a0 msgr2=0x7f68d80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 -- 172.21.15.104:0/3837181370 shutdown_connections 2023-06-10T18:45:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.647+0000 7f68de907700 1 -- 172.21.15.104:0/3837181370 wait complete. 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.648+0000 7f68de907700 1 Processor -- start 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.648+0000 7f68de907700 1 -- start start 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d815a2f0 0x7f68d80b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 0x7f68d80af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8159250 con 0x7f68c0004850 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8159530 con 0x7f68d80b1a00 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8159810 con 0x7f68d815a2f0 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57754/0 (socket says 172.21.15.104:57754) 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 -- 172.21.15.104:0/461649037 learned_addr learned my addr 172.21.15.104:0/461649037 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68dcea4700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 0x7f68d80af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 -- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d815a2f0 msgr2=0x7f68d80b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d77fe700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d815a2f0 0x7f68d80b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d815a2f0 0x7f68d80b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 -- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 msgr2=0x7f68d80af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 0x7f68d80af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 -- 172.21.15.104:0/461649037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d8159af0 con 0x7f68c0004850 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68dcea4700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 0x7f68d80af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:45:59.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d7fff700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00f770 tx=0x7f68cc00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0052e0 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc005440 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0038f0 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d8159dd0 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.649+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d80b3230 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.650+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d8159f60 con 0x7f68c0004850 2023-06-10T18:45:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.651+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f68cc003a50 con 0x7f68c0004850 2023-06-10T18:45:59.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.651+0000 7f68d4ff9700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.651+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f68cc089380 con 0x7f68c0004850 2023-06-10T18:45:59.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.651+0000 7f68d77fe700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.652+0000 7f68d77fe700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 0x7f68c406d190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f68c8015040 tx=0x7f68c8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.655+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f68cc0542b0 con 0x7f68c0004850 2023-06-10T18:45:59.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.841+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f68d8065f70 con 0x7f68c0004850 2023-06-10T18:45:59.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.841+0000 7f68d4ff9700 1 -- 172.21.15.104:0/461649037 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7f68cc0578d0 con 0x7f68c0004850 2023-06-10T18:45:59.848 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:45:59.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 msgr2=0x7f68c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 0x7f68c406d190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f68c8015040 tx=0x7f68c8007000 comp rx=0 tx=0).stop 2023-06-10T18:45:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 msgr2=0x7f68d81f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00f770 tx=0x7f68cc00fb30 comp rx=0 tx=0).stop 2023-06-10T18:45:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 shutdown_connections 2023-06-10T18:45:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68d80b1a00 0x7f68d80af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c0004850 0x7f68d81f1860 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 --2- 172.21.15.104:0/461649037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d815a2f0 0x7f68d80b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 >> 172.21.15.104:0/461649037 conn(0x7f68d806b8a0 msgr2=0x7f68d80ace00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 shutdown_connections 2023-06-10T18:45:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.844+0000 7f68de907700 1 -- 172.21.15.104:0/461649037 wait complete. 2023-06-10T18:45:59.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:45:59.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.977+0000 7f37c73c0700 1 Processor -- start 2023-06-10T18:45:59.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.977+0000 7f37c73c0700 1 -- start start 2023-06-10T18:45:59.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c73c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f37c015c580 0x7f37c00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c73c0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 0x7f37c00ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c73c0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c006e250 con 0x7f37c006e6e0 2023-06-10T18:45:59.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c73c0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c015c950 con 0x7f37c015c580 2023-06-10T18:45:59.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c73c0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c00b1e80 con 0x7f37c006cde0 2023-06-10T18:45:59.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c515c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f37c015c580 0x7f37c00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c495b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f37c006e6e0 0x7f37c00b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55590/0 (socket says 172.21.15.104:55590) 2023-06-10T18:45:59.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.978+0000 7f37c495b700 1 -- 172.21.15.104:0/2081494050 learned_addr learned my addr 172.21.15.104:0/2081494050 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:45:59.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c595d700 1 --2- 172.21.15.104:0/2081494050 >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 0x7f37c00ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c515c700 1 -- 172.21.15.104:0/2081494050 >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 msgr2=0x7f37c00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:45:59.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c515c700 1 --2- 172.21.15.104:0/2081494050 >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 0x7f37c00ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c515c700 1 -- 172.21.15.104:0/2081494050 >> v1:172.21.15.104:6789/0 conn(0x7f37c006e6e0 legacy=0x7f37c00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37b7fff700 1 -- 172.21.15.104:0/2081494050 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395950364 0 0) 0x7f37c006e250 con 0x7f37c006e6e0 2023-06-10T18:45:59.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c515c700 1 -- 172.21.15.104:0/2081494050 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c006d920 con 0x7f37c015c580 2023-06-10T18:45:59.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c595d700 1 --2- 172.21.15.104:0/2081494050 >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 0x7f37c00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:59.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c515c700 1 --2- 172.21.15.104:0/2081494050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c00b1070 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f37c00cbc60 tx=0x7f37b0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d431bd82cb16de server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37b7fff700 1 -- 172.21.15.104:0/2081494050 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b0005e30 con 0x7f37c015c580 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37b7fff700 1 -- 172.21.15.104:0/2081494050 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b0006030 con 0x7f37c015c580 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37b7fff700 1 -- 172.21.15.104:0/2081494050 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00063c0 con 0x7f37c015c580 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c73c0700 1 -- 172.21.15.104:0/2081494050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 msgr2=0x7f37c00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.979+0000 7f37c73c0700 1 --2- 172.21.15.104:0/2081494050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c00b1070 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f37c00cbc60 tx=0x7f37b0009b00 comp rx=0 tx=0).stop 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 -- 172.21.15.104:0/2081494050 shutdown_connections 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 --2- 172.21.15.104:0/2081494050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c00b1070 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 --2- 172.21.15.104:0/2081494050 >> v2:172.21.15.114:3300/0 conn(0x7f37c006cde0 0x7f37c00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 -- 172.21.15.104:0/2081494050 >> 172.21.15.104:0/2081494050 conn(0x7f37c006b8a0 msgr2=0x7f37c00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:45:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 -- 172.21.15.104:0/2081494050 shutdown_connections 2023-06-10T18:45:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.980+0000 7f37c73c0700 1 -- 172.21.15.104:0/2081494050 wait complete. 2023-06-10T18:45:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.981+0000 7f37c73c0700 1 Processor -- start 2023-06-10T18:45:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.981+0000 7f37c73c0700 1 -- start start 2023-06-10T18:45:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.981+0000 7f37c73c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 0x7f37c00b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.981+0000 7f37c73c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37c006e6e0 0x7f37c00b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c0154310 con 0x7f37c015c580 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c01e24e0 con 0x7f37c006cde0 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c01e27c0 con 0x7f37c006e6e0 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57772/0 (socket says 172.21.15.104:57772) 2023-06-10T18:45:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 -- 172.21.15.104:0/4249863639 learned_addr learned my addr 172.21.15.104:0/4249863639 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:45:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c515c700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 0x7f37c00b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c495b700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37c006e6e0 0x7f37c00b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 -- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37c006e6e0 msgr2=0x7f37c00b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37c006e6e0 0x7f37c00b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 -- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 msgr2=0x7f37c00b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:45:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 0x7f37c00b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:45:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 -- 172.21.15.104:0/4249863639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b0009430 con 0x7f37c015c580 2023-06-10T18:45:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c515c700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 0x7f37c00b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:45:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c595d700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f37b8007ef0 tx=0x7f37b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b800de60 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b800f040 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c01e2aa0 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8003c60 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.982+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c01e3060 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.983+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c01e3280 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.984+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f37b8004030 con 0x7f37c015c580 2023-06-10T18:45:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.984+0000 7f37b5ffb700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 0x7f37a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:45:59.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.984+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f37b8089800 con 0x7f37c015c580 2023-06-10T18:45:59.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.984+0000 7f37c515c700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 0x7f37a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:45:59.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.985+0000 7f37c515c700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 0x7f37a406d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009b40 tx=0x7f37b0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:45:59.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:45:59.988+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f37b8054730 con 0x7f37c015c580 2023-06-10T18:46:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.173+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37c0074050 con 0x7f37c015c580 2023-06-10T18:46:00.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.174+0000 7f37b5ffb700 1 -- 172.21.15.104:0/4249863639 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37b8057d50 con 0x7f37c015c580 2023-06-10T18:46:00.180 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:46:00.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 msgr2=0x7f37a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 0x7f37a406d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009b40 tx=0x7f37b0008070 comp rx=0 tx=0).stop 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 msgr2=0x7f37c01f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f37b8007ef0 tx=0x7f37b800d3b0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 shutdown_connections 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f37a406ace0 0x7f37a406d190 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37c006cde0 0x7f37c00b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f37c015c580 0x7f37c01f5440 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 --2- 172.21.15.104:0/4249863639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37c006e6e0 0x7f37c00b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 >> 172.21.15.104:0/4249863639 conn(0x7f37c006b8a0 msgr2=0x7f37c0155b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 shutdown_connections 2023-06-10T18:46:00.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.176+0000 7f37c73c0700 1 -- 172.21.15.104:0/4249863639 wait complete. 2023-06-10T18:46:00.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:46:00.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.312+0000 7f3c039d5700 1 Processor -- start 2023-06-10T18:46:00.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.312+0000 7f3c039d5700 1 -- start start 2023-06-10T18:46:00.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c039d5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3bfc0b1780 0x7f3bfc06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c039d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 0x7f3bfc06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c039d5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0ae7e0 con 0x7f3bfc15c6f0 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c039d5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc06df90 con 0x7f3bfc0b1780 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c039d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc06e270 con 0x7f3bfc06ea70 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c01771700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3bfc15c6f0 0x7f3bfc0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55604/0 (socket says 172.21.15.104:55604) 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c01771700 1 -- 172.21.15.104:0/4015381693 learned_addr learned my addr 172.21.15.104:0/4015381693 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c00f70700 1 --2- 172.21.15.104:0/4015381693 >> v2:172.21.15.114:3300/0 conn(0x7f3bfc0b1780 0x7f3bfc06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3c01f72700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 0x7f3bfc06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3bf3fff700 1 -- 172.21.15.104:0/4015381693 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526086813 0 0) 0x7f3bfc0ae7e0 con 0x7f3bfc15c6f0 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.313+0000 7f3bf3fff700 1 -- 172.21.15.104:0/4015381693 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be80035b0 con 0x7f3bfc15c6f0 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c00f70700 1 -- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 msgr2=0x7f3bfc06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c00f70700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 0x7f3bfc06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c00f70700 1 -- 172.21.15.104:0/4015381693 >> v1:172.21.15.104:6789/0 conn(0x7f3bfc15c6f0 legacy=0x7f3bfc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c00f70700 1 -- 172.21.15.104:0/4015381693 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc06e550 con 0x7f3bfc0b1780 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c01f72700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 0x7f3bfc06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3c00f70700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc0b1780 0x7f3bfc06cde0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009840 tx=0x7f3bf8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad4053b1ff6f065b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3bf3fff700 1 -- 172.21.15.104:0/4015381693 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8011830 con 0x7f3bfc0b1780 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3bf3fff700 1 -- 172.21.15.104:0/4015381693 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf8011990 con 0x7f3bfc0b1780 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.314+0000 7f3bf3fff700 1 -- 172.21.15.104:0/4015381693 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8011d20 con 0x7f3bfc0b1780 2023-06-10T18:46:00.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 -- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc0b1780 msgr2=0x7f3bfc06cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc0b1780 0x7f3bfc06cde0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009840 tx=0x7f3bf8009b50 comp rx=0 tx=0).stop 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 -- 172.21.15.104:0/4015381693 shutdown_connections 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc06ea70 0x7f3bfc06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 --2- 172.21.15.104:0/4015381693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc0b1780 0x7f3bfc06cde0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 -- 172.21.15.104:0/4015381693 >> 172.21.15.104:0/4015381693 conn(0x7f3bfc06b8a0 msgr2=0x7f3bfc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 -- 172.21.15.104:0/4015381693 shutdown_connections 2023-06-10T18:46:00.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.315+0000 7f3c039d5700 1 -- 172.21.15.104:0/4015381693 wait complete. 2023-06-10T18:46:00.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.316+0000 7f3c039d5700 1 Processor -- start 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.316+0000 7f3c039d5700 1 -- start start 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc0b1780 0x7f3bfc0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc15c6f0 0x7f3bfc157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc150960 con 0x7f3bfc06ea70 2023-06-10T18:46:00.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b35c0 con 0x7f3bfc15c6f0 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0c31c0 con 0x7f3bfc0b1780 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57782/0 (socket says 172.21.15.104:57782) 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 -- 172.21.15.104:0/12108210 learned_addr learned my addr 172.21.15.104:0/12108210 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01f72700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc15c6f0 0x7f3bfc157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c00f70700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc0b1780 0x7f3bfc0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 -- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc0b1780 msgr2=0x7f3bfc0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc0b1780 0x7f3bfc0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 -- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc15c6f0 msgr2=0x7f3bfc157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc15c6f0 0x7f3bfc157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 -- 172.21.15.104:0/12108210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4009a70 con 0x7f3bfc06ea70 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c01771700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3bec001260 tx=0x7f3bec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0074e0 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec007640 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf80094a0 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.317+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec005a50 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.318+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc0b3000 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.318+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc074050 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.319+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3bec005bb0 con 0x7f3bfc06ea70 2023-06-10T18:46:00.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.319+0000 7f3bf1ffb700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 0x7f3be006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.319+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3bec088b60 con 0x7f3bfc06ea70 2023-06-10T18:46:00.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.319+0000 7f3c00f70700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 0x7f3be006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.320+0000 7f3c00f70700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 0x7f3be006d190 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0b1b70 tx=0x7f3bf8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.323+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3bec053aa0 con 0x7f3bfc06ea70 2023-06-10T18:46:00.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.509+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3bfc0d0030 con 0x7f3bfc06ea70 2023-06-10T18:46:00.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.509+0000 7f3bf1ffb700 1 -- 172.21.15.104:0/12108210 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3bec007e60 con 0x7f3bfc06ea70 2023-06-10T18:46:00.515 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 msgr2=0x7f3be006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 0x7f3be006d190 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0b1b70 tx=0x7f3bf8008030 comp rx=0 tx=0).stop 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 msgr2=0x7f3bfc150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3bec001260 tx=0x7f3bec00c860 comp rx=0 tx=0).stop 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 shutdown_connections 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3be006ace0 0x7f3be006d190 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3bfc15c6f0 0x7f3bfc157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bfc06ea70 0x7f3bfc150420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 --2- 172.21.15.104:0/12108210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3bfc0b1780 0x7f3bfc0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 >> 172.21.15.104:0/12108210 conn(0x7f3bfc06b8a0 msgr2=0x7f3bfc0ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 shutdown_connections 2023-06-10T18:46:00.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.511+0000 7f3c039d5700 1 -- 172.21.15.104:0/12108210 wait complete. 2023-06-10T18:46:00.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:46:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.643+0000 7f7e79654700 1 Processor -- start 2023-06-10T18:46:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.643+0000 7f7e79654700 1 -- start start 2023-06-10T18:46:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e79654700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e74156740 0x7f7e7415d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e79654700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e7406a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e79654700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7406a810 con 0x7f7e740cec20 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e79654700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7415cdf0 con 0x7f7e74156740 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e79654700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740cf0a0 con 0x7f7e74069e70 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e72ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7e74156740 0x7f7e7415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e727fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7e740cec20 0x7f7e740cd250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55618/0 (socket says 172.21.15.104:55618) 2023-06-10T18:46:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.644+0000 7f7e727fc700 1 -- 172.21.15.104:0/2776493052 learned_addr learned my addr 172.21.15.104:0/2776493052 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e72ffd700 1 -- 172.21.15.104:0/2776493052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 msgr2=0x7f7e7406a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e737fe700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e7406a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e71ffb700 1 -- 172.21.15.104:0/2776493052 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775215565 0 0) 0x7f7e7406a810 con 0x7f7e740cec20 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e7406a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e72ffd700 1 -- 172.21.15.104:0/2776493052 >> v1:172.21.15.104:6789/0 conn(0x7f7e740cec20 legacy=0x7f7e740cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e72ffd700 1 -- 172.21.15.104:0/2776493052 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740cf380 con 0x7f7e74156740 2023-06-10T18:46:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e7415d280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009840 tx=0x7f7e68009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47b772239aa9254 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e71ffb700 1 -- 172.21.15.104:0/2776493052 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68011830 con 0x7f7e74156740 2023-06-10T18:46:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e71ffb700 1 -- 172.21.15.104:0/2776493052 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68011990 con 0x7f7e74156740 2023-06-10T18:46:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.645+0000 7f7e71ffb700 1 -- 172.21.15.104:0/2776493052 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68011d20 con 0x7f7e74156740 2023-06-10T18:46:00.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 -- 172.21.15.104:0/2776493052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 msgr2=0x7f7e7415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e7415d280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009840 tx=0x7f7e68009b50 comp rx=0 tx=0).stop 2023-06-10T18:46:00.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 -- 172.21.15.104:0/2776493052 shutdown_connections 2023-06-10T18:46:00.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e7415d280 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 --2- 172.21.15.104:0/2776493052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e7406a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 -- 172.21.15.104:0/2776493052 >> 172.21.15.104:0/2776493052 conn(0x7f7e7406d240 msgr2=0x7f7e7406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 -- 172.21.15.104:0/2776493052 shutdown_connections 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.646+0000 7f7e79654700 1 -- 172.21.15.104:0/2776493052 wait complete. 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.647+0000 7f7e79654700 1 Processor -- start 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.647+0000 7f7e79654700 1 -- start start 2023-06-10T18:46:00.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e741f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 0x7f7e740c3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741f2420 con 0x7f7e74156740 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740c4110 con 0x7f7e740c31b0 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740c3b90 con 0x7f7e74069e70 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57798/0 (socket says 172.21.15.104:57798) 2023-06-10T18:46:00.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 -- 172.21.15.104:0/1883457741 learned_addr learned my addr 172.21.15.104:0/1883457741 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e737fe700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 0x7f7e740c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e741f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 -- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 msgr2=0x7f7e741f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e741f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 -- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 msgr2=0x7f7e740c3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 0x7f7e740c3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 -- 172.21.15.104:0/1883457741 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680094a0 con 0x7f7e74156740 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e741f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e737fe700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 0x7f7e740c3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e727fc700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e640011e0 tx=0x7f7e6400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64004c80 con 0x7f7e74156740 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64004e00 con 0x7f7e74156740 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64007310 con 0x7f7e74156740 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e740c3e30 con 0x7f7e74156740 2023-06-10T18:46:00.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.648+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741f2040 con 0x7f7e74156740 2023-06-10T18:46:00.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.649+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e741f2560 con 0x7f7e74156740 2023-06-10T18:46:00.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.650+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7e64007470 con 0x7f7e74156740 2023-06-10T18:46:00.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.650+0000 7f7e5b7fe700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 0x7f7e6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:00.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.650+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f7e640892f0 con 0x7f7e74156740 2023-06-10T18:46:00.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.650+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 0x7f7e6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:00.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.651+0000 7f7e72ffd700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 0x7f7e6006d190 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7e680090b0 tx=0x7f7e6801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:00.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.654+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7e741f2560 con 0x7f7e74156740 2023-06-10T18:46:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.855+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7e741f2560 con 0x7f7e74156740 2023-06-10T18:46:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.856+0000 7f7e5b7fe700 1 -- 172.21.15.104:0/1883457741 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7e741f2560 con 0x7f7e74156740 2023-06-10T18:46:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 msgr2=0x7f7e6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 0x7f7e6006d190 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7e680090b0 tx=0x7f7e6801c040 comp rx=0 tx=0).stop 2023-06-10T18:46:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 msgr2=0x7f7e740cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e640011e0 tx=0x7f7e6400cae0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 shutdown_connections 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6006ace0 0x7f7e6006d190 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e740c31b0 0x7f7e740c3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74069e70 0x7f7e741f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 --2- 172.21.15.104:0/1883457741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e74156740 0x7f7e740cf8f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 >> 172.21.15.104:0/1883457741 conn(0x7f7e7406d240 msgr2=0x7f7e740cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 shutdown_connections 2023-06-10T18:46:00.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:00.858+0000 7f7e79654700 1 -- 172.21.15.104:0/1883457741 wait complete. 2023-06-10T18:46:00.866 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:46:00.875 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:46:00.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:46:00.876 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:46:00.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[119644]: fuse finished with error 0 and tester_r 0 2023-06-10T18:46:00.911 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:46:07.013 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:46:07.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:46:07.014 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:46:07.036 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:46:07.037 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:46:07.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:46:07.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.207+0000 7f887070d700 1 Processor -- start 2023-06-10T18:46:07.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.207+0000 7f887070d700 1 -- start start 2023-06-10T18:46:07.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.208+0000 7f887070d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8868070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.208+0000 7f887070d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88680ce510 0x7f88680ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.208+0000 7f887070d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886815c050 con 0x7f88680c8c40 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.208+0000 7f887070d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88680ceea0 con 0x7f88680ce510 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.208+0000 7f887070d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88680c9970 con 0x7f886815bc80 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886ecaa700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f88680c8c40 0x7f88680c90c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55636/0 (socket says 172.21.15.104:55636) 2023-06-10T18:46:07.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886ecaa700 1 -- 172.21.15.104:0/2063911203 learned_addr learned my addr 172.21.15.104:0/2063911203 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:07.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886dca8700 1 --2- 172.21.15.104:0/2063911203 >> v2:172.21.15.114:3300/0 conn(0x7f88680ce510 0x7f88680ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8868070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312410197 0 0) 0x7f886815c050 con 0x7f88680c8c40 2023-06-10T18:46:07.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88540035b0 con 0x7f88680c8c40 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488618681 0 0) 0x7f88540035b0 con 0x7f88680c8c40 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 msgr2=0x7f8868070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8868070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 >> v2:172.21.15.114:3300/0 conn(0x7f88680ce510 msgr2=0x7f88680ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 --2- 172.21.15.104:0/2063911203 >> v2:172.21.15.114:3300/0 conn(0x7f88680ce510 0x7f88680ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88680ccb70 con 0x7f88680c8c40 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f88600019f0 con 0x7f88680c8c40 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 >> v1:172.21.15.104:6789/0 conn(0x7f88680c8c40 legacy=0x7f88680c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8854003a00 0x7f8854003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8854004430 0x7f88540048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8854004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88680ceea0 con 0x7f8854004430 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88680c9970 con 0x7f8854003a00 2023-06-10T18:46:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885400dc20 con 0x7f886815bc80 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886dca8700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8854004430 0x7f88540048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8854003a00 0x7f8854003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.209+0000 7f886ecaa700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8854004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 -- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 msgr2=0x7f8854004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886815bc80 0x7f8854004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 -- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8854003a00 msgr2=0x7f8854003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8854003a00 0x7f8854003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 -- 172.21.15.104:0/2063911203 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88540035b0 con 0x7f8854004430 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886dca8700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8854004430 0x7f88540048f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8864009b60 tx=0x7f8854005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bfc452a8e855c1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400e6c0 con 0x7f8854004430 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886400e820 con 0x7f8854004430 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886d4a7700 1 -- 172.21.15.104:0/2063911203 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400eb90 con 0x7f8854004430 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f887070d700 1 -- 172.21.15.104:0/2063911203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8854004430 msgr2=0x7f88540048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f887070d700 1 --2- 172.21.15.104:0/2063911203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8854004430 0x7f88540048f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8864009b60 tx=0x7f8854005880 comp rx=0 tx=0).stop 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f886ecaa700 1 -- 172.21.15.104:0/2063911203 reap_dead start 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f887070d700 1 -- 172.21.15.104:0/2063911203 shutdown_connections 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.210+0000 7f887070d700 1 -- 172.21.15.104:0/2063911203 >> 172.21.15.104:0/2063911203 conn(0x7f886806ab00 msgr2=0x7f88680b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.211+0000 7f887070d700 1 -- 172.21.15.104:0/2063911203 shutdown_connections 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.211+0000 7f887070d700 1 -- 172.21.15.104:0/2063911203 wait complete. 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 Processor -- start 2023-06-10T18:46:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 -- start start 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 0x7f88681523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8868070650 0x7f8868070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868071000 con 0x7f88681f22b0 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.212+0000 7f887070d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88680b1be0 con 0x7f8868070650 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f887070d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88680b1ec0 con 0x7f8854004430 2023-06-10T18:46:07.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57818/0 (socket says 172.21.15.104:57818) 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 -- 172.21.15.104:0/2701400573 learned_addr learned my addr 172.21.15.104:0/2701400573 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 0x7f88681523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886ecaa700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8868070650 0x7f8868070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 -- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 msgr2=0x7f88681523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 0x7f88681523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 -- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8868070650 msgr2=0x7f8868070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8868070650 0x7f8868070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 -- 172.21.15.104:0/2701400573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88680b21a0 con 0x7f88681f22b0 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 0x7f88681523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f886dca8700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8854005760 tx=0x7f88640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400e6c0 con 0x7f88681f22b0 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8864003ab0 con 0x7f88681f22b0 2023-06-10T18:46:07.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864003e60 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88680b2480 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.213+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88681ee0d0 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.214+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88680c5860 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.215+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8864013470 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.215+0000 7f885effd700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 0x7f884c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.215+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f886408af40 con 0x7f88681f22b0 2023-06-10T18:46:07.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.215+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 0x7f884c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.216+0000 7f886e4a9700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 0x7f884c06d190 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8858006dd0 tx=0x7f8858008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.219+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8864056020 con 0x7f88681f22b0 2023-06-10T18:46:07.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.406+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8868074050 con 0x7f88681f22b0 2023-06-10T18:46:07.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.407+0000 7f885effd700 1 -- 172.21.15.104:0/2701400573 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7f8864059640 con 0x7f88681f22b0 2023-06-10T18:46:07.414 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:46:07.414 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 msgr2=0x7f884c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 0x7f884c06d190 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8858006dd0 tx=0x7f8858008040 comp rx=0 tx=0).stop 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 msgr2=0x7f88680b13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8854005760 tx=0x7f88640047c0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 shutdown_connections 2023-06-10T18:46:07.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f884c06ace0 0x7f884c06d190 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8868070650 0x7f8868070ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8854004430 0x7f88681523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 --2- 172.21.15.104:0/2701400573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88681f22b0 0x7f88680b13b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 >> 172.21.15.104:0/2701400573 conn(0x7f886806ab00 msgr2=0x7f8868159e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 shutdown_connections 2023-06-10T18:46:07.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.409+0000 7f887070d700 1 -- 172.21.15.104:0/2701400573 wait complete. 2023-06-10T18:46:07.427 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:45:39.741633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:45:40.245539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:46:07.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:46:07.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.546+0000 7f7c9ce33700 1 Processor -- start 2023-06-10T18:46:07.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.546+0000 7f7c9ce33700 1 -- start start 2023-06-10T18:46:07.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9ce33700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c9806cde0 0x7f7c980b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9ce33700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9806e250 con 0x7f7c9815c580 2023-06-10T18:46:07.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9ce33700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9815c950 con 0x7f7c9806e6e0 2023-06-10T18:46:07.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9ce33700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c980b1e80 con 0x7f7c9806cde0 2023-06-10T18:46:07.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c96d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c9806cde0 0x7f7c980b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7c9815c580 0x7f7c980b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55652/0 (socket says 172.21.15.104:55652) 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c96d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7c9806cde0 0x7f7c980b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57828/0 (socket says 172.21.15.104:57828) 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.547+0000 7f7c9659c700 1 -- 172.21.15.104:0/660379594 learned_addr learned my addr 172.21.15.104:0/660379594 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c96d9d700 1 -- 172.21.15.104:0/660379594 >> v1:172.21.15.114:6789/0 conn(0x7f7c9806e6e0 legacy=0x7f7c980ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c96d9d700 1 -- 172.21.15.104:0/660379594 >> v1:172.21.15.104:6789/0 conn(0x7f7c9815c580 legacy=0x7f7c980b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c96d9d700 1 -- 172.21.15.104:0/660379594 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9806d920 con 0x7f7c9806cde0 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c96d9d700 1 --2- 172.21.15.104:0/660379594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806cde0 0x7f7c980b1780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009d90 tx=0x7f7c9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7424905e48e115 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c9559a700 1 -- 172.21.15.104:0/660379594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9000e070 con 0x7f7c9806cde0 2023-06-10T18:46:07.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c9559a700 1 -- 172.21.15.104:0/660379594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9000bbd0 con 0x7f7c9806cde0 2023-06-10T18:46:07.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c9559a700 1 -- 172.21.15.104:0/660379594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90012430 con 0x7f7c9806cde0 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c9ce33700 1 -- 172.21.15.104:0/660379594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806cde0 msgr2=0x7f7c980b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.548+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/660379594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806cde0 0x7f7c980b1780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009d90 tx=0x7f7c9000b330 comp rx=0 tx=0).stop 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.549+0000 7f7c9ce33700 1 -- 172.21.15.104:0/660379594 shutdown_connections 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.549+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/660379594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806cde0 0x7f7c980b1780 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.549+0000 7f7c9ce33700 1 -- 172.21.15.104:0/660379594 >> 172.21.15.104:0/660379594 conn(0x7f7c9806b8a0 msgr2=0x7f7c980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.549+0000 7f7c9ce33700 1 -- 172.21.15.104:0/660379594 shutdown_connections 2023-06-10T18:46:07.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.549+0000 7f7c9ce33700 1 -- 172.21.15.104:0/660379594 wait complete. 2023-06-10T18:46:07.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.550+0000 7f7c9ce33700 1 Processor -- start 2023-06-10T18:46:07.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.550+0000 7f7c9ce33700 1 -- start start 2023-06-10T18:46:07.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 0x7f7c98155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9815c580 0x7f7c981e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c980b3530 con 0x7f7c9806e6e0 2023-06-10T18:46:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c980aded0 con 0x7f7c9815c580 2023-06-10T18:46:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c980b2e70 con 0x7f7c9806cde0 2023-06-10T18:46:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57832/0 (socket says 172.21.15.104:57832) 2023-06-10T18:46:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 -- 172.21.15.104:0/3449197998 learned_addr learned my addr 172.21.15.104:0/3449197998 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c96d9d700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9815c580 0x7f7c981e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9659c700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 0x7f7c98155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 -- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 msgr2=0x7f7c98155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 0x7f7c98155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 -- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9815c580 msgr2=0x7f7c981e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9815c580 0x7f7c981e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 -- 172.21.15.104:0/3449197998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90009a70 con 0x7f7c9806e6e0 2023-06-10T18:46:07.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9659c700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 0x7f7c98155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:07.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c95d9b700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800c4b0 tx=0x7f7c8800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c880052d0 con 0x7f7c9806e6e0 2023-06-10T18:46:07.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88005430 con 0x7f7c9806e6e0 2023-06-10T18:46:07.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c880038c0 con 0x7f7c9806e6e0 2023-06-10T18:46:07.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c980b3150 con 0x7f7c9806e6e0 2023-06-10T18:46:07.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.551+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c980c3270 con 0x7f7c9806e6e0 2023-06-10T18:46:07.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.552+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c980b2cd0 con 0x7f7c9806e6e0 2023-06-10T18:46:07.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.553+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7c88003a20 con 0x7f7c9806e6e0 2023-06-10T18:46:07.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.553+0000 7f7c86ffd700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 0x7f7c7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.553+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f7c880889c0 con 0x7f7c9806e6e0 2023-06-10T18:46:07.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.553+0000 7f7c9659c700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 0x7f7c7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.554+0000 7f7c9659c700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 0x7f7c7806d190 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0010e0 tx=0x7f7c8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.557+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7c980b2cd0 con 0x7f7c9806e6e0 2023-06-10T18:46:07.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:46:07.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:46:07.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.745+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c980b3670 con 0x7f7c9806e6e0 2023-06-10T18:46:07.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.745+0000 7f7c86ffd700 1 -- 172.21.15.104:0/3449197998 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c88017020 con 0x7f7c9806e6e0 2023-06-10T18:46:07.751 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:46:07.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 msgr2=0x7f7c7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 0x7f7c7806d190 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0010e0 tx=0x7f7c8c009040 comp rx=0 tx=0).stop 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 msgr2=0x7f7c981e3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800c4b0 tx=0x7f7c8800c7c0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 shutdown_connections 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7c7806ace0 0x7f7c7806d190 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c9815c580 0x7f7c981e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c9806cde0 0x7f7c98155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 --2- 172.21.15.104:0/3449197998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7c9806e6e0 0x7f7c981e3b50 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 >> 172.21.15.104:0/3449197998 conn(0x7f7c9806b8a0 msgr2=0x7f7c981599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:07.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 shutdown_connections 2023-06-10T18:46:07.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.747+0000 7f7c9ce33700 1 -- 172.21.15.104:0/3449197998 wait complete. 2023-06-10T18:46:07.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:46:07.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.957+0000 7f4e522b6700 1 Processor -- start 2023-06-10T18:46:07.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.957+0000 7f4e522b6700 1 -- start start 2023-06-10T18:46:07.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e522b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 0x7f4e4c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e522b6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e522b6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4c06e250 con 0x7f4e4c06cde0 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e522b6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c15c950 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e522b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0cf0e0 con 0x7f4e4c15c580 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e4b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e4b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57838/0 (socket says 172.21.15.104:57838) 2023-06-10T18:46:07.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.958+0000 7f4e4b7fe700 1 -- 172.21.15.104:0/3579917410 learned_addr learned my addr 172.21.15.104:0/3579917410 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4bfff700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 0x7f4e4c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4b7fe700 1 -- 172.21.15.104:0/3579917410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 msgr2=0x7f4e4c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4b7fe700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 0x7f4e4c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4b7fe700 1 -- 172.21.15.104:0/3579917410 >> v1:172.21.15.114:6789/0 conn(0x7f4e4c06cde0 legacy=0x7f4e4c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4b7fe700 1 -- 172.21.15.104:0/3579917410 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c0b1e80 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4bfff700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 0x7f4e4c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:46:07.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4b7fe700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0098f0 tx=0x7f4e3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfefbde1298cd3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4affd700 1 -- 172.21.15.104:0/3579917410 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c004230 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4affd700 1 -- 172.21.15.104:0/3579917410 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c004390 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e4affd700 1 -- 172.21.15.104:0/3579917410 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0036a0 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 -- 172.21.15.104:0/3579917410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c06e6e0 msgr2=0x7f4e4c0b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0098f0 tx=0x7f4e3c009c00 comp rx=0 tx=0).stop 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 -- 172.21.15.104:0/3579917410 shutdown_connections 2023-06-10T18:46:07.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c15c580 0x7f4e4c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 --2- 172.21.15.104:0/3579917410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c06e6e0 0x7f4e4c0b15b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 -- 172.21.15.104:0/3579917410 >> 172.21.15.104:0/3579917410 conn(0x7f4e4c06b8a0 msgr2=0x7f4e4c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:07.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.959+0000 7f4e522b6700 1 -- 172.21.15.104:0/3579917410 shutdown_connections 2023-06-10T18:46:07.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.960+0000 7f4e522b6700 1 -- 172.21.15.104:0/3579917410 wait complete. 2023-06-10T18:46:07.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.961+0000 7f4e522b6700 1 Processor -- start 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.961+0000 7f4e522b6700 1 -- start start 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e4c06cde0 0x7f4e4c1f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c06e6e0 0x7f4e4c1f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0af050 con 0x7f4e4c15c580 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0af260 con 0x7f4e4c06cde0 2023-06-10T18:46:07.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0af540 con 0x7f4e4c06e6e0 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57848/0 (socket says 172.21.15.104:57848) 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e4b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c06e6e0 0x7f4e4c1f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 -- 172.21.15.104:0/2034515270 learned_addr learned my addr 172.21.15.104:0/2034515270 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e4bfff700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e4c06cde0 0x7f4e4c1f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 -- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c06e6e0 msgr2=0x7f4e4c1f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c06e6e0 0x7f4e4c1f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 -- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e4c06cde0 msgr2=0x7f4e4c1f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e4c06cde0 0x7f4e4c1f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 -- 172.21.15.104:0/2034515270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c009550 con 0x7f4e4c15c580 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e50853700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b990 tx=0x7f4e4400bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4400d890 con 0x7f4e4c15c580 2023-06-10T18:46:07.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4400d9f0 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c0af820 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4400dda0 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.962+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c1f5d20 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.963+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c1f5f10 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.964+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4e44006d50 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.964+0000 7f4e48ff9700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 0x7f4e3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.964+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4e44089520 con 0x7f4e4c15c580 2023-06-10T18:46:07.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.964+0000 7f4e4bfff700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 0x7f4e3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:07.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.965+0000 7f4e4bfff700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 0x7f4e3806d190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000abe0 tx=0x7f4e40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:07.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:07.968+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4e44054450 con 0x7f4e4c15c580 2023-06-10T18:46:08.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.152+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4e4c0afd40 con 0x7f4e4c15c580 2023-06-10T18:46:08.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.152+0000 7f4e48ff9700 1 -- 172.21.15.104:0/2034515270 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e4c0afd40 con 0x7f4e4c15c580 2023-06-10T18:46:08.158 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 msgr2=0x7f4e3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 0x7f4e3806d190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000abe0 tx=0x7f4e40008040 comp rx=0 tx=0).stop 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 msgr2=0x7f4e4c0b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b990 tx=0x7f4e4400bca0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 shutdown_connections 2023-06-10T18:46:08.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e3806ace0 0x7f4e3806d190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e4c06cde0 0x7f4e4c1f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e4c15c580 0x7f4e4c0b0020 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 --2- 172.21.15.104:0/2034515270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e4c06e6e0 0x7f4e4c1f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 >> 172.21.15.104:0/2034515270 conn(0x7f4e4c06b8a0 msgr2=0x7f4e4c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 shutdown_connections 2023-06-10T18:46:08.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.154+0000 7f4e522b6700 1 -- 172.21.15.104:0/2034515270 wait complete. 2023-06-10T18:46:08.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:46:08.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.290+0000 7f5a040e9700 1 Processor -- start 2023-06-10T18:46:08.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 -- start start 2023-06-10T18:46:08.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f59fc15c580 0x7f59fc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59fc0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc06e250 con 0x7f59fc06e6e0 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc15c950 con 0x7f59fc15c580 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a040e9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b1e80 con 0x7f59fc06cde0 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.291+0000 7f5a01684700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f59fc06e6e0 0x7f59fc0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55680/0 (socket says 172.21.15.104:55680) 2023-06-10T18:46:08.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a01684700 1 -- 172.21.15.104:0/1857506037 learned_addr learned my addr 172.21.15.104:0/1857506037 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a02686700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59fc0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a01e85700 1 --2- 172.21.15.104:0/1857506037 >> v2:172.21.15.114:3300/0 conn(0x7f59fc15c580 0x7f59fc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271876427 0 0) 0x7f59fc06e250 con 0x7f59fc06e6e0 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e80035b0 con 0x7f59fc06e6e0 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754657967 0 0) 0x7f59e80035b0 con 0x7f59fc06e6e0 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 msgr2=0x7f59fc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59fc0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 >> v2:172.21.15.114:3300/0 conn(0x7f59fc15c580 msgr2=0x7f59fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 --2- 172.21.15.104:0/1857506037 >> v2:172.21.15.114:3300/0 conn(0x7f59fc15c580 0x7f59fc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc06d920 con 0x7f59fc06e6e0 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f59f8001670 con 0x7f59fc06e6e0 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 >> v1:172.21.15.104:6789/0 conn(0x7f59fc06e6e0 legacy=0x7f59fc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e8003a00 0x7f59e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8004d90 0x7f59e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15c950 con 0x7f59e8004d90 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b1e80 con 0x7f59e8003a00 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e800ba10 con 0x7f59fc06cde0 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a02686700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8004d90 0x7f59e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a01e85700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e8003a00 0x7f59e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a01684700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a02686700 1 -- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 msgr2=0x7f59e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.292+0000 7f5a02686700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59fc06cde0 0x7f59e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a02686700 1 -- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e8003a00 msgr2=0x7f59e8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a02686700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e8003a00 0x7f59e8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a02686700 1 -- 172.21.15.104:0/1857506037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80035b0 con 0x7f59e8004d90 2023-06-10T18:46:08.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a02686700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8004d90 0x7f59e8005200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009fc0 tx=0x7f59e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fde9ddb414ab02 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400ec30 con 0x7f59e8004d90 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f400f040 con 0x7f59e8004d90 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a00e83700 1 -- 172.21.15.104:0/1857506037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4014480 con 0x7f59e8004d90 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 -- 172.21.15.104:0/1857506037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8004d90 msgr2=0x7f59e8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 --2- 172.21.15.104:0/1857506037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e8004d90 0x7f59e8005200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009fc0 tx=0x7f59e8005880 comp rx=0 tx=0).stop 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a02686700 1 -- 172.21.15.104:0/1857506037 reap_dead start 2023-06-10T18:46:08.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 -- 172.21.15.104:0/1857506037 shutdown_connections 2023-06-10T18:46:08.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 -- 172.21.15.104:0/1857506037 >> 172.21.15.104:0/1857506037 conn(0x7f59fc06b8a0 msgr2=0x7f59fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 -- 172.21.15.104:0/1857506037 shutdown_connections 2023-06-10T18:46:08.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.293+0000 7f5a040e9700 1 -- 172.21.15.104:0/1857506037 wait complete. 2023-06-10T18:46:08.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 Processor -- start 2023-06-10T18:46:08.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 -- start start 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59e8004d90 0x7f59fc155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59fc0ad210 0x7f59fc0ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0adbf0 con 0x7f59fc0b28d0 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b3950 con 0x7f59fc0ad210 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a040e9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b2d40 con 0x7f59e8004d90 2023-06-10T18:46:08.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a01684700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a01684700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57864/0 (socket says 172.21.15.104:57864) 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a01684700 1 -- 172.21.15.104:0/4269476237 learned_addr learned my addr 172.21.15.104:0/4269476237 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a01e85700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59e8004d90 0x7f59fc155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.295+0000 7f5a02686700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59fc0ad210 0x7f59fc0ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 -- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59e8004d90 msgr2=0x7f59fc155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59e8004d90 0x7f59fc155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 -- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59fc0ad210 msgr2=0x7f59fc0ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59fc0ad210 0x7f59fc0ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 -- 172.21.15.104:0/4269476237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0b3020 con 0x7f59fc0b28d0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a01684700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f59f8000fb0 tx=0x7f59f80075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800f630 con 0x7f59fc0b28d0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f800f7d0 con 0x7f59fc0b28d0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc0b3300 con 0x7f59fc0b28d0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f800fb80 con 0x7f59fc0b28d0 2023-06-10T18:46:08.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.296+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc1560a0 con 0x7f59fc0b28d0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.297+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc0b3a90 con 0x7f59fc0b28d0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.297+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f59f800fce0 con 0x7f59fc0b28d0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.298+0000 7f59f27fc700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 0x7f59e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.298+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f59f8089550 con 0x7f59fc0b28d0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.298+0000 7f5a01e85700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 0x7f59e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.298+0000 7f5a01e85700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 0x7f59e006d190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009c40 tx=0x7f59ec0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.302+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f59fc0b3a90 con 0x7f59fc0b28d0 2023-06-10T18:46:08.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.487+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f59fc0b3a90 con 0x7f59fc0b28d0 2023-06-10T18:46:08.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.487+0000 7f59f27fc700 1 -- 172.21.15.104:0/4269476237 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f59fc0b3a90 con 0x7f59fc0b28d0 2023-06-10T18:46:08.493 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:46:08.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 msgr2=0x7f59e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 0x7f59e006d190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009c40 tx=0x7f59ec0091c0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 msgr2=0x7f59fc158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f59f8000fb0 tx=0x7f59f80075f0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 shutdown_connections 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59e006ace0 0x7f59e006d190 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59fc0ad210 0x7f59fc0ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59e8004d90 0x7f59fc155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 --2- 172.21.15.104:0/4269476237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59fc0b28d0 0x7f59fc158020 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.489+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 >> 172.21.15.104:0/4269476237 conn(0x7f59fc06b8a0 msgr2=0x7f59fc159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.490+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 shutdown_connections 2023-06-10T18:46:08.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.490+0000 7f5a040e9700 1 -- 172.21.15.104:0/4269476237 wait complete. 2023-06-10T18:46:08.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:46:08.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.625+0000 7fa316e8e700 1 Processor -- start 2023-06-10T18:46:08.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.625+0000 7fa316e8e700 1 -- start start 2023-06-10T18:46:08.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3101545a0 0x7fa31015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 0x7fa31015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa31015e720 0x7fa3100cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31015dab0 con 0x7fa31015e720 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31015aa00 con 0x7fa3100af940 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa316e8e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa310154970 con 0x7fa3101545a0 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa31015e720 0x7fa3100cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa31015e720 0x7fa3100cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57874/0 (socket says 172.21.15.104:57874) 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 -- 172.21.15.104:0/2330689414 learned_addr learned my addr 172.21.15.104:0/2330689414 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa314c2a700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3101545a0 0x7fa31015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa30ffff700 1 --2- 172.21.15.104:0/2330689414 >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 0x7fa31015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 -- 172.21.15.104:0/2330689414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3101545a0 msgr2=0x7fa31015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3101545a0 0x7fa31015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 -- 172.21.15.104:0/2330689414 >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 msgr2=0x7fa31015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 --2- 172.21.15.104:0/2330689414 >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 0x7fa31015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 -- 172.21.15.104:0/2330689414 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100cb0b0 con 0x7fa31015e720 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa30ffff700 1 --2- 172.21.15.104:0/2330689414 >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 0x7fa31015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:08.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa31542b700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa31015e720 0x7fa3100cd150 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa3100d1790 tx=0x7fa30800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a033d07ad2c1f3e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa30f7fe700 1 -- 172.21.15.104:0/2330689414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800e070 con 0x7fa31015e720 2023-06-10T18:46:08.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa30f7fe700 1 -- 172.21.15.104:0/2330689414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30800b910 con 0x7fa31015e720 2023-06-10T18:46:08.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.626+0000 7fa30f7fe700 1 -- 172.21.15.104:0/2330689414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30800bcc0 con 0x7fa31015e720 2023-06-10T18:46:08.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 -- 172.21.15.104:0/2330689414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa31015e720 msgr2=0x7fa3100cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa31015e720 0x7fa3100cd150 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa3100d1790 tx=0x7fa30800b330 comp rx=0 tx=0).stop 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 -- 172.21.15.104:0/2330689414 shutdown_connections 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa31015e720 0x7fa3100cd150 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2330689414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3101545a0 0x7fa31015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2330689414 >> v2:172.21.15.114:3300/0 conn(0x7fa3100af940 0x7fa31015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 -- 172.21.15.104:0/2330689414 >> 172.21.15.104:0/2330689414 conn(0x7fa31006e370 msgr2=0x7fa31006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 -- 172.21.15.104:0/2330689414 shutdown_connections 2023-06-10T18:46:08.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.627+0000 7fa316e8e700 1 -- 172.21.15.104:0/2330689414 wait complete. 2023-06-10T18:46:08.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.628+0000 7fa316e8e700 1 Processor -- start 2023-06-10T18:46:08.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.628+0000 7fa316e8e700 1 -- start start 2023-06-10T18:46:08.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3100af940 0x7fa3100b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa31015e720 0x7fa3101514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c5f50 con 0x7fa3101545a0 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c6210 con 0x7fa3100af940 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa316e8e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100c64f0 con 0x7fa31015e720 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57878/0 (socket says 172.21.15.104:57878) 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 -- 172.21.15.104:0/2294352747 learned_addr learned my addr 172.21.15.104:0/2294352747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa314c2a700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3100af940 0x7fa3100b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa31542b700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa31015e720 0x7fa3101514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 -- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa31015e720 msgr2=0x7fa3101514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa31015e720 0x7fa3101514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 -- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3100af940 msgr2=0x7fa3100b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3100af940 0x7fa3100b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.629+0000 7fa30ffff700 1 -- 172.21.15.104:0/2294352747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009b30 con 0x7fa3101545a0 2023-06-10T18:46:08.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa30ffff700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00d800 tx=0x7fa2fc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc004030 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc004190 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3100c67d0 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc004520 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.630+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31014c390 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.631+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31014c5e0 con 0x7fa3101545a0 2023-06-10T18:46:08.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.631+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa2fc0036a0 con 0x7fa3101545a0 2023-06-10T18:46:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.631+0000 7fa30d7fa700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 0x7fa2f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.632+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fa2fc088e50 con 0x7fa3101545a0 2023-06-10T18:46:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.632+0000 7fa314c2a700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 0x7fa2f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.632+0000 7fa314c2a700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 0x7fa2f406d190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cacf0 tx=0x7fa304008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.636+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2fc057950 con 0x7fa3101545a0 2023-06-10T18:46:08.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.822+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa310074050 con 0x7fa3101545a0 2023-06-10T18:46:08.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.822+0000 7fa30d7fa700 1 -- 172.21.15.104:0/2294352747 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa2fc019020 con 0x7fa3101545a0 2023-06-10T18:46:08.828 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:46:08.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 msgr2=0x7fa2f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 0x7fa2f406d190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cacf0 tx=0x7fa304008040 comp rx=0 tx=0).stop 2023-06-10T18:46:08.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 msgr2=0x7fa3100c5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00d800 tx=0x7fa2fc00dbc0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 shutdown_connections 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2f406ace0 0x7fa2f406d190 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3100af940 0x7fa3100b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa31015e720 0x7fa3101514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 --2- 172.21.15.104:0/2294352747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3101545a0 0x7fa3100c5a10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 >> 172.21.15.104:0/2294352747 conn(0x7fa31006e370 msgr2=0x7fa310152690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 shutdown_connections 2023-06-10T18:46:08.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.824+0000 7fa316e8e700 1 -- 172.21.15.104:0/2294352747 wait complete. 2023-06-10T18:46:08.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:46:08.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.957+0000 7f4423f10700 1 Processor -- start 2023-06-10T18:46:08.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.957+0000 7f4423f10700 1 -- start start 2023-06-10T18:46:08.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f4423f10700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f4423f10700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f441c0ce890 0x7f441c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f4423f10700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0b6070 con 0x7f441c15a420 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f4423f10700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c0ced10 con 0x7f441c0ce890 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f4423f10700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b1d70 con 0x7f441c0b4ff0 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f44224ad700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f441c0ce890 0x7f441c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f44224ad700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f441c0ce890 0x7f441c15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57888/0 (socket says 172.21.15.104:57888) 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.958+0000 7f44224ad700 1 -- 172.21.15.104:0/2073157665 learned_addr learned my addr 172.21.15.104:0/2073157665 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44214ab700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44224ad700 1 -- 172.21.15.104:0/2073157665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 msgr2=0x7f441c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44224ad700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c0ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44224ad700 1 -- 172.21.15.104:0/2073157665 >> v1:172.21.15.114:6789/0 conn(0x7f441c15a420 legacy=0x7f441c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44224ad700 1 -- 172.21.15.104:0/2073157665 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c0b5460 con 0x7f441c0ce890 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f44224ad700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c15a7f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4414009e50 tx=0x7f441400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc71e6f0c21de9bf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f4420caa700 1 -- 172.21.15.104:0/2073157665 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441400e070 con 0x7f441c0ce890 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f4420caa700 1 -- 172.21.15.104:0/2073157665 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4414003a20 con 0x7f441c0ce890 2023-06-10T18:46:08.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.959+0000 7f4420caa700 1 -- 172.21.15.104:0/2073157665 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414003dd0 con 0x7f441c0ce890 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 -- 172.21.15.104:0/2073157665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 msgr2=0x7f441c15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c15a7f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4414009e50 tx=0x7f441400b330 comp rx=0 tx=0).stop 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 -- 172.21.15.104:0/2073157665 shutdown_connections 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c15a7f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 --2- 172.21.15.104:0/2073157665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 -- 172.21.15.104:0/2073157665 >> 172.21.15.104:0/2073157665 conn(0x7f441c089fd0 msgr2=0x7f441c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:08.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 -- 172.21.15.104:0/2073157665 shutdown_connections 2023-06-10T18:46:08.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.960+0000 7f4423f10700 1 -- 172.21.15.104:0/2073157665 wait complete. 2023-06-10T18:46:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.961+0000 7f4423f10700 1 Processor -- start 2023-06-10T18:46:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.961+0000 7f4423f10700 1 -- start start 2023-06-10T18:46:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.961+0000 7f4423f10700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f441c15a420 0x7f441c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c14ca80 con 0x7f441c0ce890 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0c4450 con 0x7f441c15a420 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0c4730 con 0x7f441c0b4ff0 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4421cac700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57892/0 (socket says 172.21.15.104:57892) 2023-06-10T18:46:08.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44224ad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f441c15a420 0x7f441c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 -- 172.21.15.104:0/1862672646 learned_addr learned my addr 172.21.15.104:0/1862672646 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 -- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 msgr2=0x7f441c14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c14c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 -- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f441c15a420 msgr2=0x7f441c1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f441c15a420 0x7f441c1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 -- 172.21.15.104:0/1862672646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4414009b30 con 0x7f441c0ce890 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44214ab700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f441800c7f0 tx=0x7f441800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418004220 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418004380 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c0c4a10 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180036a0 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.962+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c14cf20 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.963+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c0c4f30 con 0x7f441c0ce890 2023-06-10T18:46:08.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.964+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4418003800 con 0x7f441c0ce890 2023-06-10T18:46:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.964+0000 7f44127fc700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 0x7f440006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.964+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4418089060 con 0x7f441c0ce890 2023-06-10T18:46:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.964+0000 7f4421cac700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 0x7f440006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.965+0000 7f4421cac700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 0x7f440006d190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f441c155c30 tx=0x7f440c0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:08.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:08.968+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f441c0c4f30 con 0x7f441c0ce890 2023-06-10T18:46:09.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.169+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f441c0c4f30 con 0x7f441c0ce890 2023-06-10T18:46:09.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.169+0000 7f44127fc700 1 -- 172.21.15.104:0/1862672646 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7f441c0c4f30 con 0x7f441c0ce890 2023-06-10T18:46:09.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 msgr2=0x7f440006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 0x7f440006d190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f441c155c30 tx=0x7f440c0090e0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 msgr2=0x7f441c0cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f441800c7f0 tx=0x7f441800cb00 comp rx=0 tx=0).stop 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 shutdown_connections 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f440006ace0 0x7f440006d190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f441c15a420 0x7f441c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f441c0b4ff0 0x7f441c14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 --2- 172.21.15.104:0/1862672646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f441c0ce890 0x7f441c0cd240 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.171+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 >> 172.21.15.104:0/1862672646 conn(0x7f441c089fd0 msgr2=0x7f441c0b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:09.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.172+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 shutdown_connections 2023-06-10T18:46:09.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.172+0000 7f4423f10700 1 -- 172.21.15.104:0/1862672646 wait complete. 2023-06-10T18:46:09.179 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:46:09.189 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:46:09.189 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:46:09.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:46:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.324+0000 7fbd33af0700 1 Processor -- start 2023-06-10T18:46:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.324+0000 7fbd33af0700 1 -- start start 2023-06-10T18:46:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd33af0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd33af0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd33af0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0b6070 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd33af0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2c0b5460 con 0x7fbd2c0cea60 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd33af0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0b5740 con 0x7fbd2c15a420 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd3208d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd3208d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57900/0 (socket says 172.21.15.104:57900) 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd3208d700 1 -- 172.21.15.104:0/92328410 learned_addr learned my addr 172.21.15.104:0/92328410 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.325+0000 7fbd3188c700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3208d700 1 -- 172.21.15.104:0/92328410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 msgr2=0x7fbd2c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:09.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3208d700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3208d700 1 -- 172.21.15.104:0/92328410 >> v1:172.21.15.114:6789/0 conn(0x7fbd2c0b4ff0 legacy=0x7fbd2c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3208d700 1 -- 172.21.15.104:0/92328410 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0b5a20 con 0x7fbd2c0cea60 2023-06-10T18:46:09.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3188c700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:09.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3208d700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009e50 tx=0x7fbd2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc7899f4d1cc28e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3088a700 1 -- 172.21.15.104:0/92328410 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2400e070 con 0x7fbd2c0cea60 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3088a700 1 -- 172.21.15.104:0/92328410 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2400b9c0 con 0x7fbd2c0cea60 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd3088a700 1 -- 172.21.15.104:0/92328410 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2400bd70 con 0x7fbd2c0cea60 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 -- 172.21.15.104:0/92328410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0cea60 msgr2=0x7fbd2c15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009e50 tx=0x7fbd2400b330 comp rx=0 tx=0).stop 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 -- 172.21.15.104:0/92328410 shutdown_connections 2023-06-10T18:46:09.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c15a7f0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 --2- 172.21.15.104:0/92328410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.326+0000 7fbd33af0700 1 -- 172.21.15.104:0/92328410 >> 172.21.15.104:0/92328410 conn(0x7fbd2c089fd0 msgr2=0x7fbd2c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.327+0000 7fbd33af0700 1 -- 172.21.15.104:0/92328410 shutdown_connections 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.327+0000 7fbd33af0700 1 -- 172.21.15.104:0/92328410 wait complete. 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 Processor -- start 2023-06-10T18:46:09.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 -- start start 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c1f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0c5260 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c1f3dd0 con 0x7fbd2c0cea60 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.328+0000 7fbd33af0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c0c4ca0 con 0x7fbd2c15a420 2023-06-10T18:46:09.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57902/0 (socket says 172.21.15.104:57902) 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 -- 172.21.15.104:0/2545535277 learned_addr learned my addr 172.21.15.104:0/2545535277 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3108b700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3208d700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c1f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 -- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 msgr2=0x7fbd2c1f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c1f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 -- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd2c0cea60 msgr2=0x7fbd2c149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 -- 172.21.15.104:0/2545535277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24009b30 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd3188c700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00e810 tx=0x7fbd1c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c00c020 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c004250 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0036a0 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c0c4f80 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.329+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c0c7cd0 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.330+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c0c8040 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.331+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbd1c003800 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.331+0000 7fbd227fc700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 0x7fbd1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.331+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fbd1c088f00 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.331+0000 7fbd3108b700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 0x7fbd1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.332+0000 7fbd3108b700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 0x7fbd1006d190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c15df60 tx=0x7fbd28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.335+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbd1c053ec0 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.523+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd2c0c7830 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.524+0000 7fbd227fc700 1 -- 172.21.15.104:0/2545535277 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7fbd1c0574e0 con 0x7fbd2c0b4ff0 2023-06-10T18:46:09.532 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:46:09.533 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:46:09.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 msgr2=0x7fbd1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 0x7fbd1006d190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c15df60 tx=0x7fbd28009040 comp rx=0 tx=0).stop 2023-06-10T18:46:09.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 msgr2=0x7fbd2c0c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00e810 tx=0x7fbd1c00eb20 comp rx=0 tx=0).stop 2023-06-10T18:46:09.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 shutdown_connections 2023-06-10T18:46:09.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd2c15a420 0x7fbd2c1f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd2c0b4ff0 0x7fbd2c0c42b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd1006ace0 0x7fbd1006d190 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 --2- 172.21.15.104:0/2545535277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd2c0cea60 0x7fbd2c149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 >> 172.21.15.104:0/2545535277 conn(0x7fbd2c089fd0 msgr2=0x7fbd2c1579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:09.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 shutdown_connections 2023-06-10T18:46:09.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.526+0000 7fbd33af0700 1 -- 172.21.15.104:0/2545535277 wait complete. 2023-06-10T18:46:09.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:45:39.741633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:45:40.245539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:46:09.545 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:46:09.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.663+0000 7fd27262e700 1 Processor -- start 2023-06-10T18:46:09.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.663+0000 7fd27262e700 1 -- start start 2023-06-10T18:46:09.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd27262e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd26c0d0a00 0x7fd26c164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd27262e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c0ad3c0 con 0x7fd26c0cea50 2023-06-10T18:46:09.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd27262e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c157f00 con 0x7fd26c157b30 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd27262e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26c0d0e60 con 0x7fd26c0d0a00 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd270bcb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd26c0d0a00 0x7fd26c164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd26c0cea50 0x7fd26c0d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55740/0 (socket says 172.21.15.104:55740) 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26b7fe700 1 -- 172.21.15.104:0/1304864005 learned_addr learned my addr 172.21.15.104:0/1304864005 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd270bcb700 1 -- 172.21.15.104:0/1304864005 >> v1:172.21.15.114:6789/0 conn(0x7fd26c157b30 legacy=0x7fd26c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd270bcb700 1 -- 172.21.15.104:0/1304864005 >> v1:172.21.15.104:6789/0 conn(0x7fd26c0cea50 legacy=0x7fd26c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd270bcb700 1 -- 172.21.15.104:0/1304864005 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0b0170 con 0x7fd26c0d0a00 2023-06-10T18:46:09.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26affd700 1 -- 172.21.15.104:0/1304864005 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961747469 0 0) 0x7fd26c0ad3c0 con 0x7fd26c0cea50 2023-06-10T18:46:09.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd270bcb700 1 --2- 172.21.15.104:0/1304864005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c0d0a00 0x7fd26c164e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd264009d90 tx=0x7fd26400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe26352f04480f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26affd700 1 -- 172.21.15.104:0/1304864005 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26400e070 con 0x7fd26c0d0a00 2023-06-10T18:46:09.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26affd700 1 -- 172.21.15.104:0/1304864005 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26400bb90 con 0x7fd26c0d0a00 2023-06-10T18:46:09.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.664+0000 7fd26affd700 1 -- 172.21.15.104:0/1304864005 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264003a20 con 0x7fd26c0d0a00 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 -- 172.21.15.104:0/1304864005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c0d0a00 msgr2=0x7fd26c164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 --2- 172.21.15.104:0/1304864005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c0d0a00 0x7fd26c164e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd264009d90 tx=0x7fd26400b330 comp rx=0 tx=0).stop 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 -- 172.21.15.104:0/1304864005 shutdown_connections 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 --2- 172.21.15.104:0/1304864005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c0d0a00 0x7fd26c164e60 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 -- 172.21.15.104:0/1304864005 >> 172.21.15.104:0/1304864005 conn(0x7fd26c089fd0 msgr2=0x7fd26c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:09.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 -- 172.21.15.104:0/1304864005 shutdown_connections 2023-06-10T18:46:09.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.665+0000 7fd27262e700 1 -- 172.21.15.104:0/1304864005 wait complete. 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.666+0000 7fd27262e700 1 Processor -- start 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.666+0000 7fd27262e700 1 -- start start 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 0x7fd26c0b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 0x7fd26c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0b3c50 con 0x7fd26c157b30 2023-06-10T18:46:09.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0b2b50 con 0x7fd26c0cea50 2023-06-10T18:46:09.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd27262e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0b3080 con 0x7fd26c0d0a00 2023-06-10T18:46:09.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57920/0 (socket says 172.21.15.104:57920) 2023-06-10T18:46:09.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 -- 172.21.15.104:0/2926318483 learned_addr learned my addr 172.21.15.104:0/2926318483 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd26bfff700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 0x7fd26c0b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 -- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 msgr2=0x7fd26c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd26b7fe700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 0x7fd26c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 0x7fd26c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 -- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 msgr2=0x7fd26c0b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 0x7fd26c0b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd270bcb700 1 -- 172.21.15.104:0/2926318483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264009a70 con 0x7fd26c157b30 2023-06-10T18:46:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd26bfff700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 0x7fd26c0b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:46:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.667+0000 7fd26b7fe700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 0x7fd26c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:46:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd270bcb700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd26c066690 tx=0x7fd264004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264012630 con 0x7fd26c157b30 2023-06-10T18:46:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2640127b0 con 0x7fd26c157b30 2023-06-10T18:46:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c0b36d0 con 0x7fd26c157b30 2023-06-10T18:46:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264012b60 con 0x7fd26c157b30 2023-06-10T18:46:09.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.668+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c0b3970 con 0x7fd26c157b30 2023-06-10T18:46:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.669+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c074050 con 0x7fd26c157b30 2023-06-10T18:46:09.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.669+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd264012cc0 con 0x7fd26c157b30 2023-06-10T18:46:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.669+0000 7fd268ff9700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 0x7fd24c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.670+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd26408bc90 con 0x7fd26c157b30 2023-06-10T18:46:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.670+0000 7fd26bfff700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 0x7fd24c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:09.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.670+0000 7fd26bfff700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 0x7fd24c06d190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:09.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.674+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd26405a6e0 con 0x7fd26c157b30 2023-06-10T18:46:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.859+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd26c15a2f0 con 0x7fd26c157b30 2023-06-10T18:46:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.860+0000 7fd268ff9700 1 -- 172.21.15.104:0/2926318483 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd26405a270 con 0x7fd26c157b30 2023-06-10T18:46:09.865 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:46:09.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.861+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 msgr2=0x7fd24c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.861+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 0x7fd24c06d190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254008040 comp rx=0 tx=0).stop 2023-06-10T18:46:09.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.861+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 msgr2=0x7fd26c158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.861+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd26c066690 tx=0x7fd264004ad0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 shutdown_connections 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd24c06ace0 0x7fd24c06d190 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd26c0cea50 0x7fd26c0b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd26c157b30 0x7fd26c158640 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 --2- 172.21.15.104:0/2926318483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd26c0d0a00 0x7fd26c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 >> 172.21.15.104:0/2926318483 conn(0x7fd26c089fd0 msgr2=0x7fd26c152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 shutdown_connections 2023-06-10T18:46:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.862+0000 7fd27262e700 1 -- 172.21.15.104:0/2926318483 wait complete. 2023-06-10T18:46:09.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:46:10.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.995+0000 7ffbcadc3700 1 Processor -- start 2023-06-10T18:46:10.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.995+0000 7ffbcadc3700 1 -- start start 2023-06-10T18:46:10.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbcadc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc40d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbcadc3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbc40b05d0 0x7ffbc40b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbcadc3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40af530 con 0x7ffbc40d04f0 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbcadc3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc40af810 con 0x7ffbc40b05d0 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbcadc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40afaf0 con 0x7ffbc40d25c0 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbc40b05d0 0x7ffbc40b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbc40b05d0 0x7ffbc40b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57928/0 (socket says 172.21.15.104:57928) 2023-06-10T18:46:10.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 -- 172.21.15.104:0/1568952427 learned_addr learned my addr 172.21.15.104:0/1568952427 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:10.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc8b5f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ffbc40d04f0 0x7ffbc40d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55752/0 (socket says 172.21.15.104:55752) 2023-06-10T18:46:10.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc3fff700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc40d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 -- 172.21.15.104:0/1568952427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 msgr2=0x7ffbc40d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:10.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc40d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 -- 172.21.15.104:0/1568952427 >> v1:172.21.15.104:6789/0 conn(0x7ffbc40d04f0 legacy=0x7ffbc40d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc37fe700 1 -- 172.21.15.104:0/1568952427 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361999651 0 0) 0x7ffbc40af530 con 0x7ffbc40d04f0 2023-06-10T18:46:10.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.996+0000 7ffbc9360700 1 -- 172.21.15.104:0/1568952427 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40afdd0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbc9360700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40b0a50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009e50 tx=0x7ffbbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62ae8a69d784245 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbc37fe700 1 -- 172.21.15.104:0/1568952427 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00e070 con 0x7ffbc40b05d0 2023-06-10T18:46:10.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbc37fe700 1 -- 172.21.15.104:0/1568952427 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc00b9c0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbc37fe700 1 -- 172.21.15.104:0/1568952427 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00bd70 con 0x7ffbc40b05d0 2023-06-10T18:46:10.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 -- 172.21.15.104:0/1568952427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 msgr2=0x7ffbc40b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40b0a50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009e50 tx=0x7ffbbc00b330 comp rx=0 tx=0).stop 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 -- 172.21.15.104:0/1568952427 shutdown_connections 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40b0a50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/1568952427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc40d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 -- 172.21.15.104:0/1568952427 >> 172.21.15.104:0/1568952427 conn(0x7ffbc4071ec0 msgr2=0x7ffbc409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 -- 172.21.15.104:0/1568952427 shutdown_connections 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.997+0000 7ffbcadc3700 1 -- 172.21.15.104:0/1568952427 wait complete. 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 Processor -- start 2023-06-10T18:46:10.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 -- start start 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc40d04f0 0x7ffbc414e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc414ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc414f3b0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40cfbb0 con 0x7ffbc40d04f0 2023-06-10T18:46:10.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:09.999+0000 7ffbcadc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41e3a10 con 0x7ffbc40d25c0 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57930/0 (socket says 172.21.15.104:57930) 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 -- 172.21.15.104:0/3437699792 learned_addr learned my addr 172.21.15.104:0/3437699792 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc9360700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc414ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 -- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 msgr2=0x7ffbc414ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc414ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 -- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc40d04f0 msgr2=0x7ffbc414e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc40d04f0 0x7ffbc414e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 -- 172.21.15.104:0/3437699792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc009b30 con 0x7ffbc40b05d0 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc8b5f700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c570 tx=0x7ffbb800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80074e0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb8007660 con 0x7ffbc40b05d0 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41e3cf0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8005a70 con 0x7ffbc40b05d0 2023-06-10T18:46:10.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.000+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41e42b0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.001+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4067010 con 0x7ffbc40b05d0 2023-06-10T18:46:10.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.002+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffbb8005bd0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.002+0000 7ffbc17fa700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 0x7ffba806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.002+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ffbb8088b70 con 0x7ffbc40b05d0 2023-06-10T18:46:10.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.002+0000 7ffbc3fff700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 0x7ffba806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.003+0000 7ffbc3fff700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 0x7ffba806d190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffbc41e6530 tx=0x7ffbb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.006+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffbb8053b30 con 0x7ffbc40b05d0 2023-06-10T18:46:10.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.191+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffbc4074050 con 0x7ffbc40b05d0 2023-06-10T18:46:10.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.192+0000 7ffbc17fa700 1 -- 172.21.15.104:0/3437699792 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffbb80170a0 con 0x7ffbc40b05d0 2023-06-10T18:46:10.197 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:46:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.193+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 msgr2=0x7ffba806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.193+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 0x7ffba806d190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffbc41e6530 tx=0x7ffbb0008040 comp rx=0 tx=0).stop 2023-06-10T18:46:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.193+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 msgr2=0x7ffbc40cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.193+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c570 tx=0x7ffbb800c930 comp rx=0 tx=0).stop 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 shutdown_connections 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc40d25c0 0x7ffbc414ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbc40b05d0 0x7ffbc40cd030 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffba806ace0 0x7ffba806d190 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 --2- 172.21.15.104:0/3437699792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc40d04f0 0x7ffbc414e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 >> 172.21.15.104:0/3437699792 conn(0x7ffbc4071ec0 msgr2=0x7ffbc40ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 shutdown_connections 2023-06-10T18:46:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.194+0000 7ffbcadc3700 1 -- 172.21.15.104:0/3437699792 wait complete. 2023-06-10T18:46:10.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:46:10.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.328+0000 7fba9e461700 1 Processor -- start 2023-06-10T18:46:10.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.328+0000 7fba9e461700 1 -- start start 2023-06-10T18:46:10.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba980aebe0 0x7fba98159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 0x7fba980cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 0x7fba980cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba98071450 con 0x7fba980aebe0 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9815d0d0 con 0x7fba980cd9a0 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba9e461700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9815d3b0 con 0x7fba9815c930 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba97fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba980aebe0 0x7fba98159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba97fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba980aebe0 0x7fba98159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57938/0 (socket says 172.21.15.104:57938) 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba97fff700 1 -- 172.21.15.104:0/1623211556 learned_addr learned my addr 172.21.15.104:0/1623211556 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.329+0000 7fba977fe700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 0x7fba980cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba9c9fe700 1 --2- 172.21.15.104:0/1623211556 >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 0x7fba980cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 -- 172.21.15.104:0/1623211556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 msgr2=0x7fba980cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 0x7fba980cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 -- 172.21.15.104:0/1623211556 >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 msgr2=0x7fba980cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 --2- 172.21.15.104:0/1623211556 >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 0x7fba980cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 -- 172.21.15.104:0/1623211556 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9815d690 con 0x7fba980aebe0 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba977fe700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 0x7fba980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:10.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba9c9fe700 1 --2- 172.21.15.104:0/1623211556 >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 0x7fba980cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba97fff700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba980aebe0 0x7fba98159ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba9806f300 tx=0x7fba88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9c993b7baf2558 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba96ffd700 1 -- 172.21.15.104:0/1623211556 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8800b870 con 0x7fba980aebe0 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba96ffd700 1 -- 172.21.15.104:0/1623211556 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8800b9d0 con 0x7fba980aebe0 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.330+0000 7fba96ffd700 1 -- 172.21.15.104:0/1623211556 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8800bd60 con 0x7fba980aebe0 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 -- 172.21.15.104:0/1623211556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba980aebe0 msgr2=0x7fba98159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba980aebe0 0x7fba98159ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba9806f300 tx=0x7fba88009bd0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 -- 172.21.15.104:0/1623211556 shutdown_connections 2023-06-10T18:46:10.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba980aebe0 0x7fba98159ee0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 --2- 172.21.15.104:0/1623211556 >> v2:172.21.15.114:3300/0 conn(0x7fba980cd9a0 0x7fba980cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 --2- 172.21.15.104:0/1623211556 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c930 0x7fba980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 -- 172.21.15.104:0/1623211556 >> 172.21.15.104:0/1623211556 conn(0x7fba9806d240 msgr2=0x7fba9806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 -- 172.21.15.104:0/1623211556 shutdown_connections 2023-06-10T18:46:10.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.331+0000 7fba9e461700 1 -- 172.21.15.104:0/1623211556 wait complete. 2023-06-10T18:46:10.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.332+0000 7fba9e461700 1 Processor -- start 2023-06-10T18:46:10.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.332+0000 7fba9e461700 1 -- start start 2023-06-10T18:46:10.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba980aebe0 0x7fba980c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba980cd9a0 0x7fba9813d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c4870 con 0x7fba9815c930 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c4b10 con 0x7fba980cd9a0 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c4df0 con 0x7fba980aebe0 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57948/0 (socket says 172.21.15.104:57948) 2023-06-10T18:46:10.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 -- 172.21.15.104:0/1063574035 learned_addr learned my addr 172.21.15.104:0/1063574035 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:10.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba977fe700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba980cd9a0 0x7fba9813d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba97fff700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba980aebe0 0x7fba980c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 -- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba980aebe0 msgr2=0x7fba980c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba980aebe0 0x7fba980c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 -- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba980cd9a0 msgr2=0x7fba9813d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba980cd9a0 0x7fba9813d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 -- 172.21.15.104:0/1063574035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88009550 con 0x7fba9815c930 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9c9fe700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fba90007dc0 tx=0x7fba9000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000ddd0 con 0x7fba9815c930 2023-06-10T18:46:10.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9000f040 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba980c50d0 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90015640 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.333+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba980c5690 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.334+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba980c5880 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.335+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba9001e3b0 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.335+0000 7fba94ff9700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 0x7fba8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.335+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fba90089860 con 0x7fba9815c930 2023-06-10T18:46:10.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.336+0000 7fba97fff700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 0x7fba8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.336+0000 7fba97fff700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 0x7fba8406d190 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fba88000c00 tx=0x7fba88008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.340+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba90054680 con 0x7fba9815c930 2023-06-10T18:46:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.523+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba98074050 con 0x7fba9815c930 2023-06-10T18:46:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.524+0000 7fba94ff9700 1 -- 172.21.15.104:0/1063574035 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba90057ca0 con 0x7fba9815c930 2023-06-10T18:46:10.530 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 msgr2=0x7fba8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 0x7fba8406d190 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fba88000c00 tx=0x7fba88008030 comp rx=0 tx=0).stop 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 msgr2=0x7fba980c2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fba90007dc0 tx=0x7fba9000d3b0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 shutdown_connections 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9815c930 0x7fba980c2370 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba980aebe0 0x7fba980c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba8406ace0 0x7fba8406d190 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 --2- 172.21.15.104:0/1063574035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba980cd9a0 0x7fba9813d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 >> 172.21.15.104:0/1063574035 conn(0x7fba9806d240 msgr2=0x7fba980ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 shutdown_connections 2023-06-10T18:46:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.526+0000 7fba9e461700 1 -- 172.21.15.104:0/1063574035 wait complete. 2023-06-10T18:46:10.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:46:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.661+0000 7fb5044dd700 1 Processor -- start 2023-06-10T18:46:10.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.661+0000 7fb5044dd700 1 -- start start 2023-06-10T18:46:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.662+0000 7fb5044dd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4fc0aea00 0x7fb4fc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.662+0000 7fb5044dd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.662+0000 7fb5044dd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc06ebb0 con 0x7fb4fc06cd50 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.662+0000 7fb5044dd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc159500 con 0x7fb4fc0caea0 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.662+0000 7fb5044dd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc071520 con 0x7fb4fc0aea00 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501a78700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb4fc06cd50 0x7fb4fc0cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55780/0 (socket says 172.21.15.104:55780) 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57954/0 (socket says 172.21.15.104:57954) 2023-06-10T18:46:10.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501a78700 1 -- 172.21.15.104:0/1828387754 learned_addr learned my addr 172.21.15.104:0/1828387754 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502279700 1 --2- 172.21.15.104:0/1828387754 >> v2:172.21.15.114:3300/0 conn(0x7fb4fc0aea00 0x7fb4fc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 -- 172.21.15.104:0/1828387754 >> v2:172.21.15.114:3300/0 conn(0x7fb4fc0aea00 msgr2=0x7fb4fc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 --2- 172.21.15.104:0/1828387754 >> v2:172.21.15.114:3300/0 conn(0x7fb4fc0aea00 0x7fb4fc0cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 -- 172.21.15.104:0/1828387754 >> v1:172.21.15.104:6789/0 conn(0x7fb4fc06cd50 legacy=0x7fb4fc0cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501277700 1 -- 172.21.15.104:0/1828387754 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897093990 0 0) 0x7fb4fc06ebb0 con 0x7fb4fc06cd50 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 -- 172.21.15.104:0/1828387754 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc06dfa0 con 0x7fb4fc0caea0 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb502a7a700 1 --2- 172.21.15.104:0/1828387754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc070270 tx=0x7fb4f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5245bfa5f93cb2f9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501277700 1 -- 172.21.15.104:0/1828387754 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f400e070 con 0x7fb4fc0caea0 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501277700 1 -- 172.21.15.104:0/1828387754 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f400bb20 con 0x7fb4fc0caea0 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.663+0000 7fb501277700 1 -- 172.21.15.104:0/1828387754 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4012430 con 0x7fb4fc0caea0 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 -- 172.21.15.104:0/1828387754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 msgr2=0x7fb4fc0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 --2- 172.21.15.104:0/1828387754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc070270 tx=0x7fb4f400b330 comp rx=0 tx=0).stop 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 -- 172.21.15.104:0/1828387754 shutdown_connections 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 --2- 172.21.15.104:0/1828387754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0cd7a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 --2- 172.21.15.104:0/1828387754 >> v2:172.21.15.114:3300/0 conn(0x7fb4fc0aea00 0x7fb4fc0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 -- 172.21.15.104:0/1828387754 >> 172.21.15.104:0/1828387754 conn(0x7fb4fc06b8a0 msgr2=0x7fb4fc0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 -- 172.21.15.104:0/1828387754 shutdown_connections 2023-06-10T18:46:10.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.664+0000 7fb5044dd700 1 -- 172.21.15.104:0/1828387754 wait complete. 2023-06-10T18:46:10.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.665+0000 7fb5044dd700 1 Processor -- start 2023-06-10T18:46:10.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.665+0000 7fb5044dd700 1 -- start start 2023-06-10T18:46:10.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4fc06cd50 0x7fb4fc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4fc0aea00 0x7fb4fc0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c9500 con 0x7fb4fc0caea0 2023-06-10T18:46:10.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c8b30 con 0x7fb4fc06cd50 2023-06-10T18:46:10.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb5044dd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c8200 con 0x7fb4fc0aea00 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb502a7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb502a7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57960/0 (socket says 172.21.15.104:57960) 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.666+0000 7fb502a7a700 1 -- 172.21.15.104:0/864512348 learned_addr learned my addr 172.21.15.104:0/864512348 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 -- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4fc0aea00 msgr2=0x7fb4fc0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4fc0aea00 0x7fb4fc0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 -- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4fc06cd50 msgr2=0x7fb4fc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502279700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4fc06cd50 0x7fb4fc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4fc06cd50 0x7fb4fc0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 -- 172.21.15.104:0/864512348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4009a70 con 0x7fb4fc0caea0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb502a7a700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0b2280 tx=0x7fb4f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4012430 con 0x7fb4fc0caea0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f4020070 con 0x7fb4fc0caea0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc0c84e0 con 0x7fb4fc0caea0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4004280 con 0x7fb4fc0caea0 2023-06-10T18:46:10.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.667+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc158200 con 0x7fb4fc0caea0 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.668+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc0ce980 con 0x7fb4fc0caea0 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.668+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb4f40043e0 con 0x7fb4fc0caea0 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.669+0000 7fb4f2ffd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 0x7fb4e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.669+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fb4f4022070 con 0x7fb4fc0caea0 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.669+0000 7fb502279700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 0x7fb4e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:10.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.669+0000 7fb502279700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 0x7fb4e006d190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0099f0 tx=0x7fb4ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:10.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.673+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4f40565e0 con 0x7fb4fc0caea0 2023-06-10T18:46:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.857+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4fc074050 con 0x7fb4fc0caea0 2023-06-10T18:46:10.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.858+0000 7fb4f2ffd700 1 -- 172.21.15.104:0/864512348 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb4f4059c00 con 0x7fb4fc0caea0 2023-06-10T18:46:10.864 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:46:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.859+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 msgr2=0x7fb4e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.859+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 0x7fb4e006d190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0099f0 tx=0x7fb4ec008040 comp rx=0 tx=0).stop 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.859+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 msgr2=0x7fb4fc0b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.859+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0b2280 tx=0x7fb4f4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 shutdown_connections 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb4e006ace0 0x7fb4e006d190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4fc06cd50 0x7fb4fc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4fc0caea0 0x7fb4fc0b4c10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 --2- 172.21.15.104:0/864512348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4fc0aea00 0x7fb4fc0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 >> 172.21.15.104:0/864512348 conn(0x7fb4fc06b8a0 msgr2=0x7fb4fc0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 shutdown_connections 2023-06-10T18:46:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.860+0000 7fb5044dd700 1 -- 172.21.15.104:0/864512348 wait complete. 2023-06-10T18:46:10.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:46:10.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 Processor -- start 2023-06-10T18:46:10.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 -- start start 2023-06-10T18:46:11.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0cdae0 con 0x7fc97c0b1080 2023-06-10T18:46:11.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0c9e60 con 0x7fc97c0cd680 2023-06-10T18:46:11.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.992+0000 7fc9824b3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0c9bc0 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc97c0b1080 0x7fc97c0ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55792/0 (socket says 172.21.15.104:55792) 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97b7fe700 1 -- 172.21.15.104:0/2752267933 learned_addr learned my addr 172.21.15.104:0/2752267933 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97bfff700 1 --2- 172.21.15.104:0/2752267933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637144700 0 0) 0x7fc97c0cdae0 con 0x7fc97c0b1080 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9640035b0 con 0x7fc97c0b1080 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773949297 0 0) 0x7fc97c0c9e60 con 0x7fc97c0cd680 2023-06-10T18:46:11.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c0cdae0 con 0x7fc97c0cd680 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002753652 0 0) 0x7fc9640035b0 con 0x7fc97c0b1080 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97bfff700 1 -- 172.21.15.104:0/2752267933 >> v1:172.21.15.114:6789/0 conn(0x7fc97c0cd680 legacy=0x7fc97c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97bfff700 1 -- 172.21.15.104:0/2752267933 >> v1:172.21.15.104:6789/0 conn(0x7fc97c0b1080 legacy=0x7fc97c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97bfff700 1 -- 172.21.15.104:0/2752267933 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c0aee70 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97bfff700 1 --2- 172.21.15.104:0/2752267933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c0cd140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009850 tx=0x7fc96c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c7b2c68a3593be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.993+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970001670 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c005c30 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc97affd700 1 -- 172.21.15.104:0/2752267933 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c004030 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 -- 172.21.15.104:0/2752267933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 msgr2=0x7fc97c0cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 --2- 172.21.15.104:0/2752267933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c0cd140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009850 tx=0x7fc96c009b60 comp rx=0 tx=0).stop 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 -- 172.21.15.104:0/2752267933 shutdown_connections 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 --2- 172.21.15.104:0/2752267933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c0cd140 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 -- 172.21.15.104:0/2752267933 >> 172.21.15.104:0/2752267933 conn(0x7fc97c06b8a0 msgr2=0x7fc97c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 -- 172.21.15.104:0/2752267933 shutdown_connections 2023-06-10T18:46:11.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.994+0000 7fc9824b3700 1 -- 172.21.15.104:0/2752267933 wait complete. 2023-06-10T18:46:11.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.995+0000 7fc9824b3700 1 Processor -- start 2023-06-10T18:46:11.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.995+0000 7fc9824b3700 1 -- start start 2023-06-10T18:46:11.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 0x7fc97c0c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0c52c0 con 0x7fc97c0b1080 2023-06-10T18:46:11.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c15c380 con 0x7fc97c0cd680 2023-06-10T18:46:11.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc9824b3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c152aa0 con 0x7fc97c0aeaa0 2023-06-10T18:46:11.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc97b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc97b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57972/0 (socket says 172.21.15.104:57972) 2023-06-10T18:46:11.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc97b7fe700 1 -- 172.21.15.104:0/3753567754 learned_addr learned my addr 172.21.15.104:0/3753567754 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:11.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc97bfff700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.996+0000 7fc980a50700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 0x7fc97c0c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 -- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 msgr2=0x7fc97c152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 -- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 msgr2=0x7fc97c0c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 0x7fc97c0c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 -- 172.21.15.104:0/3753567754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c009430 con 0x7fc97c0b1080 2023-06-10T18:46:11.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc980a50700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 0x7fc97c0c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:46:11.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc97b7fe700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc97000c7b0 tx=0x7fc97000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970007560 con 0x7fc97c0b1080 2023-06-10T18:46:11.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9700076c0 con 0x7fc97c0b1080 2023-06-10T18:46:11.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c153300 con 0x7fc97c0b1080 2023-06-10T18:46:11.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970004ab0 con 0x7fc97c0b1080 2023-06-10T18:46:11.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.997+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c0c4c00 con 0x7fc97c0b1080 2023-06-10T18:46:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.998+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c0c5400 con 0x7fc97c0b1080 2023-06-10T18:46:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.998+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc970004c10 con 0x7fc97c0b1080 2023-06-10T18:46:11.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.999+0000 7fc978ff9700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.999+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc970089380 con 0x7fc97c0b1080 2023-06-10T18:46:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.999+0000 7fc97bfff700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:10.999+0000 7fc97bfff700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009ba0 tx=0x7fc96c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.003+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc97c0c5400 con 0x7fc97c0b1080 2023-06-10T18:46:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.204+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc97c0c5400 con 0x7fc97c0b1080 2023-06-10T18:46:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.205+0000 7fc978ff9700 1 -- 172.21.15.104:0/3753567754 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc97c0c5400 con 0x7fc97c0b1080 2023-06-10T18:46:11.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 msgr2=0x7fc96806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009ba0 tx=0x7fc96c008070 comp rx=0 tx=0).stop 2023-06-10T18:46:11.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 msgr2=0x7fc97c0c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc97000c7b0 tx=0x7fc97000cac0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 shutdown_connections 2023-06-10T18:46:11.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ace0 0x7fc96806d190 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc97c0cd680 0x7fc97c0c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc97c0aeaa0 0x7fc97c152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 --2- 172.21.15.104:0/3753567754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc97c0b1080 0x7fc97c0c41d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 >> 172.21.15.104:0/3753567754 conn(0x7fc97c06b8a0 msgr2=0x7fc97c0cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:11.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 shutdown_connections 2023-06-10T18:46:11.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.207+0000 7fc9824b3700 1 -- 172.21.15.104:0/3753567754 wait complete. 2023-06-10T18:46:11.218 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:46:11.224 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:46:11.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:46:11.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:46:11.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[108371]: fuse finished with error 0 and tester_r 0 2023-06-10T18:46:11.270 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:46:11.342 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:46:11.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:46:11.343 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:46:11.363 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:46:11.363 DEBUG:teuthology.orchestra.run.smithi104:> import cephfs as libcephfs 2023-06-10T18:46:11.363 DEBUG:teuthology.orchestra.run.smithi104:> cephfs = libcephfs.LibCephFS(conffile='"'"''"'"') 2023-06-10T18:46:11.364 DEBUG:teuthology.orchestra.run.smithi104:> cephfs.mount() 2023-06-10T18:46:11.364 DEBUG:teuthology.orchestra.run.smithi104:> client_id = cephfs.get_instance_id() 2023-06-10T18:46:11.364 DEBUG:teuthology.orchestra.run.smithi104:> cephfs.abort_conn() 2023-06-10T18:46:11.365 DEBUG:teuthology.orchestra.run.smithi104:> print(client_id) 2023-06-10T18:46:11.365 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:46:11.475 INFO:teuthology.orchestra.run.smithi104.stdout:6024 2023-06-10T18:46:11.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:46:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.774+0000 7fab59831700 1 Processor -- start 2023-06-10T18:46:11.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 -- start start 2023-06-10T18:46:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 0x7fab5406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 0x7fab540cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5415cb00 con 0x7fab5406cd70 2023-06-10T18:46:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab540aed80 con 0x7fab5415c730 2023-06-10T18:46:11.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.775+0000 7fab59831700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab540c9c60 con 0x7fab540cc000 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 0x7fab540cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab52ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 0x7fab5406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab537fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fab5406cd70 0x7fab5406d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:33442/0 (socket says 172.21.15.104:33442) 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab537fe700 1 -- 172.21.15.104:0/3473597499 learned_addr learned my addr 172.21.15.104:0/3473597499 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 -- 172.21.15.104:0/3473597499 >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 msgr2=0x7fab5406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 --2- 172.21.15.104:0/3473597499 >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 0x7fab5406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 -- 172.21.15.104:0/3473597499 >> v1:172.21.15.114:6789/0 conn(0x7fab5406cd70 legacy=0x7fab5406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 -- 172.21.15.104:0/3473597499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5406ecd0 con 0x7fab540cc000 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab52ffd700 1 --2- 172.21.15.104:0/3473597499 >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 0x7fab5406e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab527fc700 1 --2- 172.21.15.104:0/3473597499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 0x7fab540cc470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab44009840 tx=0x7fab44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359b2a5ce588902b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab51ffb700 1 -- 172.21.15.104:0/3473597499 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011830 con 0x7fab540cc000 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab51ffb700 1 -- 172.21.15.104:0/3473597499 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab44011990 con 0x7fab540cc000 2023-06-10T18:46:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.776+0000 7fab51ffb700 1 -- 172.21.15.104:0/3473597499 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011d20 con 0x7fab540cc000 2023-06-10T18:46:11.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 -- 172.21.15.104:0/3473597499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 msgr2=0x7fab540cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 --2- 172.21.15.104:0/3473597499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 0x7fab540cc470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab44009840 tx=0x7fab44009b50 comp rx=0 tx=0).stop 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 -- 172.21.15.104:0/3473597499 shutdown_connections 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 --2- 172.21.15.104:0/3473597499 >> v2:172.21.15.114:3300/0 conn(0x7fab5415c730 0x7fab5406e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 --2- 172.21.15.104:0/3473597499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab540cc000 0x7fab540cc470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 -- 172.21.15.104:0/3473597499 >> 172.21.15.104:0/3473597499 conn(0x7fab5406b8a0 msgr2=0x7fab540cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 -- 172.21.15.104:0/3473597499 shutdown_connections 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.777+0000 7fab59831700 1 -- 172.21.15.104:0/3473597499 wait complete. 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.778+0000 7fab59831700 1 Processor -- start 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.778+0000 7fab59831700 1 -- start start 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab540cc000 0x7fab540c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab5415c730 0x7fab541557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b0e30 con 0x7fab5406cd70 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b1ba0 con 0x7fab540cc000 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab59831700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab540b1e80 con 0x7fab5415c730 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58002/0 (socket says 172.21.15.104:58002) 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 -- 172.21.15.104:0/2402002026 learned_addr learned my addr 172.21.15.104:0/2402002026 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab537fe700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab5415c730 0x7fab541557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab527fc700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab540cc000 0x7fab540c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 -- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab5415c730 msgr2=0x7fab541557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab5415c730 0x7fab541557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 -- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab540cc000 msgr2=0x7fab540c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab540cc000 0x7fab540c2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.779+0000 7fab52ffd700 1 -- 172.21.15.104:0/2402002026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c009430 con 0x7fab5406cd70 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab52ffd700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00ebe0 tx=0x7fab3c009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c005a20 con 0x7fab5406cd70 2023-06-10T18:46:11.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c005b80 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab440094a0 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00c910 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.780+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab540b0a50 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.781+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab54074050 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.781+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fab3c00ca70 con 0x7fab5406cd70 2023-06-10T18:46:11.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.781+0000 7fab4bfff700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 0x7fab3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:46:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.782+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fab3c0898c0 con 0x7fab5406cd70 2023-06-10T18:46:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.782+0000 7fab527fc700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 0x7fab3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:46:11.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.782+0000 7fab527fc700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 0x7fab3406d190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab44008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:46:11.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:11.786+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fab3c058310 con 0x7fab5406cd70 2023-06-10T18:46:12.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.013+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab540c2d20 con 0x7fab5406cd70 2023-06-10T18:46:12.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.014+0000 7fab4bfff700 1 -- 172.21.15.104:0/2402002026 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7fab3c057ea0 con 0x7fab5406cd70 2023-06-10T18:46:12.020 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:46:12.020 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:46:12.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 msgr2=0x7fab3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 0x7fab3406d190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab44008030 comp rx=0 tx=0).stop 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 msgr2=0x7fab5414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00ebe0 tx=0x7fab3c009ea0 comp rx=0 tx=0).stop 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 shutdown_connections 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab5415c730 0x7fab541557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab5406cd70 0x7fab5414cb60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab3406ace0 0x7fab3406d190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 --2- 172.21.15.104:0/2402002026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab540cc000 0x7fab540c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 >> 172.21.15.104:0/2402002026 conn(0x7fab5406b8a0 msgr2=0x7fab540cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 shutdown_connections 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:46:12.016+0000 7fab59831700 1 -- 172.21.15.104:0/2402002026 wait complete. 2023-06-10T18:46:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:46:12.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2023-06-10T18:46:12.186 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":6024,"entity":{"name":{"type":"client","num":6024},"addr":{"type":"any","addr":"172.21.15.104:0","nonce":2619019344}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":0.70198433999999998,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.99838399979605619,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6024 172.21.15.104:0/2619019344","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"admin","hostname":"smithi104","pid":"121214","root":"/"}}] 2023-06-10T18:46:12.186 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "admin", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "121214", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.104:0", "nonce": 2619019344, "type": "any" }, "name": { "num": 6024, "type": "client" } }, "id": 6024, "inst": "client.6024 172.21.15.104:0/2619019344", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9983839997960562 }, "state": "open", "uptime": 0.70198434 } ] 2023-06-10T18:46:37.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:46:37.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:47:07.932 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:47:07.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:47:38.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:47:38.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:47:42.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.315+0000 7fd31cade700 1 Processor -- start 2023-06-10T18:47:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.315+0000 7fd31cade700 1 -- start start 2023-06-10T18:47:42.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31cade700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd31815c500 0x7fd3180cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31cade700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31806dcb0 con 0x7fd318069e70 2023-06-10T18:47:42.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31cade700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3180ce470 con 0x7fd3180ce040 2023-06-10T18:47:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31cade700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31806e850 con 0x7fd31815c500 2023-06-10T18:47:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd316d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd318069e70 0x7fd3180c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55842/0 (socket says 172.21.15.104:55842) 2023-06-10T18:47:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd316d9d700 1 -- 172.21.15.104:0/1085776887 learned_addr learned my addr 172.21.15.104:0/1085776887 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045137742 0 0) 0x7fd31806dcb0 con 0x7fd318069e70 2023-06-10T18:47:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31659c700 1 --2- 172.21.15.104:0/1085776887 >> v2:172.21.15.114:3300/0 conn(0x7fd31815c500 0x7fd3180cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3000035b0 con 0x7fd318069e70 2023-06-10T18:47:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300930503 0 0) 0x7fd3000035b0 con 0x7fd318069e70 2023-06-10T18:47:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 >> v2:172.21.15.114:3300/0 conn(0x7fd31815c500 msgr2=0x7fd3180cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 --2- 172.21.15.104:0/1085776887 >> v2:172.21.15.114:3300/0 conn(0x7fd31815c500 0x7fd3180cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 >> v1:172.21.15.114:6789/0 conn(0x7fd3180ce040 legacy=0x7fd31806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31806d0a0 con 0x7fd318069e70 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.316+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd3100019f0 con 0x7fd318069e70 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 >> v1:172.21.15.104:6789/0 conn(0x7fd318069e70 legacy=0x7fd3180c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd300003a00 0x7fd300003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd300004850 0x7fd300004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd30000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31806e850 con 0x7fd300005200 2023-06-10T18:47:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000c060 con 0x7fd300003a00 2023-06-10T18:47:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000c220 con 0x7fd300004850 2023-06-10T18:47:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd30000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31659c700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd300003a00 0x7fd300003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd315d9b700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd300004850 0x7fd300004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 -- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd300004850 msgr2=0x7fd300004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd300004850 0x7fd300004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 -- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd300003a00 msgr2=0x7fd300003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd300003a00 0x7fd300003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 -- 172.21.15.104:0/1085776887 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000035b0 con 0x7fd300005200 2023-06-10T18:47:42.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 -- 172.21.15.104:0/1085776887 reap_dead start 2023-06-10T18:47:42.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd316d9d700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd30000ba10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd3100013d0 tx=0x7fd300005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93383d84d5bf3a00 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:42.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310012600 con 0x7fd300005200 2023-06-10T18:47:42.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3100127a0 con 0x7fd300005200 2023-06-10T18:47:42.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.317+0000 7fd31559a700 1 -- 172.21.15.104:0/1085776887 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310012b50 con 0x7fd300005200 2023-06-10T18:47:42.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 -- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 msgr2=0x7fd30000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd30000ba10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd3100013d0 tx=0x7fd300005880 comp rx=0 tx=0).stop 2023-06-10T18:47:42.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 -- 172.21.15.104:0/1085776887 shutdown_connections 2023-06-10T18:47:42.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 --2- 172.21.15.104:0/1085776887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd30000ba10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 -- 172.21.15.104:0/1085776887 >> 172.21.15.104:0/1085776887 conn(0x7fd31806b8a0 msgr2=0x7fd3180cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:42.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 -- 172.21.15.104:0/1085776887 shutdown_connections 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.318+0000 7fd31cade700 1 -- 172.21.15.104:0/1085776887 wait complete. 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.319+0000 7fd31cade700 1 Processor -- start 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.319+0000 7fd31cade700 1 -- start start 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3181576f0 0x7fd318158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd318158960 0x7fd31809ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318157260 con 0x7fd300005200 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318156650 con 0x7fd3181576f0 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31cade700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd318156930 con 0x7fd318158960 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58022/0 (socket says 172.21.15.104:58022) 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 -- 172.21.15.104:0/3451008260 learned_addr learned my addr 172.21.15.104:0/3451008260 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 -- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd318158960 msgr2=0x7fd31809ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd318158960 0x7fd31809ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 -- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3181576f0 msgr2=0x7fd318158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3181576f0 0x7fd318158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.320+0000 7fd31659c700 1 -- 172.21.15.104:0/3451008260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318156c10 con 0x7fd300005200 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd31659c700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd308009fd0 tx=0x7fd308011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308009610 con 0x7fd300005200 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308005c30 con 0x7fd300005200 2023-06-10T18:47:42.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd318156ef0 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800f730 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.321+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31809b040 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.322+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318157540 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.322+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd30800f890 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.323+0000 7fd306ffd700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 0x7fd2f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.323+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd308089ed0 con 0x7fd300005200 2023-06-10T18:47:42.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.323+0000 7fd315d9b700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 0x7fd2f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:42.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.324+0000 7fd315d9b700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 0x7fd2f806d190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:42.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.327+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd308058940 con 0x7fd300005200 2023-06-10T18:47:42.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.554+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd318074050 con 0x7fd300005200 2023-06-10T18:47:42.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.555+0000 7fd306ffd700 1 -- 172.21.15.104:0/3451008260 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7fd3080584d0 con 0x7fd300005200 2023-06-10T18:47:42.561 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 msgr2=0x7fd2f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 0x7fd2f806d190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c009040 comp rx=0 tx=0).stop 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 msgr2=0x7fd318147a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd308009fd0 tx=0x7fd308011d30 comp rx=0 tx=0).stop 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 shutdown_connections 2023-06-10T18:47:42.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd318158960 0x7fd31809ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd300005200 0x7fd318147a30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd2f806ace0 0x7fd2f806d190 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 --2- 172.21.15.104:0/3451008260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3181576f0 0x7fd318158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 >> 172.21.15.104:0/3451008260 conn(0x7fd31806b8a0 msgr2=0x7fd3180cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 shutdown_connections 2023-06-10T18:47:42.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:42.557+0000 7fd31cade700 1 -- 172.21.15.104:0/3451008260 wait complete. 2023-06-10T18:47:42.565 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:47:42.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2023-06-10T18:47:42.727 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":6024,"entity":{"name":{"type":"client","num":6024},"addr":{"type":"any","addr":"172.21.15.104:0","nonce":2619019344}},"state":"stale","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":91.243976771000007,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.80992491760506002,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6024 172.21.15.104:0/2619019344","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"admin","hostname":"smithi104","pid":"121214","root":"/"}}] 2023-06-10T18:47:42.728 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "admin", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "121214", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.104:0", "nonce": 2619019344, "type": "any" }, "name": { "num": 6024, "type": "client" } }, "id": 6024, "inst": "client.6024 172.21.15.104:0/2619019344", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.80992491760506 }, "state": "stale", "uptime": 91.243976771 } ] 2023-06-10T18:47:42.728 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:47:42.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:47:42.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:47:42.810 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:47:42.811 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:47:42.811 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:42.829 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:47:42.830 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:47:42.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:47:42.830 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:47:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:47:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-06-10T18:47:42.886 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:47:42.887 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:42.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:47:42.904 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:47:42.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:47:42.905 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:47:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:47:42.961 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-06-10T18:47:42.961 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:47:42.962 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:42.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:47:42.979 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:47:42.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:47:42.979 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:47:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:47:43.035 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:47:43.036 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:47:43.036 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:43.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:47:43.055 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:47:43.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:47:43.056 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:47:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:47:43.111 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:47:43.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:43.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.225+0000 7f6b04358700 1 Processor -- start 2023-06-10T18:47:43.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.225+0000 7f6b04358700 1 -- start start 2023-06-10T18:47:43.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b04358700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6afc070a20 0x7f6afc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b04358700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6afc070df0 0x7f6afc1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b04358700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc06ae00 con 0x7f6afc158ce0 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b04358700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc069e70 con 0x7f6afc070a20 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b04358700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc1595f0 con 0x7f6afc070df0 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b020f4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6afc070a20 0x7f6afc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b028f5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6afc158ce0 0x7f6afc159c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55862/0 (socket says 172.21.15.104:55862) 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b028f5700 1 -- 172.21.15.104:0/269636719 learned_addr learned my addr 172.21.15.104:0/269636719 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:43.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b018f3700 1 --2- 172.21.15.104:0/269636719 >> v2:172.21.15.114:3300/0 conn(0x7f6afc070df0 0x7f6afc1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b020f4700 1 -- 172.21.15.104:0/269636719 >> v2:172.21.15.114:3300/0 conn(0x7f6afc070df0 msgr2=0x7f6afc1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b020f4700 1 --2- 172.21.15.104:0/269636719 >> v2:172.21.15.114:3300/0 conn(0x7f6afc070df0 0x7f6afc1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b020f4700 1 -- 172.21.15.104:0/269636719 >> v1:172.21.15.104:6789/0 conn(0x7f6afc158ce0 legacy=0x7f6afc159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b020f4700 1 -- 172.21.15.104:0/269636719 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc1598d0 con 0x7f6afc070a20 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.226+0000 7f6b010f2700 1 -- 172.21.15.104:0/269636719 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407601362 0 0) 0x7f6af4001490 con 0x7f6afc158ce0 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b020f4700 1 --2- 172.21.15.104:0/269636719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070a20 0x7f6afc06a8c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6afc06df00 tx=0x7f6aec00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf57f3ebd14117ae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b010f2700 1 -- 172.21.15.104:0/269636719 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00db60 con 0x7f6afc070a20 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b010f2700 1 -- 172.21.15.104:0/269636719 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec00dcc0 con 0x7f6afc070a20 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b010f2700 1 -- 172.21.15.104:0/269636719 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec017630 con 0x7f6afc070a20 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 -- 172.21.15.104:0/269636719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070a20 msgr2=0x7f6afc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 --2- 172.21.15.104:0/269636719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070a20 0x7f6afc06a8c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6afc06df00 tx=0x7f6aec00bdc0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 -- 172.21.15.104:0/269636719 shutdown_connections 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 --2- 172.21.15.104:0/269636719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070a20 0x7f6afc06a8c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 --2- 172.21.15.104:0/269636719 >> v2:172.21.15.114:3300/0 conn(0x7f6afc070df0 0x7f6afc1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 -- 172.21.15.104:0/269636719 >> 172.21.15.104:0/269636719 conn(0x7f6afc0b1610 msgr2=0x7f6afc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:43.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 -- 172.21.15.104:0/269636719 shutdown_connections 2023-06-10T18:47:43.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.227+0000 7f6b04358700 1 -- 172.21.15.104:0/269636719 wait complete. 2023-06-10T18:47:43.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 Processor -- start 2023-06-10T18:47:43.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 -- start start 2023-06-10T18:47:43.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6afc070a20 0x7f6afc152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 0x7f6afc0b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc1f1ca0 con 0x7f6afc070df0 2023-06-10T18:47:43.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0b06e0 con 0x7f6afc070a20 2023-06-10T18:47:43.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b04358700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0b0400 con 0x7f6afc158ce0 2023-06-10T18:47:43.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b018f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b018f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58038/0 (socket says 172.21.15.104:58038) 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b018f3700 1 -- 172.21.15.104:0/338400544 learned_addr learned my addr 172.21.15.104:0/338400544 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.229+0000 7f6b020f4700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6afc070a20 0x7f6afc152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b028f5700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 0x7f6afc0b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 -- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 msgr2=0x7f6afc0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 0x7f6afc0b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 -- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6afc070a20 msgr2=0x7f6afc152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6afc070a20 0x7f6afc152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 -- 172.21.15.104:0/338400544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec00b740 con 0x7f6afc070df0 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b028f5700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 0x7f6afc0b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b018f3700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e810 tx=0x7f6af800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8013610 con 0x7f6afc070df0 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8013770 con 0x7f6afc070df0 2023-06-10T18:47:43.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc0c7960 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8013b20 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.230+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc152d80 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.231+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc0c7fb0 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.232+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6af8013c80 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.232+0000 7f6af2ffd700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 0x7f6ae006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.232+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f6af80891f0 con 0x7f6afc070df0 2023-06-10T18:47:43.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.232+0000 7f6b020f4700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 0x7f6ae006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.232+0000 7f6b020f4700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 0x7f6ae006d190 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6aec019040 tx=0x7f6aec014000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.236+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6af8057bc0 con 0x7f6afc070df0 2023-06-10T18:47:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.465+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6afc071780 con 0x7f6afc070df0 2023-06-10T18:47:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.466+0000 7f6af2ffd700 1 -- 172.21.15.104:0/338400544 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f6af8057750 con 0x7f6afc070df0 2023-06-10T18:47:43.471 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:43.472 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:47:43.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.467+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 msgr2=0x7f6ae006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.467+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 0x7f6ae006d190 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6aec019040 tx=0x7f6aec014000 comp rx=0 tx=0).stop 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.467+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 msgr2=0x7f6afc0c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.467+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e810 tx=0x7f6af800eb20 comp rx=0 tx=0).stop 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 shutdown_connections 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ae006ace0 0x7f6ae006d190 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6afc070a20 0x7f6afc152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6afc158ce0 0x7f6afc0b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 --2- 172.21.15.104:0/338400544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6afc070df0 0x7f6afc0c8160 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 >> 172.21.15.104:0/338400544 conn(0x7f6afc0b1610 msgr2=0x7f6afc06d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 shutdown_connections 2023-06-10T18:47:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.468+0000 7f6b04358700 1 -- 172.21.15.104:0/338400544 wait complete. 2023-06-10T18:47:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:47:43.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:43.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.599+0000 7fbb494b4700 1 Processor -- start 2023-06-10T18:47:43.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.599+0000 7fbb494b4700 1 -- start start 2023-06-10T18:47:43.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb494b4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4415a420 0x7fbb440b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb494b4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 0x7fbb440ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb494b4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb440b6070 con 0x7fbb440ce890 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb494b4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440cecb0 con 0x7fbb4415a420 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb494b4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440b5460 con 0x7fbb440b4ff0 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb42ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4415a420 0x7fbb440b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb42ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4415a420 0x7fbb440b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58044/0 (socket says 172.21.15.104:58044) 2023-06-10T18:47:43.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb42ffd700 1 -- 172.21.15.104:0/768500309 learned_addr learned my addr 172.21.15.104:0/768500309 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.600+0000 7fbb427fc700 1 --2- 172.21.15.104:0/768500309 >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 0x7fbb440ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb42ffd700 1 -- 172.21.15.104:0/768500309 >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 msgr2=0x7fbb440ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb42ffd700 1 --2- 172.21.15.104:0/768500309 >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 0x7fbb440ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb42ffd700 1 -- 172.21.15.104:0/768500309 >> v1:172.21.15.114:6789/0 conn(0x7fbb440ce890 legacy=0x7fbb440b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb42ffd700 1 -- 172.21.15.104:0/768500309 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb440b5740 con 0x7fbb4415a420 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb427fc700 1 --2- 172.21.15.104:0/768500309 >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 0x7fbb440ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:43.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb42ffd700 1 --2- 172.21.15.104:0/768500309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb440b1660 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbb44066590 tx=0x7fbb38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bc27cc7ac3ded5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb41ffb700 1 -- 172.21.15.104:0/768500309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800b870 con 0x7fbb4415a420 2023-06-10T18:47:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb41ffb700 1 -- 172.21.15.104:0/768500309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3800b9d0 con 0x7fbb4415a420 2023-06-10T18:47:43.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb41ffb700 1 -- 172.21.15.104:0/768500309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800bd60 con 0x7fbb4415a420 2023-06-10T18:47:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 -- 172.21.15.104:0/768500309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 msgr2=0x7fbb440b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 --2- 172.21.15.104:0/768500309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb440b1660 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbb44066590 tx=0x7fbb38009bd0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 -- 172.21.15.104:0/768500309 shutdown_connections 2023-06-10T18:47:43.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 --2- 172.21.15.104:0/768500309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb440b1660 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 --2- 172.21.15.104:0/768500309 >> v2:172.21.15.114:3300/0 conn(0x7fbb440b4ff0 0x7fbb440ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 -- 172.21.15.104:0/768500309 >> 172.21.15.104:0/768500309 conn(0x7fbb44089fd0 msgr2=0x7fbb440d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.601+0000 7fbb494b4700 1 -- 172.21.15.104:0/768500309 shutdown_connections 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.602+0000 7fbb494b4700 1 -- 172.21.15.104:0/768500309 wait complete. 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.603+0000 7fbb494b4700 1 Processor -- start 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.603+0000 7fbb494b4700 1 -- start start 2023-06-10T18:47:43.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb440b4ff0 0x7fbb4414f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb440ce890 0x7fbb441f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441f09f0 con 0x7fbb4415a420 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441f03e0 con 0x7fbb440ce890 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb44151dc0 con 0x7fbb440b4ff0 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58052/0 (socket says 172.21.15.104:58052) 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 -- 172.21.15.104:0/22447833 learned_addr learned my addr 172.21.15.104:0/22447833 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb427fc700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb440ce890 0x7fbb441f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb42ffd700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb440b4ff0 0x7fbb4414f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 -- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb440b4ff0 msgr2=0x7fbb4414f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb440b4ff0 0x7fbb4414f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 -- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb440ce890 msgr2=0x7fbb441f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb440ce890 0x7fbb441f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 -- 172.21.15.104:0/22447833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38009550 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb437fe700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00cc50 tx=0x7fbb3c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c003c00 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c003d60 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb441f0680 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00aab0 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.604+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb441f13c0 con 0x7fbb4415a420 2023-06-10T18:47:43.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.605+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb440b64e0 con 0x7fbb4415a420 2023-06-10T18:47:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.606+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbb3c00ac10 con 0x7fbb4415a420 2023-06-10T18:47:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.606+0000 7fbb2b7fe700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 0x7fbb3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.606+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fbb3c089620 con 0x7fbb4415a420 2023-06-10T18:47:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.606+0000 7fbb42ffd700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 0x7fbb3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.607+0000 7fbb42ffd700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 0x7fbb3006d190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbb38014040 tx=0x7fbb38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.610+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbb440b64e0 con 0x7fbb4415a420 2023-06-10T18:47:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.799+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb440b64e0 con 0x7fbb4415a420 2023-06-10T18:47:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.800+0000 7fbb2b7fe700 1 -- 172.21.15.104:0/22447833 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7fbb440b64e0 con 0x7fbb4415a420 2023-06-10T18:47:43.806 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:43.807 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:43.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 msgr2=0x7fbb3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 0x7fbb3006d190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbb38014040 tx=0x7fbb38008030 comp rx=0 tx=0).stop 2023-06-10T18:47:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 msgr2=0x7fbb4406f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00cc50 tx=0x7fbb3c00cf60 comp rx=0 tx=0).stop 2023-06-10T18:47:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 shutdown_connections 2023-06-10T18:47:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4415a420 0x7fbb4406f570 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb440b4ff0 0x7fbb4414f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb3006ace0 0x7fbb3006d190 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 --2- 172.21.15.104:0/22447833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb440ce890 0x7fbb441f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 >> 172.21.15.104:0/22447833 conn(0x7fbb44089fd0 msgr2=0x7fbb441503c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:43.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 shutdown_connections 2023-06-10T18:47:43.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.802+0000 7fbb494b4700 1 -- 172.21.15.104:0/22447833 wait complete. 2023-06-10T18:47:43.820 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:45:39.741633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:45:40.245539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:47:43.821 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:47:43.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.939+0000 7f0e58514700 1 Processor -- start 2023-06-10T18:47:43.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.939+0000 7f0e58514700 1 -- start start 2023-06-10T18:47:43.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e58514700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e500b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e58514700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 0x7f0e500b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e58514700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5006e250 con 0x7f0e5006cde0 2023-06-10T18:47:43.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e58514700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5015c950 con 0x7f0e5006e6e0 2023-06-10T18:47:43.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e58514700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e500cf0e0 con 0x7f0e5015c580 2023-06-10T18:47:43.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e55aaf700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 0x7f0e500b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e562b0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e500b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e56ab1700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0e5006cde0 0x7f0e50069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:33500/0 (socket says 172.21.15.104:33500) 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e56ab1700 1 -- 172.21.15.104:0/4046546399 learned_addr learned my addr 172.21.15.104:0/4046546399 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.940+0000 7f0e552ae700 1 -- 172.21.15.104:0/4046546399 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786020604 0 0) 0x7f0e5006e250 con 0x7f0e5006cde0 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e552ae700 1 -- 172.21.15.104:0/4046546399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e3c0035b0 con 0x7f0e5006cde0 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e562b0700 1 -- 172.21.15.104:0/4046546399 >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 msgr2=0x7f0e500b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:43.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e562b0700 1 --2- 172.21.15.104:0/4046546399 >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 0x7f0e500b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e562b0700 1 -- 172.21.15.104:0/4046546399 >> v1:172.21.15.114:6789/0 conn(0x7f0e5006cde0 legacy=0x7f0e50069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e562b0700 1 -- 172.21.15.104:0/4046546399 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e500b1e80 con 0x7f0e5015c580 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e55aaf700 1 --2- 172.21.15.104:0/4046546399 >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 0x7f0e500b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e562b0700 1 --2- 172.21.15.104:0/4046546399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e500b1070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009820 tx=0x7f0e4c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366ce75275dad079 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e552ae700 1 -- 172.21.15.104:0/4046546399 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c005e30 con 0x7f0e5015c580 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e552ae700 1 -- 172.21.15.104:0/4046546399 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c006030 con 0x7f0e5015c580 2023-06-10T18:47:43.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e552ae700 1 -- 172.21.15.104:0/4046546399 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0063c0 con 0x7f0e5015c580 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e58514700 1 -- 172.21.15.104:0/4046546399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 msgr2=0x7f0e500b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.941+0000 7f0e58514700 1 --2- 172.21.15.104:0/4046546399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e500b1070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009820 tx=0x7f0e4c009b30 comp rx=0 tx=0).stop 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 -- 172.21.15.104:0/4046546399 shutdown_connections 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 --2- 172.21.15.104:0/4046546399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e500b1070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 --2- 172.21.15.104:0/4046546399 >> v2:172.21.15.114:3300/0 conn(0x7f0e5006e6e0 0x7f0e500b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 -- 172.21.15.104:0/4046546399 >> 172.21.15.104:0/4046546399 conn(0x7f0e5006b8a0 msgr2=0x7f0e500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 -- 172.21.15.104:0/4046546399 shutdown_connections 2023-06-10T18:47:43.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.942+0000 7f0e58514700 1 -- 172.21.15.104:0/4046546399 wait complete. 2023-06-10T18:47:43.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.943+0000 7f0e58514700 1 Processor -- start 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.943+0000 7f0e58514700 1 -- start start 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 0x7f0e500c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e5015f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e500af0c0 con 0x7f0e5006e6e0 2023-06-10T18:47:43.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015fb90 con 0x7f0e5006cde0 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e500c55a0 con 0x7f0e5015c580 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e562b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 0x7f0e500c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58062/0 (socket says 172.21.15.104:58062) 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 -- 172.21.15.104:0/2752833388 learned_addr learned my addr 172.21.15.104:0/2752833388 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e562b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 0x7f0e500c4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:45562/0 (socket says 172.21.15.104:45562) 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e56ab1700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e5015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 -- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 msgr2=0x7f0e5015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e5015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 -- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 msgr2=0x7f0e500c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 0x7f0e500c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 -- 172.21.15.104:0/2752833388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e400094a0 con 0x7f0e5006e6e0 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e55aaf700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000ec10 tx=0x7f0e4000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40009760 con 0x7f0e5006e6e0 2023-06-10T18:47:43.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000c850 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c009430 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4000cbe0 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.944+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e500c5b60 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.945+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e500af200 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.946+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0e4000cd40 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.946+0000 7f0e46ffd700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 0x7f0e3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.947+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f0e40089a80 con 0x7f0e5006e6e0 2023-06-10T18:47:43.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.947+0000 7f0e562b0700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 0x7f0e3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:43.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.947+0000 7f0e562b0700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 0x7f0e3406d190 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c000c00 tx=0x7f0e4c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:43.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:43.951+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0e500af200 con 0x7f0e5006e6e0 2023-06-10T18:47:44.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.135+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e500af200 con 0x7f0e5006e6e0 2023-06-10T18:47:44.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.136+0000 7f0e46ffd700 1 -- 172.21.15.104:0/2752833388 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e500af200 con 0x7f0e5006e6e0 2023-06-10T18:47:44.142 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:47:44.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 msgr2=0x7f0e3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 0x7f0e3406d190 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c000c00 tx=0x7f0e4c008070 comp rx=0 tx=0).stop 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 msgr2=0x7f0e50158d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000ec10 tx=0x7f0e4000ef20 comp rx=0 tx=0).stop 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 shutdown_connections 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e3406ace0 0x7f0e3406d190 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e5006cde0 0x7f0e500c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e5015c580 0x7f0e5015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 --2- 172.21.15.104:0/2752833388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e5006e6e0 0x7f0e50158d40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 >> 172.21.15.104:0/2752833388 conn(0x7f0e5006b8a0 msgr2=0x7f0e50159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 shutdown_connections 2023-06-10T18:47:44.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.138+0000 7f0e58514700 1 -- 172.21.15.104:0/2752833388 wait complete. 2023-06-10T18:47:44.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:47:44.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.271+0000 7f3b14ba7700 1 Processor -- start 2023-06-10T18:47:44.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.271+0000 7f3b14ba7700 1 -- start start 2023-06-10T18:47:44.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b14ba7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b1006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b14ba7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100ae7e0 con 0x7f3b1006ea70 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b14ba7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1006df90 con 0x7f3b1015c6f0 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b14ba7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1006e270 con 0x7f3b100b1780 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b0ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3b1006ea70 0x7f3b1006d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55900/0 (socket says 172.21.15.104:55900) 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b0ed9d700 1 -- 172.21.15.104:0/589694783 learned_addr learned my addr 172.21.15.104:0/589694783 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.272+0000 7f3b0dd9b700 1 --2- 172.21.15.104:0/589694783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b1006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091583953 0 0) 0x7f3b100ae7e0 con 0x7f3b1006ea70 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af80035b0 con 0x7f3b1006ea70 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515310890 0 0) 0x7f3b1006df90 con 0x7f3b1015c6f0 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b100ae7e0 con 0x7f3b1015c6f0 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442283503 0 0) 0x7f3af80035b0 con 0x7f3b1006ea70 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0dd9b700 1 -- 172.21.15.104:0/589694783 >> v1:172.21.15.114:6789/0 conn(0x7f3b1015c6f0 legacy=0x7f3b100b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0dd9b700 1 -- 172.21.15.104:0/589694783 >> v1:172.21.15.104:6789/0 conn(0x7f3b1006ea70 legacy=0x7f3b1006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0dd9b700 1 -- 172.21.15.104:0/589694783 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1006e550 con 0x7f3b100b1780 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0dd9b700 1 --2- 172.21.15.104:0/589694783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b1006cde0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009870 tx=0x7f3b00009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782b9a910cb4a46c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b080019f0 con 0x7f3b100b1780 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00004030 con 0x7f3b100b1780 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.273+0000 7f3b0d59a700 1 -- 172.21.15.104:0/589694783 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000043c0 con 0x7f3b100b1780 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 -- 172.21.15.104:0/589694783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 msgr2=0x7f3b1006cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/589694783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b1006cde0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009870 tx=0x7f3b00009b80 comp rx=0 tx=0).stop 2023-06-10T18:47:44.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 -- 172.21.15.104:0/589694783 shutdown_connections 2023-06-10T18:47:44.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/589694783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b1006cde0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 -- 172.21.15.104:0/589694783 >> 172.21.15.104:0/589694783 conn(0x7f3b1006b8a0 msgr2=0x7f3b100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 -- 172.21.15.104:0/589694783 shutdown_connections 2023-06-10T18:47:44.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.274+0000 7f3b14ba7700 1 -- 172.21.15.104:0/589694783 wait complete. 2023-06-10T18:47:44.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.275+0000 7f3b14ba7700 1 Processor -- start 2023-06-10T18:47:44.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.275+0000 7f3b14ba7700 1 -- start start 2023-06-10T18:47:44.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b10069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b1015c6f0 0x7f3b1006a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100b0550 con 0x7f3b1006ea70 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101540d0 con 0x7f3b1015c6f0 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b14ba7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100b3a10 con 0x7f3b100b1780 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58074/0 (socket says 172.21.15.104:58074) 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 -- 172.21.15.104:0/2659936032 learned_addr learned my addr 172.21.15.104:0/2659936032 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0ed9d700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b1015c6f0 0x7f3b1006a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0dd9b700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b10069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 -- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 msgr2=0x7f3b10069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b10069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 -- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b1015c6f0 msgr2=0x7f3b1006a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b1015c6f0 0x7f3b1006a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 -- 172.21.15.104:0/2659936032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000094a0 con 0x7f3b1006ea70 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.276+0000 7f3b0e59c700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3b04001f00 tx=0x7f3b0400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040052d0 con 0x7f3b1006ea70 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b04005430 con 0x7f3b1006ea70 2023-06-10T18:47:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b100b3cf0 con 0x7f3b1006ea70 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040038f0 con 0x7f3b1006ea70 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b100b42b0 con 0x7f3b1006ea70 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.277+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b10074050 con 0x7f3b1006ea70 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.278+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3b04003a50 con 0x7f3b1006ea70 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.278+0000 7f3afeffd700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 0x7f3af006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.278+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3b04088be0 con 0x7f3b1006ea70 2023-06-10T18:47:44.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.278+0000 7f3b0dd9b700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 0x7f3af006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.279+0000 7f3b0dd9b700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 0x7f3af006d190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b100b1b70 tx=0x7f3b00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.283+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3b04053a90 con 0x7f3b1006ea70 2023-06-10T18:47:44.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.469+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3b101595b0 con 0x7f3b1006ea70 2023-06-10T18:47:44.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.469+0000 7f3afeffd700 1 -- 172.21.15.104:0/2659936032 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v44) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3b040570b0 con 0x7f3b1006ea70 2023-06-10T18:47:44.475 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:47:44.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 msgr2=0x7f3af006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 0x7f3af006d190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b100b1b70 tx=0x7f3b00008030 comp rx=0 tx=0).stop 2023-06-10T18:47:44.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 msgr2=0x7f3b10153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3b04001f00 tx=0x7f3b0400c860 comp rx=0 tx=0).stop 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 shutdown_connections 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3af006ace0 0x7f3af006d190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b1015c6f0 0x7f3b1006a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b1006ea70 0x7f3b10153b90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 --2- 172.21.15.104:0/2659936032 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b100b1780 0x7f3b10069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 >> 172.21.15.104:0/2659936032 conn(0x7f3b1006b8a0 msgr2=0x7f3b101582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 shutdown_connections 2023-06-10T18:47:44.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.471+0000 7f3b14ba7700 1 -- 172.21.15.104:0/2659936032 wait complete. 2023-06-10T18:47:44.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:47:44.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.605+0000 7fe9a09c1700 1 Processor -- start 2023-06-10T18:47:44.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.605+0000 7fe9a09c1700 1 -- start start 2023-06-10T18:47:44.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe9a09c1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe9a09c1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe99c06cde0 0x7fe99c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe9a09c1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c06e250 con 0x7fe99c15c580 2023-06-10T18:47:44.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe9a09c1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99c15c950 con 0x7fe99c06cde0 2023-06-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe9a09c1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0b1e80 con 0x7fe99c06e6e0 2023-06-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.606+0000 7fe99ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe99c06cde0 0x7fe99c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe99c06cde0 0x7fe99c0ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58086/0 (socket says 172.21.15.104:58086) 2023-06-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 -- 172.21.15.104:0/488969333 learned_addr learned my addr 172.21.15.104:0/488969333 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:44.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe999d9b700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 -- 172.21.15.104:0/488969333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 msgr2=0x7fe99c0b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:44.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 -- 172.21.15.104:0/488969333 >> v1:172.21.15.114:6789/0 conn(0x7fe99c15c580 legacy=0x7fe99c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 -- 172.21.15.104:0/488969333 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c06d920 con 0x7fe99c06cde0 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99ad9d700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c06cde0 0x7fe99c0ce0d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe994009e50 tx=0x7fe99400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6885958a922656f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99959a700 1 -- 172.21.15.104:0/488969333 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99400e070 con 0x7fe99c06cde0 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99959a700 1 -- 172.21.15.104:0/488969333 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99400b9c0 con 0x7fe99c06cde0 2023-06-10T18:47:44.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.607+0000 7fe99959a700 1 -- 172.21.15.104:0/488969333 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99400bd70 con 0x7fe99c06cde0 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 -- 172.21.15.104:0/488969333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c06cde0 msgr2=0x7fe99c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c06cde0 0x7fe99c0ce0d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe994009e50 tx=0x7fe99400b330 comp rx=0 tx=0).stop 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 -- 172.21.15.104:0/488969333 shutdown_connections 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c06cde0 0x7fe99c0ce0d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/488969333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 -- 172.21.15.104:0/488969333 >> 172.21.15.104:0/488969333 conn(0x7fe99c06b8a0 msgr2=0x7fe99c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 -- 172.21.15.104:0/488969333 shutdown_connections 2023-06-10T18:47:44.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.608+0000 7fe9a09c1700 1 -- 172.21.15.104:0/488969333 wait complete. 2023-06-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.609+0000 7fe9a09c1700 1 Processor -- start 2023-06-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 -- start start 2023-06-10T18:47:44.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe99c06cde0 0x7fe99c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0b3910 con 0x7fe99c15c580 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0b4e40 con 0x7fe99c06cde0 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.610+0000 7fe9a09c1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0c6140 con 0x7fe99c06e6e0 2023-06-10T18:47:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58092/0 (socket says 172.21.15.104:58092) 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 -- 172.21.15.104:0/3970487400 learned_addr learned my addr 172.21.15.104:0/3970487400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99a59c700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe99c06cde0 0x7fe99c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe999d9b700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 -- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 msgr2=0x7fe99c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 -- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe99c06cde0 msgr2=0x7fe99c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe99c06cde0 0x7fe99c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 -- 172.21.15.104:0/3970487400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994009b30 con 0x7fe99c15c580 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe99ad9d700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe994004270 tx=0x7fe994004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940099b0 con 0x7fe99c15c580 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9940092e0 con 0x7fe99c15c580 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99401c640 con 0x7fe99c15c580 2023-06-10T18:47:44.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c0b4230 con 0x7fe99c15c580 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.611+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c0b47f0 con 0x7fe99c15c580 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.612+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99c074050 con 0x7fe99c15c580 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.613+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe99401c7a0 con 0x7fe99c15c580 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.613+0000 7fe98affd700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 0x7fe97c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.613+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fe99408b1d0 con 0x7fe99c15c580 2023-06-10T18:47:44.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.613+0000 7fe99a59c700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 0x7fe97c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.614+0000 7fe99a59c700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 0x7fe97c06d1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe990000fb0 tx=0x7fe990009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.617+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe994056270 con 0x7fe99c15c580 2023-06-10T18:47:44.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.804+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe99c0b5450 con 0x7fe99c15c580 2023-06-10T18:47:44.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.804+0000 7fe98affd700 1 -- 172.21.15.104:0/3970487400 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe994059890 con 0x7fe99c15c580 2023-06-10T18:47:44.810 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:47:44.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 msgr2=0x7fe97c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 0x7fe97c06d1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe990000fb0 tx=0x7fe990009040 comp rx=0 tx=0).stop 2023-06-10T18:47:44.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 msgr2=0x7fe99c0cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe994004270 tx=0x7fe994004ad0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 shutdown_connections 2023-06-10T18:47:44.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe97c06ad30 0x7fe97c06d1e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe99c06cde0 0x7fe99c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe99c15c580 0x7fe99c0cf280 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 --2- 172.21.15.104:0/3970487400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe99c06e6e0 0x7fe99c0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 >> 172.21.15.104:0/3970487400 conn(0x7fe99c06b8a0 msgr2=0x7fe99c0af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 shutdown_connections 2023-06-10T18:47:44.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.806+0000 7fe9a09c1700 1 -- 172.21.15.104:0/3970487400 wait complete. 2023-06-10T18:47:44.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:47:44.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 Processor -- start 2023-06-10T18:47:44.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 -- start start 2023-06-10T18:47:44.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f62480d0a00 0x7f6248164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62480ad3c0 con 0x7f6248157b30 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6248157f00 con 0x7f62480cea50 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.940+0000 7f624dda7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62480d0e60 con 0x7f62480d0a00 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f6247fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f62480d0a00 0x7f6248164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62477fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6248157b30 0x7f62480ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55924/0 (socket says 172.21.15.104:55924) 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62477fe700 1 -- 172.21.15.104:0/342718807 learned_addr learned my addr 172.21.15.104:0/342718807 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:44.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f6247fff700 1 -- 172.21.15.104:0/342718807 >> v1:172.21.15.114:6789/0 conn(0x7f62480cea50 legacy=0x7f62480d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f6247fff700 1 -- 172.21.15.104:0/342718807 >> v1:172.21.15.104:6789/0 conn(0x7f6248157b30 legacy=0x7f62480ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f6247fff700 1 -- 172.21.15.104:0/342718807 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62480b0170 con 0x7f62480d0a00 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62467fc700 1 -- 172.21.15.104:0/342718807 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506774693 0 0) 0x7f62480ad3c0 con 0x7f6248157b30 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f6247fff700 1 --2- 172.21.15.104:0/342718807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480d0a00 0x7f6248164e60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6240009d90 tx=0x7f624000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51c86ac732bfa54 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62467fc700 1 -- 172.21.15.104:0/342718807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624000e070 con 0x7f62480d0a00 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62467fc700 1 -- 172.21.15.104:0/342718807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624000bb90 con 0x7f62480d0a00 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.941+0000 7f62467fc700 1 -- 172.21.15.104:0/342718807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6240003a20 con 0x7f62480d0a00 2023-06-10T18:47:44.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 -- 172.21.15.104:0/342718807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480d0a00 msgr2=0x7f6248164e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 --2- 172.21.15.104:0/342718807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480d0a00 0x7f6248164e60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6240009d90 tx=0x7f624000b330 comp rx=0 tx=0).stop 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 -- 172.21.15.104:0/342718807 shutdown_connections 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 --2- 172.21.15.104:0/342718807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480d0a00 0x7f6248164e60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 -- 172.21.15.104:0/342718807 >> 172.21.15.104:0/342718807 conn(0x7f6248089fd0 msgr2=0x7f624808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 -- 172.21.15.104:0/342718807 shutdown_connections 2023-06-10T18:47:44.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.942+0000 7f624dda7700 1 -- 172.21.15.104:0/342718807 wait complete. 2023-06-10T18:47:44.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.943+0000 7f624dda7700 1 Processor -- start 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.943+0000 7f624dda7700 1 -- start start 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62480d0a00 0x7f6248142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6248157b30 0x7f62481428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6248141f60 con 0x7f62480cea50 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62481430e0 con 0x7f6248157b30 2023-06-10T18:47:44.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f624dda7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62481433c0 con 0x7f62480d0a00 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58102/0 (socket says 172.21.15.104:58102) 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 -- 172.21.15.104:0/3833439796 learned_addr learned my addr 172.21.15.104:0/3833439796 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 -- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62480d0a00 msgr2=0x7f6248142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f6247fff700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6248157b30 0x7f62481428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f6246ffd700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62480d0a00 0x7f6248142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62480d0a00 0x7f6248142370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 -- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6248157b30 msgr2=0x7f62481428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6248157b30 0x7f62481428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 -- 172.21.15.104:0/3833439796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240009a70 con 0x7f62480cea50 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f62477fe700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f623800c550 tx=0x7f623800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.944+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238004c80 con 0x7f62480cea50 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.945+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6238004e00 con 0x7f62480cea50 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.945+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62481436a0 con 0x7f62480cea50 2023-06-10T18:47:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.945+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238007310 con 0x7f62480cea50 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.945+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6248143c60 con 0x7f62480cea50 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.946+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6248143e80 con 0x7f62480cea50 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.946+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6238007470 con 0x7f62480cea50 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.946+0000 7f622ffff700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 0x7f623406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.946+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f6238089360 con 0x7f62480cea50 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.947+0000 7f6246ffd700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 0x7f623406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:44.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.947+0000 7f6246ffd700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 0x7f623406d190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f624806f4f0 tx=0x7f623c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:44.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:44.951+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6238054290 con 0x7f62480cea50 2023-06-10T18:47:45.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.137+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6248074050 con 0x7f62480cea50 2023-06-10T18:47:45.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.137+0000 7f622ffff700 1 -- 172.21.15.104:0/3833439796 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62380578b0 con 0x7f62480cea50 2023-06-10T18:47:45.143 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:47:45.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 msgr2=0x7f623406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 0x7f623406d190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f624806f4f0 tx=0x7f623c009040 comp rx=0 tx=0).stop 2023-06-10T18:47:45.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 msgr2=0x7f6248141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f623800c550 tx=0x7f623800c910 comp rx=0 tx=0).stop 2023-06-10T18:47:45.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 shutdown_connections 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f623406ace0 0x7f623406d190 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6248157b30 0x7f62481428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62480cea50 0x7f6248141a20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 --2- 172.21.15.104:0/3833439796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62480d0a00 0x7f6248142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 >> 172.21.15.104:0/3833439796 conn(0x7f6248089fd0 msgr2=0x7f6248152cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 shutdown_connections 2023-06-10T18:47:45.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.139+0000 7f624dda7700 1 -- 172.21.15.104:0/3833439796 wait complete. 2023-06-10T18:47:45.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:45.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.271+0000 7f030af8e700 1 Processor -- start 2023-06-10T18:47:45.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.271+0000 7f030af8e700 1 -- start start 2023-06-10T18:47:45.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030af8e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03041567f0 0x7f03040cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030af8e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03040701a0 0x7f03040705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030af8e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304070b20 con 0x7f030415c7a0 2023-06-10T18:47:45.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030af8e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03040cb2d0 con 0x7f03040701a0 2023-06-10T18:47:45.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030af8e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030415cca0 con 0x7f03041567f0 2023-06-10T18:47:45.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03040701a0 0x7f03040705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f0303fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f030415c7a0 0x7f03040cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55940/0 (socket says 172.21.15.104:55940) 2023-06-10T18:47:45.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03040701a0 0x7f03040705e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58114/0 (socket says 172.21.15.104:58114) 2023-06-10T18:47:45.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f0303fff700 1 -- 172.21.15.104:0/1304408709 learned_addr learned my addr 172.21.15.104:0/1304408709 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:45.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f0308d2a700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03041567f0 0x7f03040cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 -- 172.21.15.104:0/1304408709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03041567f0 msgr2=0x7f03040cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03041567f0 0x7f03040cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 -- 172.21.15.104:0/1304408709 >> v1:172.21.15.104:6789/0 conn(0x7f030415c7a0 legacy=0x7f03040cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f03037fe700 1 -- 172.21.15.104:0/1304408709 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362985799 0 0) 0x7f0304070b20 con 0x7f030415c7a0 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 -- 172.21.15.104:0/1304408709 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030415cf80 con 0x7f03040701a0 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f030952b700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f03040705e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009d90 tx=0x7f02fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc43f21670ac125 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.272+0000 7f03037fe700 1 -- 172.21.15.104:0/1304408709 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00e070 con 0x7f03040701a0 2023-06-10T18:47:45.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f03037fe700 1 -- 172.21.15.104:0/1304408709 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00bbd0 con 0x7f03040701a0 2023-06-10T18:47:45.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f03037fe700 1 -- 172.21.15.104:0/1304408709 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc012430 con 0x7f03040701a0 2023-06-10T18:47:45.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 -- 172.21.15.104:0/1304408709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 msgr2=0x7f03040705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f03040705e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009d90 tx=0x7f02fc00b330 comp rx=0 tx=0).stop 2023-06-10T18:47:45.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 -- 172.21.15.104:0/1304408709 shutdown_connections 2023-06-10T18:47:45.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f03040705e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 --2- 172.21.15.104:0/1304408709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03041567f0 0x7f03040cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 -- 172.21.15.104:0/1304408709 >> 172.21.15.104:0/1304408709 conn(0x7f030406e370 msgr2=0x7f030406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 -- 172.21.15.104:0/1304408709 shutdown_connections 2023-06-10T18:47:45.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.273+0000 7f030af8e700 1 -- 172.21.15.104:0/1304408709 wait complete. 2023-06-10T18:47:45.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.275+0000 7f030af8e700 1 Processor -- start 2023-06-10T18:47:45.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.275+0000 7f030af8e700 1 -- start start 2023-06-10T18:47:45.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03041567f0 0x7f03040b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f030415c7a0 0x7f03040c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040c4bd0 con 0x7f03040701a0 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040c7140 con 0x7f03041567f0 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030af8e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b26c0 con 0x7f030415c7a0 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58116/0 (socket says 172.21.15.104:58116) 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 -- 172.21.15.104:0/2849156101 learned_addr learned my addr 172.21.15.104:0/2849156101 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0303fff700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03041567f0 0x7f03040b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f030952b700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f030415c7a0 0x7f03040c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 -- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f030415c7a0 msgr2=0x7f03040c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f030415c7a0 0x7f03040c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 -- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03041567f0 msgr2=0x7f03040b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03041567f0 0x7f03040b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 -- 172.21.15.104:0/2849156101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc009a70 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f0308d2a700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e810 tx=0x7f02f800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.276+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8013610 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.277+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f8013770 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.277+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03040b29a0 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.277+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8013b20 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.277+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03040c50c0 con 0x7f03040701a0 2023-06-10T18:47:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.277+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03040c5430 con 0x7f03040701a0 2023-06-10T18:47:45.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.278+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f02f8013c80 con 0x7f03040701a0 2023-06-10T18:47:45.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.278+0000 7f03017fa700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.278+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f02f8089330 con 0x7f03040701a0 2023-06-10T18:47:45.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.278+0000 7f0303fff700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.279+0000 7f0303fff700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f03040c44b0 tx=0x7f02f0007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.283+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f02f8054270 con 0x7f03040701a0 2023-06-10T18:47:45.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.508+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03040b2b30 con 0x7f03040701a0 2023-06-10T18:47:45.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.509+0000 7f03017fa700 1 -- 172.21.15.104:0/2849156101 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f02f8057890 con 0x7f03040701a0 2023-06-10T18:47:45.514 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:45.515 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:47:45.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 msgr2=0x7f02e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f03040c44b0 tx=0x7f02f0007be0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 msgr2=0x7f0304151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e810 tx=0x7f02f800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 shutdown_connections 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f030415c7a0 0x7f03040c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03040701a0 0x7f0304151bf0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02e806ace0 0x7f02e806d190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 --2- 172.21.15.104:0/2849156101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03041567f0 0x7f03040b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 >> 172.21.15.104:0/2849156101 conn(0x7f030406e370 msgr2=0x7f030415f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 shutdown_connections 2023-06-10T18:47:45.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.511+0000 7f030af8e700 1 -- 172.21.15.104:0/2849156101 wait complete. 2023-06-10T18:47:45.518 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:47:45.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.641+0000 7f9a56448700 1 Processor -- start 2023-06-10T18:47:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.641+0000 7f9a56448700 1 -- start start 2023-06-10T18:47:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a5015a250 0x7f9a500cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 0x7f9a500b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a500ccca0 con 0x7f9a5015a250 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a500d0310 con 0x7f9a500cad90 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a56448700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a500cee50 con 0x7f9a500cfa00 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a4f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a5015a250 0x7f9a500cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a4f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a5015a250 0x7f9a500cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58126/0 (socket says 172.21.15.104:58126) 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a4f7fe700 1 -- 172.21.15.104:0/2483330326 learned_addr learned my addr 172.21.15.104:0/2483330326 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.642+0000 7f9a549e5700 1 --2- 172.21.15.104:0/2483330326 >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 0x7f9a500b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 -- 172.21.15.104:0/2483330326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 msgr2=0x7f9a500701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 -- 172.21.15.104:0/2483330326 >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 msgr2=0x7f9a500b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 --2- 172.21.15.104:0/2483330326 >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 0x7f9a500b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 -- 172.21.15.104:0/2483330326 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500cf130 con 0x7f9a5015a250 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a549e5700 1 --2- 172.21.15.104:0/2483330326 >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 0x7f9a500b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4f7fe700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a500cfdd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9a5015dab0 tx=0x7f9a4000f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f261a45e0a218a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4effd700 1 -- 172.21.15.104:0/2483330326 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40012070 con 0x7f9a5015a250 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4effd700 1 -- 172.21.15.104:0/2483330326 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4000d2e0 con 0x7f9a5015a250 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a4effd700 1 -- 172.21.15.104:0/2483330326 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4000fa00 con 0x7f9a5015a250 2023-06-10T18:47:45.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a56448700 1 -- 172.21.15.104:0/2483330326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 msgr2=0x7f9a500cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.643+0000 7f9a56448700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a500cfdd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9a5015dab0 tx=0x7f9a4000f050 comp rx=0 tx=0).stop 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 -- 172.21.15.104:0/2483330326 shutdown_connections 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 --2- 172.21.15.104:0/2483330326 >> v2:172.21.15.114:3300/0 conn(0x7f9a500cad90 0x7f9a500b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 --2- 172.21.15.104:0/2483330326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a500cfdd0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 -- 172.21.15.104:0/2483330326 >> 172.21.15.104:0/2483330326 conn(0x7f9a5006e370 msgr2=0x7f9a5006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 -- 172.21.15.104:0/2483330326 shutdown_connections 2023-06-10T18:47:45.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.644+0000 7f9a56448700 1 -- 172.21.15.104:0/2483330326 wait complete. 2023-06-10T18:47:45.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.645+0000 7f9a56448700 1 Processor -- start 2023-06-10T18:47:45.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.645+0000 7f9a56448700 1 -- start start 2023-06-10T18:47:45.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 0x7f9a500c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a500c71f0 con 0x7f9a5015a250 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a501ed6e0 con 0x7f9a500cad90 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a56448700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a501f1060 con 0x7f9a500cfa00 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a549e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a549e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58134/0 (socket says 172.21.15.104:58134) 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.646+0000 7f9a549e5700 1 -- 172.21.15.104:0/1679908507 learned_addr learned my addr 172.21.15.104:0/1679908507 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:45.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4f7fe700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 0x7f9a500c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 -- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 msgr2=0x7f9a500b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 -- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 msgr2=0x7f9a500c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 0x7f9a500c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 -- 172.21.15.104:0/1679908507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4000d9f0 con 0x7f9a5015a250 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4f7fe700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:45.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 0x7f9a500c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a549e5700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9a48007db0 tx=0x7f9a4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800ddd0 con 0x7f9a5015a250 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4800f040 con 0x7f9a5015a250 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a501f1340 con 0x7f9a5015a250 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48003c60 con 0x7f9a5015a250 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.647+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a501f1900 con 0x7f9a5015a250 2023-06-10T18:47:45.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.648+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a501ed850 con 0x7f9a5015a250 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.649+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a48004030 con 0x7f9a5015a250 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.649+0000 7f9a4cff9700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 0x7f9a3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.649+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f9a48089640 con 0x7f9a5015a250 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.649+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 0x7f9a3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.649+0000 7f9a4ffff700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 0x7f9a3c06d190 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009960 tx=0x7f9a44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.653+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a48054600 con 0x7f9a5015a250 2023-06-10T18:47:45.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.843+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a50074050 con 0x7f9a5015a250 2023-06-10T18:47:45.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.844+0000 7f9a4cff9700 1 -- 172.21.15.104:0/1679908507 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7f9a48057c20 con 0x7f9a5015a250 2023-06-10T18:47:45.851 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:45.851 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:45.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 msgr2=0x7f9a3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 0x7f9a3c06d190 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009960 tx=0x7f9a44008040 comp rx=0 tx=0).stop 2023-06-10T18:47:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 msgr2=0x7f9a5015a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9a48007db0 tx=0x7f9a4800d3b0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 shutdown_connections 2023-06-10T18:47:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a3c06ace0 0x7f9a3c06d190 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a500cad90 0x7f9a500c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a5015a250 0x7f9a5015a650 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 --2- 172.21.15.104:0/1679908507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a500cfa00 0x7f9a500b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 >> 172.21.15.104:0/1679908507 conn(0x7f9a5006e370 msgr2=0x7f9a500cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 shutdown_connections 2023-06-10T18:47:45.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.846+0000 7f9a56448700 1 -- 172.21.15.104:0/1679908507 wait complete. 2023-06-10T18:47:45.864 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:47:45.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.977+0000 7f1b67fd8700 1 Processor -- start 2023-06-10T18:47:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 -- start start 2023-06-10T18:47:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 0x7f1b600cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1b600ce040 0x7f1b6006e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b6006a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6006a830 con 0x7f1b600ce040 2023-06-10T18:47:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6006dcb0 con 0x7f1b6015c500 2023-06-10T18:47:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b67fd8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600ce470 con 0x7f1b60069e70 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.978+0000 7f1b65573700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1b600ce040 0x7f1b6006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1b600ce040 0x7f1b6006e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58140/0 (socket says 172.21.15.104:58140) 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 -- 172.21.15.104:0/3578408600 learned_addr learned my addr 172.21.15.104:0/3578408600 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b66575700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b6006a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65d74700 1 --2- 172.21.15.104:0/3578408600 >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 0x7f1b600cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 -- 172.21.15.104:0/3578408600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 msgr2=0x7f1b6006a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b6006a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 -- 172.21.15.104:0/3578408600 >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 msgr2=0x7f1b600cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 --2- 172.21.15.104:0/3578408600 >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 0x7f1b600cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 -- 172.21.15.104:0/3578408600 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6006d0a0 con 0x7f1b600ce040 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65d74700 1 --2- 172.21.15.104:0/3578408600 >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 0x7f1b600cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b65573700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b600ce040 0x7f1b6006e140 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1b60067130 tx=0x7f1b5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a834726b76cbf9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b64d72700 1 -- 172.21.15.104:0/3578408600 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c00b870 con 0x7f1b600ce040 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b64d72700 1 -- 172.21.15.104:0/3578408600 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c00b9d0 con 0x7f1b600ce040 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.979+0000 7f1b64d72700 1 -- 172.21.15.104:0/3578408600 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c00bd60 con 0x7f1b600ce040 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 -- 172.21.15.104:0/3578408600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b600ce040 msgr2=0x7f1b6006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b600ce040 0x7f1b6006e140 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1b60067130 tx=0x7f1b5c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 -- 172.21.15.104:0/3578408600 shutdown_connections 2023-06-10T18:47:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b6006a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/3578408600 >> v2:172.21.15.114:3300/0 conn(0x7f1b6015c500 0x7f1b600cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/3578408600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b600ce040 0x7f1b6006e140 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 -- 172.21.15.104:0/3578408600 >> 172.21.15.104:0/3578408600 conn(0x7f1b6006b8a0 msgr2=0x7f1b600cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 -- 172.21.15.104:0/3578408600 shutdown_connections 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.980+0000 7f1b67fd8700 1 -- 172.21.15.104:0/3578408600 wait complete. 2023-06-10T18:47:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.981+0000 7f1b67fd8700 1 Processor -- start 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.981+0000 7f1b67fd8700 1 -- start start 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b600c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b600ce040 0x7f1b6015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600cc540 con 0x7f1b6015c500 2023-06-10T18:47:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6015b5f0 con 0x7f1b600ce040 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b600c6de0 con 0x7f1b60069e70 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58148/0 (socket says 172.21.15.104:58148) 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 -- 172.21.15.104:0/1797055621 learned_addr learned my addr 172.21.15.104:0/1797055621 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b65d74700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b600c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b65573700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b600ce040 0x7f1b6015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 -- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 msgr2=0x7f1b600c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b600c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 -- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b600ce040 msgr2=0x7f1b6015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b600ce040 0x7f1b6015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 -- 172.21.15.104:0/1797055621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c009550 con 0x7f1b6015c500 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b66575700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1b58007e60 tx=0x7f1b5800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800ed30 con 0x7f1b6015c500 2023-06-10T18:47:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b58014420 con 0x7f1b6015c500 2023-06-10T18:47:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b600cbc10 con 0x7f1b6015c500 2023-06-10T18:47:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b580147d0 con 0x7f1b6015c500 2023-06-10T18:47:45.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.982+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b600cc1d0 con 0x7f1b6015c500 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.983+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b600cc6b0 con 0x7f1b6015c500 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.984+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1b58014930 con 0x7f1b6015c500 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.984+0000 7f1b567fc700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 0x7f1b4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.984+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1b5808aa80 con 0x7f1b6015c500 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.985+0000 7f1b65d74700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 0x7f1b4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.985+0000 7f1b65d74700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 0x7f1b4406d190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000ab60 tx=0x7f1b50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:45.988+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1b58055a90 con 0x7f1b6015c500 2023-06-10T18:47:46.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.213+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b60074050 con 0x7f1b6015c500 2023-06-10T18:47:46.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.214+0000 7f1b567fc700 1 -- 172.21.15.104:0/1797055621 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f1b580590b0 con 0x7f1b6015c500 2023-06-10T18:47:46.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:46.220 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:47:46.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 msgr2=0x7f1b4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 0x7f1b4406d190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000ab60 tx=0x7f1b50008040 comp rx=0 tx=0).stop 2023-06-10T18:47:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 msgr2=0x7f1b6014d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1b58007e60 tx=0x7f1b5800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 shutdown_connections 2023-06-10T18:47:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b6015c500 0x7f1b6014d960 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b60069e70 0x7f1b600c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b4406ace0 0x7f1b4406d190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 --2- 172.21.15.104:0/1797055621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b600ce040 0x7f1b6015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 >> 172.21.15.104:0/1797055621 conn(0x7f1b6006b8a0 msgr2=0x7f1b600b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 shutdown_connections 2023-06-10T18:47:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.216+0000 7f1b67fd8700 1 -- 172.21.15.104:0/1797055621 wait complete. 2023-06-10T18:47:46.225 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:47:46.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:46.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.347+0000 7f4a2c03b700 1 Processor -- start 2023-06-10T18:47:46.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.347+0000 7f4a2c03b700 1 -- start start 2023-06-10T18:47:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a2c03b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4a2415c040 0x7f4a240ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a2c03b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a240ae6f0 con 0x7f4a24071130 2023-06-10T18:47:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a2c03b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a240ce2c0 con 0x7f4a240ae280 2023-06-10T18:47:46.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a2c03b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a240cf750 con 0x7f4a2415c040 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4a2415c040 0x7f4a240ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4a2415c040 0x7f4a240ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58152/0 (socket says 172.21.15.104:58152) 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 -- 172.21.15.104:0/1649922644 learned_addr learned my addr 172.21.15.104:0/1649922644 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 -- 172.21.15.104:0/1649922644 >> v1:172.21.15.114:6789/0 conn(0x7f4a240ae280 legacy=0x7f4a24070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 -- 172.21.15.104:0/1649922644 >> v1:172.21.15.104:6789/0 conn(0x7f4a24071130 legacy=0x7f4a240ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a29dd7700 1 -- 172.21.15.104:0/1649922644 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240cfa30 con 0x7f4a2415c040 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.348+0000 7f4a28dd5700 1 -- 172.21.15.104:0/1649922644 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804049754 0 0) 0x7f4a240ae6f0 con 0x7f4a24071130 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a29dd7700 1 --2- 172.21.15.104:0/1649922644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240ce780 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098f0 tx=0x7f4a14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50ad0a5bde7b6d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a28dd5700 1 -- 172.21.15.104:0/1649922644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400b8a0 con 0x7f4a2415c040 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a28dd5700 1 -- 172.21.15.104:0/1649922644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1400ba00 con 0x7f4a2415c040 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a28dd5700 1 -- 172.21.15.104:0/1649922644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400bd90 con 0x7f4a2415c040 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 -- 172.21.15.104:0/1649922644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 msgr2=0x7f4a240ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/1649922644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240ce780 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098f0 tx=0x7f4a14009c00 comp rx=0 tx=0).stop 2023-06-10T18:47:46.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 -- 172.21.15.104:0/1649922644 shutdown_connections 2023-06-10T18:47:46.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/1649922644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240ce780 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 -- 172.21.15.104:0/1649922644 >> 172.21.15.104:0/1649922644 conn(0x7f4a240b1610 msgr2=0x7f4a240ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:46.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 -- 172.21.15.104:0/1649922644 shutdown_connections 2023-06-10T18:47:46.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.349+0000 7f4a2c03b700 1 -- 172.21.15.104:0/1649922644 wait complete. 2023-06-10T18:47:46.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.351+0000 7f4a2c03b700 1 Processor -- start 2023-06-10T18:47:46.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.351+0000 7f4a2c03b700 1 -- start start 2023-06-10T18:47:46.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a24071130 0x7f4a240b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a240ae280 0x7f4a240c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240cbc90 con 0x7f4a2415c040 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2415f650 con 0x7f4a24071130 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2c03b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2415f090 con 0x7f4a240ae280 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58162/0 (socket says 172.21.15.104:58162) 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 -- 172.21.15.104:0/2611454388 learned_addr learned my addr 172.21.15.104:0/2611454388 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 -- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a240ae280 msgr2=0x7f4a240c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a240ae280 0x7f4a240c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 -- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a24071130 msgr2=0x7f4a240b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a24071130 0x7f4a240b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 -- 172.21.15.104:0/2611454388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14009550 con 0x7f4a2415c040 2023-06-10T18:47:46.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.352+0000 7f4a2a5d8700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c850 tx=0x7f4a1c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.353+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c003e30 con 0x7f4a2415c040 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.353+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c00ce50 con 0x7f4a2415c040 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.353+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2415f370 con 0x7f4a2415c040 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.353+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c005330 con 0x7f4a2415c040 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.353+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a240b1c80 con 0x7f4a2415c040 2023-06-10T18:47:46.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.354+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a24074050 con 0x7f4a2415c040 2023-06-10T18:47:46.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.354+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4a1c005490 con 0x7f4a2415c040 2023-06-10T18:47:46.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.355+0000 7f4a1a7fc700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 0x7f4a0806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.355+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4a1c0891c0 con 0x7f4a2415c040 2023-06-10T18:47:46.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.355+0000 7f4a29dd7700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 0x7f4a0806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.356+0000 7f4a29dd7700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 0x7f4a0806d1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009130 tx=0x7f4a14010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.359+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4a1c0541d0 con 0x7f4a2415c040 2023-06-10T18:47:46.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.586+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a240cbdd0 con 0x7f4a2415c040 2023-06-10T18:47:46.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.586+0000 7f4a1a7fc700 1 -- 172.21.15.104:0/2611454388 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4677 (secure 0 0 0) 0x7f4a240cbdd0 con 0x7f4a2415c040 2023-06-10T18:47:46.593 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:46.593 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:45:41.438525+0000","modified":"2023-06-10T18:45:44.530451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5622":{"gid":5622,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/698196467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":698196467},{"type":"v1","addr":"172.21.15.114:6836","nonce":698196467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-06-10T18:47:46.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 msgr2=0x7f4a0806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 0x7f4a0806d1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009130 tx=0x7f4a14010000 comp rx=0 tx=0).stop 2023-06-10T18:47:46.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 msgr2=0x7f4a240c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c850 tx=0x7f4a1c00cc10 comp rx=0 tx=0).stop 2023-06-10T18:47:46.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 shutdown_connections 2023-06-10T18:47:46.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a0806ad10 0x7f4a0806d1c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a24071130 0x7f4a240b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a2415c040 0x7f4a240c7f70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 --2- 172.21.15.104:0/2611454388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a240ae280 0x7f4a240c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 >> 172.21.15.104:0/2611454388 conn(0x7f4a240b1610 msgr2=0x7f4a24069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 shutdown_connections 2023-06-10T18:47:46.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.588+0000 7f4a2c03b700 1 -- 172.21.15.104:0/2611454388 wait complete. 2023-06-10T18:47:46.597 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 17 2023-06-10T18:47:46.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:46.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.721+0000 7f06768f0700 1 Processor -- start 2023-06-10T18:47:46.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.721+0000 7f06768f0700 1 -- start start 2023-06-10T18:47:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f06768f0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f06768f0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 0x7f06700cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f06768f0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06700ccea0 con 0x7f0670070a20 2023-06-10T18:47:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f06768f0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06700ae6f0 con 0x7f06700ae280 2023-06-10T18:47:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f06768f0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06700ce2c0 con 0x7f067015c040 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f066f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 0x7f06700cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f066ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f0674e8d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0670070a20 0x7f06700ca3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:33610/0 (socket says 172.21.15.104:33610) 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f066ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:49318/0 (socket says 172.21.15.104:49318) 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.722+0000 7f0674e8d700 1 -- 172.21.15.104:0/4000833352 learned_addr learned my addr 172.21.15.104:0/4000833352 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066ffff700 1 -- 172.21.15.104:0/4000833352 >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 msgr2=0x7f06700cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066ffff700 1 --2- 172.21.15.104:0/4000833352 >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 0x7f06700cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066ffff700 1 -- 172.21.15.104:0/4000833352 >> v1:172.21.15.114:6789/0 conn(0x7f0670070a20 legacy=0x7f06700ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066ffff700 1 -- 172.21.15.104:0/4000833352 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06700cf750 con 0x7f067015c040 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066f7fe700 1 --2- 172.21.15.104:0/4000833352 >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 0x7f06700cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066ffff700 1 --2- 172.21.15.104:0/4000833352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0664009840 tx=0x7f0664009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a47d205593ce94b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066effd700 1 -- 172.21.15.104:0/4000833352 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664011830 con 0x7f067015c040 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066effd700 1 -- 172.21.15.104:0/4000833352 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0664011990 con 0x7f067015c040 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.723+0000 7f066effd700 1 -- 172.21.15.104:0/4000833352 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664011d20 con 0x7f067015c040 2023-06-10T18:47:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 -- 172.21.15.104:0/4000833352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 msgr2=0x7f06700ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 --2- 172.21.15.104:0/4000833352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0664009840 tx=0x7f0664009b50 comp rx=0 tx=0).stop 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 -- 172.21.15.104:0/4000833352 shutdown_connections 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 --2- 172.21.15.104:0/4000833352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f06700ce780 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 --2- 172.21.15.104:0/4000833352 >> v2:172.21.15.114:3300/0 conn(0x7f06700ae280 0x7f06700cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 -- 172.21.15.104:0/4000833352 >> 172.21.15.104:0/4000833352 conn(0x7f06700b1610 msgr2=0x7f06700ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 -- 172.21.15.104:0/4000833352 shutdown_connections 2023-06-10T18:47:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.724+0000 7f06768f0700 1 -- 172.21.15.104:0/4000833352 wait complete. 2023-06-10T18:47:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.725+0000 7f06768f0700 1 Processor -- start 2023-06-10T18:47:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.725+0000 7f06768f0700 1 -- start start 2023-06-10T18:47:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06700ae280 0x7f067014f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f0670069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06700cb5f0 con 0x7f0670070a20 2023-06-10T18:47:46.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067014fe90 con 0x7f06700ae280 2023-06-10T18:47:46.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f06768f0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06700c2ad0 con 0x7f067015c040 2023-06-10T18:47:46.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58172/0 (socket says 172.21.15.104:58172) 2023-06-10T18:47:46.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 -- 172.21.15.104:0/1974248127 learned_addr learned my addr 172.21.15.104:0/1974248127 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:46.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066f7fe700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06700ae280 0x7f067014f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f0674e8d700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f0670069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 -- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 msgr2=0x7f0670069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f0670069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 -- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06700ae280 msgr2=0x7f067014f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06700ae280 0x7f067014f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f066ffff700 1 -- 172.21.15.104:0/1974248127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600094a0 con 0x7f0670070a20 2023-06-10T18:47:46.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.726+0000 7f0674e8d700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f0670069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:46.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f066ffff700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fd0 tx=0x7f0664004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640038a0 con 0x7f0670070a20 2023-06-10T18:47:46.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066401f070 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06640094a0 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664003c20 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.727+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06700b3a10 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.728+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0670074050 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.728+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f066401a430 con 0x7f0670070a20 2023-06-10T18:47:46.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.728+0000 7f066cff9700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 0x7f065c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:46.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.729+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f0664021070 con 0x7f0670070a20 2023-06-10T18:47:46.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.729+0000 7f066f7fe700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 0x7f065c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:46.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.729+0000 7f066f7fe700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 0x7f065c06d190 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f06700c5830 tx=0x7f0660008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:46.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.733+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0664055ef0 con 0x7f0670070a20 2023-06-10T18:47:46.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.922+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06700b3cf0 con 0x7f0670070a20 2023-06-10T18:47:46.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.923+0000 7f066cff9700 1 -- 172.21.15.104:0/1974248127 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+18699 (secure 0 0 0) 0x7f0664059510 con 0x7f0670070a20 2023-06-10T18:47:46.929 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:46.929 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:45:51.563754+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:45:39.741633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-06-10T18:45:40.245539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:46.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 msgr2=0x7f065c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 0x7f065c06d190 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f06700c5830 tx=0x7f0660008040 comp rx=0 tx=0).stop 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 msgr2=0x7f06700cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fd0 tx=0x7f0664004750 comp rx=0 tx=0).stop 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 shutdown_connections 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f067015c040 0x7f0670069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0670070a20 0x7f06700cb0b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f065c06ace0 0x7f065c06d190 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 --2- 172.21.15.104:0/1974248127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06700ae280 0x7f067014f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:46.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 >> 172.21.15.104:0/1974248127 conn(0x7f06700b1610 msgr2=0x7f067015a190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:46.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 shutdown_connections 2023-06-10T18:47:46.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:46.925+0000 7f06768f0700 1 -- 172.21.15.104:0/1974248127 wait complete. 2023-06-10T18:47:46.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:47:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.058+0000 7f2d0af46700 1 Processor -- start 2023-06-10T18:47:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.058+0000 7f2d0af46700 1 -- start start 2023-06-10T18:47:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d0af46700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d0406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d0af46700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d04069e70 0x7f2d040ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d0af46700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0406d290 con 0x7f2d0406e4b0 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d0af46700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d04071640 con 0x7f2d04069e70 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d0af46700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040aef00 con 0x7f2d0406e0e0 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d094e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d04069e70 0x7f2d040ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d094e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d04069e70 0x7f2d040ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58184/0 (socket says 172.21.15.104:58184) 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d03fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2d0406e4b0 0x7f2d040c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56010/0 (socket says 172.21.15.104:56010) 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d094e3700 1 -- 172.21.15.104:0/508196325 learned_addr learned my addr 172.21.15.104:0/508196325 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.059+0000 7f2d08ce2700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d0406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d094e3700 1 -- 172.21.15.104:0/508196325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 msgr2=0x7f2d0406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d094e3700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d0406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d094e3700 1 -- 172.21.15.104:0/508196325 >> v1:172.21.15.104:6789/0 conn(0x7f2d0406e4b0 legacy=0x7f2d040c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d094e3700 1 -- 172.21.15.104:0/508196325 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040ca2d0 con 0x7f2d04069e70 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d037fe700 1 -- 172.21.15.104:0/508196325 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197179441 0 0) 0x7f2d0406d290 con 0x7f2d0406e4b0 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d094e3700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040ce9f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009e50 tx=0x7f2cfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da295f7e3bd6c656 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d037fe700 1 -- 172.21.15.104:0/508196325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc00e070 con 0x7f2d04069e70 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d037fe700 1 -- 172.21.15.104:0/508196325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00b9c0 con 0x7f2d04069e70 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d037fe700 1 -- 172.21.15.104:0/508196325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc00bd70 con 0x7f2d04069e70 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 -- 172.21.15.104:0/508196325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 msgr2=0x7f2d040ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040ce9f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009e50 tx=0x7f2cfc00b330 comp rx=0 tx=0).stop 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 -- 172.21.15.104:0/508196325 shutdown_connections 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040ce9f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 --2- 172.21.15.104:0/508196325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d0406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 -- 172.21.15.104:0/508196325 >> 172.21.15.104:0/508196325 conn(0x7f2d0406b8a0 msgr2=0x7f2d040cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 -- 172.21.15.104:0/508196325 shutdown_connections 2023-06-10T18:47:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.060+0000 7f2d0af46700 1 -- 172.21.15.104:0/508196325 wait complete. 2023-06-10T18:47:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 Processor -- start 2023-06-10T18:47:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 -- start start 2023-06-10T18:47:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d040b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 0x7f2d041f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040afbf0 con 0x7f2d04069e70 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040afed0 con 0x7f2d0406e4b0 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.062+0000 7f2d0af46700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040b01b0 con 0x7f2d0406e0e0 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58186/0 (socket says 172.21.15.104:58186) 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 -- 172.21.15.104:0/2816386814 learned_addr learned my addr 172.21.15.104:0/2816386814 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d03fff700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d040b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d094e3700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 0x7f2d041f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 -- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 msgr2=0x7f2d040b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d040b1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 -- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 msgr2=0x7f2d041f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 0x7f2d041f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 -- 172.21.15.104:0/2816386814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc009b30 con 0x7f2d04069e70 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d094e3700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 0x7f2d041f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d08ce2700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800d7d0 tx=0x7f2cf800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf80125f0 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8012750 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d040b0490 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8012b00 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.063+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d040b3690 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.064+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d040b0ae0 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.065+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2cf8012c60 con 0x7f2d04069e70 2023-06-10T18:47:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.065+0000 7f2d017fa700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 0x7f2ce806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.065+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2cf8089610 con 0x7f2d04069e70 2023-06-10T18:47:47.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.065+0000 7f2d03fff700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 0x7f2ce806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.066+0000 7f2d03fff700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 0x7f2ce806d190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2d04156720 tx=0x7f2cf0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.069+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2cf8054540 con 0x7f2d04069e70 2023-06-10T18:47:47.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.296+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2d04074050 con 0x7f2d04069e70 2023-06-10T18:47:47.794 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:47:47.789+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:47:47.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.797+0000 7f2d017fa700 1 -- 172.21.15.104:0/2816386814 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2cf8057b60 con 0x7f2d04069e70 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 msgr2=0x7f2ce806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 0x7f2ce806d190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2d04156720 tx=0x7f2cf0007de0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 msgr2=0x7f2d040b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800d7d0 tx=0x7f2cf800dae0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 shutdown_connections 2023-06-10T18:47:47.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ce806ace0 0x7f2ce806d190 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0406e4b0 0x7f2d041f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d04069e70 0x7f2d040b4380 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 --2- 172.21.15.104:0/2816386814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d0406e0e0 0x7f2d040b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 >> 172.21.15.104:0/2816386814 conn(0x7f2d0406b8a0 msgr2=0x7f2d040ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 shutdown_connections 2023-06-10T18:47:47.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.799+0000 7f2d0af46700 1 -- 172.21.15.104:0/2816386814 wait complete. 2023-06-10T18:47:47.807 INFO:tasks.ceph.mds.b.smithi114.stderr: -3775> 2023-06-10T18:45:44.863+0000 7f647c7be700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:47:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:47:47.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:47:47.836 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:47:47.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 Processor -- start 2023-06-10T18:47:47.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 -- start start 2023-06-10T18:47:47.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe2e806e6e0 0x7fe2e80b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e806e250 con 0x7fe2e815c580 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e815c950 con 0x7fe2e806cde0 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.936+0000 7fe2ee0a0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e80b1e80 con 0x7fe2e806e6e0 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe2e806e6e0 0x7fe2e80b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe2e815c580 0x7fe2e80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56022/0 (socket says 172.21.15.104:56022) 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e77fe700 1 -- 172.21.15.104:0/2287962044 learned_addr learned my addr 172.21.15.104:0/2287962044 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e67fc700 1 -- 172.21.15.104:0/2287962044 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057125826 0 0) 0x7fe2e806e250 con 0x7fe2e815c580 2023-06-10T18:47:47.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e67fc700 1 -- 172.21.15.104:0/2287962044 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d00035b0 con 0x7fe2e815c580 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e6ffd700 1 -- 172.21.15.104:0/2287962044 >> v1:172.21.15.114:6789/0 conn(0x7fe2e806cde0 legacy=0x7fe2e80ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e6ffd700 1 -- 172.21.15.104:0/2287962044 >> v1:172.21.15.104:6789/0 conn(0x7fe2e815c580 legacy=0x7fe2e80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e6ffd700 1 -- 172.21.15.104:0/2287962044 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e806d920 con 0x7fe2e806e6e0 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e6ffd700 1 --2- 172.21.15.104:0/2287962044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e806e6e0 0x7fe2e80b1780 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009840 tx=0x7fe2dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6748bbfa13ec37 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e67fc700 1 -- 172.21.15.104:0/2287962044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc011830 con 0x7fe2e806e6e0 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e67fc700 1 -- 172.21.15.104:0/2287962044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc011990 con 0x7fe2e806e6e0 2023-06-10T18:47:47.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.937+0000 7fe2e67fc700 1 -- 172.21.15.104:0/2287962044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc011d20 con 0x7fe2e806e6e0 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2287962044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e806e6e0 msgr2=0x7fe2e80b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2287962044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e806e6e0 0x7fe2e80b1780 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009840 tx=0x7fe2dc009b50 comp rx=0 tx=0).stop 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2287962044 shutdown_connections 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2287962044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e806e6e0 0x7fe2e80b1780 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2287962044 >> 172.21.15.104:0/2287962044 conn(0x7fe2e806b8a0 msgr2=0x7fe2e80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2287962044 shutdown_connections 2023-06-10T18:47:47.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.938+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2287962044 wait complete. 2023-06-10T18:47:47.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.939+0000 7fe2ee0a0700 1 Processor -- start 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.939+0000 7fe2ee0a0700 1 -- start start 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e806cde0 0x7fe2e81557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2e806e6e0 0x7fe2e8150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80b0fd0 con 0x7fe2e815c580 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80afdf0 con 0x7fe2e806cde0 2023-06-10T18:47:47.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2ee0a0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80b0cf0 con 0x7fe2e806e6e0 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58202/0 (socket says 172.21.15.104:58202) 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 -- 172.21.15.104:0/2490674778 learned_addr learned my addr 172.21.15.104:0/2490674778 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 -- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2e806e6e0 msgr2=0x7fe2e8150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2e806e6e0 0x7fe2e8150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 -- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e806cde0 msgr2=0x7fe2e81557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e77fe700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e806cde0 0x7fe2e81557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e806cde0 0x7fe2e81557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 -- 172.21.15.104:0/2490674778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0094a0 con 0x7fe2e815c580 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2e7fff700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000cc50 tx=0x7fe2e000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.940+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000ad80 con 0x7fe2e815c580 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.941+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e000ce50 con 0x7fe2e815c580 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.941+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0003c40 con 0x7fe2e815c580 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.941+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e80af830 con 0x7fe2e815c580 2023-06-10T18:47:47.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.941+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e80b5180 con 0x7fe2e815c580 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.942+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e81e2250 con 0x7fe2e815c580 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.942+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe2e00050d0 con 0x7fe2e815c580 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.942+0000 7fe2cffff700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 0x7fe2d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.942+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6526+0+0 (secure 0 0 0) 0x7fe2e0089860 con 0x7fe2e815c580 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.943+0000 7fe2e77fe700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 0x7fe2d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:47.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.943+0000 7fe2e77fe700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 0x7fe2d406d190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80010e0 tx=0x7fe2d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:47.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:47.947+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe2e81e2250 con 0x7fe2e815c580 2023-06-10T18:47:48.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.179+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe2e81e2250 con 0x7fe2e815c580 2023-06-10T18:47:48.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.802+0000 7fe2cffff700 1 -- 172.21.15.104:0/2490674778 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe2e81e2250 con 0x7fe2e815c580 2023-06-10T18:47:48.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 msgr2=0x7fe2d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:48.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 0x7fe2d406d190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80010e0 tx=0x7fe2d8009040 comp rx=0 tx=0).stop 2023-06-10T18:47:48.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 msgr2=0x7fe2e80b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:48.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000cc50 tx=0x7fe2e000a340 comp rx=0 tx=0).stop 2023-06-10T18:47:48.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 shutdown_connections 2023-06-10T18:47:48.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe2d406ace0 0x7fe2d406d190 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e806cde0 0x7fe2e81557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2e815c580 0x7fe2e80b0120 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 --2- 172.21.15.104:0/2490674778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2e806e6e0 0x7fe2e8150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 >> 172.21.15.104:0/2490674778 conn(0x7fe2e806b8a0 msgr2=0x7fe2e80ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:48.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 shutdown_connections 2023-06-10T18:47:48.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.804+0000 7fe2ee0a0700 1 -- 172.21.15.104:0/2490674778 wait complete. 2023-06-10T18:47:48.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:47:48.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.959+0000 7f54a56e2700 1 Processor -- start 2023-06-10T18:47:48.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.959+0000 7f54a56e2700 1 -- start start 2023-06-10T18:47:48.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54a00d04f0 0x7f54a00d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00d25c0 0x7f54a00d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 0x7f54a006b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a00d0e00 con 0x7f54a00d04f0 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a00af530 con 0x7f54a006b510 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f54a56e2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00af810 con 0x7f54a00d25c0 2023-06-10T18:47:48.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54a00d04f0 0x7f54a00d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54a00d04f0 0x7f54a00d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58206/0 (socket says 172.21.15.104:58206) 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 -- 172.21.15.104:0/3565379942 learned_addr learned my addr 172.21.15.104:0/3565379942 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549f7fe700 1 --2- 172.21.15.104:0/3565379942 >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 0x7f54a006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549e7fc700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00d25c0 0x7f54a00d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 -- 172.21.15.104:0/3565379942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00d25c0 msgr2=0x7f54a00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00d25c0 0x7f54a00d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 -- 172.21.15.104:0/3565379942 >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 msgr2=0x7f54a006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:48.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 --2- 172.21.15.104:0/3565379942 >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 0x7f54a006b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 -- 172.21.15.104:0/3565379942 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00afaf0 con 0x7f54a00d04f0 2023-06-10T18:47:48.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549f7fe700 1 --2- 172.21.15.104:0/3565379942 >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 0x7f54a006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:48.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.960+0000 7f549effd700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00d34d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f54a006d130 tx=0x7f5494009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36a307f33860895 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:48.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f549dffb700 1 -- 172.21.15.104:0/3565379942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494005e30 con 0x7f54a00d04f0 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f549dffb700 1 -- 172.21.15.104:0/3565379942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5494004030 con 0x7f54a00d04f0 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f549dffb700 1 -- 172.21.15.104:0/3565379942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940043e0 con 0x7f54a00d04f0 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 -- 172.21.15.104:0/3565379942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 msgr2=0x7f54a00d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00d34d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f54a006d130 tx=0x7f5494009bd0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 -- 172.21.15.104:0/3565379942 shutdown_connections 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00d34d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 --2- 172.21.15.104:0/3565379942 >> v2:172.21.15.114:3300/0 conn(0x7f54a006b510 0x7f54a006b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 --2- 172.21.15.104:0/3565379942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00d25c0 0x7f54a00d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 -- 172.21.15.104:0/3565379942 >> 172.21.15.104:0/3565379942 conn(0x7f54a0071ec0 msgr2=0x7f54a009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 -- 172.21.15.104:0/3565379942 shutdown_connections 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.961+0000 7f54a56e2700 1 -- 172.21.15.104:0/3565379942 wait complete. 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.962+0000 7f54a56e2700 1 Processor -- start 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 -- start start 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a006b510 0x7f54a0152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 0x7f54a01520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00b73a0 con 0x7f54a00d04f0 2023-06-10T18:47:48.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00b7680 con 0x7f54a00d25c0 2023-06-10T18:47:48.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f54a56e2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00b7960 con 0x7f54a006b510 2023-06-10T18:47:48.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f549e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f549e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58214/0 (socket says 172.21.15.104:58214) 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.963+0000 7f549e7fc700 1 -- 172.21.15.104:0/1145428330 learned_addr learned my addr 172.21.15.104:0/1145428330 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549f7fe700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 0x7f54a01520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549effd700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a006b510 0x7f54a0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 -- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a006b510 msgr2=0x7f54a0152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:48.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a006b510 0x7f54a0152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 -- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 msgr2=0x7f54a01520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:48.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 0x7f54a01520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:48.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 -- 172.21.15.104:0/1145428330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5494009500 con 0x7f54a00d04f0 2023-06-10T18:47:48.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549f7fe700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 0x7f54a01520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f549e7fc700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f549000c7f0 tx=0x7f549000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490011800 con 0x7f54a00d04f0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5490011960 con 0x7f54a00d04f0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490011cf0 con 0x7f54a00d04f0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a00b7c40 con 0x7f54a00d04f0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.964+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a0152610 con 0x7f54a00d04f0 2023-06-10T18:47:48.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.965+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0067010 con 0x7f54a00d04f0 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.965+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5490018360 con 0x7f54a00d04f0 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.966+0000 7f54877fe700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 0x7f548c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.966+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f5490089620 con 0x7f54a00d04f0 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.966+0000 7f549effd700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 0x7f548c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.966+0000 7f549effd700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 0x7f548c06d190 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f54a006d130 tx=0x7f5494008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:48.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:48.970+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5490054510 con 0x7f54a00d04f0 2023-06-10T18:47:49.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.158+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54a0074050 con 0x7f54a00d04f0 2023-06-10T18:47:49.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.808+0000 7f54877fe700 1 -- 172.21.15.104:0/1145428330 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5490057b30 con 0x7f54a00d04f0 2023-06-10T18:47:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 msgr2=0x7f548c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 0x7f548c06d190 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f54a006d130 tx=0x7f5494008070 comp rx=0 tx=0).stop 2023-06-10T18:47:49.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 msgr2=0x7f54a00b8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f549000c7f0 tx=0x7f549000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 shutdown_connections 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f548c06ace0 0x7f548c06d190 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a00d25c0 0x7f54a01520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a006b510 0x7f54a0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 --2- 172.21.15.104:0/1145428330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54a00d04f0 0x7f54a00b8440 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 >> 172.21.15.104:0/1145428330 conn(0x7f54a0071ec0 msgr2=0x7f54a0072940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:49.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 shutdown_connections 2023-06-10T18:47:49.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.810+0000 7f54a56e2700 1 -- 172.21.15.104:0/1145428330 wait complete. 2023-06-10T18:47:49.818 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:47:49.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:47:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.958+0000 7f7d78842700 1 Processor -- start 2023-06-10T18:47:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.958+0000 7f7d78842700 1 -- start start 2023-06-10T18:47:49.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.959+0000 7f7d78842700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d740b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.959+0000 7f7d78842700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d7406e6e0 0x7f7d740b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.959+0000 7f7d78842700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7406e250 con 0x7f7d7406cde0 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.959+0000 7f7d78842700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7415c950 con 0x7f7d7406e6e0 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.959+0000 7f7d78842700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740cf0e0 con 0x7f7d7415c580 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d7406e6e0 0x7f7d740b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d7406e6e0 0x7f7d740b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58220/0 (socket says 172.21.15.104:58220) 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 -- 172.21.15.104:0/4141984017 learned_addr learned my addr 172.21.15.104:0/4141984017 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d7259c700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d740b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 -- 172.21.15.104:0/4141984017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 msgr2=0x7f7d740b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d740b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 -- 172.21.15.104:0/4141984017 >> v1:172.21.15.114:6789/0 conn(0x7f7d7406cde0 legacy=0x7f7d74069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 -- 172.21.15.104:0/4141984017 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740b1e80 con 0x7f7d7406e6e0 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d71d9b700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406e6e0 0x7f7d740b15b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098f0 tx=0x7f7d64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dccab47f8cf91ff8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d7159a700 1 -- 172.21.15.104:0/4141984017 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400b8a0 con 0x7f7d7406e6e0 2023-06-10T18:47:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d7159a700 1 -- 172.21.15.104:0/4141984017 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6400ba00 con 0x7f7d7406e6e0 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.960+0000 7f7d7159a700 1 -- 172.21.15.104:0/4141984017 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400bd90 con 0x7f7d7406e6e0 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 -- 172.21.15.104:0/4141984017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406e6e0 msgr2=0x7f7d740b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406e6e0 0x7f7d740b15b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098f0 tx=0x7f7d64009c00 comp rx=0 tx=0).stop 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 -- 172.21.15.104:0/4141984017 shutdown_connections 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d740b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 --2- 172.21.15.104:0/4141984017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406e6e0 0x7f7d740b15b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 -- 172.21.15.104:0/4141984017 >> 172.21.15.104:0/4141984017 conn(0x7f7d7406b8a0 msgr2=0x7f7d740cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 -- 172.21.15.104:0/4141984017 shutdown_connections 2023-06-10T18:47:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.961+0000 7f7d78842700 1 -- 172.21.15.104:0/4141984017 wait complete. 2023-06-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.963+0000 7f7d78842700 1 Processor -- start 2023-06-10T18:47:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.963+0000 7f7d78842700 1 -- start start 2023-06-10T18:47:49.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.963+0000 7f7d78842700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.963+0000 7f7d78842700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7406e6e0 0x7f7d740c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d741563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74154c60 con 0x7f7d7406cde0 2023-06-10T18:47:49.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74154f50 con 0x7f7d7406e6e0 2023-06-10T18:47:49.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740c4aa0 con 0x7f7d7415c580 2023-06-10T18:47:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58226/0 (socket says 172.21.15.104:58226) 2023-06-10T18:47:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 -- 172.21.15.104:0/3537164145 learned_addr learned my addr 172.21.15.104:0/3537164145 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d71d9b700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7406e6e0 0x7f7d740c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d72d9d700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d741563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 -- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 msgr2=0x7f7d741563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d741563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 -- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7406e6e0 msgr2=0x7f7d740c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7406e6e0 0x7f7d740c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 -- 172.21.15.104:0/3537164145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64009550 con 0x7f7d7406cde0 2023-06-10T18:47:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d72d9d700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d741563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d7259c700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e730 tx=0x7f7d6800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800c070 con 0x7f7d7406cde0 2023-06-10T18:47:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d68013610 con 0x7f7d7406cde0 2023-06-10T18:47:49.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d74156c10 con 0x7f7d7406cde0 2023-06-10T18:47:49.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680139c0 con 0x7f7d7406cde0 2023-06-10T18:47:49.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.964+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d741571d0 con 0x7f7d7406cde0 2023-06-10T18:47:49.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.965+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d74074050 con 0x7f7d7406cde0 2023-06-10T18:47:49.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.966+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7d68013b20 con 0x7f7d7406cde0 2023-06-10T18:47:49.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.966+0000 7f7d62ffd700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 0x7f7d5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:49.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.966+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6088+0+0 (secure 0 0 0) 0x7f7d68088f70 con 0x7f7d7406cde0 2023-06-10T18:47:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.966+0000 7f7d71d9b700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 0x7f7d5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.967+0000 7f7d71d9b700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 0x7f7d5406d190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7d64014040 tx=0x7f7d64008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:49.970+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7d68057aa0 con 0x7f7d7406cde0 2023-06-10T18:47:50.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.164+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d740cfd40 con 0x7f7d7406cde0 2023-06-10T18:47:50.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.818+0000 7f7d62ffd700 1 -- 172.21.15.104:0/3537164145 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7d68057630 con 0x7f7d7406cde0 2023-06-10T18:47:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 msgr2=0x7f7d5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 0x7f7d5406d190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7d64014040 tx=0x7f7d64008030 comp rx=0 tx=0).stop 2023-06-10T18:47:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 msgr2=0x7f7d74154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e730 tx=0x7f7d6800ea40 comp rx=0 tx=0).stop 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 shutdown_connections 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7415c580 0x7f7d741563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d7406cde0 0x7f7d74154720 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d5406ace0 0x7f7d5406d190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 --2- 172.21.15.104:0/3537164145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7406e6e0 0x7f7d740c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 >> 172.21.15.104:0/3537164145 conn(0x7f7d7406b8a0 msgr2=0x7f7d7415a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:50.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 shutdown_connections 2023-06-10T18:47:50.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.821+0000 7f7d78842700 1 -- 172.21.15.104:0/3537164145 wait complete. 2023-06-10T18:47:50.829 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:47:50.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:47:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.960+0000 7f6f556a9700 1 Processor -- start 2023-06-10T18:47:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.960+0000 7f6f556a9700 1 -- start start 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f556a9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6f5015c580 0x7f6f500b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f556a9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5006e250 con 0x7f6f5006cde0 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f556a9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5015c950 con 0x7f6f5006e6e0 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f556a9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f500b1e80 con 0x7f6f5015c580 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f4f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6f5006cde0 0x7f6f500ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56064/0 (socket says 172.21.15.104:56064) 2023-06-10T18:47:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f4f7fe700 1 -- 172.21.15.104:0/1993292942 learned_addr learned my addr 172.21.15.104:0/1993292942 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683702908 0 0) 0x7f6f5006e250 con 0x7f6f5006cde0 2023-06-10T18:47:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f380035b0 con 0x7f6f5006cde0 2023-06-10T18:47:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.961+0000 7f6f4effd700 1 --2- 172.21.15.104:0/1993292942 >> v2:172.21.15.114:3300/0 conn(0x7f6f5015c580 0x7f6f500b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059470609 0 0) 0x7f6f380035b0 con 0x7f6f5006cde0 2023-06-10T18:47:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 >> v2:172.21.15.114:3300/0 conn(0x7f6f5015c580 msgr2=0x7f6f500b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 --2- 172.21.15.104:0/1993292942 >> v2:172.21.15.114:3300/0 conn(0x7f6f5015c580 0x7f6f500b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 >> v1:172.21.15.114:6789/0 conn(0x7f6f5006e6e0 legacy=0x7f6f500b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5006d920 con 0x7f6f5006cde0 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6f480019f0 con 0x7f6f5006cde0 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 >> v1:172.21.15.104:6789/0 conn(0x7f6f5006cde0 legacy=0x7f6f500ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f38003a00 0x7f6f38003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38004850 0x7f6f38004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 0x7f6f3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500b1e80 con 0x7f6f38005200 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3800c080 con 0x7f6f38003a00 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3800c240 con 0x7f6f38004850 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 0x7f6f3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4effd700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f38003a00 0x7f6f38003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4e7fc700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38004850 0x7f6f38004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 -- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38004850 msgr2=0x7f6f38004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38004850 0x7f6f38004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 -- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f38003a00 msgr2=0x7f6f38003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f38003a00 0x7f6f38003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 -- 172.21.15.104:0/1993292942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380035b0 con 0x7f6f38005200 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 -- 172.21.15.104:0/1993292942 reap_dead start 2023-06-10T18:47:50.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 0x7f6f3800ba10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800a830 tx=0x7f6f38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde965f974a98a53 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800a2f0 con 0x7f6f38005200 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48012600 con 0x7f6f38005200 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.962+0000 7f6f4dffb700 1 -- 172.21.15.104:0/1993292942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480129b0 con 0x7f6f38005200 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 -- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 msgr2=0x7f6f3800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 0x7f6f3800ba10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800a830 tx=0x7f6f38005880 comp rx=0 tx=0).stop 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 -- 172.21.15.104:0/1993292942 shutdown_connections 2023-06-10T18:47:50.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 --2- 172.21.15.104:0/1993292942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f38005200 0x7f6f3800ba10 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 -- 172.21.15.104:0/1993292942 >> 172.21.15.104:0/1993292942 conn(0x7f6f5006b8a0 msgr2=0x7f6f500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:50.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 -- 172.21.15.104:0/1993292942 shutdown_connections 2023-06-10T18:47:50.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.963+0000 7f6f556a9700 1 -- 172.21.15.104:0/1993292942 wait complete. 2023-06-10T18:47:50.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.964+0000 7f6f556a9700 1 Processor -- start 2023-06-10T18:47:50.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.964+0000 7f6f556a9700 1 -- start start 2023-06-10T18:47:50.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38005200 0x7f6f500c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 0x7f6f501567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c4de0 con 0x7f6f5014e440 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c6000 con 0x7f6f50156380 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f556a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f500c41d0 con 0x7f6f38005200 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f4e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f4e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58248/0 (socket says 172.21.15.104:58248) 2023-06-10T18:47:50.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f4e7fc700 1 -- 172.21.15.104:0/2486324443 learned_addr learned my addr 172.21.15.104:0/2486324443 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 0x7f6f501567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.965+0000 7f6f4effd700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38005200 0x7f6f500c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 -- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38005200 msgr2=0x7f6f500c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38005200 0x7f6f500c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 -- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 msgr2=0x7f6f501567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 0x7f6f501567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:50.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 -- 172.21.15.104:0/2486324443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500c44b0 con 0x7f6f5014e440 2023-06-10T18:47:50.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4f7fe700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 0x7f6f501567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:50.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f4e7fc700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e4c0 tx=0x7f6f4000e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:50.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f400052e0 con 0x7f6f5014e440 2023-06-10T18:47:50.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f40005440 con 0x7f6f5014e440 2023-06-10T18:47:50.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f500c4790 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f400038c0 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.966+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f500c5270 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.967+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f500c4f90 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.967+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6f40003a20 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.968+0000 7f6f377fe700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 0x7f6f3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.968+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f6f400888d0 con 0x7f6f5014e440 2023-06-10T18:47:50.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.968+0000 7f6f4effd700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 0x7f6f3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:50.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.968+0000 7f6f4effd700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 0x7f6f3c06d190 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009b00 tx=0x7f6f44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:50.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:50.972+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6f500c4f90 con 0x7f6f5014e440 2023-06-10T18:47:51.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.171+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6f50074050 con 0x7f6f5014e440 2023-06-10T18:47:51.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.174+0000 7f6f377fe700 1 -- 172.21.15.104:0/2486324443 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6f40053bf0 con 0x7f6f5014e440 2023-06-10T18:47:51.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 msgr2=0x7f6f3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 0x7f6f3c06d190 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009b00 tx=0x7f6f44008040 comp rx=0 tx=0).stop 2023-06-10T18:47:51.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 msgr2=0x7f6f500afcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e4c0 tx=0x7f6f4000e880 comp rx=0 tx=0).stop 2023-06-10T18:47:51.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 shutdown_connections 2023-06-10T18:47:51.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6f3c06ace0 0x7f6f3c06d190 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f50156380 0x7f6f501567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f38005200 0x7f6f500c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 --2- 172.21.15.104:0/2486324443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f5014e440 0x7f6f500afcc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 >> 172.21.15.104:0/2486324443 conn(0x7f6f5006b8a0 msgr2=0x7f6f500cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:51.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 shutdown_connections 2023-06-10T18:47:51.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.176+0000 7f6f556a9700 1 -- 172.21.15.104:0/2486324443 wait complete. 2023-06-10T18:47:51.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn' 2023-06-10T18:47:51.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.350+0000 7fb70d3c0700 1 Processor -- start 2023-06-10T18:47:51.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 -- start start 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 0x7fb7080ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7080cae10 0x7fb7080cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7080d0340 con 0x7fb7080ce390 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080d1660 con 0x7fb7080cae10 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.351+0000 7fb70d3c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70815cb60 con 0x7fb70806c720 2023-06-10T18:47:51.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7067fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 0x7fb7080ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7080cae10 0x7fb7080cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7080cae10 0x7fb7080cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:45760/0 (socket says 172.21.15.104:45760) 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 -- 172.21.15.104:0/1218880030 learned_addr learned my addr 172.21.15.104:0/1218880030 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 -- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 msgr2=0x7fb7080ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 0x7fb7080ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 -- 172.21.15.104:0/1218880030 >> v1:172.21.15.114:6789/0 conn(0x7fb7080ce390 legacy=0x7fb708160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 -- 172.21.15.104:0/1218880030 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080ceca0 con 0x7fb7080cae10 2023-06-10T18:47:51.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7067fc700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 0x7fb7080ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:47:51.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb7077fe700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7080cae10 0x7fb7080cb270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb700009e50 tx=0x7fb70000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a369d7e46620114 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.352+0000 7fb705ffb700 1 -- 172.21.15.104:0/1218880030 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70000e070 con 0x7fb7080cae10 2023-06-10T18:47:51.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb705ffb700 1 -- 172.21.15.104:0/1218880030 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb700003a20 con 0x7fb7080cae10 2023-06-10T18:47:51.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb705ffb700 1 -- 172.21.15.104:0/1218880030 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700003dd0 con 0x7fb7080cae10 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 -- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7080cae10 msgr2=0x7fb7080cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7080cae10 0x7fb7080cb270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb700009e50 tx=0x7fb70000b330 comp rx=0 tx=0).stop 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 -- 172.21.15.104:0/1218880030 shutdown_connections 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7080cae10 0x7fb7080cb270 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/1218880030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb70806c720 0x7fb7080ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 -- 172.21.15.104:0/1218880030 >> 172.21.15.104:0/1218880030 conn(0x7fb708069e70 msgr2=0x7fb70806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:51.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 -- 172.21.15.104:0/1218880030 shutdown_connections 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.353+0000 7fb70d3c0700 1 -- 172.21.15.104:0/1218880030 wait complete. 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.354+0000 7fb70d3c0700 1 Processor -- start 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 -- start start 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 0x7fb70815f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 0x7fb7080c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080c7a30 con 0x7fb7080ce390 2023-06-10T18:47:51.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080c7d10 con 0x7fb70806c720 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb70d3c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080c7ff0 con 0x7fb7080cae10 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb7077fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58264/0 (socket says 172.21.15.104:58264) 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.355+0000 7fb7077fe700 1 -- 172.21.15.104:0/264910409 learned_addr learned my addr 172.21.15.104:0/264910409 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7067fc700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 0x7fb7080c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb706ffd700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 0x7fb70815f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 -- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 msgr2=0x7fb7080c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 0x7fb7080c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 -- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 msgr2=0x7fb70815f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 0x7fb70815f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 -- 172.21.15.104:0/264910409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f80094a0 con 0x7fb7080ce390 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb706ffd700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 0x7fb70815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7067fc700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 0x7fb7080c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:51.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb7077fe700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb70806ad60 tx=0x7fb700004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700004440 con 0x7fb7080ce390 2023-06-10T18:47:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7000045c0 con 0x7fb7080ce390 2023-06-10T18:47:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb700009b30 con 0x7fb7080ce390 2023-06-10T18:47:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700004970 con 0x7fb7080ce390 2023-06-10T18:47:51.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.356+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7080c85b0 con 0x7fb7080ce390 2023-06-10T18:47:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.357+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708074050 con 0x7fb7080ce390 2023-06-10T18:47:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.357+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb700019070 con 0x7fb7080ce390 2023-06-10T18:47:51.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.358+0000 7fb6ef7fe700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 0x7fb6f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.358+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fb70008a300 con 0x7fb7080ce390 2023-06-10T18:47:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.358+0000 7fb706ffd700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 0x7fb6f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.359+0000 7fb706ffd700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 0x7fb6f406d190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc001940 tx=0x7fb6fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.363+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb700059090 con 0x7fb7080ce390 2023-06-10T18:47:51.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.623+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn"]} v 0) v1 -- 0x7fb7080b2e10 con 0x7fb7080ce390 2023-06-10T18:47:51.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.819+0000 7fb6ef7fe700 1 -- 172.21.15.104:0/264910409 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_abort_conn"]}]=0 v190) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb700011390 con 0x7fb7080ce390 2023-06-10T18:47:51.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 msgr2=0x7fb6f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 0x7fb6f406d190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc001940 tx=0x7fb6fc009040 comp rx=0 tx=0).stop 2023-06-10T18:47:51.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 msgr2=0x7fb7081501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb70806ad60 tx=0x7fb700004ad0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 shutdown_connections 2023-06-10T18:47:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb6f406ace0 0x7fb6f406d190 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70806c720 0x7fb70815f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7080ce390 0x7fb7081501c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 --2- 172.21.15.104:0/264910409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7080cae10 0x7fb7080c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 >> 172.21.15.104:0/264910409 conn(0x7fb708069e70 msgr2=0x7fb70806fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 shutdown_connections 2023-06-10T18:47:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.822+0000 7fb70d3c0700 1 -- 172.21.15.104:0/264910409 wait complete. 2023-06-10T18:47:51.839 INFO:tasks.cephfs_test_runner:test_abort_conn (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:47:51.840 INFO:tasks.cephfs_test_runner:Starting test: test_basic (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:47:51.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic' 2023-06-10T18:47:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.954+0000 7fc56e432700 1 Processor -- start 2023-06-10T18:47:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.954+0000 7fc56e432700 1 -- start start 2023-06-10T18:47:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5680b4ff0 0x7fc5680ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 0x7fc56815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680cecf0 con 0x7fc56815a420 2023-06-10T18:47:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5680b6070 con 0x7fc5680b4ff0 2023-06-10T18:47:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680b1d70 con 0x7fc5680ce890 2023-06-10T18:47:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56c9cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 0x7fc56815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5680b4ff0 0x7fc5680ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc567fff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc56815a420 0x7fc5680b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:33710/0 (socket says 172.21.15.104:33710) 2023-06-10T18:47:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc567fff700 1 -- 172.21.15.104:0/1073328237 learned_addr learned my addr 172.21.15.104:0/1073328237 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 -- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 msgr2=0x7fc56815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 0x7fc56815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 -- 172.21.15.104:0/1073328237 >> v1:172.21.15.114:6789/0 conn(0x7fc56815a420 legacy=0x7fc5680b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 -- 172.21.15.104:0/1073328237 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680b5460 con 0x7fc5680b4ff0 2023-06-10T18:47:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56c9cf700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 0x7fc56815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:47:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680b4ff0 0x7fc5680ce350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc558009840 tx=0x7fc558009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2285ef2c841b8f13 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc566ffd700 1 -- 172.21.15.104:0/1073328237 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558011830 con 0x7fc5680b4ff0 2023-06-10T18:47:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc566ffd700 1 -- 172.21.15.104:0/1073328237 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558011990 con 0x7fc5680b4ff0 2023-06-10T18:47:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc566ffd700 1 -- 172.21.15.104:0/1073328237 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558011d20 con 0x7fc5680b4ff0 2023-06-10T18:47:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 -- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680b4ff0 msgr2=0x7fc5680ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680b4ff0 0x7fc5680ce350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc558009840 tx=0x7fc558009b50 comp rx=0 tx=0).stop 2023-06-10T18:47:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 -- 172.21.15.104:0/1073328237 shutdown_connections 2023-06-10T18:47:51.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5680ce890 0x7fc56815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.955+0000 7fc56e432700 1 --2- 172.21.15.104:0/1073328237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680b4ff0 0x7fc5680ce350 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.956+0000 7fc56e432700 1 -- 172.21.15.104:0/1073328237 >> 172.21.15.104:0/1073328237 conn(0x7fc568089fd0 msgr2=0x7fc5680d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:51.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.956+0000 7fc56e432700 1 -- 172.21.15.104:0/1073328237 shutdown_connections 2023-06-10T18:47:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.956+0000 7fc56e432700 1 -- 172.21.15.104:0/1073328237 wait complete. 2023-06-10T18:47:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.958+0000 7fc56e432700 1 Processor -- start 2023-06-10T18:47:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.958+0000 7fc56e432700 1 -- start start 2023-06-10T18:47:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 0x7fc5681e1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56815a420 0x7fc568152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5681e1e50 con 0x7fc5680b4ff0 2023-06-10T18:47:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568152890 con 0x7fc5680ce890 2023-06-10T18:47:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc568155f60 con 0x7fc56815a420 2023-06-10T18:47:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58274/0 (socket says 172.21.15.104:58274) 2023-06-10T18:47:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 -- 172.21.15.104:0/1682250137 learned_addr learned my addr 172.21.15.104:0/1682250137 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 0x7fc5681e1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56c9cf700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56815a420 0x7fc568152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 -- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56815a420 msgr2=0x7fc568152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56815a420 0x7fc568152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 -- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 msgr2=0x7fc5681e1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 0x7fc5681e1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 -- 172.21.15.104:0/1682250137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560009a70 con 0x7fc5680b4ff0 2023-06-10T18:47:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 0x7fc5681e1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:47:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc567fff700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c480 tx=0x7fc55c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c005d70 con 0x7fc5680b4ff0 2023-06-10T18:47:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c00ce80 con 0x7fc5680b4ff0 2023-06-10T18:47:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580094a0 con 0x7fc5680b4ff0 2023-06-10T18:47:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0072d0 con 0x7fc5680b4ff0 2023-06-10T18:47:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.959+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5681e2370 con 0x7fc5680b4ff0 2023-06-10T18:47:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.960+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc568074050 con 0x7fc5680b4ff0 2023-06-10T18:47:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.961+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc55c007430 con 0x7fc5680b4ff0 2023-06-10T18:47:51.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.961+0000 7fc564ff9700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 0x7fc55406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:51.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.961+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fc55c088440 con 0x7fc5680b4ff0 2023-06-10T18:47:51.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.961+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 0x7fc55406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:51.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.962+0000 7fc5677fe700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 0x7fc55406d190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc558008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:51.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:51.965+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc55c0537f0 con 0x7fc5680b4ff0 2023-06-10T18:47:52.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.227+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic"]} v 0) v1 -- 0x7fc56806f6c0 con 0x7fc5680b4ff0 2023-06-10T18:47:52.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.822+0000 7fc564ff9700 1 -- 172.21.15.104:0/1682250137 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic"]}]=0 v191) v1 ==== 145+0+0 (secure 0 0 0) 0x7fc55c056e10 con 0x7fc5680b4ff0 2023-06-10T18:47:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 msgr2=0x7fc55406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 0x7fc55406d190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc558008030 comp rx=0 tx=0).stop 2023-06-10T18:47:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 msgr2=0x7fc5681f3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c480 tx=0x7fc55c00c790 comp rx=0 tx=0).stop 2023-06-10T18:47:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 shutdown_connections 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56815a420 0x7fc568152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5680b4ff0 0x7fc5681f3a50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc55406ace0 0x7fc55406d190 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 --2- 172.21.15.104:0/1682250137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5680ce890 0x7fc5681e1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 >> 172.21.15.104:0/1682250137 conn(0x7fc568089fd0 msgr2=0x7fc5680b48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 shutdown_connections 2023-06-10T18:47:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.824+0000 7fc56e432700 1 -- 172.21.15.104:0/1682250137 wait complete. 2023-06-10T18:47:52.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:47:52.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.960+0000 7f91adf8d700 1 Processor -- start 2023-06-10T18:47:52.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 -- start start 2023-06-10T18:47:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91a815c730 0x7f91a80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f91a80cd720 0x7f91a806cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a806d2b0 0x7f91a806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a80cdbc0 con 0x7f91a80cd720 2023-06-10T18:47:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a815cb00 con 0x7f91a815c730 2023-06-10T18:47:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.961+0000 7f91adf8d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80acd90 con 0x7f91a806d2b0 2023-06-10T18:47:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f91a80cd720 0x7f91a806cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f91a80cd720 0x7f91a806cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58284/0 (socket says 172.21.15.104:58284) 2023-06-10T18:47:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 -- 172.21.15.104:0/2817433431 learned_addr learned my addr 172.21.15.104:0/2817433431 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a7fff700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a806d2b0 0x7f91a806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a77fe700 1 --2- 172.21.15.104:0/2817433431 >> v2:172.21.15.114:3300/0 conn(0x7f91a815c730 0x7f91a80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 -- 172.21.15.104:0/2817433431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a806d2b0 msgr2=0x7f91a806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a806d2b0 0x7f91a806d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 -- 172.21.15.104:0/2817433431 >> v2:172.21.15.114:3300/0 conn(0x7f91a815c730 msgr2=0x7f91a80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/2817433431 >> v2:172.21.15.114:3300/0 conn(0x7f91a815c730 0x7f91a80cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 -- 172.21.15.104:0/2817433431 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a80ac460 con 0x7f91a80cd720 2023-06-10T18:47:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a80cd720 0x7f91a806cd70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f91980098f0 tx=0x7f9198009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c7de6db49bbcd9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a67fc700 1 -- 172.21.15.104:0/2817433431 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919800b860 con 0x7f91a80cd720 2023-06-10T18:47:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a67fc700 1 -- 172.21.15.104:0/2817433431 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919800b9c0 con 0x7f91a80cd720 2023-06-10T18:47:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.962+0000 7f91a67fc700 1 -- 172.21.15.104:0/2817433431 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919800bd50 con 0x7f91a80cd720 2023-06-10T18:47:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 -- 172.21.15.104:0/2817433431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a80cd720 msgr2=0x7f91a806cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a80cd720 0x7f91a806cd70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f91980098f0 tx=0x7f9198009c00 comp rx=0 tx=0).stop 2023-06-10T18:47:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 -- 172.21.15.104:0/2817433431 shutdown_connections 2023-06-10T18:47:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a806d2b0 0x7f91a806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 --2- 172.21.15.104:0/2817433431 >> v2:172.21.15.114:3300/0 conn(0x7f91a815c730 0x7f91a80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 --2- 172.21.15.104:0/2817433431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a80cd720 0x7f91a806cd70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 -- 172.21.15.104:0/2817433431 >> 172.21.15.104:0/2817433431 conn(0x7f91a806b8a0 msgr2=0x7f91a80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 -- 172.21.15.104:0/2817433431 shutdown_connections 2023-06-10T18:47:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.963+0000 7f91adf8d700 1 -- 172.21.15.104:0/2817433431 wait complete. 2023-06-10T18:47:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.964+0000 7f91adf8d700 1 Processor -- start 2023-06-10T18:47:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.964+0000 7f91adf8d700 1 -- start start 2023-06-10T18:47:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 0x7f91a80c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 0x7f91a806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81f4fa0 con 0x7f91a806d2b0 2023-06-10T18:47:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81f5b20 con 0x7f91a815c730 2023-06-10T18:47:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91adf8d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81f51f0 con 0x7f91a80cd720 2023-06-10T18:47:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58288/0 (socket says 172.21.15.104:58288) 2023-06-10T18:47:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 -- 172.21.15.104:0/128275830 learned_addr learned my addr 172.21.15.104:0/128275830 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a7fff700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 0x7f91a806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 0x7f91a80c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 -- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 msgr2=0x7f91a80c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 0x7f91a80c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 -- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 msgr2=0x7f91a806e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 0x7f91a806e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 -- 172.21.15.104:0/128275830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198009550 con 0x7f91a806d2b0 2023-06-10T18:47:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a7fff700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 0x7f91a806e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 0x7f91a80c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.965+0000 7f91a77fe700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f919000c7f0 tx=0x7f919000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:52.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190011800 con 0x7f91a806d2b0 2023-06-10T18:47:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190011960 con 0x7f91a806d2b0 2023-06-10T18:47:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a81f54d0 con 0x7f91a806d2b0 2023-06-10T18:47:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190011cf0 con 0x7f91a806d2b0 2023-06-10T18:47:52.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a8157160 con 0x7f91a806d2b0 2023-06-10T18:47:52.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.966+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a8074050 con 0x7f91a806d2b0 2023-06-10T18:47:52.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.968+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9190024480 con 0x7f91a806d2b0 2023-06-10T18:47:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.968+0000 7f919ffff700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 0x7f918806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.968+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f9190089520 con 0x7f91a806d2b0 2023-06-10T18:47:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.968+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 0x7f918806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:52.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.969+0000 7f91a6ffd700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 0x7f918806d190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f91a809d610 tx=0x7f9198008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:52.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:52.972+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9190054920 con 0x7f91a806d2b0 2023-06-10T18:47:53.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.159+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f91a80ce7e0 con 0x7f91a806d2b0 2023-06-10T18:47:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.165+0000 7f919ffff700 1 -- 172.21.15.104:0/128275830 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9190057f40 con 0x7f91a806d2b0 2023-06-10T18:47:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 msgr2=0x7f918806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 0x7f918806d190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f91a809d610 tx=0x7f9198008030 comp rx=0 tx=0).stop 2023-06-10T18:47:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 msgr2=0x7f91a8156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f919000c7f0 tx=0x7f919000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 shutdown_connections 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918806ace0 0x7f918806d190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a815c730 0x7f91a806e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a806d2b0 0x7f91a8156c20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 --2- 172.21.15.104:0/128275830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a80cd720 0x7f91a80c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 >> 172.21.15.104:0/128275830 conn(0x7f91a806b8a0 msgr2=0x7f91a80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:53.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 shutdown_connections 2023-06-10T18:47:53.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.167+0000 7f91adf8d700 1 -- 172.21.15.104:0/128275830 wait complete. 2023-06-10T18:47:53.184 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:47:53.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:47:53.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:47:53.278 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:47:53.278 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:47:53.279 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:53.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:47:53.296 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:47:53.297 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:53.316 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:47:53.317 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:47:53.317 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:53.353 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:47:53.354 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:47:53.355 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:47:53.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:47:53.379 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:47:53.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:47:53.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.507+0000 7facf5233700 1 Processor -- start 2023-06-10T18:47:53.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.507+0000 7facf5233700 1 -- start start 2023-06-10T18:47:53.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facf5233700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facf00cd850 0x7facf006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facf5233700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 0x7facf006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facf5233700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf00c9e60 con 0x7facf00aeaa0 2023-06-10T18:47:53.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facf5233700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf00c9bc0 con 0x7facf00cd850 2023-06-10T18:47:53.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facf5233700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00aee70 con 0x7facf00cb300 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facee59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facf00cd850 0x7facf006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facee59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facf00cd850 0x7facf006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58298/0 (socket says 172.21.15.104:58298) 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facee59c700 1 -- 172.21.15.104:0/897231625 learned_addr learned my addr 172.21.15.104:0/897231625 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.508+0000 7facef59e700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 0x7facf006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facee59c700 1 -- 172.21.15.104:0/897231625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 msgr2=0x7facf006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facee59c700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 0x7facf006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facee59c700 1 -- 172.21.15.104:0/897231625 >> v1:172.21.15.114:6789/0 conn(0x7facf00aeaa0 legacy=0x7facf00cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facee59c700 1 -- 172.21.15.104:0/897231625 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf006e540 con 0x7facf00cd850 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facef59e700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 0x7facf006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facee59c700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cd850 0x7facf006cd50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7face00098f0 tx=0x7face0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c323d6e23f551bfb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facedd9b700 1 -- 172.21.15.104:0/897231625 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face000b8a0 con 0x7facf00cd850 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facedd9b700 1 -- 172.21.15.104:0/897231625 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face000ba00 con 0x7facf00cd850 2023-06-10T18:47:53.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facedd9b700 1 -- 172.21.15.104:0/897231625 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face000bd90 con 0x7facf00cd850 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 -- 172.21.15.104:0/897231625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cd850 msgr2=0x7facf006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cd850 0x7facf006cd50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7face00098f0 tx=0x7face0009c00 comp rx=0 tx=0).stop 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 -- 172.21.15.104:0/897231625 shutdown_connections 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00cb300 0x7facf006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 --2- 172.21.15.104:0/897231625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cd850 0x7facf006cd50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 -- 172.21.15.104:0/897231625 >> 172.21.15.104:0/897231625 conn(0x7facf006b8a0 msgr2=0x7facf00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.509+0000 7facf5233700 1 -- 172.21.15.104:0/897231625 shutdown_connections 2023-06-10T18:47:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.510+0000 7facf5233700 1 -- 172.21.15.104:0/897231625 wait complete. 2023-06-10T18:47:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.511+0000 7facf5233700 1 Processor -- start 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.511+0000 7facf5233700 1 -- start start 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00aeaa0 0x7facf015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 0x7facf015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf00c86a0 con 0x7facf00cb300 2023-06-10T18:47:53.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0141490 con 0x7facf00cd850 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00c6640 con 0x7facf00aeaa0 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58304/0 (socket says 172.21.15.104:58304) 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 -- 172.21.15.104:0/3249623197 learned_addr learned my addr 172.21.15.104:0/3249623197 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facef59e700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 0x7facf015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7faceed9d700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00aeaa0 0x7facf015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 -- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00aeaa0 msgr2=0x7facf015a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00aeaa0 0x7facf015a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 -- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 msgr2=0x7facf015bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 0x7facf015bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 -- 172.21.15.104:0/3249623197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face0009550 con 0x7facf00cb300 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facef59e700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 0x7facf015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facee59c700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7face0014040 tx=0x7face0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face00036a0 con 0x7facf00cb300 2023-06-10T18:47:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face001e070 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0003a40 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.512+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf015c060 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.513+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf01416a0 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.513+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf0074050 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.514+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7face0003ba0 con 0x7facf00cb300 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.514+0000 7facd77fe700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 0x7facdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.514+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5616+0+0 (secure 0 0 0) 0x7face0020070 con 0x7facf00cb300 2023-06-10T18:47:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.515+0000 7faceed9d700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 0x7facdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.515+0000 7faceed9d700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 0x7facdc06d190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7face4000fb0 tx=0x7face4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.519+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7face0055a10 con 0x7facf00cb300 2023-06-10T18:47:53.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.751+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facf006d3b0 con 0x7facf00cb300 2023-06-10T18:47:53.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.752+0000 7facd77fe700 1 -- 172.21.15.104:0/3249623197 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3538 (secure 0 0 0) 0x7face0015df0 con 0x7facf00cb300 2023-06-10T18:47:53.757 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:53.757 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3426813332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3426813332},{"type":"v1","addr":"172.21.15.114:6837","nonce":3426813332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/518241314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":518241314},{"type":"v1","addr":"172.21.15.104:6834","nonce":518241314}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1498009362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1498009362},{"type":"v1","addr":"172.21.15.104:6835","nonce":1498009362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3235427694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3235427694},{"type":"v1","addr":"172.21.15.114:6836","nonce":3235427694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2023-06-10T18:47:53.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 msgr2=0x7facdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 0x7facdc06d190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7face4000fb0 tx=0x7face4009040 comp rx=0 tx=0).stop 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 msgr2=0x7facf00c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7face0014040 tx=0x7face0004750 comp rx=0 tx=0).stop 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 shutdown_connections 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facdc06ace0 0x7facdc06d190 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facf00cd850 0x7facf015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facf00aeaa0 0x7facf015a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 --2- 172.21.15.104:0/3249623197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facf00cb300 0x7facf00c8160 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 >> 172.21.15.104:0/3249623197 conn(0x7facf006b8a0 msgr2=0x7facf00b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 shutdown_connections 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.754+0000 7facf5233700 1 -- 172.21.15.104:0/3249623197 wait complete. 2023-06-10T18:47:53.760 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 19 2023-06-10T18:47:53.770 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:47:53.771 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:47:53.771 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:47:53.771 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:47:53.772 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:47:53.772 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:47:53.772 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:47:53.772 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:47:53.773 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:47:53.773 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:47:53.773 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:47:53.773 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:47:53.773 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:47:53.774 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:47:53.774 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:47:53.774 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:47:53.782 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:47:53.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:47:53.783 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:47:53.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:47:53.784 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:47:53.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:47:53.785 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:47:53.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:47:53.824 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:47:53.825 DEBUG:teuthology.parallel:result is None 2023-06-10T18:47:53.826 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:47:53.827 DEBUG:teuthology.parallel:result is None 2023-06-10T18:47:53.827 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:47:53.828 DEBUG:teuthology.parallel:result is None 2023-06-10T18:47:53.828 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:47:53.829 DEBUG:teuthology.parallel:result is None 2023-06-10T18:47:53.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:47:53.903 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:47:53.904 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:47:53.904 INFO:tasks.ceph.mds.c.smithi104.stderr:2023-06-10T18:47:53.899+0000 7f50935c3700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-06-10T18:47:53.917 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:47:53.917 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:47:53.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.958+0000 7f4010870700 1 Processor -- start 2023-06-10T18:47:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.958+0000 7f4010870700 1 -- start start 2023-06-10T18:47:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.959+0000 7f4010870700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c0ade00 0x7f400c0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.959+0000 7f4010870700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0cee70 con 0x7f400c0ada30 2023-06-10T18:47:53.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.959+0000 7f4010870700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0cf150 con 0x7f400c0aeb00 2023-06-10T18:47:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.959+0000 7f4010870700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0cf430 con 0x7f400c0ade00 2023-06-10T18:47:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f400c0ada30 0x7f400c0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56162/0 (socket says 172.21.15.104:56162) 2023-06-10T18:47:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400a59c700 1 -- 172.21.15.104:0/247599244 learned_addr learned my addr 172.21.15.104:0/247599244 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f4009d9b700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c0ade00 0x7f400c0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678885806 0 0) 0x7f400c0cee70 con 0x7f400c0ada30 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff40035b0 con 0x7f400c0ada30 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984847650 0 0) 0x7f400c0cf150 con 0x7f400c0aeb00 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c0cee70 con 0x7f400c0aeb00 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033737689 0 0) 0x7f3ff40035b0 con 0x7f400c0ada30 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c0ade00 msgr2=0x7f400c0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c0ade00 0x7f400c0ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 >> v1:172.21.15.114:6789/0 conn(0x7f400c0aeb00 legacy=0x7f400c0ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c06cf10 con 0x7f400c0ada30 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f40000017f0 con 0x7f400c0ada30 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 >> v1:172.21.15.104:6789/0 conn(0x7f400c0ada30 legacy=0x7f400c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ff4003bc0 0x7f3ff4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f3ff4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ff4005200 0x7f3ff400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0cf430 con 0x7f3ff4004850 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff400bf80 con 0x7f3ff4005200 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff400c140 con 0x7f3ff4003bc0 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f4009d9b700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f3ff4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400a59c700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ff4003bc0 0x7f3ff4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.960+0000 7f400ad9d700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ff4005200 0x7f3ff400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 -- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ff4003bc0 msgr2=0x7f3ff4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ff4003bc0 0x7f3ff4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 -- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ff4005200 msgr2=0x7f3ff400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ff4005200 0x7f3ff400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 -- 172.21.15.104:0/247599244 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff40035b0 con 0x7f3ff4004850 2023-06-10T18:47:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4009d9b700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f3ff4004cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009b30 tx=0x7f3ff4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e8f9d93733930f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00e6c0 con 0x7f3ff4004850 2023-06-10T18:47:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc00e820 con 0x7f3ff4004850 2023-06-10T18:47:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f400959a700 1 -- 172.21.15.104:0/247599244 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00eb90 con 0x7f3ff4004850 2023-06-10T18:47:53.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 -- 172.21.15.104:0/247599244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 msgr2=0x7f3ff4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 --2- 172.21.15.104:0/247599244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f3ff4004cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009b30 tx=0x7f3ff4005880 comp rx=0 tx=0).stop 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f400ad9d700 1 -- 172.21.15.104:0/247599244 reap_dead start 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 -- 172.21.15.104:0/247599244 shutdown_connections 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 -- 172.21.15.104:0/247599244 >> 172.21.15.104:0/247599244 conn(0x7f400c06b8a0 msgr2=0x7f400c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 -- 172.21.15.104:0/247599244 shutdown_connections 2023-06-10T18:47:53.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.961+0000 7f4010870700 1 -- 172.21.15.104:0/247599244 wait complete. 2023-06-10T18:47:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.963+0000 7f4010870700 1 Processor -- start 2023-06-10T18:47:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.963+0000 7f4010870700 1 -- start start 2023-06-10T18:47:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.963+0000 7f4010870700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c09c4b0 0x7f400c157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400c0b3470 0x7f400c0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0b3e50 con 0x7f3ff4004850 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c09cc20 con 0x7f400c0b3470 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c09cf00 con 0x7f400c09c4b0 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58346/0 (socket says 172.21.15.104:58346) 2023-06-10T18:47:53.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 -- 172.21.15.104:0/1175501269 learned_addr learned my addr 172.21.15.104:0/1175501269 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4009d9b700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c09c4b0 0x7f400c157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400ad9d700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400c0b3470 0x7f400c0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 -- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c09c4b0 msgr2=0x7f400c157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c09c4b0 0x7f400c157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 -- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400c0b3470 msgr2=0x7f400c0b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400c0b3470 0x7f400c0b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:53.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 -- 172.21.15.104:0/1175501269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c09d1e0 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f400a59c700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f400000f7a0 tx=0x7f400000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40000072e0 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4000007440 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c1594e0 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4000005830 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.964+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c158050 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.965+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c159a00 con 0x7f3ff4004850 2023-06-10T18:47:53.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.965+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4000005990 con 0x7f3ff4004850 2023-06-10T18:47:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.966+0000 7f3ffaffd700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 0x7f3fec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.966+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f4000088f70 con 0x7f3ff4004850 2023-06-10T18:47:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.966+0000 7f4009d9b700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 0x7f3fec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.966+0000 7f4009d9b700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 0x7f3fec06d190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0090b0 tx=0x7f3ffc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:53.970+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40000bb310 con 0x7f3ff4004850 2023-06-10T18:47:54.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:54.171+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f400c066f70 con 0x7f3ff4004850 2023-06-10T18:47:55.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.168+0000 7f3ffaffd700 1 -- 172.21.15.104:0/1175501269 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7f400c159a00 con 0x7f3ff4004850 2023-06-10T18:47:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 msgr2=0x7f3fec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 0x7f3fec06d190 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0090b0 tx=0x7f3ffc008030 comp rx=0 tx=0).stop 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 msgr2=0x7f400c09a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f400000f7a0 tx=0x7f400000fab0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 shutdown_connections 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3fec06ace0 0x7f3fec06d190 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400c0b3470 0x7f400c0b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ff4004850 0x7f400c09a090 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 --2- 172.21.15.104:0/1175501269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f400c09c4b0 0x7f400c157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.170+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 >> 172.21.15.104:0/1175501269 conn(0x7f400c06b8a0 msgr2=0x7f400c0b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:55.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.171+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 shutdown_connections 2023-06-10T18:47:55.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.171+0000 7f4010870700 1 -- 172.21.15.104:0/1175501269 wait complete. 2023-06-10T18:47:55.177 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:47:55.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:47:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.310+0000 7fea18628700 1 Processor -- start 2023-06-10T18:47:55.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.310+0000 7fea18628700 1 -- start start 2023-06-10T18:47:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea18628700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fea100cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea18628700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100cdae0 con 0x7fea100cd680 2023-06-10T18:47:55.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea18628700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100c9e60 con 0x7fea100b1080 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea18628700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea100c9bc0 con 0x7fea100aeaa0 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea16bc5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fea100cd680 0x7fea1006dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56190/0 (socket says 172.21.15.104:56190) 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.311+0000 7fea16bc5700 1 -- 172.21.15.104:0/872392886 learned_addr learned my addr 172.21.15.104:0/872392886 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea163c4700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fea100cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181128495 0 0) 0x7fea100cdae0 con 0x7fea100cd680 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc0035b0 con 0x7fea100cd680 2023-06-10T18:47:55.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116874383 0 0) 0x7fe9fc0035b0 con 0x7fea100cd680 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 msgr2=0x7fea100cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fea100cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 >> v1:172.21.15.114:6789/0 conn(0x7fea100b1080 legacy=0x7fea100ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100aee70 con 0x7fea100cd680 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fea080019f0 con 0x7fea100cd680 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 >> v1:172.21.15.104:6789/0 conn(0x7fea100cd680 legacy=0x7fea1006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fe9fc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9fc003f70 0x7fe9fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc004d90 0x7fe9fc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100c9bc0 con 0x7fe9fc003f70 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc00bac0 con 0x7fe9fc004d90 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc00bc80 con 0x7fea100aeaa0 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea15bc3700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9fc003f70 0x7fe9fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea163c4700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fe9fc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea16bc5700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc004d90 0x7fe9fc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea15bc3700 1 -- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 msgr2=0x7fe9fc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.312+0000 7fea15bc3700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea100aeaa0 0x7fe9fc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea15bc3700 1 -- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc004d90 msgr2=0x7fe9fc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea15bc3700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc004d90 0x7fe9fc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea15bc3700 1 -- 172.21.15.104:0/872392886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0035b0 con 0x7fe9fc003f70 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea15bc3700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9fc003f70 0x7fe9fc004850 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007fd0 tx=0x7fe9fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f850ceff23b043 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00c660 con 0x7fe9fc003f70 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c00c7c0 con 0x7fe9fc003f70 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea153c2700 1 -- 172.21.15.104:0/872392886 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00cb70 con 0x7fe9fc003f70 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 -- 172.21.15.104:0/872392886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9fc003f70 msgr2=0x7fe9fc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 --2- 172.21.15.104:0/872392886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9fc003f70 0x7fe9fc004850 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007fd0 tx=0x7fe9fc005880 comp rx=0 tx=0).stop 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea16bc5700 1 -- 172.21.15.104:0/872392886 reap_dead start 2023-06-10T18:47:55.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 -- 172.21.15.104:0/872392886 shutdown_connections 2023-06-10T18:47:55.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 -- 172.21.15.104:0/872392886 >> 172.21.15.104:0/872392886 conn(0x7fea1006b8a0 msgr2=0x7fea100cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:55.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 -- 172.21.15.104:0/872392886 shutdown_connections 2023-06-10T18:47:55.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.313+0000 7fea18628700 1 -- 172.21.15.104:0/872392886 wait complete. 2023-06-10T18:47:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 Processor -- start 2023-06-10T18:47:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 -- start start 2023-06-10T18:47:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc003f70 0x7fea100c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea100adf00 0x7fea100ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100b3670 con 0x7fea10156170 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100b3470 con 0x7fea100adf00 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea18628700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea100ac770 con 0x7fe9fc003f70 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea16bc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea16bc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58374/0 (socket says 172.21.15.104:58374) 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.315+0000 7fea16bc5700 1 -- 172.21.15.104:0/371628409 learned_addr learned my addr 172.21.15.104:0/371628409 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:55.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea15bc3700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea100adf00 0x7fea100ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea163c4700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc003f70 0x7fea100c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 -- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc003f70 msgr2=0x7fea100c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc003f70 0x7fea100c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 -- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea100adf00 msgr2=0x7fea100ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea100adf00 0x7fea100ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 -- 172.21.15.104:0/371628409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100aca10 con 0x7fea10156170 2023-06-10T18:47:55.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea16bc5700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fea1006ff90 tx=0x7fea0800a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08015410 con 0x7fea10156170 2023-06-10T18:47:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08015590 con 0x7fea10156170 2023-06-10T18:47:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08015940 con 0x7fea10156170 2023-06-10T18:47:55.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea100accf0 con 0x7fea10156170 2023-06-10T18:47:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.316+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea100ad2b0 con 0x7fea10156170 2023-06-10T18:47:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.317+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10066f70 con 0x7fea10156170 2023-06-10T18:47:55.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.317+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fea08015aa0 con 0x7fea10156170 2023-06-10T18:47:55.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.318+0000 7fea06ffd700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 0x7fe9f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.318+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fea0808bfa0 con 0x7fea10156170 2023-06-10T18:47:55.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.318+0000 7fea163c4700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 0x7fe9f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.318+0000 7fea163c4700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 0x7fe9f406d190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea00009a80 tx=0x7fea00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.322+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fea10066f70 con 0x7fea10156170 2023-06-10T18:47:55.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.583+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fea10066f70 con 0x7fea10156170 2023-06-10T18:47:55.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.583+0000 7fea06ffd700 1 -- 172.21.15.104:0/371628409 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fea10066f70 con 0x7fea10156170 2023-06-10T18:47:55.588 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:47:55.589 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:47:55.591 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.592 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:47:55.592 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:47:55.593 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.593 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.593 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:47:55.593 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:47:55.594 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.594 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.594 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:47:55.595 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.597 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.598 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.598 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:47:55.599 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:47:55.599 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:47:55.600 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.601 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:47:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.604 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:47:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:47:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.606 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.606 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.606 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:47:55.608 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:47:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:47:55.612 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:47:55.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:55.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:47:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.619 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:55.619 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:47:55.619 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:47:55.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 msgr2=0x7fe9f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 0x7fe9f406d190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea00009a80 tx=0x7fea00008040 comp rx=0 tx=0).stop 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 msgr2=0x7fea1006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fea1006ff90 tx=0x7fea0800a810 comp rx=0 tx=0).stop 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 shutdown_connections 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea10156170 0x7fea1006d680 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc003f70 0x7fea100c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9f406ace0 0x7fe9f406d190 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 --2- 172.21.15.104:0/371628409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea100adf00 0x7fea100ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 >> 172.21.15.104:0/371628409 conn(0x7fea1006b8a0 msgr2=0x7fea100af300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:55.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 shutdown_connections 2023-06-10T18:47:55.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.585+0000 7fea18628700 1 -- 172.21.15.104:0/371628409 wait complete. 2023-06-10T18:47:55.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.736+0000 7f73c3334700 1 Processor -- start 2023-06-10T18:47:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.736+0000 7f73c3334700 1 -- start start 2023-06-10T18:47:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c3334700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73bc0cb300 0x7f73bc163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c3334700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0c9e60 con 0x7f73bc0cd850 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c3334700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0aee70 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c3334700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73bc0c9bc0 con 0x7f73bc0cb300 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c18d1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73bc0cb300 0x7f73bc163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c08cf700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f73bc0cd850 0x7f73bc06cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56208/0 (socket says 172.21.15.104:56208) 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.737+0000 7f73c08cf700 1 -- 172.21.15.104:0/137982516 learned_addr learned my addr 172.21.15.104:0/137982516 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c18d1700 1 -- 172.21.15.104:0/137982516 >> v1:172.21.15.114:6789/0 conn(0x7f73bc0aeaa0 legacy=0x7f73bc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c18d1700 1 -- 172.21.15.104:0/137982516 >> v1:172.21.15.104:6789/0 conn(0x7f73bc0cd850 legacy=0x7f73bc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c18d1700 1 -- 172.21.15.104:0/137982516 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc06e540 con 0x7f73bc0cb300 2023-06-10T18:47:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c18d1700 1 --2- 172.21.15.104:0/137982516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0cb300 0x7f73bc163c30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009d90 tx=0x7f73b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eaf5433f6ee09e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73b3fff700 1 -- 172.21.15.104:0/137982516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b400bdd0 con 0x7f73bc0cb300 2023-06-10T18:47:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73b3fff700 1 -- 172.21.15.104:0/137982516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b4011460 con 0x7f73bc0cb300 2023-06-10T18:47:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73b3fff700 1 -- 172.21.15.104:0/137982516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4011810 con 0x7f73bc0cb300 2023-06-10T18:47:55.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c3334700 1 -- 172.21.15.104:0/137982516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0cb300 msgr2=0x7f73bc163c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.738+0000 7f73c3334700 1 --2- 172.21.15.104:0/137982516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0cb300 0x7f73bc163c30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009d90 tx=0x7f73b400b330 comp rx=0 tx=0).stop 2023-06-10T18:47:55.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.739+0000 7f73c3334700 1 -- 172.21.15.104:0/137982516 shutdown_connections 2023-06-10T18:47:55.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.739+0000 7f73c3334700 1 --2- 172.21.15.104:0/137982516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0cb300 0x7f73bc163c30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.739+0000 7f73c3334700 1 -- 172.21.15.104:0/137982516 >> 172.21.15.104:0/137982516 conn(0x7f73bc06b8a0 msgr2=0x7f73bc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:55.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.739+0000 7f73c3334700 1 -- 172.21.15.104:0/137982516 shutdown_connections 2023-06-10T18:47:55.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.739+0000 7f73c3334700 1 -- 172.21.15.104:0/137982516 wait complete. 2023-06-10T18:47:55.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.740+0000 7f73c3334700 1 Processor -- start 2023-06-10T18:47:55.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.740+0000 7f73c3334700 1 -- start start 2023-06-10T18:47:55.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73bc0cb300 0x7f73bc0c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73bc0cd850 0x7f73bc0b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0c5f50 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0b4be0 con 0x7f73bc0cd850 2023-06-10T18:47:55.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c3334700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0b42b0 con 0x7f73bc0cb300 2023-06-10T18:47:55.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58384/0 (socket says 172.21.15.104:58384) 2023-06-10T18:47:55.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 -- 172.21.15.104:0/3659002920 learned_addr learned my addr 172.21.15.104:0/3659002920 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:55.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 -- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73bc0cb300 msgr2=0x7f73bc0c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73bc0cb300 0x7f73bc0c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 -- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73bc0cd850 msgr2=0x7f73bc0b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73bc0cd850 0x7f73bc0b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.741+0000 7f73c10d0700 1 -- 172.21.15.104:0/3659002920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b4009a70 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73c10d0700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c4d0 tx=0x7f73ac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0052d0 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac005430 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc0b4590 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac003910 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.742+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc1465b0 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.743+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc074050 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.743+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f73ac003a70 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.743+0000 7f73b1ffb700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 0x7f73a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.743+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f73ac088800 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.744+0000 7f73c08cf700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 0x7f73a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:55.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.744+0000 7f73c08cf700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 0x7f73a006d190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f73bc149d70 tx=0x7f73b8007ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:55.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.748+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f73ac053ac0 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.937+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73bc0b1030 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.938+0000 7f73b1ffb700 1 -- 172.21.15.104:0/3659002920 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+14892 (secure 0 0 0) 0x7f73ac0570e0 con 0x7f73bc0aeaa0 2023-06-10T18:47:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:55.943 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":52,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:47:55.164770+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:55.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 msgr2=0x7f73a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 0x7f73a006d190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f73bc149d70 tx=0x7f73b8007ad0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 msgr2=0x7f73bc0b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c4d0 tx=0x7f73ac00c890 comp rx=0 tx=0).stop 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 shutdown_connections 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73a006ace0 0x7f73a006d190 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73bc0cd850 0x7f73bc0b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73bc0aeaa0 0x7f73bc0b5070 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 --2- 172.21.15.104:0/3659002920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73bc0cb300 0x7f73bc0c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:55.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 >> 172.21.15.104:0/3659002920 conn(0x7f73bc06b8a0 msgr2=0x7f73bc157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:55.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 shutdown_connections 2023-06-10T18:47:55.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:55.940+0000 7f73c3334700 1 -- 172.21.15.104:0/3659002920 wait complete. 2023-06-10T18:47:55.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:47:55.958 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:47:56.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.076+0000 7f1a62de8700 1 Processor -- start 2023-06-10T18:47:56.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.076+0000 7f1a62de8700 1 -- start start 2023-06-10T18:47:56.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a62de8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a5c069e70 0x7f1a5c06a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a62de8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06dcd0 0x7f1a5c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a62de8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c06e690 con 0x7f1a5c15c500 2023-06-10T18:47:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a62de8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0aef20 con 0x7f1a5c069e70 2023-06-10T18:47:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a62de8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15c8d0 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a5c069e70 0x7f1a5c06a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1a5c069e70 0x7f1a5c06a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58394/0 (socket says 172.21.15.104:58394) 2023-06-10T18:47:56.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 -- 172.21.15.104:0/1639308351 learned_addr learned my addr 172.21.15.104:0/1639308351 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:56.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a61385700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06dcd0 0x7f1a5c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 -- 172.21.15.104:0/1639308351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06dcd0 msgr2=0x7f1a5c06e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:56.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06dcd0 0x7f1a5c06e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 -- 172.21.15.104:0/1639308351 >> v1:172.21.15.114:6789/0 conn(0x7f1a5c15c500 legacy=0x7f1a5c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 -- 172.21.15.104:0/1639308351 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c06abd0 con 0x7f1a5c069e70 2023-06-10T18:47:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5bfff700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c069e70 0x7f1a5c06a310 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009920 tx=0x7f1a48009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878f7a9f9b3b6733 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5b7fe700 1 -- 172.21.15.104:0/1639308351 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48011830 con 0x7f1a5c069e70 2023-06-10T18:47:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5b7fe700 1 -- 172.21.15.104:0/1639308351 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a48011990 con 0x7f1a5c069e70 2023-06-10T18:47:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.077+0000 7f1a5b7fe700 1 -- 172.21.15.104:0/1639308351 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48011d20 con 0x7f1a5c069e70 2023-06-10T18:47:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 -- 172.21.15.104:0/1639308351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c069e70 msgr2=0x7f1a5c06a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c069e70 0x7f1a5c06a310 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009920 tx=0x7f1a48009c30 comp rx=0 tx=0).stop 2023-06-10T18:47:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 -- 172.21.15.104:0/1639308351 shutdown_connections 2023-06-10T18:47:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c06dcd0 0x7f1a5c06e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 --2- 172.21.15.104:0/1639308351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c069e70 0x7f1a5c06a310 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 -- 172.21.15.104:0/1639308351 >> 172.21.15.104:0/1639308351 conn(0x7f1a5c06b8a0 msgr2=0x7f1a5c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 -- 172.21.15.104:0/1639308351 shutdown_connections 2023-06-10T18:47:56.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.078+0000 7f1a62de8700 1 -- 172.21.15.104:0/1639308351 wait complete. 2023-06-10T18:47:56.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.079+0000 7f1a62de8700 1 Processor -- start 2023-06-10T18:47:56.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.079+0000 7f1a62de8700 1 -- start start 2023-06-10T18:47:56.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 0x7f1a5c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 0x7f1a5c0c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c3680 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c4e90 con 0x7f1a5c069e70 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a62de8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c38b0 con 0x7f1a5c15c500 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58400/0 (socket says 172.21.15.104:58400) 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 -- 172.21.15.104:0/2530402319 learned_addr learned my addr 172.21.15.104:0/2530402319 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:56.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a60b84700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 0x7f1a5c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 -- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 msgr2=0x7f1a5c0c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a61385700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 0x7f1a5c0c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 0x7f1a5c0c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 -- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 msgr2=0x7f1a5c0b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 0x7f1a5c0b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a5bfff700 1 -- 172.21.15.104:0/2530402319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48009550 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a60b84700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 0x7f1a5c0b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.080+0000 7f1a61385700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 0x7f1a5c0c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a5bfff700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009130 tx=0x7f1a48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a480036a0 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4801e070 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003a60 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0c3b90 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.081+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0c4150 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.082+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c066f10 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.082+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a48003bc0 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.083+0000 7f1a597fa700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.083+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f1a4808a4c0 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.083+0000 7f1a60b84700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.083+0000 7f1a60b84700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000fb0 tx=0x7f1a50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.087+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1a480558f0 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.273+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a5c074050 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.273+0000 7f1a597fa700 1 -- 172.21.15.104:0/2530402319 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a48058f10 con 0x7f1a5c06dcd0 2023-06-10T18:47:56.279 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:47:56.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 msgr2=0x7f1a4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000fb0 tx=0x7f1a50009040 comp rx=0 tx=0).stop 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 msgr2=0x7f1a5c0c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009130 tx=0x7f1a48004750 comp rx=0 tx=0).stop 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 shutdown_connections 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a4006ace0 0x7f1a4006d190 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a5c069e70 0x7f1a5c0b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a5c15c500 0x7f1a5c0c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 --2- 172.21.15.104:0/2530402319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a5c06dcd0 0x7f1a5c0c3140 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 >> 172.21.15.104:0/2530402319 conn(0x7f1a5c06b8a0 msgr2=0x7f1a5c0ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 shutdown_connections 2023-06-10T18:47:56.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.276+0000 7f1a62de8700 1 -- 172.21.15.104:0/2530402319 wait complete. 2023-06-10T18:47:56.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:47:56.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.410+0000 7fe091067700 1 Processor -- start 2023-06-10T18:47:56.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.410+0000 7fe091067700 1 -- start start 2023-06-10T18:47:56.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.410+0000 7fe091067700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe08c070a10 0x7fe08c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.410+0000 7fe091067700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe08c06a5e0 0x7fe08c06a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe091067700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c0c4770 con 0x7fe08c145cc0 2023-06-10T18:47:56.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe091067700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08c0713d0 con 0x7fe08c06a5e0 2023-06-10T18:47:56.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe091067700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08c069e70 con 0x7fe08c070a10 2023-06-10T18:47:56.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe08c06a5e0 0x7fe08c06a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe08c145cc0 0x7fe08c06c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56234/0 (socket says 172.21.15.104:56234) 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08ad9d700 1 -- 172.21.15.104:0/3038192577 learned_addr learned my addr 172.21.15.104:0/3038192577 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08a59c700 1 --2- 172.21.15.104:0/3038192577 >> v2:172.21.15.114:3300/0 conn(0x7fe08c070a10 0x7fe08c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 -- 172.21.15.104:0/3038192577 >> v2:172.21.15.114:3300/0 conn(0x7fe08c070a10 msgr2=0x7fe08c070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 --2- 172.21.15.104:0/3038192577 >> v2:172.21.15.114:3300/0 conn(0x7fe08c070a10 0x7fe08c070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 -- 172.21.15.104:0/3038192577 >> v1:172.21.15.104:6789/0 conn(0x7fe08c145cc0 legacy=0x7fe08c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 -- 172.21.15.104:0/3038192577 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c06cf10 con 0x7fe08c06a5e0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe089d9b700 1 -- 172.21.15.104:0/3038192577 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347090808 0 0) 0x7fe08c0c4770 con 0x7fe08c145cc0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe08b59e700 1 --2- 172.21.15.104:0/3038192577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c06a9e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0cfe30 tx=0x7fe08400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a303f43a252201d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe089d9b700 1 -- 172.21.15.104:0/3038192577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08400e070 con 0x7fe08c06a5e0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe089d9b700 1 -- 172.21.15.104:0/3038192577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08400bbd0 con 0x7fe08c06a5e0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.411+0000 7fe089d9b700 1 -- 172.21.15.104:0/3038192577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084012430 con 0x7fe08c06a5e0 2023-06-10T18:47:56.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 -- 172.21.15.104:0/3038192577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 msgr2=0x7fe08c06a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 --2- 172.21.15.104:0/3038192577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c06a9e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0cfe30 tx=0x7fe08400b330 comp rx=0 tx=0).stop 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 -- 172.21.15.104:0/3038192577 shutdown_connections 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 --2- 172.21.15.104:0/3038192577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c06a9e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 --2- 172.21.15.104:0/3038192577 >> v2:172.21.15.114:3300/0 conn(0x7fe08c070a10 0x7fe08c070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 -- 172.21.15.104:0/3038192577 >> 172.21.15.104:0/3038192577 conn(0x7fe08c0ceec0 msgr2=0x7fe08c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 -- 172.21.15.104:0/3038192577 shutdown_connections 2023-06-10T18:47:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.412+0000 7fe091067700 1 -- 172.21.15.104:0/3038192577 wait complete. 2023-06-10T18:47:56.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.414+0000 7fe091067700 1 Processor -- start 2023-06-10T18:47:56.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.414+0000 7fe091067700 1 -- start start 2023-06-10T18:47:56.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe08c070a10 0x7fe08c0ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 0x7fe08c0adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1444a0 con 0x7fe08c06a5e0 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1438b0 con 0x7fe08c070a10 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c143b00 con 0x7fe08c145cc0 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58412/0 (socket says 172.21.15.104:58412) 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 -- 172.21.15.104:0/1452936661 learned_addr learned my addr 172.21.15.104:0/1452936661 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:56.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08a59c700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe08c070a10 0x7fe08c0ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08b59e700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 0x7fe08c0adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 -- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 msgr2=0x7fe08c0adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 0x7fe08c0adc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 -- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe08c070a10 msgr2=0x7fe08c0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe08c070a10 0x7fe08c0ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 -- 172.21.15.104:0/1452936661 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084009a70 con 0x7fe08c06a5e0 2023-06-10T18:47:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08b59e700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 0x7fe08c0adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe08ad9d700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c550 tx=0x7fe07c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c004c80 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c004e00 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c143de0 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c007310 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.415+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c0c7670 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.416+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c074050 con 0x7fe08c06a5e0 2023-06-10T18:47:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.418+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe07c007470 con 0x7fe08c06a5e0 2023-06-10T18:47:56.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.418+0000 7fe0737fe700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 0x7fe07806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.418+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fe07c089500 con 0x7fe08c06a5e0 2023-06-10T18:47:56.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.418+0000 7fe08a59c700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 0x7fe07806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.419+0000 7fe08a59c700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 0x7fe07806d190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe08c1f55c0 tx=0x7fe080008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.422+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe07c054930 con 0x7fe08c06a5e0 2023-06-10T18:47:56.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.608+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe08c141d50 con 0x7fe08c06a5e0 2023-06-10T18:47:56.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.608+0000 7fe0737fe700 1 -- 172.21.15.104:0/1452936661 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v52) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe07c057f50 con 0x7fe08c06a5e0 2023-06-10T18:47:56.614 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:47:56.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 msgr2=0x7fe07806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 0x7fe07806d190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe08c1f55c0 tx=0x7fe080008040 comp rx=0 tx=0).stop 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 msgr2=0x7fe08c138ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c550 tx=0x7fe07c00c910 comp rx=0 tx=0).stop 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 shutdown_connections 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe08c145cc0 0x7fe08c0adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe08c06a5e0 0x7fe08c138ba0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe07806ace0 0x7fe07806d190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 --2- 172.21.15.104:0/1452936661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe08c070a10 0x7fe08c0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 >> 172.21.15.104:0/1452936661 conn(0x7fe08c0ceec0 msgr2=0x7fe08c0cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 shutdown_connections 2023-06-10T18:47:56.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.610+0000 7fe091067700 1 -- 172.21.15.104:0/1452936661 wait complete. 2023-06-10T18:47:56.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:47:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.741+0000 7f4f49a2a700 1 Processor -- start 2023-06-10T18:47:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.741+0000 7f4f49a2a700 1 -- start start 2023-06-10T18:47:56.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f49a2a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4f4406b1f0 0x7f4f440cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f49a2a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f440cadd0 con 0x7f4f4408a3d0 2023-06-10T18:47:56.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f49a2a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f440cb0b0 con 0x7f4f440ce350 2023-06-10T18:47:56.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f49a2a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f440cfeb0 con 0x7f4f4406b1f0 2023-06-10T18:47:56.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f437fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4f4408a3d0 0x7f4f440cb540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56250/0 (socket says 172.21.15.104:56250) 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f437fe700 1 -- 172.21.15.104:0/2520213259 learned_addr learned my addr 172.21.15.104:0/2520213259 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/2520213259 >> v2:172.21.15.114:3300/0 conn(0x7f4f4406b1f0 0x7f4f440cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161510221 0 0) 0x7f4f440cadd0 con 0x7f4f4408a3d0 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.742+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c0035b0 con 0x7f4f4408a3d0 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472637998 0 0) 0x7f4f2c0035b0 con 0x7f4f4408a3d0 2023-06-10T18:47:56.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 >> v2:172.21.15.114:3300/0 conn(0x7f4f4406b1f0 msgr2=0x7f4f440cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 --2- 172.21.15.104:0/2520213259 >> v2:172.21.15.114:3300/0 conn(0x7f4f4406b1f0 0x7f4f440cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 >> v1:172.21.15.114:6789/0 conn(0x7f4f440ce350 legacy=0x7f4f440cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440cf550 con 0x7f4f4408a3d0 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4f3c0019f0 con 0x7f4f4408a3d0 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 >> v1:172.21.15.104:6789/0 conn(0x7f4f4408a3d0 legacy=0x7f4f440cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c003a00 0x7f4f2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 0x7f4f2c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f2c005200 0x7f4f2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440cfeb0 con 0x7f4f2c004850 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c00c110 con 0x7f4f2c005200 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c00c2d0 con 0x7f4f2c003a00 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 0x7f4f2c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f437fe700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f2c005200 0x7f4f2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c003a00 0x7f4f2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 -- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c003a00 msgr2=0x7f4f2c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c003a00 0x7f4f2c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 -- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f2c005200 msgr2=0x7f4f2c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f2c005200 0x7f4f2c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 -- 172.21.15.104:0/2520213259 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0035b0 con 0x7f4f2c004850 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f437fe700 1 -- 172.21.15.104:0/2520213259 reap_dead start 2023-06-10T18:47:56.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f427fc700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 0x7f4f2c004cc0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4f44067a90 tx=0x7f4f2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644729ede7085641 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340052d0 con 0x7f4f2c004850 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f34005450 con 0x7f4f2c004850 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.743+0000 7f4f41ffb700 1 -- 172.21.15.104:0/2520213259 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340038c0 con 0x7f4f2c004850 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 -- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 msgr2=0x7f4f2c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 0x7f4f2c004cc0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4f44067a90 tx=0x7f4f2c005880 comp rx=0 tx=0).stop 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 -- 172.21.15.104:0/2520213259 shutdown_connections 2023-06-10T18:47:56.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/2520213259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f2c004850 0x7f4f2c004cc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 -- 172.21.15.104:0/2520213259 >> 172.21.15.104:0/2520213259 conn(0x7f4f44089fd0 msgr2=0x7f4f440d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 -- 172.21.15.104:0/2520213259 shutdown_connections 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.744+0000 7f4f49a2a700 1 -- 172.21.15.104:0/2520213259 wait complete. 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.745+0000 7f4f49a2a700 1 Processor -- start 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.745+0000 7f4f49a2a700 1 -- start start 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c004850 0x7f4f441565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44156b00 0x7f4f440c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c5090 con 0x7f4f440c9dc0 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c41e0 con 0x7f4f44156b00 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f49a2a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440c4480 con 0x7f4f2c004850 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58434/0 (socket says 172.21.15.104:58434) 2023-06-10T18:47:56.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 -- 172.21.15.104:0/596254139 learned_addr learned my addr 172.21.15.104:0/596254139 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c004850 0x7f4f441565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f427fc700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44156b00 0x7f4f440c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 -- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c004850 msgr2=0x7f4f441565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c004850 0x7f4f441565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 -- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44156b00 msgr2=0x7f4f440c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44156b00 0x7f4f440c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.746+0000 7f4f437fe700 1 -- 172.21.15.104:0/596254139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440c4760 con 0x7f4f440c9dc0 2023-06-10T18:47:56.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f437fe700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4f440d06b0 tx=0x7f4f3c00a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00ad80 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c0133f0 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f440c4a40 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0137a0 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.747+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f440c8e60 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.748+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44074050 con 0x7f4f440c9dc0 2023-06-10T18:47:56.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.748+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4f3c013900 con 0x7f4f440c9dc0 2023-06-10T18:47:56.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.748+0000 7f4f2bfff700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:56.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.749+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4f3c089bd0 con 0x7f4f440c9dc0 2023-06-10T18:47:56.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.749+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:56.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.749+0000 7f4f42ffd700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009ba0 tx=0x7f4f38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:56.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.753+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4f3c054e90 con 0x7f4f440c9dc0 2023-06-10T18:47:56.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.945+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4f440ce8a0 con 0x7f4f440c9dc0 2023-06-10T18:47:56.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.945+0000 7f4f2bfff700 1 -- 172.21.15.104:0/596254139 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+126 (secure 0 0 0) 0x7f4f3c0584b0 con 0x7f4f440c9dc0 2023-06-10T18:47:56.951 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:56.951 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:47:56.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:56.952 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:47:56.952 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:47:56.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:47:56.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:47:56.953 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:47:56.953 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:47:56.953 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:47:56.953 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:47:56.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 msgr2=0x7f4f3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009ba0 tx=0x7f4f38008040 comp rx=0 tx=0).stop 2023-06-10T18:47:56.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 msgr2=0x7f4f440c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:56.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4f440d06b0 tx=0x7f4f3c00a5f0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 shutdown_connections 2023-06-10T18:47:56.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f440c9dc0 0x7f4f440c8920 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4f2c004850 0x7f4f441565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4f3006ace0 0x7f4f3006d190 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 --2- 172.21.15.104:0/596254139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f44156b00 0x7f4f440c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:56.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 >> 172.21.15.104:0/596254139 conn(0x7f4f44089fd0 msgr2=0x7f4f440cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:56.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 shutdown_connections 2023-06-10T18:47:56.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:56.947+0000 7f4f49a2a700 1 -- 172.21.15.104:0/596254139 wait complete. 2023-06-10T18:47:56.964 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:47:56.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:47:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.084+0000 7fcde5818700 1 Processor -- start 2023-06-10T18:47:57.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.084+0000 7fcde5818700 1 -- start start 2023-06-10T18:47:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcde5818700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d04f0 0x7fcde00d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcde5818700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcde00b05d0 0x7fcde00b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcde5818700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde00af530 con 0x7fcde00d25c0 2023-06-10T18:47:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcde5818700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde00af810 con 0x7fcde00b05d0 2023-06-10T18:47:57.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcde5818700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00afaf0 con 0x7fcde00d04f0 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcde00b05d0 0x7fcde00b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcde00b05d0 0x7fcde00b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58440/0 (socket says 172.21.15.104:58440) 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 -- 172.21.15.104:0/2278466914 learned_addr learned my addr 172.21.15.104:0/2278466914 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddeffd700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d04f0 0x7fcde00d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 -- 172.21.15.104:0/2278466914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d04f0 msgr2=0x7fcde00d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d04f0 0x7fcde00d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 -- 172.21.15.104:0/2278466914 >> v1:172.21.15.114:6789/0 conn(0x7fcde00d25c0 legacy=0x7fcde00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.085+0000 7fcddf7fe700 1 -- 172.21.15.104:0/2278466914 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00afdd0 con 0x7fcde00b05d0 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcddf7fe700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00b05d0 0x7fcde00b0a50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009e50 tx=0x7fcdd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e114116a54b9ffa6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcdddffb700 1 -- 172.21.15.104:0/2278466914 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd800e070 con 0x7fcde00b05d0 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcdddffb700 1 -- 172.21.15.104:0/2278466914 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd800b9c0 con 0x7fcde00b05d0 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcdddffb700 1 -- 172.21.15.104:0/2278466914 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd800bd70 con 0x7fcde00b05d0 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 -- 172.21.15.104:0/2278466914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00b05d0 msgr2=0x7fcde00b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00b05d0 0x7fcde00b0a50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009e50 tx=0x7fcdd800b330 comp rx=0 tx=0).stop 2023-06-10T18:47:57.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 -- 172.21.15.104:0/2278466914 shutdown_connections 2023-06-10T18:47:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00b05d0 0x7fcde00b0a50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 --2- 172.21.15.104:0/2278466914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d04f0 0x7fcde00d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 -- 172.21.15.104:0/2278466914 >> 172.21.15.104:0/2278466914 conn(0x7fcde0071ec0 msgr2=0x7fcde009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 -- 172.21.15.104:0/2278466914 shutdown_connections 2023-06-10T18:47:57.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.086+0000 7fcde5818700 1 -- 172.21.15.104:0/2278466914 wait complete. 2023-06-10T18:47:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.088+0000 7fcde5818700 1 Processor -- start 2023-06-10T18:47:57.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.088+0000 7fcde5818700 1 -- start start 2023-06-10T18:47:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.088+0000 7fcde5818700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde00b05d0 0x7fcde0147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.088+0000 7fcde5818700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.088+0000 7fcde5818700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d25c0 0x7fcde0148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcde5818700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde0147000 con 0x7fcde00d04f0 2023-06-10T18:47:57.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcde5818700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01463f0 con 0x7fcde00b05d0 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcde5818700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01466d0 con 0x7fcde00d25c0 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58444/0 (socket says 172.21.15.104:58444) 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 -- 172.21.15.104:0/2999770733 learned_addr learned my addr 172.21.15.104:0/2999770733 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcddeffd700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde00b05d0 0x7fcde0147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcddf7fe700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d25c0 0x7fcde0148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 -- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d25c0 msgr2=0x7fcde0148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d25c0 0x7fcde0148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 -- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde00b05d0 msgr2=0x7fcde0147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde00b05d0 0x7fcde0147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 -- 172.21.15.104:0/2999770733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8009b30 con 0x7fcde00d04f0 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdde7fc700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000c450 tx=0x7fcdd000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00052d0 con 0x7fcde00d04f0 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0005430 con 0x7fcde00d04f0 2023-06-10T18:47:57.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00038c0 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde01469b0 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.089+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0146030 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.090+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde0067010 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.091+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcdd0003a20 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.091+0000 7fcdc7fff700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 0x7fcdcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.091+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fcdd0088590 con 0x7fcde00d04f0 2023-06-10T18:47:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.091+0000 7fcddeffd700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 0x7fcdcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.091+0000 7fcddeffd700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 0x7fcdcc06d190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcde015b8c0 tx=0x7fcdd4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.095+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcdd0053850 con 0x7fcde00d04f0 2023-06-10T18:47:57.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.281+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcde0074050 con 0x7fcde00d04f0 2023-06-10T18:47:57.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.842+0000 7fcdc7fff700 1 -- 172.21.15.104:0/2999770733 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcdd000ce60 con 0x7fcde00d04f0 2023-06-10T18:47:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.845+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 msgr2=0x7fcdcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.845+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 0x7fcdcc06d190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcde015b8c0 tx=0x7fcdd4008040 comp rx=0 tx=0).stop 2023-06-10T18:47:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 msgr2=0x7fcde0145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000c450 tx=0x7fcdd000c760 comp rx=0 tx=0).stop 2023-06-10T18:47:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 shutdown_connections 2023-06-10T18:47:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdcc06ace0 0x7fcdcc06d190 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcde00b05d0 0x7fcde0147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcde00d25c0 0x7fcde0148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 --2- 172.21.15.104:0/2999770733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcde00d04f0 0x7fcde0145af0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.846+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 >> 172.21.15.104:0/2999770733 conn(0x7fcde0071ec0 msgr2=0x7fcde00c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.847+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 shutdown_connections 2023-06-10T18:47:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.847+0000 7fcde5818700 1 -- 172.21.15.104:0/2999770733 wait complete. 2023-06-10T18:47:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:47:57.867 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:47:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.992+0000 7f81f53a6700 1 Processor -- start 2023-06-10T18:47:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.992+0000 7f81f53a6700 1 -- start start 2023-06-10T18:47:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81f53a6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f81f00d04f0 0x7f81f00d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81f53a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81f00d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81f53a6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f00d0e00 con 0x7f81f006b510 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81f53a6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f00af530 con 0x7f81f00d04f0 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81f53a6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00af810 con 0x7f81f00d25c0 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81ef7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f81f006b510 0x7f81f006b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56282/0 (socket says 172.21.15.104:56282) 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81ef7fe700 1 -- 172.21.15.104:0/2165159516 learned_addr learned my addr 172.21.15.104:0/2165159516 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81eeffd700 1 --2- 172.21.15.104:0/2165159516 >> v2:172.21.15.114:3300/0 conn(0x7f81f00d04f0 0x7f81f00d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81f00d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131539171 0 0) 0x7f81f00d0e00 con 0x7f81f006b510 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d80035b0 con 0x7f81f006b510 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.993+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446363327 0 0) 0x7f81d80035b0 con 0x7f81f006b510 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 msgr2=0x7f81f00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81f00d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 >> v2:172.21.15.114:3300/0 conn(0x7f81f00d04f0 msgr2=0x7f81f00d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 --2- 172.21.15.104:0/2165159516 >> v2:172.21.15.114:3300/0 conn(0x7f81f00d04f0 0x7f81f00d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f00afaf0 con 0x7f81f006b510 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f81e80019f0 con 0x7f81f006b510 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 >> v1:172.21.15.104:6789/0 conn(0x7f81f006b510 legacy=0x7f81f006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8003a00 0x7f81d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81d8004430 0x7f81d80048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81d8005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00af530 con 0x7f81d8004430 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00af810 con 0x7f81d8003a00 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d800ba10 con 0x7f81f00d25c0 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81d8004430 0x7f81d80048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81eeffd700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8003a00 0x7f81d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ef7fe700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81d8005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 -- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 msgr2=0x7f81d8005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f00d25c0 0x7f81d8005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 -- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8003a00 msgr2=0x7f81d8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8003a00 0x7f81d8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 -- 172.21.15.104:0/2165159516 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d80035b0 con 0x7f81d8004430 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81d8004430 0x7f81d80048f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009b90 tx=0x7f81d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da83cd949c501cb4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000e6c0 con 0x7f81d8004430 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e000e820 con 0x7f81d8004430 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.994+0000 7f81edffb700 1 -- 172.21.15.104:0/2165159516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000eb90 con 0x7f81d8004430 2023-06-10T18:47:58.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 -- 172.21.15.104:0/2165159516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81d8004430 msgr2=0x7f81d80048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 --2- 172.21.15.104:0/2165159516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81d8004430 0x7f81d80048f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009b90 tx=0x7f81d8005880 comp rx=0 tx=0).stop 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81ef7fe700 1 -- 172.21.15.104:0/2165159516 reap_dead start 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 -- 172.21.15.104:0/2165159516 shutdown_connections 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 -- 172.21.15.104:0/2165159516 >> 172.21.15.104:0/2165159516 conn(0x7f81f0071ec0 msgr2=0x7f81f009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 -- 172.21.15.104:0/2165159516 shutdown_connections 2023-06-10T18:47:58.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.995+0000 7f81f53a6700 1 -- 172.21.15.104:0/2165159516 wait complete. 2023-06-10T18:47:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.996+0000 7f81f53a6700 1 Processor -- start 2023-06-10T18:47:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.996+0000 7f81f53a6700 1 -- start start 2023-06-10T18:47:58.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8004430 0x7f81f00b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f0146f50 0x7f81f01473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01e27a0 con 0x7f81f01e1df0 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01e4980 con 0x7f81d8004430 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81f53a6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00b7810 con 0x7f81f0146f50 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58464/0 (socket says 172.21.15.104:58464) 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 -- 172.21.15.104:0/814247757 learned_addr learned my addr 172.21.15.104:0/814247757 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 -- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f0146f50 msgr2=0x7f81f01473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f0146f50 0x7f81f01473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 -- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8004430 msgr2=0x7f81f00b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8004430 0x7f81f00b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.997+0000 7f81ee7fc700 1 -- 172.21.15.104:0/814247757 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f00b7af0 con 0x7f81f01e1df0 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81ee7fc700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005770 tx=0x7f81e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:58.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0021070 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e0004550 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f00b7dd0 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000e6c0 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.998+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f00b8390 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.999+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f00b8550 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.999+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f81e0025070 con 0x7f81f01e1df0 2023-06-10T18:47:58.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:57.999+0000 7f81d77fe700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 0x7f81dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:58.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.000+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f81e008a560 con 0x7f81f01e1df0 2023-06-10T18:47:58.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.000+0000 7f81eeffd700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 0x7f81dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:58.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.000+0000 7f81eeffd700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 0x7f81dc06d190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009bb0 tx=0x7f81e4009670 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:58.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.004+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f81e00557c0 con 0x7f81f01e1df0 2023-06-10T18:47:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.191+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f81f00b8700 con 0x7f81f01e1df0 2023-06-10T18:47:58.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.846+0000 7f81d77fe700 1 -- 172.21.15.104:0/814247757 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) v1 ==== 170+0+0 (secure 0 0 0) 0x7f81e0058de0 con 0x7f81f01e1df0 2023-06-10T18:47:58.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.852+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 msgr2=0x7f81dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.852+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 0x7f81dc06d190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009bb0 tx=0x7f81e4009670 comp rx=0 tx=0).stop 2023-06-10T18:47:58.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.852+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 msgr2=0x7f81f01e2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.852+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005770 tx=0x7f81e00047c0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 shutdown_connections 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81dc06ace0 0x7f81dc06d190 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81d8004430 0x7f81f00b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81f0146f50 0x7f81f01473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 --2- 172.21.15.104:0/814247757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81f01e1df0 0x7f81f01e2260 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:58.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 >> 172.21.15.104:0/814247757 conn(0x7f81f0071ec0 msgr2=0x7f81f00d2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:58.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 shutdown_connections 2023-06-10T18:47:58.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.853+0000 7f81f53a6700 1 -- 172.21.15.104:0/814247757 wait complete. 2023-06-10T18:47:58.862 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:47:58.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:47:59.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.997+0000 7f723c2b8700 1 Processor -- start 2023-06-10T18:47:59.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.997+0000 7f723c2b8700 1 -- start start 2023-06-10T18:47:59.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723c2b8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f723415c580 0x7f72340b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723c2b8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723406e250 con 0x7f723406cde0 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723c2b8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723415c950 con 0x7f723406e6e0 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723c2b8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72340b1e80 con 0x7f723415c580 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723a054700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f723415c580 0x7f72340b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723a855700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f723406cde0 0x7f72340ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56302/0 (socket says 172.21.15.104:56302) 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.998+0000 7f723a855700 1 -- 172.21.15.104:0/3045282566 learned_addr learned my addr 172.21.15.104:0/3045282566 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:59.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723a054700 1 -- 172.21.15.104:0/3045282566 >> v1:172.21.15.114:6789/0 conn(0x7f723406e6e0 legacy=0x7f72340b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723a054700 1 -- 172.21.15.104:0/3045282566 >> v1:172.21.15.104:6789/0 conn(0x7f723406cde0 legacy=0x7f72340ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723a054700 1 -- 172.21.15.104:0/3045282566 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723406d920 con 0x7f723415c580 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f7239052700 1 -- 172.21.15.104:0/3045282566 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198238650 0 0) 0x7f723406e250 con 0x7f723406cde0 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723a054700 1 --2- 172.21.15.104:0/3045282566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723415c580 0x7f72340b1070 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7224009820 tx=0x7f7224009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c06a68d816cad server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f7239052700 1 -- 172.21.15.104:0/3045282566 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224005e30 con 0x7f723415c580 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f7239052700 1 -- 172.21.15.104:0/3045282566 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224004030 con 0x7f723415c580 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f7239052700 1 -- 172.21.15.104:0/3045282566 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240043c0 con 0x7f723415c580 2023-06-10T18:47:59.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 -- 172.21.15.104:0/3045282566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723415c580 msgr2=0x7f72340b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 --2- 172.21.15.104:0/3045282566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723415c580 0x7f72340b1070 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7224009820 tx=0x7f7224009b30 comp rx=0 tx=0).stop 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 -- 172.21.15.104:0/3045282566 shutdown_connections 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 --2- 172.21.15.104:0/3045282566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723415c580 0x7f72340b1070 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 -- 172.21.15.104:0/3045282566 >> 172.21.15.104:0/3045282566 conn(0x7f723406b8a0 msgr2=0x7f72340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:58.999+0000 7f723c2b8700 1 -- 172.21.15.104:0/3045282566 shutdown_connections 2023-06-10T18:47:59.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.000+0000 7f723c2b8700 1 -- 172.21.15.104:0/3045282566 wait complete. 2023-06-10T18:47:59.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 Processor -- start 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 -- start start 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f723406cde0 0x7f72340b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f723415c580 0x7f72340b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72341e44b0 con 0x7f723406e6e0 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72340b5280 con 0x7f723415c580 2023-06-10T18:47:59.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f723c2b8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72340c97a0 con 0x7f723406cde0 2023-06-10T18:47:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.001+0000 7f7239853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58478/0 (socket says 172.21.15.104:58478) 2023-06-10T18:47:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 -- 172.21.15.104:0/1072860311 learned_addr learned my addr 172.21.15.104:0/1072860311 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f723a054700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f723406cde0 0x7f72340b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 -- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f723406cde0 msgr2=0x7f72340b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f723406cde0 0x7f72340b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 -- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f723415c580 msgr2=0x7f72340b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f723a855700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f723415c580 0x7f72340b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f723415c580 0x7f72340b4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 -- 172.21.15.104:0/1072860311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7224009430 con 0x7f723406e6e0 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f7239853700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f723000c4d0 tx=0x7f723000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72300054e0 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230003680 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7230003a10 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72341e4700 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.002+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72341e4cc0 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.003+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7234074050 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.003+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7230003b70 con 0x7f723406e6e0 2023-06-10T18:47:59.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.004+0000 7f722affd700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 0x7f721806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.004+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f72300886c0 con 0x7f723406e6e0 2023-06-10T18:47:59.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.004+0000 7f723a054700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 0x7f721806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.005+0000 7f723a054700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 0x7f721806d190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f72340d0380 tx=0x7f7224008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.008+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7230053720 con 0x7f723406e6e0 2023-06-10T18:47:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.236+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7234159a90 con 0x7f723406e6e0 2023-06-10T18:47:59.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.241+0000 7f722affd700 1 -- 172.21.15.104:0/1072860311 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7230056d40 con 0x7f723406e6e0 2023-06-10T18:47:59.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.245+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 msgr2=0x7f721806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.245+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 0x7f721806d190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f72340d0380 tx=0x7f7224008070 comp rx=0 tx=0).stop 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.245+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 msgr2=0x7f72340b4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.245+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f723000c4d0 tx=0x7f723000c890 comp rx=0 tx=0).stop 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 shutdown_connections 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f721806ace0 0x7f721806d190 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f723415c580 0x7f72340b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f723406cde0 0x7f72340b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 --2- 172.21.15.104:0/1072860311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f723406e6e0 0x7f72340b4800 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.247+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 >> 172.21.15.104:0/1072860311 conn(0x7f723406b8a0 msgr2=0x7f723415b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.249+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 shutdown_connections 2023-06-10T18:47:59.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.250+0000 7f723c2b8700 1 -- 172.21.15.104:0/1072860311 wait complete. 2023-06-10T18:47:59.262 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 8 and data pool 9 2023-06-10T18:47:59.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:59.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.392+0000 7f6b6cf7b700 1 Processor -- start 2023-06-10T18:47:59.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.392+0000 7f6b6cf7b700 1 -- start start 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b6cf7b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b68164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b6cf7b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6815a680 con 0x7f6b680ac230 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b6cf7b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b680ac680 con 0x7f6b6815a2b0 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b6cf7b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680d0510 con 0x7f6b68069e70 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b65d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6b680ac230 0x7f6b680cb860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56312/0 (socket says 172.21.15.104:56312) 2023-06-10T18:47:59.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b65d9b700 1 -- 172.21.15.104:0/1938096891 learned_addr learned my addr 172.21.15.104:0/1938096891 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:47:59.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.393+0000 7f6b66d9d700 1 --2- 172.21.15.104:0/1938096891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b68164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375831235 0 0) 0x7f6b6815a680 con 0x7f6b680ac230 2023-06-10T18:47:59.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b500035b0 con 0x7f6b680ac230 2023-06-10T18:47:59.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857212890 0 0) 0x7f6b5c001160 con 0x7f6b6815a2b0 2023-06-10T18:47:59.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6815a680 con 0x7f6b6815a2b0 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b66d9d700 1 -- 172.21.15.104:0/1938096891 >> v1:172.21.15.114:6789/0 conn(0x7f6b6815a2b0 legacy=0x7f6b680cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b66d9d700 1 -- 172.21.15.104:0/1938096891 >> v1:172.21.15.104:6789/0 conn(0x7f6b680ac230 legacy=0x7f6b680cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b66d9d700 1 -- 172.21.15.104:0/1938096891 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6806a5e0 con 0x7f6b68069e70 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b66d9d700 1 --2- 172.21.15.104:0/1938096891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b68164d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009dc0 tx=0x7f6b6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280020eba0e2b2e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000e070 con 0x7f6b68069e70 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6000bb20 con 0x7f6b68069e70 2023-06-10T18:47:59.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.394+0000 7f6b6559a700 1 -- 172.21.15.104:0/1938096891 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60012430 con 0x7f6b68069e70 2023-06-10T18:47:59.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/1938096891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 msgr2=0x7f6b68164d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/1938096891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b68164d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009dc0 tx=0x7f6b6000b330 comp rx=0 tx=0).stop 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/1938096891 shutdown_connections 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/1938096891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b68164d50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/1938096891 >> 172.21.15.104:0/1938096891 conn(0x7f6b6806b8a0 msgr2=0x7f6b680cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/1938096891 shutdown_connections 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.395+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/1938096891 wait complete. 2023-06-10T18:47:59.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.396+0000 7f6b6cf7b700 1 Processor -- start 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.396+0000 7f6b6cf7b700 1 -- start start 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b6815b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b680ac230 0x7f6b680c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6815cfa0 con 0x7f6b6815a2b0 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681f2d90 con 0x7f6b680ac230 2023-06-10T18:47:59.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6cf7b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681efce0 con 0x7f6b68069e70 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58492/0 (socket says 172.21.15.104:58492) 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 -- 172.21.15.104:0/3081998452 learned_addr learned my addr 172.21.15.104:0/3081998452 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b6659c700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b6815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b65d9b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b680ac230 0x7f6b680c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 -- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 msgr2=0x7f6b6815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b6815b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 -- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b680ac230 msgr2=0x7f6b680c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b680ac230 0x7f6b680c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.397+0000 7f6b66d9d700 1 -- 172.21.15.104:0/3081998452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b60009a70 con 0x7f6b6815a2b0 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b66d9d700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000b370 tx=0x7f6b60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6001b810 con 0x7f6b6815a2b0 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60020070 con 0x7f6b6815a2b0 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b681effc0 con 0x7f6b6815a2b0 2023-06-10T18:47:59.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6001bbb0 con 0x7f6b6815a2b0 2023-06-10T18:47:59.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.398+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b681f0580 con 0x7f6b6815a2b0 2023-06-10T18:47:59.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.399+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b68074050 con 0x7f6b6815a2b0 2023-06-10T18:47:59.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.399+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b6001bd10 con 0x7f6b6815a2b0 2023-06-10T18:47:59.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.400+0000 7f6b56ffd700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.400+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6b60022070 con 0x7f6b6815a2b0 2023-06-10T18:47:59.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.400+0000 7f6b6659c700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.400+0000 7f6b6659c700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007fd0 tx=0x7f6b5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.404+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6b60056880 con 0x7f6b6815a2b0 2023-06-10T18:47:59.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.592+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b681f2ed0 con 0x7f6b6815a2b0 2023-06-10T18:47:59.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.593+0000 7f6b56ffd700 1 -- 172.21.15.104:0/3081998452 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18613 (secure 0 0 0) 0x7f6b60059ea0 con 0x7f6b6815a2b0 2023-06-10T18:47:59.598 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:59.599 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":55,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:47:59.237478+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:59.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 msgr2=0x7f6b4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007fd0 tx=0x7f6b5c009040 comp rx=0 tx=0).stop 2023-06-10T18:47:59.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 msgr2=0x7f6b681f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000b370 tx=0x7f6b60004ad0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 shutdown_connections 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6815a2b0 0x7f6b681f3220 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b68069e70 0x7f6b6815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 --2- 172.21.15.104:0/3081998452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b680ac230 0x7f6b680c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 >> 172.21.15.104:0/3081998452 conn(0x7f6b6806b8a0 msgr2=0x7f6b681576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 shutdown_connections 2023-06-10T18:47:59.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.595+0000 7f6b6cf7b700 1 -- 172.21.15.104:0/3081998452 wait complete. 2023-06-10T18:47:59.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:47:59.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.727+0000 7fc6f5995700 1 Processor -- start 2023-06-10T18:47:59.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.727+0000 7fc6f5995700 1 -- start start 2023-06-10T18:47:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6f5995700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6f006a8c0 0x7fc6f01587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6f5995700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f0069e70 con 0x7fc6f0070a20 2023-06-10T18:47:59.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6f5995700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f01595f0 con 0x7fc6f0158ce0 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6f5995700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f01598d0 con 0x7fc6f006a8c0 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6ee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6f006a8c0 0x7fc6f01587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6ee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc6f006a8c0 0x7fc6f01587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58498/0 (socket says 172.21.15.104:58498) 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/647650417 learned_addr learned my addr 172.21.15.104:0/647650417 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.728+0000 7fc6eeffd700 1 --1- 172.21.15.104:0/647650417 >> v1:172.21.15.104:6789/0 conn(0x7fc6f0070a20 0x7fc6f0070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56324/0 (socket says 172.21.15.104:56324) 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/647650417 >> v1:172.21.15.114:6789/0 conn(0x7fc6f0158ce0 legacy=0x7fc6f0159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/647650417 >> v1:172.21.15.104:6789/0 conn(0x7fc6f0070a20 legacy=0x7fc6f0070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/647650417 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f006a0c0 con 0x7fc6f006a8c0 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6edffb700 1 -- 172.21.15.104:0/647650417 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852800799 0 0) 0x7fc6f0069e70 con 0x7fc6f0070a20 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/647650417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01587a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400bb90 tx=0x7fc6e400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d29570cb5804754 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6edffb700 1 -- 172.21.15.104:0/647650417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e400dc40 con 0x7fc6f006a8c0 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6edffb700 1 -- 172.21.15.104:0/647650417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e400dda0 con 0x7fc6f006a8c0 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6edffb700 1 -- 172.21.15.104:0/647650417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4017630 con 0x7fc6f006a8c0 2023-06-10T18:47:59.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 -- 172.21.15.104:0/647650417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f006a8c0 msgr2=0x7fc6f01587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 --2- 172.21.15.104:0/647650417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01587a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400bb90 tx=0x7fc6e400bea0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 -- 172.21.15.104:0/647650417 shutdown_connections 2023-06-10T18:47:59.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 --2- 172.21.15.104:0/647650417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01587a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 -- 172.21.15.104:0/647650417 >> 172.21.15.104:0/647650417 conn(0x7fc6f00b1610 msgr2=0x7fc6f00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 -- 172.21.15.104:0/647650417 shutdown_connections 2023-06-10T18:47:59.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.729+0000 7fc6f5995700 1 -- 172.21.15.104:0/647650417 wait complete. 2023-06-10T18:47:59.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 Processor -- start 2023-06-10T18:47:59.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 -- start start 2023-06-10T18:47:59.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6f0158ce0 0x7fc6f006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0154be0 con 0x7fc6f0070a20 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.731+0000 7fc6f5995700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f01e2910 con 0x7fc6f006a8c0 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6f5995700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00b1c80 con 0x7fc6f0158ce0 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58504/0 (socket says 172.21.15.104:58504) 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/2687419671 learned_addr learned my addr 172.21.15.104:0/2687419671 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6eeffd700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6f0158ce0 msgr2=0x7fc6f006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6f0158ce0 0x7fc6f006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f006a8c0 msgr2=0x7fc6f01546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 -- 172.21.15.104:0/2687419671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e400b7f0 con 0x7fc6f0070a20 2023-06-10T18:47:59.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc6f006c890 tx=0x7fc6e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4022670 con 0x7fc6f0070a20 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e40227d0 con 0x7fc6f0070a20 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f00b1f60 con 0x7fc6f0070a20 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4022b80 con 0x7fc6f0070a20 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.732+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f00b2520 con 0x7fc6f0070a20 2023-06-10T18:47:59.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.733+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f0071780 con 0x7fc6f0070a20 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.734+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc6e4015070 con 0x7fc6f0070a20 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.734+0000 7fc6d7fff700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 0x7fc6dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.734+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc6e408d5a0 con 0x7fc6f0070a20 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.734+0000 7fc6eeffd700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 0x7fc6dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.735+0000 7fc6eeffd700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 0x7fc6dc06d190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0007b10 tx=0x7fc6e0007a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:47:59.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.738+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc6e40585a0 con 0x7fc6f0070a20 2023-06-10T18:47:59.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.928+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6f00cd6a0 con 0x7fc6f0070a20 2023-06-10T18:47:59.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.930+0000 7fc6d7fff700 1 -- 172.21.15.104:0/2687419671 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18613 (secure 0 0 0) 0x7fc6e405bbc0 con 0x7fc6f0070a20 2023-06-10T18:47:59.934 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:47:59.935 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":55,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:47:59.237478+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 msgr2=0x7fc6dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 0x7fc6dc06d190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0007b10 tx=0x7fc6e0007a60 comp rx=0 tx=0).stop 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 msgr2=0x7fc6f00c3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc6f006c890 tx=0x7fc6e4004750 comp rx=0 tx=0).stop 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 shutdown_connections 2023-06-10T18:47:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc6dc06ace0 0x7fc6dc06d190 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f006a8c0 0x7fc6f01546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6f0158ce0 0x7fc6f006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 --2- 172.21.15.104:0/2687419671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f0070a20 0x7fc6f00c3160 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 >> 172.21.15.104:0/2687419671 conn(0x7fc6f00b1610 msgr2=0x7fc6f0070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 shutdown_connections 2023-06-10T18:47:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:47:59.932+0000 7fc6f5995700 1 -- 172.21.15.104:0/2687419671 wait complete. 2023-06-10T18:47:59.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:48:00.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.068+0000 7f80df132700 1 Processor -- start 2023-06-10T18:48:00.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.069+0000 7f80df132700 1 -- start start 2023-06-10T18:48:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80df132700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80d80b1780 0x7f80d806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80df132700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80df132700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d80ae7e0 con 0x7f80d815c6f0 2023-06-10T18:48:00.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80df132700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d806df90 con 0x7f80d80b1780 2023-06-10T18:48:00.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80df132700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d806e270 con 0x7f80d806ea70 2023-06-10T18:48:00.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80d80b1780 0x7f80d806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80dd6cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80dd6cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:49666/0 (socket says 172.21.15.104:49666) 2023-06-10T18:48:00.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80dcece700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f80d815c6f0 0x7f80d80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:33952/0 (socket says 172.21.15.104:33952) 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80d80b1780 0x7f80d806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:46014/0 (socket says 172.21.15.104:46014) 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80dd6cf700 1 -- 172.21.15.104:0/2904609217 learned_addr learned my addr 172.21.15.104:0/2904609217 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 -- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 msgr2=0x7f80d806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 -- 172.21.15.104:0/2904609217 >> v1:172.21.15.114:6789/0 conn(0x7f80d815c6f0 legacy=0x7f80d80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d7fff700 1 -- 172.21.15.104:0/2904609217 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d806e550 con 0x7f80d80b1780 2023-06-10T18:48:00.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80d77fe700 1 -- 172.21.15.104:0/2904609217 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700378885 0 0) 0x7f80d80ae7e0 con 0x7f80d815c6f0 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.070+0000 7f80dd6cf700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80d7fff700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d80b1780 0x7f80d806cde0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009840 tx=0x7f80c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9779c123970965ae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80d77fe700 1 -- 172.21.15.104:0/2904609217 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011830 con 0x7f80d80b1780 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80d77fe700 1 -- 172.21.15.104:0/2904609217 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c4011990 con 0x7f80d80b1780 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80d77fe700 1 -- 172.21.15.104:0/2904609217 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4011d20 con 0x7f80d80b1780 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 -- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d80b1780 msgr2=0x7f80d806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:00.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d80b1780 0x7f80d806cde0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009840 tx=0x7f80c4009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 -- 172.21.15.104:0/2904609217 shutdown_connections 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d806ea70 0x7f80d806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 --2- 172.21.15.104:0/2904609217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d80b1780 0x7f80d806cde0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 -- 172.21.15.104:0/2904609217 >> 172.21.15.104:0/2904609217 conn(0x7f80d806b8a0 msgr2=0x7f80d80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 -- 172.21.15.104:0/2904609217 shutdown_connections 2023-06-10T18:48:00.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.071+0000 7f80df132700 1 -- 172.21.15.104:0/2904609217 wait complete. 2023-06-10T18:48:00.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 Processor -- start 2023-06-10T18:48:00.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 -- start start 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d806ea70 0x7f80d81533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d815c6f0 0x7f80d80c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80b32f0 con 0x7f80d80b1780 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80b52a0 con 0x7f80d806ea70 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.073+0000 7f80df132700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80b2530 con 0x7f80d815c6f0 2023-06-10T18:48:00.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58518/0 (socket says 172.21.15.104:58518) 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 -- 172.21.15.104:0/3759680239 learned_addr learned my addr 172.21.15.104:0/3759680239 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80dcece700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d806ea70 0x7f80d81533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80dd6cf700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d815c6f0 0x7f80d80c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 -- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d815c6f0 msgr2=0x7f80d80c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d815c6f0 0x7f80d80c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 -- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d806ea70 msgr2=0x7f80d81533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:00.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d806ea70 0x7f80d81533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 -- 172.21.15.104:0/3759680239 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d0009b30 con 0x7f80d80b1780 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d7fff700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f80c40090b0 tx=0x7f80c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c40036a0 con 0x7f80d80b1780 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c401f070 con 0x7f80d80b1780 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c40094a0 con 0x7f80d80b1780 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4003a20 con 0x7f80d80b1780 2023-06-10T18:48:00.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.074+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d80b2af0 con 0x7f80d80b1780 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.075+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d8066f10 con 0x7f80d80b1780 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.076+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f80c4003b80 con 0x7f80d80b1780 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.076+0000 7f80d57fa700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 0x7f80bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.076+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f80c4021070 con 0x7f80d80b1780 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.076+0000 7f80dcece700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 0x7f80bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:00.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.077+0000 7f80dcece700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 0x7f80bc06d190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f80cc000fb0 tx=0x7f80cc007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:00.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.081+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f80c4055d10 con 0x7f80d80b1780 2023-06-10T18:48:00.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:00.313+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f80d8074050 con 0x7f80d80b1780 2023-06-10T18:48:01.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.254+0000 7f80d57fa700 1 -- 172.21.15.104:0/3759680239 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f80c40150b0 con 0x7f80d80b1780 2023-06-10T18:48:01.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 msgr2=0x7f80bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 0x7f80bc06d190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f80cc000fb0 tx=0x7f80cc007ec0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 msgr2=0x7f80d80c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f80c40090b0 tx=0x7f80c4004750 comp rx=0 tx=0).stop 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 shutdown_connections 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80bc06ace0 0x7f80bc06d190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80d806ea70 0x7f80d81533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d815c6f0 0x7f80d80c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 --2- 172.21.15.104:0/3759680239 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80d80b1780 0x7f80d80c88e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 >> 172.21.15.104:0/3759680239 conn(0x7f80d806b8a0 msgr2=0x7f80d80afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:01.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.257+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 shutdown_connections 2023-06-10T18:48:01.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.258+0000 7f80df132700 1 -- 172.21.15.104:0/3759680239 wait complete. 2023-06-10T18:48:01.277 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:48:01.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:48:01.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.401+0000 7fcf8fc05700 1 Processor -- start 2023-06-10T18:48:01.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.402+0000 7fcf8fc05700 1 -- start start 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8fc05700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf880b6030 0x7fcf880cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8fc05700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcf880cad90 0x7fcf880cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8fc05700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880ce310 con 0x7fcf880ced60 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8fc05700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf880cf870 con 0x7fcf880cad90 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8fc05700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf880ce8d0 con 0x7fcf880b6030 2023-06-10T18:48:01.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcf880cad90 0x7fcf880cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8e1a2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcf880ced60 0x7fcf880cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56356/0 (socket says 172.21.15.104:56356) 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8e1a2700 1 -- 172.21.15.104:0/2021738408 learned_addr learned my addr 172.21.15.104:0/2021738408 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d9a1700 1 --2- 172.21.15.104:0/2021738408 >> v2:172.21.15.114:3300/0 conn(0x7fcf880b6030 0x7fcf880cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 -- 172.21.15.104:0/2021738408 >> v2:172.21.15.114:3300/0 conn(0x7fcf880b6030 msgr2=0x7fcf880cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 --2- 172.21.15.104:0/2021738408 >> v2:172.21.15.114:3300/0 conn(0x7fcf880b6030 0x7fcf880cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 -- 172.21.15.104:0/2021738408 >> v1:172.21.15.104:6789/0 conn(0x7fcf880ced60 legacy=0x7fcf880cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 -- 172.21.15.104:0/2021738408 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8815e600 con 0x7fcf880cad90 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8c99f700 1 -- 172.21.15.104:0/2021738408 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920707643 0 0) 0x7fcf880ce310 con 0x7fcf880ced60 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8d1a0700 1 --2- 172.21.15.104:0/2021738408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880cad90 0x7fcf880cff40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcf880d17b0 tx=0x7fcf84009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bf7a89250fdb56 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8c99f700 1 -- 172.21.15.104:0/2021738408 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011830 con 0x7fcf880cad90 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8c99f700 1 -- 172.21.15.104:0/2021738408 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf84011990 con 0x7fcf880cad90 2023-06-10T18:48:01.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.403+0000 7fcf8c99f700 1 -- 172.21.15.104:0/2021738408 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011d20 con 0x7fcf880cad90 2023-06-10T18:48:01.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 -- 172.21.15.104:0/2021738408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880cad90 msgr2=0x7fcf880cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/2021738408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880cad90 0x7fcf880cff40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcf880d17b0 tx=0x7fcf84009b20 comp rx=0 tx=0).stop 2023-06-10T18:48:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 -- 172.21.15.104:0/2021738408 shutdown_connections 2023-06-10T18:48:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/2021738408 >> v2:172.21.15.114:3300/0 conn(0x7fcf880b6030 0x7fcf880cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/2021738408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880cad90 0x7fcf880cff40 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 -- 172.21.15.104:0/2021738408 >> 172.21.15.104:0/2021738408 conn(0x7fcf8806e370 msgr2=0x7fcf8806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 -- 172.21.15.104:0/2021738408 shutdown_connections 2023-06-10T18:48:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.404+0000 7fcf8fc05700 1 -- 172.21.15.104:0/2021738408 wait complete. 2023-06-10T18:48:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.405+0000 7fcf8fc05700 1 Processor -- start 2023-06-10T18:48:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.405+0000 7fcf8fc05700 1 -- start start 2023-06-10T18:48:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf880cad90 0x7fcf88157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf880ced60 0x7fcf880c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881ef9e0 con 0x7fcf880b6030 2023-06-10T18:48:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881f3f10 con 0x7fcf880ced60 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8fc05700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880c8e10 con 0x7fcf880cad90 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58530/0 (socket says 172.21.15.104:58530) 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 -- 172.21.15.104:0/3992719719 learned_addr learned my addr 172.21.15.104:0/3992719719 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 -- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf880cad90 msgr2=0x7fcf88157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf880cad90 0x7fcf88157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 -- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf880ced60 msgr2=0x7fcf880c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf880ced60 0x7fcf880c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 -- 172.21.15.104:0/3992719719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840094a0 con 0x7fcf880b6030 2023-06-10T18:48:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.406+0000 7fcf8d9a1700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800d800 tx=0x7fcf7800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf780125f0 con 0x7fcf880b6030 2023-06-10T18:48:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78012750 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf880c57b0 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78012b00 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881f3950 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.407+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88074050 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.408+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcf78012c60 con 0x7fcf880b6030 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.408+0000 7fcf7e7fc700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 0x7fcf6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.409+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcf78089a50 con 0x7fcf880b6030 2023-06-10T18:48:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.409+0000 7fcf8d1a0700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 0x7fcf6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.409+0000 7fcf8d1a0700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 0x7fcf6c06d190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf880b2ee0 tx=0x7fcf84010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.413+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcf78054920 con 0x7fcf880b6030 2023-06-10T18:48:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:01.639+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcf880703f0 con 0x7fcf880b6030 2023-06-10T18:48:02.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.263+0000 7fcf7e7fc700 1 -- 172.21.15.104:0/3992719719 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf78057f40 con 0x7fcf880b6030 2023-06-10T18:48:02.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.265+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 msgr2=0x7fcf6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 0x7fcf6c06d190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf880b2ee0 tx=0x7fcf84010000 comp rx=0 tx=0).stop 2023-06-10T18:48:02.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 msgr2=0x7fcf880c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800d800 tx=0x7fcf7800dbc0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 shutdown_connections 2023-06-10T18:48:02.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcf6c06ace0 0x7fcf6c06d190 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf880ced60 0x7fcf880c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf880b6030 0x7fcf880c88d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 --2- 172.21.15.104:0/3992719719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf880cad90 0x7fcf88157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 >> 172.21.15.104:0/3992719719 conn(0x7fcf8806e370 msgr2=0x7fcf880cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 shutdown_connections 2023-06-10T18:48:02.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.266+0000 7fcf8fc05700 1 -- 172.21.15.104:0/3992719719 wait complete. 2023-06-10T18:48:02.282 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:48:02.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:02.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.399+0000 7fa840e70700 1 Processor -- start 2023-06-10T18:48:02.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.399+0000 7fa840e70700 1 -- start start 2023-06-10T18:48:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa840e70700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa83c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa840e70700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa83c069e70 0x7fa83c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa840e70700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83c06d290 con 0x7fa83c06e4b0 2023-06-10T18:48:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa840e70700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c071640 con 0x7fa83c069e70 2023-06-10T18:48:02.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa840e70700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0aef00 con 0x7fa83c06e0e0 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa839d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa83c06e4b0 0x7fa83c0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56368/0 (socket says 172.21.15.104:56368) 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa839d9b700 1 -- 172.21.15.104:0/3776766834 learned_addr learned my addr 172.21.15.104:0/3776766834 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa83ad9d700 1 --2- 172.21.15.104:0/3776766834 >> v2:172.21.15.114:3300/0 conn(0x7fa83c069e70 0x7fa83c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa83a59c700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa83c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787845357 0 0) 0x7fa83c06d290 con 0x7fa83c06e4b0 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.400+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8240035b0 con 0x7fa83c06e4b0 2023-06-10T18:48:02.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3518900606 0 0) 0x7fa8240035b0 con 0x7fa83c06e4b0 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 msgr2=0x7fa83c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa83c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 >> v2:172.21.15.114:3300/0 conn(0x7fa83c069e70 msgr2=0x7fa83c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 --2- 172.21.15.104:0/3776766834 >> v2:172.21.15.114:3300/0 conn(0x7fa83c069e70 0x7fa83c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0ca2d0 con 0x7fa83c06e4b0 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa82c001670 con 0x7fa83c06e4b0 2023-06-10T18:48:02.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 >> v1:172.21.15.104:6789/0 conn(0x7fa83c06e4b0 legacy=0x7fa83c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa824003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa824003f80 0x7fa824004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa824004d90 0x7fa824005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c071640 con 0x7fa824004d90 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0aef00 con 0x7fa824003f80 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82400baf0 con 0x7fa83c06e0e0 2023-06-10T18:48:02.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa824004d90 0x7fa824005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83a59c700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa824003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa839d9b700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa824003f80 0x7fa824004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 -- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 msgr2=0x7fa824003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa83c06e0e0 0x7fa824003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 -- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa824003f80 msgr2=0x7fa824004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa824003f80 0x7fa824004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 -- 172.21.15.104:0/3776766834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240035b0 con 0x7fa824004d90 2023-06-10T18:48:02.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83ad9d700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa824004d90 0x7fa824005200 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa834009f90 tx=0x7fa824005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b29c7aedb7a5a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83400ec30 con 0x7fa824004d90 2023-06-10T18:48:02.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83400f040 con 0x7fa824004d90 2023-06-10T18:48:02.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.401+0000 7fa83959a700 1 -- 172.21.15.104:0/3776766834 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834014480 con 0x7fa824004d90 2023-06-10T18:48:02.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 -- 172.21.15.104:0/3776766834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa824004d90 msgr2=0x7fa824005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 --2- 172.21.15.104:0/3776766834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa824004d90 0x7fa824005200 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa834009f90 tx=0x7fa824005880 comp rx=0 tx=0).stop 2023-06-10T18:48:02.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa83ad9d700 1 -- 172.21.15.104:0/3776766834 reap_dead start 2023-06-10T18:48:02.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 -- 172.21.15.104:0/3776766834 shutdown_connections 2023-06-10T18:48:02.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 -- 172.21.15.104:0/3776766834 >> 172.21.15.104:0/3776766834 conn(0x7fa83c06b8a0 msgr2=0x7fa83c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:02.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 -- 172.21.15.104:0/3776766834 shutdown_connections 2023-06-10T18:48:02.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.402+0000 7fa840e70700 1 -- 172.21.15.104:0/3776766834 wait complete. 2023-06-10T18:48:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.403+0000 7fa840e70700 1 Processor -- start 2023-06-10T18:48:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.403+0000 7fa840e70700 1 -- start start 2023-06-10T18:48:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa824004d90 0x7fa83c0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa83c0b3380 0x7fa83c0c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b4af0 con 0x7fa83c158fb0 2023-06-10T18:48:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b3f00 con 0x7fa83c0b3380 2023-06-10T18:48:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa840e70700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b5450 con 0x7fa824004d90 2023-06-10T18:48:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58552/0 (socket says 172.21.15.104:58552) 2023-06-10T18:48:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 -- 172.21.15.104:0/270595269 learned_addr learned my addr 172.21.15.104:0/270595269 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa83a59c700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa824004d90 0x7fa83c0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 -- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa824004d90 msgr2=0x7fa83c0b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa824004d90 0x7fa83c0b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 -- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa83c0b3380 msgr2=0x7fa83c0c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa83c0b3380 0x7fa83c0c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 -- 172.21.15.104:0/270595269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0b4150 con 0x7fa83c158fb0 2023-06-10T18:48:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.404+0000 7fa839d9b700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0010e0 tx=0x7fa82c0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.405+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0077c0 con 0x7fa83c158fb0 2023-06-10T18:48:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.405+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c00f8a0 con 0x7fa83c158fb0 2023-06-10T18:48:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.405+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00fc30 con 0x7fa83c158fb0 2023-06-10T18:48:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.405+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c0b4430 con 0x7fa83c158fb0 2023-06-10T18:48:02.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.405+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c156080 con 0x7fa83c158fb0 2023-06-10T18:48:02.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.406+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83c074050 con 0x7fa83c158fb0 2023-06-10T18:48:02.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.406+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa82c00fd90 con 0x7fa83c158fb0 2023-06-10T18:48:02.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.406+0000 7fa82affd700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 0x7fa81c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.406+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa82c089540 con 0x7fa83c158fb0 2023-06-10T18:48:02.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.407+0000 7fa83a59c700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 0x7fa81c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.407+0000 7fa83a59c700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 0x7fa81c06d190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa830009a80 tx=0x7fa830008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.411+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa82c054540 con 0x7fa83c158fb0 2023-06-10T18:48:02.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.637+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa83c0cf8a0 con 0x7fa83c158fb0 2023-06-10T18:48:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.638+0000 7fa82affd700 1 -- 172.21.15.104:0/270595269 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7fa82c057b60 con 0x7fa83c158fb0 2023-06-10T18:48:02.755 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:02.756 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 msgr2=0x7fa81c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 0x7fa81c06d190 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa830009a80 tx=0x7fa830008040 comp rx=0 tx=0).stop 2023-06-10T18:48:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 msgr2=0x7fa83c0b2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0010e0 tx=0x7fa82c0076a0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 shutdown_connections 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa81c06ace0 0x7fa81c06d190 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa83c0b3380 0x7fa83c0c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa824004d90 0x7fa83c0b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 --2- 172.21.15.104:0/270595269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa83c158fb0 0x7fa83c0b2e40 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 >> 172.21.15.104:0/270595269 conn(0x7fa83c06b8a0 msgr2=0x7fa83c0ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:02.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 shutdown_connections 2023-06-10T18:48:02.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.640+0000 7fa840e70700 1 -- 172.21.15.104:0/270595269 wait complete. 2023-06-10T18:48:02.759 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:02.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:02.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.876+0000 7fc58272a700 1 Processor -- start 2023-06-10T18:48:02.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.876+0000 7fc58272a700 1 -- start start 2023-06-10T18:48:02.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc58272a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc57c08a3d0 0x7fc57c167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc58272a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c0cadd0 con 0x7fc57c0ce350 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc58272a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c0cb0b0 con 0x7fc57c06b1f0 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc58272a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c0cfeb0 con 0x7fc57c08a3d0 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc57c0ce350 0x7fc57c0cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56388/0 (socket says 172.21.15.104:56388) 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57b7fe700 1 -- 172.21.15.104:0/2719422400 learned_addr learned my addr 172.21.15.104:0/2719422400 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc580cc7700 1 --2- 172.21.15.104:0/2719422400 >> v2:172.21.15.114:3300/0 conn(0x7fc57c08a3d0 0x7fc57c167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791785073 0 0) 0x7fc57c0cadd0 con 0x7fc57c0ce350 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5640035b0 con 0x7fc57c0ce350 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360391870 0 0) 0x7fc5640035b0 con 0x7fc57c0ce350 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 >> v2:172.21.15.114:3300/0 conn(0x7fc57c08a3d0 msgr2=0x7fc57c167940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 --2- 172.21.15.104:0/2719422400 >> v2:172.21.15.114:3300/0 conn(0x7fc57c08a3d0 0x7fc57c167940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 >> v1:172.21.15.114:6789/0 conn(0x7fc57c06b1f0 legacy=0x7fc57c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0cdeb0 con 0x7fc57c0ce350 2023-06-10T18:48:02.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc56c001670 con 0x7fc57c0ce350 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 >> v1:172.21.15.104:6789/0 conn(0x7fc57c0ce350 legacy=0x7fc57c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564003a00 0x7fc564003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc564004850 0x7fc564004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 0x7fc56400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0cfeb0 con 0x7fc564005200 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c110 con 0x7fc564004850 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.877+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c2d0 con 0x7fc564003a00 2023-06-10T18:48:02.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 0x7fc56400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57b7fe700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc564004850 0x7fc564004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57bfff700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564003a00 0x7fc564003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 -- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564003a00 msgr2=0x7fc564003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564003a00 0x7fc564003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 -- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc564004850 msgr2=0x7fc564004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc564004850 0x7fc564004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 -- 172.21.15.104:0/2719422400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640035b0 con 0x7fc564005200 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57b7fe700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc564004850 0x7fc564004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 -- 172.21.15.104:0/2719422400 reap_dead start 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc580cc7700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 0x7fc56400ba10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc574009fc0 tx=0x7fc564005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec22a65bfd5e878 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574003a20 con 0x7fc564005200 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57400f040 con 0x7fc564005200 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc57affd700 1 -- 172.21.15.104:0/2719422400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574003de0 con 0x7fc564005200 2023-06-10T18:48:02.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc58272a700 1 -- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 msgr2=0x7fc56400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.878+0000 7fc58272a700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 0x7fc56400ba10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc574009fc0 tx=0x7fc564005880 comp rx=0 tx=0).stop 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.879+0000 7fc58272a700 1 -- 172.21.15.104:0/2719422400 shutdown_connections 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.879+0000 7fc58272a700 1 --2- 172.21.15.104:0/2719422400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc564005200 0x7fc56400ba10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.879+0000 7fc58272a700 1 -- 172.21.15.104:0/2719422400 >> 172.21.15.104:0/2719422400 conn(0x7fc57c089fd0 msgr2=0x7fc57c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.879+0000 7fc58272a700 1 -- 172.21.15.104:0/2719422400 shutdown_connections 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.879+0000 7fc58272a700 1 -- 172.21.15.104:0/2719422400 wait complete. 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.880+0000 7fc58272a700 1 Processor -- start 2023-06-10T18:48:02.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.880+0000 7fc58272a700 1 -- start start 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 0x7fc57c09de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57c0ca570 0x7fc57c151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0ca0e0 con 0x7fc57c09b8e0 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c09e4c0 con 0x7fc57c0ca570 2023-06-10T18:48:02.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0c97b0 con 0x7fc564005200 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58574/0 (socket says 172.21.15.104:58574) 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 -- 172.21.15.104:0/3536272873 learned_addr learned my addr 172.21.15.104:0/3536272873 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc57b7fe700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57c0ca570 0x7fc57c151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc57bfff700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 0x7fc57c09de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 -- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 msgr2=0x7fc57c09de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 0x7fc57c09de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 -- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57c0ca570 msgr2=0x7fc57c151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57c0ca570 0x7fc57c151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 -- 172.21.15.104:0/3536272873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0c9a90 con 0x7fc57c09b8e0 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc57bfff700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 0x7fc57c09de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc580cc7700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc57c066590 tx=0x7fc574009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574003e90 con 0x7fc57c09b8e0 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574004230 con 0x7fc57c09b8e0 2023-06-10T18:48:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c0c9d70 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5740045e0 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.881+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c152200 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.882+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c0cabc0 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.883+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc574004740 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.883+0000 7fc578ff9700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 0x7fc56806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.883+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc57408a7c0 con 0x7fc57c09b8e0 2023-06-10T18:48:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.884+0000 7fc57bfff700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 0x7fc56806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.884+0000 7fc57bfff700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 0x7fc56806d190 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc570009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:02.888+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc57c0cabc0 con 0x7fc57c09b8e0 2023-06-10T18:48:03.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.073+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc57c0cabc0 con 0x7fc57c09b8e0 2023-06-10T18:48:03.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.075+0000 7fc578ff9700 1 -- 172.21.15.104:0/3536272873 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7fc57c0cabc0 con 0x7fc57c09b8e0 2023-06-10T18:48:03.080 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:03.080 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:03.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 msgr2=0x7fc56806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 0x7fc56806d190 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc570009040 comp rx=0 tx=0).stop 2023-06-10T18:48:03.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 msgr2=0x7fc57c09a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc57c066590 tx=0x7fc574009750 comp rx=0 tx=0).stop 2023-06-10T18:48:03.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 shutdown_connections 2023-06-10T18:48:03.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc57c09b8e0 0x7fc57c09a480 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc564005200 0x7fc57c09de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc56806ace0 0x7fc56806d190 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 --2- 172.21.15.104:0/3536272873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57c0ca570 0x7fc57c151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 >> 172.21.15.104:0/3536272873 conn(0x7fc57c089fd0 msgr2=0x7fc57c15fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 shutdown_connections 2023-06-10T18:48:03.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.077+0000 7fc58272a700 1 -- 172.21.15.104:0/3536272873 wait complete. 2023-06-10T18:48:03.094 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:48:03.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:48:03.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.210+0000 7fc90b30c700 1 Processor -- start 2023-06-10T18:48:03.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.210+0000 7fc90b30c700 1 -- start start 2023-06-10T18:48:03.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc90b30c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc90406af80 0x7fc90406b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc90b30c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9040b1a20 0x7fc9040cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc90b30c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90406b890 con 0x7fc9040b15b0 2023-06-10T18:48:03.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc90b30c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040b1ea0 con 0x7fc90406af80 2023-06-10T18:48:03.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc90b30c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040cb880 con 0x7fc9040b1a20 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc9090a8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc90406af80 0x7fc90406b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc9088a7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc9040b15b0 0x7fc9040cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56406/0 (socket says 172.21.15.104:56406) 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc9088a7700 1 -- 172.21.15.104:0/4069983567 learned_addr learned my addr 172.21.15.104:0/4069983567 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.211+0000 7fc9098a9700 1 --2- 172.21.15.104:0/4069983567 >> v2:172.21.15.114:3300/0 conn(0x7fc9040b1a20 0x7fc9040cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc9090a8700 1 -- 172.21.15.104:0/4069983567 >> v2:172.21.15.114:3300/0 conn(0x7fc9040b1a20 msgr2=0x7fc9040cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc9090a8700 1 --2- 172.21.15.104:0/4069983567 >> v2:172.21.15.114:3300/0 conn(0x7fc9040b1a20 0x7fc9040cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc9090a8700 1 -- 172.21.15.104:0/4069983567 >> v1:172.21.15.104:6789/0 conn(0x7fc9040b15b0 legacy=0x7fc9040cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc8fbfff700 1 -- 172.21.15.104:0/4069983567 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819869502 0 0) 0x7fc90406b890 con 0x7fc9040b15b0 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc9090a8700 1 -- 172.21.15.104:0/4069983567 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040cbb60 con 0x7fc90406af80 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc9090a8700 1 --2- 172.21.15.104:0/4069983567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc90406b350 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009840 tx=0x7fc8f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb63bf3435e60bf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc8fbfff700 1 -- 172.21.15.104:0/4069983567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4011830 con 0x7fc90406af80 2023-06-10T18:48:03.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc8fbfff700 1 -- 172.21.15.104:0/4069983567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4011990 con 0x7fc90406af80 2023-06-10T18:48:03.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc8fbfff700 1 -- 172.21.15.104:0/4069983567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4011d20 con 0x7fc90406af80 2023-06-10T18:48:03.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc90b30c700 1 -- 172.21.15.104:0/4069983567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 msgr2=0x7fc90406b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc90b30c700 1 --2- 172.21.15.104:0/4069983567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc90406b350 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009840 tx=0x7fc8f4009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:03.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc90b30c700 1 -- 172.21.15.104:0/4069983567 shutdown_connections 2023-06-10T18:48:03.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.212+0000 7fc90b30c700 1 --2- 172.21.15.104:0/4069983567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc90406b350 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.213+0000 7fc90b30c700 1 --2- 172.21.15.104:0/4069983567 >> v2:172.21.15.114:3300/0 conn(0x7fc9040b1a20 0x7fc9040cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.213+0000 7fc90b30c700 1 -- 172.21.15.104:0/4069983567 >> 172.21.15.104:0/4069983567 conn(0x7fc904089fd0 msgr2=0x7fc9040d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.213+0000 7fc90b30c700 1 -- 172.21.15.104:0/4069983567 shutdown_connections 2023-06-10T18:48:03.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.213+0000 7fc90b30c700 1 -- 172.21.15.104:0/4069983567 wait complete. 2023-06-10T18:48:03.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.214+0000 7fc90b30c700 1 Processor -- start 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.214+0000 7fc90b30c700 1 -- start start 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.214+0000 7fc90b30c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9040b15b0 0x7fc9041523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9040b1a20 0x7fc9041e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041e5990 con 0x7fc90406af80 2023-06-10T18:48:03.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041e5ec0 con 0x7fc9040b15b0 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040b4a40 con 0x7fc9040b1a20 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58584/0 (socket says 172.21.15.104:58584) 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 -- 172.21.15.104:0/3111927826 learned_addr learned my addr 172.21.15.104:0/3111927826 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9088a7700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9040b15b0 0x7fc9041523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9098a9700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9040b1a20 0x7fc9041e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 -- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9040b1a20 msgr2=0x7fc9041e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9040b1a20 0x7fc9041e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 -- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9040b15b0 msgr2=0x7fc9041523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9040b15b0 0x7fc9041523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 -- 172.21.15.104:0/3111927826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40094a0 con 0x7fc90406af80 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc9090a8700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc9040b3410 tx=0x7fc8f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40036a0 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f401e070 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9041e5be0 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003a20 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.215+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9040b46d0 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.216+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9040b4d20 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.217+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8f4003b80 con 0x7fc90406af80 2023-06-10T18:48:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.217+0000 7fc8f9ffb700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 0x7fc8e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.217+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc8f408a680 con 0x7fc90406af80 2023-06-10T18:48:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.217+0000 7fc9088a7700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 0x7fc8e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.218+0000 7fc9088a7700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 0x7fc8e806d190 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c52f0 tx=0x7fc900007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.222+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8f4055680 con 0x7fc90406af80 2023-06-10T18:48:03.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.405+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9040cc120 con 0x7fc8e806ace0 2023-06-10T18:48:03.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.415+0000 7fc8f9ffb700 1 -- 172.21.15.104:0/3111927826 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311620 (secure 0 0 0) 0x7fc900012380 con 0x7fc8e806ace0 2023-06-10T18:48:03.423 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:03.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 msgr2=0x7fc8e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 0x7fc8e806d190 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c52f0 tx=0x7fc900007c90 comp rx=0 tx=0).stop 2023-06-10T18:48:03.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 msgr2=0x7fc9040b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc9040b3410 tx=0x7fc8f4004750 comp rx=0 tx=0).stop 2023-06-10T18:48:03.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 shutdown_connections 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9040b1a20 0x7fc9041e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc90406af80 0x7fc9040b4ed0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8e806ace0 0x7fc8e806d190 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 --2- 172.21.15.104:0/3111927826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9040b15b0 0x7fc9041523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 >> 172.21.15.104:0/3111927826 conn(0x7fc904089fd0 msgr2=0x7fc90406fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 shutdown_connections 2023-06-10T18:48:03.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.418+0000 7fc90b30c700 1 -- 172.21.15.104:0/3111927826 wait complete. 2023-06-10T18:48:03.427 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:48:03.434 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2023-06-10T18:48:01.409457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222040,"kb_used_data":6424,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754752680,"statfs":{"total":773094113280,"available":772866744320,"internally_reserved":0,"allocated":6578176,"data_stored":3653616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001049"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400978+0000","last_change":"2023-06-10T18:47:59.270757+0000","last_active":"2023-06-10T18:48:00.400978+0000","last_peered":"2023-06-10T18:48:00.400978+0000","last_clean":"2023-06-10T18:48:00.400978+0000","last_became_active":"2023-06-10T18:47:59.270515+0000","last_became_peered":"2023-06-10T18:47:59.270515+0000","last_unstale":"2023-06-10T18:48:00.400978+0000","last_undegraded":"2023-06-10T18:48:00.400978+0000","last_fullsized":"2023-06-10T18:48:00.400978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:06.126008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254848+0000","last_change":"2023-06-10T18:47:59.255348+0000","last_active":"2023-06-10T18:48:00.254848+0000","last_peered":"2023-06-10T18:48:00.254848+0000","last_clean":"2023-06-10T18:48:00.254848+0000","last_became_active":"2023-06-10T18:47:59.254464+0000","last_became_peered":"2023-06-10T18:47:59.254464+0000","last_unstale":"2023-06-10T18:48:00.254848+0000","last_undegraded":"2023-06-10T18:48:00.254848+0000","last_fullsized":"2023-06-10T18:48:00.254848+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:04:57.552360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401945+0000","last_change":"2023-06-10T18:47:59.258124+0000","last_active":"2023-06-10T18:48:00.401945+0000","last_peered":"2023-06-10T18:48:00.401945+0000","last_clean":"2023-06-10T18:48:00.401945+0000","last_became_active":"2023-06-10T18:47:59.257849+0000","last_became_peered":"2023-06-10T18:47:59.257849+0000","last_unstale":"2023-06-10T18:48:00.401945+0000","last_undegraded":"2023-06-10T18:48:00.401945+0000","last_fullsized":"2023-06-10T18:48:00.401945+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:46:38.601166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401646+0000","last_change":"2023-06-10T18:47:59.271245+0000","last_active":"2023-06-10T18:48:00.401646+0000","last_peered":"2023-06-10T18:48:00.401646+0000","last_clean":"2023-06-10T18:48:00.401646+0000","last_became_active":"2023-06-10T18:47:59.270943+0000","last_became_peered":"2023-06-10T18:47:59.270943+0000","last_unstale":"2023-06-10T18:48:00.401646+0000","last_undegraded":"2023-06-10T18:48:00.401646+0000","last_fullsized":"2023-06-10T18:48:00.401646+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:32.792889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400739+0000","last_change":"2023-06-10T18:47:59.270430+0000","last_active":"2023-06-10T18:48:00.400739+0000","last_peered":"2023-06-10T18:48:00.400739+0000","last_clean":"2023-06-10T18:48:00.400739+0000","last_became_active":"2023-06-10T18:47:59.270065+0000","last_became_peered":"2023-06-10T18:47:59.270065+0000","last_unstale":"2023-06-10T18:48:00.400739+0000","last_undegraded":"2023-06-10T18:48:00.400739+0000","last_fullsized":"2023-06-10T18:48:00.400739+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:24.430578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254008+0000","last_change":"2023-06-10T18:47:59.270812+0000","last_active":"2023-06-10T18:48:00.254008+0000","last_peered":"2023-06-10T18:48:00.254008+0000","last_clean":"2023-06-10T18:48:00.254008+0000","last_became_active":"2023-06-10T18:47:59.270515+0000","last_became_peered":"2023-06-10T18:47:59.270515+0000","last_unstale":"2023-06-10T18:48:00.254008+0000","last_undegraded":"2023-06-10T18:48:00.254008+0000","last_fullsized":"2023-06-10T18:48:00.254008+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:09.734260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255580+0000","last_change":"2023-06-10T18:47:59.261582+0000","last_active":"2023-06-10T18:48:00.255580+0000","last_peered":"2023-06-10T18:48:00.255580+0000","last_clean":"2023-06-10T18:48:00.255580+0000","last_became_active":"2023-06-10T18:47:59.261273+0000","last_became_peered":"2023-06-10T18:47:59.261273+0000","last_unstale":"2023-06-10T18:48:00.255580+0000","last_undegraded":"2023-06-10T18:48:00.255580+0000","last_fullsized":"2023-06-10T18:48:00.255580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253842+0000","last_change":"2023-06-10T18:47:59.256478+0000","last_active":"2023-06-10T18:48:00.253842+0000","last_peered":"2023-06-10T18:48:00.253842+0000","last_clean":"2023-06-10T18:48:00.253842+0000","last_became_active":"2023-06-10T18:47:59.256207+0000","last_became_peered":"2023-06-10T18:47:59.256207+0000","last_unstale":"2023-06-10T18:48:00.253842+0000","last_undegraded":"2023-06-10T18:48:00.253842+0000","last_fullsized":"2023-06-10T18:48:00.253842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:42.947644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256438+0000","last_change":"2023-06-10T18:47:59.258747+0000","last_active":"2023-06-10T18:48:00.256438+0000","last_peered":"2023-06-10T18:48:00.256438+0000","last_clean":"2023-06-10T18:48:00.256438+0000","last_became_active":"2023-06-10T18:47:59.258474+0000","last_became_peered":"2023-06-10T18:47:59.258474+0000","last_unstale":"2023-06-10T18:48:00.256438+0000","last_undegraded":"2023-06-10T18:48:00.256438+0000","last_fullsized":"2023-06-10T18:48:00.256438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:30.306301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401171+0000","last_change":"2023-06-10T18:47:59.260235+0000","last_active":"2023-06-10T18:48:00.401171+0000","last_peered":"2023-06-10T18:48:00.401171+0000","last_clean":"2023-06-10T18:48:00.401171+0000","last_became_active":"2023-06-10T18:47:59.259924+0000","last_became_peered":"2023-06-10T18:47:59.259924+0000","last_unstale":"2023-06-10T18:48:00.401171+0000","last_undegraded":"2023-06-10T18:48:00.401171+0000","last_fullsized":"2023-06-10T18:48:00.401171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:49:38.478392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401262+0000","last_change":"2023-06-10T18:47:59.256416+0000","last_active":"2023-06-10T18:48:00.401262+0000","last_peered":"2023-06-10T18:48:00.401262+0000","last_clean":"2023-06-10T18:48:00.401262+0000","last_became_active":"2023-06-10T18:47:59.256161+0000","last_became_peered":"2023-06-10T18:47:59.256161+0000","last_unstale":"2023-06-10T18:48:00.401262+0000","last_undegraded":"2023-06-10T18:48:00.401262+0000","last_fullsized":"2023-06-10T18:48:00.401262+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:20:38.752799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401828+0000","last_change":"2023-06-10T18:47:59.266083+0000","last_active":"2023-06-10T18:48:00.401828+0000","last_peered":"2023-06-10T18:48:00.401828+0000","last_clean":"2023-06-10T18:48:00.401828+0000","last_became_active":"2023-06-10T18:47:59.265817+0000","last_became_peered":"2023-06-10T18:47:59.265817+0000","last_unstale":"2023-06-10T18:48:00.401828+0000","last_undegraded":"2023-06-10T18:48:00.401828+0000","last_fullsized":"2023-06-10T18:48:00.401828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:28.808249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255792+0000","last_change":"2023-06-10T18:47:59.259652+0000","last_active":"2023-06-10T18:48:00.255792+0000","last_peered":"2023-06-10T18:48:00.255792+0000","last_clean":"2023-06-10T18:48:00.255792+0000","last_became_active":"2023-06-10T18:47:59.259396+0000","last_became_peered":"2023-06-10T18:47:59.259396+0000","last_unstale":"2023-06-10T18:48:00.255792+0000","last_undegraded":"2023-06-10T18:48:00.255792+0000","last_fullsized":"2023-06-10T18:48:00.255792+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:59:29.107540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256016+0000","last_change":"2023-06-10T18:47:59.270952+0000","last_active":"2023-06-10T18:48:00.256016+0000","last_peered":"2023-06-10T18:48:00.256016+0000","last_clean":"2023-06-10T18:48:00.256016+0000","last_became_active":"2023-06-10T18:47:59.270649+0000","last_became_peered":"2023-06-10T18:47:59.270649+0000","last_unstale":"2023-06-10T18:48:00.256016+0000","last_undegraded":"2023-06-10T18:48:00.256016+0000","last_fullsized":"2023-06-10T18:48:00.256016+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:08:40.378289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255813+0000","last_change":"2023-06-10T18:47:59.256978+0000","last_active":"2023-06-10T18:48:00.255813+0000","last_peered":"2023-06-10T18:48:00.255813+0000","last_clean":"2023-06-10T18:48:00.255813+0000","last_became_active":"2023-06-10T18:47:59.256675+0000","last_became_peered":"2023-06-10T18:47:59.256675+0000","last_unstale":"2023-06-10T18:48:00.255813+0000","last_undegraded":"2023-06-10T18:48:00.255813+0000","last_fullsized":"2023-06-10T18:48:00.255813+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:03.380984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254733+0000","last_change":"2023-06-10T18:47:59.259833+0000","last_active":"2023-06-10T18:48:00.254733+0000","last_peered":"2023-06-10T18:48:00.254733+0000","last_clean":"2023-06-10T18:48:00.254733+0000","last_became_active":"2023-06-10T18:47:59.259568+0000","last_became_peered":"2023-06-10T18:47:59.259568+0000","last_unstale":"2023-06-10T18:48:00.254733+0000","last_undegraded":"2023-06-10T18:48:00.254733+0000","last_fullsized":"2023-06-10T18:48:00.254733+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254225+0000","last_change":"2023-06-10T18:47:59.264346+0000","last_active":"2023-06-10T18:48:00.254225+0000","last_peered":"2023-06-10T18:48:00.254225+0000","last_clean":"2023-06-10T18:48:00.254225+0000","last_became_active":"2023-06-10T18:47:59.264063+0000","last_became_peered":"2023-06-10T18:47:59.264063+0000","last_unstale":"2023-06-10T18:48:00.254225+0000","last_undegraded":"2023-06-10T18:48:00.254225+0000","last_fullsized":"2023-06-10T18:48:00.254225+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:17.078369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401707+0000","last_change":"2023-06-10T18:47:59.253060+0000","last_active":"2023-06-10T18:48:00.401707+0000","last_peered":"2023-06-10T18:48:00.401707+0000","last_clean":"2023-06-10T18:48:00.401707+0000","last_became_active":"2023-06-10T18:47:59.252812+0000","last_became_peered":"2023-06-10T18:47:59.252812+0000","last_unstale":"2023-06-10T18:48:00.401707+0000","last_undegraded":"2023-06-10T18:48:00.401707+0000","last_fullsized":"2023-06-10T18:48:00.401707+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:00.483660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400768+0000","last_change":"2023-06-10T18:47:59.270792+0000","last_active":"2023-06-10T18:48:00.400768+0000","last_peered":"2023-06-10T18:48:00.400768+0000","last_clean":"2023-06-10T18:48:00.400768+0000","last_became_active":"2023-06-10T18:47:59.270563+0000","last_became_peered":"2023-06-10T18:47:59.270563+0000","last_unstale":"2023-06-10T18:48:00.400768+0000","last_undegraded":"2023-06-10T18:48:00.400768+0000","last_fullsized":"2023-06-10T18:48:00.400768+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:47:58.932650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254200+0000","last_change":"2023-06-10T18:47:59.258843+0000","last_active":"2023-06-10T18:48:00.254200+0000","last_peered":"2023-06-10T18:48:00.254200+0000","last_clean":"2023-06-10T18:48:00.254200+0000","last_became_active":"2023-06-10T18:47:59.258566+0000","last_became_peered":"2023-06-10T18:47:59.258566+0000","last_unstale":"2023-06-10T18:48:00.254200+0000","last_undegraded":"2023-06-10T18:48:00.254200+0000","last_fullsized":"2023-06-10T18:48:00.254200+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:08:40.378289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401411+0000","last_change":"2023-06-10T18:47:59.260327+0000","last_active":"2023-06-10T18:48:00.401411+0000","last_peered":"2023-06-10T18:48:00.401411+0000","last_clean":"2023-06-10T18:48:00.401411+0000","last_became_active":"2023-06-10T18:47:59.260046+0000","last_became_peered":"2023-06-10T18:47:59.260046+0000","last_unstale":"2023-06-10T18:48:00.401411+0000","last_undegraded":"2023-06-10T18:48:00.401411+0000","last_fullsized":"2023-06-10T18:48:00.401411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:22.297934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254314+0000","last_change":"2023-06-10T18:47:59.271695+0000","last_active":"2023-06-10T18:48:00.254314+0000","last_peered":"2023-06-10T18:48:00.254314+0000","last_clean":"2023-06-10T18:48:00.254314+0000","last_became_active":"2023-06-10T18:47:59.271109+0000","last_became_peered":"2023-06-10T18:47:59.271109+0000","last_unstale":"2023-06-10T18:48:00.254314+0000","last_undegraded":"2023-06-10T18:48:00.254314+0000","last_fullsized":"2023-06-10T18:48:00.254314+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:28:22.113139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401888+0000","last_change":"2023-06-10T18:47:59.259022+0000","last_active":"2023-06-10T18:48:00.401888+0000","last_peered":"2023-06-10T18:48:00.401888+0000","last_clean":"2023-06-10T18:48:00.401888+0000","last_became_active":"2023-06-10T18:47:59.258727+0000","last_became_peered":"2023-06-10T18:47:59.258727+0000","last_unstale":"2023-06-10T18:48:00.401888+0000","last_undegraded":"2023-06-10T18:48:00.401888+0000","last_fullsized":"2023-06-10T18:48:00.401888+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:49.543959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255271+0000","last_change":"2023-06-10T18:47:59.263274+0000","last_active":"2023-06-10T18:48:00.255271+0000","last_peered":"2023-06-10T18:48:00.255271+0000","last_clean":"2023-06-10T18:48:00.255271+0000","last_became_active":"2023-06-10T18:47:59.263020+0000","last_became_peered":"2023-06-10T18:47:59.263020+0000","last_unstale":"2023-06-10T18:48:00.255271+0000","last_undegraded":"2023-06-10T18:48:00.255271+0000","last_fullsized":"2023-06-10T18:48:00.255271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:26:24.637191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401105+0000","last_change":"2023-06-10T18:47:59.270306+0000","last_active":"2023-06-10T18:48:00.401105+0000","last_peered":"2023-06-10T18:48:00.401105+0000","last_clean":"2023-06-10T18:48:00.401105+0000","last_became_active":"2023-06-10T18:47:59.270025+0000","last_became_peered":"2023-06-10T18:47:59.270025+0000","last_unstale":"2023-06-10T18:48:00.401105+0000","last_undegraded":"2023-06-10T18:48:00.401105+0000","last_fullsized":"2023-06-10T18:48:00.401105+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:18:12.102366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401949+0000","last_change":"2023-06-10T18:47:59.259768+0000","last_active":"2023-06-10T18:48:00.401949+0000","last_peered":"2023-06-10T18:48:00.401949+0000","last_clean":"2023-06-10T18:48:00.401949+0000","last_became_active":"2023-06-10T18:47:59.259498+0000","last_became_peered":"2023-06-10T18:47:59.259498+0000","last_unstale":"2023-06-10T18:48:00.401949+0000","last_undegraded":"2023-06-10T18:48:00.401949+0000","last_fullsized":"2023-06-10T18:48:00.401949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:22.297934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253974+0000","last_change":"2023-06-10T18:47:59.259734+0000","last_active":"2023-06-10T18:48:00.253974+0000","last_peered":"2023-06-10T18:48:00.253974+0000","last_clean":"2023-06-10T18:48:00.253974+0000","last_became_active":"2023-06-10T18:47:59.259382+0000","last_became_peered":"2023-06-10T18:47:59.259382+0000","last_unstale":"2023-06-10T18:48:00.253974+0000","last_undegraded":"2023-06-10T18:48:00.253974+0000","last_fullsized":"2023-06-10T18:48:00.253974+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:28:28.123158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400867+0000","last_change":"2023-06-10T18:47:59.271191+0000","last_active":"2023-06-10T18:48:00.400867+0000","last_peered":"2023-06-10T18:48:00.400867+0000","last_clean":"2023-06-10T18:48:00.400867+0000","last_became_active":"2023-06-10T18:47:59.270956+0000","last_became_peered":"2023-06-10T18:47:59.270956+0000","last_unstale":"2023-06-10T18:48:00.400867+0000","last_undegraded":"2023-06-10T18:48:00.400867+0000","last_fullsized":"2023-06-10T18:48:00.400867+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:05.090087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254704+0000","last_change":"2023-06-10T18:47:59.259234+0000","last_active":"2023-06-10T18:48:00.254704+0000","last_peered":"2023-06-10T18:48:00.254704+0000","last_clean":"2023-06-10T18:48:00.254704+0000","last_became_active":"2023-06-10T18:47:59.259003+0000","last_became_peered":"2023-06-10T18:47:59.259003+0000","last_unstale":"2023-06-10T18:48:00.254704+0000","last_undegraded":"2023-06-10T18:48:00.254704+0000","last_fullsized":"2023-06-10T18:48:00.254704+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:16.936511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401687+0000","last_change":"2023-06-10T18:47:59.258089+0000","last_active":"2023-06-10T18:48:00.401687+0000","last_peered":"2023-06-10T18:48:00.401687+0000","last_clean":"2023-06-10T18:48:00.401687+0000","last_became_active":"2023-06-10T18:47:59.257778+0000","last_became_peered":"2023-06-10T18:47:59.257778+0000","last_unstale":"2023-06-10T18:48:00.401687+0000","last_undegraded":"2023-06-10T18:48:00.401687+0000","last_fullsized":"2023-06-10T18:48:00.401687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:49:39.701679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400467+0000","last_change":"2023-06-10T18:47:59.270406+0000","last_active":"2023-06-10T18:48:00.400467+0000","last_peered":"2023-06-10T18:48:00.400467+0000","last_clean":"2023-06-10T18:48:00.400467+0000","last_became_active":"2023-06-10T18:47:59.270156+0000","last_became_peered":"2023-06-10T18:47:59.270156+0000","last_unstale":"2023-06-10T18:48:00.400467+0000","last_undegraded":"2023-06-10T18:48:00.400467+0000","last_fullsized":"2023-06-10T18:48:00.400467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:25:02.081758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253863+0000","last_change":"2023-06-10T18:47:59.258426+0000","last_active":"2023-06-10T18:48:00.253863+0000","last_peered":"2023-06-10T18:48:00.253863+0000","last_clean":"2023-06-10T18:48:00.253863+0000","last_became_active":"2023-06-10T18:47:59.258156+0000","last_became_peered":"2023-06-10T18:47:59.258156+0000","last_unstale":"2023-06-10T18:48:00.253863+0000","last_undegraded":"2023-06-10T18:48:00.253863+0000","last_fullsized":"2023-06-10T18:48:00.253863+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:09:30.096718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254556+0000","last_change":"2023-06-10T18:47:59.253101+0000","last_active":"2023-06-10T18:48:00.254556+0000","last_peered":"2023-06-10T18:48:00.254556+0000","last_clean":"2023-06-10T18:48:00.254556+0000","last_became_active":"2023-06-10T18:47:59.252851+0000","last_became_peered":"2023-06-10T18:47:59.252851+0000","last_unstale":"2023-06-10T18:48:00.254556+0000","last_undegraded":"2023-06-10T18:48:00.254556+0000","last_fullsized":"2023-06-10T18:48:00.254556+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:59.652589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254657+0000","last_change":"2023-06-10T18:47:58.867330+0000","last_active":"2023-06-10T18:48:00.254657+0000","last_peered":"2023-06-10T18:48:00.254657+0000","last_clean":"2023-06-10T18:48:00.254657+0000","last_became_active":"2023-06-10T18:47:58.867029+0000","last_became_peered":"2023-06-10T18:47:58.867029+0000","last_unstale":"2023-06-10T18:48:00.254657+0000","last_undegraded":"2023-06-10T18:48:00.254657+0000","last_fullsized":"2023-06-10T18:48:00.254657+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:20.841046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401304+0000","last_change":"2023-06-10T18:47:59.267409+0000","last_active":"2023-06-10T18:48:00.401304+0000","last_peered":"2023-06-10T18:48:00.401304+0000","last_clean":"2023-06-10T18:48:00.401304+0000","last_became_active":"2023-06-10T18:47:59.267043+0000","last_became_peered":"2023-06-10T18:47:59.267043+0000","last_unstale":"2023-06-10T18:48:00.401304+0000","last_undegraded":"2023-06-10T18:48:00.401304+0000","last_fullsized":"2023-06-10T18:48:00.401304+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:40.094000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254717+0000","last_change":"2023-06-10T18:47:58.858136+0000","last_active":"2023-06-10T18:48:00.254717+0000","last_peered":"2023-06-10T18:48:00.254717+0000","last_clean":"2023-06-10T18:48:00.254717+0000","last_became_active":"2023-06-10T18:47:58.857856+0000","last_became_peered":"2023-06-10T18:47:58.857856+0000","last_unstale":"2023-06-10T18:48:00.254717+0000","last_undegraded":"2023-06-10T18:48:00.254717+0000","last_fullsized":"2023-06-10T18:48:00.254717+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:39.397363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254918+0000","last_change":"2023-06-10T18:47:59.253824+0000","last_active":"2023-06-10T18:48:00.254918+0000","last_peered":"2023-06-10T18:48:00.254918+0000","last_clean":"2023-06-10T18:48:00.254918+0000","last_became_active":"2023-06-10T18:47:59.253570+0000","last_became_peered":"2023-06-10T18:47:59.253570+0000","last_unstale":"2023-06-10T18:48:00.254918+0000","last_undegraded":"2023-06-10T18:48:00.254918+0000","last_fullsized":"2023-06-10T18:48:00.254918+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:03.380984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254359+0000","last_change":"2023-06-10T18:47:58.861996+0000","last_active":"2023-06-10T18:48:00.254359+0000","last_peered":"2023-06-10T18:48:00.254359+0000","last_clean":"2023-06-10T18:48:00.254359+0000","last_became_active":"2023-06-10T18:47:58.861765+0000","last_became_peered":"2023-06-10T18:47:58.861765+0000","last_unstale":"2023-06-10T18:48:00.254359+0000","last_undegraded":"2023-06-10T18:48:00.254359+0000","last_fullsized":"2023-06-10T18:48:00.254359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:46.896527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254102+0000","last_change":"2023-06-10T18:47:59.262983+0000","last_active":"2023-06-10T18:48:00.254102+0000","last_peered":"2023-06-10T18:48:00.254102+0000","last_clean":"2023-06-10T18:48:00.254102+0000","last_became_active":"2023-06-10T18:47:59.262724+0000","last_became_peered":"2023-06-10T18:47:59.262724+0000","last_unstale":"2023-06-10T18:48:00.254102+0000","last_undegraded":"2023-06-10T18:48:00.254102+0000","last_fullsized":"2023-06-10T18:48:00.254102+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:24:37.026536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402121+0000","last_change":"2023-06-10T18:47:58.869907+0000","last_active":"2023-06-10T18:48:00.402121+0000","last_peered":"2023-06-10T18:48:00.402121+0000","last_clean":"2023-06-10T18:48:00.402121+0000","last_became_active":"2023-06-10T18:47:58.869623+0000","last_became_peered":"2023-06-10T18:47:58.869623+0000","last_unstale":"2023-06-10T18:48:00.402121+0000","last_undegraded":"2023-06-10T18:48:00.402121+0000","last_fullsized":"2023-06-10T18:48:00.402121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:05.006193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254432+0000","last_change":"2023-06-10T18:47:59.271824+0000","last_active":"2023-06-10T18:48:00.254432+0000","last_peered":"2023-06-10T18:48:00.254432+0000","last_clean":"2023-06-10T18:48:00.254432+0000","last_became_active":"2023-06-10T18:47:59.271467+0000","last_became_peered":"2023-06-10T18:47:59.271467+0000","last_unstale":"2023-06-10T18:48:00.254432+0000","last_undegraded":"2023-06-10T18:48:00.254432+0000","last_fullsized":"2023-06-10T18:48:00.254432+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:08:37.981895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256077+0000","last_change":"2023-06-10T18:47:58.863602+0000","last_active":"2023-06-10T18:48:00.256077+0000","last_peered":"2023-06-10T18:48:00.256077+0000","last_clean":"2023-06-10T18:48:00.256077+0000","last_became_active":"2023-06-10T18:47:58.863323+0000","last_became_peered":"2023-06-10T18:47:58.863323+0000","last_unstale":"2023-06-10T18:48:00.256077+0000","last_undegraded":"2023-06-10T18:48:00.256077+0000","last_fullsized":"2023-06-10T18:48:00.256077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:16.073766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254325+0000","last_change":"2023-06-10T18:47:59.259732+0000","last_active":"2023-06-10T18:48:00.254325+0000","last_peered":"2023-06-10T18:48:00.254325+0000","last_clean":"2023-06-10T18:48:00.254325+0000","last_became_active":"2023-06-10T18:47:59.258654+0000","last_became_peered":"2023-06-10T18:47:59.258654+0000","last_unstale":"2023-06-10T18:48:00.254325+0000","last_undegraded":"2023-06-10T18:48:00.254325+0000","last_fullsized":"2023-06-10T18:48:00.254325+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:30.306301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254133+0000","last_change":"2023-06-10T18:47:58.863792+0000","last_active":"2023-06-10T18:48:00.254133+0000","last_peered":"2023-06-10T18:48:00.254133+0000","last_clean":"2023-06-10T18:48:00.254133+0000","last_became_active":"2023-06-10T18:47:58.863542+0000","last_became_peered":"2023-06-10T18:47:58.863542+0000","last_unstale":"2023-06-10T18:48:00.254133+0000","last_undegraded":"2023-06-10T18:48:00.254133+0000","last_fullsized":"2023-06-10T18:48:00.254133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:07.030546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255416+0000","last_change":"2023-06-10T18:47:59.267928+0000","last_active":"2023-06-10T18:48:00.255416+0000","last_peered":"2023-06-10T18:48:00.255416+0000","last_clean":"2023-06-10T18:48:00.255416+0000","last_became_active":"2023-06-10T18:47:59.267603+0000","last_became_peered":"2023-06-10T18:47:59.267603+0000","last_unstale":"2023-06-10T18:48:00.255416+0000","last_undegraded":"2023-06-10T18:48:00.255416+0000","last_fullsized":"2023-06-10T18:48:00.255416+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:20.818818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401641+0000","last_change":"2023-06-10T18:47:58.868802+0000","last_active":"2023-06-10T18:48:00.401641+0000","last_peered":"2023-06-10T18:48:00.401641+0000","last_clean":"2023-06-10T18:48:00.401641+0000","last_became_active":"2023-06-10T18:47:58.868529+0000","last_became_peered":"2023-06-10T18:47:58.868529+0000","last_unstale":"2023-06-10T18:48:00.401641+0000","last_undegraded":"2023-06-10T18:48:00.401641+0000","last_fullsized":"2023-06-10T18:48:00.401641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:04:56.547758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254028+0000","last_change":"2023-06-10T18:47:59.266772+0000","last_active":"2023-06-10T18:48:00.254028+0000","last_peered":"2023-06-10T18:48:00.254028+0000","last_clean":"2023-06-10T18:48:00.254028+0000","last_became_active":"2023-06-10T18:47:59.266544+0000","last_became_peered":"2023-06-10T18:47:59.266544+0000","last_unstale":"2023-06-10T18:48:00.254028+0000","last_undegraded":"2023-06-10T18:48:00.254028+0000","last_fullsized":"2023-06-10T18:48:00.254028+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255033+0000","last_change":"2023-06-10T18:47:58.861723+0000","last_active":"2023-06-10T18:48:00.255033+0000","last_peered":"2023-06-10T18:48:00.255033+0000","last_clean":"2023-06-10T18:48:00.255033+0000","last_became_active":"2023-06-10T18:47:58.861352+0000","last_became_peered":"2023-06-10T18:47:58.861352+0000","last_unstale":"2023-06-10T18:48:00.255033+0000","last_undegraded":"2023-06-10T18:48:00.255033+0000","last_fullsized":"2023-06-10T18:48:00.255033+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:15.931908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256436+0000","last_change":"2023-06-10T18:47:59.260410+0000","last_active":"2023-06-10T18:48:00.256436+0000","last_peered":"2023-06-10T18:48:00.256436+0000","last_clean":"2023-06-10T18:48:00.256436+0000","last_became_active":"2023-06-10T18:47:59.260128+0000","last_became_peered":"2023-06-10T18:47:59.260128+0000","last_unstale":"2023-06-10T18:48:00.256436+0000","last_undegraded":"2023-06-10T18:48:00.256436+0000","last_fullsized":"2023-06-10T18:48:00.256436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:28:22.113139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402220+0000","last_change":"2023-06-10T18:47:58.860628+0000","last_active":"2023-06-10T18:48:00.402220+0000","last_peered":"2023-06-10T18:48:00.402220+0000","last_clean":"2023-06-10T18:48:00.402220+0000","last_became_active":"2023-06-10T18:47:58.860160+0000","last_became_peered":"2023-06-10T18:47:58.860160+0000","last_unstale":"2023-06-10T18:48:00.402220+0000","last_undegraded":"2023-06-10T18:48:00.402220+0000","last_fullsized":"2023-06-10T18:48:00.402220+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:00.601382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402465+0000","last_change":"2023-06-10T18:47:59.259859+0000","last_active":"2023-06-10T18:48:00.402465+0000","last_peered":"2023-06-10T18:48:00.402465+0000","last_clean":"2023-06-10T18:48:00.402465+0000","last_became_active":"2023-06-10T18:47:59.259589+0000","last_became_peered":"2023-06-10T18:47:59.259589+0000","last_unstale":"2023-06-10T18:48:00.402465+0000","last_undegraded":"2023-06-10T18:48:00.402465+0000","last_fullsized":"2023-06-10T18:48:00.402465+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:06.146139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254794+0000","last_change":"2023-06-10T18:47:58.865983+0000","last_active":"2023-06-10T18:48:00.254794+0000","last_peered":"2023-06-10T18:48:00.254794+0000","last_clean":"2023-06-10T18:48:00.254794+0000","last_became_active":"2023-06-10T18:47:58.865626+0000","last_became_peered":"2023-06-10T18:47:58.865626+0000","last_unstale":"2023-06-10T18:48:00.254794+0000","last_undegraded":"2023-06-10T18:48:00.254794+0000","last_fullsized":"2023-06-10T18:48:00.254794+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:34:53.814935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400914+0000","last_change":"2023-06-10T18:47:59.263747+0000","last_active":"2023-06-10T18:48:00.400914+0000","last_peered":"2023-06-10T18:48:00.400914+0000","last_clean":"2023-06-10T18:48:00.400914+0000","last_became_active":"2023-06-10T18:47:59.263244+0000","last_became_peered":"2023-06-10T18:47:59.263244+0000","last_unstale":"2023-06-10T18:48:00.400914+0000","last_undegraded":"2023-06-10T18:48:00.400914+0000","last_fullsized":"2023-06-10T18:48:00.400914+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:03.042115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402222+0000","last_change":"2023-06-10T18:47:58.861989+0000","last_active":"2023-06-10T18:48:00.402222+0000","last_peered":"2023-06-10T18:48:00.402222+0000","last_clean":"2023-06-10T18:48:00.402222+0000","last_became_active":"2023-06-10T18:47:58.861702+0000","last_became_peered":"2023-06-10T18:47:58.861702+0000","last_unstale":"2023-06-10T18:48:00.402222+0000","last_undegraded":"2023-06-10T18:48:00.402222+0000","last_fullsized":"2023-06-10T18:48:00.402222+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:49.751137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402381+0000","last_change":"2023-06-10T18:47:59.261189+0000","last_active":"2023-06-10T18:48:00.402381+0000","last_peered":"2023-06-10T18:48:00.402381+0000","last_clean":"2023-06-10T18:48:00.402381+0000","last_became_active":"2023-06-10T18:47:59.260920+0000","last_became_peered":"2023-06-10T18:47:59.260920+0000","last_unstale":"2023-06-10T18:48:00.402381+0000","last_undegraded":"2023-06-10T18:48:00.402381+0000","last_fullsized":"2023-06-10T18:48:00.402381+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:41:27.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256456+0000","last_change":"2023-06-10T18:47:58.861104+0000","last_active":"2023-06-10T18:48:00.256456+0000","last_peered":"2023-06-10T18:48:00.256456+0000","last_clean":"2023-06-10T18:48:00.256456+0000","last_became_active":"2023-06-10T18:47:58.860789+0000","last_became_peered":"2023-06-10T18:47:58.860789+0000","last_unstale":"2023-06-10T18:48:00.256456+0000","last_undegraded":"2023-06-10T18:48:00.256456+0000","last_fullsized":"2023-06-10T18:48:00.256456+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:33:50.813268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401409+0000","last_change":"2023-06-10T18:47:59.263588+0000","last_active":"2023-06-10T18:48:00.401409+0000","last_peered":"2023-06-10T18:48:00.401409+0000","last_clean":"2023-06-10T18:48:00.401409+0000","last_became_active":"2023-06-10T18:47:59.263147+0000","last_became_peered":"2023-06-10T18:47:59.263147+0000","last_unstale":"2023-06-10T18:48:00.401409+0000","last_undegraded":"2023-06-10T18:48:00.401409+0000","last_fullsized":"2023-06-10T18:48:00.401409+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:40.401965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254615+0000","last_change":"2023-06-10T18:47:58.865915+0000","last_active":"2023-06-10T18:48:00.254615+0000","last_peered":"2023-06-10T18:48:00.254615+0000","last_clean":"2023-06-10T18:48:00.254615+0000","last_became_active":"2023-06-10T18:47:58.865645+0000","last_became_peered":"2023-06-10T18:47:58.865645+0000","last_unstale":"2023-06-10T18:48:00.254615+0000","last_undegraded":"2023-06-10T18:48:00.254615+0000","last_fullsized":"2023-06-10T18:48:00.254615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:14.661882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401569+0000","last_change":"2023-06-10T18:47:59.262624+0000","last_active":"2023-06-10T18:48:00.401569+0000","last_peered":"2023-06-10T18:48:00.401569+0000","last_clean":"2023-06-10T18:48:00.401569+0000","last_became_active":"2023-06-10T18:47:59.261891+0000","last_became_peered":"2023-06-10T18:47:59.261891+0000","last_unstale":"2023-06-10T18:48:00.401569+0000","last_undegraded":"2023-06-10T18:48:00.401569+0000","last_fullsized":"2023-06-10T18:48:00.401569+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:43:56.816497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255592+0000","last_change":"2023-06-10T18:47:58.862528+0000","last_active":"2023-06-10T18:48:00.255592+0000","last_peered":"2023-06-10T18:48:00.255592+0000","last_clean":"2023-06-10T18:48:00.255592+0000","last_became_active":"2023-06-10T18:47:58.862241+0000","last_became_peered":"2023-06-10T18:47:58.862241+0000","last_unstale":"2023-06-10T18:48:00.255592+0000","last_undegraded":"2023-06-10T18:48:00.255592+0000","last_fullsized":"2023-06-10T18:48:00.255592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:17:41.370481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254985+0000","last_change":"2023-06-10T18:47:59.255838+0000","last_active":"2023-06-10T18:48:00.254985+0000","last_peered":"2023-06-10T18:48:00.254985+0000","last_clean":"2023-06-10T18:48:00.254985+0000","last_became_active":"2023-06-10T18:47:59.255506+0000","last_became_peered":"2023-06-10T18:47:59.255506+0000","last_unstale":"2023-06-10T18:48:00.254985+0000","last_undegraded":"2023-06-10T18:48:00.254985+0000","last_fullsized":"2023-06-10T18:48:00.254985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:21.845649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401793+0000","last_change":"2023-06-10T18:47:58.869572+0000","last_active":"2023-06-10T18:48:00.401793+0000","last_peered":"2023-06-10T18:48:00.401793+0000","last_clean":"2023-06-10T18:48:00.401793+0000","last_became_active":"2023-06-10T18:47:58.869238+0000","last_became_peered":"2023-06-10T18:47:58.869238+0000","last_unstale":"2023-06-10T18:48:00.401793+0000","last_undegraded":"2023-06-10T18:48:00.401793+0000","last_fullsized":"2023-06-10T18:48:00.401793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:03:26.529546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402100+0000","last_change":"2023-06-10T18:47:59.258884+0000","last_active":"2023-06-10T18:48:00.402100+0000","last_peered":"2023-06-10T18:48:00.402100+0000","last_clean":"2023-06-10T18:48:00.402100+0000","last_became_active":"2023-06-10T18:47:59.258599+0000","last_became_peered":"2023-06-10T18:47:59.258599+0000","last_unstale":"2023-06-10T18:48:00.402100+0000","last_undegraded":"2023-06-10T18:48:00.402100+0000","last_fullsized":"2023-06-10T18:48:00.402100+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:43:32.660038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401236+0000","last_change":"2023-06-10T18:47:58.867385+0000","last_active":"2023-06-10T18:48:00.401236+0000","last_peered":"2023-06-10T18:48:00.401236+0000","last_clean":"2023-06-10T18:48:00.401236+0000","last_became_active":"2023-06-10T18:47:58.867085+0000","last_became_peered":"2023-06-10T18:47:58.867085+0000","last_unstale":"2023-06-10T18:48:00.401236+0000","last_undegraded":"2023-06-10T18:48:00.401236+0000","last_fullsized":"2023-06-10T18:48:00.401236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:31.788286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402159+0000","last_change":"2023-06-10T18:47:59.271202+0000","last_active":"2023-06-10T18:48:00.402159+0000","last_peered":"2023-06-10T18:48:00.402159+0000","last_clean":"2023-06-10T18:48:00.402159+0000","last_became_active":"2023-06-10T18:47:59.270921+0000","last_became_peered":"2023-06-10T18:47:59.270921+0000","last_unstale":"2023-06-10T18:48:00.402159+0000","last_undegraded":"2023-06-10T18:48:00.402159+0000","last_fullsized":"2023-06-10T18:48:00.402159+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:35:31.848299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401347+0000","last_change":"2023-06-10T18:47:58.860453+0000","last_active":"2023-06-10T18:48:00.401347+0000","last_peered":"2023-06-10T18:48:00.401347+0000","last_clean":"2023-06-10T18:48:00.401347+0000","last_became_active":"2023-06-10T18:47:58.860184+0000","last_became_peered":"2023-06-10T18:47:58.860184+0000","last_unstale":"2023-06-10T18:48:00.401347+0000","last_undegraded":"2023-06-10T18:48:00.401347+0000","last_fullsized":"2023-06-10T18:48:00.401347+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:59:08.710821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401986+0000","last_change":"2023-06-10T18:47:59.262671+0000","last_active":"2023-06-10T18:48:00.401986+0000","last_peered":"2023-06-10T18:48:00.401986+0000","last_clean":"2023-06-10T18:48:00.401986+0000","last_became_active":"2023-06-10T18:47:59.262399+0000","last_became_peered":"2023-06-10T18:47:59.262399+0000","last_unstale":"2023-06-10T18:48:00.401986+0000","last_undegraded":"2023-06-10T18:48:00.401986+0000","last_fullsized":"2023-06-10T18:48:00.401986+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:00.483660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254670+0000","last_change":"2023-06-10T18:47:58.864550+0000","last_active":"2023-06-10T18:48:00.254670+0000","last_peered":"2023-06-10T18:48:00.254670+0000","last_clean":"2023-06-10T18:48:00.254670+0000","last_became_active":"2023-06-10T18:47:58.864301+0000","last_became_peered":"2023-06-10T18:47:58.864301+0000","last_unstale":"2023-06-10T18:48:00.254670+0000","last_undegraded":"2023-06-10T18:48:00.254670+0000","last_fullsized":"2023-06-10T18:48:00.254670+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:15.931908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401051+0000","last_change":"2023-06-10T18:47:59.263553+0000","last_active":"2023-06-10T18:48:00.401051+0000","last_peered":"2023-06-10T18:48:00.401051+0000","last_clean":"2023-06-10T18:48:00.401051+0000","last_became_active":"2023-06-10T18:47:59.262877+0000","last_became_peered":"2023-06-10T18:47:59.262877+0000","last_unstale":"2023-06-10T18:48:00.401051+0000","last_undegraded":"2023-06-10T18:48:00.401051+0000","last_fullsized":"2023-06-10T18:48:00.401051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:48.210843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255706+0000","last_change":"2023-06-10T18:47:58.862717+0000","last_active":"2023-06-10T18:48:00.255706+0000","last_peered":"2023-06-10T18:48:00.255706+0000","last_clean":"2023-06-10T18:48:00.255706+0000","last_became_active":"2023-06-10T18:47:58.862331+0000","last_became_peered":"2023-06-10T18:47:58.862331+0000","last_unstale":"2023-06-10T18:48:00.255706+0000","last_undegraded":"2023-06-10T18:48:00.255706+0000","last_fullsized":"2023-06-10T18:48:00.255706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:39.043042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254224+0000","last_change":"2023-06-10T18:47:59.262842+0000","last_active":"2023-06-10T18:48:00.254224+0000","last_peered":"2023-06-10T18:48:00.254224+0000","last_clean":"2023-06-10T18:48:00.254224+0000","last_became_active":"2023-06-10T18:47:59.262588+0000","last_became_peered":"2023-06-10T18:47:59.262588+0000","last_unstale":"2023-06-10T18:48:00.254224+0000","last_undegraded":"2023-06-10T18:48:00.254224+0000","last_fullsized":"2023-06-10T18:48:00.254224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:50.987450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401961+0000","last_change":"2023-06-10T18:47:58.866633+0000","last_active":"2023-06-10T18:48:00.401961+0000","last_peered":"2023-06-10T18:48:00.401961+0000","last_clean":"2023-06-10T18:48:00.401961+0000","last_became_active":"2023-06-10T18:47:58.866369+0000","last_became_peered":"2023-06-10T18:47:58.866369+0000","last_unstale":"2023-06-10T18:48:00.401961+0000","last_undegraded":"2023-06-10T18:48:00.401961+0000","last_fullsized":"2023-06-10T18:48:00.401961+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:49:38.697076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254547+0000","last_change":"2023-06-10T18:47:59.254653+0000","last_active":"2023-06-10T18:48:00.254547+0000","last_peered":"2023-06-10T18:48:00.254547+0000","last_clean":"2023-06-10T18:48:00.254547+0000","last_became_active":"2023-06-10T18:47:59.254400+0000","last_became_peered":"2023-06-10T18:47:59.254400+0000","last_unstale":"2023-06-10T18:48:00.254547+0000","last_undegraded":"2023-06-10T18:48:00.254547+0000","last_fullsized":"2023-06-10T18:48:00.254547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:10:22.206584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401027+0000","last_change":"2023-06-10T18:47:58.867265+0000","last_active":"2023-06-10T18:48:00.401027+0000","last_peered":"2023-06-10T18:48:00.401027+0000","last_clean":"2023-06-10T18:48:00.401027+0000","last_became_active":"2023-06-10T18:47:58.866983+0000","last_became_peered":"2023-06-10T18:47:58.866983+0000","last_unstale":"2023-06-10T18:48:00.401027+0000","last_undegraded":"2023-06-10T18:48:00.401027+0000","last_fullsized":"2023-06-10T18:48:00.401027+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:51:57.937926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401720+0000","last_change":"2023-06-10T18:47:59.260756+0000","last_active":"2023-06-10T18:48:00.401720+0000","last_peered":"2023-06-10T18:48:00.401720+0000","last_clean":"2023-06-10T18:48:00.401720+0000","last_became_active":"2023-06-10T18:47:59.259768+0000","last_became_peered":"2023-06-10T18:47:59.259768+0000","last_unstale":"2023-06-10T18:48:00.401720+0000","last_undegraded":"2023-06-10T18:48:00.401720+0000","last_fullsized":"2023-06-10T18:48:00.401720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:06.010795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255999+0000","last_change":"2023-06-10T18:47:58.861747+0000","last_active":"2023-06-10T18:48:00.255999+0000","last_peered":"2023-06-10T18:48:00.255999+0000","last_clean":"2023-06-10T18:48:00.255999+0000","last_became_active":"2023-06-10T18:47:58.861368+0000","last_became_peered":"2023-06-10T18:47:58.861368+0000","last_unstale":"2023-06-10T18:48:00.255999+0000","last_undegraded":"2023-06-10T18:48:00.255999+0000","last_fullsized":"2023-06-10T18:48:00.255999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:21.187245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400720+0000","last_change":"2023-06-10T18:47:59.262613+0000","last_active":"2023-06-10T18:48:00.400720+0000","last_peered":"2023-06-10T18:48:00.400720+0000","last_clean":"2023-06-10T18:48:00.400720+0000","last_became_active":"2023-06-10T18:47:59.262351+0000","last_became_peered":"2023-06-10T18:47:59.262351+0000","last_unstale":"2023-06-10T18:48:00.400720+0000","last_undegraded":"2023-06-10T18:48:00.400720+0000","last_fullsized":"2023-06-10T18:48:00.400720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:49.262983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400913+0000","last_change":"2023-06-10T18:47:58.862597+0000","last_active":"2023-06-10T18:48:00.400913+0000","last_peered":"2023-06-10T18:48:00.400913+0000","last_clean":"2023-06-10T18:48:00.400913+0000","last_became_active":"2023-06-10T18:47:58.862345+0000","last_became_peered":"2023-06-10T18:47:58.862345+0000","last_unstale":"2023-06-10T18:48:00.400913+0000","last_undegraded":"2023-06-10T18:48:00.400913+0000","last_fullsized":"2023-06-10T18:48:00.400913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:58:26.726084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254162+0000","last_change":"2023-06-10T18:47:59.266712+0000","last_active":"2023-06-10T18:48:00.254162+0000","last_peered":"2023-06-10T18:48:00.254162+0000","last_clean":"2023-06-10T18:48:00.254162+0000","last_became_active":"2023-06-10T18:47:59.266549+0000","last_became_peered":"2023-06-10T18:47:59.266549+0000","last_unstale":"2023-06-10T18:48:00.254162+0000","last_undegraded":"2023-06-10T18:48:00.254162+0000","last_fullsized":"2023-06-10T18:48:00.254162+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:22.191847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254102+0000","last_change":"2023-06-10T18:47:58.867264+0000","last_active":"2023-06-10T18:48:00.254102+0000","last_peered":"2023-06-10T18:48:00.254102+0000","last_clean":"2023-06-10T18:48:00.254102+0000","last_became_active":"2023-06-10T18:47:58.867016+0000","last_became_peered":"2023-06-10T18:47:58.867016+0000","last_unstale":"2023-06-10T18:48:00.254102+0000","last_undegraded":"2023-06-10T18:48:00.254102+0000","last_fullsized":"2023-06-10T18:48:00.254102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:02.376382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402347+0000","last_change":"2023-06-10T18:47:59.264466+0000","last_active":"2023-06-10T18:48:00.402347+0000","last_peered":"2023-06-10T18:48:00.402347+0000","last_clean":"2023-06-10T18:48:00.402347+0000","last_became_active":"2023-06-10T18:47:59.264213+0000","last_became_peered":"2023-06-10T18:47:59.264213+0000","last_unstale":"2023-06-10T18:48:00.402347+0000","last_undegraded":"2023-06-10T18:48:00.402347+0000","last_fullsized":"2023-06-10T18:48:00.402347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:15.666485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254900+0000","last_change":"2023-06-10T18:47:58.866173+0000","last_active":"2023-06-10T18:48:00.254900+0000","last_peered":"2023-06-10T18:48:00.254900+0000","last_clean":"2023-06-10T18:48:00.254900+0000","last_became_active":"2023-06-10T18:47:58.865625+0000","last_became_peered":"2023-06-10T18:47:58.865625+0000","last_unstale":"2023-06-10T18:48:00.254900+0000","last_undegraded":"2023-06-10T18:48:00.254900+0000","last_fullsized":"2023-06-10T18:48:00.254900+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:41.943041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401097+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:48:00.401097+0000","last_peered":"2023-06-10T18:48:00.401097+0000","last_clean":"2023-06-10T18:48:00.401097+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:48:00.401097+0000","last_undegraded":"2023-06-10T18:48:00.401097+0000","last_fullsized":"2023-06-10T18:48:00.401097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:55:59.087724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253917+0000","last_change":"2023-06-10T18:47:59.253010+0000","last_active":"2023-06-10T18:48:00.253917+0000","last_peered":"2023-06-10T18:48:00.253917+0000","last_clean":"2023-06-10T18:48:00.253917+0000","last_became_active":"2023-06-10T18:47:59.252724+0000","last_became_peered":"2023-06-10T18:47:59.252724+0000","last_unstale":"2023-06-10T18:48:00.253917+0000","last_undegraded":"2023-06-10T18:48:00.253917+0000","last_fullsized":"2023-06-10T18:48:00.253917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:48.345875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401187+0000","last_change":"2023-06-10T18:47:58.869350+0000","last_active":"2023-06-10T18:48:00.401187+0000","last_peered":"2023-06-10T18:48:00.401187+0000","last_clean":"2023-06-10T18:48:00.401187+0000","last_became_active":"2023-06-10T18:47:58.868873+0000","last_became_peered":"2023-06-10T18:47:58.868873+0000","last_unstale":"2023-06-10T18:48:00.401187+0000","last_undegraded":"2023-06-10T18:48:00.401187+0000","last_fullsized":"2023-06-10T18:48:00.401187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:10:21.201982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401637+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:48:00.401637+0000","last_peered":"2023-06-10T18:48:00.401637+0000","last_clean":"2023-06-10T18:48:00.401637+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:48:00.401637+0000","last_undegraded":"2023-06-10T18:48:00.401637+0000","last_fullsized":"2023-06-10T18:48:00.401637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:42.280363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401557+0000","last_change":"2023-06-10T18:47:59.263421+0000","last_active":"2023-06-10T18:48:00.401557+0000","last_peered":"2023-06-10T18:48:00.401557+0000","last_clean":"2023-06-10T18:48:00.401557+0000","last_became_active":"2023-06-10T18:47:59.262823+0000","last_became_peered":"2023-06-10T18:47:59.262823+0000","last_unstale":"2023-06-10T18:48:00.401557+0000","last_undegraded":"2023-06-10T18:48:00.401557+0000","last_fullsized":"2023-06-10T18:48:00.401557+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:34:54.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254433+0000","last_change":"2023-06-10T18:47:58.860988+0000","last_active":"2023-06-10T18:48:00.254433+0000","last_peered":"2023-06-10T18:48:00.254433+0000","last_clean":"2023-06-10T18:48:00.254433+0000","last_became_active":"2023-06-10T18:47:58.860748+0000","last_became_peered":"2023-06-10T18:47:58.860748+0000","last_unstale":"2023-06-10T18:48:00.254433+0000","last_undegraded":"2023-06-10T18:48:00.254433+0000","last_fullsized":"2023-06-10T18:48:00.254433+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:18:15.896296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400418+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:48:00.400418+0000","last_peered":"2023-06-10T18:48:00.400418+0000","last_clean":"2023-06-10T18:48:00.400418+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:48:00.400418+0000","last_undegraded":"2023-06-10T18:48:00.400418+0000","last_fullsized":"2023-06-10T18:48:00.400418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:56.642449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253983+0000","last_change":"2023-06-10T18:47:59.262130+0000","last_active":"2023-06-10T18:48:00.253983+0000","last_peered":"2023-06-10T18:48:00.253983+0000","last_clean":"2023-06-10T18:48:00.253983+0000","last_became_active":"2023-06-10T18:47:59.261897+0000","last_became_peered":"2023-06-10T18:47:59.261897+0000","last_unstale":"2023-06-10T18:48:00.253983+0000","last_undegraded":"2023-06-10T18:48:00.253983+0000","last_fullsized":"2023-06-10T18:48:00.253983+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:21.845649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255259+0000","last_change":"2023-06-10T18:47:58.870783+0000","last_active":"2023-06-10T18:48:00.255259+0000","last_peered":"2023-06-10T18:48:00.255259+0000","last_clean":"2023-06-10T18:48:00.255259+0000","last_became_active":"2023-06-10T18:47:58.870474+0000","last_became_peered":"2023-06-10T18:47:58.870474+0000","last_unstale":"2023-06-10T18:48:00.255259+0000","last_undegraded":"2023-06-10T18:48:00.255259+0000","last_fullsized":"2023-06-10T18:48:00.255259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:30:57.976057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401900+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:48:00.401900+0000","last_peered":"2023-06-10T18:48:00.401900+0000","last_clean":"2023-06-10T18:48:00.401900+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:48:00.401900+0000","last_undegraded":"2023-06-10T18:48:00.401900+0000","last_fullsized":"2023-06-10T18:48:00.401900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:00.495694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401295+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:48:00.401295+0000","last_peered":"2023-06-10T18:48:00.401295+0000","last_clean":"2023-06-10T18:48:00.401295+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:48:00.401295+0000","last_undegraded":"2023-06-10T18:48:00.401295+0000","last_fullsized":"2023-06-10T18:48:00.401295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:59:25.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254425+0000","last_change":"2023-06-10T18:47:59.260160+0000","last_active":"2023-06-10T18:48:00.254425+0000","last_peered":"2023-06-10T18:48:00.254425+0000","last_clean":"2023-06-10T18:48:00.254425+0000","last_became_active":"2023-06-10T18:47:59.259907+0000","last_became_peered":"2023-06-10T18:47:59.259907+0000","last_unstale":"2023-06-10T18:48:00.254425+0000","last_undegraded":"2023-06-10T18:48:00.254425+0000","last_fullsized":"2023-06-10T18:48:00.254425+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:31.431220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401640+0000","last_change":"2023-06-10T18:47:58.860831+0000","last_active":"2023-06-10T18:48:00.401640+0000","last_peered":"2023-06-10T18:48:00.401640+0000","last_clean":"2023-06-10T18:48:00.401640+0000","last_became_active":"2023-06-10T18:47:58.860555+0000","last_became_peered":"2023-06-10T18:47:58.860555+0000","last_unstale":"2023-06-10T18:48:00.401640+0000","last_undegraded":"2023-06-10T18:48:00.401640+0000","last_fullsized":"2023-06-10T18:48:00.401640+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:36:46.516888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401349+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:48:00.401349+0000","last_peered":"2023-06-10T18:48:00.401349+0000","last_clean":"2023-06-10T18:48:00.401349+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:48:00.401349+0000","last_undegraded":"2023-06-10T18:48:00.401349+0000","last_fullsized":"2023-06-10T18:48:00.401349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:22:32.487446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401861+0000","last_change":"2023-06-10T18:47:59.258960+0000","last_active":"2023-06-10T18:48:00.401861+0000","last_peered":"2023-06-10T18:48:00.401861+0000","last_clean":"2023-06-10T18:48:00.401861+0000","last_became_active":"2023-06-10T18:47:59.258637+0000","last_became_peered":"2023-06-10T18:47:59.258637+0000","last_unstale":"2023-06-10T18:48:00.401861+0000","last_undegraded":"2023-06-10T18:48:00.401861+0000","last_fullsized":"2023-06-10T18:48:00.401861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:35:31.848299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254359+0000","last_change":"2023-06-10T18:47:58.866007+0000","last_active":"2023-06-10T18:48:00.254359+0000","last_peered":"2023-06-10T18:48:00.254359+0000","last_clean":"2023-06-10T18:48:00.254359+0000","last_became_active":"2023-06-10T18:47:58.865696+0000","last_became_peered":"2023-06-10T18:47:58.865696+0000","last_unstale":"2023-06-10T18:48:00.254359+0000","last_undegraded":"2023-06-10T18:48:00.254359+0000","last_fullsized":"2023-06-10T18:48:00.254359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:39.011202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402472+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:48:00.402472+0000","last_peered":"2023-06-10T18:48:00.402472+0000","last_clean":"2023-06-10T18:48:00.402472+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:48:00.402472+0000","last_undegraded":"2023-06-10T18:48:00.402472+0000","last_fullsized":"2023-06-10T18:48:00.402472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:17.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402610+0000","last_change":"2023-06-10T18:47:59.260870+0000","last_active":"2023-06-10T18:48:00.402610+0000","last_peered":"2023-06-10T18:48:00.402610+0000","last_clean":"2023-06-10T18:48:00.402610+0000","last_became_active":"2023-06-10T18:47:59.260577+0000","last_became_peered":"2023-06-10T18:47:59.260577+0000","last_unstale":"2023-06-10T18:48:00.402610+0000","last_undegraded":"2023-06-10T18:48:00.402610+0000","last_fullsized":"2023-06-10T18:48:00.402610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:33.650753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254951+0000","last_change":"2023-06-10T18:47:58.860101+0000","last_active":"2023-06-10T18:48:00.254951+0000","last_peered":"2023-06-10T18:48:00.254951+0000","last_clean":"2023-06-10T18:48:00.254951+0000","last_became_active":"2023-06-10T18:47:58.859822+0000","last_became_peered":"2023-06-10T18:47:58.859822+0000","last_unstale":"2023-06-10T18:48:00.254951+0000","last_undegraded":"2023-06-10T18:48:00.254951+0000","last_fullsized":"2023-06-10T18:48:00.254951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:21.187245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401081+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:48:00.401081+0000","last_peered":"2023-06-10T18:48:00.401081+0000","last_clean":"2023-06-10T18:48:00.401081+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:48:00.401081+0000","last_undegraded":"2023-06-10T18:48:00.401081+0000","last_fullsized":"2023-06-10T18:48:00.401081+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:32:59.634148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254334+0000","last_change":"2023-06-10T18:47:59.257647+0000","last_active":"2023-06-10T18:48:00.254334+0000","last_peered":"2023-06-10T18:48:00.254334+0000","last_clean":"2023-06-10T18:48:00.254334+0000","last_became_active":"2023-06-10T18:47:59.257322+0000","last_became_peered":"2023-06-10T18:47:59.257322+0000","last_unstale":"2023-06-10T18:48:00.254334+0000","last_undegraded":"2023-06-10T18:48:00.254334+0000","last_fullsized":"2023-06-10T18:48:00.254334+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:33:51.817871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402214+0000","last_change":"2023-06-10T18:47:58.869544+0000","last_active":"2023-06-10T18:48:00.402214+0000","last_peered":"2023-06-10T18:48:00.402214+0000","last_clean":"2023-06-10T18:48:00.402214+0000","last_became_active":"2023-06-10T18:47:58.869249+0000","last_became_peered":"2023-06-10T18:47:58.869249+0000","last_unstale":"2023-06-10T18:48:00.402214+0000","last_undegraded":"2023-06-10T18:48:00.402214+0000","last_fullsized":"2023-06-10T18:48:00.402214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:48.539356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255771+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:48:00.255771+0000","last_peered":"2023-06-10T18:48:00.255771+0000","last_clean":"2023-06-10T18:48:00.255771+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:48:00.255771+0000","last_undegraded":"2023-06-10T18:48:00.255771+0000","last_fullsized":"2023-06-10T18:48:00.255771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:56:26.622598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1269760,"data_stored":899557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.39100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.40300000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.70899999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.48199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73499999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.443}]},{"osd":2,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48799999999999999,"15min":0.47299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67200000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.625}]},{"osd":4,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47499999999999998,"15min":0.46700000000000003},"min":{"1min":0.32500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63900000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.752,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67500000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54900000000000004,"15min":0.55000000000000004},"min":{"1min":0.373,"5min":0.311,"15min":0.311},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":663552,"data_stored":308010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.43099999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.75700000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52000000000000002,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.752},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.72399999999999998,"15min":0.89000000000000001},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65400000000000003,"15min":0.77400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.316,"15min":0.316},"max":{"1min":0.71299999999999997,"5min":1.252,"15min":1.252},"last":0.621}]},{"osd":3,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.69099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.52300000000000002},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.40899999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.78600000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":667648,"data_stored":308100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.46300000000000002,"5min":0.753,"15min":0.753},"last":0.246},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.33000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56599999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.309}]},{"osd":3,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55900000000000005,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.62},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86799999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":1.008,"15min":1.008},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.43099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77100000000000002,"5min":0.876,"15min":0.876},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1273856,"data_stored":900524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34699999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.625,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.36099999999999999},"min":{"1min":0.30199999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.54300000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38800000000000001},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65500000000000003,"5min":1.405,"15min":1.405},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39600000000000002},"min":{"1min":0.33000000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.505,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.27200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60499999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56699999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.32600000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.371,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79900000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.495}]},{"osd":5,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85899999999999999,"5min":0.999,"15min":0.999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80600000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.623,"15min":0.63},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91400000000000003,"5min":1.363,"15min":1.363},"last":0.47899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89200000000000002,"5min":1.034,"15min":1.034},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.33000000000000002},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.23799999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.34799999999999998},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.35399999999999998,"15min":0.34300000000000003},"min":{"1min":0.33000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.29199999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.32100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.376}]},{"osd":4,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.47599999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.622},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.502,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.498}]},{"osd":5,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.52100000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74199999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.498,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77200000000000002,"5min":0.872,"15min":0.872},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.496,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.71299999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.55100000000000005,"5min":0.433,"15min":0.433},"max":{"1min":0.83199999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.51600000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.50800000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.53400000000000003,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.51700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.312,"5min":0.245,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":1.3,"15min":1.3},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.33100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58699999999999997,"5min":1.109,"15min":1.109},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.49199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56000000000000005,"5min":1.599,"15min":1.599},"last":0.378}]},{"osd":4,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.745,"5min":1.925,"15min":1.925},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.45400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81299999999999994,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.49299999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84299999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.72099999999999997,"15min":0.78000000000000003},"min":{"1min":0.43099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89900000000000002,"5min":3.802,"15min":3.802},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71899999999999997,"15min":0.78000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.622,"15min":0.628},"min":{"1min":0.46000000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88600000000000001,"5min":1.03,"15min":1.03},"last":0.55500000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94199999999999995,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85299999999999998,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.47199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51800000000000002,"15min":0.50700000000000001},"min":{"1min":0.40999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.40000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.44900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72199999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70499999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.47899999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.45800000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54400000000000004,"15min":0.53400000000000003},"min":{"1min":0.44900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.38800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70699999999999996,"5min":1.042,"15min":1.042},"last":0.38100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.069,"15min":1.069},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50800000000000001,"15min":0.51600000000000001},"min":{"1min":0.32500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65500000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53800000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77200000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53100000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.71999999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59199999999999997,"15min":0.60099999999999998},"min":{"1min":0.41899999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.58399999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67900000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.38300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65200000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42399999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.46400000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71299999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.42499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.436,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68100000000000005,"5min":0.745,"15min":0.745},"last":0.503},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54600000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67500000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.32900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77800000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":0.754},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:48:03.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:48:03.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.590+0000 7fd5f40da700 1 Processor -- start 2023-06-10T18:48:03.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.591+0000 7fd5f40da700 1 -- start start 2023-06-10T18:48:03.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.591+0000 7fd5f40da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5ec0cf100 0x7fd5ec06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.591+0000 7fd5f40da700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec06d290 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.591+0000 7fd5f40da700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec0ca2d0 con 0x7fd5ec06e4b0 2023-06-10T18:48:03.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f40da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec071640 con 0x7fd5ec0cf100 2023-06-10T18:48:03.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f1e76700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd5ec06e0e0 0x7fd5ec0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56420/0 (socket says 172.21.15.104:56420) 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f1e76700 1 -- 172.21.15.104:0/2779363151 learned_addr learned my addr 172.21.15.104:0/2779363151 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f2677700 1 --2- 172.21.15.104:0/2779363151 >> v2:172.21.15.114:3300/0 conn(0x7fd5ec0cf100 0x7fd5ec06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255822028 0 0) 0x7fd5ec06d290 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d80035b0 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077093300 0 0) 0x7fd5d80035b0 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 >> v2:172.21.15.114:3300/0 conn(0x7fd5ec0cf100 msgr2=0x7fd5ec06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 --2- 172.21.15.104:0/2779363151 >> v2:172.21.15.114:3300/0 conn(0x7fd5ec0cf100 0x7fd5ec06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 >> v1:172.21.15.114:6789/0 conn(0x7fd5ec06e4b0 legacy=0x7fd5ec0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec06d4c0 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd5dc0017f0 con 0x7fd5ec06e0e0 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 >> v1:172.21.15.104:6789/0 conn(0x7fd5ec06e0e0 legacy=0x7fd5ec0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 0x7fd5d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8004850 0x7fd5d8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5d8005200 0x7fd5d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec071640 con 0x7fd5d8003a00 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d800c110 con 0x7fd5d8004850 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d800c2d0 con 0x7fd5d8005200 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 0x7fd5d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f1675700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8004850 0x7fd5d8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.592+0000 7fd5f2677700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5d8005200 0x7fd5d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 -- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5d8005200 msgr2=0x7fd5d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5d8005200 0x7fd5d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 -- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8004850 msgr2=0x7fd5d8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8004850 0x7fd5d8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 -- 172.21.15.104:0/2779363151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d80035b0 con 0x7fd5d8003a00 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f2677700 1 -- 172.21.15.104:0/2779363151 reap_dead start 2023-06-10T18:48:03.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1675700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8004850 0x7fd5d8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 0x7fd5d8003e10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c240 tx=0x7fd5d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376a534fed5ec144 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc00cde0 con 0x7fd5d8003a00 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc007b80 con 0x7fd5d8003a00 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f0e74700 1 -- 172.21.15.104:0/2779363151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0072f0 con 0x7fd5d8003a00 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 -- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 msgr2=0x7fd5d8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 0x7fd5d8003e10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c240 tx=0x7fd5d8005880 comp rx=0 tx=0).stop 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 -- 172.21.15.104:0/2779363151 shutdown_connections 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 --2- 172.21.15.104:0/2779363151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5d8003a00 0x7fd5d8003e10 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 -- 172.21.15.104:0/2779363151 >> 172.21.15.104:0/2779363151 conn(0x7fd5ec06b8a0 msgr2=0x7fd5ec0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 -- 172.21.15.104:0/2779363151 shutdown_connections 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.593+0000 7fd5f40da700 1 -- 172.21.15.104:0/2779363151 wait complete. 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.595+0000 7fd5f40da700 1 Processor -- start 2023-06-10T18:48:03.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.595+0000 7fd5f40da700 1 -- start start 2023-06-10T18:48:03.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 0x7fd5ec14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5ec0c8b00 0x7fd5ec0c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c94b0 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c6640 con 0x7fd5d8003a00 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f40da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0b4860 con 0x7fd5ec0c8b00 2023-06-10T18:48:03.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58606/0 (socket says 172.21.15.104:58606) 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 -- 172.21.15.104:0/3644595750 learned_addr learned my addr 172.21.15.104:0/3644595750 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f1675700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5ec0c8b00 0x7fd5ec0c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 0x7fd5ec14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 -- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5ec0c8b00 msgr2=0x7fd5ec0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5ec0c8b00 0x7fd5ec0c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 -- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 msgr2=0x7fd5ec14cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 0x7fd5ec14cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f2677700 1 -- 172.21.15.104:0/3644595750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0b4b40 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.596+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 0x7fd5ec14cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5f2677700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0cf560 tx=0x7fd5e40111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400e070 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e4003cf0 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4003ea0 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec0c6890 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.597+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec0b4510 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.598+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec0c6db0 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.598+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd5e4011880 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.599+0000 7fd5e27fc700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 0x7fd5d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.599+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd5e4089870 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.599+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 0x7fd5d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.599+0000 7fd5f1e76700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 0x7fd5d006d190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc015040 tx=0x7fd5dc007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.603+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd5ec0c6db0 con 0x7fd5ec0b3b30 2023-06-10T18:48:03.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.785+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5ec06da80 con 0x7fd5d006ace0 2023-06-10T18:48:03.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.794+0000 7fd5e27fc700 1 -- 172.21.15.104:0/3644595750 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311620 (secure 0 0 0) 0x7fd5dc00cea0 con 0x7fd5d006ace0 2023-06-10T18:48:03.802 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:03.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 msgr2=0x7fd5d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 0x7fd5d006d190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc015040 tx=0x7fd5dc007000 comp rx=0 tx=0).stop 2023-06-10T18:48:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 msgr2=0x7fd5ec0b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0cf560 tx=0x7fd5e40111d0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 shutdown_connections 2023-06-10T18:48:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5d006ace0 0x7fd5d006d190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8003a00 0x7fd5ec14cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5ec0b3b30 0x7fd5ec0b3fd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 --2- 172.21.15.104:0/3644595750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5ec0c8b00 0x7fd5ec0c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 >> 172.21.15.104:0/3644595750 conn(0x7fd5ec06b8a0 msgr2=0x7fd5ec0caed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 shutdown_connections 2023-06-10T18:48:03.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.797+0000 7fd5f40da700 1 -- 172.21.15.104:0/3644595750 wait complete. 2023-06-10T18:48:03.808 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:48:03.814 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2023-06-10T18:48:03.414829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222040,"kb_used_data":6424,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754752680,"statfs":{"total":773094113280,"available":772866744320,"internally_reserved":0,"allocated":6578176,"data_stored":3653616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573325"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400978+0000","last_change":"2023-06-10T18:47:59.270757+0000","last_active":"2023-06-10T18:48:00.400978+0000","last_peered":"2023-06-10T18:48:00.400978+0000","last_clean":"2023-06-10T18:48:00.400978+0000","last_became_active":"2023-06-10T18:47:59.270515+0000","last_became_peered":"2023-06-10T18:47:59.270515+0000","last_unstale":"2023-06-10T18:48:00.400978+0000","last_undegraded":"2023-06-10T18:48:00.400978+0000","last_fullsized":"2023-06-10T18:48:00.400978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:06.126008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254848+0000","last_change":"2023-06-10T18:47:59.255348+0000","last_active":"2023-06-10T18:48:00.254848+0000","last_peered":"2023-06-10T18:48:00.254848+0000","last_clean":"2023-06-10T18:48:00.254848+0000","last_became_active":"2023-06-10T18:47:59.254464+0000","last_became_peered":"2023-06-10T18:47:59.254464+0000","last_unstale":"2023-06-10T18:48:00.254848+0000","last_undegraded":"2023-06-10T18:48:00.254848+0000","last_fullsized":"2023-06-10T18:48:00.254848+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:04:57.552360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401945+0000","last_change":"2023-06-10T18:47:59.258124+0000","last_active":"2023-06-10T18:48:00.401945+0000","last_peered":"2023-06-10T18:48:00.401945+0000","last_clean":"2023-06-10T18:48:00.401945+0000","last_became_active":"2023-06-10T18:47:59.257849+0000","last_became_peered":"2023-06-10T18:47:59.257849+0000","last_unstale":"2023-06-10T18:48:00.401945+0000","last_undegraded":"2023-06-10T18:48:00.401945+0000","last_fullsized":"2023-06-10T18:48:00.401945+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:46:38.601166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401646+0000","last_change":"2023-06-10T18:47:59.271245+0000","last_active":"2023-06-10T18:48:00.401646+0000","last_peered":"2023-06-10T18:48:00.401646+0000","last_clean":"2023-06-10T18:48:00.401646+0000","last_became_active":"2023-06-10T18:47:59.270943+0000","last_became_peered":"2023-06-10T18:47:59.270943+0000","last_unstale":"2023-06-10T18:48:00.401646+0000","last_undegraded":"2023-06-10T18:48:00.401646+0000","last_fullsized":"2023-06-10T18:48:00.401646+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:32.792889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400739+0000","last_change":"2023-06-10T18:47:59.270430+0000","last_active":"2023-06-10T18:48:00.400739+0000","last_peered":"2023-06-10T18:48:00.400739+0000","last_clean":"2023-06-10T18:48:00.400739+0000","last_became_active":"2023-06-10T18:47:59.270065+0000","last_became_peered":"2023-06-10T18:47:59.270065+0000","last_unstale":"2023-06-10T18:48:00.400739+0000","last_undegraded":"2023-06-10T18:48:00.400739+0000","last_fullsized":"2023-06-10T18:48:00.400739+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:24.430578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254008+0000","last_change":"2023-06-10T18:47:59.270812+0000","last_active":"2023-06-10T18:48:00.254008+0000","last_peered":"2023-06-10T18:48:00.254008+0000","last_clean":"2023-06-10T18:48:00.254008+0000","last_became_active":"2023-06-10T18:47:59.270515+0000","last_became_peered":"2023-06-10T18:47:59.270515+0000","last_unstale":"2023-06-10T18:48:00.254008+0000","last_undegraded":"2023-06-10T18:48:00.254008+0000","last_fullsized":"2023-06-10T18:48:00.254008+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:09.734260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255580+0000","last_change":"2023-06-10T18:47:59.261582+0000","last_active":"2023-06-10T18:48:00.255580+0000","last_peered":"2023-06-10T18:48:00.255580+0000","last_clean":"2023-06-10T18:48:00.255580+0000","last_became_active":"2023-06-10T18:47:59.261273+0000","last_became_peered":"2023-06-10T18:47:59.261273+0000","last_unstale":"2023-06-10T18:48:00.255580+0000","last_undegraded":"2023-06-10T18:48:00.255580+0000","last_fullsized":"2023-06-10T18:48:00.255580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253842+0000","last_change":"2023-06-10T18:47:59.256478+0000","last_active":"2023-06-10T18:48:00.253842+0000","last_peered":"2023-06-10T18:48:00.253842+0000","last_clean":"2023-06-10T18:48:00.253842+0000","last_became_active":"2023-06-10T18:47:59.256207+0000","last_became_peered":"2023-06-10T18:47:59.256207+0000","last_unstale":"2023-06-10T18:48:00.253842+0000","last_undegraded":"2023-06-10T18:48:00.253842+0000","last_fullsized":"2023-06-10T18:48:00.253842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:42.947644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256438+0000","last_change":"2023-06-10T18:47:59.258747+0000","last_active":"2023-06-10T18:48:00.256438+0000","last_peered":"2023-06-10T18:48:00.256438+0000","last_clean":"2023-06-10T18:48:00.256438+0000","last_became_active":"2023-06-10T18:47:59.258474+0000","last_became_peered":"2023-06-10T18:47:59.258474+0000","last_unstale":"2023-06-10T18:48:00.256438+0000","last_undegraded":"2023-06-10T18:48:00.256438+0000","last_fullsized":"2023-06-10T18:48:00.256438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:30.306301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401171+0000","last_change":"2023-06-10T18:47:59.260235+0000","last_active":"2023-06-10T18:48:00.401171+0000","last_peered":"2023-06-10T18:48:00.401171+0000","last_clean":"2023-06-10T18:48:00.401171+0000","last_became_active":"2023-06-10T18:47:59.259924+0000","last_became_peered":"2023-06-10T18:47:59.259924+0000","last_unstale":"2023-06-10T18:48:00.401171+0000","last_undegraded":"2023-06-10T18:48:00.401171+0000","last_fullsized":"2023-06-10T18:48:00.401171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:49:38.478392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401262+0000","last_change":"2023-06-10T18:47:59.256416+0000","last_active":"2023-06-10T18:48:00.401262+0000","last_peered":"2023-06-10T18:48:00.401262+0000","last_clean":"2023-06-10T18:48:00.401262+0000","last_became_active":"2023-06-10T18:47:59.256161+0000","last_became_peered":"2023-06-10T18:47:59.256161+0000","last_unstale":"2023-06-10T18:48:00.401262+0000","last_undegraded":"2023-06-10T18:48:00.401262+0000","last_fullsized":"2023-06-10T18:48:00.401262+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:20:38.752799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401828+0000","last_change":"2023-06-10T18:47:59.266083+0000","last_active":"2023-06-10T18:48:00.401828+0000","last_peered":"2023-06-10T18:48:00.401828+0000","last_clean":"2023-06-10T18:48:00.401828+0000","last_became_active":"2023-06-10T18:47:59.265817+0000","last_became_peered":"2023-06-10T18:47:59.265817+0000","last_unstale":"2023-06-10T18:48:00.401828+0000","last_undegraded":"2023-06-10T18:48:00.401828+0000","last_fullsized":"2023-06-10T18:48:00.401828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:26:28.808249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255792+0000","last_change":"2023-06-10T18:47:59.259652+0000","last_active":"2023-06-10T18:48:00.255792+0000","last_peered":"2023-06-10T18:48:00.255792+0000","last_clean":"2023-06-10T18:48:00.255792+0000","last_became_active":"2023-06-10T18:47:59.259396+0000","last_became_peered":"2023-06-10T18:47:59.259396+0000","last_unstale":"2023-06-10T18:48:00.255792+0000","last_undegraded":"2023-06-10T18:48:00.255792+0000","last_fullsized":"2023-06-10T18:48:00.255792+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:59:29.107540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256016+0000","last_change":"2023-06-10T18:47:59.270952+0000","last_active":"2023-06-10T18:48:00.256016+0000","last_peered":"2023-06-10T18:48:00.256016+0000","last_clean":"2023-06-10T18:48:00.256016+0000","last_became_active":"2023-06-10T18:47:59.270649+0000","last_became_peered":"2023-06-10T18:47:59.270649+0000","last_unstale":"2023-06-10T18:48:00.256016+0000","last_undegraded":"2023-06-10T18:48:00.256016+0000","last_fullsized":"2023-06-10T18:48:00.256016+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:08:40.378289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255813+0000","last_change":"2023-06-10T18:47:59.256978+0000","last_active":"2023-06-10T18:48:00.255813+0000","last_peered":"2023-06-10T18:48:00.255813+0000","last_clean":"2023-06-10T18:48:00.255813+0000","last_became_active":"2023-06-10T18:47:59.256675+0000","last_became_peered":"2023-06-10T18:47:59.256675+0000","last_unstale":"2023-06-10T18:48:00.255813+0000","last_undegraded":"2023-06-10T18:48:00.255813+0000","last_fullsized":"2023-06-10T18:48:00.255813+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:03.380984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254733+0000","last_change":"2023-06-10T18:47:59.259833+0000","last_active":"2023-06-10T18:48:00.254733+0000","last_peered":"2023-06-10T18:48:00.254733+0000","last_clean":"2023-06-10T18:48:00.254733+0000","last_became_active":"2023-06-10T18:47:59.259568+0000","last_became_peered":"2023-06-10T18:47:59.259568+0000","last_unstale":"2023-06-10T18:48:00.254733+0000","last_undegraded":"2023-06-10T18:48:00.254733+0000","last_fullsized":"2023-06-10T18:48:00.254733+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254225+0000","last_change":"2023-06-10T18:47:59.264346+0000","last_active":"2023-06-10T18:48:00.254225+0000","last_peered":"2023-06-10T18:48:00.254225+0000","last_clean":"2023-06-10T18:48:00.254225+0000","last_became_active":"2023-06-10T18:47:59.264063+0000","last_became_peered":"2023-06-10T18:47:59.264063+0000","last_unstale":"2023-06-10T18:48:00.254225+0000","last_undegraded":"2023-06-10T18:48:00.254225+0000","last_fullsized":"2023-06-10T18:48:00.254225+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:17.078369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401707+0000","last_change":"2023-06-10T18:47:59.253060+0000","last_active":"2023-06-10T18:48:00.401707+0000","last_peered":"2023-06-10T18:48:00.401707+0000","last_clean":"2023-06-10T18:48:00.401707+0000","last_became_active":"2023-06-10T18:47:59.252812+0000","last_became_peered":"2023-06-10T18:47:59.252812+0000","last_unstale":"2023-06-10T18:48:00.401707+0000","last_undegraded":"2023-06-10T18:48:00.401707+0000","last_fullsized":"2023-06-10T18:48:00.401707+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:00.483660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400768+0000","last_change":"2023-06-10T18:47:59.270792+0000","last_active":"2023-06-10T18:48:00.400768+0000","last_peered":"2023-06-10T18:48:00.400768+0000","last_clean":"2023-06-10T18:48:00.400768+0000","last_became_active":"2023-06-10T18:47:59.270563+0000","last_became_peered":"2023-06-10T18:47:59.270563+0000","last_unstale":"2023-06-10T18:48:00.400768+0000","last_undegraded":"2023-06-10T18:48:00.400768+0000","last_fullsized":"2023-06-10T18:48:00.400768+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:47:58.932650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254200+0000","last_change":"2023-06-10T18:47:59.258843+0000","last_active":"2023-06-10T18:48:00.254200+0000","last_peered":"2023-06-10T18:48:00.254200+0000","last_clean":"2023-06-10T18:48:00.254200+0000","last_became_active":"2023-06-10T18:47:59.258566+0000","last_became_peered":"2023-06-10T18:47:59.258566+0000","last_unstale":"2023-06-10T18:48:00.254200+0000","last_undegraded":"2023-06-10T18:48:00.254200+0000","last_fullsized":"2023-06-10T18:48:00.254200+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:08:40.378289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401411+0000","last_change":"2023-06-10T18:47:59.260327+0000","last_active":"2023-06-10T18:48:00.401411+0000","last_peered":"2023-06-10T18:48:00.401411+0000","last_clean":"2023-06-10T18:48:00.401411+0000","last_became_active":"2023-06-10T18:47:59.260046+0000","last_became_peered":"2023-06-10T18:47:59.260046+0000","last_unstale":"2023-06-10T18:48:00.401411+0000","last_undegraded":"2023-06-10T18:48:00.401411+0000","last_fullsized":"2023-06-10T18:48:00.401411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:22.297934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254314+0000","last_change":"2023-06-10T18:47:59.271695+0000","last_active":"2023-06-10T18:48:00.254314+0000","last_peered":"2023-06-10T18:48:00.254314+0000","last_clean":"2023-06-10T18:48:00.254314+0000","last_became_active":"2023-06-10T18:47:59.271109+0000","last_became_peered":"2023-06-10T18:47:59.271109+0000","last_unstale":"2023-06-10T18:48:00.254314+0000","last_undegraded":"2023-06-10T18:48:00.254314+0000","last_fullsized":"2023-06-10T18:48:00.254314+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:28:22.113139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401888+0000","last_change":"2023-06-10T18:47:59.259022+0000","last_active":"2023-06-10T18:48:00.401888+0000","last_peered":"2023-06-10T18:48:00.401888+0000","last_clean":"2023-06-10T18:48:00.401888+0000","last_became_active":"2023-06-10T18:47:59.258727+0000","last_became_peered":"2023-06-10T18:47:59.258727+0000","last_unstale":"2023-06-10T18:48:00.401888+0000","last_undegraded":"2023-06-10T18:48:00.401888+0000","last_fullsized":"2023-06-10T18:48:00.401888+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:49.543959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255271+0000","last_change":"2023-06-10T18:47:59.263274+0000","last_active":"2023-06-10T18:48:00.255271+0000","last_peered":"2023-06-10T18:48:00.255271+0000","last_clean":"2023-06-10T18:48:00.255271+0000","last_became_active":"2023-06-10T18:47:59.263020+0000","last_became_peered":"2023-06-10T18:47:59.263020+0000","last_unstale":"2023-06-10T18:48:00.255271+0000","last_undegraded":"2023-06-10T18:48:00.255271+0000","last_fullsized":"2023-06-10T18:48:00.255271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:26:24.637191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401105+0000","last_change":"2023-06-10T18:47:59.270306+0000","last_active":"2023-06-10T18:48:00.401105+0000","last_peered":"2023-06-10T18:48:00.401105+0000","last_clean":"2023-06-10T18:48:00.401105+0000","last_became_active":"2023-06-10T18:47:59.270025+0000","last_became_peered":"2023-06-10T18:47:59.270025+0000","last_unstale":"2023-06-10T18:48:00.401105+0000","last_undegraded":"2023-06-10T18:48:00.401105+0000","last_fullsized":"2023-06-10T18:48:00.401105+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:18:12.102366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401949+0000","last_change":"2023-06-10T18:47:59.259768+0000","last_active":"2023-06-10T18:48:00.401949+0000","last_peered":"2023-06-10T18:48:00.401949+0000","last_clean":"2023-06-10T18:48:00.401949+0000","last_became_active":"2023-06-10T18:47:59.259498+0000","last_became_peered":"2023-06-10T18:47:59.259498+0000","last_unstale":"2023-06-10T18:48:00.401949+0000","last_undegraded":"2023-06-10T18:48:00.401949+0000","last_fullsized":"2023-06-10T18:48:00.401949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:53:22.297934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253974+0000","last_change":"2023-06-10T18:47:59.259734+0000","last_active":"2023-06-10T18:48:00.253974+0000","last_peered":"2023-06-10T18:48:00.253974+0000","last_clean":"2023-06-10T18:48:00.253974+0000","last_became_active":"2023-06-10T18:47:59.259382+0000","last_became_peered":"2023-06-10T18:47:59.259382+0000","last_unstale":"2023-06-10T18:48:00.253974+0000","last_undegraded":"2023-06-10T18:48:00.253974+0000","last_fullsized":"2023-06-10T18:48:00.253974+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:28:28.123158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400867+0000","last_change":"2023-06-10T18:47:59.271191+0000","last_active":"2023-06-10T18:48:00.400867+0000","last_peered":"2023-06-10T18:48:00.400867+0000","last_clean":"2023-06-10T18:48:00.400867+0000","last_became_active":"2023-06-10T18:47:59.270956+0000","last_became_peered":"2023-06-10T18:47:59.270956+0000","last_unstale":"2023-06-10T18:48:00.400867+0000","last_undegraded":"2023-06-10T18:48:00.400867+0000","last_fullsized":"2023-06-10T18:48:00.400867+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:48:05.090087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254704+0000","last_change":"2023-06-10T18:47:59.259234+0000","last_active":"2023-06-10T18:48:00.254704+0000","last_peered":"2023-06-10T18:48:00.254704+0000","last_clean":"2023-06-10T18:48:00.254704+0000","last_became_active":"2023-06-10T18:47:59.259003+0000","last_became_peered":"2023-06-10T18:47:59.259003+0000","last_unstale":"2023-06-10T18:48:00.254704+0000","last_undegraded":"2023-06-10T18:48:00.254704+0000","last_fullsized":"2023-06-10T18:48:00.254704+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:16.936511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401687+0000","last_change":"2023-06-10T18:47:59.258089+0000","last_active":"2023-06-10T18:48:00.401687+0000","last_peered":"2023-06-10T18:48:00.401687+0000","last_clean":"2023-06-10T18:48:00.401687+0000","last_became_active":"2023-06-10T18:47:59.257778+0000","last_became_peered":"2023-06-10T18:47:59.257778+0000","last_unstale":"2023-06-10T18:48:00.401687+0000","last_undegraded":"2023-06-10T18:48:00.401687+0000","last_fullsized":"2023-06-10T18:48:00.401687+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:49:39.701679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400467+0000","last_change":"2023-06-10T18:47:59.270406+0000","last_active":"2023-06-10T18:48:00.400467+0000","last_peered":"2023-06-10T18:48:00.400467+0000","last_clean":"2023-06-10T18:48:00.400467+0000","last_became_active":"2023-06-10T18:47:59.270156+0000","last_became_peered":"2023-06-10T18:47:59.270156+0000","last_unstale":"2023-06-10T18:48:00.400467+0000","last_undegraded":"2023-06-10T18:48:00.400467+0000","last_fullsized":"2023-06-10T18:48:00.400467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:25:02.081758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253863+0000","last_change":"2023-06-10T18:47:59.258426+0000","last_active":"2023-06-10T18:48:00.253863+0000","last_peered":"2023-06-10T18:48:00.253863+0000","last_clean":"2023-06-10T18:48:00.253863+0000","last_became_active":"2023-06-10T18:47:59.258156+0000","last_became_peered":"2023-06-10T18:47:59.258156+0000","last_unstale":"2023-06-10T18:48:00.253863+0000","last_undegraded":"2023-06-10T18:48:00.253863+0000","last_fullsized":"2023-06-10T18:48:00.253863+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:09:30.096718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254556+0000","last_change":"2023-06-10T18:47:59.253101+0000","last_active":"2023-06-10T18:48:00.254556+0000","last_peered":"2023-06-10T18:48:00.254556+0000","last_clean":"2023-06-10T18:48:00.254556+0000","last_became_active":"2023-06-10T18:47:59.252851+0000","last_became_peered":"2023-06-10T18:47:59.252851+0000","last_unstale":"2023-06-10T18:48:00.254556+0000","last_undegraded":"2023-06-10T18:48:00.254556+0000","last_fullsized":"2023-06-10T18:48:00.254556+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:59.652589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254657+0000","last_change":"2023-06-10T18:47:58.867330+0000","last_active":"2023-06-10T18:48:00.254657+0000","last_peered":"2023-06-10T18:48:00.254657+0000","last_clean":"2023-06-10T18:48:00.254657+0000","last_became_active":"2023-06-10T18:47:58.867029+0000","last_became_peered":"2023-06-10T18:47:58.867029+0000","last_unstale":"2023-06-10T18:48:00.254657+0000","last_undegraded":"2023-06-10T18:48:00.254657+0000","last_fullsized":"2023-06-10T18:48:00.254657+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:20.841046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401304+0000","last_change":"2023-06-10T18:47:59.267409+0000","last_active":"2023-06-10T18:48:00.401304+0000","last_peered":"2023-06-10T18:48:00.401304+0000","last_clean":"2023-06-10T18:48:00.401304+0000","last_became_active":"2023-06-10T18:47:59.267043+0000","last_became_peered":"2023-06-10T18:47:59.267043+0000","last_unstale":"2023-06-10T18:48:00.401304+0000","last_undegraded":"2023-06-10T18:48:00.401304+0000","last_fullsized":"2023-06-10T18:48:00.401304+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:40.094000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254717+0000","last_change":"2023-06-10T18:47:58.858136+0000","last_active":"2023-06-10T18:48:00.254717+0000","last_peered":"2023-06-10T18:48:00.254717+0000","last_clean":"2023-06-10T18:48:00.254717+0000","last_became_active":"2023-06-10T18:47:58.857856+0000","last_became_peered":"2023-06-10T18:47:58.857856+0000","last_unstale":"2023-06-10T18:48:00.254717+0000","last_undegraded":"2023-06-10T18:48:00.254717+0000","last_fullsized":"2023-06-10T18:48:00.254717+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:39.397363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254918+0000","last_change":"2023-06-10T18:47:59.253824+0000","last_active":"2023-06-10T18:48:00.254918+0000","last_peered":"2023-06-10T18:48:00.254918+0000","last_clean":"2023-06-10T18:48:00.254918+0000","last_became_active":"2023-06-10T18:47:59.253570+0000","last_became_peered":"2023-06-10T18:47:59.253570+0000","last_unstale":"2023-06-10T18:48:00.254918+0000","last_undegraded":"2023-06-10T18:48:00.254918+0000","last_fullsized":"2023-06-10T18:48:00.254918+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:03.380984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254359+0000","last_change":"2023-06-10T18:47:58.861996+0000","last_active":"2023-06-10T18:48:00.254359+0000","last_peered":"2023-06-10T18:48:00.254359+0000","last_clean":"2023-06-10T18:48:00.254359+0000","last_became_active":"2023-06-10T18:47:58.861765+0000","last_became_peered":"2023-06-10T18:47:58.861765+0000","last_unstale":"2023-06-10T18:48:00.254359+0000","last_undegraded":"2023-06-10T18:48:00.254359+0000","last_fullsized":"2023-06-10T18:48:00.254359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:46.896527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254102+0000","last_change":"2023-06-10T18:47:59.262983+0000","last_active":"2023-06-10T18:48:00.254102+0000","last_peered":"2023-06-10T18:48:00.254102+0000","last_clean":"2023-06-10T18:48:00.254102+0000","last_became_active":"2023-06-10T18:47:59.262724+0000","last_became_peered":"2023-06-10T18:47:59.262724+0000","last_unstale":"2023-06-10T18:48:00.254102+0000","last_undegraded":"2023-06-10T18:48:00.254102+0000","last_fullsized":"2023-06-10T18:48:00.254102+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:24:37.026536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402121+0000","last_change":"2023-06-10T18:47:58.869907+0000","last_active":"2023-06-10T18:48:00.402121+0000","last_peered":"2023-06-10T18:48:00.402121+0000","last_clean":"2023-06-10T18:48:00.402121+0000","last_became_active":"2023-06-10T18:47:58.869623+0000","last_became_peered":"2023-06-10T18:47:58.869623+0000","last_unstale":"2023-06-10T18:48:00.402121+0000","last_undegraded":"2023-06-10T18:48:00.402121+0000","last_fullsized":"2023-06-10T18:48:00.402121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:05.006193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254432+0000","last_change":"2023-06-10T18:47:59.271824+0000","last_active":"2023-06-10T18:48:00.254432+0000","last_peered":"2023-06-10T18:48:00.254432+0000","last_clean":"2023-06-10T18:48:00.254432+0000","last_became_active":"2023-06-10T18:47:59.271467+0000","last_became_peered":"2023-06-10T18:47:59.271467+0000","last_unstale":"2023-06-10T18:48:00.254432+0000","last_undegraded":"2023-06-10T18:48:00.254432+0000","last_fullsized":"2023-06-10T18:48:00.254432+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:08:37.981895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256077+0000","last_change":"2023-06-10T18:47:58.863602+0000","last_active":"2023-06-10T18:48:00.256077+0000","last_peered":"2023-06-10T18:48:00.256077+0000","last_clean":"2023-06-10T18:48:00.256077+0000","last_became_active":"2023-06-10T18:47:58.863323+0000","last_became_peered":"2023-06-10T18:47:58.863323+0000","last_unstale":"2023-06-10T18:48:00.256077+0000","last_undegraded":"2023-06-10T18:48:00.256077+0000","last_fullsized":"2023-06-10T18:48:00.256077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:15:16.073766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254325+0000","last_change":"2023-06-10T18:47:59.259732+0000","last_active":"2023-06-10T18:48:00.254325+0000","last_peered":"2023-06-10T18:48:00.254325+0000","last_clean":"2023-06-10T18:48:00.254325+0000","last_became_active":"2023-06-10T18:47:59.258654+0000","last_became_peered":"2023-06-10T18:47:59.258654+0000","last_unstale":"2023-06-10T18:48:00.254325+0000","last_undegraded":"2023-06-10T18:48:00.254325+0000","last_fullsized":"2023-06-10T18:48:00.254325+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:36:30.306301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254133+0000","last_change":"2023-06-10T18:47:58.863792+0000","last_active":"2023-06-10T18:48:00.254133+0000","last_peered":"2023-06-10T18:48:00.254133+0000","last_clean":"2023-06-10T18:48:00.254133+0000","last_became_active":"2023-06-10T18:47:58.863542+0000","last_became_peered":"2023-06-10T18:47:58.863542+0000","last_unstale":"2023-06-10T18:48:00.254133+0000","last_undegraded":"2023-06-10T18:48:00.254133+0000","last_fullsized":"2023-06-10T18:48:00.254133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:07.030546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255416+0000","last_change":"2023-06-10T18:47:59.267928+0000","last_active":"2023-06-10T18:48:00.255416+0000","last_peered":"2023-06-10T18:48:00.255416+0000","last_clean":"2023-06-10T18:48:00.255416+0000","last_became_active":"2023-06-10T18:47:59.267603+0000","last_became_peered":"2023-06-10T18:47:59.267603+0000","last_unstale":"2023-06-10T18:48:00.255416+0000","last_undegraded":"2023-06-10T18:48:00.255416+0000","last_fullsized":"2023-06-10T18:48:00.255416+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:20.818818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401641+0000","last_change":"2023-06-10T18:47:58.868802+0000","last_active":"2023-06-10T18:48:00.401641+0000","last_peered":"2023-06-10T18:48:00.401641+0000","last_clean":"2023-06-10T18:48:00.401641+0000","last_became_active":"2023-06-10T18:47:58.868529+0000","last_became_peered":"2023-06-10T18:47:58.868529+0000","last_unstale":"2023-06-10T18:48:00.401641+0000","last_undegraded":"2023-06-10T18:48:00.401641+0000","last_fullsized":"2023-06-10T18:48:00.401641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:04:56.547758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254028+0000","last_change":"2023-06-10T18:47:59.266772+0000","last_active":"2023-06-10T18:48:00.254028+0000","last_peered":"2023-06-10T18:48:00.254028+0000","last_clean":"2023-06-10T18:48:00.254028+0000","last_became_active":"2023-06-10T18:47:59.266544+0000","last_became_peered":"2023-06-10T18:47:59.266544+0000","last_unstale":"2023-06-10T18:48:00.254028+0000","last_undegraded":"2023-06-10T18:48:00.254028+0000","last_fullsized":"2023-06-10T18:48:00.254028+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:52.432809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255033+0000","last_change":"2023-06-10T18:47:58.861723+0000","last_active":"2023-06-10T18:48:00.255033+0000","last_peered":"2023-06-10T18:48:00.255033+0000","last_clean":"2023-06-10T18:48:00.255033+0000","last_became_active":"2023-06-10T18:47:58.861352+0000","last_became_peered":"2023-06-10T18:47:58.861352+0000","last_unstale":"2023-06-10T18:48:00.255033+0000","last_undegraded":"2023-06-10T18:48:00.255033+0000","last_fullsized":"2023-06-10T18:48:00.255033+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:15.931908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256436+0000","last_change":"2023-06-10T18:47:59.260410+0000","last_active":"2023-06-10T18:48:00.256436+0000","last_peered":"2023-06-10T18:48:00.256436+0000","last_clean":"2023-06-10T18:48:00.256436+0000","last_became_active":"2023-06-10T18:47:59.260128+0000","last_became_peered":"2023-06-10T18:47:59.260128+0000","last_unstale":"2023-06-10T18:48:00.256436+0000","last_undegraded":"2023-06-10T18:48:00.256436+0000","last_fullsized":"2023-06-10T18:48:00.256436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:28:22.113139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402220+0000","last_change":"2023-06-10T18:47:58.860628+0000","last_active":"2023-06-10T18:48:00.402220+0000","last_peered":"2023-06-10T18:48:00.402220+0000","last_clean":"2023-06-10T18:48:00.402220+0000","last_became_active":"2023-06-10T18:47:58.860160+0000","last_became_peered":"2023-06-10T18:47:58.860160+0000","last_unstale":"2023-06-10T18:48:00.402220+0000","last_undegraded":"2023-06-10T18:48:00.402220+0000","last_fullsized":"2023-06-10T18:48:00.402220+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:00.601382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402465+0000","last_change":"2023-06-10T18:47:59.259859+0000","last_active":"2023-06-10T18:48:00.402465+0000","last_peered":"2023-06-10T18:48:00.402465+0000","last_clean":"2023-06-10T18:48:00.402465+0000","last_became_active":"2023-06-10T18:47:59.259589+0000","last_became_peered":"2023-06-10T18:47:59.259589+0000","last_unstale":"2023-06-10T18:48:00.402465+0000","last_undegraded":"2023-06-10T18:48:00.402465+0000","last_fullsized":"2023-06-10T18:48:00.402465+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:06.146139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254794+0000","last_change":"2023-06-10T18:47:58.865983+0000","last_active":"2023-06-10T18:48:00.254794+0000","last_peered":"2023-06-10T18:48:00.254794+0000","last_clean":"2023-06-10T18:48:00.254794+0000","last_became_active":"2023-06-10T18:47:58.865626+0000","last_became_peered":"2023-06-10T18:47:58.865626+0000","last_unstale":"2023-06-10T18:48:00.254794+0000","last_undegraded":"2023-06-10T18:48:00.254794+0000","last_fullsized":"2023-06-10T18:48:00.254794+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:34:53.814935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400914+0000","last_change":"2023-06-10T18:47:59.263747+0000","last_active":"2023-06-10T18:48:00.400914+0000","last_peered":"2023-06-10T18:48:00.400914+0000","last_clean":"2023-06-10T18:48:00.400914+0000","last_became_active":"2023-06-10T18:47:59.263244+0000","last_became_peered":"2023-06-10T18:47:59.263244+0000","last_unstale":"2023-06-10T18:48:00.400914+0000","last_undegraded":"2023-06-10T18:48:00.400914+0000","last_fullsized":"2023-06-10T18:48:00.400914+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:03.042115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402222+0000","last_change":"2023-06-10T18:47:58.861989+0000","last_active":"2023-06-10T18:48:00.402222+0000","last_peered":"2023-06-10T18:48:00.402222+0000","last_clean":"2023-06-10T18:48:00.402222+0000","last_became_active":"2023-06-10T18:47:58.861702+0000","last_became_peered":"2023-06-10T18:47:58.861702+0000","last_unstale":"2023-06-10T18:48:00.402222+0000","last_undegraded":"2023-06-10T18:48:00.402222+0000","last_fullsized":"2023-06-10T18:48:00.402222+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:49.751137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402381+0000","last_change":"2023-06-10T18:47:59.261189+0000","last_active":"2023-06-10T18:48:00.402381+0000","last_peered":"2023-06-10T18:48:00.402381+0000","last_clean":"2023-06-10T18:48:00.402381+0000","last_became_active":"2023-06-10T18:47:59.260920+0000","last_became_peered":"2023-06-10T18:47:59.260920+0000","last_unstale":"2023-06-10T18:48:00.402381+0000","last_undegraded":"2023-06-10T18:48:00.402381+0000","last_fullsized":"2023-06-10T18:48:00.402381+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:41:27.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.256456+0000","last_change":"2023-06-10T18:47:58.861104+0000","last_active":"2023-06-10T18:48:00.256456+0000","last_peered":"2023-06-10T18:48:00.256456+0000","last_clean":"2023-06-10T18:48:00.256456+0000","last_became_active":"2023-06-10T18:47:58.860789+0000","last_became_peered":"2023-06-10T18:47:58.860789+0000","last_unstale":"2023-06-10T18:48:00.256456+0000","last_undegraded":"2023-06-10T18:48:00.256456+0000","last_fullsized":"2023-06-10T18:48:00.256456+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:33:50.813268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401409+0000","last_change":"2023-06-10T18:47:59.263588+0000","last_active":"2023-06-10T18:48:00.401409+0000","last_peered":"2023-06-10T18:48:00.401409+0000","last_clean":"2023-06-10T18:48:00.401409+0000","last_became_active":"2023-06-10T18:47:59.263147+0000","last_became_peered":"2023-06-10T18:47:59.263147+0000","last_unstale":"2023-06-10T18:48:00.401409+0000","last_undegraded":"2023-06-10T18:48:00.401409+0000","last_fullsized":"2023-06-10T18:48:00.401409+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:40.401965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254615+0000","last_change":"2023-06-10T18:47:58.865915+0000","last_active":"2023-06-10T18:48:00.254615+0000","last_peered":"2023-06-10T18:48:00.254615+0000","last_clean":"2023-06-10T18:48:00.254615+0000","last_became_active":"2023-06-10T18:47:58.865645+0000","last_became_peered":"2023-06-10T18:47:58.865645+0000","last_unstale":"2023-06-10T18:48:00.254615+0000","last_undegraded":"2023-06-10T18:48:00.254615+0000","last_fullsized":"2023-06-10T18:48:00.254615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:14.661882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401569+0000","last_change":"2023-06-10T18:47:59.262624+0000","last_active":"2023-06-10T18:48:00.401569+0000","last_peered":"2023-06-10T18:48:00.401569+0000","last_clean":"2023-06-10T18:48:00.401569+0000","last_became_active":"2023-06-10T18:47:59.261891+0000","last_became_peered":"2023-06-10T18:47:59.261891+0000","last_unstale":"2023-06-10T18:48:00.401569+0000","last_undegraded":"2023-06-10T18:48:00.401569+0000","last_fullsized":"2023-06-10T18:48:00.401569+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:43:56.816497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255592+0000","last_change":"2023-06-10T18:47:58.862528+0000","last_active":"2023-06-10T18:48:00.255592+0000","last_peered":"2023-06-10T18:48:00.255592+0000","last_clean":"2023-06-10T18:48:00.255592+0000","last_became_active":"2023-06-10T18:47:58.862241+0000","last_became_peered":"2023-06-10T18:47:58.862241+0000","last_unstale":"2023-06-10T18:48:00.255592+0000","last_undegraded":"2023-06-10T18:48:00.255592+0000","last_fullsized":"2023-06-10T18:48:00.255592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:17:41.370481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254985+0000","last_change":"2023-06-10T18:47:59.255838+0000","last_active":"2023-06-10T18:48:00.254985+0000","last_peered":"2023-06-10T18:48:00.254985+0000","last_clean":"2023-06-10T18:48:00.254985+0000","last_became_active":"2023-06-10T18:47:59.255506+0000","last_became_peered":"2023-06-10T18:47:59.255506+0000","last_unstale":"2023-06-10T18:48:00.254985+0000","last_undegraded":"2023-06-10T18:48:00.254985+0000","last_fullsized":"2023-06-10T18:48:00.254985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:21.845649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401793+0000","last_change":"2023-06-10T18:47:58.869572+0000","last_active":"2023-06-10T18:48:00.401793+0000","last_peered":"2023-06-10T18:48:00.401793+0000","last_clean":"2023-06-10T18:48:00.401793+0000","last_became_active":"2023-06-10T18:47:58.869238+0000","last_became_peered":"2023-06-10T18:47:58.869238+0000","last_unstale":"2023-06-10T18:48:00.401793+0000","last_undegraded":"2023-06-10T18:48:00.401793+0000","last_fullsized":"2023-06-10T18:48:00.401793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:03:26.529546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402100+0000","last_change":"2023-06-10T18:47:59.258884+0000","last_active":"2023-06-10T18:48:00.402100+0000","last_peered":"2023-06-10T18:48:00.402100+0000","last_clean":"2023-06-10T18:48:00.402100+0000","last_became_active":"2023-06-10T18:47:59.258599+0000","last_became_peered":"2023-06-10T18:47:59.258599+0000","last_unstale":"2023-06-10T18:48:00.402100+0000","last_undegraded":"2023-06-10T18:48:00.402100+0000","last_fullsized":"2023-06-10T18:48:00.402100+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:43:32.660038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401236+0000","last_change":"2023-06-10T18:47:58.867385+0000","last_active":"2023-06-10T18:48:00.401236+0000","last_peered":"2023-06-10T18:48:00.401236+0000","last_clean":"2023-06-10T18:48:00.401236+0000","last_became_active":"2023-06-10T18:47:58.867085+0000","last_became_peered":"2023-06-10T18:47:58.867085+0000","last_unstale":"2023-06-10T18:48:00.401236+0000","last_undegraded":"2023-06-10T18:48:00.401236+0000","last_fullsized":"2023-06-10T18:48:00.401236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:05:31.788286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402159+0000","last_change":"2023-06-10T18:47:59.271202+0000","last_active":"2023-06-10T18:48:00.402159+0000","last_peered":"2023-06-10T18:48:00.402159+0000","last_clean":"2023-06-10T18:48:00.402159+0000","last_became_active":"2023-06-10T18:47:59.270921+0000","last_became_peered":"2023-06-10T18:47:59.270921+0000","last_unstale":"2023-06-10T18:48:00.402159+0000","last_undegraded":"2023-06-10T18:48:00.402159+0000","last_fullsized":"2023-06-10T18:48:00.402159+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:35:31.848299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401347+0000","last_change":"2023-06-10T18:47:58.860453+0000","last_active":"2023-06-10T18:48:00.401347+0000","last_peered":"2023-06-10T18:48:00.401347+0000","last_clean":"2023-06-10T18:48:00.401347+0000","last_became_active":"2023-06-10T18:47:58.860184+0000","last_became_peered":"2023-06-10T18:47:58.860184+0000","last_unstale":"2023-06-10T18:48:00.401347+0000","last_undegraded":"2023-06-10T18:48:00.401347+0000","last_fullsized":"2023-06-10T18:48:00.401347+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:59:08.710821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401986+0000","last_change":"2023-06-10T18:47:59.262671+0000","last_active":"2023-06-10T18:48:00.401986+0000","last_peered":"2023-06-10T18:48:00.401986+0000","last_clean":"2023-06-10T18:48:00.401986+0000","last_became_active":"2023-06-10T18:47:59.262399+0000","last_became_peered":"2023-06-10T18:47:59.262399+0000","last_unstale":"2023-06-10T18:48:00.401986+0000","last_undegraded":"2023-06-10T18:48:00.401986+0000","last_fullsized":"2023-06-10T18:48:00.401986+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:00.483660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254670+0000","last_change":"2023-06-10T18:47:58.864550+0000","last_active":"2023-06-10T18:48:00.254670+0000","last_peered":"2023-06-10T18:48:00.254670+0000","last_clean":"2023-06-10T18:48:00.254670+0000","last_became_active":"2023-06-10T18:47:58.864301+0000","last_became_peered":"2023-06-10T18:47:58.864301+0000","last_unstale":"2023-06-10T18:48:00.254670+0000","last_undegraded":"2023-06-10T18:48:00.254670+0000","last_fullsized":"2023-06-10T18:48:00.254670+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:15.931908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401051+0000","last_change":"2023-06-10T18:47:59.263553+0000","last_active":"2023-06-10T18:48:00.401051+0000","last_peered":"2023-06-10T18:48:00.401051+0000","last_clean":"2023-06-10T18:48:00.401051+0000","last_became_active":"2023-06-10T18:47:59.262877+0000","last_became_peered":"2023-06-10T18:47:59.262877+0000","last_unstale":"2023-06-10T18:48:00.401051+0000","last_undegraded":"2023-06-10T18:48:00.401051+0000","last_fullsized":"2023-06-10T18:48:00.401051+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:48.210843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255706+0000","last_change":"2023-06-10T18:47:58.862717+0000","last_active":"2023-06-10T18:48:00.255706+0000","last_peered":"2023-06-10T18:48:00.255706+0000","last_clean":"2023-06-10T18:48:00.255706+0000","last_became_active":"2023-06-10T18:47:58.862331+0000","last_became_peered":"2023-06-10T18:47:58.862331+0000","last_unstale":"2023-06-10T18:48:00.255706+0000","last_undegraded":"2023-06-10T18:48:00.255706+0000","last_fullsized":"2023-06-10T18:48:00.255706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:42:39.043042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254224+0000","last_change":"2023-06-10T18:47:59.262842+0000","last_active":"2023-06-10T18:48:00.254224+0000","last_peered":"2023-06-10T18:48:00.254224+0000","last_clean":"2023-06-10T18:48:00.254224+0000","last_became_active":"2023-06-10T18:47:59.262588+0000","last_became_peered":"2023-06-10T18:47:59.262588+0000","last_unstale":"2023-06-10T18:48:00.254224+0000","last_undegraded":"2023-06-10T18:48:00.254224+0000","last_fullsized":"2023-06-10T18:48:00.254224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:50.987450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401961+0000","last_change":"2023-06-10T18:47:58.866633+0000","last_active":"2023-06-10T18:48:00.401961+0000","last_peered":"2023-06-10T18:48:00.401961+0000","last_clean":"2023-06-10T18:48:00.401961+0000","last_became_active":"2023-06-10T18:47:58.866369+0000","last_became_peered":"2023-06-10T18:47:58.866369+0000","last_unstale":"2023-06-10T18:48:00.401961+0000","last_undegraded":"2023-06-10T18:48:00.401961+0000","last_fullsized":"2023-06-10T18:48:00.401961+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:49:38.697076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254547+0000","last_change":"2023-06-10T18:47:59.254653+0000","last_active":"2023-06-10T18:48:00.254547+0000","last_peered":"2023-06-10T18:48:00.254547+0000","last_clean":"2023-06-10T18:48:00.254547+0000","last_became_active":"2023-06-10T18:47:59.254400+0000","last_became_peered":"2023-06-10T18:47:59.254400+0000","last_unstale":"2023-06-10T18:48:00.254547+0000","last_undegraded":"2023-06-10T18:48:00.254547+0000","last_fullsized":"2023-06-10T18:48:00.254547+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:10:22.206584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401027+0000","last_change":"2023-06-10T18:47:58.867265+0000","last_active":"2023-06-10T18:48:00.401027+0000","last_peered":"2023-06-10T18:48:00.401027+0000","last_clean":"2023-06-10T18:48:00.401027+0000","last_became_active":"2023-06-10T18:47:58.866983+0000","last_became_peered":"2023-06-10T18:47:58.866983+0000","last_unstale":"2023-06-10T18:48:00.401027+0000","last_undegraded":"2023-06-10T18:48:00.401027+0000","last_fullsized":"2023-06-10T18:48:00.401027+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:51:57.937926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401720+0000","last_change":"2023-06-10T18:47:59.260756+0000","last_active":"2023-06-10T18:48:00.401720+0000","last_peered":"2023-06-10T18:48:00.401720+0000","last_clean":"2023-06-10T18:48:00.401720+0000","last_became_active":"2023-06-10T18:47:59.259768+0000","last_became_peered":"2023-06-10T18:47:59.259768+0000","last_unstale":"2023-06-10T18:48:00.401720+0000","last_undegraded":"2023-06-10T18:48:00.401720+0000","last_fullsized":"2023-06-10T18:48:00.401720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:06.010795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255999+0000","last_change":"2023-06-10T18:47:58.861747+0000","last_active":"2023-06-10T18:48:00.255999+0000","last_peered":"2023-06-10T18:48:00.255999+0000","last_clean":"2023-06-10T18:48:00.255999+0000","last_became_active":"2023-06-10T18:47:58.861368+0000","last_became_peered":"2023-06-10T18:47:58.861368+0000","last_unstale":"2023-06-10T18:48:00.255999+0000","last_undegraded":"2023-06-10T18:48:00.255999+0000","last_fullsized":"2023-06-10T18:48:00.255999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:21.187245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400720+0000","last_change":"2023-06-10T18:47:59.262613+0000","last_active":"2023-06-10T18:48:00.400720+0000","last_peered":"2023-06-10T18:48:00.400720+0000","last_clean":"2023-06-10T18:48:00.400720+0000","last_became_active":"2023-06-10T18:47:59.262351+0000","last_became_peered":"2023-06-10T18:47:59.262351+0000","last_unstale":"2023-06-10T18:48:00.400720+0000","last_undegraded":"2023-06-10T18:48:00.400720+0000","last_fullsized":"2023-06-10T18:48:00.400720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:49.262983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400913+0000","last_change":"2023-06-10T18:47:58.862597+0000","last_active":"2023-06-10T18:48:00.400913+0000","last_peered":"2023-06-10T18:48:00.400913+0000","last_clean":"2023-06-10T18:48:00.400913+0000","last_became_active":"2023-06-10T18:47:58.862345+0000","last_became_peered":"2023-06-10T18:47:58.862345+0000","last_unstale":"2023-06-10T18:48:00.400913+0000","last_undegraded":"2023-06-10T18:48:00.400913+0000","last_fullsized":"2023-06-10T18:48:00.400913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:58:26.726084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254162+0000","last_change":"2023-06-10T18:47:59.266712+0000","last_active":"2023-06-10T18:48:00.254162+0000","last_peered":"2023-06-10T18:48:00.254162+0000","last_clean":"2023-06-10T18:48:00.254162+0000","last_became_active":"2023-06-10T18:47:59.266549+0000","last_became_peered":"2023-06-10T18:47:59.266549+0000","last_unstale":"2023-06-10T18:48:00.254162+0000","last_undegraded":"2023-06-10T18:48:00.254162+0000","last_fullsized":"2023-06-10T18:48:00.254162+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:22.191847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254102+0000","last_change":"2023-06-10T18:47:58.867264+0000","last_active":"2023-06-10T18:48:00.254102+0000","last_peered":"2023-06-10T18:48:00.254102+0000","last_clean":"2023-06-10T18:48:00.254102+0000","last_became_active":"2023-06-10T18:47:58.867016+0000","last_became_peered":"2023-06-10T18:47:58.867016+0000","last_unstale":"2023-06-10T18:48:00.254102+0000","last_undegraded":"2023-06-10T18:48:00.254102+0000","last_fullsized":"2023-06-10T18:48:00.254102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:02.376382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402347+0000","last_change":"2023-06-10T18:47:59.264466+0000","last_active":"2023-06-10T18:48:00.402347+0000","last_peered":"2023-06-10T18:48:00.402347+0000","last_clean":"2023-06-10T18:48:00.402347+0000","last_became_active":"2023-06-10T18:47:59.264213+0000","last_became_peered":"2023-06-10T18:47:59.264213+0000","last_unstale":"2023-06-10T18:48:00.402347+0000","last_undegraded":"2023-06-10T18:48:00.402347+0000","last_fullsized":"2023-06-10T18:48:00.402347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:15.666485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254900+0000","last_change":"2023-06-10T18:47:58.866173+0000","last_active":"2023-06-10T18:48:00.254900+0000","last_peered":"2023-06-10T18:48:00.254900+0000","last_clean":"2023-06-10T18:48:00.254900+0000","last_became_active":"2023-06-10T18:47:58.865625+0000","last_became_peered":"2023-06-10T18:47:58.865625+0000","last_unstale":"2023-06-10T18:48:00.254900+0000","last_undegraded":"2023-06-10T18:48:00.254900+0000","last_fullsized":"2023-06-10T18:48:00.254900+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:02:41.943041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401097+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:48:00.401097+0000","last_peered":"2023-06-10T18:48:00.401097+0000","last_clean":"2023-06-10T18:48:00.401097+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:48:00.401097+0000","last_undegraded":"2023-06-10T18:48:00.401097+0000","last_fullsized":"2023-06-10T18:48:00.401097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:55:59.087724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253917+0000","last_change":"2023-06-10T18:47:59.253010+0000","last_active":"2023-06-10T18:48:00.253917+0000","last_peered":"2023-06-10T18:48:00.253917+0000","last_clean":"2023-06-10T18:48:00.253917+0000","last_became_active":"2023-06-10T18:47:59.252724+0000","last_became_peered":"2023-06-10T18:47:59.252724+0000","last_unstale":"2023-06-10T18:48:00.253917+0000","last_undegraded":"2023-06-10T18:48:00.253917+0000","last_fullsized":"2023-06-10T18:48:00.253917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:48.345875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401187+0000","last_change":"2023-06-10T18:47:58.869350+0000","last_active":"2023-06-10T18:48:00.401187+0000","last_peered":"2023-06-10T18:48:00.401187+0000","last_clean":"2023-06-10T18:48:00.401187+0000","last_became_active":"2023-06-10T18:47:58.868873+0000","last_became_peered":"2023-06-10T18:47:58.868873+0000","last_unstale":"2023-06-10T18:48:00.401187+0000","last_undegraded":"2023-06-10T18:48:00.401187+0000","last_fullsized":"2023-06-10T18:48:00.401187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:10:21.201982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401637+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:48:00.401637+0000","last_peered":"2023-06-10T18:48:00.401637+0000","last_clean":"2023-06-10T18:48:00.401637+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:48:00.401637+0000","last_undegraded":"2023-06-10T18:48:00.401637+0000","last_fullsized":"2023-06-10T18:48:00.401637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:42.280363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401557+0000","last_change":"2023-06-10T18:47:59.263421+0000","last_active":"2023-06-10T18:48:00.401557+0000","last_peered":"2023-06-10T18:48:00.401557+0000","last_clean":"2023-06-10T18:48:00.401557+0000","last_became_active":"2023-06-10T18:47:59.262823+0000","last_became_peered":"2023-06-10T18:47:59.262823+0000","last_unstale":"2023-06-10T18:48:00.401557+0000","last_undegraded":"2023-06-10T18:48:00.401557+0000","last_fullsized":"2023-06-10T18:48:00.401557+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:34:54.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254433+0000","last_change":"2023-06-10T18:47:58.860988+0000","last_active":"2023-06-10T18:48:00.254433+0000","last_peered":"2023-06-10T18:48:00.254433+0000","last_clean":"2023-06-10T18:48:00.254433+0000","last_became_active":"2023-06-10T18:47:58.860748+0000","last_became_peered":"2023-06-10T18:47:58.860748+0000","last_unstale":"2023-06-10T18:48:00.254433+0000","last_undegraded":"2023-06-10T18:48:00.254433+0000","last_fullsized":"2023-06-10T18:48:00.254433+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:18:15.896296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.400418+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:48:00.400418+0000","last_peered":"2023-06-10T18:48:00.400418+0000","last_clean":"2023-06-10T18:48:00.400418+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:48:00.400418+0000","last_undegraded":"2023-06-10T18:48:00.400418+0000","last_fullsized":"2023-06-10T18:48:00.400418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:56.642449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.253983+0000","last_change":"2023-06-10T18:47:59.262130+0000","last_active":"2023-06-10T18:48:00.253983+0000","last_peered":"2023-06-10T18:48:00.253983+0000","last_clean":"2023-06-10T18:48:00.253983+0000","last_became_active":"2023-06-10T18:47:59.261897+0000","last_became_peered":"2023-06-10T18:47:59.261897+0000","last_unstale":"2023-06-10T18:48:00.253983+0000","last_undegraded":"2023-06-10T18:48:00.253983+0000","last_fullsized":"2023-06-10T18:48:00.253983+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:21.845649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255259+0000","last_change":"2023-06-10T18:47:58.870783+0000","last_active":"2023-06-10T18:48:00.255259+0000","last_peered":"2023-06-10T18:48:00.255259+0000","last_clean":"2023-06-10T18:48:00.255259+0000","last_became_active":"2023-06-10T18:47:58.870474+0000","last_became_peered":"2023-06-10T18:47:58.870474+0000","last_unstale":"2023-06-10T18:48:00.255259+0000","last_undegraded":"2023-06-10T18:48:00.255259+0000","last_fullsized":"2023-06-10T18:48:00.255259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:30:57.976057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401900+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:48:00.401900+0000","last_peered":"2023-06-10T18:48:00.401900+0000","last_clean":"2023-06-10T18:48:00.401900+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:48:00.401900+0000","last_undegraded":"2023-06-10T18:48:00.401900+0000","last_fullsized":"2023-06-10T18:48:00.401900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:00.495694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":237,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401295+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:48:00.401295+0000","last_peered":"2023-06-10T18:48:00.401295+0000","last_clean":"2023-06-10T18:48:00.401295+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:48:00.401295+0000","last_undegraded":"2023-06-10T18:48:00.401295+0000","last_fullsized":"2023-06-10T18:48:00.401295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:59:25.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254425+0000","last_change":"2023-06-10T18:47:59.260160+0000","last_active":"2023-06-10T18:48:00.254425+0000","last_peered":"2023-06-10T18:48:00.254425+0000","last_clean":"2023-06-10T18:48:00.254425+0000","last_became_active":"2023-06-10T18:47:59.259907+0000","last_became_peered":"2023-06-10T18:47:59.259907+0000","last_unstale":"2023-06-10T18:48:00.254425+0000","last_undegraded":"2023-06-10T18:48:00.254425+0000","last_fullsized":"2023-06-10T18:48:00.254425+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:31.431220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401640+0000","last_change":"2023-06-10T18:47:58.860831+0000","last_active":"2023-06-10T18:48:00.401640+0000","last_peered":"2023-06-10T18:48:00.401640+0000","last_clean":"2023-06-10T18:48:00.401640+0000","last_became_active":"2023-06-10T18:47:58.860555+0000","last_became_peered":"2023-06-10T18:47:58.860555+0000","last_unstale":"2023-06-10T18:48:00.401640+0000","last_undegraded":"2023-06-10T18:48:00.401640+0000","last_fullsized":"2023-06-10T18:48:00.401640+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:36:46.516888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401349+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:48:00.401349+0000","last_peered":"2023-06-10T18:48:00.401349+0000","last_clean":"2023-06-10T18:48:00.401349+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:48:00.401349+0000","last_undegraded":"2023-06-10T18:48:00.401349+0000","last_fullsized":"2023-06-10T18:48:00.401349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:22:32.487446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401861+0000","last_change":"2023-06-10T18:47:59.258960+0000","last_active":"2023-06-10T18:48:00.401861+0000","last_peered":"2023-06-10T18:48:00.401861+0000","last_clean":"2023-06-10T18:48:00.401861+0000","last_became_active":"2023-06-10T18:47:59.258637+0000","last_became_peered":"2023-06-10T18:47:59.258637+0000","last_unstale":"2023-06-10T18:48:00.401861+0000","last_undegraded":"2023-06-10T18:48:00.401861+0000","last_fullsized":"2023-06-10T18:48:00.401861+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:35:31.848299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254359+0000","last_change":"2023-06-10T18:47:58.866007+0000","last_active":"2023-06-10T18:48:00.254359+0000","last_peered":"2023-06-10T18:48:00.254359+0000","last_clean":"2023-06-10T18:48:00.254359+0000","last_became_active":"2023-06-10T18:47:58.865696+0000","last_became_peered":"2023-06-10T18:47:58.865696+0000","last_unstale":"2023-06-10T18:48:00.254359+0000","last_undegraded":"2023-06-10T18:48:00.254359+0000","last_fullsized":"2023-06-10T18:48:00.254359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:54:39.011202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402472+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:48:00.402472+0000","last_peered":"2023-06-10T18:48:00.402472+0000","last_clean":"2023-06-10T18:48:00.402472+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:48:00.402472+0000","last_undegraded":"2023-06-10T18:48:00.402472+0000","last_fullsized":"2023-06-10T18:48:00.402472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:17.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402610+0000","last_change":"2023-06-10T18:47:59.260870+0000","last_active":"2023-06-10T18:48:00.402610+0000","last_peered":"2023-06-10T18:48:00.402610+0000","last_clean":"2023-06-10T18:48:00.402610+0000","last_became_active":"2023-06-10T18:47:59.260577+0000","last_became_peered":"2023-06-10T18:47:59.260577+0000","last_unstale":"2023-06-10T18:48:00.402610+0000","last_undegraded":"2023-06-10T18:48:00.402610+0000","last_fullsized":"2023-06-10T18:48:00.402610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:33.650753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254951+0000","last_change":"2023-06-10T18:47:58.860101+0000","last_active":"2023-06-10T18:48:00.254951+0000","last_peered":"2023-06-10T18:48:00.254951+0000","last_clean":"2023-06-10T18:48:00.254951+0000","last_became_active":"2023-06-10T18:47:58.859822+0000","last_became_peered":"2023-06-10T18:47:58.859822+0000","last_unstale":"2023-06-10T18:48:00.254951+0000","last_undegraded":"2023-06-10T18:48:00.254951+0000","last_fullsized":"2023-06-10T18:48:00.254951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:36:21.187245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.401081+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:48:00.401081+0000","last_peered":"2023-06-10T18:48:00.401081+0000","last_clean":"2023-06-10T18:48:00.401081+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:48:00.401081+0000","last_undegraded":"2023-06-10T18:48:00.401081+0000","last_fullsized":"2023-06-10T18:48:00.401081+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:32:59.634148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.254334+0000","last_change":"2023-06-10T18:47:59.257647+0000","last_active":"2023-06-10T18:48:00.254334+0000","last_peered":"2023-06-10T18:48:00.254334+0000","last_clean":"2023-06-10T18:48:00.254334+0000","last_became_active":"2023-06-10T18:47:59.257322+0000","last_became_peered":"2023-06-10T18:47:59.257322+0000","last_unstale":"2023-06-10T18:48:00.254334+0000","last_undegraded":"2023-06-10T18:48:00.254334+0000","last_fullsized":"2023-06-10T18:48:00.254334+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:58.840830+0000","last_clean_scrub_stamp":"2023-06-10T18:47:58.840830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:33:51.817871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.402214+0000","last_change":"2023-06-10T18:47:58.869544+0000","last_active":"2023-06-10T18:48:00.402214+0000","last_peered":"2023-06-10T18:48:00.402214+0000","last_clean":"2023-06-10T18:48:00.402214+0000","last_became_active":"2023-06-10T18:47:58.869249+0000","last_became_peered":"2023-06-10T18:47:58.869249+0000","last_unstale":"2023-06-10T18:48:00.402214+0000","last_undegraded":"2023-06-10T18:48:00.402214+0000","last_fullsized":"2023-06-10T18:48:00.402214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:47:57.836227+0000","last_clean_scrub_stamp":"2023-06-10T18:47:57.836227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:48.539356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-06-10T18:48:00.255771+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:48:00.255771+0000","last_peered":"2023-06-10T18:48:00.255771+0000","last_clean":"2023-06-10T18:48:00.255771+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:48:00.255771+0000","last_undegraded":"2023-06-10T18:48:00.255771+0000","last_fullsized":"2023-06-10T18:48:00.255771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:56:26.622598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1269760,"data_stored":899557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53300000000000003,"15min":0.52100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.39100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.40300000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.70899999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.48199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73499999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.443}]},{"osd":2,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48799999999999999,"15min":0.47299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67200000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.33900000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.625}]},{"osd":4,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47499999999999998,"15min":0.46700000000000003},"min":{"1min":0.32500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63900000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.752,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36199999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67500000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54900000000000004,"15min":0.55000000000000004},"min":{"1min":0.373,"5min":0.311,"15min":0.311},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":663552,"data_stored":308010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.43099999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.75700000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52000000000000002,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.752},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.72399999999999998,"15min":0.89000000000000001},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65400000000000003,"15min":0.77400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.316,"15min":0.316},"max":{"1min":0.71299999999999997,"5min":1.252,"15min":1.252},"last":0.621}]},{"osd":3,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.69099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.52300000000000002},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.40899999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.78600000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":667648,"data_stored":308100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.46300000000000002,"5min":0.753,"15min":0.753},"last":0.246},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.33000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56599999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.309}]},{"osd":3,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55900000000000005,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.62},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86799999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":1.008,"15min":1.008},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.43099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77100000000000002,"5min":0.876,"15min":0.876},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1273856,"data_stored":900524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34699999999999998,"15min":0.35399999999999998},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.625,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.36099999999999999},"min":{"1min":0.30199999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.54300000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38800000000000001},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65500000000000003,"5min":1.405,"15min":1.405},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.39600000000000002},"min":{"1min":0.33000000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.505,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.27200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60499999999999998,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56699999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.32600000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.371,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79900000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.495}]},{"osd":5,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85899999999999999,"5min":0.999,"15min":0.999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80600000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.46200000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.623,"15min":0.63},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91400000000000003,"5min":1.363,"15min":1.363},"last":0.47899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89200000000000002,"5min":1.034,"15min":1.034},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.33000000000000002},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.23799999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.34799999999999998},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.35399999999999998,"15min":0.34300000000000003},"min":{"1min":0.33000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.29199999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.32100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.376}]},{"osd":4,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.47599999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.622},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.502,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.498}]},{"osd":5,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.52100000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.52100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74199999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.498,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77200000000000002,"5min":0.872,"15min":0.872},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.496,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.71299999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.55100000000000005,"5min":0.433,"15min":0.433},"max":{"1min":0.83199999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.51600000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.50800000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.53400000000000003,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.51700000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.312,"5min":0.245,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":1.3,"15min":1.3},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.33100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58699999999999997,"5min":1.109,"15min":1.109},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.46500000000000002,"15min":0.49199999999999999},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56000000000000005,"5min":1.599,"15min":1.599},"last":0.378}]},{"osd":4,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.61399999999999999,"15min":0.63400000000000001},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.745,"5min":1.925,"15min":1.925},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.45400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81299999999999994,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.49299999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84299999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.72099999999999997,"15min":0.78000000000000003},"min":{"1min":0.43099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89900000000000002,"5min":3.802,"15min":3.802},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71899999999999997,"15min":0.78000000000000003},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.622,"15min":0.628},"min":{"1min":0.46000000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88600000000000001,"5min":1.03,"15min":1.03},"last":0.55500000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:46:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94199999999999995,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.64000000000000001,"15min":0.65800000000000003},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85299999999999998,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.47199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51800000000000002,"15min":0.50700000000000001},"min":{"1min":0.40999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.40000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.44900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72199999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.36699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70499999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.47899999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.45800000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54400000000000004,"15min":0.53400000000000003},"min":{"1min":0.44900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.38800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70699999999999996,"5min":1.042,"15min":1.042},"last":0.38100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.069,"15min":1.069},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.50800000000000001,"15min":0.51600000000000001},"min":{"1min":0.32500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65500000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53800000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77200000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53100000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.71999999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59199999999999997,"15min":0.60099999999999998},"min":{"1min":0.41899999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.58399999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":675840,"data_stored":309681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67900000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.495,"15min":0.48099999999999998},"min":{"1min":0.38300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65200000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42399999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.51800000000000002,"15min":0.504},"min":{"1min":0.46400000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71299999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.42499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.436,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68100000000000005,"5min":0.745,"15min":0.745},"last":0.503},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54600000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67500000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.32900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:47:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77800000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":0.754},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:48:03.821 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:48:03.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:48:03.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.977+0000 7fc3a9014700 1 Processor -- start 2023-06-10T18:48:03.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.977+0000 7fc3a9014700 1 -- start start 2023-06-10T18:48:03.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 0x7fc3a40d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a40ccff0 0x7fc3a406b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3a415fd00 0x7fc3a4160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40cd460 con 0x7fc3a415fd00 2023-06-10T18:48:03.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40ce340 con 0x7fc3a406af80 2023-06-10T18:48:03.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a9014700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40d00d0 con 0x7fc3a40ccff0 2023-06-10T18:48:03.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3a415fd00 0x7fc3a4160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3a415fd00 0x7fc3a4160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58614/0 (socket says 172.21.15.104:58614) 2023-06-10T18:48:03.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 -- 172.21.15.104:0/2872191961 learned_addr learned my addr 172.21.15.104:0/2872191961 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:03.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a2d9d700 1 --2- 172.21.15.104:0/2872191961 >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 0x7fc3a40d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a259c700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a40ccff0 0x7fc3a406b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 -- 172.21.15.104:0/2872191961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a40ccff0 msgr2=0x7fc3a406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a40ccff0 0x7fc3a406b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 -- 172.21.15.104:0/2872191961 >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 msgr2=0x7fc3a40d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:03.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 --2- 172.21.15.104:0/2872191961 >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 0x7fc3a40d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 -- 172.21.15.104:0/2872191961 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a406b890 con 0x7fc3a415fd00 2023-06-10T18:48:03.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a2d9d700 1 --2- 172.21.15.104:0/2872191961 >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 0x7fc3a40d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:03.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a359e700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a415fd00 0x7fc3a4160180 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009e50 tx=0x7fc39c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94014a7a717e9a45 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a1d9b700 1 -- 172.21.15.104:0/2872191961 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c00e070 con 0x7fc3a415fd00 2023-06-10T18:48:03.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a1d9b700 1 -- 172.21.15.104:0/2872191961 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c00b980 con 0x7fc3a415fd00 2023-06-10T18:48:03.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.978+0000 7fc3a1d9b700 1 -- 172.21.15.104:0/2872191961 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c00bd30 con 0x7fc3a415fd00 2023-06-10T18:48:03.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 -- 172.21.15.104:0/2872191961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a415fd00 msgr2=0x7fc3a4160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a415fd00 0x7fc3a4160180 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009e50 tx=0x7fc39c00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 -- 172.21.15.104:0/2872191961 shutdown_connections 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a415fd00 0x7fc3a4160180 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 --2- 172.21.15.104:0/2872191961 >> v2:172.21.15.114:3300/0 conn(0x7fc3a406af80 0x7fc3a40d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 --2- 172.21.15.104:0/2872191961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a40ccff0 0x7fc3a406b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 -- 172.21.15.104:0/2872191961 >> 172.21.15.104:0/2872191961 conn(0x7fc3a4089fd0 msgr2=0x7fc3a408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:03.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 -- 172.21.15.104:0/2872191961 shutdown_connections 2023-06-10T18:48:03.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.979+0000 7fc3a9014700 1 -- 172.21.15.104:0/2872191961 wait complete. 2023-06-10T18:48:03.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.981+0000 7fc3a9014700 1 Processor -- start 2023-06-10T18:48:03.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.981+0000 7fc3a9014700 1 -- start start 2023-06-10T18:48:03.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3a40ccff0 0x7fc3a40b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 0x7fc3a40b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40c70a0 con 0x7fc3a406af80 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b4790 con 0x7fc3a40ccff0 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b4a70 con 0x7fc3a415fd00 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58616/0 (socket says 172.21.15.104:58616) 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 -- 172.21.15.104:0/1639194621 learned_addr learned my addr 172.21.15.104:0/1639194621 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a259c700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3a40ccff0 0x7fc3a40b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a359e700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 0x7fc3a40b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 -- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 msgr2=0x7fc3a40b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 0x7fc3a40b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 -- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3a40ccff0 msgr2=0x7fc3a40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3a40ccff0 0x7fc3a40b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 -- 172.21.15.104:0/1639194621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009b30 con 0x7fc3a406af80 2023-06-10T18:48:03.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a359e700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 0x7fc3a40b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a2d9d700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc39800d800 tx=0x7fc39800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398004030 con 0x7fc3a406af80 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398004190 con 0x7fc3a406af80 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.982+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a40b4d50 con 0x7fc3a406af80 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.983+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398004520 con 0x7fc3a406af80 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.983+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a415b650 con 0x7fc3a406af80 2023-06-10T18:48:03.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.983+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a40b53a0 con 0x7fc3a406af80 2023-06-10T18:48:03.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.984+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc3980036a0 con 0x7fc3a406af80 2023-06-10T18:48:03.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.984+0000 7fc38b7fe700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 0x7fc39006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:03.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.984+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc398089a90 con 0x7fc3a406af80 2023-06-10T18:48:03.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.984+0000 7fc3a259c700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 0x7fc39006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:03.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.985+0000 7fc3a259c700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 0x7fc39006d190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40b4260 tx=0x7fc394008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:03.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:03.989+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc398058520 con 0x7fc3a406af80 2023-06-10T18:48:04.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.255+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc3a406c600 con 0x7fc3a406af80 2023-06-10T18:48:04.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.259+0000 7fc38b7fe700 1 -- 172.21.15.104:0/1639194621 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7fc3980580b0 con 0x7fc3a406af80 2023-06-10T18:48:04.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 msgr2=0x7fc39006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 0x7fc39006d190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40b4260 tx=0x7fc394008040 comp rx=0 tx=0).stop 2023-06-10T18:48:04.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 msgr2=0x7fc3a4157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc39800d800 tx=0x7fc39800dbc0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 shutdown_connections 2023-06-10T18:48:04.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3a415fd00 0x7fc3a40b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3a406af80 0x7fc3a4157c10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc39006ace0 0x7fc39006d190 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 --2- 172.21.15.104:0/1639194621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3a40ccff0 0x7fc3a40b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 >> 172.21.15.104:0/1639194621 conn(0x7fc3a4089fd0 msgr2=0x7fc3a406ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:04.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 shutdown_connections 2023-06-10T18:48:04.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.261+0000 7fc3a9014700 1 -- 172.21.15.104:0/1639194621 wait complete. 2023-06-10T18:48:04.270 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:48:04.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.392+0000 7fcdd0423700 1 Processor -- start 2023-06-10T18:48:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.392+0000 7fcdd0423700 1 -- start start 2023-06-10T18:48:04.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdd0423700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcdc806e4b0 0x7fcdc806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdd0423700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc8069e70 0x7fcdc80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdd0423700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc806d290 con 0x7fcdc806e0e0 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdd0423700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc8071640 con 0x7fcdc8069e70 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdd0423700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc80aef00 con 0x7fcdc806e4b0 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc8069e70 0x7fcdc80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcdc8069e70 0x7fcdc80ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58628/0 (socket says 172.21.15.104:58628) 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 -- 172.21.15.104:0/3032356813 learned_addr learned my addr 172.21.15.104:0/3032356813 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdcd9be700 1 --2- 172.21.15.104:0/3032356813 >> v2:172.21.15.114:3300/0 conn(0x7fcdc806e4b0 0x7fcdc806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 -- 172.21.15.104:0/3032356813 >> v2:172.21.15.114:3300/0 conn(0x7fcdc806e4b0 msgr2=0x7fcdc806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 --2- 172.21.15.104:0/3032356813 >> v2:172.21.15.114:3300/0 conn(0x7fcdc806e4b0 0x7fcdc806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 -- 172.21.15.104:0/3032356813 >> v1:172.21.15.114:6789/0 conn(0x7fcdc806e0e0 legacy=0x7fcdc80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 -- 172.21.15.104:0/3032356813 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc80ca2d0 con 0x7fcdc8069e70 2023-06-10T18:48:04.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdce9c0700 1 --2- 172.21.15.104:0/3032356813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc8069e70 0x7fcdc80ce9f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8070320 tx=0x7fcdc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1356ba0ef87f709 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdcd1bd700 1 -- 172.21.15.104:0/3032356813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc000e070 con 0x7fcdc8069e70 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdcd1bd700 1 -- 172.21.15.104:0/3032356813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc000b910 con 0x7fcdc8069e70 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.393+0000 7fcdcd1bd700 1 -- 172.21.15.104:0/3032356813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc000bcc0 con 0x7fcdc8069e70 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 -- 172.21.15.104:0/3032356813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc8069e70 msgr2=0x7fcdc80ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 --2- 172.21.15.104:0/3032356813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc8069e70 0x7fcdc80ce9f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8070320 tx=0x7fcdc000b330 comp rx=0 tx=0).stop 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 -- 172.21.15.104:0/3032356813 shutdown_connections 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 --2- 172.21.15.104:0/3032356813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc8069e70 0x7fcdc80ce9f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 --2- 172.21.15.104:0/3032356813 >> v2:172.21.15.114:3300/0 conn(0x7fcdc806e4b0 0x7fcdc806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 -- 172.21.15.104:0/3032356813 >> 172.21.15.104:0/3032356813 conn(0x7fcdc806b8a0 msgr2=0x7fcdc80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 -- 172.21.15.104:0/3032356813 shutdown_connections 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.394+0000 7fcdd0423700 1 -- 172.21.15.104:0/3032356813 wait complete. 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.395+0000 7fcdd0423700 1 Processor -- start 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.395+0000 7fcdd0423700 1 -- start start 2023-06-10T18:48:04.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 0x7fcdc8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 0x7fcdc80adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc80b2430 con 0x7fcdc806e4b0 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc80b1dd0 con 0x7fcdc8069e70 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdd0423700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc80b2680 con 0x7fcdc806e0e0 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58634/0 (socket says 172.21.15.104:58634) 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 -- 172.21.15.104:0/351193443 learned_addr learned my addr 172.21.15.104:0/351193443 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdcd9be700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 0x7fcdc80adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce1bf700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 0x7fcdc8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 -- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 msgr2=0x7fcdc80adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 0x7fcdc80adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 -- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 msgr2=0x7fcdc8155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 0x7fcdc8155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 -- 172.21.15.104:0/351193443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc0009b30 con 0x7fcdc806e4b0 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce1bf700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 0x7fcdc8155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdcd9be700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 0x7fcdc80adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdce9c0700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000b370 tx=0x7fcdc0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00099b0 con 0x7fcdc806e4b0 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.396+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc00092e0 con 0x7fcdc806e4b0 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.397+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc80b2960 con 0x7fcdc806e4b0 2023-06-10T18:48:04.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.397+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc001b650 con 0x7fcdc806e4b0 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.397+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc81559a0 con 0x7fcdc806e4b0 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.398+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc8074050 con 0x7fcdc806e4b0 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.398+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcdc001b7b0 con 0x7fcdc806e4b0 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.398+0000 7fcdbeffd700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 0x7fcdac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.398+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcdc008ae50 con 0x7fcdc806e4b0 2023-06-10T18:48:04.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.399+0000 7fcdce1bf700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 0x7fcdac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:04.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.399+0000 7fcdce1bf700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 0x7fcdac06d190 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80c8670 tx=0x7fcdc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:04.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.403+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcdc0055e50 con 0x7fcdc806e4b0 2023-06-10T18:48:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.632+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdc80ad6d0 con 0x7fcdc806e4b0 2023-06-10T18:48:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.632+0000 7fcdbeffd700 1 -- 172.21.15.104:0/351193443 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7fcdc0059470 con 0x7fcdc806e4b0 2023-06-10T18:48:04.637 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:04.637 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 msgr2=0x7fcdac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 0x7fcdac06d190 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80c8670 tx=0x7fcdc4009040 comp rx=0 tx=0).stop 2023-06-10T18:48:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 msgr2=0x7fcdc80ae130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000b370 tx=0x7fcdc0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 shutdown_connections 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcdac06ace0 0x7fcdac06d190 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdc8069e70 0x7fcdc8155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdc806e4b0 0x7fcdc80ae130 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 --2- 172.21.15.104:0/351193443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdc806e0e0 0x7fcdc80adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 >> 172.21.15.104:0/351193443 conn(0x7fcdc806b8a0 msgr2=0x7fcdc80cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 shutdown_connections 2023-06-10T18:48:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:04.634+0000 7fcdd0423700 1 -- 172.21.15.104:0/351193443 wait complete. 2023-06-10T18:48:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:04.844 INFO:tasks.ceph.mds.b.smithi114.stderr:2023-06-10T18:48:04.839+0000 7fcfbb8f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:48:05.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:47:59.237789+0000', 'modified': '2023-06-10T18:48:02.253448+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6189}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6189': {'gid': 6189, 'name': 'b', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/1791317446', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 1791317446}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 1791317446}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:48:05.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:48:05.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-06-10T18:48:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.171+0000 7f60ac388700 1 Processor -- start 2023-06-10T18:48:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.172+0000 7f60ac388700 1 -- start start 2023-06-10T18:48:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.172+0000 7f60ac388700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f60a406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60ac388700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a415cb00 con 0x7f60a40cc000 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60ac388700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a406ecd0 con 0x7f60a40cb0c0 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60ac388700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40aed80 con 0x7f60a415c730 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9923700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f60a40cc000 0x7f60a406cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56466/0 (socket says 172.21.15.104:56466) 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9923700 1 -- 172.21.15.104:0/603251487 learned_addr learned my addr 172.21.15.104:0/603251487 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814311492 0 0) 0x7f60a0001110 con 0x7f60a40cc000 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60900035b0 con 0x7f60a40cc000 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60aa124700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f60a406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3119681528 0 0) 0x7f60900035b0 con 0x7f60a40cc000 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 msgr2=0x7f60a406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f60a406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 >> v1:172.21.15.114:6789/0 conn(0x7f60a40cb0c0 legacy=0x7f60a406d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a406db90 con 0x7f60a40cc000 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f60a0001800 con 0x7f60a40cc000 2023-06-10T18:48:05.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 >> v1:172.21.15.104:6789/0 conn(0x7f60a40cc000 legacy=0x7f60a406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6090003a30 0x7f6090003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f6090004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6090004d90 0x7f6090005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40aed80 con 0x7f6090003a30 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609000bac0 con 0x7f6090004d90 2023-06-10T18:48:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609000bc50 con 0x7f60a415c730 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60aa124700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6090003a30 0x7f6090003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.173+0000 7f60a9923700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f6090004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa925700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6090004d90 0x7f6090005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 -- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 msgr2=0x7f6090004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60a415c730 0x7f6090004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 -- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6090004d90 msgr2=0x7f6090005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6090004d90 0x7f6090005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 -- 172.21.15.104:0/603251487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60900035b0 con 0x7f6090003a30 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa925700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6090004d90 0x7f6090005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa124700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6090003a30 0x7f6090003e00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6094009b40 tx=0x7f6090005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32081c389ccd1df server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094005a20 con 0x7f6090003a30 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094005b80 con 0x7f6090003a30 2023-06-10T18:48:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60a9122700 1 -- 172.21.15.104:0/603251487 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609400e6c0 con 0x7f6090003a30 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60ac388700 1 -- 172.21.15.104:0/603251487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6090003a30 msgr2=0x7f6090003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60ac388700 1 --2- 172.21.15.104:0/603251487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6090003a30 0x7f6090003e00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6094009b40 tx=0x7f6090005880 comp rx=0 tx=0).stop 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.174+0000 7f60aa925700 1 -- 172.21.15.104:0/603251487 reap_dead start 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.175+0000 7f60ac388700 1 -- 172.21.15.104:0/603251487 shutdown_connections 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.175+0000 7f60ac388700 1 -- 172.21.15.104:0/603251487 >> 172.21.15.104:0/603251487 conn(0x7f60a406b8a0 msgr2=0x7f60a40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.175+0000 7f60ac388700 1 -- 172.21.15.104:0/603251487 shutdown_connections 2023-06-10T18:48:05.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.175+0000 7f60ac388700 1 -- 172.21.15.104:0/603251487 wait complete. 2023-06-10T18:48:05.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.176+0000 7f60ac388700 1 Processor -- start 2023-06-10T18:48:05.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.176+0000 7f60ac388700 1 -- start start 2023-06-10T18:48:05.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6090003a30 0x7f60a4157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40b1270 0x7f60a40b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40b1c00 con 0x7f60a40c4360 2023-06-10T18:48:05.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40b2e50 con 0x7f60a40b1270 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60ac388700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40b3130 con 0x7f6090003a30 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58652/0 (socket says 172.21.15.104:58652) 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 -- 172.21.15.104:0/1381293001 learned_addr learned my addr 172.21.15.104:0/1381293001 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa124700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6090003a30 0x7f60a4157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60a9923700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40b1270 0x7f60a40b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 -- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6090003a30 msgr2=0x7f60a4157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6090003a30 0x7f60a4157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 -- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40b1270 msgr2=0x7f60a40b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40b1270 0x7f60a40b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.177+0000 7f60aa925700 1 -- 172.21.15.104:0/1381293001 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a40b3410 con 0x7f60a40c4360 2023-06-10T18:48:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f60aa925700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b21a0 tx=0x7f609c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f609affd700 1 -- 172.21.15.104:0/1381293001 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c003c10 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f609affd700 1 -- 172.21.15.104:0/1381293001 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c003d90 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a40b36f0 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f609affd700 1 -- 172.21.15.104:0/1381293001 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c00c950 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.178+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40b3cb0 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.180+0000 7f609affd700 1 -- 172.21.15.104:0/1381293001 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f609c00cab0 con 0x7f60a40c4360 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.180+0000 7f609affd700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 0x7f608806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.180+0000 7f609affd700 1 -- 172.21.15.104:0/1381293001 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f609c089710 con 0x7f60a40c4360 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.180+0000 7f60aa124700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 0x7f608806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.181+0000 7f60aa124700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 0x7f608806d190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b2070 tx=0x7f6094008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.182+0000 7f60b850db80 1 Processor -- start 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.182+0000 7f60b850db80 1 -- start start 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651feca6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651fecccf00 0x5651fecd01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651febaac60 con 0x5651feca3740 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651febfd9f0 con 0x5651feca6270 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651febf6e80 con 0x5651fecccf00 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58658/0 (socket says 172.21.15.104:58658) 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 -- 172.21.15.104:0/1820458410 learned_addr learned my addr 172.21.15.104:0/1820458410 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60aa124700 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651feca6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60aa925700 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651fecccf00 0x5651fecd01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 -- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651fecccf00 msgr2=0x5651fecd01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651fecccf00 0x5651fecd01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 -- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 msgr2=0x5651feca6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651feca6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 -- 172.21.15.104:0/1820458410 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651fea785e0 con 0x5651feca3740 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f60a9923700 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f60a00018d0 tx=0x7f60a00011b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa57571a8ed59015 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f6087fff700 1 -- 172.21.15.104:0/1820458410 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0004e50 con 0x5651feca3740 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f6087fff700 1 -- 172.21.15.104:0/1820458410 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0007390 con 0x5651feca3740 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.183+0000 7f6087fff700 1 -- 172.21.15.104:0/1820458410 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0007720 con 0x5651feca3740 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 -- 172.21.15.104:0/1820458410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 msgr2=0x5651feccc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f60a00018d0 tx=0x7f60a00011b0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 -- 172.21.15.104:0/1820458410 shutdown_connections 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651feca6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651fecccf00 0x5651fecd01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 --2- 172.21.15.104:0/1820458410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651feccc9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 -- 172.21.15.104:0/1820458410 >> 172.21.15.104:0/1820458410 conn(0x5651fec9afc0 msgr2=0x5651feceb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 -- 172.21.15.104:0/1820458410 shutdown_connections 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.184+0000 7f60b850db80 1 -- 172.21.15.104:0/1820458410 wait complete. 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.185+0000 7f60b850db80 1 -- start start 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.185+0000 7f60b850db80 1 Processor -- start 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fececcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 0x5651feced210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651feb96a70 con 0x5651feca3740 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651feb96770 con 0x5651fecccf00 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651fea7a8d0 con 0x5651feca6270 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60b850db80 10 client.0 _ll_register_callbacks cb 0x5651feb9c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58662/0 (socket says 172.21.15.104:58662) 2023-06-10T18:48:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 -- 172.21.15.104:0/4122088968 learned_addr learned my addr 172.21.15.104:0/4122088968 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60a9923700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fececcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa925700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 0x5651feced210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 -- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 msgr2=0x5651fececcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fececcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 -- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 msgr2=0x5651feced210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 0x5651feced210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 -- 172.21.15.104:0/4122088968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a406db90 con 0x5651feca3740 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa925700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 0x5651feced210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60a9923700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fececcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60aa124700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6094006000 tx=0x7f6094004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d44140cebf1247d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60857fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.186+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094006230 con 0x5651feca3740 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6094035d90 con 0x5651feca3740 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651fec98780 con 0x5651feca3740 2023-06-10T18:48:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6094006390 con 0x5651feca3740 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 10 client.6330 fetch_fsmap learned FSMap version 25 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5651fea24d90 con 0x5651feca3740 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5651fec98780 con 0x5651feca3740 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f60940064f0 con 0x5651feca3740 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 10 client.6330 fetch_fsmap finished waiting for FSMap version 25 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 10 client.6330 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:48:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 20 client.6330 populate_metadata read hostname 'smithi104' 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651feca40d0 0x5651feca44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 4 client.6330 mds_command: new command op to 6189 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 --> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651febefe60 con 0x5651feca40d0 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.187+0000 7f60a9923700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651feca40d0 0x5651feca44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60857fa700 10 client.6330 ms_handle_connect on v2:172.21.15.114:6834/1791317446 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60a9923700 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651feca40d0 0x5651feca44a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e79b165ecf4a9285 server_cookie=6a98624d4a5b86d7 in_seq=0 out_seq=0 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60857fa700 1 -- 172.21.15.104:0/4122088968 <== mds.0 v2:172.21.15.114:6834/1791317446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5651febefe60 con 0x5651feca40d0 2023-06-10T18:48:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60857fa700 10 client.6330 handle_command_reply: tid=0 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 client.6330 shutdown 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 msgr2=0x5651fecc4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6094006000 tx=0x7f6094004750 comp rx=0 tx=0).stop 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 shutdown_connections 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651feca40d0 0x5651feca44a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651fecccf00 0x5651feced210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651feca3740 0x5651fecc4e20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 --2- 172.21.15.104:0/4122088968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fececcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60857fa700 0 client.6330 ms_handle_reset on v2:172.21.15.114:6834/1791317446 2023-06-10T18:48:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.188+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 >> 172.21.15.104:0/4122088968 conn(0x5651fec9afc0 msgr2=0x5651feccc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.189+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 shutdown_connections 2023-06-10T18:48:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.189+0000 7f60b850db80 1 -- 172.21.15.104:0/4122088968 wait complete. 2023-06-10T18:48:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.189+0000 7f60b850db80 20 client.6330 trim_cache size 0 max 16384 2023-06-10T18:48:05.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.206+0000 7f60b850db80 1 Processor -- start 2023-06-10T18:48:05.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.206+0000 7f60b850db80 1 -- start start 2023-06-10T18:48:05.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca3740 0x5651feceebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fec9f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fecec920 con 0x5651fecccf00 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651feccc810 con 0x5651feca3740 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651feced480 con 0x5651feca6270 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58672/0 (socket says 172.21.15.104:58672) 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 -- 172.21.15.104:0/3779164760 learned_addr learned my addr 172.21.15.104:0/3779164760 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 -- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 msgr2=0x5651fec9f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fec9f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 -- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca3740 msgr2=0x5651feceebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca3740 0x5651feceebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 -- 172.21.15.104:0/3779164760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651fea0dc10 con 0x5651fecccf00 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f60aa925700 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f609c058040 tx=0x7f609c089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9162d9ac1bf85064 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.207+0000 7f6098dac700 1 -- 172.21.15.104:0/3779164760 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c060690 con 0x5651fecccf00 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f6098dac700 1 -- 172.21.15.104:0/3779164760 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0566e0 con 0x5651fecccf00 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f6098dac700 1 -- 172.21.15.104:0/3779164760 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c057830 con 0x5651fecccf00 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 -- 172.21.15.104:0/3779164760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 msgr2=0x5651feca27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f609c058040 tx=0x7f609c089040 comp rx=0 tx=0).stop 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 -- 172.21.15.104:0/3779164760 shutdown_connections 2023-06-10T18:48:05.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca3740 0x5651feceebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651feca27a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 --2- 172.21.15.104:0/3779164760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca6270 0x5651fec9f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 -- 172.21.15.104:0/3779164760 >> 172.21.15.104:0/3779164760 conn(0x5651fec9afc0 msgr2=0x5651fecd3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 -- 172.21.15.104:0/3779164760 shutdown_connections 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.208+0000 7f60b850db80 1 -- 172.21.15.104:0/3779164760 wait complete. 2023-06-10T18:48:05.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.209+0000 7f60b850db80 1 -- start start 2023-06-10T18:48:05.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.209+0000 7f60b850db80 1 Processor -- start 2023-06-10T18:48:05.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca3740 0x5651fecfda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651fecd02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fecfe040 con 0x5651fecccf00 2023-06-10T18:48:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651feca50c0 con 0x5651feca6270 2023-06-10T18:48:05.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651fecef0a0 con 0x5651feca3740 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 10 client.0 _ll_register_callbacks cb 0x5651feb9c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58680/0 (socket says 172.21.15.104:58680) 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 -- 172.21.15.104:0/186586149 learned_addr learned my addr 172.21.15.104:0/186586149 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa124700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca3740 0x5651fecfda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60a9923700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651fecd02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 -- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca3740 msgr2=0x5651fecfda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca3740 0x5651fecfda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 -- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 msgr2=0x5651fecd02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651fecd02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 -- 172.21.15.104:0/186586149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40b3cb0 con 0x5651fecccf00 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60aa925700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f609c0879f0 tx=0x7f609c001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8becd90ea73b5d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60857fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:48:05.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c057a30 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f609c075030 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a0007390 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f609c060200 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.210+0000 7f60b850db80 10 client.6336 fetch_fsmap learned FSMap version 25 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5651fea4b200 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0007390 con 0x5651fecccf00 2023-06-10T18:48:05.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f609c071d20 con 0x5651fecccf00 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 10 client.6336 fetch_fsmap finished waiting for FSMap version 25 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 10 client.6336 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 20 client.6336 populate_metadata read hostname 'smithi104' 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651fecca450 0x5651fecca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 4 client.6336 mds_command: new command op to 6189 tid=0[{"prefix": "status"}] 2023-06-10T18:48:05.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 --> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5651febb2000 con 0x5651fecca450 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60aa124700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651fecca450 0x5651fecca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60857fa700 10 client.6336 ms_handle_connect on v2:172.21.15.114:6834/1791317446 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60aa124700 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651fecca450 0x5651fecca820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db3f0b80a320ce5 server_cookie=64987eea9d085eb7 in_seq=0 out_seq=0 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60857fa700 1 -- 172.21.15.104:0/186586149 <== mds.0 v2:172.21.15.114:6834/1791317446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5651febb2000 con 0x5651fecca450 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.211+0000 7f60857fa700 10 client.6336 handle_command_reply: tid=0 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 client.6336 shutdown 2023-06-10T18:48:05.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 msgr2=0x5651fecea290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f609c0879f0 tx=0x7f609c001490 comp rx=0 tx=0).stop 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 shutdown_connections 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:6834/1791317446,v1:172.21.15.114:6836/1791317446] conn(0x5651fecca450 0x5651fecca820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5651fecccf00 0x5651fecea290 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5651feca3740 0x5651fecfda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 --2- 172.21.15.104:0/186586149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651feca6270 0x5651fecd02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60857fa700 0 client.6336 ms_handle_reset on v2:172.21.15.114:6834/1791317446 2023-06-10T18:48:05.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 >> 172.21.15.104:0/186586149 conn(0x5651fec9afc0 msgr2=0x5651fec9f530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 shutdown_connections 2023-06-10T18:48:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 1 -- 172.21.15.104:0/186586149 wait complete. 2023-06-10T18:48:05.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60b850db80 20 client.6336 trim_cache size 0 max 16384 2023-06-10T18:48:05.230 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:05.230 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6189, 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.9721049869999998, 2023-06-10T18:48:05.231 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 25, 2023-06-10T18:48:05.232 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 56, 2023-06-10T18:48:05.232 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 56, 2023-06-10T18:48:05.232 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.297424762 2023-06-10T18:48:05.232 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:05.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 msgr2=0x7f608806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 0x7f608806d190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b2070 tx=0x7f6094008030 comp rx=0 tx=0).stop 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 msgr2=0x7f60a40c47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b21a0 tx=0x7f609c00ef20 comp rx=0 tx=0).stop 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 shutdown_connections 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60a40c4360 0x7f60a40c47e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6090003a30 0x7f60a4157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f608806ace0 0x7f608806d190 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 --2- 172.21.15.104:0/1381293001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40b1270 0x7f60a40b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 >> 172.21.15.104:0/1381293001 conn(0x7f60a406b8a0 msgr2=0x7f60a40ad580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 shutdown_connections 2023-06-10T18:48:05.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.212+0000 7f60ac388700 1 -- 172.21.15.104:0/1381293001 wait complete. 2023-06-10T18:48:05.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:48:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.351+0000 7fc41733f700 1 Processor -- start 2023-06-10T18:48:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.351+0000 7fc41733f700 1 -- start start 2023-06-10T18:48:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc41733f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc4100cb300 0x7fc410163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc41733f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4100c9e60 con 0x7fc4100aeaa0 2023-06-10T18:48:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc41733f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4100aee70 con 0x7fc4100cd850 2023-06-10T18:48:05.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc41733f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4100c9bc0 con 0x7fc4100cb300 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc4150db700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc4100aeaa0 0x7fc4100cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56512/0 (socket says 172.21.15.104:56512) 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc4150db700 1 -- 172.21.15.104:0/3109890755 learned_addr learned my addr 172.21.15.104:0/3109890755 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc4158dc700 1 --2- 172.21.15.104:0/3109890755 >> v2:172.21.15.114:3300/0 conn(0x7fc4100cb300 0x7fc410163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571260069 0 0) 0x7fc4100c9e60 con 0x7fc4100aeaa0 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.352+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc0035b0 con 0x7fc4100aeaa0 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451900845 0 0) 0x7fc3fc0035b0 con 0x7fc4100aeaa0 2023-06-10T18:48:05.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 >> v2:172.21.15.114:3300/0 conn(0x7fc4100cb300 msgr2=0x7fc410163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 --2- 172.21.15.104:0/3109890755 >> v2:172.21.15.114:3300/0 conn(0x7fc4100cb300 0x7fc410163c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 >> v1:172.21.15.114:6789/0 conn(0x7fc4100cd850 legacy=0x7fc41006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41006e540 con 0x7fc4100aeaa0 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc4000017f0 con 0x7fc4100aeaa0 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 >> v1:172.21.15.104:6789/0 conn(0x7fc4100aeaa0 legacy=0x7fc4100cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc3fc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3fc004850 0x7fc3fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3fc005200 0x7fc3fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4100c9bc0 con 0x7fc3fc003a00 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc00c080 con 0x7fc3fc004850 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc00c240 con 0x7fc3fc005200 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc3fc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4148da700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3fc004850 0x7fc3fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4158dc700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3fc005200 0x7fc3fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 -- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3fc005200 msgr2=0x7fc3fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3fc005200 0x7fc3fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 -- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3fc004850 msgr2=0x7fc3fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3fc004850 0x7fc3fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 -- 172.21.15.104:0/3109890755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc0035b0 con 0x7fc3fc003a00 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4158dc700 1 -- 172.21.15.104:0/3109890755 reap_dead start 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc4150db700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc3fc003e10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc400007d20 tx=0x7fc3fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8648b72c86c30da6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40000cd30 con 0x7fc3fc003a00 2023-06-10T18:48:05.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40000ce90 con 0x7fc3fc003a00 2023-06-10T18:48:05.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.353+0000 7fc407fff700 1 -- 172.21.15.104:0/3109890755 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000072d0 con 0x7fc3fc003a00 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 -- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 msgr2=0x7fc3fc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc3fc003e10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc400007d20 tx=0x7fc3fc005880 comp rx=0 tx=0).stop 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 -- 172.21.15.104:0/3109890755 shutdown_connections 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 --2- 172.21.15.104:0/3109890755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc3fc003e10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 -- 172.21.15.104:0/3109890755 >> 172.21.15.104:0/3109890755 conn(0x7fc41006b8a0 msgr2=0x7fc4100cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 -- 172.21.15.104:0/3109890755 shutdown_connections 2023-06-10T18:48:05.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.354+0000 7fc41733f700 1 -- 172.21.15.104:0/3109890755 wait complete. 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.355+0000 7fc41733f700 1 Processor -- start 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.355+0000 7fc41733f700 1 -- start start 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc410151ac0 0x7fc4100c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100c9420 0x7fc4101f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4100c9890 con 0x7fc3fc003a00 2023-06-10T18:48:05.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410159850 con 0x7fc410151ac0 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc41733f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc410159b30 con 0x7fc4100c9420 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58696/0 (socket says 172.21.15.104:58696) 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 -- 172.21.15.104:0/3460608912 learned_addr learned my addr 172.21.15.104:0/3460608912 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 -- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100c9420 msgr2=0x7fc4101f52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100c9420 0x7fc4101f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 -- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc410151ac0 msgr2=0x7fc4100c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4148da700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc410151ac0 0x7fc4100c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc410151ac0 0x7fc4100c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.356+0000 7fc4150db700 1 -- 172.21.15.104:0/3460608912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410159e10 con 0x7fc3fc003a00 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc4150db700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc40000c800 tx=0x7fc400001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000015b0 con 0x7fc3fc003a00 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc400001730 con 0x7fc3fc003a00 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400005a70 con 0x7fc3fc003a00 2023-06-10T18:48:05.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41015a0f0 con 0x7fc3fc003a00 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.357+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc410159470 con 0x7fc3fc003a00 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.358+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41015afa0 con 0x7fc3fc003a00 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.359+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc400005bd0 con 0x7fc3fc003a00 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.359+0000 7fc405ffb700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.359+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc400089d30 con 0x7fc3fc003a00 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.359+0000 7fc4148da700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:05.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.360+0000 7fc4148da700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000c00 tx=0x7fc40c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:05.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.363+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc41015afa0 con 0x7fc3fc003a00 2023-06-10T18:48:05.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.595+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc41015afa0 con 0x7fc3fc003a00 2023-06-10T18:48:05.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.595+0000 7fc405ffb700 1 -- 172.21.15.104:0/3460608912 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc400029060 con 0x7fc3fc003a00 2023-06-10T18:48:05.600 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:48:05.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 msgr2=0x7fc3f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000c00 tx=0x7fc40c009040 comp rx=0 tx=0).stop 2023-06-10T18:48:05.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 msgr2=0x7fc4100ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:05.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc40000c800 tx=0x7fc400001b80 comp rx=0 tx=0).stop 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 shutdown_connections 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4100c9420 0x7fc4101f52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3fc003a00 0x7fc4100ad9d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc3f406ad10 0x7fc3f406d1c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 --2- 172.21.15.104:0/3460608912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc410151ac0 0x7fc4100c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 >> 172.21.15.104:0/3460608912 conn(0x7fc41006b8a0 msgr2=0x7fc4100ca7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:05.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 shutdown_connections 2023-06-10T18:48:05.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:05.597+0000 7fc41733f700 1 -- 172.21.15.104:0/3460608912 wait complete. 2023-06-10T18:48:05.614 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:48:05.614 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:48:05.614 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:48:05.615 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:48:05.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:05.615 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:48:05.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:48:05.616 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:48:05.616 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:05.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.617 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:48:05.634 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:48:05.634 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:05.635 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42199sec preferred_lft 42199sec 2023-06-10T18:48:05.636 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.637 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:48:05.638 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:48:05.639 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:05.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.639 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:48:05.703 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:48:05.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.705 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:48:05.767 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:48:05.767 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:48:05.767 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:48:05.767 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:48:05.767 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:48:05.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.768 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:48:05.769 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:48:05.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:48:05.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:48:05.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:48:05.770 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:48:05.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.881 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:48:05.900 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:48:05.900 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:48:05.901 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:05.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:05.901 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:48:05.967 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:48:05.967 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:05.968 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:06.028 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:06.028 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:06.029 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:06.088 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:06.155 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:06.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:06.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:06.225 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:06.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:48:06.286 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:48:06.329 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:06.359 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:06.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:06.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:48:06.411+0000 7f98317be5c0 -1 init, newargv = 0x561b2be26420 newargc=15 2023-06-10T18:48:06.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[123861]: starting ceph client 2023-06-10T18:48:06.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[123861]: starting fuse 2023-06-10T18:48:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:06.432 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:06.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:06.434 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:06.498 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:48:06.499 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:48:06.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:06.500 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:48:06.500 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:48:06.500 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:48:06.501 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:48:06.501 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:06.501 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:48:06.502 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:48:06.502 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:48:06.503 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:48:06.504 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:06.504 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:48:06.505 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:48:06.506 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:48:06.507 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:48:06.508 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:06.508 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:48:06.509 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:06.509 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:06.510 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:06.510 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:48:06.511 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:48:06.511 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:48:06.511 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:06.511 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:06.512 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:48:06.513 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:48:06.628 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.123861.asok 2023-06-10T18:48:06.811 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.123861.asok 2023-06-10T18:48:06.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:06.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.123861.asok status 2023-06-10T18:48:06.971 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:06.971 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "123861", 2023-06-10T18:48:06.972 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6348, 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:48:06.973 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6348 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3450172328 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:06.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3450172328 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:06.975 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.6348 192.168.0.1:0/3450172328", 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3450172328", 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 25, 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 56, 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:48:06.976 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:06.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:06.981 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:48:06.982 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:06.998 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:48:06.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:06.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.000 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:07.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:48:07.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.234+0000 7f56d2cf8700 1 Processor -- start 2023-06-10T18:48:07.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 -- start start 2023-06-10T18:48:07.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56cc0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56cc0ce890 0x7f56cc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0b6070 con 0x7f56cc15a420 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0ced10 con 0x7f56cc0ce890 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.235+0000 7f56d2cf8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b1d70 con 0x7f56cc0b4ff0 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56d0a94700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f56cc15a420 0x7f56cc0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56552/0 (socket says 172.21.15.104:56552) 2023-06-10T18:48:07.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56d0a94700 1 -- 172.21.15.104:0/1390239012 learned_addr learned my addr 172.21.15.104:0/1390239012 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:07.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291390550 0 0) 0x7f56cc0b6070 con 0x7f56cc15a420 2023-06-10T18:48:07.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56cc0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56bc0035b0 con 0x7f56cc15a420 2023-06-10T18:48:07.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56d1295700 1 --2- 172.21.15.104:0/1390239012 >> v2:172.21.15.114:3300/0 conn(0x7f56cc0ce890 0x7f56cc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3829323448 0 0) 0x7f56bc0035b0 con 0x7f56cc15a420 2023-06-10T18:48:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 msgr2=0x7f56cc0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56cc0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 >> v2:172.21.15.114:3300/0 conn(0x7f56cc0ce890 msgr2=0x7f56cc15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:07.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 --2- 172.21.15.104:0/1390239012 >> v2:172.21.15.114:3300/0 conn(0x7f56cc0ce890 0x7f56cc15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0b5460 con 0x7f56cc15a420 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f56c0001670 con 0x7f56cc15a420 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 >> v1:172.21.15.104:6789/0 conn(0x7f56cc15a420 legacy=0x7f56cc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56bc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56bc003f80 0x7f56bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc004d90 0x7f56bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0ced10 con 0x7f56bc003f80 2023-06-10T18:48:07.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b1d70 con 0x7f56bc004d90 2023-06-10T18:48:07.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc00ba10 con 0x7f56cc0b4ff0 2023-06-10T18:48:07.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56bc003f80 0x7f56bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56d1295700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc004d90 0x7f56bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56d0a94700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56bc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 -- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 msgr2=0x7f56bc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc0b4ff0 0x7f56bc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 -- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc004d90 msgr2=0x7f56bc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc004d90 0x7f56bc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.236+0000 7f56cbfff700 1 -- 172.21.15.104:0/1390239012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc0035b0 con 0x7f56bc003f80 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56cbfff700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56bc003f80 0x7f56bc004850 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009b60 tx=0x7f56bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9addc9abc251ba1e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:07.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800e6c0 con 0x7f56bc003f80 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b800e820 con 0x7f56bc003f80 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56cb7fe700 1 -- 172.21.15.104:0/1390239012 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800eb90 con 0x7f56bc003f80 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 -- 172.21.15.104:0/1390239012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56bc003f80 msgr2=0x7f56bc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/1390239012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56bc003f80 0x7f56bc004850 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009b60 tx=0x7f56bc005880 comp rx=0 tx=0).stop 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d1295700 1 -- 172.21.15.104:0/1390239012 reap_dead start 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 -- 172.21.15.104:0/1390239012 shutdown_connections 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 -- 172.21.15.104:0/1390239012 >> 172.21.15.104:0/1390239012 conn(0x7f56cc089fd0 msgr2=0x7f56cc0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:07.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 -- 172.21.15.104:0/1390239012 shutdown_connections 2023-06-10T18:48:07.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.237+0000 7f56d2cf8700 1 -- 172.21.15.104:0/1390239012 wait complete. 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.239+0000 7f56d2cf8700 1 Processor -- start 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.239+0000 7f56d2cf8700 1 -- start start 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc003f80 0x7f56cc06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc06e4a0 0x7f56cc0c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc06e9f0 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc06ecd0 con 0x7f56bc003f80 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d2cf8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc06efb0 con 0x7f56cc06e4a0 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58740/0 (socket says 172.21.15.104:58740) 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 -- 172.21.15.104:0/2507320019 learned_addr learned my addr 172.21.15.104:0/2507320019 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 -- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc06e4a0 msgr2=0x7f56cc0c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc06e4a0 0x7f56cc0c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 -- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc003f80 msgr2=0x7f56cc06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc003f80 0x7f56cc06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 -- 172.21.15.104:0/2507320019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc06f290 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.240+0000 7f56d1295700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f56cc072200 tx=0x7f56c40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.241+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c400e070 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.241+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4003c30 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.241+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc06f570 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.241+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4003e20 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.241+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc0c81e0 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.242+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc06f760 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.242+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f56c4011a10 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.242+0000 7f56c97fa700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 0x7f56b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:07.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.243+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f56c40898b0 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.243+0000 7f56d0a94700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 0x7f56b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:07.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.243+0000 7f56d0a94700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 0x7f56b006d190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f56c0001eb0 tx=0x7f56c0001e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:07.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.247+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f56c4054820 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.479+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f56cc0c81e0 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.479+0000 7f56c97fa700 1 -- 172.21.15.104:0/2507320019 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f56cc0c81e0 con 0x7f56cc0cb5f0 2023-06-10T18:48:07.485 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.481+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 msgr2=0x7f56b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 0x7f56b006d190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f56c0001eb0 tx=0x7f56c0001e40 comp rx=0 tx=0).stop 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 msgr2=0x7f56cc0cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f56cc072200 tx=0x7f56c40114b0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 shutdown_connections 2023-06-10T18:48:07.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f56b006ace0 0x7f56b006d190 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc003f80 0x7f56cc06fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56cc0cb5f0 0x7f56cc0cba40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 --2- 172.21.15.104:0/2507320019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56cc06e4a0 0x7f56cc0c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:07.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 >> 172.21.15.104:0/2507320019 conn(0x7f56cc089fd0 msgr2=0x7f56cc1e37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:07.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 shutdown_connections 2023-06-10T18:48:07.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:07.482+0000 7f56d2cf8700 1 -- 172.21.15.104:0/2507320019 wait complete. 2023-06-10T18:48:07.498 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:48:07.499 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:48:07.499 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:48:07.500 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:48:07.500 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:07.500 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:48:07.501 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:48:07.501 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:48:07.501 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:07.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.502 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:48:07.525 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:48:07.525 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:48:07.525 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:48:07.525 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.525 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:48:07.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.526 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:07.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:07.526 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:07.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:07.527 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:07.527 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:07.527 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:48:07.527 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:07.527 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:48:07.528 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42193sec preferred_lft 42193sec 2023-06-10T18:48:07.528 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:48:07.528 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.529 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:48:07.529 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:07.529 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:48:07.529 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.530 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:07.531 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:48:07.531 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:48:07.531 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:07.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.532 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:48:07.611 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:48:07.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.613 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:48:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:48:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:48:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:48:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:48:07.681 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:48:07.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.682 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:48:07.682 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:48:07.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:48:07.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:48:07.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:48:07.684 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:48:07.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.791 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:48:07.815 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:48:07.815 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:48:07.815 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:07.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:07.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:48:07.882 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:48:07.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:07.883 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:07.939 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:07.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:07.940 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:07.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:48:08.067 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:08.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:08.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:48:08.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:48:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:08.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:08.136 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:48:08.193 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:48:08.194 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:48:08.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:48:08.263 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:08.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:08.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:48:08.315+0000 7f0f2321b5c0 -1 init, newargv = 0x5570eca72ad0 newargc=15 2023-06-10T18:48:08.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[109373]: starting ceph client 2023-06-10T18:48:08.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[109373]: starting fuse 2023-06-10T18:48:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:08.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:08.331 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:48:08.387 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:48:08.387 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:48:08.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:08.388 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:48:08.389 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:48:08.389 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:48:08.389 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:48:08.390 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:08.390 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:48:08.390 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:48:08.390 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:48:08.391 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:48:08.391 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:08.391 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:48:08.392 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:48:08.392 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:48:08.392 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:48:08.393 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:08.393 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:48:08.394 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:08.394 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:08.394 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:08.395 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:48:08.395 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:48:08.395 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:48:08.396 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:08.396 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:08.396 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:48:08.397 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:48:08.509 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.109373.asok 2023-06-10T18:48:08.691 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.109373.asok 2023-06-10T18:48:08.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:08.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.109373.asok status 2023-06-10T18:48:08.846 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:48:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:48:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:48:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "109373", 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:48:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5366, 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5366 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4139818742 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4139818742 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.5366 192.168.0.1:0/4139818742", 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/4139818742", 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 25, 2023-06-10T18:48:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 56, 2023-06-10T18:48:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:48:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:48:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:48:08.851 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:48:08.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:08.856 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:48:08.856 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:08.873 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:48:08.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:08.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:08.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:08.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:48:09.085 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:48:09.086 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:48:09.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:48:09.221 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:48:09.221 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:48:09.222 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:09.222 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:09.239 INFO:tasks.cephfs.mount:Creating file a 2023-06-10T18:48:09.240 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.0/a 2023-06-10T18:48:09.300 INFO:tasks.cephfs.mount:Creating file b 2023-06-10T18:48:09.301 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.0/b 2023-06-10T18:48:09.360 INFO:tasks.cephfs.mount:Creating file c 2023-06-10T18:48:09.360 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.0/c 2023-06-10T18:48:09.419 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:09.419 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:09.476 INFO:tasks.cephfs.mount:Checking file a 2023-06-10T18:48:09.477 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.0/a 2023-06-10T18:48:09.532 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.0/a 2023-06-10T18:48:09.533 INFO:tasks.cephfs.mount:Checking file b 2023-06-10T18:48:09.533 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.0/b 2023-06-10T18:48:09.588 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.0/b 2023-06-10T18:48:09.589 INFO:tasks.cephfs.mount:Checking file c 2023-06-10T18:48:09.589 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.0/c 2023-06-10T18:48:09.645 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.0/c 2023-06-10T18:48:09.645 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:09.645 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:09.702 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 Processor -- start 2023-06-10T18:48:09.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 -- start start 2023-06-10T18:48:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc15a420 0x7f96cc0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc0b6070 con 0x7f96cc0b4ff0 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc0b5460 con 0x7f96cc0cea60 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96d1eb5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0b5740 con 0x7f96cc15a420 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96caffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f96cc0b4ff0 0x7f96cc0ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56572/0 (socket says 172.21.15.104:56572) 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96caffd700 1 -- 172.21.15.104:0/3906641664 learned_addr learned my addr 172.21.15.104:0/3906641664 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.857+0000 7f96cbfff700 1 --2- 172.21.15.104:0/3906641664 >> v2:172.21.15.104:3300/0 conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cb7fe700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc15a420 0x7f96cc0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cbfff700 1 -- 172.21.15.104:0/3906641664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc15a420 msgr2=0x7f96cc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cbfff700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc15a420 0x7f96cc0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cbfff700 1 -- 172.21.15.104:0/3906641664 >> v1:172.21.15.104:6789/0 conn(0x7f96cc0b4ff0 legacy=0x7f96cc0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cbfff700 1 -- 172.21.15.104:0/3906641664 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0b5a20 con 0x7f96cc0cea60 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96ca7fc700 1 -- 172.21.15.104:0/3906641664 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674406596 0 0) 0x7f96cc0b6070 con 0x7f96cc0b4ff0 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96cbfff700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009d90 tx=0x7f96c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2508440eef38029f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96ca7fc700 1 -- 172.21.15.104:0/3906641664 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c400e070 con 0x7f96cc0cea60 2023-06-10T18:48:09.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96ca7fc700 1 -- 172.21.15.104:0/3906641664 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c400bbd0 con 0x7f96cc0cea60 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.858+0000 7f96ca7fc700 1 -- 172.21.15.104:0/3906641664 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4012430 con 0x7f96cc0cea60 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 -- 172.21.15.104:0/3906641664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0cea60 msgr2=0x7f96cc15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009d90 tx=0x7f96c400b330 comp rx=0 tx=0).stop 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 -- 172.21.15.104:0/3906641664 shutdown_connections 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/3906641664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc15a420 0x7f96cc0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:09.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 -- 172.21.15.104:0/3906641664 >> 172.21.15.104:0/3906641664 conn(0x7f96cc089fd0 msgr2=0x7f96cc0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:09.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 -- 172.21.15.104:0/3906641664 shutdown_connections 2023-06-10T18:48:09.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.859+0000 7f96d1eb5700 1 -- 172.21.15.104:0/3906641664 wait complete. 2023-06-10T18:48:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.860+0000 7f96d1eb5700 1 Processor -- start 2023-06-10T18:48:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.860+0000 7f96d1eb5700 1 -- start start 2023-06-10T18:48:09.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 0x7f96cc0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc14c740 con 0x7f96cc15a420 2023-06-10T18:48:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15c570 con 0x7f96cc0b4ff0 2023-06-10T18:48:09.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15ad30 con 0x7f96cc0cea60 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58754/0 (socket says 172.21.15.104:58754) 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 -- 172.21.15.104:0/295441327 learned_addr learned my addr 172.21.15.104:0/295441327 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cb7fe700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 0x7f96cc0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96caffd700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 -- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 msgr2=0x7f96cc15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:09.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 -- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 msgr2=0x7f96cc0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 0x7f96cc0cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 -- 172.21.15.104:0/295441327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4009a70 con 0x7f96cc15a420 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cb7fe700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 0x7f96cc0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96caffd700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96cbfff700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f96c4013000 tx=0x7f96c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4012430 con 0x7f96cc15a420 2023-06-10T18:48:09.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4020070 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc15afd0 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.861+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4004280 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.862+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc15b590 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.862+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc15c340 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.863+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96c40043e0 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.863+0000 7f96b3fff700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 0x7f96b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.863+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f96c4022070 con 0x7f96cc15a420 2023-06-10T18:48:09.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.864+0000 7f96cb7fe700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 0x7f96b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.864+0000 7f96cb7fe700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 0x7f96b806d190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0c53e0 tx=0x7f96c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:09.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:09.868+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96cc15c340 con 0x7f96cc15a420 2023-06-10T18:48:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.053+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96cc15c340 con 0x7f96cc15a420 2023-06-10T18:48:10.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.054+0000 7f96b3fff700 1 -- 172.21.15.104:0/295441327 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f96cc15c340 con 0x7f96cc15a420 2023-06-10T18:48:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:10.060 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:10.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 msgr2=0x7f96b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 0x7f96b806d190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0c53e0 tx=0x7f96c0008040 comp rx=0 tx=0).stop 2023-06-10T18:48:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 msgr2=0x7f96cc0cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f96c4013000 tx=0x7f96c4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 shutdown_connections 2023-06-10T18:48:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96b806ace0 0x7f96b806d190 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96cc0b4ff0 0x7f96cc0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96cc15a420 0x7f96cc0cb100 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 --2- 172.21.15.104:0/295441327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96cc0cea60 0x7f96cc15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 >> 172.21.15.104:0/295441327 conn(0x7f96cc089fd0 msgr2=0x7f96cc157aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 shutdown_connections 2023-06-10T18:48:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.056+0000 7f96d1eb5700 1 -- 172.21.15.104:0/295441327 wait complete. 2023-06-10T18:48:10.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:47:57.299455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:47:58.209852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:10.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:10.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.193+0000 7f2786197700 1 Processor -- start 2023-06-10T18:48:10.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.193+0000 7f2786197700 1 -- start start 2023-06-10T18:48:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.193+0000 7f2786197700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.193+0000 7f2786197700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27800b21c0 0x7f27800b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f2786197700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800b2b40 con 0x7f278015d1a0 2023-06-10T18:48:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f2786197700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27800b30a0 con 0x7f27800b21c0 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f2786197700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278015d740 con 0x7f2780156760 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27800b21c0 0x7f27800b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27800b21c0 0x7f27800b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:46264/0 (socket says 172.21.15.104:46264) 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:49912/0 (socket says 172.21.15.104:49912) 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 -- 172.21.15.104:0/3715213351 learned_addr learned my addr 172.21.15.104:0/3715213351 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277effd700 1 --1- 172.21.15.104:0/3715213351 >> v1:172.21.15.114:6789/0 conn(0x7f278015d1a0 0x7f27800cad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:34200/0 (socket says 172.21.15.104:34200) 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 -- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 msgr2=0x7f27800ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 -- 172.21.15.104:0/3715213351 >> v1:172.21.15.114:6789/0 conn(0x7f278015d1a0 legacy=0x7f27800cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 -- 172.21.15.104:0/3715213351 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800ce850 con 0x7f27800b21c0 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277e7fc700 1 -- 172.21.15.104:0/3715213351 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550014633 0 0) 0x7f27800b2b40 con 0x7f278015d1a0 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277f7fe700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.194+0000 7f277ffff700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800b21c0 0x7f27800b2600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2778009e50 tx=0x7f277800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad6f4b4615bc8eb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f277e7fc700 1 -- 172.21.15.104:0/3715213351 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800e070 con 0x7f27800b21c0 2023-06-10T18:48:10.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f277e7fc700 1 -- 172.21.15.104:0/3715213351 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277800b910 con 0x7f27800b21c0 2023-06-10T18:48:10.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f277e7fc700 1 -- 172.21.15.104:0/3715213351 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800bcc0 con 0x7f27800b21c0 2023-06-10T18:48:10.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 -- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800b21c0 msgr2=0x7f27800b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800b21c0 0x7f27800b2600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2778009e50 tx=0x7f277800b330 comp rx=0 tx=0).stop 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 -- 172.21.15.104:0/3715213351 shutdown_connections 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800b21c0 0x7f27800b2600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 --2- 172.21.15.104:0/3715213351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2780156760 0x7f27800ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 -- 172.21.15.104:0/3715213351 >> 172.21.15.104:0/3715213351 conn(0x7f278006e370 msgr2=0x7f278006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 -- 172.21.15.104:0/3715213351 shutdown_connections 2023-06-10T18:48:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.195+0000 7f2786197700 1 -- 172.21.15.104:0/3715213351 wait complete. 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.196+0000 7f2786197700 1 Processor -- start 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 -- start start 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800b21c0 0x7f27800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 0x7f2780152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c8660 con 0x7f2780156760 2023-06-10T18:48:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c8940 con 0x7f278015d1a0 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f2786197700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c8c20 con 0x7f27800b21c0 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f277effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f277effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58766/0 (socket says 172.21.15.104:58766) 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.197+0000 7f277effd700 1 -- 172.21.15.104:0/3804405246 learned_addr learned my addr 172.21.15.104:0/3804405246 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277f7fe700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800b21c0 0x7f27800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277ffff700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 0x7f2780152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 -- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800b21c0 msgr2=0x7f27800b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800b21c0 0x7f27800b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 -- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 msgr2=0x7f2780152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 0x7f2780152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 -- 172.21.15.104:0/3804405246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680094a0 con 0x7f2780156760 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277ffff700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 0x7f2780152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f277effd700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f277000c550 tx=0x7f277000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770004c80 con 0x7f2780156760 2023-06-10T18:48:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2778009b30 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770004e00 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770007310 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.198+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27800c91e0 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.199+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800c8a80 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.199+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2770007470 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.200+0000 7f2777fff700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 0x7f276006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.200+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f27700892b0 con 0x7f2780156760 2023-06-10T18:48:10.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.200+0000 7f277f7fe700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 0x7f276006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.201+0000 7f277f7fe700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 0x7f276006d190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2768009d70 tx=0x7f2768009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.204+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f27800c8a80 con 0x7f2780156760 2023-06-10T18:48:10.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.387+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27800c8a80 con 0x7f2780156760 2023-06-10T18:48:10.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.387+0000 7f2777fff700 1 -- 172.21.15.104:0/3804405246 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27800c8a80 con 0x7f2780156760 2023-06-10T18:48:10.392 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:10.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 msgr2=0x7f276006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 0x7f276006d190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2768009d70 tx=0x7f2768009210 comp rx=0 tx=0).stop 2023-06-10T18:48:10.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 msgr2=0x7f27800c7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f277000c550 tx=0x7f277000c860 comp rx=0 tx=0).stop 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 shutdown_connections 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276006ace0 0x7f276006d190 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f278015d1a0 0x7f2780152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800b21c0 0x7f27800b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 --2- 172.21.15.104:0/3804405246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780156760 0x7f27800c7f60 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 >> 172.21.15.104:0/3804405246 conn(0x7f278006e370 msgr2=0x7f278006af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 shutdown_connections 2023-06-10T18:48:10.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.389+0000 7f2786197700 1 -- 172.21.15.104:0/3804405246 wait complete. 2023-06-10T18:48:10.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:10.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.524+0000 7f4129d0b700 1 Processor -- start 2023-06-10T18:48:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.524+0000 7f4129d0b700 1 -- start start 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4129d0b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f412415c6f0 0x7f41240b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4129d0b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f412406ea70 0x7f41240b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4129d0b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240ae7e0 con 0x7f41240aec00 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4129d0b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412406df90 con 0x7f412415c6f0 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4129d0b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412406e270 con 0x7f412406ea70 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f412415c6f0 0x7f41240b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f412415c6f0 0x7f41240b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58772/0 (socket says 172.21.15.104:58772) 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 -- 172.21.15.104:0/385254185 learned_addr learned my addr 172.21.15.104:0/385254185 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f4122ffd700 1 --2- 172.21.15.104:0/385254185 >> v2:172.21.15.114:3300/0 conn(0x7f412406ea70 0x7f41240b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 -- 172.21.15.104:0/385254185 >> v2:172.21.15.114:3300/0 conn(0x7f412406ea70 msgr2=0x7f41240b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 --2- 172.21.15.104:0/385254185 >> v2:172.21.15.114:3300/0 conn(0x7f412406ea70 0x7f41240b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 -- 172.21.15.104:0/385254185 >> v1:172.21.15.114:6789/0 conn(0x7f41240aec00 legacy=0x7f41240ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 -- 172.21.15.104:0/385254185 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412406e550 con 0x7f412415c6f0 2023-06-10T18:48:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41237fe700 1 --2- 172.21.15.104:0/385254185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b1070 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4124070990 tx=0x7f4118009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db75494a746bc0a2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41227fc700 1 -- 172.21.15.104:0/385254185 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118005e30 con 0x7f412415c6f0 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41227fc700 1 -- 172.21.15.104:0/385254185 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118006030 con 0x7f412415c6f0 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.525+0000 7f41227fc700 1 -- 172.21.15.104:0/385254185 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41180063e0 con 0x7f412415c6f0 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 -- 172.21.15.104:0/385254185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 msgr2=0x7f41240b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 --2- 172.21.15.104:0/385254185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b1070 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4124070990 tx=0x7f4118009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 -- 172.21.15.104:0/385254185 shutdown_connections 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 --2- 172.21.15.104:0/385254185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b1070 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 --2- 172.21.15.104:0/385254185 >> v2:172.21.15.114:3300/0 conn(0x7f412406ea70 0x7f41240b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 -- 172.21.15.104:0/385254185 >> 172.21.15.104:0/385254185 conn(0x7f412406b8a0 msgr2=0x7f41240cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 -- 172.21.15.104:0/385254185 shutdown_connections 2023-06-10T18:48:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.526+0000 7f4129d0b700 1 -- 172.21.15.104:0/385254185 wait complete. 2023-06-10T18:48:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.527+0000 7f4129d0b700 1 Processor -- start 2023-06-10T18:48:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.527+0000 7f4129d0b700 1 -- start start 2023-06-10T18:48:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 0x7f4124153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41240aec00 0x7f41240c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41241f3e40 con 0x7f412415c6f0 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41240c31b0 con 0x7f412406ea70 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4129d0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41241f31c0 con 0x7f41240aec00 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58782/0 (socket says 172.21.15.104:58782) 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 -- 172.21.15.104:0/386741567 learned_addr learned my addr 172.21.15.104:0/386741567 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f41237fe700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 0x7f4124153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4122ffd700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41240aec00 0x7f41240c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 -- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41240aec00 msgr2=0x7f41240c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41240aec00 0x7f41240c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 -- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 msgr2=0x7f4124153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 0x7f4124153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f4123fff700 1 -- 172.21.15.104:0/386741567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009500 con 0x7f412415c6f0 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.528+0000 7f41237fe700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 0x7f4124153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f4123fff700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f411c00cc10 tx=0x7f411c004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c00ab80 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41241f34a0 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c00ace0 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41241f3a60 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.529+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c012660 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.530+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41240ce380 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.530+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f411c0127c0 con 0x7f412415c6f0 2023-06-10T18:48:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.531+0000 7f410bfff700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 0x7f411006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.531+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f411c089df0 con 0x7f412415c6f0 2023-06-10T18:48:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.531+0000 7f41237fe700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 0x7f411006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.531+0000 7f41237fe700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 0x7f411006d190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4124070990 tx=0x7f4118008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.535+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f411c054cd0 con 0x7f412415c6f0 2023-06-10T18:48:10.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.721+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f41240cdb20 con 0x7f412415c6f0 2023-06-10T18:48:10.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.721+0000 7f410bfff700 1 -- 172.21.15.104:0/386741567 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f41240cdb20 con 0x7f412415c6f0 2023-06-10T18:48:10.726 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 msgr2=0x7f411006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 0x7f411006d190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4124070990 tx=0x7f4118008070 comp rx=0 tx=0).stop 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 msgr2=0x7f41240b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f411c00cc10 tx=0x7f411c004990 comp rx=0 tx=0).stop 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 shutdown_connections 2023-06-10T18:48:10.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f411006ace0 0x7f411006d190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412406ea70 0x7f4124153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412415c6f0 0x7f41240b01d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 --2- 172.21.15.104:0/386741567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41240aec00 0x7f41240c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 >> 172.21.15.104:0/386741567 conn(0x7f412406b8a0 msgr2=0x7f4124159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 shutdown_connections 2023-06-10T18:48:10.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.723+0000 7f4129d0b700 1 -- 172.21.15.104:0/386741567 wait complete. 2023-06-10T18:48:10.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:48:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.857+0000 7f574dc32700 1 Processor -- start 2023-06-10T18:48:10.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.857+0000 7f574dc32700 1 -- start start 2023-06-10T18:48:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f574dc32700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5748163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f574dc32700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574806d7d0 con 0x7f57480ce390 2023-06-10T18:48:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f574dc32700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480cd6c0 con 0x7f57481586a0 2023-06-10T18:48:10.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f574dc32700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c95d0 con 0x7f5748071120 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f5746ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f57480ce390 0x7f5748070a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56616/0 (socket says 172.21.15.104:56616) 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.858+0000 7f5746ffd700 1 -- 172.21.15.104:0/1863715162 learned_addr learned my addr 172.21.15.104:0/1863715162 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955755972 0 0) 0x7f574806d7d0 con 0x7f57480ce390 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f5747fff700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5748163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57300046c0 con 0x7f57480ce390 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1713711675 0 0) 0x7f57300046c0 con 0x7f57480ce390 2023-06-10T18:48:10.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 msgr2=0x7f5748163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5748163160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 >> v1:172.21.15.114:6789/0 conn(0x7f57481586a0 legacy=0x7f574806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480cd910 con 0x7f57480ce390 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5738001670 con 0x7f57480ce390 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 >> v1:172.21.15.104:6789/0 conn(0x7f57480ce390 legacy=0x7f5748070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f5730004f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5730005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5730005ea0 0x7f5730006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c95d0 con 0x7f5730004b40 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573000cc60 con 0x7f5730005ea0 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573000cdf0 con 0x7f5748071120 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f5730004f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f5746ffd700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5730005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f5747fff700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5730005ea0 0x7f5730006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 -- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 msgr2=0x7f5730005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5748071120 0x7f5730005960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 -- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5730005ea0 msgr2=0x7f5730006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5730005ea0 0x7f5730006310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.859+0000 7f57477fe700 1 -- 172.21.15.104:0/1863715162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300046c0 con 0x7f5730004b40 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f57477fe700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f5730004f50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f573c001800 tx=0x7f5730006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5a8874da015c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0012f0 con 0x7f5730004b40 2023-06-10T18:48:10.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c0052f0 con 0x7f5730004b40 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f57467fc700 1 -- 172.21.15.104:0/1863715162 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c006860 con 0x7f5730004b40 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 -- 172.21.15.104:0/1863715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 msgr2=0x7f5730004f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 --2- 172.21.15.104:0/1863715162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f5730004f50 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f573c001800 tx=0x7f5730006990 comp rx=0 tx=0).stop 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f5747fff700 1 -- 172.21.15.104:0/1863715162 reap_dead start 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 -- 172.21.15.104:0/1863715162 shutdown_connections 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 -- 172.21.15.104:0/1863715162 >> 172.21.15.104:0/1863715162 conn(0x7f57480ceec0 msgr2=0x7f57480cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:10.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 -- 172.21.15.104:0/1863715162 shutdown_connections 2023-06-10T18:48:10.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.860+0000 7f574dc32700 1 -- 172.21.15.104:0/1863715162 wait complete. 2023-06-10T18:48:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.861+0000 7f574dc32700 1 Processor -- start 2023-06-10T18:48:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 -- start start 2023-06-10T18:48:10.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57481f5ad0 0x7f574814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57480ae7b0 0x7f57480c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480acca0 con 0x7f5730004b40 2023-06-10T18:48:10.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481f6090 con 0x7f57481f5ad0 2023-06-10T18:48:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f574dc32700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480abbf0 con 0x7f57480ae7b0 2023-06-10T18:48:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f57477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f57477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58798/0 (socket says 172.21.15.104:58798) 2023-06-10T18:48:10.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f57477fe700 1 -- 172.21.15.104:0/25593850 learned_addr learned my addr 172.21.15.104:0/25593850 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.862+0000 7f5747fff700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57480ae7b0 0x7f57480c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f5746ffd700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57481f5ad0 0x7f574814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 -- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57480ae7b0 msgr2=0x7f57480c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57480ae7b0 0x7f57480c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 -- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57481f5ad0 msgr2=0x7f574814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:10.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57481f5ad0 0x7f574814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:10.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 -- 172.21.15.104:0/25593850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574814af50 con 0x7f5730004b40 2023-06-10T18:48:10.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f57477fe700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f57300058b0 tx=0x7f573c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c006c70 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c006dd0 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480b0390 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0038a0 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.863+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480ad540 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.864+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57480ad700 con 0x7f5730004b40 2023-06-10T18:48:10.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.864+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f573c003a00 con 0x7f5730004b40 2023-06-10T18:48:10.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.865+0000 7f572ffff700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 0x7f573406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:10.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.865+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f573c08a780 con 0x7f5730004b40 2023-06-10T18:48:10.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.865+0000 7f5746ffd700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 0x7f573406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:10.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.865+0000 7f5746ffd700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 0x7f573406d190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5738007dd0 tx=0x7f5738001020 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:10.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:10.869+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f573c055700 con 0x7f5730004b40 2023-06-10T18:48:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.051+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5748074050 con 0x7f5730004b40 2023-06-10T18:48:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.051+0000 7f572ffff700 1 -- 172.21.15.104:0/25593850 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f573c058d20 con 0x7f5730004b40 2023-06-10T18:48:11.057 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:48:11.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 msgr2=0x7f573406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 0x7f573406d190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5738007dd0 tx=0x7f5738001020 comp rx=0 tx=0).stop 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 msgr2=0x7f574815b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f57300058b0 tx=0x7f573c001120 comp rx=0 tx=0).stop 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 shutdown_connections 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57480ae7b0 0x7f57480c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5730004b40 0x7f574815b4c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f573406ace0 0x7f573406d190 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 --2- 172.21.15.104:0/25593850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57481f5ad0 0x7f574814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.053+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 >> 172.21.15.104:0/25593850 conn(0x7f57480ceec0 msgr2=0x7f5748157850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.054+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 shutdown_connections 2023-06-10T18:48:11.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.054+0000 7f574dc32700 1 -- 172.21.15.104:0/25593850 wait complete. 2023-06-10T18:48:11.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:48:11.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.191+0000 7f4dcfdb2700 1 Processor -- start 2023-06-10T18:48:11.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.191+0000 7f4dcfdb2700 1 -- start start 2023-06-10T18:48:11.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcfdb2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 0x7f4dc80cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcfdb2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 0x7f4dc80cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcfdb2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc80cab60 con 0x7f4dc80ae950 2023-06-10T18:48:11.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcfdb2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc80d0540 con 0x7f4dc806e0b0 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcfdb2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc80cc880 con 0x7f4dc806a880 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcdb4e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4dc80ae950 0x7f4dc806d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56634/0 (socket says 172.21.15.104:56634) 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.192+0000 7f4dcdb4e700 1 -- 172.21.15.104:0/2667334874 learned_addr learned my addr 172.21.15.104:0/2667334874 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dcd34d700 1 --2- 172.21.15.104:0/2667334874 >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 0x7f4dc80cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 --2- 172.21.15.104:0/2667334874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 0x7f4dc80cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570126164 0 0) 0x7f4dc80cab60 con 0x7f4dc80ae950 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db40035b0 con 0x7f4dc80ae950 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959927414 0 0) 0x7f4db40035b0 con 0x7f4dc80ae950 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 -- 172.21.15.104:0/2667334874 >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 msgr2=0x7f4dc80cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 --2- 172.21.15.104:0/2667334874 >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 0x7f4dc80cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 -- 172.21.15.104:0/2667334874 >> v1:172.21.15.104:6789/0 conn(0x7f4dc80ae950 legacy=0x7f4dc806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 -- 172.21.15.104:0/2667334874 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc806cea0 con 0x7f4dc806a880 2023-06-10T18:48:11.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dcd34d700 1 --2- 172.21.15.104:0/2667334874 >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 0x7f4dc80cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:11.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dce34f700 1 --2- 172.21.15.104:0/2667334874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 0x7f4dc80cbb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009840 tx=0x7f4dc4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a22d2766fc39cc51 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0001b70 con 0x7f4dc806a880 2023-06-10T18:48:11.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4004030 con 0x7f4dc806a880 2023-06-10T18:48:11.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.193+0000 7f4dccb4c700 1 -- 172.21.15.104:0/2667334874 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc40043a0 con 0x7f4dc806a880 2023-06-10T18:48:11.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/2667334874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 msgr2=0x7f4dc80cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/2667334874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 0x7f4dc80cbb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009840 tx=0x7f4dc4009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/2667334874 shutdown_connections 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/2667334874 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806a880 0x7f4dc80cbb10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/2667334874 >> v2:172.21.15.114:3300/0 conn(0x7f4dc806e0b0 0x7f4dc80cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/2667334874 >> 172.21.15.104:0/2667334874 conn(0x7f4dc806b8a0 msgr2=0x7f4dc80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/2667334874 shutdown_connections 2023-06-10T18:48:11.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.194+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/2667334874 wait complete. 2023-06-10T18:48:11.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.195+0000 7f4dcfdb2700 1 Processor -- start 2023-06-10T18:48:11.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.195+0000 7f4dcfdb2700 1 -- start start 2023-06-10T18:48:11.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 0x7f4dc81ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806e0b0 0x7f4dc80c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc80af830 con 0x7f4dc80ae950 2023-06-10T18:48:11.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc815b6b0 con 0x7f4dc806a880 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcfdb2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc815b990 con 0x7f4dc806e0b0 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58816/0 (socket says 172.21.15.104:58816) 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 -- 172.21.15.104:0/3444568917 learned_addr learned my addr 172.21.15.104:0/3444568917 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcdb4e700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 0x7f4dc81ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcd34d700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806e0b0 0x7f4dc80c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 -- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806e0b0 msgr2=0x7f4dc80c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806e0b0 0x7f4dc80c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 -- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 msgr2=0x7f4dc81ef6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 0x7f4dc81ef6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 -- 172.21.15.104:0/3444568917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80094a0 con 0x7f4dc80ae950 2023-06-10T18:48:11.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dcdb4e700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 0x7f4dc81ef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.196+0000 7f4dce34f700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80cc9c0 tx=0x7f4dc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.197+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4003e70 con 0x7f4dc80ae950 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.197+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc401f070 con 0x7f4dc80ae950 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.197+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc40094a0 con 0x7f4dc80ae950 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.197+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4011870 con 0x7f4dc80ae950 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.197+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc815bf50 con 0x7f4dc80ae950 2023-06-10T18:48:11.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.198+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc8066f10 con 0x7f4dc80ae950 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.198+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4dc4021030 con 0x7f4dc80ae950 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.198+0000 7f4dbe7fc700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 0x7f4dac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.198+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4dc408a8a0 con 0x7f4dc80ae950 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.199+0000 7f4dcdb4e700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 0x7f4dac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.199+0000 7f4dcdb4e700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 0x7f4dac06d190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0001270 tx=0x7f4dc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.203+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4dc40558a0 con 0x7f4dc80ae950 2023-06-10T18:48:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.387+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4dc815c0e0 con 0x7f4dc80ae950 2023-06-10T18:48:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.387+0000 7f4dbe7fc700 1 -- 172.21.15.104:0/3444568917 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4dc4058ec0 con 0x7f4dc80ae950 2023-06-10T18:48:11.393 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:48:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 msgr2=0x7f4dac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 0x7f4dac06d190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0001270 tx=0x7f4dc0009040 comp rx=0 tx=0).stop 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 msgr2=0x7f4dc80c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80cc9c0 tx=0x7f4dc4004750 comp rx=0 tx=0).stop 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 shutdown_connections 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dac06ace0 0x7f4dac06d190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dc806a880 0x7f4dc81ef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc80ae950 0x7f4dc80c8a30 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 --2- 172.21.15.104:0/3444568917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4dc806e0b0 0x7f4dc80c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.389+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 >> 172.21.15.104:0/3444568917 conn(0x7f4dc806b8a0 msgr2=0x7f4dc80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.390+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 shutdown_connections 2023-06-10T18:48:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.390+0000 7f4dcfdb2700 1 -- 172.21.15.104:0/3444568917 wait complete. 2023-06-10T18:48:11.406 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:48:11.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.522+0000 7fca1407e700 1 Processor -- start 2023-06-10T18:48:11.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.523+0000 7fca1407e700 1 -- start start 2023-06-10T18:48:11.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.523+0000 7fca1407e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.523+0000 7fca1407e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c0b4ff0 0x7fca0c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.523+0000 7fca1407e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b6070 con 0x7fca0c0ce890 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca1407e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0ced10 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca1407e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0b5460 con 0x7fca0c15a420 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c0b4ff0 0x7fca0c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c0b4ff0 0x7fca0c0ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58820/0 (socket says 172.21.15.104:58820) 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 -- 172.21.15.104:0/2777221866 learned_addr learned my addr 172.21.15.104:0/2777221866 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11e1a700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 -- 172.21.15.104:0/2777221866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 msgr2=0x7fca0c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 -- 172.21.15.104:0/2777221866 >> v1:172.21.15.114:6789/0 conn(0x7fca0c0ce890 legacy=0x7fca0c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 -- 172.21.15.104:0/2777221866 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0b5740 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca11619700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0ce350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fca080098f0 tx=0x7fca08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937b1f42a2997a3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca10e18700 1 -- 172.21.15.104:0/2777221866 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08011830 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca10e18700 1 -- 172.21.15.104:0/2777221866 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca08011990 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.524+0000 7fca10e18700 1 -- 172.21.15.104:0/2777221866 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08011d20 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 -- 172.21.15.104:0/2777221866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 msgr2=0x7fca0c0ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0ce350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fca080098f0 tx=0x7fca08009c00 comp rx=0 tx=0).stop 2023-06-10T18:48:11.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 -- 172.21.15.104:0/2777221866 shutdown_connections 2023-06-10T18:48:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 --2- 172.21.15.104:0/2777221866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0ce350 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 -- 172.21.15.104:0/2777221866 >> 172.21.15.104:0/2777221866 conn(0x7fca0c089fd0 msgr2=0x7fca0c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 -- 172.21.15.104:0/2777221866 shutdown_connections 2023-06-10T18:48:11.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.525+0000 7fca1407e700 1 -- 172.21.15.104:0/2777221866 wait complete. 2023-06-10T18:48:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.526+0000 7fca1407e700 1 Processor -- start 2023-06-10T18:48:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.526+0000 7fca1407e700 1 -- start start 2023-06-10T18:48:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce890 0x7fca0c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c155f60 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c155350 con 0x7fca0c0ce890 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c155630 con 0x7fca0c15a420 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58826/0 (socket says 172.21.15.104:58826) 2023-06-10T18:48:11.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 -- 172.21.15.104:0/1172915463 learned_addr learned my addr 172.21.15.104:0/1172915463 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1261b700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11619700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce890 0x7fca0c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 -- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 msgr2=0x7fca0c0c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 -- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce890 msgr2=0x7fca0c1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce890 0x7fca0c1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 -- 172.21.15.104:0/1172915463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08009550 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca11e1a700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e9b0 tx=0x7fc9fc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc009710 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc00c850 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.527+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c155910 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.528+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00cbc0 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.528+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c154bc0 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.529+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c0ce190 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.529+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc9fc00cd20 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.529+0000 7fca027fc700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 0x7fc9f006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.529+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc9fc089650 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.529+0000 7fca11619700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 0x7fc9f006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.530+0000 7fca11619700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 0x7fc9f006d1c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fca08014040 tx=0x7fca08008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.534+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fca0c0ce190 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.736+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca0c0ce190 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.736+0000 7fca027fc700 1 -- 172.21.15.104:0/1172915463 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7fca0c0ce190 con 0x7fca0c0b4ff0 2023-06-10T18:48:11.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 msgr2=0x7fc9f006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 0x7fc9f006d1c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fca08014040 tx=0x7fca08008030 comp rx=0 tx=0).stop 2023-06-10T18:48:11.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 msgr2=0x7fca0c0cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e9b0 tx=0x7fc9fc00ed70 comp rx=0 tx=0).stop 2023-06-10T18:48:11.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 shutdown_connections 2023-06-10T18:48:11.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c15a420 0x7fca0c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c0b4ff0 0x7fca0c0cdc50 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f006ad10 0x7fc9f006d1c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 --2- 172.21.15.104:0/1172915463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce890 0x7fca0c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 >> 172.21.15.104:0/1172915463 conn(0x7fca0c089fd0 msgr2=0x7fca0c0b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 shutdown_connections 2023-06-10T18:48:11.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.738+0000 7fca1407e700 1 -- 172.21.15.104:0/1172915463 wait complete. 2023-06-10T18:48:11.746 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:48:11.755 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:11.755 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:11.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:11.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 Processor -- start 2023-06-10T18:48:11.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 -- start start 2023-06-10T18:48:11.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 0x7f08100cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08100ce040 0x7f081006e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081006dcb0 con 0x7f0810069e70 2023-06-10T18:48:11.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.928+0000 7f0815e4d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08100ce470 con 0x7f08100ce040 2023-06-10T18:48:11.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f0815e4d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081006d0a0 con 0x7f081015c500 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08100ce040 0x7f081006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0810069e70 0x7f08100c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56666/0 (socket says 172.21.15.104:56666) 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080ffff700 1 -- 172.21.15.104:0/1877440837 learned_addr learned my addr 172.21.15.104:0/1877440837 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080f7fe700 1 --2- 172.21.15.104:0/1877440837 >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 0x7f08100cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 -- 172.21.15.104:0/1877440837 >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 msgr2=0x7f08100cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 --2- 172.21.15.104:0/1877440837 >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 0x7f08100cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 -- 172.21.15.104:0/1877440837 >> v1:172.21.15.104:6789/0 conn(0x7f0810069e70 legacy=0x7f08100c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080e7fc700 1 -- 172.21.15.104:0/1877440837 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149979117 0 0) 0x7f081006dcb0 con 0x7f0810069e70 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 -- 172.21.15.104:0/1877440837 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081006d380 con 0x7f08100ce040 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080f7fe700 1 --2- 172.21.15.104:0/1877440837 >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 0x7f08100cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080effd700 1 --2- 172.21.15.104:0/1877440837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08100ce040 0x7f081006e140 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0810067130 tx=0x7f0804009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b4304b42cf11f1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080e7fc700 1 -- 172.21.15.104:0/1877440837 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804011830 con 0x7f08100ce040 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080e7fc700 1 -- 172.21.15.104:0/1877440837 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0804011990 con 0x7f08100ce040 2023-06-10T18:48:11.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.929+0000 7f080e7fc700 1 -- 172.21.15.104:0/1877440837 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804011d20 con 0x7f08100ce040 2023-06-10T18:48:11.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 -- 172.21.15.104:0/1877440837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08100ce040 msgr2=0x7f081006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 --2- 172.21.15.104:0/1877440837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08100ce040 0x7f081006e140 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0810067130 tx=0x7f0804009b20 comp rx=0 tx=0).stop 2023-06-10T18:48:11.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 -- 172.21.15.104:0/1877440837 shutdown_connections 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 --2- 172.21.15.104:0/1877440837 >> v2:172.21.15.114:3300/0 conn(0x7f081015c500 0x7f08100cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 --2- 172.21.15.104:0/1877440837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08100ce040 0x7f081006e140 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 -- 172.21.15.104:0/1877440837 >> 172.21.15.104:0/1877440837 conn(0x7f081006b8a0 msgr2=0x7f08100cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 -- 172.21.15.104:0/1877440837 shutdown_connections 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.930+0000 7f0815e4d700 1 -- 172.21.15.104:0/1877440837 wait complete. 2023-06-10T18:48:11.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.931+0000 7f0815e4d700 1 Processor -- start 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.931+0000 7f0815e4d700 1 -- start start 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0810069e70 0x7f08101f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08100ce040 0x7f08100c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081009b5c0 con 0x7f081015c500 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081009bb40 con 0x7f08100ce040 2023-06-10T18:48:11.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f0815e4d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08100c90a0 con 0x7f0810069e70 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58844/0 (socket says 172.21.15.104:58844) 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 -- 172.21.15.104:0/69336279 learned_addr learned my addr 172.21.15.104:0/69336279 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080f7fe700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0810069e70 0x7f08101f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080effd700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08100ce040 0x7f08100c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 -- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0810069e70 msgr2=0x7f08101f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0810069e70 0x7f08101f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 -- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08100ce040 msgr2=0x7f08100c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08100ce040 0x7f08100c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 -- 172.21.15.104:0/69336279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040094a0 con 0x7f081015c500 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.932+0000 7f080ffff700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f080800cb90 tx=0x7f080800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.933+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800ab30 con 0x7f081015c500 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.933+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080800ac90 con 0x7f081015c500 2023-06-10T18:48:11.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.933+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08100c8ae0 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.933+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080070d0 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.933+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081009b7d0 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.934+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081009bcf0 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.934+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0808007230 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.934+0000 7f07f7fff700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 0x7f07fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.935+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f08080893c0 con 0x7f081015c500 2023-06-10T18:48:11.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.935+0000 7f080f7fe700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 0x7f07fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:11.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.935+0000 7f080f7fe700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 0x7f07fc06d190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f080000a970 tx=0x7f0800008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:11.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:11.939+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f081009bcf0 con 0x7f081015c500 2023-06-10T18:48:12.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.124+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f081009bcf0 con 0x7f081015c500 2023-06-10T18:48:12.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.125+0000 7f07f7fff700 1 -- 172.21.15.104:0/69336279 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f081009bcf0 con 0x7f081015c500 2023-06-10T18:48:12.130 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:12.130 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 msgr2=0x7f07fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 0x7f07fc06d190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f080000a970 tx=0x7f0800008040 comp rx=0 tx=0).stop 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 msgr2=0x7f0810099560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f080800cb90 tx=0x7f080800cf50 comp rx=0 tx=0).stop 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 shutdown_connections 2023-06-10T18:48:12.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f081015c500 0x7f0810099560 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0810069e70 0x7f08101f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f07fc06ace0 0x7f07fc06d190 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 --2- 172.21.15.104:0/69336279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08100ce040 0x7f08100c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 >> 172.21.15.104:0/69336279 conn(0x7f081006b8a0 msgr2=0x7f08100ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 shutdown_connections 2023-06-10T18:48:12.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.127+0000 7f0815e4d700 1 -- 172.21.15.104:0/69336279 wait complete. 2023-06-10T18:48:12.147 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:47:57.299455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:47:58.209852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:12.149 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.267+0000 7f9d6f7da700 1 Processor -- start 2023-06-10T18:48:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.268+0000 7f9d6f7da700 1 -- start start 2023-06-10T18:48:12.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.268+0000 7f9d6f7da700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d6806e6e0 0x7f9d680b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.268+0000 7f9d6f7da700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6806e250 con 0x7f9d6806cde0 2023-06-10T18:48:12.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.268+0000 7f9d6f7da700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6815c950 con 0x7f9d6815c580 2023-06-10T18:48:12.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.268+0000 7f9d6f7da700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d680b1e80 con 0x7f9d6806e6e0 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6cd75700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d6806e6e0 0x7f9d680b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6dd77700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9d6806cde0 0x7f9d680ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56680/0 (socket says 172.21.15.104:56680) 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6dd77700 1 -- 172.21.15.104:0/1526235295 learned_addr learned my addr 172.21.15.104:0/1526235295 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6cd75700 1 -- 172.21.15.104:0/1526235295 >> v1:172.21.15.114:6789/0 conn(0x7f9d6815c580 legacy=0x7f9d680b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6cd75700 1 -- 172.21.15.104:0/1526235295 >> v1:172.21.15.104:6789/0 conn(0x7f9d6806cde0 legacy=0x7f9d680ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6cd75700 1 -- 172.21.15.104:0/1526235295 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6806d920 con 0x7f9d6806e6e0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d5ffff700 1 -- 172.21.15.104:0/1526235295 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470456983 0 0) 0x7f9d6806e250 con 0x7f9d6806cde0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d6cd75700 1 --2- 172.21.15.104:0/1526235295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b1780 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009840 tx=0x7f9d64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95472b4f11be5680 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d5ffff700 1 -- 172.21.15.104:0/1526235295 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64011830 con 0x7f9d6806e6e0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d5ffff700 1 -- 172.21.15.104:0/1526235295 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64011990 con 0x7f9d6806e6e0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.269+0000 7f9d5ffff700 1 -- 172.21.15.104:0/1526235295 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64011d20 con 0x7f9d6806e6e0 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 -- 172.21.15.104:0/1526235295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 msgr2=0x7f9d680b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/1526235295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b1780 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009840 tx=0x7f9d64009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:12.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 -- 172.21.15.104:0/1526235295 shutdown_connections 2023-06-10T18:48:12.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/1526235295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b1780 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 -- 172.21.15.104:0/1526235295 >> 172.21.15.104:0/1526235295 conn(0x7f9d6806b8a0 msgr2=0x7f9d680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 -- 172.21.15.104:0/1526235295 shutdown_connections 2023-06-10T18:48:12.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.270+0000 7f9d6f7da700 1 -- 172.21.15.104:0/1526235295 wait complete. 2023-06-10T18:48:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.271+0000 7f9d6f7da700 1 Processor -- start 2023-06-10T18:48:12.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.271+0000 7f9d6f7da700 1 -- start start 2023-06-10T18:48:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d6806cde0 0x7f9d680ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6815c580 0x7f9d681e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6814a3a0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680b27d0 con 0x7f9d6815c580 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d681e3be0 con 0x7f9d6806cde0 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58856/0 (socket says 172.21.15.104:58856) 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 -- 172.21.15.104:0/2674545028 learned_addr learned my addr 172.21.15.104:0/2674545028 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6dd77700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6815c580 0x7f9d681e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6d576700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d6806cde0 0x7f9d680ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 -- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d6806cde0 msgr2=0x7f9d680ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d6806cde0 0x7f9d680ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 -- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6815c580 msgr2=0x7f9d681e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6815c580 0x7f9d681e5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 -- 172.21.15.104:0/2674545028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640094a0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6cd75700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9d640090b0 tx=0x7f9d64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6401a860 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6401a9e0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6401ad90 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.272+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d681e3ec0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.273+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d68158b10 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.273+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d68158ca0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.274+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9d64022070 con 0x7f9d6806e6e0 2023-06-10T18:48:12.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.274+0000 7f9d5dffb700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 0x7f9d4c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.274+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9d6408a460 con 0x7f9d6806e6e0 2023-06-10T18:48:12.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.275+0000 7f9d6d576700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 0x7f9d4c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.275+0000 7f9d6d576700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 0x7f9d4c06d190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d58000fb0 tx=0x7f9d58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.279+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9d64055580 con 0x7f9d6806e6e0 2023-06-10T18:48:12.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.462+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d68066f10 con 0x7f9d6806e6e0 2023-06-10T18:48:12.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.463+0000 7f9d5dffb700 1 -- 172.21.15.104:0/2674545028 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d64058ba0 con 0x7f9d6806e6e0 2023-06-10T18:48:12.468 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 msgr2=0x7f9d4c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 0x7f9d4c06d190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d58000fb0 tx=0x7f9d58009040 comp rx=0 tx=0).stop 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 msgr2=0x7f9d680b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9d640090b0 tx=0x7f9d64004750 comp rx=0 tx=0).stop 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 shutdown_connections 2023-06-10T18:48:12.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d4c06ace0 0x7f9d4c06d190 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6815c580 0x7f9d681e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d6806cde0 0x7f9d680ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 --2- 172.21.15.104:0/2674545028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d6806e6e0 0x7f9d680b4e50 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 >> 172.21.15.104:0/2674545028 conn(0x7f9d6806b8a0 msgr2=0x7f9d681e32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 shutdown_connections 2023-06-10T18:48:12.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.465+0000 7f9d6f7da700 1 -- 172.21.15.104:0/2674545028 wait complete. 2023-06-10T18:48:12.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:12.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.599+0000 7feadf0d3700 1 Processor -- start 2023-06-10T18:48:12.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.599+0000 7feadf0d3700 1 -- start start 2023-06-10T18:48:12.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadf0d3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead815c580 0x7fead80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadf0d3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fead806e6e0 0x7fead80b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadf0d3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead806e250 con 0x7fead806cde0 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadf0d3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead815c950 con 0x7fead806e6e0 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadf0d3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead80cf0e0 con 0x7fead815c580 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7fead7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fead806e6e0 0x7fead80b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadd670700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fead806cde0 0x7fead8069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56694/0 (socket says 172.21.15.104:56694) 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.600+0000 7feadd670700 1 -- 172.21.15.104:0/2160211933 learned_addr learned my addr 172.21.15.104:0/2160211933 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:12.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7feadce6f700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead815c580 0x7fead80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead77fe700 1 -- 172.21.15.104:0/2160211933 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666353054 0 0) 0x7fead806e250 con 0x7fead806cde0 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead77fe700 1 -- 172.21.15.104:0/2160211933 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac80035b0 con 0x7fead806cde0 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead7fff700 1 -- 172.21.15.104:0/2160211933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead815c580 msgr2=0x7fead80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead7fff700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead815c580 0x7fead80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead7fff700 1 -- 172.21.15.104:0/2160211933 >> v1:172.21.15.104:6789/0 conn(0x7fead806cde0 legacy=0x7fead8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead7fff700 1 -- 172.21.15.104:0/2160211933 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead80b1e80 con 0x7fead806e6e0 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead7fff700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead806e6e0 0x7fead80b15b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feac4009840 tx=0x7feac4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23b4ebb5e7178b6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead77fe700 1 -- 172.21.15.104:0/2160211933 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4011830 con 0x7fead806e6e0 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead77fe700 1 -- 172.21.15.104:0/2160211933 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac4011990 con 0x7fead806e6e0 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.601+0000 7fead77fe700 1 -- 172.21.15.104:0/2160211933 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4011d00 con 0x7fead806e6e0 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 -- 172.21.15.104:0/2160211933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead806e6e0 msgr2=0x7fead80b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead806e6e0 0x7fead80b15b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7feac4009840 tx=0x7feac4009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:12.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 -- 172.21.15.104:0/2160211933 shutdown_connections 2023-06-10T18:48:12.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead815c580 0x7fead80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 --2- 172.21.15.104:0/2160211933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead806e6e0 0x7fead80b15b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 -- 172.21.15.104:0/2160211933 >> 172.21.15.104:0/2160211933 conn(0x7fead806b8a0 msgr2=0x7fead80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 -- 172.21.15.104:0/2160211933 shutdown_connections 2023-06-10T18:48:12.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.602+0000 7feadf0d3700 1 -- 172.21.15.104:0/2160211933 wait complete. 2023-06-10T18:48:12.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.603+0000 7feadf0d3700 1 Processor -- start 2023-06-10T18:48:12.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.603+0000 7feadf0d3700 1 -- start start 2023-06-10T18:48:12.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead806cde0 0x7fead8157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fead806e6e0 0x7fead80afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead80b2550 con 0x7fead815c580 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead8157940 con 0x7fead806e6e0 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead80b27f0 con 0x7fead806cde0 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58872/0 (socket says 172.21.15.104:58872) 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 -- 172.21.15.104:0/1340607422 learned_addr learned my addr 172.21.15.104:0/1340607422 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadce6f700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead806cde0 0x7fead8157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7fead7fff700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fead806e6e0 0x7fead80afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 -- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead806cde0 msgr2=0x7fead8157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead806cde0 0x7fead8157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 -- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fead806e6e0 msgr2=0x7fead80afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fead806e6e0 0x7fead80afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 -- 172.21.15.104:0/1340607422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac40094a0 con 0x7fead815c580 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadd670700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fead000ca20 tx=0x7fead000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead000aaa0 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000ac00 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0012400 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.604+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead80b2ad0 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.605+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead80b3090 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.605+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead80b3280 con 0x7fead815c580 2023-06-10T18:48:12.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.606+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fead001b3b0 con 0x7fead815c580 2023-06-10T18:48:12.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.606+0000 7fead57fa700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 0x7feabc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.606+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fead0089250 con 0x7fead815c580 2023-06-10T18:48:12.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.606+0000 7feadce6f700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 0x7feabc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.607+0000 7feadce6f700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 0x7feabc06d190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feacc009b50 tx=0x7feacc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.611+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fead0057ce0 con 0x7fead815c580 2023-06-10T18:48:12.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.795+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fead8074050 con 0x7fead815c580 2023-06-10T18:48:12.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.796+0000 7fead57fa700 1 -- 172.21.15.104:0/1340607422 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7fead0001ae0 con 0x7fead815c580 2023-06-10T18:48:12.801 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:12.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 msgr2=0x7feabc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 0x7feabc06d190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feacc009b50 tx=0x7feacc008040 comp rx=0 tx=0).stop 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 msgr2=0x7fead81e5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fead000ca20 tx=0x7fead000cde0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 shutdown_connections 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fead815c580 0x7fead81e5e50 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fead806cde0 0x7fead8157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feabc06ace0 0x7feabc06d190 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 --2- 172.21.15.104:0/1340607422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fead806e6e0 0x7fead80afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 >> 172.21.15.104:0/1340607422 conn(0x7fead806b8a0 msgr2=0x7fead80c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 shutdown_connections 2023-06-10T18:48:12.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.798+0000 7feadf0d3700 1 -- 172.21.15.104:0/1340607422 wait complete. 2023-06-10T18:48:12.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:48:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.932+0000 7f0587b9b700 1 Processor -- start 2023-06-10T18:48:12.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.932+0000 7f0587b9b700 1 -- start start 2023-06-10T18:48:12.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0587b9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f058015c730 0x7f05800cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0587b9b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05800cd720 0x7f058006cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0587b9b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058015cb00 con 0x7f058006d2b0 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0587b9b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800cdbc0 con 0x7f058015c730 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0587b9b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800acd90 con 0x7f05800cd720 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f058015c730 0x7f05800cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f058015c730 0x7f05800cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58876/0 (socket says 172.21.15.104:58876) 2023-06-10T18:48:12.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 -- 172.21.15.104:0/2983982340 learned_addr learned my addr 172.21.15.104:0/2983982340 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585136700 1 --2- 172.21.15.104:0/2983982340 >> v2:172.21.15.114:3300/0 conn(0x7f05800cd720 0x7f058006cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 -- 172.21.15.104:0/2983982340 >> v2:172.21.15.114:3300/0 conn(0x7f05800cd720 msgr2=0x7f058006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 --2- 172.21.15.104:0/2983982340 >> v2:172.21.15.114:3300/0 conn(0x7f05800cd720 0x7f058006cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 -- 172.21.15.104:0/2983982340 >> v1:172.21.15.114:6789/0 conn(0x7f058006d2b0 legacy=0x7f058006d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:12.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.933+0000 7f0585937700 1 -- 172.21.15.104:0/2983982340 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800ac460 con 0x7f058015c730 2023-06-10T18:48:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0585937700 1 --2- 172.21.15.104:0/2983982340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800cd1e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0570006fd0 tx=0x7f0570009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c217c00643cf26 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0584935700 1 -- 172.21.15.104:0/2983982340 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0570005e30 con 0x7f058015c730 2023-06-10T18:48:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0584935700 1 -- 172.21.15.104:0/2983982340 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0570004030 con 0x7f058015c730 2023-06-10T18:48:12.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0584935700 1 -- 172.21.15.104:0/2983982340 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700043e0 con 0x7f058015c730 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 -- 172.21.15.104:0/2983982340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 msgr2=0x7f05800cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 --2- 172.21.15.104:0/2983982340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800cd1e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0570006fd0 tx=0x7f0570009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 -- 172.21.15.104:0/2983982340 shutdown_connections 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 --2- 172.21.15.104:0/2983982340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800cd1e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 --2- 172.21.15.104:0/2983982340 >> v2:172.21.15.114:3300/0 conn(0x7f05800cd720 0x7f058006cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 -- 172.21.15.104:0/2983982340 >> 172.21.15.104:0/2983982340 conn(0x7f058006b8a0 msgr2=0x7f05800cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 -- 172.21.15.104:0/2983982340 shutdown_connections 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.934+0000 7f0587b9b700 1 -- 172.21.15.104:0/2983982340 wait complete. 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 Processor -- start 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 -- start start 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058006d2b0 0x7f0580158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05800cd720 0x7f05800c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c7fe0 con 0x7f058015c730 2023-06-10T18:48:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801592a0 con 0x7f058006d2b0 2023-06-10T18:48:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0587b9b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c70f0 con 0x7f05800cd720 2023-06-10T18:48:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0586138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0586138700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58886/0 (socket says 172.21.15.104:58886) 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0586138700 1 -- 172.21.15.104:0/1898238033 learned_addr learned my addr 172.21.15.104:0/1898238033 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0585136700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05800cd720 0x7f05800c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.936+0000 7f0585937700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058006d2b0 0x7f0580158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 -- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05800cd720 msgr2=0x7f05800c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05800cd720 0x7f05800c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 -- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058006d2b0 msgr2=0x7f0580158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058006d2b0 0x7f0580158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:12.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 -- 172.21.15.104:0/1898238033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009500 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0586138700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f057800cc50 tx=0x7f057800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578003c00 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578003d60 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800c73d0 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057800aab0 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.937+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800c7990 con 0x7f058015c730 2023-06-10T18:48:12.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.938+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05800c8190 con 0x7f058015c730 2023-06-10T18:48:12.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.938+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f057800ac10 con 0x7f058015c730 2023-06-10T18:48:12.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.939+0000 7f05767fc700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 0x7f056406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:12.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.939+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0578089360 con 0x7f058015c730 2023-06-10T18:48:12.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.939+0000 7f0585937700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 0x7f056406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:12.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.939+0000 7f0585937700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 0x7f056406d190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0570009c10 tx=0x7f0570008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:12.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:12.943+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05800c8190 con 0x7f058015c730 2023-06-10T18:48:13.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.128+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f058015cb00 con 0x7f058015c730 2023-06-10T18:48:13.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.128+0000 7f05767fc700 1 -- 172.21.15.104:0/1898238033 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f058015cb00 con 0x7f058015c730 2023-06-10T18:48:13.134 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:48:13.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.130+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 msgr2=0x7f056406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.130+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 0x7f056406d190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0570009c10 tx=0x7f0570008070 comp rx=0 tx=0).stop 2023-06-10T18:48:13.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 msgr2=0x7f05800b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f057800cc50 tx=0x7f057800cf60 comp rx=0 tx=0).stop 2023-06-10T18:48:13.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 shutdown_connections 2023-06-10T18:48:13.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f056406ace0 0x7f056406d190 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058006d2b0 0x7f0580158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f058015c730 0x7f05800b4560 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 --2- 172.21.15.104:0/1898238033 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05800cd720 0x7f05800c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 >> 172.21.15.104:0/1898238033 conn(0x7f058006b8a0 msgr2=0x7f05800b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 shutdown_connections 2023-06-10T18:48:13.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.131+0000 7f0587b9b700 1 -- 172.21.15.104:0/1898238033 wait complete. 2023-06-10T18:48:13.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:48:13.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.265+0000 7f432e4c3700 1 Processor -- start 2023-06-10T18:48:13.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.265+0000 7f432e4c3700 1 -- start start 2023-06-10T18:48:13.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432e4c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d04f0 0x7f43280d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432e4c3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43280b05d0 0x7f43280b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432e4c3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280af530 con 0x7f43280d25c0 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432e4c3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280af810 con 0x7f43280b05d0 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432e4c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280afaf0 con 0x7f43280d04f0 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432ca60700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43280b05d0 0x7f43280b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432ca60700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43280b05d0 0x7f43280b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58894/0 (socket says 172.21.15.104:58894) 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.266+0000 7f432ca60700 1 -- 172.21.15.104:0/2929657104 learned_addr learned my addr 172.21.15.104:0/2929657104 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f4327fff700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d04f0 0x7f43280d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432ca60700 1 -- 172.21.15.104:0/2929657104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d04f0 msgr2=0x7f43280d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432ca60700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d04f0 0x7f43280d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432ca60700 1 -- 172.21.15.104:0/2929657104 >> v1:172.21.15.114:6789/0 conn(0x7f43280d25c0 legacy=0x7f43280d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432ca60700 1 -- 172.21.15.104:0/2929657104 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280afdd0 con 0x7f43280b05d0 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432ca60700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f43280b0a50 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4320009e50 tx=0x7f432000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c119cd4585833d27 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f4326ffd700 1 -- 172.21.15.104:0/2929657104 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000e070 con 0x7f43280b05d0 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f4326ffd700 1 -- 172.21.15.104:0/2929657104 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4320003a20 con 0x7f43280b05d0 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f4326ffd700 1 -- 172.21.15.104:0/2929657104 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320003dd0 con 0x7f43280b05d0 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 -- 172.21.15.104:0/2929657104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 msgr2=0x7f43280b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f43280b0a50 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4320009e50 tx=0x7f432000b330 comp rx=0 tx=0).stop 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 -- 172.21.15.104:0/2929657104 shutdown_connections 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f43280b0a50 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 --2- 172.21.15.104:0/2929657104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d04f0 0x7f43280d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 -- 172.21.15.104:0/2929657104 >> 172.21.15.104:0/2929657104 conn(0x7f4328071ec0 msgr2=0x7f432809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:13.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 -- 172.21.15.104:0/2929657104 shutdown_connections 2023-06-10T18:48:13.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.267+0000 7f432e4c3700 1 -- 172.21.15.104:0/2929657104 wait complete. 2023-06-10T18:48:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.269+0000 7f432e4c3700 1 Processor -- start 2023-06-10T18:48:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.269+0000 7f432e4c3700 1 -- start start 2023-06-10T18:48:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 0x7f43280b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d25c0 0x7f43280b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432814fb50 con 0x7f43280b05d0 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280b5220 con 0x7f43280d04f0 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432814e850 con 0x7f43280d25c0 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58896/0 (socket says 172.21.15.104:58896) 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 -- 172.21.15.104:0/3553673419 learned_addr learned my addr 172.21.15.104:0/3553673419 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:13.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432ca60700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d25c0 0x7f43280b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 -- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d25c0 msgr2=0x7f43280b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d25c0 0x7f43280b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f43277fe700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 0x7f43280b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 -- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 msgr2=0x7f43280b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 0x7f43280b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 -- 172.21.15.104:0/3553673419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320009b30 con 0x7f43280b05d0 2023-06-10T18:48:13.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f43277fe700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 0x7f43280b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4327fff700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f431800c7f0 tx=0x7f431800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318011800 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318011960 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432814eb30 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318011d10 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.270+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432814f0f0 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.271+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4328067010 con 0x7f43280b05d0 2023-06-10T18:48:13.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.272+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4318018360 con 0x7f43280b05d0 2023-06-10T18:48:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.272+0000 7f4324ff9700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 0x7f430806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.272+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f43180895b0 con 0x7f43280b05d0 2023-06-10T18:48:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.272+0000 7f43277fe700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 0x7f430806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.273+0000 7f43277fe700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 0x7f430806d190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4310006fd0 tx=0x7f4310008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.276+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4318057fc0 con 0x7f43280b05d0 2023-06-10T18:48:13.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.464+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4328074050 con 0x7f43280b05d0 2023-06-10T18:48:13.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.465+0000 7f4324ff9700 1 -- 172.21.15.104:0/3553673419 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4318057b50 con 0x7f43280b05d0 2023-06-10T18:48:13.470 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:48:13.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 msgr2=0x7f430806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 0x7f430806d190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4310006fd0 tx=0x7f4310008040 comp rx=0 tx=0).stop 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 msgr2=0x7f432814f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f431800c7f0 tx=0x7f431800cbb0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 shutdown_connections 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43280d25c0 0x7f43280b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43280b05d0 0x7f432814f610 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f430806ace0 0x7f430806d190 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 --2- 172.21.15.104:0/3553673419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43280d04f0 0x7f43280b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 >> 172.21.15.104:0/3553673419 conn(0x7f4328071ec0 msgr2=0x7f432815b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:13.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 shutdown_connections 2023-06-10T18:48:13.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.467+0000 7f432e4c3700 1 -- 172.21.15.104:0/3553673419 wait complete. 2023-06-10T18:48:13.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:48:13.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.600+0000 7f5de3199700 1 Processor -- start 2023-06-10T18:48:13.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.600+0000 7f5de3199700 1 -- start start 2023-06-10T18:48:13.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de3199700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de3199700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ddc0c93e0 0x7f5ddc15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de3199700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc06b2e0 con 0x7f5ddc070a20 2023-06-10T18:48:13.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de3199700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc15e150 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de3199700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc15db40 con 0x7f5ddc0c93e0 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de0f35700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de0f35700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58908/0 (socket says 172.21.15.104:58908) 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.601+0000 7f5de0f35700 1 -- 172.21.15.104:0/979978107 learned_addr learned my addr 172.21.15.104:0/979978107 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5ddbfff700 1 --2- 172.21.15.104:0/979978107 >> v2:172.21.15.114:3300/0 conn(0x7f5ddc0c93e0 0x7f5ddc15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de0f35700 1 -- 172.21.15.104:0/979978107 >> v2:172.21.15.114:3300/0 conn(0x7f5ddc0c93e0 msgr2=0x7f5ddc15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:13.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de0f35700 1 --2- 172.21.15.104:0/979978107 >> v2:172.21.15.114:3300/0 conn(0x7f5ddc0c93e0 0x7f5ddc15e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de0f35700 1 -- 172.21.15.104:0/979978107 >> v1:172.21.15.114:6789/0 conn(0x7f5ddc070a20 legacy=0x7f5ddc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de0f35700 1 -- 172.21.15.104:0/979978107 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc06b530 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de0f35700 1 --2- 172.21.15.104:0/979978107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0cc8e0 tx=0x7f5dd0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596cd8a18c52a582 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5ddb7fe700 1 -- 172.21.15.104:0/979978107 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0004230 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5ddb7fe700 1 -- 172.21.15.104:0/979978107 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd0004390 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5ddb7fe700 1 -- 172.21.15.104:0/979978107 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd00036a0 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 -- 172.21.15.104:0/979978107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 msgr2=0x7f5ddc0717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 --2- 172.21.15.104:0/979978107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0cc8e0 tx=0x7f5dd0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 -- 172.21.15.104:0/979978107 shutdown_connections 2023-06-10T18:48:13.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 --2- 172.21.15.104:0/979978107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0717d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 --2- 172.21.15.104:0/979978107 >> v2:172.21.15.114:3300/0 conn(0x7f5ddc0c93e0 0x7f5ddc15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.602+0000 7f5de3199700 1 -- 172.21.15.104:0/979978107 >> 172.21.15.104:0/979978107 conn(0x7f5ddc0b1610 msgr2=0x7f5ddc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:13.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.603+0000 7f5de3199700 1 -- 172.21.15.104:0/979978107 shutdown_connections 2023-06-10T18:48:13.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.603+0000 7f5de3199700 1 -- 172.21.15.104:0/979978107 wait complete. 2023-06-10T18:48:13.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.604+0000 7f5de3199700 1 Processor -- start 2023-06-10T18:48:13.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.604+0000 7f5de3199700 1 -- start start 2023-06-10T18:48:13.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ddc070a20 0x7f5ddc0c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc0c93e0 0x7f5ddc152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc152570 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1e32f0 con 0x7f5ddc0c93e0 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1e35d0 con 0x7f5ddc070a20 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58914/0 (socket says 172.21.15.104:58914) 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 -- 172.21.15.104:0/2701598717 learned_addr learned my addr 172.21.15.104:0/2701598717 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5ddbfff700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc0c93e0 0x7f5ddc152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de0f35700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ddc070a20 0x7f5ddc0c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 -- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ddc070a20 msgr2=0x7f5ddc0c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ddc070a20 0x7f5ddc0c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 -- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc0c93e0 msgr2=0x7f5ddc152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc0c93e0 0x7f5ddc152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 -- 172.21.15.104:0/2701598717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd0009550 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de1736700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400c7a0 tx=0x7f5dd400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4003e10 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd4005070 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc1e38b0 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4005420 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.605+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc1e3e70 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.606+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc1e4090 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.607+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5dd4005580 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.607+0000 7f5dd97fa700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 0x7f5dc006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:13.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.607+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5dd4088bc0 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.607+0000 7f5de0f35700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 0x7f5dc006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:13.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.608+0000 7f5de0f35700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 0x7f5dc006d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009130 tx=0x7f5dd0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:13.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.611+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5dd4053b30 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.811+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5ddc074050 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.811+0000 7f5dd97fa700 1 -- 172.21.15.104:0/2701598717 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5dd4017020 con 0x7f5ddc15a0b0 2023-06-10T18:48:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 msgr2=0x7f5dc006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 0x7f5dc006d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009130 tx=0x7f5dd0008030 comp rx=0 tx=0).stop 2023-06-10T18:48:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 msgr2=0x7f5ddc0c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400c7a0 tx=0x7f5dd400cab0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 shutdown_connections 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ddc15a0b0 0x7f5ddc0c5390 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ddc070a20 0x7f5ddc0c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5dc006ace0 0x7f5dc006d190 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 --2- 172.21.15.104:0/2701598717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc0c93e0 0x7f5ddc152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 >> 172.21.15.104:0/2701598717 conn(0x7f5ddc0b1610 msgr2=0x7f5ddc15cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 shutdown_connections 2023-06-10T18:48:13.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:13.813+0000 7f5de3199700 1 -- 172.21.15.104:0/2701598717 wait complete. 2023-06-10T18:48:13.820 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:48:13.830 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:48:13.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:13.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:13.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[123861]: fuse finished with error 0 and tester_r 0 2023-06-10T18:48:13.872 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:48:19.973 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:48:19.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:19.974 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:48:19.995 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:48:19.996 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:20.017 INFO:tasks.cephfs.mount:Checking file a 2023-06-10T18:48:20.018 DEBUG:teuthology.orchestra.run.smithi114:> ls /home/ubuntu/cephtest/mnt.1/a 2023-06-10T18:48:20.078 INFO:teuthology.orchestra.run.smithi114.stdout:/home/ubuntu/cephtest/mnt.1/a 2023-06-10T18:48:20.079 INFO:tasks.cephfs.mount:Checking file b 2023-06-10T18:48:20.079 DEBUG:teuthology.orchestra.run.smithi114:> ls /home/ubuntu/cephtest/mnt.1/b 2023-06-10T18:48:20.140 INFO:teuthology.orchestra.run.smithi114.stdout:/home/ubuntu/cephtest/mnt.1/b 2023-06-10T18:48:20.140 INFO:tasks.cephfs.mount:Checking file c 2023-06-10T18:48:20.141 DEBUG:teuthology.orchestra.run.smithi114:> ls /home/ubuntu/cephtest/mnt.1/c 2023-06-10T18:48:20.200 INFO:teuthology.orchestra.run.smithi114.stdout:/home/ubuntu/cephtest/mnt.1/c 2023-06-10T18:48:20.287 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:48:20.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.422+0000 7faf3623d700 1 Processor -- start 2023-06-10T18:48:20.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.422+0000 7faf3623d700 1 -- start start 2023-06-10T18:48:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.422+0000 7faf3623d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 0x7faf300d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf3623d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf3006b9f0 0x7faf300cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf3623d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3006bfd0 con 0x7faf300d0a30 2023-06-10T18:48:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf3623d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf300d0ea0 con 0x7faf3006b9f0 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf3623d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3006c2b0 con 0x7faf300b1e30 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf3006b9f0 0x7faf300cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf3006b9f0 0x7faf300cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58922/0 (socket says 172.21.15.104:58922) 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 -- 172.21.15.104:0/1685970619 learned_addr learned my addr 172.21.15.104:0/1685970619 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 0x7faf300d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 -- 172.21.15.104:0/1685970619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 msgr2=0x7faf300d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 0x7faf300d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 -- 172.21.15.104:0/1685970619 >> v1:172.21.15.114:6789/0 conn(0x7faf300d0a30 legacy=0x7faf3006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 -- 172.21.15.104:0/1685970619 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300ccf70 con 0x7faf3006b9f0 2023-06-10T18:48:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 0x7faf300d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2ffff700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3006b9f0 0x7faf300cda50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faf28009e50 tx=0x7faf2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4ff3840ebbbd7b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2e7fc700 1 -- 172.21.15.104:0/1685970619 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2800e070 con 0x7faf3006b9f0 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2e7fc700 1 -- 172.21.15.104:0/1685970619 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28003a20 con 0x7faf3006b9f0 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.423+0000 7faf2e7fc700 1 -- 172.21.15.104:0/1685970619 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28003dd0 con 0x7faf3006b9f0 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 -- 172.21.15.104:0/1685970619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3006b9f0 msgr2=0x7faf300cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3006b9f0 0x7faf300cda50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faf28009e50 tx=0x7faf2800b330 comp rx=0 tx=0).stop 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 -- 172.21.15.104:0/1685970619 shutdown_connections 2023-06-10T18:48:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf3006b9f0 0x7faf300cda50 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 --2- 172.21.15.104:0/1685970619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300b1e30 0x7faf300d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 -- 172.21.15.104:0/1685970619 >> 172.21.15.104:0/1685970619 conn(0x7faf30071ec0 msgr2=0x7faf3009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 -- 172.21.15.104:0/1685970619 shutdown_connections 2023-06-10T18:48:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.424+0000 7faf3623d700 1 -- 172.21.15.104:0/1685970619 wait complete. 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.425+0000 7faf3623d700 1 Processor -- start 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 -- start start 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 0x7faf3015ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 0x7faf300b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300cc1e0 con 0x7faf300b1e30 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3015e7a0 con 0x7faf3006b9f0 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf3623d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3015f570 con 0x7faf300d0a30 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf2effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf2effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58926/0 (socket says 172.21.15.104:58926) 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.426+0000 7faf2effd700 1 -- 172.21.15.104:0/2582091608 learned_addr learned my addr 172.21.15.104:0/2582091608 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 0x7faf3015ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2ffff700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 0x7faf300b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 -- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 msgr2=0x7faf300b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 0x7faf300b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 -- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 msgr2=0x7faf3015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 0x7faf3015ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 -- 172.21.15.104:0/2582091608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28009b30 con 0x7faf300b1e30 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 0x7faf3015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2ffff700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 0x7faf300b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf2effd700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7faf2400b470 tx=0x7faf2400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24011830 con 0x7faf300b1e30 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf24011990 con 0x7faf300b1e30 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3015b250 con 0x7faf300b1e30 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24011d40 con 0x7faf300b1e30 2023-06-10T18:48:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.427+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3015ecd0 con 0x7faf300b1e30 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.428+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf300d1860 con 0x7faf300b1e30 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.428+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf2401a400 con 0x7faf300b1e30 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.429+0000 7faf17fff700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 0x7faf1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.429+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faf24089620 con 0x7faf300b1e30 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.429+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 0x7faf1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.429+0000 7faf2f7fe700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 0x7faf1c06d190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faf20006fd0 tx=0x7faf20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:20.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.433+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf24054590 con 0x7faf300b1e30 2023-06-10T18:48:20.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.662+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faf30067090 con 0x7faf300b1e30 2023-06-10T18:48:20.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.663+0000 7faf17fff700 1 -- 172.21.15.104:0/2582091608 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7faf30067090 con 0x7faf300b1e30 2023-06-10T18:48:20.669 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:48:20.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 msgr2=0x7faf1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 0x7faf1c06d190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faf20006fd0 tx=0x7faf20008040 comp rx=0 tx=0).stop 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 msgr2=0x7faf300b7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7faf2400b470 tx=0x7faf2400b830 comp rx=0 tx=0).stop 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 shutdown_connections 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf1c06ace0 0x7faf1c06d190 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf3006b9f0 0x7faf3015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf300d0a30 0x7faf300b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 --2- 172.21.15.104:0/2582091608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300b1e30 0x7faf300b7dd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 >> 172.21.15.104:0/2582091608 conn(0x7faf30071ec0 msgr2=0x7faf3015be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 shutdown_connections 2023-06-10T18:48:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:20.665+0000 7faf3623d700 1 -- 172.21.15.104:0/2582091608 wait complete. 2023-06-10T18:48:20.683 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:48:20.683 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:48:20.684 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:48:20.684 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:48:20.685 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:20.686 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:48:20.686 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:48:20.687 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:48:20.687 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:20.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.689 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:48:20.712 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:48:20.712 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:48:20.712 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:48:20.712 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.713 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:48:20.713 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.713 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:20.713 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:20.713 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:20.714 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:20.714 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:20.714 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:48:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:48:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42184sec preferred_lft 42184sec 2023-06-10T18:48:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:48:20.715 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.716 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:48:20.716 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:20.716 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:48:20.716 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.716 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:48:20.717 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:48:20.718 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:20.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.719 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:48:20.779 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:48:20.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.781 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:48:20.840 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:48:20.840 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:48:20.841 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:48:20.841 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:48:20.841 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:48:20.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.842 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:48:20.842 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:48:20.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:48:20.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:48:20.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:48:20.843 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:48:20.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.945 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:48:20.962 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:48:20.962 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:48:20.962 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:20.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:20.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:48:21.028 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:48:21.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:21.029 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:21.086 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:21.086 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:21.087 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:21.143 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:21.209 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:21.210 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:21.275 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:21.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:21.277 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:21.333 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:48:21.334 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:48:21.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:21.402 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:21.403 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:21.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:48:21.456+0000 7fbf96d865c0 -1 init, newargv = 0x56503ed64980 newargc=15 2023-06-10T18:48:21.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[125265]: starting ceph client 2023-06-10T18:48:21.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[125265]: starting fuse 2023-06-10T18:48:21.471 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:21.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:21.473 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:21.529 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:48:21.529 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:48:21.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:21.530 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:48:21.530 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:48:21.531 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:48:21.531 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:48:21.531 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:21.532 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:48:21.532 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:48:21.532 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:48:21.533 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:48:21.533 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:21.534 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:48:21.534 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:48:21.534 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:48:21.534 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:48:21.535 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:21.535 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:48:21.535 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:21.536 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:21.536 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:21.536 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:48:21.537 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:48:21.537 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:48:21.537 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:21.538 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:21.538 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:48:21.538 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:48:21.652 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.125265.asok 2023-06-10T18:48:21.834 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125265.asok 2023-06-10T18:48:21.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:21.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125265.asok status 2023-06-10T18:48:21.989 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:21.989 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:48:21.989 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "125265", 2023-06-10T18:48:21.990 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6453, 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:48:21.991 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:48:21.992 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6453 2023-06-10T18:48:21.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:21.992 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3130832381 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:21.993 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3130832381 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.6453 192.168.0.1:0/3130832381", 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3130832381", 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:48:21.994 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 25, 2023-06-10T18:48:21.995 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 56, 2023-06-10T18:48:21.995 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:48:21.995 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:48:21.995 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:48:21.995 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:21.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:21.999 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:48:22.000 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:22.016 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:48:22.017 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:22.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:22.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:22.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:22.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.237+0000 7f34742cb700 1 Processor -- start 2023-06-10T18:48:22.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.237+0000 7f34742cb700 1 -- start start 2023-06-10T18:48:22.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f34742cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f346c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f34742cb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f346c06b820 0x7f346c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f34742cb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c06bfd0 con 0x7f346c0b1e30 2023-06-10T18:48:22.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f34742cb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346c06c2b0 con 0x7f346c06b820 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f34742cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ccf70 con 0x7f346c0d0c00 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f3472067700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f346c0b1e30 0x7f346c0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56774/0 (socket says 172.21.15.104:56774) 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.238+0000 7f3472067700 1 -- 172.21.15.104:0/765799836 learned_addr learned my addr 172.21.15.104:0/765799836 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313994162 0 0) 0x7f346c06bfd0 con 0x7f346c0b1e30 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34580035b0 con 0x7f346c0b1e30 2023-06-10T18:48:22.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3472868700 1 --2- 172.21.15.104:0/765799836 >> v2:172.21.15.114:3300/0 conn(0x7f346c06b820 0x7f346c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f346c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379977704 0 0) 0x7f34580035b0 con 0x7f346c0b1e30 2023-06-10T18:48:22.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 msgr2=0x7f346c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:22.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f346c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 >> v2:172.21.15.114:3300/0 conn(0x7f346c06b820 msgr2=0x7f346c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:22.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 --2- 172.21.15.104:0/765799836 >> v2:172.21.15.114:3300/0 conn(0x7f346c06b820 0x7f346c0cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0cd250 con 0x7f346c0b1e30 2023-06-10T18:48:22.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3468001670 con 0x7f346c0b1e30 2023-06-10T18:48:22.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 >> v1:172.21.15.104:6789/0 conn(0x7f346c0b1e30 legacy=0x7f346c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3458003a00 0x7f3458003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3458004430 0x7f34580048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f3458005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c06c2b0 con 0x7f3458004430 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ccf70 con 0x7f3458003a00 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345800ba10 con 0x7f346c0d0c00 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3458004430 0x7f34580048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3472067700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3458003a00 0x7f3458003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3472868700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f3458005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 -- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 msgr2=0x7f3458005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f3458005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 -- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3458003a00 msgr2=0x7f3458003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3458003a00 0x7f3458003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3471866700 1 -- 172.21.15.104:0/765799836 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34580035b0 con 0x7f3458004430 2023-06-10T18:48:22.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.239+0000 7f3472868700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346c0d0c00 0x7f3458005060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:22.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f3471866700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3458004430 0x7f34580048f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3460009b60 tx=0x7f3458005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784478b04d8ef821 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:22.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346000e6c0 con 0x7f3458004430 2023-06-10T18:48:22.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346000e820 con 0x7f3458004430 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f3471065700 1 -- 172.21.15.104:0/765799836 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346000eb90 con 0x7f3458004430 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 -- 172.21.15.104:0/765799836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3458004430 msgr2=0x7f34580048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 --2- 172.21.15.104:0/765799836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3458004430 0x7f34580048f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3460009b60 tx=0x7f3458005880 comp rx=0 tx=0).stop 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f3472868700 1 -- 172.21.15.104:0/765799836 reap_dead start 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 -- 172.21.15.104:0/765799836 shutdown_connections 2023-06-10T18:48:22.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 -- 172.21.15.104:0/765799836 >> 172.21.15.104:0/765799836 conn(0x7f346c071ec0 msgr2=0x7f346c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:22.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 -- 172.21.15.104:0/765799836 shutdown_connections 2023-06-10T18:48:22.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.240+0000 7f34742cb700 1 -- 172.21.15.104:0/765799836 wait complete. 2023-06-10T18:48:22.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 Processor -- start 2023-06-10T18:48:22.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 -- start start 2023-06-10T18:48:22.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3458004430 0x7f346c15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346c0cc430 0x7f346c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ccc40 con 0x7f346c0cb080 2023-06-10T18:48:22.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0cc960 con 0x7f346c0cc430 2023-06-10T18:48:22.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f34742cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ca5a0 con 0x7f3458004430 2023-06-10T18:48:22.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f3472868700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f3472868700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58962/0 (socket says 172.21.15.104:58962) 2023-06-10T18:48:22.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.242+0000 7f3472868700 1 -- 172.21.15.104:0/3186014856 learned_addr learned my addr 172.21.15.104:0/3186014856 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:22.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3471866700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346c0cc430 0x7f346c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472067700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3458004430 0x7f346c15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 -- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3458004430 msgr2=0x7f346c15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3458004430 0x7f346c15ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 -- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346c0cc430 msgr2=0x7f346c15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346c0cc430 0x7f346c15cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 -- 172.21.15.104:0/3186014856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0ca880 con 0x7f346c0cb080 2023-06-10T18:48:22.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f3472868700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f346c0d1f00 tx=0x7f34640104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:22.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346400e050 con 0x7f346c0cb080 2023-06-10T18:48:22.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346400f040 con 0x7f346c0cb080 2023-06-10T18:48:22.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c0cab60 con 0x7f346c0cb080 2023-06-10T18:48:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464010e50 con 0x7f346c0cb080 2023-06-10T18:48:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.243+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c15a8c0 con 0x7f346c0cb080 2023-06-10T18:48:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.244+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c0d1860 con 0x7f346c0cb080 2023-06-10T18:48:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.244+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3464015440 con 0x7f346c0cb080 2023-06-10T18:48:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.245+0000 7f345effd700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 0x7f345006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.245+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3464089770 con 0x7f346c0cb080 2023-06-10T18:48:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.245+0000 7f3472067700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 0x7f345006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:22.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.246+0000 7f3472067700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 0x7f345006d190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3468001550 tx=0x7f3468009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:22.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.249+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3464054770 con 0x7f346c0cb080 2023-06-10T18:48:22.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.478+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f346c0ca400 con 0x7f346c0cb080 2023-06-10T18:48:22.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.478+0000 7f345effd700 1 -- 172.21.15.104:0/3186014856 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f346c0ca400 con 0x7f346c0cb080 2023-06-10T18:48:22.484 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:22.484 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:22.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 msgr2=0x7f345006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 0x7f345006d190 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3468001550 tx=0x7f3468009040 comp rx=0 tx=0).stop 2023-06-10T18:48:22.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 msgr2=0x7f346c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f346c0d1f00 tx=0x7f34640104a0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 shutdown_connections 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f346c0cb080 0x7f346c15a380 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3458004430 0x7f346c15ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f345006ace0 0x7f345006d190 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 --2- 172.21.15.104:0/3186014856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346c0cc430 0x7f346c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:22.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 >> 172.21.15.104:0/3186014856 conn(0x7f346c071ec0 msgr2=0x7f346c0d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:22.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 shutdown_connections 2023-06-10T18:48:22.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:22.480+0000 7f34742cb700 1 -- 172.21.15.104:0/3186014856 wait complete. 2023-06-10T18:48:22.488 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:22.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2023-06-10T18:48:22.651 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":6453,"entity":{"name":{"type":"client","num":6453},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":3130832381}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":1.1769740609999999,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.99728891501952333,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6453 192.168.0.1:0/3130832381","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"0","hostname":"smithi104","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"125265","root":"/"}},{"id":5366,"entity":{"name":{"type":"client","num":5366},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":4139818742}},"state":"open","num_leases":3,"num_caps":4,"request_load_avg":3,"uptime":14.317684459000001,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":8.8721557154844248,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":4,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5366 192.168.0.1:0/4139818742","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"109373","root":"/"}}] 2023-06-10T18:48:22.652 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "0", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "125265", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 3130832381, "type": "any" }, "name": { "num": 6453, "type": "client" } }, "id": 6453, "inst": "client.6453 192.168.0.1:0/3130832381", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9972889150195233 }, "state": "open", "uptime": 1.176974061 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "109373", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 4139818742, "type": "any" }, "name": { "num": 5366, "type": "client" } }, "id": 5366, "inst": "client.5366 192.168.0.1:0/4139818742", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 4, "num_caps": 4, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 3, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 3, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 8.872155715484425 }, "state": "open", "uptime": 14.317684459 } ] 2023-06-10T18:48:22.653 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:22.653 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:48:22.655 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:48:22.655 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:48:22.656 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:48:22.656 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:22.657 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:48:22.658 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:48:22.658 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:48:22.658 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:48:22.659 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:22.659 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:48:22.659 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:48:22.660 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:48:22.660 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:48:22.660 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:22.661 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:48:22.661 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:22.662 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:22.662 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:22.662 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:48:22.663 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:48:22.663 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:48:22.664 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:22.664 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:22.665 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:48:22.665 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:48:22.748 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.125265.asok 2023-06-10T18:48:22.930 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125265.asok 2023-06-10T18:48:22.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:22.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125265.asok mds_sessions 2023-06-10T18:48:23.084 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6453, 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6453 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3130832381 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.6453 v1:192.168.0.1:0/3130832381", 2023-06-10T18:48:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/3130832381", 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:48:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6834", 2023-06-10T18:48:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1791317446 2023-06-10T18:48:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6836", 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1791317446 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:49:22.458929+0000", 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:48:22.458929+0000", 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 1, 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:48:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:48:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 25 2023-06-10T18:48:23.091 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:23.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:23.095 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:48:23.095 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:48:23.095 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:48:23.096 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:48:23.096 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:23.096 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:48:23.097 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:48:23.097 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:48:23.097 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:48:23.098 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:23.098 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:48:23.098 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:48:23.099 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:48:23.099 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:48:23.099 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:23.100 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:48:23.100 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:23.100 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:23.101 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:23.102 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:48:23.102 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:48:23.103 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:48:23.104 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:23.104 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:23.105 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:48:23.107 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:48:23.196 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.109373.asok 2023-06-10T18:48:23.378 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.109373.asok 2023-06-10T18:48:23.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:23.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.109373.asok mds_sessions 2023-06-10T18:48:23.534 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5366, 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5366 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:23.535 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 4139818742 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.5366 v1:192.168.0.1:0/4139818742", 2023-06-10T18:48:23.536 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "v1:192.168.0.1:0/4139818742", 2023-06-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions": [ 2023-06-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2023-06-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": 0, 2023-06-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi114.stdout: "addrs": { 2023-06-10T18:48:23.537 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6834", 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1791317446 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2023-06-10T18:48:23.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6836", 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1791317446 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: "seq": 1, 2023-06-10T18:48:23.539 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_gen": 0, 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_ttl": "2023-06-10T18:49:09.318647+0000", 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "last_cap_renew_request": "2023-06-10T18:48:09.318647+0000", 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_renew_seq": 2, 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "num_caps": 4, 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "open" 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2023-06-10T18:48:23.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 25 2023-06-10T18:48:23.541 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:48:23.544 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:48:23.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:48:23.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:48:23.637 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:48:23.637 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:23.638 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:23.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:23.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.809+0000 7f94b6072700 1 Processor -- start 2023-06-10T18:48:23.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.809+0000 7f94b6072700 1 -- start start 2023-06-10T18:48:23.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94b6072700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94b00aeaa0 0x7f94b00cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:23.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94b6072700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00c9e60 con 0x7f94b00cd680 2023-06-10T18:48:23.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94b6072700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00cdaa0 con 0x7f94b00b1080 2023-06-10T18:48:23.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94b6072700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b00c9bc0 con 0x7f94b00aeaa0 2023-06-10T18:48:23.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94af7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94b00aeaa0 0x7f94b00cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:23.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94affff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f94b00cd680 0x7f94b006dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56798/0 (socket says 172.21.15.104:56798) 2023-06-10T18:48:23.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94affff700 1 -- 172.21.15.104:0/1311808970 learned_addr learned my addr 172.21.15.104:0/1311808970 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:23.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94af7fe700 1 -- 172.21.15.104:0/1311808970 >> v1:172.21.15.114:6789/0 conn(0x7f94b00b1080 legacy=0x7f94b00ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:23.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94af7fe700 1 -- 172.21.15.104:0/1311808970 >> v1:172.21.15.104:6789/0 conn(0x7f94b00cd680 legacy=0x7f94b006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:23.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.810+0000 7f94ae7fc700 1 -- 172.21.15.104:0/1311808970 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667972649 0 0) 0x7f94b00c9e60 con 0x7f94b00cd680 2023-06-10T18:48:23.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94af7fe700 1 -- 172.21.15.104:0/1311808970 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00aee70 con 0x7f94b00aeaa0 2023-06-10T18:48:23.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94af7fe700 1 --2- 172.21.15.104:0/1311808970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00aeaa0 0x7f94b00cd140 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9498009820 tx=0x7f9498009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fda0fb3eb125a6a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:23.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94ae7fc700 1 -- 172.21.15.104:0/1311808970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9498005e30 con 0x7f94b00aeaa0 2023-06-10T18:48:23.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94ae7fc700 1 -- 172.21.15.104:0/1311808970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9498006030 con 0x7f94b00aeaa0 2023-06-10T18:48:23.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94ae7fc700 1 -- 172.21.15.104:0/1311808970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980063e0 con 0x7f94b00aeaa0 2023-06-10T18:48:23.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 -- 172.21.15.104:0/1311808970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00aeaa0 msgr2=0x7f94b00cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:23.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 --2- 172.21.15.104:0/1311808970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00aeaa0 0x7f94b00cd140 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9498009820 tx=0x7f9498009b30 comp rx=0 tx=0).stop 2023-06-10T18:48:23.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 -- 172.21.15.104:0/1311808970 shutdown_connections 2023-06-10T18:48:23.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 --2- 172.21.15.104:0/1311808970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00aeaa0 0x7f94b00cd140 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:23.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 -- 172.21.15.104:0/1311808970 >> 172.21.15.104:0/1311808970 conn(0x7f94b006b8a0 msgr2=0x7f94b00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:23.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 -- 172.21.15.104:0/1311808970 shutdown_connections 2023-06-10T18:48:23.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.811+0000 7f94b6072700 1 -- 172.21.15.104:0/1311808970 wait complete. 2023-06-10T18:48:23.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.813+0000 7f94b6072700 1 Processor -- start 2023-06-10T18:48:23.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.813+0000 7f94b6072700 1 -- start start 2023-06-10T18:48:23.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.813+0000 7f94b6072700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94b00aeaa0 0x7f94b00b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:23.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.813+0000 7f94b6072700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:23.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94b00cd680 0x7f94b015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:23.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00b2970 con 0x7f94b00b1080 2023-06-10T18:48:23.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0157fb0 con 0x7f94b00cd680 2023-06-10T18:48:23.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b015c280 con 0x7f94b00aeaa0 2023-06-10T18:48:23.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:23.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58974/0 (socket says 172.21.15.104:58974) 2023-06-10T18:48:23.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94af7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94b00aeaa0 0x7f94b00b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:23.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 -- 172.21.15.104:0/4131702780 learned_addr learned my addr 172.21.15.104:0/4131702780 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:23.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94affff700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94b00cd680 0x7f94b015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:23.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 -- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94b00aeaa0 msgr2=0x7f94b00b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:23.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94b00aeaa0 0x7f94b00b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:23.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 -- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94b00cd680 msgr2=0x7f94b015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94b00cd680 0x7f94b015bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 -- 172.21.15.104:0/4131702780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9498009430 con 0x7f94b00b1080 2023-06-10T18:48:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94aeffd700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f94a000c4e0 tx=0x7f94a000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a00052d0 con 0x7f94b00b1080 2023-06-10T18:48:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a0005430 con 0x7f94b00b1080 2023-06-10T18:48:23.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b00b2ba0 con 0x7f94b00b1080 2023-06-10T18:48:23.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a00038c0 con 0x7f94b00b1080 2023-06-10T18:48:23.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.814+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b00b3160 con 0x7f94b00b1080 2023-06-10T18:48:23.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.815+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b0074050 con 0x7f94b00b1080 2023-06-10T18:48:23.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.816+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94a0003a20 con 0x7f94b00b1080 2023-06-10T18:48:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.816+0000 7f94a7fff700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.816+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f94a00888b0 con 0x7f94b00b1080 2023-06-10T18:48:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.816+0000 7f94af7fe700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.817+0000 7f94af7fe700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9498013040 tx=0x7f9498008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:23.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:23.820+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94a00537a0 con 0x7f94b00b1080 2023-06-10T18:48:24.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.008+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94b00704c0 con 0x7f94b00b1080 2023-06-10T18:48:24.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.010+0000 7f94a7fff700 1 -- 172.21.15.104:0/4131702780 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f94a0056dc0 con 0x7f94b00b1080 2023-06-10T18:48:24.016 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:24.017 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:24.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 msgr2=0x7f949006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9498013040 tx=0x7f9498008070 comp rx=0 tx=0).stop 2023-06-10T18:48:24.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 msgr2=0x7f94b0157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f94a000c4e0 tx=0x7f94a000c7f0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 shutdown_connections 2023-06-10T18:48:24.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94b00cd680 0x7f94b015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94b00aeaa0 0x7f94b00b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 --2- 172.21.15.104:0/4131702780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94b00b1080 0x7f94b0157a70 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 >> 172.21.15.104:0/4131702780 conn(0x7f94b006b8a0 msgr2=0x7f94b00cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 shutdown_connections 2023-06-10T18:48:24.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.012+0000 7f94b6072700 1 -- 172.21.15.104:0/4131702780 wait complete. 2023-06-10T18:48:24.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:47:57.299455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:47:58.209852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:24.032 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:24.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.151+0000 7f8e302fb700 1 Processor -- start 2023-06-10T18:48:24.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.151+0000 7f8e302fb700 1 -- start start 2023-06-10T18:48:24.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e302fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2815c580 0x7f8e280b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e302fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e2806e6e0 0x7f8e280b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e302fb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2806e250 con 0x7f8e2806cde0 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e302fb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2815c950 con 0x7f8e2806e6e0 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e302fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280cf0e0 con 0x7f8e2815c580 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e2d896700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e2806e6e0 0x7f8e280b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e2d896700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e2806e6e0 0x7f8e280b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58980/0 (socket says 172.21.15.104:58980) 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e2d896700 1 -- 172.21.15.104:0/2531316756 learned_addr learned my addr 172.21.15.104:0/2531316756 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.152+0000 7f8e2e097700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2815c580 0x7f8e280b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d896700 1 -- 172.21.15.104:0/2531316756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2815c580 msgr2=0x7f8e280b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d896700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2815c580 0x7f8e280b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d896700 1 -- 172.21.15.104:0/2531316756 >> v1:172.21.15.114:6789/0 conn(0x7f8e2806cde0 legacy=0x7f8e28069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d896700 1 -- 172.21.15.104:0/2531316756 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e280b1e80 con 0x7f8e2806e6e0 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d896700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806e6e0 0x7f8e280b15b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8e240098f0 tx=0x7f8e24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5a9d8946cf43f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d095700 1 -- 172.21.15.104:0/2531316756 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24011830 con 0x7f8e2806e6e0 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d095700 1 -- 172.21.15.104:0/2531316756 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e24011990 con 0x7f8e2806e6e0 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e2d095700 1 -- 172.21.15.104:0/2531316756 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24011d20 con 0x7f8e2806e6e0 2023-06-10T18:48:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 -- 172.21.15.104:0/2531316756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806e6e0 msgr2=0x7f8e280b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806e6e0 0x7f8e280b15b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8e240098f0 tx=0x7f8e24009c00 comp rx=0 tx=0).stop 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 -- 172.21.15.104:0/2531316756 shutdown_connections 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2815c580 0x7f8e280b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 --2- 172.21.15.104:0/2531316756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806e6e0 0x7f8e280b15b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.153+0000 7f8e302fb700 1 -- 172.21.15.104:0/2531316756 >> 172.21.15.104:0/2531316756 conn(0x7f8e2806b8a0 msgr2=0x7f8e280cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.154+0000 7f8e302fb700 1 -- 172.21.15.104:0/2531316756 shutdown_connections 2023-06-10T18:48:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.154+0000 7f8e302fb700 1 -- 172.21.15.104:0/2531316756 wait complete. 2023-06-10T18:48:24.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 Processor -- start 2023-06-10T18:48:24.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 -- start start 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2806e6e0 0x7f8e280c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e2815c580 0x7f8e281563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28154c60 con 0x7f8e2806cde0 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28154f50 con 0x7f8e2815c580 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.155+0000 7f8e302fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280c4aa0 con 0x7f8e2806e6e0 2023-06-10T18:48:24.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58982/0 (socket says 172.21.15.104:58982) 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 -- 172.21.15.104:0/1467607008 learned_addr learned my addr 172.21.15.104:0/1467607008 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2d896700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2806e6e0 0x7f8e280c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e898700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e2815c580 0x7f8e281563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 -- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2806e6e0 msgr2=0x7f8e280c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2806e6e0 0x7f8e280c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 -- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e2815c580 msgr2=0x7f8e281563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e2815c580 0x7f8e281563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 -- 172.21.15.104:0/1467607008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24009550 con 0x7f8e2806cde0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e2e097700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800e990 tx=0x7f8e1800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180096f0 con 0x7f8e2806cde0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18013680 con 0x7f8e2806cde0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180139f0 con 0x7f8e2806cde0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e28156c10 con 0x7f8e2806cde0 2023-06-10T18:48:24.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.156+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e281571d0 con 0x7f8e2806cde0 2023-06-10T18:48:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.157+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e28074050 con 0x7f8e2806cde0 2023-06-10T18:48:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.158+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e18013b50 con 0x7f8e2806cde0 2023-06-10T18:48:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.158+0000 7f8e1effd700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 0x7f8e0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.158+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8e180898d0 con 0x7f8e2806cde0 2023-06-10T18:48:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.158+0000 7f8e2d896700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 0x7f8e0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.159+0000 7f8e2d896700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 0x7f8e0c06d190 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8e24014040 tx=0x7f8e24008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.162+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e180582d0 con 0x7f8e2806cde0 2023-06-10T18:48:24.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.346+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e280cfd40 con 0x7f8e2806cde0 2023-06-10T18:48:24.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.346+0000 7f8e1effd700 1 -- 172.21.15.104:0/1467607008 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e18017020 con 0x7f8e2806cde0 2023-06-10T18:48:24.352 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:24.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 msgr2=0x7f8e0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 0x7f8e0c06d190 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8e24014040 tx=0x7f8e24008030 comp rx=0 tx=0).stop 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 msgr2=0x7f8e28154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800e990 tx=0x7f8e1800eca0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 shutdown_connections 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e0c06ace0 0x7f8e0c06d190 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e2815c580 0x7f8e281563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e2806cde0 0x7f8e28154720 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 --2- 172.21.15.104:0/1467607008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e2806e6e0 0x7f8e280c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 >> 172.21.15.104:0/1467607008 conn(0x7f8e2806b8a0 msgr2=0x7f8e2815a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 shutdown_connections 2023-06-10T18:48:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.348+0000 7f8e302fb700 1 -- 172.21.15.104:0/1467607008 wait complete. 2023-06-10T18:48:24.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:24.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.486+0000 7fdff5461700 1 Processor -- start 2023-06-10T18:48:24.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.486+0000 7fdff5461700 1 -- start start 2023-06-10T18:48:24.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 0x7fdff006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdff00ce390 0x7fdff0070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 0x7fdff0069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff00713f0 con 0x7fdff00ce390 2023-06-10T18:48:24.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff006d7d0 con 0x7fdff01586a0 2023-06-10T18:48:24.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdff5461700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff00cd6c0 con 0x7fdff0070f50 2023-06-10T18:48:24.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdff00ce390 0x7fdff0070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdff00ce390 0x7fdff0070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58990/0 (socket says 172.21.15.104:58990) 2023-06-10T18:48:24.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 -- 172.21.15.104:0/1891862013 learned_addr learned my addr 172.21.15.104:0/1891862013 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfef7fe700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 0x7fdff0069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfeeffd700 1 --2- 172.21.15.104:0/1891862013 >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 0x7fdff006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 -- 172.21.15.104:0/1891862013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 msgr2=0x7fdff0069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:24.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 0x7fdff0069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.487+0000 7fdfee7fc700 1 -- 172.21.15.104:0/1891862013 >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 msgr2=0x7fdff006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1891862013 >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 0x7fdff006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfee7fc700 1 -- 172.21.15.104:0/1891862013 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff00c95d0 con 0x7fdff00ce390 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfef7fe700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 0x7fdff0069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfeeffd700 1 --2- 172.21.15.104:0/1891862013 >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 0x7fdff006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff00ce390 0x7fdff0070a10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdff006ca00 tx=0x7fdfe000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4436a205acb4f2a8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfedffb700 1 -- 172.21.15.104:0/1891862013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe000dd30 con 0x7fdff00ce390 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfedffb700 1 -- 172.21.15.104:0/1891862013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe000de90 con 0x7fdff00ce390 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdfedffb700 1 -- 172.21.15.104:0/1891862013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0010690 con 0x7fdff00ce390 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 -- 172.21.15.104:0/1891862013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff00ce390 msgr2=0x7fdff0070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff00ce390 0x7fdff0070a10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdff006ca00 tx=0x7fdfe000be70 comp rx=0 tx=0).stop 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 -- 172.21.15.104:0/1891862013 shutdown_connections 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff0070f50 0x7fdff0069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 --2- 172.21.15.104:0/1891862013 >> v2:172.21.15.114:3300/0 conn(0x7fdff01586a0 0x7fdff006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 --2- 172.21.15.104:0/1891862013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff00ce390 0x7fdff0070a10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 -- 172.21.15.104:0/1891862013 >> 172.21.15.104:0/1891862013 conn(0x7fdff00ceec0 msgr2=0x7fdff00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 -- 172.21.15.104:0/1891862013 shutdown_connections 2023-06-10T18:48:24.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.488+0000 7fdff5461700 1 -- 172.21.15.104:0/1891862013 wait complete. 2023-06-10T18:48:24.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.490+0000 7fdff5461700 1 Processor -- start 2023-06-10T18:48:24.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.490+0000 7fdff5461700 1 -- start start 2023-06-10T18:48:24.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff00ce390 0x7fdff014b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff01586a0 0x7fdff014b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff00c68b0 con 0x7fdff0070f50 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff00c6b90 con 0x7fdff00ce390 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdff5461700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0159050 con 0x7fdff01586a0 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58996/0 (socket says 172.21.15.104:58996) 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfeeffd700 1 -- 172.21.15.104:0/1223530097 learned_addr learned my addr 172.21.15.104:0/1223530097 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfef7fe700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff01586a0 0x7fdff014b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff00ce390 0x7fdff014b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.491+0000 7fdfeeffd700 1 -- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff01586a0 msgr2=0x7fdff014b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfeeffd700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff01586a0 0x7fdff014b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfeeffd700 1 -- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff00ce390 msgr2=0x7fdff014b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfeeffd700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff00ce390 0x7fdff014b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfeeffd700 1 -- 172.21.15.104:0/1223530097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe000b7f0 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfeeffd700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c7f0 tx=0x7fdfe400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4011800 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe4011960 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff0159330 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4011d10 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.492+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff014c120 con 0x7fdff0070f50 2023-06-10T18:48:24.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.493+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff0074050 con 0x7fdff0070f50 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.493+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdfe4018360 con 0x7fdff0070f50 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.493+0000 7fdfd77fe700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 0x7fdfdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.494+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdfe4089520 con 0x7fdff0070f50 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.494+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 0x7fdfdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.494+0000 7fdfee7fc700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 0x7fdfdc06d190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000beb0 tx=0x7fdfe0012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.498+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdfe4054490 con 0x7fdff0070f50 2023-06-10T18:48:24.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.683+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdff0070ab0 con 0x7fdff0070f50 2023-06-10T18:48:24.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.683+0000 7fdfd77fe700 1 -- 172.21.15.104:0/1223530097 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdfe4057ab0 con 0x7fdff0070f50 2023-06-10T18:48:24.689 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:24.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 msgr2=0x7fdfdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 0x7fdfdc06d190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000beb0 tx=0x7fdfe0012000 comp rx=0 tx=0).stop 2023-06-10T18:48:24.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 msgr2=0x7fdff0158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c7f0 tx=0x7fdfe400cbb0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 shutdown_connections 2023-06-10T18:48:24.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdff01586a0 0x7fdff014b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdff0070f50 0x7fdff0158a70 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfdc06ace0 0x7fdfdc06d190 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 --2- 172.21.15.104:0/1223530097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff00ce390 0x7fdff014b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 >> 172.21.15.104:0/1223530097 conn(0x7fdff00ceec0 msgr2=0x7fdff00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 shutdown_connections 2023-06-10T18:48:24.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.685+0000 7fdff5461700 1 -- 172.21.15.104:0/1223530097 wait complete. 2023-06-10T18:48:24.703 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:48:24.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.825+0000 7f5f8c2ec700 1 Processor -- start 2023-06-10T18:48:24.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.825+0000 7f5f8c2ec700 1 -- start start 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8c2ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8c2ec700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8c2ec700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8415c8d0 con 0x7f5f8406a3b0 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8c2ec700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f840aef20 con 0x7f5f840cf200 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8c2ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8406abd0 con 0x7f5f8415c500 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f89887700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:50156/0 (socket says 172.21.15.104:50156) 2023-06-10T18:48:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a889700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5f8406a3b0 0x7f5f8406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:34450/0 (socket says 172.21.15.104:34450) 2023-06-10T18:48:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 -- 172.21.15.104:0/900293358 learned_addr learned my addr 172.21.15.104:0/900293358 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f89887700 1 --2- 172.21.15.104:0/900293358 >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:46506/0 (socket says 172.21.15.104:46506) 2023-06-10T18:48:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 -- 172.21.15.104:0/900293358 >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 msgr2=0x7f5f84069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 --2- 172.21.15.104:0/900293358 >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 -- 172.21.15.104:0/900293358 >> v1:172.21.15.114:6789/0 conn(0x7f5f8406a3b0 legacy=0x7f5f8406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f8a088700 1 -- 172.21.15.104:0/900293358 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8406a8f0 con 0x7f5f8415c500 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.826+0000 7f5f89887700 1 --2- 172.21.15.104:0/900293358 >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8a088700 1 --2- 172.21.15.104:0/900293358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f80006fd0 tx=0x7f5f80009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad3abe292270154 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f89086700 1 -- 172.21.15.104:0/900293358 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80005e30 con 0x7f5f8415c500 2023-06-10T18:48:24.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f89086700 1 -- 172.21.15.104:0/900293358 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80004030 con 0x7f5f8415c500 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f89086700 1 -- 172.21.15.104:0/900293358 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f800043e0 con 0x7f5f8415c500 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/900293358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 msgr2=0x7f5f840cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/900293358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f80006fd0 tx=0x7f5f80009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/900293358 shutdown_connections 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/900293358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840cecc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/900293358 >> v2:172.21.15.114:3300/0 conn(0x7f5f840cf200 0x7f5f84069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/900293358 >> 172.21.15.104:0/900293358 conn(0x7f5f8406b8a0 msgr2=0x7f5f840cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:24.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/900293358 shutdown_connections 2023-06-10T18:48:24.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.827+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/900293358 wait complete. 2023-06-10T18:48:24.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 Processor -- start 2023-06-10T18:48:24.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 -- start start 2023-06-10T18:48:24.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 0x7f5f84155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840c79e0 con 0x7f5f840cf200 2023-06-10T18:48:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840cc930 con 0x7f5f8406a3b0 2023-06-10T18:48:24.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f8c2ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840c66e0 con 0x7f5f8415c500 2023-06-10T18:48:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f89887700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f89887700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59012/0 (socket says 172.21.15.104:59012) 2023-06-10T18:48:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.829+0000 7f5f89887700 1 -- 172.21.15.104:0/2712623792 learned_addr learned my addr 172.21.15.104:0/2712623792 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8a088700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 0x7f5f84155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8a889700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 -- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 msgr2=0x7f5f840c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 -- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 msgr2=0x7f5f84155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 0x7f5f84155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 -- 172.21.15.104:0/2712623792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740094a0 con 0x7f5f840cf200 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8a088700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 0x7f5f84155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:24.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8a889700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f89887700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400ebe0 tx=0x7f5f74009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400c8b0 con 0x7f5f840cf200 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7400ca10 con 0x7f5f840cf200 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f80009500 con 0x7f5f840cf200 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400cda0 con 0x7f5f840cf200 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.830+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f840c6ca0 con 0x7f5f840cf200 2023-06-10T18:48:24.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.831+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84074050 con 0x7f5f840cf200 2023-06-10T18:48:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.832+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5f74013470 con 0x7f5f840cf200 2023-06-10T18:48:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.832+0000 7f5f7affd700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 0x7f5f6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.832+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5f74089710 con 0x7f5f840cf200 2023-06-10T18:48:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.832+0000 7f5f8a088700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 0x7f5f6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:24.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.833+0000 7f5f8a088700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 0x7f5f6806d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5f80006fd0 tx=0x7f5f80008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:24.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:24.836+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5f74054680 con 0x7f5f840cf200 2023-06-10T18:48:25.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.021+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f8406d510 con 0x7f5f840cf200 2023-06-10T18:48:25.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.021+0000 7f5f7affd700 1 -- 172.21.15.104:0/2712623792 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f74057ca0 con 0x7f5f840cf200 2023-06-10T18:48:25.028 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:48:25.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 msgr2=0x7f5f6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 0x7f5f6806d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5f80006fd0 tx=0x7f5f80008070 comp rx=0 tx=0).stop 2023-06-10T18:48:25.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 msgr2=0x7f5f841550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400ebe0 tx=0x7f5f74009e20 comp rx=0 tx=0).stop 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 shutdown_connections 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5f6806ace0 0x7f5f6806d190 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8406a3b0 0x7f5f84155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f8415c500 0x7f5f840c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 --2- 172.21.15.104:0/2712623792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f840cf200 0x7f5f841550e0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 >> 172.21.15.104:0/2712623792 conn(0x7f5f8406b8a0 msgr2=0x7f5f840b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:25.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 shutdown_connections 2023-06-10T18:48:25.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.023+0000 7f5f8c2ec700 1 -- 172.21.15.104:0/2712623792 wait complete. 2023-06-10T18:48:25.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:48:25.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.200+0000 7f80899a9700 1 Processor -- start 2023-06-10T18:48:25.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.200+0000 7f80899a9700 1 -- start start 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 0x7f80840d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80840d25c0 0x7f80840d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808406b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80840d0e00 con 0x7f80840d25c0 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80840af530 con 0x7f80840d04f0 2023-06-10T18:48:25.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80899a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80840af810 con 0x7f808406b510 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80840d25c0 0x7f80840d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80840d25c0 0x7f80840d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59020/0 (socket says 172.21.15.104:59020) 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 -- 172.21.15.104:0/2124928362 learned_addr learned my addr 172.21.15.104:0/2124928362 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80837fe700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f8082ffd700 1 --2- 172.21.15.104:0/2124928362 >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 0x7f80840d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 -- 172.21.15.104:0/2124928362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 msgr2=0x7f808406b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808406b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 -- 172.21.15.104:0/2124928362 >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 msgr2=0x7f80840d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 --2- 172.21.15.104:0/2124928362 >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 0x7f80840d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f80827fc700 1 -- 172.21.15.104:0/2124928362 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840afaf0 con 0x7f80840d25c0 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.201+0000 7f8082ffd700 1 --2- 172.21.15.104:0/2124928362 >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 0x7f80840d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80827fc700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f80840d08c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8084066690 tx=0x7f8074009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a67e31a4b3f987e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f8081ffb700 1 -- 172.21.15.104:0/2124928362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807400b870 con 0x7f80840d25c0 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f8081ffb700 1 -- 172.21.15.104:0/2124928362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807400b9d0 con 0x7f80840d25c0 2023-06-10T18:48:25.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f8081ffb700 1 -- 172.21.15.104:0/2124928362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807400bd60 con 0x7f80840d25c0 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 -- 172.21.15.104:0/2124928362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 msgr2=0x7f80840d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f80840d08c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8084066690 tx=0x7f8074009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 -- 172.21.15.104:0/2124928362 shutdown_connections 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808406b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 --2- 172.21.15.104:0/2124928362 >> v2:172.21.15.114:3300/0 conn(0x7f80840d04f0 0x7f80840d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 --2- 172.21.15.104:0/2124928362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f80840d08c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 -- 172.21.15.104:0/2124928362 >> 172.21.15.104:0/2124928362 conn(0x7f8084071ec0 msgr2=0x7f808409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 -- 172.21.15.104:0/2124928362 shutdown_connections 2023-06-10T18:48:25.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.202+0000 7f80899a9700 1 -- 172.21.15.104:0/2124928362 wait complete. 2023-06-10T18:48:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 Processor -- start 2023-06-10T18:48:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 -- start start 2023-06-10T18:48:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808409dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80840d04f0 0x7f8084157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80840cdf90 con 0x7f80840d25c0 2023-06-10T18:48:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80840ce270 con 0x7f80840d04f0 2023-06-10T18:48:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.204+0000 7f80899a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80840ce550 con 0x7f808406b510 2023-06-10T18:48:25.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59028/0 (socket says 172.21.15.104:59028) 2023-06-10T18:48:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 -- 172.21.15.104:0/212952123 learned_addr learned my addr 172.21.15.104:0/212952123 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f8082ffd700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808409dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80827fc700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80840d04f0 0x7f8084157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 -- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 msgr2=0x7f808409dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808409dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 -- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80840d04f0 msgr2=0x7f8084157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80840d04f0 0x7f8084157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 -- 172.21.15.104:0/212952123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009550 con 0x7f80840d25c0 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80837fe700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f807c007e60 tx=0x7f807c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c00ed30 con 0x7f80840d25c0 2023-06-10T18:48:25.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c014420 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0147d0 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80840ce830 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.205+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8084155090 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.206+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8084155220 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.207+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f807c014930 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.207+0000 7f806bfff700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.207+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f807c08a890 con 0x7f80840d25c0 2023-06-10T18:48:25.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.208+0000 7f8082ffd700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.208+0000 7f8082ffd700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f807800ab60 tx=0x7f8078008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.211+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f807c0558e0 con 0x7f80840d25c0 2023-06-10T18:48:25.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.397+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80840ced50 con 0x7f80840d25c0 2023-06-10T18:48:25.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.397+0000 7f806bfff700 1 -- 172.21.15.104:0/212952123 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f80840ced50 con 0x7f80840d25c0 2023-06-10T18:48:25.404 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 msgr2=0x7f807006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f807800ab60 tx=0x7f8078008040 comp rx=0 tx=0).stop 2023-06-10T18:48:25.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 msgr2=0x7f8084158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f807c007e60 tx=0x7f807c00e3c0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 shutdown_connections 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80840d25c0 0x7f8084158040 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808406b510 0x7f808409dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 --2- 172.21.15.104:0/212952123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80840d04f0 0x7f8084157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 >> 172.21.15.104:0/212952123 conn(0x7f8084071ec0 msgr2=0x7f80840cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:25.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 shutdown_connections 2023-06-10T18:48:25.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.399+0000 7f80899a9700 1 -- 172.21.15.104:0/212952123 wait complete. 2023-06-10T18:48:25.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:48:25.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.532+0000 7f60cb5b7700 1 Processor -- start 2023-06-10T18:48:25.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.532+0000 7f60cb5b7700 1 -- start start 2023-06-10T18:48:25.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60cb5b7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f60c406e790 0x7f60c40ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60cb5b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c415cb00 con 0x7f60c40cc710 2023-06-10T18:48:25.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60cb5b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40cd280 con 0x7f60c415c730 2023-06-10T18:48:25.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60cb5b7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c40aed80 con 0x7f60c406e790 2023-06-10T18:48:25.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c9b54700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f60c406e790 0x7f60c40ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c8b52700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f60c40cc710 0x7f60c40ca0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56862/0 (socket says 172.21.15.104:56862) 2023-06-10T18:48:25.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c8b52700 1 -- 172.21.15.104:0/1992974188 learned_addr learned my addr 172.21.15.104:0/1992974188 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:25.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c9b54700 1 -- 172.21.15.104:0/1992974188 >> v1:172.21.15.114:6789/0 conn(0x7f60c415c730 legacy=0x7f60c40cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:25.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c9b54700 1 -- 172.21.15.104:0/1992974188 >> v1:172.21.15.104:6789/0 conn(0x7f60c40cc710 legacy=0x7f60c40ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c9b54700 1 -- 172.21.15.104:0/1992974188 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40cd560 con 0x7f60c406e790 2023-06-10T18:48:25.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60bbfff700 1 -- 172.21.15.104:0/1992974188 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619544736 0 0) 0x7f60b4001110 con 0x7f60c40cc710 2023-06-10T18:48:25.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.533+0000 7f60c9b54700 1 --2- 172.21.15.104:0/1992974188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c406e790 0x7f60c40ca7c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009d90 tx=0x7f60bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c9bb2cd11b8ed1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60bbfff700 1 -- 172.21.15.104:0/1992974188 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00e070 con 0x7f60c406e790 2023-06-10T18:48:25.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60bbfff700 1 -- 172.21.15.104:0/1992974188 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc00bb90 con 0x7f60c406e790 2023-06-10T18:48:25.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60bbfff700 1 -- 172.21.15.104:0/1992974188 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc003a20 con 0x7f60c406e790 2023-06-10T18:48:25.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 -- 172.21.15.104:0/1992974188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c406e790 msgr2=0x7f60c40ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/1992974188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c406e790 0x7f60c40ca7c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009d90 tx=0x7f60bc00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 -- 172.21.15.104:0/1992974188 shutdown_connections 2023-06-10T18:48:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/1992974188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c406e790 0x7f60c40ca7c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 -- 172.21.15.104:0/1992974188 >> 172.21.15.104:0/1992974188 conn(0x7f60c406b8a0 msgr2=0x7f60c40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 -- 172.21.15.104:0/1992974188 shutdown_connections 2023-06-10T18:48:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.534+0000 7f60cb5b7700 1 -- 172.21.15.104:0/1992974188 wait complete. 2023-06-10T18:48:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.535+0000 7f60cb5b7700 1 Processor -- start 2023-06-10T18:48:25.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.535+0000 7f60cb5b7700 1 -- start start 2023-06-10T18:48:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c406e790 0x7f60c4149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60c415c730 0x7f60c414a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40af8f0 con 0x7f60c40cc710 2023-06-10T18:48:25.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40af020 con 0x7f60c406e790 2023-06-10T18:48:25.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60cb5b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4148370 con 0x7f60c415c730 2023-06-10T18:48:25.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59040/0 (socket says 172.21.15.104:59040) 2023-06-10T18:48:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 -- 172.21.15.104:0/2396688155 learned_addr learned my addr 172.21.15.104:0/2396688155 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c9353700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c406e790 0x7f60c4149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c9b54700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60c415c730 0x7f60c414a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 -- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60c415c730 msgr2=0x7f60c414a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60c415c730 0x7f60c414a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 -- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c406e790 msgr2=0x7f60c4149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c406e790 0x7f60c4149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.536+0000 7f60c8b52700 1 -- 172.21.15.104:0/2396688155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc009a70 con 0x7f60c40cc710 2023-06-10T18:48:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60c8b52700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c550 tx=0x7f60b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4004c80 con 0x7f60c40cc710 2023-06-10T18:48:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b4004e00 con 0x7f60c40cc710 2023-06-10T18:48:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c414a2c0 con 0x7f60c40cc710 2023-06-10T18:48:25.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4007310 con 0x7f60c40cc710 2023-06-10T18:48:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.537+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c40af510 con 0x7f60c40cc710 2023-06-10T18:48:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.538+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c4074050 con 0x7f60c40cc710 2023-06-10T18:48:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.538+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f60b4007470 con 0x7f60c40cc710 2023-06-10T18:48:25.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.539+0000 7f60b9ffb700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 0x7f60a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:25.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.539+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f60b40891b0 con 0x7f60c40cc710 2023-06-10T18:48:25.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.539+0000 7f60c9353700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 0x7f60a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:25.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.539+0000 7f60c9353700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 0x7f60a806d190 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f60c0006fd0 tx=0x7f60c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:25.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.543+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f60b4054110 con 0x7f60c40cc710 2023-06-10T18:48:25.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.745+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f60c415cb00 con 0x7f60c40cc710 2023-06-10T18:48:25.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.745+0000 7f60b9ffb700 1 -- 172.21.15.104:0/2396688155 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7f60c415cb00 con 0x7f60c40cc710 2023-06-10T18:48:25.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 msgr2=0x7f60a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 0x7f60a806d190 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f60c0006fd0 tx=0x7f60c0008040 comp rx=0 tx=0).stop 2023-06-10T18:48:25.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 msgr2=0x7f60c4149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:25.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c550 tx=0x7f60b400c910 comp rx=0 tx=0).stop 2023-06-10T18:48:25.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 shutdown_connections 2023-06-10T18:48:25.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f60a806ace0 0x7f60a806d190 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c406e790 0x7f60c4149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60c415c730 0x7f60c414a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 --2- 172.21.15.104:0/2396688155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60c40cc710 0x7f60c4149a40 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:25.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 >> 172.21.15.104:0/2396688155 conn(0x7f60c406b8a0 msgr2=0x7f60c40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:25.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 shutdown_connections 2023-06-10T18:48:25.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:25.747+0000 7f60cb5b7700 1 -- 172.21.15.104:0/2396688155 wait complete. 2023-06-10T18:48:25.757 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:48:25.764 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:48:25.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:25.765 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:25.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[125265]: fuse finished with error 0 and tester_r 0 2023-06-10T18:48:25.799 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:48:25.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:25.800 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:48:25.861 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:48:25.862 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:25.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:26.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.017+0000 7f6c4b81e700 1 Processor -- start 2023-06-10T18:48:26.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.017+0000 7f6c4b81e700 1 -- start start 2023-06-10T18:48:26.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6c4415c500 0x7f6c440cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c440ce040 0x7f6c4406e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c44069e70 0x7f6c4406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4406a830 con 0x7f6c4415c500 2023-06-10T18:48:26.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4406dcb0 con 0x7f6c440ce040 2023-06-10T18:48:26.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c4b81e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440ce470 con 0x7f6c44069e70 2023-06-10T18:48:26.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6c4415c500 0x7f6c440cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6c4415c500 0x7f6c440cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59046/0 (socket says 172.21.15.104:59046) 2023-06-10T18:48:26.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 -- 172.21.15.104:0/1880160987 learned_addr learned my addr 172.21.15.104:0/1880160987 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:26.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c48db9700 1 --2- 172.21.15.104:0/1880160987 >> v2:172.21.15.114:3300/0 conn(0x7f6c440ce040 0x7f6c4406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c49dbb700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c44069e70 0x7f6c4406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 -- 172.21.15.104:0/1880160987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c44069e70 msgr2=0x7f6c4406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:26.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c44069e70 0x7f6c4406a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 -- 172.21.15.104:0/1880160987 >> v2:172.21.15.114:3300/0 conn(0x7f6c440ce040 msgr2=0x7f6c4406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:26.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 --2- 172.21.15.104:0/1880160987 >> v2:172.21.15.114:3300/0 conn(0x7f6c440ce040 0x7f6c4406e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.018+0000 7f6c495ba700 1 -- 172.21.15.104:0/1880160987 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4406d0a0 con 0x7f6c4415c500 2023-06-10T18:48:26.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c495ba700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c4415c500 0x7f6c440cf030 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6c440b21a0 tx=0x7f6c34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8ddfac067b7edc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c3bfff700 1 -- 172.21.15.104:0/1880160987 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34005e30 con 0x7f6c4415c500 2023-06-10T18:48:26.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c3bfff700 1 -- 172.21.15.104:0/1880160987 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c34004030 con 0x7f6c4415c500 2023-06-10T18:48:26.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c3bfff700 1 -- 172.21.15.104:0/1880160987 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340043e0 con 0x7f6c4415c500 2023-06-10T18:48:26.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 -- 172.21.15.104:0/1880160987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c4415c500 msgr2=0x7f6c440cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c4415c500 0x7f6c440cf030 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6c440b21a0 tx=0x7f6c34009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 -- 172.21.15.104:0/1880160987 shutdown_connections 2023-06-10T18:48:26.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c4415c500 0x7f6c440cf030 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/1880160987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c44069e70 0x7f6c4406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/1880160987 >> v2:172.21.15.114:3300/0 conn(0x7f6c440ce040 0x7f6c4406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 -- 172.21.15.104:0/1880160987 >> 172.21.15.104:0/1880160987 conn(0x7f6c4406b8a0 msgr2=0x7f6c440cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 -- 172.21.15.104:0/1880160987 shutdown_connections 2023-06-10T18:48:26.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.019+0000 7f6c4b81e700 1 -- 172.21.15.104:0/1880160987 wait complete. 2023-06-10T18:48:26.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.021+0000 7f6c4b81e700 1 Processor -- start 2023-06-10T18:48:26.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.021+0000 7f6c4b81e700 1 -- start start 2023-06-10T18:48:26.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c44069e70 0x7f6c441e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c4415c500 0x7f6c44144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b4db0 con 0x7f6c440ce040 2023-06-10T18:48:26.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b41a0 con 0x7f6c44069e70 2023-06-10T18:48:26.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b4480 con 0x7f6c4415c500 2023-06-10T18:48:26.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59054/0 (socket says 172.21.15.104:59054) 2023-06-10T18:48:26.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 -- 172.21.15.104:0/3449835929 learned_addr learned my addr 172.21.15.104:0/3449835929 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:26.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c495ba700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c44069e70 0x7f6c441e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c49dbb700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c4415c500 0x7f6c44144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 -- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c4415c500 msgr2=0x7f6c44144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c4415c500 0x7f6c44144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 -- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c44069e70 msgr2=0x7f6c441e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c44069e70 0x7f6c441e62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 -- 172.21.15.104:0/3449835929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c34009500 con 0x7f6c440ce040 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c48db9700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000d800 tx=0x7f6c4000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c400125f0 con 0x7f6c440ce040 2023-06-10T18:48:26.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40012750 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c440b4760 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.022+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40012b00 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.023+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c440b5240 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.023+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c440b4f60 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.024+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6c40012c60 con 0x7f6c440ce040 2023-06-10T18:48:26.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.024+0000 7f6c39ffb700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 0x7f6c2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.024+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6c40089570 con 0x7f6c440ce040 2023-06-10T18:48:26.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.024+0000 7f6c495ba700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 0x7f6c2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.025+0000 7f6c495ba700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 0x7f6c2806d190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c440b21a0 tx=0x7f6c34008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.029+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6c440b4f60 con 0x7f6c440ce040 2023-06-10T18:48:26.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.217+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c44074050 con 0x7f6c440ce040 2023-06-10T18:48:26.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.219+0000 7f6c39ffb700 1 -- 172.21.15.104:0/3449835929 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f6c400544d0 con 0x7f6c440ce040 2023-06-10T18:48:26.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:26.226 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 msgr2=0x7f6c2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 0x7f6c2806d190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c440b21a0 tx=0x7f6c34008070 comp rx=0 tx=0).stop 2023-06-10T18:48:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 msgr2=0x7f6c441445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000d800 tx=0x7f6c4000db10 comp rx=0 tx=0).stop 2023-06-10T18:48:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 shutdown_connections 2023-06-10T18:48:26.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c2806ace0 0x7f6c2806d190 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c44069e70 0x7f6c441e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c4415c500 0x7f6c44144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 --2- 172.21.15.104:0/3449835929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c440ce040 0x7f6c441445f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 >> 172.21.15.104:0/3449835929 conn(0x7f6c4406b8a0 msgr2=0x7f6c440ad970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 shutdown_connections 2023-06-10T18:48:26.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.221+0000 7f6c4b81e700 1 -- 172.21.15.104:0/3449835929 wait complete. 2023-06-10T18:48:26.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:47:57.299455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:47:58.209852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:26.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:26.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.357+0000 7fbc3e1e1700 1 Processor -- start 2023-06-10T18:48:26.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.357+0000 7fbc3e1e1700 1 -- start start 2023-06-10T18:48:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc3e1e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc38070a10 0x7fbc3806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc3e1e1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc3806c800 0x7fbc3806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc3e1e1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380cec40 con 0x7fbc38143a40 2023-06-10T18:48:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc3e1e1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380cca90 con 0x7fbc38070a10 2023-06-10T18:48:26.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc3e1e1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380ce960 con 0x7fbc3806c800 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc38070a10 0x7fbc3806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc38070a10 0x7fbc3806e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59060/0 (socket says 172.21.15.104:59060) 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 -- 172.21.15.104:0/506452598 learned_addr learned my addr 172.21.15.104:0/506452598 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc37fff700 1 --2- 172.21.15.104:0/506452598 >> v2:172.21.15.114:3300/0 conn(0x7fbc3806c800 0x7fbc3806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 -- 172.21.15.104:0/506452598 >> v2:172.21.15.114:3300/0 conn(0x7fbc3806c800 msgr2=0x7fbc3806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 --2- 172.21.15.104:0/506452598 >> v2:172.21.15.114:3300/0 conn(0x7fbc3806c800 0x7fbc3806cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 -- 172.21.15.104:0/506452598 >> v1:172.21.15.114:6789/0 conn(0x7fbc38143a40 legacy=0x7fbc38070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.358+0000 7fbc377fe700 1 -- 172.21.15.104:0/506452598 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3806a9b0 con 0x7fbc38070a10 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc377fe700 1 --2- 172.21.15.104:0/506452598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc3806e560 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbc380707c0 tx=0x7fbc2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8298078d8c08c9b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc367fc700 1 -- 172.21.15.104:0/506452598 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c00b870 con 0x7fbc38070a10 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc367fc700 1 -- 172.21.15.104:0/506452598 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c00b9d0 con 0x7fbc38070a10 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc367fc700 1 -- 172.21.15.104:0/506452598 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c00bd60 con 0x7fbc38070a10 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/506452598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 msgr2=0x7fbc3806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/506452598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc3806e560 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbc380707c0 tx=0x7fbc2c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/506452598 shutdown_connections 2023-06-10T18:48:26.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/506452598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc3806e560 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/506452598 >> v2:172.21.15.114:3300/0 conn(0x7fbc3806c800 0x7fbc3806cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/506452598 >> 172.21.15.104:0/506452598 conn(0x7fbc380ceec0 msgr2=0x7fbc380cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/506452598 shutdown_connections 2023-06-10T18:48:26.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.359+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/506452598 wait complete. 2023-06-10T18:48:26.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.360+0000 7fbc3e1e1700 1 Processor -- start 2023-06-10T18:48:26.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 -- start start 2023-06-10T18:48:26.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806c800 0x7fbc380b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc38143a40 0x7fbc380c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b2e40 con 0x7fbc38070a10 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b3090 con 0x7fbc3806c800 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.361+0000 7fbc3e1e1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b3370 con 0x7fbc38143a40 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59066/0 (socket says 172.21.15.104:59066) 2023-06-10T18:48:26.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1429005315 learned_addr learned my addr 172.21.15.104:0/1429005315 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc377fe700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806c800 0x7fbc380b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc37fff700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc38143a40 0x7fbc380c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc38143a40 msgr2=0x7fbc380c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc38143a40 0x7fbc380c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806c800 msgr2=0x7fbc380b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806c800 0x7fbc380b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1429005315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c009550 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc36ffd700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c4e0 tx=0x7fbc2800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280052d0 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28005430 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc380cd0a0 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280038c0 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.362+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc380b3930 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.363+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc38074050 con 0x7fbc38070a10 2023-06-10T18:48:26.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.363+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbc28003a20 con 0x7fbc38070a10 2023-06-10T18:48:26.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.364+0000 7fbc1ffff700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.364+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbc280889a0 con 0x7fbc38070a10 2023-06-10T18:48:26.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.364+0000 7fbc377fe700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.365+0000 7fbc377fe700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c014040 tx=0x7fbc2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.368+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbc28053910 con 0x7fbc38070a10 2023-06-10T18:48:26.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.553+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc3806b030 con 0x7fbc38070a10 2023-06-10T18:48:26.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.553+0000 7fbc1ffff700 1 -- 172.21.15.104:0/1429005315 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc28017020 con 0x7fbc38070a10 2023-06-10T18:48:26.559 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:26.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 msgr2=0x7fbc2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c014040 tx=0x7fbc2c008030 comp rx=0 tx=0).stop 2023-06-10T18:48:26.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 msgr2=0x7fbc381595a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c4e0 tx=0x7fbc2800c7f0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 shutdown_connections 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806c800 0x7fbc380b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc38143a40 0x7fbc380c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 --2- 172.21.15.104:0/1429005315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc38070a10 0x7fbc381595a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 >> 172.21.15.104:0/1429005315 conn(0x7fbc380ceec0 msgr2=0x7fbc38070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 shutdown_connections 2023-06-10T18:48:26.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.555+0000 7fbc3e1e1700 1 -- 172.21.15.104:0/1429005315 wait complete. 2023-06-10T18:48:26.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:26.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.691+0000 7f6d3bd7c700 1 Processor -- start 2023-06-10T18:48:26.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.691+0000 7f6d3bd7c700 1 -- start start 2023-06-10T18:48:26.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d3bd7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d3415a330 0x7f6d340ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d3bd7c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6d340b1780 0x7f6d3406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d3bd7c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3415a700 con 0x7f6d340cb420 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d3bd7c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3406d7b0 con 0x7f6d3415a330 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d3bd7c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d340cbe00 con 0x7f6d340b1780 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d39b18700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d3415a330 0x7f6d340ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d39317700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6d340cb420 0x7f6d340b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56902/0 (socket says 172.21.15.104:56902) 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d39b18700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d3415a330 0x7f6d340ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59072/0 (socket says 172.21.15.104:59072) 2023-06-10T18:48:26.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.692+0000 7f6d39317700 1 -- 172.21.15.104:0/2737090506 learned_addr learned my addr 172.21.15.104:0/2737090506 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3a319700 1 --2- 172.21.15.104:0/2737090506 >> v2:172.21.15.114:3300/0 conn(0x7f6d340b1780 0x7f6d3406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d39b18700 1 -- 172.21.15.104:0/2737090506 >> v2:172.21.15.114:3300/0 conn(0x7f6d340b1780 msgr2=0x7f6d3406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d39b18700 1 --2- 172.21.15.104:0/2737090506 >> v2:172.21.15.114:3300/0 conn(0x7f6d340b1780 0x7f6d3406dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d39b18700 1 -- 172.21.15.104:0/2737090506 >> v1:172.21.15.104:6789/0 conn(0x7f6d340cb420 legacy=0x7f6d340b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d39b18700 1 -- 172.21.15.104:0/2737090506 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3406ce80 con 0x7f6d3415a330 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d38b16700 1 -- 172.21.15.104:0/2737090506 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647413650 0 0) 0x7f6d3415a700 con 0x7f6d340cb420 2023-06-10T18:48:26.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d39b18700 1 --2- 172.21.15.104:0/2737090506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d3415a330 0x7f6d340ac1c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6d340670c0 tx=0x7f6d24009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d31af1f7cff4ea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d38b16700 1 -- 172.21.15.104:0/2737090506 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24005e30 con 0x7f6d3415a330 2023-06-10T18:48:26.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d38b16700 1 -- 172.21.15.104:0/2737090506 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24006030 con 0x7f6d3415a330 2023-06-10T18:48:26.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d38b16700 1 -- 172.21.15.104:0/2737090506 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240063c0 con 0x7f6d3415a330 2023-06-10T18:48:26.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/2737090506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d3415a330 msgr2=0x7f6d340ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/2737090506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d3415a330 0x7f6d340ac1c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6d340670c0 tx=0x7f6d24009b00 comp rx=0 tx=0).stop 2023-06-10T18:48:26.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/2737090506 shutdown_connections 2023-06-10T18:48:26.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/2737090506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d3415a330 0x7f6d340ac1c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/2737090506 >> v2:172.21.15.114:3300/0 conn(0x7f6d340b1780 0x7f6d3406dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.693+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/2737090506 >> 172.21.15.104:0/2737090506 conn(0x7f6d3406b8a0 msgr2=0x7f6d340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.694+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/2737090506 shutdown_connections 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.694+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/2737090506 wait complete. 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 Processor -- start 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 -- start start 2023-06-10T18:48:26.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 0x7f6d34156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d3415a330 0x7f6d340c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d341572b0 con 0x7f6d340cb420 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d341efcf0 con 0x7f6d340b1780 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.695+0000 7f6d3bd7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d341effd0 con 0x7f6d3415a330 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59080/0 (socket says 172.21.15.104:59080) 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 -- 172.21.15.104:0/779884273 learned_addr learned my addr 172.21.15.104:0/779884273 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39b18700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 0x7f6d34156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d3a319700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d3415a330 0x7f6d340c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 -- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d3415a330 msgr2=0x7f6d340c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d3415a330 0x7f6d340c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 -- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 msgr2=0x7f6d34156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 0x7f6d34156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 -- 172.21.15.104:0/779884273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24009430 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39b18700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 0x7f6d34156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d39317700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000c550 tx=0x7f6d3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30004c80 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30004e00 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d341f02b0 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30007310 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.696+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d341f0870 con 0x7f6d340cb420 2023-06-10T18:48:26.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.697+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d340c3090 con 0x7f6d340cb420 2023-06-10T18:48:26.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.698+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6d30007470 con 0x7f6d340cb420 2023-06-10T18:48:26.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.698+0000 7f6d2a7fc700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 0x7f6d1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:26.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.698+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6d30089270 con 0x7f6d340cb420 2023-06-10T18:48:26.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.698+0000 7f6d39b18700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 0x7f6d1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:26.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.699+0000 7f6d39b18700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 0x7f6d1806d190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6d24000c00 tx=0x7f6d24008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:26.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.703+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6d340c3090 con 0x7f6d340cb420 2023-06-10T18:48:26.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.889+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6d340c3090 con 0x7f6d340cb420 2023-06-10T18:48:26.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.890+0000 7f6d2a7fc700 1 -- 172.21.15.104:0/779884273 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6d340c3090 con 0x7f6d340cb420 2023-06-10T18:48:26.895 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:26.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 msgr2=0x7f6d1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 0x7f6d1806d190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6d24000c00 tx=0x7f6d24008070 comp rx=0 tx=0).stop 2023-06-10T18:48:26.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 msgr2=0x7f6d340c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000c550 tx=0x7f6d3000c910 comp rx=0 tx=0).stop 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 shutdown_connections 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d1806ace0 0x7f6d1806d190 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d340b1780 0x7f6d34156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d3415a330 0x7f6d340c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 --2- 172.21.15.104:0/779884273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d340cb420 0x7f6d340c8950 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.891+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 >> 172.21.15.104:0/779884273 conn(0x7f6d3406b8a0 msgr2=0x7f6d34069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:26.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.892+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 shutdown_connections 2023-06-10T18:48:26.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:26.892+0000 7f6d3bd7c700 1 -- 172.21.15.104:0/779884273 wait complete. 2023-06-10T18:48:26.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:48:27.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.026+0000 7fed25e8f700 1 Processor -- start 2023-06-10T18:48:27.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.026+0000 7fed25e8f700 1 -- start start 2023-06-10T18:48:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2015c500 0x7fed200cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fed20069e70 0x7fed2006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fed2006dcd0 0x7fed2006e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2006e690 con 0x7fed20069e70 2023-06-10T18:48:27.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed200ce040 con 0x7fed2006dcd0 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed25e8f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2015c8d0 con 0x7fed2015c500 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fed20069e70 0x7fed2006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fed20069e70 0x7fed2006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59086/0 (socket says 172.21.15.104:59086) 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 -- 172.21.15.104:0/4144503715 learned_addr learned my addr 172.21.15.104:0/4144503715 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1f7fe700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2015c500 0x7fed200cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1ffff700 1 --2- 172.21.15.104:0/4144503715 >> v2:172.21.15.114:3300/0 conn(0x7fed2006dcd0 0x7fed2006e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 -- 172.21.15.104:0/4144503715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2015c500 msgr2=0x7fed200cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2015c500 0x7fed200cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 -- 172.21.15.104:0/4144503715 >> v2:172.21.15.114:3300/0 conn(0x7fed2006dcd0 msgr2=0x7fed2006e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 --2- 172.21.15.104:0/4144503715 >> v2:172.21.15.114:3300/0 conn(0x7fed2006dcd0 0x7fed2006e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.027+0000 7fed1effd700 1 -- 172.21.15.104:0/4144503715 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2006da40 con 0x7fed20069e70 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed1effd700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed20069e70 0x7fed2006a2e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fed200cd010 tx=0x7fed10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6cceb9913d6d59 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed1e7fc700 1 -- 172.21.15.104:0/4144503715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000b870 con 0x7fed20069e70 2023-06-10T18:48:27.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed1e7fc700 1 -- 172.21.15.104:0/4144503715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000b9d0 con 0x7fed20069e70 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed1e7fc700 1 -- 172.21.15.104:0/4144503715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000bd60 con 0x7fed20069e70 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 -- 172.21.15.104:0/4144503715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed20069e70 msgr2=0x7fed2006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed20069e70 0x7fed2006a2e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fed200cd010 tx=0x7fed10009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 -- 172.21.15.104:0/4144503715 shutdown_connections 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 --2- 172.21.15.104:0/4144503715 >> v2:172.21.15.114:3300/0 conn(0x7fed2006dcd0 0x7fed2006e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2015c500 0x7fed200cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 --2- 172.21.15.104:0/4144503715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed20069e70 0x7fed2006a2e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 -- 172.21.15.104:0/4144503715 >> 172.21.15.104:0/4144503715 conn(0x7fed2006b8a0 msgr2=0x7fed200cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 -- 172.21.15.104:0/4144503715 shutdown_connections 2023-06-10T18:48:27.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.028+0000 7fed25e8f700 1 -- 172.21.15.104:0/4144503715 wait complete. 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.029+0000 7fed25e8f700 1 Processor -- start 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 -- start start 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed20069e70 0x7fed20154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2006dcd0 0x7fed200b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200b2980 con 0x7fed2015c500 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200c6810 con 0x7fed20069e70 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.030+0000 7fed25e8f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed200c6af0 con 0x7fed2006dcd0 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59096/0 (socket says 172.21.15.104:59096) 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 -- 172.21.15.104:0/3531045525 learned_addr learned my addr 172.21.15.104:0/3531045525 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1effd700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2006dcd0 0x7fed200b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1f7fe700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed20069e70 0x7fed20154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 -- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2006dcd0 msgr2=0x7fed200b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2006dcd0 0x7fed200b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 -- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed20069e70 msgr2=0x7fed20154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed20069e70 0x7fed20154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 -- 172.21.15.104:0/3531045525 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009550 con 0x7fed2015c500 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed1ffff700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fed18007d20 tx=0x7fed1800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1800ede0 con 0x7fed2015c500 2023-06-10T18:48:27.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18007e70 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed200c6dd0 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18014640 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.031+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed200c7390 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.032+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200c75b0 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.033+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fed180147a0 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.033+0000 7fed07fff700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 0x7fed0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.033+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fed18089bb0 con 0x7fed2015c500 2023-06-10T18:48:27.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.033+0000 7fed1f7fe700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 0x7fed0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.034+0000 7fed1f7fe700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 0x7fed0c06d190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed200b1720 tx=0x7fed14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.037+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fed18054bb0 con 0x7fed2015c500 2023-06-10T18:48:27.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.223+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed20074050 con 0x7fed2015c500 2023-06-10T18:48:27.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.223+0000 7fed07fff700 1 -- 172.21.15.104:0/3531045525 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed180581d0 con 0x7fed2015c500 2023-06-10T18:48:27.229 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:48:27.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 msgr2=0x7fed0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 0x7fed0c06d190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed200b1720 tx=0x7fed14008040 comp rx=0 tx=0).stop 2023-06-10T18:48:27.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 msgr2=0x7fed200b0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fed18007d20 tx=0x7fed1800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 shutdown_connections 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fed0c06ace0 0x7fed0c06d190 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed20069e70 0x7fed20154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed2015c500 0x7fed200b0330 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 --2- 172.21.15.104:0/3531045525 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed2006dcd0 0x7fed200b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 >> 172.21.15.104:0/3531045525 conn(0x7fed2006b8a0 msgr2=0x7fed200cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 shutdown_connections 2023-06-10T18:48:27.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.225+0000 7fed25e8f700 1 -- 172.21.15.104:0/3531045525 wait complete. 2023-06-10T18:48:27.244 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:48:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.360+0000 7fcb6b979700 1 Processor -- start 2023-06-10T18:48:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.360+0000 7fcb6b979700 1 -- start start 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb6b979700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb6415c580 0x7fcb640b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb6b979700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb6406cde0 0x7fcb640ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb6b979700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6406e250 con 0x7fcb6406e6e0 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb6b979700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6415c950 con 0x7fcb6406cde0 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb6b979700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb640b1e80 con 0x7fcb6415c580 2023-06-10T18:48:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb6406cde0 0x7fcb640ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb6406cde0 0x7fcb640ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59104/0 (socket says 172.21.15.104:59104) 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 -- 172.21.15.104:0/4214778022 learned_addr learned my addr 172.21.15.104:0/4214778022 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69715700 1 --2- 172.21.15.104:0/4214778022 >> v2:172.21.15.114:3300/0 conn(0x7fcb6415c580 0x7fcb640b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 -- 172.21.15.104:0/4214778022 >> v2:172.21.15.114:3300/0 conn(0x7fcb6415c580 msgr2=0x7fcb640b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 --2- 172.21.15.104:0/4214778022 >> v2:172.21.15.114:3300/0 conn(0x7fcb6415c580 0x7fcb640b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 -- 172.21.15.104:0/4214778022 >> v1:172.21.15.114:6789/0 conn(0x7fcb6406e6e0 legacy=0x7fcb640b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 -- 172.21.15.104:0/4214778022 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6406d920 con 0x7fcb6406cde0 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.361+0000 7fcb69f16700 1 --2- 172.21.15.104:0/4214778022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ce0d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fcb64070300 tx=0x7fcb5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8361b6c8facbcf0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb5bfff700 1 -- 172.21.15.104:0/4214778022 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00e070 con 0x7fcb6406cde0 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb5bfff700 1 -- 172.21.15.104:0/4214778022 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c003a20 con 0x7fcb6406cde0 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb5bfff700 1 -- 172.21.15.104:0/4214778022 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003dd0 con 0x7fcb6406cde0 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 -- 172.21.15.104:0/4214778022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 msgr2=0x7fcb640ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 --2- 172.21.15.104:0/4214778022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ce0d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fcb64070300 tx=0x7fcb5c00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 -- 172.21.15.104:0/4214778022 shutdown_connections 2023-06-10T18:48:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 --2- 172.21.15.104:0/4214778022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ce0d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 --2- 172.21.15.104:0/4214778022 >> v2:172.21.15.114:3300/0 conn(0x7fcb6415c580 0x7fcb640b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 -- 172.21.15.104:0/4214778022 >> 172.21.15.104:0/4214778022 conn(0x7fcb6406b8a0 msgr2=0x7fcb640cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 -- 172.21.15.104:0/4214778022 shutdown_connections 2023-06-10T18:48:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.362+0000 7fcb6b979700 1 -- 172.21.15.104:0/4214778022 wait complete. 2023-06-10T18:48:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.363+0000 7fcb6b979700 1 Processor -- start 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.363+0000 7fcb6b979700 1 -- start start 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb6406e6e0 0x7fcb640b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 0x7fcb640c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640ac1c0 con 0x7fcb6406cde0 2023-06-10T18:48:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640ac480 con 0x7fcb6415c580 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb6b979700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640ac760 con 0x7fcb6406e6e0 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb69715700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb69715700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59106/0 (socket says 172.21.15.104:59106) 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb69715700 1 -- 172.21.15.104:0/2634084079 learned_addr learned my addr 172.21.15.104:0/2634084079 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb69f16700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 0x7fcb640c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.364+0000 7fcb68f14700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb6406e6e0 0x7fcb640b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 -- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb6406e6e0 msgr2=0x7fcb640b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb6406e6e0 0x7fcb640b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 -- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 msgr2=0x7fcb640c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 0x7fcb640c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 -- 172.21.15.104:0/2634084079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c009b30 con 0x7fcb6406cde0 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69f16700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 0x7fcb640c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb69715700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c7f0 tx=0x7fcb5400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54005c20 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54005da0 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb640aca40 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54011840 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.365+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb640ad000 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.366+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb640ad220 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.366+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb540119a0 con 0x7fcb6406cde0 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.367+0000 7fcb59ffb700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 0x7fcb4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.367+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcb540892d0 con 0x7fcb6406cde0 2023-06-10T18:48:27.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.367+0000 7fcb68f14700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 0x7fcb4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.367+0000 7fcb68f14700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 0x7fcb4806d190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb640c2f10 tx=0x7fcb60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.371+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb540542d0 con 0x7fcb6406cde0 2023-06-10T18:48:27.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.556+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb64074050 con 0x7fcb6406cde0 2023-06-10T18:48:27.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.556+0000 7fcb59ffb700 1 -- 172.21.15.104:0/2634084079 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb540578f0 con 0x7fcb6406cde0 2023-06-10T18:48:27.562 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:48:27.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 msgr2=0x7fcb4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 0x7fcb4806d190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb640c2f10 tx=0x7fcb60009040 comp rx=0 tx=0).stop 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 msgr2=0x7fcb640ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c7f0 tx=0x7fcb5400cbb0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 shutdown_connections 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb4806ace0 0x7fcb4806d190 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6415c580 0x7fcb640c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb6406cde0 0x7fcb640ad520 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 --2- 172.21.15.104:0/2634084079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb6406e6e0 0x7fcb640b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 >> 172.21.15.104:0/2634084079 conn(0x7fcb6406b8a0 msgr2=0x7fcb6415a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 shutdown_connections 2023-06-10T18:48:27.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.559+0000 7fcb6b979700 1 -- 172.21.15.104:0/2634084079 wait complete. 2023-06-10T18:48:27.577 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:48:27.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.693+0000 7f9dde6a6700 1 Processor -- start 2023-06-10T18:48:27.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.693+0000 7f9dde6a6700 1 -- start start 2023-06-10T18:48:27.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dde6a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd80701a0 0x7f9dd80705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dde6a6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd8070b00 con 0x7f9dd815c7a0 2023-06-10T18:48:27.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dde6a6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd815cca0 con 0x7f9dd81567f0 2023-06-10T18:48:27.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dde6a6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd815cf80 con 0x7f9dd80701a0 2023-06-10T18:48:27.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9dd815c7a0 0x7f9dd80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56944/0 (socket says 172.21.15.104:56944) 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd77fe700 1 -- 172.21.15.104:0/3110495513 learned_addr learned my addr 172.21.15.104:0/3110495513 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579825632 0 0) 0x7f9dd8070b00 con 0x7f9dd815c7a0 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc00035b0 con 0x7f9dd815c7a0 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9ddcc43700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd80701a0 0x7f9dd80705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628166699 0 0) 0x7f9dc00035b0 con 0x7f9dd815c7a0 2023-06-10T18:48:27.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd80701a0 msgr2=0x7f9dd80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd80701a0 0x7f9dd80705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 >> v1:172.21.15.114:6789/0 conn(0x7f9dd81567f0 legacy=0x7f9dd80cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd815d260 con 0x7f9dd815c7a0 2023-06-10T18:48:27.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9dc8001670 con 0x7f9dd815c7a0 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 >> v1:172.21.15.104:6789/0 conn(0x7f9dd815c7a0 legacy=0x7f9dd80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc0003a30 0x7f9dc0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.694+0000 7f9dd6ffd700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dc0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc0005250 0x7f9dc000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd815cf80 con 0x7f9dc0004850 2023-06-10T18:48:27.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000c140 con 0x7f9dc0005250 2023-06-10T18:48:27.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000c300 con 0x7f9dc0003a30 2023-06-10T18:48:27.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9ddcc43700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd80701a0 0x7f9dd80705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:27.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dc0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd7fff700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc0003a30 0x7f9dc0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9ddcc43700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc0005250 0x7f9dc000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 -- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc0003a30 msgr2=0x7f9dc0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc0003a30 0x7f9dc0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 -- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc0005250 msgr2=0x7f9dc000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc0005250 0x7f9dc000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 -- 172.21.15.104:0/3110495513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc00035b0 con 0x7f9dc0004850 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dc0004cc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c210 tx=0x7f9dc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530ee1349f8c5278 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8007c10 con 0x7f9dc0004850 2023-06-10T18:48:27.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc800cdd0 con 0x7f9dc0004850 2023-06-10T18:48:27.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.695+0000 7f9dd6ffd700 1 -- 172.21.15.104:0/3110495513 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8005ba0 con 0x7f9dc0004850 2023-06-10T18:48:27.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3110495513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 msgr2=0x7f9dc0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3110495513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dc0004cc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c210 tx=0x7f9dc0005880 comp rx=0 tx=0).stop 2023-06-10T18:48:27.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9ddcc43700 1 -- 172.21.15.104:0/3110495513 reap_dead start 2023-06-10T18:48:27.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3110495513 shutdown_connections 2023-06-10T18:48:27.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3110495513 >> 172.21.15.104:0/3110495513 conn(0x7f9dd806e370 msgr2=0x7f9dd806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3110495513 shutdown_connections 2023-06-10T18:48:27.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.696+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3110495513 wait complete. 2023-06-10T18:48:27.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.697+0000 7f9dde6a6700 1 Processor -- start 2023-06-10T18:48:27.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.697+0000 7f9dde6a6700 1 -- start start 2023-06-10T18:48:27.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 0x7f9dd80c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd806bfd0 0x7f9dd806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd806c9b0 con 0x7f9dc0004850 2023-06-10T18:48:27.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80c9db0 con 0x7f9dd806bfd0 2023-06-10T18:48:27.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dde6a6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80ca090 con 0x7f9dd81f14c0 2023-06-10T18:48:27.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dd7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dd7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59126/0 (socket says 172.21.15.104:59126) 2023-06-10T18:48:27.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.698+0000 7f9dd7fff700 1 -- 172.21.15.104:0/3812555543 learned_addr learned my addr 172.21.15.104:0/3812555543 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:27.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 0x7f9dd80c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9ddcc43700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd806bfd0 0x7f9dd806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 -- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 msgr2=0x7f9dd80c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 0x7f9dd80c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 -- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd806bfd0 msgr2=0x7f9dd806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd806bfd0 0x7f9dd806c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 -- 172.21.15.104:0/3812555543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd80ca370 con 0x7f9dc0004850 2023-06-10T18:48:27.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 0x7f9dd80c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:27.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd7fff700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e510 tx=0x7f9dcc00e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0052e0 con 0x7f9dc0004850 2023-06-10T18:48:27.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc005440 con 0x7f9dc0004850 2023-06-10T18:48:27.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd80ca650 con 0x7f9dc0004850 2023-06-10T18:48:27.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0038a0 con 0x7f9dc0004850 2023-06-10T18:48:27.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.699+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd80c9aa0 con 0x7f9dc0004850 2023-06-10T18:48:27.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.700+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd81445f0 con 0x7f9dc0004850 2023-06-10T18:48:27.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.700+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9dcc003a00 con 0x7f9dc0004850 2023-06-10T18:48:27.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.701+0000 7f9dd4ff9700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 0x7f9dc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:27.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.701+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9dcc088c80 con 0x7f9dc0004850 2023-06-10T18:48:27.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.701+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 0x7f9dc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:27.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.701+0000 7f9dd77fe700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 0x7f9dc406d190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c4030 tx=0x7f9dc8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:27.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.705+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9dcc053c00 con 0x7f9dc0004850 2023-06-10T18:48:27.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.906+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9dd8074050 con 0x7f9dc0004850 2023-06-10T18:48:27.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.907+0000 7f9dd4ff9700 1 -- 172.21.15.104:0/3812555543 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9dcc019080 con 0x7f9dc0004850 2023-06-10T18:48:27.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 msgr2=0x7f9dc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 0x7f9dc406d190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c4030 tx=0x7f9dc8007000 comp rx=0 tx=0).stop 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 msgr2=0x7f9dd8147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e510 tx=0x7f9dcc00e8d0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 shutdown_connections 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9dc406ace0 0x7f9dc406d190 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd806bfd0 0x7f9dd806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9dc0004850 0x7f9dd8147170 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 --2- 172.21.15.104:0/3812555543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dd81f14c0 0x7f9dd80c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 >> 172.21.15.104:0/3812555543 conn(0x7f9dd806e370 msgr2=0x7f9dd806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 shutdown_connections 2023-06-10T18:48:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:27.909+0000 7f9dde6a6700 1 -- 172.21.15.104:0/3812555543 wait complete. 2023-06-10T18:48:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:48:27.926 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:48:27.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:27.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:27.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[109373]: fuse finished with error 0 and tester_r 0 2023-06-10T18:48:27.976 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:48:27.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:27.977 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:48:28.107 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:28.108 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:28.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:48:28.125 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:48:28.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:28.126 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:48:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:48:28.182 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:48:28.182 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:48:28.183 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:28.202 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:48:28.202 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:48:28.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:28.203 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:48:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:48:28.261 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:48:28.261 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:28.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.375+0000 7fb17bfaa700 1 Processor -- start 2023-06-10T18:48:28.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.375+0000 7fb17bfaa700 1 -- start start 2023-06-10T18:48:28.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb17bfaa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb17408a3d0 0x7fb174167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb17bfaa700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1740cadd0 con 0x7fb17406b1f0 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb17bfaa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1740cb0b0 con 0x7fb1740ce350 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb17bfaa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1740cfeb0 con 0x7fb17408a3d0 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb179d46700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb17406b1f0 0x7fb1740cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56962/0 (socket says 172.21.15.104:56962) 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb179d46700 1 -- 172.21.15.104:0/933833756 learned_addr learned my addr 172.21.15.104:0/933833756 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb17a547700 1 --2- 172.21.15.104:0/933833756 >> v2:172.21.15.114:3300/0 conn(0x7fb17408a3d0 0x7fb174167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728055975 0 0) 0x7fb1740cadd0 con 0x7fb17406b1f0 2023-06-10T18:48:28.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1600035b0 con 0x7fb17406b1f0 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28048766 0 0) 0x7fb1600035b0 con 0x7fb17406b1f0 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 >> v2:172.21.15.114:3300/0 conn(0x7fb17408a3d0 msgr2=0x7fb174167940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 --2- 172.21.15.104:0/933833756 >> v2:172.21.15.114:3300/0 conn(0x7fb17408a3d0 0x7fb174167940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 >> v1:172.21.15.114:6789/0 conn(0x7fb1740ce350 legacy=0x7fb1740cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740cdeb0 con 0x7fb17406b1f0 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb170001670 con 0x7fb17406b1f0 2023-06-10T18:48:28.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 >> v1:172.21.15.104:6789/0 conn(0x7fb17406b1f0 legacy=0x7fb1740cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb160003a00 0x7fb160003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb160004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.376+0000 7fb178d44700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb160005200 0x7fb16000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740cfeb0 con 0x7fb160004850 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16000c110 con 0x7fb160003a00 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16000c2d0 con 0x7fb160005200 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb160004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179d46700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb160003a00 0x7fb160003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb17a547700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb160005200 0x7fb16000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 -- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb160005200 msgr2=0x7fb16000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb160005200 0x7fb16000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 -- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb160003a00 msgr2=0x7fb160003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb160003a00 0x7fb160003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 -- 172.21.15.104:0/933833756 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1600035b0 con 0x7fb160004850 2023-06-10T18:48:28.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb17a547700 1 -- 172.21.15.104:0/933833756 reap_dead start 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb179545700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb160004cc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb174067a90 tx=0x7fb160005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a123a66dd27f32b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1640052d0 con 0x7fb160004850 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb164005470 con 0x7fb160004850 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.377+0000 7fb178d44700 1 -- 172.21.15.104:0/933833756 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1640038c0 con 0x7fb160004850 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 -- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 msgr2=0x7fb160004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb160004cc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb174067a90 tx=0x7fb160005880 comp rx=0 tx=0).stop 2023-06-10T18:48:28.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 -- 172.21.15.104:0/933833756 shutdown_connections 2023-06-10T18:48:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/933833756 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb160004cc0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 -- 172.21.15.104:0/933833756 >> 172.21.15.104:0/933833756 conn(0x7fb174089fd0 msgr2=0x7fb1740d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 -- 172.21.15.104:0/933833756 shutdown_connections 2023-06-10T18:48:28.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.378+0000 7fb17bfaa700 1 -- 172.21.15.104:0/933833756 wait complete. 2023-06-10T18:48:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.379+0000 7fb17bfaa700 1 Processor -- start 2023-06-10T18:48:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.379+0000 7fb17bfaa700 1 -- start start 2023-06-10T18:48:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb17409b190 0x7fb174151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb17409b600 0x7fb174151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174155600 con 0x7fb160004850 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17409d040 con 0x7fb17409b190 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1741f62a0 con 0x7fb17409b600 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59146/0 (socket says 172.21.15.104:59146) 2023-06-10T18:48:28.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 -- 172.21.15.104:0/1647806091 learned_addr learned my addr 172.21.15.104:0/1647806091 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179545700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb17409b190 0x7fb174151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17a547700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb17409b600 0x7fb174151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 -- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb17409b600 msgr2=0x7fb174151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb17409b600 0x7fb174151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 -- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb17409b190 msgr2=0x7fb174151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb17409b190 0x7fb174151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 -- 172.21.15.104:0/1647806091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1741521a0 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb179d46700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb170001e60 tx=0x7fb1700075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17000f6c0 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17000f820 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1741f5720 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17000fbd0 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.380+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1741f5ce0 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.381+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb174074050 con 0x7fb160004850 2023-06-10T18:48:28.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.382+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb17000fd30 con 0x7fb160004850 2023-06-10T18:48:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.382+0000 7fb16a7fc700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 0x7fb15806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.382+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb170089610 con 0x7fb160004850 2023-06-10T18:48:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.383+0000 7fb179545700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 0x7fb15806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.383+0000 7fb179545700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 0x7fb15806d190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb174067a90 tx=0x7fb164009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.387+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb170054630 con 0x7fb160004850 2023-06-10T18:48:28.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.616+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17409d180 con 0x7fb160004850 2023-06-10T18:48:28.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.617+0000 7fb16a7fc700 1 -- 172.21.15.104:0/1647806091 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7fb170057c50 con 0x7fb160004850 2023-06-10T18:48:28.623 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:28.623 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:28.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 msgr2=0x7fb15806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 0x7fb15806d190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb174067a90 tx=0x7fb164009030 comp rx=0 tx=0).stop 2023-06-10T18:48:28.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 msgr2=0x7fb17409a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb170001e60 tx=0x7fb1700075f0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 shutdown_connections 2023-06-10T18:48:28.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb17409b600 0x7fb174151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb160004850 0x7fb17409a5a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb15806ace0 0x7fb15806d190 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 --2- 172.21.15.104:0/1647806091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb17409b190 0x7fb174151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 >> 172.21.15.104:0/1647806091 conn(0x7fb174089fd0 msgr2=0x7fb17415fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:28.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 shutdown_connections 2023-06-10T18:48:28.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.619+0000 7fb17bfaa700 1 -- 172.21.15.104:0/1647806091 wait complete. 2023-06-10T18:48:28.630 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:28.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.752+0000 7f53ae73e700 1 Processor -- start 2023-06-10T18:48:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.752+0000 7f53ae73e700 1 -- start start 2023-06-10T18:48:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53ae73e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53a815c730 0x7f53a806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53ae73e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a815cb00 con 0x7f53a80cb0c0 2023-06-10T18:48:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53ae73e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a806ecd0 con 0x7f53a80cc000 2023-06-10T18:48:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53ae73e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a80aed80 con 0x7f53a815c730 2023-06-10T18:48:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53accdb700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f53a80cb0c0 0x7f53a806d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56986/0 (socket says 172.21.15.104:56986) 2023-06-10T18:48:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53accdb700 1 -- 172.21.15.104:0/3665381289 learned_addr learned my addr 172.21.15.104:0/3665381289 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3665381289 >> v2:172.21.15.114:3300/0 conn(0x7f53a815c730 0x7f53a806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881188535 0 0) 0x7f53a0001490 con 0x7f53a80cb0c0 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53900035b0 con 0x7f53a80cb0c0 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1859371936 0 0) 0x7f53900035b0 con 0x7f53a80cb0c0 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 >> v2:172.21.15.114:3300/0 conn(0x7f53a815c730 msgr2=0x7f53a806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 --2- 172.21.15.104:0/3665381289 >> v2:172.21.15.114:3300/0 conn(0x7f53a815c730 0x7f53a806e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 >> v1:172.21.15.114:6789/0 conn(0x7f53a80cc000 legacy=0x7f53a806cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a806db90 con 0x7f53a80cb0c0 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f53a0001b80 con 0x7f53a80cb0c0 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 >> v1:172.21.15.104:6789/0 conn(0x7f53a80cb0c0 legacy=0x7f53a806d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 0x7f5390003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390004850 0x7f5390004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390005200 0x7f539000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80aed80 con 0x7f5390003a00 2023-06-10T18:48:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c110 con 0x7f5390005200 2023-06-10T18:48:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000c2d0 con 0x7f5390004850 2023-06-10T18:48:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 0x7f5390003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53accdb700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390005200 0x7f539000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.753+0000 7f53a77fe700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390004850 0x7f5390004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 -- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390004850 msgr2=0x7f5390004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390004850 0x7f5390004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 -- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390005200 msgr2=0x7f539000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390005200 0x7f539000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53accdb700 1 -- 172.21.15.104:0/3665381289 reap_dead start 2023-06-10T18:48:28.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 -- 172.21.15.104:0/3665381289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900035b0 con 0x7f5390003a00 2023-06-10T18:48:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 0x7f5390003e10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5398009b40 tx=0x7f5390005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b45fa67315fc33c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398005a20 con 0x7f5390003a00 2023-06-10T18:48:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398005b80 con 0x7f5390003a00 2023-06-10T18:48:28.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53a6ffd700 1 -- 172.21.15.104:0/3665381289 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539800e6c0 con 0x7f5390003a00 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53ae73e700 1 -- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 msgr2=0x7f5390003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 0x7f5390003e10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5398009b40 tx=0x7f5390005880 comp rx=0 tx=0).stop 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53ae73e700 1 -- 172.21.15.104:0/3665381289 shutdown_connections 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3665381289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5390003a00 0x7f5390003e10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.754+0000 7f53ae73e700 1 -- 172.21.15.104:0/3665381289 >> 172.21.15.104:0/3665381289 conn(0x7f53a806b8a0 msgr2=0x7f53a80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:28.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.755+0000 7f53ae73e700 1 -- 172.21.15.104:0/3665381289 shutdown_connections 2023-06-10T18:48:28.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.755+0000 7f53ae73e700 1 -- 172.21.15.104:0/3665381289 wait complete. 2023-06-10T18:48:28.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.756+0000 7f53ae73e700 1 Processor -- start 2023-06-10T18:48:28.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.756+0000 7f53ae73e700 1 -- start start 2023-06-10T18:48:28.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 0x7f53a806a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a80c1a70 0x7f53a8153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80c1f30 con 0x7f53a80c51c0 2023-06-10T18:48:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80c2210 con 0x7f53a80c1a70 2023-06-10T18:48:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53ae73e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80c24f0 con 0x7f5390003a00 2023-06-10T18:48:28.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59170/0 (socket says 172.21.15.104:59170) 2023-06-10T18:48:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 -- 172.21.15.104:0/3383736806 learned_addr learned my addr 172.21.15.104:0/3383736806 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 0x7f53a806a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53a77fe700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a80c1a70 0x7f53a8153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 -- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 msgr2=0x7f53a806a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 0x7f53a806a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 -- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a80c1a70 msgr2=0x7f53a8153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a80c1a70 0x7f53a8153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53accdb700 1 -- 172.21.15.104:0/3383736806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a80c27d0 con 0x7f53a80c51c0 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.757+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 0x7f53a806a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:28.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53accdb700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f53a00016f0 tx=0x7f53a0007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00133f0 con 0x7f53a80c51c0 2023-06-10T18:48:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0013550 con 0x7f53a80c51c0 2023-06-10T18:48:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a80b0290 con 0x7f53a80c51c0 2023-06-10T18:48:28.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0013900 con 0x7f53a80c51c0 2023-06-10T18:48:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.758+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a80c5b80 con 0x7f53a80c51c0 2023-06-10T18:48:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.759+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a80b04d0 con 0x7f53a80c51c0 2023-06-10T18:48:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.759+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f53a0013a60 con 0x7f53a80c51c0 2023-06-10T18:48:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.759+0000 7f53a4ff9700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 0x7f539406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.760+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f53a0089d80 con 0x7f53a80c51c0 2023-06-10T18:48:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.760+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 0x7f539406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.760+0000 7f53a7fff700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 0x7f539406d190 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f539800a010 tx=0x7f5398008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:28.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.764+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f53a80b04d0 con 0x7f53a80c51c0 2023-06-10T18:48:28.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.952+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a815cb00 con 0x7f53a80c51c0 2023-06-10T18:48:28.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.953+0000 7f53a4ff9700 1 -- 172.21.15.104:0/3383736806 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f53a815cb00 con 0x7f53a80c51c0 2023-06-10T18:48:28.959 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:28.959 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:28.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 msgr2=0x7f539406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 0x7f539406d190 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f539800a010 tx=0x7f5398008030 comp rx=0 tx=0).stop 2023-06-10T18:48:28.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 msgr2=0x7f53a80c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:28.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f53a00016f0 tx=0x7f53a0007340 comp rx=0 tx=0).stop 2023-06-10T18:48:28.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 shutdown_connections 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a80c51c0 0x7f53a80c5640 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5390003a00 0x7f53a806a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f539406ace0 0x7f539406d190 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 --2- 172.21.15.104:0/3383736806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a80c1a70 0x7f53a8153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 >> 172.21.15.104:0/3383736806 conn(0x7f53a806b8a0 msgr2=0x7f53a806d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:28.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 shutdown_connections 2023-06-10T18:48:28.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:28.955+0000 7f53ae73e700 1 -- 172.21.15.104:0/3383736806 wait complete. 2023-06-10T18:48:28.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:47:57.299455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:47:58.209852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:28.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.091+0000 7ff4622b7700 1 Processor -- start 2023-06-10T18:48:29.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.091+0000 7ff4622b7700 1 -- start start 2023-06-10T18:48:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff4622b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff45c0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff4622b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff45c0d0830 0x7ff45c0b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff4622b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0ad3c0 con 0x7ff45c157b30 2023-06-10T18:48:29.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff4622b7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c157f00 con 0x7ff45c0d0830 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff4622b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b0170 con 0x7ff45c0cea50 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff45bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff45c157b30 0x7ff45c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57002/0 (socket says 172.21.15.104:57002) 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.092+0000 7ff45bfff700 1 -- 172.21.15.104:0/706332803 learned_addr learned my addr 172.21.15.104:0/706332803 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445180493 0 0) 0x7ff45c0ad3c0 con 0x7ff45c157b30 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff460854700 1 --2- 172.21.15.104:0/706332803 >> v2:172.21.15.114:3300/0 conn(0x7ff45c0d0830 0x7ff45c0b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4440035b0 con 0x7ff45c157b30 2023-06-10T18:48:29.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45b7fe700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff45c0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009034970 0 0) 0x7ff4440035b0 con 0x7ff45c157b30 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 msgr2=0x7ff45c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff45c0d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 >> v2:172.21.15.114:3300/0 conn(0x7ff45c0d0830 msgr2=0x7ff45c0b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 --2- 172.21.15.104:0/706332803 >> v2:172.21.15.114:3300/0 conn(0x7ff45c0d0830 0x7ff45c0b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c0b0410 con 0x7ff45c157b30 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff44c001670 con 0x7ff45c157b30 2023-06-10T18:48:29.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 >> v1:172.21.15.104:6789/0 conn(0x7ff45c157b30 legacy=0x7ff45c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff444003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff444004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff444004d90 0x7ff444005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c157f00 con 0x7ff444003a00 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b0170 con 0x7ff444004d90 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44400ba10 con 0x7ff45c0cea50 2023-06-10T18:48:29.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff444003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45b7fe700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff444004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff460854700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff444004d90 0x7ff444005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 -- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 msgr2=0x7ff444004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c0cea50 0x7ff444004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 -- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff444004d90 msgr2=0x7ff444005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff444004d90 0x7ff444005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 -- 172.21.15.104:0/706332803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4440035b0 con 0x7ff444003a00 2023-06-10T18:48:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.093+0000 7ff45bfff700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff444003e10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c2f0 tx=0x7ff444005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c63883b037ded9a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c007c40 con 0x7ff444003a00 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c00ce30 con 0x7ff444003a00 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff45affd700 1 -- 172.21.15.104:0/706332803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c005ba0 con 0x7ff444003a00 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 -- 172.21.15.104:0/706332803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 msgr2=0x7ff444003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 --2- 172.21.15.104:0/706332803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff444003e10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c2f0 tx=0x7ff444005880 comp rx=0 tx=0).stop 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff460854700 1 -- 172.21.15.104:0/706332803 reap_dead start 2023-06-10T18:48:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 -- 172.21.15.104:0/706332803 shutdown_connections 2023-06-10T18:48:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 -- 172.21.15.104:0/706332803 >> 172.21.15.104:0/706332803 conn(0x7ff45c089fd0 msgr2=0x7ff45c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 -- 172.21.15.104:0/706332803 shutdown_connections 2023-06-10T18:48:29.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.094+0000 7ff4622b7700 1 -- 172.21.15.104:0/706332803 wait complete. 2023-06-10T18:48:29.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.096+0000 7ff4622b7700 1 Processor -- start 2023-06-10T18:48:29.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.096+0000 7ff4622b7700 1 -- start start 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c14bb40 0x7ff45c0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff45c0ae3d0 0x7ff45c0ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0aed90 con 0x7ff444003a00 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b50a0 con 0x7ff45c0ae3d0 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b4000 con 0x7ff45c14bb40 2023-06-10T18:48:29.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59186/0 (socket says 172.21.15.104:59186) 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 -- 172.21.15.104:0/4125144224 learned_addr learned my addr 172.21.15.104:0/4125144224 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff460854700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff45c0ae3d0 0x7ff45c0ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45b7fe700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c14bb40 0x7ff45c0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 -- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c14bb40 msgr2=0x7ff45c0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c14bb40 0x7ff45c0b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 -- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff45c0ae3d0 msgr2=0x7ff45c0ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff45c0ae3d0 0x7ff45c0ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 -- 172.21.15.104:0/4125144224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c0b42e0 con 0x7ff444003a00 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff45bfff700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff444004700 tx=0x7ff45c15a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0070d0 con 0x7ff444003a00 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c007230 con 0x7ff444003a00 2023-06-10T18:48:29.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c0b45c0 con 0x7ff444003a00 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.097+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0075e0 con 0x7ff444003a00 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.098+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c0b4b80 con 0x7ff444003a00 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.098+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c074050 con 0x7ff444003a00 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.099+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff44c01e430 con 0x7ff444003a00 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.099+0000 7ff458ff9700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 0x7ff44806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.099+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff44c08a0b0 con 0x7ff444003a00 2023-06-10T18:48:29.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.100+0000 7ff45b7fe700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 0x7ff44806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.100+0000 7ff45b7fe700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 0x7ff44806d190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0cee40 tx=0x7ff450008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.104+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff44c0550b0 con 0x7ff444003a00 2023-06-10T18:48:29.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.290+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff45c0afad0 con 0x7ff444003a00 2023-06-10T18:48:29.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.290+0000 7ff458ff9700 1 -- 172.21.15.104:0/4125144224 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff44c007da0 con 0x7ff444003a00 2023-06-10T18:48:29.296 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 msgr2=0x7ff44806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 0x7ff44806d190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0cee40 tx=0x7ff450008040 comp rx=0 tx=0).stop 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 msgr2=0x7ff45c14b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff444004700 tx=0x7ff45c15a4e0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 shutdown_connections 2023-06-10T18:48:29.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff44806ace0 0x7ff44806d190 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff45c0ae3d0 0x7ff45c0ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff444003a00 0x7ff45c14b390 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 --2- 172.21.15.104:0/4125144224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff45c14bb40 0x7ff45c0b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 >> 172.21.15.104:0/4125144224 conn(0x7ff45c089fd0 msgr2=0x7ff45c14c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 shutdown_connections 2023-06-10T18:48:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.292+0000 7ff4622b7700 1 -- 172.21.15.104:0/4125144224 wait complete. 2023-06-10T18:48:29.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:29.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.425+0000 7f10a8f69700 1 Processor -- start 2023-06-10T18:48:29.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.425+0000 7f10a8f69700 1 -- start start 2023-06-10T18:48:29.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10a415c580 0x7f10a40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 0x7f10a40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10a406cde0 0x7f10a4069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a406e250 con 0x7f10a406cde0 2023-06-10T18:48:29.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a415c950 con 0x7f10a415c580 2023-06-10T18:48:29.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a8f69700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40b1e80 con 0x7f10a406e6e0 2023-06-10T18:48:29.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10a406cde0 0x7f10a4069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10a406cde0 0x7f10a4069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59194/0 (socket says 172.21.15.104:59194) 2023-06-10T18:48:29.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a2d9d700 1 -- 172.21.15.104:0/188244954 learned_addr learned my addr 172.21.15.104:0/188244954 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:29.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 0x7f10a40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.426+0000 7f10a259c700 1 --2- 172.21.15.104:0/188244954 >> v2:172.21.15.114:3300/0 conn(0x7f10a415c580 0x7f10a40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 -- 172.21.15.104:0/188244954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 msgr2=0x7f10a40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 0x7f10a40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 -- 172.21.15.104:0/188244954 >> v2:172.21.15.114:3300/0 conn(0x7f10a415c580 msgr2=0x7f10a40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/188244954 >> v2:172.21.15.114:3300/0 conn(0x7f10a415c580 0x7f10a40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 -- 172.21.15.104:0/188244954 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a406d920 con 0x7f10a406cde0 2023-06-10T18:48:29.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 0x7f10a40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:29.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406cde0 0x7f10a4069e70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f10a4070300 tx=0x7f109c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33638948ca06c21d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a159a700 1 -- 172.21.15.104:0/188244954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c00e070 con 0x7f10a406cde0 2023-06-10T18:48:29.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a159a700 1 -- 172.21.15.104:0/188244954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c00b9c0 con 0x7f10a406cde0 2023-06-10T18:48:29.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a159a700 1 -- 172.21.15.104:0/188244954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c00bd70 con 0x7f10a406cde0 2023-06-10T18:48:29.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 -- 172.21.15.104:0/188244954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406cde0 msgr2=0x7f10a4069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406cde0 0x7f10a4069e70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f10a4070300 tx=0x7f109c00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:29.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 -- 172.21.15.104:0/188244954 shutdown_connections 2023-06-10T18:48:29.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406cde0 0x7f10a4069e70 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 --2- 172.21.15.104:0/188244954 >> v2:172.21.15.114:3300/0 conn(0x7f10a415c580 0x7f10a40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 --2- 172.21.15.104:0/188244954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406e6e0 0x7f10a40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 -- 172.21.15.104:0/188244954 >> 172.21.15.104:0/188244954 conn(0x7f10a406b8a0 msgr2=0x7f10a40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.427+0000 7f10a8f69700 1 -- 172.21.15.104:0/188244954 shutdown_connections 2023-06-10T18:48:29.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.428+0000 7f10a8f69700 1 -- 172.21.15.104:0/188244954 wait complete. 2023-06-10T18:48:29.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.429+0000 7f10a8f69700 1 Processor -- start 2023-06-10T18:48:29.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.429+0000 7f10a8f69700 1 -- start start 2023-06-10T18:48:29.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406cde0 0x7f10a4157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a415c580 0x7f10a40ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40c36e0 con 0x7f10a406e6e0 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a415eb80 con 0x7f10a415c580 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a415ee60 con 0x7f10a406cde0 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59200/0 (socket says 172.21.15.104:59200) 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 -- 172.21.15.104:0/3082765794 learned_addr learned my addr 172.21.15.104:0/3082765794 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a259c700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406cde0 0x7f10a4157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a415c580 0x7f10a40ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 -- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406cde0 msgr2=0x7f10a4157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406cde0 0x7f10a4157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 -- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a415c580 msgr2=0x7f10a40ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a415c580 0x7f10a40ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 -- 172.21.15.104:0/3082765794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c009b30 con 0x7f10a406e6e0 2023-06-10T18:48:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f109400c740 tx=0x7f109400ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094011800 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094011960 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a40c3090 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094011cd0 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.430+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a4157c90 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.431+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a40c3850 con 0x7f10a406e6e0 2023-06-10T18:48:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.432+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1094021350 con 0x7f10a406e6e0 2023-06-10T18:48:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.432+0000 7f1092ffd700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.432+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1094089420 con 0x7f10a406e6e0 2023-06-10T18:48:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.432+0000 7f10a259c700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.433+0000 7f10a259c700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f10a4158270 tx=0x7f1098008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.436+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1094057da0 con 0x7f10a406e6e0 2023-06-10T18:48:29.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.621+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10a4074050 con 0x7f10a406e6e0 2023-06-10T18:48:29.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.622+0000 7f1092ffd700 1 -- 172.21.15.104:0/3082765794 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1094057930 con 0x7f10a406e6e0 2023-06-10T18:48:29.628 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:29.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 msgr2=0x7f108406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f10a4158270 tx=0x7f1098008040 comp rx=0 tx=0).stop 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 msgr2=0x7f10a415d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f109400c740 tx=0x7f109400ca50 comp rx=0 tx=0).stop 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 shutdown_connections 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a415c580 0x7f10a40ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406cde0 0x7f10a4157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 --2- 172.21.15.104:0/3082765794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a406e6e0 0x7f10a415d4a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 >> 172.21.15.104:0/3082765794 conn(0x7f10a406b8a0 msgr2=0x7f10a40ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 shutdown_connections 2023-06-10T18:48:29.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.624+0000 7f10a8f69700 1 -- 172.21.15.104:0/3082765794 wait complete. 2023-06-10T18:48:29.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:48:29.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.757+0000 7f2f274da700 1 Processor -- start 2023-06-10T18:48:29.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.758+0000 7f2f274da700 1 -- start start 2023-06-10T18:48:29.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.758+0000 7f2f274da700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2f2015c580 0x7f2f200b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.758+0000 7f2f274da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 0x7f2f200b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.758+0000 7f2f274da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f20069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.758+0000 7f2f274da700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2006e250 con 0x7f2f2015c580 2023-06-10T18:48:29.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f274da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2015c950 con 0x7f2f2006e6e0 2023-06-10T18:48:29.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f274da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200b1e80 con 0x7f2f2006cde0 2023-06-10T18:48:29.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2f2015c580 0x7f2f200b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2f2015c580 0x7f2f200b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59206/0 (socket says 172.21.15.104:59206) 2023-06-10T18:48:29.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 -- 172.21.15.104:0/1021333310 learned_addr learned my addr 172.21.15.104:0/1021333310 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:29.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 -- 172.21.15.104:0/1021333310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 msgr2=0x7f2f20069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f24a75700 1 --2- 172.21.15.104:0/1021333310 >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 0x7f2f200b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25a77700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f20069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f20069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 -- 172.21.15.104:0/1021333310 >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 msgr2=0x7f2f200b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 --2- 172.21.15.104:0/1021333310 >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 0x7f2f200b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 -- 172.21.15.104:0/1021333310 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2006d920 con 0x7f2f2015c580 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f24a75700 1 --2- 172.21.15.104:0/1021333310 >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 0x7f2f200b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f25276700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2015c580 0x7f2f200b1070 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2f2006af30 tx=0x7f2f10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d520ac7d53dd251 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f17fff700 1 -- 172.21.15.104:0/1021333310 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10005e30 con 0x7f2f2015c580 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f17fff700 1 -- 172.21.15.104:0/1021333310 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f10004030 con 0x7f2f2015c580 2023-06-10T18:48:29.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.759+0000 7f2f17fff700 1 -- 172.21.15.104:0/1021333310 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f100043e0 con 0x7f2f2015c580 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 -- 172.21.15.104:0/1021333310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2015c580 msgr2=0x7f2f200b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2015c580 0x7f2f200b1070 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2f2006af30 tx=0x7f2f10009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 -- 172.21.15.104:0/1021333310 shutdown_connections 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2015c580 0x7f2f200b1070 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 --2- 172.21.15.104:0/1021333310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f20069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 --2- 172.21.15.104:0/1021333310 >> v2:172.21.15.114:3300/0 conn(0x7f2f2006e6e0 0x7f2f200b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 -- 172.21.15.104:0/1021333310 >> 172.21.15.104:0/1021333310 conn(0x7f2f2006b8a0 msgr2=0x7f2f200cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 -- 172.21.15.104:0/1021333310 shutdown_connections 2023-06-10T18:48:29.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.760+0000 7f2f274da700 1 -- 172.21.15.104:0/1021333310 wait complete. 2023-06-10T18:48:29.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.761+0000 7f2f274da700 1 Processor -- start 2023-06-10T18:48:29.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.761+0000 7f2f274da700 1 -- start start 2023-06-10T18:48:29.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f201f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 0x7f2f200af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200c9410 con 0x7f2f2006e6e0 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201f6060 con 0x7f2f2015c580 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f274da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200c7050 con 0x7f2f2006cde0 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59214/0 (socket says 172.21.15.104:59214) 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 -- 172.21.15.104:0/2705526841 learned_addr learned my addr 172.21.15.104:0/2705526841 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f25276700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f201f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f25a77700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 0x7f2f200af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 -- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 msgr2=0x7f2f201f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f201f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 -- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 msgr2=0x7f2f200af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 0x7f2f200af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 -- 172.21.15.104:0/2705526841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10009500 con 0x7f2f2006e6e0 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f25a77700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 0x7f2f200af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.762+0000 7f2f24a75700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c7f0 tx=0x7f2f1c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.763+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c011800 con 0x7f2f2006e6e0 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.763+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c011960 con 0x7f2f2006e6e0 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.763+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f200c7330 con 0x7f2f2006e6e0 2023-06-10T18:48:29.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.763+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c011cf0 con 0x7f2f2006e6e0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.763+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f200c78f0 con 0x7f2f2006e6e0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.764+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f20074050 con 0x7f2f2006e6e0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.764+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2f1c018360 con 0x7f2f2006e6e0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.764+0000 7f2f15ffb700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 0x7f2f0406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.764+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2f1c0895a0 con 0x7f2f2006e6e0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.765+0000 7f2f25276700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 0x7f2f0406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:29.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.765+0000 7f2f25276700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 0x7f2f0406d190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2f2006af30 tx=0x7f2f10008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:29.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.769+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2f1c0545a0 con 0x7f2f2006e6e0 2023-06-10T18:48:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.954+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f200b0850 con 0x7f2f2006e6e0 2023-06-10T18:48:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.954+0000 7f2f15ffb700 1 -- 172.21.15.104:0/2705526841 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f1c057bc0 con 0x7f2f2006e6e0 2023-06-10T18:48:29.961 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:48:29.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 msgr2=0x7f2f0406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 0x7f2f0406d190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2f2006af30 tx=0x7f2f10008070 comp rx=0 tx=0).stop 2023-06-10T18:48:29.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 msgr2=0x7f2f2015abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:29.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c7f0 tx=0x7f2f1c00cbb0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 shutdown_connections 2023-06-10T18:48:29.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2f0406ace0 0x7f2f0406d190 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2015c580 0x7f2f200af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2006cde0 0x7f2f201f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 --2- 172.21.15.104:0/2705526841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f2006e6e0 0x7f2f2015abd0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:29.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 >> 172.21.15.104:0/2705526841 conn(0x7f2f2006b8a0 msgr2=0x7f2f200cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:29.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 shutdown_connections 2023-06-10T18:48:29.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:29.956+0000 7f2f274da700 1 -- 172.21.15.104:0/2705526841 wait complete. 2023-06-10T18:48:29.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:48:30.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.090+0000 7f5c047ee700 1 Processor -- start 2023-06-10T18:48:30.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.090+0000 7f5c047ee700 1 -- start start 2023-06-10T18:48:30.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c047ee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 0x7f5bfc06a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c047ee700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc15c8d0 con 0x7f5bfc15c500 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c047ee700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc06abd0 con 0x7f5bfc069e70 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c047ee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0aef20 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c0258a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5bfc15c500 0x7f5bfc0cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57048/0 (socket says 172.21.15.104:57048) 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c0258a700 1 -- 172.21.15.104:0/2364227082 learned_addr learned my addr 172.21.15.104:0/2364227082 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:30.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.091+0000 7f5c02d8b700 1 --2- 172.21.15.104:0/2364227082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 0x7f5bfc06a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203495415 0 0) 0x7f5bfc15c8d0 con 0x7f5bfc15c500 2023-06-10T18:48:30.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be80035b0 con 0x7f5bfc15c500 2023-06-10T18:48:30.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189631541 0 0) 0x7f5bfc06abd0 con 0x7f5bfc069e70 2023-06-10T18:48:30.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bfc15c8d0 con 0x7f5bfc069e70 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c02d8b700 1 -- 172.21.15.104:0/2364227082 >> v1:172.21.15.114:6789/0 conn(0x7f5bfc069e70 legacy=0x7f5bfc06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c02d8b700 1 -- 172.21.15.104:0/2364227082 >> v1:172.21.15.104:6789/0 conn(0x7f5bfc15c500 legacy=0x7f5bfc0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c02d8b700 1 -- 172.21.15.104:0/2364227082 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc06a8f0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c02d8b700 1 --2- 172.21.15.104:0/2364227082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 0x7f5bfc06a310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009df0 tx=0x7f5bf400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133c466067700ec7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400e070 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf400bae0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.092+0000 7f5c01588700 1 -- 172.21.15.104:0/2364227082 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400be90 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 -- 172.21.15.104:0/2364227082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 msgr2=0x7f5bfc06a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2364227082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 0x7f5bfc06a310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009df0 tx=0x7f5bf400b330 comp rx=0 tx=0).stop 2023-06-10T18:48:30.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 -- 172.21.15.104:0/2364227082 shutdown_connections 2023-06-10T18:48:30.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2364227082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc06e3e0 0x7f5bfc06a310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 -- 172.21.15.104:0/2364227082 >> 172.21.15.104:0/2364227082 conn(0x7f5bfc06b8a0 msgr2=0x7f5bfc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 -- 172.21.15.104:0/2364227082 shutdown_connections 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.093+0000 7f5c047ee700 1 -- 172.21.15.104:0/2364227082 wait complete. 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.094+0000 7f5c047ee700 1 Processor -- start 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.094+0000 7f5c047ee700 1 -- start start 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc069e70 0x7f5bfc14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 0x7f5bfc0caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc14f950 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc15e700 con 0x7f5bfc15c500 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc15e9e0 con 0x7f5bfc069e70 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59228/0 (socket says 172.21.15.104:59228) 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 -- 172.21.15.104:0/2645129614 learned_addr learned my addr 172.21.15.104:0/2645129614 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:30.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c02d8b700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 0x7f5bfc0caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c0258a700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc069e70 0x7f5bfc14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 -- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc069e70 msgr2=0x7f5bfc14f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc069e70 0x7f5bfc14f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 -- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 msgr2=0x7f5bfc0caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 0x7f5bfc0caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 -- 172.21.15.104:0/2645129614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf4009a70 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c02d8b700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 0x7f5bfc0caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c01d89700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c500 tx=0x7f5bf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80052d0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf8005450 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc15ecc0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80038c0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.095+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc15f280 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.096+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc074050 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.097+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5bf8003a20 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.097+0000 7f5bf2ffd700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 0x7f5be006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.097+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5bf8088b60 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.097+0000 7f5c0258a700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 0x7f5be006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.098+0000 7f5c0258a700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 0x7f5be006d190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0010e0 tx=0x7f5bec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.101+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5bf8053a50 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.288+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5bfc0c2ef0 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.288+0000 7f5bf2ffd700 1 -- 172.21.15.104:0/2645129614 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5bf8057070 con 0x7f5bfc06e3e0 2023-06-10T18:48:30.294 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:48:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.290+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 msgr2=0x7f5be006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.290+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 0x7f5be006d190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0010e0 tx=0x7f5bec009040 comp rx=0 tx=0).stop 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.290+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 msgr2=0x7f5bfc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.290+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c500 tx=0x7f5bf800c8c0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 shutdown_connections 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5be006ace0 0x7f5be006d190 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bfc15c500 0x7f5bfc0caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bfc069e70 0x7f5bfc14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 --2- 172.21.15.104:0/2645129614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5bfc06e3e0 0x7f5bfc15dca0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 >> 172.21.15.104:0/2645129614 conn(0x7f5bfc06b8a0 msgr2=0x7f5bfc0c9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:30.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 shutdown_connections 2023-06-10T18:48:30.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.291+0000 7f5c047ee700 1 -- 172.21.15.104:0/2645129614 wait complete. 2023-06-10T18:48:30.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:30.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.425+0000 7f5422ab6700 1 Processor -- start 2023-06-10T18:48:30.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.425+0000 7f5422ab6700 1 -- start start 2023-06-10T18:48:30.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.426+0000 7f5422ab6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.426+0000 7f5422ab6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f541c070df0 0x7f541c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.426+0000 7f5422ab6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f541c158ce0 0x7f541c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.426+0000 7f5422ab6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c06ae00 con 0x7f541c158ce0 2023-06-10T18:48:30.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5422ab6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c069e70 con 0x7f541c070df0 2023-06-10T18:48:30.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5422ab6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c1595f0 con 0x7f541c070a20 2023-06-10T18:48:30.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f541c158ce0 0x7f541c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f541c158ce0 0x7f541c159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59238/0 (socket says 172.21.15.104:59238) 2023-06-10T18:48:30.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 -- 172.21.15.104:0/1187707435 learned_addr learned my addr 172.21.15.104:0/1187707435 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:30.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5420852700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f541bfff700 1 --2- 172.21.15.104:0/1187707435 >> v2:172.21.15.114:3300/0 conn(0x7f541c070df0 0x7f541c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 -- 172.21.15.104:0/1187707435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 msgr2=0x7f541c06a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:30.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c06a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 -- 172.21.15.104:0/1187707435 >> v2:172.21.15.114:3300/0 conn(0x7f541c070df0 msgr2=0x7f541c1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:30.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 --2- 172.21.15.104:0/1187707435 >> v2:172.21.15.114:3300/0 conn(0x7f541c070df0 0x7f541c1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 -- 172.21.15.104:0/1187707435 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c1598d0 con 0x7f541c158ce0 2023-06-10T18:48:30.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f5421053700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c159c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f541c070670 tx=0x7f541400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524804b7a33c9e8a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f541b7fe700 1 -- 172.21.15.104:0/1187707435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400f070 con 0x7f541c158ce0 2023-06-10T18:48:30.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f541b7fe700 1 -- 172.21.15.104:0/1187707435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541400a2e0 con 0x7f541c158ce0 2023-06-10T18:48:30.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.427+0000 7f541b7fe700 1 -- 172.21.15.104:0/1187707435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541400cc10 con 0x7f541c158ce0 2023-06-10T18:48:30.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 -- 172.21.15.104:0/1187707435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 msgr2=0x7f541c159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c159c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f541c070670 tx=0x7f541400c330 comp rx=0 tx=0).stop 2023-06-10T18:48:30.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 -- 172.21.15.104:0/1187707435 shutdown_connections 2023-06-10T18:48:30.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c159c40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1187707435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c06a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1187707435 >> v2:172.21.15.114:3300/0 conn(0x7f541c070df0 0x7f541c1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 -- 172.21.15.104:0/1187707435 >> 172.21.15.104:0/1187707435 conn(0x7f541c0b1610 msgr2=0x7f541c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 -- 172.21.15.104:0/1187707435 shutdown_connections 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.428+0000 7f5422ab6700 1 -- 172.21.15.104:0/1187707435 wait complete. 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.429+0000 7f5422ab6700 1 Processor -- start 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.429+0000 7f5422ab6700 1 -- start start 2023-06-10T18:48:30.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 0x7f541c0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b3ea0 con 0x7f541c158ce0 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b24e0 con 0x7f541c070df0 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5422ab6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0b4100 con 0x7f541c070a20 2023-06-10T18:48:30.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59244/0 (socket says 172.21.15.104:59244) 2023-06-10T18:48:30.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 -- 172.21.15.104:0/1685673612 learned_addr learned my addr 172.21.15.104:0/1685673612 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:30.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f541bfff700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 0x7f541c0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5420852700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 -- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 msgr2=0x7f541c156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 -- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 msgr2=0x7f541c0b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 0x7f541c0b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 -- 172.21.15.104:0/1685673612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541400ac40 con 0x7f541c158ce0 2023-06-10T18:48:30.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f541bfff700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 0x7f541c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:30.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.430+0000 7f5421053700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f541c06c890 tx=0x7f5414004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.431+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414003e10 con 0x7f541c158ce0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.431+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414004250 con 0x7f541c158ce0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.431+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0b43e0 con 0x7f541c158ce0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.431+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414004600 con 0x7f541c158ce0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.431+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c0b49a0 con 0x7f541c158ce0 2023-06-10T18:48:30.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.432+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c071780 con 0x7f541c158ce0 2023-06-10T18:48:30.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.432+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5414004760 con 0x7f541c158ce0 2023-06-10T18:48:30.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.432+0000 7f54197fa700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 0x7f540006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.432+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f541408bb00 con 0x7f541c158ce0 2023-06-10T18:48:30.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.433+0000 7f5420852700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 0x7f540006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.433+0000 7f5420852700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 0x7f540006d190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5410009ad0 tx=0x7f5410008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.437+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5414056a80 con 0x7f541c158ce0 2023-06-10T18:48:30.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.662+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f541c074050 con 0x7f541c158ce0 2023-06-10T18:48:30.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.663+0000 7f54197fa700 1 -- 172.21.15.104:0/1685673612 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f541405a0a0 con 0x7f541c158ce0 2023-06-10T18:48:30.669 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:30.669 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:30.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 msgr2=0x7f540006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 0x7f540006d190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5410009ad0 tx=0x7f5410008040 comp rx=0 tx=0).stop 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 msgr2=0x7f541c0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f541c06c890 tx=0x7f5414004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 shutdown_connections 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f541c158ce0 0x7f541c0b1fa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f541c070a20 0x7f541c156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f540006ace0 0x7f540006d190 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 --2- 172.21.15.104:0/1685673612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541c070df0 0x7f541c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 >> 172.21.15.104:0/1685673612 conn(0x7f541c0b1610 msgr2=0x7f541c06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 shutdown_connections 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.665+0000 7f5422ab6700 1 -- 172.21.15.104:0/1685673612 wait complete. 2023-06-10T18:48:30.672 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:30.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.796+0000 7f6743403700 1 Processor -- start 2023-06-10T18:48:30.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f6743403700 1 -- start start 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f6743403700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f673c06e0e0 0x7f673c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f6743403700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c06d290 con 0x7f673c069e70 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f6743403700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0ca2d0 con 0x7f673c06e4b0 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f6743403700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c071640 con 0x7f673c06e0e0 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f674119f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f673c06e0e0 0x7f673c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f67419a0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f673c069e70 0x7f673c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57078/0 (socket says 172.21.15.104:57078) 2023-06-10T18:48:30.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.797+0000 7f67419a0700 1 -- 172.21.15.104:0/2308534185 learned_addr learned my addr 172.21.15.104:0/2308534185 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f674119f700 1 -- 172.21.15.104:0/2308534185 >> v1:172.21.15.114:6789/0 conn(0x7f673c06e4b0 legacy=0x7f673c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f674119f700 1 -- 172.21.15.104:0/2308534185 >> v1:172.21.15.104:6789/0 conn(0x7f673c069e70 legacy=0x7f673c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6733fff700 1 -- 172.21.15.104:0/2308534185 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835411900 0 0) 0x7f673c06d290 con 0x7f673c069e70 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f674119f700 1 -- 172.21.15.104:0/2308534185 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c06d4c0 con 0x7f673c06e0e0 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f674119f700 1 --2- 172.21.15.104:0/2308534185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c06e0e0 0x7f673c06cd50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f672c009820 tx=0x7f672c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76401e22fe342df server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6733fff700 1 -- 172.21.15.104:0/2308534185 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c005e30 con 0x7f673c06e0e0 2023-06-10T18:48:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6733fff700 1 -- 172.21.15.104:0/2308534185 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c004030 con 0x7f673c06e0e0 2023-06-10T18:48:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6733fff700 1 -- 172.21.15.104:0/2308534185 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c0043c0 con 0x7f673c06e0e0 2023-06-10T18:48:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6743403700 1 -- 172.21.15.104:0/2308534185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c06e0e0 msgr2=0x7f673c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6743403700 1 --2- 172.21.15.104:0/2308534185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c06e0e0 0x7f673c06cd50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f672c009820 tx=0x7f672c009b30 comp rx=0 tx=0).stop 2023-06-10T18:48:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.798+0000 7f6743403700 1 -- 172.21.15.104:0/2308534185 shutdown_connections 2023-06-10T18:48:30.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.799+0000 7f6743403700 1 --2- 172.21.15.104:0/2308534185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c06e0e0 0x7f673c06cd50 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.799+0000 7f6743403700 1 -- 172.21.15.104:0/2308534185 >> 172.21.15.104:0/2308534185 conn(0x7f673c06b8a0 msgr2=0x7f673c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.799+0000 7f6743403700 1 -- 172.21.15.104:0/2308534185 shutdown_connections 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.799+0000 7f6743403700 1 -- 172.21.15.104:0/2308534185 wait complete. 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 Processor -- start 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 -- start start 2023-06-10T18:48:30.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 0x7f673c099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f673c06e4b0 0x7f673c0b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c099c90 con 0x7f673c069e70 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c099f70 con 0x7f673c06e0e0 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.800+0000 7f6743403700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b2430 con 0x7f673c06e4b0 2023-06-10T18:48:30.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59252/0 (socket says 172.21.15.104:59252) 2023-06-10T18:48:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 -- 172.21.15.104:0/2408492057 learned_addr learned my addr 172.21.15.104:0/2408492057 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674099e700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 0x7f673c099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 -- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f673c06e4b0 msgr2=0x7f673c0b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f673c06e4b0 0x7f673c0b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 -- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 msgr2=0x7f673c099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:30.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 0x7f673c099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:30.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 -- 172.21.15.104:0/2408492057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c009430 con 0x7f673c069e70 2023-06-10T18:48:30.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674099e700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 0x7f673c099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:30.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f674119f700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f672c000c00 tx=0x7f672c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c01d630 con 0x7f673c069e70 2023-06-10T18:48:30.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c01d7b0 con 0x7f673c069e70 2023-06-10T18:48:30.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c0b2710 con 0x7f673c069e70 2023-06-10T18:48:30.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c01db60 con 0x7f673c069e70 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.801+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c099880 con 0x7f673c069e70 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.802+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c0b2c30 con 0x7f673c069e70 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.803+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f672c01dcc0 con 0x7f673c069e70 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.803+0000 7f6731ffb700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 0x7f672006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.803+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f672c08ace0 con 0x7f673c069e70 2023-06-10T18:48:30.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.803+0000 7f674099e700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 0x7f672006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:30.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.804+0000 7f674099e700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 0x7f672006d190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f673c0b1810 tx=0x7f6738009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:30.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.807+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f673c0b2c30 con 0x7f673c069e70 2023-06-10T18:48:31.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.997+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f673c0b2c30 con 0x7f673c069e70 2023-06-10T18:48:31.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:30.999+0000 7f6731ffb700 1 -- 172.21.15.104:0/2408492057 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f673c0b2c30 con 0x7f673c069e70 2023-06-10T18:48:31.005 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:31.005 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:31.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 msgr2=0x7f672006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 0x7f672006d190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f673c0b1810 tx=0x7f6738009040 comp rx=0 tx=0).stop 2023-06-10T18:48:31.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 msgr2=0x7f673c15a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f672c000c00 tx=0x7f672c004750 comp rx=0 tx=0).stop 2023-06-10T18:48:31.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 shutdown_connections 2023-06-10T18:48:31.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f673c06e4b0 0x7f673c0b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f673c069e70 0x7f673c15a670 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f672006ace0 0x7f672006d190 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 --2- 172.21.15.104:0/2408492057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f673c06e0e0 0x7f673c099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 >> 172.21.15.104:0/2408492057 conn(0x7f673c06b8a0 msgr2=0x7f673c0ad070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 shutdown_connections 2023-06-10T18:48:31.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.001+0000 7f6743403700 1 -- 172.21.15.104:0/2408492057 wait complete. 2023-06-10T18:48:31.019 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:48:31.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:31.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.134+0000 7f4c67218700 1 Processor -- start 2023-06-10T18:48:31.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 -- start start 2023-06-10T18:48:31.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c6015a420 0x7f4c600b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c600cea60 0x7f4c6015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600b6070 con 0x7f4c600b4ff0 2023-06-10T18:48:31.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600b5460 con 0x7f4c600cea60 2023-06-10T18:48:31.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c67218700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600b5740 con 0x7f4c6015a420 2023-06-10T18:48:31.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c657b5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c600cea60 0x7f4c6015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c657b5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4c600cea60 0x7f4c6015a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59264/0 (socket says 172.21.15.104:59264) 2023-06-10T18:48:31.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.135+0000 7f4c657b5700 1 -- 172.21.15.104:0/3843065794 learned_addr learned my addr 172.21.15.104:0/3843065794 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c64fb4700 1 --2- 172.21.15.104:0/3843065794 >> v2:172.21.15.114:3300/0 conn(0x7f4c6015a420 0x7f4c600b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c657b5700 1 -- 172.21.15.104:0/3843065794 >> v2:172.21.15.114:3300/0 conn(0x7f4c6015a420 msgr2=0x7f4c600b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c657b5700 1 --2- 172.21.15.104:0/3843065794 >> v2:172.21.15.114:3300/0 conn(0x7f4c6015a420 0x7f4c600b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c657b5700 1 -- 172.21.15.104:0/3843065794 >> v1:172.21.15.114:6789/0 conn(0x7f4c600b4ff0 legacy=0x7f4c600ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c657b5700 1 -- 172.21.15.104:0/3843065794 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600b5a20 con 0x7f4c600cea60 2023-06-10T18:48:31.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c657b5700 1 --2- 172.21.15.104:0/3843065794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600cea60 0x7f4c6015a7f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4c600b32e0 tx=0x7f4c5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c983c273ac9aaf14 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c5f7fe700 1 -- 172.21.15.104:0/3843065794 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800e070 con 0x7f4c600cea60 2023-06-10T18:48:31.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c5f7fe700 1 -- 172.21.15.104:0/3843065794 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58003a20 con 0x7f4c600cea60 2023-06-10T18:48:31.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.136+0000 7f4c5f7fe700 1 -- 172.21.15.104:0/3843065794 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58003dd0 con 0x7f4c600cea60 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 -- 172.21.15.104:0/3843065794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600cea60 msgr2=0x7f4c6015a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 --2- 172.21.15.104:0/3843065794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600cea60 0x7f4c6015a7f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f4c600b32e0 tx=0x7f4c5800b330 comp rx=0 tx=0).stop 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 -- 172.21.15.104:0/3843065794 shutdown_connections 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 --2- 172.21.15.104:0/3843065794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600cea60 0x7f4c6015a7f0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 --2- 172.21.15.104:0/3843065794 >> v2:172.21.15.114:3300/0 conn(0x7f4c6015a420 0x7f4c600b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 -- 172.21.15.104:0/3843065794 >> 172.21.15.104:0/3843065794 conn(0x7f4c60089fd0 msgr2=0x7f4c600d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 -- 172.21.15.104:0/3843065794 shutdown_connections 2023-06-10T18:48:31.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.137+0000 7f4c67218700 1 -- 172.21.15.104:0/3843065794 wait complete. 2023-06-10T18:48:31.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.138+0000 7f4c67218700 1 Processor -- start 2023-06-10T18:48:31.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.138+0000 7f4c67218700 1 -- start start 2023-06-10T18:48:31.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 0x7f4c6014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c6015a420 0x7f4c6014cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c601e4ba0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c601e3b20 con 0x7f4c600cea60 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c601e3e00 con 0x7f4c6015a420 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59266/0 (socket says 172.21.15.104:59266) 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 -- 172.21.15.104:0/435604039 learned_addr learned my addr 172.21.15.104:0/435604039 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c5ffff700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 0x7f4c6014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c657b5700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c6015a420 0x7f4c6014cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 -- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c6015a420 msgr2=0x7f4c6014cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c6015a420 0x7f4c6014cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 -- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 msgr2=0x7f4c6014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 0x7f4c6014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 -- 172.21.15.104:0/435604039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58009b30 con 0x7f4c600b4ff0 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c5ffff700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 0x7f4c6014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c64fb4700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400d800 tx=0x7f4c5400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c540125f0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54012750 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c601e40e0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54012b00 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.139+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c601e46a0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.140+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c601e4830 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.141+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4c54012c60 con 0x7f4c600b4ff0 2023-06-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.141+0000 7f4c5d7fa700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.141+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4c540898f0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.141+0000 7f4c5ffff700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.142+0000 7f4c5ffff700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c003f40 tx=0x7f4c4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.145+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4c540548f0 con 0x7f4c600b4ff0 2023-06-10T18:48:31.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.380+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c60074050 con 0x7f4c600b4ff0 2023-06-10T18:48:31.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.380+0000 7f4c5d7fa700 1 -- 172.21.15.104:0/435604039 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f4c54057f10 con 0x7f4c600b4ff0 2023-06-10T18:48:31.387 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:31.387 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:31.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 msgr2=0x7f4c4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c003f40 tx=0x7f4c4c009040 comp rx=0 tx=0).stop 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 msgr2=0x7f4c6015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400d800 tx=0x7f4c5400dbc0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 shutdown_connections 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c6015a420 0x7f4c6014cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c600b4ff0 0x7f4c6015cd50 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 --2- 172.21.15.104:0/435604039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c600cea60 0x7f4c6014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 >> 172.21.15.104:0/435604039 conn(0x7f4c60089fd0 msgr2=0x7f4c600ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.382+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 shutdown_connections 2023-06-10T18:48:31.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.383+0000 7f4c67218700 1 -- 172.21.15.104:0/435604039 wait complete. 2023-06-10T18:48:31.390 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:31.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.514+0000 7f8cac3c9700 1 Processor -- start 2023-06-10T18:48:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.514+0000 7f8cac3c9700 1 -- start start 2023-06-10T18:48:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8cac3c9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ca40aeaa0 0x7f8ca40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8cac3c9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 0x7f8ca406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8cac3c9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40c9e60 con 0x7f8ca40b1080 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8cac3c9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca40c9bc0 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8cac3c9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca40cdaa0 con 0x7f8ca40cd680 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8caa165700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ca40aeaa0 0x7f8ca40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.515+0000 7f8caa165700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ca40aeaa0 0x7f8ca40cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59274/0 (socket says 172.21.15.104:59274) 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 -- 172.21.15.104:0/1841385016 learned_addr learned my addr 172.21.15.104:0/1841385016 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa966700 1 --2- 172.21.15.104:0/1841385016 >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 0x7f8ca406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 -- 172.21.15.104:0/1841385016 >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 msgr2=0x7f8ca406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 --2- 172.21.15.104:0/1841385016 >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 0x7f8ca406dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 -- 172.21.15.104:0/1841385016 >> v1:172.21.15.114:6789/0 conn(0x7f8ca40b1080 legacy=0x7f8ca40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 -- 172.21.15.104:0/1841385016 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40aee70 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa966700 1 --2- 172.21.15.104:0/1841385016 >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 0x7f8ca406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:31.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8caa165700 1 --2- 172.21.15.104:0/1841385016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca40cd140 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4071030 tx=0x7f8c9c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d26b0af19b5241 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8ca9163700 1 -- 172.21.15.104:0/1841385016 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c005e30 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8ca9163700 1 -- 172.21.15.104:0/1841385016 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c00d8b0 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.516+0000 7f8ca9163700 1 -- 172.21.15.104:0/1841385016 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c00dc40 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 -- 172.21.15.104:0/1841385016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40aeaa0 msgr2=0x7f8ca40cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/1841385016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca40cd140 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4071030 tx=0x7f8c9c00be70 comp rx=0 tx=0).stop 2023-06-10T18:48:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 -- 172.21.15.104:0/1841385016 shutdown_connections 2023-06-10T18:48:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/1841385016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca40cd140 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/1841385016 >> v2:172.21.15.114:3300/0 conn(0x7f8ca40cd680 0x7f8ca406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 -- 172.21.15.104:0/1841385016 >> 172.21.15.104:0/1841385016 conn(0x7f8ca406b8a0 msgr2=0x7f8ca40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 -- 172.21.15.104:0/1841385016 shutdown_connections 2023-06-10T18:48:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.517+0000 7f8cac3c9700 1 -- 172.21.15.104:0/1841385016 wait complete. 2023-06-10T18:48:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.518+0000 7f8cac3c9700 1 Processor -- start 2023-06-10T18:48:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.518+0000 7f8cac3c9700 1 -- start start 2023-06-10T18:48:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca4157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ca40b1080 0x7f8ca415c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40c8a30 con 0x7f8ca40cd680 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4157ad0 con 0x7f8ca40aeaa0 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40c7db0 con 0x7f8ca40b1080 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59284/0 (socket says 172.21.15.104:59284) 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 -- 172.21.15.104:0/2121988445 learned_addr learned my addr 172.21.15.104:0/2121988445 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa165700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca4157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8ca9964700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ca40b1080 0x7f8ca415c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 -- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ca40b1080 msgr2=0x7f8ca415c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ca40b1080 0x7f8ca415c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 -- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca40aeaa0 msgr2=0x7f8ca4157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca4157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 -- 172.21.15.104:0/2121988445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c00b7a0 con 0x7f8ca40cd680 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8caa966700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400cd50 tx=0x7f8c9400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400cea0 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9400f480 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca40c8090 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400f830 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.519+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca40c8650 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.520+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca4074050 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.521+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c9400b040 con 0x7f8ca40cd680 2023-06-10T18:48:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.521+0000 7f8ca2ffd700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 0x7f8c9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.521+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8c94089c80 con 0x7f8ca40cd680 2023-06-10T18:48:31.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.521+0000 7f8caa165700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 0x7f8c9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.522+0000 7f8caa165700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 0x7f8c9006d190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c000c00 tx=0x7f8c9c012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.525+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c94054c80 con 0x7f8ca40cd680 2023-06-10T18:48:31.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.755+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ca40af550 con 0x7f8ca40cd680 2023-06-10T18:48:31.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.756+0000 7f8ca2ffd700 1 -- 172.21.15.104:0/2121988445 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4683 (secure 0 0 0) 0x7f8c940582a0 con 0x7f8ca40cd680 2023-06-10T18:48:31.763 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:31.763 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:47:59.237789+0000","modified":"2023-06-10T18:48:02.253448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1791317446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1791317446},{"type":"v1","addr":"172.21.15.114:6836","nonce":1791317446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-06-10T18:48:31.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 msgr2=0x7f8c9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 0x7f8c9006d190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c000c00 tx=0x7f8c9c012000 comp rx=0 tx=0).stop 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 msgr2=0x7f8ca41ecf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400cd50 tx=0x7f8c9400a560 comp rx=0 tx=0).stop 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 shutdown_connections 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c9006ace0 0x7f8c9006d190 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca40aeaa0 0x7f8ca4157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ca40cd680 0x7f8ca41ecf30 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 --2- 172.21.15.104:0/2121988445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ca40b1080 0x7f8ca415c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 >> 172.21.15.104:0/2121988445 conn(0x7f8ca406b8a0 msgr2=0x7f8ca40cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 shutdown_connections 2023-06-10T18:48:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.758+0000 7f8cac3c9700 1 -- 172.21.15.104:0/2121988445 wait complete. 2023-06-10T18:48:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 25 2023-06-10T18:48:31.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:31.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.890+0000 7f64a48ef700 1 Processor -- start 2023-06-10T18:48:31.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.890+0000 7f64a48ef700 1 -- start start 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070a20 0x7f64a006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f64a0070df0 0x7f64a01587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f64a0158ce0 0x7f64a0159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a006ae00 con 0x7f64a0158ce0 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a0069e70 con 0x7f64a0070df0 2023-06-10T18:48:31.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f64a48ef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01595f0 con 0x7f64a0070a20 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f64a0158ce0 0x7f64a0159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f64a0158ce0 0x7f64a0159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59292/0 (socket says 172.21.15.104:59292) 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 -- 172.21.15.104:0/3767836481 learned_addr learned my addr 172.21.15.104:0/3767836481 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649dd9b700 1 --2- 172.21.15.104:0/3767836481 >> v2:172.21.15.114:3300/0 conn(0x7f64a0070df0 0x7f64a01587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649e59c700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070a20 0x7f64a006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 -- 172.21.15.104:0/3767836481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070a20 msgr2=0x7f64a006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070a20 0x7f64a006a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 -- 172.21.15.104:0/3767836481 >> v2:172.21.15.114:3300/0 conn(0x7f64a0070df0 msgr2=0x7f64a01587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 --2- 172.21.15.104:0/3767836481 >> v2:172.21.15.114:3300/0 conn(0x7f64a0070df0 0x7f64a01587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 -- 172.21.15.104:0/3767836481 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a01598d0 con 0x7f64a0158ce0 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.891+0000 7f649ed9d700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a0159c40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f64a0070670 tx=0x7f649800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9d623ae4c4f0cf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f649d59a700 1 -- 172.21.15.104:0/3767836481 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498010070 con 0x7f64a0158ce0 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f649d59a700 1 -- 172.21.15.104:0/3767836481 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649800dad0 con 0x7f64a0158ce0 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f649d59a700 1 -- 172.21.15.104:0/3767836481 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649800de80 con 0x7f64a0158ce0 2023-06-10T18:48:31.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 -- 172.21.15.104:0/3767836481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 msgr2=0x7f64a0159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a0159c40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f64a0070670 tx=0x7f649800d430 comp rx=0 tx=0).stop 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 -- 172.21.15.104:0/3767836481 shutdown_connections 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a0159c40 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 --2- 172.21.15.104:0/3767836481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070a20 0x7f64a006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 --2- 172.21.15.104:0/3767836481 >> v2:172.21.15.114:3300/0 conn(0x7f64a0070df0 0x7f64a01587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 -- 172.21.15.104:0/3767836481 >> 172.21.15.104:0/3767836481 conn(0x7f64a00b1610 msgr2=0x7f64a00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 -- 172.21.15.104:0/3767836481 shutdown_connections 2023-06-10T18:48:31.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.892+0000 7f64a48ef700 1 -- 172.21.15.104:0/3767836481 wait complete. 2023-06-10T18:48:31.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 Processor -- start 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 -- start start 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64a0070a20 0x7f64a00b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070df0 0x7f64a0155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00c5290 con 0x7f64a0158ce0 2023-06-10T18:48:31.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00c4960 con 0x7f64a0070a20 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f64a48ef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00c4c40 con 0x7f64a0070df0 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f649ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f649ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59298/0 (socket says 172.21.15.104:59298) 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.894+0000 7f649ed9d700 1 -- 172.21.15.104:0/4160511684 learned_addr learned my addr 172.21.15.104:0/4160511684 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649dd9b700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070df0 0x7f64a0155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649e59c700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64a0070a20 0x7f64a00b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 -- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070df0 msgr2=0x7f64a0155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070df0 0x7f64a0155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 -- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64a0070a20 msgr2=0x7f64a00b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64a0070a20 0x7f64a00b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 -- 172.21.15.104:0/4160511684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649800bd40 con 0x7f64a0158ce0 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f649ed9d700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f64a006c890 tx=0x7f6498004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498004040 con 0x7f64a0158ce0 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64980041c0 con 0x7f64a0158ce0 2023-06-10T18:48:31.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a00c4f20 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498004570 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.895+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a00c45a0 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.896+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0071780 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.896+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f64980046d0 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.897+0000 7f648effd700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 0x7f648006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.897+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f649808aa40 con 0x7f64a0158ce0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.897+0000 7f649e59c700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 0x7f648006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:31.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.898+0000 7f649e59c700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 0x7f648006d190 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f64a00cfc60 tx=0x7f6494009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:31.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:31.901+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f649808ce60 con 0x7f64a0158ce0 2023-06-10T18:48:32.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.087+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64a0074050 con 0x7f64a0158ce0 2023-06-10T18:48:32.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.088+0000 7f648effd700 1 -- 172.21.15.104:0/4160511684 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18600 (secure 0 0 0) 0x7f6498055a40 con 0x7f64a0158ce0 2023-06-10T18:48:32.094 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:32.095 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:00.239792+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:47:57.299455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-06-10T18:47:58.209852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:32.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.090+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 msgr2=0x7f648006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.090+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 0x7f648006d190 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f64a00cfc60 tx=0x7f6494009210 comp rx=0 tx=0).stop 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.090+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 msgr2=0x7f64a00add80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.090+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f64a006c890 tx=0x7f6498004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 shutdown_connections 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f648006ace0 0x7f648006d190 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64a0070a20 0x7f64a00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64a0158ce0 0x7f64a00add80 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 --2- 172.21.15.104:0/4160511684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f64a0070df0 0x7f64a0155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 >> 172.21.15.104:0/4160511684 conn(0x7f64a00b1610 msgr2=0x7f64a006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:32.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 shutdown_connections 2023-06-10T18:48:32.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.091+0000 7f64a48ef700 1 -- 172.21.15.104:0/4160511684 wait complete. 2023-06-10T18:48:32.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:48:32.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.224+0000 7f9e39470700 1 Processor -- start 2023-06-10T18:48:32.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.224+0000 7f9e39470700 1 -- start start 2023-06-10T18:48:32.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.224+0000 7f9e39470700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e34156740 0x7f9e3415d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.224+0000 7f9e39470700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e340cec20 0x7f9e340cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e39470700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e34069e70 0x7f9e3406a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e39470700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e3406a810 con 0x7f9e34069e70 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e39470700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e3415cdf0 con 0x7f9e34156740 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e39470700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340cd790 con 0x7f9e340cec20 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e34069e70 0x7f9e3406a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e34069e70 0x7f9e3406a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59306/0 (socket says 172.21.15.104:59306) 2023-06-10T18:48:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 -- 172.21.15.104:0/3831403000 learned_addr learned my addr 172.21.15.104:0/3831403000 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e327fc700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e340cec20 0x7f9e340cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e32ffd700 1 --2- 172.21.15.104:0/3831403000 >> v2:172.21.15.114:3300/0 conn(0x7f9e34156740 0x7f9e3415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 -- 172.21.15.104:0/3831403000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e340cec20 msgr2=0x7f9e340cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:32.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e340cec20 0x7f9e340cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 -- 172.21.15.104:0/3831403000 >> v2:172.21.15.114:3300/0 conn(0x7f9e34156740 msgr2=0x7f9e3415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 --2- 172.21.15.104:0/3831403000 >> v2:172.21.15.114:3300/0 conn(0x7f9e34156740 0x7f9e3415d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 -- 172.21.15.104:0/3831403000 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340cf0a0 con 0x7f9e34069e70 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e337fe700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e34069e70 0x7f9e3406a2d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9e3406ae70 tx=0x7f9e2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9937cc2bd7cf6fa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e31ffb700 1 -- 172.21.15.104:0/3831403000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c00e070 con 0x7f9e34069e70 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e31ffb700 1 -- 172.21.15.104:0/3831403000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2c00b9c0 con 0x7f9e34069e70 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.225+0000 7f9e31ffb700 1 -- 172.21.15.104:0/3831403000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c00bd70 con 0x7f9e34069e70 2023-06-10T18:48:32.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 -- 172.21.15.104:0/3831403000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e34069e70 msgr2=0x7f9e3406a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e34069e70 0x7f9e3406a2d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9e3406ae70 tx=0x7f9e2c00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 -- 172.21.15.104:0/3831403000 shutdown_connections 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e34069e70 0x7f9e3406a2d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 --2- 172.21.15.104:0/3831403000 >> v2:172.21.15.114:3300/0 conn(0x7f9e34156740 0x7f9e3415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 --2- 172.21.15.104:0/3831403000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e340cec20 0x7f9e340cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 -- 172.21.15.104:0/3831403000 >> 172.21.15.104:0/3831403000 conn(0x7f9e3406d240 msgr2=0x7f9e3406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 -- 172.21.15.104:0/3831403000 shutdown_connections 2023-06-10T18:48:32.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.226+0000 7f9e39470700 1 -- 172.21.15.104:0/3831403000 wait complete. 2023-06-10T18:48:32.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.227+0000 7f9e39470700 1 Processor -- start 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.227+0000 7f9e39470700 1 -- start start 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e34069e70 0x7f9e34153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 0x7f9e340c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e34154130 con 0x7f9e340c5830 2023-06-10T18:48:32.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340b2650 con 0x7f9e34156740 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e39470700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340b2930 con 0x7f9e34069e70 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59312/0 (socket says 172.21.15.104:59312) 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 -- 172.21.15.104:0/2018499477 learned_addr learned my addr 172.21.15.104:0/2018499477 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e327fc700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 0x7f9e340c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e32ffd700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e34069e70 0x7f9e34153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 -- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e34069e70 msgr2=0x7f9e34153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e34069e70 0x7f9e34153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 -- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 msgr2=0x7f9e340c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 0x7f9e340c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e337fe700 1 -- 172.21.15.104:0/2018499477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c009b30 con 0x7f9e340c5830 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.228+0000 7f9e327fc700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 0x7f9e340c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e337fe700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9e3406acf0 tx=0x7f9e2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c004250 con 0x7f9e340c5830 2023-06-10T18:48:32.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2c0043d0 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e340b2c10 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c004780 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.229+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e340b31d0 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.230+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e34074050 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.230+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9e2c01b430 con 0x7f9e340c5830 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.230+0000 7f9e1b7fe700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 0x7f9e2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.231+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9e2c08a6f0 con 0x7f9e340c5830 2023-06-10T18:48:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.231+0000 7f9e32ffd700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 0x7f9e2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.231+0000 7f9e32ffd700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 0x7f9e2006d190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009950 tx=0x7f9e24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.235+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9e2c0556f0 con 0x7f9e340c5830 2023-06-10T18:48:32.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.461+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9e340b1b10 con 0x7f9e340c5830 2023-06-10T18:48:32.649 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:48:32.643+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:48:32.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.646+0000 7f9e1b7fe700 1 -- 172.21.15.104:0/2018499477 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9e2c058d10 con 0x7f9e340c5830 2023-06-10T18:48:32.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 msgr2=0x7f9e2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 0x7f9e2006d190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009950 tx=0x7f9e24008040 comp rx=0 tx=0).stop 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 msgr2=0x7f9e340b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9e3406acf0 tx=0x7f9e2c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 shutdown_connections 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e340c5830 0x7f9e340b36f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e34069e70 0x7f9e34153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e2006ace0 0x7f9e2006d190 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 --2- 172.21.15.104:0/2018499477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e34156740 0x7f9e340c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.648+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 >> 172.21.15.104:0/2018499477 conn(0x7f9e3406d240 msgr2=0x7f9e340cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:32.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.649+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 shutdown_connections 2023-06-10T18:48:32.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.649+0000 7f9e39470700 1 -- 172.21.15.104:0/2018499477 wait complete. 2023-06-10T18:48:32.658 INFO:tasks.ceph.mds.b.smithi114.stderr: -4193> 2023-06-10T18:48:04.839+0000 7fcfbb8f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:48:32.659 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:48:32.667 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:48:32.686 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:48:32.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.783+0000 7f8acfdf1700 1 Processor -- start 2023-06-10T18:48:32.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.783+0000 7f8acfdf1700 1 -- start start 2023-06-10T18:48:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acfdf1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ac80b1780 0x7f8ac806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acfdf1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ac806ea70 0x7f8ac806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acfdf1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac80ae7e0 con 0x7f8ac815c6f0 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acfdf1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac806df90 con 0x7f8ac80b1780 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acfdf1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac806e270 con 0x7f8ac806ea70 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ac80b1780 0x7f8ac806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ac80b1780 0x7f8ac806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59318/0 (socket says 172.21.15.104:59318) 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 -- 172.21.15.104:0/2664445075 learned_addr learned my addr 172.21.15.104:0/2664445075 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8ace38e700 1 --2- 172.21.15.104:0/2664445075 >> v2:172.21.15.114:3300/0 conn(0x7f8ac806ea70 0x7f8ac806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 -- 172.21.15.104:0/2664445075 >> v2:172.21.15.114:3300/0 conn(0x7f8ac806ea70 msgr2=0x7f8ac806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 --2- 172.21.15.104:0/2664445075 >> v2:172.21.15.114:3300/0 conn(0x7f8ac806ea70 0x7f8ac806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 -- 172.21.15.104:0/2664445075 >> v1:172.21.15.114:6789/0 conn(0x7f8ac815c6f0 legacy=0x7f8ac80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.784+0000 7f8acd38c700 1 -- 172.21.15.104:0/2664445075 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac806e550 con 0x7f8ac80b1780 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acd38c700 1 --2- 172.21.15.104:0/2664445075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac80b1780 0x7f8ac806cde0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80c2fc0 tx=0x7f8ac4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2b4d248b015f0b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8accb8b700 1 -- 172.21.15.104:0/2664445075 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4011830 con 0x7f8ac80b1780 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8accb8b700 1 -- 172.21.15.104:0/2664445075 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac4011990 con 0x7f8ac80b1780 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8accb8b700 1 -- 172.21.15.104:0/2664445075 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4011d20 con 0x7f8ac80b1780 2023-06-10T18:48:32.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 -- 172.21.15.104:0/2664445075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac80b1780 msgr2=0x7f8ac806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/2664445075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac80b1780 0x7f8ac806cde0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80c2fc0 tx=0x7f8ac4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 -- 172.21.15.104:0/2664445075 shutdown_connections 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/2664445075 >> v2:172.21.15.114:3300/0 conn(0x7f8ac806ea70 0x7f8ac806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/2664445075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac80b1780 0x7f8ac806cde0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 -- 172.21.15.104:0/2664445075 >> 172.21.15.104:0/2664445075 conn(0x7f8ac806b8a0 msgr2=0x7f8ac80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 -- 172.21.15.104:0/2664445075 shutdown_connections 2023-06-10T18:48:32.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.785+0000 7f8acfdf1700 1 -- 172.21.15.104:0/2664445075 wait complete. 2023-06-10T18:48:32.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 Processor -- start 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 -- start start 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ac80b1780 0x7f8ac80c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ac815c6f0 0x7f8ac8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8158fd0 con 0x7f8ac806ea70 2023-06-10T18:48:32.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80ca570 con 0x7f8ac80b1780 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acfdf1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80ca850 con 0x7f8ac815c6f0 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acdb8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acdb8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59322/0 (socket says 172.21.15.104:59322) 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.787+0000 7f8acdb8d700 1 -- 172.21.15.104:0/1740087218 learned_addr learned my addr 172.21.15.104:0/1740087218 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acd38c700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ac80b1780 0x7f8ac80c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8ace38e700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ac815c6f0 0x7f8ac8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 -- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ac815c6f0 msgr2=0x7f8ac8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ac815c6f0 0x7f8ac8069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 -- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ac80b1780 msgr2=0x7f8ac80c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ac80b1780 0x7f8ac80c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 -- 172.21.15.104:0/1740087218 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4009550 con 0x7f8ac806ea70 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acdb8d700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c430 tx=0x7f8ab800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab80052d0 con 0x7f8ac806ea70 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8005450 con 0x7f8ac806ea70 2023-06-10T18:48:32.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac80cab30 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8006af0 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.788+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac80c6b80 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.789+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8074050 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.789+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8ab80036d0 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.790+0000 7f8abe7fc700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 0x7f8aac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.790+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8ab8088730 con 0x7f8ac806ea70 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.790+0000 7f8acd38c700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 0x7f8aac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:32.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.790+0000 7f8acd38c700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 0x7f8aac06d190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80b1b70 tx=0x7f8ac4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:32.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:32.794+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ab8053640 con 0x7f8ac806ea70 2023-06-10T18:48:33.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.028+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8ac80d0030 con 0x7f8ac806ea70 2023-06-10T18:48:33.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.657+0000 7f8abe7fc700 1 -- 172.21.15.104:0/1740087218 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ab8056c60 con 0x7f8ac806ea70 2023-06-10T18:48:33.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.659+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 msgr2=0x7f8aac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.659+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 0x7f8aac06d190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80b1b70 tx=0x7f8ac4008030 comp rx=0 tx=0).stop 2023-06-10T18:48:33.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.659+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 msgr2=0x7f8ac8158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.659+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c430 tx=0x7f8ab800c740 comp rx=0 tx=0).stop 2023-06-10T18:48:33.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 shutdown_connections 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ac815c6f0 0x7f8ac8069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ac806ea70 0x7f8ac8158a90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aac06ace0 0x7f8aac06d190 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 --2- 172.21.15.104:0/1740087218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ac80b1780 0x7f8ac80c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 >> 172.21.15.104:0/1740087218 conn(0x7f8ac806b8a0 msgr2=0x7f8ac815b470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:33.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 shutdown_connections 2023-06-10T18:48:33.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.660+0000 7f8acfdf1700 1 -- 172.21.15.104:0/1740087218 wait complete. 2023-06-10T18:48:33.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:48:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.803+0000 7ffb10f3f700 1 Processor -- start 2023-06-10T18:48:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.803+0000 7ffb10f3f700 1 -- start start 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb10f3f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffb0c15c500 0x7ffb0c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb10f3f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0c0cf200 0x7ffb0c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb10f3f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0c15c8d0 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb10f3f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c0aef20 con 0x7ffb0c0cf200 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb10f3f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c06abd0 con 0x7ffb0c15c500 2023-06-10T18:48:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0c0cf200 0x7ffb0c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ffb0c06a3b0 0x7ffb0c06dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57162/0 (socket says 172.21.15.104:57162) 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0ad9d700 1 -- 172.21.15.104:0/192547283 learned_addr learned my addr 172.21.15.104:0/192547283 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0a59c700 1 --2- 172.21.15.104:0/192547283 >> v2:172.21.15.114:3300/0 conn(0x7ffb0c15c500 0x7ffb0c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 -- 172.21.15.104:0/192547283 >> v2:172.21.15.114:3300/0 conn(0x7ffb0c15c500 msgr2=0x7ffb0c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/192547283 >> v2:172.21.15.114:3300/0 conn(0x7ffb0c15c500 0x7ffb0c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 -- 172.21.15.104:0/192547283 >> v1:172.21.15.104:6789/0 conn(0x7ffb0c06a3b0 legacy=0x7ffb0c06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0959a700 1 -- 172.21.15.104:0/192547283 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416636230 0 0) 0x7ffb0c15c8d0 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 -- 172.21.15.104:0/192547283 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c06a8f0 con 0x7ffb0c0cf200 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/192547283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf200 0x7ffb0c069e70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b2170 tx=0x7ffb00009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305b060e8e69e097 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0959a700 1 -- 172.21.15.104:0/192547283 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011830 con 0x7ffb0c0cf200 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0959a700 1 -- 172.21.15.104:0/192547283 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00011990 con 0x7ffb0c0cf200 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.804+0000 7ffb0959a700 1 -- 172.21.15.104:0/192547283 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011d20 con 0x7ffb0c0cf200 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 -- 172.21.15.104:0/192547283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf200 msgr2=0x7ffb0c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/192547283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf200 0x7ffb0c069e70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b2170 tx=0x7ffb00009b20 comp rx=0 tx=0).stop 2023-06-10T18:48:33.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 -- 172.21.15.104:0/192547283 shutdown_connections 2023-06-10T18:48:33.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/192547283 >> v2:172.21.15.114:3300/0 conn(0x7ffb0c15c500 0x7ffb0c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/192547283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf200 0x7ffb0c069e70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 -- 172.21.15.104:0/192547283 >> 172.21.15.104:0/192547283 conn(0x7ffb0c06b8a0 msgr2=0x7ffb0c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:33.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 -- 172.21.15.104:0/192547283 shutdown_connections 2023-06-10T18:48:33.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.805+0000 7ffb10f3f700 1 -- 172.21.15.104:0/192547283 wait complete. 2023-06-10T18:48:33.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.806+0000 7ffb10f3f700 1 Processor -- start 2023-06-10T18:48:33.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.806+0000 7ffb10f3f700 1 -- start start 2023-06-10T18:48:33.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 0x7ffb0c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 0x7ffb0c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c9c40 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c15ef10 con 0x7ffb0c15c500 2023-06-10T18:48:33.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb10f3f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c15f1f0 con 0x7ffb0c0cf200 2023-06-10T18:48:33.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb0a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb0a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59336/0 (socket says 172.21.15.104:59336) 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb0a59c700 1 -- 172.21.15.104:0/2082188679 learned_addr learned my addr 172.21.15.104:0/2082188679 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.807+0000 7ffb0ad9d700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 0x7ffb0c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 0x7ffb0c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 -- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 msgr2=0x7ffb0c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 0x7ffb0c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 -- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 msgr2=0x7ffb0c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 0x7ffb0c0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 -- 172.21.15.104:0/2082188679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb000094a0 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0ad9d700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 0x7ffb0c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 0x7ffb0c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb0a59c700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00e730 tx=0x7ffafc00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:33.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc013610 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc013770 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c15f4d0 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc013b20 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.808+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c151ec0 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.809+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c074050 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.809+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffafc013c80 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.810+0000 7ffafaffd700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:33.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.810+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ffafc089000 con 0x7ffb0c06a3b0 2023-06-10T18:48:33.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.810+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:33.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.811+0000 7ffb09d9b700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b0a50 tx=0x7ffb00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:33.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:33.814+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffafc0579a0 con 0x7ffb0c06a3b0 2023-06-10T18:48:34.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.000+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb0c0c9f20 con 0x7ffb0c06a3b0 2023-06-10T18:48:34.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.702+0000 7ffafaffd700 1 -- 172.21.15.104:0/2082188679 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7ffafc057530 con 0x7ffb0c06a3b0 2023-06-10T18:48:34.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 msgr2=0x7ffaec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b0a50 tx=0x7ffb00008030 comp rx=0 tx=0).stop 2023-06-10T18:48:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 msgr2=0x7ffb0c151980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00e730 tx=0x7ffafc00eaf0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 shutdown_connections 2023-06-10T18:48:34.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c15c500 0x7ffb0c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c06a3b0 0x7ffb0c151980 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 --2- 172.21.15.104:0/2082188679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf200 0x7ffb0c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.704+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 >> 172.21.15.104:0/2082188679 conn(0x7ffb0c06b8a0 msgr2=0x7ffb0c0aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.705+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 shutdown_connections 2023-06-10T18:48:34.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.705+0000 7ffb10f3f700 1 -- 172.21.15.104:0/2082188679 wait complete. 2023-06-10T18:48:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:48:34.724 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:48:34.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.846+0000 7fae82054700 1 Processor -- start 2023-06-10T18:48:34.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.846+0000 7fae82054700 1 -- start start 2023-06-10T18:48:34.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae82054700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae7c0ce390 0x7fae7c070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae82054700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae7c070f50 0x7fae7c0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae82054700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c06d7d0 con 0x7fae7c1586a0 2023-06-10T18:48:34.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae82054700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0cd6c0 con 0x7fae7c070f50 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae82054700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0c95d0 con 0x7fae7c0ce390 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae7c070f50 0x7fae7c0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae7c070f50 0x7fae7c0c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59348/0 (socket says 172.21.15.104:59348) 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 -- 172.21.15.104:0/1841488744 learned_addr learned my addr 172.21.15.104:0/1841488744 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 -- 172.21.15.104:0/1841488744 >> v2:172.21.15.114:3300/0 conn(0x7fae7c0ce390 msgr2=0x7fae7c070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 --2- 172.21.15.104:0/1841488744 >> v2:172.21.15.114:3300/0 conn(0x7fae7c0ce390 0x7fae7c070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 -- 172.21.15.104:0/1841488744 >> v1:172.21.15.114:6789/0 conn(0x7fae7c1586a0 legacy=0x7fae7c06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:34.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 -- 172.21.15.104:0/1841488744 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c071390 con 0x7fae7c070f50 2023-06-10T18:48:34.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7bfff700 1 --2- 172.21.15.104:0/1841488744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c070f50 0x7fae7c0c9a60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0cb910 tx=0x7fae7400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff30d68fc20fa8d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:34.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7a7fc700 1 -- 172.21.15.104:0/1841488744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74010070 con 0x7fae7c070f50 2023-06-10T18:48:34.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7a7fc700 1 -- 172.21.15.104:0/1841488744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7400db40 con 0x7fae7c070f50 2023-06-10T18:48:34.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.847+0000 7fae7a7fc700 1 -- 172.21.15.104:0/1841488744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74003a20 con 0x7fae7c070f50 2023-06-10T18:48:34.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 -- 172.21.15.104:0/1841488744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c070f50 msgr2=0x7fae7c0c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:34.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 --2- 172.21.15.104:0/1841488744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c070f50 0x7fae7c0c9a60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0cb910 tx=0x7fae7400d430 comp rx=0 tx=0).stop 2023-06-10T18:48:34.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 -- 172.21.15.104:0/1841488744 shutdown_connections 2023-06-10T18:48:34.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 --2- 172.21.15.104:0/1841488744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c070f50 0x7fae7c0c9a60 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 --2- 172.21.15.104:0/1841488744 >> v2:172.21.15.114:3300/0 conn(0x7fae7c0ce390 0x7fae7c070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 -- 172.21.15.104:0/1841488744 >> 172.21.15.104:0/1841488744 conn(0x7fae7c0ceec0 msgr2=0x7fae7c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 -- 172.21.15.104:0/1841488744 shutdown_connections 2023-06-10T18:48:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.848+0000 7fae82054700 1 -- 172.21.15.104:0/1841488744 wait complete. 2023-06-10T18:48:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.849+0000 7fae82054700 1 Processor -- start 2023-06-10T18:48:34.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.849+0000 7fae82054700 1 -- start start 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 0x7fae7c0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 0x7fae7c0c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1495b0 con 0x7fae7c0ce390 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c151f90 con 0x7fae7c070f50 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae82054700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1486e0 con 0x7fae7c1586a0 2023-06-10T18:48:34.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae7affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae7affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59352/0 (socket says 172.21.15.104:59352) 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae7affd700 1 -- 172.21.15.104:0/991774733 learned_addr learned my addr 172.21.15.104:0/991774733 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae7bfff700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 0x7fae7c0c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.850+0000 7fae7b7fe700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 0x7fae7c0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 -- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 msgr2=0x7fae7c0c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 0x7fae7c0c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 -- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 msgr2=0x7fae7c0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 0x7fae7c0b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 -- 172.21.15.104:0/991774733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7400bd40 con 0x7fae7c0ce390 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7b7fe700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 0x7fae7c0b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7bfff700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 0x7fae7c0c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae7affd700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fae7000b530 tx=0x7fae7000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:34.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70011830 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae700119b0 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c148930 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70011d60 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.851+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c148ef0 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.852+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c074050 con 0x7fae7c0ce390 2023-06-10T18:48:34.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.852+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fae7001a400 con 0x7fae7c0ce390 2023-06-10T18:48:34.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.853+0000 7fae63fff700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 0x7fae6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:34.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.853+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6030+0+0 (secure 0 0 0) 0x7fae70089520 con 0x7fae7c0ce390 2023-06-10T18:48:34.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.853+0000 7fae7b7fe700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 0x7fae6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:34.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.853+0000 7fae7b7fe700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 0x7fae6806d190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fae7c1f5690 tx=0x7fae6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:34.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:34.857+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fae700545f0 con 0x7fae7c0ce390 2023-06-10T18:48:35.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.042+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fae7c06e200 con 0x7fae7c0ce390 2023-06-10T18:48:35.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.710+0000 7fae63fff700 1 -- 172.21.15.104:0/991774733 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7fae70057c10 con 0x7fae7c0ce390 2023-06-10T18:48:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 msgr2=0x7fae6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 0x7fae6806d190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fae7c1f5690 tx=0x7fae6c008040 comp rx=0 tx=0).stop 2023-06-10T18:48:35.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 msgr2=0x7fae7c0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fae7000b530 tx=0x7fae7000b8f0 comp rx=0 tx=0).stop 2023-06-10T18:48:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 shutdown_connections 2023-06-10T18:48:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae6806ace0 0x7fae6806d190 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:35.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7c070f50 0x7fae7c0b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae7c1586a0 0x7fae7c0c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 --2- 172.21.15.104:0/991774733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0ce390 0x7fae7c0b20a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:35.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 >> 172.21.15.104:0/991774733 conn(0x7fae7c0ceec0 msgr2=0x7fae7c0c53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:35.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 shutdown_connections 2023-06-10T18:48:35.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.713+0000 7fae82054700 1 -- 172.21.15.104:0/991774733 wait complete. 2023-06-10T18:48:35.863 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:48:35.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:48:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.986+0000 7f41892cc700 1 Processor -- start 2023-06-10T18:48:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.986+0000 7f41892cc700 1 -- start start 2023-06-10T18:48:35.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f41892cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f418406e6e0 0x7f41840b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f41892cc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418406e250 con 0x7f418415c580 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f41892cc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418415c950 con 0x7f418406cde0 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f41892cc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840b1e80 con 0x7f418406e6e0 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f41827fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f418406e6e0 0x7f41840b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f4182ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f418415c580 0x7f41840b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57186/0 (socket says 172.21.15.104:57186) 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.987+0000 7f4182ffd700 1 -- 172.21.15.104:0/2563036064 learned_addr learned my addr 172.21.15.104:0/2563036064 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41827fc700 1 -- 172.21.15.104:0/2563036064 >> v1:172.21.15.114:6789/0 conn(0x7f418406cde0 legacy=0x7f41840ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41827fc700 1 -- 172.21.15.104:0/2563036064 >> v1:172.21.15.104:6789/0 conn(0x7f418415c580 legacy=0x7f41840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41827fc700 1 -- 172.21.15.104:0/2563036064 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418406d920 con 0x7f418406e6e0 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f4181ffb700 1 -- 172.21.15.104:0/2563036064 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368636504 0 0) 0x7f418406e250 con 0x7f418415c580 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41827fc700 1 --2- 172.21.15.104:0/2563036064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418406e6e0 0x7f41840b1780 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4174009840 tx=0x7f4174009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce73899accf9701 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f4181ffb700 1 -- 172.21.15.104:0/2563036064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174004230 con 0x7f418406e6e0 2023-06-10T18:48:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f4181ffb700 1 -- 172.21.15.104:0/2563036064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174004390 con 0x7f418406e6e0 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f4181ffb700 1 -- 172.21.15.104:0/2563036064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41740036a0 con 0x7f418406e6e0 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 -- 172.21.15.104:0/2563036064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418406e6e0 msgr2=0x7f41840b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 --2- 172.21.15.104:0/2563036064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418406e6e0 0x7f41840b1780 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4174009840 tx=0x7f4174009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 -- 172.21.15.104:0/2563036064 shutdown_connections 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 --2- 172.21.15.104:0/2563036064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418406e6e0 0x7f41840b1780 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 -- 172.21.15.104:0/2563036064 >> 172.21.15.104:0/2563036064 conn(0x7f418406b8a0 msgr2=0x7f41840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 -- 172.21.15.104:0/2563036064 shutdown_connections 2023-06-10T18:48:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.988+0000 7f41892cc700 1 -- 172.21.15.104:0/2563036064 wait complete. 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.990+0000 7f41892cc700 1 Processor -- start 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.990+0000 7f41892cc700 1 -- start start 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f418406cde0 0x7f41840af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f418406e6e0 0x7f4184157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840afc50 con 0x7f418415c580 2023-06-10T18:48:36.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840aff30 con 0x7f418406cde0 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41892cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b0210 con 0x7f418406e6e0 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f4182ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f418406cde0 0x7f41840af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59366/0 (socket says 172.21.15.104:59366) 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 -- 172.21.15.104:0/1437512320 learned_addr learned my addr 172.21.15.104:0/1437512320 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41827fc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f418406e6e0 0x7f4184157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 -- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f418406e6e0 msgr2=0x7f4184157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f418406e6e0 0x7f4184157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 -- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f418406cde0 msgr2=0x7f41840af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f418406cde0 0x7f41840af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 -- 172.21.15.104:0/1437512320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740094a0 con 0x7f418415c580 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.991+0000 7f41837fe700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f417c00cc10 tx=0x7f417c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.992+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c00aa60 con 0x7f418415c580 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.992+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41840b04f0 con 0x7f418415c580 2023-06-10T18:48:36.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.992+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41841576b0 con 0x7f418415c580 2023-06-10T18:48:36.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.992+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c00abc0 con 0x7f418415c580 2023-06-10T18:48:36.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.992+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c01c460 con 0x7f418415c580 2023-06-10T18:48:36.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:35.993+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41640052f0 con 0x7f418415c580 2023-06-10T18:48:36.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.001+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f417c012420 con 0x7f418415c580 2023-06-10T18:48:36.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.002+0000 7f416b7fe700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 0x7f417006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.002+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f417c089cc0 con 0x7f418415c580 2023-06-10T18:48:36.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.003+0000 7f4182ffd700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 0x7f417006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.004+0000 7f4182ffd700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 0x7f417006d190 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4178007fd0 tx=0x7f4178009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.013+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f417c054eb0 con 0x7f418415c580 2023-06-10T18:48:36.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.206+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4164005b20 con 0x7f418415c580 2023-06-10T18:48:36.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.209+0000 7f416b7fe700 1 -- 172.21.15.104:0/1437512320 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f417c0584d0 con 0x7f418415c580 2023-06-10T18:48:36.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 msgr2=0x7f417006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 0x7f417006d190 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4178007fd0 tx=0x7f4178009040 comp rx=0 tx=0).stop 2023-06-10T18:48:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 msgr2=0x7f418414e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f417c00cc10 tx=0x7f417c00cf20 comp rx=0 tx=0).stop 2023-06-10T18:48:36.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 shutdown_connections 2023-06-10T18:48:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f417006ace0 0x7f417006d190 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f418406cde0 0x7f41840af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f418415c580 0x7f418414e630 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 --2- 172.21.15.104:0/1437512320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f418406e6e0 0x7f4184157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.211+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 >> 172.21.15.104:0/1437512320 conn(0x7f418406b8a0 msgr2=0x7f4184159890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.212+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 shutdown_connections 2023-06-10T18:48:36.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.212+0000 7f41892cc700 1 -- 172.21.15.104:0/1437512320 wait complete. 2023-06-10T18:48:36.229 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic' 2023-06-10T18:48:36.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.344+0000 7f0fca819700 1 Processor -- start 2023-06-10T18:48:36.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.344+0000 7f0fca819700 1 -- start start 2023-06-10T18:48:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0fc406e0e0 0x7f0fc406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 0x7f0fc40c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc4071640 con 0x7f0fc406e0e0 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc40aef00 con 0x7f0fc406e4b0 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fca819700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc406d290 con 0x7f0fc40ca100 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0fc406e0e0 0x7f0fc406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0fc406e0e0 0x7f0fc406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59372/0 (socket says 172.21.15.104:59372) 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 -- 172.21.15.104:0/3400720163 learned_addr learned my addr 172.21.15.104:0/3400720163 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:36.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc37fe700 1 --2- 172.21.15.104:0/3400720163 >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 0x7f0fc40c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc8db6700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 -- 172.21.15.104:0/3400720163 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 msgr2=0x7f0fc40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 -- 172.21.15.104:0/3400720163 >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 msgr2=0x7f0fc40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/3400720163 >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 0x7f0fc40c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 -- 172.21.15.104:0/3400720163 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc406d4c0 con 0x7f0fc406e0e0 2023-06-10T18:48:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc37fe700 1 --2- 172.21.15.104:0/3400720163 >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 0x7f0fc40c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:36.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e0e0 0x7f0fc406cd50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4070320 tx=0x7f0fb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10169dff39637add server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.345+0000 7f0fc2ffd700 1 -- 172.21.15.104:0/3400720163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4005e30 con 0x7f0fc406e0e0 2023-06-10T18:48:36.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fc2ffd700 1 -- 172.21.15.104:0/3400720163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb4004030 con 0x7f0fc406e0e0 2023-06-10T18:48:36.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fc2ffd700 1 -- 172.21.15.104:0/3400720163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb40043e0 con 0x7f0fc406e0e0 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 -- 172.21.15.104:0/3400720163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e0e0 msgr2=0x7f0fc406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e0e0 0x7f0fc406cd50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4070320 tx=0x7f0fb4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 -- 172.21.15.104:0/3400720163 shutdown_connections 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e0e0 0x7f0fc406cd50 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 --2- 172.21.15.104:0/3400720163 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 --2- 172.21.15.104:0/3400720163 >> v2:172.21.15.114:3300/0 conn(0x7f0fc406e4b0 0x7f0fc40c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 -- 172.21.15.104:0/3400720163 >> 172.21.15.104:0/3400720163 conn(0x7f0fc406b8a0 msgr2=0x7f0fc40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 -- 172.21.15.104:0/3400720163 shutdown_connections 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.346+0000 7f0fca819700 1 -- 172.21.15.104:0/3400720163 wait complete. 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.347+0000 7f0fca819700 1 Processor -- start 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 -- start start 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc406e0e0 0x7f0fc40b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40b3e80 con 0x7f0fc406e4b0 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40b0b10 con 0x7f0fc406e0e0 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fca819700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40b0db0 con 0x7f0fc40ca100 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fc37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fc37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59380/0 (socket says 172.21.15.104:59380) 2023-06-10T18:48:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fc37fe700 1 -- 172.21.15.104:0/888882864 learned_addr learned my addr 172.21.15.104:0/888882864 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc406e0e0 0x7f0fc40b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.348+0000 7f0fc8db6700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 -- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 msgr2=0x7f0fc40c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 -- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc406e0e0 msgr2=0x7f0fc40b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc406e0e0 0x7f0fc40b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 -- 172.21.15.104:0/888882864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4009500 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc37fe700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800d800 tx=0x7f0fb800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8004030 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8004190 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc40b1090 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8004520 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.349+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc40b1650 con 0x7f0fc406e4b0 2023-06-10T18:48:36.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.350+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc4074050 con 0x7f0fc406e4b0 2023-06-10T18:48:36.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.350+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0fb80036a0 con 0x7f0fc406e4b0 2023-06-10T18:48:36.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.351+0000 7f0fc0ff9700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 0x7f0fb006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.351+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f0fb8088ae0 con 0x7f0fc406e4b0 2023-06-10T18:48:36.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.351+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 0x7f0fb006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.352+0000 7f0fc3fff700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 0x7f0fb006d190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4070320 tx=0x7f0fb4008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.355+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0fb8057890 con 0x7f0fc406e4b0 2023-06-10T18:48:36.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.618+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic"]} v 0) v1 -- 0x7f0fc40ca720 con 0x7f0fc406e4b0 2023-06-10T18:48:36.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.712+0000 7f0fc0ff9700 1 -- 172.21.15.104:0/888882864 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_basic"]}]=0 v229) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0fb8019020 con 0x7f0fc406e4b0 2023-06-10T18:48:36.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 msgr2=0x7f0fb006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 0x7f0fb006d190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4070320 tx=0x7f0fb4008070 comp rx=0 tx=0).stop 2023-06-10T18:48:36.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 msgr2=0x7f0fc41f1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800d800 tx=0x7f0fb800db10 comp rx=0 tx=0).stop 2023-06-10T18:48:36.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 shutdown_connections 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0fb006ace0 0x7f0fb006d190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc406e0e0 0x7f0fc40b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0fc40ca100 0x7f0fc40c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 --2- 172.21.15.104:0/888882864 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fc406e4b0 0x7f0fc41f1840 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 >> 172.21.15.104:0/888882864 conn(0x7f0fc406b8a0 msgr2=0x7f0fc415aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:36.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 shutdown_connections 2023-06-10T18:48:36.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.714+0000 7f0fca819700 1 -- 172.21.15.104:0/888882864 wait complete. 2023-06-10T18:48:36.732 INFO:tasks.cephfs_test_runner:test_basic (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:48:36.732 INFO:tasks.cephfs_test_runner:Starting test: test_config_session_timeout (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:48:36.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout' 2023-06-10T18:48:36.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 Processor -- start 2023-06-10T18:48:36.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 -- start start 2023-06-10T18:48:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f274015a230 0x7f27400ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f274006dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400cc050 con 0x7f2740069e70 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274006e290 con 0x7f274015a230 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.847+0000 7f2745e66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27400cba00 con 0x7f274006d810 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f274015a230 0x7f27400ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f274015a230 0x7f27400ac2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59386/0 (socket says 172.21.15.104:59386) 2023-06-10T18:48:36.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2740069e70 0x7f27400cc4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57216/0 (socket says 172.21.15.104:57216) 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 -- 172.21.15.104:0/3888017189 learned_addr learned my addr 172.21.15.104:0/3888017189 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 -- 172.21.15.104:0/3888017189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 msgr2=0x7f274006dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 --2- 172.21.15.104:0/3888017189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f274006dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 -- 172.21.15.104:0/3888017189 >> v1:172.21.15.104:6789/0 conn(0x7f2740069e70 legacy=0x7f27400cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 -- 172.21.15.104:0/3888017189 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400cbce0 con 0x7f274015a230 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273e7fc700 1 -- 172.21.15.104:0/3888017189 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551404437 0 0) 0x7f27400cc050 con 0x7f2740069e70 2023-06-10T18:48:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273f7fe700 1 --2- 172.21.15.104:0/3888017189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27400ac2c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2730006fd0 tx=0x7f2730009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04fa6ea59735364 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273e7fc700 1 -- 172.21.15.104:0/3888017189 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730005e30 con 0x7f274015a230 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273e7fc700 1 -- 172.21.15.104:0/3888017189 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730004030 con 0x7f274015a230 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.848+0000 7f273e7fc700 1 -- 172.21.15.104:0/3888017189 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27300043e0 con 0x7f274015a230 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 -- 172.21.15.104:0/3888017189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 msgr2=0x7f27400ac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 --2- 172.21.15.104:0/3888017189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27400ac2c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2730006fd0 tx=0x7f2730009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 -- 172.21.15.104:0/3888017189 shutdown_connections 2023-06-10T18:48:36.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 --2- 172.21.15.104:0/3888017189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27400ac2c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 --2- 172.21.15.104:0/3888017189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f274006dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 -- 172.21.15.104:0/3888017189 >> 172.21.15.104:0/3888017189 conn(0x7f274006b8a0 msgr2=0x7f27400cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 -- 172.21.15.104:0/3888017189 shutdown_connections 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.849+0000 7f2745e66700 1 -- 172.21.15.104:0/3888017189 wait complete. 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.850+0000 7f2745e66700 1 Processor -- start 2023-06-10T18:48:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.850+0000 7f2745e66700 1 -- start start 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2740069e70 0x7f27401df5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f2740157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27401e0df0 con 0x7f274015a230 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27401dfb90 con 0x7f2740069e70 2023-06-10T18:48:36.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27401dfe70 con 0x7f274006d810 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59396/0 (socket says 172.21.15.104:59396) 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 -- 172.21.15.104:0/3135524909 learned_addr learned my addr 172.21.15.104:0/3135524909 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273effd700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f2740157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273f7fe700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2740069e70 0x7f27401df5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 -- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 msgr2=0x7f2740157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f2740157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 -- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2740069e70 msgr2=0x7f27401df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2740069e70 0x7f27401df5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 -- 172.21.15.104:0/3135524909 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730009500 con 0x7f274015a230 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f273ffff700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2738002230 tx=0x7f273800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273800ac20 con 0x7f274015a230 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273800ada0 con 0x7f274015a230 2023-06-10T18:48:36.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27401e0150 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738007310 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.851+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27401e0430 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.852+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27401f5160 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.853+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2738007470 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.853+0000 7f2727fff700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 0x7f272c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.853+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f27380891b0 con 0x7f274015a230 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.853+0000 7f273f7fe700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 0x7f272c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:36.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.854+0000 7f273f7fe700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 0x7f272c06d190 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f2730008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:36.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:36.858+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f27401f5160 con 0x7f274015a230 2023-06-10T18:48:37.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.118+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout"]} v 0) v1 -- 0x7f27401f5160 con 0x7f274015a230 2023-06-10T18:48:37.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.718+0000 7f2727fff700 1 -- 172.21.15.104:0/3135524909 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout"]}]=0 v230) v1 ==== 162+0+0 (secure 0 0 0) 0x7f27401f5160 con 0x7f274015a230 2023-06-10T18:48:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 msgr2=0x7f272c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 0x7f272c06d190 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f2730008070 comp rx=0 tx=0).stop 2023-06-10T18:48:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 msgr2=0x7f27401e08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2738002230 tx=0x7f273800a060 comp rx=0 tx=0).stop 2023-06-10T18:48:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 shutdown_connections 2023-06-10T18:48:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f272c06ace0 0x7f272c06d190 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2740069e70 0x7f27401df5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f274015a230 0x7f27401e08b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 --2- 172.21.15.104:0/3135524909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f274006d810 0x7f2740157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 >> 172.21.15.104:0/3135524909 conn(0x7f274006b8a0 msgr2=0x7f27400ad590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:37.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 shutdown_connections 2023-06-10T18:48:37.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.721+0000 7f2745e66700 1 -- 172.21.15.104:0/3135524909 wait complete. 2023-06-10T18:48:37.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:48:37.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.865+0000 7f9498f79700 1 Processor -- start 2023-06-10T18:48:37.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.865+0000 7f9498f79700 1 -- start start 2023-06-10T18:48:37.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9498f79700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f949415c730 0x7f94940cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9498f79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94940cd720 0x7f949406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9498f79700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949415cb00 con 0x7f949406d2b0 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9498f79700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94940cdbc0 con 0x7f94940cd720 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9498f79700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94940acd90 con 0x7f949415c730 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94940cd720 0x7f949406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94940cd720 0x7f949406cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59400/0 (socket says 172.21.15.104:59400) 2023-06-10T18:48:37.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 -- 172.21.15.104:0/2945331906 learned_addr learned my addr 172.21.15.104:0/2945331906 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f949259c700 1 --2- 172.21.15.104:0/2945331906 >> v2:172.21.15.114:3300/0 conn(0x7f949415c730 0x7f94940cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 -- 172.21.15.104:0/2945331906 >> v2:172.21.15.114:3300/0 conn(0x7f949415c730 msgr2=0x7f94940cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 --2- 172.21.15.104:0/2945331906 >> v2:172.21.15.114:3300/0 conn(0x7f949415c730 0x7f94940cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 -- 172.21.15.104:0/2945331906 >> v1:172.21.15.114:6789/0 conn(0x7f949406d2b0 legacy=0x7f949406d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 -- 172.21.15.104:0/2945331906 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940ac460 con 0x7f94940cd720 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f9491d9b700 1 --2- 172.21.15.104:0/2945331906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94940cd720 0x7f949406cd70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f94880098f0 tx=0x7f9488009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad7ca722efa8061 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f949159a700 1 -- 172.21.15.104:0/2945331906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011830 con 0x7f94940cd720 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f949159a700 1 -- 172.21.15.104:0/2945331906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9488011990 con 0x7f94940cd720 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.866+0000 7f949159a700 1 -- 172.21.15.104:0/2945331906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488011d20 con 0x7f94940cd720 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 -- 172.21.15.104:0/2945331906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94940cd720 msgr2=0x7f949406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 --2- 172.21.15.104:0/2945331906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94940cd720 0x7f949406cd70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f94880098f0 tx=0x7f9488009c00 comp rx=0 tx=0).stop 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 -- 172.21.15.104:0/2945331906 shutdown_connections 2023-06-10T18:48:37.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 --2- 172.21.15.104:0/2945331906 >> v2:172.21.15.114:3300/0 conn(0x7f949415c730 0x7f94940cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 --2- 172.21.15.104:0/2945331906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94940cd720 0x7f949406cd70 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 -- 172.21.15.104:0/2945331906 >> 172.21.15.104:0/2945331906 conn(0x7f949406b8a0 msgr2=0x7f94940cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:37.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 -- 172.21.15.104:0/2945331906 shutdown_connections 2023-06-10T18:48:37.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.867+0000 7f9498f79700 1 -- 172.21.15.104:0/2945331906 wait complete. 2023-06-10T18:48:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.868+0000 7f9498f79700 1 Processor -- start 2023-06-10T18:48:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.868+0000 7f9498f79700 1 -- start start 2023-06-10T18:48:37.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 0x7f9494150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 0x7f94940b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9494151190 con 0x7f949406d2b0 2023-06-10T18:48:37.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940b0d40 con 0x7f94940cd720 2023-06-10T18:48:37.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f9498f79700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94940b1020 con 0x7f949415c730 2023-06-10T18:48:37.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f949259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f949259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59406/0 (socket says 172.21.15.104:59406) 2023-06-10T18:48:37.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.869+0000 7f949259c700 1 -- 172.21.15.104:0/1010093609 learned_addr learned my addr 172.21.15.104:0/1010093609 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:37.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9491d9b700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 0x7f9494150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9492d9d700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 0x7f94940b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 -- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 msgr2=0x7f94940b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 0x7f94940b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 -- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 msgr2=0x7f9494150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:37.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 0x7f9494150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:37.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 -- 172.21.15.104:0/1010093609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009550 con 0x7f949406d2b0 2023-06-10T18:48:37.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9491d9b700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 0x7f9494150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:37.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9492d9d700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 0x7f94940b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:37.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f949259c700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f948400e900 tx=0x7f948400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:37.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484009650 con 0x7f949406d2b0 2023-06-10T18:48:37.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948400c850 con 0x7f949406d2b0 2023-06-10T18:48:37.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948400cbc0 con 0x7f949406d2b0 2023-06-10T18:48:37.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94940b1300 con 0x7f949406d2b0 2023-06-10T18:48:37.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.870+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94940b0960 con 0x7f949406d2b0 2023-06-10T18:48:37.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.871+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94940b1950 con 0x7f949406d2b0 2023-06-10T18:48:37.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.871+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f948400cd20 con 0x7f949406d2b0 2023-06-10T18:48:37.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.872+0000 7f9482ffd700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 0x7f947406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:37.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.872+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f94840896b0 con 0x7f949406d2b0 2023-06-10T18:48:37.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.872+0000 7f9491d9b700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 0x7f947406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:37.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.872+0000 7f9491d9b700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 0x7f947406d190 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f94940c2ee0 tx=0x7f9488008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:37.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:37.876+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94840549b0 con 0x7f949406d2b0 2023-06-10T18:48:38.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.063+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9494074050 con 0x7f949406d2b0 2023-06-10T18:48:38.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.067+0000 7f9482ffd700 1 -- 172.21.15.104:0/1010093609 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9484057fd0 con 0x7f949406d2b0 2023-06-10T18:48:38.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 msgr2=0x7f947406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 0x7f947406d190 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f94940c2ee0 tx=0x7f9488008030 comp rx=0 tx=0).stop 2023-06-10T18:48:38.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 msgr2=0x7f94940b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f948400e900 tx=0x7f948400ecc0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 shutdown_connections 2023-06-10T18:48:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f949415c730 0x7f94940b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f949406d2b0 0x7f94940b1b00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f947406ace0 0x7f947406d190 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 --2- 172.21.15.104:0/1010093609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94940cd720 0x7f9494150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 >> 172.21.15.104:0/1010093609 conn(0x7f949406b8a0 msgr2=0x7f94940c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 shutdown_connections 2023-06-10T18:48:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.069+0000 7f9498f79700 1 -- 172.21.15.104:0/1010093609 wait complete. 2023-06-10T18:48:38.086 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:48:38.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:48:38.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:48:38.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:48:38.185 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:48:38.186 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:38.186 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:38.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:48:38.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:48:38.205 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:48:38.205 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:38.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:48:38.252 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:48:38.253 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:38.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:48:38.301 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:48:38.302 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:48:38.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:48:38.342 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:48:38.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.462+0000 7f22943fd700 1 Processor -- start 2023-06-10T18:48:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.462+0000 7f22943fd700 1 -- start start 2023-06-10T18:48:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f22943fd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228c0b1e30 0x7f228c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f22943fd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c06b9f0 0x7f228c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f22943fd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c06bfd0 con 0x7f228c0d0a30 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f22943fd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0d0ed0 con 0x7f228c0b1e30 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f22943fd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c06c2b0 con 0x7f228c06b9f0 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228c0b1e30 0x7f228c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228c0b1e30 0x7f228c0d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59414/0 (socket says 172.21.15.104:59414) 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 -- 172.21.15.104:0/3946146601 learned_addr learned my addr 172.21.15.104:0/3946146601 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 -- 172.21.15.104:0/3946146601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c06b9f0 msgr2=0x7f228c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 --2- 172.21.15.104:0/3946146601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c06b9f0 0x7f228c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 -- 172.21.15.104:0/3946146601 >> v1:172.21.15.114:6789/0 conn(0x7f228c0d0a30 legacy=0x7f228c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 -- 172.21.15.104:0/3946146601 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0ccf70 con 0x7f228c0b1e30 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2292199700 1 --2- 172.21.15.104:0/3946146601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c0b1e30 0x7f228c0d04f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f22880098f0 tx=0x7f2288009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd419704c54005c4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.463+0000 7f2291197700 1 -- 172.21.15.104:0/3946146601 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288004230 con 0x7f228c0b1e30 2023-06-10T18:48:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f2291197700 1 -- 172.21.15.104:0/3946146601 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288004390 con 0x7f228c0b1e30 2023-06-10T18:48:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f2291197700 1 -- 172.21.15.104:0/3946146601 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22880036a0 con 0x7f228c0b1e30 2023-06-10T18:48:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 -- 172.21.15.104:0/3946146601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c0b1e30 msgr2=0x7f228c0d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 --2- 172.21.15.104:0/3946146601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c0b1e30 0x7f228c0d04f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f22880098f0 tx=0x7f2288009c00 comp rx=0 tx=0).stop 2023-06-10T18:48:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 -- 172.21.15.104:0/3946146601 shutdown_connections 2023-06-10T18:48:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 --2- 172.21.15.104:0/3946146601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c0b1e30 0x7f228c0d04f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 --2- 172.21.15.104:0/3946146601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c06b9f0 0x7f228c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 -- 172.21.15.104:0/3946146601 >> 172.21.15.104:0/3946146601 conn(0x7f228c071ec0 msgr2=0x7f228c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 -- 172.21.15.104:0/3946146601 shutdown_connections 2023-06-10T18:48:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.464+0000 7f22943fd700 1 -- 172.21.15.104:0/3946146601 wait complete. 2023-06-10T18:48:38.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.465+0000 7f22943fd700 1 Processor -- start 2023-06-10T18:48:38.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.465+0000 7f22943fd700 1 -- start start 2023-06-10T18:48:38.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228c0b1e30 0x7f228c0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c0d0a30 0x7f228c0cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0b7680 con 0x7f228c06b9f0 2023-06-10T18:48:38.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0cc300 con 0x7f228c0b1e30 2023-06-10T18:48:38.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f22943fd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0cbd40 con 0x7f228c0d0a30 2023-06-10T18:48:38.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59422/0 (socket says 172.21.15.104:59422) 2023-06-10T18:48:38.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 -- 172.21.15.104:0/2088765298 learned_addr learned my addr 172.21.15.104:0/2088765298 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:38.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f229299a700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c0d0a30 0x7f228c0cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2291998700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228c0b1e30 0x7f228c0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 -- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c0d0a30 msgr2=0x7f228c0cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c0d0a30 0x7f228c0cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 -- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228c0b1e30 msgr2=0x7f228c0b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228c0b1e30 0x7f228c0b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.466+0000 7f2292199700 1 -- 172.21.15.104:0/2088765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288009550 con 0x7f228c06b9f0 2023-06-10T18:48:38.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f2292199700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f228c0b5510 tx=0x7f2288004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288003e70 con 0x7f228c06b9f0 2023-06-10T18:48:38.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228801f070 con 0x7f228c06b9f0 2023-06-10T18:48:38.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c0cc020 con 0x7f228c06b9f0 2023-06-10T18:48:38.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228800b8c0 con 0x7f228c06b9f0 2023-06-10T18:48:38.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.467+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c0b6c20 con 0x7f228c06b9f0 2023-06-10T18:48:38.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.468+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c0d1860 con 0x7f228c06b9f0 2023-06-10T18:48:38.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.468+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2288021030 con 0x7f228c06b9f0 2023-06-10T18:48:38.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.469+0000 7f2282ffd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.469+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f228808a960 con 0x7f228c06b9f0 2023-06-10T18:48:38.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.469+0000 7f2291998700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.469+0000 7f2291998700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f228c0a0790 tx=0x7f227c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.473+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2288055d80 con 0x7f228c06b9f0 2023-06-10T18:48:38.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.698+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228c0d1860 con 0x7f228c06b9f0 2023-06-10T18:48:38.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.699+0000 7f2282ffd700 1 -- 172.21.15.104:0/2088765298 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f22880593a0 con 0x7f228c06b9f0 2023-06-10T18:48:38.705 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:38.705 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5516,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2708374731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2708374731},{"type":"v1","addr":"172.21.15.114:6836","nonce":2708374731}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6174,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1236194386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1236194386},{"type":"v1","addr":"172.21.15.104:6835","nonce":1236194386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6177,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1092508684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1092508684},{"type":"v1","addr":"172.21.15.104:6834","nonce":1092508684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3553086704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3553086704},{"type":"v1","addr":"172.21.15.114:6837","nonce":3553086704}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[]} 2023-06-10T18:48:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 msgr2=0x7f227006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f228c0a0790 tx=0x7f227c008040 comp rx=0 tx=0).stop 2023-06-10T18:48:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 msgr2=0x7f228c15a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f228c0b5510 tx=0x7f2288004750 comp rx=0 tx=0).stop 2023-06-10T18:48:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 shutdown_connections 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f228c0d0a30 0x7f228c0cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228c06b9f0 0x7f228c15a1c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 --2- 172.21.15.104:0/2088765298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228c0b1e30 0x7f228c0b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 >> 172.21.15.104:0/2088765298 conn(0x7f228c071ec0 msgr2=0x7f228c0b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 shutdown_connections 2023-06-10T18:48:38.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.701+0000 7f22943fd700 1 -- 172.21.15.104:0/2088765298 wait complete. 2023-06-10T18:48:38.709 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 27 2023-06-10T18:48:38.719 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:48:38.719 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:48:38.719 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:48:38.719 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:48:38.720 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:48:38.720 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:48:38.720 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:48:38.720 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:48:38.721 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:48:38.721 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:48:38.721 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:48:38.721 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:48:38.721 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:48:38.722 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:48:38.722 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:48:38.722 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:48:38.729 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:48:38.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:48:38.730 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:48:38.730 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:48:38.732 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:48:38.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:48:38.733 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:48:38.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:48:38.771 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:48:38.772 DEBUG:teuthology.parallel:result is None 2023-06-10T18:48:38.773 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:48:38.773 DEBUG:teuthology.parallel:result is None 2023-06-10T18:48:38.775 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:48:38.775 DEBUG:teuthology.parallel:result is None 2023-06-10T18:48:38.776 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:48:38.777 DEBUG:teuthology.parallel:result is None 2023-06-10T18:48:38.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:48:38.849 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:48:38.850 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:48:38.860 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:48:38.860 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:48:38.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.902+0000 7f8c7bb29700 1 Processor -- start 2023-06-10T18:48:38.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.903+0000 7f8c7bb29700 1 -- start start 2023-06-10T18:48:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.903+0000 7f8c7bb29700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c740aec00 0x7f8c7406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7bb29700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7406e5e0 con 0x7f8c740b1070 2023-06-10T18:48:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7bb29700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740ae7e0 con 0x7f8c7406ea70 2023-06-10T18:48:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7bb29700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7406df90 con 0x7f8c740aec00 2023-06-10T18:48:38.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c798c5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8c740b1070 0x7f8c740b1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57284/0 (socket says 172.21.15.104:57284) 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c798c5700 1 -- 172.21.15.104:0/3649199933 learned_addr learned my addr 172.21.15.104:0/3649199933 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7a0c6700 1 --2- 172.21.15.104:0/3649199933 >> v2:172.21.15.114:3300/0 conn(0x7f8c740aec00 0x7f8c7406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994635520 0 0) 0x7f8c7406e5e0 con 0x7f8c740b1070 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c600035b0 con 0x7f8c740b1070 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/3649199933 >> v1:172.21.15.114:6789/0 conn(0x7f8c7406ea70 legacy=0x7f8c7406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2461004124 0 0) 0x7f8c600035b0 con 0x7f8c740b1070 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/3649199933 >> v1:172.21.15.104:6789/0 conn(0x7f8c740b1070 legacy=0x7f8c740b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/3649199933 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7406e270 con 0x7f8c740aec00 2023-06-10T18:48:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c7a0c6700 1 --2- 172.21.15.104:0/3649199933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c740aec00 0x7f8c7406d4f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009d90 tx=0x7f8c6c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e3981d40a105a1e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00e070 con 0x7f8c740aec00 2023-06-10T18:48:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c003a20 con 0x7f8c740aec00 2023-06-10T18:48:38.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.904+0000 7f8c788c3700 1 -- 172.21.15.104:0/3649199933 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640017f0 con 0x7f8c740aec00 2023-06-10T18:48:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 -- 172.21.15.104:0/3649199933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c740aec00 msgr2=0x7f8c7406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/3649199933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c740aec00 0x7f8c7406d4f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009d90 tx=0x7f8c6c00b330 comp rx=0 tx=0).stop 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 -- 172.21.15.104:0/3649199933 shutdown_connections 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/3649199933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c740aec00 0x7f8c7406d4f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 -- 172.21.15.104:0/3649199933 >> 172.21.15.104:0/3649199933 conn(0x7f8c7406b8a0 msgr2=0x7f8c740cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 -- 172.21.15.104:0/3649199933 shutdown_connections 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.905+0000 7f8c7bb29700 1 -- 172.21.15.104:0/3649199933 wait complete. 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.906+0000 7f8c7bb29700 1 Processor -- start 2023-06-10T18:48:38.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.906+0000 7f8c7bb29700 1 -- start start 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c7406ea70 0x7f8c741ee400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c740aec00 0x7f8c741514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741ee940 con 0x7f8c740b1070 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74151d10 con 0x7f8c7406ea70 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7bb29700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74153130 con 0x7f8c740aec00 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59466/0 (socket says 172.21.15.104:59466) 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/1850578357 learned_addr learned my addr 172.21.15.104:0/1850578357 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c798c5700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c7406ea70 0x7f8c741ee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c790c4700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c740aec00 0x7f8c741514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c740aec00 msgr2=0x7f8c741514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c740aec00 0x7f8c741514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c7406ea70 msgr2=0x7f8c741ee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:38.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c7406ea70 0x7f8c741ee400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.907+0000 7f8c7a0c6700 1 -- 172.21.15.104:0/1850578357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c009a70 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c7a0c6700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c004000 tx=0x7f8c6c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c004540 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c0046a0 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c741533b0 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c012450 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.908+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c74153970 con 0x7f8c740b1070 2023-06-10T18:48:38.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.909+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c740ae4d0 con 0x7f8c740b1070 2023-06-10T18:48:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.909+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c6c0125b0 con 0x7f8c740b1070 2023-06-10T18:48:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.909+0000 7f8c6a7fc700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 0x7f8c5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:38.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.910+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f8c6c08aab0 con 0x7f8c740b1070 2023-06-10T18:48:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.910+0000 7f8c798c5700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 0x7f8c5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.910+0000 7f8c798c5700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 0x7f8c5806d190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000ef0 tx=0x7f8c64001290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:38.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:38.914+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c6c055db0 con 0x7f8c740b1070 2023-06-10T18:48:39.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.117+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8c74074050 con 0x7f8c740b1070 2023-06-10T18:48:39.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.895+0000 7f8c6a7fc700 1 -- 172.21.15.104:0/1850578357 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8c6c0593d0 con 0x7f8c740b1070 2023-06-10T18:48:39.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 msgr2=0x7f8c5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:39.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 0x7f8c5806d190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000ef0 tx=0x7f8c64001290 comp rx=0 tx=0).stop 2023-06-10T18:48:39.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 msgr2=0x7f8c74154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c004000 tx=0x7f8c6c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 shutdown_connections 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5806ace0 0x7f8c5806d190 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c7406ea70 0x7f8c741ee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c740b1070 0x7f8c74154170 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:39.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 --2- 172.21.15.104:0/1850578357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c740aec00 0x7f8c741514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:39.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 >> 172.21.15.104:0/1850578357 conn(0x7f8c7406b8a0 msgr2=0x7f8c740ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:39.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 shutdown_connections 2023-06-10T18:48:39.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:39.897+0000 7f8c7bb29700 1 -- 172.21.15.104:0/1850578357 wait complete. 2023-06-10T18:48:39.905 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:48:39.917 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:48:40.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.036+0000 7fc9e041a700 1 Processor -- start 2023-06-10T18:48:40.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.036+0000 7fc9e041a700 1 -- start start 2023-06-10T18:48:40.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9e041a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9d80cd850 0x7fc9d806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9e041a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9d80cb300 0x7fc9d806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9e041a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80c9e60 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9e041a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80c9bc0 con 0x7fc9d80cb300 2023-06-10T18:48:40.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9e041a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80aee70 con 0x7fc9d80cd850 2023-06-10T18:48:40.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9de1b6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc9d80aeaa0 0x7fc9d80cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57302/0 (socket says 172.21.15.104:57302) 2023-06-10T18:48:40.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9de1b6700 1 -- 172.21.15.104:0/321057456 learned_addr learned my addr 172.21.15.104:0/321057456 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:40.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698474292 0 0) 0x7fc9d80c9e60 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c40035b0 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/321057456 >> v2:172.21.15.114:3300/0 conn(0x7fc9d80cb300 0x7fc9d806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.037+0000 7fc9dd9b5700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9d80cd850 0x7fc9d806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 509176550 0 0) 0x7fc9c40035b0 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9d80cd850 msgr2=0x7fc9d806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9d80cd850 0x7fc9d806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 >> v2:172.21.15.114:3300/0 conn(0x7fc9d80cb300 msgr2=0x7fc9d806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 --2- 172.21.15.104:0/321057456 >> v2:172.21.15.114:3300/0 conn(0x7fc9d80cb300 0x7fc9d806d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d806e540 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc9c80017f0 con 0x7fc9d80aeaa0 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 >> v1:172.21.15.104:6789/0 conn(0x7fc9d80aeaa0 legacy=0x7fc9d80cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c4003a00 0x7fc9c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4004850 0x7fc9c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c4005200 0x7fc9c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80c9bc0 con 0x7fc9c4003a00 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80aee70 con 0x7fc9c4005200 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c400bfd0 con 0x7fc9c4004850 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/321057456 >> v2:172.21.15.114:3300/0 conn(0x7fc9d80cb300 0x7fc9d806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd9b5700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9d80cd850 0x7fc9d806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:40.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c4003a00 0x7fc9c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd9b5700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4004850 0x7fc9c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c4005200 0x7fc9c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 -- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4004850 msgr2=0x7fc9c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4004850 0x7fc9c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 -- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c4005200 msgr2=0x7fc9c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c4005200 0x7fc9c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 -- 172.21.15.104:0/321057456 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40035b0 con 0x7fc9c4003a00 2023-06-10T18:48:40.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c4005200 0x7fc9c400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:40.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c4003a00 0x7fc9c4003e10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c2f0 tx=0x7fc9c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f926f34573c2b44b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800ce30 con 0x7fc9c4003a00 2023-06-10T18:48:40.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c80070d0 con 0x7fc9c4003a00 2023-06-10T18:48:40.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.038+0000 7fc9dd1b4700 1 -- 172.21.15.104:0/321057456 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8007480 con 0x7fc9c4003a00 2023-06-10T18:48:40.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 -- 172.21.15.104:0/321057456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c4003a00 msgr2=0x7fc9c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 --2- 172.21.15.104:0/321057456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9c4003a00 0x7fc9c4003e10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c2f0 tx=0x7fc9c4005880 comp rx=0 tx=0).stop 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9de9b7700 1 -- 172.21.15.104:0/321057456 reap_dead start 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 -- 172.21.15.104:0/321057456 shutdown_connections 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 -- 172.21.15.104:0/321057456 >> 172.21.15.104:0/321057456 conn(0x7fc9d806b8a0 msgr2=0x7fc9d80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 -- 172.21.15.104:0/321057456 shutdown_connections 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.039+0000 7fc9e041a700 1 -- 172.21.15.104:0/321057456 wait complete. 2023-06-10T18:48:40.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.040+0000 7fc9e041a700 1 Processor -- start 2023-06-10T18:48:40.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.040+0000 7fc9e041a700 1 -- start start 2023-06-10T18:48:40.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4003a00 0x7fc9d81ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9d80b41a0 0x7fc9d81eab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80c5f10 con 0x7fc9d81f0da0 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d81f1750 con 0x7fc9d80b41a0 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80b5300 con 0x7fc9c4003a00 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59490/0 (socket says 172.21.15.104:59490) 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 -- 172.21.15.104:0/3615395679 learned_addr learned my addr 172.21.15.104:0/3615395679 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9dd9b5700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9d80b41a0 0x7fc9d81eab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 -- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4003a00 msgr2=0x7fc9d81ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4003a00 0x7fc9d81ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 -- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9d80b41a0 msgr2=0x7fc9d81eab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9d80b41a0 0x7fc9d81eab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 -- 172.21.15.104:0/3615395679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d81ee770 con 0x7fc9d81f0da0 2023-06-10T18:48:40.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9de9b7700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00045e0 tx=0x7fc9d00049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0007ad0 con 0x7fc9d81f0da0 2023-06-10T18:48:40.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0007c30 con 0x7fc9d81f0da0 2023-06-10T18:48:40.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0014400 con 0x7fc9d81f0da0 2023-06-10T18:48:40.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.041+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d81f1b70 con 0x7fc9d81f0da0 2023-06-10T18:48:40.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.042+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d80b49d0 con 0x7fc9d81f0da0 2023-06-10T18:48:40.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.042+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d8074050 con 0x7fc9d81f0da0 2023-06-10T18:48:40.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.043+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc9d001e440 con 0x7fc9d81f0da0 2023-06-10T18:48:40.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.043+0000 7fc9ceffd700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 0x7fc9bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.043+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc9d0088d50 con 0x7fc9d81f0da0 2023-06-10T18:48:40.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.043+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 0x7fc9bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.044+0000 7fc9de1b6700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 0x7fc9bc06d190 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000ef0 tx=0x7fc9c800f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.048+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc9d008c030 con 0x7fc9d81f0da0 2023-06-10T18:48:40.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.310+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc9d80b1060 con 0x7fc9d81f0da0 2023-06-10T18:48:40.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.311+0000 7fc9ceffd700 1 -- 172.21.15.104:0/3615395679 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc9d80b1060 con 0x7fc9d81f0da0 2023-06-10T18:48:40.317 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:40.318 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:40.319 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:48:40.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.320 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:48:40.320 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:48:40.321 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.321 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.321 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:48:40.321 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:48:40.322 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.323 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:48:40.323 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:48:40.323 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.323 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:48:40.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:48:40.324 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.324 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.325 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:48:40.325 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:48:40.325 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.325 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:48:40.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:48:40.326 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.327 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:48:40.327 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:48:40.327 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:48:40.330 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:48:40.330 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.330 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.331 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.333 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:48:40.333 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:48:40.333 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.334 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.335 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.336 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:48:40.336 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:48:40.336 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.337 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.337 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.337 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.339 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:48:40.339 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:48:40.339 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.340 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.341 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:48:40.341 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:48:40.341 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.341 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.341 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:48:40.342 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:48:40.343 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.343 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:48:40.344 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.345 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:48:40.348 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.350 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:48:40.351 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:48:40.352 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:48:40.352 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.353 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:48:40.353 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:48:40.353 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.353 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:48:40.354 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:48:40.354 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:48:40.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:48:40.355 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.356 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:48:40.356 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:48:40.356 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.356 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:40.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:40.359 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:48:40.359 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:48:40.359 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:48:40.360 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:48:40.360 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:48:40.360 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:48:40.360 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.361 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:40.361 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:48:40.361 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 msgr2=0x7fc9bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 0x7fc9bc06d190 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000ef0 tx=0x7fc9c800f000 comp rx=0 tx=0).stop 2023-06-10T18:48:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 msgr2=0x7fc9d81f1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00045e0 tx=0x7fc9d00049a0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 shutdown_connections 2023-06-10T18:48:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d81f0da0 0x7fc9d81f1210 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9c4003a00 0x7fc9d81ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9bc06ace0 0x7fc9bc06d190 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 --2- 172.21.15.104:0/3615395679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9d80b41a0 0x7fc9d81eab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 >> 172.21.15.104:0/3615395679 conn(0x7fc9d806b8a0 msgr2=0x7fc9d80cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:40.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 shutdown_connections 2023-06-10T18:48:40.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.313+0000 7fc9e041a700 1 -- 172.21.15.104:0/3615395679 wait complete. 2023-06-10T18:48:40.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:40.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.497+0000 7f8fdb31e700 1 Processor -- start 2023-06-10T18:48:40.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.497+0000 7f8fdb31e700 1 -- start start 2023-06-10T18:48:40.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fdb31e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fd40ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fdb31e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40cee70 con 0x7f8fd40ada30 2023-06-10T18:48:40.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fdb31e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40cf150 con 0x7f8fd40ade00 2023-06-10T18:48:40.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fdb31e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40cf430 con 0x7f8fd40aeb00 2023-06-10T18:48:40.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fd90ba700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8fd40ada30 0x7f8fd40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57322/0 (socket says 172.21.15.104:57322) 2023-06-10T18:48:40.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.498+0000 7f8fd90ba700 1 -- 172.21.15.104:0/4200803956 learned_addr learned my addr 172.21.15.104:0/4200803956 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:40.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd98bb700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fd40ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908663638 0 0) 0x7f8fd40cee70 con 0x7f8fd40ada30 2023-06-10T18:48:40.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc00035b0 con 0x7f8fd40ada30 2023-06-10T18:48:40.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888558161 0 0) 0x7f8fd40cf150 con 0x7f8fd40ade00 2023-06-10T18:48:40.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd40cee70 con 0x7f8fd40ade00 2023-06-10T18:48:40.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 13757187 0 0) 0x7f8fc00035b0 con 0x7f8fd40ada30 2023-06-10T18:48:40.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 msgr2=0x7f8fd40ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fd40ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 >> v1:172.21.15.114:6789/0 conn(0x7f8fd40ade00 legacy=0x7f8fd40ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd406cf10 con 0x7f8fd40ada30 2023-06-10T18:48:40.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8fc40017f0 con 0x7f8fd40ada30 2023-06-10T18:48:40.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 >> v1:172.21.15.104:6789/0 conn(0x7f8fd40ada30 legacy=0x7f8fd40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fc0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0004850 0x7f8fc0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fc0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40cf430 con 0x7f8fc0003bc0 2023-06-10T18:48:40.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc000baf0 con 0x7f8fc0004850 2023-06-10T18:48:40.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc000bcb0 con 0x7f8fd40aeb00 2023-06-10T18:48:40.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fc0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd88b9700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0004850 0x7f8fc0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd98bb700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fc0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 -- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 msgr2=0x7f8fc0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40aeb00 0x7f8fc0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 -- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0004850 msgr2=0x7f8fc0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0004850 0x7f8fc0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.499+0000 7f8fd90ba700 1 -- 172.21.15.104:0/4200803956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc00035b0 con 0x7f8fc0003bc0 2023-06-10T18:48:40.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fc0003fb0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c240 tx=0x7f8fc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e22fac65c71da82 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400cd80 con 0x7f8fc0003bc0 2023-06-10T18:48:40.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4001c70 con 0x7f8fc0003bc0 2023-06-10T18:48:40.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fcbfff700 1 -- 172.21.15.104:0/4200803956 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4007310 con 0x7f8fc0003bc0 2023-06-10T18:48:40.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 -- 172.21.15.104:0/4200803956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 msgr2=0x7f8fc0003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/4200803956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fc0003fb0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c240 tx=0x7f8fc0005880 comp rx=0 tx=0).stop 2023-06-10T18:48:40.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fd98bb700 1 -- 172.21.15.104:0/4200803956 reap_dead start 2023-06-10T18:48:40.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 -- 172.21.15.104:0/4200803956 shutdown_connections 2023-06-10T18:48:40.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 -- 172.21.15.104:0/4200803956 >> 172.21.15.104:0/4200803956 conn(0x7f8fd406b8a0 msgr2=0x7f8fd40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:40.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 -- 172.21.15.104:0/4200803956 shutdown_connections 2023-06-10T18:48:40.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.500+0000 7f8fdb31e700 1 -- 172.21.15.104:0/4200803956 wait complete. 2023-06-10T18:48:40.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 Processor -- start 2023-06-10T18:48:40.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 -- start start 2023-06-10T18:48:40.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 0x7f8fd40b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fd40b5150 0x7f8fd40b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b4220 con 0x7f8fc0003bc0 2023-06-10T18:48:40.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4099f70 con 0x7f8fd40b5150 2023-06-10T18:48:40.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fdb31e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3610 con 0x7f8fd40b4ce0 2023-06-10T18:48:40.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fd90ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fd90ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59506/0 (socket says 172.21.15.104:59506) 2023-06-10T18:48:40.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.502+0000 7f8fd90ba700 1 -- 172.21.15.104:0/729911754 learned_addr learned my addr 172.21.15.104:0/729911754 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:40.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd88b9700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 0x7f8fd40b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd98bb700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fd40b5150 0x7f8fd40b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 -- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 msgr2=0x7f8fd40b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 0x7f8fd40b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 -- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fd40b5150 msgr2=0x7f8fd40b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fd40b5150 0x7f8fd40b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 -- 172.21.15.104:0/729911754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40b38f0 con 0x7f8fc0003bc0 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd88b9700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 0x7f8fd40b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fd90ba700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000bac0 tx=0x7f8fc4001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4005c40 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4005da0 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40b3bd0 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4007310 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.503+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd415e590 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.504+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4066f70 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.505+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8fc4019070 con 0x7f8fc0003bc0 2023-06-10T18:48:40.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.505+0000 7f8fc9ffb700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 0x7f8fb806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.505+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8fc408a5c0 con 0x7f8fc0003bc0 2023-06-10T18:48:40.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.505+0000 7f8fd88b9700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 0x7f8fb806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.506+0000 7f8fd88b9700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 0x7f8fb806d190 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40b4c70 tx=0x7f8fd0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.509+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8fc4055a30 con 0x7f8fc0003bc0 2023-06-10T18:48:40.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.697+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd4074050 con 0x7f8fc0003bc0 2023-06-10T18:48:40.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.698+0000 7f8fc9ffb700 1 -- 172.21.15.104:0/729911754 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14892 (secure 0 0 0) 0x7f8fc4059050 con 0x7f8fc0003bc0 2023-06-10T18:48:40.704 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:40.704 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":64,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:39.890654+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:40.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 msgr2=0x7f8fb806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 0x7f8fb806d190 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40b4c70 tx=0x7f8fd0009040 comp rx=0 tx=0).stop 2023-06-10T18:48:40.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 msgr2=0x7f8fd409c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000bac0 tx=0x7f8fc4001b00 comp rx=0 tx=0).stop 2023-06-10T18:48:40.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 shutdown_connections 2023-06-10T18:48:40.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fb806ace0 0x7f8fb806d190 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fd40b5150 0x7f8fd40b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fc0003bc0 0x7f8fd409c830 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 --2- 172.21.15.104:0/729911754 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fd40b4ce0 0x7f8fd40b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 >> 172.21.15.104:0/729911754 conn(0x7f8fd406b8a0 msgr2=0x7f8fd40ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 shutdown_connections 2023-06-10T18:48:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.700+0000 7f8fdb31e700 1 -- 172.21.15.104:0/729911754 wait complete. 2023-06-10T18:48:40.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:48:40.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:48:40.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.842+0000 7f976c159700 1 Processor -- start 2023-06-10T18:48:40.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.842+0000 7f976c159700 1 -- start start 2023-06-10T18:48:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f976c159700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f97640add90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f976c159700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976415cb00 con 0x7f97640cd8f0 2023-06-10T18:48:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f976c159700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97640acd90 con 0x7f976415c730 2023-06-10T18:48:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f976c159700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97640ac460 con 0x7f97640ad930 2023-06-10T18:48:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f97696f4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f97640cd8f0 0x7f97640ad220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57344/0 (socket says 172.21.15.104:57344) 2023-06-10T18:48:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.843+0000 7f97696f4700 1 -- 172.21.15.104:0/82360408 learned_addr learned my addr 172.21.15.104:0/82360408 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f976a6f6700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f97640add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366148465 0 0) 0x7f9760001110 con 0x7f97640cd8f0 2023-06-10T18:48:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97500035b0 con 0x7f97640cd8f0 2023-06-10T18:48:40.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1148428673 0 0) 0x7f97500035b0 con 0x7f97640cd8f0 2023-06-10T18:48:40.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 msgr2=0x7f97640add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f97640add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 >> v1:172.21.15.114:6789/0 conn(0x7f976415c730 legacy=0x7f97640cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97640ac740 con 0x7f97640cd8f0 2023-06-10T18:48:40.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9760001800 con 0x7f97640cd8f0 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 >> v1:172.21.15.104:6789/0 conn(0x7f97640cd8f0 legacy=0x7f97640ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f9750003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f9750004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9750004d90 0x7f9750005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97640ac460 con 0x7f9750003a30 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975000bac0 con 0x7f9750004d90 2023-06-10T18:48:40.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975000de60 con 0x7f97640ad930 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f9750003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f97696f4700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f9750004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f976a6f6700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9750004d90 0x7f9750005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 -- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 msgr2=0x7f9750004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640ad930 0x7f9750004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 -- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9750004d90 msgr2=0x7f9750005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:40.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9750004d90 0x7f9750005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f9769ef5700 1 -- 172.21.15.104:0/82360408 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500035b0 con 0x7f9750003a30 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.844+0000 7f976a6f6700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9750004d90 0x7f9750005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f9769ef5700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f9750003e00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9754001930 tx=0x7f9750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7a4859ac62af7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540013a0 con 0x7f9750003a30 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975400c690 con 0x7f9750003a30 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f9768ef3700 1 -- 172.21.15.104:0/82360408 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975400ca40 con 0x7f9750003a30 2023-06-10T18:48:40.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 -- 172.21.15.104:0/82360408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 msgr2=0x7f9750003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 --2- 172.21.15.104:0/82360408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f9750003e00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9754001930 tx=0x7f9750005880 comp rx=0 tx=0).stop 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976a6f6700 1 -- 172.21.15.104:0/82360408 reap_dead start 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 -- 172.21.15.104:0/82360408 shutdown_connections 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 -- 172.21.15.104:0/82360408 >> 172.21.15.104:0/82360408 conn(0x7f976406b8a0 msgr2=0x7f97640cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 -- 172.21.15.104:0/82360408 shutdown_connections 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.845+0000 7f976c159700 1 -- 172.21.15.104:0/82360408 wait complete. 2023-06-10T18:48:40.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.846+0000 7f976c159700 1 Processor -- start 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.846+0000 7f976c159700 1 -- start start 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97640b0b50 0x7f97640b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640b1920 0x7f97640c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97640b4450 con 0x7f9750003a30 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97640b46c0 con 0x7f97640b0b50 2023-06-10T18:48:40.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f976c159700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97640b49a0 con 0x7f97640b1920 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f9769ef5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f9769ef5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59526/0 (socket says 172.21.15.104:59526) 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.847+0000 7f9769ef5700 1 -- 172.21.15.104:0/1862617958 learned_addr learned my addr 172.21.15.104:0/1862617958 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 -- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640b1920 msgr2=0x7f97640c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640b1920 0x7f97640c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 -- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97640b0b50 msgr2=0x7f97640b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:40.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97640b0b50 0x7f97640b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 -- 172.21.15.104:0/1862617958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976406f150 con 0x7f9750003a30 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f9769ef5700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f97500047a0 tx=0x7f9754001280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754001320 con 0x7f9750003a30 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975400cdf0 con 0x7f9750003a30 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97640b4c80 con 0x7f9750003a30 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540052b0 con 0x7f9750003a30 2023-06-10T18:48:40.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.848+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97640b1da0 con 0x7f9750003a30 2023-06-10T18:48:40.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.849+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976406bee0 con 0x7f9750003a30 2023-06-10T18:48:40.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.849+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9754022070 con 0x7f9750003a30 2023-06-10T18:48:40.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.850+0000 7f975a7fc700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:40.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.850+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f975408a7f0 con 0x7f9750003a30 2023-06-10T18:48:40.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.850+0000 7f97696f4700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:40.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.850+0000 7f97696f4700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f97640b3150 tx=0x7f97600015e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:40.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:40.854+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9754055b40 con 0x7f9750003a30 2023-06-10T18:48:41.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.038+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9764074050 con 0x7f9750003a30 2023-06-10T18:48:41.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.038+0000 7f975a7fc700 1 -- 172.21.15.104:0/1862617958 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9754059160 con 0x7f9750003a30 2023-06-10T18:48:41.044 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:48:41.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 msgr2=0x7f974806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f97640b3150 tx=0x7f97600015e0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 msgr2=0x7f97640b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f97500047a0 tx=0x7f9754001280 comp rx=0 tx=0).stop 2023-06-10T18:48:41.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 shutdown_connections 2023-06-10T18:48:41.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97640b1920 0x7f97640c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9750003a30 0x7f97640b0610 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 --2- 172.21.15.104:0/1862617958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97640b0b50 0x7f97640b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 >> 172.21.15.104:0/1862617958 conn(0x7f976406b8a0 msgr2=0x7f976406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 shutdown_connections 2023-06-10T18:48:41.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.040+0000 7f976c159700 1 -- 172.21.15.104:0/1862617958 wait complete. 2023-06-10T18:48:41.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:48:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.180+0000 7ffb0ee19700 1 Processor -- start 2023-06-10T18:48:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.180+0000 7ffb0ee19700 1 -- start start 2023-06-10T18:48:41.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0ee19700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffb0806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0ee19700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb080ae7e0 con 0x7ffb0806ea70 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0ee19700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0806df90 con 0x7ffb0815c6f0 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0ee19700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0806e270 con 0x7ffb080b1780 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0d3b6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ffb0806ea70 0x7ffb0806d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57364/0 (socket says 172.21.15.104:57364) 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/2150273761 learned_addr learned my addr 172.21.15.104:0/2150273761 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb07fff700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffb0806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540114256 0 0) 0x7ffb080ae7e0 con 0x7ffb0806ea70 2023-06-10T18:48:41.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.181+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf80035b0 con 0x7ffb0806ea70 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979895231 0 0) 0x7ffb0806df90 con 0x7ffb0815c6f0 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb080ae7e0 con 0x7ffb0815c6f0 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697397297 0 0) 0x7ffaf80035b0 con 0x7ffb0806ea70 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 msgr2=0x7ffb0806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffb0806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 >> v1:172.21.15.114:6789/0 conn(0x7ffb0815c6f0 legacy=0x7ffb080b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0806e550 con 0x7ffb0806ea70 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ffb000019f0 con 0x7ffb0806ea70 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 >> v1:172.21.15.104:6789/0 conn(0x7ffb0806ea70 legacy=0x7ffb0806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf8003bc0 0x7ffaf8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffaf8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaf8004d90 0x7ffaf8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0806e270 con 0x7ffaf8004d90 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf800bba0 con 0x7ffaf8003bc0 2023-06-10T18:48:41.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf800bd10 con 0x7ffb080b1780 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaf8004d90 0x7ffaf8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb07fff700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffaf8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0cbb5700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf8003bc0 0x7ffaf8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 msgr2=0x7ffaf8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080b1780 0x7ffaf8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf8003bc0 msgr2=0x7ffaf8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf8003bc0 0x7ffaf8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/2150273761 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80035b0 con 0x7ffaf8004d90 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0cbb5700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf8003bc0 0x7ffaf8003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaf8004d90 0x7ffaf8005200 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ffb00002230 tx=0x7ffaf8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fdc316a032a714 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb000072d0 con 0x7ffaf8004d90 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00007450 con 0x7ffaf8004d90 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.182+0000 7ffb077fe700 1 -- 172.21.15.104:0/2150273761 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00007800 con 0x7ffaf8004d90 2023-06-10T18:48:41.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 -- 172.21.15.104:0/2150273761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaf8004d90 msgr2=0x7ffaf8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/2150273761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaf8004d90 0x7ffaf8005200 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ffb00002230 tx=0x7ffaf8005880 comp rx=0 tx=0).stop 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/2150273761 reap_dead start 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 -- 172.21.15.104:0/2150273761 shutdown_connections 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 -- 172.21.15.104:0/2150273761 >> 172.21.15.104:0/2150273761 conn(0x7ffb0806b8a0 msgr2=0x7ffb080cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 -- 172.21.15.104:0/2150273761 shutdown_connections 2023-06-10T18:48:41.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.183+0000 7ffb0ee19700 1 -- 172.21.15.104:0/2150273761 wait complete. 2023-06-10T18:48:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.184+0000 7ffb0ee19700 1 Processor -- start 2023-06-10T18:48:41.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.184+0000 7ffb0ee19700 1 -- start start 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb08157f30 0x7ffb080b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb08158300 0x7ffb080b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080c7a10 con 0x7ffb08158740 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08158fb0 con 0x7ffb08158300 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0ee19700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080b3b30 con 0x7ffb08157f30 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59550/0 (socket says 172.21.15.104:59550) 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/1500433121 learned_addr learned my addr 172.21.15.104:0/1500433121 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb08157f30 msgr2=0x7ffb080b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb08157f30 0x7ffb080b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb08158300 msgr2=0x7ffb080b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb08158300 0x7ffb080b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 -- 172.21.15.104:0/1500433121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080c70e0 con 0x7ffb08158740 2023-06-10T18:48:41.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.185+0000 7ffb0d3b6700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ffb00000f80 tx=0x7ffb00001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.186+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00022070 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.186+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00001db0 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.186+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00007a10 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.186+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb080c73c0 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.186+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb080c7ea0 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.187+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb08066f10 con 0x7ffb08158740 2023-06-10T18:48:41.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.187+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb0001b490 con 0x7ffb08158740 2023-06-10T18:48:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.187+0000 7ffb057fa700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.187+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ffb0008ba10 con 0x7ffb08158740 2023-06-10T18:48:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.188+0000 7ffb0cbb5700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.188+0000 7ffb0cbb5700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007c40 tx=0x7ffafc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.192+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb00056d60 con 0x7ffb08158740 2023-06-10T18:48:41.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.377+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffb080c7b50 con 0x7ffb08158740 2023-06-10T18:48:41.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.378+0000 7ffb057fa700 1 -- 172.21.15.104:0/1500433121 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffb080c7b50 con 0x7ffb08158740 2023-06-10T18:48:41.384 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:48:41.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 msgr2=0x7ffaec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007c40 tx=0x7ffafc009040 comp rx=0 tx=0).stop 2023-06-10T18:48:41.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 msgr2=0x7ffb080c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ffb00000f80 tx=0x7ffb00001cb0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 shutdown_connections 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb08158740 0x7ffb080c4560 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb08157f30 0x7ffb080b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 --2- 172.21.15.104:0/1500433121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb08158300 0x7ffb080b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 >> 172.21.15.104:0/1500433121 conn(0x7ffb0806b8a0 msgr2=0x7ffb080ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 shutdown_connections 2023-06-10T18:48:41.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.380+0000 7ffb0ee19700 1 -- 172.21.15.104:0/1500433121 wait complete. 2023-06-10T18:48:41.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:48:41.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.515+0000 7f58bdc6a700 1 Processor -- start 2023-06-10T18:48:41.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 -- start start 2023-06-10T18:48:41.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58b81567f0 0x7f58b80cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58b80705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b8070b20 con 0x7f58b815c7a0 2023-06-10T18:48:41.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b80cb2d0 con 0x7f58b81567f0 2023-06-10T18:48:41.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58bdc6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b815cca0 con 0x7f58b80701a0 2023-06-10T18:48:41.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58b6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f58b815c7a0 0x7f58b80cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57384/0 (socket says 172.21.15.104:57384) 2023-06-10T18:48:41.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.516+0000 7f58b6ffd700 1 -- 172.21.15.104:0/874878843 learned_addr learned my addr 172.21.15.104:0/874878843 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:41.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33843900 0 0) 0x7f58b8070b20 con 0x7f58b815c7a0 2023-06-10T18:48:41.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a40035b0 con 0x7f58b815c7a0 2023-06-10T18:48:41.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b77fe700 1 --2- 172.21.15.104:0/874878843 >> v2:172.21.15.114:3300/0 conn(0x7f58b81567f0 0x7f58b80cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58b80705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013690031 0 0) 0x7f58a40035b0 con 0x7f58b815c7a0 2023-06-10T18:48:41.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 msgr2=0x7f58b80705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58b80705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 >> v2:172.21.15.114:3300/0 conn(0x7f58b81567f0 msgr2=0x7f58b80cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 --2- 172.21.15.104:0/874878843 >> v2:172.21.15.114:3300/0 conn(0x7f58b81567f0 0x7f58b80cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b815cf80 con 0x7f58b815c7a0 2023-06-10T18:48:41.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f58a0001670 con 0x7f58b815c7a0 2023-06-10T18:48:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 >> v1:172.21.15.104:6789/0 conn(0x7f58b815c7a0 legacy=0x7f58b80cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58a4003a00 0x7f58a4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80cb2d0 con 0x7f58a4004d90 2023-06-10T18:48:41.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b815cca0 con 0x7f58a4003a00 2023-06-10T18:48:41.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a400ba10 con 0x7f58b80701a0 2023-06-10T18:48:41.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b77fe700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58a4003a00 0x7f58a4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b6ffd700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 -- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 msgr2=0x7f58a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80701a0 0x7f58a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 -- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58a4003a00 msgr2=0x7f58a4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.517+0000 7f58b7fff700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58a4003a00 0x7f58a4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b7fff700 1 -- 172.21.15.104:0/874878843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a40035b0 con 0x7f58a4004d90 2023-06-10T18:48:41.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b7fff700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58a4005200 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009fc0 tx=0x7f58a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889b74a689a3434c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b000ec30 con 0x7f58a4004d90 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b000f040 con 0x7f58a4004d90 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b67fc700 1 -- 172.21.15.104:0/874878843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b0014480 con 0x7f58a4004d90 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 -- 172.21.15.104:0/874878843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 msgr2=0x7f58a4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/874878843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58a4005200 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009fc0 tx=0x7f58a4005880 comp rx=0 tx=0).stop 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58b7fff700 1 -- 172.21.15.104:0/874878843 reap_dead start 2023-06-10T18:48:41.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 -- 172.21.15.104:0/874878843 shutdown_connections 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 -- 172.21.15.104:0/874878843 >> 172.21.15.104:0/874878843 conn(0x7f58b806e370 msgr2=0x7f58b806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 -- 172.21.15.104:0/874878843 shutdown_connections 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.518+0000 7f58bdc6a700 1 -- 172.21.15.104:0/874878843 wait complete. 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 Processor -- start 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 -- start start 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b80aff50 0x7f58b80c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80c5620 0x7f58b80c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80ca680 con 0x7f58a4004d90 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8155590 con 0x7f58b80aff50 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.520+0000 7f58bdc6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80c9e70 con 0x7f58b80c5620 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59566/0 (socket says 172.21.15.104:59566) 2023-06-10T18:48:41.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 -- 172.21.15.104:0/2676161902 learned_addr learned my addr 172.21.15.104:0/2676161902 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b7fff700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80c5620 0x7f58b80c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b6ffd700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b80aff50 0x7f58b80c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 -- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80c5620 msgr2=0x7f58b80c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80c5620 0x7f58b80c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 -- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b80aff50 msgr2=0x7f58b80c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b80aff50 0x7f58b80c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 -- 172.21.15.104:0/2676161902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b80c9a20 con 0x7f58a4004d90 2023-06-10T18:48:41.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58b77fe700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f58a8011aa0 tx=0x7f58a8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8011ee0 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a800f940 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b80ca0c0 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a800fcd0 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.521+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b80c4e20 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.522+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b80c4c50 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.522+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f58a8014440 con 0x7f58a4004d90 2023-06-10T18:48:41.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.523+0000 7f58affff700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 0x7f589806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.523+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f58a8089950 con 0x7f58a4004d90 2023-06-10T18:48:41.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.523+0000 7f58b6ffd700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 0x7f589806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.524+0000 7f58b6ffd700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 0x7f589806d190 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f58a0000c00 tx=0x7f58a0007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.527+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f58a8058760 con 0x7f58a4004d90 2023-06-10T18:48:41.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.715+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f58b8074050 con 0x7f58a4004d90 2023-06-10T18:48:41.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.715+0000 7f58affff700 1 -- 172.21.15.104:0/2676161902 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+126 (secure 0 0 0) 0x7f58a80582f0 con 0x7f58a4004d90 2023-06-10T18:48:41.721 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:41.722 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:48:41.723 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:48:41.723 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:41.723 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:48:41.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 msgr2=0x7f589806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 0x7f589806d190 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f58a0000c00 tx=0x7f58a0007980 comp rx=0 tx=0).stop 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 msgr2=0x7f58b8155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f58a8011aa0 tx=0x7f58a8011db0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 shutdown_connections 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b80c5620 0x7f58b80c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58a4004d90 0x7f58b8155050 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f589806ace0 0x7f589806d190 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 --2- 172.21.15.104:0/2676161902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b80aff50 0x7f58b80c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 >> 172.21.15.104:0/2676161902 conn(0x7f58b806e370 msgr2=0x7f58b806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 shutdown_connections 2023-06-10T18:48:41.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.717+0000 7f58bdc6a700 1 -- 172.21.15.104:0/2676161902 wait complete. 2023-06-10T18:48:41.736 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:48:41.737 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:48:41.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.855+0000 7f8a59722700 1 Processor -- start 2023-06-10T18:48:41.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.855+0000 7f8a59722700 1 -- start start 2023-06-10T18:48:41.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.855+0000 7f8a59722700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a540ce510 0x7f8a540ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.855+0000 7f8a59722700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415b190 0x7f8a54070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.855+0000 7f8a59722700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5415bcc0 con 0x7f8a5415dad0 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a59722700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a54071c90 con 0x7f8a540ce510 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a59722700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a54157e60 con 0x7f8a5415b190 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a540ce510 0x7f8a540ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a537fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8a5415dad0 0x7f8a5415df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57406/0 (socket says 172.21.15.104:57406) 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a537fe700 1 -- 172.21.15.104:0/2726252277 learned_addr learned my addr 172.21.15.104:0/2726252277 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a527fc700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415b190 0x7f8a54070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 -- 172.21.15.104:0/2726252277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415b190 msgr2=0x7f8a54070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415b190 0x7f8a54070130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 -- 172.21.15.104:0/2726252277 >> v1:172.21.15.104:6789/0 conn(0x7f8a5415dad0 legacy=0x7f8a5415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 -- 172.21.15.104:0/2726252277 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540cee20 con 0x7f8a540ce510 2023-06-10T18:48:41.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a51ffb700 1 -- 172.21.15.104:0/2726252277 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571349752 0 0) 0x7f8a5415bcc0 con 0x7f8a5415dad0 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a540ce510 0x7f8a540ce8e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009840 tx=0x7f8a44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d8980370671b31 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a51ffb700 1 -- 172.21.15.104:0/2726252277 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44011830 con 0x7f8a540ce510 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a51ffb700 1 -- 172.21.15.104:0/2726252277 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44011990 con 0x7f8a540ce510 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.856+0000 7f8a51ffb700 1 -- 172.21.15.104:0/2726252277 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44011d20 con 0x7f8a540ce510 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 -- 172.21.15.104:0/2726252277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a540ce510 msgr2=0x7f8a540ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a540ce510 0x7f8a540ce8e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009840 tx=0x7f8a44009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 -- 172.21.15.104:0/2726252277 shutdown_connections 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a540ce510 0x7f8a540ce8e0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 --2- 172.21.15.104:0/2726252277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415b190 0x7f8a54070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 -- 172.21.15.104:0/2726252277 >> 172.21.15.104:0/2726252277 conn(0x7f8a5406aac0 msgr2=0x7f8a540b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 -- 172.21.15.104:0/2726252277 shutdown_connections 2023-06-10T18:48:41.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.857+0000 7f8a59722700 1 -- 172.21.15.104:0/2726252277 wait complete. 2023-06-10T18:48:41.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.858+0000 7f8a59722700 1 Processor -- start 2023-06-10T18:48:41.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 -- start start 2023-06-10T18:48:41.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 0x7f8a54149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 0x7f8a540c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c8bb0 con 0x7f8a5415b190 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c5970 con 0x7f8a540ce510 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.859+0000 7f8a59722700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5414a110 con 0x7f8a5415dad0 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59582/0 (socket says 172.21.15.104:59582) 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 -- 172.21.15.104:0/4038798295 learned_addr learned my addr 172.21.15.104:0/4038798295 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a537fe700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 0x7f8a540c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 0x7f8a54149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 -- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 msgr2=0x7f8a540c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 0x7f8a540c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 -- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 msgr2=0x7f8a54149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 0x7f8a54149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 -- 172.21.15.104:0/4038798295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a440094a0 con 0x7f8a5415b190 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 0x7f8a54149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:41.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a537fe700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 0x7f8a540c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a527fc700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d830 tx=0x7f8a4800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a480125f0 con 0x7f8a5415b190 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48012750 con 0x7f8a5415b190 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a540b4140 con 0x7f8a5415b190 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48012ae0 con 0x7f8a5415b190 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.860+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a540c90e0 con 0x7f8a5415b190 2023-06-10T18:48:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.861+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a54074050 con 0x7f8a5415b190 2023-06-10T18:48:41.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.862+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8a48012c40 con 0x7f8a5415b190 2023-06-10T18:48:41.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.862+0000 7f8a3b7fe700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 0x7f8a4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:41.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.862+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8a48089620 con 0x7f8a5415b190 2023-06-10T18:48:41.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.862+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 0x7f8a4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:41.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.863+0000 7f8a52ffd700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 0x7f8a4006d190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8a440090b0 tx=0x7f8a4401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:41.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:41.866+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8a480548d0 con 0x7f8a5415b190 2023-06-10T18:48:42.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.053+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8a540b0980 con 0x7f8a5415b190 2023-06-10T18:48:42.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.741+0000 7f8a3b7fe700 1 -- 172.21.15.104:0/4038798295 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8a48057ef0 con 0x7f8a5415b190 2023-06-10T18:48:42.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.743+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 msgr2=0x7f8a4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.743+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 0x7f8a4006d190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8a440090b0 tx=0x7f8a4401c040 comp rx=0 tx=0).stop 2023-06-10T18:48:42.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.743+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 msgr2=0x7f8a540b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.743+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d830 tx=0x7f8a4800dbf0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 shutdown_connections 2023-06-10T18:48:42.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a4006ace0 0x7f8a4006d190 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540ce510 0x7f8a54149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a5415dad0 0x7f8a540c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 --2- 172.21.15.104:0/4038798295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a5415b190 0x7f8a540b3910 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 >> 172.21.15.104:0/4038798295 conn(0x7f8a5406aac0 msgr2=0x7f8a54159650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 shutdown_connections 2023-06-10T18:48:42.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.744+0000 7f8a59722700 1 -- 172.21.15.104:0/4038798295 wait complete. 2023-06-10T18:48:42.752 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:48:42.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:48:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.893+0000 7f5c6fd33700 1 Processor -- start 2023-06-10T18:48:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.893+0000 7f5c6fd33700 1 -- start start 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6fd33700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c680d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6fd33700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5c680d0830 0x7f5c680b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6fd33700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c680ad3c0 con 0x7f5c68157b30 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6fd33700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c68157f00 con 0x7f5c680d0830 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6fd33700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680b0170 con 0x7f5c680cea50 2023-06-10T18:48:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6dacf700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5c68157b30 0x7f5c680ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57416/0 (socket says 172.21.15.104:57416) 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.894+0000 7f5c6dacf700 1 -- 172.21.15.104:0/847257544 learned_addr learned my addr 172.21.15.104:0/847257544 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204099762 0 0) 0x7f5c680ad3c0 con 0x7f5c68157b30 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540035b0 con 0x7f5c68157b30 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c680d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6e2d0700 1 --2- 172.21.15.104:0/847257544 >> v2:172.21.15.114:3300/0 conn(0x7f5c680d0830 0x7f5c680b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187153690 0 0) 0x7f5c540035b0 con 0x7f5c68157b30 2023-06-10T18:48:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 msgr2=0x7f5c680d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c680d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 >> v2:172.21.15.114:3300/0 conn(0x7f5c680d0830 msgr2=0x7f5c680b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 --2- 172.21.15.104:0/847257544 >> v2:172.21.15.114:3300/0 conn(0x7f5c680d0830 0x7f5c680b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c680b0410 con 0x7f5c68157b30 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5c58001670 con 0x7f5c68157b30 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 >> v1:172.21.15.104:6789/0 conn(0x7f5c68157b30 legacy=0x7f5c680ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54003a00 0x7f5c54003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c54004430 0x7f5c540048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c54004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c68157f00 con 0x7f5c54004430 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680b0170 con 0x7f5c54003a00 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5400ba10 con 0x7f5c680cea50 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c54004430 0x7f5c540048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6dacf700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54003a00 0x7f5c54003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6e2d0700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c54004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 -- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 msgr2=0x7f5c54004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c680cea50 0x7f5c54004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 -- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54003a00 msgr2=0x7f5c54003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54003a00 0x7f5c54003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.895+0000 7f5c6d2ce700 1 -- 172.21.15.104:0/847257544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540035b0 con 0x7f5c54004430 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c54004430 0x7f5c540048f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009b30 tx=0x7f5c54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7b78b3b6781e92 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400e6c0 con 0x7f5c54004430 2023-06-10T18:48:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400e820 con 0x7f5c54004430 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6cacd700 1 -- 172.21.15.104:0/847257544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400eb90 con 0x7f5c54004430 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 -- 172.21.15.104:0/847257544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c54004430 msgr2=0x7f5c540048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/847257544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c54004430 0x7f5c540048f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009b30 tx=0x7f5c54005880 comp rx=0 tx=0).stop 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6e2d0700 1 -- 172.21.15.104:0/847257544 reap_dead start 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 -- 172.21.15.104:0/847257544 shutdown_connections 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 -- 172.21.15.104:0/847257544 >> 172.21.15.104:0/847257544 conn(0x7f5c68089fd0 msgr2=0x7f5c6808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 -- 172.21.15.104:0/847257544 shutdown_connections 2023-06-10T18:48:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.896+0000 7f5c6fd33700 1 -- 172.21.15.104:0/847257544 wait complete. 2023-06-10T18:48:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 Processor -- start 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 -- start start 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54004430 0x7f5c680cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c6806c460 0x7f5c6814bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c680b2c10 con 0x7f5c680b2230 2023-06-10T18:48:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c681587f0 con 0x7f5c54004430 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6fd33700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6814c290 con 0x7f5c6806c460 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6e2d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6e2d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59604/0 (socket says 172.21.15.104:59604) 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.898+0000 7f5c6e2d0700 1 -- 172.21.15.104:0/4073553124 learned_addr learned my addr 172.21.15.104:0/4073553124 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6dacf700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54004430 0x7f5c680cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6d2ce700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c6806c460 0x7f5c6814bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 -- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c6806c460 msgr2=0x7f5c6814bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c6806c460 0x7f5c6814bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 -- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54004430 msgr2=0x7f5c680cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54004430 0x7f5c680cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 -- 172.21.15.104:0/4073553124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c680ce440 con 0x7f5c680b2230 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6e2d0700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5c680713f0 tx=0x7f5c600111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000e070 con 0x7f5c680b2230 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60003cf0 con 0x7f5c680b2230 2023-06-10T18:48:42.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c68158a60 con 0x7f5c680b2230 2023-06-10T18:48:42.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60003ea0 con 0x7f5c680b2230 2023-06-10T18:48:42.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.899+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6806c8d0 con 0x7f5c680b2230 2023-06-10T18:48:42.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.900+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c68158f80 con 0x7f5c680b2230 2023-06-10T18:48:42.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.901+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5c600117a0 con 0x7f5c680b2230 2023-06-10T18:48:42.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.901+0000 7f5c5e7fc700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 0x7f5c4c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:42.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.901+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f5c60089720 con 0x7f5c680b2230 2023-06-10T18:48:42.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.901+0000 7f5c6dacf700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 0x7f5c4c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:42.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.902+0000 7f5c6dacf700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 0x7f5c4c06d1c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c58001050 tx=0x7f5c58007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:42.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:42.906+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5c68158f80 con 0x7f5c680b2230 2023-06-10T18:48:43.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.092+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c68158f80 con 0x7f5c680b2230 2023-06-10T18:48:43.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.752+0000 7f5c5e7fc700 1 -- 172.21.15.104:0/4073553124 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c68158f80 con 0x7f5c680b2230 2023-06-10T18:48:43.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 msgr2=0x7f5c4c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 0x7f5c4c06d1c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5c58001050 tx=0x7f5c58007980 comp rx=0 tx=0).stop 2023-06-10T18:48:43.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 msgr2=0x7f5c680b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5c680713f0 tx=0x7f5c600111d0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 shutdown_connections 2023-06-10T18:48:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5c4c06ad10 0x7f5c4c06d1c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c54004430 0x7f5c680cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c680b2230 0x7f5c680b26d0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 --2- 172.21.15.104:0/4073553124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5c6806c460 0x7f5c6814bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.758+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 >> 172.21.15.104:0/4073553124 conn(0x7f5c68089fd0 msgr2=0x7f5c680ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.761+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 shutdown_connections 2023-06-10T18:48:43.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.761+0000 7f5c6fd33700 1 -- 172.21.15.104:0/4073553124 wait complete. 2023-06-10T18:48:43.772 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:48:43.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:48:43.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.904+0000 7faf92d4d700 1 Processor -- start 2023-06-10T18:48:43.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.904+0000 7faf92d4d700 1 -- start start 2023-06-10T18:48:43.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf92d4d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf8c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf92d4d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0ad3c0 con 0x7faf8c0d0a00 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf92d4d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c157f00 con 0x7faf8c0ce880 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf92d4d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d0e40 con 0x7faf8c157b30 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf912ea700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faf8c0d0a00 0x7faf8c0b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57438/0 (socket says 172.21.15.104:57438) 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf912ea700 1 -- 172.21.15.104:0/1710447209 learned_addr learned my addr 172.21.15.104:0/1710447209 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:43.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf8c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768254388 0 0) 0x7faf8c0ad3c0 con 0x7faf8c0d0a00 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7c0035b0 con 0x7faf8c0d0a00 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808476086 0 0) 0x7faf7c0035b0 con 0x7faf8c0d0a00 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 msgr2=0x7faf8c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf8c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 >> v1:172.21.15.114:6789/0 conn(0x7faf8c0ce880 legacy=0x7faf8c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0b0170 con 0x7faf8c0d0a00 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7faf840019f0 con 0x7faf8c0d0a00 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 >> v1:172.21.15.104:6789/0 conn(0x7faf8c0d0a00 legacy=0x7faf8c0b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf7c003a30 0x7faf7c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c0043d0 0x7faf7c004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf7c0050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d0e40 con 0x7faf7c003a30 2023-06-10T18:48:43.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.905+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c00ba10 con 0x7faf7c0043d0 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c00bba0 con 0x7faf8c157b30 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf7c003a30 0x7faf7c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf8bfff700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c0043d0 0x7faf7c004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf912ea700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf7c0050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 -- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 msgr2=0x7faf7c0050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c157b30 0x7faf7c0050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 -- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c0043d0 msgr2=0x7faf7c004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c0043d0 0x7faf7c004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 -- 172.21.15.104:0/1710447209 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0035b0 con 0x7faf7c003a30 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf7c003a30 0x7faf7c003e00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7faf80009b60 tx=0x7faf7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e77a9940a5e6355 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8000e6c0 con 0x7faf7c003a30 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8000e820 con 0x7faf7c003a30 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.906+0000 7faf8b7fe700 1 -- 172.21.15.104:0/1710447209 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8000eb90 con 0x7faf7c003a30 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 -- 172.21.15.104:0/1710447209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf7c003a30 msgr2=0x7faf7c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1710447209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf7c003a30 0x7faf7c003e00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7faf80009b60 tx=0x7faf7c005880 comp rx=0 tx=0).stop 2023-06-10T18:48:43.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf912ea700 1 -- 172.21.15.104:0/1710447209 reap_dead start 2023-06-10T18:48:43.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 -- 172.21.15.104:0/1710447209 shutdown_connections 2023-06-10T18:48:43.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 -- 172.21.15.104:0/1710447209 >> 172.21.15.104:0/1710447209 conn(0x7faf8c089fd0 msgr2=0x7faf8c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:43.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 -- 172.21.15.104:0/1710447209 shutdown_connections 2023-06-10T18:48:43.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.907+0000 7faf92d4d700 1 -- 172.21.15.104:0/1710447209 wait complete. 2023-06-10T18:48:43.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.908+0000 7faf92d4d700 1 Processor -- start 2023-06-10T18:48:43.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.908+0000 7faf92d4d700 1 -- start start 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c003a30 0x7faf8c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 0x7faf8c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c154d50 con 0x7faf8c154390 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c06cd10 con 0x7faf7c003a30 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf92d4d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0cd9d0 con 0x7faf8c1527d0 2023-06-10T18:48:43.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59624/0 (socket says 172.21.15.104:59624) 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 -- 172.21.15.104:0/1403579081 learned_addr learned my addr 172.21.15.104:0/1403579081 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf8bfff700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 0x7faf8c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c003a30 0x7faf8c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 -- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 msgr2=0x7faf8c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 0x7faf8c152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 -- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c003a30 msgr2=0x7faf8c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c003a30 0x7faf8c152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 -- 172.21.15.104:0/1403579081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0cdc70 con 0x7faf8c154390 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf8bfff700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 0x7faf8c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf912ea700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7faf8c071360 tx=0x7faf8400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.909+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8400ad20 con 0x7faf8c154390 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.910+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf840133f0 con 0x7faf8c154390 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.910+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c0b26f0 con 0x7faf8c154390 2023-06-10T18:48:43.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.910+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf840137a0 con 0x7faf8c154390 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.910+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c0b23e0 con 0x7faf8c154390 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.911+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c0b2c10 con 0x7faf8c154390 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.911+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf84013900 con 0x7faf8c154390 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.911+0000 7faf897fa700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.912+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6282+0+0 (secure 0 0 0) 0x7faf84089d00 con 0x7faf8c154390 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.912+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:43.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.912+0000 7faf90ae9700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0cc5e0 tx=0x7faf80008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:43.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:43.916+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf8c0b2c10 con 0x7faf8c154390 2023-06-10T18:48:44.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.144+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faf8c0b2c10 con 0x7faf8c154390 2023-06-10T18:48:44.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.153+0000 7faf897fa700 1 -- 172.21.15.104:0/1403579081 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v29) v1 ==== 174+0+0 (secure 0 0 0) 0x7faf8c0b2c10 con 0x7faf8c154390 2023-06-10T18:48:44.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 msgr2=0x7faf7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0cc5e0 tx=0x7faf80008030 comp rx=0 tx=0).stop 2023-06-10T18:48:44.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 msgr2=0x7faf8c154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7faf8c071360 tx=0x7faf8400a5b0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 shutdown_connections 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf7c003a30 0x7faf8c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c154390 0x7faf8c154810 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 --2- 172.21.15.104:0/1403579081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c1527d0 0x7faf8c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 >> 172.21.15.104:0/1403579081 conn(0x7faf8c089fd0 msgr2=0x7faf8c0ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.157+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 shutdown_connections 2023-06-10T18:48:44.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.158+0000 7faf92d4d700 1 -- 172.21.15.104:0/1403579081 wait complete. 2023-06-10T18:48:44.168 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 10 and data pool 11 2023-06-10T18:48:44.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:44.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.302+0000 7f6e04d71700 1 Processor -- start 2023-06-10T18:48:44.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.302+0000 7f6e04d71700 1 -- start start 2023-06-10T18:48:44.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e000aeaa0 0x7f6e000cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 0x7f6e000cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 0x7f6e000ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000c9bc0 con 0x7f6e000aeaa0 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000c9e60 con 0x7f6e000ca2f0 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6e04d71700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000aee70 con 0x7f6e000b1080 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e000aeaa0 0x7f6e000cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e000aeaa0 0x7f6e000cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59628/0 (socket says 172.21.15.104:59628) 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 -- 172.21.15.104:0/2123879140 learned_addr learned my addr 172.21.15.104:0/2123879140 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfdd9b700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 0x7f6e000cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfed9d700 1 --2- 172.21.15.104:0/2123879140 >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 0x7f6e000ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 -- 172.21.15.104:0/2123879140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 msgr2=0x7f6e000cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 0x7f6e000cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 -- 172.21.15.104:0/2123879140 >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 msgr2=0x7f6e000ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:44.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/2123879140 >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 0x7f6e000ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 -- 172.21.15.104:0/2123879140 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0006e540 con 0x7f6e000aeaa0 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfdd9b700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 0x7f6e000cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfed9d700 1 --2- 172.21.15.104:0/2123879140 >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 0x7f6e000ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000aeaa0 0x7f6e000cd140 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6e0006fa30 tx=0x7f6df0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fffa0f4cf545c9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfd59a700 1 -- 172.21.15.104:0/2123879140 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0005e30 con 0x7f6e000aeaa0 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfd59a700 1 -- 172.21.15.104:0/2123879140 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0004030 con 0x7f6e000aeaa0 2023-06-10T18:48:44.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.303+0000 7f6dfd59a700 1 -- 172.21.15.104:0/2123879140 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df00043e0 con 0x7f6e000aeaa0 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 -- 172.21.15.104:0/2123879140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000aeaa0 msgr2=0x7f6e000cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000aeaa0 0x7f6e000cd140 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6e0006fa30 tx=0x7f6df0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 -- 172.21.15.104:0/2123879140 shutdown_connections 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000aeaa0 0x7f6e000cd140 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 --2- 172.21.15.104:0/2123879140 >> v2:172.21.15.114:3300/0 conn(0x7f6e000ca2f0 0x7f6e000ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 --2- 172.21.15.104:0/2123879140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000b1080 0x7f6e000cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 -- 172.21.15.104:0/2123879140 >> 172.21.15.104:0/2123879140 conn(0x7f6e0006b8a0 msgr2=0x7f6e000cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 -- 172.21.15.104:0/2123879140 shutdown_connections 2023-06-10T18:48:44.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.304+0000 7f6e04d71700 1 -- 172.21.15.104:0/2123879140 wait complete. 2023-06-10T18:48:44.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.305+0000 7f6e04d71700 1 Processor -- start 2023-06-10T18:48:44.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.305+0000 7f6e04d71700 1 -- start start 2023-06-10T18:48:44.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 0x7f6e00156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000ca2f0 0x7f6e000b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000b3d70 con 0x7f6e000b1080 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000b3160 con 0x7f6e000aeaa0 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6e04d71700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000b3440 con 0x7f6e000ca2f0 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59636/0 (socket says 172.21.15.104:59636) 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 -- 172.21.15.104:0/798433220 learned_addr learned my addr 172.21.15.104:0/798433220 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:44.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 0x7f6e00156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfed9d700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000ca2f0 0x7f6e000b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 -- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000ca2f0 msgr2=0x7f6e000b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000ca2f0 0x7f6e000b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 -- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 msgr2=0x7f6e00156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 0x7f6e00156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfdd9b700 1 -- 172.21.15.104:0/798433220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0009500 con 0x7f6e000b1080 2023-06-10T18:48:44.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.306+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 0x7f6e00156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6dfdd9b700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d800 tx=0x7f6df400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4004030 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df4004190 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e000b3720 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4004520 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.307+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0015c2d0 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.308+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00074050 con 0x7f6e000b1080 2023-06-10T18:48:44.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.308+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6df40036a0 con 0x7f6e000b1080 2023-06-10T18:48:44.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.308+0000 7f6deeffd700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 0x7f6de006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.309+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6df4089b50 con 0x7f6e000b1080 2023-06-10T18:48:44.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.309+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 0x7f6de006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.309+0000 7f6dfe59c700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 0x7f6de006d190 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6e0006fa30 tx=0x7f6df0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.313+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6df4058550 con 0x7f6e000b1080 2023-06-10T18:48:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.500+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e000c2de0 con 0x7f6e000b1080 2023-06-10T18:48:44.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.501+0000 7f6deeffd700 1 -- 172.21.15.104:0/798433220 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+18616 (secure 0 0 0) 0x7f6df40580e0 con 0x7f6e000b1080 2023-06-10T18:48:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:44.507 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":67,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:44.145966+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:44.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 msgr2=0x7f6de006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 0x7f6de006d190 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6e0006fa30 tx=0x7f6df0008070 comp rx=0 tx=0).stop 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 msgr2=0x7f6e00156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d800 tx=0x7f6df400dbc0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 shutdown_connections 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6de006ace0 0x7f6de006d190 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e000aeaa0 0x7f6e00156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e000ca2f0 0x7f6e000b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 --2- 172.21.15.104:0/798433220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e000b1080 0x7f6e00156fa0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 >> 172.21.15.104:0/798433220 conn(0x7f6e0006b8a0 msgr2=0x7f6e0006a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 shutdown_connections 2023-06-10T18:48:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.503+0000 7f6e04d71700 1 -- 172.21.15.104:0/798433220 wait complete. 2023-06-10T18:48:44.521 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.634+0000 7fafbcd03700 1 Processor -- start 2023-06-10T18:48:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.634+0000 7fafbcd03700 1 -- start start 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafbcd03700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafb815a420 0x7fafb80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafbcd03700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 0x7fafb80ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafbcd03700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb80b6070 con 0x7fafb80ce890 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafbcd03700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80cecb0 con 0x7fafb815a420 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafbcd03700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80b5460 con 0x7fafb80b4ff0 2023-06-10T18:48:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafb659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fafb815a420 0x7fafb80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafb6d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fafb80ce890 0x7fafb80b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57474/0 (socket says 172.21.15.104:57474) 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.635+0000 7fafb6d9d700 1 -- 172.21.15.104:0/1962111184 learned_addr learned my addr 172.21.15.104:0/1962111184 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb5d9b700 1 --2- 172.21.15.104:0/1962111184 >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 0x7fafb80ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb559a700 1 -- 172.21.15.104:0/1962111184 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957826197 0 0) 0x7fafb80b6070 con 0x7fafb80ce890 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb559a700 1 -- 172.21.15.104:0/1962111184 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa00035b0 con 0x7fafb80ce890 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb659c700 1 -- 172.21.15.104:0/1962111184 >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 msgr2=0x7fafb80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb659c700 1 --2- 172.21.15.104:0/1962111184 >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 0x7fafb80ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb659c700 1 -- 172.21.15.104:0/1962111184 >> v1:172.21.15.104:6789/0 conn(0x7fafb80ce890 legacy=0x7fafb80b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb659c700 1 -- 172.21.15.104:0/1962111184 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80b5740 con 0x7fafb815a420 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb5d9b700 1 --2- 172.21.15.104:0/1962111184 >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 0x7fafb80ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb659c700 1 --2- 172.21.15.104:0/1962111184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb80b1660 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fafb8066590 tx=0x7fafac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ff0e2e41fc9c8d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb559a700 1 -- 172.21.15.104:0/1962111184 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011830 con 0x7fafb815a420 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb559a700 1 -- 172.21.15.104:0/1962111184 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac011990 con 0x7fafb815a420 2023-06-10T18:48:44.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.636+0000 7fafb559a700 1 -- 172.21.15.104:0/1962111184 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011d20 con 0x7fafb815a420 2023-06-10T18:48:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 -- 172.21.15.104:0/1962111184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 msgr2=0x7fafb80b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 --2- 172.21.15.104:0/1962111184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb80b1660 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fafb8066590 tx=0x7fafac009b20 comp rx=0 tx=0).stop 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 -- 172.21.15.104:0/1962111184 shutdown_connections 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 --2- 172.21.15.104:0/1962111184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb80b1660 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 --2- 172.21.15.104:0/1962111184 >> v2:172.21.15.114:3300/0 conn(0x7fafb80b4ff0 0x7fafb80ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 -- 172.21.15.104:0/1962111184 >> 172.21.15.104:0/1962111184 conn(0x7fafb8089fd0 msgr2=0x7fafb80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 -- 172.21.15.104:0/1962111184 shutdown_connections 2023-06-10T18:48:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.637+0000 7fafbcd03700 1 -- 172.21.15.104:0/1962111184 wait complete. 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.638+0000 7fafbcd03700 1 Processor -- start 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 -- start start 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 0x7fafb81f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafb80ce890 0x7fafb80c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81e6b40 con 0x7fafb815a420 2023-06-10T18:48:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81f0bd0 con 0x7fafb80b4ff0 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.639+0000 7fafbcd03700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81e5c50 con 0x7fafb80ce890 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59650/0 (socket says 172.21.15.104:59650) 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 -- 172.21.15.104:0/3940989389 learned_addr learned my addr 172.21.15.104:0/3940989389 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb5d9b700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafb80ce890 0x7fafb80c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb659c700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 0x7fafb81f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 -- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafb80ce890 msgr2=0x7fafb80c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafb80ce890 0x7fafb80c4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 -- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 msgr2=0x7fafb81f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 0x7fafb81f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 -- 172.21.15.104:0/3940989389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0094a0 con 0x7fafb815a420 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb659c700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 0x7fafb81f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafb6d9d700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fafb000cb40 tx=0x7fafb000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000ab50 con 0x7fafb815a420 2023-06-10T18:48:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb000acb0 con 0x7fafb815a420 2023-06-10T18:48:44.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb81e5f30 con 0x7fafb815a420 2023-06-10T18:48:44.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0012620 con 0x7fafb815a420 2023-06-10T18:48:44.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.640+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb81e64f0 con 0x7fafb815a420 2023-06-10T18:48:44.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.641+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb80b64e0 con 0x7fafb815a420 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.642+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fafb001b3b0 con 0x7fafb815a420 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.642+0000 7fafa6ffd700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.642+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fafb0089350 con 0x7fafb815a420 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.642+0000 7fafb659c700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.643+0000 7fafb659c700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fafac0090b0 tx=0x7fafac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.646+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fafb80b64e0 con 0x7fafb815a420 2023-06-10T18:48:44.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.836+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafb80b64e0 con 0x7fafb815a420 2023-06-10T18:48:44.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.837+0000 7fafa6ffd700 1 -- 172.21.15.104:0/3940989389 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+18616 (secure 0 0 0) 0x7fafb80b64e0 con 0x7fafb815a420 2023-06-10T18:48:44.843 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:44.843 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":67,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:44.145966+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:44.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 msgr2=0x7faf9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fafac0090b0 tx=0x7fafac008030 comp rx=0 tx=0).stop 2023-06-10T18:48:44.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 msgr2=0x7fafb81e3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fafb000cb40 tx=0x7fafb000cf00 comp rx=0 tx=0).stop 2023-06-10T18:48:44.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 shutdown_connections 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb80b4ff0 0x7fafb81f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb815a420 0x7fafb81e3c80 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 --2- 172.21.15.104:0/3940989389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafb80ce890 0x7fafb80c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 >> 172.21.15.104:0/3940989389 conn(0x7fafb8089fd0 msgr2=0x7fafb80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 shutdown_connections 2023-06-10T18:48:44.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.839+0000 7fafbcd03700 1 -- 172.21.15.104:0/3940989389 wait complete. 2023-06-10T18:48:44.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:48:44.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.974+0000 7fcd347b6700 1 Processor -- start 2023-06-10T18:48:44.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.974+0000 7fcd347b6700 1 -- start start 2023-06-10T18:48:44.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.975+0000 7fcd347b6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd2c145be0 0x7fcd2c06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.975+0000 7fcd347b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c071120 0x7fcd2c0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.975+0000 7fcd347b6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c06cef0 con 0x7fcd2c06e4d0 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.975+0000 7fcd347b6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c06ae00 con 0x7fcd2c145be0 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.975+0000 7fcd347b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c9510 con 0x7fcd2c071120 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd2c145be0 0x7fcd2c06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31d51700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcd2c06e4d0 0x7fcd2c070a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57484/0 (socket says 172.21.15.104:57484) 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31d51700 1 -- 172.21.15.104:0/2095634611 learned_addr learned my addr 172.21.15.104:0/2095634611 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32d53700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c071120 0x7fcd2c0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 -- 172.21.15.104:0/2095634611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c071120 msgr2=0x7fcd2c0c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c071120 0x7fcd2c0c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 -- 172.21.15.104:0/2095634611 >> v1:172.21.15.104:6789/0 conn(0x7fcd2c06e4d0 legacy=0x7fcd2c070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31550700 1 -- 172.21.15.104:0/2095634611 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869994665 0 0) 0x7fcd2c06cef0 con 0x7fcd2c06e4d0 2023-06-10T18:48:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 -- 172.21.15.104:0/2095634611 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c06c930 con 0x7fcd2c145be0 2023-06-10T18:48:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd32552700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c145be0 0x7fcd2c06b1c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009840 tx=0x7fcd1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347b77eb5a0ee5b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31550700 1 -- 172.21.15.104:0/2095634611 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c011830 con 0x7fcd2c145be0 2023-06-10T18:48:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31550700 1 -- 172.21.15.104:0/2095634611 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c011990 con 0x7fcd2c145be0 2023-06-10T18:48:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.976+0000 7fcd31550700 1 -- 172.21.15.104:0/2095634611 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c011d20 con 0x7fcd2c145be0 2023-06-10T18:48:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 -- 172.21.15.104:0/2095634611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c145be0 msgr2=0x7fcd2c06b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c145be0 0x7fcd2c06b1c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009840 tx=0x7fcd1c009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 -- 172.21.15.104:0/2095634611 shutdown_connections 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c145be0 0x7fcd2c06b1c0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 --2- 172.21.15.104:0/2095634611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c071120 0x7fcd2c0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 -- 172.21.15.104:0/2095634611 >> 172.21.15.104:0/2095634611 conn(0x7fcd2c0ceec0 msgr2=0x7fcd2c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 -- 172.21.15.104:0/2095634611 shutdown_connections 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.977+0000 7fcd347b6700 1 -- 172.21.15.104:0/2095634611 wait complete. 2023-06-10T18:48:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.978+0000 7fcd347b6700 1 Processor -- start 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.978+0000 7fcd347b6700 1 -- start start 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2c06e4d0 0x7fcd2c0ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c145be0 0x7fcd2c15b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0ae790 con 0x7fcd2c071120 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c1e6030 con 0x7fcd2c06e4d0 2023-06-10T18:48:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0ab8f0 con 0x7fcd2c145be0 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59664/0 (socket says 172.21.15.104:59664) 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 -- 172.21.15.104:0/388484424 learned_addr learned my addr 172.21.15.104:0/388484424 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd32552700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2c06e4d0 0x7fcd2c0ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd32d53700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c145be0 0x7fcd2c15b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 -- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c145be0 msgr2=0x7fcd2c15b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c145be0 0x7fcd2c15b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 -- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2c06e4d0 msgr2=0x7fcd2c0ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:44.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2c06e4d0 0x7fcd2c0ae250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:44.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 -- 172.21.15.104:0/388484424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0094a0 con 0x7fcd2c071120 2023-06-10T18:48:44.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd31d51700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c580 tx=0x7fcd2800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28004c80 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28004e00 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c143fb0 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28007310 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.979+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0abb90 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.980+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c074050 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.981+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcd28007470 con 0x7fcd2c071120 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.981+0000 7fcd22ffd700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 0x7fcd1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:44.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.981+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcd280892a0 con 0x7fcd2c071120 2023-06-10T18:48:44.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.981+0000 7fcd32552700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 0x7fcd1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:44.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.982+0000 7fcd32552700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 0x7fcd1006d190 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c143ee0 tx=0x7fcd1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:44.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:44.985+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcd28054210 con 0x7fcd2c071120 2023-06-10T18:48:45.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:45.215+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcd2c160a80 con 0x7fcd2c071120 2023-06-10T18:48:46.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.157+0000 7fcd22ffd700 1 -- 172.21.15.104:0/388484424 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcd28057830 con 0x7fcd2c071120 2023-06-10T18:48:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 msgr2=0x7fcd1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 0x7fcd1006d190 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c143ee0 tx=0x7fcd1c008030 comp rx=0 tx=0).stop 2023-06-10T18:48:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 msgr2=0x7fcd2c1e5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c580 tx=0x7fcd2800c940 comp rx=0 tx=0).stop 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 shutdown_connections 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd1006ace0 0x7fcd1006d190 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2c06e4d0 0x7fcd2c0ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2c145be0 0x7fcd2c15b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 --2- 172.21.15.104:0/388484424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c071120 0x7fcd2c1e5af0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 >> 172.21.15.104:0/388484424 conn(0x7fcd2c0ceec0 msgr2=0x7fcd2c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:46.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 shutdown_connections 2023-06-10T18:48:46.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.160+0000 7fcd347b6700 1 -- 172.21.15.104:0/388484424 wait complete. 2023-06-10T18:48:46.181 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:48:46.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:48:46.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.308+0000 7f60b3198700 1 Processor -- start 2023-06-10T18:48:46.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.308+0000 7f60b3198700 1 -- start start 2023-06-10T18:48:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b3198700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f60ac15c6f0 0x7f60ac0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b3198700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b3198700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac0ae7e0 con 0x7f60ac06ea70 2023-06-10T18:48:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b3198700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac06df90 con 0x7f60ac15c6f0 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b3198700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac06e270 con 0x7f60ac0aec00 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b0f34700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f60ac15c6f0 0x7f60ac0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60abfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f60ac06ea70 0x7f60ac0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57498/0 (socket says 172.21.15.104:57498) 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60abfff700 1 -- 172.21.15.104:0/2503429975 learned_addr learned my addr 172.21.15.104:0/2503429975 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b1735700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b0f34700 1 -- 172.21.15.104:0/2503429975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 msgr2=0x7f60ac0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.309+0000 7f60b0f34700 1 -- 172.21.15.104:0/2503429975 >> v1:172.21.15.104:6789/0 conn(0x7f60ac06ea70 legacy=0x7f60ac0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b0f34700 1 -- 172.21.15.104:0/2503429975 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac06e550 con 0x7f60ac15c6f0 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60ab7fe700 1 -- 172.21.15.104:0/2503429975 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236052466 0 0) 0x7f60ac0ae7e0 con 0x7f60ac06ea70 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b1735700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b1070 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009820 tx=0x7f60a0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c532c4d1ff4d42 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60ab7fe700 1 -- 172.21.15.104:0/2503429975 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0005e30 con 0x7f60ac15c6f0 2023-06-10T18:48:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60ab7fe700 1 -- 172.21.15.104:0/2503429975 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0006030 con 0x7f60ac15c6f0 2023-06-10T18:48:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60ab7fe700 1 -- 172.21.15.104:0/2503429975 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00063e0 con 0x7f60ac15c6f0 2023-06-10T18:48:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 -- 172.21.15.104:0/2503429975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 msgr2=0x7f60ac0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b1070 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009820 tx=0x7f60a0009b30 comp rx=0 tx=0).stop 2023-06-10T18:48:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 -- 172.21.15.104:0/2503429975 shutdown_connections 2023-06-10T18:48:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b1070 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 --2- 172.21.15.104:0/2503429975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 -- 172.21.15.104:0/2503429975 >> 172.21.15.104:0/2503429975 conn(0x7f60ac06b8a0 msgr2=0x7f60ac0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 -- 172.21.15.104:0/2503429975 shutdown_connections 2023-06-10T18:48:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.310+0000 7f60b3198700 1 -- 172.21.15.104:0/2503429975 wait complete. 2023-06-10T18:48:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.312+0000 7f60b3198700 1 Processor -- start 2023-06-10T18:48:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.312+0000 7f60b3198700 1 -- start start 2023-06-10T18:48:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 0x7f60ac149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac14c2d0 con 0x7f60ac15c6f0 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac150630 con 0x7f60ac06ea70 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b3198700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac14a300 con 0x7f60ac0aec00 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b1735700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b1735700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59680/0 (socket says 172.21.15.104:59680) 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b1735700 1 -- 172.21.15.104:0/2880566574 learned_addr learned my addr 172.21.15.104:0/2880566574 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60abfff700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 0x7f60ac149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b1735700 1 -- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 msgr2=0x7f60ac0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.313+0000 7f60b1735700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b1735700 1 -- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 msgr2=0x7f60ac149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b1735700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 0x7f60ac149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b1735700 1 -- 172.21.15.104:0/2880566574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0009430 con 0x7f60ac15c6f0 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 0x7f60ac149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b1735700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f60a400ce80 tx=0x7f60a400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400ae20 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4015440 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac14b6c0 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a40157f0 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.314+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac14bc80 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.315+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac14c480 con 0x7f60ac15c6f0 2023-06-10T18:48:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.315+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f60a400d430 con 0x7f60ac15c6f0 2023-06-10T18:48:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.316+0000 7f60a97fa700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 0x7f609006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.316+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f60a4089b50 con 0x7f60ac15c6f0 2023-06-10T18:48:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.316+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 0x7f609006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.316+0000 7f60b0f34700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 0x7f609006d1e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f60ac14e520 tx=0x7f60a0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:46.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.320+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f60ac14c480 con 0x7f60ac15c6f0 2023-06-10T18:48:46.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:46.551+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60ac074050 con 0x7f60ac15c6f0 2023-06-10T18:48:47.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.163+0000 7f60a97fa700 1 -- 172.21.15.104:0/2880566574 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60a4054ba0 con 0x7f60ac15c6f0 2023-06-10T18:48:47.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 msgr2=0x7f609006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 0x7f609006d1e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f60ac14e520 tx=0x7f60a0008070 comp rx=0 tx=0).stop 2023-06-10T18:48:47.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 msgr2=0x7f60ac0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f60a400ce80 tx=0x7f60a400cf60 comp rx=0 tx=0).stop 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 shutdown_connections 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f609006ad30 0x7f609006d1e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60ac06ea70 0x7f60ac149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60ac15c6f0 0x7f60ac0b05e0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 --2- 172.21.15.104:0/2880566574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f60ac0aec00 0x7f60ac0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.165+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 >> 172.21.15.104:0/2880566574 conn(0x7f60ac06b8a0 msgr2=0x7f60ac069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:47.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.166+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 shutdown_connections 2023-06-10T18:48:47.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.166+0000 7f60b3198700 1 -- 172.21.15.104:0/2880566574 wait complete. 2023-06-10T18:48:47.186 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:48:47.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:47.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.304+0000 7facc58d8700 1 Processor -- start 2023-06-10T18:48:47.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.304+0000 7facc58d8700 1 -- start start 2023-06-10T18:48:47.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facc58d8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facc00ce390 0x7facc0160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facc58d8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 0x7facc00cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facc58d8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00d0340 con 0x7facc006c720 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facc58d8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc00d1660 con 0x7facc00ce390 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facc58d8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc015cb60 con 0x7facc00cae10 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facc00ce390 0x7facc0160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facc00ce390 0x7facc0160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59684/0 (socket says 172.21.15.104:59684) 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 -- 172.21.15.104:0/3954551533 learned_addr learned my addr 172.21.15.104:0/3954551533 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:47.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbf7fe700 1 --2- 172.21.15.104:0/3954551533 >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 0x7facc00cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 -- 172.21.15.104:0/3954551533 >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 msgr2=0x7facc00cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 --2- 172.21.15.104:0/3954551533 >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 0x7facc00cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 -- 172.21.15.104:0/3954551533 >> v1:172.21.15.114:6789/0 conn(0x7facc006c720 legacy=0x7facc00ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbeffd700 1 -- 172.21.15.104:0/3954551533 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0161230 con 0x7facc00ce390 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.305+0000 7facbf7fe700 1 --2- 172.21.15.104:0/3954551533 >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 0x7facc00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:47.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facbeffd700 1 --2- 172.21.15.104:0/3954551533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc0160cf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7facc006b1a0 tx=0x7facb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65350a2565b3eaf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facbdffb700 1 -- 172.21.15.104:0/3954551533 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4004230 con 0x7facc00ce390 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facbdffb700 1 -- 172.21.15.104:0/3954551533 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb4004390 con 0x7facc00ce390 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facbdffb700 1 -- 172.21.15.104:0/3954551533 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb40036a0 con 0x7facc00ce390 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 -- 172.21.15.104:0/3954551533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 msgr2=0x7facc0160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 --2- 172.21.15.104:0/3954551533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc0160cf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7facc006b1a0 tx=0x7facb4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 -- 172.21.15.104:0/3954551533 shutdown_connections 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 --2- 172.21.15.104:0/3954551533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc0160cf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 --2- 172.21.15.104:0/3954551533 >> v2:172.21.15.114:3300/0 conn(0x7facc00cae10 0x7facc00cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 -- 172.21.15.104:0/3954551533 >> 172.21.15.104:0/3954551533 conn(0x7facc0069e70 msgr2=0x7facc006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:47.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 -- 172.21.15.104:0/3954551533 shutdown_connections 2023-06-10T18:48:47.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.306+0000 7facc58d8700 1 -- 172.21.15.104:0/3954551533 wait complete. 2023-06-10T18:48:47.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.307+0000 7facc58d8700 1 Processor -- start 2023-06-10T18:48:47.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 -- start start 2023-06-10T18:48:47.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facc006c720 0x7facc0157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 0x7facc00c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00b6340 con 0x7facc00ce390 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00b5a10 con 0x7facc00cae10 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facc58d8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc00b5cf0 con 0x7facc006c720 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.308+0000 7facbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59694/0 (socket says 172.21.15.104:59694) 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 -- 172.21.15.104:0/2543662165 learned_addr learned my addr 172.21.15.104:0/2543662165 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:47.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbeffd700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facc006c720 0x7facc0157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbe7fc700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 0x7facc00c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 -- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facc006c720 msgr2=0x7facc0157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facc006c720 0x7facc0157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 -- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 msgr2=0x7facc00c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 0x7facc00c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 -- 172.21.15.104:0/2543662165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb4009550 con 0x7facc00ce390 2023-06-10T18:48:47.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbe7fc700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 0x7facc00c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facbf7fe700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7facb800cef0 tx=0x7facb800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb800adb0 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb8015440 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc00b5fd0 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80157f0 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.309+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc00b5700 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.310+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc0074050 con 0x7facc00ce390 2023-06-10T18:48:47.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.310+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7facb800d430 con 0x7facc00ce390 2023-06-10T18:48:47.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.311+0000 7faca7fff700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 0x7facac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.311+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7facb8089a90 con 0x7facc00ce390 2023-06-10T18:48:47.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.311+0000 7facbeffd700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 0x7facac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.312+0000 7facbeffd700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 0x7facac06d190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7facb4009130 tx=0x7facb4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.315+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7facb8054a90 con 0x7facc00ce390 2023-06-10T18:48:47.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.542+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc00b6480 con 0x7facc00ce390 2023-06-10T18:48:47.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.542+0000 7faca7fff700 1 -- 172.21.15.104:0/2543662165 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7facc00b6480 con 0x7facc00ce390 2023-06-10T18:48:47.549 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:47.549 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 msgr2=0x7facac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 0x7facac06d190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7facb4009130 tx=0x7facb4008030 comp rx=0 tx=0).stop 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 msgr2=0x7facc00cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7facb800cef0 tx=0x7facb800cf20 comp rx=0 tx=0).stop 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 shutdown_connections 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc00ce390 0x7facc00cbe60 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facc006c720 0x7facc0157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facac06ace0 0x7facac06d190 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 --2- 172.21.15.104:0/2543662165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc00cae10 0x7facc00c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 >> 172.21.15.104:0/2543662165 conn(0x7facc0069e70 msgr2=0x7facc00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.544+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 shutdown_connections 2023-06-10T18:48:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.545+0000 7facc58d8700 1 -- 172.21.15.104:0/2543662165 wait complete. 2023-06-10T18:48:47.552 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:47.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:48:47.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.676+0000 7f63deaaf700 1 Processor -- start 2023-06-10T18:48:47.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.676+0000 7f63deaaf700 1 -- start start 2023-06-10T18:48:47.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63deaaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e0e0 0x7f63d806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63deaaf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f63d8069e70 0x7f63d80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63deaaf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d806d290 con 0x7f63d806e4b0 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63deaaf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d8071640 con 0x7f63d8069e70 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63deaaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80aef00 con 0x7f63d806e0e0 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f63d8069e70 0x7f63d80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63d7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f63d806e4b0 0x7f63d80c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57528/0 (socket says 172.21.15.104:57528) 2023-06-10T18:48:47.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63d7fff700 1 -- 172.21.15.104:0/731638218 learned_addr learned my addr 172.21.15.104:0/731638218 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dc84b700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e0e0 0x7f63d806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 -- 172.21.15.104:0/731638218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e0e0 msgr2=0x7f63d806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e0e0 0x7f63d806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 -- 172.21.15.104:0/731638218 >> v1:172.21.15.104:6789/0 conn(0x7f63d806e4b0 legacy=0x7f63d80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 -- 172.21.15.104:0/731638218 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80ca2d0 con 0x7f63d8069e70 2023-06-10T18:48:47.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63d77fe700 1 -- 172.21.15.104:0/731638218 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098835568 0 0) 0x7f63d806d290 con 0x7f63d806e4b0 2023-06-10T18:48:47.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63dd04c700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d8069e70 0x7f63d80ce9f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009d90 tx=0x7f63d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3dfc35edfb383d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.677+0000 7f63d77fe700 1 -- 172.21.15.104:0/731638218 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d000e070 con 0x7f63d8069e70 2023-06-10T18:48:47.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63d77fe700 1 -- 172.21.15.104:0/731638218 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d000bbd0 con 0x7f63d8069e70 2023-06-10T18:48:47.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63d77fe700 1 -- 172.21.15.104:0/731638218 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0012430 con 0x7f63d8069e70 2023-06-10T18:48:47.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 -- 172.21.15.104:0/731638218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d8069e70 msgr2=0x7f63d80ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d8069e70 0x7f63d80ce9f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009d90 tx=0x7f63d000b330 comp rx=0 tx=0).stop 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 -- 172.21.15.104:0/731638218 shutdown_connections 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d8069e70 0x7f63d80ce9f0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 --2- 172.21.15.104:0/731638218 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e0e0 0x7f63d806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 -- 172.21.15.104:0/731638218 >> 172.21.15.104:0/731638218 conn(0x7f63d806b8a0 msgr2=0x7f63d80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 -- 172.21.15.104:0/731638218 shutdown_connections 2023-06-10T18:48:47.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.678+0000 7f63deaaf700 1 -- 172.21.15.104:0/731638218 wait complete. 2023-06-10T18:48:47.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.679+0000 7f63deaaf700 1 Processor -- start 2023-06-10T18:48:47.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.679+0000 7f63deaaf700 1 -- start start 2023-06-10T18:48:47.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 0x7f63d80b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e4b0 0x7f63d81f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80afbf0 con 0x7f63d806e0e0 2023-06-10T18:48:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80afed0 con 0x7f63d8069e70 2023-06-10T18:48:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63deaaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d80b01b0 con 0x7f63d806e4b0 2023-06-10T18:48:47.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59706/0 (socket says 172.21.15.104:59706) 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63d7fff700 1 -- 172.21.15.104:0/4002083702 learned_addr learned my addr 172.21.15.104:0/4002083702 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63dc84b700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 0x7f63d80b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.680+0000 7f63dd04c700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e4b0 0x7f63d81f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 -- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e4b0 msgr2=0x7f63d81f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e4b0 0x7f63d81f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 -- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 msgr2=0x7f63d80b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 0x7f63d80b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 -- 172.21.15.104:0/4002083702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0009a70 con 0x7f63d806e0e0 2023-06-10T18:48:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63dc84b700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 0x7f63d80b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d7fff700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f63c40011e0 tx=0x7f63c400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4004c80 con 0x7f63d806e0e0 2023-06-10T18:48:47.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4004e00 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d80b0490 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4007310 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.681+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d80b3690 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.682+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d80b09b0 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.682+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f63c4007470 con 0x7f63d806e0e0 2023-06-10T18:48:47.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.683+0000 7f63d57fa700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 0x7f63bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.683+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f63c4089430 con 0x7f63d806e0e0 2023-06-10T18:48:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.683+0000 7f63dc84b700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 0x7f63bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.683+0000 7f63dc84b700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 0x7f63bc06d190 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63d80b3d70 tx=0x7f63cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:47.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.687+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f63d80b09b0 con 0x7f63d806e0e0 2023-06-10T18:48:47.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.876+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d80b09b0 con 0x7f63d806e0e0 2023-06-10T18:48:47.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.877+0000 7f63d57fa700 1 -- 172.21.15.104:0/4002083702 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+18603 (secure 0 0 0) 0x7f63d80b09b0 con 0x7f63d806e0e0 2023-06-10T18:48:47.883 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:47.883 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":68,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:48:45.150166+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:48:47.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 msgr2=0x7f63bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 0x7f63bc06d190 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f63d80b3d70 tx=0x7f63cc008040 comp rx=0 tx=0).stop 2023-06-10T18:48:47.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 msgr2=0x7f63d80b1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:47.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f63c40011e0 tx=0x7f63c400cae0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 shutdown_connections 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f63bc06ace0 0x7f63bc06d190 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63d8069e70 0x7f63d80b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63d806e4b0 0x7f63d81f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 --2- 172.21.15.104:0/4002083702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d806e0e0 0x7f63d80b1450 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 >> 172.21.15.104:0/4002083702 conn(0x7f63d806b8a0 msgr2=0x7f63d80ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 shutdown_connections 2023-06-10T18:48:47.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:47.879+0000 7f63deaaf700 1 -- 172.21.15.104:0/4002083702 wait complete. 2023-06-10T18:48:47.897 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:48:47.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:48:48.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 Processor -- start 2023-06-10T18:48:48.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 -- start start 2023-06-10T18:48:48.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f632406e0e0 0x7f632406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6324069e70 0x7f63240ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406d290 con 0x7f632406e4b0 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6324071640 con 0x7f632406e0e0 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.011+0000 7f632a171700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240aef00 con 0x7f6324069e70 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f632406e0e0 0x7f632406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f6322ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f632406e4b0 0x7f63240c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57540/0 (socket says 172.21.15.104:57540) 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f6322ffd700 1 -- 172.21.15.104:0/116479690 learned_addr learned my addr 172.21.15.104:0/116479690 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:48.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f6323fff700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6324069e70 0x7f63240ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 -- 172.21.15.104:0/116479690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6324069e70 msgr2=0x7f63240ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6324069e70 0x7f63240ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 -- 172.21.15.104:0/116479690 >> v1:172.21.15.104:6789/0 conn(0x7f632406e4b0 legacy=0x7f63240c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 -- 172.21.15.104:0/116479690 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240ca2d0 con 0x7f632406e0e0 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63227fc700 1 -- 172.21.15.104:0/116479690 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504979635 0 0) 0x7f632406d290 con 0x7f632406e4b0 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63237fe700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f632406cd50 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6314009820 tx=0x7f6314009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaea5d51d5a4c78 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63227fc700 1 -- 172.21.15.104:0/116479690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314005e30 con 0x7f632406e0e0 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63227fc700 1 -- 172.21.15.104:0/116479690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314006030 con 0x7f632406e0e0 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.012+0000 7f63227fc700 1 -- 172.21.15.104:0/116479690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140063e0 con 0x7f632406e0e0 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 -- 172.21.15.104:0/116479690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 msgr2=0x7f632406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f632406cd50 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6314009820 tx=0x7f6314009b30 comp rx=0 tx=0).stop 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 -- 172.21.15.104:0/116479690 shutdown_connections 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f632406cd50 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 --2- 172.21.15.104:0/116479690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6324069e70 0x7f63240ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 -- 172.21.15.104:0/116479690 >> 172.21.15.104:0/116479690 conn(0x7f632406b8a0 msgr2=0x7f63240cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:48.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 -- 172.21.15.104:0/116479690 shutdown_connections 2023-06-10T18:48:48.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.013+0000 7f632a171700 1 -- 172.21.15.104:0/116479690 wait complete. 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.014+0000 7f632a171700 1 Processor -- start 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 -- start start 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6324069e70 0x7f63240af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f632406e4b0 0x7f63240c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63240b1320 con 0x7f632406e0e0 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63240c87b0 con 0x7f6324069e70 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.015+0000 7f632a171700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240b15c0 con 0x7f632406e4b0 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59718/0 (socket says 172.21.15.104:59718) 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 -- 172.21.15.104:0/242994388 learned_addr learned my addr 172.21.15.104:0/242994388 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6323fff700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f632406e4b0 0x7f63240c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f63237fe700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6324069e70 0x7f63240af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 -- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f632406e4b0 msgr2=0x7f63240c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f632406e4b0 0x7f63240c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 -- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6324069e70 msgr2=0x7f63240af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6324069e70 0x7f63240af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 -- 172.21.15.104:0/242994388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314009430 con 0x7f632406e0e0 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f6322ffd700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f63180011e0 tx=0x7f631800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318004c80 con 0x7f632406e0e0 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6318004e00 con 0x7f632406e0e0 2023-06-10T18:48:48.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63240b18a0 con 0x7f632406e0e0 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318007310 con 0x7f632406e0e0 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.016+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240c8480 con 0x7f632406e0e0 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.017+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63240b1180 con 0x7f632406e0e0 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.018+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6318007470 con 0x7f632406e0e0 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.018+0000 7f630bfff700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 0x7f631006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.018+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6318089630 con 0x7f632406e0e0 2023-06-10T18:48:48.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.019+0000 7f63237fe700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 0x7f631006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.019+0000 7f63237fe700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 0x7f631006d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6314013040 tx=0x7f6314008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.022+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6318054680 con 0x7f632406e0e0 2023-06-10T18:48:48.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.204+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f632406d7a0 con 0x7f631006ace0 2023-06-10T18:48:48.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.214+0000 7f630bfff700 1 -- 172.21.15.104:0/242994388 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311729 (secure 0 0 0) 0x7f6314005d30 con 0x7f631006ace0 2023-06-10T18:48:48.226 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:48.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 msgr2=0x7f631006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 0x7f631006d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6314013040 tx=0x7f6314008070 comp rx=0 tx=0).stop 2023-06-10T18:48:48.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 msgr2=0x7f63240c77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f63180011e0 tx=0x7f631800ca30 comp rx=0 tx=0).stop 2023-06-10T18:48:48.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 shutdown_connections 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f631006ace0 0x7f631006d190 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6324069e70 0x7f63240af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f632406e4b0 0x7f63240c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 --2- 172.21.15.104:0/242994388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f632406e0e0 0x7f63240c77c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 >> 172.21.15.104:0/242994388 conn(0x7f632406b8a0 msgr2=0x7f63240acdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 shutdown_connections 2023-06-10T18:48:48.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.217+0000 7f632a171700 1 -- 172.21.15.104:0/242994388 wait complete. 2023-06-10T18:48:48.230 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:48:48.235 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2023-06-10T18:48:47.425603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223208,"kb_used_data":7592,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754751512,"statfs":{"total":773094113280,"available":772865548288,"internally_reserved":0,"allocated":7774208,"data_stored":4238452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677658"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161363+0000","last_change":"2023-06-10T18:48:44.172499+0000","last_active":"2023-06-10T18:48:45.161363+0000","last_peered":"2023-06-10T18:48:45.161363+0000","last_clean":"2023-06-10T18:48:45.161363+0000","last_became_active":"2023-06-10T18:48:44.172204+0000","last_became_peered":"2023-06-10T18:48:44.172204+0000","last_unstale":"2023-06-10T18:48:45.161363+0000","last_undegraded":"2023-06-10T18:48:45.161363+0000","last_fullsized":"2023-06-10T18:48:45.161363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:11.376824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189186+0000","last_change":"2023-06-10T18:48:44.182176+0000","last_active":"2023-06-10T18:48:44.189186+0000","last_peered":"2023-06-10T18:48:44.189186+0000","last_clean":"2023-06-10T18:48:44.189186+0000","last_became_active":"2023-06-10T18:48:44.181840+0000","last_became_peered":"2023-06-10T18:48:44.181840+0000","last_unstale":"2023-06-10T18:48:44.189186+0000","last_undegraded":"2023-06-10T18:48:44.189186+0000","last_fullsized":"2023-06-10T18:48:44.189186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:13:01.836306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349250+0000","last_change":"2023-06-10T18:48:44.164500+0000","last_active":"2023-06-10T18:48:46.349250+0000","last_peered":"2023-06-10T18:48:46.349250+0000","last_clean":"2023-06-10T18:48:46.349250+0000","last_became_active":"2023-06-10T18:48:44.164061+0000","last_became_peered":"2023-06-10T18:48:44.164061+0000","last_unstale":"2023-06-10T18:48:46.349250+0000","last_undegraded":"2023-06-10T18:48:46.349250+0000","last_fullsized":"2023-06-10T18:48:46.349250+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:04.927420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188775+0000","last_change":"2023-06-10T18:48:44.183737+0000","last_active":"2023-06-10T18:48:44.188775+0000","last_peered":"2023-06-10T18:48:44.188775+0000","last_clean":"2023-06-10T18:48:44.188775+0000","last_became_active":"2023-06-10T18:48:44.183489+0000","last_became_peered":"2023-06-10T18:48:44.183489+0000","last_unstale":"2023-06-10T18:48:44.188775+0000","last_undegraded":"2023-06-10T18:48:44.188775+0000","last_fullsized":"2023-06-10T18:48:44.188775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:04:49.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160893+0000","last_change":"2023-06-10T18:48:44.170182+0000","last_active":"2023-06-10T18:48:45.160893+0000","last_peered":"2023-06-10T18:48:45.160893+0000","last_clean":"2023-06-10T18:48:45.160893+0000","last_became_active":"2023-06-10T18:48:44.169893+0000","last_became_peered":"2023-06-10T18:48:44.169893+0000","last_unstale":"2023-06-10T18:48:45.160893+0000","last_undegraded":"2023-06-10T18:48:45.160893+0000","last_fullsized":"2023-06-10T18:48:45.160893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:25:07.136698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188614+0000","last_change":"2023-06-10T18:48:44.179750+0000","last_active":"2023-06-10T18:48:44.188614+0000","last_peered":"2023-06-10T18:48:44.188614+0000","last_clean":"2023-06-10T18:48:44.188614+0000","last_became_active":"2023-06-10T18:48:44.179464+0000","last_became_peered":"2023-06-10T18:48:44.179464+0000","last_unstale":"2023-06-10T18:48:44.188614+0000","last_undegraded":"2023-06-10T18:48:44.188614+0000","last_fullsized":"2023-06-10T18:48:44.188614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:44.552384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162159+0000","last_change":"2023-06-10T18:48:44.175621+0000","last_active":"2023-06-10T18:48:45.162159+0000","last_peered":"2023-06-10T18:48:45.162159+0000","last_clean":"2023-06-10T18:48:45.162159+0000","last_became_active":"2023-06-10T18:48:44.175335+0000","last_became_peered":"2023-06-10T18:48:44.175335+0000","last_unstale":"2023-06-10T18:48:45.162159+0000","last_undegraded":"2023-06-10T18:48:45.162159+0000","last_fullsized":"2023-06-10T18:48:45.162159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:09:05.696574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188738+0000","last_change":"2023-06-10T18:48:44.180429+0000","last_active":"2023-06-10T18:48:44.188738+0000","last_peered":"2023-06-10T18:48:44.188738+0000","last_clean":"2023-06-10T18:48:44.188738+0000","last_became_active":"2023-06-10T18:48:44.179965+0000","last_became_peered":"2023-06-10T18:48:44.179965+0000","last_unstale":"2023-06-10T18:48:44.188738+0000","last_undegraded":"2023-06-10T18:48:44.188738+0000","last_fullsized":"2023-06-10T18:48:44.188738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:38:16.331016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161307+0000","last_change":"2023-06-10T18:48:44.170136+0000","last_active":"2023-06-10T18:48:45.161307+0000","last_peered":"2023-06-10T18:48:45.161307+0000","last_clean":"2023-06-10T18:48:45.161307+0000","last_became_active":"2023-06-10T18:48:44.168891+0000","last_became_peered":"2023-06-10T18:48:44.168891+0000","last_unstale":"2023-06-10T18:48:45.161307+0000","last_undegraded":"2023-06-10T18:48:45.161307+0000","last_fullsized":"2023-06-10T18:48:45.161307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:31.686125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756627+0000","last_change":"2023-06-10T18:48:44.168880+0000","last_active":"2023-06-10T18:48:45.756627+0000","last_peered":"2023-06-10T18:48:45.756627+0000","last_clean":"2023-06-10T18:48:45.756627+0000","last_became_active":"2023-06-10T18:48:44.168567+0000","last_became_peered":"2023-06-10T18:48:44.168567+0000","last_unstale":"2023-06-10T18:48:45.756627+0000","last_undegraded":"2023-06-10T18:48:45.756627+0000","last_fullsized":"2023-06-10T18:48:45.756627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:15.117741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161067+0000","last_change":"2023-06-10T18:48:44.170830+0000","last_active":"2023-06-10T18:48:45.161067+0000","last_peered":"2023-06-10T18:48:45.161067+0000","last_clean":"2023-06-10T18:48:45.161067+0000","last_became_active":"2023-06-10T18:48:44.170541+0000","last_became_peered":"2023-06-10T18:48:44.170541+0000","last_unstale":"2023-06-10T18:48:45.161067+0000","last_undegraded":"2023-06-10T18:48:45.161067+0000","last_fullsized":"2023-06-10T18:48:45.161067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:55:55.933554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.180420+0000","last_change":"2023-06-10T18:48:44.175356+0000","last_active":"2023-06-10T18:48:44.180420+0000","last_peered":"2023-06-10T18:48:44.180420+0000","last_clean":"2023-06-10T18:48:44.180420+0000","last_became_active":"2023-06-10T18:48:44.174639+0000","last_became_peered":"2023-06-10T18:48:44.174639+0000","last_unstale":"2023-06-10T18:48:44.180420+0000","last_undegraded":"2023-06-10T18:48:44.180420+0000","last_fullsized":"2023-06-10T18:48:44.180420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:49:00.933651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631938+0000","last_change":"2023-06-10T18:48:44.182041+0000","last_active":"2023-06-10T18:48:45.631938+0000","last_peered":"2023-06-10T18:48:45.631938+0000","last_clean":"2023-06-10T18:48:45.631938+0000","last_became_active":"2023-06-10T18:48:44.181832+0000","last_became_peered":"2023-06-10T18:48:44.181832+0000","last_unstale":"2023-06-10T18:48:45.631938+0000","last_undegraded":"2023-06-10T18:48:45.631938+0000","last_fullsized":"2023-06-10T18:48:45.631938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:50.498103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160931+0000","last_change":"2023-06-10T18:48:44.169875+0000","last_active":"2023-06-10T18:48:45.160931+0000","last_peered":"2023-06-10T18:48:45.160931+0000","last_clean":"2023-06-10T18:48:45.160931+0000","last_became_active":"2023-06-10T18:48:44.169272+0000","last_became_peered":"2023-06-10T18:48:44.169272+0000","last_unstale":"2023-06-10T18:48:45.160931+0000","last_undegraded":"2023-06-10T18:48:45.160931+0000","last_fullsized":"2023-06-10T18:48:45.160931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:38.787140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631742+0000","last_change":"2023-06-10T18:48:44.166899+0000","last_active":"2023-06-10T18:48:45.631742+0000","last_peered":"2023-06-10T18:48:45.631742+0000","last_clean":"2023-06-10T18:48:45.631742+0000","last_became_active":"2023-06-10T18:48:44.166629+0000","last_became_peered":"2023-06-10T18:48:44.166629+0000","last_unstale":"2023-06-10T18:48:45.631742+0000","last_undegraded":"2023-06-10T18:48:45.631742+0000","last_fullsized":"2023-06-10T18:48:45.631742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:26:51.798101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756108+0000","last_change":"2023-06-10T18:48:44.182782+0000","last_active":"2023-06-10T18:48:45.756108+0000","last_peered":"2023-06-10T18:48:45.756108+0000","last_clean":"2023-06-10T18:48:45.756108+0000","last_became_active":"2023-06-10T18:48:44.182497+0000","last_became_peered":"2023-06-10T18:48:44.182497+0000","last_unstale":"2023-06-10T18:48:45.756108+0000","last_undegraded":"2023-06-10T18:48:45.756108+0000","last_fullsized":"2023-06-10T18:48:45.756108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:38.295549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161315+0000","last_change":"2023-06-10T18:48:44.170563+0000","last_active":"2023-06-10T18:48:45.161315+0000","last_peered":"2023-06-10T18:48:45.161315+0000","last_clean":"2023-06-10T18:48:45.161315+0000","last_became_active":"2023-06-10T18:48:44.169922+0000","last_became_peered":"2023-06-10T18:48:44.169922+0000","last_unstale":"2023-06-10T18:48:45.161315+0000","last_undegraded":"2023-06-10T18:48:45.161315+0000","last_fullsized":"2023-06-10T18:48:45.161315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:20:57.438010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188772+0000","last_change":"2023-06-10T18:48:44.183674+0000","last_active":"2023-06-10T18:48:44.188772+0000","last_peered":"2023-06-10T18:48:44.188772+0000","last_clean":"2023-06-10T18:48:44.188772+0000","last_became_active":"2023-06-10T18:48:44.183481+0000","last_became_peered":"2023-06-10T18:48:44.183481+0000","last_unstale":"2023-06-10T18:48:44.188772+0000","last_undegraded":"2023-06-10T18:48:44.188772+0000","last_fullsized":"2023-06-10T18:48:44.188772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:06:59.290841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183452+0000","last_change":"2023-06-10T18:48:44.177093+0000","last_active":"2023-06-10T18:48:44.183452+0000","last_peered":"2023-06-10T18:48:44.183452+0000","last_clean":"2023-06-10T18:48:44.183452+0000","last_became_active":"2023-06-10T18:48:44.176810+0000","last_became_peered":"2023-06-10T18:48:44.176810+0000","last_unstale":"2023-06-10T18:48:44.183452+0000","last_undegraded":"2023-06-10T18:48:44.183452+0000","last_fullsized":"2023-06-10T18:48:44.183452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:36:32.800925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631858+0000","last_change":"2023-06-10T18:48:44.167750+0000","last_active":"2023-06-10T18:48:45.631858+0000","last_peered":"2023-06-10T18:48:45.631858+0000","last_clean":"2023-06-10T18:48:45.631858+0000","last_became_active":"2023-06-10T18:48:44.167495+0000","last_became_peered":"2023-06-10T18:48:44.167495+0000","last_unstale":"2023-06-10T18:48:45.631858+0000","last_undegraded":"2023-06-10T18:48:45.631858+0000","last_fullsized":"2023-06-10T18:48:45.631858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:12.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756473+0000","last_change":"2023-06-10T18:48:44.163065+0000","last_active":"2023-06-10T18:48:45.756473+0000","last_peered":"2023-06-10T18:48:45.756473+0000","last_clean":"2023-06-10T18:48:45.756473+0000","last_became_active":"2023-06-10T18:48:44.162778+0000","last_became_peered":"2023-06-10T18:48:44.162778+0000","last_unstale":"2023-06-10T18:48:45.756473+0000","last_undegraded":"2023-06-10T18:48:45.756473+0000","last_fullsized":"2023-06-10T18:48:45.756473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:55.566902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756426+0000","last_change":"2023-06-10T18:48:44.169732+0000","last_active":"2023-06-10T18:48:45.756426+0000","last_peered":"2023-06-10T18:48:45.756426+0000","last_clean":"2023-06-10T18:48:45.756426+0000","last_became_active":"2023-06-10T18:48:44.168972+0000","last_became_peered":"2023-06-10T18:48:44.168972+0000","last_unstale":"2023-06-10T18:48:45.756426+0000","last_undegraded":"2023-06-10T18:48:45.756426+0000","last_fullsized":"2023-06-10T18:48:45.756426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:38.665134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160584+0000","last_change":"2023-06-10T18:48:44.170224+0000","last_active":"2023-06-10T18:48:45.160584+0000","last_peered":"2023-06-10T18:48:45.160584+0000","last_clean":"2023-06-10T18:48:45.160584+0000","last_became_active":"2023-06-10T18:48:44.169691+0000","last_became_peered":"2023-06-10T18:48:44.169691+0000","last_unstale":"2023-06-10T18:48:45.160584+0000","last_undegraded":"2023-06-10T18:48:45.160584+0000","last_fullsized":"2023-06-10T18:48:45.160584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:36:47.630806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160171+0000","last_change":"2023-06-10T18:48:44.185120+0000","last_active":"2023-06-10T18:48:45.160171+0000","last_peered":"2023-06-10T18:48:45.160171+0000","last_clean":"2023-06-10T18:48:45.160171+0000","last_became_active":"2023-06-10T18:48:44.184720+0000","last_became_peered":"2023-06-10T18:48:44.184720+0000","last_unstale":"2023-06-10T18:48:45.160171+0000","last_undegraded":"2023-06-10T18:48:45.160171+0000","last_fullsized":"2023-06-10T18:48:45.160171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:30.637597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756772+0000","last_change":"2023-06-10T18:48:44.169546+0000","last_active":"2023-06-10T18:48:45.756772+0000","last_peered":"2023-06-10T18:48:45.756772+0000","last_clean":"2023-06-10T18:48:45.756772+0000","last_became_active":"2023-06-10T18:48:44.169251+0000","last_became_peered":"2023-06-10T18:48:44.169251+0000","last_unstale":"2023-06-10T18:48:45.756772+0000","last_undegraded":"2023-06-10T18:48:45.756772+0000","last_fullsized":"2023-06-10T18:48:45.756772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:04.927420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162554+0000","last_change":"2023-06-10T18:48:44.182352+0000","last_active":"2023-06-10T18:48:45.162554+0000","last_peered":"2023-06-10T18:48:45.162554+0000","last_clean":"2023-06-10T18:48:45.162554+0000","last_became_active":"2023-06-10T18:48:44.182061+0000","last_became_peered":"2023-06-10T18:48:44.182061+0000","last_unstale":"2023-06-10T18:48:45.162554+0000","last_undegraded":"2023-06-10T18:48:45.162554+0000","last_fullsized":"2023-06-10T18:48:45.162554+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:44:49.218305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349209+0000","last_change":"2023-06-10T18:48:44.164321+0000","last_active":"2023-06-10T18:48:46.349209+0000","last_peered":"2023-06-10T18:48:46.349209+0000","last_clean":"2023-06-10T18:48:46.349209+0000","last_became_active":"2023-06-10T18:48:44.163730+0000","last_became_peered":"2023-06-10T18:48:44.163730+0000","last_unstale":"2023-06-10T18:48:46.349209+0000","last_undegraded":"2023-06-10T18:48:46.349209+0000","last_fullsized":"2023-06-10T18:48:46.349209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:15.773141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756677+0000","last_change":"2023-06-10T18:48:44.169076+0000","last_active":"2023-06-10T18:48:45.756677+0000","last_peered":"2023-06-10T18:48:45.756677+0000","last_clean":"2023-06-10T18:48:45.756677+0000","last_became_active":"2023-06-10T18:48:44.168675+0000","last_became_peered":"2023-06-10T18:48:44.168675+0000","last_unstale":"2023-06-10T18:48:45.756677+0000","last_undegraded":"2023-06-10T18:48:45.756677+0000","last_fullsized":"2023-06-10T18:48:45.756677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:15.773141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349058+0000","last_change":"2023-06-10T18:48:44.169421+0000","last_active":"2023-06-10T18:48:46.349058+0000","last_peered":"2023-06-10T18:48:46.349058+0000","last_clean":"2023-06-10T18:48:46.349058+0000","last_became_active":"2023-06-10T18:48:44.169191+0000","last_became_peered":"2023-06-10T18:48:44.169191+0000","last_unstale":"2023-06-10T18:48:46.349058+0000","last_undegraded":"2023-06-10T18:48:46.349058+0000","last_fullsized":"2023-06-10T18:48:46.349058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:07:42.169057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162156+0000","last_change":"2023-06-10T18:48:44.180378+0000","last_active":"2023-06-10T18:48:45.162156+0000","last_peered":"2023-06-10T18:48:45.162156+0000","last_clean":"2023-06-10T18:48:45.162156+0000","last_became_active":"2023-06-10T18:48:44.179895+0000","last_became_peered":"2023-06-10T18:48:44.179895+0000","last_unstale":"2023-06-10T18:48:45.162156+0000","last_undegraded":"2023-06-10T18:48:45.162156+0000","last_fullsized":"2023-06-10T18:48:45.162156+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:18:57.325990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161954+0000","last_change":"2023-06-10T18:48:44.169505+0000","last_active":"2023-06-10T18:48:45.161954+0000","last_peered":"2023-06-10T18:48:45.161954+0000","last_clean":"2023-06-10T18:48:45.161954+0000","last_became_active":"2023-06-10T18:48:44.169209+0000","last_became_peered":"2023-06-10T18:48:44.169209+0000","last_unstale":"2023-06-10T18:48:45.161954+0000","last_undegraded":"2023-06-10T18:48:45.161954+0000","last_fullsized":"2023-06-10T18:48:45.161954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:20.666277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188718+0000","last_change":"2023-06-10T18:48:44.180193+0000","last_active":"2023-06-10T18:48:44.188718+0000","last_peered":"2023-06-10T18:48:44.188718+0000","last_clean":"2023-06-10T18:48:44.188718+0000","last_became_active":"2023-06-10T18:48:44.179663+0000","last_became_peered":"2023-06-10T18:48:44.179663+0000","last_unstale":"2023-06-10T18:48:44.188718+0000","last_undegraded":"2023-06-10T18:48:44.188718+0000","last_fullsized":"2023-06-10T18:48:44.188718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:03:27.847439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161335+0000","last_change":"2023-06-10T18:48:44.184312+0000","last_active":"2023-06-10T18:48:45.161335+0000","last_peered":"2023-06-10T18:48:45.161335+0000","last_clean":"2023-06-10T18:48:45.161335+0000","last_became_active":"2023-06-10T18:48:44.183450+0000","last_became_peered":"2023-06-10T18:48:44.183450+0000","last_unstale":"2023-06-10T18:48:45.161335+0000","last_undegraded":"2023-06-10T18:48:45.161335+0000","last_fullsized":"2023-06-10T18:48:45.161335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:42.590220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161767+0000","last_change":"2023-06-10T18:48:43.761342+0000","last_active":"2023-06-10T18:48:45.161767+0000","last_peered":"2023-06-10T18:48:45.161767+0000","last_clean":"2023-06-10T18:48:45.161767+0000","last_became_active":"2023-06-10T18:48:43.760962+0000","last_became_peered":"2023-06-10T18:48:43.760962+0000","last_unstale":"2023-06-10T18:48:45.161767+0000","last_undegraded":"2023-06-10T18:48:45.161767+0000","last_fullsized":"2023-06-10T18:48:45.161767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:02:44.052313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161413+0000","last_change":"2023-06-10T18:48:44.167129+0000","last_active":"2023-06-10T18:48:45.161413+0000","last_peered":"2023-06-10T18:48:45.161413+0000","last_clean":"2023-06-10T18:48:45.161413+0000","last_became_active":"2023-06-10T18:48:44.166777+0000","last_became_peered":"2023-06-10T18:48:44.166777+0000","last_unstale":"2023-06-10T18:48:45.161413+0000","last_undegraded":"2023-06-10T18:48:45.161413+0000","last_fullsized":"2023-06-10T18:48:45.161413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:43.661938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632056+0000","last_change":"2023-06-10T18:48:43.768043+0000","last_active":"2023-06-10T18:48:45.632056+0000","last_peered":"2023-06-10T18:48:45.632056+0000","last_clean":"2023-06-10T18:48:45.632056+0000","last_became_active":"2023-06-10T18:48:43.767749+0000","last_became_peered":"2023-06-10T18:48:43.767749+0000","last_unstale":"2023-06-10T18:48:45.632056+0000","last_undegraded":"2023-06-10T18:48:45.632056+0000","last_fullsized":"2023-06-10T18:48:45.632056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:14.288793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189352+0000","last_change":"2023-06-10T18:48:44.184295+0000","last_active":"2023-06-10T18:48:44.189352+0000","last_peered":"2023-06-10T18:48:44.189352+0000","last_clean":"2023-06-10T18:48:44.189352+0000","last_became_active":"2023-06-10T18:48:44.183805+0000","last_became_peered":"2023-06-10T18:48:44.183805+0000","last_unstale":"2023-06-10T18:48:44.189352+0000","last_undegraded":"2023-06-10T18:48:44.189352+0000","last_fullsized":"2023-06-10T18:48:44.189352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:57:37.332605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756720+0000","last_change":"2023-06-10T18:48:43.772302+0000","last_active":"2023-06-10T18:48:45.756720+0000","last_peered":"2023-06-10T18:48:45.756720+0000","last_clean":"2023-06-10T18:48:45.756720+0000","last_became_active":"2023-06-10T18:48:43.771946+0000","last_became_peered":"2023-06-10T18:48:43.771946+0000","last_unstale":"2023-06-10T18:48:45.756720+0000","last_undegraded":"2023-06-10T18:48:45.756720+0000","last_fullsized":"2023-06-10T18:48:45.756720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:56.612107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161140+0000","last_change":"2023-06-10T18:48:44.170473+0000","last_active":"2023-06-10T18:48:45.161140+0000","last_peered":"2023-06-10T18:48:45.161140+0000","last_clean":"2023-06-10T18:48:45.161140+0000","last_became_active":"2023-06-10T18:48:44.170008+0000","last_became_peered":"2023-06-10T18:48:44.170008+0000","last_unstale":"2023-06-10T18:48:45.161140+0000","last_undegraded":"2023-06-10T18:48:45.161140+0000","last_fullsized":"2023-06-10T18:48:45.161140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:35.785565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.184024+0000","last_change":"2023-06-10T18:48:43.770656+0000","last_active":"2023-06-10T18:48:44.184024+0000","last_peered":"2023-06-10T18:48:44.184024+0000","last_clean":"2023-06-10T18:48:44.184024+0000","last_became_active":"2023-06-10T18:48:43.770267+0000","last_became_peered":"2023-06-10T18:48:43.770267+0000","last_unstale":"2023-06-10T18:48:44.184024+0000","last_undegraded":"2023-06-10T18:48:44.184024+0000","last_fullsized":"2023-06-10T18:48:44.184024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:16:00.969421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162191+0000","last_change":"2023-06-10T18:48:44.167166+0000","last_active":"2023-06-10T18:48:45.162191+0000","last_peered":"2023-06-10T18:48:45.162191+0000","last_clean":"2023-06-10T18:48:45.162191+0000","last_became_active":"2023-06-10T18:48:44.166894+0000","last_became_peered":"2023-06-10T18:48:44.166894+0000","last_unstale":"2023-06-10T18:48:45.162191+0000","last_undegraded":"2023-06-10T18:48:45.162191+0000","last_fullsized":"2023-06-10T18:48:45.162191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:26:17.522851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349222+0000","last_change":"2023-06-10T18:48:43.764019+0000","last_active":"2023-06-10T18:48:46.349222+0000","last_peered":"2023-06-10T18:48:46.349222+0000","last_clean":"2023-06-10T18:48:46.349222+0000","last_became_active":"2023-06-10T18:48:43.763743+0000","last_became_peered":"2023-06-10T18:48:43.763743+0000","last_unstale":"2023-06-10T18:48:46.349222+0000","last_undegraded":"2023-06-10T18:48:46.349222+0000","last_fullsized":"2023-06-10T18:48:46.349222+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:56.612107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756461+0000","last_change":"2023-06-10T18:48:44.169566+0000","last_active":"2023-06-10T18:48:45.756461+0000","last_peered":"2023-06-10T18:48:45.756461+0000","last_clean":"2023-06-10T18:48:45.756461+0000","last_became_active":"2023-06-10T18:48:44.169261+0000","last_became_peered":"2023-06-10T18:48:44.169261+0000","last_unstale":"2023-06-10T18:48:45.756461+0000","last_undegraded":"2023-06-10T18:48:45.756461+0000","last_fullsized":"2023-06-10T18:48:45.756461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:07:42.169057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183175+0000","last_change":"2023-06-10T18:48:43.771888+0000","last_active":"2023-06-10T18:48:44.183175+0000","last_peered":"2023-06-10T18:48:44.183175+0000","last_clean":"2023-06-10T18:48:44.183175+0000","last_became_active":"2023-06-10T18:48:43.771572+0000","last_became_peered":"2023-06-10T18:48:43.771572+0000","last_unstale":"2023-06-10T18:48:44.183175+0000","last_undegraded":"2023-06-10T18:48:44.183175+0000","last_fullsized":"2023-06-10T18:48:44.183175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:06.082899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160742+0000","last_change":"2023-06-10T18:48:44.172292+0000","last_active":"2023-06-10T18:48:45.160742+0000","last_peered":"2023-06-10T18:48:45.160742+0000","last_clean":"2023-06-10T18:48:45.160742+0000","last_became_active":"2023-06-10T18:48:44.172014+0000","last_became_peered":"2023-06-10T18:48:44.172014+0000","last_unstale":"2023-06-10T18:48:45.160742+0000","last_undegraded":"2023-06-10T18:48:45.160742+0000","last_fullsized":"2023-06-10T18:48:45.160742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:23.652595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163229+0000","last_change":"2023-06-10T18:48:43.765302+0000","last_active":"2023-06-10T18:48:44.163229+0000","last_peered":"2023-06-10T18:48:44.163229+0000","last_clean":"2023-06-10T18:48:44.163229+0000","last_became_active":"2023-06-10T18:48:43.765050+0000","last_became_peered":"2023-06-10T18:48:43.765050+0000","last_unstale":"2023-06-10T18:48:44.163229+0000","last_undegraded":"2023-06-10T18:48:44.163229+0000","last_fullsized":"2023-06-10T18:48:44.163229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:55.628445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631780+0000","last_change":"2023-06-10T18:48:44.167451+0000","last_active":"2023-06-10T18:48:45.631780+0000","last_peered":"2023-06-10T18:48:45.631780+0000","last_clean":"2023-06-10T18:48:45.631780+0000","last_became_active":"2023-06-10T18:48:44.167157+0000","last_became_peered":"2023-06-10T18:48:44.167157+0000","last_unstale":"2023-06-10T18:48:45.631780+0000","last_undegraded":"2023-06-10T18:48:45.631780+0000","last_fullsized":"2023-06-10T18:48:45.631780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:18:22.227910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160783+0000","last_change":"2023-06-10T18:48:43.763361+0000","last_active":"2023-06-10T18:48:45.160783+0000","last_peered":"2023-06-10T18:48:45.160783+0000","last_clean":"2023-06-10T18:48:45.160783+0000","last_became_active":"2023-06-10T18:48:43.763093+0000","last_became_peered":"2023-06-10T18:48:43.763093+0000","last_unstale":"2023-06-10T18:48:45.160783+0000","last_undegraded":"2023-06-10T18:48:45.160783+0000","last_fullsized":"2023-06-10T18:48:45.160783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:23.985051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162533+0000","last_change":"2023-06-10T18:48:44.167148+0000","last_active":"2023-06-10T18:48:45.162533+0000","last_peered":"2023-06-10T18:48:45.162533+0000","last_clean":"2023-06-10T18:48:45.162533+0000","last_became_active":"2023-06-10T18:48:44.166861+0000","last_became_peered":"2023-06-10T18:48:44.166861+0000","last_unstale":"2023-06-10T18:48:45.162533+0000","last_undegraded":"2023-06-10T18:48:45.162533+0000","last_fullsized":"2023-06-10T18:48:45.162533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:14.075412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756841+0000","last_change":"2023-06-10T18:48:43.765639+0000","last_active":"2023-06-10T18:48:45.756841+0000","last_peered":"2023-06-10T18:48:45.756841+0000","last_clean":"2023-06-10T18:48:45.756841+0000","last_became_active":"2023-06-10T18:48:43.765356+0000","last_became_peered":"2023-06-10T18:48:43.765356+0000","last_unstale":"2023-06-10T18:48:45.756841+0000","last_undegraded":"2023-06-10T18:48:45.756841+0000","last_fullsized":"2023-06-10T18:48:45.756841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:01:26.026568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161634+0000","last_change":"2023-06-10T18:48:44.183821+0000","last_active":"2023-06-10T18:48:45.161634+0000","last_peered":"2023-06-10T18:48:45.161634+0000","last_clean":"2023-06-10T18:48:45.161634+0000","last_became_active":"2023-06-10T18:48:44.183288+0000","last_became_peered":"2023-06-10T18:48:44.183288+0000","last_unstale":"2023-06-10T18:48:45.161634+0000","last_undegraded":"2023-06-10T18:48:45.161634+0000","last_fullsized":"2023-06-10T18:48:45.161634+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:16.928954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.178984+0000","last_change":"2023-06-10T18:48:43.771799+0000","last_active":"2023-06-10T18:48:44.178984+0000","last_peered":"2023-06-10T18:48:44.178984+0000","last_clean":"2023-06-10T18:48:44.178984+0000","last_became_active":"2023-06-10T18:48:43.771118+0000","last_became_peered":"2023-06-10T18:48:43.771118+0000","last_unstale":"2023-06-10T18:48:44.178984+0000","last_undegraded":"2023-06-10T18:48:44.178984+0000","last_fullsized":"2023-06-10T18:48:44.178984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:06.097636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189172+0000","last_change":"2023-06-10T18:48:44.181778+0000","last_active":"2023-06-10T18:48:44.189172+0000","last_peered":"2023-06-10T18:48:44.189172+0000","last_clean":"2023-06-10T18:48:44.189172+0000","last_became_active":"2023-06-10T18:48:44.181302+0000","last_became_peered":"2023-06-10T18:48:44.181302+0000","last_unstale":"2023-06-10T18:48:44.189172+0000","last_undegraded":"2023-06-10T18:48:44.189172+0000","last_fullsized":"2023-06-10T18:48:44.189172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:48:06.745444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349347+0000","last_change":"2023-06-10T18:48:43.760029+0000","last_active":"2023-06-10T18:48:46.349347+0000","last_peered":"2023-06-10T18:48:46.349347+0000","last_clean":"2023-06-10T18:48:46.349347+0000","last_became_active":"2023-06-10T18:48:43.759791+0000","last_became_peered":"2023-06-10T18:48:43.759791+0000","last_unstale":"2023-06-10T18:48:46.349347+0000","last_undegraded":"2023-06-10T18:48:46.349347+0000","last_fullsized":"2023-06-10T18:48:46.349347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:56:58.400363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161215+0000","last_change":"2023-06-10T18:48:44.166130+0000","last_active":"2023-06-10T18:48:45.161215+0000","last_peered":"2023-06-10T18:48:45.161215+0000","last_clean":"2023-06-10T18:48:45.161215+0000","last_became_active":"2023-06-10T18:48:44.165948+0000","last_became_peered":"2023-06-10T18:48:44.165948+0000","last_unstale":"2023-06-10T18:48:45.161215+0000","last_undegraded":"2023-06-10T18:48:45.161215+0000","last_fullsized":"2023-06-10T18:48:45.161215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:42:53.225771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188819+0000","last_change":"2023-06-10T18:48:43.763986+0000","last_active":"2023-06-10T18:48:44.188819+0000","last_peered":"2023-06-10T18:48:44.188819+0000","last_clean":"2023-06-10T18:48:44.188819+0000","last_became_active":"2023-06-10T18:48:43.763556+0000","last_became_peered":"2023-06-10T18:48:43.763556+0000","last_unstale":"2023-06-10T18:48:44.188819+0000","last_undegraded":"2023-06-10T18:48:44.188819+0000","last_fullsized":"2023-06-10T18:48:44.188819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:18:50.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160620+0000","last_change":"2023-06-10T18:48:44.172599+0000","last_active":"2023-06-10T18:48:45.160620+0000","last_peered":"2023-06-10T18:48:45.160620+0000","last_clean":"2023-06-10T18:48:45.160620+0000","last_became_active":"2023-06-10T18:48:44.172287+0000","last_became_peered":"2023-06-10T18:48:44.172287+0000","last_unstale":"2023-06-10T18:48:45.160620+0000","last_undegraded":"2023-06-10T18:48:45.160620+0000","last_fullsized":"2023-06-10T18:48:45.160620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:52.593526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.175694+0000","last_change":"2023-06-10T18:48:43.760038+0000","last_active":"2023-06-10T18:48:44.175694+0000","last_peered":"2023-06-10T18:48:44.175694+0000","last_clean":"2023-06-10T18:48:44.175694+0000","last_became_active":"2023-06-10T18:48:43.759766+0000","last_became_peered":"2023-06-10T18:48:43.759766+0000","last_unstale":"2023-06-10T18:48:44.175694+0000","last_undegraded":"2023-06-10T18:48:44.175694+0000","last_fullsized":"2023-06-10T18:48:44.175694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:46:02.650662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162418+0000","last_change":"2023-06-10T18:48:44.167863+0000","last_active":"2023-06-10T18:48:45.162418+0000","last_peered":"2023-06-10T18:48:45.162418+0000","last_clean":"2023-06-10T18:48:45.162418+0000","last_became_active":"2023-06-10T18:48:44.167599+0000","last_became_peered":"2023-06-10T18:48:44.167599+0000","last_unstale":"2023-06-10T18:48:45.162418+0000","last_undegraded":"2023-06-10T18:48:45.162418+0000","last_fullsized":"2023-06-10T18:48:45.162418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:32:05.923639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.172931+0000","last_change":"2023-06-10T18:48:43.771772+0000","last_active":"2023-06-10T18:48:44.172931+0000","last_peered":"2023-06-10T18:48:44.172931+0000","last_clean":"2023-06-10T18:48:44.172931+0000","last_became_active":"2023-06-10T18:48:43.771470+0000","last_became_peered":"2023-06-10T18:48:43.771470+0000","last_unstale":"2023-06-10T18:48:44.172931+0000","last_undegraded":"2023-06-10T18:48:44.172931+0000","last_fullsized":"2023-06-10T18:48:44.172931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:24.269341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631728+0000","last_change":"2023-06-10T18:48:44.179966+0000","last_active":"2023-06-10T18:48:45.631728+0000","last_peered":"2023-06-10T18:48:45.631728+0000","last_clean":"2023-06-10T18:48:45.631728+0000","last_became_active":"2023-06-10T18:48:44.179665+0000","last_became_peered":"2023-06-10T18:48:44.179665+0000","last_unstale":"2023-06-10T18:48:45.631728+0000","last_undegraded":"2023-06-10T18:48:45.631728+0000","last_fullsized":"2023-06-10T18:48:45.631728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:40:12.464774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162361+0000","last_change":"2023-06-10T18:48:43.768991+0000","last_active":"2023-06-10T18:48:45.162361+0000","last_peered":"2023-06-10T18:48:45.162361+0000","last_clean":"2023-06-10T18:48:45.162361+0000","last_became_active":"2023-06-10T18:48:43.766837+0000","last_became_peered":"2023-06-10T18:48:43.766837+0000","last_unstale":"2023-06-10T18:48:45.162361+0000","last_undegraded":"2023-06-10T18:48:45.162361+0000","last_fullsized":"2023-06-10T18:48:45.162361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:28.349761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160854+0000","last_change":"2023-06-10T18:48:44.168794+0000","last_active":"2023-06-10T18:48:45.160854+0000","last_peered":"2023-06-10T18:48:45.160854+0000","last_clean":"2023-06-10T18:48:45.160854+0000","last_became_active":"2023-06-10T18:48:44.168334+0000","last_became_peered":"2023-06-10T18:48:44.168334+0000","last_unstale":"2023-06-10T18:48:45.160854+0000","last_undegraded":"2023-06-10T18:48:45.160854+0000","last_fullsized":"2023-06-10T18:48:45.160854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:59:54.615219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756164+0000","last_change":"2023-06-10T18:48:43.768871+0000","last_active":"2023-06-10T18:48:45.756164+0000","last_peered":"2023-06-10T18:48:45.756164+0000","last_clean":"2023-06-10T18:48:45.756164+0000","last_became_active":"2023-06-10T18:48:43.768597+0000","last_became_peered":"2023-06-10T18:48:43.768597+0000","last_unstale":"2023-06-10T18:48:45.756164+0000","last_undegraded":"2023-06-10T18:48:45.756164+0000","last_fullsized":"2023-06-10T18:48:45.756164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:59.649340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632005+0000","last_change":"2023-06-10T18:48:44.166852+0000","last_active":"2023-06-10T18:48:45.632005+0000","last_peered":"2023-06-10T18:48:45.632005+0000","last_clean":"2023-06-10T18:48:45.632005+0000","last_became_active":"2023-06-10T18:48:44.166581+0000","last_became_peered":"2023-06-10T18:48:44.166581+0000","last_unstale":"2023-06-10T18:48:45.632005+0000","last_undegraded":"2023-06-10T18:48:45.632005+0000","last_fullsized":"2023-06-10T18:48:45.632005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:52.840640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188686+0000","last_change":"2023-06-10T18:48:43.771160+0000","last_active":"2023-06-10T18:48:44.188686+0000","last_peered":"2023-06-10T18:48:44.188686+0000","last_clean":"2023-06-10T18:48:44.188686+0000","last_became_active":"2023-06-10T18:48:43.770878+0000","last_became_peered":"2023-06-10T18:48:43.770878+0000","last_unstale":"2023-06-10T18:48:44.188686+0000","last_undegraded":"2023-06-10T18:48:44.188686+0000","last_fullsized":"2023-06-10T18:48:44.188686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:35.708922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162569+0000","last_change":"2023-06-10T18:48:44.174979+0000","last_active":"2023-06-10T18:48:45.162569+0000","last_peered":"2023-06-10T18:48:45.162569+0000","last_clean":"2023-06-10T18:48:45.162569+0000","last_became_active":"2023-06-10T18:48:44.174607+0000","last_became_peered":"2023-06-10T18:48:44.174607+0000","last_unstale":"2023-06-10T18:48:45.162569+0000","last_undegraded":"2023-06-10T18:48:45.162569+0000","last_fullsized":"2023-06-10T18:48:45.162569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:18.079380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162641+0000","last_change":"2023-06-10T18:48:43.767032+0000","last_active":"2023-06-10T18:48:45.162641+0000","last_peered":"2023-06-10T18:48:45.162641+0000","last_clean":"2023-06-10T18:48:45.162641+0000","last_became_active":"2023-06-10T18:48:43.766740+0000","last_became_peered":"2023-06-10T18:48:43.766740+0000","last_unstale":"2023-06-10T18:48:45.162641+0000","last_undegraded":"2023-06-10T18:48:45.162641+0000","last_fullsized":"2023-06-10T18:48:45.162641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:16:16.416115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160597+0000","last_change":"2023-06-10T18:48:44.183869+0000","last_active":"2023-06-10T18:48:45.160597+0000","last_peered":"2023-06-10T18:48:45.160597+0000","last_clean":"2023-06-10T18:48:45.160597+0000","last_became_active":"2023-06-10T18:48:44.183617+0000","last_became_peered":"2023-06-10T18:48:44.183617+0000","last_unstale":"2023-06-10T18:48:45.160597+0000","last_undegraded":"2023-06-10T18:48:45.160597+0000","last_fullsized":"2023-06-10T18:48:45.160597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:50:50.167706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161269+0000","last_change":"2023-06-10T18:48:43.766273+0000","last_active":"2023-06-10T18:48:45.161269+0000","last_peered":"2023-06-10T18:48:45.161269+0000","last_clean":"2023-06-10T18:48:45.161269+0000","last_became_active":"2023-06-10T18:48:43.765568+0000","last_became_peered":"2023-06-10T18:48:43.765568+0000","last_unstale":"2023-06-10T18:48:45.161269+0000","last_undegraded":"2023-06-10T18:48:45.161269+0000","last_fullsized":"2023-06-10T18:48:45.161269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:47.987218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161201+0000","last_change":"2023-06-10T18:48:44.170289+0000","last_active":"2023-06-10T18:48:45.161201+0000","last_peered":"2023-06-10T18:48:45.161201+0000","last_clean":"2023-06-10T18:48:45.161201+0000","last_became_active":"2023-06-10T18:48:44.169663+0000","last_became_peered":"2023-06-10T18:48:44.169663+0000","last_unstale":"2023-06-10T18:48:45.161201+0000","last_undegraded":"2023-06-10T18:48:45.161201+0000","last_fullsized":"2023-06-10T18:48:45.161201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:41.042057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632076+0000","last_change":"2023-06-10T18:48:43.769455+0000","last_active":"2023-06-10T18:48:45.632076+0000","last_peered":"2023-06-10T18:48:45.632076+0000","last_clean":"2023-06-10T18:48:45.632076+0000","last_became_active":"2023-06-10T18:48:43.769166+0000","last_became_peered":"2023-06-10T18:48:43.769166+0000","last_unstale":"2023-06-10T18:48:45.632076+0000","last_undegraded":"2023-06-10T18:48:45.632076+0000","last_fullsized":"2023-06-10T18:48:45.632076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:35:36.279401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160966+0000","last_change":"2023-06-10T18:48:44.163927+0000","last_active":"2023-06-10T18:48:45.160966+0000","last_peered":"2023-06-10T18:48:45.160966+0000","last_clean":"2023-06-10T18:48:45.160966+0000","last_became_active":"2023-06-10T18:48:44.163655+0000","last_became_peered":"2023-06-10T18:48:44.163655+0000","last_unstale":"2023-06-10T18:48:45.160966+0000","last_undegraded":"2023-06-10T18:48:45.160966+0000","last_fullsized":"2023-06-10T18:48:45.160966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:34.155054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161073+0000","last_change":"2023-06-10T18:48:43.766111+0000","last_active":"2023-06-10T18:48:45.161073+0000","last_peered":"2023-06-10T18:48:45.161073+0000","last_clean":"2023-06-10T18:48:45.161073+0000","last_became_active":"2023-06-10T18:48:43.765829+0000","last_became_peered":"2023-06-10T18:48:43.765829+0000","last_unstale":"2023-06-10T18:48:45.161073+0000","last_undegraded":"2023-06-10T18:48:45.161073+0000","last_fullsized":"2023-06-10T18:48:45.161073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:05:00.015824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183187+0000","last_change":"2023-06-10T18:48:44.177244+0000","last_active":"2023-06-10T18:48:44.183187+0000","last_peered":"2023-06-10T18:48:44.183187+0000","last_clean":"2023-06-10T18:48:44.183187+0000","last_became_active":"2023-06-10T18:48:44.176941+0000","last_became_peered":"2023-06-10T18:48:44.176941+0000","last_unstale":"2023-06-10T18:48:44.183187+0000","last_undegraded":"2023-06-10T18:48:44.183187+0000","last_fullsized":"2023-06-10T18:48:44.183187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:52.934944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163575+0000","last_change":"2023-06-10T18:48:43.762463+0000","last_active":"2023-06-10T18:48:44.163575+0000","last_peered":"2023-06-10T18:48:44.163575+0000","last_clean":"2023-06-10T18:48:44.163575+0000","last_became_active":"2023-06-10T18:48:43.762174+0000","last_became_peered":"2023-06-10T18:48:43.762174+0000","last_unstale":"2023-06-10T18:48:44.163575+0000","last_undegraded":"2023-06-10T18:48:44.163575+0000","last_fullsized":"2023-06-10T18:48:44.163575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:12:35.388801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.177484+0000","last_change":"2023-06-10T18:48:44.172046+0000","last_active":"2023-06-10T18:48:44.177484+0000","last_peered":"2023-06-10T18:48:44.177484+0000","last_clean":"2023-06-10T18:48:44.177484+0000","last_became_active":"2023-06-10T18:48:44.171791+0000","last_became_peered":"2023-06-10T18:48:44.171791+0000","last_unstale":"2023-06-10T18:48:44.177484+0000","last_undegraded":"2023-06-10T18:48:44.177484+0000","last_fullsized":"2023-06-10T18:48:44.177484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:58:30.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756381+0000","last_change":"2023-06-10T18:48:43.768949+0000","last_active":"2023-06-10T18:48:45.756381+0000","last_peered":"2023-06-10T18:48:45.756381+0000","last_clean":"2023-06-10T18:48:45.756381+0000","last_became_active":"2023-06-10T18:48:43.768666+0000","last_became_peered":"2023-06-10T18:48:43.768666+0000","last_unstale":"2023-06-10T18:48:45.756381+0000","last_undegraded":"2023-06-10T18:48:45.756381+0000","last_fullsized":"2023-06-10T18:48:45.756381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:38:33.251347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.181738+0000","last_change":"2023-06-10T18:48:44.172457+0000","last_active":"2023-06-10T18:48:44.181738+0000","last_peered":"2023-06-10T18:48:44.181738+0000","last_clean":"2023-06-10T18:48:44.181738+0000","last_became_active":"2023-06-10T18:48:44.171706+0000","last_became_peered":"2023-06-10T18:48:44.171706+0000","last_unstale":"2023-06-10T18:48:44.181738+0000","last_undegraded":"2023-06-10T18:48:44.181738+0000","last_fullsized":"2023-06-10T18:48:44.181738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:35.887246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173169+0000","last_change":"2023-06-10T18:48:43.771036+0000","last_active":"2023-06-10T18:48:44.173169+0000","last_peered":"2023-06-10T18:48:44.173169+0000","last_clean":"2023-06-10T18:48:44.173169+0000","last_became_active":"2023-06-10T18:48:43.770451+0000","last_became_peered":"2023-06-10T18:48:43.770451+0000","last_unstale":"2023-06-10T18:48:44.173169+0000","last_undegraded":"2023-06-10T18:48:44.173169+0000","last_fullsized":"2023-06-10T18:48:44.173169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:14.197352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161158+0000","last_change":"2023-06-10T18:48:44.170127+0000","last_active":"2023-06-10T18:48:45.161158+0000","last_peered":"2023-06-10T18:48:45.161158+0000","last_clean":"2023-06-10T18:48:45.161158+0000","last_became_active":"2023-06-10T18:48:44.168766+0000","last_became_peered":"2023-06-10T18:48:44.168766+0000","last_unstale":"2023-06-10T18:48:45.161158+0000","last_undegraded":"2023-06-10T18:48:45.161158+0000","last_fullsized":"2023-06-10T18:48:45.161158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:44:25.275961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160990+0000","last_change":"2023-06-10T18:48:43.767740+0000","last_active":"2023-06-10T18:48:45.160990+0000","last_peered":"2023-06-10T18:48:45.160990+0000","last_clean":"2023-06-10T18:48:45.160990+0000","last_became_active":"2023-06-10T18:48:43.767216+0000","last_became_peered":"2023-06-10T18:48:43.767216+0000","last_unstale":"2023-06-10T18:48:45.160990+0000","last_undegraded":"2023-06-10T18:48:45.160990+0000","last_fullsized":"2023-06-10T18:48:45.160990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:25.367008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":95,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163903+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:48:44.163903+0000","last_peered":"2023-06-10T18:48:44.163903+0000","last_clean":"2023-06-10T18:48:44.163903+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:48:44.163903+0000","last_undegraded":"2023-06-10T18:48:44.163903+0000","last_fullsized":"2023-06-10T18:48:44.163903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:36:56.258036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349173+0000","last_change":"2023-06-10T18:48:44.173610+0000","last_active":"2023-06-10T18:48:46.349173+0000","last_peered":"2023-06-10T18:48:46.349173+0000","last_clean":"2023-06-10T18:48:46.349173+0000","last_became_active":"2023-06-10T18:48:44.173331+0000","last_became_peered":"2023-06-10T18:48:44.173331+0000","last_unstale":"2023-06-10T18:48:46.349173+0000","last_undegraded":"2023-06-10T18:48:46.349173+0000","last_fullsized":"2023-06-10T18:48:46.349173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:15.117741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173018+0000","last_change":"2023-06-10T18:48:43.770945+0000","last_active":"2023-06-10T18:48:44.173018+0000","last_peered":"2023-06-10T18:48:44.173018+0000","last_clean":"2023-06-10T18:48:44.173018+0000","last_became_active":"2023-06-10T18:48:43.770401+0000","last_became_peered":"2023-06-10T18:48:43.770401+0000","last_unstale":"2023-06-10T18:48:44.173018+0000","last_undegraded":"2023-06-10T18:48:44.173018+0000","last_fullsized":"2023-06-10T18:48:44.173018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:29:06.004191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162247+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:48:45.162247+0000","last_peered":"2023-06-10T18:48:45.162247+0000","last_clean":"2023-06-10T18:48:45.162247+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:48:45.162247+0000","last_undegraded":"2023-06-10T18:48:45.162247+0000","last_fullsized":"2023-06-10T18:48:45.162247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:26:17.186332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349033+0000","last_change":"2023-06-10T18:48:44.181640+0000","last_active":"2023-06-10T18:48:46.349033+0000","last_peered":"2023-06-10T18:48:46.349033+0000","last_clean":"2023-06-10T18:48:46.349033+0000","last_became_active":"2023-06-10T18:48:44.181411+0000","last_became_peered":"2023-06-10T18:48:44.181411+0000","last_unstale":"2023-06-10T18:48:46.349033+0000","last_undegraded":"2023-06-10T18:48:46.349033+0000","last_fullsized":"2023-06-10T18:48:46.349033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:38.665134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162537+0000","last_change":"2023-06-10T18:48:43.764233+0000","last_active":"2023-06-10T18:48:45.162537+0000","last_peered":"2023-06-10T18:48:45.162537+0000","last_clean":"2023-06-10T18:48:45.162537+0000","last_became_active":"2023-06-10T18:48:43.763961+0000","last_became_peered":"2023-06-10T18:48:43.763961+0000","last_unstale":"2023-06-10T18:48:45.162537+0000","last_undegraded":"2023-06-10T18:48:45.162537+0000","last_fullsized":"2023-06-10T18:48:45.162537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:34:49.536536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160873+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:48:45.160873+0000","last_peered":"2023-06-10T18:48:45.160873+0000","last_clean":"2023-06-10T18:48:45.160873+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:48:45.160873+0000","last_undegraded":"2023-06-10T18:48:45.160873+0000","last_fullsized":"2023-06-10T18:48:45.160873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:11.185165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349263+0000","last_change":"2023-06-10T18:48:44.181179+0000","last_active":"2023-06-10T18:48:46.349263+0000","last_peered":"2023-06-10T18:48:46.349263+0000","last_clean":"2023-06-10T18:48:46.349263+0000","last_became_active":"2023-06-10T18:48:44.180946+0000","last_became_peered":"2023-06-10T18:48:44.180946+0000","last_unstale":"2023-06-10T18:48:46.349263+0000","last_undegraded":"2023-06-10T18:48:46.349263+0000","last_fullsized":"2023-06-10T18:48:46.349263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:24.624705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.181572+0000","last_change":"2023-06-10T18:48:43.770788+0000","last_active":"2023-06-10T18:48:44.181572+0000","last_peered":"2023-06-10T18:48:44.181572+0000","last_clean":"2023-06-10T18:48:44.181572+0000","last_became_active":"2023-06-10T18:48:43.770183+0000","last_became_peered":"2023-06-10T18:48:43.770183+0000","last_unstale":"2023-06-10T18:48:44.181572+0000","last_undegraded":"2023-06-10T18:48:44.181572+0000","last_fullsized":"2023-06-10T18:48:44.181572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:47.272036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":109,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162208+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:48:45.162208+0000","last_peered":"2023-06-10T18:48:45.162208+0000","last_clean":"2023-06-10T18:48:45.162208+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:48:45.162208+0000","last_undegraded":"2023-06-10T18:48:45.162208+0000","last_fullsized":"2023-06-10T18:48:45.162208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:03:10.822794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":257,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160167+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:48:45.160167+0000","last_peered":"2023-06-10T18:48:45.160167+0000","last_clean":"2023-06-10T18:48:45.160167+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:48:45.160167+0000","last_undegraded":"2023-06-10T18:48:45.160167+0000","last_fullsized":"2023-06-10T18:48:45.160167+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:58:23.329731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349120+0000","last_change":"2023-06-10T18:48:44.179828+0000","last_active":"2023-06-10T18:48:46.349120+0000","last_peered":"2023-06-10T18:48:46.349120+0000","last_clean":"2023-06-10T18:48:46.349120+0000","last_became_active":"2023-06-10T18:48:44.179591+0000","last_became_peered":"2023-06-10T18:48:44.179591+0000","last_unstale":"2023-06-10T18:48:46.349120+0000","last_undegraded":"2023-06-10T18:48:46.349120+0000","last_fullsized":"2023-06-10T18:48:46.349120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:55.566902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631814+0000","last_change":"2023-06-10T18:48:43.765451+0000","last_active":"2023-06-10T18:48:45.631814+0000","last_peered":"2023-06-10T18:48:45.631814+0000","last_clean":"2023-06-10T18:48:45.631814+0000","last_became_active":"2023-06-10T18:48:43.763981+0000","last_became_peered":"2023-06-10T18:48:43.763981+0000","last_unstale":"2023-06-10T18:48:45.631814+0000","last_undegraded":"2023-06-10T18:48:45.631814+0000","last_fullsized":"2023-06-10T18:48:45.631814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:12.374835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":101,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160261+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:48:45.160261+0000","last_peered":"2023-06-10T18:48:45.160261+0000","last_clean":"2023-06-10T18:48:45.160261+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:48:45.160261+0000","last_undegraded":"2023-06-10T18:48:45.160261+0000","last_fullsized":"2023-06-10T18:48:45.160261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:15:55.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.180366+0000","last_change":"2023-06-10T18:48:44.175016+0000","last_active":"2023-06-10T18:48:44.180366+0000","last_peered":"2023-06-10T18:48:44.180366+0000","last_clean":"2023-06-10T18:48:44.180366+0000","last_became_active":"2023-06-10T18:48:44.174079+0000","last_became_peered":"2023-06-10T18:48:44.174079+0000","last_unstale":"2023-06-10T18:48:44.180366+0000","last_undegraded":"2023-06-10T18:48:44.180366+0000","last_fullsized":"2023-06-10T18:48:44.180366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:56:49.791107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183405+0000","last_change":"2023-06-10T18:48:43.770650+0000","last_active":"2023-06-10T18:48:44.183405+0000","last_peered":"2023-06-10T18:48:44.183405+0000","last_clean":"2023-06-10T18:48:44.183405+0000","last_became_active":"2023-06-10T18:48:43.770271+0000","last_became_peered":"2023-06-10T18:48:43.770271+0000","last_unstale":"2023-06-10T18:48:44.183405+0000","last_undegraded":"2023-06-10T18:48:44.183405+0000","last_fullsized":"2023-06-10T18:48:44.183405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:23.906856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161243+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:48:45.161243+0000","last_peered":"2023-06-10T18:48:45.161243+0000","last_clean":"2023-06-10T18:48:45.161243+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:48:45.161243+0000","last_undegraded":"2023-06-10T18:48:45.161243+0000","last_fullsized":"2023-06-10T18:48:45.161243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:36.895969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161503+0000","last_change":"2023-06-10T18:48:44.174386+0000","last_active":"2023-06-10T18:48:45.161503+0000","last_peered":"2023-06-10T18:48:45.161503+0000","last_clean":"2023-06-10T18:48:45.161503+0000","last_became_active":"2023-06-10T18:48:44.174019+0000","last_became_peered":"2023-06-10T18:48:44.174019+0000","last_unstale":"2023-06-10T18:48:45.161503+0000","last_undegraded":"2023-06-10T18:48:45.161503+0000","last_fullsized":"2023-06-10T18:48:45.161503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:23.594227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756805+0000","last_change":"2023-06-10T18:48:43.771185+0000","last_active":"2023-06-10T18:48:45.756805+0000","last_peered":"2023-06-10T18:48:45.756805+0000","last_clean":"2023-06-10T18:48:45.756805+0000","last_became_active":"2023-06-10T18:48:43.770925+0000","last_became_peered":"2023-06-10T18:48:43.770925+0000","last_unstale":"2023-06-10T18:48:45.756805+0000","last_undegraded":"2023-06-10T18:48:45.756805+0000","last_fullsized":"2023-06-10T18:48:45.756805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:56:58.400363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.164091+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:48:44.164091+0000","last_peered":"2023-06-10T18:48:44.164091+0000","last_clean":"2023-06-10T18:48:44.164091+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:48:44.164091+0000","last_undegraded":"2023-06-10T18:48:44.164091+0000","last_fullsized":"2023-06-10T18:48:44.164091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:45.226104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756706+0000","last_change":"2023-06-10T18:48:44.164870+0000","last_active":"2023-06-10T18:48:45.756706+0000","last_peered":"2023-06-10T18:48:45.756706+0000","last_clean":"2023-06-10T18:48:45.756706+0000","last_became_active":"2023-06-10T18:48:44.164573+0000","last_became_peered":"2023-06-10T18:48:44.164573+0000","last_unstale":"2023-06-10T18:48:45.756706+0000","last_undegraded":"2023-06-10T18:48:45.756706+0000","last_fullsized":"2023-06-10T18:48:45.756706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:24.624705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183116+0000","last_change":"2023-06-10T18:48:43.771408+0000","last_active":"2023-06-10T18:48:44.183116+0000","last_peered":"2023-06-10T18:48:44.183116+0000","last_clean":"2023-06-10T18:48:44.183116+0000","last_became_active":"2023-06-10T18:48:43.770509+0000","last_became_peered":"2023-06-10T18:48:43.770509+0000","last_unstale":"2023-06-10T18:48:44.183116+0000","last_undegraded":"2023-06-10T18:48:44.183116+0000","last_fullsized":"2023-06-10T18:48:44.183116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:21.872946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":93,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173444+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:48:44.173444+0000","last_peered":"2023-06-10T18:48:44.173444+0000","last_clean":"2023-06-10T18:48:44.173444+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:48:44.173444+0000","last_undegraded":"2023-06-10T18:48:44.173444+0000","last_fullsized":"2023-06-10T18:48:44.173444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:14:20.580095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1425408,"data_stored":976604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.46899999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.41399999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.443,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47699999999999998},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67300000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.433}]},{"osd":3,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.497},"min":{"1min":0.41999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73899999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63500000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59099999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.71499999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67700000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55300000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":815104,"data_stored":377720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.43099999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.75700000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52000000000000002,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52400000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.72399999999999998,"15min":0.89000000000000001},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65400000000000003,"15min":0.77400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.316,"15min":0.316},"max":{"1min":0.71299999999999997,"5min":1.252,"15min":1.252},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.69099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.52300000000000002},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.40899999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.623},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":379808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.46300000000000002,"5min":0.753,"15min":0.753},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.33000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.247},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56599999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32600000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55900000000000005,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.373}]},{"osd":4,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.61199999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86799999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":1.008,"15min":1.008},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.43099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77100000000000002,"5min":0.876,"15min":0.876},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1400832,"data_stored":967996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.35099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.435,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.51900000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.373,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.45200000000000001,"5min":1.405,"15min":1.405},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51500000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.372,"15min":0.378},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70999999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39600000000000002,"15min":0.39700000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52100000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.313}]},{"osd":4,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73999999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.58499999999999996}]},{"osd":5,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.999,"15min":0.999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.436,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":1.363,"15min":1.363},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.40600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93500000000000005,"5min":1.034,"15min":1.034},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":835584,"data_stored":387184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.496,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.249}]},{"osd":1,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.34699999999999998},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.28199999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73799999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.42599999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.48599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.47799999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77400000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83699999999999997,"5min":0.872,"15min":0.872},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84999999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.92400000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":831488,"data_stored":385732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47799999999999998,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41399999999999998},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.505,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.377,"15min":0.39700000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41499999999999998,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":1.3,"15min":1.3},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.28399999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53000000000000003,"5min":1.109,"15min":1.109},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44700000000000001,"15min":0.48599999999999999},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.599,"15min":1.599},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60199999999999998,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.746,"5min":1.925,"15min":1.925},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84499999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90100000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.68200000000000005,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77500000000000002,"5min":3.802,"15min":3.802},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67500000000000004,"15min":0.76500000000000001},"min":{"1min":0.438,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78600000000000003,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.628},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81999999999999995,"5min":1.03,"15min":1.03},"last":0.505}]},{"osd":7,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86799999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.47199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82099999999999995,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":811008,"data_stored":377698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.50800000000000001},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.77000000000000002,"5min":1.119,"15min":1.119},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.45600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.67200000000000004,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52200000000000002,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65700000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.50900000000000001},"min":{"1min":0.34999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68700000000000006,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.39900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72899999999999998,"5min":1.042,"15min":1.042},"last":0.624}]},{"osd":5,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72899999999999998,"5min":1.069,"15min":1.069},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51300000000000001,"15min":0.54000000000000004},"min":{"1min":0.34499999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.54800000000000004},"min":{"1min":0.36499999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76200000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81000000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.81599999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":385710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35899999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.72499999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.48499999999999999},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70699999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.69999999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.42399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68999999999999995,"5min":0.745,"15min":0.745},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.47799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64700000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.33700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67200000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.502,"15min":0.505},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61799999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52800000000000002,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69699999999999995,"5min":0.879,"15min":0.879},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69499999999999995,"5min":0.754,"15min":0.754},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:48:48.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:48:48.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.390+0000 7fdeb6eb1700 1 Processor -- start 2023-06-10T18:48:48.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.390+0000 7fdeb6eb1700 1 -- start start 2023-06-10T18:48:48.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 0x7fdeb006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 0x7fdeb00ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb0071640 con 0x7fdeb006e4b0 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb00aef00 con 0x7fdeb00ca100 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb6eb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb006d290 con 0x7fdeb006e0e0 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59728/0 (socket says 172.21.15.104:59728) 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 -- 172.21.15.104:0/496868806 learned_addr learned my addr 172.21.15.104:0/496868806 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb544e700 1 --2- 172.21.15.104:0/496868806 >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 0x7fdeb00ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb4c4d700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 0x7fdeb006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 -- 172.21.15.104:0/496868806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 msgr2=0x7fdeb006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 0x7fdeb006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 -- 172.21.15.104:0/496868806 >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 msgr2=0x7fdeb00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 --2- 172.21.15.104:0/496868806 >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 0x7fdeb00ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeaffff700 1 -- 172.21.15.104:0/496868806 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb006d4c0 con 0x7fdeb006e4b0 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb544e700 1 --2- 172.21.15.104:0/496868806 >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 0x7fdeb00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.391+0000 7fdeb4c4d700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 0x7fdeb006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeaffff700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0071210 tx=0x7fde9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b059bf90e742a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeaf7fe700 1 -- 172.21.15.104:0/496868806 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00b860 con 0x7fdeb006e4b0 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeaf7fe700 1 -- 172.21.15.104:0/496868806 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c00b9c0 con 0x7fdeb006e4b0 2023-06-10T18:48:48.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeaf7fe700 1 -- 172.21.15.104:0/496868806 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00bd50 con 0x7fdeb006e4b0 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/496868806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb006e4b0 msgr2=0x7fdeb00c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0071210 tx=0x7fde9c009c00 comp rx=0 tx=0).stop 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/496868806 shutdown_connections 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/496868806 >> v2:172.21.15.114:3300/0 conn(0x7fdeb00ca100 0x7fdeb00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e0e0 0x7fdeb006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/496868806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb006e4b0 0x7fdeb00c9bc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/496868806 >> 172.21.15.104:0/496868806 conn(0x7fdeb006b8a0 msgr2=0x7fdeb00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:48.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/496868806 shutdown_connections 2023-06-10T18:48:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.392+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/496868806 wait complete. 2023-06-10T18:48:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 Processor -- start 2023-06-10T18:48:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 -- start start 2023-06-10T18:48:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb006e0e0 0x7fdeb00b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e4b0 0x7fdeb0099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb009b1b0 con 0x7fdeb00ca100 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00b39a0 con 0x7fdeb006e0e0 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.394+0000 7fdeb6eb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00b3c80 con 0x7fdeb006e4b0 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59734/0 (socket says 172.21.15.104:59734) 2023-06-10T18:48:48.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 -- 172.21.15.104:0/3056776051 learned_addr learned my addr 172.21.15.104:0/3056776051 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb4c4d700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb006e0e0 0x7fdeb00b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeaffff700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e4b0 0x7fdeb0099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 -- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e4b0 msgr2=0x7fdeb0099570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e4b0 0x7fdeb0099570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 -- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb006e0e0 msgr2=0x7fdeb00b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb006e0e0 0x7fdeb00b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 -- 172.21.15.104:0/3056776051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c009550 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb544e700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdea800cef0 tx=0x7fdea800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800ad00 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea800ae60 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb00b3f60 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8015680 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.395+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb0099ab0 con 0x7fdeb00ca100 2023-06-10T18:48:48.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.396+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb0072410 con 0x7fdeb00ca100 2023-06-10T18:48:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.397+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdea800d430 con 0x7fdeb00ca100 2023-06-10T18:48:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.397+0000 7fdead7fa700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 0x7fde9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.397+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdea80899e0 con 0x7fdeb00ca100 2023-06-10T18:48:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.397+0000 7fdeb4c4d700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 0x7fde9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.398+0000 7fdeb4c4d700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 0x7fde9406d190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00c8ce0 tx=0x7fdea4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.401+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdea80549e0 con 0x7fdeb00ca100 2023-06-10T18:48:48.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.582+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdeb006da80 con 0x7fde9406ace0 2023-06-10T18:48:48.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.592+0000 7fdead7fa700 1 -- 172.21.15.104:0/3056776051 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311729 (secure 0 0 0) 0x7fdea400c040 con 0x7fde9406ace0 2023-06-10T18:48:48.604 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:48.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 msgr2=0x7fde9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 0x7fde9406d190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00c8ce0 tx=0x7fdea4008040 comp rx=0 tx=0).stop 2023-06-10T18:48:48.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 msgr2=0x7fdeb015a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdea800cef0 tx=0x7fdea800cf20 comp rx=0 tx=0).stop 2023-06-10T18:48:48.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 shutdown_connections 2023-06-10T18:48:48.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde9406ace0 0x7fde9406d190 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb006e0e0 0x7fdeb00b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb00ca100 0x7fdeb015a870 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 --2- 172.21.15.104:0/3056776051 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb006e4b0 0x7fdeb0099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 >> 172.21.15.104:0/3056776051 conn(0x7fdeb006b8a0 msgr2=0x7fdeb00ac230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:48.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 shutdown_connections 2023-06-10T18:48:48.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.595+0000 7fdeb6eb1700 1 -- 172.21.15.104:0/3056776051 wait complete. 2023-06-10T18:48:48.611 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:48:48.613 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2023-06-10T18:48:47.425603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223208,"kb_used_data":7592,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754751512,"statfs":{"total":773094113280,"available":772865548288,"internally_reserved":0,"allocated":7774208,"data_stored":4238452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677658"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161363+0000","last_change":"2023-06-10T18:48:44.172499+0000","last_active":"2023-06-10T18:48:45.161363+0000","last_peered":"2023-06-10T18:48:45.161363+0000","last_clean":"2023-06-10T18:48:45.161363+0000","last_became_active":"2023-06-10T18:48:44.172204+0000","last_became_peered":"2023-06-10T18:48:44.172204+0000","last_unstale":"2023-06-10T18:48:45.161363+0000","last_undegraded":"2023-06-10T18:48:45.161363+0000","last_fullsized":"2023-06-10T18:48:45.161363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:11.376824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189186+0000","last_change":"2023-06-10T18:48:44.182176+0000","last_active":"2023-06-10T18:48:44.189186+0000","last_peered":"2023-06-10T18:48:44.189186+0000","last_clean":"2023-06-10T18:48:44.189186+0000","last_became_active":"2023-06-10T18:48:44.181840+0000","last_became_peered":"2023-06-10T18:48:44.181840+0000","last_unstale":"2023-06-10T18:48:44.189186+0000","last_undegraded":"2023-06-10T18:48:44.189186+0000","last_fullsized":"2023-06-10T18:48:44.189186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:13:01.836306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349250+0000","last_change":"2023-06-10T18:48:44.164500+0000","last_active":"2023-06-10T18:48:46.349250+0000","last_peered":"2023-06-10T18:48:46.349250+0000","last_clean":"2023-06-10T18:48:46.349250+0000","last_became_active":"2023-06-10T18:48:44.164061+0000","last_became_peered":"2023-06-10T18:48:44.164061+0000","last_unstale":"2023-06-10T18:48:46.349250+0000","last_undegraded":"2023-06-10T18:48:46.349250+0000","last_fullsized":"2023-06-10T18:48:46.349250+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:04.927420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188775+0000","last_change":"2023-06-10T18:48:44.183737+0000","last_active":"2023-06-10T18:48:44.188775+0000","last_peered":"2023-06-10T18:48:44.188775+0000","last_clean":"2023-06-10T18:48:44.188775+0000","last_became_active":"2023-06-10T18:48:44.183489+0000","last_became_peered":"2023-06-10T18:48:44.183489+0000","last_unstale":"2023-06-10T18:48:44.188775+0000","last_undegraded":"2023-06-10T18:48:44.188775+0000","last_fullsized":"2023-06-10T18:48:44.188775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:04:49.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160893+0000","last_change":"2023-06-10T18:48:44.170182+0000","last_active":"2023-06-10T18:48:45.160893+0000","last_peered":"2023-06-10T18:48:45.160893+0000","last_clean":"2023-06-10T18:48:45.160893+0000","last_became_active":"2023-06-10T18:48:44.169893+0000","last_became_peered":"2023-06-10T18:48:44.169893+0000","last_unstale":"2023-06-10T18:48:45.160893+0000","last_undegraded":"2023-06-10T18:48:45.160893+0000","last_fullsized":"2023-06-10T18:48:45.160893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:25:07.136698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188614+0000","last_change":"2023-06-10T18:48:44.179750+0000","last_active":"2023-06-10T18:48:44.188614+0000","last_peered":"2023-06-10T18:48:44.188614+0000","last_clean":"2023-06-10T18:48:44.188614+0000","last_became_active":"2023-06-10T18:48:44.179464+0000","last_became_peered":"2023-06-10T18:48:44.179464+0000","last_unstale":"2023-06-10T18:48:44.188614+0000","last_undegraded":"2023-06-10T18:48:44.188614+0000","last_fullsized":"2023-06-10T18:48:44.188614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:44.552384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162159+0000","last_change":"2023-06-10T18:48:44.175621+0000","last_active":"2023-06-10T18:48:45.162159+0000","last_peered":"2023-06-10T18:48:45.162159+0000","last_clean":"2023-06-10T18:48:45.162159+0000","last_became_active":"2023-06-10T18:48:44.175335+0000","last_became_peered":"2023-06-10T18:48:44.175335+0000","last_unstale":"2023-06-10T18:48:45.162159+0000","last_undegraded":"2023-06-10T18:48:45.162159+0000","last_fullsized":"2023-06-10T18:48:45.162159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:09:05.696574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188738+0000","last_change":"2023-06-10T18:48:44.180429+0000","last_active":"2023-06-10T18:48:44.188738+0000","last_peered":"2023-06-10T18:48:44.188738+0000","last_clean":"2023-06-10T18:48:44.188738+0000","last_became_active":"2023-06-10T18:48:44.179965+0000","last_became_peered":"2023-06-10T18:48:44.179965+0000","last_unstale":"2023-06-10T18:48:44.188738+0000","last_undegraded":"2023-06-10T18:48:44.188738+0000","last_fullsized":"2023-06-10T18:48:44.188738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:38:16.331016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161307+0000","last_change":"2023-06-10T18:48:44.170136+0000","last_active":"2023-06-10T18:48:45.161307+0000","last_peered":"2023-06-10T18:48:45.161307+0000","last_clean":"2023-06-10T18:48:45.161307+0000","last_became_active":"2023-06-10T18:48:44.168891+0000","last_became_peered":"2023-06-10T18:48:44.168891+0000","last_unstale":"2023-06-10T18:48:45.161307+0000","last_undegraded":"2023-06-10T18:48:45.161307+0000","last_fullsized":"2023-06-10T18:48:45.161307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:31.686125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756627+0000","last_change":"2023-06-10T18:48:44.168880+0000","last_active":"2023-06-10T18:48:45.756627+0000","last_peered":"2023-06-10T18:48:45.756627+0000","last_clean":"2023-06-10T18:48:45.756627+0000","last_became_active":"2023-06-10T18:48:44.168567+0000","last_became_peered":"2023-06-10T18:48:44.168567+0000","last_unstale":"2023-06-10T18:48:45.756627+0000","last_undegraded":"2023-06-10T18:48:45.756627+0000","last_fullsized":"2023-06-10T18:48:45.756627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:15.117741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161067+0000","last_change":"2023-06-10T18:48:44.170830+0000","last_active":"2023-06-10T18:48:45.161067+0000","last_peered":"2023-06-10T18:48:45.161067+0000","last_clean":"2023-06-10T18:48:45.161067+0000","last_became_active":"2023-06-10T18:48:44.170541+0000","last_became_peered":"2023-06-10T18:48:44.170541+0000","last_unstale":"2023-06-10T18:48:45.161067+0000","last_undegraded":"2023-06-10T18:48:45.161067+0000","last_fullsized":"2023-06-10T18:48:45.161067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:55:55.933554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.180420+0000","last_change":"2023-06-10T18:48:44.175356+0000","last_active":"2023-06-10T18:48:44.180420+0000","last_peered":"2023-06-10T18:48:44.180420+0000","last_clean":"2023-06-10T18:48:44.180420+0000","last_became_active":"2023-06-10T18:48:44.174639+0000","last_became_peered":"2023-06-10T18:48:44.174639+0000","last_unstale":"2023-06-10T18:48:44.180420+0000","last_undegraded":"2023-06-10T18:48:44.180420+0000","last_fullsized":"2023-06-10T18:48:44.180420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:49:00.933651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631938+0000","last_change":"2023-06-10T18:48:44.182041+0000","last_active":"2023-06-10T18:48:45.631938+0000","last_peered":"2023-06-10T18:48:45.631938+0000","last_clean":"2023-06-10T18:48:45.631938+0000","last_became_active":"2023-06-10T18:48:44.181832+0000","last_became_peered":"2023-06-10T18:48:44.181832+0000","last_unstale":"2023-06-10T18:48:45.631938+0000","last_undegraded":"2023-06-10T18:48:45.631938+0000","last_fullsized":"2023-06-10T18:48:45.631938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:50.498103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160931+0000","last_change":"2023-06-10T18:48:44.169875+0000","last_active":"2023-06-10T18:48:45.160931+0000","last_peered":"2023-06-10T18:48:45.160931+0000","last_clean":"2023-06-10T18:48:45.160931+0000","last_became_active":"2023-06-10T18:48:44.169272+0000","last_became_peered":"2023-06-10T18:48:44.169272+0000","last_unstale":"2023-06-10T18:48:45.160931+0000","last_undegraded":"2023-06-10T18:48:45.160931+0000","last_fullsized":"2023-06-10T18:48:45.160931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:15:38.787140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631742+0000","last_change":"2023-06-10T18:48:44.166899+0000","last_active":"2023-06-10T18:48:45.631742+0000","last_peered":"2023-06-10T18:48:45.631742+0000","last_clean":"2023-06-10T18:48:45.631742+0000","last_became_active":"2023-06-10T18:48:44.166629+0000","last_became_peered":"2023-06-10T18:48:44.166629+0000","last_unstale":"2023-06-10T18:48:45.631742+0000","last_undegraded":"2023-06-10T18:48:45.631742+0000","last_fullsized":"2023-06-10T18:48:45.631742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:26:51.798101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756108+0000","last_change":"2023-06-10T18:48:44.182782+0000","last_active":"2023-06-10T18:48:45.756108+0000","last_peered":"2023-06-10T18:48:45.756108+0000","last_clean":"2023-06-10T18:48:45.756108+0000","last_became_active":"2023-06-10T18:48:44.182497+0000","last_became_peered":"2023-06-10T18:48:44.182497+0000","last_unstale":"2023-06-10T18:48:45.756108+0000","last_undegraded":"2023-06-10T18:48:45.756108+0000","last_fullsized":"2023-06-10T18:48:45.756108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:38.295549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161315+0000","last_change":"2023-06-10T18:48:44.170563+0000","last_active":"2023-06-10T18:48:45.161315+0000","last_peered":"2023-06-10T18:48:45.161315+0000","last_clean":"2023-06-10T18:48:45.161315+0000","last_became_active":"2023-06-10T18:48:44.169922+0000","last_became_peered":"2023-06-10T18:48:44.169922+0000","last_unstale":"2023-06-10T18:48:45.161315+0000","last_undegraded":"2023-06-10T18:48:45.161315+0000","last_fullsized":"2023-06-10T18:48:45.161315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:20:57.438010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188772+0000","last_change":"2023-06-10T18:48:44.183674+0000","last_active":"2023-06-10T18:48:44.188772+0000","last_peered":"2023-06-10T18:48:44.188772+0000","last_clean":"2023-06-10T18:48:44.188772+0000","last_became_active":"2023-06-10T18:48:44.183481+0000","last_became_peered":"2023-06-10T18:48:44.183481+0000","last_unstale":"2023-06-10T18:48:44.188772+0000","last_undegraded":"2023-06-10T18:48:44.188772+0000","last_fullsized":"2023-06-10T18:48:44.188772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:06:59.290841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183452+0000","last_change":"2023-06-10T18:48:44.177093+0000","last_active":"2023-06-10T18:48:44.183452+0000","last_peered":"2023-06-10T18:48:44.183452+0000","last_clean":"2023-06-10T18:48:44.183452+0000","last_became_active":"2023-06-10T18:48:44.176810+0000","last_became_peered":"2023-06-10T18:48:44.176810+0000","last_unstale":"2023-06-10T18:48:44.183452+0000","last_undegraded":"2023-06-10T18:48:44.183452+0000","last_fullsized":"2023-06-10T18:48:44.183452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:36:32.800925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631858+0000","last_change":"2023-06-10T18:48:44.167750+0000","last_active":"2023-06-10T18:48:45.631858+0000","last_peered":"2023-06-10T18:48:45.631858+0000","last_clean":"2023-06-10T18:48:45.631858+0000","last_became_active":"2023-06-10T18:48:44.167495+0000","last_became_peered":"2023-06-10T18:48:44.167495+0000","last_unstale":"2023-06-10T18:48:45.631858+0000","last_undegraded":"2023-06-10T18:48:45.631858+0000","last_fullsized":"2023-06-10T18:48:45.631858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:12.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756473+0000","last_change":"2023-06-10T18:48:44.163065+0000","last_active":"2023-06-10T18:48:45.756473+0000","last_peered":"2023-06-10T18:48:45.756473+0000","last_clean":"2023-06-10T18:48:45.756473+0000","last_became_active":"2023-06-10T18:48:44.162778+0000","last_became_peered":"2023-06-10T18:48:44.162778+0000","last_unstale":"2023-06-10T18:48:45.756473+0000","last_undegraded":"2023-06-10T18:48:45.756473+0000","last_fullsized":"2023-06-10T18:48:45.756473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:55.566902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756426+0000","last_change":"2023-06-10T18:48:44.169732+0000","last_active":"2023-06-10T18:48:45.756426+0000","last_peered":"2023-06-10T18:48:45.756426+0000","last_clean":"2023-06-10T18:48:45.756426+0000","last_became_active":"2023-06-10T18:48:44.168972+0000","last_became_peered":"2023-06-10T18:48:44.168972+0000","last_unstale":"2023-06-10T18:48:45.756426+0000","last_undegraded":"2023-06-10T18:48:45.756426+0000","last_fullsized":"2023-06-10T18:48:45.756426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:38.665134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160584+0000","last_change":"2023-06-10T18:48:44.170224+0000","last_active":"2023-06-10T18:48:45.160584+0000","last_peered":"2023-06-10T18:48:45.160584+0000","last_clean":"2023-06-10T18:48:45.160584+0000","last_became_active":"2023-06-10T18:48:44.169691+0000","last_became_peered":"2023-06-10T18:48:44.169691+0000","last_unstale":"2023-06-10T18:48:45.160584+0000","last_undegraded":"2023-06-10T18:48:45.160584+0000","last_fullsized":"2023-06-10T18:48:45.160584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:36:47.630806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160171+0000","last_change":"2023-06-10T18:48:44.185120+0000","last_active":"2023-06-10T18:48:45.160171+0000","last_peered":"2023-06-10T18:48:45.160171+0000","last_clean":"2023-06-10T18:48:45.160171+0000","last_became_active":"2023-06-10T18:48:44.184720+0000","last_became_peered":"2023-06-10T18:48:44.184720+0000","last_unstale":"2023-06-10T18:48:45.160171+0000","last_undegraded":"2023-06-10T18:48:45.160171+0000","last_fullsized":"2023-06-10T18:48:45.160171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:30.637597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756772+0000","last_change":"2023-06-10T18:48:44.169546+0000","last_active":"2023-06-10T18:48:45.756772+0000","last_peered":"2023-06-10T18:48:45.756772+0000","last_clean":"2023-06-10T18:48:45.756772+0000","last_became_active":"2023-06-10T18:48:44.169251+0000","last_became_peered":"2023-06-10T18:48:44.169251+0000","last_unstale":"2023-06-10T18:48:45.756772+0000","last_undegraded":"2023-06-10T18:48:45.756772+0000","last_fullsized":"2023-06-10T18:48:45.756772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:04.927420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162554+0000","last_change":"2023-06-10T18:48:44.182352+0000","last_active":"2023-06-10T18:48:45.162554+0000","last_peered":"2023-06-10T18:48:45.162554+0000","last_clean":"2023-06-10T18:48:45.162554+0000","last_became_active":"2023-06-10T18:48:44.182061+0000","last_became_peered":"2023-06-10T18:48:44.182061+0000","last_unstale":"2023-06-10T18:48:45.162554+0000","last_undegraded":"2023-06-10T18:48:45.162554+0000","last_fullsized":"2023-06-10T18:48:45.162554+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:44:49.218305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349209+0000","last_change":"2023-06-10T18:48:44.164321+0000","last_active":"2023-06-10T18:48:46.349209+0000","last_peered":"2023-06-10T18:48:46.349209+0000","last_clean":"2023-06-10T18:48:46.349209+0000","last_became_active":"2023-06-10T18:48:44.163730+0000","last_became_peered":"2023-06-10T18:48:44.163730+0000","last_unstale":"2023-06-10T18:48:46.349209+0000","last_undegraded":"2023-06-10T18:48:46.349209+0000","last_fullsized":"2023-06-10T18:48:46.349209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:15.773141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756677+0000","last_change":"2023-06-10T18:48:44.169076+0000","last_active":"2023-06-10T18:48:45.756677+0000","last_peered":"2023-06-10T18:48:45.756677+0000","last_clean":"2023-06-10T18:48:45.756677+0000","last_became_active":"2023-06-10T18:48:44.168675+0000","last_became_peered":"2023-06-10T18:48:44.168675+0000","last_unstale":"2023-06-10T18:48:45.756677+0000","last_undegraded":"2023-06-10T18:48:45.756677+0000","last_fullsized":"2023-06-10T18:48:45.756677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:15.773141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349058+0000","last_change":"2023-06-10T18:48:44.169421+0000","last_active":"2023-06-10T18:48:46.349058+0000","last_peered":"2023-06-10T18:48:46.349058+0000","last_clean":"2023-06-10T18:48:46.349058+0000","last_became_active":"2023-06-10T18:48:44.169191+0000","last_became_peered":"2023-06-10T18:48:44.169191+0000","last_unstale":"2023-06-10T18:48:46.349058+0000","last_undegraded":"2023-06-10T18:48:46.349058+0000","last_fullsized":"2023-06-10T18:48:46.349058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:07:42.169057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162156+0000","last_change":"2023-06-10T18:48:44.180378+0000","last_active":"2023-06-10T18:48:45.162156+0000","last_peered":"2023-06-10T18:48:45.162156+0000","last_clean":"2023-06-10T18:48:45.162156+0000","last_became_active":"2023-06-10T18:48:44.179895+0000","last_became_peered":"2023-06-10T18:48:44.179895+0000","last_unstale":"2023-06-10T18:48:45.162156+0000","last_undegraded":"2023-06-10T18:48:45.162156+0000","last_fullsized":"2023-06-10T18:48:45.162156+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:18:57.325990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161954+0000","last_change":"2023-06-10T18:48:44.169505+0000","last_active":"2023-06-10T18:48:45.161954+0000","last_peered":"2023-06-10T18:48:45.161954+0000","last_clean":"2023-06-10T18:48:45.161954+0000","last_became_active":"2023-06-10T18:48:44.169209+0000","last_became_peered":"2023-06-10T18:48:44.169209+0000","last_unstale":"2023-06-10T18:48:45.161954+0000","last_undegraded":"2023-06-10T18:48:45.161954+0000","last_fullsized":"2023-06-10T18:48:45.161954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:20.666277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188718+0000","last_change":"2023-06-10T18:48:44.180193+0000","last_active":"2023-06-10T18:48:44.188718+0000","last_peered":"2023-06-10T18:48:44.188718+0000","last_clean":"2023-06-10T18:48:44.188718+0000","last_became_active":"2023-06-10T18:48:44.179663+0000","last_became_peered":"2023-06-10T18:48:44.179663+0000","last_unstale":"2023-06-10T18:48:44.188718+0000","last_undegraded":"2023-06-10T18:48:44.188718+0000","last_fullsized":"2023-06-10T18:48:44.188718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:03:27.847439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161335+0000","last_change":"2023-06-10T18:48:44.184312+0000","last_active":"2023-06-10T18:48:45.161335+0000","last_peered":"2023-06-10T18:48:45.161335+0000","last_clean":"2023-06-10T18:48:45.161335+0000","last_became_active":"2023-06-10T18:48:44.183450+0000","last_became_peered":"2023-06-10T18:48:44.183450+0000","last_unstale":"2023-06-10T18:48:45.161335+0000","last_undegraded":"2023-06-10T18:48:45.161335+0000","last_fullsized":"2023-06-10T18:48:45.161335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:42.590220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161767+0000","last_change":"2023-06-10T18:48:43.761342+0000","last_active":"2023-06-10T18:48:45.161767+0000","last_peered":"2023-06-10T18:48:45.161767+0000","last_clean":"2023-06-10T18:48:45.161767+0000","last_became_active":"2023-06-10T18:48:43.760962+0000","last_became_peered":"2023-06-10T18:48:43.760962+0000","last_unstale":"2023-06-10T18:48:45.161767+0000","last_undegraded":"2023-06-10T18:48:45.161767+0000","last_fullsized":"2023-06-10T18:48:45.161767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:02:44.052313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161413+0000","last_change":"2023-06-10T18:48:44.167129+0000","last_active":"2023-06-10T18:48:45.161413+0000","last_peered":"2023-06-10T18:48:45.161413+0000","last_clean":"2023-06-10T18:48:45.161413+0000","last_became_active":"2023-06-10T18:48:44.166777+0000","last_became_peered":"2023-06-10T18:48:44.166777+0000","last_unstale":"2023-06-10T18:48:45.161413+0000","last_undegraded":"2023-06-10T18:48:45.161413+0000","last_fullsized":"2023-06-10T18:48:45.161413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:43.661938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632056+0000","last_change":"2023-06-10T18:48:43.768043+0000","last_active":"2023-06-10T18:48:45.632056+0000","last_peered":"2023-06-10T18:48:45.632056+0000","last_clean":"2023-06-10T18:48:45.632056+0000","last_became_active":"2023-06-10T18:48:43.767749+0000","last_became_peered":"2023-06-10T18:48:43.767749+0000","last_unstale":"2023-06-10T18:48:45.632056+0000","last_undegraded":"2023-06-10T18:48:45.632056+0000","last_fullsized":"2023-06-10T18:48:45.632056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:14:14.288793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189352+0000","last_change":"2023-06-10T18:48:44.184295+0000","last_active":"2023-06-10T18:48:44.189352+0000","last_peered":"2023-06-10T18:48:44.189352+0000","last_clean":"2023-06-10T18:48:44.189352+0000","last_became_active":"2023-06-10T18:48:44.183805+0000","last_became_peered":"2023-06-10T18:48:44.183805+0000","last_unstale":"2023-06-10T18:48:44.189352+0000","last_undegraded":"2023-06-10T18:48:44.189352+0000","last_fullsized":"2023-06-10T18:48:44.189352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:57:37.332605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756720+0000","last_change":"2023-06-10T18:48:43.772302+0000","last_active":"2023-06-10T18:48:45.756720+0000","last_peered":"2023-06-10T18:48:45.756720+0000","last_clean":"2023-06-10T18:48:45.756720+0000","last_became_active":"2023-06-10T18:48:43.771946+0000","last_became_peered":"2023-06-10T18:48:43.771946+0000","last_unstale":"2023-06-10T18:48:45.756720+0000","last_undegraded":"2023-06-10T18:48:45.756720+0000","last_fullsized":"2023-06-10T18:48:45.756720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:56.612107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161140+0000","last_change":"2023-06-10T18:48:44.170473+0000","last_active":"2023-06-10T18:48:45.161140+0000","last_peered":"2023-06-10T18:48:45.161140+0000","last_clean":"2023-06-10T18:48:45.161140+0000","last_became_active":"2023-06-10T18:48:44.170008+0000","last_became_peered":"2023-06-10T18:48:44.170008+0000","last_unstale":"2023-06-10T18:48:45.161140+0000","last_undegraded":"2023-06-10T18:48:45.161140+0000","last_fullsized":"2023-06-10T18:48:45.161140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:35.785565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.184024+0000","last_change":"2023-06-10T18:48:43.770656+0000","last_active":"2023-06-10T18:48:44.184024+0000","last_peered":"2023-06-10T18:48:44.184024+0000","last_clean":"2023-06-10T18:48:44.184024+0000","last_became_active":"2023-06-10T18:48:43.770267+0000","last_became_peered":"2023-06-10T18:48:43.770267+0000","last_unstale":"2023-06-10T18:48:44.184024+0000","last_undegraded":"2023-06-10T18:48:44.184024+0000","last_fullsized":"2023-06-10T18:48:44.184024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:16:00.969421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162191+0000","last_change":"2023-06-10T18:48:44.167166+0000","last_active":"2023-06-10T18:48:45.162191+0000","last_peered":"2023-06-10T18:48:45.162191+0000","last_clean":"2023-06-10T18:48:45.162191+0000","last_became_active":"2023-06-10T18:48:44.166894+0000","last_became_peered":"2023-06-10T18:48:44.166894+0000","last_unstale":"2023-06-10T18:48:45.162191+0000","last_undegraded":"2023-06-10T18:48:45.162191+0000","last_fullsized":"2023-06-10T18:48:45.162191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:26:17.522851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349222+0000","last_change":"2023-06-10T18:48:43.764019+0000","last_active":"2023-06-10T18:48:46.349222+0000","last_peered":"2023-06-10T18:48:46.349222+0000","last_clean":"2023-06-10T18:48:46.349222+0000","last_became_active":"2023-06-10T18:48:43.763743+0000","last_became_peered":"2023-06-10T18:48:43.763743+0000","last_unstale":"2023-06-10T18:48:46.349222+0000","last_undegraded":"2023-06-10T18:48:46.349222+0000","last_fullsized":"2023-06-10T18:48:46.349222+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:56.612107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756461+0000","last_change":"2023-06-10T18:48:44.169566+0000","last_active":"2023-06-10T18:48:45.756461+0000","last_peered":"2023-06-10T18:48:45.756461+0000","last_clean":"2023-06-10T18:48:45.756461+0000","last_became_active":"2023-06-10T18:48:44.169261+0000","last_became_peered":"2023-06-10T18:48:44.169261+0000","last_unstale":"2023-06-10T18:48:45.756461+0000","last_undegraded":"2023-06-10T18:48:45.756461+0000","last_fullsized":"2023-06-10T18:48:45.756461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:07:42.169057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183175+0000","last_change":"2023-06-10T18:48:43.771888+0000","last_active":"2023-06-10T18:48:44.183175+0000","last_peered":"2023-06-10T18:48:44.183175+0000","last_clean":"2023-06-10T18:48:44.183175+0000","last_became_active":"2023-06-10T18:48:43.771572+0000","last_became_peered":"2023-06-10T18:48:43.771572+0000","last_unstale":"2023-06-10T18:48:44.183175+0000","last_undegraded":"2023-06-10T18:48:44.183175+0000","last_fullsized":"2023-06-10T18:48:44.183175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:06.082899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160742+0000","last_change":"2023-06-10T18:48:44.172292+0000","last_active":"2023-06-10T18:48:45.160742+0000","last_peered":"2023-06-10T18:48:45.160742+0000","last_clean":"2023-06-10T18:48:45.160742+0000","last_became_active":"2023-06-10T18:48:44.172014+0000","last_became_peered":"2023-06-10T18:48:44.172014+0000","last_unstale":"2023-06-10T18:48:45.160742+0000","last_undegraded":"2023-06-10T18:48:45.160742+0000","last_fullsized":"2023-06-10T18:48:45.160742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:21:23.652595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163229+0000","last_change":"2023-06-10T18:48:43.765302+0000","last_active":"2023-06-10T18:48:44.163229+0000","last_peered":"2023-06-10T18:48:44.163229+0000","last_clean":"2023-06-10T18:48:44.163229+0000","last_became_active":"2023-06-10T18:48:43.765050+0000","last_became_peered":"2023-06-10T18:48:43.765050+0000","last_unstale":"2023-06-10T18:48:44.163229+0000","last_undegraded":"2023-06-10T18:48:44.163229+0000","last_fullsized":"2023-06-10T18:48:44.163229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:55.628445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631780+0000","last_change":"2023-06-10T18:48:44.167451+0000","last_active":"2023-06-10T18:48:45.631780+0000","last_peered":"2023-06-10T18:48:45.631780+0000","last_clean":"2023-06-10T18:48:45.631780+0000","last_became_active":"2023-06-10T18:48:44.167157+0000","last_became_peered":"2023-06-10T18:48:44.167157+0000","last_unstale":"2023-06-10T18:48:45.631780+0000","last_undegraded":"2023-06-10T18:48:45.631780+0000","last_fullsized":"2023-06-10T18:48:45.631780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:18:22.227910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160783+0000","last_change":"2023-06-10T18:48:43.763361+0000","last_active":"2023-06-10T18:48:45.160783+0000","last_peered":"2023-06-10T18:48:45.160783+0000","last_clean":"2023-06-10T18:48:45.160783+0000","last_became_active":"2023-06-10T18:48:43.763093+0000","last_became_peered":"2023-06-10T18:48:43.763093+0000","last_unstale":"2023-06-10T18:48:45.160783+0000","last_undegraded":"2023-06-10T18:48:45.160783+0000","last_fullsized":"2023-06-10T18:48:45.160783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:23.985051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162533+0000","last_change":"2023-06-10T18:48:44.167148+0000","last_active":"2023-06-10T18:48:45.162533+0000","last_peered":"2023-06-10T18:48:45.162533+0000","last_clean":"2023-06-10T18:48:45.162533+0000","last_became_active":"2023-06-10T18:48:44.166861+0000","last_became_peered":"2023-06-10T18:48:44.166861+0000","last_unstale":"2023-06-10T18:48:45.162533+0000","last_undegraded":"2023-06-10T18:48:45.162533+0000","last_fullsized":"2023-06-10T18:48:45.162533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:14.075412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756841+0000","last_change":"2023-06-10T18:48:43.765639+0000","last_active":"2023-06-10T18:48:45.756841+0000","last_peered":"2023-06-10T18:48:45.756841+0000","last_clean":"2023-06-10T18:48:45.756841+0000","last_became_active":"2023-06-10T18:48:43.765356+0000","last_became_peered":"2023-06-10T18:48:43.765356+0000","last_unstale":"2023-06-10T18:48:45.756841+0000","last_undegraded":"2023-06-10T18:48:45.756841+0000","last_fullsized":"2023-06-10T18:48:45.756841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:01:26.026568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161634+0000","last_change":"2023-06-10T18:48:44.183821+0000","last_active":"2023-06-10T18:48:45.161634+0000","last_peered":"2023-06-10T18:48:45.161634+0000","last_clean":"2023-06-10T18:48:45.161634+0000","last_became_active":"2023-06-10T18:48:44.183288+0000","last_became_peered":"2023-06-10T18:48:44.183288+0000","last_unstale":"2023-06-10T18:48:45.161634+0000","last_undegraded":"2023-06-10T18:48:45.161634+0000","last_fullsized":"2023-06-10T18:48:45.161634+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:16.928954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.178984+0000","last_change":"2023-06-10T18:48:43.771799+0000","last_active":"2023-06-10T18:48:44.178984+0000","last_peered":"2023-06-10T18:48:44.178984+0000","last_clean":"2023-06-10T18:48:44.178984+0000","last_became_active":"2023-06-10T18:48:43.771118+0000","last_became_peered":"2023-06-10T18:48:43.771118+0000","last_unstale":"2023-06-10T18:48:44.178984+0000","last_undegraded":"2023-06-10T18:48:44.178984+0000","last_fullsized":"2023-06-10T18:48:44.178984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:06.097636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.189172+0000","last_change":"2023-06-10T18:48:44.181778+0000","last_active":"2023-06-10T18:48:44.189172+0000","last_peered":"2023-06-10T18:48:44.189172+0000","last_clean":"2023-06-10T18:48:44.189172+0000","last_became_active":"2023-06-10T18:48:44.181302+0000","last_became_peered":"2023-06-10T18:48:44.181302+0000","last_unstale":"2023-06-10T18:48:44.189172+0000","last_undegraded":"2023-06-10T18:48:44.189172+0000","last_fullsized":"2023-06-10T18:48:44.189172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:48:06.745444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349347+0000","last_change":"2023-06-10T18:48:43.760029+0000","last_active":"2023-06-10T18:48:46.349347+0000","last_peered":"2023-06-10T18:48:46.349347+0000","last_clean":"2023-06-10T18:48:46.349347+0000","last_became_active":"2023-06-10T18:48:43.759791+0000","last_became_peered":"2023-06-10T18:48:43.759791+0000","last_unstale":"2023-06-10T18:48:46.349347+0000","last_undegraded":"2023-06-10T18:48:46.349347+0000","last_fullsized":"2023-06-10T18:48:46.349347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:56:58.400363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161215+0000","last_change":"2023-06-10T18:48:44.166130+0000","last_active":"2023-06-10T18:48:45.161215+0000","last_peered":"2023-06-10T18:48:45.161215+0000","last_clean":"2023-06-10T18:48:45.161215+0000","last_became_active":"2023-06-10T18:48:44.165948+0000","last_became_peered":"2023-06-10T18:48:44.165948+0000","last_unstale":"2023-06-10T18:48:45.161215+0000","last_undegraded":"2023-06-10T18:48:45.161215+0000","last_fullsized":"2023-06-10T18:48:45.161215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:42:53.225771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188819+0000","last_change":"2023-06-10T18:48:43.763986+0000","last_active":"2023-06-10T18:48:44.188819+0000","last_peered":"2023-06-10T18:48:44.188819+0000","last_clean":"2023-06-10T18:48:44.188819+0000","last_became_active":"2023-06-10T18:48:43.763556+0000","last_became_peered":"2023-06-10T18:48:43.763556+0000","last_unstale":"2023-06-10T18:48:44.188819+0000","last_undegraded":"2023-06-10T18:48:44.188819+0000","last_fullsized":"2023-06-10T18:48:44.188819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:18:50.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160620+0000","last_change":"2023-06-10T18:48:44.172599+0000","last_active":"2023-06-10T18:48:45.160620+0000","last_peered":"2023-06-10T18:48:45.160620+0000","last_clean":"2023-06-10T18:48:45.160620+0000","last_became_active":"2023-06-10T18:48:44.172287+0000","last_became_peered":"2023-06-10T18:48:44.172287+0000","last_unstale":"2023-06-10T18:48:45.160620+0000","last_undegraded":"2023-06-10T18:48:45.160620+0000","last_fullsized":"2023-06-10T18:48:45.160620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:52.593526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.175694+0000","last_change":"2023-06-10T18:48:43.760038+0000","last_active":"2023-06-10T18:48:44.175694+0000","last_peered":"2023-06-10T18:48:44.175694+0000","last_clean":"2023-06-10T18:48:44.175694+0000","last_became_active":"2023-06-10T18:48:43.759766+0000","last_became_peered":"2023-06-10T18:48:43.759766+0000","last_unstale":"2023-06-10T18:48:44.175694+0000","last_undegraded":"2023-06-10T18:48:44.175694+0000","last_fullsized":"2023-06-10T18:48:44.175694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:46:02.650662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162418+0000","last_change":"2023-06-10T18:48:44.167863+0000","last_active":"2023-06-10T18:48:45.162418+0000","last_peered":"2023-06-10T18:48:45.162418+0000","last_clean":"2023-06-10T18:48:45.162418+0000","last_became_active":"2023-06-10T18:48:44.167599+0000","last_became_peered":"2023-06-10T18:48:44.167599+0000","last_unstale":"2023-06-10T18:48:45.162418+0000","last_undegraded":"2023-06-10T18:48:45.162418+0000","last_fullsized":"2023-06-10T18:48:45.162418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:32:05.923639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.172931+0000","last_change":"2023-06-10T18:48:43.771772+0000","last_active":"2023-06-10T18:48:44.172931+0000","last_peered":"2023-06-10T18:48:44.172931+0000","last_clean":"2023-06-10T18:48:44.172931+0000","last_became_active":"2023-06-10T18:48:43.771470+0000","last_became_peered":"2023-06-10T18:48:43.771470+0000","last_unstale":"2023-06-10T18:48:44.172931+0000","last_undegraded":"2023-06-10T18:48:44.172931+0000","last_fullsized":"2023-06-10T18:48:44.172931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:24.269341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631728+0000","last_change":"2023-06-10T18:48:44.179966+0000","last_active":"2023-06-10T18:48:45.631728+0000","last_peered":"2023-06-10T18:48:45.631728+0000","last_clean":"2023-06-10T18:48:45.631728+0000","last_became_active":"2023-06-10T18:48:44.179665+0000","last_became_peered":"2023-06-10T18:48:44.179665+0000","last_unstale":"2023-06-10T18:48:45.631728+0000","last_undegraded":"2023-06-10T18:48:45.631728+0000","last_fullsized":"2023-06-10T18:48:45.631728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:40:12.464774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162361+0000","last_change":"2023-06-10T18:48:43.768991+0000","last_active":"2023-06-10T18:48:45.162361+0000","last_peered":"2023-06-10T18:48:45.162361+0000","last_clean":"2023-06-10T18:48:45.162361+0000","last_became_active":"2023-06-10T18:48:43.766837+0000","last_became_peered":"2023-06-10T18:48:43.766837+0000","last_unstale":"2023-06-10T18:48:45.162361+0000","last_undegraded":"2023-06-10T18:48:45.162361+0000","last_fullsized":"2023-06-10T18:48:45.162361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:28.349761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160854+0000","last_change":"2023-06-10T18:48:44.168794+0000","last_active":"2023-06-10T18:48:45.160854+0000","last_peered":"2023-06-10T18:48:45.160854+0000","last_clean":"2023-06-10T18:48:45.160854+0000","last_became_active":"2023-06-10T18:48:44.168334+0000","last_became_peered":"2023-06-10T18:48:44.168334+0000","last_unstale":"2023-06-10T18:48:45.160854+0000","last_undegraded":"2023-06-10T18:48:45.160854+0000","last_fullsized":"2023-06-10T18:48:45.160854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:59:54.615219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756164+0000","last_change":"2023-06-10T18:48:43.768871+0000","last_active":"2023-06-10T18:48:45.756164+0000","last_peered":"2023-06-10T18:48:45.756164+0000","last_clean":"2023-06-10T18:48:45.756164+0000","last_became_active":"2023-06-10T18:48:43.768597+0000","last_became_peered":"2023-06-10T18:48:43.768597+0000","last_unstale":"2023-06-10T18:48:45.756164+0000","last_undegraded":"2023-06-10T18:48:45.756164+0000","last_fullsized":"2023-06-10T18:48:45.756164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:59.649340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632005+0000","last_change":"2023-06-10T18:48:44.166852+0000","last_active":"2023-06-10T18:48:45.632005+0000","last_peered":"2023-06-10T18:48:45.632005+0000","last_clean":"2023-06-10T18:48:45.632005+0000","last_became_active":"2023-06-10T18:48:44.166581+0000","last_became_peered":"2023-06-10T18:48:44.166581+0000","last_unstale":"2023-06-10T18:48:45.632005+0000","last_undegraded":"2023-06-10T18:48:45.632005+0000","last_fullsized":"2023-06-10T18:48:45.632005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:52.840640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.188686+0000","last_change":"2023-06-10T18:48:43.771160+0000","last_active":"2023-06-10T18:48:44.188686+0000","last_peered":"2023-06-10T18:48:44.188686+0000","last_clean":"2023-06-10T18:48:44.188686+0000","last_became_active":"2023-06-10T18:48:43.770878+0000","last_became_peered":"2023-06-10T18:48:43.770878+0000","last_unstale":"2023-06-10T18:48:44.188686+0000","last_undegraded":"2023-06-10T18:48:44.188686+0000","last_fullsized":"2023-06-10T18:48:44.188686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:34:35.708922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162569+0000","last_change":"2023-06-10T18:48:44.174979+0000","last_active":"2023-06-10T18:48:45.162569+0000","last_peered":"2023-06-10T18:48:45.162569+0000","last_clean":"2023-06-10T18:48:45.162569+0000","last_became_active":"2023-06-10T18:48:44.174607+0000","last_became_peered":"2023-06-10T18:48:44.174607+0000","last_unstale":"2023-06-10T18:48:45.162569+0000","last_undegraded":"2023-06-10T18:48:45.162569+0000","last_fullsized":"2023-06-10T18:48:45.162569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:18.079380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162641+0000","last_change":"2023-06-10T18:48:43.767032+0000","last_active":"2023-06-10T18:48:45.162641+0000","last_peered":"2023-06-10T18:48:45.162641+0000","last_clean":"2023-06-10T18:48:45.162641+0000","last_became_active":"2023-06-10T18:48:43.766740+0000","last_became_peered":"2023-06-10T18:48:43.766740+0000","last_unstale":"2023-06-10T18:48:45.162641+0000","last_undegraded":"2023-06-10T18:48:45.162641+0000","last_fullsized":"2023-06-10T18:48:45.162641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:16:16.416115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160597+0000","last_change":"2023-06-10T18:48:44.183869+0000","last_active":"2023-06-10T18:48:45.160597+0000","last_peered":"2023-06-10T18:48:45.160597+0000","last_clean":"2023-06-10T18:48:45.160597+0000","last_became_active":"2023-06-10T18:48:44.183617+0000","last_became_peered":"2023-06-10T18:48:44.183617+0000","last_unstale":"2023-06-10T18:48:45.160597+0000","last_undegraded":"2023-06-10T18:48:45.160597+0000","last_fullsized":"2023-06-10T18:48:45.160597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:50:50.167706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161269+0000","last_change":"2023-06-10T18:48:43.766273+0000","last_active":"2023-06-10T18:48:45.161269+0000","last_peered":"2023-06-10T18:48:45.161269+0000","last_clean":"2023-06-10T18:48:45.161269+0000","last_became_active":"2023-06-10T18:48:43.765568+0000","last_became_peered":"2023-06-10T18:48:43.765568+0000","last_unstale":"2023-06-10T18:48:45.161269+0000","last_undegraded":"2023-06-10T18:48:45.161269+0000","last_fullsized":"2023-06-10T18:48:45.161269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:47.987218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161201+0000","last_change":"2023-06-10T18:48:44.170289+0000","last_active":"2023-06-10T18:48:45.161201+0000","last_peered":"2023-06-10T18:48:45.161201+0000","last_clean":"2023-06-10T18:48:45.161201+0000","last_became_active":"2023-06-10T18:48:44.169663+0000","last_became_peered":"2023-06-10T18:48:44.169663+0000","last_unstale":"2023-06-10T18:48:45.161201+0000","last_undegraded":"2023-06-10T18:48:45.161201+0000","last_fullsized":"2023-06-10T18:48:45.161201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:41.042057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.632076+0000","last_change":"2023-06-10T18:48:43.769455+0000","last_active":"2023-06-10T18:48:45.632076+0000","last_peered":"2023-06-10T18:48:45.632076+0000","last_clean":"2023-06-10T18:48:45.632076+0000","last_became_active":"2023-06-10T18:48:43.769166+0000","last_became_peered":"2023-06-10T18:48:43.769166+0000","last_unstale":"2023-06-10T18:48:45.632076+0000","last_undegraded":"2023-06-10T18:48:45.632076+0000","last_fullsized":"2023-06-10T18:48:45.632076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:35:36.279401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160966+0000","last_change":"2023-06-10T18:48:44.163927+0000","last_active":"2023-06-10T18:48:45.160966+0000","last_peered":"2023-06-10T18:48:45.160966+0000","last_clean":"2023-06-10T18:48:45.160966+0000","last_became_active":"2023-06-10T18:48:44.163655+0000","last_became_peered":"2023-06-10T18:48:44.163655+0000","last_unstale":"2023-06-10T18:48:45.160966+0000","last_undegraded":"2023-06-10T18:48:45.160966+0000","last_fullsized":"2023-06-10T18:48:45.160966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:34.155054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161073+0000","last_change":"2023-06-10T18:48:43.766111+0000","last_active":"2023-06-10T18:48:45.161073+0000","last_peered":"2023-06-10T18:48:45.161073+0000","last_clean":"2023-06-10T18:48:45.161073+0000","last_became_active":"2023-06-10T18:48:43.765829+0000","last_became_peered":"2023-06-10T18:48:43.765829+0000","last_unstale":"2023-06-10T18:48:45.161073+0000","last_undegraded":"2023-06-10T18:48:45.161073+0000","last_fullsized":"2023-06-10T18:48:45.161073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:05:00.015824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183187+0000","last_change":"2023-06-10T18:48:44.177244+0000","last_active":"2023-06-10T18:48:44.183187+0000","last_peered":"2023-06-10T18:48:44.183187+0000","last_clean":"2023-06-10T18:48:44.183187+0000","last_became_active":"2023-06-10T18:48:44.176941+0000","last_became_peered":"2023-06-10T18:48:44.176941+0000","last_unstale":"2023-06-10T18:48:44.183187+0000","last_undegraded":"2023-06-10T18:48:44.183187+0000","last_fullsized":"2023-06-10T18:48:44.183187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:52.934944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163575+0000","last_change":"2023-06-10T18:48:43.762463+0000","last_active":"2023-06-10T18:48:44.163575+0000","last_peered":"2023-06-10T18:48:44.163575+0000","last_clean":"2023-06-10T18:48:44.163575+0000","last_became_active":"2023-06-10T18:48:43.762174+0000","last_became_peered":"2023-06-10T18:48:43.762174+0000","last_unstale":"2023-06-10T18:48:44.163575+0000","last_undegraded":"2023-06-10T18:48:44.163575+0000","last_fullsized":"2023-06-10T18:48:44.163575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:12:35.388801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.177484+0000","last_change":"2023-06-10T18:48:44.172046+0000","last_active":"2023-06-10T18:48:44.177484+0000","last_peered":"2023-06-10T18:48:44.177484+0000","last_clean":"2023-06-10T18:48:44.177484+0000","last_became_active":"2023-06-10T18:48:44.171791+0000","last_became_peered":"2023-06-10T18:48:44.171791+0000","last_unstale":"2023-06-10T18:48:44.177484+0000","last_undegraded":"2023-06-10T18:48:44.177484+0000","last_fullsized":"2023-06-10T18:48:44.177484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:58:30.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756381+0000","last_change":"2023-06-10T18:48:43.768949+0000","last_active":"2023-06-10T18:48:45.756381+0000","last_peered":"2023-06-10T18:48:45.756381+0000","last_clean":"2023-06-10T18:48:45.756381+0000","last_became_active":"2023-06-10T18:48:43.768666+0000","last_became_peered":"2023-06-10T18:48:43.768666+0000","last_unstale":"2023-06-10T18:48:45.756381+0000","last_undegraded":"2023-06-10T18:48:45.756381+0000","last_fullsized":"2023-06-10T18:48:45.756381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:38:33.251347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.181738+0000","last_change":"2023-06-10T18:48:44.172457+0000","last_active":"2023-06-10T18:48:44.181738+0000","last_peered":"2023-06-10T18:48:44.181738+0000","last_clean":"2023-06-10T18:48:44.181738+0000","last_became_active":"2023-06-10T18:48:44.171706+0000","last_became_peered":"2023-06-10T18:48:44.171706+0000","last_unstale":"2023-06-10T18:48:44.181738+0000","last_undegraded":"2023-06-10T18:48:44.181738+0000","last_fullsized":"2023-06-10T18:48:44.181738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:34:35.887246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173169+0000","last_change":"2023-06-10T18:48:43.771036+0000","last_active":"2023-06-10T18:48:44.173169+0000","last_peered":"2023-06-10T18:48:44.173169+0000","last_clean":"2023-06-10T18:48:44.173169+0000","last_became_active":"2023-06-10T18:48:43.770451+0000","last_became_peered":"2023-06-10T18:48:43.770451+0000","last_unstale":"2023-06-10T18:48:44.173169+0000","last_undegraded":"2023-06-10T18:48:44.173169+0000","last_fullsized":"2023-06-10T18:48:44.173169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:37:14.197352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161158+0000","last_change":"2023-06-10T18:48:44.170127+0000","last_active":"2023-06-10T18:48:45.161158+0000","last_peered":"2023-06-10T18:48:45.161158+0000","last_clean":"2023-06-10T18:48:45.161158+0000","last_became_active":"2023-06-10T18:48:44.168766+0000","last_became_peered":"2023-06-10T18:48:44.168766+0000","last_unstale":"2023-06-10T18:48:45.161158+0000","last_undegraded":"2023-06-10T18:48:45.161158+0000","last_fullsized":"2023-06-10T18:48:45.161158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:44:25.275961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160990+0000","last_change":"2023-06-10T18:48:43.767740+0000","last_active":"2023-06-10T18:48:45.160990+0000","last_peered":"2023-06-10T18:48:45.160990+0000","last_clean":"2023-06-10T18:48:45.160990+0000","last_became_active":"2023-06-10T18:48:43.767216+0000","last_became_peered":"2023-06-10T18:48:43.767216+0000","last_unstale":"2023-06-10T18:48:45.160990+0000","last_undegraded":"2023-06-10T18:48:45.160990+0000","last_fullsized":"2023-06-10T18:48:45.160990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:25.367008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":95,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.163903+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:48:44.163903+0000","last_peered":"2023-06-10T18:48:44.163903+0000","last_clean":"2023-06-10T18:48:44.163903+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:48:44.163903+0000","last_undegraded":"2023-06-10T18:48:44.163903+0000","last_fullsized":"2023-06-10T18:48:44.163903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:36:56.258036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349173+0000","last_change":"2023-06-10T18:48:44.173610+0000","last_active":"2023-06-10T18:48:46.349173+0000","last_peered":"2023-06-10T18:48:46.349173+0000","last_clean":"2023-06-10T18:48:46.349173+0000","last_became_active":"2023-06-10T18:48:44.173331+0000","last_became_peered":"2023-06-10T18:48:44.173331+0000","last_unstale":"2023-06-10T18:48:46.349173+0000","last_undegraded":"2023-06-10T18:48:46.349173+0000","last_fullsized":"2023-06-10T18:48:46.349173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:34:15.117741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173018+0000","last_change":"2023-06-10T18:48:43.770945+0000","last_active":"2023-06-10T18:48:44.173018+0000","last_peered":"2023-06-10T18:48:44.173018+0000","last_clean":"2023-06-10T18:48:44.173018+0000","last_became_active":"2023-06-10T18:48:43.770401+0000","last_became_peered":"2023-06-10T18:48:43.770401+0000","last_unstale":"2023-06-10T18:48:44.173018+0000","last_undegraded":"2023-06-10T18:48:44.173018+0000","last_fullsized":"2023-06-10T18:48:44.173018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:29:06.004191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162247+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:48:45.162247+0000","last_peered":"2023-06-10T18:48:45.162247+0000","last_clean":"2023-06-10T18:48:45.162247+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:48:45.162247+0000","last_undegraded":"2023-06-10T18:48:45.162247+0000","last_fullsized":"2023-06-10T18:48:45.162247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:26:17.186332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349033+0000","last_change":"2023-06-10T18:48:44.181640+0000","last_active":"2023-06-10T18:48:46.349033+0000","last_peered":"2023-06-10T18:48:46.349033+0000","last_clean":"2023-06-10T18:48:46.349033+0000","last_became_active":"2023-06-10T18:48:44.181411+0000","last_became_peered":"2023-06-10T18:48:44.181411+0000","last_unstale":"2023-06-10T18:48:46.349033+0000","last_undegraded":"2023-06-10T18:48:46.349033+0000","last_fullsized":"2023-06-10T18:48:46.349033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:38.665134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162537+0000","last_change":"2023-06-10T18:48:43.764233+0000","last_active":"2023-06-10T18:48:45.162537+0000","last_peered":"2023-06-10T18:48:45.162537+0000","last_clean":"2023-06-10T18:48:45.162537+0000","last_became_active":"2023-06-10T18:48:43.763961+0000","last_became_peered":"2023-06-10T18:48:43.763961+0000","last_unstale":"2023-06-10T18:48:45.162537+0000","last_undegraded":"2023-06-10T18:48:45.162537+0000","last_fullsized":"2023-06-10T18:48:45.162537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:34:49.536536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160873+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:48:45.160873+0000","last_peered":"2023-06-10T18:48:45.160873+0000","last_clean":"2023-06-10T18:48:45.160873+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:48:45.160873+0000","last_undegraded":"2023-06-10T18:48:45.160873+0000","last_fullsized":"2023-06-10T18:48:45.160873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:11.185165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349263+0000","last_change":"2023-06-10T18:48:44.181179+0000","last_active":"2023-06-10T18:48:46.349263+0000","last_peered":"2023-06-10T18:48:46.349263+0000","last_clean":"2023-06-10T18:48:46.349263+0000","last_became_active":"2023-06-10T18:48:44.180946+0000","last_became_peered":"2023-06-10T18:48:44.180946+0000","last_unstale":"2023-06-10T18:48:46.349263+0000","last_undegraded":"2023-06-10T18:48:46.349263+0000","last_fullsized":"2023-06-10T18:48:46.349263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:24.624705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.181572+0000","last_change":"2023-06-10T18:48:43.770788+0000","last_active":"2023-06-10T18:48:44.181572+0000","last_peered":"2023-06-10T18:48:44.181572+0000","last_clean":"2023-06-10T18:48:44.181572+0000","last_became_active":"2023-06-10T18:48:43.770183+0000","last_became_peered":"2023-06-10T18:48:43.770183+0000","last_unstale":"2023-06-10T18:48:44.181572+0000","last_undegraded":"2023-06-10T18:48:44.181572+0000","last_fullsized":"2023-06-10T18:48:44.181572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:43:47.272036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":109,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.162208+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:48:45.162208+0000","last_peered":"2023-06-10T18:48:45.162208+0000","last_clean":"2023-06-10T18:48:45.162208+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:48:45.162208+0000","last_undegraded":"2023-06-10T18:48:45.162208+0000","last_fullsized":"2023-06-10T18:48:45.162208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:03:10.822794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":257,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160167+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:48:45.160167+0000","last_peered":"2023-06-10T18:48:45.160167+0000","last_clean":"2023-06-10T18:48:45.160167+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:48:45.160167+0000","last_undegraded":"2023-06-10T18:48:45.160167+0000","last_fullsized":"2023-06-10T18:48:45.160167+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:58:23.329731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:46.349120+0000","last_change":"2023-06-10T18:48:44.179828+0000","last_active":"2023-06-10T18:48:46.349120+0000","last_peered":"2023-06-10T18:48:46.349120+0000","last_clean":"2023-06-10T18:48:46.349120+0000","last_became_active":"2023-06-10T18:48:44.179591+0000","last_became_peered":"2023-06-10T18:48:44.179591+0000","last_unstale":"2023-06-10T18:48:46.349120+0000","last_undegraded":"2023-06-10T18:48:46.349120+0000","last_fullsized":"2023-06-10T18:48:46.349120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:55.566902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.631814+0000","last_change":"2023-06-10T18:48:43.765451+0000","last_active":"2023-06-10T18:48:45.631814+0000","last_peered":"2023-06-10T18:48:45.631814+0000","last_clean":"2023-06-10T18:48:45.631814+0000","last_became_active":"2023-06-10T18:48:43.763981+0000","last_became_peered":"2023-06-10T18:48:43.763981+0000","last_unstale":"2023-06-10T18:48:45.631814+0000","last_undegraded":"2023-06-10T18:48:45.631814+0000","last_fullsized":"2023-06-10T18:48:45.631814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:12.374835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":101,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.160261+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:48:45.160261+0000","last_peered":"2023-06-10T18:48:45.160261+0000","last_clean":"2023-06-10T18:48:45.160261+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:48:45.160261+0000","last_undegraded":"2023-06-10T18:48:45.160261+0000","last_fullsized":"2023-06-10T18:48:45.160261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:15:55.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.180366+0000","last_change":"2023-06-10T18:48:44.175016+0000","last_active":"2023-06-10T18:48:44.180366+0000","last_peered":"2023-06-10T18:48:44.180366+0000","last_clean":"2023-06-10T18:48:44.180366+0000","last_became_active":"2023-06-10T18:48:44.174079+0000","last_became_peered":"2023-06-10T18:48:44.174079+0000","last_unstale":"2023-06-10T18:48:44.180366+0000","last_undegraded":"2023-06-10T18:48:44.180366+0000","last_fullsized":"2023-06-10T18:48:44.180366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:56:49.791107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183405+0000","last_change":"2023-06-10T18:48:43.770650+0000","last_active":"2023-06-10T18:48:44.183405+0000","last_peered":"2023-06-10T18:48:44.183405+0000","last_clean":"2023-06-10T18:48:44.183405+0000","last_became_active":"2023-06-10T18:48:43.770271+0000","last_became_peered":"2023-06-10T18:48:43.770271+0000","last_unstale":"2023-06-10T18:48:44.183405+0000","last_undegraded":"2023-06-10T18:48:44.183405+0000","last_fullsized":"2023-06-10T18:48:44.183405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:23.906856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161243+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:48:45.161243+0000","last_peered":"2023-06-10T18:48:45.161243+0000","last_clean":"2023-06-10T18:48:45.161243+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:48:45.161243+0000","last_undegraded":"2023-06-10T18:48:45.161243+0000","last_fullsized":"2023-06-10T18:48:45.161243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:36.895969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.161503+0000","last_change":"2023-06-10T18:48:44.174386+0000","last_active":"2023-06-10T18:48:45.161503+0000","last_peered":"2023-06-10T18:48:45.161503+0000","last_clean":"2023-06-10T18:48:45.161503+0000","last_became_active":"2023-06-10T18:48:44.174019+0000","last_became_peered":"2023-06-10T18:48:44.174019+0000","last_unstale":"2023-06-10T18:48:45.161503+0000","last_undegraded":"2023-06-10T18:48:45.161503+0000","last_fullsized":"2023-06-10T18:48:45.161503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:35:23.594227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756805+0000","last_change":"2023-06-10T18:48:43.771185+0000","last_active":"2023-06-10T18:48:45.756805+0000","last_peered":"2023-06-10T18:48:45.756805+0000","last_clean":"2023-06-10T18:48:45.756805+0000","last_became_active":"2023-06-10T18:48:43.770925+0000","last_became_peered":"2023-06-10T18:48:43.770925+0000","last_unstale":"2023-06-10T18:48:45.756805+0000","last_undegraded":"2023-06-10T18:48:45.756805+0000","last_fullsized":"2023-06-10T18:48:45.756805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:56:58.400363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.164091+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:48:44.164091+0000","last_peered":"2023-06-10T18:48:44.164091+0000","last_clean":"2023-06-10T18:48:44.164091+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:48:44.164091+0000","last_undegraded":"2023-06-10T18:48:44.164091+0000","last_fullsized":"2023-06-10T18:48:44.164091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:45.226104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-06-10T18:48:45.756706+0000","last_change":"2023-06-10T18:48:44.164870+0000","last_active":"2023-06-10T18:48:45.756706+0000","last_peered":"2023-06-10T18:48:45.756706+0000","last_clean":"2023-06-10T18:48:45.756706+0000","last_became_active":"2023-06-10T18:48:44.164573+0000","last_became_peered":"2023-06-10T18:48:44.164573+0000","last_unstale":"2023-06-10T18:48:45.756706+0000","last_undegraded":"2023-06-10T18:48:45.756706+0000","last_fullsized":"2023-06-10T18:48:45.756706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:43.740625+0000","last_clean_scrub_stamp":"2023-06-10T18:48:43.740625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:53:24.624705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.183116+0000","last_change":"2023-06-10T18:48:43.771408+0000","last_active":"2023-06-10T18:48:44.183116+0000","last_peered":"2023-06-10T18:48:44.183116+0000","last_clean":"2023-06-10T18:48:44.183116+0000","last_became_active":"2023-06-10T18:48:43.770509+0000","last_became_peered":"2023-06-10T18:48:43.770509+0000","last_unstale":"2023-06-10T18:48:44.183116+0000","last_undegraded":"2023-06-10T18:48:44.183116+0000","last_fullsized":"2023-06-10T18:48:44.183116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:48:42.731881+0000","last_clean_scrub_stamp":"2023-06-10T18:48:42.731881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:21.872946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":93,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-06-10T18:48:44.173444+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:48:44.173444+0000","last_peered":"2023-06-10T18:48:44.173444+0000","last_clean":"2023-06-10T18:48:44.173444+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:48:44.173444+0000","last_undegraded":"2023-06-10T18:48:44.173444+0000","last_fullsized":"2023-06-10T18:48:44.173444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:14:20.580095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1425408,"data_stored":976604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.46899999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.41399999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.80900000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.443,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47699999999999998},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67300000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.433}]},{"osd":3,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.497},"min":{"1min":0.41999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73899999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63500000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59099999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.71499999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:48:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67700000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55300000000000005,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":815104,"data_stored":377720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.43099999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.75700000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52000000000000002,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52400000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.72399999999999998,"15min":0.89000000000000001},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":9.6850000000000005,"15min":9.6850000000000005},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65400000000000003,"15min":0.77400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73099999999999998,"5min":6.6689999999999996,"15min":6.6689999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.437,"5min":0.316,"15min":0.316},"max":{"1min":0.71299999999999997,"5min":1.252,"15min":1.252},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.69099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.52300000000000002},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.40899999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78600000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.623},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.55800000000000005},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:48:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.442,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":379808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.46300000000000002,"5min":0.753,"15min":0.753},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.33000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57499999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.247},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56599999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32600000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55900000000000005,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.373}]},{"osd":4,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.61199999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86799999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":1.008,"15min":1.008},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.43099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77100000000000002,"5min":0.876,"15min":0.876},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1400832,"data_stored":967996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.35099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.435,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.35499999999999998,"15min":0.35899999999999999},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.51900000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.373,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.45200000000000001,"5min":1.405,"15min":1.405},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51500000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.372,"15min":0.378},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70999999999999996,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39600000000000002,"15min":0.39700000000000002},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52100000000000002,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.313}]},{"osd":4,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73999999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.40000000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.58499999999999996}]},{"osd":5,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90800000000000003,"5min":0.999,"15min":0.999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81499999999999995,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.436,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":1.363,"15min":1.363},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.40600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93500000000000005,"5min":1.034,"15min":1.034},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":835584,"data_stored":387184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.496,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.249}]},{"osd":1,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.34699999999999998},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.28199999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73799999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.42599999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.48599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.47799999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77400000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.36499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83699999999999997,"5min":0.872,"15min":0.872},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84999999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63700000000000001,"15min":0.63800000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61099999999999999,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.92400000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":831488,"data_stored":385732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.38500000000000001},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47799999999999998,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.41399999999999998},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.505,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.377,"15min":0.39700000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41499999999999998,"15min":0.438},"min":{"1min":0.28199999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":1.3,"15min":1.3},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.42499999999999999},"min":{"1min":0.28399999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53000000000000003,"5min":1.109,"15min":1.109},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44700000000000001,"15min":0.48599999999999999},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.599,"15min":1.599},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60199999999999998,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.746,"5min":1.925,"15min":1.925},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84499999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.47999999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90100000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.68200000000000005,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77500000000000002,"5min":3.802,"15min":3.802},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67500000000000004,"15min":0.76500000000000001},"min":{"1min":0.438,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78600000000000003,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.622,"15min":0.628},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81999999999999995,"5min":1.03,"15min":1.03},"last":0.505}]},{"osd":7,"last update":"Sat Jun 10 18:48:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.623,"15min":0.64100000000000001},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86799999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.47199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82099999999999995,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":811008,"data_stored":377698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52100000000000002,"15min":0.50800000000000001},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.77000000000000002,"5min":1.119,"15min":1.119},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.45600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.67200000000000004,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52200000000000002,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65700000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.50900000000000001},"min":{"1min":0.34999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68700000000000006,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.39900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72899999999999998,"5min":1.042,"15min":1.042},"last":0.624}]},{"osd":5,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72899999999999998,"5min":1.069,"15min":1.069},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.502,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77600000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51300000000000001,"15min":0.54000000000000004},"min":{"1min":0.34499999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.54800000000000004},"min":{"1min":0.36499999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76200000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81000000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.81599999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":385710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35899999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.72499999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.48499999999999999},"min":{"1min":0.39600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70699999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.69999999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.40100000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.42399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68999999999999995,"5min":0.745,"15min":0.745},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.47799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64700000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.33700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67200000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.502,"15min":0.505},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.61799999999999999,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52800000000000002,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69699999999999995,"5min":0.879,"15min":0.879},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69499999999999995,"5min":0.754,"15min":0.754},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:48:48.625 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:48:48.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:48:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.775+0000 7f2043279700 1 Processor -- start 2023-06-10T18:48:48.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.775+0000 7f2043279700 1 -- start start 2023-06-10T18:48:48.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2043279700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f203c15c580 0x7f203c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2043279700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c06cde0 0x7f203c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2043279700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c06e250 con 0x7f203c06e6e0 2023-06-10T18:48:48.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2043279700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203c15c950 con 0x7f203c15c580 2023-06-10T18:48:48.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2043279700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0b1e80 con 0x7f203c06cde0 2023-06-10T18:48:48.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2040814700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f203c06e6e0 0x7f203c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57566/0 (socket says 172.21.15.104:57566) 2023-06-10T18:48:48.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2040814700 1 -- 172.21.15.104:0/4043826186 learned_addr learned my addr 172.21.15.104:0/4043826186 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:48.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2041816700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c06cde0 0x7f203c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> v2:172.21.15.114:3300/0 conn(0x7f203c15c580 0x7f203c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.776+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378792560 0 0) 0x7f203c06e250 con 0x7f203c06e6e0 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20280035b0 con 0x7f203c06e6e0 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554975436 0 0) 0x7f20280035b0 con 0x7f203c06e6e0 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c06cde0 msgr2=0x7f203c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c06cde0 0x7f203c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 >> v2:172.21.15.114:3300/0 conn(0x7f203c15c580 msgr2=0x7f203c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 --2- 172.21.15.104:0/4043826186 >> v2:172.21.15.114:3300/0 conn(0x7f203c15c580 0x7f203c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c06d920 con 0x7f203c06e6e0 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2038001670 con 0x7f203c06e6e0 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 >> v1:172.21.15.104:6789/0 conn(0x7f203c06e6e0 legacy=0x7f203c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f2028003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2028004430 0x7f20280048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2028004f60 0x7f202800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c15c950 con 0x7f2028003a00 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0b1e80 con 0x7f2028004430 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202800bf50 con 0x7f2028004f60 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041816700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c06cde0 0x7f203c0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> v2:172.21.15.114:3300/0 conn(0x7f203c15c580 0x7f203c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f2028003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2040814700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2028004430 0x7f20280048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041816700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2028004f60 0x7f202800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 -- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2028004f60 msgr2=0x7f202800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2028004f60 0x7f202800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 -- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2028004430 msgr2=0x7f20280048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2028004430 0x7f20280048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 -- 172.21.15.104:0/4043826186 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280035b0 con 0x7f2028003a00 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.777+0000 7f2041015700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f2028003e10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f202c009cf0 tx=0x7f2028005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb557e36c68a9aa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004030 con 0x7f2028003a00 2023-06-10T18:48:48.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c004190 con 0x7f2028003a00 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2033fff700 1 -- 172.21.15.104:0/4043826186 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004500 con 0x7f2028003a00 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 -- 172.21.15.104:0/4043826186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 msgr2=0x7f2028003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 --2- 172.21.15.104:0/4043826186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f2028003e10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f202c009cf0 tx=0x7f2028005880 comp rx=0 tx=0).stop 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2041816700 1 -- 172.21.15.104:0/4043826186 reap_dead start 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 -- 172.21.15.104:0/4043826186 shutdown_connections 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 -- 172.21.15.104:0/4043826186 >> 172.21.15.104:0/4043826186 conn(0x7f203c06b8a0 msgr2=0x7f203c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 -- 172.21.15.104:0/4043826186 shutdown_connections 2023-06-10T18:48:48.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.778+0000 7f2043279700 1 -- 172.21.15.104:0/4043826186 wait complete. 2023-06-10T18:48:48.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.779+0000 7f2043279700 1 Processor -- start 2023-06-10T18:48:48.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.779+0000 7f2043279700 1 -- start start 2023-06-10T18:48:48.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 0x7f203c1e1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c157860 0x7f203c0b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1570b0 con 0x7f2028003a00 2023-06-10T18:48:48.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0b0ff0 con 0x7f203c0b0b60 2023-06-10T18:48:48.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2043279700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1f6220 con 0x7f203c157860 2023-06-10T18:48:48.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59750/0 (socket says 172.21.15.104:59750) 2023-06-10T18:48:48.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 -- 172.21.15.104:0/3949571474 learned_addr learned my addr 172.21.15.104:0/3949571474 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:48.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2040814700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 0x7f203c1e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 -- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c157860 msgr2=0x7f203c0b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041816700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c157860 0x7f203c0b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c157860 0x7f203c0b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 -- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 msgr2=0x7f203c1e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 0x7f203c1e1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2041015700 1 -- 172.21.15.104:0/3949571474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0b24c0 con 0x7f2028003a00 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.780+0000 7f2040814700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 0x7f203c1e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2041015700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e4b0 tx=0x7f202c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c021070 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0038b0 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c0b27a0 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004030 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.781+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c1e1d50 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.782+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c0b2960 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.782+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f202c003c40 con 0x7f2028003a00 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.783+0000 7f2031ffb700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 0x7f202006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:48.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.783+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f202c08a7c0 con 0x7f2028003a00 2023-06-10T18:48:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.783+0000 7f2040814700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 0x7f202006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.783+0000 7f2040814700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 0x7f202006d190 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2038000c00 tx=0x7f2038009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:48.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:48.787+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f202c055860 con 0x7f2028003a00 2023-06-10T18:48:49.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.047+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f203c0b2b10 con 0x7f2028003a00 2023-06-10T18:48:49.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.053+0000 7f2031ffb700 1 -- 172.21.15.104:0/3949571474 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f202c058e80 con 0x7f2028003a00 2023-06-10T18:48:49.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 msgr2=0x7f202006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 0x7f202006d190 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2038000c00 tx=0x7f2038009040 comp rx=0 tx=0).stop 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 msgr2=0x7f203c156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e4b0 tx=0x7f202c0047c0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 shutdown_connections 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f203c157860 0x7f203c0b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2028003a00 0x7f203c156b70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f202006ace0 0x7f202006d190 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 --2- 172.21.15.104:0/3949571474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f203c0b0b60 0x7f203c1e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 >> 172.21.15.104:0/3949571474 conn(0x7f203c06b8a0 msgr2=0x7f203c159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 shutdown_connections 2023-06-10T18:48:49.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.055+0000 7f2043279700 1 -- 172.21.15.104:0/3949571474 wait complete. 2023-06-10T18:48:49.063 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:48:49.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:49.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.185+0000 7fafbb6a4700 1 Processor -- start 2023-06-10T18:48:49.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.185+0000 7fafbb6a4700 1 -- start start 2023-06-10T18:48:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafbb6a4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafb40cc710 0x7fafb406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafbb6a4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb415cb00 con 0x7fafb415c730 2023-06-10T18:48:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafbb6a4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb406ecd0 con 0x7fafb40ca0b0 2023-06-10T18:48:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafbb6a4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40aed80 con 0x7fafb40cc710 2023-06-10T18:48:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafb9440700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fafb415c730 0x7fafb40cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57586/0 (socket says 172.21.15.104:57586) 2023-06-10T18:48:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafb9440700 1 -- 172.21.15.104:0/3328168558 learned_addr learned my addr 172.21.15.104:0/3328168558 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.186+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3328168558 >> v2:172.21.15.114:3300/0 conn(0x7fafb40cc710 0x7fafb406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083639387 0 0) 0x7fafa4001290 con 0x7fafb415c730 2023-06-10T18:48:49.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa00035b0 con 0x7fafb415c730 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366649723 0 0) 0x7fafa00035b0 con 0x7fafb415c730 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 >> v2:172.21.15.114:3300/0 conn(0x7fafb40cc710 msgr2=0x7fafb406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 --2- 172.21.15.104:0/3328168558 >> v2:172.21.15.114:3300/0 conn(0x7fafb40cc710 0x7fafb406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 >> v1:172.21.15.114:6789/0 conn(0x7fafb40ca0b0 legacy=0x7fafb40ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40cd280 con 0x7fafb415c730 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fafa4001980 con 0x7fafb415c730 2023-06-10T18:48:49.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 >> v1:172.21.15.104:6789/0 conn(0x7fafb415c730 legacy=0x7fafb40cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa0003a00 0x7fafa0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0004850 0x7fafa0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 0x7fafa000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40aed80 con 0x7fafa0005200 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa000c110 con 0x7fafa0003a00 2023-06-10T18:48:49.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa000c2d0 con 0x7fafa0004850 2023-06-10T18:48:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 0x7fafa000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9440700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa0003a00 0x7fafa0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0004850 0x7fafa0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 -- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0004850 msgr2=0x7fafa0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0004850 0x7fafa0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 -- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa0003a00 msgr2=0x7fafa0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafa0003a00 0x7fafa0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 -- 172.21.15.104:0/3328168558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00035b0 con 0x7fafa0005200 2023-06-10T18:48:49.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0004850 0x7fafa0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 -- 172.21.15.104:0/3328168558 reap_dead start 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.187+0000 7fafb9c41700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 0x7fafa000ba10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c190 tx=0x7fafa0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b353c652b4b69221 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac001db0 con 0x7fafa0005200 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac00ccf0 con 0x7fafa0005200 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafabfff700 1 -- 172.21.15.104:0/3328168558 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac003c40 con 0x7fafa0005200 2023-06-10T18:48:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 msgr2=0x7fafa000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 0x7fafa000ba10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c190 tx=0x7fafa0005880 comp rx=0 tx=0).stop 2023-06-10T18:48:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3328168558 shutdown_connections 2023-06-10T18:48:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3328168558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa0005200 0x7fafa000ba10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3328168558 >> 172.21.15.104:0/3328168558 conn(0x7fafb406b8a0 msgr2=0x7fafb40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3328168558 shutdown_connections 2023-06-10T18:48:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.188+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3328168558 wait complete. 2023-06-10T18:48:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.189+0000 7fafbb6a4700 1 Processor -- start 2023-06-10T18:48:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.189+0000 7fafbb6a4700 1 -- start start 2023-06-10T18:48:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0005200 0x7fafb41f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb40ac2c0 0x7fafb40ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40c4070 con 0x7fafb40c66e0 2023-06-10T18:48:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40acca0 con 0x7fafb40ac2c0 2023-06-10T18:48:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafbb6a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40c78b0 con 0x7fafa0005200 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafb8c3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafb8c3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59772/0 (socket says 172.21.15.104:59772) 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafb8c3f700 1 -- 172.21.15.104:0/3359840920 learned_addr learned my addr 172.21.15.104:0/3359840920 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafb9440700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0005200 0x7fafb41f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.190+0000 7fafb9c41700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb40ac2c0 0x7fafb40ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 -- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0005200 msgr2=0x7fafb41f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0005200 0x7fafb41f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 -- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb40ac2c0 msgr2=0x7fafb40ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb40ac2c0 0x7fafb40ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 -- 172.21.15.104:0/3359840920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40c6ca0 con 0x7fafb40c66e0 2023-06-10T18:48:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafb8c3f700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fafb00108c0 tx=0x7fafb0010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0010e60 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb0004020 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb40c6f80 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00043b0 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.191+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb40c7540 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.192+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb40c7a40 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.192+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fafb000e8d0 con 0x7fafb40c66e0 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.193+0000 7fafa9ffb700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.193+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fafb0089570 con 0x7fafb40c66e0 2023-06-10T18:48:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.193+0000 7fafb9440700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.194+0000 7fafb9440700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fafa4000ef0 tx=0x7fafa4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.197+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fafb00544e0 con 0x7fafb40c66e0 2023-06-10T18:48:49.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.427+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafb4074050 con 0x7fafb40c66e0 2023-06-10T18:48:49.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.428+0000 7fafa9ffb700 1 -- 172.21.15.104:0/3359840920 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fafb0057b00 con 0x7fafb40c66e0 2023-06-10T18:48:49.434 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:49.435 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:49.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.429+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 msgr2=0x7faf9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.429+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fafa4000ef0 tx=0x7fafa4009040 comp rx=0 tx=0).stop 2023-06-10T18:48:49.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.429+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 msgr2=0x7fafb4151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.429+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fafb00108c0 tx=0x7fafb0010c80 comp rx=0 tx=0).stop 2023-06-10T18:48:49.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 shutdown_connections 2023-06-10T18:48:49.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf9806ace0 0x7faf9806d190 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafb40ac2c0 0x7fafb40ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafa0005200 0x7fafb41f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 --2- 172.21.15.104:0/3359840920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafb40c66e0 0x7fafb4151450 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 >> 172.21.15.104:0/3359840920 conn(0x7fafb406b8a0 msgr2=0x7fafb40ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:49.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 shutdown_connections 2023-06-10T18:48:49.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.430+0000 7fafbb6a4700 1 -- 172.21.15.104:0/3359840920 wait complete. 2023-06-10T18:48:49.439 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:49.734 INFO:tasks.ceph.mds.d.smithi114.stderr:2023-06-10T18:48:49.729+0000 7fd5acee0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:48:49.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:48:44.146291+0000', 'modified': '2023-06-10T18:48:47.161449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6699': {'gid': 6699, 'name': 'd', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/1280497888', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 1280497888}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 1280497888}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:48:49.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:48:49.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:48:49.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.971+0000 7f9661f95700 1 Processor -- start 2023-06-10T18:48:49.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.971+0000 7f9661f95700 1 -- start start 2023-06-10T18:48:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0aeaa0 0x7f965c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f965c0b1080 0x7f965c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f965c0ca2f0 0x7f965c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965c0c9bc0 con 0x7f965c0b1080 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965c0c9e60 con 0x7f965c0ca2f0 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9661f95700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0aee70 con 0x7f965c0aeaa0 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9652ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f965c0b1080 0x7f965c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9652ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f965c0b1080 0x7f965c0cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59778/0 (socket says 172.21.15.104:59778) 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.972+0000 7f9652ffd700 1 -- 172.21.15.104:0/1986926063 learned_addr learned my addr 172.21.15.104:0/1986926063 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f965bfff700 1 --2- 172.21.15.104:0/1986926063 >> v2:172.21.15.114:3300/0 conn(0x7f965c0ca2f0 0x7f965c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f965b7fe700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0aeaa0 0x7f965c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 -- 172.21.15.104:0/1986926063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0aeaa0 msgr2=0x7f965c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0aeaa0 0x7f965c0cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 -- 172.21.15.104:0/1986926063 >> v2:172.21.15.114:3300/0 conn(0x7f965c0ca2f0 msgr2=0x7f965c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 --2- 172.21.15.104:0/1986926063 >> v2:172.21.15.114:3300/0 conn(0x7f965c0ca2f0 0x7f965c0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 -- 172.21.15.104:0/1986926063 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c06e540 con 0x7f965c0b1080 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9652ffd700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0b1080 0x7f965c0cd680 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f965c06fa30 tx=0x7f9644009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e0911433292d8f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f965affd700 1 -- 172.21.15.104:0/1986926063 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964400b870 con 0x7f965c0b1080 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f965affd700 1 -- 172.21.15.104:0/1986926063 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964400b9d0 con 0x7f965c0b1080 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f965affd700 1 -- 172.21.15.104:0/1986926063 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964400bd60 con 0x7f965c0b1080 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9661f95700 1 -- 172.21.15.104:0/1986926063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0b1080 msgr2=0x7f965c0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.973+0000 7f9661f95700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0b1080 0x7f965c0cd680 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f965c06fa30 tx=0x7f9644009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 -- 172.21.15.104:0/1986926063 shutdown_connections 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 --2- 172.21.15.104:0/1986926063 >> v2:172.21.15.114:3300/0 conn(0x7f965c0ca2f0 0x7f965c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0aeaa0 0x7f965c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 --2- 172.21.15.104:0/1986926063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0b1080 0x7f965c0cd680 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 -- 172.21.15.104:0/1986926063 >> 172.21.15.104:0/1986926063 conn(0x7f965c06b8a0 msgr2=0x7f965c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 -- 172.21.15.104:0/1986926063 shutdown_connections 2023-06-10T18:48:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.974+0000 7f9661f95700 1 -- 172.21.15.104:0/1986926063 wait complete. 2023-06-10T18:48:49.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.975+0000 7f9661f95700 1 Processor -- start 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.975+0000 7f9661f95700 1 -- start start 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0b1080 0x7f965c0b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 0x7f965c0b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c155350 con 0x7f965c0aeaa0 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0b4270 con 0x7f965c0ca2f0 2023-06-10T18:48:49.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0b4550 con 0x7f965c0b1080 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59784/0 (socket says 172.21.15.104:59784) 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 -- 172.21.15.104:0/3745228495 learned_addr learned my addr 172.21.15.104:0/3745228495 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0b1080 0x7f965c0b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965bfff700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 0x7f965c0b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 -- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0b1080 msgr2=0x7f965c0b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0b1080 0x7f965c0b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 -- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 msgr2=0x7f965c0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:49.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 0x7f965c0b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 -- 172.21.15.104:0/3745228495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644009550 con 0x7f965c0aeaa0 2023-06-10T18:48:49.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965bfff700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 0x7f965c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f964c00aeb0 tx=0x7f964c00cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9658ff9700 1 -- 172.21.15.104:0/3745228495 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c010910 con 0x7f965c0aeaa0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9658ff9700 1 -- 172.21.15.104:0/3745228495 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c010a70 con 0x7f965c0aeaa0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c0b4830 con 0x7f965c0aeaa0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9658ff9700 1 -- 172.21.15.104:0/3745228495 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c010e20 con 0x7f965c0aeaa0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.976+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c0b4df0 con 0x7f965c0aeaa0 2023-06-10T18:48:49.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.978+0000 7f9658ff9700 1 -- 172.21.15.104:0/3745228495 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f964c00f430 con 0x7f965c0aeaa0 2023-06-10T18:48:49.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.978+0000 7f9658ff9700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 0x7f964006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.979+0000 7f9658ff9700 1 -- 172.21.15.104:0/3745228495 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f964c014070 con 0x7f965c0aeaa0 2023-06-10T18:48:49.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.979+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 0x7f964006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.979+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 0x7f964006d190 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9644000c00 tx=0x7f9644008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.980+0000 7f966e11ab80 1 Processor -- start 2023-06-10T18:48:49.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.980+0000 7f966e11ab80 1 -- start start 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x5569022669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902266f00 0x55690226a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556902144c60 con 0x556902240270 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569021979f0 con 0x55690223d740 2023-06-10T18:48:49.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x556902190e80 con 0x556902266f00 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f965b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f965b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59792/0 (socket says 172.21.15.104:59792) 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.981+0000 7f965b7fe700 1 -- 172.21.15.104:0/4286037236 learned_addr learned my addr 172.21.15.104:0/4286037236 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f9652ffd700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x5569022669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965bfff700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902266f00 0x55690226a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 -- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902266f00 msgr2=0x55690226a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902266f00 0x55690226a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 -- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 msgr2=0x5569022669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x5569022669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 -- 172.21.15.104:0/4286037236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569020125e0 con 0x556902240270 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f9652ffd700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x5569022669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f965b7fe700 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f964c061dd0 tx=0x7f964c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20622341946959f3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f96517fa700 1 -- 172.21.15.104:0/4286037236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c062300 con 0x556902240270 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f96517fa700 1 -- 172.21.15.104:0/4286037236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c0582c0 con 0x556902240270 2023-06-10T18:48:49.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.982+0000 7f96517fa700 1 -- 172.21.15.104:0/4286037236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c061e70 con 0x556902240270 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- 172.21.15.104:0/4286037236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 msgr2=0x556902240640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f964c061dd0 tx=0x7f964c004750 comp rx=0 tx=0).stop 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- 172.21.15.104:0/4286037236 shutdown_connections 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902266f00 0x55690226a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902240640 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 --2- 172.21.15.104:0/4286037236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x5569022669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- 172.21.15.104:0/4286037236 >> 172.21.15.104:0/4286037236 conn(0x556902234fc0 msgr2=0x5569022851a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- 172.21.15.104:0/4286037236 shutdown_connections 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- 172.21.15.104:0/4286037236 wait complete. 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 -- start start 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.983+0000 7f966e11ab80 1 Processor -- start 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x556902285710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 0x556902265910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556902130a70 con 0x556902266f00 2023-06-10T18:48:50.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x556902130770 con 0x55690223d740 2023-06-10T18:48:50.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569020148d0 con 0x556902240270 2023-06-10T18:48:50.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f966e11ab80 10 client.0 _ll_register_callbacks cb 0x556902136210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:48:50.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f965bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f965bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59802/0 (socket says 172.21.15.104:59802) 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f965bfff700 1 -- 172.21.15.104:0/2965403692 learned_addr learned my addr 172.21.15.104:0/2965403692 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f965b7fe700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x556902285710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f9652ffd700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 0x556902265910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.984+0000 7f965bfff700 1 -- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 msgr2=0x556902265910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965bfff700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 0x556902265910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965bfff700 1 -- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 msgr2=0x556902285710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965bfff700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x556902285710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965bfff700 1 -- 172.21.15.104:0/2965403692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c0b4df0 con 0x556902266f00 2023-06-10T18:48:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f9652ffd700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 0x556902265910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965bfff700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9654010b80 tx=0x7f9654010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3bcc5a42ddcd76 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965400eb10 con 0x556902266f00 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f965400ec70 con 0x556902266f00 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556902232780 con 0x556902266f00 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f965400ee30 con 0x556902266f00 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 10 client.6846 fetch_fsmap learned FSMap version 33 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x556901fbed90 con 0x556902266f00 2023-06-10T18:48:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556902232780 con 0x556902266f00 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f965401a030 con 0x556902266f00 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 10 client.6846 fetch_fsmap finished waiting for FSMap version 33 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 10 client.6846 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 20 client.6846 populate_metadata read hostname 'smithi104' 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x556902258960 0x556902258d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 4 client.6846 mds_command: new command op to 6699 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 --> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556902189e60 con 0x556902258960 2023-06-10T18:48:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.985+0000 7f965b7fe700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x556902258960 0x556902258d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.986+0000 7f964a7fc700 10 client.6846 ms_handle_connect on v2:172.21.15.114:6834/1280497888 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.986+0000 7f965b7fe700 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x556902258960 0x556902258d30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f58609a08fcc708 server_cookie=12b5adaf60655813 in_seq=0 out_seq=0 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.986+0000 7f964a7fc700 1 -- 172.21.15.104:0/2965403692 <== mds.0 v2:172.21.15.114:6834/1280497888 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556902189e60 con 0x556902258960 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.986+0000 7f964a7fc700 10 client.6846 handle_command_reply: tid=0 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 client.6846 shutdown 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 msgr2=0x556902265e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9654010b80 tx=0x7f9654010e90 comp rx=0 tx=0).stop 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 shutdown_connections 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x556902258960 0x556902258d30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55690223d740 0x556902285710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902266f00 0x556902265e50 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2965403692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556902240270 0x556902265910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 >> 172.21.15.104:0/2965403692 conn(0x556902234fc0 msgr2=0x556902239600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f964a7fc700 0 client.6846 ms_handle_reset on v2:172.21.15.114:6834/1280497888 2023-06-10T18:48:50.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 shutdown_connections 2023-06-10T18:48:50.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 1 -- 172.21.15.104:0/2965403692 wait complete. 2023-06-10T18:48:50.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:49.987+0000 7f966e11ab80 20 client.6846 trim_cache size 0 max 16384 2023-06-10T18:48:50.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.004+0000 7f966e11ab80 1 Processor -- start 2023-06-10T18:48:50.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.004+0000 7f966e11ab80 1 -- start start 2023-06-10T18:48:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x55690226a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902265f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556902285650 con 0x556902240270 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55690225f6f0 con 0x556902266f00 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x556902257940 con 0x55690223d740 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59808/0 (socket says 172.21.15.104:59808) 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 -- 172.21.15.104:0/2382804463 learned_addr learned my addr 172.21.15.104:0/2382804463 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 -- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 msgr2=0x55690226a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x55690226a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 -- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 msgr2=0x556902265f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902265f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 -- 172.21.15.104:0/2382804463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556901fa7c10 con 0x556902240270 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.005+0000 7f9652ffd700 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9644013000 tx=0x7f9644010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a320f5a708910a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f9651ffb700 1 -- 172.21.15.104:0/2382804463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964402e070 con 0x556902240270 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f9651ffb700 1 -- 172.21.15.104:0/2382804463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96440037a0 con 0x556902240270 2023-06-10T18:48:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f9651ffb700 1 -- 172.21.15.104:0/2382804463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644020070 con 0x556902240270 2023-06-10T18:48:50.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 -- 172.21.15.104:0/2382804463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 msgr2=0x556902236590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9644013000 tx=0x7f9644010000 comp rx=0 tx=0).stop 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 -- 172.21.15.104:0/2382804463 shutdown_connections 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902265f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x55690226a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 --2- 172.21.15.104:0/2382804463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x556902236590 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 -- 172.21.15.104:0/2382804463 >> 172.21.15.104:0/2382804463 conn(0x556902234fc0 msgr2=0x556902267b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 -- 172.21.15.104:0/2382804463 shutdown_connections 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.006+0000 7f966e11ab80 1 -- 172.21.15.104:0/2382804463 wait complete. 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.007+0000 7f966e11ab80 1 -- start start 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.007+0000 7f966e11ab80 1 Processor -- start 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x5569022ef950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902288c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556902256c70 con 0x556902240270 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569022887a0 con 0x556902266f00 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x556902236990 con 0x55690223d740 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 10 client.0 _ll_register_callbacks cb 0x7f9654010400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59814/0 (socket says 172.21.15.104:59814) 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 -- 172.21.15.104:0/3562242221 learned_addr learned my addr 172.21.15.104:0/3562242221 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:50.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x5569022ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f965bfff700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902288c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 -- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 msgr2=0x5569022ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x5569022ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 -- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 msgr2=0x556902288c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902288c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 -- 172.21.15.104:0/3562242221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c06e540 con 0x556902240270 2023-06-10T18:48:50.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f9652ffd700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f965c06cc20 tx=0x7f96440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbd1948489bd3c0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f964a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644033660 con 0x556902240270 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f96440337c0 con 0x556902240270 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f964c0582c0 con 0x556902240270 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9644033980 con 0x556902240270 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.008+0000 7f966e11ab80 10 client.6852 fetch_fsmap learned FSMap version 33 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x556901fe5200 con 0x556902240270 2023-06-10T18:48:50.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c0582c0 con 0x556902240270 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9644033ae0 con 0x556902240270 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 10 client.6852 fetch_fsmap finished waiting for FSMap version 33 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 10 client.6852 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 20 client.6852 populate_metadata read hostname 'smithi104' 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x55690223c5b0 0x55690223c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 4 client.6852 mds_command: new command op to 6699 tid=0[{"prefix": "status"}] 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 --> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55690214c000 con 0x55690223c5b0 2023-06-10T18:48:50.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x55690223c5b0 0x55690223c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f964a7fc700 10 client.6852 ms_handle_connect on v2:172.21.15.114:6834/1280497888 2023-06-10T18:48:50.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.009+0000 7f965b7fe700 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x55690223c5b0 0x55690223c980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae15395f2e93ec4 server_cookie=2812cc03b3d56fea in_seq=0 out_seq=0 2023-06-10T18:48:50.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f964a7fc700 1 -- 172.21.15.104:0/3562242221 <== mds.0 v2:172.21.15.114:6834/1280497888 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55690214c000 con 0x55690223c5b0 2023-06-10T18:48:50.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f964a7fc700 10 client.6852 handle_command_reply: tid=0 2023-06-10T18:48:50.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 client.6852 shutdown 2023-06-10T18:48:50.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 msgr2=0x5569022563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f965c06cc20 tx=0x7f96440047c0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 shutdown_connections 2023-06-10T18:48:50.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:6834/1280497888,v1:172.21.15.114:6836/1280497888] conn(0x55690223c5b0 0x55690223c980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556902266f00 0x556902288c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55690223d740 0x5569022ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f964a7fc700 0 client.6852 ms_handle_reset on v2:172.21.15.114:6834/1280497888 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 --2- 172.21.15.104:0/3562242221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556902240270 0x5569022563e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 >> 172.21.15.104:0/3562242221 conn(0x556902234fc0 msgr2=0x556902238f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 shutdown_connections 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 1 -- 172.21.15.104:0/3562242221 wait complete. 2023-06-10T18:48:50.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f966e11ab80 20 client.6852 trim_cache size 0 max 16384 2023-06-10T18:48:50.027 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:50.027 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:48:50.027 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:48:50.027 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6699, 2023-06-10T18:48:50.027 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8607882309999999, 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 33, 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 68, 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 68, 2023-06-10T18:48:50.028 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.153637914000001 2023-06-10T18:48:50.029 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:50.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 msgr2=0x7f964006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 0x7f964006d190 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9644000c00 tx=0x7f9644008030 comp rx=0 tx=0).stop 2023-06-10T18:48:50.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 msgr2=0x7f965c154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f964c00aeb0 tx=0x7f964c00cad0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 shutdown_connections 2023-06-10T18:48:50.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f964006ace0 0x7f964006d190 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965c0ca2f0 0x7f965c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f965c0aeaa0 0x7f965c154e10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 --2- 172.21.15.104:0/3745228495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965c0b1080 0x7f965c0b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.010+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 >> 172.21.15.104:0/3745228495 conn(0x7f965c06b8a0 msgr2=0x7f965c0af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.011+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 shutdown_connections 2023-06-10T18:48:50.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.011+0000 7f9661f95700 1 -- 172.21.15.104:0/3745228495 wait complete. 2023-06-10T18:48:50.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:48:50.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.149+0000 7f1b7902f700 1 Processor -- start 2023-06-10T18:48:50.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.149+0000 7f1b7902f700 1 -- start start 2023-06-10T18:48:50.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7902f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 0x7f1b7406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7902f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7406d290 con 0x7f1b74069e70 2023-06-10T18:48:50.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7902f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740ca2d0 con 0x7f1b7406e0e0 2023-06-10T18:48:50.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7902f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74071640 con 0x7f1b7406e4b0 2023-06-10T18:48:50.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7359e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1b74069e70 0x7f1b740ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57650/0 (socket says 172.21.15.104:57650) 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7359e700 1 -- 172.21.15.104:0/3551610602 learned_addr learned my addr 172.21.15.104:0/3551610602 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7259c700 1 --2- 172.21.15.104:0/3551610602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 0x7f1b7406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191937691 0 0) 0x7f1b7406d290 con 0x7f1b74069e70 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b5c0035b0 con 0x7f1b74069e70 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588150464 0 0) 0x7f1b740ca2d0 con 0x7f1b7406e0e0 2023-06-10T18:48:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7406d290 con 0x7f1b7406e0e0 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1161575137 0 0) 0x7f1b5c0035b0 con 0x7f1b74069e70 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7259c700 1 -- 172.21.15.104:0/3551610602 >> v1:172.21.15.114:6789/0 conn(0x7f1b7406e0e0 legacy=0x7f1b740c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7259c700 1 -- 172.21.15.104:0/3551610602 >> v1:172.21.15.104:6789/0 conn(0x7f1b74069e70 legacy=0x7f1b740ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.150+0000 7f1b7259c700 1 -- 172.21.15.104:0/3551610602 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7406d4c0 con 0x7f1b7406e4b0 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7259c700 1 --2- 172.21.15.104:0/3551610602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 0x7f1b7406cd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009870 tx=0x7f1b64009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aedd79ede16fdd3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c0019f0 con 0x7f1b7406e4b0 2023-06-10T18:48:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64004030 con 0x7f1b7406e4b0 2023-06-10T18:48:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b71d9b700 1 -- 172.21.15.104:0/3551610602 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640043c0 con 0x7f1b7406e4b0 2023-06-10T18:48:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 -- 172.21.15.104:0/3551610602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 msgr2=0x7f1b7406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 --2- 172.21.15.104:0/3551610602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 0x7f1b7406cd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009870 tx=0x7f1b64009b80 comp rx=0 tx=0).stop 2023-06-10T18:48:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 -- 172.21.15.104:0/3551610602 shutdown_connections 2023-06-10T18:48:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 --2- 172.21.15.104:0/3551610602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e4b0 0x7f1b7406cd50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 -- 172.21.15.104:0/3551610602 >> 172.21.15.104:0/3551610602 conn(0x7f1b7406b8a0 msgr2=0x7f1b740cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 -- 172.21.15.104:0/3551610602 shutdown_connections 2023-06-10T18:48:50.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.151+0000 7f1b7902f700 1 -- 172.21.15.104:0/3551610602 wait complete. 2023-06-10T18:48:50.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.153+0000 7f1b7902f700 1 Processor -- start 2023-06-10T18:48:50.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.153+0000 7f1b7902f700 1 -- start start 2023-06-10T18:48:50.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 0x7f1b740b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b7406e4b0 0x7f1b740b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7415a1c0 con 0x7f1b74069e70 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740b5410 con 0x7f1b7406e4b0 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740b2ee0 con 0x7f1b7406e0e0 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59826/0 (socket says 172.21.15.104:59826) 2023-06-10T18:48:50.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 -- 172.21.15.104:0/133016405 learned_addr learned my addr 172.21.15.104:0/133016405 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7259c700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 0x7f1b740b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7359e700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b7406e4b0 0x7f1b740b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 -- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 msgr2=0x7f1b740b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 0x7f1b740b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 -- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b7406e4b0 msgr2=0x7f1b740b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b7406e4b0 0x7f1b740b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 -- 172.21.15.104:0/133016405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640094a0 con 0x7f1b74069e70 2023-06-10T18:48:50.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7259c700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 0x7f1b740b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b72d9d700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1b68001f00 tx=0x7f1b6800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b680052d0 con 0x7f1b74069e70 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b68005430 con 0x7f1b74069e70 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b740b31c0 con 0x7f1b74069e70 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b680038f0 con 0x7f1b74069e70 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.154+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b740b3780 con 0x7f1b74069e70 2023-06-10T18:48:50.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.155+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74074050 con 0x7f1b74069e70 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.156+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1b68003a50 con 0x7f1b74069e70 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.156+0000 7f1b5b7fe700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 0x7f1b6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.156+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1b68088a90 con 0x7f1b74069e70 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.157+0000 7f1b7259c700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 0x7f1b6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.157+0000 7f1b7259c700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 0x7f1b6006d190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b64013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:50.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.161+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1b68053a90 con 0x7f1b74069e70 2023-06-10T18:48:50.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.389+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b7406da50 con 0x7f1b74069e70 2023-06-10T18:48:50.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.389+0000 7f1b5b7fe700 1 -- 172.21.15.104:0/133016405 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1b68017060 con 0x7f1b74069e70 2023-06-10T18:48:50.396 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:48:50.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 msgr2=0x7f1b6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 0x7f1b6006d190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b64013040 comp rx=0 tx=0).stop 2023-06-10T18:48:50.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 msgr2=0x7f1b74159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:50.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1b68001f00 tx=0x7f1b6800c860 comp rx=0 tx=0).stop 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 shutdown_connections 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1b6006ace0 0x7f1b6006d190 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b7406e4b0 0x7f1b740b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b74069e70 0x7f1b74159920 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 --2- 172.21.15.104:0/133016405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b7406e0e0 0x7f1b740b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 >> 172.21.15.104:0/133016405 conn(0x7f1b7406b8a0 msgr2=0x7f1b740c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:50.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 shutdown_connections 2023-06-10T18:48:50.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:50.391+0000 7f1b7902f700 1 -- 172.21.15.104:0/133016405 wait complete. 2023-06-10T18:48:50.409 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:48:50.409 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:48:50.410 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:48:50.410 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:48:50.410 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:50.410 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:48:50.411 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:48:50.411 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:48:50.411 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:50.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.413 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:48:50.431 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:48:50.432 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:48:50.432 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:48:50.432 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.432 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:48:50.432 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:50.433 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:48:50.434 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:50.434 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:48:50.434 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42154sec preferred_lft 42154sec 2023-06-10T18:48:50.434 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:48:50.435 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.435 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:48:50.435 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:50.435 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:48:50.435 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.436 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:48:50.436 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.436 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:50.436 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:48:50.436 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:48:50.437 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.437 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:50.437 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:48:50.438 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:48:50.438 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:50.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.439 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:48:50.496 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:48:50.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.499 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:48:50.557 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:48:50.557 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:48:50.557 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:48:50.558 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:48:50.558 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:48:50.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.559 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:48:50.559 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:48:50.559 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:48:50.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:48:50.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:48:50.560 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:48:50.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.662 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:48:50.678 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:48:50.678 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:48:50.679 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:50.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:48:50.744 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:48:50.745 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:50.746 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:50.801 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:50.802 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:50.802 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:50.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:50.924 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:50.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:50.994 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:50.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:50.995 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:51.051 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:48:51.052 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:48:51.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:48:51.118 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:51.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:51.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:48:51.172+0000 7fead6e5a5c0 -1 init, newargv = 0x561ac5f2ea40 newargc=15 2023-06-10T18:48:51.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[128563]: starting ceph client 2023-06-10T18:48:51.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[128563]: starting fuse 2023-06-10T18:48:51.185 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:51.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:51.187 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:48:51.244 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:48:51.244 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:48:51.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:51.245 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:48:51.245 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:48:51.246 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:48:51.246 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:48:51.246 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:51.247 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:48:51.247 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:48:51.247 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:48:51.247 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:48:51.248 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:51.248 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:48:51.249 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:48:51.249 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:48:51.249 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:48:51.250 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:51.250 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:48:51.251 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:51.251 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:51.251 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:51.252 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:48:51.252 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:48:51.252 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:48:51.253 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:51.253 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:51.254 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:48:51.254 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:48:51.366 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.128563.asok 2023-06-10T18:48:51.548 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.128563.asok 2023-06-10T18:48:51.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:51.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.128563.asok status 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:48:51.703 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "128563", 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:48:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6864, 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6864 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:51.705 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:51.706 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2562980824 2023-06-10T18:48:51.706 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:51.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:51.707 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:51.707 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:51.707 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:51.707 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2562980824 2023-06-10T18:48:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.6864 192.168.0.1:0/2562980824", 2023-06-10T18:48:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2562980824", 2023-06-10T18:48:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:48:51.708 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 33, 2023-06-10T18:48:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 68, 2023-06-10T18:48:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:48:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:48:51.709 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:48:51.709 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:51.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:51.713 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:48:51.713 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:51.730 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:48:51.730 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:51.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:51.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:48:51.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:48:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.951+0000 7ff014062700 1 Processor -- start 2023-06-10T18:48:51.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.951+0000 7ff014062700 1 -- start start 2023-06-10T18:48:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff014062700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff014062700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff00c06b820 0x7ff00c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff014062700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c06bfd0 con 0x7ff00c0b1e30 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff014062700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00c06c2b0 con 0x7ff00c06b820 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff014062700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0ccf70 con 0x7ff00c0d0c00 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff0125ff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff00c06b820 0x7ff00c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff011dfe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff00c0b1e30 0x7ff00c0d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57676/0 (socket says 172.21.15.104:57676) 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff0125ff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff00c06b820 0x7ff00c0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59852/0 (socket says 172.21.15.104:59852) 2023-06-10T18:48:51.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.952+0000 7ff011dfe700 1 -- 172.21.15.104:0/785367943 learned_addr learned my addr 172.21.15.104:0/785367943 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0115fd700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0125ff700 1 -- 172.21.15.104:0/785367943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 msgr2=0x7ff00c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0125ff700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0125ff700 1 -- 172.21.15.104:0/785367943 >> v1:172.21.15.104:6789/0 conn(0x7ff00c0b1e30 legacy=0x7ff00c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0125ff700 1 -- 172.21.15.104:0/785367943 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c0cd250 con 0x7ff00c06b820 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff010dfc700 1 -- 172.21.15.104:0/785367943 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056765917 0 0) 0x7ff00c06bfd0 con 0x7ff00c0b1e30 2023-06-10T18:48:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff0125ff700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c06b820 0x7ff00c0cda50 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff004009d90 tx=0x7ff00400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfe48e436d171b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff010dfc700 1 -- 172.21.15.104:0/785367943 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00400e070 con 0x7ff00c06b820 2023-06-10T18:48:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff010dfc700 1 -- 172.21.15.104:0/785367943 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00400bbd0 con 0x7ff00c06b820 2023-06-10T18:48:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.953+0000 7ff010dfc700 1 -- 172.21.15.104:0/785367943 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004012430 con 0x7ff00c06b820 2023-06-10T18:48:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 -- 172.21.15.104:0/785367943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c06b820 msgr2=0x7ff00c0cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c06b820 0x7ff00c0cda50 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff004009d90 tx=0x7ff00400b330 comp rx=0 tx=0).stop 2023-06-10T18:48:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 -- 172.21.15.104:0/785367943 shutdown_connections 2023-06-10T18:48:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c06b820 0x7ff00c0cda50 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 --2- 172.21.15.104:0/785367943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 -- 172.21.15.104:0/785367943 >> 172.21.15.104:0/785367943 conn(0x7ff00c071ec0 msgr2=0x7ff00c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 -- 172.21.15.104:0/785367943 shutdown_connections 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.954+0000 7ff014062700 1 -- 172.21.15.104:0/785367943 wait complete. 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.955+0000 7ff014062700 1 Processor -- start 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.955+0000 7ff014062700 1 -- start start 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c06b820 0x7ff00c1f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c1f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c1f1c60 con 0x7ff00c0b1e30 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c1f1e50 con 0x7ff00c06b820 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c1f5ca0 con 0x7ff00c0d0c00 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59856/0 (socket says 172.21.15.104:59856) 2023-06-10T18:48:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 -- 172.21.15.104:0/216798882 learned_addr learned my addr 172.21.15.104:0/216798882 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff011dfe700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c06b820 0x7ff00c1f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0125ff700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c1f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 -- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 msgr2=0x7ff00c1f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c1f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 -- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c06b820 msgr2=0x7ff00c1f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c06b820 0x7ff00c1f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 -- 172.21.15.104:0/216798882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff004009a70 con 0x7ff00c0b1e30 2023-06-10T18:48:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff0115fd700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff00800d800 tx=0x7ff00800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080125f0 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff008012750 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008012ae0 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c1f5f80 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.956+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c1f2b50 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.957+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00c0d1860 con 0x7ff00c0b1e30 2023-06-10T18:48:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.958+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff008012c40 con 0x7ff00c0b1e30 2023-06-10T18:48:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.958+0000 7feffe7fc700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 0x7feff006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.958+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff0080899d0 con 0x7ff00c0b1e30 2023-06-10T18:48:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.958+0000 7ff011dfe700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 0x7feff006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.959+0000 7ff011dfe700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 0x7feff006d190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff000000f60 tx=0x7ff000007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:51.962+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff008054900 con 0x7ff00c0b1e30 2023-06-10T18:48:52.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.194+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff00c1f61c0 con 0x7ff00c0b1e30 2023-06-10T18:48:52.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.195+0000 7feffe7fc700 1 -- 172.21.15.104:0/216798882 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff008057f20 con 0x7ff00c0b1e30 2023-06-10T18:48:52.201 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:48:52.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 msgr2=0x7feff006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 0x7feff006d190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff000000f60 tx=0x7ff000007be0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 msgr2=0x7ff00c0b8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff00800d800 tx=0x7ff00800dbc0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 shutdown_connections 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feff006ace0 0x7feff006d190 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c06b820 0x7ff00c1f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff00c0d0c00 0x7ff00c1f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 --2- 172.21.15.104:0/216798882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff00c0b1e30 0x7ff00c0b8190 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:52.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 >> 172.21.15.104:0/216798882 conn(0x7ff00c071ec0 msgr2=0x7ff00c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:52.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 shutdown_connections 2023-06-10T18:48:52.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:52.197+0000 7ff014062700 1 -- 172.21.15.104:0/216798882 wait complete. 2023-06-10T18:48:52.215 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:48:52.215 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:48:52.216 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:48:52.217 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:48:52.217 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:52.218 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:48:52.218 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:48:52.218 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:48:52.219 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:52.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.220 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:48:52.241 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:48:52.241 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:48:52.241 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:48:52.242 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.242 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:48:52.242 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.242 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:52.243 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:52.243 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:52.243 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:52.243 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:48:52.244 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:52.244 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:48:52.244 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:52.245 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:48:52.245 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42148sec preferred_lft 42148sec 2023-06-10T18:48:52.245 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:48:52.245 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.246 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:48:52.246 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:48:52.246 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:48:52.246 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.247 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:48:52.247 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.247 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:48:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:48:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.248 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:48:52.249 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:48:52.249 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:48:52.249 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:48:52.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.250 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:48:52.312 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:48:52.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.314 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:48:52.375 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:48:52.376 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:48:52.376 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:48:52.378 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:48:52.378 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:48:52.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.380 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:48:52.381 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:48:52.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:48:52.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:48:52.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:48:52.384 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:48:52.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.490 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:48:52.508 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:48:52.509 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:48:52.509 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:52.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:48:52.576 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:48:52.577 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:52.577 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:52.633 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:48:52.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:48:52.634 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:52.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:48:52.759 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:52.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:52.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:52.827 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:48:52.884 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:48:52.884 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:48:52.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:48:52.953 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:48:52.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:48:53.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:48:53.008+0000 7f6f696415c0 -1 init, newargv = 0x562a7f329dd0 newargc=15 2023-06-10T18:48:53.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[110297]: starting ceph client 2023-06-10T18:48:53.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[110297]: starting fuse 2023-06-10T18:48:53.020 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:48:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:48:53.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:53.022 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:48:53.079 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:48:53.079 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:48:53.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:53.082 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:48:53.083 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:48:53.083 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:48:53.084 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:48:53.085 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:53.085 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:48:53.086 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:48:53.087 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:48:53.087 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:48:53.088 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:53.089 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:48:53.090 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:48:53.090 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:48:53.091 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:48:53.092 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:53.093 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:48:53.093 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:53.094 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:53.095 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:53.096 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:48:53.096 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:48:53.097 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:48:53.098 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:53.098 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:48:53.099 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:48:53.100 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:48:53.201 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.110297.asok 2023-06-10T18:48:53.383 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.110297.asok 2023-06-10T18:48:53.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:53.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.110297.asok status 2023-06-10T18:48:53.539 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:48:53.540 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:48:53.540 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:48:53.540 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:48:53.540 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:48:53.540 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:48:53.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "110297", 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5660, 2023-06-10T18:48:53.542 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5660 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:53.543 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2960984225 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2960984225 2023-06-10T18:48:53.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.5660 192.168.0.1:0/2960984225", 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/2960984225", 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 33, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 70, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:48:53.545 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:48:53.546 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:48:53.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:53.549 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:48:53.550 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:53.568 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:48:53.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:53.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:53.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:48:53.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:48:53.778 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:48:53.779 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:48:53.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:48:53.914 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:48:53.914 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:48:53.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:54.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.029+0000 7fa9eeb42700 1 Processor -- start 2023-06-10T18:48:54.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 -- start start 2023-06-10T18:48:54.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e81606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 0x7fa9e806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80ce2f0 con 0x7fa9e80ad8e0 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80cfd30 con 0x7fa9e806c9b0 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.030+0000 7fa9eeb42700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e806d360 con 0x7fa9e815c040 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ed0df700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa9e80ad8e0 0x7fa9e80add60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57694/0 (socket says 172.21.15.104:57694) 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ed0df700 1 -- 172.21.15.104:0/2541089747 learned_addr learned my addr 172.21.15.104:0/2541089747 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:54.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e7fff700 1 --2- 172.21.15.104:0/2541089747 >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 0x7fa9e806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/2541089747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e81606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e77fe700 1 -- 172.21.15.104:0/2541089747 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454049610 0 0) 0x7fa9e80ce2f0 con 0x7fa9e80ad8e0 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e77fe700 1 -- 172.21.15.104:0/2541089747 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d80035b0 con 0x7fa9e80ad8e0 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 -- 172.21.15.104:0/2541089747 >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 msgr2=0x7fa9e806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/2541089747 >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 0x7fa9e806ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 -- 172.21.15.104:0/2541089747 >> v1:172.21.15.104:6789/0 conn(0x7fa9e80ad8e0 legacy=0x7fa9e80add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 -- 172.21.15.104:0/2541089747 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8160c20 con 0x7fa9e815c040 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e7fff700 1 --2- 172.21.15.104:0/2541089747 >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 0x7fa9e806ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/2541089747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e81606e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009840 tx=0x7fa9dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f3fb4fbe240f70 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e77fe700 1 -- 172.21.15.104:0/2541089747 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc011830 con 0x7fa9e815c040 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e77fe700 1 -- 172.21.15.104:0/2541089747 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc011990 con 0x7fa9e815c040 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.031+0000 7fa9e77fe700 1 -- 172.21.15.104:0/2541089747 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc011d00 con 0x7fa9e815c040 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 -- 172.21.15.104:0/2541089747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 msgr2=0x7fa9e81606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/2541089747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e81606e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009840 tx=0x7fa9dc009b50 comp rx=0 tx=0).stop 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 -- 172.21.15.104:0/2541089747 shutdown_connections 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/2541089747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e81606e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/2541089747 >> v2:172.21.15.114:3300/0 conn(0x7fa9e806c9b0 0x7fa9e806ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 -- 172.21.15.104:0/2541089747 >> 172.21.15.104:0/2541089747 conn(0x7fa9e80b1570 msgr2=0x7fa9e80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 -- 172.21.15.104:0/2541089747 shutdown_connections 2023-06-10T18:48:54.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.032+0000 7fa9eeb42700 1 -- 172.21.15.104:0/2541089747 wait complete. 2023-06-10T18:48:54.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.033+0000 7fa9eeb42700 1 Processor -- start 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.033+0000 7fa9eeb42700 1 -- start start 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e80ad8e0 0x7fa9e81f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81f5590 con 0x7fa9e806c9b0 2023-06-10T18:48:54.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e815c410 con 0x7fa9e80ad8e0 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9eeb42700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81f57c0 con 0x7fa9e815c040 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59868/0 (socket says 172.21.15.104:59868) 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 -- 172.21.15.104:0/1496306322 learned_addr learned my addr 172.21.15.104:0/1496306322 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ed0df700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9e7fff700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e80ad8e0 0x7fa9e81f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 -- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 msgr2=0x7fa9e815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e815cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 -- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e80ad8e0 msgr2=0x7fa9e81f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e80ad8e0 0x7fa9e81f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 -- 172.21.15.104:0/1496306322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40094a0 con 0x7fa9e806c9b0 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.034+0000 7fa9ec8de700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009fd0 tx=0x7fa9dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.035+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc004550 con 0x7fa9e806c9b0 2023-06-10T18:48:54.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.035+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc01e070 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.035+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc0094a0 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.035+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0038e0 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.035+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81f5d80 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.036+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8074050 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.036+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa9dc003a40 con 0x7fa9e806c9b0 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.036+0000 7fa9e57fa700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.037+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fa9dc08a430 con 0x7fa9e806c9b0 2023-06-10T18:48:54.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.037+0000 7fa9e7fff700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.037+0000 7fa9e7fff700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80b4720 tx=0x7fa9d4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.041+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa9dc0553f0 con 0x7fa9e806c9b0 2023-06-10T18:48:54.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.274+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e80b4d10 con 0x7fa9e806c9b0 2023-06-10T18:48:54.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.275+0000 7fa9e57fa700 1 -- 172.21.15.104:0/1496306322 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fa9dc058a10 con 0x7fa9e806c9b0 2023-06-10T18:48:54.281 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:54.282 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:54.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 msgr2=0x7fa9cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80b4720 tx=0x7fa9d4009210 comp rx=0 tx=0).stop 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 msgr2=0x7fa9e80c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009fd0 tx=0x7fa9dc004750 comp rx=0 tx=0).stop 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 shutdown_connections 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e806c9b0 0x7fa9e80c6100 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 --2- 172.21.15.104:0/1496306322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e80ad8e0 0x7fa9e81f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 >> 172.21.15.104:0/1496306322 conn(0x7fa9e80b1570 msgr2=0x7fa9e80cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:54.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 shutdown_connections 2023-06-10T18:48:54.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.277+0000 7fa9eeb42700 1 -- 172.21.15.104:0/1496306322 wait complete. 2023-06-10T18:48:54.284 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:54.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config set mds_defer_session_stale false 2023-06-10T18:48:54.431 INFO:teuthology.orchestra.run.smithi114.stdout:{"success":"mds_defer_session_stale = 'false' (not observed, change may require restart) "} 2023-06-10T18:48:54.432 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_defer_session_stale = 'false' (not observed, change may require restart) " } 2023-06-10T18:48:54.432 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:54.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.547+0000 7ff7f70fb700 1 Processor -- start 2023-06-10T18:48:54.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 -- start start 2023-06-10T18:48:54.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 0x7ff7f00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 0x7ff7f00b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f00b17a0 con 0x7ff7f00cd9e0 2023-06-10T18:48:54.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f015a8c0 con 0x7ff7f00b0d90 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.548+0000 7ff7f70fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00b14c0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7effff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff7f00cd9e0 0x7ff7f00ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57706/0 (socket says 172.21.15.104:57706) 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7effff700 1 -- 172.21.15.104:0/3918117634 learned_addr learned my addr 172.21.15.104:0/3918117634 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 --2- 172.21.15.104:0/3918117634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 0x7ff7f00b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/3918117634 >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 0x7ff7f00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819724380 0 0) 0x7ff7f00b17a0 con 0x7ff7f00cd9e0 2023-06-10T18:48:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e00035b0 con 0x7ff7f00cd9e0 2023-06-10T18:48:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 -- 172.21.15.104:0/3918117634 >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 msgr2=0x7ff7f00cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262239472 0 0) 0x7ff7e00035b0 con 0x7ff7f00cd9e0 2023-06-10T18:48:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 --2- 172.21.15.104:0/3918117634 >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 0x7ff7f00cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 -- 172.21.15.104:0/3918117634 >> v1:172.21.15.104:6789/0 conn(0x7ff7f00cd9e0 legacy=0x7ff7f00ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 -- 172.21.15.104:0/3918117634 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f00cc9c0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/3918117634 >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 0x7ff7f00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:48:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7f5698700 1 --2- 172.21.15.104:0/3918117634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 0x7ff7f00b3670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009d90 tx=0x7ff7e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440e4c01fd084afc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e800e070 con 0x7ff7f00ceb00 2023-06-10T18:48:54.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e8003a20 con 0x7ff7f00ceb00 2023-06-10T18:48:54.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.549+0000 7ff7ef7fe700 1 -- 172.21.15.104:0/3918117634 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc001670 con 0x7ff7f00ceb00 2023-06-10T18:48:54.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 -- 172.21.15.104:0/3918117634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 msgr2=0x7ff7f00b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/3918117634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 0x7ff7f00b3670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009d90 tx=0x7ff7e800b330 comp rx=0 tx=0).stop 2023-06-10T18:48:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 -- 172.21.15.104:0/3918117634 shutdown_connections 2023-06-10T18:48:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/3918117634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00ceb00 0x7ff7f00b3670 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/3918117634 >> v2:172.21.15.114:3300/0 conn(0x7ff7f00b0d90 0x7ff7f00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 -- 172.21.15.104:0/3918117634 >> 172.21.15.104:0/3918117634 conn(0x7ff7f0089fd0 msgr2=0x7ff7f00d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 -- 172.21.15.104:0/3918117634 shutdown_connections 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.550+0000 7ff7f70fb700 1 -- 172.21.15.104:0/3918117634 wait complete. 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.551+0000 7ff7f70fb700 1 Processor -- start 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.551+0000 7ff7f70fb700 1 -- start start 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 0x7ff7f00cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00cd9e0 0x7ff7f00aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00af530 con 0x7ff7f00ceb00 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00af760 con 0x7ff7f00b0d90 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f70fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f009bac0 con 0x7ff7f00cd9e0 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59886/0 (socket says 172.21.15.104:59886) 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 -- 172.21.15.104:0/2463005096 learned_addr learned my addr 172.21.15.104:0/2463005096 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:54.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7effff700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00cd9e0 0x7ff7f00aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 0x7ff7f00cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 -- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00cd9e0 msgr2=0x7ff7f00aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00cd9e0 0x7ff7f00aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 -- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 msgr2=0x7ff7f00cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 0x7ff7f00cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f5698700 1 -- 172.21.15.104:0/2463005096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e40094a0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.552+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 0x7ff7f00cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:48:54.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7f5698700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00b1600 tx=0x7ff7e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8009410 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e80044e0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e8009a70 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8004890 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.553+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f009c080 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.554+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f0074050 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.554+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff7e8012430 con 0x7ff7f00ceb00 2023-06-10T18:48:54.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.555+0000 7ff7ed7fa700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 0x7ff7d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.555+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff7e808b1f0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.555+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 0x7ff7d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.555+0000 7ff7f4e97700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 0x7ff7d406d190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00cb1f0 tx=0x7ff7e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:54.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.559+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff7e8059c40 con 0x7ff7f00ceb00 2023-06-10T18:48:54.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.787+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7f00cffc0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.788+0000 7ff7ed7fa700 1 -- 172.21.15.104:0/2463005096 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7ff7e80597d0 con 0x7ff7f00ceb00 2023-06-10T18:48:54.794 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:54.794 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 msgr2=0x7ff7d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 0x7ff7d406d190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00cb1f0 tx=0x7ff7e4008040 comp rx=0 tx=0).stop 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 msgr2=0x7ff7f009b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00b1600 tx=0x7ff7e8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 shutdown_connections 2023-06-10T18:48:54.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d406ace0 0x7ff7d406d190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f00b0d90 0x7ff7f00cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f00ceb00 0x7ff7f009b210 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 --2- 172.21.15.104:0/2463005096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f00cd9e0 0x7ff7f00aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 >> 172.21.15.104:0/2463005096 conn(0x7ff7f0089fd0 msgr2=0x7ff7f0159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 shutdown_connections 2023-06-10T18:48:54.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:54.790+0000 7ff7f70fb700 1 -- 172.21.15.104:0/2463005096 wait complete. 2023-06-10T18:48:54.798 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:54.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:48:54.808 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:48:54.809 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:48:54.809 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:48:54.809 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:48:54.810 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:54.810 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:48:54.810 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:48:54.811 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:48:54.811 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:48:54.811 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:54.812 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:48:54.812 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:48:54.812 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:48:54.813 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:48:54.813 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:54.813 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:48:54.814 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:48:54.814 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:48:54.814 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:48:54.815 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:48:54.815 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:48:54.815 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:48:54.815 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:48:54.816 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:48:54.816 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:48:54.816 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:48:54.899 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.128563.asok 2023-06-10T18:48:55.081 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.128563.asok 2023-06-10T18:48:55.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:55.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.128563.asok mds_sessions 2023-06-10T18:48:55.235 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:48:55.236 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 6864, 2023-06-10T18:48:55.236 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:48:55.236 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:48:55.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:48:55.236 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 6864 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2562980824 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:55.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.6864 v1:192.168.0.1:0/2562980824", 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/2562980824", 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:48:55.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6834", 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1280497888 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:48:55.239 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6836", 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1280497888 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:49:52.175929+0000", 2023-06-10T18:48:55.240 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:48:52.175929+0000", 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 1, 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 33 2023-06-10T18:48:55.241 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:48:55.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:55.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.358+0000 7f959298d700 1 Processor -- start 2023-06-10T18:48:55.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.358+0000 7f959298d700 1 -- start start 2023-06-10T18:48:55.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f958c1567f0 0x7f958c0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 0x7f958c0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958c070b00 con 0x7f958c1567f0 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958c0cb2d0 con 0x7f958c0701a0 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f959298d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c15cca0 con 0x7f958c15c7a0 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f958c1567f0 0x7f958c0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f958c1567f0 0x7f958c0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59890/0 (socket says 172.21.15.104:59890) 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 -- 172.21.15.104:0/2355487875 learned_addr learned my addr 172.21.15.104:0/2355487875 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:55.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958b7fe700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f9590f2a700 1 --2- 172.21.15.104:0/2355487875 >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 0x7f958c0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 -- 172.21.15.104:0/2355487875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 msgr2=0x7f958c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 -- 172.21.15.104:0/2355487875 >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 msgr2=0x7f958c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 --2- 172.21.15.104:0/2355487875 >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 0x7f958c0705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 -- 172.21.15.104:0/2355487875 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c15cf80 con 0x7f958c1567f0 2023-06-10T18:48:55.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f9590f2a700 1 --2- 172.21.15.104:0/2355487875 >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 0x7f958c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958bfff700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c0cad90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f958c0c77a0 tx=0x7f9580009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8a0ba423df59a3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.359+0000 7f958affd700 1 -- 172.21.15.104:0/2355487875 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000b870 con 0x7f958c1567f0 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f958affd700 1 -- 172.21.15.104:0/2355487875 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000b9d0 con 0x7f958c1567f0 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f958affd700 1 -- 172.21.15.104:0/2355487875 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000bd60 con 0x7f958c1567f0 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 -- 172.21.15.104:0/2355487875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 msgr2=0x7f958c0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c0cad90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f958c0c77a0 tx=0x7f9580009bd0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 -- 172.21.15.104:0/2355487875 shutdown_connections 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c0cad90 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 --2- 172.21.15.104:0/2355487875 >> v2:172.21.15.114:3300/0 conn(0x7f958c0701a0 0x7f958c0705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 --2- 172.21.15.104:0/2355487875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 -- 172.21.15.104:0/2355487875 >> 172.21.15.104:0/2355487875 conn(0x7f958c06e370 msgr2=0x7f958c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 -- 172.21.15.104:0/2355487875 shutdown_connections 2023-06-10T18:48:55.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.360+0000 7f959298d700 1 -- 172.21.15.104:0/2355487875 wait complete. 2023-06-10T18:48:55.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.361+0000 7f959298d700 1 Processor -- start 2023-06-10T18:48:55.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 -- start start 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f958c0701a0 0x7f958c0c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c154b60 con 0x7f958c1567f0 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c152240 con 0x7f958c0701a0 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f959298d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c151840 con 0x7f958c15c7a0 2023-06-10T18:48:55.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f958b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59898/0 (socket says 172.21.15.104:59898) 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.362+0000 7f958b7fe700 1 -- 172.21.15.104:0/3111818030 learned_addr learned my addr 172.21.15.104:0/3111818030 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f9590f2a700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958bfff700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f958c0701a0 0x7f958c0c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 -- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 msgr2=0x7f958c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 -- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f958c0701a0 msgr2=0x7f958c0c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f958c0701a0 0x7f958c0c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 -- 172.21.15.104:0/3111818030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009550 con 0x7f958c1567f0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f958b7fe700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e810 tx=0x7f957c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c013610 con 0x7f958c1567f0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c013770 con 0x7f958c1567f0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c0c9910 con 0x7f958c1567f0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c013b00 con 0x7f958c1567f0 2023-06-10T18:48:55.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.363+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c151bf0 con 0x7f958c1567f0 2023-06-10T18:48:55.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.364+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c074050 con 0x7f958c1567f0 2023-06-10T18:48:55.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.364+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f957c013c60 con 0x7f958c1567f0 2023-06-10T18:48:55.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.365+0000 7f9588ff9700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 0x7f957806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.365+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f957c089220 con 0x7f958c1567f0 2023-06-10T18:48:55.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.365+0000 7f958bfff700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 0x7f957806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.365+0000 7f958bfff700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 0x7f957806d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9580009c10 tx=0x7f9580008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.369+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f957c054150 con 0x7f958c1567f0 2023-06-10T18:48:55.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.598+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f958c0b66c0 con 0x7f958c1567f0 2023-06-10T18:48:55.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.599+0000 7f9588ff9700 1 -- 172.21.15.104:0/3111818030 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f957c057770 con 0x7f958c1567f0 2023-06-10T18:48:55.605 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:55.605 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:55.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 msgr2=0x7f957806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 0x7f957806d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9580009c10 tx=0x7f9580008030 comp rx=0 tx=0).stop 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 msgr2=0x7f958c154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e810 tx=0x7f957c00ebd0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 shutdown_connections 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f957806ace0 0x7f957806d190 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f958c0701a0 0x7f958c0c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f958c15c7a0 0x7f958c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 --2- 172.21.15.104:0/3111818030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f958c1567f0 0x7f958c154620 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 >> 172.21.15.104:0/3111818030 conn(0x7f958c06e370 msgr2=0x7f958c155c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:55.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 shutdown_connections 2023-06-10T18:48:55.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.601+0000 7f959298d700 1 -- 172.21.15.104:0/3111818030 wait complete. 2023-06-10T18:48:55.608 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:55.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session config 6864 timeout 120 2023-06-10T18:48:55.772 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2023-06-10T18:48:55.773 INFO:tasks.cephfs.mount:Killing connection on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:48:55.773 INFO:tasks.cephfs.mount:Suspending the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:48:55.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:48:55.774 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 down 2023-06-10T18:48:55.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:48:55.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 Processor -- start 2023-06-10T18:48:55.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 -- start start 2023-06-10T18:48:55.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88140ce510 0x7f88140ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881415c050 con 0x7f881415bc80 2023-06-10T18:48:55.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140c9970 con 0x7f88140c8c40 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.962+0000 7f881adc8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88140ceed0 con 0x7f88140ce510 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8818b64700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f881415bc80 0x7f8814070310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57732/0 (socket says 172.21.15.104:57732) 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8818b64700 1 -- 172.21.15.104:0/3976458665 learned_addr learned my addr 172.21.15.104:0/3976458665 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 --2- 172.21.15.104:0/3976458665 >> v2:172.21.15.114:3300/0 conn(0x7f88140ce510 0x7f88140ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864070086 0 0) 0x7f881415c050 con 0x7f881415bc80 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88040046c0 con 0x7f881415bc80 2023-06-10T18:48:55.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269602379 0 0) 0x7f88040046c0 con 0x7f881415bc80 2023-06-10T18:48:55.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 >> v2:172.21.15.114:3300/0 conn(0x7f88140ce510 msgr2=0x7f88140ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:55.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 --2- 172.21.15.104:0/3976458665 >> v2:172.21.15.114:3300/0 conn(0x7f88140ce510 0x7f88140ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 >> v1:172.21.15.114:6789/0 conn(0x7f88140c8c40 legacy=0x7f88140c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140ccb70 con 0x7f881415bc80 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8808001670 con 0x7f881415bc80 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 >> v1:172.21.15.104:6789/0 conn(0x7f881415bc80 legacy=0x7f8814070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804004b10 0x7f8804004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 0x7f8804005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804006310 0x7f880400cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140ceed0 con 0x7f8804005960 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880400d170 con 0x7f8804006310 2023-06-10T18:48:55.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880400d330 con 0x7f8804004b10 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 0x7f8804005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8818b64700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804004b10 0x7f8804004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8819365700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804006310 0x7f880400cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 -- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804004b10 msgr2=0x7f8804004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804004b10 0x7f8804004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 -- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804006310 msgr2=0x7f880400cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:48:55.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.963+0000 7f8813fff700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804006310 0x7f880400cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f8819365700 1 -- 172.21.15.104:0/3976458665 reap_dead start 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f8813fff700 1 -- 172.21.15.104:0/3976458665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88040046c0 con 0x7f8804005960 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f8813fff700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 0x7f8804005dd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8800009b60 tx=0x7f8804006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13d8bcc5a5723ac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880000e6c0 con 0x7f8804005960 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880000e820 con 0x7f8804005960 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f88137fe700 1 -- 172.21.15.104:0/3976458665 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880000eb90 con 0x7f8804005960 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 -- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 msgr2=0x7f8804005dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 0x7f8804005dd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8800009b60 tx=0x7f8804006990 comp rx=0 tx=0).stop 2023-06-10T18:48:55.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 -- 172.21.15.104:0/3976458665 shutdown_connections 2023-06-10T18:48:55.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 --2- 172.21.15.104:0/3976458665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8804005960 0x7f8804005dd0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 -- 172.21.15.104:0/3976458665 >> 172.21.15.104:0/3976458665 conn(0x7f881406ab00 msgr2=0x7f88140b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:55.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 -- 172.21.15.104:0/3976458665 shutdown_connections 2023-06-10T18:48:55.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.964+0000 7f881adc8700 1 -- 172.21.15.104:0/3976458665 wait complete. 2023-06-10T18:48:55.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.966+0000 7f881adc8700 1 Processor -- start 2023-06-10T18:48:55.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.966+0000 7f881adc8700 1 -- start start 2023-06-10T18:48:55.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.966+0000 7f881adc8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804005960 0x7f88140accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.966+0000 7f881adc8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88140c3d20 0x7f88140af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140c41a0 con 0x7f88140c2b90 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140ad1f0 con 0x7f88140c3d20 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88140b0370 con 0x7f8804005960 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59918/0 (socket says 172.21.15.104:59918) 2023-06-10T18:48:55.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 -- 172.21.15.104:0/3592464286 learned_addr learned my addr 172.21.15.104:0/3592464286 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8818b64700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804005960 0x7f88140accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8819365700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88140c3d20 0x7f88140af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 -- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804005960 msgr2=0x7f88140accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804005960 0x7f88140accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 -- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88140c3d20 msgr2=0x7f88140af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88140c3d20 0x7f88140af6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:55.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 -- 172.21.15.104:0/3592464286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140b0650 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f8813fff700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f880400d140 tx=0x7f88000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880000e6c0 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800003ad0 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800003e80 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88140c20b0 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.967+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88140c2670 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.968+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88140c4120 con 0x7f88140c2b90 2023-06-10T18:48:55.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.969+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8800013470 con 0x7f88140c2b90 2023-06-10T18:48:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.969+0000 7f88117fa700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 0x7f87f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:48:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.969+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f880008ad70 con 0x7f88140c2b90 2023-06-10T18:48:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.969+0000 7f8818b64700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 0x7f87f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:48:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.970+0000 7f8818b64700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 0x7f87f806d190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88140cd430 tx=0x7f8808000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:48:55.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:55.974+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8800055e50 con 0x7f88140c2b90 2023-06-10T18:48:56.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.202+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88140c5450 con 0x7f88140c2b90 2023-06-10T18:48:56.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.202+0000 7f88117fa700 1 -- 172.21.15.104:0/3592464286 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f8800017090 con 0x7f88140c2b90 2023-06-10T18:48:56.208 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:48:56.208 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:48:56.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 msgr2=0x7f87f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:56.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 0x7f87f806d190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f88140cd430 tx=0x7f8808000ed0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 msgr2=0x7f88140c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:48:56.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f880400d140 tx=0x7f88000047c0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 shutdown_connections 2023-06-10T18:48:56.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87f806ace0 0x7f87f806d190 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88140c3d20 0x7f88140af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8804005960 0x7f88140accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 --2- 172.21.15.104:0/3592464286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88140c2b90 0x7f88140c37e0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:48:56.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 >> 172.21.15.104:0/3592464286 conn(0x7f881406ab00 msgr2=0x7f8814071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:48:56.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 shutdown_connections 2023-06-10T18:48:56.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:48:56.204+0000 7f881adc8700 1 -- 172.21.15.104:0/3592464286 wait complete. 2023-06-10T18:48:56.213 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:48:56.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-06-10T18:48:56.363 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":5660,"entity":{"name":{"type":"client","num":5660},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":2960984225}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":3.3379260660000001,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.99232200165472695,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5660 192.168.0.1:0/2960984225","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"110297","root":"/"}},{"id":6864,"entity":{"name":{"type":"client","num":6864},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":2562980824}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":1,"uptime":5.1728854230000003,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.988123803127964,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6864 192.168.0.1:0/2562980824","completed_requests":[{"tid":2,"created_ino":"0x0"}],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"0","hostname":"smithi104","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"128563","root":"/","timeout":"120"}}] 2023-06-10T18:48:56.364 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "110297", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 2960984225, "type": "any" }, "name": { "num": 5660, "type": "client" } }, "id": 5660, "inst": "client.5660 192.168.0.1:0/2960984225", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.992322001654727 }, "state": "open", "uptime": 3.337926066 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "0", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "128563", "root": "/", "timeout": "120" }, "completed_requests": [ { "created_ino": "0x0", "tid": 2 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 2562980824, "type": "any" }, "name": { "num": 6864, "type": "client" } }, "id": 6864, "inst": "client.6864 192.168.0.1:0/2562980824", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 1, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 1, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.988123803127964 }, "state": "open", "uptime": 5.172885423 } ] 2023-06-10T18:49:08.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:49:08.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:49:38.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:49:38.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:50:08.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:50:08.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:50:26.366 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:50:26.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.494+0000 7fd6994bd700 1 Processor -- start 2023-06-10T18:50:26.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 -- start start 2023-06-10T18:50:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd694070130 0x7fd6940c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6940cee20 con 0x7fd6940705a0 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6940caa90 con 0x7fd694070130 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.495+0000 7fd6994bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd694071fe0 con 0x7fd6940ce510 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd692ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd694070130 0x7fd6940c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd694070130 0x7fd6940c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:47536/0 (socket says 172.21.15.104:47536) 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd692ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:51186/0 (socket says 172.21.15.104:51186) 2023-06-10T18:50:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 -- 172.21.15.104:0/2668849357 learned_addr learned my addr 172.21.15.104:0/2668849357 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:50:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6937fe700 1 --1- 172.21.15.104:0/2668849357 >> v1:172.21.15.114:6789/0 conn(0x7fd6940705a0 0x7fd6940cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:35478/0 (socket says 172.21.15.104:35478) 2023-06-10T18:50:26.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 -- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 msgr2=0x7fd6940ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:50:26.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 -- 172.21.15.104:0/2668849357 >> v1:172.21.15.114:6789/0 conn(0x7fd6940705a0 legacy=0x7fd6940cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 -- 172.21.15.104:0/2668849357 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940722c0 con 0x7fd694070130 2023-06-10T18:50:26.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd692ffd700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd6927fc700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd6940c8bd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd6840098f0 tx=0x7fd684009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=288ef0785e5fd93c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd691ffb700 1 -- 172.21.15.104:0/2668849357 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684011830 con 0x7fd694070130 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd691ffb700 1 -- 172.21.15.104:0/2668849357 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684011990 con 0x7fd694070130 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.496+0000 7fd691ffb700 1 -- 172.21.15.104:0/2668849357 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684011d20 con 0x7fd694070130 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 -- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 msgr2=0x7fd6940c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd6940c8bd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd6840098f0 tx=0x7fd684009c00 comp rx=0 tx=0).stop 2023-06-10T18:50:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 -- 172.21.15.104:0/2668849357 shutdown_connections 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940ce510 0x7fd6940ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 --2- 172.21.15.104:0/2668849357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd6940c8bd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 -- 172.21.15.104:0/2668849357 >> 172.21.15.104:0/2668849357 conn(0x7fd69406ab00 msgr2=0x7fd6940b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 -- 172.21.15.104:0/2668849357 shutdown_connections 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.497+0000 7fd6994bd700 1 -- 172.21.15.104:0/2668849357 wait complete. 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.498+0000 7fd6994bd700 1 Processor -- start 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 -- start start 2023-06-10T18:50:26.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd69414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940705a0 0x7fd6940c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6941560e0 con 0x7fd6940ce510 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694155460 con 0x7fd694070130 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6994bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd694155740 con 0x7fd6940705a0 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60042/0 (socket says 172.21.15.104:60042) 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.499+0000 7fd6937fe700 1 -- 172.21.15.104:0/41180775 learned_addr learned my addr 172.21.15.104:0/41180775 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6927fc700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940705a0 0x7fd6940c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd692ffd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd69414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 -- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940705a0 msgr2=0x7fd6940c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940705a0 0x7fd6940c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 -- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 msgr2=0x7fd69414dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd69414dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 -- 172.21.15.104:0/41180775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880094a0 con 0x7fd6940ce510 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6937fe700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00cbd0 tx=0x7fd68c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c00aaa0 con 0x7fd6940ce510 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c00ac00 con 0x7fd6940ce510 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd684009550 con 0x7fd6940ce510 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c007070 con 0x7fd6940ce510 2023-06-10T18:50:26.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.500+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd694155d00 con 0x7fd6940ce510 2023-06-10T18:50:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.501+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd694074050 con 0x7fd6940ce510 2023-06-10T18:50:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.501+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd68c0071d0 con 0x7fd6940ce510 2023-06-10T18:50:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.502+0000 7fd67b7fe700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 0x7fd68006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:50:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.502+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd68c089230 con 0x7fd6940ce510 2023-06-10T18:50:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.502+0000 7fd692ffd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 0x7fd68006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:50:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.502+0000 7fd692ffd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 0x7fd68006d190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd688009d70 tx=0x7fd688009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:50:26.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.506+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd68c054210 con 0x7fd6940ce510 2023-06-10T18:50:26.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.733+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6940cec00 con 0x7fd6940ce510 2023-06-10T18:50:26.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.733+0000 7fd67b7fe700 1 -- 172.21.15.104:0/41180775 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fd68c017090 con 0x7fd6940ce510 2023-06-10T18:50:26.739 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:50:26.739 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:50:26.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 msgr2=0x7fd68006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 0x7fd68006d190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd688009d70 tx=0x7fd688009210 comp rx=0 tx=0).stop 2023-06-10T18:50:26.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 msgr2=0x7fd6940c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:50:26.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00cbd0 tx=0x7fd68c00cee0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 shutdown_connections 2023-06-10T18:50:26.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd68006ace0 0x7fd68006d190 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd694070130 0x7fd69414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6940ce510 0x7fd6940c9fa0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 --2- 172.21.15.104:0/41180775 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6940705a0 0x7fd6940c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:50:26.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 >> 172.21.15.104:0/41180775 conn(0x7fd69406ab00 msgr2=0x7fd694157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:50:26.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 shutdown_connections 2023-06-10T18:50:26.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:50:26.735+0000 7fd6994bd700 1 -- 172.21.15.104:0/41180775 wait complete. 2023-06-10T18:50:26.743 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:50:26.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-06-10T18:50:26.906 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":5660,"entity":{"name":{"type":"client","num":5660},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":2960984225}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":93.880894702999996,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.80500539153535577,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5660 192.168.0.1:0/2960984225","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"110297","root":"/"}},{"id":6864,"entity":{"name":{"type":"client","num":6864},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":2562980824}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":95.715854059999998,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.80159966996197041,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6864 192.168.0.1:0/2562980824","completed_requests":[{"tid":2,"created_ino":"0x0"}],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"0","hostname":"smithi104","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"128563","root":"/","timeout":"120"}}] 2023-06-10T18:50:26.907 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "110297", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 2960984225, "type": "any" }, "name": { "num": 5660, "type": "client" } }, "id": 5660, "inst": "client.5660 192.168.0.1:0/2960984225", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.8050053915353558 }, "state": "open", "uptime": 93.880894703 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "0", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "128563", "root": "/", "timeout": "120" }, "completed_requests": [ { "created_ino": "0x0", "tid": 2 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 2562980824, "type": "any" }, "name": { "num": 6864, "type": "client" } }, "id": 6864, "inst": "client.6864 192.168.0.1:0/2562980824", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 1, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.8015996699619704 }, "state": "open", "uptime": 95.71585406 } ] 2023-06-10T18:50:38.483 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:50:38.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:51:08.548 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:51:08.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:51:26.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.035+0000 7fabf5a1c700 1 Processor -- start 2023-06-10T18:51:27.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.035+0000 7fabf5a1c700 1 -- start start 2023-06-10T18:51:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabf5a1c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006e0e0 0x7fabf006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabf5a1c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf006e4b0 0x7fabf00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabf5a1c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf006d290 con 0x7fabf00ca100 2023-06-10T18:51:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabf5a1c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf0071640 con 0x7fabf006e4b0 2023-06-10T18:51:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabf5a1c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00aef00 con 0x7fabf006e0e0 2023-06-10T18:51:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf006e4b0 0x7fabf00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabee7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf006e4b0 0x7fabf00c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60124/0 (socket says 172.21.15.104:60124) 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.036+0000 7fabee7fc700 1 -- 172.21.15.104:0/2905414281 learned_addr learned my addr 172.21.15.104:0/2905414281 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabeeffd700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006e0e0 0x7fabf006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabee7fc700 1 -- 172.21.15.104:0/2905414281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006e0e0 msgr2=0x7fabf006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabee7fc700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006e0e0 0x7fabf006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabee7fc700 1 -- 172.21.15.104:0/2905414281 >> v1:172.21.15.114:6789/0 conn(0x7fabf00ca100 legacy=0x7fabf00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabee7fc700 1 -- 172.21.15.104:0/2905414281 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf006d4c0 con 0x7fabf006e4b0 2023-06-10T18:51:27.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabee7fc700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e4b0 0x7fabf00c9bc0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fabe00098f0 tx=0x7fabe0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611006c04217df4e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabedffb700 1 -- 172.21.15.104:0/2905414281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe000b8a0 con 0x7fabf006e4b0 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabedffb700 1 -- 172.21.15.104:0/2905414281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe000ba00 con 0x7fabf006e4b0 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabedffb700 1 -- 172.21.15.104:0/2905414281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe000bd90 con 0x7fabf006e4b0 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabf5a1c700 1 -- 172.21.15.104:0/2905414281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e4b0 msgr2=0x7fabf00c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.037+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e4b0 0x7fabf00c9bc0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fabe00098f0 tx=0x7fabe0009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 -- 172.21.15.104:0/2905414281 shutdown_connections 2023-06-10T18:51:27.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006e0e0 0x7fabf006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/2905414281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e4b0 0x7fabf00c9bc0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 -- 172.21.15.104:0/2905414281 >> 172.21.15.104:0/2905414281 conn(0x7fabf006b8a0 msgr2=0x7fabf00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 -- 172.21.15.104:0/2905414281 shutdown_connections 2023-06-10T18:51:27.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.038+0000 7fabf5a1c700 1 -- 172.21.15.104:0/2905414281 wait complete. 2023-06-10T18:51:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.039+0000 7fabf5a1c700 1 Processor -- start 2023-06-10T18:51:27.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.039+0000 7fabf5a1c700 1 -- start start 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 0x7fabf00cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf00ca100 0x7fabf0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00c9550 con 0x7fabf006e0e0 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00b0260 con 0x7fabf006e4b0 2023-06-10T18:51:27.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00c7c10 con 0x7fabf00ca100 2023-06-10T18:51:27.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60128/0 (socket says 172.21.15.104:60128) 2023-06-10T18:51:27.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 -- 172.21.15.104:0/3066461116 learned_addr learned my addr 172.21.15.104:0/3066461116 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:27.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabef7fe700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf00ca100 0x7fabf0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabee7fc700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 0x7fabf00cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 -- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf00ca100 msgr2=0x7fabf0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf00ca100 0x7fabf0155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 -- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 msgr2=0x7fabf00cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 0x7fabf00cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 -- 172.21.15.104:0/3066461116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0009550 con 0x7fabf006e0e0 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabee7fc700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 0x7fabf00cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabeeffd700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fabe400d800 tx=0x7fabe400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40125f0 con 0x7fabf006e0e0 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4012750 con 0x7fabf006e0e0 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf00c7ef0 con 0x7fabf006e0e0 2023-06-10T18:51:27.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4012b00 con 0x7fabf006e0e0 2023-06-10T18:51:27.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.040+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf00c84b0 con 0x7fabf006e0e0 2023-06-10T18:51:27.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.041+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf0074050 con 0x7fabf006e0e0 2023-06-10T18:51:27.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.042+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fabe4012c60 con 0x7fabf006e0e0 2023-06-10T18:51:27.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.042+0000 7fabd7fff700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 0x7fabdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.042+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fabe4089810 con 0x7fabf006e0e0 2023-06-10T18:51:27.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.042+0000 7fabee7fc700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 0x7fabdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.043+0000 7fabee7fc700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 0x7fabdc06d190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fabe0000c00 tx=0x7fabe0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.047+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fabe4054690 con 0x7fabf006e0e0 2023-06-10T18:51:27.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.273+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf00b03a0 con 0x7fabf006e0e0 2023-06-10T18:51:27.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.273+0000 7fabd7fff700 1 -- 172.21.15.104:0/3066461116 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fabe4057cb0 con 0x7fabf006e0e0 2023-06-10T18:51:27.279 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:27.279 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:51:27.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 msgr2=0x7fabdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 0x7fabdc06d190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fabe0000c00 tx=0x7fabe0008030 comp rx=0 tx=0).stop 2023-06-10T18:51:27.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 msgr2=0x7fabf01f9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fabe400d800 tx=0x7fabe400dbc0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 shutdown_connections 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf00ca100 0x7fabf0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006e0e0 0x7fabf01f9e00 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabdc06ace0 0x7fabdc06d190 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 --2- 172.21.15.104:0/3066461116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf006e4b0 0x7fabf00cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 >> 172.21.15.104:0/3066461116 conn(0x7fabf006b8a0 msgr2=0x7fabf00b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 shutdown_connections 2023-06-10T18:51:27.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.275+0000 7fabf5a1c700 1 -- 172.21.15.104:0/3066461116 wait complete. 2023-06-10T18:51:27.283 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:51:27.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2023-06-10T18:51:27.446 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":5660,"entity":{"name":{"type":"client","num":5660},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":2960984225}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":154.42053189800001,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.69992466804669851,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5660 192.168.0.1:0/2960984225","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"110297","root":"/"}}] 2023-06-10T18:51:27.447 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "110297", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 2960984225, "type": "any" }, "name": { "num": 5660, "type": "client" } }, "id": 5660, "inst": "client.5660 192.168.0.1:0/2960984225", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.6999246680466985 }, "state": "open", "uptime": 154.420531898 } ] 2023-06-10T18:51:27.447 INFO:tasks.cephfs.mount:Cleaning up killed connection on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:51:27.448 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:51:27.449 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:27.466 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.620+0000 7f3571299700 1 Processor -- start 2023-06-10T18:51:27.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f3571299700 1 -- start start 2023-06-10T18:51:27.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f3571299700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f356c06b160 0x7f356c06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f3571299700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0ce2f0 con 0x7f356c15c040 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f3571299700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0cfa50 con 0x7f356c06c9b0 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f3571299700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0cfd30 con 0x7f356c06b160 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f356b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f356c06b160 0x7f356c06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f356affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f356c15c040 0x7f356c1606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57966/0 (socket says 172.21.15.104:57966) 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f356b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f356c06b160 0x7f356c06ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60142/0 (socket says 172.21.15.104:60142) 2023-06-10T18:51:27.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.621+0000 7f356affd700 1 -- 172.21.15.104:0/1423744076 learned_addr learned my addr 172.21.15.104:0/1423744076 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:27.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f356b7fe700 1 -- 172.21.15.104:0/1423744076 >> v1:172.21.15.114:6789/0 conn(0x7f356c06c9b0 legacy=0x7f356c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f356b7fe700 1 -- 172.21.15.104:0/1423744076 >> v1:172.21.15.104:6789/0 conn(0x7f356c15c040 legacy=0x7f356c1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f356b7fe700 1 -- 172.21.15.104:0/1423744076 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0ce590 con 0x7f356c06b160 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f3569ffb700 1 -- 172.21.15.104:0/1423744076 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461436237 0 0) 0x7f356c0ce2f0 con 0x7f356c15c040 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f356b7fe700 1 --2- 172.21.15.104:0/1423744076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06b160 0x7f356c06ce20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f3564009d90 tx=0x7f356400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab4c3d5d879da27 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f3569ffb700 1 -- 172.21.15.104:0/1423744076 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400e070 con 0x7f356c06b160 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f3569ffb700 1 -- 172.21.15.104:0/1423744076 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356400bbd0 con 0x7f356c06b160 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.622+0000 7f3569ffb700 1 -- 172.21.15.104:0/1423744076 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564012430 con 0x7f356c06b160 2023-06-10T18:51:27.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 -- 172.21.15.104:0/1423744076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06b160 msgr2=0x7f356c06ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 --2- 172.21.15.104:0/1423744076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06b160 0x7f356c06ce20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f3564009d90 tx=0x7f356400b330 comp rx=0 tx=0).stop 2023-06-10T18:51:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 -- 172.21.15.104:0/1423744076 shutdown_connections 2023-06-10T18:51:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 --2- 172.21.15.104:0/1423744076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06b160 0x7f356c06ce20 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 -- 172.21.15.104:0/1423744076 >> 172.21.15.104:0/1423744076 conn(0x7f356c0b1570 msgr2=0x7f356c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:27.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 -- 172.21.15.104:0/1423744076 shutdown_connections 2023-06-10T18:51:27.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.623+0000 7f3571299700 1 -- 172.21.15.104:0/1423744076 wait complete. 2023-06-10T18:51:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.624+0000 7f3571299700 1 Processor -- start 2023-06-10T18:51:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.624+0000 7f3571299700 1 -- start start 2023-06-10T18:51:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f356c06b160 0x7f356c154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356c15c040 0x7f356c155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c1e87f0 con 0x7f356c06c9b0 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c154ca0 con 0x7f356c15c040 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c1e7b70 con 0x7f356c06b160 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60146/0 (socket says 172.21.15.104:60146) 2023-06-10T18:51:27.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 -- 172.21.15.104:0/2056411723 learned_addr learned my addr 172.21.15.104:0/2056411723 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356affd700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f356c06b160 0x7f356c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356b7fe700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356c15c040 0x7f356c155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 -- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f356c06b160 msgr2=0x7f356c154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f356c06b160 0x7f356c154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 -- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356c15c040 msgr2=0x7f356c155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356c15c040 0x7f356c155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 -- 172.21.15.104:0/2056411723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3564009a70 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f356a7fc700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c4e0 tx=0x7f355c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0052d0 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c005430 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c1e7e50 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.625+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0038c0 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.626+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c1e8410 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.627+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c1e8930 con 0x7f356c06c9b0 2023-06-10T18:51:27.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.627+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f355c003a20 con 0x7f356c06c9b0 2023-06-10T18:51:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.627+0000 7f35537fe700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 0x7f355806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.627+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f355c088a10 con 0x7f356c06c9b0 2023-06-10T18:51:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.627+0000 7f356affd700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 0x7f355806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.628+0000 7f356affd700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 0x7f355806d190 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3560007c40 tx=0x7f3560007b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.632+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f356c1e8930 con 0x7f356c06c9b0 2023-06-10T18:51:27.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.819+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f356c1e8930 con 0x7f356c06c9b0 2023-06-10T18:51:27.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.821+0000 7f35537fe700 1 -- 172.21.15.104:0/2056411723 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18760 (secure 0 0 0) 0x7f356c1e8930 con 0x7f356c06c9b0 2023-06-10T18:51:27.827 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:27.827 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":71,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:50:54.764730+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.a","mappings":[{"from":6,"to":0}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/2562980824":"2023-06-10T19:50:53.863403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:27.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 msgr2=0x7f355806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 0x7f355806d190 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3560007c40 tx=0x7f3560007b90 comp rx=0 tx=0).stop 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 msgr2=0x7f356c154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c4e0 tx=0x7f355c00c7f0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 shutdown_connections 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355806ace0 0x7f355806d190 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356c15c040 0x7f356c155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f356c06b160 0x7f356c154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 --2- 172.21.15.104:0/2056411723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f356c06c9b0 0x7f356c154e30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 >> 172.21.15.104:0/2056411723 conn(0x7f356c0b1570 msgr2=0x7f356c0afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:27.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 shutdown_connections 2023-06-10T18:51:27.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.823+0000 7f3571299700 1 -- 172.21.15.104:0/2056411723 wait complete. 2023-06-10T18:51:27.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:48:42.072415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:48:43.111638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:51:27.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:51:27.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.961+0000 7f39bd01d700 1 Processor -- start 2023-06-10T18:51:27.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.961+0000 7f39bd01d700 1 -- start start 2023-06-10T18:51:27.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39bd01d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 0x7f39b80ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39bd01d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f39b80ad760 0x7f39b80adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39bd01d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b815cb00 con 0x7f39b815c730 2023-06-10T18:51:27.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39bd01d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80acd90 con 0x7f39b80ad760 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39bd01d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80ac460 con 0x7f39b80cd8f0 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39b759e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f39b80ad760 0x7f39b80adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39b759e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f39b80ad760 0x7f39b80adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60156/0 (socket says 172.21.15.104:60156) 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39b6d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f39b815c730 0x7f39b80cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57980/0 (socket says 172.21.15.104:57980) 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39b759e700 1 -- 172.21.15.104:0/4145164898 learned_addr learned my addr 172.21.15.104:0/4145164898 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.962+0000 7f39b659c700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 0x7f39b80ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b759e700 1 -- 172.21.15.104:0/4145164898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 msgr2=0x7f39b80ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:27.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b759e700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 0x7f39b80ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b759e700 1 -- 172.21.15.104:0/4145164898 >> v1:172.21.15.104:6789/0 conn(0x7f39b815c730 legacy=0x7f39b80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b759e700 1 -- 172.21.15.104:0/4145164898 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80ac740 con 0x7f39b80ad760 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b5d9b700 1 -- 172.21.15.104:0/4145164898 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099562982 0 0) 0x7f39ac0011d0 con 0x7f39b815c730 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b659c700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 0x7f39b80ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b759e700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80adc00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8a56adaa5887a3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b5d9b700 1 -- 172.21.15.104:0/4145164898 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000e070 con 0x7f39b80ad760 2023-06-10T18:51:27.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b5d9b700 1 -- 172.21.15.104:0/4145164898 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b000b9c0 con 0x7f39b80ad760 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39b5d9b700 1 -- 172.21.15.104:0/4145164898 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000bd70 con 0x7f39b80ad760 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 -- 172.21.15.104:0/4145164898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 msgr2=0x7f39b80adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80adc00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009e50 tx=0x7f39b000b330 comp rx=0 tx=0).stop 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 -- 172.21.15.104:0/4145164898 shutdown_connections 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80adc00 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 --2- 172.21.15.104:0/4145164898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b80cd8f0 0x7f39b80ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.963+0000 7f39bd01d700 1 -- 172.21.15.104:0/4145164898 >> 172.21.15.104:0/4145164898 conn(0x7f39b806b8a0 msgr2=0x7f39b80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:27.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.964+0000 7f39bd01d700 1 -- 172.21.15.104:0/4145164898 shutdown_connections 2023-06-10T18:51:27.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.964+0000 7f39bd01d700 1 -- 172.21.15.104:0/4145164898 wait complete. 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.965+0000 7f39bd01d700 1 Processor -- start 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.965+0000 7f39bd01d700 1 -- start start 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 0x7f39b80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b815c730 0x7f39b80b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80c5f10 con 0x7f39b80ad760 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81432e0 con 0x7f39b80cd8f0 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80c89e0 con 0x7f39b815c730 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60158/0 (socket says 172.21.15.104:60158) 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 -- 172.21.15.104:0/2229122557 learned_addr learned my addr 172.21.15.104:0/2229122557 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b759e700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b815c730 0x7f39b80b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b659c700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 0x7f39b80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 -- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b815c730 msgr2=0x7f39b80b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b815c730 0x7f39b80b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 -- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 msgr2=0x7f39b80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 0x7f39b80b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 -- 172.21.15.104:0/2229122557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0009b30 con 0x7f39b80ad760 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b659c700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 0x7f39b80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39b6d9d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c4d0 tx=0x7f39ac00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0074e0 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac007640 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b806f150 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.966+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac005a70 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.967+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80c8fa0 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.967+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8074050 con 0x7f39b80ad760 2023-06-10T18:51:27.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.968+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f39ac005bd0 con 0x7f39b80ad760 2023-06-10T18:51:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.968+0000 7f399f7fe700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 0x7f39a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.968+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f39ac0888e0 con 0x7f39b80ad760 2023-06-10T18:51:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.968+0000 7f39b659c700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 0x7f39a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.969+0000 7f39b659c700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 0x7f39a406d190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f39b815b6f0 tx=0x7f39a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:27.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:27.973+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f39ac0537e0 con 0x7f39b80ad760 2023-06-10T18:51:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.157+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39b815cb00 con 0x7f39b80ad760 2023-06-10T18:51:28.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.158+0000 7f399f7fe700 1 -- 172.21.15.104:0/2229122557 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39ac017020 con 0x7f39b80ad760 2023-06-10T18:51:28.163 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:51:28.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 msgr2=0x7f39a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 0x7f39a406d190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f39b815b6f0 tx=0x7f39a8008040 comp rx=0 tx=0).stop 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 msgr2=0x7f39b80c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c4d0 tx=0x7f39ac00c7e0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 shutdown_connections 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39b815c730 0x7f39b80b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39b80ad760 0x7f39b80c6e10 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f39a406ace0 0x7f39a406d190 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 --2- 172.21.15.104:0/2229122557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80cd8f0 0x7f39b80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 >> 172.21.15.104:0/2229122557 conn(0x7f39b806b8a0 msgr2=0x7f39b80cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 shutdown_connections 2023-06-10T18:51:28.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.160+0000 7f39bd01d700 1 -- 172.21.15.104:0/2229122557 wait complete. 2023-06-10T18:51:28.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:51:28.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.293+0000 7f535019e700 1 Processor -- start 2023-06-10T18:51:28.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f535019e700 1 -- start start 2023-06-10T18:51:28.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f535019e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5348157b30 0x7f53480ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f535019e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53480ad3c0 con 0x7f53480ce880 2023-06-10T18:51:28.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f535019e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348157f00 con 0x7f53480d0a00 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f535019e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53480ced20 con 0x7f5348157b30 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.294+0000 7f534d739700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f53480ce880 0x7f53480d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:57994/0 (socket says 172.21.15.104:57994) 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534d739700 1 -- 172.21.15.104:0/942628246 learned_addr learned my addr 172.21.15.104:0/942628246 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 --2- 172.21.15.104:0/942628246 >> v2:172.21.15.114:3300/0 conn(0x7f5348157b30 0x7f53480ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094201648 0 0) 0x7f53480ad3c0 con 0x7f53480ce880 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53340035b0 con 0x7f53480ce880 2023-06-10T18:51:28.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3948824985 0 0) 0x7f53340035b0 con 0x7f53480ce880 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 >> v2:172.21.15.114:3300/0 conn(0x7f5348157b30 msgr2=0x7f53480ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 --2- 172.21.15.104:0/942628246 >> v2:172.21.15.114:3300/0 conn(0x7f5348157b30 0x7f53480ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 >> v1:172.21.15.114:6789/0 conn(0x7f53480d0a00 legacy=0x7f53480b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480b0170 con 0x7f53480ce880 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5344001670 con 0x7f53480ce880 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 >> v1:172.21.15.104:6789/0 conn(0x7f53480ce880 legacy=0x7f53480d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 0x7f5334003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334004850 0x7f5334004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5334005200 0x7f533400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533400c050 con 0x7f5334003a00 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533400c1c0 con 0x7f5334005200 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533400c380 con 0x7f5334004850 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 0x7f5334003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534d739700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334004850 0x7f5334004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534e73b700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5334005200 0x7f533400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 -- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334004850 msgr2=0x7f5334004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334004850 0x7f5334004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 -- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5334005200 msgr2=0x7f533400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.295+0000 7f534df3a700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5334005200 0x7f533400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534e73b700 1 -- 172.21.15.104:0/942628246 reap_dead start 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534df3a700 1 -- 172.21.15.104:0/942628246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53340035b0 con 0x7f5334003a00 2023-06-10T18:51:28.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534df3a700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 0x7f5334003e10 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5338009b60 tx=0x7f5334005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6e546e631e724 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800e6c0 con 0x7f5334003a00 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533800e820 con 0x7f5334003a00 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f534cf38700 1 -- 172.21.15.104:0/942628246 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800eb90 con 0x7f5334003a00 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 -- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 msgr2=0x7f5334003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 0x7f5334003e10 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5338009b60 tx=0x7f5334005880 comp rx=0 tx=0).stop 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 -- 172.21.15.104:0/942628246 shutdown_connections 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 --2- 172.21.15.104:0/942628246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5334003a00 0x7f5334003e10 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 -- 172.21.15.104:0/942628246 >> 172.21.15.104:0/942628246 conn(0x7f5348089fd0 msgr2=0x7f534808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 -- 172.21.15.104:0/942628246 shutdown_connections 2023-06-10T18:51:28.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.296+0000 7f535019e700 1 -- 172.21.15.104:0/942628246 wait complete. 2023-06-10T18:51:28.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.298+0000 7f535019e700 1 Processor -- start 2023-06-10T18:51:28.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.298+0000 7f535019e700 1 -- start start 2023-06-10T18:51:28.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334003a00 0x7f53481592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 0x7f534815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53480c64b0 con 0x7f53480c4fc0 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534815bef0 con 0x7f534815b510 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f535019e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53480c5830 con 0x7f5334003a00 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60180/0 (socket says 172.21.15.104:60180) 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 -- 172.21.15.104:0/3022324665 learned_addr learned my addr 172.21.15.104:0/3022324665 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:28.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534df3a700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334003a00 0x7f53481592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534e73b700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 0x7f534815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 -- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334003a00 msgr2=0x7f53481592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334003a00 0x7f53481592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 -- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 msgr2=0x7f534815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 0x7f534815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 -- 172.21.15.104:0/3022324665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480c5b10 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534e73b700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 0x7f534815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.299+0000 7f534d739700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5344000ff0 tx=0x7f534400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344012590 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5344012730 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53480c5df0 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344012ae0 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53480c5430 con 0x7f53480c4fc0 2023-06-10T18:51:28.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.300+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53480c55f0 con 0x7f53480c4fc0 2023-06-10T18:51:28.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.301+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5344012c40 con 0x7f53480c4fc0 2023-06-10T18:51:28.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.301+0000 7f533e7fc700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 0x7f532c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.301+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5344089390 con 0x7f53480c4fc0 2023-06-10T18:51:28.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.302+0000 7f534df3a700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 0x7f532c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.302+0000 7f534df3a700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 0x7f532c06d190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f53380090b0 tx=0x7f533801e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.306+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f53480c55f0 con 0x7f53480c4fc0 2023-06-10T18:51:28.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.490+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f53480c65f0 con 0x7f53480c4fc0 2023-06-10T18:51:28.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.490+0000 7f533e7fc700 1 -- 172.21.15.104:0/3022324665 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7f53480c65f0 con 0x7f53480c4fc0 2023-06-10T18:51:28.497 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:51:28.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 msgr2=0x7f532c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 0x7f532c06d190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f53380090b0 tx=0x7f533801e040 comp rx=0 tx=0).stop 2023-06-10T18:51:28.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 msgr2=0x7f5348150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5344000ff0 tx=0x7f534400eb20 comp rx=0 tx=0).stop 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 shutdown_connections 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532c06ace0 0x7f532c06d190 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534815b510 0x7f534815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5334003a00 0x7f53481592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 --2- 172.21.15.104:0/3022324665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53480c4fc0 0x7f5348150590 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.492+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 >> 172.21.15.104:0/3022324665 conn(0x7f5348089fd0 msgr2=0x7f53480aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.493+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 shutdown_connections 2023-06-10T18:51:28.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.493+0000 7f535019e700 1 -- 172.21.15.104:0/3022324665 wait complete. 2023-06-10T18:51:28.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:51:28.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.628+0000 7f786ef53700 1 Processor -- start 2023-06-10T18:51:28.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.628+0000 7f786ef53700 1 -- start start 2023-06-10T18:51:28.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ef53700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f786815c580 0x7f78680b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ef53700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ef53700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786806e250 con 0x7f786806cde0 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ef53700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786815c950 con 0x7f786815c580 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ef53700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78680cf0e0 con 0x7f786806e6e0 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ccef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f786815c580 0x7f78680b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786d4f0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f786806cde0 0x7f7868069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58018/0 (socket says 172.21.15.104:58018) 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786d4f0700 1 -- 172.21.15.104:0/1370233843 learned_addr learned my addr 172.21.15.104:0/1370233843 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:28.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ccef700 1 --2- 172.21.15.104:0/1370233843 >> v2:172.21.15.104:3300/0 conn(0x7f786815c580 0x7f78680b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60186/0 (socket says 172.21.15.104:60186) 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f7867fff700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ccef700 1 -- 172.21.15.104:0/1370233843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 msgr2=0x7f78680b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.629+0000 7f786ccef700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ccef700 1 -- 172.21.15.104:0/1370233843 >> v1:172.21.15.104:6789/0 conn(0x7f786806cde0 legacy=0x7f7868069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ccef700 1 -- 172.21.15.104:0/1370233843 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78680b1e80 con 0x7f786815c580 2023-06-10T18:51:28.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f78677fe700 1 -- 172.21.15.104:0/1370233843 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555095948 0 0) 0x7f786806e250 con 0x7f786806cde0 2023-06-10T18:51:28.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ccef700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680b1070 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f785c009820 tx=0x7f785c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e1edc24dfcb0ae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f78677fe700 1 -- 172.21.15.104:0/1370233843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c005e30 con 0x7f786815c580 2023-06-10T18:51:28.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f78677fe700 1 -- 172.21.15.104:0/1370233843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c006030 con 0x7f786815c580 2023-06-10T18:51:28.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f78677fe700 1 -- 172.21.15.104:0/1370233843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c0063c0 con 0x7f786815c580 2023-06-10T18:51:28.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 -- 172.21.15.104:0/1370233843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 msgr2=0x7f78680b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680b1070 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f785c009820 tx=0x7f785c009b30 comp rx=0 tx=0).stop 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 -- 172.21.15.104:0/1370233843 shutdown_connections 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680b1070 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 --2- 172.21.15.104:0/1370233843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 -- 172.21.15.104:0/1370233843 >> 172.21.15.104:0/1370233843 conn(0x7f786806b8a0 msgr2=0x7f78680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 -- 172.21.15.104:0/1370233843 shutdown_connections 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.630+0000 7f786ef53700 1 -- 172.21.15.104:0/1370233843 wait complete. 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.632+0000 7f786ef53700 1 Processor -- start 2023-06-10T18:51:28.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.632+0000 7f786ef53700 1 -- start start 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806cde0 0x7f78680ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680c33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680af050 con 0x7f786815c580 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680afbd0 con 0x7f786806cde0 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ef53700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78680af2a0 con 0x7f786806e6e0 2023-06-10T18:51:28.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786d4f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786d4f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60194/0 (socket says 172.21.15.104:60194) 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786d4f0700 1 -- 172.21.15.104:0/1908944695 learned_addr learned my addr 172.21.15.104:0/1908944695 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f7867fff700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680c33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.633+0000 7f786ccef700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806cde0 0x7f78680ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 -- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 msgr2=0x7f78680c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680c33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 -- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806cde0 msgr2=0x7f78680ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806cde0 0x7f78680ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 -- 172.21.15.104:0/1908944695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c009430 con 0x7f786815c580 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786d4f0700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f786000cba0 tx=0x7f786000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786000aaa0 con 0x7f786815c580 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000ac20 con 0x7f786815c580 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78680af580 con 0x7f786815c580 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78600070d0 con 0x7f786815c580 2023-06-10T18:51:28.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.634+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78680b0060 con 0x7f786815c580 2023-06-10T18:51:28.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.635+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78680afd80 con 0x7f786815c580 2023-06-10T18:51:28.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.635+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7860007230 con 0x7f786815c580 2023-06-10T18:51:28.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.636+0000 7f78657fa700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 0x7f784c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.636+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f78600893e0 con 0x7f786815c580 2023-06-10T18:51:28.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.636+0000 7f786ccef700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 0x7f784c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.636+0000 7f786ccef700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 0x7f784c06d190 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f785c000c00 tx=0x7f785c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.640+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f78680afd80 con 0x7f786815c580 2023-06-10T18:51:28.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.826+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78680afd40 con 0x7f786815c580 2023-06-10T18:51:28.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.827+0000 7f78657fa700 1 -- 172.21.15.104:0/1908944695 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7860054240 con 0x7f786815c580 2023-06-10T18:51:28.833 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:51:28.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 msgr2=0x7f784c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 0x7f784c06d190 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f785c000c00 tx=0x7f785c008070 comp rx=0 tx=0).stop 2023-06-10T18:51:28.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 msgr2=0x7f78680c3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f786000cba0 tx=0x7f786000ceb0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 shutdown_connections 2023-06-10T18:51:28.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f784c06ace0 0x7f784c06d190 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806cde0 0x7f78680ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f786815c580 0x7f78680c3910 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 --2- 172.21.15.104:0/1908944695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f786806e6e0 0x7f78680c33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 >> 172.21.15.104:0/1908944695 conn(0x7f786806b8a0 msgr2=0x7f7868158ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 shutdown_connections 2023-06-10T18:51:28.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.829+0000 7f786ef53700 1 -- 172.21.15.104:0/1908944695 wait complete. 2023-06-10T18:51:28.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:51:28.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.964+0000 7f42c3e13700 1 Processor -- start 2023-06-10T18:51:28.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.964+0000 7f42c3e13700 1 -- start start 2023-06-10T18:51:28.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c3e13700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42bc0d0a00 0x7f42bc164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c3e13700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0ad3c0 con 0x7f42bc157b30 2023-06-10T18:51:28.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c3e13700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc157f00 con 0x7f42bc0cea50 2023-06-10T18:51:28.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c3e13700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc0d0e60 con 0x7f42bc0d0a00 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c1baf700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f42bc157b30 0x7f42bc0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58026/0 (socket says 172.21.15.104:58026) 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c1baf700 1 -- 172.21.15.104:0/2965702475 learned_addr learned my addr 172.21.15.104:0/2965702475 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c23b0700 1 --2- 172.21.15.104:0/2965702475 >> v2:172.21.15.114:3300/0 conn(0x7f42bc0d0a00 0x7f42bc164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451069588 0 0) 0x7f42bc0ad3c0 con 0x7f42bc157b30 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a80035b0 con 0x7f42bc157b30 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3350838300 0 0) 0x7f42a80035b0 con 0x7f42bc157b30 2023-06-10T18:51:28.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 >> v2:172.21.15.114:3300/0 conn(0x7f42bc0d0a00 msgr2=0x7f42bc164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 --2- 172.21.15.104:0/2965702475 >> v2:172.21.15.114:3300/0 conn(0x7f42bc0d0a00 0x7f42bc164e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 >> v1:172.21.15.114:6789/0 conn(0x7f42bc0cea50 legacy=0x7f42bc0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0b0170 con 0x7f42bc157b30 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f42ac001670 con 0x7f42bc157b30 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c23b0700 1 --2- 172.21.15.104:0/2965702475 >> v2:172.21.15.114:3300/0 conn(0x7f42bc0d0a00 0x7f42bc164e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 >> v1:172.21.15.104:6789/0 conn(0x7f42bc157b30 legacy=0x7f42bc0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 0x7f42a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8004850 0x7f42a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42a8005200 0x7f42a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0d0e60 con 0x7f42a8003a00 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a800c080 con 0x7f42a8004850 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a800c240 con 0x7f42a8005200 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c1baf700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 0x7f42a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.965+0000 7f42c13ae700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8004850 0x7f42a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c23b0700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42a8005200 0x7f42a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 -- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42a8005200 msgr2=0x7f42a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42a8005200 0x7f42a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 -- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8004850 msgr2=0x7f42a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8004850 0x7f42a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 -- 172.21.15.104:0/2965702475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a80035b0 con 0x7f42a8003a00 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c23b0700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42a8005200 0x7f42a800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c13ae700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8004850 0x7f42a8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:28.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c23b0700 1 -- 172.21.15.104:0/2965702475 reap_dead start 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c1baf700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 0x7f42a8003e10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c2a0 tx=0x7f42a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce2dc88d072024 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac007c10 con 0x7f42a8003a00 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac00cde0 con 0x7f42a8003a00 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c0bad700 1 -- 172.21.15.104:0/2965702475 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac005ba0 con 0x7f42a8003a00 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c3e13700 1 -- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 msgr2=0x7f42a8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c3e13700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 0x7f42a8003e10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00c2a0 tx=0x7f42a8005880 comp rx=0 tx=0).stop 2023-06-10T18:51:28.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c3e13700 1 -- 172.21.15.104:0/2965702475 shutdown_connections 2023-06-10T18:51:28.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c3e13700 1 --2- 172.21.15.104:0/2965702475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42a8003a00 0x7f42a8003e10 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.966+0000 7f42c3e13700 1 -- 172.21.15.104:0/2965702475 >> 172.21.15.104:0/2965702475 conn(0x7f42bc089fd0 msgr2=0x7f42bc08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:28.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.967+0000 7f42c3e13700 1 -- 172.21.15.104:0/2965702475 shutdown_connections 2023-06-10T18:51:28.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.967+0000 7f42c3e13700 1 -- 172.21.15.104:0/2965702475 wait complete. 2023-06-10T18:51:28.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 Processor -- start 2023-06-10T18:51:28.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 -- start start 2023-06-10T18:51:28.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8003a00 0x7f42bc06c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc0ad3c0 0x7f42bc0aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0ad930 con 0x7f42bc0b35a0 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0adc10 con 0x7f42a8003a00 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.968+0000 7f42c3e13700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0adef0 con 0x7f42bc0ad3c0 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60212/0 (socket says 172.21.15.104:60212) 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 -- 172.21.15.104:0/1863468675 learned_addr learned my addr 172.21.15.104:0/1863468675 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:28.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c1baf700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8003a00 0x7f42bc06c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c23b0700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc0ad3c0 0x7f42bc0aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 -- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc0ad3c0 msgr2=0x7f42bc0aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc0ad3c0 0x7f42bc0aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 -- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8003a00 msgr2=0x7f42bc06c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8003a00 0x7f42bc06c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 -- 172.21.15.104:0/1863468675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0ae1d0 con 0x7f42bc0b35a0 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c13ae700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f42b800e550 tx=0x7f42b800e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b800ee80 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b80052e0 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc0cd4b0 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8003680 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.969+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc0b63f0 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.970+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc0cd310 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.971+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f42b80037e0 con 0x7f42bc0b35a0 2023-06-10T18:51:28.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.971+0000 7f42b27fc700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 0x7f42a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.971+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f42b8088b50 con 0x7f42bc0b35a0 2023-06-10T18:51:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.971+0000 7f42c1baf700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 0x7f42a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.972+0000 7f42c1baf700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 0x7f42a006d190 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f42ac000c00 tx=0x7f42ac007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:28.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:28.976+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f42bc0cd310 con 0x7f42bc0b35a0 2023-06-10T18:51:29.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.159+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f42bc0cd310 con 0x7f42bc0b35a0 2023-06-10T18:51:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.159+0000 7f42b27fc700 1 -- 172.21.15.104:0/1863468675 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f42bc0cd310 con 0x7f42bc0b35a0 2023-06-10T18:51:29.165 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 msgr2=0x7f42a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 0x7f42a006d190 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f42ac000c00 tx=0x7f42ac007000 comp rx=0 tx=0).stop 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 msgr2=0x7f42bc0b5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f42b800e550 tx=0x7f42b800e860 comp rx=0 tx=0).stop 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 shutdown_connections 2023-06-10T18:51:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f42a006ace0 0x7f42a006d190 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42a8003a00 0x7f42bc06c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc0ad3c0 0x7f42bc0aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 --2- 172.21.15.104:0/1863468675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc0b35a0 0x7f42bc0b5eb0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 >> 172.21.15.104:0/1863468675 conn(0x7f42bc089fd0 msgr2=0x7f42bc0b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 shutdown_connections 2023-06-10T18:51:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.161+0000 7f42c3e13700 1 -- 172.21.15.104:0/1863468675 wait complete. 2023-06-10T18:51:29.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:51:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.293+0000 7ff23e9c9700 1 Processor -- start 2023-06-10T18:51:29.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.293+0000 7ff23e9c9700 1 -- start start 2023-06-10T18:51:29.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23e9c9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff238071130 0x7ff2380717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23e9c9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff23815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23e9c9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23806b2e0 con 0x7ff23815a0b0 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23e9c9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23815e150 con 0x7ff238071130 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23e9c9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23815db40 con 0x7ff2380c93e0 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff2377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff238071130 0x7ff2380717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff2377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff238071130 0x7ff2380717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60220/0 (socket says 172.21.15.104:60220) 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff237fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff23815a0b0 0x7ff238070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58046/0 (socket says 172.21.15.104:58046) 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff2377fe700 1 -- 172.21.15.104:0/2211069099 learned_addr learned my addr 172.21.15.104:0/2211069099 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:29.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.294+0000 7ff23cf66700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff23815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff2377fe700 1 -- 172.21.15.104:0/2211069099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 msgr2=0x7ff23815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff2377fe700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff23815e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff2377fe700 1 -- 172.21.15.104:0/2211069099 >> v1:172.21.15.104:6789/0 conn(0x7ff23815a0b0 legacy=0x7ff238070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff2377fe700 1 -- 172.21.15.104:0/2211069099 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23806b530 con 0x7ff238071130 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff236ffd700 1 -- 172.21.15.104:0/2211069099 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550014431 0 0) 0x7ff23806b2e0 con 0x7ff23815a0b0 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff2377fe700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff238071130 0x7ff2380717d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff2280098f0 tx=0x7ff228009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a1e16f05294693 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff236ffd700 1 -- 172.21.15.104:0/2211069099 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800b8a0 con 0x7ff238071130 2023-06-10T18:51:29.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff236ffd700 1 -- 172.21.15.104:0/2211069099 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22800ba00 con 0x7ff238071130 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff236ffd700 1 -- 172.21.15.104:0/2211069099 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800bd90 con 0x7ff238071130 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 -- 172.21.15.104:0/2211069099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff238071130 msgr2=0x7ff2380717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff238071130 0x7ff2380717d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff2280098f0 tx=0x7ff228009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 -- 172.21.15.104:0/2211069099 shutdown_connections 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff23815e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/2211069099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff238071130 0x7ff2380717d0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.295+0000 7ff23e9c9700 1 -- 172.21.15.104:0/2211069099 >> 172.21.15.104:0/2211069099 conn(0x7ff2380b1610 msgr2=0x7ff2380ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:29.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.296+0000 7ff23e9c9700 1 -- 172.21.15.104:0/2211069099 shutdown_connections 2023-06-10T18:51:29.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.296+0000 7ff23e9c9700 1 -- 172.21.15.104:0/2211069099 wait complete. 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.297+0000 7ff23e9c9700 1 Processor -- start 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.297+0000 7ff23e9c9700 1 -- start start 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff238071130 0x7ff238157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff2380aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380cc960 con 0x7ff23815a0b0 2023-06-10T18:51:29.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380ccbb0 con 0x7ff238071130 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380cce90 con 0x7ff2380c93e0 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60228/0 (socket says 172.21.15.104:60228) 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 -- 172.21.15.104:0/4088587539 learned_addr learned my addr 172.21.15.104:0/4088587539 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff2377fe700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff2380aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff237fff700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff238071130 0x7ff238157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 -- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 msgr2=0x7ff2380aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff2380aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 -- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff238071130 msgr2=0x7ff238157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff238071130 0x7ff238157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 -- 172.21.15.104:0/4088587539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228009550 con 0x7ff23815a0b0 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23cf66700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff230007e40 tx=0x7ff23000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23000ed30 con 0x7ff23815a0b0 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff230014420 con 0x7ff23815a0b0 2023-06-10T18:51:29.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380cd170 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2300147d0 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.298+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2380cd970 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.299+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff238074050 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.300+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff230014930 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.300+0000 7ff234ff9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 0x7ff22406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.300+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7ff230089c90 con 0x7ff23815a0b0 2023-06-10T18:51:29.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.300+0000 7ff237fff700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 0x7ff22406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.301+0000 7ff237fff700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 0x7ff22406d190 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff22c000f60 tx=0x7ff22c007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.304+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff230054cd0 con 0x7ff23815a0b0 2023-06-10T18:51:29.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.507+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff238071550 con 0x7ff23815a0b0 2023-06-10T18:51:29.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.507+0000 7ff234ff9700 1 -- 172.21.15.104:0/4088587539 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v71) v1 ==== 80+0+57 (secure 0 0 0) 0x7ff2300582f0 con 0x7ff23815a0b0 2023-06-10T18:51:29.513 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/2562980824 2023-06-10T19:50:53.863403+0000 2023-06-10T18:51:29.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.509+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 msgr2=0x7ff22406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.509+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 0x7ff22406d190 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff22c000f60 tx=0x7ff22c007e70 comp rx=0 tx=0).stop 2023-06-10T18:51:29.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.509+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 msgr2=0x7ff2380af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.509+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff230007e40 tx=0x7ff23000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 shutdown_connections 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff22406ace0 0x7ff22406d190 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff238071130 0x7ff238157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff23815a0b0 0x7ff2380af140 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 --2- 172.21.15.104:0/4088587539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2380c93e0 0x7ff2380aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 >> 172.21.15.104:0/4088587539 conn(0x7ff2380b1610 msgr2=0x7ff238159310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:29.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 shutdown_connections 2023-06-10T18:51:29.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.510+0000 7ff23e9c9700 1 -- 172.21.15.104:0/4088587539 wait complete. 2023-06-10T18:51:29.519 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:51:29.527 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T18:51:29.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:29.528 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:29.583 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:51:29.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:29.585 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:51:29.601 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:51:29.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:51:29.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:51:29.731 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:51:29.731 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:51:29.732 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:29.748 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:51:29.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:51:29.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:29.749 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:51:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-06-10T18:51:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:51:29.805 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:51:29.805 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:29.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:29.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.959+0000 7f066c727700 1 Processor -- start 2023-06-10T18:51:29.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.959+0000 7f066c727700 1 -- start start 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066c727700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f066415c730 0x7f066406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066c727700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066c727700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066415cb00 con 0x7f066406cd70 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066c727700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06640aed80 con 0x7f066415c730 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066c727700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640c9c60 con 0x7f06640cc000 2023-06-10T18:51:29.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f0669cc2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066a4c3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f066415c730 0x7f066406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f0669cc2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:51384/0 (socket says 172.21.15.104:51384) 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066a4c3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f066415c730 0x7f066406e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:47734/0 (socket says 172.21.15.104:47734) 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f066acc4700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f066406cd70 0x7f066406d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:35678/0 (socket says 172.21.15.104:35678) 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.960+0000 7f0669cc2700 1 -- 172.21.15.104:0/584310816 learned_addr learned my addr 172.21.15.104:0/584310816 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f06694c1700 1 -- 172.21.15.104:0/584310816 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664729909 0 0) 0x7f065c001490 con 0x7f066406cd70 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f06694c1700 1 -- 172.21.15.104:0/584310816 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06500035b0 con 0x7f066406cd70 2023-06-10T18:51:29.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f066a4c3700 1 -- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 msgr2=0x7f06640cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f066a4c3700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f066a4c3700 1 -- 172.21.15.104:0/584310816 >> v1:172.21.15.114:6789/0 conn(0x7f066406cd70 legacy=0x7f066406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f066a4c3700 1 -- 172.21.15.104:0/584310816 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066406ecd0 con 0x7f066415c730 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f0669cc2700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f066a4c3700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f066406e790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0654009820 tx=0x7f0654009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1f91be86b6faeb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f06694c1700 1 -- 172.21.15.104:0/584310816 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654005e30 con 0x7f066415c730 2023-06-10T18:51:29.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f06694c1700 1 -- 172.21.15.104:0/584310816 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654006030 con 0x7f066415c730 2023-06-10T18:51:29.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.961+0000 7f06694c1700 1 -- 172.21.15.104:0/584310816 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540063c0 con 0x7f066415c730 2023-06-10T18:51:29.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 -- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 msgr2=0x7f066406e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f066406e790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0654009820 tx=0x7f0654009b30 comp rx=0 tx=0).stop 2023-06-10T18:51:29.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 -- 172.21.15.104:0/584310816 shutdown_connections 2023-06-10T18:51:29.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f066406e790 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 --2- 172.21.15.104:0/584310816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 -- 172.21.15.104:0/584310816 >> 172.21.15.104:0/584310816 conn(0x7f066406b8a0 msgr2=0x7f06640cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 -- 172.21.15.104:0/584310816 shutdown_connections 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.962+0000 7f066c727700 1 -- 172.21.15.104:0/584310816 wait complete. 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.963+0000 7f066c727700 1 Processor -- start 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.963+0000 7f066c727700 1 -- start start 2023-06-10T18:51:29.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f06640cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640b1f70 con 0x7f066406cd70 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640afaf0 con 0x7f066415c730 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640af1c0 con 0x7f06640cc000 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60238/0 (socket says 172.21.15.104:60238) 2023-06-10T18:51:29.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 -- 172.21.15.104:0/1216622813 learned_addr learned my addr 172.21.15.104:0/1216622813 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f0669cc2700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066acc4700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f06640cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 -- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 msgr2=0x7f06640cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 -- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 msgr2=0x7f06640cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f06640cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 -- 172.21.15.104:0/1216622813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066000b7f0 con 0x7f066406cd70 2023-06-10T18:51:29.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066acc4700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f06640cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066a4c3700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0654000c00 tx=0x7f0654004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540063a0 con 0x7f066406cd70 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654020070 con 0x7f066406cd70 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654006760 con 0x7f066406cd70 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.964+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0654009430 con 0x7f066406cd70 2023-06-10T18:51:29.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.965+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06640af780 con 0x7f066406cd70 2023-06-10T18:51:29.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.965+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0664074050 con 0x7f066406cd70 2023-06-10T18:51:29.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.966+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0654022030 con 0x7f066406cd70 2023-06-10T18:51:29.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.966+0000 7f065affd700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 0x7f064806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:29.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.966+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f065408aaa0 con 0x7f066406cd70 2023-06-10T18:51:29.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.967+0000 7f0669cc2700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 0x7f064806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:29.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.967+0000 7f0669cc2700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 0x7f064806d190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f066000b240 tx=0x7f066000b4d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:29.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:29.971+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f06540559b0 con 0x7f066406cd70 2023-06-10T18:51:30.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.161+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0664066f10 con 0x7f066406cd70 2023-06-10T18:51:30.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.162+0000 7f065affd700 1 -- 172.21.15.104:0/1216622813 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18760 (secure 0 0 0) 0x7f0654058fd0 con 0x7f066406cd70 2023-06-10T18:51:30.168 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:30.168 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":71,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:50:54.764730+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.a","mappings":[{"from":6,"to":0}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/2562980824":"2023-06-10T19:50:53.863403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 msgr2=0x7f064806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 0x7f064806d190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f066000b240 tx=0x7f066000b4d0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 msgr2=0x7f06640b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0654000c00 tx=0x7f0654004750 comp rx=0 tx=0).stop 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 shutdown_connections 2023-06-10T18:51:30.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064806ace0 0x7f064806d190 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f066415c730 0x7f06640cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f066406cd70 0x7f06640b1520 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 --2- 172.21.15.104:0/1216622813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06640cc000 0x7f06640cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 >> 172.21.15.104:0/1216622813 conn(0x7f066406b8a0 msgr2=0x7f066415b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 shutdown_connections 2023-06-10T18:51:30.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.164+0000 7f066c727700 1 -- 172.21.15.104:0/1216622813 wait complete. 2023-06-10T18:51:30.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:48:42.072415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:48:43.111638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:51:30.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:51:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.299+0000 7fac83e51700 1 Processor -- start 2023-06-10T18:51:30.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.299+0000 7fac83e51700 1 -- start start 2023-06-10T18:51:30.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac83e51700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac7c15c6f0 0x7fac7c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac83e51700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac7c0aec00 0x7fac7c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac83e51700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c0ae7e0 con 0x7fac7c06ea70 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac83e51700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7c06df90 con 0x7fac7c15c6f0 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac83e51700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7c06e270 con 0x7fac7c0aec00 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac7c15c6f0 0x7fac7c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac7c15c6f0 0x7fac7c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60246/0 (socket says 172.21.15.104:60246) 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 -- 172.21.15.104:0/2441286079 learned_addr learned my addr 172.21.15.104:0/2441286079 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac823ee700 1 --2- 172.21.15.104:0/2441286079 >> v2:172.21.15.114:3300/0 conn(0x7fac7c0aec00 0x7fac7c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 -- 172.21.15.104:0/2441286079 >> v2:172.21.15.114:3300/0 conn(0x7fac7c0aec00 msgr2=0x7fac7c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 --2- 172.21.15.104:0/2441286079 >> v2:172.21.15.114:3300/0 conn(0x7fac7c0aec00 0x7fac7c0b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 -- 172.21.15.104:0/2441286079 >> v1:172.21.15.114:6789/0 conn(0x7fac7c06ea70 legacy=0x7fac7c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 -- 172.21.15.104:0/2441286079 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c06e550 con 0x7fac7c15c6f0 2023-06-10T18:51:30.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac81bed700 1 --2- 172.21.15.104:0/2441286079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c15c6f0 0x7fac7c0b1070 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fac7c06ff70 tx=0x7fac78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacca4397734560b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac80beb700 1 -- 172.21.15.104:0/2441286079 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78005e30 con 0x7fac7c15c6f0 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac80beb700 1 -- 172.21.15.104:0/2441286079 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78006030 con 0x7fac7c15c6f0 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.300+0000 7fac80beb700 1 -- 172.21.15.104:0/2441286079 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780063e0 con 0x7fac7c15c6f0 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 -- 172.21.15.104:0/2441286079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c15c6f0 msgr2=0x7fac7c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 --2- 172.21.15.104:0/2441286079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c15c6f0 0x7fac7c0b1070 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fac7c06ff70 tx=0x7fac78009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 -- 172.21.15.104:0/2441286079 shutdown_connections 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 --2- 172.21.15.104:0/2441286079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c15c6f0 0x7fac7c0b1070 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 --2- 172.21.15.104:0/2441286079 >> v2:172.21.15.114:3300/0 conn(0x7fac7c0aec00 0x7fac7c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 -- 172.21.15.104:0/2441286079 >> 172.21.15.104:0/2441286079 conn(0x7fac7c06b8a0 msgr2=0x7fac7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 -- 172.21.15.104:0/2441286079 shutdown_connections 2023-06-10T18:51:30.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.301+0000 7fac83e51700 1 -- 172.21.15.104:0/2441286079 wait complete. 2023-06-10T18:51:30.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.302+0000 7fac83e51700 1 Processor -- start 2023-06-10T18:51:30.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.302+0000 7fac83e51700 1 -- start start 2023-06-10T18:51:30.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 0x7fac7c06a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac7c0ca820 0x7fac7c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0b2960 con 0x7fac7c06ea70 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0c4530 con 0x7fac7c0aec00 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac83e51700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0c4810 con 0x7fac7c0ca820 2023-06-10T18:51:30.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60250/0 (socket says 172.21.15.104:60250) 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 -- 172.21.15.104:0/56325013 learned_addr learned my addr 172.21.15.104:0/56325013 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac823ee700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac7c0ca820 0x7fac7c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac813ec700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 0x7fac7c06a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 -- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac7c0ca820 msgr2=0x7fac7c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac7c0ca820 0x7fac7c0caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 -- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 msgr2=0x7fac7c06a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 0x7fac7c06a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 -- 172.21.15.104:0/56325013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78009500 con 0x7fac7c06ea70 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac813ec700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 0x7fac7c06a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac81bed700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fac78006000 tx=0x7fac78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.303+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780068d0 con 0x7fac7c06ea70 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.304+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78020070 con 0x7fac7c06ea70 2023-06-10T18:51:30.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.304+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c0c4af0 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.304+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78025670 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.304+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c0cb390 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.304+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c0cdb20 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.305+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac78022030 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.305+0000 7fac727fc700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 0x7fac6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.305+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fac7808ac20 con 0x7fac7c06ea70 2023-06-10T18:51:30.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.306+0000 7fac813ec700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 0x7fac6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.306+0000 7fac813ec700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 0x7fac6006d190 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0c4150 tx=0x7fac6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.310+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac7c0cdb20 con 0x7fac7c06ea70 2023-06-10T18:51:30.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.494+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac7c0c4950 con 0x7fac7c06ea70 2023-06-10T18:51:30.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.494+0000 7fac727fc700 1 -- 172.21.15.104:0/56325013 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac7c0c4950 con 0x7fac7c06ea70 2023-06-10T18:51:30.500 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 msgr2=0x7fac6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 0x7fac6006d190 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0c4150 tx=0x7fac6c009040 comp rx=0 tx=0).stop 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 msgr2=0x7fac7c0b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fac78006000 tx=0x7fac78004750 comp rx=0 tx=0).stop 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 shutdown_connections 2023-06-10T18:51:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac7c0ca820 0x7fac7c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac7c06ea70 0x7fac7c0b2420 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac6006ace0 0x7fac6006d190 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 --2- 172.21.15.104:0/56325013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac7c0aec00 0x7fac7c06a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 >> 172.21.15.104:0/56325013 conn(0x7fac7c06b8a0 msgr2=0x7fac7c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 shutdown_connections 2023-06-10T18:51:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.496+0000 7fac83e51700 1 -- 172.21.15.104:0/56325013 wait complete. 2023-06-10T18:51:30.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:51:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.631+0000 7fb4a7bad700 1 Processor -- start 2023-06-10T18:51:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.631+0000 7fb4a7bad700 1 -- start start 2023-06-10T18:51:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a7bad700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0156760 0x7fb4a00ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a7bad700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 0x7fb4a00b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a7bad700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a00ce850 con 0x7fb4a015d1a0 2023-06-10T18:51:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a7bad700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a00b30a0 con 0x7fb4a0156760 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a7bad700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a015d740 con 0x7fb4a00b21c0 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0156760 0x7fb4a00ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5148700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb4a015d1a0 0x7fb4a00cad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58086/0 (socket says 172.21.15.104:58086) 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0156760 0x7fb4a00ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60258/0 (socket says 172.21.15.104:60258) 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5148700 1 -- 172.21.15.104:0/3716769785 learned_addr learned my addr 172.21.15.104:0/3716769785 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a614a700 1 --2- 172.21.15.104:0/3716769785 >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 0x7fb4a00b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 -- 172.21.15.104:0/3716769785 >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 msgr2=0x7fb4a00b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 --2- 172.21.15.104:0/3716769785 >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 0x7fb4a00b25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 -- 172.21.15.104:0/3716769785 >> v1:172.21.15.104:6789/0 conn(0x7fb4a015d1a0 legacy=0x7fb4a00cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 -- 172.21.15.104:0/3716769785 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00b2b20 con 0x7fb4a0156760 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a614a700 1 --2- 172.21.15.104:0/3716769785 >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 0x7fb4a00b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a5949700 1 --2- 172.21.15.104:0/3716769785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0156760 0x7fb4a00ce310 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb4a006f340 tx=0x7fb49c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e007d55ec75011 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a4947700 1 -- 172.21.15.104:0/3716769785 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c011830 con 0x7fb4a0156760 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a4947700 1 -- 172.21.15.104:0/3716769785 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c011990 con 0x7fb4a0156760 2023-06-10T18:51:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.632+0000 7fb4a4947700 1 -- 172.21.15.104:0/3716769785 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c011d20 con 0x7fb4a0156760 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 -- 172.21.15.104:0/3716769785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0156760 msgr2=0x7fb4a00ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/3716769785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0156760 0x7fb4a00ce310 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb4a006f340 tx=0x7fb49c009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 -- 172.21.15.104:0/3716769785 shutdown_connections 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/3716769785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0156760 0x7fb4a00ce310 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/3716769785 >> v2:172.21.15.114:3300/0 conn(0x7fb4a00b21c0 0x7fb4a00b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 -- 172.21.15.104:0/3716769785 >> 172.21.15.104:0/3716769785 conn(0x7fb4a006e370 msgr2=0x7fb4a006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 -- 172.21.15.104:0/3716769785 shutdown_connections 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.633+0000 7fb4a7bad700 1 -- 172.21.15.104:0/3716769785 wait complete. 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.634+0000 7fb4a7bad700 1 Processor -- start 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.634+0000 7fb4a7bad700 1 -- start start 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 0x7fb4a00ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 0x7fb4a01f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00c8860 con 0x7fb4a00b21c0 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00ae780 con 0x7fb4a015d1a0 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a7bad700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00aea00 con 0x7fb4a0156760 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60262/0 (socket says 172.21.15.104:60262) 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 -- 172.21.15.104:0/4026202847 learned_addr learned my addr 172.21.15.104:0/4026202847 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5148700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 0x7fb4a00ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a614a700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 0x7fb4a01f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 -- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 msgr2=0x7fb4a00ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 0x7fb4a00ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 -- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 msgr2=0x7fb4a01f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 0x7fb4a01f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 -- 172.21.15.104:0/4026202847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0094a0 con 0x7fb4a00b21c0 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5148700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 0x7fb4a00ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a614a700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 0x7fb4a01f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.635+0000 7fb4a5949700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0097c0 tx=0x7fb49c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0038a0 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c01f070 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a00aece0 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c003c40 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a00c8d90 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.636+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0074050 con 0x7fb4a00b21c0 2023-06-10T18:51:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.637+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb49c026480 con 0x7fb4a00b21c0 2023-06-10T18:51:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.637+0000 7fb4967fc700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.637+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fb49c021070 con 0x7fb4a00b21c0 2023-06-10T18:51:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.638+0000 7fb4a5148700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.638+0000 7fb4a5148700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00af200 tx=0x7fb490009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.642+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb49c055ff0 con 0x7fb4a00b21c0 2023-06-10T18:51:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.826+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb4a015f770 con 0x7fb4a00b21c0 2023-06-10T18:51:30.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.827+0000 7fb4967fc700 1 -- 172.21.15.104:0/4026202847 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb49c0150a0 con 0x7fb4a00b21c0 2023-06-10T18:51:30.833 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 msgr2=0x7fb48406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00af200 tx=0x7fb490009040 comp rx=0 tx=0).stop 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 msgr2=0x7fb4a00b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0097c0 tx=0x7fb49c004750 comp rx=0 tx=0).stop 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 shutdown_connections 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a015d1a0 0x7fb4a01f1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a00b21c0 0x7fb4a00b3d20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 --2- 172.21.15.104:0/4026202847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0156760 0x7fb4a00ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 >> 172.21.15.104:0/4026202847 conn(0x7fb4a006e370 msgr2=0x7fb4a006b260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 shutdown_connections 2023-06-10T18:51:30.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.829+0000 7fb4a7bad700 1 -- 172.21.15.104:0/4026202847 wait complete. 2023-06-10T18:51:30.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:51:30.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.964+0000 7f9627dab700 1 Processor -- start 2023-06-10T18:51:30.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.964+0000 7f9627dab700 1 -- start start 2023-06-10T18:51:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9627dab700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9620156760 0x7f96200ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9627dab700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200ce850 con 0x7f96200b21c0 2023-06-10T18:51:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9627dab700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200cb4a0 con 0x7f962015d1a0 2023-06-10T18:51:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9627dab700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96200b30a0 con 0x7f9620156760 2023-06-10T18:51:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9626348700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f96200b21c0 0x7f96200b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58102/0 (socket says 172.21.15.104:58102) 2023-06-10T18:51:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9626348700 1 -- 172.21.15.104:0/423406848 learned_addr learned my addr 172.21.15.104:0/423406848 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9625b47700 1 --2- 172.21.15.104:0/423406848 >> v2:172.21.15.114:3300/0 conn(0x7f9620156760 0x7f96200ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.965+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137175953 0 0) 0x7f96200ce850 con 0x7f96200b21c0 2023-06-10T18:51:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f960c0035b0 con 0x7f96200b21c0 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659917242 0 0) 0x7f960c0035b0 con 0x7f96200b21c0 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 >> v2:172.21.15.114:3300/0 conn(0x7f9620156760 msgr2=0x7f96200ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 --2- 172.21.15.104:0/423406848 >> v2:172.21.15.114:3300/0 conn(0x7f9620156760 0x7f96200ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 >> v1:172.21.15.114:6789/0 conn(0x7f962015d1a0 legacy=0x7f96200cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200cfca0 con 0x7f96200b21c0 2023-06-10T18:51:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f96180019f0 con 0x7f96200b21c0 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 >> v1:172.21.15.104:6789/0 conn(0x7f96200b21c0 legacy=0x7f96200b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f960c003a00 0x7f960c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f960c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f960c005200 0x7f960c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b30a0 con 0x7f960c004850 2023-06-10T18:51:30.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c00c080 con 0x7f960c005200 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c00c240 con 0x7f960c003a00 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f960c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625b47700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f960c003a00 0x7f960c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9626348700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f960c005200 0x7f960c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 -- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f960c003a00 msgr2=0x7f960c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f960c003a00 0x7f960c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 -- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f960c005200 msgr2=0x7f960c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f960c005200 0x7f960c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9626348700 1 -- 172.21.15.104:0/423406848 reap_dead start 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 -- 172.21.15.104:0/423406848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c0035b0 con 0x7f960c004850 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.966+0000 7f9625346700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f960c004cc0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f961c0017c0 tx=0x7f960c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f998a6c7460d1e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c0052d0 con 0x7f960c004850 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c005450 con 0x7f960c004850 2023-06-10T18:51:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9624b45700 1 -- 172.21.15.104:0/423406848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c0038c0 con 0x7f960c004850 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 -- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 msgr2=0x7f960c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f960c004cc0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f961c0017c0 tx=0x7f960c005880 comp rx=0 tx=0).stop 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 -- 172.21.15.104:0/423406848 shutdown_connections 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 --2- 172.21.15.104:0/423406848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f960c004cc0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 -- 172.21.15.104:0/423406848 >> 172.21.15.104:0/423406848 conn(0x7f962006e370 msgr2=0x7f962006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 -- 172.21.15.104:0/423406848 shutdown_connections 2023-06-10T18:51:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.967+0000 7f9627dab700 1 -- 172.21.15.104:0/423406848 wait complete. 2023-06-10T18:51:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.968+0000 7f9627dab700 1 Processor -- start 2023-06-10T18:51:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.968+0000 7f9627dab700 1 -- start start 2023-06-10T18:51:30.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 0x7f96200b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96200b3fb0 0x7f96200b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b5040 con 0x7f960c004850 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b5d80 con 0x7f96200b53f0 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9627dab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b4d60 con 0x7f96200b3fb0 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60282/0 (socket says 172.21.15.104:60282) 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 -- 172.21.15.104:0/2104006282 learned_addr learned my addr 172.21.15.104:0/2104006282 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625346700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 0x7f96200b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9626348700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96200b3fb0 0x7f96200b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 -- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96200b3fb0 msgr2=0x7f96200b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96200b3fb0 0x7f96200b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 -- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 msgr2=0x7f96200b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:30.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.969+0000 7f9625b47700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 0x7f96200b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:30.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f9625b47700 1 -- 172.21.15.104:0/2104006282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200c79b0 con 0x7f960c004850 2023-06-10T18:51:30.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f9625346700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 0x7f96200b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:30.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f9625b47700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f96100119a0 tx=0x7f9610011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610011ee0 con 0x7f960c004850 2023-06-10T18:51:30.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961000f8e0 con 0x7f960c004850 2023-06-10T18:51:30.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96200c7c90 con 0x7f960c004850 2023-06-10T18:51:30.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961000fc70 con 0x7f960c004850 2023-06-10T18:51:30.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.970+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96201541c0 con 0x7f960c004850 2023-06-10T18:51:30.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.971+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96200c81b0 con 0x7f960c004850 2023-06-10T18:51:30.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.971+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9610014440 con 0x7f960c004850 2023-06-10T18:51:30.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.972+0000 7f96167fc700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 0x7f960406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:30.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.972+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9610089be0 con 0x7f960c004850 2023-06-10T18:51:30.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.972+0000 7f9625346700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 0x7f960406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:30.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.972+0000 7f9625346700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 0x7f960406d1c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:30.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:30.976+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96200c81b0 con 0x7f960c004850 2023-06-10T18:51:31.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.163+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96200c81b0 con 0x7f960c004850 2023-06-10T18:51:31.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.163+0000 7f96167fc700 1 -- 172.21.15.104:0/2104006282 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96200c81b0 con 0x7f960c004850 2023-06-10T18:51:31.169 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:51:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 msgr2=0x7f960406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 0x7f960406d1c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c009030 comp rx=0 tx=0).stop 2023-06-10T18:51:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 msgr2=0x7f9620153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f96100119a0 tx=0x7f9610011d60 comp rx=0 tx=0).stop 2023-06-10T18:51:31.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 shutdown_connections 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96200b3fb0 0x7f96200b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f960c004850 0x7f9620153c80 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f960406ad10 0x7f960406d1c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 --2- 172.21.15.104:0/2104006282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96200b53f0 0x7f96200b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 >> 172.21.15.104:0/2104006282 conn(0x7f962006e370 msgr2=0x7f962006b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 shutdown_connections 2023-06-10T18:51:31.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.165+0000 7f9627dab700 1 -- 172.21.15.104:0/2104006282 wait complete. 2023-06-10T18:51:31.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:51:31.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.299+0000 7f18e48a3700 1 Processor -- start 2023-06-10T18:51:31.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.299+0000 7f18e48a3700 1 -- start start 2023-06-10T18:51:31.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18e48a3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18e015c580 0x7f18e00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18e48a3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 0x7f18e00ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18e48a3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e006e250 con 0x7f18e006e6e0 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18e48a3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e015c950 con 0x7f18e015c580 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18e48a3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e00b1e80 con 0x7f18e006cde0 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18de59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18e015c580 0x7f18e00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18de59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18e015c580 0x7f18e00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60290/0 (socket says 172.21.15.104:60290) 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18de59c700 1 -- 172.21.15.104:0/3441551379 learned_addr learned my addr 172.21.15.104:0/3441551379 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:31.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.300+0000 7f18ded9d700 1 --2- 172.21.15.104:0/3441551379 >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 0x7f18e00ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18de59c700 1 -- 172.21.15.104:0/3441551379 >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 msgr2=0x7f18e00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18de59c700 1 --2- 172.21.15.104:0/3441551379 >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 0x7f18e00ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18de59c700 1 -- 172.21.15.104:0/3441551379 >> v1:172.21.15.114:6789/0 conn(0x7f18e006e6e0 legacy=0x7f18e00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18de59c700 1 -- 172.21.15.104:0/3441551379 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e006d920 con 0x7f18e015c580 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18ded9d700 1 --2- 172.21.15.104:0/3441551379 >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 0x7f18e00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18de59c700 1 --2- 172.21.15.104:0/3441551379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e015c580 0x7f18e00b1070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbc60 tx=0x7f18d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef408a692fac656 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18dd59a700 1 -- 172.21.15.104:0/3441551379 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4005e30 con 0x7f18e015c580 2023-06-10T18:51:31.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18dd59a700 1 -- 172.21.15.104:0/3441551379 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d4004030 con 0x7f18e015c580 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18dd59a700 1 -- 172.21.15.104:0/3441551379 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40043e0 con 0x7f18e015c580 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18e48a3700 1 -- 172.21.15.104:0/3441551379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e015c580 msgr2=0x7f18e00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.301+0000 7f18e48a3700 1 --2- 172.21.15.104:0/3441551379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e015c580 0x7f18e00b1070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbc60 tx=0x7f18d4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 -- 172.21.15.104:0/3441551379 shutdown_connections 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 --2- 172.21.15.104:0/3441551379 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e015c580 0x7f18e00b1070 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 --2- 172.21.15.104:0/3441551379 >> v2:172.21.15.114:3300/0 conn(0x7f18e006cde0 0x7f18e00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 -- 172.21.15.104:0/3441551379 >> 172.21.15.104:0/3441551379 conn(0x7f18e006b8a0 msgr2=0x7f18e00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:31.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 -- 172.21.15.104:0/3441551379 shutdown_connections 2023-06-10T18:51:31.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.302+0000 7f18e48a3700 1 -- 172.21.15.104:0/3441551379 wait complete. 2023-06-10T18:51:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.303+0000 7f18e48a3700 1 Processor -- start 2023-06-10T18:51:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.303+0000 7f18e48a3700 1 -- start start 2023-06-10T18:51:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 0x7f18e01578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18e015c580 0x7f18e00c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c70a0 con 0x7f18e006e6e0 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00ac970 con 0x7f18e006cde0 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18e48a3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00acc50 con 0x7f18e015c580 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18ddd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18ddd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60298/0 (socket says 172.21.15.104:60298) 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.304+0000 7f18ddd9b700 1 -- 172.21.15.104:0/448103654 learned_addr learned my addr 172.21.15.104:0/448103654 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ded9d700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18e015c580 0x7f18e00c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18de59c700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 0x7f18e01578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 -- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18e015c580 msgr2=0x7f18e00c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18e015c580 0x7f18e00c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 -- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 msgr2=0x7f18e01578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 0x7f18e01578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 -- 172.21.15.104:0/448103654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4009500 con 0x7f18e006e6e0 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18de59c700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 0x7f18e01578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:31.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ddd9b700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c4b0 tx=0x7f18d000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00052d0 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0005430 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e00acf30 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00038c0 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.305+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e0157fb0 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.306+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e0152e30 con 0x7f18e006e6e0 2023-06-10T18:51:31.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.306+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f18d0003a20 con 0x7f18e006e6e0 2023-06-10T18:51:31.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.307+0000 7f18ceffd700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 0x7f18c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.307+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f18d00889f0 con 0x7f18e006e6e0 2023-06-10T18:51:31.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.307+0000 7f18de59c700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 0x7f18c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.308+0000 7f18de59c700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 0x7f18c006d190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbc60 tx=0x7f18d4008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.311+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f18d00538f0 con 0x7f18e006e6e0 2023-06-10T18:51:31.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.500+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18e0074050 con 0x7f18e006e6e0 2023-06-10T18:51:31.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.500+0000 7f18ceffd700 1 -- 172.21.15.104:0/448103654 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18d0017020 con 0x7f18e006e6e0 2023-06-10T18:51:31.506 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:51:31.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 msgr2=0x7f18c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 0x7f18c006d190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbc60 tx=0x7f18d4008070 comp rx=0 tx=0).stop 2023-06-10T18:51:31.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 msgr2=0x7f18e00b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f18d000c4b0 tx=0x7f18d000c7c0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 shutdown_connections 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18c006ace0 0x7f18c006d190 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18e006cde0 0x7f18e01578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18e015c580 0x7f18e00c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 --2- 172.21.15.104:0/448103654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18e006e6e0 0x7f18e00b3050 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 >> 172.21.15.104:0/448103654 conn(0x7f18e006b8a0 msgr2=0x7f18e00c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:31.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 shutdown_connections 2023-06-10T18:51:31.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.502+0000 7f18e48a3700 1 -- 172.21.15.104:0/448103654 wait complete. 2023-06-10T18:51:31.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:51:31.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.635+0000 7fc0e7c1e700 1 Processor -- start 2023-06-10T18:51:31.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.635+0000 7fc0e7c1e700 1 -- start start 2023-06-10T18:51:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e7c1e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0e00ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e7c1e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e00cc050 con 0x7fc0e006a580 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e7c1e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e006e290 con 0x7fc0e006d810 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e7c1e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00cba00 con 0x7fc0e015a230 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e61bb700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc0e006a580 0x7fc0e00cc4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58136/0 (socket says 172.21.15.104:58136) 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.636+0000 7fc0e61bb700 1 -- 172.21.15.104:0/4170478352 learned_addr learned my addr 172.21.15.104:0/4170478352 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0e00ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140519917 0 0) 0x7fc0e00cc050 con 0x7fc0e006a580 2023-06-10T18:51:31.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0cc0035b0 con 0x7fc0e006a580 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580025702 0 0) 0x7fc0cc0035b0 con 0x7fc0e006a580 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 msgr2=0x7fc0e00ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0e00ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 >> v1:172.21.15.114:6789/0 conn(0x7fc0e006d810 legacy=0x7fc0e0069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e00cbce0 con 0x7fc0e006a580 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc0d80019f0 con 0x7fc0e006a580 2023-06-10T18:51:31.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 >> v1:172.21.15.104:6789/0 conn(0x7fc0e006a580 legacy=0x7fc0e00cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0cc003a30 0x7fc0cc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0cc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0cc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00cba00 con 0x7fc0cc004850 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc00bac0 con 0x7fc0cc003a30 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc00bc50 con 0x7fc0e015a230 2023-06-10T18:51:31.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0cc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0cc003a30 0x7fc0cc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e61bb700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0cc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 -- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 msgr2=0x7fc0cc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0cc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 -- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0cc003a30 msgr2=0x7fc0cc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0cc003a30 0x7fc0cc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e51b9700 1 -- 172.21.15.104:0/4170478352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0035b0 con 0x7fc0cc004850 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0cc003a30 0x7fc0cc003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.637+0000 7fc0e61bb700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e015a230 0x7fc0cc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0cc004cc0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc001800 tx=0x7fc0cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e3ce0fb24ccb37 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc0012f0 con 0x7fc0cc004850 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc0052f0 con 0x7fc0cc004850 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e49b8700 1 -- 172.21.15.104:0/4170478352 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc006860 con 0x7fc0cc004850 2023-06-10T18:51:31.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/4170478352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 msgr2=0x7fc0cc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/4170478352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0cc004cc0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc001800 tx=0x7fc0cc005880 comp rx=0 tx=0).stop 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e61bb700 1 -- 172.21.15.104:0/4170478352 reap_dead start 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/4170478352 shutdown_connections 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/4170478352 >> 172.21.15.104:0/4170478352 conn(0x7fc0e006b8a0 msgr2=0x7fc0e00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/4170478352 shutdown_connections 2023-06-10T18:51:31.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.638+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/4170478352 wait complete. 2023-06-10T18:51:31.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.639+0000 7fc0e7c1e700 1 Processor -- start 2023-06-10T18:51:31.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 -- start start 2023-06-10T18:51:31.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0e00c22e0 0x7fc0e0157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e00c2750 0x7fc0e0157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e01e4560 con 0x7fc0cc004850 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00c6930 con 0x7fc0e00c22e0 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e7c1e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00c6c10 con 0x7fc0e00c2750 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e59ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e59ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60316/0 (socket says 172.21.15.104:60316) 2023-06-10T18:51:31.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.640+0000 7fc0e59ba700 1 -- 172.21.15.104:0/3347309933 learned_addr learned my addr 172.21.15.104:0/3347309933 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e61bb700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e00c2750 0x7fc0e0157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 -- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e00c2750 msgr2=0x7fc0e0157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e00c2750 0x7fc0e0157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 -- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0e00c22e0 msgr2=0x7fc0e0157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0e00c22e0 0x7fc0e0157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0e00c22e0 0x7fc0e0157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 -- 172.21.15.104:0/3347309933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e00c6ef0 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e59ba700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00118d0 tx=0x7fc0d0011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0005c20 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0005d80 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e01e1e80 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0004240 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.641+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e01581a0 con 0x7fc0cc004850 2023-06-10T18:51:31.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.642+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e01583c0 con 0x7fc0cc004850 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.642+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc0d00043a0 con 0x7fc0cc004850 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.643+0000 7fc0d67fc700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 0x7fc0c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.643+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc0d00898e0 con 0x7fc0cc004850 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.643+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 0x7fc0c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.643+0000 7fc0e51b9700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 0x7fc0c406d190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc014040 tx=0x7fc0dc009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:31.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.647+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc0d00547e0 con 0x7fc0cc004850 2023-06-10T18:51:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.851+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc0e00c7260 con 0x7fc0cc004850 2023-06-10T18:51:31.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.852+0000 7fc0d67fc700 1 -- 172.21.15.104:0/3347309933 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v71) v1 ==== 80+0+57 (secure 0 0 0) 0x7fc0d0022050 con 0x7fc0cc004850 2023-06-10T18:51:31.858 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/2562980824 2023-06-10T19:50:53.863403+0000 2023-06-10T18:51:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 msgr2=0x7fc0c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 0x7fc0c406d190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc014040 tx=0x7fc0dc009030 comp rx=0 tx=0).stop 2023-06-10T18:51:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 msgr2=0x7fc0e01e1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00118d0 tx=0x7fc0d0011c90 comp rx=0 tx=0).stop 2023-06-10T18:51:31.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 shutdown_connections 2023-06-10T18:51:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0e00c2750 0x7fc0e0157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0cc004850 0x7fc0e01e1940 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc0c406ace0 0x7fc0c406d190 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 --2- 172.21.15.104:0/3347309933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0e00c22e0 0x7fc0e0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 >> 172.21.15.104:0/3347309933 conn(0x7fc0e006b8a0 msgr2=0x7fc0e00ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 shutdown_connections 2023-06-10T18:51:31.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:31.854+0000 7fc0e7c1e700 1 -- 172.21.15.104:0/3347309933 wait complete. 2023-06-10T18:51:31.865 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:51:31.871 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:51:31.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:31.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:31.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[110297]: fuse finished with error 0 and tester_r 0 2023-06-10T18:51:31.913 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:51:31.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:31.913 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:51:32.023 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:51:32.023 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:32.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:51:32.041 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:51:32.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:32.042 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:51:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:51:32.098 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:51:32.098 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:51:32.099 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:32.118 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:51:32.118 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:51:32.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:32.119 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:51:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:51:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:51:32.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:32.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.291+0000 7fafc2266700 1 Processor -- start 2023-06-10T18:51:32.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.291+0000 7fafc2266700 1 -- start start 2023-06-10T18:51:32.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc2266700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 0x7fafbc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc2266700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc06d290 con 0x7fafbc069e70 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc2266700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc0ca2d0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc2266700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc071640 con 0x7fafbc06e0e0 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc0803700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fafbc069e70 0x7fafbc0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58154/0 (socket says 172.21.15.104:58154) 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafc0803700 1 -- 172.21.15.104:0/3362613533 learned_addr learned my addr 172.21.15.104:0/3362613533 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbbfff700 1 --2- 172.21.15.104:0/3362613533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 0x7fafbc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642754754 0 0) 0x7fafbc06d290 con 0x7fafbc069e70 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa40035b0 con 0x7fafbc069e70 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547944912 0 0) 0x7fafbc0ca2d0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc06d290 con 0x7fafbc06e4b0 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbbfff700 1 -- 172.21.15.104:0/3362613533 >> v1:172.21.15.114:6789/0 conn(0x7fafbc06e4b0 legacy=0x7fafbc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbbfff700 1 -- 172.21.15.104:0/3362613533 >> v1:172.21.15.104:6789/0 conn(0x7fafbc069e70 legacy=0x7fafbc0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.292+0000 7fafbbfff700 1 -- 172.21.15.104:0/3362613533 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc06d4c0 con 0x7fafbc06e0e0 2023-06-10T18:51:32.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafbbfff700 1 --2- 172.21.15.104:0/3362613533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 0x7fafbc06cd50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009850 tx=0x7fafb0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a297a9a09d525a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0005e30 con 0x7fafbc06e0e0 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb0006030 con 0x7fafbc06e0e0 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafbaffd700 1 -- 172.21.15.104:0/3362613533 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00063e0 con 0x7fafbc06e0e0 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 -- 172.21.15.104:0/3362613533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 msgr2=0x7fafbc06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 --2- 172.21.15.104:0/3362613533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 0x7fafbc06cd50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009850 tx=0x7fafb0009b60 comp rx=0 tx=0).stop 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 -- 172.21.15.104:0/3362613533 shutdown_connections 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 --2- 172.21.15.104:0/3362613533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc06e0e0 0x7fafbc06cd50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 -- 172.21.15.104:0/3362613533 >> 172.21.15.104:0/3362613533 conn(0x7fafbc06b8a0 msgr2=0x7fafbc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:32.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 -- 172.21.15.104:0/3362613533 shutdown_connections 2023-06-10T18:51:32.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.293+0000 7fafc2266700 1 -- 172.21.15.104:0/3362613533 wait complete. 2023-06-10T18:51:32.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.295+0000 7fafc2266700 1 Processor -- start 2023-06-10T18:51:32.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.295+0000 7fafc2266700 1 -- start start 2023-06-10T18:51:32.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 0x7fafbc0b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 0x7fafbc155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0aecd0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0afaf0 con 0x7fafbc06e0e0 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc2266700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1560b0 con 0x7fafbc069e70 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc0803700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc0803700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60332/0 (socket says 172.21.15.104:60332) 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafc0803700 1 -- 172.21.15.104:0/947833760 learned_addr learned my addr 172.21.15.104:0/947833760 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:32.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafbb7fe700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 0x7fafbc155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.296+0000 7fafbbfff700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 0x7fafbc0b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 -- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 msgr2=0x7fafbc0b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 0x7fafbc0b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 -- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 msgr2=0x7fafbc155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 0x7fafbc155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 -- 172.21.15.104:0/947833760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0009430 con 0x7fafbc06e4b0 2023-06-10T18:51:32.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafbb7fe700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 0x7fafbc155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafbbfff700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 0x7fafbc0b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc0803700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fafb400cb20 tx=0x7fafb400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400aac0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb400ac20 con 0x7fafbc06e4b0 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4012400 con 0x7fafbc06e4b0 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc0aeee0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.297+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc0af4a0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.298+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc074050 con 0x7fafbc06e4b0 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.298+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fafb40113a0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.299+0000 7fafb8ff9700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 0x7fafa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.299+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fafb4089450 con 0x7fafbc06e4b0 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.299+0000 7fafbbfff700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 0x7fafa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.299+0000 7fafbbfff700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 0x7fafa806d190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fafb0000c00 tx=0x7fafb0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.303+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fafb40542c0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.531+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafbc0afc30 con 0x7fafbc06e4b0 2023-06-10T18:51:32.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.531+0000 7fafb8ff9700 1 -- 172.21.15.104:0/947833760 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fafb40578e0 con 0x7fafbc06e4b0 2023-06-10T18:51:32.537 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:32.538 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:51:32.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 msgr2=0x7fafa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 0x7fafa806d190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fafb0000c00 tx=0x7fafb0008070 comp rx=0 tx=0).stop 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 msgr2=0x7fafbc0cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fafb400cb20 tx=0x7fafb400cee0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 shutdown_connections 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafbc06e4b0 0x7fafbc0cf0e0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fafbc069e70 0x7fafbc0b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.533+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafa806ace0 0x7fafa806d190 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.534+0000 7fafc2266700 1 --2- 172.21.15.104:0/947833760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc06e0e0 0x7fafbc155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.534+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 >> 172.21.15.104:0/947833760 conn(0x7fafbc06b8a0 msgr2=0x7fafbc0c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:32.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.534+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 shutdown_connections 2023-06-10T18:51:32.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.534+0000 7fafc2266700 1 -- 172.21.15.104:0/947833760 wait complete. 2023-06-10T18:51:32.541 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:51:32.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:32.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.665+0000 7feffd7a9700 1 Processor -- start 2023-06-10T18:51:32.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.665+0000 7feffd7a9700 1 -- start start 2023-06-10T18:51:32.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.666+0000 7feffd7a9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feff80cd850 0x7feff806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.666+0000 7feffd7a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.666+0000 7feffd7a9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80c9e60 con 0x7feff80aeaa0 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.666+0000 7feffd7a9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff80c9bc0 con 0x7feff80cd850 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.666+0000 7feffd7a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff80aee70 con 0x7feff80cb300 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feff80cd850 0x7feff806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feff80cd850 0x7feff806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60336/0 (socket says 172.21.15.104:60336) 2023-06-10T18:51:32.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feff80aeaa0 0x7feff80cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58164/0 (socket says 172.21.15.104:58164) 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 -- 172.21.15.104:0/527415422 learned_addr learned my addr 172.21.15.104:0/527415422 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff77fe700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 -- 172.21.15.104:0/527415422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 msgr2=0x7feff806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff806d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 -- 172.21.15.104:0/527415422 >> v1:172.21.15.104:6789/0 conn(0x7feff80aeaa0 legacy=0x7feff80cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 -- 172.21.15.104:0/527415422 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff806e540 con 0x7feff80cd850 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff5ffb700 1 -- 172.21.15.104:0/527415422 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282437723 0 0) 0x7feff80c9e60 con 0x7feff80aeaa0 2023-06-10T18:51:32.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff67fc700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff806cd50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fefe80098f0 tx=0x7fefe8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45e8dad25d7d4ec server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff5ffb700 1 -- 172.21.15.104:0/527415422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800b8a0 con 0x7feff80cd850 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff5ffb700 1 -- 172.21.15.104:0/527415422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe800ba00 con 0x7feff80cd850 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.667+0000 7feff5ffb700 1 -- 172.21.15.104:0/527415422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800bd90 con 0x7feff80cd850 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 -- 172.21.15.104:0/527415422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 msgr2=0x7feff806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff806cd50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fefe80098f0 tx=0x7fefe8009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 -- 172.21.15.104:0/527415422 shutdown_connections 2023-06-10T18:51:32.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 --2- 172.21.15.104:0/527415422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff806cd50 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 -- 172.21.15.104:0/527415422 >> 172.21.15.104:0/527415422 conn(0x7feff806b8a0 msgr2=0x7feff80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:32.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 -- 172.21.15.104:0/527415422 shutdown_connections 2023-06-10T18:51:32.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.668+0000 7feffd7a9700 1 -- 172.21.15.104:0/527415422 wait complete. 2023-06-10T18:51:32.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.669+0000 7feffd7a9700 1 Processor -- start 2023-06-10T18:51:32.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.669+0000 7feffd7a9700 1 -- start start 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff80aeaa0 0x7feff80c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff80b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c83c0 con 0x7feff80cd850 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c86a0 con 0x7feff80aeaa0 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feffd7a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c8980 con 0x7feff80cb300 2023-06-10T18:51:32.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60346/0 (socket says 172.21.15.104:60346) 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 -- 172.21.15.104:0/2347012391 learned_addr learned my addr 172.21.15.104:0/2347012391 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff67fc700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff80b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff6ffd700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff80aeaa0 0x7feff80c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 -- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 msgr2=0x7feff80b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff80b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 -- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff80aeaa0 msgr2=0x7feff80c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff80aeaa0 0x7feff80c22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 -- 172.21.15.104:0/2347012391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8009550 con 0x7feff80cd850 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7feff77fe700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7feff000cd70 tx=0x7feff000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003a20 con 0x7feff80cd850 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0003b80 con 0x7feff80cd850 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.670+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0004020 con 0x7feff80cd850 2023-06-10T18:51:32.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.671+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff80c8c60 con 0x7feff80cd850 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.671+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff80c2820 con 0x7feff80cd850 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.671+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff8074050 con 0x7feff80cd850 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.672+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feff0004180 con 0x7feff80cd850 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.672+0000 7fefdf7fe700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 0x7fefe406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.672+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7feff0012070 con 0x7feff80cd850 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.673+0000 7feff6ffd700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 0x7fefe406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:32.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.673+0000 7feff6ffd700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 0x7fefe406d190 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fefec0018e0 tx=0x7fefec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:32.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.677+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feff0054c50 con 0x7feff80cd850 2023-06-10T18:51:32.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.864+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feff806df10 con 0x7feff80cd850 2023-06-10T18:51:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.866+0000 7fefdf7fe700 1 -- 172.21.15.104:0/2347012391 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18760 (secure 0 0 0) 0x7feff0058270 con 0x7feff80cd850 2023-06-10T18:51:32.871 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:32.872 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":71,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:50:54.764730+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.a","mappings":[{"from":6,"to":0}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/2562980824":"2023-06-10T19:50:53.863403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 msgr2=0x7fefe406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 0x7fefe406d190 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fefec0018e0 tx=0x7fefec009040 comp rx=0 tx=0).stop 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 msgr2=0x7feff80b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7feff000cd70 tx=0x7feff000a570 comp rx=0 tx=0).stop 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 shutdown_connections 2023-06-10T18:51:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe406ace0 0x7fefe406d190 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff80aeaa0 0x7feff80c22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff80cd850 0x7feff80b2a50 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 --2- 172.21.15.104:0/2347012391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff80cb300 0x7feff80b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 >> 172.21.15.104:0/2347012391 conn(0x7feff806b8a0 msgr2=0x7feff806a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 shutdown_connections 2023-06-10T18:51:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:32.868+0000 7feffd7a9700 1 -- 172.21.15.104:0/2347012391 wait complete. 2023-06-10T18:51:32.886 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:48:42.072415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:48:43.111638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:51:32.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:51:33.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.002+0000 7f10c7b33700 1 Processor -- start 2023-06-10T18:51:33.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 -- start start 2023-06-10T18:51:33.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10c015c6f0 0x7f10c00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00ae7e0 con 0x7f10c00aec00 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c006df90 con 0x7f10c015c6f0 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.003+0000 7f10c7b33700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c006e270 con 0x7f10c006ea70 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10c015c6f0 0x7f10c00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10c015c6f0 0x7f10c00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60350/0 (socket says 172.21.15.104:60350) 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 -- 172.21.15.104:0/764921868 learned_addr learned my addr 172.21.15.104:0/764921868 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:33.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c50ce700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 -- 172.21.15.104:0/764921868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 msgr2=0x7f10c00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 -- 172.21.15.104:0/764921868 >> v1:172.21.15.114:6789/0 conn(0x7f10c00aec00 legacy=0x7f10c00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 -- 172.21.15.104:0/764921868 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c006e550 con 0x7f10c015c6f0 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c58cf700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c015c6f0 0x7f10c00b1070 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3ee0cd6b39985c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c48cd700 1 -- 172.21.15.104:0/764921868 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0005e30 con 0x7f10c015c6f0 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c48cd700 1 -- 172.21.15.104:0/764921868 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0004030 con 0x7f10c015c6f0 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.004+0000 7f10c48cd700 1 -- 172.21.15.104:0/764921868 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b00043e0 con 0x7f10c015c6f0 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 -- 172.21.15.104:0/764921868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c015c6f0 msgr2=0x7f10c00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c015c6f0 0x7f10c00b1070 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 -- 172.21.15.104:0/764921868 shutdown_connections 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c015c6f0 0x7f10c00b1070 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 --2- 172.21.15.104:0/764921868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 -- 172.21.15.104:0/764921868 >> 172.21.15.104:0/764921868 conn(0x7f10c006b8a0 msgr2=0x7f10c00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 -- 172.21.15.104:0/764921868 shutdown_connections 2023-06-10T18:51:33.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.005+0000 7f10c7b33700 1 -- 172.21.15.104:0/764921868 wait complete. 2023-06-10T18:51:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.006+0000 7f10c7b33700 1 Processor -- start 2023-06-10T18:51:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.006+0000 7f10c7b33700 1 -- start start 2023-06-10T18:51:33.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00c2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10c015c6f0 0x7f10c006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c0069e70 con 0x7f10c00aec00 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00b4b20 con 0x7f10c015c6f0 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00b4e00 con 0x7f10c006ea70 2023-06-10T18:51:33.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60358/0 (socket says 172.21.15.104:60358) 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 -- 172.21.15.104:0/1602323600 learned_addr learned my addr 172.21.15.104:0/1602323600 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c58cf700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c60d0700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10c015c6f0 0x7f10c006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 -- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 msgr2=0x7f10c00c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00c2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 -- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10c015c6f0 msgr2=0x7f10c006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10c015c6f0 0x7f10c006a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 -- 172.21.15.104:0/1602323600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b0009500 con 0x7f10c00aec00 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c50ce700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e810 tx=0x7f10bc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc013610 con 0x7f10c00aec00 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc013770 con 0x7f10c00aec00 2023-06-10T18:51:33.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c00b50e0 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.007+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc013b00 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.008+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c00c2210 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.008+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c006a2c0 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.009+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f10bc013c60 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.009+0000 7f10b67fc700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 0x7f10a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.009+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f10bc089230 con 0x7f10c00aec00 2023-06-10T18:51:33.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.009+0000 7f10c58cf700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 0x7f10a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.010+0000 7f10c58cf700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 0x7f10a406d190 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f10b0014040 tx=0x7f10b0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.013+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10c006a2c0 con 0x7f10c00aec00 2023-06-10T18:51:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.200+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10c006a2c0 con 0x7f10c00aec00 2023-06-10T18:51:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.201+0000 7f10b67fc700 1 -- 172.21.15.104:0/1602323600 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10c006a2c0 con 0x7f10c00aec00 2023-06-10T18:51:33.206 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:51:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.202+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 msgr2=0x7f10a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.202+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 0x7f10a406d190 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f10b0014040 tx=0x7f10b0008070 comp rx=0 tx=0).stop 2023-06-10T18:51:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.202+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 msgr2=0x7f10c015ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.202+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e810 tx=0x7f10bc00ebd0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 shutdown_connections 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f10a406ace0 0x7f10a406d190 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10c015c6f0 0x7f10c006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10c006ea70 0x7f10c00c2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 --2- 172.21.15.104:0/1602323600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10c00aec00 0x7f10c015ab30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 >> 172.21.15.104:0/1602323600 conn(0x7f10c006b8a0 msgr2=0x7f10c015b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 shutdown_connections 2023-06-10T18:51:33.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.203+0000 7f10c7b33700 1 -- 172.21.15.104:0/1602323600 wait complete. 2023-06-10T18:51:33.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:51:33.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.337+0000 7f9e2761b700 1 Processor -- start 2023-06-10T18:51:33.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.337+0000 7f9e2761b700 1 -- start start 2023-06-10T18:51:33.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e2761b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e2015c730 0x7f9e200cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e2761b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e200ae230 0x7f9e2006d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e2761b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2015cb00 con 0x7f9e200cd720 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e2761b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e200cdb90 con 0x7f9e200ae230 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e2761b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200acd90 con 0x7f9e2015c730 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e200ae230 0x7f9e2006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e24bb6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9e200cd720 0x7f9e2006cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58192/0 (socket says 172.21.15.104:58192) 2023-06-10T18:51:33.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e200ae230 0x7f9e2006d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60368/0 (socket says 172.21.15.104:60368) 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e24bb6700 1 -- 172.21.15.104:0/818924882 learned_addr learned my addr 172.21.15.104:0/818924882 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e253b7700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e2015c730 0x7f9e200cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 -- 172.21.15.104:0/818924882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e2015c730 msgr2=0x7f9e200cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e2015c730 0x7f9e200cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 -- 172.21.15.104:0/818924882 >> v1:172.21.15.104:6789/0 conn(0x7f9e200cd720 legacy=0x7f9e2006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 -- 172.21.15.104:0/818924882 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200ac460 con 0x7f9e200ae230 2023-06-10T18:51:33.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e17fff700 1 -- 172.21.15.104:0/818924882 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283656241 0 0) 0x7f9e10001110 con 0x7f9e200cd720 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e25bb8700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e200ae230 0x7f9e2006d480 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84474905b6fb3f51 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e17fff700 1 -- 172.21.15.104:0/818924882 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1800e070 con 0x7f9e200ae230 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e17fff700 1 -- 172.21.15.104:0/818924882 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1800bbd0 con 0x7f9e200ae230 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.338+0000 7f9e17fff700 1 -- 172.21.15.104:0/818924882 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18012430 con 0x7f9e200ae230 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 -- 172.21.15.104:0/818924882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e200ae230 msgr2=0x7f9e2006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e200ae230 0x7f9e2006d480 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009d90 tx=0x7f9e1800b330 comp rx=0 tx=0).stop 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 -- 172.21.15.104:0/818924882 shutdown_connections 2023-06-10T18:51:33.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e200ae230 0x7f9e2006d480 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 --2- 172.21.15.104:0/818924882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e2015c730 0x7f9e200cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 -- 172.21.15.104:0/818924882 >> 172.21.15.104:0/818924882 conn(0x7f9e2006b8a0 msgr2=0x7f9e200cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 -- 172.21.15.104:0/818924882 shutdown_connections 2023-06-10T18:51:33.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.339+0000 7f9e2761b700 1 -- 172.21.15.104:0/818924882 wait complete. 2023-06-10T18:51:33.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.340+0000 7f9e2761b700 1 Processor -- start 2023-06-10T18:51:33.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.340+0000 7f9e2761b700 1 -- start start 2023-06-10T18:51:33.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 0x7f9e201f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e200cd720 0x7f9e2014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200c96c0 con 0x7f9e2015c730 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200b3600 con 0x7f9e200cd720 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e2761b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e201f2ec0 con 0x7f9e200ae230 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60372/0 (socket says 172.21.15.104:60372) 2023-06-10T18:51:33.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 -- 172.21.15.104:0/3733754790 learned_addr learned my addr 172.21.15.104:0/3733754790 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e253b7700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 0x7f9e201f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e24bb6700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e200cd720 0x7f9e2014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 -- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 msgr2=0x7f9e201f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 0x7f9e201f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 -- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e200cd720 msgr2=0x7f9e2014ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e200cd720 0x7f9e2014ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e25bb8700 1 -- 172.21.15.104:0/3733754790 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18009a70 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.341+0000 7f9e253b7700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 0x7f9e201f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e25bb8700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9e200b2200 tx=0x7f9e18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18012430 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18020070 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18004280 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e200c93e0 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.342+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e200b38b0 con 0x7f9e2015c730 2023-06-10T18:51:33.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.343+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e20066f10 con 0x7f9e2015c730 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.343+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9e180043e0 con 0x7f9e2015c730 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.344+0000 7f9e15ffb700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 0x7f9e0406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.344+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9e18022070 con 0x7f9e2015c730 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.344+0000 7f9e253b7700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 0x7f9e0406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.344+0000 7f9e253b7700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 0x7f9e0406d190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e200d0260 tx=0x7f9e1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.348+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9e18056400 con 0x7f9e2015c730 2023-06-10T18:51:33.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.534+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9e20074050 con 0x7f9e2015c730 2023-06-10T18:51:33.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.534+0000 7f9e15ffb700 1 -- 172.21.15.104:0/3733754790 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9e18059a20 con 0x7f9e2015c730 2023-06-10T18:51:33.540 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:51:33.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 msgr2=0x7f9e0406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 0x7f9e0406d190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e200d0260 tx=0x7f9e1c008040 comp rx=0 tx=0).stop 2023-06-10T18:51:33.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 msgr2=0x7f9e200c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9e200b2200 tx=0x7f9e18004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 shutdown_connections 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e2015c730 0x7f9e200c9c40 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e200ae230 0x7f9e201f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e0406ace0 0x7f9e0406d190 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 --2- 172.21.15.104:0/3733754790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e200cd720 0x7f9e2014ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 >> 172.21.15.104:0/3733754790 conn(0x7f9e2006b8a0 msgr2=0x7f9e20072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 shutdown_connections 2023-06-10T18:51:33.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.536+0000 7f9e2761b700 1 -- 172.21.15.104:0/3733754790 wait complete. 2023-06-10T18:51:33.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:51:33.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.671+0000 7fa3c67ab700 1 Processor -- start 2023-06-10T18:51:33.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.671+0000 7fa3c67ab700 1 -- start start 2023-06-10T18:51:33.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c67ab700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 0x7fa3c006a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c67ab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3c006dcd0 0x7fa3c006e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c67ab700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c006e650 con 0x7fa3c015c500 2023-06-10T18:51:33.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c67ab700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c00aef20 con 0x7fa3c006dcd0 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c67ab700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c015c8d0 con 0x7fa3c0069e70 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c4d48700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3c006dcd0 0x7fa3c006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c4d48700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3c006dcd0 0x7fa3c006e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60382/0 (socket says 172.21.15.104:60382) 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3c4d48700 1 -- 172.21.15.104:0/1138556691 learned_addr learned my addr 172.21.15.104:0/1138556691 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.672+0000 7fa3bf7fe700 1 --2- 172.21.15.104:0/1138556691 >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 0x7fa3c006a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c4d48700 1 -- 172.21.15.104:0/1138556691 >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 msgr2=0x7fa3c006a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:33.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c4d48700 1 --2- 172.21.15.104:0/1138556691 >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 0x7fa3c006a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c4d48700 1 -- 172.21.15.104:0/1138556691 >> v1:172.21.15.114:6789/0 conn(0x7fa3c015c500 legacy=0x7fa3c00cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c4d48700 1 -- 172.21.15.104:0/1138556691 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c006abd0 con 0x7fa3c006dcd0 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3bf7fe700 1 --2- 172.21.15.104:0/1138556691 >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 0x7fa3c006a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c4d48700 1 --2- 172.21.15.104:0/1138556691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c006dcd0 0x7fa3c006e110 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009e50 tx=0x7fa3b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787151684d6bc57f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3beffd700 1 -- 172.21.15.104:0/1138556691 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b800e070 con 0x7fa3c006dcd0 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3beffd700 1 -- 172.21.15.104:0/1138556691 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b800b980 con 0x7fa3c006dcd0 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3beffd700 1 -- 172.21.15.104:0/1138556691 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b800bd30 con 0x7fa3c006dcd0 2023-06-10T18:51:33.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 -- 172.21.15.104:0/1138556691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c006dcd0 msgr2=0x7fa3c006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/1138556691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c006dcd0 0x7fa3c006e110 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009e50 tx=0x7fa3b800b330 comp rx=0 tx=0).stop 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 -- 172.21.15.104:0/1138556691 shutdown_connections 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/1138556691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c006dcd0 0x7fa3c006e110 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/1138556691 >> v2:172.21.15.114:3300/0 conn(0x7fa3c0069e70 0x7fa3c006a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.673+0000 7fa3c67ab700 1 -- 172.21.15.104:0/1138556691 >> 172.21.15.104:0/1138556691 conn(0x7fa3c006b8a0 msgr2=0x7fa3c00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.674+0000 7fa3c67ab700 1 -- 172.21.15.104:0/1138556691 shutdown_connections 2023-06-10T18:51:33.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.674+0000 7fa3c67ab700 1 -- 172.21.15.104:0/1138556691 wait complete. 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.675+0000 7fa3c67ab700 1 Processor -- start 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.675+0000 7fa3c67ab700 1 -- start start 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3c006dcd0 0x7fa3c014b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 0x7fa3c00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00ca1b0 con 0x7fa3c0069e70 2023-06-10T18:51:33.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00c8690 con 0x7fa3c006dcd0 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00ccdb0 con 0x7fa3c015c500 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60384/0 (socket says 172.21.15.104:60384) 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 -- 172.21.15.104:0/3783632932 learned_addr learned my addr 172.21.15.104:0/3783632932 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bf7fe700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3c006dcd0 0x7fa3c014b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c4d48700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 0x7fa3c00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 -- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 msgr2=0x7fa3c00b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 0x7fa3c00b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 -- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3c006dcd0 msgr2=0x7fa3c014b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3c006dcd0 0x7fa3c014b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 -- 172.21.15.104:0/3783632932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009b30 con 0x7fa3c0069e70 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c4d48700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 0x7fa3c00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bffff700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c460 tx=0x7fa3a800c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80052d0 con 0x7fa3c0069e70 2023-06-10T18:51:33.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a8005450 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c014b7c0 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8006af0 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.676+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c015b170 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.677+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c006e440 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.678+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3a80036d0 con 0x7fa3c0069e70 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.678+0000 7fa3bcff9700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 0x7fa3a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:33.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.678+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fa3a80888b0 con 0x7fa3c0069e70 2023-06-10T18:51:33.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.678+0000 7fa3bf7fe700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 0x7fa3a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:33.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.679+0000 7fa3bf7fe700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 0x7fa3a006d190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa3c006d5a0 tx=0x7fa3b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:33.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.683+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3a8053720 con 0x7fa3c0069e70 2023-06-10T18:51:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.869+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3c0074050 con 0x7fa3c0069e70 2023-06-10T18:51:33.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.869+0000 7fa3bcff9700 1 -- 172.21.15.104:0/3783632932 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3a8056d40 con 0x7fa3c0069e70 2023-06-10T18:51:33.875 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:51:33.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 msgr2=0x7fa3a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 0x7fa3a006d190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa3c006d5a0 tx=0x7fa3b0008040 comp rx=0 tx=0).stop 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 msgr2=0x7fa3c00cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c460 tx=0x7fa3a800c770 comp rx=0 tx=0).stop 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 shutdown_connections 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3c015c500 0x7fa3c00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3c0069e70 0x7fa3c00cc870 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3a006ace0 0x7fa3a006d190 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 --2- 172.21.15.104:0/3783632932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3c006dcd0 0x7fa3c014b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 >> 172.21.15.104:0/3783632932 conn(0x7fa3c006b8a0 msgr2=0x7fa3c00af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 shutdown_connections 2023-06-10T18:51:33.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:33.871+0000 7fa3c67ab700 1 -- 172.21.15.104:0/3783632932 wait complete. 2023-06-10T18:51:33.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:51:34.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.005+0000 7f7a95351700 1 Processor -- start 2023-06-10T18:51:34.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.005+0000 7f7a95351700 1 -- start start 2023-06-10T18:51:34.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a95351700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a900d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a95351700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900af530 con 0x7f7a900d25c0 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a95351700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900af810 con 0x7f7a900b05d0 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a95351700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900afaf0 con 0x7f7a900d04f0 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8e7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7a900d25c0 0x7f7a900d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58220/0 (socket says 172.21.15.104:58220) 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8e7fc700 1 -- 172.21.15.104:0/3241202362 learned_addr learned my addr 172.21.15.104:0/3241202362 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8effd700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a900d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581954988 0 0) 0x7f7a900af530 con 0x7f7a900d25c0 2023-06-10T18:51:34.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a780035b0 con 0x7f7a900d25c0 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348426362 0 0) 0x7f7a780035b0 con 0x7f7a900d25c0 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 msgr2=0x7f7a900d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a900d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 >> v1:172.21.15.114:6789/0 conn(0x7f7a900b05d0 legacy=0x7f7a900b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900afdd0 con 0x7f7a900d25c0 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7a80001670 con 0x7f7a900d25c0 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 >> v1:172.21.15.104:6789/0 conn(0x7f7a900d25c0 legacy=0x7f7a900d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a78003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a78004d90 0x7f7a78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900afaf0 con 0x7f7a78003fb0 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7800bba0 con 0x7f7a78004d90 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.006+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7800bd60 con 0x7f7a900d04f0 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8effd700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a78003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8f7fe700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a78004d90 0x7f7a78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 -- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 msgr2=0x7f7a78003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900d04f0 0x7f7a78003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 -- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a78004d90 msgr2=0x7f7a78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a78004d90 0x7f7a78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 -- 172.21.15.104:0/3241202362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a780035b0 con 0x7f7a78003fb0 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a78004850 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c210 tx=0x7f7a78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1388fe563fc01e32 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80007c10 con 0x7f7a78003fb0 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8000cdb0 con 0x7f7a78003fb0 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.007+0000 7f7a8dffb700 1 -- 172.21.15.104:0/3241202362 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80005ba0 con 0x7f7a78003fb0 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 -- 172.21.15.104:0/3241202362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 msgr2=0x7f7a78004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 --2- 172.21.15.104:0/3241202362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a78004850 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c210 tx=0x7f7a78005880 comp rx=0 tx=0).stop 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a8f7fe700 1 -- 172.21.15.104:0/3241202362 reap_dead start 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 -- 172.21.15.104:0/3241202362 shutdown_connections 2023-06-10T18:51:34.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 -- 172.21.15.104:0/3241202362 >> 172.21.15.104:0/3241202362 conn(0x7f7a90071ec0 msgr2=0x7f7a9009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 -- 172.21.15.104:0/3241202362 shutdown_connections 2023-06-10T18:51:34.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.008+0000 7f7a95351700 1 -- 172.21.15.104:0/3241202362 wait complete. 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.009+0000 7f7a95351700 1 Processor -- start 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.009+0000 7f7a95351700 1 -- start start 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900b7230 0x7f7a90158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a90159290 0x7f7a90157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90159700 con 0x7f7a78003fb0 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900b8420 con 0x7f7a90159290 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900b5ba0 con 0x7f7a900b7230 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60404/0 (socket says 172.21.15.104:60404) 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 -- 172.21.15.104:0/1946246996 learned_addr learned my addr 172.21.15.104:0/1946246996 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8f7fe700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a90159290 0x7f7a90157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900b7230 0x7f7a90158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 -- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900b7230 msgr2=0x7f7a90158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900b7230 0x7f7a90158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 -- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a90159290 msgr2=0x7f7a90157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a90159290 0x7f7a90157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 -- 172.21.15.104:0/1946246996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900b5e80 con 0x7f7a78003fb0 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a8effd700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7a840119f0 tx=0x7f7a84011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84005c30 con 0x7f7a78003fb0 2023-06-10T18:51:34.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84005d90 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a900b6160 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8400f950 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.010+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a900b6720 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.011+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a900b68e0 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.012+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a8400fab0 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.012+0000 7f7a777fe700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.012+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f7a84089e80 con 0x7f7a78003fb0 2023-06-10T18:51:34.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.012+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.013+0000 7f7a8e7fc700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7a80000c00 tx=0x7f7a80007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.017+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a84054e10 con 0x7f7a78003fb0 2023-06-10T18:51:34.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.202+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a90067010 con 0x7f7a78003fb0 2023-06-10T18:51:34.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.202+0000 7f7a777fe700 1 -- 172.21.15.104:0/1946246996 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a84058430 con 0x7f7a78003fb0 2023-06-10T18:51:34.208 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:51:34.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 msgr2=0x7f7a7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7a80000c00 tx=0x7f7a80007000 comp rx=0 tx=0).stop 2023-06-10T18:51:34.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 msgr2=0x7f7a901f46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7a840119f0 tx=0x7f7a84011db0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 shutdown_connections 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a90159290 0x7f7a90157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78003fb0 0x7f7a901f46c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 --2- 172.21.15.104:0/1946246996 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a900b7230 0x7f7a90158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 >> 172.21.15.104:0/1946246996 conn(0x7f7a90071ec0 msgr2=0x7f7a9015a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.204+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 shutdown_connections 2023-06-10T18:51:34.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.205+0000 7f7a95351700 1 -- 172.21.15.104:0/1946246996 wait complete. 2023-06-10T18:51:34.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.336+0000 7ff2dd9e5700 1 Processor -- start 2023-06-10T18:51:34.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.336+0000 7ff2dd9e5700 1 -- start start 2023-06-10T18:51:34.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2dd9e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2d80d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2dd9e5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2d80d0a30 0x7ff2d806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2dd9e5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d806bfd0 con 0x7ff2d806b820 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2dd9e5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d80d0ea0 con 0x7ff2d80d0a30 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2dd9e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d806c2b0 con 0x7ff2d80b1e30 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff2d806b820 0x7ff2d80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58244/0 (socket says 172.21.15.104:58244) 2023-06-10T18:51:34.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d77fe700 1 -- 172.21.15.104:0/1259371653 learned_addr learned my addr 172.21.15.104:0/1259371653 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d6ffd700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2d80d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936523938 0 0) 0x7ff2d806bfd0 con 0x7ff2d806b820 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/1259371653 >> v2:172.21.15.114:3300/0 conn(0x7ff2d80d0a30 0x7ff2d806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.337+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c00035b0 con 0x7ff2d806b820 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2764854905 0 0) 0x7ff2c00035b0 con 0x7ff2d806b820 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 msgr2=0x7ff2d80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2d80d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 >> v2:172.21.15.114:3300/0 conn(0x7ff2d80d0a30 msgr2=0x7ff2d806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 --2- 172.21.15.104:0/1259371653 >> v2:172.21.15.114:3300/0 conn(0x7ff2d80d0a30 0x7ff2d806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80ccf70 con 0x7ff2d806b820 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff2d00019f0 con 0x7ff2d806b820 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 >> v1:172.21.15.104:6789/0 conn(0x7ff2d806b820 legacy=0x7ff2d80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c0003a00 0x7ff2c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2c00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2c0005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80d0ea0 con 0x7ff2c0004430 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d806c2b0 con 0x7ff2c0003a00 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c000ba10 con 0x7ff2d80b1e30 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2c00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d6ffd700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c0003a00 0x7ff2c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d77fe700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2c0005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 -- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 msgr2=0x7ff2c0005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d80b1e30 0x7ff2c0005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 -- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c0003a00 msgr2=0x7ff2c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c0003a00 0x7ff2c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 -- 172.21.15.104:0/1259371653 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c00035b0 con 0x7ff2c0004430 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2c00048f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009b90 tx=0x7ff2c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc46645c71b7268d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800e6c0 con 0x7ff2c0004430 2023-06-10T18:51:34.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c800e820 con 0x7ff2c0004430 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.338+0000 7ff2d5ffb700 1 -- 172.21.15.104:0/1259371653 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800eb90 con 0x7ff2c0004430 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/1259371653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 msgr2=0x7ff2c00048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/1259371653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2c00048f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009b90 tx=0x7ff2c0005880 comp rx=0 tx=0).stop 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2d77fe700 1 -- 172.21.15.104:0/1259371653 reap_dead start 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/1259371653 shutdown_connections 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/1259371653 >> 172.21.15.104:0/1259371653 conn(0x7ff2d8071ec0 msgr2=0x7ff2d809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/1259371653 shutdown_connections 2023-06-10T18:51:34.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.339+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/1259371653 wait complete. 2023-06-10T18:51:34.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.340+0000 7ff2dd9e5700 1 Processor -- start 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.340+0000 7ff2dd9e5700 1 -- start start 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 0x7ff2d80caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 0x7ff2d80cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80b5960 con 0x7ff2c0004430 2023-06-10T18:51:34.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80cb9b0 con 0x7ff2d80cb000 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2dd9e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d815dc30 con 0x7ff2d815d400 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60424/0 (socket says 172.21.15.104:60424) 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 -- 172.21.15.104:0/4263325569 learned_addr learned my addr 172.21.15.104:0/4263325569 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d77fe700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 0x7ff2d80cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 0x7ff2d80caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 -- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 msgr2=0x7ff2d80caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 0x7ff2d80caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 -- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 msgr2=0x7ff2d80cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 0x7ff2d80cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d6ffd700 1 -- 172.21.15.104:0/4263325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d815df10 con 0x7ff2c0004430 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d77fe700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 0x7ff2d80cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.341+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 0x7ff2d80caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:34.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2d6ffd700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc011aa0 tx=0x7ff2cc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00f980 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc00fae0 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d815e1f0 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00fe90 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.342+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d815d870 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.343+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d80d1860 con 0x7ff2c0004430 2023-06-10T18:51:34.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.343+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff2cc014440 con 0x7ff2c0004430 2023-06-10T18:51:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.343+0000 7ff2bffff700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 0x7ff2c406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.343+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7ff2cc089d20 con 0x7ff2c0004430 2023-06-10T18:51:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.344+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 0x7ff2c406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.344+0000 7ff2d67fc700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 0x7ff2c406d1c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80d0000 tx=0x7ff2c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.348+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff2cc054ce0 con 0x7ff2c0004430 2023-06-10T18:51:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.573+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2d815e430 con 0x7ff2c0004430 2023-06-10T18:51:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.574+0000 7ff2bffff700 1 -- 172.21.15.104:0/4263325569 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7ff2d815e430 con 0x7ff2c0004430 2023-06-10T18:51:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:34.580 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:51:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 msgr2=0x7ff2c406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 0x7ff2c406d1c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80d0000 tx=0x7ff2c8008030 comp rx=0 tx=0).stop 2023-06-10T18:51:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 msgr2=0x7ff2d80cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc011aa0 tx=0x7ff2cc011e60 comp rx=0 tx=0).stop 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 shutdown_connections 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2c406ad10 0x7ff2c406d1c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2d80cb000 0x7ff2d80cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2c0004430 0x7ff2d80cbe80 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 --2- 172.21.15.104:0/4263325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2d815d400 0x7ff2d80caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 >> 172.21.15.104:0/4263325569 conn(0x7ff2d8071ec0 msgr2=0x7ff2d80b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 shutdown_connections 2023-06-10T18:51:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.576+0000 7ff2dd9e5700 1 -- 172.21.15.104:0/4263325569 wait complete. 2023-06-10T18:51:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:51:34.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.707+0000 7eff6d9e3700 1 Processor -- start 2023-06-10T18:51:34.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.707+0000 7eff6d9e3700 1 -- start start 2023-06-10T18:51:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff6d9e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff680d0c00 0x7eff6806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff6d9e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff6806b820 0x7eff680cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff6d9e3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6806bfd0 con 0x7eff680b1e30 2023-06-10T18:51:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff6d9e3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6806c2b0 con 0x7eff6806b820 2023-06-10T18:51:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff6d9e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff680ccf70 con 0x7eff680d0c00 2023-06-10T18:51:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff6806b820 0x7eff680cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff66ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7eff680b1e30 0x7eff680d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58260/0 (socket says 172.21.15.104:58260) 2023-06-10T18:51:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff66ffd700 1 -- 172.21.15.104:0/2046886571 learned_addr learned my addr 172.21.15.104:0/2046886571 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff667fc700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff680d0c00 0x7eff6806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 -- 172.21.15.104:0/2046886571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff680d0c00 msgr2=0x7eff6806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff680d0c00 0x7eff6806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 -- 172.21.15.104:0/2046886571 >> v1:172.21.15.104:6789/0 conn(0x7eff680b1e30 legacy=0x7eff680d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 -- 172.21.15.104:0/2046886571 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680cd250 con 0x7eff6806b820 2023-06-10T18:51:34.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff65ffb700 1 -- 172.21.15.104:0/2046886571 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323029988 0 0) 0x7eff6806bfd0 con 0x7eff680b1e30 2023-06-10T18:51:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.708+0000 7eff677fe700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff6806b820 0x7eff680cda50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7eff60009d90 tx=0x7eff6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c6b8f78a1a7e17 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff65ffb700 1 -- 172.21.15.104:0/2046886571 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6000e070 con 0x7eff6806b820 2023-06-10T18:51:34.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff65ffb700 1 -- 172.21.15.104:0/2046886571 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6000bbd0 con 0x7eff6806b820 2023-06-10T18:51:34.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff65ffb700 1 -- 172.21.15.104:0/2046886571 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff60012430 con 0x7eff6806b820 2023-06-10T18:51:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 -- 172.21.15.104:0/2046886571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff6806b820 msgr2=0x7eff680cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff6806b820 0x7eff680cda50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7eff60009d90 tx=0x7eff6000b330 comp rx=0 tx=0).stop 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 -- 172.21.15.104:0/2046886571 shutdown_connections 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff6806b820 0x7eff680cda50 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/2046886571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff680d0c00 0x7eff6806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 -- 172.21.15.104:0/2046886571 >> 172.21.15.104:0/2046886571 conn(0x7eff68071ec0 msgr2=0x7eff6809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 -- 172.21.15.104:0/2046886571 shutdown_connections 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.709+0000 7eff6d9e3700 1 -- 172.21.15.104:0/2046886571 wait complete. 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.710+0000 7eff6d9e3700 1 Processor -- start 2023-06-10T18:51:34.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 -- start start 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff6806b820 0x7eff681f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff680d0c00 0x7eff6815d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff680b7310 con 0x7eff680b1e30 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff680ca090 con 0x7eff680d0c00 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff6d9e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff680ca370 con 0x7eff6806b820 2023-06-10T18:51:34.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff667fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff667fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60440/0 (socket says 172.21.15.104:60440) 2023-06-10T18:51:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.711+0000 7eff667fc700 1 -- 172.21.15.104:0/1581040622 learned_addr learned my addr 172.21.15.104:0/1581040622 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 -- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff6806b820 msgr2=0x7eff681f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff6806b820 0x7eff681f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 -- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff680d0c00 msgr2=0x7eff6815d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff680d0c00 0x7eff6815d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 -- 172.21.15.104:0/1581040622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60009a70 con 0x7eff680b1e30 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff667fc700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7eff5800d830 tx=0x7eff5800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580125f0 con 0x7eff680b1e30 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58012750 con 0x7eff680b1e30 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6815e230 con 0x7eff680b1e30 2023-06-10T18:51:34.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58012ae0 con 0x7eff680b1e30 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.712+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6815e7f0 con 0x7eff680b1e30 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.713+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff680d1860 con 0x7eff680b1e30 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.713+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff58012c40 con 0x7eff680b1e30 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.714+0000 7eff4ffff700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 0x7eff5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.714+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7eff58089940 con 0x7eff680b1e30 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.714+0000 7eff66ffd700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 0x7eff5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:34.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.714+0000 7eff66ffd700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 0x7eff5406d190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0011e0 tx=0x7eff5c0079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:34.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.718+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff580548d0 con 0x7eff680b1e30 2023-06-10T18:51:34.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.906+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff680ca4b0 con 0x7eff680b1e30 2023-06-10T18:51:34.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.907+0000 7eff4ffff700 1 -- 172.21.15.104:0/1581040622 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18760 (secure 0 0 0) 0x7eff680ca4b0 con 0x7eff680b1e30 2023-06-10T18:51:34.913 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:34.913 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":71,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:50:54.764730+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.a","mappings":[{"from":6,"to":0}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/2562980824":"2023-06-10T19:50:53.863403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:34.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 msgr2=0x7eff5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 0x7eff5406d190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0011e0 tx=0x7eff5c0079b0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 msgr2=0x7eff680b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7eff5800d830 tx=0x7eff5800dbf0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 shutdown_connections 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5406ace0 0x7eff5406d190 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff680d0c00 0x7eff6815d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff6806b820 0x7eff681f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 --2- 172.21.15.104:0/1581040622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff680b1e30 0x7eff680b6dd0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:34.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 >> 172.21.15.104:0/1581040622 conn(0x7eff68071ec0 msgr2=0x7eff68072a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:34.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 shutdown_connections 2023-06-10T18:51:34.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:34.909+0000 7eff6d9e3700 1 -- 172.21.15.104:0/1581040622 wait complete. 2023-06-10T18:51:34.927 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:51:34.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:35.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.041+0000 7f35afe5f700 1 Processor -- start 2023-06-10T18:51:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.041+0000 7f35afe5f700 1 -- start start 2023-06-10T18:51:35.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35afe5f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35afe5f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 0x7f35a80cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35afe5f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a806bfd0 con 0x7f35a80b1e30 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35afe5f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a806c2b0 con 0x7f35a806b820 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35afe5f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80ccf70 con 0x7f35a80d0c00 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35ad3fa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35ae3fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 0x7f35a80cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35adbfb700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f35a80b1e30 0x7f35a80d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:35888/0 (socket says 172.21.15.104:35888) 2023-06-10T18:51:35.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35ad3fa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:51600/0 (socket says 172.21.15.104:51600) 2023-06-10T18:51:35.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.042+0000 7f35adbfb700 1 -- 172.21.15.104:0/2223677200 learned_addr learned my addr 172.21.15.104:0/2223677200 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:35.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ad3fa700 1 -- 172.21.15.104:0/2223677200 >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 msgr2=0x7f35a80cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ad3fa700 1 --2- 172.21.15.104:0/2223677200 >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 0x7f35a80cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ad3fa700 1 -- 172.21.15.104:0/2223677200 >> v1:172.21.15.114:6789/0 conn(0x7f35a80b1e30 legacy=0x7f35a80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35acbf9700 1 -- 172.21.15.104:0/2223677200 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499317629 0 0) 0x7f35a806bfd0 con 0x7f35a80b1e30 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ad3fa700 1 -- 172.21.15.104:0/2223677200 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80cd250 con 0x7f35a80d0c00 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ae3fc700 1 --2- 172.21.15.104:0/2223677200 >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 0x7f35a80cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35ad3fa700 1 --2- 172.21.15.104:0/2223677200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009840 tx=0x7f35a4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e6414afdfaf704 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35acbf9700 1 -- 172.21.15.104:0/2223677200 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4011830 con 0x7f35a80d0c00 2023-06-10T18:51:35.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35acbf9700 1 -- 172.21.15.104:0/2223677200 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a4011990 con 0x7f35a80d0c00 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.043+0000 7f35acbf9700 1 -- 172.21.15.104:0/2223677200 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4011d20 con 0x7f35a80d0c00 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 -- 172.21.15.104:0/2223677200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 msgr2=0x7f35a806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 --2- 172.21.15.104:0/2223677200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009840 tx=0x7f35a4009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 -- 172.21.15.104:0/2223677200 shutdown_connections 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 --2- 172.21.15.104:0/2223677200 >> v2:172.21.15.114:3300/0 conn(0x7f35a806b820 0x7f35a80cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 --2- 172.21.15.104:0/2223677200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a80d0c00 0x7f35a806b2e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 -- 172.21.15.104:0/2223677200 >> 172.21.15.104:0/2223677200 conn(0x7f35a8071ec0 msgr2=0x7f35a809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:35.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 -- 172.21.15.104:0/2223677200 shutdown_connections 2023-06-10T18:51:35.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.044+0000 7f35afe5f700 1 -- 172.21.15.104:0/2223677200 wait complete. 2023-06-10T18:51:35.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.045+0000 7f35afe5f700 1 Processor -- start 2023-06-10T18:51:35.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.045+0000 7f35afe5f700 1 -- start start 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a806b820 0x7f35a80b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a80d0c00 0x7f35a8142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80b6230 con 0x7f35a80b1e30 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8145550 con 0x7f35a80d0c00 2023-06-10T18:51:35.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8145830 con 0x7f35a806b820 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60454/0 (socket says 172.21.15.104:60454) 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 -- 172.21.15.104:0/1781701710 learned_addr learned my addr 172.21.15.104:0/1781701710 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35adbfb700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a806b820 0x7f35a80b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ae3fc700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a80d0c00 0x7f35a8142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 -- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a806b820 msgr2=0x7f35a80b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a806b820 0x7f35a80b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 -- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a80d0c00 msgr2=0x7f35a8142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a80d0c00 0x7f35a8142c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 -- 172.21.15.104:0/1781701710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0009a70 con 0x7f35a80b1e30 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35ad3fa700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009fd0 tx=0x7f35a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a40036a0 con 0x7f35a80b1e30 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a401e070 con 0x7f35a80b1e30 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a40094a0 con 0x7f35a80b1e30 2023-06-10T18:51:35.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4003a20 con 0x7f35a80b1e30 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.046+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a80b79c0 con 0x7f35a80b1e30 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.047+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a80d1860 con 0x7f35a80b1e30 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.048+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f35a4003b80 con 0x7f35a80b1e30 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.048+0000 7f359e7fc700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.048+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f35a408a4d0 con 0x7f35a80b1e30 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.048+0000 7f35adbfb700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.049+0000 7f35adbfb700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3598007fd0 tx=0x7f3598007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.052+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f35a4055460 con 0x7f35a80b1e30 2023-06-10T18:51:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.279+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a8145970 con 0x7f35a80b1e30 2023-06-10T18:51:35.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.279+0000 7f359e7fc700 1 -- 172.21.15.104:0/1781701710 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f35a8145970 con 0x7f35a80b1e30 2023-06-10T18:51:35.285 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:35.285 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:51:35.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 msgr2=0x7f358c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3598007fd0 tx=0x7f3598007de0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 msgr2=0x7f35a8144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009fd0 tx=0x7f35a4004750 comp rx=0 tx=0).stop 2023-06-10T18:51:35.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 shutdown_connections 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f358c06ace0 0x7f358c06d190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35a80d0c00 0x7f35a8142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35a806b820 0x7f35a80b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 --2- 172.21.15.104:0/1781701710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a80b1e30 0x7f35a8144cc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 >> 172.21.15.104:0/1781701710 conn(0x7f35a8071ec0 msgr2=0x7f35a80d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 shutdown_connections 2023-06-10T18:51:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.281+0000 7f35afe5f700 1 -- 172.21.15.104:0/1781701710 wait complete. 2023-06-10T18:51:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:51:35.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:35.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.413+0000 7faf59882700 1 Processor -- start 2023-06-10T18:51:35.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.413+0000 7faf59882700 1 -- start start 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf59882700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf5406b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf59882700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf5406b820 0x7faf540cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf59882700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5406bfd0 con 0x7faf540b1e30 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf59882700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf5406c2b0 con 0x7faf5406b820 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf59882700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf540ccf70 con 0x7faf540d0c00 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf5406b820 0x7faf540cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf52ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faf540b1e30 0x7faf540d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58288/0 (socket says 172.21.15.104:58288) 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf5406b820 0x7faf540cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60464/0 (socket says 172.21.15.104:60464) 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf52ffd700 1 -- 172.21.15.104:0/3324832117 learned_addr learned my addr 172.21.15.104:0/3324832117 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf527fc700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf5406b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 -- 172.21.15.104:0/3324832117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 msgr2=0x7faf5406b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf5406b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 -- 172.21.15.104:0/3324832117 >> v1:172.21.15.104:6789/0 conn(0x7faf540b1e30 legacy=0x7faf540d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 -- 172.21.15.104:0/3324832117 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf540cd250 con 0x7faf5406b820 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf51ffb700 1 -- 172.21.15.104:0/3324832117 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998207909 0 0) 0x7faf5406bfd0 con 0x7faf540b1e30 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf527fc700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf5406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf537fe700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf5406b820 0x7faf540cda50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d90 tx=0x7faf4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87dc0915de42f79d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf51ffb700 1 -- 172.21.15.104:0/3324832117 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c00e070 con 0x7faf5406b820 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf51ffb700 1 -- 172.21.15.104:0/3324832117 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c00bbd0 con 0x7faf5406b820 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.414+0000 7faf51ffb700 1 -- 172.21.15.104:0/3324832117 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c012430 con 0x7faf5406b820 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 -- 172.21.15.104:0/3324832117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf5406b820 msgr2=0x7faf540cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf5406b820 0x7faf540cda50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d90 tx=0x7faf4c00b330 comp rx=0 tx=0).stop 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 -- 172.21.15.104:0/3324832117 shutdown_connections 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf5406b820 0x7faf540cda50 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 --2- 172.21.15.104:0/3324832117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf5406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 -- 172.21.15.104:0/3324832117 >> 172.21.15.104:0/3324832117 conn(0x7faf54071ec0 msgr2=0x7faf5409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 -- 172.21.15.104:0/3324832117 shutdown_connections 2023-06-10T18:51:35.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.415+0000 7faf59882700 1 -- 172.21.15.104:0/3324832117 wait complete. 2023-06-10T18:51:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.416+0000 7faf59882700 1 Processor -- start 2023-06-10T18:51:35.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.416+0000 7faf59882700 1 -- start start 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 0x7faf541f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf540b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf540c9c60 con 0x7faf540b1e30 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541f57a0 con 0x7faf5406b820 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf59882700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf541f5a80 con 0x7faf540d0c00 2023-06-10T18:51:35.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60468/0 (socket says 172.21.15.104:60468) 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf527fc700 1 -- 172.21.15.104:0/3599396450 learned_addr learned my addr 172.21.15.104:0/3599396450 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf52ffd700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 0x7faf541f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.417+0000 7faf537fe700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf540b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 -- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 msgr2=0x7faf540b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf540b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 -- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 msgr2=0x7faf541f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 0x7faf541f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 -- 172.21.15.104:0/3599396450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c009a70 con 0x7faf540b1e30 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf52ffd700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 0x7faf541f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf527fc700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c7f0 tx=0x7faf4400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44011800 con 0x7faf540b1e30 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44011960 con 0x7faf540b1e30 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf541f5d60 con 0x7faf540b1e30 2023-06-10T18:51:35.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44011d10 con 0x7faf540b1e30 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.418+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf540c9f40 con 0x7faf540b1e30 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.419+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf540d1860 con 0x7faf540b1e30 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.419+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf44018360 con 0x7faf540b1e30 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.420+0000 7faf3bfff700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 0x7faf4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.420+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7faf44089630 con 0x7faf540b1e30 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.420+0000 7faf52ffd700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 0x7faf4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.420+0000 7faf52ffd700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 0x7faf4006d190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7faf48000f60 tx=0x7faf48007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.424+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf44057fc0 con 0x7faf540b1e30 2023-06-10T18:51:35.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.653+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf54074050 con 0x7faf540b1e30 2023-06-10T18:51:35.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.653+0000 7faf3bfff700 1 -- 172.21.15.104:0/3599396450 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4685 (secure 0 0 0) 0x7faf44057b50 con 0x7faf540b1e30 2023-06-10T18:51:35.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:35.660 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:48:44.146291+0000","modified":"2023-06-10T18:48:47.161449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6699":{"gid":6699,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/1280497888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1280497888},{"type":"v1","addr":"172.21.15.114:6836","nonce":1280497888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-06-10T18:51:35.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 msgr2=0x7faf4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 0x7faf4006d190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7faf48000f60 tx=0x7faf48007ca0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 msgr2=0x7faf540ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c7f0 tx=0x7faf4400cbb0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 shutdown_connections 2023-06-10T18:51:35.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4006ace0 0x7faf4006d190 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf5406b820 0x7faf541f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf540d0c00 0x7faf540b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 --2- 172.21.15.104:0/3599396450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf540b1e30 0x7faf540ce5b0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 >> 172.21.15.104:0/3599396450 conn(0x7faf54071ec0 msgr2=0x7faf54072a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:35.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 shutdown_connections 2023-06-10T18:51:35.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.655+0000 7faf59882700 1 -- 172.21.15.104:0/3599396450 wait complete. 2023-06-10T18:51:35.665 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 33 2023-06-10T18:51:35.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.787+0000 7fa298874700 1 Processor -- start 2023-06-10T18:51:35.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.787+0000 7fa298874700 1 -- start start 2023-06-10T18:51:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa298874700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa2940b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa298874700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2940b4ff0 0x7fa2940ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa298874700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b6070 con 0x7fa2940ce890 2023-06-10T18:51:35.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa298874700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940ced10 con 0x7fa2940b4ff0 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa298874700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940b5460 con 0x7fa29415a420 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa292d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa2940ce890 0x7fa2940b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58306/0 (socket says 172.21.15.104:58306) 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa292d9d700 1 -- 172.21.15.104:0/1534663008 learned_addr learned my addr 172.21.15.104:0/1534663008 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa291d9b700 1 --2- 172.21.15.104:0/1534663008 >> v2:172.21.15.114:3300/0 conn(0x7fa2940b4ff0 0x7fa2940ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa29259c700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa2940b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.788+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369132500 0 0) 0x7fa2940b6070 con 0x7fa2940ce890 2023-06-10T18:51:35.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c0035b0 con 0x7fa2940ce890 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085766139 0 0) 0x7fa27c0035b0 con 0x7fa2940ce890 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 msgr2=0x7fa2940b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa2940b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 >> v2:172.21.15.114:3300/0 conn(0x7fa2940b4ff0 msgr2=0x7fa2940ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 --2- 172.21.15.104:0/1534663008 >> v2:172.21.15.114:3300/0 conn(0x7fa2940b4ff0 0x7fa2940ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940b5740 con 0x7fa2940ce890 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa28c0019f0 con 0x7fa2940ce890 2023-06-10T18:51:35.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 >> v1:172.21.15.104:6789/0 conn(0x7fa2940ce890 legacy=0x7fa2940b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa27c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa27c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c004d90 0x7fa27c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940ced10 con 0x7fa27c003f80 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940b5460 con 0x7fa27c004d90 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c00ba10 con 0x7fa29415a420 2023-06-10T18:51:35.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa27c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29259c700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa27c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa292d9d700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c004d90 0x7fa27c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 -- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 msgr2=0x7fa27c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa27c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 -- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c004d90 msgr2=0x7fa27c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c004d90 0x7fa27c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 -- 172.21.15.104:0/1534663008 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0035b0 con 0x7fa27c003f80 2023-06-10T18:51:35.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa292d9d700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c004d90 0x7fa27c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29259c700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa29415a420 0x7fa27c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa291d9b700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa27c004850 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa284009b60 tx=0x7fa27c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7373b61f95dbd93 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400e6c0 con 0x7fa27c003f80 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28400e820 con 0x7fa27c003f80 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.789+0000 7fa29159a700 1 -- 172.21.15.104:0/1534663008 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400eb90 con 0x7fa27c003f80 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 -- 172.21.15.104:0/1534663008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 msgr2=0x7fa27c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 --2- 172.21.15.104:0/1534663008 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa27c004850 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa284009b60 tx=0x7fa27c005880 comp rx=0 tx=0).stop 2023-06-10T18:51:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa292d9d700 1 -- 172.21.15.104:0/1534663008 reap_dead start 2023-06-10T18:51:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 -- 172.21.15.104:0/1534663008 shutdown_connections 2023-06-10T18:51:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 -- 172.21.15.104:0/1534663008 >> 172.21.15.104:0/1534663008 conn(0x7fa294089fd0 msgr2=0x7fa2940d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 -- 172.21.15.104:0/1534663008 shutdown_connections 2023-06-10T18:51:35.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.790+0000 7fa298874700 1 -- 172.21.15.104:0/1534663008 wait complete. 2023-06-10T18:51:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.791+0000 7fa298874700 1 Processor -- start 2023-06-10T18:51:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.791+0000 7fa298874700 1 -- start start 2023-06-10T18:51:35.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 0x7fa2941df770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 0x7fa294157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294158260 con 0x7fa27c003f80 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940cb690 con 0x7fa2940c9670 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa298874700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940cb970 con 0x7fa294157880 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa29259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa29259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60486/0 (socket says 172.21.15.104:60486) 2023-06-10T18:51:35.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.792+0000 7fa29259c700 1 -- 172.21.15.104:0/2286180240 learned_addr learned my addr 172.21.15.104:0/2286180240 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa292d9d700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 0x7fa294157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 -- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 msgr2=0x7fa294157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa291d9b700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 0x7fa2941df770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 0x7fa294157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 -- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 msgr2=0x7fa2941df770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 0x7fa2941df770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:35.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 -- 172.21.15.104:0/2286180240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940cbc50 con 0x7fa27c003f80 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa292d9d700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 0x7fa294157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa291d9b700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 0x7fa2941df770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa29259c700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa2880118d0 tx=0x7fa288011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288004220 con 0x7fa27c003f80 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288004380 con 0x7fa27c003f80 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2880036a0 con 0x7fa27c003f80 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2940cbf30 con 0x7fa27c003f80 2023-06-10T18:51:35.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.793+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941dfcb0 con 0x7fa27c003f80 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.794+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294074050 con 0x7fa27c003f80 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.794+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa288003800 con 0x7fa27c003f80 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.795+0000 7fa282ffd700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 0x7fa27406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.795+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fa2880899e0 con 0x7fa27c003f80 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.795+0000 7fa291d9b700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 0x7fa27406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.795+0000 7fa291d9b700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 0x7fa27406d190 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa2940cc9d0 tx=0x7fa284008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:35.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.799+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2880548e0 con 0x7fa27c003f80 2023-06-10T18:51:35.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.990+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2940cab20 con 0x7fa27c003f80 2023-06-10T18:51:35.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.991+0000 7fa282ffd700 1 -- 172.21.15.104:0/2286180240 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+18760 (secure 0 0 0) 0x7fa288022090 con 0x7fa27c003f80 2023-06-10T18:51:35.997 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:35.998 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":71,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:50:54.764730+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:48:42.072415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-06-10T18:48:43.111638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.a","mappings":[{"from":6,"to":0}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/2562980824":"2023-06-10T19:50:53.863403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.993+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 msgr2=0x7fa27406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:35.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.993+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 0x7fa27406d190 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa2940cc9d0 tx=0x7fa284008030 comp rx=0 tx=0).stop 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.993+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 msgr2=0x7fa2940cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.993+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa2880118d0 tx=0x7fa288011c90 comp rx=0 tx=0).stop 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 shutdown_connections 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa294157880 0x7fa294157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c003f80 0x7fa2940cad90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa27406ace0 0x7fa27406d190 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 --2- 172.21.15.104:0/2286180240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2940c9670 0x7fa2941df770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 >> 172.21.15.104:0/2286180240 conn(0x7fa294089fd0 msgr2=0x7fa2941e0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 shutdown_connections 2023-06-10T18:51:36.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:35.994+0000 7fa298874700 1 -- 172.21.15.104:0/2286180240 wait complete. 2023-06-10T18:51:36.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:51:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.126+0000 7f77ffa22700 1 Processor -- start 2023-06-10T18:51:36.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.126+0000 7f77ffa22700 1 -- start start 2023-06-10T18:51:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77ffa22700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f77f80cf100 0x7f77f806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77ffa22700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f806d290 con 0x7f77f806e0e0 2023-06-10T18:51:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77ffa22700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f80ca2d0 con 0x7f77f806e4b0 2023-06-10T18:51:36.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77ffa22700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f8071640 con 0x7f77f80cf100 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77fd7be700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f77f806e0e0 0x7f77f80c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58322/0 (socket says 172.21.15.104:58322) 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.127+0000 7f77fd7be700 1 -- 172.21.15.104:0/402586674 learned_addr learned my addr 172.21.15.104:0/402586674 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592827256 0 0) 0x7f77f806d290 con 0x7f77f806e0e0 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fdfbf700 1 --2- 172.21.15.104:0/402586674 >> v2:172.21.15.114:3300/0 conn(0x7f77f80cf100 0x7f77f806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e40035b0 con 0x7f77f806e0e0 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560724329 0 0) 0x7f77e40035b0 con 0x7f77f806e0e0 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 >> v2:172.21.15.114:3300/0 conn(0x7f77f80cf100 msgr2=0x7f77f806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:36.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 --2- 172.21.15.104:0/402586674 >> v2:172.21.15.114:3300/0 conn(0x7f77f80cf100 0x7f77f806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 >> v1:172.21.15.114:6789/0 conn(0x7f77f806e4b0 legacy=0x7f77f80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f806d4c0 con 0x7f77f806e0e0 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f77f40017f0 con 0x7f77f806e0e0 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 >> v1:172.21.15.104:6789/0 conn(0x7f77f806e0e0 legacy=0x7f77f80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 0x7f77e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e4004850 0x7f77e4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4005200 0x7f77e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f8071640 con 0x7f77e4003a00 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e400c110 con 0x7f77e4004850 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e400c2d0 con 0x7f77e4005200 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 0x7f77e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fcfbd700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e4004850 0x7f77e4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fdfbf700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4005200 0x7f77e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 -- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4005200 msgr2=0x7f77e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:36.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4005200 0x7f77e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 -- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e4004850 msgr2=0x7f77e4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e4004850 0x7f77e4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fd7be700 1 -- 172.21.15.104:0/402586674 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40035b0 con 0x7f77e4003a00 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.128+0000 7f77fdfbf700 1 -- 172.21.15.104:0/402586674 reap_dead start 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77fd7be700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 0x7f77e4003e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c240 tx=0x7f77e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7471244f8ec0e4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400cde0 con 0x7f77e4003a00 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f4007b80 con 0x7f77e4003a00 2023-06-10T18:51:36.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77effff700 1 -- 172.21.15.104:0/402586674 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f40072f0 con 0x7f77e4003a00 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 -- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 msgr2=0x7f77e4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 0x7f77e4003e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f77f400c240 tx=0x7f77e4005880 comp rx=0 tx=0).stop 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 -- 172.21.15.104:0/402586674 shutdown_connections 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 --2- 172.21.15.104:0/402586674 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77e4003a00 0x7f77e4003e10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 -- 172.21.15.104:0/402586674 >> 172.21.15.104:0/402586674 conn(0x7f77f806b8a0 msgr2=0x7f77f80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 -- 172.21.15.104:0/402586674 shutdown_connections 2023-06-10T18:51:36.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.129+0000 7f77ffa22700 1 -- 172.21.15.104:0/402586674 wait complete. 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.130+0000 7f77ffa22700 1 Processor -- start 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.130+0000 7f77ffa22700 1 -- start start 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4003a00 0x7f77f814cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 0x7f77f80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80c94b0 con 0x7f77f80c8b00 2023-06-10T18:51:36.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80c6640 con 0x7f77f80b3b30 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77ffa22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80b4860 con 0x7f77e4003a00 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77fcfbd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77fcfbd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60508/0 (socket says 172.21.15.104:60508) 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77fcfbd700 1 -- 172.21.15.104:0/3642835821 learned_addr learned my addr 172.21.15.104:0/3642835821 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77fd7be700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4003a00 0x7f77f814cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.131+0000 7f77fdfbf700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 0x7f77f80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 -- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4003a00 msgr2=0x7f77f814cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4003a00 0x7f77f814cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 -- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 msgr2=0x7f77f80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 0x7f77f80b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 -- 172.21.15.104:0/3642835821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f80b4b40 con 0x7f77f80c8b00 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fdfbf700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 0x7f77f80b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77fcfbd700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f77e800f850 tx=0x7f77e800fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e80050d0 con 0x7f77f80c8b00 2023-06-10T18:51:36.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8005250 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f80c6890 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8003680 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.132+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f80b4510 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.133+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f8066f70 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.133+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f77e80037e0 con 0x7f77f80c8b00 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.134+0000 7f77edffb700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 0x7f77dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:36.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.134+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f77e8089370 con 0x7f77f80c8b00 2023-06-10T18:51:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.134+0000 7f77fd7be700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 0x7f77dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.134+0000 7f77fd7be700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 0x7f77dc06d190 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f77f4015040 tx=0x7f77f4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:36.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.138+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f77f8066f70 con 0x7f77f80c8b00 2023-06-10T18:51:36.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.366+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f77f8066f70 con 0x7f77f80c8b00 2023-06-10T18:51:36.861 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:51:36.856+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:51:36.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.866+0000 7f77edffb700 1 -- 172.21.15.104:0/3642835821 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f77f8066f70 con 0x7f77f80c8b00 2023-06-10T18:51:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 msgr2=0x7f77dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 0x7f77dc06d190 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f77f4015040 tx=0x7f77f4007000 comp rx=0 tx=0).stop 2023-06-10T18:51:36.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 msgr2=0x7f77f80c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f77e800f850 tx=0x7f77e800fc10 comp rx=0 tx=0).stop 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 shutdown_connections 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f77dc06ace0 0x7f77dc06d190 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f80b3b30 0x7f77f80b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4003a00 0x7f77f814cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 --2- 172.21.15.104:0/3642835821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77f80c8b00 0x7f77f80c8f70 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:36.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 >> 172.21.15.104:0/3642835821 conn(0x7f77f806b8a0 msgr2=0x7f77f80caed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:36.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 shutdown_connections 2023-06-10T18:51:36.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:36.868+0000 7f77ffa22700 1 -- 172.21.15.104:0/3642835821 wait complete. 2023-06-10T18:51:36.876 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:51:36.877 INFO:tasks.ceph.mds.d.smithi114.stderr: -6660> 2023-06-10T18:48:49.729+0000 7fd5acee0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:51:36.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:51:36.908 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:51:37.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 Processor -- start 2023-06-10T18:51:37.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 -- start start 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc31806e6e0 0x7fc3180b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806cde0 0x7fc3180ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31806e250 con 0x7fc31815c580 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31815c950 con 0x7fc31806e6e0 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.005+0000 7fc31ecba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180b1e80 con 0x7fc31806cde0 2023-06-10T18:51:37.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc31806e6e0 0x7fc3180b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc31806e6e0 0x7fc3180b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60516/0 (socket says 172.21.15.104:60516) 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 -- 172.21.15.104:0/3377244807 learned_addr learned my addr 172.21.15.104:0/3377244807 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc31d257700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806cde0 0x7fc3180ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 -- 172.21.15.104:0/3377244807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806cde0 msgr2=0x7fc3180ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806cde0 0x7fc3180ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 -- 172.21.15.104:0/3377244807 >> v1:172.21.15.114:6789/0 conn(0x7fc31815c580 legacy=0x7fc3180b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 -- 172.21.15.104:0/3377244807 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31806d920 con 0x7fc31806e6e0 2023-06-10T18:51:37.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc317fff700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806e6e0 0x7fc3180b1780 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098f0 tx=0x7fc304009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d984953c38b48899 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:37.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc3177fe700 1 -- 172.21.15.104:0/3377244807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304004230 con 0x7fc31806e6e0 2023-06-10T18:51:37.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc3177fe700 1 -- 172.21.15.104:0/3377244807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc304004390 con 0x7fc31806e6e0 2023-06-10T18:51:37.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.006+0000 7fc3177fe700 1 -- 172.21.15.104:0/3377244807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3040036a0 con 0x7fc31806e6e0 2023-06-10T18:51:37.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 -- 172.21.15.104:0/3377244807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806e6e0 msgr2=0x7fc3180b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806e6e0 0x7fc3180b1780 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098f0 tx=0x7fc304009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 -- 172.21.15.104:0/3377244807 shutdown_connections 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806cde0 0x7fc3180ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 --2- 172.21.15.104:0/3377244807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806e6e0 0x7fc3180b1780 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 -- 172.21.15.104:0/3377244807 >> 172.21.15.104:0/3377244807 conn(0x7fc31806b8a0 msgr2=0x7fc3180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 -- 172.21.15.104:0/3377244807 shutdown_connections 2023-06-10T18:51:37.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.007+0000 7fc31ecba700 1 -- 172.21.15.104:0/3377244807 wait complete. 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.008+0000 7fc31ecba700 1 Processor -- start 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.008+0000 7fc31ecba700 1 -- start start 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806e6e0 0x7fc3180c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 0x7fc3180b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180b4550 con 0x7fc31806cde0 2023-06-10T18:51:37.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180c9610 con 0x7fc31815c580 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ecba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180b3bb0 con 0x7fc31806e6e0 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60520/0 (socket says 172.21.15.104:60520) 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 -- 172.21.15.104:0/409434896 learned_addr learned my addr 172.21.15.104:0/409434896 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc317fff700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806e6e0 0x7fc3180c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31d257700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 0x7fc3180b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 -- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806e6e0 msgr2=0x7fc3180c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806e6e0 0x7fc3180c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 -- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 msgr2=0x7fc3180b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 0x7fc3180b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 -- 172.21.15.104:0/409434896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc304009550 con 0x7fc31806cde0 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31d257700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 0x7fc3180b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.009+0000 7fc31ca56700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b450 tx=0x7fc30c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:37.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.010+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c004020 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.010+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c004180 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.010+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3180b3e90 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.010+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c004530 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.010+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3180c8130 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.011+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc318074050 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.011+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc30c0036d0 con 0x7fc31806cde0 2023-06-10T18:51:37.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.011+0000 7fc3157fa700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 0x7fc2fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:37.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.011+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6569+0+0 (secure 0 0 0) 0x7fc30c088f60 con 0x7fc31806cde0 2023-06-10T18:51:37.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.012+0000 7fc317fff700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 0x7fc2fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:37.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.012+0000 7fc317fff700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 0x7fc2fc06d190 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc31806eb20 tx=0x7fc304008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:37.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.016+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc30c053e10 con 0x7fc31806cde0 2023-06-10T18:51:37.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.247+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc3180cd8c0 con 0x7fc31806cde0 2023-06-10T18:51:37.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.869+0000 7fc3157fa700 1 -- 172.21.15.104:0/409434896 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc30c057430 con 0x7fc31806cde0 2023-06-10T18:51:37.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 msgr2=0x7fc2fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:37.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 0x7fc2fc06d190 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc31806eb20 tx=0x7fc304008030 comp rx=0 tx=0).stop 2023-06-10T18:51:37.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 msgr2=0x7fc3181e32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b450 tx=0x7fc30c00b810 comp rx=0 tx=0).stop 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 shutdown_connections 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2fc06ace0 0x7fc2fc06d190 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc31815c580 0x7fc3180b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc31806cde0 0x7fc3181e32f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 --2- 172.21.15.104:0/409434896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc31806e6e0 0x7fc3180c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:37.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 >> 172.21.15.104:0/409434896 conn(0x7fc31806b8a0 msgr2=0x7fc3180ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:37.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 shutdown_connections 2023-06-10T18:51:37.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:37.872+0000 7fc31ecba700 1 -- 172.21.15.104:0/409434896 wait complete. 2023-06-10T18:51:37.892 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:51:38.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.012+0000 7f57a8a73700 1 Processor -- start 2023-06-10T18:51:38.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.012+0000 7f57a8a73700 1 -- start start 2023-06-10T18:51:38.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a8a73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 0x7f57a4163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a8a73700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a406d7d0 con 0x7f57a41586a0 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a8a73700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40cd6c0 con 0x7f57a40ce390 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a8a73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a40c95d0 con 0x7f57a4071120 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f57a41586a0 0x7f57a406cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58356/0 (socket says 172.21.15.104:58356) 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a259c700 1 -- 172.21.15.104:0/3286801740 learned_addr learned my addr 172.21.15.104:0/3286801740 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.013+0000 7f57a2d9d700 1 --2- 172.21.15.104:0/3286801740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 0x7f57a4163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:38.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293576735 0 0) 0x7f57a406d7d0 con 0x7f57a41586a0 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f578c0046c0 con 0x7f57a41586a0 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142700014 0 0) 0x7f57a40cd6c0 con 0x7f57a40ce390 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a406d7d0 con 0x7f57a40ce390 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a2d9d700 1 -- 172.21.15.104:0/3286801740 >> v1:172.21.15.114:6789/0 conn(0x7f57a40ce390 legacy=0x7f57a4070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a2d9d700 1 -- 172.21.15.104:0/3286801740 >> v1:172.21.15.104:6789/0 conn(0x7f57a41586a0 legacy=0x7f57a406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a2d9d700 1 -- 172.21.15.104:0/3286801740 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40cd910 con 0x7f57a4071120 2023-06-10T18:51:38.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a2d9d700 1 --2- 172.21.15.104:0/3286801740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 0x7f57a4163160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f579c009dc0 tx=0x7f579c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a79c5f5bf8025db6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c00e070 con 0x7f57a4071120 2023-06-10T18:51:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c00bb20 con 0x7f57a4071120 2023-06-10T18:51:38.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.014+0000 7f57a159a700 1 -- 172.21.15.104:0/3286801740 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c012430 con 0x7f57a4071120 2023-06-10T18:51:38.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 -- 172.21.15.104:0/3286801740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 msgr2=0x7f57a4163160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 --2- 172.21.15.104:0/3286801740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 0x7f57a4163160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f579c009dc0 tx=0x7f579c00b330 comp rx=0 tx=0).stop 2023-06-10T18:51:38.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 -- 172.21.15.104:0/3286801740 shutdown_connections 2023-06-10T18:51:38.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 --2- 172.21.15.104:0/3286801740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a4071120 0x7f57a4163160 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 -- 172.21.15.104:0/3286801740 >> 172.21.15.104:0/3286801740 conn(0x7f57a40ceec0 msgr2=0x7f57a40cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 -- 172.21.15.104:0/3286801740 shutdown_connections 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.015+0000 7f57a8a73700 1 -- 172.21.15.104:0/3286801740 wait complete. 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.016+0000 7f57a8a73700 1 Processor -- start 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.016+0000 7f57a8a73700 1 -- start start 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:38.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57a40ce390 0x7f57a40c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a41586a0 0x7f57a40ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a40ac900 con 0x7f57a4071120 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a414a900 con 0x7f57a40ce390 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a8a73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a414abe0 con 0x7f57a41586a0 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60534/0 (socket says 172.21.15.104:60534) 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 -- 172.21.15.104:0/1024521216 learned_addr learned my addr 172.21.15.104:0/1024521216 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:38.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a2d9d700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a41586a0 0x7f57a40ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a1d9b700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57a40ce390 0x7f57a40c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 -- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a41586a0 msgr2=0x7f57a40ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a41586a0 0x7f57a40ad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 -- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57a40ce390 msgr2=0x7f57a40c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57a40ce390 0x7f57a40c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 -- 172.21.15.104:0/1024521216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c009a70 con 0x7f57a4071120 2023-06-10T18:51:38.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.017+0000 7f57a259c700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f579800c4d0 tx=0x7f579800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.018+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798005d70 con 0x7f57a4071120 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.018+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57980070d0 con 0x7f57a4071120 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.018+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798007480 con 0x7f57a4071120 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.018+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a414aec0 con 0x7f57a4071120 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.018+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a40c64d0 con 0x7f57a4071120 2023-06-10T18:51:38.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.019+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a40c66f0 con 0x7f57a4071120 2023-06-10T18:51:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.019+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f57980075e0 con 0x7f57a4071120 2023-06-10T18:51:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.020+0000 7f5792ffd700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 0x7f578406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.020+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f5798088c30 con 0x7f57a4071120 2023-06-10T18:51:38.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.020+0000 7f57a1d9b700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 0x7f578406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:38.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.020+0000 7f57a1d9b700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 0x7f578406d190 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f57a40c4c00 tx=0x7f5794009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:38.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.024+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5798053ae0 con 0x7f57a4071120 2023-06-10T18:51:38.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.210+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57a4074050 con 0x7f57a4071120 2023-06-10T18:51:38.612 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:51:38.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:51:38.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.932+0000 7f5792ffd700 1 -- 172.21.15.104:0/1024521216 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5798057100 con 0x7f57a4071120 2023-06-10T18:51:38.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 msgr2=0x7f578406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 0x7f578406d190 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f57a40c4c00 tx=0x7f5794009040 comp rx=0 tx=0).stop 2023-06-10T18:51:38.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 msgr2=0x7f57a40c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:38.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f579800c4d0 tx=0x7f579800c890 comp rx=0 tx=0).stop 2023-06-10T18:51:38.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 shutdown_connections 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57a41586a0 0x7f57a40ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a4071120 0x7f57a40c46c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f578406ace0 0x7f578406d190 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 --2- 172.21.15.104:0/1024521216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57a40ce390 0x7f57a40c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 >> 172.21.15.104:0/1024521216 conn(0x7f57a40ceec0 msgr2=0x7f57a40cafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 shutdown_connections 2023-06-10T18:51:38.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:38.935+0000 7f57a8a73700 1 -- 172.21.15.104:0/1024521216 wait complete. 2023-06-10T18:51:38.943 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:51:38.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:51:39.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.071+0000 7f51f7901700 1 Processor -- start 2023-06-10T18:51:39.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 -- start start 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f015c6f0 0x7f51f00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f51f006ea70 0x7f51f00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51f00aec00 0x7f51f00ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00ae7e0 con 0x7f51f00aec00 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f006df90 con 0x7f51f006ea70 2023-06-10T18:51:39.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f7901700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f006e270 con 0x7f51f015c6f0 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.072+0000 7f51f5e9e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51f00aec00 0x7f51f00ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51f00aec00 0x7f51f00ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60542/0 (socket says 172.21.15.104:60542) 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 -- 172.21.15.104:0/1634806711 learned_addr learned my addr 172.21.15.104:0/1634806711 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f569d700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f015c6f0 0x7f51f00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 -- 172.21.15.104:0/1634806711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f015c6f0 msgr2=0x7f51f00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f4e9c700 1 --2- 172.21.15.104:0/1634806711 >> v2:172.21.15.114:3300/0 conn(0x7f51f006ea70 0x7f51f00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f015c6f0 0x7f51f00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 -- 172.21.15.104:0/1634806711 >> v2:172.21.15.114:3300/0 conn(0x7f51f006ea70 msgr2=0x7f51f00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 --2- 172.21.15.104:0/1634806711 >> v2:172.21.15.114:3300/0 conn(0x7f51f006ea70 0x7f51f00b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 -- 172.21.15.104:0/1634806711 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f006e550 con 0x7f51f00aec00 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51f5e9e700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f00ac1c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f51f0067240 tx=0x7f51e800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f97f9b7a6b81bb8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51e7fff700 1 -- 172.21.15.104:0/1634806711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8010070 con 0x7f51f00aec00 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51e7fff700 1 -- 172.21.15.104:0/1634806711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e800dd90 con 0x7f51f00aec00 2023-06-10T18:51:39.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.073+0000 7f51e7fff700 1 -- 172.21.15.104:0/1634806711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e800b310 con 0x7f51f00aec00 2023-06-10T18:51:39.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 -- 172.21.15.104:0/1634806711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 msgr2=0x7f51f00ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:39.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f00ac1c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f51f0067240 tx=0x7f51e800d430 comp rx=0 tx=0).stop 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 -- 172.21.15.104:0/1634806711 shutdown_connections 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f00ac1c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 --2- 172.21.15.104:0/1634806711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f015c6f0 0x7f51f00b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 --2- 172.21.15.104:0/1634806711 >> v2:172.21.15.114:3300/0 conn(0x7f51f006ea70 0x7f51f00b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 -- 172.21.15.104:0/1634806711 >> 172.21.15.104:0/1634806711 conn(0x7f51f006b8a0 msgr2=0x7f51f00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:39.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 -- 172.21.15.104:0/1634806711 shutdown_connections 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.074+0000 7f51f7901700 1 -- 172.21.15.104:0/1634806711 wait complete. 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.075+0000 7f51f7901700 1 Processor -- start 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.075+0000 7f51f7901700 1 -- start start 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f006ea70 0x7f51f0152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 0x7f51f015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f01597c0 con 0x7f51f00aec00 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0159aa0 con 0x7f51f015c6f0 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0159d80 con 0x7f51f006ea70 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60548/0 (socket says 172.21.15.104:60548) 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 -- 172.21.15.104:0/124823857 learned_addr learned my addr 172.21.15.104:0/124823857 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:39.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f569d700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f006ea70 0x7f51f0152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f5e9e700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 0x7f51f015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 -- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f006ea70 msgr2=0x7f51f0152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f006ea70 0x7f51f0152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 -- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 msgr2=0x7f51f015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 0x7f51f015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 -- 172.21.15.104:0/124823857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e800bd40 con 0x7f51f00aec00 2023-06-10T18:51:39.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f5e9e700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 0x7f51f015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f4e9c700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00c7f0 tx=0x7f51ec00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec011800 con 0x7f51f00aec00 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec011960 con 0x7f51f00aec00 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec011d10 con 0x7f51f00aec00 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.076+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f015a060 con 0x7f51f00aec00 2023-06-10T18:51:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.077+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01f2a30 con 0x7f51f00aec00 2023-06-10T18:51:39.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.077+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f0074050 con 0x7f51f00aec00 2023-06-10T18:51:39.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.078+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f51ec018360 con 0x7f51f00aec00 2023-06-10T18:51:39.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.078+0000 7f51e5ffb700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 0x7f51d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:39.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.078+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6131+0+0 (secure 0 0 0) 0x7f51ec089430 con 0x7f51f00aec00 2023-06-10T18:51:39.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.078+0000 7f51f569d700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 0x7f51d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:39.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.079+0000 7f51f569d700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 0x7f51d406d190 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51e000aaa0 tx=0x7f51e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:39.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.083+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f51ec054410 con 0x7f51f00aec00 2023-06-10T18:51:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.267+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f51f006ad60 con 0x7f51f00aec00 2023-06-10T18:51:39.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.935+0000 7f51e5ffb700 1 -- 172.21.15.104:0/124823857 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7f51ec057a30 con 0x7f51f00aec00 2023-06-10T18:51:39.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 msgr2=0x7f51d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:39.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 0x7f51d406d190 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51e000aaa0 tx=0x7f51e0008040 comp rx=0 tx=0).stop 2023-06-10T18:51:39.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 msgr2=0x7f51f01f24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00c7f0 tx=0x7f51ec00cb00 comp rx=0 tx=0).stop 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 shutdown_connections 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51d406ace0 0x7f51d406d190 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51f015c6f0 0x7f51f015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51f006ea70 0x7f51f0152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 --2- 172.21.15.104:0/124823857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51f00aec00 0x7f51f01f24f0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:39.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 >> 172.21.15.104:0/124823857 conn(0x7f51f006b8a0 msgr2=0x7f51f0155850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:39.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 shutdown_connections 2023-06-10T18:51:39.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:39.937+0000 7f51f7901700 1 -- 172.21.15.104:0/124823857 wait complete. 2023-06-10T18:51:39.945 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:51:39.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:51:40.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.076+0000 7f34fcccc700 1 Processor -- start 2023-06-10T18:51:40.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.076+0000 7f34fcccc700 1 -- start start 2023-06-10T18:51:40.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34fcccc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f815c580 0x7f34f80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34fcccc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f34f806cde0 0x7f34f80ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34fcccc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f806e250 con 0x7f34f806e6e0 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34fcccc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f815c950 con 0x7f34f806cde0 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34fcccc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b1e80 con 0x7f34f815c580 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f34f806cde0 0x7f34f80ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f34f806cde0 0x7f34f80ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60558/0 (socket says 172.21.15.104:60558) 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f5d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f34f806e6e0 0x7f34f80b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58384/0 (socket says 172.21.15.104:58384) 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 -- 172.21.15.104:0/1205644597 learned_addr learned my addr 172.21.15.104:0/1205644597 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:40.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f659c700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f815c580 0x7f34f80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 -- 172.21.15.104:0/1205644597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f815c580 msgr2=0x7f34f80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f815c580 0x7f34f80b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 -- 172.21.15.104:0/1205644597 >> v1:172.21.15.104:6789/0 conn(0x7f34f806e6e0 legacy=0x7f34f80b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f559a700 1 -- 172.21.15.104:0/1205644597 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690453648 0 0) 0x7f34f806e250 con 0x7f34f806e6e0 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 -- 172.21.15.104:0/1205644597 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f806d920 con 0x7f34f806cde0 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f6d9d700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806cde0 0x7f34f80ce0d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009e50 tx=0x7f34f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6651fc7b07e79dc1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.077+0000 7f34f559a700 1 -- 172.21.15.104:0/1205644597 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f000e070 con 0x7f34f806cde0 2023-06-10T18:51:40.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34f559a700 1 -- 172.21.15.104:0/1205644597 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f000b9c0 con 0x7f34f806cde0 2023-06-10T18:51:40.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34f559a700 1 -- 172.21.15.104:0/1205644597 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f000bd70 con 0x7f34f806cde0 2023-06-10T18:51:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 -- 172.21.15.104:0/1205644597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806cde0 msgr2=0x7f34f80ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806cde0 0x7f34f80ce0d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009e50 tx=0x7f34f000b330 comp rx=0 tx=0).stop 2023-06-10T18:51:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 -- 172.21.15.104:0/1205644597 shutdown_connections 2023-06-10T18:51:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806cde0 0x7f34f80ce0d0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1205644597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f815c580 0x7f34f80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 -- 172.21.15.104:0/1205644597 >> 172.21.15.104:0/1205644597 conn(0x7f34f806b8a0 msgr2=0x7f34f80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 -- 172.21.15.104:0/1205644597 shutdown_connections 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.078+0000 7f34fcccc700 1 -- 172.21.15.104:0/1205644597 wait complete. 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 Processor -- start 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 -- start start 2023-06-10T18:51:40.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 0x7f34f815e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f815c580 0x7f34f80cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f815ea50 con 0x7f34f806e6e0 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f815fa10 con 0x7f34f815c580 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34fcccc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80ac360 con 0x7f34f806cde0 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34f5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34f5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60564/0 (socket says 172.21.15.104:60564) 2023-06-10T18:51:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.080+0000 7f34f5d9b700 1 -- 172.21.15.104:0/1792732578 learned_addr learned my addr 172.21.15.104:0/1792732578 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f659c700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 0x7f34f815e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 -- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 msgr2=0x7f34f815e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 0x7f34f815e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 -- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f815c580 msgr2=0x7f34f80cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f815c580 0x7f34f80cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 -- 172.21.15.104:0/1792732578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f0009b30 con 0x7f34f806e6e0 2023-06-10T18:51:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f659c700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 0x7f34f815e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34f5d9b700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c750 tx=0x7f34ec00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec004c80 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec004e00 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f80ac640 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec007310 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.081+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f80acc00 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.082+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8074050 con 0x7f34f806e6e0 2023-06-10T18:51:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.082+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f34ec007470 con 0x7f34f806e6e0 2023-06-10T18:51:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.083+0000 7f34e6ffd700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 0x7f34d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.083+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f34ec089000 con 0x7f34f806e6e0 2023-06-10T18:51:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.083+0000 7f34f659c700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 0x7f34d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.083+0000 7f34f659c700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 0x7f34d806d190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f34e8008870 tx=0x7f34e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.087+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f34ec0542d0 con 0x7f34f806e6e0 2023-06-10T18:51:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.279+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f34f80cd250 con 0x7f34f806e6e0 2023-06-10T18:51:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.282+0000 7f34e6ffd700 1 -- 172.21.15.104:0/1792732578 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34ec0578f0 con 0x7f34f806e6e0 2023-06-10T18:51:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 msgr2=0x7f34d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 0x7f34d806d190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f34e8008870 tx=0x7f34e8008040 comp rx=0 tx=0).stop 2023-06-10T18:51:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 msgr2=0x7f34f815f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c750 tx=0x7f34ec00cb10 comp rx=0 tx=0).stop 2023-06-10T18:51:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 shutdown_connections 2023-06-10T18:51:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f34d806ace0 0x7f34d806d190 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f815c580 0x7f34f80cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34f806cde0 0x7f34f815e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 --2- 172.21.15.104:0/1792732578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34f806e6e0 0x7f34f815f4d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 >> 172.21.15.104:0/1792732578 conn(0x7f34f806b8a0 msgr2=0x7f34f8157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 shutdown_connections 2023-06-10T18:51:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.286+0000 7f34fcccc700 1 -- 172.21.15.104:0/1792732578 wait complete. 2023-06-10T18:51:40.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout' 2023-06-10T18:51:40.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.420+0000 7f7720d74700 1 Processor -- start 2023-06-10T18:51:40.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.420+0000 7f7720d74700 1 -- start start 2023-06-10T18:51:40.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7720d74700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f771c06d150 0x7f771c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7720d74700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c0cccd0 con 0x7f771c06b180 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7720d74700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c06a010 con 0x7f771c158c50 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7720d74700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c0ccfb0 con 0x7f771c06d150 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f771ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f771c06b180 0x7f771c0c9b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58400/0 (socket says 172.21.15.104:58400) 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f771ad9d700 1 -- 172.21.15.104:0/2898661224 learned_addr learned my addr 172.21.15.104:0/2898661224 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7719d9b700 1 --2- 172.21.15.104:0/2898661224 >> v2:172.21.15.114:3300/0 conn(0x7f771c06d150 0x7f771c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086330279 0 0) 0x7f771c0cccd0 con 0x7f771c06b180 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77040035b0 con 0x7f771c06b180 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7719d9b700 1 -- 172.21.15.104:0/2898661224 >> v1:172.21.15.114:6789/0 conn(0x7f771c158c50 legacy=0x7f771c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7719d9b700 1 -- 172.21.15.104:0/2898661224 >> v1:172.21.15.104:6789/0 conn(0x7f771c06b180 legacy=0x7f771c0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065376334 0 0) 0x7f77040035b0 con 0x7f771c06b180 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.421+0000 7f7719d9b700 1 -- 172.21.15.104:0/2898661224 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c06a2f0 con 0x7f771c06d150 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7719d9b700 1 --2- 172.21.15.104:0/2898661224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06d150 0x7f771c06aaf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7710009840 tx=0x7f7710009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23326734f763a26 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140019f0 con 0x7f771c06d150 2023-06-10T18:51:40.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710004030 con 0x7f771c06d150 2023-06-10T18:51:40.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f771959a700 1 -- 172.21.15.104:0/2898661224 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100043a0 con 0x7f771c06d150 2023-06-10T18:51:40.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 -- 172.21.15.104:0/2898661224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06d150 msgr2=0x7f771c06aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 --2- 172.21.15.104:0/2898661224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06d150 0x7f771c06aaf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7710009840 tx=0x7f7710009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:40.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 -- 172.21.15.104:0/2898661224 shutdown_connections 2023-06-10T18:51:40.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 --2- 172.21.15.104:0/2898661224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06d150 0x7f771c06aaf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 -- 172.21.15.104:0/2898661224 >> 172.21.15.104:0/2898661224 conn(0x7f771c0b1610 msgr2=0x7f771c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 -- 172.21.15.104:0/2898661224 shutdown_connections 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.422+0000 7f7720d74700 1 -- 172.21.15.104:0/2898661224 wait complete. 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 Processor -- start 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 -- start start 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06b180 0x7f771c154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f771c158c50 0x7f771c0c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1558e0 con 0x7f771c06d150 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0c71c0 con 0x7f771c06b180 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.424+0000 7f7720d74700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0c3ae0 con 0x7f771c158c50 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60576/0 (socket says 172.21.15.104:60576) 2023-06-10T18:51:40.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 -- 172.21.15.104:0/2254700648 learned_addr learned my addr 172.21.15.104:0/2254700648 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f771a59c700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06b180 0x7f771c154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f771ad9d700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f771c158c50 0x7f771c0c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 -- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f771c158c50 msgr2=0x7f771c0c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f771c158c50 0x7f771c0c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 -- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06b180 msgr2=0x7f771c154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06b180 0x7f771c154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 -- 172.21.15.104:0/2254700648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100094a0 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7719d9b700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7710009790 tx=0x7f7710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710003ea0 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771001f070 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c15a300 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710011870 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.425+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c15a8c0 con 0x7f771c06d150 2023-06-10T18:51:40.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.426+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c0c3ca0 con 0x7f771c06d150 2023-06-10T18:51:40.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.427+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f77100119d0 con 0x7f771c06d150 2023-06-10T18:51:40.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.427+0000 7f770affd700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:40.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.427+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7710021070 con 0x7f771c06d150 2023-06-10T18:51:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.427+0000 7f771a59c700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.428+0000 7f771a59c700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f771c0c59f0 tx=0x7f770c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:40.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.433+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7710055dd0 con 0x7f771c06d150 2023-06-10T18:51:40.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.700+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout"]} v 0) v1 -- 0x7f771c074050 con 0x7f771c06d150 2023-06-10T18:51:40.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.937+0000 7f770affd700 1 -- 172.21.15.104:0/2254700648 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_config_session_timeout"]}]=0 v339) v1 ==== 159+0+0 (secure 0 0 0) 0x7f77100593f0 con 0x7f771c06d150 2023-06-10T18:51:40.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 msgr2=0x7f76fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f771c0c59f0 tx=0x7f770c009040 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 msgr2=0x7f771c1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7710009790 tx=0x7f7710004750 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 shutdown_connections 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f771c06b180 0x7f771c154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f771c158c50 0x7f771c0c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 --2- 172.21.15.104:0/2254700648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f771c06d150 0x7f771c1553a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:40.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 >> 172.21.15.104:0/2254700648 conn(0x7f771c0b1610 msgr2=0x7f771c0cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:40.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 shutdown_connections 2023-06-10T18:51:40.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:40.940+0000 7f7720d74700 1 -- 172.21.15.104:0/2254700648 wait complete. 2023-06-10T18:51:40.958 INFO:tasks.cephfs_test_runner:test_config_session_timeout (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:51:40.958 INFO:tasks.cephfs_test_runner:Starting test: test_create_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:51:40.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync' 2023-06-10T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.074+0000 7f4efeec4700 1 Processor -- start 2023-06-10T18:51:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.074+0000 7f4efeec4700 1 -- start start 2023-06-10T18:51:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efeec4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef815a0b0 0x7f4ef80717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efeec4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4ef80c93e0 0x7f4ef815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efeec4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef806b2e0 con 0x7f4ef8070a20 2023-06-10T18:51:41.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efeec4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef815e150 con 0x7f4ef80c93e0 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efeec4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef815db40 con 0x7f4ef815a0b0 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4ef80c93e0 0x7f4ef815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efd461700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4ef8070a20 0x7f4ef815a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58414/0 (socket says 172.21.15.104:58414) 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efd461700 1 -- 172.21.15.104:0/3840261105 learned_addr learned my addr 172.21.15.104:0/3840261105 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4ef80c93e0 0x7f4ef815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60584/0 (socket says 172.21.15.104:60584) 2023-06-10T18:51:41.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4efcc60700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef815a0b0 0x7f4ef80717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 -- 172.21.15.104:0/3840261105 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef815a0b0 msgr2=0x7f4ef80717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef815a0b0 0x7f4ef80717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 -- 172.21.15.104:0/3840261105 >> v1:172.21.15.104:6789/0 conn(0x7f4ef8070a20 legacy=0x7f4ef815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 -- 172.21.15.104:0/3840261105 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef806b530 con 0x7f4ef80c93e0 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef77fe700 1 -- 172.21.15.104:0/3840261105 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399364086 0 0) 0x7f4ef806b2e0 con 0x7f4ef8070a20 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.075+0000 7f4ef7fff700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef815e5e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009840 tx=0x7f4ee4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fedd7352fe2616f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4ef77fe700 1 -- 172.21.15.104:0/3840261105 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4011830 con 0x7f4ef80c93e0 2023-06-10T18:51:41.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4ef77fe700 1 -- 172.21.15.104:0/3840261105 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee4011990 con 0x7f4ef80c93e0 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4ef77fe700 1 -- 172.21.15.104:0/3840261105 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4011d20 con 0x7f4ef80c93e0 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 -- 172.21.15.104:0/3840261105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 msgr2=0x7f4ef815e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef815e5e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009840 tx=0x7f4ee4009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 -- 172.21.15.104:0/3840261105 shutdown_connections 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef815a0b0 0x7f4ef80717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 --2- 172.21.15.104:0/3840261105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef815e5e0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 -- 172.21.15.104:0/3840261105 >> 172.21.15.104:0/3840261105 conn(0x7f4ef80b1610 msgr2=0x7f4ef80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 -- 172.21.15.104:0/3840261105 shutdown_connections 2023-06-10T18:51:41.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.076+0000 7f4efeec4700 1 -- 172.21.15.104:0/3840261105 wait complete. 2023-06-10T18:51:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.077+0000 7f4efeec4700 1 Processor -- start 2023-06-10T18:51:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.077+0000 7f4efeec4700 1 -- start start 2023-06-10T18:51:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef8070a20 0x7f4ef80cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 0x7f4ef80b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef80c2fc0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef80b4690 con 0x7f4ef815a0b0 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4efeec4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef80abfb0 con 0x7f4ef8070a20 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4ef7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4ef7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60590/0 (socket says 172.21.15.104:60590) 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.078+0000 7f4ef7fff700 1 -- 172.21.15.104:0/1848437433 learned_addr learned my addr 172.21.15.104:0/1848437433 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4efcc60700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef8070a20 0x7f4ef80cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4efd461700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 0x7f4ef80b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 -- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef8070a20 msgr2=0x7f4ef80cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef8070a20 0x7f4ef80cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 -- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 msgr2=0x7f4ef80b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 0x7f4ef80b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 -- 172.21.15.104:0/1848437433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee40094a0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4efd461700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 0x7f4ef80b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:41.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef7fff700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40090b0 tx=0x7f4ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40038a0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee401f070 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef80cda70 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4003c40 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.079+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef80ab960 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.080+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef8074050 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.080+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4ee4026480 con 0x7f4ef80c93e0 2023-06-10T18:51:41.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.081+0000 7f4ef57fa700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 0x7f4edc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.081+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4ee4021070 con 0x7f4ef80c93e0 2023-06-10T18:51:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.081+0000 7f4efcc60700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 0x7f4edc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.082+0000 7f4efcc60700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 0x7f4edc06d190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00a950 tx=0x7f4eec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:41.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.085+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4ee40560b0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.346+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync"]} v 0) v1 -- 0x7f4ef80ac0f0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.939+0000 7f4ef57fa700 1 -- 172.21.15.104:0/1848437433 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync"]}]=0 v340) v1 ==== 152+0+0 (secure 0 0 0) 0x7f4ee40596d0 con 0x7f4ef80c93e0 2023-06-10T18:51:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.941+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 msgr2=0x7f4edc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.941+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 0x7f4edc06d190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00a950 tx=0x7f4eec008040 comp rx=0 tx=0).stop 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.941+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 msgr2=0x7f4ef8157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.941+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40090b0 tx=0x7f4ee4004750 comp rx=0 tx=0).stop 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 shutdown_connections 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4edc06ace0 0x7f4edc06d190 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ef815a0b0 0x7f4ef80b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ef8070a20 0x7f4ef80cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 --2- 172.21.15.104:0/1848437433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ef80c93e0 0x7f4ef8157e50 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:41.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 >> 172.21.15.104:0/1848437433 conn(0x7f4ef80b1610 msgr2=0x7f4ef81e3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:41.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 shutdown_connections 2023-06-10T18:51:41.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:41.942+0000 7f4efeec4700 1 -- 172.21.15.104:0/1848437433 wait complete. 2023-06-10T18:51:41.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:51:42.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 Processor -- start 2023-06-10T18:51:42.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 -- start start 2023-06-10T18:51:42.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f05bc0d04f0 0x7f05bc0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05bc0b05d0 0x7f05bc0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc0af530 con 0x7f05bc0d25c0 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0af810 con 0x7f05bc0d04f0 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.082+0000 7f05c1354700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0afaf0 con 0x7f05bc0b05d0 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f05bc0d04f0 0x7f05bc0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f05bc0d04f0 0x7f05bc0d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60596/0 (socket says 172.21.15.104:60596) 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 -- 172.21.15.104:0/1008297291 learned_addr learned my addr 172.21.15.104:0/1008297291 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05bb7fe700 1 --2- 172.21.15.104:0/1008297291 >> v2:172.21.15.114:3300/0 conn(0x7f05bc0b05d0 0x7f05bc0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 -- 172.21.15.104:0/1008297291 >> v2:172.21.15.114:3300/0 conn(0x7f05bc0b05d0 msgr2=0x7f05bc0b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 --2- 172.21.15.104:0/1008297291 >> v2:172.21.15.114:3300/0 conn(0x7f05bc0b05d0 0x7f05bc0b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 -- 172.21.15.104:0/1008297291 >> v1:172.21.15.114:6789/0 conn(0x7f05bc0d25c0 legacy=0x7f05bc0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 -- 172.21.15.104:0/1008297291 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0afdd0 con 0x7f05bc0d04f0 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05baffd700 1 --2- 172.21.15.104:0/1008297291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d04f0 0x7f05bc0d34d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c9970 tx=0x7f05b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e08c558eb90dd4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05b9ffb700 1 -- 172.21.15.104:0/1008297291 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0005e30 con 0x7f05bc0d04f0 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05b9ffb700 1 -- 172.21.15.104:0/1008297291 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0006030 con 0x7f05bc0d04f0 2023-06-10T18:51:42.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.083+0000 7f05b9ffb700 1 -- 172.21.15.104:0/1008297291 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00063e0 con 0x7f05bc0d04f0 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 -- 172.21.15.104:0/1008297291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d04f0 msgr2=0x7f05bc0d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 --2- 172.21.15.104:0/1008297291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d04f0 0x7f05bc0d34d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c9970 tx=0x7f05b0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 -- 172.21.15.104:0/1008297291 shutdown_connections 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 --2- 172.21.15.104:0/1008297291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d04f0 0x7f05bc0d34d0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 --2- 172.21.15.104:0/1008297291 >> v2:172.21.15.114:3300/0 conn(0x7f05bc0b05d0 0x7f05bc0b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 -- 172.21.15.104:0/1008297291 >> 172.21.15.104:0/1008297291 conn(0x7f05bc071ec0 msgr2=0x7f05bc09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 -- 172.21.15.104:0/1008297291 shutdown_connections 2023-06-10T18:51:42.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.084+0000 7f05c1354700 1 -- 172.21.15.104:0/1008297291 wait complete. 2023-06-10T18:51:42.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.085+0000 7f05c1354700 1 Processor -- start 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.085+0000 7f05c1354700 1 -- start start 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05bc0b05d0 0x7f05bc15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 0x7f05bc09f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc09e140 con 0x7f05bc0d25c0 2023-06-10T18:51:42.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b3a00 con 0x7f05bc0d04f0 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05c1354700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b3fc0 con 0x7f05bc0b05d0 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60606/0 (socket says 172.21.15.104:60606) 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 -- 172.21.15.104:0/1838159390 learned_addr learned my addr 172.21.15.104:0/1838159390 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05baffd700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05bc0b05d0 0x7f05bc15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05ba7fc700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 0x7f05bc09f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 -- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05bc0b05d0 msgr2=0x7f05bc15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05bc0b05d0 0x7f05bc15ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 -- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 msgr2=0x7f05bc09f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 0x7f05bc09f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 -- 172.21.15.104:0/1838159390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009500 con 0x7f05bc0d25c0 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05ba7fc700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 0x7f05bc09f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05bb7fe700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f05b4007e40 tx=0x7f05b400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.086+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400eea0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.087+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4014420 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.087+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc15b1c0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.087+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40147b0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.087+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc0a0150 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.087+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc0b4170 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.088+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f05b4014910 con 0x7f05bc0d25c0 2023-06-10T18:51:42.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.089+0000 7f05a37fe700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 0x7f05a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.089+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f05b408a720 con 0x7f05bc0d25c0 2023-06-10T18:51:42.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.089+0000 7f05baffd700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 0x7f05a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.090+0000 7f05baffd700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 0x7f05a806d190 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c9970 tx=0x7f05b0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.093+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05b4055ad0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.279+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f05bc0a05c0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.283+0000 7f05a37fe700 1 -- 172.21.15.104:0/1838159390 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f05b40590f0 con 0x7f05bc0d25c0 2023-06-10T18:51:42.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 msgr2=0x7f05a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 0x7f05a806d190 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c9970 tx=0x7f05b0008070 comp rx=0 tx=0).stop 2023-06-10T18:51:42.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 msgr2=0x7f05bc09fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f05b4007e40 tx=0x7f05b400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 shutdown_connections 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05bc0d25c0 0x7f05bc09fa60 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05bc0b05d0 0x7f05bc15ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05a806ace0 0x7f05a806d190 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 --2- 172.21.15.104:0/1838159390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05bc0d04f0 0x7f05bc09f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 >> 172.21.15.104:0/1838159390 conn(0x7f05bc071ec0 msgr2=0x7f05bc0b6080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 shutdown_connections 2023-06-10T18:51:42.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.285+0000 7f05c1354700 1 -- 172.21.15.104:0/1838159390 wait complete. 2023-06-10T18:51:42.303 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:51:42.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:51:42.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:51:42.400 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:51:42.400 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:51:42.401 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:42.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:51:42.418 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:51:42.418 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:42.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:51:42.437 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:51:42.437 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:42.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:51:42.474 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:51:42.474 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:42.496 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:51:42.496 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:51:42.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.627+0000 7fd4770fd700 1 Processor -- start 2023-06-10T18:51:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.627+0000 7fd4770fd700 1 -- start start 2023-06-10T18:51:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd4770fd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd47006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd4770fd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4700cb0c0 0x7fd4700cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd4770fd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47015cb00 con 0x7fd4700cc000 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd4770fd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4700aed80 con 0x7fd4700cb0c0 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd4770fd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700c9c60 con 0x7fd47015c730 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd4700cc000 0x7fd47006cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58438/0 (socket says 172.21.15.104:58438) 2023-06-10T18:51:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46ffff700 1 -- 172.21.15.104:0/595302191 learned_addr learned my addr 172.21.15.104:0/595302191 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809343466 0 0) 0x7fd45c001110 con 0x7fd4700cc000 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4600035b0 con 0x7fd4700cc000 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd474e99700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd47006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd47569a700 1 --2- 172.21.15.104:0/595302191 >> v2:172.21.15.114:3300/0 conn(0x7fd4700cb0c0 0x7fd4700cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819256647 0 0) 0x7fd4600035b0 con 0x7fd4700cc000 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 msgr2=0x7fd47006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd47006e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 >> v2:172.21.15.114:3300/0 conn(0x7fd4700cb0c0 msgr2=0x7fd4700cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 --2- 172.21.15.104:0/595302191 >> v2:172.21.15.114:3300/0 conn(0x7fd4700cb0c0 0x7fd4700cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47006ecd0 con 0x7fd4700cc000 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.628+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd45c001800 con 0x7fd4700cc000 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 >> v1:172.21.15.104:6789/0 conn(0x7fd4700cc000 legacy=0x7fd47006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd460003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd460003f80 0x7fd460004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd460004d90 0x7fd460005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700aed80 con 0x7fd460003f80 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700c9c60 con 0x7fd460004d90 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46000ba40 con 0x7fd47015c730 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd460003f80 0x7fd460004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd474e99700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd460003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd47569a700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd460004d90 0x7fd460005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 -- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 msgr2=0x7fd460003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd460003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 -- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd460004d90 msgr2=0x7fd460005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd460004d90 0x7fd460005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 -- 172.21.15.104:0/595302191 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600035b0 con 0x7fd460003f80 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd474e99700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd47015c730 0x7fd460003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46ffff700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd460003f80 0x7fd460004850 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c2f0 tx=0x7fd460005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aee847396bb6be0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00cdd0 con 0x7fd460003f80 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c005960 con 0x7fd460003f80 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.629+0000 7fd46f7fe700 1 -- 172.21.15.104:0/595302191 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c005d10 con 0x7fd460003f80 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 -- 172.21.15.104:0/595302191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd460003f80 msgr2=0x7fd460004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 --2- 172.21.15.104:0/595302191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd460003f80 0x7fd460004850 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c2f0 tx=0x7fd460005880 comp rx=0 tx=0).stop 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd47569a700 1 -- 172.21.15.104:0/595302191 reap_dead start 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 -- 172.21.15.104:0/595302191 shutdown_connections 2023-06-10T18:51:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 -- 172.21.15.104:0/595302191 >> 172.21.15.104:0/595302191 conn(0x7fd47006b8a0 msgr2=0x7fd4700cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 -- 172.21.15.104:0/595302191 shutdown_connections 2023-06-10T18:51:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.630+0000 7fd4770fd700 1 -- 172.21.15.104:0/595302191 wait complete. 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.631+0000 7fd4770fd700 1 Processor -- start 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.631+0000 7fd4770fd700 1 -- start start 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 0x7fd4700b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4700b4930 0x7fd4700aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700af810 con 0x7fd4700c2330 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700afaf0 con 0x7fd4700b4930 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd4770fd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700afdd0 con 0x7fd460003f80 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60626/0 (socket says 172.21.15.104:60626) 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 -- 172.21.15.104:0/3365425227 learned_addr learned my addr 172.21.15.104:0/3365425227 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd474e99700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 0x7fd4700b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd46ffff700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4700b4930 0x7fd4700aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 -- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 msgr2=0x7fd4700b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 0x7fd4700b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 -- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4700b4930 msgr2=0x7fd4700aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4700b4930 0x7fd4700aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 -- 172.21.15.104:0/3365425227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4700b00b0 con 0x7fd4700c2330 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd474e99700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 0x7fd4700b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.632+0000 7fd47569a700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd4700aeec0 tx=0x7fd468011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46800e040 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46800f040 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4700b0390 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46800fdb0 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4700c27a0 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.633+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd470074050 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.634+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd468022460 con 0x7fd4700c2330 2023-06-10T18:51:42.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.634+0000 7fd46d7fa700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 0x7fd45406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:42.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.634+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd468089710 con 0x7fd4700c2330 2023-06-10T18:51:42.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.635+0000 7fd474e99700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 0x7fd45406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:42.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.635+0000 7fd474e99700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 0x7fd45406d190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd464009b60 tx=0x7fd464008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:42.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.639+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd468054950 con 0x7fd4700c2330 2023-06-10T18:51:42.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.867+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4700cf260 con 0x7fd4700c2330 2023-06-10T18:51:42.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.868+0000 7fd46d7fa700 1 -- 172.21.15.104:0/3365425227 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fd468057f70 con 0x7fd4700c2330 2023-06-10T18:51:42.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:42.874 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5543,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3076885008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3076885008},{"type":"v1","addr":"172.21.15.104:6835","nonce":3076885008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2984842729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2984842729},{"type":"v1","addr":"172.21.15.114:6837","nonce":2984842729}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5786,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2620737663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2620737663},{"type":"v1","addr":"172.21.15.114:6836","nonce":2620737663}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/4228970313","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":4228970313},{"type":"v1","addr":"172.21.15.104:6833","nonce":4228970313}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[]} 2023-06-10T18:51:42.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 msgr2=0x7fd45406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 0x7fd45406d190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd464009b60 tx=0x7fd464008040 comp rx=0 tx=0).stop 2023-06-10T18:51:42.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 msgr2=0x7fd470069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:42.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd4700aeec0 tx=0x7fd468011f80 comp rx=0 tx=0).stop 2023-06-10T18:51:42.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 shutdown_connections 2023-06-10T18:51:42.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4700c2330 0x7fd470069e70 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd460003f80 0x7fd4700b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd45406ace0 0x7fd45406d190 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 --2- 172.21.15.104:0/3365425227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4700b4930 0x7fd4700aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 >> 172.21.15.104:0/3365425227 conn(0x7fd47006b8a0 msgr2=0x7fd470072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 shutdown_connections 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:42.870+0000 7fd4770fd700 1 -- 172.21.15.104:0/3365425227 wait complete. 2023-06-10T18:51:42.877 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 35 2023-06-10T18:51:42.888 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:51:42.888 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:51:42.888 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:51:42.889 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:51:42.889 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:51:42.889 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:51:42.890 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:51:42.890 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:51:42.890 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:51:42.890 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:51:42.891 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:51:42.891 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:51:42.891 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:51:42.892 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:51:42.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:51:42.892 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:51:42.898 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:51:42.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:51:42.899 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:51:42.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:51:42.901 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:51:42.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:51:42.903 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:51:42.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:51:42.939 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:51:42.940 DEBUG:teuthology.parallel:result is None 2023-06-10T18:51:42.940 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:51:42.941 DEBUG:teuthology.parallel:result is None 2023-06-10T18:51:42.945 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:51:42.946 DEBUG:teuthology.parallel:result is None 2023-06-10T18:51:42.947 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:51:42.948 DEBUG:teuthology.parallel:result is None 2023-06-10T18:51:42.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:51:43.018 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:51:43.018 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:51:43.031 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:51:43.031 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:51:43.032 INFO:tasks.ceph.mds.b.smithi114.stderr:2023-06-10T18:51:43.027+0000 7ff26bc26700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2023-06-10T18:51:43.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.073+0000 7fb679c1a700 1 Processor -- start 2023-06-10T18:51:43.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.073+0000 7fb679c1a700 1 -- start start 2023-06-10T18:51:43.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb679c1a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb6740ce390 0x7fb674160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb679c1a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6740cae10 0x7fb6740cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb679c1a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6740d0340 con 0x7fb67406c720 2023-06-10T18:51:43.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb679c1a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6740d1660 con 0x7fb6740ce390 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb679c1a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67415cb60 con 0x7fb6740cae10 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb6737fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb6740ce390 0x7fb674160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb6737fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb6740ce390 0x7fb674160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60652/0 (socket says 172.21.15.104:60652) 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb6737fe700 1 -- 172.21.15.104:0/3946075773 learned_addr learned my addr 172.21.15.104:0/3946075773 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.074+0000 7fb673fff700 1 --2- 172.21.15.104:0/3946075773 >> v2:172.21.15.114:3300/0 conn(0x7fb6740cae10 0x7fb6740cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6737fe700 1 -- 172.21.15.104:0/3946075773 >> v2:172.21.15.114:3300/0 conn(0x7fb6740cae10 msgr2=0x7fb6740cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:43.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6737fe700 1 --2- 172.21.15.104:0/3946075773 >> v2:172.21.15.114:3300/0 conn(0x7fb6740cae10 0x7fb6740cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6737fe700 1 -- 172.21.15.104:0/3946075773 >> v1:172.21.15.114:6789/0 conn(0x7fb67406c720 legacy=0x7fb6740ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6737fe700 1 -- 172.21.15.104:0/3946075773 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674161230 con 0x7fb6740ce390 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6737fe700 1 --2- 172.21.15.104:0/3946075773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6740ce390 0x7fb674160cf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fb67406b1a0 tx=0x7fb668009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08cc388b0c2b4f4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6727fc700 1 -- 172.21.15.104:0/3946075773 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668011830 con 0x7fb6740ce390 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6727fc700 1 -- 172.21.15.104:0/3946075773 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb668011990 con 0x7fb6740ce390 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb6727fc700 1 -- 172.21.15.104:0/3946075773 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668011d20 con 0x7fb6740ce390 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 -- 172.21.15.104:0/3946075773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6740ce390 msgr2=0x7fb674160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:43.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3946075773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6740ce390 0x7fb674160cf0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fb67406b1a0 tx=0x7fb668009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 -- 172.21.15.104:0/3946075773 shutdown_connections 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3946075773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6740ce390 0x7fb674160cf0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3946075773 >> v2:172.21.15.114:3300/0 conn(0x7fb6740cae10 0x7fb6740cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.075+0000 7fb679c1a700 1 -- 172.21.15.104:0/3946075773 >> 172.21.15.104:0/3946075773 conn(0x7fb674069e70 msgr2=0x7fb67406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.076+0000 7fb679c1a700 1 -- 172.21.15.104:0/3946075773 shutdown_connections 2023-06-10T18:51:43.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.076+0000 7fb679c1a700 1 -- 172.21.15.104:0/3946075773 wait complete. 2023-06-10T18:51:43.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.077+0000 7fb679c1a700 1 Processor -- start 2023-06-10T18:51:43.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.077+0000 7fb679c1a700 1 -- start start 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6740cae10 0x7fb6741e8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 0x7fb674150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6740ca310 con 0x7fb67406c720 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6740c9a70 con 0x7fb6740ce390 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb679c1a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6740c9d50 con 0x7fb6740cae10 2023-06-10T18:51:43.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb6737fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb6737fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60656/0 (socket says 172.21.15.104:60656) 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb6737fe700 1 -- 172.21.15.104:0/3381089623 learned_addr learned my addr 172.21.15.104:0/3381089623 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb672ffd700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6740cae10 0x7fb6741e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.078+0000 7fb673fff700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 0x7fb674150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 -- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6740cae10 msgr2=0x7fb6741e8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6740cae10 0x7fb6741e8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 -- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 msgr2=0x7fb674150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:43.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 0x7fb674150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 -- 172.21.15.104:0/3381089623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668009550 con 0x7fb67406c720 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb673fff700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 0x7fb674150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb6737fe700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb668004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668004380 con 0x7fb67406c720 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66801f070 con 0x7fb67406c720 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6740ca030 con 0x7fb67406c720 2023-06-10T18:51:43.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6680036a0 con 0x7fb67406c720 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.079+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6740b5ca0 con 0x7fb67406c720 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.080+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb674074050 con 0x7fb67406c720 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.080+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb668021030 con 0x7fb67406c720 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.081+0000 7fb65bfff700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.081+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb66808a770 con 0x7fb67406c720 2023-06-10T18:51:43.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.081+0000 7fb672ffd700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:43.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.081+0000 7fb672ffd700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 0x7fb66006d190 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb664007fd0 tx=0x7fb664009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:43.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.085+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb668055a70 con 0x7fb67406c720 2023-06-10T18:51:43.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:43.284+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb6740d0e50 con 0x7fb67406c720 2023-06-10T18:51:44.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.063+0000 7fb65bfff700 1 -- 172.21.15.104:0/3381089623 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb668059090 con 0x7fb67406c720 2023-06-10T18:51:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 msgr2=0x7fb66006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 0x7fb66006d190 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb664007fd0 tx=0x7fb664009040 comp rx=0 tx=0).stop 2023-06-10T18:51:44.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 msgr2=0x7fb6740b5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb668004750 comp rx=0 tx=0).stop 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 shutdown_connections 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6740ce390 0x7fb674150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb67406c720 0x7fb6740b5760 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.066+0000 7fb679c1a700 1 --2- 172.21.15.104:0/3381089623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6740cae10 0x7fb6741e8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.067+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 >> 172.21.15.104:0/3381089623 conn(0x7fb674069e70 msgr2=0x7fb6740ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.067+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 shutdown_connections 2023-06-10T18:51:44.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.067+0000 7fb679c1a700 1 -- 172.21.15.104:0/3381089623 wait complete. 2023-06-10T18:51:44.074 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:51:44.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:51:44.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.206+0000 7ff3f5c65700 1 Processor -- start 2023-06-10T18:51:44.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 -- start start 2023-06-10T18:51:44.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff3f006e4b0 0x7ff3f006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f006d290 con 0x7ff3f006e0e0 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f0071640 con 0x7ff3f006e4b0 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.207+0000 7ff3f5c65700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00aef00 con 0x7ff3f0069e70 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3effff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff3f006e4b0 0x7ff3f006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff3f006e4b0 0x7ff3f006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:48172/0 (socket says 172.21.15.104:48172) 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3072655095 learned_addr learned my addr 172.21.15.104:0/3072655095 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3effff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:51824/0 (socket says 172.21.15.104:51824) 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3ef7fe700 1 --1- 172.21.15.104:0/3072655095 >> v1:172.21.15.114:6789/0 conn(0x7ff3f006e0e0 0x7ff3f00c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:36110/0 (socket says 172.21.15.104:36110) 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 msgr2=0x7ff3f00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3072655095 >> v1:172.21.15.114:6789/0 conn(0x7ff3f006e0e0 legacy=0x7ff3f00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3072655095 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00ca2d0 con 0x7ff3f006e4b0 2023-06-10T18:51:44.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3effff700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:44.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3eeffd700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f006e4b0 0x7ff3f006cd50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00098f0 tx=0x7ff3e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676122ba16daea0d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.208+0000 7ff3ee7fc700 1 -- 172.21.15.104:0/3072655095 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0011830 con 0x7ff3f006e4b0 2023-06-10T18:51:44.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3ee7fc700 1 -- 172.21.15.104:0/3072655095 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0011990 con 0x7ff3f006e4b0 2023-06-10T18:51:44.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3ee7fc700 1 -- 172.21.15.104:0/3072655095 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0011d20 con 0x7ff3f006e4b0 2023-06-10T18:51:44.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f006e4b0 msgr2=0x7ff3f006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f006e4b0 0x7ff3f006cd50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00098f0 tx=0x7ff3e0009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3072655095 shutdown_connections 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f0069e70 0x7ff3f00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3072655095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f006e4b0 0x7ff3f006cd50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3072655095 >> 172.21.15.104:0/3072655095 conn(0x7ff3f006b8a0 msgr2=0x7ff3f00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3072655095 shutdown_connections 2023-06-10T18:51:44.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.209+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3072655095 wait complete. 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.210+0000 7ff3f5c65700 1 Processor -- start 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.210+0000 7ff3f5c65700 1 -- start start 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 0x7ff3f00b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f006e4b0 0x7ff3f00b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00af170 con 0x7ff3f006e0e0 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00af450 con 0x7ff3f0069e70 2023-06-10T18:51:44.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3f5c65700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00af730 con 0x7ff3f006e4b0 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60676/0 (socket says 172.21.15.104:60676) 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3832977987 learned_addr learned my addr 172.21.15.104:0/3832977987 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3ef7fe700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 0x7ff3f00b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.211+0000 7ff3effff700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f006e4b0 0x7ff3f00b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f006e4b0 msgr2=0x7ff3f00b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f006e4b0 0x7ff3f00b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 msgr2=0x7ff3f00b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 0x7ff3f00b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 -- 172.21.15.104:0/3832977987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8009a70 con 0x7ff3f006e0e0 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3ef7fe700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 0x7ff3f00b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3eeffd700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009c40 tx=0x7ff3e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00036a0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0003800 con 0x7ff3f006e0e0 2023-06-10T18:51:44.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0009550 con 0x7ff3f006e0e0 2023-06-10T18:51:44.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0003bb0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.212+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f00afcf0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.213+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f0074050 con 0x7ff3f006e0e0 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.213+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff3e0003d10 con 0x7ff3f006e0e0 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.214+0000 7ff3d7fff700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 0x7ff3dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.214+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff3e008a4c0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.214+0000 7ff3ef7fe700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 0x7ff3dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.214+0000 7ff3ef7fe700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 0x7ff3dc06d190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4007fd0 tx=0x7ff3e4007d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.218+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3e0055790 con 0x7ff3f006e0e0 2023-06-10T18:51:44.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.482+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff3f015c0f0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.482+0000 7ff3d7fff700 1 -- 172.21.15.104:0/3832977987 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff3e0058db0 con 0x7ff3f006e0e0 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:51:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:51:44.491 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:51:44.492 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:51:44.494 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.495 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:51:44.496 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.497 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.501 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:51:44.502 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:51:44.503 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:51:44.508 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:51:44.511 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:51:44.512 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:51:44.513 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 msgr2=0x7ff3dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 0x7ff3dc06d190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4007fd0 tx=0x7ff3e4007d80 comp rx=0 tx=0).stop 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 msgr2=0x7ff3f00b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009c40 tx=0x7ff3e0004750 comp rx=0 tx=0).stop 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 shutdown_connections 2023-06-10T18:51:44.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff3dc06ace0 0x7ff3dc06d190 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3f0069e70 0x7ff3f00b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3f006e4b0 0x7ff3f00b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 --2- 172.21.15.104:0/3832977987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f006e0e0 0x7ff3f00b1870 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 >> 172.21.15.104:0/3832977987 conn(0x7ff3f006b8a0 msgr2=0x7ff3f00ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 shutdown_connections 2023-06-10T18:51:44.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.484+0000 7ff3f5c65700 1 -- 172.21.15.104:0/3832977987 wait complete. 2023-06-10T18:51:44.517 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:44.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.631+0000 7f8004bb3700 1 Processor -- start 2023-06-10T18:51:44.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.631+0000 7f8004bb3700 1 -- start start 2023-06-10T18:51:44.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f8004bb3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80000cd850 0x7f800006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f8004bb3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80000cb300 0x7f800006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f8004bb3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80000c9e60 con 0x7f80000aeaa0 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f8004bb3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000c9bc0 con 0x7f80000cb300 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f8004bb3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000aee70 con 0x7f80000cd850 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f7ffed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80000cb300 0x7f800006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f7ffe59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f80000aeaa0 0x7f80000cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58510/0 (socket says 172.21.15.104:58510) 2023-06-10T18:51:44.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f7ffe59c700 1 -- 172.21.15.104:0/3637707460 learned_addr learned my addr 172.21.15.104:0/3637707460 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.632+0000 7f7ffdd9b700 1 --2- 172.21.15.104:0/3637707460 >> v2:172.21.15.114:3300/0 conn(0x7f80000cd850 0x7f800006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffed9d700 1 -- 172.21.15.104:0/3637707460 >> v2:172.21.15.114:3300/0 conn(0x7f80000cd850 msgr2=0x7f800006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffed9d700 1 --2- 172.21.15.104:0/3637707460 >> v2:172.21.15.114:3300/0 conn(0x7f80000cd850 0x7f800006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffed9d700 1 -- 172.21.15.104:0/3637707460 >> v1:172.21.15.104:6789/0 conn(0x7f80000aeaa0 legacy=0x7f80000cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffed9d700 1 -- 172.21.15.104:0/3637707460 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800006e540 con 0x7f80000cb300 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffd59a700 1 -- 172.21.15.104:0/3637707460 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105285727 0 0) 0x7f80000c9e60 con 0x7f80000aeaa0 2023-06-10T18:51:44.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffed9d700 1 --2- 172.21.15.104:0/3637707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cb300 0x7f800006d290 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f8000070110 tx=0x7f7ff800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a4a08aedbe9070 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffd59a700 1 -- 172.21.15.104:0/3637707460 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800e070 con 0x7f80000cb300 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffd59a700 1 -- 172.21.15.104:0/3637707460 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff800bbd0 con 0x7f80000cb300 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f7ffd59a700 1 -- 172.21.15.104:0/3637707460 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8012430 con 0x7f80000cb300 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 -- 172.21.15.104:0/3637707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cb300 msgr2=0x7f800006d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 --2- 172.21.15.104:0/3637707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cb300 0x7f800006d290 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f8000070110 tx=0x7f7ff800b330 comp rx=0 tx=0).stop 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 -- 172.21.15.104:0/3637707460 shutdown_connections 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 --2- 172.21.15.104:0/3637707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cb300 0x7f800006d290 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 --2- 172.21.15.104:0/3637707460 >> v2:172.21.15.114:3300/0 conn(0x7f80000cd850 0x7f800006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.633+0000 7f8004bb3700 1 -- 172.21.15.104:0/3637707460 >> 172.21.15.104:0/3637707460 conn(0x7f800006b8a0 msgr2=0x7f80000cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.634+0000 7f8004bb3700 1 -- 172.21.15.104:0/3637707460 shutdown_connections 2023-06-10T18:51:44.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.634+0000 7f8004bb3700 1 -- 172.21.15.104:0/3637707460 wait complete. 2023-06-10T18:51:44.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.635+0000 7f8004bb3700 1 Processor -- start 2023-06-10T18:51:44.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.635+0000 7f8004bb3700 1 -- start start 2023-06-10T18:51:44.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80000aeaa0 0x7f800015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80000cb300 0x7f80001581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800015b5e0 con 0x7f80000cd850 2023-06-10T18:51:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8000159470 con 0x7f80000cb300 2023-06-10T18:51:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f8004bb3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8000159750 con 0x7f80000aeaa0 2023-06-10T18:51:44.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f7ffed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f7ffed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60692/0 (socket says 172.21.15.104:60692) 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.636+0000 7f7ffed9d700 1 -- 172.21.15.104:0/4106642859 learned_addr learned my addr 172.21.15.104:0/4106642859 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffe59c700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80000aeaa0 0x7f800015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffdd9b700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80000cb300 0x7f80001581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 -- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80000aeaa0 msgr2=0x7f800015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80000aeaa0 0x7f800015b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 -- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80000cb300 msgr2=0x7f80001581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80000cb300 0x7f80001581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 -- 172.21.15.104:0/4106642859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009a70 con 0x7f80000cd850 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7ffed9d700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f800006cc20 tx=0x7f7ff8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8012430 con 0x7f80000cd850 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff8020070 con 0x7f80000cd850 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8000159a30 con 0x7f80000cd850 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8004280 con 0x7f80000cd850 2023-06-10T18:51:44.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.637+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8000159ff0 con 0x7f80000cd850 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.638+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8000074050 con 0x7f80000cd850 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.638+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7ff80043e0 con 0x7f80000cd850 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.639+0000 7f7feeffd700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 0x7f7fe006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.639+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7ff8022070 con 0x7f80000cd850 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.639+0000 7f7ffe59c700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 0x7f7fe006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.639+0000 7f7ffe59c700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 0x7f7fe006d190 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4001360 tx=0x7f7ff4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.643+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7ff8056740 con 0x7f80000cd850 2023-06-10T18:51:44.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.830+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8000071560 con 0x7f80000cd850 2023-06-10T18:51:44.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.831+0000 7f7feeffd700 1 -- 172.21.15.104:0/4106642859 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+14893 (secure 0 0 0) 0x7f7ff8059d60 con 0x7f80000cd850 2023-06-10T18:51:44.837 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:44.837 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":79,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:44.055358+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:44.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 msgr2=0x7f7fe006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 0x7f7fe006d190 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4001360 tx=0x7f7ff4009040 comp rx=0 tx=0).stop 2023-06-10T18:51:44.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 msgr2=0x7f8000158720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f800006cc20 tx=0x7f7ff8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 shutdown_connections 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80000cd850 0x7f8000158720 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80000aeaa0 0x7f800015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fe006ace0 0x7f7fe006d190 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 --2- 172.21.15.104:0/4106642859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80000cb300 0x7f80001581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 >> 172.21.15.104:0/4106642859 conn(0x7f800006b8a0 msgr2=0x7f8000069ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 shutdown_connections 2023-06-10T18:51:44.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.833+0000 7f8004bb3700 1 -- 172.21.15.104:0/4106642859 wait complete. 2023-06-10T18:51:44.851 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:51:44.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:51:44.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.969+0000 7f7fee4df700 1 Processor -- start 2023-06-10T18:51:44.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.969+0000 7f7fee4df700 1 -- start start 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fee4df700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fe806e6e0 0x7f7fe80b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fee4df700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe806e250 con 0x7f7fe815c580 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fee4df700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe815c950 con 0x7f7fe806cde0 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fee4df700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80b1e80 con 0x7f7fe806e6e0 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe77fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fe806e6e0 0x7f7fe80b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7fe815c580 0x7f7fe80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58524/0 (socket says 172.21.15.104:58524) 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe7fff700 1 -- 172.21.15.104:0/2142284461 learned_addr learned my addr 172.21.15.104:0/2142284461 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe6ffd700 1 -- 172.21.15.104:0/2142284461 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578130790 0 0) 0x7f7fe806e250 con 0x7f7fe815c580 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe6ffd700 1 -- 172.21.15.104:0/2142284461 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd00035b0 con 0x7f7fe815c580 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe77fe700 1 -- 172.21.15.104:0/2142284461 >> v1:172.21.15.114:6789/0 conn(0x7f7fe806cde0 legacy=0x7f7fe80ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe77fe700 1 -- 172.21.15.104:0/2142284461 >> v1:172.21.15.104:6789/0 conn(0x7f7fe815c580 legacy=0x7f7fe80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.970+0000 7f7fe77fe700 1 -- 172.21.15.104:0/2142284461 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe806d920 con 0x7f7fe806e6e0 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fe77fe700 1 --2- 172.21.15.104:0/2142284461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe806e6e0 0x7f7fe80b1780 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009840 tx=0x7f7fd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a32ba0267db6586 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fe6ffd700 1 -- 172.21.15.104:0/2142284461 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8004230 con 0x7f7fe806e6e0 2023-06-10T18:51:44.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fe6ffd700 1 -- 172.21.15.104:0/2142284461 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8004390 con 0x7f7fe806e6e0 2023-06-10T18:51:44.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fe6ffd700 1 -- 172.21.15.104:0/2142284461 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80036c0 con 0x7f7fe806e6e0 2023-06-10T18:51:44.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 -- 172.21.15.104:0/2142284461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe806e6e0 msgr2=0x7f7fe80b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2142284461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe806e6e0 0x7f7fe80b1780 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009840 tx=0x7f7fd8009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:44.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 -- 172.21.15.104:0/2142284461 shutdown_connections 2023-06-10T18:51:44.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2142284461 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe806e6e0 0x7f7fe80b1780 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 -- 172.21.15.104:0/2142284461 >> 172.21.15.104:0/2142284461 conn(0x7f7fe806b8a0 msgr2=0x7f7fe80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:44.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 -- 172.21.15.104:0/2142284461 shutdown_connections 2023-06-10T18:51:44.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.971+0000 7f7fee4df700 1 -- 172.21.15.104:0/2142284461 wait complete. 2023-06-10T18:51:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.972+0000 7f7fee4df700 1 Processor -- start 2023-06-10T18:51:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 -- start start 2023-06-10T18:51:44.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fe806cde0 0x7f7fe814f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fe806e6e0 0x7f7fe80b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe81e1840 con 0x7f7fe815c580 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe81e1b20 con 0x7f7fe806cde0 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7fee4df700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe81f1cc0 con 0x7f7fe806e6e0 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7feca7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.973+0000 7f7feca7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60706/0 (socket says 172.21.15.104:60706) 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 -- 172.21.15.104:0/2528765854 learned_addr learned my addr 172.21.15.104:0/2528765854 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fe7fff700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fe806cde0 0x7f7fe814f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fe77fe700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fe806e6e0 0x7f7fe80b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 -- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fe806e6e0 msgr2=0x7f7fe80b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fe806e6e0 0x7f7fe80b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 -- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fe806cde0 msgr2=0x7f7fe814f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:44.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fe806cde0 0x7f7fe814f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 -- 172.21.15.104:0/2528765854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd80094a0 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7feca7c700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000cc10 tx=0x7f7fe000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe000abb0 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe000ad10 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe80b0930 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0012660 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.974+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe80b0ef0 con 0x7f7fe815c580 2023-06-10T18:51:44.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.975+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe81e1c90 con 0x7f7fe815c580 2023-06-10T18:51:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.975+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7fe00127c0 con 0x7f7fe815c580 2023-06-10T18:51:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.976+0000 7f7fe4ff9700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 0x7f7fd406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.976+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7fe0089a10 con 0x7f7fe815c580 2023-06-10T18:51:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.976+0000 7f7fe7fff700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 0x7f7fd406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:44.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.976+0000 7f7fe7fff700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 0x7f7fd406d190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc007fd0 tx=0x7f7fdc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:44.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:44.980+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7fe0054d60 con 0x7f7fe815c580 2023-06-10T18:51:45.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.166+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fe8074050 con 0x7f7fe815c580 2023-06-10T18:51:45.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.167+0000 7f7fe4ff9700 1 -- 172.21.15.104:0/2528765854 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fe0016020 con 0x7f7fe815c580 2023-06-10T18:51:45.173 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:51:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 msgr2=0x7f7fd406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 0x7f7fd406d190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc007fd0 tx=0x7f7fdc009040 comp rx=0 tx=0).stop 2023-06-10T18:51:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 msgr2=0x7f7fe80b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000cc10 tx=0x7f7fe000a060 comp rx=0 tx=0).stop 2023-06-10T18:51:45.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 shutdown_connections 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd406ace0 0x7f7fd406d190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fe806cde0 0x7f7fe814f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fe815c580 0x7f7fe80b3530 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 --2- 172.21.15.104:0/2528765854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fe806e6e0 0x7f7fe80b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 >> 172.21.15.104:0/2528765854 conn(0x7f7fe806b8a0 msgr2=0x7f7fe80ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 shutdown_connections 2023-06-10T18:51:45.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.169+0000 7f7fee4df700 1 -- 172.21.15.104:0/2528765854 wait complete. 2023-06-10T18:51:45.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:51:45.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.303+0000 7f2d0e8d1700 1 Processor -- start 2023-06-10T18:51:45.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.303+0000 7f2d0e8d1700 1 -- start start 2023-06-10T18:51:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d0e8d1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d080b1780 0x7f2d0806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d0e8d1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d080ae7e0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d0e8d1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0806df90 con 0x7f2d0806ea70 2023-06-10T18:51:45.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d0e8d1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0806e270 con 0x7f2d080b1780 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d080b1780 0x7f2d0806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d080b1780 0x7f2d0806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60712/0 (socket says 172.21.15.104:60712) 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 -- 172.21.15.104:0/950608027 learned_addr learned my addr 172.21.15.104:0/950608027 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d07fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2d0815c6f0 0x7f2d080b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58538/0 (socket says 172.21.15.104:58538) 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 -- 172.21.15.104:0/950608027 >> v1:172.21.15.114:6789/0 conn(0x7f2d0806ea70 legacy=0x7f2d0806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 -- 172.21.15.104:0/950608027 >> v1:172.21.15.104:6789/0 conn(0x7f2d0815c6f0 legacy=0x7f2d080b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 -- 172.21.15.104:0/950608027 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0806e550 con 0x7f2d080b1780 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d06ffd700 1 -- 172.21.15.104:0/950608027 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159480211 0 0) 0x7f2d080ae7e0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d077fe700 1 --2- 172.21.15.104:0/950608027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d080b1780 0x7f2d0806cde0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80098f0 tx=0x7f2cf8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fc4c97b621ad35 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d06ffd700 1 -- 172.21.15.104:0/950608027 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8004230 con 0x7f2d080b1780 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d06ffd700 1 -- 172.21.15.104:0/950608027 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8004390 con 0x7f2d080b1780 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.304+0000 7f2d06ffd700 1 -- 172.21.15.104:0/950608027 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf80036a0 con 0x7f2d080b1780 2023-06-10T18:51:45.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/950608027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d080b1780 msgr2=0x7f2d0806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/950608027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d080b1780 0x7f2d0806cde0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80098f0 tx=0x7f2cf8009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/950608027 shutdown_connections 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/950608027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d080b1780 0x7f2d0806cde0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/950608027 >> 172.21.15.104:0/950608027 conn(0x7f2d0806b8a0 msgr2=0x7f2d080cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/950608027 shutdown_connections 2023-06-10T18:51:45.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.305+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/950608027 wait complete. 2023-06-10T18:51:45.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.307+0000 7f2d0e8d1700 1 Processor -- start 2023-06-10T18:51:45.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.307+0000 7f2d0e8d1700 1 -- start start 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 0x7f2d081ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d080b1780 0x7f2d080ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080afdc0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080b00a0 con 0x7f2d0806ea70 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080b0380 con 0x7f2d080b1780 2023-06-10T18:51:45.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60720/0 (socket says 172.21.15.104:60720) 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 -- 172.21.15.104:0/3698121344 learned_addr learned my addr 172.21.15.104:0/3698121344 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d07fff700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 0x7f2d081ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d077fe700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d080b1780 0x7f2d080ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 -- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d080b1780 msgr2=0x7f2d080ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d080b1780 0x7f2d080ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 -- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 msgr2=0x7f2d081ee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 0x7f2d081ee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 -- 172.21.15.104:0/3698121344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8009550 con 0x7f2d0815c6f0 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d07fff700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 0x7f2d081ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0ce6e700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000cc10 tx=0x7f2d0000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00004b90 con 0x7f2d0815c6f0 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00004cf0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d080b0660 con 0x7f2d0815c6f0 2023-06-10T18:51:45.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000ab30 con 0x7f2d0815c6f0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.308+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d081ee5e0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.309+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d080b2420 con 0x7f2d0815c6f0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.310+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2d0000ac90 con 0x7f2d0815c6f0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.310+0000 7f2d04ff9700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 0x7f2cf406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.310+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f2d00088be0 con 0x7f2d0815c6f0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.310+0000 7f2d07fff700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 0x7f2cf406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.311+0000 7f2d07fff700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 0x7f2cf406d190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc007bf0 tx=0x7f2cfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.315+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2d080b2420 con 0x7f2d0815c6f0 2023-06-10T18:51:45.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.499+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2d080b2420 con 0x7f2d0815c6f0 2023-06-10T18:51:45.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.499+0000 7f2d04ff9700 1 -- 172.21.15.104:0/3698121344 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2d080b2420 con 0x7f2d0815c6f0 2023-06-10T18:51:45.506 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:51:45.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 msgr2=0x7f2cf406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 0x7f2cf406d190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc007bf0 tx=0x7f2cfc009040 comp rx=0 tx=0).stop 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 msgr2=0x7f2d080cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000cc10 tx=0x7f2d0000cf20 comp rx=0 tx=0).stop 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 shutdown_connections 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf406ace0 0x7f2cf406d190 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d0806ea70 0x7f2d081ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d0815c6f0 0x7f2d080cadb0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 --2- 172.21.15.104:0/3698121344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d080b1780 0x7f2d080ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 >> 172.21.15.104:0/3698121344 conn(0x7f2d0806b8a0 msgr2=0x7f2d080adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.501+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 shutdown_connections 2023-06-10T18:51:45.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.502+0000 7f2d0e8d1700 1 -- 172.21.15.104:0/3698121344 wait complete. 2023-06-10T18:51:45.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:51:45.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.632+0000 7f63a1ec8700 1 Processor -- start 2023-06-10T18:51:45.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.632+0000 7f63a1ec8700 1 -- start start 2023-06-10T18:51:45.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 0x7f639c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 0x7f639c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f639c0ca2f0 0x7f639c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639c0c9bc0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639c0c9e60 con 0x7f639c0b1080 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f63a1ec8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639c0aee70 con 0x7f639c0aeaa0 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f639bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f639c0ca2f0 0x7f639c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f639bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f639c0ca2f0 0x7f639c0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60728/0 (socket says 172.21.15.104:60728) 2023-06-10T18:51:45.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.633+0000 7f639bfff700 1 -- 172.21.15.104:0/2233696805 learned_addr learned my addr 172.21.15.104:0/2233696805 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639affd700 1 --2- 172.21.15.104:0/2233696805 >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 0x7f639c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639b7fe700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 0x7f639c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 -- 172.21.15.104:0/2233696805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 msgr2=0x7f639c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 0x7f639c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 -- 172.21.15.104:0/2233696805 >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 msgr2=0x7f639c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 --2- 172.21.15.104:0/2233696805 >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 0x7f639c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 -- 172.21.15.104:0/2233696805 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c06e540 con 0x7f639c0ca2f0 2023-06-10T18:51:45.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639affd700 1 --2- 172.21.15.104:0/2233696805 >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 0x7f639c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639b7fe700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 0x7f639c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639bfff700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0ca750 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f639c066f40 tx=0x7f639400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3285b290050604bf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639a7fc700 1 -- 172.21.15.104:0/2233696805 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639400e070 con 0x7f639c0ca2f0 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639a7fc700 1 -- 172.21.15.104:0/2233696805 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639400b9c0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f639a7fc700 1 -- 172.21.15.104:0/2233696805 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639400bd70 con 0x7f639c0ca2f0 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f63a1ec8700 1 -- 172.21.15.104:0/2233696805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 msgr2=0x7f639c0ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.634+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0ca750 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f639c066f40 tx=0x7f639400b330 comp rx=0 tx=0).stop 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 -- 172.21.15.104:0/2233696805 shutdown_connections 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0ca750 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/2233696805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0aeaa0 0x7f639c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/2233696805 >> v2:172.21.15.114:3300/0 conn(0x7f639c0b1080 0x7f639c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 -- 172.21.15.104:0/2233696805 >> 172.21.15.104:0/2233696805 conn(0x7f639c06b8a0 msgr2=0x7f639c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 -- 172.21.15.104:0/2233696805 shutdown_connections 2023-06-10T18:51:45.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.635+0000 7f63a1ec8700 1 -- 172.21.15.104:0/2233696805 wait complete. 2023-06-10T18:51:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.636+0000 7f63a1ec8700 1 Processor -- start 2023-06-10T18:51:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.636+0000 7f63a1ec8700 1 -- start start 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f639c0aeaa0 0x7f639c0ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0b1080 0x7f639c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c157220 con 0x7f639c0ca2f0 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c1f4f90 con 0x7f639c0aeaa0 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639c0b2810 con 0x7f639c0b1080 2023-06-10T18:51:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60734/0 (socket says 172.21.15.104:60734) 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 -- 172.21.15.104:0/3685903926 learned_addr learned my addr 172.21.15.104:0/3685903926 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639b7fe700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f639c0aeaa0 0x7f639c0ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639affd700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0b1080 0x7f639c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 -- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0b1080 msgr2=0x7f639c1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0b1080 0x7f639c1558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 -- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f639c0aeaa0 msgr2=0x7f639c0ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f639c0aeaa0 0x7f639c0ac820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 -- 172.21.15.104:0/3685903926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394009b30 con 0x7f639c0ca2f0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f639bfff700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f639c06cc20 tx=0x7f6394004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394004250 con 0x7f639c0ca2f0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63940043d0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394004780 con 0x7f639c0ca2f0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639c0b2af0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.637+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639c0b30b0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.638+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639c0b3270 con 0x7f639c0ca2f0 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.639+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f639401b430 con 0x7f639c0ca2f0 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.639+0000 7f6393fff700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.639+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f639408a3e0 con 0x7f639c0ca2f0 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.639+0000 7f639b7fe700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.640+0000 7f639b7fe700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f639c0c6060 tx=0x7f6384008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.644+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6394055730 con 0x7f639c0ca2f0 2023-06-10T18:51:45.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.832+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f639c1f5100 con 0x7f639c0ca2f0 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.832+0000 7f6393fff700 1 -- 172.21.15.104:0/3685903926 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+126 (secure 0 0 0) 0x7f6394058d50 con 0x7f639c0ca2f0 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:51:45.838 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:45.839 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:51:45.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 msgr2=0x7f637c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f639c0c6060 tx=0x7f6384008040 comp rx=0 tx=0).stop 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 msgr2=0x7f639c0b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f639c06cc20 tx=0x7f6394004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 shutdown_connections 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f639c0aeaa0 0x7f639c0ac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f639c0ca2f0 0x7f639c0b4bd0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 --2- 172.21.15.104:0/3685903926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639c0b1080 0x7f639c1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.834+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 >> 172.21.15.104:0/3685903926 conn(0x7f639c06b8a0 msgr2=0x7f639c0af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.835+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 shutdown_connections 2023-06-10T18:51:45.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.835+0000 7f63a1ec8700 1 -- 172.21.15.104:0/3685903926 wait complete. 2023-06-10T18:51:45.852 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:51:45.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:51:45.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.971+0000 7feb0631c700 1 Processor -- start 2023-06-10T18:51:45.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.971+0000 7feb0631c700 1 -- start start 2023-06-10T18:51:45.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feb0631c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb0015c500 0x7feb000cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feb0631c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feb000cf200 0x7feb00069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feb0631c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0015c8d0 con 0x7feb0006a3b0 2023-06-10T18:51:45.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feb0631c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb000aef20 con 0x7feb000cf200 2023-06-10T18:51:45.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feb0631c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0006abd0 con 0x7feb0015c500 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feb000cf200 0x7feb00069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feb000cf200 0x7feb00069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60738/0 (socket says 172.21.15.104:60738) 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 -- 172.21.15.104:0/351118507 learned_addr learned my addr 172.21.15.104:0/351118507 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feafffff700 1 --2- 172.21.15.104:0/351118507 >> v2:172.21.15.114:3300/0 conn(0x7feb0015c500 0x7feb000cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 -- 172.21.15.104:0/351118507 >> v2:172.21.15.114:3300/0 conn(0x7feb0015c500 msgr2=0x7feb000cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 --2- 172.21.15.104:0/351118507 >> v2:172.21.15.114:3300/0 conn(0x7feb0015c500 0x7feb000cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 -- 172.21.15.104:0/351118507 >> v1:172.21.15.114:6789/0 conn(0x7feb0006a3b0 legacy=0x7feb0006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 -- 172.21.15.104:0/351118507 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0006a8f0 con 0x7feb000cf200 2023-06-10T18:51:45.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feaff7fe700 1 --2- 172.21.15.104:0/351118507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb000cf200 0x7feb00069e70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7feb000b2170 tx=0x7feaf4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e5a6f2a4d2ac6f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.972+0000 7feafeffd700 1 -- 172.21.15.104:0/351118507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4011830 con 0x7feb000cf200 2023-06-10T18:51:45.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feafeffd700 1 -- 172.21.15.104:0/351118507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf4011990 con 0x7feb000cf200 2023-06-10T18:51:45.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feafeffd700 1 -- 172.21.15.104:0/351118507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf4011d20 con 0x7feb000cf200 2023-06-10T18:51:45.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 -- 172.21.15.104:0/351118507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb000cf200 msgr2=0x7feb00069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 --2- 172.21.15.104:0/351118507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb000cf200 0x7feb00069e70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7feb000b2170 tx=0x7feaf4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 -- 172.21.15.104:0/351118507 shutdown_connections 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 --2- 172.21.15.104:0/351118507 >> v2:172.21.15.114:3300/0 conn(0x7feb0015c500 0x7feb000cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 --2- 172.21.15.104:0/351118507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb000cf200 0x7feb00069e70 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 -- 172.21.15.104:0/351118507 >> 172.21.15.104:0/351118507 conn(0x7feb0006b8a0 msgr2=0x7feb000cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 -- 172.21.15.104:0/351118507 shutdown_connections 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.973+0000 7feb0631c700 1 -- 172.21.15.104:0/351118507 wait complete. 2023-06-10T18:51:45.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.974+0000 7feb0631c700 1 Processor -- start 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.974+0000 7feb0631c700 1 -- start start 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 0x7feb001f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 0x7feb001f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb000b3ec0 con 0x7feb0015c500 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb000c7990 con 0x7feb000cf200 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb0631c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb000c7c70 con 0x7feb0006a3b0 2023-06-10T18:51:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb048b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb048b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60748/0 (socket says 172.21.15.104:60748) 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.975+0000 7feb048b9700 1 -- 172.21.15.104:0/1835601824 learned_addr learned my addr 172.21.15.104:0/1835601824 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feaff7fe700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 0x7feb001f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feafffff700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 0x7feb001f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 -- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 msgr2=0x7feb001f48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 0x7feb001f48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 -- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 msgr2=0x7feb001f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 0x7feb001f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 -- 172.21.15.104:0/1835601824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4009550 con 0x7feb0015c500 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feaff7fe700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 0x7feb001f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feafffff700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 0x7feb001f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb048b9700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7feaf800cb50 tx=0x7feaf800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8003c00 con 0x7feb0015c500 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf8003d60 con 0x7feb0015c500 2023-06-10T18:51:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb000b3b50 con 0x7feb0015c500 2023-06-10T18:51:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf800ab70 con 0x7feb0015c500 2023-06-10T18:51:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.976+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb000ada20 con 0x7feb0015c500 2023-06-10T18:51:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.977+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb000c9890 con 0x7feb0015c500 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.977+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feaf800acd0 con 0x7feb0015c500 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.978+0000 7feafcff9700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 0x7feaec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.978+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5530+0+0 (secure 0 0 0) 0x7feaf8089280 con 0x7feb0015c500 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.978+0000 7feafffff700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 0x7feaec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.978+0000 7feafffff700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 0x7feaec06d190 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009b40 tx=0x7feaf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:45.982+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feaf80545d0 con 0x7feb0015c500 2023-06-10T18:51:46.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.166+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feb000ce1b0 con 0x7feb0015c500 2023-06-10T18:51:46.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.965+0000 7feafcff9700 1 -- 172.21.15.104:0/1835601824 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7feaf8057bf0 con 0x7feb0015c500 2023-06-10T18:51:46.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 msgr2=0x7feaec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 0x7feaec06d190 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009b40 tx=0x7feaf0008040 comp rx=0 tx=0).stop 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 msgr2=0x7feb000ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7feaf800cb50 tx=0x7feaf800cf10 comp rx=0 tx=0).stop 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 shutdown_connections 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feb0015c500 0x7feb000ad4e0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0006a3b0 0x7feb001f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:46.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feaec06ace0 0x7feaec06d190 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:46.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 --2- 172.21.15.104:0/1835601824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb000cf200 0x7feb001f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:46.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 >> 172.21.15.104:0/1835601824 conn(0x7feb0006b8a0 msgr2=0x7feb000c9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:46.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 shutdown_connections 2023-06-10T18:51:46.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:46.969+0000 7feb0631c700 1 -- 172.21.15.104:0/1835601824 wait complete. 2023-06-10T18:51:46.978 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:51:46.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:51:47.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.120+0000 7faa75269700 1 Processor -- start 2023-06-10T18:51:47.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 -- start start 2023-06-10T18:51:47.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa701545a0 0x7faa7015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7015aa00 con 0x7faa700af940 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7015dab0 con 0x7faa701545a0 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.121+0000 7faa75269700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa700cda90 con 0x7faa700cd150 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa701545a0 0x7faa7015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa701545a0 0x7faa7015a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60752/0 (socket says 172.21.15.104:60752) 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 -- 172.21.15.104:0/415579347 learned_addr learned my addr 172.21.15.104:0/415579347 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6f7fe700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 -- 172.21.15.104:0/415579347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 msgr2=0x7faa700cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 -- 172.21.15.104:0/415579347 >> v1:172.21.15.114:6789/0 conn(0x7faa700af940 legacy=0x7faa7015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 -- 172.21.15.104:0/415579347 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa70154970 con 0x7faa701545a0 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6effd700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa701545a0 0x7faa7015a4c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7faa640098f0 tx=0x7faa64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c7c3cd06f2d1f3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6dffb700 1 -- 172.21.15.104:0/415579347 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64011830 con 0x7faa701545a0 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6dffb700 1 -- 172.21.15.104:0/415579347 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64011990 con 0x7faa701545a0 2023-06-10T18:51:47.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.122+0000 7faa6dffb700 1 -- 172.21.15.104:0/415579347 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64011d20 con 0x7faa701545a0 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 -- 172.21.15.104:0/415579347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa701545a0 msgr2=0x7faa7015a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa701545a0 0x7faa7015a4c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7faa640098f0 tx=0x7faa64009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 -- 172.21.15.104:0/415579347 shutdown_connections 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa701545a0 0x7faa7015a4c0 secure :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7faa640098f0 tx=0x7faa64009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 --2- 172.21.15.104:0/415579347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 -- 172.21.15.104:0/415579347 >> 172.21.15.104:0/415579347 conn(0x7faa7006e370 msgr2=0x7faa7006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 -- 172.21.15.104:0/415579347 shutdown_connections 2023-06-10T18:51:47.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.123+0000 7faa75269700 1 -- 172.21.15.104:0/415579347 wait complete. 2023-06-10T18:51:47.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.124+0000 7faa75269700 1 Processor -- start 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.124+0000 7faa75269700 1 -- start start 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa700b2390 0x7faa700b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa70146ff0 con 0x7faa700af940 2023-06-10T18:51:47.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa70154b80 con 0x7faa700b2390 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa75269700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa70154e60 con 0x7faa700cd150 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa6effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.125+0000 7faa6effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60758/0 (socket says 172.21.15.104:60758) 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 -- 172.21.15.104:0/3652373980 learned_addr learned my addr 172.21.15.104:0/3652373980 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6e7fc700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6f7fe700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa700b2390 0x7faa700b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 -- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 msgr2=0x7faa700b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 -- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa700b2390 msgr2=0x7faa700b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa700b2390 0x7faa700b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 -- 172.21.15.104:0/3652373980 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa64009550 con 0x7faa700af940 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa6effd700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7faa64013000 tx=0x7faa64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64004550 con 0x7faa700af940 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6401f070 con 0x7faa700af940 2023-06-10T18:51:47.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa700b2900 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa640038e0 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.126+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa700b2ec0 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.127+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa70074050 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.128+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faa64003a40 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.128+0000 7faa577fe700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 0x7faa5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.128+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5900+0+0 (secure 0 0 0) 0x7faa64021070 con 0x7faa700af940 2023-06-10T18:51:47.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.128+0000 7faa6e7fc700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 0x7faa5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:47.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.129+0000 7faa6e7fc700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 0x7faa5c06d190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa70154300 tx=0x7faa60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:47.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.132+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faa64055c80 con 0x7faa700af940 2023-06-10T18:51:47.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.318+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faa70157de0 con 0x7faa700af940 2023-06-10T18:51:47.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.972+0000 7faa577fe700 1 -- 172.21.15.104:0/3652373980 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7faa6402b0d0 con 0x7faa700af940 2023-06-10T18:51:47.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.977+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 msgr2=0x7faa5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.977+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 0x7faa5c06d190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa70154300 tx=0x7faa60009040 comp rx=0 tx=0).stop 2023-06-10T18:51:47.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.977+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 msgr2=0x7faa70146ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:47.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.977+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7faa64013000 tx=0x7faa64004750 comp rx=0 tx=0).stop 2023-06-10T18:51:47.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 shutdown_connections 2023-06-10T18:51:47.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa5c06ace0 0x7faa5c06d190 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa700b2390 0x7faa700b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa700af940 0x7faa70146ab0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 --2- 172.21.15.104:0/3652373980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa700cd150 0x7faa700b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 >> 172.21.15.104:0/3652373980 conn(0x7faa7006e370 msgr2=0x7faa70158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.983+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 shutdown_connections 2023-06-10T18:51:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:47.984+0000 7faa75269700 1 -- 172.21.15.104:0/3652373980 wait complete. 2023-06-10T18:51:47.992 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:51:48.007 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:51:48.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.125+0000 7f0550d67700 1 Processor -- start 2023-06-10T18:51:48.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.125+0000 7f0550d67700 1 -- start start 2023-06-10T18:51:48.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0550d67700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f054c15a330 0x7f054c0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0550d67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f054c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0550d67700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c15a700 con 0x7f054c0cb890 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0550d67700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c06d7b0 con 0x7f054c15a330 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0550d67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0cbe00 con 0x7f054c0cb420 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f054ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f054c0cb890 0x7f054c06dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58594/0 (socket says 172.21.15.104:58594) 2023-06-10T18:51:48.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f054ad9d700 1 -- 172.21.15.104:0/3756622208 learned_addr learned my addr 172.21.15.104:0/3756622208 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f0549d9b700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f054c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f054a59c700 1 --2- 172.21.15.104:0/3756622208 >> v2:172.21.15.114:3300/0 conn(0x7f054c15a330 0x7f054c0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170610149 0 0) 0x7f054c15a700 con 0x7f054c0cb890 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.126+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05340035b0 con 0x7f054c0cb890 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216625970 0 0) 0x7f05340035b0 con 0x7f054c0cb890 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 msgr2=0x7f054c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f054c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 >> v2:172.21.15.114:3300/0 conn(0x7f054c15a330 msgr2=0x7f054c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 --2- 172.21.15.104:0/3756622208 >> v2:172.21.15.114:3300/0 conn(0x7f054c15a330 0x7f054c0ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c06ce80 con 0x7f054c0cb890 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f05440019f0 con 0x7f054c0cb890 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 >> v1:172.21.15.104:6789/0 conn(0x7f054c0cb890 legacy=0x7f054c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f0534003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534003f80 0x7f0534004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0534004d90 0x7f0534005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c06d7b0 con 0x7f0534004d90 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0cbe00 con 0x7f0534003f80 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053400ba10 con 0x7f054c0cb420 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054a59c700 1 --2- 172.21.15.104:0/3756622208 >> v2:172.21.15.114:3300/0 conn(0x7f054c15a330 0x7f054c0ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0534004d90 0x7f0534005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f0549d9b700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534003f80 0x7f0534004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054a59c700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f0534003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 -- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 msgr2=0x7f0534003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c0cb420 0x7f0534003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 -- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534003f80 msgr2=0x7f0534004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534003f80 0x7f0534004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 -- 172.21.15.104:0/3756622208 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340035b0 con 0x7f0534004d90 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054ad9d700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0534004d90 0x7f0534005200 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f054400a860 tx=0x7f0534005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fd5ca0cba382c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544012600 con 0x7f0534004d90 2023-06-10T18:51:48.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0544012780 con 0x7f0534004d90 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.127+0000 7f054959a700 1 -- 172.21.15.104:0/3756622208 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544012b30 con 0x7f0534004d90 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 -- 172.21.15.104:0/3756622208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0534004d90 msgr2=0x7f0534005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 --2- 172.21.15.104:0/3756622208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0534004d90 0x7f0534005200 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f054400a860 tx=0x7f0534005880 comp rx=0 tx=0).stop 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f054ad9d700 1 -- 172.21.15.104:0/3756622208 reap_dead start 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 -- 172.21.15.104:0/3756622208 shutdown_connections 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 -- 172.21.15.104:0/3756622208 >> 172.21.15.104:0/3756622208 conn(0x7f054c06b8a0 msgr2=0x7f054c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:48.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 -- 172.21.15.104:0/3756622208 shutdown_connections 2023-06-10T18:51:48.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.128+0000 7f0550d67700 1 -- 172.21.15.104:0/3756622208 wait complete. 2023-06-10T18:51:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.129+0000 7f0550d67700 1 Processor -- start 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.129+0000 7f0550d67700 1 -- start start 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534004d90 0x7f054c1f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c154860 0x7f054c0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c153fd0 con 0x7f054c1543f0 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c153950 con 0x7f0534004d90 2023-06-10T18:51:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0550d67700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0cdf10 con 0x7f054c154860 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0549d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0549d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60776/0 (socket says 172.21.15.104:60776) 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f0549d9b700 1 -- 172.21.15.104:0/3760613187 learned_addr learned my addr 172.21.15.104:0/3760613187 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f054a59c700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534004d90 0x7f054c1f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.130+0000 7f054ad9d700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c154860 0x7f054c0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 -- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c154860 msgr2=0x7f054c0b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c154860 0x7f054c0b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 -- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534004d90 msgr2=0x7f054c1f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534004d90 0x7f054c1f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 -- 172.21.15.104:0/3760613187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c153bf0 con 0x7f054c1543f0 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0549d9b700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f053c0119a0 tx=0x7f053c011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c00f780 con 0x7f054c1543f0 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c00f8e0 con 0x7f054c1543f0 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0ce1f0 con 0x7f054c1543f0 2023-06-10T18:51:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c00fc70 con 0x7f054c1543f0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.131+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c15a700 con 0x7f054c1543f0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.132+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c154240 con 0x7f054c1543f0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.132+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f053c014440 con 0x7f054c1543f0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.133+0000 7f053affd700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.133+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f053c089a70 con 0x7f054c1543f0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.133+0000 7f054a59c700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.134+0000 7f054a59c700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 0x7f052c06d190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f054c1f52d0 tx=0x7f05400091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.137+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f053c058560 con 0x7f054c1543f0 2023-06-10T18:51:48.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.368+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f054c074050 con 0x7f054c1543f0 2023-06-10T18:51:48.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.375+0000 7f053affd700 1 -- 172.21.15.104:0/3760613187 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f053c0580f0 con 0x7f054c1543f0 2023-06-10T18:51:48.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 msgr2=0x7f052c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 0x7f052c06d190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f054c1f52d0 tx=0x7f05400091c0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 msgr2=0x7f054c0b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f053c0119a0 tx=0x7f053c011cb0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 shutdown_connections 2023-06-10T18:51:48.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0534004d90 0x7f054c1f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f054c154860 0x7f054c0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 --2- 172.21.15.104:0/3760613187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f054c1543f0 0x7f054c0b0e40 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.382+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 >> 172.21.15.104:0/3760613187 conn(0x7f054c06b8a0 msgr2=0x7f054c1e2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:48.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.383+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 shutdown_connections 2023-06-10T18:51:48.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.383+0000 7f0550d67700 1 -- 172.21.15.104:0/3760613187 wait complete. 2023-06-10T18:51:48.393 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 12 and data pool 13 2023-06-10T18:51:48.411 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:48.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.525+0000 7f5b98317700 1 Processor -- start 2023-06-10T18:51:48.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.525+0000 7f5b98317700 1 -- start start 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b98317700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b900cad90 0x7f5b900701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b98317700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900ccca0 con 0x7f5b9015a250 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b98317700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900cee50 con 0x7f5b900cfa00 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b98317700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b900cf130 con 0x7f5b900cad90 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b958b2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5b9015a250 0x7f5b900b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58610/0 (socket says 172.21.15.104:58610) 2023-06-10T18:51:48.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b958b2700 1 -- 172.21.15.104:0/2849169770 learned_addr learned my addr 172.21.15.104:0/2849169770 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784298396 0 0) 0x7f5b900ccca0 con 0x7f5b9015a250 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c0035b0 con 0x7f5b9015a250 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.526+0000 7f5b968b4700 1 --2- 172.21.15.104:0/2849169770 >> v2:172.21.15.114:3300/0 conn(0x7f5b900cad90 0x7f5b900701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404168546 0 0) 0x7f5b7c0035b0 con 0x7f5b9015a250 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 >> v2:172.21.15.114:3300/0 conn(0x7f5b900cad90 msgr2=0x7f5b900701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 --2- 172.21.15.104:0/2849169770 >> v2:172.21.15.114:3300/0 conn(0x7f5b900cad90 0x7f5b900701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 >> v1:172.21.15.114:6789/0 conn(0x7f5b900cfa00 legacy=0x7f5b900cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900cc930 con 0x7f5b9015a250 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5b8c001670 con 0x7f5b9015a250 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 >> v1:172.21.15.104:6789/0 conn(0x7f5b9015a250 legacy=0x7f5b900b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c004850 0x7f5b7c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c005200 0x7f5b7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900cf130 con 0x7f5b7c003a00 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c00c110 con 0x7f5b7c004850 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c00c2d0 con 0x7f5b7c005200 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b958b2700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c004850 0x7f5b7c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b968b4700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c005200 0x7f5b7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 -- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c005200 msgr2=0x7f5b7c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c005200 0x7f5b7c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 -- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c004850 msgr2=0x7f5b7c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c004850 0x7f5b7c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 -- 172.21.15.104:0/2849169770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0035b0 con 0x7f5b7c003a00 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b968b4700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c005200 0x7f5b7c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b968b4700 1 -- 172.21.15.104:0/2849169770 reap_dead start 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b960b3700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b7c003e10 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5b800017c0 tx=0x7f5b7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2e3dfc6f16d11b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b800052d0 con 0x7f5b7c003a00 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b80005430 con 0x7f5b7c003a00 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.527+0000 7f5b950b1700 1 -- 172.21.15.104:0/2849169770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b800038c0 con 0x7f5b7c003a00 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 -- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 msgr2=0x7f5b7c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b7c003e10 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5b800017c0 tx=0x7f5b7c005880 comp rx=0 tx=0).stop 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 -- 172.21.15.104:0/2849169770 shutdown_connections 2023-06-10T18:51:48.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 --2- 172.21.15.104:0/2849169770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b7c003e10 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 -- 172.21.15.104:0/2849169770 >> 172.21.15.104:0/2849169770 conn(0x7f5b9006e370 msgr2=0x7f5b9006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 -- 172.21.15.104:0/2849169770 shutdown_connections 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.528+0000 7f5b98317700 1 -- 172.21.15.104:0/2849169770 wait complete. 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.529+0000 7f5b98317700 1 Processor -- start 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.529+0000 7f5b98317700 1 -- start start 2023-06-10T18:51:48.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 0x7f5b9015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 0x7f5b901e4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900b2620 con 0x7f5b7c003a00 2023-06-10T18:51:48.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b901e4790 con 0x7f5b9014e730 2023-06-10T18:51:48.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b98317700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900b19a0 con 0x7f5b901e3de0 2023-06-10T18:51:48.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60796/0 (socket says 172.21.15.104:60796) 2023-06-10T18:51:48.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 -- 172.21.15.104:0/1680154167 learned_addr learned my addr 172.21.15.104:0/1680154167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:48.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b968b4700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 0x7f5b901e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 -- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 msgr2=0x7f5b901e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 0x7f5b901e4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b958b2700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 0x7f5b9015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 -- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 msgr2=0x7f5b9015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 0x7f5b9015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 -- 172.21.15.104:0/1680154167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900b1c80 con 0x7f5b7c003a00 2023-06-10T18:51:48.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b968b4700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 0x7f5b901e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:48.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b958b2700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 0x7f5b9015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:48.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.530+0000 7f5b960b3700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5b80015040 tx=0x7f5b80001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b80001290 con 0x7f5b7c003a00 2023-06-10T18:51:48.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8001d480 con 0x7f5b7c003a00 2023-06-10T18:51:48.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b900b1f60 con 0x7f5b7c003a00 2023-06-10T18:51:48.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8001d810 con 0x7f5b7c003a00 2023-06-10T18:51:48.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9015d500 con 0x7f5b7c003a00 2023-06-10T18:51:48.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.531+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b90074050 con 0x7f5b7c003a00 2023-06-10T18:51:48.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.532+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5b80019040 con 0x7f5b7c003a00 2023-06-10T18:51:48.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.532+0000 7f5b86ffd700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 0x7f5b7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.532+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5b8008a160 con 0x7f5b7c003a00 2023-06-10T18:51:48.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.533+0000 7f5b958b2700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 0x7f5b7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.533+0000 7f5b958b2700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 0x7f5b7406d190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5b900b12a0 tx=0x7f5b8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.537+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5b800550c0 con 0x7f5b7c003a00 2023-06-10T18:51:48.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.728+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b90154d50 con 0x7f5b7c003a00 2023-06-10T18:51:48.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.729+0000 7f5b86ffd700 1 -- 172.21.15.104:0/1680154167 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18608 (secure 0 0 0) 0x7f5b800586e0 con 0x7f5b7c003a00 2023-06-10T18:51:48.735 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:48.735 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":82,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:48.369869+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:48.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 msgr2=0x7f5b7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 0x7f5b7406d190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5b900b12a0 tx=0x7f5b8c009040 comp rx=0 tx=0).stop 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 msgr2=0x7f5b900b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5b80015040 tx=0x7f5b80001120 comp rx=0 tx=0).stop 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 shutdown_connections 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b901e3de0 0x7f5b901e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c003a00 0x7f5b900b1380 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b7406ace0 0x7f5b7406d190 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 --2- 172.21.15.104:0/1680154167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b9014e730 0x7f5b9015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 >> 172.21.15.104:0/1680154167 conn(0x7f5b9006e370 msgr2=0x7f5b900b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:48.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 shutdown_connections 2023-06-10T18:51:48.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.732+0000 7f5b98317700 1 -- 172.21.15.104:0/1680154167 wait complete. 2023-06-10T18:51:48.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.863+0000 7f0d0f2c1700 1 Processor -- start 2023-06-10T18:51:48.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.863+0000 7f0d0f2c1700 1 -- start start 2023-06-10T18:51:48.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0f2c1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0f2c1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0d080ae280 0x7f0d080cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0f2c1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080ccea0 con 0x7f0d08070a20 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0f2c1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080ae6f0 con 0x7f0d080ae280 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0f2c1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080ce2c0 con 0x7f0d0815c040 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0d080ae280 0x7f0d080cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0d080ae280 0x7f0d080cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60802/0 (socket says 172.21.15.104:60802) 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0d85e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0d08070a20 0x7f0d080ca3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58634/0 (socket says 172.21.15.104:58634) 2023-06-10T18:51:48.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 -- 172.21.15.104:0/3860467463 learned_addr learned my addr 172.21.15.104:0/3860467463 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0d05d700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 -- 172.21.15.104:0/3860467463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 msgr2=0x7f0d080ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 -- 172.21.15.104:0/3860467463 >> v1:172.21.15.104:6789/0 conn(0x7f0d08070a20 legacy=0x7f0d080ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0d0c85c700 1 -- 172.21.15.104:0/3860467463 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080cf750 con 0x7f0d080ae280 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.864+0000 7f0cfffff700 1 -- 172.21.15.104:0/3860467463 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033502975 0 0) 0x7f0d00001490 con 0x7f0d08070a20 2023-06-10T18:51:48.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0c85c700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d080cc960 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0d040098f0 tx=0x7f0d04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51eff1b3780c1308 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0cfffff700 1 -- 172.21.15.104:0/3860467463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0400b8a0 con 0x7f0d080ae280 2023-06-10T18:51:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0cfffff700 1 -- 172.21.15.104:0/3860467463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0400ba00 con 0x7f0d080ae280 2023-06-10T18:51:48.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0cfffff700 1 -- 172.21.15.104:0/3860467463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0400bd90 con 0x7f0d080ae280 2023-06-10T18:51:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/3860467463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 msgr2=0x7f0d080cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d080cc960 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0d040098f0 tx=0x7f0d04009c00 comp rx=0 tx=0).stop 2023-06-10T18:51:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/3860467463 shutdown_connections 2023-06-10T18:51:48.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/3860467463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d080cc960 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/3860467463 >> 172.21.15.104:0/3860467463 conn(0x7f0d080b1610 msgr2=0x7f0d080ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/3860467463 shutdown_connections 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.865+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/3860467463 wait complete. 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 Processor -- start 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 -- start start 2023-06-10T18:51:48.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 0x7f0d080b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d081507f0 con 0x7f0d080ae280 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080c9e40 con 0x7f0d08070a20 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0f2c1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080ca120 con 0x7f0d0815c040 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0c85c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0c85c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60810/0 (socket says 172.21.15.104:60810) 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.867+0000 7f0d0c85c700 1 -- 172.21.15.104:0/1963487818 learned_addr learned my addr 172.21.15.104:0/1963487818 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 -- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 msgr2=0x7f0d080c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0d85e700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0d05d700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 0x7f0d080b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 -- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 msgr2=0x7f0d080b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 0x7f0d080b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:48.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 -- 172.21.15.104:0/1963487818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04009550 con 0x7f0d080ae280 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0d05d700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 0x7f0d080b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0c85c700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0d0806c890 tx=0x7f0d04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04004400 con 0x7f0d080ae280 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0401f070 con 0x7f0d080ae280 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d080ca400 con 0x7f0d080ae280 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d040038a0 con 0x7f0d080ae280 2023-06-10T18:51:48.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.868+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080cb260 con 0x7f0d080ae280 2023-06-10T18:51:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.869+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d080c77a0 con 0x7f0d080ae280 2023-06-10T18:51:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.870+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0d04021030 con 0x7f0d080ae280 2023-06-10T18:51:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.870+0000 7f0cfdffb700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 0x7f0cec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:48.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.870+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0d0408aa30 con 0x7f0d080ae280 2023-06-10T18:51:48.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.870+0000 7f0d0d05d700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 0x7f0cec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:48.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.871+0000 7f0d0d05d700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 0x7f0cec06d190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0d081603c0 tx=0x7f0cf8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:48.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:48.874+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0d080c77a0 con 0x7f0d080ae280 2023-06-10T18:51:48.961 INFO:tasks.ceph.mds.d.smithi114.stderr:2023-06-10T18:51:48.956+0000 7f49a9309700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:51:49.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.061+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d080c77a0 con 0x7f0d080ae280 2023-06-10T18:51:49.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.062+0000 7f0cfdffb700 1 -- 172.21.15.104:0/1963487818 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18608 (secure 0 0 0) 0x7f0d080c77a0 con 0x7f0d080ae280 2023-06-10T18:51:49.068 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:49.069 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":82,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:48.369869+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:49.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 msgr2=0x7f0cec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 0x7f0cec06d190 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0d081603c0 tx=0x7f0cf8008040 comp rx=0 tx=0).stop 2023-06-10T18:51:49.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 msgr2=0x7f0d081502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0d0806c890 tx=0x7f0d04004750 comp rx=0 tx=0).stop 2023-06-10T18:51:49.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 shutdown_connections 2023-06-10T18:51:49.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cec06ace0 0x7f0cec06d190 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d08070a20 0x7f0d080b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0815c040 0x7f0d080c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 --2- 172.21.15.104:0/1963487818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d080ae280 0x7f0d081502b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 >> 172.21.15.104:0/1963487818 conn(0x7f0d080b1610 msgr2=0x7f0d0806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:49.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 shutdown_connections 2023-06-10T18:51:49.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.064+0000 7f0d0f2c1700 1 -- 172.21.15.104:0/1963487818 wait complete. 2023-06-10T18:51:49.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:51:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.199+0000 7ff25e59c700 1 Processor -- start 2023-06-10T18:51:49.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.199+0000 7ff25e59c700 1 -- start start 2023-06-10T18:51:49.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25e59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff2580d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25e59c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580ad3c0 con 0x7ff2580d0830 2023-06-10T18:51:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25e59c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff258157f00 con 0x7ff258157b30 2023-06-10T18:51:49.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25e59c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b0170 con 0x7ff2580cea50 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25cb39700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff2580d0830 0x7ff2580b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58648/0 (socket says 172.21.15.104:58648) 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff25cb39700 1 -- 172.21.15.104:0/2220457603 learned_addr learned my addr 172.21.15.104:0/2220457603 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff2577fe700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff2580d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649830153 0 0) 0x7ff2580ad3c0 con 0x7ff2580d0830 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2440035b0 con 0x7ff2580d0830 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649627645 0 0) 0x7ff258157f00 con 0x7ff258157b30 2023-06-10T18:51:49.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2580ad3c0 con 0x7ff258157b30 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 568675246 0 0) 0x7ff2440035b0 con 0x7ff2580d0830 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 msgr2=0x7ff2580d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff2580d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.200+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 >> v1:172.21.15.114:6789/0 conn(0x7ff258157b30 legacy=0x7ff2580ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580b0410 con 0x7ff2580d0830 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff2500019f0 con 0x7ff2580d0830 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 >> v1:172.21.15.104:6789/0 conn(0x7ff2580d0830 legacy=0x7ff2580b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff244003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff244004170 0x7ff244004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff244004d90 0x7ff244005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b0170 con 0x7ff244004d90 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24400bb20 con 0x7ff244004170 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24400bce0 con 0x7ff2580cea50 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff244004d90 0x7ff244005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff2577fe700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff244004170 0x7ff244004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff257fff700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff244003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 -- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 msgr2=0x7ff244003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2580cea50 0x7ff244003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 -- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff244004170 msgr2=0x7ff244004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff244004170 0x7ff244004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 -- 172.21.15.104:0/2220457603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2440035b0 con 0x7ff244004d90 2023-06-10T18:51:49.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff25cb39700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff244004d90 0x7ff244005200 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ff25000a6f0 tx=0x7ff244005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02f39785812ac5e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500072d0 con 0x7ff244004d90 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250007470 con 0x7ff244004d90 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.201+0000 7ff256ffd700 1 -- 172.21.15.104:0/2220457603 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250007820 con 0x7ff244004d90 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 -- 172.21.15.104:0/2220457603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff244004d90 msgr2=0x7ff244005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2220457603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff244004d90 0x7ff244005200 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ff25000a6f0 tx=0x7ff244005880 comp rx=0 tx=0).stop 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25cb39700 1 -- 172.21.15.104:0/2220457603 reap_dead start 2023-06-10T18:51:49.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 -- 172.21.15.104:0/2220457603 shutdown_connections 2023-06-10T18:51:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 -- 172.21.15.104:0/2220457603 >> 172.21.15.104:0/2220457603 conn(0x7ff258089fd0 msgr2=0x7ff25808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 -- 172.21.15.104:0/2220457603 shutdown_connections 2023-06-10T18:51:49.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.202+0000 7ff25e59c700 1 -- 172.21.15.104:0/2220457603 wait complete. 2023-06-10T18:51:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.203+0000 7ff25e59c700 1 Processor -- start 2023-06-10T18:51:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.203+0000 7ff25e59c700 1 -- start start 2023-06-10T18:51:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2581567b0 0x7ff2580cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2580cc3d0 0x7ff25806af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b1b60 con 0x7ff2580b5e60 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b35a0 con 0x7ff2580cc3d0 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25e59c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff258156c20 con 0x7ff2581567b0 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60828/0 (socket says 172.21.15.104:60828) 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 -- 172.21.15.104:0/2054608571 learned_addr learned my addr 172.21.15.104:0/2054608571 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff2577fe700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2581567b0 0x7ff2580cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff25cb39700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2580cc3d0 0x7ff25806af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 -- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2581567b0 msgr2=0x7ff2580cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2581567b0 0x7ff2580cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 -- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2580cc3d0 msgr2=0x7ff25806af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2580cc3d0 0x7ff25806af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 -- 172.21.15.104:0/2054608571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580b0ee0 con 0x7ff2580b5e60 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.204+0000 7ff257fff700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff24000f870 tx=0x7ff24000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:49.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.205+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24000fe70 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.205+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2400052e0 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.205+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2580b11c0 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.205+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240003680 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.205+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2580b1780 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.206+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2580b1ca0 con 0x7ff2580b5e60 2023-06-10T18:51:49.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.206+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff2400037e0 con 0x7ff2580b5e60 2023-06-10T18:51:49.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.206+0000 7ff254ff9700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 0x7ff23806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:49.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.206+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff240089630 con 0x7ff2580b5e60 2023-06-10T18:51:49.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.207+0000 7ff2577fe700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 0x7ff23806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:49.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.207+0000 7ff2577fe700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 0x7ff23806d190 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff2580cee40 tx=0x7ff248008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:49.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.211+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff2580b1ca0 con 0x7ff2580b5e60 2023-06-10T18:51:49.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:49.443+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff2580b1ca0 con 0x7ff2580b5e60 2023-06-10T18:51:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.378+0000 7ff254ff9700 1 -- 172.21.15.104:0/2054608571 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff2580b1ca0 con 0x7ff2580b5e60 2023-06-10T18:51:50.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 msgr2=0x7ff23806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 0x7ff23806d190 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff2580cee40 tx=0x7ff248008040 comp rx=0 tx=0).stop 2023-06-10T18:51:50.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 msgr2=0x7ff258156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff24000f870 tx=0x7ff24000fc30 comp rx=0 tx=0).stop 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 shutdown_connections 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff23806ace0 0x7ff23806d190 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2580cc3d0 0x7ff25806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2580b5e60 0x7ff258156270 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 --2- 172.21.15.104:0/2054608571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2581567b0 0x7ff2580cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.381+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 >> 172.21.15.104:0/2054608571 conn(0x7ff258089fd0 msgr2=0x7ff2581553e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.382+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 shutdown_connections 2023-06-10T18:51:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.382+0000 7ff25e59c700 1 -- 172.21.15.104:0/2054608571 wait complete. 2023-06-10T18:51:50.402 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:51:50.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:51:50.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.525+0000 7ff9dc50a700 1 Processor -- start 2023-06-10T18:51:50.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.525+0000 7ff9dc50a700 1 -- start start 2023-06-10T18:51:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.526+0000 7ff9dc50a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.526+0000 7ff9dc50a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff9d4071120 0x7ff9d40c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.526+0000 7ff9dc50a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d406d7d0 con 0x7ff9d40ce390 2023-06-10T18:51:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.526+0000 7ff9dc50a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40cd6c0 con 0x7ff9d4071120 2023-06-10T18:51:50.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.526+0000 7ff9dc50a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40c95d0 con 0x7ff9d41586a0 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff9d4071120 0x7ff9d40c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9da2a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff9d4071120 0x7ff9d40c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:48342/0 (socket says 172.21.15.104:48342) 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9d9aa5700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff9d40ce390 0x7ff9d4070a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:36278/0 (socket says 172.21.15.104:36278) 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 -- 172.21.15.104:0/3174728103 learned_addr learned my addr 172.21.15.104:0/3174728103 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9da2a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:51990/0 (socket says 172.21.15.104:51990) 2023-06-10T18:51:50.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 -- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 msgr2=0x7ff9d406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 -- 172.21.15.104:0/3174728103 >> v1:172.21.15.114:6789/0 conn(0x7ff9d40ce390 legacy=0x7ff9d4070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 -- 172.21.15.104:0/3174728103 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d406d530 con 0x7ff9d4071120 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9da2a6700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9daaa7700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d4071120 0x7ff9d40c9a60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009e50 tx=0x7ff9cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0ced204d144bb88 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9d92a4700 1 -- 172.21.15.104:0/3174728103 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc00e070 con 0x7ff9d4071120 2023-06-10T18:51:50.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9d92a4700 1 -- 172.21.15.104:0/3174728103 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc00b910 con 0x7ff9d4071120 2023-06-10T18:51:50.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.527+0000 7ff9d92a4700 1 -- 172.21.15.104:0/3174728103 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc00bcc0 con 0x7ff9d4071120 2023-06-10T18:51:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 -- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d4071120 msgr2=0x7ff9d40c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d4071120 0x7ff9d40c9a60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009e50 tx=0x7ff9cc00b330 comp rx=0 tx=0).stop 2023-06-10T18:51:50.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 -- 172.21.15.104:0/3174728103 shutdown_connections 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d4071120 0x7ff9d40c9a60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/3174728103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d41586a0 0x7ff9d406cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 -- 172.21.15.104:0/3174728103 >> 172.21.15.104:0/3174728103 conn(0x7ff9d40ceec0 msgr2=0x7ff9d40cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 -- 172.21.15.104:0/3174728103 shutdown_connections 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.528+0000 7ff9dc50a700 1 -- 172.21.15.104:0/3174728103 wait complete. 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.529+0000 7ff9dc50a700 1 Processor -- start 2023-06-10T18:51:50.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.529+0000 7ff9dc50a700 1 -- start start 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d4071120 0x7ff9d41414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d40ce390 0x7ff9d40b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4141be0 con 0x7ff9d41586a0 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d41efb20 con 0x7ff9d40ce390 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9dc50a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4141e30 con 0x7ff9d4071120 2023-06-10T18:51:50.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60846/0 (socket says 172.21.15.104:60846) 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 -- 172.21.15.104:0/957371452 learned_addr learned my addr 172.21.15.104:0/957371452 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9d9aa5700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d40ce390 0x7ff9d40b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9da2a6700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d4071120 0x7ff9d41414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 -- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d4071120 msgr2=0x7ff9d41414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d4071120 0x7ff9d41414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 -- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d40ce390 msgr2=0x7ff9d40b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d40ce390 0x7ff9d40b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 -- 172.21.15.104:0/957371452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c400a640 con 0x7ff9d41586a0 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.530+0000 7ff9daaa7700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc012000 tx=0x7ff9cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0099b0 con 0x7ff9d41586a0 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc0092e0 con 0x7ff9d41586a0 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc009b30 con 0x7ff9d41586a0 2023-06-10T18:51:50.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc01c640 con 0x7ff9d41586a0 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d41423f0 con 0x7ff9d41586a0 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.531+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d4074050 con 0x7ff9d41586a0 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.532+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff9cc01c7a0 con 0x7ff9d41586a0 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.532+0000 7ff9caffd700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 0x7ff9b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.532+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff9cc08af30 con 0x7ff9d41586a0 2023-06-10T18:51:50.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.533+0000 7ff9da2a6700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 0x7ff9b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.533+0000 7ff9da2a6700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 0x7ff9b806d190 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400ad20 tx=0x7ff9c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:50.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.537+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff9cc055ea0 con 0x7ff9d41586a0 2023-06-10T18:51:50.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:50.771+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff9d40c53d0 con 0x7ff9d41586a0 2023-06-10T18:51:51.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.385+0000 7ff9caffd700 1 -- 172.21.15.104:0/957371452 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff9cc0594c0 con 0x7ff9d41586a0 2023-06-10T18:51:51.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 msgr2=0x7ff9b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 0x7ff9b806d190 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400ad20 tx=0x7ff9c4008040 comp rx=0 tx=0).stop 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 msgr2=0x7ff9d4143c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc012000 tx=0x7ff9cc004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 shutdown_connections 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d41586a0 0x7ff9d4143c70 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9d4071120 0x7ff9d41414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9b806ace0 0x7ff9b806d190 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 --2- 172.21.15.104:0/957371452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9d40ce390 0x7ff9d40b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 >> 172.21.15.104:0/957371452 conn(0x7ff9d40ceec0 msgr2=0x7ff9d4155a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:51.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 shutdown_connections 2023-06-10T18:51:51.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.388+0000 7ff9dc50a700 1 -- 172.21.15.104:0/957371452 wait complete. 2023-06-10T18:51:51.406 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:51:51.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:51.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.522+0000 7efc847c0700 1 Processor -- start 2023-06-10T18:51:51.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.522+0000 7efc847c0700 1 -- start start 2023-06-10T18:51:51.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc847c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc7c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc847c0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c06d290 con 0x7efc7c069e70 2023-06-10T18:51:51.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc847c0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0ca2d0 con 0x7efc7c06e0e0 2023-06-10T18:51:51.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc847c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c071640 con 0x7efc7c06e4b0 2023-06-10T18:51:51.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc82d5d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efc7c069e70 0x7efc7c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58682/0 (socket says 172.21.15.104:58682) 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc82d5d700 1 -- 172.21.15.104:0/1559668413 learned_addr learned my addr 172.21.15.104:0/1559668413 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc81d5b700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc7c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447421282 0 0) 0x7efc7c06d290 con 0x7efc7c069e70 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc680035b0 con 0x7efc7c069e70 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303204654 0 0) 0x7efc680035b0 con 0x7efc7c069e70 2023-06-10T18:51:51.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 msgr2=0x7efc7c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc7c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 >> v1:172.21.15.114:6789/0 conn(0x7efc7c06e0e0 legacy=0x7efc7c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c06d4c0 con 0x7efc7c069e70 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.523+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7efc740019f0 con 0x7efc7c069e70 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 >> v1:172.21.15.104:6789/0 conn(0x7efc7c069e70 legacy=0x7efc7c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68003a30 0x7efc68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc68004d90 0x7efc68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c071640 con 0x7efc68004d90 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6800bb20 con 0x7efc68003a30 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6800bce0 con 0x7efc7c06e4b0 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc68004d90 0x7efc68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc81d5b700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8255c700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68003a30 0x7efc68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 -- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 msgr2=0x7efc68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c06e4b0 0x7efc68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 -- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68003a30 msgr2=0x7efc68003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68003a30 0x7efc68003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 -- 172.21.15.104:0/1559668413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680035b0 con 0x7efc68004d90 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc82d5d700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc68004d90 0x7efc68005200 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7efc7400a830 tx=0x7efc68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a38230ace68344 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74012600 con 0x7efc68004d90 2023-06-10T18:51:51.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74012780 con 0x7efc68004d90 2023-06-10T18:51:51.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.524+0000 7efc8155a700 1 -- 172.21.15.104:0/1559668413 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74012b30 con 0x7efc68004d90 2023-06-10T18:51:51.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 -- 172.21.15.104:0/1559668413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc68004d90 msgr2=0x7efc68005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 --2- 172.21.15.104:0/1559668413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc68004d90 0x7efc68005200 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7efc7400a830 tx=0x7efc68005880 comp rx=0 tx=0).stop 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc82d5d700 1 -- 172.21.15.104:0/1559668413 reap_dead start 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 -- 172.21.15.104:0/1559668413 shutdown_connections 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 -- 172.21.15.104:0/1559668413 >> 172.21.15.104:0/1559668413 conn(0x7efc7c06b8a0 msgr2=0x7efc7c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 -- 172.21.15.104:0/1559668413 shutdown_connections 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.525+0000 7efc847c0700 1 -- 172.21.15.104:0/1559668413 wait complete. 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.526+0000 7efc847c0700 1 Processor -- start 2023-06-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.526+0000 7efc847c0700 1 -- start start 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 0x7efc7c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c0b2430 0x7efc7c0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b1190 con 0x7efc7c0b07e0 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1558a0 con 0x7efc68004d90 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc847c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b2fa0 con 0x7efc7c0b2430 2023-06-10T18:51:51.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60864/0 (socket says 172.21.15.104:60864) 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 -- 172.21.15.104:0/405121122 learned_addr learned my addr 172.21.15.104:0/405121122 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc81d5b700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c0b2430 0x7efc7c0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc8255c700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 0x7efc7c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 -- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c0b2430 msgr2=0x7efc7c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c0b2430 0x7efc7c0b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 -- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 msgr2=0x7efc7c155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 0x7efc7c155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 -- 172.21.15.104:0/405121122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0b3280 con 0x7efc7c0b07e0 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc8255c700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 0x7efc7c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc82d5d700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7efc74012000 tx=0x7efc74001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7401e840 con 0x7efc7c0b07e0 2023-06-10T18:51:51.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.527+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74022070 con 0x7efc7c0b07e0 2023-06-10T18:51:51.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.528+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c0b3560 con 0x7efc7c0b07e0 2023-06-10T18:51:51.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.528+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7401ebe0 con 0x7efc7c0b07e0 2023-06-10T18:51:51.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.528+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c0b3b20 con 0x7efc7c0b07e0 2023-06-10T18:51:51.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.528+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c0b3d40 con 0x7efc7c0b07e0 2023-06-10T18:51:51.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.529+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc7401ed40 con 0x7efc7c0b07e0 2023-06-10T18:51:51.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.529+0000 7efc72ffd700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 0x7efc6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.529+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6382+0+0 (secure 0 0 0) 0x7efc7408bca0 con 0x7efc7c0b07e0 2023-06-10T18:51:51.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.530+0000 7efc8255c700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 0x7efc6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.530+0000 7efc8255c700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 0x7efc6006d190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efc78000c00 tx=0x7efc78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.534+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc74056c00 con 0x7efc7c0b07e0 2023-06-10T18:51:51.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.761+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc7c074050 con 0x7efc7c0b07e0 2023-06-10T18:51:51.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.762+0000 7efc72ffd700 1 -- 172.21.15.104:0/405121122 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4682 (secure 0 0 0) 0x7efc7402d090 con 0x7efc7c0b07e0 2023-06-10T18:51:51.768 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:51.768 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:51:51.378743+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/414069859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":414069859},{"type":"v1","addr":"172.21.15.114:6836","nonce":414069859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 msgr2=0x7efc6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 0x7efc6006d190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efc78000c00 tx=0x7efc78009040 comp rx=0 tx=0).stop 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 msgr2=0x7efc7c0b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7efc74012000 tx=0x7efc74001d80 comp rx=0 tx=0).stop 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 shutdown_connections 2023-06-10T18:51:51.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc6006ace0 0x7efc6006d190 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc68004d90 0x7efc7c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc7c0b07e0 0x7efc7c0b0c50 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 --2- 172.21.15.104:0/405121122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc7c0b2430 0x7efc7c0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 >> 172.21.15.104:0/405121122 conn(0x7efc7c06b8a0 msgr2=0x7efc7c072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 shutdown_connections 2023-06-10T18:51:51.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.764+0000 7efc847c0700 1 -- 172.21.15.104:0/405121122 wait complete. 2023-06-10T18:51:51.772 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 40 2023-06-10T18:51:51.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:51.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 Processor -- start 2023-06-10T18:51:51.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 -- start start 2023-06-10T18:51:51.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18d015bc80 0x7f18d0070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d015c050 con 0x7f18d00ce510 2023-06-10T18:51:51.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d00c9970 con 0x7f18d00c9350 2023-06-10T18:51:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18d554e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d00ccb70 con 0x7f18d015bc80 2023-06-10T18:51:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18ceffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18d015bc80 0x7f18d0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f18d015bc80 0x7f18d0070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60870/0 (socket says 172.21.15.104:60870) 2023-06-10T18:51:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.896+0000 7f18ce7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f18d00ce510 0x7f18d00c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58700/0 (socket says 172.21.15.104:58700) 2023-06-10T18:51:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 -- 172.21.15.104:0/2816341750 learned_addr learned my addr 172.21.15.104:0/2816341750 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:51.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 -- 172.21.15.104:0/2816341750 >> v1:172.21.15.114:6789/0 conn(0x7f18d00c9350 legacy=0x7f18d00cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 -- 172.21.15.104:0/2816341750 >> v1:172.21.15.104:6789/0 conn(0x7f18d00ce510 legacy=0x7f18d00c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 -- 172.21.15.104:0/2816341750 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d00cce50 con 0x7f18d015bc80 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18cdffb700 1 -- 172.21.15.104:0/2816341750 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768477313 0 0) 0x7f18d015c050 con 0x7f18d00ce510 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18ceffd700 1 --2- 172.21.15.104:0/2816341750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d015bc80 0x7f18d0070310 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f18c400bb90 tx=0x7f18c400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91851463a9e93012 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18cdffb700 1 -- 172.21.15.104:0/2816341750 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400dd10 con 0x7f18d015bc80 2023-06-10T18:51:51.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18cdffb700 1 -- 172.21.15.104:0/2816341750 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c400de70 con 0x7f18d015bc80 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.897+0000 7f18cdffb700 1 -- 172.21.15.104:0/2816341750 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004230 con 0x7f18d015bc80 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 -- 172.21.15.104:0/2816341750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d015bc80 msgr2=0x7f18d0070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 --2- 172.21.15.104:0/2816341750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d015bc80 0x7f18d0070310 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f18c400bb90 tx=0x7f18c400bea0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 -- 172.21.15.104:0/2816341750 shutdown_connections 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 --2- 172.21.15.104:0/2816341750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d015bc80 0x7f18d0070310 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 -- 172.21.15.104:0/2816341750 >> 172.21.15.104:0/2816341750 conn(0x7f18d006ab00 msgr2=0x7f18d00b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 -- 172.21.15.104:0/2816341750 shutdown_connections 2023-06-10T18:51:51.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.898+0000 7f18d554e700 1 -- 172.21.15.104:0/2816341750 wait complete. 2023-06-10T18:51:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.899+0000 7f18d554e700 1 Processor -- start 2023-06-10T18:51:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.899+0000 7f18d554e700 1 -- start start 2023-06-10T18:51:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d00c9350 0x7f18d00c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d015bc80 0x7f18d01f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d00b2770 con 0x7f18d00ce510 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d00b2180 con 0x7f18d015bc80 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18d554e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d01f30d0 con 0x7f18d00c9350 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18ce7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18ce7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60878/0 (socket says 172.21.15.104:60878) 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.900+0000 7f18ce7fc700 1 -- 172.21.15.104:0/3954804194 learned_addr learned my addr 172.21.15.104:0/3954804194 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:51.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18cf7fe700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d015bc80 0x7f18d01f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ceffd700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d00c9350 0x7f18d00c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 -- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d00c9350 msgr2=0x7f18d00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d00c9350 0x7f18d00c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 -- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d015bc80 msgr2=0x7f18d01f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d015bc80 0x7f18d01f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 -- 172.21.15.104:0/3954804194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c400b7f0 con 0x7f18d00ce510 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18ce7fc700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f18c00011e0 tx=0x7f18c000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0004c80 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0004e00 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d01f33b0 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0007310 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.901+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d00c49c0 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.902+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d0067940 con 0x7f18d00ce510 2023-06-10T18:51:51.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.902+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f18c0007470 con 0x7f18d00ce510 2023-06-10T18:51:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.903+0000 7f18b77fe700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 0x7f18bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.903+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f18c0089360 con 0x7f18d00ce510 2023-06-10T18:51:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.903+0000 7f18ceffd700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 0x7f18bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.903+0000 7f18ceffd700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 0x7f18bc06d190 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f18c400b340 tx=0x7f18c4013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:51.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:51.907+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f18c00542d0 con 0x7f18d00ce510 2023-06-10T18:51:52.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.092+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18d0074050 con 0x7f18d00ce510 2023-06-10T18:51:52.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.093+0000 7f18b77fe700 1 -- 172.21.15.104:0/3954804194 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18595 (secure 0 0 0) 0x7f18c00578f0 con 0x7f18d00ce510 2023-06-10T18:51:52.100 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:52.100 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":83,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:49.371827+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:52.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 msgr2=0x7f18bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 0x7f18bc06d190 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f18c400b340 tx=0x7f18c4013000 comp rx=0 tx=0).stop 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 msgr2=0x7f18d00c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f18c00011e0 tx=0x7f18c000cae0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 shutdown_connections 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f18bc06ace0 0x7f18bc06d190 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d015bc80 0x7f18d01f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d00c9350 0x7f18d00c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 --2- 172.21.15.104:0/3954804194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f18d00ce510 0x7f18d00c20b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.095+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 >> 172.21.15.104:0/3954804194 conn(0x7f18d006ab00 msgr2=0x7f18d00ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:52.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.096+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 shutdown_connections 2023-06-10T18:51:52.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.096+0000 7f18d554e700 1 -- 172.21.15.104:0/3954804194 wait complete. 2023-06-10T18:51:52.113 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:51:52.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:51:52.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.228+0000 7efd77ef9700 1 Processor -- start 2023-06-10T18:51:52.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.228+0000 7efd77ef9700 1 -- start start 2023-06-10T18:51:52.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd77ef9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd77ef9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 0x7efd7006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd77ef9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd700cdae0 con 0x7efd700b1080 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd77ef9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700c9bc0 con 0x7efd700cd680 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd77ef9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700c9e60 con 0x7efd700aeaa0 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd76496700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 0x7efd7006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:52036/0 (socket says 172.21.15.104:52036) 2023-06-10T18:51:52.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 -- 172.21.15.104:0/3250643544 learned_addr learned my addr 172.21.15.104:0/3250643544 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:52.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 -- 172.21.15.104:0/3250643544 >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 msgr2=0x7efd7006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:52.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 --2- 172.21.15.104:0/3250643544 >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 0x7efd7006dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 -- 172.21.15.104:0/3250643544 >> v1:172.21.15.114:6789/0 conn(0x7efd700b1080 legacy=0x7efd700ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.229+0000 7efd75c95700 1 -- 172.21.15.104:0/3250643544 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700aee70 con 0x7efd700aeaa0 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd76496700 1 --2- 172.21.15.104:0/3250643544 >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 0x7efd7006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd75c95700 1 --2- 172.21.15.104:0/3250643544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006fd0 tx=0x7efd6c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a3e38106cb84b6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd74c93700 1 -- 172.21.15.104:0/3250643544 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c005e30 con 0x7efd700aeaa0 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd74c93700 1 -- 172.21.15.104:0/3250643544 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c004030 con 0x7efd700aeaa0 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd74c93700 1 -- 172.21.15.104:0/3250643544 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0043e0 con 0x7efd700aeaa0 2023-06-10T18:51:52.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 -- 172.21.15.104:0/3250643544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 msgr2=0x7efd700cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 --2- 172.21.15.104:0/3250643544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006fd0 tx=0x7efd6c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 -- 172.21.15.104:0/3250643544 shutdown_connections 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 --2- 172.21.15.104:0/3250643544 >> v2:172.21.15.114:3300/0 conn(0x7efd700cd680 0x7efd7006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 --2- 172.21.15.104:0/3250643544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700aeaa0 0x7efd700cd140 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 -- 172.21.15.104:0/3250643544 >> 172.21.15.104:0/3250643544 conn(0x7efd7006b8a0 msgr2=0x7efd700cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.230+0000 7efd77ef9700 1 -- 172.21.15.104:0/3250643544 shutdown_connections 2023-06-10T18:51:52.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.231+0000 7efd77ef9700 1 -- 172.21.15.104:0/3250643544 wait complete. 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.232+0000 7efd77ef9700 1 Processor -- start 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.232+0000 7efd77ef9700 1 -- start start 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 0x7efd70153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 0x7efd70152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700c7bc0 con 0x7efd700cd680 2023-06-10T18:51:52.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700c66e0 con 0x7efd700aeaa0 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd77ef9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd70153d60 con 0x7efd700b1080 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60894/0 (socket says 172.21.15.104:60894) 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 -- 172.21.15.104:0/2969762400 learned_addr learned my addr 172.21.15.104:0/2969762400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd75494700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 0x7efd70152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd75c95700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 0x7efd70153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 -- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 msgr2=0x7efd70152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 0x7efd70152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 -- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 msgr2=0x7efd70153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 0x7efd70153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd76496700 1 -- 172.21.15.104:0/2969762400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009a70 con 0x7efd700cd680 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd75494700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 0x7efd70152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.233+0000 7efd75c95700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 0x7efd70153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd76496700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efd6800efc0 tx=0x7efd6800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68009cd0 con 0x7efd700cd680 2023-06-10T18:51:52.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800f040 con 0x7efd700cd680 2023-06-10T18:51:52.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800cd60 con 0x7efd700cd680 2023-06-10T18:51:52.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c009500 con 0x7efd700cd680 2023-06-10T18:51:52.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.234+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd700c83d0 con 0x7efd700cd680 2023-06-10T18:51:52.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.235+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd700c68a0 con 0x7efd700cd680 2023-06-10T18:51:52.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.235+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd68014480 con 0x7efd700cd680 2023-06-10T18:51:52.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.235+0000 7efd667fc700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 0x7efd5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.236+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6382+0+0 (secure 0 0 0) 0x7efd68089e50 con 0x7efd700cd680 2023-06-10T18:51:52.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.236+0000 7efd75c95700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 0x7efd5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.236+0000 7efd75c95700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 0x7efd5406d190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006fd0 tx=0x7efd6c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.240+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd68054c90 con 0x7efd700cd680 2023-06-10T18:51:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.430+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd7006e540 con 0x7efd5406ace0 2023-06-10T18:51:52.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.440+0000 7efd667fc700 1 -- 172.21.15.104:0/2969762400 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311925 (secure 0 0 0) 0x7efd6c004460 con 0x7efd5406ace0 2023-06-10T18:51:52.449 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 msgr2=0x7efd5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 0x7efd5406d190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efd6c006fd0 tx=0x7efd6c008070 comp rx=0 tx=0).stop 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 msgr2=0x7efd700b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efd6800efc0 tx=0x7efd6800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 shutdown_connections 2023-06-10T18:51:52.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efd5406ace0 0x7efd5406d190 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd700aeaa0 0x7efd70153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd700cd680 0x7efd700b42a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 --2- 172.21.15.104:0/2969762400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd700b1080 0x7efd70152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 >> 172.21.15.104:0/2969762400 conn(0x7efd7006b8a0 msgr2=0x7efd700cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 shutdown_connections 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.444+0000 7efd77ef9700 1 -- 172.21.15.104:0/2969762400 wait complete. 2023-06-10T18:51:52.452 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:51:52.461 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-06-10T18:51:52.396168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224760,"kb_used_data":9144,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754749960,"statfs":{"total":773094113280,"available":772863959040,"internally_reserved":0,"allocated":9363456,"data_stored":5045456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.923073"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383798+0000","last_change":"2023-06-10T18:51:48.399525+0000","last_active":"2023-06-10T18:51:49.383798+0000","last_peered":"2023-06-10T18:51:49.383798+0000","last_clean":"2023-06-10T18:51:49.383798+0000","last_became_active":"2023-06-10T18:51:48.399164+0000","last_became_peered":"2023-06-10T18:51:48.399164+0000","last_unstale":"2023-06-10T18:51:49.383798+0000","last_undegraded":"2023-06-10T18:51:49.383798+0000","last_fullsized":"2023-06-10T18:51:49.383798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:09.337408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380763+0000","last_change":"2023-06-10T18:51:48.399833+0000","last_active":"2023-06-10T18:51:49.380763+0000","last_peered":"2023-06-10T18:51:49.380763+0000","last_clean":"2023-06-10T18:51:49.380763+0000","last_became_active":"2023-06-10T18:51:48.399578+0000","last_became_peered":"2023-06-10T18:51:48.399578+0000","last_unstale":"2023-06-10T18:51:49.380763+0000","last_undegraded":"2023-06-10T18:51:49.380763+0000","last_fullsized":"2023-06-10T18:51:49.380763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:54.726570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377812+0000","last_change":"2023-06-10T18:51:48.395619+0000","last_active":"2023-06-10T18:51:49.377812+0000","last_peered":"2023-06-10T18:51:49.377812+0000","last_clean":"2023-06-10T18:51:49.377812+0000","last_became_active":"2023-06-10T18:51:48.395361+0000","last_became_peered":"2023-06-10T18:51:48.395361+0000","last_unstale":"2023-06-10T18:51:49.377812+0000","last_undegraded":"2023-06-10T18:51:49.377812+0000","last_fullsized":"2023-06-10T18:51:49.377812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:14.847162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624696+0000","last_change":"2023-06-10T18:51:48.403461+0000","last_active":"2023-06-10T18:51:49.624696+0000","last_peered":"2023-06-10T18:51:49.624696+0000","last_clean":"2023-06-10T18:51:49.624696+0000","last_became_active":"2023-06-10T18:51:48.403184+0000","last_became_peered":"2023-06-10T18:51:48.403184+0000","last_unstale":"2023-06-10T18:51:49.624696+0000","last_undegraded":"2023-06-10T18:51:49.624696+0000","last_fullsized":"2023-06-10T18:51:49.624696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:44:35.982855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381873+0000","last_change":"2023-06-10T18:51:48.403100+0000","last_active":"2023-06-10T18:51:49.381873+0000","last_peered":"2023-06-10T18:51:49.381873+0000","last_clean":"2023-06-10T18:51:49.381873+0000","last_became_active":"2023-06-10T18:51:48.402775+0000","last_became_peered":"2023-06-10T18:51:48.402775+0000","last_unstale":"2023-06-10T18:51:49.381873+0000","last_undegraded":"2023-06-10T18:51:49.381873+0000","last_fullsized":"2023-06-10T18:51:49.381873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:10.156603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380538+0000","last_change":"2023-06-10T18:51:48.392649+0000","last_active":"2023-06-10T18:51:49.380538+0000","last_peered":"2023-06-10T18:51:49.380538+0000","last_clean":"2023-06-10T18:51:49.380538+0000","last_became_active":"2023-06-10T18:51:48.392384+0000","last_became_peered":"2023-06-10T18:51:48.392384+0000","last_unstale":"2023-06-10T18:51:49.380538+0000","last_undegraded":"2023-06-10T18:51:49.380538+0000","last_fullsized":"2023-06-10T18:51:49.380538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:16.961282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624303+0000","last_change":"2023-06-10T18:51:48.391101+0000","last_active":"2023-06-10T18:51:49.624303+0000","last_peered":"2023-06-10T18:51:49.624303+0000","last_clean":"2023-06-10T18:51:49.624303+0000","last_became_active":"2023-06-10T18:51:48.390794+0000","last_became_peered":"2023-06-10T18:51:48.390794+0000","last_unstale":"2023-06-10T18:51:49.624303+0000","last_undegraded":"2023-06-10T18:51:49.624303+0000","last_fullsized":"2023-06-10T18:51:49.624303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:27.376706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380490+0000","last_change":"2023-06-10T18:51:48.394711+0000","last_active":"2023-06-10T18:51:49.380490+0000","last_peered":"2023-06-10T18:51:49.380490+0000","last_clean":"2023-06-10T18:51:49.380490+0000","last_became_active":"2023-06-10T18:51:48.394352+0000","last_became_peered":"2023-06-10T18:51:48.394352+0000","last_unstale":"2023-06-10T18:51:49.380490+0000","last_undegraded":"2023-06-10T18:51:49.380490+0000","last_fullsized":"2023-06-10T18:51:49.380490+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:58:13.773736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380930+0000","last_change":"2023-06-10T18:51:48.388605+0000","last_active":"2023-06-10T18:51:49.380930+0000","last_peered":"2023-06-10T18:51:49.380930+0000","last_clean":"2023-06-10T18:51:49.380930+0000","last_became_active":"2023-06-10T18:51:48.388337+0000","last_became_peered":"2023-06-10T18:51:48.388337+0000","last_unstale":"2023-06-10T18:51:49.380930+0000","last_undegraded":"2023-06-10T18:51:49.380930+0000","last_fullsized":"2023-06-10T18:51:49.380930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:11:27.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383731+0000","last_change":"2023-06-10T18:51:48.398604+0000","last_active":"2023-06-10T18:51:49.383731+0000","last_peered":"2023-06-10T18:51:49.383731+0000","last_clean":"2023-06-10T18:51:49.383731+0000","last_became_active":"2023-06-10T18:51:48.398187+0000","last_became_peered":"2023-06-10T18:51:48.398187+0000","last_unstale":"2023-06-10T18:51:49.383731+0000","last_undegraded":"2023-06-10T18:51:49.383731+0000","last_fullsized":"2023-06-10T18:51:49.383731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380842+0000","last_change":"2023-06-10T18:51:48.401796+0000","last_active":"2023-06-10T18:51:49.380842+0000","last_peered":"2023-06-10T18:51:49.380842+0000","last_clean":"2023-06-10T18:51:49.380842+0000","last_became_active":"2023-06-10T18:51:48.401574+0000","last_became_peered":"2023-06-10T18:51:48.401574+0000","last_unstale":"2023-06-10T18:51:49.380842+0000","last_undegraded":"2023-06-10T18:51:49.380842+0000","last_fullsized":"2023-06-10T18:51:49.380842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383304+0000","last_change":"2023-06-10T18:51:48.399880+0000","last_active":"2023-06-10T18:51:49.383304+0000","last_peered":"2023-06-10T18:51:49.383304+0000","last_clean":"2023-06-10T18:51:49.383304+0000","last_became_active":"2023-06-10T18:51:48.399574+0000","last_became_peered":"2023-06-10T18:51:48.399574+0000","last_unstale":"2023-06-10T18:51:49.383304+0000","last_undegraded":"2023-06-10T18:51:49.383304+0000","last_fullsized":"2023-06-10T18:51:49.383304+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:54:58.346559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377767+0000","last_change":"2023-06-10T18:51:48.393189+0000","last_active":"2023-06-10T18:51:49.377767+0000","last_peered":"2023-06-10T18:51:49.377767+0000","last_clean":"2023-06-10T18:51:49.377767+0000","last_became_active":"2023-06-10T18:51:48.391396+0000","last_became_peered":"2023-06-10T18:51:48.391396+0000","last_unstale":"2023-06-10T18:51:49.377767+0000","last_undegraded":"2023-06-10T18:51:49.377767+0000","last_fullsized":"2023-06-10T18:51:49.377767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:15:15.817356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624648+0000","last_change":"2023-06-10T18:51:48.403589+0000","last_active":"2023-06-10T18:51:49.624648+0000","last_peered":"2023-06-10T18:51:49.624648+0000","last_clean":"2023-06-10T18:51:49.624648+0000","last_became_active":"2023-06-10T18:51:48.403352+0000","last_became_peered":"2023-06-10T18:51:48.403352+0000","last_unstale":"2023-06-10T18:51:49.624648+0000","last_undegraded":"2023-06-10T18:51:49.624648+0000","last_fullsized":"2023-06-10T18:51:49.624648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:36:55.439108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623832+0000","last_change":"2023-06-10T18:51:48.390644+0000","last_active":"2023-06-10T18:51:49.623832+0000","last_peered":"2023-06-10T18:51:49.623832+0000","last_clean":"2023-06-10T18:51:49.623832+0000","last_became_active":"2023-06-10T18:51:48.390409+0000","last_became_peered":"2023-06-10T18:51:48.390409+0000","last_unstale":"2023-06-10T18:51:49.623832+0000","last_undegraded":"2023-06-10T18:51:49.623832+0000","last_fullsized":"2023-06-10T18:51:49.623832+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381807+0000","last_change":"2023-06-10T18:51:48.397500+0000","last_active":"2023-06-10T18:51:49.381807+0000","last_peered":"2023-06-10T18:51:49.381807+0000","last_clean":"2023-06-10T18:51:49.381807+0000","last_became_active":"2023-06-10T18:51:48.397179+0000","last_became_peered":"2023-06-10T18:51:48.397179+0000","last_unstale":"2023-06-10T18:51:49.381807+0000","last_undegraded":"2023-06-10T18:51:49.381807+0000","last_fullsized":"2023-06-10T18:51:49.381807+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:30:44.869672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382168+0000","last_change":"2023-06-10T18:51:48.388377+0000","last_active":"2023-06-10T18:51:49.382168+0000","last_peered":"2023-06-10T18:51:49.382168+0000","last_clean":"2023-06-10T18:51:49.382168+0000","last_became_active":"2023-06-10T18:51:48.386618+0000","last_became_peered":"2023-06-10T18:51:48.386618+0000","last_unstale":"2023-06-10T18:51:49.382168+0000","last_undegraded":"2023-06-10T18:51:49.382168+0000","last_fullsized":"2023-06-10T18:51:49.382168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:51.791581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625256+0000","last_change":"2023-06-10T18:51:48.399189+0000","last_active":"2023-06-10T18:51:49.625256+0000","last_peered":"2023-06-10T18:51:49.625256+0000","last_clean":"2023-06-10T18:51:49.625256+0000","last_became_active":"2023-06-10T18:51:48.398915+0000","last_became_peered":"2023-06-10T18:51:48.398915+0000","last_unstale":"2023-06-10T18:51:49.625256+0000","last_undegraded":"2023-06-10T18:51:49.625256+0000","last_fullsized":"2023-06-10T18:51:49.625256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:09.165115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625138+0000","last_change":"2023-06-10T18:51:48.391744+0000","last_active":"2023-06-10T18:51:49.625138+0000","last_peered":"2023-06-10T18:51:49.625138+0000","last_clean":"2023-06-10T18:51:49.625138+0000","last_became_active":"2023-06-10T18:51:48.391495+0000","last_became_peered":"2023-06-10T18:51:48.391495+0000","last_unstale":"2023-06-10T18:51:49.625138+0000","last_undegraded":"2023-06-10T18:51:49.625138+0000","last_fullsized":"2023-06-10T18:51:49.625138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:59:51.205176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624891+0000","last_change":"2023-06-10T18:51:48.394708+0000","last_active":"2023-06-10T18:51:49.624891+0000","last_peered":"2023-06-10T18:51:49.624891+0000","last_clean":"2023-06-10T18:51:49.624891+0000","last_became_active":"2023-06-10T18:51:48.394442+0000","last_became_peered":"2023-06-10T18:51:48.394442+0000","last_unstale":"2023-06-10T18:51:49.624891+0000","last_undegraded":"2023-06-10T18:51:49.624891+0000","last_fullsized":"2023-06-10T18:51:49.624891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:19:39.745961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624955+0000","last_change":"2023-06-10T18:51:48.396079+0000","last_active":"2023-06-10T18:51:49.624955+0000","last_peered":"2023-06-10T18:51:49.624955+0000","last_clean":"2023-06-10T18:51:49.624955+0000","last_became_active":"2023-06-10T18:51:48.393682+0000","last_became_peered":"2023-06-10T18:51:48.393682+0000","last_unstale":"2023-06-10T18:51:49.624955+0000","last_undegraded":"2023-06-10T18:51:49.624955+0000","last_fullsized":"2023-06-10T18:51:49.624955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:29:31.472028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380554+0000","last_change":"2023-06-10T18:51:48.401012+0000","last_active":"2023-06-10T18:51:49.380554+0000","last_peered":"2023-06-10T18:51:49.380554+0000","last_clean":"2023-06-10T18:51:49.380554+0000","last_became_active":"2023-06-10T18:51:48.400756+0000","last_became_peered":"2023-06-10T18:51:48.400756+0000","last_unstale":"2023-06-10T18:51:49.380554+0000","last_undegraded":"2023-06-10T18:51:49.380554+0000","last_fullsized":"2023-06-10T18:51:49.380554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:11.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383216+0000","last_change":"2023-06-10T18:51:48.397643+0000","last_active":"2023-06-10T18:51:49.383216+0000","last_peered":"2023-06-10T18:51:49.383216+0000","last_clean":"2023-06-10T18:51:49.383216+0000","last_became_active":"2023-06-10T18:51:48.397334+0000","last_became_peered":"2023-06-10T18:51:48.397334+0000","last_unstale":"2023-06-10T18:51:49.383216+0000","last_undegraded":"2023-06-10T18:51:49.383216+0000","last_fullsized":"2023-06-10T18:51:49.383216+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:16.685695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624100+0000","last_change":"2023-06-10T18:51:48.399459+0000","last_active":"2023-06-10T18:51:49.624100+0000","last_peered":"2023-06-10T18:51:49.624100+0000","last_clean":"2023-06-10T18:51:49.624100+0000","last_became_active":"2023-06-10T18:51:48.399176+0000","last_became_peered":"2023-06-10T18:51:48.399176+0000","last_unstale":"2023-06-10T18:51:49.624100+0000","last_undegraded":"2023-06-10T18:51:49.624100+0000","last_fullsized":"2023-06-10T18:51:49.624100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:24.130103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380970+0000","last_change":"2023-06-10T18:51:48.400544+0000","last_active":"2023-06-10T18:51:49.380970+0000","last_peered":"2023-06-10T18:51:49.380970+0000","last_clean":"2023-06-10T18:51:49.380970+0000","last_became_active":"2023-06-10T18:51:48.400277+0000","last_became_peered":"2023-06-10T18:51:48.400277+0000","last_unstale":"2023-06-10T18:51:49.380970+0000","last_undegraded":"2023-06-10T18:51:49.380970+0000","last_fullsized":"2023-06-10T18:51:49.380970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:47.547020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378028+0000","last_change":"2023-06-10T18:51:48.401372+0000","last_active":"2023-06-10T18:51:49.378028+0000","last_peered":"2023-06-10T18:51:49.378028+0000","last_clean":"2023-06-10T18:51:49.378028+0000","last_became_active":"2023-06-10T18:51:48.401092+0000","last_became_peered":"2023-06-10T18:51:48.401092+0000","last_unstale":"2023-06-10T18:51:49.378028+0000","last_undegraded":"2023-06-10T18:51:49.378028+0000","last_fullsized":"2023-06-10T18:51:49.378028+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:39.409186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382269+0000","last_change":"2023-06-10T18:51:48.393667+0000","last_active":"2023-06-10T18:51:49.382269+0000","last_peered":"2023-06-10T18:51:49.382269+0000","last_clean":"2023-06-10T18:51:49.382269+0000","last_became_active":"2023-06-10T18:51:48.393342+0000","last_became_peered":"2023-06-10T18:51:48.393342+0000","last_unstale":"2023-06-10T18:51:49.382269+0000","last_undegraded":"2023-06-10T18:51:49.382269+0000","last_fullsized":"2023-06-10T18:51:49.382269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:12.665569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383498+0000","last_change":"2023-06-10T18:51:48.396836+0000","last_active":"2023-06-10T18:51:49.383498+0000","last_peered":"2023-06-10T18:51:49.383498+0000","last_clean":"2023-06-10T18:51:49.383498+0000","last_became_active":"2023-06-10T18:51:48.396395+0000","last_became_peered":"2023-06-10T18:51:48.396395+0000","last_unstale":"2023-06-10T18:51:49.383498+0000","last_undegraded":"2023-06-10T18:51:49.383498+0000","last_fullsized":"2023-06-10T18:51:49.383498+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:06.492459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380732+0000","last_change":"2023-06-10T18:51:48.396566+0000","last_active":"2023-06-10T18:51:49.380732+0000","last_peered":"2023-06-10T18:51:49.380732+0000","last_clean":"2023-06-10T18:51:49.380732+0000","last_became_active":"2023-06-10T18:51:48.396294+0000","last_became_peered":"2023-06-10T18:51:48.396294+0000","last_unstale":"2023-06-10T18:51:49.380732+0000","last_undegraded":"2023-06-10T18:51:49.380732+0000","last_fullsized":"2023-06-10T18:51:49.380732+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:06.492459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380751+0000","last_change":"2023-06-10T18:51:48.396468+0000","last_active":"2023-06-10T18:51:49.380751+0000","last_peered":"2023-06-10T18:51:49.380751+0000","last_clean":"2023-06-10T18:51:49.380751+0000","last_became_active":"2023-06-10T18:51:48.396207+0000","last_became_peered":"2023-06-10T18:51:48.396207+0000","last_unstale":"2023-06-10T18:51:49.380751+0000","last_undegraded":"2023-06-10T18:51:49.380751+0000","last_fullsized":"2023-06-10T18:51:49.380751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:21:28.376204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382130+0000","last_change":"2023-06-10T18:51:48.386674+0000","last_active":"2023-06-10T18:51:49.382130+0000","last_peered":"2023-06-10T18:51:49.382130+0000","last_clean":"2023-06-10T18:51:49.382130+0000","last_became_active":"2023-06-10T18:51:48.386084+0000","last_became_peered":"2023-06-10T18:51:48.386084+0000","last_unstale":"2023-06-10T18:51:49.382130+0000","last_undegraded":"2023-06-10T18:51:49.382130+0000","last_fullsized":"2023-06-10T18:51:49.382130+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:52.150020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383209+0000","last_change":"2023-06-10T18:51:48.403182+0000","last_active":"2023-06-10T18:51:49.383209+0000","last_peered":"2023-06-10T18:51:49.383209+0000","last_clean":"2023-06-10T18:51:49.383209+0000","last_became_active":"2023-06-10T18:51:48.402915+0000","last_became_peered":"2023-06-10T18:51:48.402915+0000","last_unstale":"2023-06-10T18:51:49.383209+0000","last_undegraded":"2023-06-10T18:51:49.383209+0000","last_fullsized":"2023-06-10T18:51:49.383209+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:16.066951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382423+0000","last_change":"2023-06-10T18:51:48.399859+0000","last_active":"2023-06-10T18:51:49.382423+0000","last_peered":"2023-06-10T18:51:49.382423+0000","last_clean":"2023-06-10T18:51:49.382423+0000","last_became_active":"2023-06-10T18:51:48.399589+0000","last_became_peered":"2023-06-10T18:51:48.399589+0000","last_unstale":"2023-06-10T18:51:49.382423+0000","last_undegraded":"2023-06-10T18:51:49.382423+0000","last_fullsized":"2023-06-10T18:51:49.382423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:29.310115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383983+0000","last_change":"2023-06-10T18:51:47.987681+0000","last_active":"2023-06-10T18:51:49.383983+0000","last_peered":"2023-06-10T18:51:49.383983+0000","last_clean":"2023-06-10T18:51:49.383983+0000","last_became_active":"2023-06-10T18:51:47.987266+0000","last_became_peered":"2023-06-10T18:51:47.987266+0000","last_unstale":"2023-06-10T18:51:49.383983+0000","last_undegraded":"2023-06-10T18:51:49.383983+0000","last_fullsized":"2023-06-10T18:51:49.383983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:20.641457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.384089+0000","last_change":"2023-06-10T18:51:48.399497+0000","last_active":"2023-06-10T18:51:49.384089+0000","last_peered":"2023-06-10T18:51:49.384089+0000","last_clean":"2023-06-10T18:51:49.384089+0000","last_became_active":"2023-06-10T18:51:48.398830+0000","last_became_peered":"2023-06-10T18:51:48.398830+0000","last_unstale":"2023-06-10T18:51:49.384089+0000","last_undegraded":"2023-06-10T18:51:49.384089+0000","last_fullsized":"2023-06-10T18:51:49.384089+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:46.167170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382344+0000","last_change":"2023-06-10T18:51:47.991139+0000","last_active":"2023-06-10T18:51:49.382344+0000","last_peered":"2023-06-10T18:51:49.382344+0000","last_clean":"2023-06-10T18:51:49.382344+0000","last_became_active":"2023-06-10T18:51:47.990905+0000","last_became_peered":"2023-06-10T18:51:47.990905+0000","last_unstale":"2023-06-10T18:51:49.382344+0000","last_undegraded":"2023-06-10T18:51:49.382344+0000","last_fullsized":"2023-06-10T18:51:49.382344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:57.547464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383455+0000","last_change":"2023-06-10T18:51:48.401666+0000","last_active":"2023-06-10T18:51:49.383455+0000","last_peered":"2023-06-10T18:51:49.383455+0000","last_clean":"2023-06-10T18:51:49.383455+0000","last_became_active":"2023-06-10T18:51:48.401343+0000","last_became_peered":"2023-06-10T18:51:48.401343+0000","last_unstale":"2023-06-10T18:51:49.383455+0000","last_undegraded":"2023-06-10T18:51:49.383455+0000","last_fullsized":"2023-06-10T18:51:49.383455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:21:28.376204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625450+0000","last_change":"2023-06-10T18:51:47.987428+0000","last_active":"2023-06-10T18:51:49.625450+0000","last_peered":"2023-06-10T18:51:49.625450+0000","last_clean":"2023-06-10T18:51:49.625450+0000","last_became_active":"2023-06-10T18:51:47.987205+0000","last_became_peered":"2023-06-10T18:51:47.987205+0000","last_unstale":"2023-06-10T18:51:49.625450+0000","last_undegraded":"2023-06-10T18:51:49.625450+0000","last_fullsized":"2023-06-10T18:51:49.625450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:15.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377930+0000","last_change":"2023-06-10T18:51:48.392214+0000","last_active":"2023-06-10T18:51:49.377930+0000","last_peered":"2023-06-10T18:51:49.377930+0000","last_clean":"2023-06-10T18:51:49.377930+0000","last_became_active":"2023-06-10T18:51:48.391937+0000","last_became_peered":"2023-06-10T18:51:48.391937+0000","last_unstale":"2023-06-10T18:51:49.377930+0000","last_undegraded":"2023-06-10T18:51:49.377930+0000","last_fullsized":"2023-06-10T18:51:49.377930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:43.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625176+0000","last_change":"2023-06-10T18:51:47.987865+0000","last_active":"2023-06-10T18:51:49.625176+0000","last_peered":"2023-06-10T18:51:49.625176+0000","last_clean":"2023-06-10T18:51:49.625176+0000","last_became_active":"2023-06-10T18:51:47.987614+0000","last_became_peered":"2023-06-10T18:51:47.987614+0000","last_unstale":"2023-06-10T18:51:49.625176+0000","last_undegraded":"2023-06-10T18:51:49.625176+0000","last_fullsized":"2023-06-10T18:51:49.625176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:56.618982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380631+0000","last_change":"2023-06-10T18:51:48.400984+0000","last_active":"2023-06-10T18:51:49.380631+0000","last_peered":"2023-06-10T18:51:49.380631+0000","last_clean":"2023-06-10T18:51:49.380631+0000","last_became_active":"2023-06-10T18:51:48.400402+0000","last_became_peered":"2023-06-10T18:51:48.400402+0000","last_unstale":"2023-06-10T18:51:49.380631+0000","last_undegraded":"2023-06-10T18:51:49.380631+0000","last_fullsized":"2023-06-10T18:51:49.380631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:34.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382095+0000","last_change":"2023-06-10T18:51:47.990886+0000","last_active":"2023-06-10T18:51:49.382095+0000","last_peered":"2023-06-10T18:51:49.382095+0000","last_clean":"2023-06-10T18:51:49.382095+0000","last_became_active":"2023-06-10T18:51:47.990649+0000","last_became_peered":"2023-06-10T18:51:47.990649+0000","last_unstale":"2023-06-10T18:51:49.382095+0000","last_undegraded":"2023-06-10T18:51:49.382095+0000","last_fullsized":"2023-06-10T18:51:49.382095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:38:32.258439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382098+0000","last_change":"2023-06-10T18:51:48.386807+0000","last_active":"2023-06-10T18:51:49.382098+0000","last_peered":"2023-06-10T18:51:49.382098+0000","last_clean":"2023-06-10T18:51:49.382098+0000","last_became_active":"2023-06-10T18:51:48.386503+0000","last_became_peered":"2023-06-10T18:51:48.386503+0000","last_unstale":"2023-06-10T18:51:49.382098+0000","last_undegraded":"2023-06-10T18:51:49.382098+0000","last_fullsized":"2023-06-10T18:51:49.382098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:47:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381611+0000","last_change":"2023-06-10T18:51:47.995836+0000","last_active":"2023-06-10T18:51:49.381611+0000","last_peered":"2023-06-10T18:51:49.381611+0000","last_clean":"2023-06-10T18:51:49.381611+0000","last_became_active":"2023-06-10T18:51:47.995594+0000","last_became_peered":"2023-06-10T18:51:47.995594+0000","last_unstale":"2023-06-10T18:51:49.381611+0000","last_undegraded":"2023-06-10T18:51:49.381611+0000","last_fullsized":"2023-06-10T18:51:49.381611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:26:22.505078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377656+0000","last_change":"2023-06-10T18:51:48.394078+0000","last_active":"2023-06-10T18:51:49.377656+0000","last_peered":"2023-06-10T18:51:49.377656+0000","last_clean":"2023-06-10T18:51:49.377656+0000","last_became_active":"2023-06-10T18:51:48.391671+0000","last_became_peered":"2023-06-10T18:51:48.391671+0000","last_unstale":"2023-06-10T18:51:49.377656+0000","last_undegraded":"2023-06-10T18:51:49.377656+0000","last_fullsized":"2023-06-10T18:51:49.377656+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:53:37.194624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623983+0000","last_change":"2023-06-10T18:51:47.985868+0000","last_active":"2023-06-10T18:51:49.623983+0000","last_peered":"2023-06-10T18:51:49.623983+0000","last_clean":"2023-06-10T18:51:49.623983+0000","last_became_active":"2023-06-10T18:51:47.985572+0000","last_became_peered":"2023-06-10T18:51:47.985572+0000","last_unstale":"2023-06-10T18:51:49.623983+0000","last_undegraded":"2023-06-10T18:51:49.623983+0000","last_fullsized":"2023-06-10T18:51:49.623983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:11.403051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623960+0000","last_change":"2023-06-10T18:51:48.387610+0000","last_active":"2023-06-10T18:51:49.623960+0000","last_peered":"2023-06-10T18:51:49.623960+0000","last_clean":"2023-06-10T18:51:49.623960+0000","last_became_active":"2023-06-10T18:51:48.387276+0000","last_became_peered":"2023-06-10T18:51:48.387276+0000","last_unstale":"2023-06-10T18:51:49.623960+0000","last_undegraded":"2023-06-10T18:51:49.623960+0000","last_fullsized":"2023-06-10T18:51:49.623960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:03.200969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380505+0000","last_change":"2023-06-10T18:51:47.991946+0000","last_active":"2023-06-10T18:51:49.380505+0000","last_peered":"2023-06-10T18:51:49.380505+0000","last_clean":"2023-06-10T18:51:49.380505+0000","last_became_active":"2023-06-10T18:51:47.991654+0000","last_became_peered":"2023-06-10T18:51:47.991654+0000","last_unstale":"2023-06-10T18:51:49.380505+0000","last_undegraded":"2023-06-10T18:51:49.380505+0000","last_fullsized":"2023-06-10T18:51:49.380505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:54:57.340884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624960+0000","last_change":"2023-06-10T18:51:48.395577+0000","last_active":"2023-06-10T18:51:49.624960+0000","last_peered":"2023-06-10T18:51:49.624960+0000","last_clean":"2023-06-10T18:51:49.624960+0000","last_became_active":"2023-06-10T18:51:48.395287+0000","last_became_peered":"2023-06-10T18:51:48.395287+0000","last_unstale":"2023-06-10T18:51:49.624960+0000","last_undegraded":"2023-06-10T18:51:49.624960+0000","last_fullsized":"2023-06-10T18:51:49.624960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:47.547020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383953+0000","last_change":"2023-06-10T18:51:47.990334+0000","last_active":"2023-06-10T18:51:49.383953+0000","last_peered":"2023-06-10T18:51:49.383953+0000","last_clean":"2023-06-10T18:51:49.383953+0000","last_became_active":"2023-06-10T18:51:47.990065+0000","last_became_peered":"2023-06-10T18:51:47.990065+0000","last_unstale":"2023-06-10T18:51:49.383953+0000","last_undegraded":"2023-06-10T18:51:49.383953+0000","last_fullsized":"2023-06-10T18:51:49.383953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:31.385016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381021+0000","last_change":"2023-06-10T18:51:48.391076+0000","last_active":"2023-06-10T18:51:49.381021+0000","last_peered":"2023-06-10T18:51:49.381021+0000","last_clean":"2023-06-10T18:51:49.381021+0000","last_became_active":"2023-06-10T18:51:48.390814+0000","last_became_peered":"2023-06-10T18:51:48.390814+0000","last_unstale":"2023-06-10T18:51:49.381021+0000","last_undegraded":"2023-06-10T18:51:49.381021+0000","last_fullsized":"2023-06-10T18:51:49.381021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:25.986113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383773+0000","last_change":"2023-06-10T18:51:47.992194+0000","last_active":"2023-06-10T18:51:49.383773+0000","last_peered":"2023-06-10T18:51:49.383773+0000","last_clean":"2023-06-10T18:51:49.383773+0000","last_became_active":"2023-06-10T18:51:47.991936+0000","last_became_peered":"2023-06-10T18:51:47.991936+0000","last_unstale":"2023-06-10T18:51:49.383773+0000","last_undegraded":"2023-06-10T18:51:49.383773+0000","last_fullsized":"2023-06-10T18:51:49.383773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:02.195295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624508+0000","last_change":"2023-06-10T18:51:48.397516+0000","last_active":"2023-06-10T18:51:49.624508+0000","last_peered":"2023-06-10T18:51:49.624508+0000","last_clean":"2023-06-10T18:51:49.624508+0000","last_became_active":"2023-06-10T18:51:48.397263+0000","last_became_peered":"2023-06-10T18:51:48.397263+0000","last_unstale":"2023-06-10T18:51:49.624508+0000","last_undegraded":"2023-06-10T18:51:49.624508+0000","last_fullsized":"2023-06-10T18:51:49.624508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:50.234704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383572+0000","last_change":"2023-06-10T18:51:47.988055+0000","last_active":"2023-06-10T18:51:49.383572+0000","last_peered":"2023-06-10T18:51:49.383572+0000","last_clean":"2023-06-10T18:51:49.383572+0000","last_became_active":"2023-06-10T18:51:47.987779+0000","last_became_peered":"2023-06-10T18:51:47.987779+0000","last_unstale":"2023-06-10T18:51:49.383572+0000","last_undegraded":"2023-06-10T18:51:49.383572+0000","last_fullsized":"2023-06-10T18:51:49.383572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:17.219417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381827+0000","last_change":"2023-06-10T18:51:48.391908+0000","last_active":"2023-06-10T18:51:49.381827+0000","last_peered":"2023-06-10T18:51:49.381827+0000","last_clean":"2023-06-10T18:51:49.381827+0000","last_became_active":"2023-06-10T18:51:48.391468+0000","last_became_peered":"2023-06-10T18:51:48.391468+0000","last_unstale":"2023-06-10T18:51:49.381827+0000","last_undegraded":"2023-06-10T18:51:49.381827+0000","last_fullsized":"2023-06-10T18:51:49.381827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:43:51.612691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380919+0000","last_change":"2023-06-10T18:51:47.981335+0000","last_active":"2023-06-10T18:51:49.380919+0000","last_peered":"2023-06-10T18:51:49.380919+0000","last_clean":"2023-06-10T18:51:49.380919+0000","last_became_active":"2023-06-10T18:51:47.981022+0000","last_became_peered":"2023-06-10T18:51:47.981022+0000","last_unstale":"2023-06-10T18:51:49.380919+0000","last_undegraded":"2023-06-10T18:51:49.380919+0000","last_fullsized":"2023-06-10T18:51:49.380919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:34:48.939304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624262+0000","last_change":"2023-06-10T18:51:48.390023+0000","last_active":"2023-06-10T18:51:49.624262+0000","last_peered":"2023-06-10T18:51:49.624262+0000","last_clean":"2023-06-10T18:51:49.624262+0000","last_became_active":"2023-06-10T18:51:48.389749+0000","last_became_peered":"2023-06-10T18:51:48.389749+0000","last_unstale":"2023-06-10T18:51:49.624262+0000","last_undegraded":"2023-06-10T18:51:49.624262+0000","last_fullsized":"2023-06-10T18:51:49.624262+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:28.528107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381036+0000","last_change":"2023-06-10T18:51:47.985509+0000","last_active":"2023-06-10T18:51:49.381036+0000","last_peered":"2023-06-10T18:51:49.381036+0000","last_clean":"2023-06-10T18:51:49.381036+0000","last_became_active":"2023-06-10T18:51:47.985211+0000","last_became_peered":"2023-06-10T18:51:47.985211+0000","last_unstale":"2023-06-10T18:51:49.381036+0000","last_undegraded":"2023-06-10T18:51:49.381036+0000","last_fullsized":"2023-06-10T18:51:49.381036+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:31.385016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377808+0000","last_change":"2023-06-10T18:51:48.392517+0000","last_active":"2023-06-10T18:51:49.377808+0000","last_peered":"2023-06-10T18:51:49.377808+0000","last_clean":"2023-06-10T18:51:49.377808+0000","last_became_active":"2023-06-10T18:51:48.391882+0000","last_became_peered":"2023-06-10T18:51:48.391882+0000","last_unstale":"2023-06-10T18:51:49.377808+0000","last_undegraded":"2023-06-10T18:51:49.377808+0000","last_fullsized":"2023-06-10T18:51:49.377808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:39:45.638924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624402+0000","last_change":"2023-06-10T18:51:47.987992+0000","last_active":"2023-06-10T18:51:49.624402+0000","last_peered":"2023-06-10T18:51:49.624402+0000","last_clean":"2023-06-10T18:51:49.624402+0000","last_became_active":"2023-06-10T18:51:47.987738+0000","last_became_peered":"2023-06-10T18:51:47.987738+0000","last_unstale":"2023-06-10T18:51:49.624402+0000","last_undegraded":"2023-06-10T18:51:49.624402+0000","last_fullsized":"2023-06-10T18:51:49.624402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:08.331733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380499+0000","last_change":"2023-06-10T18:51:48.402300+0000","last_active":"2023-06-10T18:51:49.380499+0000","last_peered":"2023-06-10T18:51:49.380499+0000","last_clean":"2023-06-10T18:51:49.380499+0000","last_became_active":"2023-06-10T18:51:48.402037+0000","last_became_peered":"2023-06-10T18:51:48.402037+0000","last_unstale":"2023-06-10T18:51:49.380499+0000","last_undegraded":"2023-06-10T18:51:49.380499+0000","last_fullsized":"2023-06-10T18:51:49.380499+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:22.801629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380618+0000","last_change":"2023-06-10T18:51:47.992514+0000","last_active":"2023-06-10T18:51:49.380618+0000","last_peered":"2023-06-10T18:51:49.380618+0000","last_clean":"2023-06-10T18:51:49.380618+0000","last_became_active":"2023-06-10T18:51:47.992269+0000","last_became_peered":"2023-06-10T18:51:47.992269+0000","last_unstale":"2023-06-10T18:51:49.380618+0000","last_undegraded":"2023-06-10T18:51:49.380618+0000","last_fullsized":"2023-06-10T18:51:49.380618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:57:49.167698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380566+0000","last_change":"2023-06-10T18:51:48.398094+0000","last_active":"2023-06-10T18:51:49.380566+0000","last_peered":"2023-06-10T18:51:49.380566+0000","last_clean":"2023-06-10T18:51:49.380566+0000","last_became_active":"2023-06-10T18:51:48.397819+0000","last_became_peered":"2023-06-10T18:51:48.397819+0000","last_unstale":"2023-06-10T18:51:49.380566+0000","last_undegraded":"2023-06-10T18:51:49.380566+0000","last_fullsized":"2023-06-10T18:51:49.380566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:16.685695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624252+0000","last_change":"2023-06-10T18:51:47.981834+0000","last_active":"2023-06-10T18:51:49.624252+0000","last_peered":"2023-06-10T18:51:49.624252+0000","last_clean":"2023-06-10T18:51:49.624252+0000","last_became_active":"2023-06-10T18:51:47.981587+0000","last_became_peered":"2023-06-10T18:51:47.981587+0000","last_unstale":"2023-06-10T18:51:49.624252+0000","last_undegraded":"2023-06-10T18:51:49.624252+0000","last_fullsized":"2023-06-10T18:51:49.624252+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:39.669408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382202+0000","last_change":"2023-06-10T18:51:48.382463+0000","last_active":"2023-06-10T18:51:49.382202+0000","last_peered":"2023-06-10T18:51:49.382202+0000","last_clean":"2023-06-10T18:51:49.382202+0000","last_became_active":"2023-06-10T18:51:48.382190+0000","last_became_peered":"2023-06-10T18:51:48.382190+0000","last_unstale":"2023-06-10T18:51:49.382202+0000","last_undegraded":"2023-06-10T18:51:49.382202+0000","last_fullsized":"2023-06-10T18:51:49.382202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:13:02.105582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380959+0000","last_change":"2023-06-10T18:51:47.989294+0000","last_active":"2023-06-10T18:51:49.380959+0000","last_peered":"2023-06-10T18:51:49.380959+0000","last_clean":"2023-06-10T18:51:49.380959+0000","last_became_active":"2023-06-10T18:51:47.989022+0000","last_became_peered":"2023-06-10T18:51:47.989022+0000","last_unstale":"2023-06-10T18:51:49.380959+0000","last_undegraded":"2023-06-10T18:51:49.380959+0000","last_fullsized":"2023-06-10T18:51:49.380959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:08.331733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378105+0000","last_change":"2023-06-10T18:51:48.399670+0000","last_active":"2023-06-10T18:51:49.378105+0000","last_peered":"2023-06-10T18:51:49.378105+0000","last_clean":"2023-06-10T18:51:49.378105+0000","last_became_active":"2023-06-10T18:51:48.399378+0000","last_became_peered":"2023-06-10T18:51:48.399378+0000","last_unstale":"2023-06-10T18:51:49.378105+0000","last_undegraded":"2023-06-10T18:51:49.378105+0000","last_fullsized":"2023-06-10T18:51:49.378105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:32:03.498809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380863+0000","last_change":"2023-06-10T18:51:47.984973+0000","last_active":"2023-06-10T18:51:49.380863+0000","last_peered":"2023-06-10T18:51:49.380863+0000","last_clean":"2023-06-10T18:51:49.380863+0000","last_became_active":"2023-06-10T18:51:47.984697+0000","last_became_peered":"2023-06-10T18:51:47.984697+0000","last_unstale":"2023-06-10T18:51:49.380863+0000","last_undegraded":"2023-06-10T18:51:49.380863+0000","last_fullsized":"2023-06-10T18:51:49.380863+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:02.195295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383309+0000","last_change":"2023-06-10T18:51:48.398836+0000","last_active":"2023-06-10T18:51:49.383309+0000","last_peered":"2023-06-10T18:51:49.383309+0000","last_clean":"2023-06-10T18:51:49.383309+0000","last_became_active":"2023-06-10T18:51:48.398514+0000","last_became_peered":"2023-06-10T18:51:48.398514+0000","last_unstale":"2023-06-10T18:51:49.383309+0000","last_undegraded":"2023-06-10T18:51:49.383309+0000","last_fullsized":"2023-06-10T18:51:49.383309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:11.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383607+0000","last_change":"2023-06-10T18:51:47.985586+0000","last_active":"2023-06-10T18:51:49.383607+0000","last_peered":"2023-06-10T18:51:49.383607+0000","last_clean":"2023-06-10T18:51:49.383607+0000","last_became_active":"2023-06-10T18:51:47.985284+0000","last_became_peered":"2023-06-10T18:51:47.985284+0000","last_unstale":"2023-06-10T18:51:49.383607+0000","last_undegraded":"2023-06-10T18:51:49.383607+0000","last_fullsized":"2023-06-10T18:51:49.383607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:56.914900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382151+0000","last_change":"2023-06-10T18:51:48.391743+0000","last_active":"2023-06-10T18:51:49.382151+0000","last_peered":"2023-06-10T18:51:49.382151+0000","last_clean":"2023-06-10T18:51:49.382151+0000","last_became_active":"2023-06-10T18:51:48.391115+0000","last_became_peered":"2023-06-10T18:51:48.391115+0000","last_unstale":"2023-06-10T18:51:49.382151+0000","last_undegraded":"2023-06-10T18:51:49.382151+0000","last_fullsized":"2023-06-10T18:51:49.382151+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:16:42.405449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383616+0000","last_change":"2023-06-10T18:51:47.984028+0000","last_active":"2023-06-10T18:51:49.383616+0000","last_peered":"2023-06-10T18:51:49.383616+0000","last_clean":"2023-06-10T18:51:49.383616+0000","last_became_active":"2023-06-10T18:51:47.983744+0000","last_became_peered":"2023-06-10T18:51:47.983744+0000","last_unstale":"2023-06-10T18:51:49.383616+0000","last_undegraded":"2023-06-10T18:51:49.383616+0000","last_fullsized":"2023-06-10T18:51:49.383616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:53.720896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380623+0000","last_change":"2023-06-10T18:51:48.385157+0000","last_active":"2023-06-10T18:51:49.380623+0000","last_peered":"2023-06-10T18:51:49.380623+0000","last_clean":"2023-06-10T18:51:49.380623+0000","last_became_active":"2023-06-10T18:51:48.384841+0000","last_became_peered":"2023-06-10T18:51:48.384841+0000","last_unstale":"2023-06-10T18:51:49.380623+0000","last_undegraded":"2023-06-10T18:51:49.380623+0000","last_fullsized":"2023-06-10T18:51:49.380623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:22:49.512567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625209+0000","last_change":"2023-06-10T18:51:47.983812+0000","last_active":"2023-06-10T18:51:49.625209+0000","last_peered":"2023-06-10T18:51:49.625209+0000","last_clean":"2023-06-10T18:51:49.625209+0000","last_became_active":"2023-06-10T18:51:47.983100+0000","last_became_peered":"2023-06-10T18:51:47.983100+0000","last_unstale":"2023-06-10T18:51:49.625209+0000","last_undegraded":"2023-06-10T18:51:49.625209+0000","last_fullsized":"2023-06-10T18:51:49.625209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:45:32.401099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625156+0000","last_change":"2023-06-10T18:51:48.390131+0000","last_active":"2023-06-10T18:51:49.625156+0000","last_peered":"2023-06-10T18:51:49.625156+0000","last_clean":"2023-06-10T18:51:49.625156+0000","last_became_active":"2023-06-10T18:51:48.389863+0000","last_became_peered":"2023-06-10T18:51:48.389863+0000","last_unstale":"2023-06-10T18:51:49.625156+0000","last_undegraded":"2023-06-10T18:51:49.625156+0000","last_fullsized":"2023-06-10T18:51:49.625156+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:15.857938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381766+0000","last_change":"2023-06-10T18:51:47.990817+0000","last_active":"2023-06-10T18:51:49.381766+0000","last_peered":"2023-06-10T18:51:49.381766+0000","last_clean":"2023-06-10T18:51:49.381766+0000","last_became_active":"2023-06-10T18:51:47.990556+0000","last_became_peered":"2023-06-10T18:51:47.990556+0000","last_unstale":"2023-06-10T18:51:49.381766+0000","last_undegraded":"2023-06-10T18:51:49.381766+0000","last_fullsized":"2023-06-10T18:51:49.381766+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:24:29.305759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377661+0000","last_change":"2023-06-10T18:51:48.403094+0000","last_active":"2023-06-10T18:51:49.377661+0000","last_peered":"2023-06-10T18:51:49.377661+0000","last_clean":"2023-06-10T18:51:49.377661+0000","last_became_active":"2023-06-10T18:51:48.402822+0000","last_became_peered":"2023-06-10T18:51:48.402822+0000","last_unstale":"2023-06-10T18:51:49.377661+0000","last_undegraded":"2023-06-10T18:51:49.377661+0000","last_fullsized":"2023-06-10T18:51:49.377661+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:47:32.336237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383392+0000","last_change":"2023-06-10T18:51:47.984261+0000","last_active":"2023-06-10T18:51:49.383392+0000","last_peered":"2023-06-10T18:51:49.383392+0000","last_clean":"2023-06-10T18:51:49.383392+0000","last_became_active":"2023-06-10T18:51:47.983679+0000","last_became_peered":"2023-06-10T18:51:47.983679+0000","last_unstale":"2023-06-10T18:51:49.383392+0000","last_undegraded":"2023-06-10T18:51:49.383392+0000","last_fullsized":"2023-06-10T18:51:49.383392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:33.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383325+0000","last_change":"2023-06-10T18:51:48.397113+0000","last_active":"2023-06-10T18:51:49.383325+0000","last_peered":"2023-06-10T18:51:49.383325+0000","last_clean":"2023-06-10T18:51:49.383325+0000","last_became_active":"2023-06-10T18:51:48.396810+0000","last_became_peered":"2023-06-10T18:51:48.396810+0000","last_unstale":"2023-06-10T18:51:49.383325+0000","last_undegraded":"2023-06-10T18:51:49.383325+0000","last_fullsized":"2023-06-10T18:51:49.383325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:22.801629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381669+0000","last_change":"2023-06-10T18:51:47.991411+0000","last_active":"2023-06-10T18:51:49.381669+0000","last_peered":"2023-06-10T18:51:49.381669+0000","last_clean":"2023-06-10T18:51:49.381669+0000","last_became_active":"2023-06-10T18:51:47.991126+0000","last_became_peered":"2023-06-10T18:51:47.991126+0000","last_unstale":"2023-06-10T18:51:49.381669+0000","last_undegraded":"2023-06-10T18:51:49.381669+0000","last_fullsized":"2023-06-10T18:51:49.381669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:40.430246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380789+0000","last_change":"2023-06-10T18:51:48.391781+0000","last_active":"2023-06-10T18:51:49.380789+0000","last_peered":"2023-06-10T18:51:49.380789+0000","last_clean":"2023-06-10T18:51:49.380789+0000","last_became_active":"2023-06-10T18:51:48.391530+0000","last_became_peered":"2023-06-10T18:51:48.391530+0000","last_unstale":"2023-06-10T18:51:49.380789+0000","last_undegraded":"2023-06-10T18:51:49.380789+0000","last_fullsized":"2023-06-10T18:51:49.380789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:14:22.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624953+0000","last_change":"2023-06-10T18:51:47.982022+0000","last_active":"2023-06-10T18:51:49.624953+0000","last_peered":"2023-06-10T18:51:49.624953+0000","last_clean":"2023-06-10T18:51:49.624953+0000","last_became_active":"2023-06-10T18:51:47.981720+0000","last_became_peered":"2023-06-10T18:51:47.981720+0000","last_unstale":"2023-06-10T18:51:49.624953+0000","last_undegraded":"2023-06-10T18:51:49.624953+0000","last_fullsized":"2023-06-10T18:51:49.624953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:22.267003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380673+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:51:49.380673+0000","last_peered":"2023-06-10T18:51:49.380673+0000","last_clean":"2023-06-10T18:51:49.380673+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:51:49.380673+0000","last_undegraded":"2023-06-10T18:51:49.380673+0000","last_fullsized":"2023-06-10T18:51:49.380673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:58:12.475609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381982+0000","last_change":"2023-06-10T18:51:48.402926+0000","last_active":"2023-06-10T18:51:49.381982+0000","last_peered":"2023-06-10T18:51:49.381982+0000","last_clean":"2023-06-10T18:51:49.381982+0000","last_became_active":"2023-06-10T18:51:48.402639+0000","last_became_peered":"2023-06-10T18:51:48.402639+0000","last_unstale":"2023-06-10T18:51:49.381982+0000","last_undegraded":"2023-06-10T18:51:49.381982+0000","last_fullsized":"2023-06-10T18:51:49.381982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:05.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624622+0000","last_change":"2023-06-10T18:51:47.985853+0000","last_active":"2023-06-10T18:51:49.624622+0000","last_peered":"2023-06-10T18:51:49.624622+0000","last_clean":"2023-06-10T18:51:49.624622+0000","last_became_active":"2023-06-10T18:51:47.985583+0000","last_became_peered":"2023-06-10T18:51:47.985583+0000","last_unstale":"2023-06-10T18:51:49.624622+0000","last_undegraded":"2023-06-10T18:51:49.624622+0000","last_fullsized":"2023-06-10T18:51:49.624622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:26.364949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380657+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:51:49.380657+0000","last_peered":"2023-06-10T18:51:49.380657+0000","last_clean":"2023-06-10T18:51:49.380657+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:51:49.380657+0000","last_undegraded":"2023-06-10T18:51:49.380657+0000","last_fullsized":"2023-06-10T18:51:49.380657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:55:13.554861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378065+0000","last_change":"2023-06-10T18:51:48.393893+0000","last_active":"2023-06-10T18:51:49.378065+0000","last_peered":"2023-06-10T18:51:49.378065+0000","last_clean":"2023-06-10T18:51:49.378065+0000","last_became_active":"2023-06-10T18:51:48.393627+0000","last_became_peered":"2023-06-10T18:51:48.393627+0000","last_unstale":"2023-06-10T18:51:49.378065+0000","last_undegraded":"2023-06-10T18:51:49.378065+0000","last_fullsized":"2023-06-10T18:51:49.378065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:11.871412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383888+0000","last_change":"2023-06-10T18:51:47.986705+0000","last_active":"2023-06-10T18:51:49.383888+0000","last_peered":"2023-06-10T18:51:49.383888+0000","last_clean":"2023-06-10T18:51:49.383888+0000","last_became_active":"2023-06-10T18:51:47.986423+0000","last_became_peered":"2023-06-10T18:51:47.986423+0000","last_unstale":"2023-06-10T18:51:49.383888+0000","last_undegraded":"2023-06-10T18:51:49.383888+0000","last_fullsized":"2023-06-10T18:51:49.383888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:46.541345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624376+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:51:49.624376+0000","last_peered":"2023-06-10T18:51:49.624376+0000","last_clean":"2023-06-10T18:51:49.624376+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:51:49.624376+0000","last_undegraded":"2023-06-10T18:51:49.624376+0000","last_fullsized":"2023-06-10T18:51:49.624376+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:48:32.852209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380727+0000","last_change":"2023-06-10T18:51:48.394754+0000","last_active":"2023-06-10T18:51:49.380727+0000","last_peered":"2023-06-10T18:51:49.380727+0000","last_clean":"2023-06-10T18:51:49.380727+0000","last_became_active":"2023-06-10T18:51:48.394550+0000","last_became_peered":"2023-06-10T18:51:48.394550+0000","last_unstale":"2023-06-10T18:51:49.380727+0000","last_undegraded":"2023-06-10T18:51:49.380727+0000","last_fullsized":"2023-06-10T18:51:49.380727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:18.225092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377973+0000","last_change":"2023-06-10T18:51:47.989368+0000","last_active":"2023-06-10T18:51:49.377973+0000","last_peered":"2023-06-10T18:51:49.377973+0000","last_clean":"2023-06-10T18:51:49.377973+0000","last_became_active":"2023-06-10T18:51:47.989082+0000","last_became_peered":"2023-06-10T18:51:47.989082+0000","last_unstale":"2023-06-10T18:51:49.377973+0000","last_undegraded":"2023-06-10T18:51:49.377973+0000","last_fullsized":"2023-06-10T18:51:49.377973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:16.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380563+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:51:49.380563+0000","last_peered":"2023-06-10T18:51:49.380563+0000","last_clean":"2023-06-10T18:51:49.380563+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:51:49.380563+0000","last_undegraded":"2023-06-10T18:51:49.380563+0000","last_fullsized":"2023-06-10T18:51:49.380563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:39.628192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":284,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377748+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:51:49.377748+0000","last_peered":"2023-06-10T18:51:49.377748+0000","last_clean":"2023-06-10T18:51:49.377748+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:51:49.377748+0000","last_undegraded":"2023-06-10T18:51:49.377748+0000","last_fullsized":"2023-06-10T18:51:49.377748+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:41:01.529544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.384060+0000","last_change":"2023-06-10T18:51:48.399675+0000","last_active":"2023-06-10T18:51:49.384060+0000","last_peered":"2023-06-10T18:51:49.384060+0000","last_clean":"2023-06-10T18:51:49.384060+0000","last_became_active":"2023-06-10T18:51:48.399176+0000","last_became_peered":"2023-06-10T18:51:48.399176+0000","last_unstale":"2023-06-10T18:51:49.384060+0000","last_undegraded":"2023-06-10T18:51:49.384060+0000","last_fullsized":"2023-06-10T18:51:49.384060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:40.997736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625482+0000","last_change":"2023-06-10T18:51:47.992465+0000","last_active":"2023-06-10T18:51:49.625482+0000","last_peered":"2023-06-10T18:51:49.625482+0000","last_clean":"2023-06-10T18:51:49.625482+0000","last_became_active":"2023-06-10T18:51:47.992180+0000","last_became_peered":"2023-06-10T18:51:47.992180+0000","last_unstale":"2023-06-10T18:51:49.625482+0000","last_undegraded":"2023-06-10T18:51:49.625482+0000","last_fullsized":"2023-06-10T18:51:49.625482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:15.680020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377821+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:51:49.377821+0000","last_peered":"2023-06-10T18:51:49.377821+0000","last_clean":"2023-06-10T18:51:49.377821+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:51:49.377821+0000","last_undegraded":"2023-06-10T18:51:49.377821+0000","last_fullsized":"2023-06-10T18:51:49.377821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:06:40.909431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624779+0000","last_change":"2023-06-10T18:51:48.403247+0000","last_active":"2023-06-10T18:51:49.624779+0000","last_peered":"2023-06-10T18:51:49.624779+0000","last_clean":"2023-06-10T18:51:49.624779+0000","last_became_active":"2023-06-10T18:51:48.402885+0000","last_became_peered":"2023-06-10T18:51:48.402885+0000","last_unstale":"2023-06-10T18:51:49.624779+0000","last_undegraded":"2023-06-10T18:51:49.624779+0000","last_fullsized":"2023-06-10T18:51:49.624779+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:52:20.034259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623976+0000","last_change":"2023-06-10T18:51:47.980733+0000","last_active":"2023-06-10T18:51:49.623976+0000","last_peered":"2023-06-10T18:51:49.623976+0000","last_clean":"2023-06-10T18:51:49.623976+0000","last_became_active":"2023-06-10T18:51:47.980388+0000","last_became_peered":"2023-06-10T18:51:47.980388+0000","last_unstale":"2023-06-10T18:51:49.623976+0000","last_undegraded":"2023-06-10T18:51:49.623976+0000","last_fullsized":"2023-06-10T18:51:49.623976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:55.969246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378062+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:51:49.378062+0000","last_peered":"2023-06-10T18:51:49.378062+0000","last_clean":"2023-06-10T18:51:49.378062+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:51:49.378062+0000","last_undegraded":"2023-06-10T18:51:49.378062+0000","last_fullsized":"2023-06-10T18:51:49.378062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:13.460094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380652+0000","last_change":"2023-06-10T18:51:48.388213+0000","last_active":"2023-06-10T18:51:49.380652+0000","last_peered":"2023-06-10T18:51:49.380652+0000","last_clean":"2023-06-10T18:51:49.380652+0000","last_became_active":"2023-06-10T18:51:48.387943+0000","last_became_peered":"2023-06-10T18:51:48.387943+0000","last_unstale":"2023-06-10T18:51:49.380652+0000","last_undegraded":"2023-06-10T18:51:49.380652+0000","last_fullsized":"2023-06-10T18:51:49.380652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:57.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623711+0000","last_change":"2023-06-10T18:51:47.980408+0000","last_active":"2023-06-10T18:51:49.623711+0000","last_peered":"2023-06-10T18:51:49.623711+0000","last_clean":"2023-06-10T18:51:49.623711+0000","last_became_active":"2023-06-10T18:51:47.980134+0000","last_became_peered":"2023-06-10T18:51:47.980134+0000","last_unstale":"2023-06-10T18:51:49.623711+0000","last_undegraded":"2023-06-10T18:51:49.623711+0000","last_fullsized":"2023-06-10T18:51:49.623711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:53.720896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380857+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:51:49.380857+0000","last_peered":"2023-06-10T18:51:49.380857+0000","last_clean":"2023-06-10T18:51:49.380857+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:51:49.380857+0000","last_undegraded":"2023-06-10T18:51:49.380857+0000","last_fullsized":"2023-06-10T18:51:49.380857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:55:11.529955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624234+0000","last_change":"2023-06-10T18:51:48.393080+0000","last_active":"2023-06-10T18:51:49.624234+0000","last_peered":"2023-06-10T18:51:49.624234+0000","last_clean":"2023-06-10T18:51:49.624234+0000","last_became_active":"2023-06-10T18:51:48.392801+0000","last_became_peered":"2023-06-10T18:51:48.392801+0000","last_unstale":"2023-06-10T18:51:49.624234+0000","last_undegraded":"2023-06-10T18:51:49.624234+0000","last_fullsized":"2023-06-10T18:51:49.624234+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:06.319540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624158+0000","last_change":"2023-06-10T18:51:47.981311+0000","last_active":"2023-06-10T18:51:49.624158+0000","last_peered":"2023-06-10T18:51:49.624158+0000","last_clean":"2023-06-10T18:51:49.624158+0000","last_became_active":"2023-06-10T18:51:47.980879+0000","last_became_peered":"2023-06-10T18:51:47.980879+0000","last_unstale":"2023-06-10T18:51:49.624158+0000","last_undegraded":"2023-06-10T18:51:49.624158+0000","last_fullsized":"2023-06-10T18:51:49.624158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:33.095897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":120,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383735+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:51:49.383735+0000","last_peered":"2023-06-10T18:51:49.383735+0000","last_clean":"2023-06-10T18:51:49.383735+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:51:49.383735+0000","last_undegraded":"2023-06-10T18:51:49.383735+0000","last_fullsized":"2023-06-10T18:51:49.383735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:55:11.529955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28528,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343312,"statfs":{"total":96636764160,"available":96607551488,"internally_reserved":0,"allocated":1613824,"data_stored":1073447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.53900000000000003},"min":{"1min":0.39000000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.53000000000000003},"min":{"1min":0.48199999999999998,"5min":0.34899999999999998,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.88100000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.38,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60899999999999999,"15min":0.57099999999999995},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.34799999999999998},"max":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.81200000000000006},"last":0.80600000000000005}]},{"osd":2,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55500000000000005,"15min":0.498},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.42199999999999999,"5min":0.33900000000000002,"15min":0.27300000000000002},"max":{"1min":0.77900000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67800000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56899999999999995,"15min":0.51600000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.55800000000000005},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.32700000000000001},"max":{"1min":0.80100000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.78100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.35099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.42899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68899999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.83799999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69899999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.754}]},{"osd":6,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.40300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.32600000000000001},"max":{"1min":0.72599999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.71599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":1024000,"data_stored":482684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.52200000000000002},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.81100000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55000000000000004,"15min":0.50800000000000001},"min":{"1min":0.47299999999999998,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.69499999999999995,"5min":0.746,"15min":0.78700000000000003},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.52400000000000002},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.67100000000000004,"5min":0.752,"15min":0.80600000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55900000000000005,"15min":0.80500000000000005},"min":{"1min":0.38500000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.71699999999999997},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.442,"5min":0.374,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.76300000000000001,"15min":1.252},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.46200000000000002,"5min":0.371,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.745,"15min":0.82899999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69099999999999995,"5min":0.746,"15min":0.77500000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46600000000000003,"15min":0.51000000000000001},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.59699999999999998,"5min":0.80700000000000005,"15min":0.82999999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68500000000000005,"5min":0.78600000000000003,"15min":0.89600000000000002},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46600000000000003,"15min":0.50900000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28799999999999998},"max":{"1min":0.54800000000000004,"5min":0.72699999999999998,"15min":0.86699999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.504,"15min":0.54500000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.628,"5min":0.76700000000000002,"15min":0.79100000000000004},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64000000000000001,"5min":0.78000000000000003,"15min":0.92700000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.79700000000000004,"15min":0.81399999999999995},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":1024000,"data_stored":482184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35199999999999998,"15min":0.34100000000000003},"min":{"1min":0.27200000000000002,"5min":0.221,"15min":0.19},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.753},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.59599999999999997},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36599999999999999,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.215},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.38700000000000001},"min":{"1min":0.313,"5min":0.215,"15min":0.215},"max":{"1min":0.59199999999999997,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.24299999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.40799999999999997},"min":{"1min":0.34499999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.42499999999999999,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.501,"5min":0.378,"15min":0.378},"max":{"1min":0.753,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.441},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.53800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.38200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.52500000000000002,"5min":0.36499999999999999,"15min":0.33500000000000002},"max":{"1min":0.78000000000000003,"5min":0.89900000000000002,"15min":0.93000000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.53400000000000003,"5min":0.42199999999999999,"15min":0.39500000000000002},"max":{"1min":0.83299999999999996,"5min":0.871,"15min":1.008},"last":0.46600000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60399999999999998,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.41999999999999998,"15min":0.40000000000000002},"max":{"1min":0.79500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.41399999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.56699999999999995,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76800000000000002,"5min":0.85899999999999999,"15min":0.95099999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.439,"5min":0.434,"15min":0.434},"max":{"1min":0.85499999999999998,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.48099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574947,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1609728,"data_stored":1072392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34899999999999998},"min":{"1min":0.26200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.46500000000000002,"5min":0.625,"15min":0.72699999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.36699999999999999,"15min":0.35999999999999999},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.186},"max":{"1min":0.45200000000000001,"5min":0.56599999999999995,"15min":0.88900000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.38100000000000001},"min":{"1min":0.29999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.65500000000000003,"15min":1.405},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":0.496,"5min":0.51500000000000001,"15min":0.95499999999999996},"last":0.26900000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38300000000000001,"15min":0.378},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.439,"5min":0.70999999999999996,"15min":0.93100000000000005},"last":0.249},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42399999999999999,"15min":0.40400000000000003},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.25900000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.88100000000000001,"15min":1.0489999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.53700000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68100000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.503,"5min":0.39000000000000001,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.90800000000000003,"15min":0.999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.44600000000000001,"5min":0.38300000000000001,"15min":0.35399999999999998},"max":{"1min":0.71199999999999997,"5min":0.88400000000000001,"15min":0.98199999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.498,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":1.363},"last":0.434}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.48099999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80700000000000005,"5min":0.93500000000000005,"15min":1.034},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.38900000000000001},"max":{"1min":0.76400000000000001,"5min":0.86099999999999999,"15min":1.0149999999999999},"last":0.50700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":1019904,"data_stored":483591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35399999999999998,"15min":0.33700000000000002},"min":{"1min":0.26900000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.52600000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.36199999999999999},"min":{"1min":0.30299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55700000000000005,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.251}]},{"osd":1,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39100000000000001,"15min":0.35999999999999999},"min":{"1min":0.32800000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.36599999999999999},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.436,"15min":0.39500000000000002},"min":{"1min":0.38400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57999999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64200000000000002,"15min":0.59099999999999997},"min":{"1min":0.51400000000000001,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":0.77000000000000002,"5min":5.452,"15min":5.452},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.47099999999999997,"5min":0.38,"15min":0.373},"max":{"1min":0.89800000000000002,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90900000000000003,"5min":1.194,"15min":1.194},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65400000000000003,"15min":0.628},"min":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.38400000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54900000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.60999999999999999},"min":{"1min":0.49399999999999999,"5min":0.36499999999999999,"15min":0.34599999999999997},"max":{"1min":0.88400000000000001,"5min":5.4640000000000004,"15min":5.4640000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.60899999999999999},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":0.875,"15min":0.92600000000000005},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.64000000000000001},"min":{"1min":0.503,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92600000000000005,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65600000000000003,"15min":0.62},"min":{"1min":0.57799999999999996,"5min":0.38300000000000001,"15min":0.37},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574947,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":1028096,"data_stored":484573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.373},"min":{"1min":0.26000000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.36599999999999999,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.219},"max":{"1min":0.505,"5min":0.56999999999999995,"15min":1.8600000000000001},"last":0.307}]},{"osd":1,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.371,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":0.99299999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.42799999999999999},"min":{"1min":0.309,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53800000000000003,"5min":0.61499999999999999,"15min":1.3},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.41899999999999998},"min":{"1min":0.32600000000000001,"5min":0.28100000000000003,"15min":0.26900000000000002},"max":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":1.109},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.46899999999999997},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":1.599},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.59699999999999998,"15min":0.623},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.378},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.925},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":1.0860000000000001},"last":0.38400000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61099999999999999,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.40699999999999997,"15min":0.40600000000000003},"max":{"1min":0.86899999999999999,"5min":0.90100000000000002,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61199999999999999,"15min":0.72699999999999998},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80400000000000005,"5min":0.97599999999999998,"15min":3.802},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.60799999999999998,"15min":0.72599999999999998},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":3.8180000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.03},"last":0.40200000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":1.2769999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.627,"15min":0.64700000000000002},"min":{"1min":0.48199999999999998,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":1.8939999999999999},"last":0.498}]}]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":1011712,"data_stored":481990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54400000000000004,"15min":0.51500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28299999999999997,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82599999999999996,"15min":1.119},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.74099999999999999,"15min":1.1639999999999999},"last":0.54700000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55500000000000005,"15min":0.51000000000000001},"min":{"1min":0.377,"5min":0.36499999999999999,"15min":0.27100000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.77800000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56399999999999995,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.51700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68500000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74399999999999999,"5min":0.80100000000000005,"15min":1.0229999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54700000000000004,"15min":0.52600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.79800000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":1.042},"last":0.41699999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46999999999999997,"15min":0.495},"min":{"1min":0.35799999999999998,"5min":0.29799999999999999,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":0.752,"15min":1.069},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71999999999999997,"5min":0.84099999999999997,"15min":0.88300000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48499999999999999,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":1.1910000000000001},"last":2.1579999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.32100000000000001},"max":{"1min":0.77000000000000002,"5min":0.78000000000000003,"15min":1.3169999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.78100000000000003,"5min":0.81499999999999995,"15min":0.85899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.81599999999999995,"15min":1.1459999999999999},"last":0.69999999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":1032192,"data_stored":484595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.46600000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.72099999999999997,"5min":0.752,"15min":0.76800000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.505},"min":{"1min":0.437,"5min":0.34399999999999997,"15min":0.32300000000000001},"max":{"1min":0.70899999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.51600000000000001},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.68300000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.52100000000000002},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53100000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54500000000000004,"15min":0.51600000000000001},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.85099999999999998},"last":0.495},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.53100000000000003},"min":{"1min":0.45700000000000002,"5min":0.38500000000000001,"15min":0.34999999999999998},"max":{"1min":0.68300000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.32400000000000001},"max":{"1min":0.69299999999999995,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57999999999999996,"15min":0.55100000000000005},"min":{"1min":0.52000000000000002,"5min":0.38800000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.76800000000000002},"last":0.435}]},{"osd":4,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.39100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.879,"15min":0.879},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38,"15min":0.30299999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:51:52.469 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:51:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.615+0000 7f6b6c733700 1 Processor -- start 2023-06-10T18:51:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.615+0000 7f6b6c733700 1 -- start start 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6c733700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b6415a420 0x7f6b640b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6c733700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b640cea60 0x7f6b6415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6c733700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b640b6070 con 0x7f6b640b4ff0 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6c733700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640b5460 con 0x7f6b6415a420 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6c733700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640b5740 con 0x7f6b640cea60 2023-06-10T18:51:52.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6a4cf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b6415a420 0x7f6b640b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b69cce700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6b640b4ff0 0x7f6b640ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58728/0 (socket says 172.21.15.104:58728) 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b6a4cf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6b6415a420 0x7f6b640b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60898/0 (socket says 172.21.15.104:60898) 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.616+0000 7f6b69cce700 1 -- 172.21.15.104:0/1557365388 learned_addr learned my addr 172.21.15.104:0/1557365388 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6acd0700 1 --2- 172.21.15.104:0/1557365388 >> v2:172.21.15.114:3300/0 conn(0x7f6b640cea60 0x7f6b6415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6a4cf700 1 -- 172.21.15.104:0/1557365388 >> v2:172.21.15.114:3300/0 conn(0x7f6b640cea60 msgr2=0x7f6b6415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6a4cf700 1 --2- 172.21.15.104:0/1557365388 >> v2:172.21.15.114:3300/0 conn(0x7f6b640cea60 0x7f6b6415a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6a4cf700 1 -- 172.21.15.104:0/1557365388 >> v1:172.21.15.104:6789/0 conn(0x7f6b640b4ff0 legacy=0x7f6b640ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6a4cf700 1 -- 172.21.15.104:0/1557365388 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b640b5a20 con 0x7f6b6415a420 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b694cd700 1 -- 172.21.15.104:0/1557365388 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540781145 0 0) 0x7f6b640b6070 con 0x7f6b640b4ff0 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6a4cf700 1 --2- 172.21.15.104:0/1557365388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6415a420 0x7f6b640b1660 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6b640cad60 tx=0x7f6b54009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa682e8a91fd1d0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b694cd700 1 -- 172.21.15.104:0/1557365388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54011830 con 0x7f6b6415a420 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b694cd700 1 -- 172.21.15.104:0/1557365388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54011990 con 0x7f6b6415a420 2023-06-10T18:51:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b694cd700 1 -- 172.21.15.104:0/1557365388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54011d20 con 0x7f6b6415a420 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6c733700 1 -- 172.21.15.104:0/1557365388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6415a420 msgr2=0x7f6b640b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.617+0000 7f6b6c733700 1 --2- 172.21.15.104:0/1557365388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6415a420 0x7f6b640b1660 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6b640cad60 tx=0x7f6b54009b20 comp rx=0 tx=0).stop 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 -- 172.21.15.104:0/1557365388 shutdown_connections 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 --2- 172.21.15.104:0/1557365388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b6415a420 0x7f6b640b1660 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 --2- 172.21.15.104:0/1557365388 >> v2:172.21.15.114:3300/0 conn(0x7f6b640cea60 0x7f6b6415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 -- 172.21.15.104:0/1557365388 >> 172.21.15.104:0/1557365388 conn(0x7f6b64089fd0 msgr2=0x7f6b640d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 -- 172.21.15.104:0/1557365388 shutdown_connections 2023-06-10T18:51:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.618+0000 7f6b6c733700 1 -- 172.21.15.104:0/1557365388 wait complete. 2023-06-10T18:51:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.619+0000 7f6b6c733700 1 Processor -- start 2023-06-10T18:51:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.619+0000 7f6b6c733700 1 -- start start 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b640b4ff0 0x7f6b641e2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 0x7f6b641efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641ed800 con 0x7f6b640cea60 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641f02a0 con 0x7f6b6415a420 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6c733700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641f0500 con 0x7f6b640b4ff0 2023-06-10T18:51:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60906/0 (socket says 172.21.15.104:60906) 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 -- 172.21.15.104:0/2189418879 learned_addr learned my addr 172.21.15.104:0/2189418879 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6acd0700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 0x7f6b641efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6a4cf700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b640b4ff0 0x7f6b641e2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 -- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b640b4ff0 msgr2=0x7f6b641e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b640b4ff0 0x7f6b641e2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 -- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 msgr2=0x7f6b641efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 0x7f6b641efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b69cce700 1 -- 172.21.15.104:0/2189418879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540094a0 con 0x7f6b640cea60 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.620+0000 7f6b6acd0700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 0x7f6b641efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b69cce700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f6b600011e0 tx=0x7f6b6000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60004c80 con 0x7f6b640cea60 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60004e00 con 0x7f6b640cea60 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b641f07e0 con 0x7f6b640cea60 2023-06-10T18:51:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60007310 con 0x7f6b640cea60 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.621+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b640cc150 con 0x7f6b640cea60 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.622+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b641f0d00 con 0x7f6b640cea60 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.622+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b60007470 con 0x7f6b640cea60 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.622+0000 7f6b5affd700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.623+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f6b60089360 con 0x7f6b640cea60 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.623+0000 7f6b6a4cf700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.623+0000 7f6b6a4cf700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6b540090b0 tx=0x7f6b54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.627+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6b641f0d00 con 0x7f6b640cea60 2023-06-10T18:51:52.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.806+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b640b5d00 con 0x7f6b4806ace0 2023-06-10T18:51:52.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.815+0000 7f6b5affd700 1 -- 172.21.15.104:0/2189418879 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311925 (secure 0 0 0) 0x7f6b54011b00 con 0x7f6b4806ace0 2023-06-10T18:51:52.826 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:52.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 msgr2=0x7f6b4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6b540090b0 tx=0x7f6b54008030 comp rx=0 tx=0).stop 2023-06-10T18:51:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 msgr2=0x7f6b641ed2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f6b600011e0 tx=0x7f6b6000cae0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 shutdown_connections 2023-06-10T18:51:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6b4806ace0 0x7f6b4806d190 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6415a420 0x7f6b641efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b640b4ff0 0x7f6b641e2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 --2- 172.21.15.104:0/2189418879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b640cea60 0x7f6b641ed2c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 >> 172.21.15.104:0/2189418879 conn(0x7f6b64089fd0 msgr2=0x7f6b6406fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:52.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 shutdown_connections 2023-06-10T18:51:52.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.819+0000 7f6b6c733700 1 -- 172.21.15.104:0/2189418879 wait complete. 2023-06-10T18:51:52.831 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:51:52.836 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-06-10T18:51:52.396168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224760,"kb_used_data":9144,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754749960,"statfs":{"total":773094113280,"available":772863959040,"internally_reserved":0,"allocated":9363456,"data_stored":5045456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.923073"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383798+0000","last_change":"2023-06-10T18:51:48.399525+0000","last_active":"2023-06-10T18:51:49.383798+0000","last_peered":"2023-06-10T18:51:49.383798+0000","last_clean":"2023-06-10T18:51:49.383798+0000","last_became_active":"2023-06-10T18:51:48.399164+0000","last_became_peered":"2023-06-10T18:51:48.399164+0000","last_unstale":"2023-06-10T18:51:49.383798+0000","last_undegraded":"2023-06-10T18:51:49.383798+0000","last_fullsized":"2023-06-10T18:51:49.383798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:09.337408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380763+0000","last_change":"2023-06-10T18:51:48.399833+0000","last_active":"2023-06-10T18:51:49.380763+0000","last_peered":"2023-06-10T18:51:49.380763+0000","last_clean":"2023-06-10T18:51:49.380763+0000","last_became_active":"2023-06-10T18:51:48.399578+0000","last_became_peered":"2023-06-10T18:51:48.399578+0000","last_unstale":"2023-06-10T18:51:49.380763+0000","last_undegraded":"2023-06-10T18:51:49.380763+0000","last_fullsized":"2023-06-10T18:51:49.380763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:54.726570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377812+0000","last_change":"2023-06-10T18:51:48.395619+0000","last_active":"2023-06-10T18:51:49.377812+0000","last_peered":"2023-06-10T18:51:49.377812+0000","last_clean":"2023-06-10T18:51:49.377812+0000","last_became_active":"2023-06-10T18:51:48.395361+0000","last_became_peered":"2023-06-10T18:51:48.395361+0000","last_unstale":"2023-06-10T18:51:49.377812+0000","last_undegraded":"2023-06-10T18:51:49.377812+0000","last_fullsized":"2023-06-10T18:51:49.377812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:12:14.847162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624696+0000","last_change":"2023-06-10T18:51:48.403461+0000","last_active":"2023-06-10T18:51:49.624696+0000","last_peered":"2023-06-10T18:51:49.624696+0000","last_clean":"2023-06-10T18:51:49.624696+0000","last_became_active":"2023-06-10T18:51:48.403184+0000","last_became_peered":"2023-06-10T18:51:48.403184+0000","last_unstale":"2023-06-10T18:51:49.624696+0000","last_undegraded":"2023-06-10T18:51:49.624696+0000","last_fullsized":"2023-06-10T18:51:49.624696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:44:35.982855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381873+0000","last_change":"2023-06-10T18:51:48.403100+0000","last_active":"2023-06-10T18:51:49.381873+0000","last_peered":"2023-06-10T18:51:49.381873+0000","last_clean":"2023-06-10T18:51:49.381873+0000","last_became_active":"2023-06-10T18:51:48.402775+0000","last_became_peered":"2023-06-10T18:51:48.402775+0000","last_unstale":"2023-06-10T18:51:49.381873+0000","last_undegraded":"2023-06-10T18:51:49.381873+0000","last_fullsized":"2023-06-10T18:51:49.381873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:10.156603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380538+0000","last_change":"2023-06-10T18:51:48.392649+0000","last_active":"2023-06-10T18:51:49.380538+0000","last_peered":"2023-06-10T18:51:49.380538+0000","last_clean":"2023-06-10T18:51:49.380538+0000","last_became_active":"2023-06-10T18:51:48.392384+0000","last_became_peered":"2023-06-10T18:51:48.392384+0000","last_unstale":"2023-06-10T18:51:49.380538+0000","last_undegraded":"2023-06-10T18:51:49.380538+0000","last_fullsized":"2023-06-10T18:51:49.380538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:16.961282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624303+0000","last_change":"2023-06-10T18:51:48.391101+0000","last_active":"2023-06-10T18:51:49.624303+0000","last_peered":"2023-06-10T18:51:49.624303+0000","last_clean":"2023-06-10T18:51:49.624303+0000","last_became_active":"2023-06-10T18:51:48.390794+0000","last_became_peered":"2023-06-10T18:51:48.390794+0000","last_unstale":"2023-06-10T18:51:49.624303+0000","last_undegraded":"2023-06-10T18:51:49.624303+0000","last_fullsized":"2023-06-10T18:51:49.624303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:27.376706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380490+0000","last_change":"2023-06-10T18:51:48.394711+0000","last_active":"2023-06-10T18:51:49.380490+0000","last_peered":"2023-06-10T18:51:49.380490+0000","last_clean":"2023-06-10T18:51:49.380490+0000","last_became_active":"2023-06-10T18:51:48.394352+0000","last_became_peered":"2023-06-10T18:51:48.394352+0000","last_unstale":"2023-06-10T18:51:49.380490+0000","last_undegraded":"2023-06-10T18:51:49.380490+0000","last_fullsized":"2023-06-10T18:51:49.380490+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:58:13.773736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380930+0000","last_change":"2023-06-10T18:51:48.388605+0000","last_active":"2023-06-10T18:51:49.380930+0000","last_peered":"2023-06-10T18:51:49.380930+0000","last_clean":"2023-06-10T18:51:49.380930+0000","last_became_active":"2023-06-10T18:51:48.388337+0000","last_became_peered":"2023-06-10T18:51:48.388337+0000","last_unstale":"2023-06-10T18:51:49.380930+0000","last_undegraded":"2023-06-10T18:51:49.380930+0000","last_fullsized":"2023-06-10T18:51:49.380930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:11:27.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383731+0000","last_change":"2023-06-10T18:51:48.398604+0000","last_active":"2023-06-10T18:51:49.383731+0000","last_peered":"2023-06-10T18:51:49.383731+0000","last_clean":"2023-06-10T18:51:49.383731+0000","last_became_active":"2023-06-10T18:51:48.398187+0000","last_became_peered":"2023-06-10T18:51:48.398187+0000","last_unstale":"2023-06-10T18:51:49.383731+0000","last_undegraded":"2023-06-10T18:51:49.383731+0000","last_fullsized":"2023-06-10T18:51:49.383731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380842+0000","last_change":"2023-06-10T18:51:48.401796+0000","last_active":"2023-06-10T18:51:49.380842+0000","last_peered":"2023-06-10T18:51:49.380842+0000","last_clean":"2023-06-10T18:51:49.380842+0000","last_became_active":"2023-06-10T18:51:48.401574+0000","last_became_peered":"2023-06-10T18:51:48.401574+0000","last_unstale":"2023-06-10T18:51:49.380842+0000","last_undegraded":"2023-06-10T18:51:49.380842+0000","last_fullsized":"2023-06-10T18:51:49.380842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383304+0000","last_change":"2023-06-10T18:51:48.399880+0000","last_active":"2023-06-10T18:51:49.383304+0000","last_peered":"2023-06-10T18:51:49.383304+0000","last_clean":"2023-06-10T18:51:49.383304+0000","last_became_active":"2023-06-10T18:51:48.399574+0000","last_became_peered":"2023-06-10T18:51:48.399574+0000","last_unstale":"2023-06-10T18:51:49.383304+0000","last_undegraded":"2023-06-10T18:51:49.383304+0000","last_fullsized":"2023-06-10T18:51:49.383304+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:54:58.346559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377767+0000","last_change":"2023-06-10T18:51:48.393189+0000","last_active":"2023-06-10T18:51:49.377767+0000","last_peered":"2023-06-10T18:51:49.377767+0000","last_clean":"2023-06-10T18:51:49.377767+0000","last_became_active":"2023-06-10T18:51:48.391396+0000","last_became_peered":"2023-06-10T18:51:48.391396+0000","last_unstale":"2023-06-10T18:51:49.377767+0000","last_undegraded":"2023-06-10T18:51:49.377767+0000","last_fullsized":"2023-06-10T18:51:49.377767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:15:15.817356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624648+0000","last_change":"2023-06-10T18:51:48.403589+0000","last_active":"2023-06-10T18:51:49.624648+0000","last_peered":"2023-06-10T18:51:49.624648+0000","last_clean":"2023-06-10T18:51:49.624648+0000","last_became_active":"2023-06-10T18:51:48.403352+0000","last_became_peered":"2023-06-10T18:51:48.403352+0000","last_unstale":"2023-06-10T18:51:49.624648+0000","last_undegraded":"2023-06-10T18:51:49.624648+0000","last_fullsized":"2023-06-10T18:51:49.624648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:36:55.439108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623832+0000","last_change":"2023-06-10T18:51:48.390644+0000","last_active":"2023-06-10T18:51:49.623832+0000","last_peered":"2023-06-10T18:51:49.623832+0000","last_clean":"2023-06-10T18:51:49.623832+0000","last_became_active":"2023-06-10T18:51:48.390409+0000","last_became_peered":"2023-06-10T18:51:48.390409+0000","last_unstale":"2023-06-10T18:51:49.623832+0000","last_undegraded":"2023-06-10T18:51:49.623832+0000","last_fullsized":"2023-06-10T18:51:49.623832+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:15.169524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381807+0000","last_change":"2023-06-10T18:51:48.397500+0000","last_active":"2023-06-10T18:51:49.381807+0000","last_peered":"2023-06-10T18:51:49.381807+0000","last_clean":"2023-06-10T18:51:49.381807+0000","last_became_active":"2023-06-10T18:51:48.397179+0000","last_became_peered":"2023-06-10T18:51:48.397179+0000","last_unstale":"2023-06-10T18:51:49.381807+0000","last_undegraded":"2023-06-10T18:51:49.381807+0000","last_fullsized":"2023-06-10T18:51:49.381807+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:30:44.869672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382168+0000","last_change":"2023-06-10T18:51:48.388377+0000","last_active":"2023-06-10T18:51:49.382168+0000","last_peered":"2023-06-10T18:51:49.382168+0000","last_clean":"2023-06-10T18:51:49.382168+0000","last_became_active":"2023-06-10T18:51:48.386618+0000","last_became_peered":"2023-06-10T18:51:48.386618+0000","last_unstale":"2023-06-10T18:51:49.382168+0000","last_undegraded":"2023-06-10T18:51:49.382168+0000","last_fullsized":"2023-06-10T18:51:49.382168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:51.791581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625256+0000","last_change":"2023-06-10T18:51:48.399189+0000","last_active":"2023-06-10T18:51:49.625256+0000","last_peered":"2023-06-10T18:51:49.625256+0000","last_clean":"2023-06-10T18:51:49.625256+0000","last_became_active":"2023-06-10T18:51:48.398915+0000","last_became_peered":"2023-06-10T18:51:48.398915+0000","last_unstale":"2023-06-10T18:51:49.625256+0000","last_undegraded":"2023-06-10T18:51:49.625256+0000","last_fullsized":"2023-06-10T18:51:49.625256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:09.165115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625138+0000","last_change":"2023-06-10T18:51:48.391744+0000","last_active":"2023-06-10T18:51:49.625138+0000","last_peered":"2023-06-10T18:51:49.625138+0000","last_clean":"2023-06-10T18:51:49.625138+0000","last_became_active":"2023-06-10T18:51:48.391495+0000","last_became_peered":"2023-06-10T18:51:48.391495+0000","last_unstale":"2023-06-10T18:51:49.625138+0000","last_undegraded":"2023-06-10T18:51:49.625138+0000","last_fullsized":"2023-06-10T18:51:49.625138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:59:51.205176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624891+0000","last_change":"2023-06-10T18:51:48.394708+0000","last_active":"2023-06-10T18:51:49.624891+0000","last_peered":"2023-06-10T18:51:49.624891+0000","last_clean":"2023-06-10T18:51:49.624891+0000","last_became_active":"2023-06-10T18:51:48.394442+0000","last_became_peered":"2023-06-10T18:51:48.394442+0000","last_unstale":"2023-06-10T18:51:49.624891+0000","last_undegraded":"2023-06-10T18:51:49.624891+0000","last_fullsized":"2023-06-10T18:51:49.624891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:19:39.745961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624955+0000","last_change":"2023-06-10T18:51:48.396079+0000","last_active":"2023-06-10T18:51:49.624955+0000","last_peered":"2023-06-10T18:51:49.624955+0000","last_clean":"2023-06-10T18:51:49.624955+0000","last_became_active":"2023-06-10T18:51:48.393682+0000","last_became_peered":"2023-06-10T18:51:48.393682+0000","last_unstale":"2023-06-10T18:51:49.624955+0000","last_undegraded":"2023-06-10T18:51:49.624955+0000","last_fullsized":"2023-06-10T18:51:49.624955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:29:31.472028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380554+0000","last_change":"2023-06-10T18:51:48.401012+0000","last_active":"2023-06-10T18:51:49.380554+0000","last_peered":"2023-06-10T18:51:49.380554+0000","last_clean":"2023-06-10T18:51:49.380554+0000","last_became_active":"2023-06-10T18:51:48.400756+0000","last_became_peered":"2023-06-10T18:51:48.400756+0000","last_unstale":"2023-06-10T18:51:49.380554+0000","last_undegraded":"2023-06-10T18:51:49.380554+0000","last_fullsized":"2023-06-10T18:51:49.380554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:11.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383216+0000","last_change":"2023-06-10T18:51:48.397643+0000","last_active":"2023-06-10T18:51:49.383216+0000","last_peered":"2023-06-10T18:51:49.383216+0000","last_clean":"2023-06-10T18:51:49.383216+0000","last_became_active":"2023-06-10T18:51:48.397334+0000","last_became_peered":"2023-06-10T18:51:48.397334+0000","last_unstale":"2023-06-10T18:51:49.383216+0000","last_undegraded":"2023-06-10T18:51:49.383216+0000","last_fullsized":"2023-06-10T18:51:49.383216+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:16.685695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624100+0000","last_change":"2023-06-10T18:51:48.399459+0000","last_active":"2023-06-10T18:51:49.624100+0000","last_peered":"2023-06-10T18:51:49.624100+0000","last_clean":"2023-06-10T18:51:49.624100+0000","last_became_active":"2023-06-10T18:51:48.399176+0000","last_became_peered":"2023-06-10T18:51:48.399176+0000","last_unstale":"2023-06-10T18:51:49.624100+0000","last_undegraded":"2023-06-10T18:51:49.624100+0000","last_fullsized":"2023-06-10T18:51:49.624100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:24.130103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380970+0000","last_change":"2023-06-10T18:51:48.400544+0000","last_active":"2023-06-10T18:51:49.380970+0000","last_peered":"2023-06-10T18:51:49.380970+0000","last_clean":"2023-06-10T18:51:49.380970+0000","last_became_active":"2023-06-10T18:51:48.400277+0000","last_became_peered":"2023-06-10T18:51:48.400277+0000","last_unstale":"2023-06-10T18:51:49.380970+0000","last_undegraded":"2023-06-10T18:51:49.380970+0000","last_fullsized":"2023-06-10T18:51:49.380970+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:47.547020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378028+0000","last_change":"2023-06-10T18:51:48.401372+0000","last_active":"2023-06-10T18:51:49.378028+0000","last_peered":"2023-06-10T18:51:49.378028+0000","last_clean":"2023-06-10T18:51:49.378028+0000","last_became_active":"2023-06-10T18:51:48.401092+0000","last_became_peered":"2023-06-10T18:51:48.401092+0000","last_unstale":"2023-06-10T18:51:49.378028+0000","last_undegraded":"2023-06-10T18:51:49.378028+0000","last_fullsized":"2023-06-10T18:51:49.378028+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:39.409186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382269+0000","last_change":"2023-06-10T18:51:48.393667+0000","last_active":"2023-06-10T18:51:49.382269+0000","last_peered":"2023-06-10T18:51:49.382269+0000","last_clean":"2023-06-10T18:51:49.382269+0000","last_became_active":"2023-06-10T18:51:48.393342+0000","last_became_peered":"2023-06-10T18:51:48.393342+0000","last_unstale":"2023-06-10T18:51:49.382269+0000","last_undegraded":"2023-06-10T18:51:49.382269+0000","last_fullsized":"2023-06-10T18:51:49.382269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:12.665569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383498+0000","last_change":"2023-06-10T18:51:48.396836+0000","last_active":"2023-06-10T18:51:49.383498+0000","last_peered":"2023-06-10T18:51:49.383498+0000","last_clean":"2023-06-10T18:51:49.383498+0000","last_became_active":"2023-06-10T18:51:48.396395+0000","last_became_peered":"2023-06-10T18:51:48.396395+0000","last_unstale":"2023-06-10T18:51:49.383498+0000","last_undegraded":"2023-06-10T18:51:49.383498+0000","last_fullsized":"2023-06-10T18:51:49.383498+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:06.492459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380732+0000","last_change":"2023-06-10T18:51:48.396566+0000","last_active":"2023-06-10T18:51:49.380732+0000","last_peered":"2023-06-10T18:51:49.380732+0000","last_clean":"2023-06-10T18:51:49.380732+0000","last_became_active":"2023-06-10T18:51:48.396294+0000","last_became_peered":"2023-06-10T18:51:48.396294+0000","last_unstale":"2023-06-10T18:51:49.380732+0000","last_undegraded":"2023-06-10T18:51:49.380732+0000","last_fullsized":"2023-06-10T18:51:49.380732+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:06.492459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380751+0000","last_change":"2023-06-10T18:51:48.396468+0000","last_active":"2023-06-10T18:51:49.380751+0000","last_peered":"2023-06-10T18:51:49.380751+0000","last_clean":"2023-06-10T18:51:49.380751+0000","last_became_active":"2023-06-10T18:51:48.396207+0000","last_became_peered":"2023-06-10T18:51:48.396207+0000","last_unstale":"2023-06-10T18:51:49.380751+0000","last_undegraded":"2023-06-10T18:51:49.380751+0000","last_fullsized":"2023-06-10T18:51:49.380751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:21:28.376204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382130+0000","last_change":"2023-06-10T18:51:48.386674+0000","last_active":"2023-06-10T18:51:49.382130+0000","last_peered":"2023-06-10T18:51:49.382130+0000","last_clean":"2023-06-10T18:51:49.382130+0000","last_became_active":"2023-06-10T18:51:48.386084+0000","last_became_peered":"2023-06-10T18:51:48.386084+0000","last_unstale":"2023-06-10T18:51:49.382130+0000","last_undegraded":"2023-06-10T18:51:49.382130+0000","last_fullsized":"2023-06-10T18:51:49.382130+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:52.150020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383209+0000","last_change":"2023-06-10T18:51:48.403182+0000","last_active":"2023-06-10T18:51:49.383209+0000","last_peered":"2023-06-10T18:51:49.383209+0000","last_clean":"2023-06-10T18:51:49.383209+0000","last_became_active":"2023-06-10T18:51:48.402915+0000","last_became_peered":"2023-06-10T18:51:48.402915+0000","last_unstale":"2023-06-10T18:51:49.383209+0000","last_undegraded":"2023-06-10T18:51:49.383209+0000","last_fullsized":"2023-06-10T18:51:49.383209+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:16.066951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382423+0000","last_change":"2023-06-10T18:51:48.399859+0000","last_active":"2023-06-10T18:51:49.382423+0000","last_peered":"2023-06-10T18:51:49.382423+0000","last_clean":"2023-06-10T18:51:49.382423+0000","last_became_active":"2023-06-10T18:51:48.399589+0000","last_became_peered":"2023-06-10T18:51:48.399589+0000","last_unstale":"2023-06-10T18:51:49.382423+0000","last_undegraded":"2023-06-10T18:51:49.382423+0000","last_fullsized":"2023-06-10T18:51:49.382423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:29.310115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383983+0000","last_change":"2023-06-10T18:51:47.987681+0000","last_active":"2023-06-10T18:51:49.383983+0000","last_peered":"2023-06-10T18:51:49.383983+0000","last_clean":"2023-06-10T18:51:49.383983+0000","last_became_active":"2023-06-10T18:51:47.987266+0000","last_became_peered":"2023-06-10T18:51:47.987266+0000","last_unstale":"2023-06-10T18:51:49.383983+0000","last_undegraded":"2023-06-10T18:51:49.383983+0000","last_fullsized":"2023-06-10T18:51:49.383983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:20.641457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.384089+0000","last_change":"2023-06-10T18:51:48.399497+0000","last_active":"2023-06-10T18:51:49.384089+0000","last_peered":"2023-06-10T18:51:49.384089+0000","last_clean":"2023-06-10T18:51:49.384089+0000","last_became_active":"2023-06-10T18:51:48.398830+0000","last_became_peered":"2023-06-10T18:51:48.398830+0000","last_unstale":"2023-06-10T18:51:49.384089+0000","last_undegraded":"2023-06-10T18:51:49.384089+0000","last_fullsized":"2023-06-10T18:51:49.384089+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:50:46.167170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382344+0000","last_change":"2023-06-10T18:51:47.991139+0000","last_active":"2023-06-10T18:51:49.382344+0000","last_peered":"2023-06-10T18:51:49.382344+0000","last_clean":"2023-06-10T18:51:49.382344+0000","last_became_active":"2023-06-10T18:51:47.990905+0000","last_became_peered":"2023-06-10T18:51:47.990905+0000","last_unstale":"2023-06-10T18:51:49.382344+0000","last_undegraded":"2023-06-10T18:51:49.382344+0000","last_fullsized":"2023-06-10T18:51:49.382344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:57.547464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383455+0000","last_change":"2023-06-10T18:51:48.401666+0000","last_active":"2023-06-10T18:51:49.383455+0000","last_peered":"2023-06-10T18:51:49.383455+0000","last_clean":"2023-06-10T18:51:49.383455+0000","last_became_active":"2023-06-10T18:51:48.401343+0000","last_became_peered":"2023-06-10T18:51:48.401343+0000","last_unstale":"2023-06-10T18:51:49.383455+0000","last_undegraded":"2023-06-10T18:51:49.383455+0000","last_fullsized":"2023-06-10T18:51:49.383455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:21:28.376204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625450+0000","last_change":"2023-06-10T18:51:47.987428+0000","last_active":"2023-06-10T18:51:49.625450+0000","last_peered":"2023-06-10T18:51:49.625450+0000","last_clean":"2023-06-10T18:51:49.625450+0000","last_became_active":"2023-06-10T18:51:47.987205+0000","last_became_peered":"2023-06-10T18:51:47.987205+0000","last_unstale":"2023-06-10T18:51:49.625450+0000","last_undegraded":"2023-06-10T18:51:49.625450+0000","last_fullsized":"2023-06-10T18:51:49.625450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:09:15.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377930+0000","last_change":"2023-06-10T18:51:48.392214+0000","last_active":"2023-06-10T18:51:49.377930+0000","last_peered":"2023-06-10T18:51:49.377930+0000","last_clean":"2023-06-10T18:51:49.377930+0000","last_became_active":"2023-06-10T18:51:48.391937+0000","last_became_peered":"2023-06-10T18:51:48.391937+0000","last_unstale":"2023-06-10T18:51:49.377930+0000","last_undegraded":"2023-06-10T18:51:49.377930+0000","last_fullsized":"2023-06-10T18:51:49.377930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:06:43.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625176+0000","last_change":"2023-06-10T18:51:47.987865+0000","last_active":"2023-06-10T18:51:49.625176+0000","last_peered":"2023-06-10T18:51:49.625176+0000","last_clean":"2023-06-10T18:51:49.625176+0000","last_became_active":"2023-06-10T18:51:47.987614+0000","last_became_peered":"2023-06-10T18:51:47.987614+0000","last_unstale":"2023-06-10T18:51:49.625176+0000","last_undegraded":"2023-06-10T18:51:49.625176+0000","last_fullsized":"2023-06-10T18:51:49.625176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:56.618982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380631+0000","last_change":"2023-06-10T18:51:48.400984+0000","last_active":"2023-06-10T18:51:49.380631+0000","last_peered":"2023-06-10T18:51:49.380631+0000","last_clean":"2023-06-10T18:51:49.380631+0000","last_became_active":"2023-06-10T18:51:48.400402+0000","last_became_peered":"2023-06-10T18:51:48.400402+0000","last_unstale":"2023-06-10T18:51:49.380631+0000","last_undegraded":"2023-06-10T18:51:49.380631+0000","last_fullsized":"2023-06-10T18:51:49.380631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:34.966942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382095+0000","last_change":"2023-06-10T18:51:47.990886+0000","last_active":"2023-06-10T18:51:49.382095+0000","last_peered":"2023-06-10T18:51:49.382095+0000","last_clean":"2023-06-10T18:51:49.382095+0000","last_became_active":"2023-06-10T18:51:47.990649+0000","last_became_peered":"2023-06-10T18:51:47.990649+0000","last_unstale":"2023-06-10T18:51:49.382095+0000","last_undegraded":"2023-06-10T18:51:49.382095+0000","last_fullsized":"2023-06-10T18:51:49.382095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:38:32.258439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382098+0000","last_change":"2023-06-10T18:51:48.386807+0000","last_active":"2023-06-10T18:51:49.382098+0000","last_peered":"2023-06-10T18:51:49.382098+0000","last_clean":"2023-06-10T18:51:49.382098+0000","last_became_active":"2023-06-10T18:51:48.386503+0000","last_became_peered":"2023-06-10T18:51:48.386503+0000","last_unstale":"2023-06-10T18:51:49.382098+0000","last_undegraded":"2023-06-10T18:51:49.382098+0000","last_fullsized":"2023-06-10T18:51:49.382098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:47:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381611+0000","last_change":"2023-06-10T18:51:47.995836+0000","last_active":"2023-06-10T18:51:49.381611+0000","last_peered":"2023-06-10T18:51:49.381611+0000","last_clean":"2023-06-10T18:51:49.381611+0000","last_became_active":"2023-06-10T18:51:47.995594+0000","last_became_peered":"2023-06-10T18:51:47.995594+0000","last_unstale":"2023-06-10T18:51:49.381611+0000","last_undegraded":"2023-06-10T18:51:49.381611+0000","last_fullsized":"2023-06-10T18:51:49.381611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:26:22.505078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377656+0000","last_change":"2023-06-10T18:51:48.394078+0000","last_active":"2023-06-10T18:51:49.377656+0000","last_peered":"2023-06-10T18:51:49.377656+0000","last_clean":"2023-06-10T18:51:49.377656+0000","last_became_active":"2023-06-10T18:51:48.391671+0000","last_became_peered":"2023-06-10T18:51:48.391671+0000","last_unstale":"2023-06-10T18:51:49.377656+0000","last_undegraded":"2023-06-10T18:51:49.377656+0000","last_fullsized":"2023-06-10T18:51:49.377656+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:53:37.194624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623983+0000","last_change":"2023-06-10T18:51:47.985868+0000","last_active":"2023-06-10T18:51:49.623983+0000","last_peered":"2023-06-10T18:51:49.623983+0000","last_clean":"2023-06-10T18:51:49.623983+0000","last_became_active":"2023-06-10T18:51:47.985572+0000","last_became_peered":"2023-06-10T18:51:47.985572+0000","last_unstale":"2023-06-10T18:51:49.623983+0000","last_undegraded":"2023-06-10T18:51:49.623983+0000","last_fullsized":"2023-06-10T18:51:49.623983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:11.403051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623960+0000","last_change":"2023-06-10T18:51:48.387610+0000","last_active":"2023-06-10T18:51:49.623960+0000","last_peered":"2023-06-10T18:51:49.623960+0000","last_clean":"2023-06-10T18:51:49.623960+0000","last_became_active":"2023-06-10T18:51:48.387276+0000","last_became_peered":"2023-06-10T18:51:48.387276+0000","last_unstale":"2023-06-10T18:51:49.623960+0000","last_undegraded":"2023-06-10T18:51:49.623960+0000","last_fullsized":"2023-06-10T18:51:49.623960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:03.200969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380505+0000","last_change":"2023-06-10T18:51:47.991946+0000","last_active":"2023-06-10T18:51:49.380505+0000","last_peered":"2023-06-10T18:51:49.380505+0000","last_clean":"2023-06-10T18:51:49.380505+0000","last_became_active":"2023-06-10T18:51:47.991654+0000","last_became_peered":"2023-06-10T18:51:47.991654+0000","last_unstale":"2023-06-10T18:51:49.380505+0000","last_undegraded":"2023-06-10T18:51:49.380505+0000","last_fullsized":"2023-06-10T18:51:49.380505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:54:57.340884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624960+0000","last_change":"2023-06-10T18:51:48.395577+0000","last_active":"2023-06-10T18:51:49.624960+0000","last_peered":"2023-06-10T18:51:49.624960+0000","last_clean":"2023-06-10T18:51:49.624960+0000","last_became_active":"2023-06-10T18:51:48.395287+0000","last_became_peered":"2023-06-10T18:51:48.395287+0000","last_unstale":"2023-06-10T18:51:49.624960+0000","last_undegraded":"2023-06-10T18:51:49.624960+0000","last_fullsized":"2023-06-10T18:51:49.624960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:47.547020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383953+0000","last_change":"2023-06-10T18:51:47.990334+0000","last_active":"2023-06-10T18:51:49.383953+0000","last_peered":"2023-06-10T18:51:49.383953+0000","last_clean":"2023-06-10T18:51:49.383953+0000","last_became_active":"2023-06-10T18:51:47.990065+0000","last_became_peered":"2023-06-10T18:51:47.990065+0000","last_unstale":"2023-06-10T18:51:49.383953+0000","last_undegraded":"2023-06-10T18:51:49.383953+0000","last_fullsized":"2023-06-10T18:51:49.383953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:31.385016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381021+0000","last_change":"2023-06-10T18:51:48.391076+0000","last_active":"2023-06-10T18:51:49.381021+0000","last_peered":"2023-06-10T18:51:49.381021+0000","last_clean":"2023-06-10T18:51:49.381021+0000","last_became_active":"2023-06-10T18:51:48.390814+0000","last_became_peered":"2023-06-10T18:51:48.390814+0000","last_unstale":"2023-06-10T18:51:49.381021+0000","last_undegraded":"2023-06-10T18:51:49.381021+0000","last_fullsized":"2023-06-10T18:51:49.381021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:25.986113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383773+0000","last_change":"2023-06-10T18:51:47.992194+0000","last_active":"2023-06-10T18:51:49.383773+0000","last_peered":"2023-06-10T18:51:49.383773+0000","last_clean":"2023-06-10T18:51:49.383773+0000","last_became_active":"2023-06-10T18:51:47.991936+0000","last_became_peered":"2023-06-10T18:51:47.991936+0000","last_unstale":"2023-06-10T18:51:49.383773+0000","last_undegraded":"2023-06-10T18:51:49.383773+0000","last_fullsized":"2023-06-10T18:51:49.383773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:02.195295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624508+0000","last_change":"2023-06-10T18:51:48.397516+0000","last_active":"2023-06-10T18:51:49.624508+0000","last_peered":"2023-06-10T18:51:49.624508+0000","last_clean":"2023-06-10T18:51:49.624508+0000","last_became_active":"2023-06-10T18:51:48.397263+0000","last_became_peered":"2023-06-10T18:51:48.397263+0000","last_unstale":"2023-06-10T18:51:49.624508+0000","last_undegraded":"2023-06-10T18:51:49.624508+0000","last_fullsized":"2023-06-10T18:51:49.624508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:50.234704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383572+0000","last_change":"2023-06-10T18:51:47.988055+0000","last_active":"2023-06-10T18:51:49.383572+0000","last_peered":"2023-06-10T18:51:49.383572+0000","last_clean":"2023-06-10T18:51:49.383572+0000","last_became_active":"2023-06-10T18:51:47.987779+0000","last_became_peered":"2023-06-10T18:51:47.987779+0000","last_unstale":"2023-06-10T18:51:49.383572+0000","last_undegraded":"2023-06-10T18:51:49.383572+0000","last_fullsized":"2023-06-10T18:51:49.383572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:17.219417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381827+0000","last_change":"2023-06-10T18:51:48.391908+0000","last_active":"2023-06-10T18:51:49.381827+0000","last_peered":"2023-06-10T18:51:49.381827+0000","last_clean":"2023-06-10T18:51:49.381827+0000","last_became_active":"2023-06-10T18:51:48.391468+0000","last_became_peered":"2023-06-10T18:51:48.391468+0000","last_unstale":"2023-06-10T18:51:49.381827+0000","last_undegraded":"2023-06-10T18:51:49.381827+0000","last_fullsized":"2023-06-10T18:51:49.381827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:43:51.612691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380919+0000","last_change":"2023-06-10T18:51:47.981335+0000","last_active":"2023-06-10T18:51:49.380919+0000","last_peered":"2023-06-10T18:51:49.380919+0000","last_clean":"2023-06-10T18:51:49.380919+0000","last_became_active":"2023-06-10T18:51:47.981022+0000","last_became_peered":"2023-06-10T18:51:47.981022+0000","last_unstale":"2023-06-10T18:51:49.380919+0000","last_undegraded":"2023-06-10T18:51:49.380919+0000","last_fullsized":"2023-06-10T18:51:49.380919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:34:48.939304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624262+0000","last_change":"2023-06-10T18:51:48.390023+0000","last_active":"2023-06-10T18:51:49.624262+0000","last_peered":"2023-06-10T18:51:49.624262+0000","last_clean":"2023-06-10T18:51:49.624262+0000","last_became_active":"2023-06-10T18:51:48.389749+0000","last_became_peered":"2023-06-10T18:51:48.389749+0000","last_unstale":"2023-06-10T18:51:49.624262+0000","last_undegraded":"2023-06-10T18:51:49.624262+0000","last_fullsized":"2023-06-10T18:51:49.624262+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:24:28.528107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381036+0000","last_change":"2023-06-10T18:51:47.985509+0000","last_active":"2023-06-10T18:51:49.381036+0000","last_peered":"2023-06-10T18:51:49.381036+0000","last_clean":"2023-06-10T18:51:49.381036+0000","last_became_active":"2023-06-10T18:51:47.985211+0000","last_became_peered":"2023-06-10T18:51:47.985211+0000","last_unstale":"2023-06-10T18:51:49.381036+0000","last_undegraded":"2023-06-10T18:51:49.381036+0000","last_fullsized":"2023-06-10T18:51:49.381036+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:31.385016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377808+0000","last_change":"2023-06-10T18:51:48.392517+0000","last_active":"2023-06-10T18:51:49.377808+0000","last_peered":"2023-06-10T18:51:49.377808+0000","last_clean":"2023-06-10T18:51:49.377808+0000","last_became_active":"2023-06-10T18:51:48.391882+0000","last_became_peered":"2023-06-10T18:51:48.391882+0000","last_unstale":"2023-06-10T18:51:49.377808+0000","last_undegraded":"2023-06-10T18:51:49.377808+0000","last_fullsized":"2023-06-10T18:51:49.377808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:39:45.638924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624402+0000","last_change":"2023-06-10T18:51:47.987992+0000","last_active":"2023-06-10T18:51:49.624402+0000","last_peered":"2023-06-10T18:51:49.624402+0000","last_clean":"2023-06-10T18:51:49.624402+0000","last_became_active":"2023-06-10T18:51:47.987738+0000","last_became_peered":"2023-06-10T18:51:47.987738+0000","last_unstale":"2023-06-10T18:51:49.624402+0000","last_undegraded":"2023-06-10T18:51:49.624402+0000","last_fullsized":"2023-06-10T18:51:49.624402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:08.331733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380499+0000","last_change":"2023-06-10T18:51:48.402300+0000","last_active":"2023-06-10T18:51:49.380499+0000","last_peered":"2023-06-10T18:51:49.380499+0000","last_clean":"2023-06-10T18:51:49.380499+0000","last_became_active":"2023-06-10T18:51:48.402037+0000","last_became_peered":"2023-06-10T18:51:48.402037+0000","last_unstale":"2023-06-10T18:51:49.380499+0000","last_undegraded":"2023-06-10T18:51:49.380499+0000","last_fullsized":"2023-06-10T18:51:49.380499+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:22.801629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380618+0000","last_change":"2023-06-10T18:51:47.992514+0000","last_active":"2023-06-10T18:51:49.380618+0000","last_peered":"2023-06-10T18:51:49.380618+0000","last_clean":"2023-06-10T18:51:49.380618+0000","last_became_active":"2023-06-10T18:51:47.992269+0000","last_became_peered":"2023-06-10T18:51:47.992269+0000","last_unstale":"2023-06-10T18:51:49.380618+0000","last_undegraded":"2023-06-10T18:51:49.380618+0000","last_fullsized":"2023-06-10T18:51:49.380618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:57:49.167698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380566+0000","last_change":"2023-06-10T18:51:48.398094+0000","last_active":"2023-06-10T18:51:49.380566+0000","last_peered":"2023-06-10T18:51:49.380566+0000","last_clean":"2023-06-10T18:51:49.380566+0000","last_became_active":"2023-06-10T18:51:48.397819+0000","last_became_peered":"2023-06-10T18:51:48.397819+0000","last_unstale":"2023-06-10T18:51:49.380566+0000","last_undegraded":"2023-06-10T18:51:49.380566+0000","last_fullsized":"2023-06-10T18:51:49.380566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:16.685695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624252+0000","last_change":"2023-06-10T18:51:47.981834+0000","last_active":"2023-06-10T18:51:49.624252+0000","last_peered":"2023-06-10T18:51:49.624252+0000","last_clean":"2023-06-10T18:51:49.624252+0000","last_became_active":"2023-06-10T18:51:47.981587+0000","last_became_peered":"2023-06-10T18:51:47.981587+0000","last_unstale":"2023-06-10T18:51:49.624252+0000","last_undegraded":"2023-06-10T18:51:49.624252+0000","last_fullsized":"2023-06-10T18:51:49.624252+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:39.669408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382202+0000","last_change":"2023-06-10T18:51:48.382463+0000","last_active":"2023-06-10T18:51:49.382202+0000","last_peered":"2023-06-10T18:51:49.382202+0000","last_clean":"2023-06-10T18:51:49.382202+0000","last_became_active":"2023-06-10T18:51:48.382190+0000","last_became_peered":"2023-06-10T18:51:48.382190+0000","last_unstale":"2023-06-10T18:51:49.382202+0000","last_undegraded":"2023-06-10T18:51:49.382202+0000","last_fullsized":"2023-06-10T18:51:49.382202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:13:02.105582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380959+0000","last_change":"2023-06-10T18:51:47.989294+0000","last_active":"2023-06-10T18:51:49.380959+0000","last_peered":"2023-06-10T18:51:49.380959+0000","last_clean":"2023-06-10T18:51:49.380959+0000","last_became_active":"2023-06-10T18:51:47.989022+0000","last_became_peered":"2023-06-10T18:51:47.989022+0000","last_unstale":"2023-06-10T18:51:49.380959+0000","last_undegraded":"2023-06-10T18:51:49.380959+0000","last_fullsized":"2023-06-10T18:51:49.380959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:08.331733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378105+0000","last_change":"2023-06-10T18:51:48.399670+0000","last_active":"2023-06-10T18:51:49.378105+0000","last_peered":"2023-06-10T18:51:49.378105+0000","last_clean":"2023-06-10T18:51:49.378105+0000","last_became_active":"2023-06-10T18:51:48.399378+0000","last_became_peered":"2023-06-10T18:51:48.399378+0000","last_unstale":"2023-06-10T18:51:49.378105+0000","last_undegraded":"2023-06-10T18:51:49.378105+0000","last_fullsized":"2023-06-10T18:51:49.378105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:32:03.498809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380863+0000","last_change":"2023-06-10T18:51:47.984973+0000","last_active":"2023-06-10T18:51:49.380863+0000","last_peered":"2023-06-10T18:51:49.380863+0000","last_clean":"2023-06-10T18:51:49.380863+0000","last_became_active":"2023-06-10T18:51:47.984697+0000","last_became_peered":"2023-06-10T18:51:47.984697+0000","last_unstale":"2023-06-10T18:51:49.380863+0000","last_undegraded":"2023-06-10T18:51:49.380863+0000","last_fullsized":"2023-06-10T18:51:49.380863+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:02.195295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383309+0000","last_change":"2023-06-10T18:51:48.398836+0000","last_active":"2023-06-10T18:51:49.383309+0000","last_peered":"2023-06-10T18:51:49.383309+0000","last_clean":"2023-06-10T18:51:49.383309+0000","last_became_active":"2023-06-10T18:51:48.398514+0000","last_became_peered":"2023-06-10T18:51:48.398514+0000","last_unstale":"2023-06-10T18:51:49.383309+0000","last_undegraded":"2023-06-10T18:51:49.383309+0000","last_fullsized":"2023-06-10T18:51:49.383309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:11.072483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383607+0000","last_change":"2023-06-10T18:51:47.985586+0000","last_active":"2023-06-10T18:51:49.383607+0000","last_peered":"2023-06-10T18:51:49.383607+0000","last_clean":"2023-06-10T18:51:49.383607+0000","last_became_active":"2023-06-10T18:51:47.985284+0000","last_became_peered":"2023-06-10T18:51:47.985284+0000","last_unstale":"2023-06-10T18:51:49.383607+0000","last_undegraded":"2023-06-10T18:51:49.383607+0000","last_fullsized":"2023-06-10T18:51:49.383607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:56.914900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.382151+0000","last_change":"2023-06-10T18:51:48.391743+0000","last_active":"2023-06-10T18:51:49.382151+0000","last_peered":"2023-06-10T18:51:49.382151+0000","last_clean":"2023-06-10T18:51:49.382151+0000","last_became_active":"2023-06-10T18:51:48.391115+0000","last_became_peered":"2023-06-10T18:51:48.391115+0000","last_unstale":"2023-06-10T18:51:49.382151+0000","last_undegraded":"2023-06-10T18:51:49.382151+0000","last_fullsized":"2023-06-10T18:51:49.382151+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:16:42.405449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383616+0000","last_change":"2023-06-10T18:51:47.984028+0000","last_active":"2023-06-10T18:51:49.383616+0000","last_peered":"2023-06-10T18:51:49.383616+0000","last_clean":"2023-06-10T18:51:49.383616+0000","last_became_active":"2023-06-10T18:51:47.983744+0000","last_became_peered":"2023-06-10T18:51:47.983744+0000","last_unstale":"2023-06-10T18:51:49.383616+0000","last_undegraded":"2023-06-10T18:51:49.383616+0000","last_fullsized":"2023-06-10T18:51:49.383616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:53.720896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380623+0000","last_change":"2023-06-10T18:51:48.385157+0000","last_active":"2023-06-10T18:51:49.380623+0000","last_peered":"2023-06-10T18:51:49.380623+0000","last_clean":"2023-06-10T18:51:49.380623+0000","last_became_active":"2023-06-10T18:51:48.384841+0000","last_became_peered":"2023-06-10T18:51:48.384841+0000","last_unstale":"2023-06-10T18:51:49.380623+0000","last_undegraded":"2023-06-10T18:51:49.380623+0000","last_fullsized":"2023-06-10T18:51:49.380623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:22:49.512567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625209+0000","last_change":"2023-06-10T18:51:47.983812+0000","last_active":"2023-06-10T18:51:49.625209+0000","last_peered":"2023-06-10T18:51:49.625209+0000","last_clean":"2023-06-10T18:51:49.625209+0000","last_became_active":"2023-06-10T18:51:47.983100+0000","last_became_peered":"2023-06-10T18:51:47.983100+0000","last_unstale":"2023-06-10T18:51:49.625209+0000","last_undegraded":"2023-06-10T18:51:49.625209+0000","last_fullsized":"2023-06-10T18:51:49.625209+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:45:32.401099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625156+0000","last_change":"2023-06-10T18:51:48.390131+0000","last_active":"2023-06-10T18:51:49.625156+0000","last_peered":"2023-06-10T18:51:49.625156+0000","last_clean":"2023-06-10T18:51:49.625156+0000","last_became_active":"2023-06-10T18:51:48.389863+0000","last_became_peered":"2023-06-10T18:51:48.389863+0000","last_unstale":"2023-06-10T18:51:49.625156+0000","last_undegraded":"2023-06-10T18:51:49.625156+0000","last_fullsized":"2023-06-10T18:51:49.625156+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:09:15.857938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381766+0000","last_change":"2023-06-10T18:51:47.990817+0000","last_active":"2023-06-10T18:51:49.381766+0000","last_peered":"2023-06-10T18:51:49.381766+0000","last_clean":"2023-06-10T18:51:49.381766+0000","last_became_active":"2023-06-10T18:51:47.990556+0000","last_became_peered":"2023-06-10T18:51:47.990556+0000","last_unstale":"2023-06-10T18:51:49.381766+0000","last_undegraded":"2023-06-10T18:51:49.381766+0000","last_fullsized":"2023-06-10T18:51:49.381766+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:24:29.305759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377661+0000","last_change":"2023-06-10T18:51:48.403094+0000","last_active":"2023-06-10T18:51:49.377661+0000","last_peered":"2023-06-10T18:51:49.377661+0000","last_clean":"2023-06-10T18:51:49.377661+0000","last_became_active":"2023-06-10T18:51:48.402822+0000","last_became_peered":"2023-06-10T18:51:48.402822+0000","last_unstale":"2023-06-10T18:51:49.377661+0000","last_undegraded":"2023-06-10T18:51:49.377661+0000","last_fullsized":"2023-06-10T18:51:49.377661+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:47:32.336237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383392+0000","last_change":"2023-06-10T18:51:47.984261+0000","last_active":"2023-06-10T18:51:49.383392+0000","last_peered":"2023-06-10T18:51:49.383392+0000","last_clean":"2023-06-10T18:51:49.383392+0000","last_became_active":"2023-06-10T18:51:47.983679+0000","last_became_peered":"2023-06-10T18:51:47.983679+0000","last_unstale":"2023-06-10T18:51:49.383392+0000","last_undegraded":"2023-06-10T18:51:49.383392+0000","last_fullsized":"2023-06-10T18:51:49.383392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:33.961267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383325+0000","last_change":"2023-06-10T18:51:48.397113+0000","last_active":"2023-06-10T18:51:49.383325+0000","last_peered":"2023-06-10T18:51:49.383325+0000","last_clean":"2023-06-10T18:51:49.383325+0000","last_became_active":"2023-06-10T18:51:48.396810+0000","last_became_peered":"2023-06-10T18:51:48.396810+0000","last_unstale":"2023-06-10T18:51:49.383325+0000","last_undegraded":"2023-06-10T18:51:49.383325+0000","last_fullsized":"2023-06-10T18:51:49.383325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:22.801629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381669+0000","last_change":"2023-06-10T18:51:47.991411+0000","last_active":"2023-06-10T18:51:49.381669+0000","last_peered":"2023-06-10T18:51:49.381669+0000","last_clean":"2023-06-10T18:51:49.381669+0000","last_became_active":"2023-06-10T18:51:47.991126+0000","last_became_peered":"2023-06-10T18:51:47.991126+0000","last_unstale":"2023-06-10T18:51:49.381669+0000","last_undegraded":"2023-06-10T18:51:49.381669+0000","last_fullsized":"2023-06-10T18:51:49.381669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:49:40.430246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380789+0000","last_change":"2023-06-10T18:51:48.391781+0000","last_active":"2023-06-10T18:51:49.380789+0000","last_peered":"2023-06-10T18:51:49.380789+0000","last_clean":"2023-06-10T18:51:49.380789+0000","last_became_active":"2023-06-10T18:51:48.391530+0000","last_became_peered":"2023-06-10T18:51:48.391530+0000","last_unstale":"2023-06-10T18:51:49.380789+0000","last_undegraded":"2023-06-10T18:51:49.380789+0000","last_fullsized":"2023-06-10T18:51:49.380789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:14:22.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624953+0000","last_change":"2023-06-10T18:51:47.982022+0000","last_active":"2023-06-10T18:51:49.624953+0000","last_peered":"2023-06-10T18:51:49.624953+0000","last_clean":"2023-06-10T18:51:49.624953+0000","last_became_active":"2023-06-10T18:51:47.981720+0000","last_became_peered":"2023-06-10T18:51:47.981720+0000","last_unstale":"2023-06-10T18:51:49.624953+0000","last_undegraded":"2023-06-10T18:51:49.624953+0000","last_fullsized":"2023-06-10T18:51:49.624953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:22.267003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380673+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:51:49.380673+0000","last_peered":"2023-06-10T18:51:49.380673+0000","last_clean":"2023-06-10T18:51:49.380673+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:51:49.380673+0000","last_undegraded":"2023-06-10T18:51:49.380673+0000","last_fullsized":"2023-06-10T18:51:49.380673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:58:12.475609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.381982+0000","last_change":"2023-06-10T18:51:48.402926+0000","last_active":"2023-06-10T18:51:49.381982+0000","last_peered":"2023-06-10T18:51:49.381982+0000","last_clean":"2023-06-10T18:51:49.381982+0000","last_became_active":"2023-06-10T18:51:48.402639+0000","last_became_peered":"2023-06-10T18:51:48.402639+0000","last_unstale":"2023-06-10T18:51:49.381982+0000","last_undegraded":"2023-06-10T18:51:49.381982+0000","last_fullsized":"2023-06-10T18:51:49.381982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:05.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624622+0000","last_change":"2023-06-10T18:51:47.985853+0000","last_active":"2023-06-10T18:51:49.624622+0000","last_peered":"2023-06-10T18:51:49.624622+0000","last_clean":"2023-06-10T18:51:49.624622+0000","last_became_active":"2023-06-10T18:51:47.985583+0000","last_became_peered":"2023-06-10T18:51:47.985583+0000","last_unstale":"2023-06-10T18:51:49.624622+0000","last_undegraded":"2023-06-10T18:51:49.624622+0000","last_fullsized":"2023-06-10T18:51:49.624622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:26.364949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380657+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:51:49.380657+0000","last_peered":"2023-06-10T18:51:49.380657+0000","last_clean":"2023-06-10T18:51:49.380657+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:51:49.380657+0000","last_undegraded":"2023-06-10T18:51:49.380657+0000","last_fullsized":"2023-06-10T18:51:49.380657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:55:13.554861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378065+0000","last_change":"2023-06-10T18:51:48.393893+0000","last_active":"2023-06-10T18:51:49.378065+0000","last_peered":"2023-06-10T18:51:49.378065+0000","last_clean":"2023-06-10T18:51:49.378065+0000","last_became_active":"2023-06-10T18:51:48.393627+0000","last_became_peered":"2023-06-10T18:51:48.393627+0000","last_unstale":"2023-06-10T18:51:49.378065+0000","last_undegraded":"2023-06-10T18:51:49.378065+0000","last_fullsized":"2023-06-10T18:51:49.378065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:11.871412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383888+0000","last_change":"2023-06-10T18:51:47.986705+0000","last_active":"2023-06-10T18:51:49.383888+0000","last_peered":"2023-06-10T18:51:49.383888+0000","last_clean":"2023-06-10T18:51:49.383888+0000","last_became_active":"2023-06-10T18:51:47.986423+0000","last_became_peered":"2023-06-10T18:51:47.986423+0000","last_unstale":"2023-06-10T18:51:49.383888+0000","last_undegraded":"2023-06-10T18:51:49.383888+0000","last_fullsized":"2023-06-10T18:51:49.383888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:46.541345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624376+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:51:49.624376+0000","last_peered":"2023-06-10T18:51:49.624376+0000","last_clean":"2023-06-10T18:51:49.624376+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:51:49.624376+0000","last_undegraded":"2023-06-10T18:51:49.624376+0000","last_fullsized":"2023-06-10T18:51:49.624376+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:48:32.852209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380727+0000","last_change":"2023-06-10T18:51:48.394754+0000","last_active":"2023-06-10T18:51:49.380727+0000","last_peered":"2023-06-10T18:51:49.380727+0000","last_clean":"2023-06-10T18:51:49.380727+0000","last_became_active":"2023-06-10T18:51:48.394550+0000","last_became_peered":"2023-06-10T18:51:48.394550+0000","last_unstale":"2023-06-10T18:51:49.380727+0000","last_undegraded":"2023-06-10T18:51:49.380727+0000","last_fullsized":"2023-06-10T18:51:49.380727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:18.225092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377973+0000","last_change":"2023-06-10T18:51:47.989368+0000","last_active":"2023-06-10T18:51:49.377973+0000","last_peered":"2023-06-10T18:51:49.377973+0000","last_clean":"2023-06-10T18:51:49.377973+0000","last_became_active":"2023-06-10T18:51:47.989082+0000","last_became_peered":"2023-06-10T18:51:47.989082+0000","last_unstale":"2023-06-10T18:51:49.377973+0000","last_undegraded":"2023-06-10T18:51:49.377973+0000","last_fullsized":"2023-06-10T18:51:49.377973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:16.663218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380563+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:51:49.380563+0000","last_peered":"2023-06-10T18:51:49.380563+0000","last_clean":"2023-06-10T18:51:49.380563+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:51:49.380563+0000","last_undegraded":"2023-06-10T18:51:49.380563+0000","last_fullsized":"2023-06-10T18:51:49.380563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:39.628192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":284,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377748+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:51:49.377748+0000","last_peered":"2023-06-10T18:51:49.377748+0000","last_clean":"2023-06-10T18:51:49.377748+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:51:49.377748+0000","last_undegraded":"2023-06-10T18:51:49.377748+0000","last_fullsized":"2023-06-10T18:51:49.377748+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:41:01.529544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.384060+0000","last_change":"2023-06-10T18:51:48.399675+0000","last_active":"2023-06-10T18:51:49.384060+0000","last_peered":"2023-06-10T18:51:49.384060+0000","last_clean":"2023-06-10T18:51:49.384060+0000","last_became_active":"2023-06-10T18:51:48.399176+0000","last_became_peered":"2023-06-10T18:51:48.399176+0000","last_unstale":"2023-06-10T18:51:49.384060+0000","last_undegraded":"2023-06-10T18:51:49.384060+0000","last_fullsized":"2023-06-10T18:51:49.384060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:40.997736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.625482+0000","last_change":"2023-06-10T18:51:47.992465+0000","last_active":"2023-06-10T18:51:49.625482+0000","last_peered":"2023-06-10T18:51:49.625482+0000","last_clean":"2023-06-10T18:51:49.625482+0000","last_became_active":"2023-06-10T18:51:47.992180+0000","last_became_peered":"2023-06-10T18:51:47.992180+0000","last_unstale":"2023-06-10T18:51:49.625482+0000","last_undegraded":"2023-06-10T18:51:49.625482+0000","last_fullsized":"2023-06-10T18:51:49.625482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:15.680020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.377821+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:51:49.377821+0000","last_peered":"2023-06-10T18:51:49.377821+0000","last_clean":"2023-06-10T18:51:49.377821+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:51:49.377821+0000","last_undegraded":"2023-06-10T18:51:49.377821+0000","last_fullsized":"2023-06-10T18:51:49.377821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:06:40.909431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624779+0000","last_change":"2023-06-10T18:51:48.403247+0000","last_active":"2023-06-10T18:51:49.624779+0000","last_peered":"2023-06-10T18:51:49.624779+0000","last_clean":"2023-06-10T18:51:49.624779+0000","last_became_active":"2023-06-10T18:51:48.402885+0000","last_became_peered":"2023-06-10T18:51:48.402885+0000","last_unstale":"2023-06-10T18:51:49.624779+0000","last_undegraded":"2023-06-10T18:51:49.624779+0000","last_fullsized":"2023-06-10T18:51:49.624779+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:52:20.034259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623976+0000","last_change":"2023-06-10T18:51:47.980733+0000","last_active":"2023-06-10T18:51:49.623976+0000","last_peered":"2023-06-10T18:51:49.623976+0000","last_clean":"2023-06-10T18:51:49.623976+0000","last_became_active":"2023-06-10T18:51:47.980388+0000","last_became_peered":"2023-06-10T18:51:47.980388+0000","last_unstale":"2023-06-10T18:51:49.623976+0000","last_undegraded":"2023-06-10T18:51:49.623976+0000","last_fullsized":"2023-06-10T18:51:49.623976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:55.969246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.378062+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:51:49.378062+0000","last_peered":"2023-06-10T18:51:49.378062+0000","last_clean":"2023-06-10T18:51:49.378062+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:51:49.378062+0000","last_undegraded":"2023-06-10T18:51:49.378062+0000","last_fullsized":"2023-06-10T18:51:49.378062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:13.460094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380652+0000","last_change":"2023-06-10T18:51:48.388213+0000","last_active":"2023-06-10T18:51:49.380652+0000","last_peered":"2023-06-10T18:51:49.380652+0000","last_clean":"2023-06-10T18:51:49.380652+0000","last_became_active":"2023-06-10T18:51:48.387943+0000","last_became_peered":"2023-06-10T18:51:48.387943+0000","last_unstale":"2023-06-10T18:51:49.380652+0000","last_undegraded":"2023-06-10T18:51:49.380652+0000","last_fullsized":"2023-06-10T18:51:49.380652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:57.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.623711+0000","last_change":"2023-06-10T18:51:47.980408+0000","last_active":"2023-06-10T18:51:49.623711+0000","last_peered":"2023-06-10T18:51:49.623711+0000","last_clean":"2023-06-10T18:51:49.623711+0000","last_became_active":"2023-06-10T18:51:47.980134+0000","last_became_peered":"2023-06-10T18:51:47.980134+0000","last_unstale":"2023-06-10T18:51:49.623711+0000","last_undegraded":"2023-06-10T18:51:49.623711+0000","last_fullsized":"2023-06-10T18:51:49.623711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:53.720896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.380857+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:51:49.380857+0000","last_peered":"2023-06-10T18:51:49.380857+0000","last_clean":"2023-06-10T18:51:49.380857+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:51:49.380857+0000","last_undegraded":"2023-06-10T18:51:49.380857+0000","last_fullsized":"2023-06-10T18:51:49.380857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:55:11.529955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624234+0000","last_change":"2023-06-10T18:51:48.393080+0000","last_active":"2023-06-10T18:51:49.624234+0000","last_peered":"2023-06-10T18:51:49.624234+0000","last_clean":"2023-06-10T18:51:49.624234+0000","last_became_active":"2023-06-10T18:51:48.392801+0000","last_became_peered":"2023-06-10T18:51:48.392801+0000","last_unstale":"2023-06-10T18:51:49.624234+0000","last_undegraded":"2023-06-10T18:51:49.624234+0000","last_fullsized":"2023-06-10T18:51:49.624234+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:47.964992+0000","last_clean_scrub_stamp":"2023-06-10T18:51:47.964992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:06.319540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.624158+0000","last_change":"2023-06-10T18:51:47.981311+0000","last_active":"2023-06-10T18:51:49.624158+0000","last_peered":"2023-06-10T18:51:49.624158+0000","last_clean":"2023-06-10T18:51:49.624158+0000","last_became_active":"2023-06-10T18:51:47.980879+0000","last_became_peered":"2023-06-10T18:51:47.980879+0000","last_unstale":"2023-06-10T18:51:49.624158+0000","last_undegraded":"2023-06-10T18:51:49.624158+0000","last_fullsized":"2023-06-10T18:51:49.624158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:51:46.959317+0000","last_clean_scrub_stamp":"2023-06-10T18:51:46.959317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:48:33.095897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":120,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-06-10T18:51:49.383735+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:51:49.383735+0000","last_peered":"2023-06-10T18:51:49.383735+0000","last_clean":"2023-06-10T18:51:49.383735+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:51:49.383735+0000","last_undegraded":"2023-06-10T18:51:49.383735+0000","last_fullsized":"2023-06-10T18:51:49.383735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:55:11.529955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28528,"kb_used_data":1576,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343312,"statfs":{"total":96636764160,"available":96607551488,"internally_reserved":0,"allocated":1613824,"data_stored":1073447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.53900000000000003},"min":{"1min":0.39000000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.74099999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.53000000000000003},"min":{"1min":0.48199999999999998,"5min":0.34899999999999998,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.88100000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.38,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60899999999999999,"15min":0.57099999999999995},"min":{"1min":0.443,"5min":0.39400000000000002,"15min":0.34799999999999998},"max":{"1min":0.77100000000000002,"5min":0.80100000000000005,"15min":0.81200000000000006},"last":0.80600000000000005}]},{"osd":2,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55500000000000005,"15min":0.498},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.42199999999999999,"5min":0.33900000000000002,"15min":0.27300000000000002},"max":{"1min":0.77900000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67800000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56899999999999995,"15min":0.51600000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.72899999999999998,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.55800000000000005},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.32700000000000001},"max":{"1min":0.80100000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.78100000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.35099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61499999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.42899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68899999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.83799999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69899999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.311},"max":{"1min":0.65600000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.754}]},{"osd":6,"last update":"Sat Jun 10 18:51:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.40300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.38200000000000001,"5min":0.373,"15min":0.32600000000000001},"max":{"1min":0.72599999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.71599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":1024000,"data_stored":482684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.52200000000000002},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.81100000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55000000000000004,"15min":0.50800000000000001},"min":{"1min":0.47299999999999998,"5min":0.32600000000000001,"15min":0.24399999999999999},"max":{"1min":0.69499999999999995,"5min":0.746,"15min":0.78700000000000003},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.52400000000000002},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.67100000000000004,"5min":0.752,"15min":0.80600000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55900000000000005,"15min":0.80500000000000005},"min":{"1min":0.38500000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54800000000000004,"15min":0.71699999999999997},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.442,"5min":0.374,"15min":0.316},"max":{"1min":0.72099999999999997,"5min":0.76300000000000001,"15min":1.252},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.46200000000000002,"5min":0.371,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.745,"15min":0.82899999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69099999999999995,"5min":0.746,"15min":0.77500000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.46600000000000003,"15min":0.51000000000000001},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.59699999999999998,"5min":0.80700000000000005,"15min":0.82999999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68500000000000005,"5min":0.78600000000000003,"15min":0.89600000000000002},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46600000000000003,"15min":0.50900000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28799999999999998},"max":{"1min":0.54800000000000004,"5min":0.72699999999999998,"15min":0.86699999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.504,"15min":0.54500000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.628,"5min":0.76700000000000002,"15min":0.79100000000000004},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64000000000000001,"5min":0.78000000000000003,"15min":0.92700000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.79700000000000004,"15min":0.81399999999999995},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":1024000,"data_stored":482184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35199999999999998,"15min":0.34100000000000003},"min":{"1min":0.27200000000000002,"5min":0.221,"15min":0.19},"max":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.753},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.59599999999999997},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36599999999999999,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.23599999999999999,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.215},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.38700000000000001},"min":{"1min":0.313,"5min":0.215,"15min":0.215},"max":{"1min":0.59199999999999997,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.24299999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42899999999999999,"15min":0.40799999999999997},"min":{"1min":0.34499999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.42499999999999999,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.501,"5min":0.378,"15min":0.378},"max":{"1min":0.753,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.441},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.53800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.38200000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.52500000000000002,"5min":0.36499999999999999,"15min":0.33500000000000002},"max":{"1min":0.78000000000000003,"5min":0.89900000000000002,"15min":0.93000000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.53400000000000003,"5min":0.42199999999999999,"15min":0.39500000000000002},"max":{"1min":0.83299999999999996,"5min":0.871,"15min":1.008},"last":0.46600000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60399999999999998,"15min":0.58499999999999996},"min":{"1min":0.501,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.41999999999999998,"15min":0.40000000000000002},"max":{"1min":0.79500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.41399999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.56699999999999995,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76800000000000002,"5min":0.85899999999999999,"15min":0.95099999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.439,"5min":0.434,"15min":0.434},"max":{"1min":0.85499999999999998,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.48099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574947,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1609728,"data_stored":1072392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34899999999999998},"min":{"1min":0.26200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.46500000000000002,"5min":0.625,"15min":0.72699999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.36699999999999999,"15min":0.35999999999999999},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.186},"max":{"1min":0.45200000000000001,"5min":0.56599999999999995,"15min":0.88900000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.38100000000000001},"min":{"1min":0.29999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47999999999999998,"5min":0.65500000000000003,"15min":1.405},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":0.496,"5min":0.51500000000000001,"15min":0.95499999999999996},"last":0.26900000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38300000000000001,"15min":0.378},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.439,"5min":0.70999999999999996,"15min":0.93100000000000005},"last":0.249},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42399999999999999,"15min":0.40400000000000003},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.25900000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.376,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67900000000000005,"5min":0.88100000000000001,"15min":1.0489999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.53700000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68100000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.503,"5min":0.39000000000000001,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.90800000000000003,"15min":0.999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.44600000000000001,"5min":0.38300000000000001,"15min":0.35399999999999998},"max":{"1min":0.71199999999999997,"5min":0.88400000000000001,"15min":0.98199999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.498,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":1.363},"last":0.434}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.48099999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80700000000000005,"5min":0.93500000000000005,"15min":1.034},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.38900000000000001},"max":{"1min":0.76400000000000001,"5min":0.86099999999999999,"15min":1.0149999999999999},"last":0.50700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":1019904,"data_stored":483591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35399999999999998,"15min":0.33700000000000002},"min":{"1min":0.26900000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.52600000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.36199999999999999},"min":{"1min":0.30299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55700000000000005,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.251}]},{"osd":1,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39100000000000001,"15min":0.35999999999999999},"min":{"1min":0.32800000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.36599999999999999},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.436,"15min":0.39500000000000002},"min":{"1min":0.38400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57999999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64200000000000002,"15min":0.59099999999999997},"min":{"1min":0.51400000000000001,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":0.77000000000000002,"5min":5.452,"15min":5.452},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.47099999999999997,"5min":0.38,"15min":0.373},"max":{"1min":0.89800000000000002,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.47099999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90900000000000003,"5min":1.194,"15min":1.194},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65400000000000003,"15min":0.628},"min":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.38400000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54900000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.60999999999999999},"min":{"1min":0.49399999999999999,"5min":0.36499999999999999,"15min":0.34599999999999997},"max":{"1min":0.88400000000000001,"5min":5.4640000000000004,"15min":5.4640000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.60899999999999999},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":0.875,"15min":0.92600000000000005},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:51:08 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.64000000000000001},"min":{"1min":0.503,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92600000000000005,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65600000000000003,"15min":0.62},"min":{"1min":0.57799999999999996,"5min":0.38300000000000001,"15min":0.37},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574947,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":1028096,"data_stored":484573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.373},"min":{"1min":0.26000000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.36599999999999999,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.22900000000000001,"15min":0.219},"max":{"1min":0.505,"5min":0.56999999999999995,"15min":1.8600000000000001},"last":0.307}]},{"osd":1,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.371,"15min":0.39000000000000001},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":0.99299999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.40100000000000002,"15min":0.42799999999999999},"min":{"1min":0.309,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53800000000000003,"5min":0.61499999999999999,"15min":1.3},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.41899999999999998},"min":{"1min":0.32600000000000001,"5min":0.28100000000000003,"15min":0.26900000000000002},"max":{"1min":0.50800000000000001,"5min":0.58899999999999997,"15min":1.109},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.46899999999999997},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":1.599},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.59699999999999998,"15min":0.623},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.378},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.925},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40000000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":1.0860000000000001},"last":0.38400000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61099999999999999,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.40699999999999997,"15min":0.40600000000000003},"max":{"1min":0.86899999999999999,"5min":0.90100000000000002,"15min":1.0589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61199999999999999,"15min":0.72699999999999998},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80400000000000005,"5min":0.97599999999999998,"15min":3.802},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.60799999999999998,"15min":0.72599999999999998},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":3.8180000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.03},"last":0.40200000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":1.2769999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.627,"15min":0.64700000000000002},"min":{"1min":0.48199999999999998,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":1.8939999999999999},"last":0.498}]}]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":1011712,"data_stored":481990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54400000000000004,"15min":0.51500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28299999999999997,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82599999999999996,"15min":1.119},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.74099999999999999,"15min":1.1639999999999999},"last":0.54700000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55500000000000005,"15min":0.51000000000000001},"min":{"1min":0.377,"5min":0.36499999999999999,"15min":0.27100000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.77800000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56399999999999995,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.51700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68500000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74399999999999999,"5min":0.80100000000000005,"15min":1.0229999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54700000000000004,"15min":0.52600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.79800000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":1.042},"last":0.41699999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46999999999999997,"15min":0.495},"min":{"1min":0.35799999999999998,"5min":0.29799999999999999,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":0.752,"15min":1.069},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71999999999999997,"5min":0.84099999999999997,"15min":0.88300000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48499999999999999,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":1.1910000000000001},"last":2.1579999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.36199999999999999,"5min":0.35599999999999998,"15min":0.32100000000000001},"max":{"1min":0.77000000000000002,"5min":0.78000000000000003,"15min":1.3169999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:51:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.78100000000000003,"5min":0.81499999999999995,"15min":0.85899999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.755,"5min":0.81599999999999995,"15min":1.1459999999999999},"last":0.69999999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":1032192,"data_stored":484595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.52800000000000002},"min":{"1min":0.46600000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.72099999999999997,"5min":0.752,"15min":0.76800000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.505},"min":{"1min":0.437,"5min":0.34399999999999997,"15min":0.32300000000000001},"max":{"1min":0.70899999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.51600000000000001},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.68300000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.52100000000000002},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53100000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54500000000000004,"15min":0.51600000000000001},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.85099999999999998},"last":0.495},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.53100000000000003},"min":{"1min":0.45700000000000002,"5min":0.38500000000000001,"15min":0.34999999999999998},"max":{"1min":0.68300000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.32400000000000001},"max":{"1min":0.69299999999999995,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57999999999999996,"15min":0.55100000000000005},"min":{"1min":0.52000000000000002,"5min":0.38800000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.76800000000000002},"last":0.435}]},{"osd":4,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.39100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67400000000000004,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73799999999999999,"5min":0.879,"15min":0.879},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38,"15min":0.30299999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:51:52.846 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:51:52.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:51:53.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.998+0000 7f868a1a7700 1 Processor -- start 2023-06-10T18:51:53.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.998+0000 7f868a1a7700 1 -- start start 2023-06-10T18:51:53.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f868a1a7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86840b1780 0x7f868406cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f868a1a7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f868406ea70 0x7f868406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f868a1a7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840ae7e0 con 0x7f868415c6f0 2023-06-10T18:51:53.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f868a1a7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868406df90 con 0x7f868406ea70 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f868a1a7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868406e270 con 0x7f86840b1780 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f8683fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f868406ea70 0x7f868406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f86837fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f868415c6f0 0x7f86840b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58740/0 (socket says 172.21.15.104:58740) 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:52.999+0000 7f86837fe700 1 -- 172.21.15.104:0/356692302 learned_addr learned my addr 172.21.15.104:0/356692302 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8682ffd700 1 --2- 172.21.15.104:0/356692302 >> v2:172.21.15.114:3300/0 conn(0x7f86840b1780 0x7f868406cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8683fff700 1 -- 172.21.15.104:0/356692302 >> v2:172.21.15.114:3300/0 conn(0x7f86840b1780 msgr2=0x7f868406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:53.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8683fff700 1 --2- 172.21.15.104:0/356692302 >> v2:172.21.15.114:3300/0 conn(0x7f86840b1780 0x7f868406cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8683fff700 1 -- 172.21.15.104:0/356692302 >> v1:172.21.15.104:6789/0 conn(0x7f868415c6f0 legacy=0x7f86840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8683fff700 1 -- 172.21.15.104:0/356692302 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868406e550 con 0x7f868406ea70 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f86827fc700 1 -- 172.21.15.104:0/356692302 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233990082 0 0) 0x7f86840ae7e0 con 0x7f868415c6f0 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f8683fff700 1 --2- 172.21.15.104:0/356692302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868406ea70 0x7f868406d320 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f8684070990 tx=0x7f867c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee984591e3a19914 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f86827fc700 1 -- 172.21.15.104:0/356692302 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00e070 con 0x7f868406ea70 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f86827fc700 1 -- 172.21.15.104:0/356692302 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00bbd0 con 0x7f868406ea70 2023-06-10T18:51:53.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f86827fc700 1 -- 172.21.15.104:0/356692302 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c012430 con 0x7f868406ea70 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f868a1a7700 1 -- 172.21.15.104:0/356692302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868406ea70 msgr2=0x7f868406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f868a1a7700 1 --2- 172.21.15.104:0/356692302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868406ea70 0x7f868406d320 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f8684070990 tx=0x7f867c00b330 comp rx=0 tx=0).stop 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f868a1a7700 1 -- 172.21.15.104:0/356692302 shutdown_connections 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f868a1a7700 1 --2- 172.21.15.104:0/356692302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868406ea70 0x7f868406d320 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.000+0000 7f868a1a7700 1 --2- 172.21.15.104:0/356692302 >> v2:172.21.15.114:3300/0 conn(0x7f86840b1780 0x7f868406cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.001+0000 7f868a1a7700 1 -- 172.21.15.104:0/356692302 >> 172.21.15.104:0/356692302 conn(0x7f868406b8a0 msgr2=0x7f86840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.001+0000 7f868a1a7700 1 -- 172.21.15.104:0/356692302 shutdown_connections 2023-06-10T18:51:53.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.001+0000 7f868a1a7700 1 -- 172.21.15.104:0/356692302 wait complete. 2023-06-10T18:51:53.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.002+0000 7f868a1a7700 1 Processor -- start 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.002+0000 7f868a1a7700 1 -- start start 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 0x7f86840c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86840b1780 0x7f86840b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840c9c30 con 0x7f868415c6f0 2023-06-10T18:51:53.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840c56f0 con 0x7f86840b1780 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840c59d0 con 0x7f868406ea70 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60922/0 (socket says 172.21.15.104:60922) 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 -- 172.21.15.104:0/474905012 learned_addr learned my addr 172.21.15.104:0/474905012 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8682ffd700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86840b1780 0x7f86840b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f86837fe700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 0x7f86840c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 -- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 msgr2=0x7f86840c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 0x7f86840c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 -- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86840b1780 msgr2=0x7f86840b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86840b1780 0x7f86840b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 -- 172.21.15.104:0/474905012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c009a70 con 0x7f868415c6f0 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f86837fe700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 0x7f86840c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f8683fff700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f867c013000 tx=0x7f867c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c020070 con 0x7f868415c6f0 2023-06-10T18:51:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c0098f0 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86840c5cb0 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c012430 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.003+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86840c5f90 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.004+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86840c9e00 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.005+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f867c024070 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.005+0000 7f867bfff700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 0x7f866406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.005+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f867c08c5b0 con 0x7f868415c6f0 2023-06-10T18:51:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.005+0000 7f86837fe700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 0x7f866406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.006+0000 7f86837fe700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 0x7f866406d190 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f866c0017b0 tx=0x7f866c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.009+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f86840c9e00 con 0x7f868415c6f0 2023-06-10T18:51:53.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.271+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f86840c9e00 con 0x7f868415c6f0 2023-06-10T18:51:53.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.275+0000 7f867bfff700 1 -- 172.21.15.104:0/474905012 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f86840c9e00 con 0x7f868415c6f0 2023-06-10T18:51:53.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 msgr2=0x7f866406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 0x7f866406d190 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f866c0017b0 tx=0x7f866c009040 comp rx=0 tx=0).stop 2023-06-10T18:51:53.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 msgr2=0x7f86840b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f867c013000 tx=0x7f867c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 shutdown_connections 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f868415c6f0 0x7f86840b2960 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f868406ea70 0x7f86840c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f866406ace0 0x7f866406d190 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 --2- 172.21.15.104:0/474905012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86840b1780 0x7f86840b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 >> 172.21.15.104:0/474905012 conn(0x7f868406b8a0 msgr2=0x7f86840ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 shutdown_connections 2023-06-10T18:51:53.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.277+0000 7f868a1a7700 1 -- 172.21.15.104:0/474905012 wait complete. 2023-06-10T18:51:53.285 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:51:53.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:51:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.410+0000 7f62c7380700 1 Processor -- start 2023-06-10T18:51:53.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.410+0000 7f62c7380700 1 -- start start 2023-06-10T18:51:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c7380700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62c006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c7380700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c006d290 con 0x7f62c006e4b0 2023-06-10T18:51:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c7380700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00ca2d0 con 0x7f62c006e0e0 2023-06-10T18:51:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c7380700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0071640 con 0x7f62c00cf100 2023-06-10T18:51:53.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c491b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f62c006e4b0 0x7f62c00ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58754/0 (socket says 172.21.15.104:58754) 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.411+0000 7f62c491b700 1 -- 172.21.15.104:0/2581807104 learned_addr learned my addr 172.21.15.104:0/2581807104 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62c006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310900171 0 0) 0x7f62c006d290 con 0x7f62c006e4b0 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac0035b0 con 0x7f62c006e4b0 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949935702 0 0) 0x7f62c00ca2d0 con 0x7f62c006e0e0 2023-06-10T18:51:53.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c006d290 con 0x7f62c006e0e0 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876414110 0 0) 0x7f62ac0035b0 con 0x7f62c006e4b0 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 msgr2=0x7f62c006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62c006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 >> v1:172.21.15.114:6789/0 conn(0x7f62c006e0e0 legacy=0x7f62c00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c006d4c0 con 0x7f62c006e4b0 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62c006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f62b0001670 con 0x7f62c006e4b0 2023-06-10T18:51:53.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 >> v1:172.21.15.104:6789/0 conn(0x7f62c006e4b0 legacy=0x7f62c00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62ac003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac004100 0x7f62ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62ac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0071640 con 0x7f62ac004d90 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00bb50 con 0x7f62ac004100 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00bd10 con 0x7f62c00cf100 2023-06-10T18:51:53.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62ac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c511c700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62ac003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c491b700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac004100 0x7f62ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 -- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 msgr2=0x7f62ac003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cf100 0x7f62ac003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 -- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac004100 msgr2=0x7f62ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac004100 0x7f62ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.412+0000 7f62c591d700 1 -- 172.21.15.104:0/2581807104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b8009a70 con 0x7f62ac004d90 2023-06-10T18:51:53.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c591d700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62ac005200 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f62c00cf560 tx=0x7f62ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91ea21aefd21c01 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b800edb0 con 0x7f62ac004d90 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b800f040 con 0x7f62ac004d90 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62b7fff700 1 -- 172.21.15.104:0/2581807104 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8003c60 con 0x7f62ac004d90 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 -- 172.21.15.104:0/2581807104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 msgr2=0x7f62ac005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 --2- 172.21.15.104:0/2581807104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62ac005200 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f62c00cf560 tx=0x7f62ac005880 comp rx=0 tx=0).stop 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c591d700 1 -- 172.21.15.104:0/2581807104 reap_dead start 2023-06-10T18:51:53.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 -- 172.21.15.104:0/2581807104 shutdown_connections 2023-06-10T18:51:53.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 -- 172.21.15.104:0/2581807104 >> 172.21.15.104:0/2581807104 conn(0x7f62c006b8a0 msgr2=0x7f62c00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:53.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 -- 172.21.15.104:0/2581807104 shutdown_connections 2023-06-10T18:51:53.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.413+0000 7f62c7380700 1 -- 172.21.15.104:0/2581807104 wait complete. 2023-06-10T18:51:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 Processor -- start 2023-06-10T18:51:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 -- start start 2023-06-10T18:51:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c00b1580 0x7f62c0159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cb030 0x7f62c00cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00cb9e0 con 0x7f62ac004d90 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00af210 con 0x7f62c00b1580 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c7380700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0159750 con 0x7f62c00cb030 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c511c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c511c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60938/0 (socket says 172.21.15.104:60938) 2023-06-10T18:51:53.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.415+0000 7f62c511c700 1 -- 172.21.15.104:0/1376374615 learned_addr learned my addr 172.21.15.104:0/1376374615 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c591d700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cb030 0x7f62c00cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c491b700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c00b1580 0x7f62c0159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 -- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cb030 msgr2=0x7f62c00cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cb030 0x7f62c00cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 -- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c00b1580 msgr2=0x7f62c0159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c00b1580 0x7f62c0159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 -- 172.21.15.104:0/1376374615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00af440 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c511c700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00f8e0 tx=0x7f62bc00fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00fe80 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc0052e0 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c00af720 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc0036d0 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.416+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c00afce0 con 0x7f62ac004d90 2023-06-10T18:51:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.417+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c0074050 con 0x7f62ac004d90 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.417+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f62bc003830 con 0x7f62ac004d90 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.418+0000 7f62b5ffb700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 0x7f62a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.418+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f62bc089640 con 0x7f62ac004d90 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.418+0000 7f62c491b700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 0x7f62a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.419+0000 7f62c491b700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 0x7f62a406d190 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f62c00b07a0 tx=0x7f62b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:53.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.422+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f62bc058020 con 0x7f62ac004d90 2023-06-10T18:51:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.656+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62c00cdd60 con 0x7f62ac004d90 2023-06-10T18:51:53.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.656+0000 7f62b5ffb700 1 -- 172.21.15.104:0/1376374615 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4682 (secure 0 0 0) 0x7f62bc057bb0 con 0x7f62ac004d90 2023-06-10T18:51:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:53.663 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:51:51.378743+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/414069859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":414069859},{"type":"v1","addr":"172.21.15.114:6836","nonce":414069859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:51:53.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 msgr2=0x7f62a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 0x7f62a406d190 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f62c00b07a0 tx=0x7f62b0009040 comp rx=0 tx=0).stop 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 msgr2=0x7f62c00aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00f8e0 tx=0x7f62bc00fca0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 shutdown_connections 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62c00cb030 0x7f62c00cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62ac004d90 0x7f62c00aecd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f62a406ace0 0x7f62a406d190 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 --2- 172.21.15.104:0/1376374615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c00b1580 0x7f62c0159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:53.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 >> 172.21.15.104:0/1376374615 conn(0x7f62c006b8a0 msgr2=0x7f62c015bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:53.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 shutdown_connections 2023-06-10T18:51:53.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:53.659+0000 7f62c7380700 1 -- 172.21.15.104:0/1376374615 wait complete. 2023-06-10T18:51:53.666 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 40 2023-06-10T18:51:54.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:51:48.370220+0000', 'modified': '2023-06-10T18:51:51.378743+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7158}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7158': {'gid': 7158, 'name': 'd', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/414069859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 414069859}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 414069859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:51:54.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:51:54.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:51:54.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 Processor -- start 2023-06-10T18:51:54.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 -- start start 2023-06-10T18:51:54.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f13f80d0530 0x7f13f80cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 0x7f13f80d2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f80b1c40 con 0x7f13f806c260 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80b1680 con 0x7f13f80d0530 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.196+0000 7f13fd6e3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80b1960 con 0x7f13f80b1e80 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f13f80d0530 0x7f13f80cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f67fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f13f806c260 0x7f13f80d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58776/0 (socket says 172.21.15.104:58776) 2023-06-10T18:51:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f67fc700 1 -- 172.21.15.104:0/3747018062 learned_addr learned my addr 172.21.15.104:0/3747018062 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:54.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3747018062 >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 msgr2=0x7f13f80d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f77fe700 1 --2- 172.21.15.104:0/3747018062 >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 0x7f13f80d2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/3747018062 >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 0x7f13f80d2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f5ffb700 1 -- 172.21.15.104:0/3747018062 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404093395 0 0) 0x7f13f80b1c40 con 0x7f13f806c260 2023-06-10T18:51:54.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3747018062 >> v1:172.21.15.104:6789/0 conn(0x7f13f806c260 legacy=0x7f13f80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3747018062 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f806b780 con 0x7f13f80d0530 2023-06-10T18:51:54.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f77fe700 1 --2- 172.21.15.104:0/3747018062 >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 0x7f13f80d2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:54.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/3747018062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80d0530 0x7f13f80cfd80 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f13f80c99c0 tx=0x7f13ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c1188a39616682 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f5ffb700 1 -- 172.21.15.104:0/3747018062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec011830 con 0x7f13f80d0530 2023-06-10T18:51:54.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f5ffb700 1 -- 172.21.15.104:0/3747018062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec011990 con 0x7f13f80d0530 2023-06-10T18:51:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.197+0000 7f13f5ffb700 1 -- 172.21.15.104:0/3747018062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec011d20 con 0x7f13f80d0530 2023-06-10T18:51:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 -- 172.21.15.104:0/3747018062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80d0530 msgr2=0x7f13f80cfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/3747018062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80d0530 0x7f13f80cfd80 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f13f80c99c0 tx=0x7f13ec009b20 comp rx=0 tx=0).stop 2023-06-10T18:51:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 -- 172.21.15.104:0/3747018062 shutdown_connections 2023-06-10T18:51:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/3747018062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80d0530 0x7f13f80cfd80 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/3747018062 >> v2:172.21.15.114:3300/0 conn(0x7f13f80b1e80 0x7f13f80d2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 -- 172.21.15.104:0/3747018062 >> 172.21.15.104:0/3747018062 conn(0x7f13f8071ec0 msgr2=0x7f13f809bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 -- 172.21.15.104:0/3747018062 shutdown_connections 2023-06-10T18:51:54.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.198+0000 7f13fd6e3700 1 -- 172.21.15.104:0/3747018062 wait complete. 2023-06-10T18:51:54.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.199+0000 7f13fd6e3700 1 Processor -- start 2023-06-10T18:51:54.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.199+0000 7f13fd6e3700 1 -- start start 2023-06-10T18:51:54.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13f806c260 0x7f13f81e7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13f80d0530 0x7f13f81e1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80c8750 con 0x7f13f80b1e80 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80c8a30 con 0x7f13f806c260 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13fd6e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80c8d10 con 0x7f13f80d0530 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60954/0 (socket says 172.21.15.104:60954) 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 -- 172.21.15.104:0/970803847 learned_addr learned my addr 172.21.15.104:0/970803847 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13f806c260 0x7f13f81e7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f77fe700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13f80d0530 0x7f13f81e1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 -- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13f80d0530 msgr2=0x7f13f81e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13f80d0530 0x7f13f81e1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 -- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13f806c260 msgr2=0x7f13f81e7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13f806c260 0x7f13f81e7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 -- 172.21.15.104:0/970803847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec0094a0 con 0x7f13f80b1e80 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13f67fc700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c520 tx=0x7f13e800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13df7fe700 1 -- 172.21.15.104:0/970803847 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8004c80 con 0x7f13f80b1e80 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.200+0000 7f13df7fe700 1 -- 172.21.15.104:0/970803847 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e8004de0 con 0x7f13f80b1e80 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.201+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f80c8ff0 con 0x7f13f80b1e80 2023-06-10T18:51:54.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.201+0000 7f13df7fe700 1 -- 172.21.15.104:0/970803847 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e80072d0 con 0x7f13f80b1e80 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.201+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f81e7f70 con 0x7f13f80b1e80 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.202+0000 7f13df7fe700 1 -- 172.21.15.104:0/970803847 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f13e8007430 con 0x7f13f80b1e80 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.202+0000 7f13df7fe700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 0x7f13e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.202+0000 7f13df7fe700 1 -- 172.21.15.104:0/970803847 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f13e80890d0 con 0x7f13f80b1e80 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.203+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 0x7f13e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.203+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 0x7f13e406d190 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0090b0 tx=0x7f13ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.204+0000 7f1409868b80 1 Processor -- start 2023-06-10T18:51:54.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.204+0000 7f1409868b80 1 -- start start 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b97820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978b9b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978a75c60 con 0x563978b70f50 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978ac89f0 con 0x563978b98d80 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x563978ac1e80 con 0x563978b6e740 2023-06-10T18:51:54.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60960/0 (socket says 172.21.15.104:60960) 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3868723497 learned_addr learned my addr 172.21.15.104:0/3868723497 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f77fe700 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978b9b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f67fc700 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b97820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 msgr2=0x563978b9b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978b9b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 msgr2=0x563978b97820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b97820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.205+0000 7f13f6ffd700 1 -- 172.21.15.104:0/3868723497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639789435e0 con 0x563978b70f50 2023-06-10T18:51:54.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009f20 tx=0x7f13ec010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da87c6e42609dc33 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f13dcff9700 1 -- 172.21.15.104:0/3868723497 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec02c070 con 0x563978b70f50 2023-06-10T18:51:54.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f13dcff9700 1 -- 172.21.15.104:0/3868723497 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0037e0 con 0x563978b70f50 2023-06-10T18:51:54.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f13dcff9700 1 -- 172.21.15.104:0/3868723497 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec020070 con 0x563978b70f50 2023-06-10T18:51:54.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 -- 172.21.15.104:0/3868723497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 msgr2=0x563978b71320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009f20 tx=0x7f13ec010000 comp rx=0 tx=0).stop 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 -- 172.21.15.104:0/3868723497 shutdown_connections 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978b9b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b70f50 0x563978b71320 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 --2- 172.21.15.104:0/3868723497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b97820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 -- 172.21.15.104:0/3868723497 >> 172.21.15.104:0/3868723497 conn(0x563978b65fc0 msgr2=0x563978bb61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.206+0000 7f1409868b80 1 -- 172.21.15.104:0/3868723497 shutdown_connections 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.207+0000 7f1409868b80 1 -- 172.21.15.104:0/3868723497 wait complete. 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.207+0000 7f1409868b80 1 -- start start 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.207+0000 7f1409868b80 1 Processor -- start 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b918f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978a61a70 con 0x563978b98d80 2023-06-10T18:51:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978a61770 con 0x563978b70f50 2023-06-10T18:51:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639789458d0 con 0x563978b6e740 2023-06-10T18:51:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f1409868b80 10 client.0 _ll_register_callbacks cb 0x563978a67210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:51:54.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60970/0 (socket says 172.21.15.104:60970) 2023-06-10T18:51:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 -- 172.21.15.104:0/836097177 learned_addr learned my addr 172.21.15.104:0/836097177 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f67fc700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 -- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 msgr2=0x563978bb8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 -- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 msgr2=0x563978b918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 -- 172.21.15.104:0/836097177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f806b780 con 0x563978b98d80 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13f77fe700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f13f000fbf0 tx=0x7f13f000ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4478a68e881c5d86 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.208+0000 7f13d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0014070 con 0x563978b98d80 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f13f000dab0 con 0x563978b98d80 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563978b63780 con 0x563978b98d80 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13f000dc70 con 0x563978b98d80 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 10 client.7302 fetch_fsmap learned FSMap version 40 2023-06-10T18:51:54.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x5639788efd90 con 0x563978b98d80 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563978b63780 con 0x563978b98d80 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 40) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f13f0018410 con 0x563978b98d80 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 10 client.7302 fetch_fsmap finished waiting for FSMap version 40 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 10 client.7302 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 20 client.7302 populate_metadata read hostname 'smithi104' 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978bb9fe0 0x563978b997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 4 client.7302 mds_command: new command op to 7158 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:51:54.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 --> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563978abae60 con 0x563978bb9fe0 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.209+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978bb9fe0 0x563978b997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f13d5ffb700 10 client.7302 ms_handle_connect on v2:172.21.15.114:6834/414069859 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978bb9fe0 0x563978b997c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cef52c9b48c6042 server_cookie=ad68556303dd16ae in_seq=0 out_seq=0 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f13d5ffb700 1 -- 172.21.15.104:0/836097177 <== mds.0 v2:172.21.15.114:6834/414069859 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563978abae60 con 0x563978bb9fe0 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f13d5ffb700 10 client.7302 handle_command_reply: tid=0 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f1409868b80 1 client.7302 shutdown 2023-06-10T18:51:54.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 msgr2=0x563978bb39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.210+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f13f000fbf0 tx=0x7f13f000ff00 comp rx=0 tx=0).stop 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 shutdown_connections 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978bb9fe0 0x563978b997c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978bb39b0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f13d5ffb700 0 client.7302 ms_handle_reset on v2:172.21.15.114:6834/414069859 2023-06-10T18:51:54.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 --2- 172.21.15.104:0/836097177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 >> 172.21.15.104:0/836097177 conn(0x563978b65fc0 msgr2=0x563978b93a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 shutdown_connections 2023-06-10T18:51:54.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 1 -- 172.21.15.104:0/836097177 wait complete. 2023-06-10T18:51:54.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.211+0000 7f1409868b80 20 client.7302 trim_cache size 0 max 16384 2023-06-10T18:51:54.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.228+0000 7f1409868b80 1 Processor -- start 2023-06-10T18:51:54.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.228+0000 7f1409868b80 1 -- start start 2023-06-10T18:51:54.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b6b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978b6b120 con 0x563978b98d80 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978b982c0 con 0x563978b70f50 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x563978b72690 con 0x563978b6e740 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60978/0 (socket says 172.21.15.104:60978) 2023-06-10T18:51:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 -- 172.21.15.104:0/2748372749 learned_addr learned my addr 172.21.15.104:0/2748372749 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f67fc700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b6b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 -- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 msgr2=0x563978bb5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 -- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 msgr2=0x563978b6b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b6b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 -- 172.21.15.104:0/2748372749 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639788d8c10 con 0x563978b98d80 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f67fc700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b6b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13f77fe700 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d700 tx=0x7f13f0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07a3cfdb08eafb3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13dd7fa700 1 -- 172.21.15.104:0/2748372749 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0018410 con 0x563978b98d80 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13dd7fa700 1 -- 172.21.15.104:0/2748372749 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0018bb0 con 0x563978b98d80 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.229+0000 7f13dd7fa700 1 -- 172.21.15.104:0/2748372749 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0018590 con 0x563978b98d80 2023-06-10T18:51:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 -- 172.21.15.104:0/2748372749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 msgr2=0x563978b6bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d700 tx=0x7f13f0015000 comp rx=0 tx=0).stop 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 -- 172.21.15.104:0/2748372749 shutdown_connections 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b98d80 0x563978b6bbe0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b6e740 0x563978bb5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 --2- 172.21.15.104:0/2748372749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b70f50 0x563978b6b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 -- 172.21.15.104:0/2748372749 >> 172.21.15.104:0/2748372749 conn(0x563978b65fc0 msgr2=0x563978bb39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 -- 172.21.15.104:0/2748372749 shutdown_connections 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.230+0000 7f1409868b80 1 -- 172.21.15.104:0/2748372749 wait complete. 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 -- start start 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 Processor -- start 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b70f50 0x563978b6a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b6a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978b6ab10 con 0x563978b6e740 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563978bd6910 con 0x563978b98d80 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.231+0000 7f1409868b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x563978bd50f0 con 0x563978b70f50 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 10 client.0 _ll_register_callbacks cb 0x563978a67210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60980/0 (socket says 172.21.15.104:60980) 2023-06-10T18:51:54.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 -- 172.21.15.104:0/544125593 learned_addr learned my addr 172.21.15.104:0/544125593 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f67fc700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b70f50 0x563978b6a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f77fe700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b6a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 -- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b70f50 msgr2=0x563978b6a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b70f50 0x563978b6a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 -- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 msgr2=0x563978b6a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:54.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b6a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 -- 172.21.15.104:0/544125593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639788efd90 con 0x563978b6e740 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13f6ffd700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f13ec034040 tx=0x7f13ec00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c02f5e61cbc740 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec02c040 con 0x563978b6e740 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f13ec03ea40 con 0x563978b6e740 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13ec0037e0 con 0x563978b6e740 2023-06-10T18:51:54.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13ec03ec00 con 0x563978b6e740 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 10 client.7308 fetch_fsmap learned FSMap version 40 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x563978916200 con 0x563978b6e740 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0037e0 con 0x563978b6e740 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 40) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f13ec03e040 con 0x563978b6e740 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 10 client.7308 fetch_fsmap finished waiting for FSMap version 40 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 10 client.7308 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.232+0000 7f1409868b80 20 client.7308 populate_metadata read hostname 'smithi104' 2023-06-10T18:51:54.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978b999c0 0x563978b99d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 4 client.7308 mds_command: new command op to 7158 tid=0[{"prefix": "status"}] 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 --> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563978a7d000 con 0x563978b999c0 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13f67fc700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978b999c0 0x563978b99d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13d5ffb700 10 client.7308 ms_handle_connect on v2:172.21.15.114:6834/414069859 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13f67fc700 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978b999c0 0x563978b99d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad5196ae59b398ea server_cookie=4679adaf230e1079 in_seq=0 out_seq=0 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13d5ffb700 1 -- 172.21.15.104:0/544125593 <== mds.0 v2:172.21.15.114:6834/414069859 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x563978a7d000 con 0x563978b999c0 2023-06-10T18:51:54.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13d5ffb700 10 client.7308 handle_command_reply: tid=0 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 client.7308 shutdown 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 msgr2=0x563978b92150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f13ec034040 tx=0x7f13ec00e000 comp rx=0 tx=0).stop 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 shutdown_connections 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:6834/414069859,v1:172.21.15.114:6836/414069859] conn(0x563978b999c0 0x563978b99d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563978b98d80 0x563978b6a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563978b6e740 0x563978b92150 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 --2- 172.21.15.104:0/544125593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563978b70f50 0x563978b6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 >> 172.21.15.104:0/544125593 conn(0x563978b65fc0 msgr2=0x563978bb39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f13d5ffb700 0 client.7308 ms_handle_reset on v2:172.21.15.114:6834/414069859 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 shutdown_connections 2023-06-10T18:51:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.233+0000 7f1409868b80 1 -- 172.21.15.104:0/544125593 wait complete. 2023-06-10T18:51:54.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f1409868b80 20 client.7308 trim_cache size 0 max 16384 2023-06-10T18:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:51:54.254 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 7158, 2023-06-10T18:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:51:54.255 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8626111349999999, 2023-06-10T18:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 40, 2023-06-10T18:51:54.256 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 85, 2023-06-10T18:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 83, 2023-06-10T18:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.206651528 2023-06-10T18:51:54.257 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:51:54.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 msgr2=0x7f13e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 0x7f13e406d190 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0090b0 tx=0x7f13ec008030 comp rx=0 tx=0).stop 2023-06-10T18:51:54.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 msgr2=0x7f13f81e1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c520 tx=0x7f13e800c830 comp rx=0 tx=0).stop 2023-06-10T18:51:54.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 shutdown_connections 2023-06-10T18:51:54.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f13e406ace0 0x7f13e406d190 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13f806c260 0x7f13f81e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13f80d0530 0x7f13f81e1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 --2- 172.21.15.104:0/970803847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13f80b1e80 0x7f13f81e1880 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 >> 172.21.15.104:0/970803847 conn(0x7f13f8071ec0 msgr2=0x7f13f80b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 shutdown_connections 2023-06-10T18:51:54.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.234+0000 7f13fd6e3700 1 -- 172.21.15.104:0/970803847 wait complete. 2023-06-10T18:51:54.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:51:54.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.378+0000 7fd257321700 1 Processor -- start 2023-06-10T18:51:54.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.378+0000 7fd257321700 1 -- start start 2023-06-10T18:51:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd257321700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd250070a20 0x7fd25006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd257321700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25006ae00 con 0x7fd250070df0 2023-06-10T18:51:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd257321700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd250069e70 con 0x7fd250158eb0 2023-06-10T18:51:54.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd257321700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2501595f0 con 0x7fd250070a20 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2550bd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd250070a20 0x7fd25006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2548bc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd250070df0 0x7fd2501587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58816/0 (socket says 172.21.15.104:58816) 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2548bc700 1 -- 172.21.15.104:0/553806385 learned_addr learned my addr 172.21.15.104:0/553806385 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2550bd700 1 -- 172.21.15.104:0/553806385 >> v1:172.21.15.114:6789/0 conn(0x7fd250158eb0 legacy=0x7fd250159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2550bd700 1 -- 172.21.15.104:0/553806385 >> v1:172.21.15.104:6789/0 conn(0x7fd250070df0 legacy=0x7fd2501587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd247fff700 1 -- 172.21.15.104:0/553806385 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997019193 0 0) 0x7fd240001110 con 0x7fd250070df0 2023-06-10T18:51:54.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2550bd700 1 -- 172.21.15.104:0/553806385 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2501598d0 con 0x7fd250070a20 2023-06-10T18:51:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd2550bd700 1 --2- 172.21.15.104:0/553806385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070a20 0x7fd25006a8c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009840 tx=0x7fd24c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4285c49dae48b8dc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.379+0000 7fd247fff700 1 -- 172.21.15.104:0/553806385 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c011830 con 0x7fd250070a20 2023-06-10T18:51:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd247fff700 1 -- 172.21.15.104:0/553806385 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c011990 con 0x7fd250070a20 2023-06-10T18:51:54.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd247fff700 1 -- 172.21.15.104:0/553806385 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c011d20 con 0x7fd250070a20 2023-06-10T18:51:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 -- 172.21.15.104:0/553806385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070a20 msgr2=0x7fd25006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 --2- 172.21.15.104:0/553806385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070a20 0x7fd25006a8c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009840 tx=0x7fd24c009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 -- 172.21.15.104:0/553806385 shutdown_connections 2023-06-10T18:51:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 --2- 172.21.15.104:0/553806385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070a20 0x7fd25006a8c0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 -- 172.21.15.104:0/553806385 >> 172.21.15.104:0/553806385 conn(0x7fd2500b1610 msgr2=0x7fd2500ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 -- 172.21.15.104:0/553806385 shutdown_connections 2023-06-10T18:51:54.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.380+0000 7fd257321700 1 -- 172.21.15.104:0/553806385 wait complete. 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.381+0000 7fd257321700 1 Processor -- start 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.381+0000 7fd257321700 1 -- start start 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 0x7fd2500c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 0x7fd250148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500b1d40 con 0x7fd250070df0 2023-06-10T18:51:54.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500c2100 con 0x7fd250158eb0 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd257321700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2501426d0 con 0x7fd250070a20 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd2548bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd2548bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60996/0 (socket says 172.21.15.104:60996) 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd2548bc700 1 -- 172.21.15.104:0/489362007 learned_addr learned my addr 172.21.15.104:0/489362007 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd2550bd700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 0x7fd2500c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.382+0000 7fd2558be700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 0x7fd250148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 -- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 msgr2=0x7fd2500c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 0x7fd2500c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 -- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 msgr2=0x7fd250148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 0x7fd250148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 -- 172.21.15.104:0/489362007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0094a0 con 0x7fd250070df0 2023-06-10T18:51:54.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2550bd700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 0x7fd2500c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:54.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2558be700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 0x7fd250148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:54.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd2548bc700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c520 tx=0x7fd24000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240004c80 con 0x7fd250070df0 2023-06-10T18:51:54.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240004e20 con 0x7fd250070df0 2023-06-10T18:51:54.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2501429b0 con 0x7fd250070df0 2023-06-10T18:51:54.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240007310 con 0x7fd250070df0 2023-06-10T18:51:54.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.383+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd250142f70 con 0x7fd250070df0 2023-06-10T18:51:54.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.384+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd250071780 con 0x7fd250070df0 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.384+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd240007470 con 0x7fd250070df0 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.385+0000 7fd245ffb700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 0x7fd23406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.385+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fd2400891b0 con 0x7fd250070df0 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.385+0000 7fd2550bd700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 0x7fd23406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.386+0000 7fd2550bd700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 0x7fd23406d190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd24c000c00 tx=0x7fd24c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:54.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.389+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd240054190 con 0x7fd250070df0 2023-06-10T18:51:54.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.620+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd25006ae00 con 0x7fd250070df0 2023-06-10T18:51:54.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.620+0000 7fd245ffb700 1 -- 172.21.15.104:0/489362007 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd25006ae00 con 0x7fd250070df0 2023-06-10T18:51:54.626 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:51:54.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 msgr2=0x7fd23406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 0x7fd23406d190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd24c000c00 tx=0x7fd24c01c040 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 msgr2=0x7fd250147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c520 tx=0x7fd24000c8e0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 shutdown_connections 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd23406ace0 0x7fd23406d190 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250158eb0 0x7fd250148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd250070a20 0x7fd2500c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 --2- 172.21.15.104:0/489362007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd250070df0 0x7fd250147b00 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:54.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 >> 172.21.15.104:0/489362007 conn(0x7fd2500b1610 msgr2=0x7fd25006c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:54.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 shutdown_connections 2023-06-10T18:51:54.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:54.622+0000 7fd257321700 1 -- 172.21.15.104:0/489362007 wait complete. 2023-06-10T18:51:54.640 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:51:54.640 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:51:54.640 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:51:54.641 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:51:54.641 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:54.641 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:51:54.641 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:51:54.642 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:51:54.642 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:51:54.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.643 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:51:54.659 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:51:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:51:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:51:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:51:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:54.661 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41970sec preferred_lft 41970sec 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state DOWN group default qlen 1000 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:51:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:51:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:51:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:54.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.664 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:51:54.724 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:51:54.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.726 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:51:54.784 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:51:54.784 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:51:54.784 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:51:54.785 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:51:54.785 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:51:54.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.785 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:51:54.786 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:51:54.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:51:54.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:51:54.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:51:54.787 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:51:54.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.890 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:51:54.906 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:51:54.907 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:51:54.907 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:51:54.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:54.908 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:51:54.979 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:51:54.979 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:51:54.980 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:55.036 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:51:55.036 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:51:55.037 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:55.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:51:55.159 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:51:55.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:51:55.225 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:51:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:51:55.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:55.228 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:51:55.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:51:55.285 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:51:55.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:51:55.353 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:51:55.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:51:55.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:51:55.410+0000 7f069f0eb5c0 -1 init, newargv = 0x55f97b735ab0 newargc=15 2023-06-10T18:51:55.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[132382]: starting ceph client 2023-06-10T18:51:55.420 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:51:55.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[132382]: starting fuse 2023-06-10T18:51:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:51:55.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:55.422 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:51:55.478 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:51:55.479 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:51:55.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:51:55.479 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:51:55.480 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:51:55.480 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:51:55.480 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:51:55.481 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:51:55.481 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:51:55.481 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:51:55.482 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:51:55.482 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:51:55.482 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:51:55.483 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:51:55.483 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:51:55.483 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:51:55.484 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:51:55.484 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:51:55.484 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:51:55.484 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:51:55.485 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:51:55.485 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:51:55.485 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:51:55.486 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:51:55.486 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:51:55.486 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:51:55.487 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:51:55.487 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:51:55.487 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:51:55.600 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.132382.asok 2023-06-10T18:51:55.783 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.132382.asok 2023-06-10T18:51:55.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:55.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.132382.asok status 2023-06-10T18:51:55.939 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:51:55.940 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:51:55.940 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:51:55.940 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "132382", 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:51:55.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 7320, 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:51:55.942 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 7320 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1823366359 2023-06-10T18:51:55.943 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1823366359 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:51:55.944 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.7320 192.168.0.1:0/1823366359", 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/1823366359", 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 40, 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 85, 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:51:55.945 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:51:55.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:55.949 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:51:55.950 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:55.967 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:51:55.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:55.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:55.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:51:56.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:51:56.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.188+0000 7fd5a86a3700 1 Processor -- start 2023-06-10T18:51:56.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.188+0000 7fd5a86a3700 1 -- start start 2023-06-10T18:51:56.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a86a3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 0x7fd5a00ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a86a3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 0x7fd5a00cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a86a3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00d0340 con 0x7fd5a00ce390 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a86a3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a00d1660 con 0x7fd5a00cae10 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a86a3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a015cb60 con 0x7fd5a006c720 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a5c3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 0x7fd5a00ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a6c40700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 0x7fd5a00cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a643f700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd5a00ce390 0x7fd5a0160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:36458/0 (socket says 172.21.15.104:36458) 2023-06-10T18:51:56.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.189+0000 7fd5a643f700 1 -- 172.21.15.104:0/2860909440 learned_addr learned my addr 172.21.15.104:0/2860909440 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:56.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a543d700 1 -- 172.21.15.104:0/2860909440 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310910661 0 0) 0x7fd5a00d0340 con 0x7fd5a00ce390 2023-06-10T18:51:56.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a543d700 1 -- 172.21.15.104:0/2860909440 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c0035b0 con 0x7fd5a00ce390 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a5c3e700 1 -- 172.21.15.104:0/2860909440 >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 msgr2=0x7fd5a00cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a5c3e700 1 --2- 172.21.15.104:0/2860909440 >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 0x7fd5a00cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a5c3e700 1 -- 172.21.15.104:0/2860909440 >> v1:172.21.15.114:6789/0 conn(0x7fd5a00ce390 legacy=0x7fd5a0160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a5c3e700 1 -- 172.21.15.104:0/2860909440 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00ceca0 con 0x7fd5a006c720 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a6c40700 1 --2- 172.21.15.104:0/2860909440 >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 0x7fd5a00cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a5c3e700 1 --2- 172.21.15.104:0/2860909440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 0x7fd5a00ce760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009840 tx=0x7fd59c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d554ea881eb52b9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:56.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a543d700 1 -- 172.21.15.104:0/2860909440 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011830 con 0x7fd5a006c720 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a543d700 1 -- 172.21.15.104:0/2860909440 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c011990 con 0x7fd5a006c720 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a543d700 1 -- 172.21.15.104:0/2860909440 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011d00 con 0x7fd5a006c720 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a86a3700 1 -- 172.21.15.104:0/2860909440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 msgr2=0x7fd5a00ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.190+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/2860909440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 0x7fd5a00ce760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009840 tx=0x7fd59c009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 -- 172.21.15.104:0/2860909440 shutdown_connections 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/2860909440 >> v2:172.21.15.114:3300/0 conn(0x7fd5a00cae10 0x7fd5a00cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/2860909440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a006c720 0x7fd5a00ce760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 -- 172.21.15.104:0/2860909440 >> 172.21.15.104:0/2860909440 conn(0x7fd5a0069e70 msgr2=0x7fd5a006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 -- 172.21.15.104:0/2860909440 shutdown_connections 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.191+0000 7fd5a86a3700 1 -- 172.21.15.104:0/2860909440 wait complete. 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 Processor -- start 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 -- start start 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a006c720 0x7fd5a01f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a00cae10 0x7fd5a01f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00c7890 con 0x7fd5a00ce390 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a01584f0 con 0x7fd5a006c720 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.192+0000 7fd5a86a3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a01587d0 con 0x7fd5a00cae10 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a643f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a006c720 0x7fd5a01f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32794/0 (socket says 172.21.15.104:32794) 2023-06-10T18:51:56.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 -- 172.21.15.104:0/3539869421 learned_addr learned my addr 172.21.15.104:0/3539869421 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a5c3e700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a00cae10 0x7fd5a01f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 -- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a00cae10 msgr2=0x7fd5a01f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a00cae10 0x7fd5a01f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 -- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a006c720 msgr2=0x7fd5a01f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a006c720 0x7fd5a01f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 -- 172.21.15.104:0/3539869421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598009a70 con 0x7fd5a00ce390 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a6c40700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fd59800efc0 tx=0x7fd59800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59800ca90 con 0x7fd5a00ce390 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59800f040 con 0x7fd5a00ce390 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0094a0 con 0x7fd5a00ce390 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59800ce30 con 0x7fd5a00ce390 2023-06-10T18:51:56.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.193+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a0158d90 con 0x7fd5a00ce390 2023-06-10T18:51:56.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.194+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a0074050 con 0x7fd5a00ce390 2023-06-10T18:51:56.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.195+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd598014480 con 0x7fd5a00ce390 2023-06-10T18:51:56.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.195+0000 7fd596ffd700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 0x7fd58406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:56.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.195+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fd598089c20 con 0x7fd5a00ce390 2023-06-10T18:51:56.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.195+0000 7fd5a643f700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 0x7fd58406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:56.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.196+0000 7fd5a643f700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 0x7fd58406d190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd590007fd0 tx=0x7fd590009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:56.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.200+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd598054b70 con 0x7fd5a00ce390 2023-06-10T18:51:56.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.433+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd5a00b2e10 con 0x7fd5a00ce390 2023-06-10T18:51:56.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.433+0000 7fd596ffd700 1 -- 172.21.15.104:0/3539869421 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd598018070 con 0x7fd5a00ce390 2023-06-10T18:51:56.439 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:51:56.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 msgr2=0x7fd58406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 0x7fd58406d190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd590007fd0 tx=0x7fd590009040 comp rx=0 tx=0).stop 2023-06-10T18:51:56.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 msgr2=0x7fd5a01f65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fd59800efc0 tx=0x7fd59800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 shutdown_connections 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd58406ace0 0x7fd58406d190 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a006c720 0x7fd5a01f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5a00ce390 0x7fd5a01f65e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 --2- 172.21.15.104:0/3539869421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5a00cae10 0x7fd5a01f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:56.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 >> 172.21.15.104:0/3539869421 conn(0x7fd5a0069e70 msgr2=0x7fd5a015aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:56.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.435+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 shutdown_connections 2023-06-10T18:51:56.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:56.436+0000 7fd5a86a3700 1 -- 172.21.15.104:0/3539869421 wait complete. 2023-06-10T18:51:56.452 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:51:56.453 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:51:56.453 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:51:56.453 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:51:56.454 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:56.454 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:51:56.454 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:51:56.454 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:51:56.455 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:51:56.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.455 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:51:56.474 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:51:56.475 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:51:56.475 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:51:56.475 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.475 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:51:56.475 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:56.476 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41964sec preferred_lft 41964sec 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:51:56.477 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:51:56.478 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:51:56.478 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.478 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:51:56.478 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.479 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:51:56.479 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:51:56.479 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:51:56.479 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.479 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:51:56.480 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:51:56.480 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:51:56.480 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:51:56.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.481 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:51:56.547 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:51:56.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.548 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:51:56.607 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:51:56.608 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:51:56.608 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:51:56.608 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:51:56.608 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:51:56.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.609 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:51:56.609 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:51:56.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:51:56.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:51:56.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:51:56.611 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:51:56.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.715 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:51:56.731 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:51:56.731 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:51:56.732 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:51:56.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:56.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:51:56.797 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:51:56.798 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:51:56.798 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:56.854 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:51:56.854 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:51:56.854 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:56.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:51:56.976 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:51:56.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:51:57.042 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:51:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:51:57.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:57.044 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:51:57.099 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:51:57.100 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:51:57.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:51:57.167 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:51:57.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:51:57.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:51:57.222+0000 7fca7db5c5c0 -1 init, newargv = 0x55b7c039ef40 newargc=15 2023-06-10T18:51:57.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111326]: starting ceph client 2023-06-10T18:51:57.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111326]: starting fuse 2023-06-10T18:51:57.233 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:51:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:51:57.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:57.236 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:51:57.292 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:51:57.292 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:51:57.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:51:57.293 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:51:57.293 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:51:57.294 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:51:57.294 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:51:57.294 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:51:57.294 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:51:57.295 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:51:57.295 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:51:57.295 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:51:57.296 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:51:57.296 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:51:57.296 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:51:57.297 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:51:57.297 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:51:57.297 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:51:57.298 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:51:57.298 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:51:57.298 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:51:57.298 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:51:57.299 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:51:57.299 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:51:57.299 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:51:57.300 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:51:57.300 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:51:57.300 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:51:57.301 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:51:57.413 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.111326.asok 2023-06-10T18:51:57.596 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.111326.asok 2023-06-10T18:51:57.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:57.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.111326.asok status 2023-06-10T18:51:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:51:57.752 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:51:57.752 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:51:57.752 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:51:57.753 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:51:57.753 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:51:57.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "111326", 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5927, 2023-06-10T18:51:57.754 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:51:57.755 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:51:57.755 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:51:57.755 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5927 2023-06-10T18:51:57.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1730509434 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:51:57.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1730509434 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.5927 192.168.0.1:0/1730509434", 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1730509434", 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:51:57.757 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 40, 2023-06-10T18:51:57.758 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 85, 2023-06-10T18:51:57.758 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:51:57.758 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:51:57.758 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:51:57.758 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:51:57.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:57.762 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:51:57.762 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:57.779 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:51:57.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:57.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:51:57.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:51:57.846 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:51:57.993 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:51:57.993 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:51:57.994 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:51:58.129 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:51:58.129 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:51:58.130 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:51:58.130 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:51:58.148 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:51:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.261+0000 7fdccbaaf700 1 Processor -- start 2023-06-10T18:51:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.261+0000 7fdccbaaf700 1 -- start start 2023-06-10T18:51:58.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdccbaaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc41567f0 0x7fdcc40cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdccbaaf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdcc415c7a0 0x7fdcc40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdccbaaf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc40cb2d0 con 0x7fdcc40701a0 2023-06-10T18:51:58.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdccbaaf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc415cca0 con 0x7fdcc415c7a0 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdccbaaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc415cf80 con 0x7fdcc41567f0 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdcc904a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdcc415c7a0 0x7fdcc40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdcca04c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdcc40701a0 0x7fdcc4160150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58858/0 (socket says 172.21.15.104:58858) 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.262+0000 7fdcca04c700 1 -- 172.21.15.104:0/880329741 learned_addr learned my addr 172.21.15.104:0/880329741 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc904a700 1 -- 172.21.15.104:0/880329741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc41567f0 msgr2=0x7fdcc40cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc984b700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc41567f0 0x7fdcc40cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc904a700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc41567f0 0x7fdcc40cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc8849700 1 -- 172.21.15.104:0/880329741 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568187497 0 0) 0x7fdcc40cb2d0 con 0x7fdcc40701a0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc904a700 1 -- 172.21.15.104:0/880329741 >> v1:172.21.15.104:6789/0 conn(0x7fdcc40701a0 legacy=0x7fdcc4160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc904a700 1 -- 172.21.15.104:0/880329741 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc415d260 con 0x7fdcc415c7a0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc904a700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc415c7a0 0x7fdcc40cd560 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009840 tx=0x7fdcc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5388a7171306051 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc8849700 1 -- 172.21.15.104:0/880329741 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0011830 con 0x7fdcc415c7a0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc8849700 1 -- 172.21.15.104:0/880329741 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0011990 con 0x7fdcc415c7a0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdcc8849700 1 -- 172.21.15.104:0/880329741 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0011d20 con 0x7fdcc415c7a0 2023-06-10T18:51:58.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 -- 172.21.15.104:0/880329741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc415c7a0 msgr2=0x7fdcc40cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc415c7a0 0x7fdcc40cd560 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009840 tx=0x7fdcc0009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:58.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 -- 172.21.15.104:0/880329741 shutdown_connections 2023-06-10T18:51:58.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc41567f0 0x7fdcc40cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/880329741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc415c7a0 0x7fdcc40cd560 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 -- 172.21.15.104:0/880329741 >> 172.21.15.104:0/880329741 conn(0x7fdcc406e370 msgr2=0x7fdcc406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:58.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 -- 172.21.15.104:0/880329741 shutdown_connections 2023-06-10T18:51:58.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.263+0000 7fdccbaaf700 1 -- 172.21.15.104:0/880329741 wait complete. 2023-06-10T18:51:58.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 Processor -- start 2023-06-10T18:51:58.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 -- start start 2023-06-10T18:51:58.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 0x7fdcc40c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc415c7a0 0x7fdcc4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc40b1680 con 0x7fdcc41567f0 2023-06-10T18:51:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc40b3d00 con 0x7fdcc415c7a0 2023-06-10T18:51:58.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdccbaaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc40b0030 con 0x7fdcc40701a0 2023-06-10T18:51:58.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdcc904a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdcc904a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32804/0 (socket says 172.21.15.104:32804) 2023-06-10T18:51:58.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.265+0000 7fdcc904a700 1 -- 172.21.15.104:0/2260319358 learned_addr learned my addr 172.21.15.104:0/2260319358 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:58.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcca04c700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc415c7a0 0x7fdcc4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc984b700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 0x7fdcc40c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 -- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 msgr2=0x7fdcc40c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 0x7fdcc40c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 -- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc415c7a0 msgr2=0x7fdcc4154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc415c7a0 0x7fdcc4154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 -- 172.21.15.104:0/2260319358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00094a0 con 0x7fdcc41567f0 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc984b700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 0x7fdcc40c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:58.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcc904a700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00090b0 tx=0x7fdcc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00036a0 con 0x7fdcc41567f0 2023-06-10T18:51:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0003800 con 0x7fdcc41567f0 2023-06-10T18:51:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc40b3740 con 0x7fdcc41567f0 2023-06-10T18:51:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0003b70 con 0x7fdcc41567f0 2023-06-10T18:51:58.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.266+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc40b18b0 con 0x7fdcc41567f0 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.267+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc40b1aa0 con 0x7fdcc41567f0 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.267+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdcc0003cd0 con 0x7fdcc41567f0 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.268+0000 7fdcba7fc700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 0x7fdca806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.268+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fdcc008aac0 con 0x7fdcc41567f0 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.268+0000 7fdcc984b700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 0x7fdca806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.269+0000 7fdcc984b700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 0x7fdca806d190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400aa50 tx=0x7fdcb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.272+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdcc0055bc0 con 0x7fdcc41567f0 2023-06-10T18:51:58.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.460+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc4074050 con 0x7fdcc41567f0 2023-06-10T18:51:58.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.461+0000 7fdcba7fc700 1 -- 172.21.15.104:0/2260319358 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+18645 (secure 0 0 0) 0x7fdcc00591e0 con 0x7fdcc41567f0 2023-06-10T18:51:58.467 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:51:58.467 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":85,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:53.390543+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 msgr2=0x7fdca806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 0x7fdca806d190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400aa50 tx=0x7fdcb4008040 comp rx=0 tx=0).stop 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 msgr2=0x7fdcc40c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00090b0 tx=0x7fdcc0004750 comp rx=0 tx=0).stop 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 shutdown_connections 2023-06-10T18:51:58.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdca806ace0 0x7fdca806d190 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc415c7a0 0x7fdcc4154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcc40701a0 0x7fdcc40c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 --2- 172.21.15.104:0/2260319358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcc41567f0 0x7fdcc40c9270 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 >> 172.21.15.104:0/2260319358 conn(0x7fdcc406e370 msgr2=0x7fdcc415d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 shutdown_connections 2023-06-10T18:51:58.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.463+0000 7fdccbaaf700 1 -- 172.21.15.104:0/2260319358 wait complete. 2023-06-10T18:51:58.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:51:58.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:51:58.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.599+0000 7f3b98f82700 1 Processor -- start 2023-06-10T18:51:58.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.599+0000 7f3b98f82700 1 -- start start 2023-06-10T18:51:58.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b98f82700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3b94070a10 0x7f3b94070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b98f82700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940c4770 con 0x7f3b94145cc0 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b98f82700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b94069e70 con 0x7f3b9406a5e0 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b98f82700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b940713d0 con 0x7f3b94070a10 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b9259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3b94145cc0 0x7f3b9406c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58870/0 (socket says 172.21.15.104:58870) 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b9259c700 1 -- 172.21.15.104:0/3111466143 learned_addr learned my addr 172.21.15.104:0/3111466143 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:58.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b91d9b700 1 --2- 172.21.15.104:0/3111466143 >> v2:172.21.15.114:3300/0 conn(0x7f3b94070a10 0x7f3b94070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831573189 0 0) 0x7f3b94069e70 con 0x7f3b9406a5e0 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b7c0035b0 con 0x7f3b9406a5e0 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672093136 0 0) 0x7f3b940c4770 con 0x7f3b94145cc0 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b91d9b700 1 -- 172.21.15.104:0/3111466143 >> v1:172.21.15.114:6789/0 conn(0x7f3b9406a5e0 legacy=0x7f3b9406a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b91d9b700 1 -- 172.21.15.104:0/3111466143 >> v1:172.21.15.104:6789/0 conn(0x7f3b94145cc0 legacy=0x7f3b9406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.600+0000 7f3b91d9b700 1 -- 172.21.15.104:0/3111466143 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9406cf10 con 0x7f3b94070a10 2023-06-10T18:51:58.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b91d9b700 1 --2- 172.21.15.104:0/3111466143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94070a10 0x7f3b94070e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009840 tx=0x7f3b84009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77c4bf70ccd7ecc3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84011830 con 0x7f3b94070a10 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b84011990 con 0x7f3b94070a10 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b9159a700 1 -- 172.21.15.104:0/3111466143 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84011d20 con 0x7f3b94070a10 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 -- 172.21.15.104:0/3111466143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94070a10 msgr2=0x7f3b94070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 --2- 172.21.15.104:0/3111466143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94070a10 0x7f3b94070e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009840 tx=0x7f3b84009b50 comp rx=0 tx=0).stop 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 -- 172.21.15.104:0/3111466143 shutdown_connections 2023-06-10T18:51:58.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 --2- 172.21.15.104:0/3111466143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94070a10 0x7f3b94070e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 -- 172.21.15.104:0/3111466143 >> 172.21.15.104:0/3111466143 conn(0x7f3b940ceec0 msgr2=0x7f3b940cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:58.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 -- 172.21.15.104:0/3111466143 shutdown_connections 2023-06-10T18:51:58.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.601+0000 7f3b98f82700 1 -- 172.21.15.104:0/3111466143 wait complete. 2023-06-10T18:51:58.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.603+0000 7f3b98f82700 1 Processor -- start 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.603+0000 7f3b98f82700 1 -- start start 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b94070a10 0x7f3b941f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94145cc0 0x7f3b9415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940ca770 con 0x7f3b9406a5e0 2023-06-10T18:51:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941f5b40 con 0x7f3b94145cc0 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b98f82700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940ca9a0 con 0x7f3b94070a10 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32816/0 (socket says 172.21.15.104:32816) 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 -- 172.21.15.104:0/2461816204 learned_addr learned my addr 172.21.15.104:0/2461816204 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b92d9d700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94145cc0 0x7f3b9415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b91d9b700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b94070a10 0x7f3b941f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 -- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b94070a10 msgr2=0x7f3b941f0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b94070a10 0x7f3b941f0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 -- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94145cc0 msgr2=0x7f3b9415a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94145cc0 0x7f3b9415a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 -- 172.21.15.104:0/2461816204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840094a0 con 0x7f3b9406a5e0 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.604+0000 7f3b9259c700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c520 tx=0x7f3b8800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.605+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88004c80 con 0x7f3b9406a5e0 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.605+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b88004de0 con 0x7f3b9406a5e0 2023-06-10T18:51:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.605+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b940cac80 con 0x7f3b9406a5e0 2023-06-10T18:51:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.605+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88007310 con 0x7f3b9406a5e0 2023-06-10T18:51:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.605+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b940cb240 con 0x7f3b9406a5e0 2023-06-10T18:51:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.606+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b94074050 con 0x7f3b9406a5e0 2023-06-10T18:51:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.606+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3b88007470 con 0x7f3b9406a5e0 2023-06-10T18:51:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.607+0000 7f3b82ffd700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 0x7f3b7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.607+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f3b88089180 con 0x7f3b9406a5e0 2023-06-10T18:51:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.607+0000 7f3b91d9b700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 0x7f3b7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.607+0000 7f3b91d9b700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 0x7f3b7406d190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009fd0 tx=0x7f3b8401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.611+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3b880540d0 con 0x7f3b9406a5e0 2023-06-10T18:51:58.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.793+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b94071930 con 0x7f3b9406a5e0 2023-06-10T18:51:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.793+0000 7f3b82ffd700 1 -- 172.21.15.104:0/2461816204 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b880576f0 con 0x7f3b9406a5e0 2023-06-10T18:51:58.799 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:51:58.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 msgr2=0x7f3b7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 0x7f3b7406d190 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009fd0 tx=0x7f3b8401c040 comp rx=0 tx=0).stop 2023-06-10T18:51:58.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 msgr2=0x7f3b941f5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c520 tx=0x7f3b8800c830 comp rx=0 tx=0).stop 2023-06-10T18:51:58.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 shutdown_connections 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b7406ace0 0x7f3b7406d190 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b94145cc0 0x7f3b9415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b9406a5e0 0x7f3b941f5600 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 --2- 172.21.15.104:0/2461816204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b94070a10 0x7f3b941f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 >> 172.21.15.104:0/2461816204 conn(0x7f3b940ceec0 msgr2=0x7f3b9406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 shutdown_connections 2023-06-10T18:51:58.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.795+0000 7f3b98f82700 1 -- 172.21.15.104:0/2461816204 wait complete. 2023-06-10T18:51:58.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:51:58.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.929+0000 7f90adb7c700 1 Processor -- start 2023-06-10T18:51:58.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.929+0000 7f90adb7c700 1 -- start start 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90adb7c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 0x7f90a80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90adb7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90a80ae230 0x7f90a806d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90adb7c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a815cb00 con 0x7f90a80cd720 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90adb7c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a80cdbc0 con 0x7f90a80ae230 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90adb7c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a80acd90 con 0x7f90a815c730 2023-06-10T18:51:58.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90a80ae230 0x7f90a806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90a80ae230 0x7f90a806d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32828/0 (socket says 172.21.15.104:32828) 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/1622268808 learned_addr learned my addr 172.21.15.104:0/1622268808 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a77fe700 1 --2- 172.21.15.104:0/1622268808 >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 0x7f90a80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/1622268808 >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 msgr2=0x7f90a80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 --2- 172.21.15.104:0/1622268808 >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 0x7f90a80cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/1622268808 >> v1:172.21.15.114:6789/0 conn(0x7f90a80cd720 legacy=0x7f90a806cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:58.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/1622268808 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80ac460 con 0x7f90a80ae230 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a77fe700 1 --2- 172.21.15.104:0/1622268808 >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 0x7f90a80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.930+0000 7f90a7fff700 1 --2- 172.21.15.104:0/1622268808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a80ae230 0x7f90a806d480 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f90a000d150 tx=0x7f90a000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c82d0e72eb94581 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90a67fc700 1 -- 172.21.15.104:0/1622268808 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0010070 con 0x7f90a80ae230 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90a67fc700 1 -- 172.21.15.104:0/1622268808 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a000dd40 con 0x7f90a80ae230 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90a67fc700 1 -- 172.21.15.104:0/1622268808 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a000b310 con 0x7f90a80ae230 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 -- 172.21.15.104:0/1622268808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a80ae230 msgr2=0x7f90a806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1622268808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a80ae230 0x7f90a806d480 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f90a000d150 tx=0x7f90a000d460 comp rx=0 tx=0).stop 2023-06-10T18:51:58.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 -- 172.21.15.104:0/1622268808 shutdown_connections 2023-06-10T18:51:58.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1622268808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a80ae230 0x7f90a806d480 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1622268808 >> v2:172.21.15.114:3300/0 conn(0x7f90a815c730 0x7f90a80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 -- 172.21.15.104:0/1622268808 >> 172.21.15.104:0/1622268808 conn(0x7f90a806b8a0 msgr2=0x7f90a80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:58.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 -- 172.21.15.104:0/1622268808 shutdown_connections 2023-06-10T18:51:58.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.931+0000 7f90adb7c700 1 -- 172.21.15.104:0/1622268808 wait complete. 2023-06-10T18:51:58.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.932+0000 7f90adb7c700 1 Processor -- start 2023-06-10T18:51:58.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 -- start start 2023-06-10T18:51:58.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90a80ae230 0x7f90a81f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a80cd720 0x7f90a81501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80c8f60 con 0x7f90a815c730 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81f5e20 con 0x7f90a80cd720 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90adb7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80b35c0 con 0x7f90a80ae230 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32834/0 (socket says 172.21.15.104:32834) 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.933+0000 7f90a7fff700 1 -- 172.21.15.104:0/1765576372 learned_addr learned my addr 172.21.15.104:0/1765576372 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a6ffd700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a80cd720 0x7f90a81501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a77fe700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90a80ae230 0x7f90a81f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 -- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90a80ae230 msgr2=0x7f90a81f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90a80ae230 0x7f90a81f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 -- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a80cd720 msgr2=0x7f90a81501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a80cd720 0x7f90a81501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:58.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 -- 172.21.15.104:0/1765576372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a000bd40 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90a7fff700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f90a0017040 tx=0x7f90a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0009a10 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a0009b70 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a80b38a0 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0004030 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.934+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a80b3e60 con 0x7f90a815c730 2023-06-10T18:51:58.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.935+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8074050 con 0x7f90a815c730 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.935+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90a001b070 con 0x7f90a815c730 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.936+0000 7f908ffff700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 0x7f909406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.936+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f90a008a8f0 con 0x7f90a815c730 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.936+0000 7f90a77fe700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 0x7f909406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.936+0000 7f90a77fe700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 0x7f909406d190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f909c00a970 tx=0x7f909c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:58.940+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90a00558d0 con 0x7f90a815c730 2023-06-10T18:51:59.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.127+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f90a815cb00 con 0x7f90a815c730 2023-06-10T18:51:59.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.127+0000 7f908ffff700 1 -- 172.21.15.104:0/1765576372 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v85) v1 ==== 92+0+10 (secure 0 0 0) 0x7f90a815cb00 con 0x7f90a815c730 2023-06-10T18:51:59.133 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:51:59.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 msgr2=0x7f909406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 0x7f909406d190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f909c00a970 tx=0x7f909c008040 comp rx=0 tx=0).stop 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 msgr2=0x7f90a81f4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f90a0017040 tx=0x7f90a0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 shutdown_connections 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a815c730 0x7f90a81f4fc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90a80ae230 0x7f90a81f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909406ace0 0x7f909406d190 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 --2- 172.21.15.104:0/1765576372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a80cd720 0x7f90a81501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 >> 172.21.15.104:0/1765576372 conn(0x7f90a806b8a0 msgr2=0x7f90a815a250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 shutdown_connections 2023-06-10T18:51:59.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.129+0000 7f90adb7c700 1 -- 172.21.15.104:0/1765576372 wait complete. 2023-06-10T18:51:59.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:51:59.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.263+0000 7f22f332e700 1 Processor -- start 2023-06-10T18:51:59.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.264+0000 7f22f332e700 1 -- start start 2023-06-10T18:51:59.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.264+0000 7f22f332e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22ec15c500 0x7f22ec0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.264+0000 7f22f332e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec15c8d0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f332e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06abd0 con 0x7f22ec0cf200 2023-06-10T18:51:59.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f332e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ec0aef20 con 0x7f22ec15c500 2023-06-10T18:51:59.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f18cb700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f22ec06dcd0 0x7f22ec06e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58902/0 (socket says 172.21.15.104:58902) 2023-06-10T18:51:59.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f18cb700 1 -- 172.21.15.104:0/2698665082 learned_addr learned my addr 172.21.15.104:0/2698665082 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:59.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f10ca700 1 --2- 172.21.15.104:0/2698665082 >> v2:172.21.15.114:3300/0 conn(0x7f22ec15c500 0x7f22ec0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22e3fff700 1 -- 172.21.15.104:0/2698665082 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396866779 0 0) 0x7f22ec15c8d0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22e3fff700 1 -- 172.21.15.104:0/2698665082 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d80035b0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f10ca700 1 -- 172.21.15.104:0/2698665082 >> v1:172.21.15.114:6789/0 conn(0x7f22ec0cf200 legacy=0x7f22ec069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f10ca700 1 -- 172.21.15.104:0/2698665082 >> v1:172.21.15.104:6789/0 conn(0x7f22ec06dcd0 legacy=0x7f22ec06e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f10ca700 1 -- 172.21.15.104:0/2698665082 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec06a8f0 con 0x7f22ec15c500 2023-06-10T18:51:59.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22f10ca700 1 --2- 172.21.15.104:0/2698665082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec15c500 0x7f22ec0cecc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009820 tx=0x7f22dc009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f744687f7b65a01 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22e3fff700 1 -- 172.21.15.104:0/2698665082 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc005e30 con 0x7f22ec15c500 2023-06-10T18:51:59.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22e3fff700 1 -- 172.21.15.104:0/2698665082 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc006030 con 0x7f22ec15c500 2023-06-10T18:51:59.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.265+0000 7f22e3fff700 1 -- 172.21.15.104:0/2698665082 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc0063c0 con 0x7f22ec15c500 2023-06-10T18:51:59.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 -- 172.21.15.104:0/2698665082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec15c500 msgr2=0x7f22ec0cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 --2- 172.21.15.104:0/2698665082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec15c500 0x7f22ec0cecc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009820 tx=0x7f22dc009b30 comp rx=0 tx=0).stop 2023-06-10T18:51:59.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 -- 172.21.15.104:0/2698665082 shutdown_connections 2023-06-10T18:51:59.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 --2- 172.21.15.104:0/2698665082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec15c500 0x7f22ec0cecc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 -- 172.21.15.104:0/2698665082 >> 172.21.15.104:0/2698665082 conn(0x7f22ec06b8a0 msgr2=0x7f22ec0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 -- 172.21.15.104:0/2698665082 shutdown_connections 2023-06-10T18:51:59.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.266+0000 7f22f332e700 1 -- 172.21.15.104:0/2698665082 wait complete. 2023-06-10T18:51:59.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.267+0000 7f22f332e700 1 Processor -- start 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.267+0000 7f22f332e700 1 -- start start 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec0cf200 0x7f22ec0c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 0x7f22ec0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0c7510 con 0x7f22ec06dcd0 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0c77f0 con 0x7f22ec0cf200 2023-06-10T18:51:59.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f332e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0c7ad0 con 0x7f22ec15c500 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32844/0 (socket says 172.21.15.104:32844) 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 -- 172.21.15.104:0/958457507 learned_addr learned my addr 172.21.15.104:0/958457507 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f18cb700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 0x7f22ec0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f08c9700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec0cf200 0x7f22ec0c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 -- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 msgr2=0x7f22ec0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 0x7f22ec0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 -- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec0cf200 msgr2=0x7f22ec0c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec0cf200 0x7f22ec0c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 -- 172.21.15.104:0/958457507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc009430 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f18cb700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 0x7f22ec0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.268+0000 7f22f10ca700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009fd0 tx=0x7f22dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.269+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc004240 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.269+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc0043a0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.269+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec0c7db0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.269+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc0036a0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.269+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec0c70f0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.270+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec0b2ec0 con 0x7f22ec06dcd0 2023-06-10T18:51:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.270+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f22dc022070 con 0x7f22ec06dcd0 2023-06-10T18:51:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.270+0000 7f22e1ffb700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 0x7f22d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.271+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f22dc08ad70 con 0x7f22ec06dcd0 2023-06-10T18:51:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.271+0000 7f22f08c9700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 0x7f22d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.271+0000 7f22f08c9700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 0x7f22d006d190 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0cd7e0 tx=0x7f22e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.275+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22dc055d50 con 0x7f22ec06dcd0 2023-06-10T18:51:59.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.458+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f22ec066f10 con 0x7f22ec06dcd0 2023-06-10T18:51:59.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.459+0000 7f22e1ffb700 1 -- 172.21.15.104:0/958457507 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f22dc059370 con 0x7f22ec06dcd0 2023-06-10T18:51:59.465 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 msgr2=0x7f22d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 0x7f22d006d190 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0cd7e0 tx=0x7f22e8009040 comp rx=0 tx=0).stop 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 msgr2=0x7f22ec14c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009fd0 tx=0x7f22dc004750 comp rx=0 tx=0).stop 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 shutdown_connections 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22ec15c500 0x7f22ec0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22ec06dcd0 0x7f22ec14c750 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22d006ace0 0x7f22d006d190 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 --2- 172.21.15.104:0/958457507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22ec0cf200 0x7f22ec0c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 >> 172.21.15.104:0/958457507 conn(0x7f22ec06b8a0 msgr2=0x7f22ec0ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 shutdown_connections 2023-06-10T18:51:59.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.461+0000 7f22f332e700 1 -- 172.21.15.104:0/958457507 wait complete. 2023-06-10T18:51:59.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:51:59.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.595+0000 7fb43639b700 1 Processor -- start 2023-06-10T18:51:59.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.595+0000 7fb43639b700 1 -- start start 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb43639b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb43015c500 0x7fb4300cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb43639b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43015c8d0 con 0x7fb43006dcd0 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb43639b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43006abd0 con 0x7fb4300cf200 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb43639b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4300aef20 con 0x7fb43015c500 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb434938700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb43006dcd0 0x7fb43006e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58914/0 (socket says 172.21.15.104:58914) 2023-06-10T18:51:59.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb434938700 1 -- 172.21.15.104:0/1923780606 learned_addr learned my addr 172.21.15.104:0/1923780606 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777450522 0 0) 0x7fb43015c8d0 con 0x7fb43006dcd0 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4180035b0 con 0x7fb43006dcd0 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42ffff700 1 --2- 172.21.15.104:0/1923780606 >> v2:172.21.15.114:3300/0 conn(0x7fb43015c500 0x7fb4300cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2082355979 0 0) 0x7fb4180035b0 con 0x7fb43006dcd0 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 >> v2:172.21.15.114:3300/0 conn(0x7fb43015c500 msgr2=0x7fb4300cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 --2- 172.21.15.104:0/1923780606 >> v2:172.21.15.114:3300/0 conn(0x7fb43015c500 0x7fb4300cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 >> v1:172.21.15.114:6789/0 conn(0x7fb4300cf200 legacy=0x7fb430069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43006a8f0 con 0x7fb43006dcd0 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb4280019f0 con 0x7fb43006dcd0 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 >> v1:172.21.15.104:6789/0 conn(0x7fb43006dcd0 legacy=0x7fb43006e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 0x7fb418003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418004850 0x7fb418004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.596+0000 7fb42effd700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb418005200 0x7fb41800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41800c160 con 0x7fb418003a00 2023-06-10T18:51:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41800c2d0 con 0x7fb418004850 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41800c490 con 0x7fb418005200 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 0x7fb418003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42f7fe700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418004850 0x7fb418004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb434938700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb418005200 0x7fb41800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 -- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb418005200 msgr2=0x7fb41800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb418005200 0x7fb41800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 -- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418004850 msgr2=0x7fb418004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:51:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418004850 0x7fb418004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 -- 172.21.15.104:0/1923780606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180035b0 con 0x7fb418003a00 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb434938700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb418005200 0x7fb41800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb434938700 1 -- 172.21.15.104:0/1923780606 reap_dead start 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42ffff700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 0x7fb418003e10 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb424006fd0 tx=0x7fb418005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb98e961f9a4708a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400c070 con 0x7fb418003a00 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb424005c20 con 0x7fb418003a00 2023-06-10T18:51:59.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.597+0000 7fb42effd700 1 -- 172.21.15.104:0/1923780606 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424006050 con 0x7fb418003a00 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 -- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 msgr2=0x7fb418003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 0x7fb418003e10 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb424006fd0 tx=0x7fb418005880 comp rx=0 tx=0).stop 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 -- 172.21.15.104:0/1923780606 shutdown_connections 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 --2- 172.21.15.104:0/1923780606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb418003a00 0x7fb418003e10 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 -- 172.21.15.104:0/1923780606 >> 172.21.15.104:0/1923780606 conn(0x7fb43006b8a0 msgr2=0x7fb4300cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 -- 172.21.15.104:0/1923780606 shutdown_connections 2023-06-10T18:51:59.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.598+0000 7fb43639b700 1 -- 172.21.15.104:0/1923780606 wait complete. 2023-06-10T18:51:59.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.599+0000 7fb43639b700 1 Processor -- start 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.599+0000 7fb43639b700 1 -- start start 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418003a00 0x7fb43014b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 0x7fb4300c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300c70a0 con 0x7fb4300c6c20 2023-06-10T18:51:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300c95d0 con 0x7fb418003a00 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb43639b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300b27f0 con 0x7fb4301f47b0 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32866/0 (socket says 172.21.15.104:32866) 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 -- 172.21.15.104:0/730034967 learned_addr learned my addr 172.21.15.104:0/730034967 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb42f7fe700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 0x7fb4300c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb42ffff700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418003a00 0x7fb43014b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 -- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 msgr2=0x7fb4300c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 0x7fb4300c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 -- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418003a00 msgr2=0x7fb43014b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418003a00 0x7fb43014b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 -- 172.21.15.104:0/730034967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300b2ad0 con 0x7fb4300c6c20 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb42f7fe700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 0x7fb4300c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.600+0000 7fb434938700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb43006f040 tx=0x7fb42800e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.601+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428017070 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.601+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42800ed80 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.601+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4300b2db0 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.601+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428013420 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.601+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43014bc10 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.602+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4300b32d0 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.603+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb428013580 con 0x7fb4300c6c20 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.603+0000 7fb42cff9700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 0x7fb41c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.603+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fb4280896d0 con 0x7fb4300c6c20 2023-06-10T18:51:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.603+0000 7fb42ffff700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 0x7fb41c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.604+0000 7fb42ffff700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 0x7fb41c06d1c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb424008010 tx=0x7fb42401e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.607+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb428054790 con 0x7fb4300c6c20 2023-06-10T18:51:59.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.791+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4300b32d0 con 0x7fb4300c6c20 2023-06-10T18:51:59.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.792+0000 7fb42cff9700 1 -- 172.21.15.104:0/730034967 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb4300b32d0 con 0x7fb4300c6c20 2023-06-10T18:51:59.798 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:51:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 msgr2=0x7fb41c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 0x7fb41c06d1c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb424008010 tx=0x7fb42401e040 comp rx=0 tx=0).stop 2023-06-10T18:51:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 msgr2=0x7fb4300c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb43006f040 tx=0x7fb42800e4f0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 shutdown_connections 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb41c06ad10 0x7fb41c06d1c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb418003a00 0x7fb43014b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4300c6c20 0x7fb4300c8860 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 --2- 172.21.15.104:0/730034967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4301f47b0 0x7fb4300c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 >> 172.21.15.104:0/730034967 conn(0x7fb43006b8a0 msgr2=0x7fb4300acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 shutdown_connections 2023-06-10T18:51:59.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.794+0000 7fb43639b700 1 -- 172.21.15.104:0/730034967 wait complete. 2023-06-10T18:51:59.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:51:59.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.926+0000 7f9477270700 1 Processor -- start 2023-06-10T18:51:59.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.926+0000 7f9477270700 1 -- start start 2023-06-10T18:51:59.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f9477270700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f94700cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f9477270700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94700cadd0 con 0x7f947008a3d0 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f9477270700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94700cb0b0 con 0x7f94700ce350 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f9477270700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700cfeb0 con 0x7f947006b1f0 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f947580d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f947008a3d0 0x7f94700cb540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58932/0 (socket says 172.21.15.104:58932) 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.927+0000 7f947580d700 1 -- 172.21.15.104:0/1334578144 learned_addr learned my addr 172.21.15.104:0/1334578144 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f947500c700 1 --2- 172.21.15.104:0/1334578144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f94700cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876428729 0 0) 0x7f94700cadd0 con 0x7f947008a3d0 2023-06-10T18:51:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c0035b0 con 0x7f947008a3d0 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f947500c700 1 -- 172.21.15.104:0/1334578144 >> v1:172.21.15.114:6789/0 conn(0x7f94700ce350 legacy=0x7f94700cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849581181 0 0) 0x7f94700cb0b0 con 0x7f94700ce350 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f947500c700 1 -- 172.21.15.104:0/1334578144 >> v1:172.21.15.104:6789/0 conn(0x7f947008a3d0 legacy=0x7f94700cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f947500c700 1 -- 172.21.15.104:0/1334578144 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94700cf550 con 0x7f947006b1f0 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f947500c700 1 --2- 172.21.15.104:0/1334578144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f94700cd090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f9460009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52934bf53e73f0c4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460011830 con 0x7f947006b1f0 2023-06-10T18:51:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9460011990 con 0x7f947006b1f0 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.928+0000 7f9467fff700 1 -- 172.21.15.104:0/1334578144 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460011d20 con 0x7f947006b1f0 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 -- 172.21.15.104:0/1334578144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 msgr2=0x7f94700cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 --2- 172.21.15.104:0/1334578144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f94700cd090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f9460009b80 comp rx=0 tx=0).stop 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 -- 172.21.15.104:0/1334578144 shutdown_connections 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 --2- 172.21.15.104:0/1334578144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f94700cd090 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 -- 172.21.15.104:0/1334578144 >> 172.21.15.104:0/1334578144 conn(0x7f9470089fd0 msgr2=0x7f94700d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:51:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 -- 172.21.15.104:0/1334578144 shutdown_connections 2023-06-10T18:51:59.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.929+0000 7f9477270700 1 -- 172.21.15.104:0/1334578144 wait complete. 2023-06-10T18:51:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.930+0000 7f9477270700 1 Processor -- start 2023-06-10T18:51:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.930+0000 7f9477270700 1 -- start start 2023-06-10T18:51:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f947014dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94700ce350 0x7f94700c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700b4230 con 0x7f947008a3d0 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700cab50 con 0x7f94700ce350 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f9477270700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700c7080 con 0x7f947006b1f0 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32876/0 (socket says 172.21.15.104:32876) 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 -- 172.21.15.104:0/2485690027 learned_addr learned my addr 172.21.15.104:0/2485690027 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:51:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947500c700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f947014dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947580d700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94700ce350 0x7f94700c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 -- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 msgr2=0x7f947014dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f947014dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 -- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94700ce350 msgr2=0x7f94700c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94700ce350 0x7f94700c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 -- 172.21.15.104:0/2485690027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600094a0 con 0x7f947008a3d0 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.931+0000 7f947480b700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f946c00c530 tx=0x7f946c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.932+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c0054e0 con 0x7f947008a3d0 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.932+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c003680 con 0x7f947008a3d0 2023-06-10T18:51:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.932+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94700b35b0 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.932+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c003a30 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.932+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94700b3b70 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.933+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94700b4370 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.933+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f946c003b90 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.934+0000 7f9465ffb700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 0x7f945406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.934+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f946c088920 con 0x7f947008a3d0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.934+0000 7f947500c700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 0x7f945406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:51:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.934+0000 7f947500c700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 0x7f945406d190 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f94600090b0 tx=0x7f946001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:51:59.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:51:59.938+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94700b4370 con 0x7f947008a3d0 2023-06-10T18:52:00.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.137+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f94700b4370 con 0x7f947008a3d0 2023-06-10T18:52:00.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.137+0000 7f9465ffb700 1 -- 172.21.15.104:0/2485690027 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v85) v1 ==== 80+0+0 (secure 0 0 0) 0x7f94700b4370 con 0x7f947008a3d0 2023-06-10T18:52:00.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 msgr2=0x7f945406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 0x7f945406d190 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f94600090b0 tx=0x7f946001c040 comp rx=0 tx=0).stop 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 msgr2=0x7f947014a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f946c00c530 tx=0x7f946c00c840 comp rx=0 tx=0).stop 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 shutdown_connections 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f945406ace0 0x7f945406d190 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94700ce350 0x7f94700c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f947006b1f0 0x7f947014dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 --2- 172.21.15.104:0/2485690027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f947008a3d0 0x7f947014a770 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 >> 172.21.15.104:0/2485690027 conn(0x7f9470089fd0 msgr2=0x7f94700b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 shutdown_connections 2023-06-10T18:52:00.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.139+0000 7f9477270700 1 -- 172.21.15.104:0/2485690027 wait complete. 2023-06-10T18:52:00.148 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:52:00.156 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:52:00.157 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:52:00.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:52:00.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.292+0000 7f235d571700 1 Processor -- start 2023-06-10T18:52:00.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.292+0000 7f235d571700 1 -- start start 2023-06-10T18:52:00.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f235d571700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23580cea60 0x7f235806b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f235d571700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235806bfd0 con 0x7f23580b1e30 2023-06-10T18:52:00.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f235d571700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235806c2b0 con 0x7f23580d0c00 2023-06-10T18:52:00.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f235d571700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580ccf70 con 0x7f23580cea60 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f2356ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f23580b1e30 0x7f23580d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58942/0 (socket says 172.21.15.104:58942) 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f2356ffd700 1 -- 172.21.15.104:0/3905363135 learned_addr learned my addr 172.21.15.104:0/3905363135 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f23577fe700 1 --2- 172.21.15.104:0/3905363135 >> v2:172.21.15.114:3300/0 conn(0x7f23580cea60 0x7f235806b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026067784 0 0) 0x7f235806bfd0 con 0x7f23580b1e30 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.293+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23400035b0 con 0x7f23580b1e30 2023-06-10T18:52:00.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457515128 0 0) 0x7f23400035b0 con 0x7f23580b1e30 2023-06-10T18:52:00.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 >> v2:172.21.15.114:3300/0 conn(0x7f23580cea60 msgr2=0x7f235806b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 --2- 172.21.15.104:0/3905363135 >> v2:172.21.15.114:3300/0 conn(0x7f23580cea60 0x7f235806b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 >> v1:172.21.15.114:6789/0 conn(0x7f23580d0c00 legacy=0x7f235806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580cd250 con 0x7f23580b1e30 2023-06-10T18:52:00.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f234c001670 con 0x7f23580b1e30 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 >> v1:172.21.15.104:6789/0 conn(0x7f23580b1e30 legacy=0x7f23580d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340003a00 0x7f2340003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 0x7f2340004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2340005200 0x7f234000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580ccf70 con 0x7f2340004850 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234000c110 con 0x7f2340003a00 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234000c2d0 con 0x7f2340005200 2023-06-10T18:52:00.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 0x7f2340004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2356ffd700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340003a00 0x7f2340003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23577fe700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2340005200 0x7f234000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 -- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2340005200 msgr2=0x7f234000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2340005200 0x7f234000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 -- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340003a00 msgr2=0x7f2340003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340003a00 0x7f2340003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 -- 172.21.15.104:0/3905363135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400035b0 con 0x7f2340004850 2023-06-10T18:52:00.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23577fe700 1 -- 172.21.15.104:0/3905363135 reap_dead start 2023-06-10T18:52:00.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f23567fc700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 0x7f2340004cc0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f23480017c0 tx=0x7f2340005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cdf70172f2f934 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23480052d0 con 0x7f2340004850 2023-06-10T18:52:00.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2348005430 con 0x7f2340004850 2023-06-10T18:52:00.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.294+0000 7f2355ffb700 1 -- 172.21.15.104:0/3905363135 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23480038c0 con 0x7f2340004850 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 -- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 msgr2=0x7f2340004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 0x7f2340004cc0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f23480017c0 tx=0x7f2340005880 comp rx=0 tx=0).stop 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 -- 172.21.15.104:0/3905363135 shutdown_connections 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 --2- 172.21.15.104:0/3905363135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2340004850 0x7f2340004cc0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 -- 172.21.15.104:0/3905363135 >> 172.21.15.104:0/3905363135 conn(0x7f2358071ec0 msgr2=0x7f235809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 -- 172.21.15.104:0/3905363135 shutdown_connections 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.295+0000 7f235d571700 1 -- 172.21.15.104:0/3905363135 wait complete. 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.296+0000 7f235d571700 1 Processor -- start 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.296+0000 7f235d571700 1 -- start start 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340004850 0x7f235815e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23580b7500 0x7f23580b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c99f0 con 0x7f23581f14b0 2023-06-10T18:52:00.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580ca290 con 0x7f2340004850 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f235d571700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b8810 con 0x7f23580b7500 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32898/0 (socket says 172.21.15.104:32898) 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 -- 172.21.15.104:0/1906662171 learned_addr learned my addr 172.21.15.104:0/1906662171 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f2356ffd700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340004850 0x7f235815e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 -- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23580b7500 msgr2=0x7f23580b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23580b7500 0x7f23580b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 -- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340004850 msgr2=0x7f235815e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340004850 0x7f235815e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.297+0000 7f23577fe700 1 -- 172.21.15.104:0/1906662171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235815efe0 con 0x7f23581f14b0 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f23577fe700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f23580d0fd0 tx=0x7f2350011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000e040 con 0x7f23581f14b0 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000f040 con 0x7f23581f14b0 2023-06-10T18:52:00.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23580c9c40 con 0x7f23581f14b0 2023-06-10T18:52:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000fdb0 con 0x7f23581f14b0 2023-06-10T18:52:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.298+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23581f1950 con 0x7f23581f14b0 2023-06-10T18:52:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.299+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580d1860 con 0x7f23581f14b0 2023-06-10T18:52:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.299+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2350022460 con 0x7f23581f14b0 2023-06-10T18:52:00.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.300+0000 7f233f7fe700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 0x7f234406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.300+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f2350089c20 con 0x7f23581f14b0 2023-06-10T18:52:00.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.300+0000 7f2356ffd700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 0x7f234406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.300+0000 7f2356ffd700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 0x7f234406d190 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2358063ac0 tx=0x7f234c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.304+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2350054b70 con 0x7f23581f14b0 2023-06-10T18:52:00.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.496+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23580cd510 con 0x7f23581f14b0 2023-06-10T18:52:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.497+0000 7f233f7fe700 1 -- 172.21.15.104:0/1906662171 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+18645 (secure 0 0 0) 0x7f23580cd510 con 0x7f23581f14b0 2023-06-10T18:52:00.504 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:00.504 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":85,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:51:53.390543+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:52:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 msgr2=0x7f234406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 0x7f234406d190 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2358063ac0 tx=0x7f234c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 msgr2=0x7f23580b7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f23580d0fd0 tx=0x7f2350011f80 comp rx=0 tx=0).stop 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 shutdown_connections 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f234406ace0 0x7f234406d190 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2340004850 0x7f235815e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23581f14b0 0x7f23580b7a50 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 --2- 172.21.15.104:0/1906662171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23580b7500 0x7f23580b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 >> 172.21.15.104:0/1906662171 conn(0x7f2358071ec0 msgr2=0x7f235815be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 shutdown_connections 2023-06-10T18:52:00.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.500+0000 7f235d571700 1 -- 172.21.15.104:0/1906662171 wait complete. 2023-06-10T18:52:00.517 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:52:00.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:52:00.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.635+0000 7fb4a6c14700 1 Processor -- start 2023-06-10T18:52:00.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.635+0000 7fb4a6c14700 1 -- start start 2023-06-10T18:52:00.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a6c14700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0069e70 0x7fb4a006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a6c14700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4a006dcd0 0x7fb4a006e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a6c14700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a006e650 con 0x7fb4a015c500 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a6c14700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a00cf4d0 con 0x7fb4a0069e70 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a6c14700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a00ce040 con 0x7fb4a006dcd0 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0069e70 0x7fb4a006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a49b0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb4a015c500 0x7fb4a00cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58962/0 (socket says 172.21.15.104:58962) 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a49b0700 1 -- 172.21.15.104:0/1717480352 learned_addr learned my addr 172.21.15.104:0/1717480352 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb4a0069e70 0x7fb4a006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32904/0 (socket says 172.21.15.104:32904) 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb4a51b1700 1 --2- 172.21.15.104:0/1717480352 >> v2:172.21.15.114:3300/0 conn(0x7fb4a006dcd0 0x7fb4a006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 -- 172.21.15.104:0/1717480352 >> v2:172.21.15.114:3300/0 conn(0x7fb4a006dcd0 msgr2=0x7fb4a006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 --2- 172.21.15.104:0/1717480352 >> v2:172.21.15.114:3300/0 conn(0x7fb4a006dcd0 0x7fb4a006e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 -- 172.21.15.104:0/1717480352 >> v1:172.21.15.104:6789/0 conn(0x7fb4a015c500 legacy=0x7fb4a00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 -- 172.21.15.104:0/1717480352 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a015c8d0 con 0x7fb4a0069e70 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49f7fe700 1 -- 172.21.15.104:0/1717480352 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353838213 0 0) 0x7fb4a006e650 con 0x7fb4a015c500 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49ffff700 1 --2- 172.21.15.104:0/1717480352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0069e70 0x7fb4a006a2e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00cd0a0 tx=0x7fb48c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4822484266c822 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49f7fe700 1 -- 172.21.15.104:0/1717480352 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011830 con 0x7fb4a0069e70 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49f7fe700 1 -- 172.21.15.104:0/1717480352 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c011990 con 0x7fb4a0069e70 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.636+0000 7fb49f7fe700 1 -- 172.21.15.104:0/1717480352 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011d20 con 0x7fb4a0069e70 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 -- 172.21.15.104:0/1717480352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0069e70 msgr2=0x7fb4a006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/1717480352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0069e70 0x7fb4a006a2e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00cd0a0 tx=0x7fb48c009b20 comp rx=0 tx=0).stop 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 -- 172.21.15.104:0/1717480352 shutdown_connections 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/1717480352 >> v2:172.21.15.114:3300/0 conn(0x7fb4a006dcd0 0x7fb4a006e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/1717480352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a0069e70 0x7fb4a006a2e0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 -- 172.21.15.104:0/1717480352 >> 172.21.15.104:0/1717480352 conn(0x7fb4a006b8a0 msgr2=0x7fb4a00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 -- 172.21.15.104:0/1717480352 shutdown_connections 2023-06-10T18:52:00.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.637+0000 7fb4a6c14700 1 -- 172.21.15.104:0/1717480352 wait complete. 2023-06-10T18:52:00.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.638+0000 7fb4a6c14700 1 Processor -- start 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.638+0000 7fb4a6c14700 1 -- start start 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 0x7fb4a0150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 0x7fb4a00c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0150e40 con 0x7fb4a015c500 2023-06-10T18:52:00.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00b4c10 con 0x7fb4a006dcd0 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a6c14700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00b4ef0 con 0x7fb4a0069e70 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32912/0 (socket says 172.21.15.104:32912) 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 -- 172.21.15.104:0/2010369361 learned_addr learned my addr 172.21.15.104:0/2010369361 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb49ffff700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 0x7fb4a00c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a49b0700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 0x7fb4a0150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 -- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 msgr2=0x7fb4a0150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 0x7fb4a0150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 -- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 msgr2=0x7fb4a00c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 0x7fb4a00c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a51b1700 1 -- 172.21.15.104:0/2010369361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0094a0 con 0x7fb4a015c500 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb4a49b0700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 0x7fb4a0150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.639+0000 7fb49ffff700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 0x7fb4a00c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:00.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb4a51b1700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb498007e40 tx=0x7fb49800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49800eea0 con 0x7fb4a015c500 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498014420 con 0x7fb4a015c500 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a00b51d0 con 0x7fb4a015c500 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4980147b0 con 0x7fb4a015c500 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.640+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a00b2840 con 0x7fb4a015c500 2023-06-10T18:52:00.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.641+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0074050 con 0x7fb4a015c500 2023-06-10T18:52:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.641+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb498014910 con 0x7fb4a015c500 2023-06-10T18:52:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.641+0000 7fb49d7fa700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.641+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7fb498089d90 con 0x7fb4a015c500 2023-06-10T18:52:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.642+0000 7fb4a49b0700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.642+0000 7fb4a49b0700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb494001360 tx=0x7fb494009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.646+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb498054d70 con 0x7fb4a015c500 2023-06-10T18:52:00.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.831+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb4a006c320 con 0x7fb4a015c500 2023-06-10T18:52:00.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.832+0000 7fb49d7fa700 1 -- 172.21.15.104:0/2010369361 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb498058390 con 0x7fb4a015c500 2023-06-10T18:52:00.838 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:52:00.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 msgr2=0x7fb48406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb494001360 tx=0x7fb494009040 comp rx=0 tx=0).stop 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 msgr2=0x7fb4a01f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb498007e40 tx=0x7fb49800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 shutdown_connections 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a015c500 0x7fb4a01f4c20 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4a0069e70 0x7fb4a0150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb48406ace0 0x7fb48406d190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 --2- 172.21.15.104:0/2010369361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4a006dcd0 0x7fb4a00c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 >> 172.21.15.104:0/2010369361 conn(0x7fb4a006b8a0 msgr2=0x7fb4a00b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 shutdown_connections 2023-06-10T18:52:00.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.834+0000 7fb4a6c14700 1 -- 172.21.15.104:0/2010369361 wait complete. 2023-06-10T18:52:00.851 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:52:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.967+0000 7f393fae6700 1 Processor -- start 2023-06-10T18:52:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.968+0000 7f393fae6700 1 -- start start 2023-06-10T18:52:00.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.968+0000 7f393fae6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f393806cde0 0x7f39380b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.968+0000 7f393fae6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393806e250 con 0x7f393815c580 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.968+0000 7f393fae6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393815c950 con 0x7f393806e6e0 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.968+0000 7f393fae6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39380b1e80 con 0x7f393806cde0 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393d882700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f393815c580 0x7f39380b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58976/0 (socket says 172.21.15.104:58976) 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393d882700 1 -- 172.21.15.104:0/2058359115 learned_addr learned my addr 172.21.15.104:0/2058359115 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> v2:172.21.15.114:3300/0 conn(0x7f393806cde0 0x7f39380b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383889029 0 0) 0x7f393806e250 con 0x7f393815c580 2023-06-10T18:52:00.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39240035b0 con 0x7f393815c580 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098563676 0 0) 0x7f39240035b0 con 0x7f393815c580 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 >> v2:172.21.15.114:3300/0 conn(0x7f393806cde0 msgr2=0x7f39380b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 --2- 172.21.15.104:0/2058359115 >> v2:172.21.15.114:3300/0 conn(0x7f393806cde0 0x7f39380b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 >> v1:172.21.15.114:6789/0 conn(0x7f393806e6e0 legacy=0x7f39380ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393806d920 con 0x7f393815c580 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f39280017f0 con 0x7f393815c580 2023-06-10T18:52:00.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> v2:172.21.15.114:3300/0 conn(0x7f393806cde0 0x7f39380b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 >> v1:172.21.15.104:6789/0 conn(0x7f393815c580 legacy=0x7f39380b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3924003a00 0x7f3924003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39240043f0 0x7f39240048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f392400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39380b1e80 con 0x7f39240050b0 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392400c000 con 0x7f3924003a00 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392400c1c0 con 0x7f39240043f0 2023-06-10T18:52:00.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f392400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393d882700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3924003a00 0x7f3924003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.969+0000 7f393d081700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39240043f0 0x7f39240048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 -- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39240043f0 msgr2=0x7f39240048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39240043f0 0x7f39240048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 -- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3924003a00 msgr2=0x7f3924003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:00.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3924003a00 0x7f3924003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 -- 172.21.15.104:0/2058359115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39240035b0 con 0x7f39240050b0 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 -- 172.21.15.104:0/2058359115 reap_dead start 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393e083700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f392400ba10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f393000e350 tx=0x7f3924005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20511afa58405eb6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000edf0 con 0x7f39240050b0 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393000f040 con 0x7f39240050b0 2023-06-10T18:52:00.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393c880700 1 -- 172.21.15.104:0/2058359115 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39300146c0 con 0x7f39240050b0 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393fae6700 1 -- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 msgr2=0x7f392400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393fae6700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f392400ba10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f393000e350 tx=0x7f3924005880 comp rx=0 tx=0).stop 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393fae6700 1 -- 172.21.15.104:0/2058359115 shutdown_connections 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393fae6700 1 --2- 172.21.15.104:0/2058359115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f392400ba10 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.970+0000 7f393fae6700 1 -- 172.21.15.104:0/2058359115 >> 172.21.15.104:0/2058359115 conn(0x7f393806b8a0 msgr2=0x7f39380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:00.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.971+0000 7f393fae6700 1 -- 172.21.15.104:0/2058359115 shutdown_connections 2023-06-10T18:52:00.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.971+0000 7f393fae6700 1 -- 172.21.15.104:0/2058359115 wait complete. 2023-06-10T18:52:00.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 Processor -- start 2023-06-10T18:52:00.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 -- start start 2023-06-10T18:52:00.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f393814d8d0 0x7f393815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f393815f280 0x7f393815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39380c7e90 con 0x7f39240050b0 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393815f6f0 con 0x7f393814d8d0 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393fae6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39380ae5d0 con 0x7f393815f280 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.972+0000 7f393d882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32928/0 (socket says 172.21.15.104:32928) 2023-06-10T18:52:00.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 -- 172.21.15.104:0/2037057077 learned_addr learned my addr 172.21.15.104:0/2037057077 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 -- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f393815f280 msgr2=0x7f393815d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f393815f280 0x7f393815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 -- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f393814d8d0 msgr2=0x7f393815ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f393814d8d0 0x7f393815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 -- 172.21.15.104:0/2037057077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393815df70 con 0x7f39240050b0 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393d882700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f392800f770 tx=0x7f392800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280072e0 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3928007440 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39380ac360 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928005830 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.973+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39380ac920 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.974+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39380ac1c0 con 0x7f39240050b0 2023-06-10T18:52:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.974+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3928005990 con 0x7f39240050b0 2023-06-10T18:52:00.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.975+0000 7f392e7fc700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 0x7f391c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:00.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.975+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f3928089540 con 0x7f39240050b0 2023-06-10T18:52:00.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.975+0000 7f393d081700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 0x7f391c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:00.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.975+0000 7f393d081700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 0x7f391c06d190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3934007fd0 tx=0x7f3934009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:00.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:00.979+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f39380ac1c0 con 0x7f39240050b0 2023-06-10T18:52:01.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.163+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f39380ac1c0 con 0x7f39240050b0 2023-06-10T18:52:01.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.164+0000 7f392e7fc700 1 -- 172.21.15.104:0/2037057077 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v85) v1 ==== 92+0+10 (secure 0 0 0) 0x7f39380ac1c0 con 0x7f39240050b0 2023-06-10T18:52:01.170 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:52:01.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 msgr2=0x7f391c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 0x7f391c06d190 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3934007fd0 tx=0x7f3934009040 comp rx=0 tx=0).stop 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 msgr2=0x7f39380c7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f392800f770 tx=0x7f392800fb30 comp rx=0 tx=0).stop 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 shutdown_connections 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f393815f280 0x7f393815d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39240050b0 0x7f39380c7950 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f391c06ace0 0x7f391c06d190 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 --2- 172.21.15.104:0/2037057077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f393814d8d0 0x7f393815ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 >> 172.21.15.104:0/2037057077 conn(0x7f393806b8a0 msgr2=0x7f39381e5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 shutdown_connections 2023-06-10T18:52:01.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.166+0000 7f393fae6700 1 -- 172.21.15.104:0/2037057077 wait complete. 2023-06-10T18:52:01.183 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:52:01.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.300+0000 7f3f8ed4f700 1 Processor -- start 2023-06-10T18:52:01.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.300+0000 7f3f8ed4f700 1 -- start start 2023-06-10T18:52:01.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8ed4f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f88160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8ed4f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880d0340 con 0x7f3f880cae10 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8ed4f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8815cb60 con 0x7f3f8806c720 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8ed4f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880d1660 con 0x7f3f880ce390 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8d2ec700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3f880cae10 0x7f3f880cb250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59000/0 (socket says 172.21.15.104:59000) 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3326062563 learned_addr learned my addr 172.21.15.104:0/3326062563 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f88160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264450903 0 0) 0x7f3f880d0340 con 0x7f3f880cae10 2023-06-10T18:52:01.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f780035b0 con 0x7f3f880cae10 2023-06-10T18:52:01.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21682328 0 0) 0x7f3f8815cb60 con 0x7f3f8806c720 2023-06-10T18:52:01.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f880d0340 con 0x7f3f8806c720 2023-06-10T18:52:01.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292012775 0 0) 0x7f3f780035b0 con 0x7f3f880cae10 2023-06-10T18:52:01.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 msgr2=0x7f3f88160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f88160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 >> v1:172.21.15.114:6789/0 conn(0x7f3f8806c720 legacy=0x7f3f880ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f88161230 con 0x7f3f880cae10 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3f800019f0 con 0x7f3f880cae10 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 >> v1:172.21.15.104:6789/0 conn(0x7f3f880cae10 legacy=0x7f3f880cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f78003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f78004170 0x7f3f78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.301+0000 7f3f877fe700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004d90 0x7f3f78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880d1660 con 0x7f3f78004170 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7800bb20 con 0x7f3f78004d90 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7800bce0 con 0x7f3f880ce390 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f78004170 0x7f3f78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f78003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f8d2ec700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004d90 0x7f3f78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 -- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 msgr2=0x7f3f78003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f880ce390 0x7f3f78003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 -- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004d90 msgr2=0x7f3f78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004d90 0x7f3f78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 -- 172.21.15.104:0/3326062563 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780035b0 con 0x7f3f78004170 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f87fff700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f78004170 0x7f3f78004850 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f3f74001980 tx=0x7f3f78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff67674dbde8c72 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74001470 con 0x7f3f78004170 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f740052d0 con 0x7f3f78004170 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.302+0000 7f3f877fe700 1 -- 172.21.15.104:0/3326062563 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74003680 con 0x7f3f78004170 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3326062563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f78004170 msgr2=0x7f3f78004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3326062563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f78004170 0x7f3f78004850 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f3f74001980 tx=0x7f3f78005880 comp rx=0 tx=0).stop 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3326062563 reap_dead start 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3326062563 shutdown_connections 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3326062563 >> 172.21.15.104:0/3326062563 conn(0x7f3f88069e70 msgr2=0x7f3f8806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3326062563 shutdown_connections 2023-06-10T18:52:01.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.303+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3326062563 wait complete. 2023-06-10T18:52:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.304+0000 7f3f8ed4f700 1 Processor -- start 2023-06-10T18:52:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.304+0000 7f3f8ed4f700 1 -- start start 2023-06-10T18:52:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 0x7f3f88159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f88159800 0x7f3f88159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881596a0 con 0x7f3f881e6620 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881e6390 con 0x7f3f78004170 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880c57b0 con 0x7f3f88159800 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32952/0 (socket says 172.21.15.104:32952) 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3049809869 learned_addr learned my addr 172.21.15.104:0/3049809869 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:01.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f87fff700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f88159800 0x7f3f88159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 0x7f3f88159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f88159800 msgr2=0x7f3f88159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f88159800 0x7f3f88159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 msgr2=0x7f3f88159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 0x7f3f88159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 -- 172.21.15.104:0/3049809869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f881e73a0 con 0x7f3f881e6620 2023-06-10T18:52:01.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 0x7f3f88159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8d2ec700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3f80002230 tx=0x7f3f8000a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8000ad10 con 0x7f3f881e6620 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8000ae70 con 0x7f3f881e6620 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f880c5a20 con 0x7f3f881e6620 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.305+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80013610 con 0x7f3f881e6620 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.306+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f881e7680 con 0x7f3f881e6620 2023-06-10T18:52:01.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.307+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f880c5610 con 0x7f3f881e6620 2023-06-10T18:52:01.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.307+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3f80013770 con 0x7f3f881e6620 2023-06-10T18:52:01.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.307+0000 7f3f857fa700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 0x7f3f6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.307+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f3f80089e20 con 0x7f3f881e6620 2023-06-10T18:52:01.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.307+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 0x7f3f6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.308+0000 7f3f8caeb700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 0x7f3f6c06d190 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f880b65c0 tx=0x7f3f7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.311+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3f880c5610 con 0x7f3f881e6620 2023-06-10T18:52:01.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.499+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f880c5f40 con 0x7f3f881e6620 2023-06-10T18:52:01.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.500+0000 7f3f857fa700 1 -- 172.21.15.104:0/3049809869 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f880c5f40 con 0x7f3f881e6620 2023-06-10T18:52:01.506 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:52:01.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 msgr2=0x7f3f6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 0x7f3f6c06d190 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f880b65c0 tx=0x7f3f7c008040 comp rx=0 tx=0).stop 2023-06-10T18:52:01.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 msgr2=0x7f3f881e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3f80002230 tx=0x7f3f8000a510 comp rx=0 tx=0).stop 2023-06-10T18:52:01.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 shutdown_connections 2023-06-10T18:52:01.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f6c06ace0 0x7f3f6c06d190 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f78004170 0x7f3f88159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f881e6620 0x7f3f881e6aa0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 --2- 172.21.15.104:0/3049809869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f88159800 0x7f3f88159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 >> 172.21.15.104:0/3049809869 conn(0x7f3f88069e70 msgr2=0x7f3f880cc050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 shutdown_connections 2023-06-10T18:52:01.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.502+0000 7f3f8ed4f700 1 -- 172.21.15.104:0/3049809869 wait complete. 2023-06-10T18:52:01.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:52:01.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.636+0000 7f565e56c700 1 Processor -- start 2023-06-10T18:52:01.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.636+0000 7f565e56c700 1 -- start start 2023-06-10T18:52:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565e56c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56580d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565e56c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f56580ce7d0 0x7f56580cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565e56c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580ce340 con 0x7f56580ccff0 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565e56c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56580cd460 con 0x7f56580ce7d0 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565e56c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56580d00d0 con 0x7f565806af80 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f56580ce7d0 0x7f56580cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f56580ce7d0 0x7f56580cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32960/0 (socket says 172.21.15.104:32960) 2023-06-10T18:52:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 -- 172.21.15.104:0/4116293167 learned_addr learned my addr 172.21.15.104:0/4116293167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:01.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f5657fff700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56580d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 -- 172.21.15.104:0/4116293167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 msgr2=0x7f56580d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56580d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 -- 172.21.15.104:0/4116293167 >> v1:172.21.15.114:6789/0 conn(0x7f56580ccff0 legacy=0x7f565806b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.637+0000 7f565cb09700 1 -- 172.21.15.104:0/4116293167 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56580cd680 con 0x7f56580ce7d0 2023-06-10T18:52:01.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565cb09700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f56580cec50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5650009e80 tx=0x7f565000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f4bbe46ae4a51c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f5656ffd700 1 -- 172.21.15.104:0/4116293167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565000e070 con 0x7f56580ce7d0 2023-06-10T18:52:01.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f5656ffd700 1 -- 172.21.15.104:0/4116293167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650003a20 con 0x7f56580ce7d0 2023-06-10T18:52:01.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f5656ffd700 1 -- 172.21.15.104:0/4116293167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650003dd0 con 0x7f56580ce7d0 2023-06-10T18:52:01.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 -- 172.21.15.104:0/4116293167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 msgr2=0x7f56580cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f56580cec50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f5650009e80 tx=0x7f565000b330 comp rx=0 tx=0).stop 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 -- 172.21.15.104:0/4116293167 shutdown_connections 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f56580cec50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 --2- 172.21.15.104:0/4116293167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56580d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 -- 172.21.15.104:0/4116293167 >> 172.21.15.104:0/4116293167 conn(0x7f5658089fd0 msgr2=0x7f565808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 -- 172.21.15.104:0/4116293167 shutdown_connections 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.638+0000 7f565e56c700 1 -- 172.21.15.104:0/4116293167 wait complete. 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.639+0000 7f565e56c700 1 Processor -- start 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 -- start start 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56581f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56580ccff0 0x7f56580c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658159110 con 0x7f56580ce7d0 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658158540 con 0x7f56580ccff0 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565e56c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56581587e0 con 0x7f565806af80 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565cb09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565cb09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32966/0 (socket says 172.21.15.104:32966) 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.640+0000 7f565cb09700 1 -- 172.21.15.104:0/3554957619 learned_addr learned my addr 172.21.15.104:0/3554957619 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:01.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f5657fff700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56581f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f56577fe700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56580ccff0 0x7f56580c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 -- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 msgr2=0x7f56581f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56581f1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 -- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56580ccff0 msgr2=0x7f56580c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56580ccff0 0x7f56580c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 -- 172.21.15.104:0/3554957619 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650009b30 con 0x7f56580ce7d0 2023-06-10T18:52:01.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565cb09700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5650015040 tx=0x7f5650004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650004240 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56500043a0 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5658158ac0 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650004750 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.641+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5658150e70 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.642+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565806c600 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.642+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5650019070 con 0x7f56580ce7d0 2023-06-10T18:52:01.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.643+0000 7f5654ff9700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 0x7f564406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.643+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f565008a4e0 con 0x7f56580ce7d0 2023-06-10T18:52:01.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.643+0000 7f5657fff700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 0x7f564406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.643+0000 7f5657fff700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 0x7f564406d190 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5648009950 tx=0x7f5648008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.647+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f56500554c0 con 0x7f56580ce7d0 2023-06-10T18:52:01.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.832+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5658151000 con 0x7f56580ce7d0 2023-06-10T18:52:01.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.832+0000 7f5654ff9700 1 -- 172.21.15.104:0/3554957619 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5650017020 con 0x7f56580ce7d0 2023-06-10T18:52:01.838 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:52:01.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 msgr2=0x7f564406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 0x7f564406d190 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5648009950 tx=0x7f5648008040 comp rx=0 tx=0).stop 2023-06-10T18:52:01.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 msgr2=0x7f5658154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f5650015040 tx=0x7f5650004ad0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 shutdown_connections 2023-06-10T18:52:01.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56580ce7d0 0x7f5658154c70 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f565806af80 0x7f56581f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f564406ace0 0x7f564406d190 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 --2- 172.21.15.104:0/3554957619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56580ccff0 0x7f56580c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 >> 172.21.15.104:0/3554957619 conn(0x7f5658089fd0 msgr2=0x7f56580b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 shutdown_connections 2023-06-10T18:52:01.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.834+0000 7f565e56c700 1 -- 172.21.15.104:0/3554957619 wait complete. 2023-06-10T18:52:01.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:52:01.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.966+0000 7f9b84934700 1 Processor -- start 2023-06-10T18:52:01.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.966+0000 7f9b84934700 1 -- start start 2023-06-10T18:52:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.967+0000 7f9b84934700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 0x7f9b800ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.967+0000 7f9b84934700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 0x7f9b800c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.967+0000 7f9b84934700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b800cee20 con 0x7f9b80070580 2023-06-10T18:52:01.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.967+0000 7f9b84934700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b800caa90 con 0x7f9b800ce510 2023-06-10T18:52:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.967+0000 7f9b84934700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b80071fe0 con 0x7f9b80070130 2023-06-10T18:52:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7e59c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 0x7f9b800ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7ed9d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9b80070580 0x7f9b800cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:36644/0 (socket says 172.21.15.104:36644) 2023-06-10T18:52:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 0x7f9b800c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7ed9d700 1 -- 172.21.15.104:0/33334718 learned_addr learned my addr 172.21.15.104:0/33334718 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 -- 172.21.15.104:0/33334718 >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 msgr2=0x7f9b800ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 --2- 172.21.15.104:0/33334718 >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 0x7f9b800ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 -- 172.21.15.104:0/33334718 >> v1:172.21.15.114:6789/0 conn(0x7f9b80070580 legacy=0x7f9b800cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 -- 172.21.15.104:0/33334718 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800722c0 con 0x7f9b80070130 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7e59c700 1 --2- 172.21.15.104:0/33334718 >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 0x7f9b800ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7dd9b700 1 --2- 172.21.15.104:0/33334718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 0x7f9b800c8bd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009840 tx=0x7f9b70009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74df51088f068615 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7d59a700 1 -- 172.21.15.104:0/33334718 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70004230 con 0x7f9b80070130 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7d59a700 1 -- 172.21.15.104:0/33334718 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b70004390 con 0x7f9b80070130 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.968+0000 7f9b7d59a700 1 -- 172.21.15.104:0/33334718 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b700036a0 con 0x7f9b80070130 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 -- 172.21.15.104:0/33334718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 msgr2=0x7f9b800c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 --2- 172.21.15.104:0/33334718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 0x7f9b800c8bd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009840 tx=0x7f9b70009b50 comp rx=0 tx=0).stop 2023-06-10T18:52:01.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 -- 172.21.15.104:0/33334718 shutdown_connections 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 --2- 172.21.15.104:0/33334718 >> v2:172.21.15.114:3300/0 conn(0x7f9b800ce510 0x7f9b800ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 --2- 172.21.15.104:0/33334718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b80070130 0x7f9b800c8bd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 -- 172.21.15.104:0/33334718 >> 172.21.15.104:0/33334718 conn(0x7f9b8006ab00 msgr2=0x7f9b800b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 -- 172.21.15.104:0/33334718 shutdown_connections 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.969+0000 7f9b84934700 1 -- 172.21.15.104:0/33334718 wait complete. 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.970+0000 7f9b84934700 1 Processor -- start 2023-06-10T18:52:01.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.970+0000 7f9b84934700 1 -- start start 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b80070580 0x7f9b801f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b800ce510 0x7f9b801f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801e4920 con 0x7f9b80070130 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8015a9d0 con 0x7f9b80070580 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b84934700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8014e8e0 con 0x7f9b800ce510 2023-06-10T18:52:01.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32974/0 (socket says 172.21.15.104:32974) 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7e59c700 1 -- 172.21.15.104:0/3246568763 learned_addr learned my addr 172.21.15.104:0/3246568763 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7ed9d700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b800ce510 0x7f9b801f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7dd9b700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b80070580 0x7f9b801f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.971+0000 7f9b7e59c700 1 -- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b800ce510 msgr2=0x7f9b801f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b7e59c700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b800ce510 0x7f9b801f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b7e59c700 1 -- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b80070580 msgr2=0x7f9b801f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b7e59c700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b80070580 0x7f9b801f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b7e59c700 1 -- 172.21.15.104:0/3246568763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740094a0 con 0x7f9b80070130 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b7e59c700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400ec10 tx=0x7f9b7400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400c910 con 0x7f9b80070130 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7400ca70 con 0x7f9b80070130 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b700094a0 con 0x7f9b80070130 2023-06-10T18:52:01.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400cde0 con 0x7f9b80070130 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.972+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b801e4640 con 0x7f9b80070130 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.973+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b80074050 con 0x7f9b80070130 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.973+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9b74013470 con 0x7f9b80070130 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.974+0000 7f9b6effd700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 0x7f9b6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.974+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f9b74089780 con 0x7f9b80070130 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.974+0000 7f9b7dd9b700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 0x7f9b6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:01.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.974+0000 7f9b7dd9b700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 0x7f9b6006d190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009840 tx=0x7f9b70013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:01.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:01.978+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9b740546c0 con 0x7f9b80070130 2023-06-10T18:52:02.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.178+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9b800b0910 con 0x7f9b80070130 2023-06-10T18:52:02.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.178+0000 7f9b6effd700 1 -- 172.21.15.104:0/3246568763 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v85) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9b74057ce0 con 0x7f9b80070130 2023-06-10T18:52:02.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 msgr2=0x7f9b6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:02.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 0x7f9b6006d190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009840 tx=0x7f9b70013040 comp rx=0 tx=0).stop 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 msgr2=0x7f9b8014e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400ec10 tx=0x7f9b7400ef20 comp rx=0 tx=0).stop 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 shutdown_connections 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b800ce510 0x7f9b801f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b80070130 0x7f9b8014e3a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9b6006ace0 0x7f9b6006d190 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:02.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 --2- 172.21.15.104:0/3246568763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b80070580 0x7f9b801f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:02.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 >> 172.21.15.104:0/3246568763 conn(0x7f9b8006ab00 msgr2=0x7f9b800b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:02.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 shutdown_connections 2023-06-10T18:52:02.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:02.181+0000 7f9b84934700 1 -- 172.21.15.104:0/3246568763 wait complete. 2023-06-10T18:52:02.188 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:52:02.198 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:52:02.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:52:02.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:52:02.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111326]: fuse finished with error 0 and tester_r 0 2023-06-10T18:52:02.248 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:52:08.349 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:52:08.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:52:08.350 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:52:08.372 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:52:08.372 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:52:08.372 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:52:08.373 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:52:08.373 DEBUG:teuthology.orchestra.run.smithi104:> path = "/home/ubuntu/cephtest/mnt.0/subdir" 2023-06-10T18:52:08.373 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:52:08.374 DEBUG:teuthology.orchestra.run.smithi104:> print("Starting creation...") 2023-06-10T18:52:08.374 DEBUG:teuthology.orchestra.run.smithi104:> start = time.time() 2023-06-10T18:52:08.374 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:52:08.374 DEBUG:teuthology.orchestra.run.smithi104:> os.mkdir(path) 2023-06-10T18:52:08.375 DEBUG:teuthology.orchestra.run.smithi104:> dfd = os.open(path, os.O_DIRECTORY) 2023-06-10T18:52:08.375 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:52:08.375 DEBUG:teuthology.orchestra.run.smithi104:> fd = open(os.path.join(path, "childfile"), "w") 2023-06-10T18:52:08.376 DEBUG:teuthology.orchestra.run.smithi104:> print("Finished creation in {0}s".format(time.time() - start)) 2023-06-10T18:52:08.376 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:52:08.376 DEBUG:teuthology.orchestra.run.smithi104:> print("Starting fsync...") 2023-06-10T18:52:08.376 DEBUG:teuthology.orchestra.run.smithi104:> start = time.time() 2023-06-10T18:52:08.377 DEBUG:teuthology.orchestra.run.smithi104:> if False: 2023-06-10T18:52:08.377 DEBUG:teuthology.orchestra.run.smithi104:> os.fsync(dfd) 2023-06-10T18:52:08.377 DEBUG:teuthology.orchestra.run.smithi104:> else: 2023-06-10T18:52:08.378 DEBUG:teuthology.orchestra.run.smithi104:> os.fsync(fd) 2023-06-10T18:52:08.378 DEBUG:teuthology.orchestra.run.smithi104:> print("Finished fsync in {0}s".format(time.time() - start)) 2023-06-10T18:52:08.378 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:52:08.464 INFO:teuthology.orchestra.run.smithi104.stdout:Starting creation... 2023-06-10T18:52:08.464 INFO:teuthology.orchestra.run.smithi104.stdout:Finished creation in 0.007350444793701172s 2023-06-10T18:52:08.464 INFO:teuthology.orchestra.run.smithi104.stdout:Starting fsync... 2023-06-10T18:52:08.465 INFO:teuthology.orchestra.run.smithi104.stdout:Finished fsync in 0.001958131790161133s 2023-06-10T18:52:08.642 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:52:08.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:52:08.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:52:08.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 Processor -- start 2023-06-10T18:52:08.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 -- start start 2023-06-10T18:52:08.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32f40b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40b6070 con 0x7f32f40b4ff0 2023-06-10T18:52:08.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40b5460 con 0x7f32f40cea60 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.804+0000 7f32fa416700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40b5740 con 0x7f32f415a420 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32ebfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f32f40b4ff0 0x7f32f40ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59042/0 (socket says 172.21.15.104:59042) 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32ebfff700 1 -- 172.21.15.104:0/1368180781 learned_addr learned my addr 172.21.15.104:0/1368180781 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386030976 0 0) 0x7f32f40b6070 con 0x7f32f40b4ff0 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00035b0 con 0x7f32f40b4ff0 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f3fff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32f40b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 918209547 0 0) 0x7f32e00035b0 con 0x7f32f40b4ff0 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 msgr2=0x7f32f40b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32f40b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 >> v1:172.21.15.114:6789/0 conn(0x7f32f40cea60 legacy=0x7f32f415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40b5a20 con 0x7f32f40b4ff0 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f32dc001670 con 0x7f32f40b4ff0 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 >> v1:172.21.15.104:6789/0 conn(0x7f32f40b4ff0 legacy=0x7f32f40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32e0003a30 0x7f32e0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0004850 0x7f32e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32e0005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40b5740 con 0x7f32e0003a30 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e000ff50 con 0x7f32e0004850 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0010110 con 0x7f32f415a420 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f3fff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32e0003a30 0x7f32e0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32ebfff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0004850 0x7f32e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.805+0000 7f32f89b3700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32e0005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 -- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 msgr2=0x7f32e0005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32e0005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 -- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0004850 msgr2=0x7f32e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0004850 0x7f32e0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 -- 172.21.15.104:0/1368180781 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00035b0 con 0x7f32e0003a30 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32ebfff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0004850 0x7f32e0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:08.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f89b3700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f415a420 0x7f32e0005240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f3fff700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32e0003a30 0x7f32e0003e40 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009890 tx=0x7f32e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39446da211e254de server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e400c070 con 0x7f32e0003a30 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e4013610 con 0x7f32e0003a30 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f37fe700 1 -- 172.21.15.104:0/1368180781 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e40139a0 con 0x7f32e0003a30 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 -- 172.21.15.104:0/1368180781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32e0003a30 msgr2=0x7f32e0003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 --2- 172.21.15.104:0/1368180781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32e0003a30 0x7f32e0003e40 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009890 tx=0x7f32e0005880 comp rx=0 tx=0).stop 2023-06-10T18:52:08.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32f89b3700 1 -- 172.21.15.104:0/1368180781 reap_dead start 2023-06-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 -- 172.21.15.104:0/1368180781 shutdown_connections 2023-06-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 -- 172.21.15.104:0/1368180781 >> 172.21.15.104:0/1368180781 conn(0x7f32f4089fd0 msgr2=0x7f32f40d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 -- 172.21.15.104:0/1368180781 shutdown_connections 2023-06-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.806+0000 7f32fa416700 1 -- 172.21.15.104:0/1368180781 wait complete. 2023-06-10T18:52:08.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 Processor -- start 2023-06-10T18:52:08.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 -- start start 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0003a30 0x7f32f4153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40cc840 0x7f32f40c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40cb010 con 0x7f32f4154770 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40c4500 con 0x7f32e0003a30 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32fa416700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40cb260 con 0x7f32f40cc840 2023-06-10T18:52:08.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:32996/0 (socket says 172.21.15.104:32996) 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.808+0000 7f32ebfff700 1 -- 172.21.15.104:0/2823780512 learned_addr learned my addr 172.21.15.104:0/2823780512 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32f3fff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0003a30 0x7f32f4153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32f89b3700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40cc840 0x7f32f40c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 -- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40cc840 msgr2=0x7f32f40c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40cc840 0x7f32f40c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 -- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0003a30 msgr2=0x7f32f4153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:08.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0003a30 0x7f32f4153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 -- 172.21.15.104:0/2823780512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40cb540 con 0x7f32f4154770 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32ebfff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00eaf0 tx=0x7f32dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00c7b0 con 0x7f32f4154770 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc00c910 con 0x7f32f4154770 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f40cb820 con 0x7f32f4154770 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00cc80 con 0x7f32f4154770 2023-06-10T18:52:08.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.809+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f40cbde0 con 0x7f32f4154770 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.810+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f40ce1a0 con 0x7f32f4154770 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.810+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f32dc01d460 con 0x7f32f4154770 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.811+0000 7f32f17fa700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 0x7f32d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.811+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6411+0+0 (secure 0 0 0) 0x7f32dc089430 con 0x7f32f4154770 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.811+0000 7f32f3fff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 0x7f32d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:08.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.811+0000 7f32f3fff700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 0x7f32d406d190 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f32e4016040 tx=0x7f32e400c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:08.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:08.815+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f32dc054410 con 0x7f32f4154770 2023-06-10T18:52:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.041+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f32f40cc000 con 0x7f32f4154770 2023-06-10T18:52:09.615 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:52:09.611+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:52:09.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.621+0000 7f32f17fa700 1 -- 172.21.15.104:0/2823780512 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7f32dc057a30 con 0x7f32f4154770 2023-06-10T18:52:09.628 INFO:tasks.ceph.mds.d.smithi114.stderr: -1746> 2023-06-10T18:51:48.956+0000 7f49a9309700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:52:09.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 msgr2=0x7f32d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 0x7f32d406d190 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f32e4016040 tx=0x7f32e400c000 comp rx=0 tx=0).stop 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 msgr2=0x7f32f40cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00eaf0 tx=0x7f32dc00eeb0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 shutdown_connections 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f32d406ace0 0x7f32d406d190 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32e0003a30 0x7f32f4153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40cc840 0x7f32f40c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 --2- 172.21.15.104:0/2823780512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32f4154770 0x7f32f40cc300 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 >> 172.21.15.104:0/2823780512 conn(0x7f32f4089fd0 msgr2=0x7f32f41556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:09.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 shutdown_connections 2023-06-10T18:52:09.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.624+0000 7f32fa416700 1 -- 172.21.15.104:0/2823780512 wait complete. 2023-06-10T18:52:09.630 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:52:09.642 INFO:tasks.cephfs.mount:Killing connection on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:52:09.642 INFO:tasks.cephfs.mount:Suspending the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:52:09.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:52:09.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 down 2023-06-10T18:52:09.654 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:52:09.681 INFO:tasks.cephfs.mount:Cleaning up killed connection on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:52:09.681 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:52:09.682 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:52:09.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:52:09.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.893+0000 7f10a359e700 1 Processor -- start 2023-06-10T18:52:09.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.893+0000 7f10a359e700 1 -- start start 2023-06-10T18:52:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a359e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f10a406b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a359e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10a40b0c30 0x7f10a40b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a359e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a40b1eb0 con 0x7f10a415fc90 2023-06-10T18:52:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a359e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a415c640 con 0x7f10a40b0c30 2023-06-10T18:52:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a359e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a406b890 con 0x7f10a406af80 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a2d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f10a415fc90 0x7f10a4160110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59064/0 (socket says 172.21.15.104:59064) 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.894+0000 7f10a2d9d700 1 -- 172.21.15.104:0/2349649739 learned_addr learned my addr 172.21.15.104:0/2349649739 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a259c700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f10a406b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/2349649739 >> v2:172.21.15.114:3300/0 conn(0x7f10a40b0c30 0x7f10a40b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35100701 0 0) 0x7f10a40b1eb0 con 0x7f10a415fc90 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c0035b0 con 0x7f10a415fc90 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724112296 0 0) 0x7f108c0035b0 con 0x7f10a415fc90 2023-06-10T18:52:09.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 msgr2=0x7f10a406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f10a406b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 >> v2:172.21.15.114:3300/0 conn(0x7f10a40b0c30 msgr2=0x7f10a40b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 --2- 172.21.15.104:0/2349649739 >> v2:172.21.15.114:3300/0 conn(0x7f10a40b0c30 0x7f10a40b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a40b18f0 con 0x7f10a415fc90 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f109c0019f0 con 0x7f10a415fc90 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 >> v1:172.21.15.104:6789/0 conn(0x7f10a415fc90 legacy=0x7f10a4160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f108c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f108c003f80 0x7f108c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a406b890 con 0x7f108c003f80 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c00ba10 con 0x7f108c004d90 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c00bb50 con 0x7f10a406af80 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f108c003f80 0x7f108c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a259c700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f108c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 -- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 msgr2=0x7f108c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10a406af80 0x7f108c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 -- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f108c004d90 msgr2=0x7f108c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a1d9b700 1 -- 172.21.15.104:0/2349649739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0035b0 con 0x7f108c003f80 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.895+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:09.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f108c003f80 0x7f108c004850 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1094009b60 tx=0x7f108c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd026beb8021124 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400e6c0 con 0x7f108c003f80 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109400e820 con 0x7f108c003f80 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a159a700 1 -- 172.21.15.104:0/2349649739 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400eb90 con 0x7f108c003f80 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 -- 172.21.15.104:0/2349649739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f108c003f80 msgr2=0x7f108c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 --2- 172.21.15.104:0/2349649739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f108c003f80 0x7f108c004850 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1094009b60 tx=0x7f108c005880 comp rx=0 tx=0).stop 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a2d9d700 1 -- 172.21.15.104:0/2349649739 reap_dead start 2023-06-10T18:52:09.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 -- 172.21.15.104:0/2349649739 shutdown_connections 2023-06-10T18:52:09.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 -- 172.21.15.104:0/2349649739 >> 172.21.15.104:0/2349649739 conn(0x7f10a4099980 msgr2=0x7f10a40d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:09.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 -- 172.21.15.104:0/2349649739 shutdown_connections 2023-06-10T18:52:09.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.896+0000 7f10a359e700 1 -- 172.21.15.104:0/2349649739 wait complete. 2023-06-10T18:52:09.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 Processor -- start 2023-06-10T18:52:09.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 -- start start 2023-06-10T18:52:09.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f108c003f80 0x7f10a40c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a40b3e50 0x7f10a415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40b5020 con 0x7f10a4157760 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40b4410 con 0x7f10a40b3e50 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a359e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40b46f0 con 0x7f108c003f80 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33016/0 (socket says 172.21.15.104:33016) 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a2d9d700 1 -- 172.21.15.104:0/1751820020 learned_addr learned my addr 172.21.15.104:0/1751820020 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:09.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.898+0000 7f10a1d9b700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a40b3e50 0x7f10a415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a259c700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f108c003f80 0x7f10a40c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 -- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f108c003f80 msgr2=0x7f10a40c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f108c003f80 0x7f10a40c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 -- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a40b3e50 msgr2=0x7f10a415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a40b3e50 0x7f10a415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 -- 172.21.15.104:0/1751820020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a40b49d0 con 0x7f10a4157760 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a2d9d700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f109c001330 tx=0x7f109c00e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:09.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c0019d0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c00ede0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a40b4cb0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c013660 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.899+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a4157bd0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.900+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a4157dc0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.900+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f109c0137c0 con 0x7f10a4157760 2023-06-10T18:52:09.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.901+0000 7f1092ffd700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:09.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.901+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f109c089b80 con 0x7f10a4157760 2023-06-10T18:52:09.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.901+0000 7f10a259c700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:09.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.901+0000 7f10a259c700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1098009b80 tx=0x7f1098009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:09.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:09.905+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10a4157dc0 con 0x7f10a4157760 2023-06-10T18:52:10.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.092+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a4157dc0 con 0x7f10a4157760 2023-06-10T18:52:10.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.093+0000 7f1092ffd700 1 -- 172.21.15.104:0/1751820020 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+18774 (secure 0 0 0) 0x7f10a4157dc0 con 0x7f10a4157760 2023-06-10T18:52:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:10.098 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":86,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:52:09.042293+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:52:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 msgr2=0x7f108406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1098009b80 tx=0x7f1098009210 comp rx=0 tx=0).stop 2023-06-10T18:52:10.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 msgr2=0x7f10a415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f109c001330 tx=0x7f109c00e670 comp rx=0 tx=0).stop 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 shutdown_connections 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10a4157760 0x7f10a415ae50 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f108c003f80 0x7f10a40c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f108406ace0 0x7f108406d190 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 --2- 172.21.15.104:0/1751820020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10a40b3e50 0x7f10a415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.095+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 >> 172.21.15.104:0/1751820020 conn(0x7f10a4099980 msgr2=0x7f10a40ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.096+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 shutdown_connections 2023-06-10T18:52:10.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.096+0000 7f10a359e700 1 -- 172.21.15.104:0/1751820020 wait complete. 2023-06-10T18:52:10.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:52:10.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:52:10.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.234+0000 7ff450608700 1 Processor -- start 2023-06-10T18:52:10.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.234+0000 7ff450608700 1 -- start start 2023-06-10T18:52:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff450608700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4480cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff450608700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480cab60 con 0x7ff4480ae950 2023-06-10T18:52:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff450608700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480d0540 con 0x7ff44806e0b0 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff450608700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480cc880 con 0x7ff44806a880 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44e3a4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff4480ae950 0x7ff44806d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59080/0 (socket says 172.21.15.104:59080) 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44e3a4700 1 -- 172.21.15.104:0/735764700 learned_addr learned my addr 172.21.15.104:0/735764700 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941547564 0 0) 0x7ff4480cab60 con 0x7ff4480ae950 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4340035b0 con 0x7ff4480ae950 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44eba5700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4480cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766355647 0 0) 0x7ff4340035b0 con 0x7ff4480ae950 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 msgr2=0x7ff4480cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4480cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 >> v1:172.21.15.114:6789/0 conn(0x7ff44806e0b0 legacy=0x7ff4480cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44806cea0 con 0x7ff4480ae950 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff4440017f0 con 0x7ff4480ae950 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 >> v1:172.21.15.104:6789/0 conn(0x7ff4480ae950 legacy=0x7ff44806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff434003a30 0x7ff434003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434004850 0x7ff434004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.235+0000 7ff44d3a2700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4340052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480cc880 con 0x7ff434003a30 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43400bbd0 con 0x7ff434004850 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43400bd90 con 0x7ff44806a880 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff434003a30 0x7ff434003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44eba5700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4340052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44dba3700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434004850 0x7ff434004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 -- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 msgr2=0x7ff4340052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff44806a880 0x7ff4340052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 -- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434004850 msgr2=0x7ff434004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434004850 0x7ff434004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 -- 172.21.15.104:0/735764700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340035b0 con 0x7ff434003a30 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff434003a30 0x7ff434003e40 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c210 tx=0x7ff434005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934ad460de5169ab server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400cd50 con 0x7ff434003a30 2023-06-10T18:52:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4440070d0 con 0x7ff434003a30 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.236+0000 7ff44d3a2700 1 -- 172.21.15.104:0/735764700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444007440 con 0x7ff434003a30 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 -- 172.21.15.104:0/735764700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff434003a30 msgr2=0x7ff434003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 --2- 172.21.15.104:0/735764700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff434003a30 0x7ff434003e40 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c210 tx=0x7ff434005880 comp rx=0 tx=0).stop 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff44eba5700 1 -- 172.21.15.104:0/735764700 reap_dead start 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 -- 172.21.15.104:0/735764700 shutdown_connections 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 -- 172.21.15.104:0/735764700 >> 172.21.15.104:0/735764700 conn(0x7ff44806b8a0 msgr2=0x7ff4480cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 -- 172.21.15.104:0/735764700 shutdown_connections 2023-06-10T18:52:10.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.237+0000 7ff450608700 1 -- 172.21.15.104:0/735764700 wait complete. 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.238+0000 7ff450608700 1 Processor -- start 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.238+0000 7ff450608700 1 -- start start 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 0x7ff448155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 0x7ff4480c5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480b0d20 con 0x7ff4480afba0 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448156530 con 0x7ff434003a30 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480b2940 con 0x7ff4480af730 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33036/0 (socket says 172.21.15.104:33036) 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 -- 172.21.15.104:0/816822842 learned_addr learned my addr 172.21.15.104:0/816822842 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44dba3700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 0x7ff4480c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 0x7ff448155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 -- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 msgr2=0x7ff4480c5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 0x7ff4480c5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 -- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 msgr2=0x7ff448155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 0x7ff448155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 -- 172.21.15.104:0/816822842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480b0110 con 0x7ff4480afba0 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44dba3700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 0x7ff4480c5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 0x7ff448155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:10.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff44eba5700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff44809d610 tx=0x7ff440004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400079c0 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff440007b20 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4480b03f0 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.239+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440015420 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.240+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4480b09b0 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.240+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480cb090 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.241+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff440015580 con 0x7ff4480afba0 2023-06-10T18:52:10.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.241+0000 7ff43effd700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.241+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7ff440089c60 con 0x7ff4480afba0 2023-06-10T18:52:10.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.242+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.242+0000 7ff44e3a4700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 0x7ff42c06d190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff444000ef0 tx=0x7ff444007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.246+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff4480cb090 con 0x7ff4480afba0 2023-06-10T18:52:10.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.430+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4480cb090 con 0x7ff4480afba0 2023-06-10T18:52:10.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.430+0000 7ff43effd700 1 -- 172.21.15.104:0/816822842 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff4480cb090 con 0x7ff4480afba0 2023-06-10T18:52:10.435 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:52:10.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 msgr2=0x7ff42c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 0x7ff42c06d190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff444000ef0 tx=0x7ff444007000 comp rx=0 tx=0).stop 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 msgr2=0x7ff4480c5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff44809d610 tx=0x7ff440004970 comp rx=0 tx=0).stop 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 shutdown_connections 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff434003a30 0x7ff448155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4480afba0 0x7ff4480c5990 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 --2- 172.21.15.104:0/816822842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4480af730 0x7ff4480c5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 >> 172.21.15.104:0/816822842 conn(0x7ff44806b8a0 msgr2=0x7ff4480cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 shutdown_connections 2023-06-10T18:52:10.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.432+0000 7ff450608700 1 -- 172.21.15.104:0/816822842 wait complete. 2023-06-10T18:52:10.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:52:10.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 Processor -- start 2023-06-10T18:52:10.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 -- start start 2023-06-10T18:52:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e6806c720 0x7f5e680ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5e680cae10 0x7f5e680cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e680d0340 con 0x7f5e680ce390 2023-06-10T18:52:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e680d1660 con 0x7f5e6806c720 2023-06-10T18:52:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.566+0000 7f5e6d68a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6815cb60 con 0x7f5e680cae10 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e6806c720 0x7f5e680ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e6806c720 0x7f5e680ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33040/0 (socket says 172.21.15.104:33040) 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 -- 172.21.15.104:0/3262422358 learned_addr learned my addr 172.21.15.104:0/3262422358 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e677fe700 1 --2- 172.21.15.104:0/3262422358 >> v2:172.21.15.114:3300/0 conn(0x7f5e680cae10 0x7f5e680cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 -- 172.21.15.104:0/3262422358 >> v2:172.21.15.114:3300/0 conn(0x7f5e680cae10 msgr2=0x7f5e680cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 --2- 172.21.15.104:0/3262422358 >> v2:172.21.15.114:3300/0 conn(0x7f5e680cae10 0x7f5e680cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 -- 172.21.15.104:0/3262422358 >> v1:172.21.15.114:6789/0 conn(0x7f5e680ce390 legacy=0x7f5e68160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 -- 172.21.15.104:0/3262422358 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680ceca0 con 0x7f5e6806c720 2023-06-10T18:52:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e667fc700 1 --2- 172.21.15.104:0/3262422358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e6806c720 0x7f5e680ce760 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5e680b2a90 tx=0x7f5e58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377dfb1801e89928 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e65ffb700 1 -- 172.21.15.104:0/3262422358 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58011830 con 0x7f5e6806c720 2023-06-10T18:52:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e65ffb700 1 -- 172.21.15.104:0/3262422358 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e58011990 con 0x7f5e6806c720 2023-06-10T18:52:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.567+0000 7f5e65ffb700 1 -- 172.21.15.104:0/3262422358 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58011d20 con 0x7f5e6806c720 2023-06-10T18:52:10.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 -- 172.21.15.104:0/3262422358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e6806c720 msgr2=0x7f5e680ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/3262422358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e6806c720 0x7f5e680ce760 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5e680b2a90 tx=0x7f5e58009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 -- 172.21.15.104:0/3262422358 shutdown_connections 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/3262422358 >> v2:172.21.15.114:3300/0 conn(0x7f5e680cae10 0x7f5e680cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/3262422358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e6806c720 0x7f5e680ce760 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 -- 172.21.15.104:0/3262422358 >> 172.21.15.104:0/3262422358 conn(0x7f5e68069e70 msgr2=0x7f5e6806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 -- 172.21.15.104:0/3262422358 shutdown_connections 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.568+0000 7f5e6d68a700 1 -- 172.21.15.104:0/3262422358 wait complete. 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.569+0000 7f5e6d68a700 1 Processor -- start 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.569+0000 7f5e6d68a700 1 -- start start 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e6806c720 0x7f5e680c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 0x7f5e6815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e680c6850 con 0x7f5e680cae10 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e681e7c80 con 0x7f5e680ce390 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e6d68a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e681e7f60 con 0x7f5e6806c720 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33046/0 (socket says 172.21.15.104:33046) 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 -- 172.21.15.104:0/1328841850 learned_addr learned my addr 172.21.15.104:0/1328841850 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e677fe700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 0x7f5e6815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 -- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e6806c720 msgr2=0x7f5e680c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e66ffd700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e6806c720 0x7f5e680c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e6806c720 0x7f5e680c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 -- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 msgr2=0x7f5e6815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 0x7f5e6815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 -- 172.21.15.104:0/1328841850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e58009550 con 0x7f5e680cae10 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e677fe700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 0x7f5e6815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.570+0000 7f5e667fc700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009c10 tx=0x7f5e58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.571+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580036a0 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.571+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5801e070 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.571+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e681e8240 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.571+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58003a40 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.571+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e681e8800 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.572+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6815bd00 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.572+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5e58003ba0 con 0x7f5e680cae10 2023-06-10T18:52:10.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.572+0000 7f5e4f7fe700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 0x7f5e5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.572+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f5e5808a530 con 0x7f5e680cae10 2023-06-10T18:52:10.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.572+0000 7f5e66ffd700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 0x7f5e5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.573+0000 7f5e66ffd700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 0x7f5e5406d190 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c000f60 tx=0x7f5e5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.577+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5e580554b0 con 0x7f5e680cae10 2023-06-10T18:52:10.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.763+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5e68074050 con 0x7f5e680cae10 2023-06-10T18:52:10.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.764+0000 7f5e4f7fe700 1 -- 172.21.15.104:0/1328841850 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5e58058ad0 con 0x7f5e680cae10 2023-06-10T18:52:10.769 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:52:10.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 msgr2=0x7f5e5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 0x7f5e5406d190 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c000f60 tx=0x7f5e5c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:10.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 msgr2=0x7f5e68159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009c10 tx=0x7f5e58004750 comp rx=0 tx=0).stop 2023-06-10T18:52:10.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 shutdown_connections 2023-06-10T18:52:10.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e5406ace0 0x7f5e5406d190 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e680ce390 0x7f5e6815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e6806c720 0x7f5e680c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 --2- 172.21.15.104:0/1328841850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e680cae10 0x7f5e68159e30 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 >> 172.21.15.104:0/1328841850 conn(0x7f5e68069e70 msgr2=0x7f5e68072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 shutdown_connections 2023-06-10T18:52:10.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.766+0000 7f5e6d68a700 1 -- 172.21.15.104:0/1328841850 wait complete. 2023-06-10T18:52:10.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:52:10.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.899+0000 7f238818f700 1 Processor -- start 2023-06-10T18:52:10.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.900+0000 7f238818f700 1 -- start start 2023-06-10T18:52:10.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.900+0000 7f238818f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2380069e70 0x7f238006a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.900+0000 7f238818f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238015c8d0 con 0x7f238015c500 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.900+0000 7f238818f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238006abd0 con 0x7f238006dcd0 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.900+0000 7f238818f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23800aef20 con 0x7f2380069e70 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f238572a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2380069e70 0x7f238006a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2385f2b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f238015c500 0x7f23800cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59112/0 (socket says 172.21.15.104:59112) 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2385f2b700 1 -- 172.21.15.104:0/1293279444 learned_addr learned my addr 172.21.15.104:0/1293279444 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f238572a700 1 -- 172.21.15.104:0/1293279444 >> v1:172.21.15.114:6789/0 conn(0x7f238006dcd0 legacy=0x7f238006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f238572a700 1 -- 172.21.15.104:0/1293279444 >> v1:172.21.15.104:6789/0 conn(0x7f238015c500 legacy=0x7f23800cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f238572a700 1 -- 172.21.15.104:0/1293279444 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238006a8f0 con 0x7f2380069e70 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2384f29700 1 -- 172.21.15.104:0/1293279444 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889529850 0 0) 0x7f238015c8d0 con 0x7f238015c500 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f238572a700 1 --2- 172.21.15.104:0/1293279444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2380069e70 0x7f238006a310 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2370009870 tx=0x7f2370009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c869d5a0170e5b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2384f29700 1 -- 172.21.15.104:0/1293279444 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370004230 con 0x7f2380069e70 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2384f29700 1 -- 172.21.15.104:0/1293279444 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370004390 con 0x7f2380069e70 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.901+0000 7f2384f29700 1 -- 172.21.15.104:0/1293279444 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700036a0 con 0x7f2380069e70 2023-06-10T18:52:10.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 -- 172.21.15.104:0/1293279444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2380069e70 msgr2=0x7f238006a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 --2- 172.21.15.104:0/1293279444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2380069e70 0x7f238006a310 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2370009870 tx=0x7f2370009b80 comp rx=0 tx=0).stop 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 -- 172.21.15.104:0/1293279444 shutdown_connections 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 --2- 172.21.15.104:0/1293279444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2380069e70 0x7f238006a310 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 -- 172.21.15.104:0/1293279444 >> 172.21.15.104:0/1293279444 conn(0x7f238006b8a0 msgr2=0x7f23800cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 -- 172.21.15.104:0/1293279444 shutdown_connections 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.902+0000 7f238818f700 1 -- 172.21.15.104:0/1293279444 wait complete. 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.903+0000 7f238818f700 1 Processor -- start 2023-06-10T18:52:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.903+0000 7f238818f700 1 -- start start 2023-06-10T18:52:10.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2380069e70 0x7f23801f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f238015c500 0x7f23800cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23801f5000 con 0x7f238006dcd0 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23800b1170 con 0x7f238015c500 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23800b1450 con 0x7f2380069e70 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33060/0 (socket says 172.21.15.104:33060) 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 -- 172.21.15.104:0/2467381937 learned_addr learned my addr 172.21.15.104:0/2467381937 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 -- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2380069e70 msgr2=0x7f23801f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2380069e70 0x7f23801f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 -- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f238015c500 msgr2=0x7f23800cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238672c700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f238015c500 0x7f23800cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f238015c500 0x7f23800cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:10.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 -- 172.21.15.104:0/2467381937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23700094a0 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238572a700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f23700097c0 tx=0x7f23700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370011830 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23700119b0 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23800b1730 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370011d60 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.904+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23800b1cf0 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.905+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2380074050 con 0x7f238006dcd0 2023-06-10T18:52:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.906+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f237001b440 con 0x7f238006dcd0 2023-06-10T18:52:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.906+0000 7f23767fc700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 0x7f236406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.906+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f237008afd0 con 0x7f238006dcd0 2023-06-10T18:52:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.906+0000 7f2385f2b700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 0x7f236406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.907+0000 7f2385f2b700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 0x7f236406d190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f237c0010e0 tx=0x7f237c007850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:10.910+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2370055f50 con 0x7f238006dcd0 2023-06-10T18:52:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.095+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23800ac2c0 con 0x7f238006dcd0 2023-06-10T18:52:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.096+0000 7f23767fc700 1 -- 172.21.15.104:0/2467381937 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2370059570 con 0x7f238006dcd0 2023-06-10T18:52:11.101 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 msgr2=0x7f236406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 0x7f236406d190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f237c0010e0 tx=0x7f237c007850 comp rx=0 tx=0).stop 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 msgr2=0x7f23800cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f23700097c0 tx=0x7f23700047c0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 shutdown_connections 2023-06-10T18:52:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f236406ace0 0x7f236406d190 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f238015c500 0x7f23800cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2380069e70 0x7f23801f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 --2- 172.21.15.104:0/2467381937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f238006dcd0 0x7f23800cb060 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 >> 172.21.15.104:0/2467381937 conn(0x7f238006b8a0 msgr2=0x7f23800ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 shutdown_connections 2023-06-10T18:52:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.098+0000 7f238818f700 1 -- 172.21.15.104:0/2467381937 wait complete. 2023-06-10T18:52:11.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:52:11.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.232+0000 7f96f1802700 1 Processor -- start 2023-06-10T18:52:11.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.232+0000 7f96f1802700 1 -- start start 2023-06-10T18:52:11.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96f1802700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96ec0d04f0 0x7f96ec0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96f1802700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96f1802700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0d0e00 con 0x7f96ec06b510 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96f1802700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec0af530 con 0x7f96ec0d04f0 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96f1802700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0af810 con 0x7f96ec0d25c0 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96ec0d04f0 0x7f96ec0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f96ec06b510 0x7f96ec06b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59130/0 (socket says 172.21.15.104:59130) 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eb7fe700 1 -- 172.21.15.104:0/1553371407 learned_addr learned my addr 172.21.15.104:0/1553371407 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:11.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96ea7fc700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eaffd700 1 -- 172.21.15.104:0/1553371407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 msgr2=0x7f96ec0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eaffd700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec0d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eaffd700 1 -- 172.21.15.104:0/1553371407 >> v1:172.21.15.104:6789/0 conn(0x7f96ec06b510 legacy=0x7f96ec06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96e9ffb700 1 -- 172.21.15.104:0/1553371407 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658528826 0 0) 0x7f96ec0d0e00 con 0x7f96ec06b510 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.233+0000 7f96eaffd700 1 -- 172.21.15.104:0/1553371407 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0afaf0 con 0x7f96ec0d04f0 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96eaffd700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec0d04f0 0x7f96ec0d34d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009820 tx=0x7f96e0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fabfe4acaeb0da server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96e9ffb700 1 -- 172.21.15.104:0/1553371407 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0005e30 con 0x7f96ec0d04f0 2023-06-10T18:52:11.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96e9ffb700 1 -- 172.21.15.104:0/1553371407 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0006030 con 0x7f96ec0d04f0 2023-06-10T18:52:11.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96e9ffb700 1 -- 172.21.15.104:0/1553371407 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00063c0 con 0x7f96ec0d04f0 2023-06-10T18:52:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 -- 172.21.15.104:0/1553371407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec0d04f0 msgr2=0x7f96ec0d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec0d04f0 0x7f96ec0d34d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009820 tx=0x7f96e0009b30 comp rx=0 tx=0).stop 2023-06-10T18:52:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 -- 172.21.15.104:0/1553371407 shutdown_connections 2023-06-10T18:52:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec0d04f0 0x7f96ec0d34d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 --2- 172.21.15.104:0/1553371407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec0d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 -- 172.21.15.104:0/1553371407 >> 172.21.15.104:0/1553371407 conn(0x7f96ec071ec0 msgr2=0x7f96ec09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 -- 172.21.15.104:0/1553371407 shutdown_connections 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.234+0000 7f96f1802700 1 -- 172.21.15.104:0/1553371407 wait complete. 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.236+0000 7f96f1802700 1 Processor -- start 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.236+0000 7f96f1802700 1 -- start start 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96ec0d04f0 0x7f96ec1f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0b6ec0 con 0x7f96ec06b510 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0b3ab0 con 0x7f96ec0d04f0 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0b3d90 con 0x7f96ec0d25c0 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33072/0 (socket says 172.21.15.104:33072) 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 -- 172.21.15.104:0/1972965977 learned_addr learned my addr 172.21.15.104:0/1972965977 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:11.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96ea7fc700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96ec0d04f0 0x7f96ec1f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eb7fe700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 -- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 msgr2=0x7f96ec153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 -- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96ec0d04f0 msgr2=0x7f96ec1f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96ec0d04f0 0x7f96ec1f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 -- 172.21.15.104:0/1972965977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e0009430 con 0x7f96ec06b510 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96eaffd700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00061a0 con 0x7f96ec06b510 2023-06-10T18:52:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0020070 con 0x7f96ec06b510 2023-06-10T18:52:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec0b4070 con 0x7f96ec06b510 2023-06-10T18:52:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0006540 con 0x7f96ec06b510 2023-06-10T18:52:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.237+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec0b33b0 con 0x7f96ec06b510 2023-06-10T18:52:11.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.238+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec0b4590 con 0x7f96ec06b510 2023-06-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.239+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96e0022030 con 0x7f96ec06b510 2023-06-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.239+0000 7f96d3fff700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 0x7f96d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.239+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f96e008aab0 con 0x7f96ec06b510 2023-06-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.239+0000 7f96ea7fc700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 0x7f96d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.240+0000 7f96ea7fc700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 0x7f96d806d190 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0cb1d0 tx=0x7f96dc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.243+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96e0055a30 con 0x7f96ec06b510 2023-06-10T18:52:11.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.430+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f96ec0b7030 con 0x7f96ec06b510 2023-06-10T18:52:11.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.430+0000 7f96d3fff700 1 -- 172.21.15.104:0/1972965977 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f96ec0b7030 con 0x7f96ec06b510 2023-06-10T18:52:11.435 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:52:11.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 msgr2=0x7f96d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 0x7f96d806d190 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0cb1d0 tx=0x7f96dc009210 comp rx=0 tx=0).stop 2023-06-10T18:52:11.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 msgr2=0x7f96ec1e4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e0004750 comp rx=0 tx=0).stop 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 shutdown_connections 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96ec0d25c0 0x7f96ec153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ec06b510 0x7f96ec1e4560 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96d806ace0 0x7f96d806d190 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 --2- 172.21.15.104:0/1972965977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96ec0d04f0 0x7f96ec1f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 >> 172.21.15.104:0/1972965977 conn(0x7f96ec071ec0 msgr2=0x7f96ec0cb790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 shutdown_connections 2023-06-10T18:52:11.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.432+0000 7f96f1802700 1 -- 172.21.15.104:0/1972965977 wait complete. 2023-06-10T18:52:11.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:52:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.565+0000 7fdd53641700 1 Processor -- start 2023-06-10T18:52:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.565+0000 7fdd53641700 1 -- start start 2023-06-10T18:52:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd53641700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd4c163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd53641700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c0c9e60 con 0x7fdd4c0cd850 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd53641700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c0aee70 con 0x7fdd4c0aeaa0 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd53641700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c9bc0 con 0x7fdd4c0cb300 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd50bdc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdd4c0cd850 0x7fdd4c06cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59142/0 (socket says 172.21.15.104:59142) 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd50bdc700 1 -- 172.21.15.104:0/1578867691 learned_addr learned my addr 172.21.15.104:0/1578867691 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd51bde700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd4c163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264781883 0 0) 0x7fdd4c0c9e60 con 0x7fdd4c0cd850 2023-06-10T18:52:11.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd380035b0 con 0x7fdd4c0cd850 2023-06-10T18:52:11.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337768881 0 0) 0x7fdd380035b0 con 0x7fdd4c0cd850 2023-06-10T18:52:11.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 msgr2=0x7fdd4c163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd4c163c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 >> v1:172.21.15.114:6789/0 conn(0x7fdd4c0aeaa0 legacy=0x7fdd4c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c06e540 con 0x7fdd4c0cd850 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.566+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdd48001670 con 0x7fdd4c0cd850 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 >> v1:172.21.15.104:6789/0 conn(0x7fdd4c0cd850 legacy=0x7fdd4c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd38003a30 0x7fdd38003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd38004850 0x7fdd38004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd38005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c9bc0 con 0x7fdd38004850 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3800bac0 con 0x7fdd38003a30 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3800bc50 con 0x7fdd4c0cb300 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd38004850 0x7fdd38004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd51bde700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd38005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd513dd700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd38003a30 0x7fdd38003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 -- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 msgr2=0x7fdd38005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0cb300 0x7fdd38005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 -- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd38003a30 msgr2=0x7fdd38003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd38003a30 0x7fdd38003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 -- 172.21.15.104:0/1578867691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380035b0 con 0x7fdd38004850 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd38004850 0x7fdd38004cc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800c240 tx=0x7fdd38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558ee0e75864d7c7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48007c10 con 0x7fdd38004850 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4800ce00 con 0x7fdd38004850 2023-06-10T18:52:11.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.567+0000 7fdd43fff700 1 -- 172.21.15.104:0/1578867691 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48005ba0 con 0x7fdd38004850 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 -- 172.21.15.104:0/1578867691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd38004850 msgr2=0x7fdd38004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 --2- 172.21.15.104:0/1578867691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd38004850 0x7fdd38004cc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800c240 tx=0x7fdd38005880 comp rx=0 tx=0).stop 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd51bde700 1 -- 172.21.15.104:0/1578867691 reap_dead start 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 -- 172.21.15.104:0/1578867691 shutdown_connections 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 -- 172.21.15.104:0/1578867691 >> 172.21.15.104:0/1578867691 conn(0x7fdd4c06b8a0 msgr2=0x7fdd4c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 -- 172.21.15.104:0/1578867691 shutdown_connections 2023-06-10T18:52:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.568+0000 7fdd53641700 1 -- 172.21.15.104:0/1578867691 wait complete. 2023-06-10T18:52:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.569+0000 7fdd53641700 1 Processor -- start 2023-06-10T18:52:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.569+0000 7fdd53641700 1 -- start start 2023-06-10T18:52:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0b2be0 0x7fdd4c15ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd4c15bfc0 0x7fdd4c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c14f3d0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c3fe0 con 0x7fdd4c15bfc0 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd53641700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c33d0 con 0x7fdd4c0b2be0 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33092/0 (socket says 172.21.15.104:33092) 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 -- 172.21.15.104:0/203081522 learned_addr learned my addr 172.21.15.104:0/203081522 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 -- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0b2be0 msgr2=0x7fdd4c15ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0b2be0 0x7fdd4c15ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 -- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd4c15bfc0 msgr2=0x7fdd4c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd4c15bfc0 0x7fdd4c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 -- 172.21.15.104:0/203081522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0c36b0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd513dd700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00f820 tx=0x7fdd3c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.570+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0052e0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.571+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c005440 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.571+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c0c3990 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.571+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0038f0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.571+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c0c3060 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.572+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c0c42c0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.572+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdd3c003a50 con 0x7fdd4c0b4830 2023-06-10T18:52:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.573+0000 7fdd41ffb700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 0x7fdd3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.573+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fdd3c0893b0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.573+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 0x7fdd3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.573+0000 7fdd50bdc700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 0x7fdd3006d190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0c24b0 tx=0x7fdd4800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.577+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdd3c0542b0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.780+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdd4c074050 con 0x7fdd4c0b4830 2023-06-10T18:52:11.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.780+0000 7fdd41ffb700 1 -- 172.21.15.104:0/203081522 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v86) v1 ==== 80+0+122 (secure 0 0 0) 0x7fdd3c0578d0 con 0x7fdd4c0b4830 2023-06-10T18:52:11.785 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6836/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:52:11.785 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6834/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:52:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.782+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 msgr2=0x7fdd3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.782+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 0x7fdd3006d190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0c24b0 tx=0x7fdd4800a000 comp rx=0 tx=0).stop 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.782+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 msgr2=0x7fdd4c0b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.782+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00f820 tx=0x7fdd3c00fbe0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 shutdown_connections 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd3006ace0 0x7fdd3006d190 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd4c15bfc0 0x7fdd4c069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd4c0b4830 0x7fdd4c0b26a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 --2- 172.21.15.104:0/203081522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd4c0b2be0 0x7fdd4c15ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 >> 172.21.15.104:0/203081522 conn(0x7fdd4c06b8a0 msgr2=0x7fdd4c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 shutdown_connections 2023-06-10T18:52:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.783+0000 7fdd53641700 1 -- 172.21.15.104:0/203081522 wait complete. 2023-06-10T18:52:11.789 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:52:11.799 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:52:11.799 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:52:11.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:52:11.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.970+0000 7f0659a64700 1 Processor -- start 2023-06-10T18:52:11.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.970+0000 7f0659a64700 1 -- start start 2023-06-10T18:52:11.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f065406e0e0 0x7f065406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f065406e4b0 0x7f06540c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0654071640 con 0x7f065406e0e0 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06540aef00 con 0x7f065406e4b0 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0659a64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065406d290 con 0x7f06540ca100 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f065406e0e0 0x7f065406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f065406e0e0 0x7f065406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33100/0 (socket says 172.21.15.104:33100) 2023-06-10T18:52:11.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 -- 172.21.15.104:0/2824164764 learned_addr learned my addr 172.21.15.104:0/2824164764 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0653fff700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f0652ffd700 1 --2- 172.21.15.104:0/2824164764 >> v2:172.21.15.114:3300/0 conn(0x7f065406e4b0 0x7f06540c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 -- 172.21.15.104:0/2824164764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 msgr2=0x7f06540ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 -- 172.21.15.104:0/2824164764 >> v2:172.21.15.114:3300/0 conn(0x7f065406e4b0 msgr2=0x7f06540c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 --2- 172.21.15.104:0/2824164764 >> v2:172.21.15.114:3300/0 conn(0x7f065406e4b0 0x7f06540c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.971+0000 7f06537fe700 1 -- 172.21.15.104:0/2824164764 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065406d4c0 con 0x7f065406e0e0 2023-06-10T18:52:11.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f06537fe700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e0e0 0x7f065406cd50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0654070320 tx=0x7f0648009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f149bd49fb315f07 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f06527fc700 1 -- 172.21.15.104:0/2824164764 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648005e30 con 0x7f065406e0e0 2023-06-10T18:52:11.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f06527fc700 1 -- 172.21.15.104:0/2824164764 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648004030 con 0x7f065406e0e0 2023-06-10T18:52:11.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f06527fc700 1 -- 172.21.15.104:0/2824164764 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480043e0 con 0x7f065406e0e0 2023-06-10T18:52:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 -- 172.21.15.104:0/2824164764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e0e0 msgr2=0x7f065406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e0e0 0x7f065406cd50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0654070320 tx=0x7f0648009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 -- 172.21.15.104:0/2824164764 shutdown_connections 2023-06-10T18:52:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e0e0 0x7f065406cd50 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 --2- 172.21.15.104:0/2824164764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 --2- 172.21.15.104:0/2824164764 >> v2:172.21.15.114:3300/0 conn(0x7f065406e4b0 0x7f06540c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 -- 172.21.15.104:0/2824164764 >> 172.21.15.104:0/2824164764 conn(0x7f065406b8a0 msgr2=0x7f06540cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 -- 172.21.15.104:0/2824164764 shutdown_connections 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.972+0000 7f0659a64700 1 -- 172.21.15.104:0/2824164764 wait complete. 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.973+0000 7f0659a64700 1 Processor -- start 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 -- start start 2023-06-10T18:52:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065406e0e0 0x7f06540b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b3e80 con 0x7f065406e4b0 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b0b10 con 0x7f065406e0e0 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0659a64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06540b0db0 con 0x7f06540ca100 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0652ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33110/0 (socket says 172.21.15.104:33110) 2023-06-10T18:52:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.974+0000 7f0652ffd700 1 -- 172.21.15.104:0/311047376 learned_addr learned my addr 172.21.15.104:0/311047376 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f06537fe700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065406e0e0 0x7f06540b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0653fff700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 -- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 msgr2=0x7f06540c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 -- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065406e0e0 msgr2=0x7f06540b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065406e0e0 0x7f06540b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:11.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 -- 172.21.15.104:0/311047376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648009500 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0652ffd700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f064400e810 tx=0x7f064400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400c070 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644013610 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06540b1090 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440139a0 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.975+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540b1650 con 0x7f065406e4b0 2023-06-10T18:52:11.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.976+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0654074050 con 0x7f065406e4b0 2023-06-10T18:52:11.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.977+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0644013b00 con 0x7f065406e4b0 2023-06-10T18:52:11.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.977+0000 7f063bfff700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 0x7f064006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:11.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.977+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f0644089760 con 0x7f065406e4b0 2023-06-10T18:52:11.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.977+0000 7f06537fe700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 0x7f064006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:11.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.978+0000 7f06537fe700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 0x7f064006d190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0654070320 tx=0x7f0648008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:11.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:11.981+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0644054730 con 0x7f065406e4b0 2023-06-10T18:52:12.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.170+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06540ca720 con 0x7f065406e4b0 2023-06-10T18:52:12.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.171+0000 7f063bfff700 1 -- 172.21.15.104:0/311047376 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+18774 (secure 0 0 0) 0x7f0644057d50 con 0x7f065406e4b0 2023-06-10T18:52:12.176 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:12.176 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":86,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:52:09.042293+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:52:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 msgr2=0x7f064006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 0x7f064006d190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0654070320 tx=0x7f0648008070 comp rx=0 tx=0).stop 2023-06-10T18:52:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 msgr2=0x7f06541f1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f064400e810 tx=0x7f064400eb20 comp rx=0 tx=0).stop 2023-06-10T18:52:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 shutdown_connections 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f064006ace0 0x7f064006d190 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065406e0e0 0x7f06540b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06540ca100 0x7f06540c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 --2- 172.21.15.104:0/311047376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f065406e4b0 0x7f06541f1840 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 >> 172.21.15.104:0/311047376 conn(0x7f065406b8a0 msgr2=0x7f065415aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 shutdown_connections 2023-06-10T18:52:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.173+0000 7f0659a64700 1 -- 172.21.15.104:0/311047376 wait complete. 2023-06-10T18:52:12.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:52:12.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:52:12.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.309+0000 7fde09dc6700 1 Processor -- start 2023-06-10T18:52:12.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.309+0000 7fde09dc6700 1 -- start start 2023-06-10T18:52:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde09dc6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde04070a10 0x7fde04070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde09dc6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fde0406a5e0 0x7fde0406aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde09dc6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde040713a0 con 0x7fde04145cc0 2023-06-10T18:52:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde09dc6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde040c4770 con 0x7fde0406a5e0 2023-06-10T18:52:12.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde09dc6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde04069e70 con 0x7fde04070a10 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fde0406a5e0 0x7fde0406aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fde0406a5e0 0x7fde0406aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33118/0 (socket says 172.21.15.104:33118) 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 -- 172.21.15.104:0/2741999161 learned_addr learned my addr 172.21.15.104:0/2741999161 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde02ffd700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde04070a10 0x7fde04070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 -- 172.21.15.104:0/2741999161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde04070a10 msgr2=0x7fde04070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde04070a10 0x7fde04070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 -- 172.21.15.104:0/2741999161 >> v1:172.21.15.114:6789/0 conn(0x7fde04145cc0 legacy=0x7fde0406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:12.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 -- 172.21.15.104:0/2741999161 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0406cf10 con 0x7fde0406a5e0 2023-06-10T18:52:12.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde03fff700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0406a5e0 0x7fde0406aa20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b865eccd77d198 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde027fc700 1 -- 172.21.15.104:0/2741999161 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00e070 con 0x7fde0406a5e0 2023-06-10T18:52:12.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde027fc700 1 -- 172.21.15.104:0/2741999161 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc003a20 con 0x7fde0406a5e0 2023-06-10T18:52:12.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.310+0000 7fde027fc700 1 -- 172.21.15.104:0/2741999161 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc003dd0 con 0x7fde0406a5e0 2023-06-10T18:52:12.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 -- 172.21.15.104:0/2741999161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0406a5e0 msgr2=0x7fde0406aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0406a5e0 0x7fde0406aa20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc00b330 comp rx=0 tx=0).stop 2023-06-10T18:52:12.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 -- 172.21.15.104:0/2741999161 shutdown_connections 2023-06-10T18:52:12.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0406a5e0 0x7fde0406aa20 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 --2- 172.21.15.104:0/2741999161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde04070a10 0x7fde04070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 -- 172.21.15.104:0/2741999161 >> 172.21.15.104:0/2741999161 conn(0x7fde040ceec0 msgr2=0x7fde040cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 -- 172.21.15.104:0/2741999161 shutdown_connections 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.311+0000 7fde09dc6700 1 -- 172.21.15.104:0/2741999161 wait complete. 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.312+0000 7fde09dc6700 1 Processor -- start 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.313+0000 7fde09dc6700 1 -- start start 2023-06-10T18:52:12.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde0406a5e0 0x7fde0415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde04070a10 0x7fde040ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde041f1940 con 0x7fde04145cc0 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0415d400 con 0x7fde04070a10 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0415c810 con 0x7fde0406a5e0 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33124/0 (socket says 172.21.15.104:33124) 2023-06-10T18:52:12.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 -- 172.21.15.104:0/369014903 learned_addr learned my addr 172.21.15.104:0/369014903 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde02ffd700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde04070a10 0x7fde040ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde037fe700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde0406a5e0 0x7fde0415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 -- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde0406a5e0 msgr2=0x7fde0415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde0406a5e0 0x7fde0415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 -- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde04070a10 msgr2=0x7fde040ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde04070a10 0x7fde040ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 -- 172.21.15.104:0/369014903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc009b30 con 0x7fde04145cc0 2023-06-10T18:52:12.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde03fff700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fde04065c30 tx=0x7fddfc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004440 con 0x7fde04145cc0 2023-06-10T18:52:12.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0045a0 con 0x7fde04145cc0 2023-06-10T18:52:12.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0415cad0 con 0x7fde04145cc0 2023-06-10T18:52:12.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004950 con 0x7fde04145cc0 2023-06-10T18:52:12.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.314+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0415d090 con 0x7fde04145cc0 2023-06-10T18:52:12.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.315+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0415d5b0 con 0x7fde04145cc0 2023-06-10T18:52:12.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.316+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fddfc025450 con 0x7fde04145cc0 2023-06-10T18:52:12.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.316+0000 7fddebfff700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 0x7fddf006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.316+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fddfc08add0 con 0x7fde04145cc0 2023-06-10T18:52:12.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.316+0000 7fde037fe700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 0x7fddf006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.317+0000 7fde037fe700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 0x7fddf006d190 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fde040ae190 tx=0x7fddf4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.320+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde0415d5b0 con 0x7fde04145cc0 2023-06-10T18:52:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.512+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde040713a0 con 0x7fde04145cc0 2023-06-10T18:52:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.512+0000 7fddebfff700 1 -- 172.21.15.104:0/369014903 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde040713a0 con 0x7fde04145cc0 2023-06-10T18:52:12.517 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 msgr2=0x7fddf006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 0x7fddf006d190 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fde040ae190 tx=0x7fddf4008040 comp rx=0 tx=0).stop 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 msgr2=0x7fde040ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fde04065c30 tx=0x7fddfc004ad0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 shutdown_connections 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde04145cc0 0x7fde040ad860 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde0406a5e0 0x7fde0415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fddf006ace0 0x7fddf006d190 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 --2- 172.21.15.104:0/369014903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde04070a10 0x7fde040ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 >> 172.21.15.104:0/369014903 conn(0x7fde040ceec0 msgr2=0x7fde04071690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 shutdown_connections 2023-06-10T18:52:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.514+0000 7fde09dc6700 1 -- 172.21.15.104:0/369014903 wait complete. 2023-06-10T18:52:12.530 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:52:12.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.649+0000 7fda5fdfb700 1 Processor -- start 2023-06-10T18:52:12.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5fdfb700 1 -- start start 2023-06-10T18:52:12.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5fdfb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda5806ea70 0x7fda5806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5fdfb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580ae7e0 con 0x7fda580b1780 2023-06-10T18:52:12.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5fdfb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5806df90 con 0x7fda5815c6f0 2023-06-10T18:52:12.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5fdfb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5806e270 con 0x7fda5806ea70 2023-06-10T18:52:12.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.650+0000 7fda5e398700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fda5806ea70 0x7fda5806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5d396700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fda580b1780 0x7fda5806cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59188/0 (socket says 172.21.15.104:59188) 2023-06-10T18:52:12.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5d396700 1 -- 172.21.15.104:0/1852715729 learned_addr learned my addr 172.21.15.104:0/1852715729 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:12.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5e398700 1 -- 172.21.15.104:0/1852715729 >> v1:172.21.15.114:6789/0 conn(0x7fda5815c6f0 legacy=0x7fda580b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:12.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5e398700 1 -- 172.21.15.104:0/1852715729 >> v1:172.21.15.104:6789/0 conn(0x7fda580b1780 legacy=0x7fda5806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5cb95700 1 -- 172.21.15.104:0/1852715729 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639019754 0 0) 0x7fda580ae7e0 con 0x7fda580b1780 2023-06-10T18:52:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5e398700 1 -- 172.21.15.104:0/1852715729 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5806e550 con 0x7fda5806ea70 2023-06-10T18:52:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5e398700 1 --2- 172.21.15.104:0/1852715729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda5806d4f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fda50009d90 tx=0x7fda5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa66ebc36c67ecfe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5cb95700 1 -- 172.21.15.104:0/1852715729 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000e070 con 0x7fda5806ea70 2023-06-10T18:52:12.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5cb95700 1 -- 172.21.15.104:0/1852715729 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5000bb90 con 0x7fda5806ea70 2023-06-10T18:52:12.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.651+0000 7fda5cb95700 1 -- 172.21.15.104:0/1852715729 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50003a20 con 0x7fda5806ea70 2023-06-10T18:52:12.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 -- 172.21.15.104:0/1852715729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 msgr2=0x7fda5806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/1852715729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda5806d4f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fda50009d90 tx=0x7fda5000b330 comp rx=0 tx=0).stop 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 -- 172.21.15.104:0/1852715729 shutdown_connections 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/1852715729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda5806d4f0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 -- 172.21.15.104:0/1852715729 >> 172.21.15.104:0/1852715729 conn(0x7fda5806b8a0 msgr2=0x7fda580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 -- 172.21.15.104:0/1852715729 shutdown_connections 2023-06-10T18:52:12.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.652+0000 7fda5fdfb700 1 -- 172.21.15.104:0/1852715729 wait complete. 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.653+0000 7fda5fdfb700 1 Processor -- start 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.653+0000 7fda5fdfb700 1 -- start start 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580b1780 0x7fda580c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 0x7fda580c2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b3c30 con 0x7fda5806ea70 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580c5f40 con 0x7fda580b1780 2023-06-10T18:52:12.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda580c2d50 con 0x7fda5815c6f0 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33134/0 (socket says 172.21.15.104:33134) 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 -- 172.21.15.104:0/3797559312 learned_addr learned my addr 172.21.15.104:0/3797559312 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5d396700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580b1780 0x7fda580c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5e398700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 0x7fda580c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 -- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 msgr2=0x7fda580c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 0x7fda580c2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 -- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580b1780 msgr2=0x7fda580c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580b1780 0x7fda580c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 -- 172.21.15.104:0/3797559312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50009a70 con 0x7fda5806ea70 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5e398700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 0x7fda580c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5db97700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fda4800b450 tx=0x7fda4800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48004020 con 0x7fda5806ea70 2023-06-10T18:52:12.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48004180 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda580b2fb0 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48004530 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.654+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda580b3570 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.655+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda58074050 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.656+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fda480036d0 con 0x7fda5806ea70 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.656+0000 7fda4e7fc700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 0x7fda3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.656+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fda48088e10 con 0x7fda5806ea70 2023-06-10T18:52:12.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.657+0000 7fda5d396700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 0x7fda3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.657+0000 7fda5d396700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 0x7fda3c06d190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fda5809d510 tx=0x7fda54007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.661+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fda48053d00 con 0x7fda5806ea70 2023-06-10T18:52:12.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.847+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fda580aebd0 con 0x7fda5806ea70 2023-06-10T18:52:12.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.848+0000 7fda4e7fc700 1 -- 172.21.15.104:0/3797559312 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7fda480150a0 con 0x7fda5806ea70 2023-06-10T18:52:12.852 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:52:12.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.849+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 msgr2=0x7fda3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.849+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 0x7fda3c06d190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fda5809d510 tx=0x7fda54007de0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.849+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 msgr2=0x7fda580b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.849+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fda4800b450 tx=0x7fda4800b760 comp rx=0 tx=0).stop 2023-06-10T18:52:12.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 shutdown_connections 2023-06-10T18:52:12.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda5815c6f0 0x7fda580c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda5806ea70 0x7fda580b28f0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda3c06ace0 0x7fda3c06d190 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 --2- 172.21.15.104:0/3797559312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda580b1780 0x7fda580c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 >> 172.21.15.104:0/3797559312 conn(0x7fda5806b8a0 msgr2=0x7fda5815b890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 shutdown_connections 2023-06-10T18:52:12.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.850+0000 7fda5fdfb700 1 -- 172.21.15.104:0/3797559312 wait complete. 2023-06-10T18:52:12.866 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:52:12.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.984+0000 7f5e9e13e700 1 Processor -- start 2023-06-10T18:52:12.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.984+0000 7f5e9e13e700 1 -- start start 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e9e13e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 0x7f5e9815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e9e13e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e980ce670 0x7f5e9806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e9e13e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980c4980 con 0x7f5e98070a40 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e9e13e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9815d0b0 con 0x7f5e980ce670 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e9e13e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9815c810 con 0x7f5e98158830 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5e980ce670 0x7f5e9806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e97fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5e98070a40 0x7f5e98070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59206/0 (socket says 172.21.15.104:59206) 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e97fff700 1 -- 172.21.15.104:0/1705137207 learned_addr learned my addr 172.21.15.104:0/1705137207 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e977fe700 1 --2- 172.21.15.104:0/1705137207 >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 0x7f5e9815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 -- 172.21.15.104:0/1705137207 >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 msgr2=0x7f5e9815c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 --2- 172.21.15.104:0/1705137207 >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 0x7f5e9815c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 -- 172.21.15.104:0/1705137207 >> v1:172.21.15.104:6789/0 conn(0x7f5e98070a40 legacy=0x7f5e98070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 -- 172.21.15.104:0/1705137207 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9815caf0 con 0x7f5e980ce670 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e967fc700 1 -- 172.21.15.104:0/1705137207 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845791436 0 0) 0x7f5e980c4980 con 0x7f5e98070a40 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e977fe700 1 --2- 172.21.15.104:0/1705137207 >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 0x7f5e9815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e96ffd700 1 --2- 172.21.15.104:0/1705137207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980ce670 0x7f5e9806cf30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5e9806ea40 tx=0x7f5e88009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d00f4321f3d2ff9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.985+0000 7f5e967fc700 1 -- 172.21.15.104:0/1705137207 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88011830 con 0x7f5e980ce670 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e967fc700 1 -- 172.21.15.104:0/1705137207 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e88011990 con 0x7f5e980ce670 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e967fc700 1 -- 172.21.15.104:0/1705137207 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88011d20 con 0x7f5e980ce670 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 -- 172.21.15.104:0/1705137207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980ce670 msgr2=0x7f5e9806cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/1705137207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980ce670 0x7f5e9806cf30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5e9806ea40 tx=0x7f5e88009b20 comp rx=0 tx=0).stop 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 -- 172.21.15.104:0/1705137207 shutdown_connections 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/1705137207 >> v2:172.21.15.114:3300/0 conn(0x7f5e98158830 0x7f5e9815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/1705137207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e980ce670 0x7f5e9806cf30 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 -- 172.21.15.104:0/1705137207 >> 172.21.15.104:0/1705137207 conn(0x7f5e980cedc0 msgr2=0x7f5e980cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 -- 172.21.15.104:0/1705137207 shutdown_connections 2023-06-10T18:52:12.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.986+0000 7f5e9e13e700 1 -- 172.21.15.104:0/1705137207 wait complete. 2023-06-10T18:52:12.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.987+0000 7f5e9e13e700 1 Processor -- start 2023-06-10T18:52:12.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 -- start start 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e98070a40 0x7f5e981f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e980ce670 0x7f5e981f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e981f6680 con 0x7f5e981575e0 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98157d50 con 0x7f5e980ce670 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e9e13e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98158030 con 0x7f5e98070a40 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33150/0 (socket says 172.21.15.104:33150) 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.988+0000 7f5e97fff700 1 -- 172.21.15.104:0/3982336110 learned_addr learned my addr 172.21.15.104:0/3982336110 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e977fe700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e98070a40 0x7f5e981f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 -- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e98070a40 msgr2=0x7f5e981f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e98070a40 0x7f5e981f2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 -- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e980ce670 msgr2=0x7f5e981f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e980ce670 0x7f5e981f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:12.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 -- 172.21.15.104:0/3982336110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880094a0 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e97fff700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000cca0 tx=0x7f5e9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9000cea0 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9000ade0 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e981f2250 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90007310 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.989+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e981f2810 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.990+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e981f2a30 con 0x7f5e981575e0 2023-06-10T18:52:12.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.991+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5e90007470 con 0x7f5e981575e0 2023-06-10T18:52:13.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.991+0000 7f5e7ffff700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 0x7f5e8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.991+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f5e90089810 con 0x7f5e981575e0 2023-06-10T18:52:13.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.991+0000 7f5e977fe700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 0x7f5e8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.992+0000 7f5e977fe700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 0x7f5e8406d190 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009750 tx=0x7f5e8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:12.995+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5e90054870 con 0x7f5e981575e0 2023-06-10T18:52:13.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.180+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e98074050 con 0x7f5e981575e0 2023-06-10T18:52:13.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.180+0000 7f5e7ffff700 1 -- 172.21.15.104:0/3982336110 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e90057e90 con 0x7f5e981575e0 2023-06-10T18:52:13.185 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:52:13.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.182+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 msgr2=0x7f5e8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.182+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 0x7f5e8406d190 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009750 tx=0x7f5e8c008040 comp rx=0 tx=0).stop 2023-06-10T18:52:13.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.182+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 msgr2=0x7f5e981ef470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.182+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000cca0 tx=0x7f5e9000a340 comp rx=0 tx=0).stop 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 shutdown_connections 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e981575e0 0x7f5e981ef470 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5e98070a40 0x7f5e981f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5e8406ace0 0x7f5e8406d190 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 --2- 172.21.15.104:0/3982336110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e980ce670 0x7f5e981f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 >> 172.21.15.104:0/3982336110 conn(0x7f5e980cedc0 msgr2=0x7f5e980cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:13.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 shutdown_connections 2023-06-10T18:52:13.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.183+0000 7f5e9e13e700 1 -- 172.21.15.104:0/3982336110 wait complete. 2023-06-10T18:52:13.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:52:13.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.317+0000 7f17a741a700 1 Processor -- start 2023-06-10T18:52:13.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.317+0000 7f17a741a700 1 -- start start 2023-06-10T18:52:13.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.317+0000 7f17a741a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17a006ea70 0x7f17a006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.317+0000 7f17a741a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a00ae7e0 con 0x7f17a015c6f0 2023-06-10T18:52:13.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a741a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a006df90 con 0x7f17a00b1780 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a741a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a006e270 con 0x7f17a006ea70 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a51b6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f17a015c6f0 0x7f17a00b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59214/0 (socket says 172.21.15.104:59214) 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a51b6700 1 -- 172.21.15.104:0/2598707441 learned_addr learned my addr 172.21.15.104:0/2598707441 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402885006 0 0) 0x7f17a00ae7e0 con 0x7f17a015c6f0 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a59b7700 1 --2- 172.21.15.104:0/2598707441 >> v2:172.21.15.114:3300/0 conn(0x7f17a006ea70 0x7f17a006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f178c0035b0 con 0x7f17a015c6f0 2023-06-10T18:52:13.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041227964 0 0) 0x7f178c0035b0 con 0x7f17a015c6f0 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 >> v2:172.21.15.114:3300/0 conn(0x7f17a006ea70 msgr2=0x7f17a006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 --2- 172.21.15.104:0/2598707441 >> v2:172.21.15.114:3300/0 conn(0x7f17a006ea70 0x7f17a006d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 >> v1:172.21.15.114:6789/0 conn(0x7f17a00b1780 legacy=0x7f17a006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a006e550 con 0x7f17a015c6f0 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f17900017f0 con 0x7f17a015c6f0 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 >> v1:172.21.15.104:6789/0 conn(0x7f17a015c6f0 legacy=0x7f17a00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 0x7f178c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f178c004850 0x7f178c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c005200 0x7f178c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a006e270 con 0x7f178c003a00 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c00c110 con 0x7f178c005200 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c00c2d0 con 0x7f178c004850 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a51b6700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 0x7f178c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a49b5700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f178c004850 0x7f178c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.318+0000 7f17a59b7700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c005200 0x7f178c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 -- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f178c004850 msgr2=0x7f178c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f178c004850 0x7f178c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 -- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c005200 msgr2=0x7f178c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c005200 0x7f178c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 -- 172.21.15.104:0/2598707441 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0035b0 con 0x7f178c003a00 2023-06-10T18:52:13.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a59b7700 1 -- 172.21.15.104:0/2598707441 reap_dead start 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a51b6700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 0x7f178c003e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f179000c240 tx=0x7f178c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a742927b6a3cac3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179000cd60 con 0x7f178c003a00 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790007b80 con 0x7f178c003a00 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f1797fff700 1 -- 172.21.15.104:0/2598707441 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17900072d0 con 0x7f178c003a00 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a741a700 1 -- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 msgr2=0x7f178c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a741a700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 0x7f178c003e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f179000c240 tx=0x7f178c005880 comp rx=0 tx=0).stop 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a741a700 1 -- 172.21.15.104:0/2598707441 shutdown_connections 2023-06-10T18:52:13.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a741a700 1 --2- 172.21.15.104:0/2598707441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f178c003a00 0x7f178c003e10 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.319+0000 7f17a741a700 1 -- 172.21.15.104:0/2598707441 >> 172.21.15.104:0/2598707441 conn(0x7f17a006b8a0 msgr2=0x7f17a00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:13.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.320+0000 7f17a741a700 1 -- 172.21.15.104:0/2598707441 shutdown_connections 2023-06-10T18:52:13.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.320+0000 7f17a741a700 1 -- 172.21.15.104:0/2598707441 wait complete. 2023-06-10T18:52:13.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 Processor -- start 2023-06-10T18:52:13.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 -- start start 2023-06-10T18:52:13.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c003a00 0x7f17a01ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17a015b7d0 0x7f17a00c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b3220 con 0x7f17a00c6980 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a015bc40 con 0x7f178c003a00 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.321+0000 7f17a741a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01ef960 con 0x7f17a015b7d0 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33166/0 (socket says 172.21.15.104:33166) 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 -- 172.21.15.104:0/1961415942 learned_addr learned my addr 172.21.15.104:0/1961415942 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:13.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a59b7700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17a015b7d0 0x7f17a00c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a51b6700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c003a00 0x7f17a01ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 -- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17a015b7d0 msgr2=0x7f17a00c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17a015b7d0 0x7f17a00c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 -- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c003a00 msgr2=0x7f17a01ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c003a00 0x7f17a01ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 -- 172.21.15.104:0/1961415942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00b25a0 con 0x7f17a00c6980 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a49b5700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f179c00f670 tx=0x7f179c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0052e0 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c005440 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a00b2880 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0038a0 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.322+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00b2e40 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.323+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a00b3360 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.323+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f179c003a00 con 0x7f17a00c6980 2023-06-10T18:52:13.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.324+0000 7f1795ffb700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 0x7f178406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.324+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f179c0892d0 con 0x7f17a00c6980 2023-06-10T18:52:13.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.324+0000 7f17a51b6700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 0x7f178406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.324+0000 7f17a51b6700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 0x7f178406d190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1790000c00 tx=0x7f1790007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.328+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f17a00b3360 con 0x7f17a00c6980 2023-06-10T18:52:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.515+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17a00b3360 con 0x7f17a00c6980 2023-06-10T18:52:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.515+0000 7f1795ffb700 1 -- 172.21.15.104:0/1961415942 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17a00b3360 con 0x7f17a00c6980 2023-06-10T18:52:13.521 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:52:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 msgr2=0x7f178406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 0x7f178406d190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1790000c00 tx=0x7f1790007000 comp rx=0 tx=0).stop 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 msgr2=0x7f17a015b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f179c00f670 tx=0x7f179c00fa30 comp rx=0 tx=0).stop 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 shutdown_connections 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f178406ace0 0x7f178406d190 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f178c003a00 0x7f17a01ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17a015b7d0 0x7f17a00c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 --2- 172.21.15.104:0/1961415942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17a00c6980 0x7f17a015b290 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 >> 172.21.15.104:0/1961415942 conn(0x7f17a006b8a0 msgr2=0x7f17a00cac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 shutdown_connections 2023-06-10T18:52:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.518+0000 7f17a741a700 1 -- 172.21.15.104:0/1961415942 wait complete. 2023-06-10T18:52:13.535 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:52:13.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.652+0000 7f90c1994700 1 Processor -- start 2023-06-10T18:52:13.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.652+0000 7f90c1994700 1 -- start start 2023-06-10T18:52:13.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90c1994700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90bc15bc80 0x7f90bc070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90c1994700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 0x7f90bc0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90c1994700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc15c050 con 0x7f90bc0ce510 2023-06-10T18:52:13.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90c1994700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc0ceed0 con 0x7f90bc15bc80 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90c1994700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc0c9970 con 0x7f90bc0c9350 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90bc15bc80 0x7f90bc070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90bc15bc80 0x7f90bc070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33174/0 (socket says 172.21.15.104:33174) 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 -- 172.21.15.104:0/1730354404 learned_addr learned my addr 172.21.15.104:0/1730354404 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90bb7fe700 1 --2- 172.21.15.104:0/1730354404 >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 0x7f90bc0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 -- 172.21.15.104:0/1730354404 >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 msgr2=0x7f90bc0ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:13.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 --2- 172.21.15.104:0/1730354404 >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 0x7f90bc0ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 -- 172.21.15.104:0/1730354404 >> v1:172.21.15.114:6789/0 conn(0x7f90bc0ce510 legacy=0x7f90bc0c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90baffd700 1 -- 172.21.15.104:0/1730354404 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc0ccb70 con 0x7f90bc15bc80 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.653+0000 7f90bb7fe700 1 --2- 172.21.15.104:0/1730354404 >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 0x7f90bc0ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90baffd700 1 --2- 172.21.15.104:0/1730354404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc15bc80 0x7f90bc070310 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0adfb0 tx=0x7f90b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b515d65cc2f50 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90b9ffb700 1 -- 172.21.15.104:0/1730354404 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400e070 con 0x7f90bc15bc80 2023-06-10T18:52:13.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90b9ffb700 1 -- 172.21.15.104:0/1730354404 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b400b9c0 con 0x7f90bc15bc80 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90b9ffb700 1 -- 172.21.15.104:0/1730354404 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400bd70 con 0x7f90bc15bc80 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 -- 172.21.15.104:0/1730354404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc15bc80 msgr2=0x7f90bc070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 --2- 172.21.15.104:0/1730354404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc15bc80 0x7f90bc070310 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0adfb0 tx=0x7f90b400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 -- 172.21.15.104:0/1730354404 shutdown_connections 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 --2- 172.21.15.104:0/1730354404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc15bc80 0x7f90bc070310 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 --2- 172.21.15.104:0/1730354404 >> v2:172.21.15.114:3300/0 conn(0x7f90bc0c9350 0x7f90bc0ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 -- 172.21.15.104:0/1730354404 >> 172.21.15.104:0/1730354404 conn(0x7f90bc06ab00 msgr2=0x7f90bc0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:13.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 -- 172.21.15.104:0/1730354404 shutdown_connections 2023-06-10T18:52:13.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.654+0000 7f90c1994700 1 -- 172.21.15.104:0/1730354404 wait complete. 2023-06-10T18:52:13.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.655+0000 7f90c1994700 1 Processor -- start 2023-06-10T18:52:13.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 -- start start 2023-06-10T18:52:13.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90bc0c9350 0x7f90bc0ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90bc15bc80 0x7f90bc0c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc148490 con 0x7f90bc0ce510 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc1f6ce0 con 0x7f90bc15bc80 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90c1994700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0c41b0 con 0x7f90bc0c9350 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90ba7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90ba7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33182/0 (socket says 172.21.15.104:33182) 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.656+0000 7f90ba7fc700 1 -- 172.21.15.104:0/1410418818 learned_addr learned my addr 172.21.15.104:0/1410418818 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90baffd700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90bc0c9350 0x7f90bc0ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90bb7fe700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90bc15bc80 0x7f90bc0c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 -- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90bc0c9350 msgr2=0x7f90bc0ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90bc0c9350 0x7f90bc0ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 -- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90bc15bc80 msgr2=0x7f90bc0c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90bc15bc80 0x7f90bc0c2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 -- 172.21.15.104:0/1410418818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4009b30 con 0x7f90bc0ce510 2023-06-10T18:52:13.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90ba7fc700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c510 tx=0x7f90a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40052d0 con 0x7f90bc0ce510 2023-06-10T18:52:13.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4005430 con 0x7f90bc0ce510 2023-06-10T18:52:13.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc0c35a0 con 0x7f90bc0ce510 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40038c0 con 0x7f90bc0ce510 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.657+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc0c3b60 con 0x7f90bc0ce510 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.658+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc074050 con 0x7f90bc0ce510 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.658+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90a4003a20 con 0x7f90bc0ce510 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.659+0000 7f90b3fff700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 0x7f909c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:13.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.659+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f90a4088bb0 con 0x7f90bc0ce510 2023-06-10T18:52:13.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.659+0000 7f90baffd700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 0x7f909c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:13.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.660+0000 7f90baffd700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 0x7f909c06d190 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b370 tx=0x7f90b4009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:13.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.663+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90a4053a00 con 0x7f90bc0ce510 2023-06-10T18:52:13.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.863+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90bc0c42f0 con 0x7f90bc0ce510 2023-06-10T18:52:13.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.864+0000 7f90b3fff700 1 -- 172.21.15.104:0/1410418818 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v86) v1 ==== 80+0+122 (secure 0 0 0) 0x7f90a4017090 con 0x7f90bc0ce510 2023-06-10T18:52:13.869 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6836/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:52:13.869 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6834/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:52:13.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 msgr2=0x7f909c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 0x7f909c06d190 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b370 tx=0x7f90b4009000 comp rx=0 tx=0).stop 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 msgr2=0x7f90bc1f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c510 tx=0x7f90a400c8d0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 shutdown_connections 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f909c06ace0 0x7f909c06d190 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90bc15bc80 0x7f90bc0c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90bc0c9350 0x7f90bc0ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 --2- 172.21.15.104:0/1410418818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90bc0ce510 0x7f90bc1f67a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 >> 172.21.15.104:0/1410418818 conn(0x7f90bc06ab00 msgr2=0x7f90bc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 shutdown_connections 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:13.866+0000 7f90c1994700 1 -- 172.21.15.104:0/1410418818 wait complete. 2023-06-10T18:52:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:52:13.882 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:52:13.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:52:13.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:52:13.928 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:52:13.928 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:52:13.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:52:13.929 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:52:13.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2023-06-10T18:52:14.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 Processor -- start 2023-06-10T18:52:14.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 -- start start 2023-06-10T18:52:14.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcebc06af80 0x7fcebc0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc06b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcebc15fd00 0x7fcebc160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc0cd460 con 0x7fcebc06af80 2023-06-10T18:52:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc0ce340 con 0x7fcebc15fd00 2023-06-10T18:52:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.105+0000 7fcec29c8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0d00d0 con 0x7fcebc0ccff0 2023-06-10T18:52:14.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcebc06af80 0x7fcebc0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcebc06af80 0x7fcebc0d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33188/0 (socket says 172.21.15.104:33188) 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 -- 172.21.15.104:0/2732906223 learned_addr learned my addr 172.21.15.104:0/2732906223 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcec0f65700 1 --2- 172.21.15.104:0/2732906223 >> v2:172.21.15.114:3300/0 conn(0x7fcebc15fd00 0x7fcebc160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebb7fe700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc06b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 -- 172.21.15.104:0/2732906223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 msgr2=0x7fcebc06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc06b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 -- 172.21.15.104:0/2732906223 >> v2:172.21.15.114:3300/0 conn(0x7fcebc15fd00 msgr2=0x7fcebc160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:14.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 --2- 172.21.15.104:0/2732906223 >> v2:172.21.15.114:3300/0 conn(0x7fcebc15fd00 0x7fcebc160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 -- 172.21.15.104:0/2732906223 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc06b890 con 0x7fcebc06af80 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebbfff700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc06af80 0x7fcebc0d0d20 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fceac0098f0 tx=0x7fceac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c23b2f186e3fee2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebaffd700 1 -- 172.21.15.104:0/2732906223 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac00b860 con 0x7fcebc06af80 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebaffd700 1 -- 172.21.15.104:0/2732906223 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac00b9c0 con 0x7fcebc06af80 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.106+0000 7fcebaffd700 1 -- 172.21.15.104:0/2732906223 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac00bd50 con 0x7fcebc06af80 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 -- 172.21.15.104:0/2732906223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc06af80 msgr2=0x7fcebc0d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc06af80 0x7fcebc0d0d20 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fceac0098f0 tx=0x7fceac009c00 comp rx=0 tx=0).stop 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 -- 172.21.15.104:0/2732906223 shutdown_connections 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc06af80 0x7fcebc0d0d20 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 --2- 172.21.15.104:0/2732906223 >> v2:172.21.15.114:3300/0 conn(0x7fcebc15fd00 0x7fcebc160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 --2- 172.21.15.104:0/2732906223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc06b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 -- 172.21.15.104:0/2732906223 >> 172.21.15.104:0/2732906223 conn(0x7fcebc089fd0 msgr2=0x7fcebc08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 -- 172.21.15.104:0/2732906223 shutdown_connections 2023-06-10T18:52:14.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.107+0000 7fcec29c8700 1 -- 172.21.15.104:0/2732906223 wait complete. 2023-06-10T18:52:14.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.108+0000 7fcec29c8700 1 Processor -- start 2023-06-10T18:52:14.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.108+0000 7fcec29c8700 1 -- start start 2023-06-10T18:52:14.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 0x7fcebc1f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc1f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1544f0 con 0x7fcebc15fd00 2023-06-10T18:52:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0b25f0 con 0x7fcebc06af80 2023-06-10T18:52:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1f6050 con 0x7fcebc0ccff0 2023-06-10T18:52:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33198/0 (socket says 172.21.15.104:33198) 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 -- 172.21.15.104:0/1390529353 learned_addr learned my addr 172.21.15.104:0/1390529353 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcebbfff700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 0x7fcebc1f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 -- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 msgr2=0x7fcebc1f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcebb7fe700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc1f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc1f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 -- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 msgr2=0x7fcebc1f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 0x7fcebc1f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 -- 172.21.15.104:0/1390529353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac009550 con 0x7fcebc15fd00 2023-06-10T18:52:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcebbfff700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 0x7fcebc1f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:14.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec0f65700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fceb4007e90 tx=0x7fceb400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb400ede0 con 0x7fcebc15fd00 2023-06-10T18:52:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4014420 con 0x7fcebc15fd00 2023-06-10T18:52:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.109+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc1e62c0 con 0x7fcebc15fd00 2023-06-10T18:52:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.110+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40147d0 con 0x7fcebc15fd00 2023-06-10T18:52:14.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.110+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1f1b00 con 0x7fcebc15fd00 2023-06-10T18:52:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.110+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc06c600 con 0x7fcebc15fd00 2023-06-10T18:52:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.111+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fceb4014930 con 0x7fcebc15fd00 2023-06-10T18:52:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.111+0000 7fceb8ff9700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 0x7fcea806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.111+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fceb4089de0 con 0x7fcebc15fd00 2023-06-10T18:52:14.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.112+0000 7fcebbfff700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 0x7fcea806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.112+0000 7fcebbfff700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 0x7fcea806d190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fceac009c40 tx=0x7fceac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.116+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fceb4054cc0 con 0x7fcebc15fd00 2023-06-10T18:52:14.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.342+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fcebc1e6450 con 0x7fcebc15fd00 2023-06-10T18:52:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.643+0000 7fceb8ff9700 1 -- 172.21.15.104:0/1390529353 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v43) v1 ==== 162+0+0 (secure 0 0 0) 0x7fceb40582e0 con 0x7fcebc15fd00 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 msgr2=0x7fcea806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 0x7fcea806d190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fceac009c40 tx=0x7fceac008030 comp rx=0 tx=0).stop 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 msgr2=0x7fcebc1f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fceb4007e90 tx=0x7fceb400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 shutdown_connections 2023-06-10T18:52:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcea806ace0 0x7fcea806d190 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcebc06af80 0x7fcebc1f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcebc15fd00 0x7fcebc1f56f0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 --2- 172.21.15.104:0/1390529353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcebc0ccff0 0x7fcebc1f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 >> 172.21.15.104:0/1390529353 conn(0x7fcebc089fd0 msgr2=0x7fcebc06e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 shutdown_connections 2023-06-10T18:52:14.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.645+0000 7fcec29c8700 1 -- 172.21.15.104:0/1390529353 wait complete. 2023-06-10T18:52:14.652 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked joinable; MDS may join as newly active. 2023-06-10T18:52:14.664 INFO:tasks.cephfs.test_client_recovery:Waiting for reconnect... 2023-06-10T18:52:14.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:14.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.780+0000 7fe636ce9700 1 Processor -- start 2023-06-10T18:52:14.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.780+0000 7fe636ce9700 1 -- start start 2023-06-10T18:52:14.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe636ce9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe630158e50 0x7fe6300cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe636ce9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6300cdbb0 0x7fe6300caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe636ce9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300c8de0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe636ce9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300ccc90 con 0x7fe6300cdbb0 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe636ce9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300ccf30 con 0x7fe630158e50 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6300cdbb0 0x7fe6300caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe635286700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6300cb3e0 0x7fe63015cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59266/0 (socket says 172.21.15.104:59266) 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe635286700 1 -- 172.21.15.104:0/3141708305 learned_addr learned my addr 172.21.15.104:0/3141708305 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe634a85700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe630158e50 0x7fe6300cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 -- 172.21.15.104:0/3141708305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe630158e50 msgr2=0x7fe6300cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe630158e50 0x7fe6300cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 -- 172.21.15.104:0/3141708305 >> v1:172.21.15.104:6789/0 conn(0x7fe6300cb3e0 legacy=0x7fe63015cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62f7fe700 1 -- 172.21.15.104:0/3141708305 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989036841 0 0) 0x7fe6300c8de0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 -- 172.21.15.104:0/3141708305 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300cd210 con 0x7fe6300cdbb0 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62ffff700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cdbb0 0x7fe6300caea0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009840 tx=0x7fe61c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9cee429d4af176 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.781+0000 7fe62f7fe700 1 -- 172.21.15.104:0/3141708305 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c011830 con 0x7fe6300cdbb0 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe62f7fe700 1 -- 172.21.15.104:0/3141708305 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c011990 con 0x7fe6300cdbb0 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe62f7fe700 1 -- 172.21.15.104:0/3141708305 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c011d20 con 0x7fe6300cdbb0 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 -- 172.21.15.104:0/3141708305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cdbb0 msgr2=0x7fe6300caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cdbb0 0x7fe6300caea0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009840 tx=0x7fe61c009b50 comp rx=0 tx=0).stop 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 -- 172.21.15.104:0/3141708305 shutdown_connections 2023-06-10T18:52:14.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe630158e50 0x7fe6300cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3141708305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cdbb0 0x7fe6300caea0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 -- 172.21.15.104:0/3141708305 >> 172.21.15.104:0/3141708305 conn(0x7fe63006ab00 msgr2=0x7fe6300b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:14.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 -- 172.21.15.104:0/3141708305 shutdown_connections 2023-06-10T18:52:14.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.782+0000 7fe636ce9700 1 -- 172.21.15.104:0/3141708305 wait complete. 2023-06-10T18:52:14.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.783+0000 7fe636ce9700 1 Processor -- start 2023-06-10T18:52:14.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.783+0000 7fe636ce9700 1 -- start start 2023-06-10T18:52:14.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6300cdbb0 0x7fe6300c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 0x7fe6300c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630145850 con 0x7fe6300cb3e0 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6301474f0 con 0x7fe630158e50 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe636ce9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6301468e0 con 0x7fe6300cdbb0 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe634a85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe634a85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33208/0 (socket says 172.21.15.104:33208) 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe634a85700 1 -- 172.21.15.104:0/3143620159 learned_addr learned my addr 172.21.15.104:0/3143620159 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe62ffff700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6300cdbb0 0x7fe6300c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.784+0000 7fe635286700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 0x7fe6300c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 -- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6300cdbb0 msgr2=0x7fe6300c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6300cdbb0 0x7fe6300c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 -- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 msgr2=0x7fe6300c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 0x7fe6300c4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:14.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 -- 172.21.15.104:0/3143620159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c0094a0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe635286700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 0x7fe6300c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe634a85700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fe62400d830 tx=0x7fe62400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6240125f0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624012750 con 0x7fe6300cb3e0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe630146bc0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624012b00 con 0x7fe6300cb3e0 2023-06-10T18:52:14.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.785+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630147180 con 0x7fe6300cb3e0 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.786+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6301477d0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.786+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe624012c60 con 0x7fe6300cb3e0 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.787+0000 7fe62d7fa700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 0x7fe61406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.787+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fe6240899e0 con 0x7fe6300cb3e0 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.787+0000 7fe62ffff700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 0x7fe61406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:14.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.787+0000 7fe62ffff700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 0x7fe61406d190 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0090b0 tx=0x7fe61c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:14.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:14.791+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe6240548e0 con 0x7fe6300cb3e0 2023-06-10T18:52:15.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.020+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe630074050 con 0x7fe6300cb3e0 2023-06-10T18:52:15.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.020+0000 7fe62d7fa700 1 -- 172.21.15.104:0/3143620159 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4686 (secure 0 0 0) 0x7fe624057f00 con 0x7fe6300cb3e0 2023-06-10T18:52:15.025 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:15.026 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:14.643455+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:15.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 msgr2=0x7fe61406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:15.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 0x7fe61406d190 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0090b0 tx=0x7fe61c008030 comp rx=0 tx=0).stop 2023-06-10T18:52:15.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 msgr2=0x7fe630145310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:15.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fe62400d830 tx=0x7fe62400db40 comp rx=0 tx=0).stop 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 shutdown_connections 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe61406ace0 0x7fe61406d190 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe630158e50 0x7fe6300c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6300cb3e0 0x7fe630145310 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 --2- 172.21.15.104:0/3143620159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6300cdbb0 0x7fe6300c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:15.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 >> 172.21.15.104:0/3143620159 conn(0x7fe63006ab00 msgr2=0x7fe630070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:15.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 shutdown_connections 2023-06-10T18:52:15.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:15.022+0000 7fe636ce9700 1 -- 172.21.15.104:0/3143620159 wait complete. 2023-06-10T18:52:15.029 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 44 2023-06-10T18:52:15.039 DEBUG:tasks.cephfs.filesystem:mapped states ['up:replay'] to None 2023-06-10T18:52:16.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:16.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.156+0000 7fe18210a700 1 Processor -- start 2023-06-10T18:52:16.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 -- start start 2023-06-10T18:52:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c070a20 0x7fe17c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe17c158eb0 0x7fe17c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c06ae00 con 0x7fe17c070df0 2023-06-10T18:52:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17c069e70 con 0x7fe17c158eb0 2023-06-10T18:52:16.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe18210a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1595f0 con 0x7fe17c070a20 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe17affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe17c070df0 0x7fe17c1587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59276/0 (socket says 172.21.15.104:59276) 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.157+0000 7fe17affd700 1 -- 172.21.15.104:0/986337606 learned_addr learned my addr 172.21.15.104:0/986337606 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081495698 0 0) 0x7fe16c001110 con 0x7fe17c070df0 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17bfff700 1 --2- 172.21.15.104:0/986337606 >> v2:172.21.15.114:3300/0 conn(0x7fe17c158eb0 0x7fe17c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1680046c0 con 0x7fe17c070df0 2023-06-10T18:52:16.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c070a20 0x7fe17c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162968293 0 0) 0x7fe1680046c0 con 0x7fe17c070df0 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c070a20 msgr2=0x7fe17c06a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c070a20 0x7fe17c06a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 >> v2:172.21.15.114:3300/0 conn(0x7fe17c158eb0 msgr2=0x7fe17c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 --2- 172.21.15.104:0/986337606 >> v2:172.21.15.114:3300/0 conn(0x7fe17c158eb0 0x7fe17c159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c1598d0 con 0x7fe17c070df0 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fe16c001800 con 0x7fe17c070df0 2023-06-10T18:52:16.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 >> v1:172.21.15.104:6789/0 conn(0x7fe17c070df0 legacy=0x7fe17c1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe168004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe168005960 0x7fe168005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe168006310 0x7fe16800cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c069e70 con 0x7fe168004b10 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1595f0 con 0x7fe168006310 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16800d110 con 0x7fe168005960 2023-06-10T18:52:16.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17bfff700 1 --2- 172.21.15.104:0/986337606 >> v2:172.21.15.114:3300/0 conn(0x7fe17c158eb0 0x7fe17c159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c070a20 0x7fe17c06a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe168004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17affd700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe168005960 0x7fe168005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17bfff700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe168006310 0x7fe16800cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 -- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe168005960 msgr2=0x7fe168005dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe168005960 0x7fe168005dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 -- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe168006310 msgr2=0x7fe16800cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:16.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe168006310 0x7fe16800cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17b7fe700 1 -- 172.21.15.104:0/986337606 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680046c0 con 0x7fe168004b10 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.158+0000 7fe17bfff700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe168006310 0x7fe16800cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe168004f20 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0cc8e0 tx=0x7fe168006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581f623e8859c28e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16400c070 con 0x7fe168004b10 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe164013630 con 0x7fe168004b10 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe17a7fc700 1 -- 172.21.15.104:0/986337606 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640093d0 con 0x7fe168004b10 2023-06-10T18:52:16.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 -- 172.21.15.104:0/986337606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 msgr2=0x7fe168004f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 --2- 172.21.15.104:0/986337606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe168004f20 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0cc8e0 tx=0x7fe168006990 comp rx=0 tx=0).stop 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe17bfff700 1 -- 172.21.15.104:0/986337606 reap_dead start 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 -- 172.21.15.104:0/986337606 shutdown_connections 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 -- 172.21.15.104:0/986337606 >> 172.21.15.104:0/986337606 conn(0x7fe17c0b1610 msgr2=0x7fe17c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 -- 172.21.15.104:0/986337606 shutdown_connections 2023-06-10T18:52:16.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.159+0000 7fe18210a700 1 -- 172.21.15.104:0/986337606 wait complete. 2023-06-10T18:52:16.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 Processor -- start 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 -- start start 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe17c0adf50 0x7fe17c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c1565b0 0x7fe17c1f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c156a20 con 0x7fe168004b10 2023-06-10T18:52:16.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0c8850 con 0x7fe17c0adf50 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe18210a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1f2700 con 0x7fe17c1565b0 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.161+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33228/0 (socket says 172.21.15.104:33228) 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 -- 172.21.15.104:0/2808676194 learned_addr learned my addr 172.21.15.104:0/2808676194 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17bfff700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c1565b0 0x7fe17c1f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17affd700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe17c0adf50 0x7fe17c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 -- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c1565b0 msgr2=0x7fe17c1f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c1565b0 0x7fe17c1f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 -- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe17c0adf50 msgr2=0x7fe17c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe17c0adf50 0x7fe17c156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 -- 172.21.15.104:0/2808676194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c1f29e0 con 0x7fe168004b10 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe17b7fe700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fe164009d00 tx=0x7fe1640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16401d810 con 0x7fe168004b10 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16401d970 con 0x7fe168004b10 2023-06-10T18:52:16.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c0c6f50 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16401dd00 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.162+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c0c7510 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.163+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c071780 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.164+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe1640273e0 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.164+0000 7fe173fff700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 0x7fe15c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.164+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fe164023030 con 0x7fe168004b10 2023-06-10T18:52:16.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.164+0000 7fe17affd700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 0x7fe15c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.165+0000 7fe17affd700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 0x7fe15c06d190 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe16c001050 tx=0x7fe16c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.168+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe164055930 con 0x7fe168004b10 2023-06-10T18:52:16.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.399+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe17c074050 con 0x7fe168004b10 2023-06-10T18:52:16.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.399+0000 7fe173fff700 1 -- 172.21.15.104:0/2808676194 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fe16401de60 con 0x7fe168004b10 2023-06-10T18:52:16.404 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:16.404 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 msgr2=0x7fe15c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 0x7fe15c06d190 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe16c001050 tx=0x7fe16c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 msgr2=0x7fe17c0c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fe164009d00 tx=0x7fe1640047c0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 shutdown_connections 2023-06-10T18:52:16.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe17c1565b0 0x7fe17c1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe168004b10 0x7fe17c0c7a30 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe15c06ace0 0x7fe15c06d190 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 --2- 172.21.15.104:0/2808676194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe17c0adf50 0x7fe17c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 >> 172.21.15.104:0/2808676194 conn(0x7fe17c0b1610 msgr2=0x7fe17c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 shutdown_connections 2023-06-10T18:52:16.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.401+0000 7fe18210a700 1 -- 172.21.15.104:0/2808676194 wait complete. 2023-06-10T18:52:16.408 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:16.418 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to up:reconnect 2023-06-10T18:52:16.418 DEBUG:tasks.cephfs.filesystem:reached state 'up:reconnect' in 1.7542848587036133s 2023-06-10T18:52:16.419 INFO:tasks.cephfs.test_client_recovery:Waiting for active... 2023-06-10T18:52:16.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:16.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.535+0000 7fc0033c5700 1 Processor -- start 2023-06-10T18:52:16.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.535+0000 7fc0033c5700 1 -- start start 2023-06-10T18:52:16.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.535+0000 7fc0033c5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbffc06a580 0x7fbffc0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc0033c5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc06dcb0 con 0x7fbffc0cf030 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc0033c5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc0cf460 con 0x7fbffc15c500 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc0033c5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc06e850 con 0x7fbffc06a580 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc000960700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbffc0cf030 0x7fbffc069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59298/0 (socket says 172.21.15.104:59298) 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc000960700 1 -- 172.21.15.104:0/905594411 learned_addr learned my addr 172.21.15.104:0/905594411 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc001962700 1 --2- 172.21.15.104:0/905594411 >> v2:172.21.15.114:3300/0 conn(0x7fbffc06a580 0x7fbffc0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fbff3fff700 1 -- 172.21.15.104:0/905594411 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875803398 0 0) 0x7fbffc06dcb0 con 0x7fbffc0cf030 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fbff3fff700 1 -- 172.21.15.104:0/905594411 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe80035b0 con 0x7fbffc0cf030 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc001962700 1 -- 172.21.15.104:0/905594411 >> v1:172.21.15.114:6789/0 conn(0x7fbffc15c500 legacy=0x7fbffc06e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc001962700 1 -- 172.21.15.104:0/905594411 >> v1:172.21.15.104:6789/0 conn(0x7fbffc0cf030 legacy=0x7fbffc069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc001962700 1 -- 172.21.15.104:0/905594411 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc06d0a0 con 0x7fbffc06a580 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fc001962700 1 --2- 172.21.15.104:0/905594411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc06a580 0x7fbffc0ce040 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009d90 tx=0x7fbff400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85a3b0df64d59215 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fbff3fff700 1 -- 172.21.15.104:0/905594411 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff400e070 con 0x7fbffc06a580 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fbff3fff700 1 -- 172.21.15.104:0/905594411 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff400bb20 con 0x7fbffc06a580 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.536+0000 7fbff3fff700 1 -- 172.21.15.104:0/905594411 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4012430 con 0x7fbffc06a580 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 -- 172.21.15.104:0/905594411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc06a580 msgr2=0x7fbffc0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 --2- 172.21.15.104:0/905594411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc06a580 0x7fbffc0ce040 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009d90 tx=0x7fbff400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 -- 172.21.15.104:0/905594411 shutdown_connections 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 --2- 172.21.15.104:0/905594411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc06a580 0x7fbffc0ce040 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 -- 172.21.15.104:0/905594411 >> 172.21.15.104:0/905594411 conn(0x7fbffc06b8a0 msgr2=0x7fbffc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:16.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 -- 172.21.15.104:0/905594411 shutdown_connections 2023-06-10T18:52:16.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.537+0000 7fc0033c5700 1 -- 172.21.15.104:0/905594411 wait complete. 2023-06-10T18:52:16.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.538+0000 7fc0033c5700 1 Processor -- start 2023-06-10T18:52:16.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.538+0000 7fc0033c5700 1 -- start start 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbffc0cf030 0x7fbffc158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc15c500 0x7fbffc0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b4b30 con 0x7fbffc06a580 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b4e10 con 0x7fbffc15c500 2023-06-10T18:52:16.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc0033c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0b50f0 con 0x7fbffc0cf030 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33242/0 (socket says 172.21.15.104:33242) 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 -- 172.21.15.104:0/1935373354 learned_addr learned my addr 172.21.15.104:0/1935373354 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 -- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbffc0cf030 msgr2=0x7fbffc158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbffc0cf030 0x7fbffc158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 -- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc15c500 msgr2=0x7fbffc0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc15c500 0x7fbffc0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 -- 172.21.15.104:0/1935373354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4009a70 con 0x7fbffc06a580 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.539+0000 7fc001161700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00c4d0 tx=0x7fbfec00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.540+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0052d0 con 0x7fbffc06a580 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.540+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec005430 con 0x7fbffc06a580 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.540+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0038f0 con 0x7fbffc06a580 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.540+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc0b3630 con 0x7fbffc06a580 2023-06-10T18:52:16.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.540+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc1590b0 con 0x7fbffc06a580 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.541+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc0b3490 con 0x7fbffc06a580 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.541+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbfec003a50 con 0x7fbffc06a580 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.541+0000 7fbff1ffb700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 0x7fbfe006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.542+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fbfec088b50 con 0x7fbffc06a580 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.542+0000 7fc000960700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 0x7fbfe006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:16.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.542+0000 7fc000960700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 0x7fbfe006d190 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbffc15abc0 tx=0x7fbff80079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:16.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.546+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbffc0b3490 con 0x7fbffc06a580 2023-06-10T18:52:16.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.773+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbffc0b3870 con 0x7fbffc06a580 2023-06-10T18:52:16.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.774+0000 7fbff1ffb700 1 -- 172.21.15.104:0/1935373354 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fbffc0b3870 con 0x7fbffc06a580 2023-06-10T18:52:16.778 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:16.779 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:16.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 msgr2=0x7fbfe006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 0x7fbfe006d190 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbffc15abc0 tx=0x7fbff80079c0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 msgr2=0x7fbffc1587f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00c4d0 tx=0x7fbfec00c890 comp rx=0 tx=0).stop 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 shutdown_connections 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbfe006ace0 0x7fbfe006d190 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbffc15c500 0x7fbffc0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbffc06a580 0x7fbffc1587f0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 --2- 172.21.15.104:0/1935373354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbffc0cf030 0x7fbffc158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 >> 172.21.15.104:0/1935373354 conn(0x7fbffc06b8a0 msgr2=0x7fbffc06e040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 shutdown_connections 2023-06-10T18:52:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:16.776+0000 7fc0033c5700 1 -- 172.21.15.104:0/1935373354 wait complete. 2023-06-10T18:52:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:16.792 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:17.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:17.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.921+0000 7f7bc3e6d700 1 Processor -- start 2023-06-10T18:52:17.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.921+0000 7f7bc3e6d700 1 -- start start 2023-06-10T18:52:17.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc3e6d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc3e6d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7bbc15a250 0x7f7bbc0cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc3e6d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc0ccca0 con 0x7f7bbc0cad90 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc3e6d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bbc0d0310 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc3e6d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bbc0cee50 con 0x7f7bbc15a250 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc1c09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc240a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7bbc0cad90 0x7f7bbc0b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59310/0 (socket says 172.21.15.104:59310) 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc1c09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33250/0 (socket says 172.21.15.104:33250) 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.922+0000 7f7bc240a700 1 -- 172.21.15.104:0/3271353979 learned_addr learned my addr 172.21.15.104:0/3271353979 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1408700 1 --2- 172.21.15.104:0/3271353979 >> v2:172.21.15.114:3300/0 conn(0x7f7bbc15a250 0x7f7bbc0cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1c09700 1 -- 172.21.15.104:0/3271353979 >> v2:172.21.15.114:3300/0 conn(0x7f7bbc15a250 msgr2=0x7f7bbc0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1c09700 1 --2- 172.21.15.104:0/3271353979 >> v2:172.21.15.114:3300/0 conn(0x7f7bbc15a250 0x7f7bbc0cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:17.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1c09700 1 -- 172.21.15.104:0/3271353979 >> v1:172.21.15.104:6789/0 conn(0x7f7bbc0cad90 legacy=0x7f7bbc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1c09700 1 -- 172.21.15.104:0/3271353979 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc0cf130 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc0c07700 1 -- 172.21.15.104:0/3271353979 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036882660 0 0) 0x7f7bbc0ccca0 con 0x7f7bbc0cad90 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc1c09700 1 --2- 172.21.15.104:0/3271353979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc071ed0 tx=0x7f7bac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320bcecd823da689 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc0c07700 1 -- 172.21.15.104:0/3271353979 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac011830 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc0c07700 1 -- 172.21.15.104:0/3271353979 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac011990 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.923+0000 7f7bc0c07700 1 -- 172.21.15.104:0/3271353979 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac011d20 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/3271353979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc0cfa00 msgr2=0x7f7bbc0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:17.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/3271353979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc071ed0 tx=0x7f7bac009b20 comp rx=0 tx=0).stop 2023-06-10T18:52:17.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/3271353979 shutdown_connections 2023-06-10T18:52:17.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/3271353979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc0701e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:17.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/3271353979 >> v2:172.21.15.114:3300/0 conn(0x7f7bbc15a250 0x7f7bbc0cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:17.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/3271353979 >> 172.21.15.104:0/3271353979 conn(0x7f7bbc06e370 msgr2=0x7f7bbc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:17.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/3271353979 shutdown_connections 2023-06-10T18:52:17.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.924+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/3271353979 wait complete. 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.925+0000 7f7bc3e6d700 1 Processor -- start 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.925+0000 7f7bc3e6d700 1 -- start start 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bbc0cad90 0x7f7bbc15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc1560e0 con 0x7f7bbc15a250 2023-06-10T18:52:17.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc15d500 con 0x7f7bbc0cfa00 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc15b3d0 con 0x7f7bbc0cad90 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33260/0 (socket says 172.21.15.104:33260) 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 -- 172.21.15.104:0/534798801 learned_addr learned my addr 172.21.15.104:0/534798801 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc1408700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc1c09700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bbc0cad90 0x7f7bbc15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 -- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bbc0cad90 msgr2=0x7f7bbc15cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bbc0cad90 0x7f7bbc15cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 -- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 msgr2=0x7f7bbc154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 -- 172.21.15.104:0/534798801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0094a0 con 0x7f7bbc15a250 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc1408700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc240a700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400cb60 tx=0x7f7bb400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb400ab50 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb400acb0 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc0c8a70 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4007310 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.926+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc15b030 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.927+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc074050 con 0x7f7bbc15a250 2023-06-10T18:52:17.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.928+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7bb4007470 con 0x7f7bbc15a250 2023-06-10T18:52:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.928+0000 7f7bb27fc700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 0x7f7ba006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.928+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f7bb4089150 con 0x7f7bbc15a250 2023-06-10T18:52:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.928+0000 7f7bc1c09700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 0x7f7ba006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.929+0000 7f7bc1c09700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 0x7f7ba006d190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7bac000c00 tx=0x7f7bac01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:17.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:17.932+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7bb4054040 con 0x7f7bbc15a250 2023-06-10T18:52:18.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.161+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bbc15b510 con 0x7f7bbc15a250 2023-06-10T18:52:18.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.162+0000 7f7bb27fc700 1 -- 172.21.15.104:0/534798801 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f7bb4057660 con 0x7f7bbc15a250 2023-06-10T18:52:18.167 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:18.167 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:18.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 msgr2=0x7f7ba006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 0x7f7ba006d190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7bac000c00 tx=0x7f7bac01c040 comp rx=0 tx=0).stop 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 msgr2=0x7f7bbc15b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400cb60 tx=0x7f7bb400cf20 comp rx=0 tx=0).stop 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 shutdown_connections 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bbc15a250 0x7f7bbc15b860 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:18.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bbc0cad90 0x7f7bbc15cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ba006ace0 0x7f7ba006d190 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 --2- 172.21.15.104:0/534798801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0cfa00 0x7f7bbc154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 >> 172.21.15.104:0/534798801 conn(0x7f7bbc06e370 msgr2=0x7f7bbc156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 shutdown_connections 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:18.164+0000 7f7bc3e6d700 1 -- 172.21.15.104:0/534798801 wait complete. 2023-06-10T18:52:18.170 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:18.180 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:19.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.309+0000 7f7ab8488700 1 Processor -- start 2023-06-10T18:52:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.309+0000 7f7ab8488700 1 -- start start 2023-06-10T18:52:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab8488700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7ab006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab8488700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab006d290 con 0x7f7ab006e0e0 2023-06-10T18:52:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab8488700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab00ca2d0 con 0x7f7ab006e4b0 2023-06-10T18:52:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab8488700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab0071640 con 0x7f7ab00cf100 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab6224700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7ab006e0e0 0x7f7ab00c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59326/0 (socket says 172.21.15.104:59326) 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.310+0000 7f7ab6224700 1 -- 172.21.15.104:0/2679677364 learned_addr learned my addr 172.21.15.104:0/2679677364 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab6a25700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7ab006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464685606 0 0) 0x7f7ab006d290 con 0x7f7ab006e0e0 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a9c0035b0 con 0x7f7ab006e0e0 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573693955 0 0) 0x7f7ab00ca2d0 con 0x7f7ab006e4b0 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab006d290 con 0x7f7ab006e4b0 2023-06-10T18:52:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1497567338 0 0) 0x7f7a9c0035b0 con 0x7f7ab006e0e0 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 msgr2=0x7f7ab006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7ab006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 >> v1:172.21.15.114:6789/0 conn(0x7f7ab006e4b0 legacy=0x7f7ab00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab006d4c0 con 0x7f7ab006e0e0 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7aa00017f0 con 0x7f7ab006e0e0 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 >> v1:172.21.15.104:6789/0 conn(0x7f7ab006e0e0 legacy=0x7f7ab00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c003bc0 0x7f7a9c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a9c004850 0x7f7a9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7a9c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab0071640 con 0x7f7a9c003bc0 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c00bb20 con 0x7f7a9c004850 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c00bce0 con 0x7f7ab00cf100 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab6224700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c003bc0 0x7f7a9c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab5a23700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a9c004850 0x7f7a9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.311+0000 7f7ab6a25700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7a9c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 -- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 msgr2=0x7f7a9c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ab00cf100 0x7f7a9c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 -- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a9c004850 msgr2=0x7f7a9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a9c004850 0x7f7a9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 -- 172.21.15.104:0/2679677364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c0035b0 con 0x7f7a9c003bc0 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6224700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c003bc0 0x7f7a9c003fb0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c240 tx=0x7f7a9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1049b37bec26a834 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000cd80 con 0x7f7a9c003bc0 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0007b50 con 0x7f7a9c003bc0 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab5222700 1 -- 172.21.15.104:0/2679677364 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0007310 con 0x7f7a9c003bc0 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab8488700 1 -- 172.21.15.104:0/2679677364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c003bc0 msgr2=0x7f7a9c003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab8488700 1 --2- 172.21.15.104:0/2679677364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9c003bc0 0x7f7a9c003fb0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c240 tx=0x7f7a9c005880 comp rx=0 tx=0).stop 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab6a25700 1 -- 172.21.15.104:0/2679677364 reap_dead start 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab8488700 1 -- 172.21.15.104:0/2679677364 shutdown_connections 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.312+0000 7f7ab8488700 1 -- 172.21.15.104:0/2679677364 >> 172.21.15.104:0/2679677364 conn(0x7f7ab006b8a0 msgr2=0x7f7ab00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.313+0000 7f7ab8488700 1 -- 172.21.15.104:0/2679677364 shutdown_connections 2023-06-10T18:52:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.313+0000 7f7ab8488700 1 -- 172.21.15.104:0/2679677364 wait complete. 2023-06-10T18:52:19.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 Processor -- start 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 -- start start 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 0x7f7ab00b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 0x7f7ab0152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab0155a80 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab00b25d0 con 0x7f7ab00b17c0 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.314+0000 7f7ab8488700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab00ad280 con 0x7f7a9c003bc0 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33280/0 (socket says 172.21.15.104:33280) 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 -- 172.21.15.104:0/3109350414 learned_addr learned my addr 172.21.15.104:0/3109350414 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab5a23700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 0x7f7ab0152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6224700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 0x7f7ab00b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 -- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 msgr2=0x7f7ab00b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 0x7f7ab00b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 -- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 msgr2=0x7f7ab0152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 0x7f7ab0152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 -- 172.21.15.104:0/3109350414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab00b1c30 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab5a23700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 0x7f7ab0152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6224700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 0x7f7ab00b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab6a25700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0072080 tx=0x7f7aa800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa800e040 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa800f040 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab00b0bc0 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8015430 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.315+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab00afa60 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.316+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab0074050 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.317+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7aa8015590 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.317+0000 7f7aa6ffd700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 0x7f7a9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.317+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f7aa8089cf0 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.317+0000 7f7ab6224700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 0x7f7a9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.318+0000 7f7ab6224700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 0x7f7a9406d190 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c240 tx=0x7f7aa000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.322+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7aa808e070 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.554+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ab015c180 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.554+0000 7f7aa6ffd700 1 -- 172.21.15.104:0/3109350414 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f7aa8054b50 con 0x7f7ab00ac8d0 2023-06-10T18:52:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:19.559 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 msgr2=0x7f7a9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 0x7f7a9406d190 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c240 tx=0x7f7aa000a000 comp rx=0 tx=0).stop 2023-06-10T18:52:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 msgr2=0x7f7ab00acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0072080 tx=0x7f7aa800f490 comp rx=0 tx=0).stop 2023-06-10T18:52:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 shutdown_connections 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ab00ac8d0 0x7f7ab00acd40 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a9c003bc0 0x7f7ab00b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9406ace0 0x7f7a9406d190 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 --2- 172.21.15.104:0/3109350414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00b17c0 0x7f7ab0152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 >> 172.21.15.104:0/3109350414 conn(0x7f7ab006b8a0 msgr2=0x7f7ab00c9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 shutdown_connections 2023-06-10T18:52:19.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:19.556+0000 7f7ab8488700 1 -- 172.21.15.104:0/3109350414 wait complete. 2023-06-10T18:52:19.563 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:19.573 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:20.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:20.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.694+0000 7fc2859c0700 1 Processor -- start 2023-06-10T18:52:20.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.694+0000 7fc2859c0700 1 -- start start 2023-06-10T18:52:20.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.695+0000 7fc2859c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2800aeaa0 0x7fc2800cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:20.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc2859c0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800c9e60 con 0x7fc2800b1080 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc2859c0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800cdaa0 con 0x7fc2800cd680 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc2859c0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2800c9bc0 con 0x7fc2800aeaa0 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2800aeaa0 0x7fc2800cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27e7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc2800b1080 0x7fc2800ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59344/0 (socket says 172.21.15.104:59344) 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27e7fc700 1 -- 172.21.15.104:0/2208022156 learned_addr learned my addr 172.21.15.104:0/2208022156 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27effd700 1 -- 172.21.15.104:0/2208022156 >> v1:172.21.15.114:6789/0 conn(0x7fc2800cd680 legacy=0x7fc28006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27effd700 1 -- 172.21.15.104:0/2208022156 >> v1:172.21.15.104:6789/0 conn(0x7fc2800b1080 legacy=0x7fc2800ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27dffb700 1 -- 172.21.15.104:0/2208022156 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489657721 0 0) 0x7fc2800c9e60 con 0x7fc2800b1080 2023-06-10T18:52:20.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27effd700 1 -- 172.21.15.104:0/2208022156 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2800aee70 con 0x7fc2800aeaa0 2023-06-10T18:52:20.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27effd700 1 --2- 172.21.15.104:0/2208022156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800aeaa0 0x7fc2800cd140 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc274009820 tx=0x7fc274009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e50310aec7b0c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27dffb700 1 -- 172.21.15.104:0/2208022156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274005e30 con 0x7fc2800aeaa0 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27dffb700 1 -- 172.21.15.104:0/2208022156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274006030 con 0x7fc2800aeaa0 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.696+0000 7fc27dffb700 1 -- 172.21.15.104:0/2208022156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2740063e0 con 0x7fc2800aeaa0 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 -- 172.21.15.104:0/2208022156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800aeaa0 msgr2=0x7fc2800cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 --2- 172.21.15.104:0/2208022156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800aeaa0 0x7fc2800cd140 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc274009820 tx=0x7fc274009b30 comp rx=0 tx=0).stop 2023-06-10T18:52:20.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 -- 172.21.15.104:0/2208022156 shutdown_connections 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 --2- 172.21.15.104:0/2208022156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800aeaa0 0x7fc2800cd140 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 -- 172.21.15.104:0/2208022156 >> 172.21.15.104:0/2208022156 conn(0x7fc28006b8a0 msgr2=0x7fc2800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 -- 172.21.15.104:0/2208022156 shutdown_connections 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.697+0000 7fc2859c0700 1 -- 172.21.15.104:0/2208022156 wait complete. 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.698+0000 7fc2859c0700 1 Processor -- start 2023-06-10T18:52:20.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.698+0000 7fc2859c0700 1 -- start start 2023-06-10T18:52:20.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2800aeaa0 0x7fc280157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:20.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 0x7fc28015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:20.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2800c8970 con 0x7fc2800cd680 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280157a80 con 0x7fc2800b1080 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc2859c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2800c9790 con 0x7fc2800aeaa0 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33294/0 (socket says 172.21.15.104:33294) 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 -- 172.21.15.104:0/90406277 learned_addr learned my addr 172.21.15.104:0/90406277 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:20.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27effd700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2800aeaa0 0x7fc280157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 -- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2800aeaa0 msgr2=0x7fc280157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2800aeaa0 0x7fc280157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 -- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 msgr2=0x7fc28015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27e7fc700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 0x7fc28015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 0x7fc28015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 -- 172.21.15.104:0/90406277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274009430 con 0x7fc2800cd680 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27e7fc700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 0x7fc28015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:20.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.699+0000 7fc27f7fe700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc27800cbc0 tx=0x7fc27800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.700+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278003c00 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.700+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc278003d60 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.700+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2800c8b80 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.700+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27800ab90 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.700+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2800c9140 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.701+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2800c9900 con 0x7fc2800cd680 2023-06-10T18:52:20.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.701+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc27800acf0 con 0x7fc2800cd680 2023-06-10T18:52:20.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.701+0000 7fc267fff700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 0x7fc26c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:20.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.702+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fc278089730 con 0x7fc2800cd680 2023-06-10T18:52:20.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.702+0000 7fc27effd700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 0x7fc26c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:20.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.702+0000 7fc27effd700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 0x7fc26c06d190 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc28015b510 tx=0x7fc274008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:20.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.706+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc278054580 con 0x7fc2800cd680 2023-06-10T18:52:20.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.936+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc280074050 con 0x7fc2800cd680 2023-06-10T18:52:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.937+0000 7fc267fff700 1 -- 172.21.15.104:0/90406277 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fc278057ba0 con 0x7fc2800cd680 2023-06-10T18:52:20.941 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:20.942 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 msgr2=0x7fc26c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 0x7fc26c06d190 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc28015b510 tx=0x7fc274008070 comp rx=0 tx=0).stop 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 msgr2=0x7fc2800b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc27800cbc0 tx=0x7fc27800cf80 comp rx=0 tx=0).stop 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 shutdown_connections 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2800cd680 0x7fc2800b4640 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2800aeaa0 0x7fc280157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc26c06ace0 0x7fc26c06d190 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 --2- 172.21.15.104:0/90406277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2800b1080 0x7fc28015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 >> 172.21.15.104:0/90406277 conn(0x7fc28006b8a0 msgr2=0x7fc2800b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 shutdown_connections 2023-06-10T18:52:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:20.939+0000 7fc2859c0700 1 -- 172.21.15.104:0/90406277 wait complete. 2023-06-10T18:52:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:20.955 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:21.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:22.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.084+0000 7f10c1f47700 1 Processor -- start 2023-06-10T18:52:22.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.084+0000 7f10c1f47700 1 -- start start 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10c1f47700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10bc1567f0 0x7f10bc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10c1f47700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10c1f47700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc070b20 con 0x7f10bc15c7a0 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10c1f47700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc0cb2d0 con 0x7f10bc1567f0 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10c1f47700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc15cca0 con 0x7f10bc0701a0 2023-06-10T18:52:22.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10bb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10bc1567f0 0x7f10bc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10bb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f10bc1567f0 0x7f10bc0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33298/0 (socket says 172.21.15.104:33298) 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.085+0000 7f10bb7fe700 1 -- 172.21.15.104:0/4224529497 learned_addr learned my addr 172.21.15.104:0/4224529497 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bbfff700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bb7fe700 1 -- 172.21.15.104:0/4224529497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 msgr2=0x7f10bc0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bb7fe700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bb7fe700 1 -- 172.21.15.104:0/4224529497 >> v1:172.21.15.114:6789/0 conn(0x7f10bc15c7a0 legacy=0x7f10bc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bb7fe700 1 -- 172.21.15.104:0/4224529497 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc15cf80 con 0x7f10bc1567f0 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bbfff700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10bb7fe700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc1567f0 0x7f10bc0cad90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f10a40098f0 tx=0x7f10a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e217f3859c8d6f24 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10ba7fc700 1 -- 172.21.15.104:0/4224529497 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4004230 con 0x7f10bc1567f0 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10ba7fc700 1 -- 172.21.15.104:0/4224529497 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4004390 con 0x7f10bc1567f0 2023-06-10T18:52:22.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.086+0000 7f10ba7fc700 1 -- 172.21.15.104:0/4224529497 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40036a0 con 0x7f10bc1567f0 2023-06-10T18:52:22.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 -- 172.21.15.104:0/4224529497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc1567f0 msgr2=0x7f10bc0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:22.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc1567f0 0x7f10bc0cad90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f10a40098f0 tx=0x7f10a4009c00 comp rx=0 tx=0).stop 2023-06-10T18:52:22.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 -- 172.21.15.104:0/4224529497 shutdown_connections 2023-06-10T18:52:22.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc1567f0 0x7f10bc0cad90 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 --2- 172.21.15.104:0/4224529497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 -- 172.21.15.104:0/4224529497 >> 172.21.15.104:0/4224529497 conn(0x7f10bc06e370 msgr2=0x7f10bc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:22.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 -- 172.21.15.104:0/4224529497 shutdown_connections 2023-06-10T18:52:22.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.087+0000 7f10c1f47700 1 -- 172.21.15.104:0/4224529497 wait complete. 2023-06-10T18:52:22.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.088+0000 7f10c1f47700 1 Processor -- start 2023-06-10T18:52:22.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 -- start start 2023-06-10T18:52:22.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10bc1567f0 0x7f10bc0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc156e20 con 0x7f10bc15c7a0 2023-06-10T18:52:22.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0b4bc0 con 0x7f10bc1567f0 2023-06-10T18:52:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.089+0000 7f10c1f47700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0b4ea0 con 0x7f10bc0701a0 2023-06-10T18:52:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33306/0 (socket says 172.21.15.104:33306) 2023-06-10T18:52:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 -- 172.21.15.104:0/3512945654 learned_addr learned my addr 172.21.15.104:0/3512945654 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:22.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bb7fe700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10baffd700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10bc1567f0 0x7f10bc0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 -- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 msgr2=0x7f10bc0b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:22.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 -- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10bc1567f0 msgr2=0x7f10bc0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10bc1567f0 0x7f10bc0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 -- 172.21.15.104:0/3512945654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a4009550 con 0x7f10bc15c7a0 2023-06-10T18:52:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10bbfff700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f10b400cd80 tx=0x7f10b400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:22.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400f480 con 0x7f10bc15c7a0 2023-06-10T18:52:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b400f600 con 0x7f10bc15c7a0 2023-06-10T18:52:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400f9b0 con 0x7f10bc15c7a0 2023-06-10T18:52:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc0b0e60 con 0x7f10bc15c7a0 2023-06-10T18:52:22.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.090+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0b1420 con 0x7f10bc15c7a0 2023-06-10T18:52:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.091+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc0b1610 con 0x7f10bc15c7a0 2023-06-10T18:52:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.092+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f10b400b040 con 0x7f10bc15c7a0 2023-06-10T18:52:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.092+0000 7f10b3fff700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 0x7f109c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:22.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.092+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f10b4089da0 con 0x7f10bc15c7a0 2023-06-10T18:52:22.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.092+0000 7f10bb7fe700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 0x7f109c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:22.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.093+0000 7f10bb7fe700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 0x7f109c06d190 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009130 tx=0x7f10a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:22.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.096+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10b4054c90 con 0x7f10bc15c7a0 2023-06-10T18:52:22.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.323+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10bc074050 con 0x7f10bc15c7a0 2023-06-10T18:52:22.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.323+0000 7f10b3fff700 1 -- 172.21.15.104:0/3512945654 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f10b401d090 con 0x7f10bc15c7a0 2023-06-10T18:52:22.329 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:22.329 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 msgr2=0x7f109c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 0x7f109c06d190 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009130 tx=0x7f10a4008030 comp rx=0 tx=0).stop 2023-06-10T18:52:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 msgr2=0x7f10bc154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:22.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f10b400cd80 tx=0x7f10b400a560 comp rx=0 tx=0).stop 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 shutdown_connections 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10bc15c7a0 0x7f10bc154550 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10bc0701a0 0x7f10bc0b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f109c06ace0 0x7f109c06d190 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 --2- 172.21.15.104:0/3512945654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10bc1567f0 0x7f10bc0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 >> 172.21.15.104:0/3512945654 conn(0x7f10bc06e370 msgr2=0x7f10bc15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:22.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 shutdown_connections 2023-06-10T18:52:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:22.325+0000 7f10c1f47700 1 -- 172.21.15.104:0/3512945654 wait complete. 2023-06-10T18:52:22.332 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:22.342 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:23.343 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.470+0000 7f8949750700 1 Processor -- start 2023-06-10T18:52:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.470+0000 7f8949750700 1 -- start start 2023-06-10T18:52:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f8949750700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f89440b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f8949750700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440b6070 con 0x7f89440b4ff0 2023-06-10T18:52:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f8949750700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440b5460 con 0x7f89440cea60 2023-06-10T18:52:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f8949750700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89440b5740 con 0x7f894415a420 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f89427fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f89440b4ff0 0x7f89440ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59370/0 (socket says 172.21.15.104:59370) 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.471+0000 7f89427fc700 1 -- 172.21.15.104:0/1448477060 learned_addr learned my addr 172.21.15.104:0/1448477060 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8942ffd700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f89440b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033667707 0 0) 0x7f89440b6070 con 0x7f89440b4ff0 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0035b0 con 0x7f89440b4ff0 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1644179136 0 0) 0x7f892c0035b0 con 0x7f89440b4ff0 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 msgr2=0x7f89440b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f89440b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 >> v1:172.21.15.114:6789/0 conn(0x7f89440cea60 legacy=0x7f894415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440b5a20 con 0x7f89440b4ff0 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8934001670 con 0x7f89440b4ff0 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 >> v1:172.21.15.104:6789/0 conn(0x7f89440b4ff0 legacy=0x7f89440ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f892c003a30 0x7f892c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f892c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892c004d90 0x7f892c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89440b5740 con 0x7f892c004d90 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c00ba10 con 0x7f892c003a30 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c00bba0 con 0x7f894415a420 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892c004d90 0x7f892c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f8942ffd700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f892c003a30 0x7f892c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89427fc700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f892c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 -- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 msgr2=0x7f892c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f894415a420 0x7f892c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 -- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f892c003a30 msgr2=0x7f892c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f892c003a30 0x7f892c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.472+0000 7f89437fe700 1 -- 172.21.15.104:0/1448477060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0035b0 con 0x7f892c004d90 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f89437fe700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892c004d90 0x7f892c005200 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f89440cad80 tx=0x7f892c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05c8d345706775d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c001e70 con 0x7f892c004d90 2023-06-10T18:52:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c00cdf0 con 0x7f892c004d90 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8941ffb700 1 -- 172.21.15.104:0/1448477060 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c012660 con 0x7f892c004d90 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 -- 172.21.15.104:0/1448477060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892c004d90 msgr2=0x7f892c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 --2- 172.21.15.104:0/1448477060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892c004d90 0x7f892c005200 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f89440cad80 tx=0x7f892c005880 comp rx=0 tx=0).stop 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f89437fe700 1 -- 172.21.15.104:0/1448477060 reap_dead start 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 -- 172.21.15.104:0/1448477060 shutdown_connections 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 -- 172.21.15.104:0/1448477060 >> 172.21.15.104:0/1448477060 conn(0x7f8944089fd0 msgr2=0x7f89440d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 -- 172.21.15.104:0/1448477060 shutdown_connections 2023-06-10T18:52:23.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.473+0000 7f8949750700 1 -- 172.21.15.104:0/1448477060 wait complete. 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 Processor -- start 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 -- start start 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 0x7f89440c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89441f0020 0x7f89441e3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441542c0 con 0x7f89440c42b0 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441545a0 con 0x7f89441ef710 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f8949750700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89441f0500 con 0x7f89441f0020 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f89427fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.475+0000 7f89427fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33324/0 (socket says 172.21.15.104:33324) 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 -- 172.21.15.104:0/219481062 learned_addr learned my addr 172.21.15.104:0/219481062 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89437fe700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89441f0020 0x7f89441e3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f8942ffd700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 0x7f89440c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 -- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89441f0020 msgr2=0x7f89441e3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89441f0020 0x7f89441e3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 -- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 msgr2=0x7f89440c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 0x7f89440c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 -- 172.21.15.104:0/219481062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89441f07e0 con 0x7f89440c42b0 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f8942ffd700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 0x7f89440c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f89427fc700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8934004fd0 tx=0x7f8934007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:23.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893400f840 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893400f9c0 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89441f0ac0 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893400fd70 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.476+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89440c4900 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.477+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8944074050 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.478+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f893401d460 con 0x7f89440c42b0 2023-06-10T18:52:23.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.478+0000 7f892b7fe700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 0x7f893006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:23.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.478+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f893408a250 con 0x7f89440c42b0 2023-06-10T18:52:23.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.479+0000 7f8942ffd700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 0x7f893006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:23.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.479+0000 7f8942ffd700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 0x7f893006d190 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f894409e6c0 tx=0x7f8938008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:23.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.482+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8934055120 con 0x7f89440c42b0 2023-06-10T18:52:23.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.715+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89441546e0 con 0x7f89440c42b0 2023-06-10T18:52:23.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.715+0000 7f892b7fe700 1 -- 172.21.15.104:0/219481062 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f8934058740 con 0x7f89440c42b0 2023-06-10T18:52:23.720 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:23.720 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 msgr2=0x7f893006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 0x7f893006d190 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f894409e6c0 tx=0x7f8938008040 comp rx=0 tx=0).stop 2023-06-10T18:52:23.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 msgr2=0x7f89441efae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:23.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8934004fd0 tx=0x7f8934007620 comp rx=0 tx=0).stop 2023-06-10T18:52:23.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 shutdown_connections 2023-06-10T18:52:23.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f893006ace0 0x7f893006d190 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89441ef710 0x7f89440c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89441f0020 0x7f89441e3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 --2- 172.21.15.104:0/219481062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89440c42b0 0x7f89441efae0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 >> 172.21.15.104:0/219481062 conn(0x7f8944089fd0 msgr2=0x7f894406fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 shutdown_connections 2023-06-10T18:52:23.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:23.717+0000 7f8949750700 1 -- 172.21.15.104:0/219481062 wait complete. 2023-06-10T18:52:23.725 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:23.734 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:24.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.864+0000 7fb55d942700 1 Processor -- start 2023-06-10T18:52:24.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.864+0000 7fb55d942700 1 -- start start 2023-06-10T18:52:24.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb55d942700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb5580cea60 0x7fb55806b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:24.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb55d942700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806bfd0 con 0x7fb5580b1e30 2023-06-10T18:52:24.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb55d942700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806c2b0 con 0x7fb5580d0c00 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb55d942700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580ccf70 con 0x7fb5580cea60 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb5577fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb5580cea60 0x7fb55806b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb556ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb5580b1e30 0x7fb5580d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59390/0 (socket says 172.21.15.104:59390) 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb556ffd700 1 -- 172.21.15.104:0/3772407320 learned_addr learned my addr 172.21.15.104:0/3772407320 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb5577fe700 1 -- 172.21.15.104:0/3772407320 >> v1:172.21.15.114:6789/0 conn(0x7fb5580d0c00 legacy=0x7fb55806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb5577fe700 1 -- 172.21.15.104:0/3772407320 >> v1:172.21.15.104:6789/0 conn(0x7fb5580b1e30 legacy=0x7fb5580d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb5577fe700 1 -- 172.21.15.104:0/3772407320 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580cd250 con 0x7fb5580cea60 2023-06-10T18:52:24.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.865+0000 7fb5577fe700 1 --2- 172.21.15.104:0/3772407320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580cea60 0x7fb55806b9f0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fb550009d90 tx=0x7fb55000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea1dbeb4e56db7d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:24.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb555ffb700 1 -- 172.21.15.104:0/3772407320 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55000e070 con 0x7fb5580cea60 2023-06-10T18:52:24.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb555ffb700 1 -- 172.21.15.104:0/3772407320 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55000bb90 con 0x7fb5580cea60 2023-06-10T18:52:24.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb555ffb700 1 -- 172.21.15.104:0/3772407320 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550003a20 con 0x7fb5580cea60 2023-06-10T18:52:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 -- 172.21.15.104:0/3772407320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580cea60 msgr2=0x7fb55806b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 --2- 172.21.15.104:0/3772407320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580cea60 0x7fb55806b9f0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fb550009d90 tx=0x7fb55000b330 comp rx=0 tx=0).stop 2023-06-10T18:52:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 -- 172.21.15.104:0/3772407320 shutdown_connections 2023-06-10T18:52:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 --2- 172.21.15.104:0/3772407320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580cea60 0x7fb55806b9f0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:24.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 -- 172.21.15.104:0/3772407320 >> 172.21.15.104:0/3772407320 conn(0x7fb558071ec0 msgr2=0x7fb55809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 -- 172.21.15.104:0/3772407320 shutdown_connections 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.866+0000 7fb55d942700 1 -- 172.21.15.104:0/3772407320 wait complete. 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.867+0000 7fb55d942700 1 Processor -- start 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 -- start start 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5580cea60 0x7fb558155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:24.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5580d0c00 0x7fb5580b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581572f0 con 0x7fb5580b1e30 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581566e0 con 0x7fb5580cea60 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb55d942700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581569c0 con 0x7fb5580d0c00 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb556ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb556ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33336/0 (socket says 172.21.15.104:33336) 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.868+0000 7fb556ffd700 1 -- 172.21.15.104:0/2031240116 learned_addr learned my addr 172.21.15.104:0/2031240116 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:24.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb5577fe700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5580d0c00 0x7fb5580b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb5567fc700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5580cea60 0x7fb558155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 -- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5580d0c00 msgr2=0x7fb5580b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5580d0c00 0x7fb5580b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 -- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5580cea60 msgr2=0x7fb558155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5580cea60 0x7fb558155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 -- 172.21.15.104:0/2031240116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550009a70 con 0x7fb5580b1e30 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb556ffd700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c430 tx=0x7fb54c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:24.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c005d70 con 0x7fb5580b1e30 2023-06-10T18:52:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c0070d0 con 0x7fb5580b1e30 2023-06-10T18:52:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb558156ca0 con 0x7fb5580b1e30 2023-06-10T18:52:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c007460 con 0x7fb5580b1e30 2023-06-10T18:52:24.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.869+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5581563f0 con 0x7fb5580b1e30 2023-06-10T18:52:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.870+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5580d1860 con 0x7fb5580b1e30 2023-06-10T18:52:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.870+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb54c0075c0 con 0x7fb5580b1e30 2023-06-10T18:52:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.871+0000 7fb53ffff700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 0x7fb54406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.871+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fb54c088ab0 con 0x7fb5580b1e30 2023-06-10T18:52:24.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.871+0000 7fb5567fc700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 0x7fb54406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.872+0000 7fb5567fc700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 0x7fb54406d190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb55806ba80 tx=0x7fb548008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:24.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:24.875+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb54c0539a0 con 0x7fb5580b1e30 2023-06-10T18:52:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.105+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb558074050 con 0x7fb5580b1e30 2023-06-10T18:52:25.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.105+0000 7fb53ffff700 1 -- 172.21.15.104:0/2031240116 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fb54c017090 con 0x7fb5580b1e30 2023-06-10T18:52:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:25.110 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.107+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 msgr2=0x7fb54406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.107+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 0x7fb54406d190 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb55806ba80 tx=0x7fb548008040 comp rx=0 tx=0).stop 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.107+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 msgr2=0x7fb558150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.107+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c430 tx=0x7fb54c00c7f0 comp rx=0 tx=0).stop 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 shutdown_connections 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5580d0c00 0x7fb5580b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5580b1e30 0x7fb558150950 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54406ace0 0x7fb54406d190 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 --2- 172.21.15.104:0/2031240116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5580cea60 0x7fb558155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 >> 172.21.15.104:0/2031240116 conn(0x7fb558071ec0 msgr2=0x7fb5580d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 shutdown_connections 2023-06-10T18:52:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:25.108+0000 7fb55d942700 1 -- 172.21.15.104:0/2031240116 wait complete. 2023-06-10T18:52:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:25.124 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:26.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:26.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.251+0000 7f51079b6700 1 Processor -- start 2023-06-10T18:52:26.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 -- start start 2023-06-10T18:52:26.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f51000701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f510015a250 0x7f51000cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000ccca0 con 0x7f51000cad90 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000d0310 con 0x7f510015a250 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.252+0000 7f51079b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51000cee50 con 0x7f51000cfa00 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5105f53700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f51000cad90 0x7f51000b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59406/0 (socket says 172.21.15.104:59406) 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5105f53700 1 -- 172.21.15.104:0/3994891427 learned_addr learned my addr 172.21.15.104:0/3994891427 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:26.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5104f51700 1 --2- 172.21.15.104:0/3994891427 >> v2:172.21.15.114:3300/0 conn(0x7f510015a250 0x7f51000cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5105752700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f51000701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669860531 0 0) 0x7f51000ccca0 con 0x7f51000cad90 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec0035b0 con 0x7f51000cad90 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490925688 0 0) 0x7f50ec0035b0 con 0x7f51000cad90 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 msgr2=0x7f51000701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f51000701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 >> v2:172.21.15.114:3300/0 conn(0x7f510015a250 msgr2=0x7f51000cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 --2- 172.21.15.104:0/3994891427 >> v2:172.21.15.114:3300/0 conn(0x7f510015a250 0x7f51000cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000cf130 con 0x7f51000cad90 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f50f80019f0 con 0x7f51000cad90 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 >> v1:172.21.15.104:6789/0 conn(0x7f51000cad90 legacy=0x7f51000b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f50ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f50ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50ec004d90 0x7f50ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000d0310 con 0x7f50ec003a00 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000cee50 con 0x7f50ec004d90 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec00ba10 con 0x7f51000cfa00 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5105752700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f50ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5104f51700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f50ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.253+0000 7f5105f53700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50ec004d90 0x7f50ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 -- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 msgr2=0x7f50ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51000cfa00 0x7f50ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 -- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50ec004d90 msgr2=0x7f50ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50ec004d90 0x7f50ec005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 -- 172.21.15.104:0/3994891427 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0035b0 con 0x7f50ec003a00 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105f53700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50ec004d90 0x7f50ec005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105752700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f50ec003e10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009b60 tx=0x7f50ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c414e831e0dfcc6d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400e6c0 con 0x7f50ec003a00 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f400e820 con 0x7f50ec003a00 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f50f3fff700 1 -- 172.21.15.104:0/3994891427 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400eb90 con 0x7f50ec003a00 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f51079b6700 1 -- 172.21.15.104:0/3994891427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 msgr2=0x7f50ec003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f51079b6700 1 --2- 172.21.15.104:0/3994891427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f50ec003e10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009b60 tx=0x7f50ec005880 comp rx=0 tx=0).stop 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f5105f53700 1 -- 172.21.15.104:0/3994891427 reap_dead start 2023-06-10T18:52:26.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f51079b6700 1 -- 172.21.15.104:0/3994891427 shutdown_connections 2023-06-10T18:52:26.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f51079b6700 1 -- 172.21.15.104:0/3994891427 >> 172.21.15.104:0/3994891427 conn(0x7f510006e370 msgr2=0x7f510006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:26.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.254+0000 7f51079b6700 1 -- 172.21.15.104:0/3994891427 shutdown_connections 2023-06-10T18:52:26.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.255+0000 7f51079b6700 1 -- 172.21.15.104:0/3994891427 wait complete. 2023-06-10T18:52:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 Processor -- start 2023-06-10T18:52:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 -- start start 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 0x7f5100156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 0x7f510006ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5100157a60 con 0x7f50ec003a00 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5100157d40 con 0x7f5100158b00 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f51079b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5100158020 con 0x7f51001574e0 2023-06-10T18:52:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f5105752700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f5105752700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33354/0 (socket says 172.21.15.104:33354) 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.256+0000 7f5105752700 1 -- 172.21.15.104:0/3217856567 learned_addr learned my addr 172.21.15.104:0/3217856567 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105f53700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 0x7f510006ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5104f51700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 0x7f5100156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 -- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 msgr2=0x7f510006ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 0x7f510006ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 -- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 msgr2=0x7f5100156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 0x7f5100156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 -- 172.21.15.104:0/3217856567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100158300 con 0x7f50ec003a00 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105f53700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 0x7f510006ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5104f51700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 0x7f5100156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f5105752700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f50ec004720 tx=0x7f50f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4021070 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f40045d0 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400e6c0 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51001585e0 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.257+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510014f390 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.258+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5100158950 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.259+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f50f4025070 con 0x7f50ec003a00 2023-06-10T18:52:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.259+0000 7f50f1ffb700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 0x7f50e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.259+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f50f408a840 con 0x7f50ec003a00 2023-06-10T18:52:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.259+0000 7f5104f51700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 0x7f50e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.260+0000 7f5104f51700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 0x7f50e406d190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51000b32a0 tx=0x7f50fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.263+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f50f40558e0 con 0x7f50ec003a00 2023-06-10T18:52:26.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.494+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51000cc910 con 0x7f50ec003a00 2023-06-10T18:52:26.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.494+0000 7f50f1ffb700 1 -- 172.21.15.104:0/3217856567 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f51000cc910 con 0x7f50ec003a00 2023-06-10T18:52:26.499 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:26.499 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:26.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.496+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 msgr2=0x7f50e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 0x7f50e406d190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51000b32a0 tx=0x7f50fc008040 comp rx=0 tx=0).stop 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 msgr2=0x7f510014ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f50ec004720 tx=0x7f50f40047c0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 shutdown_connections 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51001574e0 0x7f510006ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50ec003a00 0x7f510014ee50 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f50e406ace0 0x7f50e406d190 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 --2- 172.21.15.104:0/3217856567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5100158b00 0x7f5100156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 >> 172.21.15.104:0/3217856567 conn(0x7f510006e370 msgr2=0x7f51001544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 shutdown_connections 2023-06-10T18:52:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:26.497+0000 7f51079b6700 1 -- 172.21.15.104:0/3217856567 wait complete. 2023-06-10T18:52:26.503 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:26.513 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:27.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:27.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.642+0000 7f1628fca700 1 Processor -- start 2023-06-10T18:52:27.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.642+0000 7f1628fca700 1 -- start start 2023-06-10T18:52:27.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1628fca700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16240b1070 0x7f16240b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1628fca700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16240aec00 0x7f16240ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1628fca700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162406e5e0 con 0x7f162406ea70 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1628fca700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16240b1980 con 0x7f16240b1070 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1628fca700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16240ae7e0 con 0x7f16240aec00 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16240b1070 0x7f16240b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f16240b1070 0x7f16240b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33362/0 (socket says 172.21.15.104:33362) 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 -- 172.21.15.104:0/2202208157 learned_addr learned my addr 172.21.15.104:0/2202208157 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 -- 172.21.15.104:0/2202208157 >> v2:172.21.15.114:3300/0 conn(0x7f16240aec00 msgr2=0x7f16240ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:27.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f1622d9d700 1 --2- 172.21.15.104:0/2202208157 >> v2:172.21.15.114:3300/0 conn(0x7f16240aec00 0x7f16240ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 --2- 172.21.15.104:0/2202208157 >> v2:172.21.15.114:3300/0 conn(0x7f16240aec00 0x7f16240ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 -- 172.21.15.104:0/2202208157 >> v1:172.21.15.114:6789/0 conn(0x7f162406ea70 legacy=0x7f16240ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 -- 172.21.15.104:0/2202208157 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162406df90 con 0x7f16240b1070 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.643+0000 7f162259c700 1 --2- 172.21.15.104:0/2202208157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240b1070 0x7f16240b1440 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f16240cd0c0 tx=0x7f161400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4a04fca3f3fa50 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f162159a700 1 -- 172.21.15.104:0/2202208157 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614005e30 con 0x7f16240b1070 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f162159a700 1 -- 172.21.15.104:0/2202208157 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161400d8b0 con 0x7f16240b1070 2023-06-10T18:52:27.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f162159a700 1 -- 172.21.15.104:0/2202208157 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161400dc40 con 0x7f16240b1070 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 -- 172.21.15.104:0/2202208157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240b1070 msgr2=0x7f16240b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 --2- 172.21.15.104:0/2202208157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240b1070 0x7f16240b1440 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f16240cd0c0 tx=0x7f161400be70 comp rx=0 tx=0).stop 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 -- 172.21.15.104:0/2202208157 shutdown_connections 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 --2- 172.21.15.104:0/2202208157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240b1070 0x7f16240b1440 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 --2- 172.21.15.104:0/2202208157 >> v2:172.21.15.114:3300/0 conn(0x7f16240aec00 0x7f16240ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 -- 172.21.15.104:0/2202208157 >> 172.21.15.104:0/2202208157 conn(0x7f162406b8a0 msgr2=0x7f16240cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 -- 172.21.15.104:0/2202208157 shutdown_connections 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.644+0000 7f1628fca700 1 -- 172.21.15.104:0/2202208157 wait complete. 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 Processor -- start 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 -- start start 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f162406ea70 0x7f16240b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 0x7f1624155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16240b4ea0 con 0x7f16240aec00 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16240b4570 con 0x7f162406ea70 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1628fca700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16240b4850 con 0x7f16240b1070 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.646+0000 7f1621d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33370/0 (socket says 172.21.15.104:33370) 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 -- 172.21.15.104:0/975949932 learned_addr learned my addr 172.21.15.104:0/975949932 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f162259c700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f162406ea70 0x7f16240b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1622d9d700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 0x7f1624155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 -- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 msgr2=0x7f1624155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 0x7f1624155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 -- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f162406ea70 msgr2=0x7f16240b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f162406ea70 0x7f16240b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 -- 172.21.15.104:0/975949932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161400b7a0 con 0x7f16240aec00 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1622d9d700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 0x7f1624155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:27.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1621d9b700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f161800c4d0 tx=0x7f161800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618011790 con 0x7f16240aec00 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16180118f0 con 0x7f16240aec00 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16240b4b30 con 0x7f16240aec00 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618011ca0 con 0x7f16240aec00 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.647+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16240b41b0 con 0x7f16240aec00 2023-06-10T18:52:27.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.648+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16240ceeb0 con 0x7f16240aec00 2023-06-10T18:52:27.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.649+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f161801c460 con 0x7f16240aec00 2023-06-10T18:52:27.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.649+0000 7f1612ffd700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 0x7f160406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:27.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.649+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f1618089480 con 0x7f16240aec00 2023-06-10T18:52:27.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.649+0000 7f162259c700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 0x7f160406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:27.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.650+0000 7f162259c700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 0x7f160406d190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1614000c00 tx=0x7f1614012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:27.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.653+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f16180542e0 con 0x7f16240aec00 2023-06-10T18:52:27.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.887+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16240b4340 con 0x7f16240aec00 2023-06-10T18:52:27.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.887+0000 7f1612ffd700 1 -- 172.21.15.104:0/975949932 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f1618057900 con 0x7f16240aec00 2023-06-10T18:52:27.892 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:27.893 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.889+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 msgr2=0x7f160406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.889+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 0x7f160406d190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1614000c00 tx=0x7f1614012000 comp rx=0 tx=0).stop 2023-06-10T18:52:27.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.889+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 msgr2=0x7f1624154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.889+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f161800c4d0 tx=0x7f161800c7e0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 shutdown_connections 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f160406ace0 0x7f160406d190 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f162406ea70 0x7f16240b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16240b1070 0x7f1624155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 --2- 172.21.15.104:0/975949932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f16240aec00 0x7f1624154c00 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:27.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 >> 172.21.15.104:0/975949932 conn(0x7f162406b8a0 msgr2=0x7f16240b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 shutdown_connections 2023-06-10T18:52:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:27.890+0000 7f1628fca700 1 -- 172.21.15.104:0/975949932 wait complete. 2023-06-10T18:52:27.896 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:27.906 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:28.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:29.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.026+0000 7f29a6f3e700 1 Processor -- start 2023-06-10T18:52:29.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.026+0000 7f29a6f3e700 1 -- start start 2023-06-10T18:52:29.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a6f3e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29a006e0e0 0x7f29a006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a6f3e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a006d290 con 0x7f29a0069e70 2023-06-10T18:52:29.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a6f3e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a00ca2d0 con 0x7f29a006e4b0 2023-06-10T18:52:29.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a6f3e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a0071640 con 0x7f29a006e0e0 2023-06-10T18:52:29.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a54db700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f29a0069e70 0x7f29a00ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59440/0 (socket says 172.21.15.104:59440) 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a54db700 1 -- 172.21.15.104:0/2108471407 learned_addr learned my addr 172.21.15.104:0/2108471407 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044916677 0 0) 0x7f29a006d290 con 0x7f29a0069e70 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29900035b0 con 0x7f29a0069e70 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f29a4cda700 1 --2- 172.21.15.104:0/2108471407 >> v2:172.21.15.114:3300/0 conn(0x7f29a006e0e0 0x7f29a006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417470023 0 0) 0x7f29900035b0 con 0x7f29a0069e70 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 >> v2:172.21.15.114:3300/0 conn(0x7f29a006e0e0 msgr2=0x7f29a006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:29.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 --2- 172.21.15.104:0/2108471407 >> v2:172.21.15.114:3300/0 conn(0x7f29a006e0e0 0x7f29a006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 >> v1:172.21.15.114:6789/0 conn(0x7f29a006e4b0 legacy=0x7f29a00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a006d4c0 con 0x7f29a0069e70 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f29980019f0 con 0x7f29a0069e70 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 >> v1:172.21.15.104:6789/0 conn(0x7f29a0069e70 legacy=0x7f29a00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 0x7f2990003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990004850 0x7f2990004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.027+0000 7f299f7fe700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2990005320 0x7f299000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0071640 con 0x7f2990003a00 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299000c250 con 0x7f2990005320 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299000c410 con 0x7f2990004850 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 0x7f2990003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a54db700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2990005320 0x7f299000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299ffff700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990004850 0x7f2990004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 -- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990004850 msgr2=0x7f2990004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:29.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990004850 0x7f2990004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 -- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2990005320 msgr2=0x7f299000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2990005320 0x7f299000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 -- 172.21.15.104:0/2108471407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29900035b0 con 0x7f2990003a00 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a54db700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2990005320 0x7f299000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a54db700 1 -- 172.21.15.104:0/2108471407 reap_dead start 2023-06-10T18:52:29.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f29a4cda700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 0x7f2990003df0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2994009880 tx=0x7f2990005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a192a5343ef00ff4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299400c070 con 0x7f2990003a00 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994005c20 con 0x7f2990003a00 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.028+0000 7f299f7fe700 1 -- 172.21.15.104:0/2108471407 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994006050 con 0x7f2990003a00 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 -- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 msgr2=0x7f2990003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 0x7f2990003df0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2994009880 tx=0x7f2990005880 comp rx=0 tx=0).stop 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 -- 172.21.15.104:0/2108471407 shutdown_connections 2023-06-10T18:52:29.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/2108471407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2990003a00 0x7f2990003df0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 -- 172.21.15.104:0/2108471407 >> 172.21.15.104:0/2108471407 conn(0x7f29a006b8a0 msgr2=0x7f29a00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:29.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 -- 172.21.15.104:0/2108471407 shutdown_connections 2023-06-10T18:52:29.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.029+0000 7f29a6f3e700 1 -- 172.21.15.104:0/2108471407 wait complete. 2023-06-10T18:52:29.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.030+0000 7f29a6f3e700 1 Processor -- start 2023-06-10T18:52:29.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.030+0000 7f29a6f3e700 1 -- start start 2023-06-10T18:52:29.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 0x7f29a00b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 0x7f29a00b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a01e68f0 con 0x7f29a014a660 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a01e6610 con 0x7f29a01e60e0 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a00b2a90 con 0x7f2990003a00 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33392/0 (socket says 172.21.15.104:33392) 2023-06-10T18:52:29.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 -- 172.21.15.104:0/626245119 learned_addr learned my addr 172.21.15.104:0/626245119 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a4cda700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 0x7f29a00b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f299ffff700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 0x7f29a00b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 -- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 msgr2=0x7f29a00b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 0x7f29a00b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 -- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 msgr2=0x7f29a00b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 0x7f29a00b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 -- 172.21.15.104:0/626245119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a00b2d70 con 0x7f29a014a660 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f299ffff700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 0x7f29a00b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a4cda700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 0x7f29a00b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a54db700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f29a006e880 tx=0x7f2998007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299800ec60 con 0x7f29a014a660 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299800edc0 con 0x7f29a014a660 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a00b3050 con 0x7f29a014a660 2023-06-10T18:52:29.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998013640 con 0x7f29a014a660 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.031+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a00b0e70 con 0x7f29a014a660 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.032+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a00b28f0 con 0x7f29a014a660 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.033+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f29980137a0 con 0x7f29a014a660 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.033+0000 7f299d7fa700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 0x7f298406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.033+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f2998089c80 con 0x7f29a014a660 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.033+0000 7f29a4cda700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 0x7f298406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.034+0000 7f29a4cda700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 0x7f298406d190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2994008010 tx=0x7f299400c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:29.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.037+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f29a00b28f0 con 0x7f29a014a660 2023-06-10T18:52:29.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.261+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a00b28f0 con 0x7f29a014a660 2023-06-10T18:52:29.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.262+0000 7f299d7fa700 1 -- 172.21.15.104:0/626245119 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f29a00b28f0 con 0x7f29a014a660 2023-06-10T18:52:29.268 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:29.268 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:29.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 msgr2=0x7f298406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 0x7f298406d190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2994008010 tx=0x7f299400c000 comp rx=0 tx=0).stop 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 msgr2=0x7f29a00b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f29a006e880 tx=0x7f2998007c90 comp rx=0 tx=0).stop 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 shutdown_connections 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a014a660 0x7f29a00b3ab0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2990003a00 0x7f29a00b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f298406ace0 0x7f298406d190 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 --2- 172.21.15.104:0/626245119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29a01e60e0 0x7f29a00b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 >> 172.21.15.104:0/626245119 conn(0x7f29a006b8a0 msgr2=0x7f29a00b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 shutdown_connections 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:29.264+0000 7f29a6f3e700 1 -- 172.21.15.104:0/626245119 wait complete. 2023-06-10T18:52:29.271 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:29.281 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:30.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:30.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.409+0000 7f70e1913700 1 Processor -- start 2023-06-10T18:52:30.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.409+0000 7f70e1913700 1 -- start start 2023-06-10T18:52:30.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70e1913700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70dc070a20 0x7f70dc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70e1913700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 0x7f70dc159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70e1913700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc06ae00 con 0x7f70dc070df0 2023-06-10T18:52:30.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70e1913700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc069e70 con 0x7f70dc070a20 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70e1913700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc1595f0 con 0x7f70dc158eb0 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70daffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70dc070a20 0x7f70dc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70daffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f70dc070a20 0x7f70dc06a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33396/0 (socket says 172.21.15.104:33396) 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.410+0000 7f70daffd700 1 -- 172.21.15.104:0/1090381870 learned_addr learned my addr 172.21.15.104:0/1090381870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70db7fe700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 0x7f70dc159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70daffd700 1 -- 172.21.15.104:0/1090381870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 msgr2=0x7f70dc159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70daffd700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 0x7f70dc159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70daffd700 1 -- 172.21.15.104:0/1090381870 >> v1:172.21.15.114:6789/0 conn(0x7f70dc070df0 legacy=0x7f70dc1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70daffd700 1 -- 172.21.15.104:0/1090381870 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc1598d0 con 0x7f70dc070a20 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70db7fe700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 0x7f70dc159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70daffd700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc070a20 0x7f70dc06a8c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f70d00098f0 tx=0x7f70d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b86bc21e73825b7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70d9ffb700 1 -- 172.21.15.104:0/1090381870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0004230 con 0x7f70dc070a20 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70d9ffb700 1 -- 172.21.15.104:0/1090381870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d0004390 con 0x7f70dc070a20 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70d9ffb700 1 -- 172.21.15.104:0/1090381870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d00036a0 con 0x7f70dc070a20 2023-06-10T18:52:30.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70e1913700 1 -- 172.21.15.104:0/1090381870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc070a20 msgr2=0x7f70dc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70e1913700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc070a20 0x7f70dc06a8c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f70d00098f0 tx=0x7f70d0009c00 comp rx=0 tx=0).stop 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.411+0000 7f70e1913700 1 -- 172.21.15.104:0/1090381870 shutdown_connections 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.412+0000 7f70e1913700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc070a20 0x7f70dc06a8c0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.412+0000 7f70e1913700 1 --2- 172.21.15.104:0/1090381870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc158eb0 0x7f70dc159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.412+0000 7f70e1913700 1 -- 172.21.15.104:0/1090381870 >> 172.21.15.104:0/1090381870 conn(0x7f70dc0b1610 msgr2=0x7f70dc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.412+0000 7f70e1913700 1 -- 172.21.15.104:0/1090381870 shutdown_connections 2023-06-10T18:52:30.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.412+0000 7f70e1913700 1 -- 172.21.15.104:0/1090381870 wait complete. 2023-06-10T18:52:30.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.413+0000 7f70e1913700 1 Processor -- start 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.413+0000 7f70e1913700 1 -- start start 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 0x7f70dc153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70dc070df0 0x7f70dc0c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc153a40 con 0x7f70dc158eb0 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0c4bc0 con 0x7f70dc070df0 2023-06-10T18:52:30.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0c4040 con 0x7f70dc070a20 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33406/0 (socket says 172.21.15.104:33406) 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 -- 172.21.15.104:0/4048681612 learned_addr learned my addr 172.21.15.104:0/4048681612 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70da7fc700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70dc070df0 0x7f70dc0c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70daffd700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 0x7f70dc153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 -- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 msgr2=0x7f70dc153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:30.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 0x7f70dc153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 -- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70dc070df0 msgr2=0x7f70dc0c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70dc070df0 0x7f70dc0c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 -- 172.21.15.104:0/4048681612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0009550 con 0x7f70dc158eb0 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70daffd700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 0x7f70dc153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70db7fe700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f70d400cd50 tx=0x7f70d400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400cea0 con 0x7f70dc158eb0 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400f480 con 0x7f70dc158eb0 2023-06-10T18:52:30.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc0c4320 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.414+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400f810 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.415+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc0c48e0 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.415+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc071780 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.416+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f70d400b040 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.416+0000 7f70c3fff700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 0x7f70c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.416+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f70d4089d10 con 0x7f70dc158eb0 2023-06-10T18:52:30.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.416+0000 7f70daffd700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 0x7f70c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:30.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.417+0000 7f70daffd700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 0x7f70c806d190 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f70dc1f0d00 tx=0x7f70d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:30.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.421+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f70d4054c90 con 0x7f70dc158eb0 2023-06-10T18:52:30.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.647+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70dc0b1c80 con 0x7f70dc158eb0 2023-06-10T18:52:30.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.647+0000 7f70c3fff700 1 -- 172.21.15.104:0/4048681612 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f70dc0b1c80 con 0x7f70dc158eb0 2023-06-10T18:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:30.653 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:30.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 msgr2=0x7f70c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:30.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 0x7f70c806d190 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f70dc1f0d00 tx=0x7f70d0008030 comp rx=0 tx=0).stop 2023-06-10T18:52:30.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 msgr2=0x7f70dc156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f70d400cd50 tx=0x7f70d400a560 comp rx=0 tx=0).stop 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 shutdown_connections 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70dc158eb0 0x7f70dc156100 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70dc070a20 0x7f70dc153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f70c806ace0 0x7f70c806d190 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 --2- 172.21.15.104:0/4048681612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70dc070df0 0x7f70dc0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:30.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.649+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 >> 172.21.15.104:0/4048681612 conn(0x7f70dc0b1610 msgr2=0x7f70dc06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:30.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.650+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 shutdown_connections 2023-06-10T18:52:30.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:30.650+0000 7f70e1913700 1 -- 172.21.15.104:0/4048681612 wait complete. 2023-06-10T18:52:30.657 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:30.667 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:31.668 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.795+0000 7f72fc3b5700 1 Processor -- start 2023-06-10T18:52:31.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.795+0000 7f72fc3b5700 1 -- start start 2023-06-10T18:52:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fc3b5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72f406e0e0 0x7f72f406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fc3b5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f406d290 con 0x7f72f4069e70 2023-06-10T18:52:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fc3b5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f40ca2d0 con 0x7f72f406e4b0 2023-06-10T18:52:31.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fc3b5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f4071640 con 0x7f72f406e0e0 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fa952700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f72f4069e70 0x7f72f40ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59470/0 (socket says 172.21.15.104:59470) 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.796+0000 7f72fa952700 1 -- 172.21.15.104:0/102992481 learned_addr learned my addr 172.21.15.104:0/102992481 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125273331 0 0) 0x7f72f406d290 con 0x7f72f4069e70 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e00035b0 con 0x7f72f4069e70 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72fa151700 1 --2- 172.21.15.104:0/102992481 >> v2:172.21.15.114:3300/0 conn(0x7f72f406e0e0 0x7f72f406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2764130237 0 0) 0x7f72e00035b0 con 0x7f72f4069e70 2023-06-10T18:52:31.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 >> v2:172.21.15.114:3300/0 conn(0x7f72f406e0e0 msgr2=0x7f72f406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 --2- 172.21.15.104:0/102992481 >> v2:172.21.15.114:3300/0 conn(0x7f72f406e0e0 0x7f72f406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 >> v1:172.21.15.114:6789/0 conn(0x7f72f406e4b0 legacy=0x7f72f40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f406d4c0 con 0x7f72f4069e70 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f72ec0019f0 con 0x7f72f4069e70 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 >> v1:172.21.15.104:6789/0 conn(0x7f72f4069e70 legacy=0x7f72f40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e0003a00 0x7f72e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 0x7f72e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0005320 0x7f72e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f4071640 con 0x7f72e0004850 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000c250 con 0x7f72e0005320 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000c410 con 0x7f72e0003a00 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 0x7f72e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72fa151700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e0003a00 0x7f72e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72fa952700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0005320 0x7f72e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 -- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e0003a00 msgr2=0x7f72e0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e0003a00 0x7f72e0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 -- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0005320 msgr2=0x7f72e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0005320 0x7f72e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72f9950700 1 -- 172.21.15.104:0/102992481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00035b0 con 0x7f72e0004850 2023-06-10T18:52:31.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72fa952700 1 -- 172.21.15.104:0/102992481 reap_dead start 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.797+0000 7f72fa151700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e0003a00 0x7f72e0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72f9950700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 0x7f72e0004cc0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f72f00017c0 tx=0x7f72e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abeddf546c1db16 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00052d0 con 0x7f72e0004850 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0005450 con 0x7f72e0004850 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72f914f700 1 -- 172.21.15.104:0/102992481 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00038c0 con 0x7f72e0004850 2023-06-10T18:52:31.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 -- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 msgr2=0x7f72e0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 0x7f72e0004cc0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f72f00017c0 tx=0x7f72e0005880 comp rx=0 tx=0).stop 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 -- 172.21.15.104:0/102992481 shutdown_connections 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/102992481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72e0004850 0x7f72e0004cc0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 -- 172.21.15.104:0/102992481 >> 172.21.15.104:0/102992481 conn(0x7f72f406b8a0 msgr2=0x7f72f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 -- 172.21.15.104:0/102992481 shutdown_connections 2023-06-10T18:52:31.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.798+0000 7f72fc3b5700 1 -- 172.21.15.104:0/102992481 wait complete. 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.799+0000 7f72fc3b5700 1 Processor -- start 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 -- start start 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 0x7f72f40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72f4159380 0x7f72f40cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41582e0 con 0x7f72f4156f30 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41585c0 con 0x7f72e0004850 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fc3b5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41588a0 con 0x7f72f4159380 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fa952700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fa952700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33426/0 (socket says 172.21.15.104:33426) 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fa952700 1 -- 172.21.15.104:0/549839375 learned_addr learned my addr 172.21.15.104:0/549839375 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72f9950700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72f4159380 0x7f72f40cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.800+0000 7f72fa151700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 0x7f72f40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 -- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72f4159380 msgr2=0x7f72f40cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72f4159380 0x7f72f40cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 -- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 msgr2=0x7f72f40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 0x7f72f40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 -- 172.21.15.104:0/549839375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4158b80 con 0x7f72f4156f30 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa151700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 0x7f72f40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fa952700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f72f4072080 tx=0x7f72ec00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:31.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00ec30 con 0x7f72f4156f30 2023-06-10T18:52:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec00ed90 con 0x7f72f4156f30 2023-06-10T18:52:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f4158e60 con 0x7f72f4156f30 2023-06-10T18:52:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec013660 con 0x7f72f4156f30 2023-06-10T18:52:31.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.801+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f41578e0 con 0x7f72f4156f30 2023-06-10T18:52:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.802+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f40cdd80 con 0x7f72f4156f30 2023-06-10T18:52:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.802+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f72ec0137c0 con 0x7f72f4156f30 2023-06-10T18:52:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.803+0000 7f72eaffd700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 0x7f72d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:31.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.803+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f72ec089c20 con 0x7f72f4156f30 2023-06-10T18:52:31.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.803+0000 7f72fa151700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 0x7f72d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:31.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.803+0000 7f72fa151700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 0x7f72d806d190 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f72f4063ac0 tx=0x7f72e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:31.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:31.807+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f72ec054b10 con 0x7f72f4156f30 2023-06-10T18:52:32.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.035+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72f4159050 con 0x7f72f4156f30 2023-06-10T18:52:32.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.036+0000 7f72eaffd700 1 -- 172.21.15.104:0/549839375 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f72ec058130 con 0x7f72f4156f30 2023-06-10T18:52:32.042 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:32.042 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:32.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.037+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 msgr2=0x7f72d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:32.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.037+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 0x7f72d806d190 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f72f4063ac0 tx=0x7f72e4008040 comp rx=0 tx=0).stop 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.037+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 msgr2=0x7f72f41573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.037+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f72f4072080 tx=0x7f72ec00e440 comp rx=0 tx=0).stop 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 shutdown_connections 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f72d806ace0 0x7f72d806d190 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e0004850 0x7f72f40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72f4156f30 0x7f72f41573a0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:32.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 --2- 172.21.15.104:0/549839375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72f4159380 0x7f72f40cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:32.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 >> 172.21.15.104:0/549839375 conn(0x7f72f406b8a0 msgr2=0x7f72f406a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:32.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 shutdown_connections 2023-06-10T18:52:32.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:32.038+0000 7f72fc3b5700 1 -- 172.21.15.104:0/549839375 wait complete. 2023-06-10T18:52:32.045 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:32.055 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:33.057 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:33.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.178+0000 7ff950ab3700 1 Processor -- start 2023-06-10T18:52:33.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.178+0000 7ff950ab3700 1 -- start start 2023-06-10T18:52:33.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff950ab3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff94c06c720 0x7ff94c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff950ab3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff94c0cae10 0x7ff94c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff950ab3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0d0340 con 0x7ff94c0ce390 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff950ab3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0d1660 con 0x7ff94c0cae10 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff950ab3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c15cb60 con 0x7ff94c06c720 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff94c0cae10 0x7ff94c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff94c0ce390 0x7ff94c160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59490/0 (socket says 172.21.15.104:59490) 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94a59c700 1 -- 172.21.15.104:0/3976929636 learned_addr learned my addr 172.21.15.104:0/3976929636 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 -- 172.21.15.104:0/3976929636 >> v2:172.21.15.114:3300/0 conn(0x7ff94c06c720 msgr2=0x7ff94c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 --2- 172.21.15.104:0/3976929636 >> v2:172.21.15.114:3300/0 conn(0x7ff94c06c720 0x7ff94c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 -- 172.21.15.104:0/3976929636 >> v1:172.21.15.104:6789/0 conn(0x7ff94c0ce390 legacy=0x7ff94c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 -- 172.21.15.104:0/3976929636 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0ceca0 con 0x7ff94c0cae10 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.179+0000 7ff94ad9d700 1 --2- 172.21.15.104:0/3976929636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c0cb210 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0ce340 tx=0x7ff94400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253bd390e08ec2c2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff94959a700 1 -- 172.21.15.104:0/3976929636 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400e070 con 0x7ff94c0cae10 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff94959a700 1 -- 172.21.15.104:0/3976929636 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94400bbd0 con 0x7ff94c0cae10 2023-06-10T18:52:33.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff94959a700 1 -- 172.21.15.104:0/3976929636 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944012430 con 0x7ff94c0cae10 2023-06-10T18:52:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 -- 172.21.15.104:0/3976929636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 msgr2=0x7ff94c0cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 --2- 172.21.15.104:0/3976929636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c0cb210 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0ce340 tx=0x7ff94400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 -- 172.21.15.104:0/3976929636 shutdown_connections 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 --2- 172.21.15.104:0/3976929636 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c0cb210 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 --2- 172.21.15.104:0/3976929636 >> v2:172.21.15.114:3300/0 conn(0x7ff94c06c720 0x7ff94c0ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 -- 172.21.15.104:0/3976929636 >> 172.21.15.104:0/3976929636 conn(0x7ff94c069e70 msgr2=0x7ff94c06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 -- 172.21.15.104:0/3976929636 shutdown_connections 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.180+0000 7ff950ab3700 1 -- 172.21.15.104:0/3976929636 wait complete. 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 Processor -- start 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 -- start start 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 0x7ff94c1f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 0x7ff94c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c158520 con 0x7ff94c0cae10 2023-06-10T18:52:33.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0b3810 con 0x7ff94c06c720 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff950ab3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c1578a0 con 0x7ff94c0ce390 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff949d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff949d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33436/0 (socket says 172.21.15.104:33436) 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.182+0000 7ff949d9b700 1 -- 172.21.15.104:0/4009504567 learned_addr learned my addr 172.21.15.104:0/4009504567 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff94a59c700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 0x7ff94c1f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff94ad9d700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 0x7ff94c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:33.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 -- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 msgr2=0x7ff94c0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 0x7ff94c0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 -- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 msgr2=0x7ff94c1f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 0x7ff94c1f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 -- 172.21.15.104:0/4009504567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009a70 con 0x7ff94c0cae10 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff94ad9d700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 0x7ff94c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff94a59c700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 0x7ff94c1f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff949d9b700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00b4a0 tx=0x7ff93c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:33.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c011830 con 0x7ff94c0cae10 2023-06-10T18:52:33.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c011990 con 0x7ff94c0cae10 2023-06-10T18:52:33.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c157b80 con 0x7ff94c0cae10 2023-06-10T18:52:33.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c011d40 con 0x7ff94c0cae10 2023-06-10T18:52:33.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.183+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c158140 con 0x7ff94c0cae10 2023-06-10T18:52:33.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.184+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c074050 con 0x7ff94c0cae10 2023-06-10T18:52:33.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.184+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff93c01a400 con 0x7ff94c0cae10 2023-06-10T18:52:33.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.185+0000 7ff93affd700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 0x7ff92c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:33.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.185+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7ff93c089630 con 0x7ff94c0cae10 2023-06-10T18:52:33.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.185+0000 7ff94a59c700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 0x7ff92c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:33.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.185+0000 7ff94a59c700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 0x7ff92c06d190 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff940007fd0 tx=0x7ff940009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:33.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.189+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff93c054530 con 0x7ff94c0cae10 2023-06-10T18:52:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.417+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff94c160570 con 0x7ff94c0cae10 2023-06-10T18:52:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.417+0000 7ff93affd700 1 -- 172.21.15.104:0/4009504567 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7ff93c057b50 con 0x7ff94c0cae10 2023-06-10T18:52:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:33.424 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:33.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.419+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 msgr2=0x7ff92c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.419+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 0x7ff92c06d190 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff940007fd0 tx=0x7ff940009040 comp rx=0 tx=0).stop 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.419+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 msgr2=0x7ff94c158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.419+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00b4a0 tx=0x7ff93c00b860 comp rx=0 tx=0).stop 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 shutdown_connections 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff92c06ace0 0x7ff92c06d190 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff94c06c720 0x7ff94c1f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff94c0ce390 0x7ff94c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 --2- 172.21.15.104:0/4009504567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff94c0cae10 0x7ff94c158950 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:33.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 >> 172.21.15.104:0/4009504567 conn(0x7ff94c069e70 msgr2=0x7ff94c06fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:33.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 shutdown_connections 2023-06-10T18:52:33.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:33.420+0000 7ff950ab3700 1 -- 172.21.15.104:0/4009504567 wait complete. 2023-06-10T18:52:33.427 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:33.437 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:34.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.566+0000 7f4fc795e700 1 Processor -- start 2023-06-10T18:52:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.566+0000 7f4fc795e700 1 -- start start 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc795e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fc00cf030 0x7f4fc00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc795e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc006dcb0 con 0x7f4fc0069e70 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc795e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc00cf460 con 0x7f4fc015c500 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc795e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc006e850 con 0x7f4fc00cf030 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc5efb700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4fc0069e70 0x7f4fc00c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59508/0 (socket says 172.21.15.104:59508) 2023-06-10T18:52:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.567+0000 7f4fc5efb700 1 -- 172.21.15.104:0/2324622393 learned_addr learned my addr 172.21.15.104:0/2324622393 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc4ef9700 1 --2- 172.21.15.104:0/2324622393 >> v2:172.21.15.114:3300/0 conn(0x7f4fc00cf030 0x7f4fc00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075822194 0 0) 0x7f4fc006dcb0 con 0x7f4fc0069e70 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fac0035b0 con 0x7f4fc0069e70 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913181303 0 0) 0x7f4fac0035b0 con 0x7f4fc0069e70 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 >> v2:172.21.15.114:3300/0 conn(0x7f4fc00cf030 msgr2=0x7f4fc00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 --2- 172.21.15.104:0/2324622393 >> v2:172.21.15.114:3300/0 conn(0x7f4fc00cf030 0x7f4fc00ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 >> v1:172.21.15.114:6789/0 conn(0x7f4fc015c500 legacy=0x7f4fc006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc006d0a0 con 0x7f4fc0069e70 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4fb80019f0 con 0x7f4fc0069e70 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 >> v1:172.21.15.104:6789/0 conn(0x7f4fc0069e70 legacy=0x7f4fc00c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac003a00 0x7f4fac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac004850 0x7f4fac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 0x7f4fac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc006e850 con 0x7f4fac005200 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac00c060 con 0x7f4fac003a00 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac00c220 con 0x7f4fac004850 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 0x7f4fac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc4ef9700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac004850 0x7f4fac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc56fa700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac003a00 0x7f4fac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 -- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac004850 msgr2=0x7f4fac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac004850 0x7f4fac004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 -- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac003a00 msgr2=0x7f4fac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac003a00 0x7f4fac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.568+0000 7f4fc5efb700 1 -- 172.21.15.104:0/2324622393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0035b0 con 0x7f4fac005200 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc5efb700 1 -- 172.21.15.104:0/2324622393 reap_dead start 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 0x7f4fac00ba10 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800a830 tx=0x7f4fac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2d4428ecfa86bc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800a2f0 con 0x7f4fac005200 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8012620 con 0x7f4fac005200 2023-06-10T18:52:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fb7fff700 1 -- 172.21.15.104:0/2324622393 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb80129d0 con 0x7f4fac005200 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 -- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 msgr2=0x7f4fac00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 0x7f4fac00ba10 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800a830 tx=0x7f4fac005880 comp rx=0 tx=0).stop 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 -- 172.21.15.104:0/2324622393 shutdown_connections 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 --2- 172.21.15.104:0/2324622393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fac005200 0x7f4fac00ba10 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 -- 172.21.15.104:0/2324622393 >> 172.21.15.104:0/2324622393 conn(0x7f4fc006b8a0 msgr2=0x7f4fc00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 -- 172.21.15.104:0/2324622393 shutdown_connections 2023-06-10T18:52:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.569+0000 7f4fc795e700 1 -- 172.21.15.104:0/2324622393 wait complete. 2023-06-10T18:52:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 Processor -- start 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 -- start start 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac005200 0x7f4fc01e2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc0149be0 0x7f4fc0156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc01e60e0 con 0x7f4fc01e5730 2023-06-10T18:52:34.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0156650 con 0x7f4fc0149be0 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc795e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc01568f0 con 0x7f4fac005200 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc5efb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc5efb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33460/0 (socket says 172.21.15.104:33460) 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.571+0000 7f4fc5efb700 1 -- 172.21.15.104:0/389019878 learned_addr learned my addr 172.21.15.104:0/389019878 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc56fa700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac005200 0x7f4fc01e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc4ef9700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc0149be0 0x7f4fc0156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 -- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac005200 msgr2=0x7f4fc01e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac005200 0x7f4fc01e2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 -- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc0149be0 msgr2=0x7f4fc0156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc0149be0 0x7f4fc0156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 -- 172.21.15.104:0/389019878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0156bd0 con 0x7f4fc01e5730 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc5efb700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0071770 tx=0x7f4fb8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb80017b0 con 0x7f4fc01e5730 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8001910 con 0x7f4fc01e5730 2023-06-10T18:52:34.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb801d660 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc0156eb0 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.572+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc0157470 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.573+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc0149fb0 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.574+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4fb801d9d0 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.574+0000 7f4fb5ffb700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 0x7f4fa406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.574+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f4fb808b7c0 con 0x7f4fc01e5730 2023-06-10T18:52:34.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.574+0000 7f4fc56fa700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 0x7f4fa406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:34.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.575+0000 7f4fc56fa700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 0x7f4fa406d190 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000c00 tx=0x7f4fb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:34.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.579+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4fb8056740 con 0x7f4fc01e5730 2023-06-10T18:52:34.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.807+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fc00b1f10 con 0x7f4fc01e5730 2023-06-10T18:52:34.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.808+0000 7f4fb5ffb700 1 -- 172.21.15.104:0/389019878 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f4fb8059d60 con 0x7f4fc01e5730 2023-06-10T18:52:34.813 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:34.814 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:34.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.809+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 msgr2=0x7f4fa406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.809+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 0x7f4fa406d190 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000c00 tx=0x7f4fb0009040 comp rx=0 tx=0).stop 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.809+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 msgr2=0x7f4fc01e5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.809+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0071770 tx=0x7f4fb8001d80 comp rx=0 tx=0).stop 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 shutdown_connections 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc01e5730 0x7f4fc01e5ba0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac005200 0x7f4fc01e2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4fa406ace0 0x7f4fa406d190 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 --2- 172.21.15.104:0/389019878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc0149be0 0x7f4fc0156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:34.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 >> 172.21.15.104:0/389019878 conn(0x7f4fc006b8a0 msgr2=0x7f4fc00b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:34.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 shutdown_connections 2023-06-10T18:52:34.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:34.810+0000 7f4fc795e700 1 -- 172.21.15.104:0/389019878 wait complete. 2023-06-10T18:52:34.817 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:34.827 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:35.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:35.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.956+0000 7fe4e38f9700 1 Processor -- start 2023-06-10T18:52:35.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.956+0000 7fe4e38f9700 1 -- start start 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e38f9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e38f9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e38f9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc0ccca0 con 0x7fe4dc15a250 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e38f9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc0cee50 con 0x7fe4dc0cfa00 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e38f9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0cf130 con 0x7fe4dc0cad90 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e1695700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e1e96700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e1695700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:49198/0 (socket says 172.21.15.104:49198) 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e0e94700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe4dc15a250 0x7fe4dc0cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37140/0 (socket says 172.21.15.104:37140) 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e1e96700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:52852/0 (socket says 172.21.15.104:52852) 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.957+0000 7fe4e1695700 1 -- 172.21.15.104:0/1744146384 learned_addr learned my addr 172.21.15.104:0/1744146384 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1e96700 1 -- 172.21.15.104:0/1744146384 >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 msgr2=0x7fe4dc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1e96700 1 --2- 172.21.15.104:0/1744146384 >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:35.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1e96700 1 -- 172.21.15.104:0/1744146384 >> v1:172.21.15.114:6789/0 conn(0x7fe4dc15a250 legacy=0x7fe4dc0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1e96700 1 -- 172.21.15.104:0/1744146384 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc0cc930 con 0x7fe4dc0cad90 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1695700 1 --2- 172.21.15.104:0/1744146384 >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4e1e96700 1 --2- 172.21.15.104:0/1744146384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009d90 tx=0x7fe4d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=580859e5b86c0f7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4d3fff700 1 -- 172.21.15.104:0/1744146384 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400e070 con 0x7fe4dc0cad90 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4d3fff700 1 -- 172.21.15.104:0/1744146384 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d400bb20 con 0x7fe4dc0cad90 2023-06-10T18:52:35.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.958+0000 7fe4d3fff700 1 -- 172.21.15.104:0/1744146384 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4012430 con 0x7fe4dc0cad90 2023-06-10T18:52:35.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 -- 172.21.15.104:0/1744146384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 msgr2=0x7fe4dc0b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:35.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/1744146384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009d90 tx=0x7fe4d400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 -- 172.21.15.104:0/1744146384 shutdown_connections 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/1744146384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cad90 0x7fe4dc0b3590 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/1744146384 >> v2:172.21.15.114:3300/0 conn(0x7fe4dc0cfa00 0x7fe4dc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 -- 172.21.15.104:0/1744146384 >> 172.21.15.104:0/1744146384 conn(0x7fe4dc06e370 msgr2=0x7fe4dc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 -- 172.21.15.104:0/1744146384 shutdown_connections 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.959+0000 7fe4e38f9700 1 -- 172.21.15.104:0/1744146384 wait complete. 2023-06-10T18:52:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.960+0000 7fe4e38f9700 1 Processor -- start 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.960+0000 7fe4e38f9700 1 -- start start 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4dc0cad90 0x7fe4dc1ef100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cfa00 0x7fe4dc0b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0b6410 con 0x7fe4dc15a250 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0b1f30 con 0x7fe4dc0cad90 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e38f9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0b1320 con 0x7fe4dc0cfa00 2023-06-10T18:52:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33474/0 (socket says 172.21.15.104:33474) 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 -- 172.21.15.104:0/3515804924 learned_addr learned my addr 172.21.15.104:0/3515804924 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e0e94700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cfa00 0x7fe4dc0b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1695700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4dc0cad90 0x7fe4dc1ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 -- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cfa00 msgr2=0x7fe4dc0b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cfa00 0x7fe4dc0b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 -- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4dc0cad90 msgr2=0x7fe4dc1ef100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4dc0cad90 0x7fe4dc1ef100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 -- 172.21.15.104:0/3515804924 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8009550 con 0x7fe4dc15a250 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.961+0000 7fe4e1e96700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4014040 tx=0x7fe4d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.962+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4012430 con 0x7fe4dc15a250 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.962+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4020070 con 0x7fe4dc15a250 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.962+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d4009a70 con 0x7fe4dc15a250 2023-06-10T18:52:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.962+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4004280 con 0x7fe4dc15a250 2023-06-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.962+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc0b18e0 con 0x7fe4dc15a250 2023-06-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.963+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc0b20e0 con 0x7fe4dc15a250 2023-06-10T18:52:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.963+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe4d40043e0 con 0x7fe4dc15a250 2023-06-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.964+0000 7fe4d1ffb700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 0x7fe4c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.964+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fe4d4022070 con 0x7fe4dc15a250 2023-06-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.964+0000 7fe4e1695700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 0x7fe4c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.965+0000 7fe4e1695700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 0x7fe4c006d190 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8006d30 tx=0x7fe4d80092c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:35.968+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe4dc0b20e0 con 0x7fe4dc15a250 2023-06-10T18:52:36.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.196+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4dc0b20a0 con 0x7fe4dc15a250 2023-06-10T18:52:36.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.196+0000 7fe4d1ffb700 1 -- 172.21.15.104:0/3515804924 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fe4d4056180 con 0x7fe4dc15a250 2023-06-10T18:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:36.202 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 msgr2=0x7fe4c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 0x7fe4c006d190 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8006d30 tx=0x7fe4d80092c0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 msgr2=0x7fe4dc15b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4014040 tx=0x7fe4d4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 shutdown_connections 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe4c006ace0 0x7fe4c006d190 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4dc0cad90 0x7fe4dc1ef100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4dc15a250 0x7fe4dc15b060 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 --2- 172.21.15.104:0/3515804924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4dc0cfa00 0x7fe4dc0b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 >> 172.21.15.104:0/3515804924 conn(0x7fe4dc06e370 msgr2=0x7fe4dc0cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 shutdown_connections 2023-06-10T18:52:36.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:36.198+0000 7fe4e38f9700 1 -- 172.21.15.104:0/3515804924 wait complete. 2023-06-10T18:52:36.206 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:36.216 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:37.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:37.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.344+0000 7f04a27e3700 1 Processor -- start 2023-06-10T18:52:37.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.344+0000 7f04a27e3700 1 -- start start 2023-06-10T18:52:37.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a27e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c06e680 0x7f049c0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a27e3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f049c0ae310 0x7f049c0ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a27e3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c06a840 con 0x7f049c15c2d0 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a27e3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0aeab0 con 0x7f049c0ae310 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a27e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c069f10 con 0x7f049c06e680 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f049c0ae310 0x7f049c0ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f049c0ae310 0x7f049c0ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33482/0 (socket says 172.21.15.104:33482) 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 -- 172.21.15.104:0/713386041 learned_addr learned my addr 172.21.15.104:0/713386041 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f049b7fe700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c06e680 0x7f049c0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 -- 172.21.15.104:0/713386041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c06e680 msgr2=0x7f049c0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c06e680 0x7f049c0b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 -- 172.21.15.104:0/713386041 >> v1:172.21.15.114:6789/0 conn(0x7f049c15c2d0 legacy=0x7f049c06df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 -- 172.21.15.104:0/713386041 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c06a1f0 con 0x7f049c0ae310 2023-06-10T18:52:37.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.345+0000 7f04a0d80700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0ce210 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f0494009e50 tx=0x7f049400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b836d0b14ca5f1f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f049affd700 1 -- 172.21.15.104:0/713386041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400e070 con 0x7f049c0ae310 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f049affd700 1 -- 172.21.15.104:0/713386041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0494003a20 con 0x7f049c0ae310 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f049affd700 1 -- 172.21.15.104:0/713386041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494003dd0 con 0x7f049c0ae310 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 -- 172.21.15.104:0/713386041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 msgr2=0x7f049c0ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0ce210 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f0494009e50 tx=0x7f049400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 -- 172.21.15.104:0/713386041 shutdown_connections 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0ce210 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 --2- 172.21.15.104:0/713386041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c06e680 0x7f049c0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 -- 172.21.15.104:0/713386041 >> 172.21.15.104:0/713386041 conn(0x7f049c06b8a0 msgr2=0x7f049c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:37.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 -- 172.21.15.104:0/713386041 shutdown_connections 2023-06-10T18:52:37.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.346+0000 7f04a27e3700 1 -- 172.21.15.104:0/713386041 wait complete. 2023-06-10T18:52:37.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.347+0000 7f04a27e3700 1 Processor -- start 2023-06-10T18:52:37.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 -- start start 2023-06-10T18:52:37.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c06e680 0x7f049c151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c15c2d0 0x7f049c0c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0b14b0 con 0x7f049c0ae310 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0af070 con 0x7f049c06e680 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f04a27e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0af350 con 0x7f049c15c2d0 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f049b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f049b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33486/0 (socket says 172.21.15.104:33486) 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.348+0000 7f049b7fe700 1 -- 172.21.15.104:0/3347489543 learned_addr learned my addr 172.21.15.104:0/3347489543 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:37.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049bfff700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c06e680 0x7f049c151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f04a0d80700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c15c2d0 0x7f049c0c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 -- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c15c2d0 msgr2=0x7f049c0c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c15c2d0 0x7f049c0c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 -- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c06e680 msgr2=0x7f049c151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c06e680 0x7f049c151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 -- 172.21.15.104:0/3347489543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494009b30 con 0x7f049c0ae310 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f049b7fe700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f048c00c7f0 tx=0x7f048c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:37.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c011800 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c011960 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c0af630 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c011d10 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.349+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c0afbf0 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.350+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c0afe10 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.351+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f048c018360 con 0x7f049c0ae310 2023-06-10T18:52:37.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.351+0000 7f0498ff9700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 0x7f048806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:37.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.351+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f048c089570 con 0x7f049c0ae310 2023-06-10T18:52:37.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.351+0000 7f049bfff700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 0x7f048806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:37.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.352+0000 7f049bfff700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 0x7f048806d190 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0490003e90 tx=0x7f0490009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:37.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.355+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f048c054460 con 0x7f049c0ae310 2023-06-10T18:52:37.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.581+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f049c06acb0 con 0x7f049c0ae310 2023-06-10T18:52:37.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.581+0000 7f0498ff9700 1 -- 172.21.15.104:0/3347489543 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f049c06acb0 con 0x7f049c0ae310 2023-06-10T18:52:37.587 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:37.587 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:37.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 msgr2=0x7f048806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:37.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 0x7f048806d190 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f0490003e90 tx=0x7f0490009040 comp rx=0 tx=0).stop 2023-06-10T18:52:37.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 msgr2=0x7f049c0b0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:37.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f048c00c7f0 tx=0x7f048c00cbb0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 shutdown_connections 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f048806ace0 0x7f048806d190 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c06e680 0x7f049c151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f049c15c2d0 0x7f049c0c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 --2- 172.21.15.104:0/3347489543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c0ae310 0x7f049c0b0110 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 >> 172.21.15.104:0/3347489543 conn(0x7f049c06b8a0 msgr2=0x7f049c0c9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 shutdown_connections 2023-06-10T18:52:37.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:37.583+0000 7f04a27e3700 1 -- 172.21.15.104:0/3347489543 wait complete. 2023-06-10T18:52:37.591 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:37.601 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:38.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:38.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.719+0000 7f8084d2a700 1 Processor -- start 2023-06-10T18:52:38.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.719+0000 7f8084d2a700 1 -- start start 2023-06-10T18:52:38.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f8084d2a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f808015a420 0x7f80800b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f8084d2a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80800cea60 0x7f808015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f8084d2a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80800b6070 con 0x7f80800b4ff0 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f8084d2a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80800b5460 con 0x7f808015a420 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f8084d2a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80800b5740 con 0x7f80800cea60 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f807e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f808015a420 0x7f80800b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f807e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f808015a420 0x7f80800b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33492/0 (socket says 172.21.15.104:33492) 2023-06-10T18:52:38.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.720+0000 7f807e59c700 1 -- 172.21.15.104:0/734879752 learned_addr learned my addr 172.21.15.104:0/734879752 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807e59c700 1 -- 172.21.15.104:0/734879752 >> v2:172.21.15.114:3300/0 conn(0x7f80800cea60 msgr2=0x7f808015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807e59c700 1 --2- 172.21.15.104:0/734879752 >> v2:172.21.15.114:3300/0 conn(0x7f80800cea60 0x7f808015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807e59c700 1 -- 172.21.15.104:0/734879752 >> v1:172.21.15.114:6789/0 conn(0x7f80800b4ff0 legacy=0x7f80800ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807e59c700 1 -- 172.21.15.104:0/734879752 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80800b5a20 con 0x7f808015a420 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807e59c700 1 --2- 172.21.15.104:0/734879752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f808015a420 0x7f80800b1660 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f80800cad60 tx=0x7f8074009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22fa59de389843b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807d59a700 1 -- 172.21.15.104:0/734879752 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074004230 con 0x7f808015a420 2023-06-10T18:52:38.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807d59a700 1 -- 172.21.15.104:0/734879752 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074004390 con 0x7f808015a420 2023-06-10T18:52:38.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f807d59a700 1 -- 172.21.15.104:0/734879752 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80740036a0 con 0x7f808015a420 2023-06-10T18:52:38.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f8084d2a700 1 -- 172.21.15.104:0/734879752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f808015a420 msgr2=0x7f80800b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.721+0000 7f8084d2a700 1 --2- 172.21.15.104:0/734879752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f808015a420 0x7f80800b1660 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f80800cad60 tx=0x7f8074009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 -- 172.21.15.104:0/734879752 shutdown_connections 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 --2- 172.21.15.104:0/734879752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f808015a420 0x7f80800b1660 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 --2- 172.21.15.104:0/734879752 >> v2:172.21.15.114:3300/0 conn(0x7f80800cea60 0x7f808015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 -- 172.21.15.104:0/734879752 >> 172.21.15.104:0/734879752 conn(0x7f8080089fd0 msgr2=0x7f80800d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 -- 172.21.15.104:0/734879752 shutdown_connections 2023-06-10T18:52:38.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.722+0000 7f8084d2a700 1 -- 172.21.15.104:0/734879752 wait complete. 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.723+0000 7f8084d2a700 1 Processor -- start 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.723+0000 7f8084d2a700 1 -- start start 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 0x7f80800c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808015a420 0x7f80800c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80800c5300 con 0x7f80800b4ff0 2023-06-10T18:52:38.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80800c5550 con 0x7f80800cea60 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f8084d2a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80800c5830 con 0x7f808015a420 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33498/0 (socket says 172.21.15.104:33498) 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 -- 172.21.15.104:0/4212712075 learned_addr learned my addr 172.21.15.104:0/4212712075 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807ed9d700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808015a420 0x7f80800c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807dd9b700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 0x7f80800c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 -- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808015a420 msgr2=0x7f80800c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:38.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808015a420 0x7f80800c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 -- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 msgr2=0x7f80800c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 0x7f80800c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 -- 172.21.15.104:0/4212712075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009550 con 0x7f80800b4ff0 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807dd9b700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 0x7f80800c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f807e59c700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8074009130 tx=0x7f8074004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011630 con 0x7f80800b4ff0 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.724+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074011790 con 0x7f80800b4ff0 2023-06-10T18:52:38.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.725+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80800c5b10 con 0x7f80800b4ff0 2023-06-10T18:52:38.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.725+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011b40 con 0x7f80800b4ff0 2023-06-10T18:52:38.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.725+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80800c60d0 con 0x7f80800b4ff0 2023-06-10T18:52:38.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.725+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8080074050 con 0x7f80800b4ff0 2023-06-10T18:52:38.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.726+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8074011ca0 con 0x7f80800b4ff0 2023-06-10T18:52:38.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.727+0000 7f806effd700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 0x7f806006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:38.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.727+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f807408ac60 con 0x7f80800b4ff0 2023-06-10T18:52:38.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.727+0000 7f807dd9b700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 0x7f806006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:38.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.727+0000 7f807dd9b700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 0x7f806006d190 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f80800cad10 tx=0x7f8070009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:38.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.732+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8074055be0 con 0x7f80800b4ff0 2023-06-10T18:52:38.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:52:38.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:52:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.968+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80800702a0 con 0x7f80800b4ff0 2023-06-10T18:52:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.968+0000 7f806effd700 1 -- 172.21.15.104:0/4212712075 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f8074059200 con 0x7f80800b4ff0 2023-06-10T18:52:38.974 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:38.974 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 msgr2=0x7f806006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 0x7f806006d190 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f80800cad10 tx=0x7f8070009040 comp rx=0 tx=0).stop 2023-06-10T18:52:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 msgr2=0x7f808014fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8074009130 tx=0x7f8074004750 comp rx=0 tx=0).stop 2023-06-10T18:52:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 shutdown_connections 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f808015a420 0x7f80800c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80800b4ff0 0x7f808014fda0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f806006ace0 0x7f806006d190 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 --2- 172.21.15.104:0/4212712075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80800cea60 0x7f80800c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 >> 172.21.15.104:0/4212712075 conn(0x7f8080089fd0 msgr2=0x7f80800b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 shutdown_connections 2023-06-10T18:52:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:38.970+0000 7f8084d2a700 1 -- 172.21.15.104:0/4212712075 wait complete. 2023-06-10T18:52:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:38.988 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:39.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.116+0000 7ff088f6c700 1 Processor -- start 2023-06-10T18:52:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.116+0000 7ff088f6c700 1 -- start start 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff088f6c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0840cad90 0x7ff0840701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff088f6c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0840ccca0 con 0x7ff0840cfa00 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff088f6c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0840cee50 con 0x7ff08415a250 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff088f6c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0840cf130 con 0x7ff0840cad90 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff082d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0840cad90 0x7ff0840701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff08259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff0840cfa00 0x7ff0840cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59566/0 (socket says 172.21.15.104:59566) 2023-06-10T18:52:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.117+0000 7ff08259c700 1 -- 172.21.15.104:0/1076021013 learned_addr learned my addr 172.21.15.104:0/1076021013 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff082d9d700 1 -- 172.21.15.104:0/1076021013 >> v1:172.21.15.114:6789/0 conn(0x7ff08415a250 legacy=0x7ff0840b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff082d9d700 1 -- 172.21.15.104:0/1076021013 >> v1:172.21.15.104:6789/0 conn(0x7ff0840cfa00 legacy=0x7ff0840cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff08159a700 1 -- 172.21.15.104:0/1076021013 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143402644 0 0) 0x7ff0840ccca0 con 0x7ff0840cfa00 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff082d9d700 1 -- 172.21.15.104:0/1076021013 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0840cc930 con 0x7ff0840cad90 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff082d9d700 1 --2- 172.21.15.104:0/1076021013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0840cad90 0x7ff0840701e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009d90 tx=0x7ff07c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3576daed4a31c2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff08159a700 1 -- 172.21.15.104:0/1076021013 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00e070 con 0x7ff0840cad90 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff08159a700 1 -- 172.21.15.104:0/1076021013 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c00bbd0 con 0x7ff0840cad90 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff08159a700 1 -- 172.21.15.104:0/1076021013 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c012430 con 0x7ff0840cad90 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff088f6c700 1 -- 172.21.15.104:0/1076021013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0840cad90 msgr2=0x7ff0840701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.118+0000 7ff088f6c700 1 --2- 172.21.15.104:0/1076021013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0840cad90 0x7ff0840701e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009d90 tx=0x7ff07c00b330 comp rx=0 tx=0).stop 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.119+0000 7ff088f6c700 1 -- 172.21.15.104:0/1076021013 shutdown_connections 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.119+0000 7ff088f6c700 1 --2- 172.21.15.104:0/1076021013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0840cad90 0x7ff0840701e0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.119+0000 7ff088f6c700 1 -- 172.21.15.104:0/1076021013 >> 172.21.15.104:0/1076021013 conn(0x7ff08406e370 msgr2=0x7ff08406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.119+0000 7ff088f6c700 1 -- 172.21.15.104:0/1076021013 shutdown_connections 2023-06-10T18:52:40.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.119+0000 7ff088f6c700 1 -- 172.21.15.104:0/1076021013 wait complete. 2023-06-10T18:52:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.120+0000 7ff088f6c700 1 Processor -- start 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.120+0000 7ff088f6c700 1 -- start start 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0840cad90 0x7ff0840c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0840cfa00 0x7ff0840c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840c4980 con 0x7ff08415a250 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff084143780 con 0x7ff0840cfa00 2023-06-10T18:52:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff088f6c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff084143a60 con 0x7ff0840cad90 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33516/0 (socket says 172.21.15.104:33516) 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 -- 172.21.15.104:0/893445834 learned_addr learned my addr 172.21.15.104:0/893445834 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff08259c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0840cad90 0x7ff0840c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff081d9b700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0840cfa00 0x7ff0840c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 -- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0840cad90 msgr2=0x7ff0840c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0840cad90 0x7ff0840c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 -- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0840cfa00 msgr2=0x7ff0840c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0840cfa00 0x7ff0840c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 -- 172.21.15.104:0/893445834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c009a70 con 0x7ff08415a250 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff082d9d700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7ff07c014040 tx=0x7ff07c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.121+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c01b810 con 0x7ff08415a250 2023-06-10T18:52:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.122+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c020070 con 0x7ff08415a250 2023-06-10T18:52:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.122+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff084143d40 con 0x7ff08415a250 2023-06-10T18:52:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.122+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c01bbb0 con 0x7ff08415a250 2023-06-10T18:52:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.122+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0840ca440 con 0x7ff08415a250 2023-06-10T18:52:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.122+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff084144260 con 0x7ff08415a250 2023-06-10T18:52:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.123+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff07c01bd10 con 0x7ff08415a250 2023-06-10T18:52:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.123+0000 7ff072ffd700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.124+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7ff07c022070 con 0x7ff08415a250 2023-06-10T18:52:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.124+0000 7ff08259c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.124+0000 7ff08259c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff078000f60 tx=0x7ff078004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.128+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff084144260 con 0x7ff08415a250 2023-06-10T18:52:40.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.354+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff084144260 con 0x7ff08415a250 2023-06-10T18:52:40.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.354+0000 7ff072ffd700 1 -- 172.21.15.104:0/893445834 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7ff084144260 con 0x7ff08415a250 2023-06-10T18:52:40.360 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:40.360 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 msgr2=0x7ff06406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff078000f60 tx=0x7ff078004a50 comp rx=0 tx=0).stop 2023-06-10T18:52:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 msgr2=0x7ff0840c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7ff07c014040 tx=0x7ff07c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 shutdown_connections 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08415a250 0x7ff0840c9f00 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0840cad90 0x7ff0840c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 --2- 172.21.15.104:0/893445834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0840cfa00 0x7ff0840c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 >> 172.21.15.104:0/893445834 conn(0x7ff08406e370 msgr2=0x7ff0840b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 shutdown_connections 2023-06-10T18:52:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:40.356+0000 7ff088f6c700 1 -- 172.21.15.104:0/893445834 wait complete. 2023-06-10T18:52:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:40.374 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:41.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:41.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.501+0000 7fece5515700 1 Processor -- start 2023-06-10T18:52:41.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.501+0000 7fece5515700 1 -- start start 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.501+0000 7fece5515700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece00d0c00 0x7fece006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.501+0000 7fece5515700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fece006b820 0x7fece00cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fece5515700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece006bfd0 con 0x7fece00b1e30 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fece5515700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece006c2b0 con 0x7fece00d0c00 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fece5515700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece00ccf70 con 0x7fece006b820 2023-06-10T18:52:41.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece00d0c00 0x7fece006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece00d0c00 0x7fece006b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33522/0 (socket says 172.21.15.104:33522) 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 -- 172.21.15.104:0/3546351563 learned_addr learned my addr 172.21.15.104:0/3546351563 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecdf7fe700 1 --2- 172.21.15.104:0/3546351563 >> v2:172.21.15.114:3300/0 conn(0x7fece006b820 0x7fece00cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 -- 172.21.15.104:0/3546351563 >> v2:172.21.15.114:3300/0 conn(0x7fece006b820 msgr2=0x7fece00cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 --2- 172.21.15.104:0/3546351563 >> v2:172.21.15.114:3300/0 conn(0x7fece006b820 0x7fece00cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 -- 172.21.15.104:0/3546351563 >> v1:172.21.15.114:6789/0 conn(0x7fece00b1e30 legacy=0x7fece00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 -- 172.21.15.104:0/3546351563 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece00cd250 con 0x7fece00d0c00 2023-06-10T18:52:41.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecde7fc700 1 --2- 172.21.15.104:0/3546351563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece00d0c00 0x7fece006b2e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fece008b0f0 tx=0x7fecd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf3bc8183ee3303 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:41.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecddffb700 1 -- 172.21.15.104:0/3546351563 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd400b870 con 0x7fece00d0c00 2023-06-10T18:52:41.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecddffb700 1 -- 172.21.15.104:0/3546351563 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd400b9d0 con 0x7fece00d0c00 2023-06-10T18:52:41.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.502+0000 7fecddffb700 1 -- 172.21.15.104:0/3546351563 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd400bd60 con 0x7fece00d0c00 2023-06-10T18:52:41.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 -- 172.21.15.104:0/3546351563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece00d0c00 msgr2=0x7fece006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 --2- 172.21.15.104:0/3546351563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece00d0c00 0x7fece006b2e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fece008b0f0 tx=0x7fecd4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 -- 172.21.15.104:0/3546351563 shutdown_connections 2023-06-10T18:52:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 --2- 172.21.15.104:0/3546351563 >> v2:172.21.15.114:3300/0 conn(0x7fece006b820 0x7fece00cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 --2- 172.21.15.104:0/3546351563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece00d0c00 0x7fece006b2e0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 -- 172.21.15.104:0/3546351563 >> 172.21.15.104:0/3546351563 conn(0x7fece0071ec0 msgr2=0x7fece009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 -- 172.21.15.104:0/3546351563 shutdown_connections 2023-06-10T18:52:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.503+0000 7fece5515700 1 -- 172.21.15.104:0/3546351563 wait complete. 2023-06-10T18:52:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.504+0000 7fece5515700 1 Processor -- start 2023-06-10T18:52:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.504+0000 7fece5515700 1 -- start start 2023-06-10T18:52:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 0x7fece00b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece00d0c00 0x7fece00b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece015a8a0 con 0x7fece006b820 2023-06-10T18:52:41.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b5fa0 con 0x7fece00d0c00 2023-06-10T18:52:41.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fece5515700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b6280 con 0x7fece00b1e30 2023-06-10T18:52:41.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33526/0 (socket says 172.21.15.104:33526) 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 -- 172.21.15.104:0/2240673374 learned_addr learned my addr 172.21.15.104:0/2240673374 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecde7fc700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 0x7fece00b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdf7fe700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece00d0c00 0x7fece00b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 -- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 msgr2=0x7fece00b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 0x7fece00b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 -- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece00d0c00 msgr2=0x7fece00b7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.505+0000 7fecdeffd700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece00d0c00 0x7fece00b7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecdeffd700 1 -- 172.21.15.104:0/2240673374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4009550 con 0x7fece006b820 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecde7fc700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 0x7fece00b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecdeffd700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fecd000c4e0 tx=0x7fecd000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00052d0 con 0x7fece006b820 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0005430 con 0x7fece006b820 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece00b6560 con 0x7fece006b820 2023-06-10T18:52:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00038c0 con 0x7fece006b820 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.506+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece00b6b20 con 0x7fece006b820 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.507+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece00d1860 con 0x7fece006b820 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.507+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fecd0003a20 con 0x7fece006b820 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.507+0000 7fecc77fe700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 0x7feccc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.508+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fecd0088b70 con 0x7fece006b820 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.508+0000 7fecde7fc700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 0x7feccc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:41.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.508+0000 7fecde7fc700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 0x7feccc06d190 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fecd4014040 tx=0x7fecd4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:41.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.512+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fecd0053a50 con 0x7fece006b820 2023-06-10T18:52:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.741+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fece0074050 con 0x7fece006b820 2023-06-10T18:52:41.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.742+0000 7fecc77fe700 1 -- 172.21.15.104:0/2240673374 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fecd0057070 con 0x7fece006b820 2023-06-10T18:52:41.748 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:41.748 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:41.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 msgr2=0x7feccc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 0x7feccc06d190 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fecd4014040 tx=0x7fecd4008030 comp rx=0 tx=0).stop 2023-06-10T18:52:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 msgr2=0x7fece015a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fecd000c4e0 tx=0x7fecd000c8a0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 shutdown_connections 2023-06-10T18:52:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feccc06ace0 0x7feccc06d190 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece00d0c00 0x7fece00b7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece006b820 0x7fece015a360 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 --2- 172.21.15.104:0/2240673374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece00b1e30 0x7fece00b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 >> 172.21.15.104:0/2240673374 conn(0x7fece0071ec0 msgr2=0x7fece0072940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 shutdown_connections 2023-06-10T18:52:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:41.744+0000 7fece5515700 1 -- 172.21.15.104:0/2240673374 wait complete. 2023-06-10T18:52:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:41.761 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:42.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:42.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.890+0000 7fc076ac6700 1 Processor -- start 2023-06-10T18:52:42.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.890+0000 7fc076ac6700 1 -- start start 2023-06-10T18:52:42.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc076ac6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc07015c040 0x7fc0700ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc076ac6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700ae6f0 con 0x7fc0700ae280 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc076ac6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700ce2c0 con 0x7fc070071130 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc076ac6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0700cf750 con 0x7fc07015c040 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc06ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc0700ae280 0x7fc070070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59594/0 (socket says 172.21.15.104:59594) 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc06ffff700 1 -- 172.21.15.104:0/4004236536 learned_addr learned my addr 172.21.15.104:0/4004236536 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:42.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331480273 0 0) 0x7fc0700ae6f0 con 0x7fc0700ae280 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc074862700 1 --2- 172.21.15.104:0/4004236536 >> v2:172.21.15.114:3300/0 conn(0x7fc07015c040 0x7fc0700ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.891+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0600035b0 con 0x7fc0700ae280 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4054433260 0 0) 0x7fc0600035b0 con 0x7fc0700ae280 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 >> v2:172.21.15.114:3300/0 conn(0x7fc07015c040 msgr2=0x7fc0700ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 --2- 172.21.15.104:0/4004236536 >> v2:172.21.15.114:3300/0 conn(0x7fc07015c040 0x7fc0700ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 >> v1:172.21.15.114:6789/0 conn(0x7fc070071130 legacy=0x7fc0700ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0700cfa30 con 0x7fc0700ae280 2023-06-10T18:52:42.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc05c001670 con 0x7fc0700ae280 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 >> v1:172.21.15.104:6789/0 conn(0x7fc0700ae280 legacy=0x7fc070070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060003a00 0x7fc060003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc060004850 0x7fc060004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc06000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700cf750 con 0x7fc060005200 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06000c110 con 0x7fc060003a00 2023-06-10T18:52:42.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06000c2d0 con 0x7fc060004850 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc06000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc074862700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060003a00 0x7fc060003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc06ffff700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc060004850 0x7fc060004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 -- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc060004850 msgr2=0x7fc060004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc060004850 0x7fc060004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 -- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060003a00 msgr2=0x7fc060003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060003a00 0x7fc060003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 -- 172.21.15.104:0/4004236536 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600035b0 con 0x7fc060005200 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 -- 172.21.15.104:0/4004236536 reap_dead start 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.892+0000 7fc075063700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc06000ba10 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc06800c160 tx=0x7fc060005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8052450d314bc5d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068001db0 con 0x7fc060005200 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800ccc0 con 0x7fc060005200 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc06f7fe700 1 -- 172.21.15.104:0/4004236536 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068003c40 con 0x7fc060005200 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 -- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 msgr2=0x7fc06000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:42.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc06000ba10 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc06800c160 tx=0x7fc060005880 comp rx=0 tx=0).stop 2023-06-10T18:52:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 -- 172.21.15.104:0/4004236536 shutdown_connections 2023-06-10T18:52:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 --2- 172.21.15.104:0/4004236536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc06000ba10 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 -- 172.21.15.104:0/4004236536 >> 172.21.15.104:0/4004236536 conn(0x7fc0700b1610 msgr2=0x7fc0700ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 -- 172.21.15.104:0/4004236536 shutdown_connections 2023-06-10T18:52:42.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.893+0000 7fc076ac6700 1 -- 172.21.15.104:0/4004236536 wait complete. 2023-06-10T18:52:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.894+0000 7fc076ac6700 1 Processor -- start 2023-06-10T18:52:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.894+0000 7fc076ac6700 1 -- start start 2023-06-10T18:52:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 0x7fc0700b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc07015c880 0x7fc07015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700b2610 con 0x7fc060005200 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07015e150 con 0x7fc07015c880 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc076ac6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07015d5d0 con 0x7fc0700b1c80 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc074862700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc074862700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33546/0 (socket says 172.21.15.104:33546) 2023-06-10T18:52:42.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc074862700 1 -- 172.21.15.104:0/2997397197 learned_addr learned my addr 172.21.15.104:0/2997397197 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.895+0000 7fc075063700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc07015c880 0x7fc07015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc06ffff700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 0x7fc0700b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 -- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 msgr2=0x7fc0700b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 0x7fc0700b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 -- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc07015c880 msgr2=0x7fc07015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc07015c880 0x7fc07015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:42.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 -- 172.21.15.104:0/2997397197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07015d8b0 con 0x7fc060005200 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc06ffff700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 0x7fc0700b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc074862700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc0640119a0 tx=0x7fc064011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06400f940 con 0x7fc060005200 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06400faa0 con 0x7fc060005200 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06400fe30 con 0x7fc060005200 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07015db90 con 0x7fc060005200 2023-06-10T18:52:42.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.896+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07014a6b0 con 0x7fc060005200 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.897+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc070160660 con 0x7fc060005200 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.897+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc064014440 con 0x7fc060005200 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.898+0000 7fc06d7fa700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 0x7fc05406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.898+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fc064089c10 con 0x7fc060005200 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.898+0000 7fc06ffff700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 0x7fc05406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:42.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.898+0000 7fc06ffff700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 0x7fc05406d190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc05c007d10 tx=0x7fc05c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:42.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:42.902+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc064054c40 con 0x7fc060005200 2023-06-10T18:52:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.130+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc070074050 con 0x7fc060005200 2023-06-10T18:52:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.130+0000 7fc06d7fa700 1 -- 172.21.15.104:0/2997397197 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fc064058260 con 0x7fc060005200 2023-06-10T18:52:43.136 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:43.136 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 msgr2=0x7fc05406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 0x7fc05406d190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc05c007d10 tx=0x7fc05c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 msgr2=0x7fc07014a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc0640119a0 tx=0x7fc064011cb0 comp rx=0 tx=0).stop 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 shutdown_connections 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc05406ace0 0x7fc05406d190 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc07015c880 0x7fc07015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc060005200 0x7fc07014a170 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 --2- 172.21.15.104:0/2997397197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc0700b1c80 0x7fc0700b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 >> 172.21.15.104:0/2997397197 conn(0x7fc0700b1610 msgr2=0x7fc070069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 shutdown_connections 2023-06-10T18:52:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:43.132+0000 7fc076ac6700 1 -- 172.21.15.104:0/2997397197 wait complete. 2023-06-10T18:52:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:43.150 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:44.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:44.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.278+0000 7fa3b2038700 1 Processor -- start 2023-06-10T18:52:44.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.278+0000 7fa3b2038700 1 -- start start 2023-06-10T18:52:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.278+0000 7fa3b2038700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 0x7fa3ac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3b2038700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3b2038700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac06d290 con 0x7fa3ac06e4b0 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3b2038700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac071640 con 0x7fa3ac069e70 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3b2038700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0aef00 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33558/0 (socket says 172.21.15.104:33558) 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 -- 172.21.15.104:0/2545545602 learned_addr learned my addr 172.21.15.104:0/2545545602 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3ab7fe700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 0x7fa3ac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 -- 172.21.15.104:0/2545545602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 msgr2=0x7fa3ac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 0x7fa3ac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 -- 172.21.15.104:0/2545545602 >> v1:172.21.15.114:6789/0 conn(0x7fa3ac06e4b0 legacy=0x7fa3ac0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 -- 172.21.15.104:0/2545545602 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0ca2d0 con 0x7fa3ac069e70 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3ab7fe700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 0x7fa3ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3abfff700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e077b6f4a53db88f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3aa7fc700 1 -- 172.21.15.104:0/2545545602 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400e070 con 0x7fa3ac069e70 2023-06-10T18:52:44.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3aa7fc700 1 -- 172.21.15.104:0/2545545602 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a400b9c0 con 0x7fa3ac069e70 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.279+0000 7fa3aa7fc700 1 -- 172.21.15.104:0/2545545602 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400bd70 con 0x7fa3ac069e70 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 -- 172.21.15.104:0/2545545602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac069e70 msgr2=0x7fa3ac0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 -- 172.21.15.104:0/2545545602 shutdown_connections 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0ce9f0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 --2- 172.21.15.104:0/2545545602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e0e0 0x7fa3ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 -- 172.21.15.104:0/2545545602 >> 172.21.15.104:0/2545545602 conn(0x7fa3ac06b8a0 msgr2=0x7fa3ac0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:44.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 -- 172.21.15.104:0/2545545602 shutdown_connections 2023-06-10T18:52:44.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.280+0000 7fa3b2038700 1 -- 172.21.15.104:0/2545545602 wait complete. 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.281+0000 7fa3b2038700 1 Processor -- start 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.281+0000 7fa3b2038700 1 -- start start 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e4b0 0x7fa3ac099880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac09b680 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0b48e0 con 0x7fa3ac069e70 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3b2038700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0b4bc0 con 0x7fa3ac06e4b0 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33562/0 (socket says 172.21.15.104:33562) 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 -- 172.21.15.104:0/392743215 learned_addr learned my addr 172.21.15.104:0/392743215 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3abfff700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e4b0 0x7fa3ac099880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3ab7fe700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 -- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e4b0 msgr2=0x7fa3ac099880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e4b0 0x7fa3ac099880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.282+0000 7fa3aaffd700 1 -- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ac069e70 msgr2=0x7fa3ac0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa3aaffd700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa3aaffd700 1 -- 172.21.15.104:0/392743215 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4009b30 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa3aaffd700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c4e0 tx=0x7fa3a000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00052d0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0005430 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac0b4ea0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00038c0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.283+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac1f1640 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.284+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac09b830 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.284+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3a0003a20 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.285+0000 7fa393fff700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 0x7fa39806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:44.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.285+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fa3a00889d0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.285+0000 7fa3ab7fe700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 0x7fa39806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:44.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.285+0000 7fa3ab7fe700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 0x7fa39806d190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c960 tx=0x7fa39c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:44.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.289+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3ac09b830 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.528+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3ac09b3f0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.529+0000 7fa393fff700 1 -- 172.21.15.104:0/392743215 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fa3a00538b0 con 0x7fa3ac06e0e0 2023-06-10T18:52:44.535 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:44.535 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:44.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 msgr2=0x7fa39806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 0x7fa39806d190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c960 tx=0x7fa39c008040 comp rx=0 tx=0).stop 2023-06-10T18:52:44.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 msgr2=0x7fa3ac099340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c4e0 tx=0x7fa3a000c7f0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 shutdown_connections 2023-06-10T18:52:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa39806ace0 0x7fa39806d190 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ac069e70 0x7fa3ac0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ac06e4b0 0x7fa3ac099880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 --2- 172.21.15.104:0/392743215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ac06e0e0 0x7fa3ac099340 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:44.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 >> 172.21.15.104:0/392743215 conn(0x7fa3ac06b8a0 msgr2=0x7fa3ac0cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:44.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 shutdown_connections 2023-06-10T18:52:44.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:44.531+0000 7fa3b2038700 1 -- 172.21.15.104:0/392743215 wait complete. 2023-06-10T18:52:44.540 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:44.549 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:45.549 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:45.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.677+0000 7f42c097a700 1 Processor -- start 2023-06-10T18:52:45.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.677+0000 7f42c097a700 1 -- start start 2023-06-10T18:52:45.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42c097a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 0x7f42bc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42c097a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42c097a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc06d290 con 0x7f42bc06e0e0 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42c097a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc071640 con 0x7f42bc06e4b0 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42c097a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0aef00 con 0x7f42bc069e70 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42b9d9b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 0x7f42bc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42ba59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f42bc06e0e0 0x7f42bc0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37242/0 (socket says 172.21.15.104:37242) 2023-06-10T18:52:45.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:52956/0 (socket says 172.21.15.104:52956) 2023-06-10T18:52:45.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42ba59c700 1 -- 172.21.15.104:0/240781344 learned_addr learned my addr 172.21.15.104:0/240781344 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:45.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 -- 172.21.15.104:0/240781344 >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 msgr2=0x7f42bc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:45.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 --2- 172.21.15.104:0/240781344 >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 0x7f42bc06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 -- 172.21.15.104:0/240781344 >> v1:172.21.15.114:6789/0 conn(0x7f42bc06e0e0 legacy=0x7f42bc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42bad9d700 1 -- 172.21.15.104:0/240781344 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0ca2d0 con 0x7f42bc069e70 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.678+0000 7f42b9d9b700 1 --2- 172.21.15.104:0/240781344 >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 0x7f42bc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42bad9d700 1 --2- 172.21.15.104:0/240781344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009d90 tx=0x7f42b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3638b63cb1e42b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42b959a700 1 -- 172.21.15.104:0/240781344 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400e070 con 0x7f42bc069e70 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42b959a700 1 -- 172.21.15.104:0/240781344 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400bb20 con 0x7f42bc069e70 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42b959a700 1 -- 172.21.15.104:0/240781344 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4012430 con 0x7f42bc069e70 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 -- 172.21.15.104:0/240781344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 msgr2=0x7f42bc0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 --2- 172.21.15.104:0/240781344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009d90 tx=0x7f42b400b330 comp rx=0 tx=0).stop 2023-06-10T18:52:45.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 -- 172.21.15.104:0/240781344 shutdown_connections 2023-06-10T18:52:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 --2- 172.21.15.104:0/240781344 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc069e70 0x7f42bc0ce9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 --2- 172.21.15.104:0/240781344 >> v2:172.21.15.114:3300/0 conn(0x7f42bc06e4b0 0x7f42bc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 -- 172.21.15.104:0/240781344 >> 172.21.15.104:0/240781344 conn(0x7f42bc06b8a0 msgr2=0x7f42bc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 -- 172.21.15.104:0/240781344 shutdown_connections 2023-06-10T18:52:45.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.679+0000 7f42c097a700 1 -- 172.21.15.104:0/240781344 wait complete. 2023-06-10T18:52:45.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.681+0000 7f42c097a700 1 Processor -- start 2023-06-10T18:52:45.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.681+0000 7f42c097a700 1 -- start start 2023-06-10T18:52:45.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 0x7f42bc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 0x7f42bc0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0af930 con 0x7f42bc06e0e0 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc156760 con 0x7f42bc069e70 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42c097a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0af040 con 0x7f42bc06e4b0 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42b9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42b9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33578/0 (socket says 172.21.15.104:33578) 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42b9d9b700 1 -- 172.21.15.104:0/3222218662 learned_addr learned my addr 172.21.15.104:0/3222218662 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.682+0000 7f42bad9d700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 0x7f42bc0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42ba59c700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 0x7f42bc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 -- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 msgr2=0x7f42bc0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 0x7f42bc0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 -- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 msgr2=0x7f42bc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 0x7f42bc1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 -- 172.21.15.104:0/3222218662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b00094a0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42bad9d700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 0x7f42bc0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42ba59c700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 0x7f42bc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42b9d9b700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ebe0 tx=0x7f42b0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b000c910 con 0x7f42bc06e0e0 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b000ca70 con 0x7f42bc06e0e0 2023-06-10T18:52:45.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b4009a70 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b000cde0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.683+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc0af2e0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.684+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc0afaa0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.684+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f42b0013470 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.685+0000 7f42aaffd700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 0x7f429c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.685+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f42b0089850 con 0x7f42bc06e0e0 2023-06-10T18:52:45.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.685+0000 7f42ba59c700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 0x7f429c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:45.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.685+0000 7f42ba59c700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 0x7f429c06d190 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f42ac005960 tx=0x7f42ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:45.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.689+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f42b00547d0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.916+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42bc074050 con 0x7f42bc06e0e0 2023-06-10T18:52:45.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.916+0000 7f42aaffd700 1 -- 172.21.15.104:0/3222218662 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f42b0057df0 con 0x7f42bc06e0e0 2023-06-10T18:52:45.922 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:45.922 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 msgr2=0x7f429c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 0x7f429c06d190 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f42ac005960 tx=0x7f42ac009040 comp rx=0 tx=0).stop 2023-06-10T18:52:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 msgr2=0x7f42bc1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ebe0 tx=0x7f42b0009ec0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 shutdown_connections 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f429c06ace0 0x7f429c06d190 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc069e70 0x7f42bc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42bc06e4b0 0x7f42bc0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 --2- 172.21.15.104:0/3222218662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42bc06e0e0 0x7f42bc1559e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 >> 172.21.15.104:0/3222218662 conn(0x7f42bc06b8a0 msgr2=0x7f42bc0c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 shutdown_connections 2023-06-10T18:52:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:45.918+0000 7f42c097a700 1 -- 172.21.15.104:0/3222218662 wait complete. 2023-06-10T18:52:45.926 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:45.936 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:46.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.061+0000 7f8d1bb7d700 1 Processor -- start 2023-06-10T18:52:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 -- start start 2023-06-10T18:52:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d140cf030 0x7f8d140ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d14069e70 0x7f8d1406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1406dcb0 con 0x7f8d1415c500 2023-06-10T18:52:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1406e850 con 0x7f8d140cf030 2023-06-10T18:52:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.062+0000 7f8d1bb7d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1406a830 con 0x7f8d14069e70 2023-06-10T18:52:47.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d140cf030 0x7f8d140ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:47.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d140cf030 0x7f8d140ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33584/0 (socket says 172.21.15.104:33584) 2023-06-10T18:52:47.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 -- 172.21.15.104:0/3123107848 learned_addr learned my addr 172.21.15.104:0/3123107848 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:47.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 -- 172.21.15.104:0/3123107848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d14069e70 msgr2=0x7f8d1406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 --2- 172.21.15.104:0/3123107848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d14069e70 0x7f8d1406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 -- 172.21.15.104:0/3123107848 >> v1:172.21.15.114:6789/0 conn(0x7f8d1415c500 legacy=0x7f8d1406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:47.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 -- 172.21.15.104:0/3123107848 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140cf460 con 0x7f8d140cf030 2023-06-10T18:52:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d19118700 1 --2- 172.21.15.104:0/3123107848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d140cf030 0x7f8d140ce040 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8d100098f0 tx=0x7f8d10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280f2372d38482fc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d18917700 1 -- 172.21.15.104:0/3123107848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10004230 con 0x7f8d140cf030 2023-06-10T18:52:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d18917700 1 -- 172.21.15.104:0/3123107848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10004390 con 0x7f8d140cf030 2023-06-10T18:52:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.063+0000 7f8d18917700 1 -- 172.21.15.104:0/3123107848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d100036a0 con 0x7f8d140cf030 2023-06-10T18:52:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3123107848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d140cf030 msgr2=0x7f8d140ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3123107848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d140cf030 0x7f8d140ce040 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f8d100098f0 tx=0x7f8d10009c00 comp rx=0 tx=0).stop 2023-06-10T18:52:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3123107848 shutdown_connections 2023-06-10T18:52:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3123107848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d14069e70 0x7f8d1406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3123107848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d140cf030 0x7f8d140ce040 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3123107848 >> 172.21.15.104:0/3123107848 conn(0x7f8d1406b8a0 msgr2=0x7f8d140cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3123107848 shutdown_connections 2023-06-10T18:52:47.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.064+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3123107848 wait complete. 2023-06-10T18:52:47.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.065+0000 7f8d1bb7d700 1 Processor -- start 2023-06-10T18:52:47.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.065+0000 7f8d1bb7d700 1 -- start start 2023-06-10T18:52:47.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d140cf030 0x7f8d141e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 0x7f8d14157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14157800 con 0x7f8d14069e70 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14156b80 con 0x7f8d1415c500 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14156e60 con 0x7f8d140cf030 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33588/0 (socket says 172.21.15.104:33588) 2023-06-10T18:52:47.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 -- 172.21.15.104:0/3159079216 learned_addr learned my addr 172.21.15.104:0/3159079216 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19118700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d140cf030 0x7f8d141e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1a11a700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 0x7f8d14157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 -- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d140cf030 msgr2=0x7f8d141e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d140cf030 0x7f8d141e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 -- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 msgr2=0x7f8d14157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 0x7f8d14157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 -- 172.21.15.104:0/3159079216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10009550 con 0x7f8d14069e70 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1a11a700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 0x7f8d14157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d19919700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800b450 tx=0x7f8d0800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d080059b0 con 0x7f8d14069e70 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08005b30 con 0x7f8d14069e70 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d14157140 con 0x7f8d14069e70 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.066+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08011630 con 0x7f8d14069e70 2023-06-10T18:52:47.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.067+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d140b5290 con 0x7f8d14069e70 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.067+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d14074050 con 0x7f8d14069e70 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.068+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d08011790 con 0x7f8d14069e70 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.068+0000 7f8d067fc700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 0x7f8cf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.068+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f8d08088e70 con 0x7f8d14069e70 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.069+0000 7f8d19118700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 0x7f8cf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.069+0000 7f8d19118700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 0x7f8cf806d190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009840 tx=0x7f8d10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:47.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.073+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d08053df0 con 0x7f8d14069e70 2023-06-10T18:52:47.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.300+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d140b1c30 con 0x7f8d14069e70 2023-06-10T18:52:47.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.300+0000 7f8d067fc700 1 -- 172.21.15.104:0/3159079216 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f8d08015090 con 0x7f8d14069e70 2023-06-10T18:52:47.306 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:47.306 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 msgr2=0x7f8cf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 0x7f8cf806d190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009840 tx=0x7f8d10008030 comp rx=0 tx=0).stop 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 msgr2=0x7f8d140b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800b450 tx=0x7f8d0800b810 comp rx=0 tx=0).stop 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 shutdown_connections 2023-06-10T18:52:47.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8cf806ace0 0x7f8cf806d190 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d1415c500 0x7f8d14157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d14069e70 0x7f8d140b4d50 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 --2- 172.21.15.104:0/3159079216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d140cf030 0x7f8d141e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 >> 172.21.15.104:0/3159079216 conn(0x7f8d1406b8a0 msgr2=0x7f8d1415ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 shutdown_connections 2023-06-10T18:52:47.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:47.302+0000 7f8d1bb7d700 1 -- 172.21.15.104:0/3159079216 wait complete. 2023-06-10T18:52:47.310 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:47.319 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:48.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:48.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.444+0000 7f661a4e8700 1 Processor -- start 2023-06-10T18:52:48.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.444+0000 7f661a4e8700 1 -- start start 2023-06-10T18:52:48.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f661a4e8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f66140ada30 0x7f66140ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f661a4e8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f661a4e8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66140aed70 con 0x7f66140ade00 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f661a4e8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66140cf430 con 0x7f66140ada30 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f661a4e8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66140cee70 con 0x7f66140ae930 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f66140ada30 0x7f66140ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f66140ada30 0x7f66140ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33596/0 (socket says 172.21.15.104:33596) 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 -- 172.21.15.104:0/1930866448 learned_addr learned my addr 172.21.15.104:0/1930866448 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6618a85700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 -- 172.21.15.104:0/1930866448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 msgr2=0x7f66140ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 -- 172.21.15.104:0/1930866448 >> v1:172.21.15.114:6789/0 conn(0x7f66140ade00 legacy=0x7f66140ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 -- 172.21.15.104:0/1930866448 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66140cf150 con 0x7f66140ada30 2023-06-10T18:52:48.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6613fff700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140ada30 0x7f66140ae3f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6604006fd0 tx=0x7f6604009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b245a5292bd937 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6612ffd700 1 -- 172.21.15.104:0/1930866448 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604005e30 con 0x7f66140ada30 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6612ffd700 1 -- 172.21.15.104:0/1930866448 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6604004030 con 0x7f66140ada30 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.445+0000 7f6612ffd700 1 -- 172.21.15.104:0/1930866448 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66040043e0 con 0x7f66140ada30 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 -- 172.21.15.104:0/1930866448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140ada30 msgr2=0x7f66140ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140ada30 0x7f66140ae3f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f6604006fd0 tx=0x7f6604009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 -- 172.21.15.104:0/1930866448 shutdown_connections 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140ada30 0x7f66140ae3f0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 --2- 172.21.15.104:0/1930866448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 -- 172.21.15.104:0/1930866448 >> 172.21.15.104:0/1930866448 conn(0x7f661406b8a0 msgr2=0x7f66140cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 -- 172.21.15.104:0/1930866448 shutdown_connections 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.446+0000 7f661a4e8700 1 -- 172.21.15.104:0/1930866448 wait complete. 2023-06-10T18:52:48.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.447+0000 7f661a4e8700 1 Processor -- start 2023-06-10T18:52:48.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.447+0000 7f661a4e8700 1 -- start start 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66140ada30 0x7f661414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661414e170 con 0x7f66140b1a00 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6614099340 con 0x7f66140ada30 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f661a4e8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6614099590 con 0x7f66140ae930 2023-06-10T18:52:48.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f6618a85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f6618a85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33606/0 (socket says 172.21.15.104:33606) 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f6618a85700 1 -- 172.21.15.104:0/2721682369 learned_addr learned my addr 172.21.15.104:0/2721682369 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f6613fff700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66140ada30 0x7f661414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.448+0000 7f66137fe700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 -- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 msgr2=0x7f66140aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 -- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66140ada30 msgr2=0x7f661414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66140ada30 0x7f661414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 -- 172.21.15.104:0/2721682369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604009500 con 0x7f66140b1a00 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6618a85700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f660c007e40 tx=0x7f660c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c00eea0 con 0x7f66140b1a00 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c014420 con 0x7f66140b1a00 2023-06-10T18:52:48.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66141e4040 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c0147b0 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.449+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66141e4600 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.450+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6614074050 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.451+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f660c014910 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.451+0000 7f6610ff9700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 0x7f660006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.451+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f660c089e40 con 0x7f66140b1a00 2023-06-10T18:52:48.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.451+0000 7f6613fff700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 0x7f660006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:48.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.452+0000 7f6613fff700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 0x7f660006d190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6604000c00 tx=0x7f6604008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:48.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.455+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f660c0587c0 con 0x7f66140b1a00 2023-06-10T18:52:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.682+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66140b2630 con 0x7f66140b1a00 2023-06-10T18:52:48.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.683+0000 7f6610ff9700 1 -- 172.21.15.104:0/2721682369 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f660c058350 con 0x7f66140b1a00 2023-06-10T18:52:48.689 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:48.689 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:48.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 msgr2=0x7f660006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 0x7f660006d190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6604000c00 tx=0x7f6604008070 comp rx=0 tx=0).stop 2023-06-10T18:52:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 msgr2=0x7f66140b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:48.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f660c007e40 tx=0x7f660c00e3c0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 shutdown_connections 2023-06-10T18:52:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f660006ace0 0x7f660006d190 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66140ada30 0x7f661414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66140b1a00 0x7f66140b0490 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 --2- 172.21.15.104:0/2721682369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66140ae930 0x7f66140aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 >> 172.21.15.104:0/2721682369 conn(0x7f661406b8a0 msgr2=0x7f66140b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 shutdown_connections 2023-06-10T18:52:48.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:48.685+0000 7f661a4e8700 1 -- 172.21.15.104:0/2721682369 wait complete. 2023-06-10T18:52:48.695 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:48.702 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:49.704 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:49.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.830+0000 7fcc8f10b700 1 Processor -- start 2023-06-10T18:52:49.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.831+0000 7fcc8f10b700 1 -- start start 2023-06-10T18:52:49.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.831+0000 7fcc8f10b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc8806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.831+0000 7fcc8f10b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8806d290 con 0x7fcc88069e70 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.831+0000 7fcc8f10b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880ca2d0 con 0x7fcc8806e0e0 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.831+0000 7fcc8f10b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88071640 con 0x7fcc8806e4b0 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc8d6a8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcc88069e70 0x7fcc880ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59674/0 (socket says 172.21.15.104:59674) 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/2827664614 learned_addr learned my addr 172.21.15.104:0/2827664614 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc87fff700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc8806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270322267 0 0) 0x7fcc8806d290 con 0x7fcc88069e70 2023-06-10T18:52:49.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc780035b0 con 0x7fcc88069e70 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881293747 0 0) 0x7fcc780035b0 con 0x7fcc88069e70 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 msgr2=0x7fcc8806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc8806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 >> v1:172.21.15.114:6789/0 conn(0x7fcc8806e0e0 legacy=0x7fcc880c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8806d4c0 con 0x7fcc88069e70 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcc800019f0 con 0x7fcc88069e70 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 >> v1:172.21.15.104:6789/0 conn(0x7fcc88069e70 legacy=0x7fcc880ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:49.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78003a30 0x7fcc78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78004d90 0x7fcc78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88071640 con 0x7fcc78004d90 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7800bb20 con 0x7fcc78003a30 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7800bce0 con 0x7fcc8806e4b0 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78004d90 0x7fcc78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc8cea7700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78003a30 0x7fcc78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.832+0000 7fcc87fff700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 msgr2=0x7fcc78004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc8806e4b0 0x7fcc78004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78003a30 msgr2=0x7fcc78003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78003a30 0x7fcc78003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/2827664614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780035b0 con 0x7fcc78004d90 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78004d90 0x7fcc78005200 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000a830 tx=0x7fcc78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b78c7163ecb746e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:49.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80012600 con 0x7fcc78004d90 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80012780 con 0x7fcc78004d90 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc877fe700 1 -- 172.21.15.104:0/2827664614 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80012b30 con 0x7fcc78004d90 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8f10b700 1 -- 172.21.15.104:0/2827664614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78004d90 msgr2=0x7fcc78005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/2827664614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc78004d90 0x7fcc78005200 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000a830 tx=0x7fcc78005880 comp rx=0 tx=0).stop 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/2827664614 reap_dead start 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8f10b700 1 -- 172.21.15.104:0/2827664614 shutdown_connections 2023-06-10T18:52:49.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.833+0000 7fcc8f10b700 1 -- 172.21.15.104:0/2827664614 >> 172.21.15.104:0/2827664614 conn(0x7fcc8806b8a0 msgr2=0x7fcc880cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:49.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.834+0000 7fcc8f10b700 1 -- 172.21.15.104:0/2827664614 shutdown_connections 2023-06-10T18:52:49.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.834+0000 7fcc8f10b700 1 -- 172.21.15.104:0/2827664614 wait complete. 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.835+0000 7fcc8f10b700 1 Processor -- start 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.835+0000 7fcc8f10b700 1 -- start start 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78004d90 0x7fcc880b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc880b3040 0x7fcc880aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b0250 con 0x7fcc88154f60 2023-06-10T18:52:49.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88155940 con 0x7fcc78004d90 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880af640 con 0x7fcc880b3040 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33626/0 (socket says 172.21.15.104:33626) 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/280517470 learned_addr learned my addr 172.21.15.104:0/280517470 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8cea7700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78004d90 0x7fcc880b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc87fff700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc880b3040 0x7fcc880aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc880b3040 msgr2=0x7fcc880aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc880b3040 0x7fcc880aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78004d90 msgr2=0x7fcc880b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78004d90 0x7fcc880b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 -- 172.21.15.104:0/280517470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880af920 con 0x7fcc88154f60 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8d6a8700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800baf0 tx=0x7fcc80001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800019b0 con 0x7fcc88154f60 2023-06-10T18:52:49.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80012610 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc880afc00 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8001e650 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.836+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc880b06e0 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.837+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc880b0400 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.838+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcc80024020 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.838+0000 7fcc857fa700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 0x7fcc6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.839+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fcc8008b8e0 con 0x7fcc88154f60 2023-06-10T18:52:49.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.839+0000 7fcc8cea7700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 0x7fcc6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:49.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.839+0000 7fcc8cea7700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 0x7fcc6c06d190 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcc8815b200 tx=0x7fcc7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:49.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:49.843+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcc880b0400 con 0x7fcc88154f60 2023-06-10T18:52:50.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.071+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc88074050 con 0x7fcc88154f60 2023-06-10T18:52:50.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.072+0000 7fcc857fa700 1 -- 172.21.15.104:0/280517470 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fcc8002c090 con 0x7fcc88154f60 2023-06-10T18:52:50.078 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:50.078 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:50.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 msgr2=0x7fcc6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 0x7fcc6c06d190 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcc8815b200 tx=0x7fcc7c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 msgr2=0x7fcc88155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800baf0 tx=0x7fcc80001d80 comp rx=0 tx=0).stop 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 shutdown_connections 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc6c06ace0 0x7fcc6c06d190 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:50.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc78004d90 0x7fcc880b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc88154f60 0x7fcc88155400 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 --2- 172.21.15.104:0/280517470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc880b3040 0x7fcc880aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 >> 172.21.15.104:0/280517470 conn(0x7fcc8806b8a0 msgr2=0x7fcc88072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 shutdown_connections 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:50.074+0000 7fcc8f10b700 1 -- 172.21.15.104:0/280517470 wait complete. 2023-06-10T18:52:50.081 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:50.091 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:51.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:51.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.220+0000 7f71c1f0b700 1 Processor -- start 2023-06-10T18:52:51.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.220+0000 7f71c1f0b700 1 -- start start 2023-06-10T18:52:51.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71c1f0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 0x7f71bc06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71c1f0b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71bc0aec00 0x7f71bc06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71c1f0b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71bc06e5e0 con 0x7f71bc0b1070 2023-06-10T18:52:51.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71c1f0b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71bc0ae7e0 con 0x7f71bc0aec00 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71c1f0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc06df90 con 0x7f71bc06ea70 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71bc0aec00 0x7f71bc06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71bc0aec00 0x7f71bc06d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33634/0 (socket says 172.21.15.104:33634) 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 -- 172.21.15.104:0/1242076211 learned_addr learned my addr 172.21.15.104:0/1242076211 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71baffd700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 0x7f71bc06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 -- 172.21.15.104:0/1242076211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 msgr2=0x7f71bc06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:51.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 0x7f71bc06cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 -- 172.21.15.104:0/1242076211 >> v1:172.21.15.114:6789/0 conn(0x7f71bc0b1070 legacy=0x7f71bc0b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71bbfff700 1 -- 172.21.15.104:0/1242076211 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc06e270 con 0x7f71bc0aec00 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.221+0000 7f71baffd700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 0x7f71bc06cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71bbfff700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc06d320 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f71b400d150 tx=0x7f71b400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eee3009137089b1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71ba7fc700 1 -- 172.21.15.104:0/1242076211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4010070 con 0x7f71bc0aec00 2023-06-10T18:52:51.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71ba7fc700 1 -- 172.21.15.104:0/1242076211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b400ddc0 con 0x7f71bc0aec00 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71ba7fc700 1 -- 172.21.15.104:0/1242076211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b400b310 con 0x7f71bc0aec00 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 -- 172.21.15.104:0/1242076211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 msgr2=0x7f71bc06d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc06d320 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f71b400d150 tx=0x7f71b400d460 comp rx=0 tx=0).stop 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 -- 172.21.15.104:0/1242076211 shutdown_connections 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc06d320 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/1242076211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc06ea70 0x7f71bc06cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 -- 172.21.15.104:0/1242076211 >> 172.21.15.104:0/1242076211 conn(0x7f71bc06b8a0 msgr2=0x7f71bc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:51.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 -- 172.21.15.104:0/1242076211 shutdown_connections 2023-06-10T18:52:51.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.222+0000 7f71c1f0b700 1 -- 172.21.15.104:0/1242076211 wait complete. 2023-06-10T18:52:51.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.224+0000 7f71c1f0b700 1 Processor -- start 2023-06-10T18:52:51.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.224+0000 7f71c1f0b700 1 -- start start 2023-06-10T18:52:51.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71bc06ea70 0x7f71bc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc0b1070 0x7f71bc1546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc0b3880 con 0x7f71bc0aec00 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc0b3b60 con 0x7f71bc06ea70 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc0b3e40 con 0x7f71bc0b1070 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33638/0 (socket says 172.21.15.104:33638) 2023-06-10T18:52:51.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 -- 172.21.15.104:0/3871898368 learned_addr learned my addr 172.21.15.104:0/3871898368 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71bb7fe700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71bc06ea70 0x7f71bc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71bbfff700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc0b1070 0x7f71bc1546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 -- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc0b1070 msgr2=0x7f71bc1546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc0b1070 0x7f71bc1546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 -- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71bc06ea70 msgr2=0x7f71bc0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71bc06ea70 0x7f71bc0b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 -- 172.21.15.104:0/3871898368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b400bd40 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71baffd700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c7f0 tx=0x7f71ac00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac011800 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac011960 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc0b4120 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac011d10 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.225+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc0b5350 con 0x7f71bc0aec00 2023-06-10T18:52:51.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.226+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc074050 con 0x7f71bc0aec00 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.227+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f71ac024480 con 0x7f71bc0aec00 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.227+0000 7f71b3fff700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 0x7f719c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.227+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f71ac089890 con 0x7f71bc0aec00 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.228+0000 7f71bb7fe700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 0x7f719c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.228+0000 7f71bb7fe700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 0x7f719c06d190 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000fb0 tx=0x7f71a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:51.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.231+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f71ac0582f0 con 0x7f71bc0aec00 2023-06-10T18:52:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.459+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71bc0b42b0 con 0x7f71bc0aec00 2023-06-10T18:52:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.459+0000 7f71b3fff700 1 -- 172.21.15.104:0/3871898368 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f71ac057e80 con 0x7f71bc0aec00 2023-06-10T18:52:51.465 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:51.466 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 msgr2=0x7f719c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 0x7f719c06d190 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000fb0 tx=0x7f71a4009040 comp rx=0 tx=0).stop 2023-06-10T18:52:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 msgr2=0x7f71bc0cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c7f0 tx=0x7f71ac00cb00 comp rx=0 tx=0).stop 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 shutdown_connections 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f719c06ace0 0x7f719c06d190 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71bc06ea70 0x7f71bc0b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71bc0b1070 0x7f71bc1546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 --2- 172.21.15.104:0/3871898368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71bc0aec00 0x7f71bc0cace0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 >> 172.21.15.104:0/3871898368 conn(0x7f71bc06b8a0 msgr2=0x7f71bc06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:51.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 shutdown_connections 2023-06-10T18:52:51.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:51.461+0000 7f71c1f0b700 1 -- 172.21.15.104:0/3871898368 wait complete. 2023-06-10T18:52:51.469 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:51.479 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:52.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:52.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.608+0000 7f0190b12700 1 Processor -- start 2023-06-10T18:52:52.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.608+0000 7f0190b12700 1 -- start start 2023-06-10T18:52:52.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f0190b12700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f018c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f0190b12700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c06d290 con 0x7f018c069e70 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f0190b12700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c0ca2d0 con 0x7f018c06e0e0 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f0190b12700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c071640 con 0x7f018c06e4b0 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f018c069e70 0x7f018c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59708/0 (socket says 172.21.15.104:59708) 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018ad9d700 1 -- 172.21.15.104:0/1145368485 learned_addr learned my addr 172.21.15.104:0/1145368485 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:52.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f0189d9b700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f018c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706666504 0 0) 0x7f018c06d290 con 0x7f018c069e70 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01740035b0 con 0x7f018c069e70 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910737446 0 0) 0x7f018c0ca2d0 con 0x7f018c06e0e0 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c06d290 con 0x7f018c06e0e0 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913927535 0 0) 0x7f01740035b0 con 0x7f018c069e70 2023-06-10T18:52:52.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 msgr2=0x7f018c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f018c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 >> v1:172.21.15.114:6789/0 conn(0x7f018c06e0e0 legacy=0x7f018c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c06d4c0 con 0x7f018c069e70 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.609+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f01840019f0 con 0x7f018c069e70 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 >> v1:172.21.15.104:6789/0 conn(0x7f018c069e70 legacy=0x7f018c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f0174003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0174004120 0x7f0174004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0174004d90 0x7f0174005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c071640 con 0x7f0174004120 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017400bc00 con 0x7f0174004d90 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017400bdc0 con 0x7f018c06e4b0 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0174004120 0x7f0174004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018ad9d700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0174004d90 0x7f0174005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018a59c700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f0174003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 -- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 msgr2=0x7f0174003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f0174003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 -- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0174004d90 msgr2=0x7f0174005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0174004d90 0x7f0174005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 -- 172.21.15.104:0/1145368485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01740035b0 con 0x7f0174004120 2023-06-10T18:52:52.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018a59c700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f018c06e4b0 0x7f0174003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f0189d9b700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0174004120 0x7f0174004850 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0180009b30 tx=0x7f0174005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafae197cdfac7c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018000e6c0 con 0x7f0174004120 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018000e820 con 0x7f0174004120 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.610+0000 7f018959a700 1 -- 172.21.15.104:0/1145368485 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018000eb90 con 0x7f0174004120 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 -- 172.21.15.104:0/1145368485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0174004120 msgr2=0x7f0174004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 --2- 172.21.15.104:0/1145368485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0174004120 0x7f0174004850 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0180009b30 tx=0x7f0174005880 comp rx=0 tx=0).stop 2023-06-10T18:52:52.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f018ad9d700 1 -- 172.21.15.104:0/1145368485 reap_dead start 2023-06-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 -- 172.21.15.104:0/1145368485 shutdown_connections 2023-06-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 -- 172.21.15.104:0/1145368485 >> 172.21.15.104:0/1145368485 conn(0x7f018c06b8a0 msgr2=0x7f018c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 -- 172.21.15.104:0/1145368485 shutdown_connections 2023-06-10T18:52:52.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.611+0000 7f0190b12700 1 -- 172.21.15.104:0/1145368485 wait complete. 2023-06-10T18:52:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.612+0000 7f0190b12700 1 Processor -- start 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.612+0000 7f0190b12700 1 -- start start 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0174004120 0x7f018c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 0x7f018c15a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c15aa40 con 0x7f018c0b2850 2023-06-10T18:52:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0cc960 con 0x7f018c15a080 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0190b12700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c155330 con 0x7f0174004120 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33658/0 (socket says 172.21.15.104:33658) 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 -- 172.21.15.104:0/4023624400 learned_addr learned my addr 172.21.15.104:0/4023624400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f018a59c700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0174004120 0x7f018c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f018ad9d700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 0x7f018c15a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 -- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0174004120 msgr2=0x7f018c0b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0174004120 0x7f018c0b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 -- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 msgr2=0x7f018c15a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 0x7f018c15a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f0189d9b700 1 -- 172.21.15.104:0/4023624400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0cce90 con 0x7f018c0b2850 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.613+0000 7f018ad9d700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 0x7f018c15a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f0189d9b700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f018000e4b0 tx=0x7f01800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:52.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018000e6c0 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180003ab0 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c0ccbb0 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180003e60 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.614+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c0b31b0 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.615+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c0b33d0 con 0x7f018c0b2850 2023-06-10T18:52:52.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.615+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0180013470 con 0x7f018c0b2850 2023-06-10T18:52:52.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.615+0000 7f017affd700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 0x7f016c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:52.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.616+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f018008ad50 con 0x7f018c0b2850 2023-06-10T18:52:52.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.616+0000 7f018a59c700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 0x7f016c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:52.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.616+0000 7f018a59c700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 0x7f016c06d190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f017c0055b0 tx=0x7f017c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:52.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.620+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f018008e030 con 0x7f018c0b2850 2023-06-10T18:52:52.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.847+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f018c15aa40 con 0x7f018c0b2850 2023-06-10T18:52:52.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.848+0000 7f017affd700 1 -- 172.21.15.104:0/4023624400 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f018c15aa40 con 0x7f018c0b2850 2023-06-10T18:52:52.854 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:52.854 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:52.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 msgr2=0x7f016c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 0x7f016c06d190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f017c0055b0 tx=0x7f017c009040 comp rx=0 tx=0).stop 2023-06-10T18:52:52.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 msgr2=0x7f018c154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:52.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f018000e4b0 tx=0x7f01800047c0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 shutdown_connections 2023-06-10T18:52:52.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f016c06ace0 0x7f016c06d190 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c15a080 0x7f018c15a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0174004120 0x7f018c0b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 --2- 172.21.15.104:0/4023624400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f018c0b2850 0x7f018c154df0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:52.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 >> 172.21.15.104:0/4023624400 conn(0x7f018c06b8a0 msgr2=0x7f018c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:52.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 shutdown_connections 2023-06-10T18:52:52.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:52.850+0000 7f0190b12700 1 -- 172.21.15.104:0/4023624400 wait complete. 2023-06-10T18:52:52.859 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:52.868 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:53.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:54.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.996+0000 7feef83c2700 1 Processor -- start 2023-06-10T18:52:54.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.996+0000 7feef83c2700 1 -- start start 2023-06-10T18:52:54.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef83c2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feef00cea60 0x7feef0165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef83c2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00b6070 con 0x7feef00b4ff0 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef83c2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00b1d70 con 0x7feef015a420 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef83c2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef00b5460 con 0x7feef00cea60 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef695f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feef00cea60 0x7feef0165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef595d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feef00b4ff0 0x7feef00ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59724/0 (socket says 172.21.15.104:59724) 2023-06-10T18:52:54.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef595d700 1 -- 172.21.15.104:0/103376610 learned_addr learned my addr 172.21.15.104:0/103376610 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef695f700 1 -- 172.21.15.104:0/103376610 >> v1:172.21.15.114:6789/0 conn(0x7feef015a420 legacy=0x7feef00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef695f700 1 -- 172.21.15.104:0/103376610 >> v1:172.21.15.104:6789/0 conn(0x7feef00b4ff0 legacy=0x7feef00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef515c700 1 -- 172.21.15.104:0/103376610 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156244614 0 0) 0x7feef00b6070 con 0x7feef00b4ff0 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.997+0000 7feef695f700 1 -- 172.21.15.104:0/103376610 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00b5740 con 0x7feef00cea60 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef695f700 1 --2- 172.21.15.104:0/103376610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef00cea60 0x7feef0165060 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5b2df004df73ea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:54.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef515c700 1 -- 172.21.15.104:0/103376610 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee800e070 con 0x7feef00cea60 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef515c700 1 -- 172.21.15.104:0/103376610 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee800bb90 con 0x7feef00cea60 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef515c700 1 -- 172.21.15.104:0/103376610 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8003a20 con 0x7feef00cea60 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 -- 172.21.15.104:0/103376610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef00cea60 msgr2=0x7feef0165060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 --2- 172.21.15.104:0/103376610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef00cea60 0x7feef0165060 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee800b330 comp rx=0 tx=0).stop 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 -- 172.21.15.104:0/103376610 shutdown_connections 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 --2- 172.21.15.104:0/103376610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef00cea60 0x7feef0165060 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 -- 172.21.15.104:0/103376610 >> 172.21.15.104:0/103376610 conn(0x7feef0089fd0 msgr2=0x7feef00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:54.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 -- 172.21.15.104:0/103376610 shutdown_connections 2023-06-10T18:52:54.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:53.998+0000 7feef83c2700 1 -- 172.21.15.104:0/103376610 wait complete. 2023-06-10T18:52:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 Processor -- start 2023-06-10T18:52:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 -- start start 2023-06-10T18:52:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feef00b4ff0 0x7feef01567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 0x7feef00adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef01f32f0 con 0x7feef015a420 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef01f0660 con 0x7feef00b4ff0 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.000+0000 7feef83c2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00ad410 con 0x7feef00cea60 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33674/0 (socket says 172.21.15.104:33674) 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 -- 172.21.15.104:0/3425371395 learned_addr learned my addr 172.21.15.104:0/3425371395 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:54.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef615e700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feef00b4ff0 0x7feef01567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef595d700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 0x7feef00adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 -- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 msgr2=0x7feef00adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 0x7feef00adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 -- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feef00b4ff0 msgr2=0x7feef01567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feef00b4ff0 0x7feef01567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 -- 172.21.15.104:0/3425371395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee8009a70 con 0x7feef015a420 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef595d700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 0x7feef00adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef695f700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7feee800b730 tx=0x7feee8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8012840 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee80129a0 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00ad6f0 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8012d50 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.001+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef01f0940 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.002+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00b64e0 con 0x7feef015a420 2023-06-10T18:52:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.003+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feee801d420 con 0x7feef015a420 2023-06-10T18:52:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.003+0000 7feee6ffd700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 0x7feed406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.003+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7feee808bac0 con 0x7feef015a420 2023-06-10T18:52:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.003+0000 7feef615e700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 0x7feed406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.004+0000 7feef615e700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 0x7feed406d190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feee0009870 tx=0x7feee0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:54.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.008+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feef00b64e0 con 0x7feef015a420 2023-06-10T18:52:54.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.233+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feef00b64e0 con 0x7feef015a420 2023-06-10T18:52:54.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.234+0000 7feee6ffd700 1 -- 172.21.15.104:0/3425371395 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7feef00b64e0 con 0x7feef015a420 2023-06-10T18:52:54.240 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:54.240 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 msgr2=0x7feed406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 0x7feed406d190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feee0009870 tx=0x7feee0008040 comp rx=0 tx=0).stop 2023-06-10T18:52:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 msgr2=0x7feef015c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7feee800b730 tx=0x7feee8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 shutdown_connections 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feed406ace0 0x7feed406d190 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feef00b4ff0 0x7feef01567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef015a420 0x7feef015c9c0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 --2- 172.21.15.104:0/3425371395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feef00cea60 0x7feef00adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 >> 172.21.15.104:0/3425371395 conn(0x7feef0089fd0 msgr2=0x7feef01e5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 shutdown_connections 2023-06-10T18:52:54.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:54.236+0000 7feef83c2700 1 -- 172.21.15.104:0/3425371395 wait complete. 2023-06-10T18:52:54.244 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:54.254 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:55.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:55.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.381+0000 7f6064040700 1 Processor -- start 2023-06-10T18:52:55.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.381+0000 7f6064040700 1 -- start start 2023-06-10T18:52:55.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6064040700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f605c15a420 0x7f605c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6064040700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 0x7f605c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6064040700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0b6070 con 0x7f605c0b4ff0 2023-06-10T18:52:55.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6064040700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0b5460 con 0x7f605c15a420 2023-06-10T18:52:55.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6064040700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b5740 con 0x7f605c0cea60 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f605c15a420 0x7f605c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f60615db700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f605c0b4ff0 0x7f605c0ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59738/0 (socket says 172.21.15.104:59738) 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f60615db700 1 -- 172.21.15.104:0/3378169974 learned_addr learned my addr 172.21.15.104:0/3378169974 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f60625dd700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 0x7f605c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 -- 172.21.15.104:0/3378169974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 msgr2=0x7f605c15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 0x7f605c15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 -- 172.21.15.104:0/3378169974 >> v1:172.21.15.104:6789/0 conn(0x7f605c0b4ff0 legacy=0x7f605c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6060dda700 1 -- 172.21.15.104:0/3378169974 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481743081 0 0) 0x7f605c0b6070 con 0x7f605c0b4ff0 2023-06-10T18:52:55.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 -- 172.21.15.104:0/3378169974 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0b5a20 con 0x7f605c15a420 2023-06-10T18:52:55.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f60625dd700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 0x7f605c15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:52:55.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.382+0000 7f6061ddc700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c15a420 0x7f605c0b1660 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6058009840 tx=0x7f6058009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaea2d2abcd08a4a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:55.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6060dda700 1 -- 172.21.15.104:0/3378169974 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058011830 con 0x7f605c15a420 2023-06-10T18:52:55.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6060dda700 1 -- 172.21.15.104:0/3378169974 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6058011990 con 0x7f605c15a420 2023-06-10T18:52:55.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6060dda700 1 -- 172.21.15.104:0/3378169974 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058011d20 con 0x7f605c15a420 2023-06-10T18:52:55.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 -- 172.21.15.104:0/3378169974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c15a420 msgr2=0x7f605c0b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c15a420 0x7f605c0b1660 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6058009840 tx=0x7f6058009b50 comp rx=0 tx=0).stop 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 -- 172.21.15.104:0/3378169974 shutdown_connections 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c15a420 0x7f605c0b1660 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 --2- 172.21.15.104:0/3378169974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c0cea60 0x7f605c15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 -- 172.21.15.104:0/3378169974 >> 172.21.15.104:0/3378169974 conn(0x7f605c089fd0 msgr2=0x7f605c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 -- 172.21.15.104:0/3378169974 shutdown_connections 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.383+0000 7f6064040700 1 -- 172.21.15.104:0/3378169974 wait complete. 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.385+0000 7f6064040700 1 Processor -- start 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.385+0000 7f6064040700 1 -- start start 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c0b4ff0 0x7f605c1f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c15a420 0x7f605c1e0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1f1330 con 0x7f605c0cea60 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c1511b0 con 0x7f605c0b4ff0 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6064040700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c151490 con 0x7f605c15a420 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33686/0 (socket says 172.21.15.104:33686) 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 -- 172.21.15.104:0/3412382533 learned_addr learned my addr 172.21.15.104:0/3412382533 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f6061ddc700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c0b4ff0 0x7f605c1f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60625dd700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c15a420 0x7f605c1e0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 -- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c15a420 msgr2=0x7f605c1e0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c15a420 0x7f605c1e0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 -- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c0b4ff0 msgr2=0x7f605c1f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c0b4ff0 0x7f605c1f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 -- 172.21.15.104:0/3412382533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580094a0 con 0x7f605c0cea60 2023-06-10T18:52:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60615db700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f604c0011e0 tx=0x7f604c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.386+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c004c80 con 0x7f605c0cea60 2023-06-10T18:52:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.387+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c004e00 con 0x7f605c0cea60 2023-06-10T18:52:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.387+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c151770 con 0x7f605c0cea60 2023-06-10T18:52:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.387+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c007310 con 0x7f605c0cea60 2023-06-10T18:52:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.387+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c1ef960 con 0x7f605c0cea60 2023-06-10T18:52:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.387+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c074050 con 0x7f605c0cea60 2023-06-10T18:52:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.388+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f604c007470 con 0x7f605c0cea60 2023-06-10T18:52:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.389+0000 7f60527fc700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.389+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f604c089850 con 0x7f605c0cea60 2023-06-10T18:52:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.389+0000 7f6061ddc700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.389+0000 7f6061ddc700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60580090b0 tx=0x7f605801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.393+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f604c054820 con 0x7f605c0cea60 2023-06-10T18:52:55.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.619+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f605c070140 con 0x7f605c0cea60 2023-06-10T18:52:55.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.619+0000 7f60527fc700 1 -- 172.21.15.104:0/3412382533 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f604c057e40 con 0x7f605c0cea60 2023-06-10T18:52:55.625 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:55.626 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 msgr2=0x7f604006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60580090b0 tx=0x7f605801c040 comp rx=0 tx=0).stop 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 msgr2=0x7f605c155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f604c0011e0 tx=0x7f604c00ca30 comp rx=0 tx=0).stop 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 shutdown_connections 2023-06-10T18:52:55.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c0b4ff0 0x7f605c1f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c15a420 0x7f605c1e0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 --2- 172.21.15.104:0/3412382533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0cea60 0x7f605c155510 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 >> 172.21.15.104:0/3412382533 conn(0x7f605c089fd0 msgr2=0x7f605c158bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 shutdown_connections 2023-06-10T18:52:55.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:55.621+0000 7f6064040700 1 -- 172.21.15.104:0/3412382533 wait complete. 2023-06-10T18:52:55.629 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:55.638 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:56.640 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:56.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.767+0000 7f79e2d1f700 1 Processor -- start 2023-06-10T18:52:56.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.767+0000 7f79e2d1f700 1 -- start start 2023-06-10T18:52:56.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e2d1f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79dc0cd850 0x7f79dc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e2d1f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79dc0cb300 0x7f79dc06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e2d1f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0c9e60 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e2d1f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc0c9bc0 con 0x7f79dc0cd850 2023-06-10T18:52:56.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e2d1f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc0aee70 con 0x7f79dc0cb300 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79dc0cd850 0x7f79dc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f79dc0cd850 0x7f79dc06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33694/0 (socket says 172.21.15.104:33694) 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e0abb700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f79dc0aeaa0 0x7f79dc0cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59752/0 (socket says 172.21.15.104:59752) 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 -- 172.21.15.104:0/3209507069 learned_addr learned my addr 172.21.15.104:0/3209507069 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79e12bc700 1 --2- 172.21.15.104:0/3209507069 >> v2:172.21.15.114:3300/0 conn(0x7f79dc0cb300 0x7f79dc06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 -- 172.21.15.104:0/3209507069 >> v2:172.21.15.114:3300/0 conn(0x7f79dc0cb300 msgr2=0x7f79dc06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:56.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 --2- 172.21.15.104:0/3209507069 >> v2:172.21.15.114:3300/0 conn(0x7f79dc0cb300 0x7f79dc06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 -- 172.21.15.104:0/3209507069 >> v1:172.21.15.104:6789/0 conn(0x7f79dc0aeaa0 legacy=0x7f79dc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 -- 172.21.15.104:0/3209507069 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc06e540 con 0x7f79dc0cd850 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79db7fe700 1 -- 172.21.15.104:0/3209507069 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002316153 0 0) 0x7f79dc0c9e60 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.768+0000 7f79dbfff700 1 --2- 172.21.15.104:0/3209507069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0cd850 0x7f79dc06cd50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0722a0 tx=0x7f79c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0d3f8b775a27e5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79db7fe700 1 -- 172.21.15.104:0/3209507069 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800b870 con 0x7f79dc0cd850 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79db7fe700 1 -- 172.21.15.104:0/3209507069 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c800b9d0 con 0x7f79dc0cd850 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79db7fe700 1 -- 172.21.15.104:0/3209507069 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800bd60 con 0x7f79dc0cd850 2023-06-10T18:52:56.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3209507069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0cd850 msgr2=0x7f79dc06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3209507069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0cd850 0x7f79dc06cd50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0722a0 tx=0x7f79c8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3209507069 shutdown_connections 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3209507069 >> v2:172.21.15.114:3300/0 conn(0x7f79dc0cb300 0x7f79dc06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3209507069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0cd850 0x7f79dc06cd50 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3209507069 >> 172.21.15.104:0/3209507069 conn(0x7f79dc06b8a0 msgr2=0x7f79dc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3209507069 shutdown_connections 2023-06-10T18:52:56.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.769+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3209507069 wait complete. 2023-06-10T18:52:56.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.770+0000 7f79e2d1f700 1 Processor -- start 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 -- start start 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79dc0cb300 0x7f79dc15b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f79dc0cd850 0x7f79dc1f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc159820 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc1f5e10 con 0x7f79dc0cb300 2023-06-10T18:52:56.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.771+0000 7f79e2d1f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc15c580 con 0x7f79dc0cd850 2023-06-10T18:52:56.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33698/0 (socket says 172.21.15.104:33698) 2023-06-10T18:52:56.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 -- 172.21.15.104:0/3537599639 learned_addr learned my addr 172.21.15.104:0/3537599639 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:56.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79dbfff700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79dc0cb300 0x7f79dc15b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e12bc700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f79dc0cd850 0x7f79dc1f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 -- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f79dc0cd850 msgr2=0x7f79dc1f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f79dc0cd850 0x7f79dc1f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 -- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79dc0cb300 msgr2=0x7f79dc15b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79dc0cb300 0x7f79dc15b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 -- 172.21.15.104:0/3537599639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8009550 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e0abb700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c570 tx=0x7f79d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00074e0 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0007660 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc15ba00 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0005a70 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.772+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc15bfc0 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.773+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc074050 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.774+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f79d0005bd0 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.774+0000 7f79d97fa700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 0x7f79c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.774+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f79d0088b40 con 0x7f79dc0aeaa0 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.774+0000 7f79dbfff700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 0x7f79c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.775+0000 7f79dbfff700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 0x7f79c006d190 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f79c8014040 tx=0x7f79c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:56.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:56.778+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f79d0053a30 con 0x7f79dc0aeaa0 2023-06-10T18:52:57.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.010+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79dc06a960 con 0x7f79dc0aeaa0 2023-06-10T18:52:57.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.011+0000 7f79d97fa700 1 -- 172.21.15.104:0/3537599639 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f79d0017090 con 0x7f79dc0aeaa0 2023-06-10T18:52:57.017 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:57.017 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:57.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 msgr2=0x7f79c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 0x7f79c006d190 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f79c8014040 tx=0x7f79c8008030 comp rx=0 tx=0).stop 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 msgr2=0x7f79dc1f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c570 tx=0x7f79d000c880 comp rx=0 tx=0).stop 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 shutdown_connections 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f79dc0cd850 0x7f79dc1f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79dc0aeaa0 0x7f79dc1f29e0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:57.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f79c006ace0 0x7f79c006d190 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:57.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 --2- 172.21.15.104:0/3537599639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79dc0cb300 0x7f79dc15b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:57.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 >> 172.21.15.104:0/3537599639 conn(0x7f79dc06b8a0 msgr2=0x7f79dc06d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:57.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 shutdown_connections 2023-06-10T18:52:57.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:57.013+0000 7f79e2d1f700 1 -- 172.21.15.104:0/3537599639 wait complete. 2023-06-10T18:52:57.020 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:57.030 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:58.031 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.159+0000 7f68dff7f700 1 Processor -- start 2023-06-10T18:52:58.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.159+0000 7f68dff7f700 1 -- start start 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dff7f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68d815a420 0x7f68d80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dff7f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d80b6070 con 0x7f68d80b4ff0 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dff7f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d80b5460 con 0x7f68d80cea60 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dff7f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d80b5740 con 0x7f68d815a420 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dd51a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f68d80b4ff0 0x7f68d80ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59770/0 (socket says 172.21.15.104:59770) 2023-06-10T18:52:58.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68dd51a700 1 -- 172.21.15.104:0/3437878369 learned_addr learned my addr 172.21.15.104:0/3437878369 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.160+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3437878369 >> v2:172.21.15.114:3300/0 conn(0x7f68d815a420 0x7f68d80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033088923 0 0) 0x7f68d80b6070 con 0x7f68d80b4ff0 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c40035b0 con 0x7f68d80b4ff0 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103840079 0 0) 0x7f68c40035b0 con 0x7f68d80b4ff0 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 >> v2:172.21.15.114:3300/0 conn(0x7f68d815a420 msgr2=0x7f68d80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 --2- 172.21.15.104:0/3437878369 >> v2:172.21.15.114:3300/0 conn(0x7f68d815a420 0x7f68d80b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 >> v1:172.21.15.114:6789/0 conn(0x7f68d80cea60 legacy=0x7f68d815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80b5a20 con 0x7f68d80b4ff0 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f68d4001670 con 0x7f68d80b4ff0 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 >> v1:172.21.15.104:6789/0 conn(0x7f68d80b4ff0 legacy=0x7f68d80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 0x7f68c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c4004850 0x7f68c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4005200 0x7f68c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80b5740 con 0x7f68c4003a00 2023-06-10T18:52:58.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c400c080 con 0x7f68c4005200 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c400c240 con 0x7f68c4004850 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 0x7f68c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dd51a700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c4004850 0x7f68c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68de51c700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4005200 0x7f68c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 -- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c4004850 msgr2=0x7f68c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c4004850 0x7f68c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 -- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4005200 msgr2=0x7f68c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4005200 0x7f68c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 -- 172.21.15.104:0/3437878369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40035b0 con 0x7f68c4003a00 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68dd51a700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68c4004850 0x7f68c4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68de51c700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4005200 0x7f68c400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68de51c700 1 -- 172.21.15.104:0/3437878369 reap_dead start 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.161+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 0x7f68c4003e10 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009b60 tx=0x7f68c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d3429354242935 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:58.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800e6c0 con 0x7f68c4003a00 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c800e820 con 0x7f68c4003a00 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dcd19700 1 -- 172.21.15.104:0/3437878369 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800eb90 con 0x7f68c4003a00 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 -- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 msgr2=0x7f68c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 0x7f68c4003e10 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009b60 tx=0x7f68c4005880 comp rx=0 tx=0).stop 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 -- 172.21.15.104:0/3437878369 shutdown_connections 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3437878369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68c4003a00 0x7f68c4003e10 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 -- 172.21.15.104:0/3437878369 >> 172.21.15.104:0/3437878369 conn(0x7f68d8089fd0 msgr2=0x7f68d80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:58.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 -- 172.21.15.104:0/3437878369 shutdown_connections 2023-06-10T18:52:58.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.162+0000 7f68dff7f700 1 -- 172.21.15.104:0/3437878369 wait complete. 2023-06-10T18:52:58.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.164+0000 7f68dff7f700 1 Processor -- start 2023-06-10T18:52:58.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.164+0000 7f68dff7f700 1 -- start start 2023-06-10T18:52:58.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4003a00 0x7f68d8154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 0x7f68d8155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80c6780 con 0x7f68d80c5dd0 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8156aa0 con 0x7f68c4003a00 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dff7f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8155c60 con 0x7f68d814dd70 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33722/0 (socket says 172.21.15.104:33722) 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 -- 172.21.15.104:0/3025298450 learned_addr learned my addr 172.21.15.104:0/3025298450 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4003a00 0x7f68d8154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dd51a700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 0x7f68d8155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 -- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 msgr2=0x7f68d8155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 0x7f68d8155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 -- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4003a00 msgr2=0x7f68d8154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4003a00 0x7f68d8154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 -- 172.21.15.104:0/3025298450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d81556a0 con 0x7f68d80c5dd0 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68dd51a700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 0x7f68d8155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:52:58.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68de51c700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f68d8070e70 tx=0x7f68d000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.165+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0003c10 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.166+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d0003d70 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.166+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0005310 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.166+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d8155980 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.166+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d814e5d0 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.167+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d80c3f50 con 0x7f68d80c5dd0 2023-06-10T18:52:58.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.167+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f68d0005470 con 0x7f68d80c5dd0 2023-06-10T18:52:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.167+0000 7f68ce7fc700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 0x7f68bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.167+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f68d0089570 con 0x7f68d80c5dd0 2023-06-10T18:52:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.168+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 0x7f68bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.168+0000 7f68ddd1b700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 0x7f68bc06d190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f68c80090b0 tx=0x7f68c801e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:58.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.172+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f68d0054460 con 0x7f68d80c5dd0 2023-06-10T18:52:58.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.403+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d8074050 con 0x7f68d80c5dd0 2023-06-10T18:52:58.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.404+0000 7f68ce7fc700 1 -- 172.21.15.104:0/3025298450 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f68d0057a80 con 0x7f68d80c5dd0 2023-06-10T18:52:58.409 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:58.410 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:58.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 msgr2=0x7f68bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 0x7f68bc06d190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f68c80090b0 tx=0x7f68c801e040 comp rx=0 tx=0).stop 2023-06-10T18:52:58.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 msgr2=0x7f68d80c6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:58.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f68d8070e70 tx=0x7f68d000ed50 comp rx=0 tx=0).stop 2023-06-10T18:52:58.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 shutdown_connections 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f68bc06ace0 0x7f68bc06d190 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68c4003a00 0x7f68d8154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d80c5dd0 0x7f68d80c6240 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 --2- 172.21.15.104:0/3025298450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68d814dd70 0x7f68d8155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 >> 172.21.15.104:0/3025298450 conn(0x7f68d8089fd0 msgr2=0x7f68d80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:58.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 shutdown_connections 2023-06-10T18:52:58.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:58.406+0000 7f68dff7f700 1 -- 172.21.15.104:0/3025298450 wait complete. 2023-06-10T18:52:58.414 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:58.425 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:52:59.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:52:59.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.553+0000 7f9015618700 1 Processor -- start 2023-06-10T18:52:59.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.553+0000 7f9015618700 1 -- start start 2023-06-10T18:52:59.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f9015618700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90100b1e30 0x7f90100d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f9015618700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f901006b9f0 0x7f90100cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f9015618700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901006bfd0 con 0x7f90100d0a30 2023-06-10T18:52:59.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f9015618700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90100d0ed0 con 0x7f90100b1e30 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f9015618700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901006c2b0 con 0x7f901006b9f0 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90100b1e30 0x7f90100d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900e7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f90100d0a30 0x7f901006b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59786/0 (socket says 172.21.15.104:59786) 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900e7fc700 1 -- 172.21.15.104:0/3269532421 learned_addr learned my addr 172.21.15.104:0/3269532421 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900f7fe700 1 --2- 172.21.15.104:0/3269532421 >> v2:172.21.15.114:3300/0 conn(0x7f901006b9f0 0x7f90100cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 -- 172.21.15.104:0/3269532421 >> v2:172.21.15.114:3300/0 conn(0x7f901006b9f0 msgr2=0x7f90100cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:52:59.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 --2- 172.21.15.104:0/3269532421 >> v2:172.21.15.114:3300/0 conn(0x7f901006b9f0 0x7f90100cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 -- 172.21.15.104:0/3269532421 >> v1:172.21.15.104:6789/0 conn(0x7f90100d0a30 legacy=0x7f901006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 -- 172.21.15.104:0/3269532421 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90100ccf70 con 0x7f90100b1e30 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900dffb700 1 -- 172.21.15.104:0/3269532421 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964740940 0 0) 0x7f901006bfd0 con 0x7f90100d0a30 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.554+0000 7f900effd700 1 --2- 172.21.15.104:0/3269532421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100b1e30 0x7f90100d04f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f9010067360 tx=0x7f9004009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f938e7fa292c0fb0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f900dffb700 1 -- 172.21.15.104:0/3269532421 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004011830 con 0x7f90100b1e30 2023-06-10T18:52:59.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f900dffb700 1 -- 172.21.15.104:0/3269532421 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9004011990 con 0x7f90100b1e30 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f900dffb700 1 -- 172.21.15.104:0/3269532421 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004011d20 con 0x7f90100b1e30 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 -- 172.21.15.104:0/3269532421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100b1e30 msgr2=0x7f90100d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 --2- 172.21.15.104:0/3269532421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100b1e30 0x7f90100d04f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f9010067360 tx=0x7f9004009b20 comp rx=0 tx=0).stop 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 -- 172.21.15.104:0/3269532421 shutdown_connections 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 --2- 172.21.15.104:0/3269532421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100b1e30 0x7f90100d04f0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 --2- 172.21.15.104:0/3269532421 >> v2:172.21.15.114:3300/0 conn(0x7f901006b9f0 0x7f90100cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 -- 172.21.15.104:0/3269532421 >> 172.21.15.104:0/3269532421 conn(0x7f9010071ec0 msgr2=0x7f901009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:59.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 -- 172.21.15.104:0/3269532421 shutdown_connections 2023-06-10T18:52:59.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.555+0000 7f9015618700 1 -- 172.21.15.104:0/3269532421 wait complete. 2023-06-10T18:52:59.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.557+0000 7f9015618700 1 Processor -- start 2023-06-10T18:52:59.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.557+0000 7f9015618700 1 -- start start 2023-06-10T18:52:59.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.557+0000 7f9015618700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901006b9f0 0x7f90101f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.557+0000 7f9015618700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90100b1e30 0x7f9010158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.557+0000 7f9015618700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f9015618700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90100c9cc0 con 0x7f90100d0a30 2023-06-10T18:52:59.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f9015618700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90101f5420 con 0x7f90100b1e30 2023-06-10T18:52:59.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f9015618700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90101591a0 con 0x7f901006b9f0 2023-06-10T18:52:59.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33736/0 (socket says 172.21.15.104:33736) 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 -- 172.21.15.104:0/279164407 learned_addr learned my addr 172.21.15.104:0/279164407 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900effd700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901006b9f0 0x7f90101f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900e7fc700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90100b1e30 0x7f9010158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 -- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901006b9f0 msgr2=0x7f90101f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901006b9f0 0x7f90101f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 -- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90100b1e30 msgr2=0x7f9010158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90100b1e30 0x7f9010158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 -- 172.21.15.104:0/279164407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040094a0 con 0x7f90100d0a30 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f900f7fe700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9008007e40 tx=0x7f900800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900800ed30 con 0x7f90100d0a30 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008014420 con 0x7f90100d0a30 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90101f56c0 con 0x7f90100d0a30 2023-06-10T18:52:59.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90080147d0 con 0x7f90100d0a30 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.558+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90101f5c80 con 0x7f90100d0a30 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.559+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90100d1860 con 0x7f90100d0a30 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.560+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9008014930 con 0x7f90100d0a30 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.560+0000 7f8ff77fe700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 0x7f8ffc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.560+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f9008089d30 con 0x7f90100d0a30 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.560+0000 7f900effd700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 0x7f8ffc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:52:59.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.561+0000 7f900effd700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 0x7f8ffc06d190 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f90040090b0 tx=0x7f9004008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:52:59.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.565+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9008054cd0 con 0x7f90100d0a30 2023-06-10T18:52:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.794+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9010074050 con 0x7f90100d0a30 2023-06-10T18:52:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.795+0000 7f8ff77fe700 1 -- 172.21.15.104:0/279164407 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f90080582f0 con 0x7f90100d0a30 2023-06-10T18:52:59.801 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:52:59.801 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:52:59.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 msgr2=0x7f8ffc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 0x7f8ffc06d190 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f90040090b0 tx=0x7f9004008030 comp rx=0 tx=0).stop 2023-06-10T18:52:59.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 msgr2=0x7f90100ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:52:59.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f9008007e40 tx=0x7f900800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 shutdown_connections 2023-06-10T18:52:59.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90100d0a30 0x7f90100ce300 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901006b9f0 0x7f90101f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ffc06ace0 0x7f8ffc06d190 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 --2- 172.21.15.104:0/279164407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90100b1e30 0x7f9010158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:52:59.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 >> 172.21.15.104:0/279164407 conn(0x7f9010071ec0 msgr2=0x7f90100b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:52:59.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 shutdown_connections 2023-06-10T18:52:59.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:52:59.797+0000 7f9015618700 1 -- 172.21.15.104:0/279164407 wait complete. 2023-06-10T18:52:59.806 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:52:59.815 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:53:00.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.944+0000 7fb36ee01700 1 Processor -- start 2023-06-10T18:53:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.944+0000 7fb36ee01700 1 -- start start 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36ee01700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 0x7fb3681587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36ee01700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 0x7fb368159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36ee01700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368069e70 con 0x7fb368070a20 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36ee01700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3681595f0 con 0x7fb368158ce0 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36ee01700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681598d0 con 0x7fb36806a8c0 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb367fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 0x7fb3681587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36d39e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 0x7fb368159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36cb9d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb368070a20 0x7fb368070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37414/0 (socket says 172.21.15.104:37414) 2023-06-10T18:53:00.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.945+0000 7fb36cb9d700 1 -- 172.21.15.104:0/3877379140 learned_addr learned my addr 172.21.15.104:0/3877379140 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb367fff700 1 -- 172.21.15.104:0/3877379140 >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 msgr2=0x7fb368159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb367fff700 1 --2- 172.21.15.104:0/3877379140 >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 0x7fb368159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb367fff700 1 -- 172.21.15.104:0/3877379140 >> v1:172.21.15.114:6789/0 conn(0x7fb368070a20 legacy=0x7fb368070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb3677fe700 1 -- 172.21.15.104:0/3877379140 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168780565 0 0) 0x7fb368069e70 con 0x7fb368070a20 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb367fff700 1 -- 172.21.15.104:0/3877379140 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36806a0c0 con 0x7fb36806a8c0 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb36d39e700 1 --2- 172.21.15.104:0/3877379140 >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 0x7fb368159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb367fff700 1 --2- 172.21.15.104:0/3877379140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 0x7fb3681587a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb354009840 tx=0x7fb354009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff80e27ecf66842 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb3677fe700 1 -- 172.21.15.104:0/3877379140 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354011830 con 0x7fb36806a8c0 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb3677fe700 1 -- 172.21.15.104:0/3877379140 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb354011990 con 0x7fb36806a8c0 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.946+0000 7fb3677fe700 1 -- 172.21.15.104:0/3877379140 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354011d20 con 0x7fb36806a8c0 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 -- 172.21.15.104:0/3877379140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 msgr2=0x7fb3681587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 --2- 172.21.15.104:0/3877379140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 0x7fb3681587a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb354009840 tx=0x7fb354009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 -- 172.21.15.104:0/3877379140 shutdown_connections 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 --2- 172.21.15.104:0/3877379140 >> v2:172.21.15.114:3300/0 conn(0x7fb368158ce0 0x7fb368159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 --2- 172.21.15.104:0/3877379140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb36806a8c0 0x7fb3681587a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 -- 172.21.15.104:0/3877379140 >> 172.21.15.104:0/3877379140 conn(0x7fb3680b1610 msgr2=0x7fb3680ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:00.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 -- 172.21.15.104:0/3877379140 shutdown_connections 2023-06-10T18:53:00.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.947+0000 7fb36ee01700 1 -- 172.21.15.104:0/3877379140 wait complete. 2023-06-10T18:53:00.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.948+0000 7fb36ee01700 1 Processor -- start 2023-06-10T18:53:00.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.948+0000 7fb36ee01700 1 -- start start 2023-06-10T18:53:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 0x7fb36814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb368158ce0 0x7fb368148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680c78c0 con 0x7fb36806a8c0 2023-06-10T18:53:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680b21c0 con 0x7fb368158ce0 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36ee01700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680c7ad0 con 0x7fb368070a20 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36cb9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36cb9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33746/0 (socket says 172.21.15.104:33746) 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36cb9d700 1 -- 172.21.15.104:0/188982552 learned_addr learned my addr 172.21.15.104:0/188982552 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.949+0000 7fb36d39e700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb368158ce0 0x7fb368148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb367fff700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 0x7fb36814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 -- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 msgr2=0x7fb36814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 0x7fb36814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 -- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb368158ce0 msgr2=0x7fb368148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb368158ce0 0x7fb368148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 -- 172.21.15.104:0/188982552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36000ab80 con 0x7fb36806a8c0 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb367fff700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 0x7fb36814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36cb9d700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0011e0 tx=0x7fb35c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c004c80 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c004e00 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c007310 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3540094a0 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.950+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3680c8090 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.951+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368071780 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.951+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb35c007470 con 0x7fb36806a8c0 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.952+0000 7fb3657fa700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.952+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7fb35c0893c0 con 0x7fb36806a8c0 2023-06-10T18:53:00.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.952+0000 7fb367fff700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:00.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.952+0000 7fb367fff700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb3540090b0 tx=0x7fb354008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:00.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:00.956+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb35c0542b0 con 0x7fb36806a8c0 2023-06-10T18:53:01.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.181+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3680cd6a0 con 0x7fb36806a8c0 2023-06-10T18:53:01.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.182+0000 7fb3657fa700 1 -- 172.21.15.104:0/188982552 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fb35c0578d0 con 0x7fb36806a8c0 2023-06-10T18:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:01.188 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:01.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 msgr2=0x7fb34c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb3540090b0 tx=0x7fb354008030 comp rx=0 tx=0).stop 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 msgr2=0x7fb3680b1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0011e0 tx=0x7fb35c00cae0 comp rx=0 tx=0).stop 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 shutdown_connections 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb368158ce0 0x7fb368148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:01.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb36806a8c0 0x7fb3680b1c80 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 --2- 172.21.15.104:0/188982552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb368070a20 0x7fb36814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 >> 172.21.15.104:0/188982552 conn(0x7fb3680b1610 msgr2=0x7fb368072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 shutdown_connections 2023-06-10T18:53:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:01.184+0000 7fb36ee01700 1 -- 172.21.15.104:0/188982552 wait complete. 2023-06-10T18:53:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:53:01.201 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:53:02.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:02.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.330+0000 7f0f6175e700 1 Processor -- start 2023-06-10T18:53:02.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.330+0000 7f0f6175e700 1 -- start start 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f6175e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 0x7f0f5c160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f6175e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f5c0cae10 0x7f0f5c0cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f6175e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c0d0340 con 0x7f0f5c06c720 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f6175e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c0d1660 con 0x7f0f5c0cae10 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f6175e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c15cb60 con 0x7f0f5c0ce390 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f5c0cae10 0x7f0f5c0cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5a7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0f5c06c720 0x7f0f5c0ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59814/0 (socket says 172.21.15.104:59814) 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5a7fc700 1 -- 172.21.15.104:0/1226627077 learned_addr learned my addr 172.21.15.104:0/1226627077 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5affd700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 0x7f0f5c160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 -- 172.21.15.104:0/1226627077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 msgr2=0x7f0f5c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 0x7f0f5c160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 -- 172.21.15.104:0/1226627077 >> v1:172.21.15.104:6789/0 conn(0x7f0f5c06c720 legacy=0x7f0f5c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f59ffb700 1 -- 172.21.15.104:0/1226627077 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928974327 0 0) 0x7f0f5c0d0340 con 0x7f0f5c06c720 2023-06-10T18:53:02.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 -- 172.21.15.104:0/1226627077 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c161230 con 0x7f0f5c0cae10 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5affd700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 0x7f0f5c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.331+0000 7f0f5b7fe700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0cae10 0x7f0f5c0cb250 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009d90 tx=0x7f0f5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0bf4f4edd1f422 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f59ffb700 1 -- 172.21.15.104:0/1226627077 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400e070 con 0x7f0f5c0cae10 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f59ffb700 1 -- 172.21.15.104:0/1226627077 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5400bbd0 con 0x7f0f5c0cae10 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f59ffb700 1 -- 172.21.15.104:0/1226627077 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54012430 con 0x7f0f5c0cae10 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 -- 172.21.15.104:0/1226627077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0cae10 msgr2=0x7f0f5c0cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0cae10 0x7f0f5c0cb250 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009d90 tx=0x7f0f5400b330 comp rx=0 tx=0).stop 2023-06-10T18:53:02.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 -- 172.21.15.104:0/1226627077 shutdown_connections 2023-06-10T18:53:02.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0cae10 0x7f0f5c0cb250 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 --2- 172.21.15.104:0/1226627077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0ce390 0x7f0f5c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 -- 172.21.15.104:0/1226627077 >> 172.21.15.104:0/1226627077 conn(0x7f0f5c069e70 msgr2=0x7f0f5c06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:02.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 -- 172.21.15.104:0/1226627077 shutdown_connections 2023-06-10T18:53:02.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.332+0000 7f0f6175e700 1 -- 172.21.15.104:0/1226627077 wait complete. 2023-06-10T18:53:02.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 Processor -- start 2023-06-10T18:53:02.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 -- start start 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0cae10 0x7f0f5c1e6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f5c0ce390 0x7f0f5c1f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.334+0000 7f0f6175e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c15cd30 con 0x7f0f5c06c720 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f6175e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0c6bd0 con 0x7f0f5c0ce390 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f6175e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0c6eb0 con 0x7f0f5c0cae10 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33760/0 (socket says 172.21.15.104:33760) 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 -- 172.21.15.104:0/372982716 learned_addr learned my addr 172.21.15.104:0/372982716 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5a7fc700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0cae10 0x7f0f5c1e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5b7fe700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f5c0ce390 0x7f0f5c1f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 -- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0cae10 msgr2=0x7f0f5c1e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0cae10 0x7f0f5c1e6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 -- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f5c0ce390 msgr2=0x7f0f5c1f5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f5c0ce390 0x7f0f5c1f5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 -- 172.21.15.104:0/372982716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54009a70 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f5affd700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000c750 tx=0x7f0f5000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50011800 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50011960 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c15cf80 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50011cd0 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.335+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c1f6050 con 0x7f0f5c06c720 2023-06-10T18:53:02.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.336+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c15d4a0 con 0x7f0f5c06c720 2023-06-10T18:53:02.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.337+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0f50021350 con 0x7f0f5c06c720 2023-06-10T18:53:02.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.337+0000 7f0f437fe700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 0x7f0f4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:02.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.337+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f0f500894f0 con 0x7f0f5c06c720 2023-06-10T18:53:02.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.337+0000 7f0f5a7fc700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 0x7f0f4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:02.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.338+0000 7f0f5a7fc700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 0x7f0f4806d190 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c000f60 tx=0x7f0f4c007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:02.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.341+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0f5c15d4a0 con 0x7f0f5c06c720 2023-06-10T18:53:02.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.568+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f5c15d4a0 con 0x7f0f5c06c720 2023-06-10T18:53:02.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.569+0000 7f0f437fe700 1 -- 172.21.15.104:0/372982716 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f0f5c15d4a0 con 0x7f0f5c06c720 2023-06-10T18:53:02.575 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:02.576 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:02.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 msgr2=0x7f0f4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 0x7f0f4806d190 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c000f60 tx=0x7f0f4c007e70 comp rx=0 tx=0).stop 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 msgr2=0x7f0f5c1e6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000c750 tx=0x7f0f5000ca60 comp rx=0 tx=0).stop 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 shutdown_connections 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f4806ace0 0x7f0f4806d190 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f5c0ce390 0x7f0f5c1f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c06c720 0x7f0f5c1e6140 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 --2- 172.21.15.104:0/372982716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f5c0cae10 0x7f0f5c1e6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:02.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 >> 172.21.15.104:0/372982716 conn(0x7f0f5c069e70 msgr2=0x7f0f5c15bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:02.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 shutdown_connections 2023-06-10T18:53:02.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:02.571+0000 7f0f6175e700 1 -- 172.21.15.104:0/372982716 wait complete. 2023-06-10T18:53:02.578 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:53:02.588 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:53:03.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:03.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.716+0000 7f4056457700 1 Processor -- start 2023-06-10T18:53:03.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.716+0000 7f4056457700 1 -- start start 2023-06-10T18:53:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f4056457700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40500d2b20 0x7f40500d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f4056457700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500cd430 con 0x7f40500d0520 2023-06-10T18:53:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f4056457700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500cd710 con 0x7f40500ccfa0 2023-06-10T18:53:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f4056457700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40500cd9f0 con 0x7f40500d2b20 2023-06-10T18:53:03.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40500d2b20 0x7f40500d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f40549f4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f40500d0520 0x7f40500ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59832/0 (socket says 172.21.15.104:59832) 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f40549f4700 1 -- 172.21.15.104:0/2267821290 learned_addr learned my addr 172.21.15.104:0/2267821290 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404ffff700 1 -- 172.21.15.104:0/2267821290 >> v1:172.21.15.114:6789/0 conn(0x7f40500ccfa0 legacy=0x7f40500b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404ffff700 1 -- 172.21.15.104:0/2267821290 >> v1:172.21.15.104:6789/0 conn(0x7f40500d0520 legacy=0x7f40500ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404effd700 1 -- 172.21.15.104:0/2267821290 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230138066 0 0) 0x7f40500cd430 con 0x7f40500d0520 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404ffff700 1 -- 172.21.15.104:0/2267821290 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500cdcd0 con 0x7f40500d2b20 2023-06-10T18:53:03.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.717+0000 7f404ffff700 1 --2- 172.21.15.104:0/2267821290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40500d2b20 0x7f40500d2f00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4038009840 tx=0x7f4038009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e8b413f16ca2c9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f404effd700 1 -- 172.21.15.104:0/2267821290 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038011830 con 0x7f40500d2b20 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f404effd700 1 -- 172.21.15.104:0/2267821290 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4038011990 con 0x7f40500d2b20 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f404effd700 1 -- 172.21.15.104:0/2267821290 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038011d20 con 0x7f40500d2b20 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 -- 172.21.15.104:0/2267821290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40500d2b20 msgr2=0x7f40500d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 --2- 172.21.15.104:0/2267821290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40500d2b20 0x7f40500d2f00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4038009840 tx=0x7f4038009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:03.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 -- 172.21.15.104:0/2267821290 shutdown_connections 2023-06-10T18:53:03.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 --2- 172.21.15.104:0/2267821290 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40500d2b20 0x7f40500d2f00 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 -- 172.21.15.104:0/2267821290 >> 172.21.15.104:0/2267821290 conn(0x7f4050071ec0 msgr2=0x7f405009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:03.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 -- 172.21.15.104:0/2267821290 shutdown_connections 2023-06-10T18:53:03.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.718+0000 7f4056457700 1 -- 172.21.15.104:0/2267821290 wait complete. 2023-06-10T18:53:03.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.719+0000 7f4056457700 1 Processor -- start 2023-06-10T18:53:03.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 -- start start 2023-06-10T18:53:03.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 0x7f40500b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:03.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40500d0520 0x7f40500c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015d810 con 0x7f405015aac0 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015c920 con 0x7f40500d0520 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f4056457700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405015cc00 con 0x7f40500ccfa0 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.720+0000 7f40549f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33778/0 (socket says 172.21.15.104:33778) 2023-06-10T18:53:03.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 -- 172.21.15.104:0/1936554652 learned_addr learned my addr 172.21.15.104:0/1936554652 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404ffff700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 0x7f40500b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404f7fe700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40500d0520 0x7f40500c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 -- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 msgr2=0x7f40500b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 0x7f40500b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 -- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40500d0520 msgr2=0x7f40500c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40500d0520 0x7f40500c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 -- 172.21.15.104:0/1936554652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380094a0 con 0x7f405015aac0 2023-06-10T18:53:03.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404ffff700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 0x7f40500b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f40549f4700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f404800cca0 tx=0x7f404800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404800cea0 con 0x7f405015aac0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404800ade0 con 0x7f405015aac0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405015cee0 con 0x7f405015aac0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048007310 con 0x7f405015aac0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.721+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405015d4a0 con 0x7f405015aac0 2023-06-10T18:53:03.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.722+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405015d9c0 con 0x7f405015aac0 2023-06-10T18:53:03.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.723+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4048007470 con 0x7f405015aac0 2023-06-10T18:53:03.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.723+0000 7f404cff9700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:03.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.723+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f4048089770 con 0x7f405015aac0 2023-06-10T18:53:03.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.723+0000 7f404ffff700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:03.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.724+0000 7f404ffff700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 0x7f403006d190 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f40501f5950 tx=0x7f4038008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:03.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.727+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4048054640 con 0x7f405015aac0 2023-06-10T18:53:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.958+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40500b2920 con 0x7f405015aac0 2023-06-10T18:53:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.959+0000 7f404cff9700 1 -- 172.21.15.104:0/1936554652 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7f4048057c60 con 0x7f405015aac0 2023-06-10T18:53:03.965 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:03.965 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 msgr2=0x7f403006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 0x7f403006d190 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f40501f5950 tx=0x7f4038008030 comp rx=0 tx=0).stop 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 msgr2=0x7f405015af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f404800cca0 tx=0x7f404800a340 comp rx=0 tx=0).stop 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 shutdown_connections 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405015aac0 0x7f405015af40 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500ccfa0 0x7f40500b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 --2- 172.21.15.104:0/1936554652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40500d0520 0x7f40500c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 >> 172.21.15.104:0/1936554652 conn(0x7f4050071ec0 msgr2=0x7f405006b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 shutdown_connections 2023-06-10T18:53:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:03.961+0000 7f4056457700 1 -- 172.21.15.104:0/1936554652 wait complete. 2023-06-10T18:53:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:53:03.978 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:53:04.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:05.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 Processor -- start 2023-06-10T18:53:05.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 -- start start 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb3040cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb3040b1e80 0x7fb3040d2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3040b1c40 con 0x7fb3040d0530 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3040b1680 con 0x7fb3040b1e80 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.107+0000 7fb3093d2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3040b1960 con 0x7fb30406c260 2023-06-10T18:53:05.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb3040b1e80 0x7fb3040d2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb3040b1e80 0x7fb3040d2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33784/0 (socket says 172.21.15.104:33784) 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb302ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb3040d0530 0x7fb3040d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59842/0 (socket says 172.21.15.104:59842) 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 -- 172.21.15.104:0/3540909273 learned_addr learned my addr 172.21.15.104:0/3540909273 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3027fc700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb3040cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 -- 172.21.15.104:0/3540909273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 msgr2=0x7fb3040cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb3040cfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 -- 172.21.15.104:0/3540909273 >> v1:172.21.15.104:6789/0 conn(0x7fb3040d0530 legacy=0x7fb3040d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb301ffb700 1 -- 172.21.15.104:0/3540909273 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248958731 0 0) 0x7fb3040b1c40 con 0x7fb3040d0530 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 -- 172.21.15.104:0/3540909273 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30406b780 con 0x7fb3040b1e80 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb3037fe700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb3040d2520 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009e50 tx=0x7fb2fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadff8b5a106da15 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb301ffb700 1 -- 172.21.15.104:0/3540909273 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc00e070 con 0x7fb3040b1e80 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb301ffb700 1 -- 172.21.15.104:0/3540909273 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc00b9c0 con 0x7fb3040b1e80 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.108+0000 7fb301ffb700 1 -- 172.21.15.104:0/3540909273 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc00bd70 con 0x7fb3040b1e80 2023-06-10T18:53:05.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 -- 172.21.15.104:0/3540909273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 msgr2=0x7fb3040d2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb3040d2520 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009e50 tx=0x7fb2fc00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 -- 172.21.15.104:0/3540909273 shutdown_connections 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb3040d2520 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 --2- 172.21.15.104:0/3540909273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb3040cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 -- 172.21.15.104:0/3540909273 >> 172.21.15.104:0/3540909273 conn(0x7fb304071ec0 msgr2=0x7fb30409bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 -- 172.21.15.104:0/3540909273 shutdown_connections 2023-06-10T18:53:05.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.109+0000 7fb3093d2700 1 -- 172.21.15.104:0/3540909273 wait complete. 2023-06-10T18:53:05.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.111+0000 7fb3093d2700 1 Processor -- start 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.111+0000 7fb3093d2700 1 -- start start 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb304155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3040d0530 0x7fb3040c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3040c8e60 con 0x7fb3040b1e80 2023-06-10T18:53:05.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3040b8890 con 0x7fb3040d0530 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3040c7de0 con 0x7fb30406c260 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33790/0 (socket says 172.21.15.104:33790) 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 -- 172.21.15.104:0/1514063600 learned_addr learned my addr 172.21.15.104:0/1514063600 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb302ffd700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb304155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3037fe700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3040d0530 0x7fb3040c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 -- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 msgr2=0x7fb304155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb304155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 -- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3040d0530 msgr2=0x7fb3040c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3040d0530 0x7fb3040c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 -- 172.21.15.104:0/1514063600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc009b30 con 0x7fb3040b1e80 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3027fc700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400d800 tx=0x7fb2f400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40126b0 con 0x7fb3040b1e80 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f4012810 con 0x7fb3040b1e80 2023-06-10T18:53:05.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3040c8120 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4012ba0 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.112+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3040c8960 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.114+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb304074050 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.114+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb2f4012d00 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.114+0000 7fb2eb7fe700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 0x7fb2f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.114+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fb2f40897c0 con 0x7fb3040b1e80 2023-06-10T18:53:05.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.115+0000 7fb302ffd700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 0x7fb2f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.115+0000 7fb302ffd700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 0x7fb2f006d190 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:05.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.119+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb2f4058120 con 0x7fb3040b1e80 2023-06-10T18:53:05.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.346+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3040c82b0 con 0x7fb3040b1e80 2023-06-10T18:53:05.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.347+0000 7fb2eb7fe700 1 -- 172.21.15.104:0/1514063600 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4689 (secure 0 0 0) 0x7fb2f4057cb0 con 0x7fb3040b1e80 2023-06-10T18:53:05.353 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:05.353 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:52:15.647949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 msgr2=0x7fb2f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 0x7fb2f006d190 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8007e70 comp rx=0 tx=0).stop 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 msgr2=0x7fb304155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400d800 tx=0x7fb2f400db10 comp rx=0 tx=0).stop 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 shutdown_connections 2023-06-10T18:53:05.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb2f006ace0 0x7fb2f006d190 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3040d0530 0x7fb3040c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb30406c260 0x7fb304155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 --2- 172.21.15.104:0/1514063600 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3040b1e80 0x7fb304155460 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 >> 172.21.15.104:0/1514063600 conn(0x7fb304071ec0 msgr2=0x7fb30415a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 shutdown_connections 2023-06-10T18:53:05.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:05.349+0000 7fb3093d2700 1 -- 172.21.15.104:0/1514063600 wait complete. 2023-06-10T18:53:05.357 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 45 2023-06-10T18:53:05.366 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:53:06.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:06.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 Processor -- start 2023-06-10T18:53:06.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 -- start start 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd87c0ce390 0x7fd87c160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c0d0340 con 0x7fd87c06b410 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c0d1660 con 0x7fd87c0ce390 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd8816c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c15cb60 con 0x7fd87c06c720 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd87affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd87c0ce390 0x7fd87c160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.495+0000 7fd87b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd87c06b410 0x7fd87c06b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59856/0 (socket says 172.21.15.104:59856) 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87b7fe700 1 -- 172.21.15.104:0/310474349 learned_addr learned my addr 172.21.15.104:0/310474349 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87a7fc700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87affd700 1 -- 172.21.15.104:0/310474349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 msgr2=0x7fd87c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87affd700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c0ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87affd700 1 -- 172.21.15.104:0/310474349 >> v1:172.21.15.104:6789/0 conn(0x7fd87c06b410 legacy=0x7fd87c06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd879ffb700 1 -- 172.21.15.104:0/310474349 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754378773 0 0) 0x7fd87c0d0340 con 0x7fd87c06b410 2023-06-10T18:53:06.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87affd700 1 -- 172.21.15.104:0/310474349 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c161230 con 0x7fd87c0ce390 2023-06-10T18:53:06.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd87affd700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c0ce390 0x7fd87c160cf0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd870009840 tx=0x7fd870009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e47cff4803f11a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:06.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd879ffb700 1 -- 172.21.15.104:0/310474349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870011830 con 0x7fd87c0ce390 2023-06-10T18:53:06.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd879ffb700 1 -- 172.21.15.104:0/310474349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870011990 con 0x7fd87c0ce390 2023-06-10T18:53:06.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.496+0000 7fd879ffb700 1 -- 172.21.15.104:0/310474349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870011d20 con 0x7fd87c0ce390 2023-06-10T18:53:06.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 -- 172.21.15.104:0/310474349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c0ce390 msgr2=0x7fd87c160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c0ce390 0x7fd87c160cf0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd870009840 tx=0x7fd870009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 -- 172.21.15.104:0/310474349 shutdown_connections 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c0ce390 0x7fd87c160cf0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 --2- 172.21.15.104:0/310474349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c0ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 -- 172.21.15.104:0/310474349 >> 172.21.15.104:0/310474349 conn(0x7fd87c069e70 msgr2=0x7fd87c06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 -- 172.21.15.104:0/310474349 shutdown_connections 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.497+0000 7fd8816c0700 1 -- 172.21.15.104:0/310474349 wait complete. 2023-06-10T18:53:06.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.498+0000 7fd8816c0700 1 Processor -- start 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.499+0000 7fd8816c0700 1 -- start start 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.499+0000 7fd8816c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.499+0000 7fd8816c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 0x7fd87c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0b55a0 con 0x7fd87c06b410 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0b4990 con 0x7fd87c0ce390 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c0b4c70 con 0x7fd87c06c720 2023-06-10T18:53:06.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33800/0 (socket says 172.21.15.104:33800) 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 -- 172.21.15.104:0/4194862283 learned_addr learned my addr 172.21.15.104:0/4194862283 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87a7fc700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87b7fe700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 0x7fd87c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 -- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 msgr2=0x7fd87c15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 -- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 msgr2=0x7fd87c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 0x7fd87c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 -- 172.21.15.104:0/4194862283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700094a0 con 0x7fd87c06b410 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87b7fe700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 0x7fd87c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd87affd700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd870000c00 tx=0x7fd870004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700045d0 con 0x7fd87c06b410 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87001e070 con 0x7fd87c06b410 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c0b4f50 con 0x7fd87c06b410 2023-06-10T18:53:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700038e0 con 0x7fd87c06b410 2023-06-10T18:53:06.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.500+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c0c4520 con 0x7fd87c06b410 2023-06-10T18:53:06.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.501+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c074050 con 0x7fd87c06b410 2023-06-10T18:53:06.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.502+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd870003a40 con 0x7fd87c06b410 2023-06-10T18:53:06.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.502+0000 7fd8637fe700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 0x7fd86806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:06.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.502+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fd87008a670 con 0x7fd87c06b410 2023-06-10T18:53:06.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.502+0000 7fd87a7fc700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 0x7fd86806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:06.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.503+0000 7fd87a7fc700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 0x7fd86806d190 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd86c006b20 tx=0x7fd86c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:06.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.506+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd8700555d0 con 0x7fd87c06b410 2023-06-10T18:53:06.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.733+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd87c06b1f0 con 0x7fd87c06b410 2023-06-10T18:53:06.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.734+0000 7fd8637fe700 1 -- 172.21.15.104:0/4194862283 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4687 (secure 0 0 0) 0x7fd870058bf0 con 0x7fd87c06b410 2023-06-10T18:53:06.740 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:06.740 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:05.803547+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:06.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 msgr2=0x7fd86806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 0x7fd86806d190 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd86c006b20 tx=0x7fd86c008040 comp rx=0 tx=0).stop 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 msgr2=0x7fd87c152e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd870000c00 tx=0x7fd870004750 comp rx=0 tx=0).stop 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 shutdown_connections 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86806ace0 0x7fd86806d190 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd87c0ce390 0x7fd87c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd87c06b410 0x7fd87c152e10 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 --2- 172.21.15.104:0/4194862283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd87c06c720 0x7fd87c15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:06.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 >> 172.21.15.104:0/4194862283 conn(0x7fd87c069e70 msgr2=0x7fd87c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:06.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 shutdown_connections 2023-06-10T18:53:06.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:06.736+0000 7fd8816c0700 1 -- 172.21.15.104:0/4194862283 wait complete. 2023-06-10T18:53:06.744 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 46 2023-06-10T18:53:06.753 DEBUG:tasks.cephfs.filesystem:mapped states ['up:rejoin'] to None 2023-06-10T18:53:07.754 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:07.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.883+0000 7f4095e93700 1 Processor -- start 2023-06-10T18:53:07.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.883+0000 7f4095e93700 1 -- start start 2023-06-10T18:53:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.883+0000 7f4095e93700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 0x7f40900b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.883+0000 7f4095e93700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 0x7f409015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f4095e93700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40900b6070 con 0x7f40900b4ff0 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f4095e93700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40900b5460 con 0x7f40900cea60 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f4095e93700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900b5740 con 0x7f409015a420 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 0x7f409015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 0x7f40900b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408effd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f40900b4ff0 0x7f40900ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37482/0 (socket says 172.21.15.104:37482) 2023-06-10T18:53:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408effd700 1 -- 172.21.15.104:0/680472617 learned_addr learned my addr 172.21.15.104:0/680472617 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 -- 172.21.15.104:0/680472617 >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 msgr2=0x7f409015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 --2- 172.21.15.104:0/680472617 >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 0x7f409015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 -- 172.21.15.104:0/680472617 >> v1:172.21.15.114:6789/0 conn(0x7f40900b4ff0 legacy=0x7f40900ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 -- 172.21.15.104:0/680472617 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40900b5a20 con 0x7f409015a420 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408ffff700 1 --2- 172.21.15.104:0/680472617 >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 0x7f409015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.884+0000 7f408f7fe700 1 --2- 172.21.15.104:0/680472617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 0x7f40900b1660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4080009840 tx=0x7f4080009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536f567c977211ab server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:07.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f408e7fc700 1 -- 172.21.15.104:0/680472617 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080011830 con 0x7f409015a420 2023-06-10T18:53:07.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f408e7fc700 1 -- 172.21.15.104:0/680472617 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080011990 con 0x7f409015a420 2023-06-10T18:53:07.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f408e7fc700 1 -- 172.21.15.104:0/680472617 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080011d20 con 0x7f409015a420 2023-06-10T18:53:07.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 -- 172.21.15.104:0/680472617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 msgr2=0x7f40900b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:07.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 --2- 172.21.15.104:0/680472617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 0x7f40900b1660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4080009840 tx=0x7f4080009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:07.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 -- 172.21.15.104:0/680472617 shutdown_connections 2023-06-10T18:53:07.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 --2- 172.21.15.104:0/680472617 >> v2:172.21.15.114:3300/0 conn(0x7f40900cea60 0x7f409015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 --2- 172.21.15.104:0/680472617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409015a420 0x7f40900b1660 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 -- 172.21.15.104:0/680472617 >> 172.21.15.104:0/680472617 conn(0x7f4090089fd0 msgr2=0x7f40900d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 -- 172.21.15.104:0/680472617 shutdown_connections 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.885+0000 7f4095e93700 1 -- 172.21.15.104:0/680472617 wait complete. 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.886+0000 7f4095e93700 1 Processor -- start 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 -- start start 2023-06-10T18:53:07.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40900b4ff0 0x7f40900cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40900cea60 0x7f40900ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40900cadb0 con 0x7f409015a420 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40900cafc0 con 0x7f40900cea60 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.887+0000 7f4095e93700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900cb2a0 con 0x7f40900b4ff0 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33818/0 (socket says 172.21.15.104:33818) 2023-06-10T18:53:07.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 -- 172.21.15.104:0/408694612 learned_addr learned my addr 172.21.15.104:0/408694612 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408effd700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40900cea60 0x7f40900ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408f7fe700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40900b4ff0 0x7f40900cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 -- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40900b4ff0 msgr2=0x7f40900cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40900b4ff0 0x7f40900cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 -- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40900cea60 msgr2=0x7f40900ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40900cea60 0x7f40900ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 -- 172.21.15.104:0/408694612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088009a70 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f408ffff700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f408800efc0 tx=0x7f408800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408800cc90 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408800f040 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40800094a0 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880146c0 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.888+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40900cb860 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.889+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409006e070 con 0x7f409015a420 2023-06-10T18:53:07.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.890+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4088014820 con 0x7f409015a420 2023-06-10T18:53:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.890+0000 7f4077fff700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 0x7f407c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.890+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f408808a1f0 con 0x7f409015a420 2023-06-10T18:53:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.890+0000 7f408f7fe700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 0x7f407c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.891+0000 7f408f7fe700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 0x7f407c06d190 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4080000c00 tx=0x7f408001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:07.894+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4088055150 con 0x7f409015a420 2023-06-10T18:53:08.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.124+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4090074050 con 0x7f409015a420 2023-06-10T18:53:08.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.124+0000 7f4077fff700 1 -- 172.21.15.104:0/408694612 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4687 (secure 0 0 0) 0x7f4088058770 con 0x7f409015a420 2023-06-10T18:53:08.131 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:08.131 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:06.805017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":17,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:08.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 msgr2=0x7f407c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 0x7f407c06d190 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4080000c00 tx=0x7f408001c040 comp rx=0 tx=0).stop 2023-06-10T18:53:08.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 msgr2=0x7f40900c6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f408800efc0 tx=0x7f408800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 shutdown_connections 2023-06-10T18:53:08.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409015a420 0x7f40900c6b20 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40900b4ff0 0x7f40900cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407c06ace0 0x7f407c06d190 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 --2- 172.21.15.104:0/408694612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40900cea60 0x7f40900ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 >> 172.21.15.104:0/408694612 conn(0x7f4090089fd0 msgr2=0x7f4090072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 shutdown_connections 2023-06-10T18:53:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.126+0000 7f4095e93700 1 -- 172.21.15.104:0/408694612 wait complete. 2023-06-10T18:53:08.136 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 47 2023-06-10T18:53:08.143 DEBUG:tasks.cephfs.filesystem:mapped states ['up:active'] to up:active 2023-06-10T18:53:08.144 DEBUG:tasks.cephfs.filesystem:reached state 'up:active' in 51.72423839569092s 2023-06-10T18:53:08.144 INFO:tasks.cephfs.test_client_recovery:Reached active... 2023-06-10T18:53:08.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:53:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.268+0000 7fcb24b57700 1 Processor -- start 2023-06-10T18:53:08.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.268+0000 7fcb24b57700 1 -- start start 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb24b57700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb2006a580 0x7fcb200ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb24b57700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2006dcb0 con 0x7fcb200cf030 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb24b57700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200cf460 con 0x7fcb2015c500 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb24b57700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2006e850 con 0x7fcb2006a580 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1dd9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcb200cf030 0x7fcb20069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59882/0 (socket says 172.21.15.104:59882) 2023-06-10T18:53:08.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/170855557 learned_addr learned my addr 172.21.15.104:0/170855557 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148641187 0 0) 0x7fcb2006dcb0 con 0x7fcb200cf030 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1ed9d700 1 --2- 172.21.15.104:0/170855557 >> v2:172.21.15.114:3300/0 conn(0x7fcb2006a580 0x7fcb200ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb080035b0 con 0x7fcb200cf030 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208124835 0 0) 0x7fcb080035b0 con 0x7fcb200cf030 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 >> v2:172.21.15.114:3300/0 conn(0x7fcb2006a580 msgr2=0x7fcb200ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 --2- 172.21.15.104:0/170855557 >> v2:172.21.15.114:3300/0 conn(0x7fcb2006a580 0x7fcb200ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 >> v1:172.21.15.114:6789/0 conn(0x7fcb2015c500 legacy=0x7fcb2006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2006d0a0 con 0x7fcb200cf030 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcb14001670 con 0x7fcb200cf030 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 >> v1:172.21.15.104:6789/0 conn(0x7fcb200cf030 legacy=0x7fcb20069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb08003a00 0x7fcb08003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 0x7fcb08004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08005200 0x7fcb0800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2006e850 con 0x7fcb08004850 2023-06-10T18:53:08.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c060 con 0x7fcb08003a00 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800c220 con 0x7fcb08005200 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.269+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 0x7fcb08004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1e59c700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb08003a00 0x7fcb08003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1ed9d700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08005200 0x7fcb0800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08005200 msgr2=0x7fcb0800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08005200 0x7fcb0800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb08003a00 msgr2=0x7fcb08003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb08003a00 0x7fcb08003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/170855557 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080035b0 con 0x7fcb08004850 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1ed9d700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08005200 0x7fcb0800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1e59c700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb08003a00 0x7fcb08003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1ed9d700 1 -- 172.21.15.104:0/170855557 reap_dead start 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 0x7fcb08004cc0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400c240 tx=0x7fcb08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf0c78a1db85d8f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:08.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14007c10 con 0x7fcb08004850 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1400cde0 con 0x7fcb08004850 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb1d59a700 1 -- 172.21.15.104:0/170855557 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb14005ba0 con 0x7fcb08004850 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb24b57700 1 -- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 msgr2=0x7fcb08004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.270+0000 7fcb24b57700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 0x7fcb08004cc0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400c240 tx=0x7fcb08005880 comp rx=0 tx=0).stop 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.271+0000 7fcb24b57700 1 -- 172.21.15.104:0/170855557 shutdown_connections 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.271+0000 7fcb24b57700 1 --2- 172.21.15.104:0/170855557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb08004850 0x7fcb08004cc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.271+0000 7fcb24b57700 1 -- 172.21.15.104:0/170855557 >> 172.21.15.104:0/170855557 conn(0x7fcb2006b8a0 msgr2=0x7fcb200cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:08.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.271+0000 7fcb24b57700 1 -- 172.21.15.104:0/170855557 shutdown_connections 2023-06-10T18:53:08.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.271+0000 7fcb24b57700 1 -- 172.21.15.104:0/170855557 wait complete. 2023-06-10T18:53:08.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.272+0000 7fcb24b57700 1 Processor -- start 2023-06-10T18:53:08.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.272+0000 7fcb24b57700 1 -- start start 2023-06-10T18:53:08.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08004850 0x7fcb20155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb200c5c20 0x7fcb200b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200c64e0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20157dc0 con 0x7fcb200c5c20 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb201580a0 con 0x7fcb08004850 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33836/0 (socket says 172.21.15.104:33836) 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/3521322673 learned_addr learned my addr 172.21.15.104:0/3521322673 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:08.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1ed9d700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb200c5c20 0x7fcb200b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1e59c700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08004850 0x7fcb20155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08004850 msgr2=0x7fcb20155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08004850 0x7fcb20155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb200c5c20 msgr2=0x7fcb200b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb200c5c20 0x7fcb200b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 -- 172.21.15.104:0/3521322673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb20158380 con 0x7fcb201f1cf0 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb1dd9b700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c030 tx=0x7fcb14001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:08.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140070d0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb14007230 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb20158660 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140075c0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.273+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb20155600 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.274+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb201589d0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.275+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb1401e430 con 0x7fcb201f1cf0 2023-06-10T18:53:08.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.275+0000 7fcb0effd700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 0x7fcb0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:08.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.275+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fcb1408a990 con 0x7fcb201f1cf0 2023-06-10T18:53:08.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.275+0000 7fcb1e59c700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 0x7fcb0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:08.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.276+0000 7fcb1e59c700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 0x7fcb0006d190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcb10000c00 tx=0x7fcb10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:08.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.280+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb140558f0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.512+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcb200c64e0 con 0x7fcb201f1cf0 2023-06-10T18:53:08.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.512+0000 7fcb0effd700 1 -- 172.21.15.104:0/3521322673 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcb14058f10 con 0x7fcb201f1cf0 2023-06-10T18:53:08.518 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:53:08.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 msgr2=0x7fcb0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 0x7fcb0006d190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcb10000c00 tx=0x7fcb10009040 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 msgr2=0x7fcb201550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c030 tx=0x7fcb14001a00 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 shutdown_connections 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb0006ace0 0x7fcb0006d190 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb200c5c20 0x7fcb200b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb08004850 0x7fcb20155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 --2- 172.21.15.104:0/3521322673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb201f1cf0 0x7fcb201550c0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:08.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 >> 172.21.15.104:0/3521322673 conn(0x7fcb2006b8a0 msgr2=0x7fcb200b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:08.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 shutdown_connections 2023-06-10T18:53:08.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:08.514+0000 7fcb24b57700 1 -- 172.21.15.104:0/3521322673 wait complete. 2023-06-10T18:53:08.531 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:53:08.532 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:53:08.532 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:53:08.532 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:53:08.533 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:08.533 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:53:08.533 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:53:08.534 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:53:08.534 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:08.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.535 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:53:08.556 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:53:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:53:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:53:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:08.557 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41891sec preferred_lft 41891sec 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:53:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:53:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:53:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:08.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.561 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:53:08.637 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:53:08.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.639 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:53:08.699 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:53:08.699 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:53:08.699 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:53:08.699 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:53:08.699 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:53:08.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.700 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:53:08.701 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:53:08.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:53:08.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:53:08.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:53:08.702 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:53:08.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.808 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:53:08.825 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:53:08.825 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:53:08.825 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:08.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:08.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:53:08.832 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:53:08.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:53:08.893 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:53:08.894 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:08.894 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:08.975 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:08.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:08.976 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:09.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:53:09.098 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:09.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:09.163 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:09.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.165 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:53:09.222 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:53:09.222 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:53:09.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:53:09.289 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:09.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:09.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:53:09.345+0000 7f2d45a445c0 -1 init, newargv = 0x557c1ff88c00 newargc=15 2023-06-10T18:53:09.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111958]: starting ceph client 2023-06-10T18:53:09.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111958]: starting fuse 2023-06-10T18:53:09.356 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:09.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.358 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:53:09.415 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:53:09.415 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:53:09.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:53:09.416 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:53:09.416 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:53:09.416 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:53:09.417 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:53:09.417 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:09.417 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:53:09.418 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:53:09.418 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:53:09.418 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:53:09.419 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:09.419 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:53:09.419 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:53:09.419 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:53:09.420 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:53:09.420 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:09.420 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:53:09.421 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:53:09.421 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:53:09.421 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:53:09.422 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:53:09.422 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:53:09.422 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:53:09.423 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:53:09.423 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:09.423 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:53:09.423 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:53:09.536 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.111958.asok 2023-06-10T18:53:09.719 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.111958.asok 2023-06-10T18:53:09.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.111958.asok status 2023-06-10T18:53:09.876 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:53:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:53:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:53:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "111958", 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 6862, 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6862 2023-06-10T18:53:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 422631595 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:53:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 422631595 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.6862 192.168.0.1:0/422631595", 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/422631595", 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:53:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 47, 2023-06-10T18:53:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 87, 2023-06-10T18:53:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:53:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:53:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:53:09.881 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:53:09.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.886 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:53:09.886 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:09.903 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:53:09.903 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:09.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:09.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:09.971 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec ls subdir/childfile) 2023-06-10T18:53:10.029 INFO:teuthology.orchestra.run.smithi114.stdout:subdir/childfile 2023-06-10T18:53:10.030 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:53:10.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:53:10.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:53:10.119 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:53:10.120 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:10.121 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:10.141 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:53:10.142 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:53:10.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:10.142 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:53:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-06-10T18:53:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:53:10.202 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:10.202 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:10.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.367+0000 7f6486bc3700 1 Processor -- start 2023-06-10T18:53:10.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.367+0000 7f6486bc3700 1 -- start start 2023-06-10T18:53:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.367+0000 7f6486bc3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6480158610 0x7f64800ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6486bc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 0x7f64800c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6486bc3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64801589e0 con 0x7f6480070a10 2023-06-10T18:53:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6486bc3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648006d0b0 con 0x7f6480158610 2023-06-10T18:53:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6486bc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648006d390 con 0x7f648015cb10 2023-06-10T18:53:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6480158610 0x7f64800ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f647ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 0x7f64800c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6485160700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6480070a10 0x7f64800cd550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37520/0 (socket says 172.21.15.104:37520) 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f6485160700 1 -- 172.21.15.104:0/1721568100 learned_addr learned my addr 172.21.15.104:0/1721568100 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 -- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 msgr2=0x7f64800c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 0x7f64800c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 -- 172.21.15.104:0/1721568100 >> v1:172.21.15.114:6789/0 conn(0x7f6480070a10 legacy=0x7f64800cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f647f7fe700 1 -- 172.21.15.104:0/1721568100 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379068724 0 0) 0x7f6478001490 con 0x7f6480070a10 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 -- 172.21.15.104:0/1721568100 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648006d670 con 0x7f6480158610 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f647ffff700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 0x7f64800c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.368+0000 7f648495f700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480158610 0x7f64800ce570 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6474009840 tx=0x7f6474009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2cf5abfcb6446b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f647f7fe700 1 -- 172.21.15.104:0/1721568100 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474011830 con 0x7f6480158610 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f647f7fe700 1 -- 172.21.15.104:0/1721568100 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474011990 con 0x7f6480158610 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f647f7fe700 1 -- 172.21.15.104:0/1721568100 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474011d20 con 0x7f6480158610 2023-06-10T18:53:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 -- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480158610 msgr2=0x7f64800ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480158610 0x7f64800ce570 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6474009840 tx=0x7f6474009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 -- 172.21.15.104:0/1721568100 shutdown_connections 2023-06-10T18:53:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480158610 0x7f64800ce570 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 --2- 172.21.15.104:0/1721568100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f648015cb10 0x7f64800c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 -- 172.21.15.104:0/1721568100 >> 172.21.15.104:0/1721568100 conn(0x7f64800ceec0 msgr2=0x7f64800cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:10.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 -- 172.21.15.104:0/1721568100 shutdown_connections 2023-06-10T18:53:10.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.369+0000 7f6486bc3700 1 -- 172.21.15.104:0/1721568100 wait complete. 2023-06-10T18:53:10.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.370+0000 7f6486bc3700 1 Processor -- start 2023-06-10T18:53:10.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.370+0000 7f6486bc3700 1 -- start start 2023-06-10T18:53:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 0x7f6480150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 0x7f64801448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648014e4b0 con 0x7f648015cb10 2023-06-10T18:53:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480150670 con 0x7f6480070a10 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6486bc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6480144df0 con 0x7f6480158610 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6485160700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.371+0000 7f6485160700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33852/0 (socket says 172.21.15.104:33852) 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 -- 172.21.15.104:0/2942010804 learned_addr learned my addr 172.21.15.104:0/2942010804 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f648495f700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 0x7f6480150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f647ffff700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 0x7f64801448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 -- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 msgr2=0x7f64801448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 0x7f64801448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 -- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 msgr2=0x7f6480150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 0x7f6480150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 -- 172.21.15.104:0/2942010804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0094a0 con 0x7f648015cb10 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f647ffff700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 0x7f64801448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f648495f700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 0x7f6480150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6485160700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f647800cca0 tx=0x7f647800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647800ab10 con 0x7f648015cb10 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800ac90 con 0x7f648015cb10 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740094a0 con 0x7f648015cb10 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64780070d0 con 0x7f648015cb10 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.372+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648014e9a0 con 0x7f648015cb10 2023-06-10T18:53:10.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.373+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64800cc790 con 0x7f648015cb10 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.374+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6478007250 con 0x7f648015cb10 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.374+0000 7f647d7fa700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 0x7f646406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.374+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f6478089250 con 0x7f648015cb10 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.374+0000 7f648495f700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 0x7f646406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.375+0000 7f648495f700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 0x7f646406d190 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6474009fd0 tx=0x7f6474008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.379+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f64780541b0 con 0x7f648015cb10 2023-06-10T18:53:10.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.565+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64801589e0 con 0x7f648015cb10 2023-06-10T18:53:10.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.566+0000 7f647d7fa700 1 -- 172.21.15.104:0/2942010804 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+18835 (secure 0 0 0) 0x7f64801589e0 con 0x7f648015cb10 2023-06-10T18:53:10.572 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:10.573 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":87,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:04.801983+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1823366359":"2023-06-10T19:53:04.656002+0000","172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 msgr2=0x7f646406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 0x7f646406d190 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6474009fd0 tx=0x7f6474008030 comp rx=0 tx=0).stop 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 msgr2=0x7f64801f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f647800cca0 tx=0x7f647800a060 comp rx=0 tx=0).stop 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 shutdown_connections 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f646406ace0 0x7f646406d190 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6480070a10 0x7f6480150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f648015cb10 0x7f64801f3030 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 --2- 172.21.15.104:0/2942010804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6480158610 0x7f64801448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 >> 172.21.15.104:0/2942010804 conn(0x7f64800ceec0 msgr2=0x7f64800c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:10.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 shutdown_connections 2023-06-10T18:53:10.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.569+0000 7f6486bc3700 1 -- 172.21.15.104:0/2942010804 wait complete. 2023-06-10T18:53:10.586 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:10.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.704+0000 7f5d2ae89700 1 Processor -- start 2023-06-10T18:53:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.704+0000 7f5d2ae89700 1 -- start start 2023-06-10T18:53:10.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d2ae89700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d240b1780 0x7f5d2406cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d2ae89700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5d2406ea70 0x7f5d2406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d2ae89700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240ae7e0 con 0x7f5d2415c6f0 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d2ae89700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2406df90 con 0x7f5d2406ea70 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d2ae89700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2406e270 con 0x7f5d240b1780 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d29426700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5d2406ea70 0x7f5d2406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d28c25700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5d2415c6f0 0x7f5d240b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59916/0 (socket says 172.21.15.104:59916) 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d28c25700 1 -- 172.21.15.104:0/1779045515 learned_addr learned my addr 172.21.15.104:0/1779045515 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d23fff700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d240b1780 0x7f5d2406cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d29426700 1 -- 172.21.15.104:0/1779045515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d240b1780 msgr2=0x7f5d2406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d29426700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d240b1780 0x7f5d2406cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d29426700 1 -- 172.21.15.104:0/1779045515 >> v1:172.21.15.104:6789/0 conn(0x7f5d2415c6f0 legacy=0x7f5d240b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.705+0000 7f5d29426700 1 -- 172.21.15.104:0/1779045515 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2406e550 con 0x7f5d2406ea70 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d29426700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2406ea70 0x7f5d2406d320 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009d90 tx=0x7f5d1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9e01e58ca86d93 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d237fe700 1 -- 172.21.15.104:0/1779045515 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00e070 con 0x7f5d2406ea70 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d237fe700 1 -- 172.21.15.104:0/1779045515 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c00bbd0 con 0x7f5d2406ea70 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d237fe700 1 -- 172.21.15.104:0/1779045515 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c012430 con 0x7f5d2406ea70 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 -- 172.21.15.104:0/1779045515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2406ea70 msgr2=0x7f5d2406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2406ea70 0x7f5d2406d320 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009d90 tx=0x7f5d1c00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 -- 172.21.15.104:0/1779045515 shutdown_connections 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2406ea70 0x7f5d2406d320 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/1779045515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d240b1780 0x7f5d2406cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 -- 172.21.15.104:0/1779045515 >> 172.21.15.104:0/1779045515 conn(0x7f5d2406b8a0 msgr2=0x7f5d240cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:10.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 -- 172.21.15.104:0/1779045515 shutdown_connections 2023-06-10T18:53:10.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.706+0000 7f5d2ae89700 1 -- 172.21.15.104:0/1779045515 wait complete. 2023-06-10T18:53:10.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.707+0000 7f5d2ae89700 1 Processor -- start 2023-06-10T18:53:10.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 -- start start 2023-06-10T18:53:10.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d2406ea70 0x7f5d24155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 0x7f5d240ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24156210 con 0x7f5d240b1780 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240b25a0 con 0x7f5d2415c6f0 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d2ae89700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240cacc0 con 0x7f5d2406ea70 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.708+0000 7f5d23fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33862/0 (socket says 172.21.15.104:33862) 2023-06-10T18:53:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 -- 172.21.15.104:0/535894397 learned_addr learned my addr 172.21.15.104:0/535894397 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d29426700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 0x7f5d240ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d28c25700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d2406ea70 0x7f5d24155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 -- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d2406ea70 msgr2=0x7f5d24155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d2406ea70 0x7f5d24155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 -- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 msgr2=0x7f5d240ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 0x7f5d240ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 -- 172.21.15.104:0/535894397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c009a70 con 0x7f5d240b1780 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d29426700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 0x7f5d240ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d23fff700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e810 tx=0x7f5d1000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000c070 con 0x7f5d240b1780 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d10013610 con 0x7f5d240b1780 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d240cafa0 con 0x7f5d240b1780 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d100139a0 con 0x7f5d240b1780 2023-06-10T18:53:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.709+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240b3cd0 con 0x7f5d240b1780 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.710+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240b3e60 con 0x7f5d240b1780 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.711+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d10013b00 con 0x7f5d240b1780 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.711+0000 7f5d217fa700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 0x7f5d0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.711+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f5d100890c0 con 0x7f5d240b1780 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.711+0000 7f5d28c25700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 0x7f5d0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.712+0000 7f5d28c25700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 0x7f5d0806d190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007fd0 tx=0x7f5d18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.715+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d10057a20 con 0x7f5d240b1780 2023-06-10T18:53:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.902+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d240cee50 con 0x7f5d240b1780 2023-06-10T18:53:10.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.902+0000 7f5d217fa700 1 -- 172.21.15.104:0/535894397 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d10013db0 con 0x7f5d240b1780 2023-06-10T18:53:10.909 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.904+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 msgr2=0x7f5d0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.904+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 0x7f5d0806d190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007fd0 tx=0x7f5d18009040 comp rx=0 tx=0).stop 2023-06-10T18:53:10.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.904+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 msgr2=0x7f5d240c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.904+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e810 tx=0x7f5d1000eb20 comp rx=0 tx=0).stop 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 shutdown_connections 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d0806ace0 0x7f5d0806d190 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d2415c6f0 0x7f5d240ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d2406ea70 0x7f5d24155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 --2- 172.21.15.104:0/535894397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d240b1780 0x7f5d240c4030 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 >> 172.21.15.104:0/535894397 conn(0x7f5d2406b8a0 msgr2=0x7f5d24159220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:10.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 shutdown_connections 2023-06-10T18:53:10.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:10.905+0000 7f5d2ae89700 1 -- 172.21.15.104:0/535894397 wait complete. 2023-06-10T18:53:10.922 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.039+0000 7f812c600700 1 Processor -- start 2023-06-10T18:53:11.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.039+0000 7f812c600700 1 -- start start 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812c600700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f812415c580 0x7f81240b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812c600700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f812406cde0 0x7f81240ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812c600700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812406e250 con 0x7f812406e6e0 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812c600700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812415c950 con 0x7f812406cde0 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812c600700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81240b1e80 con 0x7f812415c580 2023-06-10T18:53:11.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f812406cde0 0x7f81240ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f812406cde0 0x7f81240ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33874/0 (socket says 172.21.15.104:33874) 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 -- 172.21.15.104:0/3849162880 learned_addr learned my addr 172.21.15.104:0/3849162880 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812a39c700 1 --2- 172.21.15.104:0/3849162880 >> v2:172.21.15.114:3300/0 conn(0x7f812415c580 0x7f81240b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 -- 172.21.15.104:0/3849162880 >> v2:172.21.15.114:3300/0 conn(0x7f812415c580 msgr2=0x7f81240b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 --2- 172.21.15.104:0/3849162880 >> v2:172.21.15.114:3300/0 conn(0x7f812415c580 0x7f81240b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 -- 172.21.15.104:0/3849162880 >> v1:172.21.15.114:6789/0 conn(0x7f812406e6e0 legacy=0x7f81240b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 -- 172.21.15.104:0/3849162880 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812406d920 con 0x7f812406cde0 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812ab9d700 1 --2- 172.21.15.104:0/3849162880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240ce0d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f8124070300 tx=0x7f811c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1291a65166361cc2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812939a700 1 -- 172.21.15.104:0/3849162880 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c00e070 con 0x7f812406cde0 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812939a700 1 -- 172.21.15.104:0/3849162880 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c00b910 con 0x7f812406cde0 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.040+0000 7f812939a700 1 -- 172.21.15.104:0/3849162880 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c00bcc0 con 0x7f812406cde0 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 -- 172.21.15.104:0/3849162880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 msgr2=0x7f81240ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 --2- 172.21.15.104:0/3849162880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240ce0d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f8124070300 tx=0x7f811c00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 -- 172.21.15.104:0/3849162880 shutdown_connections 2023-06-10T18:53:11.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 --2- 172.21.15.104:0/3849162880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240ce0d0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 --2- 172.21.15.104:0/3849162880 >> v2:172.21.15.114:3300/0 conn(0x7f812415c580 0x7f81240b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 -- 172.21.15.104:0/3849162880 >> 172.21.15.104:0/3849162880 conn(0x7f812406b8a0 msgr2=0x7f81240cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 -- 172.21.15.104:0/3849162880 shutdown_connections 2023-06-10T18:53:11.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.041+0000 7f812c600700 1 -- 172.21.15.104:0/3849162880 wait complete. 2023-06-10T18:53:11.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.042+0000 7f812c600700 1 Processor -- start 2023-06-10T18:53:11.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.042+0000 7f812c600700 1 -- start start 2023-06-10T18:53:11.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 0x7f81240c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f812415c580 0x7f81240c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240c5fc0 con 0x7f812406cde0 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240c5720 con 0x7f812406e6e0 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812c600700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81240c5a00 con 0x7f812415c580 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33876/0 (socket says 172.21.15.104:33876) 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 -- 172.21.15.104:0/2316532370 learned_addr learned my addr 172.21.15.104:0/2316532370 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f8129b9b700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 0x7f81240c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812ab9d700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f812415c580 0x7f81240c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 -- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f812415c580 msgr2=0x7f81240c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f812415c580 0x7f81240c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 -- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 msgr2=0x7f81240c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 0x7f81240c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f812a39c700 1 -- 172.21.15.104:0/2316532370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c009b30 con 0x7f812406cde0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.043+0000 7f8129b9b700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 0x7f81240c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f812a39c700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f812000e900 tx=0x7f812000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120009650 con 0x7f812406cde0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812000c850 con 0x7f812406cde0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81240c5ce0 con 0x7f812406cde0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000cbc0 con 0x7f812406cde0 2023-06-10T18:53:11.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.044+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81241f61e0 con 0x7f812406cde0 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.045+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81240ac1c0 con 0x7f812406cde0 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.045+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f812000cd20 con 0x7f812406cde0 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.045+0000 7f811affd700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 0x7f810806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.045+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f8120089ad0 con 0x7f812406cde0 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.046+0000 7f8129b9b700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 0x7f810806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.046+0000 7f8129b9b700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 0x7f810806d1c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f81240b5420 tx=0x7f8114009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.050+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f81240ac1c0 con 0x7f812406cde0 2023-06-10T18:53:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.234+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f81240ac5a0 con 0x7f812406cde0 2023-06-10T18:53:11.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.235+0000 7f811affd700 1 -- 172.21.15.104:0/2316532370 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7f81240ac5a0 con 0x7f812406cde0 2023-06-10T18:53:11.241 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:11.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 msgr2=0x7f810806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 0x7f810806d1c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f81240b5420 tx=0x7f8114009040 comp rx=0 tx=0).stop 2023-06-10T18:53:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 msgr2=0x7f81240b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f812000e900 tx=0x7f812000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 shutdown_connections 2023-06-10T18:53:11.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f812415c580 0x7f81240c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f812406cde0 0x7f81240b3c90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f810806ad10 0x7f810806d1c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 --2- 172.21.15.104:0/2316532370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f812406e6e0 0x7f81240c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 >> 172.21.15.104:0/2316532370 conn(0x7f812406b8a0 msgr2=0x7f812415a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 shutdown_connections 2023-06-10T18:53:11.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.237+0000 7f812c600700 1 -- 172.21.15.104:0/2316532370 wait complete. 2023-06-10T18:53:11.254 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.371+0000 7fd8869cc700 1 Processor -- start 2023-06-10T18:53:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.371+0000 7fd8869cc700 1 -- start start 2023-06-10T18:53:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd8869cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd88015c580 0x7fd8800b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd8869cc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd88006cde0 0x7fd8800ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd8869cc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88006e250 con 0x7fd88006e6e0 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd8869cc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88015c950 con 0x7fd88015c580 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd8869cc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800b1e80 con 0x7fd88006cde0 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd88015c580 0x7fd8800b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd88015c580 0x7fd8800b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33884/0 (socket says 172.21.15.104:33884) 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 -- 172.21.15.104:0/1064982781 learned_addr learned my addr 172.21.15.104:0/1064982781 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd884f69700 1 --2- 172.21.15.104:0/1064982781 >> v2:172.21.15.114:3300/0 conn(0x7fd88006cde0 0x7fd8800ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 -- 172.21.15.104:0/1064982781 >> v2:172.21.15.114:3300/0 conn(0x7fd88006cde0 msgr2=0x7fd8800ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 --2- 172.21.15.104:0/1064982781 >> v2:172.21.15.114:3300/0 conn(0x7fd88006cde0 0x7fd8800ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 -- 172.21.15.104:0/1064982781 >> v1:172.21.15.114:6789/0 conn(0x7fd88006e6e0 legacy=0x7fd8800b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 -- 172.21.15.104:0/1064982781 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88006d920 con 0x7fd88015c580 2023-06-10T18:53:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87ffff700 1 --2- 172.21.15.104:0/1064982781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88015c580 0x7fd8800b1070 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd8800cbc60 tx=0x7fd870009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecbfda8c9c635b0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87effd700 1 -- 172.21.15.104:0/1064982781 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870005e30 con 0x7fd88015c580 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87effd700 1 -- 172.21.15.104:0/1064982781 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870004030 con 0x7fd88015c580 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.372+0000 7fd87effd700 1 -- 172.21.15.104:0/1064982781 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700043e0 con 0x7fd88015c580 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 -- 172.21.15.104:0/1064982781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88015c580 msgr2=0x7fd8800b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 --2- 172.21.15.104:0/1064982781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88015c580 0x7fd8800b1070 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd8800cbc60 tx=0x7fd870009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 -- 172.21.15.104:0/1064982781 shutdown_connections 2023-06-10T18:53:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 --2- 172.21.15.104:0/1064982781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88015c580 0x7fd8800b1070 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 --2- 172.21.15.104:0/1064982781 >> v2:172.21.15.114:3300/0 conn(0x7fd88006cde0 0x7fd8800ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 -- 172.21.15.104:0/1064982781 >> 172.21.15.104:0/1064982781 conn(0x7fd88006b8a0 msgr2=0x7fd8800cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 -- 172.21.15.104:0/1064982781 shutdown_connections 2023-06-10T18:53:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.373+0000 7fd8869cc700 1 -- 172.21.15.104:0/1064982781 wait complete. 2023-06-10T18:53:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.374+0000 7fd8869cc700 1 Processor -- start 2023-06-10T18:53:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.374+0000 7fd8869cc700 1 -- start start 2023-06-10T18:53:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88006cde0 0x7fd8800c3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd88015c580 0x7fd8800c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b24c0 con 0x7fd88006e6e0 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b27a0 con 0x7fd88006cde0 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd8869cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b2a80 con 0x7fd88015c580 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33892/0 (socket says 172.21.15.104:33892) 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 -- 172.21.15.104:0/220872010 learned_addr learned my addr 172.21.15.104:0/220872010 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd884f69700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd88015c580 0x7fd8800c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87ffff700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88006cde0 0x7fd8800c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 -- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd88015c580 msgr2=0x7fd8800c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd88015c580 0x7fd8800c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 -- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88006cde0 msgr2=0x7fd8800c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88006cde0 0x7fd8800c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.375+0000 7fd87f7fe700 1 -- 172.21.15.104:0/220872010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870009500 con 0x7fd88006e6e0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd87f7fe700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fd87400c4b0 tx=0x7fd87400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740052d0 con 0x7fd88006e6e0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd874005430 con 0x7fd88006e6e0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800b2d60 con 0x7fd88006e6e0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740038c0 con 0x7fd88006e6e0 2023-06-10T18:53:11.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.376+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800c86c0 con 0x7fd88006e6e0 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.377+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800b3280 con 0x7fd88006e6e0 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.377+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd874003a20 con 0x7fd88006e6e0 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.378+0000 7fd87cff9700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 0x7fd86c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.378+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fd874088b40 con 0x7fd88006e6e0 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.378+0000 7fd87ffff700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 0x7fd86c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.378+0000 7fd87ffff700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 0x7fd86c06d190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd8800cbc60 tx=0x7fd870008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.382+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd8800b3280 con 0x7fd88006e6e0 2023-06-10T18:53:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.569+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8800b3280 con 0x7fd88006e6e0 2023-06-10T18:53:11.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.569+0000 7fd87cff9700 1 -- 172.21.15.104:0/220872010 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8800b3280 con 0x7fd88006e6e0 2023-06-10T18:53:11.575 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 msgr2=0x7fd86c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 0x7fd86c06d190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd8800cbc60 tx=0x7fd870008070 comp rx=0 tx=0).stop 2023-06-10T18:53:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 msgr2=0x7fd880158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fd87400c4b0 tx=0x7fd87400c870 comp rx=0 tx=0).stop 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 shutdown_connections 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd86c06ace0 0x7fd86c06d190 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88006cde0 0x7fd8800c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd88015c580 0x7fd8800c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 --2- 172.21.15.104:0/220872010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd88006e6e0 0x7fd880158c20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 >> 172.21.15.104:0/220872010 conn(0x7fd88006b8a0 msgr2=0x7fd88015b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 shutdown_connections 2023-06-10T18:53:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.571+0000 7fd8869cc700 1 -- 172.21.15.104:0/220872010 wait complete. 2023-06-10T18:53:11.589 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:11.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.706+0000 7fbc96433700 1 Processor -- start 2023-06-10T18:53:11.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.706+0000 7fbc96433700 1 -- start start 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.706+0000 7fbc96433700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9006af80 0x7fbc900d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc96433700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc900ccff0 0x7fbc9006b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc96433700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc900ce340 con 0x7fbc9015fd00 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc96433700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc900cd460 con 0x7fbc900ccff0 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc96433700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900d00d0 con 0x7fbc9006af80 2023-06-10T18:53:11.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc900ccff0 0x7fbc9006b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc900ccff0 0x7fbc9006b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33898/0 (socket says 172.21.15.104:33898) 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 -- 172.21.15.104:0/2662002069 learned_addr learned my addr 172.21.15.104:0/2662002069 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8ffff700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9006af80 0x7fbc900d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 -- 172.21.15.104:0/2662002069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9006af80 msgr2=0x7fbc900d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9006af80 0x7fbc900d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 -- 172.21.15.104:0/2662002069 >> v1:172.21.15.114:6789/0 conn(0x7fbc9015fd00 legacy=0x7fbc90160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 -- 172.21.15.104:0/2662002069 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9006b890 con 0x7fbc900ccff0 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8f7fe700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc900ccff0 0x7fbc9006b350 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009920 tx=0x7fbc84009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47da028ad452ebd5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8effd700 1 -- 172.21.15.104:0/2662002069 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84011830 con 0x7fbc900ccff0 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8effd700 1 -- 172.21.15.104:0/2662002069 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc84011990 con 0x7fbc900ccff0 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.707+0000 7fbc8effd700 1 -- 172.21.15.104:0/2662002069 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84011d20 con 0x7fbc900ccff0 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 -- 172.21.15.104:0/2662002069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc900ccff0 msgr2=0x7fbc9006b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc900ccff0 0x7fbc9006b350 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009920 tx=0x7fbc84009c30 comp rx=0 tx=0).stop 2023-06-10T18:53:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 -- 172.21.15.104:0/2662002069 shutdown_connections 2023-06-10T18:53:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9006af80 0x7fbc900d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 --2- 172.21.15.104:0/2662002069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc900ccff0 0x7fbc9006b350 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 -- 172.21.15.104:0/2662002069 >> 172.21.15.104:0/2662002069 conn(0x7fbc90089fd0 msgr2=0x7fbc9008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 -- 172.21.15.104:0/2662002069 shutdown_connections 2023-06-10T18:53:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.708+0000 7fbc96433700 1 -- 172.21.15.104:0/2662002069 wait complete. 2023-06-10T18:53:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.709+0000 7fbc96433700 1 Processor -- start 2023-06-10T18:53:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.710+0000 7fbc96433700 1 -- start start 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.710+0000 7fbc96433700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900ccff0 0x7fbc90150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9015fd00 0x7fbc900b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900c6280 con 0x7fbc9006af80 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900b15c0 con 0x7fbc900ccff0 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900b12e0 con 0x7fbc9015fd00 2023-06-10T18:53:11.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33904/0 (socket says 172.21.15.104:33904) 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 -- 172.21.15.104:0/1592043061 learned_addr learned my addr 172.21.15.104:0/1592043061 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8f7fe700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900ccff0 0x7fbc90150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc949d0700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9015fd00 0x7fbc900b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 -- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9015fd00 msgr2=0x7fbc900b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9015fd00 0x7fbc900b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 -- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900ccff0 msgr2=0x7fbc90150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900ccff0 0x7fbc90150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 -- 172.21.15.104:0/1592043061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc84009550 con 0x7fbc9006af80 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8ffff700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e9b0 tx=0x7fbc8000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80009710 con 0x7fbc9006af80 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8000ee50 con 0x7fbc9006af80 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc900c5c30 con 0x7fbc9006af80 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8000ca50 con 0x7fbc9006af80 2023-06-10T18:53:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.711+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc900c5740 con 0x7fbc9006af80 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.712+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc900c6560 con 0x7fbc9006af80 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.713+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbc8000cbb0 con 0x7fbc9006af80 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.713+0000 7fbc8cff9700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 0x7fbc7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.713+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fbc80089630 con 0x7fbc9006af80 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.713+0000 7fbc8f7fe700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 0x7fbc7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.714+0000 7fbc8f7fe700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 0x7fbc7c06d190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009840 tx=0x7fbc84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.717+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbc80057fa0 con 0x7fbc9006af80 2023-06-10T18:53:11.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.906+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc9006c600 con 0x7fbc9006af80 2023-06-10T18:53:11.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.906+0000 7fbc8cff9700 1 -- 172.21.15.104:0/1592043061 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc8000ce60 con 0x7fbc9006af80 2023-06-10T18:53:11.912 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:11.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 msgr2=0x7fbc7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 0x7fbc7c06d190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009840 tx=0x7fbc84008030 comp rx=0 tx=0).stop 2023-06-10T18:53:11.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 msgr2=0x7fbc901514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:11.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e9b0 tx=0x7fbc8000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 shutdown_connections 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc9015fd00 0x7fbc900b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc9006af80 0x7fbc901514c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc7c06ace0 0x7fbc7c06d190 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 --2- 172.21.15.104:0/1592043061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900ccff0 0x7fbc90150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 >> 172.21.15.104:0/1592043061 conn(0x7fbc90089fd0 msgr2=0x7fbc901566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:11.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 shutdown_connections 2023-06-10T18:53:11.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:11.908+0000 7fbc96433700 1 -- 172.21.15.104:0/1592043061 wait complete. 2023-06-10T18:53:11.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:53:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.172+0000 7ff286d39700 1 Processor -- start 2023-06-10T18:53:12.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.172+0000 7ff286d39700 1 -- start start 2023-06-10T18:53:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff286d39700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff28006e0e0 0x7ff28006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff286d39700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 0x7ff2800c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff286d39700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28006d290 con 0x7ff2800ca100 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff286d39700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff280071640 con 0x7ff28006e0e0 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff286d39700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800aef00 con 0x7ff28006e4b0 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff28006e0e0 0x7ff28006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff28006e0e0 0x7ff28006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33912/0 (socket says 172.21.15.104:33912) 2023-06-10T18:53:12.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 -- 172.21.15.104:0/177432349 learned_addr learned my addr 172.21.15.104:0/177432349 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 -- 172.21.15.104:0/177432349 >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 msgr2=0x7ff2800c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff27ffff700 1 --2- 172.21.15.104:0/177432349 >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 0x7ff2800c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 --2- 172.21.15.104:0/177432349 >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 0x7ff2800c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 -- 172.21.15.104:0/177432349 >> v1:172.21.15.114:6789/0 conn(0x7ff2800ca100 legacy=0x7ff2800ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff284ad5700 1 -- 172.21.15.104:0/177432349 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28006d4c0 con 0x7ff28006e0e0 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.173+0000 7ff27ffff700 1 --2- 172.21.15.104:0/177432349 >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 0x7ff2800c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff284ad5700 1 --2- 172.21.15.104:0/177432349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e0e0 0x7ff28006cd50 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff280070320 tx=0x7ff274009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db10f38aff9f82e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff27f7fe700 1 -- 172.21.15.104:0/177432349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274005e30 con 0x7ff28006e0e0 2023-06-10T18:53:12.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff27f7fe700 1 -- 172.21.15.104:0/177432349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff274006030 con 0x7ff28006e0e0 2023-06-10T18:53:12.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff27f7fe700 1 -- 172.21.15.104:0/177432349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2740063c0 con 0x7ff28006e0e0 2023-06-10T18:53:12.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 -- 172.21.15.104:0/177432349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e0e0 msgr2=0x7ff28006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 --2- 172.21.15.104:0/177432349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e0e0 0x7ff28006cd50 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff280070320 tx=0x7ff274009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 -- 172.21.15.104:0/177432349 shutdown_connections 2023-06-10T18:53:12.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 --2- 172.21.15.104:0/177432349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e0e0 0x7ff28006cd50 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 --2- 172.21.15.104:0/177432349 >> v2:172.21.15.114:3300/0 conn(0x7ff28006e4b0 0x7ff2800c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 -- 172.21.15.104:0/177432349 >> 172.21.15.104:0/177432349 conn(0x7ff28006b8a0 msgr2=0x7ff2800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 -- 172.21.15.104:0/177432349 shutdown_connections 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.174+0000 7ff286d39700 1 -- 172.21.15.104:0/177432349 wait complete. 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.175+0000 7ff286d39700 1 Processor -- start 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 -- start start 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff28006e0e0 0x7ff28015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 0x7ff2800adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c8240 con 0x7ff28006e4b0 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c9510 con 0x7ff2800ca100 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.176+0000 7ff286d39700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c8470 con 0x7ff28006e0e0 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33920/0 (socket says 172.21.15.104:33920) 2023-06-10T18:53:12.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 -- 172.21.15.104:0/1552106562 learned_addr learned my addr 172.21.15.104:0/1552106562 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff2852d6700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 0x7ff2800adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff284ad5700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff28006e0e0 0x7ff28015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 -- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff28006e0e0 msgr2=0x7ff28015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff28006e0e0 0x7ff28015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 -- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 msgr2=0x7ff2800adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 0x7ff2800adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 -- 172.21.15.104:0/1552106562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff274009500 con 0x7ff28006e4b0 2023-06-10T18:53:12.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff2852d6700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 0x7ff2800adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27ffff700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c750 tx=0x7ff26c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c011800 con 0x7ff28006e4b0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c011960 con 0x7ff28006e4b0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2800c8750 con 0x7ff28006e4b0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c011cd0 con 0x7ff28006e4b0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.177+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2800c8d10 con 0x7ff28006e4b0 2023-06-10T18:53:12.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.178+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280074050 con 0x7ff28006e4b0 2023-06-10T18:53:12.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.179+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff26c021350 con 0x7ff28006e4b0 2023-06-10T18:53:12.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.179+0000 7ff27d7fa700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 0x7ff26406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.179+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7ff26c089640 con 0x7ff28006e4b0 2023-06-10T18:53:12.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.179+0000 7ff284ad5700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 0x7ff26406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.180+0000 7ff284ad5700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 0x7ff26406d190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff274000c00 tx=0x7ff274008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.183+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff26c057fa0 con 0x7ff28006e4b0 2023-06-10T18:53:12.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.384+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff2800cb380 con 0x7ff28006e4b0 2023-06-10T18:53:12.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.385+0000 7ff27d7fa700 1 -- 172.21.15.104:0/1552106562 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 3 entries v87) v1 ==== 80+0+179 (secure 0 0 0) 0x7ff26c057b30 con 0x7ff28006e4b0 2023-06-10T18:53:12.390 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/1823366359 2023-06-10T19:53:04.656002+0000 2023-06-10T18:53:12.391 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6836/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:53:12.391 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6834/414069859 2023-06-11T18:52:09.042261+0000 2023-06-10T18:53:12.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.386+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 msgr2=0x7ff26406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.386+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 0x7ff26406d190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff274000c00 tx=0x7ff274008070 comp rx=0 tx=0).stop 2023-06-10T18:53:12.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.386+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 msgr2=0x7ff2800c6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.386+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c750 tx=0x7ff26c00cb10 comp rx=0 tx=0).stop 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 shutdown_connections 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26406ace0 0x7ff26406d190 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800ca100 0x7ff2800adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff28006e0e0 0x7ff28015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 --2- 172.21.15.104:0/1552106562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff28006e4b0 0x7ff2800c6640 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 >> 172.21.15.104:0/1552106562 conn(0x7ff28006b8a0 msgr2=0x7ff2800ace10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:12.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 shutdown_connections 2023-06-10T18:53:12.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.387+0000 7ff286d39700 1 -- 172.21.15.104:0/1552106562 wait complete. 2023-06-10T18:53:12.394 INFO:teuthology.orchestra.run.smithi104.stderr:listed 3 entries 2023-06-10T18:53:12.404 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:53:12.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:12.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:12.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[111958]: fuse finished with error 0 and tester_r 0 2023-06-10T18:53:12.452 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:53:12.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:12.452 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:53:12.544 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:12.545 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:12.562 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:53:12.562 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:53:12.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:12.563 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:53:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:53:12.619 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:53:12.619 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:12.619 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:12.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:53:12.639 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:53:12.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:12.640 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:53:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:53:12.699 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:53:12.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.813+0000 7f86f6c78700 1 Processor -- start 2023-06-10T18:53:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.813+0000 7f86f6c78700 1 -- start start 2023-06-10T18:53:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f6c78700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cfa00 0x7f86f00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f6c78700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f86f015a250 0x7f86f00cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f6c78700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f00ccca0 con 0x7f86f00cad90 2023-06-10T18:53:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f6c78700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f00d0310 con 0x7f86f015a250 2023-06-10T18:53:12.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f6c78700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00cee50 con 0x7f86f00cfa00 2023-06-10T18:53:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86effff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f86f015a250 0x7f86f00cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f5215700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f86f00cad90 0x7f86f00b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59990/0 (socket says 172.21.15.104:59990) 2023-06-10T18:53:12.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.814+0000 7f86f5215700 1 -- 172.21.15.104:0/3375448510 learned_addr learned my addr 172.21.15.104:0/3375448510 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86effff700 1 -- 172.21.15.104:0/3375448510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cfa00 msgr2=0x7f86f00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86effff700 1 --2- 172.21.15.104:0/3375448510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cfa00 0x7f86f00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86effff700 1 -- 172.21.15.104:0/3375448510 >> v1:172.21.15.104:6789/0 conn(0x7f86f00cad90 legacy=0x7f86f00b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86effff700 1 -- 172.21.15.104:0/3375448510 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f00cf130 con 0x7f86f015a250 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86ef7fe700 1 -- 172.21.15.104:0/3375448510 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550154460 0 0) 0x7f86f00ccca0 con 0x7f86f00cad90 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86effff700 1 --2- 172.21.15.104:0/3375448510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f00cfdd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009840 tx=0x7f86dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79832421bb679cb7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86ef7fe700 1 -- 172.21.15.104:0/3375448510 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc011830 con 0x7f86f015a250 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86ef7fe700 1 -- 172.21.15.104:0/3375448510 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc011990 con 0x7f86f015a250 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86ef7fe700 1 -- 172.21.15.104:0/3375448510 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc011d20 con 0x7f86f015a250 2023-06-10T18:53:12.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 -- 172.21.15.104:0/3375448510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 msgr2=0x7f86f00cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 --2- 172.21.15.104:0/3375448510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f00cfdd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009840 tx=0x7f86dc009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 -- 172.21.15.104:0/3375448510 shutdown_connections 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 --2- 172.21.15.104:0/3375448510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cfa00 0x7f86f00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 --2- 172.21.15.104:0/3375448510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f00cfdd0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 -- 172.21.15.104:0/3375448510 >> 172.21.15.104:0/3375448510 conn(0x7f86f006e370 msgr2=0x7f86f006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 -- 172.21.15.104:0/3375448510 shutdown_connections 2023-06-10T18:53:12.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.815+0000 7f86f6c78700 1 -- 172.21.15.104:0/3375448510 wait complete. 2023-06-10T18:53:12.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 Processor -- start 2023-06-10T18:53:12.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 -- start start 2023-06-10T18:53:12.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cad90 0x7f86f01f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 0x7f86f00c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f01f3f30 con 0x7f86f015a250 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.817+0000 7f86f6c78700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0153b00 con 0x7f86f00cfa00 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f6c78700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00c5ec0 con 0x7f86f00cad90 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33936/0 (socket says 172.21.15.104:33936) 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 -- 172.21.15.104:0/1850292711 learned_addr learned my addr 172.21.15.104:0/1850292711 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86effff700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 0x7f86f00c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f4a14700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cad90 0x7f86f01f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 -- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cad90 msgr2=0x7f86f01f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cad90 0x7f86f01f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 -- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 msgr2=0x7f86f00c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 0x7f86f00c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 -- 172.21.15.104:0/1850292711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0094a0 con 0x7f86f015a250 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86effff700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 0x7f86f00c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:12.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f5215700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f86e800cbd0 tx=0x7f86e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e800ab70 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e800acf0 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f00c5630 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e8007310 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.818+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f00b1570 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.819+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f00b1790 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.820+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f86e8007470 con 0x7f86f015a250 2023-06-10T18:53:12.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.820+0000 7f86ed7fa700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 0x7f86d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:12.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.820+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f86e8089430 con 0x7f86f015a250 2023-06-10T18:53:12.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.820+0000 7f86f4a14700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 0x7f86d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:12.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.821+0000 7f86f4a14700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 0x7f86d406d190 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:12.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:12.824+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f86e8054390 con 0x7f86f015a250 2023-06-10T18:53:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.059+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86f00c5990 con 0x7f86f015a250 2023-06-10T18:53:13.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.059+0000 7f86ed7fa700 1 -- 172.21.15.104:0/1850292711 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4687 (secure 0 0 0) 0x7f86e800ae60 con 0x7f86f015a250 2023-06-10T18:53:13.065 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:13.065 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:06.805017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":17,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 msgr2=0x7f86d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 0x7f86d406d190 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4008040 comp rx=0 tx=0).stop 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 msgr2=0x7f86f0151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f86e800cbd0 tx=0x7f86e800cf90 comp rx=0 tx=0).stop 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 shutdown_connections 2023-06-10T18:53:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86f015a250 0x7f86f0151c10 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f86f00cad90 0x7f86f01f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f86d406ace0 0x7f86d406d190 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 --2- 172.21.15.104:0/1850292711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86f00cfa00 0x7f86f00c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 >> 172.21.15.104:0/1850292711 conn(0x7f86f006e370 msgr2=0x7f86f00cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 shutdown_connections 2023-06-10T18:53:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.061+0000 7f86f6c78700 1 -- 172.21.15.104:0/1850292711 wait complete. 2023-06-10T18:53:13.069 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 47 2023-06-10T18:53:13.078 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:13.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.192+0000 7f233272e700 1 Processor -- start 2023-06-10T18:53:13.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.192+0000 7f233272e700 1 -- start start 2023-06-10T18:53:13.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f233272e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 0x7f232c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f233272e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f233272e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c06bfd0 con 0x7f232c0d0a30 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f233272e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232c0d0ea0 con 0x7f232c06b9f0 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f233272e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c06c2b0 con 0x7f232c0b1e30 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f2330ccb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f232b7fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f232c0d0a30 0x7f232c06b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37614/0 (socket says 172.21.15.104:37614) 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f232bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 0x7f232c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f2330ccb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:49678/0 (socket says 172.21.15.104:49678) 2023-06-10T18:53:13.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.193+0000 7f232b7fe700 1 -- 172.21.15.104:0/2130981589 learned_addr learned my addr 172.21.15.104:0/2130981589 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:13.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232bfff700 1 -- 172.21.15.104:0/2130981589 >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 msgr2=0x7f232c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232bfff700 1 --2- 172.21.15.104:0/2130981589 >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232bfff700 1 -- 172.21.15.104:0/2130981589 >> v1:172.21.15.114:6789/0 conn(0x7f232c0d0a30 legacy=0x7f232c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232bfff700 1 -- 172.21.15.104:0/2130981589 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0ccf70 con 0x7f232c0b1e30 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f2330ccb700 1 --2- 172.21.15.104:0/2130981589 >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232bfff700 1 --2- 172.21.15.104:0/2130981589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 0x7f232c0d04f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2320009840 tx=0x7f2320009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a40d800ed4712528 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232affd700 1 -- 172.21.15.104:0/2130981589 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320011830 con 0x7f232c0b1e30 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232affd700 1 -- 172.21.15.104:0/2130981589 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320011990 con 0x7f232c0b1e30 2023-06-10T18:53:13.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.194+0000 7f232affd700 1 -- 172.21.15.104:0/2130981589 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320011d20 con 0x7f232c0b1e30 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 -- 172.21.15.104:0/2130981589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 msgr2=0x7f232c0d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 --2- 172.21.15.104:0/2130981589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 0x7f232c0d04f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2320009840 tx=0x7f2320009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 -- 172.21.15.104:0/2130981589 shutdown_connections 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 --2- 172.21.15.104:0/2130981589 >> v2:172.21.15.114:3300/0 conn(0x7f232c06b9f0 0x7f232c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 --2- 172.21.15.104:0/2130981589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0b1e30 0x7f232c0d04f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 -- 172.21.15.104:0/2130981589 >> 172.21.15.104:0/2130981589 conn(0x7f232c071ec0 msgr2=0x7f232c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 -- 172.21.15.104:0/2130981589 shutdown_connections 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.195+0000 7f233272e700 1 -- 172.21.15.104:0/2130981589 wait complete. 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.196+0000 7f233272e700 1 Processor -- start 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.196+0000 7f233272e700 1 -- start start 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c0b1e30 0x7f232c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0d0a30 0x7f232c0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0c6e60 con 0x7f232c06b9f0 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0caa40 con 0x7f232c0b1e30 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f233272e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0c94a0 con 0x7f232c0d0a30 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33946/0 (socket says 172.21.15.104:33946) 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 -- 172.21.15.104:0/3466891041 learned_addr learned my addr 172.21.15.104:0/3466891041 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232b7fe700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c0b1e30 0x7f232c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f2330ccb700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0d0a30 0x7f232c0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 -- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0d0a30 msgr2=0x7f232c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0d0a30 0x7f232c0c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 -- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c0b1e30 msgr2=0x7f232c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c0b1e30 0x7f232c0c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 -- 172.21.15.104:0/3466891041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2324009a70 con 0x7f232c06b9f0 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f232bfff700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2320009fd0 tx=0x7f2320004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200045d0 con 0x7f232c06b9f0 2023-06-10T18:53:13.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.197+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232001f070 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.198+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23200094a0 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.198+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200038c0 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.198+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c0c7390 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.198+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232c0d1860 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.199+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2320003a20 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.199+0000 7f2328ff9700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 0x7f231806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.199+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f2320021070 con 0x7f232c06b9f0 2023-06-10T18:53:13.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.199+0000 7f232b7fe700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 0x7f231806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.200+0000 7f232b7fe700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 0x7f231806d190 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f232c0b5c90 tx=0x7f231c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.203+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2320055d90 con 0x7f232c06b9f0 2023-06-10T18:53:13.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.392+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f232c074050 con 0x7f232c06b9f0 2023-06-10T18:53:13.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.393+0000 7f2328ff9700 1 -- 172.21.15.104:0/3466891041 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+18835 (secure 0 0 0) 0x7f2320015090 con 0x7f232c06b9f0 2023-06-10T18:53:13.399 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:13.399 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":87,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:04.801983+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1823366359":"2023-06-10T19:53:04.656002+0000","172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 msgr2=0x7f231806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 0x7f231806d190 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f232c0b5c90 tx=0x7f231c009040 comp rx=0 tx=0).stop 2023-06-10T18:53:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 msgr2=0x7f232c0b64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2320009fd0 tx=0x7f2320004750 comp rx=0 tx=0).stop 2023-06-10T18:53:13.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 shutdown_connections 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232c0d0a30 0x7f232c0c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f232c06b9f0 0x7f232c0b64b0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f231806ace0 0x7f231806d190 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 --2- 172.21.15.104:0/3466891041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f232c0b1e30 0x7f232c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 >> 172.21.15.104:0/3466891041 conn(0x7f232c071ec0 msgr2=0x7f232c06b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 shutdown_connections 2023-06-10T18:53:13.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.395+0000 7f233272e700 1 -- 172.21.15.104:0/3466891041 wait complete. 2023-06-10T18:53:13.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:51:46.185506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:51:47.337720+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:13.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.530+0000 7f7afd21e700 1 Processor -- start 2023-06-10T18:53:13.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.530+0000 7f7afd21e700 1 -- start start 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7afd21e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7af806e3e0 0x7f7af806a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7afd21e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af815c8d0 con 0x7f7af815c500 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7afd21e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af806abd0 con 0x7f7af8069e70 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7afd21e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af80aef20 con 0x7f7af806e3e0 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af6d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7af815c500 0x7f7af80cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60014/0 (socket says 172.21.15.104:60014) 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af6d9d700 1 -- 172.21.15.104:0/2755919789 learned_addr learned my addr 172.21.15.104:0/2755919789 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:13.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af759e700 1 --2- 172.21.15.104:0/2755919789 >> v2:172.21.15.114:3300/0 conn(0x7f7af806e3e0 0x7f7af806a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387011803 0 0) 0x7f7af815c8d0 con 0x7f7af815c500 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae00035b0 con 0x7f7af815c500 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416972709 0 0) 0x7f7ae00035b0 con 0x7f7af815c500 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 >> v2:172.21.15.114:3300/0 conn(0x7f7af806e3e0 msgr2=0x7f7af806a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 --2- 172.21.15.104:0/2755919789 >> v2:172.21.15.114:3300/0 conn(0x7f7af806e3e0 0x7f7af806a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 >> v1:172.21.15.114:6789/0 conn(0x7f7af8069e70 legacy=0x7f7af806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af806a8f0 con 0x7f7af815c500 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.531+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7aec0017f0 con 0x7f7af815c500 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 >> v1:172.21.15.104:6789/0 conn(0x7f7af815c500 legacy=0x7f7af80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7ae0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ae0004850 0x7f7ae0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ae0005200 0x7f7ae000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae000c050 con 0x7f7ae0003a00 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae000c1c0 con 0x7f7ae0005200 2023-06-10T18:53:13.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae000c380 con 0x7f7ae0004850 2023-06-10T18:53:13.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af759e700 1 --2- 172.21.15.104:0/2755919789 >> v2:172.21.15.114:3300/0 conn(0x7f7af806e3e0 0x7f7af806a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7ae0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af659c700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ae0004850 0x7f7ae0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af759e700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ae0005200 0x7f7ae000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 -- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ae0004850 msgr2=0x7f7ae0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ae0004850 0x7f7ae0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 -- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ae0005200 msgr2=0x7f7ae000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ae0005200 0x7f7ae000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 -- 172.21.15.104:0/2755919789 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00035b0 con 0x7f7ae0003a00 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af759e700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ae0005200 0x7f7ae000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:13.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af759e700 1 -- 172.21.15.104:0/2755919789 reap_dead start 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7ae0003e10 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c240 tx=0x7f7ae0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cfbb3ae2dbfe3e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00cde0 con 0x7f7ae0003a00 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec007b80 con 0x7f7ae0003a00 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.532+0000 7f7af5d9b700 1 -- 172.21.15.104:0/2755919789 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0072f0 con 0x7f7ae0003a00 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 -- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 msgr2=0x7f7ae0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7ae0003e10 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c240 tx=0x7f7ae0005880 comp rx=0 tx=0).stop 2023-06-10T18:53:13.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 -- 172.21.15.104:0/2755919789 shutdown_connections 2023-06-10T18:53:13.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 --2- 172.21.15.104:0/2755919789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7ae0003e10 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 -- 172.21.15.104:0/2755919789 >> 172.21.15.104:0/2755919789 conn(0x7f7af806b8a0 msgr2=0x7f7af80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 -- 172.21.15.104:0/2755919789 shutdown_connections 2023-06-10T18:53:13.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.533+0000 7f7afd21e700 1 -- 172.21.15.104:0/2755919789 wait complete. 2023-06-10T18:53:13.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.535+0000 7f7afd21e700 1 Processor -- start 2023-06-10T18:53:13.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.535+0000 7f7afd21e700 1 -- start start 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7af80c3140 0x7f7af80c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7af8159000 0x7f7af80b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80c9710 con 0x7f7ae0003a00 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af815dc90 con 0x7f7af8159000 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af815df70 con 0x7f7af80c3140 2023-06-10T18:53:13.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33966/0 (socket says 172.21.15.104:33966) 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 -- 172.21.15.104:0/965345079 learned_addr learned my addr 172.21.15.104:0/965345079 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af759e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7af8159000 0x7f7af80b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af659c700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7af80c3140 0x7f7af80c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 -- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7af80c3140 msgr2=0x7f7af80c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7af80c3140 0x7f7af80c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 -- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7af8159000 msgr2=0x7f7af80b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7af8159000 0x7f7af80b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 -- 172.21.15.104:0/965345079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af815e250 con 0x7f7ae0003a00 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7af6d9d700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f7aec000c00 tx=0x7f7aec001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0072e0 con 0x7f7ae0003a00 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec007440 con 0x7f7ae0003a00 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec005830 con 0x7f7ae0003a00 2023-06-10T18:53:13.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af815e530 con 0x7f7ae0003a00 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.536+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af8159470 con 0x7f7ae0003a00 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.537+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af815e770 con 0x7f7ae0003a00 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.538+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7aec005990 con 0x7f7ae0003a00 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.538+0000 7f7adf7fe700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 0x7f7ae406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.539+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f7aec08a090 con 0x7f7ae0003a00 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.539+0000 7f7af659c700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 0x7f7ae406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.539+0000 7f7af659c700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 0x7f7ae406d1c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7af80c6c50 tx=0x7f7ae8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.543+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7af815e770 con 0x7f7ae0003a00 2023-06-10T18:53:13.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.728+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7af815e770 con 0x7f7ae0003a00 2023-06-10T18:53:13.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.729+0000 7f7adf7fe700 1 -- 172.21.15.104:0/965345079 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7af815e770 con 0x7f7ae0003a00 2023-06-10T18:53:13.734 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 msgr2=0x7f7ae406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 0x7f7ae406d1c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7af80c6c50 tx=0x7f7ae8009040 comp rx=0 tx=0).stop 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 msgr2=0x7f7af8159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f7aec000c00 tx=0x7f7aec001b80 comp rx=0 tx=0).stop 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 shutdown_connections 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ae406ad10 0x7f7ae406d1c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7af8159000 0x7f7af80b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ae0003a00 0x7f7af8159850 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 --2- 172.21.15.104:0/965345079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7af80c3140 0x7f7af80c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 >> 172.21.15.104:0/965345079 conn(0x7f7af806b8a0 msgr2=0x7f7af80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 shutdown_connections 2023-06-10T18:53:13.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.731+0000 7f7afd21e700 1 -- 172.21.15.104:0/965345079 wait complete. 2023-06-10T18:53:13.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:13.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.866+0000 7f8924e76700 1 Processor -- start 2023-06-10T18:53:13.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.866+0000 7f8924e76700 1 -- start start 2023-06-10T18:53:13.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f8924e76700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f89200ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f8924e76700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f892015d1a0 0x7f89200cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f8924e76700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200ce850 con 0x7f89200b21c0 2023-06-10T18:53:13.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f8924e76700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89200b30a0 con 0x7f892015d1a0 2023-06-10T18:53:13.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f8924e76700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892015d740 con 0x7f8920156760 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f892015d1a0 0x7f89200cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f892015d1a0 0x7f89200cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33974/0 (socket says 172.21.15.104:33974) 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 -- 172.21.15.104:0/4041537868 learned_addr learned my addr 172.21.15.104:0/4041537868 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891e59c700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f89200ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 -- 172.21.15.104:0/4041537868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 msgr2=0x7f89200ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f89200ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 -- 172.21.15.104:0/4041537868 >> v1:172.21.15.114:6789/0 conn(0x7f89200b21c0 legacy=0x7f8920160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 -- 172.21.15.104:0/4041537868 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200cfca0 con 0x7f892015d1a0 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891dd9b700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f89200cad90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f89140098f0 tx=0x7f8914009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87319d00f7b13b82 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.867+0000 7f891d59a700 1 -- 172.21.15.104:0/4041537868 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891400b8a0 con 0x7f892015d1a0 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f891d59a700 1 -- 172.21.15.104:0/4041537868 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891400ba00 con 0x7f892015d1a0 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f891d59a700 1 -- 172.21.15.104:0/4041537868 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891400bd90 con 0x7f892015d1a0 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 -- 172.21.15.104:0/4041537868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 msgr2=0x7f89200cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f89200cad90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f89140098f0 tx=0x7f8914009c00 comp rx=0 tx=0).stop 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 -- 172.21.15.104:0/4041537868 shutdown_connections 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f89200ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 --2- 172.21.15.104:0/4041537868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f89200cad90 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 -- 172.21.15.104:0/4041537868 >> 172.21.15.104:0/4041537868 conn(0x7f892006e370 msgr2=0x7f892006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 -- 172.21.15.104:0/4041537868 shutdown_connections 2023-06-10T18:53:13.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.868+0000 7f8924e76700 1 -- 172.21.15.104:0/4041537868 wait complete. 2023-06-10T18:53:13.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.870+0000 7f8924e76700 1 Processor -- start 2023-06-10T18:53:13.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.870+0000 7f8924e76700 1 -- start start 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89200b21c0 0x7f892014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f892006b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200c71d0 con 0x7f892015d1a0 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200c79b0 con 0x7f89200b21c0 2023-06-10T18:53:13.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f8924e76700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b3a60 con 0x7f8920156760 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33982/0 (socket says 172.21.15.104:33982) 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 -- 172.21.15.104:0/3916757885 learned_addr learned my addr 172.21.15.104:0/3916757885 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891dd9b700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f892006b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891e59c700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89200b21c0 0x7f892014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 -- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 msgr2=0x7f892006b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f892006b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 -- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89200b21c0 msgr2=0x7f892014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89200b21c0 0x7f892014d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 -- 172.21.15.104:0/3916757885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8914009550 con 0x7f892015d1a0 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f891ed9d700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f891800cc80 tx=0x7f891800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.871+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891800ad40 con 0x7f892015d1a0 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.872+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800ce80 con 0x7f892015d1a0 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.872+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89200c6c10 con 0x7f892015d1a0 2023-06-10T18:53:13.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.872+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891800aea0 con 0x7f892015d1a0 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.872+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89200b36f0 con 0x7f892015d1a0 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.873+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8920074050 con 0x7f892015d1a0 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.873+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8918003c40 con 0x7f892015d1a0 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.873+0000 7f890effd700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 0x7f890006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.873+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f8918089760 con 0x7f892015d1a0 2023-06-10T18:53:13.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.874+0000 7f891e59c700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 0x7f890006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:13.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.874+0000 7f891e59c700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 0x7f890006d190 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8910009950 tx=0x7f8910008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:13.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:13.878+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8918054630 con 0x7f892015d1a0 2023-06-10T18:53:14.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.059+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f89200b3ba0 con 0x7f892015d1a0 2023-06-10T18:53:14.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.060+0000 7f890effd700 1 -- 172.21.15.104:0/3916757885 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8918057c50 con 0x7f892015d1a0 2023-06-10T18:53:14.065 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:14.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 msgr2=0x7f890006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 0x7f890006d190 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8910009950 tx=0x7f8910008040 comp rx=0 tx=0).stop 2023-06-10T18:53:14.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 msgr2=0x7f892006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f891800cc80 tx=0x7f891800a340 comp rx=0 tx=0).stop 2023-06-10T18:53:14.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 shutdown_connections 2023-06-10T18:53:14.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890006ace0 0x7f890006d190 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89200b21c0 0x7f892014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f892015d1a0 0x7f892006b9e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 --2- 172.21.15.104:0/3916757885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8920156760 0x7f892006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 >> 172.21.15.104:0/3916757885 conn(0x7f892006e370 msgr2=0x7f892015bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 shutdown_connections 2023-06-10T18:53:14.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.061+0000 7f8924e76700 1 -- 172.21.15.104:0/3916757885 wait complete. 2023-06-10T18:53:14.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:14.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.195+0000 7f0444aba700 1 Processor -- start 2023-06-10T18:53:14.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.195+0000 7f0444aba700 1 -- start start 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f0444aba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f044015a330 0x7f04400ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f0444aba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04400b1780 0x7f044006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f0444aba700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044015a700 con 0x7f04400cb420 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f0444aba700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044006d7b0 con 0x7f044015a330 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f0444aba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04400cbe00 con 0x7f04400b1780 2023-06-10T18:53:14.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f044015a330 0x7f04400ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f044015a330 0x7f04400ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33988/0 (socket says 172.21.15.104:33988) 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 -- 172.21.15.104:0/667288424 learned_addr learned my addr 172.21.15.104:0/667288424 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043ed9d700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04400b1780 0x7f044006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 -- 172.21.15.104:0/667288424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04400b1780 msgr2=0x7f044006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04400b1780 0x7f044006dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 -- 172.21.15.104:0/667288424 >> v1:172.21.15.114:6789/0 conn(0x7f04400cb420 legacy=0x7f04400b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 -- 172.21.15.104:0/667288424 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044006ce80 con 0x7f044015a330 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.196+0000 7f043e59c700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044015a330 0x7f04400ac1c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f0430006fd0 tx=0x7f0430009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e24f3619f74172 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f043d59a700 1 -- 172.21.15.104:0/667288424 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430005e30 con 0x7f044015a330 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f043d59a700 1 -- 172.21.15.104:0/667288424 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0430004030 con 0x7f044015a330 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f043d59a700 1 -- 172.21.15.104:0/667288424 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04300043e0 con 0x7f044015a330 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 -- 172.21.15.104:0/667288424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044015a330 msgr2=0x7f04400ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044015a330 0x7f04400ac1c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f0430006fd0 tx=0x7f0430009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 -- 172.21.15.104:0/667288424 shutdown_connections 2023-06-10T18:53:14.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044015a330 0x7f04400ac1c0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 --2- 172.21.15.104:0/667288424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04400b1780 0x7f044006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 -- 172.21.15.104:0/667288424 >> 172.21.15.104:0/667288424 conn(0x7f044006b8a0 msgr2=0x7f04400cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 -- 172.21.15.104:0/667288424 shutdown_connections 2023-06-10T18:53:14.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.197+0000 7f0444aba700 1 -- 172.21.15.104:0/667288424 wait complete. 2023-06-10T18:53:14.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.198+0000 7f0444aba700 1 Processor -- start 2023-06-10T18:53:14.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 -- start start 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 0x7f04400c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 0x7f04400cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400c3fe0 con 0x7f04400b1780 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400c41d0 con 0x7f04400cb420 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f0444aba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04400c44b0 con 0x7f044015a330 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f043e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f043e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33994/0 (socket says 172.21.15.104:33994) 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f043e59c700 1 -- 172.21.15.104:0/2222777505 learned_addr learned my addr 172.21.15.104:0/2222777505 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f043ed9d700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 0x7f04400cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.199+0000 7f043dd9b700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 0x7f04400c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 -- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 msgr2=0x7f04400cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 0x7f04400cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 -- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 msgr2=0x7f04400c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 0x7f04400c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 -- 172.21.15.104:0/2222777505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0430009500 con 0x7f04400b1780 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043dd9b700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 0x7f04400c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043ed9d700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 0x7f04400cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f043e59c700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0430009c10 tx=0x7f0430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430003dd0 con 0x7f04400b1780 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043001c420 con 0x7f04400b1780 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043001c7d0 con 0x7f04400b1780 2023-06-10T18:53:14.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04400c4790 con 0x7f04400b1780 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.200+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04400cc6f0 con 0x7f04400b1780 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.201+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044015c1c0 con 0x7f04400b1780 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.201+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f043001c930 con 0x7f04400b1780 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.202+0000 7f042effd700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 0x7f042006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.202+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f043008a6d0 con 0x7f04400b1780 2023-06-10T18:53:14.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.202+0000 7f043dd9b700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 0x7f042006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.202+0000 7f043dd9b700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 0x7f042006d190 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0434000f60 tx=0x7f0434009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.206+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f044015c1c0 con 0x7f04400b1780 2023-06-10T18:53:14.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.392+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f044015c1c0 con 0x7f04400b1780 2023-06-10T18:53:14.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.392+0000 7f042effd700 1 -- 172.21.15.104:0/2222777505 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f044015c1c0 con 0x7f04400b1780 2023-06-10T18:53:14.399 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:14.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 msgr2=0x7f042006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 0x7f042006d190 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0434000f60 tx=0x7f0434009040 comp rx=0 tx=0).stop 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 msgr2=0x7f04400c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0430009c10 tx=0x7f0430004750 comp rx=0 tx=0).stop 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 shutdown_connections 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f044015a330 0x7f04400cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04400b1780 0x7f04400c5a80 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f042006ace0 0x7f042006d190 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 --2- 172.21.15.104:0/2222777505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04400cb420 0x7f04400c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 >> 172.21.15.104:0/2222777505 conn(0x7f044006b8a0 msgr2=0x7f04400b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 shutdown_connections 2023-06-10T18:53:14.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.394+0000 7f0444aba700 1 -- 172.21.15.104:0/2222777505 wait complete. 2023-06-10T18:53:14.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:14.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.530+0000 7f900ad44700 1 Processor -- start 2023-06-10T18:53:14.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.530+0000 7f900ad44700 1 -- start start 2023-06-10T18:53:14.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f900ad44700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 0x7f90040d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f900ad44700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90040d0830 0x7f90040b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f900ad44700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040ad3c0 con 0x7f9004157b30 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f900ad44700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9004157f00 con 0x7f90040d0830 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f900ad44700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b0170 con 0x7f90040cea50 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f90092e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90040d0830 0x7f90040b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f90092e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90040d0830 0x7f90040b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34006/0 (socket says 172.21.15.104:34006) 2023-06-10T18:53:14.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.531+0000 7f90092e1700 1 -- 172.21.15.104:0/4210692730 learned_addr learned my addr 172.21.15.104:0/4210692730 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f9003fff700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 0x7f90040d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90092e1700 1 -- 172.21.15.104:0/4210692730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 msgr2=0x7f90040d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:14.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90092e1700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 0x7f90040d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90092e1700 1 -- 172.21.15.104:0/4210692730 >> v1:172.21.15.114:6789/0 conn(0x7f9004157b30 legacy=0x7f90040ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90092e1700 1 -- 172.21.15.104:0/4210692730 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040b0410 con 0x7f90040d0830 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f9003fff700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 0x7f90040d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90092e1700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90040d0830 0x7f90040b0ef0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e50 tx=0x7f8ffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172c80f6f8dc05d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90037fe700 1 -- 172.21.15.104:0/4210692730 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00e070 con 0x7f90040d0830 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90037fe700 1 -- 172.21.15.104:0/4210692730 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc003a20 con 0x7f90040d0830 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f90037fe700 1 -- 172.21.15.104:0/4210692730 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc003dd0 con 0x7f90040d0830 2023-06-10T18:53:14.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f900ad44700 1 -- 172.21.15.104:0/4210692730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90040d0830 msgr2=0x7f90040b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f900ad44700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90040d0830 0x7f90040b0ef0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009e50 tx=0x7f8ffc00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f900ad44700 1 -- 172.21.15.104:0/4210692730 shutdown_connections 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f900ad44700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90040d0830 0x7f90040b0ef0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.532+0000 7f900ad44700 1 --2- 172.21.15.104:0/4210692730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040cea50 0x7f90040d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.533+0000 7f900ad44700 1 -- 172.21.15.104:0/4210692730 >> 172.21.15.104:0/4210692730 conn(0x7f9004089fd0 msgr2=0x7f900408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.533+0000 7f900ad44700 1 -- 172.21.15.104:0/4210692730 shutdown_connections 2023-06-10T18:53:14.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.533+0000 7f900ad44700 1 -- 172.21.15.104:0/4210692730 wait complete. 2023-06-10T18:53:14.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.534+0000 7f900ad44700 1 Processor -- start 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.534+0000 7f900ad44700 1 -- start start 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90040cea50 0x7f90040b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040d0830 0x7f9004147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b20e0 con 0x7f9004157b30 2023-06-10T18:53:14.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004158690 con 0x7f90040cea50 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9004158970 con 0x7f90040d0830 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34012/0 (socket says 172.21.15.104:34012) 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 -- 172.21.15.104:0/3206548155 learned_addr learned my addr 172.21.15.104:0/3206548155 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f9003fff700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040d0830 0x7f9004147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f9008ae0700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90040cea50 0x7f90040b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 -- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040d0830 msgr2=0x7f9004147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040d0830 0x7f9004147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 -- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90040cea50 msgr2=0x7f90040b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90040cea50 0x7f90040b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 -- 172.21.15.104:0/3206548155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009b30 con 0x7f9004157b30 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90092e1700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc015040 tx=0x7f8ffc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc004440 con 0x7f9004157b30 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc0045c0 con 0x7f9004157b30 2023-06-10T18:53:14.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9004158c50 con 0x7f9004157b30 2023-06-10T18:53:14.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc004970 con 0x7f9004157b30 2023-06-10T18:53:14.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.535+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040cd090 con 0x7f9004157b30 2023-06-10T18:53:14.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.536+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004159170 con 0x7f9004157b30 2023-06-10T18:53:14.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.537+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8ffc012430 con 0x7f9004157b30 2023-06-10T18:53:14.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.537+0000 7f90017fa700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 0x7f8fe806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.537+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f8ffc08aa10 con 0x7f9004157b30 2023-06-10T18:53:14.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.537+0000 7f9008ae0700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 0x7f8fe806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.538+0000 7f9008ae0700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 0x7f8fe806d190 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009960 tx=0x7f8ff80090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.541+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9004159170 con 0x7f9004157b30 2023-06-10T18:53:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.729+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9004159170 con 0x7f9004157b30 2023-06-10T18:53:14.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.730+0000 7f90017fa700 1 -- 172.21.15.104:0/3206548155 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ffc017020 con 0x7f9004157b30 2023-06-10T18:53:14.736 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:14.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 msgr2=0x7f8fe806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 0x7f8fe806d190 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009960 tx=0x7f8ff80090e0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 msgr2=0x7f900406c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc015040 tx=0x7f8ffc004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 shutdown_connections 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fe806ace0 0x7f8fe806d190 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90040cea50 0x7f90040b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9004157b30 0x7f900406c3f0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 --2- 172.21.15.104:0/3206548155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90040d0830 0x7f9004147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 >> 172.21.15.104:0/3206548155 conn(0x7f9004089fd0 msgr2=0x7f900408a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 shutdown_connections 2023-06-10T18:53:14.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.732+0000 7f900ad44700 1 -- 172.21.15.104:0/3206548155 wait complete. 2023-06-10T18:53:14.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:14.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.863+0000 7fd05840c700 1 Processor -- start 2023-06-10T18:53:14.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.863+0000 7fd05840c700 1 -- start start 2023-06-10T18:53:14.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd05840c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0500ada30 0x7fd0500ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd05840c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 0x7fd0500ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd05840c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0500aed20 con 0x7fd0500ade00 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd05840c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0500cf430 con 0x7fd0500ada30 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd05840c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0500cee70 con 0x7fd0500ae930 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd0561a8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0500ada30 0x7fd0500ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd0561a8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0500ada30 0x7fd0500ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34016/0 (socket says 172.21.15.104:34016) 2023-06-10T18:53:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.864+0000 7fd0561a8700 1 -- 172.21.15.104:0/3417304073 learned_addr learned my addr 172.21.15.104:0/3417304073 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0569a9700 1 --2- 172.21.15.104:0/3417304073 >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 0x7fd0500ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0561a8700 1 -- 172.21.15.104:0/3417304073 >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 msgr2=0x7fd0500ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0561a8700 1 --2- 172.21.15.104:0/3417304073 >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 0x7fd0500ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0561a8700 1 -- 172.21.15.104:0/3417304073 >> v1:172.21.15.114:6789/0 conn(0x7fd0500ade00 legacy=0x7fd0500ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0561a8700 1 -- 172.21.15.104:0/3417304073 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500cf150 con 0x7fd0500ada30 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0569a9700 1 --2- 172.21.15.104:0/3417304073 >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 0x7fd0500ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0561a8700 1 --2- 172.21.15.104:0/3417304073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd0500ae3f0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fd050071030 tx=0x7fd040009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54747b7384d9271d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0551a6700 1 -- 172.21.15.104:0/3417304073 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040005e30 con 0x7fd0500ada30 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0551a6700 1 -- 172.21.15.104:0/3417304073 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd040006030 con 0x7fd0500ada30 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.865+0000 7fd0551a6700 1 -- 172.21.15.104:0/3417304073 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0400063e0 con 0x7fd0500ada30 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 -- 172.21.15.104:0/3417304073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 msgr2=0x7fd0500ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 --2- 172.21.15.104:0/3417304073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd0500ae3f0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fd050071030 tx=0x7fd040009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 -- 172.21.15.104:0/3417304073 shutdown_connections 2023-06-10T18:53:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 --2- 172.21.15.104:0/3417304073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd0500ae3f0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 --2- 172.21.15.104:0/3417304073 >> v2:172.21.15.114:3300/0 conn(0x7fd0500ae930 0x7fd0500ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 -- 172.21.15.104:0/3417304073 >> 172.21.15.104:0/3417304073 conn(0x7fd05006b8a0 msgr2=0x7fd0500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 -- 172.21.15.104:0/3417304073 shutdown_connections 2023-06-10T18:53:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.866+0000 7fd05840c700 1 -- 172.21.15.104:0/3417304073 wait complete. 2023-06-10T18:53:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.867+0000 7fd05840c700 1 Processor -- start 2023-06-10T18:53:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.867+0000 7fd05840c700 1 -- start start 2023-06-10T18:53:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0500ae930 0x7fd050099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 0x7fd05014d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050099880 con 0x7fd0500ada30 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050099ae0 con 0x7fd0501e5a00 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd050099dc0 con 0x7fd0500ae930 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34022/0 (socket says 172.21.15.104:34022) 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 -- 172.21.15.104:0/1682520190 learned_addr learned my addr 172.21.15.104:0/1682520190 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0569a9700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 0x7fd05014d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0559a7700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0500ae930 0x7fd050099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 -- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0500ae930 msgr2=0x7fd050099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0500ae930 0x7fd050099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 -- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 msgr2=0x7fd05014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 0x7fd05014d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 -- 172.21.15.104:0/1682520190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd040009500 con 0x7fd0500ada30 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0569a9700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 0x7fd05014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd0561a8700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fd040006000 tx=0x7fd040004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040004250 con 0x7fd0500ada30 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0400043b0 con 0x7fd0500ada30 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05009a0a0 con 0x7fd0500ada30 2023-06-10T18:53:14.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0400036a0 con 0x7fd0500ada30 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.868+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05009a660 con 0x7fd0500ada30 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.869+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd050071dc0 con 0x7fd0500ada30 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.870+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd040022070 con 0x7fd0500ada30 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.871+0000 7fd046ffd700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 0x7fd03406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.871+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fd04008b160 con 0x7fd0500ada30 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.871+0000 7fd0559a7700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 0x7fd03406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:14.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.871+0000 7fd0559a7700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 0x7fd03406d190 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd0501e2c50 tx=0x7fd04c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:14.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:14.874+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd0400560a0 con 0x7fd0500ada30 2023-06-10T18:53:15.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.101+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd050066f70 con 0x7fd0500ada30 2023-06-10T18:53:15.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.101+0000 7fd046ffd700 1 -- 172.21.15.104:0/1682520190 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4687 (secure 0 0 0) 0x7fd0400596c0 con 0x7fd0500ada30 2023-06-10T18:53:15.107 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:15.108 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:06.805017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":17,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:15.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.103+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 msgr2=0x7fd03406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.103+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 0x7fd03406d190 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd0501e2c50 tx=0x7fd04c007ec0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.103+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 msgr2=0x7fd05009ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.103+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fd040006000 tx=0x7fd040004750 comp rx=0 tx=0).stop 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 shutdown_connections 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd03406ace0 0x7fd03406d190 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0501e5a00 0x7fd05014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0500ada30 0x7fd05009ceb0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 --2- 172.21.15.104:0/1682520190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0500ae930 0x7fd050099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 >> 172.21.15.104:0/1682520190 conn(0x7fd05006b8a0 msgr2=0x7fd0500b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 shutdown_connections 2023-06-10T18:53:15.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.104+0000 7fd05840c700 1 -- 172.21.15.104:0/1682520190 wait complete. 2023-06-10T18:53:15.111 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 47 2023-06-10T18:53:15.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:15.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.234+0000 7fe96b0d3700 1 Processor -- start 2023-06-10T18:53:15.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.234+0000 7fe96b0d3700 1 -- start start 2023-06-10T18:53:15.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe96b0d3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe96415a420 0x7fe9640b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe96b0d3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9640cea60 0x7fe96415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe96b0d3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9640b6070 con 0x7fe9640b4ff0 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe96b0d3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9640b5460 con 0x7fe9640cea60 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe96b0d3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640b5740 con 0x7fe96415a420 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9640cea60 0x7fe96415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9640cea60 0x7fe96415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34034/0 (socket says 172.21.15.104:34034) 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 -- 172.21.15.104:0/2873790791 learned_addr learned my addr 172.21.15.104:0/2873790791 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe968e6f700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe96415a420 0x7fe9640b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 -- 172.21.15.104:0/2873790791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe96415a420 msgr2=0x7fe9640b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe96415a420 0x7fe9640b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 -- 172.21.15.104:0/2873790791 >> v1:172.21.15.114:6789/0 conn(0x7fe9640b4ff0 legacy=0x7fe9640ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.235+0000 7fe969670700 1 -- 172.21.15.104:0/2873790791 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9640b5a20 con 0x7fe9640cea60 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe969670700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9640cea60 0x7fe96415a7f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009e50 tx=0x7fe95c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5aae540d82c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe9637fe700 1 -- 172.21.15.104:0/2873790791 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c00e070 con 0x7fe9640cea60 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe9637fe700 1 -- 172.21.15.104:0/2873790791 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c003a20 con 0x7fe9640cea60 2023-06-10T18:53:15.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe9637fe700 1 -- 172.21.15.104:0/2873790791 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c003dd0 con 0x7fe9640cea60 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2873790791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9640cea60 msgr2=0x7fe96415a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9640cea60 0x7fe96415a7f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009e50 tx=0x7fe95c00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2873790791 shutdown_connections 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9640cea60 0x7fe96415a7f0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2873790791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe96415a420 0x7fe9640b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2873790791 >> 172.21.15.104:0/2873790791 conn(0x7fe964089fd0 msgr2=0x7fe9640d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2873790791 shutdown_connections 2023-06-10T18:53:15.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.236+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2873790791 wait complete. 2023-06-10T18:53:15.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 Processor -- start 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 -- start start 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9640b4ff0 0x7fe9641f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9640cea60 0x7fe9641e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9641f1e50 con 0x7fe96415a420 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9641f6460 con 0x7fe9640b4ff0 2023-06-10T18:53:15.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.238+0000 7fe96b0d3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640c9bd0 con 0x7fe9640cea60 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34038/0 (socket says 172.21.15.104:34038) 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 -- 172.21.15.104:0/2096373289 learned_addr learned my addr 172.21.15.104:0/2096373289 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe968e6f700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9640b4ff0 0x7fe9641f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe963fff700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9640cea60 0x7fe9641e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 -- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9640cea60 msgr2=0x7fe9641e3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9640cea60 0x7fe9641e3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 -- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9640b4ff0 msgr2=0x7fe9641f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9640b4ff0 0x7fe9641f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 -- 172.21.15.104:0/2096373289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c009b30 con 0x7fe96415a420 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe969670700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe95c004000 tx=0x7fe95c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c004440 con 0x7fe96415a420 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c0045a0 con 0x7fe96415a420 2023-06-10T18:53:15.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9640c9eb0 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c004950 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.239+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9640ca470 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.240+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9640cf6f0 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.241+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe95c012430 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.241+0000 7fe9617fa700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 0x7fe94806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.241+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7fe95c08abb0 con 0x7fe96415a420 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.241+0000 7fe968e6f700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 0x7fe94806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.242+0000 7fe968e6f700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 0x7fe94806d190 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe96406cc90 tx=0x7fe958008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.245+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe95c055a90 con 0x7fe96415a420 2023-06-10T18:53:15.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.429+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe964074050 con 0x7fe96415a420 2023-06-10T18:53:15.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.431+0000 7fe9617fa700 1 -- 172.21.15.104:0/2096373289 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+18835 (secure 0 0 0) 0x7fe95c0590b0 con 0x7fe96415a420 2023-06-10T18:53:15.437 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:15.437 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":87,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:04.801983+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1823366359":"2023-06-10T19:53:04.656002+0000","172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:15.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 msgr2=0x7fe94806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 0x7fe94806d190 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe96406cc90 tx=0x7fe958008040 comp rx=0 tx=0).stop 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 msgr2=0x7fe9641e3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe95c004000 tx=0x7fe95c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 shutdown_connections 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe94806ace0 0x7fe94806d190 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9640b4ff0 0x7fe9641f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe96415a420 0x7fe9641e3fb0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 --2- 172.21.15.104:0/2096373289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9640cea60 0x7fe9641e3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 >> 172.21.15.104:0/2096373289 conn(0x7fe964089fd0 msgr2=0x7fe96406f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 shutdown_connections 2023-06-10T18:53:15.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.433+0000 7fe96b0d3700 1 -- 172.21.15.104:0/2096373289 wait complete. 2023-06-10T18:53:15.451 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:53:15.451 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:15.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.565+0000 7f16022c6700 1 Processor -- start 2023-06-10T18:53:15.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.565+0000 7f16022c6700 1 -- start start 2023-06-10T18:53:15.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15fc0cfa00 0x7f15fc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 0x7f15fc0cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cad90 0x7f15fc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15fc0ccca0 con 0x7f15fc0cfa00 2023-06-10T18:53:15.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15fc0d0310 con 0x7f15fc15a250 2023-06-10T18:53:15.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f16022c6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc0cee50 con 0x7f15fc0cad90 2023-06-10T18:53:15.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15fc0cfa00 0x7f15fc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15fc0cfa00 0x7f15fc0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34044/0 (socket says 172.21.15.104:34044) 2023-06-10T18:53:15.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 -- 172.21.15.104:0/1150345134 learned_addr learned my addr 172.21.15.104:0/1150345134 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f1600863700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cad90 0x7f15fc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fb7fe700 1 --2- 172.21.15.104:0/1150345134 >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 0x7f15fc0cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 -- 172.21.15.104:0/1150345134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cad90 msgr2=0x7f15fc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cad90 0x7f15fc0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 -- 172.21.15.104:0/1150345134 >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 msgr2=0x7f15fc0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 --2- 172.21.15.104:0/1150345134 >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 0x7f15fc0cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 -- 172.21.15.104:0/1150345134 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc0cf130 con 0x7f15fc0cfa00 2023-06-10T18:53:15.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fb7fe700 1 --2- 172.21.15.104:0/1150345134 >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 0x7f15fc0cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:15.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.566+0000 7f15fbfff700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cfa00 0x7f15fc0701e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f15fc071ed0 tx=0x7f15f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd82443710c7908 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f15faffd700 1 -- 172.21.15.104:0/1150345134 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011830 con 0x7f15fc0cfa00 2023-06-10T18:53:15.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f15faffd700 1 -- 172.21.15.104:0/1150345134 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0011990 con 0x7f15fc0cfa00 2023-06-10T18:53:15.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f15faffd700 1 -- 172.21.15.104:0/1150345134 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011d20 con 0x7f15fc0cfa00 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 -- 172.21.15.104:0/1150345134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cfa00 msgr2=0x7f15fc0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cfa00 0x7f15fc0701e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f15fc071ed0 tx=0x7f15f0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 -- 172.21.15.104:0/1150345134 shutdown_connections 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cfa00 0x7f15fc0701e0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 --2- 172.21.15.104:0/1150345134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cad90 0x7f15fc0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 --2- 172.21.15.104:0/1150345134 >> v2:172.21.15.114:3300/0 conn(0x7f15fc15a250 0x7f15fc0cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 -- 172.21.15.104:0/1150345134 >> 172.21.15.104:0/1150345134 conn(0x7f15fc06e370 msgr2=0x7f15fc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 -- 172.21.15.104:0/1150345134 shutdown_connections 2023-06-10T18:53:15.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.567+0000 7f16022c6700 1 -- 172.21.15.104:0/1150345134 wait complete. 2023-06-10T18:53:15.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.568+0000 7f16022c6700 1 Processor -- start 2023-06-10T18:53:15.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.568+0000 7f16022c6700 1 -- start start 2023-06-10T18:53:15.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cfa00 0x7f15fc0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15fc15a250 0x7f15fc0c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc0c8440 con 0x7f15fc0cad90 2023-06-10T18:53:15.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc0b4320 con 0x7f15fc15a250 2023-06-10T18:53:15.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f16022c6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc0b3710 con 0x7f15fc0cfa00 2023-06-10T18:53:15.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34050/0 (socket says 172.21.15.104:34050) 2023-06-10T18:53:15.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 -- 172.21.15.104:0/258134098 learned_addr learned my addr 172.21.15.104:0/258134098 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:15.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 -- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cfa00 msgr2=0x7f15fc0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cfa00 0x7f15fc0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f15fbfff700 1 -- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15fc15a250 msgr2=0x7f15fc0c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.569+0000 7f1600863700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15fc15a250 0x7f15fc0c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15fbfff700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15fc15a250 0x7f15fc0c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15fbfff700 1 -- 172.21.15.104:0/258134098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009550 con 0x7f15fc0cad90 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15fbfff700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009c10 tx=0x7f15f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f00036a0 con 0x7f15fc0cad90 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f001e070 con 0x7f15fc0cad90 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc0b39f0 con 0x7f15fc0cad90 2023-06-10T18:53:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0003a60 con 0x7f15fc0cad90 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.570+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc0b3fb0 con 0x7f15fc0cad90 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.571+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc074050 con 0x7f15fc0cad90 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.571+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f15f0003bc0 con 0x7f15fc0cad90 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.572+0000 7f15f8ff9700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 0x7f15e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.572+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f15f008a590 con 0x7f15fc0cad90 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.572+0000 7f15fb7fe700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 0x7f15e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.572+0000 7f15fb7fe700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 0x7f15e806d190 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006fd0 tx=0x7f15ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.576+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f15f00554f0 con 0x7f15fc0cad90 2023-06-10T18:53:15.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.805+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15fc0cc1c0 con 0x7f15fc0cad90 2023-06-10T18:53:15.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.806+0000 7f15f8ff9700 1 -- 172.21.15.104:0/258134098 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4687 (secure 0 0 0) 0x7f15f002c3b0 con 0x7f15fc0cad90 2023-06-10T18:53:15.812 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:15.812 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:06.805017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":17,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 msgr2=0x7f15e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 0x7f15e806d190 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006fd0 tx=0x7f15ec008040 comp rx=0 tx=0).stop 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 msgr2=0x7f15fc1ef4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009c10 tx=0x7f15f0004750 comp rx=0 tx=0).stop 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 shutdown_connections 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15e806ace0 0x7f15e806d190 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15fc15a250 0x7f15fc0c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15fc0cad90 0x7f15fc1ef4c0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 --2- 172.21.15.104:0/258134098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15fc0cfa00 0x7f15fc0c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 >> 172.21.15.104:0/258134098 conn(0x7f15fc06e370 msgr2=0x7f15fc154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 shutdown_connections 2023-06-10T18:53:15.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.808+0000 7f16022c6700 1 -- 172.21.15.104:0/258134098 wait complete. 2023-06-10T18:53:15.816 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 47 2023-06-10T18:53:15.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.940+0000 7f5046b56700 1 Processor -- start 2023-06-10T18:53:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.940+0000 7f5046b56700 1 -- start start 2023-06-10T18:53:15.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f5046b56700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f504015a420 0x7f50400b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f5046b56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f504015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f5046b56700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50400b6070 con 0x7f50400b4ff0 2023-06-10T18:53:15.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f5046b56700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50400b5460 con 0x7f504015a420 2023-06-10T18:53:15.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f5046b56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50400b5740 con 0x7f50400cea60 2023-06-10T18:53:15.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f50400b4ff0 0x7f50400ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60120/0 (socket says 172.21.15.104:60120) 2023-06-10T18:53:15.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503ffff700 1 -- 172.21.15.104:0/2882062310 learned_addr learned my addr 172.21.15.104:0/2882062310 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:15.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f50448f2700 1 --2- 172.21.15.104:0/2882062310 >> v2:172.21.15.114:3300/0 conn(0x7f504015a420 0x7f50400b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f50450f3700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f504015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975076330 0 0) 0x7f50400b6070 con 0x7f50400b4ff0 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50300035b0 con 0x7f50400b4ff0 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88616293 0 0) 0x7f50300035b0 con 0x7f50400b4ff0 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 msgr2=0x7f504015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f504015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 >> v2:172.21.15.114:3300/0 conn(0x7f504015a420 msgr2=0x7f50400b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 --2- 172.21.15.104:0/2882062310 >> v2:172.21.15.114:3300/0 conn(0x7f504015a420 0x7f50400b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400b5a20 con 0x7f50400b4ff0 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f502c001670 con 0x7f50400b4ff0 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.941+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 >> v1:172.21.15.104:6789/0 conn(0x7f50400b4ff0 legacy=0x7f50400ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5030003a00 0x7f5030003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f5030004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5030004d90 0x7f5030005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50400b5460 con 0x7f5030004d90 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50400b5740 con 0x7f5030003a00 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503000ba10 con 0x7f50400cea60 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5030004d90 0x7f5030005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50448f2700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5030003a00 0x7f5030003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503ffff700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f5030004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 -- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 msgr2=0x7f5030004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50400cea60 0x7f5030004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 -- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5030003a00 msgr2=0x7f5030003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5030003a00 0x7f5030003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 -- 172.21.15.104:0/2882062310 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300035b0 con 0x7f5030004d90 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f50450f3700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5030004d90 0x7f5030005200 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5038009f90 tx=0x7f5030005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77733fd68137f57 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503800ebd0 con 0x7f5030004d90 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503800f040 con 0x7f5030004d90 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.942+0000 7f503f7fe700 1 -- 172.21.15.104:0/2882062310 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038014480 con 0x7f5030004d90 2023-06-10T18:53:15.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 -- 172.21.15.104:0/2882062310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5030004d90 msgr2=0x7f5030005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 --2- 172.21.15.104:0/2882062310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5030004d90 0x7f5030005200 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5038009f90 tx=0x7f5030005880 comp rx=0 tx=0).stop 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f50450f3700 1 -- 172.21.15.104:0/2882062310 reap_dead start 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 -- 172.21.15.104:0/2882062310 shutdown_connections 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 -- 172.21.15.104:0/2882062310 >> 172.21.15.104:0/2882062310 conn(0x7f5040089fd0 msgr2=0x7f50400d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 -- 172.21.15.104:0/2882062310 shutdown_connections 2023-06-10T18:53:15.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.943+0000 7f5046b56700 1 -- 172.21.15.104:0/2882062310 wait complete. 2023-06-10T18:53:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.944+0000 7f5046b56700 1 Processor -- start 2023-06-10T18:53:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.944+0000 7f5046b56700 1 -- start start 2023-06-10T18:53:15.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50401e3ae0 0x7f50400c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50401e3eb0 0x7f504006af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50400c6c50 con 0x7f504006b4c0 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50400c5cf0 con 0x7f50401e3ae0 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f5046b56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50400c5fd0 con 0x7f50401e3eb0 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34074/0 (socket says 172.21.15.104:34074) 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 -- 172.21.15.104:0/4294176810 learned_addr learned my addr 172.21.15.104:0/4294176810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f503ffff700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50401e3eb0 0x7f504006af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50448f2700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50401e3ae0 0x7f50400c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 -- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50401e3eb0 msgr2=0x7f504006af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50401e3eb0 0x7f504006af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 -- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50401e3ae0 msgr2=0x7f50400c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50401e3ae0 0x7f50400c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 -- 172.21.15.104:0/4294176810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400c62b0 con 0x7f504006b4c0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.945+0000 7f50450f3700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5038016000 tx=0x7f5038009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.946+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038007000 con 0x7f504006b4c0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.946+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5038003e70 con 0x7f504006b4c0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.946+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50400c6590 con 0x7f504006b4c0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.946+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038015b40 con 0x7f504006b4c0 2023-06-10T18:53:15.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.946+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50400c4490 con 0x7f504006b4c0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.947+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50400c6d90 con 0x7f504006b4c0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.947+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5038024070 con 0x7f504006b4c0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.947+0000 7f503d7fa700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 0x7f502406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.947+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f503808b280 con 0x7f504006b4c0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.948+0000 7f50448f2700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 0x7f502406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.948+0000 7f50448f2700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 0x7f502406d190 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f50400ce8b0 tx=0x7f5034008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:15.952+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f50400c6d90 con 0x7f504006b4c0 2023-06-10T18:53:16.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.182+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5040074050 con 0x7f504006b4c0 2023-06-10T18:53:16.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.183+0000 7f503d7fa700 1 -- 172.21.15.104:0/4294176810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4687 (secure 0 0 0) 0x7f5038059d90 con 0x7f504006b4c0 2023-06-10T18:53:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:16.189 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:51:48.370220+0000","modified":"2023-06-10T18:53:06.805017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":17,"addr":"172.21.15.114:6836/3102272294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3102272294},{"type":"v1","addr":"172.21.15.114:6836","nonce":3102272294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-06-10T18:53:16.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 msgr2=0x7f502406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 0x7f502406d190 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f50400ce8b0 tx=0x7f5034008040 comp rx=0 tx=0).stop 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 msgr2=0x7f504006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5038016000 tx=0x7f5038009750 comp rx=0 tx=0).stop 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 shutdown_connections 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502406ace0 0x7f502406d190 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50401e3ae0 0x7f50400c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f504006b4c0 0x7f504006b930 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 --2- 172.21.15.104:0/4294176810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50401e3eb0 0x7f504006af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 >> 172.21.15.104:0/4294176810 conn(0x7f5040089fd0 msgr2=0x7f504006e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 shutdown_connections 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.185+0000 7f5046b56700 1 -- 172.21.15.104:0/4294176810 wait complete. 2023-06-10T18:53:16.192 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 47 2023-06-10T18:53:16.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:16.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.316+0000 7f718aa1a700 1 Processor -- start 2023-06-10T18:53:16.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.316+0000 7f718aa1a700 1 -- start start 2023-06-10T18:53:16.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f718aa1a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f718406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f718aa1a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f718406e4b0 0x7f71840c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f718aa1a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718406d290 con 0x7f71840ca100 2023-06-10T18:53:16.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f718aa1a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7184071640 con 0x7f718406e4b0 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f718aa1a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840aef00 con 0x7f718406e0e0 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7188fb7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f71840ca100 0x7f71840ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60142/0 (socket says 172.21.15.104:60142) 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7188fb7700 1 -- 172.21.15.104:0/2641721389 learned_addr learned my addr 172.21.15.104:0/2641721389 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683876797 0 0) 0x7f718406d290 con 0x7f71840ca100 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f71837fe700 1 --2- 172.21.15.104:0/2641721389 >> v2:172.21.15.114:3300/0 conn(0x7f718406e4b0 0x7f71840c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716c0035b0 con 0x7f71840ca100 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7183fff700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f718406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114202281 0 0) 0x7f716c0035b0 con 0x7f71840ca100 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 msgr2=0x7f718406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f718406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 >> v2:172.21.15.114:3300/0 conn(0x7f718406e4b0 msgr2=0x7f71840c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 --2- 172.21.15.104:0/2641721389 >> v2:172.21.15.114:3300/0 conn(0x7f718406e4b0 0x7f71840c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718406d4c0 con 0x7f71840ca100 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f717c0019f0 con 0x7f71840ca100 2023-06-10T18:53:16.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.317+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 >> v1:172.21.15.104:6789/0 conn(0x7f71840ca100 legacy=0x7f71840ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f716c003a00 0x7f716c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f716c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f716c004d90 0x7f716c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7184071640 con 0x7f716c004d90 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840aef00 con 0x7f716c003a00 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c00ba10 con 0x7f718406e0e0 2023-06-10T18:53:16.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f716c004d90 0x7f716c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f71837fe700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f716c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7183fff700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f716c003a00 0x7f716c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 -- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 msgr2=0x7f716c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f718406e0e0 0x7f716c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 -- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f716c003a00 msgr2=0x7f716c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f716c003a00 0x7f716c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 -- 172.21.15.104:0/2641721389 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0035b0 con 0x7f716c004d90 2023-06-10T18:53:16.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7188fb7700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f716c004d90 0x7f716c005200 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f717c00a8d0 tx=0x7f716c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f691ee60732e1873 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c00a340 con 0x7f716c004d90 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c012600 con 0x7f716c004d90 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.318+0000 7f7182ffd700 1 -- 172.21.15.104:0/2641721389 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0129b0 con 0x7f716c004d90 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 -- 172.21.15.104:0/2641721389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f716c004d90 msgr2=0x7f716c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 --2- 172.21.15.104:0/2641721389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f716c004d90 0x7f716c005200 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f717c00a8d0 tx=0x7f716c005880 comp rx=0 tx=0).stop 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f7188fb7700 1 -- 172.21.15.104:0/2641721389 reap_dead start 2023-06-10T18:53:16.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 -- 172.21.15.104:0/2641721389 shutdown_connections 2023-06-10T18:53:16.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 -- 172.21.15.104:0/2641721389 >> 172.21.15.104:0/2641721389 conn(0x7f718406b8a0 msgr2=0x7f71840cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:16.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 -- 172.21.15.104:0/2641721389 shutdown_connections 2023-06-10T18:53:16.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.319+0000 7f718aa1a700 1 -- 172.21.15.104:0/2641721389 wait complete. 2023-06-10T18:53:16.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.320+0000 7f718aa1a700 1 Processor -- start 2023-06-10T18:53:16.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.320+0000 7f718aa1a700 1 -- start start 2023-06-10T18:53:16.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840c4900 0x7f71840af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840c4cd0 0x7f71840aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840c8e80 con 0x7f7184154860 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7184155210 con 0x7f71840c4900 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f718aa1a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840c8200 con 0x7f71840c4cd0 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34094/0 (socket says 172.21.15.104:34094) 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 -- 172.21.15.104:0/3664425724 learned_addr learned my addr 172.21.15.104:0/3664425724 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:16.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7183fff700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840c4900 0x7f71840af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f71837fe700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840c4cd0 0x7f71840aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 -- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840c4cd0 msgr2=0x7f71840aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840c4cd0 0x7f71840aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 -- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840c4900 msgr2=0x7f71840af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840c4900 0x7f71840af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 -- 172.21.15.104:0/3664425724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71840c84e0 con 0x7f7184154860 2023-06-10T18:53:16.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.321+0000 7f7188fb7700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f717c012000 tx=0x7f717c001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.322+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0019c0 con 0x7f7184154860 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.322+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c0011c0 con 0x7f7184154860 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.322+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c027410 con 0x7f7184154860 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.322+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71840c87c0 con 0x7f7184154860 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.322+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71840b1a70 con 0x7f7184154860 2023-06-10T18:53:16.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.323+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71840c8fc0 con 0x7f7184154860 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.323+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f717c019070 con 0x7f7184154860 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.324+0000 7f7180ff9700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 0x7f717006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.324+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f717c08ab40 con 0x7f7184154860 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.324+0000 7f7183fff700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 0x7f717006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.324+0000 7f7183fff700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 0x7f717006d1c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7178009b80 tx=0x7f7178008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.328+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f71840c8fc0 con 0x7f7184154860 2023-06-10T18:53:16.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.518+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71840c8fc0 con 0x7f7184154860 2023-06-10T18:53:16.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.519+0000 7f7180ff9700 1 -- 172.21.15.104:0/3664425724 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+18835 (secure 0 0 0) 0x7f71840c8fc0 con 0x7f7184154860 2023-06-10T18:53:16.525 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:16.525 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":87,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:04.801983+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:51:46.185506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-06-10T18:51:47.337720+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.38","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1823366359":"2023-06-10T19:53:04.656002+0000","172.21.15.114:6836/414069859":"2023-06-11T18:52:09.042261+0000","172.21.15.114:6834/414069859":"2023-06-11T18:52:09.042261+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:16.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 msgr2=0x7f717006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 0x7f717006d1c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7178009b80 tx=0x7f7178008040 comp rx=0 tx=0).stop 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 msgr2=0x7f7184154cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f717c012000 tx=0x7f717c001360 comp rx=0 tx=0).stop 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 shutdown_connections 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f717006ad10 0x7f717006d1c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840c4900 0x7f71840af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7184154860 0x7f7184154cd0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 --2- 172.21.15.104:0/3664425724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840c4cd0 0x7f71840aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 >> 172.21.15.104:0/3664425724 conn(0x7f718406b8a0 msgr2=0x7f71840ac440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:16.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 shutdown_connections 2023-06-10T18:53:16.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.522+0000 7f718aa1a700 1 -- 172.21.15.104:0/3664425724 wait complete. 2023-06-10T18:53:16.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:53:16.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.653+0000 7f16a4674700 1 Processor -- start 2023-06-10T18:53:16.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.653+0000 7f16a4674700 1 -- start start 2023-06-10T18:53:16.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a4674700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f169c06a580 0x7f169c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a4674700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c06dcb0 con 0x7f169c0cf030 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a4674700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c0cf460 con 0x7f169c15c500 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a4674700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c06e850 con 0x7f169c06a580 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a2c11700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f169c06a580 0x7f169c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a1c0f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f169c0cf030 0x7f169c069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60160/0 (socket says 172.21.15.104:60160) 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a1c0f700 1 -- 172.21.15.104:0/293486254 learned_addr learned my addr 172.21.15.104:0/293486254 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a140e700 1 -- 172.21.15.104:0/293486254 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495300452 0 0) 0x7f169c06dcb0 con 0x7f169c0cf030 2023-06-10T18:53:16.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a140e700 1 -- 172.21.15.104:0/293486254 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16880035b0 con 0x7f169c0cf030 2023-06-10T18:53:16.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a2c11700 1 -- 172.21.15.104:0/293486254 >> v1:172.21.15.114:6789/0 conn(0x7f169c15c500 legacy=0x7f169c06e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a2c11700 1 -- 172.21.15.104:0/293486254 >> v1:172.21.15.104:6789/0 conn(0x7f169c0cf030 legacy=0x7f169c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a2c11700 1 -- 172.21.15.104:0/293486254 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c06d0a0 con 0x7f169c06a580 2023-06-10T18:53:16.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.654+0000 7f16a2c11700 1 --2- 172.21.15.104:0/293486254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c06a580 0x7f169c0ce040 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1694009d90 tx=0x7f169400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32df99af2047b111 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a140e700 1 -- 172.21.15.104:0/293486254 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400e070 con 0x7f169c06a580 2023-06-10T18:53:16.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a140e700 1 -- 172.21.15.104:0/293486254 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169400bb20 con 0x7f169c06a580 2023-06-10T18:53:16.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a140e700 1 -- 172.21.15.104:0/293486254 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694012430 con 0x7f169c06a580 2023-06-10T18:53:16.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 -- 172.21.15.104:0/293486254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c06a580 msgr2=0x7f169c0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 --2- 172.21.15.104:0/293486254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c06a580 0x7f169c0ce040 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1694009d90 tx=0x7f169400b330 comp rx=0 tx=0).stop 2023-06-10T18:53:16.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 -- 172.21.15.104:0/293486254 shutdown_connections 2023-06-10T18:53:16.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 --2- 172.21.15.104:0/293486254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c06a580 0x7f169c0ce040 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 -- 172.21.15.104:0/293486254 >> 172.21.15.104:0/293486254 conn(0x7f169c06b8a0 msgr2=0x7f169c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:16.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 -- 172.21.15.104:0/293486254 shutdown_connections 2023-06-10T18:53:16.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.655+0000 7f16a4674700 1 -- 172.21.15.104:0/293486254 wait complete. 2023-06-10T18:53:16.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.656+0000 7f16a4674700 1 Processor -- start 2023-06-10T18:53:16.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.656+0000 7f16a4674700 1 -- start start 2023-06-10T18:53:16.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 0x7f169c1587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 0x7f169c158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b4b30 con 0x7f169c15c500 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b4e10 con 0x7f169c0cf030 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a4674700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b50f0 con 0x7f169c06a580 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2c11700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2c11700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34108/0 (socket says 172.21.15.104:34108) 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2c11700 1 -- 172.21.15.104:0/3630566296 learned_addr learned my addr 172.21.15.104:0/3630566296 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a1c0f700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 0x7f169c158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2410700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 0x7f169c1587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2c11700 1 -- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 msgr2=0x7f169c1587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.657+0000 7f16a2c11700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 0x7f169c1587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a2c11700 1 -- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 msgr2=0x7f169c158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a2c11700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 0x7f169c158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a2c11700 1 -- 172.21.15.104:0/3630566296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1694009a70 con 0x7f169c15c500 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a1c0f700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 0x7f169c158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:16.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a2410700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 0x7f169c1587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a2c11700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f1694013000 tx=0x7f1694004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694012430 con 0x7f169c15c500 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694020070 con 0x7f169c15c500 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c0b3630 con 0x7f169c15c500 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694004280 con 0x7f169c15c500 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.658+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c1590b0 con 0x7f169c15c500 2023-06-10T18:53:16.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.659+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c0b3490 con 0x7f169c15c500 2023-06-10T18:53:16.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.659+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f16940043e0 con 0x7f169c15c500 2023-06-10T18:53:16.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.660+0000 7f1692ffd700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 0x7f168006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:16.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.660+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6540+0+0 (secure 0 0 0) 0x7f1694022070 con 0x7f169c15c500 2023-06-10T18:53:16.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.660+0000 7f16a2410700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 0x7f168006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:16.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.660+0000 7f16a2410700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 0x7f168006d190 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f169c0c7810 tx=0x7f1698009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:16.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.664+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f169c0b3490 con 0x7f169c15c500 2023-06-10T18:53:16.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:16.896+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f169c0b3870 con 0x7f169c15c500 2023-06-10T18:53:17.866 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:53:17.861+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:53:17.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.871+0000 7f1692ffd700 1 -- 172.21.15.104:0/3630566296 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v48) v1 ==== 161+0+0 (secure 0 0 0) 0x7f169c0b3870 con 0x7f169c15c500 2023-06-10T18:53:17.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 msgr2=0x7f168006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:17.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 0x7f168006d190 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f169c0c7810 tx=0x7f1698009040 comp rx=0 tx=0).stop 2023-06-10T18:53:17.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 msgr2=0x7f169c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f1694013000 tx=0x7f1694004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 shutdown_connections 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f169c15c500 0x7f169c0b3b50 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f169c06a580 0x7f169c1587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f168006ace0 0x7f168006d190 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 --2- 172.21.15.104:0/3630566296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f169c0cf030 0x7f169c158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:17.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.874+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 >> 172.21.15.104:0/3630566296 conn(0x7f169c06b8a0 msgr2=0x7f169c06e040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:17.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.875+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 shutdown_connections 2023-06-10T18:53:17.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:17.875+0000 7f16a4674700 1 -- 172.21.15.104:0/3630566296 wait complete. 2023-06-10T18:53:17.882 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:53:17.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:53:17.912 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:53:18.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.016+0000 7fa602719700 1 Processor -- start 2023-06-10T18:53:18.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.016+0000 7fa602719700 1 -- start start 2023-06-10T18:53:18.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa602719700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:18.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa602719700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc06e250 con 0x7fa5fc15c580 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa602719700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc15c950 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa602719700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0b1e80 con 0x7fa5fc06cde0 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa5fbfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa5fc15c580 0x7fa5fc0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60172/0 (socket says 172.21.15.104:60172) 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa5fbfff700 1 -- 172.21.15.104:0/1231535730 learned_addr learned my addr 172.21.15.104:0/1231535730 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.017+0000 7fa600cb6700 1 --2- 172.21.15.104:0/1231535730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855042452 0 0) 0x7fa5fc06e250 con 0x7fa5fc15c580 2023-06-10T18:53:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e40035b0 con 0x7fa5fc15c580 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495165442 0 0) 0x7fa5fc15c950 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa600cb6700 1 -- 172.21.15.104:0/1231535730 >> v1:172.21.15.114:6789/0 conn(0x7fa5fc06e6e0 legacy=0x7fa5fc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa600cb6700 1 -- 172.21.15.104:0/1231535730 >> v1:172.21.15.104:6789/0 conn(0x7fa5fc15c580 legacy=0x7fa5fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa600cb6700 1 -- 172.21.15.104:0/1231535730 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc06d920 con 0x7fa5fc06cde0 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa600cb6700 1 --2- 172.21.15.104:0/1231535730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc0b1780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4009dc0 tx=0x7fa5f400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882aebfed32dca48 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:18.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400e070 con 0x7fa5fc06cde0 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f400bb20 con 0x7fa5fc06cde0 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.018+0000 7fa5faffd700 1 -- 172.21.15.104:0/1231535730 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400bed0 con 0x7fa5fc06cde0 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 -- 172.21.15.104:0/1231535730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 msgr2=0x7fa5fc0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 --2- 172.21.15.104:0/1231535730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc0b1780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4009dc0 tx=0x7fa5f400b330 comp rx=0 tx=0).stop 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 -- 172.21.15.104:0/1231535730 shutdown_connections 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 --2- 172.21.15.104:0/1231535730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc0b1780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 -- 172.21.15.104:0/1231535730 >> 172.21.15.104:0/1231535730 conn(0x7fa5fc06b8a0 msgr2=0x7fa5fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:18.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 -- 172.21.15.104:0/1231535730 shutdown_connections 2023-06-10T18:53:18.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.019+0000 7fa602719700 1 -- 172.21.15.104:0/1231535730 wait complete. 2023-06-10T18:53:18.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.020+0000 7fa602719700 1 Processor -- start 2023-06-10T18:53:18.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.020+0000 7fa602719700 1 -- start start 2023-06-10T18:53:18.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc1f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:18.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5fc15c580 0x7fa5fc0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc1e1840 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc1f1c10 con 0x7fa5fc15c580 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc1f5d80 con 0x7fa5fc06cde0 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34120/0 (socket says 172.21.15.104:34120) 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 -- 172.21.15.104:0/3805573366 learned_addr learned my addr 172.21.15.104:0/3805573366 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa600cb6700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5fc15c580 0x7fa5fc0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fbfff700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc1f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 -- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 msgr2=0x7fa5fc1f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc1f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 -- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5fc15c580 msgr2=0x7fa5fc0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5fc15c580 0x7fa5fc0c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 -- 172.21.15.104:0/3805573366 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f4009a70 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5fb7fe700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c4d0 tx=0x7fa5ec00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0054e0 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec003680 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc1f6060 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec003a30 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.021+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc1e1d00 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.022+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc0c6010 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.023+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa5ec003b90 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.023+0000 7fa5f8ff9700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 0x7fa5e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.023+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6626+0+0 (secure 0 0 0) 0x7fa5ec088b00 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.023+0000 7fa5fbfff700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 0x7fa5e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.024+0000 7fa5fbfff700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 0x7fa5e806d190 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0001360 tx=0x7fa5f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:18.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.028+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5ec08ae60 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.255+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa5fc1f6250 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.905+0000 7fa5f8ff9700 1 -- 172.21.15.104:0/3805573366 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa5ec053a20 con 0x7fa5fc06e6e0 2023-06-10T18:53:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 msgr2=0x7fa5e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 0x7fa5e806d190 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0001360 tx=0x7fa5f0009040 comp rx=0 tx=0).stop 2023-06-10T18:53:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 msgr2=0x7fa5fc0c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c4d0 tx=0x7fa5ec00c7e0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 shutdown_connections 2023-06-10T18:53:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5e806ace0 0x7fa5e806d190 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5fc15c580 0x7fa5fc0c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5fc06cde0 0x7fa5fc1f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 --2- 172.21.15.104:0/3805573366 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5fc06e6e0 0x7fa5fc0c3ff0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 >> 172.21.15.104:0/3805573366 conn(0x7fa5fc06b8a0 msgr2=0x7fa5fc159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 shutdown_connections 2023-06-10T18:53:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:18.908+0000 7fa602719700 1 -- 172.21.15.104:0/3805573366 wait complete. 2023-06-10T18:53:18.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:53:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.044+0000 7fa3f11cb700 1 Processor -- start 2023-06-10T18:53:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.044+0000 7fa3f11cb700 1 -- start start 2023-06-10T18:53:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3f11cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3ec06a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3f11cb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec0d0530 con 0x7fa3ec06d450 2023-06-10T18:53:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3f11cb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec06cf10 con 0x7fa3ec06e0b0 2023-06-10T18:53:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3f11cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec0cb220 con 0x7fa3ec0ae9c0 2023-06-10T18:53:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3ea59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa3ec06d450 0x7fa3ec06d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60188/0 (socket says 172.21.15.104:60188) 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3ea59c700 1 -- 172.21.15.104:0/4243371728 learned_addr learned my addr 172.21.15.104:0/4243371728 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.045+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255731268 0 0) 0x7fa3ec0d0530 con 0x7fa3ec06d450 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d40035b0 con 0x7fa3ec06d450 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3ec06a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074327946 0 0) 0x7fa3d40035b0 con 0x7fa3ec06d450 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 msgr2=0x7fa3ec06a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3ec06a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 >> v1:172.21.15.114:6789/0 conn(0x7fa3ec06e0b0 legacy=0x7fa3ec0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec06d1f0 con 0x7fa3ec06d450 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa3dc001670 con 0x7fa3ec06d450 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3ec06a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 >> v1:172.21.15.104:6789/0 conn(0x7fa3ec06d450 legacy=0x7fa3ec06d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3d4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3d4004020 0x7fa3d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3d4004d90 0x7fa3d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec0cb220 con 0x7fa3d4004d90 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d400bb50 con 0x7fa3d4004020 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d400bd10 con 0x7fa3ec0ae9c0 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3d4004d90 0x7fa3d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3d4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3ea59c700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3d4004020 0x7fa3d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 -- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 msgr2=0x7fa3d4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0ae9c0 0x7fa3d4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 -- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3d4004020 msgr2=0x7fa3d4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3d4004020 0x7fa3d4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 -- 172.21.15.104:0/4243371728 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40035b0 con 0x7fa3d4004d90 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.046+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3d4004d90 0x7fa3d4005200 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400c160 tx=0x7fa3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad76f0f201b03b13 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4001db0 con 0x7fa3d4004d90 2023-06-10T18:53:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e400cca0 con 0x7fa3d4004d90 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3e9d9b700 1 -- 172.21.15.104:0/4243371728 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4003c40 con 0x7fa3d4004d90 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 -- 172.21.15.104:0/4243371728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3d4004d90 msgr2=0x7fa3d4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/4243371728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3d4004d90 0x7fa3d4005200 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400c160 tx=0x7fa3d4005880 comp rx=0 tx=0).stop 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3eb59e700 1 -- 172.21.15.104:0/4243371728 reap_dead start 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 -- 172.21.15.104:0/4243371728 shutdown_connections 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 -- 172.21.15.104:0/4243371728 >> 172.21.15.104:0/4243371728 conn(0x7fa3ec06b8a0 msgr2=0x7fa3ec0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 -- 172.21.15.104:0/4243371728 shutdown_connections 2023-06-10T18:53:19.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.047+0000 7fa3f11cb700 1 -- 172.21.15.104:0/4243371728 wait complete. 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 Processor -- start 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 -- start start 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 0x7fa3ec1f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 0x7fa3ec0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec0acb90 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec0ac530 con 0x7fa3ec1f3f20 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.049+0000 7fa3f11cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec1f4460 con 0x7fa3ec0c3960 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34140/0 (socket says 172.21.15.104:34140) 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 -- 172.21.15.104:0/3091297654 learned_addr learned my addr 172.21.15.104:0/3091297654 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3ea59c700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 0x7fa3ec0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 0x7fa3ec1f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 -- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 msgr2=0x7fa3ec1f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 0x7fa3ec1f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 -- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 msgr2=0x7fa3ec0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 0x7fa3ec0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 -- 172.21.15.104:0/3091297654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec1f4740 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3ea59c700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 0x7fa3ec0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:19.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 0x7fa3ec1f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3eb59e700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4005040 tx=0x7fa3e40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4003e10 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e40050d0 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ec0ac7b0 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40054a0 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.050+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ec0c5c60 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.051+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ec0ac940 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.052+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3e4019040 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.052+0000 7fa3d37fe700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 0x7fa3d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:19.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.052+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6626+0+0 (secure 0 0 0) 0x7fa3e408a500 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.052+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 0x7fa3d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:19.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.053+0000 7fa3ead9d700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 0x7fa3d806d190 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009930 tx=0x7fa3e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:19.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.056+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3e4055400 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.242+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3ec074050 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.909+0000 7fa3d37fe700 1 -- 172.21.15.104:0/3091297654 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa3e40170d0 con 0x7fa3ec0c57f0 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 msgr2=0x7fa3d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 0x7fa3d806d190 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009930 tx=0x7fa3e0008040 comp rx=0 tx=0).stop 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 msgr2=0x7fa3ec1f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4005040 tx=0x7fa3e40014a0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 shutdown_connections 2023-06-10T18:53:19.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3ec0c57f0 0x7fa3ec1f0400 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3ec0c3960 0x7fa3ec1f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa3d806ace0 0x7fa3d806d190 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 --2- 172.21.15.104:0/3091297654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3ec1f3f20 0x7fa3ec0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 >> 172.21.15.104:0/3091297654 conn(0x7fa3ec06b8a0 msgr2=0x7fa3ec1e3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 shutdown_connections 2023-06-10T18:53:19.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:19.912+0000 7fa3f11cb700 1 -- 172.21.15.104:0/3091297654 wait complete. 2023-06-10T18:53:19.920 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:53:19.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:53:20.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.052+0000 7f77160ac700 1 Processor -- start 2023-06-10T18:53:20.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.052+0000 7f77160ac700 1 -- start start 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f77160ac700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7710143810 0x7f771006e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f77160ac700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7710071150 0x7f771006ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f77160ac700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771006a880 con 0x7f771006b6b0 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f77160ac700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77100ce540 con 0x7f7710071150 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f77160ac700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77100ccd10 con 0x7f7710143810 2023-06-10T18:53:20.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7710071150 0x7f771006ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f771006b6b0 0x7f7710070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60204/0 (socket says 172.21.15.104:60204) 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7710071150 0x7f771006ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34148/0 (socket says 172.21.15.104:34148) 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770effd700 1 -- 172.21.15.104:0/296333808 learned_addr learned my addr 172.21.15.104:0/296333808 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770f7fe700 1 --2- 172.21.15.104:0/296333808 >> v2:172.21.15.114:3300/0 conn(0x7f7710143810 0x7f771006e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 -- 172.21.15.104:0/296333808 >> v2:172.21.15.114:3300/0 conn(0x7f7710143810 msgr2=0x7f771006e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 --2- 172.21.15.104:0/296333808 >> v2:172.21.15.114:3300/0 conn(0x7f7710143810 0x7f771006e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 -- 172.21.15.104:0/296333808 >> v1:172.21.15.104:6789/0 conn(0x7f771006b6b0 legacy=0x7f7710070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 -- 172.21.15.104:0/296333808 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771006aad0 con 0x7f7710071150 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770e7fc700 1 -- 172.21.15.104:0/296333808 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452463045 0 0) 0x7f771006a880 con 0x7f771006b6b0 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.053+0000 7f770ffff700 1 --2- 172.21.15.104:0/296333808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f771006ca60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f77100cc940 tx=0x7f770800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1906b103a884927b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f770e7fc700 1 -- 172.21.15.104:0/296333808 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770800e070 con 0x7f7710071150 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f770e7fc700 1 -- 172.21.15.104:0/296333808 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770800bbd0 con 0x7f7710071150 2023-06-10T18:53:20.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f770e7fc700 1 -- 172.21.15.104:0/296333808 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708012430 con 0x7f7710071150 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 -- 172.21.15.104:0/296333808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 msgr2=0x7f771006ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 --2- 172.21.15.104:0/296333808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f771006ca60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f77100cc940 tx=0x7f770800b330 comp rx=0 tx=0).stop 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 -- 172.21.15.104:0/296333808 shutdown_connections 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 --2- 172.21.15.104:0/296333808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f771006ca60 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 --2- 172.21.15.104:0/296333808 >> v2:172.21.15.114:3300/0 conn(0x7f7710143810 0x7f771006e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 -- 172.21.15.104:0/296333808 >> 172.21.15.104:0/296333808 conn(0x7f77100cedc0 msgr2=0x7f77100cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 -- 172.21.15.104:0/296333808 shutdown_connections 2023-06-10T18:53:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.054+0000 7f77160ac700 1 -- 172.21.15.104:0/296333808 wait complete. 2023-06-10T18:53:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.055+0000 7f77160ac700 1 Processor -- start 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.055+0000 7f77160ac700 1 -- start start 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710143810 0x7f77100c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77100c5e90 0x7f771013b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7710139d40 con 0x7f7710071150 2023-06-10T18:53:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77100c7dc0 con 0x7f7710143810 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f77160ac700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771014e810 con 0x7f77100c5e90 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f770f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f770f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34150/0 (socket says 172.21.15.104:34150) 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.056+0000 7f770f7fe700 1 -- 172.21.15.104:0/1518745399 learned_addr learned my addr 172.21.15.104:0/1518745399 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770effd700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710143810 0x7f77100c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 -- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77100c5e90 msgr2=0x7f771013b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77100c5e90 0x7f771013b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 -- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710143810 msgr2=0x7f77100c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710143810 0x7f77100c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 -- 172.21.15.104:0/1518745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7708009a70 con 0x7f7710071150 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f770f7fe700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f770400d800 tx=0x7f770400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040125f0 con 0x7f7710071150 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7704012750 con 0x7f7710071150 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77101421c0 con 0x7f7710071150 2023-06-10T18:53:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704012b00 con 0x7f7710071150 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.057+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7710142780 con 0x7f7710071150 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.058+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77100c80a0 con 0x7f7710071150 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.058+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7704012c60 con 0x7f7710071150 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.059+0000 7f76f7fff700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.059+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f7704089960 con 0x7f7710071150 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.059+0000 7f770effd700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.059+0000 7f770effd700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7700000f60 tx=0x7f7700009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:20.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.063+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7704054990 con 0x7f7710071150 2023-06-10T18:53:20.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.250+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7710074050 con 0x7f7710071150 2023-06-10T18:53:20.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.912+0000 7f76f7fff700 1 -- 172.21.15.104:0/1518745399 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7704057fb0 con 0x7f7710071150 2023-06-10T18:53:20.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 msgr2=0x7f76fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7700000f60 tx=0x7f7700009040 comp rx=0 tx=0).stop 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 msgr2=0x7f7710139800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f770400d800 tx=0x7f770400dbc0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 shutdown_connections 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77100c5e90 0x7f771013b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710071150 0x7f7710139800 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f76fc06ace0 0x7f76fc06d190 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 --2- 172.21.15.104:0/1518745399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710143810 0x7f77100c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 >> 172.21.15.104:0/1518745399 conn(0x7f77100cedc0 msgr2=0x7f7710072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 shutdown_connections 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:20.915+0000 7f77160ac700 1 -- 172.21.15.104:0/1518745399 wait complete. 2023-06-10T18:53:20.922 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:53:20.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:53:21.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.052+0000 7f315b945700 1 Processor -- start 2023-06-10T18:53:21.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.052+0000 7f315b945700 1 -- start start 2023-06-10T18:53:21.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f315b945700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f315406e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f315b945700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540cec40 con 0x7f315406c800 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f315b945700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540cca90 con 0x7f3154143a40 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f315b945700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540ce960 con 0x7f3154070a10 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f3159ee2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f315406c800 0x7f315406cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60220/0 (socket says 172.21.15.104:60220) 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f3159ee2700 1 -- 172.21.15.104:0/2257054656 learned_addr learned my addr 172.21.15.104:0/2257054656 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041107491 0 0) 0x7f31540cec40 con 0x7f315406c800 2023-06-10T18:53:21.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31400035b0 con 0x7f315406c800 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.053+0000 7f31596e1700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f315406e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891893228 0 0) 0x7f31400035b0 con 0x7f315406c800 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 msgr2=0x7f315406e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f315406e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 >> v1:172.21.15.114:6789/0 conn(0x7f3154143a40 legacy=0x7f3154070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315406a9b0 con 0x7f315406c800 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f314c0019f0 con 0x7f315406c800 2023-06-10T18:53:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 >> v1:172.21.15.104:6789/0 conn(0x7f315406c800 legacy=0x7f315406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140003a30 0x7f3140003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140004850 0x7f3140004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f31400052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540ce960 con 0x7f3140004850 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314000bbd0 con 0x7f3140003a30 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314000bd90 con 0x7f3154070a10 2023-06-10T18:53:21.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140004850 0x7f3140004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f31596e1700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140003a30 0x7f3140003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3159ee2700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f31400052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 -- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 msgr2=0x7f31400052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3154070a10 0x7f31400052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 -- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140003a30 msgr2=0x7f3140003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140003a30 0x7f3140003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 -- 172.21.15.104:0/2257054656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400035b0 con 0x7f3140004850 2023-06-10T18:53:21.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f3158ee0700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140004850 0x7f3140004cc0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fd0 tx=0x7f3140005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ca4f445f0c6442 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31500052d0 con 0x7f3140004850 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3150005470 con 0x7f3140004850 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.054+0000 7f314bfff700 1 -- 172.21.15.104:0/2257054656 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31500038c0 con 0x7f3140004850 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 -- 172.21.15.104:0/2257054656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140004850 msgr2=0x7f3140004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 --2- 172.21.15.104:0/2257054656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3140004850 0x7f3140004cc0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fd0 tx=0x7f3140005880 comp rx=0 tx=0).stop 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f3159ee2700 1 -- 172.21.15.104:0/2257054656 reap_dead start 2023-06-10T18:53:21.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 -- 172.21.15.104:0/2257054656 shutdown_connections 2023-06-10T18:53:21.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 -- 172.21.15.104:0/2257054656 >> 172.21.15.104:0/2257054656 conn(0x7f31540ceec0 msgr2=0x7f31540cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:21.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 -- 172.21.15.104:0/2257054656 shutdown_connections 2023-06-10T18:53:21.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.055+0000 7f315b945700 1 -- 172.21.15.104:0/2257054656 wait complete. 2023-06-10T18:53:21.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.056+0000 7f315b945700 1 Processor -- start 2023-06-10T18:53:21.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.056+0000 7f315b945700 1 -- start start 2023-06-10T18:53:21.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140004850 0x7f31541f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31540c71c0 0x7f3154135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3154135b80 con 0x7f31541e4b20 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540c7630 con 0x7f3140004850 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f315b945700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31541e3980 con 0x7f31540c71c0 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34172/0 (socket says 172.21.15.104:34172) 2023-06-10T18:53:21.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 -- 172.21.15.104:0/2249423080 learned_addr learned my addr 172.21.15.104:0/2249423080 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3158ee0700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31540c71c0 0x7f3154135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f31596e1700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140004850 0x7f31541f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 -- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31540c71c0 msgr2=0x7f3154135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31540c71c0 0x7f3154135640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 -- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140004850 msgr2=0x7f31541f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140004850 0x7f31541f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 -- 172.21.15.104:0/2249423080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540cd0a0 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.057+0000 7f3159ee2700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f31540709a0 tx=0x7f314c007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.058+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c017070 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.058+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c00ebf0 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.058+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31541f0610 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.058+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00eda0 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.058+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31541f0bd0 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.059+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3154074050 con 0x7f31541e4b20 2023-06-10T18:53:21.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.059+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f314c013420 con 0x7f31541e4b20 2023-06-10T18:53:21.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.059+0000 7f3149ffb700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 0x7f313806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.060+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f314c088eb0 con 0x7f31541e4b20 2023-06-10T18:53:21.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.060+0000 7f31596e1700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 0x7f313806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.060+0000 7f31596e1700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 0x7f313806d190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3144009c60 tx=0x7f3144009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.064+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f314c0541e0 con 0x7f31541e4b20 2023-06-10T18:53:21.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.250+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f31540c9280 con 0x7f31541e4b20 2023-06-10T18:53:21.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.253+0000 7f3149ffb700 1 -- 172.21.15.104:0/2249423080 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f314c057800 con 0x7f31541e4b20 2023-06-10T18:53:21.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.255+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 msgr2=0x7f313806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.255+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 0x7f313806d190 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3144009c60 tx=0x7f3144009210 comp rx=0 tx=0).stop 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 msgr2=0x7f315413d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f31540709a0 tx=0x7f314c007c90 comp rx=0 tx=0).stop 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 shutdown_connections 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313806ace0 0x7f313806d190 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3140004850 0x7f31541f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31541e4b20 0x7f315413d870 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 --2- 172.21.15.104:0/2249423080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31540c71c0 0x7f3154135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 >> 172.21.15.104:0/2249423080 conn(0x7f31540ceec0 msgr2=0x7f31540cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 shutdown_connections 2023-06-10T18:53:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.256+0000 7f315b945700 1 -- 172.21.15.104:0/2249423080 wait complete. 2023-06-10T18:53:21.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync' 2023-06-10T18:53:21.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.386+0000 7fb4163d0700 1 Processor -- start 2023-06-10T18:53:21.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.386+0000 7fb4163d0700 1 -- start start 2023-06-10T18:53:21.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb4163d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb410163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb4163d0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4100c9e60 con 0x7fb4100aeaa0 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb4163d0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4100aee70 con 0x7fb4100cd850 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb4163d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100c9bc0 con 0x7fb4100cb300 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb40ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb4100aeaa0 0x7fb4100cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60236/0 (socket says 172.21.15.104:60236) 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb40ffff700 1 -- 172.21.15.104:0/2506715513 learned_addr learned my addr 172.21.15.104:0/2506715513 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959333694 0 0) 0x7fb4100c9e60 con 0x7fb4100aeaa0 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb41496d700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb410163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.387+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f80057c0 con 0x7fb4100aeaa0 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603349362 0 0) 0x7fb3f80057c0 con 0x7fb4100aeaa0 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 msgr2=0x7fb410163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb410163c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 >> v1:172.21.15.114:6789/0 conn(0x7fb4100cd850 legacy=0x7fb41006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41006e540 con 0x7fb4100aeaa0 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb4040017f0 con 0x7fb4100aeaa0 2023-06-10T18:53:21.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 >> v1:172.21.15.104:6789/0 conn(0x7fb4100aeaa0 legacy=0x7fb4100cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb3f8005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3f80061c0 0x7fb3f8006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f8006fa0 0x7fb3f8007410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100c9bc0 con 0x7fb3f80061c0 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f800dd00 con 0x7fb3f8006fa0 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f800dec0 con 0x7fb4100cb300 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3f80061c0 0x7fb3f8006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40ffff700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb3f8005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb41496d700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f8006fa0 0x7fb3f8007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 -- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 msgr2=0x7fb3f8005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4100cb300 0x7fb3f8005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 -- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f8006fa0 msgr2=0x7fb3f8007410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f8006fa0 0x7fb3f8007410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 -- 172.21.15.104:0/2506715513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80057c0 con 0x7fb3f80061c0 2023-06-10T18:53:21.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3f80061c0 0x7fb3f8006a60 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb400007fd0 tx=0x7fb3f8007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b10bbf0c33dad3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4000012f0 con 0x7fb3f80061c0 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4000052f0 con 0x7fb3f80061c0 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.388+0000 7fb40effd700 1 -- 172.21.15.104:0/2506715513 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400006860 con 0x7fb3f80061c0 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 -- 172.21.15.104:0/2506715513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3f80061c0 msgr2=0x7fb3f8006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 --2- 172.21.15.104:0/2506715513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3f80061c0 0x7fb3f8006a60 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb400007fd0 tx=0x7fb3f8007a90 comp rx=0 tx=0).stop 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb41496d700 1 -- 172.21.15.104:0/2506715513 reap_dead start 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 -- 172.21.15.104:0/2506715513 shutdown_connections 2023-06-10T18:53:21.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 -- 172.21.15.104:0/2506715513 >> 172.21.15.104:0/2506715513 conn(0x7fb41006b8a0 msgr2=0x7fb4100cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:21.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 -- 172.21.15.104:0/2506715513 shutdown_connections 2023-06-10T18:53:21.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.389+0000 7fb4163d0700 1 -- 172.21.15.104:0/2506715513 wait complete. 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.390+0000 7fb4163d0700 1 Processor -- start 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.390+0000 7fb4163d0700 1 -- start start 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f80061c0 0x7fb410146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4101f13d0 0x7fb41006d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4101f0fb0 con 0x7fb410149220 2023-06-10T18:53:21.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100c5640 con 0x7fb3f80061c0 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb4163d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb410146640 con 0x7fb4101f13d0 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34190/0 (socket says 172.21.15.104:34190) 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 -- 172.21.15.104:0/3258963572 learned_addr learned my addr 172.21.15.104:0/3258963572 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40ffff700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f80061c0 0x7fb410146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb41496d700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4101f13d0 0x7fb41006d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 -- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4101f13d0 msgr2=0x7fb41006d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4101f13d0 0x7fb41006d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 -- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f80061c0 msgr2=0x7fb410146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f80061c0 0x7fb410146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.391+0000 7fb40f7fe700 1 -- 172.21.15.104:0/3258963572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4101f0c10 con 0x7fb410149220 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb40f7fe700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb400005690 tx=0x7fb400001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400003880 con 0x7fb410149220 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb400003a00 con 0x7fb410149220 2023-06-10T18:53:21.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4100c9560 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400003db0 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.392+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4100c5b30 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.393+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb410074050 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.393+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb40001e450 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.393+0000 7fb40cff9700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 0x7fb3fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.394+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fb40008a610 con 0x7fb410149220 2023-06-10T18:53:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.394+0000 7fb40ffff700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 0x7fb3fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:21.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.394+0000 7fb40ffff700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 0x7fb3fc06d190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb404000ef0 tx=0x7fb404009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:21.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.398+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb400055880 con 0x7fb410149220 2023-06-10T18:53:21.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.662+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync"]} v 0) v1 -- 0x7fb41006a1f0 con 0x7fb410149220 2023-06-10T18:53:21.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.913+0000 7fb40cff9700 1 -- 172.21.15.104:0/3258963572 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_create_fsync"]}]=0 v426) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb400058ea0 con 0x7fb410149220 2023-06-10T18:53:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 msgr2=0x7fb3fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 0x7fb3fc06d190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb404000ef0 tx=0x7fb404009040 comp rx=0 tx=0).stop 2023-06-10T18:53:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 msgr2=0x7fb4100c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb400005690 tx=0x7fb400001120 comp rx=0 tx=0).stop 2023-06-10T18:53:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 shutdown_connections 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3fc06ace0 0x7fb3fc06d190 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3f80061c0 0x7fb410146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4101f13d0 0x7fb41006d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 --2- 172.21.15.104:0/3258963572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410149220 0x7fb4100c9020 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 >> 172.21.15.104:0/3258963572 conn(0x7fb41006b8a0 msgr2=0x7fb4100cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 shutdown_connections 2023-06-10T18:53:21.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:21.915+0000 7fb4163d0700 1 -- 172.21.15.104:0/3258963572 wait complete. 2023-06-10T18:53:21.933 INFO:tasks.cephfs_test_runner:test_create_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:53:21.933 INFO:tasks.cephfs_test_runner:Starting test: test_dir_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:53:21.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync' 2023-06-10T18:53:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.047+0000 7f522c814700 1 Processor -- start 2023-06-10T18:53:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.047+0000 7f522c814700 1 -- start start 2023-06-10T18:53:22.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f522c814700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f522806e4b0 0x7f522806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f522c814700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5228069e70 0x7f52280ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f522c814700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522806d290 con 0x7f522806e0e0 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f522c814700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5228071640 con 0x7f522806e4b0 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f522c814700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52280aef00 con 0x7f5228069e70 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f522806e4b0 0x7f522806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f522806e4b0 0x7f522806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34200/0 (socket says 172.21.15.104:34200) 2023-06-10T18:53:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 -- 172.21.15.104:0/18757394 learned_addr learned my addr 172.21.15.104:0/18757394 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 -- 172.21.15.104:0/18757394 >> v2:172.21.15.114:3300/0 conn(0x7f5228069e70 msgr2=0x7f52280ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 --2- 172.21.15.104:0/18757394 >> v2:172.21.15.114:3300/0 conn(0x7f5228069e70 0x7f52280ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.048+0000 7f5225d9b700 1 -- 172.21.15.104:0/18757394 >> v1:172.21.15.114:6789/0 conn(0x7f522806e0e0 legacy=0x7f52280c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f5225d9b700 1 -- 172.21.15.104:0/18757394 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280ca2d0 con 0x7f522806e4b0 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f5225d9b700 1 --2- 172.21.15.104:0/18757394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e4b0 0x7f522806cd50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5228071210 tx=0x7f5218009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42794e2c4a126780 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522559a700 1 -- 172.21.15.104:0/18757394 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218004230 con 0x7f522806e4b0 2023-06-10T18:53:22.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522559a700 1 -- 172.21.15.104:0/18757394 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5218004390 con 0x7f522806e4b0 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522559a700 1 -- 172.21.15.104:0/18757394 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52180036a0 con 0x7f522806e4b0 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 -- 172.21.15.104:0/18757394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e4b0 msgr2=0x7f522806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 --2- 172.21.15.104:0/18757394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e4b0 0x7f522806cd50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5228071210 tx=0x7f5218009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 -- 172.21.15.104:0/18757394 shutdown_connections 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 --2- 172.21.15.104:0/18757394 >> v2:172.21.15.114:3300/0 conn(0x7f5228069e70 0x7f52280ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 --2- 172.21.15.104:0/18757394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e4b0 0x7f522806cd50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 -- 172.21.15.104:0/18757394 >> 172.21.15.104:0/18757394 conn(0x7f522806b8a0 msgr2=0x7f52280cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:22.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 -- 172.21.15.104:0/18757394 shutdown_connections 2023-06-10T18:53:22.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.049+0000 7f522c814700 1 -- 172.21.15.104:0/18757394 wait complete. 2023-06-10T18:53:22.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.051+0000 7f522c814700 1 Processor -- start 2023-06-10T18:53:22.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.051+0000 7f522c814700 1 -- start start 2023-06-10T18:53:22.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5228069e70 0x7f5228148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522806e4b0 0x7f52280b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280b04a0 con 0x7f522806e0e0 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228148f70 con 0x7f5228069e70 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52280af890 con 0x7f522806e4b0 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34204/0 (socket says 172.21.15.104:34204) 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 -- 172.21.15.104:0/340235873 learned_addr learned my addr 172.21.15.104:0/340235873 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522659c700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5228069e70 0x7f5228148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5226d9d700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522806e4b0 0x7f52280b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 -- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522806e4b0 msgr2=0x7f52280b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522806e4b0 0x7f52280b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 -- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5228069e70 msgr2=0x7f5228148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5228069e70 0x7f5228148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 -- 172.21.15.104:0/340235873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218009550 con 0x7f522806e0e0 2023-06-10T18:53:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5225d9b700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5218009130 tx=0x7f5218004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218003dc0 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521801f070 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52280afb70 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218011870 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.052+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52280b0130 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.053+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5228074050 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.054+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5218021030 con 0x7f522806e0e0 2023-06-10T18:53:22.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.054+0000 7f5216ffd700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 0x7f520806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:22.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.054+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f521808a7e0 con 0x7f522806e0e0 2023-06-10T18:53:22.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.054+0000 7f522659c700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 0x7f520806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:22.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.055+0000 7f522659c700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 0x7f520806d1c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f521c006fd0 tx=0x7f521c0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:22.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.058+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5218055a50 con 0x7f522806e0e0 2023-06-10T18:53:22.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.320+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync"]} v 0) v1 -- 0x7f52280cd7a0 con 0x7f522806e0e0 2023-06-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.917+0000 7f5216ffd700 1 -- 172.21.15.104:0/340235873 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync"]}]=0 v427) v1 ==== 149+0+0 (secure 0 0 0) 0x7f52180119d0 con 0x7f522806e0e0 2023-06-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 msgr2=0x7f520806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 0x7f520806d1c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f521c006fd0 tx=0x7f521c0090e0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 msgr2=0x7f52280af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5218009130 tx=0x7f5218004750 comp rx=0 tx=0).stop 2023-06-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 shutdown_connections 2023-06-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f520806ad10 0x7f520806d1c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5228069e70 0x7f5228148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522806e4b0 0x7f52280b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 --2- 172.21.15.104:0/340235873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f522806e0e0 0x7f52280af040 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 >> 172.21.15.104:0/340235873 conn(0x7f522806b8a0 msgr2=0x7f52280ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 shutdown_connections 2023-06-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:22.919+0000 7f522c814700 1 -- 172.21.15.104:0/340235873 wait complete. 2023-06-10T18:53:22.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:53:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.053+0000 7fa50237d700 1 Processor -- start 2023-06-10T18:53:23.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.053+0000 7fa50237d700 1 -- start start 2023-06-10T18:53:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50237d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50237d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4fc06cde0 0x7fa4fc0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50237d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc06e250 con 0x7fa4fc15c580 2023-06-10T18:53:23.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50237d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc15c950 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50237d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0b1e80 con 0x7fa4fc06cde0 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34212/0 (socket says 172.21.15.104:34212) 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 -- 172.21.15.104:0/1907880217 learned_addr learned my addr 172.21.15.104:0/1907880217 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 -- 172.21.15.104:0/1907880217 >> v2:172.21.15.114:3300/0 conn(0x7fa4fc06cde0 msgr2=0x7fa4fc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa50091a700 1 --2- 172.21.15.104:0/1907880217 >> v2:172.21.15.114:3300/0 conn(0x7fa4fc06cde0 0x7fa4fc0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 --2- 172.21.15.104:0/1907880217 >> v2:172.21.15.114:3300/0 conn(0x7fa4fc06cde0 0x7fa4fc0ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 -- 172.21.15.104:0/1907880217 >> v1:172.21.15.114:6789/0 conn(0x7fa4fc15c580 legacy=0x7fa4fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 -- 172.21.15.104:0/1907880217 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc06d920 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4fb7fe700 1 --2- 172.21.15.104:0/1907880217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0cbc60 tx=0x7fa4ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812a0e7d5b94772a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.054+0000 7fa4faffd700 1 -- 172.21.15.104:0/1907880217 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec004230 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa4faffd700 1 -- 172.21.15.104:0/1907880217 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec004390 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa4faffd700 1 -- 172.21.15.104:0/1907880217 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0036a0 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 -- 172.21.15.104:0/1907880217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc06e6e0 msgr2=0x7fa4fc0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 --2- 172.21.15.104:0/1907880217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0cbc60 tx=0x7fa4ec009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 -- 172.21.15.104:0/1907880217 shutdown_connections 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 --2- 172.21.15.104:0/1907880217 >> v2:172.21.15.114:3300/0 conn(0x7fa4fc06cde0 0x7fa4fc0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 --2- 172.21.15.104:0/1907880217 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b1780 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 -- 172.21.15.104:0/1907880217 >> 172.21.15.104:0/1907880217 conn(0x7fa4fc06b8a0 msgr2=0x7fa4fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 -- 172.21.15.104:0/1907880217 shutdown_connections 2023-06-10T18:53:23.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.055+0000 7fa50237d700 1 -- 172.21.15.104:0/1907880217 wait complete. 2023-06-10T18:53:23.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.056+0000 7fa50237d700 1 Processor -- start 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.056+0000 7fa50237d700 1 -- start start 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 0x7fa4fc159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1f61e0 con 0x7fa4fc15c580 2023-06-10T18:53:23.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc15a320 con 0x7fa4fc06e6e0 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50237d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1e2e90 con 0x7fa4fc06cde0 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50091a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50091a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34220/0 (socket says 172.21.15.104:34220) 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.057+0000 7fa50091a700 1 -- 172.21.15.104:0/1116179732 learned_addr learned my addr 172.21.15.104:0/1116179732 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4fbfff700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 0x7fa4fc159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4fb7fe700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 -- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 msgr2=0x7fa4fc159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 0x7fa4fc159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 -- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 msgr2=0x7fa4fc0b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 -- 172.21.15.104:0/1116179732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec009550 con 0x7fa4fc15c580 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4fbfff700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 0x7fa4fc159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4fb7fe700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50091a700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400cd70 tx=0x7fa4f400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4003a20 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f4003b80 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc1e3170 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4004020 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.058+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc0c4160 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.059+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc074050 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.059+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa4f4004180 con 0x7fa4fc15c580 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.060+0000 7fa4f8ff9700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 0x7fa4e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.060+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fa4f4012070 con 0x7fa4fc15c580 2023-06-10T18:53:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.060+0000 7fa4fbfff700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 0x7fa4e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.060+0000 7fa4fbfff700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 0x7fa4e806d190 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000a8b0 tx=0x7fa4f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.064+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa4f4054d60 con 0x7fa4fc15c580 2023-06-10T18:53:23.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.253+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa4fc0afbd0 con 0x7fa4fc15c580 2023-06-10T18:53:23.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.258+0000 7fa4f8ff9700 1 -- 172.21.15.104:0/1116179732 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa4f4058380 con 0x7fa4fc15c580 2023-06-10T18:53:23.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 msgr2=0x7fa4e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 0x7fa4e806d190 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000a8b0 tx=0x7fa4f0008040 comp rx=0 tx=0).stop 2023-06-10T18:53:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 msgr2=0x7fa4fc1f5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400cd70 tx=0x7fa4f400a570 comp rx=0 tx=0).stop 2023-06-10T18:53:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 shutdown_connections 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4fc15c580 0x7fa4fc1f5370 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4fc06cde0 0x7fa4fc159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4e806ace0 0x7fa4e806d190 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 --2- 172.21.15.104:0/1116179732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4fc06e6e0 0x7fa4fc0b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 >> 172.21.15.104:0/1116179732 conn(0x7fa4fc06b8a0 msgr2=0x7fa4fc0b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 shutdown_connections 2023-06-10T18:53:23.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.260+0000 7fa50237d700 1 -- 172.21.15.104:0/1116179732 wait complete. 2023-06-10T18:53:23.277 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:53:23.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:53:23.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:53:23.360 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:53:23.361 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:23.361 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:23.379 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:53:23.380 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:23.380 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:23.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:53:23.419 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:23.420 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:23.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:53:23.437 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:23.437 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:23.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:53:23.478 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:53:23.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:23.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.593+0000 7fd0f2a91700 1 Processor -- start 2023-06-10T18:53:23.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.593+0000 7fd0f2a91700 1 -- start start 2023-06-10T18:53:23.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0ec070a20 0x7fd0ec06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec070df0 0x7fd0ec1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0ec158ce0 0x7fd0ec159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec06ae00 con 0x7fd0ec158ce0 2023-06-10T18:53:23.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec069e70 con 0x7fd0ec070a20 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f2a91700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1595f0 con 0x7fd0ec070df0 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0ec158ce0 0x7fd0ec159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd0ec158ce0 0x7fd0ec159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34228/0 (socket says 172.21.15.104:34228) 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 -- 172.21.15.104:0/4203204567 learned_addr learned my addr 172.21.15.104:0/4203204567 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f082d700 1 --2- 172.21.15.104:0/4203204567 >> v2:172.21.15.114:3300/0 conn(0x7fd0ec070a20 0x7fd0ec06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0ebfff700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec070df0 0x7fd0ec1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 -- 172.21.15.104:0/4203204567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec070df0 msgr2=0x7fd0ec1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec070df0 0x7fd0ec1587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 -- 172.21.15.104:0/4203204567 >> v2:172.21.15.114:3300/0 conn(0x7fd0ec070a20 msgr2=0x7fd0ec06a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 --2- 172.21.15.104:0/4203204567 >> v2:172.21.15.114:3300/0 conn(0x7fd0ec070a20 0x7fd0ec06a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 -- 172.21.15.104:0/4203204567 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec1598d0 con 0x7fd0ec158ce0 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.594+0000 7fd0f102e700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec158ce0 0x7fd0ec159c40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec072330 tx=0x7fd0e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1b7eba91cd6613 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0eb7fe700 1 -- 172.21.15.104:0/4203204567 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400e070 con 0x7fd0ec158ce0 2023-06-10T18:53:23.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0eb7fe700 1 -- 172.21.15.104:0/4203204567 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e400b980 con 0x7fd0ec158ce0 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0eb7fe700 1 -- 172.21.15.104:0/4203204567 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400bd30 con 0x7fd0ec158ce0 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 -- 172.21.15.104:0/4203204567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec158ce0 msgr2=0x7fd0ec159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec158ce0 0x7fd0ec159c40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec072330 tx=0x7fd0e400b330 comp rx=0 tx=0).stop 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 -- 172.21.15.104:0/4203204567 shutdown_connections 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec158ce0 0x7fd0ec159c40 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/4203204567 >> v2:172.21.15.114:3300/0 conn(0x7fd0ec070a20 0x7fd0ec06a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/4203204567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec070df0 0x7fd0ec1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 -- 172.21.15.104:0/4203204567 >> 172.21.15.104:0/4203204567 conn(0x7fd0ec0b1610 msgr2=0x7fd0ec0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:23.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 -- 172.21.15.104:0/4203204567 shutdown_connections 2023-06-10T18:53:23.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.595+0000 7fd0f2a91700 1 -- 172.21.15.104:0/4203204567 wait complete. 2023-06-10T18:53:23.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.596+0000 7fd0f2a91700 1 Processor -- start 2023-06-10T18:53:23.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.596+0000 7fd0f2a91700 1 -- start start 2023-06-10T18:53:23.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 0x7fd0ec1467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec158ce0 0x7fd0ec1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1564c0 con 0x7fd0ec070df0 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec155c20 con 0x7fd0ec070a20 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0f2a91700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec155f00 con 0x7fd0ec158ce0 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0ebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34232/0 (socket says 172.21.15.104:34232) 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.597+0000 7fd0ebfff700 1 -- 172.21.15.104:0/2168926221 learned_addr learned my addr 172.21.15.104:0/2168926221 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:23.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0f082d700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 0x7fd0ec1467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0f102e700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec158ce0 0x7fd0ec1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 -- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec158ce0 msgr2=0x7fd0ec1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec158ce0 0x7fd0ec1555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 -- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 msgr2=0x7fd0ec1467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 0x7fd0ec1467f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 -- 172.21.15.104:0/2168926221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4009b30 con 0x7fd0ec070df0 2023-06-10T18:53:23.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0f082d700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 0x7fd0ec1467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0ebfff700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c7f0 tx=0x7fd0d800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8011800 con 0x7fd0ec070df0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8011960 con 0x7fd0ec070df0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec1561e0 con 0x7fd0ec070df0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8011cf0 con 0x7fd0ec070df0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.598+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec14ec80 con 0x7fd0ec070df0 2023-06-10T18:53:23.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.599+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec071780 con 0x7fd0ec070df0 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.599+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd0d8018360 con 0x7fd0ec070df0 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.600+0000 7fd0e97fa700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 0x7fd0d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.600+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fd0d8089180 con 0x7fd0ec070df0 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.600+0000 7fd0f082d700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 0x7fd0d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.601+0000 7fd0f082d700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 0x7fd0d006d190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000a9b0 tx=0x7fd0e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:23.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.604+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd0d80543f0 con 0x7fd0ec070df0 2023-06-10T18:53:23.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.831+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0ec0c6ee0 con 0x7fd0ec070df0 2023-06-10T18:53:23.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.831+0000 7fd0e97fa700 1 -- 172.21.15.104:0/2168926221 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fd0d801fdf0 con 0x7fd0ec070df0 2023-06-10T18:53:23.837 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:23.837 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3615198358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3615198358},{"type":"v1","addr":"172.21.15.114:6836","nonce":3615198358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49},{"gid":6376,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4269844457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4269844457},{"type":"v1","addr":"172.21.15.114:6837","nonce":4269844457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2153970483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2153970483},{"type":"v1","addr":"172.21.15.104:6834","nonce":2153970483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7152,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3001331585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3001331585},{"type":"v1","addr":"172.21.15.104:6835","nonce":3001331585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[]} 2023-06-10T18:53:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 msgr2=0x7fd0d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 0x7fd0d006d190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000a9b0 tx=0x7fd0e0008040 comp rx=0 tx=0).stop 2023-06-10T18:53:23.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 msgr2=0x7fd0ec155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:23.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c7f0 tx=0x7fd0d800cbb0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 shutdown_connections 2023-06-10T18:53:23.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd0d006ace0 0x7fd0d006d190 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0ec070a20 0x7fd0ec1467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0ec158ce0 0x7fd0ec1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 --2- 172.21.15.104:0/2168926221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd0ec070df0 0x7fd0ec155080 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 >> 172.21.15.104:0/2168926221 conn(0x7fd0ec0b1610 msgr2=0x7fd0ec072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 shutdown_connections 2023-06-10T18:53:23.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:23.833+0000 7fd0f2a91700 1 -- 172.21.15.104:0/2168926221 wait complete. 2023-06-10T18:53:23.842 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 49 2023-06-10T18:53:23.851 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:53:23.852 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:53:23.852 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:53:23.852 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:23.852 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:53:23.852 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:53:23.853 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:53:23.853 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:23.853 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:53:23.853 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:53:23.854 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:53:23.854 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:23.854 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:53:23.854 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:53:23.854 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:53:23.855 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:23.862 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:53:23.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:53:23.863 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:53:23.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:53:23.865 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:53:23.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:53:23.866 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:53:23.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:53:23.904 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:53:23.905 DEBUG:teuthology.parallel:result is None 2023-06-10T18:53:23.906 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:53:23.906 DEBUG:teuthology.parallel:result is None 2023-06-10T18:53:23.908 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:53:23.908 DEBUG:teuthology.parallel:result is None 2023-06-10T18:53:23.909 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:53:23.909 DEBUG:teuthology.parallel:result is None 2023-06-10T18:53:23.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:53:23.984 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:53:23.984 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:53:23.996 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:53:23.996 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:53:24.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.036+0000 7f47fd788700 1 Processor -- start 2023-06-10T18:53:24.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.036+0000 7f47fd788700 1 -- start start 2023-06-10T18:53:24.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.037+0000 7f47fd788700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.037+0000 7f47fd788700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 0x7f47f815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.037+0000 7f47fd788700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80b6070 con 0x7f47f815a420 2023-06-10T18:53:24.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.037+0000 7f47fd788700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80ced10 con 0x7f47f80ce890 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.037+0000 7f47fd788700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80b1d70 con 0x7f47f80b4ff0 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f77fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 0x7f47f815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f6ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f47f815a420 0x7f47f80b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:37934/0 (socket says 172.21.15.104:37934) 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f77fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 0x7f47f815a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:49998/0 (socket says 172.21.15.104:49998) 2023-06-10T18:53:24.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f6ffd700 1 -- 172.21.15.104:0/959896802 learned_addr learned my addr 172.21.15.104:0/959896802 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:24.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:53644/0 (socket says 172.21.15.104:53644) 2023-06-10T18:53:24.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f5ffb700 1 -- 172.21.15.104:0/959896802 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192885020 0 0) 0x7f47f80b6070 con 0x7f47f815a420 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 -- 172.21.15.104:0/959896802 >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 msgr2=0x7f47f815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 --2- 172.21.15.104:0/959896802 >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 0x7f47f815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 -- 172.21.15.104:0/959896802 >> v1:172.21.15.114:6789/0 conn(0x7f47f815a420 legacy=0x7f47f80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 -- 172.21.15.104:0/959896802 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80b5460 con 0x7f47f80b4ff0 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f67fc700 1 --2- 172.21.15.104:0/959896802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009840 tx=0x7f47e8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2037102cfd3216b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:24.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f5ffb700 1 -- 172.21.15.104:0/959896802 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8011830 con 0x7f47f80b4ff0 2023-06-10T18:53:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f5ffb700 1 -- 172.21.15.104:0/959896802 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e8011990 con 0x7f47f80b4ff0 2023-06-10T18:53:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.038+0000 7f47f5ffb700 1 -- 172.21.15.104:0/959896802 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8011d20 con 0x7f47f80b4ff0 2023-06-10T18:53:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 -- 172.21.15.104:0/959896802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 msgr2=0x7f47f80ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 --2- 172.21.15.104:0/959896802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009840 tx=0x7f47e8009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:24.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 -- 172.21.15.104:0/959896802 shutdown_connections 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 --2- 172.21.15.104:0/959896802 >> v2:172.21.15.114:3300/0 conn(0x7f47f80ce890 0x7f47f815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 --2- 172.21.15.104:0/959896802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80b4ff0 0x7f47f80ce350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 -- 172.21.15.104:0/959896802 >> 172.21.15.104:0/959896802 conn(0x7f47f8089fd0 msgr2=0x7f47f80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 -- 172.21.15.104:0/959896802 shutdown_connections 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.039+0000 7f47fd788700 1 -- 172.21.15.104:0/959896802 wait complete. 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.041+0000 7f47fd788700 1 Processor -- start 2023-06-10T18:53:24.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.041+0000 7f47fd788700 1 -- start start 2023-06-10T18:53:24.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.041+0000 7f47fd788700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.041+0000 7f47fd788700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f80ce890 0x7f47f80cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.041+0000 7f47fd788700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f815a420 0x7f47f80c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47fd788700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80ce120 con 0x7f47f80b4ff0 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47fd788700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f814d4b0 con 0x7f47f80ce890 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47fd788700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80c87a0 con 0x7f47f815a420 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34266/0 (socket says 172.21.15.104:34266) 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 -- 172.21.15.104:0/4173794821 learned_addr learned my addr 172.21.15.104:0/4173794821 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:24.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f77fe700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f815a420 0x7f47f80c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f67fc700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f80ce890 0x7f47f80cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 -- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f815a420 msgr2=0x7f47f80c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f815a420 0x7f47f80c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 -- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f80ce890 msgr2=0x7f47f80cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f80ce890 0x7f47f80cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 -- 172.21.15.104:0/4173794821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80094a0 con 0x7f47f80b4ff0 2023-06-10T18:53:24.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47f6ffd700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0011e0 tx=0x7f47ec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec004c80 con 0x7f47f80b4ff0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec004e00 con 0x7f47f80b4ff0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f80c8a40 con 0x7f47f80b4ff0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec007310 con 0x7f47f80b4ff0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.042+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f80c9000 con 0x7f47f80b4ff0 2023-06-10T18:53:24.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.043+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f8074050 con 0x7f47f80b4ff0 2023-06-10T18:53:24.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.044+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f47ec007470 con 0x7f47f80b4ff0 2023-06-10T18:53:24.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.044+0000 7f47df7fe700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 0x7f47e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:24.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.044+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f47ec089210 con 0x7f47f80b4ff0 2023-06-10T18:53:24.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.044+0000 7f47f67fc700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 0x7f47e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:24.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.045+0000 7f47f67fc700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 0x7f47e406d190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f47e8000c00 tx=0x7f47e801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:24.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.048+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f47ec054370 con 0x7f47f80b4ff0 2023-06-10T18:53:24.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:24.252+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f47f80b45a0 con 0x7f47f80b4ff0 2023-06-10T18:53:25.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.035+0000 7f47df7fe700 1 -- 172.21.15.104:0/4173794821 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7f47ec057990 con 0x7f47f80b4ff0 2023-06-10T18:53:25.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 msgr2=0x7f47e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 0x7f47e406d190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f47e8000c00 tx=0x7f47e801c040 comp rx=0 tx=0).stop 2023-06-10T18:53:25.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 msgr2=0x7f47f80c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0011e0 tx=0x7f47ec00cae0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 shutdown_connections 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f815a420 0x7f47f80c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47f80b4ff0 0x7f47f80c68d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47e406ace0 0x7f47e406d190 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 --2- 172.21.15.104:0/4173794821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f80ce890 0x7f47f80cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 >> 172.21.15.104:0/4173794821 conn(0x7f47f8089fd0 msgr2=0x7f47f8157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 shutdown_connections 2023-06-10T18:53:25.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.037+0000 7f47fd788700 1 -- 172.21.15.104:0/4173794821 wait complete. 2023-06-10T18:53:25.045 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:53:25.056 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:53:25.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.171+0000 7f789d6cb700 1 Processor -- start 2023-06-10T18:53:25.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.171+0000 7f789d6cb700 1 -- start start 2023-06-10T18:53:25.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f789d6cb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78980b1e30 0x7f78980d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f789d6cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 0x7f78980cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f789d6cb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789806bfd0 con 0x7f78980d0a30 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f789d6cb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78980d0ed0 con 0x7f78980b1e30 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f789d6cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789806c2b0 con 0x7f789806b9f0 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78980b1e30 0x7f78980d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f78980b1e30 0x7f78980d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34278/0 (socket says 172.21.15.104:34278) 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 -- 172.21.15.104:0/331212945 learned_addr learned my addr 172.21.15.104:0/331212945 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f78977fe700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 0x7f78980cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 -- 172.21.15.104:0/331212945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 msgr2=0x7f78980cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 0x7f78980cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 -- 172.21.15.104:0/331212945 >> v1:172.21.15.114:6789/0 conn(0x7f78980d0a30 legacy=0x7f789806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 -- 172.21.15.104:0/331212945 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78980ccf70 con 0x7f78980b1e30 2023-06-10T18:53:25.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f78977fe700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 0x7f78980cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7896ffd700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f78980d04f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f788c0098f0 tx=0x7f788c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859e8bccc1606337 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7895ffb700 1 -- 172.21.15.104:0/331212945 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c004230 con 0x7f78980b1e30 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7895ffb700 1 -- 172.21.15.104:0/331212945 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c004390 con 0x7f78980b1e30 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.172+0000 7f7895ffb700 1 -- 172.21.15.104:0/331212945 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0036a0 con 0x7f78980b1e30 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 -- 172.21.15.104:0/331212945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 msgr2=0x7f78980d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f78980d04f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f788c0098f0 tx=0x7f788c009c00 comp rx=0 tx=0).stop 2023-06-10T18:53:25.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 -- 172.21.15.104:0/331212945 shutdown_connections 2023-06-10T18:53:25.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f78980d04f0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 --2- 172.21.15.104:0/331212945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f789806b9f0 0x7f78980cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 -- 172.21.15.104:0/331212945 >> 172.21.15.104:0/331212945 conn(0x7f7898071ec0 msgr2=0x7f789809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 -- 172.21.15.104:0/331212945 shutdown_connections 2023-06-10T18:53:25.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.173+0000 7f789d6cb700 1 -- 172.21.15.104:0/331212945 wait complete. 2023-06-10T18:53:25.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.174+0000 7f789d6cb700 1 Processor -- start 2023-06-10T18:53:25.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.174+0000 7f789d6cb700 1 -- start start 2023-06-10T18:53:25.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f789806b9f0 0x7f78980b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78980d0a30 0x7f78980cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981f6230 con 0x7f78980b1e30 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789815a720 con 0x7f789806b9f0 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f789d6cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78981f5620 con 0x7f78980d0a30 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78967fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78967fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34286/0 (socket says 172.21.15.104:34286) 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78967fc700 1 -- 172.21.15.104:0/523694414 learned_addr learned my addr 172.21.15.104:0/523694414 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:25.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f7896ffd700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f789806b9f0 0x7f78980b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78967fc700 1 -- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78980d0a30 msgr2=0x7f78980cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78977fe700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78980d0a30 0x7f78980cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.175+0000 7f78967fc700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78980d0a30 0x7f78980cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f78967fc700 1 -- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f789806b9f0 msgr2=0x7f78980b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f78967fc700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f789806b9f0 0x7f78980b7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f78967fc700 1 -- 172.21.15.104:0/523694414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c009550 con 0x7f78980b1e30 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f78967fc700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f788800b4a0 tx=0x7f788800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888011830 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7888011990 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78981f5900 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888011d40 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.176+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78981f5ec0 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.177+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78980d1860 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.177+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f788801a400 con 0x7f78980b1e30 2023-06-10T18:53:25.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.178+0000 7f787f7fe700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 0x7f788406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.178+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7888089010 con 0x7f78980b1e30 2023-06-10T18:53:25.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.178+0000 7f7896ffd700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 0x7f788406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.178+0000 7f7896ffd700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 0x7f788406d190 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f788c009c40 tx=0x7f788c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.182+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f78880542c0 con 0x7f78980b1e30 2023-06-10T18:53:25.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.443+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7898067090 con 0x7f78980b1e30 2023-06-10T18:53:25.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.443+0000 7f787f7fe700 1 -- 172.21.15.104:0/523694414 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7898067090 con 0x7f78980b1e30 2023-06-10T18:53:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:25.449 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:53:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:53:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:53:25.450 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.451 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:53:25.452 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:53:25.454 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.455 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:53:25.456 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:53:25.457 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.458 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.460 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.461 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.462 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:53:25.463 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.464 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.466 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:53:25.466 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:53:25.466 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.466 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.466 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:53:25.467 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:53:25.469 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:53:25.470 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:53:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:53:25.473 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:53:25.474 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:53:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 msgr2=0x7f788406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 0x7f788406d190 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f788c009c40 tx=0x7f788c008030 comp rx=0 tx=0).stop 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 msgr2=0x7f789815a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f788800b4a0 tx=0x7f788800b860 comp rx=0 tx=0).stop 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 shutdown_connections 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f788406ace0 0x7f788406d190 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f789806b9f0 0x7f78980b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78980d0a30 0x7f78980cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 --2- 172.21.15.104:0/523694414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78980b1e30 0x7f789815a1e0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 >> 172.21.15.104:0/523694414 conn(0x7f7898071ec0 msgr2=0x7f78980ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 shutdown_connections 2023-06-10T18:53:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.445+0000 7f789d6cb700 1 -- 172.21.15.104:0/523694414 wait complete. 2023-06-10T18:53:25.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:25.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.592+0000 7fcd0887d700 1 Processor -- start 2023-06-10T18:53:25.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.592+0000 7fcd0887d700 1 -- start start 2023-06-10T18:53:25.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd0887d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd0406a580 0x7fcd040ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd0887d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0406dcb0 con 0x7fcd040cf030 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd0887d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd040cf460 con 0x7fcd0415c500 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd0887d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0406e850 con 0x7fcd0406a580 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd02d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd0406a580 0x7fcd040ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd02d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd0406a580 0x7fcd040ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34296/0 (socket says 172.21.15.104:34296) 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd01d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcd040cf030 0x7fcd04069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60354/0 (socket says 172.21.15.104:60354) 2023-06-10T18:53:25.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.593+0000 7fcd02d9d700 1 -- 172.21.15.104:0/1674705870 learned_addr learned my addr 172.21.15.104:0/1674705870 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd02d9d700 1 -- 172.21.15.104:0/1674705870 >> v1:172.21.15.114:6789/0 conn(0x7fcd0415c500 legacy=0x7fcd0406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd02d9d700 1 -- 172.21.15.104:0/1674705870 >> v1:172.21.15.104:6789/0 conn(0x7fcd040cf030 legacy=0x7fcd04069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd02d9d700 1 -- 172.21.15.104:0/1674705870 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0406d0a0 con 0x7fcd0406a580 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd02d9d700 1 --2- 172.21.15.104:0/1674705870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0406a580 0x7fcd040ce040 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009e50 tx=0x7fccfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54dfcf5f0fd0ec1e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0159a700 1 -- 172.21.15.104:0/1674705870 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc00e070 con 0x7fcd0406a580 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0159a700 1 -- 172.21.15.104:0/1674705870 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc003a20 con 0x7fcd0406a580 2023-06-10T18:53:25.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0159a700 1 -- 172.21.15.104:0/1674705870 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc003dd0 con 0x7fcd0406a580 2023-06-10T18:53:25.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0887d700 1 -- 172.21.15.104:0/1674705870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0406a580 msgr2=0x7fcd040ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0887d700 1 --2- 172.21.15.104:0/1674705870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0406a580 0x7fcd040ce040 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009e50 tx=0x7fccfc00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.594+0000 7fcd0887d700 1 -- 172.21.15.104:0/1674705870 shutdown_connections 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.595+0000 7fcd0887d700 1 --2- 172.21.15.104:0/1674705870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0406a580 0x7fcd040ce040 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.595+0000 7fcd0887d700 1 -- 172.21.15.104:0/1674705870 >> 172.21.15.104:0/1674705870 conn(0x7fcd0406b8a0 msgr2=0x7fcd040cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.595+0000 7fcd0887d700 1 -- 172.21.15.104:0/1674705870 shutdown_connections 2023-06-10T18:53:25.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.595+0000 7fcd0887d700 1 -- 172.21.15.104:0/1674705870 wait complete. 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.596+0000 7fcd0887d700 1 Processor -- start 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.596+0000 7fcd0887d700 1 -- start start 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0406a580 0x7fcd040b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 0x7fcd040c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040b3d80 con 0x7fcd0415c500 2023-06-10T18:53:25.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040c68a0 con 0x7fcd040cf030 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040b5250 con 0x7fcd0406a580 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34302/0 (socket says 172.21.15.104:34302) 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 -- 172.21.15.104:0/416668044 learned_addr learned my addr 172.21.15.104:0/416668044 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd01d9b700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 0x7fcd040c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0259c700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0406a580 0x7fcd040b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 -- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0406a580 msgr2=0x7fcd040b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0406a580 0x7fcd040b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 -- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 msgr2=0x7fcd040c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 0x7fcd040c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 -- 172.21.15.104:0/416668044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc009b30 con 0x7fcd0415c500 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd01d9b700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 0x7fcd040c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd02d9d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fccfc004000 tx=0x7fccfc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc004240 con 0x7fcd0415c500 2023-06-10T18:53:25.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc0043a0 con 0x7fcd0415c500 2023-06-10T18:53:25.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd040b34e0 con 0x7fcd0415c500 2023-06-10T18:53:25.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc004750 con 0x7fcd0415c500 2023-06-10T18:53:25.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.597+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd040b3aa0 con 0x7fcd0415c500 2023-06-10T18:53:25.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.598+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd04074050 con 0x7fcd0415c500 2023-06-10T18:53:25.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.599+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fccfc019070 con 0x7fcd0415c500 2023-06-10T18:53:25.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.599+0000 7fccf2ffd700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 0x7fcce406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.599+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fccfc08a2b0 con 0x7fcd0415c500 2023-06-10T18:53:25.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.599+0000 7fcd0259c700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 0x7fcce406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.600+0000 7fcd0259c700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 0x7fcce406d190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcd0415c980 tx=0x7fccf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.603+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fccfc055580 con 0x7fcd0415c500 2023-06-10T18:53:25.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.788+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0415fb70 con 0x7fcd0415c500 2023-06-10T18:53:25.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.789+0000 7fccf2ffd700 1 -- 172.21.15.104:0/416668044 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+14893 (secure 0 0 0) 0x7fccfc02c300 con 0x7fcd0415c500 2023-06-10T18:53:25.794 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:25.795 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":95,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:25.032485+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:25.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.790+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 msgr2=0x7fcce406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.790+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 0x7fcce406d190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcd0415c980 tx=0x7fccf8009040 comp rx=0 tx=0).stop 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.790+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 msgr2=0x7fcd040c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fccfc004000 tx=0x7fccfc004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 shutdown_connections 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd0415c500 0x7fcd040c7270 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0406a580 0x7fcd040b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcce406ace0 0x7fcce406d190 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 --2- 172.21.15.104:0/416668044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd040cf030 0x7fcd040c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 >> 172.21.15.104:0/416668044 conn(0x7fcd0406b8a0 msgr2=0x7fcd040ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 shutdown_connections 2023-06-10T18:53:25.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.791+0000 7fcd0887d700 1 -- 172.21.15.104:0/416668044 wait complete. 2023-06-10T18:53:25.808 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:25.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:25.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.928+0000 7f414a5b7700 1 Processor -- start 2023-06-10T18:53:25.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.928+0000 7f414a5b7700 1 -- start start 2023-06-10T18:53:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f414a5b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c4470 0x7f41441607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f414a5b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414406b800 con 0x7f414406b280 2023-06-10T18:53:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f414a5b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4144069e70 con 0x7f4144145c70 2023-06-10T18:53:25.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f414a5b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4144160cf0 con 0x7f41440c4470 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f41437fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f414406b280 0x7f41440c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60368/0 (socket says 172.21.15.104:60368) 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f41437fe700 1 -- 172.21.15.104:0/1680136672 learned_addr learned my addr 172.21.15.104:0/1680136672 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275956459 0 0) 0x7f4134001110 con 0x7f414406b280 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f412c0035b0 con 0x7f414406b280 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c4470 0x7f41441607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.929+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3851543842 0 0) 0x7f412c0035b0 con 0x7f414406b280 2023-06-10T18:53:25.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c4470 msgr2=0x7f41441607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c4470 0x7f41441607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 >> v1:172.21.15.114:6789/0 conn(0x7f4144145c70 legacy=0x7f414406a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414406c9a0 con 0x7f414406b280 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4134001800 con 0x7f414406b280 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 >> v1:172.21.15.104:6789/0 conn(0x7f414406b280 legacy=0x7f41440c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f412c003a30 0x7f412c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412c004850 0x7f412c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f412c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144160cf0 con 0x7f412c005270 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c00c190 con 0x7f412c004850 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c00c350 con 0x7f412c003a30 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c4470 0x7f41441607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f412c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4143fff700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f412c003a30 0x7f412c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f41437fe700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412c004850 0x7f412c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 -- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f412c003a30 msgr2=0x7f412c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f412c003a30 0x7f412c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 -- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412c004850 msgr2=0x7f412c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f412c004850 0x7f412c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 -- 172.21.15.104:0/1680136672 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c0035b0 con 0x7f412c005270 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4148b54700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f412c00ba10 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f41440cc9e0 tx=0x7f412c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a5c3475eda6f3d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c007d90 con 0x7f412c005270 2023-06-10T18:53:25.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c00f040 con 0x7f412c005270 2023-06-10T18:53:25.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.930+0000 7f4142ffd700 1 -- 172.21.15.104:0/1680136672 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c003c60 con 0x7f412c005270 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 -- 172.21.15.104:0/1680136672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 msgr2=0x7f412c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 --2- 172.21.15.104:0/1680136672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f412c00ba10 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f41440cc9e0 tx=0x7f412c005880 comp rx=0 tx=0).stop 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f4148b54700 1 -- 172.21.15.104:0/1680136672 reap_dead start 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 -- 172.21.15.104:0/1680136672 shutdown_connections 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 -- 172.21.15.104:0/1680136672 >> 172.21.15.104:0/1680136672 conn(0x7f41440ceec0 msgr2=0x7f41440cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:25.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 -- 172.21.15.104:0/1680136672 shutdown_connections 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.931+0000 7f414a5b7700 1 -- 172.21.15.104:0/1680136672 wait complete. 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.932+0000 7f414a5b7700 1 Processor -- start 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.932+0000 7f414a5b7700 1 -- start start 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f414415b870 0x7f41440c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c7800 0x7f41440c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41441e7280 con 0x7f412c005270 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144159290 con 0x7f414415b870 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f414a5b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41441e6fa0 con 0x7f41440c7800 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f4143fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f4143fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34320/0 (socket says 172.21.15.104:34320) 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.933+0000 7f4143fff700 1 -- 172.21.15.104:0/2290489569 learned_addr learned my addr 172.21.15.104:0/2290489569 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 -- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c7800 msgr2=0x7f41440c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c7800 0x7f41440c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 -- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f414415b870 msgr2=0x7f41440c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f414415b870 0x7f41440c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 -- 172.21.15.104:0/2290489569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440acb50 con 0x7f412c005270 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4143fff700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f413800f770 tx=0x7f413800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41380013a0 con 0x7f412c005270 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41380052e0 con 0x7f412c005270 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41440ace30 con 0x7f412c005270 2023-06-10T18:53:25.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138003680 con 0x7f412c005270 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.934+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4144159000 con 0x7f412c005270 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.935+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4144074050 con 0x7f412c005270 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.935+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f41380037e0 con 0x7f412c005270 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.936+0000 7f4140ff9700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 0x7f413006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.936+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4138089040 con 0x7f412c005270 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.936+0000 7f41437fe700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 0x7f413006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:25.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.936+0000 7f41437fe700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 0x7f413006d190 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4134001270 tx=0x7f4134001200 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:25.940+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4138054300 con 0x7f412c005270 2023-06-10T18:53:26.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.126+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f414406b060 con 0x7f412c005270 2023-06-10T18:53:26.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.126+0000 7f4140ff9700 1 -- 172.21.15.104:0/2290489569 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4138057920 con 0x7f412c005270 2023-06-10T18:53:26.133 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:26.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 msgr2=0x7f413006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 0x7f413006d190 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4134001270 tx=0x7f4134001200 comp rx=0 tx=0).stop 2023-06-10T18:53:26.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 msgr2=0x7f4144158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f413800f770 tx=0x7f413800fb30 comp rx=0 tx=0).stop 2023-06-10T18:53:26.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 shutdown_connections 2023-06-10T18:53:26.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41440c7800 0x7f41440c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f412c005270 0x7f4144158ac0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f413006ace0 0x7f413006d190 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 --2- 172.21.15.104:0/2290489569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f414415b870 0x7f41440c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 >> 172.21.15.104:0/2290489569 conn(0x7f41440ceec0 msgr2=0x7f41440cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 shutdown_connections 2023-06-10T18:53:26.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.128+0000 7f414a5b7700 1 -- 172.21.15.104:0/2290489569 wait complete. 2023-06-10T18:53:26.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.262+0000 7fba2e309700 1 Processor -- start 2023-06-10T18:53:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.263+0000 7fba2e309700 1 -- start start 2023-06-10T18:53:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2e309700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba281586a0 0x7fba2806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2e309700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 0x7fba28070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2e309700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2806d7d0 con 0x7fba28070f50 2023-06-10T18:53:26.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2e309700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba280713d0 con 0x7fba281586a0 2023-06-10T18:53:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2e309700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba280cd6c0 con 0x7fba280ce390 2023-06-10T18:53:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba281586a0 0x7fba2806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2c8a6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba28070f50 0x7fba28069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60390/0 (socket says 172.21.15.104:60390) 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba2c8a6700 1 -- 172.21.15.104:0/2498867935 learned_addr learned my addr 172.21.15.104:0/2498867935 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 -- 172.21.15.104:0/2498867935 >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 msgr2=0x7fba28070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba277fe700 1 --2- 172.21.15.104:0/2498867935 >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 0x7fba28070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba26ffd700 1 -- 172.21.15.104:0/2498867935 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047837824 0 0) 0x7fba2806d7d0 con 0x7fba28070f50 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 --2- 172.21.15.104:0/2498867935 >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 0x7fba28070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 -- 172.21.15.104:0/2498867935 >> v1:172.21.15.104:6789/0 conn(0x7fba28070f50 legacy=0x7fba28069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 -- 172.21.15.104:0/2498867935 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280c95d0 con 0x7fba281586a0 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba277fe700 1 --2- 172.21.15.104:0/2498867935 >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 0x7fba28070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba27fff700 1 --2- 172.21.15.104:0/2498867935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba2806cff0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fba2806e670 tx=0x7fba10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1568486d8d354fc3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba26ffd700 1 -- 172.21.15.104:0/2498867935 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10011830 con 0x7fba281586a0 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba26ffd700 1 -- 172.21.15.104:0/2498867935 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10011990 con 0x7fba281586a0 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.264+0000 7fba26ffd700 1 -- 172.21.15.104:0/2498867935 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10011d20 con 0x7fba281586a0 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 -- 172.21.15.104:0/2498867935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 msgr2=0x7fba2806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 --2- 172.21.15.104:0/2498867935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba2806cff0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fba2806e670 tx=0x7fba10009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 -- 172.21.15.104:0/2498867935 shutdown_connections 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 --2- 172.21.15.104:0/2498867935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba2806cff0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 --2- 172.21.15.104:0/2498867935 >> v2:172.21.15.114:3300/0 conn(0x7fba280ce390 0x7fba28070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 -- 172.21.15.104:0/2498867935 >> 172.21.15.104:0/2498867935 conn(0x7fba280ceec0 msgr2=0x7fba280cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 -- 172.21.15.104:0/2498867935 shutdown_connections 2023-06-10T18:53:26.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.265+0000 7fba2e309700 1 -- 172.21.15.104:0/2498867935 wait complete. 2023-06-10T18:53:26.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.266+0000 7fba2e309700 1 Processor -- start 2023-06-10T18:53:26.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.266+0000 7fba2e309700 1 -- start start 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba28070f50 0x7fba28159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba280ce390 0x7fba2815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2815ad60 con 0x7fba281586a0 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28150db0 con 0x7fba28070f50 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2e309700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2815a150 con 0x7fba280ce390 2023-06-10T18:53:26.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34336/0 (socket says 172.21.15.104:34336) 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 -- 172.21.15.104:0/1785904564 learned_addr learned my addr 172.21.15.104:0/1785904564 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba277fe700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba280ce390 0x7fba2815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba27fff700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba28070f50 0x7fba28159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 -- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba280ce390 msgr2=0x7fba2815b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba280ce390 0x7fba2815b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 -- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba28070f50 msgr2=0x7fba28159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba28070f50 0x7fba28159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.267+0000 7fba2c8a6700 1 -- 172.21.15.104:0/1785904564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100094a0 con 0x7fba281586a0 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba2c8a6700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fba2000cba0 tx=0x7fba2000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000ab80 con 0x7fba281586a0 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2000ad00 con 0x7fba281586a0 2023-06-10T18:53:26.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2815a430 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20007310 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.268+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2815a9f0 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.269+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2815aed0 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.269+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba20007470 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.269+0000 7fba24ff9700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 0x7fba0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.270+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fba200890d0 con 0x7fba281586a0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.270+0000 7fba27fff700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 0x7fba0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.270+0000 7fba27fff700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 0x7fba0806d190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fba100090b0 tx=0x7fba10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.274+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba20054390 con 0x7fba281586a0 2023-06-10T18:53:26.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.458+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fba28074050 con 0x7fba281586a0 2023-06-10T18:53:26.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.458+0000 7fba24ff9700 1 -- 172.21.15.104:0/1785904564 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v95) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba200579b0 con 0x7fba281586a0 2023-06-10T18:53:26.464 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:26.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 msgr2=0x7fba0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 0x7fba0806d190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fba100090b0 tx=0x7fba10008030 comp rx=0 tx=0).stop 2023-06-10T18:53:26.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 msgr2=0x7fba280b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fba2000cba0 tx=0x7fba2000cf60 comp rx=0 tx=0).stop 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 shutdown_connections 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba0806ace0 0x7fba0806d190 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba28070f50 0x7fba28159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba281586a0 0x7fba280b4000 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 --2- 172.21.15.104:0/1785904564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba280ce390 0x7fba2815b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 >> 172.21.15.104:0/1785904564 conn(0x7fba280ceec0 msgr2=0x7fba280ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.460+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 shutdown_connections 2023-06-10T18:53:26.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.461+0000 7fba2e309700 1 -- 172.21.15.104:0/1785904564 wait complete. 2023-06-10T18:53:26.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:53:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.592+0000 7fc830a64700 1 Processor -- start 2023-06-10T18:53:26.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.592+0000 7fc830a64700 1 -- start start 2023-06-10T18:53:26.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.592+0000 7fc830a64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc82c0aeaa0 0x7fc82c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.592+0000 7fc830a64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0cd680 0x7fc82c06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc830a64700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c0cdae0 con 0x7fc82c0b1080 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc830a64700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82c0c9bc0 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc830a64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0c9e60 con 0x7fc82c0cd680 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc82c0aeaa0 0x7fc82c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc82c0aeaa0 0x7fc82c0cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34340/0 (socket says 172.21.15.104:34340) 2023-06-10T18:53:26.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 -- 172.21.15.104:0/1691802808 learned_addr learned my addr 172.21.15.104:0/1691802808 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82ad9d700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0cd680 0x7fc82c06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 -- 172.21.15.104:0/1691802808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0cd680 msgr2=0x7fc82c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0cd680 0x7fc82c06dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 -- 172.21.15.104:0/1691802808 >> v1:172.21.15.114:6789/0 conn(0x7fc82c0b1080 legacy=0x7fc82c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 -- 172.21.15.104:0/1691802808 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c0aee70 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82a59c700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0cd140 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc820006fd0 tx=0x7fc820009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1250a1700deb4ed0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82959a700 1 -- 172.21.15.104:0/1691802808 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc820005e30 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82959a700 1 -- 172.21.15.104:0/1691802808 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc820004030 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.593+0000 7fc82959a700 1 -- 172.21.15.104:0/1691802808 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8200043e0 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 -- 172.21.15.104:0/1691802808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 msgr2=0x7fc82c0cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0cd140 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc820006fd0 tx=0x7fc820009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 -- 172.21.15.104:0/1691802808 shutdown_connections 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0cd140 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 --2- 172.21.15.104:0/1691802808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0cd680 0x7fc82c06dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 -- 172.21.15.104:0/1691802808 >> 172.21.15.104:0/1691802808 conn(0x7fc82c06b8a0 msgr2=0x7fc82c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 -- 172.21.15.104:0/1691802808 shutdown_connections 2023-06-10T18:53:26.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.594+0000 7fc830a64700 1 -- 172.21.15.104:0/1691802808 wait complete. 2023-06-10T18:53:26.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.595+0000 7fc830a64700 1 Processor -- start 2023-06-10T18:53:26.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.595+0000 7fc830a64700 1 -- start start 2023-06-10T18:53:26.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 0x7fc82c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc82c0cd680 0x7fc82c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c152270 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0c35d0 con 0x7fc82c0cd680 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc830a64700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0c38b0 con 0x7fc82c0b1080 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34348/0 (socket says 172.21.15.104:34348) 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 -- 172.21.15.104:0/2161297881 learned_addr learned my addr 172.21.15.104:0/2161297881 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc829d9b700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 0x7fc82c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82ad9d700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc82c0cd680 0x7fc82c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 -- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 msgr2=0x7fc82c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 0x7fc82c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 -- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc82c0cd680 msgr2=0x7fc82c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.596+0000 7fc82a59c700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc82c0cd680 0x7fc82c069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc82a59c700 1 -- 172.21.15.104:0/2161297881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc820009500 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc829d9b700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 0x7fc82c15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc82a59c700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc820009c10 tx=0x7fc820004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc820003dd0 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82001c420 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c0c3b90 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82001c7b0 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.597+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c0b5170 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.598+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c074050 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.598+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc82001c910 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.598+0000 7fc81affd700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 0x7fc80c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.599+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc82008a320 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.599+0000 7fc829d9b700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 0x7fc80c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.599+0000 7fc829d9b700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 0x7fc80c06d190 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc82c158f50 tx=0x7fc81c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.603+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc820055670 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.793+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc82c06e270 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.793+0000 7fc81affd700 1 -- 172.21.15.104:0/2161297881 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+126 (secure 0 0 0) 0x7fc820058c90 con 0x7fc82c0aeaa0 2023-06-10T18:53:26.799 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:26.799 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:53:26.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:26.800 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:53:26.800 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:53:26.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:26.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:53:26.801 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:53:26.801 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:53:26.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:26.801 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:53:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 msgr2=0x7fc80c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 0x7fc80c06d190 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc82c158f50 tx=0x7fc81c009040 comp rx=0 tx=0).stop 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 msgr2=0x7fc82c0c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc820009c10 tx=0x7fc820004750 comp rx=0 tx=0).stop 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 shutdown_connections 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc80c06ace0 0x7fc80c06d190 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc82c0cd680 0x7fc82c069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc82c0aeaa0 0x7fc82c0c4240 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 --2- 172.21.15.104:0/2161297881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc82c0b1080 0x7fc82c15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 >> 172.21.15.104:0/2161297881 conn(0x7fc82c06b8a0 msgr2=0x7fc82c06d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 shutdown_connections 2023-06-10T18:53:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.795+0000 7fc830a64700 1 -- 172.21.15.104:0/2161297881 wait complete. 2023-06-10T18:53:26.813 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:53:26.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:53:26.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.932+0000 7fa1a5a87700 1 Processor -- start 2023-06-10T18:53:26.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.932+0000 7fa1a5a87700 1 -- start start 2023-06-10T18:53:26.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa1a5a87700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa1a015c6f0 0x7fa1a00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa1a5a87700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1a006ea70 0x7fa1a00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa1a5a87700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00ae7e0 con 0x7fa1a00aec00 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa1a5a87700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a006df90 con 0x7fa1a015c6f0 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa1a5a87700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a006e270 con 0x7fa1a006ea70 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa1a015c6f0 0x7fa1a00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa1a00aec00 0x7fa1a00ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60416/0 (socket says 172.21.15.104:60416) 2023-06-10T18:53:26.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19ffff700 1 -- 172.21.15.104:0/1796944364 learned_addr learned my addr 172.21.15.104:0/1796944364 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/1796944364 >> v2:172.21.15.104:3300/0 conn(0x7fa1a015c6f0 0x7fa1a00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34354/0 (socket says 172.21.15.104:34354) 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 -- 172.21.15.104:0/1796944364 >> v2:172.21.15.114:3300/0 conn(0x7fa1a006ea70 msgr2=0x7fa1a00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19effd700 1 --2- 172.21.15.104:0/1796944364 >> v2:172.21.15.114:3300/0 conn(0x7fa1a006ea70 0x7fa1a00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19e7fc700 1 -- 172.21.15.104:0/1796944364 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056946176 0 0) 0x7fa1a00ae7e0 con 0x7fa1a00aec00 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/1796944364 >> v2:172.21.15.114:3300/0 conn(0x7fa1a006ea70 0x7fa1a00b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 -- 172.21.15.104:0/1796944364 >> v1:172.21.15.104:6789/0 conn(0x7fa1a00aec00 legacy=0x7fa1a00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 -- 172.21.15.104:0/1796944364 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a006e550 con 0x7fa1a015c6f0 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/1796944364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a00b1070 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0070990 tx=0x7fa190009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15b7daed5c648d3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19e7fc700 1 -- 172.21.15.104:0/1796944364 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190005e30 con 0x7fa1a015c6f0 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19e7fc700 1 -- 172.21.15.104:0/1796944364 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa190006030 con 0x7fa1a015c6f0 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.933+0000 7fa19e7fc700 1 -- 172.21.15.104:0/1796944364 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1900063c0 con 0x7fa1a015c6f0 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 -- 172.21.15.104:0/1796944364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 msgr2=0x7fa1a00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/1796944364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a00b1070 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0070990 tx=0x7fa190009b00 comp rx=0 tx=0).stop 2023-06-10T18:53:26.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 -- 172.21.15.104:0/1796944364 shutdown_connections 2023-06-10T18:53:26.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/1796944364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a00b1070 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/1796944364 >> v2:172.21.15.114:3300/0 conn(0x7fa1a006ea70 0x7fa1a00b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 -- 172.21.15.104:0/1796944364 >> 172.21.15.104:0/1796944364 conn(0x7fa1a006b8a0 msgr2=0x7fa1a00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:26.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 -- 172.21.15.104:0/1796944364 shutdown_connections 2023-06-10T18:53:26.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.934+0000 7fa1a5a87700 1 -- 172.21.15.104:0/1796944364 wait complete. 2023-06-10T18:53:26.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.935+0000 7fa1a5a87700 1 Processor -- start 2023-06-10T18:53:26.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.935+0000 7fa1a5a87700 1 -- start start 2023-06-10T18:53:26.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1a006ea70 0x7fa1a00b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1a00aec00 0x7fa1a00af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00b0b50 con 0x7fa1a015c6f0 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00b52e0 con 0x7fa1a006ea70 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa1a5a87700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00afca0 con 0x7fa1a00aec00 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa19ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa19ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34360/0 (socket says 172.21.15.104:34360) 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa19ffff700 1 -- 172.21.15.104:0/3436272052 learned_addr learned my addr 172.21.15.104:0/3436272052 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1a006ea70 0x7fa1a00b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.936+0000 7fa19effd700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1a00aec00 0x7fa1a00af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 -- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1a00aec00 msgr2=0x7fa1a00af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1a00aec00 0x7fa1a00af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 -- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1a006ea70 msgr2=0x7fa1a00b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1a006ea70 0x7fa1a00b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:26.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 -- 172.21.15.104:0/3436272052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190009430 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa19ffff700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fa19800cbe0 tx=0x7fa19800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19800aaa0 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19800ac20 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980070d0 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a0148c00 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.937+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a00b01b0 con 0x7fa1a015c6f0 2023-06-10T18:53:26.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.938+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a00cdb20 con 0x7fa1a015c6f0 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.938+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa198007230 con 0x7fa1a015c6f0 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.939+0000 7fa187fff700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 0x7fa18c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.939+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fa198088d10 con 0x7fa1a015c6f0 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.939+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 0x7fa18c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.939+0000 7fa19f7fe700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 0x7fa18c06d190 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa190008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:26.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:26.943+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa1a00cdb20 con 0x7fa1a015c6f0 2023-06-10T18:53:27.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:27.132+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa1a00b0c90 con 0x7fa1a015c6f0 2023-06-10T18:53:28.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.047+0000 7fa187fff700 1 -- 172.21.15.104:0/3436272052 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa1a00b0c90 con 0x7fa1a015c6f0 2023-06-10T18:53:28.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 msgr2=0x7fa18c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 0x7fa18c06d190 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa190008070 comp rx=0 tx=0).stop 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 msgr2=0x7fa1a01486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fa19800cbe0 tx=0x7fa19800cef0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 shutdown_connections 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa18c06ace0 0x7fa18c06d190 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1a006ea70 0x7fa1a00b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1a015c6f0 0x7fa1a01486c0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 --2- 172.21.15.104:0/3436272052 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1a00aec00 0x7fa1a00af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 >> 172.21.15.104:0/3436272052 conn(0x7fa1a006b8a0 msgr2=0x7fa1a014f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:28.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 shutdown_connections 2023-06-10T18:53:28.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.052+0000 7fa1a5a87700 1 -- 172.21.15.104:0/3436272052 wait complete. 2023-06-10T18:53:28.059 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:53:28.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:53:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.194+0000 7fa6cee79700 1 Processor -- start 2023-06-10T18:53:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.194+0000 7fa6cee79700 1 -- start start 2023-06-10T18:53:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cee79700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa6c8070a10 0x7fa6c806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cee79700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c806cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cee79700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80cec40 con 0x7fa6c8143a40 2023-06-10T18:53:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cee79700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80cca90 con 0x7fa6c8070a10 2023-06-10T18:53:28.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cee79700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80ce960 con 0x7fa6c806c800 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6ccc15700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa6c8070a10 0x7fa6c806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6c7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa6c8143a40 0x7fa6c8070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60430/0 (socket says 172.21.15.104:60430) 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6c7fff700 1 -- 172.21.15.104:0/1990204662 learned_addr learned my addr 172.21.15.104:0/1990204662 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6cd416700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c806cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6ccc15700 1 -- 172.21.15.104:0/1990204662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 msgr2=0x7fa6c806cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c806cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6ccc15700 1 -- 172.21.15.104:0/1990204662 >> v1:172.21.15.104:6789/0 conn(0x7fa6c8143a40 legacy=0x7fa6c8070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6c77fe700 1 -- 172.21.15.104:0/1990204662 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407168533 0 0) 0x7fa6c80cec40 con 0x7fa6c8143a40 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.195+0000 7fa6ccc15700 1 -- 172.21.15.104:0/1990204662 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c806a9b0 con 0x7fa6c8070a10 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8070a10 0x7fa6c806e560 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009840 tx=0x7fa6bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813cba47eee2bf46 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6c77fe700 1 -- 172.21.15.104:0/1990204662 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc011830 con 0x7fa6c8070a10 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6c77fe700 1 -- 172.21.15.104:0/1990204662 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc011990 con 0x7fa6c8070a10 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6c77fe700 1 -- 172.21.15.104:0/1990204662 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc011d20 con 0x7fa6c8070a10 2023-06-10T18:53:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 -- 172.21.15.104:0/1990204662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8070a10 msgr2=0x7fa6c806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8070a10 0x7fa6c806e560 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009840 tx=0x7fa6bc009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 -- 172.21.15.104:0/1990204662 shutdown_connections 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8070a10 0x7fa6c806e560 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 --2- 172.21.15.104:0/1990204662 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c806cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 -- 172.21.15.104:0/1990204662 >> 172.21.15.104:0/1990204662 conn(0x7fa6c80ceec0 msgr2=0x7fa6c80cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 -- 172.21.15.104:0/1990204662 shutdown_connections 2023-06-10T18:53:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.196+0000 7fa6cee79700 1 -- 172.21.15.104:0/1990204662 wait complete. 2023-06-10T18:53:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 Processor -- start 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 -- start start 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c80b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 0x7fa6c81e4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c81efd20 con 0x7fa6c8143a40 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80b3fa0 con 0x7fa6c8070a10 2023-06-10T18:53:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.198+0000 7fa6cee79700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80b4a50 con 0x7fa6c806c800 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34376/0 (socket says 172.21.15.104:34376) 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 -- 172.21.15.104:0/710678080 learned_addr learned my addr 172.21.15.104:0/710678080 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c80b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 -- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 msgr2=0x7fa6c80b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6c7fff700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 0x7fa6c81e4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c80b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 -- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 msgr2=0x7fa6c81e4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 0x7fa6c81e4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 -- 172.21.15.104:0/710678080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0094a0 con 0x7fa6c8143a40 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c80b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6c7fff700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 0x7fa6c81e4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cd416700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000efc0 tx=0x7fa6c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0009ba0 con 0x7fa6c8143a40 2023-06-10T18:53:28.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c000f040 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c80cd0a0 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0014480 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.199+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80b4490 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.200+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c8074050 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.201+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa6c00145e0 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.201+0000 7fa6c57fa700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.201+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fa6c0089ca0 con 0x7fa6c8143a40 2023-06-10T18:53:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.201+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.202+0000 7fa6ccc15700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 0x7fa6ac06d190 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc000c00 tx=0x7fa6bc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.206+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa6c0054df0 con 0x7fa6c8143a40 2023-06-10T18:53:28.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:28.391+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa6c806b030 con 0x7fa6c8143a40 2023-06-10T18:53:29.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.047+0000 7fa6c57fa700 1 -- 172.21.15.104:0/710678080 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa6c00180d0 con 0x7fa6c8143a40 2023-06-10T18:53:29.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 msgr2=0x7fa6ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 0x7fa6ac06d190 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc000c00 tx=0x7fa6bc008030 comp rx=0 tx=0).stop 2023-06-10T18:53:29.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 msgr2=0x7fa6c8159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000efc0 tx=0x7fa6c000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 shutdown_connections 2023-06-10T18:53:29.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6c8143a40 0x7fa6c8159940 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6c806c800 0x7fa6c80b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 --2- 172.21.15.104:0/710678080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6c8070a10 0x7fa6c81e4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.051+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 >> 172.21.15.104:0/710678080 conn(0x7fa6c80ceec0 msgr2=0x7fa6c813f120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.053+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 shutdown_connections 2023-06-10T18:53:29.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.053+0000 7fa6cee79700 1 -- 172.21.15.104:0/710678080 wait complete. 2023-06-10T18:53:29.061 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:53:29.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:53:29.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 Processor -- start 2023-06-10T18:53:29.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 -- start start 2023-06-10T18:53:29.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c58072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c580cbee0 0x7f4c580cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580cf460 con 0x7f4c58069e70 2023-06-10T18:53:29.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c580cec00 con 0x7f4c580cbee0 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.203+0000 7f4c5d87c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5806ee30 con 0x7f4c580cd250 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c56ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4c58069e70 0x7f4c5806a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60440/0 (socket says 172.21.15.104:60440) 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c56ffd700 1 -- 172.21.15.104:0/2181788764 learned_addr learned my addr 172.21.15.104:0/2181788764 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c58072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c577fe700 1 --2- 172.21.15.104:0/2181788764 >> v2:172.21.15.114:3300/0 conn(0x7f4c580cbee0 0x7f4c580cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633891745 0 0) 0x7f4c580cf460 con 0x7f4c58069e70 2023-06-10T18:53:29.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c400035b0 con 0x7f4c58069e70 2023-06-10T18:53:29.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222353968 0 0) 0x7f4c400035b0 con 0x7f4c58069e70 2023-06-10T18:53:29.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 msgr2=0x7f4c58072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c58072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 >> v2:172.21.15.114:3300/0 conn(0x7f4c580cbee0 msgr2=0x7f4c580cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 --2- 172.21.15.104:0/2181788764 >> v2:172.21.15.114:3300/0 conn(0x7f4c580cbee0 0x7f4c580cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5815c490 con 0x7f4c58069e70 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4c48001670 con 0x7f4c58069e70 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 >> v1:172.21.15.104:6789/0 conn(0x7f4c58069e70 legacy=0x7f4c5806a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c40003a00 0x7f4c40003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c400048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c40004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c580cec00 con 0x7f4c40004430 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5806ee30 con 0x7f4c40003a00 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4000ba10 con 0x7f4c580cd250 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c400048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c56ffd700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c40003a00 0x7f4c40003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c577fe700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c40004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 -- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 msgr2=0x7f4c40004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c40004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 -- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c40003a00 msgr2=0x7f4c40003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.204+0000 7f4c567fc700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c40003a00 0x7f4c40003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c567fc700 1 -- 172.21.15.104:0/2181788764 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400035b0 con 0x7f4c40004430 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c577fe700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c580cd250 0x7f4c40004ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c567fc700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c400048f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009b30 tx=0x7f4c40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b3a7767ca3cd9c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00e6c0 con 0x7f4c40004430 2023-06-10T18:53:29.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00e820 con 0x7f4c40004430 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c55ffb700 1 -- 172.21.15.104:0/2181788764 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00eb90 con 0x7f4c40004430 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 -- 172.21.15.104:0/2181788764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 msgr2=0x7f4c400048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/2181788764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c400048f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009b30 tx=0x7f4c40005880 comp rx=0 tx=0).stop 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c577fe700 1 -- 172.21.15.104:0/2181788764 reap_dead start 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 -- 172.21.15.104:0/2181788764 shutdown_connections 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 -- 172.21.15.104:0/2181788764 >> 172.21.15.104:0/2181788764 conn(0x7f4c5806d240 msgr2=0x7f4c5806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 -- 172.21.15.104:0/2181788764 shutdown_connections 2023-06-10T18:53:29.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.205+0000 7f4c5d87c700 1 -- 172.21.15.104:0/2181788764 wait complete. 2023-06-10T18:53:29.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 Processor -- start 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 -- start start 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5815d6f0 0x7f4c580b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580b2720 0x7f4c580c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5815e800 con 0x7f4c40004430 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5815dbf0 con 0x7f4c580b2720 2023-06-10T18:53:29.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.207+0000 7f4c5d87c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5815ded0 con 0x7f4c5815d6f0 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34392/0 (socket says 172.21.15.104:34392) 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 -- 172.21.15.104:0/918414535 learned_addr learned my addr 172.21.15.104:0/918414535 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c567fc700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5815d6f0 0x7f4c580b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c577fe700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580b2720 0x7f4c580c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 -- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5815d6f0 msgr2=0x7f4c580b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5815d6f0 0x7f4c580b5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 -- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580b2720 msgr2=0x7f4c580c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580b2720 0x7f4c580c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 -- 172.21.15.104:0/918414535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5815e1b0 con 0x7f4c40004430 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c56ffd700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4c480072c0 tx=0x7f4c480075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011580 con 0x7f4c40004430 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c480116e0 con 0x7f4c40004430 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5815e490 con 0x7f4c40004430 2023-06-10T18:53:29.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011a90 con 0x7f4c40004430 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.208+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c580b2bc0 con 0x7f4c40004430 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.209+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5815eae0 con 0x7f4c40004430 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.210+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4c48011bf0 con 0x7f4c40004430 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.210+0000 7f4c3ffff700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.210+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f4c480893f0 con 0x7f4c40004430 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.210+0000 7f4c567fc700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.211+0000 7f4c567fc700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0090b0 tx=0x7f4c4c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.215+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4c48054470 con 0x7f4c40004430 2023-06-10T18:53:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.445+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4c58074050 con 0x7f4c40004430 2023-06-10T18:53:29.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.452+0000 7f4c3ffff700 1 -- 172.21.15.104:0/918414535 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v51) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4c48057a90 con 0x7f4c40004430 2023-06-10T18:53:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.463+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 msgr2=0x7f4c4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.463+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0090b0 tx=0x7f4c4c008030 comp rx=0 tx=0).stop 2023-06-10T18:53:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.463+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 msgr2=0x7f4c5815ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.463+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4c480072c0 tx=0x7f4c480075d0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 shutdown_connections 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4c4406ace0 0x7f4c4406d190 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580b2720 0x7f4c580c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c40004430 0x7f4c5815ec90 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 --2- 172.21.15.104:0/918414535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5815d6f0 0x7f4c580b5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.464+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 >> 172.21.15.104:0/918414535 conn(0x7f4c5806d240 msgr2=0x7f4c580cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.465+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 shutdown_connections 2023-06-10T18:53:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.465+0000 7f4c5d87c700 1 -- 172.21.15.104:0/918414535 wait complete. 2023-06-10T18:53:29.479 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 14 and data pool 15 2023-06-10T18:53:29.490 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:29.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.605+0000 7fb8b86b7700 1 Processor -- start 2023-06-10T18:53:29.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.605+0000 7fb8b86b7700 1 -- start start 2023-06-10T18:53:29.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b86b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb8b00cd850 0x7fb8b006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b86b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00c9e60 con 0x7fb8b00cb300 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b86b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00aee70 con 0x7fb8b00aeaa0 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b86b7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b00c9bc0 con 0x7fb8b00cd850 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b6c54700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb8b00cb300 0x7fb8b006d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60462/0 (socket says 172.21.15.104:60462) 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b6c54700 1 -- 172.21.15.104:0/1819914537 learned_addr learned my addr 172.21.15.104:0/1819914537 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:29.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/1819914537 >> v2:172.21.15.114:3300/0 conn(0x7fb8b00cd850 0x7fb8b006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.606+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511990767 0 0) 0x7fb8b00c9e60 con 0x7fb8b00cb300 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb89c0035b0 con 0x7fb8b00cb300 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506086909 0 0) 0x7fb89c0035b0 con 0x7fb8b00cb300 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 >> v2:172.21.15.114:3300/0 conn(0x7fb8b00cd850 msgr2=0x7fb8b006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 --2- 172.21.15.104:0/1819914537 >> v2:172.21.15.114:3300/0 conn(0x7fb8b00cd850 0x7fb8b006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 >> v1:172.21.15.114:6789/0 conn(0x7fb8b00aeaa0 legacy=0x7fb8b00cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b006e540 con 0x7fb8b00cb300 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb8a80019f0 con 0x7fb8b00cb300 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 >> v1:172.21.15.104:6789/0 conn(0x7fb8b00cb300 legacy=0x7fb8b006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb89c003a00 0x7fb89c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 0x7fb89c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c005200 0x7fb89c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b00c9bc0 con 0x7fb89c004850 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c00c060 con 0x7fb89c003a00 2023-06-10T18:53:29.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c00e430 con 0x7fb89c005200 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 0x7fb89c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b6453700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb89c003a00 0x7fb89c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b6c54700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c005200 0x7fb89c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 -- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c005200 msgr2=0x7fb89c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c005200 0x7fb89c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 -- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb89c003a00 msgr2=0x7fb89c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb89c003a00 0x7fb89c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 -- 172.21.15.104:0/1819914537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0035b0 con 0x7fb89c004850 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b6c54700 1 -- 172.21.15.104:0/1819914537 reap_dead start 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b6453700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb89c003a00 0x7fb89c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 0x7fb89c004cc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009b60 tx=0x7fb89c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f43d8b401b552 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00e6c0 con 0x7fb89c004850 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac00e820 con 0x7fb89c004850 2023-06-10T18:53:29.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.607+0000 7fb8b5451700 1 -- 172.21.15.104:0/1819914537 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00ebb0 con 0x7fb89c004850 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 -- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 msgr2=0x7fb89c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 0x7fb89c004cc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009b60 tx=0x7fb89c005880 comp rx=0 tx=0).stop 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 -- 172.21.15.104:0/1819914537 shutdown_connections 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/1819914537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb89c004850 0x7fb89c004cc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 -- 172.21.15.104:0/1819914537 >> 172.21.15.104:0/1819914537 conn(0x7fb8b006b8a0 msgr2=0x7fb8b00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 -- 172.21.15.104:0/1819914537 shutdown_connections 2023-06-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.608+0000 7fb8b86b7700 1 -- 172.21.15.104:0/1819914537 wait complete. 2023-06-10T18:53:29.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.610+0000 7fb8b86b7700 1 Processor -- start 2023-06-10T18:53:29.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.610+0000 7fb8b86b7700 1 -- start start 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.610+0000 7fb8b86b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c004850 0x7fb8b00b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 0x7fb8b0159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b01e2a60 con 0x7fb8b015a4d0 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b01f19a0 con 0x7fb8b01f1430 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b01f1c80 con 0x7fb89c004850 2023-06-10T18:53:29.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34414/0 (socket says 172.21.15.104:34414) 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 -- 172.21.15.104:0/3309823132 learned_addr learned my addr 172.21.15.104:0/3309823132 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6453700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c004850 0x7fb8b00b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 0x7fb8b0159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 -- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c004850 msgr2=0x7fb8b00b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c004850 0x7fb8b00b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 -- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 msgr2=0x7fb8b0159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 0x7fb8b0159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 -- 172.21.15.104:0/3309823132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b01f1f60 con 0x7fb8b015a4d0 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b5c52700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 0x7fb8b0159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b6c54700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00cbf40 tx=0x7fb8a8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a800ed90 con 0x7fb8b015a4d0 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a8013420 con 0x7fb8b015a4d0 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b01f2240 con 0x7fb8b015a4d0 2023-06-10T18:53:29.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80137d0 con 0x7fb8b015a4d0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b015a940 con 0x7fb8b015a4d0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.612+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b015ab60 con 0x7fb8b015a4d0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.613+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb8a8013930 con 0x7fb8b015a4d0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.613+0000 7fb8a6ffd700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 0x7fb89406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.613+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb8a8089cf0 con 0x7fb8b015a4d0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.613+0000 7fb8b6453700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 0x7fb89406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.614+0000 7fb8b6453700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 0x7fb89406d190 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.618+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb8a808e070 con 0x7fb8b015a4d0 2023-06-10T18:53:29.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.805+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b00cdc70 con 0x7fb8b015a4d0 2023-06-10T18:53:29.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.806+0000 7fb8a6ffd700 1 -- 172.21.15.104:0/3309823132 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18556 (secure 0 0 0) 0x7fb8a8054d20 con 0x7fb8b015a4d0 2023-06-10T18:53:29.812 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:29.813 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":98,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:29.447072+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:29.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 msgr2=0x7fb89406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 0x7fb89406d190 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2023-06-10T18:53:29.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 msgr2=0x7fb8b00c4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00cbf40 tx=0x7fb8a8007c90 comp rx=0 tx=0).stop 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 shutdown_connections 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b015a4d0 0x7fb8b00c4590 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb89c004850 0x7fb8b00b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb89406ace0 0x7fb89406d190 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.808+0000 7fb8b86b7700 1 --2- 172.21.15.104:0/3309823132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b01f1430 0x7fb8b0159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.809+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 >> 172.21.15.104:0/3309823132 conn(0x7fb8b006b8a0 msgr2=0x7fb8b00cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.809+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 shutdown_connections 2023-06-10T18:53:29.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.809+0000 7fb8b86b7700 1 -- 172.21.15.104:0/3309823132 wait complete. 2023-06-10T18:53:29.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:29.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.940+0000 7f9617a91700 1 Processor -- start 2023-06-10T18:53:29.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.940+0000 7f9617a91700 1 -- start start 2023-06-10T18:53:29.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f9617a91700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f96100cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f9617a91700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100cadd0 con 0x7f961006b1f0 2023-06-10T18:53:29.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f9617a91700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96100cb0b0 con 0x7f961008a3d0 2023-06-10T18:53:29.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f9617a91700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96100cfeb0 con 0x7f96100ce350 2023-06-10T18:53:29.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f961582d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f961006b1f0 0x7f96100cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60478/0 (socket says 172.21.15.104:60478) 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.941+0000 7f961582d700 1 -- 172.21.15.104:0/2094154826 learned_addr learned my addr 172.21.15.104:0/2094154826 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080340640 0 0) 0x7f96100cadd0 con 0x7f961006b1f0 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc0035b0 con 0x7f961006b1f0 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961502c700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f96100cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372814944 0 0) 0x7f95fc0035b0 con 0x7f961006b1f0 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 msgr2=0x7f96100cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f96100cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 >> v1:172.21.15.114:6789/0 conn(0x7f961008a3d0 legacy=0x7f96100cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100cf550 con 0x7f961006b1f0 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9600001670 con 0x7f961006b1f0 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 >> v1:172.21.15.104:6789/0 conn(0x7f961006b1f0 legacy=0x7f96100cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95fc003a30 0x7f95fc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc004850 0x7f95fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f95fc0052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96100cfeb0 con 0x7f95fc003a30 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc00bc00 con 0x7f95fc004850 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc00bdc0 con 0x7f96100ce350 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95fc003a30 0x7f95fc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961502c700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc004850 0x7f95fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961602e700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f95fc0052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 -- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 msgr2=0x7f95fc0052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96100ce350 0x7f95fc0052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 -- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc004850 msgr2=0x7f95fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc004850 0x7f95fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.942+0000 7f961582d700 1 -- 172.21.15.104:0/2094154826 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc0035b0 con 0x7f95fc003a30 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f961582d700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95fc003a30 0x7f95fc003e40 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f960000c210 tx=0x7f95fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32e0b753b2ee364 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9600007c10 con 0x7f95fc003a30 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960000cdd0 con 0x7f95fc003a30 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f961482b700 1 -- 172.21.15.104:0/2094154826 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9600005ba0 con 0x7f95fc003a30 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 -- 172.21.15.104:0/2094154826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95fc003a30 msgr2=0x7f95fc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 --2- 172.21.15.104:0/2094154826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95fc003a30 0x7f95fc003e40 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f960000c210 tx=0x7f95fc005880 comp rx=0 tx=0).stop 2023-06-10T18:53:29.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f961602e700 1 -- 172.21.15.104:0/2094154826 reap_dead start 2023-06-10T18:53:29.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 -- 172.21.15.104:0/2094154826 shutdown_connections 2023-06-10T18:53:29.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 -- 172.21.15.104:0/2094154826 >> 172.21.15.104:0/2094154826 conn(0x7f9610089fd0 msgr2=0x7f96100d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:29.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 -- 172.21.15.104:0/2094154826 shutdown_connections 2023-06-10T18:53:29.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.943+0000 7f9617a91700 1 -- 172.21.15.104:0/2094154826 wait complete. 2023-06-10T18:53:29.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.944+0000 7f9617a91700 1 Processor -- start 2023-06-10T18:53:29.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 -- start start 2023-06-10T18:53:29.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95fc003a30 0x7f96101e5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96100c9ad0 0x7f96101f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96100b4820 con 0x7f96101f2640 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96100b5680 con 0x7f96100c9ad0 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f9617a91700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96100b4a70 con 0x7f95fc003a30 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f961602e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f961602e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34434/0 (socket says 172.21.15.104:34434) 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.945+0000 7f961602e700 1 -- 172.21.15.104:0/3777848467 learned_addr learned my addr 172.21.15.104:0/3777848467 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961582d700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95fc003a30 0x7f96101e5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 -- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95fc003a30 msgr2=0x7f96101e5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95fc003a30 0x7f96101e5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961502c700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96100c9ad0 0x7f96101f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 -- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96100c9ad0 msgr2=0x7f96101f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96100c9ad0 0x7f96101f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:29.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 -- 172.21.15.104:0/3777848467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100b4d50 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f961602e700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f9610067a70 tx=0x7f960800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608002100 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9608003c10 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960800d990 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96100b5030 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.946+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961015e930 con 0x7f96101f2640 2023-06-10T18:53:29.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.947+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96100b57f0 con 0x7f96101f2640 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.947+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f960800daf0 con 0x7f96101f2640 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.948+0000 7f96067fc700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 0x7f95f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.948+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9608089810 con 0x7f96101f2640 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.948+0000 7f961582d700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 0x7f95f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.948+0000 7f961582d700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 0x7f95f406d1c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9600015040 tx=0x7f9600007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:29.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:29.953+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9608054780 con 0x7f96101f2640 2023-06-10T18:53:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.140+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9610074050 con 0x7f96101f2640 2023-06-10T18:53:30.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.142+0000 7f96067fc700 1 -- 172.21.15.104:0/3777848467 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+18556 (secure 0 0 0) 0x7f9608057da0 con 0x7f96101f2640 2023-06-10T18:53:30.149 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:30.149 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":98,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:29.447072+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:30.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 msgr2=0x7f95f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 0x7f95f406d1c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9600015040 tx=0x7f9600007000 comp rx=0 tx=0).stop 2023-06-10T18:53:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 msgr2=0x7f961015db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f9610067a70 tx=0x7f960800d430 comp rx=0 tx=0).stop 2023-06-10T18:53:30.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 shutdown_connections 2023-06-10T18:53:30.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96101f2640 0x7f961015db60 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95fc003a30 0x7f96101e5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95f406ad10 0x7f95f406d1c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 --2- 172.21.15.104:0/3777848467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96100c9ad0 0x7f96101f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 >> 172.21.15.104:0/3777848467 conn(0x7f9610089fd0 msgr2=0x7f961006e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 shutdown_connections 2023-06-10T18:53:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.144+0000 7f9617a91700 1 -- 172.21.15.104:0/3777848467 wait complete. 2023-06-10T18:53:30.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:53:30.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 Processor -- start 2023-06-10T18:53:30.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 -- start start 2023-06-10T18:53:30.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 0x7f5b880b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5b8806e6e0 0x7f5b880b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8806e250 con 0x7f5b8806cde0 2023-06-10T18:53:30.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8815c950 con 0x7f5b8806e6e0 2023-06-10T18:53:30.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8cf0d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b880cf0e0 con 0x7f5b8815c580 2023-06-10T18:53:30.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5b8806e6e0 0x7f5b880b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b86d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5b8806cde0 0x7f5b88069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60502/0 (socket says 172.21.15.104:60502) 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b86d9d700 1 -- 172.21.15.104:0/3481238828 learned_addr learned my addr 172.21.15.104:0/3481238828 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8659c700 1 --2- 172.21.15.104:0/3481238828 >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 0x7f5b880b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 -- 172.21.15.104:0/3481238828 >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 msgr2=0x7f5b880b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 --2- 172.21.15.104:0/3481238828 >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 0x7f5b880b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 -- 172.21.15.104:0/3481238828 >> v1:172.21.15.104:6789/0 conn(0x7f5b8806cde0 legacy=0x7f5b88069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 -- 172.21.15.104:0/3481238828 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b880b1e80 con 0x7f5b8806e6e0 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b8659c700 1 --2- 172.21.15.104:0/3481238828 >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 0x7f5b880b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.280+0000 7f5b85d9b700 1 --2- 172.21.15.104:0/3481238828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880b15b0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5b88070300 tx=0x7f5b78009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0e2b6e0855aa6d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8559a700 1 -- 172.21.15.104:0/3481238828 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011830 con 0x7f5b8806e6e0 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8559a700 1 -- 172.21.15.104:0/3481238828 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78011990 con 0x7f5b8806e6e0 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8559a700 1 -- 172.21.15.104:0/3481238828 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011d20 con 0x7f5b8806e6e0 2023-06-10T18:53:30.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/3481238828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 msgr2=0x7f5b880b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/3481238828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880b15b0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5b88070300 tx=0x7f5b78009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/3481238828 shutdown_connections 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/3481238828 >> v2:172.21.15.114:3300/0 conn(0x7f5b8815c580 0x7f5b880b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/3481238828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880b15b0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/3481238828 >> 172.21.15.104:0/3481238828 conn(0x7f5b8806b8a0 msgr2=0x7f5b880cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/3481238828 shutdown_connections 2023-06-10T18:53:30.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.281+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/3481238828 wait complete. 2023-06-10T18:53:30.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.282+0000 7f5b8cf0d700 1 Processor -- start 2023-06-10T18:53:30.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 -- start start 2023-06-10T18:53:30.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b8806cde0 0x7f5b88155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b8815c580 0x7f5b880b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880c3470 con 0x7f5b8806e6e0 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881f5540 con 0x7f5b8815c580 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b8cf0d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881560c0 con 0x7f5b8806cde0 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b85d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:30.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b85d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34444/0 (socket says 172.21.15.104:34444) 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.283+0000 7f5b85d9b700 1 -- 172.21.15.104:0/1377099517 learned_addr learned my addr 172.21.15.104:0/1377099517 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 -- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b8806cde0 msgr2=0x7f5b88155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b8806cde0 0x7f5b88155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 -- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b8815c580 msgr2=0x7f5b880b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b8815c580 0x7f5b880b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 -- 172.21.15.104:0/1377099517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b780094a0 con 0x7f5b8806e6e0 2023-06-10T18:53:30.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b85d9b700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000c00 tx=0x7f5b78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780038a0 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7801e070 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b880c5770 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78003c60 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.284+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b880c5d30 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.285+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b880c5f20 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.285+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5b7802d450 con 0x7f5b8806e6e0 2023-06-10T18:53:30.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.287+0000 7f5b76ffd700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 0x7f5b6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.287+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5b7808a6d0 con 0x7f5b8806e6e0 2023-06-10T18:53:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.287+0000 7f5b8659c700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 0x7f5b6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.287+0000 7f5b8659c700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 0x7f5b6806d190 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:30.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.291+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5b880c5f20 con 0x7f5b8806e6e0 2023-06-10T18:53:30.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:30.523+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5b880c5f20 con 0x7f5b8806e6e0 2023-06-10T18:53:31.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.456+0000 7f5b76ffd700 1 -- 172.21.15.104:0/1377099517 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5b880c5f20 con 0x7f5b8806e6e0 2023-06-10T18:53:31.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 msgr2=0x7f5b6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 0x7f5b6806d190 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c008040 comp rx=0 tx=0).stop 2023-06-10T18:53:31.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 msgr2=0x7f5b880c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000c00 tx=0x7f5b78004750 comp rx=0 tx=0).stop 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 shutdown_connections 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ace0 0x7f5b6806d190 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b8815c580 0x7f5b880b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b8806cde0 0x7f5b88155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 --2- 172.21.15.104:0/1377099517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b8806e6e0 0x7f5b880c7c50 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 >> 172.21.15.104:0/1377099517 conn(0x7f5b8806b8a0 msgr2=0x7f5b880ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:31.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 shutdown_connections 2023-06-10T18:53:31.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.459+0000 7f5b8cf0d700 1 -- 172.21.15.104:0/1377099517 wait complete. 2023-06-10T18:53:31.479 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:53:31.479 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:53:31.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 Processor -- start 2023-06-10T18:53:31.608 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 -- start start 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb94c15c500 0x7fb94c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb94c0cf200 0x7fb94c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c15c8d0 con 0x7fb94c06a3b0 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0aef20 con 0x7fb94c0cf200 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.604+0000 7fb953236700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c06abd0 con 0x7fb94c15c500 2023-06-10T18:53:31.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb94c0cf200 0x7fb94c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb9517d3700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb94c06a3b0 0x7fb94c06dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60516/0 (socket says 172.21.15.104:60516) 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb9517d3700 1 -- 172.21.15.104:0/628605644 learned_addr learned my addr 172.21.15.104:0/628605644 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb950fd2700 1 --2- 172.21.15.104:0/628605644 >> v2:172.21.15.114:3300/0 conn(0x7fb94c15c500 0x7fb94c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 -- 172.21.15.104:0/628605644 >> v2:172.21.15.114:3300/0 conn(0x7fb94c15c500 msgr2=0x7fb94c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 --2- 172.21.15.104:0/628605644 >> v2:172.21.15.114:3300/0 conn(0x7fb94c15c500 0x7fb94c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 -- 172.21.15.104:0/628605644 >> v1:172.21.15.104:6789/0 conn(0x7fb94c06a3b0 legacy=0x7fb94c06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 -- 172.21.15.104:0/628605644 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c06a8f0 con 0x7fb94c0cf200 2023-06-10T18:53:31.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94b7fe700 1 -- 172.21.15.104:0/628605644 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400787519 0 0) 0x7fb94c15c8d0 con 0x7fb94c06a3b0 2023-06-10T18:53:31.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94bfff700 1 --2- 172.21.15.104:0/628605644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c0cf200 0x7fb94c069e70 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b2170 tx=0x7fb938009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a068aeb3f02a80 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:31.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94b7fe700 1 -- 172.21.15.104:0/628605644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938011830 con 0x7fb94c0cf200 2023-06-10T18:53:31.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94b7fe700 1 -- 172.21.15.104:0/628605644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938011990 con 0x7fb94c0cf200 2023-06-10T18:53:31.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.605+0000 7fb94b7fe700 1 -- 172.21.15.104:0/628605644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938011d20 con 0x7fb94c0cf200 2023-06-10T18:53:31.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 -- 172.21.15.104:0/628605644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c0cf200 msgr2=0x7fb94c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 --2- 172.21.15.104:0/628605644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c0cf200 0x7fb94c069e70 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b2170 tx=0x7fb938009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 -- 172.21.15.104:0/628605644 shutdown_connections 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 --2- 172.21.15.104:0/628605644 >> v2:172.21.15.114:3300/0 conn(0x7fb94c15c500 0x7fb94c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 --2- 172.21.15.104:0/628605644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c0cf200 0x7fb94c069e70 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 -- 172.21.15.104:0/628605644 >> 172.21.15.104:0/628605644 conn(0x7fb94c06b8a0 msgr2=0x7fb94c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 -- 172.21.15.104:0/628605644 shutdown_connections 2023-06-10T18:53:31.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.606+0000 7fb953236700 1 -- 172.21.15.104:0/628605644 wait complete. 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.607+0000 7fb953236700 1 Processor -- start 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.607+0000 7fb953236700 1 -- start start 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94c06a3b0 0x7fb94c151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94c0cf200 0x7fb94c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0c9c40 con 0x7fb94c15c500 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c15ef10 con 0x7fb94c0cf200 2023-06-10T18:53:31.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb953236700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c15f1f0 con 0x7fb94c06a3b0 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.608+0000 7fb9517d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34462/0 (socket says 172.21.15.104:34462) 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 -- 172.21.15.104:0/3510286282 learned_addr learned my addr 172.21.15.104:0/3510286282 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb94bfff700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94c0cf200 0x7fb94c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb950fd2700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94c06a3b0 0x7fb94c151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 -- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94c06a3b0 msgr2=0x7fb94c151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94c06a3b0 0x7fb94c151980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 -- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94c0cf200 msgr2=0x7fb94c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94c0cf200 0x7fb94c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 -- 172.21.15.104:0/3510286282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380094a0 con 0x7fb94c15c500 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9517d3700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb94400cba0 tx=0x7fb94400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94400ab80 con 0x7fb94c15c500 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94400ad00 con 0x7fb94c15c500 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c15f4d0 con 0x7fb94c15c500 2023-06-10T18:53:31.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944007310 con 0x7fb94c15c500 2023-06-10T18:53:31.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.609+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c151ec0 con 0x7fb94c15c500 2023-06-10T18:53:31.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.610+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c0c9df0 con 0x7fb94c15c500 2023-06-10T18:53:31.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.610+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb944007470 con 0x7fb94c15c500 2023-06-10T18:53:31.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.611+0000 7fb9497fa700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 0x7fb93006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:31.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.611+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb944089420 con 0x7fb94c15c500 2023-06-10T18:53:31.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.611+0000 7fb950fd2700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 0x7fb93006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:31.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.611+0000 7fb950fd2700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 0x7fb93006d190 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb940006fd0 tx=0x7fb940008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:31.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.615+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb944054390 con 0x7fb94c15c500 2023-06-10T18:53:31.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:31.847+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb94c074050 con 0x7fb94c15c500 2023-06-10T18:53:32.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.462+0000 7fb9497fa700 1 -- 172.21.15.104:0/3510286282 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb9440579b0 con 0x7fb94c15c500 2023-06-10T18:53:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 msgr2=0x7fb93006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 0x7fb93006d190 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb940006fd0 tx=0x7fb940008040 comp rx=0 tx=0).stop 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 msgr2=0x7fb94c0c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb94400cba0 tx=0x7fb94400cf60 comp rx=0 tx=0).stop 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 shutdown_connections 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c15c500 0x7fb94c0c5690 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94c06a3b0 0x7fb94c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb93006ace0 0x7fb93006d190 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 --2- 172.21.15.104:0/3510286282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94c0cf200 0x7fb94c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 >> 172.21.15.104:0/3510286282 conn(0x7fb94c06b8a0 msgr2=0x7fb94c0aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:32.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 shutdown_connections 2023-06-10T18:53:32.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.465+0000 7fb953236700 1 -- 172.21.15.104:0/3510286282 wait complete. 2023-06-10T18:53:32.484 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:53:32.485 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.605+0000 7ff7d0e15700 1 Processor -- start 2023-06-10T18:53:32.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.606+0000 7ff7d0e15700 1 -- start start 2023-06-10T18:53:32.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.606+0000 7ff7d0e15700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff7cc0aeaa0 0x7ff7cc0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7d0e15700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff7cc0b1080 0x7ff7cc0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7d0e15700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0c9e60 con 0x7ff7cc0ca2f0 2023-06-10T18:53:32.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7d0e15700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c9bc0 con 0x7ff7cc0b1080 2023-06-10T18:53:32.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7d0e15700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc0aee70 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff7cc0b1080 0x7ff7cc0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7cad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff7cc0ca2f0 0x7ff7cc0ca710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60530/0 (socket says 172.21.15.104:60530) 2023-06-10T18:53:32.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7cad9d700 1 -- 172.21.15.104:0/327788244 learned_addr learned my addr 172.21.15.104:0/327788244 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:32.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7ca59c700 1 --2- 172.21.15.104:0/327788244 >> v2:172.21.15.114:3300/0 conn(0x7ff7cc0aeaa0 0x7ff7cc0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 -- 172.21.15.104:0/327788244 >> v2:172.21.15.114:3300/0 conn(0x7ff7cc0aeaa0 msgr2=0x7ff7cc0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:32.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/327788244 >> v2:172.21.15.114:3300/0 conn(0x7ff7cc0aeaa0 0x7ff7cc0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 -- 172.21.15.104:0/327788244 >> v1:172.21.15.104:6789/0 conn(0x7ff7cc0ca2f0 legacy=0x7ff7cc0ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 -- 172.21.15.104:0/327788244 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc06e540 con 0x7ff7cc0b1080 2023-06-10T18:53:32.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c959a700 1 -- 172.21.15.104:0/327788244 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100927103 0 0) 0x7ff7cc0c9e60 con 0x7ff7cc0ca2f0 2023-06-10T18:53:32.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/327788244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0b1080 0x7ff7cc0cd680 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0cf5d0 tx=0x7ff7b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6973753c905f8d66 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c959a700 1 -- 172.21.15.104:0/327788244 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4011830 con 0x7ff7cc0b1080 2023-06-10T18:53:32.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c959a700 1 -- 172.21.15.104:0/327788244 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b4011990 con 0x7ff7cc0b1080 2023-06-10T18:53:32.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.607+0000 7ff7c959a700 1 -- 172.21.15.104:0/327788244 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4011d20 con 0x7ff7cc0b1080 2023-06-10T18:53:32.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 -- 172.21.15.104:0/327788244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0b1080 msgr2=0x7ff7cc0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/327788244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0b1080 0x7ff7cc0cd680 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0cf5d0 tx=0x7ff7b4009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:32.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 -- 172.21.15.104:0/327788244 shutdown_connections 2023-06-10T18:53:32.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/327788244 >> v2:172.21.15.114:3300/0 conn(0x7ff7cc0aeaa0 0x7ff7cc0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/327788244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0b1080 0x7ff7cc0cd680 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 -- 172.21.15.104:0/327788244 >> 172.21.15.104:0/327788244 conn(0x7ff7cc06b8a0 msgr2=0x7ff7cc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 -- 172.21.15.104:0/327788244 shutdown_connections 2023-06-10T18:53:32.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.608+0000 7ff7d0e15700 1 -- 172.21.15.104:0/327788244 wait complete. 2023-06-10T18:53:32.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.610+0000 7ff7d0e15700 1 Processor -- start 2023-06-10T18:53:32.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.610+0000 7ff7d0e15700 1 -- start start 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 0x7ff7cc0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7cc0ca2f0 0x7ff7cc0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c93d0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c8b30 con 0x7ff7cc0ca2f0 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c8e10 con 0x7ff7cc0b1080 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34472/0 (socket says 172.21.15.104:34472) 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 -- 172.21.15.104:0/875541167 learned_addr learned my addr 172.21.15.104:0/875541167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7cad9d700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7cc0ca2f0 0x7ff7cc0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 0x7ff7cc0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 -- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 msgr2=0x7ff7cc0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 0x7ff7cc0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 -- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7cc0ca2f0 msgr2=0x7ff7cc0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7cc0ca2f0 0x7ff7cc0cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 -- 172.21.15.104:0/875541167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40094a0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 0x7ff7cc0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7ca59c700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc010ba0 tx=0x7ff7bc010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0096f0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc00ee00 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc0c90f0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc011650 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.611+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc155430 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.612+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc06e1e0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.613+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff7bc0117b0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.613+0000 7ff7c2ffd700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 0x7ff7ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.613+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff7bc08b8e0 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.613+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 0x7ff7ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.614+0000 7ff7c9d9b700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 0x7ff7ac06d190 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0d0090 tx=0x7ff7b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.618+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff7bc056900 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.845+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7cc074050 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.845+0000 7ff7c2ffd700 1 -- 172.21.15.104:0/875541167 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4682 (secure 0 0 0) 0x7ff7bc01e090 con 0x7ff7cc0aeaa0 2023-06-10T18:53:32.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:32.850 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":55,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:32.455645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7983":{"gid":7983,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3243355534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3243355534},{"type":"v1","addr":"172.21.15.114:6837","nonce":3243355534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:53:32.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 msgr2=0x7ff7ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 0x7ff7ac06d190 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0d0090 tx=0x7ff7b4008030 comp rx=0 tx=0).stop 2023-06-10T18:53:32.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 msgr2=0x7ff7cc154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc010ba0 tx=0x7ff7bc010f60 comp rx=0 tx=0).stop 2023-06-10T18:53:32.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 shutdown_connections 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7ac06ace0 0x7ff7ac06d190 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7cc0ca2f0 0x7ff7cc0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7cc0aeaa0 0x7ff7cc154ef0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 --2- 172.21.15.104:0/875541167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7cc0b1080 0x7ff7cc0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 >> 172.21.15.104:0/875541167 conn(0x7ff7cc06b8a0 msgr2=0x7ff7cc0af060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 shutdown_connections 2023-06-10T18:53:32.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.847+0000 7ff7d0e15700 1 -- 172.21.15.104:0/875541167 wait complete. 2023-06-10T18:53:32.855 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 55 2023-06-10T18:53:32.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:32.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 Processor -- start 2023-06-10T18:53:32.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 -- start start 2023-06-10T18:53:32.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc8c8069e70 0x7fc8c80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c806d290 con 0x7fc8c806e0e0 2023-06-10T18:53:32.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c8071640 con 0x7fc8c8069e70 2023-06-10T18:53:32.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.979+0000 7fc8ce8f4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80aef00 con 0x7fc8c806e4b0 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc8c8069e70 0x7fc8c80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8c77fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc8c8069e70 0x7fc8c80ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:50216/0 (socket says 172.21.15.104:50216) 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 -- 172.21.15.104:0/3452443742 learned_addr learned my addr 172.21.15.104:0/3452443742 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 -- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 msgr2=0x7fc8c806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 -- 172.21.15.104:0/3452443742 >> v1:172.21.15.114:6789/0 conn(0x7fc8c806e0e0 legacy=0x7fc8c80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 -- 172.21.15.104:0/3452443742 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80ca2d0 con 0x7fc8c8069e70 2023-06-10T18:53:32.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8c77fe700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:32.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.980+0000 7fc8cce91700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c8069e70 0x7fc8c80ce9f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009e50 tx=0x7fc8c000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=701b9b16885caa6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8c6ffd700 1 -- 172.21.15.104:0/3452443742 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000e070 con 0x7fc8c8069e70 2023-06-10T18:53:32.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8c6ffd700 1 -- 172.21.15.104:0/3452443742 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000b910 con 0x7fc8c8069e70 2023-06-10T18:53:32.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8c6ffd700 1 -- 172.21.15.104:0/3452443742 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c000bcc0 con 0x7fc8c8069e70 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c8069e70 msgr2=0x7fc8c80ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c8069e70 0x7fc8c80ce9f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009e50 tx=0x7fc8c000b330 comp rx=0 tx=0).stop 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/3452443742 shutdown_connections 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c8069e70 0x7fc8c80ce9f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/3452443742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/3452443742 >> 172.21.15.104:0/3452443742 conn(0x7fc8c806b8a0 msgr2=0x7fc8c80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:32.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/3452443742 shutdown_connections 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.981+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/3452443742 wait complete. 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.982+0000 7fc8ce8f4700 1 Processor -- start 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 -- start start 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c806e0e0 0x7fc8c8156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c80b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8159480 con 0x7fc8c8069e70 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80b37b0 con 0x7fc8c806e0e0 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8ce8f4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80af170 con 0x7fc8c806e4b0 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8c7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34486/0 (socket says 172.21.15.104:34486) 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.983+0000 7fc8c7fff700 1 -- 172.21.15.104:0/1599658241 learned_addr learned my addr 172.21.15.104:0/1599658241 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c77fe700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c806e0e0 0x7fc8c8156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8cce91700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c80b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 -- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 msgr2=0x7fc8c80b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c80b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 -- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c806e0e0 msgr2=0x7fc8c8156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c806e0e0 0x7fc8c8156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 -- 172.21.15.104:0/1599658241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0094a0 con 0x7fc8c8069e70 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c7fff700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800c4d0 tx=0x7fc8b800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80052d0 con 0x7fc8c8069e70 2023-06-10T18:53:32.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b8005430 con 0x7fc8c8069e70 2023-06-10T18:53:32.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80038f0 con 0x7fc8c8069e70 2023-06-10T18:53:32.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c0009b30 con 0x7fc8c8069e70 2023-06-10T18:53:32.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.984+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c80af730 con 0x7fc8c8069e70 2023-06-10T18:53:32.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.985+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c8074050 con 0x7fc8c8069e70 2023-06-10T18:53:32.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.985+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8b8003a50 con 0x7fc8c8069e70 2023-06-10T18:53:32.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.986+0000 7fc8c4ff9700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 0x7fc8b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:32.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.986+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc8b80889c0 con 0x7fc8c8069e70 2023-06-10T18:53:32.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.986+0000 7fc8c77fe700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 0x7fc8b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:32.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.986+0000 7fc8c77fe700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 0x7fc8b406d190 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80c65f0 tx=0x7fc8bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:32.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:32.990+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8b80539c0 con 0x7fc8c8069e70 2023-06-10T18:53:33.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.180+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8c80ca090 con 0x7fc8c8069e70 2023-06-10T18:53:33.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.181+0000 7fc8c4ff9700 1 -- 172.21.15.104:0/1599658241 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+18543 (secure 0 0 0) 0x7fc8b8056fe0 con 0x7fc8c8069e70 2023-06-10T18:53:33.186 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:33.187 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:30.449397+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:33.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 msgr2=0x7fc8b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 0x7fc8b406d190 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80c65f0 tx=0x7fc8bc008040 comp rx=0 tx=0).stop 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 msgr2=0x7fc8c8158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800c4d0 tx=0x7fc8b800c890 comp rx=0 tx=0).stop 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 shutdown_connections 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8c806e4b0 0x7fc8c80b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8c8069e70 0x7fc8c8158f40 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8b406ace0 0x7fc8b406d190 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.183+0000 7fc8ce8f4700 1 --2- 172.21.15.104:0/1599658241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8c806e0e0 0x7fc8c8156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.184+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 >> 172.21.15.104:0/1599658241 conn(0x7fc8c806b8a0 msgr2=0x7fc8c80cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.184+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 shutdown_connections 2023-06-10T18:53:33.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.184+0000 7fc8ce8f4700 1 -- 172.21.15.104:0/1599658241 wait complete. 2023-06-10T18:53:33.200 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:53:33.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:53:33.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.316+0000 7f55236c2700 1 Processor -- start 2023-06-10T18:53:33.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.316+0000 7f55236c2700 1 -- start start 2023-06-10T18:53:33.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f55236c2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 0x7f551c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f55236c2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f551c06e4b0 0x7f551c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f55236c2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551c06d290 con 0x7f551c0ca100 2023-06-10T18:53:33.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f55236c2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c071640 con 0x7f551c06e4b0 2023-06-10T18:53:33.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f55236c2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c0aef00 con 0x7f551c06e0e0 2023-06-10T18:53:33.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f551c06e4b0 0x7f551c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5521c5f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f551c0ca100 0x7f551c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60556/0 (socket says 172.21.15.104:60556) 2023-06-10T18:53:33.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f551c06e4b0 0x7f551c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34494/0 (socket says 172.21.15.104:34494) 2023-06-10T18:53:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5521c5f700 1 -- 172.21.15.104:0/4292302954 learned_addr learned my addr 172.21.15.104:0/4292302954 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f552145e700 1 --2- 172.21.15.104:0/4292302954 >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 0x7f551c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 -- 172.21.15.104:0/4292302954 >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 msgr2=0x7f551c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 --2- 172.21.15.104:0/4292302954 >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 0x7f551c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 -- 172.21.15.104:0/4292302954 >> v1:172.21.15.104:6789/0 conn(0x7f551c0ca100 legacy=0x7f551c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 -- 172.21.15.104:0/4292302954 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c06d4c0 con 0x7f551c06e4b0 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5513fff700 1 -- 172.21.15.104:0/4292302954 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992827975 0 0) 0x7f551c06d290 con 0x7f551c0ca100 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f552145e700 1 --2- 172.21.15.104:0/4292302954 >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 0x7f551c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5520c5d700 1 --2- 172.21.15.104:0/4292302954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e4b0 0x7f551c0c9bc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f551c06af50 tx=0x7f550c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84da962e420b471d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5513fff700 1 -- 172.21.15.104:0/4292302954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c011830 con 0x7f551c06e4b0 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5513fff700 1 -- 172.21.15.104:0/4292302954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c011990 con 0x7f551c06e4b0 2023-06-10T18:53:33.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.317+0000 7f5513fff700 1 -- 172.21.15.104:0/4292302954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c011d20 con 0x7f551c06e4b0 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 -- 172.21.15.104:0/4292302954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e4b0 msgr2=0x7f551c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 --2- 172.21.15.104:0/4292302954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e4b0 0x7f551c0c9bc0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f551c06af50 tx=0x7f550c009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 -- 172.21.15.104:0/4292302954 shutdown_connections 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 --2- 172.21.15.104:0/4292302954 >> v2:172.21.15.114:3300/0 conn(0x7f551c06e0e0 0x7f551c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 --2- 172.21.15.104:0/4292302954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e4b0 0x7f551c0c9bc0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 -- 172.21.15.104:0/4292302954 >> 172.21.15.104:0/4292302954 conn(0x7f551c06b8a0 msgr2=0x7f551c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:33.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 -- 172.21.15.104:0/4292302954 shutdown_connections 2023-06-10T18:53:33.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.318+0000 7f55236c2700 1 -- 172.21.15.104:0/4292302954 wait complete. 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.319+0000 7f55236c2700 1 Processor -- start 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.319+0000 7f55236c2700 1 -- start start 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 0x7f551c0b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f551c0ca100 0x7f551c0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c1f5c70 con 0x7f551c06e0e0 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c1f1b50 con 0x7f551c06e4b0 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f55236c2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c1f5990 con 0x7f551c0ca100 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34498/0 (socket says 172.21.15.104:34498) 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 -- 172.21.15.104:0/438525485 learned_addr learned my addr 172.21.15.104:0/438525485 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f5520c5d700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 0x7f551c0b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f5521c5f700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f551c0ca100 0x7f551c0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 -- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f551c0ca100 msgr2=0x7f551c0b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f551c0ca100 0x7f551c0b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 -- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 msgr2=0x7f551c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 0x7f551c0b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 -- 172.21.15.104:0/438525485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c0094a0 con 0x7f551c06e0e0 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f5520c5d700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 0x7f551c0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f552145e700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f551800e810 tx=0x7f551800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.320+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518005c20 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.321+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518005d80 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.321+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518006230 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.321+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c0af740 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.321+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c157d30 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.322+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c074050 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.322+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5518006390 con 0x7f551c06e0e0 2023-06-10T18:53:33.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.322+0000 7f5511ffb700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 0x7f550006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.322+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5518089090 con 0x7f551c06e0e0 2023-06-10T18:53:33.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.323+0000 7f5520c5d700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 0x7f550006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.323+0000 7f5520c5d700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 0x7f550006d190 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f551c0ac960 tx=0x7f550c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.327+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5518054090 con 0x7f551c06e0e0 2023-06-10T18:53:33.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.509+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f551c06da80 con 0x7f550006ace0 2023-06-10T18:53:33.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.518+0000 7f5511ffb700 1 -- 172.21.15.104:0/438525485 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312079 (secure 0 0 0) 0x7f550c011b80 con 0x7f550006ace0 2023-06-10T18:53:33.526 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:33.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 msgr2=0x7f550006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 0x7f550006d190 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f551c0ac960 tx=0x7f550c008030 comp rx=0 tx=0).stop 2023-06-10T18:53:33.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 msgr2=0x7f551c1577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f551800e810 tx=0x7f551800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 shutdown_connections 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f551c0ca100 0x7f551c0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f551c06e0e0 0x7f551c1577f0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f550006ace0 0x7f550006d190 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 --2- 172.21.15.104:0/438525485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f551c06e4b0 0x7f551c0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 >> 172.21.15.104:0/438525485 conn(0x7f551c06b8a0 msgr2=0x7f551c0acd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:33.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 shutdown_connections 2023-06-10T18:53:33.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.522+0000 7f55236c2700 1 -- 172.21.15.104:0/438525485 wait complete. 2023-06-10T18:53:33.530 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:53:33.538 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2023-06-10T18:53:33.501211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226348,"kb_used_data":10732,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754748372,"statfs":{"total":773094113280,"available":772862332928,"internally_reserved":0,"allocated":10989568,"data_stored":5875635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.455934"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456131+0000","last_change":"2023-06-10T18:53:29.473468+0000","last_active":"2023-06-10T18:53:30.456131+0000","last_peered":"2023-06-10T18:53:30.456131+0000","last_clean":"2023-06-10T18:53:30.456131+0000","last_became_active":"2023-06-10T18:53:29.472929+0000","last_became_peered":"2023-06-10T18:53:29.472929+0000","last_unstale":"2023-06-10T18:53:30.456131+0000","last_undegraded":"2023-06-10T18:53:30.456131+0000","last_fullsized":"2023-06-10T18:53:30.456131+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:08.442260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832063+0000","last_change":"2023-06-10T18:53:29.468899+0000","last_active":"2023-06-10T18:53:30.832063+0000","last_peered":"2023-06-10T18:53:30.832063+0000","last_clean":"2023-06-10T18:53:30.832063+0000","last_became_active":"2023-06-10T18:53:29.468356+0000","last_became_peered":"2023-06-10T18:53:29.468356+0000","last_unstale":"2023-06-10T18:53:30.832063+0000","last_undegraded":"2023-06-10T18:53:30.832063+0000","last_fullsized":"2023-06-10T18:53:30.832063+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:54:31.797527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457010+0000","last_change":"2023-06-10T18:53:29.476168+0000","last_active":"2023-06-10T18:53:30.457010+0000","last_peered":"2023-06-10T18:53:30.457010+0000","last_clean":"2023-06-10T18:53:30.457010+0000","last_became_active":"2023-06-10T18:53:29.473795+0000","last_became_peered":"2023-06-10T18:53:29.473795+0000","last_unstale":"2023-06-10T18:53:30.457010+0000","last_undegraded":"2023-06-10T18:53:30.457010+0000","last_fullsized":"2023-06-10T18:53:30.457010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:30.502910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456308+0000","last_change":"2023-06-10T18:53:29.471411+0000","last_active":"2023-06-10T18:53:30.456308+0000","last_peered":"2023-06-10T18:53:30.456308+0000","last_clean":"2023-06-10T18:53:30.456308+0000","last_became_active":"2023-06-10T18:53:29.471090+0000","last_became_peered":"2023-06-10T18:53:29.471090+0000","last_unstale":"2023-06-10T18:53:30.456308+0000","last_undegraded":"2023-06-10T18:53:30.456308+0000","last_fullsized":"2023-06-10T18:53:30.456308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:25:34.855338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833208+0000","last_change":"2023-06-10T18:53:29.476471+0000","last_active":"2023-06-10T18:53:30.833208+0000","last_peered":"2023-06-10T18:53:30.833208+0000","last_clean":"2023-06-10T18:53:30.833208+0000","last_became_active":"2023-06-10T18:53:29.476082+0000","last_became_peered":"2023-06-10T18:53:29.476082+0000","last_unstale":"2023-06-10T18:53:30.833208+0000","last_undegraded":"2023-06-10T18:53:30.833208+0000","last_fullsized":"2023-06-10T18:53:30.833208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:47:50.940134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831933+0000","last_change":"2023-06-10T18:53:29.467713+0000","last_active":"2023-06-10T18:53:30.831933+0000","last_peered":"2023-06-10T18:53:30.831933+0000","last_clean":"2023-06-10T18:53:30.831933+0000","last_became_active":"2023-06-10T18:53:29.467419+0000","last_became_peered":"2023-06-10T18:53:29.467419+0000","last_unstale":"2023-06-10T18:53:30.831933+0000","last_undegraded":"2023-06-10T18:53:30.831933+0000","last_fullsized":"2023-06-10T18:53:30.831933+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:30:13.582711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831962+0000","last_change":"2023-06-10T18:53:29.467663+0000","last_active":"2023-06-10T18:53:30.831962+0000","last_peered":"2023-06-10T18:53:30.831962+0000","last_clean":"2023-06-10T18:53:30.831962+0000","last_became_active":"2023-06-10T18:53:29.467363+0000","last_became_peered":"2023-06-10T18:53:29.467363+0000","last_unstale":"2023-06-10T18:53:30.831962+0000","last_undegraded":"2023-06-10T18:53:30.831962+0000","last_fullsized":"2023-06-10T18:53:30.831962+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:41:22.872171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831717+0000","last_change":"2023-06-10T18:53:29.467193+0000","last_active":"2023-06-10T18:53:30.831717+0000","last_peered":"2023-06-10T18:53:30.831717+0000","last_clean":"2023-06-10T18:53:30.831717+0000","last_became_active":"2023-06-10T18:53:29.466952+0000","last_became_peered":"2023-06-10T18:53:29.466952+0000","last_unstale":"2023-06-10T18:53:30.831717+0000","last_undegraded":"2023-06-10T18:53:30.831717+0000","last_fullsized":"2023-06-10T18:53:30.831717+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:46:20.365273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832453+0000","last_change":"2023-06-10T18:53:29.464465+0000","last_active":"2023-06-10T18:53:30.832453+0000","last_peered":"2023-06-10T18:53:30.832453+0000","last_clean":"2023-06-10T18:53:30.832453+0000","last_became_active":"2023-06-10T18:53:29.463818+0000","last_became_peered":"2023-06-10T18:53:29.463818+0000","last_unstale":"2023-06-10T18:53:30.832453+0000","last_undegraded":"2023-06-10T18:53:30.832453+0000","last_fullsized":"2023-06-10T18:53:30.832453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:40:42.181154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833136+0000","last_change":"2023-06-10T18:53:29.479289+0000","last_active":"2023-06-10T18:53:30.833136+0000","last_peered":"2023-06-10T18:53:30.833136+0000","last_clean":"2023-06-10T18:53:30.833136+0000","last_became_active":"2023-06-10T18:53:29.479008+0000","last_became_peered":"2023-06-10T18:53:29.479008+0000","last_unstale":"2023-06-10T18:53:30.833136+0000","last_undegraded":"2023-06-10T18:53:30.833136+0000","last_fullsized":"2023-06-10T18:53:30.833136+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:53:14.016600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832649+0000","last_change":"2023-06-10T18:53:29.468294+0000","last_active":"2023-06-10T18:53:30.832649+0000","last_peered":"2023-06-10T18:53:30.832649+0000","last_clean":"2023-06-10T18:53:30.832649+0000","last_became_active":"2023-06-10T18:53:29.467728+0000","last_became_peered":"2023-06-10T18:53:29.467728+0000","last_unstale":"2023-06-10T18:53:30.832649+0000","last_undegraded":"2023-06-10T18:53:30.832649+0000","last_fullsized":"2023-06-10T18:53:30.832649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:13.984214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832125+0000","last_change":"2023-06-10T18:53:29.468947+0000","last_active":"2023-06-10T18:53:30.832125+0000","last_peered":"2023-06-10T18:53:30.832125+0000","last_clean":"2023-06-10T18:53:30.832125+0000","last_became_active":"2023-06-10T18:53:29.468510+0000","last_became_peered":"2023-06-10T18:53:29.468510+0000","last_unstale":"2023-06-10T18:53:30.832125+0000","last_undegraded":"2023-06-10T18:53:30.832125+0000","last_fullsized":"2023-06-10T18:53:30.832125+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:16.232278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455933+0000","last_change":"2023-06-10T18:53:29.466512+0000","last_active":"2023-06-10T18:53:30.455933+0000","last_peered":"2023-06-10T18:53:30.455933+0000","last_clean":"2023-06-10T18:53:30.455933+0000","last_became_active":"2023-06-10T18:53:29.466271+0000","last_became_peered":"2023-06-10T18:53:29.466271+0000","last_unstale":"2023-06-10T18:53:30.455933+0000","last_undegraded":"2023-06-10T18:53:30.455933+0000","last_fullsized":"2023-06-10T18:53:30.455933+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:46.872625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831960+0000","last_change":"2023-06-10T18:53:29.471775+0000","last_active":"2023-06-10T18:53:30.831960+0000","last_peered":"2023-06-10T18:53:30.831960+0000","last_clean":"2023-06-10T18:53:30.831960+0000","last_became_active":"2023-06-10T18:53:29.471588+0000","last_became_peered":"2023-06-10T18:53:29.471588+0000","last_unstale":"2023-06-10T18:53:30.831960+0000","last_undegraded":"2023-06-10T18:53:30.831960+0000","last_fullsized":"2023-06-10T18:53:30.831960+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:25:14.966837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456237+0000","last_change":"2023-06-10T18:53:29.468656+0000","last_active":"2023-06-10T18:53:30.456237+0000","last_peered":"2023-06-10T18:53:30.456237+0000","last_clean":"2023-06-10T18:53:30.456237+0000","last_became_active":"2023-06-10T18:53:29.468442+0000","last_became_peered":"2023-06-10T18:53:29.468442+0000","last_unstale":"2023-06-10T18:53:30.456237+0000","last_undegraded":"2023-06-10T18:53:30.456237+0000","last_fullsized":"2023-06-10T18:53:30.456237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:10:30.321365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831735+0000","last_change":"2023-06-10T18:53:29.474996+0000","last_active":"2023-06-10T18:53:30.831735+0000","last_peered":"2023-06-10T18:53:30.831735+0000","last_clean":"2023-06-10T18:53:30.831735+0000","last_became_active":"2023-06-10T18:53:29.474740+0000","last_became_peered":"2023-06-10T18:53:29.474740+0000","last_unstale":"2023-06-10T18:53:30.831735+0000","last_undegraded":"2023-06-10T18:53:30.831735+0000","last_fullsized":"2023-06-10T18:53:30.831735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:33:10.398542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831928+0000","last_change":"2023-06-10T18:53:29.462171+0000","last_active":"2023-06-10T18:53:30.831928+0000","last_peered":"2023-06-10T18:53:30.831928+0000","last_clean":"2023-06-10T18:53:30.831928+0000","last_became_active":"2023-06-10T18:53:29.461879+0000","last_became_peered":"2023-06-10T18:53:29.461879+0000","last_unstale":"2023-06-10T18:53:30.831928+0000","last_undegraded":"2023-06-10T18:53:30.831928+0000","last_fullsized":"2023-06-10T18:53:30.831928+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:59:28.264281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456609+0000","last_change":"2023-06-10T18:53:29.471759+0000","last_active":"2023-06-10T18:53:30.456609+0000","last_peered":"2023-06-10T18:53:30.456609+0000","last_clean":"2023-06-10T18:53:30.456609+0000","last_became_active":"2023-06-10T18:53:29.471532+0000","last_became_peered":"2023-06-10T18:53:29.471532+0000","last_unstale":"2023-06-10T18:53:30.456609+0000","last_undegraded":"2023-06-10T18:53:30.456609+0000","last_fullsized":"2023-06-10T18:53:30.456609+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:45.644937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832024+0000","last_change":"2023-06-10T18:53:29.473924+0000","last_active":"2023-06-10T18:53:30.832024+0000","last_peered":"2023-06-10T18:53:30.832024+0000","last_clean":"2023-06-10T18:53:30.832024+0000","last_became_active":"2023-06-10T18:53:29.473391+0000","last_became_peered":"2023-06-10T18:53:29.473391+0000","last_unstale":"2023-06-10T18:53:30.832024+0000","last_undegraded":"2023-06-10T18:53:30.832024+0000","last_fullsized":"2023-06-10T18:53:30.832024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:57:46.625564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832327+0000","last_change":"2023-06-10T18:53:29.466723+0000","last_active":"2023-06-10T18:53:30.832327+0000","last_peered":"2023-06-10T18:53:30.832327+0000","last_clean":"2023-06-10T18:53:30.832327+0000","last_became_active":"2023-06-10T18:53:29.466453+0000","last_became_peered":"2023-06-10T18:53:29.466453+0000","last_unstale":"2023-06-10T18:53:30.832327+0000","last_undegraded":"2023-06-10T18:53:30.832327+0000","last_fullsized":"2023-06-10T18:53:30.832327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:01:11.087102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831636+0000","last_change":"2023-06-10T18:53:29.468433+0000","last_active":"2023-06-10T18:53:30.831636+0000","last_peered":"2023-06-10T18:53:30.831636+0000","last_clean":"2023-06-10T18:53:30.831636+0000","last_became_active":"2023-06-10T18:53:29.468177+0000","last_became_peered":"2023-06-10T18:53:29.468177+0000","last_unstale":"2023-06-10T18:53:30.831636+0000","last_undegraded":"2023-06-10T18:53:30.831636+0000","last_fullsized":"2023-06-10T18:53:30.831636+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:10:17.772880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831581+0000","last_change":"2023-06-10T18:53:29.468054+0000","last_active":"2023-06-10T18:53:30.831581+0000","last_peered":"2023-06-10T18:53:30.831581+0000","last_clean":"2023-06-10T18:53:30.831581+0000","last_became_active":"2023-06-10T18:53:29.467787+0000","last_became_peered":"2023-06-10T18:53:29.467787+0000","last_unstale":"2023-06-10T18:53:30.831581+0000","last_undegraded":"2023-06-10T18:53:30.831581+0000","last_fullsized":"2023-06-10T18:53:30.831581+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:14.553417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832186+0000","last_change":"2023-06-10T18:53:29.467089+0000","last_active":"2023-06-10T18:53:30.832186+0000","last_peered":"2023-06-10T18:53:30.832186+0000","last_clean":"2023-06-10T18:53:30.832186+0000","last_became_active":"2023-06-10T18:53:29.466834+0000","last_became_peered":"2023-06-10T18:53:29.466834+0000","last_unstale":"2023-06-10T18:53:30.832186+0000","last_undegraded":"2023-06-10T18:53:30.832186+0000","last_fullsized":"2023-06-10T18:53:30.832186+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:51.384932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831779+0000","last_change":"2023-06-10T18:53:29.478802+0000","last_active":"2023-06-10T18:53:30.831779+0000","last_peered":"2023-06-10T18:53:30.831779+0000","last_clean":"2023-06-10T18:53:30.831779+0000","last_became_active":"2023-06-10T18:53:29.478200+0000","last_became_peered":"2023-06-10T18:53:29.478200+0000","last_unstale":"2023-06-10T18:53:30.831779+0000","last_undegraded":"2023-06-10T18:53:30.831779+0000","last_fullsized":"2023-06-10T18:53:30.831779+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:52:20.030705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833513+0000","last_change":"2023-06-10T18:53:29.475014+0000","last_active":"2023-06-10T18:53:30.833513+0000","last_peered":"2023-06-10T18:53:30.833513+0000","last_clean":"2023-06-10T18:53:30.833513+0000","last_became_active":"2023-06-10T18:53:29.474785+0000","last_became_peered":"2023-06-10T18:53:29.474785+0000","last_unstale":"2023-06-10T18:53:30.833513+0000","last_undegraded":"2023-06-10T18:53:30.833513+0000","last_fullsized":"2023-06-10T18:53:30.833513+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:39:12.553956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832008+0000","last_change":"2023-06-10T18:53:29.468742+0000","last_active":"2023-06-10T18:53:30.832008+0000","last_peered":"2023-06-10T18:53:30.832008+0000","last_clean":"2023-06-10T18:53:30.832008+0000","last_became_active":"2023-06-10T18:53:29.468084+0000","last_became_peered":"2023-06-10T18:53:29.468084+0000","last_unstale":"2023-06-10T18:53:30.832008+0000","last_undegraded":"2023-06-10T18:53:30.832008+0000","last_fullsized":"2023-06-10T18:53:30.832008+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:16:20.375143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832868+0000","last_change":"2023-06-10T18:53:29.482882+0000","last_active":"2023-06-10T18:53:30.832868+0000","last_peered":"2023-06-10T18:53:30.832868+0000","last_clean":"2023-06-10T18:53:30.832868+0000","last_became_active":"2023-06-10T18:53:29.482654+0000","last_became_peered":"2023-06-10T18:53:29.482654+0000","last_unstale":"2023-06-10T18:53:30.832868+0000","last_undegraded":"2023-06-10T18:53:30.832868+0000","last_fullsized":"2023-06-10T18:53:30.832868+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:05:38.824105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833198+0000","last_change":"2023-06-10T18:53:29.476266+0000","last_active":"2023-06-10T18:53:30.833198+0000","last_peered":"2023-06-10T18:53:30.833198+0000","last_clean":"2023-06-10T18:53:30.833198+0000","last_became_active":"2023-06-10T18:53:29.475998+0000","last_became_peered":"2023-06-10T18:53:29.475998+0000","last_unstale":"2023-06-10T18:53:30.833198+0000","last_undegraded":"2023-06-10T18:53:30.833198+0000","last_fullsized":"2023-06-10T18:53:30.833198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:05:47.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831735+0000","last_change":"2023-06-10T18:53:29.469956+0000","last_active":"2023-06-10T18:53:30.831735+0000","last_peered":"2023-06-10T18:53:30.831735+0000","last_clean":"2023-06-10T18:53:30.831735+0000","last_became_active":"2023-06-10T18:53:29.469719+0000","last_became_peered":"2023-06-10T18:53:29.469719+0000","last_unstale":"2023-06-10T18:53:30.831735+0000","last_undegraded":"2023-06-10T18:53:30.831735+0000","last_fullsized":"2023-06-10T18:53:30.831735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:10:57.831967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831837+0000","last_change":"2023-06-10T18:53:29.469920+0000","last_active":"2023-06-10T18:53:30.831837+0000","last_peered":"2023-06-10T18:53:30.831837+0000","last_clean":"2023-06-10T18:53:30.831837+0000","last_became_active":"2023-06-10T18:53:29.469669+0000","last_became_peered":"2023-06-10T18:53:29.469669+0000","last_unstale":"2023-06-10T18:53:30.831837+0000","last_undegraded":"2023-06-10T18:53:30.831837+0000","last_fullsized":"2023-06-10T18:53:30.831837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:02.054872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456371+0000","last_change":"2023-06-10T18:53:29.474173+0000","last_active":"2023-06-10T18:53:30.456371+0000","last_peered":"2023-06-10T18:53:30.456371+0000","last_clean":"2023-06-10T18:53:30.456371+0000","last_became_active":"2023-06-10T18:53:29.473900+0000","last_became_peered":"2023-06-10T18:53:29.473900+0000","last_unstale":"2023-06-10T18:53:30.456371+0000","last_undegraded":"2023-06-10T18:53:30.456371+0000","last_fullsized":"2023-06-10T18:53:30.456371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:35.490497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455832+0000","last_change":"2023-06-10T18:53:29.477745+0000","last_active":"2023-06-10T18:53:30.455832+0000","last_peered":"2023-06-10T18:53:30.455832+0000","last_clean":"2023-06-10T18:53:30.455832+0000","last_became_active":"2023-06-10T18:53:29.476736+0000","last_became_peered":"2023-06-10T18:53:29.476736+0000","last_unstale":"2023-06-10T18:53:30.455832+0000","last_undegraded":"2023-06-10T18:53:30.455832+0000","last_fullsized":"2023-06-10T18:53:30.455832+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:41:38.229832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832207+0000","last_change":"2023-06-10T18:53:29.467293+0000","last_active":"2023-06-10T18:53:30.832207+0000","last_peered":"2023-06-10T18:53:30.832207+0000","last_clean":"2023-06-10T18:53:30.832207+0000","last_became_active":"2023-06-10T18:53:29.467018+0000","last_became_peered":"2023-06-10T18:53:29.467018+0000","last_unstale":"2023-06-10T18:53:30.832207+0000","last_undegraded":"2023-06-10T18:53:30.832207+0000","last_fullsized":"2023-06-10T18:53:30.832207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:09:01.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832030+0000","last_change":"2023-06-10T18:53:29.066047+0000","last_active":"2023-06-10T18:53:30.832030+0000","last_peered":"2023-06-10T18:53:30.832030+0000","last_clean":"2023-06-10T18:53:30.832030+0000","last_became_active":"2023-06-10T18:53:29.065782+0000","last_became_peered":"2023-06-10T18:53:29.065782+0000","last_unstale":"2023-06-10T18:53:30.832030+0000","last_undegraded":"2023-06-10T18:53:30.832030+0000","last_fullsized":"2023-06-10T18:53:30.832030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:56:36.797675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456780+0000","last_change":"2023-06-10T18:53:29.470770+0000","last_active":"2023-06-10T18:53:30.456780+0000","last_peered":"2023-06-10T18:53:30.456780+0000","last_clean":"2023-06-10T18:53:30.456780+0000","last_became_active":"2023-06-10T18:53:29.470511+0000","last_became_peered":"2023-06-10T18:53:29.470511+0000","last_unstale":"2023-06-10T18:53:30.456780+0000","last_undegraded":"2023-06-10T18:53:30.456780+0000","last_fullsized":"2023-06-10T18:53:30.456780+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:55:05.441828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832367+0000","last_change":"2023-06-10T18:53:29.069373+0000","last_active":"2023-06-10T18:53:30.832367+0000","last_peered":"2023-06-10T18:53:30.832367+0000","last_clean":"2023-06-10T18:53:30.832367+0000","last_became_active":"2023-06-10T18:53:29.069093+0000","last_became_peered":"2023-06-10T18:53:29.069093+0000","last_unstale":"2023-06-10T18:53:30.832367+0000","last_undegraded":"2023-06-10T18:53:30.832367+0000","last_fullsized":"2023-06-10T18:53:30.832367+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:45:53.856459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456189+0000","last_change":"2023-06-10T18:53:29.473265+0000","last_active":"2023-06-10T18:53:30.456189+0000","last_peered":"2023-06-10T18:53:30.456189+0000","last_clean":"2023-06-10T18:53:30.456189+0000","last_became_active":"2023-06-10T18:53:29.472970+0000","last_became_peered":"2023-06-10T18:53:29.472970+0000","last_unstale":"2023-06-10T18:53:30.456189+0000","last_undegraded":"2023-06-10T18:53:30.456189+0000","last_fullsized":"2023-06-10T18:53:30.456189+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:25.641646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832082+0000","last_change":"2023-06-10T18:53:29.068224+0000","last_active":"2023-06-10T18:53:30.832082+0000","last_peered":"2023-06-10T18:53:30.832082+0000","last_clean":"2023-06-10T18:53:30.832082+0000","last_became_active":"2023-06-10T18:53:29.068029+0000","last_became_peered":"2023-06-10T18:53:29.068029+0000","last_unstale":"2023-06-10T18:53:30.832082+0000","last_undegraded":"2023-06-10T18:53:30.832082+0000","last_fullsized":"2023-06-10T18:53:30.832082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:47.062679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833185+0000","last_change":"2023-06-10T18:53:29.471357+0000","last_active":"2023-06-10T18:53:30.833185+0000","last_peered":"2023-06-10T18:53:30.833185+0000","last_clean":"2023-06-10T18:53:30.833185+0000","last_became_active":"2023-06-10T18:53:29.470949+0000","last_became_peered":"2023-06-10T18:53:29.470949+0000","last_unstale":"2023-06-10T18:53:30.833185+0000","last_undegraded":"2023-06-10T18:53:30.833185+0000","last_fullsized":"2023-06-10T18:53:30.833185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:55:48.365827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832273+0000","last_change":"2023-06-10T18:53:29.061893+0000","last_active":"2023-06-10T18:53:30.832273+0000","last_peered":"2023-06-10T18:53:30.832273+0000","last_clean":"2023-06-10T18:53:30.832273+0000","last_became_active":"2023-06-10T18:53:29.061638+0000","last_became_peered":"2023-06-10T18:53:29.061638+0000","last_unstale":"2023-06-10T18:53:30.832273+0000","last_undegraded":"2023-06-10T18:53:30.832273+0000","last_fullsized":"2023-06-10T18:53:30.832273+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:37:28.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457535+0000","last_change":"2023-06-10T18:53:29.470107+0000","last_active":"2023-06-10T18:53:30.457535+0000","last_peered":"2023-06-10T18:53:30.457535+0000","last_clean":"2023-06-10T18:53:30.457535+0000","last_became_active":"2023-06-10T18:53:29.469843+0000","last_became_peered":"2023-06-10T18:53:29.469843+0000","last_unstale":"2023-06-10T18:53:30.457535+0000","last_undegraded":"2023-06-10T18:53:30.457535+0000","last_fullsized":"2023-06-10T18:53:30.457535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:12.080153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457050+0000","last_change":"2023-06-10T18:53:29.069038+0000","last_active":"2023-06-10T18:53:30.457050+0000","last_peered":"2023-06-10T18:53:30.457050+0000","last_clean":"2023-06-10T18:53:30.457050+0000","last_became_active":"2023-06-10T18:53:29.068743+0000","last_became_peered":"2023-06-10T18:53:29.068743+0000","last_unstale":"2023-06-10T18:53:30.457050+0000","last_undegraded":"2023-06-10T18:53:30.457050+0000","last_fullsized":"2023-06-10T18:53:30.457050+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:09:00.074642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831802+0000","last_change":"2023-06-10T18:53:29.473771+0000","last_active":"2023-06-10T18:53:30.831802+0000","last_peered":"2023-06-10T18:53:30.831802+0000","last_clean":"2023-06-10T18:53:30.831802+0000","last_became_active":"2023-06-10T18:53:29.473502+0000","last_became_peered":"2023-06-10T18:53:29.473502+0000","last_unstale":"2023-06-10T18:53:30.831802+0000","last_undegraded":"2023-06-10T18:53:30.831802+0000","last_fullsized":"2023-06-10T18:53:30.831802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:51:06.717229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831677+0000","last_change":"2023-06-10T18:53:29.066567+0000","last_active":"2023-06-10T18:53:30.831677+0000","last_peered":"2023-06-10T18:53:30.831677+0000","last_clean":"2023-06-10T18:53:30.831677+0000","last_became_active":"2023-06-10T18:53:29.065453+0000","last_became_peered":"2023-06-10T18:53:29.065453+0000","last_unstale":"2023-06-10T18:53:30.831677+0000","last_undegraded":"2023-06-10T18:53:30.831677+0000","last_fullsized":"2023-06-10T18:53:30.831677+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:32.068761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456034+0000","last_change":"2023-06-10T18:53:29.469242+0000","last_active":"2023-06-10T18:53:30.456034+0000","last_peered":"2023-06-10T18:53:30.456034+0000","last_clean":"2023-06-10T18:53:30.456034+0000","last_became_active":"2023-06-10T18:53:29.468990+0000","last_became_peered":"2023-06-10T18:53:29.468990+0000","last_unstale":"2023-06-10T18:53:30.456034+0000","last_undegraded":"2023-06-10T18:53:30.456034+0000","last_fullsized":"2023-06-10T18:53:30.456034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:15:45.624718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833031+0000","last_change":"2023-06-10T18:53:29.061978+0000","last_active":"2023-06-10T18:53:30.833031+0000","last_peered":"2023-06-10T18:53:30.833031+0000","last_clean":"2023-06-10T18:53:30.833031+0000","last_became_active":"2023-06-10T18:53:29.061625+0000","last_became_peered":"2023-06-10T18:53:29.061625+0000","last_unstale":"2023-06-10T18:53:30.833031+0000","last_undegraded":"2023-06-10T18:53:30.833031+0000","last_fullsized":"2023-06-10T18:53:30.833031+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.673427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831835+0000","last_change":"2023-06-10T18:53:29.469424+0000","last_active":"2023-06-10T18:53:30.831835+0000","last_peered":"2023-06-10T18:53:30.831835+0000","last_clean":"2023-06-10T18:53:30.831835+0000","last_became_active":"2023-06-10T18:53:29.469118+0000","last_became_peered":"2023-06-10T18:53:29.469118+0000","last_unstale":"2023-06-10T18:53:30.831835+0000","last_undegraded":"2023-06-10T18:53:30.831835+0000","last_fullsized":"2023-06-10T18:53:30.831835+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:05.039746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455691+0000","last_change":"2023-06-10T18:53:29.070985+0000","last_active":"2023-06-10T18:53:30.455691+0000","last_peered":"2023-06-10T18:53:30.455691+0000","last_clean":"2023-06-10T18:53:30.455691+0000","last_became_active":"2023-06-10T18:53:29.070535+0000","last_became_peered":"2023-06-10T18:53:29.070535+0000","last_unstale":"2023-06-10T18:53:30.455691+0000","last_undegraded":"2023-06-10T18:53:30.455691+0000","last_fullsized":"2023-06-10T18:53:30.455691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:07:54.057733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832207+0000","last_change":"2023-06-10T18:53:29.469415+0000","last_active":"2023-06-10T18:53:30.832207+0000","last_peered":"2023-06-10T18:53:30.832207+0000","last_clean":"2023-06-10T18:53:30.832207+0000","last_became_active":"2023-06-10T18:53:29.469147+0000","last_became_peered":"2023-06-10T18:53:29.469147+0000","last_unstale":"2023-06-10T18:53:30.832207+0000","last_undegraded":"2023-06-10T18:53:30.832207+0000","last_fullsized":"2023-06-10T18:53:30.832207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:29.479497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456377+0000","last_change":"2023-06-10T18:53:29.071101+0000","last_active":"2023-06-10T18:53:30.456377+0000","last_peered":"2023-06-10T18:53:30.456377+0000","last_clean":"2023-06-10T18:53:30.456377+0000","last_became_active":"2023-06-10T18:53:29.070820+0000","last_became_peered":"2023-06-10T18:53:29.070820+0000","last_unstale":"2023-06-10T18:53:30.456377+0000","last_undegraded":"2023-06-10T18:53:30.456377+0000","last_fullsized":"2023-06-10T18:53:30.456377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:48:30.964846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832550+0000","last_change":"2023-06-10T18:53:29.469938+0000","last_active":"2023-06-10T18:53:30.832550+0000","last_peered":"2023-06-10T18:53:30.832550+0000","last_clean":"2023-06-10T18:53:30.832550+0000","last_became_active":"2023-06-10T18:53:29.469506+0000","last_became_peered":"2023-06-10T18:53:29.469506+0000","last_unstale":"2023-06-10T18:53:30.832550+0000","last_undegraded":"2023-06-10T18:53:30.832550+0000","last_fullsized":"2023-06-10T18:53:30.832550+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:52:50.866828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456697+0000","last_change":"2023-06-10T18:53:29.062132+0000","last_active":"2023-06-10T18:53:30.456697+0000","last_peered":"2023-06-10T18:53:30.456697+0000","last_clean":"2023-06-10T18:53:30.456697+0000","last_became_active":"2023-06-10T18:53:29.061854+0000","last_became_peered":"2023-06-10T18:53:29.061854+0000","last_unstale":"2023-06-10T18:53:30.456697+0000","last_undegraded":"2023-06-10T18:53:30.456697+0000","last_fullsized":"2023-06-10T18:53:30.456697+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:37:54.053528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833348+0000","last_change":"2023-06-10T18:53:29.474991+0000","last_active":"2023-06-10T18:53:30.833348+0000","last_peered":"2023-06-10T18:53:30.833348+0000","last_clean":"2023-06-10T18:53:30.833348+0000","last_became_active":"2023-06-10T18:53:29.474735+0000","last_became_peered":"2023-06-10T18:53:29.474735+0000","last_unstale":"2023-06-10T18:53:30.833348+0000","last_undegraded":"2023-06-10T18:53:30.833348+0000","last_fullsized":"2023-06-10T18:53:30.833348+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:55:34.768404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832791+0000","last_change":"2023-06-10T18:53:29.061307+0000","last_active":"2023-06-10T18:53:30.832791+0000","last_peered":"2023-06-10T18:53:30.832791+0000","last_clean":"2023-06-10T18:53:30.832791+0000","last_became_active":"2023-06-10T18:53:29.061005+0000","last_became_peered":"2023-06-10T18:53:29.061005+0000","last_unstale":"2023-06-10T18:53:30.832791+0000","last_undegraded":"2023-06-10T18:53:30.832791+0000","last_fullsized":"2023-06-10T18:53:30.832791+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:29:21.496833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832497+0000","last_change":"2023-06-10T18:53:29.475041+0000","last_active":"2023-06-10T18:53:30.832497+0000","last_peered":"2023-06-10T18:53:30.832497+0000","last_clean":"2023-06-10T18:53:30.832497+0000","last_became_active":"2023-06-10T18:53:29.474805+0000","last_became_peered":"2023-06-10T18:53:29.474805+0000","last_unstale":"2023-06-10T18:53:30.832497+0000","last_undegraded":"2023-06-10T18:53:30.832497+0000","last_fullsized":"2023-06-10T18:53:30.832497+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:02.031339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":19,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832488+0000","last_change":"2023-06-10T18:53:29.064193+0000","last_active":"2023-06-10T18:53:30.832488+0000","last_peered":"2023-06-10T18:53:30.832488+0000","last_clean":"2023-06-10T18:53:30.832488+0000","last_became_active":"2023-06-10T18:53:29.063959+0000","last_became_peered":"2023-06-10T18:53:29.063959+0000","last_unstale":"2023-06-10T18:53:30.832488+0000","last_undegraded":"2023-06-10T18:53:30.832488+0000","last_fullsized":"2023-06-10T18:53:30.832488+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:52:04.776737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455980+0000","last_change":"2023-06-10T18:53:29.470619+0000","last_active":"2023-06-10T18:53:30.455980+0000","last_peered":"2023-06-10T18:53:30.455980+0000","last_clean":"2023-06-10T18:53:30.455980+0000","last_became_active":"2023-06-10T18:53:29.470327+0000","last_became_peered":"2023-06-10T18:53:29.470327+0000","last_unstale":"2023-06-10T18:53:30.455980+0000","last_undegraded":"2023-06-10T18:53:30.455980+0000","last_fullsized":"2023-06-10T18:53:30.455980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:41:20.047008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831729+0000","last_change":"2023-06-10T18:53:29.064336+0000","last_active":"2023-06-10T18:53:30.831729+0000","last_peered":"2023-06-10T18:53:30.831729+0000","last_clean":"2023-06-10T18:53:30.831729+0000","last_became_active":"2023-06-10T18:53:29.064054+0000","last_became_peered":"2023-06-10T18:53:29.064054+0000","last_unstale":"2023-06-10T18:53:30.831729+0000","last_undegraded":"2023-06-10T18:53:30.831729+0000","last_fullsized":"2023-06-10T18:53:30.831729+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:57:01.876549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456001+0000","last_change":"2023-06-10T18:53:29.482222+0000","last_active":"2023-06-10T18:53:30.456001+0000","last_peered":"2023-06-10T18:53:30.456001+0000","last_clean":"2023-06-10T18:53:30.456001+0000","last_became_active":"2023-06-10T18:53:29.481963+0000","last_became_peered":"2023-06-10T18:53:29.481963+0000","last_unstale":"2023-06-10T18:53:30.456001+0000","last_undegraded":"2023-06-10T18:53:30.456001+0000","last_fullsized":"2023-06-10T18:53:30.456001+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:55.500417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832263+0000","last_change":"2023-06-10T18:53:29.068307+0000","last_active":"2023-06-10T18:53:30.832263+0000","last_peered":"2023-06-10T18:53:30.832263+0000","last_clean":"2023-06-10T18:53:30.832263+0000","last_became_active":"2023-06-10T18:53:29.068036+0000","last_became_peered":"2023-06-10T18:53:29.068036+0000","last_unstale":"2023-06-10T18:53:30.832263+0000","last_undegraded":"2023-06-10T18:53:30.832263+0000","last_fullsized":"2023-06-10T18:53:30.832263+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:38:48.451129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831978+0000","last_change":"2023-06-10T18:53:29.468918+0000","last_active":"2023-06-10T18:53:30.831978+0000","last_peered":"2023-06-10T18:53:30.831978+0000","last_clean":"2023-06-10T18:53:30.831978+0000","last_became_active":"2023-06-10T18:53:29.468517+0000","last_became_peered":"2023-06-10T18:53:29.468517+0000","last_unstale":"2023-06-10T18:53:30.831978+0000","last_undegraded":"2023-06-10T18:53:30.831978+0000","last_fullsized":"2023-06-10T18:53:30.831978+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:48:31.971495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832000+0000","last_change":"2023-06-10T18:53:29.063717+0000","last_active":"2023-06-10T18:53:30.832000+0000","last_peered":"2023-06-10T18:53:30.832000+0000","last_clean":"2023-06-10T18:53:30.832000+0000","last_became_active":"2023-06-10T18:53:29.063409+0000","last_became_peered":"2023-06-10T18:53:29.063409+0000","last_unstale":"2023-06-10T18:53:30.832000+0000","last_undegraded":"2023-06-10T18:53:30.832000+0000","last_fullsized":"2023-06-10T18:53:30.832000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:36.743372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831349+0000","last_change":"2023-06-10T18:53:29.469493+0000","last_active":"2023-06-10T18:53:30.831349+0000","last_peered":"2023-06-10T18:53:30.831349+0000","last_clean":"2023-06-10T18:53:30.831349+0000","last_became_active":"2023-06-10T18:53:29.469243+0000","last_became_peered":"2023-06-10T18:53:29.469243+0000","last_unstale":"2023-06-10T18:53:30.831349+0000","last_undegraded":"2023-06-10T18:53:30.831349+0000","last_fullsized":"2023-06-10T18:53:30.831349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:37:51.917441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":21,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455958+0000","last_change":"2023-06-10T18:53:29.066592+0000","last_active":"2023-06-10T18:53:30.455958+0000","last_peered":"2023-06-10T18:53:30.455958+0000","last_clean":"2023-06-10T18:53:30.455958+0000","last_became_active":"2023-06-10T18:53:29.066321+0000","last_became_peered":"2023-06-10T18:53:29.066321+0000","last_unstale":"2023-06-10T18:53:30.455958+0000","last_undegraded":"2023-06-10T18:53:30.455958+0000","last_fullsized":"2023-06-10T18:53:30.455958+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:37:05.822365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832141+0000","last_change":"2023-06-10T18:53:29.468591+0000","last_active":"2023-06-10T18:53:30.832141+0000","last_peered":"2023-06-10T18:53:30.832141+0000","last_clean":"2023-06-10T18:53:30.832141+0000","last_became_active":"2023-06-10T18:53:29.467797+0000","last_became_peered":"2023-06-10T18:53:29.467797+0000","last_unstale":"2023-06-10T18:53:30.832141+0000","last_undegraded":"2023-06-10T18:53:30.832141+0000","last_fullsized":"2023-06-10T18:53:30.832141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:47:19.099850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457880+0000","last_change":"2023-06-10T18:53:29.061254+0000","last_active":"2023-06-10T18:53:30.457880+0000","last_peered":"2023-06-10T18:53:30.457880+0000","last_clean":"2023-06-10T18:53:30.457880+0000","last_became_active":"2023-06-10T18:53:29.060946+0000","last_became_peered":"2023-06-10T18:53:29.060946+0000","last_unstale":"2023-06-10T18:53:30.457880+0000","last_undegraded":"2023-06-10T18:53:30.457880+0000","last_fullsized":"2023-06-10T18:53:30.457880+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:49.851976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456258+0000","last_change":"2023-06-10T18:53:29.473162+0000","last_active":"2023-06-10T18:53:30.456258+0000","last_peered":"2023-06-10T18:53:30.456258+0000","last_clean":"2023-06-10T18:53:30.456258+0000","last_became_active":"2023-06-10T18:53:29.466711+0000","last_became_peered":"2023-06-10T18:53:29.466711+0000","last_unstale":"2023-06-10T18:53:30.456258+0000","last_undegraded":"2023-06-10T18:53:30.456258+0000","last_fullsized":"2023-06-10T18:53:30.456258+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:55.739710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832277+0000","last_change":"2023-06-10T18:53:29.071073+0000","last_active":"2023-06-10T18:53:30.832277+0000","last_peered":"2023-06-10T18:53:30.832277+0000","last_clean":"2023-06-10T18:53:30.832277+0000","last_became_active":"2023-06-10T18:53:29.070784+0000","last_became_peered":"2023-06-10T18:53:29.070784+0000","last_unstale":"2023-06-10T18:53:30.832277+0000","last_undegraded":"2023-06-10T18:53:30.832277+0000","last_fullsized":"2023-06-10T18:53:30.832277+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:58.249220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456322+0000","last_change":"2023-06-10T18:53:29.476137+0000","last_active":"2023-06-10T18:53:30.456322+0000","last_peered":"2023-06-10T18:53:30.456322+0000","last_clean":"2023-06-10T18:53:30.456322+0000","last_became_active":"2023-06-10T18:53:29.473736+0000","last_became_peered":"2023-06-10T18:53:29.473736+0000","last_unstale":"2023-06-10T18:53:30.456322+0000","last_undegraded":"2023-06-10T18:53:30.456322+0000","last_fullsized":"2023-06-10T18:53:30.456322+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:05.039746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832264+0000","last_change":"2023-06-10T18:53:29.058881+0000","last_active":"2023-06-10T18:53:30.832264+0000","last_peered":"2023-06-10T18:53:30.832264+0000","last_clean":"2023-06-10T18:53:30.832264+0000","last_became_active":"2023-06-10T18:53:29.058579+0000","last_became_peered":"2023-06-10T18:53:29.058579+0000","last_unstale":"2023-06-10T18:53:30.832264+0000","last_undegraded":"2023-06-10T18:53:30.832264+0000","last_fullsized":"2023-06-10T18:53:30.832264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:11:46.849465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833314+0000","last_change":"2023-06-10T18:53:29.476133+0000","last_active":"2023-06-10T18:53:30.833314+0000","last_peered":"2023-06-10T18:53:30.833314+0000","last_clean":"2023-06-10T18:53:30.833314+0000","last_became_active":"2023-06-10T18:53:29.475776+0000","last_became_peered":"2023-06-10T18:53:29.475776+0000","last_unstale":"2023-06-10T18:53:30.833314+0000","last_undegraded":"2023-06-10T18:53:30.833314+0000","last_fullsized":"2023-06-10T18:53:30.833314+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:52:04.177125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833423+0000","last_change":"2023-06-10T18:53:29.064217+0000","last_active":"2023-06-10T18:53:30.833423+0000","last_peered":"2023-06-10T18:53:30.833423+0000","last_clean":"2023-06-10T18:53:30.833423+0000","last_became_active":"2023-06-10T18:53:29.063905+0000","last_became_peered":"2023-06-10T18:53:29.063905+0000","last_unstale":"2023-06-10T18:53:30.833423+0000","last_undegraded":"2023-06-10T18:53:30.833423+0000","last_fullsized":"2023-06-10T18:53:30.833423+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:02:15.954425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832142+0000","last_change":"2023-06-10T18:53:29.467964+0000","last_active":"2023-06-10T18:53:30.832142+0000","last_peered":"2023-06-10T18:53:30.832142+0000","last_clean":"2023-06-10T18:53:30.832142+0000","last_became_active":"2023-06-10T18:53:29.467674+0000","last_became_peered":"2023-06-10T18:53:29.467674+0000","last_unstale":"2023-06-10T18:53:30.832142+0000","last_undegraded":"2023-06-10T18:53:30.832142+0000","last_fullsized":"2023-06-10T18:53:30.832142+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:09.900799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831963+0000","last_change":"2023-06-10T18:53:29.059874+0000","last_active":"2023-06-10T18:53:30.831963+0000","last_peered":"2023-06-10T18:53:30.831963+0000","last_clean":"2023-06-10T18:53:30.831963+0000","last_became_active":"2023-06-10T18:53:29.059599+0000","last_became_peered":"2023-06-10T18:53:29.059599+0000","last_unstale":"2023-06-10T18:53:30.831963+0000","last_undegraded":"2023-06-10T18:53:30.831963+0000","last_fullsized":"2023-06-10T18:53:30.831963+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:55:22.842032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832185+0000","last_change":"2023-06-10T18:53:29.471768+0000","last_active":"2023-06-10T18:53:30.832185+0000","last_peered":"2023-06-10T18:53:30.832185+0000","last_clean":"2023-06-10T18:53:30.832185+0000","last_became_active":"2023-06-10T18:53:29.471456+0000","last_became_peered":"2023-06-10T18:53:29.471456+0000","last_unstale":"2023-06-10T18:53:30.832185+0000","last_undegraded":"2023-06-10T18:53:30.832185+0000","last_fullsized":"2023-06-10T18:53:30.832185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:30.502910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832871+0000","last_change":"2023-06-10T18:53:29.069172+0000","last_active":"2023-06-10T18:53:30.832871+0000","last_peered":"2023-06-10T18:53:30.832871+0000","last_clean":"2023-06-10T18:53:30.832871+0000","last_became_active":"2023-06-10T18:53:29.068859+0000","last_became_peered":"2023-06-10T18:53:29.068859+0000","last_unstale":"2023-06-10T18:53:30.832871+0000","last_undegraded":"2023-06-10T18:53:30.832871+0000","last_fullsized":"2023-06-10T18:53:30.832871+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:38:49.635088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831308+0000","last_change":"2023-06-10T18:53:29.468007+0000","last_active":"2023-06-10T18:53:30.831308+0000","last_peered":"2023-06-10T18:53:30.831308+0000","last_clean":"2023-06-10T18:53:30.831308+0000","last_became_active":"2023-06-10T18:53:29.467735+0000","last_became_peered":"2023-06-10T18:53:29.467735+0000","last_unstale":"2023-06-10T18:53:30.831308+0000","last_undegraded":"2023-06-10T18:53:30.831308+0000","last_fullsized":"2023-06-10T18:53:30.831308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:27:00.731981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456499+0000","last_change":"2023-06-10T18:53:29.063569+0000","last_active":"2023-06-10T18:53:30.456499+0000","last_peered":"2023-06-10T18:53:30.456499+0000","last_clean":"2023-06-10T18:53:30.456499+0000","last_became_active":"2023-06-10T18:53:29.063267+0000","last_became_peered":"2023-06-10T18:53:29.063267+0000","last_unstale":"2023-06-10T18:53:30.456499+0000","last_undegraded":"2023-06-10T18:53:30.456499+0000","last_fullsized":"2023-06-10T18:53:30.456499+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:04:51.152861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832861+0000","last_change":"2023-06-10T18:53:29.476059+0000","last_active":"2023-06-10T18:53:30.832861+0000","last_peered":"2023-06-10T18:53:30.832861+0000","last_clean":"2023-06-10T18:53:30.832861+0000","last_became_active":"2023-06-10T18:53:29.475787+0000","last_became_peered":"2023-06-10T18:53:29.475787+0000","last_unstale":"2023-06-10T18:53:30.832861+0000","last_undegraded":"2023-06-10T18:53:30.832861+0000","last_fullsized":"2023-06-10T18:53:30.832861+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:56:19.966340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831427+0000","last_change":"2023-06-10T18:53:29.069431+0000","last_active":"2023-06-10T18:53:30.831427+0000","last_peered":"2023-06-10T18:53:30.831427+0000","last_clean":"2023-06-10T18:53:30.831427+0000","last_became_active":"2023-06-10T18:53:29.069170+0000","last_became_peered":"2023-06-10T18:53:29.069170+0000","last_unstale":"2023-06-10T18:53:30.831427+0000","last_undegraded":"2023-06-10T18:53:30.831427+0000","last_fullsized":"2023-06-10T18:53:30.831427+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:12:49.704203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832872+0000","last_change":"2023-06-10T18:53:29.478789+0000","last_active":"2023-06-10T18:53:30.832872+0000","last_peered":"2023-06-10T18:53:30.832872+0000","last_clean":"2023-06-10T18:53:30.832872+0000","last_became_active":"2023-06-10T18:53:29.478528+0000","last_became_peered":"2023-06-10T18:53:29.478528+0000","last_unstale":"2023-06-10T18:53:30.832872+0000","last_undegraded":"2023-06-10T18:53:30.832872+0000","last_fullsized":"2023-06-10T18:53:30.832872+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:23.076875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831826+0000","last_change":"2023-06-10T18:53:29.070449+0000","last_active":"2023-06-10T18:53:30.831826+0000","last_peered":"2023-06-10T18:53:30.831826+0000","last_clean":"2023-06-10T18:53:30.831826+0000","last_became_active":"2023-06-10T18:53:29.070050+0000","last_became_peered":"2023-06-10T18:53:29.070050+0000","last_unstale":"2023-06-10T18:53:30.831826+0000","last_undegraded":"2023-06-10T18:53:30.831826+0000","last_fullsized":"2023-06-10T18:53:30.831826+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:22:55.184809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456372+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:53:30.456372+0000","last_peered":"2023-06-10T18:53:30.456372+0000","last_clean":"2023-06-10T18:53:30.456372+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:53:30.456372+0000","last_undegraded":"2023-06-10T18:53:30.456372+0000","last_fullsized":"2023-06-10T18:53:30.456372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:39:11.226586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833595+0000","last_change":"2023-06-10T18:53:29.467857+0000","last_active":"2023-06-10T18:53:30.833595+0000","last_peered":"2023-06-10T18:53:30.833595+0000","last_clean":"2023-06-10T18:53:30.833595+0000","last_became_active":"2023-06-10T18:53:29.467606+0000","last_became_peered":"2023-06-10T18:53:29.467606+0000","last_unstale":"2023-06-10T18:53:30.833595+0000","last_undegraded":"2023-06-10T18:53:30.833595+0000","last_fullsized":"2023-06-10T18:53:30.833595+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:43.418029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833305+0000","last_change":"2023-06-10T18:53:29.071615+0000","last_active":"2023-06-10T18:53:30.833305+0000","last_peered":"2023-06-10T18:53:30.833305+0000","last_clean":"2023-06-10T18:53:30.833305+0000","last_became_active":"2023-06-10T18:53:29.071307+0000","last_became_peered":"2023-06-10T18:53:29.071307+0000","last_unstale":"2023-06-10T18:53:30.833305+0000","last_undegraded":"2023-06-10T18:53:30.833305+0000","last_fullsized":"2023-06-10T18:53:30.833305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:09:00.074642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":171,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457601+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:53:30.457601+0000","last_peered":"2023-06-10T18:53:30.457601+0000","last_clean":"2023-06-10T18:53:30.457601+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:53:30.457601+0000","last_undegraded":"2023-06-10T18:53:30.457601+0000","last_fullsized":"2023-06-10T18:53:30.457601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:47.016165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457577+0000","last_change":"2023-06-10T18:53:29.473860+0000","last_active":"2023-06-10T18:53:30.457577+0000","last_peered":"2023-06-10T18:53:30.457577+0000","last_clean":"2023-06-10T18:53:30.457577+0000","last_became_active":"2023-06-10T18:53:29.473582+0000","last_became_peered":"2023-06-10T18:53:29.473582+0000","last_unstale":"2023-06-10T18:53:30.457577+0000","last_undegraded":"2023-06-10T18:53:30.457577+0000","last_fullsized":"2023-06-10T18:53:30.457577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:56:15.947373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833489+0000","last_change":"2023-06-10T18:53:29.063130+0000","last_active":"2023-06-10T18:53:30.833489+0000","last_peered":"2023-06-10T18:53:30.833489+0000","last_clean":"2023-06-10T18:53:30.833489+0000","last_became_active":"2023-06-10T18:53:29.062738+0000","last_became_peered":"2023-06-10T18:53:29.062738+0000","last_unstale":"2023-06-10T18:53:30.833489+0000","last_undegraded":"2023-06-10T18:53:30.833489+0000","last_fullsized":"2023-06-10T18:53:30.833489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:40:12.356866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831301+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:53:30.831301+0000","last_peered":"2023-06-10T18:53:30.831301+0000","last_clean":"2023-06-10T18:53:30.831301+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:53:30.831301+0000","last_undegraded":"2023-06-10T18:53:30.831301+0000","last_fullsized":"2023-06-10T18:53:30.831301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:30.353337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832153+0000","last_change":"2023-06-10T18:53:29.465099+0000","last_active":"2023-06-10T18:53:30.832153+0000","last_peered":"2023-06-10T18:53:30.832153+0000","last_clean":"2023-06-10T18:53:30.832153+0000","last_became_active":"2023-06-10T18:53:29.464858+0000","last_became_peered":"2023-06-10T18:53:29.464858+0000","last_unstale":"2023-06-10T18:53:30.832153+0000","last_undegraded":"2023-06-10T18:53:30.832153+0000","last_fullsized":"2023-06-10T18:53:30.832153+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:38:29.543445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832537+0000","last_change":"2023-06-10T18:53:29.065885+0000","last_active":"2023-06-10T18:53:30.832537+0000","last_peered":"2023-06-10T18:53:30.832537+0000","last_clean":"2023-06-10T18:53:30.832537+0000","last_became_active":"2023-06-10T18:53:29.065614+0000","last_became_peered":"2023-06-10T18:53:29.065614+0000","last_unstale":"2023-06-10T18:53:30.832537+0000","last_undegraded":"2023-06-10T18:53:30.832537+0000","last_fullsized":"2023-06-10T18:53:30.832537+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:35.966791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":171,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457475+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:53:30.457475+0000","last_peered":"2023-06-10T18:53:30.457475+0000","last_clean":"2023-06-10T18:53:30.457475+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:53:30.457475+0000","last_undegraded":"2023-06-10T18:53:30.457475+0000","last_fullsized":"2023-06-10T18:53:30.457475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:22:52.297361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":315,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831834+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:53:30.831834+0000","last_peered":"2023-06-10T18:53:30.831834+0000","last_clean":"2023-06-10T18:53:30.831834+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:53:30.831834+0000","last_undegraded":"2023-06-10T18:53:30.831834+0000","last_fullsized":"2023-06-10T18:53:30.831834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:39.834975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832725+0000","last_change":"2023-06-10T18:53:29.468475+0000","last_active":"2023-06-10T18:53:30.832725+0000","last_peered":"2023-06-10T18:53:30.832725+0000","last_clean":"2023-06-10T18:53:30.832725+0000","last_became_active":"2023-06-10T18:53:29.467997+0000","last_became_peered":"2023-06-10T18:53:29.467997+0000","last_unstale":"2023-06-10T18:53:30.832725+0000","last_undegraded":"2023-06-10T18:53:30.832725+0000","last_fullsized":"2023-06-10T18:53:30.832725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:32.714194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831926+0000","last_change":"2023-06-10T18:53:29.062404+0000","last_active":"2023-06-10T18:53:30.831926+0000","last_peered":"2023-06-10T18:53:30.831926+0000","last_clean":"2023-06-10T18:53:30.831926+0000","last_became_active":"2023-06-10T18:53:29.062124+0000","last_became_peered":"2023-06-10T18:53:29.062124+0000","last_unstale":"2023-06-10T18:53:30.831926+0000","last_undegraded":"2023-06-10T18:53:30.831926+0000","last_fullsized":"2023-06-10T18:53:30.831926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:24:31.583175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831965+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:53:30.831965+0000","last_peered":"2023-06-10T18:53:30.831965+0000","last_clean":"2023-06-10T18:53:30.831965+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:53:30.831965+0000","last_undegraded":"2023-06-10T18:53:30.831965+0000","last_fullsized":"2023-06-10T18:53:30.831965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:38:32.459635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831327+0000","last_change":"2023-06-10T18:53:29.464975+0000","last_active":"2023-06-10T18:53:30.831327+0000","last_peered":"2023-06-10T18:53:30.831327+0000","last_clean":"2023-06-10T18:53:30.831327+0000","last_became_active":"2023-06-10T18:53:29.464737+0000","last_became_peered":"2023-06-10T18:53:29.464737+0000","last_unstale":"2023-06-10T18:53:30.831327+0000","last_undegraded":"2023-06-10T18:53:30.831327+0000","last_fullsized":"2023-06-10T18:53:30.831327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:28:46.688002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457493+0000","last_change":"2023-06-10T18:53:29.065636+0000","last_active":"2023-06-10T18:53:30.457493+0000","last_peered":"2023-06-10T18:53:30.457493+0000","last_clean":"2023-06-10T18:53:30.457493+0000","last_became_active":"2023-06-10T18:53:29.065389+0000","last_became_peered":"2023-06-10T18:53:29.065389+0000","last_unstale":"2023-06-10T18:53:30.457493+0000","last_undegraded":"2023-06-10T18:53:30.457493+0000","last_fullsized":"2023-06-10T18:53:30.457493+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:09.107609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832879+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:53:30.832879+0000","last_peered":"2023-06-10T18:53:30.832879+0000","last_clean":"2023-06-10T18:53:30.832879+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:53:30.832879+0000","last_undegraded":"2023-06-10T18:53:30.832879+0000","last_fullsized":"2023-06-10T18:53:30.832879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:57:16.741631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831502+0000","last_change":"2023-06-10T18:53:29.470515+0000","last_active":"2023-06-10T18:53:30.831502+0000","last_peered":"2023-06-10T18:53:30.831502+0000","last_clean":"2023-06-10T18:53:30.831502+0000","last_became_active":"2023-06-10T18:53:29.470218+0000","last_became_peered":"2023-06-10T18:53:29.470218+0000","last_unstale":"2023-06-10T18:53:30.831502+0000","last_undegraded":"2023-06-10T18:53:30.831502+0000","last_fullsized":"2023-06-10T18:53:30.831502+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:50:52.860486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457530+0000","last_change":"2023-06-10T18:53:29.064050+0000","last_active":"2023-06-10T18:53:30.457530+0000","last_peered":"2023-06-10T18:53:30.457530+0000","last_clean":"2023-06-10T18:53:30.457530+0000","last_became_active":"2023-06-10T18:53:29.063779+0000","last_became_peered":"2023-06-10T18:53:29.063779+0000","last_unstale":"2023-06-10T18:53:30.457530+0000","last_undegraded":"2023-06-10T18:53:30.457530+0000","last_fullsized":"2023-06-10T18:53:30.457530+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:24:27.751928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456494+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:53:30.456494+0000","last_peered":"2023-06-10T18:53:30.456494+0000","last_clean":"2023-06-10T18:53:30.456494+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:53:30.456494+0000","last_undegraded":"2023-06-10T18:53:30.456494+0000","last_fullsized":"2023-06-10T18:53:30.456494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:15:49.708843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832137+0000","last_change":"2023-06-10T18:53:29.467403+0000","last_active":"2023-06-10T18:53:30.832137+0000","last_peered":"2023-06-10T18:53:30.832137+0000","last_clean":"2023-06-10T18:53:30.832137+0000","last_became_active":"2023-06-10T18:53:29.467127+0000","last_became_peered":"2023-06-10T18:53:29.467127+0000","last_unstale":"2023-06-10T18:53:30.832137+0000","last_undegraded":"2023-06-10T18:53:30.832137+0000","last_fullsized":"2023-06-10T18:53:30.832137+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:26:45.304396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831869+0000","last_change":"2023-06-10T18:53:29.068241+0000","last_active":"2023-06-10T18:53:30.831869+0000","last_peered":"2023-06-10T18:53:30.831869+0000","last_clean":"2023-06-10T18:53:30.831869+0000","last_became_active":"2023-06-10T18:53:29.068042+0000","last_became_peered":"2023-06-10T18:53:29.068042+0000","last_unstale":"2023-06-10T18:53:30.831869+0000","last_undegraded":"2023-06-10T18:53:30.831869+0000","last_fullsized":"2023-06-10T18:53:30.831869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:35.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833691+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:53:30.833691+0000","last_peered":"2023-06-10T18:53:30.833691+0000","last_clean":"2023-06-10T18:53:30.833691+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:53:30.833691+0000","last_undegraded":"2023-06-10T18:53:30.833691+0000","last_fullsized":"2023-06-10T18:53:30.833691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:26.238858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574967,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28720,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343120,"statfs":{"total":96636764160,"available":96607354880,"internally_reserved":0,"allocated":1810432,"data_stored":1171160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55700000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.65200000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58099999999999996,"15min":0.53800000000000003},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.247},"max":{"1min":0.66300000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.499}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.55400000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.435},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.34799999999999998},"max":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.44800000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.505},"min":{"1min":0.34499999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70599999999999996,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.27300000000000002},"max":{"1min":0.70299999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.52300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.748,"5min":0.748,"15min":0.78300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66400000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.441,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46899999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.311},"max":{"1min":0.71399999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.32600000000000001},"max":{"1min":0.73099999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.38700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67500000000000004,"5min":0.71799999999999997,"15min":0.81100000000000005},"last":0.5},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.78700000000000003},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.309},"max":{"1min":0.59099999999999997,"5min":0.68899999999999995,"15min":0.80600000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.52700000000000002,"15min":0.74099999999999999},"min":{"1min":0.377,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.67200000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.61599999999999999,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.76300000000000001,"15min":1.252},"last":0.59099999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82899999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68200000000000005,"5min":0.73999999999999999,"15min":0.77500000000000002},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48499999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67900000000000005,"5min":0.69599999999999995,"15min":0.82999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.89600000000000002},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47999999999999998,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.86699999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52100000000000002,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.79100000000000004},"last":0.38400000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.39000000000000001,"5min":0.35299999999999998,"15min":0.32100000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.41499999999999998,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.81399999999999995},"last":0.53100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28140,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343700,"statfs":{"total":96636764160,"available":96607948800,"internally_reserved":0,"allocated":1216512,"data_stored":586634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33100000000000002,"15min":0.33500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":0.51500000000000001,"15min":0.753},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.49299999999999999,"5min":0.53900000000000003,"15min":0.59599999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35299999999999998,"15min":0.34599999999999997},"min":{"1min":0.23799999999999999,"5min":0.215,"15min":0.20000000000000001},"max":{"1min":0.52200000000000002,"5min":0.57899999999999996,"15min":0.64500000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.373},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40799999999999997,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.41899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.873,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.84599999999999997,"5min":0.85999999999999999,"15min":0.93000000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":1.008},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.86299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.433}]},{"osd":7,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.95099999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60199999999999998,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.435,"15min":0.434},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.50700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574967,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28724,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343116,"statfs":{"total":96636764160,"available":96607350784,"internally_reserved":0,"allocated":1814528,"data_stored":1177584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33800000000000002,"15min":0.34399999999999997},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.51600000000000001,"5min":0.53700000000000003,"15min":0.72699999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.35399999999999998,"15min":0.35699999999999998},"min":{"1min":0.22900000000000001,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.5,"5min":0.58499999999999996,"15min":0.88900000000000001},"last":0.248}]},{"osd":2,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35999999999999999,"15min":0.375},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":1.405},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38300000000000001,"15min":0.38900000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":0.55100000000000005,"5min":0.65500000000000003,"15min":0.95499999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.61799999999999999,"15min":0.93100000000000005},"last":0.312},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.41299999999999998,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.27100000000000002,"15min":0.25900000000000001},"max":{"1min":0.63800000000000001,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":1.0489999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.59799999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.98199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44800000000000001,"5min":0.38800000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.363},"last":0.502}]},{"osd":7,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60399999999999998},"min":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.38900000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":1.0149999999999999},"last":0.44}]}]},{"osd":2,"up_from":13,"seq":55834574968,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.45300000000000001,"15min":0.38600000000000001},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":0.37},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.46600000000000003,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.23799999999999999},"max":{"1min":5.6790000000000003,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.45500000000000002,"15min":0.38900000000000001},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.23899999999999999},"max":{"1min":7.4569999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.69799999999999995,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":5.742,"5min":5.742,"15min":5.742},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.69799999999999995,"15min":0.60999999999999999},"min":{"1min":0.39800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.68500000000000005,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.593,"5min":6.593,"15min":6.593},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.71099999999999997,"15min":0.65000000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38400000000000001},"max":{"1min":7.0880000000000001,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.496}]},{"osd":6,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.73299999999999998,"15min":0.63700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":8.7850000000000001,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.433},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.66500000000000004,"15min":0.625},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.6180000000000003,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.70999999999999996,"15min":0.66200000000000003},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.40699999999999997},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.745,"15min":0.65400000000000003},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":8.7149999999999999,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.52600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574967,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1208,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1236992,"data_stored":589239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.36499999999999999,"15min":0.39200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36299999999999999,"15min":0.38200000000000001},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39400000000000002,"15min":0.41899999999999998},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":1.3},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.41499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.46200000000000002},"min":{"1min":0.33000000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.599},"last":0.375}]},{"osd":4,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.925},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59399999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.439}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.70099999999999996},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":3.802},"last":0.56599999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60999999999999999,"15min":0.69899999999999995},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.621,"15min":0.624},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.46600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.46700000000000003,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.8939999999999999},"last":0.44800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574967,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1208,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1236992,"data_stored":588322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":0.68799999999999994,"5min":0.82599999999999996,"15min":1.119},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.44500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64900000000000002,"5min":0.74099999999999999,"15min":1.1639999999999999},"last":0.51600000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54300000000000004,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.55500000000000005,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.65400000000000003,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.52300000000000002},"min":{"1min":0.372,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.54900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68500000000000005,"5min":0.80100000000000005,"15min":1.0229999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70699999999999996,"5min":0.748,"15min":0.79800000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.042},"last":0.58799999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.72199999999999998,"5min":0.78700000000000003,"15min":1.069},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.34300000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.84099999999999997,"15min":0.88300000000000001},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.38700000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.90400000000000003,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.747,"5min":0.82999999999999996,"15min":1.3169999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.85899999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.46100000000000002,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.63200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574968,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.42999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.51600000000000001},"min":{"1min":0.42299999999999999,"5min":0.34399999999999997,"15min":0.32300000000000001},"max":{"1min":0.70199999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70399999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53700000000000003,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.64800000000000002,"5min":0.70599999999999996,"15min":0.73099999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57199999999999995,"15min":0.52900000000000003},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.66100000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":0.71199999999999997,"5min":0.79000000000000004,"15min":0.85099999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55600000000000005,"15min":0.53300000000000003},"min":{"1min":0.40200000000000002,"5min":0.371,"15min":0.34999999999999998},"max":{"1min":0.69199999999999995,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56499999999999995,"15min":0.53400000000000003},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.55800000000000005},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.745}]},{"osd":4,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.47899999999999998},"min":{"1min":0.35099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68300000000000005,"5min":0.76400000000000001,"15min":0.83999999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51600000000000001,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.66600000000000004,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.40300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.79200000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.879},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.41799999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.55600000000000005},"min":{"1min":0.44500000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73099999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.64100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:53:33.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:53:33.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.693+0000 7f9c9fe08700 1 Processor -- start 2023-06-10T18:53:33.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.694+0000 7f9c9fe08700 1 -- start start 2023-06-10T18:53:33.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9c98158c50 0x7f9c9806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c9806b180 0x7f9c980cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9c9806ca40 0x7f9c9806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c980ccfb0 con 0x7f9c98158c50 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c980ce0b0 con 0x7f9c9806ca40 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9fe08700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9806d3a0 con 0x7f9c9806b180 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9c98158c50 0x7f9c9806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9c98158c50 0x7f9c9806aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34506/0 (socket says 172.21.15.104:34506) 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 -- 172.21.15.104:0/3882486533 learned_addr learned my addr 172.21.15.104:0/3882486533 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9d3a3700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c9806b180 0x7f9c980cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9e3a5700 1 --2- 172.21.15.104:0/3882486533 >> v2:172.21.15.114:3300/0 conn(0x7f9c9806ca40 0x7f9c9806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 -- 172.21.15.104:0/3882486533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c9806b180 msgr2=0x7f9c980cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c9806b180 0x7f9c980cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 -- 172.21.15.104:0/3882486533 >> v2:172.21.15.114:3300/0 conn(0x7f9c9806ca40 msgr2=0x7f9c9806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3882486533 >> v2:172.21.15.114:3300/0 conn(0x7f9c9806ca40 0x7f9c9806ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 -- 172.21.15.104:0/3882486533 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980cccd0 con 0x7f9c98158c50 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c98158c50 0x7f9c9806aaf0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9c9806b5f0 tx=0x7f9c88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e480622abf5bdc4e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9cba2700 1 -- 172.21.15.104:0/3882486533 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800b870 con 0x7f9c98158c50 2023-06-10T18:53:33.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9cba2700 1 -- 172.21.15.104:0/3882486533 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8800b9d0 con 0x7f9c98158c50 2023-06-10T18:53:33.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.695+0000 7f9c9cba2700 1 -- 172.21.15.104:0/3882486533 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800bd60 con 0x7f9c98158c50 2023-06-10T18:53:33.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3882486533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c98158c50 msgr2=0x7f9c9806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c98158c50 0x7f9c9806aaf0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9c9806b5f0 tx=0x7f9c88009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3882486533 shutdown_connections 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c98158c50 0x7f9c9806aaf0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3882486533 >> v2:172.21.15.114:3300/0 conn(0x7f9c9806ca40 0x7f9c9806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3882486533 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c9806b180 0x7f9c980cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3882486533 >> 172.21.15.104:0/3882486533 conn(0x7f9c980b1610 msgr2=0x7f9c980ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3882486533 shutdown_connections 2023-06-10T18:53:33.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.696+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3882486533 wait complete. 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.697+0000 7f9c9fe08700 1 Processor -- start 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.697+0000 7f9c9fe08700 1 -- start start 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c9806b180 0x7f9c98156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 0x7f9c981f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980c28c0 con 0x7f9c9806ca40 2023-06-10T18:53:33.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980c2b10 con 0x7f9c9806b180 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9fe08700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980c2df0 con 0x7f9c98158c50 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34514/0 (socket says 172.21.15.104:34514) 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 -- 172.21.15.104:0/3046262546 learned_addr learned my addr 172.21.15.104:0/3046262546 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c9806b180 0x7f9c98156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9e3a5700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 0x7f9c981f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 -- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 msgr2=0x7f9c981f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 0x7f9c981f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 -- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c9806b180 msgr2=0x7f9c98156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c9806b180 0x7f9c98156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 -- 172.21.15.104:0/3046262546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88009550 con 0x7f9c9806ca40 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9e3a5700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 0x7f9c981f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.698+0000 7f9c9d3a3700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c7f0 tx=0x7f9c9400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.699+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94011800 con 0x7f9c9806ca40 2023-06-10T18:53:33.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.699+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94011960 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.699+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c980c30d0 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.699+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94011cf0 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.699+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9814ecb0 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.700+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c980c35f0 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.700+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9c94018360 con 0x7f9c9806ca40 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.700+0000 7f9c8e7fc700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 0x7f9c7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:33.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.700+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9c94089510 con 0x7f9c9806ca40 2023-06-10T18:53:33.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.700+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 0x7f9c7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:33.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.701+0000 7f9c9dba4700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 0x7f9c7c06d190 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c00 tx=0x7f9c88008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:33.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.705+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9c980c35f0 con 0x7f9c9806ca40 2023-06-10T18:53:33.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.886+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c9806a2f0 con 0x7f9c7c06ace0 2023-06-10T18:53:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.896+0000 7f9c8e7fc700 1 -- 172.21.15.104:0/3046262546 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312079 (secure 0 0 0) 0x7f9c8800bcc0 con 0x7f9c7c06ace0 2023-06-10T18:53:33.904 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:33.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 msgr2=0x7f9c7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 0x7f9c7c06d190 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c00 tx=0x7f9c88008030 comp rx=0 tx=0).stop 2023-06-10T18:53:33.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 msgr2=0x7f9c9814e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:33.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c7f0 tx=0x7f9c9400cbb0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 shutdown_connections 2023-06-10T18:53:33.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c7c06ace0 0x7f9c7c06d190 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c9806b180 0x7f9c98156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c98158c50 0x7f9c981f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 --2- 172.21.15.104:0/3046262546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c9806ca40 0x7f9c9814e770 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:33.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 >> 172.21.15.104:0/3046262546 conn(0x7f9c980b1610 msgr2=0x7f9c980c9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:33.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 shutdown_connections 2023-06-10T18:53:33.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:33.899+0000 7f9c9fe08700 1 -- 172.21.15.104:0/3046262546 wait complete. 2023-06-10T18:53:33.909 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:53:33.916 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2023-06-10T18:53:33.501211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226348,"kb_used_data":10732,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754748372,"statfs":{"total":773094113280,"available":772862332928,"internally_reserved":0,"allocated":10989568,"data_stored":5875635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.455934"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456131+0000","last_change":"2023-06-10T18:53:29.473468+0000","last_active":"2023-06-10T18:53:30.456131+0000","last_peered":"2023-06-10T18:53:30.456131+0000","last_clean":"2023-06-10T18:53:30.456131+0000","last_became_active":"2023-06-10T18:53:29.472929+0000","last_became_peered":"2023-06-10T18:53:29.472929+0000","last_unstale":"2023-06-10T18:53:30.456131+0000","last_undegraded":"2023-06-10T18:53:30.456131+0000","last_fullsized":"2023-06-10T18:53:30.456131+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:08.442260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832063+0000","last_change":"2023-06-10T18:53:29.468899+0000","last_active":"2023-06-10T18:53:30.832063+0000","last_peered":"2023-06-10T18:53:30.832063+0000","last_clean":"2023-06-10T18:53:30.832063+0000","last_became_active":"2023-06-10T18:53:29.468356+0000","last_became_peered":"2023-06-10T18:53:29.468356+0000","last_unstale":"2023-06-10T18:53:30.832063+0000","last_undegraded":"2023-06-10T18:53:30.832063+0000","last_fullsized":"2023-06-10T18:53:30.832063+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:54:31.797527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457010+0000","last_change":"2023-06-10T18:53:29.476168+0000","last_active":"2023-06-10T18:53:30.457010+0000","last_peered":"2023-06-10T18:53:30.457010+0000","last_clean":"2023-06-10T18:53:30.457010+0000","last_became_active":"2023-06-10T18:53:29.473795+0000","last_became_peered":"2023-06-10T18:53:29.473795+0000","last_unstale":"2023-06-10T18:53:30.457010+0000","last_undegraded":"2023-06-10T18:53:30.457010+0000","last_fullsized":"2023-06-10T18:53:30.457010+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:30.502910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456308+0000","last_change":"2023-06-10T18:53:29.471411+0000","last_active":"2023-06-10T18:53:30.456308+0000","last_peered":"2023-06-10T18:53:30.456308+0000","last_clean":"2023-06-10T18:53:30.456308+0000","last_became_active":"2023-06-10T18:53:29.471090+0000","last_became_peered":"2023-06-10T18:53:29.471090+0000","last_unstale":"2023-06-10T18:53:30.456308+0000","last_undegraded":"2023-06-10T18:53:30.456308+0000","last_fullsized":"2023-06-10T18:53:30.456308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:25:34.855338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833208+0000","last_change":"2023-06-10T18:53:29.476471+0000","last_active":"2023-06-10T18:53:30.833208+0000","last_peered":"2023-06-10T18:53:30.833208+0000","last_clean":"2023-06-10T18:53:30.833208+0000","last_became_active":"2023-06-10T18:53:29.476082+0000","last_became_peered":"2023-06-10T18:53:29.476082+0000","last_unstale":"2023-06-10T18:53:30.833208+0000","last_undegraded":"2023-06-10T18:53:30.833208+0000","last_fullsized":"2023-06-10T18:53:30.833208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:47:50.940134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831933+0000","last_change":"2023-06-10T18:53:29.467713+0000","last_active":"2023-06-10T18:53:30.831933+0000","last_peered":"2023-06-10T18:53:30.831933+0000","last_clean":"2023-06-10T18:53:30.831933+0000","last_became_active":"2023-06-10T18:53:29.467419+0000","last_became_peered":"2023-06-10T18:53:29.467419+0000","last_unstale":"2023-06-10T18:53:30.831933+0000","last_undegraded":"2023-06-10T18:53:30.831933+0000","last_fullsized":"2023-06-10T18:53:30.831933+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:30:13.582711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831962+0000","last_change":"2023-06-10T18:53:29.467663+0000","last_active":"2023-06-10T18:53:30.831962+0000","last_peered":"2023-06-10T18:53:30.831962+0000","last_clean":"2023-06-10T18:53:30.831962+0000","last_became_active":"2023-06-10T18:53:29.467363+0000","last_became_peered":"2023-06-10T18:53:29.467363+0000","last_unstale":"2023-06-10T18:53:30.831962+0000","last_undegraded":"2023-06-10T18:53:30.831962+0000","last_fullsized":"2023-06-10T18:53:30.831962+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:41:22.872171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831717+0000","last_change":"2023-06-10T18:53:29.467193+0000","last_active":"2023-06-10T18:53:30.831717+0000","last_peered":"2023-06-10T18:53:30.831717+0000","last_clean":"2023-06-10T18:53:30.831717+0000","last_became_active":"2023-06-10T18:53:29.466952+0000","last_became_peered":"2023-06-10T18:53:29.466952+0000","last_unstale":"2023-06-10T18:53:30.831717+0000","last_undegraded":"2023-06-10T18:53:30.831717+0000","last_fullsized":"2023-06-10T18:53:30.831717+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:46:20.365273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832453+0000","last_change":"2023-06-10T18:53:29.464465+0000","last_active":"2023-06-10T18:53:30.832453+0000","last_peered":"2023-06-10T18:53:30.832453+0000","last_clean":"2023-06-10T18:53:30.832453+0000","last_became_active":"2023-06-10T18:53:29.463818+0000","last_became_peered":"2023-06-10T18:53:29.463818+0000","last_unstale":"2023-06-10T18:53:30.832453+0000","last_undegraded":"2023-06-10T18:53:30.832453+0000","last_fullsized":"2023-06-10T18:53:30.832453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:40:42.181154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833136+0000","last_change":"2023-06-10T18:53:29.479289+0000","last_active":"2023-06-10T18:53:30.833136+0000","last_peered":"2023-06-10T18:53:30.833136+0000","last_clean":"2023-06-10T18:53:30.833136+0000","last_became_active":"2023-06-10T18:53:29.479008+0000","last_became_peered":"2023-06-10T18:53:29.479008+0000","last_unstale":"2023-06-10T18:53:30.833136+0000","last_undegraded":"2023-06-10T18:53:30.833136+0000","last_fullsized":"2023-06-10T18:53:30.833136+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:53:14.016600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832649+0000","last_change":"2023-06-10T18:53:29.468294+0000","last_active":"2023-06-10T18:53:30.832649+0000","last_peered":"2023-06-10T18:53:30.832649+0000","last_clean":"2023-06-10T18:53:30.832649+0000","last_became_active":"2023-06-10T18:53:29.467728+0000","last_became_peered":"2023-06-10T18:53:29.467728+0000","last_unstale":"2023-06-10T18:53:30.832649+0000","last_undegraded":"2023-06-10T18:53:30.832649+0000","last_fullsized":"2023-06-10T18:53:30.832649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:13.984214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832125+0000","last_change":"2023-06-10T18:53:29.468947+0000","last_active":"2023-06-10T18:53:30.832125+0000","last_peered":"2023-06-10T18:53:30.832125+0000","last_clean":"2023-06-10T18:53:30.832125+0000","last_became_active":"2023-06-10T18:53:29.468510+0000","last_became_peered":"2023-06-10T18:53:29.468510+0000","last_unstale":"2023-06-10T18:53:30.832125+0000","last_undegraded":"2023-06-10T18:53:30.832125+0000","last_fullsized":"2023-06-10T18:53:30.832125+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:16.232278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455933+0000","last_change":"2023-06-10T18:53:29.466512+0000","last_active":"2023-06-10T18:53:30.455933+0000","last_peered":"2023-06-10T18:53:30.455933+0000","last_clean":"2023-06-10T18:53:30.455933+0000","last_became_active":"2023-06-10T18:53:29.466271+0000","last_became_peered":"2023-06-10T18:53:29.466271+0000","last_unstale":"2023-06-10T18:53:30.455933+0000","last_undegraded":"2023-06-10T18:53:30.455933+0000","last_fullsized":"2023-06-10T18:53:30.455933+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:38:46.872625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831960+0000","last_change":"2023-06-10T18:53:29.471775+0000","last_active":"2023-06-10T18:53:30.831960+0000","last_peered":"2023-06-10T18:53:30.831960+0000","last_clean":"2023-06-10T18:53:30.831960+0000","last_became_active":"2023-06-10T18:53:29.471588+0000","last_became_peered":"2023-06-10T18:53:29.471588+0000","last_unstale":"2023-06-10T18:53:30.831960+0000","last_undegraded":"2023-06-10T18:53:30.831960+0000","last_fullsized":"2023-06-10T18:53:30.831960+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:25:14.966837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456237+0000","last_change":"2023-06-10T18:53:29.468656+0000","last_active":"2023-06-10T18:53:30.456237+0000","last_peered":"2023-06-10T18:53:30.456237+0000","last_clean":"2023-06-10T18:53:30.456237+0000","last_became_active":"2023-06-10T18:53:29.468442+0000","last_became_peered":"2023-06-10T18:53:29.468442+0000","last_unstale":"2023-06-10T18:53:30.456237+0000","last_undegraded":"2023-06-10T18:53:30.456237+0000","last_fullsized":"2023-06-10T18:53:30.456237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:10:30.321365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831735+0000","last_change":"2023-06-10T18:53:29.474996+0000","last_active":"2023-06-10T18:53:30.831735+0000","last_peered":"2023-06-10T18:53:30.831735+0000","last_clean":"2023-06-10T18:53:30.831735+0000","last_became_active":"2023-06-10T18:53:29.474740+0000","last_became_peered":"2023-06-10T18:53:29.474740+0000","last_unstale":"2023-06-10T18:53:30.831735+0000","last_undegraded":"2023-06-10T18:53:30.831735+0000","last_fullsized":"2023-06-10T18:53:30.831735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:33:10.398542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831928+0000","last_change":"2023-06-10T18:53:29.462171+0000","last_active":"2023-06-10T18:53:30.831928+0000","last_peered":"2023-06-10T18:53:30.831928+0000","last_clean":"2023-06-10T18:53:30.831928+0000","last_became_active":"2023-06-10T18:53:29.461879+0000","last_became_peered":"2023-06-10T18:53:29.461879+0000","last_unstale":"2023-06-10T18:53:30.831928+0000","last_undegraded":"2023-06-10T18:53:30.831928+0000","last_fullsized":"2023-06-10T18:53:30.831928+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:59:28.264281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456609+0000","last_change":"2023-06-10T18:53:29.471759+0000","last_active":"2023-06-10T18:53:30.456609+0000","last_peered":"2023-06-10T18:53:30.456609+0000","last_clean":"2023-06-10T18:53:30.456609+0000","last_became_active":"2023-06-10T18:53:29.471532+0000","last_became_peered":"2023-06-10T18:53:29.471532+0000","last_unstale":"2023-06-10T18:53:30.456609+0000","last_undegraded":"2023-06-10T18:53:30.456609+0000","last_fullsized":"2023-06-10T18:53:30.456609+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:45.644937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832024+0000","last_change":"2023-06-10T18:53:29.473924+0000","last_active":"2023-06-10T18:53:30.832024+0000","last_peered":"2023-06-10T18:53:30.832024+0000","last_clean":"2023-06-10T18:53:30.832024+0000","last_became_active":"2023-06-10T18:53:29.473391+0000","last_became_peered":"2023-06-10T18:53:29.473391+0000","last_unstale":"2023-06-10T18:53:30.832024+0000","last_undegraded":"2023-06-10T18:53:30.832024+0000","last_fullsized":"2023-06-10T18:53:30.832024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:57:46.625564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832327+0000","last_change":"2023-06-10T18:53:29.466723+0000","last_active":"2023-06-10T18:53:30.832327+0000","last_peered":"2023-06-10T18:53:30.832327+0000","last_clean":"2023-06-10T18:53:30.832327+0000","last_became_active":"2023-06-10T18:53:29.466453+0000","last_became_peered":"2023-06-10T18:53:29.466453+0000","last_unstale":"2023-06-10T18:53:30.832327+0000","last_undegraded":"2023-06-10T18:53:30.832327+0000","last_fullsized":"2023-06-10T18:53:30.832327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:01:11.087102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831636+0000","last_change":"2023-06-10T18:53:29.468433+0000","last_active":"2023-06-10T18:53:30.831636+0000","last_peered":"2023-06-10T18:53:30.831636+0000","last_clean":"2023-06-10T18:53:30.831636+0000","last_became_active":"2023-06-10T18:53:29.468177+0000","last_became_peered":"2023-06-10T18:53:29.468177+0000","last_unstale":"2023-06-10T18:53:30.831636+0000","last_undegraded":"2023-06-10T18:53:30.831636+0000","last_fullsized":"2023-06-10T18:53:30.831636+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:10:17.772880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831581+0000","last_change":"2023-06-10T18:53:29.468054+0000","last_active":"2023-06-10T18:53:30.831581+0000","last_peered":"2023-06-10T18:53:30.831581+0000","last_clean":"2023-06-10T18:53:30.831581+0000","last_became_active":"2023-06-10T18:53:29.467787+0000","last_became_peered":"2023-06-10T18:53:29.467787+0000","last_unstale":"2023-06-10T18:53:30.831581+0000","last_undegraded":"2023-06-10T18:53:30.831581+0000","last_fullsized":"2023-06-10T18:53:30.831581+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:14.553417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832186+0000","last_change":"2023-06-10T18:53:29.467089+0000","last_active":"2023-06-10T18:53:30.832186+0000","last_peered":"2023-06-10T18:53:30.832186+0000","last_clean":"2023-06-10T18:53:30.832186+0000","last_became_active":"2023-06-10T18:53:29.466834+0000","last_became_peered":"2023-06-10T18:53:29.466834+0000","last_unstale":"2023-06-10T18:53:30.832186+0000","last_undegraded":"2023-06-10T18:53:30.832186+0000","last_fullsized":"2023-06-10T18:53:30.832186+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:51.384932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831779+0000","last_change":"2023-06-10T18:53:29.478802+0000","last_active":"2023-06-10T18:53:30.831779+0000","last_peered":"2023-06-10T18:53:30.831779+0000","last_clean":"2023-06-10T18:53:30.831779+0000","last_became_active":"2023-06-10T18:53:29.478200+0000","last_became_peered":"2023-06-10T18:53:29.478200+0000","last_unstale":"2023-06-10T18:53:30.831779+0000","last_undegraded":"2023-06-10T18:53:30.831779+0000","last_fullsized":"2023-06-10T18:53:30.831779+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:52:20.030705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833513+0000","last_change":"2023-06-10T18:53:29.475014+0000","last_active":"2023-06-10T18:53:30.833513+0000","last_peered":"2023-06-10T18:53:30.833513+0000","last_clean":"2023-06-10T18:53:30.833513+0000","last_became_active":"2023-06-10T18:53:29.474785+0000","last_became_peered":"2023-06-10T18:53:29.474785+0000","last_unstale":"2023-06-10T18:53:30.833513+0000","last_undegraded":"2023-06-10T18:53:30.833513+0000","last_fullsized":"2023-06-10T18:53:30.833513+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:39:12.553956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832008+0000","last_change":"2023-06-10T18:53:29.468742+0000","last_active":"2023-06-10T18:53:30.832008+0000","last_peered":"2023-06-10T18:53:30.832008+0000","last_clean":"2023-06-10T18:53:30.832008+0000","last_became_active":"2023-06-10T18:53:29.468084+0000","last_became_peered":"2023-06-10T18:53:29.468084+0000","last_unstale":"2023-06-10T18:53:30.832008+0000","last_undegraded":"2023-06-10T18:53:30.832008+0000","last_fullsized":"2023-06-10T18:53:30.832008+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:16:20.375143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832868+0000","last_change":"2023-06-10T18:53:29.482882+0000","last_active":"2023-06-10T18:53:30.832868+0000","last_peered":"2023-06-10T18:53:30.832868+0000","last_clean":"2023-06-10T18:53:30.832868+0000","last_became_active":"2023-06-10T18:53:29.482654+0000","last_became_peered":"2023-06-10T18:53:29.482654+0000","last_unstale":"2023-06-10T18:53:30.832868+0000","last_undegraded":"2023-06-10T18:53:30.832868+0000","last_fullsized":"2023-06-10T18:53:30.832868+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:05:38.824105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833198+0000","last_change":"2023-06-10T18:53:29.476266+0000","last_active":"2023-06-10T18:53:30.833198+0000","last_peered":"2023-06-10T18:53:30.833198+0000","last_clean":"2023-06-10T18:53:30.833198+0000","last_became_active":"2023-06-10T18:53:29.475998+0000","last_became_peered":"2023-06-10T18:53:29.475998+0000","last_unstale":"2023-06-10T18:53:30.833198+0000","last_undegraded":"2023-06-10T18:53:30.833198+0000","last_fullsized":"2023-06-10T18:53:30.833198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:05:47.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831735+0000","last_change":"2023-06-10T18:53:29.469956+0000","last_active":"2023-06-10T18:53:30.831735+0000","last_peered":"2023-06-10T18:53:30.831735+0000","last_clean":"2023-06-10T18:53:30.831735+0000","last_became_active":"2023-06-10T18:53:29.469719+0000","last_became_peered":"2023-06-10T18:53:29.469719+0000","last_unstale":"2023-06-10T18:53:30.831735+0000","last_undegraded":"2023-06-10T18:53:30.831735+0000","last_fullsized":"2023-06-10T18:53:30.831735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:10:57.831967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831837+0000","last_change":"2023-06-10T18:53:29.469920+0000","last_active":"2023-06-10T18:53:30.831837+0000","last_peered":"2023-06-10T18:53:30.831837+0000","last_clean":"2023-06-10T18:53:30.831837+0000","last_became_active":"2023-06-10T18:53:29.469669+0000","last_became_peered":"2023-06-10T18:53:29.469669+0000","last_unstale":"2023-06-10T18:53:30.831837+0000","last_undegraded":"2023-06-10T18:53:30.831837+0000","last_fullsized":"2023-06-10T18:53:30.831837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:02.054872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456371+0000","last_change":"2023-06-10T18:53:29.474173+0000","last_active":"2023-06-10T18:53:30.456371+0000","last_peered":"2023-06-10T18:53:30.456371+0000","last_clean":"2023-06-10T18:53:30.456371+0000","last_became_active":"2023-06-10T18:53:29.473900+0000","last_became_peered":"2023-06-10T18:53:29.473900+0000","last_unstale":"2023-06-10T18:53:30.456371+0000","last_undegraded":"2023-06-10T18:53:30.456371+0000","last_fullsized":"2023-06-10T18:53:30.456371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:35.490497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455832+0000","last_change":"2023-06-10T18:53:29.477745+0000","last_active":"2023-06-10T18:53:30.455832+0000","last_peered":"2023-06-10T18:53:30.455832+0000","last_clean":"2023-06-10T18:53:30.455832+0000","last_became_active":"2023-06-10T18:53:29.476736+0000","last_became_peered":"2023-06-10T18:53:29.476736+0000","last_unstale":"2023-06-10T18:53:30.455832+0000","last_undegraded":"2023-06-10T18:53:30.455832+0000","last_fullsized":"2023-06-10T18:53:30.455832+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:41:38.229832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832207+0000","last_change":"2023-06-10T18:53:29.467293+0000","last_active":"2023-06-10T18:53:30.832207+0000","last_peered":"2023-06-10T18:53:30.832207+0000","last_clean":"2023-06-10T18:53:30.832207+0000","last_became_active":"2023-06-10T18:53:29.467018+0000","last_became_peered":"2023-06-10T18:53:29.467018+0000","last_unstale":"2023-06-10T18:53:30.832207+0000","last_undegraded":"2023-06-10T18:53:30.832207+0000","last_fullsized":"2023-06-10T18:53:30.832207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:09:01.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832030+0000","last_change":"2023-06-10T18:53:29.066047+0000","last_active":"2023-06-10T18:53:30.832030+0000","last_peered":"2023-06-10T18:53:30.832030+0000","last_clean":"2023-06-10T18:53:30.832030+0000","last_became_active":"2023-06-10T18:53:29.065782+0000","last_became_peered":"2023-06-10T18:53:29.065782+0000","last_unstale":"2023-06-10T18:53:30.832030+0000","last_undegraded":"2023-06-10T18:53:30.832030+0000","last_fullsized":"2023-06-10T18:53:30.832030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:56:36.797675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456780+0000","last_change":"2023-06-10T18:53:29.470770+0000","last_active":"2023-06-10T18:53:30.456780+0000","last_peered":"2023-06-10T18:53:30.456780+0000","last_clean":"2023-06-10T18:53:30.456780+0000","last_became_active":"2023-06-10T18:53:29.470511+0000","last_became_peered":"2023-06-10T18:53:29.470511+0000","last_unstale":"2023-06-10T18:53:30.456780+0000","last_undegraded":"2023-06-10T18:53:30.456780+0000","last_fullsized":"2023-06-10T18:53:30.456780+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:55:05.441828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832367+0000","last_change":"2023-06-10T18:53:29.069373+0000","last_active":"2023-06-10T18:53:30.832367+0000","last_peered":"2023-06-10T18:53:30.832367+0000","last_clean":"2023-06-10T18:53:30.832367+0000","last_became_active":"2023-06-10T18:53:29.069093+0000","last_became_peered":"2023-06-10T18:53:29.069093+0000","last_unstale":"2023-06-10T18:53:30.832367+0000","last_undegraded":"2023-06-10T18:53:30.832367+0000","last_fullsized":"2023-06-10T18:53:30.832367+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:45:53.856459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456189+0000","last_change":"2023-06-10T18:53:29.473265+0000","last_active":"2023-06-10T18:53:30.456189+0000","last_peered":"2023-06-10T18:53:30.456189+0000","last_clean":"2023-06-10T18:53:30.456189+0000","last_became_active":"2023-06-10T18:53:29.472970+0000","last_became_peered":"2023-06-10T18:53:29.472970+0000","last_unstale":"2023-06-10T18:53:30.456189+0000","last_undegraded":"2023-06-10T18:53:30.456189+0000","last_fullsized":"2023-06-10T18:53:30.456189+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:25.641646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832082+0000","last_change":"2023-06-10T18:53:29.068224+0000","last_active":"2023-06-10T18:53:30.832082+0000","last_peered":"2023-06-10T18:53:30.832082+0000","last_clean":"2023-06-10T18:53:30.832082+0000","last_became_active":"2023-06-10T18:53:29.068029+0000","last_became_peered":"2023-06-10T18:53:29.068029+0000","last_unstale":"2023-06-10T18:53:30.832082+0000","last_undegraded":"2023-06-10T18:53:30.832082+0000","last_fullsized":"2023-06-10T18:53:30.832082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:47.062679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833185+0000","last_change":"2023-06-10T18:53:29.471357+0000","last_active":"2023-06-10T18:53:30.833185+0000","last_peered":"2023-06-10T18:53:30.833185+0000","last_clean":"2023-06-10T18:53:30.833185+0000","last_became_active":"2023-06-10T18:53:29.470949+0000","last_became_peered":"2023-06-10T18:53:29.470949+0000","last_unstale":"2023-06-10T18:53:30.833185+0000","last_undegraded":"2023-06-10T18:53:30.833185+0000","last_fullsized":"2023-06-10T18:53:30.833185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:55:48.365827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832273+0000","last_change":"2023-06-10T18:53:29.061893+0000","last_active":"2023-06-10T18:53:30.832273+0000","last_peered":"2023-06-10T18:53:30.832273+0000","last_clean":"2023-06-10T18:53:30.832273+0000","last_became_active":"2023-06-10T18:53:29.061638+0000","last_became_peered":"2023-06-10T18:53:29.061638+0000","last_unstale":"2023-06-10T18:53:30.832273+0000","last_undegraded":"2023-06-10T18:53:30.832273+0000","last_fullsized":"2023-06-10T18:53:30.832273+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:37:28.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457535+0000","last_change":"2023-06-10T18:53:29.470107+0000","last_active":"2023-06-10T18:53:30.457535+0000","last_peered":"2023-06-10T18:53:30.457535+0000","last_clean":"2023-06-10T18:53:30.457535+0000","last_became_active":"2023-06-10T18:53:29.469843+0000","last_became_peered":"2023-06-10T18:53:29.469843+0000","last_unstale":"2023-06-10T18:53:30.457535+0000","last_undegraded":"2023-06-10T18:53:30.457535+0000","last_fullsized":"2023-06-10T18:53:30.457535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:12.080153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457050+0000","last_change":"2023-06-10T18:53:29.069038+0000","last_active":"2023-06-10T18:53:30.457050+0000","last_peered":"2023-06-10T18:53:30.457050+0000","last_clean":"2023-06-10T18:53:30.457050+0000","last_became_active":"2023-06-10T18:53:29.068743+0000","last_became_peered":"2023-06-10T18:53:29.068743+0000","last_unstale":"2023-06-10T18:53:30.457050+0000","last_undegraded":"2023-06-10T18:53:30.457050+0000","last_fullsized":"2023-06-10T18:53:30.457050+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:09:00.074642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831802+0000","last_change":"2023-06-10T18:53:29.473771+0000","last_active":"2023-06-10T18:53:30.831802+0000","last_peered":"2023-06-10T18:53:30.831802+0000","last_clean":"2023-06-10T18:53:30.831802+0000","last_became_active":"2023-06-10T18:53:29.473502+0000","last_became_peered":"2023-06-10T18:53:29.473502+0000","last_unstale":"2023-06-10T18:53:30.831802+0000","last_undegraded":"2023-06-10T18:53:30.831802+0000","last_fullsized":"2023-06-10T18:53:30.831802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:51:06.717229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831677+0000","last_change":"2023-06-10T18:53:29.066567+0000","last_active":"2023-06-10T18:53:30.831677+0000","last_peered":"2023-06-10T18:53:30.831677+0000","last_clean":"2023-06-10T18:53:30.831677+0000","last_became_active":"2023-06-10T18:53:29.065453+0000","last_became_peered":"2023-06-10T18:53:29.065453+0000","last_unstale":"2023-06-10T18:53:30.831677+0000","last_undegraded":"2023-06-10T18:53:30.831677+0000","last_fullsized":"2023-06-10T18:53:30.831677+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:32.068761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456034+0000","last_change":"2023-06-10T18:53:29.469242+0000","last_active":"2023-06-10T18:53:30.456034+0000","last_peered":"2023-06-10T18:53:30.456034+0000","last_clean":"2023-06-10T18:53:30.456034+0000","last_became_active":"2023-06-10T18:53:29.468990+0000","last_became_peered":"2023-06-10T18:53:29.468990+0000","last_unstale":"2023-06-10T18:53:30.456034+0000","last_undegraded":"2023-06-10T18:53:30.456034+0000","last_fullsized":"2023-06-10T18:53:30.456034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:15:45.624718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833031+0000","last_change":"2023-06-10T18:53:29.061978+0000","last_active":"2023-06-10T18:53:30.833031+0000","last_peered":"2023-06-10T18:53:30.833031+0000","last_clean":"2023-06-10T18:53:30.833031+0000","last_became_active":"2023-06-10T18:53:29.061625+0000","last_became_peered":"2023-06-10T18:53:29.061625+0000","last_unstale":"2023-06-10T18:53:30.833031+0000","last_undegraded":"2023-06-10T18:53:30.833031+0000","last_fullsized":"2023-06-10T18:53:30.833031+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.673427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831835+0000","last_change":"2023-06-10T18:53:29.469424+0000","last_active":"2023-06-10T18:53:30.831835+0000","last_peered":"2023-06-10T18:53:30.831835+0000","last_clean":"2023-06-10T18:53:30.831835+0000","last_became_active":"2023-06-10T18:53:29.469118+0000","last_became_peered":"2023-06-10T18:53:29.469118+0000","last_unstale":"2023-06-10T18:53:30.831835+0000","last_undegraded":"2023-06-10T18:53:30.831835+0000","last_fullsized":"2023-06-10T18:53:30.831835+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:05.039746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455691+0000","last_change":"2023-06-10T18:53:29.070985+0000","last_active":"2023-06-10T18:53:30.455691+0000","last_peered":"2023-06-10T18:53:30.455691+0000","last_clean":"2023-06-10T18:53:30.455691+0000","last_became_active":"2023-06-10T18:53:29.070535+0000","last_became_peered":"2023-06-10T18:53:29.070535+0000","last_unstale":"2023-06-10T18:53:30.455691+0000","last_undegraded":"2023-06-10T18:53:30.455691+0000","last_fullsized":"2023-06-10T18:53:30.455691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:07:54.057733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832207+0000","last_change":"2023-06-10T18:53:29.469415+0000","last_active":"2023-06-10T18:53:30.832207+0000","last_peered":"2023-06-10T18:53:30.832207+0000","last_clean":"2023-06-10T18:53:30.832207+0000","last_became_active":"2023-06-10T18:53:29.469147+0000","last_became_peered":"2023-06-10T18:53:29.469147+0000","last_unstale":"2023-06-10T18:53:30.832207+0000","last_undegraded":"2023-06-10T18:53:30.832207+0000","last_fullsized":"2023-06-10T18:53:30.832207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:29.479497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456377+0000","last_change":"2023-06-10T18:53:29.071101+0000","last_active":"2023-06-10T18:53:30.456377+0000","last_peered":"2023-06-10T18:53:30.456377+0000","last_clean":"2023-06-10T18:53:30.456377+0000","last_became_active":"2023-06-10T18:53:29.070820+0000","last_became_peered":"2023-06-10T18:53:29.070820+0000","last_unstale":"2023-06-10T18:53:30.456377+0000","last_undegraded":"2023-06-10T18:53:30.456377+0000","last_fullsized":"2023-06-10T18:53:30.456377+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:48:30.964846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832550+0000","last_change":"2023-06-10T18:53:29.469938+0000","last_active":"2023-06-10T18:53:30.832550+0000","last_peered":"2023-06-10T18:53:30.832550+0000","last_clean":"2023-06-10T18:53:30.832550+0000","last_became_active":"2023-06-10T18:53:29.469506+0000","last_became_peered":"2023-06-10T18:53:29.469506+0000","last_unstale":"2023-06-10T18:53:30.832550+0000","last_undegraded":"2023-06-10T18:53:30.832550+0000","last_fullsized":"2023-06-10T18:53:30.832550+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:52:50.866828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456697+0000","last_change":"2023-06-10T18:53:29.062132+0000","last_active":"2023-06-10T18:53:30.456697+0000","last_peered":"2023-06-10T18:53:30.456697+0000","last_clean":"2023-06-10T18:53:30.456697+0000","last_became_active":"2023-06-10T18:53:29.061854+0000","last_became_peered":"2023-06-10T18:53:29.061854+0000","last_unstale":"2023-06-10T18:53:30.456697+0000","last_undegraded":"2023-06-10T18:53:30.456697+0000","last_fullsized":"2023-06-10T18:53:30.456697+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:37:54.053528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833348+0000","last_change":"2023-06-10T18:53:29.474991+0000","last_active":"2023-06-10T18:53:30.833348+0000","last_peered":"2023-06-10T18:53:30.833348+0000","last_clean":"2023-06-10T18:53:30.833348+0000","last_became_active":"2023-06-10T18:53:29.474735+0000","last_became_peered":"2023-06-10T18:53:29.474735+0000","last_unstale":"2023-06-10T18:53:30.833348+0000","last_undegraded":"2023-06-10T18:53:30.833348+0000","last_fullsized":"2023-06-10T18:53:30.833348+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:55:34.768404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832791+0000","last_change":"2023-06-10T18:53:29.061307+0000","last_active":"2023-06-10T18:53:30.832791+0000","last_peered":"2023-06-10T18:53:30.832791+0000","last_clean":"2023-06-10T18:53:30.832791+0000","last_became_active":"2023-06-10T18:53:29.061005+0000","last_became_peered":"2023-06-10T18:53:29.061005+0000","last_unstale":"2023-06-10T18:53:30.832791+0000","last_undegraded":"2023-06-10T18:53:30.832791+0000","last_fullsized":"2023-06-10T18:53:30.832791+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:29:21.496833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832497+0000","last_change":"2023-06-10T18:53:29.475041+0000","last_active":"2023-06-10T18:53:30.832497+0000","last_peered":"2023-06-10T18:53:30.832497+0000","last_clean":"2023-06-10T18:53:30.832497+0000","last_became_active":"2023-06-10T18:53:29.474805+0000","last_became_peered":"2023-06-10T18:53:29.474805+0000","last_unstale":"2023-06-10T18:53:30.832497+0000","last_undegraded":"2023-06-10T18:53:30.832497+0000","last_fullsized":"2023-06-10T18:53:30.832497+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:02.031339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":19,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832488+0000","last_change":"2023-06-10T18:53:29.064193+0000","last_active":"2023-06-10T18:53:30.832488+0000","last_peered":"2023-06-10T18:53:30.832488+0000","last_clean":"2023-06-10T18:53:30.832488+0000","last_became_active":"2023-06-10T18:53:29.063959+0000","last_became_peered":"2023-06-10T18:53:29.063959+0000","last_unstale":"2023-06-10T18:53:30.832488+0000","last_undegraded":"2023-06-10T18:53:30.832488+0000","last_fullsized":"2023-06-10T18:53:30.832488+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:52:04.776737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455980+0000","last_change":"2023-06-10T18:53:29.470619+0000","last_active":"2023-06-10T18:53:30.455980+0000","last_peered":"2023-06-10T18:53:30.455980+0000","last_clean":"2023-06-10T18:53:30.455980+0000","last_became_active":"2023-06-10T18:53:29.470327+0000","last_became_peered":"2023-06-10T18:53:29.470327+0000","last_unstale":"2023-06-10T18:53:30.455980+0000","last_undegraded":"2023-06-10T18:53:30.455980+0000","last_fullsized":"2023-06-10T18:53:30.455980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:41:20.047008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831729+0000","last_change":"2023-06-10T18:53:29.064336+0000","last_active":"2023-06-10T18:53:30.831729+0000","last_peered":"2023-06-10T18:53:30.831729+0000","last_clean":"2023-06-10T18:53:30.831729+0000","last_became_active":"2023-06-10T18:53:29.064054+0000","last_became_peered":"2023-06-10T18:53:29.064054+0000","last_unstale":"2023-06-10T18:53:30.831729+0000","last_undegraded":"2023-06-10T18:53:30.831729+0000","last_fullsized":"2023-06-10T18:53:30.831729+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:57:01.876549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456001+0000","last_change":"2023-06-10T18:53:29.482222+0000","last_active":"2023-06-10T18:53:30.456001+0000","last_peered":"2023-06-10T18:53:30.456001+0000","last_clean":"2023-06-10T18:53:30.456001+0000","last_became_active":"2023-06-10T18:53:29.481963+0000","last_became_peered":"2023-06-10T18:53:29.481963+0000","last_unstale":"2023-06-10T18:53:30.456001+0000","last_undegraded":"2023-06-10T18:53:30.456001+0000","last_fullsized":"2023-06-10T18:53:30.456001+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:55.500417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832263+0000","last_change":"2023-06-10T18:53:29.068307+0000","last_active":"2023-06-10T18:53:30.832263+0000","last_peered":"2023-06-10T18:53:30.832263+0000","last_clean":"2023-06-10T18:53:30.832263+0000","last_became_active":"2023-06-10T18:53:29.068036+0000","last_became_peered":"2023-06-10T18:53:29.068036+0000","last_unstale":"2023-06-10T18:53:30.832263+0000","last_undegraded":"2023-06-10T18:53:30.832263+0000","last_fullsized":"2023-06-10T18:53:30.832263+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:38:48.451129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831978+0000","last_change":"2023-06-10T18:53:29.468918+0000","last_active":"2023-06-10T18:53:30.831978+0000","last_peered":"2023-06-10T18:53:30.831978+0000","last_clean":"2023-06-10T18:53:30.831978+0000","last_became_active":"2023-06-10T18:53:29.468517+0000","last_became_peered":"2023-06-10T18:53:29.468517+0000","last_unstale":"2023-06-10T18:53:30.831978+0000","last_undegraded":"2023-06-10T18:53:30.831978+0000","last_fullsized":"2023-06-10T18:53:30.831978+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:48:31.971495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832000+0000","last_change":"2023-06-10T18:53:29.063717+0000","last_active":"2023-06-10T18:53:30.832000+0000","last_peered":"2023-06-10T18:53:30.832000+0000","last_clean":"2023-06-10T18:53:30.832000+0000","last_became_active":"2023-06-10T18:53:29.063409+0000","last_became_peered":"2023-06-10T18:53:29.063409+0000","last_unstale":"2023-06-10T18:53:30.832000+0000","last_undegraded":"2023-06-10T18:53:30.832000+0000","last_fullsized":"2023-06-10T18:53:30.832000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:36.743372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831349+0000","last_change":"2023-06-10T18:53:29.469493+0000","last_active":"2023-06-10T18:53:30.831349+0000","last_peered":"2023-06-10T18:53:30.831349+0000","last_clean":"2023-06-10T18:53:30.831349+0000","last_became_active":"2023-06-10T18:53:29.469243+0000","last_became_peered":"2023-06-10T18:53:29.469243+0000","last_unstale":"2023-06-10T18:53:30.831349+0000","last_undegraded":"2023-06-10T18:53:30.831349+0000","last_fullsized":"2023-06-10T18:53:30.831349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:37:51.917441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":21,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.455958+0000","last_change":"2023-06-10T18:53:29.066592+0000","last_active":"2023-06-10T18:53:30.455958+0000","last_peered":"2023-06-10T18:53:30.455958+0000","last_clean":"2023-06-10T18:53:30.455958+0000","last_became_active":"2023-06-10T18:53:29.066321+0000","last_became_peered":"2023-06-10T18:53:29.066321+0000","last_unstale":"2023-06-10T18:53:30.455958+0000","last_undegraded":"2023-06-10T18:53:30.455958+0000","last_fullsized":"2023-06-10T18:53:30.455958+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:37:05.822365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832141+0000","last_change":"2023-06-10T18:53:29.468591+0000","last_active":"2023-06-10T18:53:30.832141+0000","last_peered":"2023-06-10T18:53:30.832141+0000","last_clean":"2023-06-10T18:53:30.832141+0000","last_became_active":"2023-06-10T18:53:29.467797+0000","last_became_peered":"2023-06-10T18:53:29.467797+0000","last_unstale":"2023-06-10T18:53:30.832141+0000","last_undegraded":"2023-06-10T18:53:30.832141+0000","last_fullsized":"2023-06-10T18:53:30.832141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:47:19.099850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457880+0000","last_change":"2023-06-10T18:53:29.061254+0000","last_active":"2023-06-10T18:53:30.457880+0000","last_peered":"2023-06-10T18:53:30.457880+0000","last_clean":"2023-06-10T18:53:30.457880+0000","last_became_active":"2023-06-10T18:53:29.060946+0000","last_became_peered":"2023-06-10T18:53:29.060946+0000","last_unstale":"2023-06-10T18:53:30.457880+0000","last_undegraded":"2023-06-10T18:53:30.457880+0000","last_fullsized":"2023-06-10T18:53:30.457880+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:49.851976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456258+0000","last_change":"2023-06-10T18:53:29.473162+0000","last_active":"2023-06-10T18:53:30.456258+0000","last_peered":"2023-06-10T18:53:30.456258+0000","last_clean":"2023-06-10T18:53:30.456258+0000","last_became_active":"2023-06-10T18:53:29.466711+0000","last_became_peered":"2023-06-10T18:53:29.466711+0000","last_unstale":"2023-06-10T18:53:30.456258+0000","last_undegraded":"2023-06-10T18:53:30.456258+0000","last_fullsized":"2023-06-10T18:53:30.456258+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:32:55.739710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832277+0000","last_change":"2023-06-10T18:53:29.071073+0000","last_active":"2023-06-10T18:53:30.832277+0000","last_peered":"2023-06-10T18:53:30.832277+0000","last_clean":"2023-06-10T18:53:30.832277+0000","last_became_active":"2023-06-10T18:53:29.070784+0000","last_became_peered":"2023-06-10T18:53:29.070784+0000","last_unstale":"2023-06-10T18:53:30.832277+0000","last_undegraded":"2023-06-10T18:53:30.832277+0000","last_fullsized":"2023-06-10T18:53:30.832277+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:58.249220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456322+0000","last_change":"2023-06-10T18:53:29.476137+0000","last_active":"2023-06-10T18:53:30.456322+0000","last_peered":"2023-06-10T18:53:30.456322+0000","last_clean":"2023-06-10T18:53:30.456322+0000","last_became_active":"2023-06-10T18:53:29.473736+0000","last_became_peered":"2023-06-10T18:53:29.473736+0000","last_unstale":"2023-06-10T18:53:30.456322+0000","last_undegraded":"2023-06-10T18:53:30.456322+0000","last_fullsized":"2023-06-10T18:53:30.456322+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:05.039746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832264+0000","last_change":"2023-06-10T18:53:29.058881+0000","last_active":"2023-06-10T18:53:30.832264+0000","last_peered":"2023-06-10T18:53:30.832264+0000","last_clean":"2023-06-10T18:53:30.832264+0000","last_became_active":"2023-06-10T18:53:29.058579+0000","last_became_peered":"2023-06-10T18:53:29.058579+0000","last_unstale":"2023-06-10T18:53:30.832264+0000","last_undegraded":"2023-06-10T18:53:30.832264+0000","last_fullsized":"2023-06-10T18:53:30.832264+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:11:46.849465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833314+0000","last_change":"2023-06-10T18:53:29.476133+0000","last_active":"2023-06-10T18:53:30.833314+0000","last_peered":"2023-06-10T18:53:30.833314+0000","last_clean":"2023-06-10T18:53:30.833314+0000","last_became_active":"2023-06-10T18:53:29.475776+0000","last_became_peered":"2023-06-10T18:53:29.475776+0000","last_unstale":"2023-06-10T18:53:30.833314+0000","last_undegraded":"2023-06-10T18:53:30.833314+0000","last_fullsized":"2023-06-10T18:53:30.833314+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:52:04.177125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833423+0000","last_change":"2023-06-10T18:53:29.064217+0000","last_active":"2023-06-10T18:53:30.833423+0000","last_peered":"2023-06-10T18:53:30.833423+0000","last_clean":"2023-06-10T18:53:30.833423+0000","last_became_active":"2023-06-10T18:53:29.063905+0000","last_became_peered":"2023-06-10T18:53:29.063905+0000","last_unstale":"2023-06-10T18:53:30.833423+0000","last_undegraded":"2023-06-10T18:53:30.833423+0000","last_fullsized":"2023-06-10T18:53:30.833423+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:02:15.954425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832142+0000","last_change":"2023-06-10T18:53:29.467964+0000","last_active":"2023-06-10T18:53:30.832142+0000","last_peered":"2023-06-10T18:53:30.832142+0000","last_clean":"2023-06-10T18:53:30.832142+0000","last_became_active":"2023-06-10T18:53:29.467674+0000","last_became_peered":"2023-06-10T18:53:29.467674+0000","last_unstale":"2023-06-10T18:53:30.832142+0000","last_undegraded":"2023-06-10T18:53:30.832142+0000","last_fullsized":"2023-06-10T18:53:30.832142+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:09.900799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831963+0000","last_change":"2023-06-10T18:53:29.059874+0000","last_active":"2023-06-10T18:53:30.831963+0000","last_peered":"2023-06-10T18:53:30.831963+0000","last_clean":"2023-06-10T18:53:30.831963+0000","last_became_active":"2023-06-10T18:53:29.059599+0000","last_became_peered":"2023-06-10T18:53:29.059599+0000","last_unstale":"2023-06-10T18:53:30.831963+0000","last_undegraded":"2023-06-10T18:53:30.831963+0000","last_fullsized":"2023-06-10T18:53:30.831963+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:55:22.842032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832185+0000","last_change":"2023-06-10T18:53:29.471768+0000","last_active":"2023-06-10T18:53:30.832185+0000","last_peered":"2023-06-10T18:53:30.832185+0000","last_clean":"2023-06-10T18:53:30.832185+0000","last_became_active":"2023-06-10T18:53:29.471456+0000","last_became_peered":"2023-06-10T18:53:29.471456+0000","last_unstale":"2023-06-10T18:53:30.832185+0000","last_undegraded":"2023-06-10T18:53:30.832185+0000","last_fullsized":"2023-06-10T18:53:30.832185+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:58:30.502910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832871+0000","last_change":"2023-06-10T18:53:29.069172+0000","last_active":"2023-06-10T18:53:30.832871+0000","last_peered":"2023-06-10T18:53:30.832871+0000","last_clean":"2023-06-10T18:53:30.832871+0000","last_became_active":"2023-06-10T18:53:29.068859+0000","last_became_peered":"2023-06-10T18:53:29.068859+0000","last_unstale":"2023-06-10T18:53:30.832871+0000","last_undegraded":"2023-06-10T18:53:30.832871+0000","last_fullsized":"2023-06-10T18:53:30.832871+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:38:49.635088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831308+0000","last_change":"2023-06-10T18:53:29.468007+0000","last_active":"2023-06-10T18:53:30.831308+0000","last_peered":"2023-06-10T18:53:30.831308+0000","last_clean":"2023-06-10T18:53:30.831308+0000","last_became_active":"2023-06-10T18:53:29.467735+0000","last_became_peered":"2023-06-10T18:53:29.467735+0000","last_unstale":"2023-06-10T18:53:30.831308+0000","last_undegraded":"2023-06-10T18:53:30.831308+0000","last_fullsized":"2023-06-10T18:53:30.831308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:27:00.731981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456499+0000","last_change":"2023-06-10T18:53:29.063569+0000","last_active":"2023-06-10T18:53:30.456499+0000","last_peered":"2023-06-10T18:53:30.456499+0000","last_clean":"2023-06-10T18:53:30.456499+0000","last_became_active":"2023-06-10T18:53:29.063267+0000","last_became_peered":"2023-06-10T18:53:29.063267+0000","last_unstale":"2023-06-10T18:53:30.456499+0000","last_undegraded":"2023-06-10T18:53:30.456499+0000","last_fullsized":"2023-06-10T18:53:30.456499+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:04:51.152861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832861+0000","last_change":"2023-06-10T18:53:29.476059+0000","last_active":"2023-06-10T18:53:30.832861+0000","last_peered":"2023-06-10T18:53:30.832861+0000","last_clean":"2023-06-10T18:53:30.832861+0000","last_became_active":"2023-06-10T18:53:29.475787+0000","last_became_peered":"2023-06-10T18:53:29.475787+0000","last_unstale":"2023-06-10T18:53:30.832861+0000","last_undegraded":"2023-06-10T18:53:30.832861+0000","last_fullsized":"2023-06-10T18:53:30.832861+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:56:19.966340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831427+0000","last_change":"2023-06-10T18:53:29.069431+0000","last_active":"2023-06-10T18:53:30.831427+0000","last_peered":"2023-06-10T18:53:30.831427+0000","last_clean":"2023-06-10T18:53:30.831427+0000","last_became_active":"2023-06-10T18:53:29.069170+0000","last_became_peered":"2023-06-10T18:53:29.069170+0000","last_unstale":"2023-06-10T18:53:30.831427+0000","last_undegraded":"2023-06-10T18:53:30.831427+0000","last_fullsized":"2023-06-10T18:53:30.831427+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:12:49.704203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832872+0000","last_change":"2023-06-10T18:53:29.478789+0000","last_active":"2023-06-10T18:53:30.832872+0000","last_peered":"2023-06-10T18:53:30.832872+0000","last_clean":"2023-06-10T18:53:30.832872+0000","last_became_active":"2023-06-10T18:53:29.478528+0000","last_became_peered":"2023-06-10T18:53:29.478528+0000","last_unstale":"2023-06-10T18:53:30.832872+0000","last_undegraded":"2023-06-10T18:53:30.832872+0000","last_fullsized":"2023-06-10T18:53:30.832872+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:00:23.076875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831826+0000","last_change":"2023-06-10T18:53:29.070449+0000","last_active":"2023-06-10T18:53:30.831826+0000","last_peered":"2023-06-10T18:53:30.831826+0000","last_clean":"2023-06-10T18:53:30.831826+0000","last_became_active":"2023-06-10T18:53:29.070050+0000","last_became_peered":"2023-06-10T18:53:29.070050+0000","last_unstale":"2023-06-10T18:53:30.831826+0000","last_undegraded":"2023-06-10T18:53:30.831826+0000","last_fullsized":"2023-06-10T18:53:30.831826+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:22:55.184809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":155,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456372+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:53:30.456372+0000","last_peered":"2023-06-10T18:53:30.456372+0000","last_clean":"2023-06-10T18:53:30.456372+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:53:30.456372+0000","last_undegraded":"2023-06-10T18:53:30.456372+0000","last_fullsized":"2023-06-10T18:53:30.456372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:39:11.226586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833595+0000","last_change":"2023-06-10T18:53:29.467857+0000","last_active":"2023-06-10T18:53:30.833595+0000","last_peered":"2023-06-10T18:53:30.833595+0000","last_clean":"2023-06-10T18:53:30.833595+0000","last_became_active":"2023-06-10T18:53:29.467606+0000","last_became_peered":"2023-06-10T18:53:29.467606+0000","last_unstale":"2023-06-10T18:53:30.833595+0000","last_undegraded":"2023-06-10T18:53:30.833595+0000","last_fullsized":"2023-06-10T18:53:30.833595+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:57:43.418029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833305+0000","last_change":"2023-06-10T18:53:29.071615+0000","last_active":"2023-06-10T18:53:30.833305+0000","last_peered":"2023-06-10T18:53:30.833305+0000","last_clean":"2023-06-10T18:53:30.833305+0000","last_became_active":"2023-06-10T18:53:29.071307+0000","last_became_peered":"2023-06-10T18:53:29.071307+0000","last_unstale":"2023-06-10T18:53:30.833305+0000","last_undegraded":"2023-06-10T18:53:30.833305+0000","last_fullsized":"2023-06-10T18:53:30.833305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:09:00.074642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":171,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457601+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:53:30.457601+0000","last_peered":"2023-06-10T18:53:30.457601+0000","last_clean":"2023-06-10T18:53:30.457601+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:53:30.457601+0000","last_undegraded":"2023-06-10T18:53:30.457601+0000","last_fullsized":"2023-06-10T18:53:30.457601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:47.016165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457577+0000","last_change":"2023-06-10T18:53:29.473860+0000","last_active":"2023-06-10T18:53:30.457577+0000","last_peered":"2023-06-10T18:53:30.457577+0000","last_clean":"2023-06-10T18:53:30.457577+0000","last_became_active":"2023-06-10T18:53:29.473582+0000","last_became_peered":"2023-06-10T18:53:29.473582+0000","last_unstale":"2023-06-10T18:53:30.457577+0000","last_undegraded":"2023-06-10T18:53:30.457577+0000","last_fullsized":"2023-06-10T18:53:30.457577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:56:15.947373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833489+0000","last_change":"2023-06-10T18:53:29.063130+0000","last_active":"2023-06-10T18:53:30.833489+0000","last_peered":"2023-06-10T18:53:30.833489+0000","last_clean":"2023-06-10T18:53:30.833489+0000","last_became_active":"2023-06-10T18:53:29.062738+0000","last_became_peered":"2023-06-10T18:53:29.062738+0000","last_unstale":"2023-06-10T18:53:30.833489+0000","last_undegraded":"2023-06-10T18:53:30.833489+0000","last_fullsized":"2023-06-10T18:53:30.833489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:40:12.356866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831301+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:53:30.831301+0000","last_peered":"2023-06-10T18:53:30.831301+0000","last_clean":"2023-06-10T18:53:30.831301+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:53:30.831301+0000","last_undegraded":"2023-06-10T18:53:30.831301+0000","last_fullsized":"2023-06-10T18:53:30.831301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:30.353337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832153+0000","last_change":"2023-06-10T18:53:29.465099+0000","last_active":"2023-06-10T18:53:30.832153+0000","last_peered":"2023-06-10T18:53:30.832153+0000","last_clean":"2023-06-10T18:53:30.832153+0000","last_became_active":"2023-06-10T18:53:29.464858+0000","last_became_peered":"2023-06-10T18:53:29.464858+0000","last_unstale":"2023-06-10T18:53:30.832153+0000","last_undegraded":"2023-06-10T18:53:30.832153+0000","last_fullsized":"2023-06-10T18:53:30.832153+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:38:29.543445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832537+0000","last_change":"2023-06-10T18:53:29.065885+0000","last_active":"2023-06-10T18:53:30.832537+0000","last_peered":"2023-06-10T18:53:30.832537+0000","last_clean":"2023-06-10T18:53:30.832537+0000","last_became_active":"2023-06-10T18:53:29.065614+0000","last_became_peered":"2023-06-10T18:53:29.065614+0000","last_unstale":"2023-06-10T18:53:30.832537+0000","last_undegraded":"2023-06-10T18:53:30.832537+0000","last_fullsized":"2023-06-10T18:53:30.832537+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:35.966791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":171,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457475+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:53:30.457475+0000","last_peered":"2023-06-10T18:53:30.457475+0000","last_clean":"2023-06-10T18:53:30.457475+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:53:30.457475+0000","last_undegraded":"2023-06-10T18:53:30.457475+0000","last_fullsized":"2023-06-10T18:53:30.457475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:22:52.297361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":315,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831834+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:53:30.831834+0000","last_peered":"2023-06-10T18:53:30.831834+0000","last_clean":"2023-06-10T18:53:30.831834+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:53:30.831834+0000","last_undegraded":"2023-06-10T18:53:30.831834+0000","last_fullsized":"2023-06-10T18:53:30.831834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:53:39.834975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832725+0000","last_change":"2023-06-10T18:53:29.468475+0000","last_active":"2023-06-10T18:53:30.832725+0000","last_peered":"2023-06-10T18:53:30.832725+0000","last_clean":"2023-06-10T18:53:30.832725+0000","last_became_active":"2023-06-10T18:53:29.467997+0000","last_became_peered":"2023-06-10T18:53:29.467997+0000","last_unstale":"2023-06-10T18:53:30.832725+0000","last_undegraded":"2023-06-10T18:53:30.832725+0000","last_fullsized":"2023-06-10T18:53:30.832725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:32.714194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831926+0000","last_change":"2023-06-10T18:53:29.062404+0000","last_active":"2023-06-10T18:53:30.831926+0000","last_peered":"2023-06-10T18:53:30.831926+0000","last_clean":"2023-06-10T18:53:30.831926+0000","last_became_active":"2023-06-10T18:53:29.062124+0000","last_became_peered":"2023-06-10T18:53:29.062124+0000","last_unstale":"2023-06-10T18:53:30.831926+0000","last_undegraded":"2023-06-10T18:53:30.831926+0000","last_fullsized":"2023-06-10T18:53:30.831926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:24:31.583175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831965+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:53:30.831965+0000","last_peered":"2023-06-10T18:53:30.831965+0000","last_clean":"2023-06-10T18:53:30.831965+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:53:30.831965+0000","last_undegraded":"2023-06-10T18:53:30.831965+0000","last_fullsized":"2023-06-10T18:53:30.831965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:38:32.459635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831327+0000","last_change":"2023-06-10T18:53:29.464975+0000","last_active":"2023-06-10T18:53:30.831327+0000","last_peered":"2023-06-10T18:53:30.831327+0000","last_clean":"2023-06-10T18:53:30.831327+0000","last_became_active":"2023-06-10T18:53:29.464737+0000","last_became_peered":"2023-06-10T18:53:29.464737+0000","last_unstale":"2023-06-10T18:53:30.831327+0000","last_undegraded":"2023-06-10T18:53:30.831327+0000","last_fullsized":"2023-06-10T18:53:30.831327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:28:46.688002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457493+0000","last_change":"2023-06-10T18:53:29.065636+0000","last_active":"2023-06-10T18:53:30.457493+0000","last_peered":"2023-06-10T18:53:30.457493+0000","last_clean":"2023-06-10T18:53:30.457493+0000","last_became_active":"2023-06-10T18:53:29.065389+0000","last_became_peered":"2023-06-10T18:53:29.065389+0000","last_unstale":"2023-06-10T18:53:30.457493+0000","last_undegraded":"2023-06-10T18:53:30.457493+0000","last_fullsized":"2023-06-10T18:53:30.457493+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:09.107609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832879+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:53:30.832879+0000","last_peered":"2023-06-10T18:53:30.832879+0000","last_clean":"2023-06-10T18:53:30.832879+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:53:30.832879+0000","last_undegraded":"2023-06-10T18:53:30.832879+0000","last_fullsized":"2023-06-10T18:53:30.832879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:57:16.741631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831502+0000","last_change":"2023-06-10T18:53:29.470515+0000","last_active":"2023-06-10T18:53:30.831502+0000","last_peered":"2023-06-10T18:53:30.831502+0000","last_clean":"2023-06-10T18:53:30.831502+0000","last_became_active":"2023-06-10T18:53:29.470218+0000","last_became_peered":"2023-06-10T18:53:29.470218+0000","last_unstale":"2023-06-10T18:53:30.831502+0000","last_undegraded":"2023-06-10T18:53:30.831502+0000","last_fullsized":"2023-06-10T18:53:30.831502+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:50:52.860486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.457530+0000","last_change":"2023-06-10T18:53:29.064050+0000","last_active":"2023-06-10T18:53:30.457530+0000","last_peered":"2023-06-10T18:53:30.457530+0000","last_clean":"2023-06-10T18:53:30.457530+0000","last_became_active":"2023-06-10T18:53:29.063779+0000","last_became_peered":"2023-06-10T18:53:29.063779+0000","last_unstale":"2023-06-10T18:53:30.457530+0000","last_undegraded":"2023-06-10T18:53:30.457530+0000","last_fullsized":"2023-06-10T18:53:30.457530+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:24:27.751928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.456494+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:53:30.456494+0000","last_peered":"2023-06-10T18:53:30.456494+0000","last_clean":"2023-06-10T18:53:30.456494+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:53:30.456494+0000","last_undegraded":"2023-06-10T18:53:30.456494+0000","last_fullsized":"2023-06-10T18:53:30.456494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:15:49.708843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.832137+0000","last_change":"2023-06-10T18:53:29.467403+0000","last_active":"2023-06-10T18:53:30.832137+0000","last_peered":"2023-06-10T18:53:30.832137+0000","last_clean":"2023-06-10T18:53:30.832137+0000","last_became_active":"2023-06-10T18:53:29.467127+0000","last_became_peered":"2023-06-10T18:53:29.467127+0000","last_unstale":"2023-06-10T18:53:30.832137+0000","last_undegraded":"2023-06-10T18:53:30.832137+0000","last_fullsized":"2023-06-10T18:53:30.832137+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:29.043462+0000","last_clean_scrub_stamp":"2023-06-10T18:53:29.043462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:26:45.304396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.831869+0000","last_change":"2023-06-10T18:53:29.068241+0000","last_active":"2023-06-10T18:53:30.831869+0000","last_peered":"2023-06-10T18:53:30.831869+0000","last_clean":"2023-06-10T18:53:30.831869+0000","last_became_active":"2023-06-10T18:53:29.068042+0000","last_became_peered":"2023-06-10T18:53:29.068042+0000","last_unstale":"2023-06-10T18:53:30.831869+0000","last_undegraded":"2023-06-10T18:53:30.831869+0000","last_fullsized":"2023-06-10T18:53:30.831869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:53:28.036813+0000","last_clean_scrub_stamp":"2023-06-10T18:53:28.036813+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:35.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-06-10T18:53:30.833691+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:53:30.833691+0000","last_peered":"2023-06-10T18:53:30.833691+0000","last_clean":"2023-06-10T18:53:30.833691+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:53:30.833691+0000","last_undegraded":"2023-06-10T18:53:30.833691+0000","last_fullsized":"2023-06-10T18:53:30.833691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:37:26.238858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574967,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28720,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343120,"statfs":{"total":96636764160,"available":96607354880,"internally_reserved":0,"allocated":1810432,"data_stored":1171160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55700000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.65200000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58099999999999996,"15min":0.53800000000000003},"min":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.247},"max":{"1min":0.66300000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.499}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.55400000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.435},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.34799999999999998},"max":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.44800000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.505},"min":{"1min":0.34499999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70599999999999996,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.27300000000000002},"max":{"1min":0.70299999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.52300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.748,"5min":0.748,"15min":0.78300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.76400000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66400000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.441,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46899999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.311},"max":{"1min":0.71399999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.371,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80200000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.32600000000000001},"max":{"1min":0.73099999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.38700000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67500000000000004,"5min":0.71799999999999997,"15min":0.81100000000000005},"last":0.5},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.78700000000000003},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.309},"max":{"1min":0.59099999999999997,"5min":0.68899999999999995,"15min":0.80600000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.52700000000000002,"15min":0.74099999999999999},"min":{"1min":0.377,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.67200000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.61599999999999999,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.76300000000000001,"15min":1.252},"last":0.59099999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82899999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68200000000000005,"5min":0.73999999999999999,"15min":0.77500000000000002},"last":0.48599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48499999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67900000000000005,"5min":0.69599999999999995,"15min":0.82999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.89600000000000002},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47999999999999998,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.86699999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52100000000000002,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.79100000000000004},"last":0.38400000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:53:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.39000000000000001,"5min":0.35299999999999998,"15min":0.32100000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.41499999999999998,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.81399999999999995},"last":0.53100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28140,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343700,"statfs":{"total":96636764160,"available":96607948800,"internally_reserved":0,"allocated":1216512,"data_stored":586634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33100000000000002,"15min":0.33500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":0.51500000000000001,"15min":0.753},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.49299999999999999,"5min":0.53900000000000003,"15min":0.59599999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35299999999999998,"15min":0.34599999999999997},"min":{"1min":0.23799999999999999,"5min":0.215,"15min":0.20000000000000001},"max":{"1min":0.52200000000000002,"5min":0.57899999999999996,"15min":0.64500000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.373},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40799999999999997,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.41899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.873,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.84599999999999997,"5min":0.85999999999999999,"15min":0.93000000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":1.008},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.86299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.433}]},{"osd":7,"last update":"Sat Jun 10 18:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.95099999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60199999999999998,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.435,"15min":0.434},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.50700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574967,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28724,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343116,"statfs":{"total":96636764160,"available":96607350784,"internally_reserved":0,"allocated":1814528,"data_stored":1177584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33800000000000002,"15min":0.34399999999999997},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.51600000000000001,"5min":0.53700000000000003,"15min":0.72699999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.35399999999999998,"15min":0.35699999999999998},"min":{"1min":0.22900000000000001,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.5,"5min":0.58499999999999996,"15min":0.88900000000000001},"last":0.248}]},{"osd":2,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35999999999999999,"15min":0.375},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":1.405},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38300000000000001,"15min":0.38900000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":0.55100000000000005,"5min":0.65500000000000003,"15min":0.95499999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.61799999999999999,"15min":0.93100000000000005},"last":0.312},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.41299999999999998,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.27100000000000002,"15min":0.25900000000000001},"max":{"1min":0.63800000000000001,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":1.0489999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.39800000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.59799999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57899999999999996,"15min":0.59799999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.98199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44800000000000001,"5min":0.38800000000000001,"15min":0.38400000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.363},"last":0.502}]},{"osd":7,"last update":"Sat Jun 10 18:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60399999999999998},"min":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.38900000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":1.0149999999999999},"last":0.44}]}]},{"osd":2,"up_from":13,"seq":55834574968,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.45300000000000001,"15min":0.38600000000000001},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4450000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":0.37},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.46600000000000003,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.23799999999999999},"max":{"1min":5.6790000000000003,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.45500000000000002,"15min":0.38900000000000001},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.23899999999999999},"max":{"1min":7.4569999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.69799999999999995,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":5.742,"5min":5.742,"15min":5.742},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.69799999999999995,"15min":0.60999999999999999},"min":{"1min":0.39800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.68500000000000005,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.593,"5min":6.593,"15min":6.593},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.71099999999999997,"15min":0.65000000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38400000000000001},"max":{"1min":7.0880000000000001,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.496}]},{"osd":6,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.73299999999999998,"15min":0.63700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":8.7850000000000001,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.433},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.66500000000000004,"15min":0.625},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.6180000000000003,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.57599999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:53:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.70999999999999996,"15min":0.66200000000000003},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.40699999999999997},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.745,"15min":0.65400000000000003},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":8.7149999999999999,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.52600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574967,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1208,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1236992,"data_stored":589239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.36499999999999999,"15min":0.39200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36299999999999999,"15min":0.38200000000000001},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39400000000000002,"15min":0.41899999999999998},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":1.3},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.41499999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.46200000000000002},"min":{"1min":0.33000000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.599},"last":0.375}]},{"osd":4,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.925},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60599999999999998,"15min":0.59399999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.439}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.70099999999999996},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":3.802},"last":0.56599999999999995}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60999999999999999,"15min":0.69899999999999995},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.621,"15min":0.624},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.46400000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.46600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.46700000000000003,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.8939999999999999},"last":0.44800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574967,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1208,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1236992,"data_stored":588322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":0.68799999999999994,"5min":0.82599999999999996,"15min":1.119},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.44500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64900000000000002,"5min":0.74099999999999999,"15min":1.1639999999999999},"last":0.51600000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54300000000000004,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.55500000000000005,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.65400000000000003,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.52300000000000002},"min":{"1min":0.372,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.54900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68500000000000005,"5min":0.80100000000000005,"15min":1.0229999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.39400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70699999999999996,"5min":0.748,"15min":0.79800000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.042},"last":0.58799999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.72199999999999998,"5min":0.78700000000000003,"15min":1.069},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.34300000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.84099999999999997,"15min":0.88300000000000001},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.38700000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.90400000000000003,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.747,"5min":0.82999999999999996,"15min":1.3169999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:53:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.85899999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.46100000000000002,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.63200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574968,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1224704,"data_stored":587723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.42999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.51600000000000001},"min":{"1min":0.42299999999999999,"5min":0.34399999999999997,"15min":0.32300000000000001},"max":{"1min":0.70199999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70399999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53700000000000003,"15min":0.51900000000000002},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.33800000000000002},"max":{"1min":0.64800000000000002,"5min":0.70599999999999996,"15min":0.73099999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57199999999999995,"15min":0.52900000000000003},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.66100000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":0.71199999999999997,"5min":0.79000000000000004,"15min":0.85099999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55600000000000005,"15min":0.53300000000000003},"min":{"1min":0.40200000000000002,"5min":0.371,"15min":0.34999999999999998},"max":{"1min":0.69199999999999995,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56499999999999995,"15min":0.53400000000000003},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.55800000000000005},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.745}]},{"osd":4,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.47899999999999998},"min":{"1min":0.35099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68300000000000005,"5min":0.76400000000000001,"15min":0.83999999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51600000000000001,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.66600000000000004,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.40300000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.79200000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.879},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:53:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.41799999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.55600000000000005},"min":{"1min":0.44500000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73099999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.64100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:53:33.924 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:53:33.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:53:34.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 Processor -- start 2023-06-10T18:53:34.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 -- start start 2023-06-10T18:53:34.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fecfc158790 0x7fecfc0ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fecfc0caf70 0x7fecfc0cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc071580 con 0x7fecfc0ccfe0 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0cb950 con 0x7fecfc158790 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.079+0000 7fed016a0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0c9020 con 0x7fecfc0caf70 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fecfc158790 0x7fecfc0ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfa7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fecfc0ccfe0 0x7fecfc0c95c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60580/0 (socket says 172.21.15.104:60580) 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfa7fc700 1 -- 172.21.15.104:0/1125761916 learned_addr learned my addr 172.21.15.104:0/1125761916 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfb7fe700 1 --2- 172.21.15.104:0/1125761916 >> v2:172.21.15.114:3300/0 conn(0x7fecfc0caf70 0x7fecfc0cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 -- 172.21.15.104:0/1125761916 >> v2:172.21.15.114:3300/0 conn(0x7fecfc0caf70 msgr2=0x7fecfc0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 --2- 172.21.15.104:0/1125761916 >> v2:172.21.15.114:3300/0 conn(0x7fecfc0caf70 0x7fecfc0cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 -- 172.21.15.104:0/1125761916 >> v1:172.21.15.104:6789/0 conn(0x7fecfc0ccfe0 legacy=0x7fecfc0c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 -- 172.21.15.104:0/1125761916 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc070f30 con 0x7fecfc158790 2023-06-10T18:53:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecf9ffb700 1 -- 172.21.15.104:0/1125761916 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260448376 0 0) 0x7fecfc071580 con 0x7fecfc0ccfe0 2023-06-10T18:53:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecfaffd700 1 --2- 172.21.15.104:0/1125761916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc158790 0x7fecfc0ccaa0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0cf180 tx=0x7fecec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657e308ee954a50a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecf9ffb700 1 -- 172.21.15.104:0/1125761916 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec011830 con 0x7fecfc158790 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecf9ffb700 1 -- 172.21.15.104:0/1125761916 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec011990 con 0x7fecfc158790 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.080+0000 7fecf9ffb700 1 -- 172.21.15.104:0/1125761916 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec011d20 con 0x7fecfc158790 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 -- 172.21.15.104:0/1125761916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc158790 msgr2=0x7fecfc0ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 --2- 172.21.15.104:0/1125761916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc158790 0x7fecfc0ccaa0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0cf180 tx=0x7fecec009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 -- 172.21.15.104:0/1125761916 shutdown_connections 2023-06-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 --2- 172.21.15.104:0/1125761916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc158790 0x7fecfc0ccaa0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 --2- 172.21.15.104:0/1125761916 >> v2:172.21.15.114:3300/0 conn(0x7fecfc0caf70 0x7fecfc0cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 -- 172.21.15.104:0/1125761916 >> 172.21.15.104:0/1125761916 conn(0x7fecfc06aac0 msgr2=0x7fecfc0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 -- 172.21.15.104:0/1125761916 shutdown_connections 2023-06-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.081+0000 7fed016a0700 1 -- 172.21.15.104:0/1125761916 wait complete. 2023-06-10T18:53:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.082+0000 7fed016a0700 1 Processor -- start 2023-06-10T18:53:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.082+0000 7fed016a0700 1 -- start start 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fecfc0ccfe0 0x7fecfc0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecfc158790 0x7fecfc151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc164740 con 0x7fecfc0caf70 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc165490 con 0x7fecfc158790 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fed016a0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0c4fe0 con 0x7fecfc0ccfe0 2023-06-10T18:53:34.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34524/0 (socket says 172.21.15.104:34524) 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 -- 172.21.15.104:0/1450847313 learned_addr learned my addr 172.21.15.104:0/1450847313 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 -- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fecfc0ccfe0 msgr2=0x7fecfc0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fecfc0ccfe0 0x7fecfc0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 -- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecfc158790 msgr2=0x7fecfc151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecfc158790 0x7fecfc151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 -- 172.21.15.104:0/1450847313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0094a0 con 0x7fecfc0caf70 2023-06-10T18:53:34.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.083+0000 7fecfaffd700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fecec0090b0 tx=0x7fecec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.084+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0038a0 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.084+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec01e070 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.084+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc1649b0 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.084+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec003c20 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.084+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc164f70 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.085+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc165160 con 0x7fecfc0caf70 2023-06-10T18:53:34.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.085+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fecec02d450 con 0x7fecfc0caf70 2023-06-10T18:53:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.085+0000 7fece37fe700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 0x7fece806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.085+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fecec08a6e0 con 0x7fecfc0caf70 2023-06-10T18:53:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.086+0000 7fecfa7fc700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 0x7fece806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.086+0000 7fecfa7fc700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 0x7fece806d190 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fecf00011e0 tx=0x7fecf00079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.090+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fecec055650 con 0x7fecfc0caf70 2023-06-10T18:53:34.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.352+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fecfc074050 con 0x7fecfc0caf70 2023-06-10T18:53:34.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.356+0000 7fece37fe700 1 -- 172.21.15.104:0/1450847313 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7fecec058c70 con 0x7fecfc0caf70 2023-06-10T18:53:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 msgr2=0x7fece806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 0x7fece806d190 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fecf00011e0 tx=0x7fecf00079b0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 msgr2=0x7fecfc071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fecec0090b0 tx=0x7fecec004750 comp rx=0 tx=0).stop 2023-06-10T18:53:34.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 shutdown_connections 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fece806ace0 0x7fece806d190 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecfc158790 0x7fecfc151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fecfc0caf70 0x7fecfc071c80 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 --2- 172.21.15.104:0/1450847313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fecfc0ccfe0 0x7fecfc0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 >> 172.21.15.104:0/1450847313 conn(0x7fecfc06aac0 msgr2=0x7fecfc0b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:34.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 shutdown_connections 2023-06-10T18:53:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.358+0000 7fed016a0700 1 -- 172.21.15.104:0/1450847313 wait complete. 2023-06-10T18:53:34.365 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:53:34.375 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:34.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.490+0000 7fd63c644700 1 Processor -- start 2023-06-10T18:53:34.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.490+0000 7fd63c644700 1 -- start start 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.491+0000 7fd63c644700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd63415a420 0x7fd6340b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.491+0000 7fd63c644700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd6340b4ff0 0x7fd6340ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.491+0000 7fd63c644700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340b6070 con 0x7fd6340ce890 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.491+0000 7fd63c644700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340cecb0 con 0x7fd63415a420 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.491+0000 7fd63c644700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340b5460 con 0x7fd6340b4ff0 2023-06-10T18:53:34.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd63415a420 0x7fd6340b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63abe1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd6340ce890 0x7fd6340b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60592/0 (socket says 172.21.15.104:60592) 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63abe1700 1 -- 172.21.15.104:0/3896570525 learned_addr learned my addr 172.21.15.104:0/3896570525 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd639bdf700 1 --2- 172.21.15.104:0/3896570525 >> v2:172.21.15.114:3300/0 conn(0x7fd6340b4ff0 0x7fd6340ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 -- 172.21.15.104:0/3896570525 >> v2:172.21.15.114:3300/0 conn(0x7fd6340b4ff0 msgr2=0x7fd6340ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 --2- 172.21.15.104:0/3896570525 >> v2:172.21.15.114:3300/0 conn(0x7fd6340b4ff0 0x7fd6340ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 -- 172.21.15.104:0/3896570525 >> v1:172.21.15.104:6789/0 conn(0x7fd6340ce890 legacy=0x7fd6340b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd6393de700 1 -- 172.21.15.104:0/3896570525 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468954893 0 0) 0x7fd6340b6070 con 0x7fd6340ce890 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 -- 172.21.15.104:0/3896570525 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340b5740 con 0x7fd63415a420 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd63a3e0700 1 --2- 172.21.15.104:0/3896570525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6340b1660 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd634066590 tx=0x7fd624009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2724863f04d6be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd6393de700 1 -- 172.21.15.104:0/3896570525 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624011830 con 0x7fd63415a420 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd6393de700 1 -- 172.21.15.104:0/3896570525 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624011990 con 0x7fd63415a420 2023-06-10T18:53:34.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.492+0000 7fd6393de700 1 -- 172.21.15.104:0/3896570525 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624011d20 con 0x7fd63415a420 2023-06-10T18:53:34.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 -- 172.21.15.104:0/3896570525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 msgr2=0x7fd6340b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 --2- 172.21.15.104:0/3896570525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6340b1660 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd634066590 tx=0x7fd624009b20 comp rx=0 tx=0).stop 2023-06-10T18:53:34.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 -- 172.21.15.104:0/3896570525 shutdown_connections 2023-06-10T18:53:34.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 --2- 172.21.15.104:0/3896570525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6340b1660 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 --2- 172.21.15.104:0/3896570525 >> v2:172.21.15.114:3300/0 conn(0x7fd6340b4ff0 0x7fd6340ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 -- 172.21.15.104:0/3896570525 >> 172.21.15.104:0/3896570525 conn(0x7fd634089fd0 msgr2=0x7fd6340d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:34.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 -- 172.21.15.104:0/3896570525 shutdown_connections 2023-06-10T18:53:34.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.493+0000 7fd63c644700 1 -- 172.21.15.104:0/3896570525 wait complete. 2023-06-10T18:53:34.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.494+0000 7fd63c644700 1 Processor -- start 2023-06-10T18:53:34.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.494+0000 7fd63c644700 1 -- start start 2023-06-10T18:53:34.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6340b4ff0 0x7fd6341ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 0x7fd6340cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6341efe50 con 0x7fd63415a420 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340cc220 con 0x7fd6340ce890 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340cc500 con 0x7fd6340b4ff0 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34540/0 (socket says 172.21.15.104:34540) 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 -- 172.21.15.104:0/2047523003 learned_addr learned my addr 172.21.15.104:0/2047523003 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:34.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63a3e0700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6340b4ff0 0x7fd6341ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd639bdf700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 0x7fd6340cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 -- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6340b4ff0 msgr2=0x7fd6341ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6340b4ff0 0x7fd6341ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 -- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 msgr2=0x7fd6340cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 0x7fd6340cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 -- 172.21.15.104:0/2047523003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240094a0 con 0x7fd63415a420 2023-06-10T18:53:34.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd639bdf700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 0x7fd6340cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:34.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63abe1700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd62c002230 tx=0x7fd62c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c00ab70 con 0x7fd63415a420 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c00acf0 con 0x7fd63415a420 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.495+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6340cc7e0 con 0x7fd63415a420 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.496+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c007310 con 0x7fd63415a420 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.496+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6341e38c0 con 0x7fd63415a420 2023-06-10T18:53:34.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.496+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6340b64e0 con 0x7fd63415a420 2023-06-10T18:53:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.497+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd62c007470 con 0x7fd63415a420 2023-06-10T18:53:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.497+0000 7fd62affd700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 0x7fd61806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.497+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd62c089420 con 0x7fd63415a420 2023-06-10T18:53:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.498+0000 7fd63a3e0700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 0x7fd61806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:34.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.498+0000 7fd63a3e0700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 0x7fd61806d190 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd6240090b0 tx=0x7fd62401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:34.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.502+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd6340b64e0 con 0x7fd63415a420 2023-06-10T18:53:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.730+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6340b64e0 con 0x7fd63415a420 2023-06-10T18:53:34.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.731+0000 7fd62affd700 1 -- 172.21.15.104:0/2047523003 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fd6340b64e0 con 0x7fd63415a420 2023-06-10T18:53:34.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:34.736 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":55,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":7968,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:32.455645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7983":{"gid":7983,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3243355534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3243355534},{"type":"v1","addr":"172.21.15.114:6837","nonce":3243355534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:53:34.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 msgr2=0x7fd61806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 0x7fd61806d190 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd6240090b0 tx=0x7fd62401c040 comp rx=0 tx=0).stop 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 msgr2=0x7fd6341e0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd62c002230 tx=0x7fd62c00cf50 comp rx=0 tx=0).stop 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 shutdown_connections 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63415a420 0x7fd6341e0040 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6340b4ff0 0x7fd6341ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61806ace0 0x7fd61806d190 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 --2- 172.21.15.104:0/2047523003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6340ce890 0x7fd6340cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 >> 172.21.15.104:0/2047523003 conn(0x7fd634089fd0 msgr2=0x7fd63406f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 shutdown_connections 2023-06-10T18:53:34.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:34.733+0000 7fd63c644700 1 -- 172.21.15.104:0/2047523003 wait complete. 2023-06-10T18:53:34.740 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 55 2023-06-10T18:53:34.933 INFO:tasks.ceph.mds.b.smithi114.stderr:2023-06-10T18:53:34.928+0000 7f9d194c1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:53:35.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:53:29.447461+0000', 'modified': '2023-06-10T18:53:32.455645+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7983}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7983': {'gid': 7983, 'name': 'b', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/3243355534', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 3243355534}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 3243355534}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:53:35.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:53:35.152 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-06-10T18:53:35.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.270+0000 7f2d31bb5700 1 Processor -- start 2023-06-10T18:53:35.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.270+0000 7f2d31bb5700 1 -- start start 2023-06-10T18:53:35.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d31bb5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2d2c15a2b0 0x7f2d2c0ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d31bb5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d2c0cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d31bb5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c15a680 con 0x7f2d2c0cb690 2023-06-10T18:53:35.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d31bb5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d0510 con 0x7f2d2c15a2b0 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d31bb5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c06a5e0 con 0x7f2d2c069e70 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d2bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2d2c0cb690 0x7f2d2c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60606/0 (socket says 172.21.15.104:60606) 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.271+0000 7f2d2bfff700 1 -- 172.21.15.104:0/2133337624 learned_addr learned my addr 172.21.15.104:0/2133337624 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2affd700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d2c0cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/2133337624 >> v2:172.21.15.114:3300/0 conn(0x7f2d2c15a2b0 0x7f2d2c0ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340668978 0 0) 0x7f2d2c15a680 con 0x7f2d2c0cb690 2023-06-10T18:53:35.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d140035b0 con 0x7f2d2c0cb690 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3890700026 0 0) 0x7f2d140035b0 con 0x7f2d2c0cb690 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 msgr2=0x7f2d2c0cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d2c0cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 >> v2:172.21.15.114:3300/0 conn(0x7f2d2c15a2b0 msgr2=0x7f2d2c0ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 --2- 172.21.15.104:0/2133337624 >> v2:172.21.15.114:3300/0 conn(0x7f2d2c15a2b0 0x7f2d2c0ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c06a8c0 con 0x7f2d2c0cb690 2023-06-10T18:53:35.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2d240019f0 con 0x7f2d2c0cb690 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 >> v1:172.21.15.104:6789/0 conn(0x7f2d2c0cb690 legacy=0x7f2d2c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d14003a00 0x7f2d14003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14004d90 0x7f2d14005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d0510 con 0x7f2d14003a00 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c06a5e0 con 0x7f2d14004d90 2023-06-10T18:53:35.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1400ba10 con 0x7f2d2c069e70 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d14003a00 0x7f2d14003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2affd700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14004d90 0x7f2d14005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 -- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 msgr2=0x7f2d14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c069e70 0x7f2d14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 -- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14004d90 msgr2=0x7f2d14005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14004d90 0x7f2d14005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 -- 172.21.15.104:0/2133337624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d140035b0 con 0x7f2d14003a00 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14004d90 0x7f2d14005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.272+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d14003a00 0x7f2d14003e10 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009b40 tx=0x7f2d14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6323bbaa715deb13 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20005a20 con 0x7f2d14003a00 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20005b80 con 0x7f2d14003a00 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d2a7fc700 1 -- 172.21.15.104:0/2133337624 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2000e6c0 con 0x7f2d14003a00 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 -- 172.21.15.104:0/2133337624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d14003a00 msgr2=0x7f2d14003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/2133337624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d14003a00 0x7f2d14003e10 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009b40 tx=0x7f2d14005880 comp rx=0 tx=0).stop 2023-06-10T18:53:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d2bfff700 1 -- 172.21.15.104:0/2133337624 reap_dead start 2023-06-10T18:53:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 -- 172.21.15.104:0/2133337624 shutdown_connections 2023-06-10T18:53:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 -- 172.21.15.104:0/2133337624 >> 172.21.15.104:0/2133337624 conn(0x7f2d2c06b8a0 msgr2=0x7f2d2c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 -- 172.21.15.104:0/2133337624 shutdown_connections 2023-06-10T18:53:35.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.273+0000 7f2d31bb5700 1 -- 172.21.15.104:0/2133337624 wait complete. 2023-06-10T18:53:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.274+0000 7f2d31bb5700 1 Processor -- start 2023-06-10T18:53:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 -- start start 2023-06-10T18:53:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14003a00 0x7f2d2c153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c15c670 0x7f2d2c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c15d560 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b0230 con 0x7f2d14003a00 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d31bb5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b04f0 con 0x7f2d2c15c670 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34558/0 (socket says 172.21.15.104:34558) 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d2affd700 1 -- 172.21.15.104:0/3004816710 learned_addr learned my addr 172.21.15.104:0/3004816710 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.275+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14003a00 0x7f2d2c153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c15c670 0x7f2d2c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 -- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c15c670 msgr2=0x7f2d2c0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c15c670 0x7f2d2c0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 -- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14003a00 msgr2=0x7f2d2c153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14003a00 0x7f2d2c153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 -- 172.21.15.104:0/3004816710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0b07d0 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d2affd700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c011660 tx=0x7f2d1c011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d13fff700 1 -- 172.21.15.104:0/3004816710 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c015930 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d13fff700 1 -- 172.21.15.104:0/3004816710 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c015a90 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c0b0ab0 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d13fff700 1 -- 172.21.15.104:0/3004816710 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c015e00 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.276+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c153dd0 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.278+0000 7f2d13fff700 1 -- 172.21.15.104:0/3004816710 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2d1c0143e0 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.278+0000 7f2d13fff700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 0x7f2d1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.278+0000 7f2d13fff700 1 -- 172.21.15.104:0/3004816710 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2d1c089920 con 0x7f2d2c0b12b0 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.278+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 0x7f2d1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.279+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 0x7f2d1806d190 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c15af90 tx=0x7f2d20008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.280+0000 7f2d3dd3ab80 1 Processor -- start 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.280+0000 7f2d3dd3ab80 1 -- start start 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x556981579320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x55698159f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698147dc60 con 0x556981576740 2023-06-10T18:53:35.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569814d09f0 con 0x556981578f50 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569814c9e80 con 0x5569815a0d80 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34568/0 (socket says 172.21.15.104:34568) 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 -- 172.21.15.104:0/4188521634 learned_addr learned my addr 172.21.15.104:0/4188521634 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x556981579320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2affd700 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x55698159f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 -- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 msgr2=0x55698159f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x55698159f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 -- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 msgr2=0x556981579320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x556981579320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.281+0000 7f2d2bfff700 1 -- 172.21.15.104:0/4188521634 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55698134b5e0 con 0x556981576740 2023-06-10T18:53:35.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0c9b70 tx=0x7f2d24007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a8a4ae57055253 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d117fa700 1 -- 172.21.15.104:0/4188521634 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24019070 con 0x556981576740 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d117fa700 1 -- 172.21.15.104:0/4188521634 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24007ab0 con 0x556981576740 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d117fa700 1 -- 172.21.15.104:0/4188521634 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24015470 con 0x556981576740 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/4188521634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 msgr2=0x5569815a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0c9b70 tx=0x7f2d24007330 comp rx=0 tx=0).stop 2023-06-10T18:53:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/4188521634 shutdown_connections 2023-06-10T18:53:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x556981579320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981576740 0x5569815a3440 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/4188521634 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x55698159f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/4188521634 >> 172.21.15.104:0/4188521634 conn(0x55698156dfc0 msgr2=0x5569815be1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/4188521634 shutdown_connections 2023-06-10T18:53:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.282+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/4188521634 wait complete. 2023-06-10T18:53:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.283+0000 7f2d3dd3ab80 1 -- start start 2023-06-10T18:53:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.283+0000 7f2d3dd3ab80 1 Processor -- start 2023-06-10T18:53:35.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x556981599880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x556981572220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556981469a70 con 0x556981578f50 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x556981469770 con 0x556981576740 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55698134d8d0 con 0x5569815a0d80 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 10 client.0 _ll_register_callbacks cb 0x55698146f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34572/0 (socket says 172.21.15.104:34572) 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 -- 172.21.15.104:0/1625678956 learned_addr learned my addr 172.21.15.104:0/1625678956 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x556981572220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x556981599880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 -- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 msgr2=0x556981572220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x556981572220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 -- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 msgr2=0x556981599880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x556981599880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 -- 172.21.15.104:0/1625678956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2400f040 con 0x556981578f50 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x556981572220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x556981599880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d2affd700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c054ae0 tx=0x7f2d1c057db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1674d2324794ea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d0a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c05c360 con 0x556981578f50 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2d1c0543d0 con 0x556981578f50 2023-06-10T18:53:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.284+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55698156b780 con 0x556981578f50 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d1c0545b0 con 0x556981578f50 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 10 client.8133 fetch_fsmap learned FSMap version 55 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x5569812f7d90 con 0x556981578f50 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55698156b780 con 0x556981578f50 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d1c05e0b0 con 0x556981578f50 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 10 client.8133 fetch_fsmap finished waiting for FSMap version 55 2023-06-10T18:53:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 10 client.8133 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 20 client.8133 populate_metadata read hostname 'smithi104' 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x5569815d0d60 0x5569815d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 4 client.8133 mds_command: new command op to 7983 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 --> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569814c2e60 con 0x5569815d0d60 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.285+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x5569815d0d60 0x5569815d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d0a7fc700 10 client.8133 ms_handle_connect on v2:172.21.15.114:6835/3243355534 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x5569815d0d60 0x5569815d1130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dffe0891bdd4a674 server_cookie=e62d322729be3283 in_seq=0 out_seq=0 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/1625678956 <== mds.0 v2:172.21.15.114:6835/3243355534 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5569814c2e60 con 0x5569815d0d60 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d0a7fc700 10 client.8133 handle_command_reply: tid=0 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 client.8133 shutdown 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 msgr2=0x5569815a6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c054ae0 tx=0x7f2d1c057db0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 shutdown_connections 2023-06-10T18:53:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x5569815d0d60 0x5569815d1130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x556981599880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5569815a0d80 0x556981572220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d0a7fc700 0 client.8133 ms_handle_reset on v2:172.21.15.114:6835/3243355534 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/1625678956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556981578f50 0x5569815a6e80 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 >> 172.21.15.104:0/1625678956 conn(0x55698156dfc0 msgr2=0x55698159ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 shutdown_connections 2023-06-10T18:53:35.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.286+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/1625678956 wait complete. 2023-06-10T18:53:35.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.287+0000 7f2d3dd3ab80 20 client.8133 trim_cache size 0 max 16384 2023-06-10T18:53:35.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.304+0000 7f2d3dd3ab80 1 Processor -- start 2023-06-10T18:53:35.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.304+0000 7f2d3dd3ab80 1 -- start start 2023-06-10T18:53:35.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981576740 0x5569815c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x55698159ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698159fdf0 con 0x5569815a0d80 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569815990d0 con 0x556981578f50 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569815bf9c0 con 0x556981576740 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34582/0 (socket says 172.21.15.104:34582) 2023-06-10T18:53:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 -- 172.21.15.104:0/48118037 learned_addr learned my addr 172.21.15.104:0/48118037 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2affd700 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x55698159ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981576740 0x5569815c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 -- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981576740 msgr2=0x5569815c21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981576740 0x5569815c21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 -- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 msgr2=0x55698159ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x55698159ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 -- 172.21.15.104:0/48118037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569812e0c10 con 0x5569815a0d80 2023-06-10T18:53:35.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2d24001ba0 tx=0x7f2d2400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7136cdd6c52e4186 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d11ffb700 1 -- 172.21.15.104:0/48118037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24019040 con 0x5569815a0d80 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d11ffb700 1 -- 172.21.15.104:0/48118037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24015470 con 0x5569815a0d80 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.305+0000 7f2d11ffb700 1 -- 172.21.15.104:0/48118037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24015820 con 0x5569815a0d80 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/48118037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 msgr2=0x55698159bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2d24001ba0 tx=0x7f2d2400e000 comp rx=0 tx=0).stop 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/48118037 shutdown_connections 2023-06-10T18:53:35.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159bfa0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981576740 0x5569815c21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/48118037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981578f50 0x55698159ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/48118037 >> 172.21.15.104:0/48118037 conn(0x55698156dfc0 msgr2=0x5569815d7160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/48118037 shutdown_connections 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/48118037 wait complete. 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 -- start start 2023-06-10T18:53:35.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.306+0000 7f2d3dd3ab80 1 Processor -- start 2023-06-10T18:53:35.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x5569815b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 0x55698159d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556981625b80 con 0x5569815a0d80 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698159b730 con 0x556981576740 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x556981624380 con 0x556981578f50 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d3dd3ab80 10 client.0 _ll_register_callbacks cb 0x55698146f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:53:35.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34588/0 (socket says 172.21.15.104:34588) 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d2bfff700 1 -- 172.21.15.104:0/59932713 learned_addr learned my addr 172.21.15.104:0/59932713 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d2affd700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 0x55698159d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.307+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x5569815b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 -- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 msgr2=0x55698159d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 0x55698159d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 -- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 msgr2=0x5569815b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x5569815b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 -- 172.21.15.104:0/59932713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55698134b5e0 con 0x5569815a0d80 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2affd700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 0x55698159d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2bfff700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2d24022040 tx=0x7f2d24001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4307ca7c5cdf4732 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:53:35.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24019040 con 0x5569815a0d80 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2d24028730 con 0x5569815a0d80 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d24007ab0 con 0x5569815a0d80 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d240288f0 con 0x5569815a0d80 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 10 client.8139 fetch_fsmap learned FSMap version 55 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x55698131e200 con 0x5569815a0d80 2023-06-10T18:53:35.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24007ab0 con 0x5569815a0d80 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d24028a50 con 0x5569815a0d80 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 10 client.8139 fetch_fsmap finished waiting for FSMap version 55 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 10 client.8139 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 20 client.8139 populate_metadata read hostname 'smithi104' 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x556981624a60 0x556981624e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 4 client.8139 mds_command: new command op to 7983 tid=0[{"prefix": "status"}] 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 --> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556981485000 con 0x556981624a60 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.308+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x556981624a60 0x556981624e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d0a7fc700 10 client.8139 ms_handle_connect on v2:172.21.15.114:6835/3243355534 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d2b7fe700 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x556981624a60 0x556981624e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5cdc98e3bbd4fc server_cookie=35c11eca5c954079 in_seq=0 out_seq=0 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d0a7fc700 1 -- 172.21.15.104:0/59932713 <== mds.0 v2:172.21.15.114:6835/3243355534 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x556981485000 con 0x556981624a60 2023-06-10T18:53:35.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d0a7fc700 10 client.8139 handle_command_reply: tid=0 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 client.8139 shutdown 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 msgr2=0x55698159d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2d24022040 tx=0x7f2d24001df0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 shutdown_connections 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:6835/3243355534,v1:172.21.15.114:6837/3243355534] conn(0x556981624a60 0x556981624e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556981576740 0x5569815b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d0a7fc700 0 client.8139 ms_handle_reset on v2:172.21.15.114:6835/3243355534 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5569815a0d80 0x55698159d800 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 --2- 172.21.15.104:0/59932713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556981578f50 0x55698159d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 >> 172.21.15.104:0/59932713 conn(0x55698156dfc0 msgr2=0x5569815755b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 shutdown_connections 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 1 -- 172.21.15.104:0/59932713 wait complete. 2023-06-10T18:53:35.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.309+0000 7f2d3dd3ab80 20 client.8139 trim_cache size 0 max 16384 2023-06-10T18:53:35.327 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:53:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:53:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:53:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 7983, 2023-06-10T18:53:35.327 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8600705739999999, 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 55, 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 99, 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 99, 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.316106156 2023-06-10T18:53:35.328 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:53:35.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 msgr2=0x7f2d1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 0x7f2d1806d190 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c15af90 tx=0x7f2d20008030 comp rx=0 tx=0).stop 2023-06-10T18:53:35.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 msgr2=0x7f2d2c15c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c011660 tx=0x7f2d1c011970 comp rx=0 tx=0).stop 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 shutdown_connections 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d1806ace0 0x7f2d1806d190 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d14003a00 0x7f2d2c153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d2c15c670 0x7f2d2c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 --2- 172.21.15.104:0/3004816710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d2c0b12b0 0x7f2d2c15c130 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 >> 172.21.15.104:0/3004816710 conn(0x7f2d2c06b8a0 msgr2=0x7f2d2c0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 shutdown_connections 2023-06-10T18:53:35.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.310+0000 7f2d31bb5700 1 -- 172.21.15.104:0/3004816710 wait complete. 2023-06-10T18:53:35.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:53:35.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 Processor -- start 2023-06-10T18:53:35.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 -- start start 2023-06-10T18:53:35.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc68c06ea70 0x7fc68c06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc68c0aec00 0x7fc68c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c06e5e0 con 0x7fc68c0b1070 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68c0ae7e0 con 0x7fc68c06ea70 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.448+0000 7fc692254700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68c06df90 con 0x7fc68c0aec00 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc68c06ea70 0x7fc68c06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc68c06ea70 0x7fc68c06cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34594/0 (socket says 172.21.15.104:34594) 2023-06-10T18:53:35.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 -- 172.21.15.104:0/2848190403 learned_addr learned my addr 172.21.15.104:0/2848190403 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68bfff700 1 --2- 172.21.15.104:0/2848190403 >> v2:172.21.15.114:3300/0 conn(0x7fc68c0aec00 0x7fc68c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 -- 172.21.15.104:0/2848190403 >> v2:172.21.15.114:3300/0 conn(0x7fc68c0aec00 msgr2=0x7fc68c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 --2- 172.21.15.104:0/2848190403 >> v2:172.21.15.114:3300/0 conn(0x7fc68c0aec00 0x7fc68c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 -- 172.21.15.104:0/2848190403 >> v1:172.21.15.114:6789/0 conn(0x7fc68c0b1070 legacy=0x7fc68c0b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 -- 172.21.15.104:0/2848190403 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c06e270 con 0x7fc68c06ea70 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68affd700 1 --2- 172.21.15.104:0/2848190403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c06ea70 0x7fc68c06cde0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0cd0c0 tx=0x7fc680009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbe8e0cfbc813b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68a7fc700 1 -- 172.21.15.104:0/2848190403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc680004230 con 0x7fc68c06ea70 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68a7fc700 1 -- 172.21.15.104:0/2848190403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc680004390 con 0x7fc68c06ea70 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.449+0000 7fc68a7fc700 1 -- 172.21.15.104:0/2848190403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6800036a0 con 0x7fc68c06ea70 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 -- 172.21.15.104:0/2848190403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c06ea70 msgr2=0x7fc68c06cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 --2- 172.21.15.104:0/2848190403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c06ea70 0x7fc68c06cde0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0cd0c0 tx=0x7fc680009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 -- 172.21.15.104:0/2848190403 shutdown_connections 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 --2- 172.21.15.104:0/2848190403 >> v2:172.21.15.114:3300/0 conn(0x7fc68c0aec00 0x7fc68c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 --2- 172.21.15.104:0/2848190403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c06ea70 0x7fc68c06cde0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 -- 172.21.15.104:0/2848190403 >> 172.21.15.104:0/2848190403 conn(0x7fc68c06b8a0 msgr2=0x7fc68c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 -- 172.21.15.104:0/2848190403 shutdown_connections 2023-06-10T18:53:35.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.450+0000 7fc692254700 1 -- 172.21.15.104:0/2848190403 wait complete. 2023-06-10T18:53:35.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.451+0000 7fc692254700 1 Processor -- start 2023-06-10T18:53:35.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.451+0000 7fc692254700 1 -- start start 2023-06-10T18:53:35.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 0x7fc68c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 0x7fc68c1512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c1f1ef0 con 0x7fc68c0b1070 2023-06-10T18:53:35.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c153ff0 con 0x7fc68c0aec00 2023-06-10T18:53:35.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc692254700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c1542d0 con 0x7fc68c06ea70 2023-06-10T18:53:35.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34602/0 (socket says 172.21.15.104:34602) 2023-06-10T18:53:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 -- 172.21.15.104:0/1760846375 learned_addr learned my addr 172.21.15.104:0/1760846375 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68b7fe700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 0x7fc68c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68affd700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 0x7fc68c1512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 -- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 msgr2=0x7fc68c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 0x7fc68c0b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 -- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 msgr2=0x7fc68c1512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 0x7fc68c1512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 -- 172.21.15.104:0/1760846375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680009550 con 0x7fc68c0b1070 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68b7fe700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 0x7fc68c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68affd700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 0x7fc68c1512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:35.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc68bfff700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fc684007d70 tx=0x7fc68400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.452+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68400ede0 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.453+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc684014420 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.453+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c1545b0 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.453+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6840147d0 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.453+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c0b4200 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.453+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c074050 con 0x7fc68c0b1070 2023-06-10T18:53:35.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.454+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc684014930 con 0x7fc68c0b1070 2023-06-10T18:53:35.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.454+0000 7fc673fff700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 0x7fc67806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:35.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.454+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc684089da0 con 0x7fc68c0b1070 2023-06-10T18:53:35.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.455+0000 7fc68b7fe700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 0x7fc67806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:35.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.455+0000 7fc68b7fe700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 0x7fc67806d190 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc67c007fd0 tx=0x7fc67c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:35.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.459+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc684054c90 con 0x7fc68c0b1070 2023-06-10T18:53:35.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.690+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc68c0ceb70 con 0x7fc68c0b1070 2023-06-10T18:53:35.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.691+0000 7fc673fff700 1 -- 172.21.15.104:0/1760846375 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc6840582b0 con 0x7fc68c0b1070 2023-06-10T18:53:35.696 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:53:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 msgr2=0x7fc67806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 0x7fc67806d190 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc67c007fd0 tx=0x7fc67c009040 comp rx=0 tx=0).stop 2023-06-10T18:53:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 msgr2=0x7fc68c0c3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fc684007d70 tx=0x7fc68400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 shutdown_connections 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc68c0b1070 0x7fc68c0c3380 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc68c06ea70 0x7fc68c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc67806ace0 0x7fc67806d190 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 --2- 172.21.15.104:0/1760846375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc68c0aec00 0x7fc68c1512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 >> 172.21.15.104:0/1760846375 conn(0x7fc68c06b8a0 msgr2=0x7fc68c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 shutdown_connections 2023-06-10T18:53:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:35.693+0000 7fc692254700 1 -- 172.21.15.104:0/1760846375 wait complete. 2023-06-10T18:53:35.709 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:53:35.710 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:53:35.710 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:53:35.710 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:53:35.711 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:35.711 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:53:35.711 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:53:35.711 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:53:35.712 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:53:35.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.713 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:53:35.730 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:53:35.730 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:35.731 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:35.732 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41869sec preferred_lft 41869sec 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:53:35.733 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state DOWN group default qlen 1000 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:53:35.734 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:35.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.735 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:53:35.795 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:53:35.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.797 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:53:35.855 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:53:35.855 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:53:35.856 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:53:35.856 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:53:35.856 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:53:35.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.857 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:53:35.857 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:53:35.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:53:35.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:53:35.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:53:35.860 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:53:35.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.960 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:53:35.977 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:53:35.977 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:53:35.978 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:53:35.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:35.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:53:36.050 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:53:36.051 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:36.051 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:36.106 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:53:36.106 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:36.107 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:36.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:53:36.229 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:36.229 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:36.294 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:36.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:36.296 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:53:36.352 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:53:36.352 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:53:36.394 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:53:36.420 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:36.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:36.487 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:36.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:36.489 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:53:36.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:53:36.805+0000 7fc0237655c0 -1 init, newargv = 0x55dc76f3aca0 newargc=15 2023-06-10T18:53:36.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[138650]: starting ceph client 2023-06-10T18:53:36.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[138650]: starting fuse 2023-06-10T18:53:37.646 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:53:37.680 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:37.681 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:37.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:37.752 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:53:37.809 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:53:37.810 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:53:37.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:53:37.811 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:53:37.811 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:53:37.811 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:53:37.811 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:53:37.812 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:37.812 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:53:37.812 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:53:37.813 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:53:37.813 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:53:37.813 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:37.813 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:53:37.814 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:53:37.814 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:53:37.814 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:53:37.815 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:37.815 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:53:37.815 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:53:37.815 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:53:37.816 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:53:37.816 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:53:37.816 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:53:37.817 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:53:37.817 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:53:37.817 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:37.818 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:53:37.818 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:53:37.935 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.138650.asok 2023-06-10T18:53:38.118 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.138650.asok 2023-06-10T18:53:38.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.119 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.138650.asok status 2023-06-10T18:53:38.275 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:53:38.276 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "138650", 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 8151, 2023-06-10T18:53:38.277 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 8151 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:38.278 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 413723751 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 413723751 2023-06-10T18:53:38.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.8151 192.168.0.1:0/413723751", 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/413723751", 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 55, 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 99, 2023-06-10T18:53:38.280 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:53:38.281 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:53:38.281 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:53:38.281 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:53:38.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.286 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:53:38.286 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:38.303 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:53:38.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:38.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:38.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:53:38.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.525+0000 7f8f61e73700 1 Processor -- start 2023-06-10T18:53:38.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.525+0000 7f8f61e73700 1 -- start start 2023-06-10T18:53:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f61e73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f5c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f61e73700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c0b6070 con 0x7f8f5c15a420 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f61e73700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c0b1d70 con 0x7f8f5c0ce890 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f61e73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0ced10 con 0x7f8f5c0b4ff0 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8f5c15a420 0x7f8f5c0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60678/0 (socket says 172.21.15.104:60678) 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5b7fe700 1 -- 172.21.15.104:0/3151071935 learned_addr learned my addr 172.21.15.104:0/3151071935 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286325816 0 0) 0x7f8f5c0b6070 con 0x7f8f5c15a420 2023-06-10T18:53:38.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5affd700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f5c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f440035b0 con 0x7f8f5c15a420 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529839257 0 0) 0x7f8f440035b0 con 0x7f8f5c15a420 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 msgr2=0x7f8f5c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f5c0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 >> v1:172.21.15.114:6789/0 conn(0x7f8f5c0ce890 legacy=0x7f8f5c15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0b5460 con 0x7f8f5c15a420 2023-06-10T18:53:38.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8f50001670 con 0x7f8f5c15a420 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.526+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 >> v1:172.21.15.104:6789/0 conn(0x7f8f5c15a420 legacy=0x7f8f5c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f44003a30 0x7f8f44003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44004d90 0x7f8f44005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0ced10 con 0x7f8f44003a30 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400bac0 con 0x7f8f44004d90 2023-06-10T18:53:38.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400bc50 con 0x7f8f5c0b4ff0 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f44003a30 0x7f8f44003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5affd700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5bfff700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44004d90 0x7f8f44005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 -- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 msgr2=0x7f8f44004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f5c0b4ff0 0x7f8f44004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 -- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44004d90 msgr2=0x7f8f44005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44004d90 0x7f8f44005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 -- 172.21.15.104:0/3151071935 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440035b0 con 0x7f8f44003a30 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5bfff700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44004d90 0x7f8f44005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f44003a30 0x7f8f44003e40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000c240 tx=0x7f8f44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bab859ec4f55aff server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50007c10 con 0x7f8f44003a30 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000cde0 con 0x7f8f44003a30 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.527+0000 7f8f5a7fc700 1 -- 172.21.15.104:0/3151071935 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50005b80 con 0x7f8f44003a30 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 -- 172.21.15.104:0/3151071935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f44003a30 msgr2=0x7f8f44003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 --2- 172.21.15.104:0/3151071935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f44003a30 0x7f8f44003e40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000c240 tx=0x7f8f44005880 comp rx=0 tx=0).stop 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f5bfff700 1 -- 172.21.15.104:0/3151071935 reap_dead start 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 -- 172.21.15.104:0/3151071935 shutdown_connections 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 -- 172.21.15.104:0/3151071935 >> 172.21.15.104:0/3151071935 conn(0x7f8f5c089fd0 msgr2=0x7f8f5c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 -- 172.21.15.104:0/3151071935 shutdown_connections 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.528+0000 7f8f61e73700 1 -- 172.21.15.104:0/3151071935 wait complete. 2023-06-10T18:53:38.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.529+0000 7f8f61e73700 1 Processor -- start 2023-06-10T18:53:38.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.529+0000 7f8f61e73700 1 -- start start 2023-06-10T18:53:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 0x7f8f5c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5c1efad0 0x7f8f5c1dfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0c8c30 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c1e0960 con 0x7f8f5c1efad0 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0c8300 con 0x7f8f44003a30 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34634/0 (socket says 172.21.15.104:34634) 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 -- 172.21.15.104:0/941855911 learned_addr learned my addr 172.21.15.104:0/941855911 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 0x7f8f5c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5affd700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5c1efad0 0x7f8f5c1dfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 -- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 msgr2=0x7f8f5c1f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 0x7f8f5c1f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 -- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5c1efad0 msgr2=0x7f8f5c1dfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5c1efad0 0x7f8f5c1dfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 -- 172.21.15.104:0/941855911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0c85e0 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 0x7f8f5c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f5bfff700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c08aaf0 tx=0x7f8f5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54003c10 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54003d90 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c0c88c0 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.530+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54005310 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.531+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c1eff40 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.531+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c074050 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.532+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8f54005470 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.532+0000 7f8f43fff700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 0x7f8f4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:38.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.532+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8f54089600 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.532+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 0x7f8f4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.533+0000 7f8f5b7fe700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 0x7f8f4806d190 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f50015040 tx=0x7f8f50007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:38.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.537+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8f54054570 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.766+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8f5c1df980 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.766+0000 7f8f43fff700 1 -- 172.21.15.104:0/941855911 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8f54057b90 con 0x7f8f5c1e03e0 2023-06-10T18:53:38.771 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:53:38.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 msgr2=0x7f8f4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 0x7f8f4806d190 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f50015040 tx=0x7f8f50007000 comp rx=0 tx=0).stop 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 msgr2=0x7f8f5c0cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c08aaf0 tx=0x7f8f5400ee70 comp rx=0 tx=0).stop 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 shutdown_connections 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f5c1e03e0 0x7f8f5c0cb010 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f44003a30 0x7f8f5c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f4806ace0 0x7f8f4806d190 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 --2- 172.21.15.104:0/941855911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5c1efad0 0x7f8f5c1dfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:38.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 >> 172.21.15.104:0/941855911 conn(0x7f8f5c089fd0 msgr2=0x7f8f5c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:38.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 shutdown_connections 2023-06-10T18:53:38.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:38.768+0000 7f8f61e73700 1 -- 172.21.15.104:0/941855911 wait complete. 2023-06-10T18:53:38.785 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:53:38.785 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:53:38.786 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:53:38.786 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:53:38.786 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:38.787 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:53:38.787 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:53:38.787 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:53:38.787 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:38.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.788 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:53:38.809 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:53:38.810 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:53:38.810 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:53:38.810 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.810 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:53:38.810 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:38.811 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41862sec preferred_lft 41862sec 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.812 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:53:38.813 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:53:38.814 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:53:38.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.815 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:53:38.879 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:53:38.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.881 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:53:38.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:53:38.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:53:38.942 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:53:38.942 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:53:38.942 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:53:38.943 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:53:38.943 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:53:38.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:38.943 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:53:38.944 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:53:38.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:53:38.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:53:38.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:53:38.945 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:53:39.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:39.051 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:53:39.107 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:53:39.108 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:53:39.108 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:39.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:39.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:53:39.174 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:53:39.175 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:39.175 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:39.231 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:53:39.231 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:53:39.232 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:39.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:53:39.354 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:39.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:39.420 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:39.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:39.421 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:53:39.477 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:53:39.478 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:53:39.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:53:39.545 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:53:39.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:53:39.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:53:39.600+0000 7f47dfde15c0 -1 init, newargv = 0x55af1bb99800 newargc=15 2023-06-10T18:53:39.606 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[112756]: starting ceph client 2023-06-10T18:53:39.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[112756]: starting fuse 2023-06-10T18:53:39.611 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:53:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:53:39.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:39.614 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:53:39.670 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:53:39.670 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:53:39.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:53:39.671 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:53:39.672 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:53:39.673 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:53:39.673 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:53:39.674 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:39.674 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:53:39.674 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:53:39.675 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:53:39.675 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:53:39.675 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:39.676 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:53:39.676 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:53:39.676 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:53:39.677 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:53:39.677 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:39.677 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:53:39.677 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:53:39.678 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:53:39.678 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:53:39.678 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:53:39.679 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:53:39.679 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:53:39.679 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:53:39.680 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:53:39.680 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:53:39.681 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:53:39.792 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.112756.asok 2023-06-10T18:53:39.975 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.112756.asok 2023-06-10T18:53:39.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:39.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.112756.asok status 2023-06-10T18:53:40.130 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:53:40.131 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:53:40.131 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:53:40.131 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:53:40.131 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:53:40.131 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:53:40.132 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:53:40.132 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "112756", 2023-06-10T18:53:40.132 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:53:40.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:40.132 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 7120, 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 7120 2023-06-10T18:53:40.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1544708801 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:40.134 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1544708801 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.7120 192.168.0.1:0/1544708801", 2023-06-10T18:53:40.135 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1544708801", 2023-06-10T18:53:40.136 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:53:40.136 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 55, 2023-06-10T18:53:40.136 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 99, 2023-06-10T18:53:40.136 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:53:40.136 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:53:40.137 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:53:40.137 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:53:40.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:40.142 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:53:40.143 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:40.160 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:53:40.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:40.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:40.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:40.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:53:40.373 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:53:40.374 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:53:40.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:53:40.508 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:53:40.508 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:53:40.509 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:40.509 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:40.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:40.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.641+0000 7eff7cc7e700 1 Processor -- start 2023-06-10T18:53:40.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.642+0000 7eff7cc7e700 1 -- start start 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7cc7e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff780cf100 0x7eff7806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7cc7e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7806d290 con 0x7eff7806e0e0 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7cc7e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff780ca2d0 con 0x7eff7806e4b0 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7cc7e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff78071640 con 0x7eff780cf100 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7eff7806e0e0 0x7eff780c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60698/0 (socket says 172.21.15.104:60698) 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff7659c700 1 -- 172.21.15.104:0/740563935 learned_addr learned my addr 172.21.15.104:0/740563935 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:40.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff76d9d700 1 --2- 172.21.15.104:0/740563935 >> v2:172.21.15.114:3300/0 conn(0x7eff780cf100 0x7eff7806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff76d9d700 1 -- 172.21.15.104:0/740563935 >> v1:172.21.15.114:6789/0 conn(0x7eff7806e4b0 legacy=0x7eff780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff76d9d700 1 -- 172.21.15.104:0/740563935 >> v1:172.21.15.104:6789/0 conn(0x7eff7806e0e0 legacy=0x7eff780c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff76d9d700 1 -- 172.21.15.104:0/740563935 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7806d4c0 con 0x7eff780cf100 2023-06-10T18:53:40.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.643+0000 7eff76d9d700 1 --2- 172.21.15.104:0/740563935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff7806cd50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff70009d90 tx=0x7eff7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd6f19399c08c845 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.644+0000 7eff7559a700 1 -- 172.21.15.104:0/740563935 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7000e070 con 0x7eff780cf100 2023-06-10T18:53:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.644+0000 7eff7559a700 1 -- 172.21.15.104:0/740563935 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7000bbd0 con 0x7eff780cf100 2023-06-10T18:53:40.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.644+0000 7eff7559a700 1 -- 172.21.15.104:0/740563935 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70012430 con 0x7eff780cf100 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 -- 172.21.15.104:0/740563935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 msgr2=0x7eff7806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/740563935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff7806cd50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff70009d90 tx=0x7eff7000b330 comp rx=0 tx=0).stop 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 -- 172.21.15.104:0/740563935 shutdown_connections 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/740563935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff7806cd50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 -- 172.21.15.104:0/740563935 >> 172.21.15.104:0/740563935 conn(0x7eff7806b8a0 msgr2=0x7eff780cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 -- 172.21.15.104:0/740563935 shutdown_connections 2023-06-10T18:53:40.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.645+0000 7eff7cc7e700 1 -- 172.21.15.104:0/740563935 wait complete. 2023-06-10T18:53:40.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.647+0000 7eff7cc7e700 1 Processor -- start 2023-06-10T18:53:40.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.647+0000 7eff7cc7e700 1 -- start start 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff7806e0e0 0x7eff780b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff780af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff780af920 con 0x7eff7806e4b0 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff780afc00 con 0x7eff780cf100 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7cc7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff780afee0 con 0x7eff7806e0e0 2023-06-10T18:53:40.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34644/0 (socket says 172.21.15.104:34644) 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 -- 172.21.15.104:0/334241714 learned_addr learned my addr 172.21.15.104:0/334241714 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff7659c700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff7806e0e0 0x7eff780b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff76d9d700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff780af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 -- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff7806e0e0 msgr2=0x7eff780b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff7806e0e0 0x7eff780b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 -- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 msgr2=0x7eff780af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff780af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 -- 172.21.15.104:0/334241714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70009a70 con 0x7eff7806e4b0 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.648+0000 7eff75d9b700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c4e0 tx=0x7eff6c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.649+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0052d0 con 0x7eff7806e4b0 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.649+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c005430 con 0x7eff7806e4b0 2023-06-10T18:53:40.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.649+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff780b01c0 con 0x7eff7806e4b0 2023-06-10T18:53:40.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.649+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0038c0 con 0x7eff7806e4b0 2023-06-10T18:53:40.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.649+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff780b17d0 con 0x7eff7806e4b0 2023-06-10T18:53:40.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.650+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff780b1990 con 0x7eff7806e4b0 2023-06-10T18:53:40.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.650+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff6c003a20 con 0x7eff7806e4b0 2023-06-10T18:53:40.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.651+0000 7eff66ffd700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 0x7eff5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.651+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7eff6c088d70 con 0x7eff7806e4b0 2023-06-10T18:53:40.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.651+0000 7eff7659c700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 0x7eff5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.652+0000 7eff7659c700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 0x7eff5806d190 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff68007fd0 tx=0x7eff68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.655+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff6c053d40 con 0x7eff7806e4b0 2023-06-10T18:53:40.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.841+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff780b06e0 con 0x7eff7806e4b0 2023-06-10T18:53:40.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.842+0000 7eff66ffd700 1 -- 172.21.15.104:0/334241714 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+18543 (secure 0 0 0) 0x7eff6c088070 con 0x7eff7806e4b0 2023-06-10T18:53:40.847 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:40.847 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:30.449397+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 msgr2=0x7eff5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 0x7eff5806d190 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7eff68007fd0 tx=0x7eff68009040 comp rx=0 tx=0).stop 2023-06-10T18:53:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 msgr2=0x7eff780aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c4e0 tx=0x7eff6c00c7f0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 shutdown_connections 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff5806ace0 0x7eff5806d190 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff780cf100 0x7eff780af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff7806e0e0 0x7eff780b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 --2- 172.21.15.104:0/334241714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff7806e4b0 0x7eff780aecd0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 >> 172.21.15.104:0/334241714 conn(0x7eff7806b8a0 msgr2=0x7eff780c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 shutdown_connections 2023-06-10T18:53:40.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.844+0000 7eff7cc7e700 1 -- 172.21.15.104:0/334241714 wait complete. 2023-06-10T18:53:40.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:40.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:40.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.979+0000 7f427c174700 1 Processor -- start 2023-06-10T18:53:40.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.979+0000 7f427c174700 1 -- start start 2023-06-10T18:53:40.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427c174700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427415c580 0x7f42740b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427c174700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f427406cde0 0x7f42740ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427c174700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427406e250 con 0x7f427406e6e0 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427c174700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427415c950 con 0x7f427406cde0 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427c174700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42740b1e80 con 0x7f427415c580 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f427406cde0 0x7f42740ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f427406cde0 0x7f42740ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34654/0 (socket says 172.21.15.104:34654) 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 -- 172.21.15.104:0/3546223148 learned_addr learned my addr 172.21.15.104:0/3546223148 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 -- 172.21.15.104:0/3546223148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427415c580 msgr2=0x7f42740b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 --2- 172.21.15.104:0/3546223148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427415c580 0x7f42740b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 -- 172.21.15.104:0/3546223148 >> v1:172.21.15.114:6789/0 conn(0x7f427406e6e0 legacy=0x7f42740b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 -- 172.21.15.104:0/3546223148 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427406d920 con 0x7f427406cde0 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f427a711700 1 --2- 172.21.15.104:0/3546223148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427406cde0 0x7f42740ce0d0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f426c009e50 tx=0x7f426c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159b5adbbe672f69 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f4278f0e700 1 -- 172.21.15.104:0/3546223148 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c00e070 con 0x7f427406cde0 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f4278f0e700 1 -- 172.21.15.104:0/3546223148 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c003a20 con 0x7f427406cde0 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.980+0000 7f4278f0e700 1 -- 172.21.15.104:0/3546223148 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c003dd0 con 0x7f427406cde0 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 -- 172.21.15.104:0/3546223148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427406cde0 msgr2=0x7f42740ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 --2- 172.21.15.104:0/3546223148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427406cde0 0x7f42740ce0d0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f426c009e50 tx=0x7f426c00b330 comp rx=0 tx=0).stop 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 -- 172.21.15.104:0/3546223148 shutdown_connections 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 --2- 172.21.15.104:0/3546223148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427406cde0 0x7f42740ce0d0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 --2- 172.21.15.104:0/3546223148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427415c580 0x7f42740b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 -- 172.21.15.104:0/3546223148 >> 172.21.15.104:0/3546223148 conn(0x7f427406b8a0 msgr2=0x7f42740cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:40.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 -- 172.21.15.104:0/3546223148 shutdown_connections 2023-06-10T18:53:40.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.981+0000 7f427c174700 1 -- 172.21.15.104:0/3546223148 wait complete. 2023-06-10T18:53:40.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.982+0000 7f427c174700 1 Processor -- start 2023-06-10T18:53:40.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.982+0000 7f427c174700 1 -- start start 2023-06-10T18:53:40.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f427406cde0 0x7f42741556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427406e6e0 0x7f42740ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274158630 con 0x7f427415c580 2023-06-10T18:53:40.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274157590 con 0x7f427406cde0 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427c174700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4274157870 con 0x7f427406e6e0 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34660/0 (socket says 172.21.15.104:34660) 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 -- 172.21.15.104:0/2117301140 learned_addr learned my addr 172.21.15.104:0/2117301140 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427970f700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427406e6e0 0x7f42740ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f4279f10700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f427406cde0 0x7f42741556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 -- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427406e6e0 msgr2=0x7f42740ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427406e6e0 0x7f42740ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 -- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f427406cde0 msgr2=0x7f42741556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f427406cde0 0x7f42741556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 -- 172.21.15.104:0/2117301140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c009b30 con 0x7f427415c580 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.983+0000 7f427a711700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f42740b21e0 tx=0x7f426c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.984+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c004240 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.984+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c0043a0 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.984+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4274157b50 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.984+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c004750 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.984+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4274158110 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.985+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4274074050 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.985+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f426c019070 con 0x7f427415c580 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.985+0000 7f426a7fc700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 0x7f425806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:40.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.986+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f426c08a610 con 0x7f427415c580 2023-06-10T18:53:40.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.986+0000 7f4279f10700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 0x7f425806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:40.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.986+0000 7f4279f10700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 0x7f425806d190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f42740ae810 tx=0x7f4264008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:40.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:40.990+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f426c055590 con 0x7f427415c580 2023-06-10T18:53:41.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.178+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4274066f10 con 0x7f427415c580 2023-06-10T18:53:41.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.178+0000 7f426a7fc700 1 -- 172.21.15.104:0/2117301140 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f426c02c300 con 0x7f427415c580 2023-06-10T18:53:41.183 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:41.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 msgr2=0x7f425806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 0x7f425806d190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f42740ae810 tx=0x7f4264008040 comp rx=0 tx=0).stop 2023-06-10T18:53:41.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 msgr2=0x7f42740ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f42740b21e0 tx=0x7f426c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 shutdown_connections 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f425806ace0 0x7f425806d190 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f427406cde0 0x7f42741556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f427415c580 0x7f42740ad750 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 --2- 172.21.15.104:0/2117301140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f427406e6e0 0x7f42740ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 >> 172.21.15.104:0/2117301140 conn(0x7f427406b8a0 msgr2=0x7f42740cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 shutdown_connections 2023-06-10T18:53:41.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.180+0000 7f427c174700 1 -- 172.21.15.104:0/2117301140 wait complete. 2023-06-10T18:53:41.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:41.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 Processor -- start 2023-06-10T18:53:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 -- start start 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7fc15c6f0 0x7fa7fc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7fc0aec00 0x7fa7fc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0ae7e0 con 0x7fa7fc06ea70 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc06df90 con 0x7fa7fc15c6f0 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.314+0000 7fa802812700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc06e270 con 0x7fa7fc0aec00 2023-06-10T18:53:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa7fc06ea70 0x7fa7fc0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60726/0 (socket says 172.21.15.104:60726) 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fb7fe700 1 -- 172.21.15.104:0/3926577121 learned_addr learned my addr 172.21.15.104:0/3926577121 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> v2:172.21.15.114:3300/0 conn(0x7fa7fc15c6f0 0x7fa7fc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa800daf700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7fc0aec00 0x7fa7fc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180138848 0 0) 0x7fa7fc0ae7e0 con 0x7fa7fc06ea70 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e40035b0 con 0x7fa7fc06ea70 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671081212 0 0) 0x7fa7e40035b0 con 0x7fa7fc06ea70 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7fc0aec00 msgr2=0x7fa7fc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7fc0aec00 0x7fa7fc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 >> v2:172.21.15.114:3300/0 conn(0x7fa7fc15c6f0 msgr2=0x7fa7fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 --2- 172.21.15.104:0/3926577121 >> v2:172.21.15.114:3300/0 conn(0x7fa7fc15c6f0 0x7fa7fc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc06e550 con 0x7fa7fc06ea70 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa7f0001670 con 0x7fa7fc06ea70 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 >> v1:172.21.15.104:6789/0 conn(0x7fa7fc06ea70 legacy=0x7fa7fc0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7e4003a00 0x7fa7e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e4004430 0x7fa7e40048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4005000 0x7fa7e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc06df90 con 0x7fa7e4003a00 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc06e270 con 0x7fa7e4004430 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e400bf50 con 0x7fa7e4005000 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> v2:172.21.15.114:3300/0 conn(0x7fa7fc15c6f0 0x7fa7fc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa800daf700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7fc0aec00 0x7fa7fc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fb7fe700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e4004430 0x7fa7e40048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.315+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7e4003a00 0x7fa7e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa800daf700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4005000 0x7fa7e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 -- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4005000 msgr2=0x7fa7e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4005000 0x7fa7e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 -- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e4004430 msgr2=0x7fa7e40048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e4004430 0x7fa7e40048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 -- 172.21.15.104:0/3926577121 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e40035b0 con 0x7fa7e4003a00 2023-06-10T18:53:41.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7e4003a00 0x7fa7e4003e10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec009cf0 tx=0x7fa7e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372a94183c181478 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec004030 con 0x7fa7e4003a00 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec004190 con 0x7fa7e4003a00 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa7faffd700 1 -- 172.21.15.104:0/3926577121 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec004500 con 0x7fa7e4003a00 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa802812700 1 -- 172.21.15.104:0/3926577121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7e4003a00 msgr2=0x7fa7e4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa802812700 1 --2- 172.21.15.104:0/3926577121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7e4003a00 0x7fa7e4003e10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec009cf0 tx=0x7fa7e4005880 comp rx=0 tx=0).stop 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa800daf700 1 -- 172.21.15.104:0/3926577121 reap_dead start 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa802812700 1 -- 172.21.15.104:0/3926577121 shutdown_connections 2023-06-10T18:53:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.316+0000 7fa802812700 1 -- 172.21.15.104:0/3926577121 >> 172.21.15.104:0/3926577121 conn(0x7fa7fc06b8a0 msgr2=0x7fa7fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.317+0000 7fa802812700 1 -- 172.21.15.104:0/3926577121 shutdown_connections 2023-06-10T18:53:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.317+0000 7fa802812700 1 -- 172.21.15.104:0/3926577121 wait complete. 2023-06-10T18:53:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.318+0000 7fa802812700 1 Processor -- start 2023-06-10T18:53:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.318+0000 7fa802812700 1 -- start start 2023-06-10T18:53:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4003a00 0x7fa7fc06a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7fc1f2630 0x7fa7fc155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0b5060 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1f5e60 con 0x7fa7fc1f2630 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1f2aa0 con 0x7fa7e4003a00 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34678/0 (socket says 172.21.15.104:34678) 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 -- 172.21.15.104:0/221661206 learned_addr learned my addr 172.21.15.104:0/221661206 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa800daf700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7fc1f2630 0x7fa7fc155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4003a00 0x7fa7fc06a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 -- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4003a00 msgr2=0x7fa7fc06a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4003a00 0x7fa7fc06a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 -- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7fc1f2630 msgr2=0x7fa7fc155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7fc1f2630 0x7fa7fc155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 -- 172.21.15.104:0/221661206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc1f3000 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7fb7fe700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0007370 tx=0x7fa7f0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f000f6c0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f000f820 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc1f2d20 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f000fbb0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.319+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc1f5af0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.320+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc1f5fd0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.321+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa7f000fd10 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.321+0000 7fa7f8ff9700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 0x7fa7e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.321+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa7f00893c0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.321+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 0x7fa7e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.322+0000 7fa7fbfff700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 0x7fa7e806d190 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0095e0 tx=0x7fa7ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.325+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa7f00543c0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.508+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa7fc066f10 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.508+0000 7fa7f8ff9700 1 -- 172.21.15.104:0/221661206 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v99) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa7f00579e0 con 0x7fa7fc1f62f0 2023-06-10T18:53:41.513 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 msgr2=0x7fa7e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 0x7fa7e806d190 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0095e0 tx=0x7fa7ec008030 comp rx=0 tx=0).stop 2023-06-10T18:53:41.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 msgr2=0x7fa7fc1f20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0007370 tx=0x7fa7f0007680 comp rx=0 tx=0).stop 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 shutdown_connections 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7e806ace0 0x7fa7e806d190 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7fc1f2630 0x7fa7fc155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e4003a00 0x7fa7fc06a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 --2- 172.21.15.104:0/221661206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7fc1f62f0 0x7fa7fc1f20f0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.510+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 >> 172.21.15.104:0/221661206 conn(0x7fa7fc06b8a0 msgr2=0x7fa7fc0ad7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.511+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 shutdown_connections 2023-06-10T18:53:41.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.511+0000 7fa802812700 1 -- 172.21.15.104:0/221661206 wait complete. 2023-06-10T18:53:41.527 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:41.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.645+0000 7fcf03077700 1 Processor -- start 2023-06-10T18:53:41.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.645+0000 7fcf03077700 1 -- start start 2023-06-10T18:53:41.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf03077700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf03077700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0ae7e0 con 0x7fcefc15c6f0 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf03077700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc06df90 con 0x7fcefc06ea70 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf03077700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc06e270 con 0x7fcefc0b1780 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf00e13700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcefc15c6f0 0x7fcefc0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60744/0 (socket says 172.21.15.104:60744) 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcf00e13700 1 -- 172.21.15.104:0/419025460 learned_addr learned my addr 172.21.15.104:0/419025460 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcefbfff700 1 --2- 172.21.15.104:0/419025460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009845169 0 0) 0x7fcefc0ae7e0 con 0x7fcefc15c6f0 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.646+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec0035b0 con 0x7fcefc15c6f0 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefbfff700 1 -- 172.21.15.104:0/419025460 >> v1:172.21.15.114:6789/0 conn(0x7fcefc06ea70 legacy=0x7fcefc06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441027346 0 0) 0x7fceec0035b0 con 0x7fcefc15c6f0 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefbfff700 1 -- 172.21.15.104:0/419025460 >> v1:172.21.15.104:6789/0 conn(0x7fcefc15c6f0 legacy=0x7fcefc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefbfff700 1 -- 172.21.15.104:0/419025460 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc06e550 con 0x7fcefc0b1780 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefbfff700 1 --2- 172.21.15.104:0/419025460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc06cde0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96a6b5a0db4e165 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef00017f0 con 0x7fcefc0b1780 2023-06-10T18:53:41.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee8004030 con 0x7fcefc0b1780 2023-06-10T18:53:41.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.647+0000 7fcefb7fe700 1 -- 172.21.15.104:0/419025460 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80043c0 con 0x7fcefc0b1780 2023-06-10T18:53:41.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 -- 172.21.15.104:0/419025460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 msgr2=0x7fcefc06cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 --2- 172.21.15.104:0/419025460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc06cde0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee8009b80 comp rx=0 tx=0).stop 2023-06-10T18:53:41.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 -- 172.21.15.104:0/419025460 shutdown_connections 2023-06-10T18:53:41.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 --2- 172.21.15.104:0/419025460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc06cde0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 -- 172.21.15.104:0/419025460 >> 172.21.15.104:0/419025460 conn(0x7fcefc06b8a0 msgr2=0x7fcefc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 -- 172.21.15.104:0/419025460 shutdown_connections 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.648+0000 7fcf03077700 1 -- 172.21.15.104:0/419025460 wait complete. 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.649+0000 7fcf03077700 1 Processor -- start 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.649+0000 7fcf03077700 1 -- start start 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc0b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcefc15c6f0 0x7fcefc0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b2550 con 0x7fcefc06ea70 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b06d0 con 0x7fcefc15c6f0 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf03077700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b0980 con 0x7fcefc0b1780 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34690/0 (socket says 172.21.15.104:34690) 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 -- 172.21.15.104:0/3101838367 learned_addr learned my addr 172.21.15.104:0/3101838367 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf01614700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcefc15c6f0 0x7fcefc0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcefbfff700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc0b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 -- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 msgr2=0x7fcefc0b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc0b0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 -- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcefc15c6f0 msgr2=0x7fcefc0c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcefc15c6f0 0x7fcefc0c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 -- 172.21.15.104:0/3101838367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee80094a0 con 0x7fcefc06ea70 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.650+0000 7fcf00e13700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c500 tx=0x7fcef000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.651+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef00074e0 con 0x7fcefc06ea70 2023-06-10T18:53:41.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.651+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0007640 con 0x7fcefc06ea70 2023-06-10T18:53:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.651+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0b0c60 con 0x7fcefc06ea70 2023-06-10T18:53:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.651+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0005a70 con 0x7fcefc06ea70 2023-06-10T18:53:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.651+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc0b1220 con 0x7fcefc06ea70 2023-06-10T18:53:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.652+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc074050 con 0x7fcefc06ea70 2023-06-10T18:53:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.653+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcef0005bd0 con 0x7fcefc06ea70 2023-06-10T18:53:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.653+0000 7fcef97fa700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 0x7fcee006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.653+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcef0088e70 con 0x7fcefc06ea70 2023-06-10T18:53:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.653+0000 7fcefbfff700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 0x7fcee006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.654+0000 7fcefbfff700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 0x7fcee006d190 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0b1b70 tx=0x7fcee8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.657+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcef0053e50 con 0x7fcefc06ea70 2023-06-10T18:53:41.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.840+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcefc0d0030 con 0x7fcefc06ea70 2023-06-10T18:53:41.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.841+0000 7fcef97fa700 1 -- 172.21.15.104:0/3101838367 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcef0017020 con 0x7fcefc06ea70 2023-06-10T18:53:41.846 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:41.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 msgr2=0x7fcee006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 0x7fcee006d190 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0b1b70 tx=0x7fcee8008030 comp rx=0 tx=0).stop 2023-06-10T18:53:41.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 msgr2=0x7fcefc0cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c500 tx=0x7fcef000c810 comp rx=0 tx=0).stop 2023-06-10T18:53:41.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 shutdown_connections 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcee006ace0 0x7fcee006d190 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcefc15c6f0 0x7fcefc0c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc06ea70 0x7fcefc0cb0f0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 --2- 172.21.15.104:0/3101838367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcefc0b1780 0x7fcefc0b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 >> 172.21.15.104:0/3101838367 conn(0x7fcefc06b8a0 msgr2=0x7fcefc1558a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 shutdown_connections 2023-06-10T18:53:41.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.843+0000 7fcf03077700 1 -- 172.21.15.104:0/3101838367 wait complete. 2023-06-10T18:53:41.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:41.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.978+0000 7fb643039700 1 Processor -- start 2023-06-10T18:53:41.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.978+0000 7fb643039700 1 -- start start 2023-06-10T18:53:41.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb643039700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb63c15c580 0x7fb63c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb643039700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb63c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb643039700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63c06e250 con 0x7fb63c06e6e0 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb643039700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63c15c950 con 0x7fb63c15c580 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb643039700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c0b1e80 con 0x7fb63c06cde0 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb63c06e6e0 0x7fb63c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60758/0 (socket says 172.21.15.104:60758) 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63bfff700 1 -- 172.21.15.104:0/634037291 learned_addr learned my addr 172.21.15.104:0/634037291 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb6415d6700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb63c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb640dd5700 1 --2- 172.21.15.104:0/634037291 >> v2:172.21.15.114:3300/0 conn(0x7fb63c15c580 0x7fb63c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298564340 0 0) 0x7fb63c06e250 con 0x7fb63c06e6e0 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c0035b0 con 0x7fb63c06e6e0 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2288876933 0 0) 0x7fb62c0035b0 con 0x7fb63c06e6e0 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 msgr2=0x7fb63c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb63c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 >> v2:172.21.15.114:3300/0 conn(0x7fb63c15c580 msgr2=0x7fb63c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 --2- 172.21.15.104:0/634037291 >> v2:172.21.15.114:3300/0 conn(0x7fb63c15c580 0x7fb63c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c06d920 con 0x7fb63c06e6e0 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb628001670 con 0x7fb63c06e6e0 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 >> v1:172.21.15.104:6789/0 conn(0x7fb63c06e6e0 legacy=0x7fb63c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb62c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb62c003f80 0x7fb62c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb62c004d90 0x7fb62c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c15c950 con 0x7fb62c003f80 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c0b1e80 con 0x7fb62c004d90 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00ba10 con 0x7fb63c06cde0 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb640dd5700 1 --2- 172.21.15.104:0/634037291 >> v2:172.21.15.114:3300/0 conn(0x7fb63c15c580 0x7fb63c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.979+0000 7fb63bfff700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb62c003f80 0x7fb62c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb6415d6700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb62c004d90 0x7fb62c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb640dd5700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb62c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 -- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 msgr2=0x7fb62c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb62c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 -- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb62c004d90 msgr2=0x7fb62c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb62c004d90 0x7fb62c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 -- 172.21.15.104:0/634037291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0035b0 con 0x7fb62c003f80 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb640dd5700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb63c06cde0 0x7fb62c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63bfff700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb62c003f80 0x7fb62c004850 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c2f0 tx=0x7fb62c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1db7d6d3aadcc7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62800ce30 con 0x7fb62c003f80 2023-06-10T18:53:41.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628005960 con 0x7fb62c003f80 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb63b7fe700 1 -- 172.21.15.104:0/634037291 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628005d10 con 0x7fb62c003f80 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb643039700 1 -- 172.21.15.104:0/634037291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb62c003f80 msgr2=0x7fb62c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb643039700 1 --2- 172.21.15.104:0/634037291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb62c003f80 0x7fb62c004850 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c2f0 tx=0x7fb62c005880 comp rx=0 tx=0).stop 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.980+0000 7fb6415d6700 1 -- 172.21.15.104:0/634037291 reap_dead start 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.981+0000 7fb643039700 1 -- 172.21.15.104:0/634037291 shutdown_connections 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.981+0000 7fb643039700 1 -- 172.21.15.104:0/634037291 >> 172.21.15.104:0/634037291 conn(0x7fb63c06b8a0 msgr2=0x7fb63c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:41.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.981+0000 7fb643039700 1 -- 172.21.15.104:0/634037291 shutdown_connections 2023-06-10T18:53:41.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.981+0000 7fb643039700 1 -- 172.21.15.104:0/634037291 wait complete. 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.982+0000 7fb643039700 1 Processor -- start 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.982+0000 7fb643039700 1 -- start start 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 0x7fb63c1e6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb63c0b4dc0 0x7fb63c0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c14a580 con 0x7fb63c0b5230 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c1e5b40 con 0x7fb63c0b4dc0 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb643039700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c1e5e20 con 0x7fb62c003f80 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb6415d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb6415d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34714/0 (socket says 172.21.15.104:34714) 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb6415d6700 1 -- 172.21.15.104:0/3756535471 learned_addr learned my addr 172.21.15.104:0/3756535471 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb63bfff700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb63c0b4dc0 0x7fb63c0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.983+0000 7fb640dd5700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 0x7fb63c1e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 -- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 msgr2=0x7fb63c1e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 0x7fb63c1e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 -- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb63c0b4dc0 msgr2=0x7fb63c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb63c0b4dc0 0x7fb63c0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 -- 172.21.15.104:0/3756535471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c1e6100 con 0x7fb63c0b5230 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb640dd5700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 0x7fb63c1e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6415d6700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0701c0 tx=0x7fb634011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63400e040 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63400f040 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63c1e6b60 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63400fdb0 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.984+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63c1476e0 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.985+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c1e5980 con 0x7fb63c0b5230 2023-06-10T18:53:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.986+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb634022460 con 0x7fb63c0b5230 2023-06-10T18:53:41.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.986+0000 7fb6397fa700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 0x7fb62006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:41.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.986+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb634089a50 con 0x7fb63c0b5230 2023-06-10T18:53:41.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.986+0000 7fb640dd5700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 0x7fb62006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:41.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.987+0000 7fb640dd5700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 0x7fb62006d190 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb630006eb0 tx=0x7fb630008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:41.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:41.991+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb6340549c0 con 0x7fb63c0b5230 2023-06-10T18:53:42.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.176+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb63c06dc00 con 0x7fb63c0b5230 2023-06-10T18:53:42.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.176+0000 7fb6397fa700 1 -- 172.21.15.104:0/3756535471 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb634057fe0 con 0x7fb63c0b5230 2023-06-10T18:53:42.181 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:42.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 msgr2=0x7fb62006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 0x7fb62006d190 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb630006eb0 tx=0x7fb630008040 comp rx=0 tx=0).stop 2023-06-10T18:53:42.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 msgr2=0x7fb63c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0701c0 tx=0x7fb634011f80 comp rx=0 tx=0).stop 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 shutdown_connections 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb63c0b5230 0x7fb63c0cf780 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb62c003f80 0x7fb63c1e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb62006ace0 0x7fb62006d190 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 --2- 172.21.15.104:0/3756535471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb63c0b4dc0 0x7fb63c0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 >> 172.21.15.104:0/3756535471 conn(0x7fb63c06b8a0 msgr2=0x7fb63c15a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:42.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 shutdown_connections 2023-06-10T18:53:42.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.178+0000 7fb643039700 1 -- 172.21.15.104:0/3756535471 wait complete. 2023-06-10T18:53:42.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:53:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.311+0000 7f2285cb4700 1 Processor -- start 2023-06-10T18:53:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.311+0000 7f2285cb4700 1 -- start start 2023-06-10T18:53:42.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f2285cb4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228015c730 0x7f228006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f2285cb4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22800cb0c0 0x7f22800cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f2285cb4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228015cb00 con 0x7f22800cc000 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f2285cb4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22800aed80 con 0x7f228015c730 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f2285cb4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22800c9c60 con 0x7f22800cb0c0 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228015c730 0x7f228006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f228015c730 0x7f228006e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34718/0 (socket says 172.21.15.104:34718) 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 -- 172.21.15.104:0/1756486249 learned_addr learned my addr 172.21.15.104:0/1756486249 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227ffff700 1 --2- 172.21.15.104:0/1756486249 >> v2:172.21.15.114:3300/0 conn(0x7f22800cb0c0 0x7f22800cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 -- 172.21.15.104:0/1756486249 >> v2:172.21.15.114:3300/0 conn(0x7f22800cb0c0 msgr2=0x7f22800cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 --2- 172.21.15.104:0/1756486249 >> v2:172.21.15.114:3300/0 conn(0x7f22800cb0c0 0x7f22800cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 -- 172.21.15.104:0/1756486249 >> v1:172.21.15.114:6789/0 conn(0x7f22800cc000 legacy=0x7f228006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 -- 172.21.15.104:0/1756486249 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228006ecd0 con 0x7f228015c730 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227f7fe700 1 --2- 172.21.15.104:0/1756486249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228015c730 0x7f228006e790 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f22800c9bf0 tx=0x7f2268009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd01354ae0c2fe8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227e7fc700 1 -- 172.21.15.104:0/1756486249 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268005e30 con 0x7f228015c730 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227e7fc700 1 -- 172.21.15.104:0/1756486249 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2268004030 con 0x7f228015c730 2023-06-10T18:53:42.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.312+0000 7f227e7fc700 1 -- 172.21.15.104:0/1756486249 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22680043e0 con 0x7f228015c730 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 -- 172.21.15.104:0/1756486249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228015c730 msgr2=0x7f228006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 --2- 172.21.15.104:0/1756486249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228015c730 0x7f228006e790 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f22800c9bf0 tx=0x7f2268009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 -- 172.21.15.104:0/1756486249 shutdown_connections 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 --2- 172.21.15.104:0/1756486249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f228015c730 0x7f228006e790 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 --2- 172.21.15.104:0/1756486249 >> v2:172.21.15.114:3300/0 conn(0x7f22800cb0c0 0x7f22800cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 -- 172.21.15.104:0/1756486249 >> 172.21.15.104:0/1756486249 conn(0x7f228006b8a0 msgr2=0x7f22800cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:42.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 -- 172.21.15.104:0/1756486249 shutdown_connections 2023-06-10T18:53:42.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.313+0000 7f2285cb4700 1 -- 172.21.15.104:0/1756486249 wait complete. 2023-06-10T18:53:42.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.314+0000 7f2285cb4700 1 Processor -- start 2023-06-10T18:53:42.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.314+0000 7f2285cb4700 1 -- start start 2023-06-10T18:53:42.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22800cb0c0 0x7f22800af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228015c730 0x7f228014d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22800af560 con 0x7f22800cc000 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22800b0f70 con 0x7f228015c730 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f2285cb4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22800b11c0 con 0x7f22800cb0c0 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f227effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f227effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34726/0 (socket says 172.21.15.104:34726) 2023-06-10T18:53:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f227effd700 1 -- 172.21.15.104:0/657865437 learned_addr learned my addr 172.21.15.104:0/657865437 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f227f7fe700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22800cb0c0 0x7f22800af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.315+0000 7f227ffff700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228015c730 0x7f228014d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 -- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22800cb0c0 msgr2=0x7f22800af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22800cb0c0 0x7f22800af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 -- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228015c730 msgr2=0x7f228014d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228015c730 0x7f228014d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 -- 172.21.15.104:0/657865437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2268009500 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f227effd700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f227000c4e0 tx=0x7f227000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22700052d0 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2270005430 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22700038c0 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22800b14a0 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.316+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22800b1a60 con 0x7f22800cc000 2023-06-10T18:53:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.317+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2280074050 con 0x7f22800cc000 2023-06-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.317+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2270003a20 con 0x7f22800cc000 2023-06-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.318+0000 7f2277fff700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 0x7f226006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.318+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f22700887a0 con 0x7f22800cc000 2023-06-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.318+0000 7f227f7fe700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 0x7f226006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.318+0000 7f227f7fe700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 0x7f226006d190 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f22800c9bf0 tx=0x7f2268008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.322+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2270053710 con 0x7f22800cc000 2023-06-10T18:53:42.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.521+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f228015cb00 con 0x7f22800cc000 2023-06-10T18:53:42.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.522+0000 7f2277fff700 1 -- 172.21.15.104:0/657865437 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v99) v1 ==== 80+0+0 (secure 0 0 0) 0x7f228015cb00 con 0x7f22800cc000 2023-06-10T18:53:42.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 msgr2=0x7f226006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 0x7f226006d190 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f22800c9bf0 tx=0x7f2268008070 comp rx=0 tx=0).stop 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 msgr2=0x7f22800b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f227000c4e0 tx=0x7f227000c7f0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 shutdown_connections 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f226006ace0 0x7f226006d190 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f228015c730 0x7f228014d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22800cb0c0 0x7f22800af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 --2- 172.21.15.104:0/657865437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22800cc000 0x7f22800b0a30 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 >> 172.21.15.104:0/657865437 conn(0x7f228006b8a0 msgr2=0x7f22800ac4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 shutdown_connections 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.524+0000 7f2285cb4700 1 -- 172.21.15.104:0/657865437 wait complete. 2023-06-10T18:53:42.530 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:53:42.540 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:53:42.541 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:42.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:42.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.678+0000 7f3aa9d98700 1 Processor -- start 2023-06-10T18:53:42.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.678+0000 7f3aa9d98700 1 -- start start 2023-06-10T18:53:42.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa9d98700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3aa406e0e0 0x7f3aa406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa9d98700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 0x7f3aa40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa9d98700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa406d290 con 0x7f3aa406e4b0 2023-06-10T18:53:42.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa9d98700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa4071640 con 0x7f3aa406e0e0 2023-06-10T18:53:42.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa9d98700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa40aef00 con 0x7f3aa4069e70 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3aa406e0e0 0x7f3aa406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3aa406e0e0 0x7f3aa406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34732/0 (socket says 172.21.15.104:34732) 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 -- 172.21.15.104:0/3157651282 learned_addr learned my addr 172.21.15.104:0/3157651282 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa3fff700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 0x7f3aa40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 -- 172.21.15.104:0/3157651282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 msgr2=0x7f3aa40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 0x7f3aa40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.679+0000 7f3aa37fe700 1 -- 172.21.15.104:0/3157651282 >> v1:172.21.15.114:6789/0 conn(0x7f3aa406e4b0 legacy=0x7f3aa40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa37fe700 1 -- 172.21.15.104:0/3157651282 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa40ca2d0 con 0x7f3aa406e0e0 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa3fff700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 0x7f3aa40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa37fe700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e0e0 0x7f3aa406cd50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c006fd0 tx=0x7f3a8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c5286082a8d07e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa27fc700 1 -- 172.21.15.104:0/3157651282 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c005e30 con 0x7f3aa406e0e0 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa27fc700 1 -- 172.21.15.104:0/3157651282 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c004030 con 0x7f3aa406e0e0 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa27fc700 1 -- 172.21.15.104:0/3157651282 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c0043e0 con 0x7f3aa406e0e0 2023-06-10T18:53:42.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 -- 172.21.15.104:0/3157651282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e0e0 msgr2=0x7f3aa406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e0e0 0x7f3aa406cd50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c006fd0 tx=0x7f3a8c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 -- 172.21.15.104:0/3157651282 shutdown_connections 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e0e0 0x7f3aa406cd50 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/3157651282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa4069e70 0x7f3aa40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 -- 172.21.15.104:0/3157651282 >> 172.21.15.104:0/3157651282 conn(0x7f3aa406b8a0 msgr2=0x7f3aa40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:42.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 -- 172.21.15.104:0/3157651282 shutdown_connections 2023-06-10T18:53:42.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.680+0000 7f3aa9d98700 1 -- 172.21.15.104:0/3157651282 wait complete. 2023-06-10T18:53:42.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 Processor -- start 2023-06-10T18:53:42.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 -- start start 2023-06-10T18:53:42.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3aa4069e70 0x7f3aa415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa406e0e0 0x7f3aa4159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa40b4c40 con 0x7f3aa406e4b0 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa40b5450 con 0x7f3aa4069e70 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.682+0000 7f3aa9d98700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa415a0f0 con 0x7f3aa406e0e0 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34742/0 (socket says 172.21.15.104:34742) 2023-06-10T18:53:42.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 -- 172.21.15.104:0/1582040933 learned_addr learned my addr 172.21.15.104:0/1582040933 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa37fe700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3aa4069e70 0x7f3aa415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa2ffd700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa406e0e0 0x7f3aa4159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 -- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa406e0e0 msgr2=0x7f3aa4159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa406e0e0 0x7f3aa4159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 -- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3aa4069e70 msgr2=0x7f3aa415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3aa4069e70 0x7f3aa415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 -- 172.21.15.104:0/1582040933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c009500 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa3fff700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c007df0 tx=0x7f3a9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c00dd20 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9c00f040 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa40b4e90 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c003c60 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.683+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa40b34d0 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.684+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa4074050 con 0x7f3aa406e4b0 2023-06-10T18:53:42.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.685+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3a9c004030 con 0x7f3aa406e4b0 2023-06-10T18:53:42.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.685+0000 7f3a9bfff700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 0x7f3a8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:42.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.685+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3a9c0895d0 con 0x7f3aa406e4b0 2023-06-10T18:53:42.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.685+0000 7f3aa37fe700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 0x7f3a8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:42.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.686+0000 7f3aa37fe700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 0x7f3a8406d190 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c009c10 tx=0x7f3a8c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:42.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.689+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3a9c054600 con 0x7f3aa406e4b0 2023-06-10T18:53:42.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.876+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aa40707a0 con 0x7f3aa406e4b0 2023-06-10T18:53:42.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.877+0000 7f3a9bfff700 1 -- 172.21.15.104:0/1582040933 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+18543 (secure 0 0 0) 0x7f3a9c057c20 con 0x7f3aa406e4b0 2023-06-10T18:53:42.882 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:42.882 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":99,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:30.449397+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:42.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 msgr2=0x7f3a8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 0x7f3a8406d190 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c009c10 tx=0x7f3a8c008070 comp rx=0 tx=0).stop 2023-06-10T18:53:42.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 msgr2=0x7f3aa4099340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:42.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c007df0 tx=0x7f3a9c00d3b0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 shutdown_connections 2023-06-10T18:53:42.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a8406ace0 0x7f3a8406d190 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3aa4069e70 0x7f3aa415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3aa406e4b0 0x7f3aa4099340 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 --2- 172.21.15.104:0/1582040933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3aa406e0e0 0x7f3aa4159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.879+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 >> 172.21.15.104:0/1582040933 conn(0x7f3aa406b8a0 msgr2=0x7f3aa40acbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:42.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.880+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 shutdown_connections 2023-06-10T18:53:42.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:42.880+0000 7f3aa9d98700 1 -- 172.21.15.104:0/1582040933 wait complete. 2023-06-10T18:53:42.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:42.899 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:43.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.119+0000 7f0eb21e4700 1 Processor -- start 2023-06-10T18:53:43.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.119+0000 7f0eb21e4700 1 -- start start 2023-06-10T18:53:43.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eb21e4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0eac06e6e0 0x7f0eac0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eb21e4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0eac06cde0 0x7f0eac0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eb21e4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac06e250 con 0x7f0eac15c580 2023-06-10T18:53:43.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eb21e4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eac15c950 con 0x7f0eac06e6e0 2023-06-10T18:53:43.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eb21e4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eac0b1e80 con 0x7f0eac06cde0 2023-06-10T18:53:43.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0eac06e6e0 0x7f0eac0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0eac06e6e0 0x7f0eac0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34748/0 (socket says 172.21.15.104:34748) 2023-06-10T18:53:43.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eab7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0eac15c580 0x7f0eac0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60806/0 (socket says 172.21.15.104:60806) 2023-06-10T18:53:43.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 -- 172.21.15.104:0/1968397929 learned_addr learned my addr 172.21.15.104:0/1968397929 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eabfff700 1 --2- 172.21.15.104:0/1968397929 >> v2:172.21.15.114:3300/0 conn(0x7f0eac06cde0 0x7f0eac0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 -- 172.21.15.104:0/1968397929 >> v2:172.21.15.114:3300/0 conn(0x7f0eac06cde0 msgr2=0x7f0eac0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:43.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 --2- 172.21.15.104:0/1968397929 >> v2:172.21.15.114:3300/0 conn(0x7f0eac06cde0 0x7f0eac0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 -- 172.21.15.104:0/1968397929 >> v1:172.21.15.104:6789/0 conn(0x7f0eac15c580 legacy=0x7f0eac0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 -- 172.21.15.104:0/1968397929 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac06d920 con 0x7f0eac06e6e0 2023-06-10T18:53:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaa7fc700 1 -- 172.21.15.104:0/1968397929 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033421361 0 0) 0x7f0eac06e250 con 0x7f0eac15c580 2023-06-10T18:53:43.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaaffd700 1 --2- 172.21.15.104:0/1968397929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0b1780 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0cbc60 tx=0x7f0e94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3d3e31be88a6af server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaa7fc700 1 -- 172.21.15.104:0/1968397929 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400b870 con 0x7f0eac06e6e0 2023-06-10T18:53:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaa7fc700 1 -- 172.21.15.104:0/1968397929 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9400b9d0 con 0x7f0eac06e6e0 2023-06-10T18:53:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.120+0000 7f0eaa7fc700 1 -- 172.21.15.104:0/1968397929 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400bd60 con 0x7f0eac06e6e0 2023-06-10T18:53:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 -- 172.21.15.104:0/1968397929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac06e6e0 msgr2=0x7f0eac0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/1968397929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0b1780 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0cbc60 tx=0x7f0e94009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 -- 172.21.15.104:0/1968397929 shutdown_connections 2023-06-10T18:53:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/1968397929 >> v2:172.21.15.114:3300/0 conn(0x7f0eac06cde0 0x7f0eac0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/1968397929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0b1780 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 -- 172.21.15.104:0/1968397929 >> 172.21.15.104:0/1968397929 conn(0x7f0eac06b8a0 msgr2=0x7f0eac0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 -- 172.21.15.104:0/1968397929 shutdown_connections 2023-06-10T18:53:43.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.121+0000 7f0eb21e4700 1 -- 172.21.15.104:0/1968397929 wait complete. 2023-06-10T18:53:43.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.122+0000 7f0eb21e4700 1 Processor -- start 2023-06-10T18:53:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.122+0000 7f0eb21e4700 1 -- start start 2023-06-10T18:53:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 0x7f0eac155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0b2420 con 0x7f0eac15c580 2023-06-10T18:53:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0b2cc0 con 0x7f0eac06e6e0 2023-06-10T18:53:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eb21e4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0c5450 con 0x7f0eac06cde0 2023-06-10T18:53:43.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34756/0 (socket says 172.21.15.104:34756) 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 -- 172.21.15.104:0/3900794979 learned_addr learned my addr 172.21.15.104:0/3900794979 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eaaffd700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eab7fe700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 0x7f0eac155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 -- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 msgr2=0x7f0eac155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 0x7f0eac155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 -- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 msgr2=0x7f0eac0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eabfff700 1 -- 172.21.15.104:0/3900794979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94009550 con 0x7f0eac15c580 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eab7fe700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 0x7f0eac155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.123+0000 7f0eaaffd700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:43.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0eabfff700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00e840 tx=0x7f0e9c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c013610 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c013770 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac0b2670 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c013b20 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.124+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac155740 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.125+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac074050 con 0x7f0eac15c580 2023-06-10T18:53:43.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.125+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0e9c013c80 con 0x7f0eac15c580 2023-06-10T18:53:43.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.125+0000 7f0ea3fff700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 0x7f0e8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.126+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0e9c089240 con 0x7f0eac15c580 2023-06-10T18:53:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.126+0000 7f0eab7fe700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 0x7f0e8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.126+0000 7f0eab7fe700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 0x7f0e8c06d190 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40013a0 tx=0x7f0ea4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.130+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0e9c057cd0 con 0x7f0eac15c580 2023-06-10T18:53:43.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.314+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0eac0cb1f0 con 0x7f0eac15c580 2023-06-10T18:53:43.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.314+0000 7f0ea3fff700 1 -- 172.21.15.104:0/3900794979 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e9c057860 con 0x7f0eac15c580 2023-06-10T18:53:43.319 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 msgr2=0x7f0e8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 0x7f0e8c06d190 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40013a0 tx=0x7f0ea4009040 comp rx=0 tx=0).stop 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 msgr2=0x7f0eac0af980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00e840 tx=0x7f0e9c00ec00 comp rx=0 tx=0).stop 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 shutdown_connections 2023-06-10T18:53:43.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0eac15c580 0x7f0eac0af980 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0eac06cde0 0x7f0eac155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0e8c06ace0 0x7f0e8c06d190 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 --2- 172.21.15.104:0/3900794979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eac06e6e0 0x7f0eac0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 >> 172.21.15.104:0/3900794979 conn(0x7f0eac06b8a0 msgr2=0x7f0eac0cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 shutdown_connections 2023-06-10T18:53:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.316+0000 7f0eb21e4700 1 -- 172.21.15.104:0/3900794979 wait complete. 2023-06-10T18:53:43.333 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:43.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.450+0000 7f297c3ec700 1 Processor -- start 2023-06-10T18:53:43.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.450+0000 7f297c3ec700 1 -- start start 2023-06-10T18:53:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.451+0000 7f297c3ec700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2974157b30 0x7f29740ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.451+0000 7f297c3ec700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29740ce880 0x7f29740d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.451+0000 7f297c3ec700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29740ad3c0 con 0x7f29740d0830 2023-06-10T18:53:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.451+0000 7f297c3ec700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740ced20 con 0x7f29740ce880 2023-06-10T18:53:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.451+0000 7f297c3ec700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2974157f00 con 0x7f2974157b30 2023-06-10T18:53:43.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29740ce880 0x7f29740d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29740ce880 0x7f29740d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34760/0 (socket says 172.21.15.104:34760) 2023-06-10T18:53:43.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 -- 172.21.15.104:0/2788231454 learned_addr learned my addr 172.21.15.104:0/2788231454 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f297a188700 1 --2- 172.21.15.104:0/2788231454 >> v2:172.21.15.114:3300/0 conn(0x7f2974157b30 0x7f29740ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 -- 172.21.15.104:0/2788231454 >> v2:172.21.15.114:3300/0 conn(0x7f2974157b30 msgr2=0x7f29740ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:43.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 --2- 172.21.15.104:0/2788231454 >> v2:172.21.15.114:3300/0 conn(0x7f2974157b30 0x7f29740ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 -- 172.21.15.104:0/2788231454 >> v1:172.21.15.114:6789/0 conn(0x7f29740d0830 legacy=0x7f297415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 -- 172.21.15.104:0/2788231454 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29740b0170 con 0x7f29740ce880 2023-06-10T18:53:43.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979987700 1 --2- 172.21.15.104:0/2788231454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f29740d02f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f2974071360 tx=0x7f2964009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6badf435247912c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979186700 1 -- 172.21.15.104:0/2788231454 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296400b870 con 0x7f29740ce880 2023-06-10T18:53:43.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979186700 1 -- 172.21.15.104:0/2788231454 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296400b9d0 con 0x7f29740ce880 2023-06-10T18:53:43.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.452+0000 7f2979186700 1 -- 172.21.15.104:0/2788231454 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296400bd60 con 0x7f29740ce880 2023-06-10T18:53:43.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 -- 172.21.15.104:0/2788231454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 msgr2=0x7f29740d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 --2- 172.21.15.104:0/2788231454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f29740d02f0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f2974071360 tx=0x7f2964009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 -- 172.21.15.104:0/2788231454 shutdown_connections 2023-06-10T18:53:43.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 --2- 172.21.15.104:0/2788231454 >> v2:172.21.15.114:3300/0 conn(0x7f2974157b30 0x7f29740ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 --2- 172.21.15.104:0/2788231454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f29740d02f0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 -- 172.21.15.104:0/2788231454 >> 172.21.15.104:0/2788231454 conn(0x7f2974089fd0 msgr2=0x7f297408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 -- 172.21.15.104:0/2788231454 shutdown_connections 2023-06-10T18:53:43.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.453+0000 7f297c3ec700 1 -- 172.21.15.104:0/2788231454 wait complete. 2023-06-10T18:53:43.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.454+0000 7f297c3ec700 1 Processor -- start 2023-06-10T18:53:43.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.454+0000 7f297c3ec700 1 -- start start 2023-06-10T18:53:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 0x7f29740b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2974157b30 0x7f2974156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740c5a40 con 0x7f29740ce880 2023-06-10T18:53:43.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740c5c90 con 0x7f29740d0830 2023-06-10T18:53:43.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297c3ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29740c5f70 con 0x7f2974157b30 2023-06-10T18:53:43.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34766/0 (socket says 172.21.15.104:34766) 2023-06-10T18:53:43.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 -- 172.21.15.104:0/1235922917 learned_addr learned my addr 172.21.15.104:0/1235922917 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f2979987700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 0x7f29740b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 -- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2974157b30 msgr2=0x7f2974156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2974157b30 0x7f2974156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 -- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 msgr2=0x7f29740b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 0x7f29740b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 -- 172.21.15.104:0/1235922917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964009550 con 0x7f29740ce880 2023-06-10T18:53:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f2979987700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 0x7f29740b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.455+0000 7f297a188700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f297000d800 tx=0x7f297000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700125f0 con 0x7f29740ce880 2023-06-10T18:53:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970012750 con 0x7f29740ce880 2023-06-10T18:53:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29740c6250 con 0x7f29740ce880 2023-06-10T18:53:43.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970012b00 con 0x7f29740ce880 2023-06-10T18:53:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29740c6a50 con 0x7f29740ce880 2023-06-10T18:53:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.456+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29740c68a0 con 0x7f29740ce880 2023-06-10T18:53:43.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.457+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2970012c60 con 0x7f29740ce880 2023-06-10T18:53:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.457+0000 7f296affd700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 0x7f295806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.458+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f29700898f0 con 0x7f29740ce880 2023-06-10T18:53:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.458+0000 7f2979987700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 0x7f295806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.458+0000 7f2979987700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 0x7f295806d190 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2974156da0 tx=0x7f2964008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.462+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f29700548f0 con 0x7f29740ce880 2023-06-10T18:53:43.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.648+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2974074050 con 0x7f29740ce880 2023-06-10T18:53:43.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.649+0000 7f296affd700 1 -- 172.21.15.104:0/1235922917 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v99) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2970057f10 con 0x7f29740ce880 2023-06-10T18:53:43.654 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:43.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 msgr2=0x7f295806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 0x7f295806d190 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2974156da0 tx=0x7f2964008030 comp rx=0 tx=0).stop 2023-06-10T18:53:43.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 msgr2=0x7f297414d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f297000d800 tx=0x7f297000dbc0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 shutdown_connections 2023-06-10T18:53:43.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2974157b30 0x7f2974156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29740ce880 0x7f297414d340 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f295806ace0 0x7f295806d190 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 --2- 172.21.15.104:0/1235922917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29740d0830 0x7f29740b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 >> 172.21.15.104:0/1235922917 conn(0x7f2974089fd0 msgr2=0x7f297406fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 shutdown_connections 2023-06-10T18:53:43.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.651+0000 7f297c3ec700 1 -- 172.21.15.104:0/1235922917 wait complete. 2023-06-10T18:53:43.669 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:43.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.786+0000 7fb22fbdd700 1 Processor -- start 2023-06-10T18:53:43.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.786+0000 7fb22fbdd700 1 -- start start 2023-06-10T18:53:43.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22fbdd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb22815c500 0x7fb2280cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22fbdd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb2280cf200 0x7fb228069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22fbdd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22815c8d0 con 0x7fb22806a3b0 2023-06-10T18:53:43.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22fbdd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2280aef20 con 0x7fb22815c500 2023-06-10T18:53:43.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22fbdd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22806abd0 con 0x7fb2280cf200 2023-06-10T18:53:43.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22d979700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb22815c500 0x7fb2280cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22d979700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb22815c500 0x7fb2280cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34774/0 (socket says 172.21.15.104:34774) 2023-06-10T18:53:43.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.787+0000 7fb22d979700 1 -- 172.21.15.104:0/3925320909 learned_addr learned my addr 172.21.15.104:0/3925320909 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d178700 1 --2- 172.21.15.104:0/3925320909 >> v2:172.21.15.114:3300/0 conn(0x7fb2280cf200 0x7fb228069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d979700 1 -- 172.21.15.104:0/3925320909 >> v2:172.21.15.114:3300/0 conn(0x7fb2280cf200 msgr2=0x7fb228069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:43.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d979700 1 --2- 172.21.15.104:0/3925320909 >> v2:172.21.15.114:3300/0 conn(0x7fb2280cf200 0x7fb228069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d979700 1 -- 172.21.15.104:0/3925320909 >> v1:172.21.15.114:6789/0 conn(0x7fb22806a3b0 legacy=0x7fb22806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:43.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d979700 1 -- 172.21.15.104:0/3925320909 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22806a8f0 con 0x7fb22815c500 2023-06-10T18:53:43.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22d979700 1 --2- 172.21.15.104:0/3925320909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22815c500 0x7fb2280cecc0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb2280cbca0 tx=0x7fb218009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1462838c50b38 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22c977700 1 -- 172.21.15.104:0/3925320909 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb218005e30 con 0x7fb22815c500 2023-06-10T18:53:43.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22c977700 1 -- 172.21.15.104:0/3925320909 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb218006030 con 0x7fb22815c500 2023-06-10T18:53:43.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22c977700 1 -- 172.21.15.104:0/3925320909 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2180063e0 con 0x7fb22815c500 2023-06-10T18:53:43.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3925320909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22815c500 msgr2=0x7fb2280cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3925320909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22815c500 0x7fb2280cecc0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb2280cbca0 tx=0x7fb218009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3925320909 shutdown_connections 2023-06-10T18:53:43.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3925320909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22815c500 0x7fb2280cecc0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3925320909 >> v2:172.21.15.114:3300/0 conn(0x7fb2280cf200 0x7fb228069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.788+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3925320909 >> 172.21.15.104:0/3925320909 conn(0x7fb22806b8a0 msgr2=0x7fb2280cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.789+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3925320909 shutdown_connections 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.789+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3925320909 wait complete. 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.790+0000 7fb22fbdd700 1 Processor -- start 2023-06-10T18:53:43.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.790+0000 7fb22fbdd700 1 -- start start 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.790+0000 7fb22fbdd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb22806a3b0 0x7fb2281546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.790+0000 7fb22fbdd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.790+0000 7fb22fbdd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb22815c500 0x7fb2280b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22fbdd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2280c5c00 con 0x7fb2280cf200 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22fbdd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2280c7d00 con 0x7fb22806a3b0 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22fbdd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2280c7fe0 con 0x7fb22815c500 2023-06-10T18:53:43.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34782/0 (socket says 172.21.15.104:34782) 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 -- 172.21.15.104:0/3549030986 learned_addr learned my addr 172.21.15.104:0/3549030986 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d979700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb22806a3b0 0x7fb2281546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22e17a700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb22815c500 0x7fb2280b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 -- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb22815c500 msgr2=0x7fb2280b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb22815c500 0x7fb2280b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 -- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb22806a3b0 msgr2=0x7fb2281546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb22806a3b0 0x7fb2281546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 -- 172.21.15.104:0/3549030986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218009500 con 0x7fb2280cf200 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22d178700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb22400d800 tx=0x7fb22400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2240125f0 con 0x7fb2280cf200 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224012750 con 0x7fb2280cf200 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2280c82c0 con 0x7fb2280cf200 2023-06-10T18:53:43.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224012b00 con 0x7fb2280cf200 2023-06-10T18:53:43.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.791+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2280b28c0 con 0x7fb2280cf200 2023-06-10T18:53:43.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.792+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2280b2440 con 0x7fb2280cf200 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.793+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb224012c60 con 0x7fb2280cf200 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.793+0000 7fb21e7fc700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 0x7fb20c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.794+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb22408a1c0 con 0x7fb2280cf200 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.794+0000 7fb22d979700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 0x7fb20c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.794+0000 7fb22d979700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 0x7fb20c06d190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb2280cbca0 tx=0x7fb218008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:43.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.797+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb2240551a0 con 0x7fb2280cf200 2023-06-10T18:53:43.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.983+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2280ce1c0 con 0x7fb2280cf200 2023-06-10T18:53:43.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.983+0000 7fb21e7fc700 1 -- 172.21.15.104:0/3549030986 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2280ce1c0 con 0x7fb2280cf200 2023-06-10T18:53:43.988 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:43.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 msgr2=0x7fb20c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 0x7fb20c06d190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb2280cbca0 tx=0x7fb218008070 comp rx=0 tx=0).stop 2023-06-10T18:53:43.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 msgr2=0x7fb2280b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:43.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb22400d800 tx=0x7fb22400db10 comp rx=0 tx=0).stop 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 shutdown_connections 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb20c06ace0 0x7fb20c06d190 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb22806a3b0 0x7fb2281546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb22815c500 0x7fb2280b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 --2- 172.21.15.104:0/3549030986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2280cf200 0x7fb2280b2de0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 >> 172.21.15.104:0/3549030986 conn(0x7fb22806b8a0 msgr2=0x7fb2280cb260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:43.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 shutdown_connections 2023-06-10T18:53:43.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:43.985+0000 7fb22fbdd700 1 -- 172.21.15.104:0/3549030986 wait complete. 2023-06-10T18:53:44.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:44.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.121+0000 7f8d9fac9700 1 Processor -- start 2023-06-10T18:53:44.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.121+0000 7f8d9fac9700 1 -- start start 2023-06-10T18:53:44.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9fac9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 0x7f8d9806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9fac9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 0x7f8d9806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9fac9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980ae7e0 con 0x7f8d9815c6f0 2023-06-10T18:53:44.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9fac9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9806df90 con 0x7f8d980b1780 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9fac9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9806e270 con 0x7f8d9806ea70 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9d865700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8d9815c6f0 0x7f8d980b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60848/0 (socket says 172.21.15.104:60848) 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9d865700 1 -- 172.21.15.104:0/8943700 learned_addr learned my addr 172.21.15.104:0/8943700 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9d064700 1 --2- 172.21.15.104:0/8943700 >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 0x7f8d9806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 --2- 172.21.15.104:0/8943700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 0x7f8d9806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9c863700 1 -- 172.21.15.104:0/8943700 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974392851 0 0) 0x7f8d980ae7e0 con 0x7f8d9815c6f0 2023-06-10T18:53:44.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9c863700 1 -- 172.21.15.104:0/8943700 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d840035b0 con 0x7f8d9815c6f0 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 -- 172.21.15.104:0/8943700 >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 msgr2=0x7f8d9806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 --2- 172.21.15.104:0/8943700 >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 0x7f8d9806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 -- 172.21.15.104:0/8943700 >> v1:172.21.15.104:6789/0 conn(0x7f8d9815c6f0 legacy=0x7f8d980b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 -- 172.21.15.104:0/8943700 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9806e550 con 0x7f8d9806ea70 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9d064700 1 --2- 172.21.15.104:0/8943700 >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 0x7f8d9806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.122+0000 7f8d9e066700 1 --2- 172.21.15.104:0/8943700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 0x7f8d9806d320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009d90 tx=0x7f8d9000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b27f58a2c33eb94 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9c863700 1 -- 172.21.15.104:0/8943700 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000e070 con 0x7f8d9806ea70 2023-06-10T18:53:44.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9c863700 1 -- 172.21.15.104:0/8943700 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9000bb20 con 0x7f8d9806ea70 2023-06-10T18:53:44.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9c863700 1 -- 172.21.15.104:0/8943700 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90012430 con 0x7f8d9806ea70 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 -- 172.21.15.104:0/8943700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 msgr2=0x7f8d9806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/8943700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 0x7f8d9806d320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009d90 tx=0x7f8d9000b330 comp rx=0 tx=0).stop 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 -- 172.21.15.104:0/8943700 shutdown_connections 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/8943700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9806ea70 0x7f8d9806d320 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/8943700 >> v2:172.21.15.114:3300/0 conn(0x7f8d980b1780 0x7f8d9806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 -- 172.21.15.104:0/8943700 >> 172.21.15.104:0/8943700 conn(0x7f8d9806b8a0 msgr2=0x7f8d980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 -- 172.21.15.104:0/8943700 shutdown_connections 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.123+0000 7f8d9fac9700 1 -- 172.21.15.104:0/8943700 wait complete. 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.124+0000 7f8d9fac9700 1 Processor -- start 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.124+0000 7f8d9fac9700 1 -- start start 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d980b1780 0x7f8d980ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9815c6f0 0x7f8d98069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980ad2f0 con 0x7f8d9806ea70 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980c9cc0 con 0x7f8d980b1780 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9fac9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980acd30 con 0x7f8d9815c6f0 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34794/0 (socket says 172.21.15.104:34794) 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 -- 172.21.15.104:0/1212276859 learned_addr learned my addr 172.21.15.104:0/1212276859 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9e066700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9815c6f0 0x7f8d98069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d064700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d980b1780 0x7f8d980ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 -- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9815c6f0 msgr2=0x7f8d98069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9815c6f0 0x7f8d98069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 -- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d980b1780 msgr2=0x7f8d980ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d980b1780 0x7f8d980ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.125+0000 7f8d9d865700 1 -- 172.21.15.104:0/1212276859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d880094a0 con 0x7f8d9806ea70 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d9d865700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001260 tx=0x7f8d9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940074e0 con 0x7f8d9806ea70 2023-06-10T18:53:44.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d94007640 con 0x7f8d9806ea70 2023-06-10T18:53:44.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d90009a70 con 0x7f8d9806ea70 2023-06-10T18:53:44.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94005a50 con 0x7f8d9806ea70 2023-06-10T18:53:44.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.126+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d980c9f60 con 0x7f8d9806ea70 2023-06-10T18:53:44.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.127+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d98074050 con 0x7f8d9806ea70 2023-06-10T18:53:44.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.127+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d94005bb0 con 0x7f8d9806ea70 2023-06-10T18:53:44.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.128+0000 7f8d8e7fc700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 0x7f8d7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.128+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d94088aa0 con 0x7f8d9806ea70 2023-06-10T18:53:44.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.128+0000 7f8d9d064700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 0x7f8d7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.128+0000 7f8d9d064700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 0x7f8d7c06d190 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8d980c97a0 tx=0x7f8d88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.132+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d94053aa0 con 0x7f8d9806ea70 2023-06-10T18:53:44.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.318+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d980b1560 con 0x7f8d9806ea70 2023-06-10T18:53:44.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.319+0000 7f8d8e7fc700 1 -- 172.21.15.104:0/1212276859 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d940170a0 con 0x7f8d9806ea70 2023-06-10T18:53:44.324 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:44.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 msgr2=0x7f8d7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 0x7f8d7c06d190 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8d980c97a0 tx=0x7f8d88008040 comp rx=0 tx=0).stop 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 msgr2=0x7f8d980ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001260 tx=0x7f8d9400c860 comp rx=0 tx=0).stop 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 shutdown_connections 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d9815c6f0 0x7f8d98069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d9806ea70 0x7f8d980ac1c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d7c06ace0 0x7f8d7c06d190 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 --2- 172.21.15.104:0/1212276859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d980b1780 0x7f8d980ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 >> 172.21.15.104:0/1212276859 conn(0x7f8d9806b8a0 msgr2=0x7f8d9815b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:44.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 shutdown_connections 2023-06-10T18:53:44.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.321+0000 7f8d9fac9700 1 -- 172.21.15.104:0/1212276859 wait complete. 2023-06-10T18:53:44.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:53:44.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.453+0000 7f223b40f700 1 Processor -- start 2023-06-10T18:53:44.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.454+0000 7f223b40f700 1 -- start start 2023-06-10T18:53:44.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f223b40f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f223415a2b0 0x7f22340ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f223b40f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2234069e70 0x7f22340cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f223b40f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223415a680 con 0x7f22340cb690 2023-06-10T18:53:44.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f223b40f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340d0510 con 0x7f223415a2b0 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f223b40f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223406a5e0 con 0x7f2234069e70 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f223415a2b0 0x7f22340ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22399ac700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f22340cb690 0x7f22340b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60866/0 (socket says 172.21.15.104:60866) 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22399ac700 1 -- 172.21.15.104:0/4101406333 learned_addr learned my addr 172.21.15.104:0/4101406333 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22389aa700 1 --2- 172.21.15.104:0/4101406333 >> v2:172.21.15.114:3300/0 conn(0x7f2234069e70 0x7f22340cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 -- 172.21.15.104:0/4101406333 >> v2:172.21.15.114:3300/0 conn(0x7f2234069e70 msgr2=0x7f22340cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:44.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 --2- 172.21.15.104:0/4101406333 >> v2:172.21.15.114:3300/0 conn(0x7f2234069e70 0x7f22340cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 -- 172.21.15.104:0/4101406333 >> v1:172.21.15.104:6789/0 conn(0x7f22340cb690 legacy=0x7f22340b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 -- 172.21.15.104:0/4101406333 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223406a8c0 con 0x7f223415a2b0 2023-06-10T18:53:44.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f222bfff700 1 -- 172.21.15.104:0/4101406333 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620401910 0 0) 0x7f223415a680 con 0x7f22340cb690 2023-06-10T18:53:44.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f22391ab700 1 --2- 172.21.15.104:0/4101406333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f223415a2b0 0x7f22340ac230 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f22340700a0 tx=0x7f2224009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171e14db13712b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f222bfff700 1 -- 172.21.15.104:0/4101406333 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224005e30 con 0x7f223415a2b0 2023-06-10T18:53:44.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f222bfff700 1 -- 172.21.15.104:0/4101406333 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224006030 con 0x7f223415a2b0 2023-06-10T18:53:44.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.455+0000 7f222bfff700 1 -- 172.21.15.104:0/4101406333 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240063c0 con 0x7f223415a2b0 2023-06-10T18:53:44.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 -- 172.21.15.104:0/4101406333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f223415a2b0 msgr2=0x7f22340ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 --2- 172.21.15.104:0/4101406333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f223415a2b0 0x7f22340ac230 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f22340700a0 tx=0x7f2224009b00 comp rx=0 tx=0).stop 2023-06-10T18:53:44.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 -- 172.21.15.104:0/4101406333 shutdown_connections 2023-06-10T18:53:44.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 --2- 172.21.15.104:0/4101406333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f223415a2b0 0x7f22340ac230 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 --2- 172.21.15.104:0/4101406333 >> v2:172.21.15.114:3300/0 conn(0x7f2234069e70 0x7f22340cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 -- 172.21.15.104:0/4101406333 >> 172.21.15.104:0/4101406333 conn(0x7f223406b8a0 msgr2=0x7f22340cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 -- 172.21.15.104:0/4101406333 shutdown_connections 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.456+0000 7f223b40f700 1 -- 172.21.15.104:0/4101406333 wait complete. 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.457+0000 7f223b40f700 1 Processor -- start 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.457+0000 7f223b40f700 1 -- start start 2023-06-10T18:53:44.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 0x7f22340b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 0x7f22340b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8da0 con 0x7f2234069e70 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b54b0 con 0x7f223415a2b0 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f223b40f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340aff40 con 0x7f22340cb690 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34808/0 (socket says 172.21.15.104:34808) 2023-06-10T18:53:44.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 -- 172.21.15.104:0/3504587819 learned_addr learned my addr 172.21.15.104:0/3504587819 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22389aa700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 0x7f22340b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22399ac700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 0x7f22340b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 -- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 msgr2=0x7f22340b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 0x7f22340b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 -- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 msgr2=0x7f22340b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 0x7f22340b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 -- 172.21.15.104:0/3504587819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224009430 con 0x7f2234069e70 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22389aa700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 0x7f22340b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22399ac700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 0x7f22340b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.458+0000 7f22391ab700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c00 tx=0x7f2224004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.459+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224004040 con 0x7f2234069e70 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.459+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224020070 con 0x7f2234069e70 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.459+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224004400 con 0x7f2234069e70 2023-06-10T18:53:44.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.459+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22340b0490 con 0x7f2234069e70 2023-06-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.459+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22340c8f80 con 0x7f2234069e70 2023-06-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.460+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340b0770 con 0x7f2234069e70 2023-06-10T18:53:44.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.461+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2224022030 con 0x7f2234069e70 2023-06-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.461+0000 7f2229ffb700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 0x7f221806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.461+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f222408aed0 con 0x7f2234069e70 2023-06-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.461+0000 7f22389aa700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 0x7f221806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.462+0000 7f22389aa700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 0x7f221806d190 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f223414f620 tx=0x7f2230008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:44.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.465+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2224055fb0 con 0x7f2234069e70 2023-06-10T18:53:44.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.665+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2234074050 con 0x7f2234069e70 2023-06-10T18:53:44.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.666+0000 7f2229ffb700 1 -- 172.21.15.104:0/3504587819 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v99) v1 ==== 80+0+0 (secure 0 0 0) 0x7f22240595d0 con 0x7f2234069e70 2023-06-10T18:53:44.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 msgr2=0x7f221806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 0x7f221806d190 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f223414f620 tx=0x7f2230008040 comp rx=0 tx=0).stop 2023-06-10T18:53:44.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 msgr2=0x7f223414f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:44.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f2224000c00 tx=0x7f2224004750 comp rx=0 tx=0).stop 2023-06-10T18:53:44.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 shutdown_connections 2023-06-10T18:53:44.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221806ace0 0x7f221806d190 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a2b0 0x7f22340b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2234069e70 0x7f223414f0e0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 --2- 172.21.15.104:0/3504587819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340cb690 0x7f22340b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:44.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 >> 172.21.15.104:0/3504587819 conn(0x7f223406b8a0 msgr2=0x7f2234154520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:44.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 shutdown_connections 2023-06-10T18:53:44.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:44.668+0000 7f223b40f700 1 -- 172.21.15.104:0/3504587819 wait complete. 2023-06-10T18:53:44.826 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:53:44.826 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:53:44.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:44.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:53:44.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[112756]: fuse finished with error 0 and tester_r 0 2023-06-10T18:53:44.882 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:50.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:53:50.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:50.986 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:53:51.008 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:53:51.009 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:53:51.009 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:53:51.010 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:51.010 DEBUG:teuthology.orchestra.run.smithi104:> path = "/home/ubuntu/cephtest/mnt.0/subdir" 2023-06-10T18:53:51.010 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:51.011 DEBUG:teuthology.orchestra.run.smithi104:> print("Starting creation...") 2023-06-10T18:53:51.011 DEBUG:teuthology.orchestra.run.smithi104:> start = time.time() 2023-06-10T18:53:51.011 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:51.011 DEBUG:teuthology.orchestra.run.smithi104:> os.mkdir(path) 2023-06-10T18:53:51.012 DEBUG:teuthology.orchestra.run.smithi104:> dfd = os.open(path, os.O_DIRECTORY) 2023-06-10T18:53:51.012 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:51.013 DEBUG:teuthology.orchestra.run.smithi104:> fd = open(os.path.join(path, "childfile"), "w") 2023-06-10T18:53:51.013 DEBUG:teuthology.orchestra.run.smithi104:> print("Finished creation in {0}s".format(time.time() - start)) 2023-06-10T18:53:51.014 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:53:51.014 DEBUG:teuthology.orchestra.run.smithi104:> print("Starting fsync...") 2023-06-10T18:53:51.014 DEBUG:teuthology.orchestra.run.smithi104:> start = time.time() 2023-06-10T18:53:51.014 DEBUG:teuthology.orchestra.run.smithi104:> if True: 2023-06-10T18:53:51.015 DEBUG:teuthology.orchestra.run.smithi104:> os.fsync(dfd) 2023-06-10T18:53:51.015 DEBUG:teuthology.orchestra.run.smithi104:> else: 2023-06-10T18:53:51.016 DEBUG:teuthology.orchestra.run.smithi104:> os.fsync(fd) 2023-06-10T18:53:51.016 DEBUG:teuthology.orchestra.run.smithi104:> print("Finished fsync in {0}s".format(time.time() - start)) 2023-06-10T18:53:51.016 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:53:51.102 INFO:teuthology.orchestra.run.smithi104.stdout:Starting creation... 2023-06-10T18:53:51.103 INFO:teuthology.orchestra.run.smithi104.stdout:Finished creation in 0.006462574005126953s 2023-06-10T18:53:51.103 INFO:teuthology.orchestra.run.smithi104.stdout:Starting fsync... 2023-06-10T18:53:51.103 INFO:teuthology.orchestra.run.smithi104.stdout:Finished fsync in 0.002370595932006836s 2023-06-10T18:53:51.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:53:51.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.448+0000 7f4ac1d17700 1 Processor -- start 2023-06-10T18:53:51.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.448+0000 7f4ac1d17700 1 -- start start 2023-06-10T18:53:51.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4ac1d17700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4abc06e4b0 0x7f4abc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4ac1d17700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc06d290 con 0x7f4abc06e0e0 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4ac1d17700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0ca2d0 con 0x7f4abc069e70 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4ac1d17700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc071640 con 0x7f4abc06e4b0 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4abb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4abc06e0e0 0x7f4abc0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60876/0 (socket says 172.21.15.104:60876) 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4abb7fe700 1 -- 172.21.15.104:0/3946292835 learned_addr learned my addr 172.21.15.104:0/3946292835 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4abaffd700 1 --2- 172.21.15.104:0/3946292835 >> v2:172.21.15.114:3300/0 conn(0x7f4abc06e4b0 0x7f4abc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:51.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.449+0000 7f4aba7fc700 1 -- 172.21.15.104:0/3946292835 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456338887 0 0) 0x7f4abc06d290 con 0x7f4abc06e0e0 2023-06-10T18:53:51.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4aba7fc700 1 -- 172.21.15.104:0/3946292835 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa40035b0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4abaffd700 1 -- 172.21.15.104:0/3946292835 >> v1:172.21.15.114:6789/0 conn(0x7f4abc069e70 legacy=0x7f4abc0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:51.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4abaffd700 1 -- 172.21.15.104:0/3946292835 >> v1:172.21.15.104:6789/0 conn(0x7f4abc06e0e0 legacy=0x7f4abc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:51.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4abaffd700 1 -- 172.21.15.104:0/3946292835 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc06d4c0 con 0x7f4abc06e4b0 2023-06-10T18:53:51.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4abaffd700 1 --2- 172.21.15.104:0/3946292835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc06cd50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009840 tx=0x7f4aac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc31da40691becf8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:51.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4aba7fc700 1 -- 172.21.15.104:0/3946292835 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac011830 con 0x7f4abc06e4b0 2023-06-10T18:53:51.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4aba7fc700 1 -- 172.21.15.104:0/3946292835 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac011990 con 0x7f4abc06e4b0 2023-06-10T18:53:51.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.450+0000 7f4aba7fc700 1 -- 172.21.15.104:0/3946292835 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac011d00 con 0x7f4abc06e4b0 2023-06-10T18:53:51.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 -- 172.21.15.104:0/3946292835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 msgr2=0x7f4abc06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:51.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/3946292835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc06cd50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009840 tx=0x7f4aac009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:51.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 -- 172.21.15.104:0/3946292835 shutdown_connections 2023-06-10T18:53:51.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/3946292835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc06cd50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:51.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 -- 172.21.15.104:0/3946292835 >> 172.21.15.104:0/3946292835 conn(0x7f4abc06b8a0 msgr2=0x7f4abc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:51.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 -- 172.21.15.104:0/3946292835 shutdown_connections 2023-06-10T18:53:51.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.451+0000 7f4ac1d17700 1 -- 172.21.15.104:0/3946292835 wait complete. 2023-06-10T18:53:51.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.452+0000 7f4ac1d17700 1 Processor -- start 2023-06-10T18:53:51.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.452+0000 7f4ac1d17700 1 -- start start 2023-06-10T18:53:51.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4abc069e70 0x7f4abc0c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:51.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc1f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0c6640 con 0x7f4abc06e0e0 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc1f30a0 con 0x7f4abc06e4b0 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0b2560 con 0x7f4abc069e70 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34824/0 (socket says 172.21.15.104:34824) 2023-06-10T18:53:51.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 -- 172.21.15.104:0/2193189671 learned_addr learned my addr 172.21.15.104:0/2193189671 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abbfff700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc1f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abb7fe700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4abc069e70 0x7f4abc0c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 -- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4abc069e70 msgr2=0x7f4abc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4abc069e70 0x7f4abc0c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 -- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 msgr2=0x7f4abc1f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc1f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 -- 172.21.15.104:0/2193189671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0094a0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abbfff700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc1f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4abaffd700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0090b0 tx=0x7f4aac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac0045d0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac0036a0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc15ba30 con 0x7f4abc06e0e0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac003a10 con 0x7f4abc06e0e0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.453+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc1f2dc0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.454+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc0b3ea0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.455+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4aac003b70 con 0x7f4abc06e0e0 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.455+0000 7f4aa3fff700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 0x7f4aa806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.455+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4aac08aa20 con 0x7f4abc06e0e0 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.455+0000 7f4abb7fe700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 0x7f4aa806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.456+0000 7f4abb7fe700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 0x7f4aa806d1c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0007fd0 tx=0x7f4ab0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:51.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.460+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4aac055aa0 con 0x7f4abc06e0e0 2023-06-10T18:53:51.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:51.686+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4abc0ce180 con 0x7f4abc06e0e0 2023-06-10T18:53:52.683 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:53:52.678+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:53:52.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.685+0000 7f4aa3fff700 1 -- 172.21.15.104:0/2193189671 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4aac0590c0 con 0x7f4abc06e0e0 2023-06-10T18:53:52.692 INFO:tasks.ceph.mds.b.smithi114.stderr: -1634> 2023-06-10T18:53:34.928+0000 7f9d194c1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:53:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.687+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 msgr2=0x7f4aa806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.687+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 0x7f4aa806d1c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0007fd0 tx=0x7f4ab0009040 comp rx=0 tx=0).stop 2023-06-10T18:53:52.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.687+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 msgr2=0x7f4abc1f1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.687+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0090b0 tx=0x7f4aac004750 comp rx=0 tx=0).stop 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 shutdown_connections 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4aa806ad10 0x7f4aa806d1c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4abc06e4b0 0x7f4abc1f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4abc069e70 0x7f4abc0c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 --2- 172.21.15.104:0/2193189671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4abc06e0e0 0x7f4abc1f1f20 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 >> 172.21.15.104:0/2193189671 conn(0x7f4abc06b8a0 msgr2=0x7f4abc0ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:52.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 shutdown_connections 2023-06-10T18:53:52.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.688+0000 7f4ac1d17700 1 -- 172.21.15.104:0/2193189671 wait complete. 2023-06-10T18:53:52.696 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:53:52.707 INFO:tasks.cephfs.mount:Killing connection on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:53:52.707 INFO:tasks.cephfs.mount:Suspending the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:53:52.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:52.708 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 down 2023-06-10T18:53:52.720 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:53:52.745 INFO:tasks.cephfs.mount:Cleaning up killed connection on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:53:52.746 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:52.746 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:52.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:52.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.959+0000 7fe6fdf27700 1 Processor -- start 2023-06-10T18:53:52.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.959+0000 7fe6fdf27700 1 -- start start 2023-06-10T18:53:52.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6fdf27700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f80b4ff0 0x7fe6f80ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6fdf27700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f80b6070 con 0x7fe6f80ce890 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6fdf27700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f80b1d70 con 0x7fe6f815a420 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6fdf27700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80ced10 con 0x7fe6f80b4ff0 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6f7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6f80ce890 0x7fe6f815a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60894/0 (socket says 172.21.15.104:60894) 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6f7fff700 1 -- 172.21.15.104:0/3680586311 learned_addr learned my addr 172.21.15.104:0/3680586311 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f80b4ff0 0x7fe6f80ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780929966 0 0) 0x7fe6f80b6070 con 0x7fe6f80ce890 2023-06-10T18:53:52.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.960+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e00035b0 con 0x7fe6f80ce890 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778783737 0 0) 0x7fe6f80b1d70 con 0x7fe6f815a420 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f80b6070 con 0x7fe6f815a420 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892631669 0 0) 0x7fe6e00035b0 con 0x7fe6f80ce890 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f80b4ff0 msgr2=0x7fe6f80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f80b4ff0 0x7fe6f80ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 >> v1:172.21.15.114:6789/0 conn(0x7fe6f815a420 legacy=0x7fe6f80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80b5460 con 0x7fe6f80ce890 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fe6f00019f0 con 0x7fe6f80ce890 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 >> v1:172.21.15.104:6789/0 conn(0x7fe6f80ce890 legacy=0x7fe6f815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e0003bc0 0x7fe6e0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e0004850 0x7fe6e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0005200 0x7fe6e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80ced10 con 0x7fe6e0004850 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000c030 con 0x7fe6e0005200 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000c1f0 con 0x7fe6e0003bc0 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f80b4ff0 0x7fe6f80ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e0004850 0x7fe6e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f7fff700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0005200 0x7fe6e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f77fe700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e0003bc0 0x7fe6e0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 -- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e0003bc0 msgr2=0x7fe6e0003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e0003bc0 0x7fe6e0003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 -- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0005200 msgr2=0x7fe6e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0005200 0x7fe6e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 -- 172.21.15.104:0/3680586311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e00035b0 con 0x7fe6e0004850 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.961+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e0004850 0x7fe6e0004cc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80ce280 tx=0x7fe6e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac355d911650fcbf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec004030 con 0x7fe6e0004850 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec004190 con 0x7fe6e0004850 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6f67fc700 1 -- 172.21.15.104:0/3680586311 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec004540 con 0x7fe6e0004850 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 -- 172.21.15.104:0/3680586311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e0004850 msgr2=0x7fe6e0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/3680586311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6e0004850 0x7fe6e0004cc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80ce280 tx=0x7fe6e0005880 comp rx=0 tx=0).stop 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6f7fff700 1 -- 172.21.15.104:0/3680586311 reap_dead start 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 -- 172.21.15.104:0/3680586311 shutdown_connections 2023-06-10T18:53:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 -- 172.21.15.104:0/3680586311 >> 172.21.15.104:0/3680586311 conn(0x7fe6f8089fd0 msgr2=0x7fe6f80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 -- 172.21.15.104:0/3680586311 shutdown_connections 2023-06-10T18:53:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.962+0000 7fe6fdf27700 1 -- 172.21.15.104:0/3680586311 wait complete. 2023-06-10T18:53:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.964+0000 7fe6fdf27700 1 Processor -- start 2023-06-10T18:53:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.964+0000 7fe6fdf27700 1 -- start start 2023-06-10T18:53:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0004850 0x7fe6f80b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f81e18a0 0x7fe6f81e3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81e29b0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81e1d30 con 0x7fe6e0004850 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81e2010 con 0x7fe6f81e18a0 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34846/0 (socket says 172.21.15.104:34846) 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 -- 172.21.15.104:0/264071349 learned_addr learned my addr 172.21.15.104:0/264071349 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f77fe700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0004850 0x7fe6f80b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f6ffd700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f81e18a0 0x7fe6f81e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 -- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f81e18a0 msgr2=0x7fe6f81e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f81e18a0 0x7fe6f81e3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 -- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0004850 msgr2=0x7fe6f80b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0004850 0x7fe6f80b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 -- 172.21.15.104:0/264071349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f81e22f0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6f7fff700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80cad60 tx=0x7fe6f000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000ad90 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f00133f0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0013760 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.965+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f81e25d0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.966+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f80cd680 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.967+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f81e2af0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.967+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe6f00138c0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.967+0000 7fe6dffff700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 0x7fe6e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.967+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fe6f0089fb0 con 0x7fe6f81e37d0 2023-06-10T18:53:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.968+0000 7fe6f77fe700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 0x7fe6e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.968+0000 7fe6f77fe700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 0x7fe6e406d190 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8000c00 tx=0x7fe6e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:52.972+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe6f0054ec0 con 0x7fe6f81e37d0 2023-06-10T18:53:53.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.163+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6f8074050 con 0x7fe6f81e37d0 2023-06-10T18:53:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.164+0000 7fe6dffff700 1 -- 172.21.15.104:0/264071349 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+18675 (secure 0 0 0) 0x7fe6f00584e0 con 0x7fe6f81e37d0 2023-06-10T18:53:53.169 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:53.169 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":100,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:51.686953+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:53.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 msgr2=0x7fe6e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 0x7fe6e406d190 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8000c00 tx=0x7fe6e8009040 comp rx=0 tx=0).stop 2023-06-10T18:53:53.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 msgr2=0x7fe6f80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80cad60 tx=0x7fe6f000a5c0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 shutdown_connections 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe6e406ace0 0x7fe6e406d190 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e0004850 0x7fe6f80b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6f81e37d0 0x7fe6f80ccf60 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 --2- 172.21.15.104:0/264071349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6f81e18a0 0x7fe6f81e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.166+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 >> 172.21.15.104:0/264071349 conn(0x7fe6f8089fd0 msgr2=0x7fe6f80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.167+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 shutdown_connections 2023-06-10T18:53:53.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.167+0000 7fe6fdf27700 1 -- 172.21.15.104:0/264071349 wait complete. 2023-06-10T18:53:53.183 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:53.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:53.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.302+0000 7ffaaef18700 1 Processor -- start 2023-06-10T18:53:53.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.302+0000 7ffaaef18700 1 -- start start 2023-06-10T18:53:53.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaef18700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaef18700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaef18700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa80d0480 con 0x7ffaa806d320 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaef18700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa806a800 con 0x7ffaa806a3b0 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaef18700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa806d7a0 con 0x7ffaa80aec40 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaccb4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaa7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaccb4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:54234/0 (socket says 172.21.15.104:54234) 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaad4b5700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffaa806d320 0x7ffaa80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:38526/0 (socket says 172.21.15.104:38526) 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaa7fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:50586/0 (socket says 172.21.15.104:50586) 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.303+0000 7ffaaccb4700 1 -- 172.21.15.104:0/3850999561 learned_addr learned my addr 172.21.15.104:0/3850999561 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaccb4700 1 -- 172.21.15.104:0/3850999561 >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 msgr2=0x7ffaa806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaccb4700 1 --2- 172.21.15.104:0/3850999561 >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaccb4700 1 -- 172.21.15.104:0/3850999561 >> v1:172.21.15.114:6789/0 conn(0x7ffaa806d320 legacy=0x7ffaa80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaccb4700 1 -- 172.21.15.104:0/3850999561 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa815dd00 con 0x7ffaa80aec40 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaa77fe700 1 -- 172.21.15.104:0/3850999561 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044598796 0 0) 0x7ffaa80d0480 con 0x7ffaa806d320 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaa7fff700 1 --2- 172.21.15.104:0/3850999561 >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaccb4700 1 --2- 172.21.15.104:0/3850999561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c006fd0 tx=0x7ffa9c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbd3cef14e22863a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaa77fe700 1 -- 172.21.15.104:0/3850999561 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c005e30 con 0x7ffaa80aec40 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaa77fe700 1 -- 172.21.15.104:0/3850999561 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c004030 con 0x7ffaa80aec40 2023-06-10T18:53:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaa77fe700 1 -- 172.21.15.104:0/3850999561 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0043e0 con 0x7ffaa80aec40 2023-06-10T18:53:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaef18700 1 -- 172.21.15.104:0/3850999561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 msgr2=0x7ffaa8069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.304+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3850999561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c006fd0 tx=0x7ffa9c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 -- 172.21.15.104:0/3850999561 shutdown_connections 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3850999561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa80aec40 0x7ffaa8069e70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3850999561 >> v2:172.21.15.114:3300/0 conn(0x7ffaa806a3b0 0x7ffaa806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 -- 172.21.15.104:0/3850999561 >> 172.21.15.104:0/3850999561 conn(0x7ffaa806b8a0 msgr2=0x7ffaa80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 -- 172.21.15.104:0/3850999561 shutdown_connections 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.305+0000 7ffaaef18700 1 -- 172.21.15.104:0/3850999561 wait complete. 2023-06-10T18:53:53.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.306+0000 7ffaaef18700 1 Processor -- start 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.306+0000 7ffaaef18700 1 -- start start 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa806a3b0 0x7ffaa815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa81e6ac0 con 0x7ffaa80aec40 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80c9470 con 0x7ffaa806d320 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa81e3ee0 con 0x7ffaa806a3b0 2023-06-10T18:53:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:50592/0 (socket says 172.21.15.104:50592) 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34860/0 (socket says 172.21.15.104:34860) 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa7fff700 1 -- 172.21.15.104:0/3564985599 learned_addr learned my addr 172.21.15.104:0/3564985599 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaccb4700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa806a3b0 0x7ffaa815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 -- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa806a3b0 msgr2=0x7ffaa815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa806a3b0 0x7ffaa815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 -- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 msgr2=0x7ffaa81f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 -- 172.21.15.104:0/3564985599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa940094a0 con 0x7ffaa80aec40 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa7fff700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaad4b5700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000cbe0 tx=0x7ffaa000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000aaa0 con 0x7ffaa80aec40 2023-06-10T18:53:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa000ac00 con 0x7ffaa80aec40 2023-06-10T18:53:53.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c009500 con 0x7ffaa80aec40 2023-06-10T18:53:53.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.307+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00070d0 con 0x7ffaa80aec40 2023-06-10T18:53:53.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.308+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa81e4780 con 0x7ffaa80aec40 2023-06-10T18:53:53.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.308+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa8074050 con 0x7ffaa80aec40 2023-06-10T18:53:53.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.309+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffaa0007230 con 0x7ffaa80aec40 2023-06-10T18:53:53.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.309+0000 7ffaa57fa700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 0x7ffa8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.310+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ffaa0089210 con 0x7ffaa80aec40 2023-06-10T18:53:53.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.310+0000 7ffaaccb4700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 0x7ffa8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.310+0000 7ffaaccb4700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 0x7ffa8c06d190 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c006fd0 tx=0x7ffa9c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.314+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffaa00541b0 con 0x7ffaa80aec40 2023-06-10T18:53:53.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.503+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaa80cfd00 con 0x7ffaa80aec40 2023-06-10T18:53:53.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.503+0000 7ffaa57fa700 1 -- 172.21.15.104:0/3564985599 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffaa00577d0 con 0x7ffaa80aec40 2023-06-10T18:53:53.508 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 msgr2=0x7ffa8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 0x7ffa8c06d190 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c006fd0 tx=0x7ffa9c008070 comp rx=0 tx=0).stop 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 msgr2=0x7ffaa80ad440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000cbe0 tx=0x7ffaa000cef0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 shutdown_connections 2023-06-10T18:53:53.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffaa80aec40 0x7ffaa80ad440 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffaa806a3b0 0x7ffaa815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffa8c06ace0 0x7ffa8c06d190 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 --2- 172.21.15.104:0/3564985599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa806d320 0x7ffaa81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 >> 172.21.15.104:0/3564985599 conn(0x7ffaa806b8a0 msgr2=0x7ffaa80cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 shutdown_connections 2023-06-10T18:53:53.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.505+0000 7ffaaef18700 1 -- 172.21.15.104:0/3564985599 wait complete. 2023-06-10T18:53:53.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.642+0000 7fe5dd707700 1 Processor -- start 2023-06-10T18:53:53.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.642+0000 7fe5dd707700 1 -- start start 2023-06-10T18:53:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5dd707700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d8158790 0x7fe5d80ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5dd707700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5d80caf70 0x7fe5d80cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5dd707700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d8071580 con 0x7fe5d80ccfe0 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5dd707700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d80cb950 con 0x7fe5d80caf70 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5dd707700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80c9020 con 0x7fe5d8158790 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5d80caf70 0x7fe5d80cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe5d80caf70 0x7fe5d80cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34868/0 (socket says 172.21.15.104:34868) 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 -- 172.21.15.104:0/123449345 learned_addr learned my addr 172.21.15.104:0/123449345 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:53.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d6ffd700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d8158790 0x7fe5d80ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 -- 172.21.15.104:0/123449345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d8158790 msgr2=0x7fe5d80ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d8158790 0x7fe5d80ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 -- 172.21.15.104:0/123449345 >> v1:172.21.15.114:6789/0 conn(0x7fe5d80ccfe0 legacy=0x7fe5d80c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 -- 172.21.15.104:0/123449345 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8070f30 con 0x7fe5d80caf70 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d77fe700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d80caf70 0x7fe5d80cb410 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000d150 tx=0x7fe5d000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b544ca31995c8def server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.643+0000 7fe5d5ffb700 1 -- 172.21.15.104:0/123449345 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0010070 con 0x7fe5d80caf70 2023-06-10T18:53:53.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5d5ffb700 1 -- 172.21.15.104:0/123449345 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d000dd80 con 0x7fe5d80caf70 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5d5ffb700 1 -- 172.21.15.104:0/123449345 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000b310 con 0x7fe5d80caf70 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 -- 172.21.15.104:0/123449345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d80caf70 msgr2=0x7fe5d80cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d80caf70 0x7fe5d80cb410 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000d150 tx=0x7fe5d000d460 comp rx=0 tx=0).stop 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 -- 172.21.15.104:0/123449345 shutdown_connections 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d80caf70 0x7fe5d80cb410 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 --2- 172.21.15.104:0/123449345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d8158790 0x7fe5d80ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 -- 172.21.15.104:0/123449345 >> 172.21.15.104:0/123449345 conn(0x7fe5d806aac0 msgr2=0x7fe5d80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 -- 172.21.15.104:0/123449345 shutdown_connections 2023-06-10T18:53:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.644+0000 7fe5dd707700 1 -- 172.21.15.104:0/123449345 wait complete. 2023-06-10T18:53:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.645+0000 7fe5dd707700 1 Processor -- start 2023-06-10T18:53:53.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.645+0000 7fe5dd707700 1 -- start start 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 0x7fe5d80b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 0x7fe5d80c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b1990 con 0x7fe5d8158790 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b1c70 con 0x7fe5d80ccfe0 2023-06-10T18:53:53.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5dd707700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b1f50 con 0x7fe5d80caf70 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34874/0 (socket says 172.21.15.104:34874) 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.646+0000 7fe5d77fe700 1 -- 172.21.15.104:0/3934524194 learned_addr learned my addr 172.21.15.104:0/3934524194 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d6ffd700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 0x7fe5d80b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d67fc700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 0x7fe5d80c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 -- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 msgr2=0x7fe5d80b13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 0x7fe5d80b13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 -- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 msgr2=0x7fe5d80c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 0x7fe5d80c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 -- 172.21.15.104:0/3934524194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d000bd40 con 0x7fe5d8158790 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d67fc700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 0x7fe5d80c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d6ffd700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 0x7fe5d80b13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:53.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5d77fe700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe5d806fc60 tx=0x7fe5d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d000ab20 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d000ac80 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d80b2230 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0004030 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.647+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d80c83b0 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.648+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d8074050 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.648+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe5d001b050 con 0x7fe5d8158790 2023-06-10T18:53:53.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.649+0000 7fe5bf7fe700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 0x7fe5c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.649+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fe5d008cc00 con 0x7fe5d8158790 2023-06-10T18:53:53.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.649+0000 7fe5d6ffd700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 0x7fe5c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.649+0000 7fe5d6ffd700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 0x7fe5c406d190 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.653+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe5d0093080 con 0x7fe5d8158790 2023-06-10T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.840+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe5d80c1c10 con 0x7fe5d8158790 2023-06-10T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.840+0000 7fe5bf7fe700 1 -- 172.21.15.104:0/3934524194 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe5d0019020 con 0x7fe5d8158790 2023-06-10T18:53:53.846 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:53.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 msgr2=0x7fe5c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 0x7fe5c406d190 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8008040 comp rx=0 tx=0).stop 2023-06-10T18:53:53.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 msgr2=0x7fe5d80c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe5d806fc60 tx=0x7fe5d0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 shutdown_connections 2023-06-10T18:53:53.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5d8158790 0x7fe5d80c5a30 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5d80caf70 0x7fe5d80b13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe5c406ace0 0x7fe5c406d190 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 --2- 172.21.15.104:0/3934524194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d80ccfe0 0x7fe5d80c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 >> 172.21.15.104:0/3934524194 conn(0x7fe5d806aac0 msgr2=0x7fe5d8157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 shutdown_connections 2023-06-10T18:53:53.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.842+0000 7fe5dd707700 1 -- 172.21.15.104:0/3934524194 wait complete. 2023-06-10T18:53:53.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.977+0000 7f4d5b873700 1 Processor -- start 2023-06-10T18:53:53.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.977+0000 7f4d5b873700 1 -- start start 2023-06-10T18:53:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5b873700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4d5406e6e0 0x7f4d540b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5b873700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5406e250 con 0x7f4d5415c580 2023-06-10T18:53:53.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5b873700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5415c950 con 0x7f4d5406cde0 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5b873700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d540b1e80 con 0x7f4d5406e6e0 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d58e0e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4d5406e6e0 0x7f4d540b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5960f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4d5415c580 0x7f4d540b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60938/0 (socket says 172.21.15.104:60938) 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d5960f700 1 -- 172.21.15.104:0/1974676656 learned_addr learned my addr 172.21.15.104:0/1974676656 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d58e0e700 1 -- 172.21.15.104:0/1974676656 >> v1:172.21.15.114:6789/0 conn(0x7f4d5406cde0 legacy=0x7f4d540ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d58e0e700 1 -- 172.21.15.104:0/1974676656 >> v1:172.21.15.104:6789/0 conn(0x7f4d5415c580 legacy=0x7f4d540b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d4bfff700 1 -- 172.21.15.104:0/1974676656 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172118919 0 0) 0x7f4d5406e250 con 0x7f4d5415c580 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.978+0000 7f4d58e0e700 1 -- 172.21.15.104:0/1974676656 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5406d920 con 0x7f4d5406e6e0 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d58e0e700 1 --2- 172.21.15.104:0/1974676656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5406e6e0 0x7f4d540b1780 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009840 tx=0x7f4d50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649882419ebedadd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d4bfff700 1 -- 172.21.15.104:0/1974676656 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011830 con 0x7f4d5406e6e0 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d4bfff700 1 -- 172.21.15.104:0/1974676656 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50011990 con 0x7f4d5406e6e0 2023-06-10T18:53:53.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d4bfff700 1 -- 172.21.15.104:0/1974676656 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011d20 con 0x7f4d5406e6e0 2023-06-10T18:53:53.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 -- 172.21.15.104:0/1974676656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5406e6e0 msgr2=0x7f4d540b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 --2- 172.21.15.104:0/1974676656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5406e6e0 0x7f4d540b1780 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009840 tx=0x7f4d50009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 -- 172.21.15.104:0/1974676656 shutdown_connections 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 --2- 172.21.15.104:0/1974676656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5406e6e0 0x7f4d540b1780 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 -- 172.21.15.104:0/1974676656 >> 172.21.15.104:0/1974676656 conn(0x7f4d5406b8a0 msgr2=0x7f4d540cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 -- 172.21.15.104:0/1974676656 shutdown_connections 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.979+0000 7f4d5b873700 1 -- 172.21.15.104:0/1974676656 wait complete. 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.980+0000 7f4d5b873700 1 Processor -- start 2023-06-10T18:53:53.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 -- start start 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d5406cde0 0x7f4d540ad500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d5406e6e0 0x7f4d540b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540ad0e0 con 0x7f4d5415c580 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540ac1c0 con 0x7f4d5406cde0 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5b873700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540ac460 con 0x7f4d5406e6e0 2023-06-10T18:53:53.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d59e10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d59e10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34888/0 (socket says 172.21.15.104:34888) 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d59e10700 1 -- 172.21.15.104:0/2956398109 learned_addr learned my addr 172.21.15.104:0/2956398109 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.981+0000 7f4d5960f700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d5406cde0 0x7f4d540ad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 -- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d5406e6e0 msgr2=0x7f4d540b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d58e0e700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d5406e6e0 0x7f4d540b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d5406e6e0 0x7f4d540b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 -- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d5406cde0 msgr2=0x7f4d540ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:53.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d5406cde0 0x7f4d540ad500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 -- 172.21.15.104:0/2956398109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094a0 con 0x7f4d5415c580 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d59e10700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00cc50 tx=0x7f4d4c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c00ad80 con 0x7f4d5415c580 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c00ce50 con 0x7f4d5415c580 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d540ac740 con 0x7f4d5415c580 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c003c40 con 0x7f4d5415c580 2023-06-10T18:53:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.982+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d540acd00 con 0x7f4d5415c580 2023-06-10T18:53:53.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.983+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d540ad220 con 0x7f4d5415c580 2023-06-10T18:53:53.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.983+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4d4c0050d0 con 0x7f4d5415c580 2023-06-10T18:53:53.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.984+0000 7f4d49ffb700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 0x7f4d3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:53.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.984+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4d4c0896a0 con 0x7f4d5415c580 2023-06-10T18:53:53.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.984+0000 7f4d5960f700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 0x7f4d3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:53.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.984+0000 7f4d5960f700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 0x7f4d3806d190 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4d44007fd0 tx=0x7f4d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:53.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:53.988+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4d540ad220 con 0x7f4d5415c580 2023-06-10T18:53:54.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.175+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d540ad220 con 0x7f4d5415c580 2023-06-10T18:53:54.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.175+0000 7f4d49ffb700 1 -- 172.21.15.104:0/2956398109 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d540ad220 con 0x7f4d5415c580 2023-06-10T18:53:54.180 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 msgr2=0x7f4d3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 0x7f4d3806d190 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4d44007fd0 tx=0x7f4d44009040 comp rx=0 tx=0).stop 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 msgr2=0x7f4d540c3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00cc50 tx=0x7f4d4c00a340 comp rx=0 tx=0).stop 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 shutdown_connections 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d3806ace0 0x7f4d3806d190 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d5406cde0 0x7f4d540ad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d5415c580 0x7f4d540c3090 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 --2- 172.21.15.104:0/2956398109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d5406e6e0 0x7f4d540b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 >> 172.21.15.104:0/2956398109 conn(0x7f4d5406b8a0 msgr2=0x7f4d540cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:54.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 shutdown_connections 2023-06-10T18:53:54.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.177+0000 7f4d5b873700 1 -- 172.21.15.104:0/2956398109 wait complete. 2023-06-10T18:53:54.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:54.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.311+0000 7f223448c700 1 Processor -- start 2023-06-10T18:53:54.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.311+0000 7f223448c700 1 -- start start 2023-06-10T18:53:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f223448c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f222c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f223448c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c0ae7e0 con 0x7f222c0aec00 2023-06-10T18:53:54.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f223448c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c06df90 con 0x7f222c06ea70 2023-06-10T18:53:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f223448c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c06e270 con 0x7f222c15c6f0 2023-06-10T18:53:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f2232a29700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f222c0aec00 0x7f222c0ac8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60954/0 (socket says 172.21.15.104:60954) 2023-06-10T18:53:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f2232a29700 1 -- 172.21.15.104:0/1098005090 learned_addr learned my addr 172.21.15.104:0/1098005090 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.312+0000 7f2232228700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f222c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271582416 0 0) 0x7f222c0ae7e0 con 0x7f222c0aec00 2023-06-10T18:53:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22180035b0 con 0x7f222c0aec00 2023-06-10T18:53:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625133683 0 0) 0x7f222c06df90 con 0x7f222c06ea70 2023-06-10T18:53:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c0ae7e0 con 0x7f222c06ea70 2023-06-10T18:53:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38301620 0 0) 0x7f22180035b0 con 0x7f222c0aec00 2023-06-10T18:53:54.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 msgr2=0x7f222c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f222c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 >> v1:172.21.15.114:6789/0 conn(0x7f222c06ea70 legacy=0x7f222c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c06e550 con 0x7f222c0aec00 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f22240019f0 con 0x7f222c0aec00 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 >> v1:172.21.15.104:6789/0 conn(0x7f222c0aec00 legacy=0x7f222c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f2218003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2218004120 0x7f2218004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2218004d90 0x7f2218005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c06e270 con 0x7f2218004d90 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221800bba0 con 0x7f2218004120 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221800df40 con 0x7f222c15c6f0 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2218004d90 0x7f2218005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232228700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f2218003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2231a27700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2218004120 0x7f2218004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 -- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 msgr2=0x7f2218003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f222c15c6f0 0x7f2218003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 -- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2218004120 msgr2=0x7f2218004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2218004120 0x7f2218004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 -- 172.21.15.104:0/1098005090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22180035b0 con 0x7f2218004d90 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.313+0000 7f2232a29700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2218004d90 0x7f2218005200 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f222c0af020 tx=0x7f2218005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a284c4eaeb2d852 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240072d0 con 0x7f2218004d90 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224007430 con 0x7f2218004d90 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f2231226700 1 -- 172.21.15.104:0/1098005090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240077c0 con 0x7f2218004d90 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 -- 172.21.15.104:0/1098005090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2218004d90 msgr2=0x7f2218005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 --2- 172.21.15.104:0/1098005090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2218004d90 0x7f2218005200 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f222c0af020 tx=0x7f2218005880 comp rx=0 tx=0).stop 2023-06-10T18:53:54.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f2232a29700 1 -- 172.21.15.104:0/1098005090 reap_dead start 2023-06-10T18:53:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 -- 172.21.15.104:0/1098005090 shutdown_connections 2023-06-10T18:53:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 -- 172.21.15.104:0/1098005090 >> 172.21.15.104:0/1098005090 conn(0x7f222c06b8a0 msgr2=0x7f222c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 -- 172.21.15.104:0/1098005090 shutdown_connections 2023-06-10T18:53:54.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.314+0000 7f223448c700 1 -- 172.21.15.104:0/1098005090 wait complete. 2023-06-10T18:53:54.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.316+0000 7f223448c700 1 Processor -- start 2023-06-10T18:53:54.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.316+0000 7f223448c700 1 -- start start 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2218004d90 0x7f222c151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f222c0c3140 0x7f222c0b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0c5840 con 0x7f222c156930 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c1595e0 con 0x7f222c0c3140 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c156dc0 con 0x7f2218004d90 2023-06-10T18:53:54.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34904/0 (socket says 172.21.15.104:34904) 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 -- 172.21.15.104:0/573800780 learned_addr learned my addr 172.21.15.104:0/573800780 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2232228700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2218004d90 0x7f222c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2232a29700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f222c0c3140 0x7f222c0b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 -- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2218004d90 msgr2=0x7f222c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2218004d90 0x7f222c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 -- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f222c0c3140 msgr2=0x7f222c0b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f222c0c3140 0x7f222c0b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 -- 172.21.15.104:0/573800780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1570a0 con 0x7f222c156930 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2231a27700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f222800f810 tx=0x7f222800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280052e0 con 0x7f222c156930 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2228005460 con 0x7f222c156930 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c157380 con 0x7f222c156930 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.317+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280038c0 con 0x7f222c156930 2023-06-10T18:53:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.318+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c15b100 con 0x7f222c156930 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.318+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222c159790 con 0x7f222c156930 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.319+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2228003a20 con 0x7f222c156930 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.319+0000 7f2222ffd700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 0x7f221006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.319+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f22280894e0 con 0x7f222c156930 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.319+0000 7f2232228700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 0x7f221006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.320+0000 7f2232228700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 0x7f221006d190 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f221c009c40 tx=0x7f221c0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.324+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f222c159790 con 0x7f222c156930 2023-06-10T18:53:54.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.514+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f222c159790 con 0x7f222c156930 2023-06-10T18:53:54.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.514+0000 7f2222ffd700 1 -- 172.21.15.104:0/573800780 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7f222c159790 con 0x7f222c156930 2023-06-10T18:53:54.519 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 msgr2=0x7f221006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 0x7f221006d190 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f221c009c40 tx=0x7f221c0091c0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 msgr2=0x7f222c15a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f222800f810 tx=0x7f222800fb20 comp rx=0 tx=0).stop 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 shutdown_connections 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f221006ace0 0x7f221006d190 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f222c0c3140 0x7f222c0b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2218004d90 0x7f222c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 --2- 172.21.15.104:0/573800780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f222c156930 0x7f222c15a9e0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 >> 172.21.15.104:0/573800780 conn(0x7f222c06b8a0 msgr2=0x7f222c069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:54.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 shutdown_connections 2023-06-10T18:53:54.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.517+0000 7f223448c700 1 -- 172.21.15.104:0/573800780 wait complete. 2023-06-10T18:53:54.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:53:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.647+0000 7ff1e38bb700 1 Processor -- start 2023-06-10T18:53:54.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.648+0000 7ff1e38bb700 1 -- start start 2023-06-10T18:53:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.648+0000 7ff1e38bb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff1dc0cc710 0x7ff1dc06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.648+0000 7ff1e38bb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc15cb00 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.648+0000 7ff1e38bb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc06ecd0 con 0x7ff1dc15c730 2023-06-10T18:53:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.648+0000 7ff1e38bb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc0aed80 con 0x7ff1dc0cc710 2023-06-10T18:53:54.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e1e58700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff1dc0ca0b0 0x7ff1dc0ca530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60974/0 (socket says 172.21.15.104:60974) 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e1e58700 1 -- 172.21.15.104:0/2879883706 learned_addr learned my addr 172.21.15.104:0/2879883706 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/2879883706 >> v2:172.21.15.114:3300/0 conn(0x7ff1dc0cc710 0x7ff1dc06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165444646 0 0) 0x7ff1d4001490 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c80035b0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e0e56700 1 -- 172.21.15.104:0/2879883706 >> v1:172.21.15.114:6789/0 conn(0x7ff1dc15c730 legacy=0x7ff1dc0cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2087378835 0 0) 0x7ff1c80035b0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e0e56700 1 -- 172.21.15.104:0/2879883706 >> v1:172.21.15.104:6789/0 conn(0x7ff1dc0ca0b0 legacy=0x7ff1dc0ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e0e56700 1 -- 172.21.15.104:0/2879883706 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0cd280 con 0x7ff1dc0cc710 2023-06-10T18:53:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/2879883706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc06e790 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009840 tx=0x7ff1d8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=750974e1ab8cedfa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4001b80 con 0x7ff1dc0cc710 2023-06-10T18:53:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d8004030 con 0x7ff1dc0cc710 2023-06-10T18:53:54.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.649+0000 7ff1d3fff700 1 -- 172.21.15.104:0/2879883706 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80043c0 con 0x7ff1dc0cc710 2023-06-10T18:53:54.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 -- 172.21.15.104:0/2879883706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 msgr2=0x7ff1dc06e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/2879883706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc06e790 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009840 tx=0x7ff1d8009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:54.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 -- 172.21.15.104:0/2879883706 shutdown_connections 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/2879883706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc06e790 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 -- 172.21.15.104:0/2879883706 >> 172.21.15.104:0/2879883706 conn(0x7ff1dc06b8a0 msgr2=0x7ff1dc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 -- 172.21.15.104:0/2879883706 shutdown_connections 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.650+0000 7ff1e38bb700 1 -- 172.21.15.104:0/2879883706 wait complete. 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.651+0000 7ff1e38bb700 1 Processor -- start 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.651+0000 7ff1e38bb700 1 -- start start 2023-06-10T18:53:54.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc0c3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1dc15c730 0x7ff1dc0c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0ace70 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0ad150 con 0x7ff1dc0cc710 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e38bb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0ad430 con 0x7ff1dc15c730 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34916/0 (socket says 172.21.15.104:34916) 2023-06-10T18:53:54.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 -- 172.21.15.104:0/3149652400 learned_addr learned my addr 172.21.15.104:0/3149652400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc0c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1e58700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1dc15c730 0x7ff1dc0c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 -- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1dc15c730 msgr2=0x7ff1dc0c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1dc15c730 0x7ff1dc0c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 -- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 msgr2=0x7ff1dc0c3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc0c3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e1657700 1 -- 172.21.15.104:0/3149652400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80094a0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.652+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc0c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1e1657700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00c4d0 tx=0x7ff1cc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0054e0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc0036d0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc0ad710 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc003a80 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.653+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc0c4300 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.654+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc0adc30 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.654+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff1cc003be0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.654+0000 7ff1d1ffb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 0x7ff1c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:54.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.655+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff1cc088bc0 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.655+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 0x7ff1c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:54.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.655+0000 7ff1e0e56700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 0x7ff1c006d190 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ccb00 tx=0x7ff1d8008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:54.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.659+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff1dc0adc30 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.860+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1dc074050 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.860+0000 7ff1d1ffb700 1 -- 172.21.15.104:0/3149652400 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v100) v1 ==== 80+0+124 (secure 0 0 0) 0x7ff1cc017090 con 0x7ff1dc0ca0b0 2023-06-10T18:53:54.869 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6837/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:53:54.869 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6835/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:53:54.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 msgr2=0x7ff1c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 0x7ff1c006d190 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ccb00 tx=0x7ff1d8008000 comp rx=0 tx=0).stop 2023-06-10T18:53:54.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 msgr2=0x7ff1dc15a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:54.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00c4d0 tx=0x7ff1cc00c890 comp rx=0 tx=0).stop 2023-06-10T18:53:54.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 shutdown_connections 2023-06-10T18:53:54.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1dc15c730 0x7ff1dc0c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1dc0ca0b0 0x7ff1dc15a780 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff1c006ace0 0x7ff1c006d190 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 --2- 172.21.15.104:0/3149652400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1dc0cc710 0x7ff1dc0c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:54.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 >> 172.21.15.104:0/3149652400 conn(0x7ff1dc06b8a0 msgr2=0x7ff1dc0b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:54.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 shutdown_connections 2023-06-10T18:53:54.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:54.862+0000 7ff1e38bb700 1 -- 172.21.15.104:0/3149652400 wait complete. 2023-06-10T18:53:54.872 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:53:54.880 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:53:54.881 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:53:54.898 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:53:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.052+0000 7fc77db7b700 1 Processor -- start 2023-06-10T18:53:55.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.052+0000 7fc77db7b700 1 -- start start 2023-06-10T18:53:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc77db7b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc77806e0e0 0x7fc77806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc77db7b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77806d290 con 0x7fc778069e70 2023-06-10T18:53:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc77db7b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780ca2d0 con 0x7fc77806e4b0 2023-06-10T18:53:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc77db7b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc778071640 con 0x7fc77806e0e0 2023-06-10T18:53:55.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc777fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc778069e70 0x7fc7780ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60986/0 (socket says 172.21.15.104:60986) 2023-06-10T18:53:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc7777fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc77806e0e0 0x7fc77806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc777fff700 1 -- 172.21.15.104:0/2677036336 learned_addr learned my addr 172.21.15.104:0/2677036336 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc7777fe700 1 -- 172.21.15.104:0/2677036336 >> v1:172.21.15.114:6789/0 conn(0x7fc77806e4b0 legacy=0x7fc7780c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc7777fe700 1 -- 172.21.15.104:0/2677036336 >> v1:172.21.15.104:6789/0 conn(0x7fc778069e70 legacy=0x7fc7780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc7777fe700 1 -- 172.21.15.104:0/2677036336 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77806d4c0 con 0x7fc77806e0e0 2023-06-10T18:53:55.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc776ffd700 1 -- 172.21.15.104:0/2677036336 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050745308 0 0) 0x7fc77806d290 con 0x7fc778069e70 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc7777fe700 1 --2- 172.21.15.104:0/2677036336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77806e0e0 0x7fc77806cd50 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc768009820 tx=0x7fc768009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa62273f7a7792d1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc776ffd700 1 -- 172.21.15.104:0/2677036336 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768005e30 con 0x7fc77806e0e0 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc776ffd700 1 -- 172.21.15.104:0/2677036336 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc768006030 con 0x7fc77806e0e0 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.053+0000 7fc776ffd700 1 -- 172.21.15.104:0/2677036336 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680063e0 con 0x7fc77806e0e0 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 -- 172.21.15.104:0/2677036336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77806e0e0 msgr2=0x7fc77806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 --2- 172.21.15.104:0/2677036336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77806e0e0 0x7fc77806cd50 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc768009820 tx=0x7fc768009b30 comp rx=0 tx=0).stop 2023-06-10T18:53:55.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 -- 172.21.15.104:0/2677036336 shutdown_connections 2023-06-10T18:53:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 --2- 172.21.15.104:0/2677036336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc77806e0e0 0x7fc77806cd50 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 -- 172.21.15.104:0/2677036336 >> 172.21.15.104:0/2677036336 conn(0x7fc77806b8a0 msgr2=0x7fc7780cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 -- 172.21.15.104:0/2677036336 shutdown_connections 2023-06-10T18:53:55.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.054+0000 7fc77db7b700 1 -- 172.21.15.104:0/2677036336 wait complete. 2023-06-10T18:53:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.055+0000 7fc77db7b700 1 Processor -- start 2023-06-10T18:53:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.055+0000 7fc77db7b700 1 -- start start 2023-06-10T18:53:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77806e0e0 0x7fc7780b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc77806e4b0 0x7fc7780c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780b4f20 con 0x7fc778069e70 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780b42a0 con 0x7fc77806e0e0 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc77db7b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780b4580 con 0x7fc77806e4b0 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34928/0 (socket says 172.21.15.104:34928) 2023-06-10T18:53:55.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 -- 172.21.15.104:0/1315584393 learned_addr learned my addr 172.21.15.104:0/1315584393 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 -- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc77806e4b0 msgr2=0x7fc7780c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc77806e4b0 0x7fc7780c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 -- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77806e0e0 msgr2=0x7fc7780b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77806e0e0 0x7fc7780b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 -- 172.21.15.104:0/1315584393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768009430 con 0x7fc778069e70 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.056+0000 7fc7777fe700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fc768009b70 tx=0x7fc768004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.057+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768004240 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.057+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7680043c0 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.057+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680036a0 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.057+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7780b4860 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.057+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7780b3e90 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.058+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7780b4020 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.058+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc768022070 con 0x7fc778069e70 2023-06-10T18:53:55.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.058+0000 7fc774ff9700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 0x7fc75806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.058+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fc76808b040 con 0x7fc778069e70 2023-06-10T18:53:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.059+0000 7fc76ffff700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 0x7fc75806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.059+0000 7fc76ffff700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 0x7fc75806d190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc760000f60 tx=0x7fc760007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.063+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc768055fe0 con 0x7fc778069e70 2023-06-10T18:53:55.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.249+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc778074050 con 0x7fc778069e70 2023-06-10T18:53:55.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.251+0000 7fc774ff9700 1 -- 172.21.15.104:0/1315584393 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+18675 (secure 0 0 0) 0x7fc768059600 con 0x7fc778069e70 2023-06-10T18:53:55.256 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:55.257 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":100,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:53:51.686953+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:53:55.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 msgr2=0x7fc75806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 0x7fc75806d190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc760000f60 tx=0x7fc760007a40 comp rx=0 tx=0).stop 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 msgr2=0x7fc778157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fc768009b70 tx=0x7fc768004750 comp rx=0 tx=0).stop 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 shutdown_connections 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc77806e4b0 0x7fc7780c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc778069e70 0x7fc778157eb0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc75806ace0 0x7fc75806d190 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 --2- 172.21.15.104:0/1315584393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77806e0e0 0x7fc7780b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 >> 172.21.15.104:0/1315584393 conn(0x7fc77806b8a0 msgr2=0x7fc7780ac230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 shutdown_connections 2023-06-10T18:53:55.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.253+0000 7fc77db7b700 1 -- 172.21.15.104:0/1315584393 wait complete. 2023-06-10T18:53:55.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:53:55.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:53:55.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 Processor -- start 2023-06-10T18:53:55.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 -- start start 2023-06-10T18:53:55.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 0x7f8ea00ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ea00d0a00 0x7f8ea00b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00ad3c0 con 0x7f8ea00ce880 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea00d0e40 con 0x7f8ea00d0a00 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.388+0000 7f8ea51ee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00cecf0 con 0x7f8ea0157b30 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 0x7f8ea00ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9e59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8ea00ce880 0x7f8ea00d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:38610/0 (socket says 172.21.15.104:38610) 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9e59c700 1 -- 172.21.15.104:0/1927885933 learned_addr learned my addr 172.21.15.104:0/1927885933 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9f59e700 1 --2- 172.21.15.104:0/1927885933 >> v2:172.21.15.114:3300/0 conn(0x7f8ea00d0a00 0x7f8ea00b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9dd9b700 1 -- 172.21.15.104:0/1927885933 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292586825 0 0) 0x7f8ea00ad3c0 con 0x7f8ea00ce880 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9dd9b700 1 -- 172.21.15.104:0/1927885933 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e880035b0 con 0x7f8ea00ce880 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 -- 172.21.15.104:0/1927885933 >> v2:172.21.15.114:3300/0 conn(0x7f8ea00d0a00 msgr2=0x7f8ea00b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 --2- 172.21.15.104:0/1927885933 >> v2:172.21.15.114:3300/0 conn(0x7f8ea00d0a00 0x7f8ea00b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 -- 172.21.15.104:0/1927885933 >> v1:172.21.15.114:6789/0 conn(0x7f8ea00ce880 legacy=0x7f8ea00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 -- 172.21.15.104:0/1927885933 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea0157f00 con 0x7f8ea0157b30 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.389+0000 7f8e9ed9d700 1 --2- 172.21.15.104:0/1927885933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 0x7f8ea00ce340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009840 tx=0x7f8e90009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=970dbb7530e7b6f4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8e9dd9b700 1 -- 172.21.15.104:0/1927885933 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90011830 con 0x7f8ea0157b30 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8e9dd9b700 1 -- 172.21.15.104:0/1927885933 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90011990 con 0x7f8ea0157b30 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8e9dd9b700 1 -- 172.21.15.104:0/1927885933 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90011d00 con 0x7f8ea0157b30 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 -- 172.21.15.104:0/1927885933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 msgr2=0x7f8ea00ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/1927885933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 0x7f8ea00ce340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009840 tx=0x7f8e90009b50 comp rx=0 tx=0).stop 2023-06-10T18:53:55.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 -- 172.21.15.104:0/1927885933 shutdown_connections 2023-06-10T18:53:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/1927885933 >> v2:172.21.15.114:3300/0 conn(0x7f8ea00d0a00 0x7f8ea00b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/1927885933 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea0157b30 0x7f8ea00ce340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 -- 172.21.15.104:0/1927885933 >> 172.21.15.104:0/1927885933 conn(0x7f8ea0089fd0 msgr2=0x7f8ea008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 -- 172.21.15.104:0/1927885933 shutdown_connections 2023-06-10T18:53:55.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.390+0000 7f8ea51ee700 1 -- 172.21.15.104:0/1927885933 wait complete. 2023-06-10T18:53:55.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 Processor -- start 2023-06-10T18:53:55.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 -- start start 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea00ce880 0x7f8ea0151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea00d0a00 0x7f8ea00c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b35a0 con 0x7f8ea0157b30 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00b1000 con 0x7f8ea00d0a00 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8ea51ee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00c6b30 con 0x7f8ea00ce880 2023-06-10T18:53:55.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8e9f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8e9f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34946/0 (socket says 172.21.15.104:34946) 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.392+0000 7f8e9f59e700 1 -- 172.21.15.104:0/2632079247 learned_addr learned my addr 172.21.15.104:0/2632079247 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9e59c700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea00d0a00 0x7f8ea00c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9ed9d700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea00ce880 0x7f8ea0151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 -- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea00ce880 msgr2=0x7f8ea0151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea00ce880 0x7f8ea0151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 -- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea00d0a00 msgr2=0x7f8ea00c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea00d0a00 0x7f8ea00c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 -- 172.21.15.104:0/2632079247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900094a0 con 0x7f8ea0157b30 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e9f59e700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800efc0 tx=0x7f8e9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98009ea0 con 0x7f8ea0157b30 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9800f040 con 0x7f8ea0157b30 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea00c6e10 con 0x7f8ea0157b30 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9800ced0 con 0x7f8ea0157b30 2023-06-10T18:53:55.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.393+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea00b1580 con 0x7f8ea0157b30 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.394+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea00b17a0 con 0x7f8ea0157b30 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.395+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e98014480 con 0x7f8ea0157b30 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.395+0000 7f8e877fe700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 0x7f8e8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.395+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8e9808a180 con 0x7f8ea0157b30 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.395+0000 7f8e9ed9d700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 0x7f8e8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.396+0000 7f8e9ed9d700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 0x7f8e8c06d190 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8e900090b0 tx=0x7f8e90008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.399+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e98055010 con 0x7f8ea0157b30 2023-06-10T18:53:55.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.586+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ea0074050 con 0x7f8ea0157b30 2023-06-10T18:53:55.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.586+0000 7f8e877fe700 1 -- 172.21.15.104:0/2632079247 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e98058630 con 0x7f8ea0157b30 2023-06-10T18:53:55.592 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:53:55.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 msgr2=0x7f8e8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 0x7f8e8c06d190 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8e900090b0 tx=0x7f8e90008030 comp rx=0 tx=0).stop 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 msgr2=0x7f8ea00c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800efc0 tx=0x7f8e9800c3a0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 shutdown_connections 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea0157b30 0x7f8ea00c4ad0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea00ce880 0x7f8ea0151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e8c06ace0 0x7f8e8c06d190 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 --2- 172.21.15.104:0/2632079247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea00d0a00 0x7f8ea00c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.588+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 >> 172.21.15.104:0/2632079247 conn(0x7f8ea0089fd0 msgr2=0x7f8ea015b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.589+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 shutdown_connections 2023-06-10T18:53:55.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.589+0000 7f8ea51ee700 1 -- 172.21.15.104:0/2632079247 wait complete. 2023-06-10T18:53:55.606 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:53:55.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.724+0000 7fb83dd6a700 1 Processor -- start 2023-06-10T18:53:55.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.724+0000 7fb83dd6a700 1 -- start start 2023-06-10T18:53:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb83dd6a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb83806cde0 0x7fb83806d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb83dd6a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8380ca850 0x7fb8380cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb83dd6a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8380d0480 con 0x7fb8380aec40 2023-06-10T18:53:55.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb83dd6a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83806d7a0 con 0x7fb8380ca850 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb83dd6a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83815dd00 con 0x7fb83806cde0 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8380ca850 0x7fb8380cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb8377fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb8380aec40 0x7fb838069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32778/0 (socket says 172.21.15.104:32778) 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8380ca850 0x7fb8380cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34954/0 (socket says 172.21.15.104:34954) 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb8377fe700 1 -- 172.21.15.104:0/1683872173 learned_addr learned my addr 172.21.15.104:0/1683872173 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb836ffd700 1 --2- 172.21.15.104:0/1683872173 >> v2:172.21.15.114:3300/0 conn(0x7fb83806cde0 0x7fb83806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 -- 172.21.15.104:0/1683872173 >> v2:172.21.15.114:3300/0 conn(0x7fb83806cde0 msgr2=0x7fb83806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 --2- 172.21.15.104:0/1683872173 >> v2:172.21.15.114:3300/0 conn(0x7fb83806cde0 0x7fb83806d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 -- 172.21.15.104:0/1683872173 >> v1:172.21.15.104:6789/0 conn(0x7fb8380aec40 legacy=0x7fb838069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb8367fc700 1 -- 172.21.15.104:0/1683872173 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874228766 0 0) 0x7fb8380d0480 con 0x7fb8380aec40 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 -- 172.21.15.104:0/1683872173 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83815a610 con 0x7fb8380ca850 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.725+0000 7fb837fff700 1 --2- 172.21.15.104:0/1683872173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380ca850 0x7fb8380cac90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb838071210 tx=0x7fb83000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adc7b2536c036f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb8367fc700 1 -- 172.21.15.104:0/1683872173 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83000e070 con 0x7fb8380ca850 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb8367fc700 1 -- 172.21.15.104:0/1683872173 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83000bbd0 con 0x7fb8380ca850 2023-06-10T18:53:55.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb8367fc700 1 -- 172.21.15.104:0/1683872173 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb830012430 con 0x7fb8380ca850 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 -- 172.21.15.104:0/1683872173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380ca850 msgr2=0x7fb8380cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/1683872173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380ca850 0x7fb8380cac90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fb838071210 tx=0x7fb83000b330 comp rx=0 tx=0).stop 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 -- 172.21.15.104:0/1683872173 shutdown_connections 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/1683872173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380ca850 0x7fb8380cac90 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/1683872173 >> v2:172.21.15.114:3300/0 conn(0x7fb83806cde0 0x7fb83806d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 -- 172.21.15.104:0/1683872173 >> 172.21.15.104:0/1683872173 conn(0x7fb83806b8a0 msgr2=0x7fb8380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 -- 172.21.15.104:0/1683872173 shutdown_connections 2023-06-10T18:53:55.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.726+0000 7fb83dd6a700 1 -- 172.21.15.104:0/1683872173 wait complete. 2023-06-10T18:53:55.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.727+0000 7fb83dd6a700 1 Processor -- start 2023-06-10T18:53:55.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 -- start start 2023-06-10T18:53:55.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 0x7fb8380c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8380ca850 0x7fb8380b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381efb80 con 0x7fb8380aec40 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381f0b20 con 0x7fb83806cde0 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb83dd6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381efd60 con 0x7fb8380ca850 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb836ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb836ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34958/0 (socket says 172.21.15.104:34958) 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.728+0000 7fb836ffd700 1 -- 172.21.15.104:0/604052361 learned_addr learned my addr 172.21.15.104:0/604052361 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:55.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb8377fe700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 0x7fb8380c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb837fff700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8380ca850 0x7fb8380b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 -- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8380ca850 msgr2=0x7fb8380b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8380ca850 0x7fb8380b4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 -- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 msgr2=0x7fb8380c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 0x7fb8380c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 -- 172.21.15.104:0/604052361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830009a70 con 0x7fb8380aec40 2023-06-10T18:53:55.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb8377fe700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 0x7fb8380c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb836ffd700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fb82800d800 tx=0x7fb82800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280125f0 con 0x7fb8380aec40 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb828012750 con 0x7fb8380aec40 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8381f0040 con 0x7fb8380aec40 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828012b00 con 0x7fb8380aec40 2023-06-10T18:53:55.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.729+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8381f0600 con 0x7fb8380aec40 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.730+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb838074050 con 0x7fb8380aec40 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.730+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb828012c60 con 0x7fb8380aec40 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.731+0000 7fb82ffff700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 0x7fb81806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.731+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fb828089a40 con 0x7fb8380aec40 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.731+0000 7fb8377fe700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 0x7fb81806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:55.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.732+0000 7fb8377fe700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 0x7fb81806d190 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb8380c6590 tx=0x7fb820009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:55.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.735+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb82808be60 con 0x7fb8380aec40 2023-06-10T18:53:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.919+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8380cfd00 con 0x7fb8380aec40 2023-06-10T18:53:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.920+0000 7fb82ffff700 1 -- 172.21.15.104:0/604052361 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb828054950 con 0x7fb8380aec40 2023-06-10T18:53:55.926 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:53:55.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 msgr2=0x7fb81806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 0x7fb81806d190 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb8380c6590 tx=0x7fb820009040 comp rx=0 tx=0).stop 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 msgr2=0x7fb8380b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fb82800d800 tx=0x7fb82800dbc0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 shutdown_connections 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb81806ace0 0x7fb81806d190 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb83806cde0 0x7fb8380c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8380ca850 0x7fb8380b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 --2- 172.21.15.104:0/604052361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8380aec40 0x7fb8380b4880 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 >> 172.21.15.104:0/604052361 conn(0x7fb83806b8a0 msgr2=0x7fb8380adc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 shutdown_connections 2023-06-10T18:53:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:55.922+0000 7fb83dd6a700 1 -- 172.21.15.104:0/604052361 wait complete. 2023-06-10T18:53:55.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:53:56.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.056+0000 7f4d8f126700 1 Processor -- start 2023-06-10T18:53:56.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.056+0000 7f4d8f126700 1 -- start start 2023-06-10T18:53:56.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d8f126700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4d881424d0 0x7f4d880c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d8f126700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d8806d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d8f126700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880cec90 con 0x7f4d88142980 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d8f126700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8806dd70 con 0x7f4d881424d0 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d8f126700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880cd7c0 con 0x7f4d8806d3b0 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d87fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4d88142980 0x7f4d880ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32792/0 (socket says 172.21.15.104:32792) 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.057+0000 7f4d87fff700 1 -- 172.21.15.104:0/2673493480 learned_addr learned my addr 172.21.15.104:0/2673493480 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/2673493480 >> v2:172.21.15.114:3300/0 conn(0x7f4d881424d0 0x7f4d880c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d8806d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245991104 0 0) 0x7f4d880cec90 con 0x7f4d88142980 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d780035b0 con 0x7f4d88142980 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492073127 0 0) 0x7f4d780035b0 con 0x7f4d88142980 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 msgr2=0x7f4d8806d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d8806d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 >> v2:172.21.15.114:3300/0 conn(0x7f4d881424d0 msgr2=0x7f4d880c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 --2- 172.21.15.104:0/2673493480 >> v2:172.21.15.114:3300/0 conn(0x7f4d881424d0 0x7f4d880c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880cdaa0 con 0x7f4d88142980 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4d74001670 con 0x7f4d88142980 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 >> v1:172.21.15.104:6789/0 conn(0x7f4d88142980 legacy=0x7f4d880ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d78003a00 0x7f4d78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d78004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8806dd70 con 0x7f4d78004d90 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880cd7c0 con 0x7f4d78003a00 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7800ba10 con 0x7f4d8806d3b0 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/2673493480 >> v2:172.21.15.114:3300/0 conn(0x7f4d881424d0 0x7f4d880c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d87fff700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d78004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d78003a00 0x7f4d78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 -- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 msgr2=0x7f4d78004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d78004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 -- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d78003a00 msgr2=0x7f4d78003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d78003a00 0x7f4d78003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 -- 172.21.15.104:0/2673493480 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780035b0 con 0x7f4d78004d90 2023-06-10T18:53:56.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d87fff700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d8806d3b0 0x7f4d78004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.058+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d78005200 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009fc0 tx=0x7f4d78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ec95bd4904ff68 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8000ec30 con 0x7f4d78004d90 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8000f040 con 0x7f4d78004d90 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d877fe700 1 -- 172.21.15.104:0/2673493480 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80014480 con 0x7f4d78004d90 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 -- 172.21.15.104:0/2673493480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 msgr2=0x7f4d78005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 --2- 172.21.15.104:0/2673493480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d78005200 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009fc0 tx=0x7f4d78005880 comp rx=0 tx=0).stop 2023-06-10T18:53:56.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8d6c3700 1 -- 172.21.15.104:0/2673493480 reap_dead start 2023-06-10T18:53:56.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 -- 172.21.15.104:0/2673493480 shutdown_connections 2023-06-10T18:53:56.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 -- 172.21.15.104:0/2673493480 >> 172.21.15.104:0/2673493480 conn(0x7f4d880ceec0 msgr2=0x7f4d880cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 -- 172.21.15.104:0/2673493480 shutdown_connections 2023-06-10T18:53:56.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.059+0000 7f4d8f126700 1 -- 172.21.15.104:0/2673493480 wait complete. 2023-06-10T18:53:56.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.060+0000 7f4d8f126700 1 Processor -- start 2023-06-10T18:53:56.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.060+0000 7f4d8f126700 1 -- start start 2023-06-10T18:53:56.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d88151cb0 0x7f4d8815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d880ca3a0 0x7f4d880ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8813c180 con 0x7f4d78004d90 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880cad80 con 0x7f4d88151cb0 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8f126700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881f5d90 con 0x7f4d880ca3a0 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8cec2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8cec2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34974/0 (socket says 172.21.15.104:34974) 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.061+0000 7f4d8cec2700 1 -- 172.21.15.104:0/3972148595 learned_addr learned my addr 172.21.15.104:0/3972148595 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8d6c3700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d880ca3a0 0x7f4d880ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d87fff700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d88151cb0 0x7f4d8815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 -- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d880ca3a0 msgr2=0x7f4d880ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d880ca3a0 0x7f4d880ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 -- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d88151cb0 msgr2=0x7f4d8815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d88151cb0 0x7f4d8815cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 -- 172.21.15.104:0/3972148595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d881f5fc0 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8cec2700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c010a80 tx=0x7f4d7c010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00e930 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c00ea90 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d881f62a0 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00ee20 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.062+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8815a420 con 0x7f4d78004d90 2023-06-10T18:53:56.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.063+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d881f67c0 con 0x7f4d78004d90 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.063+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4d7c016460 con 0x7f4d78004d90 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.064+0000 7f4d857fa700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 0x7f4d6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.064+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4d7c089b40 con 0x7f4d78004d90 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.064+0000 7f4d87fff700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 0x7f4d6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.064+0000 7f4d87fff700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 0x7f4d6c06d190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4d88152320 tx=0x7f4d74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.068+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4d7c054a50 con 0x7f4d78004d90 2023-06-10T18:53:56.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.255+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d880cd430 con 0x7f4d78004d90 2023-06-10T18:53:56.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.255+0000 7f4d857fa700 1 -- 172.21.15.104:0/3972148595 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d7c058070 con 0x7f4d78004d90 2023-06-10T18:53:56.261 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:53:56.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 msgr2=0x7f4d6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 0x7f4d6c06d190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4d88152320 tx=0x7f4d74009040 comp rx=0 tx=0).stop 2023-06-10T18:53:56.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 msgr2=0x7f4d881f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c010a80 tx=0x7f4d7c010e40 comp rx=0 tx=0).stop 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 shutdown_connections 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d880ca3a0 0x7f4d880ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78004d90 0x7f4d881f5850 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6c06ace0 0x7f4d6c06d190 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 --2- 172.21.15.104:0/3972148595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d88151cb0 0x7f4d8815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 >> 172.21.15.104:0/3972148595 conn(0x7f4d880ceec0 msgr2=0x7f4d88159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 shutdown_connections 2023-06-10T18:53:56.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.257+0000 7f4d8f126700 1 -- 172.21.15.104:0/3972148595 wait complete. 2023-06-10T18:53:56.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:53:56.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.392+0000 7fd091b27700 1 Processor -- start 2023-06-10T18:53:56.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.392+0000 7fd091b27700 1 -- start start 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd091b27700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd08c15c580 0x7fd08c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd091b27700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 0x7fd08c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd091b27700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08c06e250 con 0x7fd08c06e6e0 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd091b27700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c15c950 con 0x7fd08c15c580 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd091b27700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0b1e80 con 0x7fd08c06cde0 2023-06-10T18:53:56.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd08c15c580 0x7fd08c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd08c15c580 0x7fd08c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34982/0 (socket says 172.21.15.104:34982) 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 -- 172.21.15.104:0/3812749204 learned_addr learned my addr 172.21.15.104:0/3812749204 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 -- 172.21.15.104:0/3812749204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 msgr2=0x7fd08c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08bfff700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 0x7fd08c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 0x7fd08c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 -- 172.21.15.104:0/3812749204 >> v1:172.21.15.114:6789/0 conn(0x7fd08c06e6e0 legacy=0x7fd08c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08b7fe700 1 -- 172.21.15.104:0/3812749204 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c06d920 con 0x7fd08c15c580 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.393+0000 7fd08bfff700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 0x7fd08c0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd08b7fe700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0b1070 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd080006fd0 tx=0x7fd080009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f413f5bcc4ac315 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd08a7fc700 1 -- 172.21.15.104:0/3812749204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080005e30 con 0x7fd08c15c580 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd08a7fc700 1 -- 172.21.15.104:0/3812749204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080006030 con 0x7fd08c15c580 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd08a7fc700 1 -- 172.21.15.104:0/3812749204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800063c0 con 0x7fd08c15c580 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 -- 172.21.15.104:0/3812749204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 msgr2=0x7fd08c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0b1070 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd080006fd0 tx=0x7fd080009bd0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 -- 172.21.15.104:0/3812749204 shutdown_connections 2023-06-10T18:53:56.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0b1070 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 --2- 172.21.15.104:0/3812749204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06cde0 0x7fd08c0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 -- 172.21.15.104:0/3812749204 >> 172.21.15.104:0/3812749204 conn(0x7fd08c06b8a0 msgr2=0x7fd08c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 -- 172.21.15.104:0/3812749204 shutdown_connections 2023-06-10T18:53:56.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.394+0000 7fd091b27700 1 -- 172.21.15.104:0/3812749204 wait complete. 2023-06-10T18:53:56.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.396+0000 7fd091b27700 1 Processor -- start 2023-06-10T18:53:56.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.396+0000 7fd091b27700 1 -- start start 2023-06-10T18:53:56.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.396+0000 7fd091b27700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd08c06cde0 0x7fd08c0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06e6e0 0x7fd08c0c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0b3cd0 con 0x7fd08c15c580 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0c3090 con 0x7fd08c06cde0 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0c72b0 con 0x7fd08c06e6e0 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34990/0 (socket says 172.21.15.104:34990) 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd08c06cde0 0x7fd08c0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 -- 172.21.15.104:0/3013341846 learned_addr learned my addr 172.21.15.104:0/3013341846 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08affd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06e6e0 0x7fd08c0c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 -- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06e6e0 msgr2=0x7fd08c0c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06e6e0 0x7fd08c0c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 -- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd08c06cde0 msgr2=0x7fd08c0b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd08c06cde0 0x7fd08c0b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 -- 172.21.15.104:0/3013341846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080009500 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd08bfff700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd08400cef0 tx=0x7fd08400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08400ad00 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08400ae60 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c155800 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084015680 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.397+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c155dc0 con 0x7fd08c15c580 2023-06-10T18:53:56.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.398+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08c074050 con 0x7fd08c15c580 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.399+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd08400d430 con 0x7fd08c15c580 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.399+0000 7fd073fff700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 0x7fd07806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.399+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fd084089a50 con 0x7fd08c15c580 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.399+0000 7fd08b7fe700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 0x7fd07806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.400+0000 7fd08b7fe700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 0x7fd07806d190 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd080013040 tx=0x7fd080008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.403+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd0840549f0 con 0x7fd08c15c580 2023-06-10T18:53:56.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.589+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd08c156180 con 0x7fd08c15c580 2023-06-10T18:53:56.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.590+0000 7fd073fff700 1 -- 172.21.15.104:0/3013341846 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd08401a0a0 con 0x7fd08c15c580 2023-06-10T18:53:56.596 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 msgr2=0x7fd07806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 0x7fd07806d190 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd080013040 tx=0x7fd080008070 comp rx=0 tx=0).stop 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 msgr2=0x7fd08c0c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd08400cef0 tx=0x7fd08400cf20 comp rx=0 tx=0).stop 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 shutdown_connections 2023-06-10T18:53:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd07806ace0 0x7fd07806d190 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd08c06cde0 0x7fd08c0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd08c15c580 0x7fd08c0c5890 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 --2- 172.21.15.104:0/3013341846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd08c06e6e0 0x7fd08c0c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 >> 172.21.15.104:0/3013341846 conn(0x7fd08c06b8a0 msgr2=0x7fd08c159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 shutdown_connections 2023-06-10T18:53:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.592+0000 7fd091b27700 1 -- 172.21.15.104:0/3013341846 wait complete. 2023-06-10T18:53:56.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:53:56.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.726+0000 7f3366ed3700 1 Processor -- start 2023-06-10T18:53:56.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.726+0000 7f3366ed3700 1 -- start start 2023-06-10T18:53:56.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3366ed3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3360158c50 0x7f336006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3366ed3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f33600cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3366ed3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600cccd0 con 0x7f336006b180 2023-06-10T18:53:56.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3366ed3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33600ccfb0 con 0x7f3360158c50 2023-06-10T18:53:56.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3366ed3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600ce0b0 con 0x7f336006d150 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f336006b180 0x7f336006ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32822/0 (socket says 172.21.15.104:32822) 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335ffff700 1 -- 172.21.15.104:0/795762571 learned_addr learned my addr 172.21.15.104:0/795762571 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659113979 0 0) 0x7f33600cccd0 con 0x7f336006b180 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33500035b0 con 0x7f336006b180 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3365470700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f33600cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f3364c6f700 1 --2- 172.21.15.104:0/795762571 >> v2:172.21.15.114:3300/0 conn(0x7f3360158c50 0x7f336006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971199589 0 0) 0x7f33500035b0 con 0x7f336006b180 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 msgr2=0x7f33600cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.727+0000 7f335f7fe700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f33600cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 >> v2:172.21.15.114:3300/0 conn(0x7f3360158c50 msgr2=0x7f336006aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 --2- 172.21.15.104:0/795762571 >> v2:172.21.15.114:3300/0 conn(0x7f3360158c50 0x7f336006aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336006a010 con 0x7f336006b180 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f334c001670 con 0x7f336006b180 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 >> v1:172.21.15.104:6789/0 conn(0x7f336006b180 legacy=0x7f336006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350003a00 0x7f3350003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3350004850 0x7f3350004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f3350005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600ccfb0 con 0x7f3350004850 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335000bac0 con 0x7f3350003a00 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335000bc80 con 0x7f336006d150 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f3364c6f700 1 --2- 172.21.15.104:0/795762571 >> v2:172.21.15.114:3300/0 conn(0x7f3360158c50 0x7f336006aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:56.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3350004850 0x7f3350004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f3365470700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f3350005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f3364c6f700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350003a00 0x7f3350003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 -- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 msgr2=0x7f3350005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f336006d150 0x7f3350005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 -- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350003a00 msgr2=0x7f3350003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350003a00 0x7f3350003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 -- 172.21.15.104:0/795762571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33500035b0 con 0x7f3350004850 2023-06-10T18:53:56.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335ffff700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3350004850 0x7f3350004cc0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c2f0 tx=0x7f3350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6562f17b702db071 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00ce30 con 0x7f3350004850 2023-06-10T18:53:56.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c005960 con 0x7f3350004850 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.728+0000 7f335f7fe700 1 -- 172.21.15.104:0/795762571 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c005d10 con 0x7f3350004850 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 -- 172.21.15.104:0/795762571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3350004850 msgr2=0x7f3350004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 --2- 172.21.15.104:0/795762571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3350004850 0x7f3350004cc0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c2f0 tx=0x7f3350005880 comp rx=0 tx=0).stop 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3365470700 1 -- 172.21.15.104:0/795762571 reap_dead start 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 -- 172.21.15.104:0/795762571 shutdown_connections 2023-06-10T18:53:56.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 -- 172.21.15.104:0/795762571 >> 172.21.15.104:0/795762571 conn(0x7f33600b1610 msgr2=0x7f33600ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 -- 172.21.15.104:0/795762571 shutdown_connections 2023-06-10T18:53:56.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.729+0000 7f3366ed3700 1 -- 172.21.15.104:0/795762571 wait complete. 2023-06-10T18:53:56.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.730+0000 7f3366ed3700 1 Processor -- start 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.730+0000 7f3366ed3700 1 -- start start 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350004850 0x7f3360153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33601ee890 0x7f33600add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c6b30 con 0x7f33600c6170 2023-06-10T18:53:56.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600b43e0 con 0x7f3350004850 2023-06-10T18:53:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3366ed3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33601eed00 con 0x7f33601ee890 2023-06-10T18:53:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35010/0 (socket says 172.21.15.104:35010) 2023-06-10T18:53:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 -- 172.21.15.104:0/3268772252 learned_addr learned my addr 172.21.15.104:0/3268772252 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:56.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3364c6f700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350004850 0x7f3360153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f335ffff700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33601ee890 0x7f33600add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 -- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33601ee890 msgr2=0x7f33600add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33601ee890 0x7f33600add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 -- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350004850 msgr2=0x7f3360153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350004850 0x7f3360153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.731+0000 7f3365470700 1 -- 172.21.15.104:0/3268772252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33601f1010 con 0x7f33600c6170 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f3365470700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f336006d540 tx=0x7f3358004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358007ab0 con 0x7f33600c6170 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358007c10 con 0x7f33600c6170 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33601f12f0 con 0x7f33600c6170 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358015420 con 0x7f33600c6170 2023-06-10T18:53:56.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.732+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33601f18b0 con 0x7f33600c6170 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.733+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3360074050 con 0x7f33600c6170 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.733+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3358015580 con 0x7f33600c6170 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.734+0000 7f335d7fa700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 0x7f334406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.734+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3358089d10 con 0x7f33600c6170 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.734+0000 7f3364c6f700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 0x7f334406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.734+0000 7f3364c6f700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 0x7f334406d190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f33601571c0 tx=0x7f3354008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:56.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.738+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3358054b80 con 0x7f33600c6170 2023-06-10T18:53:56.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.938+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3360156c30 con 0x7f33600c6170 2023-06-10T18:53:56.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.939+0000 7f335d7fa700 1 -- 172.21.15.104:0/3268772252 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v100) v1 ==== 80+0+124 (secure 0 0 0) 0x7f335808d090 con 0x7f33600c6170 2023-06-10T18:53:56.945 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6837/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:53:56.945 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6835/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:53:56.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 msgr2=0x7f334406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 0x7f334406d190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f33601571c0 tx=0x7f3354008040 comp rx=0 tx=0).stop 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 msgr2=0x7f33600c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f336006d540 tx=0x7f3358004970 comp rx=0 tx=0).stop 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 shutdown_connections 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f334406ace0 0x7f334406d190 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3350004850 0x7f3360153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33600c6170 0x7f33600c65f0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 --2- 172.21.15.104:0/3268772252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33601ee890 0x7f33600add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:56.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 >> 172.21.15.104:0/3268772252 conn(0x7f33600b1610 msgr2=0x7f33600af240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:56.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 shutdown_connections 2023-06-10T18:53:56.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:56.941+0000 7f3366ed3700 1 -- 172.21.15.104:0/3268772252 wait complete. 2023-06-10T18:53:56.949 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:53:56.958 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:53:56.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:56.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:53:56.997 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:53:56.998 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:53:56.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:53:56.998 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:53:57.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2023-06-10T18:53:57.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.211+0000 7f1ad001c700 1 Processor -- start 2023-06-10T18:53:57.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.211+0000 7f1ad001c700 1 -- start start 2023-06-10T18:53:57.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ad001c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 0x7f1ac80b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ad001c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 0x7f1ac80ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ad001c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac806e250 con 0x7f1ac815c580 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ad001c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac815c950 con 0x7f1ac806e6e0 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ad001c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80b1e80 con 0x7f1ac806cde0 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1acd5b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 0x7f1ac80b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1ace5b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 0x7f1ac80ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1acddb8700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1ac815c580 0x7f1ac80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:38688/0 (socket says 172.21.15.104:38688) 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.212+0000 7f1acddb8700 1 -- 172.21.15.104:0/1677430695 learned_addr learned my addr 172.21.15.104:0/1677430695 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1ace5b9700 1 -- 172.21.15.104:0/1677430695 >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 msgr2=0x7f1ac80b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1ace5b9700 1 --2- 172.21.15.104:0/1677430695 >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 0x7f1ac80b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1ace5b9700 1 -- 172.21.15.104:0/1677430695 >> v1:172.21.15.114:6789/0 conn(0x7f1ac815c580 legacy=0x7f1ac80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1ace5b9700 1 -- 172.21.15.104:0/1677430695 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac806d920 con 0x7f1ac806cde0 2023-06-10T18:53:57.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1acd5b7700 1 --2- 172.21.15.104:0/1677430695 >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 0x7f1ac80b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:57.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1ace5b9700 1 --2- 172.21.15.104:0/1677430695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 0x7f1ac80ce0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009d90 tx=0x7f1ac000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=190347a2689b37c8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1accdb6700 1 -- 172.21.15.104:0/1677430695 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac000e070 con 0x7f1ac806cde0 2023-06-10T18:53:57.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1accdb6700 1 -- 172.21.15.104:0/1677430695 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac000bb20 con 0x7f1ac806cde0 2023-06-10T18:53:57.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.213+0000 7f1accdb6700 1 -- 172.21.15.104:0/1677430695 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0012430 con 0x7f1ac806cde0 2023-06-10T18:53:57.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 -- 172.21.15.104:0/1677430695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 msgr2=0x7f1ac80ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 --2- 172.21.15.104:0/1677430695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 0x7f1ac80ce0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009d90 tx=0x7f1ac000b330 comp rx=0 tx=0).stop 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 -- 172.21.15.104:0/1677430695 shutdown_connections 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 --2- 172.21.15.104:0/1677430695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806cde0 0x7f1ac80ce0d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 --2- 172.21.15.104:0/1677430695 >> v2:172.21.15.114:3300/0 conn(0x7f1ac806e6e0 0x7f1ac80b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 -- 172.21.15.104:0/1677430695 >> 172.21.15.104:0/1677430695 conn(0x7f1ac806b8a0 msgr2=0x7f1ac80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 -- 172.21.15.104:0/1677430695 shutdown_connections 2023-06-10T18:53:57.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.214+0000 7f1ad001c700 1 -- 172.21.15.104:0/1677430695 wait complete. 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.215+0000 7f1ad001c700 1 Processor -- start 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.215+0000 7f1ad001c700 1 -- start start 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac806cde0 0x7f1ac81e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806e6e0 0x7f1ac81e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac81e6ce0 con 0x7f1ac815c580 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80ce200 con 0x7f1ac806cde0 2023-06-10T18:53:57.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac81e6720 con 0x7f1ac806e6e0 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35024/0 (socket says 172.21.15.104:35024) 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 -- 172.21.15.104:0/2002408939 learned_addr learned my addr 172.21.15.104:0/2002408939 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1acddb8700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac806cde0 0x7f1ac81e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1acd5b7700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806e6e0 0x7f1ac81e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 -- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806e6e0 msgr2=0x7f1ac81e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806e6e0 0x7f1ac81e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 -- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac806cde0 msgr2=0x7f1ac81e3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac806cde0 0x7f1ac81e3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 -- 172.21.15.104:0/2002408939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80094a0 con 0x7f1ac815c580 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ace5b9700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0014040 tx=0x7f1ac0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0012430 con 0x7f1ac815c580 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0009a70 con 0x7f1ac815c580 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac0020070 con 0x7f1ac815c580 2023-06-10T18:53:57.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.216+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac00098f0 con 0x7f1ac815c580 2023-06-10T18:53:57.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.217+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac80ce4a0 con 0x7f1ac815c580 2023-06-10T18:53:57.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.217+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac8074050 con 0x7f1ac815c580 2023-06-10T18:53:57.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.218+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1ac0004040 con 0x7f1ac815c580 2023-06-10T18:53:57.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.218+0000 7f1abe7fc700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 0x7f1aac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.218+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f1ac0022070 con 0x7f1ac815c580 2023-06-10T18:53:57.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.218+0000 7f1acddb8700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 0x7f1aac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.219+0000 7f1acddb8700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 0x7f1aac06d190 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4005960 tx=0x7f1ac4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.223+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1ac0056510 con 0x7f1ac815c580 2023-06-10T18:53:57.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.450+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f1ac8066f10 con 0x7f1ac815c580 2023-06-10T18:53:57.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.695+0000 7f1abe7fc700 1 -- 172.21.15.104:0/2002408939 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v58) v1 ==== 162+0+0 (secure 0 0 0) 0x7f1ac0059b30 con 0x7f1ac815c580 2023-06-10T18:53:57.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.698+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 msgr2=0x7f1aac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.698+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 0x7f1aac06d190 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4005960 tx=0x7f1ac4009040 comp rx=0 tx=0).stop 2023-06-10T18:53:57.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.698+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 msgr2=0x7f1ac81f51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.698+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0014040 tx=0x7f1ac0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 shutdown_connections 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1aac06ace0 0x7f1aac06d190 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac806cde0 0x7f1ac81e3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ac815c580 0x7f1ac81f51e0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 --2- 172.21.15.104:0/2002408939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ac806e6e0 0x7f1ac81e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 >> 172.21.15.104:0/2002408939 conn(0x7f1ac806b8a0 msgr2=0x7f1ac8159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:57.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 shutdown_connections 2023-06-10T18:53:57.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.699+0000 7f1ad001c700 1 -- 172.21.15.104:0/2002408939 wait complete. 2023-06-10T18:53:57.706 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked joinable; MDS may join as newly active. 2023-06-10T18:53:57.718 INFO:tasks.cephfs.test_client_recovery:Waiting for reconnect... 2023-06-10T18:53:57.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:57.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.835+0000 7f11768ab700 1 Processor -- start 2023-06-10T18:53:57.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.835+0000 7f11768ab700 1 -- start start 2023-06-10T18:53:57.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.835+0000 7f11768ab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f117006a8c0 0x7f11701587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.835+0000 7f11768ab700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1170069e70 con 0x7f1170158ce0 2023-06-10T18:53:57.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f11768ab700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11701595f0 con 0x7f1170070a20 2023-06-10T18:53:57.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f11768ab700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11701598d0 con 0x7f117006a8c0 2023-06-10T18:53:57.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f117006a8c0 0x7f11701587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f117006a8c0 0x7f11701587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35044/0 (socket says 172.21.15.104:35044) 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 -- 172.21.15.104:0/127284197 learned_addr learned my addr 172.21.15.104:0/127284197 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f1174e48700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1170158ce0 0x7f1170159c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32874/0 (socket says 172.21.15.104:32874) 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 -- 172.21.15.104:0/127284197 >> v1:172.21.15.114:6789/0 conn(0x7f1170070a20 legacy=0x7f1170070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 -- 172.21.15.104:0/127284197 >> v1:172.21.15.104:6789/0 conn(0x7f1170158ce0 legacy=0x7f1170159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 -- 172.21.15.104:0/127284197 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117006a0c0 con 0x7f117006a8c0 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116effd700 1 -- 172.21.15.104:0/127284197 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253695721 0 0) 0x7f1170069e70 con 0x7f1170158ce0 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116f7fe700 1 --2- 172.21.15.104:0/127284197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f117006a8c0 0x7f11701587a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f116400aa90 tx=0x7f116400ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50625df31283da4b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116effd700 1 -- 172.21.15.104:0/127284197 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400cb70 con 0x7f117006a8c0 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116effd700 1 -- 172.21.15.104:0/127284197 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116400ccd0 con 0x7f117006a8c0 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.836+0000 7f116effd700 1 -- 172.21.15.104:0/127284197 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164018620 con 0x7f117006a8c0 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 -- 172.21.15.104:0/127284197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f117006a8c0 msgr2=0x7f11701587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 --2- 172.21.15.104:0/127284197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f117006a8c0 0x7f11701587a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f116400aa90 tx=0x7f116400ada0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 -- 172.21.15.104:0/127284197 shutdown_connections 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 --2- 172.21.15.104:0/127284197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f117006a8c0 0x7f11701587a0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 -- 172.21.15.104:0/127284197 >> 172.21.15.104:0/127284197 conn(0x7f11700b1610 msgr2=0x7f11700ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 -- 172.21.15.104:0/127284197 shutdown_connections 2023-06-10T18:53:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.837+0000 7f11768ab700 1 -- 172.21.15.104:0/127284197 wait complete. 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.838+0000 7f11768ab700 1 Processor -- start 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.838+0000 7f11768ab700 1 -- start start 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f117006a8c0 0x7f11700b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1170158ce0 0x7f11700b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170155b40 con 0x7f1170070a20 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11700c2e00 con 0x7f1170158ce0 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f11768ab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700c27f0 con 0x7f117006a8c0 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35050/0 (socket says 172.21.15.104:35050) 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 -- 172.21.15.104:0/2423366281 learned_addr learned my addr 172.21.15.104:0/2423366281 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f1174e48700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1170158ce0 0x7f11700b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116ffff700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f117006a8c0 0x7f11700b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 -- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f117006a8c0 msgr2=0x7f11700b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f117006a8c0 0x7f11700b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 -- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1170158ce0 msgr2=0x7f11700b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1170158ce0 0x7f11700b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.839+0000 7f116f7fe700 1 -- 172.21.15.104:0/2423366281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116400a6f0 con 0x7f1170070a20 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f116f7fe700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f1164000c00 tx=0x7f11640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164021680 con 0x7f1170070a20 2023-06-10T18:53:57.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11640217e0 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11701f5a50 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164021b90 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.840+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11700c2a90 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.841+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1170071780 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.842+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f11640103f0 con 0x7f1170070a20 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.842+0000 7f116cff9700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:57.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.842+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f116408bba0 con 0x7f1170070a20 2023-06-10T18:53:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.842+0000 7f116ffff700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.843+0000 7f116ffff700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 0x7f115c06d190 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f117006daf0 tx=0x7f1160009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:57.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:57.846+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f116408be60 con 0x7f1170070a20 2023-06-10T18:53:58.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.077+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1170074050 con 0x7f1170070a20 2023-06-10T18:53:58.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.078+0000 7f116cff9700 1 -- 172.21.15.104:0/2423366281 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f1164056c60 con 0x7f1170070a20 2023-06-10T18:53:58.084 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:58.084 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":59,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:57.697088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 msgr2=0x7f115c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 0x7f115c06d190 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f117006daf0 tx=0x7f1160009040 comp rx=0 tx=0).stop 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 msgr2=0x7f11700b1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f1164000c00 tx=0x7f11640047c0 comp rx=0 tx=0).stop 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 shutdown_connections 2023-06-10T18:53:58.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1170158ce0 0x7f11700b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f117006a8c0 0x7f11700b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 --2- 172.21.15.104:0/2423366281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1170070a20 0x7f11700b1c80 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 >> 172.21.15.104:0/2423366281 conn(0x7f11700b1610 msgr2=0x7f1170065b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 shutdown_connections 2023-06-10T18:53:58.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:58.080+0000 7f11768ab700 1 -- 172.21.15.104:0/2423366281 wait complete. 2023-06-10T18:53:58.088 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 59 2023-06-10T18:53:58.098 DEBUG:tasks.cephfs.filesystem:mapped states ['up:replay'] to None 2023-06-10T18:53:59.099 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:59.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.226+0000 7f718ba5b700 1 Processor -- start 2023-06-10T18:53:59.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.226+0000 7f718ba5b700 1 -- start start 2023-06-10T18:53:59.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f718ba5b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71840d2b20 0x7f71840d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f718ba5b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840d0520 0x7f71840d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f718ba5b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71840cd430 con 0x7f71840ccfa0 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f718ba5b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71840d0ee0 con 0x7f71840d2b20 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f718ba5b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840cd710 con 0x7f71840d0520 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71840d2b20 0x7f71840d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f71840d2b20 0x7f71840d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35058/0 (socket says 172.21.15.104:35058) 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 -- 172.21.15.104:0/2334068355 learned_addr learned my addr 172.21.15.104:0/2334068355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 -- 172.21.15.104:0/2334068355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840d0520 msgr2=0x7f71840d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f7189ff8700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840d0520 0x7f71840d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840d0520 0x7f71840d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 -- 172.21.15.104:0/2334068355 >> v1:172.21.15.114:6789/0 conn(0x7f71840ccfa0 legacy=0x7f71840b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.227+0000 7f71897f7700 1 -- 172.21.15.104:0/2334068355 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71840cd9f0 con 0x7f71840d2b20 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f71897f7700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d2b20 0x7f71840d2f00 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f71740098f0 tx=0x7f7174009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9126e7fade2e027 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f717bfff700 1 -- 172.21.15.104:0/2334068355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174004230 con 0x7f71840d2b20 2023-06-10T18:53:59.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f717bfff700 1 -- 172.21.15.104:0/2334068355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7174004390 con 0x7f71840d2b20 2023-06-10T18:53:59.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f717bfff700 1 -- 172.21.15.104:0/2334068355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71740036a0 con 0x7f71840d2b20 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 -- 172.21.15.104:0/2334068355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d2b20 msgr2=0x7f71840d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d2b20 0x7f71840d2f00 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f71740098f0 tx=0x7f7174009c00 comp rx=0 tx=0).stop 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 -- 172.21.15.104:0/2334068355 shutdown_connections 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d2b20 0x7f71840d2f00 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 --2- 172.21.15.104:0/2334068355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840d0520 0x7f71840d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.228+0000 7f718ba5b700 1 -- 172.21.15.104:0/2334068355 >> 172.21.15.104:0/2334068355 conn(0x7f7184071ec0 msgr2=0x7f718409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.229+0000 7f718ba5b700 1 -- 172.21.15.104:0/2334068355 shutdown_connections 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.229+0000 7f718ba5b700 1 -- 172.21.15.104:0/2334068355 wait complete. 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 Processor -- start 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 -- start start 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 0x7f71841f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840b0cc0 0x7f718415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840b6ae0 con 0x7f71840d0520 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840b1160 con 0x7f71840ccfa0 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.230+0000 7f718ba5b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840b5e60 con 0x7f71840b0cc0 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35064/0 (socket says 172.21.15.104:35064) 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 -- 172.21.15.104:0/1823112639 learned_addr learned my addr 172.21.15.104:0/1823112639 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f71897f7700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 0x7f71841f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7189ff8700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840b0cc0 0x7f718415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 -- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840b0cc0 msgr2=0x7f718415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840b0cc0 0x7f718415ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 -- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 msgr2=0x7f71841f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 0x7f71841f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 -- 172.21.15.104:0/1823112639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7174009550 con 0x7f71840d0520 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f71897f7700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 0x7f71841f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7188ff6700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f718000c4d0 tx=0x7f718000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180011790 con 0x7f71840d0520 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71800118f0 con 0x7f71840d0520 2023-06-10T18:53:59.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180011ca0 con 0x7f71840d0520 2023-06-10T18:53:59.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71840b6140 con 0x7f71840d0520 2023-06-10T18:53:59.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.231+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71840b6700 con 0x7f71840d0520 2023-06-10T18:53:59.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.232+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71840b6c20 con 0x7f71840d0520 2023-06-10T18:53:59.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.233+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f718001c460 con 0x7f71840d0520 2023-06-10T18:53:59.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.233+0000 7f7179ffb700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 0x7f716806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.233+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f7180089440 con 0x7f71840d0520 2023-06-10T18:53:59.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.233+0000 7f71897f7700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 0x7f716806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.234+0000 7f71897f7700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 0x7f716806d190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7174009130 tx=0x7f7174008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.237+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f71840b6c20 con 0x7f71840d0520 2023-06-10T18:53:59.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.466+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71840b6c20 con 0x7f71840d0520 2023-06-10T18:53:59.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.466+0000 7f7179ffb700 1 -- 172.21.15.104:0/1823112639 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f71840b6c20 con 0x7f71840d0520 2023-06-10T18:53:59.472 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:59.473 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:53:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 msgr2=0x7f716806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 0x7f716806d190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7174009130 tx=0x7f7174008030 comp rx=0 tx=0).stop 2023-06-10T18:53:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 msgr2=0x7f71840b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f718000c4d0 tx=0x7f718000c890 comp rx=0 tx=0).stop 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 shutdown_connections 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f716806ace0 0x7f716806d190 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71840ccfa0 0x7f71841f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71840b0cc0 0x7f718415ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 --2- 172.21.15.104:0/1823112639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71840d0520 0x7f71840b0780 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 >> 172.21.15.104:0/1823112639 conn(0x7f7184071ec0 msgr2=0x7f71840b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:59.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 shutdown_connections 2023-06-10T18:53:59.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.468+0000 7f718ba5b700 1 -- 172.21.15.104:0/1823112639 wait complete. 2023-06-10T18:53:59.476 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:53:59.486 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to up:reconnect 2023-06-10T18:53:59.486 DEBUG:tasks.cephfs.filesystem:reached state 'up:reconnect' in 1.7674822807312012s 2023-06-10T18:53:59.487 INFO:tasks.cephfs.test_client_recovery:Waiting for active... 2023-06-10T18:53:59.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:53:59.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.600+0000 7fa284371700 1 Processor -- start 2023-06-10T18:53:59.607 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 -- start start 2023-06-10T18:53:59.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa27c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa27c0cd680 0x7fa27c06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c0cdae0 con 0x7fa27c0b1080 2023-06-10T18:53:59.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c0c9bc0 con 0x7fa27c0cd680 2023-06-10T18:53:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.601+0000 7fa284371700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0c9e60 con 0x7fa27c0aeaa0 2023-06-10T18:53:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28190c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa27c0b1080 0x7fa27c0ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32900/0 (socket says 172.21.15.104:32900) 2023-06-10T18:53:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28190c700 1 -- 172.21.15.104:0/1151713279 learned_addr learned my addr 172.21.15.104:0/1151713279 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:53:59.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28210d700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa27c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> v2:172.21.15.114:3300/0 conn(0x7fa27c0cd680 0x7fa27c06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096539498 0 0) 0x7fa27c0cdae0 con 0x7fa27c0b1080 2023-06-10T18:53:59.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2680035b0 con 0x7fa27c0b1080 2023-06-10T18:53:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1205946373 0 0) 0x7fa2680035b0 con 0x7fa27c0b1080 2023-06-10T18:53:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 msgr2=0x7fa27c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa27c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 >> v2:172.21.15.114:3300/0 conn(0x7fa27c0cd680 msgr2=0x7fa27c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 --2- 172.21.15.104:0/1151713279 >> v2:172.21.15.114:3300/0 conn(0x7fa27c0cd680 0x7fa27c06dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0aee70 con 0x7fa27c0b1080 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa278001670 con 0x7fa27c0b1080 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 >> v1:172.21.15.104:6789/0 conn(0x7fa27c0b1080 legacy=0x7fa27c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268003a00 0x7fa268003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa268004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa268004d90 0x7fa268005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0c9bc0 con 0x7fa268004d90 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0c9e60 con 0x7fa268003a00 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26800ba10 con 0x7fa27c0aeaa0 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> v2:172.21.15.114:3300/0 conn(0x7fa27c0cd680 0x7fa27c06dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa268004d90 0x7fa268005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28210d700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268003a00 0x7fa268003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.602+0000 7fa28190c700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa268004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 -- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 msgr2=0x7fa268004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0aeaa0 0x7fa268004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 -- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268003a00 msgr2=0x7fa268003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268003a00 0x7fa268003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 -- 172.21.15.104:0/1151713279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2680035b0 con 0x7fa268004d90 2023-06-10T18:53:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa268004d90 0x7fa268005200 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fa27400e350 tx=0x7fa268005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ed3693a381f98d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400edb0 con 0x7fa268004d90 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27400f040 con 0x7fa268004d90 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28110b700 1 -- 172.21.15.104:0/1151713279 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa274003c40 con 0x7fa268004d90 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa284371700 1 -- 172.21.15.104:0/1151713279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa268004d90 msgr2=0x7fa268005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa284371700 1 --2- 172.21.15.104:0/1151713279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa268004d90 0x7fa268005200 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fa27400e350 tx=0x7fa268005880 comp rx=0 tx=0).stop 2023-06-10T18:53:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa28290e700 1 -- 172.21.15.104:0/1151713279 reap_dead start 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa284371700 1 -- 172.21.15.104:0/1151713279 shutdown_connections 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa284371700 1 -- 172.21.15.104:0/1151713279 >> 172.21.15.104:0/1151713279 conn(0x7fa27c06b8a0 msgr2=0x7fa27c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.603+0000 7fa284371700 1 -- 172.21.15.104:0/1151713279 shutdown_connections 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.604+0000 7fa284371700 1 -- 172.21.15.104:0/1151713279 wait complete. 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.605+0000 7fa284371700 1 Processor -- start 2023-06-10T18:53:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.605+0000 7fa284371700 1 -- start start 2023-06-10T18:53:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268004d90 0x7fa27c14e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 0x7fa27c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c15bc60 con 0x7fa27c15b7e0 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c14eea0 con 0x7fa268004d90 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa284371700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0b3d80 con 0x7fa27c0b4da0 2023-06-10T18:53:59.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35086/0 (socket says 172.21.15.104:35086) 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 -- 172.21.15.104:0/3600511216 learned_addr learned my addr 172.21.15.104:0/3600511216 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28190c700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 0x7fa27c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28210d700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268004d90 0x7fa27c14e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 -- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 msgr2=0x7fa27c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 0x7fa27c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 -- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268004d90 msgr2=0x7fa27c14e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268004d90 0x7fa27c14e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28290e700 1 -- 172.21.15.104:0/3600511216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0b2fc0 con 0x7fa27c15b7e0 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.606+0000 7fa28190c700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 0x7fa27c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa28290e700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b010 tx=0x7fa274009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa274004230 con 0x7fa27c15b7e0 2023-06-10T18:53:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa274022070 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c0b32a0 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2740045f0 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.607+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c0b3860 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.608+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c074050 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.608+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa274024030 con 0x7fa27c15b7e0 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.608+0000 7fa272ffd700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 0x7fa26006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:53:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.609+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fa27408ae20 con 0x7fa27c15b7e0 2023-06-10T18:53:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.609+0000 7fa28210d700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 0x7fa26006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:53:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.609+0000 7fa28210d700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 0x7fa26006d190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009b00 tx=0x7fa26c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:53:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.613+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa274059850 con 0x7fa27c15b7e0 2023-06-10T18:53:59.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.844+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa27c074050 con 0x7fa27c15b7e0 2023-06-10T18:53:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.844+0000 7fa272ffd700 1 -- 172.21.15.104:0/3600511216 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fa2740593e0 con 0x7fa27c15b7e0 2023-06-10T18:53:59.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:53:59.851 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:53:59.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.846+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 msgr2=0x7fa26006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.846+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 0x7fa26006d190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009b00 tx=0x7fa26c008040 comp rx=0 tx=0).stop 2023-06-10T18:53:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 msgr2=0x7fa27c0c3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:53:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b010 tx=0x7fa274009750 comp rx=0 tx=0).stop 2023-06-10T18:53:59.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 shutdown_connections 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa26006ace0 0x7fa26006d190 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268004d90 0x7fa27c14e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa27c15b7e0 0x7fa27c0c3e80 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 --2- 172.21.15.104:0/3600511216 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa27c0b4da0 0x7fa27c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 >> 172.21.15.104:0/3600511216 conn(0x7fa27c06b8a0 msgr2=0x7fa27c0b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:53:59.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 shutdown_connections 2023-06-10T18:53:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:53:59.847+0000 7fa284371700 1 -- 172.21.15.104:0/3600511216 wait complete. 2023-06-10T18:53:59.855 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:53:59.864 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:00.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:00.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.990+0000 7f99db7db700 1 Processor -- start 2023-06-10T18:54:00.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.991+0000 7f99db7db700 1 -- start start 2023-06-10T18:54:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.991+0000 7f99db7db700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f99d415a0b0 0x7f99d40717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99db7db700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d406b2e0 con 0x7f99d4071130 2023-06-10T18:54:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99db7db700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d415e150 con 0x7f99d40c93e0 2023-06-10T18:54:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99db7db700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d415db40 con 0x7f99d415a0b0 2023-06-10T18:54:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f99d415a0b0 0x7f99d40717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f99d415a0b0 0x7f99d40717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35092/0 (socket says 172.21.15.104:35092) 2023-06-10T18:54:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9d78700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f99d4071130 0x7f99d415a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32922/0 (socket says 172.21.15.104:32922) 2023-06-10T18:54:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 -- 172.21.15.104:0/4256326930 learned_addr learned my addr 172.21.15.104:0/4256326930 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 -- 172.21.15.104:0/4256326930 >> v1:172.21.15.114:6789/0 conn(0x7f99d40c93e0 legacy=0x7f99d4070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 -- 172.21.15.104:0/4256326930 >> v1:172.21.15.104:6789/0 conn(0x7f99d4071130 legacy=0x7f99d415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 -- 172.21.15.104:0/4256326930 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d406b530 con 0x7f99d415a0b0 2023-06-10T18:54:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99c7fff700 1 -- 172.21.15.104:0/4256326930 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987598920 0 0) 0x7f99d406b2e0 con 0x7f99d4071130 2023-06-10T18:54:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99d9577700 1 --2- 172.21.15.104:0/4256326930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40717d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f99c80098f0 tx=0x7f99c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9499b65f1cc3b86e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99c7fff700 1 -- 172.21.15.104:0/4256326930 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8004230 con 0x7f99d415a0b0 2023-06-10T18:54:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99c7fff700 1 -- 172.21.15.104:0/4256326930 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c8004390 con 0x7f99d415a0b0 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.992+0000 7f99c7fff700 1 -- 172.21.15.104:0/4256326930 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c80036a0 con 0x7f99d415a0b0 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 -- 172.21.15.104:0/4256326930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 msgr2=0x7f99d40717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 --2- 172.21.15.104:0/4256326930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40717d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f99c80098f0 tx=0x7f99c8009c00 comp rx=0 tx=0).stop 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 -- 172.21.15.104:0/4256326930 shutdown_connections 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 --2- 172.21.15.104:0/4256326930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40717d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 -- 172.21.15.104:0/4256326930 >> 172.21.15.104:0/4256326930 conn(0x7f99d40b1610 msgr2=0x7f99d40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:01.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 -- 172.21.15.104:0/4256326930 shutdown_connections 2023-06-10T18:54:01.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.993+0000 7f99db7db700 1 -- 172.21.15.104:0/4256326930 wait complete. 2023-06-10T18:54:01.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.994+0000 7f99db7db700 1 Processor -- start 2023-06-10T18:54:01.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.994+0000 7f99db7db700 1 -- start start 2023-06-10T18:54:01.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 0x7f99d41502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:01.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 0x7f99d4156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41f03c0 con 0x7f99d415a0b0 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41f0640 con 0x7f99d40c93e0 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99db7db700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41f0920 con 0x7f99d4071130 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99d9d78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99d9d78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35100/0 (socket says 172.21.15.104:35100) 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99d9d78700 1 -- 172.21.15.104:0/2765154382 learned_addr learned my addr 172.21.15.104:0/2765154382 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:01.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99d8d76700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 0x7f99d4156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.995+0000 7f99d9577700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 0x7f99d41502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 -- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 msgr2=0x7f99d41502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 0x7f99d41502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 -- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 msgr2=0x7f99d4156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 0x7f99d4156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 -- 172.21.15.104:0/2765154382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8009550 con 0x7f99d415a0b0 2023-06-10T18:54:01.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9577700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 0x7f99d41502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d8d76700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 0x7f99d4156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99d9d78700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00cca0 tx=0x7f99cc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00cea0 con 0x7f99d415a0b0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00ade0 con 0x7f99d415a0b0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc007310 con 0x7f99d415a0b0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d41f0c00 con 0x7f99d415a0b0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.996+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d41f11c0 con 0x7f99d415a0b0 2023-06-10T18:54:01.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.997+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d40c3a60 con 0x7f99d415a0b0 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.997+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f99cc007470 con 0x7f99d415a0b0 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.998+0000 7f99c5ffb700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 0x7f99b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.998+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f99cc089310 con 0x7f99d415a0b0 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.998+0000 7f99d9577700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 0x7f99b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:00.998+0000 7f99d9577700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 0x7f99b806d190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009c40 tx=0x7f99c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:01.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.002+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f99cc054220 con 0x7f99d415a0b0 2023-06-10T18:54:01.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.231+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99d415e5c0 con 0x7f99d415a0b0 2023-06-10T18:54:01.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.232+0000 7f99c5ffb700 1 -- 172.21.15.104:0/2765154382 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f99d415e5c0 con 0x7f99d415a0b0 2023-06-10T18:54:01.238 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:01.238 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:01.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 msgr2=0x7f99b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 0x7f99b806d190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009c40 tx=0x7f99c8008030 comp rx=0 tx=0).stop 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 msgr2=0x7f99d40ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00cca0 tx=0x7f99cc00a340 comp rx=0 tx=0).stop 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 shutdown_connections 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99d415a0b0 0x7f99d40ae960 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99d4071130 0x7f99d41502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99b806ace0 0x7f99b806d190 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 --2- 172.21.15.104:0/2765154382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d40c93e0 0x7f99d4156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:01.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 >> 172.21.15.104:0/2765154382 conn(0x7f99d40b1610 msgr2=0x7f99d406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:01.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 shutdown_connections 2023-06-10T18:54:01.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:01.234+0000 7f99db7db700 1 -- 172.21.15.104:0/2765154382 wait complete. 2023-06-10T18:54:01.241 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:01.251 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:02.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:02.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.378+0000 7fb581248700 1 Processor -- start 2023-06-10T18:54:02.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.378+0000 7fb581248700 1 -- start start 2023-06-10T18:54:02.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb581248700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0ade00 0x7fb57c0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb581248700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb581248700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c0cee70 con 0x7fb57c0ada30 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb581248700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c0cf430 con 0x7fb57c0aeb00 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb581248700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0cf150 con 0x7fb57c0ade00 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35108/0 (socket says 172.21.15.104:35108) 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 -- 172.21.15.104:0/2415229527 learned_addr learned my addr 172.21.15.104:0/2415229527 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57a59c700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0ade00 0x7fb57c0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 -- 172.21.15.104:0/2415229527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0ade00 msgr2=0x7fb57c0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0ade00 0x7fb57c0ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 -- 172.21.15.104:0/2415229527 >> v1:172.21.15.114:6789/0 conn(0x7fb57c0ada30 legacy=0x7fb57c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 -- 172.21.15.104:0/2415229527 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c06cf10 con 0x7fb57c0aeb00 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.379+0000 7fb57b59e700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb574009e50 tx=0x7fb57400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c681367180392d6a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb579d9b700 1 -- 172.21.15.104:0/2415229527 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57400e070 con 0x7fb57c0aeb00 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb579d9b700 1 -- 172.21.15.104:0/2415229527 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb574003a20 con 0x7fb57c0aeb00 2023-06-10T18:54:02.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb579d9b700 1 -- 172.21.15.104:0/2415229527 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574003dd0 con 0x7fb57c0aeb00 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 -- 172.21.15.104:0/2415229527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0aeb00 msgr2=0x7fb57c0ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb574009e50 tx=0x7fb57400b330 comp rx=0 tx=0).stop 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 -- 172.21.15.104:0/2415229527 shutdown_connections 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0aeb00 0x7fb57c0ce5f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 --2- 172.21.15.104:0/2415229527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0ade00 0x7fb57c0ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 -- 172.21.15.104:0/2415229527 >> 172.21.15.104:0/2415229527 conn(0x7fb57c06b8a0 msgr2=0x7fb57c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 -- 172.21.15.104:0/2415229527 shutdown_connections 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.380+0000 7fb581248700 1 -- 172.21.15.104:0/2415229527 wait complete. 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 Processor -- start 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 -- start start 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 0x7fb57c1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0aeb00 0x7fb57c0b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c1f1c00 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c099340 con 0x7fb57c0ada30 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.382+0000 7fb581248700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0b1530 con 0x7fb57c0aeb00 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35114/0 (socket says 172.21.15.104:35114) 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 -- 172.21.15.104:0/1362697448 learned_addr learned my addr 172.21.15.104:0/1362697448 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57a59c700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0aeb00 0x7fb57c0b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57ad9d700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 0x7fb57c1f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 -- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0aeb00 msgr2=0x7fb57c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0aeb00 0x7fb57c0b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 -- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 msgr2=0x7fb57c1f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 0x7fb57c1f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 -- 172.21.15.104:0/1362697448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574009b30 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57ad9d700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 0x7fb57c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb57b59e700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb574004000 tx=0x7fb574004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574004440 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5740045c0 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574004970 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c0b1810 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.383+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c0b1dd0 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.384+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c074050 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.385+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb574012430 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.385+0000 7fb5637fe700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 0x7fb56806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:02.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.385+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fb57408a980 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.385+0000 7fb57ad9d700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 0x7fb56806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:02.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.386+0000 7fb57ad9d700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 0x7fb56806d190 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:02.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.390+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb574059310 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.621+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb57c066f70 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.621+0000 7fb5637fe700 1 -- 172.21.15.104:0/1362697448 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fb574058ea0 con 0x7fb57c0b0fb0 2023-06-10T18:54:02.627 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:02.628 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:02.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 msgr2=0x7fb56806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:02.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 0x7fb56806d190 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570008040 comp rx=0 tx=0).stop 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 msgr2=0x7fb57c15eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb574004000 tx=0x7fb574004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 shutdown_connections 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb56806ace0 0x7fb56806d190 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57c0ada30 0x7fb57c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57c0b0fb0 0x7fb57c15eaf0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 --2- 172.21.15.104:0/1362697448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57c0aeb00 0x7fb57c0b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:02.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.623+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 >> 172.21.15.104:0/1362697448 conn(0x7fb57c06b8a0 msgr2=0x7fb57c0b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:02.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.624+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 shutdown_connections 2023-06-10T18:54:02.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:02.624+0000 7fb581248700 1 -- 172.21.15.104:0/1362697448 wait complete. 2023-06-10T18:54:02.632 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:02.641 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:03.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:03.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.771+0000 7fceeb32b700 1 Processor -- start 2023-06-10T18:54:03.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.771+0000 7fceeb32b700 1 -- start start 2023-06-10T18:54:03.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fceeb32b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcee40b1070 0x7fcee40b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fceeb32b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcee40aec00 0x7fcee40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fceeb32b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee406e5e0 con 0x7fcee406ea70 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fceeb32b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40b1980 con 0x7fcee40b1070 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fceeb32b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40ae7e0 con 0x7fcee40aec00 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcee40b1070 0x7fcee40b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcee40b1070 0x7fcee40b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35118/0 (socket says 172.21.15.104:35118) 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 -- 172.21.15.104:0/2213368355 learned_addr learned my addr 172.21.15.104:0/2213368355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee98c8700 1 --2- 172.21.15.104:0/2213368355 >> v2:172.21.15.114:3300/0 conn(0x7fcee40aec00 0x7fcee40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 -- 172.21.15.104:0/2213368355 >> v2:172.21.15.114:3300/0 conn(0x7fcee40aec00 msgr2=0x7fcee40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:03.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 --2- 172.21.15.104:0/2213368355 >> v2:172.21.15.114:3300/0 conn(0x7fcee40aec00 0x7fcee40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 -- 172.21.15.104:0/2213368355 >> v1:172.21.15.114:6789/0 conn(0x7fcee406ea70 legacy=0x7fcee40ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 -- 172.21.15.104:0/2213368355 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee406df90 con 0x7fcee40b1070 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcee90c7700 1 --2- 172.21.15.104:0/2213368355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b1440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fcee40cd0c0 tx=0x7fcee000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eb07b152e32390 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.772+0000 7fcedbfff700 1 -- 172.21.15.104:0/2213368355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005e30 con 0x7fcee40b1070 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fcedbfff700 1 -- 172.21.15.104:0/2213368355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000d8b0 con 0x7fcee40b1070 2023-06-10T18:54:03.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fcedbfff700 1 -- 172.21.15.104:0/2213368355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000dc40 con 0x7fcee40b1070 2023-06-10T18:54:03.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 -- 172.21.15.104:0/2213368355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 msgr2=0x7fcee40b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 --2- 172.21.15.104:0/2213368355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b1440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fcee40cd0c0 tx=0x7fcee000be70 comp rx=0 tx=0).stop 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 -- 172.21.15.104:0/2213368355 shutdown_connections 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 --2- 172.21.15.104:0/2213368355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b1440 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 --2- 172.21.15.104:0/2213368355 >> v2:172.21.15.114:3300/0 conn(0x7fcee40aec00 0x7fcee40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 -- 172.21.15.104:0/2213368355 >> 172.21.15.104:0/2213368355 conn(0x7fcee406b8a0 msgr2=0x7fcee40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 -- 172.21.15.104:0/2213368355 shutdown_connections 2023-06-10T18:54:03.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.773+0000 7fceeb32b700 1 -- 172.21.15.104:0/2213368355 wait complete. 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.774+0000 7fceeb32b700 1 Processor -- start 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 -- start start 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee406ea70 0x7fcee4146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 0x7fcee41ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4099340 con 0x7fcee40b1070 2023-06-10T18:54:03.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4099590 con 0x7fcee406ea70 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fceeb32b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4099870 con 0x7fcee40aec00 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fcee98c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fcee98c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35128/0 (socket says 172.21.15.104:35128) 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.775+0000 7fcee98c8700 1 -- 172.21.15.104:0/1084167945 learned_addr learned my addr 172.21.15.104:0/1084167945 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee90c7700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee406ea70 0x7fcee4146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee88c6700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 0x7fcee41ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 -- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 msgr2=0x7fcee41ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 0x7fcee41ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 -- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee406ea70 msgr2=0x7fcee4146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee406ea70 0x7fcee4146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 -- 172.21.15.104:0/1084167945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee000b7a0 con 0x7fcee40b1070 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee88c6700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 0x7fcee41ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fcee98c8700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fcedc007de0 tx=0x7fcedc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00ede0 con 0x7fcee40b1070 2023-06-10T18:54:03.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc014420 con 0x7fcee40b1070 2023-06-10T18:54:03.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4099b50 con 0x7fcee40b1070 2023-06-10T18:54:03.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0147d0 con 0x7fcee40b1070 2023-06-10T18:54:03.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.776+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee409a110 con 0x7fcee40b1070 2023-06-10T18:54:03.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.777+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee409a330 con 0x7fcee40b1070 2023-06-10T18:54:03.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.778+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcedc014930 con 0x7fcee40b1070 2023-06-10T18:54:03.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.778+0000 7fced9ffb700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 0x7fcec806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:03.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.778+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fcedc089ee0 con 0x7fcee40b1070 2023-06-10T18:54:03.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.778+0000 7fcee90c7700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 0x7fcec806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:03.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.779+0000 7fcee90c7700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 0x7fcec806d190 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcee0000c00 tx=0x7fcee0012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:03.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:03.782+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcedc054e10 con 0x7fcee40b1070 2023-06-10T18:54:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.012+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee4074050 con 0x7fcee40b1070 2023-06-10T18:54:04.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.012+0000 7fced9ffb700 1 -- 172.21.15.104:0/1084167945 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fcedc058430 con 0x7fcee40b1070 2023-06-10T18:54:04.018 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:04.018 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:04.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.014+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 msgr2=0x7fcec806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:04.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.014+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 0x7fcec806d190 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcee0000c00 tx=0x7fcee0012000 comp rx=0 tx=0).stop 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.014+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 msgr2=0x7fcee40b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.014+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fcedc007de0 tx=0x7fcedc00e3c0 comp rx=0 tx=0).stop 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 shutdown_connections 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcec806ace0 0x7fcec806d190 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcee406ea70 0x7fcee4146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcee40b1070 0x7fcee40b3650 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 --2- 172.21.15.104:0/1084167945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcee40aec00 0x7fcee41ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 >> 172.21.15.104:0/1084167945 conn(0x7fcee406b8a0 msgr2=0x7fcee40ad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 shutdown_connections 2023-06-10T18:54:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:04.015+0000 7fceeb32b700 1 -- 172.21.15.104:0/1084167945 wait complete. 2023-06-10T18:54:04.022 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:04.032 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:05.034 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.158+0000 7fd8ede8e700 1 Processor -- start 2023-06-10T18:54:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.158+0000 7fd8ede8e700 1 -- start start 2023-06-10T18:54:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8ede8e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd8e80aeaa0 0x7fd8e80cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8ede8e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 0x7fd8e806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8ede8e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e80c9e60 con 0x7fd8e80b1080 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8ede8e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e80c9bc0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8ede8e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e80cdaa0 con 0x7fd8e80cd680 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd8e80b1080 0x7fd8e80ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32960/0 (socket says 172.21.15.104:32960) 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e77fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd8e80aeaa0 0x7fd8e80cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e6ffd700 1 -- 172.21.15.104:0/683754753 learned_addr learned my addr 172.21.15.104:0/683754753 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e7fff700 1 --2- 172.21.15.104:0/683754753 >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 0x7fd8e806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e67fc700 1 -- 172.21.15.104:0/683754753 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614307104 0 0) 0x7fd8e80c9e60 con 0x7fd8e80b1080 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e67fc700 1 -- 172.21.15.104:0/683754753 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d40035b0 con 0x7fd8e80b1080 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e77fe700 1 -- 172.21.15.104:0/683754753 >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 msgr2=0x7fd8e806dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e77fe700 1 --2- 172.21.15.104:0/683754753 >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 0x7fd8e806dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e77fe700 1 -- 172.21.15.104:0/683754753 >> v1:172.21.15.104:6789/0 conn(0x7fd8e80b1080 legacy=0x7fd8e80ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e77fe700 1 -- 172.21.15.104:0/683754753 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e80aee70 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.159+0000 7fd8e7fff700 1 --2- 172.21.15.104:0/683754753 >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 0x7fd8e806dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:05.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8e77fe700 1 --2- 172.21.15.104:0/683754753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80cd140 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8071030 tx=0x7fd8d0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92f9137bf1321de server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:05.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8e67fc700 1 -- 172.21.15.104:0/683754753 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0005e30 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8e67fc700 1 -- 172.21.15.104:0/683754753 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d0006030 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8e67fc700 1 -- 172.21.15.104:0/683754753 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00063c0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 -- 172.21.15.104:0/683754753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 msgr2=0x7fd8e80cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/683754753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80cd140 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8071030 tx=0x7fd8d0009b00 comp rx=0 tx=0).stop 2023-06-10T18:54:05.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 -- 172.21.15.104:0/683754753 shutdown_connections 2023-06-10T18:54:05.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/683754753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80cd140 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/683754753 >> v2:172.21.15.114:3300/0 conn(0x7fd8e80cd680 0x7fd8e806dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 -- 172.21.15.104:0/683754753 >> 172.21.15.104:0/683754753 conn(0x7fd8e806b8a0 msgr2=0x7fd8e80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 -- 172.21.15.104:0/683754753 shutdown_connections 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.160+0000 7fd8ede8e700 1 -- 172.21.15.104:0/683754753 wait complete. 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 Processor -- start 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 -- start start 2023-06-10T18:54:05.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 0x7fd8e814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8e80cd680 0x7fd8e8159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e80c8060 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815bc00 con 0x7fd8e80b1080 2023-06-10T18:54:05.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.162+0000 7fd8ede8e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815bee0 con 0x7fd8e80cd680 2023-06-10T18:54:05.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35138/0 (socket says 172.21.15.104:35138) 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 -- 172.21.15.104:0/1904597111 learned_addr learned my addr 172.21.15.104:0/1904597111 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e6ffd700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 0x7fd8e814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e7fff700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8e80cd680 0x7fd8e8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 -- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8e80cd680 msgr2=0x7fd8e8159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8e80cd680 0x7fd8e8159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 -- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 msgr2=0x7fd8e814fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 0x7fd8e814fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 -- 172.21.15.104:0/1904597111 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0009430 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e6ffd700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 0x7fd8e814fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8e77fe700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0004040 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d0020070 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e815c1c0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0004400 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.163+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e80b2970 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.164+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e80c7ec0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.165+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd8d0022030 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.165+0000 7fd8dffff700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 0x7fd8c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.165+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fd8d008ab70 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.165+0000 7fd8e6ffd700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 0x7fd8c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:05.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.166+0000 7fd8e6ffd700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 0x7fd8c806d190 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd8e81f6240 tx=0x7fd8d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:05.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.170+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd8e80c7ec0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.396+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8e80c7ec0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.396+0000 7fd8dffff700 1 -- 172.21.15.104:0/1904597111 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fd8e80c7ec0 con 0x7fd8e80aeaa0 2023-06-10T18:54:05.402 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:05.402 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 msgr2=0x7fd8c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 0x7fd8c806d190 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd8e81f6240 tx=0x7fd8d8009040 comp rx=0 tx=0).stop 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 msgr2=0x7fd8e80b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0000c00 tx=0x7fd8d0004750 comp rx=0 tx=0).stop 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 shutdown_connections 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8e80cd680 0x7fd8e8159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8e80aeaa0 0x7fd8e80b2430 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8c806ace0 0x7fd8c806d190 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 --2- 172.21.15.104:0/1904597111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8e80b1080 0x7fd8e814fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 >> 172.21.15.104:0/1904597111 conn(0x7fd8e806b8a0 msgr2=0x7fd8e80cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 shutdown_connections 2023-06-10T18:54:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:05.398+0000 7fd8ede8e700 1 -- 172.21.15.104:0/1904597111 wait complete. 2023-06-10T18:54:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:05.416 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:06.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:06.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.545+0000 7ff5373f0700 1 Processor -- start 2023-06-10T18:54:06.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.545+0000 7ff5373f0700 1 -- start start 2023-06-10T18:54:06.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff53006b1f0 0x7ff5300cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff5300ce350 0x7ff5300cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff53008a3d0 0x7ff5300cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300cfeb0 con 0x7ff53008a3d0 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300cadd0 con 0x7ff53006b1f0 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff5373f0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300cb0b0 con 0x7ff5300ce350 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff53008a3d0 0x7ff5300cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff53008a3d0 0x7ff5300cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35150/0 (socket says 172.21.15.104:35150) 2023-06-10T18:54:06.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 -- 172.21.15.104:0/1489186650 learned_addr learned my addr 172.21.15.104:0/1489186650 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53518c700 1 --2- 172.21.15.104:0/1489186650 >> v2:172.21.15.114:3300/0 conn(0x7ff53006b1f0 0x7ff5300cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53498b700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff5300ce350 0x7ff5300cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 -- 172.21.15.104:0/1489186650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff5300ce350 msgr2=0x7ff5300cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff5300ce350 0x7ff5300cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 -- 172.21.15.104:0/1489186650 >> v2:172.21.15.114:3300/0 conn(0x7ff53006b1f0 msgr2=0x7ff5300cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 --2- 172.21.15.104:0/1489186650 >> v2:172.21.15.114:3300/0 conn(0x7ff53006b1f0 0x7ff5300cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.546+0000 7ff53598d700 1 -- 172.21.15.104:0/1489186650 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300cf550 con 0x7ff53008a3d0 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff53598d700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff53008a3d0 0x7ff5300cdb10 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff53006e2d0 tx=0x7ff52800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f52cb224beaaa9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff527fff700 1 -- 172.21.15.104:0/1489186650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800e070 con 0x7ff53008a3d0 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff527fff700 1 -- 172.21.15.104:0/1489186650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52800b9c0 con 0x7ff53008a3d0 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff527fff700 1 -- 172.21.15.104:0/1489186650 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800bd70 con 0x7ff53008a3d0 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff5373f0700 1 -- 172.21.15.104:0/1489186650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff53008a3d0 msgr2=0x7ff5300cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:06.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.547+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff53008a3d0 0x7ff5300cdb10 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff53006e2d0 tx=0x7ff52800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 -- 172.21.15.104:0/1489186650 shutdown_connections 2023-06-10T18:54:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff53008a3d0 0x7ff5300cdb10 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1489186650 >> v2:172.21.15.114:3300/0 conn(0x7ff53006b1f0 0x7ff5300cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1489186650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff5300ce350 0x7ff5300cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 -- 172.21.15.104:0/1489186650 >> 172.21.15.104:0/1489186650 conn(0x7ff530089fd0 msgr2=0x7ff5300d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 -- 172.21.15.104:0/1489186650 shutdown_connections 2023-06-10T18:54:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.548+0000 7ff5373f0700 1 -- 172.21.15.104:0/1489186650 wait complete. 2023-06-10T18:54:06.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.549+0000 7ff5373f0700 1 Processor -- start 2023-06-10T18:54:06.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.549+0000 7ff5373f0700 1 -- start start 2023-06-10T18:54:06.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 0x7ff53009d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 0x7ff53009a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53009e510 con 0x7ff5300ce350 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53009d900 con 0x7ff53006b1f0 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53009dbe0 con 0x7ff53008a3d0 2023-06-10T18:54:06.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35156/0 (socket says 172.21.15.104:35156) 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 -- 172.21.15.104:0/1988009281 learned_addr learned my addr 172.21.15.104:0/1988009281 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53518c700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 0x7ff53009d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53498b700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 0x7ff53009a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 -- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 msgr2=0x7ff53009a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 0x7ff53009a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 -- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 msgr2=0x7ff53009d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 0x7ff53009d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 -- 172.21.15.104:0/1988009281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff528009b30 con 0x7ff5300ce350 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53518c700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 0x7ff53009d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53498b700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 0x7ff53009a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff53598d700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff52800b370 tx=0x7ff528004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528004310 con 0x7ff5300ce350 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528004470 con 0x7ff5300ce350 2023-06-10T18:54:06.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53009dec0 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.550+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528004820 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.551+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53009a9c0 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.551+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5300c5b70 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.552+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff52801b430 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.552+0000 7ff525ffb700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 0x7ff51406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.552+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff52808ab80 con 0x7ff5300ce350 2023-06-10T18:54:06.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.553+0000 7ff53518c700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 0x7ff51406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:06.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.553+0000 7ff53518c700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 0x7ff51406d190 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff520006fd0 tx=0x7ff520008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:06.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.557+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff5300c5b70 con 0x7ff5300ce350 2023-06-10T18:54:06.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.782+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5300c5b70 con 0x7ff5300ce350 2023-06-10T18:54:06.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.782+0000 7ff525ffb700 1 -- 172.21.15.104:0/1988009281 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7ff5300c5b70 con 0x7ff5300ce350 2023-06-10T18:54:06.788 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:06.789 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:06.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 msgr2=0x7ff51406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 0x7ff51406d190 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff520006fd0 tx=0x7ff520008040 comp rx=0 tx=0).stop 2023-06-10T18:54:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 msgr2=0x7ff530154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff52800b370 tx=0x7ff528004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 shutdown_connections 2023-06-10T18:54:06.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff51406ace0 0x7ff51406d190 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff53006b1f0 0x7ff53009d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5300ce350 0x7ff530154bf0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 --2- 172.21.15.104:0/1988009281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff53008a3d0 0x7ff53009a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 >> 172.21.15.104:0/1988009281 conn(0x7ff530089fd0 msgr2=0x7ff53015fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 shutdown_connections 2023-06-10T18:54:06.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:06.785+0000 7ff5373f0700 1 -- 172.21.15.104:0/1988009281 wait complete. 2023-06-10T18:54:06.793 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:06.802 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:07.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.931+0000 7f8ecf1b4700 1 Processor -- start 2023-06-10T18:54:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.931+0000 7f8ecf1b4700 1 -- start start 2023-06-10T18:54:07.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ecf1b4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ec80cf100 0x7f8ec806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ecf1b4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec806d290 con 0x7f8ec806e4b0 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ecf1b4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80ca2d0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ecf1b4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec8071640 con 0x7f8ec80cf100 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ec7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ec806e4b0 0x7f8ec80ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:32988/0 (socket says 172.21.15.104:32988) 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ecd751700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ec80cf100 0x7f8ec806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.932+0000 7f8ec7fff700 1 -- 172.21.15.104:0/3024147384 learned_addr learned my addr 172.21.15.104:0/3024147384 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ecd751700 1 -- 172.21.15.104:0/3024147384 >> v1:172.21.15.114:6789/0 conn(0x7f8ec806e0e0 legacy=0x7f8ec80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ecd751700 1 -- 172.21.15.104:0/3024147384 >> v1:172.21.15.104:6789/0 conn(0x7f8ec806e4b0 legacy=0x7f8ec80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ecd751700 1 -- 172.21.15.104:0/3024147384 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec806d4c0 con 0x7f8ec80cf100 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ecd751700 1 --2- 172.21.15.104:0/3024147384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec80cf100 0x7f8ec806cd50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009d90 tx=0x7f8ec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5b382b2a1de56f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ec77fe700 1 -- 172.21.15.104:0/3024147384 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000e070 con 0x7f8ec80cf100 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ec77fe700 1 -- 172.21.15.104:0/3024147384 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec000bbd0 con 0x7f8ec80cf100 2023-06-10T18:54:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ec77fe700 1 -- 172.21.15.104:0/3024147384 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0012430 con 0x7f8ec80cf100 2023-06-10T18:54:07.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.933+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/3024147384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec80cf100 msgr2=0x7f8ec806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:07.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/3024147384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec80cf100 0x7f8ec806cd50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009d90 tx=0x7f8ec000b330 comp rx=0 tx=0).stop 2023-06-10T18:54:07.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/3024147384 shutdown_connections 2023-06-10T18:54:07.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/3024147384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec80cf100 0x7f8ec806cd50 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:07.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/3024147384 >> 172.21.15.104:0/3024147384 conn(0x7f8ec806b8a0 msgr2=0x7f8ec80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:07.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/3024147384 shutdown_connections 2023-06-10T18:54:07.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.934+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/3024147384 wait complete. 2023-06-10T18:54:07.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.935+0000 7f8ecf1b4700 1 Processor -- start 2023-06-10T18:54:07.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.935+0000 7f8ecf1b4700 1 -- start start 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 0x7f8ec80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 0x7f8ec80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec809ccc0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec809cfa0 con 0x7f8ec80cf100 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80b0190 con 0x7f8ec806e4b0 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:07.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35166/0 (socket says 172.21.15.104:35166) 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 -- 172.21.15.104:0/2450124932 learned_addr learned my addr 172.21.15.104:0/2450124932 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ec7fff700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 0x7f8ec80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecd751700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 0x7f8ec80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 -- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 msgr2=0x7f8ec80b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 0x7f8ec80b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 -- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 msgr2=0x7f8ec80b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:07.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 0x7f8ec80b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 -- 172.21.15.104:0/2450124932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec0009a70 con 0x7f8ec806e0e0 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ec7fff700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 0x7f8ec80b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecd751700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 0x7f8ec80b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8eccf50700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c4d0 tx=0x7f8ebc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0052d0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc005430 con 0x7f8ec806e0e0 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec80b0470 con 0x7f8ec806e0e0 2023-06-10T18:54:07.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0038f0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.936+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec809bf90 con 0x7f8ec806e0e0 2023-06-10T18:54:07.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.937+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec80afff0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.938+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8ebc003a50 con 0x7f8ec806e0e0 2023-06-10T18:54:07.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.938+0000 7f8ec57fa700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 0x7f8eac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:07.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.938+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8ebc088cb0 con 0x7f8ec806e0e0 2023-06-10T18:54:07.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.939+0000 7f8ec7fff700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 0x7f8eac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:07.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.939+0000 7f8ec7fff700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 0x7f8eac06d190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8ec814e750 tx=0x7f8eb4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:07.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:07.942+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ec80afff0 con 0x7f8ec806e0e0 2023-06-10T18:54:08.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.172+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec80afff0 con 0x7f8ec806e0e0 2023-06-10T18:54:08.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.172+0000 7f8ec57fa700 1 -- 172.21.15.104:0/2450124932 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f8ec80afff0 con 0x7f8ec806e0e0 2023-06-10T18:54:08.178 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:08.178 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:08.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 msgr2=0x7f8eac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:08.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 0x7f8eac06d190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8ec814e750 tx=0x7f8eb4009040 comp rx=0 tx=0).stop 2023-06-10T18:54:08.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 msgr2=0x7f8ec814aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:08.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c4d0 tx=0x7f8ebc00c890 comp rx=0 tx=0).stop 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 shutdown_connections 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8eac06ace0 0x7f8eac06d190 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ec80cf100 0x7f8ec80b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ec806e0e0 0x7f8ec814aa50 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 --2- 172.21.15.104:0/2450124932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ec806e4b0 0x7f8ec80b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 >> 172.21.15.104:0/2450124932 conn(0x7f8ec806b8a0 msgr2=0x7f8ec80acb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:08.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 shutdown_connections 2023-06-10T18:54:08.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:08.175+0000 7f8ecf1b4700 1 -- 172.21.15.104:0/2450124932 wait complete. 2023-06-10T18:54:08.183 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:08.192 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:09.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:54:09.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:54:09.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:09.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.314+0000 7f11b8075700 1 Processor -- start 2023-06-10T18:54:09.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.314+0000 7f11b8075700 1 -- start start 2023-06-10T18:54:09.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b8075700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11b015a250 0x7f11b00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b8075700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b00ccca0 con 0x7f11b00cad90 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b8075700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b00cee50 con 0x7f11b00cfa00 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b8075700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00cf130 con 0x7f11b015a250 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b5610700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11b015a250 0x7f11b00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b6612700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f11b00cad90 0x7f11b00b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33006/0 (socket says 172.21.15.104:33006) 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b5610700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f11b015a250 0x7f11b00701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35176/0 (socket says 172.21.15.104:35176) 2023-06-10T18:54:09.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.315+0000 7f11b6612700 1 -- 172.21.15.104:0/499122936 learned_addr learned my addr 172.21.15.104:0/499122936 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b5610700 1 -- 172.21.15.104:0/499122936 >> v1:172.21.15.114:6789/0 conn(0x7f11b00cfa00 legacy=0x7f11b00cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b5610700 1 -- 172.21.15.104:0/499122936 >> v1:172.21.15.104:6789/0 conn(0x7f11b00cad90 legacy=0x7f11b00b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b5610700 1 -- 172.21.15.104:0/499122936 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00cc930 con 0x7f11b015a250 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b5610700 1 --2- 172.21.15.104:0/499122936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00701e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009840 tx=0x7f11ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96ecd734d2c4949 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b4e0f700 1 -- 172.21.15.104:0/499122936 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac004230 con 0x7f11b015a250 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b4e0f700 1 -- 172.21.15.104:0/499122936 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac004390 con 0x7f11b015a250 2023-06-10T18:54:09.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b4e0f700 1 -- 172.21.15.104:0/499122936 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0036a0 con 0x7f11b015a250 2023-06-10T18:54:09.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b8075700 1 -- 172.21.15.104:0/499122936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 msgr2=0x7f11b00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b8075700 1 --2- 172.21.15.104:0/499122936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00701e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009840 tx=0x7f11ac009b50 comp rx=0 tx=0).stop 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b8075700 1 -- 172.21.15.104:0/499122936 shutdown_connections 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b8075700 1 --2- 172.21.15.104:0/499122936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00701e0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.316+0000 7f11b8075700 1 -- 172.21.15.104:0/499122936 >> 172.21.15.104:0/499122936 conn(0x7f11b006e370 msgr2=0x7f11b006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.317+0000 7f11b8075700 1 -- 172.21.15.104:0/499122936 shutdown_connections 2023-06-10T18:54:09.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.317+0000 7f11b8075700 1 -- 172.21.15.104:0/499122936 wait complete. 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 Processor -- start 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 -- start start 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 0x7f11b0153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11b00cfa00 0x7f11b00b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b0154500 con 0x7f11b015a250 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b00b0580 con 0x7f11b00cad90 2023-06-10T18:54:09.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.318+0000 7f11b8075700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b00b0860 con 0x7f11b00cfa00 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35182/0 (socket says 172.21.15.104:35182) 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 -- 172.21.15.104:0/676796078 learned_addr learned my addr 172.21.15.104:0/676796078 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b5610700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11b00cfa00 0x7f11b00b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b5e11700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 0x7f11b0153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 -- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11b00cfa00 msgr2=0x7f11b00b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11b00cfa00 0x7f11b00b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 -- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 msgr2=0x7f11b0153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 0x7f11b0153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 -- 172.21.15.104:0/676796078 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac0094a0 con 0x7f11b015a250 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b5e11700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 0x7f11b0153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b6612700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f11a800ca80 tx=0x7f11a800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800aaa0 con 0x7f11b015a250 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a800ac00 con 0x7f11b015a250 2023-06-10T18:54:09.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b00b0b40 con 0x7f11b015a250 2023-06-10T18:54:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8012400 con 0x7f11b015a250 2023-06-10T18:54:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.319+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b00b1100 con 0x7f11b015a250 2023-06-10T18:54:09.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.320+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b00b12f0 con 0x7f11b015a250 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.321+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f11a801b3b0 con 0x7f11b015a250 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.321+0000 7f11a67fc700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 0x7f119406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.321+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f11a80892b0 con 0x7f11b015a250 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.321+0000 7f11b5e11700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 0x7f119406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.322+0000 7f11b5e11700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 0x7f119406d190 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009930 tx=0x7f11a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:09.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.325+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11a80541e0 con 0x7f11b015a250 2023-06-10T18:54:09.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.556+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b0074050 con 0x7f11b015a250 2023-06-10T18:54:09.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.557+0000 7f11a67fc700 1 -- 172.21.15.104:0/676796078 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f11a8057800 con 0x7f11b015a250 2023-06-10T18:54:09.563 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:09.563 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 msgr2=0x7f119406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 0x7f119406d190 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f11a0009930 tx=0x7f11a0008040 comp rx=0 tx=0).stop 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 msgr2=0x7f11b00b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f11a800ca80 tx=0x7f11a800ce40 comp rx=0 tx=0).stop 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 shutdown_connections 2023-06-10T18:54:09.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119406ace0 0x7f119406d190 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11b00cad90 0x7f11b0153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11b015a250 0x7f11b00b2840 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 --2- 172.21.15.104:0/676796078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11b00cfa00 0x7f11b00b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 >> 172.21.15.104:0/676796078 conn(0x7f11b006e370 msgr2=0x7f11b00cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 shutdown_connections 2023-06-10T18:54:09.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:09.559+0000 7f11b8075700 1 -- 172.21.15.104:0/676796078 wait complete. 2023-06-10T18:54:09.567 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:09.576 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:10.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:10.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.706+0000 7ff07e6b2700 1 Processor -- start 2023-06-10T18:54:10.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.706+0000 7ff07e6b2700 1 -- start start 2023-06-10T18:54:10.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07e6b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff0780cf100 0x7ff07806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07e6b2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07806d290 con 0x7ff07806e0e0 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07e6b2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0780ca2d0 con 0x7ff07806e4b0 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07e6b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff078071640 con 0x7ff0780cf100 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff077fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff07806e0e0 0x7ff0780c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33016/0 (socket says 172.21.15.104:33016) 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff077fff700 1 -- 172.21.15.104:0/3248010488 learned_addr learned my addr 172.21.15.104:0/3248010488 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:10.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07cc4f700 1 --2- 172.21.15.104:0/3248010488 >> v2:172.21.15.114:3300/0 conn(0x7ff0780cf100 0x7ff07806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308642192 0 0) 0x7ff07806d290 con 0x7ff07806e0e0 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0600035b0 con 0x7ff07806e0e0 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222158717 0 0) 0x7ff0600035b0 con 0x7ff07806e0e0 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07cc4f700 1 -- 172.21.15.104:0/3248010488 >> v1:172.21.15.114:6789/0 conn(0x7ff07806e4b0 legacy=0x7ff0780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07cc4f700 1 -- 172.21.15.104:0/3248010488 >> v1:172.21.15.104:6789/0 conn(0x7ff07806e0e0 legacy=0x7ff0780c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.707+0000 7ff07cc4f700 1 -- 172.21.15.104:0/3248010488 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07806d4c0 con 0x7ff0780cf100 2023-06-10T18:54:10.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07cc4f700 1 --2- 172.21.15.104:0/3248010488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff07806cd50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff070009d90 tx=0x7ff07000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fcf910649389e70 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:10.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000e070 con 0x7ff0780cf100 2023-06-10T18:54:10.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070003a20 con 0x7ff0780cf100 2023-06-10T18:54:10.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff076ffd700 1 -- 172.21.15.104:0/3248010488 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680017f0 con 0x7ff0780cf100 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 -- 172.21.15.104:0/3248010488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 msgr2=0x7ff07806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/3248010488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff07806cd50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff070009d90 tx=0x7ff07000b330 comp rx=0 tx=0).stop 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 -- 172.21.15.104:0/3248010488 shutdown_connections 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/3248010488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff07806cd50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 -- 172.21.15.104:0/3248010488 >> 172.21.15.104:0/3248010488 conn(0x7ff07806b8a0 msgr2=0x7ff0780cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:10.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 -- 172.21.15.104:0/3248010488 shutdown_connections 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.708+0000 7ff07e6b2700 1 -- 172.21.15.104:0/3248010488 wait complete. 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 Processor -- start 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 -- start start 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff07806e4b0 0x7ff0780b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:10.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff0780b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07809a3e0 con 0x7ff07806e0e0 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07815b130 con 0x7ff0780cf100 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.710+0000 7ff07e6b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff078099d60 con 0x7ff07806e4b0 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35194/0 (socket says 172.21.15.104:35194) 2023-06-10T18:54:10.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 -- 172.21.15.104:0/1598705318 learned_addr learned my addr 172.21.15.104:0/1598705318 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff07cc4f700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff0780b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff0777fe700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff07806e4b0 0x7ff0780b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 -- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff07806e4b0 msgr2=0x7ff0780b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff07806e4b0 0x7ff0780b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 -- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 msgr2=0x7ff0780b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff0780b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 -- 172.21.15.104:0/1598705318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff070009a70 con 0x7ff07806e0e0 2023-06-10T18:54:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff077fff700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ff068001920 tx=0x7ff06800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680074e0 con 0x7ff07806e0e0 2023-06-10T18:54:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068007640 con 0x7ff07806e0e0 2023-06-10T18:54:10.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07809ce40 con 0x7ff07806e0e0 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068005a50 con 0x7ff07806e0e0 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.711+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07809d120 con 0x7ff07806e0e0 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.712+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff078074050 con 0x7ff07806e0e0 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.713+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff068005bb0 con 0x7ff07806e0e0 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.713+0000 7ff074ff9700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.713+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff068088b90 con 0x7ff07806e0e0 2023-06-10T18:54:10.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.713+0000 7ff0777fe700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:10.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.714+0000 7ff0777fe700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0780997f0 tx=0x7ff06c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:10.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.717+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff068053ab0 con 0x7ff07806e0e0 2023-06-10T18:54:10.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.944+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0780729e0 con 0x7ff07806e0e0 2023-06-10T18:54:10.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.945+0000 7ff074ff9700 1 -- 172.21.15.104:0/1598705318 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7ff068005e60 con 0x7ff07806e0e0 2023-06-10T18:54:10.951 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:10.951 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 msgr2=0x7ff06406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0780997f0 tx=0x7ff06c009040 comp rx=0 tx=0).stop 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 msgr2=0x7ff07814abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ff068001920 tx=0x7ff06800c8b0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 shutdown_connections 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0780cf100 0x7ff0780b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07806e0e0 0x7ff07814abe0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 --2- 172.21.15.104:0/1598705318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff07806e4b0 0x7ff0780b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 >> 172.21.15.104:0/1598705318 conn(0x7ff07806b8a0 msgr2=0x7ff0780cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 shutdown_connections 2023-06-10T18:54:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:10.947+0000 7ff07e6b2700 1 -- 172.21.15.104:0/1598705318 wait complete. 2023-06-10T18:54:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:10.964 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:11.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:12.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 Processor -- start 2023-06-10T18:54:12.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 -- start start 2023-06-10T18:54:12.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7d415a250 0x7fa7d40701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d40ccca0 con 0x7fa7d40cad90 2023-06-10T18:54:12.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d40cee50 con 0x7fa7d40cfa00 2023-06-10T18:54:12.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.095+0000 7fa7dafd6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d40cf130 con 0x7fa7d415a250 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d9573700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa7d40cad90 0x7fa7d40b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33034/0 (socket says 172.21.15.104:33034) 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d9573700 1 -- 172.21.15.104:0/637560177 learned_addr learned my addr 172.21.15.104:0/637560177 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/637560177 >> v2:172.21.15.114:3300/0 conn(0x7fa7d415a250 0x7fa7d40701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480439876 0 0) 0x7fa7d40ccca0 con 0x7fa7d40cad90 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c40035b0 con 0x7fa7d40cad90 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4287886469 0 0) 0x7fa7c40035b0 con 0x7fa7d40cad90 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 >> v2:172.21.15.114:3300/0 conn(0x7fa7d415a250 msgr2=0x7fa7d40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:12.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 --2- 172.21.15.104:0/637560177 >> v2:172.21.15.114:3300/0 conn(0x7fa7d415a250 0x7fa7d40701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 >> v1:172.21.15.114:6789/0 conn(0x7fa7d40cfa00 legacy=0x7fa7d40cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40cc930 con 0x7fa7d40cad90 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa7cc0019f0 con 0x7fa7d40cad90 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 >> v1:172.21.15.104:6789/0 conn(0x7fa7d40cad90 legacy=0x7fa7d40b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4004850 0x7fa7c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c4005200 0x7fa7c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d40cf130 con 0x7fa7c4003a00 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c400c110 con 0x7fa7c4004850 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c400c2d0 con 0x7fa7c4005200 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/637560177 >> v2:172.21.15.114:3300/0 conn(0x7fa7d415a250 0x7fa7d40701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d9573700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c4005200 0x7fa7c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.096+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4004850 0x7fa7c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 -- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c4005200 msgr2=0x7fa7c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c4005200 0x7fa7c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 -- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4004850 msgr2=0x7fa7c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4004850 0x7fa7c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 -- 172.21.15.104:0/637560177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c40035b0 con 0x7fa7c4003a00 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d9573700 1 -- 172.21.15.104:0/637560177 reap_dead start 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c4004850 0x7fa7c4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:12.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7c4003e10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80017c0 tx=0x7fa7c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8faa7c65c6b517 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80052d0 con 0x7fa7c4003a00 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c8005450 con 0x7fa7c4003a00 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7d37fe700 1 -- 172.21.15.104:0/637560177 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80038c0 con 0x7fa7c4003a00 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7dafd6700 1 -- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 msgr2=0x7fa7c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.097+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7c4003e10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80017c0 tx=0x7fa7c4005880 comp rx=0 tx=0).stop 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.098+0000 7fa7dafd6700 1 -- 172.21.15.104:0/637560177 shutdown_connections 2023-06-10T18:54:12.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.098+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/637560177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7c4003e10 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.098+0000 7fa7dafd6700 1 -- 172.21.15.104:0/637560177 >> 172.21.15.104:0/637560177 conn(0x7fa7d406e370 msgr2=0x7fa7d406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:12.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.098+0000 7fa7dafd6700 1 -- 172.21.15.104:0/637560177 shutdown_connections 2023-06-10T18:54:12.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.098+0000 7fa7dafd6700 1 -- 172.21.15.104:0/637560177 wait complete. 2023-06-10T18:54:12.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.099+0000 7fa7dafd6700 1 Processor -- start 2023-06-10T18:54:12.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.099+0000 7fa7dafd6700 1 -- start start 2023-06-10T18:54:12.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 0x7fa7d414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 0x7fa7d4155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d415d6a0 con 0x7fa7c4003a00 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d4156070 con 0x7fa7d415c470 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d415a650 con 0x7fa7d41556c0 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35214/0 (socket says 172.21.15.104:35214) 2023-06-10T18:54:12.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 -- 172.21.15.104:0/3016686663 learned_addr learned my addr 172.21.15.104:0/3016686663 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d9573700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 0x7fa7d4155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 0x7fa7d414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 -- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 msgr2=0x7fa7d4155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 0x7fa7d4155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 -- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 msgr2=0x7fa7d414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 0x7fa7d414cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 -- 172.21.15.104:0/3016686663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40b2630 con 0x7fa7c4003a00 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d9573700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 0x7fa7d4155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 0x7fa7d414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d8d72700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400c0e0 tx=0x7fa7c8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8001290 con 0x7fa7c4003a00 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c8007040 con 0x7fa7c4003a00 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d40b2910 con 0x7fa7c4003a00 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80073f0 con 0x7fa7c4003a00 2023-06-10T18:54:12.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.100+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d415d3b0 con 0x7fa7c4003a00 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.101+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d4074050 con 0x7fa7c4003a00 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.102+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa7c8007550 con 0x7fa7c4003a00 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.102+0000 7fa7d17fa700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 0x7fa7b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.102+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fa7c8089f90 con 0x7fa7c4003a00 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.102+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 0x7fa7b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:12.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.103+0000 7fa7d3fff700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 0x7fa7b806d190 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4070750 tx=0x7fa7c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:12.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.107+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa7c8054e90 con 0x7fa7c4003a00 2023-06-10T18:54:12.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.334+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7d414d170 con 0x7fa7c4003a00 2023-06-10T18:54:12.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.335+0000 7fa7d17fa700 1 -- 172.21.15.104:0/3016686663 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fa7c8017090 con 0x7fa7c4003a00 2023-06-10T18:54:12.341 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:12.341 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:12.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 msgr2=0x7fa7b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 0x7fa7b806d190 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4070750 tx=0x7fa7c0008040 comp rx=0 tx=0).stop 2023-06-10T18:54:12.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 msgr2=0x7fa7d415ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:12.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400c0e0 tx=0x7fa7c8001120 comp rx=0 tx=0).stop 2023-06-10T18:54:12.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 shutdown_connections 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7d41556c0 0x7fa7d4155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c4003a00 0x7fa7d415ce70 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa7b806ace0 0x7fa7b806d190 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 --2- 172.21.15.104:0/3016686663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7d415c470 0x7fa7d414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 >> 172.21.15.104:0/3016686663 conn(0x7fa7d406e370 msgr2=0x7fa7d4154280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 shutdown_connections 2023-06-10T18:54:12.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:12.337+0000 7fa7dafd6700 1 -- 172.21.15.104:0/3016686663 wait complete. 2023-06-10T18:54:12.345 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:12.355 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:13.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.486+0000 7f54501d6700 1 Processor -- start 2023-06-10T18:54:13.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.486+0000 7f54501d6700 1 -- start start 2023-06-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f54501d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 0x7f544806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f54501d6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f54501d6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480cccd0 con 0x7f544806b180 2023-06-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f54501d6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54480ccfb0 con 0x7f544806d150 2023-06-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f54501d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54480ce0b0 con 0x7f5448158c50 2023-06-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f544e773700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f544e773700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:50960/0 (socket says 172.21.15.104:50960) 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f544df72700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 0x7f544806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f544e773700 1 -- 172.21.15.104:0/1345879535 learned_addr learned my addr 172.21.15.104:0/1345879535 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.487+0000 7f544d771700 1 --1- 172.21.15.104:0/1345879535 >> v1:172.21.15.114:6789/0 conn(0x7f544806b180 0x7f544806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:38898/0 (socket says 172.21.15.104:38898) 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544df72700 1 -- 172.21.15.104:0/1345879535 >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 msgr2=0x7f54480cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544df72700 1 --2- 172.21.15.104:0/1345879535 >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544df72700 1 -- 172.21.15.104:0/1345879535 >> v1:172.21.15.114:6789/0 conn(0x7f544806b180 legacy=0x7f544806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544df72700 1 -- 172.21.15.104:0/1345879535 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544806a010 con 0x7f5448158c50 2023-06-10T18:54:13.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544e773700 1 --2- 172.21.15.104:0/1345879535 >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544df72700 1 --2- 172.21.15.104:0/1345879535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 0x7f544806aaf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5444009840 tx=0x7f5444009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6164a8fa29927269 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544cf70700 1 -- 172.21.15.104:0/1345879535 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444011830 con 0x7f5448158c50 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544cf70700 1 -- 172.21.15.104:0/1345879535 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5444011990 con 0x7f5448158c50 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f544cf70700 1 -- 172.21.15.104:0/1345879535 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444011d20 con 0x7f5448158c50 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f54501d6700 1 -- 172.21.15.104:0/1345879535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 msgr2=0x7f544806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.488+0000 7f54501d6700 1 --2- 172.21.15.104:0/1345879535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 0x7f544806aaf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5444009840 tx=0x7f5444009b50 comp rx=0 tx=0).stop 2023-06-10T18:54:13.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 -- 172.21.15.104:0/1345879535 shutdown_connections 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 --2- 172.21.15.104:0/1345879535 >> v2:172.21.15.114:3300/0 conn(0x7f544806d150 0x7f54480cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 --2- 172.21.15.104:0/1345879535 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5448158c50 0x7f544806aaf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 -- 172.21.15.104:0/1345879535 >> 172.21.15.104:0/1345879535 conn(0x7f54480b1610 msgr2=0x7f54480ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 -- 172.21.15.104:0/1345879535 shutdown_connections 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.489+0000 7f54501d6700 1 -- 172.21.15.104:0/1345879535 wait complete. 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.490+0000 7f54501d6700 1 Processor -- start 2023-06-10T18:54:13.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.490+0000 7f54501d6700 1 -- start start 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f544806b180 0x7f544815bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f544806d150 0x7f5448159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54481f5900 con 0x7f5448158c50 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54481f4fd0 con 0x7f544806b180 2023-06-10T18:54:13.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54481f52b0 con 0x7f544806d150 2023-06-10T18:54:13.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544df72700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f544806b180 0x7f544815bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544d771700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f544806d150 0x7f5448159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35232/0 (socket says 172.21.15.104:35232) 2023-06-10T18:54:13.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 -- 172.21.15.104:0/2985301301 learned_addr learned my addr 172.21.15.104:0/2985301301 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 -- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f544806d150 msgr2=0x7f5448159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f544806d150 0x7f5448159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 -- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f544806b180 msgr2=0x7f544815bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f544806b180 0x7f544815bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 -- 172.21.15.104:0/2985301301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440009b30 con 0x7f5448158c50 2023-06-10T18:54:13.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f544e773700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f544000efc0 tx=0x7f544000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440018070 con 0x7f5448158c50 2023-06-10T18:54:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544000f040 con 0x7f5448158c50 2023-06-10T18:54:13.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440003c20 con 0x7f5448158c50 2023-06-10T18:54:13.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54440094a0 con 0x7f5448158c50 2023-06-10T18:54:13.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.491+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54481f5d90 con 0x7f5448158c50 2023-06-10T18:54:13.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.492+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54481f5a70 con 0x7f5448158c50 2023-06-10T18:54:13.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.493+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f544000c880 con 0x7f5448158c50 2023-06-10T18:54:13.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.493+0000 7f543e7fc700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 0x7f542c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:13.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.493+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f5440089720 con 0x7f5448158c50 2023-06-10T18:54:13.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.493+0000 7f544df72700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 0x7f542c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:13.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.494+0000 7f544df72700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 0x7f542c06d190 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5444009fd0 tx=0x7f544401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:13.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.497+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f54400546c0 con 0x7f5448158c50 2023-06-10T18:54:13.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.726+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5448074050 con 0x7f5448158c50 2023-06-10T18:54:13.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.727+0000 7f543e7fc700 1 -- 172.21.15.104:0/2985301301 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f5440057ce0 con 0x7f5448158c50 2023-06-10T18:54:13.733 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:13.734 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 msgr2=0x7f542c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 0x7f542c06d190 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5444009fd0 tx=0x7f544401c040 comp rx=0 tx=0).stop 2023-06-10T18:54:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 msgr2=0x7f54480cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:13.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f544000efc0 tx=0x7f544000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 shutdown_connections 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f542c06ace0 0x7f542c06d190 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f544806b180 0x7f544815bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5448158c50 0x7f54480cb5d0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 --2- 172.21.15.104:0/2985301301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f544806d150 0x7f5448159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:13.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 >> 172.21.15.104:0/2985301301 conn(0x7f54480b1610 msgr2=0x7f54480aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 shutdown_connections 2023-06-10T18:54:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:13.729+0000 7f54501d6700 1 -- 172.21.15.104:0/2985301301 wait complete. 2023-06-10T18:54:13.737 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:13.747 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:14.749 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:14.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 Processor -- start 2023-06-10T18:54:14.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 -- start start 2023-06-10T18:54:14.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7abc0b1e30 0x7f7abc0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7abc06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7abc06bfd0 con 0x7f7abc06b820 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc0d0ea0 con 0x7f7abc0b1e30 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.880+0000 7f7ac19b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc06c2b0 con 0x7f7abc0d0a30 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7abb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7abc06b820 0x7f7abc0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33066/0 (socket says 172.21.15.104:33066) 2023-06-10T18:54:14.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7abb7fe700 1 -- 172.21.15.104:0/296663583 learned_addr learned my addr 172.21.15.104:0/296663583 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7abaffd700 1 --2- 172.21.15.104:0/296663583 >> v2:172.21.15.114:3300/0 conn(0x7f7abc0b1e30 0x7f7abc0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7aba7fc700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7abc06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065875536 0 0) 0x7f7abc06bfd0 con 0x7f7abc06b820 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa80035b0 con 0x7f7abc06b820 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170122341 0 0) 0x7f7aa80035b0 con 0x7f7abc06b820 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 msgr2=0x7f7abc06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7abc06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 >> v2:172.21.15.114:3300/0 conn(0x7f7abc0b1e30 msgr2=0x7f7abc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 --2- 172.21.15.104:0/296663583 >> v2:172.21.15.114:3300/0 conn(0x7f7abc0b1e30 0x7f7abc0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0ccf70 con 0x7f7abc06b820 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7ab40019f0 con 0x7f7abc06b820 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 >> v1:172.21.15.104:6789/0 conn(0x7f7abc06b820 legacy=0x7f7abc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa8003a00 0x7f7aa8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7aa8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aa8004d90 0x7f7aa8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0d0ea0 con 0x7f7aa8004d90 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc06c2b0 con 0x7f7aa8003a00 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa800ba10 con 0x7f7abc0d0a30 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7abb7fe700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aa8004d90 0x7f7aa8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7abaffd700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa8003a00 0x7f7aa8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.881+0000 7f7aba7fc700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7aa8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 -- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 msgr2=0x7f7aa8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:14.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0d0a30 0x7f7aa8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 -- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa8003a00 msgr2=0x7f7aa8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aa8003a00 0x7f7aa8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 -- 172.21.15.104:0/296663583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80035b0 con 0x7f7aa8004d90 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aa8004d90 0x7f7aa8005200 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400a8d0 tx=0x7f7aa8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7795d5333d6abe4c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab400a340 con 0x7f7aa8004d90 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab4012600 con 0x7f7aa8004d90 2023-06-10T18:54:14.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7ab9ffb700 1 -- 172.21.15.104:0/296663583 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab40129b0 con 0x7f7aa8004d90 2023-06-10T18:54:14.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7ac19b2700 1 -- 172.21.15.104:0/296663583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aa8004d90 msgr2=0x7f7aa8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:14.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/296663583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aa8004d90 0x7f7aa8005200 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400a8d0 tx=0x7f7aa8005880 comp rx=0 tx=0).stop 2023-06-10T18:54:14.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.882+0000 7f7abb7fe700 1 -- 172.21.15.104:0/296663583 reap_dead start 2023-06-10T18:54:14.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.883+0000 7f7ac19b2700 1 -- 172.21.15.104:0/296663583 shutdown_connections 2023-06-10T18:54:14.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.883+0000 7f7ac19b2700 1 -- 172.21.15.104:0/296663583 >> 172.21.15.104:0/296663583 conn(0x7f7abc071ec0 msgr2=0x7f7abc09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:14.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.883+0000 7f7ac19b2700 1 -- 172.21.15.104:0/296663583 shutdown_connections 2023-06-10T18:54:14.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.883+0000 7f7ac19b2700 1 -- 172.21.15.104:0/296663583 wait complete. 2023-06-10T18:54:14.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.884+0000 7f7ac19b2700 1 Processor -- start 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.884+0000 7f7ac19b2700 1 -- start start 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7abc15d3f0 0x7f7abc0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0b7720 0x7f7abc0c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc15e3f0 con 0x7f7abc0b6e10 2023-06-10T18:54:14.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc152300 con 0x7f7abc15d3f0 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc15d870 con 0x7f7abc0b7720 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35248/0 (socket says 172.21.15.104:35248) 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 -- 172.21.15.104:0/3136706229 learned_addr learned my addr 172.21.15.104:0/3136706229 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7aba7fc700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7abc15d3f0 0x7f7abc0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abb7fe700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0b7720 0x7f7abc0c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 -- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0b7720 msgr2=0x7f7abc0c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0b7720 0x7f7abc0c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 -- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7abc15d3f0 msgr2=0x7f7abc0b71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7abc15d3f0 0x7f7abc0b71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 -- 172.21.15.104:0/3136706229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc15db50 con 0x7f7abc0b6e10 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7abaffd700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f7aac011aa0 tx=0x7f7aac011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac011ed0 con 0x7f7abc0b6e10 2023-06-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac00f940 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc15de30 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac00fd10 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.885+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc0b7bc0 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.886+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc0d1860 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.887+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7aac014440 con 0x7f7abc0b6e10 2023-06-10T18:54:14.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.887+0000 7f7ab3fff700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 0x7f7a9c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:14.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.887+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f7aac089c40 con 0x7f7abc0b6e10 2023-06-10T18:54:14.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.888+0000 7f7aba7fc700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 0x7f7a9c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:14.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.888+0000 7f7aba7fc700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 0x7f7a9c06d1c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0af500 tx=0x7f7aa4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:14.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:14.892+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7aac058690 con 0x7f7abc0b6e10 2023-06-10T18:54:15.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.119+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7abc074050 con 0x7f7abc0b6e10 2023-06-10T18:54:15.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.120+0000 7f7ab3fff700 1 -- 172.21.15.104:0/3136706229 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f7aac022090 con 0x7f7abc0b6e10 2023-06-10T18:54:15.126 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:15.126 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:15.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 msgr2=0x7f7a9c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:15.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 0x7f7a9c06d1c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0af500 tx=0x7f7aa4009210 comp rx=0 tx=0).stop 2023-06-10T18:54:15.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 msgr2=0x7f7abc0b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:15.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f7aac011aa0 tx=0x7f7aac011db0 comp rx=0 tx=0).stop 2023-06-10T18:54:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 shutdown_connections 2023-06-10T18:54:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7abc0b7720 0x7f7abc0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7abc0b6e10 0x7f7abc0b7fe0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9c06ad10 0x7f7a9c06d1c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:15.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 --2- 172.21.15.104:0/3136706229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7abc15d3f0 0x7f7abc0b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:15.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 >> 172.21.15.104:0/3136706229 conn(0x7f7abc071ec0 msgr2=0x7f7abc0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:15.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 shutdown_connections 2023-06-10T18:54:15.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:15.122+0000 7f7ac19b2700 1 -- 172.21.15.104:0/3136706229 wait complete. 2023-06-10T18:54:15.131 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:15.140 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:16.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:16.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 Processor -- start 2023-06-10T18:54:16.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 -- start start 2023-06-10T18:54:16.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57b406e790 0x7f57b40ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b415cb00 con 0x7f57b40cc710 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b40cd280 con 0x7f57b415c730 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.269+0000 7f57ba50c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b40aed80 con 0x7f57b406e790 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b37fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f57b40cc710 0x7f57b40ca0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33086/0 (socket says 172.21.15.104:33086) 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b37fe700 1 -- 172.21.15.104:0/4148617081 learned_addr learned my addr 172.21.15.104:0/4148617081 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b8aa9700 1 --2- 172.21.15.104:0/4148617081 >> v2:172.21.15.114:3300/0 conn(0x7f57b406e790 0x7f57b40ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266436655 0 0) 0x7f57a8001110 con 0x7f57b40cc710 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c0035b0 con 0x7f57b40cc710 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813985817 0 0) 0x7f579c0035b0 con 0x7f57b40cc710 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 >> v2:172.21.15.114:3300/0 conn(0x7f57b406e790 msgr2=0x7f57b40ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 --2- 172.21.15.104:0/4148617081 >> v2:172.21.15.114:3300/0 conn(0x7f57b406e790 0x7f57b40ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 >> v1:172.21.15.114:6789/0 conn(0x7f57b415c730 legacy=0x7f57b40cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40cd560 con 0x7f57b40cc710 2023-06-10T18:54:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f57a8001800 con 0x7f57b40cc710 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 >> v1:172.21.15.104:6789/0 conn(0x7f57b40cc710 legacy=0x7f57b40ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 0x7f579c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c0043f0 0x7f579c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f579c005150 0x7f579c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40aed80 con 0x7f579c003a00 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c00c080 con 0x7f579c005150 2023-06-10T18:54:16.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579c00c240 con 0x7f579c0043f0 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b3fff700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 0x7f579c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b37fe700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c0043f0 0x7f579c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.270+0000 7f57b8aa9700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f579c005150 0x7f579c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 -- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c0043f0 msgr2=0x7f579c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c0043f0 0x7f579c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 -- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f579c005150 msgr2=0x7f579c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f579c005150 0x7f579c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b8aa9700 1 -- 172.21.15.104:0/4148617081 reap_dead start 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 -- 172.21.15.104:0/4148617081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c0035b0 con 0x7f579c003a00 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b37fe700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c0043f0 0x7f579c0048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b3fff700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 0x7f579c003dd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57a4001930 tx=0x7f579c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba3ab10dddce9c2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a40013a0 con 0x7f579c003a00 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a40052f0 con 0x7f579c003a00 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57b2ffd700 1 -- 172.21.15.104:0/4148617081 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a40036d0 con 0x7f579c003a00 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 -- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 msgr2=0x7f579c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 0x7f579c003dd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57a4001930 tx=0x7f579c005880 comp rx=0 tx=0).stop 2023-06-10T18:54:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 -- 172.21.15.104:0/4148617081 shutdown_connections 2023-06-10T18:54:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 --2- 172.21.15.104:0/4148617081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f579c003a00 0x7f579c003dd0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 -- 172.21.15.104:0/4148617081 >> 172.21.15.104:0/4148617081 conn(0x7f57b406b8a0 msgr2=0x7f57b40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 -- 172.21.15.104:0/4148617081 shutdown_connections 2023-06-10T18:54:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.271+0000 7f57ba50c700 1 -- 172.21.15.104:0/4148617081 wait complete. 2023-06-10T18:54:16.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.273+0000 7f57ba50c700 1 Processor -- start 2023-06-10T18:54:16.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.273+0000 7f57ba50c700 1 -- start start 2023-06-10T18:54:16.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c003a00 0x7f57b41519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 0x7f57b41e2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b41e2d90 con 0x7f57b4151fa0 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4152810 con 0x7f57b41568a0 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40b0670 con 0x7f579c003a00 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35272/0 (socket says 172.21.15.104:35272) 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 -- 172.21.15.104:0/2331221060 learned_addr learned my addr 172.21.15.104:0/2331221060 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b37fe700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 0x7f57b41e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b3fff700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c003a00 0x7f57b41519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 -- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c003a00 msgr2=0x7f57b41519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c003a00 0x7f57b41519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 -- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 msgr2=0x7f57b41e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 0x7f57b41e2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 -- 172.21.15.104:0/2331221060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b41e2f80 con 0x7f57b4151fa0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b37fe700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 0x7f57b41e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b8aa9700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c2f40 tx=0x7f57ac011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00e040 con 0x7f57b4151fa0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac00f040 con 0x7f57b4151fa0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b40b0910 con 0x7f57b4151fa0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00fc40 con 0x7f57b4151fa0 2023-06-10T18:54:16.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.274+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b41e3260 con 0x7f57b4151fa0 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.275+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b40b0e30 con 0x7f57b4151fa0 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.276+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f57ac015430 con 0x7f57b4151fa0 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.276+0000 7f57b0ff9700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 0x7f57a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.276+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f57ac089e80 con 0x7f57b4151fa0 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.277+0000 7f57b3fff700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 0x7f57a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:16.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.277+0000 7f57b3fff700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 0x7f57a006d190 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c5d30 tx=0x7f57a4005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:16.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.281+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f57b40b0e30 con 0x7f57b4151fa0 2023-06-10T18:54:16.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.510+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b40b0e30 con 0x7f57b4151fa0 2023-06-10T18:54:16.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.510+0000 7f57b0ff9700 1 -- 172.21.15.104:0/2331221060 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f57b40b0e30 con 0x7f57b4151fa0 2023-06-10T18:54:16.516 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:16.517 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:16.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 msgr2=0x7f57a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 0x7f57a006d190 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c5d30 tx=0x7f57a4005000 comp rx=0 tx=0).stop 2023-06-10T18:54:16.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 msgr2=0x7f57b40c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:16.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f57b40c2f40 tx=0x7f57ac011ed0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 shutdown_connections 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57b4151fa0 0x7f57b40c46a0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f579c003a00 0x7f57b41519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f57a006ace0 0x7f57a006d190 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 --2- 172.21.15.104:0/2331221060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57b41568a0 0x7f57b41e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 >> 172.21.15.104:0/2331221060 conn(0x7f57b406b8a0 msgr2=0x7f57b40b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 shutdown_connections 2023-06-10T18:54:16.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:16.512+0000 7f57ba50c700 1 -- 172.21.15.104:0/2331221060 wait complete. 2023-06-10T18:54:16.520 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:16.530 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:17.531 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:17.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.653+0000 7fcabaedd700 1 Processor -- start 2023-06-10T18:54:17.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.653+0000 7fcabaedd700 1 -- start start 2023-06-10T18:54:17.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcabaedd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcab406e4b0 0x7fcab406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcabaedd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcab40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcabaedd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab406d290 con 0x7fcab406e0e0 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcabaedd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab4071640 con 0x7fcab406e4b0 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcabaedd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40aef00 con 0x7fcab4069e70 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab8c79700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcab406e0e0 0x7fcab40c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33104/0 (socket says 172.21.15.104:33104) 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab8c79700 1 -- 172.21.15.104:0/3561550973 learned_addr learned my addr 172.21.15.104:0/3561550973 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:17.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632415730 0 0) 0x7fcab406d290 con 0x7fcab406e0e0 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa40035b0 con 0x7fcab406e0e0 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab3fff700 1 --2- 172.21.15.104:0/3561550973 >> v2:172.21.15.114:3300/0 conn(0x7fcab406e4b0 0x7fcab406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab947a700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcab40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172613261 0 0) 0x7fcaa40035b0 con 0x7fcab406e0e0 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 msgr2=0x7fcab40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcab40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 >> v2:172.21.15.114:3300/0 conn(0x7fcab406e4b0 msgr2=0x7fcab406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 --2- 172.21.15.104:0/3561550973 >> v2:172.21.15.114:3300/0 conn(0x7fcab406e4b0 0x7fcab406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40ca2d0 con 0x7fcab406e0e0 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcaa80017f0 con 0x7fcab406e0e0 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 >> v1:172.21.15.104:6789/0 conn(0x7fcab406e0e0 legacy=0x7fcab40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa4003a00 0x7fcaa4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcaa4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcaa4004d90 0x7fcaa4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4071640 con 0x7fcaa4004d90 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40aef00 con 0x7fcaa4003a00 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa400bc20 con 0x7fcab4069e70 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.654+0000 7fcab3fff700 1 --2- 172.21.15.104:0/3561550973 >> v2:172.21.15.114:3300/0 conn(0x7fcab406e4b0 0x7fcab406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcaa4004d90 0x7fcaa4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab8c79700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa4003a00 0x7fcaa4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab3fff700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcaa4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 -- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 msgr2=0x7fcaa4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcaa4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 -- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa4003a00 msgr2=0x7fcaa4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa4003a00 0x7fcaa4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 -- 172.21.15.104:0/3561550973 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa40035b0 con 0x7fcaa4004d90 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab3fff700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab4069e70 0x7fcaa4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab947a700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcaa4004d90 0x7fcaa4005200 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcaac007360 tx=0x7fcaa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6dd56089f4292f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac007e80 con 0x7fcaa4004d90 2023-06-10T18:54:17.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac014400 con 0x7fcaa4004d90 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.655+0000 7fcab37fe700 1 -- 172.21.15.104:0/3561550973 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac014790 con 0x7fcaa4004d90 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 -- 172.21.15.104:0/3561550973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcaa4004d90 msgr2=0x7fcaa4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 --2- 172.21.15.104:0/3561550973 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcaa4004d90 0x7fcaa4005200 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcaac007360 tx=0x7fcaa4005880 comp rx=0 tx=0).stop 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcab947a700 1 -- 172.21.15.104:0/3561550973 reap_dead start 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 -- 172.21.15.104:0/3561550973 shutdown_connections 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 -- 172.21.15.104:0/3561550973 >> 172.21.15.104:0/3561550973 conn(0x7fcab406b8a0 msgr2=0x7fcab40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 -- 172.21.15.104:0/3561550973 shutdown_connections 2023-06-10T18:54:17.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.656+0000 7fcabaedd700 1 -- 172.21.15.104:0/3561550973 wait complete. 2023-06-10T18:54:17.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.657+0000 7fcabaedd700 1 Processor -- start 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.657+0000 7fcabaedd700 1 -- start start 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab40b1090 0x7fcab41f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 0x7fcab40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40af3e0 con 0x7fcab40b1460 2023-06-10T18:54:17.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40af040 con 0x7fcab40b1090 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab41f5bf0 con 0x7fcab41e6490 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35290/0 (socket says 172.21.15.104:35290) 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 -- 172.21.15.104:0/2959262393 learned_addr learned my addr 172.21.15.104:0/2959262393 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab947a700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 0x7fcab40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab8c79700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab40b1090 0x7fcab41f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 -- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 msgr2=0x7fcab40c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 0x7fcab40c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 -- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab40b1090 msgr2=0x7fcab41f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab40b1090 0x7fcab41f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 -- 172.21.15.104:0/2959262393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab41f5ed0 con 0x7fcab40b1460 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab947a700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 0x7fcab40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab3fff700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0010890 tx=0x7fcaa0010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000e860 con 0x7fcab40b1460 2023-06-10T18:54:17.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa000c050 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab41f61b0 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.658+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000ebc0 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.659+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab41f34b0 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.659+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab40af520 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.660+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcaa000ed20 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.660+0000 7fcab17fa700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 0x7fca9806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.660+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fcaa0089df0 con 0x7fcab40b1460 2023-06-10T18:54:17.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.661+0000 7fcab8c79700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 0x7fca9806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:17.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.661+0000 7fcab8c79700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 0x7fca9806d1c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000c00 tx=0x7fcaa8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:17.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.665+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcab40af520 con 0x7fcab40b1460 2023-06-10T18:54:17.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.892+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcab40af520 con 0x7fcab40b1460 2023-06-10T18:54:17.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.893+0000 7fcab17fa700 1 -- 172.21.15.104:0/2959262393 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fcab40af520 con 0x7fcab40b1460 2023-06-10T18:54:17.899 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:17.899 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 msgr2=0x7fca9806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 0x7fca9806d1c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000c00 tx=0x7fcaa8009040 comp rx=0 tx=0).stop 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 msgr2=0x7fcab41f2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0010890 tx=0x7fcaa0010c50 comp rx=0 tx=0).stop 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 shutdown_connections 2023-06-10T18:54:17.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca9806ad10 0x7fca9806d1c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab40b1090 0x7fcab41f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab41e6490 0x7fcab40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 --2- 172.21.15.104:0/2959262393 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab40b1460 0x7fcab41f2f70 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 >> 172.21.15.104:0/2959262393 conn(0x7fcab406b8a0 msgr2=0x7fcab40cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 shutdown_connections 2023-06-10T18:54:17.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:17.895+0000 7fcabaedd700 1 -- 172.21.15.104:0/2959262393 wait complete. 2023-06-10T18:54:17.903 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:17.912 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:18.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:19.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.041+0000 7f8427870700 1 Processor -- start 2023-06-10T18:54:19.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.042+0000 7f8427870700 1 -- start start 2023-06-10T18:54:19.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.042+0000 7f8427870700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84200aeaa0 0x7f84200cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.042+0000 7f8427870700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200c9e60 con 0x7f84200cd680 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.042+0000 7f8427870700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200cdaa0 con 0x7f84200b1080 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.042+0000 7f8427870700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84200c9bc0 con 0x7f84200aeaa0 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f842560c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f84200aeaa0 0x7f84200cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8425e0d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f84200cd680 0x7f842006dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33128/0 (socket says 172.21.15.104:33128) 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8425e0d700 1 -- 172.21.15.104:0/4252897851 learned_addr learned my addr 172.21.15.104:0/4252897851 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:19.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f842560c700 1 -- 172.21.15.104:0/4252897851 >> v1:172.21.15.114:6789/0 conn(0x7f84200b1080 legacy=0x7f84200ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f842560c700 1 -- 172.21.15.104:0/4252897851 >> v1:172.21.15.104:6789/0 conn(0x7f84200cd680 legacy=0x7f842006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f842560c700 1 -- 172.21.15.104:0/4252897851 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200aee70 con 0x7f84200aeaa0 2023-06-10T18:54:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8417fff700 1 -- 172.21.15.104:0/4252897851 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933776215 0 0) 0x7f84200c9e60 con 0x7f84200cd680 2023-06-10T18:54:19.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f842560c700 1 --2- 172.21.15.104:0/4252897851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200cd140 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f841000bac0 tx=0x7f841000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad165bc4798b4b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8417fff700 1 -- 172.21.15.104:0/4252897851 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410005e30 con 0x7f84200aeaa0 2023-06-10T18:54:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8417fff700 1 -- 172.21.15.104:0/4252897851 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410004030 con 0x7f84200aeaa0 2023-06-10T18:54:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.043+0000 7f8417fff700 1 -- 172.21.15.104:0/4252897851 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84100043c0 con 0x7f84200aeaa0 2023-06-10T18:54:19.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 -- 172.21.15.104:0/4252897851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 msgr2=0x7f84200cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 --2- 172.21.15.104:0/4252897851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200cd140 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f841000bac0 tx=0x7f841000bdd0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 -- 172.21.15.104:0/4252897851 shutdown_connections 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 --2- 172.21.15.104:0/4252897851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200cd140 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 -- 172.21.15.104:0/4252897851 >> 172.21.15.104:0/4252897851 conn(0x7f842006b8a0 msgr2=0x7f84200cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 -- 172.21.15.104:0/4252897851 shutdown_connections 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.044+0000 7f8427870700 1 -- 172.21.15.104:0/4252897851 wait complete. 2023-06-10T18:54:19.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.045+0000 7f8427870700 1 Processor -- start 2023-06-10T18:54:19.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.045+0000 7f8427870700 1 -- start start 2023-06-10T18:54:19.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84200b1080 0x7f84200c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 0x7f84200c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200c3b30 con 0x7f84200aeaa0 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200c3e10 con 0x7f84200cd680 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8427870700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84200c40f0 con 0x7f84200b1080 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:19.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35302/0 (socket says 172.21.15.104:35302) 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 -- 172.21.15.104:0/3889701446 learned_addr learned my addr 172.21.15.104:0/3889701446 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8425e0d700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 0x7f84200c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 -- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84200b1080 msgr2=0x7f84200c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8424e0b700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84200b1080 0x7f84200c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84200b1080 0x7f84200c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 -- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 msgr2=0x7f84200c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 0x7f84200c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f842560c700 1 -- 172.21.15.104:0/3889701446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841000b6d0 con 0x7f84200aeaa0 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.046+0000 7f8425e0d700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 0x7f84200c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f842560c700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f841000ba10 tx=0x7f8410004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841001f070 con 0x7f84200aeaa0 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410003bd0 con 0x7f84200aeaa0 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410010470 con 0x7f84200aeaa0 2023-06-10T18:54:19.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84200c43d0 con 0x7f84200aeaa0 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.047+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84200c3820 con 0x7f84200aeaa0 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.048+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84200c48f0 con 0x7f84200aeaa0 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.048+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f841002c430 con 0x7f84200aeaa0 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.049+0000 7f8415ffb700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 0x7f840406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.049+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f841008a7b0 con 0x7f84200aeaa0 2023-06-10T18:54:19.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.049+0000 7f8424e0b700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 0x7f840406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:19.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.049+0000 7f8424e0b700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 0x7f840406d190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8420157300 tx=0x7f841c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:19.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.053+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f84200c48f0 con 0x7f84200aeaa0 2023-06-10T18:54:19.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.281+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84200c48f0 con 0x7f84200aeaa0 2023-06-10T18:54:19.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.282+0000 7f8415ffb700 1 -- 172.21.15.104:0/3889701446 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f84200c48f0 con 0x7f84200aeaa0 2023-06-10T18:54:19.288 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:19.288 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:19.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 msgr2=0x7f840406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 0x7f840406d190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8420157300 tx=0x7f841c009040 comp rx=0 tx=0).stop 2023-06-10T18:54:19.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 msgr2=0x7f84200c6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:19.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f841000ba10 tx=0x7f8410004750 comp rx=0 tx=0).stop 2023-06-10T18:54:19.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 shutdown_connections 2023-06-10T18:54:19.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f840406ace0 0x7f840406d190 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84200cd680 0x7f84200c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84200aeaa0 0x7f84200c6970 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 --2- 172.21.15.104:0/3889701446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84200b1080 0x7f84200c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 >> 172.21.15.104:0/3889701446 conn(0x7f842006b8a0 msgr2=0x7f8420069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 shutdown_connections 2023-06-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:19.284+0000 7f8427870700 1 -- 172.21.15.104:0/3889701446 wait complete. 2023-06-10T18:54:19.292 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:19.301 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:20.303 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.425+0000 7f920aaf8700 1 Processor -- start 2023-06-10T18:54:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.425+0000 7f920aaf8700 1 -- start start 2023-06-10T18:54:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f920aaf8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92040aeaa0 0x7f92040cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f920aaf8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 0x7f92040cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f920aaf8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040c9e60 con 0x7f92040ca2f0 2023-06-10T18:54:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f920aaf8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92040c9bc0 con 0x7f92040aeaa0 2023-06-10T18:54:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f920aaf8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92040aee70 con 0x7f92040b1080 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92040aeaa0 0x7f92040cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9209095700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f92040ca2f0 0x7f92040ca710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33142/0 (socket says 172.21.15.104:33142) 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92040aeaa0 0x7f92040cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35310/0 (socket says 172.21.15.104:35310) 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9209095700 1 -- 172.21.15.104:0/2259479429 learned_addr learned my addr 172.21.15.104:0/2259479429 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9203fff700 1 --2- 172.21.15.104:0/2259479429 >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 0x7f92040cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 -- 172.21.15.104:0/2259479429 >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 msgr2=0x7f92040cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 --2- 172.21.15.104:0/2259479429 >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 0x7f92040cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 -- 172.21.15.104:0/2259479429 >> v1:172.21.15.104:6789/0 conn(0x7f92040ca2f0 legacy=0x7f92040ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 -- 172.21.15.104:0/2259479429 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920406e540 con 0x7f92040aeaa0 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f92037fe700 1 -- 172.21.15.104:0/2259479429 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987085290 0 0) 0x7f92040c9e60 con 0x7f92040ca2f0 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9203fff700 1 --2- 172.21.15.104:0/2259479429 >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 0x7f92040cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f9208894700 1 --2- 172.21.15.104:0/2259479429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f92040cd140 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f9204070110 tx=0x7f91f8009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6f4921de9a49f2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.426+0000 7f92037fe700 1 -- 172.21.15.104:0/2259479429 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8005e30 con 0x7f92040aeaa0 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f92037fe700 1 -- 172.21.15.104:0/2259479429 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f8006030 con 0x7f92040aeaa0 2023-06-10T18:54:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f92037fe700 1 -- 172.21.15.104:0/2259479429 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f80063c0 con 0x7f92040aeaa0 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 -- 172.21.15.104:0/2259479429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 msgr2=0x7f92040cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2259479429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f92040cd140 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f9204070110 tx=0x7f91f8009b00 comp rx=0 tx=0).stop 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 -- 172.21.15.104:0/2259479429 shutdown_connections 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2259479429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f92040cd140 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2259479429 >> v2:172.21.15.114:3300/0 conn(0x7f92040b1080 0x7f92040cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 -- 172.21.15.104:0/2259479429 >> 172.21.15.104:0/2259479429 conn(0x7f920406b8a0 msgr2=0x7f92040cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:20.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 -- 172.21.15.104:0/2259479429 shutdown_connections 2023-06-10T18:54:20.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.427+0000 7f920aaf8700 1 -- 172.21.15.104:0/2259479429 wait complete. 2023-06-10T18:54:20.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.428+0000 7f920aaf8700 1 Processor -- start 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.428+0000 7f920aaf8700 1 -- start start 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040b1080 0x7f92040c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92040ca2f0 0x7f92040b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040c96b0 con 0x7f92040aeaa0 2023-06-10T18:54:20.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040c6e60 con 0x7f92040ca2f0 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f920aaf8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92040c8e50 con 0x7f92040b1080 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f9208894700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f9208894700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35316/0 (socket says 172.21.15.104:35316) 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.429+0000 7f9208894700 1 -- 172.21.15.104:0/2285653776 learned_addr learned my addr 172.21.15.104:0/2285653776 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 -- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040b1080 msgr2=0x7f92040c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9203fff700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040b1080 0x7f92040c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9209095700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92040ca2f0 0x7f92040b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040b1080 0x7f92040c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 -- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92040ca2f0 msgr2=0x7f92040b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92040ca2f0 0x7f92040b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 -- 172.21.15.104:0/2285653776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f8009430 con 0x7f92040aeaa0 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f9208894700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009770 tx=0x7f91f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8006930 con 0x7f92040aeaa0 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f8020070 con 0x7f92040aeaa0 2023-06-10T18:54:20.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92040c90f0 con 0x7f92040aeaa0 2023-06-10T18:54:20.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f80042a0 con 0x7f92040aeaa0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.430+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92040c6af0 con 0x7f92040aeaa0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.431+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92040c6cb0 con 0x7f92040aeaa0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.431+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f91f8022030 con 0x7f92040aeaa0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.432+0000 7f92017fa700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.432+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f91f808b010 con 0x7f92040aeaa0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.432+0000 7f9203fff700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.432+0000 7f9203fff700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f92040cb7d0 tx=0x7f91f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:20.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.436+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f91f8056050 con 0x7f92040aeaa0 2023-06-10T18:54:20.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.665+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92040c7140 con 0x7f92040aeaa0 2023-06-10T18:54:20.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.666+0000 7f92017fa700 1 -- 172.21.15.104:0/2285653776 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f91f8004400 con 0x7f92040aeaa0 2023-06-10T18:54:20.672 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:20.672 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:20.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 msgr2=0x7f91e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f92040cb7d0 tx=0x7f91f0008040 comp rx=0 tx=0).stop 2023-06-10T18:54:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 msgr2=0x7f920406a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009770 tx=0x7f91f8004750 comp rx=0 tx=0).stop 2023-06-10T18:54:20.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 shutdown_connections 2023-06-10T18:54:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92040ca2f0 0x7f92040b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040aeaa0 0x7f920406a740 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 --2- 172.21.15.104:0/2285653776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040b1080 0x7f92040c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:20.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 >> 172.21.15.104:0/2285653776 conn(0x7f920406b8a0 msgr2=0x7f92040af060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:20.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 shutdown_connections 2023-06-10T18:54:20.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:20.668+0000 7f920aaf8700 1 -- 172.21.15.104:0/2285653776 wait complete. 2023-06-10T18:54:20.678 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:20.686 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:21.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:21.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.815+0000 7f956fe61700 1 Processor -- start 2023-06-10T18:54:21.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.815+0000 7f956fe61700 1 -- start start 2023-06-10T18:54:21.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956fe61700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95680cd870 0x7f95680cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956fe61700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f956815e0f0 0x7f95680ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956fe61700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95680b3610 con 0x7f956815a410 2023-06-10T18:54:21.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956fe61700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95680d0000 con 0x7f95680cd870 2023-06-10T18:54:21.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956fe61700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95680b38d0 con 0x7f956815e0f0 2023-06-10T18:54:21.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95680cd870 0x7f95680cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95680cd870 0x7f95680cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35326/0 (socket says 172.21.15.104:35326) 2023-06-10T18:54:21.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 -- 172.21.15.104:0/2918707406 learned_addr learned my addr 172.21.15.104:0/2918707406 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:21.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956e3fe700 1 --2- 172.21.15.104:0/2918707406 >> v2:172.21.15.114:3300/0 conn(0x7f956815e0f0 0x7f95680ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 -- 172.21.15.104:0/2918707406 >> v2:172.21.15.114:3300/0 conn(0x7f956815e0f0 msgr2=0x7f95680ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:21.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 --2- 172.21.15.104:0/2918707406 >> v2:172.21.15.114:3300/0 conn(0x7f956815e0f0 0x7f95680ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 -- 172.21.15.104:0/2918707406 >> v1:172.21.15.114:6789/0 conn(0x7f956815a410 legacy=0x7f95680cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:21.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 -- 172.21.15.104:0/2918707406 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956815e6b0 con 0x7f95680cd870 2023-06-10T18:54:21.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956d3fc700 1 --2- 172.21.15.104:0/2918707406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95680cd870 0x7f95680cfac0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f95680b24d0 tx=0x7f9558009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bb6a97e7b9a050 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:21.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.816+0000 7f956cbfb700 1 -- 172.21.15.104:0/2918707406 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558011830 con 0x7f95680cd870 2023-06-10T18:54:21.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956cbfb700 1 -- 172.21.15.104:0/2918707406 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558011990 con 0x7f95680cd870 2023-06-10T18:54:21.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956cbfb700 1 -- 172.21.15.104:0/2918707406 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558011d20 con 0x7f95680cd870 2023-06-10T18:54:21.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 -- 172.21.15.104:0/2918707406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95680cd870 msgr2=0x7f95680cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:21.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 --2- 172.21.15.104:0/2918707406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95680cd870 0x7f95680cfac0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f95680b24d0 tx=0x7f9558009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 -- 172.21.15.104:0/2918707406 shutdown_connections 2023-06-10T18:54:21.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 --2- 172.21.15.104:0/2918707406 >> v2:172.21.15.114:3300/0 conn(0x7f956815e0f0 0x7f95680ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 --2- 172.21.15.104:0/2918707406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95680cd870 0x7f95680cfac0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 -- 172.21.15.104:0/2918707406 >> 172.21.15.104:0/2918707406 conn(0x7f9568069e70 msgr2=0x7f956806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:21.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 -- 172.21.15.104:0/2918707406 shutdown_connections 2023-06-10T18:54:21.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.817+0000 7f956fe61700 1 -- 172.21.15.104:0/2918707406 wait complete. 2023-06-10T18:54:21.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.818+0000 7f956fe61700 1 Processor -- start 2023-06-10T18:54:21.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.818+0000 7f956fe61700 1 -- start start 2023-06-10T18:54:21.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95680cd870 0x7f956806cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f956815a410 0x7f95680aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95680af500 con 0x7f956815e0f0 2023-06-10T18:54:21.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956815c820 con 0x7f95680cd870 2023-06-10T18:54:21.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956fe61700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956815d1f0 con 0x7f956815a410 2023-06-10T18:54:21.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956e3fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956e3fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35334/0 (socket says 172.21.15.104:35334) 2023-06-10T18:54:21.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956e3fe700 1 -- 172.21.15.104:0/2893962791 learned_addr learned my addr 172.21.15.104:0/2893962791 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:21.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956d3fc700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f956815a410 0x7f95680aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.819+0000 7f956dbfd700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95680cd870 0x7f956806cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 -- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f956815a410 msgr2=0x7f95680aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:21.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f956815a410 0x7f95680aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 -- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95680cd870 msgr2=0x7f956806cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:21.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95680cd870 0x7f956806cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:21.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 -- 172.21.15.104:0/2893962791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558009550 con 0x7f956815e0f0 2023-06-10T18:54:21.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956e3fe700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f9560009dd0 tx=0x7f956000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:21.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560009c40 con 0x7f956815e0f0 2023-06-10T18:54:21.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956000f040 con 0x7f956815e0f0 2023-06-10T18:54:21.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956815bc10 con 0x7f956815e0f0 2023-06-10T18:54:21.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956000ceb0 con 0x7f956815e0f0 2023-06-10T18:54:21.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.820+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956815c1d0 con 0x7f956815e0f0 2023-06-10T18:54:21.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.821+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9568074050 con 0x7f956815e0f0 2023-06-10T18:54:21.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.821+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9560014480 con 0x7f956815e0f0 2023-06-10T18:54:21.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.822+0000 7f955e7fc700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 0x7f954c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:21.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.822+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f9560089f70 con 0x7f956815e0f0 2023-06-10T18:54:21.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.822+0000 7f956dbfd700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 0x7f954c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:21.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.822+0000 7f956dbfd700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 0x7f954c06d190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9564007fd0 tx=0x7f9564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:21.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:21.826+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f95600bc310 con 0x7f956815e0f0 2023-06-10T18:54:22.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.052+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f956815c960 con 0x7f956815e0f0 2023-06-10T18:54:22.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.053+0000 7f955e7fc700 1 -- 172.21.15.104:0/2893962791 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f9560018070 con 0x7f956815e0f0 2023-06-10T18:54:22.059 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:22.059 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:22.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 msgr2=0x7f954c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 0x7f954c06d190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9564007fd0 tx=0x7f9564009040 comp rx=0 tx=0).stop 2023-06-10T18:54:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 msgr2=0x7f956815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f9560009dd0 tx=0x7f956000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:54:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 shutdown_connections 2023-06-10T18:54:22.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f954c06ace0 0x7f954c06d190 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95680cd870 0x7f956806cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f956815e0f0 0x7f956815ccb0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 --2- 172.21.15.104:0/2893962791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f956815a410 0x7f95680aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 >> 172.21.15.104:0/2893962791 conn(0x7f9568069e70 msgr2=0x7f95680b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 shutdown_connections 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:22.055+0000 7f956fe61700 1 -- 172.21.15.104:0/2893962791 wait complete. 2023-06-10T18:54:22.063 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:22.073 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:23.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:23.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.201+0000 7f8c2ea7e700 1 Processor -- start 2023-06-10T18:54:23.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.201+0000 7f8c2ea7e700 1 -- start start 2023-06-10T18:54:23.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c2815c730 0x7f8c2806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c280cc000 0x7f8c280cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c2806cd70 0x7f8c2806d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280aed80 con 0x7f8c2815c730 2023-06-10T18:54:23.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280c9c60 con 0x7f8c280cc000 2023-06-10T18:54:23.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2ea7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280cc9b0 con 0x7f8c2806cd70 2023-06-10T18:54:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c2815c730 0x7f8c2806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c2815c730 0x7f8c2806e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35338/0 (socket says 172.21.15.104:35338) 2023-06-10T18:54:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 -- 172.21.15.104:0/2610125866 learned_addr learned my addr 172.21.15.104:0/2610125866 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c27fff700 1 --2- 172.21.15.104:0/2610125866 >> v2:172.21.15.114:3300/0 conn(0x7f8c280cc000 0x7f8c280cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 -- 172.21.15.104:0/2610125866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c2806cd70 msgr2=0x7f8c2806d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:23.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2d01b700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c2806cd70 0x7f8c2806d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c2806cd70 0x7f8c2806d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 -- 172.21.15.104:0/2610125866 >> v2:172.21.15.114:3300/0 conn(0x7f8c280cc000 msgr2=0x7f8c280cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:23.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/2610125866 >> v2:172.21.15.114:3300/0 conn(0x7f8c280cc000 0x7f8c280cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 -- 172.21.15.104:0/2610125866 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2815cb00 con 0x7f8c2815c730 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.202+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2815c730 0x7f8c2806e790 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8c2806fd10 tx=0x7f8c1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2366565a19c5aba0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c277fe700 1 -- 172.21.15.104:0/2610125866 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c005e30 con 0x7f8c2815c730 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c277fe700 1 -- 172.21.15.104:0/2610125866 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c004030 con 0x7f8c2815c730 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c277fe700 1 -- 172.21.15.104:0/2610125866 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c0043e0 con 0x7f8c2815c730 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/2610125866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2815c730 msgr2=0x7f8c2806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2815c730 0x7f8c2806e790 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8c2806fd10 tx=0x7f8c1c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/2610125866 shutdown_connections 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2815c730 0x7f8c2806e790 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/2610125866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c2806cd70 0x7f8c2806d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/2610125866 >> v2:172.21.15.114:3300/0 conn(0x7f8c280cc000 0x7f8c280cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/2610125866 >> 172.21.15.104:0/2610125866 conn(0x7f8c2806b8a0 msgr2=0x7f8c280cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/2610125866 shutdown_connections 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.203+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/2610125866 wait complete. 2023-06-10T18:54:23.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.204+0000 7f8c2ea7e700 1 Processor -- start 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.204+0000 7f8c2ea7e700 1 -- start start 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 0x7f8c280af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 0x7f8c280b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280af990 con 0x7f8c2806cd70 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280afc70 con 0x7f8c2815c730 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2ea7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280b0780 con 0x7f8c280cc000 2023-06-10T18:54:23.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2c81a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2c81a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35344/0 (socket says 172.21.15.104:35344) 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.205+0000 7f8c2c81a700 1 -- 172.21.15.104:0/939114965 learned_addr learned my addr 172.21.15.104:0/939114965 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2d01b700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 0x7f8c280b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 -- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 msgr2=0x7f8c280af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c27fff700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 0x7f8c280af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 0x7f8c280af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 -- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 msgr2=0x7f8c280b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 0x7f8c280b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 -- 172.21.15.104:0/939114965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c009500 con 0x7f8c2806cd70 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2d01b700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 0x7f8c280b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c27fff700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 0x7f8c280af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2c81a700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009c10 tx=0x7f8c1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c011620 con 0x7f8c2806cd70 2023-06-10T18:54:23.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c011780 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c280b0a60 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c011b10 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.206+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28148200 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.207+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c28066f10 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.207+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c1c011c70 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.208+0000 7f8c257fa700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 0x7f8c0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.208+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8c1c08a520 con 0x7f8c2806cd70 2023-06-10T18:54:23.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.208+0000 7f8c27fff700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 0x7f8c0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:23.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.208+0000 7f8c27fff700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 0x7f8c0c06d190 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8c14006fd0 tx=0x7f8c14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:23.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.212+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c1c0554c0 con 0x7f8c2806cd70 2023-06-10T18:54:23.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.440+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c2815cb00 con 0x7f8c2806cd70 2023-06-10T18:54:23.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.440+0000 7f8c257fa700 1 -- 172.21.15.104:0/939114965 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f8c2815cb00 con 0x7f8c2806cd70 2023-06-10T18:54:23.446 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:23.447 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:23.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 msgr2=0x7f8c0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:23.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 0x7f8c0c06d190 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8c14006fd0 tx=0x7f8c14008040 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 msgr2=0x7f8c28147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009c10 tx=0x7f8c1c004750 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 shutdown_connections 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c0c06ace0 0x7f8c0c06d190 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2815c730 0x7f8c280b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c2806cd70 0x7f8c28147aa0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 --2- 172.21.15.104:0/939114965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280cc000 0x7f8c280af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:23.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.442+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 >> 172.21.15.104:0/939114965 conn(0x7f8c2806b8a0 msgr2=0x7f8c2815abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:23.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.443+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 shutdown_connections 2023-06-10T18:54:23.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:23.443+0000 7f8c2ea7e700 1 -- 172.21.15.104:0/939114965 wait complete. 2023-06-10T18:54:23.450 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:23.460 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:24.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:24.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.582+0000 7f504e883700 1 Processor -- start 2023-06-10T18:54:24.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.582+0000 7f504e883700 1 -- start start 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504e883700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50480cd850 0x7f504806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504e883700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f50480cb300 0x7f504806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504e883700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480c9e60 con 0x7f50480aeaa0 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504e883700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50480c9bc0 con 0x7f50480cb300 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504e883700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50480aee70 con 0x7f50480cd850 2023-06-10T18:54:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.583+0000 7f504ce20700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f50480cb300 0x7f504806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f50480cb300 0x7f504806d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35356/0 (socket says 172.21.15.104:35356) 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 -- 172.21.15.104:0/3091790416 learned_addr learned my addr 172.21.15.104:0/3091790416 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f50477fe700 1 --2- 172.21.15.104:0/3091790416 >> v2:172.21.15.114:3300/0 conn(0x7f50480cd850 0x7f504806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 -- 172.21.15.104:0/3091790416 >> v2:172.21.15.114:3300/0 conn(0x7f50480cd850 msgr2=0x7f504806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 --2- 172.21.15.104:0/3091790416 >> v2:172.21.15.114:3300/0 conn(0x7f50480cd850 0x7f504806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 -- 172.21.15.104:0/3091790416 >> v1:172.21.15.114:6789/0 conn(0x7f50480aeaa0 legacy=0x7f50480cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:24.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 -- 172.21.15.104:0/3091790416 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504806e540 con 0x7f50480cb300 2023-06-10T18:54:24.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f504ce20700 1 --2- 172.21.15.104:0/3091790416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f504806d290 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f5048070110 tx=0x7f504000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec075bb3fe116246 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:24.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f5046ffd700 1 -- 172.21.15.104:0/3091790416 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504000e070 con 0x7f50480cb300 2023-06-10T18:54:24.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f5046ffd700 1 -- 172.21.15.104:0/3091790416 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040003a20 con 0x7f50480cb300 2023-06-10T18:54:24.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.584+0000 7f5046ffd700 1 -- 172.21.15.104:0/3091790416 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040003dd0 con 0x7f50480cb300 2023-06-10T18:54:24.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 -- 172.21.15.104:0/3091790416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 msgr2=0x7f504806d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:24.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 --2- 172.21.15.104:0/3091790416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f504806d290 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f5048070110 tx=0x7f504000b330 comp rx=0 tx=0).stop 2023-06-10T18:54:24.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 -- 172.21.15.104:0/3091790416 shutdown_connections 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 --2- 172.21.15.104:0/3091790416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f504806d290 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 --2- 172.21.15.104:0/3091790416 >> v2:172.21.15.114:3300/0 conn(0x7f50480cd850 0x7f504806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 -- 172.21.15.104:0/3091790416 >> 172.21.15.104:0/3091790416 conn(0x7f504806b8a0 msgr2=0x7f50480cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 -- 172.21.15.104:0/3091790416 shutdown_connections 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.585+0000 7f504e883700 1 -- 172.21.15.104:0/3091790416 wait complete. 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.586+0000 7f504e883700 1 Processor -- start 2023-06-10T18:54:24.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.586+0000 7f504e883700 1 -- start start 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50480aeaa0 0x7f504814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50480cd850 0x7f50481580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504815c490 con 0x7f50480cb300 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504815b910 con 0x7f50480aeaa0 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f504e883700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504815bbf0 con 0x7f50480cd850 2023-06-10T18:54:24.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35360/0 (socket says 172.21.15.104:35360) 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 -- 172.21.15.104:0/1125725888 learned_addr learned my addr 172.21.15.104:0/1125725888 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 -- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50480cd850 msgr2=0x7f50481580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f5047fff700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50480aeaa0 0x7f504814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50480cd850 0x7f50481580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 -- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50480aeaa0 msgr2=0x7f504814eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50480aeaa0 0x7f504814eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 -- 172.21.15.104:0/1125725888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040009b30 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f50477fe700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f503000c7f0 tx=0x7f503000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.587+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5030011800 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.588+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5030011960 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.588+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5030011cf0 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.588+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504815bed0 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.588+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504815ac20 con 0x7f50480cb300 2023-06-10T18:54:24.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.588+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50480c66d0 con 0x7f50480cb300 2023-06-10T18:54:24.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.589+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5030018360 con 0x7f50480cb300 2023-06-10T18:54:24.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.589+0000 7f5044ff9700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 0x7f502806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:24.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.589+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f50300894d0 con 0x7f50480cb300 2023-06-10T18:54:24.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.590+0000 7f5047fff700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 0x7f502806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:24.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.590+0000 7f5047fff700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 0x7f502806d190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5038003e90 tx=0x7f5038009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:24.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.594+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f50480c66d0 con 0x7f50480cb300 2023-06-10T18:54:24.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.823+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50480c66d0 con 0x7f50480cb300 2023-06-10T18:54:24.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.824+0000 7f5044ff9700 1 -- 172.21.15.104:0/1125725888 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f50480c66d0 con 0x7f50480cb300 2023-06-10T18:54:24.830 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:24.830 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 msgr2=0x7f502806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 0x7f502806d190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5038003e90 tx=0x7f5038009040 comp rx=0 tx=0).stop 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 msgr2=0x7f5048157b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f503000c7f0 tx=0x7f503000cbb0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 shutdown_connections 2023-06-10T18:54:24.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f502806ace0 0x7f502806d190 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50480aeaa0 0x7f504814eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50480cd850 0x7f50481580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 --2- 172.21.15.104:0/1125725888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50480cb300 0x7f5048157b70 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 >> 172.21.15.104:0/1125725888 conn(0x7f504806b8a0 msgr2=0x7f50480cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 shutdown_connections 2023-06-10T18:54:24.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:24.826+0000 7f504e883700 1 -- 172.21.15.104:0/1125725888 wait complete. 2023-06-10T18:54:24.834 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:24.844 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:25.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:25.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.972+0000 7fb57b864700 1 Processor -- start 2023-06-10T18:54:25.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.972+0000 7fb57b864700 1 -- start start 2023-06-10T18:54:25.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb57b864700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb57415c2d0 0x7fb5740b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb57b864700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57406a840 con 0x7fb57406e680 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb57b864700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740aeab0 con 0x7fb5740ae310 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb57b864700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb574069f10 con 0x7fb57415c2d0 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579e01700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb57406e680 0x7fb5740ce210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33198/0 (socket says 172.21.15.104:33198) 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579600700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb57415c2d0 0x7fb5740b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:25.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579e01700 1 -- 172.21.15.104:0/554756110 learned_addr learned my addr 172.21.15.104:0/554756110 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb56bfff700 1 -- 172.21.15.104:0/554756110 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298211812 0 0) 0x7fb57406a840 con 0x7fb57406e680 2023-06-10T18:54:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb56bfff700 1 -- 172.21.15.104:0/554756110 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5600035b0 con 0x7fb57406e680 2023-06-10T18:54:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579600700 1 -- 172.21.15.104:0/554756110 >> v1:172.21.15.114:6789/0 conn(0x7fb5740ae310 legacy=0x7fb57406df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579600700 1 -- 172.21.15.104:0/554756110 >> v1:172.21.15.104:6789/0 conn(0x7fb57406e680 legacy=0x7fb5740ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:25.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.973+0000 7fb579600700 1 -- 172.21.15.104:0/554756110 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57406a1f0 con 0x7fb57415c2d0 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb579600700 1 --2- 172.21.15.104:0/554756110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57415c2d0 0x7fb5740b1ad0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb570009820 tx=0x7fb570009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2873e3727ee40364 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb56bfff700 1 -- 172.21.15.104:0/554756110 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570005e30 con 0x7fb57415c2d0 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb56bfff700 1 -- 172.21.15.104:0/554756110 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb570006030 con 0x7fb57415c2d0 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb56bfff700 1 -- 172.21.15.104:0/554756110 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700063c0 con 0x7fb57415c2d0 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb57b864700 1 -- 172.21.15.104:0/554756110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57415c2d0 msgr2=0x7fb5740b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.974+0000 7fb57b864700 1 --2- 172.21.15.104:0/554756110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57415c2d0 0x7fb5740b1ad0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb570009820 tx=0x7fb570009b30 comp rx=0 tx=0).stop 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.975+0000 7fb57b864700 1 -- 172.21.15.104:0/554756110 shutdown_connections 2023-06-10T18:54:25.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.975+0000 7fb57b864700 1 --2- 172.21.15.104:0/554756110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57415c2d0 0x7fb5740b1ad0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.975+0000 7fb57b864700 1 -- 172.21.15.104:0/554756110 >> 172.21.15.104:0/554756110 conn(0x7fb57406b8a0 msgr2=0x7fb5740cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.975+0000 7fb57b864700 1 -- 172.21.15.104:0/554756110 shutdown_connections 2023-06-10T18:54:25.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.975+0000 7fb57b864700 1 -- 172.21.15.104:0/554756110 wait complete. 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.976+0000 7fb57b864700 1 Processor -- start 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.976+0000 7fb57b864700 1 -- start start 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740ae310 0x7fb5740b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 0x7fb5740c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574159070 con 0x7fb57406e680 2023-06-10T18:54:25.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b4c80 con 0x7fb5740ae310 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b4f60 con 0x7fb57415c2d0 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35372/0 (socket says 172.21.15.104:35372) 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 -- 172.21.15.104:0/983948416 learned_addr learned my addr 172.21.15.104:0/983948416 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb578dff700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740ae310 0x7fb5740b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579e01700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 0x7fb5740c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 -- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 msgr2=0x7fb5740c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 0x7fb5740c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 -- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740ae310 msgr2=0x7fb5740b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740ae310 0x7fb5740b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 -- 172.21.15.104:0/983948416 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb570009430 con 0x7fb57406e680 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579e01700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 0x7fb5740c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb579600700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb570000c00 tx=0x7fb570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570004240 con 0x7fb57406e680 2023-06-10T18:54:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5700043a0 con 0x7fb57406e680 2023-06-10T18:54:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5740b5240 con 0x7fb57406e680 2023-06-10T18:54:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.977+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700036a0 con 0x7fb57406e680 2023-06-10T18:54:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.978+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5740af370 con 0x7fb57406e680 2023-06-10T18:54:25.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.978+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb574074050 con 0x7fb57406e680 2023-06-10T18:54:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.980+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb570022070 con 0x7fb57406e680 2023-06-10T18:54:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.980+0000 7fb569ffb700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 0x7fb55806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.980+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fb57008b400 con 0x7fb57406e680 2023-06-10T18:54:25.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.980+0000 7fb578dff700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 0x7fb55806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.981+0000 7fb578dff700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 0x7fb55806d190 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb5740c7cf0 tx=0x7fb564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:25.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:25.983+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb5700563a0 con 0x7fb57406e680 2023-06-10T18:54:26.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.208+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5740c6390 con 0x7fb57406e680 2023-06-10T18:54:26.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.209+0000 7fb569ffb700 1 -- 172.21.15.104:0/983948416 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fb5700599c0 con 0x7fb57406e680 2023-06-10T18:54:26.215 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:26.215 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 msgr2=0x7fb55806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:26.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 0x7fb55806d190 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb5740c7cf0 tx=0x7fb564009040 comp rx=0 tx=0).stop 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 msgr2=0x7fb574158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb570000c00 tx=0x7fb570004750 comp rx=0 tx=0).stop 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 shutdown_connections 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57415c2d0 0x7fb5740c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb57406e680 0x7fb574158b30 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55806ace0 0x7fb55806d190 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:26.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 --2- 172.21.15.104:0/983948416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740ae310 0x7fb5740b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 >> 172.21.15.104:0/983948416 conn(0x7fb57406b8a0 msgr2=0x7fb5740c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 shutdown_connections 2023-06-10T18:54:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:26.211+0000 7fb57b864700 1 -- 172.21.15.104:0/983948416 wait complete. 2023-06-10T18:54:26.220 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:26.229 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:27.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:27.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.357+0000 7fb566470700 1 Processor -- start 2023-06-10T18:54:27.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.357+0000 7fb566470700 1 -- start start 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.357+0000 7fb566470700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560070a20 0x7fb56006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.357+0000 7fb566470700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb560070df0 0x7fb5601587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb566470700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb560158ce0 0x7fb560159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb566470700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56006ae00 con 0x7fb560158ce0 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb566470700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb560069e70 con 0x7fb560070df0 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb566470700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5601595f0 con 0x7fb560070a20 2023-06-10T18:54:27.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb560158ce0 0x7fb560159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb560158ce0 0x7fb560159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35382/0 (socket says 172.21.15.104:35382) 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 -- 172.21.15.104:0/3935144078 learned_addr learned my addr 172.21.15.104:0/3935144078 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb55ffff700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560070a20 0x7fb56006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb55f7fe700 1 --2- 172.21.15.104:0/3935144078 >> v2:172.21.15.114:3300/0 conn(0x7fb560070df0 0x7fb5601587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 -- 172.21.15.104:0/3935144078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560070a20 msgr2=0x7fb56006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560070a20 0x7fb56006a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 -- 172.21.15.104:0/3935144078 >> v2:172.21.15.114:3300/0 conn(0x7fb560070df0 msgr2=0x7fb5601587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 --2- 172.21.15.104:0/3935144078 >> v2:172.21.15.114:3300/0 conn(0x7fb560070df0 0x7fb5601587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 -- 172.21.15.104:0/3935144078 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5601598d0 con 0x7fb560158ce0 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb564a0d700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560158ce0 0x7fb560159c40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fb560070670 tx=0x7fb55800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf7fa9744ec62d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb55effd700 1 -- 172.21.15.104:0/3935144078 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55800e070 con 0x7fb560158ce0 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb55effd700 1 -- 172.21.15.104:0/3935144078 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55800b9c0 con 0x7fb560158ce0 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.358+0000 7fb55effd700 1 -- 172.21.15.104:0/3935144078 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55800bd70 con 0x7fb560158ce0 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 -- 172.21.15.104:0/3935144078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560158ce0 msgr2=0x7fb560159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:27.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560158ce0 0x7fb560159c40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fb560070670 tx=0x7fb55800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 -- 172.21.15.104:0/3935144078 shutdown_connections 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560158ce0 0x7fb560159c40 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 --2- 172.21.15.104:0/3935144078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560070a20 0x7fb56006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 --2- 172.21.15.104:0/3935144078 >> v2:172.21.15.114:3300/0 conn(0x7fb560070df0 0x7fb5601587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 -- 172.21.15.104:0/3935144078 >> 172.21.15.104:0/3935144078 conn(0x7fb5600b1610 msgr2=0x7fb5600ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 -- 172.21.15.104:0/3935144078 shutdown_connections 2023-06-10T18:54:27.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.359+0000 7fb566470700 1 -- 172.21.15.104:0/3935144078 wait complete. 2023-06-10T18:54:27.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.360+0000 7fb566470700 1 Processor -- start 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.360+0000 7fb566470700 1 -- start start 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 0x7fb5600c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 0x7fb5600c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c1d50 con 0x7fb560070a20 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c1140 con 0x7fb560070df0 2023-06-10T18:54:27.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb566470700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c1420 con 0x7fb560158ce0 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35384/0 (socket says 172.21.15.104:35384) 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 -- 172.21.15.104:0/2926145028 learned_addr learned my addr 172.21.15.104:0/2926145028 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55f7fe700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 0x7fb5600c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb564a0d700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 0x7fb5600c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 -- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 msgr2=0x7fb5600c38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 0x7fb5600c38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 -- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 msgr2=0x7fb5600c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 0x7fb5600c0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55ffff700 1 -- 172.21.15.104:0/2926145028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558009b30 con 0x7fb560070a20 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb55f7fe700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 0x7fb5600c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.361+0000 7fb564a0d700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 0x7fb5600c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb55ffff700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d800 tx=0x7fb55000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500125f0 con 0x7fb560070a20 2023-06-10T18:54:27.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb550012750 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5600c1700 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550012b00 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.362+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5600bfad0 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.363+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb560071740 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.363+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb550012c60 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.363+0000 7fb55cff9700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 0x7fb54c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.364+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fb5500899e0 con 0x7fb560070a20 2023-06-10T18:54:27.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.364+0000 7fb55f7fe700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 0x7fb54c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.364+0000 7fb55f7fe700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 0x7fb54c06d190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb5600cb560 tx=0x7fb554008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:27.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.368+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb5500548f0 con 0x7fb560070a20 2023-06-10T18:54:27.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.596+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb560074050 con 0x7fb560070a20 2023-06-10T18:54:27.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.596+0000 7fb55cff9700 1 -- 172.21.15.104:0/2926145028 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fb550057f10 con 0x7fb560070a20 2023-06-10T18:54:27.602 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:27.602 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:27.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 msgr2=0x7fb54c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:27.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 0x7fb54c06d190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb5600cb560 tx=0x7fb554008040 comp rx=0 tx=0).stop 2023-06-10T18:54:27.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 msgr2=0x7fb5600bf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:27.604 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d800 tx=0x7fb55000dbc0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 shutdown_connections 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb560158ce0 0x7fb5600c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb560070a20 0x7fb5600bf590 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb54c06ace0 0x7fb54c06d190 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 --2- 172.21.15.104:0/2926145028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb560070df0 0x7fb5600c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 >> 172.21.15.104:0/2926145028 conn(0x7fb5600b1610 msgr2=0x7fb5600ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:27.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 shutdown_connections 2023-06-10T18:54:27.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:27.598+0000 7fb566470700 1 -- 172.21.15.104:0/2926145028 wait complete. 2023-06-10T18:54:27.606 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:27.616 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:28.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.746+0000 7f820d35c700 1 Processor -- start 2023-06-10T18:54:28.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.746+0000 7f820d35c700 1 -- start start 2023-06-10T18:54:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f820d35c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82080d0c00 0x7f820806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f820d35c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f820806b820 0x7f82080cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f820d35c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820806bfd0 con 0x7f82080b1e30 2023-06-10T18:54:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f820d35c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820806c2b0 con 0x7f82080d0c00 2023-06-10T18:54:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f820d35c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82080ccf70 con 0x7f820806b820 2023-06-10T18:54:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82080d0c00 0x7f820806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f82080d0c00 0x7f820806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35394/0 (socket says 172.21.15.104:35394) 2023-06-10T18:54:28.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 -- 172.21.15.104:0/1797813485 learned_addr learned my addr 172.21.15.104:0/1797813485 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82077fe700 1 --2- 172.21.15.104:0/1797813485 >> v2:172.21.15.114:3300/0 conn(0x7f820806b820 0x7f82080cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 -- 172.21.15.104:0/1797813485 >> v2:172.21.15.114:3300/0 conn(0x7f820806b820 msgr2=0x7f82080cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 --2- 172.21.15.104:0/1797813485 >> v2:172.21.15.114:3300/0 conn(0x7f820806b820 0x7f82080cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 -- 172.21.15.104:0/1797813485 >> v1:172.21.15.114:6789/0 conn(0x7f82080b1e30 legacy=0x7f82080d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.747+0000 7f82067fc700 1 -- 172.21.15.104:0/1797813485 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080cd250 con 0x7f82080d0c00 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f82067fc700 1 --2- 172.21.15.104:0/1797813485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f820806b2e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f820808b0f0 tx=0x7f81f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e651425e2879567 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f8205ffb700 1 -- 172.21.15.104:0/1797813485 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8004230 con 0x7f82080d0c00 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f8205ffb700 1 -- 172.21.15.104:0/1797813485 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f8004390 con 0x7f82080d0c00 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f8205ffb700 1 -- 172.21.15.104:0/1797813485 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f80036a0 con 0x7f82080d0c00 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 -- 172.21.15.104:0/1797813485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 msgr2=0x7f820806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:28.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 --2- 172.21.15.104:0/1797813485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f820806b2e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f820808b0f0 tx=0x7f81f8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 -- 172.21.15.104:0/1797813485 shutdown_connections 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 --2- 172.21.15.104:0/1797813485 >> v2:172.21.15.114:3300/0 conn(0x7f820806b820 0x7f82080cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 --2- 172.21.15.104:0/1797813485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f820806b2e0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 -- 172.21.15.104:0/1797813485 >> 172.21.15.104:0/1797813485 conn(0x7f8208071ec0 msgr2=0x7f820809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 -- 172.21.15.104:0/1797813485 shutdown_connections 2023-06-10T18:54:28.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.748+0000 7f820d35c700 1 -- 172.21.15.104:0/1797813485 wait complete. 2023-06-10T18:54:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 Processor -- start 2023-06-10T18:54:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 -- start start 2023-06-10T18:54:28.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f820806b820 0x7f82080cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82080b1e30 0x7f82080b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82080b60f0 con 0x7f82080d0c00 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820815b2e0 con 0x7f820806b820 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.750+0000 7f820d35c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82080b85b0 con 0x7f82080b1e30 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35402/0 (socket says 172.21.15.104:35402) 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 -- 172.21.15.104:0/209896126 learned_addr learned my addr 172.21.15.104:0/209896126 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f8206ffd700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f820806b820 0x7f82080cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82067fc700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82080b1e30 0x7f82080b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 -- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82080b1e30 msgr2=0x7f82080b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82080b1e30 0x7f82080b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 -- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f820806b820 msgr2=0x7f82080cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:28.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f820806b820 0x7f82080cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 -- 172.21.15.104:0/209896126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f8009550 con 0x7f82080d0c00 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f82077fe700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f820000cd70 tx=0x7f820000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200003a20 con 0x7f82080d0c00 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200003b80 con 0x7f82080d0c00 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82080b5750 con 0x7f82080d0c00 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200004020 con 0x7f82080d0c00 2023-06-10T18:54:28.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.751+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82080b5d10 con 0x7f82080d0c00 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.752+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82080d1860 con 0x7f82080d0c00 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.753+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8200004180 con 0x7f82080d0c00 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.753+0000 7f81ef7fe700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 0x7f81f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.753+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8200012070 con 0x7f82080d0c00 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.753+0000 7f8206ffd700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 0x7f81f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:28.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.754+0000 7f8206ffd700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 0x7f81f406d190 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f82081e5f30 tx=0x7f81fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:28.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.758+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8200054c90 con 0x7f82080d0c00 2023-06-10T18:54:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.988+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82080b6230 con 0x7f82080d0c00 2023-06-10T18:54:28.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.988+0000 7f81ef7fe700 1 -- 172.21.15.104:0/209896126 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f82080b6230 con 0x7f82080d0c00 2023-06-10T18:54:28.994 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:28.995 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 msgr2=0x7f81f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:28.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 0x7f81f406d190 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f82081e5f30 tx=0x7f81fc009040 comp rx=0 tx=0).stop 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 msgr2=0x7f82080b7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f820000cd70 tx=0x7f820000a570 comp rx=0 tx=0).stop 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 shutdown_connections 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f81f406ace0 0x7f81f406d190 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f820806b820 0x7f82080cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82080d0c00 0x7f82080b7f10 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 --2- 172.21.15.104:0/209896126 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82080b1e30 0x7f82080b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.990+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 >> 172.21.15.104:0/209896126 conn(0x7f8208071ec0 msgr2=0x7f82080d2430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.991+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 shutdown_connections 2023-06-10T18:54:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:28.991+0000 7f820d35c700 1 -- 172.21.15.104:0/209896126 wait complete. 2023-06-10T18:54:28.998 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:29.008 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:30.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:30.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 Processor -- start 2023-06-10T18:54:30.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 -- start start 2023-06-10T18:54:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28400b4ff0 0x7f28400ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28400ce890 0x7f284015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400b6070 con 0x7f284015a420 2023-06-10T18:54:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28400b1d70 con 0x7f28400ce890 2023-06-10T18:54:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.129+0000 7f28450da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28400cecb0 con 0x7f28400b4ff0 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28400ce890 0x7f284015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f284015a420 0x7f28400b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33234/0 (socket says 172.21.15.104:33234) 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28400ce890 0x7f284015a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35410/0 (socket says 172.21.15.104:35410) 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 -- 172.21.15.104:0/1496508970 learned_addr learned my addr 172.21.15.104:0/1496508970 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283e59c700 1 --2- 172.21.15.104:0/1496508970 >> v2:172.21.15.114:3300/0 conn(0x7f28400b4ff0 0x7f28400ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 -- 172.21.15.104:0/1496508970 >> v2:172.21.15.114:3300/0 conn(0x7f28400b4ff0 msgr2=0x7f28400ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 --2- 172.21.15.104:0/1496508970 >> v2:172.21.15.114:3300/0 conn(0x7f28400b4ff0 0x7f28400ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 -- 172.21.15.104:0/1496508970 >> v1:172.21.15.104:6789/0 conn(0x7f284015a420 legacy=0x7f28400b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 -- 172.21.15.104:0/1496508970 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400b5460 con 0x7f28400ce890 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283f59e700 1 --2- 172.21.15.104:0/1496508970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400ce890 0x7f284015a7f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f2840066590 tx=0x7f283800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2229638b8de7cf2a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283dd9b700 1 -- 172.21.15.104:0/1496508970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283800e070 con 0x7f28400ce890 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283dd9b700 1 -- 172.21.15.104:0/1496508970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283800b9c0 con 0x7f28400ce890 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.130+0000 7f283dd9b700 1 -- 172.21.15.104:0/1496508970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283800bd70 con 0x7f28400ce890 2023-06-10T18:54:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 -- 172.21.15.104:0/1496508970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400ce890 msgr2=0x7f284015a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 --2- 172.21.15.104:0/1496508970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400ce890 0x7f284015a7f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f2840066590 tx=0x7f283800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 -- 172.21.15.104:0/1496508970 shutdown_connections 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 --2- 172.21.15.104:0/1496508970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400ce890 0x7f284015a7f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 --2- 172.21.15.104:0/1496508970 >> v2:172.21.15.114:3300/0 conn(0x7f28400b4ff0 0x7f28400ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 -- 172.21.15.104:0/1496508970 >> 172.21.15.104:0/1496508970 conn(0x7f2840089fd0 msgr2=0x7f28400d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 -- 172.21.15.104:0/1496508970 shutdown_connections 2023-06-10T18:54:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.131+0000 7f28450da700 1 -- 172.21.15.104:0/1496508970 wait complete. 2023-06-10T18:54:30.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.132+0000 7f28450da700 1 Processor -- start 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.132+0000 7f28450da700 1 -- start start 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 0x7f28401f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f284015a420 0x7f284006af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28401f2960 con 0x7f28400b4ff0 2023-06-10T18:54:30.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284006beb0 con 0x7f28400ce890 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f28450da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284006b8f0 con 0x7f284015a420 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35414/0 (socket says 172.21.15.104:35414) 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 -- 172.21.15.104:0/4007489690 learned_addr learned my addr 172.21.15.104:0/4007489690 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283e59c700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 0x7f28401f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283f59e700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f284015a420 0x7f284006af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 -- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f284015a420 msgr2=0x7f284006af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f284015a420 0x7f284006af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 -- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 msgr2=0x7f28401f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 0x7f28401f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283ed9d700 1 -- 172.21.15.104:0/4007489690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838009b30 con 0x7f28400b4ff0 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.133+0000 7f283e59c700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 0x7f28401f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f283ed9d700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f283000c520 tx=0x7f283000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830005cb0 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2830005e10 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284006bbd0 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300072d0 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.134+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28401efd80 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.135+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28401f23e0 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.135+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2830007430 con 0x7f28400b4ff0 2023-06-10T18:54:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.135+0000 7f28277fe700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 0x7f282c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.136+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f2830088a90 con 0x7f28400b4ff0 2023-06-10T18:54:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.136+0000 7f283e59c700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 0x7f282c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.136+0000 7f283e59c700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 0x7f282c06d190 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2834009820 tx=0x7f2834008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.140+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f28401f23e0 con 0x7f28400b4ff0 2023-06-10T18:54:30.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.367+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28401f23e0 con 0x7f28400b4ff0 2023-06-10T18:54:30.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.367+0000 7f28277fe700 1 -- 172.21.15.104:0/4007489690 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f28401f23e0 con 0x7f28400b4ff0 2023-06-10T18:54:30.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:30.373 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 msgr2=0x7f282c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 0x7f282c06d190 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2834009820 tx=0x7f2834008040 comp rx=0 tx=0).stop 2023-06-10T18:54:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 msgr2=0x7f28401ef840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:30.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f283000c520 tx=0x7f283000c8e0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 shutdown_connections 2023-06-10T18:54:30.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f284015a420 0x7f284006af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28400b4ff0 0x7f28401ef840 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282c06ace0 0x7f282c06d190 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 --2- 172.21.15.104:0/4007489690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28400ce890 0x7f28401f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:30.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 >> 172.21.15.104:0/4007489690 conn(0x7f2840089fd0 msgr2=0x7f284006e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:30.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 shutdown_connections 2023-06-10T18:54:30.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:30.369+0000 7f28450da700 1 -- 172.21.15.104:0/4007489690 wait complete. 2023-06-10T18:54:30.378 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:30.387 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:31.389 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.516+0000 7f4110a65700 1 Processor -- start 2023-06-10T18:54:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.516+0000 7f4110a65700 1 -- start start 2023-06-10T18:54:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.517+0000 7f4110a65700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f410c06e4b0 0x7f410c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.517+0000 7f4110a65700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f410c069e70 0x7f410c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.517+0000 7f4110a65700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c06d290 con 0x7f410c06e0e0 2023-06-10T18:54:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.517+0000 7f4110a65700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410c071640 con 0x7f410c06e4b0 2023-06-10T18:54:31.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.517+0000 7f4110a65700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410c0aef00 con 0x7f410c069e70 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f410c06e4b0 0x7f410c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f410c06e0e0 0x7f410c0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33248/0 (socket says 172.21.15.104:33248) 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410a59c700 1 -- 172.21.15.104:0/4103323493 learned_addr learned my addr 172.21.15.104:0/4103323493 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410ad9d700 1 --2- 172.21.15.104:0/4103323493 >> v2:172.21.15.114:3300/0 conn(0x7f410c069e70 0x7f410c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410959a700 1 -- 172.21.15.104:0/4103323493 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965115230 0 0) 0x7f410c06d290 con 0x7f410c06e0e0 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410959a700 1 -- 172.21.15.104:0/4103323493 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f40035b0 con 0x7f410c06e0e0 2023-06-10T18:54:31.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 -- 172.21.15.104:0/4103323493 >> v2:172.21.15.114:3300/0 conn(0x7f410c069e70 msgr2=0x7f410c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 --2- 172.21.15.104:0/4103323493 >> v2:172.21.15.114:3300/0 conn(0x7f410c069e70 0x7f410c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 -- 172.21.15.104:0/4103323493 >> v1:172.21.15.104:6789/0 conn(0x7f410c06e0e0 legacy=0x7f410c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 -- 172.21.15.104:0/4103323493 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0ca2d0 con 0x7f410c06e4b0 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f4109d9b700 1 --2- 172.21.15.104:0/4103323493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e4b0 0x7f410c06cd50 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f410c071210 tx=0x7f40fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb90e0735ac42c2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410959a700 1 -- 172.21.15.104:0/4103323493 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc011830 con 0x7f410c06e4b0 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410959a700 1 -- 172.21.15.104:0/4103323493 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc011990 con 0x7f410c06e4b0 2023-06-10T18:54:31.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.518+0000 7f410959a700 1 -- 172.21.15.104:0/4103323493 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc011d00 con 0x7f410c06e4b0 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 -- 172.21.15.104:0/4103323493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e4b0 msgr2=0x7f410c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 --2- 172.21.15.104:0/4103323493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e4b0 0x7f410c06cd50 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f410c071210 tx=0x7f40fc009b20 comp rx=0 tx=0).stop 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 -- 172.21.15.104:0/4103323493 shutdown_connections 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 --2- 172.21.15.104:0/4103323493 >> v2:172.21.15.114:3300/0 conn(0x7f410c069e70 0x7f410c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 --2- 172.21.15.104:0/4103323493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e4b0 0x7f410c06cd50 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 -- 172.21.15.104:0/4103323493 >> 172.21.15.104:0/4103323493 conn(0x7f410c06b8a0 msgr2=0x7f410c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 -- 172.21.15.104:0/4103323493 shutdown_connections 2023-06-10T18:54:31.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.519+0000 7f4110a65700 1 -- 172.21.15.104:0/4103323493 wait complete. 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.521+0000 7f4110a65700 1 Processor -- start 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.521+0000 7f4110a65700 1 -- start start 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f410c069e70 0x7f410c1489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f410c06e4b0 0x7f410c0b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c157170 con 0x7f410c06e0e0 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0af040 con 0x7f410c069e70 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0b3280 con 0x7f410c06e4b0 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35428/0 (socket says 172.21.15.104:35428) 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 -- 172.21.15.104:0/4195907295 learned_addr learned my addr 172.21.15.104:0/4195907295 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f410a59c700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f410c069e70 0x7f410c1489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f410ad9d700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f410c06e4b0 0x7f410c0b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 -- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f410c06e4b0 msgr2=0x7f410c0b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f410c06e4b0 0x7f410c0b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 -- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f410c069e70 msgr2=0x7f410c1489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f410c069e70 0x7f410c1489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 -- 172.21.15.104:0/4195907295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc0094a0 con 0x7f410c06e0e0 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4109d9b700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0090b0 tx=0x7f40fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0045d0 con 0x7f410c06e0e0 2023-06-10T18:54:31.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc0036a0 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c0b3560 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc003a10 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.522+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c0af290 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.523+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c0af480 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.524+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f40fc003b70 con 0x7f410c06e0e0 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.524+0000 7f40faffd700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 0x7f40ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:31.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.524+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f40fc08a990 con 0x7f410c06e0e0 2023-06-10T18:54:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.525+0000 7f410a59c700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 0x7f40ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.525+0000 7f410a59c700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 0x7f40ec06d190 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4100007fd0 tx=0x7f4100009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:31.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.529+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40fc055930 con 0x7f410c06e0e0 2023-06-10T18:54:31.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.757+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f410c074050 con 0x7f410c06e0e0 2023-06-10T18:54:31.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.757+0000 7f40faffd700 1 -- 172.21.15.104:0/4195907295 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f40fc058f50 con 0x7f410c06e0e0 2023-06-10T18:54:31.764 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:31.764 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:31.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 msgr2=0x7f40ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 0x7f40ec06d190 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4100007fd0 tx=0x7f4100009040 comp rx=0 tx=0).stop 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 msgr2=0x7f410c0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0090b0 tx=0x7f40fc004750 comp rx=0 tx=0).stop 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 shutdown_connections 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40ec06ace0 0x7f40ec06d190 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f410c069e70 0x7f410c1489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f410c06e4b0 0x7f410c0b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 --2- 172.21.15.104:0/4195907295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f410c06e0e0 0x7f410c0b15e0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 >> 172.21.15.104:0/4195907295 conn(0x7f410c06b8a0 msgr2=0x7f410c0c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 shutdown_connections 2023-06-10T18:54:31.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:31.759+0000 7f4110a65700 1 -- 172.21.15.104:0/4195907295 wait complete. 2023-06-10T18:54:31.768 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:31.778 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:32.779 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.906+0000 7f4bc2d93700 1 Processor -- start 2023-06-10T18:54:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.906+0000 7f4bc2d93700 1 -- start start 2023-06-10T18:54:32.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc2d93700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4bbc06e4b0 0x7f4bbc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc2d93700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc06d290 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc2d93700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0ca2d0 con 0x7f4bbc069e70 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc2d93700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc071640 con 0x7f4bbc06e4b0 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc0b2f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4bbc06e0e0 0x7f4bbc0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33262/0 (socket says 172.21.15.104:33262) 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bc0b2f700 1 -- 172.21.15.104:0/3394493158 learned_addr learned my addr 172.21.15.104:0/3394493158 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/3394493158 >> v2:172.21.15.114:3300/0 conn(0x7f4bbc06e4b0 0x7f4bbc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459646272 0 0) 0x7f4bbc06d290 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bac0035b0 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580847033 0 0) 0x7f4bac0035b0 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 >> v2:172.21.15.114:3300/0 conn(0x7f4bbc06e4b0 msgr2=0x7f4bbc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 --2- 172.21.15.104:0/3394493158 >> v2:172.21.15.114:3300/0 conn(0x7f4bbc06e4b0 0x7f4bbc06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 >> v1:172.21.15.114:6789/0 conn(0x7f4bbc069e70 legacy=0x7f4bbc0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc06d4c0 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4bb00017f0 con 0x7f4bbc06e0e0 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.907+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 >> v1:172.21.15.104:6789/0 conn(0x7f4bbc06e0e0 legacy=0x7f4bbc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac003a00 0x7f4bac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 0x7f4bac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bac005320 0x7f4bac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc071640 con 0x7f4bac005320 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac00c250 con 0x7f4bac004850 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac00c410 con 0x7f4bac003a00 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bc1330700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bac005320 0x7f4bac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 0x7f4bac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bc0b2f700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac003a00 0x7f4bac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 -- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac003a00 msgr2=0x7f4bac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac003a00 0x7f4bac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 -- 172.21.15.104:0/3394493158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bac005320 msgr2=0x7f4bac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bac005320 0x7f4bac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 -- 172.21.15.104:0/3394493158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac0035b0 con 0x7f4bac004850 2023-06-10T18:54:32.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bc1330700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bac005320 0x7f4bac00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bc1330700 1 -- 172.21.15.104:0/3394493158 reap_dead start 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 0x7f4bac004cc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009b60 tx=0x7f4bac005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=653e637c30f2e5f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800e6c0 con 0x7f4bac004850 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba800e820 con 0x7f4bac004850 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.908+0000 7f4bbb7fe700 1 -- 172.21.15.104:0/3394493158 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800eb90 con 0x7f4bac004850 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 -- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 msgr2=0x7f4bac004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 0x7f4bac004cc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009b60 tx=0x7f4bac005880 comp rx=0 tx=0).stop 2023-06-10T18:54:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 -- 172.21.15.104:0/3394493158 shutdown_connections 2023-06-10T18:54:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/3394493158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bac004850 0x7f4bac004cc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 -- 172.21.15.104:0/3394493158 >> 172.21.15.104:0/3394493158 conn(0x7f4bbc06b8a0 msgr2=0x7f4bbc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 -- 172.21.15.104:0/3394493158 shutdown_connections 2023-06-10T18:54:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.909+0000 7f4bc2d93700 1 -- 172.21.15.104:0/3394493158 wait complete. 2023-06-10T18:54:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.910+0000 7f4bc2d93700 1 Processor -- start 2023-06-10T18:54:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.910+0000 7f4bc2d93700 1 -- start start 2023-06-10T18:54:32.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac004850 0x7f4bbc0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bbc1450f0 0x7f4bbc1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc145590 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0b3b60 con 0x7f4bbc1450f0 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc2d93700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0b3e40 con 0x7f4bac004850 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bbbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35448/0 (socket says 172.21.15.104:35448) 2023-06-10T18:54:32.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bbbfff700 1 -- 172.21.15.104:0/2158964102 learned_addr learned my addr 172.21.15.104:0/2158964102 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc0b2f700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac004850 0x7f4bbc0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.911+0000 7f4bc1330700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bbc1450f0 0x7f4bbc1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 -- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac004850 msgr2=0x7f4bbc0b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac004850 0x7f4bbc0b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 -- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bbc1450f0 msgr2=0x7f4bbc1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bbc1450f0 0x7f4bbc1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:32.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 -- 172.21.15.104:0/2158964102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0b4120 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bbbfff700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c3e0 tx=0x7f4ba80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:32.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800e6c0 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba8003ad0 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc0b4400 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8003e80 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.912+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc0b4c00 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.913+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc0b4a50 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.913+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4ba8013470 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.914+0000 7f4bb97fa700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 0x7f4ba006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:32.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.914+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4ba808af20 con 0x7f4bbc0c5d20 2023-06-10T18:54:32.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.914+0000 7f4bc0b2f700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 0x7f4ba006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:32.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.914+0000 7f4bc0b2f700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 0x7f4ba006d190 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0001360 tx=0x7f4bb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:32.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:32.918+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4ba8055fe0 con 0x7f4bbc0c5d20 2023-06-10T18:54:33.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.148+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bbc074050 con 0x7f4bbc0c5d20 2023-06-10T18:54:33.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.148+0000 7f4bb97fa700 1 -- 172.21.15.104:0/2158964102 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f4ba8017090 con 0x7f4bbc0c5d20 2023-06-10T18:54:33.154 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:33.155 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:33.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 msgr2=0x7f4ba006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:33.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 0x7f4ba006d190 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0001360 tx=0x7f4bb0009040 comp rx=0 tx=0).stop 2023-06-10T18:54:33.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 msgr2=0x7f4bbc144bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:33.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c3e0 tx=0x7f4ba80047c0 comp rx=0 tx=0).stop 2023-06-10T18:54:33.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 shutdown_connections 2023-06-10T18:54:33.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4ba006ace0 0x7f4ba006d190 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:33.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bbc1450f0 0x7f4bbc1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:33.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bac004850 0x7f4bbc0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:33.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 --2- 172.21.15.104:0/2158964102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4bbc0c5d20 0x7f4bbc144bb0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:33.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 >> 172.21.15.104:0/2158964102 conn(0x7f4bbc06b8a0 msgr2=0x7f4bbc072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:33.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 shutdown_connections 2023-06-10T18:54:33.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:33.150+0000 7f4bc2d93700 1 -- 172.21.15.104:0/2158964102 wait complete. 2023-06-10T18:54:33.160 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:33.168 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:34.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:34.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.291+0000 7f703d878700 1 Processor -- start 2023-06-10T18:54:34.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.291+0000 7f703d878700 1 -- start start 2023-06-10T18:54:34.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.291+0000 7f703d878700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f703815c040 0x7f70381606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:34.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.291+0000 7f703d878700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380ce2f0 con 0x7f703806ce20 2023-06-10T18:54:34.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.291+0000 7f703d878700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7038160c20 con 0x7f703806c9b0 2023-06-10T18:54:34.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f703d878700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380cfd30 con 0x7f703815c040 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7036ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f703815c040 0x7f70381606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f70377fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f703806ce20 0x7f70380adff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33286/0 (socket says 172.21.15.104:33286) 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f70377fe700 1 -- 172.21.15.104:0/24034224 learned_addr learned my addr 172.21.15.104:0/24034224 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7036ffd700 1 -- 172.21.15.104:0/24034224 >> v1:172.21.15.114:6789/0 conn(0x7f703806c9b0 legacy=0x7f70380ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7036ffd700 1 -- 172.21.15.104:0/24034224 >> v1:172.21.15.104:6789/0 conn(0x7f703806ce20 legacy=0x7f70380adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7035ffb700 1 -- 172.21.15.104:0/24034224 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223564071 0 0) 0x7f70380ce2f0 con 0x7f703806ce20 2023-06-10T18:54:34.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7036ffd700 1 -- 172.21.15.104:0/24034224 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380cfa50 con 0x7f703815c040 2023-06-10T18:54:34.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7036ffd700 1 --2- 172.21.15.104:0/24034224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70381606e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7028009840 tx=0x7f7028009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b22825ee0bba07 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:34.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7035ffb700 1 -- 172.21.15.104:0/24034224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028004230 con 0x7f703815c040 2023-06-10T18:54:34.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7035ffb700 1 -- 172.21.15.104:0/24034224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7028004390 con 0x7f703815c040 2023-06-10T18:54:34.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.292+0000 7f7035ffb700 1 -- 172.21.15.104:0/24034224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280036a0 con 0x7f703815c040 2023-06-10T18:54:34.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 -- 172.21.15.104:0/24034224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 msgr2=0x7f70381606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 --2- 172.21.15.104:0/24034224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70381606e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7028009840 tx=0x7f7028009b50 comp rx=0 tx=0).stop 2023-06-10T18:54:34.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 -- 172.21.15.104:0/24034224 shutdown_connections 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 --2- 172.21.15.104:0/24034224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70381606e0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 -- 172.21.15.104:0/24034224 >> 172.21.15.104:0/24034224 conn(0x7f70380b1570 msgr2=0x7f70380ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 -- 172.21.15.104:0/24034224 shutdown_connections 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.293+0000 7f703d878700 1 -- 172.21.15.104:0/24034224 wait complete. 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.294+0000 7f703d878700 1 Processor -- start 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.294+0000 7f703d878700 1 -- start start 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f703806c9b0 0x7f7038155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:34.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f703806ce20 0x7f703815ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038155940 con 0x7f703815c040 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703815d390 con 0x7f703806ce20 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f703d878700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703815d670 con 0x7f703806c9b0 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35464/0 (socket says 172.21.15.104:35464) 2023-06-10T18:54:34.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 -- 172.21.15.104:0/3798425493 learned_addr learned my addr 172.21.15.104:0/3798425493 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f7036ffd700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f703806c9b0 0x7f7038155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70367fc700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f703806ce20 0x7f703815ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 -- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f703806c9b0 msgr2=0x7f7038155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f703806c9b0 0x7f7038155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 -- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f703806ce20 msgr2=0x7f703815ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f703806ce20 0x7f703815ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 -- 172.21.15.104:0/3798425493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70280094a0 con 0x7f703815c040 2023-06-10T18:54:34.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.295+0000 7f70377fe700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f703000cca0 tx=0x7f703000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.296+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703000ab70 con 0x7f703815c040 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.296+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000acd0 con 0x7f703815c040 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.296+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703815d950 con 0x7f703815c040 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.296+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030007310 con 0x7f703815c040 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.296+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703815f1b0 con 0x7f703815c040 2023-06-10T18:54:34.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.297+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703815db90 con 0x7f703815c040 2023-06-10T18:54:34.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.297+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7030007470 con 0x7f703815c040 2023-06-10T18:54:34.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.298+0000 7f701ffff700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 0x7f702406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:34.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.298+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f70300893d0 con 0x7f703815c040 2023-06-10T18:54:34.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.298+0000 7f7036ffd700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 0x7f702406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:34.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.298+0000 7f7036ffd700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 0x7f702406d190 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f70280090b0 tx=0x7f7028008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:34.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.302+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f703815db90 con 0x7f703815c040 2023-06-10T18:54:34.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.530+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70381571c0 con 0x7f703815c040 2023-06-10T18:54:34.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.531+0000 7f701ffff700 1 -- 172.21.15.104:0/3798425493 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f7030054370 con 0x7f703815c040 2023-06-10T18:54:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:34.537 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 msgr2=0x7f702406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 0x7f702406d190 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f70280090b0 tx=0x7f7028008030 comp rx=0 tx=0).stop 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 msgr2=0x7f70380b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f703000cca0 tx=0x7f703000a060 comp rx=0 tx=0).stop 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 shutdown_connections 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f703815c040 0x7f70380b4000 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f703806c9b0 0x7f7038155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f702406ace0 0x7f702406d190 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 --2- 172.21.15.104:0/3798425493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f703806ce20 0x7f703815ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:34.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 >> 172.21.15.104:0/3798425493 conn(0x7f70380b1570 msgr2=0x7f70380b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:34.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 shutdown_connections 2023-06-10T18:54:34.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:34.533+0000 7f703d878700 1 -- 172.21.15.104:0/3798425493 wait complete. 2023-06-10T18:54:34.541 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:34.550 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:35.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:35.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.678+0000 7fad4d070700 1 Processor -- start 2023-06-10T18:54:35.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4d070700 1 -- start start 2023-06-10T18:54:35.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4d070700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fad480b1e30 0x7fad480d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:35.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4d070700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4806bfd0 con 0x7fad4806b9f0 2023-06-10T18:54:35.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4d070700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4806c2b0 con 0x7fad480d0a30 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4d070700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad480d0ed0 con 0x7fad480b1e30 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad46d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fad480b1e30 0x7fad480d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4759e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fad4806b9f0 0x7fad480cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33296/0 (socket says 172.21.15.104:33296) 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.679+0000 7fad4759e700 1 -- 172.21.15.104:0/2263707989 learned_addr learned my addr 172.21.15.104:0/2263707989 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad46d9d700 1 -- 172.21.15.104:0/2263707989 >> v1:172.21.15.114:6789/0 conn(0x7fad480d0a30 legacy=0x7fad4806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad46d9d700 1 -- 172.21.15.104:0/2263707989 >> v1:172.21.15.104:6789/0 conn(0x7fad4806b9f0 legacy=0x7fad480cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad45d9b700 1 -- 172.21.15.104:0/2263707989 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342225060 0 0) 0x7fad4806bfd0 con 0x7fad4806b9f0 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad46d9d700 1 -- 172.21.15.104:0/2263707989 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480ccf70 con 0x7fad480b1e30 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad46d9d700 1 --2- 172.21.15.104:0/2263707989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480b1e30 0x7fad480d04f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fad38009840 tx=0x7fad38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1564339076f816 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad45d9b700 1 -- 172.21.15.104:0/2263707989 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38004230 con 0x7fad480b1e30 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad45d9b700 1 -- 172.21.15.104:0/2263707989 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38004390 con 0x7fad480b1e30 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.680+0000 7fad45d9b700 1 -- 172.21.15.104:0/2263707989 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad380036a0 con 0x7fad480b1e30 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 -- 172.21.15.104:0/2263707989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480b1e30 msgr2=0x7fad480d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 --2- 172.21.15.104:0/2263707989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480b1e30 0x7fad480d04f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fad38009840 tx=0x7fad38009b50 comp rx=0 tx=0).stop 2023-06-10T18:54:35.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 -- 172.21.15.104:0/2263707989 shutdown_connections 2023-06-10T18:54:35.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 --2- 172.21.15.104:0/2263707989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480b1e30 0x7fad480d04f0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 -- 172.21.15.104:0/2263707989 >> 172.21.15.104:0/2263707989 conn(0x7fad48071ec0 msgr2=0x7fad4809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:35.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 -- 172.21.15.104:0/2263707989 shutdown_connections 2023-06-10T18:54:35.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.681+0000 7fad4d070700 1 -- 172.21.15.104:0/2263707989 wait complete. 2023-06-10T18:54:35.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.682+0000 7fad4d070700 1 Processor -- start 2023-06-10T18:54:35.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.682+0000 7fad4d070700 1 -- start start 2023-06-10T18:54:35.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 0x7fad4815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:35.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 0x7fad480b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:35.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:35.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad481f22d0 con 0x7fad480d0a30 2023-06-10T18:54:35.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad480b5c90 con 0x7fad4806b9f0 2023-06-10T18:54:35.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4d070700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad480b5f60 con 0x7fad480b1e30 2023-06-10T18:54:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35478/0 (socket says 172.21.15.104:35478) 2023-06-10T18:54:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 -- 172.21.15.104:0/882151345 learned_addr learned my addr 172.21.15.104:0/882151345 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad46d9d700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 0x7fad4815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4659c700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 0x7fad480b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 -- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 msgr2=0x7fad480b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 0x7fad480b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 -- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 msgr2=0x7fad4815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 0x7fad4815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4759e700 1 -- 172.21.15.104:0/882151345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380094a0 con 0x7fad480d0a30 2023-06-10T18:54:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad4659c700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 0x7fad480b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.683+0000 7fad46d9d700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 0x7fad4815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad4759e700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fad4000cca0 tx=0x7fad4000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4000cea0 con 0x7fad480d0a30 2023-06-10T18:54:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4000ade0 con 0x7fad480d0a30 2023-06-10T18:54:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad480b6240 con 0x7fad480d0a30 2023-06-10T18:54:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40007310 con 0x7fad480d0a30 2023-06-10T18:54:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.684+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad480b6800 con 0x7fad480d0a30 2023-06-10T18:54:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.685+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad480d1860 con 0x7fad480d0a30 2023-06-10T18:54:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.685+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fad40007470 con 0x7fad480d0a30 2023-06-10T18:54:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.686+0000 7fad2f7fe700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 0x7fad3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.686+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fad40089260 con 0x7fad480d0a30 2023-06-10T18:54:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.686+0000 7fad46d9d700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 0x7fad3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:35.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.686+0000 7fad46d9d700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 0x7fad3406d190 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fad38009b90 tx=0x7fad38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:35.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.690+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fad40054180 con 0x7fad480d0a30 2023-06-10T18:54:35.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.920+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad480d0850 con 0x7fad480d0a30 2023-06-10T18:54:35.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.921+0000 7fad2f7fe700 1 -- 172.21.15.104:0/882151345 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fad400577a0 con 0x7fad480d0a30 2023-06-10T18:54:35.927 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:35.927 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 msgr2=0x7fad3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 0x7fad3406d190 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fad38009b90 tx=0x7fad38008030 comp rx=0 tx=0).stop 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 msgr2=0x7fad4815a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fad4000cca0 tx=0x7fad4000a340 comp rx=0 tx=0).stop 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 shutdown_connections 2023-06-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad3406ace0 0x7fad3406d190 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4806b9f0 0x7fad4815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad480d0a30 0x7fad4815a710 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 --2- 172.21.15.104:0/882151345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad480b1e30 0x7fad480b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 >> 172.21.15.104:0/882151345 conn(0x7fad48071ec0 msgr2=0x7fad480b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 shutdown_connections 2023-06-10T18:54:35.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:35.923+0000 7fad4d070700 1 -- 172.21.15.104:0/882151345 wait complete. 2023-06-10T18:54:35.931 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:35.941 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:36.943 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:37.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.070+0000 7fca113da700 1 Processor -- start 2023-06-10T18:54:37.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 -- start start 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca0c0ce390 0x7fca0c160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c06c720 0x7fca0c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0d0340 con 0x7fca0c06b410 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0d1660 con 0x7fca0c06c720 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca113da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c15cb60 con 0x7fca0c0ce390 2023-06-10T18:54:37.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.071+0000 7fca0a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c06c720 0x7fca0c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fca0c06c720 0x7fca0c0ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35484/0 (socket says 172.21.15.104:35484) 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 -- 172.21.15.104:0/82932503 learned_addr learned my addr 172.21.15.104:0/82932503 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0affd700 1 --2- 172.21.15.104:0/82932503 >> v2:172.21.15.114:3300/0 conn(0x7fca0c0ce390 0x7fca0c160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 -- 172.21.15.104:0/82932503 >> v2:172.21.15.114:3300/0 conn(0x7fca0c0ce390 msgr2=0x7fca0c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 --2- 172.21.15.104:0/82932503 >> v2:172.21.15.114:3300/0 conn(0x7fca0c0ce390 0x7fca0c160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 -- 172.21.15.104:0/82932503 >> v1:172.21.15.114:6789/0 conn(0x7fca0c06b410 legacy=0x7fca0c06b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 -- 172.21.15.104:0/82932503 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c161230 con 0x7fca0c06c720 2023-06-10T18:54:37.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca0a7fc700 1 --2- 172.21.15.104:0/82932503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06c720 0x7fca0c0ce760 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0b24b0 tx=0x7fc9fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e5d3cd8dd42cc0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca09ffb700 1 -- 172.21.15.104:0/82932503 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00b870 con 0x7fca0c06c720 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca09ffb700 1 -- 172.21.15.104:0/82932503 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc00b9d0 con 0x7fca0c06c720 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.072+0000 7fca09ffb700 1 -- 172.21.15.104:0/82932503 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00bd60 con 0x7fca0c06c720 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 -- 172.21.15.104:0/82932503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06c720 msgr2=0x7fca0c0ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 --2- 172.21.15.104:0/82932503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06c720 0x7fca0c0ce760 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0b24b0 tx=0x7fc9fc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 -- 172.21.15.104:0/82932503 shutdown_connections 2023-06-10T18:54:37.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 --2- 172.21.15.104:0/82932503 >> v2:172.21.15.114:3300/0 conn(0x7fca0c0ce390 0x7fca0c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 --2- 172.21.15.104:0/82932503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06c720 0x7fca0c0ce760 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 -- 172.21.15.104:0/82932503 >> 172.21.15.104:0/82932503 conn(0x7fca0c069e70 msgr2=0x7fca0c06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:37.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 -- 172.21.15.104:0/82932503 shutdown_connections 2023-06-10T18:54:37.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.073+0000 7fca113da700 1 -- 172.21.15.104:0/82932503 wait complete. 2023-06-10T18:54:37.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.074+0000 7fca113da700 1 Processor -- start 2023-06-10T18:54:37.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.074+0000 7fca113da700 1 -- start start 2023-06-10T18:54:37.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c06c720 0x7fca0c0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 0x7fca0c14e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0c5670 con 0x7fca0c06b410 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1e72e0 con 0x7fca0c0ce390 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca113da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1e75c0 con 0x7fca0c06c720 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35488/0 (socket says 172.21.15.104:35488) 2023-06-10T18:54:37.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 -- 172.21.15.104:0/153074901 learned_addr learned my addr 172.21.15.104:0/153074901 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0b7fe700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 0x7fca0c14e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0a7fc700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c06c720 0x7fca0c0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 -- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c06c720 msgr2=0x7fca0c0c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c06c720 0x7fca0c0c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 -- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 msgr2=0x7fca0c14e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 0x7fca0c14e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 -- 172.21.15.104:0/153074901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009550 con 0x7fca0c06b410 2023-06-10T18:54:37.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0b7fe700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 0x7fca0c14e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.075+0000 7fca0affd700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fca0000e810 tx=0x7fca0000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.076+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00013610 con 0x7fca0c06b410 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.076+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca00013770 con 0x7fca0c06b410 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.076+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c1e78a0 con 0x7fca0c06b410 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.076+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00013b20 con 0x7fca0c06b410 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.076+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c15cd30 con 0x7fca0c06b410 2023-06-10T18:54:37.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.077+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c1e7ef0 con 0x7fca0c06b410 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.077+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fca00013c80 con 0x7fca0c06b410 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.077+0000 7fc9f37fe700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 0x7fc9f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.077+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fca00089210 con 0x7fca0c06b410 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.078+0000 7fca0a7fc700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 0x7fc9f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.078+0000 7fca0a7fc700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 0x7fc9f806d190 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca0c06cb60 tx=0x7fc9fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:37.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.082+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fca00057bc0 con 0x7fca0c06b410 2023-06-10T18:54:37.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.310+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca0c0c57b0 con 0x7fca0c06b410 2023-06-10T18:54:37.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.310+0000 7fc9f37fe700 1 -- 172.21.15.104:0/153074901 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fca00057750 con 0x7fca0c06b410 2023-06-10T18:54:37.316 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:37.317 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:37.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 msgr2=0x7fc9f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 0x7fc9f806d190 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca0c06cb60 tx=0x7fc9fc008030 comp rx=0 tx=0).stop 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 msgr2=0x7fca0c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fca0000e810 tx=0x7fca0000eb20 comp rx=0 tx=0).stop 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 shutdown_connections 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc9f806ace0 0x7fc9f806d190 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca0c0ce390 0x7fca0c14e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca0c06b410 0x7fca0c0b4dc0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 --2- 172.21.15.104:0/153074901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca0c06c720 0x7fca0c0c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 >> 172.21.15.104:0/153074901 conn(0x7fca0c069e70 msgr2=0x7fca0c0afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 shutdown_connections 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:37.313+0000 7fca113da700 1 -- 172.21.15.104:0/153074901 wait complete. 2023-06-10T18:54:37.320 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:37.331 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:38.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:38.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.458+0000 7fa8bb1c6700 1 Processor -- start 2023-06-10T18:54:38.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.458+0000 7fa8bb1c6700 1 -- start start 2023-06-10T18:54:38.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8bb1c6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa8b40aeaa0 0x7fa8b40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8bb1c6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa8b40cd680 0x7fa8b406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8bb1c6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b40c9e60 con 0x7fa8b40b1080 2023-06-10T18:54:38.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8bb1c6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40c9bc0 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8bb1c6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40cdaa0 con 0x7fa8b40cd680 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b8f62700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa8b40aeaa0 0x7fa8b40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b8f62700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa8b40aeaa0 0x7fa8b40cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35496/0 (socket says 172.21.15.104:35496) 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b8f62700 1 -- 172.21.15.104:0/364490735 learned_addr learned my addr 172.21.15.104:0/364490735 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b9763700 1 --2- 172.21.15.104:0/364490735 >> v2:172.21.15.114:3300/0 conn(0x7fa8b40cd680 0x7fa8b406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b8f62700 1 -- 172.21.15.104:0/364490735 >> v2:172.21.15.114:3300/0 conn(0x7fa8b40cd680 msgr2=0x7fa8b406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:38.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.459+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/364490735 >> v2:172.21.15.114:3300/0 conn(0x7fa8b40cd680 0x7fa8b406dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b8f62700 1 -- 172.21.15.104:0/364490735 >> v1:172.21.15.114:6789/0 conn(0x7fa8b40b1080 legacy=0x7fa8b40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b8f62700 1 -- 172.21.15.104:0/364490735 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b40aee70 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/364490735 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40aeaa0 0x7fa8b40cd140 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4071030 tx=0x7fa8a800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5430a7033de3d6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b37fe700 1 -- 172.21.15.104:0/364490735 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8005e30 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b37fe700 1 -- 172.21.15.104:0/364490735 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a800d8b0 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8b37fe700 1 -- 172.21.15.104:0/364490735 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a800dc40 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/364490735 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40aeaa0 msgr2=0x7fa8b40cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/364490735 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40aeaa0 0x7fa8b40cd140 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4071030 tx=0x7fa8a800be70 comp rx=0 tx=0).stop 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/364490735 shutdown_connections 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/364490735 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40aeaa0 0x7fa8b40cd140 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/364490735 >> v2:172.21.15.114:3300/0 conn(0x7fa8b40cd680 0x7fa8b406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/364490735 >> 172.21.15.104:0/364490735 conn(0x7fa8b406b8a0 msgr2=0x7fa8b40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/364490735 shutdown_connections 2023-06-10T18:54:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.460+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/364490735 wait complete. 2023-06-10T18:54:38.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 Processor -- start 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 -- start start 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 0x7fa8b40c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 0x7fa8b415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.462+0000 7fa8bb1c6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4154450 con 0x7fa8b40b1080 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8bb1c6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c8b20 con 0x7fa8b40cd680 2023-06-10T18:54:38.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8bb1c6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c7f10 con 0x7fa8b40aeaa0 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35504/0 (socket says 172.21.15.104:35504) 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 -- 172.21.15.104:0/3127448256 learned_addr learned my addr 172.21.15.104:0/3127448256 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 0x7fa8b40c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b9763700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 0x7fa8b415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 -- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 msgr2=0x7fa8b40c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 0x7fa8b40c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 -- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 msgr2=0x7fa8b415baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 0x7fa8b415baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 -- 172.21.15.104:0/3127448256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a800b7a0 con 0x7fa8b40b1080 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 0x7fa8b40c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b9763700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 0x7fa8b415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b3fff700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000c4d0 tx=0x7fa8a000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0011790 con 0x7fa8b40b1080 2023-06-10T18:54:38.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a00118f0 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b40c81f0 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0011ca0 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.463+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b40c87b0 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.464+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40c8cd0 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.465+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa8a001c460 con 0x7fa8b40b1080 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.465+0000 7fa8b17fa700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 0x7fa89806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:38.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.465+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fa8a00893e0 con 0x7fa8b40b1080 2023-06-10T18:54:38.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.465+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 0x7fa89806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:38.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.466+0000 7fa8b8f62700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 0x7fa89806d190 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800beb0 tx=0x7fa8a8012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:38.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.470+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa8b40c8cd0 con 0x7fa8b40b1080 2023-06-10T18:54:38.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.697+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b40c8c90 con 0x7fa8b40b1080 2023-06-10T18:54:38.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.697+0000 7fa8b17fa700 1 -- 172.21.15.104:0/3127448256 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fa8a0054380 con 0x7fa8b40b1080 2023-06-10T18:54:38.703 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:38.704 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:38.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 msgr2=0x7fa89806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:38.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 0x7fa89806d190 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800beb0 tx=0x7fa8a8012000 comp rx=0 tx=0).stop 2023-06-10T18:54:38.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 msgr2=0x7fa8b40c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000c4d0 tx=0x7fa8a000c890 comp rx=0 tx=0).stop 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 shutdown_connections 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa89806ace0 0x7fa89806d190 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8b40cd680 0x7fa8b415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8b40aeaa0 0x7fa8b40c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 --2- 172.21.15.104:0/3127448256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8b40b1080 0x7fa8b40c75c0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:38.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 >> 172.21.15.104:0/3127448256 conn(0x7fa8b406b8a0 msgr2=0x7fa8b40caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.699+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 shutdown_connections 2023-06-10T18:54:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:38.700+0000 7fa8bb1c6700 1 -- 172.21.15.104:0/3127448256 wait complete. 2023-06-10T18:54:38.707 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:38.717 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:39.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:54:39.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:54:39.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.845+0000 7f6063909700 1 Processor -- start 2023-06-10T18:54:39.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.845+0000 7f6063909700 1 -- start start 2023-06-10T18:54:39.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 0x7f605c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f605c0ca100 0x7f605c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c071640 con 0x7f605c0ca100 2023-06-10T18:54:39.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0aef00 con 0x7f605c06e4b0 2023-06-10T18:54:39.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6063909700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c06d290 con 0x7f605c06e0e0 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6061ea6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f605c0ca100 0x7f605c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6061ea6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f605c0ca100 0x7f605c0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35512/0 (socket says 172.21.15.104:35512) 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.846+0000 7f6061ea6700 1 -- 172.21.15.104:0/770533023 learned_addr learned my addr 172.21.15.104:0/770533023 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6060ea4700 1 --2- 172.21.15.104:0/770533023 >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 0x7f605c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f60616a5700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 -- 172.21.15.104:0/770533023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 msgr2=0x7f605c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:39.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 -- 172.21.15.104:0/770533023 >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 msgr2=0x7f605c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 --2- 172.21.15.104:0/770533023 >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 0x7f605c0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 -- 172.21.15.104:0/770533023 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c06d4c0 con 0x7f605c0ca100 2023-06-10T18:54:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6060ea4700 1 --2- 172.21.15.104:0/770533023 >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 0x7f605c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:39.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f60616a5700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6061ea6700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0ce9f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f605c0706d0 tx=0x7f605400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2244e36028448033 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6053fff700 1 -- 172.21.15.104:0/770533023 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605400e070 con 0x7f605c0ca100 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6053fff700 1 -- 172.21.15.104:0/770533023 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605400b980 con 0x7f605c0ca100 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6053fff700 1 -- 172.21.15.104:0/770533023 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605400bd30 con 0x7f605c0ca100 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6063909700 1 -- 172.21.15.104:0/770533023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 msgr2=0x7f605c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.847+0000 7f6063909700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0ce9f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f605c0706d0 tx=0x7f605400b330 comp rx=0 tx=0).stop 2023-06-10T18:54:39.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 -- 172.21.15.104:0/770533023 shutdown_connections 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0ce9f0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 --2- 172.21.15.104:0/770533023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 --2- 172.21.15.104:0/770533023 >> v2:172.21.15.114:3300/0 conn(0x7f605c06e4b0 0x7f605c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 -- 172.21.15.104:0/770533023 >> 172.21.15.104:0/770533023 conn(0x7f605c06b8a0 msgr2=0x7f605c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 -- 172.21.15.104:0/770533023 shutdown_connections 2023-06-10T18:54:39.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.848+0000 7f6063909700 1 -- 172.21.15.104:0/770533023 wait complete. 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.849+0000 7f6063909700 1 Processor -- start 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.849+0000 7f6063909700 1 -- start start 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.849+0000 7f6063909700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c1495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.849+0000 7f6063909700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 0x7f605c159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6063909700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6063909700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b1f10 con 0x7f605c0ca100 2023-06-10T18:54:39.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6063909700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c14c460 con 0x7f605c06e4b0 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6063909700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b1c30 con 0x7f605c06e0e0 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35518/0 (socket says 172.21.15.104:35518) 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 -- 172.21.15.104:0/3869666810 learned_addr learned my addr 172.21.15.104:0/3869666810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f60616a5700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c1495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6060ea4700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 0x7f605c159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 -- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 msgr2=0x7f605c1495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c1495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 -- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 msgr2=0x7f605c159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 0x7f605c159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 -- 172.21.15.104:0/3869666810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054009b30 con 0x7f605c0ca100 2023-06-10T18:54:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6060ea4700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 0x7f605c159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6061ea6700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f605c06cc20 tx=0x7f6054004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054004930 con 0x7f605c0ca100 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60540092e0 con 0x7f605c0ca100 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c0b2560 con 0x7f605c0ca100 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.850+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605401c650 con 0x7f605c0ca100 2023-06-10T18:54:39.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.851+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c0b2b20 con 0x7f605c0ca100 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.851+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c0cabe0 con 0x7f605c0ca100 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.852+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f605401c7b0 con 0x7f605c0ca100 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.852+0000 7f6051ffb700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.852+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f605408b0c0 con 0x7f605c0ca100 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.852+0000 7f60616a5700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:39.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.853+0000 7f60616a5700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:39.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:39.857+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6054056060 con 0x7f605c0ca100 2023-06-10T18:54:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.087+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f605c0b2d10 con 0x7f605c0ca100 2023-06-10T18:54:40.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.088+0000 7f6051ffb700 1 -- 172.21.15.104:0/3869666810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f6054059680 con 0x7f605c0ca100 2023-06-10T18:54:40.094 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:40.094 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:40.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 msgr2=0x7f604006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:40.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c008040 comp rx=0 tx=0).stop 2023-06-10T18:54:40.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 msgr2=0x7f605c0b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f605c06cc20 tx=0x7f6054004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 shutdown_connections 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f605c0ca100 0x7f605c0b10c0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f605c06e0e0 0x7f605c1495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f604006ace0 0x7f604006d190 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 --2- 172.21.15.104:0/3869666810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f605c06e4b0 0x7f605c159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:40.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 >> 172.21.15.104:0/3869666810 conn(0x7f605c06b8a0 msgr2=0x7f605c0ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:40.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 shutdown_connections 2023-06-10T18:54:40.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:40.090+0000 7f6063909700 1 -- 172.21.15.104:0/3869666810 wait complete. 2023-06-10T18:54:40.098 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:40.108 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:41.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:41.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 Processor -- start 2023-06-10T18:54:41.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 -- start start 2023-06-10T18:54:41.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f20c00cea60 0x7f20c0165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c00b6070 con 0x7f20c015a420 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c00b1d70 con 0x7f20c00b4ff0 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.237+0000 7f20c6be9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c00b5460 con 0x7f20c00cea60 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20c4985700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f20c015a420 0x7f20c00b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33350/0 (socket says 172.21.15.104:33350) 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20c4985700 1 -- 172.21.15.104:0/774044974 learned_addr learned my addr 172.21.15.104:0/774044974 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:41.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634435346 0 0) 0x7f20c00b6070 con 0x7f20c015a420 2023-06-10T18:54:41.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b00035b0 con 0x7f20c015a420 2023-06-10T18:54:41.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20c5186700 1 --2- 172.21.15.104:0/774044974 >> v2:172.21.15.114:3300/0 conn(0x7f20c00cea60 0x7f20c0165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103448718 0 0) 0x7f20b00035b0 con 0x7f20c015a420 2023-06-10T18:54:41.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 >> v2:172.21.15.114:3300/0 conn(0x7f20c00cea60 msgr2=0x7f20c0165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:41.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 --2- 172.21.15.104:0/774044974 >> v2:172.21.15.114:3300/0 conn(0x7f20c00cea60 0x7f20c0165060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 >> v1:172.21.15.114:6789/0 conn(0x7f20c00b4ff0 legacy=0x7f20c00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c00b5740 con 0x7f20c015a420 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f20b4001670 con 0x7f20c015a420 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 >> v1:172.21.15.104:6789/0 conn(0x7f20c015a420 legacy=0x7f20c00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20b0003a00 0x7f20b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20b0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20b0005200 0x7f20b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00b5460 con 0x7f20b0004850 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b000c080 con 0x7f20b0005200 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b000c240 con 0x7f20b0003a00 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20bffff700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20b0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20c4985700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20b0003a00 0x7f20b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.238+0000 7f20c5186700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20b0005200 0x7f20b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 -- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20b0003a00 msgr2=0x7f20b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20b0003a00 0x7f20b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 -- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20b0005200 msgr2=0x7f20b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20b0005200 0x7f20b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 -- 172.21.15.104:0/774044974 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00035b0 con 0x7f20b0004850 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20c5186700 1 -- 172.21.15.104:0/774044974 reap_dead start 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bffff700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20b0004cc0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0017c0 tx=0x7f20b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac97a2737c19e2d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac0052d0 con 0x7f20b0004850 2023-06-10T18:54:41.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac005430 con 0x7f20b0004850 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20bf7fe700 1 -- 172.21.15.104:0/774044974 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac0038c0 con 0x7f20b0004850 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20c6be9700 1 -- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 msgr2=0x7f20b0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.239+0000 7f20c6be9700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20b0004cc0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0017c0 tx=0x7f20b0005880 comp rx=0 tx=0).stop 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.240+0000 7f20c6be9700 1 -- 172.21.15.104:0/774044974 shutdown_connections 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.240+0000 7f20c6be9700 1 --2- 172.21.15.104:0/774044974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20b0004cc0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.240+0000 7f20c6be9700 1 -- 172.21.15.104:0/774044974 >> 172.21.15.104:0/774044974 conn(0x7f20c0089fd0 msgr2=0x7f20c00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:41.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.240+0000 7f20c6be9700 1 -- 172.21.15.104:0/774044974 shutdown_connections 2023-06-10T18:54:41.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.240+0000 7f20c6be9700 1 -- 172.21.15.104:0/774044974 wait complete. 2023-06-10T18:54:41.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.241+0000 7f20c6be9700 1 Processor -- start 2023-06-10T18:54:41.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.241+0000 7f20c6be9700 1 -- start start 2023-06-10T18:54:41.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20c00c42b0 0x7f20c00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20c01e1a10 0x7f20c00c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00c53c0 con 0x7f20b0004850 2023-06-10T18:54:41.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00c47b0 con 0x7f20c01e1a10 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00c4a90 con 0x7f20c00c42b0 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35534/0 (socket says 172.21.15.104:35534) 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 -- 172.21.15.104:0/3615243338 learned_addr learned my addr 172.21.15.104:0/3615243338 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20bffff700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20c00c42b0 0x7f20c00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c5186700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20c01e1a10 0x7f20c00c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 -- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20c00c42b0 msgr2=0x7f20c00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20c00c42b0 0x7f20c00c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 -- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20c01e1a10 msgr2=0x7f20c00c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20c01e1a10 0x7f20c00c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 -- 172.21.15.104:0/3615243338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c00c4d70 con 0x7f20b0004850 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c4985700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f20b40072c0 tx=0x7f20b4007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b40077a0 con 0x7f20b0004850 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b400f8d0 con 0x7f20b0004850 2023-06-10T18:54:41.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c00c5050 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.242+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400fc80 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.243+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c00c82d0 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.244+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c00c84f0 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.244+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f20b401d460 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.244+0000 7f20bd7fa700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 0x7f20a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.244+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f20b4089680 con 0x7f20b0004850 2023-06-10T18:54:41.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.244+0000 7f20bffff700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 0x7f20a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:41.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.245+0000 7f20bffff700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 0x7f20a406d190 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f20c00c9bd0 tx=0x7f20ac009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:41.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.248+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f20b4054590 con 0x7f20b0004850 2023-06-10T18:54:41.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.477+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20c0074050 con 0x7f20b0004850 2023-06-10T18:54:41.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.477+0000 7f20bd7fa700 1 -- 172.21.15.104:0/3615243338 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f20b4057bb0 con 0x7f20b0004850 2023-06-10T18:54:41.483 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:41.484 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:41.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.479+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 msgr2=0x7f20a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.479+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 0x7f20a406d190 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f20c00c9bd0 tx=0x7f20ac009030 comp rx=0 tx=0).stop 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 msgr2=0x7f20c00c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f20b40072c0 tx=0x7f20b4007680 comp rx=0 tx=0).stop 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 shutdown_connections 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20a406ace0 0x7f20a406d190 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20c01e1a10 0x7f20c00c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20b0004850 0x7f20c00c5850 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 --2- 172.21.15.104:0/3615243338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20c00c42b0 0x7f20c00c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:41.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 >> 172.21.15.104:0/3615243338 conn(0x7f20c0089fd0 msgr2=0x7f20c00cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:41.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 shutdown_connections 2023-06-10T18:54:41.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:41.480+0000 7f20c6be9700 1 -- 172.21.15.104:0/3615243338 wait complete. 2023-06-10T18:54:41.487 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:41.498 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:42.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:42.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.622+0000 7f5da8311700 1 Processor -- start 2023-06-10T18:54:42.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.622+0000 7f5da8311700 1 -- start start 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da8311700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5da0158c50 0x7f5da006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da8311700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da00cccd0 con 0x7f5da006b180 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da8311700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da006a010 con 0x7f5da006d150 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da8311700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da00ccfb0 con 0x7f5da0158c50 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da58ac700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5da006b180 0x7f5da006ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33370/0 (socket says 172.21.15.104:33370) 2023-06-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da58ac700 1 -- 172.21.15.104:0/2123832300 learned_addr learned my addr 172.21.15.104:0/2123832300 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da60ad700 1 --2- 172.21.15.104:0/2123832300 >> v2:172.21.15.114:3300/0 conn(0x7f5da0158c50 0x7f5da006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939973187 0 0) 0x7f5da00cccd0 con 0x7f5da006b180 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.623+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c0057c0 con 0x7f5da006b180 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2087065523 0 0) 0x7f5d8c0057c0 con 0x7f5da006b180 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 >> v2:172.21.15.114:3300/0 conn(0x7f5da0158c50 msgr2=0x7f5da006aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 --2- 172.21.15.104:0/2123832300 >> v2:172.21.15.114:3300/0 conn(0x7f5da0158c50 0x7f5da006aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 >> v1:172.21.15.114:6789/0 conn(0x7f5da006d150 legacy=0x7f5da00c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da006a2f0 con 0x7f5da006b180 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5d9c001670 con 0x7f5da006b180 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 >> v1:172.21.15.104:6789/0 conn(0x7f5da006b180 legacy=0x7f5da006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c005c10 0x7f5d8c006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d8c006a60 0x7f5d8c006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 0x7f5d8c00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00ccfb0 con 0x7f5d8c007410 2023-06-10T18:54:42.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c00e320 con 0x7f5d8c006a60 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c00e4e0 con 0x7f5d8c005c10 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 0x7f5d8c00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da60ad700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c005c10 0x7f5d8c006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da58ac700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d8c006a60 0x7f5d8c006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 -- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c005c10 msgr2=0x7f5d8c006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c005c10 0x7f5d8c006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 -- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d8c006a60 msgr2=0x7f5d8c006ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:42.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d8c006a60 0x7f5d8c006ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 -- 172.21.15.104:0/2123832300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c0057c0 con 0x7f5d8c007410 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 -- 172.21.15.104:0/2123832300 reap_dead start 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 0x7f5d8c00dc20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800c160 tx=0x7f5d8c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7d97a654eada37 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.624+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98001db0 con 0x7f5d8c007410 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9800ccc0 con 0x7f5d8c007410 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da50ab700 1 -- 172.21.15.104:0/2123832300 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98003c40 con 0x7f5d8c007410 2023-06-10T18:54:42.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 -- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 msgr2=0x7f5d8c00dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 0x7f5d8c00dc20 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800c160 tx=0x7f5d8c007a90 comp rx=0 tx=0).stop 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 -- 172.21.15.104:0/2123832300 shutdown_connections 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 --2- 172.21.15.104:0/2123832300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d8c007410 0x7f5d8c00dc20 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 -- 172.21.15.104:0/2123832300 >> 172.21.15.104:0/2123832300 conn(0x7f5da00b1610 msgr2=0x7f5da00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 -- 172.21.15.104:0/2123832300 shutdown_connections 2023-06-10T18:54:42.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.625+0000 7f5da8311700 1 -- 172.21.15.104:0/2123832300 wait complete. 2023-06-10T18:54:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.626+0000 7f5da8311700 1 Processor -- start 2023-06-10T18:54:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.626+0000 7f5da8311700 1 -- start start 2023-06-10T18:54:42.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c007410 0x7f5da01546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5da015bba0 0x7f5da01f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da01562c0 con 0x7f5da015c010 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00acb40 con 0x7f5da015bba0 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da8311700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da015a2d0 con 0x7f5d8c007410 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da68ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da68ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35558/0 (socket says 172.21.15.104:35558) 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da68ae700 1 -- 172.21.15.104:0/2186267048 learned_addr learned my addr 172.21.15.104:0/2186267048 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da60ad700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c007410 0x7f5da01546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.627+0000 7f5da58ac700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5da015bba0 0x7f5da01f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 -- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c007410 msgr2=0x7f5da01546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c007410 0x7f5da01546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 -- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5da015bba0 msgr2=0x7f5da01f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5da015bba0 0x7f5da01f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 -- 172.21.15.104:0/2186267048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da00ac4f0 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da68ae700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5d98005040 tx=0x7f5d980014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d980050d0 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d98005230 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d980055a0 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da00ac7d0 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.628+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da015a790 con 0x7f5da015c010 2023-06-10T18:54:42.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.629+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da00accd0 con 0x7f5da015c010 2023-06-10T18:54:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.629+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d98005700 con 0x7f5da015c010 2023-06-10T18:54:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.630+0000 7f5d96ffd700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 0x7f5d8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.630+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f5d9808abd0 con 0x7f5da015c010 2023-06-10T18:54:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.630+0000 7f5da60ad700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 0x7f5d8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:42.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.630+0000 7f5da60ad700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 0x7f5d8406d190 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009b20 tx=0x7f5d90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:42.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.634+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d98055ad0 con 0x7f5da015c010 2023-06-10T18:54:42.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.859+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da00cc8e0 con 0x7f5da015c010 2023-06-10T18:54:42.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.860+0000 7f5d96ffd700 1 -- 172.21.15.104:0/2186267048 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f5d98016090 con 0x7f5da015c010 2023-06-10T18:54:42.866 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:42.866 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:42.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 msgr2=0x7f5d8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 0x7f5d8406d190 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009b20 tx=0x7f5d90008040 comp rx=0 tx=0).stop 2023-06-10T18:54:42.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 msgr2=0x7f5da01f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:42.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5d98005040 tx=0x7f5d980014a0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 shutdown_connections 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5da015c010 0x7f5da01f5820 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d8c007410 0x7f5da01546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d8406ace0 0x7f5d8406d190 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 --2- 172.21.15.104:0/2186267048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5da015bba0 0x7f5da01f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 >> 172.21.15.104:0/2186267048 conn(0x7f5da00b1610 msgr2=0x7f5da0157f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 shutdown_connections 2023-06-10T18:54:42.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:42.862+0000 7f5da8311700 1 -- 172.21.15.104:0/2186267048 wait complete. 2023-06-10T18:54:42.870 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:42.879 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:43.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:44.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.009+0000 7fc18c940700 1 Processor -- start 2023-06-10T18:54:44.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.009+0000 7fc18c940700 1 -- start start 2023-06-10T18:54:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18c940700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc18806e4b0 0x7fc18806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18c940700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18806d290 con 0x7fc188069e70 2023-06-10T18:54:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18c940700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1880ca2d0 con 0x7fc18806e0e0 2023-06-10T18:54:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18c940700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc188071640 con 0x7fc18806e4b0 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc186d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc188069e70 0x7fc1880ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33394/0 (socket says 172.21.15.104:33394) 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc186d9d700 1 -- 172.21.15.104:0/780098115 learned_addr learned my addr 172.21.15.104:0/780098115 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc185d9b700 1 --2- 172.21.15.104:0/780098115 >> v2:172.21.15.114:3300/0 conn(0x7fc18806e4b0 0x7fc18806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850510816 0 0) 0x7fc18806d290 con 0x7fc188069e70 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1700035b0 con 0x7fc188069e70 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1685995171 0 0) 0x7fc1700035b0 con 0x7fc188069e70 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 >> v2:172.21.15.114:3300/0 conn(0x7fc18806e4b0 msgr2=0x7fc18806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 --2- 172.21.15.104:0/780098115 >> v2:172.21.15.114:3300/0 conn(0x7fc18806e4b0 0x7fc18806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 >> v1:172.21.15.114:6789/0 conn(0x7fc18806e0e0 legacy=0x7fc1880c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18806d4c0 con 0x7fc188069e70 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc1800019f0 con 0x7fc188069e70 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 >> v1:172.21.15.104:6789/0 conn(0x7fc188069e70 legacy=0x7fc1880ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc170003a00 0x7fc170003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.010+0000 7fc18559a700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170005320 0x7fc17000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188071640 con 0x7fc170005320 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17000c250 con 0x7fc170004850 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17000c410 con 0x7fc170003a00 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc186d9d700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170005320 0x7fc17000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18659c700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc170003a00 0x7fc170003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 -- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc170003a00 msgr2=0x7fc170003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc170003a00 0x7fc170003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 -- 172.21.15.104:0/780098115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170005320 msgr2=0x7fc17000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170005320 0x7fc17000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 -- 172.21.15.104:0/780098115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700035b0 con 0x7fc170004850 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc186d9d700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170005320 0x7fc17000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc186d9d700 1 -- 172.21.15.104:0/780098115 reap_dead start 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc185d9b700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc170004cc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009b60 tx=0x7fc170005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f72944859001eec0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00e6c0 con 0x7fc170004850 2023-06-10T18:54:44.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c00e820 con 0x7fc170004850 2023-06-10T18:54:44.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.011+0000 7fc18559a700 1 -- 172.21.15.104:0/780098115 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00eb90 con 0x7fc170004850 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 -- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 msgr2=0x7fc170004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc170004cc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009b60 tx=0x7fc170005880 comp rx=0 tx=0).stop 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 -- 172.21.15.104:0/780098115 shutdown_connections 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 --2- 172.21.15.104:0/780098115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 -- 172.21.15.104:0/780098115 >> 172.21.15.104:0/780098115 conn(0x7fc18806b8a0 msgr2=0x7fc1880cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 -- 172.21.15.104:0/780098115 shutdown_connections 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.012+0000 7fc18c940700 1 -- 172.21.15.104:0/780098115 wait complete. 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.013+0000 7fc18c940700 1 Processor -- start 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.013+0000 7fc18c940700 1 -- start start 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc1880b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc188155440 0x7fc1881558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188155e00 con 0x7fc1880b2430 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880b5440 con 0x7fc170004850 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18c940700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880b43e0 con 0x7fc188155440 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc185d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc185d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35574/0 (socket says 172.21.15.104:35574) 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc185d9b700 1 -- 172.21.15.104:0/2612745736 learned_addr learned my addr 172.21.15.104:0/2612745736 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc18659c700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc1880b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.014+0000 7fc186d9d700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc188155440 0x7fc1881558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 -- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc188155440 msgr2=0x7fc1881558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc188155440 0x7fc1881558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 -- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 msgr2=0x7fc1880b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc1880b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 -- 172.21.15.104:0/2612745736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880b4680 con 0x7fc1880b2430 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc185d9b700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c3e0 tx=0x7fc17c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00e6c0 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c003ad0 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1880b4960 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c003e80 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.015+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1880b4f20 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.016+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1880b5110 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.016+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc17c013470 con 0x7fc1880b2430 2023-06-10T18:54:44.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.017+0000 7fc176ffd700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 0x7fc16806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:44.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.017+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fc17c08add0 con 0x7fc1880b2430 2023-06-10T18:54:44.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.017+0000 7fc18659c700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 0x7fc16806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:44.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.017+0000 7fc18659c700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 0x7fc16806d190 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc1880b1c40 tx=0x7fc178009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:44.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.021+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc17c055e90 con 0x7fc1880b2430 2023-06-10T18:54:44.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.251+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc188074050 con 0x7fc1880b2430 2023-06-10T18:54:44.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.252+0000 7fc176ffd700 1 -- 172.21.15.104:0/2612745736 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fc17c017090 con 0x7fc1880b2430 2023-06-10T18:54:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:44.258 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:44.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 msgr2=0x7fc16806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 0x7fc16806d190 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc1880b1c40 tx=0x7fc178009040 comp rx=0 tx=0).stop 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 msgr2=0x7fc1880caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c3e0 tx=0x7fc17c0047c0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 shutdown_connections 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc16806ace0 0x7fc16806d190 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc170004850 0x7fc1880b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc188155440 0x7fc1881558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.253+0000 7fc18c940700 1 --2- 172.21.15.104:0/2612745736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1880b2430 0x7fc1880caf30 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:44.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.254+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 >> 172.21.15.104:0/2612745736 conn(0x7fc18806b8a0 msgr2=0x7fc18806a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:44.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.254+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 shutdown_connections 2023-06-10T18:54:44.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:44.254+0000 7fc18c940700 1 -- 172.21.15.104:0/2612745736 wait complete. 2023-06-10T18:54:44.261 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:44.271 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:45.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.400+0000 7f08e81ce700 1 Processor -- start 2023-06-10T18:54:45.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.401+0000 7f08e81ce700 1 -- start start 2023-06-10T18:54:45.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e81ce700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08e00b4ff0 0x7f08e00ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e81ce700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00b6070 con 0x7f08e015a420 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e81ce700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00b1d70 con 0x7f08e00ce890 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e81ce700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e00cecb0 con 0x7f08e00b4ff0 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08e00b4ff0 0x7f08e00ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08e00b4ff0 0x7f08e00ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35584/0 (socket says 172.21.15.104:35584) 2023-06-10T18:54:45.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 -- 172.21.15.104:0/42344364 learned_addr learned my addr 172.21.15.104:0/42344364 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5f6a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f08e015a420 0x7f08e00b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33410/0 (socket says 172.21.15.104:33410) 2023-06-10T18:54:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 -- 172.21.15.104:0/42344364 >> v1:172.21.15.114:6789/0 conn(0x7f08e00ce890 legacy=0x7f08e015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 -- 172.21.15.104:0/42344364 >> v1:172.21.15.104:6789/0 conn(0x7f08e015a420 legacy=0x7f08e00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 -- 172.21.15.104:0/42344364 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e00b5460 con 0x7f08e00b4ff0 2023-06-10T18:54:45.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e4f68700 1 -- 172.21.15.104:0/42344364 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215274546 0 0) 0x7f08e00b6070 con 0x7f08e015a420 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e5769700 1 --2- 172.21.15.104:0/42344364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e00ce350 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f08d00098f0 tx=0x7f08d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd13ece97a015ad8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e4f68700 1 -- 172.21.15.104:0/42344364 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0004230 con 0x7f08e00b4ff0 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e4f68700 1 -- 172.21.15.104:0/42344364 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d0004390 con 0x7f08e00b4ff0 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.402+0000 7f08e4f68700 1 -- 172.21.15.104:0/42344364 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00036a0 con 0x7f08e00b4ff0 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 -- 172.21.15.104:0/42344364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 msgr2=0x7f08e00ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 --2- 172.21.15.104:0/42344364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e00ce350 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f08d00098f0 tx=0x7f08d0009c00 comp rx=0 tx=0).stop 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 -- 172.21.15.104:0/42344364 shutdown_connections 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 --2- 172.21.15.104:0/42344364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e00ce350 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 -- 172.21.15.104:0/42344364 >> 172.21.15.104:0/42344364 conn(0x7f08e0089fd0 msgr2=0x7f08e00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 -- 172.21.15.104:0/42344364 shutdown_connections 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.403+0000 7f08e81ce700 1 -- 172.21.15.104:0/42344364 wait complete. 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.404+0000 7f08e81ce700 1 Processor -- start 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.404+0000 7f08e81ce700 1 -- start start 2023-06-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e00ce890 0x7f08e00ad370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08e015a420 0x7f08e00ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b0150 con 0x7f08e00b4ff0 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b0430 con 0x7f08e00ce890 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e81ce700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b0710 con 0x7f08e015a420 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35588/0 (socket says 172.21.15.104:35588) 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 -- 172.21.15.104:0/102560482 learned_addr learned my addr 172.21.15.104:0/102560482 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 -- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08e015a420 msgr2=0x7f08e00ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08e015a420 0x7f08e00ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 -- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e00ce890 msgr2=0x7f08e00ad370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5769700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e00ce890 0x7f08e00ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e00ce890 0x7f08e00ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.405+0000 7f08e5f6a700 1 -- 172.21.15.104:0/102560482 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009550 con 0x7f08e00b4ff0 2023-06-10T18:54:45.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08e5f6a700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c550 tx=0x7f08dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc004c80 con 0x7f08e00b4ff0 2023-06-10T18:54:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc004e00 con 0x7f08e00b4ff0 2023-06-10T18:54:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e00b09f0 con 0x7f08e00b4ff0 2023-06-10T18:54:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc007310 con 0x7f08e00b4ff0 2023-06-10T18:54:45.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.406+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e01f09e0 con 0x7f08e00b4ff0 2023-06-10T18:54:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.407+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e00c8b40 con 0x7f08e00b4ff0 2023-06-10T18:54:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.407+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f08dc007470 con 0x7f08e00b4ff0 2023-06-10T18:54:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.407+0000 7f08d67fc700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 0x7f08c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:45.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.408+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f08dc089350 con 0x7f08e00b4ff0 2023-06-10T18:54:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.408+0000 7f08e5769700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 0x7f08c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.408+0000 7f08e5769700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 0x7f08c406d190 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f08e00c4640 tx=0x7f08d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:45.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.412+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f08e00c8b40 con 0x7f08e00b4ff0 2023-06-10T18:54:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.637+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e00c8b40 con 0x7f08e00b4ff0 2023-06-10T18:54:45.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.637+0000 7f08d67fc700 1 -- 172.21.15.104:0/102560482 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f08dc054240 con 0x7f08e00b4ff0 2023-06-10T18:54:45.643 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:45.644 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:53:58.698403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 msgr2=0x7f08c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 0x7f08c406d190 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f08e00c4640 tx=0x7f08d0008030 comp rx=0 tx=0).stop 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 msgr2=0x7f08e01f04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c550 tx=0x7f08dc00c910 comp rx=0 tx=0).stop 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 shutdown_connections 2023-06-10T18:54:45.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08e015a420 0x7f08e00ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e00b4ff0 0x7f08e01f04a0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08c406ace0 0x7f08c406d190 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 --2- 172.21.15.104:0/102560482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e00ce890 0x7f08e00ad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 >> 172.21.15.104:0/102560482 conn(0x7f08e0089fd0 msgr2=0x7f08e0157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 shutdown_connections 2023-06-10T18:54:45.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:45.640+0000 7f08e81ce700 1 -- 172.21.15.104:0/102560482 wait complete. 2023-06-10T18:54:45.648 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 60 2023-06-10T18:54:45.657 DEBUG:tasks.cephfs.filesystem:mapped states ['up:reconnect'] to None 2023-06-10T18:54:46.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:46.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.837+0000 7f3a48907700 1 Processor -- start 2023-06-10T18:54:46.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.837+0000 7f3a48907700 1 -- start start 2023-06-10T18:54:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a48907700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a440b0b70 0x7f3a44158280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a48907700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440ccc20 con 0x7f3a440c8bd0 2023-06-10T18:54:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a48907700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4415cc50 con 0x7f3a440cebb0 2023-06-10T18:54:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a48907700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a440cc2f0 con 0x7f3a440b0b70 2023-06-10T18:54:46.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a41d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3a440c8bd0 0x7f3a440ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33428/0 (socket says 172.21.15.104:33428) 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a41d9b700 1 -- 172.21.15.104:0/4021515588 learned_addr learned my addr 172.21.15.104:0/4021515588 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.838+0000 7f3a4259c700 1 --2- 172.21.15.104:0/4021515588 >> v2:172.21.15.114:3300/0 conn(0x7f3a440b0b70 0x7f3a44158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4159a700 1 -- 172.21.15.104:0/4021515588 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440249215 0 0) 0x7f3a440ccc20 con 0x7f3a440c8bd0 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4259c700 1 -- 172.21.15.104:0/4021515588 >> v1:172.21.15.114:6789/0 conn(0x7f3a440cebb0 legacy=0x7f3a440cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4259c700 1 -- 172.21.15.104:0/4021515588 >> v1:172.21.15.104:6789/0 conn(0x7f3a440c8bd0 legacy=0x7f3a440ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4259c700 1 -- 172.21.15.104:0/4021515588 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440cc5d0 con 0x7f3a440b0b70 2023-06-10T18:54:46.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4259c700 1 --2- 172.21.15.104:0/4021515588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44158280 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3a340097c0 tx=0x7f3a34009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b35b886daf9675 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:46.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4159a700 1 -- 172.21.15.104:0/4021515588 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34004230 con 0x7f3a440b0b70 2023-06-10T18:54:46.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4159a700 1 -- 172.21.15.104:0/4021515588 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34004390 con 0x7f3a440b0b70 2023-06-10T18:54:46.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.839+0000 7f3a4159a700 1 -- 172.21.15.104:0/4021515588 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a340036a0 con 0x7f3a440b0b70 2023-06-10T18:54:46.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 -- 172.21.15.104:0/4021515588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 msgr2=0x7f3a44158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:46.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 --2- 172.21.15.104:0/4021515588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44158280 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3a340097c0 tx=0x7f3a34009ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:46.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 -- 172.21.15.104:0/4021515588 shutdown_connections 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 --2- 172.21.15.104:0/4021515588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44158280 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 -- 172.21.15.104:0/4021515588 >> 172.21.15.104:0/4021515588 conn(0x7f3a440b0090 msgr2=0x7f3a440b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 -- 172.21.15.104:0/4021515588 shutdown_connections 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.840+0000 7f3a48907700 1 -- 172.21.15.104:0/4021515588 wait complete. 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 Processor -- start 2023-06-10T18:54:46.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 -- start start 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a440c8bd0 0x7f3a440c1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44140660 con 0x7f3a440cebb0 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4413fb80 con 0x7f3a440b0b70 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a48907700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4413f250 con 0x7f3a440c8bd0 2023-06-10T18:54:46.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a41d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a440c8bd0 0x7f3a440c1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.842+0000 7f3a4259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35608/0 (socket says 172.21.15.104:35608) 2023-06-10T18:54:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 -- 172.21.15.104:0/534742041 learned_addr learned my addr 172.21.15.104:0/534742041 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 -- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a440c8bd0 msgr2=0x7f3a440c1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a440c8bd0 0x7f3a440c1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 -- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 msgr2=0x7f3a44140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 -- 172.21.15.104:0/534742041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a340094a0 con 0x7f3a440cebb0 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a4259c700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:46.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a42d9d700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c850 tx=0x7f3a3c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:46.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c003e30 con 0x7f3a440cebb0 2023-06-10T18:54:46.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c005070 con 0x7f3a440cebb0 2023-06-10T18:54:46.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4413f530 con 0x7f3a440cebb0 2023-06-10T18:54:46.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c005420 con 0x7f3a440cebb0 2023-06-10T18:54:46.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.843+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a44143dc0 con 0x7f3a440cebb0 2023-06-10T18:54:46.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.844+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4413fd30 con 0x7f3a440cebb0 2023-06-10T18:54:46.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.845+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3a3c005580 con 0x7f3a440cebb0 2023-06-10T18:54:46.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.845+0000 7f3a32ffd700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 0x7f3a2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:46.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.845+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f3a3c088840 con 0x7f3a440cebb0 2023-06-10T18:54:46.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.845+0000 7f3a4259c700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 0x7f3a2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:46.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.845+0000 7f3a4259c700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 0x7f3a2406d190 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3a340090b0 tx=0x7f3a34008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:46.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:46.849+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3a4413fd30 con 0x7f3a440cebb0 2023-06-10T18:54:47.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.080+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a4413fcf0 con 0x7f3a440cebb0 2023-06-10T18:54:47.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.081+0000 7f3a32ffd700 1 -- 172.21.15.104:0/534742041 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4682 (secure 0 0 0) 0x7f3a3c0571d0 con 0x7f3a440cebb0 2023-06-10T18:54:47.087 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:47.088 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:54:46.733023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:47.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 msgr2=0x7f3a2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 0x7f3a2406d190 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3a340090b0 tx=0x7f3a34008030 comp rx=0 tx=0).stop 2023-06-10T18:54:47.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 msgr2=0x7f3a440c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c850 tx=0x7f3a3c00cb60 comp rx=0 tx=0).stop 2023-06-10T18:54:47.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 shutdown_connections 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2406ace0 0x7f3a2406d190 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a440b0b70 0x7f3a44140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a440cebb0 0x7f3a440c5eb0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 --2- 172.21.15.104:0/534742041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a440c8bd0 0x7f3a440c1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 >> 172.21.15.104:0/534742041 conn(0x7f3a440b0090 msgr2=0x7f3a4414b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:47.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 shutdown_connections 2023-06-10T18:54:47.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.083+0000 7f3a48907700 1 -- 172.21.15.104:0/534742041 wait complete. 2023-06-10T18:54:47.091 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 62 2023-06-10T18:54:47.102 DEBUG:tasks.cephfs.filesystem:mapped states ['up:active'] to up:active 2023-06-10T18:54:47.102 DEBUG:tasks.cephfs.filesystem:reached state 'up:active' in 47.61520195007324s 2023-06-10T18:54:47.102 INFO:tasks.cephfs.test_client_recovery:Reached active... 2023-06-10T18:54:47.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:54:47.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.217+0000 7fcfc3f12700 1 Processor -- start 2023-06-10T18:54:47.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.217+0000 7fcfc3f12700 1 -- start start 2023-06-10T18:54:47.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc3f12700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcfbc158c50 0x7fcfbc06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc3f12700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 0x7fcfbc0cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc3f12700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfbc0cccd0 con 0x7fcfbc06ca40 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc3f12700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfbc0ccfb0 con 0x7fcfbc158c50 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc3f12700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc0ce0b0 con 0x7fcfbc06b180 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcfbc158c50 0x7fcfbc06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc24af700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcfbc06ca40 0x7fcfbc06ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33444/0 (socket says 172.21.15.104:33444) 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc24af700 1 -- 172.21.15.104:0/3121256478 learned_addr learned my addr 172.21.15.104:0/3121256478 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc14ad700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 0x7fcfbc0cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 -- 172.21.15.104:0/3121256478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 msgr2=0x7fcfbc0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 0x7fcfbc0cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 -- 172.21.15.104:0/3121256478 >> v1:172.21.15.104:6789/0 conn(0x7fcfbc06ca40 legacy=0x7fcfbc06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 -- 172.21.15.104:0/3121256478 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc06a010 con 0x7fcfbc158c50 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc14ad700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 0x7fcfbc0cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc1cae700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc06aaf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009870 tx=0x7fcfb8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c307c7f75baa084 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc0cac700 1 -- 172.21.15.104:0/3121256478 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8011830 con 0x7fcfbc158c50 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc0cac700 1 -- 172.21.15.104:0/3121256478 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8011990 con 0x7fcfbc158c50 2023-06-10T18:54:47.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.218+0000 7fcfc0cac700 1 -- 172.21.15.104:0/3121256478 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8011d20 con 0x7fcfbc158c50 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 -- 172.21.15.104:0/3121256478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 msgr2=0x7fcfbc06aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc06aaf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009870 tx=0x7fcfb8009b80 comp rx=0 tx=0).stop 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 -- 172.21.15.104:0/3121256478 shutdown_connections 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc06aaf0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/3121256478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06b180 0x7fcfbc0cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 -- 172.21.15.104:0/3121256478 >> 172.21.15.104:0/3121256478 conn(0x7fcfbc0b1610 msgr2=0x7fcfbc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:47.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 -- 172.21.15.104:0/3121256478 shutdown_connections 2023-06-10T18:54:47.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.219+0000 7fcfc3f12700 1 -- 172.21.15.104:0/3121256478 wait complete. 2023-06-10T18:54:47.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.221+0000 7fcfc3f12700 1 Processor -- start 2023-06-10T18:54:47.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.221+0000 7fcfc3f12700 1 -- start start 2023-06-10T18:54:47.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfbc06b180 0x7fcfbc14d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06ca40 0x7fcfbc0c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc14de30 con 0x7fcfbc158c50 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc146c80 con 0x7fcfbc06b180 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc146f60 con 0x7fcfbc06ca40 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35622/0 (socket says 172.21.15.104:35622) 2023-06-10T18:54:47.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 -- 172.21.15.104:0/2579802167 learned_addr learned my addr 172.21.15.104:0/2579802167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc1cae700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfbc06b180 0x7fcfbc14d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc14ad700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06ca40 0x7fcfbc0c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 -- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06ca40 msgr2=0x7fcfbc0c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06ca40 0x7fcfbc0c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 -- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfbc06b180 msgr2=0x7fcfbc14d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfbc06b180 0x7fcfbc14d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 -- 172.21.15.104:0/2579802167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80094a0 con 0x7fcfbc158c50 2023-06-10T18:54:47.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc24af700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cbc0 tx=0x7fcfb400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400a9e0 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb400ab40 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4012400 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc147240 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.222+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc147800 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.223+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc074050 con 0x7fcfbc158c50 2023-06-10T18:54:47.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.224+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcfb401b3b0 con 0x7fcfbc158c50 2023-06-10T18:54:47.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.224+0000 7fcfb27fc700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:47.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.224+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fcfb40890d0 con 0x7fcfbc158c50 2023-06-10T18:54:47.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.224+0000 7fcfc1cae700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:47.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.225+0000 7fcfc1cae700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 0x7fcfa006d190 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097c0 tx=0x7fcfb801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:47.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.228+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcfb4054070 con 0x7fcfbc158c50 2023-06-10T18:54:47.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.456+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcfbc1639f0 con 0x7fcfbc158c50 2023-06-10T18:54:47.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.456+0000 7fcfb27fc700 1 -- 172.21.15.104:0/2579802167 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcfb4057690 con 0x7fcfbc158c50 2023-06-10T18:54:47.462 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:54:47.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 msgr2=0x7fcfa006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 0x7fcfa006d190 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097c0 tx=0x7fcfb801c040 comp rx=0 tx=0).stop 2023-06-10T18:54:47.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 msgr2=0x7fcfbc0b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cbc0 tx=0x7fcfb400ced0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 shutdown_connections 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfbc06b180 0x7fcfbc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc158c50 0x7fcfbc0b4250 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 --2- 172.21.15.104:0/2579802167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfbc06ca40 0x7fcfbc0c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:47.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 >> 172.21.15.104:0/2579802167 conn(0x7fcfbc0b1610 msgr2=0x7fcfbc0af030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:47.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 shutdown_connections 2023-06-10T18:54:47.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:47.458+0000 7fcfc3f12700 1 -- 172.21.15.104:0/2579802167 wait complete. 2023-06-10T18:54:47.476 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:54:47.476 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:54:47.476 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:54:47.477 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:54:47.477 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:47.477 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:54:47.477 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:54:47.478 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:54:47.478 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:54:47.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.479 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:54:47.500 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:54:47.500 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:54:47.500 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:54:47.500 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.500 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:54:47.501 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41792sec preferred_lft 41792sec 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:54:47.502 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:54:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.504 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:54:47.504 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:54:47.504 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:54:47.504 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:54:47.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.505 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:54:47.569 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:54:47.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.571 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:54:47.631 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:54:47.631 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:54:47.631 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:54:47.631 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:54:47.632 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:54:47.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.632 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:54:47.633 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:54:47.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:54:47.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:54:47.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:54:47.634 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:54:47.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.743 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:54:47.760 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:54:47.760 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:54:47.761 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:54:47.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:47.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:54:47.828 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:54:47.828 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:54:47.828 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:47.884 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:54:47.885 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:54:47.886 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:47.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:54:48.008 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:54:48.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:54:48.074 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:54:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:54:48.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.076 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:54:48.132 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:54:48.133 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:54:48.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:54:48.199 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:54:48.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:54:48.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:54:48.253+0000 7f577d0835c0 -1 init, newargv = 0x55a58b5f6ad0 newargc=15 2023-06-10T18:54:48.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[113355]: starting ceph client 2023-06-10T18:54:48.264 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[113355]: starting fuse 2023-06-10T18:54:48.268 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:54:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:54:48.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.270 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:54:48.327 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:54:48.327 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:54:48.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:54:48.328 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:54:48.328 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:54:48.329 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:54:48.329 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:54:48.329 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:54:48.329 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:54:48.330 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:54:48.330 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:54:48.330 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:54:48.331 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:54:48.331 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:54:48.331 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:54:48.332 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:54:48.332 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:54:48.332 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:54:48.333 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:54:48.333 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:54:48.333 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:54:48.334 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:54:48.334 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:54:48.334 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:54:48.334 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:54:48.335 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:54:48.335 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:54:48.335 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:54:48.336 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:54:48.448 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.113355.asok 2023-06-10T18:54:48.630 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.113355.asok 2023-06-10T18:54:48.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.113355.asok status 2023-06-10T18:54:48.787 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:54:48.787 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:54:48.787 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "113355", 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:54:48.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 6527, 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:54:48.789 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6527 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1948672403 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:54:48.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:54:48.791 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:54:48.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:54:48.791 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:54:48.791 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1948672403 2023-06-10T18:54:48.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:54:48.792 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.6527 192.168.0.1:0/1948672403", 2023-06-10T18:54:48.792 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1948672403", 2023-06-10T18:54:48.792 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:54:48.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 62, 2023-06-10T18:54:48.793 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 101, 2023-06-10T18:54:48.793 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:54:48.793 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:54:48.794 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:54:48.794 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:54:48.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.798 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:54:48.799 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:48.816 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:54:48.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:48.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:48.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:48.885 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec ls subdir/childfile) 2023-06-10T18:54:48.943 INFO:teuthology.orchestra.run.smithi114.stdout:subdir/childfile 2023-06-10T18:54:48.944 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:54:48.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:54:48.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:54:49.037 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:54:49.037 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:54:49.037 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:54:49.058 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:54:49.059 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:54:49.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:49.060 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:54:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:54:49.117 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-06-10T18:54:49.117 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:54:49.118 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:54:49.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:54:49.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.277+0000 7fec8706c700 1 Processor -- start 2023-06-10T18:54:49.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.277+0000 7fec8706c700 1 -- start start 2023-06-10T18:54:49.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec8706c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 0x7fec8006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec8706c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec80069e70 0x7fec800ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec8706c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8006d290 con 0x7fec8006e0e0 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec8706c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec80071640 con 0x7fec80069e70 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec8706c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800aef00 con 0x7fec8006e4b0 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec80069e70 0x7fec800ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec7ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 0x7fec8006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec84e08700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fec8006e0e0 0x7fec800c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39300/0 (socket says 172.21.15.104:39300) 2023-06-10T18:54:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec84e08700 1 -- 172.21.15.104:0/1422877324 learned_addr learned my addr 172.21.15.104:0/1422877324 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 -- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 msgr2=0x7fec8006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 0x7fec8006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 -- 172.21.15.104:0/1422877324 >> v1:172.21.15.114:6789/0 conn(0x7fec8006e0e0 legacy=0x7fec800c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 -- 172.21.15.104:0/1422877324 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800ca2d0 con 0x7fec80069e70 2023-06-10T18:54:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec7f7fe700 1 -- 172.21.15.104:0/1422877324 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446775780 0 0) 0x7fec8006d290 con 0x7fec8006e0e0 2023-06-10T18:54:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec7ffff700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 0x7fec8006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.278+0000 7fec85609700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800ce9f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fec78009d90 tx=0x7fec7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75e75216f3e140d5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec7f7fe700 1 -- 172.21.15.104:0/1422877324 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800e070 con 0x7fec80069e70 2023-06-10T18:54:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec7f7fe700 1 -- 172.21.15.104:0/1422877324 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7800bb20 con 0x7fec80069e70 2023-06-10T18:54:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec7f7fe700 1 -- 172.21.15.104:0/1422877324 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78012430 con 0x7fec80069e70 2023-06-10T18:54:49.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 -- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 msgr2=0x7fec800ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800ce9f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fec78009d90 tx=0x7fec7800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:49.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 -- 172.21.15.104:0/1422877324 shutdown_connections 2023-06-10T18:54:49.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800ce9f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 --2- 172.21.15.104:0/1422877324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e4b0 0x7fec8006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 -- 172.21.15.104:0/1422877324 >> 172.21.15.104:0/1422877324 conn(0x7fec8006b8a0 msgr2=0x7fec800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:49.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 -- 172.21.15.104:0/1422877324 shutdown_connections 2023-06-10T18:54:49.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.279+0000 7fec8706c700 1 -- 172.21.15.104:0/1422877324 wait complete. 2023-06-10T18:54:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.280+0000 7fec8706c700 1 Processor -- start 2023-06-10T18:54:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.280+0000 7fec8706c700 1 -- start start 2023-06-10T18:54:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e0e0 0x7fec801584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800b3cc0 con 0x7fec8006e4b0 2023-06-10T18:54:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801f1390 con 0x7fec80069e70 2023-06-10T18:54:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec8706c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800b39e0 con 0x7fec8006e0e0 2023-06-10T18:54:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec85609700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec85609700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35636/0 (socket says 172.21.15.104:35636) 2023-06-10T18:54:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec85609700 1 -- 172.21.15.104:0/3389181719 learned_addr learned my addr 172.21.15.104:0/3389181719 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec84e08700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.281+0000 7fec7ffff700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e0e0 0x7fec801584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 -- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e0e0 msgr2=0x7fec801584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e0e0 0x7fec801584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 -- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 msgr2=0x7fec800afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 -- 172.21.15.104:0/3389181719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0094a0 con 0x7fec8006e4b0 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec85609700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fec8006cc20 tx=0x7fec78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78012430 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78020070 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec78009a70 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78004280 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.282+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec800af4a0 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.283+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec800ccef0 con 0x7fec8006e4b0 2023-06-10T18:54:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.283+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fec780043e0 con 0x7fec8006e4b0 2023-06-10T18:54:49.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.284+0000 7fec7d7fa700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 0x7fec6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.284+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fec78022070 con 0x7fec8006e4b0 2023-06-10T18:54:49.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.284+0000 7fec84e08700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 0x7fec6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.284+0000 7fec84e08700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 0x7fec6406d190 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec74000fb0 tx=0x7fec74007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.288+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fec780565d0 con 0x7fec8006e4b0 2023-06-10T18:54:49.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.476+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec80074050 con 0x7fec8006e4b0 2023-06-10T18:54:49.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.477+0000 7fec7d7fa700 1 -- 172.21.15.104:0/3389181719 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+18735 (secure 0 0 0) 0x7fec78059bf0 con 0x7fec8006e4b0 2023-06-10T18:54:49.483 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:49.483 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":101,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:54:44.720394+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/413723751":"2023-06-10T19:54:43.986209+0000","172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:54:49.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 msgr2=0x7fec6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 0x7fec6406d190 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec74000fb0 tx=0x7fec74007ec0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 msgr2=0x7fec800b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fec8006cc20 tx=0x7fec78004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 shutdown_connections 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec6406ace0 0x7fec6406d190 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec80069e70 0x7fec800afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec8006e4b0 0x7fec800b3150 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 --2- 172.21.15.104:0/3389181719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec8006e0e0 0x7fec801584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 >> 172.21.15.104:0/3389181719 conn(0x7fec8006b8a0 msgr2=0x7fec800c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:49.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 shutdown_connections 2023-06-10T18:54:49.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.479+0000 7fec8706c700 1 -- 172.21.15.104:0/3389181719 wait complete. 2023-06-10T18:54:49.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:54:49.498 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:54:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.615+0000 7f3eafa50700 1 Processor -- start 2023-06-10T18:54:49.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.615+0000 7f3eafa50700 1 -- start start 2023-06-10T18:54:49.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea815c6f0 0x7f3ea80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ea806ea70 0x7f3ea80b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3ea80aec00 0x7f3ea80ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea80ae7e0 con 0x7f3ea806ea70 2023-06-10T18:54:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea806df90 con 0x7f3ea80aec00 2023-06-10T18:54:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eafa50700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806e270 con 0x7f3ea815c6f0 2023-06-10T18:54:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eacfeb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ea806ea70 0x7f3ea80b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eacfeb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ea806ea70 0x7f3ea80b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35640/0 (socket says 172.21.15.104:35640) 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.616+0000 7f3eacfeb700 1 -- 172.21.15.104:0/3804748605 learned_addr learned my addr 172.21.15.104:0/3804748605 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eadfed700 1 --2- 172.21.15.104:0/3804748605 >> v2:172.21.15.114:3300/0 conn(0x7f3ea80aec00 0x7f3ea80ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3ead7ec700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea815c6f0 0x7f3ea80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 -- 172.21.15.104:0/3804748605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea815c6f0 msgr2=0x7f3ea80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea815c6f0 0x7f3ea80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 -- 172.21.15.104:0/3804748605 >> v2:172.21.15.114:3300/0 conn(0x7f3ea80aec00 msgr2=0x7f3ea80ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/3804748605 >> v2:172.21.15.114:3300/0 conn(0x7f3ea80aec00 0x7f3ea80ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 -- 172.21.15.104:0/3804748605 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea806e550 con 0x7f3ea806ea70 2023-06-10T18:54:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea806ea70 0x7f3ea80b15b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8072040 tx=0x7f3ea4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188d099cc4b07938 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3e9ffff700 1 -- 172.21.15.104:0/3804748605 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400b870 con 0x7f3ea806ea70 2023-06-10T18:54:49.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3e9ffff700 1 -- 172.21.15.104:0/3804748605 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea400b9d0 con 0x7f3ea806ea70 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.617+0000 7f3e9ffff700 1 -- 172.21.15.104:0/3804748605 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400bd60 con 0x7f3ea806ea70 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 -- 172.21.15.104:0/3804748605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea806ea70 msgr2=0x7f3ea80b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea806ea70 0x7f3ea80b15b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8072040 tx=0x7f3ea4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 -- 172.21.15.104:0/3804748605 shutdown_connections 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 --2- 172.21.15.104:0/3804748605 >> v2:172.21.15.114:3300/0 conn(0x7f3ea80aec00 0x7f3ea80ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea815c6f0 0x7f3ea80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 --2- 172.21.15.104:0/3804748605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea806ea70 0x7f3ea80b15b0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 -- 172.21.15.104:0/3804748605 >> 172.21.15.104:0/3804748605 conn(0x7f3ea806b8a0 msgr2=0x7f3ea80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 -- 172.21.15.104:0/3804748605 shutdown_connections 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.618+0000 7f3eafa50700 1 -- 172.21.15.104:0/3804748605 wait complete. 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.619+0000 7f3eafa50700 1 Processor -- start 2023-06-10T18:54:49.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.619+0000 7f3eafa50700 1 -- start start 2023-06-10T18:54:49.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea806ea70 0x7f3ea8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ea815c6f0 0x7f3ea8154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80c5fc0 con 0x7f3ea80aec00 2023-06-10T18:54:49.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80c8800 con 0x7f3ea815c6f0 2023-06-10T18:54:49.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80c8a50 con 0x7f3ea806ea70 2023-06-10T18:54:49.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35648/0 (socket says 172.21.15.104:35648) 2023-06-10T18:54:49.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 -- 172.21.15.104:0/734753137 learned_addr learned my addr 172.21.15.104:0/734753137 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:49.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3ead7ec700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea806ea70 0x7f3ea8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eadfed700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ea815c6f0 0x7f3ea8154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 -- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea806ea70 msgr2=0x7f3ea8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea806ea70 0x7f3ea8155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 -- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ea815c6f0 msgr2=0x7f3ea8154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ea815c6f0 0x7f3ea8154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 -- 172.21.15.104:0/734753137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4009550 con 0x7f3ea80aec00 2023-06-10T18:54:49.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eacfeb700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4000c00 tx=0x7f3ea4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4004480 con 0x7f3ea80aec00 2023-06-10T18:54:49.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea40045e0 con 0x7f3ea80aec00 2023-06-10T18:54:49.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea80c8d30 con 0x7f3ea80aec00 2023-06-10T18:54:49.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.620+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40038e0 con 0x7f3ea80aec00 2023-06-10T18:54:49.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.621+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea80c92f0 con 0x7f3ea80aec00 2023-06-10T18:54:49.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.622+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea8066f10 con 0x7f3ea80aec00 2023-06-10T18:54:49.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.622+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3ea4021070 con 0x7f3ea80aec00 2023-06-10T18:54:49.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.622+0000 7f3e9dffb700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 0x7f3e8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.622+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f3ea408ad40 con 0x7f3ea80aec00 2023-06-10T18:54:49.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.623+0000 7f3ead7ec700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 0x7f3e8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.623+0000 7f3ead7ec700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 0x7f3e8c06d190 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80d04d0 tx=0x7f3e98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.627+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3ea4055d40 con 0x7f3ea80aec00 2023-06-10T18:54:49.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.815+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ea8074050 con 0x7f3ea80aec00 2023-06-10T18:54:49.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.816+0000 7f3e9dffb700 1 -- 172.21.15.104:0/734753137 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ea40150a0 con 0x7f3ea80aec00 2023-06-10T18:54:49.822 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:54:49.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 msgr2=0x7f3e8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 0x7f3e8c06d190 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80d04d0 tx=0x7f3e98008040 comp rx=0 tx=0).stop 2023-06-10T18:54:49.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 msgr2=0x7f3ea81f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4000c00 tx=0x7f3ea4004750 comp rx=0 tx=0).stop 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 shutdown_connections 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e8c06ace0 0x7f3e8c06d190 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ea815c6f0 0x7f3ea8154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ea806ea70 0x7f3ea8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 --2- 172.21.15.104:0/734753137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ea80aec00 0x7f3ea81f1a40 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 >> 172.21.15.104:0/734753137 conn(0x7f3ea806b8a0 msgr2=0x7f3ea80b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:49.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 shutdown_connections 2023-06-10T18:54:49.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.818+0000 7f3eafa50700 1 -- 172.21.15.104:0/734753137 wait complete. 2023-06-10T18:54:49.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:54:49.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.953+0000 7f98a854e700 1 Processor -- start 2023-06-10T18:54:49.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.953+0000 7f98a854e700 1 -- start start 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a854e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 0x7f98a006d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a854e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00cab60 con 0x7f98a006a880 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a854e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a015ec40 con 0x7f98a006d450 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a854e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00d0540 con 0x7f98a00ae950 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a5ae9700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f98a006a880 0x7f98a015de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33482/0 (socket says 172.21.15.104:33482) 2023-06-10T18:54:49.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a5ae9700 1 -- 172.21.15.104:0/2538282732 learned_addr learned my addr 172.21.15.104:0/2538282732 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.954+0000 7f98a62ea700 1 --2- 172.21.15.104:0/2538282732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 0x7f98a006d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879116848 0 0) 0x7f98a00cab60 con 0x7f98a006a880 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c0035b0 con 0x7f98a006a880 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721551007 0 0) 0x7f98a015ec40 con 0x7f98a006d450 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a00cab60 con 0x7f98a006d450 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a62ea700 1 -- 172.21.15.104:0/2538282732 >> v1:172.21.15.114:6789/0 conn(0x7f98a006d450 legacy=0x7f98a015e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a62ea700 1 -- 172.21.15.104:0/2538282732 >> v1:172.21.15.104:6789/0 conn(0x7f98a006a880 legacy=0x7f98a015de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a62ea700 1 -- 172.21.15.104:0/2538282732 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00cc5e0 con 0x7f98a00ae950 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a62ea700 1 --2- 172.21.15.104:0/2538282732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 0x7f98a006d9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9890009850 tx=0x7f9890009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31cde0ddbf95508f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890005e30 con 0x7f98a00ae950 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890006030 con 0x7f98a00ae950 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.955+0000 7f98a52e8700 1 -- 172.21.15.104:0/2538282732 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900063e0 con 0x7f98a00ae950 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 -- 172.21.15.104:0/2538282732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 msgr2=0x7f98a006d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 --2- 172.21.15.104:0/2538282732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 0x7f98a006d9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9890009850 tx=0x7f9890009b60 comp rx=0 tx=0).stop 2023-06-10T18:54:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 -- 172.21.15.104:0/2538282732 shutdown_connections 2023-06-10T18:54:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 --2- 172.21.15.104:0/2538282732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a00ae950 0x7f98a006d9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 -- 172.21.15.104:0/2538282732 >> 172.21.15.104:0/2538282732 conn(0x7f98a006b8a0 msgr2=0x7f98a00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 -- 172.21.15.104:0/2538282732 shutdown_connections 2023-06-10T18:54:49.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.956+0000 7f98a854e700 1 -- 172.21.15.104:0/2538282732 wait complete. 2023-06-10T18:54:49.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.957+0000 7f98a854e700 1 Processor -- start 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.957+0000 7f98a854e700 1 -- start start 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a006d450 0x7f98a00c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 0x7f98a015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c71a0 con 0x7f98a006a880 2023-06-10T18:54:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c7480 con 0x7f98a00ae950 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c7760 con 0x7f98a006d450 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35660/0 (socket says 172.21.15.104:35660) 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 -- 172.21.15.104:0/2150312413 learned_addr learned my addr 172.21.15.104:0/2150312413 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a5ae9700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a006d450 0x7f98a00c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a6aeb700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 0x7f98a015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 -- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a006d450 msgr2=0x7f98a00c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a006d450 0x7f98a00c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 -- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 msgr2=0x7f98a015c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 0x7f98a015c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 -- 172.21.15.104:0/2150312413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890009430 con 0x7f98a006a880 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a6aeb700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 0x7f98a015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a62ea700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f9890013040 tx=0x7f9890004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890006810 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890020070 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a00c7a40 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890025650 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.958+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a00c89a0 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.960+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a0074050 con 0x7f98a006a880 2023-06-10T18:54:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.960+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9890022030 con 0x7f98a006a880 2023-06-10T18:54:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.960+0000 7f9896ffd700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 0x7f988406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.960+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f989008aaa0 con 0x7f98a006a880 2023-06-10T18:54:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.960+0000 7f98a5ae9700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 0x7f988406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.961+0000 7f98a5ae9700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 0x7f988406d190 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f98a01f2d50 tx=0x7f989c007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:49.964+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9890055a20 con 0x7f98a006a880 2023-06-10T18:54:50.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.151+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f98a0066f10 con 0x7f98a006a880 2023-06-10T18:54:50.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.151+0000 7f9896ffd700 1 -- 172.21.15.104:0/2150312413 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9890059040 con 0x7f98a006a880 2023-06-10T18:54:50.157 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:54:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 msgr2=0x7f988406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 0x7f988406d190 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f98a01f2d50 tx=0x7f989c007e00 comp rx=0 tx=0).stop 2023-06-10T18:54:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 msgr2=0x7f98a0152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f9890013040 tx=0x7f9890004750 comp rx=0 tx=0).stop 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 shutdown_connections 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f988406ace0 0x7f988406d190 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98a00ae950 0x7f98a015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98a006a880 0x7f98a0152c70 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 --2- 172.21.15.104:0/2150312413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98a006d450 0x7f98a00c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 >> 172.21.15.104:0/2150312413 conn(0x7f98a006b8a0 msgr2=0x7f98a0159720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 shutdown_connections 2023-06-10T18:54:50.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.153+0000 7f98a854e700 1 -- 172.21.15.104:0/2150312413 wait complete. 2023-06-10T18:54:50.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:54:50.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.287+0000 7efc16ed5700 1 Processor -- start 2023-06-10T18:54:50.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.287+0000 7efc16ed5700 1 -- start start 2023-06-10T18:54:50.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc16ed5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc100b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc16ed5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efc1006cde0 0x7efc100ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc16ed5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1006e250 con 0x7efc1015c580 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc16ed5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1015c950 con 0x7efc1006cde0 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc16ed5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100b1e80 con 0x7efc1006e6e0 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efc1006cde0 0x7efc100ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc14c71700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efc1015c580 0x7efc100b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33496/0 (socket says 172.21.15.104:33496) 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efc1006cde0 0x7efc100ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35672/0 (socket says 172.21.15.104:35672) 2023-06-10T18:54:50.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc14c71700 1 -- 172.21.15.104:0/3302671703 learned_addr learned my addr 172.21.15.104:0/3302671703 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc0ffff700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc100b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 -- 172.21.15.104:0/3302671703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 msgr2=0x7efc100b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc100b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 -- 172.21.15.104:0/3302671703 >> v1:172.21.15.104:6789/0 conn(0x7efc1015c580 legacy=0x7efc100b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 -- 172.21.15.104:0/3302671703 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1006d920 con 0x7efc1006cde0 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.288+0000 7efc15472700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1006cde0 0x7efc100ce0d0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7efc08009d90 tx=0x7efc0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0337b9c874123e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc0f7fe700 1 -- 172.21.15.104:0/3302671703 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800e070 con 0x7efc1006cde0 2023-06-10T18:54:50.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc0f7fe700 1 -- 172.21.15.104:0/3302671703 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0800bbd0 con 0x7efc1006cde0 2023-06-10T18:54:50.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc0f7fe700 1 -- 172.21.15.104:0/3302671703 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08012430 con 0x7efc1006cde0 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 -- 172.21.15.104:0/3302671703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1006cde0 msgr2=0x7efc100ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1006cde0 0x7efc100ce0d0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7efc08009d90 tx=0x7efc0800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 -- 172.21.15.104:0/3302671703 shutdown_connections 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1006cde0 0x7efc100ce0d0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3302671703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc100b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 -- 172.21.15.104:0/3302671703 >> 172.21.15.104:0/3302671703 conn(0x7efc1006b8a0 msgr2=0x7efc100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 -- 172.21.15.104:0/3302671703 shutdown_connections 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.289+0000 7efc16ed5700 1 -- 172.21.15.104:0/3302671703 wait complete. 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.291+0000 7efc16ed5700 1 Processor -- start 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.291+0000 7efc16ed5700 1 -- start start 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc1006cde0 0x7efc100c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc1015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c57b0 con 0x7efc1015c580 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c4b30 con 0x7efc1006cde0 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc16ed5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c4e10 con 0x7efc1006e6e0 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35678/0 (socket says 172.21.15.104:35678) 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 -- 172.21.15.104:0/3733450678 learned_addr learned my addr 172.21.15.104:0/3733450678 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc0ffff700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc1015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc14c71700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc1006cde0 0x7efc100c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 -- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 msgr2=0x7efc1015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc1015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 -- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc1006cde0 msgr2=0x7efc100c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc1006cde0 0x7efc100c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 -- 172.21.15.104:0/3733450678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc08009a70 con 0x7efc1015c580 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.292+0000 7efc15472700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7efc08014040 tx=0x7efc08004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.293+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08003ea0 con 0x7efc1015c580 2023-06-10T18:54:50.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.293+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08020070 con 0x7efc1015c580 2023-06-10T18:54:50.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.293+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc100c50f0 con 0x7efc1015c580 2023-06-10T18:54:50.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.293+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08004280 con 0x7efc1015c580 2023-06-10T18:54:50.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.293+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc100c4770 con 0x7efc1015c580 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.294+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc100c4900 con 0x7efc1015c580 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.294+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc080043e0 con 0x7efc1015c580 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.294+0000 7efc0d7fa700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 0x7efbf406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.294+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7efc08022070 con 0x7efc1015c580 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.295+0000 7efc14c71700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 0x7efbf406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.295+0000 7efc14c71700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 0x7efbf406d190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efc040010e0 tx=0x7efc04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.299+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc080567f0 con 0x7efc1015c580 2023-06-10T18:54:50.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.483+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc100c58f0 con 0x7efc1015c580 2023-06-10T18:54:50.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.483+0000 7efc0d7fa700 1 -- 172.21.15.104:0/3733450678 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc100c58f0 con 0x7efc1015c580 2023-06-10T18:54:50.489 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:54:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 msgr2=0x7efbf406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 0x7efbf406d190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efc040010e0 tx=0x7efc04009040 comp rx=0 tx=0).stop 2023-06-10T18:54:50.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 msgr2=0x7efc1014e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7efc08014040 tx=0x7efc08004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 shutdown_connections 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efbf406ace0 0x7efbf406d190 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc1006cde0 0x7efc100c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc1015c580 0x7efc1014e670 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 --2- 172.21.15.104:0/3733450678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc1006e6e0 0x7efc1015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 >> 172.21.15.104:0/3733450678 conn(0x7efc1006b8a0 msgr2=0x7efc100ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 shutdown_connections 2023-06-10T18:54:50.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.485+0000 7efc16ed5700 1 -- 172.21.15.104:0/3733450678 wait complete. 2023-06-10T18:54:50.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:54:50.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.620+0000 7efcfe672700 1 Processor -- start 2023-06-10T18:54:50.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.620+0000 7efcfe672700 1 -- start start 2023-06-10T18:54:50.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcfe672700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcf815c580 0x7efcf80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcfe672700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf806e250 con 0x7efcf806e6e0 2023-06-10T18:54:50.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcfe672700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf815c950 con 0x7efcf806cde0 2023-06-10T18:54:50.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcfe672700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf80b1e80 con 0x7efcf815c580 2023-06-10T18:54:50.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcf77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efcf806e6e0 0x7efcf80b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33510/0 (socket says 172.21.15.104:33510) 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcf77fe700 1 -- 172.21.15.104:0/2921124374 learned_addr learned my addr 172.21.15.104:0/2921124374 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813067794 0 0) 0x7efcf806e250 con 0x7efcf806e6e0 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce00035b0 con 0x7efcf806e6e0 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.621+0000 7efcf7fff700 1 --2- 172.21.15.104:0/2921124374 >> v2:172.21.15.114:3300/0 conn(0x7efcf815c580 0x7efcf80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464609756 0 0) 0x7efce00035b0 con 0x7efcf806e6e0 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 >> v2:172.21.15.114:3300/0 conn(0x7efcf815c580 msgr2=0x7efcf80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:50.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 --2- 172.21.15.104:0/2921124374 >> v2:172.21.15.114:3300/0 conn(0x7efcf815c580 0x7efcf80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 >> v1:172.21.15.114:6789/0 conn(0x7efcf806cde0 legacy=0x7efcf80ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf806d920 con 0x7efcf806e6e0 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7efce8001670 con 0x7efcf806e6e0 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 >> v1:172.21.15.104:6789/0 conn(0x7efcf806e6e0 legacy=0x7efcf80b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efce0003a00 0x7efce0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efce0004850 0x7efce0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efce000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80b1e80 con 0x7efce0005200 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce000c080 con 0x7efce0004850 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce000c240 con 0x7efce0003a00 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf7fff700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efce0003a00 0x7efce0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efce000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf77fe700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efce0004850 0x7efce0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 -- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efce0003a00 msgr2=0x7efce0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:50.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efce0003a00 0x7efce0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 -- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efce0004850 msgr2=0x7efce0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efce0004850 0x7efce0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 -- 172.21.15.104:0/2921124374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00035b0 con 0x7efce0005200 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 -- 172.21.15.104:0/2921124374 reap_dead start 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcfcc0f700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efce000ba10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c190 tx=0x7efce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26caf3a31191f91 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.622+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0001db0 con 0x7efce0005200 2023-06-10T18:54:50.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf000ccf0 con 0x7efce0005200 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcf6ffd700 1 -- 172.21.15.104:0/2921124374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0003c40 con 0x7efce0005200 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 -- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 msgr2=0x7efce000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efce000ba10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c190 tx=0x7efce0005880 comp rx=0 tx=0).stop 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 -- 172.21.15.104:0/2921124374 shutdown_connections 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 --2- 172.21.15.104:0/2921124374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efce000ba10 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 -- 172.21.15.104:0/2921124374 >> 172.21.15.104:0/2921124374 conn(0x7efcf806b8a0 msgr2=0x7efcf80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 -- 172.21.15.104:0/2921124374 shutdown_connections 2023-06-10T18:54:50.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.623+0000 7efcfe672700 1 -- 172.21.15.104:0/2921124374 wait complete. 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.624+0000 7efcfe672700 1 Processor -- start 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.624+0000 7efcfe672700 1 -- start start 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efcf8158f90 0x7efcf80af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf814fc20 0x7efcf80afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80c67d0 con 0x7efce0005200 2023-06-10T18:54:50.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80c8b10 con 0x7efcf814fc20 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfe672700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80c8830 con 0x7efcf8158f90 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcf7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcf7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35692/0 (socket says 172.21.15.104:35692) 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcf7fff700 1 -- 172.21.15.104:0/1023010099 learned_addr learned my addr 172.21.15.104:0/1023010099 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcf77fe700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efcf8158f90 0x7efcf80af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.625+0000 7efcfcc0f700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf814fc20 0x7efcf80afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 -- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efcf8158f90 msgr2=0x7efcf80af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efcf8158f90 0x7efcf80af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 -- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf814fc20 msgr2=0x7efcf80afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf814fc20 0x7efcf80afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 -- 172.21.15.104:0/1023010099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80c6a70 con 0x7efce0005200 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf7fff700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7efcec009fd0 tx=0x7efcec011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec009610 con 0x7efce0005200 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec005c30 con 0x7efce0005200 2023-06-10T18:54:50.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf80c6d50 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec00f730 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.626+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf80c85e0 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.627+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf80c6630 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.627+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efcec00f890 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.628+0000 7efcf4ff9700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 0x7efce406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.628+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7efcec089eb0 con 0x7efce0005200 2023-06-10T18:54:50.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.628+0000 7efcf77fe700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 0x7efce406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.628+0000 7efcf77fe700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 0x7efce406d1c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efcf8159380 tx=0x7efce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.632+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efcf80c6630 con 0x7efce0005200 2023-06-10T18:54:50.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.819+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efcf80c6630 con 0x7efce0005200 2023-06-10T18:54:50.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.819+0000 7efcf4ff9700 1 -- 172.21.15.104:0/1023010099 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7efcf80c6630 con 0x7efce0005200 2023-06-10T18:54:50.825 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:54:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 msgr2=0x7efce406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 0x7efce406d1c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efcf8159380 tx=0x7efce8009040 comp rx=0 tx=0).stop 2023-06-10T18:54:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 msgr2=0x7efcf80c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7efcec009fd0 tx=0x7efcec011d30 comp rx=0 tx=0).stop 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 shutdown_connections 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efce406ad10 0x7efce406d1c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf814fc20 0x7efcf80afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efce0005200 0x7efcf80c7ec0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 --2- 172.21.15.104:0/1023010099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efcf8158f90 0x7efcf80af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 >> 172.21.15.104:0/1023010099 conn(0x7efcf806b8a0 msgr2=0x7efcf80cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 shutdown_connections 2023-06-10T18:54:50.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.821+0000 7efcfe672700 1 -- 172.21.15.104:0/1023010099 wait complete. 2023-06-10T18:54:50.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:54:50.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.953+0000 7f80eda53700 1 Processor -- start 2023-06-10T18:54:50.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.953+0000 7f80eda53700 1 -- start start 2023-06-10T18:54:50.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80eda53700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e80caea0 0x7f80e80cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80eda53700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e80c9270 0x7f80e80c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80eda53700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e80c8de0 con 0x7f80e8158e50 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80eda53700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e80ccc90 con 0x7f80e80caea0 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80eda53700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80cdd80 con 0x7f80e80c9270 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e80caea0 0x7f80e80cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80e80caea0 0x7f80e80cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35702/0 (socket says 172.21.15.104:35702) 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 -- 172.21.15.104:0/925942405 learned_addr learned my addr 172.21.15.104:0/925942405 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:50.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e77fe700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e80c9270 0x7f80e80c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 -- 172.21.15.104:0/925942405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e80c9270 msgr2=0x7f80e80c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e80c9270 0x7f80e80c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 -- 172.21.15.104:0/925942405 >> v1:172.21.15.114:6789/0 conn(0x7f80e8158e50 legacy=0x7f80e80cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 -- 172.21.15.104:0/925942405 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e80ccf30 con 0x7f80e80caea0 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e67fc700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80caea0 0x7f80e80cb2f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bb90 tx=0x7f80d800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b0228fe89bbd93 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.954+0000 7f80e5ffb700 1 -- 172.21.15.104:0/925942405 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d800dd50 con 0x7f80e80caea0 2023-06-10T18:54:50.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80e5ffb700 1 -- 172.21.15.104:0/925942405 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8010470 con 0x7f80e80caea0 2023-06-10T18:54:50.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80e5ffb700 1 -- 172.21.15.104:0/925942405 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8010800 con 0x7f80e80caea0 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 -- 172.21.15.104:0/925942405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80caea0 msgr2=0x7f80e80cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80caea0 0x7f80e80cb2f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bb90 tx=0x7f80d800bea0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 -- 172.21.15.104:0/925942405 shutdown_connections 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e80c9270 0x7f80e80c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 --2- 172.21.15.104:0/925942405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80caea0 0x7f80e80cb2f0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 -- 172.21.15.104:0/925942405 >> 172.21.15.104:0/925942405 conn(0x7f80e806ab00 msgr2=0x7f80e80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 -- 172.21.15.104:0/925942405 shutdown_connections 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.955+0000 7f80eda53700 1 -- 172.21.15.104:0/925942405 wait complete. 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.956+0000 7f80eda53700 1 Processor -- start 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.956+0000 7f80eda53700 1 -- start start 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 0x7f80e8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e8158e50 0x7f80e80718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8155a20 con 0x7f80e80c9270 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81ef0d0 con 0x7f80e80caea0 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80eda53700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81ee4c0 con 0x7f80e8158e50 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80e6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80e6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35706/0 (socket says 172.21.15.104:35706) 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80e6ffd700 1 -- 172.21.15.104:0/1689978523 learned_addr learned my addr 172.21.15.104:0/1689978523 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80e67fc700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 0x7f80e8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.957+0000 7f80e77fe700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e8158e50 0x7f80e80718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 -- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e8158e50 msgr2=0x7f80e80718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e8158e50 0x7f80e80718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 -- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 msgr2=0x7f80e8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 0x7f80e8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 -- 172.21.15.104:0/1689978523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d800b7f0 con 0x7f80e80c9270 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e67fc700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 0x7f80e8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:50.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80e6ffd700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c430 tx=0x7f80dc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0052d0 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc005450 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e81ee7a0 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc006aa0 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.958+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81eed60 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.959+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8074050 con 0x7f80e80c9270 2023-06-10T18:54:50.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.959+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f80dc003680 con 0x7f80e80c9270 2023-06-10T18:54:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.960+0000 7f80cffff700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 0x7f80d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.960+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f80dc0888f0 con 0x7f80e80c9270 2023-06-10T18:54:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.960+0000 7f80e67fc700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 0x7f80d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:50.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.960+0000 7f80e67fc700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 0x7f80d406d190 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bee0 tx=0x7f80d8013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:50.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:50.964+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f80dc0537e0 con 0x7f80e80c9270 2023-06-10T18:54:51.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.165+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f80e80b0ab0 con 0x7f80e80c9270 2023-06-10T18:54:51.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.165+0000 7f80cffff700 1 -- 172.21.15.104:0/1689978523 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 3 entries v101) v1 ==== 80+0+180 (secure 0 0 0) 0x7f80dc056e00 con 0x7f80e80c9270 2023-06-10T18:54:51.171 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/413723751 2023-06-10T19:54:43.986209+0000 2023-06-10T18:54:51.172 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6837/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:54:51.172 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6835/3243355534 2023-06-11T18:53:51.686929+0000 2023-06-10T18:54:51.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 msgr2=0x7f80d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 0x7f80d406d190 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f80d800bee0 tx=0x7f80d8013000 comp rx=0 tx=0).stop 2023-06-10T18:54:51.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 msgr2=0x7f80e8152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00c430 tx=0x7f80dc00c7f0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 shutdown_connections 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80e8158e50 0x7f80e80718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80e80c9270 0x7f80e8152bf0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f80d406ace0 0x7f80d406d190 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 --2- 172.21.15.104:0/1689978523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80e80caea0 0x7f80e8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 >> 172.21.15.104:0/1689978523 conn(0x7f80e806ab00 msgr2=0x7f80e80ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:51.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 shutdown_connections 2023-06-10T18:54:51.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.167+0000 7f80eda53700 1 -- 172.21.15.104:0/1689978523 wait complete. 2023-06-10T18:54:51.175 INFO:teuthology.orchestra.run.smithi104.stderr:listed 3 entries 2023-06-10T18:54:51.185 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:54:51.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:51.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:54:51.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[113355]: fuse finished with error 0 and tester_r 0 2023-06-10T18:54:51.229 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:54:51.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:51.230 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:54:51.312 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:54:51.313 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:54:51.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:54:51.332 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:54:51.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:51.333 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:54:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:54:51.389 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:54:51.390 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:54:51.390 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:54:51.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:54:51.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:54:51.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:54:51.410 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:54:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:54:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:54:51.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:51.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.584+0000 7fd074153700 1 Processor -- start 2023-06-10T18:54:51.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.584+0000 7fd074153700 1 -- start start 2023-06-10T18:54:51.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd074153700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd06c06d150 0x7fd06c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd074153700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c0cccd0 con 0x7fd06c158c50 2023-06-10T18:54:51.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd074153700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c06a010 con 0x7fd06c06b180 2023-06-10T18:54:51.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd074153700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06c0ccfb0 con 0x7fd06c06d150 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd06c06d150 0x7fd06c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd071eef700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd06c158c50 0x7fd06c06ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33542/0 (socket says 172.21.15.104:33542) 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd06c06d150 0x7fd06c06aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35716/0 (socket says 172.21.15.104:35716) 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd071eef700 1 -- 172.21.15.104:0/3492060086 learned_addr learned my addr 172.21.15.104:0/3492060086 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 -- 172.21.15.104:0/3492060086 >> v1:172.21.15.114:6789/0 conn(0x7fd06c06b180 legacy=0x7fd06c0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 -- 172.21.15.104:0/3492060086 >> v1:172.21.15.104:6789/0 conn(0x7fd06c158c50 legacy=0x7fd06c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 -- 172.21.15.104:0/3492060086 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c06a2f0 con 0x7fd06c06d150 2023-06-10T18:54:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd070eed700 1 -- 172.21.15.104:0/3492060086 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210909115 0 0) 0x7fd06c0cccd0 con 0x7fd06c158c50 2023-06-10T18:54:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd0716ee700 1 --2- 172.21.15.104:0/3492060086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c06aaf0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009840 tx=0x7fd05c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1392967fabc488 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd070eed700 1 -- 172.21.15.104:0/3492060086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c011830 con 0x7fd06c06d150 2023-06-10T18:54:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd070eed700 1 -- 172.21.15.104:0/3492060086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c011990 con 0x7fd06c06d150 2023-06-10T18:54:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.585+0000 7fd070eed700 1 -- 172.21.15.104:0/3492060086 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c011d20 con 0x7fd06c06d150 2023-06-10T18:54:51.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 -- 172.21.15.104:0/3492060086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 msgr2=0x7fd06c06aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 --2- 172.21.15.104:0/3492060086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c06aaf0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009840 tx=0x7fd05c009b50 comp rx=0 tx=0).stop 2023-06-10T18:54:51.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 -- 172.21.15.104:0/3492060086 shutdown_connections 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 --2- 172.21.15.104:0/3492060086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c06aaf0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 -- 172.21.15.104:0/3492060086 >> 172.21.15.104:0/3492060086 conn(0x7fd06c0b1610 msgr2=0x7fd06c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 -- 172.21.15.104:0/3492060086 shutdown_connections 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.586+0000 7fd074153700 1 -- 172.21.15.104:0/3492060086 wait complete. 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.587+0000 7fd074153700 1 Processor -- start 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.587+0000 7fd074153700 1 -- start start 2023-06-10T18:54:51.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 0x7fd06c14fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c158c50 0x7fd06c157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c0c72e0 con 0x7fd06c06d150 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c14ff70 con 0x7fd06c06b180 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd074153700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c0c6760 con 0x7fd06c158c50 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35722/0 (socket says 172.21.15.104:35722) 2023-06-10T18:54:51.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 -- 172.21.15.104:0/3925537693 learned_addr learned my addr 172.21.15.104:0/3925537693 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd071eef700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 0x7fd06c14fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0726f0700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c158c50 0x7fd06c157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 -- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c158c50 msgr2=0x7fd06c157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c158c50 0x7fd06c157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 -- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 msgr2=0x7fd06c14fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 0x7fd06c14fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 -- 172.21.15.104:0/3925537693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0094a0 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd071eef700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 0x7fd06c14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.588+0000 7fd0716ee700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0090b0 tx=0x7fd05c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0036a0 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c01f070 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c0c6a40 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003a00 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c0c7000 con 0x7fd06c06d150 2023-06-10T18:54:51.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.589+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c0c7740 con 0x7fd06c06d150 2023-06-10T18:54:51.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.590+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd05c003b60 con 0x7fd06c06d150 2023-06-10T18:54:51.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.590+0000 7fd0627fc700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 0x7fd05006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.590+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fd05c021070 con 0x7fd06c06d150 2023-06-10T18:54:51.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.591+0000 7fd071eef700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 0x7fd05006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.591+0000 7fd071eef700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 0x7fd05006d190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd068000f60 tx=0x7fd068007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.595+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd06c0c7740 con 0x7fd06c06d150 2023-06-10T18:54:51.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.821+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd06c0c7740 con 0x7fd06c06d150 2023-06-10T18:54:51.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.822+0000 7fd0627fc700 1 -- 172.21.15.104:0/3925537693 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fd06c0c7740 con 0x7fd06c06d150 2023-06-10T18:54:51.828 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:51.828 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:54:46.733023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:51.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 msgr2=0x7fd05006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 0x7fd05006d190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd068000f60 tx=0x7fd068007e50 comp rx=0 tx=0).stop 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 msgr2=0x7fd06c0c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0090b0 tx=0x7fd05c004750 comp rx=0 tx=0).stop 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 shutdown_connections 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd05006ace0 0x7fd05006d190 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c06b180 0x7fd06c14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c158c50 0x7fd06c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 --2- 172.21.15.104:0/3925537693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c06d150 0x7fd06c0c5e60 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 >> 172.21.15.104:0/3925537693 conn(0x7fd06c0b1610 msgr2=0x7fd06c0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 shutdown_connections 2023-06-10T18:54:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.824+0000 7fd074153700 1 -- 172.21.15.104:0/3925537693 wait complete. 2023-06-10T18:54:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 62 2023-06-10T18:54:51.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:54:51.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.955+0000 7f00b88c5700 1 Processor -- start 2023-06-10T18:54:51.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.955+0000 7f00b88c5700 1 -- start start 2023-06-10T18:54:51.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b88c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f00b406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b88c5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b406d290 con 0x7f00b406e4b0 2023-06-10T18:54:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b88c5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40ca2d0 con 0x7f00b4069e70 2023-06-10T18:54:51.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b88c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4071640 con 0x7f00b406e0e0 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b1d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f00b406e4b0 0x7f00b40c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33556/0 (socket says 172.21.15.104:33556) 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b1d9b700 1 -- 172.21.15.104:0/1602640090 learned_addr learned my addr 172.21.15.104:0/1602640090 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b259c700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f00b406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746893854 0 0) 0x7f00b406d290 con 0x7f00b406e4b0 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c0035b0 con 0x7f00b406e4b0 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633871455 0 0) 0x7f009c0035b0 con 0x7f00b406e4b0 2023-06-10T18:54:51.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 msgr2=0x7f00b406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f00b406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 >> v1:172.21.15.114:6789/0 conn(0x7f00b4069e70 legacy=0x7f00b40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b406d4c0 con 0x7f00b406e4b0 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f00a8001670 con 0x7f00b406e4b0 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 >> v1:172.21.15.104:6789/0 conn(0x7f00b406e4b0 legacy=0x7f00b40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.956+0000 7f00b159a700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f009c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004060 0x7f009c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f009c004de0 0x7f009c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4071640 con 0x7f009c004de0 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00bcc0 con 0x7f009c004060 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00be80 con 0x7f00b406e0e0 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f009c004de0 0x7f009c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b259c700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f009c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b1d9b700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004060 0x7f009c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 -- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 msgr2=0x7f009c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b406e0e0 0x7f009c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 -- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004060 msgr2=0x7f009c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004060 0x7f009c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 -- 172.21.15.104:0/1602640090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0035b0 con 0x7f009c004de0 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b2d9d700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f009c004de0 0x7f009c005230 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00c190 tx=0x7f009c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a278eb5ad766d08 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac001db0 con 0x7f009c004de0 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ac00ccd0 con 0x7f009c004de0 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.957+0000 7f00b159a700 1 -- 172.21.15.104:0/1602640090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac003c40 con 0x7f009c004de0 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 -- 172.21.15.104:0/1602640090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f009c004de0 msgr2=0x7f009c005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 --2- 172.21.15.104:0/1602640090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f009c004de0 0x7f009c005230 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00c190 tx=0x7f009c005880 comp rx=0 tx=0).stop 2023-06-10T18:54:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b2d9d700 1 -- 172.21.15.104:0/1602640090 reap_dead start 2023-06-10T18:54:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 -- 172.21.15.104:0/1602640090 shutdown_connections 2023-06-10T18:54:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 -- 172.21.15.104:0/1602640090 >> 172.21.15.104:0/1602640090 conn(0x7f00b406b8a0 msgr2=0x7f00b40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 -- 172.21.15.104:0/1602640090 shutdown_connections 2023-06-10T18:54:51.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.958+0000 7f00b88c5700 1 -- 172.21.15.104:0/1602640090 wait complete. 2023-06-10T18:54:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.959+0000 7f00b88c5700 1 Processor -- start 2023-06-10T18:54:51.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.959+0000 7f00b88c5700 1 -- start start 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004de0 0x7f00b4151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b40c8890 0x7f00b4159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c69b0 con 0x7f00b41e4b70 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4151640 con 0x7f009c004de0 2023-06-10T18:54:51.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b88c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c8da0 con 0x7f00b40c8890 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35740/0 (socket says 172.21.15.104:35740) 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 -- 172.21.15.104:0/2267563213 learned_addr learned my addr 172.21.15.104:0/2267563213 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 -- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b40c8890 msgr2=0x7f00b4159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b40c8890 0x7f00b4159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 -- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004de0 msgr2=0x7f00b4151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004de0 0x7f00b4151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 -- 172.21.15.104:0/2267563213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40c9080 con 0x7f00b41e4b70 2023-06-10T18:54:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00b1d9b700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f00a80017a0 tx=0x7f00a8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8007800 con 0x7f00b41e4b70 2023-06-10T18:54:51.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.960+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a800f850 con 0x7f00b41e4b70 2023-06-10T18:54:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.961+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b40c9360 con 0x7f00b41e4b70 2023-06-10T18:54:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.961+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a800fc00 con 0x7f00b41e4b70 2023-06-10T18:54:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.961+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b41e2830 con 0x7f00b41e4b70 2023-06-10T18:54:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.961+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b41e29c0 con 0x7f00b41e4b70 2023-06-10T18:54:51.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.962+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f00a800fd60 con 0x7f00b41e4b70 2023-06-10T18:54:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.962+0000 7f00a2ffd700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 0x7f009406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.962+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f00a80896a0 con 0x7f00b41e4b70 2023-06-10T18:54:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.963+0000 7f00b259c700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 0x7f009406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.963+0000 7f00b259c700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 0x7f009406d190 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f00a4006ac0 tx=0x7f00a4006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:51.967+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f00a8054590 con 0x7f00b41e4b70 2023-06-10T18:54:52.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.154+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00b4074050 con 0x7f00b41e4b70 2023-06-10T18:54:52.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.156+0000 7f00a2ffd700 1 -- 172.21.15.104:0/2267563213 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+18735 (secure 0 0 0) 0x7f00a8057bb0 con 0x7f00b41e4b70 2023-06-10T18:54:52.162 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:52.162 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":101,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:54:44.720394+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/413723751":"2023-06-10T19:54:43.986209+0000","172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:54:52.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 msgr2=0x7f009406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 0x7f009406d190 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f00a4006ac0 tx=0x7f00a4006a30 comp rx=0 tx=0).stop 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 msgr2=0x7f00b40c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f00a80017a0 tx=0x7f00a8007620 comp rx=0 tx=0).stop 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 shutdown_connections 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f009406ace0 0x7f009406d190 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c004de0 0x7f00b4151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00b40c8890 0x7f00b4159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 --2- 172.21.15.104:0/2267563213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00b41e4b70 0x7f00b40c8350 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 >> 172.21.15.104:0/2267563213 conn(0x7f00b406b8a0 msgr2=0x7f00b40ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 shutdown_connections 2023-06-10T18:54:52.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.158+0000 7f00b88c5700 1 -- 172.21.15.104:0/2267563213 wait complete. 2023-06-10T18:54:52.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:53:27.152098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:53:28.410658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:54:52.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:54:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.293+0000 7fba9cae3700 1 Processor -- start 2023-06-10T18:54:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.293+0000 7fba9cae3700 1 -- start start 2023-06-10T18:54:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba980b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba9806e6e0 0x7fba980b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba9806cde0 0x7fba98069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9806e250 con 0x7fba9806cde0 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9815c950 con 0x7fba9806e6e0 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba9cae3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba980b1e80 con 0x7fba9815c580 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba96d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba9806cde0 0x7fba98069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba96d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fba9806cde0 0x7fba98069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35750/0 (socket says 172.21.15.104:35750) 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.294+0000 7fba96d9d700 1 -- 172.21.15.104:0/1322887568 learned_addr learned my addr 172.21.15.104:0/1322887568 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 -- 172.21.15.104:0/1322887568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 msgr2=0x7fba980b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba95d9b700 1 --2- 172.21.15.104:0/1322887568 >> v2:172.21.15.114:3300/0 conn(0x7fba9806e6e0 0x7fba980b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9659c700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba980b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba980b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 -- 172.21.15.104:0/1322887568 >> v2:172.21.15.114:3300/0 conn(0x7fba9806e6e0 msgr2=0x7fba980b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 --2- 172.21.15.104:0/1322887568 >> v2:172.21.15.114:3300/0 conn(0x7fba9806e6e0 0x7fba980b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 -- 172.21.15.104:0/1322887568 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9806d920 con 0x7fba9806cde0 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba96d9d700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98069e70 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fba98070290 tx=0x7fba9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617d0999aa0fff1d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9559a700 1 -- 172.21.15.104:0/1322887568 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000e070 con 0x7fba9806cde0 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9559a700 1 -- 172.21.15.104:0/1322887568 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9000b9c0 con 0x7fba9806cde0 2023-06-10T18:54:52.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9559a700 1 -- 172.21.15.104:0/1322887568 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000bd70 con 0x7fba9806cde0 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9cae3700 1 -- 172.21.15.104:0/1322887568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 msgr2=0x7fba98069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.295+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98069e70 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fba98070290 tx=0x7fba9000b330 comp rx=0 tx=0).stop 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 -- 172.21.15.104:0/1322887568 shutdown_connections 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98069e70 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1322887568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba980b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1322887568 >> v2:172.21.15.114:3300/0 conn(0x7fba9806e6e0 0x7fba980b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 -- 172.21.15.104:0/1322887568 >> 172.21.15.104:0/1322887568 conn(0x7fba9806b8a0 msgr2=0x7fba980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 -- 172.21.15.104:0/1322887568 shutdown_connections 2023-06-10T18:54:52.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.296+0000 7fba9cae3700 1 -- 172.21.15.104:0/1322887568 wait complete. 2023-06-10T18:54:52.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.297+0000 7fba9cae3700 1 Processor -- start 2023-06-10T18:54:52.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.297+0000 7fba9cae3700 1 -- start start 2023-06-10T18:54:52.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba9806e6e0 0x7fba98145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba9815ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980af480 con 0x7fba9806cde0 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980b1870 con 0x7fba9806e6e0 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9cae3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba981f5850 con 0x7fba9815c580 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35752/0 (socket says 172.21.15.104:35752) 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.298+0000 7fba9659c700 1 -- 172.21.15.104:0/1577939604 learned_addr learned my addr 172.21.15.104:0/1577939604 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba95d9b700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba9806e6e0 0x7fba98145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba96d9d700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba9815ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 -- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 msgr2=0x7fba9815ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba9815ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 -- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba9806e6e0 msgr2=0x7fba98145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba9806e6e0 0x7fba98145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 -- 172.21.15.104:0/1577939604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90009b30 con 0x7fba9806cde0 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9659c700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c820 tx=0x7fba8c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c011800 con 0x7fba9806cde0 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c011960 con 0x7fba9806cde0 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c011cf0 con 0x7fba9806cde0 2023-06-10T18:54:52.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba981f5b30 con 0x7fba9806cde0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.299+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba980b1500 con 0x7fba9806cde0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.300+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98074050 con 0x7fba9806cde0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.300+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba8c018360 con 0x7fba9806cde0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.301+0000 7fba86ffd700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.301+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fba8c089670 con 0x7fba9806cde0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.301+0000 7fba95d9b700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.301+0000 7fba95d9b700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fba981e5900 tx=0x7fba88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.305+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba8c057ff0 con 0x7fba9806cde0 2023-06-10T18:54:52.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.491+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba981e5930 con 0x7fba9806cde0 2023-06-10T18:54:52.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.491+0000 7fba86ffd700 1 -- 172.21.15.104:0/1577939604 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba8c057b80 con 0x7fba9806cde0 2023-06-10T18:54:52.497 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:54:52.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 msgr2=0x7fba7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fba981e5900 tx=0x7fba88008040 comp rx=0 tx=0).stop 2023-06-10T18:54:52.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 msgr2=0x7fba98151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c820 tx=0x7fba8c00cbe0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 shutdown_connections 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba9815c580 0x7fba9815ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba9806cde0 0x7fba98151b50 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 --2- 172.21.15.104:0/1577939604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba9806e6e0 0x7fba98145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 >> 172.21.15.104:0/1577939604 conn(0x7fba9806b8a0 msgr2=0x7fba981e4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 shutdown_connections 2023-06-10T18:54:52.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.493+0000 7fba9cae3700 1 -- 172.21.15.104:0/1577939604 wait complete. 2023-06-10T18:54:52.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:54:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.627+0000 7f3e71136700 1 Processor -- start 2023-06-10T18:54:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.627+0000 7f3e71136700 1 -- start start 2023-06-10T18:54:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e71136700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e71136700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0cc9f0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e71136700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0cc100 con 0x7f3e6c0cb210 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e71136700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c0cb9e0 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3e6c0cd8f0 0x7f3e6c06d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33586/0 (socket says 172.21.15.104:33586) 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35762/0 (socket says 172.21.15.104:35762) 2023-06-10T18:54:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6a59c700 1 -- 172.21.15.104:0/4272976046 learned_addr learned my addr 172.21.15.104:0/4272976046 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4272976046 >> v1:172.21.15.114:6789/0 conn(0x7f3e6c0cb210 legacy=0x7f3e6c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4272976046 >> v1:172.21.15.104:6789/0 conn(0x7f3e6c0cd8f0 legacy=0x7f3e6c06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4272976046 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0cc3a0 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.628+0000 7f3e6b59e700 1 --2- 172.21.15.104:0/4272976046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009d90 tx=0x7f3e6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481cdb539c95b98 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e69d9b700 1 -- 172.21.15.104:0/4272976046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6400e070 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e69d9b700 1 -- 172.21.15.104:0/4272976046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6400bb90 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e69d9b700 1 -- 172.21.15.104:0/4272976046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64003a20 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 -- 172.21.15.104:0/4272976046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cb5e0 msgr2=0x7f3e6c0cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 --2- 172.21.15.104:0/4272976046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009d90 tx=0x7f3e6400b330 comp rx=0 tx=0).stop 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 -- 172.21.15.104:0/4272976046 shutdown_connections 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 --2- 172.21.15.104:0/4272976046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0cf0b0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 -- 172.21.15.104:0/4272976046 >> 172.21.15.104:0/4272976046 conn(0x7f3e6c06b8a0 msgr2=0x7f3e6c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 -- 172.21.15.104:0/4272976046 shutdown_connections 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.629+0000 7f3e71136700 1 -- 172.21.15.104:0/4272976046 wait complete. 2023-06-10T18:54:52.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 Processor -- start 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 -- start start 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e6c0cb210 0x7f3e6c1e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c1e53e0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c1e5650 con 0x7f3e6c0cb5e0 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.631+0000 7f3e71136700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c1e5930 con 0x7f3e6c0cb210 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35768/0 (socket says 172.21.15.104:35768) 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4084997633 learned_addr learned my addr 172.21.15.104:0/4084997633 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6a59c700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6ad9d700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e6c0cb210 0x7f3e6c1e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e6c0cb210 msgr2=0x7f3e6c1e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e6c0cb210 0x7f3e6c1e6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 msgr2=0x7f3e6c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 -- 172.21.15.104:0/4084997633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64009a70 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6a59c700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e6b59e700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0b2200 tx=0x7f3e64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64012630 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e640127b0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c1e5c10 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64012b60 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.632+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c1e6670 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.633+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c074050 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.634+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3e64012cc0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.634+0000 7f3e537fe700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 0x7f3e5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.634+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f3e6408b450 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.634+0000 7f3e6ad9d700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 0x7f3e5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.635+0000 7f3e6ad9d700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 0x7f3e5806d190 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000a7d0 tx=0x7f3e60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.638+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3e640563d0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.824+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3e6c06c320 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.825+0000 7f3e537fe700 1 -- 172.21.15.104:0/4084997633 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3e640599f0 con 0x7f3e6c0cd8f0 2023-06-10T18:54:52.831 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:54:52.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 msgr2=0x7f3e5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 0x7f3e5806d190 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000a7d0 tx=0x7f3e60008040 comp rx=0 tx=0).stop 2023-06-10T18:54:52.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 msgr2=0x7f3e6c0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0b2200 tx=0x7f3e64004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 shutdown_connections 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e6c0cd8f0 0x7f3e6c0b4300 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e6c0cb210 0x7f3e6c1e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e5806ace0 0x7f3e5806d190 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 --2- 172.21.15.104:0/4084997633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e6c0cb5e0 0x7f3e6c0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 >> 172.21.15.104:0/4084997633 conn(0x7f3e6c06b8a0 msgr2=0x7f3e6c0b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 shutdown_connections 2023-06-10T18:54:52.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.827+0000 7f3e71136700 1 -- 172.21.15.104:0/4084997633 wait complete. 2023-06-10T18:54:52.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:54:52.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.961+0000 7fd07294e700 1 Processor -- start 2023-06-10T18:54:52.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.961+0000 7fd07294e700 1 -- start start 2023-06-10T18:54:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd07294e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd06c0b1780 0x7fd06c06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd07294e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd06c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd07294e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c0ae7e0 con 0x7fd06c15c6f0 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd07294e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd06c06df90 con 0x7fd06c0b1780 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd07294e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06e270 con 0x7fd06c06ea70 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd06bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd06c15c6f0 0x7fd06c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33600/0 (socket says 172.21.15.104:33600) 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.962+0000 7fd06bfff700 1 -- 172.21.15.104:0/3522429124 learned_addr learned my addr 172.21.15.104:0/3522429124 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:52.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440702462 0 0) 0x7fd06c0ae7e0 con 0x7fd06c15c6f0 2023-06-10T18:54:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/3522429124 >> v2:172.21.15.114:3300/0 conn(0x7fd06c0b1780 0x7fd06c06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd070eeb700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd06c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0580035b0 con 0x7fd06c15c6f0 2023-06-10T18:54:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 499824100 0 0) 0x7fd0580035b0 con 0x7fd06c15c6f0 2023-06-10T18:54:52.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 msgr2=0x7fd06c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd06c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 >> v2:172.21.15.114:3300/0 conn(0x7fd06c0b1780 msgr2=0x7fd06c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 --2- 172.21.15.104:0/3522429124 >> v2:172.21.15.114:3300/0 conn(0x7fd06c0b1780 0x7fd06c06cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c06e550 con 0x7fd06c15c6f0 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd0540017f0 con 0x7fd06c15c6f0 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 >> v1:172.21.15.104:6789/0 conn(0x7fd06c15c6f0 legacy=0x7fd06c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd058003a00 0x7fd058003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd0580048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd058005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06df90 con 0x7fd058004430 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c06e270 con 0x7fd058003a00 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05800ba10 con 0x7fd06c06ea70 2023-06-10T18:54:52.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd0580048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06bfff700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd058003a00 0x7fd058003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd070eeb700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd058005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 -- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 msgr2=0x7fd058005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c06ea70 0x7fd058005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 -- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd058003a00 msgr2=0x7fd058003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd058003a00 0x7fd058003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 -- 172.21.15.104:0/3522429124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0580035b0 con 0x7fd058004430 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.963+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd0580048f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009b60 tx=0x7fd058005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1585411b68d63c2d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00e6c0 con 0x7fd058004430 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c00e820 con 0x7fd058004430 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd06affd700 1 -- 172.21.15.104:0/3522429124 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00eb90 con 0x7fd058004430 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 -- 172.21.15.104:0/3522429124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 msgr2=0x7fd0580048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 --2- 172.21.15.104:0/3522429124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd0580048f0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009b60 tx=0x7fd058005880 comp rx=0 tx=0).stop 2023-06-10T18:54:52.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd070eeb700 1 -- 172.21.15.104:0/3522429124 reap_dead start 2023-06-10T18:54:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 -- 172.21.15.104:0/3522429124 shutdown_connections 2023-06-10T18:54:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 -- 172.21.15.104:0/3522429124 >> 172.21.15.104:0/3522429124 conn(0x7fd06c06b8a0 msgr2=0x7fd06c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 -- 172.21.15.104:0/3522429124 shutdown_connections 2023-06-10T18:54:52.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.964+0000 7fd07294e700 1 -- 172.21.15.104:0/3522429124 wait complete. 2023-06-10T18:54:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.965+0000 7fd07294e700 1 Processor -- start 2023-06-10T18:54:52.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 -- start start 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c1f5a70 0x7fd06c0caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c0caf80 0x7fd06c0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c0cb3f0 con 0x7fd058004430 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c154ee0 con 0x7fd06c1f5a70 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd07294e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c157e40 con 0x7fd06c0caf80 2023-06-10T18:54:52.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd06bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd06bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35784/0 (socket says 172.21.15.104:35784) 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd06bfff700 1 -- 172.21.15.104:0/4042364554 learned_addr learned my addr 172.21.15.104:0/4042364554 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c1f5a70 0x7fd06c0caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.966+0000 7fd070eeb700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c0caf80 0x7fd06c0b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 -- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c0caf80 msgr2=0x7fd06c0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c0caf80 0x7fd06c0b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 -- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c1f5a70 msgr2=0x7fd06c0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c1f5a70 0x7fd06c0caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 -- 172.21.15.104:0/4042364554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c0ad000 con 0x7fd058004430 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd06bfff700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd05400f830 tx=0x7fd05400fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0540072e0 con 0x7fd058004430 2023-06-10T18:54:52.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c0c40e0 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd054007460 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054005830 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.967+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c0c46a0 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.968+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c066f10 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.968+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd054005990 con 0x7fd058004430 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.969+0000 7fd068ff9700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 0x7fd04c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:52.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.969+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fd054089590 con 0x7fd058004430 2023-06-10T18:54:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.969+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 0x7fd04c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.969+0000 7fd06b7fe700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 0x7fd04c06d190 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0090b0 tx=0x7fd05c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:52.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:52.973+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd054057e90 con 0x7fd058004430 2023-06-10T18:54:53.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.160+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd06c074050 con 0x7fd058004430 2023-06-10T18:54:53.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.160+0000 7fd068ff9700 1 -- 172.21.15.104:0/4042364554 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd054057a20 con 0x7fd058004430 2023-06-10T18:54:53.166 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:54:53.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 msgr2=0x7fd04c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 0x7fd04c06d190 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0090b0 tx=0x7fd05c008030 comp rx=0 tx=0).stop 2023-06-10T18:54:53.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 msgr2=0x7fd06c150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd05400f830 tx=0x7fd05400fb40 comp rx=0 tx=0).stop 2023-06-10T18:54:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 shutdown_connections 2023-06-10T18:54:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd06c0caf80 0x7fd06c0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd058004430 0x7fd06c150ff0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd04c06ace0 0x7fd04c06d190 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 --2- 172.21.15.104:0/4042364554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06c1f5a70 0x7fd06c0caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 >> 172.21.15.104:0/4042364554 conn(0x7fd06c06b8a0 msgr2=0x7fd06c159240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:53.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 shutdown_connections 2023-06-10T18:54:53.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.162+0000 7fd07294e700 1 -- 172.21.15.104:0/4042364554 wait complete. 2023-06-10T18:54:53.180 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:54:53.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.296+0000 7f7d9c346700 1 Processor -- start 2023-06-10T18:54:53.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.296+0000 7f7d9c346700 1 -- start start 2023-06-10T18:54:53.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9c346700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9415c730 0x7f7d940cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9c346700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d940cd720 0x7f7d9406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9c346700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9415cb00 con 0x7f7d9406d2b0 2023-06-10T18:54:53.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9c346700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940cdb90 con 0x7f7d940cd720 2023-06-10T18:54:53.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9c346700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940acd90 con 0x7f7d9415c730 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d998e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d940cd720 0x7f7d9406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d998e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d940cd720 0x7f7d9406cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35792/0 (socket says 172.21.15.104:35792) 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d998e1700 1 -- 172.21.15.104:0/1154722226 learned_addr learned my addr 172.21.15.104:0/1154722226 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.297+0000 7f7d9a8e3700 1 --1- 172.21.15.104:0/1154722226 >> v1:172.21.15.104:6789/0 conn(0x7f7d9406d2b0 0x7f7d9406d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33624/0 (socket says 172.21.15.104:33624) 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d9a0e2700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9415c730 0x7f7d940cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d998e1700 1 -- 172.21.15.104:0/1154722226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9415c730 msgr2=0x7f7d940cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:53.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d998e1700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9415c730 0x7f7d940cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d998e1700 1 -- 172.21.15.104:0/1154722226 >> v1:172.21.15.104:6789/0 conn(0x7f7d9406d2b0 legacy=0x7f7d9406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d990e0700 1 -- 172.21.15.104:0/1154722226 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485877082 0 0) 0x7f7d8c001490 con 0x7f7d9406d2b0 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d998e1700 1 -- 172.21.15.104:0/1154722226 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d940ac460 con 0x7f7d940cd720 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d998e1700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d940cd720 0x7f7d9406cd70 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000bb90 tx=0x7f7d9000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c828fdc4d846f7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d990e0700 1 -- 172.21.15.104:0/1154722226 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000dd50 con 0x7f7d940cd720 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d990e0700 1 -- 172.21.15.104:0/1154722226 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90010470 con 0x7f7d940cd720 2023-06-10T18:54:53.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.298+0000 7f7d990e0700 1 -- 172.21.15.104:0/1154722226 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90010800 con 0x7f7d940cd720 2023-06-10T18:54:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 -- 172.21.15.104:0/1154722226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d940cd720 msgr2=0x7f7d9406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d940cd720 0x7f7d9406cd70 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000bb90 tx=0x7f7d9000bea0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 -- 172.21.15.104:0/1154722226 shutdown_connections 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9415c730 0x7f7d940cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 --2- 172.21.15.104:0/1154722226 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d940cd720 0x7f7d9406cd70 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 -- 172.21.15.104:0/1154722226 >> 172.21.15.104:0/1154722226 conn(0x7f7d9406b8a0 msgr2=0x7f7d940cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 -- 172.21.15.104:0/1154722226 shutdown_connections 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.299+0000 7f7d9c346700 1 -- 172.21.15.104:0/1154722226 wait complete. 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.300+0000 7f7d9c346700 1 Processor -- start 2023-06-10T18:54:53.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.300+0000 7f7d9c346700 1 -- start start 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 0x7f7d9414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 0x7f7d94150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b5240 con 0x7f7d9415c730 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b4f60 con 0x7f7d9406d2b0 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940c6780 con 0x7f7d940cd720 2023-06-10T18:54:53.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35802/0 (socket says 172.21.15.104:35802) 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 -- 172.21.15.104:0/2386467083 learned_addr learned my addr 172.21.15.104:0/2386467083 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d998e1700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 0x7f7d94150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a0e2700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 0x7f7d9414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 -- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 msgr2=0x7f7d94150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 0x7f7d94150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 -- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 msgr2=0x7f7d9414f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 0x7f7d9414f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 -- 172.21.15.104:0/2386467083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9000b7f0 con 0x7f7d9415c730 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d998e1700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 0x7f7d94150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a0e2700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 0x7f7d9414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9a8e3700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00cca0 tx=0x7f7d8c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c00cea0 con 0x7f7d9415c730 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c00ad80 con 0x7f7d9415c730 2023-06-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c007310 con 0x7f7d9415c730 2023-06-10T18:54:53.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d940c6a60 con 0x7f7d9415c730 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.301+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941592f0 con 0x7f7d9415c730 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.302+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d940c6f80 con 0x7f7d9415c730 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.303+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7d8c007470 con 0x7f7d9415c730 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.303+0000 7f7d8affd700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 0x7f7d7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.303+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f7d8c088f80 con 0x7f7d9415c730 2023-06-10T18:54:53.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.303+0000 7f7d9a0e2700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 0x7f7d7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.304+0000 7f7d9a0e2700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 0x7f7d7806d190 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009820 tx=0x7f7d84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.307+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7d8c0bb310 con 0x7f7d9415c730 2023-06-10T18:54:53.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.491+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d9415cb00 con 0x7f7d9415c730 2023-06-10T18:54:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.492+0000 7f7d8affd700 1 -- 172.21.15.104:0/2386467083 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d9415cb00 con 0x7f7d9415c730 2023-06-10T18:54:53.498 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:54:53.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 msgr2=0x7f7d7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 0x7f7d7806d190 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009820 tx=0x7f7d84008040 comp rx=0 tx=0).stop 2023-06-10T18:54:53.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 msgr2=0x7f7d940b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00cca0 tx=0x7f7d8c00a340 comp rx=0 tx=0).stop 2023-06-10T18:54:53.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 shutdown_connections 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d7806ace0 0x7f7d7806d190 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9406d2b0 0x7f7d9414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d9415c730 0x7f7d940b44a0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 --2- 172.21.15.104:0/2386467083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d940cd720 0x7f7d94150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 >> 172.21.15.104:0/2386467083 conn(0x7f7d9406b8a0 msgr2=0x7f7d940b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 shutdown_connections 2023-06-10T18:54:53.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.494+0000 7f7d9c346700 1 -- 172.21.15.104:0/2386467083 wait complete. 2023-06-10T18:54:53.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:53.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.625+0000 7f903aceb700 1 Processor -- start 2023-06-10T18:54:53.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.625+0000 7f903aceb700 1 -- start start 2023-06-10T18:54:53.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 0x7f903406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90340c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90340ca100 0x7f90340ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9034071640 con 0x7f90340ca100 2023-06-10T18:54:53.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340aef00 con 0x7f903406e0e0 2023-06-10T18:54:53.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f903aceb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903406d290 con 0x7f903406e4b0 2023-06-10T18:54:53.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90340ca100 0x7f90340ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90340ca100 0x7f90340ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35810/0 (socket says 172.21.15.104:35810) 2023-06-10T18:54:53.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 -- 172.21.15.104:0/4255308256 learned_addr learned my addr 172.21.15.104:0/4255308256 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:53.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9038a87700 1 --2- 172.21.15.104:0/4255308256 >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 0x7f903406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9033fff700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90340c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 -- 172.21.15.104:0/4255308256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 msgr2=0x7f90340c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90340c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 -- 172.21.15.104:0/4255308256 >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 msgr2=0x7f903406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 --2- 172.21.15.104:0/4255308256 >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 0x7f903406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 -- 172.21.15.104:0/4255308256 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903406d4c0 con 0x7f90340ca100 2023-06-10T18:54:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9038a87700 1 --2- 172.21.15.104:0/4255308256 >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 0x7f903406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f9039288700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90340ca100 0x7f90340ce9f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f903406af50 tx=0x7f902c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffecc9fca5d2b2c9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.626+0000 7f90337fe700 1 -- 172.21.15.104:0/4255308256 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c00e070 con 0x7f90340ca100 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f90337fe700 1 -- 172.21.15.104:0/4255308256 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c00b980 con 0x7f90340ca100 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f90337fe700 1 -- 172.21.15.104:0/4255308256 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c00bd30 con 0x7f90340ca100 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 -- 172.21.15.104:0/4255308256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90340ca100 msgr2=0x7f90340ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90340ca100 0x7f90340ce9f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f903406af50 tx=0x7f902c00b330 comp rx=0 tx=0).stop 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 -- 172.21.15.104:0/4255308256 shutdown_connections 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90340ca100 0x7f90340ce9f0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 --2- 172.21.15.104:0/4255308256 >> v2:172.21.15.114:3300/0 conn(0x7f903406e0e0 0x7f903406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 --2- 172.21.15.104:0/4255308256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90340c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 -- 172.21.15.104:0/4255308256 >> 172.21.15.104:0/4255308256 conn(0x7f903406b8a0 msgr2=0x7f90340cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 -- 172.21.15.104:0/4255308256 shutdown_connections 2023-06-10T18:54:53.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.627+0000 7f903aceb700 1 -- 172.21.15.104:0/4255308256 wait complete. 2023-06-10T18:54:53.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.628+0000 7f903aceb700 1 Processor -- start 2023-06-10T18:54:53.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.628+0000 7f903aceb700 1 -- start start 2023-06-10T18:54:53.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90341f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90340ca100 0x7f90340b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90340af870 con 0x7f903406e0e0 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90340afb50 con 0x7f90340ca100 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f903aceb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90340afe30 con 0x7f903406e4b0 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f9038a87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f9038a87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35812/0 (socket says 172.21.15.104:35812) 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.629+0000 7f9038a87700 1 -- 172.21.15.104:0/1651892442 learned_addr learned my addr 172.21.15.104:0/1651892442 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:53.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9039288700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90340ca100 0x7f90340b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9033fff700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90341f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 -- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 msgr2=0x7f90341f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90341f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 -- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90340ca100 msgr2=0x7f90340b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90340ca100 0x7f90340b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 -- 172.21.15.104:0/1651892442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c009b30 con 0x7f903406e0e0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f9038a87700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f902800e810 tx=0x7f902800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028005c20 con 0x7f903406e0e0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9028005d80 con 0x7f903406e0e0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90340b24d0 con 0x7f903406e0e0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028004250 con 0x7f903406e0e0 2023-06-10T18:54:53.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.630+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90340af580 con 0x7f903406e0e0 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.631+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9034070520 con 0x7f903406e0e0 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.631+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90280043b0 con 0x7f903406e0e0 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.632+0000 7f90317fa700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 0x7f901806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.632+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f9028089370 con 0x7f903406e0e0 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.632+0000 7f9033fff700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 0x7f901806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:53.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.632+0000 7f9033fff700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 0x7f901806d190 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f90340b2120 tx=0x7f9020008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:53.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.636+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9028054260 con 0x7f903406e0e0 2023-06-10T18:54:53.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.867+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90340c60b0 con 0x7f903406e0e0 2023-06-10T18:54:53.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.867+0000 7f90317fa700 1 -- 172.21.15.104:0/1651892442 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4682 (secure 0 0 0) 0x7f9028057880 con 0x7f903406e0e0 2023-06-10T18:54:53.873 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:53.874 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:54:46.733023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:53.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 msgr2=0x7f901806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 0x7f901806d190 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f90340b2120 tx=0x7f9020008040 comp rx=0 tx=0).stop 2023-06-10T18:54:53.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 msgr2=0x7f90340af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:53.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f902800e810 tx=0x7f902800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 shutdown_connections 2023-06-10T18:54:53.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f901806ace0 0x7f901806d190 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90340ca100 0x7f90340b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f903406e0e0 0x7f90340af040 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 --2- 172.21.15.104:0/1651892442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f903406e4b0 0x7f90341f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:53.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 >> 172.21.15.104:0/1651892442 conn(0x7f903406b8a0 msgr2=0x7f90340ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:53.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 shutdown_connections 2023-06-10T18:54:53.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:53.869+0000 7f903aceb700 1 -- 172.21.15.104:0/1651892442 wait complete. 2023-06-10T18:54:53.877 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 62 2023-06-10T18:54:53.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:54:54.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.001+0000 7f9511209700 1 Processor -- start 2023-06-10T18:54:54.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.001+0000 7f9511209700 1 -- start start 2023-06-10T18:54:54.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f9511209700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f950c06e0e0 0x7f950c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f9511209700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f9511209700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c06d290 con 0x7f950c0ca100 2023-06-10T18:54:54.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f9511209700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c071640 con 0x7f950c06e0e0 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f9511209700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0aef00 con 0x7f950c06e4b0 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950ad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f950c06e0e0 0x7f950c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950a59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950b59e700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f950c0ca100 0x7f950c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39498/0 (socket says 172.21.15.104:39498) 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950b59e700 1 -- 172.21.15.104:0/1461715981 learned_addr learned my addr 172.21.15.104:0/1461715981 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950a59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55204/0 (socket says 172.21.15.104:55204) 2023-06-10T18:54:54.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.002+0000 7f950ad9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f950c06e0e0 0x7f950c06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:51554/0 (socket says 172.21.15.104:51554) 2023-06-10T18:54:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950ad9d700 1 -- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 msgr2=0x7f950c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950ad9d700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950ad9d700 1 -- 172.21.15.104:0/1461715981 >> v1:172.21.15.114:6789/0 conn(0x7f950c0ca100 legacy=0x7f950c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950ad9d700 1 -- 172.21.15.104:0/1461715981 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c06d4c0 con 0x7f950c06e0e0 2023-06-10T18:54:54.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9509d9b700 1 -- 172.21.15.104:0/1461715981 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217759312 0 0) 0x7f950c06d290 con 0x7f950c0ca100 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950a59c700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f950ad9d700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c06e0e0 0x7f950c06cd50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9500009820 tx=0x7f9500009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde4ef4f8aa6f3b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9509d9b700 1 -- 172.21.15.104:0/1461715981 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500005e30 con 0x7f950c06e0e0 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9509d9b700 1 -- 172.21.15.104:0/1461715981 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9500006030 con 0x7f950c06e0e0 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9509d9b700 1 -- 172.21.15.104:0/1461715981 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000063c0 con 0x7f950c06e0e0 2023-06-10T18:54:54.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 -- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c06e0e0 msgr2=0x7f950c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c06e0e0 0x7f950c06cd50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9500009820 tx=0x7f9500009b30 comp rx=0 tx=0).stop 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 -- 172.21.15.104:0/1461715981 shutdown_connections 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c06e0e0 0x7f950c06cd50 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 --2- 172.21.15.104:0/1461715981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e4b0 0x7f950c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.003+0000 7f9511209700 1 -- 172.21.15.104:0/1461715981 >> 172.21.15.104:0/1461715981 conn(0x7f950c06b8a0 msgr2=0x7f950c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.004+0000 7f9511209700 1 -- 172.21.15.104:0/1461715981 shutdown_connections 2023-06-10T18:54:54.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.004+0000 7f9511209700 1 -- 172.21.15.104:0/1461715981 wait complete. 2023-06-10T18:54:54.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.005+0000 7f9511209700 1 Processor -- start 2023-06-10T18:54:54.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.005+0000 7f9511209700 1 -- start start 2023-06-10T18:54:54.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e0e0 0x7f950c154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c0ca100 0x7f950c1e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0af790 con 0x7f950c06e4b0 2023-06-10T18:54:54.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0afa70 con 0x7f950c0ca100 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0afd50 con 0x7f950c06e0e0 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35828/0 (socket says 172.21.15.104:35828) 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 -- 172.21.15.104:0/2678990435 learned_addr learned my addr 172.21.15.104:0/2678990435 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950b59e700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c0ca100 0x7f950c1e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950ad9d700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e0e0 0x7f950c154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 -- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e0e0 msgr2=0x7f950c154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e0e0 0x7f950c154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 -- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c0ca100 msgr2=0x7f950c1e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c0ca100 0x7f950c1e24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 -- 172.21.15.104:0/2678990435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc0094a0 con 0x7f950c06e4b0 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f950a59c700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00ebe0 tx=0x7f94fc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00c850 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94fc00c9b0 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500009430 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00cd40 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c0b0030 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.007+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c1e2a30 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.008+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94fc013470 con 0x7f950c06e4b0 2023-06-10T18:54:54.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.008+0000 7f94f37fe700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 0x7f94f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.008+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f94fc0896f0 con 0x7f950c06e4b0 2023-06-10T18:54:54.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.009+0000 7f950ad9d700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 0x7f94f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.009+0000 7f950ad9d700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 0x7f94f806d190 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9500009b70 tx=0x7f9500008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.013+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94fc054670 con 0x7f950c06e4b0 2023-06-10T18:54:54.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.199+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f950c0c6920 con 0x7f950c06e4b0 2023-06-10T18:54:54.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.201+0000 7f94f37fe700 1 -- 172.21.15.104:0/2678990435 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+18735 (secure 0 0 0) 0x7f950c0c6920 con 0x7f950c06e4b0 2023-06-10T18:54:54.207 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:54.207 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":101,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:54:44.720394+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/413723751":"2023-06-10T19:54:43.986209+0000","172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:54:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 msgr2=0x7f94f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 0x7f94f806d190 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9500009b70 tx=0x7f9500008070 comp rx=0 tx=0).stop 2023-06-10T18:54:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 msgr2=0x7f950c0b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00ebe0 tx=0x7f94fc00eef0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 shutdown_connections 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f806ace0 0x7f94f806d190 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c0ca100 0x7f950c1e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c06e0e0 0x7f950c154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 --2- 172.21.15.104:0/2678990435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c06e4b0 0x7f950c0b2850 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 >> 172.21.15.104:0/2678990435 conn(0x7f950c06b8a0 msgr2=0x7f950c0ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 shutdown_connections 2023-06-10T18:54:54.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.203+0000 7f9511209700 1 -- 172.21.15.104:0/2678990435 wait complete. 2023-06-10T18:54:54.221 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:54:54.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:54.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 Processor -- start 2023-06-10T18:54:54.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 -- start start 2023-06-10T18:54:54.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 0x7f40dc0ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 0x7f40dc06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40dc069e70 0x7f40dc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc06a820 con 0x7f40dc069e70 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc0cc050 con 0x7f40dc06d810 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.336+0000 7f40e2a6b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc06e290 con 0x7f40dc15a230 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40dc069e70 0x7f40dc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f40dc069e70 0x7f40dc06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35834/0 (socket says 172.21.15.104:35834) 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 -- 172.21.15.104:0/563415500 learned_addr learned my addr 172.21.15.104:0/563415500 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:54.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e0807700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 0x7f40dc0ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 -- 172.21.15.104:0/563415500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 msgr2=0x7f40dc0ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40dbfff700 1 --2- 172.21.15.104:0/563415500 >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 0x7f40dc06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 0x7f40dc0ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 -- 172.21.15.104:0/563415500 >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 msgr2=0x7f40dc06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 --2- 172.21.15.104:0/563415500 >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 0x7f40dc06dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 -- 172.21.15.104:0/563415500 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc0cba00 con 0x7f40dc069e70 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40dbfff700 1 --2- 172.21.15.104:0/563415500 >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 0x7f40dc06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e0807700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 0x7f40dc0ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40e1008700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc069e70 0x7f40dc06a2e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b2200 tx=0x7f40d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a35b027ccab292a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40db7fe700 1 -- 172.21.15.104:0/563415500 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400e070 con 0x7f40dc069e70 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40db7fe700 1 -- 172.21.15.104:0/563415500 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d400b9c0 con 0x7f40dc069e70 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.337+0000 7f40db7fe700 1 -- 172.21.15.104:0/563415500 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400bd70 con 0x7f40dc069e70 2023-06-10T18:54:54.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 -- 172.21.15.104:0/563415500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc069e70 msgr2=0x7f40dc06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc069e70 0x7f40dc06a2e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b2200 tx=0x7f40d400b330 comp rx=0 tx=0).stop 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 -- 172.21.15.104:0/563415500 shutdown_connections 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc069e70 0x7f40dc06a2e0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/563415500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc15a230 0x7f40dc0ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/563415500 >> v2:172.21.15.114:3300/0 conn(0x7f40dc06d810 0x7f40dc06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 -- 172.21.15.104:0/563415500 >> 172.21.15.104:0/563415500 conn(0x7f40dc06b8a0 msgr2=0x7f40dc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 -- 172.21.15.104:0/563415500 shutdown_connections 2023-06-10T18:54:54.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.338+0000 7f40e2a6b700 1 -- 172.21.15.104:0/563415500 wait complete. 2023-06-10T18:54:54.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.339+0000 7f40e2a6b700 1 Processor -- start 2023-06-10T18:54:54.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.339+0000 7f40e2a6b700 1 -- start start 2023-06-10T18:54:54.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc069e70 0x7f40dc14bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 0x7f40dc0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc14c020 con 0x7f40dc06d810 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0c7660 con 0x7f40dc15a230 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e2a6b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc1550d0 con 0x7f40dc069e70 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35840/0 (socket says 172.21.15.104:35840) 2023-06-10T18:54:54.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 -- 172.21.15.104:0/1924070400 learned_addr learned my addr 172.21.15.104:0/1924070400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e0807700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc069e70 0x7f40dc14bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e1008700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 0x7f40dc0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 -- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc069e70 msgr2=0x7f40dc14bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc069e70 0x7f40dc14bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 -- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 msgr2=0x7f40dc0b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 0x7f40dc0b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 -- 172.21.15.104:0/1924070400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009b30 con 0x7f40dc06d810 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40e1008700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 0x7f40dc0b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.340+0000 7f40dbfff700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f40c800c7c0 tx=0x7f40c800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.341+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8004220 con 0x7f40dc06d810 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.341+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c8004380 con 0x7f40dc06d810 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.341+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c80036a0 con 0x7f40dc06d810 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.341+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc0c6a50 con 0x7f40dc06d810 2023-06-10T18:54:54.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.341+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc0c7010 con 0x7f40dc06d810 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.342+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc158460 con 0x7f40dc06d810 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.342+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f40c8003800 con 0x7f40dc06d810 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.342+0000 7f40d97fa700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 0x7f40c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.343+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f40c8089be0 con 0x7f40dc06d810 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.343+0000 7f40e0807700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 0x7f40c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.343+0000 7f40e0807700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 0x7f40c006d190 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f40d000a970 tx=0x7f40d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.348+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40c80583b0 con 0x7f40dc06d810 2023-06-10T18:54:54.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.574+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40dc067760 con 0x7f40dc06d810 2023-06-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.575+0000 7f40d97fa700 1 -- 172.21.15.104:0/1924070400 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4682 (secure 0 0 0) 0x7f40c8089020 con 0x7f40dc06d810 2023-06-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:54:46.733023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 msgr2=0x7f40c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 0x7f40c006d190 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f40d000a970 tx=0x7f40d0008040 comp rx=0 tx=0).stop 2023-06-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 msgr2=0x7f40dc0c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f40c800c7c0 tx=0x7f40c800cb80 comp rx=0 tx=0).stop 2023-06-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 shutdown_connections 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f40c006ace0 0x7f40c006d190 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc15a230 0x7f40dc0b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc069e70 0x7f40dc14bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 --2- 172.21.15.104:0/1924070400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc06d810 0x7f40dc0c7af0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 >> 172.21.15.104:0/1924070400 conn(0x7f40dc06b8a0 msgr2=0x7f40dc157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 shutdown_connections 2023-06-10T18:54:54.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.577+0000 7f40e2a6b700 1 -- 172.21.15.104:0/1924070400 wait complete. 2023-06-10T18:54:54.585 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 62 2023-06-10T18:54:54.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:54:54.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 Processor -- start 2023-06-10T18:54:54.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 -- start start 2023-06-10T18:54:54.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9c815a250 0x7fe9c80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c80b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80ccca0 con 0x7fe9c80cfa00 2023-06-10T18:54:54.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80cee50 con 0x7fe9c815a250 2023-06-10T18:54:54.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9cd52d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80cf130 con 0x7fe9c80cad90 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9c815a250 0x7fe9c80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9c815a250 0x7fe9c80701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35848/0 (socket says 172.21.15.104:35848) 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 -- 172.21.15.104:0/4061910044 learned_addr learned my addr 172.21.15.104:0/4061910044 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c77fe700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c80b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 -- 172.21.15.104:0/4061910044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 msgr2=0x7fe9c80b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c80b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 -- 172.21.15.104:0/4061910044 >> v1:172.21.15.114:6789/0 conn(0x7fe9c80cfa00 legacy=0x7fe9c80cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 -- 172.21.15.104:0/4061910044 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80cc930 con 0x7fe9c815a250 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c67fc700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80701e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098f0 tx=0x7fe9bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ef11cf8d39a2d0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c5ffb700 1 -- 172.21.15.104:0/4061910044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011830 con 0x7fe9c815a250 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c5ffb700 1 -- 172.21.15.104:0/4061910044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc011990 con 0x7fe9c815a250 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.710+0000 7fe9c5ffb700 1 -- 172.21.15.104:0/4061910044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc011d20 con 0x7fe9c815a250 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 -- 172.21.15.104:0/4061910044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 msgr2=0x7fe9c80701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80701e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098f0 tx=0x7fe9bc009c00 comp rx=0 tx=0).stop 2023-06-10T18:54:54.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 -- 172.21.15.104:0/4061910044 shutdown_connections 2023-06-10T18:54:54.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c80b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/4061910044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80701e0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 -- 172.21.15.104:0/4061910044 >> 172.21.15.104:0/4061910044 conn(0x7fe9c806e370 msgr2=0x7fe9c806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 -- 172.21.15.104:0/4061910044 shutdown_connections 2023-06-10T18:54:54.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.711+0000 7fe9cd52d700 1 -- 172.21.15.104:0/4061910044 wait complete. 2023-06-10T18:54:54.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.712+0000 7fe9cd52d700 1 Processor -- start 2023-06-10T18:54:54.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.712+0000 7fe9cd52d700 1 -- start start 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c8152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 0x7fe9c80b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c81532d0 con 0x7fe9c815a250 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c5510 con 0x7fe9c80cfa00 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9cd52d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c57f0 con 0x7fe9c80cad90 2023-06-10T18:54:54.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35856/0 (socket says 172.21.15.104:35856) 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 -- 172.21.15.104:0/3176169150 learned_addr learned my addr 172.21.15.104:0/3176169150 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c67fc700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 0x7fe9c80b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c6ffd700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c8152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 -- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 msgr2=0x7fe9c8152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c8152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 -- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 msgr2=0x7fe9c80b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 0x7fe9c80b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 -- 172.21.15.104:0/3176169150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc009550 con 0x7fe9c815a250 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c67fc700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 0x7fe9c80b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c6ffd700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c8152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.713+0000 7fe9c77fe700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.714+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c000cd40 con 0x7fe9c815a250 2023-06-10T18:54:54.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.714+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c000f040 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.714+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c00146c0 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.714+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c80c5ad0 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.714+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c80c6090 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.715+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c80c6280 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.715+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe9c0014820 con 0x7fe9c815a250 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.715+0000 7fe9af7fe700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 0x7fe9b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:54.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.715+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fe9c0089fa0 con 0x7fe9c815a250 2023-06-10T18:54:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.716+0000 7fe9c6ffd700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 0x7fe9b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.716+0000 7fe9c6ffd700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 0x7fe9b406d190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8000f60 tx=0x7fe9b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:54.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.721+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe9c00bc310 con 0x7fe9c815a250 2023-06-10T18:54:54.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.948+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c8074050 con 0x7fe9c815a250 2023-06-10T18:54:54.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.948+0000 7fe9af7fe700 1 -- 172.21.15.104:0/3176169150 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fe9c0018090 con 0x7fe9c815a250 2023-06-10T18:54:54.955 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:54.955 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:53:29.447461+0000","modified":"2023-06-10T18:54:46.733023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.104:6835/1816764519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1816764519},{"type":"v1","addr":"172.21.15.104:6835","nonce":1816764519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-06-10T18:54:54.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 msgr2=0x7fe9b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 0x7fe9b406d190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8000f60 tx=0x7fe9b8009040 comp rx=0 tx=0).stop 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 msgr2=0x7fe9c80c65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c3a0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 shutdown_connections 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c815a250 0x7fe9c80c65b0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c80cad90 0x7fe9c8152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9b406ace0 0x7fe9b406d190 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 --2- 172.21.15.104:0/3176169150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c80cfa00 0x7fe9c80b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:54.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.950+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 >> 172.21.15.104:0/3176169150 conn(0x7fe9c806e370 msgr2=0x7fe9c80ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:54.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.951+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 shutdown_connections 2023-06-10T18:54:54.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:54.951+0000 7fe9cd52d700 1 -- 172.21.15.104:0/3176169150 wait complete. 2023-06-10T18:54:54.958 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 62 2023-06-10T18:54:54.968 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:54:55.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 Processor -- start 2023-06-10T18:54:55.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 -- start start 2023-06-10T18:54:55.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2d500ada30 0x7f2d500ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d500ade00 0x7f2d500ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d500cec20 con 0x7f2d500ce0b0 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d500cf430 con 0x7f2d500ada30 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.081+0000 7f2d557de700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d500cee70 con 0x7f2d500ade00 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2d500ce0b0 0x7f2d500ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33692/0 (socket says 172.21.15.104:33692) 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/1542716241 learned_addr learned my addr 172.21.15.104:0/1542716241 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:55.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d500ade00 0x7f2d500ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4effd700 1 --2- 172.21.15.104:0/1542716241 >> v2:172.21.15.114:3300/0 conn(0x7f2d500ada30 0x7f2d500ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286059474 0 0) 0x7f2d500cec20 con 0x7f2d500ce0b0 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d3c0035b0 con 0x7f2d500ce0b0 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455392483 0 0) 0x7f2d3c0035b0 con 0x7f2d500ce0b0 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d500ade00 msgr2=0x7f2d500ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d500ade00 0x7f2d500ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 >> v2:172.21.15.114:3300/0 conn(0x7f2d500ada30 msgr2=0x7f2d500ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 --2- 172.21.15.104:0/1542716241 >> v2:172.21.15.114:3300/0 conn(0x7f2d500ada30 0x7f2d500ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500cf150 con 0x7f2d500ce0b0 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2d48001670 con 0x7f2d500ce0b0 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 >> v1:172.21.15.104:6789/0 conn(0x7f2d500ce0b0 legacy=0x7f2d500ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d3c003a00 0x7f2d3c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d3c004430 0x7f2d3c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c005000 0x7f2d3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d500cf430 con 0x7f2d3c004430 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d500cee70 con 0x7f2d3c003a00 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0056b0 con 0x7f2d3c005000 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d3c004430 0x7f2d3c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4effd700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d3c003a00 0x7f2d3c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.082+0000 7f2d4f7fe700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c005000 0x7f2d3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 -- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c005000 msgr2=0x7f2d3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c005000 0x7f2d3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 -- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d3c003a00 msgr2=0x7f2d3c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d3c003a00 0x7f2d3c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 -- 172.21.15.104:0/1542716241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0035b0 con 0x7f2d3c004430 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d3c004430 0x7f2d3c0048f0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009b30 tx=0x7f2d3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59e875fbf1c5b4d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800e6c0 con 0x7f2d3c004430 2023-06-10T18:54:55.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3800e820 con 0x7f2d3c004430 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4dffb700 1 -- 172.21.15.104:0/1542716241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800eb90 con 0x7f2d3c004430 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 -- 172.21.15.104:0/1542716241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d3c004430 msgr2=0x7f2d3c0048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 --2- 172.21.15.104:0/1542716241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d3c004430 0x7f2d3c0048f0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009b30 tx=0x7f2d3c005880 comp rx=0 tx=0).stop 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/1542716241 reap_dead start 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 -- 172.21.15.104:0/1542716241 shutdown_connections 2023-06-10T18:54:55.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 -- 172.21.15.104:0/1542716241 >> 172.21.15.104:0/1542716241 conn(0x7f2d5006b8a0 msgr2=0x7f2d500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:55.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 -- 172.21.15.104:0/1542716241 shutdown_connections 2023-06-10T18:54:55.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.083+0000 7f2d557de700 1 -- 172.21.15.104:0/1542716241 wait complete. 2023-06-10T18:54:55.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.085+0000 7f2d557de700 1 Processor -- start 2023-06-10T18:54:55.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.085+0000 7f2d557de700 1 -- start start 2023-06-10T18:54:55.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.085+0000 7f2d557de700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c004430 0x7f2d5015aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.085+0000 7f2d557de700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5009c9a0 0x7f2d5015e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5009d3c0 con 0x7f2d5015ee00 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5015f270 con 0x7f2d5009c9a0 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501f5a40 con 0x7f2d3c004430 2023-06-10T18:54:55.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35876/0 (socket says 172.21.15.104:35876) 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4effd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c004430 0x7f2d5015aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/2360268367 learned_addr learned my addr 172.21.15.104:0/2360268367 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4e7fc700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5009c9a0 0x7f2d5015e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c004430 msgr2=0x7f2d5015aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c004430 0x7f2d5015aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5009c9a0 msgr2=0x7f2d5015e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5009c9a0 0x7f2d5015e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 -- 172.21.15.104:0/2360268367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5009bbe0 con 0x7f2d5015ee00 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d4f7fe700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2d50071030 tx=0x7f2d48007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48017070 con 0x7f2d5015ee00 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d48012620 con 0x7f2d5015ee00 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5009bec0 con 0x7f2d5015ee00 2023-06-10T18:54:55.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d480127b0 con 0x7f2d5015ee00 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.086+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5009c480 con 0x7f2d5015ee00 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.087+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50074050 con 0x7f2d5015ee00 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.088+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2d48012910 con 0x7f2d5015ee00 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.088+0000 7f2d477fe700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 0x7f2d3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.088+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f2d48088de0 con 0x7f2d5015ee00 2023-06-10T18:54:55.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.088+0000 7f2d4effd700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 0x7f2d3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.089+0000 7f2d4effd700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 0x7f2d3006d190 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009fd0 tx=0x7f2d4000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.092+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2d48053ce0 con 0x7f2d5015ee00 2023-06-10T18:54:55.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.278+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d50066f70 con 0x7f2d5015ee00 2023-06-10T18:54:55.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.279+0000 7f2d477fe700 1 -- 172.21.15.104:0/2360268367 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+18735 (secure 0 0 0) 0x7f2d48007b80 con 0x7f2d5015ee00 2023-06-10T18:54:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:54:55.286 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":101,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:54:44.720394+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:53:27.152098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-06-10T18:53:28.410658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/413723751":"2023-06-10T19:54:43.986209+0000","172.21.15.114:6837/3243355534":"2023-06-11T18:53:51.686929+0000","172.21.15.114:6835/3243355534":"2023-06-11T18:53:51.686929+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:54:55.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 msgr2=0x7f2d3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 0x7f2d3006d190 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009fd0 tx=0x7f2d4000b040 comp rx=0 tx=0).stop 2023-06-10T18:54:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 msgr2=0x7f2d501f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2d50071030 tx=0x7f2d48007570 comp rx=0 tx=0).stop 2023-06-10T18:54:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 shutdown_connections 2023-06-10T18:54:55.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d5015ee00 0x7f2d501f4c80 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d3c004430 0x7f2d5015aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2d3006ace0 0x7f2d3006d190 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 --2- 172.21.15.104:0/2360268367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d5009c9a0 0x7f2d5015e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 >> 172.21.15.104:0/2360268367 conn(0x7f2d5006b8a0 msgr2=0x7f2d500b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 shutdown_connections 2023-06-10T18:54:55.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.282+0000 7f2d557de700 1 -- 172.21.15.104:0/2360268367 wait complete. 2023-06-10T18:54:55.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:54:55.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 Processor -- start 2023-06-10T18:54:55.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 -- start start 2023-06-10T18:54:55.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf030 0x7ffb0c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0c069e70 0x7ffb0c06a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0c06dcb0 con 0x7ffb0c15c500 2023-06-10T18:54:55.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c06e850 con 0x7ffb0c069e70 2023-06-10T18:54:55.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb1277f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c06a830 con 0x7ffb0c0cf030 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0c069e70 0x7ffb0c06a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0c069e70 0x7ffb0c06a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35884/0 (socket says 172.21.15.104:35884) 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 -- 172.21.15.104:0/77455242 learned_addr learned my addr 172.21.15.104:0/77455242 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 -- 172.21.15.104:0/77455242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf030 msgr2=0x7ffb0c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 --2- 172.21.15.104:0/77455242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf030 0x7ffb0c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 -- 172.21.15.104:0/77455242 >> v1:172.21.15.114:6789/0 conn(0x7ffb0c15c500 legacy=0x7ffb0c06e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 -- 172.21.15.104:0/77455242 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0cf460 con 0x7ffb0c069e70 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb10d1c700 1 --2- 172.21.15.104:0/77455242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c069e70 0x7ffb0c06a2f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e50 tx=0x7ffb0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc853c04ac8923 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.418+0000 7ffb0affd700 1 -- 172.21.15.104:0/77455242 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400e070 con 0x7ffb0c069e70 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb0affd700 1 -- 172.21.15.104:0/77455242 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04003a20 con 0x7ffb0c069e70 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb0affd700 1 -- 172.21.15.104:0/77455242 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003dd0 con 0x7ffb0c069e70 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 -- 172.21.15.104:0/77455242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c069e70 msgr2=0x7ffb0c06a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 --2- 172.21.15.104:0/77455242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c069e70 0x7ffb0c06a2f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e50 tx=0x7ffb0400b330 comp rx=0 tx=0).stop 2023-06-10T18:54:55.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 -- 172.21.15.104:0/77455242 shutdown_connections 2023-06-10T18:54:55.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 --2- 172.21.15.104:0/77455242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c069e70 0x7ffb0c06a2f0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 --2- 172.21.15.104:0/77455242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c0cf030 0x7ffb0c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 -- 172.21.15.104:0/77455242 >> 172.21.15.104:0/77455242 conn(0x7ffb0c06b8a0 msgr2=0x7ffb0c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:55.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 -- 172.21.15.104:0/77455242 shutdown_connections 2023-06-10T18:54:55.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.419+0000 7ffb1277f700 1 -- 172.21.15.104:0/77455242 wait complete. 2023-06-10T18:54:55.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.420+0000 7ffb1277f700 1 Processor -- start 2023-06-10T18:54:55.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.420+0000 7ffb1277f700 1 -- start start 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c069e70 0x7ffb0c154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c15c500 0x7ffb0c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c87e0 con 0x7ffb0c0cf030 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c154e70 con 0x7ffb0c069e70 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb1277f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c7bd0 con 0x7ffb0c15c500 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb0b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb0b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35890/0 (socket says 172.21.15.104:35890) 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.421+0000 7ffb0b7fe700 1 -- 172.21.15.104:0/4242909388 learned_addr learned my addr 172.21.15.104:0/4242909388 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0bfff700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c069e70 0x7ffb0c154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb10d1c700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c15c500 0x7ffb0c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 -- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c15c500 msgr2=0x7ffb0c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c15c500 0x7ffb0c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 -- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c069e70 msgr2=0x7ffb0c154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c069e70 0x7ffb0c154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 -- 172.21.15.104:0/4242909388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04009b30 con 0x7ffb0c0cf030 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb0b7fe700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000b530 tx=0x7ffb0000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011830 con 0x7ffb0c0cf030 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb000119b0 con 0x7ffb0c0cf030 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c0c7eb0 con 0x7ffb0c0cf030 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011d60 con 0x7ffb0c0cf030 2023-06-10T18:54:55.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.422+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c0c8470 con 0x7ffb0c0cf030 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.423+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c0c8950 con 0x7ffb0c0cf030 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.423+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb0001a400 con 0x7ffb0c0cf030 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.424+0000 7ffb08ff9700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 0x7ffaf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.424+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6511+0+0 (secure 0 0 0) 0x7ffb00089670 con 0x7ffb0c0cf030 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.424+0000 7ffb0bfff700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 0x7ffaf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.424+0000 7ffb0bfff700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 0x7ffaf806d190 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffafc006fd0 tx=0x7ffafc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.429+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb000545f0 con 0x7ffb0c0cf030 2023-06-10T18:54:55.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.657+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffb0c074050 con 0x7ffb0c0cf030 2023-06-10T18:54:55.772 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:54:55.767+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:54:55.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.773+0000 7ffb08ff9700 1 -- 172.21.15.104:0/4242909388 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffb00057c10 con 0x7ffb0c0cf030 2023-06-10T18:54:55.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 msgr2=0x7ffaf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 0x7ffaf806d190 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffafc006fd0 tx=0x7ffafc008040 comp rx=0 tx=0).stop 2023-06-10T18:54:55.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 msgr2=0x7ffb0c0c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000b530 tx=0x7ffb0000b8f0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 shutdown_connections 2023-06-10T18:54:55.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaf806ace0 0x7ffaf806d190 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0c069e70 0x7ffb0c154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0c15c500 0x7ffb0c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 --2- 172.21.15.104:0/4242909388 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0c0cf030 0x7ffb0c0c44f0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 >> 172.21.15.104:0/4242909388 conn(0x7ffb0c06b8a0 msgr2=0x7ffb0c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 shutdown_connections 2023-06-10T18:54:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.776+0000 7ffb1277f700 1 -- 172.21.15.104:0/4242909388 wait complete. 2023-06-10T18:54:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:54:55.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:54:55.809 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:54:55.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.911+0000 7f81466ae700 1 Processor -- start 2023-06-10T18:54:55.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.911+0000 7f81466ae700 1 -- start start 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f81466ae700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f814015a330 0x7f81400ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f81466ae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f81400b1780 0x7f814006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f81466ae700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f814015a700 con 0x7f81400cb420 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f81466ae700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814006d7b0 con 0x7f81400b1780 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f81466ae700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400cbe00 con 0x7f814015a330 2023-06-10T18:54:55.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f8144c4b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f81400b1780 0x7f814006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f8144c4b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f81400b1780 0x7f814006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35910/0 (socket says 172.21.15.104:35910) 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f8144c4b700 1 -- 172.21.15.104:0/414548952 learned_addr learned my addr 172.21.15.104:0/414548952 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.912+0000 7f813ffff700 1 --2- 172.21.15.104:0/414548952 >> v2:172.21.15.114:3300/0 conn(0x7f814015a330 0x7f81400ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f8144c4b700 1 -- 172.21.15.104:0/414548952 >> v2:172.21.15.114:3300/0 conn(0x7f814015a330 msgr2=0x7f81400ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f8144c4b700 1 --2- 172.21.15.104:0/414548952 >> v2:172.21.15.114:3300/0 conn(0x7f814015a330 0x7f81400ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f8144c4b700 1 -- 172.21.15.104:0/414548952 >> v1:172.21.15.114:6789/0 conn(0x7f81400cb420 legacy=0x7f81400b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f8144c4b700 1 -- 172.21.15.104:0/414548952 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814006ce80 con 0x7f81400b1780 2023-06-10T18:54:55.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f8144c4b700 1 --2- 172.21.15.104:0/414548952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400b1780 0x7f814006dc40 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8140071060 tx=0x7f813800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6030f0acd981422 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f813effd700 1 -- 172.21.15.104:0/414548952 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800e070 con 0x7f81400b1780 2023-06-10T18:54:55.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f813effd700 1 -- 172.21.15.104:0/414548952 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813800b910 con 0x7f81400b1780 2023-06-10T18:54:55.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f813effd700 1 -- 172.21.15.104:0/414548952 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800bcc0 con 0x7f81400b1780 2023-06-10T18:54:55.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 -- 172.21.15.104:0/414548952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400b1780 msgr2=0x7f814006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 --2- 172.21.15.104:0/414548952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400b1780 0x7f814006dc40 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8140071060 tx=0x7f813800b330 comp rx=0 tx=0).stop 2023-06-10T18:54:55.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 -- 172.21.15.104:0/414548952 shutdown_connections 2023-06-10T18:54:55.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 --2- 172.21.15.104:0/414548952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400b1780 0x7f814006dc40 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 --2- 172.21.15.104:0/414548952 >> v2:172.21.15.114:3300/0 conn(0x7f814015a330 0x7f81400ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 -- 172.21.15.104:0/414548952 >> 172.21.15.104:0/414548952 conn(0x7f814006b8a0 msgr2=0x7f81400cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.913+0000 7f81466ae700 1 -- 172.21.15.104:0/414548952 shutdown_connections 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.914+0000 7f81466ae700 1 -- 172.21.15.104:0/414548952 wait complete. 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.915+0000 7f81466ae700 1 Processor -- start 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.915+0000 7f81466ae700 1 -- start start 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81400b1780 0x7f81400ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814015a330 0x7f81400c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81400ce800 con 0x7f81400cb420 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81400b2b70 con 0x7f814015a330 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81400c4be0 con 0x7f81400b1780 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35914/0 (socket says 172.21.15.104:35914) 2023-06-10T18:54:55.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 -- 172.21.15.104:0/4145422384 learned_addr learned my addr 172.21.15.104:0/4145422384 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813ffff700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81400b1780 0x7f81400ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f8144c4b700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814015a330 0x7f81400c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 -- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81400b1780 msgr2=0x7f81400ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81400b1780 0x7f81400ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 -- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814015a330 msgr2=0x7f81400c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814015a330 0x7f81400c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:55.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 -- 172.21.15.104:0/4145422384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8138009b30 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813f7fe700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f813000c450 tx=0x7f813000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300052d0 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130005430 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8140151e60 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300038c0 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.916+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81400ced10 con 0x7f81400cb420 2023-06-10T18:54:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.917+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8140070a60 con 0x7f81400cb420 2023-06-10T18:54:55.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.918+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8130003a20 con 0x7f81400cb420 2023-06-10T18:54:55.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.918+0000 7f813cff9700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 0x7f812006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.918+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f81300889a0 con 0x7f81400cb420 2023-06-10T18:54:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.918+0000 7f813ffff700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 0x7f812006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.919+0000 7f813ffff700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 0x7f812006d190 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8128009bc0 tx=0x7f81280091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:55.923+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8130053830 con 0x7f81400cb420 2023-06-10T18:54:56.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.152+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8140074050 con 0x7f81400cb420 2023-06-10T18:54:56.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.779+0000 7f813cff9700 1 -- 172.21.15.104:0/4145422384 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f813000ce60 con 0x7f81400cb420 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 msgr2=0x7f812006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 0x7f812006d190 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8128009bc0 tx=0x7f81280091c0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 msgr2=0x7f81400c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f813000c450 tx=0x7f813000c760 comp rx=0 tx=0).stop 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 shutdown_connections 2023-06-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f812006ace0 0x7f812006d190 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814015a330 0x7f81400c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81400b1780 0x7f81400ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 --2- 172.21.15.104:0/4145422384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81400cb420 0x7f81400c46a0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 >> 172.21.15.104:0/4145422384 conn(0x7f814006b8a0 msgr2=0x7f8140158380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 shutdown_connections 2023-06-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.782+0000 7f81466ae700 1 -- 172.21.15.104:0/4145422384 wait complete. 2023-06-10T18:54:56.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:54:56.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.962+0000 7f39009fc700 1 Processor -- start 2023-06-10T18:54:56.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.962+0000 7f39009fc700 1 -- start start 2023-06-10T18:54:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f39009fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f38fc15c6f0 0x7f38fc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f39009fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38fc06ea70 0x7f38fc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f39009fc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc0ae7e0 con 0x7f38fc0aec00 2023-06-10T18:54:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f39009fc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38fc06df90 con 0x7f38fc15c6f0 2023-06-10T18:54:56.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f39009fc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38fc06e270 con 0x7f38fc06ea70 2023-06-10T18:54:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f38fa59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f38fc15c6f0 0x7f38fc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f38fad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f38fc0aec00 0x7f38fc0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33752/0 (socket says 172.21.15.104:33752) 2023-06-10T18:54:56.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f38fa59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f38fc15c6f0 0x7f38fc0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35922/0 (socket says 172.21.15.104:35922) 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.963+0000 7f38fad9d700 1 -- 172.21.15.104:0/1900534203 learned_addr learned my addr 172.21.15.104:0/1900534203 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38f9d9b700 1 --2- 172.21.15.104:0/1900534203 >> v2:172.21.15.114:3300/0 conn(0x7f38fc06ea70 0x7f38fc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38fa59c700 1 -- 172.21.15.104:0/1900534203 >> v2:172.21.15.114:3300/0 conn(0x7f38fc06ea70 msgr2=0x7f38fc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38f959a700 1 -- 172.21.15.104:0/1900534203 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451376749 0 0) 0x7f38fc0ae7e0 con 0x7f38fc0aec00 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38fa59c700 1 --2- 172.21.15.104:0/1900534203 >> v2:172.21.15.114:3300/0 conn(0x7f38fc06ea70 0x7f38fc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38fa59c700 1 -- 172.21.15.104:0/1900534203 >> v1:172.21.15.104:6789/0 conn(0x7f38fc0aec00 legacy=0x7f38fc0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38fa59c700 1 -- 172.21.15.104:0/1900534203 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc06e550 con 0x7f38fc15c6f0 2023-06-10T18:54:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38fa59c700 1 --2- 172.21.15.104:0/1900534203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0b1070 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f38fc070990 tx=0x7f38f0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959f709b56b88478 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38f959a700 1 -- 172.21.15.104:0/1900534203 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0005e30 con 0x7f38fc15c6f0 2023-06-10T18:54:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38f959a700 1 -- 172.21.15.104:0/1900534203 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f0006030 con 0x7f38fc15c6f0 2023-06-10T18:54:56.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f38f959a700 1 -- 172.21.15.104:0/1900534203 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f00063c0 con 0x7f38fc15c6f0 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f39009fc700 1 -- 172.21.15.104:0/1900534203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc15c6f0 msgr2=0x7f38fc0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.964+0000 7f39009fc700 1 --2- 172.21.15.104:0/1900534203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0b1070 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f38fc070990 tx=0x7f38f0009b00 comp rx=0 tx=0).stop 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 -- 172.21.15.104:0/1900534203 shutdown_connections 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 --2- 172.21.15.104:0/1900534203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0b1070 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 --2- 172.21.15.104:0/1900534203 >> v2:172.21.15.114:3300/0 conn(0x7f38fc06ea70 0x7f38fc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 -- 172.21.15.104:0/1900534203 >> 172.21.15.104:0/1900534203 conn(0x7f38fc06b8a0 msgr2=0x7f38fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 -- 172.21.15.104:0/1900534203 shutdown_connections 2023-06-10T18:54:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.965+0000 7f39009fc700 1 -- 172.21.15.104:0/1900534203 wait complete. 2023-06-10T18:54:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.966+0000 7f39009fc700 1 Processor -- start 2023-06-10T18:54:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.966+0000 7f39009fc700 1 -- start start 2023-06-10T18:54:56.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38fc06ea70 0x7f38fc150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc151300 con 0x7f38fc0aec00 2023-06-10T18:54:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc1ef960 con 0x7f38fc15c6f0 2023-06-10T18:54:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc1f3c40 con 0x7f38fc06ea70 2023-06-10T18:54:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35930/0 (socket says 172.21.15.104:35930) 2023-06-10T18:54:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 -- 172.21.15.104:0/2036115686 learned_addr learned my addr 172.21.15.104:0/2036115686 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:56.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38fa59c700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38fc06ea70 0x7f38fc150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38fad9d700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 -- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38fc06ea70 msgr2=0x7f38fc150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38fc06ea70 0x7f38fc150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 -- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38fc15c6f0 msgr2=0x7f38fc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:56.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 -- 172.21.15.104:0/2036115686 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f0009430 con 0x7f38fc0aec00 2023-06-10T18:54:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38f9d9b700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e810 tx=0x7f38ec00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec00c070 con 0x7f38fc0aec00 2023-06-10T18:54:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec013610 con 0x7f38fc0aec00 2023-06-10T18:54:56.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc1efc20 con 0x7f38fc0aec00 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0139a0 con 0x7f38fc0aec00 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.967+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc1f01e0 con 0x7f38fc0aec00 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.968+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc0cdb20 con 0x7f38fc0aec00 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.969+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f38ec013b00 con 0x7f38fc0aec00 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.969+0000 7f38eaffd700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 0x7f38dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.969+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f38ec089020 con 0x7f38fc0aec00 2023-06-10T18:54:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.969+0000 7f38fa59c700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 0x7f38dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.970+0000 7f38fa59c700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 0x7f38dc06d190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:56.973+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f38fc0cdb20 con 0x7f38fc0aec00 2023-06-10T18:54:57.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.164+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f38fc0cdb20 con 0x7f38fc0aec00 2023-06-10T18:54:57.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.782+0000 7f38eaffd700 1 -- 172.21.15.104:0/2036115686 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7f38fc0cdb20 con 0x7f38fc0aec00 2023-06-10T18:54:57.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 msgr2=0x7f38dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 0x7f38dc06d190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0008070 comp rx=0 tx=0).stop 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 msgr2=0x7f38fc0cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e810 tx=0x7f38ec00eb20 comp rx=0 tx=0).stop 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 shutdown_connections 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38dc06ace0 0x7f38dc06d190 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38fc15c6f0 0x7f38fc0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38fc06ea70 0x7f38fc150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 --2- 172.21.15.104:0/2036115686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38fc0aec00 0x7f38fc0cbd80 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 >> 172.21.15.104:0/2036115686 conn(0x7f38fc06b8a0 msgr2=0x7f38fc0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:57.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.785+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 shutdown_connections 2023-06-10T18:54:57.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.786+0000 7f39009fc700 1 -- 172.21.15.104:0/2036115686 wait complete. 2023-06-10T18:54:57.795 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:54:57.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:54:57.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.929+0000 7f4925aa7700 1 Processor -- start 2023-06-10T18:54:57.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.929+0000 7f4925aa7700 1 -- start start 2023-06-10T18:54:57.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f4925aa7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49200cd8f0 0x7f49200ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f4925aa7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49200ad760 0x7f49200adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f4925aa7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492015cb00 con 0x7f492015c730 2023-06-10T18:54:57.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f4925aa7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49200acd90 con 0x7f49200cd8f0 2023-06-10T18:54:57.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f4925aa7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49200ac460 con 0x7f49200ad760 2023-06-10T18:54:57.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f491effd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49200cd8f0 0x7f49200ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f491effd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49200cd8f0 0x7f49200ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:51672/0 (socket says 172.21.15.104:51672) 2023-06-10T18:54:57.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f491effd700 1 -- 172.21.15.104:0/2550782448 learned_addr learned my addr 172.21.15.104:0/2550782448 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:57.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f491f7fe700 1 --1- 172.21.15.104:0/2550782448 >> v1:172.21.15.114:6789/0 conn(0x7f492015c730 0x7f49200cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39610/0 (socket says 172.21.15.104:39610) 2023-06-10T18:54:57.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.930+0000 7f491ffff700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49200ad760 0x7f49200adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491effd700 1 -- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49200ad760 msgr2=0x7f49200adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491effd700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49200ad760 0x7f49200adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491effd700 1 -- 172.21.15.104:0/2550782448 >> v1:172.21.15.114:6789/0 conn(0x7f492015c730 legacy=0x7f49200cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491effd700 1 -- 172.21.15.104:0/2550782448 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49200ac740 con 0x7f49200cd8f0 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491effd700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200cd8f0 0x7f49200ad220 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f491000d150 tx=0x7f491000d460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd457893c2cc6cb6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:57.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491e7fc700 1 -- 172.21.15.104:0/2550782448 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910010070 con 0x7f49200cd8f0 2023-06-10T18:54:57.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491e7fc700 1 -- 172.21.15.104:0/2550782448 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000dd10 con 0x7f49200cd8f0 2023-06-10T18:54:57.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f491e7fc700 1 -- 172.21.15.104:0/2550782448 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000b310 con 0x7f49200cd8f0 2023-06-10T18:54:57.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 -- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200cd8f0 msgr2=0x7f49200ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200cd8f0 0x7f49200ad220 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f491000d150 tx=0x7f491000d460 comp rx=0 tx=0).stop 2023-06-10T18:54:57.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 -- 172.21.15.104:0/2550782448 shutdown_connections 2023-06-10T18:54:57.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49200ad760 0x7f49200adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 --2- 172.21.15.104:0/2550782448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200cd8f0 0x7f49200ad220 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.931+0000 7f4925aa7700 1 -- 172.21.15.104:0/2550782448 >> 172.21.15.104:0/2550782448 conn(0x7f492006b8a0 msgr2=0x7f49200cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:57.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.932+0000 7f4925aa7700 1 -- 172.21.15.104:0/2550782448 shutdown_connections 2023-06-10T18:54:57.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.932+0000 7f4925aa7700 1 -- 172.21.15.104:0/2550782448 wait complete. 2023-06-10T18:54:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.933+0000 7f4925aa7700 1 Processor -- start 2023-06-10T18:54:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.933+0000 7f4925aa7700 1 -- start start 2023-06-10T18:54:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200ad760 0x7f49200af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f492015c730 0x7f49200c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200afe40 con 0x7f49200cd8f0 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200b0120 con 0x7f49200ad760 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49200b0400 con 0x7f492015c730 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35944/0 (socket says 172.21.15.104:35944) 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 -- 172.21.15.104:0/3552330491 learned_addr learned my addr 172.21.15.104:0/3552330491 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:57.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491f7fe700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200ad760 0x7f49200af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491ffff700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f492015c730 0x7f49200c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 -- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f492015c730 msgr2=0x7f49200c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f492015c730 0x7f49200c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 -- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200ad760 msgr2=0x7f49200af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200ad760 0x7f49200af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 -- 172.21.15.104:0/3552330491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491000bd40 con 0x7f49200cd8f0 2023-06-10T18:54:57.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f491effd700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f49200b2200 tx=0x7f4910004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000bbc0 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000b4c0 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492006f150 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910004040 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.934+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49200b09c0 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.935+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4920066f10 con 0x7f49200cd8f0 2023-06-10T18:54:57.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.936+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f49100041a0 con 0x7f49200cd8f0 2023-06-10T18:54:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.936+0000 7f4907fff700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 0x7f490c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.936+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f491008cf50 con 0x7f49200cd8f0 2023-06-10T18:54:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.937+0000 7f491f7fe700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 0x7f490c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.937+0000 7f491f7fe700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 0x7f490c06d190 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f49200b56a0 tx=0x7f4914000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:57.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:57.940+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4910058030 con 0x7f49200cd8f0 2023-06-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.129+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4920074050 con 0x7f49200cd8f0 2023-06-10T18:54:58.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.789+0000 7f4907fff700 1 -- 172.21.15.104:0/3552330491 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7f491005b650 con 0x7f49200cd8f0 2023-06-10T18:54:58.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 msgr2=0x7f490c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:58.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 0x7f490c06d190 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f49200b56a0 tx=0x7f4914000bc0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 msgr2=0x7f49200b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f49200b2200 tx=0x7f4910004ad0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 shutdown_connections 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f490c06ace0 0x7f490c06d190 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49200ad760 0x7f49200af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f492015c730 0x7f49200c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 --2- 172.21.15.104:0/3552330491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49200cd8f0 0x7f49200b0ee0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 >> 172.21.15.104:0/3552330491 conn(0x7f492006b8a0 msgr2=0x7f49200adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:58.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 shutdown_connections 2023-06-10T18:54:58.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.793+0000 7f4925aa7700 1 -- 172.21.15.104:0/3552330491 wait complete. 2023-06-10T18:54:58.800 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:54:58.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:54:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.941+0000 7f23166c0700 1 Processor -- start 2023-06-10T18:54:58.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.941+0000 7f23166c0700 1 -- start start 2023-06-10T18:54:58.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f23166c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f23100d04f0 0x7f23100d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f23166c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 0x7f23100b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f23166c0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100af530 con 0x7f23100d25c0 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f23166c0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23100af810 con 0x7f23100d04f0 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f23166c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23100afaf0 con 0x7f23100b05d0 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f230ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f23100d04f0 0x7f23100d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f230ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f23100d04f0 0x7f23100d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35950/0 (socket says 172.21.15.104:35950) 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f230ffff700 1 -- 172.21.15.104:0/3350603955 learned_addr learned my addr 172.21.15.104:0/3350603955 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f230ffff700 1 -- 172.21.15.104:0/3350603955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 msgr2=0x7f23100b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.942+0000 7f2314c5d700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 0x7f23100b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230ffff700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 0x7f23100b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230ffff700 1 -- 172.21.15.104:0/3350603955 >> v1:172.21.15.114:6789/0 conn(0x7f23100d25c0 legacy=0x7f23100d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230ffff700 1 -- 172.21.15.104:0/3350603955 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23100afdd0 con 0x7f23100d04f0 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f2314c5d700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 0x7f23100b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230ffff700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d04f0 0x7f23100d34d0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f2300006fd0 tx=0x7f2300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8e004fc138893c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230effd700 1 -- 172.21.15.104:0/3350603955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300005e30 con 0x7f23100d04f0 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230effd700 1 -- 172.21.15.104:0/3350603955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300006030 con 0x7f23100d04f0 2023-06-10T18:54:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f230effd700 1 -- 172.21.15.104:0/3350603955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23000063c0 con 0x7f23100d04f0 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 -- 172.21.15.104:0/3350603955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d04f0 msgr2=0x7f23100d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d04f0 0x7f23100d34d0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f2300006fd0 tx=0x7f2300009bd0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 -- 172.21.15.104:0/3350603955 shutdown_connections 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d04f0 0x7f23100d34d0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 --2- 172.21.15.104:0/3350603955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100b05d0 0x7f23100b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 -- 172.21.15.104:0/3350603955 >> 172.21.15.104:0/3350603955 conn(0x7f2310071ec0 msgr2=0x7f231009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 -- 172.21.15.104:0/3350603955 shutdown_connections 2023-06-10T18:54:58.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.943+0000 7f23166c0700 1 -- 172.21.15.104:0/3350603955 wait complete. 2023-06-10T18:54:58.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 Processor -- start 2023-06-10T18:54:58.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 -- start start 2023-06-10T18:54:58.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23100b05d0 0x7f23100a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100d04f0 0x7f23101473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.945+0000 7f23166c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310144960 con 0x7f23100d25c0 2023-06-10T18:54:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f23166c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310147910 con 0x7f23100b05d0 2023-06-10T18:54:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f23166c0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2310144bb0 con 0x7f23100d04f0 2023-06-10T18:54:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35960/0 (socket says 172.21.15.104:35960) 2023-06-10T18:54:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 -- 172.21.15.104:0/1500901341 learned_addr learned my addr 172.21.15.104:0/1500901341 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f230ffff700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23100b05d0 0x7f23100a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f230f7fe700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100d04f0 0x7f23101473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 -- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100d04f0 msgr2=0x7f23101473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100d04f0 0x7f23101473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 -- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23100b05d0 msgr2=0x7f23100a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23100b05d0 0x7f23100a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:58.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 -- 172.21.15.104:0/1500901341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300009500 con 0x7f23100d25c0 2023-06-10T18:54:58.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f2314c5d700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f230800cef0 tx=0x7f230800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230800ad00 con 0x7f23100d25c0 2023-06-10T18:54:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230800ae60 con 0x7f23100d25c0 2023-06-10T18:54:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310144e90 con 0x7f23100d25c0 2023-06-10T18:54:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308015680 con 0x7f23100d25c0 2023-06-10T18:54:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.946+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2310145450 con 0x7f23100d25c0 2023-06-10T18:54:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.947+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2310067010 con 0x7f23100d25c0 2023-06-10T18:54:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.948+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f230800d430 con 0x7f23100d25c0 2023-06-10T18:54:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.948+0000 7f230cff9700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 0x7f22fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.948+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f2308089770 con 0x7f23100d25c0 2023-06-10T18:54:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.948+0000 7f230ffff700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 0x7f22fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.949+0000 7f230ffff700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 0x7f22fc06d190 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2300013040 tx=0x7f2300008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:58.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:58.952+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f23080549e0 con 0x7f23100d25c0 2023-06-10T18:54:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.144+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2310074050 con 0x7f23100d25c0 2023-06-10T18:54:59.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.148+0000 7f230cff9700 1 -- 172.21.15.104:0/1500901341 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2308058000 con 0x7f23100d25c0 2023-06-10T18:54:59.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 msgr2=0x7f22fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 0x7f22fc06d190 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2300013040 tx=0x7f2300008070 comp rx=0 tx=0).stop 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 msgr2=0x7f23100b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f230800cef0 tx=0x7f230800cf20 comp rx=0 tx=0).stop 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 shutdown_connections 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22fc06ace0 0x7f22fc06d190 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23100b05d0 0x7f23100a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23100d25c0 0x7f23100b12f0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 --2- 172.21.15.104:0/1500901341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100d04f0 0x7f23101473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 >> 172.21.15.104:0/1500901341 conn(0x7f2310071ec0 msgr2=0x7f23100c99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:59.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 shutdown_connections 2023-06-10T18:54:59.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.150+0000 7f23166c0700 1 -- 172.21.15.104:0/1500901341 wait complete. 2023-06-10T18:54:59.166 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync' 2023-06-10T18:54:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.281+0000 7fccb0b3e700 1 Processor -- start 2023-06-10T18:54:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.281+0000 7fccb0b3e700 1 -- start start 2023-06-10T18:54:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fccb0b3e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccac070a20 0x7fccac06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fccb0b3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fccac159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fccb0b3e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac06ae00 con 0x7fccac070df0 2023-06-10T18:54:59.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fccb0b3e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccac069e70 con 0x7fccac070a20 2023-06-10T18:54:59.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fccb0b3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1595f0 con 0x7fccac158eb0 2023-06-10T18:54:59.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fcca9d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fccac070df0 0x7fccac1587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33792/0 (socket says 172.21.15.104:33792) 2023-06-10T18:54:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.282+0000 7fcca9d9b700 1 -- 172.21.15.104:0/2446929964 learned_addr learned my addr 172.21.15.104:0/2446929964 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:54:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119896022 0 0) 0x7fcc9c001110 con 0x7fccac070df0 2023-06-10T18:54:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc940046c0 con 0x7fccac070df0 2023-06-10T18:54:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fccac159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaa59c700 1 --2- 172.21.15.104:0/2446929964 >> v2:172.21.15.114:3300/0 conn(0x7fccac070a20 0x7fccac06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1795244354 0 0) 0x7fcc940046c0 con 0x7fccac070df0 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 msgr2=0x7fccac159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fccac159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 >> v2:172.21.15.114:3300/0 conn(0x7fccac070a20 msgr2=0x7fccac06a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 --2- 172.21.15.104:0/2446929964 >> v2:172.21.15.114:3300/0 conn(0x7fccac070a20 0x7fccac06a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac1598d0 con 0x7fccac070df0 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcc9c001800 con 0x7fccac070df0 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 >> v1:172.21.15.104:6789/0 conn(0x7fccac070df0 legacy=0x7fccac1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc94004b10 0x7fcc94004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fcc94005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fcc94006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac069e70 con 0x7fcc94005ea0 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1595f0 con 0x7fcc94004b10 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9400cb20 con 0x7fccac158eb0 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaa59c700 1 --2- 172.21.15.104:0/2446929964 >> v2:172.21.15.114:3300/0 conn(0x7fccac070a20 0x7fccac06a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fcc94006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fcca9d9b700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fcc94005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaa59c700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc94004b10 0x7fcc94004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 -- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 msgr2=0x7fcc94005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac158eb0 0x7fcc94005960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 -- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc94004b10 msgr2=0x7fcc94004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc94004b10 0x7fcc94004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.283+0000 7fccaad9d700 1 -- 172.21.15.104:0/2446929964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940046c0 con 0x7fcc94005ea0 2023-06-10T18:54:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccaad9d700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fcc94006310 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009fc0 tx=0x7fcc94006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bd80f3c1c78e44 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400ec30 con 0x7fcc94005ea0 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca400f040 con 0x7fcc94005ea0 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fcca959a700 1 -- 172.21.15.104:0/2446929964 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4014480 con 0x7fcc94005ea0 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 -- 172.21.15.104:0/2446929964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 msgr2=0x7fcc94006310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/2446929964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fcc94006310 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009fc0 tx=0x7fcc94006990 comp rx=0 tx=0).stop 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccaad9d700 1 -- 172.21.15.104:0/2446929964 reap_dead start 2023-06-10T18:54:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 -- 172.21.15.104:0/2446929964 shutdown_connections 2023-06-10T18:54:59.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 -- 172.21.15.104:0/2446929964 >> 172.21.15.104:0/2446929964 conn(0x7fccac0b1610 msgr2=0x7fccac0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:59.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 -- 172.21.15.104:0/2446929964 shutdown_connections 2023-06-10T18:54:59.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.284+0000 7fccb0b3e700 1 -- 172.21.15.104:0/2446929964 wait complete. 2023-06-10T18:54:59.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 Processor -- start 2023-06-10T18:54:59.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 -- start start 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccac1de8c0 0x7fccac0ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 0x7fccac0ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac1e18a0 con 0x7fcc94005ea0 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0c9440 con 0x7fccac1de8c0 2023-06-10T18:54:59.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.286+0000 7fccb0b3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0c96e0 con 0x7fccac0adea0 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35976/0 (socket says 172.21.15.104:35976) 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 -- 172.21.15.104:0/3371732068 learned_addr learned my addr 172.21.15.104:0/3371732068 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaad9d700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 0x7fccac0ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fcca9d9b700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccac1de8c0 0x7fccac0ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 -- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 msgr2=0x7fccac0ab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 0x7fccac0ab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 -- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccac1de8c0 msgr2=0x7fccac0ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccac1de8c0 0x7fccac0ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 -- 172.21.15.104:0/3371732068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0c99c0 con 0x7fcc94005ea0 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaad9d700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 0x7fccac0ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccaa59c700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fcca0010890 tx=0x7fcca0010ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0015880 con 0x7fcc94005ea0 2023-06-10T18:54:59.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca000c050 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac0c9ca0 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0015c20 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.287+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac0ae320 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.288+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac071780 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.289+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcca0016420 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.289+0000 7fcc9affd700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 0x7fcc8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.289+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fcca0089a80 con 0x7fcc94005ea0 2023-06-10T18:54:59.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.289+0000 7fcca9d9b700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 0x7fcc8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.290+0000 7fcca9d9b700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 0x7fcc8c06d190 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fccac072b40 tx=0x7fcc9c007b00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.293+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcca0054c70 con 0x7fcc94005ea0 2023-06-10T18:54:59.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.555+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync"]} v 0) v1 -- 0x7fccac074050 con 0x7fcc94005ea0 2023-06-10T18:54:59.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.791+0000 7fcc9affd700 1 -- 172.21.15.104:0/3371732068 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dir_fsync"]}]=0 v513) v1 ==== 146+0+0 (secure 0 0 0) 0x7fcca0058290 con 0x7fcc94005ea0 2023-06-10T18:54:59.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 msgr2=0x7fcc8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 0x7fcc8c06d190 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fccac072b40 tx=0x7fcc9c007b00 comp rx=0 tx=0).stop 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 msgr2=0x7fccac1e1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fcca0010890 tx=0x7fcca0010ba0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 shutdown_connections 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccac0adea0 0x7fccac0ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc94005ea0 0x7fccac1e1360 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc8c06ace0 0x7fcc8c06d190 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.793+0000 7fccb0b3e700 1 --2- 172.21.15.104:0/3371732068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccac1de8c0 0x7fccac0ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.794+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 >> 172.21.15.104:0/3371732068 conn(0x7fccac0b1610 msgr2=0x7fccac06c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.794+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 shutdown_connections 2023-06-10T18:54:59.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.794+0000 7fccb0b3e700 1 -- 172.21.15.104:0/3371732068 wait complete. 2023-06-10T18:54:59.811 INFO:tasks.cephfs_test_runner:test_dir_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:54:59.811 INFO:tasks.cephfs_test_runner:Starting test: test_dont_mark_unresponsive_client_stale (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:54:59.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale' 2023-06-10T18:54:59.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.928+0000 7fa9ef2e5700 1 Processor -- start 2023-06-10T18:54:59.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 -- start start 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e80ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9e80ae280 0x7fa9e80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9e8070a20 0x7fa9e8070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80713e0 con 0x7fa9e8070a20 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80ccea0 con 0x7fa9e80ae280 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ef2e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80ae6f0 con 0x7fa9e815c040 2023-06-10T18:54:59.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.929+0000 7fa9ed882700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9e8070a20 0x7fa9e8070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa9e8070a20 0x7fa9e8070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35988/0 (socket says 172.21.15.104:35988) 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 -- 172.21.15.104:0/1448877053 learned_addr learned my addr 172.21.15.104:0/1448877053 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 -- 172.21.15.104:0/1448877053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 msgr2=0x7fa9e80ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 --2- 172.21.15.104:0/1448877053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e80ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 -- 172.21.15.104:0/1448877053 >> v2:172.21.15.114:3300/0 conn(0x7fa9e80ae280 msgr2=0x7fa9e80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 --2- 172.21.15.104:0/1448877053 >> v2:172.21.15.114:3300/0 conn(0x7fa9e80ae280 0x7fa9e80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 -- 172.21.15.104:0/1448877053 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80ce2c0 con 0x7fa9e8070a20 2023-06-10T18:54:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9ed882700 1 --2- 172.21.15.104:0/1448877053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e8070a20 0x7fa9e8070ea0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80663f0 tx=0x7fa9e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9debbebf4afd5f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9dffff700 1 -- 172.21.15.104:0/1448877053 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000e070 con 0x7fa9e8070a20 2023-06-10T18:54:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9dffff700 1 -- 172.21.15.104:0/1448877053 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e000b9c0 con 0x7fa9e8070a20 2023-06-10T18:54:59.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.930+0000 7fa9dffff700 1 -- 172.21.15.104:0/1448877053 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000bd70 con 0x7fa9e8070a20 2023-06-10T18:54:59.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/1448877053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e8070a20 msgr2=0x7fa9e8070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/1448877053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e8070a20 0x7fa9e8070ea0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80663f0 tx=0x7fa9e000b330 comp rx=0 tx=0).stop 2023-06-10T18:54:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/1448877053 shutdown_connections 2023-06-10T18:54:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/1448877053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e8070a20 0x7fa9e8070ea0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/1448877053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e815c040 0x7fa9e80ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/1448877053 >> v2:172.21.15.114:3300/0 conn(0x7fa9e80ae280 0x7fa9e80cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/1448877053 >> 172.21.15.104:0/1448877053 conn(0x7fa9e80b1610 msgr2=0x7fa9e80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/1448877053 shutdown_connections 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.931+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/1448877053 wait complete. 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.932+0000 7fa9ef2e5700 1 Processor -- start 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.932+0000 7fa9ef2e5700 1 -- start start 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e8070a20 0x7fa9e814a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e815c040 0x7fa9e80caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e814a690 con 0x7fa9e80ae280 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81f5660 con 0x7fa9e815c040 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81f5940 con 0x7fa9e8070a20 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35992/0 (socket says 172.21.15.104:35992) 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 -- 172.21.15.104:0/2022595856 learned_addr learned my addr 172.21.15.104:0/2022595856 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ed882700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e815c040 0x7fa9e80caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ed081700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e8070a20 0x7fa9e814a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 -- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e8070a20 msgr2=0x7fa9e814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e8070a20 0x7fa9e814a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 -- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e815c040 msgr2=0x7fa9e80caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e815c040 0x7fa9e80caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 -- 172.21.15.104:0/2022595856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0009b30 con 0x7fa9e80ae280 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ec880700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400b530 tx=0x7fa9e400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4011830 con 0x7fa9e80ae280 2023-06-10T18:54:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4011990 con 0x7fa9e80ae280 2023-06-10T18:54:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e81f5c20 con 0x7fa9e80ae280 2023-06-10T18:54:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.933+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4011d40 con 0x7fa9e80ae280 2023-06-10T18:54:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.934+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81f61e0 con 0x7fa9e80ae280 2023-06-10T18:54:59.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.935+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e81f6400 con 0x7fa9e80ae280 2023-06-10T18:54:59.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.935+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa9e401a400 con 0x7fa9e80ae280 2023-06-10T18:54:59.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.935+0000 7fa9ddffb700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:54:59.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.935+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fa9e4089280 con 0x7fa9e80ae280 2023-06-10T18:54:59.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.936+0000 7fa9ed081700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:54:59.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.936+0000 7fa9ed081700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009970 tx=0x7fa9d8009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:54:59.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:54:59.940+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa9e40545b0 con 0x7fa9e80ae280 2023-06-10T18:55:00.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.201+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale"]} v 0) v1 -- 0x7fa9e8074050 con 0x7fa9e80ae280 2023-06-10T18:55:00.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.793+0000 7fa9ddffb700 1 -- 172.21.15.104:0/2022595856 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale"]}]=0 v514) v1 ==== 175+0+0 (secure 0 0 0) 0x7fa9e4057bd0 con 0x7fa9e80ae280 2023-06-10T18:55:00.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 msgr2=0x7fa9cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009970 tx=0x7fa9d8009110 comp rx=0 tx=0).stop 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 msgr2=0x7fa9e80ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400b530 tx=0x7fa9e400b840 comp rx=0 tx=0).stop 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 shutdown_connections 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa9cc06ace0 0x7fa9cc06d190 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e815c040 0x7fa9e80caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e8070a20 0x7fa9e814a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 --2- 172.21.15.104:0/2022595856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9e80ae280 0x7fa9e80ca540 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.796+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 >> 172.21.15.104:0/2022595856 conn(0x7fa9e80b1610 msgr2=0x7fa9e80ace60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:00.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.797+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 shutdown_connections 2023-06-10T18:55:00.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.797+0000 7fa9ef2e5700 1 -- 172.21.15.104:0/2022595856 wait complete. 2023-06-10T18:55:00.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:55:00.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.940+0000 7f920b46c700 1 Processor -- start 2023-06-10T18:55:00.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.940+0000 7f920b46c700 1 -- start start 2023-06-10T18:55:00.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f920b46c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f92040cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f920b46c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040cda90 con 0x7f92040af940 2023-06-10T18:55:00.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f920b46c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9204154970 con 0x7f92041545a0 2023-06-10T18:55:00.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f920b46c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920415dab0 con 0x7f92040cd150 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f9208a07700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f92040af940 0x7f920415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33828/0 (socket says 172.21.15.104:33828) 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f9208a07700 1 -- 172.21.15.104:0/3325982803 learned_addr learned my addr 172.21.15.104:0/3325982803 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f9209a09700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f92040cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176139754 0 0) 0x7f92040cda90 con 0x7f92040af940 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.941+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f00035b0 con 0x7f92040af940 2023-06-10T18:55:00.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503808766 0 0) 0x7f9204154970 con 0x7f92041545a0 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92040cda90 con 0x7f92041545a0 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139673699 0 0) 0x7f91f00035b0 con 0x7f92040af940 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 msgr2=0x7f92040cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f92040cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 >> v1:172.21.15.114:6789/0 conn(0x7f92041545a0 legacy=0x7f920415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92040cb0b0 con 0x7f92040af940 2023-06-10T18:55:00.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9200001670 con 0x7f92040af940 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 >> v1:172.21.15.104:6789/0 conn(0x7f92040af940 legacy=0x7f920415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f91f0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91f0004120 0x7f91f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91f0004d90 0x7f91f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920415dab0 con 0x7f91f0004d90 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f000bac0 con 0x7f91f0004120 2023-06-10T18:55:00.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f000bc80 con 0x7f92040cd150 2023-06-10T18:55:00.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91f0004d90 0x7f91f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9208a07700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91f0004120 0x7f91f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209208700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f91f0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 -- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 msgr2=0x7f91f0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040cd150 0x7f91f0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 -- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91f0004120 msgr2=0x7f91f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91f0004120 0x7f91f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 -- 172.21.15.104:0/3325982803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00035b0 con 0x7f91f0004d90 2023-06-10T18:55:00.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f9209a09700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91f0004d90 0x7f91f0005200 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009f90 tx=0x7f91f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde9c936078d9b9a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00ec30 con 0x7f91f0004d90 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc00f040 con 0x7f91f0004d90 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.942+0000 7f91fbfff700 1 -- 172.21.15.104:0/3325982803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc014480 con 0x7f91f0004d90 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 -- 172.21.15.104:0/3325982803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91f0004d90 msgr2=0x7f91f0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 --2- 172.21.15.104:0/3325982803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91f0004d90 0x7f91f0005200 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009f90 tx=0x7f91f0005880 comp rx=0 tx=0).stop 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f9209a09700 1 -- 172.21.15.104:0/3325982803 reap_dead start 2023-06-10T18:55:00.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 -- 172.21.15.104:0/3325982803 shutdown_connections 2023-06-10T18:55:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 -- 172.21.15.104:0/3325982803 >> 172.21.15.104:0/3325982803 conn(0x7f920406e370 msgr2=0x7f920406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 -- 172.21.15.104:0/3325982803 shutdown_connections 2023-06-10T18:55:00.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.943+0000 7f920b46c700 1 -- 172.21.15.104:0/3325982803 wait complete. 2023-06-10T18:55:00.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.945+0000 7f920b46c700 1 Processor -- start 2023-06-10T18:55:00.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.945+0000 7f920b46c700 1 -- start start 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92041531e0 0x7f92040b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040c8da0 0x7f92040c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040b2dd0 con 0x7f92040b2960 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040b1920 con 0x7f92041531e0 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92040c4690 con 0x7f92040c8da0 2023-06-10T18:55:00.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36012/0 (socket says 172.21.15.104:36012) 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 -- 172.21.15.104:0/973955642 learned_addr learned my addr 172.21.15.104:0/973955642 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9209208700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92041531e0 0x7f92040b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9209a09700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040c8da0 0x7f92040c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 -- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040c8da0 msgr2=0x7f92040c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040c8da0 0x7f92040c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 -- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92041531e0 msgr2=0x7f92040b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:00.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92041531e0 0x7f92040b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 -- 172.21.15.104:0/973955642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92040b0b60 con 0x7f92040b2960 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f9208a07700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f9200007340 tx=0x7f9200007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920000f660 con 0x7f92040b2960 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920000f7c0 con 0x7f92040b2960 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92040b0e40 con 0x7f92040b2960 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.946+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920000fb70 con 0x7f92040b2960 2023-06-10T18:55:00.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.947+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92040b1400 con 0x7f92040b2960 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.947+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9204074050 con 0x7f92040b2960 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.948+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f920000fcd0 con 0x7f92040b2960 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.948+0000 7f91f9ffb700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.948+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f9200089290 con 0x7f92040b2960 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.948+0000 7f9209208700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.949+0000 7f9209208700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f92040ca810 tx=0x7f91f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:00.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:00.953+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f92000543d0 con 0x7f92040b2960 2023-06-10T18:55:01.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.138+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f92040cfd10 con 0x7f92040b2960 2023-06-10T18:55:01.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.142+0000 7f91f9ffb700 1 -- 172.21.15.104:0/973955642 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f92000579f0 con 0x7f92040b2960 2023-06-10T18:55:01.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 msgr2=0x7f91e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f92040ca810 tx=0x7f91f4009040 comp rx=0 tx=0).stop 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 msgr2=0x7f92040c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f9200007340 tx=0x7f9200007650 comp rx=0 tx=0).stop 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 shutdown_connections 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91e806ace0 0x7f91e806d190 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92041531e0 0x7f92040b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92040c8da0 0x7f92040c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 --2- 172.21.15.104:0/973955642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92040b2960 0x7f92040c8860 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.144+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 >> 172.21.15.104:0/973955642 conn(0x7f920406e370 msgr2=0x7f92040b3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:01.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.145+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 shutdown_connections 2023-06-10T18:55:01.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.145+0000 7f920b46c700 1 -- 172.21.15.104:0/973955642 wait complete. 2023-06-10T18:55:01.161 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:55:01.162 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:55:01.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:55:01.242 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:55:01.243 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:55:01.243 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:55:01.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:55:01.262 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:55:01.263 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:55:01.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:55:01.282 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:55:01.283 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:55:01.318 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:55:01.318 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:55:01.319 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:55:01.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:55:01.341 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:55:01.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:01.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.472+0000 7f9e758a4700 1 Processor -- start 2023-06-10T18:55:01.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.472+0000 7f9e758a4700 1 -- start start 2023-06-10T18:55:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e758a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015a410 0x7f9e700cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e758a4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e7015e0f0 0x7f9e700cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e758a4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e700cdb00 con 0x7f9e700cd6a0 2023-06-10T18:55:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e758a4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e700d0000 con 0x7f9e7015e0f0 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e758a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700b3610 con 0x7f9e7015a410 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e7015e0f0 0x7f9e700cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9e700cd6a0 0x7f9e700ce8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33848/0 (socket says 172.21.15.104:33848) 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9e7015e0f0 0x7f9e700cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36018/0 (socket says 172.21.15.104:36018) 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6f7fe700 1 -- 172.21.15.104:0/1057118909 learned_addr learned my addr 172.21.15.104:0/1057118909 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:01.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6effd700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015a410 0x7f9e700cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/1057118909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015a410 msgr2=0x7f9e700cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015a410 0x7f9e700cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/1057118909 >> v1:172.21.15.104:6789/0 conn(0x7f9e700cd6a0 legacy=0x7f9e700ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/1057118909 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700b38d0 con 0x7f9e7015e0f0 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6dffb700 1 -- 172.21.15.104:0/1057118909 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050682576 0 0) 0x7f9e700cdb00 con 0x7f9e700cd6a0 2023-06-10T18:55:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.473+0000 7f9e6e7fc700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015e0f0 0x7f9e700cd160 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009840 tx=0x7f9e60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcefc748905390d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e6dffb700 1 -- 172.21.15.104:0/1057118909 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60011830 con 0x7f9e7015e0f0 2023-06-10T18:55:01.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e6dffb700 1 -- 172.21.15.104:0/1057118909 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60011990 con 0x7f9e7015e0f0 2023-06-10T18:55:01.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e6dffb700 1 -- 172.21.15.104:0/1057118909 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60011d20 con 0x7f9e7015e0f0 2023-06-10T18:55:01.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 -- 172.21.15.104:0/1057118909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015e0f0 msgr2=0x7f9e700cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015e0f0 0x7f9e700cd160 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009840 tx=0x7f9e60009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:01.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 -- 172.21.15.104:0/1057118909 shutdown_connections 2023-06-10T18:55:01.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015a410 0x7f9e700cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 --2- 172.21.15.104:0/1057118909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015e0f0 0x7f9e700cd160 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 -- 172.21.15.104:0/1057118909 >> 172.21.15.104:0/1057118909 conn(0x7f9e70069e70 msgr2=0x7f9e7006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:01.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 -- 172.21.15.104:0/1057118909 shutdown_connections 2023-06-10T18:55:01.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.474+0000 7f9e758a4700 1 -- 172.21.15.104:0/1057118909 wait complete. 2023-06-10T18:55:01.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.475+0000 7f9e758a4700 1 Processor -- start 2023-06-10T18:55:01.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.475+0000 7f9e758a4700 1 -- start start 2023-06-10T18:55:01.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 0x7f9e70158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 0x7f9e700c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7006af80 con 0x7f9e7015a410 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015b8c0 con 0x7f9e700cd6a0 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e758a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700c6ee0 con 0x7f9e7015e0f0 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e6e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e6e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36022/0 (socket says 172.21.15.104:36022) 2023-06-10T18:55:01.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/3947337423 learned_addr learned my addr 172.21.15.104:0/3947337423 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e6effd700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 0x7f9e70158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.476+0000 7f9e6f7fe700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 0x7f9e700c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 msgr2=0x7f9e700c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 0x7f9e700c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 msgr2=0x7f9e70158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 0x7f9e70158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 -- 172.21.15.104:0/3947337423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600094a0 con 0x7f9e7015a410 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6effd700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 0x7f9e70158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6f7fe700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 0x7f9e700c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e6e7fc700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9e600090b0 tx=0x7f9e60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600036a0 con 0x7f9e7015a410 2023-06-10T18:55:01.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60003800 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e700c71c0 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60003b70 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.477+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e701598c0 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.478+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e700c76e0 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.479+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9e60003cd0 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.479+0000 7f9e57fff700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 0x7f9e5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.479+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f9e6008a800 con 0x7f9e7015a410 2023-06-10T18:55:01.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.479+0000 7f9e6effd700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 0x7f9e5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.480+0000 7f9e6effd700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 0x7f9e5c06d190 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e640099f0 tx=0x7f9e64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.483+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9e60055a70 con 0x7f9e7015a410 2023-06-10T18:55:01.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.711+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e70074050 con 0x7f9e7015a410 2023-06-10T18:55:01.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.711+0000 7f9e57fff700 1 -- 172.21.15.104:0/3947337423 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f9e60015de0 con 0x7f9e7015a410 2023-06-10T18:55:01.717 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:01.717 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":64,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1601828826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1601828826},{"type":"v1","addr":"172.21.15.114:6836","nonce":1601828826}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":6401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/525849510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":525849510},{"type":"v1","addr":"172.21.15.114:6837","nonce":525849510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57},{"gid":7965,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/868430849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":868430849},{"type":"v1","addr":"172.21.15.104:6834","nonce":868430849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":8718,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1157546614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1157546614},{"type":"v1","addr":"172.21.15.104:6835","nonce":1157546614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[]} 2023-06-10T18:55:01.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 msgr2=0x7f9e5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 0x7f9e5c06d190 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e640099f0 tx=0x7f9e64008040 comp rx=0 tx=0).stop 2023-06-10T18:55:01.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 msgr2=0x7f9e700c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f9e600090b0 tx=0x7f9e60004750 comp rx=0 tx=0).stop 2023-06-10T18:55:01.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 shutdown_connections 2023-06-10T18:55:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9e5c06ace0 0x7f9e5c06d190 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e700cd6a0 0x7f9e70158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7015e0f0 0x7f9e700c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 --2- 172.21.15.104:0/3947337423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e7015a410 0x7f9e700c4220 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 >> 172.21.15.104:0/3947337423 conn(0x7f9e70069e70 msgr2=0x7f9e700b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:01.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 shutdown_connections 2023-06-10T18:55:01.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.714+0000 7f9e758a4700 1 -- 172.21.15.104:0/3947337423 wait complete. 2023-06-10T18:55:01.721 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 64 2023-06-10T18:55:01.730 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:55:01.730 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:55:01.731 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:55:01.731 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:55:01.731 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:55:01.731 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:55:01.731 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:55:01.732 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:55:01.732 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:55:01.732 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:55:01.732 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:55:01.733 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:55:01.733 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:55:01.733 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:55:01.733 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:55:01.734 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:55:01.741 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:55:01.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:55:01.742 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:55:01.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:55:01.744 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:55:01.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:55:01.746 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:55:01.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:55:01.783 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:55:01.784 DEBUG:teuthology.parallel:result is None 2023-06-10T18:55:01.785 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:55:01.785 DEBUG:teuthology.parallel:result is None 2023-06-10T18:55:01.787 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:55:01.787 DEBUG:teuthology.parallel:result is None 2023-06-10T18:55:01.788 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:55:01.788 DEBUG:teuthology.parallel:result is None 2023-06-10T18:55:01.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:55:01.863 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:55:01.863 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:55:01.875 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:55:01.876 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:55:01.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.914+0000 7feff9700700 1 Processor -- start 2023-06-10T18:55:01.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.914+0000 7feff9700700 1 -- start start 2023-06-10T18:55:01.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff9700700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feff406a580 0x7feff415d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff9700700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff415cdf0 con 0x7feff4156740 2023-06-10T18:55:01.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff9700700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff40cf0a0 con 0x7feff40cec20 2023-06-10T18:55:01.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff9700700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff40cf380 con 0x7feff406a580 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feff406a580 0x7feff415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff2ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7feff4156740 0x7feff40cd250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33880/0 (socket says 172.21.15.104:33880) 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff2ffd700 1 -- 172.21.15.104:0/2089130194 learned_addr learned my addr 172.21.15.104:0/2089130194 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff37fe700 1 -- 172.21.15.104:0/2089130194 >> v1:172.21.15.114:6789/0 conn(0x7feff40cec20 legacy=0x7feff4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff37fe700 1 -- 172.21.15.104:0/2089130194 >> v1:172.21.15.104:6789/0 conn(0x7feff4156740 legacy=0x7feff40cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff37fe700 1 -- 172.21.15.104:0/2089130194 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff415ca80 con 0x7feff406a580 2023-06-10T18:55:01.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff1ffb700 1 -- 172.21.15.104:0/2089130194 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172799680 0 0) 0x7feff415cdf0 con 0x7feff4156740 2023-06-10T18:55:01.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff37fe700 1 --2- 172.21.15.104:0/2089130194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff406a580 0x7feff415d280 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fefec009d90 tx=0x7fefec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31da71a6b14dc410 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff1ffb700 1 -- 172.21.15.104:0/2089130194 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00e070 con 0x7feff406a580 2023-06-10T18:55:01.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff1ffb700 1 -- 172.21.15.104:0/2089130194 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec00bbd0 con 0x7feff406a580 2023-06-10T18:55:01.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.915+0000 7feff1ffb700 1 -- 172.21.15.104:0/2089130194 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec012430 con 0x7feff406a580 2023-06-10T18:55:01.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 -- 172.21.15.104:0/2089130194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff406a580 msgr2=0x7feff415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 --2- 172.21.15.104:0/2089130194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff406a580 0x7feff415d280 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fefec009d90 tx=0x7fefec00b330 comp rx=0 tx=0).stop 2023-06-10T18:55:01.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 -- 172.21.15.104:0/2089130194 shutdown_connections 2023-06-10T18:55:01.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 --2- 172.21.15.104:0/2089130194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff406a580 0x7feff415d280 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 -- 172.21.15.104:0/2089130194 >> 172.21.15.104:0/2089130194 conn(0x7feff406d240 msgr2=0x7feff406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 -- 172.21.15.104:0/2089130194 shutdown_connections 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.916+0000 7feff9700700 1 -- 172.21.15.104:0/2089130194 wait complete. 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.917+0000 7feff9700700 1 Processor -- start 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.917+0000 7feff9700700 1 -- start start 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 0x7feff40b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 0x7feff40cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40b4790 con 0x7feff4156740 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40b3b10 con 0x7feff414dd20 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff9700700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff40b3df0 con 0x7feff406a580 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36060/0 (socket says 172.21.15.104:36060) 2023-06-10T18:55:01.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 -- 172.21.15.104:0/2314829116 learned_addr learned my addr 172.21.15.104:0/2314829116 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff2ffd700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 0x7feff40b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff37fe700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 0x7feff40cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 -- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 msgr2=0x7feff40b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 0x7feff40b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 -- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 msgr2=0x7feff40cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 0x7feff40cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:01.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 -- 172.21.15.104:0/2314829116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec009a70 con 0x7feff4156740 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff2ffd700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 0x7feff40b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff37fe700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 0x7feff40cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.918+0000 7feff27fc700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c4e0 tx=0x7fefe400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe40052d0 con 0x7feff4156740 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe4005430 con 0x7feff4156740 2023-06-10T18:55:01.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff40b40d0 con 0x7feff4156740 2023-06-10T18:55:01.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe40038c0 con 0x7feff4156740 2023-06-10T18:55:01.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff414e1c0 con 0x7feff4156740 2023-06-10T18:55:01.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.919+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff40b48d0 con 0x7feff4156740 2023-06-10T18:55:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.920+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fefe4003a20 con 0x7feff4156740 2023-06-10T18:55:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.920+0000 7fefdb7fe700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 0x7fefe006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:01.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.921+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fefe40887e0 con 0x7feff4156740 2023-06-10T18:55:01.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.921+0000 7feff2ffd700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 0x7fefe006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:01.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.921+0000 7feff2ffd700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 0x7fefe006d190 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fefe8000f60 tx=0x7fefe8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:01.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:01.925+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feff40b48d0 con 0x7feff4156740 2023-06-10T18:55:02.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.126+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7feff40b48d0 con 0x7feff4156740 2023-06-10T18:55:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.905+0000 7fefdb7fe700 1 -- 172.21.15.104:0/2314829116 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v109) v1 ==== 97+0+0 (secure 0 0 0) 0x7feff40b48d0 con 0x7feff4156740 2023-06-10T18:55:02.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 msgr2=0x7fefe006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 0x7fefe006d190 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fefe8000f60 tx=0x7fefe8009040 comp rx=0 tx=0).stop 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 msgr2=0x7feff41524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c4e0 tx=0x7fefe400c8a0 comp rx=0 tx=0).stop 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 shutdown_connections 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fefe006ace0 0x7fefe006d190 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:02.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feff414dd20 0x7feff40cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feff406a580 0x7feff40b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 --2- 172.21.15.104:0/2314829116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feff4156740 0x7feff41524d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 >> 172.21.15.104:0/2314829116 conn(0x7feff406d240 msgr2=0x7feff40cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 shutdown_connections 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:02.907+0000 7feff9700700 1 -- 172.21.15.104:0/2314829116 wait complete. 2023-06-10T18:55:02.914 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:55:02.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:55:03.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 Processor -- start 2023-06-10T18:55:03.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 -- start start 2023-06-10T18:55:03.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcdf406e4b0 0x7fcdf406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 0x7fcdf40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf406d290 con 0x7fcdf406e0e0 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf4071640 con 0x7fcdf406e4b0 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.046+0000 7fcdf9ba5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40aef00 con 0x7fcdf4069e70 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf3fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 0x7fcdf40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcdf406e4b0 0x7fcdf406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcdf406e4b0 0x7fcdf406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:51806/0 (socket says 172.21.15.104:51806) 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf3fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 0x7fcdf40ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55458/0 (socket says 172.21.15.104:55458) 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/1910618014 learned_addr learned my addr 172.21.15.104:0/1910618014 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf37fe700 1 --1- 172.21.15.104:0/1910618014 >> v1:172.21.15.114:6789/0 conn(0x7fcdf406e0e0 0x7fcdf40c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39744/0 (socket says 172.21.15.104:39744) 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 msgr2=0x7fcdf40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 --2- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 0x7fcdf40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/1910618014 >> v1:172.21.15.114:6789/0 conn(0x7fcdf406e0e0 legacy=0x7fcdf40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/1910618014 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40ca2d0 con 0x7fcdf406e4b0 2023-06-10T18:55:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.047+0000 7fcdf2ffd700 1 --2- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf406e4b0 0x7fcdf406cd50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcde80098f0 tx=0x7fcde8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb6af9632ac4954 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf27fc700 1 -- 172.21.15.104:0/1910618014 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8011830 con 0x7fcdf406e4b0 2023-06-10T18:55:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf27fc700 1 -- 172.21.15.104:0/1910618014 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde8011990 con 0x7fcdf406e4b0 2023-06-10T18:55:03.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf27fc700 1 -- 172.21.15.104:0/1910618014 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8011d20 con 0x7fcdf406e4b0 2023-06-10T18:55:03.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf406e4b0 msgr2=0x7fcdf406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf406e4b0 0x7fcdf406cd50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcde80098f0 tx=0x7fcde8009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:03.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/1910618014 shutdown_connections 2023-06-10T18:55:03.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf4069e70 0x7fcdf40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/1910618014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf406e4b0 0x7fcdf406cd50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/1910618014 >> 172.21.15.104:0/1910618014 conn(0x7fcdf406b8a0 msgr2=0x7fcdf40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/1910618014 shutdown_connections 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.048+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/1910618014 wait complete. 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.050+0000 7fcdf9ba5700 1 Processor -- start 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.050+0000 7fcdf9ba5700 1 -- start start 2023-06-10T18:55:03.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4069e70 0x7fcdf40b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf406e4b0 0x7fcdf40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40b0670 con 0x7fcdf406e0e0 2023-06-10T18:55:03.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40b0950 con 0x7fcdf4069e70 2023-06-10T18:55:03.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf9ba5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40b0c30 con 0x7fcdf406e4b0 2023-06-10T18:55:03.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36078/0 (socket says 172.21.15.104:36078) 2023-06-10T18:55:03.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/3851057822 learned_addr learned my addr 172.21.15.104:0/3851057822 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:03.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf37fe700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4069e70 0x7fcdf40b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf3fff700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf406e4b0 0x7fcdf40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf406e4b0 msgr2=0x7fcdf40b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf406e4b0 0x7fcdf40b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4069e70 msgr2=0x7fcdf40b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4069e70 0x7fcdf40b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 -- 172.21.15.104:0/3851057822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8009550 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcdf2ffd700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009c40 tx=0x7fcde8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.051+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde80036a0 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.052+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde8003800 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.052+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf40b0f10 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.052+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8003bb0 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.052+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf4148ee0 con 0x7fcdf406e0e0 2023-06-10T18:55:03.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.053+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf40b1430 con 0x7fcdf406e0e0 2023-06-10T18:55:03.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.053+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcde8003d10 con 0x7fcdf406e0e0 2023-06-10T18:55:03.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.053+0000 7fcddbfff700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 0x7fcde006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.053+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fcde808a4c0 con 0x7fcdf406e0e0 2023-06-10T18:55:03.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.053+0000 7fcdf37fe700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 0x7fcde006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.054+0000 7fcdf37fe700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 0x7fcde006d190 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcde4000fb0 tx=0x7fcde4007ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.057+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcdf40b1430 con 0x7fcdf406e0e0 2023-06-10T18:55:03.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.319+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcdf40b1430 con 0x7fcdf406e0e0 2023-06-10T18:55:03.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.320+0000 7fcddbfff700 1 -- 172.21.15.104:0/3851057822 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcdf40b1430 con 0x7fcdf406e0e0 2023-06-10T18:55:03.325 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:03.325 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:55:03.325 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:55:03.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:55:03.327 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.328 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:55:03.329 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.330 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:55:03.331 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.334 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.335 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:55:03.336 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.337 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:55:03.340 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.341 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:55:03.342 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:55:03.344 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:55:03.345 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:55:03.346 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:55:03.349 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:55:03.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:03.350 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:55:03.350 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:55:03.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 msgr2=0x7fcde006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 0x7fcde006d190 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fcde4000fb0 tx=0x7fcde4007ce0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 msgr2=0x7fcdf41489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009c40 tx=0x7fcde8004750 comp rx=0 tx=0).stop 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 shutdown_connections 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcde006ace0 0x7fcde006d190 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4069e70 0x7fcdf40b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcdf406e4b0 0x7fcdf40b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 --2- 172.21.15.104:0/3851057822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcdf406e0e0 0x7fcdf41489a0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 >> 172.21.15.104:0/3851057822 conn(0x7fcdf406b8a0 msgr2=0x7fcdf40c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:03.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 shutdown_connections 2023-06-10T18:55:03.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.322+0000 7fcdf9ba5700 1 -- 172.21.15.104:0/3851057822 wait complete. 2023-06-10T18:55:03.354 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:55:03.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.470+0000 7f4406135700 1 Processor -- start 2023-06-10T18:55:03.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.470+0000 7f4406135700 1 -- start start 2023-06-10T18:55:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f4406135700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f44000cea60 0x7f4400165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f4406135700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000b6070 con 0x7f44000b4ff0 2023-06-10T18:55:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f4406135700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000b1d70 con 0x7f440015a420 2023-06-10T18:55:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f4406135700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000b5460 con 0x7f44000cea60 2023-06-10T18:55:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f43feffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f44000b4ff0 0x7f44000ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33912/0 (socket says 172.21.15.104:33912) 2023-06-10T18:55:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.471+0000 7f43feffd700 1 -- 172.21.15.104:0/754984136 learned_addr learned my addr 172.21.15.104:0/754984136 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437853898 0 0) 0x7f44000b6070 con 0x7f44000b4ff0 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ec0035b0 con 0x7f44000b4ff0 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fffff700 1 --2- 172.21.15.104:0/754984136 >> v2:172.21.15.114:3300/0 conn(0x7f44000cea60 0x7f4400165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2171292592 0 0) 0x7f43ec0035b0 con 0x7f44000b4ff0 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 >> v2:172.21.15.114:3300/0 conn(0x7f44000cea60 msgr2=0x7f4400165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 --2- 172.21.15.104:0/754984136 >> v2:172.21.15.114:3300/0 conn(0x7f44000cea60 0x7f4400165060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 >> v1:172.21.15.114:6789/0 conn(0x7f440015a420 legacy=0x7f44000b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000b5740 con 0x7f44000b4ff0 2023-06-10T18:55:03.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f43f0001670 con 0x7f44000b4ff0 2023-06-10T18:55:03.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 >> v1:172.21.15.104:6789/0 conn(0x7f44000b4ff0 legacy=0x7f44000ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43ec003a00 0x7f43ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f43ec004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43ec005200 0x7f43ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b5460 con 0x7f43ec004850 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec00c080 con 0x7f43ec005200 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec00c240 con 0x7f43ec003a00 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f43ec004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43ff7fe700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43ec003a00 0x7f43ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fffff700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43ec005200 0x7f43ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 -- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43ec003a00 msgr2=0x7f43ec003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43ec003a00 0x7f43ec003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 -- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43ec005200 msgr2=0x7f43ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43ec005200 0x7f43ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 -- 172.21.15.104:0/754984136 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec0035b0 con 0x7f43ec004850 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43fffff700 1 -- 172.21.15.104:0/754984136 reap_dead start 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43ff7fe700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43ec003a00 0x7f43ec003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:03.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.472+0000 7f43feffd700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f43ec004cc0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001050 tx=0x7f43ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e756075be4b598a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000cd00 con 0x7f43ec004850 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f000ce60 con 0x7f43ec004850 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f43fe7fc700 1 -- 172.21.15.104:0/754984136 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0005ba0 con 0x7f43ec004850 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 -- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 msgr2=0x7f43ec004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f43ec004cc0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f43f0001050 tx=0x7f43ec005880 comp rx=0 tx=0).stop 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 -- 172.21.15.104:0/754984136 shutdown_connections 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 --2- 172.21.15.104:0/754984136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f43ec004cc0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 -- 172.21.15.104:0/754984136 >> 172.21.15.104:0/754984136 conn(0x7f4400089fd0 msgr2=0x7f44000d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:03.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 -- 172.21.15.104:0/754984136 shutdown_connections 2023-06-10T18:55:03.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.473+0000 7f4406135700 1 -- 172.21.15.104:0/754984136 wait complete. 2023-06-10T18:55:03.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.474+0000 7f4406135700 1 Processor -- start 2023-06-10T18:55:03.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 -- start start 2023-06-10T18:55:03.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44000c9a90 0x7f44001e4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 0x7f44000cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000cac10 con 0x7f43ec004850 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000cd340 con 0x7f44000cc990 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f4406135700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44001576a0 con 0x7f44000c9a90 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36094/0 (socket says 172.21.15.104:36094) 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.475+0000 7f43ff7fe700 1 -- 172.21.15.104:0/2641684135 learned_addr learned my addr 172.21.15.104:0/2641684135 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 -- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44000c9a90 msgr2=0x7f44001e4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43fffff700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 0x7f44000cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43feffd700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44000c9a90 0x7f44001e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44000c9a90 0x7f44001e4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 -- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 msgr2=0x7f44000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 0x7f44000cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 -- 172.21.15.104:0/2641684135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000ca000 con 0x7f43ec004850 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43fffff700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 0x7f44000cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43ff7fe700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f43e800e460 tx=0x7f43e800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80052e0 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8005440 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000ca2e0 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80038c0 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.476+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44000ca8a0 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.477+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44000caa60 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.477+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f43e8003a20 con 0x7f43ec004850 2023-06-10T18:55:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.478+0000 7f43f7fff700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 0x7f43e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.478+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f43e80888d0 con 0x7f43ec004850 2023-06-10T18:55:03.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.478+0000 7f43feffd700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 0x7f43e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.478+0000 7f43feffd700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 0x7f43e006d190 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f440009e6c0 tx=0x7f43f000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.482+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43e8053c30 con 0x7f43ec004850 2023-06-10T18:55:03.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.674+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44001e1b40 con 0x7f43ec004850 2023-06-10T18:55:03.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.674+0000 7f43f7fff700 1 -- 172.21.15.104:0/2641684135 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+14894 (secure 0 0 0) 0x7f43e8057250 con 0x7f43ec004850 2023-06-10T18:55:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:03.680 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":109,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:02.898545+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 msgr2=0x7f43e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 0x7f43e006d190 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f440009e6c0 tx=0x7f43f000a000 comp rx=0 tx=0).stop 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 msgr2=0x7f4400157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f43e800e460 tx=0x7f43e800e820 comp rx=0 tx=0).stop 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 shutdown_connections 2023-06-10T18:55:03.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43e006ace0 0x7f43e006d190 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44000cc990 0x7f44000cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43ec004850 0x7f4400157160 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 --2- 172.21.15.104:0/2641684135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44000c9a90 0x7f44001e4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 >> 172.21.15.104:0/2641684135 conn(0x7f4400089fd0 msgr2=0x7f440006fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 shutdown_connections 2023-06-10T18:55:03.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.677+0000 7f4406135700 1 -- 172.21.15.104:0/2641684135 wait complete. 2023-06-10T18:55:03.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:55:03.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:55:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.812+0000 7f99ff979700 1 Processor -- start 2023-06-10T18:55:03.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.812+0000 7f99ff979700 1 -- start start 2023-06-10T18:55:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.813+0000 7f99ff979700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.813+0000 7f99ff979700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.813+0000 7f99ff979700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f806e250 con 0x7f99f815c580 2023-06-10T18:55:03.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.813+0000 7f99ff979700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f815c950 con 0x7f99f806e6e0 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.813+0000 7f99ff979700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80b1e80 con 0x7f99f806cde0 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fcf14700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55490/0 (socket says 172.21.15.104:55490) 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fd715700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f99f815c580 0x7f99f80b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39776/0 (socket says 172.21.15.104:39776) 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fcf14700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:51838/0 (socket says 172.21.15.104:51838) 2023-06-10T18:55:03.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 -- 172.21.15.104:0/3521607286 learned_addr learned my addr 172.21.15.104:0/3521607286 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 -- 172.21.15.104:0/3521607286 >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 msgr2=0x7f99f80b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 --2- 172.21.15.104:0/3521607286 >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 -- 172.21.15.104:0/3521607286 >> v1:172.21.15.114:6789/0 conn(0x7f99f815c580 legacy=0x7f99f80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 -- 172.21.15.104:0/3521607286 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f806d920 con 0x7f99f806cde0 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99effff700 1 -- 172.21.15.104:0/3521607286 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884574468 0 0) 0x7f99f806e250 con 0x7f99f815c580 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fcf14700 1 --2- 172.21.15.104:0/3521607286 >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99fdf16700 1 --2- 172.21.15.104:0/3521607286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009e50 tx=0x7f99f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99099cd3a80a39ce server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99effff700 1 -- 172.21.15.104:0/3521607286 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f000e070 con 0x7f99f806cde0 2023-06-10T18:55:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99effff700 1 -- 172.21.15.104:0/3521607286 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f000b910 con 0x7f99f806cde0 2023-06-10T18:55:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.814+0000 7f99effff700 1 -- 172.21.15.104:0/3521607286 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f000bcc0 con 0x7f99f806cde0 2023-06-10T18:55:03.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 -- 172.21.15.104:0/3521607286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 msgr2=0x7f99f80ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 --2- 172.21.15.104:0/3521607286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009e50 tx=0x7f99f000b330 comp rx=0 tx=0).stop 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 -- 172.21.15.104:0/3521607286 shutdown_connections 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 --2- 172.21.15.104:0/3521607286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806cde0 0x7f99f80ce0d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 --2- 172.21.15.104:0/3521607286 >> v2:172.21.15.114:3300/0 conn(0x7f99f806e6e0 0x7f99f80b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 -- 172.21.15.104:0/3521607286 >> 172.21.15.104:0/3521607286 conn(0x7f99f806b8a0 msgr2=0x7f99f80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 -- 172.21.15.104:0/3521607286 shutdown_connections 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.815+0000 7f99ff979700 1 -- 172.21.15.104:0/3521607286 wait complete. 2023-06-10T18:55:03.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.816+0000 7f99ff979700 1 Processor -- start 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.816+0000 7f99ff979700 1 -- start start 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f806cde0 0x7f99f814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 0x7f99f80c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80b2b60 con 0x7f99f815c580 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80b4dd0 con 0x7f99f806cde0 2023-06-10T18:55:03.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99ff979700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80c6ef0 con 0x7f99f806e6e0 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36112/0 (socket says 172.21.15.104:36112) 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 -- 172.21.15.104:0/4160886902 learned_addr learned my addr 172.21.15.104:0/4160886902 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fd715700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f806cde0 0x7f99f814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fcf14700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 0x7f99f80c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 -- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 msgr2=0x7f99f80c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 0x7f99f80c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 -- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f806cde0 msgr2=0x7f99f814d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f806cde0 0x7f99f814d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 -- 172.21.15.104:0/4160886902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f40094a0 con 0x7f99f815c580 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fcf14700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 0x7f99f80c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.817+0000 7f99fdf16700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f99f80b1fc0 tx=0x7f99f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.818+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f00092e0 con 0x7f99f815c580 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.818+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f00099b0 con 0x7f99f815c580 2023-06-10T18:55:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.818+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0009b30 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.818+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f001c680 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.818+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f81f1c90 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.819+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f80b4f10 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.819+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f99f001c7e0 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.819+0000 7f99edffb700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 0x7f99dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.820+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f99f008ac60 con 0x7f99f815c580 2023-06-10T18:55:03.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.820+0000 7f99fd715700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 0x7f99dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.820+0000 7f99fd715700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 0x7f99dc06d190 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f99f80c8840 tx=0x7f99e8007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:03.824+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f99f80b4f10 con 0x7f99f815c580 2023-06-10T18:55:04.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.009+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f99f80b4f10 con 0x7f99f815c580 2023-06-10T18:55:04.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.009+0000 7f99edffb700 1 -- 172.21.15.104:0/4160886902 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99f80b4f10 con 0x7f99f815c580 2023-06-10T18:55:04.015 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:55:04.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 msgr2=0x7f99dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 0x7f99dc06d190 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f99f80c8840 tx=0x7f99e8007d20 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 msgr2=0x7f99f81f1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f99f80b1fc0 tx=0x7f99f0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 shutdown_connections 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f99dc06ace0 0x7f99dc06d190 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f806cde0 0x7f99f814d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99f815c580 0x7f99f81f1750 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 --2- 172.21.15.104:0/4160886902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f99f806e6e0 0x7f99f80c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 >> 172.21.15.104:0/4160886902 conn(0x7f99f806b8a0 msgr2=0x7f99f81e17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 shutdown_connections 2023-06-10T18:55:04.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.011+0000 7f99ff979700 1 -- 172.21.15.104:0/4160886902 wait complete. 2023-06-10T18:55:04.028 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:55:04.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.145+0000 7fa4c7a45700 1 Processor -- start 2023-06-10T18:55:04.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.146+0000 7fa4c7a45700 1 -- start start 2023-06-10T18:55:04.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c7a45700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4c00b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c7a45700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c006cde0 0x7fa4c00ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c7a45700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c006e250 con 0x7fa4c015c580 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c7a45700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c015c950 con 0x7fa4c006cde0 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c7a45700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00b1e80 con 0x7fa4c006e6e0 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c57e1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa4c015c580 0x7fa4c00b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33944/0 (socket says 172.21.15.104:33944) 2023-06-10T18:55:04.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c57e1700 1 -- 172.21.15.104:0/4278779463 learned_addr learned my addr 172.21.15.104:0/4278779463 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c5fe2700 1 --2- 172.21.15.104:0/4278779463 >> v2:172.21.15.114:3300/0 conn(0x7fa4c006cde0 0x7fa4c00ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4c00b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168617788 0 0) 0x7fa4c006e250 con 0x7fa4c015c580 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ac0035b0 con 0x7fa4c015c580 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133744882 0 0) 0x7fa4ac0035b0 con 0x7fa4c015c580 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 msgr2=0x7fa4c00b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4c00b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 >> v2:172.21.15.114:3300/0 conn(0x7fa4c006cde0 msgr2=0x7fa4c00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 --2- 172.21.15.104:0/4278779463 >> v2:172.21.15.114:3300/0 conn(0x7fa4c006cde0 0x7fa4c00ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c006d920 con 0x7fa4c015c580 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa4b00017f0 con 0x7fa4c015c580 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c5fe2700 1 --2- 172.21.15.104:0/4278779463 >> v2:172.21.15.114:3300/0 conn(0x7fa4c006cde0 0x7fa4c00ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4c00b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 >> v1:172.21.15.104:6789/0 conn(0x7fa4c015c580 legacy=0x7fa4c00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4ac004430 0x7fa4ac0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4ac004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c015c950 con 0x7fa4ac003a00 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00b1e80 con 0x7fa4ac004430 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.147+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac00ba10 con 0x7fa4c006e6e0 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4ac004430 0x7fa4ac0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c5fe2700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4ac004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 -- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 msgr2=0x7fa4ac004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4ac004f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 -- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4ac004430 msgr2=0x7fa4ac0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4ac004430 0x7fa4ac0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 -- 172.21.15.104:0/4278779463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0094a0 con 0x7fa4ac003a00 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c5fe2700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c006e6e0 0x7fa4ac004f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4ac003e10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000c320 tx=0x7fa4ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35740d24d64675d3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b000ce60 con 0x7fa4ac003a00 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0007bf0 con 0x7fa4ac003a00 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.148+0000 7fa4b7fff700 1 -- 172.21.15.104:0/4278779463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00072f0 con 0x7fa4ac003a00 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 -- 172.21.15.104:0/4278779463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 msgr2=0x7fa4ac003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/4278779463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4ac003e10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000c320 tx=0x7fa4ac005880 comp rx=0 tx=0).stop 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c5fe2700 1 -- 172.21.15.104:0/4278779463 reap_dead start 2023-06-10T18:55:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 -- 172.21.15.104:0/4278779463 shutdown_connections 2023-06-10T18:55:04.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 -- 172.21.15.104:0/4278779463 >> 172.21.15.104:0/4278779463 conn(0x7fa4c006b8a0 msgr2=0x7fa4c00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 -- 172.21.15.104:0/4278779463 shutdown_connections 2023-06-10T18:55:04.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.149+0000 7fa4c7a45700 1 -- 172.21.15.104:0/4278779463 wait complete. 2023-06-10T18:55:04.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.150+0000 7fa4c7a45700 1 Processor -- start 2023-06-10T18:55:04.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.150+0000 7fa4c7a45700 1 -- start start 2023-06-10T18:55:04.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 0x7fa4c00b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c0159aa0 0x7fa4c00c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00c6d20 con 0x7fa4ac003a00 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00c6f70 con 0x7fa4c00c7d30 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00c7250 con 0x7fa4c0159aa0 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36128/0 (socket says 172.21.15.104:36128) 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 -- 172.21.15.104:0/3595622910 learned_addr learned my addr 172.21.15.104:0/3595622910 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:04.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 0x7fa4c00b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c5fe2700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c0159aa0 0x7fa4c00c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 -- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c0159aa0 msgr2=0x7fa4c00c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c0159aa0 0x7fa4c00c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 -- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 msgr2=0x7fa4c00b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 0x7fa4c00b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 -- 172.21.15.104:0/3595622910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c00c7530 con 0x7fa4ac003a00 2023-06-10T18:55:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 0x7fa4c00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c57e1700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0015040 tx=0x7fa4b0001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00072d0 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0007450 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c00c7810 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0007800 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.151+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c0159f40 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.152+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c006dbe0 con 0x7fa4ac003a00 2023-06-10T18:55:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.153+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa4b001e430 con 0x7fa4ac003a00 2023-06-10T18:55:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.153+0000 7fa4b5ffb700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.153+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fa4b008a8e0 con 0x7fa4ac003a00 2023-06-10T18:55:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.154+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.154+0000 7fa4c4fe0700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 0x7fa4a406d190 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00add90 tx=0x7fa4bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.158+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa4c006dbe0 con 0x7fa4ac003a00 2023-06-10T18:55:04.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.343+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa4c006dbe0 con 0x7fa4ac003a00 2023-06-10T18:55:04.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.344+0000 7fa4b5ffb700 1 -- 172.21.15.104:0/3595622910 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v109) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa4c006dbe0 con 0x7fa4ac003a00 2023-06-10T18:55:04.349 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:55:04.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 msgr2=0x7fa4a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 0x7fa4a406d190 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00add90 tx=0x7fa4bc008040 comp rx=0 tx=0).stop 2023-06-10T18:55:04.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 msgr2=0x7fa4c00b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0015040 tx=0x7fa4b0001bf0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 shutdown_connections 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4c0159aa0 0x7fa4c00c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4ac003a00 0x7fa4c00b4f70 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 --2- 172.21.15.104:0/3595622910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c00c7d30 0x7fa4c00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 >> 172.21.15.104:0/3595622910 conn(0x7fa4c006b8a0 msgr2=0x7fa4c00cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 shutdown_connections 2023-06-10T18:55:04.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.346+0000 7fa4c7a45700 1 -- 172.21.15.104:0/3595622910 wait complete. 2023-06-10T18:55:04.363 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:55:04.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.478+0000 7f14f43ea700 1 Processor -- start 2023-06-10T18:55:04.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.478+0000 7f14f43ea700 1 -- start start 2023-06-10T18:55:04.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f43ea700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f14ec0b1070 0x7f14ec0b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f43ea700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec06e5e0 con 0x7f14ec0aec00 2023-06-10T18:55:04.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f43ea700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec0b1980 con 0x7f14ec06ea70 2023-06-10T18:55:04.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f43ea700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec0ae7e0 con 0x7f14ec0b1070 2023-06-10T18:55:04.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f2186700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f14ec0b1070 0x7f14ec0b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f2186700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f14ec0b1070 0x7f14ec0b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36136/0 (socket says 172.21.15.104:36136) 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f2186700 1 -- 172.21.15.104:0/2698105745 learned_addr learned my addr 172.21.15.104:0/2698105745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.479+0000 7f14f2987700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f14ec0aec00 0x7f14ec0ac8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33966/0 (socket says 172.21.15.104:33966) 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f2186700 1 -- 172.21.15.104:0/2698105745 >> v1:172.21.15.114:6789/0 conn(0x7f14ec06ea70 legacy=0x7f14ec0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f2186700 1 -- 172.21.15.104:0/2698105745 >> v1:172.21.15.104:6789/0 conn(0x7f14ec0aec00 legacy=0x7f14ec0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f1184700 1 -- 172.21.15.104:0/2698105745 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952584386 0 0) 0x7f14ec06e5e0 con 0x7f14ec0aec00 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f2186700 1 -- 172.21.15.104:0/2698105745 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec06df90 con 0x7f14ec0b1070 2023-06-10T18:55:04.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f2186700 1 --2- 172.21.15.104:0/2698105745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0b1070 0x7f14ec0b1440 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f14dc006fd0 tx=0x7f14dc00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a512db3c5dbc5af server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f1184700 1 -- 172.21.15.104:0/2698105745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc005e30 con 0x7f14ec0b1070 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f1184700 1 -- 172.21.15.104:0/2698105745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc004030 con 0x7f14ec0b1070 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.480+0000 7f14f1184700 1 -- 172.21.15.104:0/2698105745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc0043e0 con 0x7f14ec0b1070 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 -- 172.21.15.104:0/2698105745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0b1070 msgr2=0x7f14ec0b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 --2- 172.21.15.104:0/2698105745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0b1070 0x7f14ec0b1440 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f14dc006fd0 tx=0x7f14dc00be70 comp rx=0 tx=0).stop 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 -- 172.21.15.104:0/2698105745 shutdown_connections 2023-06-10T18:55:04.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 --2- 172.21.15.104:0/2698105745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0b1070 0x7f14ec0b1440 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 -- 172.21.15.104:0/2698105745 >> 172.21.15.104:0/2698105745 conn(0x7f14ec06b8a0 msgr2=0x7f14ec0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 -- 172.21.15.104:0/2698105745 shutdown_connections 2023-06-10T18:55:04.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.481+0000 7f14f43ea700 1 -- 172.21.15.104:0/2698105745 wait complete. 2023-06-10T18:55:04.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.482+0000 7f14f43ea700 1 Processor -- start 2023-06-10T18:55:04.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.482+0000 7f14f43ea700 1 -- start start 2023-06-10T18:55:04.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 0x7f14ec1ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14ec0b1070 0x7f14ec1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1532e0 con 0x7f14ec0aec00 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec09bbc0 con 0x7f14ec06ea70 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec09bea0 con 0x7f14ec0b1070 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36144/0 (socket says 172.21.15.104:36144) 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 -- 172.21.15.104:0/3784154802 learned_addr learned my addr 172.21.15.104:0/3784154802 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f2186700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 0x7f14ec1ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 -- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14ec0b1070 msgr2=0x7f14ec1582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f2987700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14ec0b1070 0x7f14ec1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14ec0b1070 0x7f14ec1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 -- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 msgr2=0x7f14ec1ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 0x7f14ec1ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 -- 172.21.15.104:0/3784154802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc00b7a0 con 0x7f14ec0aec00 2023-06-10T18:55:04.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f2186700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 0x7f14ec1ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:04.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f1985700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f14e800c450 tx=0x7f14e800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80052d0 con 0x7f14ec0aec00 2023-06-10T18:55:04.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e8005430 con 0x7f14ec0aec00 2023-06-10T18:55:04.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec09c180 con 0x7f14ec0aec00 2023-06-10T18:55:04.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e80038c0 con 0x7f14ec0aec00 2023-06-10T18:55:04.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.483+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec09c740 con 0x7f14ec0aec00 2023-06-10T18:55:04.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.484+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec09c960 con 0x7f14ec0aec00 2023-06-10T18:55:04.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.485+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f14e8003a20 con 0x7f14ec0aec00 2023-06-10T18:55:04.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.485+0000 7f14e2ffd700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 0x7f14d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.485+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f14e8088580 con 0x7f14ec0aec00 2023-06-10T18:55:04.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.485+0000 7f14f2186700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 0x7f14d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.485+0000 7f14f2186700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 0x7f14d006d190 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00beb0 tx=0x7f14dc01b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.489+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f14e80538d0 con 0x7f14ec0aec00 2023-06-10T18:55:04.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.678+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f14ec074050 con 0x7f14ec0aec00 2023-06-10T18:55:04.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.679+0000 7f14e2ffd700 1 -- 172.21.15.104:0/3784154802 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+126 (secure 0 0 0) 0x7f14e8056ef0 con 0x7f14ec0aec00 2023-06-10T18:55:04.684 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:04.684 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:55:04.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:04.684 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:55:04.684 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:04.685 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:55:04.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 msgr2=0x7f14d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 0x7f14d006d190 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00beb0 tx=0x7f14dc01b040 comp rx=0 tx=0).stop 2023-06-10T18:55:04.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 msgr2=0x7f14ec157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f14e800c450 tx=0x7f14e800c810 comp rx=0 tx=0).stop 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 shutdown_connections 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f14d006ace0 0x7f14d006d190 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14ec06ea70 0x7f14ec1ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14ec0b1070 0x7f14ec1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.680+0000 7f14f43ea700 1 --2- 172.21.15.104:0/3784154802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14ec0aec00 0x7f14ec157d60 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.681+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 >> 172.21.15.104:0/3784154802 conn(0x7f14ec06b8a0 msgr2=0x7f14ec0af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.681+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 shutdown_connections 2023-06-10T18:55:04.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.681+0000 7f14f43ea700 1 -- 172.21.15.104:0/3784154802 wait complete. 2023-06-10T18:55:04.697 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:55:04.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:55:04.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.817+0000 7fcff4058700 1 Processor -- start 2023-06-10T18:55:04.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.817+0000 7fcff4058700 1 -- start start 2023-06-10T18:55:04.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff4058700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcfec0d23f0 0x7fcfec0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff4058700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff4058700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec145ba0 con 0x7fcfec0cec90 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff4058700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfec0d0370 con 0x7fcfec0d0a60 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff4058700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfec0d0e60 con 0x7fcfec0d23f0 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36154/0 (socket says 172.21.15.104:36154) 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 -- 172.21.15.104:0/504263522 learned_addr learned my addr 172.21.15.104:0/504263522 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff15f3700 1 --2- 172.21.15.104:0/504263522 >> v2:172.21.15.114:3300/0 conn(0x7fcfec0d23f0 0x7fcfec0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 -- 172.21.15.104:0/504263522 >> v2:172.21.15.114:3300/0 conn(0x7fcfec0d23f0 msgr2=0x7fcfec0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 --2- 172.21.15.104:0/504263522 >> v2:172.21.15.114:3300/0 conn(0x7fcfec0d23f0 0x7fcfec0d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 -- 172.21.15.104:0/504263522 >> v1:172.21.15.114:6789/0 conn(0x7fcfec0cec90 legacy=0x7fcfec0cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.818+0000 7fcff25f5700 1 -- 172.21.15.104:0/504263522 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec0d2890 con 0x7fcfec0d0a60 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff25f5700 1 --2- 172.21.15.104:0/504263522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0b19e0 tx=0x7fcfe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f685339b6c087b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff0df2700 1 -- 172.21.15.104:0/504263522 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400e070 con 0x7fcfec0d0a60 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff0df2700 1 -- 172.21.15.104:0/504263522 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe4003a20 con 0x7fcfec0d0a60 2023-06-10T18:55:04.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff0df2700 1 -- 172.21.15.104:0/504263522 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4003dd0 con 0x7fcfec0d0a60 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 -- 172.21.15.104:0/504263522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d0a60 msgr2=0x7fcfec0ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 --2- 172.21.15.104:0/504263522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0b19e0 tx=0x7fcfe400b330 comp rx=0 tx=0).stop 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 -- 172.21.15.104:0/504263522 shutdown_connections 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 --2- 172.21.15.104:0/504263522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d0a60 0x7fcfec0ccfa0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 --2- 172.21.15.104:0/504263522 >> v2:172.21.15.114:3300/0 conn(0x7fcfec0d23f0 0x7fcfec0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 -- 172.21.15.104:0/504263522 >> 172.21.15.104:0/504263522 conn(0x7fcfec071ec0 msgr2=0x7fcfec09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 -- 172.21.15.104:0/504263522 shutdown_connections 2023-06-10T18:55:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.819+0000 7fcff4058700 1 -- 172.21.15.104:0/504263522 wait complete. 2023-06-10T18:55:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.820+0000 7fcff4058700 1 Processor -- start 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 -- start start 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfec0cec90 0x7fcfec15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfec0d0a60 0x7fcfec15cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec15e630 con 0x7fcfec0d23f0 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec15d1c0 con 0x7fcfec0d0a60 2023-06-10T18:55:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff4058700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec15d4a0 con 0x7fcfec0cec90 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff25f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff25f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36160/0 (socket says 172.21.15.104:36160) 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.821+0000 7fcff25f5700 1 -- 172.21.15.104:0/1027313866 learned_addr learned my addr 172.21.15.104:0/1027313866 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 -- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfec0cec90 msgr2=0x7fcfec15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfec0cec90 0x7fcfec15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 -- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfec0d0a60 msgr2=0x7fcfec15cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff15f3700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfec0d0a60 0x7fcfec15cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfec0d0a60 0x7fcfec15cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 -- 172.21.15.104:0/1027313866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4009b30 con 0x7fcfec0d23f0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff25f5700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400b370 tx=0x7fcfe4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4004440 con 0x7fcfec0d23f0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe40045a0 con 0x7fcfec0d23f0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec15d780 con 0x7fcfec0d23f0 2023-06-10T18:55:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4004950 con 0x7fcfec0d23f0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.822+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec15dd40 con 0x7fcfec0d23f0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.823+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec0cde40 con 0x7fcfec0d23f0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.823+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcfe4012430 con 0x7fcfec0d23f0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.824+0000 7fcfe27fc700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 0x7fcfd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.824+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fcfe408a6a0 con 0x7fcfec0d23f0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.824+0000 7fcff1df4700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 0x7fcfd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:04.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.825+0000 7fcff1df4700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 0x7fcfd006d190 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcfec06b8b0 tx=0x7fcfdc006c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:04.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:04.828+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcfe40559f0 con 0x7fcfec0d23f0 2023-06-10T18:55:05.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.014+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcfec074050 con 0x7fcfec0d23f0 2023-06-10T18:55:05.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.818+0000 7fcfe27fc700 1 -- 172.21.15.104:0/1027313866 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcfe4059010 con 0x7fcfec0d23f0 2023-06-10T18:55:05.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 msgr2=0x7fcfd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 0x7fcfd006d190 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcfec06b8b0 tx=0x7fcfdc006c50 comp rx=0 tx=0).stop 2023-06-10T18:55:05.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 msgr2=0x7fcfec1f1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400b370 tx=0x7fcfe4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 shutdown_connections 2023-06-10T18:55:05.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfec0d23f0 0x7fcfec1f1cf0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcfec0cec90 0x7fcfec15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcfd006ace0 0x7fcfd006d190 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 --2- 172.21.15.104:0/1027313866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfec0d0a60 0x7fcfec15cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 >> 172.21.15.104:0/1027313866 conn(0x7fcfec071ec0 msgr2=0x7fcfec0b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 shutdown_connections 2023-06-10T18:55:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.825+0000 7fcff4058700 1 -- 172.21.15.104:0/1027313866 wait complete. 2023-06-10T18:55:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:55:05.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:55:05.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.974+0000 7f365852f700 1 Processor -- start 2023-06-10T18:55:05.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.974+0000 7f365852f700 1 -- start start 2023-06-10T18:55:05.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.975+0000 7f365852f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.975+0000 7f365852f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f365006cde0 0x7f36500ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.975+0000 7f365852f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365006e250 con 0x7f365006e6e0 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.975+0000 7f365852f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365015c950 con 0x7f365006cde0 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.975+0000 7f365852f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500b1e80 con 0x7f365015c580 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f365006cde0 0x7f36500ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f36562cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3655aca700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f365006e6e0 0x7f36500b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:39840/0 (socket says 172.21.15.104:39840) 2023-06-10T18:55:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3655aca700 1 -- 172.21.15.104:0/3855498089 learned_addr learned my addr 172.21.15.104:0/3855498089 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 -- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 msgr2=0x7f36500b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 -- 172.21.15.104:0/3855498089 >> v1:172.21.15.114:6789/0 conn(0x7f365006e6e0 legacy=0x7f36500b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 -- 172.21.15.104:0/3855498089 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365006d920 con 0x7f365006cde0 2023-06-10T18:55:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f36562cb700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.976+0000 7f3656acc700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006cde0 0x7f36500ce0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3648009d90 tx=0x7f364800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d801475697e3508f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f36552c9700 1 -- 172.21.15.104:0/3855498089 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800e070 con 0x7f365006cde0 2023-06-10T18:55:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f36552c9700 1 -- 172.21.15.104:0/3855498089 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364800bb20 con 0x7f365006cde0 2023-06-10T18:55:05.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f36552c9700 1 -- 172.21.15.104:0/3855498089 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648012430 con 0x7f365006cde0 2023-06-10T18:55:05.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 -- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006cde0 msgr2=0x7f36500ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006cde0 0x7f36500ce0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3648009d90 tx=0x7f364800b330 comp rx=0 tx=0).stop 2023-06-10T18:55:05.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 -- 172.21.15.104:0/3855498089 shutdown_connections 2023-06-10T18:55:05.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006cde0 0x7f36500ce0d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 --2- 172.21.15.104:0/3855498089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 -- 172.21.15.104:0/3855498089 >> 172.21.15.104:0/3855498089 conn(0x7f365006b8a0 msgr2=0x7f36500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 -- 172.21.15.104:0/3855498089 shutdown_connections 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.977+0000 7f365852f700 1 -- 172.21.15.104:0/3855498089 wait complete. 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.978+0000 7f365852f700 1 Processor -- start 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.978+0000 7f365852f700 1 -- start start 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006e6e0 0x7f36501479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650147750 con 0x7f365006cde0 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500ac1c0 con 0x7f365006e6e0 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f365852f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500ac3d0 con 0x7f365015c580 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f36562cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f36562cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36170/0 (socket says 172.21.15.104:36170) 2023-06-10T18:55:05.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f36562cb700 1 -- 172.21.15.104:0/2636139176 learned_addr learned my addr 172.21.15.104:0/2636139176 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f3656acc700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.979+0000 7f3655aca700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006e6e0 0x7f36501479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 -- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 msgr2=0x7f36500c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 -- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006e6e0 msgr2=0x7f36501479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006e6e0 0x7f36501479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 -- 172.21.15.104:0/2636139176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3640009430 con 0x7f365006cde0 2023-06-10T18:55:05.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f36562cb700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f364000ebe0 tx=0x7f3640009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640005a20 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3640005b80 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3648009a70 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364000c8b0 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.980+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36500ac990 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.981+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36500acfe0 con 0x7f365006cde0 2023-06-10T18:55:05.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.981+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f364000ca10 con 0x7f365006cde0 2023-06-10T18:55:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.982+0000 7f3646ffd700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 0x7f363406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.982+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f3640089740 con 0x7f365006cde0 2023-06-10T18:55:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.982+0000 7f3655aca700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 0x7f363406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.982+0000 7f3655aca700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 0x7f363406d160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f36500b0670 tx=0x7f364c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:05.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:05.986+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3640054810 con 0x7f365006cde0 2023-06-10T18:55:06.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.173+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3650074050 con 0x7f365006cde0 2023-06-10T18:55:06.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.815+0000 7f3646ffd700 1 -- 172.21.15.104:0/2636139176 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3640057e30 con 0x7f365006cde0 2023-06-10T18:55:06.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 msgr2=0x7f363406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 0x7f363406d160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f36500b0670 tx=0x7f364c009040 comp rx=0 tx=0).stop 2023-06-10T18:55:06.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 msgr2=0x7f365014a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f364000ebe0 tx=0x7f3640009e00 comp rx=0 tx=0).stop 2023-06-10T18:55:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 shutdown_connections 2023-06-10T18:55:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f365015c580 0x7f36500c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f365006cde0 0x7f365014a280 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f363406acb0 0x7f363406d160 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 --2- 172.21.15.104:0/2636139176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365006e6e0 0x7f36501479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.819+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 >> 172.21.15.104:0/2636139176 conn(0x7f365006b8a0 msgr2=0x7f36500b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.821+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 shutdown_connections 2023-06-10T18:55:06.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.821+0000 7f365852f700 1 -- 172.21.15.104:0/2636139176 wait complete. 2023-06-10T18:55:06.839 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:55:06.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:55:06.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.969+0000 7fcd588dd700 1 Processor -- start 2023-06-10T18:55:06.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.969+0000 7fcd588dd700 1 -- start start 2023-06-10T18:55:06.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd588dd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd540cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd588dd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540cda90 con 0x7fcd540af940 2023-06-10T18:55:06.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd588dd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd54154970 con 0x7fcd541545a0 2023-06-10T18:55:06.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd588dd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5415dab0 con 0x7fcd540cd150 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd51d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcd540af940 0x7fcd5415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34006/0 (socket says 172.21.15.104:34006) 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd51d9b700 1 -- 172.21.15.104:0/1269523205 learned_addr learned my addr 172.21.15.104:0/1269523205 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051468809 0 0) 0x7fcd540cda90 con 0x7fcd540af940 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3c0035b0 con 0x7fcd540af940 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd52d9d700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd540cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722440912 0 0) 0x7fcd3c0035b0 con 0x7fcd540af940 2023-06-10T18:55:06.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 msgr2=0x7fcd540cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd540cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 >> v1:172.21.15.114:6789/0 conn(0x7fcd541545a0 legacy=0x7fcd5415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540cb0b0 con 0x7fcd540af940 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcd48001670 con 0x7fcd540af940 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd52d9d700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd540cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:06.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 >> v1:172.21.15.104:6789/0 conn(0x7fcd540af940 legacy=0x7fcd5415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3c003a30 0x7fcd3c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd3c004850 0x7fcd3c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd3c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5415dab0 con 0x7fcd3c004850 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.970+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c00baf0 con 0x7fcd3c003a30 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c00bcb0 con 0x7fcd540cd150 2023-06-10T18:55:06.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd3c004850 0x7fcd3c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd52d9d700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd3c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5259c700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3c003a30 0x7fcd3c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 -- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 msgr2=0x7fcd3c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd540cd150 0x7fcd3c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 -- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3c003a30 msgr2=0x7fcd3c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3c003a30 0x7fcd3c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 -- 172.21.15.104:0/1269523205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c0035b0 con 0x7fcd3c004850 2023-06-10T18:55:06.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5259c700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd3c003a30 0x7fcd3c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd3c004850 0x7fcd3c004cc0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c240 tx=0x7fcd3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d1b28a1efb8507 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48007c10 con 0x7fcd3c004850 2023-06-10T18:55:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4800ce00 con 0x7fcd3c004850 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.971+0000 7fcd5159a700 1 -- 172.21.15.104:0/1269523205 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48005ba0 con 0x7fcd3c004850 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 -- 172.21.15.104:0/1269523205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd3c004850 msgr2=0x7fcd3c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 --2- 172.21.15.104:0/1269523205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd3c004850 0x7fcd3c004cc0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c240 tx=0x7fcd3c005880 comp rx=0 tx=0).stop 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd52d9d700 1 -- 172.21.15.104:0/1269523205 reap_dead start 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 -- 172.21.15.104:0/1269523205 shutdown_connections 2023-06-10T18:55:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 -- 172.21.15.104:0/1269523205 >> 172.21.15.104:0/1269523205 conn(0x7fcd5406e370 msgr2=0x7fcd5406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 -- 172.21.15.104:0/1269523205 shutdown_connections 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.972+0000 7fcd588dd700 1 -- 172.21.15.104:0/1269523205 wait complete. 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.973+0000 7fcd588dd700 1 Processor -- start 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.973+0000 7fcd588dd700 1 -- start start 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd3c004850 0x7fcd54150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd54151590 0x7fcd540cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540c72e0 con 0x7fcd54151ae0 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540b0800 con 0x7fcd54151590 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54151340 con 0x7fcd3c004850 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36192/0 (socket says 172.21.15.104:36192) 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 -- 172.21.15.104:0/62539700 learned_addr learned my addr 172.21.15.104:0/62539700 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd52d9d700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd54151590 0x7fcd540cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd5259c700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd3c004850 0x7fcd54150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 -- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd3c004850 msgr2=0x7fcd54150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd3c004850 0x7fcd54150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 -- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd54151590 msgr2=0x7fcd540cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd54151590 0x7fcd540cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 -- 172.21.15.104:0/62539700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540b2f50 con 0x7fcd54151ae0 2023-06-10T18:55:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd51d9b700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00bac0 tx=0x7fcd48001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48004c80 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48004de0 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.974+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd540b0240 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.975+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48007310 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.975+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd540b1ca0 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.975+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd540c6ea0 con 0x7fcd54151ae0 2023-06-10T18:55:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.976+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcd48019070 con 0x7fcd54151ae0 2023-06-10T18:55:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.977+0000 7fcd42ffd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 0x7fcd3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.977+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fcd48089ad0 con 0x7fcd54151ae0 2023-06-10T18:55:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.977+0000 7fcd5259c700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 0x7fcd3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.977+0000 7fcd5259c700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 0x7fcd3406d190 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007fd0 tx=0x7fcd44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:06.981+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcd48054cc0 con 0x7fcd54151ae0 2023-06-10T18:55:07.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.209+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcd54074050 con 0x7fcd54151ae0 2023-06-10T18:55:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.215+0000 7fcd42ffd700 1 -- 172.21.15.104:0/62539700 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcd48007d70 con 0x7fcd54151ae0 2023-06-10T18:55:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 msgr2=0x7fcd3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 0x7fcd3406d190 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcd44007fd0 tx=0x7fcd44009040 comp rx=0 tx=0).stop 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 msgr2=0x7fcd540ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00bac0 tx=0x7fcd48001a00 comp rx=0 tx=0).stop 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 shutdown_connections 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd3406ace0 0x7fcd3406d190 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd54151590 0x7fcd540cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd3c004850 0x7fcd54150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 --2- 172.21.15.104:0/62539700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd54151ae0 0x7fcd540ce6c0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.218+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 >> 172.21.15.104:0/62539700 conn(0x7fcd5406e370 msgr2=0x7fcd540d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.219+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 shutdown_connections 2023-06-10T18:55:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.219+0000 7fcd588dd700 1 -- 172.21.15.104:0/62539700 wait complete. 2023-06-10T18:55:07.233 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 16 and data pool 17 2023-06-10T18:55:07.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:55:07.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.376+0000 7f47ddc5f700 1 Processor -- start 2023-06-10T18:55:07.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.376+0000 7f47ddc5f700 1 -- start start 2023-06-10T18:55:07.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47ddc5f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47d80d2b20 0x7f47d80d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47ddc5f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47d80ccfa0 0x7f47d80b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47ddc5f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80cd430 con 0x7f47d80b2780 2023-06-10T18:55:07.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47ddc5f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d80cd710 con 0x7f47d80ccfa0 2023-06-10T18:55:07.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47ddc5f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80cd9f0 con 0x7f47d80d2b20 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f47d80b2780 0x7f47d80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34044/0 (socket says 172.21.15.104:34044) 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d7fff700 1 -- 172.21.15.104:0/3854653382 learned_addr learned my addr 172.21.15.104:0/3854653382 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3854653382 >> v2:172.21.15.114:3300/0 conn(0x7f47d80ccfa0 0x7f47d80b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602825839 0 0) 0x7f47d80cd430 con 0x7f47d80b2780 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c00035b0 con 0x7f47d80b2780 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.377+0000 7f47d77fe700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47d80d2b20 0x7f47d80d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420962734 0 0) 0x7f47c00035b0 con 0x7f47d80b2780 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47d80d2b20 msgr2=0x7f47d80d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47d80d2b20 0x7f47d80d2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 >> v2:172.21.15.114:3300/0 conn(0x7f47d80ccfa0 msgr2=0x7f47d80b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 --2- 172.21.15.104:0/3854653382 >> v2:172.21.15.114:3300/0 conn(0x7f47d80ccfa0 0x7f47d80b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80cdcd0 con 0x7f47d80b2780 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f47d00019f0 con 0x7f47d80b2780 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 >> v1:172.21.15.104:6789/0 conn(0x7f47d80b2780 legacy=0x7f47d80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0003a00 0x7f47c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47c0004430 0x7f47c00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004f60 0x7f47c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80cd710 con 0x7f47c0004430 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80cd9f0 con 0x7f47c0003a00 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0005610 con 0x7f47c0004f60 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47c0004430 0x7f47c00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d77fe700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0003a00 0x7f47c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d7fff700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004f60 0x7f47c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004f60 msgr2=0x7f47c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:07.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004f60 0x7f47c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0003a00 msgr2=0x7f47c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:07.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0003a00 0x7f47c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3854653382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c00035b0 con 0x7f47c0004430 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d7fff700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004f60 0x7f47c000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47c0004430 0x7f47c00048f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009b60 tx=0x7f47c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4dde4c114636fc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800e6c0 con 0x7f47c0004430 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c800e820 con 0x7f47c0004430 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.378+0000 7f47d67fc700 1 -- 172.21.15.104:0/3854653382 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800eb90 con 0x7f47c0004430 2023-06-10T18:55:07.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3854653382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47c0004430 msgr2=0x7f47c00048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3854653382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47c0004430 0x7f47c00048f0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009b60 tx=0x7f47c0005880 comp rx=0 tx=0).stop 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47d7fff700 1 -- 172.21.15.104:0/3854653382 reap_dead start 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3854653382 shutdown_connections 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3854653382 >> 172.21.15.104:0/3854653382 conn(0x7f47d8071ec0 msgr2=0x7f47d809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3854653382 shutdown_connections 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.379+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3854653382 wait complete. 2023-06-10T18:55:07.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.381+0000 7f47ddc5f700 1 Processor -- start 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.381+0000 7f47ddc5f700 1 -- start start 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004430 0x7f47d81ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47d815e250 0x7f47d815e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d81f5fc0 con 0x7f47d80b7760 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815ec10 con 0x7f47d815e250 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d81f1a80 con 0x7f47c0004430 2023-06-10T18:55:07.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36226/0 (socket says 172.21.15.104:36226) 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3460545168 learned_addr learned my addr 172.21.15.104:0/3460545168 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d77fe700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004430 0x7f47d81ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d7fff700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47d815e250 0x7f47d815e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004430 msgr2=0x7f47d81ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004430 0x7f47d81ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47d815e250 msgr2=0x7f47d815e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47d815e250 0x7f47d815e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 -- 172.21.15.104:0/3460545168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80b7c40 con 0x7f47d80b7760 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47d6ffd700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005560 tx=0x7f47c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800e6c0 con 0x7f47d80b7760 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c80038b0 con 0x7f47d80b7760 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d81f1d40 con 0x7f47d80b7760 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.382+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003c40 con 0x7f47d80b7760 2023-06-10T18:55:07.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.383+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d81f2300 con 0x7f47d80b7760 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.383+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d8074050 con 0x7f47d80b7760 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.384+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f47c8013470 con 0x7f47d80b7760 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.384+0000 7f47bffff700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 0x7f47c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.384+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f47c808aeb0 con 0x7f47d80b7760 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.384+0000 7f47d77fe700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 0x7f47c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.385+0000 7f47d77fe700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 0x7f47c406d190 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b80 tx=0x7f47cc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.388+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f47c8055f70 con 0x7f47d80b7760 2023-06-10T18:55:07.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.578+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d80b83c0 con 0x7f47d80b7760 2023-06-10T18:55:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.579+0000 7f47bffff700 1 -- 172.21.15.104:0/3460545168 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+18575 (secure 0 0 0) 0x7f47c8059590 con 0x7f47d80b7760 2023-06-10T18:55:07.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:07.586 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":112,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:07.210335+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:55:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 msgr2=0x7f47c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 0x7f47c406d190 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b80 tx=0x7f47cc009210 comp rx=0 tx=0).stop 2023-06-10T18:55:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 msgr2=0x7f47d81f5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005560 tx=0x7f47c80047c0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 shutdown_connections 2023-06-10T18:55:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f47c406ace0 0x7f47c406d190 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47d815e250 0x7f47d815e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47c0004430 0x7f47d81ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 --2- 172.21.15.104:0/3460545168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47d80b7760 0x7f47d81f5a80 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 >> 172.21.15.104:0/3460545168 conn(0x7f47d8071ec0 msgr2=0x7f47d806b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 shutdown_connections 2023-06-10T18:55:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.581+0000 7f47ddc5f700 1 -- 172.21.15.104:0/3460545168 wait complete. 2023-06-10T18:55:07.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:55:07.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.715+0000 7f3575007700 1 Processor -- start 2023-06-10T18:55:07.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.715+0000 7f3575007700 1 -- start start 2023-06-10T18:55:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f3575007700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 0x7f35700701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f3575007700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700ccca0 con 0x7f35700cad90 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f3575007700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700cee50 con 0x7f35700cfa00 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f3575007700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700cf130 con 0x7f357015a250 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f356f59e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f35700cad90 0x7f35700b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34064/0 (socket says 172.21.15.104:34064) 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.716+0000 7f356f59e700 1 -- 172.21.15.104:0/556159630 learned_addr learned my addr 172.21.15.104:0/556159630 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356e59c700 1 --2- 172.21.15.104:0/556159630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 0x7f35700701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582970505 0 0) 0x7f35700ccca0 con 0x7f35700cad90 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35580035b0 con 0x7f35700cad90 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105799594 0 0) 0x7f35700cee50 con 0x7f35700cfa00 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35700ccca0 con 0x7f35700cfa00 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356e59c700 1 -- 172.21.15.104:0/556159630 >> v1:172.21.15.114:6789/0 conn(0x7f35700cfa00 legacy=0x7f35700cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356e59c700 1 -- 172.21.15.104:0/556159630 >> v1:172.21.15.104:6789/0 conn(0x7f35700cad90 legacy=0x7f35700b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3497545227 0 0) 0x7f35580035b0 con 0x7f35700cad90 2023-06-10T18:55:07.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356e59c700 1 -- 172.21.15.104:0/556159630 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35700cc930 con 0x7f357015a250 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356e59c700 1 --2- 172.21.15.104:0/556159630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 0x7f35700701e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3560009870 tx=0x7f3560009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=637e42a75e9c6792 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560011830 con 0x7f357015a250 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560011990 con 0x7f357015a250 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.717+0000 7f356dd9b700 1 -- 172.21.15.104:0/556159630 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560011d20 con 0x7f357015a250 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 -- 172.21.15.104:0/556159630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 msgr2=0x7f35700701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 --2- 172.21.15.104:0/556159630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 0x7f35700701e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3560009870 tx=0x7f3560009b80 comp rx=0 tx=0).stop 2023-06-10T18:55:07.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 -- 172.21.15.104:0/556159630 shutdown_connections 2023-06-10T18:55:07.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 --2- 172.21.15.104:0/556159630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f357015a250 0x7f35700701e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 -- 172.21.15.104:0/556159630 >> 172.21.15.104:0/556159630 conn(0x7f357006e370 msgr2=0x7f357006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:07.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 -- 172.21.15.104:0/556159630 shutdown_connections 2023-06-10T18:55:07.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.718+0000 7f3575007700 1 -- 172.21.15.104:0/556159630 wait complete. 2023-06-10T18:55:07.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.719+0000 7f3575007700 1 Processor -- start 2023-06-10T18:55:07.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.719+0000 7f3575007700 1 -- start start 2023-06-10T18:55:07.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35700cad90 0x7f3570155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 0x7f35700c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35701560e0 con 0x7f35700cfa00 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357015b6a0 con 0x7f357015a250 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f3575007700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357015c500 con 0x7f35700cad90 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36240/0 (socket says 172.21.15.104:36240) 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 -- 172.21.15.104:0/3521859990 learned_addr learned my addr 172.21.15.104:0/3521859990 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356f59e700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 0x7f35700c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356ed9d700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35700cad90 0x7f3570155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 -- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35700cad90 msgr2=0x7f3570155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35700cad90 0x7f3570155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 -- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 msgr2=0x7f35700c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 0x7f35700c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356e59c700 1 -- 172.21.15.104:0/3521859990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600094a0 con 0x7f35700cfa00 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.720+0000 7f356f59e700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 0x7f35700c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:07.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f356e59c700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f35600097c0 tx=0x7f3560004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356001b870 con 0x7f35700cfa00 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356001b9d0 con 0x7f35700cfa00 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35701f0440 con 0x7f35700cfa00 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356001bd80 con 0x7f35700cfa00 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.721+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357015bbd0 con 0x7f35700cfa00 2023-06-10T18:55:07.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.722+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570067760 con 0x7f35700cfa00 2023-06-10T18:55:07.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.722+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3560023070 con 0x7f35700cfa00 2023-06-10T18:55:07.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.722+0000 7f35577fe700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 0x7f355c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:07.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.723+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f356008a640 con 0x7f35700cfa00 2023-06-10T18:55:07.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.723+0000 7f356ed9d700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 0x7f355c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:07.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.723+0000 7f356ed9d700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 0x7f355c06d190 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3564001220 tx=0x7f3564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:07.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.727+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3560055640 con 0x7f35700cfa00 2023-06-10T18:55:07.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.915+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f357015c670 con 0x7f35700cfa00 2023-06-10T18:55:07.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.917+0000 7f35577fe700 1 -- 172.21.15.104:0/3521859990 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+18575 (secure 0 0 0) 0x7f3560058c60 con 0x7f35700cfa00 2023-06-10T18:55:07.922 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:07.922 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":112,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:07.210335+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:55:07.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 msgr2=0x7f355c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 0x7f355c06d190 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3564001220 tx=0x7f3564009040 comp rx=0 tx=0).stop 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 msgr2=0x7f35701eff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f35600097c0 tx=0x7f3560004750 comp rx=0 tx=0).stop 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 shutdown_connections 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f355c06ace0 0x7f355c06d190 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f357015a250 0x7f35700c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35700cad90 0x7f3570155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 --2- 172.21.15.104:0/3521859990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700cfa00 0x7f35701eff00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:07.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 >> 172.21.15.104:0/3521859990 conn(0x7f357006e370 msgr2=0x7f3570156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:07.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 shutdown_connections 2023-06-10T18:55:07.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:07.919+0000 7f3575007700 1 -- 172.21.15.104:0/3521859990 wait complete. 2023-06-10T18:55:07.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:55:08.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.055+0000 7fdbc4ce3700 1 Processor -- start 2023-06-10T18:55:08.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 -- start start 2023-06-10T18:55:08.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdbc00cd8f0 0x7fdbc00ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbc00adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc015cb00 con 0x7fdbc015c730 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00acd90 con 0x7fdbc00cd8f0 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.056+0000 7fdbc4ce3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00ac460 con 0x7fdbc00ad760 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbe59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdbc015c730 0x7fdbc00cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34074/0 (socket says 172.21.15.104:34074) 2023-06-10T18:55:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbe59c700 1 -- 172.21.15.104:0/2634716175 learned_addr learned my addr 172.21.15.104:0/2634716175 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2634716175 >> v2:172.21.15.114:3300/0 conn(0x7fdbc00cd8f0 0x7fdbc00ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908457098 0 0) 0x7fdba8001290 con 0x7fdbc015c730 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbc00adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac0035b0 con 0x7fdbc015c730 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734395147 0 0) 0x7fdbac0035b0 con 0x7fdbc015c730 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 msgr2=0x7fdbc00adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbc00adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 >> v2:172.21.15.114:3300/0 conn(0x7fdbc00cd8f0 msgr2=0x7fdbc00ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 --2- 172.21.15.104:0/2634716175 >> v2:172.21.15.114:3300/0 conn(0x7fdbc00cd8f0 0x7fdbc00ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00ac740 con 0x7fdbc015c730 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdba8001980 con 0x7fdbc015c730 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 >> v1:172.21.15.104:6789/0 conn(0x7fdbc015c730 legacy=0x7fdbc00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbac003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbac003f80 0x7fdbac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00acd90 con 0x7fdbac004d90 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00ac460 con 0x7fdbac003f80 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac00ba40 con 0x7fdbc00ad760 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbac003f80 0x7fdbac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.057+0000 7fdbbe59c700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbac003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 -- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 msgr2=0x7fdbac003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc00ad760 0x7fdbac003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 -- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbac003f80 msgr2=0x7fdbac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbac003f80 0x7fdbac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 -- 172.21.15.104:0/2634716175 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac0035b0 con 0x7fdbac004d90 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbac005200 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fc0 tx=0x7fdbac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e5f2d833c07a71 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:08.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800ebd0 con 0x7fdbac004d90 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb800f040 con 0x7fdbac004d90 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbd59a700 1 -- 172.21.15.104:0/2634716175 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8014480 con 0x7fdbac004d90 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2634716175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 msgr2=0x7fdbac005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2634716175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbac005200 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fc0 tx=0x7fdbac005880 comp rx=0 tx=0).stop 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbbed9d700 1 -- 172.21.15.104:0/2634716175 reap_dead start 2023-06-10T18:55:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2634716175 shutdown_connections 2023-06-10T18:55:08.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2634716175 >> 172.21.15.104:0/2634716175 conn(0x7fdbc006b8a0 msgr2=0x7fdbc00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:08.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.058+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2634716175 shutdown_connections 2023-06-10T18:55:08.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.059+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2634716175 wait complete. 2023-06-10T18:55:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.060+0000 7fdbc4ce3700 1 Processor -- start 2023-06-10T18:55:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.060+0000 7fdbc4ce3700 1 -- start start 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 0x7fdbc014e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc00c66e0 0x7fdbc00c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00c70c0 con 0x7fdbac004d90 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc014faa0 con 0x7fdbc00c66e0 2023-06-10T18:55:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc014ee90 con 0x7fdbc014e3d0 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36258/0 (socket says 172.21.15.104:36258) 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 -- 172.21.15.104:0/2225919592 learned_addr learned my addr 172.21.15.104:0/2225919592 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbed9d700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc00c66e0 0x7fdbc00c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 0x7fdbc014e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 -- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 msgr2=0x7fdbc014e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:08.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 0x7fdbc014e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 -- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc00c66e0 msgr2=0x7fdbc00c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:08.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc00c66e0 0x7fdbc00c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:08.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 -- 172.21.15.104:0/2225919592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc006f150 con 0x7fdbac004d90 2023-06-10T18:55:08.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 0x7fdbc014e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbbe59c700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fdba80072f0 tx=0x7fdba8007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba8007840 con 0x7fdbac004d90 2023-06-10T18:55:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba800f870 con 0x7fdbac004d90 2023-06-10T18:55:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc014f170 con 0x7fdbac004d90 2023-06-10T18:55:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba800fc40 con 0x7fdbac004d90 2023-06-10T18:55:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.061+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc014f730 con 0x7fdbac004d90 2023-06-10T18:55:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.062+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc0074050 con 0x7fdbac004d90 2023-06-10T18:55:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.063+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdba80133f0 con 0x7fdbac004d90 2023-06-10T18:55:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.063+0000 7fdbb6ffd700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 0x7fdba006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.063+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdba8089490 con 0x7fdbac004d90 2023-06-10T18:55:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.063+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 0x7fdba006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.064+0000 7fdbbdd9b700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 0x7fdba006d190 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdbc01f0fb0 tx=0x7fdbb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.067+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdba8054400 con 0x7fdbac004d90 2023-06-10T18:55:08.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:08.301+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdbc015cb00 con 0x7fdbac004d90 2023-06-10T18:55:09.166 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:55:09.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:55:09.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.224+0000 7fdbb6ffd700 1 -- 172.21.15.104:0/2225919592 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdbc015cb00 con 0x7fdbac004d90 2023-06-10T18:55:09.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 msgr2=0x7fdba006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 0x7fdba006d190 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdbc01f0fb0 tx=0x7fdbb0008040 comp rx=0 tx=0).stop 2023-06-10T18:55:09.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 msgr2=0x7fdbc014dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fdba80072f0 tx=0x7fdba8007600 comp rx=0 tx=0).stop 2023-06-10T18:55:09.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 shutdown_connections 2023-06-10T18:55:09.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdba006ace0 0x7fdba006d190 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc00c66e0 0x7fdbc00c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbac004d90 0x7fdbc014dc20 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 --2- 172.21.15.104:0/2225919592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbc014e3d0 0x7fdbc014e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 >> 172.21.15.104:0/2225919592 conn(0x7fdbc006b8a0 msgr2=0x7fdbc006e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.226+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 shutdown_connections 2023-06-10T18:55:09.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.227+0000 7fdbc4ce3700 1 -- 172.21.15.104:0/2225919592 wait complete. 2023-06-10T18:55:09.246 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:55:09.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:55:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.414+0000 7fba9126a700 1 Processor -- start 2023-06-10T18:55:09.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.414+0000 7fba9126a700 1 -- start start 2023-06-10T18:55:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba9126a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba8c0d04f0 0x7fba8c0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba9126a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba8c0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba9126a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c0d0e00 con 0x7fba8c06b510 2023-06-10T18:55:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba9126a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8c0af530 con 0x7fba8c0d04f0 2023-06-10T18:55:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba9126a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0af810 con 0x7fba8c0d25c0 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba8b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba8c06b510 0x7fba8c06b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34098/0 (socket says 172.21.15.104:34098) 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba8b7fe700 1 -- 172.21.15.104:0/430203618 learned_addr learned my addr 172.21.15.104:0/430203618 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba8affd700 1 --2- 172.21.15.104:0/430203618 >> v2:172.21.15.114:3300/0 conn(0x7fba8c0d04f0 0x7fba8c0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba8c0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958445387 0 0) 0x7fba8c0d0e00 con 0x7fba8c06b510 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba740035b0 con 0x7fba8c06b510 2023-06-10T18:55:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306781423 0 0) 0x7fba740035b0 con 0x7fba8c06b510 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 msgr2=0x7fba8c0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.415+0000 7fba89ffb700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba8c0d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 >> v2:172.21.15.114:3300/0 conn(0x7fba8c0d04f0 msgr2=0x7fba8c0d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 --2- 172.21.15.104:0/430203618 >> v2:172.21.15.114:3300/0 conn(0x7fba8c0d04f0 0x7fba8c0d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0afaf0 con 0x7fba8c06b510 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fba840019f0 con 0x7fba8c06b510 2023-06-10T18:55:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 >> v1:172.21.15.104:6789/0 conn(0x7fba8c06b510 legacy=0x7fba8c06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba74003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba74003f80 0x7fba74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba74004d90 0x7fba74005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0af530 con 0x7fba74003f80 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0af810 con 0x7fba74004d90 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7400ba40 con 0x7fba8c0d25c0 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba74003f80 0x7fba74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8affd700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba74003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8b7fe700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba74004d90 0x7fba74005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 -- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 msgr2=0x7fba74003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba8c0d25c0 0x7fba74003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 -- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba74004d90 msgr2=0x7fba74005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba74004d90 0x7fba74005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 -- 172.21.15.104:0/430203618 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba740035b0 con 0x7fba74003f80 2023-06-10T18:55:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba74003f80 0x7fba74004850 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009b60 tx=0x7fba74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf117a6fccf20dd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c00e660 con 0x7fba74003f80 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7c00e7c0 con 0x7fba74003f80 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.416+0000 7fba89ffb700 1 -- 172.21.15.104:0/430203618 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c00eb30 con 0x7fba74003f80 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 -- 172.21.15.104:0/430203618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba74003f80 msgr2=0x7fba74004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 --2- 172.21.15.104:0/430203618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba74003f80 0x7fba74004850 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009b60 tx=0x7fba74005880 comp rx=0 tx=0).stop 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba8b7fe700 1 -- 172.21.15.104:0/430203618 reap_dead start 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 -- 172.21.15.104:0/430203618 shutdown_connections 2023-06-10T18:55:09.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 -- 172.21.15.104:0/430203618 >> 172.21.15.104:0/430203618 conn(0x7fba8c071ec0 msgr2=0x7fba8c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:09.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 -- 172.21.15.104:0/430203618 shutdown_connections 2023-06-10T18:55:09.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.417+0000 7fba9126a700 1 -- 172.21.15.104:0/430203618 wait complete. 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.418+0000 7fba9126a700 1 Processor -- start 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.418+0000 7fba9126a700 1 -- start start 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 0x7fba8c0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba8c1f1770 0x7fba8c0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c154680 con 0x7fba8c1e5f10 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c1e6740 con 0x7fba8c1f1770 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba9126a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c1e39f0 con 0x7fba74003f80 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36282/0 (socket says 172.21.15.104:36282) 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 -- 172.21.15.104:0/1835512396 learned_addr learned my addr 172.21.15.104:0/1835512396 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8a7fc700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba8c1f1770 0x7fba8c0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8affd700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 0x7fba8c0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 -- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 msgr2=0x7fba8c0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 0x7fba8c0b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 -- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba8c1f1770 msgr2=0x7fba8c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba8c1f1770 0x7fba8c0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 -- 172.21.15.104:0/1835512396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c1e3cd0 con 0x7fba8c1e5f10 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8affd700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 0x7fba8c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.419+0000 7fba8b7fe700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fba84001330 tx=0x7fba84007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.420+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84001f00 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.420+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8400ed90 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.420+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8c1e3fb0 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.420+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84013640 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.420+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8c1e6380 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.421+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8c1e6510 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.421+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba840137a0 con 0x7fba8c1e5f10 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.421+0000 7fba737fe700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.422+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fba84089cf0 con 0x7fba8c1e5f10 2023-06-10T18:55:09.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.422+0000 7fba8affd700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:09.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.422+0000 7fba8affd700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fba80009bb0 tx=0x7fba80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:09.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.426+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba8408e070 con 0x7fba8c1e5f10 2023-06-10T18:55:09.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:09.656+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fba8c067010 con 0x7fba8c1e5f10 2023-06-10T18:55:10.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.225+0000 7fba737fe700 1 -- 172.21.15.104:0/1835512396 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fba84054c60 con 0x7fba8c1e5f10 2023-06-10T18:55:10.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 msgr2=0x7fba7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fba80009bb0 tx=0x7fba80008040 comp rx=0 tx=0).stop 2023-06-10T18:55:10.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 msgr2=0x7fba8c0b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fba84001330 tx=0x7fba84007c90 comp rx=0 tx=0).stop 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 shutdown_connections 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba8c1e5f10 0x7fba8c0b2380 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba74003f80 0x7fba8c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba7806ace0 0x7fba7806d190 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 --2- 172.21.15.104:0/1835512396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba8c1f1770 0x7fba8c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 >> 172.21.15.104:0/1835512396 conn(0x7fba8c071ec0 msgr2=0x7fba8c06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:10.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 shutdown_connections 2023-06-10T18:55:10.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.227+0000 7fba9126a700 1 -- 172.21.15.104:0/1835512396 wait complete. 2023-06-10T18:55:10.245 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:55:10.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:10.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.365+0000 7f7ab24b2700 1 Processor -- start 2023-06-10T18:55:10.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.365+0000 7f7ab24b2700 1 -- start start 2023-06-10T18:55:10.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.365+0000 7f7ab24b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7aac0cd850 0x7f7aac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7ab24b2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0c9e60 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7ab24b2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0aee70 con 0x7f7aac0cb300 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7ab24b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0c9bc0 con 0x7f7aac0cd850 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aabfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7aac0aeaa0 0x7f7aac0cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34114/0 (socket says 172.21.15.104:34114) 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aabfff700 1 -- 172.21.15.104:0/1351305555 learned_addr learned my addr 172.21.15.104:0/1351305555 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724725129 0 0) 0x7f7aac0c9e60 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/1351305555 >> v2:172.21.15.114:3300/0 conn(0x7f7aac0cd850 0x7f7aac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a940057c0 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938296174 0 0) 0x7f7a940057c0 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 >> v2:172.21.15.114:3300/0 conn(0x7f7aac0cd850 msgr2=0x7f7aac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 --2- 172.21.15.104:0/1351305555 >> v2:172.21.15.114:3300/0 conn(0x7f7aac0cd850 0x7f7aac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 >> v1:172.21.15.114:6789/0 conn(0x7f7aac0cb300 legacy=0x7f7aac06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac06e540 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7a9c0017f0 con 0x7f7aac0aeaa0 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 >> v1:172.21.15.104:6789/0 conn(0x7f7aac0aeaa0 legacy=0x7f7aac0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94005c10 0x7f7a94006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 0x7f7a94006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a94007410 0x7f7a9400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0c9bc0 con 0x7f7a94006a60 2023-06-10T18:55:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9400e270 con 0x7f7a94007410 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9400e430 con 0x7f7a94005c10 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.366+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 0x7f7a94006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7ab0a4f700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a94007410 0x7f7a9400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aabfff700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94005c10 0x7f7a94006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 -- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94005c10 msgr2=0x7f7a94006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94005c10 0x7f7a94006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 -- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a94007410 msgr2=0x7f7a9400dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a94007410 0x7f7a9400dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 -- 172.21.15.104:0/1351305555 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a940057c0 con 0x7f7a94006a60 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7ab0a4f700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a94007410 0x7f7a9400dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7ab0a4f700 1 -- 172.21.15.104:0/1351305555 reap_dead start 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 0x7f7a94006ed0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009b60 tx=0x7f7a94007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da07c10cfe4978 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000e6c0 con 0x7f7a94006a60 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa000e820 con 0x7f7a94006a60 2023-06-10T18:55:10.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7aaaffd700 1 -- 172.21.15.104:0/1351305555 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000eb90 con 0x7f7a94006a60 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7ab24b2700 1 -- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 msgr2=0x7f7a94006ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.367+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 0x7f7a94006ed0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009b60 tx=0x7f7a94007a90 comp rx=0 tx=0).stop 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.368+0000 7f7ab24b2700 1 -- 172.21.15.104:0/1351305555 shutdown_connections 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.368+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/1351305555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a94006a60 0x7f7a94006ed0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.368+0000 7f7ab24b2700 1 -- 172.21.15.104:0/1351305555 >> 172.21.15.104:0/1351305555 conn(0x7f7aac06b8a0 msgr2=0x7f7aac0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:10.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.368+0000 7f7ab24b2700 1 -- 172.21.15.104:0/1351305555 shutdown_connections 2023-06-10T18:55:10.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.368+0000 7f7ab24b2700 1 -- 172.21.15.104:0/1351305555 wait complete. 2023-06-10T18:55:10.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.369+0000 7f7ab24b2700 1 Processor -- start 2023-06-10T18:55:10.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.369+0000 7f7ab24b2700 1 -- start start 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94006a60 0x7f7aac154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aac1565b0 0x7f7aac14da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac156120 con 0x7f7aac1f5200 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac155510 con 0x7f7aac1565b0 2023-06-10T18:55:10.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac1557f0 con 0x7f7a94006a60 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36302/0 (socket says 172.21.15.104:36302) 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 -- 172.21.15.104:0/2105556050 learned_addr learned my addr 172.21.15.104:0/2105556050 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7aab7fe700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aac1565b0 0x7f7aac14da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7aabfff700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94006a60 0x7f7aac154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 -- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94006a60 msgr2=0x7f7aac154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94006a60 0x7f7aac154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 -- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aac1565b0 msgr2=0x7f7aac14da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aac1565b0 0x7f7aac14da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 -- 172.21.15.104:0/2105556050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac155ad0 con 0x7f7aac1f5200 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab0a4f700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f7aac071030 tx=0x7f7aa400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa400eec0 con 0x7f7aac1f5200 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa4003c10 con 0x7f7aac1f5200 2023-06-10T18:55:10.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac155db0 con 0x7f7aac1f5200 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.370+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40050d0 con 0x7f7aac1f5200 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.371+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac0c98e0 con 0x7f7aac1f5200 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.371+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac074050 con 0x7f7aac1f5200 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.372+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7aa4005230 con 0x7f7aac1f5200 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.372+0000 7f7aa8ff9700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 0x7f7a9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.372+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7aa40894d0 con 0x7f7aac1f5200 2023-06-10T18:55:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.372+0000 7f7aabfff700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 0x7f7a9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.373+0000 7f7aabfff700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 0x7f7a9806d190 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c001e10 tx=0x7f7a9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.377+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7aa4057ed0 con 0x7f7aac1f5200 2023-06-10T18:55:10.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.605+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aac06a8b0 con 0x7f7aac1f5200 2023-06-10T18:55:10.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.606+0000 7f7aa8ff9700 1 -- 172.21.15.104:0/2105556050 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4682 (secure 0 0 0) 0x7f7aa4057a60 con 0x7f7aac1f5200 2023-06-10T18:55:10.611 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:10.612 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:10.222100+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:55:10.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 msgr2=0x7f7a9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 0x7f7a9806d190 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c001e10 tx=0x7f7a9c009040 comp rx=0 tx=0).stop 2023-06-10T18:55:10.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 msgr2=0x7f7aac0b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f7aac071030 tx=0x7f7aa400ed30 comp rx=0 tx=0).stop 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 shutdown_connections 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7aac1f5200 0x7f7aac0b2430 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a94006a60 0x7f7aac154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a9806ace0 0x7f7a9806d190 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 --2- 172.21.15.104:0/2105556050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7aac1565b0 0x7f7aac14da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 >> 172.21.15.104:0/2105556050 conn(0x7f7aac06b8a0 msgr2=0x7f7aac0cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 shutdown_connections 2023-06-10T18:55:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.608+0000 7f7ab24b2700 1 -- 172.21.15.104:0/2105556050 wait complete. 2023-06-10T18:55:10.616 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 70 2023-06-10T18:55:10.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:55:10.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 Processor -- start 2023-06-10T18:55:10.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 -- start start 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c06e0e0 0x7f371c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f371c06e4b0 0x7f371c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371c06d290 con 0x7f371c0ca100 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f371c071640 con 0x7f371c06e4b0 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.740+0000 7f3721fef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0aef00 con 0x7f371c06e0e0 2023-06-10T18:55:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f371c06e4b0 0x7f371c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371b7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c06e0e0 0x7f371c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f371c06e4b0 0x7f371c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36308/0 (socket says 172.21.15.104:36308) 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 -- 172.21.15.104:0/3119998726 learned_addr learned my addr 172.21.15.104:0/3119998726 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 -- 172.21.15.104:0/3119998726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c06e0e0 msgr2=0x7f371c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 --2- 172.21.15.104:0/3119998726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c06e0e0 0x7f371c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 -- 172.21.15.104:0/3119998726 >> v1:172.21.15.114:6789/0 conn(0x7f371c0ca100 legacy=0x7f371c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 -- 172.21.15.104:0/3119998726 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c06d4c0 con 0x7f371c06e4b0 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371affd700 1 --2- 172.21.15.104:0/3119998726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9bc0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f37100098f0 tx=0x7f3710009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a67f22c52fa29b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371a7fc700 1 -- 172.21.15.104:0/3119998726 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710011830 con 0x7f371c06e4b0 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371a7fc700 1 -- 172.21.15.104:0/3119998726 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3710011990 con 0x7f371c06e4b0 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.741+0000 7f371a7fc700 1 -- 172.21.15.104:0/3119998726 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710011d20 con 0x7f371c06e4b0 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 -- 172.21.15.104:0/3119998726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e4b0 msgr2=0x7f371c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 --2- 172.21.15.104:0/3119998726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9bc0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f37100098f0 tx=0x7f3710009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 -- 172.21.15.104:0/3119998726 shutdown_connections 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 --2- 172.21.15.104:0/3119998726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c06e0e0 0x7f371c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 --2- 172.21.15.104:0/3119998726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9bc0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 -- 172.21.15.104:0/3119998726 >> 172.21.15.104:0/3119998726 conn(0x7f371c06b8a0 msgr2=0x7f371c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 -- 172.21.15.104:0/3119998726 shutdown_connections 2023-06-10T18:55:10.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.742+0000 7f3721fef700 1 -- 172.21.15.104:0/3119998726 wait complete. 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.743+0000 7f3721fef700 1 Processor -- start 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.743+0000 7f3721fef700 1 -- start start 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c0ca100 0x7f371c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0c9560 con 0x7f371c06e0e0 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0c7970 con 0x7f371c06e4b0 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f3721fef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0af580 con 0x7f371c0ca100 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36312/0 (socket says 172.21.15.104:36312) 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 -- 172.21.15.104:0/2305734425 learned_addr learned my addr 172.21.15.104:0/2305734425 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371affd700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 -- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c0ca100 msgr2=0x7f371c0af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c0ca100 0x7f371c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 -- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 msgr2=0x7f371c0c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 -- 172.21.15.104:0/2305734425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710009550 con 0x7f371c06e0e0 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371affd700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.744+0000 7f371b7fe700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e900 tx=0x7f370c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.745+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c009650 con 0x7f371c06e0e0 2023-06-10T18:55:10.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.745+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c00c850 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.745+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c0af790 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.745+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c00cbc0 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.745+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c0afd50 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.746+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371c074050 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.746+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f370c00cd20 con 0x7f371c06e0e0 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.746+0000 7f3703fff700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 0x7f370806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:10.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.747+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f370c0899b0 con 0x7f371c06e0e0 2023-06-10T18:55:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.747+0000 7f371affd700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 0x7f370806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.747+0000 7f371affd700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 0x7f370806d1c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f371c0cbbe0 tx=0x7f3710008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:10.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.751+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f370c054920 con 0x7f371c06e0e0 2023-06-10T18:55:10.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.938+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f371c0c6340 con 0x7f371c06e0e0 2023-06-10T18:55:10.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.939+0000 7f3703fff700 1 -- 172.21.15.104:0/2305734425 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+18562 (secure 0 0 0) 0x7f370c017090 con 0x7f371c06e0e0 2023-06-10T18:55:10.944 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:10.945 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":113,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:08.212532+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:55:10.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 msgr2=0x7f370806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 0x7f370806d1c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f371c0cbbe0 tx=0x7f3710008030 comp rx=0 tx=0).stop 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 msgr2=0x7f371c1558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e900 tx=0x7f370c00ecc0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 shutdown_connections 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f371c0ca100 0x7f371c0af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f371c06e0e0 0x7f371c1558f0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f370806ad10 0x7f370806d1c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 --2- 172.21.15.104:0/2305734425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f371c06e4b0 0x7f371c0c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 >> 172.21.15.104:0/2305734425 conn(0x7f371c06b8a0 msgr2=0x7f371c0ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 shutdown_connections 2023-06-10T18:55:10.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:10.941+0000 7f3721fef700 1 -- 172.21.15.104:0/2305734425 wait complete. 2023-06-10T18:55:10.958 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:55:10.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:55:11.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.073+0000 7ff63f2d6700 1 Processor -- start 2023-06-10T18:55:11.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.073+0000 7ff63f2d6700 1 -- start start 2023-06-10T18:55:11.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63f2d6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff63815a420 0x7ff6380b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63f2d6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6380b6070 con 0x7ff6380cea60 2023-06-10T18:55:11.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63f2d6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6380b5460 con 0x7ff6380b4ff0 2023-06-10T18:55:11.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63f2d6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6380b5740 con 0x7ff63815a420 2023-06-10T18:55:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63d873700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff6380cea60 0x7ff63815a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34152/0 (socket says 172.21.15.104:34152) 2023-06-10T18:55:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63d873700 1 -- 172.21.15.104:0/1072789693 learned_addr learned my addr 172.21.15.104:0/1072789693 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff63d072700 1 --2- 172.21.15.104:0/1072789693 >> v2:172.21.15.114:3300/0 conn(0x7ff63815a420 0x7ff6380b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494898440 0 0) 0x7ff6380b6070 con 0x7ff6380cea60 2023-06-10T18:55:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.074+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6240035b0 con 0x7ff6380cea60 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340482011 0 0) 0x7ff6240035b0 con 0x7ff6380cea60 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 >> v2:172.21.15.114:3300/0 conn(0x7ff63815a420 msgr2=0x7ff6380b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 --2- 172.21.15.104:0/1072789693 >> v2:172.21.15.114:3300/0 conn(0x7ff63815a420 0x7ff6380b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 >> v1:172.21.15.114:6789/0 conn(0x7ff6380b4ff0 legacy=0x7ff6380ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380b5a20 con 0x7ff6380cea60 2023-06-10T18:55:11.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff6300019f0 con 0x7ff6380cea60 2023-06-10T18:55:11.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 >> v1:172.21.15.104:6789/0 conn(0x7ff6380cea60 legacy=0x7ff63815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff624003a00 0x7ff624003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff624004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff624005200 0x7ff62400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380b5740 con 0x7ff624004850 2023-06-10T18:55:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62400c080 con 0x7ff624005200 2023-06-10T18:55:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62400c240 con 0x7ff624003a00 2023-06-10T18:55:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d072700 1 --2- 172.21.15.104:0/1072789693 >> v2:172.21.15.114:3300/0 conn(0x7ff63815a420 0x7ff6380b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff624004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d873700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff624005200 0x7ff62400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d072700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff624003a00 0x7ff624003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 -- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff624003a00 msgr2=0x7ff624003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff624003a00 0x7ff624003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 -- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff624005200 msgr2=0x7ff62400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff624005200 0x7ff62400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 -- 172.21.15.104:0/1072789693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240035b0 con 0x7ff624004850 2023-06-10T18:55:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d072700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff624003a00 0x7ff624003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d873700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff624005200 0x7ff62400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63d873700 1 -- 172.21.15.104:0/1072789693 reap_dead start 2023-06-10T18:55:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff63c871700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff624004cc0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff6340017c0 tx=0x7ff624005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e38c571c973d033 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340052d0 con 0x7ff624004850 2023-06-10T18:55:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff634005430 con 0x7ff624004850 2023-06-10T18:55:11.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.075+0000 7ff62ffff700 1 -- 172.21.15.104:0/1072789693 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340038c0 con 0x7ff624004850 2023-06-10T18:55:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 msgr2=0x7ff624004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff624004cc0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff6340017c0 tx=0x7ff624005880 comp rx=0 tx=0).stop 2023-06-10T18:55:11.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1072789693 shutdown_connections 2023-06-10T18:55:11.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1072789693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff624004cc0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1072789693 >> 172.21.15.104:0/1072789693 conn(0x7ff638089fd0 msgr2=0x7ff6380d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:11.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1072789693 shutdown_connections 2023-06-10T18:55:11.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.076+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1072789693 wait complete. 2023-06-10T18:55:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.077+0000 7ff63f2d6700 1 Processor -- start 2023-06-10T18:55:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.077+0000 7ff63f2d6700 1 -- start start 2023-06-10T18:55:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6381485e0 0x7ff6380cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 0x7ff6381f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff638156500 con 0x7ff624004850 2023-06-10T18:55:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63806b680 con 0x7ff6380cadb0 2023-06-10T18:55:11.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63f2d6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380cb370 con 0x7ff6381485e0 2023-06-10T18:55:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36332/0 (socket says 172.21.15.104:36332) 2023-06-10T18:55:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 -- 172.21.15.104:0/1774525616 learned_addr learned my addr 172.21.15.104:0/1774525616 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:11.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d873700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 0x7ff6381f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 -- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6381485e0 msgr2=0x7ff6380cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63c871700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6381485e0 0x7ff6380cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6381485e0 0x7ff6380cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 -- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 msgr2=0x7ff6381f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 0x7ff6381f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 -- 172.21.15.104:0/1774525616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63806b940 con 0x7ff624004850 2023-06-10T18:55:11.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d873700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 0x7ff6381f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff63d072700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff6280108d0 tx=0x7ff628010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.078+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62800ea70 con 0x7ff624004850 2023-06-10T18:55:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.079+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff628010ed0 con 0x7ff624004850 2023-06-10T18:55:11.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.079+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63806bc20 con 0x7ff624004850 2023-06-10T18:55:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.079+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62800edd0 con 0x7ff624004850 2023-06-10T18:55:11.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.079+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6380cc280 con 0x7ff624004850 2023-06-10T18:55:11.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.080+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6380b3140 con 0x7ff624004850 2023-06-10T18:55:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.080+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff6280143f0 con 0x7ff624004850 2023-06-10T18:55:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.080+0000 7ff62dffb700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 0x7ff61c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.081+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff628089b30 con 0x7ff624004850 2023-06-10T18:55:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.081+0000 7ff63c871700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 0x7ff61c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.081+0000 7ff63c871700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 0x7ff61c06d190 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff6380caa90 tx=0x7ff634009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.085+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff628054aa0 con 0x7ff624004850 2023-06-10T18:55:11.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.269+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff63806b500 con 0x7ff61c06ace0 2023-06-10T18:55:11.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.277+0000 7ff62dffb700 1 -- 172.21.15.104:0/1774525616 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300017 (secure 0 0 0) 0x7ff634005150 con 0x7ff61c06ace0 2023-06-10T18:55:11.286 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.280+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 msgr2=0x7ff61c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.280+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 0x7ff61c06d190 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff6380caa90 tx=0x7ff634009030 comp rx=0 tx=0).stop 2023-06-10T18:55:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 msgr2=0x7ff6380cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff6280108d0 tx=0x7ff628010c90 comp rx=0 tx=0).stop 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 shutdown_connections 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff61c06ace0 0x7ff61c06d190 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6380cadb0 0x7ff6381f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff624004850 0x7ff6380cdac0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 --2- 172.21.15.104:0/1774525616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6381485e0 0x7ff6380cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 >> 172.21.15.104:0/1774525616 conn(0x7ff638089fd0 msgr2=0x7ff6380b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 shutdown_connections 2023-06-10T18:55:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.281+0000 7ff63f2d6700 1 -- 172.21.15.104:0/1774525616 wait complete. 2023-06-10T18:55:11.290 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:55:11.298 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2023-06-10T18:55:09.527753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227368,"kb_used_data":11752,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754747352,"statfs":{"total":773094113280,"available":772861288448,"internally_reserved":0,"allocated":12034048,"data_stored":6409828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000832"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813840+0000","last_change":"2023-06-10T18:55:05.813840+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813840+0000","last_undegraded":"2023-06-10T18:55:05.813840+0000","last_fullsized":"2023-06-10T18:55:05.813840+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819109+0000","last_change":"2023-06-10T18:55:05.819109+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819109+0000","last_undegraded":"2023-06-10T18:55:05.819109+0000","last_fullsized":"2023-06-10T18:55:05.819109+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819982+0000","last_change":"2023-06-10T18:55:05.819982+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819982+0000","last_undegraded":"2023-06-10T18:55:05.819982+0000","last_fullsized":"2023-06-10T18:55:05.819982+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819812+0000","last_change":"2023-06-10T18:55:05.819812+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819812+0000","last_undegraded":"2023-06-10T18:55:05.819812+0000","last_fullsized":"2023-06-10T18:55:05.819812+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.814528+0000","last_change":"2023-06-10T18:55:05.814528+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.814528+0000","last_undegraded":"2023-06-10T18:55:05.814528+0000","last_fullsized":"2023-06-10T18:55:05.814528+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819347+0000","last_change":"2023-06-10T18:55:05.819347+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819347+0000","last_undegraded":"2023-06-10T18:55:05.819347+0000","last_fullsized":"2023-06-10T18:55:05.819347+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813962+0000","last_change":"2023-06-10T18:55:05.813962+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813962+0000","last_undegraded":"2023-06-10T18:55:05.813962+0000","last_fullsized":"2023-06-10T18:55:05.813962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820839+0000","last_change":"2023-06-10T18:55:05.820839+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820839+0000","last_undegraded":"2023-06-10T18:55:05.820839+0000","last_fullsized":"2023-06-10T18:55:05.820839+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819935+0000","last_change":"2023-06-10T18:55:05.819935+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819935+0000","last_undegraded":"2023-06-10T18:55:05.819935+0000","last_fullsized":"2023-06-10T18:55:05.819935+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820212+0000","last_change":"2023-06-10T18:55:05.820212+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820212+0000","last_undegraded":"2023-06-10T18:55:05.820212+0000","last_fullsized":"2023-06-10T18:55:05.820212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.823603+0000","last_change":"2023-06-10T18:55:05.823603+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.823603+0000","last_undegraded":"2023-06-10T18:55:05.823603+0000","last_fullsized":"2023-06-10T18:55:05.823603+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813553+0000","last_change":"2023-06-10T18:55:05.813553+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813553+0000","last_undegraded":"2023-06-10T18:55:05.813553+0000","last_fullsized":"2023-06-10T18:55:05.813553+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.816400+0000","last_change":"2023-06-10T18:55:05.816400+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.816400+0000","last_undegraded":"2023-06-10T18:55:05.816400+0000","last_fullsized":"2023-06-10T18:55:05.816400+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820342+0000","last_change":"2023-06-10T18:55:05.820342+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820342+0000","last_undegraded":"2023-06-10T18:55:05.820342+0000","last_fullsized":"2023-06-10T18:55:05.820342+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.816398+0000","last_change":"2023-06-10T18:55:05.816398+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.816398+0000","last_undegraded":"2023-06-10T18:55:05.816398+0000","last_fullsized":"2023-06-10T18:55:05.816398+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820375+0000","last_change":"2023-06-10T18:55:05.820375+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820375+0000","last_undegraded":"2023-06-10T18:55:05.820375+0000","last_fullsized":"2023-06-10T18:55:05.820375+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819160+0000","last_change":"2023-06-10T18:55:05.819160+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819160+0000","last_undegraded":"2023-06-10T18:55:05.819160+0000","last_fullsized":"2023-06-10T18:55:05.819160+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.823661+0000","last_change":"2023-06-10T18:55:05.823661+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.823661+0000","last_undegraded":"2023-06-10T18:55:05.823661+0000","last_fullsized":"2023-06-10T18:55:05.823661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819272+0000","last_change":"2023-06-10T18:55:05.819272+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819272+0000","last_undegraded":"2023-06-10T18:55:05.819272+0000","last_fullsized":"2023-06-10T18:55:05.819272+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819089+0000","last_change":"2023-06-10T18:55:05.819089+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819089+0000","last_undegraded":"2023-06-10T18:55:05.819089+0000","last_fullsized":"2023-06-10T18:55:05.819089+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813898+0000","last_change":"2023-06-10T18:55:05.813898+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813898+0000","last_undegraded":"2023-06-10T18:55:05.813898+0000","last_fullsized":"2023-06-10T18:55:05.813898+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813467+0000","last_change":"2023-06-10T18:55:05.813467+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813467+0000","last_undegraded":"2023-06-10T18:55:05.813467+0000","last_fullsized":"2023-06-10T18:55:05.813467+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820479+0000","last_change":"2023-06-10T18:55:05.820479+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820479+0000","last_undegraded":"2023-06-10T18:55:05.820479+0000","last_fullsized":"2023-06-10T18:55:05.820479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819289+0000","last_change":"2023-06-10T18:55:05.819289+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819289+0000","last_undegraded":"2023-06-10T18:55:05.819289+0000","last_fullsized":"2023-06-10T18:55:05.819289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820176+0000","last_change":"2023-06-10T18:55:05.820176+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820176+0000","last_undegraded":"2023-06-10T18:55:05.820176+0000","last_fullsized":"2023-06-10T18:55:05.820176+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.817339+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:55:05.817339+0000","last_peered":"2023-06-10T18:55:05.817339+0000","last_clean":"2023-06-10T18:55:05.817339+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:55:05.817339+0000","last_undegraded":"2023-06-10T18:55:05.817339+0000","last_fullsized":"2023-06-10T18:55:05.817339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:18.756189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819741+0000","last_change":"2023-06-10T18:55:05.819741+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819741+0000","last_undegraded":"2023-06-10T18:55:05.819741+0000","last_fullsized":"2023-06-10T18:55:05.819741+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":194,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.813340+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:55:05.813340+0000","last_peered":"2023-06-10T18:55:05.813340+0000","last_clean":"2023-06-10T18:55:05.813340+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:55:05.813340+0000","last_undegraded":"2023-06-10T18:55:05.813340+0000","last_fullsized":"2023-06-10T18:55:05.813340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:49.842452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.816810+0000","last_change":"2023-06-10T18:55:05.816810+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.816810+0000","last_undegraded":"2023-06-10T18:55:05.816810+0000","last_fullsized":"2023-06-10T18:55:05.816810+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":164,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.823698+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:55:05.823698+0000","last_peered":"2023-06-10T18:55:05.823698+0000","last_clean":"2023-06-10T18:55:05.823698+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:55:05.823698+0000","last_undegraded":"2023-06-10T18:55:05.823698+0000","last_fullsized":"2023-06-10T18:55:05.823698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:52.672254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.813614+0000","last_change":"2023-06-10T18:55:05.813614+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.813614+0000","last_undegraded":"2023-06-10T18:55:05.813614+0000","last_fullsized":"2023-06-10T18:55:05.813614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":194,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.814772+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:55:05.814772+0000","last_peered":"2023-06-10T18:55:05.814772+0000","last_clean":"2023-06-10T18:55:05.814772+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:55:05.814772+0000","last_undegraded":"2023-06-10T18:55:05.814772+0000","last_fullsized":"2023-06-10T18:55:05.814772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:27.739184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":334,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.815489+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:55:05.815489+0000","last_peered":"2023-06-10T18:55:05.815489+0000","last_clean":"2023-06-10T18:55:05.815489+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:55:05.815489+0000","last_undegraded":"2023-06-10T18:55:05.815489+0000","last_fullsized":"2023-06-10T18:55:05.815489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:45:56.896704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.823664+0000","last_change":"2023-06-10T18:55:05.823664+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.823664+0000","last_undegraded":"2023-06-10T18:55:05.823664+0000","last_fullsized":"2023-06-10T18:55:05.823664+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":178,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.815479+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:55:05.815479+0000","last_peered":"2023-06-10T18:55:05.815479+0000","last_clean":"2023-06-10T18:55:05.815479+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:55:05.815479+0000","last_undegraded":"2023-06-10T18:55:05.815479+0000","last_fullsized":"2023-06-10T18:55:05.815479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:55:35.658085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820096+0000","last_change":"2023-06-10T18:55:05.820096+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820096+0000","last_undegraded":"2023-06-10T18:55:05.820096+0000","last_fullsized":"2023-06-10T18:55:05.820096+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":178,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.815477+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:55:05.815477+0000","last_peered":"2023-06-10T18:55:05.815477+0000","last_clean":"2023-06-10T18:55:05.815477+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:55:05.815477+0000","last_undegraded":"2023-06-10T18:55:05.815477+0000","last_fullsized":"2023-06-10T18:55:05.815477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:33:40.006287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.819896+0000","last_change":"2023-06-10T18:55:05.819896+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.819896+0000","last_undegraded":"2023-06-10T18:55:05.819896+0000","last_fullsized":"2023-06-10T18:55:05.819896+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":172,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.817337+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:55:05.817337+0000","last_peered":"2023-06-10T18:55:05.817337+0000","last_clean":"2023-06-10T18:55:05.817337+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:55:05.817337+0000","last_undegraded":"2023-06-10T18:55:05.817337+0000","last_fullsized":"2023-06-10T18:55:05.817337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:10:59.618403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:55:06.810285+0000","last_change":"2023-06-10T18:55:06.810285+0000","last_active":"2023-06-10T18:55:06.810285+0000","last_peered":"2023-06-10T18:55:06.810285+0000","last_clean":"2023-06-10T18:55:06.810285+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:06.810285+0000","last_undegraded":"2023-06-10T18:55:06.810285+0000","last_fullsized":"2023-06-10T18:55:06.810285+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":110,"state":"creating+peering","last_fresh":"2023-06-10T18:55:05.820358+0000","last_change":"2023-06-10T18:55:05.820358+0000","last_active":"2023-06-10T18:55:05.803572+0000","last_peered":"2023-06-10T18:55:05.803572+0000","last_clean":"2023-06-10T18:55:05.803572+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:55:05.820358+0000","last_undegraded":"2023-06-10T18:55:05.820358+0000","last_fullsized":"2023-06-10T18:55:05.820358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":166,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-06-10T18:55:05.824103+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:55:05.824103+0000","last_peered":"2023-06-10T18:55:05.824103+0000","last_clean":"2023-06-10T18:55:05.824103+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:55:05.824103+0000","last_undegraded":"2023-06-10T18:55:05.824103+0000","last_fullsized":"2023-06-10T18:55:05.824103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:12:58.716154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574986,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28864,"kb_used_data":1912,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342976,"statfs":{"total":96636764160,"available":96607207424,"internally_reserved":0,"allocated":1957888,"data_stored":1247759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57799999999999996,"15min":0.54100000000000004},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.247},"max":{"1min":0.68500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.55300000000000005},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76900000000000002,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.53000000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.54300000000000004,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.68400000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55600000000000005,"15min":0.52400000000000002},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.434,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.629,"5min":0.80100000000000005,"15min":0.91700000000000004},"last":0.48099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47599999999999998},"min":{"1min":0.32200000000000001,"5min":0.29799999999999999,"15min":0.25600000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.89700000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.55200000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.71299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.505}]},{"osd":5,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67700000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.311},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.45400000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.70999999999999996,"5min":0.73099999999999998,"15min":0.81100000000000005},"last":0.46800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574987,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1347584,"data_stored":649882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56799999999999995,"5min":0.71799999999999997,"15min":0.81100000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53000000000000003,"15min":0.51000000000000001},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.73799999999999999,"5min":0.748,"15min":0.78700000000000003},"last":0.51100000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.309},"max":{"1min":0.626,"5min":0.67100000000000004,"15min":0.80600000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.71099999999999997},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.65300000000000002},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69399999999999995,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71299999999999997,"5min":0.76300000000000001,"15min":1.252},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.38900000000000001,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.79900000000000004,"15min":0.82899999999999996},"last":0.628},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73999999999999999,"15min":0.77500000000000002},"last":0.56999999999999995}]},{"osd":4,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.504},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.66400000000000003,"5min":0.69599999999999995,"15min":0.82999999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.27600000000000002},"max":{"1min":0.68500000000000005,"5min":0.71199999999999997,"15min":0.89600000000000002},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.64600000000000002,"5min":0.73699999999999999,"15min":0.86699999999999999},"last":0.442},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.54200000000000004},"min":{"1min":0.35699999999999998,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.79100000000000004},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.76200000000000001,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.57599999999999996},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.76300000000000001,"15min":0.81399999999999995},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1347584,"data_stored":649882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.183,"15min":0.183},"max":{"1min":0.378,"5min":0.51500000000000001,"15min":0.753},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36099999999999999,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.219,"15min":0.219},"max":{"1min":0.40400000000000003,"5min":0.53900000000000003,"15min":0.59599999999999997},"last":0.37}]},{"osd":2,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.251,"5min":0.215,"15min":0.20000000000000001},"max":{"1min":0.38800000000000001,"5min":0.53100000000000003,"15min":0.64500000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38200000000000001,"15min":0.38},"min":{"1min":0.27800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.44}]},{"osd":3,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.40300000000000002,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.252,"15min":0.251},"max":{"1min":0.46300000000000002,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":0.47399999999999998,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":1.1040000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.45200000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.621,"15min":0.629},"min":{"1min":0.44,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.76000000000000001,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.83999999999999997,"5min":0.85999999999999999,"15min":0.93000000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.629},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.83499999999999996,"5min":0.871,"15min":1.008},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.84999999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":1.206},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.873,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.36899999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95099999999999996},"last":1.224},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80400000000000005,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.58499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574986,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28860,"kb_used_data":1908,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342980,"statfs":{"total":96636764160,"available":96607211520,"internally_reserved":0,"allocated":1953792,"data_stored":1246740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":0.40799999999999997,"5min":0.51600000000000001,"15min":0.72699999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35299999999999998,"15min":0.35399999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.47699999999999998,"5min":0.58499999999999996,"15min":0.88900000000000001},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35699999999999998,"15min":0.36899999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.44900000000000001,"5min":0.59899999999999998,"15min":1.405},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.43099999999999999,"5min":0.65500000000000003,"15min":0.95499999999999996},"last":0.28399999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.37},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.46700000000000003,"5min":0.61799999999999999,"15min":0.93100000000000005},"last":0.31},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40600000000000003,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51900000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.92600000000000005,"15min":1.0489999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.77800000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.623}]},{"osd":5,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69899999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69399999999999995,"5min":0.90000000000000002,"15min":0.98199999999999998},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38800000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.97699999999999998,"15min":1.363},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.72699999999999998,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.89800000000000002,"15min":1.0149999999999999},"last":0.69699999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574987,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1363968,"data_stored":657391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.40400000000000003,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.312},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45000000000000001,"15min":0.38500000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46600000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.40999999999999998,"15min":0.371},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":2.847,"15min":2.847},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.46000000000000002,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.25900000000000001,"15min":0.23799999999999999},"max":{"1min":0.51400000000000001,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.45300000000000001,"15min":0.38700000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.443,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.45200000000000001,"15min":0.40699999999999997},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.25800000000000001},"max":{"1min":0.46100000000000002,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.38700000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.65400000000000003,"15min":0.60999999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73099999999999998,"5min":5.742,"15min":5.742},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.65100000000000002,"15min":0.61099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.55000000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.68600000000000005,"15min":0.64600000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":6.593,"15min":6.593},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70299999999999996,"15min":0.64600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.78000000000000003,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.626}]},{"osd":6,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.68100000000000005,"15min":0.63300000000000001},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.79900000000000004,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.66000000000000003,"15min":0.621},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.745,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.52500000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.71399999999999997,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.81200000000000006,"5min":6.617,"15min":6.617},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.74199999999999999,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.58699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574986,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28272,"kb_used_data":1320,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343568,"statfs":{"total":96636764160,"available":96607813632,"internally_reserved":0,"allocated":1351680,"data_stored":650901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.36099999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.19400000000000001},"max":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.38500000000000001},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36399999999999999,"15min":0.378},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.31},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.255,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.501,"5min":0.68600000000000005,"15min":1.3},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.45400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.59799999999999998,"5min":1.5960000000000001,"15min":1.599},"last":0.33200000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.5049999999999999,"15min":1.925},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61199999999999999,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84099999999999997,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.437}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.50800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.86499999999999999,"5min":1.613,"15min":1.613},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61799999999999999,"15min":0.68899999999999995},"min":{"1min":0.44700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82899999999999996,"5min":1.4410000000000001,"15min":3.802},"last":0.443}]},{"osd":6,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61799999999999999,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.752,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.627,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.39600000000000002,"15min":0.375},"max":{"1min":0.83399999999999996,"5min":1.679,"15min":1.679},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.48999999999999999,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":1.47,"15min":1.47},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.548,"15min":1.8939999999999999},"last":0.54700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1328,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1359872,"data_stored":656372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.755,"15min":1.119},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.30199999999999999,"15min":0.29999999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.40799999999999997,"5min":0.37,"15min":0.27100000000000002},"max":{"1min":0.67500000000000004,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.41799999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56499999999999995,"15min":0.55000000000000004},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":1.0229999999999999},"last":0.437}]},{"osd":3,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61299999999999999,"5min":0.748,"15min":0.79800000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.85299999999999998,"15min":1.042},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49199999999999999,"15min":0.495},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.069},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.505,"15min":0.51000000000000001},"min":{"1min":0.33300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.73599999999999999,"15min":0.88300000000000001},"last":0.52000000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.28199999999999997},"max":{"1min":0.76400000000000001,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.3169999999999999},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.54000000000000004},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.70399999999999996,"5min":0.80000000000000004,"15min":0.85899999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.35799999999999998},"max":{"1min":0.79000000000000004,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.70999999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574987,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28272,"kb_used_data":1320,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343568,"statfs":{"total":96636764160,"available":96607813632,"internally_reserved":0,"allocated":1351680,"data_stored":650901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52000000000000002},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.32300000000000001},"max":{"1min":0.70699999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.52200000000000002},"min":{"1min":0.42299999999999999,"5min":0.35499999999999998,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.372},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.442},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.53400000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.68000000000000005,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.53600000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58999999999999997,"15min":0.56000000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.497}]},{"osd":4,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83999999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39200000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.79200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73499999999999999,"5min":0.73799999999999999,"15min":0.879},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.77400000000000002,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58999999999999997,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:55:11.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:55:11.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.453+0000 7f57b40e8700 1 Processor -- start 2023-06-10T18:55:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.453+0000 7f57b40e8700 1 -- start start 2023-06-10T18:55:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b40e8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b40e8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b40e8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ac06d290 con 0x7f57ac0ca100 2023-06-10T18:55:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b40e8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ac071640 con 0x7f57ac06e4b0 2023-06-10T18:55:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b40e8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac0aef00 con 0x7f57ac06e0e0 2023-06-10T18:55:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b2685700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f57ac0ca100 0x7f57ac0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34170/0 (socket says 172.21.15.104:34170) 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36342/0 (socket says 172.21.15.104:36342) 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b2685700 1 -- 172.21.15.104:0/1195072848 learned_addr learned my addr 172.21.15.104:0/1195072848 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 -- 172.21.15.104:0/1195072848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 msgr2=0x7f57ac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1e84700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b0e82700 1 -- 172.21.15.104:0/1195072848 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149059157 0 0) 0x7f57ac06d290 con 0x7f57ac0ca100 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 -- 172.21.15.104:0/1195072848 >> v1:172.21.15.104:6789/0 conn(0x7f57ac0ca100 legacy=0x7f57ac0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1683700 1 -- 172.21.15.104:0/1195072848 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac06d4c0 con 0x7f57ac06e4b0 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.454+0000 7f57b1e84700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b1683700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f57a8009840 tx=0x7f57a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb941e54e4943d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b0e82700 1 -- 172.21.15.104:0/1195072848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8011830 con 0x7f57ac06e4b0 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b0e82700 1 -- 172.21.15.104:0/1195072848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a8011990 con 0x7f57ac06e4b0 2023-06-10T18:55:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b0e82700 1 -- 172.21.15.104:0/1195072848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8011d20 con 0x7f57ac06e4b0 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 -- 172.21.15.104:0/1195072848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac06e4b0 msgr2=0x7f57ac0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f57a8009840 tx=0x7f57a8009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 -- 172.21.15.104:0/1195072848 shutdown_connections 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 --2- 172.21.15.104:0/1195072848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac06e4b0 0x7f57ac0c9bc0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 -- 172.21.15.104:0/1195072848 >> 172.21.15.104:0/1195072848 conn(0x7f57ac06b8a0 msgr2=0x7f57ac0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:11.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 -- 172.21.15.104:0/1195072848 shutdown_connections 2023-06-10T18:55:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.455+0000 7f57b40e8700 1 -- 172.21.15.104:0/1195072848 wait complete. 2023-06-10T18:55:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.456+0000 7f57b40e8700 1 Processor -- start 2023-06-10T18:55:11.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 -- start start 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac1479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 0x7f57ac1e1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac15a4b0 con 0x7f57ac0ca100 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac1e20b0 con 0x7f57ac06e4b0 2023-06-10T18:55:11.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b40e8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac1e2390 con 0x7f57ac06e0e0 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b2685700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b2685700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36350/0 (socket says 172.21.15.104:36350) 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.457+0000 7f57b2685700 1 -- 172.21.15.104:0/3444500752 learned_addr learned my addr 172.21.15.104:0/3444500752 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b1e84700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac1479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b1683700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 0x7f57ac1e1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 -- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 msgr2=0x7f57ac1479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac1479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 -- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 msgr2=0x7f57ac1e1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 0x7f57ac1e1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 -- 172.21.15.104:0/3444500752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80094a0 con 0x7f57ac0ca100 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b1683700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 0x7f57ac1e1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b2685700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f57a400cca0 tx=0x7f57a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a400cea0 con 0x7f57ac0ca100 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a400ade0 con 0x7f57ac0ca100 2023-06-10T18:55:11.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ac1e2670 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4007310 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.458+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ac1e2c30 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.459+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ac1e2e50 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.459+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f57a4007470 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.460+0000 7f57a27fc700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 0x7f579006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.460+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f57a4089510 con 0x7f57ac0ca100 2023-06-10T18:55:11.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.460+0000 7f57b1e84700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 0x7f579006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.460+0000 7f57b1e84700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 0x7f579006d190 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f579c00b840 tx=0x7f579c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.464+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f57a4054480 con 0x7f57ac0ca100 2023-06-10T18:55:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.647+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57ac06da80 con 0x7f579006ace0 2023-06-10T18:55:11.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.656+0000 7f57a27fc700 1 -- 172.21.15.104:0/3444500752 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312272 (secure 0 0 0) 0x7f579c00c040 con 0x7f579006ace0 2023-06-10T18:55:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:11.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 msgr2=0x7f579006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 0x7f579006d190 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f579c00b840 tx=0x7f579c008040 comp rx=0 tx=0).stop 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 msgr2=0x7f57ac1e1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f57a400cca0 tx=0x7f57a400a340 comp rx=0 tx=0).stop 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 shutdown_connections 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57ac0ca100 0x7f57ac1e1a80 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57ac06e0e0 0x7f57ac1479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f579006ace0 0x7f579006d190 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 --2- 172.21.15.104:0/3444500752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57ac06e4b0 0x7f57ac1e1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 >> 172.21.15.104:0/3444500752 conn(0x7f57ac06b8a0 msgr2=0x7f57ac0ac230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 shutdown_connections 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.659+0000 7f57b40e8700 1 -- 172.21.15.104:0/3444500752 wait complete. 2023-06-10T18:55:11.669 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:55:11.677 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2023-06-10T18:55:11.528132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227768,"kb_used_data":12152,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754746952,"statfs":{"total":773094113280,"available":772860878848,"internally_reserved":0,"allocated":12443648,"data_stored":6611464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.716269"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360629+0000","last_change":"2023-06-10T18:55:07.232126+0000","last_active":"2023-06-10T18:55:10.360629+0000","last_peered":"2023-06-10T18:55:10.360629+0000","last_clean":"2023-06-10T18:55:10.360629+0000","last_became_active":"2023-06-10T18:55:07.231882+0000","last_became_peered":"2023-06-10T18:55:07.231882+0000","last_unstale":"2023-06-10T18:55:10.360629+0000","last_undegraded":"2023-06-10T18:55:10.360629+0000","last_fullsized":"2023-06-10T18:55:10.360629+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:49:44.839785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512406+0000","last_change":"2023-06-10T18:55:07.235771+0000","last_active":"2023-06-10T18:55:08.512406+0000","last_peered":"2023-06-10T18:55:08.512406+0000","last_clean":"2023-06-10T18:55:08.512406+0000","last_became_active":"2023-06-10T18:55:07.235464+0000","last_became_peered":"2023-06-10T18:55:07.235464+0000","last_unstale":"2023-06-10T18:55:08.512406+0000","last_undegraded":"2023-06-10T18:55:08.512406+0000","last_fullsized":"2023-06-10T18:55:08.512406+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:58:34.221255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451427+0000","last_change":"2023-06-10T18:55:07.236386+0000","last_active":"2023-06-10T18:55:08.451427+0000","last_peered":"2023-06-10T18:55:08.451427+0000","last_clean":"2023-06-10T18:55:08.451427+0000","last_became_active":"2023-06-10T18:55:07.236152+0000","last_became_peered":"2023-06-10T18:55:07.236152+0000","last_unstale":"2023-06-10T18:55:08.451427+0000","last_undegraded":"2023-06-10T18:55:08.451427+0000","last_fullsized":"2023-06-10T18:55:08.451427+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:07:11.500618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360558+0000","last_change":"2023-06-10T18:55:07.234810+0000","last_active":"2023-06-10T18:55:10.360558+0000","last_peered":"2023-06-10T18:55:10.360558+0000","last_clean":"2023-06-10T18:55:10.360558+0000","last_became_active":"2023-06-10T18:55:07.234552+0000","last_became_peered":"2023-06-10T18:55:07.234552+0000","last_unstale":"2023-06-10T18:55:10.360558+0000","last_undegraded":"2023-06-10T18:55:10.360558+0000","last_fullsized":"2023-06-10T18:55:10.360558+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:09:32.029442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580682+0000","last_change":"2023-06-10T18:55:07.241128+0000","last_active":"2023-06-10T18:55:09.580682+0000","last_peered":"2023-06-10T18:55:09.580682+0000","last_clean":"2023-06-10T18:55:09.580682+0000","last_became_active":"2023-06-10T18:55:07.240853+0000","last_became_peered":"2023-06-10T18:55:07.240853+0000","last_unstale":"2023-06-10T18:55:09.580682+0000","last_undegraded":"2023-06-10T18:55:09.580682+0000","last_fullsized":"2023-06-10T18:55:09.580682+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:44:42.819448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219138+0000","last_change":"2023-06-10T18:55:07.224743+0000","last_active":"2023-06-10T18:55:08.219138+0000","last_peered":"2023-06-10T18:55:08.219138+0000","last_clean":"2023-06-10T18:55:08.219138+0000","last_became_active":"2023-06-10T18:55:07.224156+0000","last_became_peered":"2023-06-10T18:55:07.224156+0000","last_unstale":"2023-06-10T18:55:08.219138+0000","last_undegraded":"2023-06-10T18:55:08.219138+0000","last_fullsized":"2023-06-10T18:55:08.219138+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:58:34.221255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218960+0000","last_change":"2023-06-10T18:55:07.232520+0000","last_active":"2023-06-10T18:55:08.218960+0000","last_peered":"2023-06-10T18:55:08.218960+0000","last_clean":"2023-06-10T18:55:08.218960+0000","last_became_active":"2023-06-10T18:55:07.232095+0000","last_became_peered":"2023-06-10T18:55:07.232095+0000","last_unstale":"2023-06-10T18:55:08.218960+0000","last_undegraded":"2023-06-10T18:55:08.218960+0000","last_fullsized":"2023-06-10T18:55:08.218960+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:45:00.971205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451425+0000","last_change":"2023-06-10T18:55:07.232530+0000","last_active":"2023-06-10T18:55:08.451425+0000","last_peered":"2023-06-10T18:55:08.451425+0000","last_clean":"2023-06-10T18:55:08.451425+0000","last_became_active":"2023-06-10T18:55:07.231518+0000","last_became_peered":"2023-06-10T18:55:07.231518+0000","last_unstale":"2023-06-10T18:55:08.451425+0000","last_undegraded":"2023-06-10T18:55:08.451425+0000","last_fullsized":"2023-06-10T18:55:08.451425+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:10:56.806237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360498+0000","last_change":"2023-06-10T18:55:07.241878+0000","last_active":"2023-06-10T18:55:10.360498+0000","last_peered":"2023-06-10T18:55:10.360498+0000","last_clean":"2023-06-10T18:55:10.360498+0000","last_became_active":"2023-06-10T18:55:07.241519+0000","last_became_peered":"2023-06-10T18:55:07.241519+0000","last_unstale":"2023-06-10T18:55:10.360498+0000","last_undegraded":"2023-06-10T18:55:10.360498+0000","last_fullsized":"2023-06-10T18:55:10.360498+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:31.204253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360614+0000","last_change":"2023-06-10T18:55:07.243368+0000","last_active":"2023-06-10T18:55:10.360614+0000","last_peered":"2023-06-10T18:55:10.360614+0000","last_clean":"2023-06-10T18:55:10.360614+0000","last_became_active":"2023-06-10T18:55:07.243054+0000","last_became_peered":"2023-06-10T18:55:07.243054+0000","last_unstale":"2023-06-10T18:55:10.360614+0000","last_undegraded":"2023-06-10T18:55:10.360614+0000","last_fullsized":"2023-06-10T18:55:10.360614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:54.144303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512300+0000","last_change":"2023-06-10T18:55:07.233749+0000","last_active":"2023-06-10T18:55:08.512300+0000","last_peered":"2023-06-10T18:55:08.512300+0000","last_clean":"2023-06-10T18:55:08.512300+0000","last_became_active":"2023-06-10T18:55:07.233187+0000","last_became_peered":"2023-06-10T18:55:07.233187+0000","last_unstale":"2023-06-10T18:55:08.512300+0000","last_undegraded":"2023-06-10T18:55:08.512300+0000","last_fullsized":"2023-06-10T18:55:08.512300+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:49:44.839785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360461+0000","last_change":"2023-06-10T18:55:07.244611+0000","last_active":"2023-06-10T18:55:10.360461+0000","last_peered":"2023-06-10T18:55:10.360461+0000","last_clean":"2023-06-10T18:55:10.360461+0000","last_became_active":"2023-06-10T18:55:07.244252+0000","last_became_peered":"2023-06-10T18:55:07.244252+0000","last_unstale":"2023-06-10T18:55:10.360461+0000","last_undegraded":"2023-06-10T18:55:10.360461+0000","last_fullsized":"2023-06-10T18:55:10.360461+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:44.724578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451437+0000","last_change":"2023-06-10T18:55:07.243045+0000","last_active":"2023-06-10T18:55:08.451437+0000","last_peered":"2023-06-10T18:55:08.451437+0000","last_clean":"2023-06-10T18:55:08.451437+0000","last_became_active":"2023-06-10T18:55:07.242761+0000","last_became_peered":"2023-06-10T18:55:07.242761+0000","last_unstale":"2023-06-10T18:55:08.451437+0000","last_undegraded":"2023-06-10T18:55:08.451437+0000","last_fullsized":"2023-06-10T18:55:08.451437+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:36.979104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.450999+0000","last_change":"2023-06-10T18:55:07.231362+0000","last_active":"2023-06-10T18:55:08.450999+0000","last_peered":"2023-06-10T18:55:08.450999+0000","last_clean":"2023-06-10T18:55:08.450999+0000","last_became_active":"2023-06-10T18:55:07.231080+0000","last_became_peered":"2023-06-10T18:55:07.231080+0000","last_unstale":"2023-06-10T18:55:08.450999+0000","last_undegraded":"2023-06-10T18:55:08.450999+0000","last_fullsized":"2023-06-10T18:55:08.450999+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:17:37.097285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218810+0000","last_change":"2023-06-10T18:55:07.231182+0000","last_active":"2023-06-10T18:55:08.218810+0000","last_peered":"2023-06-10T18:55:08.218810+0000","last_clean":"2023-06-10T18:55:08.218810+0000","last_became_active":"2023-06-10T18:55:07.230908+0000","last_became_peered":"2023-06-10T18:55:07.230908+0000","last_unstale":"2023-06-10T18:55:08.218810+0000","last_undegraded":"2023-06-10T18:55:08.218810+0000","last_fullsized":"2023-06-10T18:55:08.218810+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:11:32.988068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360117+0000","last_change":"2023-06-10T18:55:07.230896+0000","last_active":"2023-06-10T18:55:10.360117+0000","last_peered":"2023-06-10T18:55:10.360117+0000","last_clean":"2023-06-10T18:55:10.360117+0000","last_became_active":"2023-06-10T18:55:07.230487+0000","last_became_peered":"2023-06-10T18:55:07.230487+0000","last_unstale":"2023-06-10T18:55:10.360117+0000","last_undegraded":"2023-06-10T18:55:10.360117+0000","last_fullsized":"2023-06-10T18:55:10.360117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:55:41.371063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512390+0000","last_change":"2023-06-10T18:55:07.234139+0000","last_active":"2023-06-10T18:55:08.512390+0000","last_peered":"2023-06-10T18:55:08.512390+0000","last_clean":"2023-06-10T18:55:08.512390+0000","last_became_active":"2023-06-10T18:55:07.233537+0000","last_became_peered":"2023-06-10T18:55:07.233537+0000","last_unstale":"2023-06-10T18:55:08.512390+0000","last_undegraded":"2023-06-10T18:55:08.512390+0000","last_fullsized":"2023-06-10T18:55:08.512390+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:57:49.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.365988+0000","last_change":"2023-06-10T18:55:07.237217+0000","last_active":"2023-06-10T18:55:08.365988+0000","last_peered":"2023-06-10T18:55:08.365988+0000","last_clean":"2023-06-10T18:55:08.365988+0000","last_became_active":"2023-06-10T18:55:07.236932+0000","last_became_peered":"2023-06-10T18:55:07.236932+0000","last_unstale":"2023-06-10T18:55:08.365988+0000","last_undegraded":"2023-06-10T18:55:08.365988+0000","last_fullsized":"2023-06-10T18:55:08.365988+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:47:05.709682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451538+0000","last_change":"2023-06-10T18:55:07.243039+0000","last_active":"2023-06-10T18:55:08.451538+0000","last_peered":"2023-06-10T18:55:08.451538+0000","last_clean":"2023-06-10T18:55:08.451538+0000","last_became_active":"2023-06-10T18:55:07.242743+0000","last_became_peered":"2023-06-10T18:55:07.242743+0000","last_unstale":"2023-06-10T18:55:08.451538+0000","last_undegraded":"2023-06-10T18:55:08.451538+0000","last_fullsized":"2023-06-10T18:55:08.451538+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:24:20.554511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512106+0000","last_change":"2023-06-10T18:55:07.234472+0000","last_active":"2023-06-10T18:55:08.512106+0000","last_peered":"2023-06-10T18:55:08.512106+0000","last_clean":"2023-06-10T18:55:08.512106+0000","last_became_active":"2023-06-10T18:55:07.233621+0000","last_became_peered":"2023-06-10T18:55:07.233621+0000","last_unstale":"2023-06-10T18:55:08.512106+0000","last_undegraded":"2023-06-10T18:55:08.512106+0000","last_fullsized":"2023-06-10T18:55:08.512106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:54.132709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512164+0000","last_change":"2023-06-10T18:55:07.238752+0000","last_active":"2023-06-10T18:55:08.512164+0000","last_peered":"2023-06-10T18:55:08.512164+0000","last_clean":"2023-06-10T18:55:08.512164+0000","last_became_active":"2023-06-10T18:55:07.238459+0000","last_became_peered":"2023-06-10T18:55:07.238459+0000","last_unstale":"2023-06-10T18:55:08.512164+0000","last_undegraded":"2023-06-10T18:55:08.512164+0000","last_fullsized":"2023-06-10T18:55:08.512164+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:33:54.144303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218590+0000","last_change":"2023-06-10T18:55:07.235810+0000","last_active":"2023-06-10T18:55:08.218590+0000","last_peered":"2023-06-10T18:55:08.218590+0000","last_clean":"2023-06-10T18:55:08.218590+0000","last_became_active":"2023-06-10T18:55:07.234186+0000","last_became_peered":"2023-06-10T18:55:07.234186+0000","last_unstale":"2023-06-10T18:55:08.218590+0000","last_undegraded":"2023-06-10T18:55:08.218590+0000","last_fullsized":"2023-06-10T18:55:08.218590+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:03:12.879229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.511921+0000","last_change":"2023-06-10T18:55:07.234350+0000","last_active":"2023-06-10T18:55:08.511921+0000","last_peered":"2023-06-10T18:55:08.511921+0000","last_clean":"2023-06-10T18:55:08.511921+0000","last_became_active":"2023-06-10T18:55:07.233378+0000","last_became_peered":"2023-06-10T18:55:07.233378+0000","last_unstale":"2023-06-10T18:55:08.511921+0000","last_undegraded":"2023-06-10T18:55:08.511921+0000","last_fullsized":"2023-06-10T18:55:08.511921+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:39:55.130588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218798+0000","last_change":"2023-06-10T18:55:07.227166+0000","last_active":"2023-06-10T18:55:08.218798+0000","last_peered":"2023-06-10T18:55:08.218798+0000","last_clean":"2023-06-10T18:55:08.218798+0000","last_became_active":"2023-06-10T18:55:07.226539+0000","last_became_peered":"2023-06-10T18:55:07.226539+0000","last_unstale":"2023-06-10T18:55:08.218798+0000","last_undegraded":"2023-06-10T18:55:08.218798+0000","last_fullsized":"2023-06-10T18:55:08.218798+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:15:23.213407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512542+0000","last_change":"2023-06-10T18:55:07.233888+0000","last_active":"2023-06-10T18:55:08.512542+0000","last_peered":"2023-06-10T18:55:08.512542+0000","last_clean":"2023-06-10T18:55:08.512542+0000","last_became_active":"2023-06-10T18:55:07.233250+0000","last_became_peered":"2023-06-10T18:55:07.233250+0000","last_unstale":"2023-06-10T18:55:08.512542+0000","last_undegraded":"2023-06-10T18:55:08.512542+0000","last_fullsized":"2023-06-10T18:55:08.512542+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:54.109180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360624+0000","last_change":"2023-06-10T18:55:07.230922+0000","last_active":"2023-06-10T18:55:10.360624+0000","last_peered":"2023-06-10T18:55:10.360624+0000","last_clean":"2023-06-10T18:55:10.360624+0000","last_became_active":"2023-06-10T18:55:07.230477+0000","last_became_peered":"2023-06-10T18:55:07.230477+0000","last_unstale":"2023-06-10T18:55:10.360624+0000","last_undegraded":"2023-06-10T18:55:10.360624+0000","last_fullsized":"2023-06-10T18:55:10.360624+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:24:46.780952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219452+0000","last_change":"2023-06-10T18:55:07.232227+0000","last_active":"2023-06-10T18:55:08.219452+0000","last_peered":"2023-06-10T18:55:08.219452+0000","last_clean":"2023-06-10T18:55:08.219452+0000","last_became_active":"2023-06-10T18:55:07.231676+0000","last_became_peered":"2023-06-10T18:55:07.231676+0000","last_unstale":"2023-06-10T18:55:08.219452+0000","last_undegraded":"2023-06-10T18:55:08.219452+0000","last_fullsized":"2023-06-10T18:55:08.219452+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:51:00.104768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219426+0000","last_change":"2023-06-10T18:55:07.235262+0000","last_active":"2023-06-10T18:55:08.219426+0000","last_peered":"2023-06-10T18:55:08.219426+0000","last_clean":"2023-06-10T18:55:08.219426+0000","last_became_active":"2023-06-10T18:55:07.234875+0000","last_became_peered":"2023-06-10T18:55:07.234875+0000","last_unstale":"2023-06-10T18:55:08.219426+0000","last_undegraded":"2023-06-10T18:55:08.219426+0000","last_fullsized":"2023-06-10T18:55:08.219426+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:43:55.467205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512166+0000","last_change":"2023-06-10T18:55:07.235676+0000","last_active":"2023-06-10T18:55:08.512166+0000","last_peered":"2023-06-10T18:55:08.512166+0000","last_clean":"2023-06-10T18:55:08.512166+0000","last_became_active":"2023-06-10T18:55:07.235362+0000","last_became_peered":"2023-06-10T18:55:07.235362+0000","last_unstale":"2023-06-10T18:55:08.512166+0000","last_undegraded":"2023-06-10T18:55:08.512166+0000","last_fullsized":"2023-06-10T18:55:08.512166+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:09:32.029442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219150+0000","last_change":"2023-06-10T18:55:07.227911+0000","last_active":"2023-06-10T18:55:08.219150+0000","last_peered":"2023-06-10T18:55:08.219150+0000","last_clean":"2023-06-10T18:55:08.219150+0000","last_became_active":"2023-06-10T18:55:07.227396+0000","last_became_peered":"2023-06-10T18:55:07.227396+0000","last_unstale":"2023-06-10T18:55:08.219150+0000","last_undegraded":"2023-06-10T18:55:08.219150+0000","last_fullsized":"2023-06-10T18:55:08.219150+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:10.452040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360008+0000","last_change":"2023-06-10T18:55:07.245315+0000","last_active":"2023-06-10T18:55:10.360008+0000","last_peered":"2023-06-10T18:55:10.360008+0000","last_clean":"2023-06-10T18:55:10.360008+0000","last_became_active":"2023-06-10T18:55:07.244806+0000","last_became_peered":"2023-06-10T18:55:07.244806+0000","last_unstale":"2023-06-10T18:55:10.360008+0000","last_undegraded":"2023-06-10T18:55:10.360008+0000","last_fullsized":"2023-06-10T18:55:10.360008+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:05.209797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218799+0000","last_change":"2023-06-10T18:55:07.224420+0000","last_active":"2023-06-10T18:55:08.218799+0000","last_peered":"2023-06-10T18:55:08.218799+0000","last_clean":"2023-06-10T18:55:08.218799+0000","last_became_active":"2023-06-10T18:55:07.222937+0000","last_became_peered":"2023-06-10T18:55:07.222937+0000","last_unstale":"2023-06-10T18:55:08.218799+0000","last_undegraded":"2023-06-10T18:55:08.218799+0000","last_fullsized":"2023-06-10T18:55:08.218799+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:57:49.210128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219434+0000","last_change":"2023-06-10T18:55:07.242957+0000","last_active":"2023-06-10T18:55:08.219434+0000","last_peered":"2023-06-10T18:55:08.219434+0000","last_clean":"2023-06-10T18:55:08.219434+0000","last_became_active":"2023-06-10T18:55:07.242687+0000","last_became_peered":"2023-06-10T18:55:07.242687+0000","last_unstale":"2023-06-10T18:55:08.219434+0000","last_undegraded":"2023-06-10T18:55:08.219434+0000","last_fullsized":"2023-06-10T18:55:08.219434+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:22:04.676894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'4","reported_seq":17,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219503+0000","last_change":"2023-06-10T18:55:06.835311+0000","last_active":"2023-06-10T18:55:08.219503+0000","last_peered":"2023-06-10T18:55:08.219503+0000","last_clean":"2023-06-10T18:55:08.219503+0000","last_became_active":"2023-06-10T18:55:06.835047+0000","last_became_peered":"2023-06-10T18:55:06.835047+0000","last_unstale":"2023-06-10T18:55:08.219503+0000","last_undegraded":"2023-06-10T18:55:08.219503+0000","last_fullsized":"2023-06-10T18:55:08.219503+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:07.951607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.366234+0000","last_change":"2023-06-10T18:55:07.232750+0000","last_active":"2023-06-10T18:55:08.366234+0000","last_peered":"2023-06-10T18:55:08.366234+0000","last_clean":"2023-06-10T18:55:08.366234+0000","last_became_active":"2023-06-10T18:55:07.232397+0000","last_became_peered":"2023-06-10T18:55:07.232397+0000","last_unstale":"2023-06-10T18:55:08.366234+0000","last_undegraded":"2023-06-10T18:55:08.366234+0000","last_fullsized":"2023-06-10T18:55:08.366234+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:19:37.745112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"112'1","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451700+0000","last_change":"2023-06-10T18:55:06.836607+0000","last_active":"2023-06-10T18:55:08.451700+0000","last_peered":"2023-06-10T18:55:08.451700+0000","last_clean":"2023-06-10T18:55:08.451700+0000","last_became_active":"2023-06-10T18:55:06.836361+0000","last_became_peered":"2023-06-10T18:55:06.836361+0000","last_unstale":"2023-06-10T18:55:08.451700+0000","last_undegraded":"2023-06-10T18:55:08.451700+0000","last_fullsized":"2023-06-10T18:55:08.451700+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:09:38.068390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219641+0000","last_change":"2023-06-10T18:55:07.222628+0000","last_active":"2023-06-10T18:55:08.219641+0000","last_peered":"2023-06-10T18:55:08.219641+0000","last_clean":"2023-06-10T18:55:08.219641+0000","last_became_active":"2023-06-10T18:55:07.222354+0000","last_became_peered":"2023-06-10T18:55:07.222354+0000","last_unstale":"2023-06-10T18:55:08.219641+0000","last_undegraded":"2023-06-10T18:55:08.219641+0000","last_fullsized":"2023-06-10T18:55:08.219641+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:28:44.054155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580652+0000","last_change":"2023-06-10T18:55:06.832583+0000","last_active":"2023-06-10T18:55:09.580652+0000","last_peered":"2023-06-10T18:55:09.580652+0000","last_clean":"2023-06-10T18:55:09.580652+0000","last_became_active":"2023-06-10T18:55:06.832318+0000","last_became_peered":"2023-06-10T18:55:06.832318+0000","last_unstale":"2023-06-10T18:55:09.580652+0000","last_undegraded":"2023-06-10T18:55:09.580652+0000","last_fullsized":"2023-06-10T18:55:09.580652+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:02.122238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451569+0000","last_change":"2023-06-10T18:55:07.239223+0000","last_active":"2023-06-10T18:55:08.451569+0000","last_peered":"2023-06-10T18:55:08.451569+0000","last_clean":"2023-06-10T18:55:08.451569+0000","last_became_active":"2023-06-10T18:55:07.238941+0000","last_became_peered":"2023-06-10T18:55:07.238941+0000","last_unstale":"2023-06-10T18:55:08.451569+0000","last_undegraded":"2023-06-10T18:55:08.451569+0000","last_fullsized":"2023-06-10T18:55:08.451569+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:49:49.618873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580586+0000","last_change":"2023-06-10T18:55:06.836065+0000","last_active":"2023-06-10T18:55:09.580586+0000","last_peered":"2023-06-10T18:55:09.580586+0000","last_clean":"2023-06-10T18:55:09.580586+0000","last_became_active":"2023-06-10T18:55:06.835785+0000","last_became_peered":"2023-06-10T18:55:06.835785+0000","last_unstale":"2023-06-10T18:55:09.580586+0000","last_undegraded":"2023-06-10T18:55:09.580586+0000","last_fullsized":"2023-06-10T18:55:09.580586+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:05:08.133915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360232+0000","last_change":"2023-06-10T18:55:07.232886+0000","last_active":"2023-06-10T18:55:10.360232+0000","last_peered":"2023-06-10T18:55:10.360232+0000","last_clean":"2023-06-10T18:55:10.360232+0000","last_became_active":"2023-06-10T18:55:07.232630+0000","last_became_peered":"2023-06-10T18:55:07.232630+0000","last_unstale":"2023-06-10T18:55:10.360232+0000","last_undegraded":"2023-06-10T18:55:10.360232+0000","last_fullsized":"2023-06-10T18:55:10.360232+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:18:29.375078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"112'1","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219006+0000","last_change":"2023-06-10T18:55:06.835842+0000","last_active":"2023-06-10T18:55:08.219006+0000","last_peered":"2023-06-10T18:55:08.219006+0000","last_clean":"2023-06-10T18:55:08.219006+0000","last_became_active":"2023-06-10T18:55:06.835590+0000","last_became_peered":"2023-06-10T18:55:06.835590+0000","last_unstale":"2023-06-10T18:55:08.219006+0000","last_undegraded":"2023-06-10T18:55:08.219006+0000","last_fullsized":"2023-06-10T18:55:08.219006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:49:56.767752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512375+0000","last_change":"2023-06-10T18:55:07.233770+0000","last_active":"2023-06-10T18:55:08.512375+0000","last_peered":"2023-06-10T18:55:08.512375+0000","last_clean":"2023-06-10T18:55:08.512375+0000","last_became_active":"2023-06-10T18:55:07.233262+0000","last_became_peered":"2023-06-10T18:55:07.233262+0000","last_unstale":"2023-06-10T18:55:08.512375+0000","last_undegraded":"2023-06-10T18:55:08.512375+0000","last_fullsized":"2023-06-10T18:55:08.512375+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:11:32.988068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"112'1","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218679+0000","last_change":"2023-06-10T18:55:06.829848+0000","last_active":"2023-06-10T18:55:08.218679+0000","last_peered":"2023-06-10T18:55:08.218679+0000","last_clean":"2023-06-10T18:55:08.218679+0000","last_became_active":"2023-06-10T18:55:06.829588+0000","last_became_peered":"2023-06-10T18:55:06.829588+0000","last_unstale":"2023-06-10T18:55:08.218679+0000","last_undegraded":"2023-06-10T18:55:08.218679+0000","last_fullsized":"2023-06-10T18:55:08.218679+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:41:18.563546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.359968+0000","last_change":"2023-06-10T18:55:07.231412+0000","last_active":"2023-06-10T18:55:10.359968+0000","last_peered":"2023-06-10T18:55:10.359968+0000","last_clean":"2023-06-10T18:55:10.359968+0000","last_became_active":"2023-06-10T18:55:07.231156+0000","last_became_peered":"2023-06-10T18:55:07.231156+0000","last_unstale":"2023-06-10T18:55:10.359968+0000","last_undegraded":"2023-06-10T18:55:10.359968+0000","last_fullsized":"2023-06-10T18:55:10.359968+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:20:41.991248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.511967+0000","last_change":"2023-06-10T18:55:06.830698+0000","last_active":"2023-06-10T18:55:08.511967+0000","last_peered":"2023-06-10T18:55:08.511967+0000","last_clean":"2023-06-10T18:55:08.511967+0000","last_became_active":"2023-06-10T18:55:06.830354+0000","last_became_peered":"2023-06-10T18:55:06.830354+0000","last_unstale":"2023-06-10T18:55:08.511967+0000","last_undegraded":"2023-06-10T18:55:08.511967+0000","last_fullsized":"2023-06-10T18:55:08.511967+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:30.197540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218996+0000","last_change":"2023-06-10T18:55:07.230948+0000","last_active":"2023-06-10T18:55:08.218996+0000","last_peered":"2023-06-10T18:55:08.218996+0000","last_clean":"2023-06-10T18:55:08.218996+0000","last_became_active":"2023-06-10T18:55:07.230668+0000","last_became_peered":"2023-06-10T18:55:07.230668+0000","last_unstale":"2023-06-10T18:55:08.218996+0000","last_undegraded":"2023-06-10T18:55:08.218996+0000","last_fullsized":"2023-06-10T18:55:08.218996+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:22.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.359909+0000","last_change":"2023-06-10T18:55:06.822960+0000","last_active":"2023-06-10T18:55:10.359909+0000","last_peered":"2023-06-10T18:55:10.359909+0000","last_clean":"2023-06-10T18:55:10.359909+0000","last_became_active":"2023-06-10T18:55:06.822630+0000","last_became_peered":"2023-06-10T18:55:06.822630+0000","last_unstale":"2023-06-10T18:55:10.359909+0000","last_undegraded":"2023-06-10T18:55:10.359909+0000","last_fullsized":"2023-06-10T18:55:10.359909+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:36:22.039735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580874+0000","last_change":"2023-06-10T18:55:07.241051+0000","last_active":"2023-06-10T18:55:09.580874+0000","last_peered":"2023-06-10T18:55:09.580874+0000","last_clean":"2023-06-10T18:55:09.580874+0000","last_became_active":"2023-06-10T18:55:07.240797+0000","last_became_peered":"2023-06-10T18:55:07.240797+0000","last_unstale":"2023-06-10T18:55:09.580874+0000","last_undegraded":"2023-06-10T18:55:09.580874+0000","last_fullsized":"2023-06-10T18:55:09.580874+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:43:57.520845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'1","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360308+0000","last_change":"2023-06-10T18:55:06.836175+0000","last_active":"2023-06-10T18:55:10.360308+0000","last_peered":"2023-06-10T18:55:10.360308+0000","last_clean":"2023-06-10T18:55:10.360308+0000","last_became_active":"2023-06-10T18:55:06.835858+0000","last_became_peered":"2023-06-10T18:55:06.835858+0000","last_unstale":"2023-06-10T18:55:10.360308+0000","last_undegraded":"2023-06-10T18:55:10.360308+0000","last_fullsized":"2023-06-10T18:55:10.360308+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:39:29.795519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219388+0000","last_change":"2023-06-10T18:55:07.231932+0000","last_active":"2023-06-10T18:55:08.219388+0000","last_peered":"2023-06-10T18:55:08.219388+0000","last_clean":"2023-06-10T18:55:08.219388+0000","last_became_active":"2023-06-10T18:55:07.231631+0000","last_became_peered":"2023-06-10T18:55:07.231631+0000","last_unstale":"2023-06-10T18:55:08.219388+0000","last_undegraded":"2023-06-10T18:55:08.219388+0000","last_fullsized":"2023-06-10T18:55:08.219388+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:47:27.817993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580908+0000","last_change":"2023-06-10T18:55:06.831597+0000","last_active":"2023-06-10T18:55:09.580908+0000","last_peered":"2023-06-10T18:55:09.580908+0000","last_clean":"2023-06-10T18:55:09.580908+0000","last_became_active":"2023-06-10T18:55:06.831334+0000","last_became_peered":"2023-06-10T18:55:06.831334+0000","last_unstale":"2023-06-10T18:55:09.580908+0000","last_undegraded":"2023-06-10T18:55:09.580908+0000","last_fullsized":"2023-06-10T18:55:09.580908+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:58.069433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451329+0000","last_change":"2023-06-10T18:55:07.233832+0000","last_active":"2023-06-10T18:55:08.451329+0000","last_peered":"2023-06-10T18:55:08.451329+0000","last_clean":"2023-06-10T18:55:08.451329+0000","last_became_active":"2023-06-10T18:55:07.232578+0000","last_became_peered":"2023-06-10T18:55:07.232578+0000","last_unstale":"2023-06-10T18:55:08.451329+0000","last_undegraded":"2023-06-10T18:55:08.451329+0000","last_fullsized":"2023-06-10T18:55:08.451329+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:16:38.192403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'1","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.366071+0000","last_change":"2023-06-10T18:55:06.837564+0000","last_active":"2023-06-10T18:55:08.366071+0000","last_peered":"2023-06-10T18:55:08.366071+0000","last_clean":"2023-06-10T18:55:08.366071+0000","last_became_active":"2023-06-10T18:55:06.837109+0000","last_became_peered":"2023-06-10T18:55:06.837109+0000","last_unstale":"2023-06-10T18:55:08.366071+0000","last_undegraded":"2023-06-10T18:55:08.366071+0000","last_fullsized":"2023-06-10T18:55:08.366071+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:01:29.922107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451633+0000","last_change":"2023-06-10T18:55:07.228664+0000","last_active":"2023-06-10T18:55:08.451633+0000","last_peered":"2023-06-10T18:55:08.451633+0000","last_clean":"2023-06-10T18:55:08.451633+0000","last_became_active":"2023-06-10T18:55:07.228355+0000","last_became_peered":"2023-06-10T18:55:07.228355+0000","last_unstale":"2023-06-10T18:55:08.451633+0000","last_undegraded":"2023-06-10T18:55:08.451633+0000","last_fullsized":"2023-06-10T18:55:08.451633+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:32:37.080859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219125+0000","last_change":"2023-06-10T18:55:06.835452+0000","last_active":"2023-06-10T18:55:08.219125+0000","last_peered":"2023-06-10T18:55:08.219125+0000","last_clean":"2023-06-10T18:55:08.219125+0000","last_became_active":"2023-06-10T18:55:06.835124+0000","last_became_peered":"2023-06-10T18:55:06.835124+0000","last_unstale":"2023-06-10T18:55:08.219125+0000","last_undegraded":"2023-06-10T18:55:08.219125+0000","last_fullsized":"2023-06-10T18:55:08.219125+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:47:04.702970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451038+0000","last_change":"2023-06-10T18:55:07.235230+0000","last_active":"2023-06-10T18:55:08.451038+0000","last_peered":"2023-06-10T18:55:08.451038+0000","last_clean":"2023-06-10T18:55:08.451038+0000","last_became_active":"2023-06-10T18:55:07.234951+0000","last_became_peered":"2023-06-10T18:55:07.234951+0000","last_unstale":"2023-06-10T18:55:08.451038+0000","last_undegraded":"2023-06-10T18:55:08.451038+0000","last_fullsized":"2023-06-10T18:55:08.451038+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:39.884295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219212+0000","last_change":"2023-06-10T18:55:06.830679+0000","last_active":"2023-06-10T18:55:08.219212+0000","last_peered":"2023-06-10T18:55:08.219212+0000","last_clean":"2023-06-10T18:55:08.219212+0000","last_became_active":"2023-06-10T18:55:06.830426+0000","last_became_peered":"2023-06-10T18:55:06.830426+0000","last_unstale":"2023-06-10T18:55:08.219212+0000","last_undegraded":"2023-06-10T18:55:08.219212+0000","last_fullsized":"2023-06-10T18:55:08.219212+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:21:42.928300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360158+0000","last_change":"2023-06-10T18:55:07.233558+0000","last_active":"2023-06-10T18:55:10.360158+0000","last_peered":"2023-06-10T18:55:10.360158+0000","last_clean":"2023-06-10T18:55:10.360158+0000","last_became_active":"2023-06-10T18:55:07.233311+0000","last_became_peered":"2023-06-10T18:55:07.233311+0000","last_unstale":"2023-06-10T18:55:10.360158+0000","last_undegraded":"2023-06-10T18:55:10.360158+0000","last_fullsized":"2023-06-10T18:55:10.360158+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:57.421041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580494+0000","last_change":"2023-06-10T18:55:06.833471+0000","last_active":"2023-06-10T18:55:09.580494+0000","last_peered":"2023-06-10T18:55:09.580494+0000","last_clean":"2023-06-10T18:55:09.580494+0000","last_became_active":"2023-06-10T18:55:06.833195+0000","last_became_peered":"2023-06-10T18:55:06.833195+0000","last_unstale":"2023-06-10T18:55:09.580494+0000","last_undegraded":"2023-06-10T18:55:09.580494+0000","last_fullsized":"2023-06-10T18:55:09.580494+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:17:47.245860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451278+0000","last_change":"2023-06-10T18:55:07.243246+0000","last_active":"2023-06-10T18:55:08.451278+0000","last_peered":"2023-06-10T18:55:08.451278+0000","last_clean":"2023-06-10T18:55:08.451278+0000","last_became_active":"2023-06-10T18:55:07.242978+0000","last_became_peered":"2023-06-10T18:55:07.242978+0000","last_unstale":"2023-06-10T18:55:08.451278+0000","last_undegraded":"2023-06-10T18:55:08.451278+0000","last_fullsized":"2023-06-10T18:55:08.451278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:25:17.370555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218923+0000","last_change":"2023-06-10T18:55:06.831258+0000","last_active":"2023-06-10T18:55:08.218923+0000","last_peered":"2023-06-10T18:55:08.218923+0000","last_clean":"2023-06-10T18:55:08.218923+0000","last_became_active":"2023-06-10T18:55:06.830383+0000","last_became_peered":"2023-06-10T18:55:06.830383+0000","last_unstale":"2023-06-10T18:55:08.218923+0000","last_undegraded":"2023-06-10T18:55:08.218923+0000","last_fullsized":"2023-06-10T18:55:08.218923+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:10.805855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218363+0000","last_change":"2023-06-10T18:55:07.245825+0000","last_active":"2023-06-10T18:55:08.218363+0000","last_peered":"2023-06-10T18:55:08.218363+0000","last_clean":"2023-06-10T18:55:08.218363+0000","last_became_active":"2023-06-10T18:55:07.245573+0000","last_became_peered":"2023-06-10T18:55:07.245573+0000","last_unstale":"2023-06-10T18:55:08.218363+0000","last_undegraded":"2023-06-10T18:55:08.218363+0000","last_fullsized":"2023-06-10T18:55:08.218363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:50:57.426384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360080+0000","last_change":"2023-06-10T18:55:06.830804+0000","last_active":"2023-06-10T18:55:10.360080+0000","last_peered":"2023-06-10T18:55:10.360080+0000","last_clean":"2023-06-10T18:55:10.360080+0000","last_became_active":"2023-06-10T18:55:06.830374+0000","last_became_peered":"2023-06-10T18:55:06.830374+0000","last_unstale":"2023-06-10T18:55:10.360080+0000","last_undegraded":"2023-06-10T18:55:10.360080+0000","last_fullsized":"2023-06-10T18:55:10.360080+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:48:09.174933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.366192+0000","last_change":"2023-06-10T18:55:07.236575+0000","last_active":"2023-06-10T18:55:08.366192+0000","last_peered":"2023-06-10T18:55:08.366192+0000","last_clean":"2023-06-10T18:55:08.366192+0000","last_became_active":"2023-06-10T18:55:07.236299+0000","last_became_peered":"2023-06-10T18:55:07.236299+0000","last_unstale":"2023-06-10T18:55:08.366192+0000","last_undegraded":"2023-06-10T18:55:08.366192+0000","last_fullsized":"2023-06-10T18:55:08.366192+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:43:29.875742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219488+0000","last_change":"2023-06-10T18:55:06.821921+0000","last_active":"2023-06-10T18:55:08.219488+0000","last_peered":"2023-06-10T18:55:08.219488+0000","last_clean":"2023-06-10T18:55:08.219488+0000","last_became_active":"2023-06-10T18:55:06.821622+0000","last_became_peered":"2023-06-10T18:55:06.821622+0000","last_unstale":"2023-06-10T18:55:08.219488+0000","last_undegraded":"2023-06-10T18:55:08.219488+0000","last_fullsized":"2023-06-10T18:55:08.219488+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:53:23.623118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360526+0000","last_change":"2023-06-10T18:55:07.236805+0000","last_active":"2023-06-10T18:55:10.360526+0000","last_peered":"2023-06-10T18:55:10.360526+0000","last_clean":"2023-06-10T18:55:10.360526+0000","last_became_active":"2023-06-10T18:55:07.236587+0000","last_became_peered":"2023-06-10T18:55:07.236587+0000","last_unstale":"2023-06-10T18:55:10.360526+0000","last_undegraded":"2023-06-10T18:55:10.360526+0000","last_fullsized":"2023-06-10T18:55:10.360526+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:39:54.132709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.365927+0000","last_change":"2023-06-10T18:55:06.835719+0000","last_active":"2023-06-10T18:55:08.365927+0000","last_peered":"2023-06-10T18:55:08.365927+0000","last_clean":"2023-06-10T18:55:08.365927+0000","last_became_active":"2023-06-10T18:55:06.835358+0000","last_became_peered":"2023-06-10T18:55:06.835358+0000","last_unstale":"2023-06-10T18:55:08.365927+0000","last_undegraded":"2023-06-10T18:55:08.365927+0000","last_fullsized":"2023-06-10T18:55:08.365927+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:04:41.534992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360356+0000","last_change":"2023-06-10T18:55:07.231813+0000","last_active":"2023-06-10T18:55:10.360356+0000","last_peered":"2023-06-10T18:55:10.360356+0000","last_clean":"2023-06-10T18:55:10.360356+0000","last_became_active":"2023-06-10T18:55:07.231446+0000","last_became_peered":"2023-06-10T18:55:07.231446+0000","last_unstale":"2023-06-10T18:55:10.360356+0000","last_undegraded":"2023-06-10T18:55:10.360356+0000","last_fullsized":"2023-06-10T18:55:10.360356+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:21:23.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360399+0000","last_change":"2023-06-10T18:55:06.831601+0000","last_active":"2023-06-10T18:55:10.360399+0000","last_peered":"2023-06-10T18:55:10.360399+0000","last_clean":"2023-06-10T18:55:10.360399+0000","last_became_active":"2023-06-10T18:55:06.831318+0000","last_became_peered":"2023-06-10T18:55:06.831318+0000","last_unstale":"2023-06-10T18:55:10.360399+0000","last_undegraded":"2023-06-10T18:55:10.360399+0000","last_fullsized":"2023-06-10T18:55:10.360399+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:39:54.123876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.365961+0000","last_change":"2023-06-10T18:55:07.244702+0000","last_active":"2023-06-10T18:55:08.365961+0000","last_peered":"2023-06-10T18:55:08.365961+0000","last_clean":"2023-06-10T18:55:08.365961+0000","last_became_active":"2023-06-10T18:55:07.244393+0000","last_became_peered":"2023-06-10T18:55:07.244393+0000","last_unstale":"2023-06-10T18:55:08.365961+0000","last_undegraded":"2023-06-10T18:55:08.365961+0000","last_fullsized":"2023-06-10T18:55:08.365961+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:10.452040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219615+0000","last_change":"2023-06-10T18:55:06.837524+0000","last_active":"2023-06-10T18:55:08.219615+0000","last_peered":"2023-06-10T18:55:08.219615+0000","last_clean":"2023-06-10T18:55:08.219615+0000","last_became_active":"2023-06-10T18:55:06.837255+0000","last_became_peered":"2023-06-10T18:55:06.837255+0000","last_unstale":"2023-06-10T18:55:08.219615+0000","last_undegraded":"2023-06-10T18:55:08.219615+0000","last_fullsized":"2023-06-10T18:55:08.219615+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:53:56.790815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.511955+0000","last_change":"2023-06-10T18:55:07.234192+0000","last_active":"2023-06-10T18:55:08.511955+0000","last_peered":"2023-06-10T18:55:08.511955+0000","last_clean":"2023-06-10T18:55:08.511955+0000","last_became_active":"2023-06-10T18:55:07.233131+0000","last_became_peered":"2023-06-10T18:55:07.233131+0000","last_unstale":"2023-06-10T18:55:08.511955+0000","last_undegraded":"2023-06-10T18:55:08.511955+0000","last_fullsized":"2023-06-10T18:55:08.511955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:44.724578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219178+0000","last_change":"2023-06-10T18:55:06.835813+0000","last_active":"2023-06-10T18:55:08.219178+0000","last_peered":"2023-06-10T18:55:08.219178+0000","last_clean":"2023-06-10T18:55:08.219178+0000","last_became_active":"2023-06-10T18:55:06.835562+0000","last_became_peered":"2023-06-10T18:55:06.835562+0000","last_unstale":"2023-06-10T18:55:08.219178+0000","last_undegraded":"2023-06-10T18:55:08.219178+0000","last_fullsized":"2023-06-10T18:55:08.219178+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:04:41.534992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512383+0000","last_change":"2023-06-10T18:55:07.244688+0000","last_active":"2023-06-10T18:55:08.512383+0000","last_peered":"2023-06-10T18:55:08.512383+0000","last_clean":"2023-06-10T18:55:08.512383+0000","last_became_active":"2023-06-10T18:55:07.244381+0000","last_became_peered":"2023-06-10T18:55:07.244381+0000","last_unstale":"2023-06-10T18:55:08.512383+0000","last_undegraded":"2023-06-10T18:55:08.512383+0000","last_fullsized":"2023-06-10T18:55:08.512383+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:11.812568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.512123+0000","last_change":"2023-06-10T18:55:06.830500+0000","last_active":"2023-06-10T18:55:08.512123+0000","last_peered":"2023-06-10T18:55:08.512123+0000","last_clean":"2023-06-10T18:55:08.512123+0000","last_became_active":"2023-06-10T18:55:06.830138+0000","last_became_peered":"2023-06-10T18:55:06.830138+0000","last_unstale":"2023-06-10T18:55:08.512123+0000","last_undegraded":"2023-06-10T18:55:08.512123+0000","last_fullsized":"2023-06-10T18:55:08.512123+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:24:45.774239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451220+0000","last_change":"2023-06-10T18:55:07.230132+0000","last_active":"2023-06-10T18:55:08.451220+0000","last_peered":"2023-06-10T18:55:08.451220+0000","last_clean":"2023-06-10T18:55:08.451220+0000","last_became_active":"2023-06-10T18:55:07.229708+0000","last_became_peered":"2023-06-10T18:55:07.229708+0000","last_unstale":"2023-06-10T18:55:08.451220+0000","last_undegraded":"2023-06-10T18:55:08.451220+0000","last_fullsized":"2023-06-10T18:55:08.451220+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:08:56.274714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580469+0000","last_change":"2023-06-10T18:55:06.832426+0000","last_active":"2023-06-10T18:55:09.580469+0000","last_peered":"2023-06-10T18:55:09.580469+0000","last_clean":"2023-06-10T18:55:09.580469+0000","last_became_active":"2023-06-10T18:55:06.832160+0000","last_became_peered":"2023-06-10T18:55:06.832160+0000","last_unstale":"2023-06-10T18:55:09.580469+0000","last_undegraded":"2023-06-10T18:55:09.580469+0000","last_fullsized":"2023-06-10T18:55:09.580469+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:26:46.744049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360092+0000","last_change":"2023-06-10T18:55:07.245101+0000","last_active":"2023-06-10T18:55:10.360092+0000","last_peered":"2023-06-10T18:55:10.360092+0000","last_clean":"2023-06-10T18:55:10.360092+0000","last_became_active":"2023-06-10T18:55:07.244528+0000","last_became_peered":"2023-06-10T18:55:07.244528+0000","last_unstale":"2023-06-10T18:55:10.360092+0000","last_undegraded":"2023-06-10T18:55:10.360092+0000","last_fullsized":"2023-06-10T18:55:10.360092+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:19:13.503734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360198+0000","last_change":"2023-06-10T18:55:06.830630+0000","last_active":"2023-06-10T18:55:10.360198+0000","last_peered":"2023-06-10T18:55:10.360198+0000","last_clean":"2023-06-10T18:55:10.360198+0000","last_became_active":"2023-06-10T18:55:06.830069+0000","last_became_peered":"2023-06-10T18:55:06.830069+0000","last_unstale":"2023-06-10T18:55:10.360198+0000","last_undegraded":"2023-06-10T18:55:10.360198+0000","last_fullsized":"2023-06-10T18:55:10.360198+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:25.416436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.366252+0000","last_change":"2023-06-10T18:55:07.237906+0000","last_active":"2023-06-10T18:55:08.366252+0000","last_peered":"2023-06-10T18:55:08.366252+0000","last_clean":"2023-06-10T18:55:08.366252+0000","last_became_active":"2023-06-10T18:55:07.237718+0000","last_became_peered":"2023-06-10T18:55:07.237718+0000","last_unstale":"2023-06-10T18:55:08.366252+0000","last_undegraded":"2023-06-10T18:55:08.366252+0000","last_fullsized":"2023-06-10T18:55:08.366252+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:08.958320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219342+0000","last_change":"2023-06-10T18:55:06.831760+0000","last_active":"2023-06-10T18:55:08.219342+0000","last_peered":"2023-06-10T18:55:08.219342+0000","last_clean":"2023-06-10T18:55:08.219342+0000","last_became_active":"2023-06-10T18:55:06.830907+0000","last_became_peered":"2023-06-10T18:55:06.830907+0000","last_unstale":"2023-06-10T18:55:08.219342+0000","last_undegraded":"2023-06-10T18:55:08.219342+0000","last_fullsized":"2023-06-10T18:55:08.219342+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:12:05.716385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219693+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:55:08.219693+0000","last_peered":"2023-06-10T18:55:08.219693+0000","last_clean":"2023-06-10T18:55:08.219693+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:55:08.219693+0000","last_undegraded":"2023-06-10T18:55:08.219693+0000","last_fullsized":"2023-06-10T18:55:08.219693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:35.567579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218311+0000","last_change":"2023-06-10T18:55:07.234301+0000","last_active":"2023-06-10T18:55:08.218311+0000","last_peered":"2023-06-10T18:55:08.218311+0000","last_clean":"2023-06-10T18:55:08.218311+0000","last_became_active":"2023-06-10T18:55:07.234049+0000","last_became_peered":"2023-06-10T18:55:07.234049+0000","last_unstale":"2023-06-10T18:55:08.218311+0000","last_undegraded":"2023-06-10T18:55:08.218311+0000","last_fullsized":"2023-06-10T18:55:08.218311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:14:47.902394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'8","reported_seq":21,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580435+0000","last_change":"2023-06-10T18:55:06.834402+0000","last_active":"2023-06-10T18:55:09.580435+0000","last_peered":"2023-06-10T18:55:09.580435+0000","last_clean":"2023-06-10T18:55:09.580435+0000","last_became_active":"2023-06-10T18:55:06.834148+0000","last_became_peered":"2023-06-10T18:55:06.834148+0000","last_unstale":"2023-06-10T18:55:09.580435+0000","last_undegraded":"2023-06-10T18:55:09.580435+0000","last_fullsized":"2023-06-10T18:55:09.580435+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:57:41.700519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":200,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219003+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:55:08.219003+0000","last_peered":"2023-06-10T18:55:08.219003+0000","last_clean":"2023-06-10T18:55:08.219003+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:55:08.219003+0000","last_undegraded":"2023-06-10T18:55:08.219003+0000","last_fullsized":"2023-06-10T18:55:08.219003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:26:23.505151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218599+0000","last_change":"2023-06-10T18:55:07.229612+0000","last_active":"2023-06-10T18:55:08.218599+0000","last_peered":"2023-06-10T18:55:08.218599+0000","last_clean":"2023-06-10T18:55:08.218599+0000","last_became_active":"2023-06-10T18:55:07.229187+0000","last_became_peered":"2023-06-10T18:55:07.229187+0000","last_unstale":"2023-06-10T18:55:08.218599+0000","last_undegraded":"2023-06-10T18:55:08.218599+0000","last_fullsized":"2023-06-10T18:55:08.218599+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:40:25.782642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'2","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219162+0000","last_change":"2023-06-10T18:55:06.830397+0000","last_active":"2023-06-10T18:55:08.219162+0000","last_peered":"2023-06-10T18:55:08.219162+0000","last_clean":"2023-06-10T18:55:08.219162+0000","last_became_active":"2023-06-10T18:55:06.830139+0000","last_became_peered":"2023-06-10T18:55:06.830139+0000","last_unstale":"2023-06-10T18:55:08.219162+0000","last_undegraded":"2023-06-10T18:55:08.219162+0000","last_fullsized":"2023-06-10T18:55:08.219162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:53.102467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":170,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580510+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:55:09.580510+0000","last_peered":"2023-06-10T18:55:09.580510+0000","last_clean":"2023-06-10T18:55:09.580510+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:55:09.580510+0000","last_undegraded":"2023-06-10T18:55:09.580510+0000","last_fullsized":"2023-06-10T18:55:09.580510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:22:03.936450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218296+0000","last_change":"2023-06-10T18:55:07.227330+0000","last_active":"2023-06-10T18:55:08.218296+0000","last_peered":"2023-06-10T18:55:08.218296+0000","last_clean":"2023-06-10T18:55:08.218296+0000","last_became_active":"2023-06-10T18:55:07.226839+0000","last_became_peered":"2023-06-10T18:55:07.226839+0000","last_unstale":"2023-06-10T18:55:08.218296+0000","last_undegraded":"2023-06-10T18:55:08.218296+0000","last_fullsized":"2023-06-10T18:55:08.218296+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:39:38.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219090+0000","last_change":"2023-06-10T18:55:06.834204+0000","last_active":"2023-06-10T18:55:08.219090+0000","last_peered":"2023-06-10T18:55:08.219090+0000","last_clean":"2023-06-10T18:55:08.219090+0000","last_became_active":"2023-06-10T18:55:06.833936+0000","last_became_peered":"2023-06-10T18:55:06.833936+0000","last_unstale":"2023-06-10T18:55:08.219090+0000","last_undegraded":"2023-06-10T18:55:08.219090+0000","last_fullsized":"2023-06-10T18:55:08.219090+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:11:17.141044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219373+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:55:08.219373+0000","last_peered":"2023-06-10T18:55:08.219373+0000","last_clean":"2023-06-10T18:55:08.219373+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:55:08.219373+0000","last_undegraded":"2023-06-10T18:55:08.219373+0000","last_fullsized":"2023-06-10T18:55:08.219373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:26.638561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":340,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451030+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:55:08.451030+0000","last_peered":"2023-06-10T18:55:08.451030+0000","last_clean":"2023-06-10T18:55:08.451030+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:55:08.451030+0000","last_undegraded":"2023-06-10T18:55:08.451030+0000","last_fullsized":"2023-06-10T18:55:08.451030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:53:54.609261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.218515+0000","last_change":"2023-06-10T18:55:07.225554+0000","last_active":"2023-06-10T18:55:08.218515+0000","last_peered":"2023-06-10T18:55:08.218515+0000","last_clean":"2023-06-10T18:55:08.218515+0000","last_became_active":"2023-06-10T18:55:07.225275+0000","last_became_peered":"2023-06-10T18:55:07.225275+0000","last_unstale":"2023-06-10T18:55:08.218515+0000","last_undegraded":"2023-06-10T18:55:08.218515+0000","last_fullsized":"2023-06-10T18:55:08.218515+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:59:25.786675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.365899+0000","last_change":"2023-06-10T18:55:06.836290+0000","last_active":"2023-06-10T18:55:08.365899+0000","last_peered":"2023-06-10T18:55:08.365899+0000","last_clean":"2023-06-10T18:55:08.365899+0000","last_became_active":"2023-06-10T18:55:06.836014+0000","last_became_peered":"2023-06-10T18:55:06.836014+0000","last_unstale":"2023-06-10T18:55:08.365899+0000","last_undegraded":"2023-06-10T18:55:08.365899+0000","last_fullsized":"2023-06-10T18:55:08.365899+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:49:56.767752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":184,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.450977+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:55:08.450977+0000","last_peered":"2023-06-10T18:55:08.450977+0000","last_clean":"2023-06-10T18:55:08.450977+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:55:08.450977+0000","last_undegraded":"2023-06-10T18:55:08.450977+0000","last_fullsized":"2023-06-10T18:55:08.450977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:21:01.762332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219480+0000","last_change":"2023-06-10T18:55:07.243134+0000","last_active":"2023-06-10T18:55:08.219480+0000","last_peered":"2023-06-10T18:55:08.219480+0000","last_clean":"2023-06-10T18:55:08.219480+0000","last_became_active":"2023-06-10T18:55:07.242806+0000","last_became_peered":"2023-06-10T18:55:07.242806+0000","last_unstale":"2023-06-10T18:55:08.219480+0000","last_undegraded":"2023-06-10T18:55:08.219480+0000","last_fullsized":"2023-06-10T18:55:08.219480+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:43:29.875742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:09.580830+0000","last_change":"2023-06-10T18:55:06.832278+0000","last_active":"2023-06-10T18:55:09.580830+0000","last_peered":"2023-06-10T18:55:09.580830+0000","last_clean":"2023-06-10T18:55:09.580830+0000","last_became_active":"2023-06-10T18:55:06.832019+0000","last_became_peered":"2023-06-10T18:55:06.832019+0000","last_unstale":"2023-06-10T18:55:09.580830+0000","last_undegraded":"2023-06-10T18:55:09.580830+0000","last_fullsized":"2023-06-10T18:55:09.580830+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:05:58.286123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":184,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.451623+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:55:08.451623+0000","last_peered":"2023-06-10T18:55:08.451623+0000","last_clean":"2023-06-10T18:55:08.451623+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:55:08.451623+0000","last_undegraded":"2023-06-10T18:55:08.451623+0000","last_fullsized":"2023-06-10T18:55:08.451623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:23.286950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219493+0000","last_change":"2023-06-10T18:55:07.227965+0000","last_active":"2023-06-10T18:55:08.219493+0000","last_peered":"2023-06-10T18:55:08.219493+0000","last_clean":"2023-06-10T18:55:08.219493+0000","last_became_active":"2023-06-10T18:55:07.227647+0000","last_became_peered":"2023-06-10T18:55:07.227647+0000","last_unstale":"2023-06-10T18:55:08.219493+0000","last_undegraded":"2023-06-10T18:55:08.219493+0000","last_fullsized":"2023-06-10T18:55:08.219493+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:19:37.745112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'6","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360154+0000","last_change":"2023-06-10T18:55:06.828598+0000","last_active":"2023-06-10T18:55:10.360154+0000","last_peered":"2023-06-10T18:55:10.360154+0000","last_clean":"2023-06-10T18:55:10.360154+0000","last_became_active":"2023-06-10T18:55:06.824884+0000","last_became_peered":"2023-06-10T18:55:06.824884+0000","last_unstale":"2023-06-10T18:55:10.360154+0000","last_undegraded":"2023-06-10T18:55:10.360154+0000","last_fullsized":"2023-06-10T18:55:10.360154+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:42:49.524787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219265+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:55:08.219265+0000","last_peered":"2023-06-10T18:55:08.219265+0000","last_clean":"2023-06-10T18:55:08.219265+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:55:08.219265+0000","last_undegraded":"2023-06-10T18:55:08.219265+0000","last_fullsized":"2023-06-10T18:55:08.219265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:12:58.716154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.219559+0000","last_change":"2023-06-10T18:55:07.235110+0000","last_active":"2023-06-10T18:55:08.219559+0000","last_peered":"2023-06-10T18:55:08.219559+0000","last_clean":"2023-06-10T18:55:08.219559+0000","last_became_active":"2023-06-10T18:55:07.234559+0000","last_became_peered":"2023-06-10T18:55:07.234559+0000","last_unstale":"2023-06-10T18:55:08.219559+0000","last_undegraded":"2023-06-10T18:55:08.219559+0000","last_fullsized":"2023-06-10T18:55:08.219559+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:06.810285+0000","last_clean_scrub_stamp":"2023-06-10T18:55:06.810285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:34:48.165365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'3","reported_seq":16,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:10.360304+0000","last_change":"2023-06-10T18:55:06.832405+0000","last_active":"2023-06-10T18:55:10.360304+0000","last_peered":"2023-06-10T18:55:10.360304+0000","last_clean":"2023-06-10T18:55:10.360304+0000","last_became_active":"2023-06-10T18:55:06.832136+0000","last_became_peered":"2023-06-10T18:55:06.832136+0000","last_unstale":"2023-06-10T18:55:10.360304+0000","last_undegraded":"2023-06-10T18:55:10.360304+0000","last_fullsized":"2023-06-10T18:55:10.360304+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:55:05.803572+0000","last_clean_scrub_stamp":"2023-06-10T18:55:05.803572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:49:09.158527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":172,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-06-10T18:55:08.366290+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:55:08.366290+0000","last_peered":"2023-06-10T18:55:08.366290+0000","last_clean":"2023-06-10T18:55:08.366290+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:55:08.366290+0000","last_undegraded":"2023-06-10T18:55:08.366290+0000","last_fullsized":"2023-06-10T18:55:08.366290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:35:07.985328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574987,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28908,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342932,"statfs":{"total":96636764160,"available":96607162368,"internally_reserved":0,"allocated":2002944,"data_stored":1269744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57799999999999996,"15min":0.54100000000000004},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.247},"max":{"1min":0.68500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.55300000000000005},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.76900000000000002,"5min":0.82199999999999995,"15min":0.86099999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.54300000000000004,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.68400000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.36099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.55600000000000005,"15min":0.52400000000000002},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.317},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.434,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.629,"5min":0.80100000000000005,"15min":0.91700000000000004},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47599999999999998},"min":{"1min":0.32200000000000001,"5min":0.29799999999999999,"15min":0.25600000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.89700000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.55200000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.71299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67700000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.311},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:54:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.70999999999999996,"5min":0.73099999999999998,"15min":0.81100000000000005},"last":0.437}]}]},{"osd":6,"up_from":13,"seq":55834574988,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1400832,"data_stored":677811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56799999999999995,"5min":0.71799999999999997,"15min":0.81100000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53000000000000003,"15min":0.51000000000000001},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.73799999999999999,"5min":0.748,"15min":0.78700000000000003},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.309},"max":{"1min":0.626,"5min":0.67100000000000004,"15min":0.80600000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.71099999999999997},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.75800000000000001,"15min":9.6850000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.65300000000000002},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69399999999999995,"5min":0.74099999999999999,"15min":6.6689999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71299999999999997,"5min":0.76300000000000001,"15min":1.252},"last":0.44}]},{"osd":3,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.38900000000000001,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.79900000000000004,"15min":0.82899999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70299999999999996,"5min":0.73999999999999999,"15min":0.77500000000000002},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47899999999999998,"15min":0.504},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.66400000000000003,"5min":0.69599999999999995,"15min":0.82999999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.27600000000000002},"max":{"1min":0.68500000000000005,"5min":0.71199999999999997,"15min":0.89600000000000002},"last":0.56200000000000006}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.64600000000000002,"5min":0.73699999999999999,"15min":0.86699999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.54200000000000004},"min":{"1min":0.35699999999999998,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.79100000000000004},"last":0.42099999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.76200000000000001,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56000000000000005,"15min":0.57599999999999996},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.76300000000000001,"15min":0.81399999999999995},"last":0.52700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28324,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343516,"statfs":{"total":96636764160,"available":96607760384,"internally_reserved":0,"allocated":1404928,"data_stored":677865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.183,"15min":0.183},"max":{"1min":0.378,"5min":0.51500000000000001,"15min":0.753},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36099999999999999,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.219,"15min":0.219},"max":{"1min":0.40400000000000003,"5min":0.53900000000000003,"15min":0.59599999999999997},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.251,"5min":0.215,"15min":0.20000000000000001},"max":{"1min":0.38800000000000001,"5min":0.53100000000000003,"15min":0.64500000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38200000000000001,"15min":0.38},"min":{"1min":0.27800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.61099999999999999,"15min":0.71899999999999997},"last":0.309}]},{"osd":3,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.40300000000000002,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.252,"15min":0.251},"max":{"1min":0.46300000000000002,"5min":0.60999999999999999,"15min":0.66600000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":0.47399999999999998,"5min":0.63100000000000001,"15min":0.73799999999999999},"last":0.24099999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.45200000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.621,"15min":0.629},"min":{"1min":0.44,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.76000000000000001,"5min":0.91200000000000003,"15min":0.96599999999999997},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.83999999999999997,"5min":0.85999999999999999,"15min":0.93000000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.629},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.83499999999999996,"5min":0.871,"15min":1.008},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.84999999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.873,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:54:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.36899999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80400000000000005,"5min":0.91100000000000003,"15min":0.97099999999999997},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574987,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28916,"kb_used_data":1964,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342924,"statfs":{"total":96636764160,"available":96607154176,"internally_reserved":0,"allocated":2011136,"data_stored":1269785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":0.40799999999999997,"5min":0.51600000000000001,"15min":0.72699999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35299999999999998,"15min":0.35399999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.47699999999999998,"5min":0.58499999999999996,"15min":0.88900000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35699999999999998,"15min":0.36899999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.44900000000000001,"5min":0.59899999999999998,"15min":1.405},"last":0.378},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.43099999999999999,"5min":0.65500000000000003,"15min":0.95499999999999996},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.37},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.46700000000000003,"5min":0.61799999999999999,"15min":0.93100000000000005},"last":0.309},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40600000000000003,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51900000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.39500000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57499999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.92600000000000005,"15min":1.0489999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.77800000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.442}]},{"osd":5,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69899999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69399999999999995,"5min":0.90000000000000002,"15min":0.98199999999999998},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38800000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.66400000000000003,"5min":0.97699999999999998,"15min":1.363},"last":0.432}]},{"osd":7,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.72699999999999998,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.89800000000000002,"15min":1.0149999999999999},"last":0.505}]}]},{"osd":2,"up_from":13,"seq":55834574988,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1400832,"data_stored":678760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.376,"15min":0.34399999999999997},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.40400000000000003,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.45000000000000001,"15min":0.38500000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46600000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.214}]},{"osd":1,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.40999999999999998,"15min":0.371},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":2.847,"15min":2.847},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.46000000000000002,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.25900000000000001,"15min":0.23799999999999999},"max":{"1min":0.51400000000000001,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.251}]},{"osd":3,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.45300000000000001,"15min":0.38700000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.443,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.45200000000000001,"15min":0.40699999999999997},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.25800000000000001},"max":{"1min":0.46100000000000002,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.65400000000000003,"15min":0.60999999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73099999999999998,"5min":5.742,"15min":5.742},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.65100000000000002,"15min":0.61099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.628}]},{"osd":5,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.68600000000000005,"15min":0.64600000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":6.593,"15min":6.593},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.70299999999999996,"15min":0.64600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.78000000000000003,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.68100000000000005,"15min":0.63300000000000001},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.79900000000000004,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.66000000000000003,"15min":0.621},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.745,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:54:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.71399999999999997,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.81200000000000006,"5min":6.617,"15min":6.617},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.74199999999999999,"15min":0.65300000000000002},"min":{"1min":0.42999999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.70199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574987,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1384,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1417216,"data_stored":680394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.36099999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.19400000000000001},"max":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":1.2170000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.38500000000000001},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36399999999999999,"15min":0.378},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51400000000000001,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.31},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.255,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.501,"5min":0.68600000000000005,"15min":1.3},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28299999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.45400000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.59799999999999998,"5min":1.5960000000000001,"15min":1.599},"last":0.33200000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.5049999999999999,"15min":1.925},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61199999999999999,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84099999999999997,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.437}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.59699999999999998},"min":{"1min":0.50800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.86499999999999999,"5min":1.613,"15min":1.613},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61799999999999999,"15min":0.68899999999999995},"min":{"1min":0.44700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82899999999999996,"5min":1.4410000000000001,"15min":3.802},"last":0.443}]},{"osd":6,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61799999999999999,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.752,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.627,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.39600000000000002,"15min":0.375},"max":{"1min":0.83399999999999996,"5min":1.679,"15min":1.679},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.48999999999999999,"5min":0.40999999999999998,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":1.47,"15min":1.47},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.548,"15min":1.8939999999999999},"last":0.54700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574987,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1396736,"data_stored":677741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.28399999999999997,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.755,"15min":1.119},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.30199999999999999,"15min":0.29999999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.40799999999999997,"5min":0.37,"15min":0.27100000000000002},"max":{"1min":0.67500000000000004,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.41799999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.81100000000000005,"15min":0.96599999999999997},"last":0.73699999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56499999999999995,"15min":0.55000000000000004},"min":{"1min":0.44,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":1.0229999999999999},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53900000000000003,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61299999999999999,"5min":0.748,"15min":0.79800000000000004},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.85299999999999998,"15min":1.042},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49199999999999999,"15min":0.495},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.069},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.505,"15min":0.51000000000000001},"min":{"1min":0.33300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67400000000000004,"5min":0.73599999999999999,"15min":0.88300000000000001},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.53000000000000003},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.28199999999999997},"max":{"1min":0.76400000000000001,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.3169999999999999},"last":0.76900000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:54:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.54000000000000004},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.70399999999999996,"5min":0.80000000000000004,"15min":0.85899999999999999},"last":0.751},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.35799999999999998},"max":{"1min":0.79000000000000004,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.72599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574988,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28328,"kb_used_data":1376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343512,"statfs":{"total":96636764160,"available":96607756288,"internally_reserved":0,"allocated":1409024,"data_stored":679364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52000000000000002},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.32300000000000001},"max":{"1min":0.70699999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54000000000000004,"15min":0.52200000000000002},"min":{"1min":0.42299999999999999,"5min":0.35499999999999998,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.53200000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55100000000000005,"15min":0.53400000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.68000000000000005,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.439}]},{"osd":3,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.53600000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58999999999999997,"15min":0.56000000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83999999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.376}]},{"osd":6,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.79200000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73499999999999999,"5min":0.73799999999999999,"15min":0.879},"last":0.54500000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.77400000000000002,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58999999999999997,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:55:11.684 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:55:11.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:55:11.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.840+0000 7fd7c6f85700 1 Processor -- start 2023-06-10T18:55:11.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.840+0000 7fd7c6f85700 1 -- start start 2023-06-10T18:55:11.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c6f85700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c6f85700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c6f85700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c00ad3c0 con 0x7fd7c0157b30 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c6f85700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c0157f00 con 0x7fd7c00d0830 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c6f85700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00b0170 con 0x7fd7c00cea50 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c5522700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55740/0 (socket says 172.21.15.104:55740) 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c5522700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:52092/0 (socket says 172.21.15.104:52092) 2023-06-10T18:55:11.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 -- 172.21.15.104:0/2086082637 learned_addr learned my addr 172.21.15.104:0/2086082637 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c4d21700 1 --1- 172.21.15.104:0/2086082637 >> v1:172.21.15.114:6789/0 conn(0x7fd7c0157b30 0x7fd7c00ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40028/0 (socket says 172.21.15.104:40028) 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 -- 172.21.15.104:0/2086082637 >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 msgr2=0x7fd7c00b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 --2- 172.21.15.104:0/2086082637 >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 -- 172.21.15.104:0/2086082637 >> v1:172.21.15.114:6789/0 conn(0x7fd7c0157b30 legacy=0x7fd7c00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7bffff700 1 -- 172.21.15.104:0/2086082637 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00b0410 con 0x7fd7c00cea50 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.841+0000 7fd7c5522700 1 --2- 172.21.15.104:0/2086082637 >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7bffff700 1 --2- 172.21.15.104:0/2086082637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0098f0 tx=0x7fd7ac009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846a6110366d91fe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7bf7fe700 1 -- 172.21.15.104:0/2086082637 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac011830 con 0x7fd7c00cea50 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7bf7fe700 1 -- 172.21.15.104:0/2086082637 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac011990 con 0x7fd7c00cea50 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7bf7fe700 1 -- 172.21.15.104:0/2086082637 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac011d20 con 0x7fd7c00cea50 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 -- 172.21.15.104:0/2086082637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 msgr2=0x7fd7c00d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/2086082637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0098f0 tx=0x7fd7ac009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 -- 172.21.15.104:0/2086082637 shutdown_connections 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/2086082637 >> v2:172.21.15.114:3300/0 conn(0x7fd7c00d0830 0x7fd7c00b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/2086082637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c00cea50 0x7fd7c00d02f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 -- 172.21.15.104:0/2086082637 >> 172.21.15.104:0/2086082637 conn(0x7fd7c0089fd0 msgr2=0x7fd7c008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:11.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 -- 172.21.15.104:0/2086082637 shutdown_connections 2023-06-10T18:55:11.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.842+0000 7fd7c6f85700 1 -- 172.21.15.104:0/2086082637 wait complete. 2023-06-10T18:55:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.843+0000 7fd7c6f85700 1 Processor -- start 2023-06-10T18:55:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 -- start start 2023-06-10T18:55:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c00d0830 0x7fd7c01549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c0157b30 0x7fd7c015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c006c740 con 0x7fd7c00cea50 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c006af80 con 0x7fd7c00d0830 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.844+0000 7fd7c6f85700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c006b220 con 0x7fd7c0157b30 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36360/0 (socket says 172.21.15.104:36360) 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 -- 172.21.15.104:0/1073304436 learned_addr learned my addr 172.21.15.104:0/1073304436 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:11.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c5522700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c0157b30 0x7fd7c015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7bffff700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c00d0830 0x7fd7c01549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 -- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c0157b30 msgr2=0x7fd7c015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c0157b30 0x7fd7c015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 -- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c00d0830 msgr2=0x7fd7c01549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c00d0830 0x7fd7c01549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 -- 172.21.15.104:0/1073304436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b8009a70 con 0x7fd7c00cea50 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c4d21700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c500 tx=0x7fd7b400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400cea0 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b4005cb0 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac009550 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40072d0 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.845+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c006b7e0 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.846+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c0074050 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.846+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd7b4007430 con 0x7fd7c00cea50 2023-06-10T18:55:11.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.847+0000 7fd7bd7fa700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 0x7fd7a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.847+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd7b40887c0 con 0x7fd7c00cea50 2023-06-10T18:55:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.847+0000 7fd7bffff700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 0x7fd7a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.847+0000 7fd7bffff700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 0x7fd7a406d190 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00cee40 tx=0x7fd7ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:11.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:11.851+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd7b40537e0 con 0x7fd7c00cea50 2023-06-10T18:55:12.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.114+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd7c0070290 con 0x7fd7c00cea50 2023-06-10T18:55:12.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.117+0000 7fd7bd7fa700 1 -- 172.21.15.104:0/1073304436 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd7b4056e00 con 0x7fd7c00cea50 2023-06-10T18:55:12.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.119+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 msgr2=0x7fd7a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.119+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 0x7fd7a406d190 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00cee40 tx=0x7fd7ac008030 comp rx=0 tx=0).stop 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.119+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 msgr2=0x7fd7c01f2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.119+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c500 tx=0x7fd7b400c810 comp rx=0 tx=0).stop 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 shutdown_connections 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7c0157b30 0x7fd7c015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7c00cea50 0x7fd7c01f2ba0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd7a406ace0 0x7fd7a406d190 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 --2- 172.21.15.104:0/1073304436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c00d0830 0x7fd7c01549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 >> 172.21.15.104:0/1073304436 conn(0x7fd7c0089fd0 msgr2=0x7fd7c00ad5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:12.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 shutdown_connections 2023-06-10T18:55:12.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.120+0000 7fd7c6f85700 1 -- 172.21.15.104:0/1073304436 wait complete. 2023-06-10T18:55:12.127 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:55:12.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:12.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.253+0000 7fc8d150a700 1 Processor -- start 2023-06-10T18:55:12.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 -- start start 2023-06-10T18:55:12.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc8cc0cec20 0x7fc8cc15d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8cc06a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc06a810 con 0x7fc8cc156740 2023-06-10T18:55:12.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc15cdf0 con 0x7fc8cc0cec20 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8d150a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cf0a0 con 0x7fc8cc069e70 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8caffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc8cc156740 0x7fc8cc0cd250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34196/0 (socket says 172.21.15.104:34196) 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.254+0000 7fc8caffd700 1 -- 172.21.15.104:0/1865907349 learned_addr learned my addr 172.21.15.104:0/1865907349 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8cb7fe700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8cc06a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/1865907349 >> v2:172.21.15.114:3300/0 conn(0x7fc8cc0cec20 0x7fc8cc15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446072375 0 0) 0x7fc8cc06a810 con 0x7fc8cc156740 2023-06-10T18:55:12.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b80035b0 con 0x7fc8cc156740 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398300953 0 0) 0x7fc8b80035b0 con 0x7fc8cc156740 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 msgr2=0x7fc8cc06a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8cc06a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 >> v2:172.21.15.114:3300/0 conn(0x7fc8cc0cec20 msgr2=0x7fc8cc15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 --2- 172.21.15.104:0/1865907349 >> v2:172.21.15.114:3300/0 conn(0x7fc8cc0cec20 0x7fc8cc15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0cf380 con 0x7fc8cc156740 2023-06-10T18:55:12.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc8bc001670 con 0x7fc8cc156740 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 >> v1:172.21.15.104:6789/0 conn(0x7fc8cc156740 legacy=0x7fc8cc0cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8b8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8b8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8004d90 0x7fc8b8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc15cdf0 con 0x7fc8b8003f80 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cf0a0 con 0x7fc8b8004d90 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b800ba10 con 0x7fc8cc069e70 2023-06-10T18:55:12.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8b8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8caffd700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8b8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8cb7fe700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8004d90 0x7fc8b8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 -- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 msgr2=0x7fc8b8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc069e70 0x7fc8b8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 -- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8004d90 msgr2=0x7fc8b8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8004d90 0x7fc8b8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.255+0000 7fc8ca7fc700 1 -- 172.21.15.104:0/1865907349 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80035b0 con 0x7fc8b8003f80 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8b8004850 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009b60 tx=0x7fc8b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9494003eaf24e96 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400e6c0 con 0x7fc8b8003f80 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b400e820 con 0x7fc8b8003f80 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8c9ffb700 1 -- 172.21.15.104:0/1865907349 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400eb90 con 0x7fc8b8003f80 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 -- 172.21.15.104:0/1865907349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 msgr2=0x7fc8b8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 --2- 172.21.15.104:0/1865907349 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8b8004850 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009b60 tx=0x7fc8b8005880 comp rx=0 tx=0).stop 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8cb7fe700 1 -- 172.21.15.104:0/1865907349 reap_dead start 2023-06-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 -- 172.21.15.104:0/1865907349 shutdown_connections 2023-06-10T18:55:12.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 -- 172.21.15.104:0/1865907349 >> 172.21.15.104:0/1865907349 conn(0x7fc8cc06d240 msgr2=0x7fc8cc06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:12.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 -- 172.21.15.104:0/1865907349 shutdown_connections 2023-06-10T18:55:12.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.256+0000 7fc8d150a700 1 -- 172.21.15.104:0/1865907349 wait complete. 2023-06-10T18:55:12.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 Processor -- start 2023-06-10T18:55:12.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 -- start start 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc1f6110 0x7fc8cc0c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8cc151b60 0x7fc8cc151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc157630 con 0x7fc8b8003f80 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c5930 con 0x7fc8cc151b60 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8d150a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc156c90 con 0x7fc8cc1f6110 2023-06-10T18:55:12.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.258+0000 7fc8caffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36380/0 (socket says 172.21.15.104:36380) 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 -- 172.21.15.104:0/3722350835 learned_addr learned my addr 172.21.15.104:0/3722350835 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8cb7fe700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8cc151b60 0x7fc8cc151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc1f6110 0x7fc8cc0c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 -- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc1f6110 msgr2=0x7fc8cc0c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc1f6110 0x7fc8cc0c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 -- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8cc151b60 msgr2=0x7fc8cc151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8cc151b60 0x7fc8cc151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 -- 172.21.15.104:0/3722350835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc156f70 con 0x7fc8b8003f80 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8caffd700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc007310 tx=0x7fc8bc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011580 con 0x7fc8b8003f80 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc011700 con 0x7fc8b8003f80 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc157250 con 0x7fc8b8003f80 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011ab0 con 0x7fc8b8003f80 2023-06-10T18:55:12.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.259+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc15aa90 con 0x7fc8b8003f80 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.260+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc15ac80 con 0x7fc8b8003f80 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.260+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8bc011c10 con 0x7fc8b8003f80 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.261+0000 7fc8c37fe700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 0x7fc8ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.261+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc8bc089430 con 0x7fc8b8003f80 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.261+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 0x7fc8ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.261+0000 7fc8ca7fc700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 0x7fc8ac06d190 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40090b0 tx=0x7fc8b401f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:12.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.265+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8bc054430 con 0x7fc8b8003f80 2023-06-10T18:55:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.497+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc157770 con 0x7fc8b8003f80 2023-06-10T18:55:12.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.498+0000 7fc8c37fe700 1 -- 172.21.15.104:0/3722350835 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fc8cc157770 con 0x7fc8b8003f80 2023-06-10T18:55:12.503 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:12.503 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:10.222100+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:55:12.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 msgr2=0x7fc8ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 0x7fc8ac06d190 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40090b0 tx=0x7fc8b401f040 comp rx=0 tx=0).stop 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 msgr2=0x7fc8cc15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc007310 tx=0x7fc8bc007620 comp rx=0 tx=0).stop 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 shutdown_connections 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc8ac06ace0 0x7fc8ac06d190 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8cc151b60 0x7fc8cc151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8b8003f80 0x7fc8cc15a550 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 --2- 172.21.15.104:0/3722350835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8cc1f6110 0x7fc8cc0c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:12.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 >> 172.21.15.104:0/3722350835 conn(0x7fc8cc06d240 msgr2=0x7fc8cc0cf5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:12.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 shutdown_connections 2023-06-10T18:55:12.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:12.500+0000 7fc8d150a700 1 -- 172.21.15.104:0/3722350835 wait complete. 2023-06-10T18:55:12.507 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 70 2023-06-10T18:55:12.809 INFO:tasks.ceph.mds.c.smithi104.stderr:2023-06-10T18:55:12.804+0000 7fc40ec1a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:55:12.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:55:07.210651+0000', 'modified': '2023-06-10T18:55:10.222100+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8781': {'gid': 8781, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6835/2511953457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6833', 'nonce': 2511953457}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 2511953457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:55:12.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:55:12.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-06-10T18:55:13.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.037+0000 7f65058f7700 1 Processor -- start 2023-06-10T18:55:13.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.037+0000 7f65058f7700 1 -- start start 2023-06-10T18:55:13.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f65058f7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65000ce390 0x7f6500160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f65058f7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000cae10 0x7f65000cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f65058f7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65000d0340 con 0x7f650006c720 2023-06-10T18:55:13.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f65058f7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65000d1660 con 0x7f65000ce390 2023-06-10T18:55:13.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f65058f7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650015cb60 con 0x7f65000cae10 2023-06-10T18:55:13.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65000ce390 0x7f6500160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65000ce390 0x7f6500160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36388/0 (socket says 172.21.15.104:36388) 2023-06-10T18:55:13.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 -- 172.21.15.104:0/2606905235 learned_addr learned my addr 172.21.15.104:0/2606905235 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000cae10 0x7f65000cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 -- 172.21.15.104:0/2606905235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000cae10 msgr2=0x7f65000cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000cae10 0x7f65000cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 -- 172.21.15.104:0/2606905235 >> v1:172.21.15.114:6789/0 conn(0x7f650006c720 legacy=0x7f65000ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 -- 172.21.15.104:0/2606905235 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500161230 con 0x7f65000ce390 2023-06-10T18:55:13.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64feffd700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65000ce390 0x7f6500160cf0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098f0 tx=0x7f64f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d39dc93aed217e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.038+0000 7f64fdffb700 1 -- 172.21.15.104:0/2606905235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f400b8a0 con 0x7f65000ce390 2023-06-10T18:55:13.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f64fdffb700 1 -- 172.21.15.104:0/2606905235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f400ba00 con 0x7f65000ce390 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f64fdffb700 1 -- 172.21.15.104:0/2606905235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f400bd90 con 0x7f65000ce390 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 -- 172.21.15.104:0/2606905235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65000ce390 msgr2=0x7f6500160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65000ce390 0x7f6500160cf0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098f0 tx=0x7f64f4009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 -- 172.21.15.104:0/2606905235 shutdown_connections 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65000ce390 0x7f6500160cf0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 --2- 172.21.15.104:0/2606905235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000cae10 0x7f65000cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 -- 172.21.15.104:0/2606905235 >> 172.21.15.104:0/2606905235 conn(0x7f6500069e70 msgr2=0x7f650006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 -- 172.21.15.104:0/2606905235 shutdown_connections 2023-06-10T18:55:13.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.039+0000 7f65058f7700 1 -- 172.21.15.104:0/2606905235 wait complete. 2023-06-10T18:55:13.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.040+0000 7f65058f7700 1 Processor -- start 2023-06-10T18:55:13.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.040+0000 7f65058f7700 1 -- start start 2023-06-10T18:55:13.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65000cae10 0x7f650006fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000ce390 0x7f65000703b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65001e61e0 con 0x7f650006c720 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65001e6450 con 0x7f65000cae10 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f65058f7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65001e6730 con 0x7f65000ce390 2023-06-10T18:55:13.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36394/0 (socket says 172.21.15.104:36394) 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f64feffd700 1 -- 172.21.15.104:0/3538988812 learned_addr learned my addr 172.21.15.104:0/3538988812 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000ce390 0x7f65000703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.041+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65000cae10 0x7f650006fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 -- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000ce390 msgr2=0x7f65000703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000ce390 0x7f65000703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 -- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65000cae10 msgr2=0x7f650006fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65000cae10 0x7f650006fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 -- 172.21.15.104:0/3538988812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4009550 con 0x7f650006c720 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64feffd700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f64f4000c00 tx=0x7f64f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64e7fff700 1 -- 172.21.15.104:0/3538988812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4004480 con 0x7f650006c720 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64e7fff700 1 -- 172.21.15.104:0/3538988812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f401f070 con 0x7f650006c720 2023-06-10T18:55:13.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65001e6a10 con 0x7f650006c720 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f64e7fff700 1 -- 172.21.15.104:0/3538988812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f40038c0 con 0x7f650006c720 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.042+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65000b3780 con 0x7f650006c720 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.043+0000 7f64e7fff700 1 -- 172.21.15.104:0/3538988812 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f64f4021030 con 0x7f650006c720 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.044+0000 7f64e7fff700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 0x7f64ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.044+0000 7f64e7fff700 1 -- 172.21.15.104:0/3538988812 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f64f408ab50 con 0x7f650006c720 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.044+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 0x7f64ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.044+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 0x7f64ec06d190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65000ce840 tx=0x7f64f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.046+0000 7f6511a7cb80 1 Processor -- start 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.046+0000 7f6511a7cb80 1 -- start start 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c36ad80 0x56200c369820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c340740 0x56200c36d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c247c60 con 0x56200c342f50 2023-06-10T18:55:13.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c29a9f0 con 0x56200c340740 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x56200c293e80 con 0x56200c36ad80 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36402/0 (socket says 172.21.15.104:36402) 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 -- 172.21.15.104:0/945636221 learned_addr learned my addr 172.21.15.104:0/945636221 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c340740 0x56200c36d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c36ad80 0x56200c369820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 -- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c36ad80 msgr2=0x56200c369820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c36ad80 0x56200c369820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 -- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c340740 msgr2=0x56200c36d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c340740 0x56200c36d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 -- 172.21.15.104:0/945636221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56200c1155e0 con 0x56200c342f50 2023-06-10T18:55:13.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64feffd700 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b160 tx=0x7f64f4059470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7188b6fdc8efbd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64e57fa700 1 -- 172.21.15.104:0/945636221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f405d8f0 con 0x56200c342f50 2023-06-10T18:55:13.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64e57fa700 1 -- 172.21.15.104:0/945636221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f4061c80 con 0x56200c342f50 2023-06-10T18:55:13.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.047+0000 7f64e57fa700 1 -- 172.21.15.104:0/945636221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4057cd0 con 0x56200c342f50 2023-06-10T18:55:13.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 -- 172.21.15.104:0/945636221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 msgr2=0x56200c343320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b160 tx=0x7f64f4059470 comp rx=0 tx=0).stop 2023-06-10T18:55:13.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 -- 172.21.15.104:0/945636221 shutdown_connections 2023-06-10T18:55:13.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c340740 0x56200c36d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c343320 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/945636221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c36ad80 0x56200c369820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 -- 172.21.15.104:0/945636221 >> 172.21.15.104:0/945636221 conn(0x56200c337fc0 msgr2=0x56200c3881a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 -- 172.21.15.104:0/945636221 shutdown_connections 2023-06-10T18:55:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.048+0000 7f6511a7cb80 1 -- 172.21.15.104:0/945636221 wait complete. 2023-06-10T18:55:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 -- start start 2023-06-10T18:55:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 Processor -- start 2023-06-10T18:55:13.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c342f50 0x56200c36b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c233a70 con 0x56200c36ad80 2023-06-10T18:55:13.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c233770 con 0x56200c342f50 2023-06-10T18:55:13.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.049+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x56200c1178d0 con 0x56200c340740 2023-06-10T18:55:13.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f6511a7cb80 10 client.0 _ll_register_callbacks cb 0x56200c239210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:55:13.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36412/0 (socket says 172.21.15.104:36412) 2023-06-10T18:55:13.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 -- 172.21.15.104:0/4111071213 learned_addr learned my addr 172.21.15.104:0/4111071213 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64fe7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c342f50 0x56200c36b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64feffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 -- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 msgr2=0x56200c3636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 -- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c342f50 msgr2=0x56200c36b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c342f50 0x56200c36b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 -- 172.21.15.104:0/4111071213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65000b3780 con 0x56200c36ad80 2023-06-10T18:55:13.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f64f8007e10 tx=0x7f64f8012520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e1f07ff3641bbe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64de7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:55:13.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8012c20 con 0x56200c36ad80 2023-06-10T18:55:13.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f64f8012d80 con 0x56200c36ad80 2023-06-10T18:55:13.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56200c335780 con 0x56200c36ad80 2023-06-10T18:55:13.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64f8018470 con 0x56200c36ad80 2023-06-10T18:55:13.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f6511a7cb80 10 client.8916 fetch_fsmap learned FSMap version 70 2023-06-10T18:55:13.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x56200c0c1d90 con 0x56200c36ad80 2023-06-10T18:55:13.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.050+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56200c335780 con 0x56200c36ad80 2023-06-10T18:55:13.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f64f8014040 con 0x56200c36ad80 2023-06-10T18:55:13.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 10 client.8916 fetch_fsmap finished waiting for FSMap version 70 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 10 client.8916 resolve_mds: resolved name 'c' to daemon mds.c 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 20 client.8916 populate_metadata read hostname 'smithi104' 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c386780 0x56200c39af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 4 client.8916 mds_command: new command op to 8781 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 --> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56200c28ce60 con 0x56200c386780 2023-06-10T18:55:13.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f64feffd700 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c386780 0x56200c39af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f64de7fc700 10 client.8916 ms_handle_connect on v2:172.21.15.104:6833/2511953457 2023-06-10T18:55:13.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.051+0000 7f64feffd700 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c386780 0x56200c39af70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30492c6e66e41845 server_cookie=f6936af8b65e260a in_seq=0 out_seq=0 2023-06-10T18:55:13.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f64de7fc700 1 -- 172.21.15.104:0/4111071213 <== mds.0 v2:172.21.15.104:6833/2511953457 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56200c28ce60 con 0x56200c386780 2023-06-10T18:55:13.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f64de7fc700 10 client.8916 handle_command_reply: tid=0 2023-06-10T18:55:13.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 client.8916 shutdown 2023-06-10T18:55:13.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 msgr2=0x56200c387fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f64f8007e10 tx=0x7f64f8012520 comp rx=0 tx=0).stop 2023-06-10T18:55:13.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 shutdown_connections 2023-06-10T18:55:13.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c386780 0x56200c39af70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c36ad80 0x56200c387fb0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/4111071213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c342f50 0x56200c36b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f64de7fc700 0 client.8916 ms_handle_reset on v2:172.21.15.104:6833/2511953457 2023-06-10T18:55:13.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 >> 172.21.15.104:0/4111071213 conn(0x56200c337fc0 msgr2=0x56200c369df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 shutdown_connections 2023-06-10T18:55:13.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 1 -- 172.21.15.104:0/4111071213 wait complete. 2023-06-10T18:55:13.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.052+0000 7f6511a7cb80 20 client.8916 trim_cache size 0 max 16384 2023-06-10T18:55:13.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.069+0000 7f6511a7cb80 1 Processor -- start 2023-06-10T18:55:13.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.069+0000 7f6511a7cb80 1 -- start start 2023-06-10T18:55:13.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c380d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c3a1420 con 0x56200c342f50 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c386620 con 0x56200c36ad80 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x56200c3a17c0 con 0x56200c340740 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f64fe7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f64fe7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36416/0 (socket says 172.21.15.104:36416) 2023-06-10T18:55:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.070+0000 7f64fe7fc700 1 -- 172.21.15.104:0/2574349938 learned_addr learned my addr 172.21.15.104:0/2574349938 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c380d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64feffd700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 -- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 msgr2=0x56200c3843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 -- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 msgr2=0x56200c380d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c380d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 -- 172.21.15.104:0/2574349938 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56200c0aac10 con 0x56200c342f50 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c380d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64feffd700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:13.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007f70 tx=0x7f64f0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c60390370feaadd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64e5ffb700 1 -- 172.21.15.104:0/2574349938 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0023470 con 0x56200c342f50 2023-06-10T18:55:13.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64e5ffb700 1 -- 172.21.15.104:0/2574349938 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f00235d0 con 0x56200c342f50 2023-06-10T18:55:13.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f64e5ffb700 1 -- 172.21.15.104:0/2574349938 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0023780 con 0x56200c342f50 2023-06-10T18:55:13.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.071+0000 7f6511a7cb80 1 -- 172.21.15.104:0/2574349938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 msgr2=0x56200c384930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007f70 tx=0x7f64f0011000 comp rx=0 tx=0).stop 2023-06-10T18:55:13.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 -- 172.21.15.104:0/2574349938 shutdown_connections 2023-06-10T18:55:13.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c380d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c340740 0x56200c3843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/2574349938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c342f50 0x56200c384930 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 -- 172.21.15.104:0/2574349938 >> 172.21.15.104:0/2574349938 conn(0x56200c337fc0 msgr2=0x56200c362b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 -- 172.21.15.104:0/2574349938 shutdown_connections 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 -- 172.21.15.104:0/2574349938 wait complete. 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 -- start start 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.072+0000 7f6511a7cb80 1 Processor -- start 2023-06-10T18:55:13.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.073+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c342f50 0x56200c36c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c3e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c363fe0 con 0x56200c340740 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56200c3f2a80 con 0x56200c36ad80 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x56200c381bb0 con 0x56200c342f50 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 10 client.0 _ll_register_callbacks cb 0x7f64f80076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:55:13.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36422/0 (socket says 172.21.15.104:36422) 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 -- 172.21.15.104:0/168444005 learned_addr learned my addr 172.21.15.104:0/168444005 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c342f50 0x56200c36c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64ff7fe700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c3e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 -- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c342f50 msgr2=0x56200c36c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c342f50 0x56200c36c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 -- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 msgr2=0x56200c3e8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c3e8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 -- 172.21.15.104:0/168444005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f000f040 con 0x56200c340740 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64feffd700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f64f4059630 tx=0x7f64f405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503141c7e2e7b4ab server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64de7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f405d970 con 0x56200c340740 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f64f4057a70 con 0x56200c340740 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64f4061c80 con 0x56200c340740 2023-06-10T18:55:13.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64f4057cd0 con 0x56200c340740 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 10 client.8922 fetch_fsmap learned FSMap version 70 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x56200c0e8200 con 0x56200c340740 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.074+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f4061c80 con 0x56200c340740 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f64f405ad20 con 0x56200c340740 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 10 client.8922 fetch_fsmap finished waiting for FSMap version 70 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 10 client.8922 resolve_mds: resolved name 'c' to daemon mds.c 2023-06-10T18:55:13.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 20 client.8922 populate_metadata read hostname 'smithi104' 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c3e90f0 0x56200c33d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 4 client.8922 mds_command: new command op to 8781 tid=0[{"prefix": "status"}] 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 --> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56200c24f000 con 0x56200c3e90f0 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c3e90f0 0x56200c33d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64de7fc700 10 client.8922 ms_handle_connect on v2:172.21.15.104:6833/2511953457 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64fe7fc700 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c3e90f0 0x56200c33d020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20163a2b263175a4 server_cookie=351693b4661cc96e in_seq=0 out_seq=0 2023-06-10T18:55:13.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64de7fc700 1 -- 172.21.15.104:0/168444005 <== mds.0 v2:172.21.15.104:6833/2511953457 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56200c24f000 con 0x56200c3e90f0 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64de7fc700 10 client.8922 handle_command_reply: tid=0 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 client.8922 shutdown 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 msgr2=0x56200c3d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f64f4059630 tx=0x7f64f405c000 comp rx=0 tx=0).stop 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 shutdown_connections 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:6833/2511953457,v1:172.21.15.104:6835/2511953457] conn(0x56200c3e90f0 0x56200c33d020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56200c36ad80 0x56200c3e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56200c340740 0x56200c3d4b20 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f64de7fc700 0 client.8922 ms_handle_reset on v2:172.21.15.104:6833/2511953457 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 --2- 172.21.15.104:0/168444005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56200c342f50 0x56200c36c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 >> 172.21.15.104:0/168444005 conn(0x56200c337fc0 msgr2=0x56200c3843f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 shutdown_connections 2023-06-10T18:55:13.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.075+0000 7f6511a7cb80 1 -- 172.21.15.104:0/168444005 wait complete. 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f6511a7cb80 20 client.8922 trim_cache size 0 max 16384 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 8781, 2023-06-10T18:55:13.099 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8633587049999996, 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 70, 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 113, 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 113, 2023-06-10T18:55:13.100 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.215957564 2023-06-10T18:55:13.101 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 msgr2=0x7f64ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 0x7f64ec06d190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65000ce840 tx=0x7f64f0009040 comp rx=0 tx=0).stop 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 msgr2=0x7f6500151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f64f4000c00 tx=0x7f64f4004750 comp rx=0 tx=0).stop 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 shutdown_connections 2023-06-10T18:55:13.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65000ce390 0x7f65000703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f650006c720 0x7f6500151a40 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f64ec06ace0 0x7f64ec06d190 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 --2- 172.21.15.104:0/3538988812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65000cae10 0x7f650006fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 >> 172.21.15.104:0/3538988812 conn(0x7f6500069e70 msgr2=0x7f6500160270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 shutdown_connections 2023-06-10T18:55:13.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.076+0000 7f65058f7700 1 -- 172.21.15.104:0/3538988812 wait complete. 2023-06-10T18:55:13.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:55:13.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.218+0000 7fb7659d8700 1 Processor -- start 2023-06-10T18:55:13.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.218+0000 7fb7659d8700 1 -- start start 2023-06-10T18:55:13.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.219+0000 7fb7659d8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.219+0000 7fb7659d8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 0x7fb76015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.219+0000 7fb7659d8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7600b6070 con 0x7fb76015a420 2023-06-10T18:55:13.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.219+0000 7fb7659d8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7600ced10 con 0x7fb7600ce890 2023-06-10T18:55:13.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.219+0000 7fb7659d8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600b1d70 con 0x7fb7600b4ff0 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 0x7fb76015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55816/0 (socket says 172.21.15.104:55816) 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75effd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb76015a420 0x7fb7600b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40104/0 (socket says 172.21.15.104:40104) 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 -- 172.21.15.104:0/4042243011 learned_addr learned my addr 172.21.15.104:0/4042243011 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75f7fe700 1 --2- 172.21.15.104:0/4042243011 >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 0x7fb76015a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:52168/0 (socket says 172.21.15.104:52168) 2023-06-10T18:55:13.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 -- 172.21.15.104:0/4042243011 >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 msgr2=0x7fb76015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:13.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 --2- 172.21.15.104:0/4042243011 >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 0x7fb76015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 -- 172.21.15.104:0/4042243011 >> v1:172.21.15.114:6789/0 conn(0x7fb76015a420 legacy=0x7fb7600b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 -- 172.21.15.104:0/4042243011 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600b5460 con 0x7fb7600b4ff0 2023-06-10T18:55:13.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75e7fc700 1 --2- 172.21.15.104:0/4042243011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7500098f0 tx=0x7fb750009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aac16eec57b4b19 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75dffb700 1 -- 172.21.15.104:0/4042243011 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011830 con 0x7fb7600b4ff0 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75dffb700 1 -- 172.21.15.104:0/4042243011 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750011990 con 0x7fb7600b4ff0 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.220+0000 7fb75dffb700 1 -- 172.21.15.104:0/4042243011 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750011d20 con 0x7fb7600b4ff0 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 -- 172.21.15.104:0/4042243011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 msgr2=0x7fb7600ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 --2- 172.21.15.104:0/4042243011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7500098f0 tx=0x7fb750009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 -- 172.21.15.104:0/4042243011 shutdown_connections 2023-06-10T18:55:13.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 --2- 172.21.15.104:0/4042243011 >> v2:172.21.15.114:3300/0 conn(0x7fb7600ce890 0x7fb76015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 --2- 172.21.15.104:0/4042243011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7600b4ff0 0x7fb7600ce350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 -- 172.21.15.104:0/4042243011 >> 172.21.15.104:0/4042243011 conn(0x7fb760089fd0 msgr2=0x7fb7600d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 -- 172.21.15.104:0/4042243011 shutdown_connections 2023-06-10T18:55:13.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.221+0000 7fb7659d8700 1 -- 172.21.15.104:0/4042243011 wait complete. 2023-06-10T18:55:13.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.222+0000 7fb7659d8700 1 Processor -- start 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.222+0000 7fb7659d8700 1 -- start start 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 0x7fb7600c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 0x7fb7601e3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76015d970 con 0x7fb7600ce890 2023-06-10T18:55:13.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb760153c20 con 0x7fb7600b4ff0 2023-06-10T18:55:13.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb7659d8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb760153f00 con 0x7fb76015a420 2023-06-10T18:55:13.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36440/0 (socket says 172.21.15.104:36440) 2023-06-10T18:55:13.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 -- 172.21.15.104:0/2970391682 learned_addr learned my addr 172.21.15.104:0/2970391682 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:13.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75f7fe700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 0x7fb7601e3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75effd700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 0x7fb7600c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 -- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 msgr2=0x7fb7601e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 0x7fb7601e3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 -- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 msgr2=0x7fb7600c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 0x7fb7600c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75e7fc700 1 -- 172.21.15.104:0/2970391682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750009550 con 0x7fb7600ce890 2023-06-10T18:55:13.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75effd700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 0x7fb7600c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:13.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.223+0000 7fb75f7fe700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 0x7fb7601e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:13.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb75e7fc700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fb750009c40 tx=0x7fb750004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500045d0 con 0x7fb7600ce890 2023-06-10T18:55:13.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75001e070 con 0x7fb7600ce890 2023-06-10T18:55:13.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7500038e0 con 0x7fb7600ce890 2023-06-10T18:55:13.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7601541e0 con 0x7fb7600ce890 2023-06-10T18:55:13.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.224+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7601547a0 con 0x7fb7600ce890 2023-06-10T18:55:13.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.225+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb760154990 con 0x7fb7600ce890 2023-06-10T18:55:13.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.225+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb750003a40 con 0x7fb7600ce890 2023-06-10T18:55:13.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.225+0000 7fb747fff700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 0x7fb74c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:13.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.226+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb75008a4f0 con 0x7fb7600ce890 2023-06-10T18:55:13.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.226+0000 7fb75effd700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 0x7fb74c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:13.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.226+0000 7fb75effd700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 0x7fb74c06d190 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb760159d40 tx=0x7fb754009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:13.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.230+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb7500554f0 con 0x7fb7600ce890 2023-06-10T18:55:13.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.460+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb760074050 con 0x7fb7600ce890 2023-06-10T18:55:13.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.460+0000 7fb747fff700 1 -- 172.21.15.104:0/2970391682 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb750058b10 con 0x7fb7600ce890 2023-06-10T18:55:13.465 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:55:13.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 msgr2=0x7fb74c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 0x7fb74c06d190 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb760159d40 tx=0x7fb754009040 comp rx=0 tx=0).stop 2023-06-10T18:55:13.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 msgr2=0x7fb76015d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:13.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fb750009c40 tx=0x7fb750004750 comp rx=0 tx=0).stop 2023-06-10T18:55:13.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 shutdown_connections 2023-06-10T18:55:13.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb74c06ace0 0x7fb74c06d190 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7600b4ff0 0x7fb7600c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb76015a420 0x7fb7601e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 --2- 172.21.15.104:0/2970391682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7600ce890 0x7fb76015d430 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:13.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 >> 172.21.15.104:0/2970391682 conn(0x7fb760089fd0 msgr2=0x7fb760155eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:13.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 shutdown_connections 2023-06-10T18:55:13.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:13.462+0000 7fb7659d8700 1 -- 172.21.15.104:0/2970391682 wait complete. 2023-06-10T18:55:13.480 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:55:13.481 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:55:13.482 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:55:13.482 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:55:13.482 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:13.482 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:55:13.483 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:55:13.483 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:55:13.483 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:55:13.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.484 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:55:13.501 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:13.502 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:55:13.503 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:13.504 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:55:13.504 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41771sec preferred_lft 41771sec 2023-06-10T18:55:13.504 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:55:13.504 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.505 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:55:13.505 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:13.505 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:55:13.505 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.506 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:55:13.506 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.506 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state DOWN group default qlen 1000 2023-06-10T18:55:13.506 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:55:13.507 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:55:13.507 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:55:13.507 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:55:13.507 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:13.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.508 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:55:13.566 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:55:13.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.568 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:55:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:55:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:55:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:55:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:55:13.626 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:55:13.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.627 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:55:13.627 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:55:13.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:55:13.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:55:13.628 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:55:13.629 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:55:13.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.730 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:55:13.746 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:55:13.747 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:55:13.747 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:55:13.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:13.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:55:13.819 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:55:13.819 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:55:13.820 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:13.881 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:55:13.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:55:13.883 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:13.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:55:14.004 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:55:14.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:55:14.070 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:55:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:55:14.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:14.072 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:55:14.128 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:55:14.128 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:55:14.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:55:14.197 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:55:14.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:55:14.265 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:55:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:55:14.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:14.267 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:55:14.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:55:14.789+0000 7f1f893ca5c0 -1 init, newargv = 0x56038eec2c30 newargc=15 2023-06-10T18:55:14.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[144895]: starting ceph client 2023-06-10T18:55:14.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[144895]: starting fuse 2023-06-10T18:55:15.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:55:15.454 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:55:15.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:55:15.521 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:55:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:55:15.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:15.523 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:55:15.579 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:55:15.580 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:55:15.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:55:15.581 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:55:15.581 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:55:15.581 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:55:15.582 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:55:15.582 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:15.582 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:55:15.583 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:55:15.583 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:55:15.583 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:55:15.583 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:15.584 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:55:15.584 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:55:15.584 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:55:15.585 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:55:15.585 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:15.585 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:55:15.585 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:55:15.586 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:55:15.586 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:55:15.586 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:55:15.587 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:55:15.587 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:55:15.587 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:55:15.588 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:15.588 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:55:15.588 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:55:15.703 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.144895.asok 2023-06-10T18:55:15.885 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.144895.asok 2023-06-10T18:55:15.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:15.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.144895.asok status 2023-06-10T18:55:16.042 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:55:16.042 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:55:16.042 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:55:16.042 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:55:16.042 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "144895", 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:55:16.043 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 8928, 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 8928 2023-06-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 48469774 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:16.045 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 48469774 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.8928 192.168.0.1:0/48469774", 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/48469774", 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:55:16.046 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 70, 2023-06-10T18:55:16.047 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 113, 2023-06-10T18:55:16.047 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:55:16.047 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:55:16.047 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:55:16.047 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:55:16.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.053 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:55:16.053 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:16.071 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:55:16.072 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:16.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:55:16.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:55:16.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.293+0000 7f75269cd700 1 Processor -- start 2023-06-10T18:55:16.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 -- start start 2023-06-10T18:55:16.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75200cfa00 0x7f75200701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75200ccca0 con 0x7f75200cad90 2023-06-10T18:55:16.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75200d0310 con 0x7f75200cfa00 2023-06-10T18:55:16.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.294+0000 7f75269cd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75200cee50 con 0x7f752015a250 2023-06-10T18:55:16.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75200cfa00 0x7f75200701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751f7fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55842/0 (socket says 172.21.15.104:55842) 2023-06-10T18:55:16.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75200cfa00 0x7f75200701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:52192/0 (socket says 172.21.15.104:52192) 2023-06-10T18:55:16.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751f7fe700 1 -- 172.21.15.104:0/2580459012 learned_addr learned my addr 172.21.15.104:0/2580459012 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:16.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f7524f6a700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f75200cad90 0x7f75200b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40136/0 (socket says 172.21.15.104:40136) 2023-06-10T18:55:16.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 -- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 msgr2=0x7f75200cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 -- 172.21.15.104:0/2580459012 >> v1:172.21.15.114:6789/0 conn(0x7f75200cad90 legacy=0x7f75200b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 -- 172.21.15.104:0/2580459012 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200cf130 con 0x7f75200cfa00 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751f7fe700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751ffff700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75200701e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7510009840 tx=0x7f7510009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd43ea56768fd66d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751effd700 1 -- 172.21.15.104:0/2580459012 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510011830 con 0x7f75200cfa00 2023-06-10T18:55:16.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.295+0000 7f751effd700 1 -- 172.21.15.104:0/2580459012 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510011990 con 0x7f75200cfa00 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f751effd700 1 -- 172.21.15.104:0/2580459012 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510011d20 con 0x7f75200cfa00 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 -- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 msgr2=0x7f75200701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75200701e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7510009840 tx=0x7f7510009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 -- 172.21.15.104:0/2580459012 shutdown_connections 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75200701e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 --2- 172.21.15.104:0/2580459012 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 -- 172.21.15.104:0/2580459012 >> 172.21.15.104:0/2580459012 conn(0x7f752006e370 msgr2=0x7f752006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:16.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 -- 172.21.15.104:0/2580459012 shutdown_connections 2023-06-10T18:55:16.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.296+0000 7f75269cd700 1 -- 172.21.15.104:0/2580459012 wait complete. 2023-06-10T18:55:16.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.297+0000 7f75269cd700 1 Processor -- start 2023-06-10T18:55:16.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.297+0000 7f75269cd700 1 -- start start 2023-06-10T18:55:16.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75201f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75201ef8d0 con 0x7f75200cad90 2023-06-10T18:55:16.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c39e0 con 0x7f75200cfa00 2023-06-10T18:55:16.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f75269cd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75201efac0 con 0x7f752015a250 2023-06-10T18:55:16.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36464/0 (socket says 172.21.15.104:36464) 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 -- 172.21.15.104:0/2334458747 learned_addr learned my addr 172.21.15.104:0/2334458747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f7524f6a700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751f7fe700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75201f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 -- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 msgr2=0x7f75200b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 -- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 msgr2=0x7f75201f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75201f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751ffff700 1 -- 172.21.15.104:0/2334458747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7514009550 con 0x7f75200cad90 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.298+0000 7f751f7fe700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75201f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f751ffff700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7510009fd0 tx=0x7f7510004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100036a0 con 0x7f75200cad90 2023-06-10T18:55:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751001e070 con 0x7f75200cad90 2023-06-10T18:55:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75100094a0 con 0x7f75200cad90 2023-06-10T18:55:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510003a60 con 0x7f75200cad90 2023-06-10T18:55:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.299+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75201f0080 con 0x7f75200cad90 2023-06-10T18:55:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.300+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7520074050 con 0x7f75200cad90 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.300+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7510003bc0 con 0x7f75200cad90 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.301+0000 7f751cff9700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.301+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f751008a580 con 0x7f75200cad90 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.301+0000 7f751f7fe700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.301+0000 7f751f7fe700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7520156c50 tx=0x7f7514008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.305+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7510055580 con 0x7f75200cad90 2023-06-10T18:55:16.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.535+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f75200cc5a0 con 0x7f75200cad90 2023-06-10T18:55:16.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.536+0000 7f751cff9700 1 -- 172.21.15.104:0/2334458747 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7510058ba0 con 0x7f75200cad90 2023-06-10T18:55:16.541 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:55:16.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.537+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 msgr2=0x7f750c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.537+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7520156c50 tx=0x7f7514008040 comp rx=0 tx=0).stop 2023-06-10T18:55:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.537+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 msgr2=0x7f75200c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.537+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7510009fd0 tx=0x7f7510004750 comp rx=0 tx=0).stop 2023-06-10T18:55:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 shutdown_connections 2023-06-10T18:55:16.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752015a250 0x7f75200b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75200cad90 0x7f75200c34a0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 --2- 172.21.15.104:0/2334458747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75200cfa00 0x7f75201f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 >> 172.21.15.104:0/2334458747 conn(0x7f752006e370 msgr2=0x7f7520159000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 shutdown_connections 2023-06-10T18:55:16.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:16.538+0000 7f75269cd700 1 -- 172.21.15.104:0/2334458747 wait complete. 2023-06-10T18:55:16.555 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:55:16.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:55:16.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:55:16.557 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:55:16.557 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:16.558 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:55:16.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:55:16.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:55:16.559 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:55:16.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.559 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:55:16.581 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:55:16.581 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:55:16.581 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:55:16.581 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.581 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:55:16.582 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41764sec preferred_lft 41764sec 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:55:16.583 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:55:16.584 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:55:16.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.586 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:55:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:55:16.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.652 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:55:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:55:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:55:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:55:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:55:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:55:16.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.714 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:55:16.715 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:55:16.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:55:16.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:55:16.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:55:16.716 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:55:16.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.821 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:55:16.838 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:55:16.839 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:55:16.839 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:55:16.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:16.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:55:16.907 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:55:16.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:55:16.907 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:16.963 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:55:16.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:55:16.964 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:17.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:55:17.087 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:55:17.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:55:17.153 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:55:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:55:17.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:17.157 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:55:17.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:55:17.214 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:55:17.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:55:17.282 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:55:17.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:55:17.341 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:55:17.336+0000 7fc79a1ae5c0 -1 init, newargv = 0x5555cce13a50 newargc=15 2023-06-10T18:55:17.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114128]: starting ceph client 2023-06-10T18:55:17.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114128]: starting fuse 2023-06-10T18:55:17.350 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:55:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:55:17.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:17.352 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:55:17.409 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:55:17.410 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:55:17.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:55:17.410 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:55:17.411 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:55:17.411 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:55:17.411 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:55:17.412 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:55:17.412 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:55:17.412 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:55:17.413 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:55:17.413 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:55:17.413 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:55:17.414 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:55:17.414 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:55:17.414 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:55:17.415 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:55:17.415 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:55:17.415 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:55:17.415 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:55:17.416 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:55:17.416 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:55:17.416 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:55:17.417 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:55:17.417 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:55:17.417 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:55:17.418 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:55:17.418 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:55:17.418 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:55:17.531 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.114128.asok 2023-06-10T18:55:17.714 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.114128.asok 2023-06-10T18:55:17.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:17.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.114128.asok status 2023-06-10T18:55:17.869 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:55:17.869 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:55:17.869 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:55:17.869 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "114128", 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:55:17.870 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:55:17.871 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:55:17.871 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 6752, 2023-06-10T18:55:17.871 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:55:17.871 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:55:17.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:55:17.872 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6752 2023-06-10T18:55:17.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:55:17.872 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:55:17.873 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:55:17.873 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:55:17.873 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1677168428 2023-06-10T18:55:17.873 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:55:17.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:55:17.874 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:55:17.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:55:17.875 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:55:17.875 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1677168428 2023-06-10T18:55:17.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:55:17.876 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.6752 192.168.0.1:0/1677168428", 2023-06-10T18:55:17.876 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1677168428", 2023-06-10T18:55:17.876 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:55:17.877 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 70, 2023-06-10T18:55:17.877 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 113, 2023-06-10T18:55:17.877 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:55:17.878 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:55:17.878 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:55:17.878 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:55:17.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:17.881 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:55:17.881 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:17.898 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:55:17.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:17.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:17.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:55:17.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:55:18.112 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:55:18.113 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:55:18.113 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:55:18.247 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:55:18.247 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:55:18.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:55:18.249 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:55:18.250 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:55:18.251 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:55:18.252 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:55:18.252 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:18.253 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:55:18.253 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:55:18.254 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:55:18.255 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:55:18.256 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:18.256 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:55:18.257 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:55:18.257 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:55:18.258 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:55:18.259 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:18.259 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:55:18.260 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:55:18.261 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:55:18.262 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:55:18.262 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:55:18.263 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:55:18.264 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:55:18.265 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:55:18.265 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:55:18.266 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:55:18.267 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:55:18.350 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.144895.asok 2023-06-10T18:55:18.531 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.144895.asok 2023-06-10T18:55:18.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:18.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.144895.asok mds_sessions 2023-06-10T18:55:18.687 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:55:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 8928, 2023-06-10T18:55:18.687 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:55:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:55:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:55:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 8928 2023-06-10T18:55:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:55:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:55:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:55:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 48469774 2023-06-10T18:55:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.8928 v1:192.168.0.1:0/48469774", 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/48469774", 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:55:18.690 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6833", 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2511953457 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:18.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6835", 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2511953457 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:55:18.692 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:56:15.791587+0000", 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:55:15.791587+0000", 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 1, 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:55:18.693 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:55:18.694 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 70 2023-06-10T18:55:18.694 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:55:18.696 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 30 2023-06-10T18:55:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.813+0000 7faa556c6700 1 Processor -- start 2023-06-10T18:55:18.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 -- start start 2023-06-10T18:55:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa500ce390 0x7faa50160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 0x7faa500cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500d0340 con 0x7faa5006c720 2023-06-10T18:55:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa500d1660 con 0x7faa500ce390 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa556c6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5015cb60 con 0x7faa500cae10 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.814+0000 7faa4effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faa500ce390 0x7faa50160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4e7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faa5006c720 0x7faa500ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34302/0 (socket says 172.21.15.104:34302) 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4e7fc700 1 -- 172.21.15.104:0/1661605389 learned_addr learned my addr 172.21.15.104:0/1661605389 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4f7fe700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 0x7faa500cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4dffb700 1 -- 172.21.15.104:0/1661605389 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741827455 0 0) 0x7faa500d0340 con 0x7faa5006c720 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4dffb700 1 -- 172.21.15.104:0/1661605389 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa380035b0 con 0x7faa5006c720 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4effd700 1 -- 172.21.15.104:0/1661605389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 msgr2=0x7faa500cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4effd700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 0x7faa500cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4effd700 1 -- 172.21.15.104:0/1661605389 >> v1:172.21.15.104:6789/0 conn(0x7faa5006c720 legacy=0x7faa500ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4effd700 1 -- 172.21.15.104:0/1661605389 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa50161230 con 0x7faa500ce390 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4f7fe700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 0x7faa500cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4effd700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500ce390 0x7faa50160cf0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7faa44009840 tx=0x7faa44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b159d5babc593c2a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4dffb700 1 -- 172.21.15.104:0/1661605389 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44011830 con 0x7faa500ce390 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4dffb700 1 -- 172.21.15.104:0/1661605389 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44011990 con 0x7faa500ce390 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.815+0000 7faa4dffb700 1 -- 172.21.15.104:0/1661605389 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44011d20 con 0x7faa500ce390 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 -- 172.21.15.104:0/1661605389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500ce390 msgr2=0x7faa50160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500ce390 0x7faa50160cf0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7faa44009840 tx=0x7faa44009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 -- 172.21.15.104:0/1661605389 shutdown_connections 2023-06-10T18:55:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500ce390 0x7faa50160cf0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 --2- 172.21.15.104:0/1661605389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500cae10 0x7faa500cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 -- 172.21.15.104:0/1661605389 >> 172.21.15.104:0/1661605389 conn(0x7faa50069e70 msgr2=0x7faa5006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 -- 172.21.15.104:0/1661605389 shutdown_connections 2023-06-10T18:55:18.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.816+0000 7faa556c6700 1 -- 172.21.15.104:0/1661605389 wait complete. 2023-06-10T18:55:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.817+0000 7faa556c6700 1 Processor -- start 2023-06-10T18:55:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.817+0000 7faa556c6700 1 -- start start 2023-06-10T18:55:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5006c720 0x7faa5015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500ce390 0x7faa500c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa500b4cb0 con 0x7faa500cae10 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa500b40a0 con 0x7faa5006c720 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa500b4380 con 0x7faa500ce390 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36480/0 (socket says 172.21.15.104:36480) 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 -- 172.21.15.104:0/361638573 learned_addr learned my addr 172.21.15.104:0/361638573 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4f7fe700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500ce390 0x7faa500c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4effd700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5006c720 0x7faa5015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 -- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500ce390 msgr2=0x7faa500c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500ce390 0x7faa500c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 -- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5006c720 msgr2=0x7faa5015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5006c720 0x7faa5015f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 -- 172.21.15.104:0/361638573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa440094a0 con 0x7faa500cae10 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa4e7fc700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7faa40005910 tx=0x7faa4000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40005d70 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa400070d0 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40007440 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa500b4660 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.818+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa501f70a0 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.819+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa500b4e60 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.820+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faa400075a0 con 0x7faa500cae10 2023-06-10T18:55:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.820+0000 7faa377fe700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 0x7faa3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.820+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faa40088960 con 0x7faa500cae10 2023-06-10T18:55:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.820+0000 7faa4effd700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 0x7faa3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.821+0000 7faa4effd700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 0x7faa3c06d190 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:18.825+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faa500b4e60 con 0x7faa500cae10 2023-06-10T18:55:19.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.056+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "30"} v 0) v1 -- 0x7faa50074050 con 0x7faa500cae10 2023-06-10T18:55:19.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.649+0000 7faa377fe700 1 -- 172.21.15.104:0/361638573 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "30"}]=0 v71) v1 ==== 114+0+0 (secure 0 0 0) 0x7faa40053980 con 0x7faa500cae10 2023-06-10T18:55:19.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 msgr2=0x7faa3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 0x7faa3c06d190 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7faa44000c00 tx=0x7faa44008030 comp rx=0 tx=0).stop 2023-06-10T18:55:19.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 msgr2=0x7faa500c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7faa40005910 tx=0x7faa4000c860 comp rx=0 tx=0).stop 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 shutdown_connections 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faa3c06ace0 0x7faa3c06d190 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5006c720 0x7faa5015f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa500ce390 0x7faa500c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 --2- 172.21.15.104:0/361638573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa500cae10 0x7faa500c8da0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 >> 172.21.15.104:0/361638573 conn(0x7faa50069e70 msgr2=0x7faa5006fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 shutdown_connections 2023-06-10T18:55:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.652+0000 7faa556c6700 1 -- 172.21.15.104:0/361638573 wait complete. 2023-06-10T18:55:19.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_autoclose 50 2023-06-10T18:55:19.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.788+0000 7faec4360700 1 Processor -- start 2023-06-10T18:55:19.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.788+0000 7faec4360700 1 -- start start 2023-06-10T18:55:19.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec4360700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faebc06e6e0 0x7faebc0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec4360700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faebc06cde0 0x7faebc0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec4360700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc06e250 con 0x7faebc15c580 2023-06-10T18:55:19.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec4360700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc15c950 con 0x7faebc06e6e0 2023-06-10T18:55:19.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec4360700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc0b1e80 con 0x7faebc06cde0 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec18fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faebc06e6e0 0x7faebc0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec18fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faebc06e6e0 0x7faebc0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36488/0 (socket says 172.21.15.104:36488) 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec18fb700 1 -- 172.21.15.104:0/2530752491 learned_addr learned my addr 172.21.15.104:0/2530752491 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec20fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faebc15c580 0x7faebc0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34314/0 (socket says 172.21.15.104:34314) 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.789+0000 7faec28fd700 1 --2- 172.21.15.104:0/2530752491 >> v2:172.21.15.114:3300/0 conn(0x7faebc06cde0 0x7faebc0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec18fb700 1 -- 172.21.15.104:0/2530752491 >> v2:172.21.15.114:3300/0 conn(0x7faebc06cde0 msgr2=0x7faebc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec18fb700 1 --2- 172.21.15.104:0/2530752491 >> v2:172.21.15.114:3300/0 conn(0x7faebc06cde0 0x7faebc0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec18fb700 1 -- 172.21.15.104:0/2530752491 >> v1:172.21.15.104:6789/0 conn(0x7faebc15c580 legacy=0x7faebc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec18fb700 1 -- 172.21.15.104:0/2530752491 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc06d920 con 0x7faebc06e6e0 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec10fa700 1 -- 172.21.15.104:0/2530752491 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626067466 0 0) 0x7faebc06e250 con 0x7faebc15c580 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec18fb700 1 --2- 172.21.15.104:0/2530752491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06e6e0 0x7faebc0b1780 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7faebc0cbc60 tx=0x7faeb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84792670c9b30172 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec10fa700 1 -- 172.21.15.104:0/2530752491 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800b870 con 0x7faebc06e6e0 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec10fa700 1 -- 172.21.15.104:0/2530752491 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb800b9d0 con 0x7faebc06e6e0 2023-06-10T18:55:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec10fa700 1 -- 172.21.15.104:0/2530752491 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800bd60 con 0x7faebc06e6e0 2023-06-10T18:55:19.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec4360700 1 -- 172.21.15.104:0/2530752491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06e6e0 msgr2=0x7faebc0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.790+0000 7faec4360700 1 --2- 172.21.15.104:0/2530752491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06e6e0 0x7faebc0b1780 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7faebc0cbc60 tx=0x7faeb8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 -- 172.21.15.104:0/2530752491 shutdown_connections 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 --2- 172.21.15.104:0/2530752491 >> v2:172.21.15.114:3300/0 conn(0x7faebc06cde0 0x7faebc0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 --2- 172.21.15.104:0/2530752491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06e6e0 0x7faebc0b1780 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 -- 172.21.15.104:0/2530752491 >> 172.21.15.104:0/2530752491 conn(0x7faebc06b8a0 msgr2=0x7faebc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 -- 172.21.15.104:0/2530752491 shutdown_connections 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.791+0000 7faec4360700 1 -- 172.21.15.104:0/2530752491 wait complete. 2023-06-10T18:55:19.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 Processor -- start 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 -- start start 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 0x7faebc0c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 0x7faebc0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0ad410 con 0x7faebc06cde0 2023-06-10T18:55:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1f5c40 con 0x7faebc06e6e0 2023-06-10T18:55:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.792+0000 7faec4360700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1f4d50 con 0x7faebc15c580 2023-06-10T18:55:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36492/0 (socket says 172.21.15.104:36492) 2023-06-10T18:55:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 -- 172.21.15.104:0/2343166583 learned_addr learned my addr 172.21.15.104:0/2343166583 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec28fd700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 0x7faebc0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec18fb700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 0x7faebc0c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 -- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 msgr2=0x7faebc0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 0x7faebc0cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 -- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 msgr2=0x7faebc0c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 0x7faebc0c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 -- 172.21.15.104:0/2343166583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8009550 con 0x7faebc06cde0 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec28fd700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 0x7faebc0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec18fb700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 0x7faebc0c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec20fc700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7faeac00c4a0 tx=0x7faeac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0074e0 con 0x7faebc06cde0 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac007640 con 0x7faebc06cde0 2023-06-10T18:55:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc1f5030 con 0x7faebc06cde0 2023-06-10T18:55:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac005a50 con 0x7faebc06cde0 2023-06-10T18:55:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.793+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc1f55f0 con 0x7faebc06cde0 2023-06-10T18:55:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.794+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc074050 con 0x7faebc06cde0 2023-06-10T18:55:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.794+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faeac005bb0 con 0x7faebc06cde0 2023-06-10T18:55:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.795+0000 7faeb2ffd700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 0x7faea006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.795+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7faeac088a70 con 0x7faebc06cde0 2023-06-10T18:55:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.795+0000 7faec18fb700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 0x7faea006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.795+0000 7faec18fb700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 0x7faea006d190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faebc06eb20 tx=0x7faeb8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:19.799+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faeac053a70 con 0x7faebc06cde0 2023-06-10T18:55:20.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.026+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_autoclose", "val": "50"} v 0) v1 -- 0x7faebc072130 con 0x7faebc06cde0 2023-06-10T18:55:20.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.654+0000 7faeb2ffd700 1 -- 172.21.15.104:0/2343166583 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_autoclose", "val": "50"}]=0 v72) v1 ==== 116+0+0 (secure 0 0 0) 0x7faeac0170b0 con 0x7faebc06cde0 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 msgr2=0x7faea006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 0x7faea006d190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faebc06eb20 tx=0x7faeb8008030 comp rx=0 tx=0).stop 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 msgr2=0x7faebc1588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7faeac00c4a0 tx=0x7faeac00c860 comp rx=0 tx=0).stop 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 shutdown_connections 2023-06-10T18:55:20.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faebc15c580 0x7faebc0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faebc06cde0 0x7faebc1588f0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faea006ace0 0x7faea006d190 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 --2- 172.21.15.104:0/2343166583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faebc06e6e0 0x7faebc0c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 >> 172.21.15.104:0/2343166583 conn(0x7faebc06b8a0 msgr2=0x7faebc159600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 shutdown_connections 2023-06-10T18:55:20.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.657+0000 7faec4360700 1 -- 172.21.15.104:0/2343166583 wait complete. 2023-06-10T18:55:20.676 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.794+0000 7ff7fb1e2700 1 Processor -- start 2023-06-10T18:55:20.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 -- start start 2023-06-10T18:55:20.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7f406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff7f406e4b0 0x7ff7f40c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f406d290 con 0x7ff7f40ca100 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f4071640 con 0x7ff7f406e4b0 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7fb1e2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40aef00 con 0x7ff7f406e0e0 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7f977f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff7f40ca100 0x7ff7f40ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34330/0 (socket says 172.21.15.104:34330) 2023-06-10T18:55:20.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.795+0000 7ff7f977f700 1 -- 172.21.15.104:0/1204679808 learned_addr learned my addr 172.21.15.104:0/1204679808 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f3fff700 1 --2- 172.21.15.104:0/1204679808 >> v2:172.21.15.114:3300/0 conn(0x7ff7f406e4b0 0x7ff7f40c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987159070 0 0) 0x7ff7f406d290 con 0x7ff7f40ca100 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e40035b0 con 0x7ff7f40ca100 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f8f7e700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7f406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439636781 0 0) 0x7ff7e40035b0 con 0x7ff7f40ca100 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 msgr2=0x7ff7f406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7f406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 >> v2:172.21.15.114:3300/0 conn(0x7ff7f406e4b0 msgr2=0x7ff7f40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 --2- 172.21.15.104:0/1204679808 >> v2:172.21.15.114:3300/0 conn(0x7ff7f406e4b0 0x7ff7f40c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f406d4c0 con 0x7ff7f40ca100 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff7ec0019f0 con 0x7ff7f40ca100 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 >> v1:172.21.15.104:6789/0 conn(0x7ff7f40ca100 legacy=0x7ff7f40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7e4003a00 0x7ff7e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f4071640 con 0x7ff7e4004d90 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40aef00 con 0x7ff7e4003a00 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e400ba10 con 0x7ff7f406e0e0 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f8f7e700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7e4003a00 0x7ff7e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f3fff700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 -- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 msgr2=0x7ff7e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:20.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f406e0e0 0x7ff7e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 -- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7e4003a00 msgr2=0x7ff7e4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7e4003a00 0x7ff7e4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.796+0000 7ff7f977f700 1 -- 172.21.15.104:0/1204679808 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e40035b0 con 0x7ff7e4004d90 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7f977f700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005200 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00a8d0 tx=0x7ff7e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac46b9987d6d6d7c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00a340 con 0x7ff7e4004d90 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec012600 con 0x7ff7e4004d90 2023-06-10T18:55:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7f37fe700 1 -- 172.21.15.104:0/1204679808 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec0129b0 con 0x7ff7e4004d90 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/1204679808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e4004d90 msgr2=0x7ff7e4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/1204679808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005200 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00a8d0 tx=0x7ff7e4005880 comp rx=0 tx=0).stop 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7f977f700 1 -- 172.21.15.104:0/1204679808 reap_dead start 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/1204679808 shutdown_connections 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/1204679808 >> 172.21.15.104:0/1204679808 conn(0x7ff7f406b8a0 msgr2=0x7ff7f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/1204679808 shutdown_connections 2023-06-10T18:55:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.797+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/1204679808 wait complete. 2023-06-10T18:55:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.798+0000 7ff7fb1e2700 1 Processor -- start 2023-06-10T18:55:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 -- start start 2023-06-10T18:55:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f40b4db0 0x7ff7f41549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f40cac90 0x7ff7f40cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40cb670 con 0x7ff7f40b1a00 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40b1570 con 0x7ff7f40b4db0 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.799+0000 7ff7fb1e2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f40b0960 con 0x7ff7f40cac90 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36514/0 (socket says 172.21.15.104:36514) 2023-06-10T18:55:20.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 -- 172.21.15.104:0/282632334 learned_addr learned my addr 172.21.15.104:0/282632334 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f977f700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f40cac90 0x7ff7f40cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f8f7e700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f40b4db0 0x7ff7f41549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 -- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f40cac90 msgr2=0x7ff7f40cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f40cac90 0x7ff7f40cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 -- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f40b4db0 msgr2=0x7ff7f41549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f40b4db0 0x7ff7f41549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:20.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 -- 172.21.15.104:0/282632334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f40b0c40 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f3fff700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0011aa0 tx=0x7ff7e0011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000f940 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e000faa0 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f40b0f20 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000fe50 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.800+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f40b1e70 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.801+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f4074050 con 0x7ff7f40b1a00 2023-06-10T18:55:20.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.802+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff7e0014440 con 0x7ff7f40b1a00 2023-06-10T18:55:20.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.802+0000 7ff7f17fa700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 0x7ff7d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:20.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.802+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff7e0089c70 con 0x7ff7f40b1a00 2023-06-10T18:55:20.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.802+0000 7ff7f8f7e700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 0x7ff7d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:20.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.803+0000 7ff7f8f7e700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 0x7ff7d806d190 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff7f41f5120 tx=0x7ff7e80091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:20.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:20.806+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff7e0054be0 con 0x7ff7f40b1a00 2023-06-10T18:55:21.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.033+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7f41e6b50 con 0x7ff7f40b1a00 2023-06-10T18:55:21.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.034+0000 7ff7f17fa700 1 -- 172.21.15.104:0/282632334 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7ff7e0058200 con 0x7ff7f40b1a00 2023-06-10T18:55:21.041 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:21.041 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:55:21.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 msgr2=0x7ff7d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:21.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 0x7ff7d806d190 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff7f41f5120 tx=0x7ff7e80091c0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 msgr2=0x7ff7f40b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:21.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0011aa0 tx=0x7ff7e0011db0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 shutdown_connections 2023-06-10T18:55:21.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff7d806ace0 0x7ff7d806d190 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7f40b4db0 0x7ff7f41549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7f40cac90 0x7ff7f40cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 --2- 172.21.15.104:0/282632334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7f40b1a00 0x7ff7f40b0080 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:21.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 >> 172.21.15.104:0/282632334 conn(0x7ff7f406b8a0 msgr2=0x7ff7f40acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:21.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 shutdown_connections 2023-06-10T18:55:21.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:21.036+0000 7ff7fb1e2700 1 -- 172.21.15.104:0/282632334 wait complete. 2023-06-10T18:55:21.045 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:55:21.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2023-06-10T18:55:21.195 INFO:teuthology.orchestra.run.smithi104.stdout:[{"id":6752,"entity":{"name":{"type":"client","num":6752},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1677168428}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":3.839954283,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.99117164347257913,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6752 192.168.0.1:0/1677168428","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"114128","root":"/"}},{"id":8928,"entity":{"name":{"type":"client","num":8928},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":48469774}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":1,"uptime":6.3879239490000002,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.98535369866763967,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.8928 192.168.0.1:0/48469774","completed_requests":[{"tid":2,"created_ino":"0x0"}],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"0","hostname":"smithi104","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"144895","root":"/"}}] 2023-06-10T18:55:21.195 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "114128", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1677168428, "type": "any" }, "name": { "num": 6752, "type": "client" } }, "id": 6752, "inst": "client.6752 192.168.0.1:0/1677168428", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9911716434725791 }, "state": "open", "uptime": 3.839954283 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "0", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "144895", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 2 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 48469774, "type": "any" }, "name": { "num": 8928, "type": "client" } }, "id": 8928, "inst": "client.8928 192.168.0.1:0/48469774", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 1, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 1, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9853536986676397 }, "state": "open", "uptime": 6.387923949 } ] 2023-06-10T18:55:21.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:21.196 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2023-06-10T18:55:21.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo kill -sigstop 144895 2023-06-10T18:55:39.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:55:39.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:55:53.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.424+0000 7fde255a6700 1 Processor -- start 2023-06-10T18:55:53.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.424+0000 7fde255a6700 1 -- start start 2023-06-10T18:55:53.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde255a6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fde200b1070 0x7fde200b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde255a6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2006e5e0 con 0x7fde200aec00 2023-06-10T18:55:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde255a6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde200b1980 con 0x7fde2006ea70 2023-06-10T18:55:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde255a6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde200ae7e0 con 0x7fde200b1070 2023-06-10T18:55:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde1effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fde200b1070 0x7fde200b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:53.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde1f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fde200aec00 0x7fde200ac8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34350/0 (socket says 172.21.15.104:34350) 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.425+0000 7fde1f7fe700 1 -- 172.21.15.104:0/1649947565 learned_addr learned my addr 172.21.15.104:0/1649947565 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1effd700 1 -- 172.21.15.104:0/1649947565 >> v1:172.21.15.114:6789/0 conn(0x7fde2006ea70 legacy=0x7fde200ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1dffb700 1 -- 172.21.15.104:0/1649947565 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923076668 0 0) 0x7fde2006e5e0 con 0x7fde200aec00 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1effd700 1 -- 172.21.15.104:0/1649947565 >> v1:172.21.15.104:6789/0 conn(0x7fde200aec00 legacy=0x7fde200ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1effd700 1 -- 172.21.15.104:0/1649947565 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2006df90 con 0x7fde200b1070 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1effd700 1 --2- 172.21.15.104:0/1649947565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde200b1070 0x7fde200b1440 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fde14009820 tx=0x7fde14009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950e38125cd01149 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:53.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1dffb700 1 -- 172.21.15.104:0/1649947565 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14005e30 con 0x7fde200b1070 2023-06-10T18:55:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1dffb700 1 -- 172.21.15.104:0/1649947565 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14006030 con 0x7fde200b1070 2023-06-10T18:55:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde1dffb700 1 -- 172.21.15.104:0/1649947565 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde140063e0 con 0x7fde200b1070 2023-06-10T18:55:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 -- 172.21.15.104:0/1649947565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde200b1070 msgr2=0x7fde200b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 --2- 172.21.15.104:0/1649947565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde200b1070 0x7fde200b1440 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fde14009820 tx=0x7fde14009b30 comp rx=0 tx=0).stop 2023-06-10T18:55:53.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 -- 172.21.15.104:0/1649947565 shutdown_connections 2023-06-10T18:55:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 --2- 172.21.15.104:0/1649947565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde200b1070 0x7fde200b1440 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 -- 172.21.15.104:0/1649947565 >> 172.21.15.104:0/1649947565 conn(0x7fde2006b8a0 msgr2=0x7fde200cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.426+0000 7fde255a6700 1 -- 172.21.15.104:0/1649947565 shutdown_connections 2023-06-10T18:55:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.427+0000 7fde255a6700 1 -- 172.21.15.104:0/1649947565 wait complete. 2023-06-10T18:55:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.428+0000 7fde255a6700 1 Processor -- start 2023-06-10T18:55:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.428+0000 7fde255a6700 1 -- start start 2023-06-10T18:55:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde200aec00 0x7fde200ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde200b1070 0x7fde200add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde201f5e60 con 0x7fde2006ea70 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde200c3810 con 0x7fde200b1070 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2014e690 con 0x7fde200aec00 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:53.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36524/0 (socket says 172.21.15.104:36524) 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 -- 172.21.15.104:0/359699811 learned_addr learned my addr 172.21.15.104:0/359699811 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1f7fe700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde200b1070 0x7fde200add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1e7fc700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde200aec00 0x7fde200ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 -- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde200aec00 msgr2=0x7fde200ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde200aec00 0x7fde200ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 -- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde200b1070 msgr2=0x7fde200add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde200b1070 0x7fde200add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 -- 172.21.15.104:0/359699811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14009430 con 0x7fde2006ea70 2023-06-10T18:55:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde1effd700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fde14013040 tx=0x7fde14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14025430 con 0x7fde2006ea70 2023-06-10T18:55:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14020070 con 0x7fde2006ea70 2023-06-10T18:55:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde20154870 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.429+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde140257d0 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.430+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde200c31c0 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.431+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde20074050 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.431+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fde14022030 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.431+0000 7fde077fe700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 0x7fde0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.431+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fde1408ab40 con 0x7fde2006ea70 2023-06-10T18:55:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.431+0000 7fde1e7fc700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 0x7fde0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.432+0000 7fde1e7fc700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 0x7fde0c06d190 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fde20066040 tx=0x7fde10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.435+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde14055b00 con 0x7fde2006ea70 2023-06-10T18:55:53.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.665+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde20066f10 con 0x7fde2006ea70 2023-06-10T18:55:53.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.665+0000 7fde077fe700 1 -- 172.21.15.104:0/359699811 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7fde14059120 con 0x7fde2006ea70 2023-06-10T18:55:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:53.672 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:55:53.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 msgr2=0x7fde0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 0x7fde0c06d190 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fde20066040 tx=0x7fde10009040 comp rx=0 tx=0).stop 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 msgr2=0x7fde2014e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fde14013040 tx=0x7fde14004750 comp rx=0 tx=0).stop 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 shutdown_connections 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fde0c06ace0 0x7fde0c06d190 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde200b1070 0x7fde200add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde2006ea70 0x7fde2014e150 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 --2- 172.21.15.104:0/359699811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde200aec00 0x7fde200ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:53.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 >> 172.21.15.104:0/359699811 conn(0x7fde2006b8a0 msgr2=0x7fde2006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:53.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 shutdown_connections 2023-06-10T18:55:53.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:53.667+0000 7fde255a6700 1 -- 172.21.15.104:0/359699811 wait complete. 2023-06-10T18:55:53.675 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:55:53.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2023-06-10T18:55:53.826 INFO:teuthology.orchestra.run.smithi104.stdout:[{"id":6752,"entity":{"name":{"type":"client","num":6752},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1677168428}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":36.471565794,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":2.7740747038963023,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6752 192.168.0.1:0/1677168428","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"114128","root":"/"}},{"id":8928,"entity":{"name":{"type":"client","num":8928},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":48469774}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":39.01953546,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":3.6961291995304193,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":4,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.8928 192.168.0.1:0/48469774","completed_requests":[{"tid":4,"created_ino":"0x10000000000"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"0","hostname":"smithi104","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"144895","root":"/"}}] 2023-06-10T18:55:53.827 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "114128", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1677168428, "type": "any" }, "name": { "num": 6752, "type": "client" } }, "id": 6752, "inst": "client.6752 192.168.0.1:0/1677168428", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 2.7740747038963023 }, "state": "open", "uptime": 36.471565794 }, { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "0", "hostname": "smithi104", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "144895", "root": "/" }, "completed_requests": [ { "created_ino": "0x10000000000", "tid": 4 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 48469774, "type": "any" }, "name": { "num": 8928, "type": "client" } }, "id": 8928, "inst": "client.8928 192.168.0.1:0/48469774", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 4, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 1, "num_leases": 0, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 3.6961291995304193 }, "state": "open", "uptime": 39.01953546 } ] 2023-06-10T18:55:53.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:55:53.828 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec stat dir) 2023-06-10T18:55:57.749 INFO:teuthology.orchestra.run.smithi114.stdout: File: dir 2023-06-10T18:55:57.749 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2023-06-10T18:55:57.749 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 2fh/47d Inode: 1099511627776 Links: 2 2023-06-10T18:55:57.749 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-10T18:55:57.749 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fusefs_t:s0 2023-06-10T18:55:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:55:21.209497945 +0000 2023-06-10T18:55:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:55:21.209497945 +0000 2023-06-10T18:55:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:55:21.209497945 +0000 2023-06-10T18:55:57.750 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:55:57.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:55:57.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.874+0000 7f8094f56700 1 Processor -- start 2023-06-10T18:55:57.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.874+0000 7f8094f56700 1 -- start start 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f8094f56700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80900cfa00 0x7f80900701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f8094f56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cad90 0x7f80900b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f8094f56700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900ccca0 con 0x7f809015a250 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f8094f56700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80900cee50 con 0x7f80900cfa00 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f8094f56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900cf130 con 0x7f80900cad90 2023-06-10T18:55:57.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80900cfa00 0x7f80900701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f80900cfa00 0x7f80900701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36532/0 (socket says 172.21.15.104:36532) 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 -- 172.21.15.104:0/611455368 learned_addr learned my addr 172.21.15.104:0/611455368 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808dd9b700 1 --1- 172.21.15.104:0/611455368 >> v1:172.21.15.104:6789/0 conn(0x7f809015a250 0x7f80900cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34360/0 (socket says 172.21.15.104:34360) 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808ed9d700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cad90 0x7f80900b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 -- 172.21.15.104:0/611455368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cad90 msgr2=0x7f80900b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:57.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808d59a700 1 -- 172.21.15.104:0/611455368 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954536595 0 0) 0x7f80900ccca0 con 0x7f809015a250 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cad90 0x7f80900b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 -- 172.21.15.104:0/611455368 >> v1:172.21.15.104:6789/0 conn(0x7f809015a250 legacy=0x7f80900cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 -- 172.21.15.104:0/611455368 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900cc930 con 0x7f80900cfa00 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.875+0000 7f808e59c700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cfa00 0x7f80900701e0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f80800098f0 tx=0x7f8080009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83128b9560cbea0f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f808d59a700 1 -- 172.21.15.104:0/611455368 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080011830 con 0x7f80900cfa00 2023-06-10T18:55:57.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f808d59a700 1 -- 172.21.15.104:0/611455368 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080011990 con 0x7f80900cfa00 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f808d59a700 1 -- 172.21.15.104:0/611455368 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080011d20 con 0x7f80900cfa00 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 -- 172.21.15.104:0/611455368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cfa00 msgr2=0x7f80900701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cfa00 0x7f80900701e0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f80800098f0 tx=0x7f8080009c00 comp rx=0 tx=0).stop 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 -- 172.21.15.104:0/611455368 shutdown_connections 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cfa00 0x7f80900701e0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 --2- 172.21.15.104:0/611455368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cad90 0x7f80900b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:57.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 -- 172.21.15.104:0/611455368 >> 172.21.15.104:0/611455368 conn(0x7f809006e370 msgr2=0x7f809006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:57.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 -- 172.21.15.104:0/611455368 shutdown_connections 2023-06-10T18:55:57.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.876+0000 7f8094f56700 1 -- 172.21.15.104:0/611455368 wait complete. 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 Processor -- start 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 -- start start 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cfa00 0x7f80900c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 0x7f80901e3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80901f5560 con 0x7f80900cad90 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80901f5840 con 0x7f809015a250 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f8094f56700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80901f5b20 con 0x7f80900cfa00 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f808e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f808e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36538/0 (socket says 172.21.15.104:36538) 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.878+0000 7f808e59c700 1 -- 172.21.15.104:0/4042700084 learned_addr learned my addr 172.21.15.104:0/4042700084 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808ed9d700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 0x7f80901e3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 -- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cfa00 msgr2=0x7f80900c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808dd9b700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cfa00 0x7f80900c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cfa00 0x7f80900c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 -- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 msgr2=0x7f80901e3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 0x7f80901e3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 -- 172.21.15.104:0/4042700084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8080009550 con 0x7f80900cad90 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808ed9d700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 0x7f80901e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:57.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f808e59c700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f8080009c40 tx=0x7f8080004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800036a0 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80901f5e00 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808001f070 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80900ca4d0 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.879+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080003a00 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.880+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80900ca660 con 0x7f80900cad90 2023-06-10T18:55:57.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.880+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8080003b60 con 0x7f80900cad90 2023-06-10T18:55:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.881+0000 7f807effd700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.881+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8080021070 con 0x7f80900cad90 2023-06-10T18:55:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.881+0000 7f808dd9b700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.882+0000 7f808dd9b700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f80900af500 tx=0x7f8084009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:57.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:57.885+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8080055ed0 con 0x7f80900cad90 2023-06-10T18:55:58.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.114+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8090074050 con 0x7f80900cad90 2023-06-10T18:55:58.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.114+0000 7f807effd700 1 -- 172.21.15.104:0/4042700084 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f8080015090 con 0x7f80900cad90 2023-06-10T18:55:58.121 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:58.121 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:55:58.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 msgr2=0x7f807006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f80900af500 tx=0x7f8084009040 comp rx=0 tx=0).stop 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 msgr2=0x7f809006cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f8080009c40 tx=0x7f8080004750 comp rx=0 tx=0).stop 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 shutdown_connections 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f807006ace0 0x7f807006d190 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f809015a250 0x7f80901e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80900cad90 0x7f809006cbd0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 --2- 172.21.15.104:0/4042700084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80900cfa00 0x7f80900c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 >> 172.21.15.104:0/4042700084 conn(0x7f809006e370 msgr2=0x7f80900cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 shutdown_connections 2023-06-10T18:55:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.116+0000 7f8094f56700 1 -- 172.21.15.104:0/4042700084 wait complete. 2023-06-10T18:55:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:55:58.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2023-06-10T18:55:58.272 INFO:teuthology.orchestra.run.smithi104.stdout:[{"id":6752,"entity":{"name":{"type":"client","num":6752},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1677168428}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":40.917512833000004,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":3.744535204234412,"halflife":300},"cap_acquisition":{"value":0,"halflife":10},"last_trim_completed_requests_tid":3,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.6752 192.168.0.1:0/1677168428","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000000fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"219d315c3a0b2075e570fb1964be2ad33fe7a9c8","ceph_version":"ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)","entity_id":"1","hostname":"smithi114","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"114128","root":"/"}}] 2023-06-10T18:55:58.273 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 10, "value": 0 }, "client_metadata": { "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", "client_features": { "feature_bits": "0x00000000000fffff" }, "entity_id": "1", "hostname": "smithi114", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "114128", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1677168428, "type": "any" }, "name": { "num": 6752, "type": "client" } }, "id": 6752, "inst": "client.6752 192.168.0.1:0/1677168428", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 3, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 3.744535204234412 }, "state": "open", "uptime": 40.917512833 } ] 2023-06-10T18:55:58.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo kill -sigcont 144895 2023-06-10T18:55:58.301 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:55:58.302 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:55:58.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:55:58.295+0000 7f1f66ffd700 -1 client.8928 I was blocklisted at osd epoch 116 2023-06-10T18:55:58.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:55:58.434 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:55:58.435 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:55:58.435 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:55:58.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:55:58.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.605+0000 7f5678ca5700 1 Processor -- start 2023-06-10T18:55:58.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.605+0000 7f5678ca5700 1 -- start start 2023-06-10T18:55:58.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5678ca5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f567406e4b0 0x7f567406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5678ca5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5678ca5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567406d290 con 0x7f567406e0e0 2023-06-10T18:55:58.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5678ca5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5674071640 con 0x7f567406e4b0 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5678ca5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740aef00 con 0x7f5674069e70 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5672d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5671d9b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f567406e4b0 0x7f567406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5672d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:55942/0 (socket says 172.21.15.104:55942) 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.606+0000 7f5672d9d700 1 -- 172.21.15.104:0/2288632217 learned_addr learned my addr 172.21.15.104:0/2288632217 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:58.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5672d9d700 1 -- 172.21.15.104:0/2288632217 >> v2:172.21.15.114:3300/0 conn(0x7f567406e4b0 msgr2=0x7f567406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5672d9d700 1 --2- 172.21.15.104:0/2288632217 >> v2:172.21.15.114:3300/0 conn(0x7f567406e4b0 0x7f567406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5672d9d700 1 -- 172.21.15.104:0/2288632217 >> v1:172.21.15.114:6789/0 conn(0x7f567406e0e0 legacy=0x7f56740c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5672d9d700 1 -- 172.21.15.104:0/2288632217 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740ca2d0 con 0x7f5674069e70 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5672d9d700 1 --2- 172.21.15.104:0/2288632217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f566c009e50 tx=0x7f566c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad68acc6b0ae43f1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f567159a700 1 -- 172.21.15.104:0/2288632217 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00e070 con 0x7f5674069e70 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f567159a700 1 -- 172.21.15.104:0/2288632217 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c00b910 con 0x7f5674069e70 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f567159a700 1 -- 172.21.15.104:0/2288632217 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00bcc0 con 0x7f5674069e70 2023-06-10T18:55:58.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5678ca5700 1 -- 172.21.15.104:0/2288632217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 msgr2=0x7f56740ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.607+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2288632217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f566c009e50 tx=0x7f566c00b330 comp rx=0 tx=0).stop 2023-06-10T18:55:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 -- 172.21.15.104:0/2288632217 shutdown_connections 2023-06-10T18:55:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2288632217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5674069e70 0x7f56740ce9f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2288632217 >> v2:172.21.15.114:3300/0 conn(0x7f567406e4b0 0x7f567406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 -- 172.21.15.104:0/2288632217 >> 172.21.15.104:0/2288632217 conn(0x7f567406b8a0 msgr2=0x7f56740cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:58.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 -- 172.21.15.104:0/2288632217 shutdown_connections 2023-06-10T18:55:58.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.608+0000 7f5678ca5700 1 -- 172.21.15.104:0/2288632217 wait complete. 2023-06-10T18:55:58.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.609+0000 7f5678ca5700 1 Processor -- start 2023-06-10T18:55:58.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.609+0000 7f5678ca5700 1 -- start start 2023-06-10T18:55:58.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 0x7f56740b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 0x7f56740b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56740c7d30 con 0x7f5674069e70 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56740b5270 con 0x7f567406e0e0 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740c7f80 con 0x7f567406e4b0 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36560/0 (socket says 172.21.15.104:36560) 2023-06-10T18:55:58.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 -- 172.21.15.104:0/2434696046 learned_addr learned my addr 172.21.15.104:0/2434696046 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5671d9b700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 0x7f56740b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5672d9d700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 0x7f56740b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 -- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 msgr2=0x7f56740b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 0x7f56740b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 -- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 msgr2=0x7f56740b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 0x7f56740b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 -- 172.21.15.104:0/2434696046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c009b30 con 0x7f5674069e70 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5672d9d700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 0x7f56740b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5671d9b700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 0x7f56740b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f567259c700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f566800c4d0 tx=0x7f566800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680052d0 con 0x7f5674069e70 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668005430 con 0x7f5674069e70 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680038f0 con 0x7f5674069e70 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56740c8260 con 0x7f5674069e70 2023-06-10T18:55:58.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.610+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56740cb1c0 con 0x7f5674069e70 2023-06-10T18:55:58.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.611+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56740cb380 con 0x7f5674069e70 2023-06-10T18:55:58.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.612+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5668003a50 con 0x7f5674069e70 2023-06-10T18:55:58.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.612+0000 7f5662ffd700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 0x7f565406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.612+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5668088ab0 con 0x7f5674069e70 2023-06-10T18:55:58.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.612+0000 7f5671d9b700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 0x7f565406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.613+0000 7f5671d9b700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 0x7f565406d190 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f56740cbf70 tx=0x7f5664008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.617+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5668053a40 con 0x7f5674069e70 2023-06-10T18:55:58.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.803+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5674074050 con 0x7f5674069e70 2023-06-10T18:55:58.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.804+0000 7f5662ffd700 1 -- 172.21.15.104:0/2434696046 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+18716 (secure 0 0 0) 0x7f5668057060 con 0x7f5674069e70 2023-06-10T18:55:58.811 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:55:58.811 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":116,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:57.725868+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/48469774":"2023-06-10T19:55:56.863350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:55:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 msgr2=0x7f565406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 0x7f565406d190 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f56740cbf70 tx=0x7f5664008040 comp rx=0 tx=0).stop 2023-06-10T18:55:58.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 msgr2=0x7f56740cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f566800c4d0 tx=0x7f566800c890 comp rx=0 tx=0).stop 2023-06-10T18:55:58.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 shutdown_connections 2023-06-10T18:55:58.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f567406e4b0 0x7f56740b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5674069e70 0x7f56740cac80 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f565406ace0 0x7f565406d190 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 --2- 172.21.15.104:0/2434696046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567406e0e0 0x7f56740b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 >> 172.21.15.104:0/2434696046 conn(0x7f567406b8a0 msgr2=0x7f56740ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:58.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 shutdown_connections 2023-06-10T18:55:58.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.806+0000 7f5678ca5700 1 -- 172.21.15.104:0/2434696046 wait complete. 2023-06-10T18:55:58.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:55:05.032515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:55:06.191668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:55:58.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:55:58.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.942+0000 7fa50f2f5700 1 Processor -- start 2023-06-10T18:55:58.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.942+0000 7fa50f2f5700 1 -- start start 2023-06-10T18:55:58.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50f2f5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 0x7fa5080ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50f2f5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 0x7fa5080b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50f2f5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080ad3c0 con 0x7fa5080ce880 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50f2f5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080d0e40 con 0x7fa5080d0a00 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50f2f5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080cecf0 con 0x7fa508157b30 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50d892700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 0x7fa5080b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50d091700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 0x7fa5080ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50c890700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa5080ce880 0x7fa5080d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40242/0 (socket says 172.21.15.104:40242) 2023-06-10T18:55:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.943+0000 7fa50c890700 1 -- 172.21.15.104:0/3127410093 learned_addr learned my addr 172.21.15.104:0/3127410093 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:58.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d091700 1 -- 172.21.15.104:0/3127410093 >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 msgr2=0x7fa5080b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:58.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d091700 1 --2- 172.21.15.104:0/3127410093 >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 0x7fa5080b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d091700 1 -- 172.21.15.104:0/3127410093 >> v1:172.21.15.114:6789/0 conn(0x7fa5080ce880 legacy=0x7fa5080d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d091700 1 -- 172.21.15.104:0/3127410093 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508157f00 con 0x7fa508157b30 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d892700 1 --2- 172.21.15.104:0/3127410093 >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 0x7fa5080b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50d091700 1 --2- 172.21.15.104:0/3127410093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 0x7fa5080ce340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009840 tx=0x7fa4f8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=232e7d1475724cd0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa4fffff700 1 -- 172.21.15.104:0/3127410093 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8011830 con 0x7fa508157b30 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa4fffff700 1 -- 172.21.15.104:0/3127410093 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f8011990 con 0x7fa508157b30 2023-06-10T18:55:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa4fffff700 1 -- 172.21.15.104:0/3127410093 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8011d20 con 0x7fa508157b30 2023-06-10T18:55:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3127410093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 msgr2=0x7fa5080ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3127410093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 0x7fa5080ce340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009840 tx=0x7fa4f8009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.944+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3127410093 shutdown_connections 2023-06-10T18:55:58.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.945+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3127410093 >> v2:172.21.15.114:3300/0 conn(0x7fa5080d0a00 0x7fa5080b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.945+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3127410093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa508157b30 0x7fa5080ce340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.945+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3127410093 >> 172.21.15.104:0/3127410093 conn(0x7fa508089fd0 msgr2=0x7fa50808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.945+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3127410093 shutdown_connections 2023-06-10T18:55:58.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.945+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3127410093 wait complete. 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.946+0000 7fa50f2f5700 1 Processor -- start 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.946+0000 7fa50f2f5700 1 -- start start 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.946+0000 7fa50f2f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.946+0000 7fa50f2f5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5080d0a00 0x7fa5080aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50f2f5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa508157b30 0x7fa5080cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50f2f5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080c44b0 con 0x7fa5080ce880 2023-06-10T18:55:58.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50f2f5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080ccbc0 con 0x7fa508157b30 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50f2f5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080ccea0 con 0x7fa5080d0a00 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36574/0 (socket says 172.21.15.104:36574) 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 -- 172.21.15.104:0/3485761096 learned_addr learned my addr 172.21.15.104:0/3485761096 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50c890700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5080d0a00 0x7fa5080aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d892700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa508157b30 0x7fa5080cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 -- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5080d0a00 msgr2=0x7fa5080aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5080d0a00 0x7fa5080aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 -- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa508157b30 msgr2=0x7fa5080cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa508157b30 0x7fa5080cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 -- 172.21.15.104:0/3485761096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500009a70 con 0x7fa5080ce880 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50d091700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009fd0 tx=0x7fa4f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80036a0 con 0x7fa5080ce880 2023-06-10T18:55:58.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f801f070 con 0x7fa5080ce880 2023-06-10T18:55:58.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f80094a0 con 0x7fa5080ce880 2023-06-10T18:55:58.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.947+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8003a20 con 0x7fa5080ce880 2023-06-10T18:55:58.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.948+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5080cd460 con 0x7fa5080ce880 2023-06-10T18:55:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.948+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508074050 con 0x7fa5080ce880 2023-06-10T18:55:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.949+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa4f8003b80 con 0x7fa5080ce880 2023-06-10T18:55:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.949+0000 7fa4fdffb700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 0x7fa4ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:58.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.949+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fa4f8021070 con 0x7fa5080ce880 2023-06-10T18:55:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.950+0000 7fa50c890700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 0x7fa4ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:58.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.950+0000 7fa50c890700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 0x7fa4ec06d190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa5080cbd70 tx=0x7fa504009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:58.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:58.954+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa4f8055c30 con 0x7fa5080ce880 2023-06-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.138+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5080ce660 con 0x7fa5080ce880 2023-06-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.139+0000 7fa4fdffb700 1 -- 172.21.15.104:0/3485761096 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4f80150a0 con 0x7fa5080ce880 2023-06-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 msgr2=0x7fa4ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 0x7fa4ec06d190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa5080cbd70 tx=0x7fa504009040 comp rx=0 tx=0).stop 2023-06-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 msgr2=0x7fa5080aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009fd0 tx=0x7fa4f8004750 comp rx=0 tx=0).stop 2023-06-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 shutdown_connections 2023-06-10T18:55:59.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa4ec06ace0 0x7fa4ec06d190 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa508157b30 0x7fa5080cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5080ce880 0x7fa5080aea30 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 --2- 172.21.15.104:0/3485761096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5080d0a00 0x7fa5080aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 >> 172.21.15.104:0/3485761096 conn(0x7fa508089fd0 msgr2=0x7fa50815b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 shutdown_connections 2023-06-10T18:55:59.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.141+0000 7fa50f2f5700 1 -- 172.21.15.104:0/3485761096 wait complete. 2023-06-10T18:55:59.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:55:59.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.274+0000 7f6af3b87700 1 Processor -- start 2023-06-10T18:55:59.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af3b87700 1 -- start start 2023-06-10T18:55:59.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af3b87700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6aec0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af3b87700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0cc9f0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af3b87700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0cc100 con 0x7f6aec0cb210 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af3b87700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0cbbd0 con 0x7f6aec0cd8f0 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.275+0000 7f6af2124700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6aec0cf0b0 0x7f6aec06d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34414/0 (socket says 172.21.15.104:34414) 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af2124700 1 -- 172.21.15.104:0/2576113231 learned_addr learned my addr 172.21.15.104:0/2576113231 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:59.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6aec0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222494669 0 0) 0x7f6aec0cc9f0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad80035b0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373984274 0 0) 0x7f6aec0cc100 con 0x7f6aec0cb210 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec0cc9f0 con 0x7f6aec0cb210 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457015970 0 0) 0x7f6ad80035b0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 msgr2=0x7f6aec0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6aec0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 >> v1:172.21.15.114:6789/0 conn(0x7f6aec0cb210 legacy=0x7f6aec0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0cc3a0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6ae40019f0 con 0x7f6aec0cf0b0 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6aec0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 >> v1:172.21.15.104:6789/0 conn(0x7f6aec0cf0b0 legacy=0x7f6aec06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ad8003bc0 0x7f6ad8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad8004850 0x7f6ad8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6ad8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0cbbd0 con 0x7f6ad8004850 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad800dcd0 con 0x7f6ad8003bc0 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad800de60 con 0x7f6aec0cd8f0 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad8004850 0x7f6ad8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1923700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ad8003bc0 0x7f6ad8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af2124700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6ad8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 -- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 msgr2=0x7f6ad8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec0cd8f0 0x7f6ad8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 -- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ad8003bc0 msgr2=0x7f6ad8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ad8003bc0 0x7f6ad8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.276+0000 7f6af1122700 1 -- 172.21.15.104:0/2576113231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80094a0 con 0x7f6ad8004850 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af1122700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad8004850 0x7f6ad8004cc0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009d30 tx=0x7f6ad8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f7fe190eb6f3c1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8009e80 con 0x7f6ad8004850 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae800e9a0 con 0x7f6ad8004850 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af0921700 1 -- 172.21.15.104:0/2576113231 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800ed10 con 0x7f6ad8004850 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 -- 172.21.15.104:0/2576113231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad8004850 msgr2=0x7f6ad8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 --2- 172.21.15.104:0/2576113231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ad8004850 0x7f6ad8004cc0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009d30 tx=0x7f6ad8005880 comp rx=0 tx=0).stop 2023-06-10T18:55:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af2124700 1 -- 172.21.15.104:0/2576113231 reap_dead start 2023-06-10T18:55:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 -- 172.21.15.104:0/2576113231 shutdown_connections 2023-06-10T18:55:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 -- 172.21.15.104:0/2576113231 >> 172.21.15.104:0/2576113231 conn(0x7f6aec06b8a0 msgr2=0x7f6aec0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 -- 172.21.15.104:0/2576113231 shutdown_connections 2023-06-10T18:55:59.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.277+0000 7f6af3b87700 1 -- 172.21.15.104:0/2576113231 wait complete. 2023-06-10T18:55:59.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 Processor -- start 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 -- start start 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ad8004850 0x7f6aec14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec1f2ca0 0x7f6aec1e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec1f2a40 con 0x7f6aec1e0a10 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0c77d0 con 0x7f6aec1f2ca0 2023-06-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af3b87700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0c7ab0 con 0x7f6ad8004850 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af2124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af2124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36598/0 (socket says 172.21.15.104:36598) 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af2124700 1 -- 172.21.15.104:0/3026733069 learned_addr learned my addr 172.21.15.104:0/3026733069 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af1122700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec1f2ca0 0x7f6aec1e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.279+0000 7f6af1923700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ad8004850 0x7f6aec14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 -- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ad8004850 msgr2=0x7f6aec14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ad8004850 0x7f6aec14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 -- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec1f2ca0 msgr2=0x7f6aec1e04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec1f2ca0 0x7f6aec1e04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 -- 172.21.15.104:0/3026733069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0c7d90 con 0x7f6aec1e0a10 2023-06-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af2124700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0c9bf0 tx=0x7f6ae400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae400acc0 con 0x7f6aec1e0a10 2023-06-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae400ae40 con 0x7f6aec1e0a10 2023-06-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4013630 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec0c8070 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.280+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec1e0e80 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.281+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec074050 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.281+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6ae4013790 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.282+0000 7f6ae27fc700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 0x7f6ad006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.282+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f6ae4089db0 con 0x7f6aec1e0a10 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.282+0000 7f6af1923700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 0x7f6ad006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.282+0000 7f6af1923700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 0x7f6ad006d1c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0019a0 tx=0x7f6adc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.286+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6ae4054cb0 con 0x7f6aec1e0a10 2023-06-10T18:55:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.472+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6aec0cfca0 con 0x7f6aec1e0a10 2023-06-10T18:55:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.472+0000 7f6ae27fc700 1 -- 172.21.15.104:0/3026733069 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v116) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6ae40582d0 con 0x7f6aec1e0a10 2023-06-10T18:55:59.479 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:55:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 msgr2=0x7f6ad006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 0x7f6ad006d1c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0019a0 tx=0x7f6adc009040 comp rx=0 tx=0).stop 2023-06-10T18:55:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 msgr2=0x7f6aec1f1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0c9bf0 tx=0x7f6ae400a5b0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 shutdown_connections 2023-06-10T18:55:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec1e0a10 0x7f6aec1f1d10 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ad8004850 0x7f6aec14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ad10 0x7f6ad006d1c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 --2- 172.21.15.104:0/3026733069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec1f2ca0 0x7f6aec1e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 >> 172.21.15.104:0/3026733069 conn(0x7f6aec06b8a0 msgr2=0x7f6aec06cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 shutdown_connections 2023-06-10T18:55:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.474+0000 7f6af3b87700 1 -- 172.21.15.104:0/3026733069 wait complete. 2023-06-10T18:55:59.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:55:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.609+0000 7fcb37e35700 1 Processor -- start 2023-06-10T18:55:59.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.609+0000 7fcb37e35700 1 -- start start 2023-06-10T18:55:59.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb37e35700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb3006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb37e35700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb300ce390 0x7fcb30070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb37e35700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3006d7d0 con 0x7fcb30070f50 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb37e35700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb300713d0 con 0x7fcb300ce390 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb37e35700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300662a0 con 0x7fcb301586a0 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb300ce390 0x7fcb30070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb363d2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcb30070f50 0x7fcb30065b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34434/0 (socket says 172.21.15.104:34434) 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb363d2700 1 -- 172.21.15.104:0/3304453044 learned_addr learned my addr 172.21.15.104:0/3304453044 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcb300ce390 0x7fcb30070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36602/0 (socket says 172.21.15.104:36602) 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb35bd1700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb3006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 -- 172.21.15.104:0/3304453044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 msgr2=0x7fcb3006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb3006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 -- 172.21.15.104:0/3304453044 >> v1:172.21.15.104:6789/0 conn(0x7fcb30070f50 legacy=0x7fcb30065b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb353d0700 1 -- 172.21.15.104:0/3304453044 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300cd6c0 con 0x7fcb300ce390 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb34bcf700 1 -- 172.21.15.104:0/3304453044 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119710793 0 0) 0x7fcb3006d7d0 con 0x7fcb30070f50 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.610+0000 7fcb35bd1700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb3006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb353d0700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb30070a10 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009840 tx=0x7fcb2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87fadd3aade85b9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb34bcf700 1 -- 172.21.15.104:0/3304453044 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c011830 con 0x7fcb300ce390 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb34bcf700 1 -- 172.21.15.104:0/3304453044 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c011990 con 0x7fcb300ce390 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb34bcf700 1 -- 172.21.15.104:0/3304453044 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c011d20 con 0x7fcb300ce390 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 -- 172.21.15.104:0/3304453044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 msgr2=0x7fcb30070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb30070a10 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009840 tx=0x7fcb2c009b50 comp rx=0 tx=0).stop 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 -- 172.21.15.104:0/3304453044 shutdown_connections 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb3006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 --2- 172.21.15.104:0/3304453044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb30070a10 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 -- 172.21.15.104:0/3304453044 >> 172.21.15.104:0/3304453044 conn(0x7fcb300ceec0 msgr2=0x7fcb300cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 -- 172.21.15.104:0/3304453044 shutdown_connections 2023-06-10T18:55:59.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.611+0000 7fcb37e35700 1 -- 172.21.15.104:0/3304453044 wait complete. 2023-06-10T18:55:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 Processor -- start 2023-06-10T18:55:59.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 -- start start 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb30070f50 0x7fcb30149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb300cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300aeac0 con 0x7fcb300ce390 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300aeda0 con 0x7fcb30070f50 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb37e35700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300af080 con 0x7fcb301586a0 2023-06-10T18:55:59.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb353d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb353d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36610/0 (socket says 172.21.15.104:36610) 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.613+0000 7fcb353d0700 1 -- 172.21.15.104:0/1504829956 learned_addr learned my addr 172.21.15.104:0/1504829956 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb363d2700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb300cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb35bd1700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb30070f50 0x7fcb30149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 -- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 msgr2=0x7fcb300cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb300cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 -- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb30070f50 msgr2=0x7fcb30149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb30070f50 0x7fcb30149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 -- 172.21.15.104:0/1504829956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c0094a0 con 0x7fcb300ce390 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb353d0700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0090b0 tx=0x7fcb2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c0036a0 con 0x7fcb300ce390 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c01f070 con 0x7fcb300ce390 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c003a40 con 0x7fcb300ce390 2023-06-10T18:55:59.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb300af360 con 0x7fcb300ce390 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.614+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb300cb330 con 0x7fcb300ce390 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.615+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30074050 con 0x7fcb300ce390 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.615+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb2c003ba0 con 0x7fcb300ce390 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.616+0000 7fcb267fc700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 0x7fcb1406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.616+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fcb2c021070 con 0x7fcb300ce390 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.616+0000 7fcb35bd1700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 0x7fcb1406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.616+0000 7fcb35bd1700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 0x7fcb1406d190 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcb20006fd0 tx=0x7fcb20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.620+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb2c055f10 con 0x7fcb300ce390 2023-06-10T18:55:59.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.807+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb300b4e20 con 0x7fcb300ce390 2023-06-10T18:55:59.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.807+0000 7fcb267fc700 1 -- 172.21.15.104:0/1504829956 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb2c059530 con 0x7fcb300ce390 2023-06-10T18:55:59.813 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:55:59.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 msgr2=0x7fcb1406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 0x7fcb1406d190 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcb20006fd0 tx=0x7fcb20008040 comp rx=0 tx=0).stop 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 msgr2=0x7fcb300afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0090b0 tx=0x7fcb2c004750 comp rx=0 tx=0).stop 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 shutdown_connections 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb1406ace0 0x7fcb1406d190 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb30070f50 0x7fcb30149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb301586a0 0x7fcb300cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 --2- 172.21.15.104:0/1504829956 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb300ce390 0x7fcb300afb60 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 >> 172.21.15.104:0/1504829956 conn(0x7fcb300ceec0 msgr2=0x7fcb30155a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 shutdown_connections 2023-06-10T18:55:59.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.809+0000 7fcb37e35700 1 -- 172.21.15.104:0/1504829956 wait complete. 2023-06-10T18:55:59.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:55:59.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.945+0000 7fa8ad1c5700 1 Processor -- start 2023-06-10T18:55:59.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.945+0000 7fa8ad1c5700 1 -- start start 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8ad1c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa8a80ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8ad1c5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa8a80ce880 0x7fa8a80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8ad1c5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a80ad3c0 con 0x7fa8a80d0830 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8ad1c5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a80cecf0 con 0x7fa8a80ce880 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8ad1c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8157f00 con 0x7fa8a8157b30 2023-06-10T18:55:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8a759e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa8a80d0830 0x7fa8a815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34448/0 (socket says 172.21.15.104:34448) 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8a759e700 1 -- 172.21.15.104:0/2201576088 learned_addr learned my addr 172.21.15.104:0/2201576088 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8a659c700 1 --2- 172.21.15.104:0/2201576088 >> v2:172.21.15.114:3300/0 conn(0x7fa8a80ce880 0x7fa8a80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa8a80ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.946+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109557054 0 0) 0x7fa8a80ad3c0 con 0x7fa8a80d0830 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8940035b0 con 0x7fa8a80d0830 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562024387 0 0) 0x7fa8940035b0 con 0x7fa8a80d0830 2023-06-10T18:55:59.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 msgr2=0x7fa8a80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa8a80ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 >> v2:172.21.15.114:3300/0 conn(0x7fa8a80ce880 msgr2=0x7fa8a80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 --2- 172.21.15.104:0/2201576088 >> v2:172.21.15.114:3300/0 conn(0x7fa8a80ce880 0x7fa8a80d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80b0170 con 0x7fa8a80d0830 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa8a00019f0 con 0x7fa8a80d0830 2023-06-10T18:55:59.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 >> v1:172.21.15.104:6789/0 conn(0x7fa8a80d0830 legacy=0x7fa8a815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa894003a00 0x7fa894003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa894004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894004d90 0x7fa894005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80cecf0 con 0x7fa894003a00 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8157f00 con 0x7fa894004d90 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89400ba10 con 0x7fa8a8157b30 2023-06-10T18:55:59.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa894003a00 0x7fa894003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a759e700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894004d90 0x7fa894005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a659c700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa894004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 -- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 msgr2=0x7fa894004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8157b30 0x7fa894004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 -- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894004d90 msgr2=0x7fa894005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894004d90 0x7fa894005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 -- 172.21.15.104:0/2201576088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940035b0 con 0x7fa894003a00 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a759e700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894004d90 0x7fa894005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa894003a00 0x7fa894003e10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa890009b60 tx=0x7fa894005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc28efeff6ee00 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000e660 con 0x7fa894003a00 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89000e7c0 con 0x7fa894003a00 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.947+0000 7fa8a5d9b700 1 -- 172.21.15.104:0/2201576088 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000eb30 con 0x7fa894003a00 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2201576088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa894003a00 msgr2=0x7fa894003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2201576088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa894003a00 0x7fa894003e10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa890009b60 tx=0x7fa894005880 comp rx=0 tx=0).stop 2023-06-10T18:55:59.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8a759e700 1 -- 172.21.15.104:0/2201576088 reap_dead start 2023-06-10T18:55:59.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2201576088 shutdown_connections 2023-06-10T18:55:59.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2201576088 >> 172.21.15.104:0/2201576088 conn(0x7fa8a8089fd0 msgr2=0x7fa8a808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:55:59.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2201576088 shutdown_connections 2023-06-10T18:55:59.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.948+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2201576088 wait complete. 2023-06-10T18:55:59.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.950+0000 7fa8ad1c5700 1 Processor -- start 2023-06-10T18:55:59.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.950+0000 7fa8ad1c5700 1 -- start start 2023-06-10T18:55:59.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894003a00 0x7fa8a80b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 0x7fa8a8154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80b3090 con 0x7fa8a80b4130 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80b2ad0 con 0x7fa894003a00 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8ad1c5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80b2db0 con 0x7fa8a8155470 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36630/0 (socket says 172.21.15.104:36630) 2023-06-10T18:55:59.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 -- 172.21.15.104:0/2096032085 learned_addr learned my addr 172.21.15.104:0/2096032085 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894003a00 0x7fa8a80b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a759e700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 0x7fa8a8154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 -- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 msgr2=0x7fa8a8154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 0x7fa8a8154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 -- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894003a00 msgr2=0x7fa8a80b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894003a00 0x7fa8a80b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:55:59.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 -- 172.21.15.104:0/2096032085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80b36d0 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a759e700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 0x7fa8a8154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa8a659c700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa8980116f0 tx=0x7fa898011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.951+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898015870 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.952+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8980159d0 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.952+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a80c6bc0 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.952+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898015d40 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.952+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a80b1de0 con 0x7fa8a80b4130 2023-06-10T18:55:59.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.952+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a80b3860 con 0x7fa8a80b4130 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.953+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa898013400 con 0x7fa8a80b4130 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.953+0000 7fa89f7fe700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 0x7fa88806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.953+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fa898089d20 con 0x7fa8a80b4130 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.954+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 0x7fa88806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.954+0000 7fa8a6d9d700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 0x7fa88806d190 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa8900090b0 tx=0x7fa890008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:55:59.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:55:59.958+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa898054cb0 con 0x7fa8a80b4130 2023-06-10T18:56:00.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.142+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa8a8074050 con 0x7fa8a80b4130 2023-06-10T18:56:00.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.143+0000 7fa89f7fe700 1 -- 172.21.15.104:0/2096032085 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa8980582d0 con 0x7fa8a80b4130 2023-06-10T18:56:00.150 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 msgr2=0x7fa88806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 0x7fa88806d190 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa8900090b0 tx=0x7fa890008030 comp rx=0 tx=0).stop 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 msgr2=0x7fa8a8154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa8980116f0 tx=0x7fa898011ab0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 shutdown_connections 2023-06-10T18:56:00.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa88806ace0 0x7fa88806d190 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa894003a00 0x7fa8a80b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8a8155470 0x7fa8a8154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 --2- 172.21.15.104:0/2096032085 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80b4130 0x7fa8a8154920 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 >> 172.21.15.104:0/2096032085 conn(0x7fa8a8089fd0 msgr2=0x7fa8a80ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 shutdown_connections 2023-06-10T18:56:00.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.145+0000 7fa8ad1c5700 1 -- 172.21.15.104:0/2096032085 wait complete. 2023-06-10T18:56:00.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:56:00.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.278+0000 7f9150029700 1 Processor -- start 2023-06-10T18:56:00.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.278+0000 7f9150029700 1 -- start start 2023-06-10T18:56:00.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f9150029700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9148069e70 0x7f9148164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f9150029700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914815a680 con 0x7f914815a2b0 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f9150029700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91480ac680 con 0x7f91480ac230 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f9150029700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91480d0510 con 0x7f9148069e70 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914e5c6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9148069e70 0x7f9148164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914ddc5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f914815a2b0 0x7f91480cb150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34464/0 (socket says 172.21.15.104:34464) 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914ddc5700 1 -- 172.21.15.104:0/95506818 learned_addr learned my addr 172.21.15.104:0/95506818 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:00.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914e5c6700 1 -- 172.21.15.104:0/95506818 >> v1:172.21.15.114:6789/0 conn(0x7f91480ac230 legacy=0x7f91480cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914e5c6700 1 -- 172.21.15.104:0/95506818 >> v1:172.21.15.104:6789/0 conn(0x7f914815a2b0 legacy=0x7f91480cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914cdc3700 1 -- 172.21.15.104:0/95506818 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786338789 0 0) 0x7f914815a680 con 0x7f914815a2b0 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.279+0000 7f914e5c6700 1 -- 172.21.15.104:0/95506818 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914806a5e0 con 0x7f9148069e70 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f914e5c6700 1 --2- 172.21.15.104:0/95506818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9148069e70 0x7f9148164d50 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f9140009d90 tx=0x7f914000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5c74fa76996413 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f914cdc3700 1 -- 172.21.15.104:0/95506818 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000e070 con 0x7f9148069e70 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f914cdc3700 1 -- 172.21.15.104:0/95506818 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914000bbd0 con 0x7f9148069e70 2023-06-10T18:56:00.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f914cdc3700 1 -- 172.21.15.104:0/95506818 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140012430 con 0x7f9148069e70 2023-06-10T18:56:00.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 -- 172.21.15.104:0/95506818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9148069e70 msgr2=0x7f9148164d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 --2- 172.21.15.104:0/95506818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9148069e70 0x7f9148164d50 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f9140009d90 tx=0x7f914000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 -- 172.21.15.104:0/95506818 shutdown_connections 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 --2- 172.21.15.104:0/95506818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9148069e70 0x7f9148164d50 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 -- 172.21.15.104:0/95506818 >> 172.21.15.104:0/95506818 conn(0x7f914806b8a0 msgr2=0x7f91480cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 -- 172.21.15.104:0/95506818 shutdown_connections 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.280+0000 7f9150029700 1 -- 172.21.15.104:0/95506818 wait complete. 2023-06-10T18:56:00.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 Processor -- start 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 -- start start 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 0x7f9148152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91480c7690 con 0x7f914815a2b0 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9148153230 con 0x7f91480ac230 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f9150029700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91480b4120 con 0x7f9148069e70 2023-06-10T18:56:00.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f914e5c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f914e5c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36646/0 (socket says 172.21.15.104:36646) 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.282+0000 7f914e5c6700 1 -- 172.21.15.104:0/351850626 learned_addr learned my addr 172.21.15.104:0/351850626 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914d5c4700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 0x7f9148152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914ddc5700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 -- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 msgr2=0x7f91480afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 -- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 msgr2=0x7f9148152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 0x7f9148152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 -- 172.21.15.104:0/351850626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140009a70 con 0x7f914815a2b0 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914d5c4700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 0x7f9148152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f914e5c6700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f9140014040 tx=0x7f9140004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914001b810 con 0x7f914815a2b0 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9140020070 con 0x7f914815a2b0 2023-06-10T18:56:00.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91480b4400 con 0x7f914815a2b0 2023-06-10T18:56:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914001bbb0 con 0x7f914815a2b0 2023-06-10T18:56:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.283+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91480c7bc0 con 0x7f914815a2b0 2023-06-10T18:56:00.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.284+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914806d230 con 0x7f914815a2b0 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.284+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f914001bd10 con 0x7f914815a2b0 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.285+0000 7f913e7fc700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.285+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9140022070 con 0x7f914815a2b0 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.285+0000 7f914ddc5700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.286+0000 7f914ddc5700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9138000c00 tx=0x7f9138009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.289+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f914005a1e0 con 0x7f914815a2b0 2023-06-10T18:56:00.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.490+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f914806ce70 con 0x7f914815a2b0 2023-06-10T18:56:00.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.490+0000 7f913e7fc700 1 -- 172.21.15.104:0/351850626 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v116) v1 ==== 80+0+55 (secure 0 0 0) 0x7f9140059d70 con 0x7f914815a2b0 2023-06-10T18:56:00.497 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/48469774 2023-06-10T19:55:56.863350+0000 2023-06-10T18:56:00.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 msgr2=0x7f912c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9138000c00 tx=0x7f9138009040 comp rx=0 tx=0).stop 2023-06-10T18:56:00.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 msgr2=0x7f91480b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f9140014040 tx=0x7f9140004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 shutdown_connections 2023-06-10T18:56:00.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914815a2b0 0x7f91480b12f0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9148069e70 0x7f91480afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f912c06ace0 0x7f912c06d190 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 --2- 172.21.15.104:0/351850626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91480ac230 0x7f9148152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 >> 172.21.15.104:0/351850626 conn(0x7f914806b8a0 msgr2=0x7f91481553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 shutdown_connections 2023-06-10T18:56:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.492+0000 7f9150029700 1 -- 172.21.15.104:0/351850626 wait complete. 2023-06-10T18:56:00.503 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:56:00.510 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T18:56:00.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:00.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:00.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[144895]: fuse finished with error 0 and tester_r 0 2023-06-10T18:56:00.552 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:56:00.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:00.553 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:56:00.707 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:00.708 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:00.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:00.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.843+0000 7fe64c4b3700 1 Processor -- start 2023-06-10T18:56:00.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.843+0000 7fe64c4b3700 1 -- start start 2023-06-10T18:56:00.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64c4b3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6440aeaa0 0x7fe6440cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64c4b3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe6440cd680 0x7fe64406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64c4b3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6440c9e60 con 0x7fe6440b1080 2023-06-10T18:56:00.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64c4b3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6440c9bc0 con 0x7fe6440aeaa0 2023-06-10T18:56:00.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64c4b3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6440cdaa0 con 0x7fe6440cd680 2023-06-10T18:56:00.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe6440aeaa0 0x7fe6440cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe649a4e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe6440b1080 0x7fe6440ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34480/0 (socket says 172.21.15.104:34480) 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe649a4e700 1 -- 172.21.15.104:0/2392084648 learned_addr learned my addr 172.21.15.104:0/2392084648 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 -- 172.21.15.104:0/2392084648 >> v2:172.21.15.114:3300/0 conn(0x7fe6440cd680 msgr2=0x7fe64406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 --2- 172.21.15.104:0/2392084648 >> v2:172.21.15.114:3300/0 conn(0x7fe6440cd680 0x7fe64406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 -- 172.21.15.104:0/2392084648 >> v1:172.21.15.104:6789/0 conn(0x7fe6440b1080 legacy=0x7fe6440ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 -- 172.21.15.104:0/2392084648 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440aee70 con 0x7fe6440aeaa0 2023-06-10T18:56:00.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64a24f700 1 --2- 172.21.15.104:0/2392084648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440cd140 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fe644071030 tx=0x7fe640009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe454df4f72bfb9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.844+0000 7fe64924d700 1 -- 172.21.15.104:0/2392084648 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640005e30 con 0x7fe6440aeaa0 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64924d700 1 -- 172.21.15.104:0/2392084648 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe640006030 con 0x7fe6440aeaa0 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64924d700 1 -- 172.21.15.104:0/2392084648 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6400063c0 con 0x7fe6440aeaa0 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 -- 172.21.15.104:0/2392084648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 msgr2=0x7fe6440cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/2392084648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440cd140 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fe644071030 tx=0x7fe640009b00 comp rx=0 tx=0).stop 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 -- 172.21.15.104:0/2392084648 shutdown_connections 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/2392084648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440cd140 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/2392084648 >> v2:172.21.15.114:3300/0 conn(0x7fe6440cd680 0x7fe64406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 -- 172.21.15.104:0/2392084648 >> 172.21.15.104:0/2392084648 conn(0x7fe64406b8a0 msgr2=0x7fe6440cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:00.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 -- 172.21.15.104:0/2392084648 shutdown_connections 2023-06-10T18:56:00.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.845+0000 7fe64c4b3700 1 -- 172.21.15.104:0/2392084648 wait complete. 2023-06-10T18:56:00.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.846+0000 7fe64c4b3700 1 Processor -- start 2023-06-10T18:56:00.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.846+0000 7fe64c4b3700 1 -- start start 2023-06-10T18:56:00.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6440b1080 0x7fe64415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 0x7fe6441ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64415aa40 con 0x7fe6440aeaa0 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440c8c00 con 0x7fe6440cd680 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64c4b3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440c7ff0 con 0x7fe6440b1080 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36658/0 (socket says 172.21.15.104:36658) 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 -- 172.21.15.104:0/3513714285 learned_addr learned my addr 172.21.15.104:0/3513714285 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:00.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64aa50700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 0x7fe6441ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 -- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6440b1080 msgr2=0x7fe64415a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6440b1080 0x7fe64415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 -- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 msgr2=0x7fe6441ecf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 0x7fe6441ecf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64a24f700 1 -- 172.21.15.104:0/3513714285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640009430 con 0x7fe6440aeaa0 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.847+0000 7fe64aa50700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 0x7fe6441ecf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:00.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe64a24f700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fe640009b40 tx=0x7fe640004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640004240 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6400043c0 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6440c82d0 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6400036a0 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.848+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6440c8890 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.849+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe644074050 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.849+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe640022070 con 0x7fe6440aeaa0 2023-06-10T18:56:00.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.850+0000 7fe63affd700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 0x7fe62806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.850+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe64008af40 con 0x7fe6440aeaa0 2023-06-10T18:56:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.850+0000 7fe649a4e700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 0x7fe62806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.850+0000 7fe649a4e700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 0x7fe62806d190 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe6441f0b30 tx=0x7fe634007ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:00.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:00.854+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe640055ed0 con 0x7fe6440aeaa0 2023-06-10T18:56:01.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.044+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6441ed490 con 0x7fe6440aeaa0 2023-06-10T18:56:01.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.045+0000 7fe63affd700 1 -- 172.21.15.104:0/3513714285 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+18716 (secure 0 0 0) 0x7fe6400594f0 con 0x7fe6440aeaa0 2023-06-10T18:56:01.052 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:01.052 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":116,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:57.725868+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/48469774":"2023-06-10T19:55:56.863350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:01.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 msgr2=0x7fe62806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 0x7fe62806d190 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe6441f0b30 tx=0x7fe634007ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 msgr2=0x7fe6440c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fe640009b40 tx=0x7fe640004750 comp rx=0 tx=0).stop 2023-06-10T18:56:01.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 shutdown_connections 2023-06-10T18:56:01.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe62806ace0 0x7fe62806d190 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6440cd680 0x7fe6441ecf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6440aeaa0 0x7fe6440c78f0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 --2- 172.21.15.104:0/3513714285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6440b1080 0x7fe64415a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 >> 172.21.15.104:0/3513714285 conn(0x7fe64406b8a0 msgr2=0x7fe64406a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.047+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 shutdown_connections 2023-06-10T18:56:01.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.048+0000 7fe64c4b3700 1 -- 172.21.15.104:0/3513714285 wait complete. 2023-06-10T18:56:01.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:55:05.032515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:55:06.191668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:01.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:01.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.183+0000 7fb33fcb1700 1 Processor -- start 2023-06-10T18:56:01.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.183+0000 7fb33fcb1700 1 -- start start 2023-06-10T18:56:01.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33fcb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb3380b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33fcb1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb33806cde0 0x7fb3380ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33fcb1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33806e250 con 0x7fb33815c580 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33fcb1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33815c950 con 0x7fb33806cde0 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33fcb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380b1e80 con 0x7fb33806e6e0 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33da4d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb33815c580 0x7fb3380b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34492/0 (socket says 172.21.15.104:34492) 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33da4d700 1 -- 172.21.15.104:0/1964089422 learned_addr learned my addr 172.21.15.104:0/1964089422 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb3380b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33e24e700 1 --2- 172.21.15.104:0/1964089422 >> v2:172.21.15.114:3300/0 conn(0x7fb33806cde0 0x7fb3380ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112469340 0 0) 0x7fb33806e250 con 0x7fb33815c580 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3240035b0 con 0x7fb33815c580 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398558158 0 0) 0x7fb3240035b0 con 0x7fb33815c580 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 msgr2=0x7fb3380b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb3380b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 >> v2:172.21.15.114:3300/0 conn(0x7fb33806cde0 msgr2=0x7fb3380ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 --2- 172.21.15.104:0/1964089422 >> v2:172.21.15.114:3300/0 conn(0x7fb33806cde0 0x7fb3380ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33806d920 con 0x7fb33815c580 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb3340017f0 con 0x7fb33815c580 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 >> v1:172.21.15.104:6789/0 conn(0x7fb33815c580 legacy=0x7fb3380b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb324003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb324004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb324004d90 0x7fb324005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33815c950 con 0x7fb324003a00 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380b1e80 con 0x7fb324004d90 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.184+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32400ba10 con 0x7fb33806e6e0 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb324003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33e24e700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb324004d90 0x7fb324005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb324004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 -- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 msgr2=0x7fb324004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb324004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 -- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb324004d90 msgr2=0x7fb324005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb324004d90 0x7fb324005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 -- 172.21.15.104:0/1964089422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3240035b0 con 0x7fb324003a00 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33e24e700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb324004d90 0x7fb324005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb33806e6e0 0x7fb324004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb324003e10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb33400c320 tx=0x7fb324005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a88a25216c9b67 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400ce00 con 0x7fb324003a00 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334007bf0 con 0x7fb324003a00 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.185+0000 7fb33ca4b700 1 -- 172.21.15.104:0/1964089422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334007310 con 0x7fb324003a00 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1964089422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 msgr2=0x7fb324003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1964089422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb324003e10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb33400c320 tx=0x7fb324005880 comp rx=0 tx=0).stop 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33e24e700 1 -- 172.21.15.104:0/1964089422 reap_dead start 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1964089422 shutdown_connections 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1964089422 >> 172.21.15.104:0/1964089422 conn(0x7fb33806b8a0 msgr2=0x7fb3380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1964089422 shutdown_connections 2023-06-10T18:56:01.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.186+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1964089422 wait complete. 2023-06-10T18:56:01.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.187+0000 7fb33fcb1700 1 Processor -- start 2023-06-10T18:56:01.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.187+0000 7fb33fcb1700 1 -- start start 2023-06-10T18:56:01.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 0x7fb3380b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb338159aa0 0x7fb3380c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380c6d20 con 0x7fb324003a00 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380c6f70 con 0x7fb3380c7d30 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380c7250 con 0x7fb338159aa0 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36676/0 (socket says 172.21.15.104:36676) 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 -- 172.21.15.104:0/1998583357 learned_addr learned my addr 172.21.15.104:0/1998583357 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:01.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33e24e700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb338159aa0 0x7fb3380c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 0x7fb3380b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 -- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb338159aa0 msgr2=0x7fb3380c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb338159aa0 0x7fb3380c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 -- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 msgr2=0x7fb3380b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 0x7fb3380b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 -- 172.21.15.104:0/1998583357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380c7530 con 0x7fb324003a00 2023-06-10T18:56:01.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 0x7fb3380b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33da4d700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb334015040 tx=0x7fb334001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334007500 con 0x7fb324003a00 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334007680 con 0x7fb324003a00 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334005a70 con 0x7fb324003a00 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3380c7810 con 0x7fb324003a00 2023-06-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.188+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb338159f40 con 0x7fb324003a00 2023-06-10T18:56:01.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.189+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33806dbe0 con 0x7fb324003a00 2023-06-10T18:56:01.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.190+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb334005bd0 con 0x7fb324003a00 2023-06-10T18:56:01.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.190+0000 7fb32e7fc700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 0x7fb31c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.190+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fb33408a8f0 con 0x7fb324003a00 2023-06-10T18:56:01.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.190+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 0x7fb31c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.191+0000 7fb33d24c700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 0x7fb31c06d190 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb3380add90 tx=0x7fb328008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.195+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb33806dbe0 con 0x7fb324003a00 2023-06-10T18:56:01.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.378+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb33806dbe0 con 0x7fb324003a00 2023-06-10T18:56:01.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.378+0000 7fb32e7fc700 1 -- 172.21.15.104:0/1998583357 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb33806dbe0 con 0x7fb324003a00 2023-06-10T18:56:01.385 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:01.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 msgr2=0x7fb31c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 0x7fb31c06d190 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb3380add90 tx=0x7fb328008040 comp rx=0 tx=0).stop 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 msgr2=0x7fb3380b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb334015040 tx=0x7fb334001bf0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 shutdown_connections 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb338159aa0 0x7fb3380c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb324003a00 0x7fb3380b4f70 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb31c06ace0 0x7fb31c06d190 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 --2- 172.21.15.104:0/1998583357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3380c7d30 0x7fb3380b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 >> 172.21.15.104:0/1998583357 conn(0x7fb33806b8a0 msgr2=0x7fb3380cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 shutdown_connections 2023-06-10T18:56:01.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.380+0000 7fb33fcb1700 1 -- 172.21.15.104:0/1998583357 wait complete. 2023-06-10T18:56:01.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:01.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.516+0000 7fe4abad4700 1 Processor -- start 2023-06-10T18:56:01.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.516+0000 7fe4abad4700 1 -- start start 2023-06-10T18:56:01.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4abad4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4a415a330 0x7fe4a40ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4abad4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 0x7fe4a40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4abad4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a415a700 con 0x7fe4a40cb890 2023-06-10T18:56:01.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4abad4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a406d7b0 con 0x7fe4a415a330 2023-06-10T18:56:01.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4abad4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40cbe00 con 0x7fe4a40cb420 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4aa071700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe4a40cb890 0x7fe4a406dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34516/0 (socket says 172.21.15.104:34516) 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4aa071700 1 -- 172.21.15.104:0/566606304 learned_addr learned my addr 172.21.15.104:0/566606304 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a906f700 1 --2- 172.21.15.104:0/566606304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 0x7fe4a40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a9870700 1 --2- 172.21.15.104:0/566606304 >> v2:172.21.15.114:3300/0 conn(0x7fe4a415a330 0x7fe4a40ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732478295 0 0) 0x7fe4a415a700 con 0x7fe4a40cb890 2023-06-10T18:56:01.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4900035b0 con 0x7fe4a40cb890 2023-06-10T18:56:01.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236175010 0 0) 0x7fe4900035b0 con 0x7fe4a40cb890 2023-06-10T18:56:01.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a906f700 1 -- 172.21.15.104:0/566606304 >> v2:172.21.15.114:3300/0 conn(0x7fe4a415a330 msgr2=0x7fe4a40ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a906f700 1 --2- 172.21.15.104:0/566606304 >> v2:172.21.15.114:3300/0 conn(0x7fe4a415a330 0x7fe4a40ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.517+0000 7fe4a906f700 1 -- 172.21.15.104:0/566606304 >> v1:172.21.15.104:6789/0 conn(0x7fe4a40cb890 legacy=0x7fe4a406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4a906f700 1 -- 172.21.15.104:0/566606304 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a406ce80 con 0x7fe4a40cb420 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4a906f700 1 --2- 172.21.15.104:0/566606304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 0x7fe4a40b1070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe494009840 tx=0x7fe494009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=556ffec081bd8361 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c0019f0 con 0x7fe4a40cb420 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494004030 con 0x7fe4a40cb420 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4a886e700 1 -- 172.21.15.104:0/566606304 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940043a0 con 0x7fe4a40cb420 2023-06-10T18:56:01.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 -- 172.21.15.104:0/566606304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 msgr2=0x7fe4a40b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 --2- 172.21.15.104:0/566606304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 0x7fe4a40b1070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe494009840 tx=0x7fe494009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 -- 172.21.15.104:0/566606304 shutdown_connections 2023-06-10T18:56:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 --2- 172.21.15.104:0/566606304 >> v2:172.21.15.114:3300/0 conn(0x7fe4a415a330 0x7fe4a40ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 --2- 172.21.15.104:0/566606304 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a40cb420 0x7fe4a40b1070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 -- 172.21.15.104:0/566606304 >> 172.21.15.104:0/566606304 conn(0x7fe4a406b8a0 msgr2=0x7fe4a40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.518+0000 7fe4abad4700 1 -- 172.21.15.104:0/566606304 shutdown_connections 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.519+0000 7fe4abad4700 1 -- 172.21.15.104:0/566606304 wait complete. 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 Processor -- start 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 -- start start 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 0x7fe4a41e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a415a330 0x7fe4a41e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41e1890 con 0x7fe4a40cb420 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40c73d0 con 0x7fe4a40cb890 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4abad4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40c76b0 con 0x7fe4a415a330 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.520+0000 7fe4a9870700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36690/0 (socket says 172.21.15.104:36690) 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 -- 172.21.15.104:0/3475838745 learned_addr learned my addr 172.21.15.104:0/3475838745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a906f700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 0x7fe4a41e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4aa071700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a415a330 0x7fe4a41e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 -- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a415a330 msgr2=0x7fe4a41e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a415a330 0x7fe4a41e4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 -- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 msgr2=0x7fe4a41e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 0x7fe4a41e3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 -- 172.21.15.104:0/3475838745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4940094a0 con 0x7fe4a40cb420 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a906f700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 0x7fe4a41e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4a9870700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000e990 tx=0x7fe4a000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00096f0 con 0x7fe4a40cb420 2023-06-10T18:56:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a000c850 con 0x7fe4a40cb420 2023-06-10T18:56:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a40c7990 con 0x7fe4a40cb420 2023-06-10T18:56:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000eee0 con 0x7fe4a40cb420 2023-06-10T18:56:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.521+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a41ef9c0 con 0x7fe4a40cb420 2023-06-10T18:56:01.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.522+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40707e0 con 0x7fe4a40cb420 2023-06-10T18:56:01.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.523+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe4a000cbc0 con 0x7fe4a40cb420 2023-06-10T18:56:01.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.523+0000 7fe49a7fc700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 0x7fe48806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.523+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe4a0089a40 con 0x7fe4a40cb420 2023-06-10T18:56:01.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.523+0000 7fe4a906f700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 0x7fe48806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.524+0000 7fe4a906f700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 0x7fe48806d190 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe494000c00 tx=0x7fe494013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.527+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe4a008be60 con 0x7fe4a40cb420 2023-06-10T18:56:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.714+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe4a4074050 con 0x7fe4a40cb420 2023-06-10T18:56:01.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.714+0000 7fe49a7fc700 1 -- 172.21.15.104:0/3475838745 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v116) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe4a00170a0 con 0x7fe4a40cb420 2023-06-10T18:56:01.721 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:01.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 msgr2=0x7fe48806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 0x7fe48806d190 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe494000c00 tx=0x7fe494013040 comp rx=0 tx=0).stop 2023-06-10T18:56:01.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 msgr2=0x7fe4a41ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000e990 tx=0x7fe4a000eca0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 shutdown_connections 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4a415a330 0x7fe4a41e4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4a40cb420 0x7fe4a41ef480 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe48806ace0 0x7fe48806d190 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 --2- 172.21.15.104:0/3475838745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a40cb890 0x7fe4a41e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 >> 172.21.15.104:0/3475838745 conn(0x7fe4a406b8a0 msgr2=0x7fe4a40b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 shutdown_connections 2023-06-10T18:56:01.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.716+0000 7fe4abad4700 1 -- 172.21.15.104:0/3475838745 wait complete. 2023-06-10T18:56:01.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:56:01.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.852+0000 7f951436f700 1 Processor -- start 2023-06-10T18:56:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.852+0000 7f951436f700 1 -- start start 2023-06-10T18:56:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951436f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f950c0af940 0x7f950c15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951436f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c15e720 0x7f950c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951436f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c154970 con 0x7f950c1545a0 2023-06-10T18:56:01.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951436f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c15dab0 con 0x7f950c0af940 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951436f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0cb0b0 con 0x7f950c15e720 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.853+0000 7f951190a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f950c0af940 0x7f950c15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f950c0af940 0x7f950c15e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36700/0 (socket says 172.21.15.104:36700) 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 -- 172.21.15.104:0/3587379404 learned_addr learned my addr 172.21.15.104:0/3587379404 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951290c700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c15e720 0x7f950c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 -- 172.21.15.104:0/3587379404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c15e720 msgr2=0x7f950c0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:01.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c15e720 0x7f950c0cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 -- 172.21.15.104:0/3587379404 >> v1:172.21.15.114:6789/0 conn(0x7f950c1545a0 legacy=0x7f950c15a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 -- 172.21.15.104:0/3587379404 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c15ac80 con 0x7f950c0af940 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f951190a700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c0af940 0x7f950c15e1e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f95080098f0 tx=0x7f9508009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b022029a5149090 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f9511109700 1 -- 172.21.15.104:0/3587379404 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800b8a0 con 0x7f950c0af940 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f9511109700 1 -- 172.21.15.104:0/3587379404 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950800ba00 con 0x7f950c0af940 2023-06-10T18:56:01.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.854+0000 7f9511109700 1 -- 172.21.15.104:0/3587379404 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800bd90 con 0x7f950c0af940 2023-06-10T18:56:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 -- 172.21.15.104:0/3587379404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c0af940 msgr2=0x7f950c15e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c0af940 0x7f950c15e1e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f95080098f0 tx=0x7f9508009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 -- 172.21.15.104:0/3587379404 shutdown_connections 2023-06-10T18:56:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c15e720 0x7f950c0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 --2- 172.21.15.104:0/3587379404 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c0af940 0x7f950c15e1e0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 -- 172.21.15.104:0/3587379404 >> 172.21.15.104:0/3587379404 conn(0x7f950c06e370 msgr2=0x7f950c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 -- 172.21.15.104:0/3587379404 shutdown_connections 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.855+0000 7f951436f700 1 -- 172.21.15.104:0/3587379404 wait complete. 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.856+0000 7f951436f700 1 Processor -- start 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.856+0000 7f951436f700 1 -- start start 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c0af940 0x7f950c0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c15e720 0x7f950c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c140960 con 0x7f950c1545a0 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0b0a80 con 0x7f950c15e720 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951436f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c1595d0 con 0x7f950c0af940 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951190a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951190a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36706/0 (socket says 172.21.15.104:36706) 2023-06-10T18:56:01.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.857+0000 7f951190a700 1 -- 172.21.15.104:0/656187508 learned_addr learned my addr 172.21.15.104:0/656187508 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951290c700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c15e720 0x7f950c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951210b700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c0af940 0x7f950c0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 -- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c0af940 msgr2=0x7f950c0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c0af940 0x7f950c0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 -- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c15e720 msgr2=0x7f950c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c15e720 0x7f950c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 -- 172.21.15.104:0/656187508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508009550 con 0x7f950c1545a0 2023-06-10T18:56:01.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951190a700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9508009c40 tx=0x7f9508004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080036a0 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950801e070 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003a60 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c0b23e0 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.858+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c0b2960 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.859+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c074050 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.859+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9508003bc0 con 0x7f950c1545a0 2023-06-10T18:56:01.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.860+0000 7f94feffd700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 0x7f94f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:01.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.860+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9508020070 con 0x7f950c1545a0 2023-06-10T18:56:01.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.860+0000 7f951210b700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 0x7f94f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:01.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.861+0000 7f951210b700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 0x7f94f006d190 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9500000f60 tx=0x7f9500009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:01.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:01.864+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9508055a50 con 0x7f950c1545a0 2023-06-10T18:56:02.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.052+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f950c159170 con 0x7f950c1545a0 2023-06-10T18:56:02.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.052+0000 7f94feffd700 1 -- 172.21.15.104:0/656187508 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9508059070 con 0x7f950c1545a0 2023-06-10T18:56:02.058 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:56:02.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 msgr2=0x7f94f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 0x7f94f006d190 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9500000f60 tx=0x7f9500009040 comp rx=0 tx=0).stop 2023-06-10T18:56:02.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 msgr2=0x7f950c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9508009c40 tx=0x7f9508004750 comp rx=0 tx=0).stop 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 shutdown_connections 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94f006ace0 0x7f94f006d190 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f950c15e720 0x7f950c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950c0af940 0x7f950c0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 --2- 172.21.15.104:0/656187508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f950c1545a0 0x7f950c0b1ea0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 >> 172.21.15.104:0/656187508 conn(0x7f950c06e370 msgr2=0x7f950c15c970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:02.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 shutdown_connections 2023-06-10T18:56:02.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.054+0000 7f951436f700 1 -- 172.21.15.104:0/656187508 wait complete. 2023-06-10T18:56:02.072 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:56:02.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.188+0000 7f5a4cf0e700 1 Processor -- start 2023-06-10T18:56:02.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.188+0000 7f5a4cf0e700 1 -- start start 2023-06-10T18:56:02.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4cf0e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5a4806d350 0x7f5a4815a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4cf0e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a4806e3d0 0x7f5a48069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4cf0e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480cc050 con 0x7f5a4815a0a0 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4cf0e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a480ac9d0 con 0x7f5a4806e3d0 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4cf0e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4806d0e0 con 0x7f5a4806d350 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a46d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a4806e3d0 0x7f5a48069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5a4815a0a0 0x7f5a480ac2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34540/0 (socket says 172.21.15.104:34540) 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4659c700 1 -- 172.21.15.104:0/3516380320 learned_addr learned my addr 172.21.15.104:0/3516380320 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:02.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a45d9b700 1 --2- 172.21.15.104:0/3516380320 >> v2:172.21.15.114:3300/0 conn(0x7f5a4806d350 0x7f5a4815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a46d9d700 1 -- 172.21.15.104:0/3516380320 >> v2:172.21.15.114:3300/0 conn(0x7f5a4806d350 msgr2=0x7f5a4815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a46d9d700 1 --2- 172.21.15.104:0/3516380320 >> v2:172.21.15.114:3300/0 conn(0x7f5a4806d350 0x7f5a4815a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a46d9d700 1 -- 172.21.15.104:0/3516380320 >> v1:172.21.15.104:6789/0 conn(0x7f5a4815a0a0 legacy=0x7f5a480ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a46d9d700 1 -- 172.21.15.104:0/3516380320 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4806ce00 con 0x7f5a4806e3d0 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.189+0000 7f5a4559a700 1 -- 172.21.15.104:0/3516380320 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615611691 0 0) 0x7f5a480cc050 con 0x7f5a4815a0a0 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a46d9d700 1 --2- 172.21.15.104:0/3516380320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4806e3d0 0x7f5a48069e70 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5a48071070 tx=0x7f5a4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224a35cae2fad281 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4559a700 1 -- 172.21.15.104:0/3516380320 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000e070 con 0x7f5a4806e3d0 2023-06-10T18:56:02.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4559a700 1 -- 172.21.15.104:0/3516380320 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4000bb20 con 0x7f5a4806e3d0 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4559a700 1 -- 172.21.15.104:0/3516380320 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40012430 con 0x7f5a4806e3d0 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/3516380320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4806e3d0 msgr2=0x7f5a48069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/3516380320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4806e3d0 0x7f5a48069e70 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5a48071070 tx=0x7f5a4000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/3516380320 shutdown_connections 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/3516380320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4806e3d0 0x7f5a48069e70 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/3516380320 >> v2:172.21.15.114:3300/0 conn(0x7f5a4806d350 0x7f5a4815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/3516380320 >> 172.21.15.104:0/3516380320 conn(0x7f5a4806b8a0 msgr2=0x7f5a480cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:02.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/3516380320 shutdown_connections 2023-06-10T18:56:02.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.190+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/3516380320 wait complete. 2023-06-10T18:56:02.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 Processor -- start 2023-06-10T18:56:02.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 -- start start 2023-06-10T18:56:02.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 0x7f5a480b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 0x7f5a480c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480c6ff0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480b41b0 con 0x7f5a4806e3d0 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.192+0000 7f5a4cf0e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480c66e0 con 0x7f5a4806d350 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36722/0 (socket says 172.21.15.104:36722) 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 -- 172.21.15.104:0/686752849 learned_addr learned my addr 172.21.15.104:0/686752849 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:02.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a45d9b700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 0x7f5a480c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a4659c700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 0x7f5a480b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 -- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 msgr2=0x7f5a480b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 0x7f5a480b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 -- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 msgr2=0x7f5a480c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 0x7f5a480c2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 -- 172.21.15.104:0/686752849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40009a70 con 0x7f5a4815a0a0 2023-06-10T18:56:02.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a45d9b700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 0x7f5a480c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a4659c700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 0x7f5a480b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a46d9d700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000b680 tx=0x7f5a40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40012430 con 0x7f5a4815a0a0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a40020070 con 0x7f5a4815a0a0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a480b4430 con 0x7f5a4815a0a0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40004280 con 0x7f5a4815a0a0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.193+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a480c6c80 con 0x7f5a4815a0a0 2023-06-10T18:56:02.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.194+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a480c71a0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.195+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a400043e0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.195+0000 7f5a36ffd700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 0x7f5a2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.195+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5a40022070 con 0x7f5a4815a0a0 2023-06-10T18:56:02.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.195+0000 7f5a4659c700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 0x7f5a2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.196+0000 7f5a4659c700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 0x7f5a2806d190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0017b0 tx=0x7f5a3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.200+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a480c71a0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.383+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a480c71a0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.384+0000 7f5a36ffd700 1 -- 172.21.15.104:0/686752849 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a480c71a0 con 0x7f5a4815a0a0 2023-06-10T18:56:02.390 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 msgr2=0x7f5a2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 0x7f5a2806d190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0017b0 tx=0x7f5a3c009040 comp rx=0 tx=0).stop 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 msgr2=0x7f5a48154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000b680 tx=0x7f5a40004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 shutdown_connections 2023-06-10T18:56:02.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4815a0a0 0x7f5a48154a60 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a4806d350 0x7f5a480b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a2806ace0 0x7f5a2806d190 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 --2- 172.21.15.104:0/686752849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a4806e3d0 0x7f5a480c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 >> 172.21.15.104:0/686752849 conn(0x7f5a4806b8a0 msgr2=0x7f5a480ae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 shutdown_connections 2023-06-10T18:56:02.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.386+0000 7f5a4cf0e700 1 -- 172.21.15.104:0/686752849 wait complete. 2023-06-10T18:56:02.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:56:02.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.518+0000 7f7ae2364700 1 Processor -- start 2023-06-10T18:56:02.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.518+0000 7f7ae2364700 1 -- start start 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae2364700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7adc15c730 0x7f7adc06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae2364700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7adc0cb0c0 0x7f7adc0cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae2364700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7adc15cb00 con 0x7f7adc0cc000 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae2364700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc0aed80 con 0x7f7adc15c730 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae2364700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc0c9c60 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7adc15c730 0x7f7adc06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7adc0cc000 0x7f7adc06cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34554/0 (socket says 172.21.15.104:34554) 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adb7fe700 1 -- 172.21.15.104:0/3656721817 learned_addr learned my addr 172.21.15.104:0/3656721817 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7ae0901700 1 --2- 172.21.15.104:0/3656721817 >> v2:172.21.15.114:3300/0 conn(0x7f7adc0cb0c0 0x7f7adc0cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 -- 172.21.15.104:0/3656721817 >> v2:172.21.15.114:3300/0 conn(0x7f7adc0cb0c0 msgr2=0x7f7adc0cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 --2- 172.21.15.104:0/3656721817 >> v2:172.21.15.114:3300/0 conn(0x7f7adc0cb0c0 0x7f7adc0cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 -- 172.21.15.104:0/3656721817 >> v1:172.21.15.104:6789/0 conn(0x7f7adc0cc000 legacy=0x7f7adc06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 -- 172.21.15.104:0/3656721817 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc06ecd0 con 0x7f7adc15c730 2023-06-10T18:56:02.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adaffd700 1 -- 172.21.15.104:0/3656721817 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541736228 0 0) 0x7f7acc001110 con 0x7f7adc0cc000 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.519+0000 7f7adbfff700 1 --2- 172.21.15.104:0/3656721817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc15c730 0x7f7adc06e790 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0c9bf0 tx=0x7f7ad0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2952e7195e39878f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7adaffd700 1 -- 172.21.15.104:0/3656721817 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0005e30 con 0x7f7adc15c730 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7adaffd700 1 -- 172.21.15.104:0/3656721817 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0006030 con 0x7f7adc15c730 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7adaffd700 1 -- 172.21.15.104:0/3656721817 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00063c0 con 0x7f7adc15c730 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 -- 172.21.15.104:0/3656721817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc15c730 msgr2=0x7f7adc06e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 --2- 172.21.15.104:0/3656721817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc15c730 0x7f7adc06e790 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0c9bf0 tx=0x7f7ad0009b00 comp rx=0 tx=0).stop 2023-06-10T18:56:02.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 -- 172.21.15.104:0/3656721817 shutdown_connections 2023-06-10T18:56:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 --2- 172.21.15.104:0/3656721817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc15c730 0x7f7adc06e790 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 --2- 172.21.15.104:0/3656721817 >> v2:172.21.15.114:3300/0 conn(0x7f7adc0cb0c0 0x7f7adc0cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 -- 172.21.15.104:0/3656721817 >> 172.21.15.104:0/3656721817 conn(0x7f7adc06b8a0 msgr2=0x7f7adc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 -- 172.21.15.104:0/3656721817 shutdown_connections 2023-06-10T18:56:02.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.520+0000 7f7ae2364700 1 -- 172.21.15.104:0/3656721817 wait complete. 2023-06-10T18:56:02.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.522+0000 7f7ae2364700 1 Processor -- start 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.522+0000 7f7ae2364700 1 -- start start 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7adc0cc000 0x7f7adc1f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7adc15c730 0x7f7adc0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc1efff0 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0c2f70 con 0x7f7adc0cc000 2023-06-10T18:56:02.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae2364700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0c3250 con 0x7f7adc15c730 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36732/0 (socket says 172.21.15.104:36732) 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 -- 172.21.15.104:0/955794224 learned_addr learned my addr 172.21.15.104:0/955794224 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adb7fe700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7adc0cc000 0x7f7adc1f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7ae0901700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7adc15c730 0x7f7adc0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 -- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7adc15c730 msgr2=0x7f7adc0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7adc15c730 0x7f7adc0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 -- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7adc0cc000 msgr2=0x7f7adc1f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7adc0cc000 0x7f7adc1f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 -- 172.21.15.104:0/955794224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad0009430 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.523+0000 7f7adbfff700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000c00 tx=0x7f7ad0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.524+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00068f0 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.524+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0020070 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.524+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0004280 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.524+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc0c3530 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.524+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc0c3af0 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.525+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc0ced00 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.525+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7ad0022030 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.526+0000 7f7ad8ff9700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 0x7f7ac806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.526+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f7ad008ab80 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.526+0000 7f7adb7fe700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 0x7f7ac806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:02.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.526+0000 7f7adb7fe700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 0x7f7ac806d190 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0c7d60 tx=0x7f7acc007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:02.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.530+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7ad0055c30 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.729+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7adc066f10 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.730+0000 7f7ad8ff9700 1 -- 172.21.15.104:0/955794224 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v116) v1 ==== 80+0+55 (secure 0 0 0) 0x7f7ad0059250 con 0x7f7adc0cb0c0 2023-06-10T18:56:02.736 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/48469774 2023-06-10T19:55:56.863350+0000 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 msgr2=0x7f7ac806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 0x7f7ac806d190 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0c7d60 tx=0x7f7acc007c20 comp rx=0 tx=0).stop 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 msgr2=0x7f7adc1efab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000c00 tx=0x7f7ad0004750 comp rx=0 tx=0).stop 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 shutdown_connections 2023-06-10T18:56:02.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7adc15c730 0x7f7adc0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7adc0cb0c0 0x7f7adc1efab0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7ac806ace0 0x7f7ac806d190 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 --2- 172.21.15.104:0/955794224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7adc0cc000 0x7f7adc1f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 >> 172.21.15.104:0/955794224 conn(0x7f7adc06b8a0 msgr2=0x7f7adc15aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 shutdown_connections 2023-06-10T18:56:02.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:02.732+0000 7f7ae2364700 1 -- 172.21.15.104:0/955794224 wait complete. 2023-06-10T18:56:02.740 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:56:02.749 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:56:02.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:02.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:02.789 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114128]: fuse finished with error 0 and tester_r 0 2023-06-10T18:56:02.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:56:02.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:02.792 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:56:02.960 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:02.960 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:02.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:56:02.978 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:56:02.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:02.979 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:56:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:56:03.034 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:56:03.035 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:03.035 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:03.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:56:03.055 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:56:03.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:03.055 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:56:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:56:03.113 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:56:03.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.227+0000 7f27754af700 1 Processor -- start 2023-06-10T18:56:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.227+0000 7f27754af700 1 -- start start 2023-06-10T18:56:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f27754af700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 0x7f27700701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f27754af700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 0x7f27700b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f27754af700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27700ccca0 con 0x7f27700cfa00 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f27754af700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700cee50 con 0x7f27700cad90 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f27754af700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27700cf130 con 0x7f277015a250 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f27700cfa00 0x7f27700cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34568/0 (socket says 172.21.15.104:34568) 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276effd700 1 -- 172.21.15.104:0/559595798 learned_addr learned my addr 172.21.15.104:0/559595798 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276f7fe700 1 --2- 172.21.15.104:0/559595798 >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 0x7f27700b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276e7fc700 1 --2- 172.21.15.104:0/559595798 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 0x7f27700701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276dffb700 1 -- 172.21.15.104:0/559595798 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603429010 0 0) 0x7f27700ccca0 con 0x7f27700cfa00 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276dffb700 1 -- 172.21.15.104:0/559595798 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27580035b0 con 0x7f27700cfa00 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276e7fc700 1 -- 172.21.15.104:0/559595798 >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 msgr2=0x7f27700b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276e7fc700 1 --2- 172.21.15.104:0/559595798 >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 0x7f27700b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276e7fc700 1 -- 172.21.15.104:0/559595798 >> v1:172.21.15.104:6789/0 conn(0x7f27700cfa00 legacy=0x7f27700cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276e7fc700 1 -- 172.21.15.104:0/559595798 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700cc930 con 0x7f277015a250 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.228+0000 7f276f7fe700 1 --2- 172.21.15.104:0/559595798 >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 0x7f27700b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f276e7fc700 1 --2- 172.21.15.104:0/559595798 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 0x7f27700701e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2764009840 tx=0x7f2764009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a4a3a5e1564c5f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f276dffb700 1 -- 172.21.15.104:0/559595798 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011830 con 0x7f277015a250 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f276dffb700 1 -- 172.21.15.104:0/559595798 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764011990 con 0x7f277015a250 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f276dffb700 1 -- 172.21.15.104:0/559595798 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011d20 con 0x7f277015a250 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 -- 172.21.15.104:0/559595798 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 msgr2=0x7f27700701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 --2- 172.21.15.104:0/559595798 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 0x7f27700701e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2764009840 tx=0x7f2764009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 -- 172.21.15.104:0/559595798 shutdown_connections 2023-06-10T18:56:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 --2- 172.21.15.104:0/559595798 >> v2:172.21.15.114:3300/0 conn(0x7f27700cad90 0x7f27700b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 --2- 172.21.15.104:0/559595798 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f277015a250 0x7f27700701e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 -- 172.21.15.104:0/559595798 >> 172.21.15.104:0/559595798 conn(0x7f277006e370 msgr2=0x7f277006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 -- 172.21.15.104:0/559595798 shutdown_connections 2023-06-10T18:56:03.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.229+0000 7f27754af700 1 -- 172.21.15.104:0/559595798 wait complete. 2023-06-10T18:56:03.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.231+0000 7f27754af700 1 Processor -- start 2023-06-10T18:56:03.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.231+0000 7f27754af700 1 -- start start 2023-06-10T18:56:03.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27700cad90 0x7f27700ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f277015a250 0x7f277015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770151440 con 0x7f27700cfa00 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277015beb0 con 0x7f277015a250 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f27754af700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277015a6f0 con 0x7f27700cad90 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36748/0 (socket says 172.21.15.104:36748) 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 -- 172.21.15.104:0/910334519 learned_addr learned my addr 172.21.15.104:0/910334519 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276effd700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27700cad90 0x7f27700ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276f7fe700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f277015a250 0x7f277015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 -- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27700cad90 msgr2=0x7f27700ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27700cad90 0x7f27700ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 -- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f277015a250 msgr2=0x7f277015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f277015a250 0x7f277015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.232+0000 7f276e7fc700 1 -- 172.21.15.104:0/910334519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2768009a70 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f276e7fc700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f27640090b0 tx=0x7f2764004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764004550 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276401f070 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27640094a0 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27640038c0 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.233+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277015a990 con 0x7f27700cfa00 2023-06-10T18:56:03.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.234+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770074050 con 0x7f27700cfa00 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.234+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2764003a20 con 0x7f27700cfa00 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.235+0000 7f27577fe700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 0x7f275c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.235+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f2764021070 con 0x7f27700cfa00 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.235+0000 7f276effd700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 0x7f275c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.235+0000 7f276effd700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 0x7f275c06d190 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2760007fd0 tx=0x7f2760007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.239+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2764055cc0 con 0x7f27700cfa00 2023-06-10T18:56:03.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.469+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27700cc1c0 con 0x7f27700cfa00 2023-06-10T18:56:03.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.470+0000 7f27577fe700 1 -- 172.21.15.104:0/910334519 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f27640592e0 con 0x7f27700cfa00 2023-06-10T18:56:03.476 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:03.477 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:56:03.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 msgr2=0x7f275c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 0x7f275c06d190 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2760007fd0 tx=0x7f2760007dc0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 msgr2=0x7f2770150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f27640090b0 tx=0x7f2764004750 comp rx=0 tx=0).stop 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 shutdown_connections 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f275c06ace0 0x7f275c06d190 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f277015a250 0x7f277015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27700cad90 0x7f27700ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 --2- 172.21.15.104:0/910334519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27700cfa00 0x7f2770150f00 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 >> 172.21.15.104:0/910334519 conn(0x7f277006e370 msgr2=0x7f27700cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 shutdown_connections 2023-06-10T18:56:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.472+0000 7f27754af700 1 -- 172.21.15.104:0/910334519 wait complete. 2023-06-10T18:56:03.480 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:56:03.491 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.604+0000 7fb3c8575700 1 Processor -- start 2023-06-10T18:56:03.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.604+0000 7fb3c8575700 1 -- start start 2023-06-10T18:56:03.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.605+0000 7fb3c8575700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb3c006e0e0 0x7fb3c006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.605+0000 7fb3c8575700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3c006e4b0 0x7fb3c00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.605+0000 7fb3c8575700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c006d290 con 0x7fb3c00ca100 2023-06-10T18:56:03.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.605+0000 7fb3c8575700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c0071640 con 0x7fb3c006e0e0 2023-06-10T18:56:03.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.605+0000 7fb3c8575700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c00aef00 con 0x7fb3c006e4b0 2023-06-10T18:56:03.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb3c006e0e0 0x7fb3c006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6b12700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb3c00ca100 0x7fb3c00ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34582/0 (socket says 172.21.15.104:34582) 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6b12700 1 -- 172.21.15.104:0/489897838 learned_addr learned my addr 172.21.15.104:0/489897838 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c5b10700 1 --2- 172.21.15.104:0/489897838 >> v2:172.21.15.114:3300/0 conn(0x7fb3c006e4b0 0x7fb3c00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c530f700 1 -- 172.21.15.104:0/489897838 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099962015 0 0) 0x7fb3c006d290 con 0x7fb3c00ca100 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c530f700 1 -- 172.21.15.104:0/489897838 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ac0035b0 con 0x7fb3c00ca100 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 -- 172.21.15.104:0/489897838 >> v2:172.21.15.114:3300/0 conn(0x7fb3c006e4b0 msgr2=0x7fb3c00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 --2- 172.21.15.104:0/489897838 >> v2:172.21.15.114:3300/0 conn(0x7fb3c006e4b0 0x7fb3c00c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 -- 172.21.15.104:0/489897838 >> v1:172.21.15.104:6789/0 conn(0x7fb3c00ca100 legacy=0x7fb3c00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 -- 172.21.15.104:0/489897838 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c006d4c0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c6311700 1 --2- 172.21.15.104:0/489897838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c006cd50 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0070320 tx=0x7fb3b0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857dc5b07286cafe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c530f700 1 -- 172.21.15.104:0/489897838 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0005e30 con 0x7fb3c006e0e0 2023-06-10T18:56:03.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c530f700 1 -- 172.21.15.104:0/489897838 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b0006030 con 0x7fb3c006e0e0 2023-06-10T18:56:03.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.606+0000 7fb3c530f700 1 -- 172.21.15.104:0/489897838 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00063c0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 -- 172.21.15.104:0/489897838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 msgr2=0x7fb3c006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 --2- 172.21.15.104:0/489897838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c006cd50 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0070320 tx=0x7fb3b0009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:03.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 -- 172.21.15.104:0/489897838 shutdown_connections 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 --2- 172.21.15.104:0/489897838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c006cd50 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 --2- 172.21.15.104:0/489897838 >> v2:172.21.15.114:3300/0 conn(0x7fb3c006e4b0 0x7fb3c00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 -- 172.21.15.104:0/489897838 >> 172.21.15.104:0/489897838 conn(0x7fb3c006b8a0 msgr2=0x7fb3c00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 -- 172.21.15.104:0/489897838 shutdown_connections 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.607+0000 7fb3c8575700 1 -- 172.21.15.104:0/489897838 wait complete. 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.608+0000 7fb3c8575700 1 Processor -- start 2023-06-10T18:56:03.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.608+0000 7fb3c8575700 1 -- start start 2023-06-10T18:56:03.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c006e4b0 0x7fb3c00c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 0x7fb3c0152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00c75a0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00b0a10 con 0x7fb3c006e4b0 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00c6640 con 0x7fb3c00ca100 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36760/0 (socket says 172.21.15.104:36760) 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 -- 172.21.15.104:0/1221598212 learned_addr learned my addr 172.21.15.104:0/1221598212 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c5b10700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c006e4b0 0x7fb3c00c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6b12700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 0x7fb3c0152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 -- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 msgr2=0x7fb3c0152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 0x7fb3c0152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 -- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c006e4b0 msgr2=0x7fb3c00c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c006e4b0 0x7fb3c00c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 -- 172.21.15.104:0/1221598212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0009430 con 0x7fb3c006e0e0 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6b12700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 0x7fb3c0152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c6311700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009b70 tx=0x7fb3b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0004240 con 0x7fb3c006e0e0 2023-06-10T18:56:03.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b00043a0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c00c6920 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00036a0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.609+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c00c6ee0 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.610+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c00ce180 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.611+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb3b0022070 con 0x7fb3c006e0e0 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.611+0000 7fb3b6ffd700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 0x7fb3a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.611+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fb3b008ad10 con 0x7fb3c006e0e0 2023-06-10T18:56:03.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.612+0000 7fb3c5b10700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 0x7fb3a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.612+0000 7fb3c5b10700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 0x7fb3a406d190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00b2090 tx=0x7fb3bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.616+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb3b0055d50 con 0x7fb3c006e0e0 2023-06-10T18:56:03.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.810+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3c0074050 con 0x7fb3c006e0e0 2023-06-10T18:56:03.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.811+0000 7fb3b6ffd700 1 -- 172.21.15.104:0/1221598212 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+18716 (secure 0 0 0) 0x7fb3b002a020 con 0x7fb3c006e0e0 2023-06-10T18:56:03.818 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:03.819 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":116,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:57.725868+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/48469774":"2023-06-10T19:55:56.863350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 msgr2=0x7fb3a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 0x7fb3a406d190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00b2090 tx=0x7fb3bc008040 comp rx=0 tx=0).stop 2023-06-10T18:56:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 msgr2=0x7fb3c014daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009b70 tx=0x7fb3b0004750 comp rx=0 tx=0).stop 2023-06-10T18:56:03.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 shutdown_connections 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3c00ca100 0x7fb3c0152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3c006e0e0 0x7fb3c014daf0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb3a406ace0 0x7fb3a406d190 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 --2- 172.21.15.104:0/1221598212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c006e4b0 0x7fb3c00c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 >> 172.21.15.104:0/1221598212 conn(0x7fb3c006b8a0 msgr2=0x7fb3c00ac230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 shutdown_connections 2023-06-10T18:56:03.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.814+0000 7fb3c8575700 1 -- 172.21.15.104:0/1221598212 wait complete. 2023-06-10T18:56:03.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:55:05.032515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:55:06.191668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:03.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:03.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.951+0000 7f5b819ba700 1 Processor -- start 2023-06-10T18:56:03.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.951+0000 7f5b819ba700 1 -- start start 2023-06-10T18:56:03.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b819ba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b819ba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b819ba700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0af530 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b819ba700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c0af810 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b819ba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0afaf0 con 0x7f5b7c0b05d0 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36770/0 (socket says 172.21.15.104:36770) 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/2682384397 learned_addr learned my addr 172.21.15.104:0/2682384397 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7b7fe700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/2682384397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 msgr2=0x7f5b7c0b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:03.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c0b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/2682384397 >> v1:172.21.15.114:6789/0 conn(0x7f5b7c0d04f0 legacy=0x7f5b7c0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/2682384397 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0afdd0 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b7a7fc700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0098f0 tx=0x7f5b6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8f9434da5ab484 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b79ffb700 1 -- 172.21.15.104:0/2682384397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00b8a0 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.952+0000 7f5b79ffb700 1 -- 172.21.15.104:0/2682384397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c00ba00 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b79ffb700 1 -- 172.21.15.104:0/2682384397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00bd90 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 -- 172.21.15.104:0/2682384397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d25c0 msgr2=0x7f5b7c0d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0098f0 tx=0x7f5b6c009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:03.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 -- 172.21.15.104:0/2682384397 shutdown_connections 2023-06-10T18:56:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c0b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 --2- 172.21.15.104:0/2682384397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c0d34d0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 -- 172.21.15.104:0/2682384397 >> 172.21.15.104:0/2682384397 conn(0x7f5b7c071ec0 msgr2=0x7f5b7c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:03.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 -- 172.21.15.104:0/2682384397 shutdown_connections 2023-06-10T18:56:03.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.953+0000 7f5b819ba700 1 -- 172.21.15.104:0/2682384397 wait complete. 2023-06-10T18:56:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.954+0000 7f5b819ba700 1 Processor -- start 2023-06-10T18:56:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.954+0000 7f5b819ba700 1 -- start start 2023-06-10T18:56:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c1e6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b5750 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0cde00 con 0x7f5b7c0d25c0 2023-06-10T18:56:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b819ba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b59a0 con 0x7f5b7c0b05d0 2023-06-10T18:56:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b7a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b7a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36776/0 (socket says 172.21.15.104:36776) 2023-06-10T18:56:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/1035189708 learned_addr learned my addr 172.21.15.104:0/1035189708 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b7b7fe700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c1e6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.955+0000 7f5b7affd700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 msgr2=0x7f5b7c14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c0d25c0 msgr2=0x7f5b7c1e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c1e6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 -- 172.21.15.104:0/1035189708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c009550 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7affd700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b7a7fc700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009c40 tx=0x7f5b6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c0036a0 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c0b5c80 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c14dc30 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c01e070 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.956+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c003a00 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.957+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c0052f0 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.957+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5b6c003b60 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.958+0000 7f5b63fff700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 0x7f5b6806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.958+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5b6c020070 con 0x7f5b7c0d04f0 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.958+0000 7f5b7affd700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 0x7f5b6806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.958+0000 7f5b7affd700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 0x7f5b6806d1c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5b70000fb0 tx=0x7f5b70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:03.962+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5b6c055a10 con 0x7f5b7c0d04f0 2023-06-10T18:56:04.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.149+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b5c005b20 con 0x7f5b7c0d04f0 2023-06-10T18:56:04.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.150+0000 7f5b63fff700 1 -- 172.21.15.104:0/1035189708 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b6c059030 con 0x7f5b7c0d04f0 2023-06-10T18:56:04.156 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:04.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.151+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 msgr2=0x7f5b6806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 0x7f5b6806d1c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5b70000fb0 tx=0x7f5b70009040 comp rx=0 tx=0).stop 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 msgr2=0x7f5b7c1e5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009c40 tx=0x7f5b6c004750 comp rx=0 tx=0).stop 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 shutdown_connections 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b6806ad10 0x7f5b6806d1c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b7c0d25c0 0x7f5b7c1e6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b7c0b05d0 0x7f5b7c14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 --2- 172.21.15.104:0/1035189708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b7c0d04f0 0x7f5b7c1e5b20 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 >> 172.21.15.104:0/1035189708 conn(0x7f5b7c071ec0 msgr2=0x7f5b7c0d32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 shutdown_connections 2023-06-10T18:56:04.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.152+0000 7f5b819ba700 1 -- 172.21.15.104:0/1035189708 wait complete. 2023-06-10T18:56:04.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:04.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 Processor -- start 2023-06-10T18:56:04.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 -- start start 2023-06-10T18:56:04.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7528156760 0x7f75280ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75280b21c0 0x7f75280b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75280ce850 con 0x7f752815d1a0 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75280b30a0 con 0x7f7528156760 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.286+0000 7f752ecad700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752815d740 con 0x7f75280b21c0 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7528156760 0x7f75280ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7528156760 0x7f75280ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36784/0 (socket says 172.21.15.104:36784) 2023-06-10T18:56:04.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 -- 172.21.15.104:0/394457684 learned_addr learned my addr 172.21.15.104:0/394457684 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752d24a700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75280b21c0 0x7f75280b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 -- 172.21.15.104:0/394457684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75280b21c0 msgr2=0x7f75280b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75280b21c0 0x7f75280b25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 -- 172.21.15.104:0/394457684 >> v1:172.21.15.114:6789/0 conn(0x7f752815d1a0 legacy=0x7f75280cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 -- 172.21.15.104:0/394457684 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75280b2b20 con 0x7f7528156760 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f752ca49700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280ce310 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f751c0098f0 tx=0x7f751c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8302cb8f4cfb5643 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f75277fe700 1 -- 172.21.15.104:0/394457684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c004230 con 0x7f7528156760 2023-06-10T18:56:04.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f75277fe700 1 -- 172.21.15.104:0/394457684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751c004390 con 0x7f7528156760 2023-06-10T18:56:04.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.287+0000 7f75277fe700 1 -- 172.21.15.104:0/394457684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c0036a0 con 0x7f7528156760 2023-06-10T18:56:04.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 -- 172.21.15.104:0/394457684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 msgr2=0x7f75280ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280ce310 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f751c0098f0 tx=0x7f751c009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:04.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 -- 172.21.15.104:0/394457684 shutdown_connections 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280ce310 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 --2- 172.21.15.104:0/394457684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75280b21c0 0x7f75280b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 -- 172.21.15.104:0/394457684 >> 172.21.15.104:0/394457684 conn(0x7f752806e370 msgr2=0x7f752806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 -- 172.21.15.104:0/394457684 shutdown_connections 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.288+0000 7f752ecad700 1 -- 172.21.15.104:0/394457684 wait complete. 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.289+0000 7f752ecad700 1 Processor -- start 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.289+0000 7f752ecad700 1 -- start start 2023-06-10T18:56:04.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75280b21c0 0x7f75281f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 0x7f75281f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75280b4ee0 con 0x7f7528156760 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75280b3f80 con 0x7f75280b21c0 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ecad700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75280b4260 con 0x7f752815d1a0 2023-06-10T18:56:04.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36790/0 (socket says 172.21.15.104:36790) 2023-06-10T18:56:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 -- 172.21.15.104:0/4080038455 learned_addr learned my addr 172.21.15.104:0/4080038455 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752ca49700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75280b21c0 0x7f75281f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752d24a700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 0x7f75281f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 -- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 msgr2=0x7f75281f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 0x7f75281f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 -- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75280b21c0 msgr2=0x7f75281f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75280b21c0 0x7f75281f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 -- 172.21.15.104:0/4080038455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c009550 con 0x7f7528156760 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f752d24a700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 0x7f75281f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.290+0000 7f7527fff700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f751400b4a0 tx=0x7f751400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.291+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514011830 con 0x7f7528156760 2023-06-10T18:56:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.291+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7514011990 con 0x7f7528156760 2023-06-10T18:56:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.291+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75280b4540 con 0x7f7528156760 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.291+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514011d40 con 0x7f7528156760 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.291+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75280b4b00 con 0x7f7528156760 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.292+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7528074050 con 0x7f7528156760 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.292+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f751401a400 con 0x7f7528156760 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.292+0000 7f75257fa700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.292+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f7514089250 con 0x7f7528156760 2023-06-10T18:56:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.293+0000 7f752ca49700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.293+0000 7f752ca49700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f751c009130 tx=0x7f751c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.297+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f75140541e0 con 0x7f7528156760 2023-06-10T18:56:04.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.485+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f75280b5020 con 0x7f7528156760 2023-06-10T18:56:04.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.486+0000 7f75257fa700 1 -- 172.21.15.104:0/4080038455 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v116) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7514057800 con 0x7f7528156760 2023-06-10T18:56:04.492 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:04.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 msgr2=0x7f750c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f751c009130 tx=0x7f751c008030 comp rx=0 tx=0).stop 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 msgr2=0x7f75280b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f751400b4a0 tx=0x7f751400b7b0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 shutdown_connections 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f750c06ace0 0x7f750c06d190 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75280b21c0 0x7f75281f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f752815d1a0 0x7f75281f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 --2- 172.21.15.104:0/4080038455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7528156760 0x7f75280b5430 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 >> 172.21.15.104:0/4080038455 conn(0x7f752806e370 msgr2=0x7f752815bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 shutdown_connections 2023-06-10T18:56:04.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.488+0000 7f752ecad700 1 -- 172.21.15.104:0/4080038455 wait complete. 2023-06-10T18:56:04.506 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:56:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 Processor -- start 2023-06-10T18:56:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 -- start start 2023-06-10T18:56:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e8006cde0 0x7f8e800ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8006e250 con 0x7f8e8006e6e0 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8015c950 con 0x7f8e8006cde0 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.622+0000 7f8e85ce3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800b1e80 con 0x7f8e8015c580 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8e8006cde0 0x7f8e800ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8e8006e6e0 0x7f8e800b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34622/0 (socket says 172.21.15.104:34622) 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7effd700 1 -- 172.21.15.104:0/3956830136 learned_addr learned my addr 172.21.15.104:0/3956830136 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7f7fe700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7e7fc700 1 -- 172.21.15.104:0/3956830136 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172314489 0 0) 0x7f8e8006e250 con 0x7f8e8006e6e0 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7e7fc700 1 -- 172.21.15.104:0/3956830136 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e6c0035b0 con 0x7f8e8006e6e0 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 -- 172.21.15.104:0/3956830136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 msgr2=0x7f8e800b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 -- 172.21.15.104:0/3956830136 >> v1:172.21.15.104:6789/0 conn(0x7f8e8006e6e0 legacy=0x7f8e800b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 -- 172.21.15.104:0/3956830136 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8006d920 con 0x7f8e8006cde0 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7f7fe700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7ffff700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e800ce0d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d90 tx=0x7f8e7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb841b3f6e24c01a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7e7fc700 1 -- 172.21.15.104:0/3956830136 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7800e070 con 0x7f8e8006cde0 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7e7fc700 1 -- 172.21.15.104:0/3956830136 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7800bb20 con 0x7f8e8006cde0 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.623+0000 7f8e7e7fc700 1 -- 172.21.15.104:0/3956830136 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78012430 con 0x7f8e8006cde0 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3956830136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 msgr2=0x7f8e800ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e800ce0d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d90 tx=0x7f8e7800b330 comp rx=0 tx=0).stop 2023-06-10T18:56:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3956830136 shutdown_connections 2023-06-10T18:56:04.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e800ce0d0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3956830136 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3956830136 >> 172.21.15.104:0/3956830136 conn(0x7f8e8006b8a0 msgr2=0x7f8e800cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3956830136 shutdown_connections 2023-06-10T18:56:04.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.624+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3956830136 wait complete. 2023-06-10T18:56:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.625+0000 7f8e85ce3700 1 Processor -- start 2023-06-10T18:56:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.625+0000 7f8e85ce3700 1 -- start start 2023-06-10T18:56:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 0x7f8e8015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800c33f0 con 0x7f8e8006cde0 2023-06-10T18:56:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e801f2b90 con 0x7f8e8006e6e0 2023-06-10T18:56:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e85ce3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e801f21c0 con 0x7f8e8015c580 2023-06-10T18:56:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36798/0 (socket says 172.21.15.104:36798) 2023-06-10T18:56:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.626+0000 7f8e7f7fe700 1 -- 172.21.15.104:0/3112170484 learned_addr learned my addr 172.21.15.104:0/3112170484 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7effd700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 0x7f8e8015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7ffff700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 -- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 msgr2=0x7f8e800b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 -- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 msgr2=0x7f8e8015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 0x7f8e8015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 -- 172.21.15.104:0/3112170484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009a70 con 0x7f8e8006cde0 2023-06-10T18:56:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7effd700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 0x7f8e8015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e7f7fe700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000e840 tx=0x7f8e7000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70005c20 con 0x7f8e8006cde0 2023-06-10T18:56:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e70005d80 con 0x7f8e8006cde0 2023-06-10T18:56:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e801f1bd0 con 0x7f8e8006cde0 2023-06-10T18:56:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70006250 con 0x7f8e8006cde0 2023-06-10T18:56:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.627+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8015f680 con 0x7f8e8006cde0 2023-06-10T18:56:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.628+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e801f24a0 con 0x7f8e8006cde0 2023-06-10T18:56:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.628+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e700063b0 con 0x7f8e8006cde0 2023-06-10T18:56:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.629+0000 7f8e77fff700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 0x7f8e6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.629+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8e70089090 con 0x7f8e8006cde0 2023-06-10T18:56:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.629+0000 7f8e7effd700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 0x7f8e6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.630+0000 7f8e7effd700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 0x7f8e6006d190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8e8009d510 tx=0x7f8e68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.633+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e70053fa0 con 0x7f8e8006cde0 2023-06-10T18:56:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.821+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e80159900 con 0x7f8e8006cde0 2023-06-10T18:56:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.822+0000 7f8e77fff700 1 -- 172.21.15.104:0/3112170484 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e7000c450 con 0x7f8e8006cde0 2023-06-10T18:56:04.828 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:56:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 msgr2=0x7f8e6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 0x7f8e6006d190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8e8009d510 tx=0x7f8e68009040 comp rx=0 tx=0).stop 2023-06-10T18:56:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 msgr2=0x7f8e801f2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000e840 tx=0x7f8e7000ec00 comp rx=0 tx=0).stop 2023-06-10T18:56:04.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 shutdown_connections 2023-06-10T18:56:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8015c580 0x7f8e800b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e8006cde0 0x7f8e801f2650 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e6006ace0 0x7f8e6006d190 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 --2- 172.21.15.104:0/3112170484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e8006e6e0 0x7f8e8015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 >> 172.21.15.104:0/3112170484 conn(0x7f8e8006b8a0 msgr2=0x7f8e800c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 shutdown_connections 2023-06-10T18:56:04.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.824+0000 7f8e85ce3700 1 -- 172.21.15.104:0/3112170484 wait complete. 2023-06-10T18:56:04.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:56:04.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.959+0000 7faf6ba5f700 1 Processor -- start 2023-06-10T18:56:04.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.959+0000 7faf6ba5f700 1 -- start start 2023-06-10T18:56:04.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf6ba5f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf640d23f0 0x7faf640d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf6ba5f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf64145ba0 con 0x7faf640d0a60 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf6ba5f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf640d0e60 con 0x7faf640cec90 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf6ba5f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf640d0370 con 0x7faf640d23f0 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf68ffa700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf640d23f0 0x7faf640d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf69ffc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faf640d0a60 0x7faf640ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34638/0 (socket says 172.21.15.104:34638) 2023-06-10T18:56:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf69ffc700 1 -- 172.21.15.104:0/859012684 learned_addr learned my addr 172.21.15.104:0/859012684 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf68ffa700 1 -- 172.21.15.104:0/859012684 >> v1:172.21.15.114:6789/0 conn(0x7faf640cec90 legacy=0x7faf640cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf68ffa700 1 -- 172.21.15.104:0/859012684 >> v1:172.21.15.104:6789/0 conn(0x7faf640d0a60 legacy=0x7faf640ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf5bfff700 1 -- 172.21.15.104:0/859012684 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330801699 0 0) 0x7faf64145ba0 con 0x7faf640d0a60 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf68ffa700 1 -- 172.21.15.104:0/859012684 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640d2890 con 0x7faf640d23f0 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf68ffa700 1 --2- 172.21.15.104:0/859012684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d23f0 0x7faf640d0520 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7faf60009840 tx=0x7faf60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb40c3b5850bb11 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf5bfff700 1 -- 172.21.15.104:0/859012684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60004230 con 0x7faf640d23f0 2023-06-10T18:56:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf5bfff700 1 -- 172.21.15.104:0/859012684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60004390 con 0x7faf640d23f0 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.960+0000 7faf5bfff700 1 -- 172.21.15.104:0/859012684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600036a0 con 0x7faf640d23f0 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 -- 172.21.15.104:0/859012684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d23f0 msgr2=0x7faf640d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/859012684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d23f0 0x7faf640d0520 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7faf60009840 tx=0x7faf60009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 -- 172.21.15.104:0/859012684 shutdown_connections 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/859012684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d23f0 0x7faf640d0520 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 -- 172.21.15.104:0/859012684 >> 172.21.15.104:0/859012684 conn(0x7faf64071ec0 msgr2=0x7faf6409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 -- 172.21.15.104:0/859012684 shutdown_connections 2023-06-10T18:56:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.961+0000 7faf6ba5f700 1 -- 172.21.15.104:0/859012684 wait complete. 2023-06-10T18:56:04.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.962+0000 7faf6ba5f700 1 Processor -- start 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.962+0000 7faf6ba5f700 1 -- start start 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 0x7faf640b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 0x7faf6409e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6409f420 con 0x7faf640d0a60 2023-06-10T18:56:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6409f700 con 0x7faf640d23f0 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf6ba5f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6409f9e0 con 0x7faf640cec90 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf68ffa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf68ffa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36814/0 (socket says 172.21.15.104:36814) 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf68ffa700 1 -- 172.21.15.104:0/3284276519 learned_addr learned my addr 172.21.15.104:0/3284276519 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf69ffc700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 0x7faf6409e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf68ffa700 1 -- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 msgr2=0x7faf640b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.963+0000 7faf697fb700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 0x7faf640b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf68ffa700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 0x7faf640b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf68ffa700 1 -- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 msgr2=0x7faf6409e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf68ffa700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 0x7faf6409e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf68ffa700 1 -- 172.21.15.104:0/3284276519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600094a0 con 0x7faf640d0a60 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf69ffc700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 0x7faf6409e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf697fb700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 0x7faf640b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf68ffa700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7faf640d04d0 tx=0x7faf600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011630 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60011790 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6409fcc0 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011b40 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.964+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6409edc0 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.965+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6409fe50 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.965+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf60011ca0 con 0x7faf640d0a60 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.966+0000 7faf59ffb700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 0x7faf4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.966+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7faf6008af70 con 0x7faf640d0a60 2023-06-10T18:56:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.966+0000 7faf697fb700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 0x7faf4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.967+0000 7faf697fb700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 0x7faf4806d190 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faf54006fd0 tx=0x7faf54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:04.970+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf600bd310 con 0x7faf640d0a60 2023-06-10T18:56:05.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.155+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf64074050 con 0x7faf640d0a60 2023-06-10T18:56:05.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.156+0000 7faf59ffb700 1 -- 172.21.15.104:0/3284276519 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf60055f00 con 0x7faf640d0a60 2023-06-10T18:56:05.162 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 msgr2=0x7faf4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 0x7faf4806d190 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faf54006fd0 tx=0x7faf54008040 comp rx=0 tx=0).stop 2023-06-10T18:56:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 msgr2=0x7faf640b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7faf640d04d0 tx=0x7faf600047c0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 shutdown_connections 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf4806ace0 0x7faf4806d190 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf640d23f0 0x7faf6409e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf640cec90 0x7faf640b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 --2- 172.21.15.104:0/3284276519 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf640d0a60 0x7faf640b46d0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 >> 172.21.15.104:0/3284276519 conn(0x7faf64071ec0 msgr2=0x7faf640b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 shutdown_connections 2023-06-10T18:56:05.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.158+0000 7faf6ba5f700 1 -- 172.21.15.104:0/3284276519 wait complete. 2023-06-10T18:56:05.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:05.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.294+0000 7f9cac77e700 1 Processor -- start 2023-06-10T18:56:05.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.294+0000 7f9cac77e700 1 -- start start 2023-06-10T18:56:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9cac77e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9ca40701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9cac77e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40ccca0 con 0x7f9ca415a250 2023-06-10T18:56:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9cac77e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40cee50 con 0x7f9ca40cad90 2023-06-10T18:56:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9cac77e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40cf130 con 0x7f9ca40cfa00 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9ca9d19700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9ca415a250 0x7f9ca40cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34648/0 (socket says 172.21.15.104:34648) 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.295+0000 7f9ca9d19700 1 -- 172.21.15.104:0/2996800096 learned_addr learned my addr 172.21.15.104:0/2996800096 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132360481 0 0) 0x7f9ca40ccca0 con 0x7f9ca415a250 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caa51a700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9ca40701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c900035b0 con 0x7f9ca415a250 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729322136 0 0) 0x7f9c900035b0 con 0x7f9ca415a250 2023-06-10T18:56:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 msgr2=0x7f9ca40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9ca40701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 >> v1:172.21.15.114:6789/0 conn(0x7f9ca40cad90 legacy=0x7f9ca40b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40cc930 con 0x7f9ca415a250 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9ca0001670 con 0x7f9ca415a250 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 >> v1:172.21.15.104:6789/0 conn(0x7f9ca415a250 legacy=0x7f9ca40cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90003a30 0x7f9c90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9c90004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c90004d90 0x7f9c90005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40cf130 con 0x7f9c90004d90 2023-06-10T18:56:05.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9000bac0 con 0x7f9c90003a30 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9000bc50 con 0x7f9ca40cfa00 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c90004d90 0x7f9c90005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caa51a700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90003a30 0x7f9c90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9ca9d19700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9c90004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 -- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 msgr2=0x7f9c90004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40cfa00 0x7f9c90004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 -- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90003a30 msgr2=0x7f9c90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90003a30 0x7f9c90003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 -- 172.21.15.104:0/2996800096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900035b0 con 0x7f9c90004d90 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.296+0000 7f9caad1b700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c90004d90 0x7f9c90005200 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00c290 tx=0x7f9c90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822f5fba5fb02ad8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c001e70 con 0x7f9c90004d90 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c00ce50 con 0x7f9c90004d90 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9ca9518700 1 -- 172.21.15.104:0/2996800096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c012660 con 0x7f9c90004d90 2023-06-10T18:56:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 -- 172.21.15.104:0/2996800096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c90004d90 msgr2=0x7f9c90005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 --2- 172.21.15.104:0/2996800096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c90004d90 0x7f9c90005200 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00c290 tx=0x7f9c90005880 comp rx=0 tx=0).stop 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9caad1b700 1 -- 172.21.15.104:0/2996800096 reap_dead start 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 -- 172.21.15.104:0/2996800096 shutdown_connections 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 -- 172.21.15.104:0/2996800096 >> 172.21.15.104:0/2996800096 conn(0x7f9ca406e370 msgr2=0x7f9ca406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 -- 172.21.15.104:0/2996800096 shutdown_connections 2023-06-10T18:56:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.297+0000 7f9cac77e700 1 -- 172.21.15.104:0/2996800096 wait complete. 2023-06-10T18:56:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.298+0000 7f9cac77e700 1 Processor -- start 2023-06-10T18:56:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.298+0000 7f9cac77e700 1 -- start start 2023-06-10T18:56:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90004d90 0x7f9ca40b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40ca3f0 0x7f9ca40b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca406b0b0 con 0x7f9ca40ca860 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b0680 con 0x7f9c90004d90 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9cac77e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b08d0 con 0x7f9ca40ca3f0 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36832/0 (socket says 172.21.15.104:36832) 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 -- 172.21.15.104:0/3346970464 learned_addr learned my addr 172.21.15.104:0/3346970464 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caa51a700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90004d90 0x7f9ca40b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9ca9d19700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40ca3f0 0x7f9ca40b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 -- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40ca3f0 msgr2=0x7f9ca40b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40ca3f0 0x7f9ca40b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.299+0000 7f9caad1b700 1 -- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90004d90 msgr2=0x7f9ca40b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9caad1b700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90004d90 0x7f9ca40b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9caad1b700 1 -- 172.21.15.104:0/3346970464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40b0bb0 con 0x7f9ca40ca860 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9caad1b700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9c900047a0 tx=0x7f9c9c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0050c0 con 0x7f9ca40ca860 2023-06-10T18:56:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c005220 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca40b0e90 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0055d0 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.300+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca40b1170 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.301+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca406b220 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.301+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9c9c023070 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.301+0000 7f9c9affd700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 0x7f9c8806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.302+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9c9c08af70 con 0x7f9ca40ca860 2023-06-10T18:56:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.302+0000 7f9caa51a700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 0x7f9c8806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.302+0000 7f9caa51a700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 0x7f9c8806d190 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009ba0 tx=0x7f9c94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.306+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9ca406b220 con 0x7f9ca40ca860 2023-06-10T18:56:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.531+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca4074050 con 0x7f9ca40ca860 2023-06-10T18:56:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.532+0000 7f9c9affd700 1 -- 172.21.15.104:0/3346970464 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f9c9c055f00 con 0x7f9ca40ca860 2023-06-10T18:56:05.538 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:05.538 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:56:05.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 msgr2=0x7f9c8806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 0x7f9c8806d190 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009ba0 tx=0x7f9c94008040 comp rx=0 tx=0).stop 2023-06-10T18:56:05.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 msgr2=0x7f9ca40b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9c900047a0 tx=0x7f9c9c0015b0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 shutdown_connections 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9c8806ace0 0x7f9c8806d190 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c90004d90 0x7f9ca40b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ca40ca860 0x7f9ca40b43f0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 --2- 172.21.15.104:0/3346970464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9ca40ca3f0 0x7f9ca40b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 >> 172.21.15.104:0/3346970464 conn(0x7f9ca406e370 msgr2=0x7f9ca40c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:05.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 shutdown_connections 2023-06-10T18:56:05.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.534+0000 7f9cac77e700 1 -- 172.21.15.104:0/3346970464 wait complete. 2023-06-10T18:56:05.542 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:56:05.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:05.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.665+0000 7fe726384700 1 Processor -- start 2023-06-10T18:56:05.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 -- start start 2023-06-10T18:56:05.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 0x7fe7200cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200c8de0 con 0x7fe7200cb3e0 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200ccc90 con 0x7fe7200cdbb0 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.666+0000 7fe726384700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200ccf30 con 0x7fe720158e50 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 0x7fe7200cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe724921700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe7200cb3e0 0x7fe72015cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40512/0 (socket says 172.21.15.104:40512) 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71f7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:52572/0 (socket says 172.21.15.104:52572) 2023-06-10T18:56:05.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe724921700 1 -- 172.21.15.104:0/1747953373 learned_addr learned my addr 172.21.15.104:0/1747953373 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 -- 172.21.15.104:0/1747953373 >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 msgr2=0x7fe7200caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 --2- 172.21.15.104:0/1747953373 >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 -- 172.21.15.104:0/1747953373 >> v1:172.21.15.114:6789/0 conn(0x7fe7200cb3e0 legacy=0x7fe72015cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 -- 172.21.15.104:0/1747953373 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200cd210 con 0x7fe720158e50 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71f7fe700 1 --2- 172.21.15.104:0/1747953373 >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71ffff700 1 --2- 172.21.15.104:0/1747953373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 0x7fe7200cd670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe710009840 tx=0x7fe710009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8988df9e4c00d64a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71effd700 1 -- 172.21.15.104:0/1747953373 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710011830 con 0x7fe720158e50 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71effd700 1 -- 172.21.15.104:0/1747953373 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710011990 con 0x7fe720158e50 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.667+0000 7fe71effd700 1 -- 172.21.15.104:0/1747953373 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710011d20 con 0x7fe720158e50 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 -- 172.21.15.104:0/1747953373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 msgr2=0x7fe7200cd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 --2- 172.21.15.104:0/1747953373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 0x7fe7200cd670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe710009840 tx=0x7fe710009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 -- 172.21.15.104:0/1747953373 shutdown_connections 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 --2- 172.21.15.104:0/1747953373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe720158e50 0x7fe7200cd670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 --2- 172.21.15.104:0/1747953373 >> v2:172.21.15.114:3300/0 conn(0x7fe7200cdbb0 0x7fe7200caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 -- 172.21.15.104:0/1747953373 >> 172.21.15.104:0/1747953373 conn(0x7fe72006ab00 msgr2=0x7fe7200b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:05.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 -- 172.21.15.104:0/1747953373 shutdown_connections 2023-06-10T18:56:05.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.668+0000 7fe726384700 1 -- 172.21.15.104:0/1747953373 wait complete. 2023-06-10T18:56:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.669+0000 7fe726384700 1 Processor -- start 2023-06-10T18:56:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.670+0000 7fe726384700 1 -- start start 2023-06-10T18:56:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200cdbb0 0x7fe7200c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe720158e50 0x7fe7200c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c3230 con 0x7fe7200cb3e0 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c3510 con 0x7fe720158e50 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c37f0 con 0x7fe7200cdbb0 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36842/0 (socket says 172.21.15.104:36842) 2023-06-10T18:56:05.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 -- 172.21.15.104:0/640643426 learned_addr learned my addr 172.21.15.104:0/640643426 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe724921700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe720158e50 0x7fe7200c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71f7fe700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200cdbb0 0x7fe7200c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 -- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200cdbb0 msgr2=0x7fe7200c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200cdbb0 0x7fe7200c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 -- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe720158e50 msgr2=0x7fe7200c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe720158e50 0x7fe7200c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 -- 172.21.15.104:0/640643426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080094a0 con 0x7fe7200cb3e0 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71ffff700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710004550 con 0x7fe7200cb3e0 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71001e070 con 0x7fe7200cb3e0 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7100094a0 con 0x7fe7200cb3e0 2023-06-10T18:56:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7100038e0 con 0x7fe7200cb3e0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.671+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7200c3db0 con 0x7fe7200cb3e0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.672+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe720074050 con 0x7fe7200cb3e0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.673+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe710003a40 con 0x7fe7200cb3e0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.673+0000 7fe71cff9700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 0x7fe70006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.673+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe71008a500 con 0x7fe7200cb3e0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.673+0000 7fe71f7fe700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 0x7fe70006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:05.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.674+0000 7fe71f7fe700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 0x7fe70006d190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe72014e550 tx=0x7fe708012040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:05.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.678+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe710055490 con 0x7fe7200cb3e0 2023-06-10T18:56:05.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.872+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe72014ef80 con 0x7fe7200cb3e0 2023-06-10T18:56:05.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.873+0000 7fe71cff9700 1 -- 172.21.15.104:0/640643426 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+18716 (secure 0 0 0) 0x7fe710020090 con 0x7fe7200cb3e0 2023-06-10T18:56:05.879 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:05.879 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":116,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:57.725868+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/48469774":"2023-06-10T19:55:56.863350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:05.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 msgr2=0x7fe70006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 0x7fe70006d190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe72014e550 tx=0x7fe708012040 comp rx=0 tx=0).stop 2023-06-10T18:56:05.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 msgr2=0x7fe720148f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:05.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710004750 comp rx=0 tx=0).stop 2023-06-10T18:56:05.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 shutdown_connections 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe70006ace0 0x7fe70006d190 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe720158e50 0x7fe7200c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe7200cb3e0 0x7fe720148f10 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 --2- 172.21.15.104:0/640643426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200cdbb0 0x7fe7200c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 >> 172.21.15.104:0/640643426 conn(0x7fe72006ab00 msgr2=0x7fe7200ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 shutdown_connections 2023-06-10T18:56:05.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:05.875+0000 7fe726384700 1 -- 172.21.15.104:0/640643426 wait complete. 2023-06-10T18:56:05.894 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:56:05.894 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:06.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.007+0000 7fef64837700 1 Processor -- start 2023-06-10T18:56:06.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.007+0000 7fef64837700 1 -- start start 2023-06-10T18:56:06.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef64837700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef600717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef64837700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6006b2e0 con 0x7fef600c93e0 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef64837700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6015e150 con 0x7fef60071130 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef64837700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015db40 con 0x7fef6015a0b0 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef5dd9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fef600c93e0 0x7fef60070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34678/0 (socket says 172.21.15.104:34678) 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.008+0000 7fef5dd9b700 1 -- 172.21.15.104:0/2302325569 learned_addr learned my addr 172.21.15.104:0/2302325569 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:06.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef600717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868414141 0 0) 0x7fef6006b2e0 con 0x7fef600c93e0 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef480035b0 con 0x7fef600c93e0 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831399056 0 0) 0x7fef480035b0 con 0x7fef600c93e0 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 msgr2=0x7fef600717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef600717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 >> v1:172.21.15.114:6789/0 conn(0x7fef60071130 legacy=0x7fef6015a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6006b530 con 0x7fef600c93e0 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fef50001670 con 0x7fef600c93e0 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 >> v1:172.21.15.104:6789/0 conn(0x7fef600c93e0 legacy=0x7fef60070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef48003a30 0x7fef48003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef48004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004d90 0x7fef48005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015db40 con 0x7fef48003a30 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800bac0 con 0x7fef48004d90 2023-06-10T18:56:06.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800bc50 con 0x7fef6015a0b0 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef48003a30 0x7fef48003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5dd9b700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef48004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5ed9d700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004d90 0x7fef48005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 -- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 msgr2=0x7fef48004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015a0b0 0x7fef48004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 -- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004d90 msgr2=0x7fef48005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004d90 0x7fef48005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 -- 172.21.15.104:0/2302325569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480035b0 con 0x7fef48003a30 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5ed9d700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004d90 0x7fef48005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.009+0000 7fef5e59c700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef48003a30 0x7fef48003e40 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fef54009b60 tx=0x7fef48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae75fe82cf60ff5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400e6c0 con 0x7fef48003a30 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5400e820 con 0x7fef48003a30 2023-06-10T18:56:06.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef5d59a700 1 -- 172.21.15.104:0/2302325569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400eb90 con 0x7fef48003a30 2023-06-10T18:56:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 -- 172.21.15.104:0/2302325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef48003a30 msgr2=0x7fef48003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 --2- 172.21.15.104:0/2302325569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef48003a30 0x7fef48003e40 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fef54009b60 tx=0x7fef48005880 comp rx=0 tx=0).stop 2023-06-10T18:56:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef5ed9d700 1 -- 172.21.15.104:0/2302325569 reap_dead start 2023-06-10T18:56:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 -- 172.21.15.104:0/2302325569 shutdown_connections 2023-06-10T18:56:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 -- 172.21.15.104:0/2302325569 >> 172.21.15.104:0/2302325569 conn(0x7fef600b1610 msgr2=0x7fef600ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 -- 172.21.15.104:0/2302325569 shutdown_connections 2023-06-10T18:56:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.010+0000 7fef64837700 1 -- 172.21.15.104:0/2302325569 wait complete. 2023-06-10T18:56:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 Processor -- start 2023-06-10T18:56:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 -- start start 2023-06-10T18:56:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48003a30 0x7fef601f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015b670 0x7fef6015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef60157140 con 0x7fef60156790 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015c030 con 0x7fef48003a30 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef64837700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015c2f0 con 0x7fef6015b670 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef5dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef5dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36862/0 (socket says 172.21.15.104:36862) 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.012+0000 7fef5dd9b700 1 -- 172.21.15.104:0/156489176 learned_addr learned my addr 172.21.15.104:0/156489176 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:06.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 -- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015b670 msgr2=0x7fef6015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015b670 0x7fef6015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 -- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48003a30 msgr2=0x7fef601f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5e59c700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48003a30 0x7fef601f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48003a30 0x7fef601f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 -- 172.21.15.104:0/156489176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6015c5d0 con 0x7fef60156790 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef5dd9b700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fef500072c0 tx=0x7fef50007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5000f8d0 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5000fa30 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6015c8b0 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5000fde0 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.013+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6015ce70 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.014+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6015e5e0 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.014+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fef5001d460 con 0x7fef60156790 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.015+0000 7fef4effd700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 0x7fef4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.015+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fef50089700 con 0x7fef60156790 2023-06-10T18:56:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.015+0000 7fef5e59c700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 0x7fef4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.015+0000 7fef5e59c700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 0x7fef4006d190 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fef600ceb40 tx=0x7fef54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.019+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fef50054600 con 0x7fef60156790 2023-06-10T18:56:06.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.245+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef6015d000 con 0x7fef60156790 2023-06-10T18:56:06.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.246+0000 7fef4effd700 1 -- 172.21.15.104:0/156489176 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7fef50057c20 con 0x7fef60156790 2023-06-10T18:56:06.252 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:06.252 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:56:06.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 msgr2=0x7fef4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 0x7fef4006d190 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fef600ceb40 tx=0x7fef54008030 comp rx=0 tx=0).stop 2023-06-10T18:56:06.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 msgr2=0x7fef60156c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fef500072c0 tx=0x7fef50007680 comp rx=0 tx=0).stop 2023-06-10T18:56:06.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 shutdown_connections 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef4006ace0 0x7fef4006d190 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48003a30 0x7fef601f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef6015b670 0x7fef6015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 --2- 172.21.15.104:0/156489176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef60156790 0x7fef60156c00 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 >> 172.21.15.104:0/156489176 conn(0x7fef600b1610 msgr2=0x7fef6006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 shutdown_connections 2023-06-10T18:56:06.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.248+0000 7fef64837700 1 -- 172.21.15.104:0/156489176 wait complete. 2023-06-10T18:56:06.257 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:56:06.266 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:06.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.380+0000 7f23aec10700 1 Processor -- start 2023-06-10T18:56:06.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.380+0000 7f23aec10700 1 -- start start 2023-06-10T18:56:06.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23aec10700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f23a80cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23aec10700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23a80b1080 0x7f23a80cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23aec10700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80c9e60 con 0x7f23a80ca2f0 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23aec10700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a80c9bc0 con 0x7f23a80b1080 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23aec10700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80aee70 con 0x7f23a80aeaa0 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23ad1ad700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f23a80ca2f0 0x7f23a80ca750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34696/0 (socket says 172.21.15.104:34696) 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.381+0000 7f23ad1ad700 1 -- 172.21.15.104:0/670347511 learned_addr learned my addr 172.21.15.104:0/670347511 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:06.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23ac9ac700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f23a80cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 --2- 172.21.15.104:0/670347511 >> v2:172.21.15.114:3300/0 conn(0x7f23a80b1080 0x7f23a80cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259672061 0 0) 0x7f23a80c9e60 con 0x7f23a80ca2f0 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23980035b0 con 0x7f23a80ca2f0 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4258026664 0 0) 0x7f23980035b0 con 0x7f23a80ca2f0 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 msgr2=0x7f23a80cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f23a80cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 >> v2:172.21.15.114:3300/0 conn(0x7f23a80b1080 msgr2=0x7f23a80cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 --2- 172.21.15.104:0/670347511 >> v2:172.21.15.114:3300/0 conn(0x7f23a80b1080 0x7f23a80cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a806e540 con 0x7f23a80ca2f0 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f23a00019f0 con 0x7f23a80ca2f0 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 >> v1:172.21.15.104:6789/0 conn(0x7f23a80ca2f0 legacy=0x7f23a80ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f2398003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2398003f80 0x7f2398004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398004d90 0x7f2398005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80c9bc0 con 0x7f2398003f80 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80aee70 con 0x7f2398004d90 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239800ba10 con 0x7f23a80aeaa0 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2398003f80 0x7f2398004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23ac9ac700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f2398003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23ad1ad700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398004d90 0x7f2398005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 -- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 msgr2=0x7f2398003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80aeaa0 0x7f2398003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 -- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398004d90 msgr2=0x7f2398005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398004d90 0x7f2398005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 -- 172.21.15.104:0/670347511 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980035b0 con 0x7f2398003f80 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.382+0000 7f23a7fff700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2398003f80 0x7f2398004850 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2394009b60 tx=0x7f2398005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0668be9778ff5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400e6c0 con 0x7f2398003f80 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239400e820 con 0x7f2398003f80 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23a77fe700 1 -- 172.21.15.104:0/670347511 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400eb90 con 0x7f2398003f80 2023-06-10T18:56:06.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 -- 172.21.15.104:0/670347511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2398003f80 msgr2=0x7f2398004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 --2- 172.21.15.104:0/670347511 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2398003f80 0x7f2398004850 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2394009b60 tx=0x7f2398005880 comp rx=0 tx=0).stop 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23ad1ad700 1 -- 172.21.15.104:0/670347511 reap_dead start 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 -- 172.21.15.104:0/670347511 shutdown_connections 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 -- 172.21.15.104:0/670347511 >> 172.21.15.104:0/670347511 conn(0x7f23a806b8a0 msgr2=0x7f23a80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 -- 172.21.15.104:0/670347511 shutdown_connections 2023-06-10T18:56:06.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.383+0000 7f23aec10700 1 -- 172.21.15.104:0/670347511 wait complete. 2023-06-10T18:56:06.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.385+0000 7f23aec10700 1 Processor -- start 2023-06-10T18:56:06.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.385+0000 7f23aec10700 1 -- start start 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398003f80 0x7f23a8148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 0x7f23a80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80c70c0 con 0x7f23a80c6b80 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80c73a0 con 0x7f2398003f80 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23aec10700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80c7680 con 0x7f23a80c8160 2023-06-10T18:56:06.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36882/0 (socket says 172.21.15.104:36882) 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 -- 172.21.15.104:0/1021644642 learned_addr learned my addr 172.21.15.104:0/1021644642 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ac9ac700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398003f80 0x7f23a8148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23a7fff700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 0x7f23a80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 -- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 msgr2=0x7f23a80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 0x7f23a80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 -- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398003f80 msgr2=0x7f23a8148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398003f80 0x7f23a8148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 -- 172.21.15.104:0/1021644642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a80c7960 con 0x7f23a80c6b80 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23a7fff700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 0x7f23a80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.386+0000 7f23ad1ad700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f23a806ff90 tx=0x7f23a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.387+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a000ad90 con 0x7f23a80c6b80 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.387+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a00133f0 con 0x7f23a80c6b80 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.387+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a80c7c40 con 0x7f23a80c6b80 2023-06-10T18:56:06.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.387+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00137a0 con 0x7f23a80c6b80 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.387+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a806dbc0 con 0x7f23a80c6b80 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.388+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a80ca010 con 0x7f23a80c6b80 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.388+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f23a0013900 con 0x7f23a80c6b80 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.388+0000 7f23a57fa700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 0x7f238c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.388+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f23a008a270 con 0x7f23a80c6b80 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.389+0000 7f23ac9ac700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 0x7f238c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.389+0000 7f23ac9ac700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 0x7f238c06d190 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f239c009b60 tx=0x7f239c0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.393+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f23a0055200 con 0x7f23a80c6b80 2023-06-10T18:56:06.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.625+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23a80c7e30 con 0x7f23a80c6b80 2023-06-10T18:56:06.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.625+0000 7f23a57fa700 1 -- 172.21.15.104:0/1021644642 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f23a0058820 con 0x7f23a80c6b80 2023-06-10T18:56:06.631 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:06.632 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:55:07.210651+0000","modified":"2023-06-10T18:55:20.650069+0000","tableserver":0,"root":0,"session_timeout":30,"session_autoclose":50,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8781":{"gid":8781,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/2511953457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2511953457},{"type":"v1","addr":"172.21.15.104:6835","nonce":2511953457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-06-10T18:56:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 msgr2=0x7f238c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 0x7f238c06d190 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f239c009b60 tx=0x7f239c0091c0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 msgr2=0x7f23a806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f23a806ff90 tx=0x7f23a000a620 comp rx=0 tx=0).stop 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 shutdown_connections 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f238c06ace0 0x7f238c06d190 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2398003f80 0x7f23a8148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23a80c6b80 0x7f23a806d680 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 --2- 172.21.15.104:0/1021644642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23a80c8160 0x7f23a80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 >> 172.21.15.104:0/1021644642 conn(0x7f23a806b8a0 msgr2=0x7f23a814ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 shutdown_connections 2023-06-10T18:56:06.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.627+0000 7f23aec10700 1 -- 172.21.15.104:0/1021644642 wait complete. 2023-06-10T18:56:06.635 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 72 2023-06-10T18:56:06.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:06.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.759+0000 7f7ce5b86700 1 Processor -- start 2023-06-10T18:56:06.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.759+0000 7f7ce5b86700 1 -- start start 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7ce5b86700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7ce006e0e0 0x7f7ce006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7ce5b86700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e4b0 0x7f7ce00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7ce5b86700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce006d290 con 0x7f7ce00ca100 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7ce5b86700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce0071640 con 0x7f7ce006e0e0 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7ce5b86700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce00aef00 con 0x7f7ce006e4b0 2023-06-10T18:56:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7cdf7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7ce006e0e0 0x7f7ce006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7cdffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7ce00ca100 0x7f7ce00ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34718/0 (socket says 172.21.15.104:34718) 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.760+0000 7f7cdffff700 1 -- 172.21.15.104:0/3626933357 learned_addr learned my addr 172.21.15.104:0/3626933357 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cdf7fe700 1 -- 172.21.15.104:0/3626933357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e4b0 msgr2=0x7f7ce00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cdf7fe700 1 --2- 172.21.15.104:0/3626933357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e4b0 0x7f7ce00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cdf7fe700 1 -- 172.21.15.104:0/3626933357 >> v1:172.21.15.104:6789/0 conn(0x7f7ce00ca100 legacy=0x7f7ce00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cdf7fe700 1 -- 172.21.15.104:0/3626933357 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce006d4c0 con 0x7f7ce006e0e0 2023-06-10T18:56:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cdf7fe700 1 --2- 172.21.15.104:0/3626933357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e0e0 0x7f7ce006cd50 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009820 tx=0x7f7cc8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607426dc51777ac6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cde7fc700 1 -- 172.21.15.104:0/3626933357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8005e30 con 0x7f7ce006e0e0 2023-06-10T18:56:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cde7fc700 1 -- 172.21.15.104:0/3626933357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc8006030 con 0x7f7ce006e0e0 2023-06-10T18:56:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7cde7fc700 1 -- 172.21.15.104:0/3626933357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80063e0 con 0x7f7ce006e0e0 2023-06-10T18:56:06.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 -- 172.21.15.104:0/3626933357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e0e0 msgr2=0x7f7ce006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/3626933357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e0e0 0x7f7ce006cd50 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009820 tx=0x7f7cc8009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:06.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 -- 172.21.15.104:0/3626933357 shutdown_connections 2023-06-10T18:56:06.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/3626933357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e0e0 0x7f7ce006cd50 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/3626933357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e4b0 0x7f7ce00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 -- 172.21.15.104:0/3626933357 >> 172.21.15.104:0/3626933357 conn(0x7f7ce006b8a0 msgr2=0x7f7ce00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 -- 172.21.15.104:0/3626933357 shutdown_connections 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.761+0000 7f7ce5b86700 1 -- 172.21.15.104:0/3626933357 wait complete. 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.763+0000 7f7ce5b86700 1 Processor -- start 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.763+0000 7f7ce5b86700 1 -- start start 2023-06-10T18:56:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e0e0 0x7f7ce014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce00ca100 0x7f7ce00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce013fa80 con 0x7f7ce006e4b0 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce013f730 con 0x7f7ce00ca100 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7ce5b86700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01504b0 con 0x7f7ce006e0e0 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36894/0 (socket says 172.21.15.104:36894) 2023-06-10T18:56:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 -- 172.21.15.104:0/2204381857 learned_addr learned my addr 172.21.15.104:0/2204381857 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdffff700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce00ca100 0x7f7ce00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdf7fe700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e0e0 0x7f7ce014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 -- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e0e0 msgr2=0x7f7ce014ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e0e0 0x7f7ce014ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 -- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce00ca100 msgr2=0x7f7ce00cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.764+0000 7f7cdeffd700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce00ca100 0x7f7ce00cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7cdeffd700 1 -- 172.21.15.104:0/2204381857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc8009430 con 0x7f7ce006e4b0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7cdeffd700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000b560 tx=0x7f7cd000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0011830 con 0x7f7ce006e4b0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd0011990 con 0x7f7ce006e4b0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce00b17d0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0011d40 con 0x7f7ce006e4b0 2023-06-10T18:56:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.765+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce00b1d90 con 0x7f7ce006e4b0 2023-06-10T18:56:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.766+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce013fbc0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.766+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7cd001a400 con 0x7f7ce006e4b0 2023-06-10T18:56:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.767+0000 7f7cd7fff700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 0x7f7cc006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.767+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f7cd00896d0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.767+0000 7f7cdf7fe700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 0x7f7cc006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:06.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.767+0000 7f7cdf7fe700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 0x7f7cc006d190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8013040 tx=0x7f7cc8008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:06.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.771+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7ce013fbc0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.957+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce013fbc0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.958+0000 7f7cd7fff700 1 -- 172.21.15.104:0/2204381857 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+18716 (secure 0 0 0) 0x7f7ce013fbc0 con 0x7f7ce006e4b0 2023-06-10T18:56:06.964 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:06.965 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":116,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:55:57.725868+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:55:05.032515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-06-10T18:55:06.191668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.e","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/48469774":"2023-06-10T19:55:56.863350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 msgr2=0x7f7cc006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 0x7f7cc006d190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8013040 tx=0x7f7cc8008070 comp rx=0 tx=0).stop 2023-06-10T18:56:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 msgr2=0x7f7ce00cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000b560 tx=0x7f7cd000b920 comp rx=0 tx=0).stop 2023-06-10T18:56:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 shutdown_connections 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7cc006ace0 0x7f7cc006d190 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ce00ca100 0x7f7ce00cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ce006e0e0 0x7f7ce014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 --2- 172.21.15.104:0/2204381857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ce006e4b0 0x7f7ce00cb110 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 >> 172.21.15.104:0/2204381857 conn(0x7f7ce006b8a0 msgr2=0x7f7ce00ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 shutdown_connections 2023-06-10T18:56:06.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:06.960+0000 7f7ce5b86700 1 -- 172.21.15.104:0/2204381857 wait complete. 2023-06-10T18:56:06.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:56:07.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.092+0000 7f2314d37700 1 Processor -- start 2023-06-10T18:56:07.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.092+0000 7f2314d37700 1 -- start start 2023-06-10T18:56:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f2314d37700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f23100ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f2314d37700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231006dcb0 con 0x7f2310069e70 2023-06-10T18:56:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f2314d37700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100cf460 con 0x7f231015c500 2023-06-10T18:56:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f2314d37700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231006e850 con 0x7f23100cf030 2023-06-10T18:56:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f230ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2310069e70 0x7f23100c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34732/0 (socket says 172.21.15.104:34732) 2023-06-10T18:56:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f230ed9d700 1 -- 172.21.15.104:0/2338386913 learned_addr learned my addr 172.21.15.104:0/2338386913 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f230dd9b700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f23100ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359215519 0 0) 0x7f231006dcb0 con 0x7f2310069e70 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.093+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80035b0 con 0x7f2310069e70 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364717000 0 0) 0x7f23100cf460 con 0x7f231015c500 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231006dcb0 con 0x7f231015c500 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1318441597 0 0) 0x7f22f80035b0 con 0x7f2310069e70 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 msgr2=0x7f23100ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f23100ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 >> v1:172.21.15.114:6789/0 conn(0x7f231015c500 legacy=0x7f231006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231006d0a0 con 0x7f2310069e70 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f23080019f0 con 0x7f2310069e70 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 >> v1:172.21.15.104:6789/0 conn(0x7f2310069e70 legacy=0x7f23100c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f8003bc0 0x7f22f8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f22f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f22f8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231006e850 con 0x7f22f8004d90 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f800bac0 con 0x7f22f8003bc0 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f800bc30 con 0x7f23100cf030 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f22f8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230e59c700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f8003bc0 0x7f22f8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230dd9b700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f22f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 -- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 msgr2=0x7f22f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23100cf030 0x7f22f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 -- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f8003bc0 msgr2=0x7f22f8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f8003bc0 0x7f22f8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 -- 172.21.15.104:0/2338386913 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80035b0 con 0x7f22f8004d90 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230ed9d700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f22f8005200 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f230800a7a0 tx=0x7f22f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75f29abf3d7f9e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080072d0 con 0x7f22f8004d90 2023-06-10T18:56:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308007430 con 0x7f22f8004d90 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.094+0000 7f230d59a700 1 -- 172.21.15.104:0/2338386913 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080077a0 con 0x7f22f8004d90 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 -- 172.21.15.104:0/2338386913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 msgr2=0x7f22f8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 --2- 172.21.15.104:0/2338386913 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f22f8005200 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f230800a7a0 tx=0x7f22f8005880 comp rx=0 tx=0).stop 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f230ed9d700 1 -- 172.21.15.104:0/2338386913 reap_dead start 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 -- 172.21.15.104:0/2338386913 shutdown_connections 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 -- 172.21.15.104:0/2338386913 >> 172.21.15.104:0/2338386913 conn(0x7f231006b8a0 msgr2=0x7f23100cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 -- 172.21.15.104:0/2338386913 shutdown_connections 2023-06-10T18:56:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.095+0000 7f2314d37700 1 -- 172.21.15.104:0/2338386913 wait complete. 2023-06-10T18:56:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.096+0000 7f2314d37700 1 Processor -- start 2023-06-10T18:56:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.096+0000 7f2314d37700 1 -- start start 2023-06-10T18:56:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 0x7f23101582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2310158820 0x7f2310155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101599d0 con 0x7f22f8004d90 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23101590a0 con 0x7f2310151ad0 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f2314d37700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2310159380 con 0x7f2310158820 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f230e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f230e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36912/0 (socket says 172.21.15.104:36912) 2023-06-10T18:56:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f230e59c700 1 -- 172.21.15.104:0/650265552 learned_addr learned my addr 172.21.15.104:0/650265552 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f230dd9b700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 0x7f23101582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.097+0000 7f230ed9d700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2310158820 0x7f2310155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 -- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2310158820 msgr2=0x7f2310155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2310158820 0x7f2310155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 -- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 msgr2=0x7f23101582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 0x7f23101582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 -- 172.21.15.104:0/650265552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310159660 con 0x7f22f8004d90 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230dd9b700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 0x7f23101582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f230e59c700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f230400f7e0 tx=0x7f230400fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23040052e0 con 0x7f22f8004d90 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2304005440 con 0x7f22f8004d90 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310147a30 con 0x7f22f8004d90 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23040055b0 con 0x7f22f8004d90 2023-06-10T18:56:07.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.098+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2310151f40 con 0x7f22f8004d90 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.099+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2310074050 con 0x7f22f8004d90 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.099+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2304003910 con 0x7f22f8004d90 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.100+0000 7f22feffd700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 0x7f22f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.100+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f2304089610 con 0x7f22f8004d90 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.100+0000 7f230dd9b700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 0x7f22f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.100+0000 7f230dd9b700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 0x7f22f006d190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2300009c60 tx=0x7f2300008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.104+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2304054510 con 0x7f22f8004d90 2023-06-10T18:56:07.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.331+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f231006aba0 con 0x7f22f8004d90 2023-06-10T18:56:07.778 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:56:07.772+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:56:07.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.775+0000 7f22feffd700 1 -- 172.21.15.104:0/650265552 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2304057b30 con 0x7f22f8004d90 2023-06-10T18:56:07.783 INFO:tasks.ceph.mds.c.smithi104.stderr: -3120> 2023-06-10T18:55:12.804+0000 7fc40ec1a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:56:07.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 msgr2=0x7f22f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 0x7f22f006d190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2300009c60 tx=0x7f2300008040 comp rx=0 tx=0).stop 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 msgr2=0x7f23101e82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f230400f7e0 tx=0x7f230400fba0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 shutdown_connections 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2310158820 0x7f2310155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22f8004d90 0x7f23101e82c0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f22f006ace0 0x7f22f006d190 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 --2- 172.21.15.104:0/650265552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2310151ad0 0x7f23101582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 >> 172.21.15.104:0/650265552 conn(0x7f231006b8a0 msgr2=0x7f23100af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:07.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 shutdown_connections 2023-06-10T18:56:07.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.778+0000 7f2314d37700 1 -- 172.21.15.104:0/650265552 wait complete. 2023-06-10T18:56:07.787 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:56:07.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:56:07.816 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:56:07.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.917+0000 7f84947a9700 1 Processor -- start 2023-06-10T18:56:07.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.917+0000 7f84947a9700 1 -- start start 2023-06-10T18:56:07.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f84947a9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f848c15c580 0x7f848c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f84947a9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f848c06cde0 0x7f848c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f84947a9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c06e250 con 0x7f848c06e6e0 2023-06-10T18:56:07.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f84947a9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c15c950 con 0x7f848c15c580 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f84947a9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c0b1e80 con 0x7f848c06cde0 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f8492545700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f848c15c580 0x7f848c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f8491d44700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f848c06e6e0 0x7f848c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34760/0 (socket says 172.21.15.104:34760) 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f8491d44700 1 -- 172.21.15.104:0/637213181 learned_addr learned my addr 172.21.15.104:0/637213181 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.918+0000 7f8492d46700 1 --2- 172.21.15.104:0/637213181 >> v2:172.21.15.114:3300/0 conn(0x7f848c06cde0 0x7f848c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8492545700 1 -- 172.21.15.104:0/637213181 >> v2:172.21.15.114:3300/0 conn(0x7f848c06cde0 msgr2=0x7f848c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:07.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8492545700 1 --2- 172.21.15.104:0/637213181 >> v2:172.21.15.114:3300/0 conn(0x7f848c06cde0 0x7f848c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8492545700 1 -- 172.21.15.104:0/637213181 >> v1:172.21.15.104:6789/0 conn(0x7f848c06e6e0 legacy=0x7f848c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8492545700 1 -- 172.21.15.104:0/637213181 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c06d920 con 0x7f848c15c580 2023-06-10T18:56:07.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8491543700 1 -- 172.21.15.104:0/637213181 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653374311 0 0) 0x7f848c06e250 con 0x7f848c06e6e0 2023-06-10T18:56:07.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8492545700 1 --2- 172.21.15.104:0/637213181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c0b1070 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f848c0cbc60 tx=0x7f8488009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a73a392322357c3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8491543700 1 -- 172.21.15.104:0/637213181 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488005e30 con 0x7f848c15c580 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8491543700 1 -- 172.21.15.104:0/637213181 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488006030 con 0x7f848c15c580 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.919+0000 7f8491543700 1 -- 172.21.15.104:0/637213181 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84880063c0 con 0x7f848c15c580 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 -- 172.21.15.104:0/637213181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 msgr2=0x7f848c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 --2- 172.21.15.104:0/637213181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c0b1070 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f848c0cbc60 tx=0x7f8488009b00 comp rx=0 tx=0).stop 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 -- 172.21.15.104:0/637213181 shutdown_connections 2023-06-10T18:56:07.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 --2- 172.21.15.104:0/637213181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c0b1070 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 --2- 172.21.15.104:0/637213181 >> v2:172.21.15.114:3300/0 conn(0x7f848c06cde0 0x7f848c0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 -- 172.21.15.104:0/637213181 >> 172.21.15.104:0/637213181 conn(0x7f848c06b8a0 msgr2=0x7f848c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 -- 172.21.15.104:0/637213181 shutdown_connections 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.920+0000 7f84947a9700 1 -- 172.21.15.104:0/637213181 wait complete. 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.921+0000 7f84947a9700 1 Processor -- start 2023-06-10T18:56:07.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.921+0000 7f84947a9700 1 -- start start 2023-06-10T18:56:07.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 0x7f848c0c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 0x7f848c0c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c0b3a00 con 0x7f848c15c580 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c0c3390 con 0x7f848c06e6e0 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848c0c4620 con 0x7f848c06cde0 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36940/0 (socket says 172.21.15.104:36940) 2023-06-10T18:56:07.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 -- 172.21.15.104:0/2768289863 learned_addr learned my addr 172.21.15.104:0/2768289863 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492545700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 0x7f848c0c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8491d44700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 0x7f848c0c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 -- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 msgr2=0x7f848c0c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 0x7f848c0c37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 -- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 msgr2=0x7f848c0c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 0x7f848c0c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:07.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 -- 172.21.15.104:0/2768289863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8488009430 con 0x7f848c15c580 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492545700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 0x7f848c0c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8491d44700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 0x7f848c0c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8492d46700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f848400efc0 tx=0x7f848400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400cd40 con 0x7f848c15c580 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848400f040 con 0x7f848c15c580 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c0c3d80 con 0x7f848c15c580 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84840146c0 con 0x7f848c15c580 2023-06-10T18:56:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.922+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c0c4340 con 0x7f848c15c580 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.923+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c074050 con 0x7f848c15c580 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.924+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8484014820 con 0x7f848c15c580 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.924+0000 7f8482ffd700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 0x7f847006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.924+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f848408a2c0 con 0x7f848c15c580 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.924+0000 7f8492545700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 0x7f847006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:07.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.925+0000 7f8492545700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 0x7f847006d190 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8488013040 tx=0x7f8488008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:07.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:07.928+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8484058c00 con 0x7f848c15c580 2023-06-10T18:56:08.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.158+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f848c0cb8b0 con 0x7f848c15c580 2023-06-10T18:56:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.783+0000 7f8482ffd700 1 -- 172.21.15.104:0/2768289863 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7f848400cea0 con 0x7f848c15c580 2023-06-10T18:56:08.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.786+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 msgr2=0x7f847006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.786+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 0x7f847006d190 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8488013040 tx=0x7f8488008070 comp rx=0 tx=0).stop 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.786+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 msgr2=0x7f848c158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.786+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f848400efc0 tx=0x7f848400c3a0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 shutdown_connections 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f848c15c580 0x7f848c158cb0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f848c06cde0 0x7f848c0c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847006ace0 0x7f847006d190 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 --2- 172.21.15.104:0/2768289863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848c06e6e0 0x7f848c0c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 >> 172.21.15.104:0/2768289863 conn(0x7f848c06b8a0 msgr2=0x7f848c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 shutdown_connections 2023-06-10T18:56:08.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.787+0000 7f84947a9700 1 -- 172.21.15.104:0/2768289863 wait complete. 2023-06-10T18:56:08.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:56:08.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.976+0000 7f2c2e2ff700 1 Processor -- start 2023-06-10T18:56:08.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.976+0000 7f2c2e2ff700 1 -- start start 2023-06-10T18:56:08.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2e2ff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2815c580 0x7f2c280b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2e2ff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2c2806cde0 0x7f2c280ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2e2ff700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2806e250 con 0x7f2c2806e6e0 2023-06-10T18:56:08.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2e2ff700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2815c950 con 0x7f2c2806cde0 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2e2ff700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280b1e80 con 0x7f2c2815c580 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2c2806cde0 0x7f2c280ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c277fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2c2806e6e0 0x7f2c280b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34776/0 (socket says 172.21.15.104:34776) 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c277fe700 1 -- 172.21.15.104:0/4280674134 learned_addr learned my addr 172.21.15.104:0/4280674134 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c27fff700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2815c580 0x7f2c280b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 -- 172.21.15.104:0/4280674134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2815c580 msgr2=0x7f2c280b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:08.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2815c580 0x7f2c280b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 -- 172.21.15.104:0/4280674134 >> v1:172.21.15.104:6789/0 conn(0x7f2c2806e6e0 legacy=0x7f2c280b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 -- 172.21.15.104:0/4280674134 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2806d920 con 0x7f2c2806cde0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c26ffd700 1 -- 172.21.15.104:0/4280674134 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095964772 0 0) 0x7f2c2806e250 con 0x7f2c2806e6e0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c2c89c700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2806cde0 0x7f2c280ce0d0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009d90 tx=0x7f2c2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da44652c17bce0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.977+0000 7f2c26ffd700 1 -- 172.21.15.104:0/4280674134 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000e070 con 0x7f2c2806cde0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c26ffd700 1 -- 172.21.15.104:0/4280674134 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2000bb20 con 0x7f2c2806cde0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c26ffd700 1 -- 172.21.15.104:0/4280674134 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20012430 con 0x7f2c2806cde0 2023-06-10T18:56:08.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/4280674134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2806cde0 msgr2=0x7f2c280ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2806cde0 0x7f2c280ce0d0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009d90 tx=0x7f2c2000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/4280674134 shutdown_connections 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2806cde0 0x7f2c280ce0d0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/4280674134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2815c580 0x7f2c280b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/4280674134 >> 172.21.15.104:0/4280674134 conn(0x7f2c2806b8a0 msgr2=0x7f2c280cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/4280674134 shutdown_connections 2023-06-10T18:56:08.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.978+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/4280674134 wait complete. 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.979+0000 7f2c2e2ff700 1 Processor -- start 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.979+0000 7f2c2e2ff700 1 -- start start 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 0x7f2c2815e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 0x7f2c2815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2815ea50 con 0x7f2c2815c580 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2815fa10 con 0x7f2c2806cde0 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2e2ff700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280ac360 con 0x7f2c2806e6e0 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2c89c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2c89c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36956/0 (socket says 172.21.15.104:36956) 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.980+0000 7f2c2c89c700 1 -- 172.21.15.104:0/1930448240 learned_addr learned my addr 172.21.15.104:0/1930448240 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c27fff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 0x7f2c2815e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c277fe700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 0x7f2c2815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 -- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 msgr2=0x7f2c2815f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 0x7f2c2815f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 -- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 msgr2=0x7f2c2815e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 0x7f2c2815e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 -- 172.21.15.104:0/1930448240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20009a70 con 0x7f2c2815c580 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c27fff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 0x7f2c2815e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c277fe700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 0x7f2c2815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:08.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2c89c700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000b680 tx=0x7f2c20004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20012430 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20020070 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c280ac640 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20004280 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.981+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c280acc00 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.982+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c28074050 con 0x7f2c2815c580 2023-06-10T18:56:08.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.982+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2c200043e0 con 0x7f2c2815c580 2023-06-10T18:56:08.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.983+0000 7f2c24ff9700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 0x7f2c1406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:08.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.983+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f2c20022070 con 0x7f2c2815c580 2023-06-10T18:56:08.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.983+0000 7f2c27fff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 0x7f2c1406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:08.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.984+0000 7f2c27fff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 0x7f2c1406d190 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009aa0 tx=0x7f2c1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:08.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:08.987+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2c200562d0 con 0x7f2c2815c580 2023-06-10T18:56:09.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.172+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2c28066f10 con 0x7f2c2815c580 2023-06-10T18:56:09.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:56:09.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:56:09.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.804+0000 7f2c24ff9700 1 -- 172.21.15.104:0/1930448240 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2c200598f0 con 0x7f2c2815c580 2023-06-10T18:56:09.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 msgr2=0x7f2c1406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 0x7f2c1406d190 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009aa0 tx=0x7f2c1c008040 comp rx=0 tx=0).stop 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 msgr2=0x7f2c280cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000b680 tx=0x7f2c20004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 shutdown_connections 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2c1406ace0 0x7f2c1406d190 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c2806cde0 0x7f2c2815e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c2815c580 0x7f2c280cf0e0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 --2- 172.21.15.104:0/1930448240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2c2806e6e0 0x7f2c2815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 >> 172.21.15.104:0/1930448240 conn(0x7f2c2806b8a0 msgr2=0x7f2c28157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 shutdown_connections 2023-06-10T18:56:09.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.807+0000 7f2c2e2ff700 1 -- 172.21.15.104:0/1930448240 wait complete. 2023-06-10T18:56:09.816 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:56:09.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:56:09.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.948+0000 7f719a407700 1 Processor -- start 2023-06-10T18:56:09.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.948+0000 7f719a407700 1 -- start start 2023-06-10T18:56:09.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f719a407700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 0x7f719406d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f719a407700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f71940cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f719a407700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71940d0480 con 0x7f71940aec40 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f719a407700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719406d7a0 con 0x7f719406cde0 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f719a407700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719415dd00 con 0x7f71940ca850 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f7193fff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f71940aec40 0x7f7194069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:40636/0 (socket says 172.21.15.104:40636) 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f7193fff700 1 -- 172.21.15.104:0/2709787429 learned_addr learned my addr 172.21.15.104:0/2709787429 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 --2- 172.21.15.104:0/2709787429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f71940cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71937fe700 1 --2- 172.21.15.104:0/2709787429 >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 0x7f719406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f7192ffd700 1 -- 172.21.15.104:0/2709787429 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158855860 0 0) 0x7f71940d0480 con 0x7f71940aec40 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f7192ffd700 1 -- 172.21.15.104:0/2709787429 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f717c0035b0 con 0x7f71940aec40 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 -- 172.21.15.104:0/2709787429 >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 msgr2=0x7f719406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 --2- 172.21.15.104:0/2709787429 >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 0x7f719406d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 -- 172.21.15.104:0/2709787429 >> v1:172.21.15.114:6789/0 conn(0x7f71940aec40 legacy=0x7f7194069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 -- 172.21.15.104:0/2709787429 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719415a610 con 0x7f71940ca850 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71937fe700 1 --2- 172.21.15.104:0/2709787429 >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 0x7f719406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:09.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.949+0000 7f71989a4700 1 --2- 172.21.15.104:0/2709787429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f71940cacd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f718c009d90 tx=0x7f718c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d59bb0666b00bb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f7192ffd700 1 -- 172.21.15.104:0/2709787429 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c00e070 con 0x7f71940ca850 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f7192ffd700 1 -- 172.21.15.104:0/2709787429 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c00bb20 con 0x7f71940ca850 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f7192ffd700 1 -- 172.21.15.104:0/2709787429 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c012430 con 0x7f71940ca850 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 -- 172.21.15.104:0/2709787429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 msgr2=0x7f71940cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 --2- 172.21.15.104:0/2709787429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f71940cacd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f718c009d90 tx=0x7f718c00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 -- 172.21.15.104:0/2709787429 shutdown_connections 2023-06-10T18:56:09.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 --2- 172.21.15.104:0/2709787429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f71940cacd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 --2- 172.21.15.104:0/2709787429 >> v2:172.21.15.114:3300/0 conn(0x7f719406cde0 0x7f719406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 -- 172.21.15.104:0/2709787429 >> 172.21.15.104:0/2709787429 conn(0x7f719406b8a0 msgr2=0x7f71940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 -- 172.21.15.104:0/2709787429 shutdown_connections 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.950+0000 7f719a407700 1 -- 172.21.15.104:0/2709787429 wait complete. 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.951+0000 7f719a407700 1 Processor -- start 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 -- start start 2023-06-10T18:56:09.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 0x7f71940c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f7194154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71940af700 con 0x7f719406cde0 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71940c8900 con 0x7f71940aec40 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f719a407700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71941553d0 con 0x7f71940ca850 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f7193fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f7193fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36968/0 (socket says 172.21.15.104:36968) 2023-06-10T18:56:09.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.952+0000 7f7193fff700 1 -- 172.21.15.104:0/1698919877 learned_addr learned my addr 172.21.15.104:0/1698919877 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f71989a4700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f7194154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f71937fe700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 0x7f71940c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 -- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 msgr2=0x7f7194154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f7194154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 -- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 msgr2=0x7f71940c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 0x7f71940c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:09.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 -- 172.21.15.104:0/1698919877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880094a0 con 0x7f719406cde0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f71937fe700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 0x7f71940c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7193fff700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f718400c700 tx=0x7f718400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184005dc0 con 0x7f719406cde0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71840070d0 con 0x7f719406cde0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c009a70 con 0x7f719406cde0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184007460 con 0x7f719406cde0 2023-06-10T18:56:09.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.953+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71940af0b0 con 0x7f719406cde0 2023-06-10T18:56:09.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.954+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7194074050 con 0x7f719406cde0 2023-06-10T18:56:09.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.954+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f71840075c0 con 0x7f719406cde0 2023-06-10T18:56:09.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.955+0000 7f7190ff9700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 0x7f718006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:09.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.955+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6131+0+0 (secure 0 0 0) 0x7f7184088b40 con 0x7f719406cde0 2023-06-10T18:56:09.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.955+0000 7f71937fe700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 0x7f718006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:09.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.956+0000 7f71937fe700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 0x7f718006d190 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f71940ceb00 tx=0x7f7188008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:09.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:09.959+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7184053b00 con 0x7f719406cde0 2023-06-10T18:56:10.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.146+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f719415e500 con 0x7f719406cde0 2023-06-10T18:56:10.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.810+0000 7f7190ff9700 1 -- 172.21.15.104:0/1698919877 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7f71840170d0 con 0x7f719406cde0 2023-06-10T18:56:10.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 msgr2=0x7f718006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:10.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 0x7f718006d190 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f71940ceb00 tx=0x7f7188008040 comp rx=0 tx=0).stop 2023-06-10T18:56:10.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 msgr2=0x7f71940c2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:10.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f718400c700 tx=0x7f718400cac0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 shutdown_connections 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71940ca850 0x7f7194154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719406cde0 0x7f71940c2490 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f718006ace0 0x7f718006d190 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 --2- 172.21.15.104:0/1698919877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71940aec40 0x7f71940c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 >> 172.21.15.104:0/1698919877 conn(0x7f719406b8a0 msgr2=0x7f71940cc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:10.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 shutdown_connections 2023-06-10T18:56:10.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.814+0000 7f719a407700 1 -- 172.21.15.104:0/1698919877 wait complete. 2023-06-10T18:56:10.822 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:56:10.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:56:10.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.955+0000 7fea71d66700 1 Processor -- start 2023-06-10T18:56:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.955+0000 7fea71d66700 1 -- start start 2023-06-10T18:56:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea6c06e6e0 0x7fea6c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fea6c06cde0 0x7fea6c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c06e250 con 0x7fea6c06cde0 2023-06-10T18:56:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c15c950 con 0x7fea6c06e6e0 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea71d66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0b1e80 con 0x7fea6c15c580 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea6bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fea6c06cde0 0x7fea6c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea6bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fea6c06cde0 0x7fea6c069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36980/0 (socket says 172.21.15.104:36980) 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.956+0000 7fea6bfff700 1 -- 172.21.15.104:0/3731915695 learned_addr learned my addr 172.21.15.104:0/3731915695 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6affd700 1 --2- 172.21.15.104:0/3731915695 >> v2:172.21.15.114:3300/0 conn(0x7fea6c06e6e0 0x7fea6c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6b7fe700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 -- 172.21.15.104:0/3731915695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 msgr2=0x7fea6c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 -- 172.21.15.104:0/3731915695 >> v2:172.21.15.114:3300/0 conn(0x7fea6c06e6e0 msgr2=0x7fea6c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 --2- 172.21.15.104:0/3731915695 >> v2:172.21.15.114:3300/0 conn(0x7fea6c06e6e0 0x7fea6c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 -- 172.21.15.104:0/3731915695 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c06d920 con 0x7fea6c06cde0 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6bfff700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06cde0 0x7fea6c069e70 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fea6c070290 tx=0x7fea6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51836ab8cf9adeb0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6a7fc700 1 -- 172.21.15.104:0/3731915695 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400e070 con 0x7fea6c06cde0 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6a7fc700 1 -- 172.21.15.104:0/3731915695 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6400b9c0 con 0x7fea6c06cde0 2023-06-10T18:56:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.957+0000 7fea6a7fc700 1 -- 172.21.15.104:0/3731915695 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400bd70 con 0x7fea6c06cde0 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 -- 172.21.15.104:0/3731915695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06cde0 msgr2=0x7fea6c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06cde0 0x7fea6c069e70 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fea6c070290 tx=0x7fea6400b330 comp rx=0 tx=0).stop 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 -- 172.21.15.104:0/3731915695 shutdown_connections 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06cde0 0x7fea6c069e70 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 --2- 172.21.15.104:0/3731915695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 --2- 172.21.15.104:0/3731915695 >> v2:172.21.15.114:3300/0 conn(0x7fea6c06e6e0 0x7fea6c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 -- 172.21.15.104:0/3731915695 >> 172.21.15.104:0/3731915695 conn(0x7fea6c06b8a0 msgr2=0x7fea6c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 -- 172.21.15.104:0/3731915695 shutdown_connections 2023-06-10T18:56:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.958+0000 7fea71d66700 1 -- 172.21.15.104:0/3731915695 wait complete. 2023-06-10T18:56:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.959+0000 7fea71d66700 1 Processor -- start 2023-06-10T18:56:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.959+0000 7fea71d66700 1 -- start start 2023-06-10T18:56:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea6c06cde0 0x7fea6c155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1582f0 con 0x7fea6c06e6e0 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c15bc80 con 0x7fea6c06cde0 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c15abe0 con 0x7fea6c15c580 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36984/0 (socket says 172.21.15.104:36984) 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 -- 172.21.15.104:0/3894499839 learned_addr learned my addr 172.21.15.104:0/3894499839 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6b7fe700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea6c06cde0 0x7fea6c155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6bfff700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 -- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 msgr2=0x7fea6c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 -- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea6c06cde0 msgr2=0x7fea6c155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea6c06cde0 0x7fea6c155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 -- 172.21.15.104:0/3894499839 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64009b30 con 0x7fea6c06e6e0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea6affd700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00e810 tx=0x7fea5c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00c070 con 0x7fea6c06e6e0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c013610 con 0x7fea6c06e6e0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c15aec0 con 0x7fea6c06e6e0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.960+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c0139a0 con 0x7fea6c06e6e0 2023-06-10T18:56:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.961+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c15b480 con 0x7fea6c06e6e0 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.961+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c074050 con 0x7fea6c06e6e0 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.962+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fea5c013b00 con 0x7fea6c06e6e0 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.962+0000 7fea53fff700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 0x7fea5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.962+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fea5c0890d0 con 0x7fea6c06e6e0 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.962+0000 7fea6b7fe700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 0x7fea5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.963+0000 7fea6b7fe700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 0x7fea5806d190 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fea60006fd0 tx=0x7fea60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:10.966+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fea5c054290 con 0x7fea6c06e6e0 2023-06-10T18:56:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.152+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fea6c0b5450 con 0x7fea6c06e6e0 2023-06-10T18:56:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.156+0000 7fea53fff700 1 -- 172.21.15.104:0/3894499839 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fea5c0578b0 con 0x7fea6c06e6e0 2023-06-10T18:56:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 msgr2=0x7fea5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 0x7fea5806d190 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fea60006fd0 tx=0x7fea60008040 comp rx=0 tx=0).stop 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 msgr2=0x7fea6c157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00e810 tx=0x7fea5c00eb20 comp rx=0 tx=0).stop 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 shutdown_connections 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fea5806ace0 0x7fea5806d190 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea6c06cde0 0x7fea6c155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea6c15c580 0x7fea6c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 --2- 172.21.15.104:0/3894499839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea6c06e6e0 0x7fea6c157db0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 >> 172.21.15.104:0/3894499839 conn(0x7fea6c06b8a0 msgr2=0x7fea6c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 shutdown_connections 2023-06-10T18:56:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.158+0000 7fea71d66700 1 -- 172.21.15.104:0/3894499839 wait complete. 2023-06-10T18:56:11.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale' 2023-06-10T18:56:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.293+0000 7f658f051700 1 Processor -- start 2023-06-10T18:56:11.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.293+0000 7f658f051700 1 -- start start 2023-06-10T18:56:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658f051700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e4b0 0x7f658806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658f051700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6588069e70 0x7f65880ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658f051700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658806d290 con 0x7f658806e0e0 2023-06-10T18:56:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658f051700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588071640 con 0x7f6588069e70 2023-06-10T18:56:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658f051700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65880aef00 con 0x7f658806e4b0 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6588069e70 0x7f65880ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658cded700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f658806e0e0 0x7f65880c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34818/0 (socket says 172.21.15.104:34818) 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658cded700 1 -- 172.21.15.104:0/311213555 learned_addr learned my addr 172.21.15.104:0/311213555 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f6587fff700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e4b0 0x7f658806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 -- 172.21.15.104:0/311213555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e4b0 msgr2=0x7f658806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e4b0 0x7f658806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 -- 172.21.15.104:0/311213555 >> v1:172.21.15.104:6789/0 conn(0x7f658806e0e0 legacy=0x7f65880c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 -- 172.21.15.104:0/311213555 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880ca2d0 con 0x7f6588069e70 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f65877fe700 1 -- 172.21.15.104:0/311213555 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293322098 0 0) 0x7f658806d290 con 0x7f658806e0e0 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f658d5ee700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f65880ce9f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f6580009d90 tx=0x7f658000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d32f022c19d1bc9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f65877fe700 1 -- 172.21.15.104:0/311213555 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658000e070 con 0x7f6588069e70 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f65877fe700 1 -- 172.21.15.104:0/311213555 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658000bbd0 con 0x7f6588069e70 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.294+0000 7f65877fe700 1 -- 172.21.15.104:0/311213555 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580012430 con 0x7f6588069e70 2023-06-10T18:56:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 -- 172.21.15.104:0/311213555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 msgr2=0x7f65880ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f65880ce9f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f6580009d90 tx=0x7f658000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 -- 172.21.15.104:0/311213555 shutdown_connections 2023-06-10T18:56:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f65880ce9f0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 --2- 172.21.15.104:0/311213555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e4b0 0x7f658806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 -- 172.21.15.104:0/311213555 >> 172.21.15.104:0/311213555 conn(0x7f658806b8a0 msgr2=0x7f65880cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 -- 172.21.15.104:0/311213555 shutdown_connections 2023-06-10T18:56:11.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.295+0000 7f658f051700 1 -- 172.21.15.104:0/311213555 wait complete. 2023-06-10T18:56:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.296+0000 7f658f051700 1 Processor -- start 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.296+0000 7f658f051700 1 -- start start 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 0x7f658815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 0x7f65880b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880afb60 con 0x7f6588069e70 2023-06-10T18:56:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880b1810 con 0x7f658806e4b0 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658f051700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65881593e0 con 0x7f658806e0e0 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36996/0 (socket says 172.21.15.104:36996) 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 -- 172.21.15.104:0/4120510426 learned_addr learned my addr 172.21.15.104:0/4120510426 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 -- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 msgr2=0x7f658815a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658d5ee700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 0x7f65880b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f6587fff700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 0x7f658815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 0x7f658815a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 -- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 msgr2=0x7f65880b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 0x7f65880b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658cded700 1 -- 172.21.15.104:0/4120510426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6580009a70 con 0x7f6588069e70 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f658d5ee700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 0x7f65880b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.297+0000 7f6587fff700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 0x7f658815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f658cded700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c4d0 tx=0x7f657c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c0074e0 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c007660 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65880af1c0 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c005a70 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.298+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65880af780 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.299+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65880ccef0 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.299+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f657c005bd0 con 0x7f6588069e70 2023-06-10T18:56:11.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.299+0000 7f65857fa700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 0x7f656c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.300+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f657c088710 con 0x7f6588069e70 2023-06-10T18:56:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.300+0000 7f6587fff700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 0x7f656c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.300+0000 7f6587fff700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 0x7f656c06d190 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f65880b5590 tx=0x7f6574008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.304+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f657c0539e0 con 0x7f6588069e70 2023-06-10T18:56:11.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.567+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale"]} v 0) v1 -- 0x7f6588074050 con 0x7f6588069e70 2023-06-10T18:56:11.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.809+0000 7f65857fa700 1 -- 172.21.15.104:0/4120510426 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_dont_mark_unresponsive_client_stale"]}]=0 v566) v1 ==== 172+0+0 (secure 0 0 0) 0x7f657c057000 con 0x7f6588069e70 2023-06-10T18:56:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 msgr2=0x7f656c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 0x7f656c06d190 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f65880b5590 tx=0x7f6574008040 comp rx=0 tx=0).stop 2023-06-10T18:56:11.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 msgr2=0x7f6588159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c4d0 tx=0x7f657c00c890 comp rx=0 tx=0).stop 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 shutdown_connections 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f656c06ace0 0x7f656c06d190 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658806e4b0 0x7f65880b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6588069e70 0x7f6588159bd0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 --2- 172.21.15.104:0/4120510426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f658806e0e0 0x7f658815a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 >> 172.21.15.104:0/4120510426 conn(0x7f658806b8a0 msgr2=0x7f65880c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 shutdown_connections 2023-06-10T18:56:11.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.811+0000 7f658f051700 1 -- 172.21.15.104:0/4120510426 wait complete. 2023-06-10T18:56:11.830 INFO:tasks.cephfs_test_runner:test_dont_mark_unresponsive_client_stale (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:56:11.830 INFO:tasks.cephfs_test_runner:Starting test: test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:56:11.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps' 2023-06-10T18:56:11.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.947+0000 7fdb1c910700 1 Processor -- start 2023-06-10T18:56:11.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.947+0000 7fdb1c910700 1 -- start start 2023-06-10T18:56:11.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1c910700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb180cb300 0x7fdb18163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1c910700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180c9e60 con 0x7fdb180aeaa0 2023-06-10T18:56:11.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1c910700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180aee70 con 0x7fdb180cd850 2023-06-10T18:56:11.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1c910700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb180c9bc0 con 0x7fdb180cb300 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb16d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb180cb300 0x7fdb18163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdb180aeaa0 0x7fdb180cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34832/0 (socket says 172.21.15.104:34832) 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1659c700 1 -- 172.21.15.104:0/664821467 learned_addr learned my addr 172.21.15.104:0/664821467 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb16d9d700 1 -- 172.21.15.104:0/664821467 >> v1:172.21.15.114:6789/0 conn(0x7fdb180cd850 legacy=0x7fdb1806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb16d9d700 1 -- 172.21.15.104:0/664821467 >> v1:172.21.15.104:6789/0 conn(0x7fdb180aeaa0 legacy=0x7fdb180cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb16d9d700 1 -- 172.21.15.104:0/664821467 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1806e540 con 0x7fdb180cb300 2023-06-10T18:56:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.948+0000 7fdb1559a700 1 -- 172.21.15.104:0/664821467 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563350846 0 0) 0x7fdb180c9e60 con 0x7fdb180aeaa0 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb16d9d700 1 --2- 172.21.15.104:0/664821467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cb300 0x7fdb18163c30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009d90 tx=0x7fdb1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761fc5866744adca server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1559a700 1 -- 172.21.15.104:0/664821467 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1000e070 con 0x7fdb180cb300 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1559a700 1 -- 172.21.15.104:0/664821467 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1000bb90 con 0x7fdb180cb300 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1559a700 1 -- 172.21.15.104:0/664821467 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10003a20 con 0x7fdb180cb300 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 -- 172.21.15.104:0/664821467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cb300 msgr2=0x7fdb18163c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 --2- 172.21.15.104:0/664821467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cb300 0x7fdb18163c30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009d90 tx=0x7fdb1000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 -- 172.21.15.104:0/664821467 shutdown_connections 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 --2- 172.21.15.104:0/664821467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cb300 0x7fdb18163c30 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 -- 172.21.15.104:0/664821467 >> 172.21.15.104:0/664821467 conn(0x7fdb1806b8a0 msgr2=0x7fdb180cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 -- 172.21.15.104:0/664821467 shutdown_connections 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.949+0000 7fdb1c910700 1 -- 172.21.15.104:0/664821467 wait complete. 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 Processor -- start 2023-06-10T18:56:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 -- start start 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 0x7fdb18151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 0x7fdb180afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18151350 con 0x7fdb180cd850 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1815a0d0 con 0x7fdb180cb300 2023-06-10T18:56:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb1c910700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18151ab0 con 0x7fdb180aeaa0 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.951+0000 7fdb16d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37012/0 (socket says 172.21.15.104:37012) 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 -- 172.21.15.104:0/2044061829 learned_addr learned my addr 172.21.15.104:0/2044061829 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb15d9b700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 0x7fdb180afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb1659c700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 0x7fdb18151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 -- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 msgr2=0x7fdb18151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 0x7fdb18151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 -- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 msgr2=0x7fdb180afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 0x7fdb180afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 -- 172.21.15.104:0/2044061829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb10009a70 con 0x7fdb180cd850 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb15d9b700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 0x7fdb180afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb1659c700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 0x7fdb18151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb16d9d700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fdb1806cc20 tx=0x7fdb10004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10012630 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb100127b0 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10012b60 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb180b2430 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.952+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb18151e00 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.953+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb18074050 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.953+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdb10012cc0 con 0x7fdb180cd850 2023-06-10T18:56:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.954+0000 7fdb06ffd700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 0x7fdaf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.954+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fdb1008b530 con 0x7fdb180cd850 2023-06-10T18:56:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.954+0000 7fdb1659c700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 0x7fdaf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.955+0000 7fdb1659c700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 0x7fdaf806d190 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c007fd0 tx=0x7fdb0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:11.958+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdb1005a290 con 0x7fdb180cd850 2023-06-10T18:56:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.226+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps"]} v 0) v1 -- 0x7fdb180723f0 con 0x7fdb180cd850 2023-06-10T18:56:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.810+0000 7fdb06ffd700 1 -- 172.21.15.104:0/2044061829 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps"]}]=0 v567) v1 ==== 152+0+0 (secure 0 0 0) 0x7fdb10059e20 con 0x7fdb180cd850 2023-06-10T18:56:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 msgr2=0x7fdaf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 0x7fdaf806d190 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c007fd0 tx=0x7fdb0c009040 comp rx=0 tx=0).stop 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 msgr2=0x7fdb180aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fdb1806cc20 tx=0x7fdb10004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 shutdown_connections 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb180cd850 0x7fdb180aff40 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb180aeaa0 0x7fdb18151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdaf806ace0 0x7fdaf806d190 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 --2- 172.21.15.104:0/2044061829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb180cb300 0x7fdb180afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 >> 172.21.15.104:0/2044061829 conn(0x7fdb1806b8a0 msgr2=0x7fdb180cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 shutdown_connections 2023-06-10T18:56:12.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.813+0000 7fdb1c910700 1 -- 172.21.15.104:0/2044061829 wait complete. 2023-06-10T18:56:12.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:56:12.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.955+0000 7efc9c944700 1 Processor -- start 2023-06-10T18:56:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.955+0000 7efc9c944700 1 -- start start 2023-06-10T18:56:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc9c944700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efc9806d260 0x7efc980caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc9c944700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc980d0480 con 0x7efc9806cde0 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc9c944700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9815dd00 con 0x7efc980aec40 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc9c944700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9815a610 con 0x7efc9806d260 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc95d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efc9806cde0 0x7efc980ca850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34846/0 (socket says 172.21.15.104:34846) 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.956+0000 7efc95d9b700 1 -- 172.21.15.104:0/2499234294 learned_addr learned my addr 172.21.15.104:0/2499234294 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918850856 0 0) 0x7efc980d0480 con 0x7efc9806cde0 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc800035b0 con 0x7efc9806cde0 2023-06-10T18:56:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc96d9d700 1 --2- 172.21.15.104:0/2499234294 >> v2:172.21.15.114:3300/0 conn(0x7efc9806d260 0x7efc980caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813455382 0 0) 0x7efc800035b0 con 0x7efc9806cde0 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 >> v2:172.21.15.114:3300/0 conn(0x7efc9806d260 msgr2=0x7efc980caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 --2- 172.21.15.104:0/2499234294 >> v2:172.21.15.114:3300/0 conn(0x7efc9806d260 0x7efc980caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 >> v1:172.21.15.114:6789/0 conn(0x7efc980aec40 legacy=0x7efc98069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9806a9e0 con 0x7efc9806cde0 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc96d9d700 1 --2- 172.21.15.104:0/2499234294 >> v2:172.21.15.114:3300/0 conn(0x7efc9806d260 0x7efc980caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7efc88001670 con 0x7efc9806cde0 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 >> v1:172.21.15.104:6789/0 conn(0x7efc9806cde0 legacy=0x7efc980ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc80003a00 0x7efc80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80005200 0x7efc8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9815a610 con 0x7efc80004850 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8000c190 con 0x7efc80005200 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8000c350 con 0x7efc80003a00 2023-06-10T18:56:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc9659c700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc80003a00 0x7efc80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc96d9d700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80005200 0x7efc8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 -- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc80003a00 msgr2=0x7efc80003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc80003a00 0x7efc80003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 -- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80005200 msgr2=0x7efc8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc80005200 0x7efc8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc95d9b700 1 -- 172.21.15.104:0/2499234294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800035b0 con 0x7efc80004850 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.957+0000 7efc96d9d700 1 -- 172.21.15.104:0/2499234294 reap_dead start 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc95d9b700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc80004cc0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7efc8800c210 tx=0x7efc80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab076d963d6012d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88007c10 con 0x7efc80004850 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8800cd90 con 0x7efc80004850 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9559a700 1 -- 172.21.15.104:0/2499234294 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88005ba0 con 0x7efc80004850 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 -- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 msgr2=0x7efc80004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc80004cc0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7efc8800c210 tx=0x7efc80005880 comp rx=0 tx=0).stop 2023-06-10T18:56:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 -- 172.21.15.104:0/2499234294 shutdown_connections 2023-06-10T18:56:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 --2- 172.21.15.104:0/2499234294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc80004cc0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 -- 172.21.15.104:0/2499234294 >> 172.21.15.104:0/2499234294 conn(0x7efc9806b8a0 msgr2=0x7efc980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 -- 172.21.15.104:0/2499234294 shutdown_connections 2023-06-10T18:56:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.958+0000 7efc9c944700 1 -- 172.21.15.104:0/2499234294 wait complete. 2023-06-10T18:56:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 Processor -- start 2023-06-10T18:56:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 -- start start 2023-06-10T18:56:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 0x7efc98151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc98151660 0x7efc981e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc981e55b0 con 0x7efc80004850 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980c73a0 con 0x7efc980af660 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.960+0000 7efc9c944700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc981e5a50 con 0x7efc98151660 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37030/0 (socket says 172.21.15.104:37030) 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 -- 172.21.15.104:0/3633392949 learned_addr learned my addr 172.21.15.104:0/3633392949 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc96d9d700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc98151660 0x7efc981e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc95d9b700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 0x7efc98151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 -- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc98151660 msgr2=0x7efc981e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc98151660 0x7efc981e5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 -- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 msgr2=0x7efc98151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 0x7efc98151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 -- 172.21.15.104:0/3633392949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc980c67d0 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc95d9b700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 0x7efc98151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9659c700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00f590 tx=0x7efc8c00f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c005320 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c005480 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c006af0 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc980c6a70 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.961+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc980c7030 con 0x7efc80004850 2023-06-10T18:56:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.962+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc980c7680 con 0x7efc80004850 2023-06-10T18:56:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.962+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc8c006c50 con 0x7efc80004850 2023-06-10T18:56:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.963+0000 7efc86ffd700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.963+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5659+0+0 (secure 0 0 0) 0x7efc8c088780 con 0x7efc80004850 2023-06-10T18:56:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.963+0000 7efc95d9b700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.964+0000 7efc95d9b700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efc88001930 tx=0x7efc88007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:12.967+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc8c053930 con 0x7efc80004850 2023-06-10T18:56:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.157+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efc980caf60 con 0x7efc80004850 2023-06-10T18:56:13.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.161+0000 7efc86ffd700 1 -- 172.21.15.104:0/3633392949 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7efc8c01a0c0 con 0x7efc80004850 2023-06-10T18:56:13.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 msgr2=0x7efc7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efc88001930 tx=0x7efc88007000 comp rx=0 tx=0).stop 2023-06-10T18:56:13.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 msgr2=0x7efc980af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00f590 tx=0x7efc8c00f8a0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 shutdown_connections 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc98151660 0x7efc981e5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc80004850 0x7efc980af120 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 --2- 172.21.15.104:0/3633392949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc980af660 0x7efc98151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 >> 172.21.15.104:0/3633392949 conn(0x7efc9806b8a0 msgr2=0x7efc980ca850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:13.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 shutdown_connections 2023-06-10T18:56:13.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.163+0000 7efc9c944700 1 -- 172.21.15.104:0/3633392949 wait complete. 2023-06-10T18:56:13.181 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:56:13.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:56:13.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:56:13.280 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:56:13.281 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:13.281 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:13.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:56:13.298 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:13.299 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:13.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:56:13.318 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:13.318 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:13.354 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:56:13.355 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:13.355 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:13.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:56:13.376 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:56:13.377 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:13.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.507+0000 7fabbbd80700 1 Processor -- start 2023-06-10T18:56:13.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.507+0000 7fabbbd80700 1 -- start start 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabbbd80700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fabb40b1e30 0x7fabb40d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabbbd80700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabb40d0a30 0x7fabb406b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabbbd80700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb406bfd0 con 0x7fabb406b820 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabbbd80700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb40d0ed0 con 0x7fabb40d0a30 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabbbd80700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb406c2b0 con 0x7fabb40b1e30 2023-06-10T18:56:13.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb931b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabb40d0a30 0x7fabb406b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabba31d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fabb406b820 0x7fabb40cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34868/0 (socket says 172.21.15.104:34868) 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabba31d700 1 -- 172.21.15.104:0/1968758023 learned_addr learned my addr 172.21.15.104:0/1968758023 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb9b1c700 1 --2- 172.21.15.104:0/1968758023 >> v2:172.21.15.114:3300/0 conn(0x7fabb40b1e30 0x7fabb40d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb931b700 1 -- 172.21.15.104:0/1968758023 >> v2:172.21.15.114:3300/0 conn(0x7fabb40b1e30 msgr2=0x7fabb40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb931b700 1 --2- 172.21.15.104:0/1968758023 >> v2:172.21.15.114:3300/0 conn(0x7fabb40b1e30 0x7fabb40d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb931b700 1 -- 172.21.15.104:0/1968758023 >> v1:172.21.15.104:6789/0 conn(0x7fabb406b820 legacy=0x7fabb40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb931b700 1 -- 172.21.15.104:0/1968758023 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40ccf70 con 0x7fabb40d0a30 2023-06-10T18:56:13.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.508+0000 7fabb8b1a700 1 -- 172.21.15.104:0/1968758023 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269971159 0 0) 0x7fabb406bfd0 con 0x7fabb406b820 2023-06-10T18:56:13.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabb931b700 1 --2- 172.21.15.104:0/1968758023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb40d0a30 0x7fabb406b2e0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fabb40b5490 tx=0x7faba4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03d3cfc47ec9f0d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabb8b1a700 1 -- 172.21.15.104:0/1968758023 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4011830 con 0x7fabb40d0a30 2023-06-10T18:56:13.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabb8b1a700 1 -- 172.21.15.104:0/1968758023 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4011990 con 0x7fabb40d0a30 2023-06-10T18:56:13.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabb8b1a700 1 -- 172.21.15.104:0/1968758023 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4011d20 con 0x7fabb40d0a30 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 -- 172.21.15.104:0/1968758023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb40d0a30 msgr2=0x7fabb406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 --2- 172.21.15.104:0/1968758023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb40d0a30 0x7fabb406b2e0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fabb40b5490 tx=0x7faba4009b20 comp rx=0 tx=0).stop 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 -- 172.21.15.104:0/1968758023 shutdown_connections 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 --2- 172.21.15.104:0/1968758023 >> v2:172.21.15.114:3300/0 conn(0x7fabb40b1e30 0x7fabb40d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 --2- 172.21.15.104:0/1968758023 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb40d0a30 0x7fabb406b2e0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 -- 172.21.15.104:0/1968758023 >> 172.21.15.104:0/1968758023 conn(0x7fabb4071ec0 msgr2=0x7fabb409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 -- 172.21.15.104:0/1968758023 shutdown_connections 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.509+0000 7fabbbd80700 1 -- 172.21.15.104:0/1968758023 wait complete. 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 Processor -- start 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 -- start start 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabb40b1e30 0x7fabb40cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 0x7fabb40cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb415f190 con 0x7fabb406b820 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40cca30 con 0x7fabb40b1e30 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabbbd80700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb41f2410 con 0x7fabb40d0a30 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabb9b1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabb9b1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37044/0 (socket says 172.21.15.104:37044) 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.511+0000 7fabb9b1c700 1 -- 172.21.15.104:0/2203652531 learned_addr learned my addr 172.21.15.104:0/2203652531 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb931b700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabb40b1e30 0x7fabb40cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabba31d700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 0x7fabb40cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 -- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 msgr2=0x7fabb40cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 0x7fabb40cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 -- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabb40b1e30 msgr2=0x7fabb40cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabb40b1e30 0x7fabb40cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 -- 172.21.15.104:0/2203652531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40094a0 con 0x7fabb406b820 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabba31d700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 0x7fabb40cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabb9b1c700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d830 tx=0x7fabb000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00125f0 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb0012750 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb415e510 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0012b00 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.512+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb415ead0 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.513+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb40d1860 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.513+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fabb0012c60 con 0x7fabb406b820 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.514+0000 7fabaa7fc700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 0x7fab9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.514+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fabb0089670 con 0x7fabb406b820 2023-06-10T18:56:13.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.514+0000 7fabb931b700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 0x7fab9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.514+0000 7fabb931b700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 0x7fab9806d190 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7faba40090b0 tx=0x7faba4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.518+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fabb0054940 con 0x7fabb406b820 2023-06-10T18:56:13.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.747+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb4074050 con 0x7fabb406b820 2023-06-10T18:56:13.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.748+0000 7fabaa7fc700 1 -- 172.21.15.104:0/2203652531 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+3538 (secure 0 0 0) 0x7fabb0057f60 con 0x7fabb406b820 2023-06-10T18:56:13.754 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:13.755 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7609,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3117211025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3117211025},{"type":"v1","addr":"172.21.15.114:6836","nonce":3117211025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":7612,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/284141586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":284141586},{"type":"v1","addr":"172.21.15.114:6837","nonce":284141586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3633862391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3633862391},{"type":"v1","addr":"172.21.15.104:6834","nonce":3633862391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3265844285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3265844285},{"type":"v1","addr":"172.21.15.104:6835","nonce":3265844285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[]} 2023-06-10T18:56:13.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 msgr2=0x7fab9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 0x7fab9806d190 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7faba40090b0 tx=0x7faba4008030 comp rx=0 tx=0).stop 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 msgr2=0x7fabb41f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d830 tx=0x7fabb000dbf0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 shutdown_connections 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabb40d0a30 0x7fabb40cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabb406b820 0x7fabb41f1ed0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fab9806ace0 0x7fab9806d190 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 --2- 172.21.15.104:0/2203652531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabb40b1e30 0x7fabb40cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 >> 172.21.15.104:0/2203652531 conn(0x7fabb4071ec0 msgr2=0x7fabb40b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 shutdown_connections 2023-06-10T18:56:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.750+0000 7fabbbd80700 1 -- 172.21.15.104:0/2203652531 wait complete. 2023-06-10T18:56:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 74 2023-06-10T18:56:13.768 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:56:13.768 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:56:13.768 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:56:13.769 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:56:13.769 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:56:13.769 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:56:13.770 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:56:13.770 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:56:13.771 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:56:13.771 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:56:13.771 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:56:13.771 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:56:13.771 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:56:13.772 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:56:13.772 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:56:13.772 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:56:13.777 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:56:13.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:56:13.779 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:56:13.779 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:56:13.782 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:56:13.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:56:13.783 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:56:13.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:56:13.820 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:56:13.821 DEBUG:teuthology.parallel:result is None 2023-06-10T18:56:13.822 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:56:13.822 DEBUG:teuthology.parallel:result is None 2023-06-10T18:56:13.824 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:56:13.825 DEBUG:teuthology.parallel:result is None 2023-06-10T18:56:13.825 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:56:13.825 DEBUG:teuthology.parallel:result is None 2023-06-10T18:56:13.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:56:13.898 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:56:13.899 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:56:13.899 INFO:tasks.ceph.mds.c.smithi104.stderr:2023-06-10T18:56:13.893+0000 7f231c091700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-06-10T18:56:13.909 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:56:13.910 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:56:13.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.949+0000 7faf93d99700 1 Processor -- start 2023-06-10T18:56:13.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.950+0000 7faf93d99700 1 -- start start 2023-06-10T18:56:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.950+0000 7faf93d99700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faf8c15bc80 0x7faf8c070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.950+0000 7faf93d99700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.950+0000 7faf93d99700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf8c0c8c40 0x7faf8c0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.950+0000 7faf93d99700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c15c050 con 0x7faf8c0c8c40 2023-06-10T18:56:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf93d99700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0ceea0 con 0x7faf8c15bc80 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf93d99700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0c9970 con 0x7faf8c0ce510 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf8c0c8c40 0x7faf8c0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7faf8c0c8c40 0x7faf8c0c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37082/0 (socket says 172.21.15.104:37082) 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 -- 172.21.15.104:0/3859702087 learned_addr learned my addr 172.21.15.104:0/3859702087 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf91b35700 1 --2- 172.21.15.104:0/3859702087 >> v2:172.21.15.114:3300/0 conn(0x7faf8c15bc80 0x7faf8c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf91334700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 -- 172.21.15.104:0/3859702087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 msgr2=0x7faf8c0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 -- 172.21.15.104:0/3859702087 >> v2:172.21.15.114:3300/0 conn(0x7faf8c15bc80 msgr2=0x7faf8c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 --2- 172.21.15.104:0/3859702087 >> v2:172.21.15.114:3300/0 conn(0x7faf8c15bc80 0x7faf8c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 -- 172.21.15.104:0/3859702087 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0ccb70 con 0x7faf8c0c8c40 2023-06-10T18:56:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf92336700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c90c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0cc710 tx=0x7faf8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e82ef3d12476a4b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf90b33700 1 -- 172.21.15.104:0/3859702087 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800e070 con 0x7faf8c0c8c40 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf90b33700 1 -- 172.21.15.104:0/3859702087 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8800b9c0 con 0x7faf8c0c8c40 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.951+0000 7faf90b33700 1 -- 172.21.15.104:0/3859702087 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800bd70 con 0x7faf8c0c8c40 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 -- 172.21.15.104:0/3859702087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c0c8c40 msgr2=0x7faf8c0c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c90c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0cc710 tx=0x7faf8800b330 comp rx=0 tx=0).stop 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 -- 172.21.15.104:0/3859702087 shutdown_connections 2023-06-10T18:56:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c90c0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 --2- 172.21.15.104:0/3859702087 >> v2:172.21.15.114:3300/0 conn(0x7faf8c15bc80 0x7faf8c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 --2- 172.21.15.104:0/3859702087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 -- 172.21.15.104:0/3859702087 >> 172.21.15.104:0/3859702087 conn(0x7faf8c06ab00 msgr2=0x7faf8c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 -- 172.21.15.104:0/3859702087 shutdown_connections 2023-06-10T18:56:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.952+0000 7faf93d99700 1 -- 172.21.15.104:0/3859702087 wait complete. 2023-06-10T18:56:13.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.953+0000 7faf93d99700 1 Processor -- start 2023-06-10T18:56:13.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.953+0000 7faf93d99700 1 -- start start 2023-06-10T18:56:13.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0c3bb0 con 0x7faf8c15bc80 2023-06-10T18:56:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c156e20 con 0x7faf8c0c8c40 2023-06-10T18:56:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c1f40d0 con 0x7faf8c0ce510 2023-06-10T18:56:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37088/0 (socket says 172.21.15.104:37088) 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 -- 172.21.15.104:0/3774548690 learned_addr learned my addr 172.21.15.104:0/3774548690 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf91b35700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf91334700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 -- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 msgr2=0x7faf8c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 -- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 msgr2=0x7faf8c0c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 -- 172.21.15.104:0/3774548690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88009b30 con 0x7faf8c15bc80 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf91b35700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf91334700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf92336700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7faf8c06fc10 tx=0x7faf88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88004250 con 0x7faf8c15bc80 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf880043d0 con 0x7faf8c15bc80 2023-06-10T18:56:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c1f43b0 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.954+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88004780 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.955+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c0b1690 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.955+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c074050 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.956+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faf8801b430 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.956+0000 7faf827fc700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.956+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5659+0+0 (secure 0 0 0) 0x7faf8808a4b0 con 0x7faf8c15bc80 2023-06-10T18:56:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.956+0000 7faf91b35700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:13.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.957+0000 7faf91b35700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7faf84009a00 tx=0x7faf84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:13.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:13.961+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faf880556f0 con 0x7faf8c15bc80 2023-06-10T18:56:14.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:14.164+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faf8c0cc330 con 0x7faf8c15bc80 2023-06-10T18:56:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.164+0000 7faf827fc700 1 -- 172.21.15.104:0/3774548690 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf88058d10 con 0x7faf8c15bc80 2023-06-10T18:56:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 msgr2=0x7faf7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7faf84009a00 tx=0x7faf84008040 comp rx=0 tx=0).stop 2023-06-10T18:56:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 msgr2=0x7faf8c1f3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7faf8c06fc10 tx=0x7faf88004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 shutdown_connections 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7faf7006ace0 0x7faf7006d190 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf8c0c8c40 0x7faf8c0c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf8c15bc80 0x7faf8c1f3560 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 --2- 172.21.15.104:0/3774548690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf8c0ce510 0x7faf8c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 >> 172.21.15.104:0/3774548690 conn(0x7faf8c06ab00 msgr2=0x7faf8c0b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 shutdown_connections 2023-06-10T18:56:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.167+0000 7faf93d99700 1 -- 172.21.15.104:0/3774548690 wait complete. 2023-06-10T18:56:15.175 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:56:15.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:56:15.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.307+0000 7f13a0ffb700 1 Processor -- start 2023-06-10T18:56:15.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.307+0000 7f13a0ffb700 1 -- start start 2023-06-10T18:56:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f13a0ffb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f139c0c93e0 0x7f139c0717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f13a0ffb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c06b2e0 con 0x7f139c15a0b0 2023-06-10T18:56:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f13a0ffb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c15e150 con 0x7f139c071130 2023-06-10T18:56:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f13a0ffb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139c15db40 con 0x7f139c0c93e0 2023-06-10T18:56:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139a59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f139c15a0b0 0x7f139c070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34924/0 (socket says 172.21.15.104:34924) 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139a59c700 1 -- 172.21.15.104:0/1970986321 learned_addr learned my addr 172.21.15.104:0/1970986321 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> v2:172.21.15.114:3300/0 conn(0x7f139c0c93e0 0x7f139c0717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744539952 0 0) 0x7f139c06b2e0 con 0x7f139c15a0b0 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13840035b0 con 0x7f139c15a0b0 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2148783529 0 0) 0x7f13840035b0 con 0x7f139c15a0b0 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 >> v2:172.21.15.114:3300/0 conn(0x7f139c0c93e0 msgr2=0x7f139c0717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 --2- 172.21.15.104:0/1970986321 >> v2:172.21.15.114:3300/0 conn(0x7f139c0c93e0 0x7f139c0717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 >> v1:172.21.15.114:6789/0 conn(0x7f139c071130 legacy=0x7f139c15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c06b530 con 0x7f139c15a0b0 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f1390001670 con 0x7f139c15a0b0 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> v2:172.21.15.114:3300/0 conn(0x7f139c0c93e0 0x7f139c0717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 >> v1:172.21.15.104:6789/0 conn(0x7f139c15a0b0 legacy=0x7f139c070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1384003a00 0x7f1384003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384004850 0x7f1384004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 0x7f138400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c15db40 con 0x7f1384005200 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138400c080 con 0x7f1384004850 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138400c240 con 0x7f1384003a00 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.308+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 0x7f138400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139a59c700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1384003a00 0x7f1384003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f1399d9b700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384004850 0x7f1384004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 -- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1384003a00 msgr2=0x7f1384003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1384003a00 0x7f1384003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 -- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384004850 msgr2=0x7f1384004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384004850 0x7f1384004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 -- 172.21.15.104:0/1970986321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13840035b0 con 0x7f1384005200 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f1399d9b700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384004850 0x7f1384004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139a59c700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1384003a00 0x7f1384003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 -- 172.21.15.104:0/1970986321 reap_dead start 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 0x7f138400ba10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f139400e320 tx=0x7f1384005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7085f88b151818a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139400edc0 con 0x7f1384005200 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139400f040 con 0x7f1384005200 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f139959a700 1 -- 172.21.15.104:0/1970986321 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394003c60 con 0x7f1384005200 2023-06-10T18:56:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 msgr2=0x7f138400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.309+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 0x7f138400ba10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f139400e320 tx=0x7f1384005880 comp rx=0 tx=0).stop 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.310+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1970986321 shutdown_connections 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.310+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1970986321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1384005200 0x7f138400ba10 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.310+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1970986321 >> 172.21.15.104:0/1970986321 conn(0x7f139c0b1610 msgr2=0x7f139c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.310+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1970986321 shutdown_connections 2023-06-10T18:56:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.310+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1970986321 wait complete. 2023-06-10T18:56:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.311+0000 7f13a0ffb700 1 Processor -- start 2023-06-10T18:56:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.311+0000 7f13a0ffb700 1 -- start start 2023-06-10T18:56:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 0x7f139c0c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f139c15ca40 0x7f139c0adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0c8850 con 0x7f139c15ceb0 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0c7270 con 0x7f1384005200 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f13a0ffb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0c7550 con 0x7f139c15ca40 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f139ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f139ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37110/0 (socket says 172.21.15.104:37110) 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.312+0000 7f139ad9d700 1 -- 172.21.15.104:0/1626941450 learned_addr learned my addr 172.21.15.104:0/1626941450 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f1399d9b700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f139c15ca40 0x7f139c0adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139a59c700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 0x7f139c0c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 -- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f139c15ca40 msgr2=0x7f139c0adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f139c15ca40 0x7f139c0adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 -- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 msgr2=0x7f139c0c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 0x7f139c0c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 -- 172.21.15.104:0/1626941450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0c7830 con 0x7f139c15ceb0 2023-06-10T18:56:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139a59c700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 0x7f139c0c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f139ad9d700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f139c15b620 tx=0x7f1394009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394004020 con 0x7f139c15ceb0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1394004180 con 0x7f139c15ceb0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c0c7b10 con 0x7f139c15ceb0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394004530 con 0x7f139c15ceb0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.313+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c0c6eb0 con 0x7f139c15ceb0 2023-06-10T18:56:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.314+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c0c7040 con 0x7f139c15ceb0 2023-06-10T18:56:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.314+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1394023070 con 0x7f139c15ceb0 2023-06-10T18:56:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.315+0000 7f138affd700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 0x7f137c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.315+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f139408aa20 con 0x7f139c15ceb0 2023-06-10T18:56:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.315+0000 7f139a59c700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 0x7f137c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.316+0000 7f139a59c700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 0x7f137c06d190 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1390001eb0 tx=0x7f1390001b40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.319+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1394055d70 con 0x7f139c15ceb0 2023-06-10T18:56:15.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.584+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f139c0c8030 con 0x7f139c15ceb0 2023-06-10T18:56:15.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.585+0000 7f138affd700 1 -- 172.21.15.104:0/1626941450 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f139c0c8030 con 0x7f139c15ceb0 2023-06-10T18:56:15.591 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:15.591 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:56:15.591 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:56:15.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:56:15.592 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:56:15.593 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:56:15.595 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.596 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.597 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.599 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.600 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:56:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:56:15.605 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.606 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:56:15.608 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.610 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:56:15.611 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:56:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:56:15.614 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.616 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:56:15.616 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:56:15.616 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:56:15.617 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:56:15.618 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:56:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.586+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 msgr2=0x7f137c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.586+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 0x7f137c06d190 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1390001eb0 tx=0x7f1390001b40 comp rx=0 tx=0).stop 2023-06-10T18:56:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.586+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 msgr2=0x7f139c150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.586+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f139c15b620 tx=0x7f1394009750 comp rx=0 tx=0).stop 2023-06-10T18:56:15.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 shutdown_connections 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f137c06ace0 0x7f137c06d190 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1384005200 0x7f139c0c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f139c15ceb0 0x7f139c150db0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 --2- 172.21.15.104:0/1626941450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f139c15ca40 0x7f139c0adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 >> 172.21.15.104:0/1626941450 conn(0x7f139c0b1610 msgr2=0x7f139c0afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 shutdown_connections 2023-06-10T18:56:15.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.587+0000 7f13a0ffb700 1 -- 172.21.15.104:0/1626941450 wait complete. 2023-06-10T18:56:15.622 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.735+0000 7fdebe2b9700 1 Processor -- start 2023-06-10T18:56:15.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 -- start start 2023-06-10T18:56:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdeb80ada30 0x7fdeb80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb80ade00 0x7fdeb80ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb80cec20 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb80cf430 con 0x7fdeb80ada30 2023-06-10T18:56:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.736+0000 7fdebe2b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cee70 con 0x7fdeb80ade00 2023-06-10T18:56:15.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdeb80ce0b0 0x7fdeb80ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34948/0 (socket says 172.21.15.104:34948) 2023-06-10T18:56:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 -- 172.21.15.104:0/1021698421 learned_addr learned my addr 172.21.15.104:0/1021698421 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb80ade00 0x7fdeb80ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962385468 0 0) 0x7fdeb80cec20 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb7fff700 1 --2- 172.21.15.104:0/1021698421 >> v2:172.21.15.114:3300/0 conn(0x7fdeb80ada30 0x7fdeb80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea00035b0 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157435705 0 0) 0x7fdea00035b0 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb80ade00 msgr2=0x7fdeb80ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb80ade00 0x7fdeb80ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 >> v2:172.21.15.114:3300/0 conn(0x7fdeb80ada30 msgr2=0x7fdeb80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 --2- 172.21.15.104:0/1021698421 >> v2:172.21.15.114:3300/0 conn(0x7fdeb80ada30 0x7fdeb80ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80cf150 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdeb00019f0 con 0x7fdeb80ce0b0 2023-06-10T18:56:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 >> v1:172.21.15.104:6789/0 conn(0x7fdeb80ce0b0 legacy=0x7fdeb80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdea0003a00 0x7fdea0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdea0004350 0x7fdea0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdea0004e80 0x7fdea00052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cf430 con 0x7fdea0004e80 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cee70 con 0x7fdea0004350 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea000be10 con 0x7fdea0003a00 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdea0004e80 0x7fdea00052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb7fff700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdea0003a00 0x7fdea0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdea0004350 0x7fdea0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 -- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdea0003a00 msgr2=0x7fdea0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdea0003a00 0x7fdea0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 -- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdea0004350 msgr2=0x7fdea0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.737+0000 7fdebc856700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdea0004350 0x7fdea0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebc856700 1 -- 172.21.15.104:0/1021698421 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea00035b0 con 0x7fdea0004e80 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebc856700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdea0004e80 0x7fdea00052f0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000a8d0 tx=0x7fdea0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370883dbc3f06d58 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000a340 con 0x7fdea0004e80 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb0012600 con 0x7fdea0004e80 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdeb6ffd700 1 -- 172.21.15.104:0/1021698421 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00129b0 con 0x7fdea0004e80 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 -- 172.21.15.104:0/1021698421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdea0004e80 msgr2=0x7fdea00052f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/1021698421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdea0004e80 0x7fdea00052f0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000a8d0 tx=0x7fdea0005880 comp rx=0 tx=0).stop 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebc856700 1 -- 172.21.15.104:0/1021698421 reap_dead start 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 -- 172.21.15.104:0/1021698421 shutdown_connections 2023-06-10T18:56:15.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 -- 172.21.15.104:0/1021698421 >> 172.21.15.104:0/1021698421 conn(0x7fdeb806b8a0 msgr2=0x7fdeb80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:15.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 -- 172.21.15.104:0/1021698421 shutdown_connections 2023-06-10T18:56:15.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.738+0000 7fdebe2b9700 1 -- 172.21.15.104:0/1021698421 wait complete. 2023-06-10T18:56:15.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 Processor -- start 2023-06-10T18:56:15.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 -- start start 2023-06-10T18:56:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 0x7fdeb80b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 0x7fdeb81575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80b0700 con 0x7fdeb81f11f0 2023-06-10T18:56:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80b1fc0 con 0x7fdeb81f15c0 2023-06-10T18:56:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.740+0000 7fdebe2b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8157b00 con 0x7fdeb8157120 2023-06-10T18:56:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37128/0 (socket says 172.21.15.104:37128) 2023-06-10T18:56:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 -- 172.21.15.104:0/2784479812 learned_addr learned my addr 172.21.15.104:0/2784479812 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdebc856700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 0x7fdeb81575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 0x7fdeb80b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 -- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 msgr2=0x7fdeb81575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 0x7fdeb81575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 -- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 msgr2=0x7fdeb80b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 0x7fdeb80b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 -- 172.21.15.104:0/2784479812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb81595c0 con 0x7fdeb81f11f0 2023-06-10T18:56:15.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdebc856700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 0x7fdeb81575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 0x7fdeb80b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb7fff700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fdea8011aa0 tx=0x7fdea8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800f980 con 0x7fdeb81f11f0 2023-06-10T18:56:15.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea800fb00 con 0x7fdeb81f11f0 2023-06-10T18:56:15.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb8159290 con 0x7fdeb81f11f0 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800feb0 con 0x7fdeb81f11f0 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.741+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80b0140 con 0x7fdeb81f11f0 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.742+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb8074050 con 0x7fdeb81f11f0 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.742+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdea8014440 con 0x7fdeb81f11f0 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.743+0000 7fdeb4ff9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 0x7fdea406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:15.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.743+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fdea8089af0 con 0x7fdeb81f11f0 2023-06-10T18:56:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.743+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 0x7fdea406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.744+0000 7fdeb77fe700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 0x7fdea406d190 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80b3550 tx=0x7fdeac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:15.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.747+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdea80587b0 con 0x7fdeb81f11f0 2023-06-10T18:56:15.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.936+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb80cd3b0 con 0x7fdeb81f11f0 2023-06-10T18:56:15.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.936+0000 7fdeb4ff9700 1 -- 172.21.15.104:0/2784479812 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+14902 (secure 0 0 0) 0x7fdea8058340 con 0x7fdeb81f11f0 2023-06-10T18:56:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:15.943 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":124,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:15.161561+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 msgr2=0x7fdea406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 0x7fdea406d190 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80b3550 tx=0x7fdeac008040 comp rx=0 tx=0).stop 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 msgr2=0x7fdeb80af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fdea8011aa0 tx=0x7fdea8011e60 comp rx=0 tx=0).stop 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 shutdown_connections 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdeb8157120 0x7fdeb81575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdeb81f11f0 0x7fdeb80af760 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdea406ace0 0x7fdea406d190 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 --2- 172.21.15.104:0/2784479812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdeb81f15c0 0x7fdeb80b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.938+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 >> 172.21.15.104:0/2784479812 conn(0x7fdeb806b8a0 msgr2=0x7fdeb80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.939+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 shutdown_connections 2023-06-10T18:56:15.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:15.939+0000 7fdebe2b9700 1 -- 172.21.15.104:0/2784479812 wait complete. 2023-06-10T18:56:15.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:15.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.075+0000 7faeb89d9700 1 Processor -- start 2023-06-10T18:56:16.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.075+0000 7faeb89d9700 1 -- start start 2023-06-10T18:56:16.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb89d9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faeb406db50 0x7faeb4163ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb89d9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40c5d80 con 0x7faeb4158c10 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb89d9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40cec80 con 0x7faeb4158fe0 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb89d9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40cdbe0 con 0x7faeb406db50 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faeb4158c10 0x7faeb40cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34964/0 (socket says 172.21.15.104:34964) 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.076+0000 7faeb259c700 1 -- 172.21.15.104:0/2392327962 learned_addr learned my addr 172.21.15.104:0/2392327962 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb2d9d700 1 --2- 172.21.15.104:0/2392327962 >> v2:172.21.15.114:3300/0 conn(0x7faeb406db50 0x7faeb4163ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124833059 0 0) 0x7faeb40c5d80 con 0x7faeb4158c10 2023-06-10T18:56:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae9c0035b0 con 0x7faeb4158c10 2023-06-10T18:56:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866323688 0 0) 0x7fae9c0035b0 con 0x7faeb4158c10 2023-06-10T18:56:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2392327962 >> v1:172.21.15.114:6789/0 conn(0x7faeb4158fe0 legacy=0x7faeb406d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2392327962 >> v1:172.21.15.104:6789/0 conn(0x7faeb4158c10 legacy=0x7faeb40cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2392327962 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40cdec0 con 0x7faeb406db50 2023-06-10T18:56:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb2d9d700 1 --2- 172.21.15.104:0/2392327962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4163ce0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faeac009d90 tx=0x7faeac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb16b186d16a852f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac00e070 con 0x7faeb406db50 2023-06-10T18:56:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac003a20 con 0x7faeb406db50 2023-06-10T18:56:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.077+0000 7faeb159a700 1 -- 172.21.15.104:0/2392327962 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8001670 con 0x7faeb406db50 2023-06-10T18:56:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 -- 172.21.15.104:0/2392327962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 msgr2=0x7faeb4163ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2392327962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4163ce0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faeac009d90 tx=0x7faeac00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 -- 172.21.15.104:0/2392327962 shutdown_connections 2023-06-10T18:56:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2392327962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4163ce0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 -- 172.21.15.104:0/2392327962 >> 172.21.15.104:0/2392327962 conn(0x7faeb40ceec0 msgr2=0x7faeb40cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 -- 172.21.15.104:0/2392327962 shutdown_connections 2023-06-10T18:56:16.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.078+0000 7faeb89d9700 1 -- 172.21.15.104:0/2392327962 wait complete. 2023-06-10T18:56:16.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.079+0000 7faeb89d9700 1 Processor -- start 2023-06-10T18:56:16.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.079+0000 7faeb89d9700 1 -- start start 2023-06-10T18:56:16.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 0x7faeb41e2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41537f0 con 0x7faeb4158fe0 2023-06-10T18:56:16.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41e4fe0 con 0x7faeb406db50 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb89d9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41e52c0 con 0x7faeb4158c10 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37146/0 (socket says 172.21.15.104:37146) 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2807011068 learned_addr learned my addr 172.21.15.104:0/2807011068 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb259c700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb1d9b700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 0x7faeb41e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 msgr2=0x7faeb41e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 0x7faeb41e2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 msgr2=0x7faeb4153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb2d9d700 1 -- 172.21.15.104:0/2807011068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac009a70 con 0x7faeb4158fe0 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.080+0000 7faeb1d9b700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 0x7faeb41e2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faeb2d9d700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7faeb40c5f10 tx=0x7faeac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac004640 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac0047a0 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb41e55a0 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac01d6b0 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.081+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb41e2d40 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.082+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb4150d00 con 0x7faeb4158fe0 2023-06-10T18:56:16.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.082+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faeac01d810 con 0x7faeb4158fe0 2023-06-10T18:56:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.082+0000 7faea2ffd700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 0x7fae9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.083+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7faeac08b390 con 0x7faeb4158fe0 2023-06-10T18:56:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.083+0000 7faeb259c700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 0x7fae9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.083+0000 7faeb259c700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 0x7fae9406d190 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7faea8001eb0 tx=0x7faea8001c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.087+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7faeb4150d00 con 0x7faeb4158fe0 2023-06-10T18:56:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.273+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faeb4150d00 con 0x7faeb4158fe0 2023-06-10T18:56:16.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.273+0000 7faea2ffd700 1 -- 172.21.15.104:0/2807011068 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7faeb4150d00 con 0x7faeb4158fe0 2023-06-10T18:56:16.280 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:16.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 msgr2=0x7fae9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 0x7fae9406d190 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7faea8001eb0 tx=0x7faea8001c60 comp rx=0 tx=0).stop 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 msgr2=0x7faeb414cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7faeb40c5f10 tx=0x7faeac004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 shutdown_connections 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae9406ace0 0x7fae9406d190 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb406db50 0x7faeb4153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faeb4158fe0 0x7faeb414cc50 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 --2- 172.21.15.104:0/2807011068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faeb4158c10 0x7faeb41e2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 >> 172.21.15.104:0/2807011068 conn(0x7faeb40ceec0 msgr2=0x7faeb40b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 shutdown_connections 2023-06-10T18:56:16.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.275+0000 7faeb89d9700 1 -- 172.21.15.104:0/2807011068 wait complete. 2023-06-10T18:56:16.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:16.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.412+0000 7f583b5bd700 1 Processor -- start 2023-06-10T18:56:16.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.412+0000 7f583b5bd700 1 -- start start 2023-06-10T18:56:16.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.412+0000 7f583b5bd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 0x7f58340cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.412+0000 7f583b5bd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5834069e70 0x7f583406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f583b5bd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58340ce040 con 0x7f583406dcd0 2023-06-10T18:56:16.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f583b5bd700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583415c8d0 con 0x7f5834069e70 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f583b5bd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583406da40 con 0x7f583415c500 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5834069e70 0x7f583406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5839b5a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f583406dcd0 0x7f583406e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34982/0 (socket says 172.21.15.104:34982) 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5839b5a700 1 -- 172.21.15.104:0/3586578022 learned_addr learned my addr 172.21.15.104:0/3586578022 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5839359700 1 --2- 172.21.15.104:0/3586578022 >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 0x7f58340cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 -- 172.21.15.104:0/3586578022 >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 msgr2=0x7f58340cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 --2- 172.21.15.104:0/3586578022 >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 0x7f58340cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 -- 172.21.15.104:0/3586578022 >> v1:172.21.15.104:6789/0 conn(0x7f583406dcd0 legacy=0x7f583406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 -- 172.21.15.104:0/3586578022 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583406a9c0 con 0x7f5834069e70 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f582bfff700 1 -- 172.21.15.104:0/3586578022 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224511203 0 0) 0x7f58340ce040 con 0x7f583406dcd0 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5839359700 1 --2- 172.21.15.104:0/3586578022 >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 0x7f58340cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f5838b58700 1 --2- 172.21.15.104:0/3586578022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5834069e70 0x7f583406a2e0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f58340b2170 tx=0x7f5830009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2974b4dcb449cb0f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f582bfff700 1 -- 172.21.15.104:0/3586578022 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830011830 con 0x7f5834069e70 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f582bfff700 1 -- 172.21.15.104:0/3586578022 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830011990 con 0x7f5834069e70 2023-06-10T18:56:16.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.413+0000 7f582bfff700 1 -- 172.21.15.104:0/3586578022 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830011d20 con 0x7f5834069e70 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 -- 172.21.15.104:0/3586578022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5834069e70 msgr2=0x7f583406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 --2- 172.21.15.104:0/3586578022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5834069e70 0x7f583406a2e0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f58340b2170 tx=0x7f5830009b20 comp rx=0 tx=0).stop 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 -- 172.21.15.104:0/3586578022 shutdown_connections 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 --2- 172.21.15.104:0/3586578022 >> v2:172.21.15.114:3300/0 conn(0x7f583415c500 0x7f58340cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 --2- 172.21.15.104:0/3586578022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5834069e70 0x7f583406a2e0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 -- 172.21.15.104:0/3586578022 >> 172.21.15.104:0/3586578022 conn(0x7f583406b8a0 msgr2=0x7f58340cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 -- 172.21.15.104:0/3586578022 shutdown_connections 2023-06-10T18:56:16.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.414+0000 7f583b5bd700 1 -- 172.21.15.104:0/3586578022 wait complete. 2023-06-10T18:56:16.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.415+0000 7f583b5bd700 1 Processor -- start 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.415+0000 7f583b5bd700 1 -- start start 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 0x7f583414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 0x7f58340ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340c4cd0 con 0x7f583406dcd0 2023-06-10T18:56:16.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340c4fb0 con 0x7f583415c500 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f583b5bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340c5290 con 0x7f5834069e70 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37158/0 (socket says 172.21.15.104:37158) 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 -- 172.21.15.104:0/1816969584 learned_addr learned my addr 172.21.15.104:0/1816969584 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5839b5a700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 0x7f58340ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5839359700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 0x7f583414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 -- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 msgr2=0x7f583414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 0x7f583414abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 -- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 msgr2=0x7f58340ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 0x7f58340ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5838b58700 1 -- 172.21.15.104:0/1816969584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58300094a0 con 0x7f583406dcd0 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5839b5a700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 0x7f58340ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.416+0000 7f5839359700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 0x7f583414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f5838b58700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f5830014040 tx=0x7f5830004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58300038a0 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583001e070 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58340c5570 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830003c40 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.417+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58340c32e0 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.418+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58340ce5a0 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.418+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f583002d450 con 0x7f583406dcd0 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.419+0000 7f5829ffb700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 0x7f581806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.419+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f583008a410 con 0x7f583406dcd0 2023-06-10T18:56:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.419+0000 7f5839359700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 0x7f581806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.419+0000 7f5839359700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 0x7f581806d190 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5824006fd0 tx=0x7f5824008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.423+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f58340ce5a0 con 0x7f583406dcd0 2023-06-10T18:56:16.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.606+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f58340c5a90 con 0x7f583406dcd0 2023-06-10T18:56:16.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.606+0000 7f5829ffb700 1 -- 172.21.15.104:0/1816969584 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v124) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5830020020 con 0x7f583406dcd0 2023-06-10T18:56:16.612 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:16.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 msgr2=0x7f581806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 0x7f581806d190 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5824006fd0 tx=0x7f5824008040 comp rx=0 tx=0).stop 2023-06-10T18:56:16.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 msgr2=0x7f58340b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f5830014040 tx=0x7f5830004750 comp rx=0 tx=0).stop 2023-06-10T18:56:16.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 shutdown_connections 2023-06-10T18:56:16.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f581806ace0 0x7f581806d190 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583415c500 0x7f58340ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5834069e70 0x7f583414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 --2- 172.21.15.104:0/1816969584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f583406dcd0 0x7f58340b3ea0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 >> 172.21.15.104:0/1816969584 conn(0x7f583406b8a0 msgr2=0x7f58340ac7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 shutdown_connections 2023-06-10T18:56:16.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.608+0000 7f583b5bd700 1 -- 172.21.15.104:0/1816969584 wait complete. 2023-06-10T18:56:16.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:56:16.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.739+0000 7f6a837b1700 1 Processor -- start 2023-06-10T18:56:16.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 -- start start 2023-06-10T18:56:16.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 0x7f6a7c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 0x7f6a7c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7c0c9bc0 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7c0c9e60 con 0x7f6a7c0b1080 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.740+0000 7f6a837b1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0aee70 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37168/0 (socket says 172.21.15.104:37168) 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 -- 172.21.15.104:0/3006663696 learned_addr learned my addr 172.21.15.104:0/3006663696 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/3006663696 >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 0x7f6a7c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a8154d700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 0x7f6a7c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 -- 172.21.15.104:0/3006663696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 msgr2=0x7f6a7c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 0x7f6a7c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 -- 172.21.15.104:0/3006663696 >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 msgr2=0x7f6a7c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 --2- 172.21.15.104:0/3006663696 >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 0x7f6a7c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 -- 172.21.15.104:0/3006663696 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c06e540 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/3006663696 >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 0x7f6a7c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a8154d700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 0x7f6a7c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:16.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a81d4e700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c066f40 tx=0x7f6a7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60041e0cdc92bfd8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a73fff700 1 -- 172.21.15.104:0/3006663696 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7400e070 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a73fff700 1 -- 172.21.15.104:0/3006663696 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7400b9c0 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.741+0000 7f6a73fff700 1 -- 172.21.15.104:0/3006663696 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7400bd70 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 -- 172.21.15.104:0/3006663696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0ca2f0 msgr2=0x7f6a7c0ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c066f40 tx=0x7f6a7400b330 comp rx=0 tx=0).stop 2023-06-10T18:56:16.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 -- 172.21.15.104:0/3006663696 shutdown_connections 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0ca2f0 0x7f6a7c0ca750 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 --2- 172.21.15.104:0/3006663696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0aeaa0 0x7f6a7c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 --2- 172.21.15.104:0/3006663696 >> v2:172.21.15.114:3300/0 conn(0x7f6a7c0b1080 0x7f6a7c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 -- 172.21.15.104:0/3006663696 >> 172.21.15.104:0/3006663696 conn(0x7f6a7c06b8a0 msgr2=0x7f6a7c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 -- 172.21.15.104:0/3006663696 shutdown_connections 2023-06-10T18:56:16.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.742+0000 7f6a837b1700 1 -- 172.21.15.104:0/3006663696 wait complete. 2023-06-10T18:56:16.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.743+0000 7f6a837b1700 1 Processor -- start 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.743+0000 7f6a837b1700 1 -- start start 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 0x7f6a7c0ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0ca2f0 0x7f6a7c1f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0acd80 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0ac770 con 0x7f6a7c0b1080 2023-06-10T18:56:16.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c1f5020 con 0x7f6a7c0ca2f0 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37170/0 (socket says 172.21.15.104:37170) 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 -- 172.21.15.104:0/889293607 learned_addr learned my addr 172.21.15.104:0/889293607 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 0x7f6a7c0ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a81d4e700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0ca2f0 0x7f6a7c1f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 -- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0ca2f0 msgr2=0x7f6a7c1f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0ca2f0 0x7f6a7c1f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 -- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 msgr2=0x7f6a7c0ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 0x7f6a7c0ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 -- 172.21.15.104:0/889293607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74009b30 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 0x7f6a7c0ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a8154d700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c006fd0 tx=0x7f6a6c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c005c20 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c005d80 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7c0c66e0 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.744+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c011840 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.745+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7c1f4cb0 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.745+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7c1f5300 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.746+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a6c0119a0 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.746+0000 7f6a71ffb700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 0x7f6a6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.746+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f6a6c089440 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.746+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 0x7f6a6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:16.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.747+0000 7f6a80d4c700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 0x7f6a6006d190 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0ce130 tx=0x7f6a78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:16.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.751+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a6c054790 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.943+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6a7c074050 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.943+0000 7f6a71ffb700 1 -- 172.21.15.104:0/889293607 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+126 (secure 0 0 0) 0x7f6a6c057db0 con 0x7f6a7c0aeaa0 2023-06-10T18:56:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:56:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:16.951 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:56:16.951 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:56:16.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:16.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:16.952 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:56:16.952 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:56:16.952 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:16.952 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:56:16.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.945+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 msgr2=0x7f6a6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.945+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 0x7f6a6006d190 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0ce130 tx=0x7f6a78008040 comp rx=0 tx=0).stop 2023-06-10T18:56:16.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.945+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 msgr2=0x7f6a7c1566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:16.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.945+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c006fd0 tx=0x7f6a6c00ca30 comp rx=0 tx=0).stop 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 shutdown_connections 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a7c0ca2f0 0x7f6a7c1f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a7c0aeaa0 0x7f6a7c1566c0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a6006ace0 0x7f6a6006d190 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 --2- 172.21.15.104:0/889293607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a7c0b1080 0x7f6a7c0ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 >> 172.21.15.104:0/889293607 conn(0x7f6a7c06b8a0 msgr2=0x7f6a7c0b1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:16.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 shutdown_connections 2023-06-10T18:56:16.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:16.946+0000 7f6a837b1700 1 -- 172.21.15.104:0/889293607 wait complete. 2023-06-10T18:56:16.964 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:56:16.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:56:17.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.082+0000 7fd6a015c700 1 Processor -- start 2023-06-10T18:56:17.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 -- start start 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd69815c500 0x7fd6980cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd698069e70 0x7fd69806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd69806e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69806e690 con 0x7fd69815c500 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6980ce040 con 0x7fd698069e70 2023-06-10T18:56:17.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.083+0000 7fd6a015c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69815c8d0 con 0x7fd69806dcd0 2023-06-10T18:56:17.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd69815c500 0x7fd6980cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd69815c500 0x7fd6980cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37178/0 (socket says 172.21.15.104:37178) 2023-06-10T18:56:17.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 -- 172.21.15.104:0/2961010181 learned_addr learned my addr 172.21.15.104:0/2961010181 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:17.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69e6f9700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd69806e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69d6f7700 1 --2- 172.21.15.104:0/2961010181 >> v2:172.21.15.114:3300/0 conn(0x7fd698069e70 0x7fd69806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 -- 172.21.15.104:0/2961010181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 msgr2=0x7fd69806e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd69806e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 -- 172.21.15.104:0/2961010181 >> v2:172.21.15.114:3300/0 conn(0x7fd698069e70 msgr2=0x7fd69806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 --2- 172.21.15.104:0/2961010181 >> v2:172.21.15.114:3300/0 conn(0x7fd698069e70 0x7fd69806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 -- 172.21.15.104:0/2961010181 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69806da40 con 0x7fd69815c500 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69def8700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980cedc0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b21d0 tx=0x7fd688009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9e0d1504622acd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69cef6700 1 -- 172.21.15.104:0/2961010181 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688005e30 con 0x7fd69815c500 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69cef6700 1 -- 172.21.15.104:0/2961010181 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd688004030 con 0x7fd69815c500 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.084+0000 7fd69cef6700 1 -- 172.21.15.104:0/2961010181 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880043e0 con 0x7fd69815c500 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 -- 172.21.15.104:0/2961010181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 msgr2=0x7fd6980cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980cedc0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b21d0 tx=0x7fd688009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 -- 172.21.15.104:0/2961010181 shutdown_connections 2023-06-10T18:56:17.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980cedc0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 --2- 172.21.15.104:0/2961010181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd69806e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 --2- 172.21.15.104:0/2961010181 >> v2:172.21.15.114:3300/0 conn(0x7fd698069e70 0x7fd69806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 -- 172.21.15.104:0/2961010181 >> 172.21.15.104:0/2961010181 conn(0x7fd69806b8a0 msgr2=0x7fd6980cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:17.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 -- 172.21.15.104:0/2961010181 shutdown_connections 2023-06-10T18:56:17.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.085+0000 7fd6a015c700 1 -- 172.21.15.104:0/2961010181 wait complete. 2023-06-10T18:56:17.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.086+0000 7fd6a015c700 1 Processor -- start 2023-06-10T18:56:17.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 -- start start 2023-06-10T18:56:17.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 0x7fd698157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd698159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980c6780 con 0x7fd69815c500 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980c6a60 con 0x7fd698069e70 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.087+0000 7fd6a015c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980c6d40 con 0x7fd69806dcd0 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37188/0 (socket says 172.21.15.104:37188) 2023-06-10T18:56:17.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 -- 172.21.15.104:0/3324393601 learned_addr learned my addr 172.21.15.104:0/3324393601 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69def8700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 0x7fd698157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69d6f7700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd698159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 -- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 msgr2=0x7fd698159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd698159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 -- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 msgr2=0x7fd698157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 0x7fd698157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 -- 172.21.15.104:0/3324393601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009500 con 0x7fd69815c500 2023-06-10T18:56:17.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69def8700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 0x7fd698157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd69e6f9700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fd690007e60 tx=0x7fd69000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000ede0 con 0x7fd69815c500 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd690014420 con 0x7fd69815c500 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6900147d0 con 0x7fd69815c500 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6980c7020 con 0x7fd69815c500 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.088+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6980b2440 con 0x7fd69815c500 2023-06-10T18:56:17.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.089+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6980c7540 con 0x7fd69815c500 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.090+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd690014930 con 0x7fd69815c500 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.090+0000 7fd68e7fc700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 0x7fd67c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.090+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fd6900899c0 con 0x7fd69815c500 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.090+0000 7fd69def8700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 0x7fd67c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.091+0000 7fd69def8700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 0x7fd67c06d190 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b21d0 tx=0x7fd688008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:17.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.094+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd6980c7540 con 0x7fd69815c500 2023-06-10T18:56:17.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.281+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd6980c7540 con 0x7fd69815c500 2023-06-10T18:56:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.830+0000 7fd68e7fc700 1 -- 172.21.15.104:0/3324393601 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd6980c7540 con 0x7fd69815c500 2023-06-10T18:56:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 msgr2=0x7fd67c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 0x7fd67c06d190 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b21d0 tx=0x7fd688008070 comp rx=0 tx=0).stop 2023-06-10T18:56:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 msgr2=0x7fd6980c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fd690007e60 tx=0x7fd69000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 shutdown_connections 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd67c06ace0 0x7fd67c06d190 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698069e70 0x7fd698157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd69815c500 0x7fd6980c8be0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 --2- 172.21.15.104:0/3324393601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd69806dcd0 0x7fd698159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.833+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 >> 172.21.15.104:0/3324393601 conn(0x7fd69806b8a0 msgr2=0x7fd6980cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.834+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 shutdown_connections 2023-06-10T18:56:17.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.834+0000 7fd6a015c700 1 -- 172.21.15.104:0/3324393601 wait complete. 2023-06-10T18:56:17.842 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:56:17.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:56:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 Processor -- start 2023-06-10T18:56:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 -- start start 2023-06-10T18:56:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e7c0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e7c0d0830 0x7f7e7c0b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7c0ad3c0 con 0x7f7e7c157b30 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c157f00 con 0x7f7e7c0d0830 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e811eb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b0170 con 0x7f7e7c0cea50 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e7ad9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7e7c157b30 0x7f7e7c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35020/0 (socket says 172.21.15.104:35020) 2023-06-10T18:56:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.976+0000 7f7e7ad9d700 1 -- 172.21.15.104:0/2001027614 learned_addr learned my addr 172.21.15.104:0/2001027614 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c0d0830 0x7f7e7c0b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e7c0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849690097 0 0) 0x7f7e7c0ad3c0 con 0x7f7e7c157b30 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e640035b0 con 0x7f7e7c157b30 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360792454 0 0) 0x7f7e640035b0 con 0x7f7e7c157b30 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 msgr2=0x7f7e7c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e7c0d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c0d0830 msgr2=0x7f7e7c0b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/2001027614 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c0d0830 0x7f7e7c0b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0b0410 con 0x7f7e7c157b30 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7e70001670 con 0x7f7e7c157b30 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e7c0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> v2:172.21.15.114:3300/0 conn(0x7f7e7c0d0830 0x7f7e7c0b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 >> v1:172.21.15.104:6789/0 conn(0x7f7e7c157b30 legacy=0x7f7e7c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e64003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e64003f40 0x7f7e64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e64004d90 0x7f7e64005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c157f00 con 0x7f7e64004d90 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b0170 con 0x7f7e64003f40 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6400ba10 con 0x7f7e7c0cea50 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e64004d90 0x7f7e64005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e64003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e64003f40 0x7f7e64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 -- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 msgr2=0x7f7e64003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c0cea50 0x7f7e64003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 -- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e64003f40 msgr2=0x7f7e64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e64003f40 0x7f7e64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.977+0000 7f7e7b59e700 1 -- 172.21.15.104:0/2001027614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0094a0 con 0x7f7e64004d90 2023-06-10T18:56:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e64004d90 0x7f7e64005200 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400e350 tx=0x7f7e64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e455fba2024d58 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400edf0 con 0x7f7e64004d90 2023-06-10T18:56:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7400f040 con 0x7f7e64004d90 2023-06-10T18:56:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e79d9b700 1 -- 172.21.15.104:0/2001027614 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e740146a0 con 0x7f7e64004d90 2023-06-10T18:56:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 -- 172.21.15.104:0/2001027614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e64004d90 msgr2=0x7f7e64005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 --2- 172.21.15.104:0/2001027614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e64004d90 0x7f7e64005200 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400e350 tx=0x7f7e64005880 comp rx=0 tx=0).stop 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e7b59e700 1 -- 172.21.15.104:0/2001027614 reap_dead start 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 -- 172.21.15.104:0/2001027614 shutdown_connections 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 -- 172.21.15.104:0/2001027614 >> 172.21.15.104:0/2001027614 conn(0x7f7e7c089fd0 msgr2=0x7f7e7c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 -- 172.21.15.104:0/2001027614 shutdown_connections 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.978+0000 7f7e811eb700 1 -- 172.21.15.104:0/2001027614 wait complete. 2023-06-10T18:56:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 Processor -- start 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 -- start start 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c158cf0 0x7f7e7c1f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c1590c0 0x7f7e7c1f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b1e80 con 0x7f7e7c0b2cb0 2023-06-10T18:56:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b1300 con 0x7f7e7c158cf0 2023-06-10T18:56:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.980+0000 7f7e811eb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b15e0 con 0x7f7e7c1590c0 2023-06-10T18:56:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37206/0 (socket says 172.21.15.104:37206) 2023-06-10T18:56:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 -- 172.21.15.104:0/3047610170 learned_addr learned my addr 172.21.15.104:0/3047610170 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c158cf0 0x7f7e7c1f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7b59e700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c1590c0 0x7f7e7c1f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 -- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c1590c0 msgr2=0x7f7e7c1f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c1590c0 0x7f7e7c1f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 -- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c158cf0 msgr2=0x7f7e7c1f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c158cf0 0x7f7e7c1f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 -- 172.21.15.104:0/3047610170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0b18c0 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e7a59c700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c011cd0 tx=0x7f7e6c009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00fa00 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c00fb60 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c0b1ba0 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c004030 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.981+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c149920 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.982+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c074050 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.983+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7e6c004190 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.983+0000 7f7e637fe700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 0x7f7e6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.983+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f7e6c089860 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.983+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 0x7f7e6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.984+0000 7f7e7ad9d700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 0x7f7e6806d190 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7e700048d0 tx=0x7f7e70007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:17.988+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7e6c0549b0 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.173+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7e7c0b35a0 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.831+0000 7f7e637fe700 1 -- 172.21.15.104:0/3047610170 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7e6c0220d0 con 0x7f7e7c0b2cb0 2023-06-10T18:56:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.836+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 msgr2=0x7f7e6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.836+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 0x7f7e6806d190 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7e700048d0 tx=0x7f7e70007980 comp rx=0 tx=0).stop 2023-06-10T18:56:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.836+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 msgr2=0x7f7e7c1f36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.836+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c011cd0 tx=0x7f7e6c009e00 comp rx=0 tx=0).stop 2023-06-10T18:56:18.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.838+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 shutdown_connections 2023-06-10T18:56:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.838+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7e6806ace0 0x7f7e6806d190 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.839+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c158cf0 0x7f7e7c1f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.839+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e7c1590c0 0x7f7e7c1f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.839+0000 7f7e811eb700 1 --2- 172.21.15.104:0/3047610170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e7c0b2cb0 0x7f7e7c1f36f0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:18.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.839+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 >> 172.21.15.104:0/3047610170 conn(0x7f7e7c089fd0 msgr2=0x7f7e7c15cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.839+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 shutdown_connections 2023-06-10T18:56:18.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.840+0000 7f7e811eb700 1 -- 172.21.15.104:0/3047610170 wait complete. 2023-06-10T18:56:18.851 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:56:18.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:56:18.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.988+0000 7f12c5bfc700 1 Processor -- start 2023-06-10T18:56:18.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.988+0000 7f12c5bfc700 1 -- start start 2023-06-10T18:56:18.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12c5bfc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12c015c580 0x7f12c00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12c5bfc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c006e250 con 0x7f12c006cde0 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12c5bfc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c015c950 con 0x7f12c006e6e0 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12c5bfc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c00b1e80 con 0x7f12c015c580 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12bffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f12c006cde0 0x7f12c00ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35040/0 (socket says 172.21.15.104:35040) 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12bffff700 1 -- 172.21.15.104:0/1740573588 learned_addr learned my addr 172.21.15.104:0/1740573588 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12bf7fe700 1 --2- 172.21.15.104:0/1740573588 >> v2:172.21.15.114:3300/0 conn(0x7f12c015c580 0x7f12c00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:18.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319320691 0 0) 0x7f12c006e250 con 0x7f12c006cde0 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.989+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a80035b0 con 0x7f12c006cde0 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12bf7fe700 1 -- 172.21.15.104:0/1740573588 >> v1:172.21.15.114:6789/0 conn(0x7f12c006e6e0 legacy=0x7f12c00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018811135 0 0) 0x7f12c015c950 con 0x7f12c006e6e0 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12bf7fe700 1 -- 172.21.15.104:0/1740573588 >> v1:172.21.15.104:6789/0 conn(0x7f12c006cde0 legacy=0x7f12c00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12bf7fe700 1 -- 172.21.15.104:0/1740573588 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c006d920 con 0x7f12c015c580 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12bf7fe700 1 --2- 172.21.15.104:0/1740573588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c015c580 0x7f12c00b1070 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009820 tx=0x7f12b4009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=200845f07e2eab87 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:18.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4005e30 con 0x7f12c015c580 2023-06-10T18:56:18.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4006030 con 0x7f12c015c580 2023-06-10T18:56:19.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12be7fc700 1 -- 172.21.15.104:0/1740573588 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b40063c0 con 0x7f12c015c580 2023-06-10T18:56:19.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12c5bfc700 1 -- 172.21.15.104:0/1740573588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c015c580 msgr2=0x7f12c00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/1740573588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c015c580 0x7f12c00b1070 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009820 tx=0x7f12b4009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:19.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12c5bfc700 1 -- 172.21.15.104:0/1740573588 shutdown_connections 2023-06-10T18:56:19.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/1740573588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c015c580 0x7f12c00b1070 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.990+0000 7f12c5bfc700 1 -- 172.21.15.104:0/1740573588 >> 172.21.15.104:0/1740573588 conn(0x7f12c006b8a0 msgr2=0x7f12c00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.991+0000 7f12c5bfc700 1 -- 172.21.15.104:0/1740573588 shutdown_connections 2023-06-10T18:56:19.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.991+0000 7f12c5bfc700 1 -- 172.21.15.104:0/1740573588 wait complete. 2023-06-10T18:56:19.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.992+0000 7f12c5bfc700 1 Processor -- start 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.992+0000 7f12c5bfc700 1 -- start start 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c006cde0 0x7f12c00b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c006e6e0 0x7f12c00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00ac480 con 0x7f12c015c580 2023-06-10T18:56:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00ac760 con 0x7f12c006e6e0 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00aca40 con 0x7f12c006cde0 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37218/0 (socket says 172.21.15.104:37218) 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 -- 172.21.15.104:0/448505823 learned_addr learned my addr 172.21.15.104:0/448505823 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bf7fe700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c006cde0 0x7f12c00b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12beffd700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c006e6e0 0x7f12c00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 -- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c006cde0 msgr2=0x7f12c00b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c006cde0 0x7f12c00b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 -- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c006e6e0 msgr2=0x7f12c00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c006e6e0 0x7f12c00c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 -- 172.21.15.104:0/448505823 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009430 con 0x7f12c015c580 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12bffff700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f12b800cb40 tx=0x7f12b800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b800a9e0 con 0x7f12c015c580 2023-06-10T18:56:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800ab40 con 0x7f12c015c580 2023-06-10T18:56:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c00acd20 con 0x7f12c015c580 2023-06-10T18:56:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8012400 con 0x7f12c015c580 2023-06-10T18:56:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.993+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c00b2960 con 0x7f12c015c580 2023-06-10T18:56:19.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.994+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c00ad240 con 0x7f12c015c580 2023-06-10T18:56:19.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.995+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12b801b3b0 con 0x7f12c015c580 2023-06-10T18:56:19.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.995+0000 7f12a7fff700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 0x7f12ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.995+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f12b80890c0 con 0x7f12c015c580 2023-06-10T18:56:19.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.995+0000 7f12bf7fe700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 0x7f12ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:18.996+0000 7f12bf7fe700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 0x7f12ac06d190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fd0 tx=0x7f12b4008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.000+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12c00ad240 con 0x7f12c015c580 2023-06-10T18:56:19.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.228+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f12c00ad240 con 0x7f12c015c580 2023-06-10T18:56:19.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.235+0000 7f12a7fff700 1 -- 172.21.15.104:0/448505823 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v76) v1 ==== 174+0+0 (secure 0 0 0) 0x7f12c00ad240 con 0x7f12c015c580 2023-06-10T18:56:19.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.238+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 msgr2=0x7f12ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.238+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 0x7f12ac06d190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009fd0 tx=0x7f12b4008070 comp rx=0 tx=0).stop 2023-06-10T18:56:19.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.238+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 msgr2=0x7f12c00c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.238+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f12b800cb40 tx=0x7f12b800ce50 comp rx=0 tx=0).stop 2023-06-10T18:56:19.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 shutdown_connections 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c015c580 0x7f12c00c7140 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c006cde0 0x7f12c00b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12ac06ace0 0x7f12ac06d190 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 --2- 172.21.15.104:0/448505823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c006e6e0 0x7f12c00c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.239+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 >> 172.21.15.104:0/448505823 conn(0x7f12c006b8a0 msgr2=0x7f12c00cd360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.240+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 shutdown_connections 2023-06-10T18:56:19.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.240+0000 7f12c5bfc700 1 -- 172.21.15.104:0/448505823 wait complete. 2023-06-10T18:56:19.250 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 18 and data pool 19 2023-06-10T18:56:19.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:19.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.382+0000 7fc17593f700 1 Processor -- start 2023-06-10T18:56:19.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 -- start start 2023-06-10T18:56:19.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700ce510 0x7fc1700ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc170070130 0x7fc1700c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1700cee20 con 0x7fc1700705a0 2023-06-10T18:56:19.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700caa90 con 0x7fc170070130 2023-06-10T18:56:19.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc17593f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc170071fe0 con 0x7fc1700ce510 2023-06-10T18:56:19.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc16e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc170070130 0x7fc1700c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc16e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc170070130 0x7fc1700c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37222/0 (socket says 172.21.15.104:37222) 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.383+0000 7fc16e7fc700 1 -- 172.21.15.104:0/1992881761 learned_addr learned my addr 172.21.15.104:0/1992881761 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16effd700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700ce510 0x7fc1700ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16e7fc700 1 -- 172.21.15.104:0/1992881761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700ce510 msgr2=0x7fc1700ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16e7fc700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700ce510 0x7fc1700ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16e7fc700 1 -- 172.21.15.104:0/1992881761 >> v1:172.21.15.114:6789/0 conn(0x7fc1700705a0 legacy=0x7fc1700cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:19.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16e7fc700 1 -- 172.21.15.104:0/1992881761 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700722c0 con 0x7fc170070130 2023-06-10T18:56:19.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16e7fc700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc1700c8bd0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098f0 tx=0x7fc160009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb74c78991f6bae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16dffb700 1 -- 172.21.15.104:0/1992881761 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc160004230 con 0x7fc170070130 2023-06-10T18:56:19.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16dffb700 1 -- 172.21.15.104:0/1992881761 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160004390 con 0x7fc170070130 2023-06-10T18:56:19.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.384+0000 7fc16dffb700 1 -- 172.21.15.104:0/1992881761 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1600036a0 con 0x7fc170070130 2023-06-10T18:56:19.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 -- 172.21.15.104:0/1992881761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 msgr2=0x7fc1700c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc1700c8bd0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fc1600098f0 tx=0x7fc160009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:19.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 -- 172.21.15.104:0/1992881761 shutdown_connections 2023-06-10T18:56:19.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700ce510 0x7fc1700ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 --2- 172.21.15.104:0/1992881761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc1700c8bd0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 -- 172.21.15.104:0/1992881761 >> 172.21.15.104:0/1992881761 conn(0x7fc17006ab00 msgr2=0x7fc1700b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 -- 172.21.15.104:0/1992881761 shutdown_connections 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.385+0000 7fc17593f700 1 -- 172.21.15.104:0/1992881761 wait complete. 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.386+0000 7fc17593f700 1 Processor -- start 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.386+0000 7fc17593f700 1 -- start start 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700705a0 0x7fc1700cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1700ce510 0x7fc1700c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170152430 con 0x7fc170070130 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170156770 con 0x7fc1700ce510 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc170157620 con 0x7fc1700705a0 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37228/0 (socket says 172.21.15.104:37228) 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 -- 172.21.15.104:0/1204645959 learned_addr learned my addr 172.21.15.104:0/1204645959 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16f7fe700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1700ce510 0x7fc1700c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16e7fc700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700705a0 0x7fc1700cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 -- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700705a0 msgr2=0x7fc1700cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700705a0 0x7fc1700cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 -- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1700ce510 msgr2=0x7fc1700c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1700ce510 0x7fc1700c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 -- 172.21.15.104:0/1204645959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160009550 con 0x7fc170070130 2023-06-10T18:56:19.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc16effd700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c7f0 tx=0x7fc16400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164004220 con 0x7fc170070130 2023-06-10T18:56:19.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc164004380 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1701569a0 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1640036a0 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.387+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170156f60 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.388+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170074050 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.389+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc164003800 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.389+0000 7fc157fff700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 0x7fc15c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.389+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc164089090 con 0x7fc170070130 2023-06-10T18:56:19.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.389+0000 7fc16e7fc700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 0x7fc15c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.390+0000 7fc16e7fc700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 0x7fc15c06d190 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc160009130 tx=0x7fc160008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.394+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc164057b20 con 0x7fc170070130 2023-06-10T18:56:19.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.582+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc170153e30 con 0x7fc170070130 2023-06-10T18:56:19.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.584+0000 7fc157fff700 1 -- 172.21.15.104:0/1204645959 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+18627 (secure 0 0 0) 0x7fc1640576b0 con 0x7fc170070130 2023-06-10T18:56:19.589 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:19.589 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":127,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:19.230373+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:19.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 msgr2=0x7fc15c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 0x7fc15c06d190 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc160009130 tx=0x7fc160008030 comp rx=0 tx=0).stop 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 msgr2=0x7fc170148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c7f0 tx=0x7fc16400cb00 comp rx=0 tx=0).stop 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 shutdown_connections 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc15c06ace0 0x7fc15c06d190 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1700ce510 0x7fc1700c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc170070130 0x7fc170148340 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 --2- 172.21.15.104:0/1204645959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1700705a0 0x7fc1700cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 >> 172.21.15.104:0/1204645959 conn(0x7fc17006ab00 msgr2=0x7fc1700ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.585+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 shutdown_connections 2023-06-10T18:56:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.586+0000 7fc17593f700 1 -- 172.21.15.104:0/1204645959 wait complete. 2023-06-10T18:56:19.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:19.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.720+0000 7f2d164aa700 1 Processor -- start 2023-06-10T18:56:19.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.720+0000 7f2d164aa700 1 -- start start 2023-06-10T18:56:19.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d1015a410 0x7f2d100cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 0x7f2d100cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d100ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d100d0000 con 0x7f2d1015a410 2023-06-10T18:56:19.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d100b3610 con 0x7f2d1015e0f0 2023-06-10T18:56:19.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d164aa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100b38d0 con 0x7f2d100cd6a0 2023-06-10T18:56:19.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d0ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d1015a410 0x7f2d100cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d0ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2d1015a410 0x7f2d100cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37236/0 (socket says 172.21.15.104:37236) 2023-06-10T18:56:19.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.721+0000 7f2d0ffff700 1 -- 172.21.15.104:0/200285759 learned_addr learned my addr 172.21.15.104:0/200285759 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:19.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0f7fe700 1 --2- 172.21.15.104:0/200285759 >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 0x7f2d100cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d14a47700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d100ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 -- 172.21.15.104:0/200285759 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 msgr2=0x7f2d100ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:19.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d100ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 -- 172.21.15.104:0/200285759 >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 msgr2=0x7f2d100cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:19.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/200285759 >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 0x7f2d100cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 -- 172.21.15.104:0/200285759 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d100cdae0 con 0x7f2d1015a410 2023-06-10T18:56:19.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0f7fe700 1 --2- 172.21.15.104:0/200285759 >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 0x7f2d100cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:19.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d14a47700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d100ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:19.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015a410 0x7f2d100cfac0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2d100afb10 tx=0x7f2d00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88445e9c602fc60f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0effd700 1 -- 172.21.15.104:0/200285759 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000b870 con 0x7f2d1015a410 2023-06-10T18:56:19.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0effd700 1 -- 172.21.15.104:0/200285759 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000b9d0 con 0x7f2d1015a410 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d0effd700 1 -- 172.21.15.104:0/200285759 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000bd60 con 0x7f2d1015a410 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 -- 172.21.15.104:0/200285759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015a410 msgr2=0x7f2d100cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015a410 0x7f2d100cfac0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2d100afb10 tx=0x7f2d00009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 -- 172.21.15.104:0/200285759 shutdown_connections 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015a410 0x7f2d100cfac0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 --2- 172.21.15.104:0/200285759 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d100ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 --2- 172.21.15.104:0/200285759 >> v2:172.21.15.114:3300/0 conn(0x7f2d1015e0f0 0x7f2d100cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.722+0000 7f2d164aa700 1 -- 172.21.15.104:0/200285759 >> 172.21.15.104:0/200285759 conn(0x7f2d10069e70 msgr2=0x7f2d1006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.723+0000 7f2d164aa700 1 -- 172.21.15.104:0/200285759 shutdown_connections 2023-06-10T18:56:19.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.723+0000 7f2d164aa700 1 -- 172.21.15.104:0/200285759 wait complete. 2023-06-10T18:56:19.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.724+0000 7f2d164aa700 1 Processor -- start 2023-06-10T18:56:19.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.724+0000 7f2d164aa700 1 -- start start 2023-06-10T18:56:19.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d101efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 0x7f2d10156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1015d6a0 con 0x7f2d1015e0f0 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100c7b30 con 0x7f2d1015a410 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d164aa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1014d560 con 0x7f2d100cd6a0 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d14a47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d14a47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37244/0 (socket says 172.21.15.104:37244) 2023-06-10T18:56:19.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d14a47700 1 -- 172.21.15.104:0/1416960995 learned_addr learned my addr 172.21.15.104:0/1416960995 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:19.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d101efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d0f7fe700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 0x7f2d10156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.725+0000 7f2d14a47700 1 -- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 msgr2=0x7f2d101efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d14a47700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d101efa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d14a47700 1 -- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 msgr2=0x7f2d10156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d14a47700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 0x7f2d10156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d14a47700 1 -- 172.21.15.104:0/1416960995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009550 con 0x7f2d1015e0f0 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d0f7fe700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 0x7f2d10156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:19.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d14a47700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800cd50 tx=0x7f2d0800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800f480 con 0x7f2d1015e0f0 2023-06-10T18:56:19.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800f600 con 0x7f2d1015e0f0 2023-06-10T18:56:19.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1015ce00 con 0x7f2d1015e0f0 2023-06-10T18:56:19.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800f9b0 con 0x7f2d1015e0f0 2023-06-10T18:56:19.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.726+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1015d3c0 con 0x7f2d1015e0f0 2023-06-10T18:56:19.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.727+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d100c7c70 con 0x7f2d1015e0f0 2023-06-10T18:56:19.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.727+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2d0800b040 con 0x7f2d1015e0f0 2023-06-10T18:56:19.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.727+0000 7f2d0cff9700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 0x7f2cf006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:19.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.728+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2d08089d60 con 0x7f2d1015e0f0 2023-06-10T18:56:19.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.728+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 0x7f2cf006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:19.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.728+0000 7f2d0ffff700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 0x7f2cf006d190 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009c10 tx=0x7f2d00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:19.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.732+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2d100c7c70 con 0x7f2d1015e0f0 2023-06-10T18:56:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.922+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d100c7c70 con 0x7f2d1015e0f0 2023-06-10T18:56:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.923+0000 7f2d0cff9700 1 -- 172.21.15.104:0/1416960995 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+18627 (secure 0 0 0) 0x7f2d100c7c70 con 0x7f2d1015e0f0 2023-06-10T18:56:19.929 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:19.929 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":127,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:19.230373+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 msgr2=0x7f2cf006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 0x7f2cf006d190 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009c10 tx=0x7f2d00008030 comp rx=0 tx=0).stop 2023-06-10T18:56:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 msgr2=0x7f2d1015bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800cd50 tx=0x7f2d0800a560 comp rx=0 tx=0).stop 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 shutdown_connections 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d1015e0f0 0x7f2d1015bdd0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d100cd6a0 0x7f2d101efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cf006ace0 0x7f2cf006d190 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 --2- 172.21.15.104:0/1416960995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d1015a410 0x7f2d10156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 >> 172.21.15.104:0/1416960995 conn(0x7f2d10069e70 msgr2=0x7f2d100b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 shutdown_connections 2023-06-10T18:56:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:19.925+0000 7f2d164aa700 1 -- 172.21.15.104:0/1416960995 wait complete. 2023-06-10T18:56:19.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:56:20.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.059+0000 7f2e81cdc700 1 Processor -- start 2023-06-10T18:56:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 -- start start 2023-06-10T18:56:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 0x7f2e7c15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2e7c0af940 0x7f2e7c15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c15aa00 con 0x7f2e7c15e720 2023-06-10T18:56:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c15dab0 con 0x7f2e7c0af940 2023-06-10T18:56:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e81cdc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c154970 con 0x7f2e7c1545a0 2023-06-10T18:56:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e72ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2e7c0af940 0x7f2e7c15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.060+0000 7f2e72ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2e7c0af940 0x7f2e7c15e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37250/0 (socket says 172.21.15.104:37250) 2023-06-10T18:56:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1138450089 learned_addr learned my addr 172.21.15.104:0/1138450089 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e7b7fe700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 0x7f2e7c15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1138450089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 msgr2=0x7f2e7c15a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:20.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 0x7f2e7c15a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:20.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1138450089 >> v1:172.21.15.114:6789/0 conn(0x7f2e7c15e720 legacy=0x7f2e7c0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:20.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1138450089 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0cb0b0 con 0x7f2e7c0af940 2023-06-10T18:56:20.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e7b7fe700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 0x7f2e7c15a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:20.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e72ffd700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c0af940 0x7f2e7c15e1e0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f2e640098f0 tx=0x7f2e64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a84fdb2585d3bf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:20.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e7affd700 1 -- 172.21.15.104:0/1138450089 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400b8a0 con 0x7f2e7c0af940 2023-06-10T18:56:20.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e7affd700 1 -- 172.21.15.104:0/1138450089 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6400ba00 con 0x7f2e7c0af940 2023-06-10T18:56:20.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.061+0000 7f2e7affd700 1 -- 172.21.15.104:0/1138450089 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400bd90 con 0x7f2e7c0af940 2023-06-10T18:56:20.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1138450089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c0af940 msgr2=0x7f2e7c15e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:20.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c0af940 0x7f2e7c15e1e0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f2e640098f0 tx=0x7f2e64009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:20.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1138450089 shutdown_connections 2023-06-10T18:56:20.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c1545a0 0x7f2e7c15a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:20.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1138450089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c0af940 0x7f2e7c15e1e0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:20.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1138450089 >> 172.21.15.104:0/1138450089 conn(0x7f2e7c06e370 msgr2=0x7f2e7c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:20.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1138450089 shutdown_connections 2023-06-10T18:56:20.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.062+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1138450089 wait complete. 2023-06-10T18:56:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.063+0000 7f2e81cdc700 1 Processor -- start 2023-06-10T18:56:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.063+0000 7f2e81cdc700 1 -- start start 2023-06-10T18:56:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c0af940 0x7f2e7c0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7c15e720 0x7f2e7c156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0b00a0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0b0c20 con 0x7f2e7c15e720 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e81cdc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0b02f0 con 0x7f2e7c0af940 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37256/0 (socket says 172.21.15.104:37256) 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1839662531 learned_addr learned my addr 172.21.15.104:0/1839662531 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:20.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e7b7fe700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c0af940 0x7f2e7c0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e7bfff700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7c15e720 0x7f2e7c156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c0af940 msgr2=0x7f2e7c0b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c0af940 0x7f2e7c0b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7c15e720 msgr2=0x7f2e7c156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7c15e720 0x7f2e7c156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 -- 172.21.15.104:0/1839662531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e64009550 con 0x7f2e7c1545a0 2023-06-10T18:56:20.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e72ffd700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f2e64014040 tx=0x7f2e64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.064+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e640036a0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.065+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6401e070 con 0x7f2e7c1545a0 2023-06-10T18:56:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.065+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c0b05d0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.065+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64003a60 con 0x7f2e7c1545a0 2023-06-10T18:56:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.065+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c156fa0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.065+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c15d8d0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.066+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2e64003bc0 con 0x7f2e7c1545a0 2023-06-10T18:56:20.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.066+0000 7f2e78ff9700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 0x7f2e5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.066+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2e64020070 con 0x7f2e7c1545a0 2023-06-10T18:56:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.067+0000 7f2e7b7fe700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 0x7f2e5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.067+0000 7f2e7b7fe700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 0x7f2e5c06d190 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c1f3740 tx=0x7f2e6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:20.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.071+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2e64055a50 con 0x7f2e7c1545a0 2023-06-10T18:56:20.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:20.305+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2e7c074050 con 0x7f2e7c1545a0 2023-06-10T18:56:21.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.253+0000 7f2e78ff9700 1 -- 172.21.15.104:0/1839662531 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v79) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2e64059070 con 0x7f2e7c1545a0 2023-06-10T18:56:21.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 msgr2=0x7f2e5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 0x7f2e5c06d190 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c1f3740 tx=0x7f2e6c008040 comp rx=0 tx=0).stop 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 msgr2=0x7f2e7c0b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f2e64014040 tx=0x7f2e64004750 comp rx=0 tx=0).stop 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 shutdown_connections 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e5c06ace0 0x7f2e5c06d190 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7c15e720 0x7f2e7c156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e7c0af940 0x7f2e7c0b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 --2- 172.21.15.104:0/1839662531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2e7c1545a0 0x7f2e7c0b10b0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.256+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 >> 172.21.15.104:0/1839662531 conn(0x7f2e7c06e370 msgr2=0x7f2e7c159910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.257+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 shutdown_connections 2023-06-10T18:56:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.257+0000 7f2e81cdc700 1 -- 172.21.15.104:0/1839662531 wait complete. 2023-06-10T18:56:21.276 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:56:21.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:56:21.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.400+0000 7fc46a310700 1 Processor -- start 2023-06-10T18:56:21.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.400+0000 7fc46a310700 1 -- start start 2023-06-10T18:56:21.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc46a310700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc46415c500 0x7fc4640cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:21.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc46a310700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4640ce040 con 0x7fc464069e70 2023-06-10T18:56:21.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc46a310700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46415c8d0 con 0x7fc46406e3e0 2023-06-10T18:56:21.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc46a310700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46406da40 con 0x7fc46415c500 2023-06-10T18:56:21.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc463fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc46415c500 0x7fc4640cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc4637fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc464069e70 0x7fc46406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35090/0 (socket says 172.21.15.104:35090) 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc4637fe700 1 -- 172.21.15.104:0/4284351884 learned_addr learned my addr 172.21.15.104:0/4284351884 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.401+0000 7fc463fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc46415c500 0x7fc4640cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37262/0 (socket says 172.21.15.104:37262) 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc463fff700 1 -- 172.21.15.104:0/4284351884 >> v1:172.21.15.114:6789/0 conn(0x7fc46406e3e0 legacy=0x7fc4640c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc463fff700 1 -- 172.21.15.104:0/4284351884 >> v1:172.21.15.104:6789/0 conn(0x7fc464069e70 legacy=0x7fc46406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc463fff700 1 -- 172.21.15.104:0/4284351884 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46406a9c0 con 0x7fc46415c500 2023-06-10T18:56:21.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc463fff700 1 --2- 172.21.15.104:0/4284351884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc46415c500 0x7fc4640cedc0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc458009820 tx=0x7fc458009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27897174c31c67 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:21.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc462ffd700 1 -- 172.21.15.104:0/4284351884 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458005e30 con 0x7fc46415c500 2023-06-10T18:56:21.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc462ffd700 1 -- 172.21.15.104:0/4284351884 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458006030 con 0x7fc46415c500 2023-06-10T18:56:21.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.402+0000 7fc462ffd700 1 -- 172.21.15.104:0/4284351884 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580063e0 con 0x7fc46415c500 2023-06-10T18:56:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 -- 172.21.15.104:0/4284351884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc46415c500 msgr2=0x7fc4640cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 --2- 172.21.15.104:0/4284351884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc46415c500 0x7fc4640cedc0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc458009820 tx=0x7fc458009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 -- 172.21.15.104:0/4284351884 shutdown_connections 2023-06-10T18:56:21.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 --2- 172.21.15.104:0/4284351884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc46415c500 0x7fc4640cedc0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 -- 172.21.15.104:0/4284351884 >> 172.21.15.104:0/4284351884 conn(0x7fc46406b8a0 msgr2=0x7fc4640cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 -- 172.21.15.104:0/4284351884 shutdown_connections 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.403+0000 7fc46a310700 1 -- 172.21.15.104:0/4284351884 wait complete. 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.404+0000 7fc46a310700 1 Processor -- start 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.404+0000 7fc46a310700 1 -- start start 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:21.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc46406e3e0 0x7fc4640c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc46415c500 0x7fc464159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640c6c20 con 0x7fc464069e70 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640c7000 con 0x7fc46415c500 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640c72e0 con 0x7fc46406e3e0 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37266/0 (socket says 172.21.15.104:37266) 2023-06-10T18:56:21.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 -- 172.21.15.104:0/1421365845 learned_addr learned my addr 172.21.15.104:0/1421365845 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 -- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc46406e3e0 msgr2=0x7fc4640c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc4637fe700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc46406e3e0 0x7fc4640c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc4688ad700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc46415c500 0x7fc464159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc46406e3e0 0x7fc4640c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 -- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc46415c500 msgr2=0x7fc464159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc46415c500 0x7fc464159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 -- 172.21.15.104:0/1421365845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458009430 con 0x7fc464069e70 2023-06-10T18:56:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc463fff700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fc458000c00 tx=0x7fc458004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458004250 con 0x7fc464069e70 2023-06-10T18:56:21.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4580043d0 con 0x7fc464069e70 2023-06-10T18:56:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4640c75c0 con 0x7fc464069e70 2023-06-10T18:56:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.405+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4580036a0 con 0x7fc464069e70 2023-06-10T18:56:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.406+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4640b34c0 con 0x7fc464069e70 2023-06-10T18:56:21.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.406+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4640b3040 con 0x7fc464069e70 2023-06-10T18:56:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.407+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc458022070 con 0x7fc464069e70 2023-06-10T18:56:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.407+0000 7fc460ff9700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 0x7fc45006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:21.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.407+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc45808aff0 con 0x7fc464069e70 2023-06-10T18:56:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.408+0000 7fc4637fe700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 0x7fc45006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.408+0000 7fc4637fe700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 0x7fc45006d190 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc4640c5d20 tx=0x7fc454009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:21.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.412+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc4640b3040 con 0x7fc464069e70 2023-06-10T18:56:21.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:21.642+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc4640b3040 con 0x7fc464069e70 2023-06-10T18:56:22.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.261+0000 7fc460ff9700 1 -- 172.21.15.104:0/1421365845 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v80) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc4640b3040 con 0x7fc464069e70 2023-06-10T18:56:22.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 msgr2=0x7fc45006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 0x7fc45006d190 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc4640c5d20 tx=0x7fc454009040 comp rx=0 tx=0).stop 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 msgr2=0x7fc46415a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fc458000c00 tx=0x7fc458004750 comp rx=0 tx=0).stop 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 shutdown_connections 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc45006ace0 0x7fc45006d190 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc46415c500 0x7fc464159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc464069e70 0x7fc46415a5e0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 --2- 172.21.15.104:0/1421365845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc46406e3e0 0x7fc4640c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 >> 172.21.15.104:0/1421365845 conn(0x7fc46406b8a0 msgr2=0x7fc4640ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 shutdown_connections 2023-06-10T18:56:22.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.263+0000 7fc46a310700 1 -- 172.21.15.104:0/1421365845 wait complete. 2023-06-10T18:56:22.282 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:56:22.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:22.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 Processor -- start 2023-06-10T18:56:22.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 -- start start 2023-06-10T18:56:22.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a5006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a5006e4b0 0x7f7a500c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5006d290 con 0x7f7a500ca100 2023-06-10T18:56:22.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a50071640 con 0x7f7a5006e4b0 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.401+0000 7f7a56c14700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500aef00 con 0x7f7a5006e0e0 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a551b1700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7a500ca100 0x7f7a500ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35106/0 (socket says 172.21.15.104:35106) 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a551b1700 1 -- 172.21.15.104:0/2588327724 learned_addr learned my addr 172.21.15.104:0/2588327724 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a549b0700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a5006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/2588327724 >> v2:172.21.15.114:3300/0 conn(0x7f7a5006e4b0 0x7f7a500c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410831056 0 0) 0x7f7a5006d290 con 0x7f7a500ca100 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a400035b0 con 0x7f7a500ca100 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338527550 0 0) 0x7f7a400035b0 con 0x7f7a500ca100 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 msgr2=0x7f7a5006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a5006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 >> v2:172.21.15.114:3300/0 conn(0x7f7a5006e4b0 msgr2=0x7f7a500c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 --2- 172.21.15.104:0/2588327724 >> v2:172.21.15.114:3300/0 conn(0x7f7a5006e4b0 0x7f7a500c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5006d4c0 con 0x7f7a500ca100 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7a480019f0 con 0x7f7a500ca100 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 >> v1:172.21.15.104:6789/0 conn(0x7f7a500ca100 legacy=0x7f7a500ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a40003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a40003f80 0x7f7a40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40004d90 0x7f7a40005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50071640 con 0x7f7a40003f80 2023-06-10T18:56:22.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500aef00 con 0x7f7a40004d90 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4000ba40 con 0x7f7a5006e0e0 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a40003f80 0x7f7a40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a549b0700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a40003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.402+0000 7f7a551b1700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40004d90 0x7f7a40005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 -- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 msgr2=0x7f7a40003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a5006e0e0 0x7f7a40003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 -- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40004d90 msgr2=0x7f7a40005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40004d90 0x7f7a40005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 -- 172.21.15.104:0/2588327724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400035b0 con 0x7f7a40003f80 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a40003f80 0x7f7a40004850 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009b60 tx=0x7f7a40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdeb107b4f6559e9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00e660 con 0x7f7a40003f80 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c00e7c0 con 0x7f7a40003f80 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a4f7fe700 1 -- 172.21.15.104:0/2588327724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00eb30 con 0x7f7a40003f80 2023-06-10T18:56:22.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a56c14700 1 -- 172.21.15.104:0/2588327724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a40003f80 msgr2=0x7f7a40004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a56c14700 1 --2- 172.21.15.104:0/2588327724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a40003f80 0x7f7a40004850 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009b60 tx=0x7f7a40005880 comp rx=0 tx=0).stop 2023-06-10T18:56:22.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a551b1700 1 -- 172.21.15.104:0/2588327724 reap_dead start 2023-06-10T18:56:22.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.403+0000 7f7a56c14700 1 -- 172.21.15.104:0/2588327724 shutdown_connections 2023-06-10T18:56:22.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.404+0000 7f7a56c14700 1 -- 172.21.15.104:0/2588327724 >> 172.21.15.104:0/2588327724 conn(0x7f7a5006b8a0 msgr2=0x7f7a500cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:22.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.404+0000 7f7a56c14700 1 -- 172.21.15.104:0/2588327724 shutdown_connections 2023-06-10T18:56:22.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.404+0000 7f7a56c14700 1 -- 172.21.15.104:0/2588327724 wait complete. 2023-06-10T18:56:22.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.405+0000 7f7a56c14700 1 Processor -- start 2023-06-10T18:56:22.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.405+0000 7f7a56c14700 1 -- start start 2023-06-10T18:56:22.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 0x7f7a50154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 0x7f7a500b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500b1380 con 0x7f7a500b0f10 2023-06-10T18:56:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a501543a0 con 0x7f7a40003f80 2023-06-10T18:56:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a501f1ac0 con 0x7f7a501f1650 2023-06-10T18:56:22.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37290/0 (socket says 172.21.15.104:37290) 2023-06-10T18:56:22.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 -- 172.21.15.104:0/1085074534 learned_addr learned my addr 172.21.15.104:0/1085074534 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:22.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a549b0700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 0x7f7a50154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 0x7f7a500b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 -- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 msgr2=0x7f7a500b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 0x7f7a500b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 -- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 msgr2=0x7f7a50154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 0x7f7a50154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 -- 172.21.15.104:0/1085074534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a501f6120 con 0x7f7a500b0f10 2023-06-10T18:56:22.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a4ffff700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 0x7f7a500b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:22.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a549b0700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 0x7f7a50154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:22.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a551b1700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cc010 tx=0x7f7a4800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4800ae10 con 0x7f7a500b0f10 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a480133f0 con 0x7f7a500b0f10 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480137a0 con 0x7f7a500b0f10 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a50153a70 con 0x7f7a500b0f10 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.406+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a50154030 con 0x7f7a500b0f10 2023-06-10T18:56:22.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.407+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a501541f0 con 0x7f7a500b0f10 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.408+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a48013900 con 0x7f7a500b0f10 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.408+0000 7f7a4d7fa700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 0x7f7a3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.408+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7a48089f60 con 0x7f7a500b0f10 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.408+0000 7f7a549b0700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 0x7f7a3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.409+0000 7f7a549b0700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 0x7f7a3406d190 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009b60 tx=0x7f7a44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.412+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a48054ed0 con 0x7f7a500b0f10 2023-06-10T18:56:22.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.638+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a500cddc0 con 0x7f7a500b0f10 2023-06-10T18:56:22.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.639+0000 7f7a4d7fa700 1 -- 172.21.15.104:0/1085074534 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f7a480584f0 con 0x7f7a500b0f10 2023-06-10T18:56:22.644 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:22.644 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.640+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 msgr2=0x7f7a3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.640+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 0x7f7a3406d190 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009b60 tx=0x7f7a44008040 comp rx=0 tx=0).stop 2023-06-10T18:56:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.640+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 msgr2=0x7f7a501f5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cc010 tx=0x7f7a4800a640 comp rx=0 tx=0).stop 2023-06-10T18:56:22.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 shutdown_connections 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a3406ace0 0x7f7a3406d190 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a40003f80 0x7f7a50154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a500b0f10 0x7f7a501f5380 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 --2- 172.21.15.104:0/1085074534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a501f1650 0x7f7a500b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 >> 172.21.15.104:0/1085074534 conn(0x7f7a5006b8a0 msgr2=0x7f7a500cb650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 shutdown_connections 2023-06-10T18:56:22.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.641+0000 7f7a56c14700 1 -- 172.21.15.104:0/1085074534 wait complete. 2023-06-10T18:56:22.648 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:22.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:22.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.772+0000 7f8f88a49700 1 Processor -- start 2023-06-10T18:56:22.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.773+0000 7f8f88a49700 1 -- start start 2023-06-10T18:56:22.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f88a49700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8f840aeaa0 0x7f8f840cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f88a49700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840c9e60 con 0x7f8f840cd680 2023-06-10T18:56:22.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f88a49700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840cdaa0 con 0x7f8f840b1080 2023-06-10T18:56:22.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f88a49700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840c9bc0 con 0x7f8f840aeaa0 2023-06-10T18:56:22.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8f840aeaa0 0x7f8f840cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f82d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8f840cd680 0x7f8f8406dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35126/0 (socket says 172.21.15.104:35126) 2023-06-10T18:56:22.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f82d9d700 1 -- 172.21.15.104:0/161146138 learned_addr learned my addr 172.21.15.104:0/161146138 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:22.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8f840aeaa0 0x7f8f840cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37294/0 (socket says 172.21.15.104:37294) 2023-06-10T18:56:22.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 -- 172.21.15.104:0/161146138 >> v1:172.21.15.114:6789/0 conn(0x7f8f840b1080 legacy=0x7f8f840ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:22.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8159a700 1 -- 172.21.15.104:0/161146138 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734610751 0 0) 0x7f8f840c9e60 con 0x7f8f840cd680 2023-06-10T18:56:22.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 -- 172.21.15.104:0/161146138 >> v1:172.21.15.104:6789/0 conn(0x7f8f840cd680 legacy=0x7f8f8406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 -- 172.21.15.104:0/161146138 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840aee70 con 0x7f8f840aeaa0 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.774+0000 7f8f8259c700 1 --2- 172.21.15.104:0/161146138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840aeaa0 0x7f8f840cd140 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009820 tx=0x7f8f78009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4e13c2e0b0dde8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f8159a700 1 -- 172.21.15.104:0/161146138 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78005e30 con 0x7f8f840aeaa0 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f8159a700 1 -- 172.21.15.104:0/161146138 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f78004030 con 0x7f8f840aeaa0 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f8159a700 1 -- 172.21.15.104:0/161146138 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f780043c0 con 0x7f8f840aeaa0 2023-06-10T18:56:22.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 -- 172.21.15.104:0/161146138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840aeaa0 msgr2=0x7f8f840cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 --2- 172.21.15.104:0/161146138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840aeaa0 0x7f8f840cd140 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009820 tx=0x7f8f78009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:22.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 -- 172.21.15.104:0/161146138 shutdown_connections 2023-06-10T18:56:22.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 --2- 172.21.15.104:0/161146138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840aeaa0 0x7f8f840cd140 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 -- 172.21.15.104:0/161146138 >> 172.21.15.104:0/161146138 conn(0x7f8f8406b8a0 msgr2=0x7f8f840cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 -- 172.21.15.104:0/161146138 shutdown_connections 2023-06-10T18:56:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.775+0000 7f8f88a49700 1 -- 172.21.15.104:0/161146138 wait complete. 2023-06-10T18:56:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.776+0000 7f8f88a49700 1 Processor -- start 2023-06-10T18:56:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 -- start start 2023-06-10T18:56:22.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f840aeaa0 0x7f8f84156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 0x7f8f8415af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84159e80 con 0x7f8f840cd680 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415a160 con 0x7f8f840b1080 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f88a49700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8415a440 con 0x7f8f840aeaa0 2023-06-10T18:56:22.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f82d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f82d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37304/0 (socket says 172.21.15.104:37304) 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.777+0000 7f8f82d9d700 1 -- 172.21.15.104:0/2358074487 learned_addr learned my addr 172.21.15.104:0/2358074487 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f81d9b700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 0x7f8f8415af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f8259c700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f840aeaa0 0x7f8f84156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 -- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f840aeaa0 msgr2=0x7f8f84156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f840aeaa0 0x7f8f84156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 -- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 msgr2=0x7f8f8415af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 0x7f8f8415af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 -- 172.21.15.104:0/2358074487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78009430 con 0x7f8f840cd680 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f81d9b700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 0x7f8f8415af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f82d9d700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00cb30 tx=0x7f8f7c00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00ab50 con 0x7f8f840cd680 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c00acb0 con 0x7f8f840cd680 2023-06-10T18:56:22.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8415a720 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c007310 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.778+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f840b4280 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.779+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8415ac40 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.779+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8f7c007470 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.780+0000 7f8f72ffd700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 0x7f8f6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.780+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8f7c0893f0 con 0x7f8f840cd680 2023-06-10T18:56:22.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.780+0000 7f8f8259c700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 0x7f8f6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:22.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.780+0000 7f8f8259c700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 0x7f8f6406d190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009770 tx=0x7f8f78008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:22.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.784+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8f8415ac40 con 0x7f8f840cd680 2023-06-10T18:56:22.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.970+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f8415ac40 con 0x7f8f840cd680 2023-06-10T18:56:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.971+0000 7f8f72ffd700 1 -- 172.21.15.104:0/2358074487 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+18614 (secure 0 0 0) 0x7f8f8415ac40 con 0x7f8f840cd680 2023-06-10T18:56:22.977 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:22.978 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":128,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:20.232508+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:22.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 msgr2=0x7f8f6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 0x7f8f6406d190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009770 tx=0x7f8f78008070 comp rx=0 tx=0).stop 2023-06-10T18:56:22.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 msgr2=0x7f8f840b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:22.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00cb30 tx=0x7f8f7c00cef0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 shutdown_connections 2023-06-10T18:56:22.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8f840cd680 0x7f8f840b3d40 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f840aeaa0 0x7f8f84156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8f6406ace0 0x7f8f6406d190 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 --2- 172.21.15.104:0/2358074487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f840b1080 0x7f8f8415af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 >> 172.21.15.104:0/2358074487 conn(0x7f8f8406b8a0 msgr2=0x7f8f84158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 shutdown_connections 2023-06-10T18:56:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:22.974+0000 7f8f88a49700 1 -- 172.21.15.104:0/2358074487 wait complete. 2023-06-10T18:56:22.991 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:56:22.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:56:23.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.107+0000 7f6e36021700 1 Processor -- start 2023-06-10T18:56:23.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.107+0000 7f6e36021700 1 -- start start 2023-06-10T18:56:23.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 0x7f6e300cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 0x7f6e300ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e300b1970 0x7f6e300ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3015a8c0 con 0x7f6e300b1970 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300b17a0 con 0x7f6e300b0d90 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e36021700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300b14c0 con 0x7f6e300cd9e0 2023-06-10T18:56:23.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e2ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e300b1970 0x7f6e300ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e2ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e300b1970 0x7f6e300ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37308/0 (socket says 172.21.15.104:37308) 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e2ffff700 1 -- 172.21.15.104:0/2146682415 learned_addr learned my addr 172.21.15.104:0/2146682415 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e2effd700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 0x7f6e300ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.108+0000 7f6e2f7fe700 1 --2- 172.21.15.104:0/2146682415 >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 0x7f6e300cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 -- 172.21.15.104:0/2146682415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 msgr2=0x7f6e300ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 0x7f6e300ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 -- 172.21.15.104:0/2146682415 >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 msgr2=0x7f6e300cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 --2- 172.21.15.104:0/2146682415 >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 0x7f6e300cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 -- 172.21.15.104:0/2146682415 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300cc9c0 con 0x7f6e300b1970 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2effd700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 0x7f6e300ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2f7fe700 1 --2- 172.21.15.104:0/2146682415 >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 0x7f6e300cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2ffff700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e300ce930 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f6e300713d0 tx=0x7f6e2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ca8f0d73c19a68 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2e7fc700 1 -- 172.21.15.104:0/2146682415 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800e070 con 0x7f6e300b1970 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2e7fc700 1 -- 172.21.15.104:0/2146682415 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800b9c0 con 0x7f6e300b1970 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e2e7fc700 1 -- 172.21.15.104:0/2146682415 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800bd70 con 0x7f6e300b1970 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 -- 172.21.15.104:0/2146682415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 msgr2=0x7f6e300ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e300ce930 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f6e300713d0 tx=0x7f6e2800b330 comp rx=0 tx=0).stop 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 -- 172.21.15.104:0/2146682415 shutdown_connections 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e300ce930 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 --2- 172.21.15.104:0/2146682415 >> v2:172.21.15.114:3300/0 conn(0x7f6e300b0d90 0x7f6e300cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 --2- 172.21.15.104:0/2146682415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300cd9e0 0x7f6e300ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 -- 172.21.15.104:0/2146682415 >> 172.21.15.104:0/2146682415 conn(0x7f6e30089fd0 msgr2=0x7f6e300d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:23.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.109+0000 7f6e36021700 1 -- 172.21.15.104:0/2146682415 shutdown_connections 2023-06-10T18:56:23.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.110+0000 7f6e36021700 1 -- 172.21.15.104:0/2146682415 wait complete. 2023-06-10T18:56:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.111+0000 7f6e36021700 1 Processor -- start 2023-06-10T18:56:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.111+0000 7f6e36021700 1 -- start start 2023-06-10T18:56:23.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300b0d90 0x7f6e300c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 0x7f6e30158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30158ba0 con 0x7f6e300b1970 2023-06-10T18:56:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30158e80 con 0x7f6e300cd9e0 2023-06-10T18:56:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e36021700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30159160 con 0x7f6e300b0d90 2023-06-10T18:56:23.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e2effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e2effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37312/0 (socket says 172.21.15.104:37312) 2023-06-10T18:56:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.112+0000 7f6e2effd700 1 -- 172.21.15.104:0/2105380435 learned_addr learned my addr 172.21.15.104:0/2105380435 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2ffff700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 0x7f6e30158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 -- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300b0d90 msgr2=0x7f6e300c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2f7fe700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300b0d90 0x7f6e300c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300b0d90 0x7f6e300c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 -- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 msgr2=0x7f6e30158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 0x7f6e30158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 -- 172.21.15.104:0/2105380435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28009b30 con 0x7f6e300b1970 2023-06-10T18:56:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2ffff700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 0x7f6e30158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e2effd700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e810 tx=0x7f6e2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e20004230 con 0x7f6e300b1970 2023-06-10T18:56:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e20004390 con 0x7f6e300b1970 2023-06-10T18:56:23.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e30159440 con 0x7f6e300b1970 2023-06-10T18:56:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200036a0 con 0x7f6e300b1970 2023-06-10T18:56:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.113+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e30159c40 con 0x7f6e300b1970 2023-06-10T18:56:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.114+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30159960 con 0x7f6e300b1970 2023-06-10T18:56:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.114+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6e20003800 con 0x7f6e300b1970 2023-06-10T18:56:23.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.115+0000 7f6e17fff700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 0x7f6e1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.115+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6e20089090 con 0x7f6e300b1970 2023-06-10T18:56:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.115+0000 7f6e2f7fe700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 0x7f6e1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.115+0000 7f6e2f7fe700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 0x7f6e1c06d190 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6e301f2ec0 tx=0x7f6e24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.119+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6e30159960 con 0x7f6e300b1970 2023-06-10T18:56:23.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.302+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e300ccf80 con 0x7f6e1c06ace0 2023-06-10T18:56:23.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.311+0000 7f6e17fff700 1 -- 172.21.15.104:0/2105380435 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312046 (secure 0 0 0) 0x7f6e240133f0 con 0x7f6e1c06ace0 2023-06-10T18:56:23.325 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 msgr2=0x7f6e1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 0x7f6e1c06d190 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6e301f2ec0 tx=0x7f6e24008040 comp rx=0 tx=0).stop 2023-06-10T18:56:23.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 msgr2=0x7f6e30158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e810 tx=0x7f6e2000ebd0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 shutdown_connections 2023-06-10T18:56:23.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e1c06ace0 0x7f6e1c06d190 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e300cd9e0 0x7f6e30158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e300b0d90 0x7f6e300c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 --2- 172.21.15.104:0/2105380435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300b1970 0x7f6e30158050 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 >> 172.21.15.104:0/2105380435 conn(0x7f6e30089fd0 msgr2=0x7f6e300b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:23.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 shutdown_connections 2023-06-10T18:56:23.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.314+0000 7f6e36021700 1 -- 172.21.15.104:0/2105380435 wait complete. 2023-06-10T18:56:23.332 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:56:23.333 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2023-06-10T18:56:21.545585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233368,"kb_used_data":13656,"kb_used_omap":0,"kb_used_meta":219648,"kb_avail":754741352,"statfs":{"total":773094113280,"available":772855144448,"internally_reserved":0,"allocated":13983744,"data_stored":7390576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":224919552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000705"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518572+0000","last_change":"2023-06-10T18:56:19.261823+0000","last_active":"2023-06-10T18:56:20.518572+0000","last_peered":"2023-06-10T18:56:20.518572+0000","last_clean":"2023-06-10T18:56:20.518572+0000","last_became_active":"2023-06-10T18:56:19.261529+0000","last_became_peered":"2023-06-10T18:56:19.261529+0000","last_unstale":"2023-06-10T18:56:20.518572+0000","last_undegraded":"2023-06-10T18:56:20.518572+0000","last_fullsized":"2023-06-10T18:56:20.518572+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:26:19.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246958+0000","last_change":"2023-06-10T18:56:19.260237+0000","last_active":"2023-06-10T18:56:20.246958+0000","last_peered":"2023-06-10T18:56:20.246958+0000","last_clean":"2023-06-10T18:56:20.246958+0000","last_became_active":"2023-06-10T18:56:19.259890+0000","last_became_peered":"2023-06-10T18:56:19.259890+0000","last_unstale":"2023-06-10T18:56:20.246958+0000","last_undegraded":"2023-06-10T18:56:20.246958+0000","last_fullsized":"2023-06-10T18:56:20.246958+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:03.034563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519368+0000","last_change":"2023-06-10T18:56:19.257478+0000","last_active":"2023-06-10T18:56:20.519368+0000","last_peered":"2023-06-10T18:56:20.519368+0000","last_clean":"2023-06-10T18:56:20.519368+0000","last_became_active":"2023-06-10T18:56:19.257212+0000","last_became_peered":"2023-06-10T18:56:19.257212+0000","last_unstale":"2023-06-10T18:56:20.519368+0000","last_undegraded":"2023-06-10T18:56:20.519368+0000","last_fullsized":"2023-06-10T18:56:20.519368+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:29.569098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518794+0000","last_change":"2023-06-10T18:56:19.256061+0000","last_active":"2023-06-10T18:56:20.518794+0000","last_peered":"2023-06-10T18:56:20.518794+0000","last_clean":"2023-06-10T18:56:20.518794+0000","last_became_active":"2023-06-10T18:56:19.255727+0000","last_became_peered":"2023-06-10T18:56:19.255727+0000","last_unstale":"2023-06-10T18:56:20.518794+0000","last_undegraded":"2023-06-10T18:56:20.518794+0000","last_fullsized":"2023-06-10T18:56:20.518794+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:35:05.467403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519756+0000","last_change":"2023-06-10T18:56:19.255086+0000","last_active":"2023-06-10T18:56:20.519756+0000","last_peered":"2023-06-10T18:56:20.519756+0000","last_clean":"2023-06-10T18:56:20.519756+0000","last_became_active":"2023-06-10T18:56:19.254530+0000","last_became_peered":"2023-06-10T18:56:19.254530+0000","last_unstale":"2023-06-10T18:56:20.519756+0000","last_undegraded":"2023-06-10T18:56:20.519756+0000","last_fullsized":"2023-06-10T18:56:20.519756+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:34:57.004440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518985+0000","last_change":"2023-06-10T18:56:19.256890+0000","last_active":"2023-06-10T18:56:20.518985+0000","last_peered":"2023-06-10T18:56:20.518985+0000","last_clean":"2023-06-10T18:56:20.518985+0000","last_became_active":"2023-06-10T18:56:19.256566+0000","last_became_peered":"2023-06-10T18:56:19.256566+0000","last_unstale":"2023-06-10T18:56:20.518985+0000","last_undegraded":"2023-06-10T18:56:20.518985+0000","last_fullsized":"2023-06-10T18:56:20.518985+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:42:31.082333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518668+0000","last_change":"2023-06-10T18:56:19.256763+0000","last_active":"2023-06-10T18:56:20.518668+0000","last_peered":"2023-06-10T18:56:20.518668+0000","last_clean":"2023-06-10T18:56:20.518668+0000","last_became_active":"2023-06-10T18:56:19.256465+0000","last_became_peered":"2023-06-10T18:56:19.256465+0000","last_unstale":"2023-06-10T18:56:20.518668+0000","last_undegraded":"2023-06-10T18:56:20.518668+0000","last_fullsized":"2023-06-10T18:56:20.518668+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:25.797334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519698+0000","last_change":"2023-06-10T18:56:19.255561+0000","last_active":"2023-06-10T18:56:20.519698+0000","last_peered":"2023-06-10T18:56:20.519698+0000","last_clean":"2023-06-10T18:56:20.519698+0000","last_became_active":"2023-06-10T18:56:19.254725+0000","last_became_peered":"2023-06-10T18:56:19.254725+0000","last_unstale":"2023-06-10T18:56:20.519698+0000","last_undegraded":"2023-06-10T18:56:20.519698+0000","last_fullsized":"2023-06-10T18:56:20.519698+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:07.648227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519385+0000","last_change":"2023-06-10T18:56:19.255312+0000","last_active":"2023-06-10T18:56:20.519385+0000","last_peered":"2023-06-10T18:56:20.519385+0000","last_clean":"2023-06-10T18:56:20.519385+0000","last_became_active":"2023-06-10T18:56:19.254333+0000","last_became_peered":"2023-06-10T18:56:19.254333+0000","last_unstale":"2023-06-10T18:56:20.519385+0000","last_undegraded":"2023-06-10T18:56:20.519385+0000","last_fullsized":"2023-06-10T18:56:20.519385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:34.855752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247134+0000","last_change":"2023-06-10T18:56:19.260100+0000","last_active":"2023-06-10T18:56:20.247134+0000","last_peered":"2023-06-10T18:56:20.247134+0000","last_clean":"2023-06-10T18:56:20.247134+0000","last_became_active":"2023-06-10T18:56:19.259652+0000","last_became_peered":"2023-06-10T18:56:19.259652+0000","last_unstale":"2023-06-10T18:56:20.247134+0000","last_undegraded":"2023-06-10T18:56:20.247134+0000","last_fullsized":"2023-06-10T18:56:20.247134+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:55:49.705761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518585+0000","last_change":"2023-06-10T18:56:19.249394+0000","last_active":"2023-06-10T18:56:20.518585+0000","last_peered":"2023-06-10T18:56:20.518585+0000","last_clean":"2023-06-10T18:56:20.518585+0000","last_became_active":"2023-06-10T18:56:19.249071+0000","last_became_peered":"2023-06-10T18:56:19.249071+0000","last_unstale":"2023-06-10T18:56:20.518585+0000","last_undegraded":"2023-06-10T18:56:20.518585+0000","last_fullsized":"2023-06-10T18:56:20.518585+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:25:51.817123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519755+0000","last_change":"2023-06-10T18:56:19.254911+0000","last_active":"2023-06-10T18:56:20.519755+0000","last_peered":"2023-06-10T18:56:20.519755+0000","last_clean":"2023-06-10T18:56:20.519755+0000","last_became_active":"2023-06-10T18:56:19.254602+0000","last_became_peered":"2023-06-10T18:56:19.254602+0000","last_unstale":"2023-06-10T18:56:20.519755+0000","last_undegraded":"2023-06-10T18:56:20.519755+0000","last_fullsized":"2023-06-10T18:56:20.519755+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:34.658391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518574+0000","last_change":"2023-06-10T18:56:19.260895+0000","last_active":"2023-06-10T18:56:20.518574+0000","last_peered":"2023-06-10T18:56:20.518574+0000","last_clean":"2023-06-10T18:56:20.518574+0000","last_became_active":"2023-06-10T18:56:19.260606+0000","last_became_peered":"2023-06-10T18:56:19.260606+0000","last_unstale":"2023-06-10T18:56:20.518574+0000","last_undegraded":"2023-06-10T18:56:20.518574+0000","last_fullsized":"2023-06-10T18:56:20.518574+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:16:32.953543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246945+0000","last_change":"2023-06-10T18:56:19.256069+0000","last_active":"2023-06-10T18:56:20.246945+0000","last_peered":"2023-06-10T18:56:20.246945+0000","last_clean":"2023-06-10T18:56:20.246945+0000","last_became_active":"2023-06-10T18:56:19.255815+0000","last_became_peered":"2023-06-10T18:56:19.255815+0000","last_unstale":"2023-06-10T18:56:20.246945+0000","last_undegraded":"2023-06-10T18:56:20.246945+0000","last_fullsized":"2023-06-10T18:56:20.246945+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:34.658391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518620+0000","last_change":"2023-06-10T18:56:19.258711+0000","last_active":"2023-06-10T18:56:20.518620+0000","last_peered":"2023-06-10T18:56:20.518620+0000","last_clean":"2023-06-10T18:56:20.518620+0000","last_became_active":"2023-06-10T18:56:19.258467+0000","last_became_peered":"2023-06-10T18:56:19.258467+0000","last_unstale":"2023-06-10T18:56:20.518620+0000","last_undegraded":"2023-06-10T18:56:20.518620+0000","last_fullsized":"2023-06-10T18:56:20.518620+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:45.582902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245947+0000","last_change":"2023-06-10T18:56:19.251934+0000","last_active":"2023-06-10T18:56:20.245947+0000","last_peered":"2023-06-10T18:56:20.245947+0000","last_clean":"2023-06-10T18:56:20.245947+0000","last_became_active":"2023-06-10T18:56:19.251504+0000","last_became_peered":"2023-06-10T18:56:19.251504+0000","last_unstale":"2023-06-10T18:56:20.245947+0000","last_undegraded":"2023-06-10T18:56:20.245947+0000","last_fullsized":"2023-06-10T18:56:20.245947+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:03:54.139860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519955+0000","last_change":"2023-06-10T18:56:19.250149+0000","last_active":"2023-06-10T18:56:20.519955+0000","last_peered":"2023-06-10T18:56:20.519955+0000","last_clean":"2023-06-10T18:56:20.519955+0000","last_became_active":"2023-06-10T18:56:19.249849+0000","last_became_peered":"2023-06-10T18:56:19.249849+0000","last_unstale":"2023-06-10T18:56:20.519955+0000","last_undegraded":"2023-06-10T18:56:20.519955+0000","last_fullsized":"2023-06-10T18:56:20.519955+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:12:10.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245670+0000","last_change":"2023-06-10T18:56:19.261227+0000","last_active":"2023-06-10T18:56:20.245670+0000","last_peered":"2023-06-10T18:56:20.245670+0000","last_clean":"2023-06-10T18:56:20.245670+0000","last_became_active":"2023-06-10T18:56:19.260963+0000","last_became_peered":"2023-06-10T18:56:19.260963+0000","last_unstale":"2023-06-10T18:56:20.245670+0000","last_undegraded":"2023-06-10T18:56:20.245670+0000","last_fullsized":"2023-06-10T18:56:20.245670+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:11.098566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518716+0000","last_change":"2023-06-10T18:56:19.253019+0000","last_active":"2023-06-10T18:56:20.518716+0000","last_peered":"2023-06-10T18:56:20.518716+0000","last_clean":"2023-06-10T18:56:20.518716+0000","last_became_active":"2023-06-10T18:56:19.252773+0000","last_became_peered":"2023-06-10T18:56:19.252773+0000","last_unstale":"2023-06-10T18:56:20.518716+0000","last_undegraded":"2023-06-10T18:56:20.518716+0000","last_fullsized":"2023-06-10T18:56:20.518716+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:26.353786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245803+0000","last_change":"2023-06-10T18:56:19.246982+0000","last_active":"2023-06-10T18:56:20.245803+0000","last_peered":"2023-06-10T18:56:20.245803+0000","last_clean":"2023-06-10T18:56:20.245803+0000","last_became_active":"2023-06-10T18:56:19.246740+0000","last_became_peered":"2023-06-10T18:56:19.246740+0000","last_unstale":"2023-06-10T18:56:20.245803+0000","last_undegraded":"2023-06-10T18:56:20.245803+0000","last_fullsized":"2023-06-10T18:56:20.245803+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:52:47.611860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519926+0000","last_change":"2023-06-10T18:56:19.255278+0000","last_active":"2023-06-10T18:56:20.519926+0000","last_peered":"2023-06-10T18:56:20.519926+0000","last_clean":"2023-06-10T18:56:20.519926+0000","last_became_active":"2023-06-10T18:56:19.254821+0000","last_became_peered":"2023-06-10T18:56:19.254821+0000","last_unstale":"2023-06-10T18:56:20.519926+0000","last_undegraded":"2023-06-10T18:56:20.519926+0000","last_fullsized":"2023-06-10T18:56:20.519926+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:49.366207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518471+0000","last_change":"2023-06-10T18:56:19.249063+0000","last_active":"2023-06-10T18:56:20.518471+0000","last_peered":"2023-06-10T18:56:20.518471+0000","last_clean":"2023-06-10T18:56:20.518471+0000","last_became_active":"2023-06-10T18:56:19.248806+0000","last_became_peered":"2023-06-10T18:56:19.248806+0000","last_unstale":"2023-06-10T18:56:20.518471+0000","last_undegraded":"2023-06-10T18:56:20.518471+0000","last_fullsized":"2023-06-10T18:56:20.518471+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:36.508790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519150+0000","last_change":"2023-06-10T18:56:19.255772+0000","last_active":"2023-06-10T18:56:20.519150+0000","last_peered":"2023-06-10T18:56:20.519150+0000","last_clean":"2023-06-10T18:56:20.519150+0000","last_became_active":"2023-06-10T18:56:19.255516+0000","last_became_peered":"2023-06-10T18:56:19.255516+0000","last_unstale":"2023-06-10T18:56:20.519150+0000","last_undegraded":"2023-06-10T18:56:20.519150+0000","last_fullsized":"2023-06-10T18:56:20.519150+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:44.558873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518151+0000","last_change":"2023-06-10T18:56:19.244392+0000","last_active":"2023-06-10T18:56:20.518151+0000","last_peered":"2023-06-10T18:56:20.518151+0000","last_clean":"2023-06-10T18:56:20.518151+0000","last_became_active":"2023-06-10T18:56:19.244118+0000","last_became_peered":"2023-06-10T18:56:19.244118+0000","last_unstale":"2023-06-10T18:56:20.518151+0000","last_undegraded":"2023-06-10T18:56:20.518151+0000","last_fullsized":"2023-06-10T18:56:20.518151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.917468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247386+0000","last_change":"2023-06-10T18:56:19.255155+0000","last_active":"2023-06-10T18:56:20.247386+0000","last_peered":"2023-06-10T18:56:20.247386+0000","last_clean":"2023-06-10T18:56:20.247386+0000","last_became_active":"2023-06-10T18:56:19.254849+0000","last_became_peered":"2023-06-10T18:56:19.254849+0000","last_unstale":"2023-06-10T18:56:20.247386+0000","last_undegraded":"2023-06-10T18:56:20.247386+0000","last_fullsized":"2023-06-10T18:56:20.247386+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:36.508790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245820+0000","last_change":"2023-06-10T18:56:19.255220+0000","last_active":"2023-06-10T18:56:20.245820+0000","last_peered":"2023-06-10T18:56:20.245820+0000","last_clean":"2023-06-10T18:56:20.245820+0000","last_became_active":"2023-06-10T18:56:19.254910+0000","last_became_peered":"2023-06-10T18:56:19.254910+0000","last_unstale":"2023-06-10T18:56:20.245820+0000","last_undegraded":"2023-06-10T18:56:20.245820+0000","last_fullsized":"2023-06-10T18:56:20.245820+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:31:08.500560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518415+0000","last_change":"2023-06-10T18:56:19.261716+0000","last_active":"2023-06-10T18:56:20.518415+0000","last_peered":"2023-06-10T18:56:20.518415+0000","last_clean":"2023-06-10T18:56:20.518415+0000","last_became_active":"2023-06-10T18:56:19.261482+0000","last_became_peered":"2023-06-10T18:56:19.261482+0000","last_unstale":"2023-06-10T18:56:20.518415+0000","last_undegraded":"2023-06-10T18:56:20.518415+0000","last_fullsized":"2023-06-10T18:56:20.518415+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:35:41.624038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246612+0000","last_change":"2023-06-10T18:56:19.252081+0000","last_active":"2023-06-10T18:56:20.246612+0000","last_peered":"2023-06-10T18:56:20.246612+0000","last_clean":"2023-06-10T18:56:20.246612+0000","last_became_active":"2023-06-10T18:56:19.251785+0000","last_became_peered":"2023-06-10T18:56:19.251785+0000","last_unstale":"2023-06-10T18:56:20.246612+0000","last_undegraded":"2023-06-10T18:56:20.246612+0000","last_fullsized":"2023-06-10T18:56:20.246612+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:00.942133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518939+0000","last_change":"2023-06-10T18:56:19.265399+0000","last_active":"2023-06-10T18:56:20.518939+0000","last_peered":"2023-06-10T18:56:20.518939+0000","last_clean":"2023-06-10T18:56:20.518939+0000","last_became_active":"2023-06-10T18:56:19.265228+0000","last_became_peered":"2023-06-10T18:56:19.265228+0000","last_unstale":"2023-06-10T18:56:20.518939+0000","last_undegraded":"2023-06-10T18:56:20.518939+0000","last_fullsized":"2023-06-10T18:56:20.518939+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:46:42.880931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518680+0000","last_change":"2023-06-10T18:56:19.258766+0000","last_active":"2023-06-10T18:56:20.518680+0000","last_peered":"2023-06-10T18:56:20.518680+0000","last_clean":"2023-06-10T18:56:20.518680+0000","last_became_active":"2023-06-10T18:56:19.258522+0000","last_became_peered":"2023-06-10T18:56:19.258522+0000","last_unstale":"2023-06-10T18:56:20.518680+0000","last_undegraded":"2023-06-10T18:56:20.518680+0000","last_fullsized":"2023-06-10T18:56:20.518680+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:41:48.275015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520172+0000","last_change":"2023-06-10T18:56:19.262850+0000","last_active":"2023-06-10T18:56:20.520172+0000","last_peered":"2023-06-10T18:56:20.520172+0000","last_clean":"2023-06-10T18:56:20.520172+0000","last_became_active":"2023-06-10T18:56:19.262347+0000","last_became_peered":"2023-06-10T18:56:19.262347+0000","last_unstale":"2023-06-10T18:56:20.520172+0000","last_undegraded":"2023-06-10T18:56:20.520172+0000","last_fullsized":"2023-06-10T18:56:20.520172+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:47:32.514870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520262+0000","last_change":"2023-06-10T18:56:19.250803+0000","last_active":"2023-06-10T18:56:20.520262+0000","last_peered":"2023-06-10T18:56:20.520262+0000","last_clean":"2023-06-10T18:56:20.520262+0000","last_became_active":"2023-06-10T18:56:19.250389+0000","last_became_peered":"2023-06-10T18:56:19.250389+0000","last_unstale":"2023-06-10T18:56:20.520262+0000","last_undegraded":"2023-06-10T18:56:20.520262+0000","last_fullsized":"2023-06-10T18:56:20.520262+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:50:33.873950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245763+0000","last_change":"2023-06-10T18:56:19.252384+0000","last_active":"2023-06-10T18:56:20.245763+0000","last_peered":"2023-06-10T18:56:20.245763+0000","last_clean":"2023-06-10T18:56:20.245763+0000","last_became_active":"2023-06-10T18:56:19.252118+0000","last_became_peered":"2023-06-10T18:56:19.252118+0000","last_unstale":"2023-06-10T18:56:20.245763+0000","last_undegraded":"2023-06-10T18:56:20.245763+0000","last_fullsized":"2023-06-10T18:56:20.245763+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:55.643000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519711+0000","last_change":"2023-06-10T18:56:18.853256+0000","last_active":"2023-06-10T18:56:20.519711+0000","last_peered":"2023-06-10T18:56:20.519711+0000","last_clean":"2023-06-10T18:56:20.519711+0000","last_became_active":"2023-06-10T18:56:18.853041+0000","last_became_peered":"2023-06-10T18:56:18.853041+0000","last_unstale":"2023-06-10T18:56:20.519711+0000","last_undegraded":"2023-06-10T18:56:20.519711+0000","last_fullsized":"2023-06-10T18:56:20.519711+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:16.005018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519542+0000","last_change":"2023-06-10T18:56:19.259353+0000","last_active":"2023-06-10T18:56:20.519542+0000","last_peered":"2023-06-10T18:56:20.519542+0000","last_clean":"2023-06-10T18:56:20.519542+0000","last_became_active":"2023-06-10T18:56:19.259030+0000","last_became_peered":"2023-06-10T18:56:19.259030+0000","last_unstale":"2023-06-10T18:56:20.519542+0000","last_undegraded":"2023-06-10T18:56:20.519542+0000","last_fullsized":"2023-06-10T18:56:20.519542+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:03.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246823+0000","last_change":"2023-06-10T18:56:18.848880+0000","last_active":"2023-06-10T18:56:20.246823+0000","last_peered":"2023-06-10T18:56:20.246823+0000","last_clean":"2023-06-10T18:56:20.246823+0000","last_became_active":"2023-06-10T18:56:18.848539+0000","last_became_peered":"2023-06-10T18:56:18.848539+0000","last_unstale":"2023-06-10T18:56:20.246823+0000","last_undegraded":"2023-06-10T18:56:20.246823+0000","last_fullsized":"2023-06-10T18:56:20.246823+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:25.762698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519997+0000","last_change":"2023-06-10T18:56:19.263608+0000","last_active":"2023-06-10T18:56:20.519997+0000","last_peered":"2023-06-10T18:56:20.519997+0000","last_clean":"2023-06-10T18:56:20.519997+0000","last_became_active":"2023-06-10T18:56:19.263363+0000","last_became_peered":"2023-06-10T18:56:19.263363+0000","last_unstale":"2023-06-10T18:56:20.519997+0000","last_undegraded":"2023-06-10T18:56:20.519997+0000","last_fullsized":"2023-06-10T18:56:20.519997+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:44.904473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518836+0000","last_change":"2023-06-10T18:56:18.843662+0000","last_active":"2023-06-10T18:56:20.518836+0000","last_peered":"2023-06-10T18:56:20.518836+0000","last_clean":"2023-06-10T18:56:20.518836+0000","last_became_active":"2023-06-10T18:56:18.843387+0000","last_became_peered":"2023-06-10T18:56:18.843387+0000","last_unstale":"2023-06-10T18:56:20.518836+0000","last_undegraded":"2023-06-10T18:56:20.518836+0000","last_fullsized":"2023-06-10T18:56:20.518836+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:07:04.352439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519150+0000","last_change":"2023-06-10T18:56:19.258632+0000","last_active":"2023-06-10T18:56:20.519150+0000","last_peered":"2023-06-10T18:56:20.519150+0000","last_clean":"2023-06-10T18:56:20.519150+0000","last_became_active":"2023-06-10T18:56:19.258265+0000","last_became_peered":"2023-06-10T18:56:19.258265+0000","last_unstale":"2023-06-10T18:56:20.519150+0000","last_undegraded":"2023-06-10T18:56:20.519150+0000","last_fullsized":"2023-06-10T18:56:20.519150+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:44:50.097515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247044+0000","last_change":"2023-06-10T18:56:18.850661+0000","last_active":"2023-06-10T18:56:20.247044+0000","last_peered":"2023-06-10T18:56:20.247044+0000","last_clean":"2023-06-10T18:56:20.247044+0000","last_became_active":"2023-06-10T18:56:18.847270+0000","last_became_peered":"2023-06-10T18:56:18.847270+0000","last_unstale":"2023-06-10T18:56:20.247044+0000","last_undegraded":"2023-06-10T18:56:20.247044+0000","last_fullsized":"2023-06-10T18:56:20.247044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:50:19.654934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245839+0000","last_change":"2023-06-10T18:56:19.261256+0000","last_active":"2023-06-10T18:56:20.245839+0000","last_peered":"2023-06-10T18:56:20.245839+0000","last_clean":"2023-06-10T18:56:20.245839+0000","last_became_active":"2023-06-10T18:56:19.261024+0000","last_became_peered":"2023-06-10T18:56:19.261024+0000","last_unstale":"2023-06-10T18:56:20.245839+0000","last_undegraded":"2023-06-10T18:56:20.245839+0000","last_fullsized":"2023-06-10T18:56:20.245839+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:28:11.857799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246795+0000","last_change":"2023-06-10T18:56:18.847863+0000","last_active":"2023-06-10T18:56:20.246795+0000","last_peered":"2023-06-10T18:56:20.246795+0000","last_clean":"2023-06-10T18:56:20.246795+0000","last_became_active":"2023-06-10T18:56:18.846987+0000","last_became_peered":"2023-06-10T18:56:18.846987+0000","last_unstale":"2023-06-10T18:56:20.246795+0000","last_undegraded":"2023-06-10T18:56:20.246795+0000","last_fullsized":"2023-06-10T18:56:20.246795+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:02.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519180+0000","last_change":"2023-06-10T18:56:19.250165+0000","last_active":"2023-06-10T18:56:20.519180+0000","last_peered":"2023-06-10T18:56:20.519180+0000","last_clean":"2023-06-10T18:56:20.519180+0000","last_became_active":"2023-06-10T18:56:19.249871+0000","last_became_peered":"2023-06-10T18:56:19.249871+0000","last_unstale":"2023-06-10T18:56:20.519180+0000","last_undegraded":"2023-06-10T18:56:20.519180+0000","last_fullsized":"2023-06-10T18:56:20.519180+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:42:24.041414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245892+0000","last_change":"2023-06-10T18:56:18.846176+0000","last_active":"2023-06-10T18:56:20.245892+0000","last_peered":"2023-06-10T18:56:20.245892+0000","last_clean":"2023-06-10T18:56:20.245892+0000","last_became_active":"2023-06-10T18:56:18.845916+0000","last_became_peered":"2023-06-10T18:56:18.845916+0000","last_unstale":"2023-06-10T18:56:20.245892+0000","last_undegraded":"2023-06-10T18:56:20.245892+0000","last_fullsized":"2023-06-10T18:56:20.245892+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:28.537547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519480+0000","last_change":"2023-06-10T18:56:19.256374+0000","last_active":"2023-06-10T18:56:20.519480+0000","last_peered":"2023-06-10T18:56:20.519480+0000","last_clean":"2023-06-10T18:56:20.519480+0000","last_became_active":"2023-06-10T18:56:19.255002+0000","last_became_peered":"2023-06-10T18:56:19.255002+0000","last_unstale":"2023-06-10T18:56:20.519480+0000","last_undegraded":"2023-06-10T18:56:20.519480+0000","last_fullsized":"2023-06-10T18:56:20.519480+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:37:33.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520237+0000","last_change":"2023-06-10T18:56:18.847846+0000","last_active":"2023-06-10T18:56:20.520237+0000","last_peered":"2023-06-10T18:56:20.520237+0000","last_clean":"2023-06-10T18:56:20.520237+0000","last_became_active":"2023-06-10T18:56:18.847574+0000","last_became_peered":"2023-06-10T18:56:18.847574+0000","last_unstale":"2023-06-10T18:56:20.520237+0000","last_undegraded":"2023-06-10T18:56:20.520237+0000","last_fullsized":"2023-06-10T18:56:20.520237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:27:46.670228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246859+0000","last_change":"2023-06-10T18:56:19.245734+0000","last_active":"2023-06-10T18:56:20.246859+0000","last_peered":"2023-06-10T18:56:20.246859+0000","last_clean":"2023-06-10T18:56:20.246859+0000","last_became_active":"2023-06-10T18:56:19.245539+0000","last_became_peered":"2023-06-10T18:56:19.245539+0000","last_unstale":"2023-06-10T18:56:20.246859+0000","last_undegraded":"2023-06-10T18:56:20.246859+0000","last_fullsized":"2023-06-10T18:56:20.246859+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:26.350809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246772+0000","last_change":"2023-06-10T18:56:18.847196+0000","last_active":"2023-06-10T18:56:20.246772+0000","last_peered":"2023-06-10T18:56:20.246772+0000","last_clean":"2023-06-10T18:56:20.246772+0000","last_became_active":"2023-06-10T18:56:18.846894+0000","last_became_peered":"2023-06-10T18:56:18.846894+0000","last_unstale":"2023-06-10T18:56:20.246772+0000","last_undegraded":"2023-06-10T18:56:20.246772+0000","last_fullsized":"2023-06-10T18:56:20.246772+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:37:32.116803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519991+0000","last_change":"2023-06-10T18:56:19.258695+0000","last_active":"2023-06-10T18:56:20.519991+0000","last_peered":"2023-06-10T18:56:20.519991+0000","last_clean":"2023-06-10T18:56:20.519991+0000","last_became_active":"2023-06-10T18:56:19.258388+0000","last_became_peered":"2023-06-10T18:56:19.258388+0000","last_unstale":"2023-06-10T18:56:20.519991+0000","last_undegraded":"2023-06-10T18:56:20.519991+0000","last_fullsized":"2023-06-10T18:56:20.519991+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:50:20.661715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247001+0000","last_change":"2023-06-10T18:56:18.850289+0000","last_active":"2023-06-10T18:56:20.247001+0000","last_peered":"2023-06-10T18:56:20.247001+0000","last_clean":"2023-06-10T18:56:20.247001+0000","last_became_active":"2023-06-10T18:56:18.850034+0000","last_became_peered":"2023-06-10T18:56:18.850034+0000","last_unstale":"2023-06-10T18:56:20.247001+0000","last_undegraded":"2023-06-10T18:56:20.247001+0000","last_fullsized":"2023-06-10T18:56:20.247001+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:05.996772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246836+0000","last_change":"2023-06-10T18:56:19.251727+0000","last_active":"2023-06-10T18:56:20.246836+0000","last_peered":"2023-06-10T18:56:20.246836+0000","last_clean":"2023-06-10T18:56:20.246836+0000","last_became_active":"2023-06-10T18:56:19.251400+0000","last_became_peered":"2023-06-10T18:56:19.251400+0000","last_unstale":"2023-06-10T18:56:20.246836+0000","last_undegraded":"2023-06-10T18:56:20.246836+0000","last_fullsized":"2023-06-10T18:56:20.246836+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:03:56.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518418+0000","last_change":"2023-06-10T18:56:18.842107+0000","last_active":"2023-06-10T18:56:20.518418+0000","last_peered":"2023-06-10T18:56:20.518418+0000","last_clean":"2023-06-10T18:56:20.518418+0000","last_became_active":"2023-06-10T18:56:18.841780+0000","last_became_peered":"2023-06-10T18:56:18.841780+0000","last_unstale":"2023-06-10T18:56:20.518418+0000","last_undegraded":"2023-06-10T18:56:20.518418+0000","last_fullsized":"2023-06-10T18:56:20.518418+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:36:33.630870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245865+0000","last_change":"2023-06-10T18:56:19.251710+0000","last_active":"2023-06-10T18:56:20.245865+0000","last_peered":"2023-06-10T18:56:20.245865+0000","last_clean":"2023-06-10T18:56:20.245865+0000","last_became_active":"2023-06-10T18:56:19.251258+0000","last_became_peered":"2023-06-10T18:56:19.251258+0000","last_unstale":"2023-06-10T18:56:20.245865+0000","last_undegraded":"2023-06-10T18:56:20.245865+0000","last_fullsized":"2023-06-10T18:56:20.245865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:13:52.908092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519279+0000","last_change":"2023-06-10T18:56:18.851511+0000","last_active":"2023-06-10T18:56:20.519279+0000","last_peered":"2023-06-10T18:56:20.519279+0000","last_clean":"2023-06-10T18:56:20.519279+0000","last_became_active":"2023-06-10T18:56:18.851186+0000","last_became_peered":"2023-06-10T18:56:18.851186+0000","last_unstale":"2023-06-10T18:56:20.519279+0000","last_undegraded":"2023-06-10T18:56:20.519279+0000","last_fullsized":"2023-06-10T18:56:20.519279+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:16.501429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247139+0000","last_change":"2023-06-10T18:56:19.246608+0000","last_active":"2023-06-10T18:56:20.247139+0000","last_peered":"2023-06-10T18:56:20.247139+0000","last_clean":"2023-06-10T18:56:20.247139+0000","last_became_active":"2023-06-10T18:56:19.246369+0000","last_became_peered":"2023-06-10T18:56:19.246369+0000","last_unstale":"2023-06-10T18:56:20.247139+0000","last_undegraded":"2023-06-10T18:56:20.247139+0000","last_fullsized":"2023-06-10T18:56:20.247139+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.917468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246701+0000","last_change":"2023-06-10T18:56:18.848967+0000","last_active":"2023-06-10T18:56:20.246701+0000","last_peered":"2023-06-10T18:56:20.246701+0000","last_clean":"2023-06-10T18:56:20.246701+0000","last_became_active":"2023-06-10T18:56:18.848459+0000","last_became_peered":"2023-06-10T18:56:18.848459+0000","last_unstale":"2023-06-10T18:56:20.246701+0000","last_undegraded":"2023-06-10T18:56:20.246701+0000","last_fullsized":"2023-06-10T18:56:20.246701+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:19.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518520+0000","last_change":"2023-06-10T18:56:19.261143+0000","last_active":"2023-06-10T18:56:20.518520+0000","last_peered":"2023-06-10T18:56:20.518520+0000","last_clean":"2023-06-10T18:56:20.518520+0000","last_became_active":"2023-06-10T18:56:19.260888+0000","last_became_peered":"2023-06-10T18:56:19.260888+0000","last_unstale":"2023-06-10T18:56:20.518520+0000","last_undegraded":"2023-06-10T18:56:20.518520+0000","last_fullsized":"2023-06-10T18:56:20.518520+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:12:27.952753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246460+0000","last_change":"2023-06-10T18:56:18.852961+0000","last_active":"2023-06-10T18:56:20.246460+0000","last_peered":"2023-06-10T18:56:20.246460+0000","last_clean":"2023-06-10T18:56:20.246460+0000","last_became_active":"2023-06-10T18:56:18.852501+0000","last_became_peered":"2023-06-10T18:56:18.852501+0000","last_unstale":"2023-06-10T18:56:20.246460+0000","last_undegraded":"2023-06-10T18:56:20.246460+0000","last_fullsized":"2023-06-10T18:56:20.246460+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:16.011497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519213+0000","last_change":"2023-06-10T18:56:19.258451+0000","last_active":"2023-06-10T18:56:20.519213+0000","last_peered":"2023-06-10T18:56:20.519213+0000","last_clean":"2023-06-10T18:56:20.519213+0000","last_became_active":"2023-06-10T18:56:19.258165+0000","last_became_peered":"2023-06-10T18:56:19.258165+0000","last_unstale":"2023-06-10T18:56:20.519213+0000","last_undegraded":"2023-06-10T18:56:20.519213+0000","last_fullsized":"2023-06-10T18:56:20.519213+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:56:20.546763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245961+0000","last_change":"2023-06-10T18:56:18.843168+0000","last_active":"2023-06-10T18:56:20.245961+0000","last_peered":"2023-06-10T18:56:20.245961+0000","last_clean":"2023-06-10T18:56:20.245961+0000","last_became_active":"2023-06-10T18:56:18.842943+0000","last_became_peered":"2023-06-10T18:56:18.842943+0000","last_unstale":"2023-06-10T18:56:20.245961+0000","last_undegraded":"2023-06-10T18:56:20.245961+0000","last_fullsized":"2023-06-10T18:56:20.245961+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:53.564832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246689+0000","last_change":"2023-06-10T18:56:19.248252+0000","last_active":"2023-06-10T18:56:20.246689+0000","last_peered":"2023-06-10T18:56:20.246689+0000","last_clean":"2023-06-10T18:56:20.246689+0000","last_became_active":"2023-06-10T18:56:19.247996+0000","last_became_peered":"2023-06-10T18:56:19.247996+0000","last_unstale":"2023-06-10T18:56:20.246689+0000","last_undegraded":"2023-06-10T18:56:20.246689+0000","last_fullsized":"2023-06-10T18:56:20.246689+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:29.569098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245558+0000","last_change":"2023-06-10T18:56:18.850692+0000","last_active":"2023-06-10T18:56:20.245558+0000","last_peered":"2023-06-10T18:56:20.245558+0000","last_clean":"2023-06-10T18:56:20.245558+0000","last_became_active":"2023-06-10T18:56:18.850314+0000","last_became_peered":"2023-06-10T18:56:18.850314+0000","last_unstale":"2023-06-10T18:56:20.245558+0000","last_undegraded":"2023-06-10T18:56:20.245558+0000","last_fullsized":"2023-06-10T18:56:20.245558+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:07:41.036551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520262+0000","last_change":"2023-06-10T18:56:19.262645+0000","last_active":"2023-06-10T18:56:20.520262+0000","last_peered":"2023-06-10T18:56:20.520262+0000","last_clean":"2023-06-10T18:56:20.520262+0000","last_became_active":"2023-06-10T18:56:19.261931+0000","last_became_peered":"2023-06-10T18:56:19.261931+0000","last_unstale":"2023-06-10T18:56:20.520262+0000","last_undegraded":"2023-06-10T18:56:20.520262+0000","last_fullsized":"2023-06-10T18:56:20.520262+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:10.184151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518910+0000","last_change":"2023-06-10T18:56:18.843402+0000","last_active":"2023-06-10T18:56:20.518910+0000","last_peered":"2023-06-10T18:56:20.518910+0000","last_clean":"2023-06-10T18:56:20.518910+0000","last_became_active":"2023-06-10T18:56:18.843091+0000","last_became_peered":"2023-06-10T18:56:18.843091+0000","last_unstale":"2023-06-10T18:56:20.518910+0000","last_undegraded":"2023-06-10T18:56:20.518910+0000","last_fullsized":"2023-06-10T18:56:20.518910+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:45:32.097648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519729+0000","last_change":"2023-06-10T18:56:19.260813+0000","last_active":"2023-06-10T18:56:20.519729+0000","last_peered":"2023-06-10T18:56:20.519729+0000","last_clean":"2023-06-10T18:56:20.519729+0000","last_became_active":"2023-06-10T18:56:19.260109+0000","last_became_peered":"2023-06-10T18:56:19.260109+0000","last_unstale":"2023-06-10T18:56:20.519729+0000","last_undegraded":"2023-06-10T18:56:20.519729+0000","last_fullsized":"2023-06-10T18:56:20.519729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:55.423532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518873+0000","last_change":"2023-06-10T18:56:18.851835+0000","last_active":"2023-06-10T18:56:20.518873+0000","last_peered":"2023-06-10T18:56:20.518873+0000","last_clean":"2023-06-10T18:56:20.518873+0000","last_became_active":"2023-06-10T18:56:18.851500+0000","last_became_peered":"2023-06-10T18:56:18.851500+0000","last_unstale":"2023-06-10T18:56:20.518873+0000","last_undegraded":"2023-06-10T18:56:20.518873+0000","last_fullsized":"2023-06-10T18:56:20.518873+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:00.197990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518742+0000","last_change":"2023-06-10T18:56:19.258700+0000","last_active":"2023-06-10T18:56:20.518742+0000","last_peered":"2023-06-10T18:56:20.518742+0000","last_clean":"2023-06-10T18:56:20.518742+0000","last_became_active":"2023-06-10T18:56:19.258444+0000","last_became_peered":"2023-06-10T18:56:19.258444+0000","last_unstale":"2023-06-10T18:56:20.518742+0000","last_undegraded":"2023-06-10T18:56:20.518742+0000","last_fullsized":"2023-06-10T18:56:20.518742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:43:06.568005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519391+0000","last_change":"2023-06-10T18:56:18.847973+0000","last_active":"2023-06-10T18:56:20.519391+0000","last_peered":"2023-06-10T18:56:20.519391+0000","last_clean":"2023-06-10T18:56:20.519391+0000","last_became_active":"2023-06-10T18:56:18.846967+0000","last_became_peered":"2023-06-10T18:56:18.846967+0000","last_unstale":"2023-06-10T18:56:20.519391+0000","last_undegraded":"2023-06-10T18:56:20.519391+0000","last_fullsized":"2023-06-10T18:56:20.519391+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:03:54.133766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247174+0000","last_change":"2023-06-10T18:56:19.256774+0000","last_active":"2023-06-10T18:56:20.247174+0000","last_peered":"2023-06-10T18:56:20.247174+0000","last_clean":"2023-06-10T18:56:20.247174+0000","last_became_active":"2023-06-10T18:56:19.256514+0000","last_became_peered":"2023-06-10T18:56:19.256514+0000","last_unstale":"2023-06-10T18:56:20.247174+0000","last_undegraded":"2023-06-10T18:56:20.247174+0000","last_fullsized":"2023-06-10T18:56:20.247174+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:37.827141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247254+0000","last_change":"2023-06-10T18:56:18.844664+0000","last_active":"2023-06-10T18:56:20.247254+0000","last_peered":"2023-06-10T18:56:20.247254+0000","last_clean":"2023-06-10T18:56:20.247254+0000","last_became_active":"2023-06-10T18:56:18.844378+0000","last_became_peered":"2023-06-10T18:56:18.844378+0000","last_unstale":"2023-06-10T18:56:20.247254+0000","last_undegraded":"2023-06-10T18:56:20.247254+0000","last_fullsized":"2023-06-10T18:56:20.247254+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:14.213391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519137+0000","last_change":"2023-06-10T18:56:19.259157+0000","last_active":"2023-06-10T18:56:20.519137+0000","last_peered":"2023-06-10T18:56:20.519137+0000","last_clean":"2023-06-10T18:56:20.519137+0000","last_became_active":"2023-06-10T18:56:19.258900+0000","last_became_peered":"2023-06-10T18:56:19.258900+0000","last_unstale":"2023-06-10T18:56:20.519137+0000","last_undegraded":"2023-06-10T18:56:20.519137+0000","last_fullsized":"2023-06-10T18:56:20.519137+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:31:06.947960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518861+0000","last_change":"2023-06-10T18:56:18.846358+0000","last_active":"2023-06-10T18:56:20.518861+0000","last_peered":"2023-06-10T18:56:20.518861+0000","last_clean":"2023-06-10T18:56:20.518861+0000","last_became_active":"2023-06-10T18:56:18.846074+0000","last_became_peered":"2023-06-10T18:56:18.846074+0000","last_unstale":"2023-06-10T18:56:20.518861+0000","last_undegraded":"2023-06-10T18:56:20.518861+0000","last_fullsized":"2023-06-10T18:56:20.518861+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:14.213391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519552+0000","last_change":"2023-06-10T18:56:19.262755+0000","last_active":"2023-06-10T18:56:20.519552+0000","last_peered":"2023-06-10T18:56:20.519552+0000","last_clean":"2023-06-10T18:56:20.519552+0000","last_became_active":"2023-06-10T18:56:19.262476+0000","last_became_peered":"2023-06-10T18:56:19.262476+0000","last_unstale":"2023-06-10T18:56:20.519552+0000","last_undegraded":"2023-06-10T18:56:20.519552+0000","last_fullsized":"2023-06-10T18:56:20.519552+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:14:21.738968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246862+0000","last_change":"2023-06-10T18:56:18.845783+0000","last_active":"2023-06-10T18:56:20.246862+0000","last_peered":"2023-06-10T18:56:20.246862+0000","last_clean":"2023-06-10T18:56:20.246862+0000","last_became_active":"2023-06-10T18:56:18.845251+0000","last_became_peered":"2023-06-10T18:56:18.845251+0000","last_unstale":"2023-06-10T18:56:20.246862+0000","last_undegraded":"2023-06-10T18:56:20.246862+0000","last_fullsized":"2023-06-10T18:56:20.246862+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:06.641446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246449+0000","last_change":"2023-06-10T18:56:19.256710+0000","last_active":"2023-06-10T18:56:20.246449+0000","last_peered":"2023-06-10T18:56:20.246449+0000","last_clean":"2023-06-10T18:56:20.246449+0000","last_became_active":"2023-06-10T18:56:19.256485+0000","last_became_peered":"2023-06-10T18:56:19.256485+0000","last_unstale":"2023-06-10T18:56:20.246449+0000","last_undegraded":"2023-06-10T18:56:20.246449+0000","last_fullsized":"2023-06-10T18:56:20.246449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:31.814128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520127+0000","last_change":"2023-06-10T18:56:18.847777+0000","last_active":"2023-06-10T18:56:20.520127+0000","last_peered":"2023-06-10T18:56:20.520127+0000","last_clean":"2023-06-10T18:56:20.520127+0000","last_became_active":"2023-06-10T18:56:18.847494+0000","last_became_peered":"2023-06-10T18:56:18.847494+0000","last_unstale":"2023-06-10T18:56:20.520127+0000","last_undegraded":"2023-06-10T18:56:20.520127+0000","last_fullsized":"2023-06-10T18:56:20.520127+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:05:30.131521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245804+0000","last_change":"2023-06-10T18:56:19.253071+0000","last_active":"2023-06-10T18:56:20.245804+0000","last_peered":"2023-06-10T18:56:20.245804+0000","last_clean":"2023-06-10T18:56:20.245804+0000","last_became_active":"2023-06-10T18:56:19.252820+0000","last_became_peered":"2023-06-10T18:56:19.252820+0000","last_unstale":"2023-06-10T18:56:20.245804+0000","last_undegraded":"2023-06-10T18:56:20.245804+0000","last_fullsized":"2023-06-10T18:56:20.245804+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:06.433739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245738+0000","last_change":"2023-06-10T18:56:18.851293+0000","last_active":"2023-06-10T18:56:20.245738+0000","last_peered":"2023-06-10T18:56:20.245738+0000","last_clean":"2023-06-10T18:56:20.245738+0000","last_became_active":"2023-06-10T18:56:18.850991+0000","last_became_peered":"2023-06-10T18:56:18.850991+0000","last_unstale":"2023-06-10T18:56:20.245738+0000","last_undegraded":"2023-06-10T18:56:20.245738+0000","last_fullsized":"2023-06-10T18:56:20.245738+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:01.094221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246701+0000","last_change":"2023-06-10T18:56:19.256825+0000","last_active":"2023-06-10T18:56:20.246701+0000","last_peered":"2023-06-10T18:56:20.246701+0000","last_clean":"2023-06-10T18:56:20.246701+0000","last_became_active":"2023-06-10T18:56:19.256533+0000","last_became_peered":"2023-06-10T18:56:19.256533+0000","last_unstale":"2023-06-10T18:56:20.246701+0000","last_undegraded":"2023-06-10T18:56:20.246701+0000","last_fullsized":"2023-06-10T18:56:20.246701+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:34.855752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518289+0000","last_change":"2023-06-10T18:56:18.845834+0000","last_active":"2023-06-10T18:56:20.518289+0000","last_peered":"2023-06-10T18:56:20.518289+0000","last_clean":"2023-06-10T18:56:20.518289+0000","last_became_active":"2023-06-10T18:56:18.845492+0000","last_became_peered":"2023-06-10T18:56:18.845492+0000","last_unstale":"2023-06-10T18:56:20.518289+0000","last_undegraded":"2023-06-10T18:56:20.518289+0000","last_fullsized":"2023-06-10T18:56:20.518289+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:36.820360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519014+0000","last_change":"2023-06-10T18:56:19.262372+0000","last_active":"2023-06-10T18:56:20.519014+0000","last_peered":"2023-06-10T18:56:20.519014+0000","last_clean":"2023-06-10T18:56:20.519014+0000","last_became_active":"2023-06-10T18:56:19.262061+0000","last_became_peered":"2023-06-10T18:56:19.262061+0000","last_unstale":"2023-06-10T18:56:20.519014+0000","last_undegraded":"2023-06-10T18:56:20.519014+0000","last_fullsized":"2023-06-10T18:56:20.519014+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:11:35.768462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":19,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519581+0000","last_change":"2023-06-10T18:56:18.845270+0000","last_active":"2023-06-10T18:56:20.519581+0000","last_peered":"2023-06-10T18:56:20.519581+0000","last_clean":"2023-06-10T18:56:20.519581+0000","last_became_active":"2023-06-10T18:56:18.845010+0000","last_became_peered":"2023-06-10T18:56:18.845010+0000","last_unstale":"2023-06-10T18:56:20.519581+0000","last_undegraded":"2023-06-10T18:56:20.519581+0000","last_fullsized":"2023-06-10T18:56:20.519581+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:30:50.870671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":202,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247242+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:56:20.247242+0000","last_peered":"2023-06-10T18:56:20.247242+0000","last_clean":"2023-06-10T18:56:20.247242+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:56:20.247242+0000","last_undegraded":"2023-06-10T18:56:20.247242+0000","last_fullsized":"2023-06-10T18:56:20.247242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:24:18.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518654+0000","last_change":"2023-06-10T18:56:19.255151+0000","last_active":"2023-06-10T18:56:20.518654+0000","last_peered":"2023-06-10T18:56:20.518654+0000","last_clean":"2023-06-10T18:56:20.518654+0000","last_became_active":"2023-06-10T18:56:19.254885+0000","last_became_peered":"2023-06-10T18:56:19.254885+0000","last_unstale":"2023-06-10T18:56:20.518654+0000","last_undegraded":"2023-06-10T18:56:20.518654+0000","last_fullsized":"2023-06-10T18:56:20.518654+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:16:14.135900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518792+0000","last_change":"2023-06-10T18:56:18.853944+0000","last_active":"2023-06-10T18:56:20.518792+0000","last_peered":"2023-06-10T18:56:20.518792+0000","last_clean":"2023-06-10T18:56:20.518792+0000","last_became_active":"2023-06-10T18:56:18.853689+0000","last_became_peered":"2023-06-10T18:56:18.853689+0000","last_unstale":"2023-06-10T18:56:20.518792+0000","last_undegraded":"2023-06-10T18:56:20.518792+0000","last_fullsized":"2023-06-10T18:56:20.518792+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:50.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":228,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246281+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:56:20.246281+0000","last_peered":"2023-06-10T18:56:20.246281+0000","last_clean":"2023-06-10T18:56:20.246281+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:56:20.246281+0000","last_undegraded":"2023-06-10T18:56:20.246281+0000","last_fullsized":"2023-06-10T18:56:20.246281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:16.797895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246654+0000","last_change":"2023-06-10T18:56:19.256478+0000","last_active":"2023-06-10T18:56:20.246654+0000","last_peered":"2023-06-10T18:56:20.246654+0000","last_clean":"2023-06-10T18:56:20.246654+0000","last_became_active":"2023-06-10T18:56:19.256199+0000","last_became_peered":"2023-06-10T18:56:19.256199+0000","last_unstale":"2023-06-10T18:56:20.246654+0000","last_undegraded":"2023-06-10T18:56:20.246654+0000","last_fullsized":"2023-06-10T18:56:20.246654+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:44.558873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246819+0000","last_change":"2023-06-10T18:56:18.845106+0000","last_active":"2023-06-10T18:56:20.246819+0000","last_peered":"2023-06-10T18:56:20.246819+0000","last_clean":"2023-06-10T18:56:20.246819+0000","last_became_active":"2023-06-10T18:56:18.844839+0000","last_became_peered":"2023-06-10T18:56:18.844839+0000","last_unstale":"2023-06-10T18:56:20.246819+0000","last_undegraded":"2023-06-10T18:56:20.246819+0000","last_fullsized":"2023-06-10T18:56:20.246819+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:30:50.870671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":200,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518706+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:56:20.518706+0000","last_peered":"2023-06-10T18:56:20.518706+0000","last_clean":"2023-06-10T18:56:20.518706+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:56:20.518706+0000","last_undegraded":"2023-06-10T18:56:20.518706+0000","last_fullsized":"2023-06-10T18:56:20.518706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:09.166839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519495+0000","last_change":"2023-06-10T18:56:19.257690+0000","last_active":"2023-06-10T18:56:20.519495+0000","last_peered":"2023-06-10T18:56:20.519495+0000","last_clean":"2023-06-10T18:56:20.519495+0000","last_became_active":"2023-06-10T18:56:19.257406+0000","last_became_peered":"2023-06-10T18:56:19.257406+0000","last_unstale":"2023-06-10T18:56:20.519495+0000","last_undegraded":"2023-06-10T18:56:20.519495+0000","last_fullsized":"2023-06-10T18:56:20.519495+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:13.798958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520026+0000","last_change":"2023-06-10T18:56:18.852820+0000","last_active":"2023-06-10T18:56:20.520026+0000","last_peered":"2023-06-10T18:56:20.520026+0000","last_clean":"2023-06-10T18:56:20.520026+0000","last_became_active":"2023-06-10T18:56:18.852540+0000","last_became_peered":"2023-06-10T18:56:18.852540+0000","last_unstale":"2023-06-10T18:56:20.520026+0000","last_undegraded":"2023-06-10T18:56:20.520026+0000","last_fullsized":"2023-06-10T18:56:20.520026+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:30.825236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":228,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246554+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:56:20.246554+0000","last_peered":"2023-06-10T18:56:20.246554+0000","last_clean":"2023-06-10T18:56:20.246554+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:56:20.246554+0000","last_undegraded":"2023-06-10T18:56:20.246554+0000","last_fullsized":"2023-06-10T18:56:20.246554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:50:41.734840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518775+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:56:20.518775+0000","last_peered":"2023-06-10T18:56:20.518775+0000","last_clean":"2023-06-10T18:56:20.518775+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:56:20.518775+0000","last_undegraded":"2023-06-10T18:56:20.518775+0000","last_fullsized":"2023-06-10T18:56:20.518775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:00:00.794334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520031+0000","last_change":"2023-06-10T18:56:19.255667+0000","last_active":"2023-06-10T18:56:20.520031+0000","last_peered":"2023-06-10T18:56:20.520031+0000","last_clean":"2023-06-10T18:56:20.520031+0000","last_became_active":"2023-06-10T18:56:19.254988+0000","last_became_peered":"2023-06-10T18:56:19.254988+0000","last_unstale":"2023-06-10T18:56:20.520031+0000","last_undegraded":"2023-06-10T18:56:20.520031+0000","last_fullsized":"2023-06-10T18:56:20.520031+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:55:49.705761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":21,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519928+0000","last_change":"2023-06-10T18:56:18.846798+0000","last_active":"2023-06-10T18:56:20.519928+0000","last_peered":"2023-06-10T18:56:20.519928+0000","last_clean":"2023-06-10T18:56:20.519928+0000","last_became_active":"2023-06-10T18:56:18.846561+0000","last_became_peered":"2023-06-10T18:56:18.846561+0000","last_unstale":"2023-06-10T18:56:20.519928+0000","last_undegraded":"2023-06-10T18:56:20.519928+0000","last_fullsized":"2023-06-10T18:56:20.519928+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:02.027782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":215,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518572+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:56:20.518572+0000","last_peered":"2023-06-10T18:56:20.518572+0000","last_clean":"2023-06-10T18:56:20.518572+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:56:20.518572+0000","last_undegraded":"2023-06-10T18:56:20.518572+0000","last_fullsized":"2023-06-10T18:56:20.518572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:06.990202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520137+0000","last_change":"2023-06-10T18:56:19.257337+0000","last_active":"2023-06-10T18:56:20.520137+0000","last_peered":"2023-06-10T18:56:20.520137+0000","last_clean":"2023-06-10T18:56:20.520137+0000","last_became_active":"2023-06-10T18:56:19.256968+0000","last_became_peered":"2023-06-10T18:56:19.256968+0000","last_unstale":"2023-06-10T18:56:20.520137+0000","last_undegraded":"2023-06-10T18:56:20.520137+0000","last_fullsized":"2023-06-10T18:56:20.520137+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:25:21.372024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246534+0000","last_change":"2023-06-10T18:56:18.846399+0000","last_active":"2023-06-10T18:56:20.246534+0000","last_peered":"2023-06-10T18:56:20.246534+0000","last_clean":"2023-06-10T18:56:20.246534+0000","last_became_active":"2023-06-10T18:56:18.846132+0000","last_became_peered":"2023-06-10T18:56:18.846132+0000","last_unstale":"2023-06-10T18:56:20.246534+0000","last_undegraded":"2023-06-10T18:56:20.246534+0000","last_fullsized":"2023-06-10T18:56:20.246534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:42:43.655486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519242+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:56:20.519242+0000","last_peered":"2023-06-10T18:56:20.519242+0000","last_clean":"2023-06-10T18:56:20.519242+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:56:20.519242+0000","last_undegraded":"2023-06-10T18:56:20.519242+0000","last_fullsized":"2023-06-10T18:56:20.519242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:14.316978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246757+0000","last_change":"2023-06-10T18:56:19.253481+0000","last_active":"2023-06-10T18:56:20.246757+0000","last_peered":"2023-06-10T18:56:20.246757+0000","last_clean":"2023-06-10T18:56:20.246757+0000","last_became_active":"2023-06-10T18:56:19.253226+0000","last_became_peered":"2023-06-10T18:56:19.253226+0000","last_unstale":"2023-06-10T18:56:20.246757+0000","last_undegraded":"2023-06-10T18:56:20.246757+0000","last_fullsized":"2023-06-10T18:56:20.246757+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:39.665478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245912+0000","last_change":"2023-06-10T18:56:18.846585+0000","last_active":"2023-06-10T18:56:20.245912+0000","last_peered":"2023-06-10T18:56:20.245912+0000","last_clean":"2023-06-10T18:56:20.245912+0000","last_became_active":"2023-06-10T18:56:18.846315+0000","last_became_peered":"2023-06-10T18:56:18.846315+0000","last_unstale":"2023-06-10T18:56:20.245912+0000","last_undegraded":"2023-06-10T18:56:20.245912+0000","last_fullsized":"2023-06-10T18:56:20.245912+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:33:42.718180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":202,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247082+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:56:20.247082+0000","last_peered":"2023-06-10T18:56:20.247082+0000","last_clean":"2023-06-10T18:56:20.247082+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:56:20.247082+0000","last_undegraded":"2023-06-10T18:56:20.247082+0000","last_fullsized":"2023-06-10T18:56:20.247082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:19:53.698784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518951+0000","last_change":"2023-06-10T18:56:19.254848+0000","last_active":"2023-06-10T18:56:20.518951+0000","last_peered":"2023-06-10T18:56:20.518951+0000","last_clean":"2023-06-10T18:56:20.518951+0000","last_became_active":"2023-06-10T18:56:19.254487+0000","last_became_peered":"2023-06-10T18:56:19.254487+0000","last_unstale":"2023-06-10T18:56:20.518951+0000","last_undegraded":"2023-06-10T18:56:20.518951+0000","last_fullsized":"2023-06-10T18:56:20.518951+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:29.585917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246895+0000","last_change":"2023-06-10T18:56:18.852946+0000","last_active":"2023-06-10T18:56:20.246895+0000","last_peered":"2023-06-10T18:56:20.246895+0000","last_clean":"2023-06-10T18:56:20.246895+0000","last_became_active":"2023-06-10T18:56:18.852523+0000","last_became_peered":"2023-06-10T18:56:18.852523+0000","last_unstale":"2023-06-10T18:56:20.246895+0000","last_undegraded":"2023-06-10T18:56:20.246895+0000","last_fullsized":"2023-06-10T18:56:20.246895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:40:04.391011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":198,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519635+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:56:20.519635+0000","last_peered":"2023-06-10T18:56:20.519635+0000","last_clean":"2023-06-10T18:56:20.519635+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:56:20.519635+0000","last_undegraded":"2023-06-10T18:56:20.519635+0000","last_fullsized":"2023-06-10T18:56:20.519635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:30:10.044276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575001,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29112,"kb_used_data":2160,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342728,"statfs":{"total":96636764160,"available":96606953472,"internally_reserved":0,"allocated":2211840,"data_stored":1368719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.442,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.74299999999999999,"15min":0.79000000000000004},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.72099999999999997,"5min":0.78100000000000003,"15min":0.91700000000000004},"last":0.505}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.89700000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70999999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73699999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.76300000000000001,"15min":0.81100000000000005},"last":0.60299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575002,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.51600000000000001},"min":{"1min":0.36099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70799999999999996,"5min":0.77100000000000002,"15min":0.81100000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.78700000000000003},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.51500000000000001},"min":{"1min":0.42599999999999999,"5min":0.316,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.65600000000000003},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.61499999999999999},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66900000000000004,"5min":0.872,"15min":6.6689999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.252},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.81100000000000005,"15min":0.82899999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53300000000000003,"15min":0.54600000000000004},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67900000000000005,"5min":0.77100000000000002,"15min":0.77500000000000002},"last":0.53700000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.498,"15min":0.498},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":0.874,"15min":0.874},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.27600000000000002},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.89600000000000002},"last":0.36599999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.68300000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.46600000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575001,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28504,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343336,"statfs":{"total":96636764160,"available":96607576064,"internally_reserved":0,"allocated":1589248,"data_stored":775130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.33000000000000002},"min":{"1min":0.20499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.498}]},{"osd":2,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54000000000000004,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.40000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59699999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61099999999999999,"15min":0.627},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.81999999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.93000000000000005},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60299999999999998,"15min":0.626},"min":{"1min":0.51500000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.78900000000000003,"5min":0.83899999999999997,"15min":1.008},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.206,"15min":1.206},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":0.873,"15min":0.88700000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.442,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":1.224,"15min":1.224},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59299999999999997,"15min":0.61599999999999999},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.56499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575001,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33180,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338660,"statfs":{"total":96636764160,"available":96602787840,"internally_reserved":0,"allocated":2183168,"data_stored":1365498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.46200000000000002,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34200000000000003,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.437,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48299999999999998,"5min":11.439,"15min":11.439},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.48099999999999998,"15min":0.42199999999999999},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.49099999999999999,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":5.766,"15min":5.766},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40500000000000003,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51400000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.75900000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":2.802,"15min":2.802},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81599999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80500000000000005,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79300000000000004,"5min":2.907,"15min":2.907},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.41299999999999998,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":1.363},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.79200000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.61299999999999999},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.38900000000000001},"max":{"1min":0.82999999999999996,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834575002,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34899999999999998,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41999999999999998,"15min":0.38500000000000001},"min":{"1min":0.27300000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57499999999999996,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52700000000000002,"5min":2.847,"15min":2.847},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.40200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59499999999999997,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.41099999999999998},"min":{"1min":0.32000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56899999999999995,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.60699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":5.742,"15min":5.742},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.64300000000000002,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":6.593,"15min":6.593},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.77600000000000002,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.66400000000000003},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.82299999999999995,"5min":6.617,"15min":6.617},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.71899999999999997,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.73099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33000000000000002,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.45000000000000001,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.25},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54400000000000004,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45700000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57199999999999995,"5min":0.68600000000000005,"15min":1.3},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54000000000000004,"5min":0.82999999999999996,"15min":1.109},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.439},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.61299999999999999,"5min":1.5960000000000001,"15min":1.599},"last":0.308}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35899999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89700000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":1.613,"15min":1.613},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.60399999999999998,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":1.4410000000000001,"15min":3.802},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.65300000000000002},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.86499999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.84699999999999998,"5min":1.679,"15min":1.679},"last":0.70699999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.45200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":1.47,"15min":1.47},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":0.91100000000000003,"5min":1.548,"15min":1.8939999999999999},"last":0.67500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575001,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1593344,"data_stored":775200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.45000000000000001,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.745,"15min":1.119},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.503}]},{"osd":1,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.63,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.873,"15min":0.96599999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.64800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.746,"15min":0.79800000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.94599999999999995,"15min":1.069},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":0.63800000000000001,"5min":0.755,"15min":0.88300000000000001},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.54300000000000004,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28199999999999997},"max":{"1min":0.60199999999999998,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.70399999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.377}]},{"osd":7,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.68400000000000005,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.36699999999999999,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.48799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575002,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66200000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55100000000000005,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32300000000000001},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.443}]},{"osd":1,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.69999999999999996,"5min":0.75800000000000001,"15min":0.78500000000000003},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.56200000000000006},"min":{"1min":0.36199999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69899999999999995,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53300000000000003,"15min":0.504},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.73099999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.879},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.754,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72999999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:56:23.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:56:23.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.488+0000 7f20e6a6e700 1 Processor -- start 2023-06-10T18:56:23.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.488+0000 7f20e6a6e700 1 -- start start 2023-06-10T18:56:23.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e6a6e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f20e006e4b0 0x7f20e006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e6a6e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e006d290 con 0x7f20e006e0e0 2023-06-10T18:56:23.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e6a6e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00ca2d0 con 0x7f20e0069e70 2023-06-10T18:56:23.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e6a6e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e0071640 con 0x7f20e006e4b0 2023-06-10T18:56:23.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e480a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f20e006e0e0 0x7f20e00c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35148/0 (socket says 172.21.15.104:35148) 2023-06-10T18:56:23.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20e480a700 1 -- 172.21.15.104:0/613236056 learned_addr learned my addr 172.21.15.104:0/613236056 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:23.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20dffff700 1 --2- 172.21.15.104:0/613236056 >> v2:172.21.15.114:3300/0 conn(0x7f20e006e4b0 0x7f20e006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.489+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479316701 0 0) 0x7f20e006d290 con 0x7f20e006e0e0 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d00035b0 con 0x7f20e006e0e0 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926939463 0 0) 0x7f20d00035b0 con 0x7f20e006e0e0 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 >> v2:172.21.15.114:3300/0 conn(0x7f20e006e4b0 msgr2=0x7f20e006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 --2- 172.21.15.104:0/613236056 >> v2:172.21.15.114:3300/0 conn(0x7f20e006e4b0 0x7f20e006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 >> v1:172.21.15.114:6789/0 conn(0x7f20e0069e70 legacy=0x7f20e00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e006d4c0 con 0x7f20e006e0e0 2023-06-10T18:56:23.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f20d40017f0 con 0x7f20e006e0e0 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 >> v1:172.21.15.104:6789/0 conn(0x7f20e006e0e0 legacy=0x7f20e00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20d0003a00 0x7f20d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20d0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d0005320 0x7f20d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0071640 con 0x7f20d0004850 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d000c250 con 0x7f20d0005320 2023-06-10T18:56:23.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d000c410 con 0x7f20d0003a00 2023-06-10T18:56:23.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20d0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20e480a700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20d0003a00 0x7f20d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20e500b700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d0005320 0x7f20d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 -- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20d0003a00 msgr2=0x7f20d0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20d0003a00 0x7f20d0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 -- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d0005320 msgr2=0x7f20d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d0005320 0x7f20d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 -- 172.21.15.104:0/613236056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00035b0 con 0x7f20d0004850 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20e480a700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20d0003a00 0x7f20d0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20e500b700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20d0005320 0x7f20d000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20e500b700 1 -- 172.21.15.104:0/613236056 reap_dead start 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20dffff700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20d0004cc0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009b60 tx=0x7f20d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8066e5a0c541d125 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00e6c0 con 0x7f20d0004850 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20cc00e820 con 0x7f20d0004850 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.490+0000 7f20df7fe700 1 -- 172.21.15.104:0/613236056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00eb90 con 0x7f20d0004850 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 -- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 msgr2=0x7f20d0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20d0004cc0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009b60 tx=0x7f20d0005880 comp rx=0 tx=0).stop 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 -- 172.21.15.104:0/613236056 shutdown_connections 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/613236056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20d0004cc0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 -- 172.21.15.104:0/613236056 >> 172.21.15.104:0/613236056 conn(0x7f20e006b8a0 msgr2=0x7f20e00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:23.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 -- 172.21.15.104:0/613236056 shutdown_connections 2023-06-10T18:56:23.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.491+0000 7f20e6a6e700 1 -- 172.21.15.104:0/613236056 wait complete. 2023-06-10T18:56:23.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.492+0000 7f20e6a6e700 1 Processor -- start 2023-06-10T18:56:23.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.492+0000 7f20e6a6e700 1 -- start start 2023-06-10T18:56:23.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20e00aeb40 0x7f20e00b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20e014b270 0x7f20e00c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00af390 con 0x7f20d0004850 2023-06-10T18:56:23.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c4bb0 con 0x7f20e014b270 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e6a6e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00c4e90 con 0x7f20e00aeb40 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37332/0 (socket says 172.21.15.104:37332) 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 -- 172.21.15.104:0/2452124622 learned_addr learned my addr 172.21.15.104:0/2452124622 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20dffff700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20e00aeb40 0x7f20e00b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 -- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20e00aeb40 msgr2=0x7f20e00b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e500b700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20e014b270 0x7f20e00c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20e00aeb40 0x7f20e00b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 -- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20e014b270 msgr2=0x7f20e00c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.493+0000 7f20e480a700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20e014b270 0x7f20e00c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20e480a700 1 -- 172.21.15.104:0/2452124622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e00c5170 con 0x7f20d0004850 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20e480a700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e7b0 tx=0x7f20d400eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d40072e0 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4007440 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4005830 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e00c5450 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.494+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e00b4db0 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.495+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e00c5690 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.495+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f20d4005990 con 0x7f20d0004850 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.495+0000 7f20dd7fa700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 0x7f20c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.496+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f20d4089530 con 0x7f20d0004850 2023-06-10T18:56:23.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.496+0000 7f20dffff700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 0x7f20c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.496+0000 7f20dffff700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 0x7f20c406d190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f20e01f5540 tx=0x7f20cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.500+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f20e00c5690 con 0x7f20d0004850 2023-06-10T18:56:23.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.681+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20e006da80 con 0x7f20c406ace0 2023-06-10T18:56:23.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.691+0000 7f20dd7fa700 1 -- 172.21.15.104:0/2452124622 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312046 (secure 0 0 0) 0x7f20cc00ea90 con 0x7f20c406ace0 2023-06-10T18:56:23.698 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:23.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 msgr2=0x7f20c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 0x7f20c406d190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f20e01f5540 tx=0x7f20cc008030 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 msgr2=0x7f20e00b1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e7b0 tx=0x7f20d400eb70 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 shutdown_connections 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f20c406ace0 0x7f20c406d190 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20e014b270 0x7f20e00c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20d0004850 0x7f20e00b1a90 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 --2- 172.21.15.104:0/2452124622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20e00aeb40 0x7f20e00b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 >> 172.21.15.104:0/2452124622 conn(0x7f20e006b8a0 msgr2=0x7f20e00c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:23.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 shutdown_connections 2023-06-10T18:56:23.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.694+0000 7f20e6a6e700 1 -- 172.21.15.104:0/2452124622 wait complete. 2023-06-10T18:56:23.703 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:56:23.711 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2023-06-10T18:56:23.546111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233368,"kb_used_data":13656,"kb_used_omap":0,"kb_used_meta":219648,"kb_avail":754741352,"statfs":{"total":773094113280,"available":772855144448,"internally_reserved":0,"allocated":13983744,"data_stored":7390576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":224919552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.716150"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518572+0000","last_change":"2023-06-10T18:56:19.261823+0000","last_active":"2023-06-10T18:56:20.518572+0000","last_peered":"2023-06-10T18:56:20.518572+0000","last_clean":"2023-06-10T18:56:20.518572+0000","last_became_active":"2023-06-10T18:56:19.261529+0000","last_became_peered":"2023-06-10T18:56:19.261529+0000","last_unstale":"2023-06-10T18:56:20.518572+0000","last_undegraded":"2023-06-10T18:56:20.518572+0000","last_fullsized":"2023-06-10T18:56:20.518572+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:26:19.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246958+0000","last_change":"2023-06-10T18:56:19.260237+0000","last_active":"2023-06-10T18:56:20.246958+0000","last_peered":"2023-06-10T18:56:20.246958+0000","last_clean":"2023-06-10T18:56:20.246958+0000","last_became_active":"2023-06-10T18:56:19.259890+0000","last_became_peered":"2023-06-10T18:56:19.259890+0000","last_unstale":"2023-06-10T18:56:20.246958+0000","last_undegraded":"2023-06-10T18:56:20.246958+0000","last_fullsized":"2023-06-10T18:56:20.246958+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:03.034563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519368+0000","last_change":"2023-06-10T18:56:19.257478+0000","last_active":"2023-06-10T18:56:20.519368+0000","last_peered":"2023-06-10T18:56:20.519368+0000","last_clean":"2023-06-10T18:56:20.519368+0000","last_became_active":"2023-06-10T18:56:19.257212+0000","last_became_peered":"2023-06-10T18:56:19.257212+0000","last_unstale":"2023-06-10T18:56:20.519368+0000","last_undegraded":"2023-06-10T18:56:20.519368+0000","last_fullsized":"2023-06-10T18:56:20.519368+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:29.569098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518794+0000","last_change":"2023-06-10T18:56:19.256061+0000","last_active":"2023-06-10T18:56:20.518794+0000","last_peered":"2023-06-10T18:56:20.518794+0000","last_clean":"2023-06-10T18:56:20.518794+0000","last_became_active":"2023-06-10T18:56:19.255727+0000","last_became_peered":"2023-06-10T18:56:19.255727+0000","last_unstale":"2023-06-10T18:56:20.518794+0000","last_undegraded":"2023-06-10T18:56:20.518794+0000","last_fullsized":"2023-06-10T18:56:20.518794+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:35:05.467403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519756+0000","last_change":"2023-06-10T18:56:19.255086+0000","last_active":"2023-06-10T18:56:20.519756+0000","last_peered":"2023-06-10T18:56:20.519756+0000","last_clean":"2023-06-10T18:56:20.519756+0000","last_became_active":"2023-06-10T18:56:19.254530+0000","last_became_peered":"2023-06-10T18:56:19.254530+0000","last_unstale":"2023-06-10T18:56:20.519756+0000","last_undegraded":"2023-06-10T18:56:20.519756+0000","last_fullsized":"2023-06-10T18:56:20.519756+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:34:57.004440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518985+0000","last_change":"2023-06-10T18:56:19.256890+0000","last_active":"2023-06-10T18:56:20.518985+0000","last_peered":"2023-06-10T18:56:20.518985+0000","last_clean":"2023-06-10T18:56:20.518985+0000","last_became_active":"2023-06-10T18:56:19.256566+0000","last_became_peered":"2023-06-10T18:56:19.256566+0000","last_unstale":"2023-06-10T18:56:20.518985+0000","last_undegraded":"2023-06-10T18:56:20.518985+0000","last_fullsized":"2023-06-10T18:56:20.518985+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:42:31.082333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518668+0000","last_change":"2023-06-10T18:56:19.256763+0000","last_active":"2023-06-10T18:56:20.518668+0000","last_peered":"2023-06-10T18:56:20.518668+0000","last_clean":"2023-06-10T18:56:20.518668+0000","last_became_active":"2023-06-10T18:56:19.256465+0000","last_became_peered":"2023-06-10T18:56:19.256465+0000","last_unstale":"2023-06-10T18:56:20.518668+0000","last_undegraded":"2023-06-10T18:56:20.518668+0000","last_fullsized":"2023-06-10T18:56:20.518668+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:25.797334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519698+0000","last_change":"2023-06-10T18:56:19.255561+0000","last_active":"2023-06-10T18:56:20.519698+0000","last_peered":"2023-06-10T18:56:20.519698+0000","last_clean":"2023-06-10T18:56:20.519698+0000","last_became_active":"2023-06-10T18:56:19.254725+0000","last_became_peered":"2023-06-10T18:56:19.254725+0000","last_unstale":"2023-06-10T18:56:20.519698+0000","last_undegraded":"2023-06-10T18:56:20.519698+0000","last_fullsized":"2023-06-10T18:56:20.519698+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:07.648227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519385+0000","last_change":"2023-06-10T18:56:19.255312+0000","last_active":"2023-06-10T18:56:20.519385+0000","last_peered":"2023-06-10T18:56:20.519385+0000","last_clean":"2023-06-10T18:56:20.519385+0000","last_became_active":"2023-06-10T18:56:19.254333+0000","last_became_peered":"2023-06-10T18:56:19.254333+0000","last_unstale":"2023-06-10T18:56:20.519385+0000","last_undegraded":"2023-06-10T18:56:20.519385+0000","last_fullsized":"2023-06-10T18:56:20.519385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:34.855752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247134+0000","last_change":"2023-06-10T18:56:19.260100+0000","last_active":"2023-06-10T18:56:20.247134+0000","last_peered":"2023-06-10T18:56:20.247134+0000","last_clean":"2023-06-10T18:56:20.247134+0000","last_became_active":"2023-06-10T18:56:19.259652+0000","last_became_peered":"2023-06-10T18:56:19.259652+0000","last_unstale":"2023-06-10T18:56:20.247134+0000","last_undegraded":"2023-06-10T18:56:20.247134+0000","last_fullsized":"2023-06-10T18:56:20.247134+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:55:49.705761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518585+0000","last_change":"2023-06-10T18:56:19.249394+0000","last_active":"2023-06-10T18:56:20.518585+0000","last_peered":"2023-06-10T18:56:20.518585+0000","last_clean":"2023-06-10T18:56:20.518585+0000","last_became_active":"2023-06-10T18:56:19.249071+0000","last_became_peered":"2023-06-10T18:56:19.249071+0000","last_unstale":"2023-06-10T18:56:20.518585+0000","last_undegraded":"2023-06-10T18:56:20.518585+0000","last_fullsized":"2023-06-10T18:56:20.518585+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:25:51.817123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519755+0000","last_change":"2023-06-10T18:56:19.254911+0000","last_active":"2023-06-10T18:56:20.519755+0000","last_peered":"2023-06-10T18:56:20.519755+0000","last_clean":"2023-06-10T18:56:20.519755+0000","last_became_active":"2023-06-10T18:56:19.254602+0000","last_became_peered":"2023-06-10T18:56:19.254602+0000","last_unstale":"2023-06-10T18:56:20.519755+0000","last_undegraded":"2023-06-10T18:56:20.519755+0000","last_fullsized":"2023-06-10T18:56:20.519755+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:34.658391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518574+0000","last_change":"2023-06-10T18:56:19.260895+0000","last_active":"2023-06-10T18:56:20.518574+0000","last_peered":"2023-06-10T18:56:20.518574+0000","last_clean":"2023-06-10T18:56:20.518574+0000","last_became_active":"2023-06-10T18:56:19.260606+0000","last_became_peered":"2023-06-10T18:56:19.260606+0000","last_unstale":"2023-06-10T18:56:20.518574+0000","last_undegraded":"2023-06-10T18:56:20.518574+0000","last_fullsized":"2023-06-10T18:56:20.518574+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:16:32.953543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246945+0000","last_change":"2023-06-10T18:56:19.256069+0000","last_active":"2023-06-10T18:56:20.246945+0000","last_peered":"2023-06-10T18:56:20.246945+0000","last_clean":"2023-06-10T18:56:20.246945+0000","last_became_active":"2023-06-10T18:56:19.255815+0000","last_became_peered":"2023-06-10T18:56:19.255815+0000","last_unstale":"2023-06-10T18:56:20.246945+0000","last_undegraded":"2023-06-10T18:56:20.246945+0000","last_fullsized":"2023-06-10T18:56:20.246945+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:01:34.658391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518620+0000","last_change":"2023-06-10T18:56:19.258711+0000","last_active":"2023-06-10T18:56:20.518620+0000","last_peered":"2023-06-10T18:56:20.518620+0000","last_clean":"2023-06-10T18:56:20.518620+0000","last_became_active":"2023-06-10T18:56:19.258467+0000","last_became_peered":"2023-06-10T18:56:19.258467+0000","last_unstale":"2023-06-10T18:56:20.518620+0000","last_undegraded":"2023-06-10T18:56:20.518620+0000","last_fullsized":"2023-06-10T18:56:20.518620+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:45.582902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245947+0000","last_change":"2023-06-10T18:56:19.251934+0000","last_active":"2023-06-10T18:56:20.245947+0000","last_peered":"2023-06-10T18:56:20.245947+0000","last_clean":"2023-06-10T18:56:20.245947+0000","last_became_active":"2023-06-10T18:56:19.251504+0000","last_became_peered":"2023-06-10T18:56:19.251504+0000","last_unstale":"2023-06-10T18:56:20.245947+0000","last_undegraded":"2023-06-10T18:56:20.245947+0000","last_fullsized":"2023-06-10T18:56:20.245947+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:03:54.139860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519955+0000","last_change":"2023-06-10T18:56:19.250149+0000","last_active":"2023-06-10T18:56:20.519955+0000","last_peered":"2023-06-10T18:56:20.519955+0000","last_clean":"2023-06-10T18:56:20.519955+0000","last_became_active":"2023-06-10T18:56:19.249849+0000","last_became_peered":"2023-06-10T18:56:19.249849+0000","last_unstale":"2023-06-10T18:56:20.519955+0000","last_undegraded":"2023-06-10T18:56:20.519955+0000","last_fullsized":"2023-06-10T18:56:20.519955+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:12:10.542695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245670+0000","last_change":"2023-06-10T18:56:19.261227+0000","last_active":"2023-06-10T18:56:20.245670+0000","last_peered":"2023-06-10T18:56:20.245670+0000","last_clean":"2023-06-10T18:56:20.245670+0000","last_became_active":"2023-06-10T18:56:19.260963+0000","last_became_peered":"2023-06-10T18:56:19.260963+0000","last_unstale":"2023-06-10T18:56:20.245670+0000","last_undegraded":"2023-06-10T18:56:20.245670+0000","last_fullsized":"2023-06-10T18:56:20.245670+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:31:11.098566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518716+0000","last_change":"2023-06-10T18:56:19.253019+0000","last_active":"2023-06-10T18:56:20.518716+0000","last_peered":"2023-06-10T18:56:20.518716+0000","last_clean":"2023-06-10T18:56:20.518716+0000","last_became_active":"2023-06-10T18:56:19.252773+0000","last_became_peered":"2023-06-10T18:56:19.252773+0000","last_unstale":"2023-06-10T18:56:20.518716+0000","last_undegraded":"2023-06-10T18:56:20.518716+0000","last_fullsized":"2023-06-10T18:56:20.518716+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:26.353786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245803+0000","last_change":"2023-06-10T18:56:19.246982+0000","last_active":"2023-06-10T18:56:20.245803+0000","last_peered":"2023-06-10T18:56:20.245803+0000","last_clean":"2023-06-10T18:56:20.245803+0000","last_became_active":"2023-06-10T18:56:19.246740+0000","last_became_peered":"2023-06-10T18:56:19.246740+0000","last_unstale":"2023-06-10T18:56:20.245803+0000","last_undegraded":"2023-06-10T18:56:20.245803+0000","last_fullsized":"2023-06-10T18:56:20.245803+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:52:47.611860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519926+0000","last_change":"2023-06-10T18:56:19.255278+0000","last_active":"2023-06-10T18:56:20.519926+0000","last_peered":"2023-06-10T18:56:20.519926+0000","last_clean":"2023-06-10T18:56:20.519926+0000","last_became_active":"2023-06-10T18:56:19.254821+0000","last_became_peered":"2023-06-10T18:56:19.254821+0000","last_unstale":"2023-06-10T18:56:20.519926+0000","last_undegraded":"2023-06-10T18:56:20.519926+0000","last_fullsized":"2023-06-10T18:56:20.519926+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:23:49.366207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518471+0000","last_change":"2023-06-10T18:56:19.249063+0000","last_active":"2023-06-10T18:56:20.518471+0000","last_peered":"2023-06-10T18:56:20.518471+0000","last_clean":"2023-06-10T18:56:20.518471+0000","last_became_active":"2023-06-10T18:56:19.248806+0000","last_became_peered":"2023-06-10T18:56:19.248806+0000","last_unstale":"2023-06-10T18:56:20.518471+0000","last_undegraded":"2023-06-10T18:56:20.518471+0000","last_fullsized":"2023-06-10T18:56:20.518471+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:36.508790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519150+0000","last_change":"2023-06-10T18:56:19.255772+0000","last_active":"2023-06-10T18:56:20.519150+0000","last_peered":"2023-06-10T18:56:20.519150+0000","last_clean":"2023-06-10T18:56:20.519150+0000","last_became_active":"2023-06-10T18:56:19.255516+0000","last_became_peered":"2023-06-10T18:56:19.255516+0000","last_unstale":"2023-06-10T18:56:20.519150+0000","last_undegraded":"2023-06-10T18:56:20.519150+0000","last_fullsized":"2023-06-10T18:56:20.519150+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:44.558873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518151+0000","last_change":"2023-06-10T18:56:19.244392+0000","last_active":"2023-06-10T18:56:20.518151+0000","last_peered":"2023-06-10T18:56:20.518151+0000","last_clean":"2023-06-10T18:56:20.518151+0000","last_became_active":"2023-06-10T18:56:19.244118+0000","last_became_peered":"2023-06-10T18:56:19.244118+0000","last_unstale":"2023-06-10T18:56:20.518151+0000","last_undegraded":"2023-06-10T18:56:20.518151+0000","last_fullsized":"2023-06-10T18:56:20.518151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.917468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247386+0000","last_change":"2023-06-10T18:56:19.255155+0000","last_active":"2023-06-10T18:56:20.247386+0000","last_peered":"2023-06-10T18:56:20.247386+0000","last_clean":"2023-06-10T18:56:20.247386+0000","last_became_active":"2023-06-10T18:56:19.254849+0000","last_became_peered":"2023-06-10T18:56:19.254849+0000","last_unstale":"2023-06-10T18:56:20.247386+0000","last_undegraded":"2023-06-10T18:56:20.247386+0000","last_fullsized":"2023-06-10T18:56:20.247386+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:36.508790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245820+0000","last_change":"2023-06-10T18:56:19.255220+0000","last_active":"2023-06-10T18:56:20.245820+0000","last_peered":"2023-06-10T18:56:20.245820+0000","last_clean":"2023-06-10T18:56:20.245820+0000","last_became_active":"2023-06-10T18:56:19.254910+0000","last_became_peered":"2023-06-10T18:56:19.254910+0000","last_unstale":"2023-06-10T18:56:20.245820+0000","last_undegraded":"2023-06-10T18:56:20.245820+0000","last_fullsized":"2023-06-10T18:56:20.245820+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:31:08.500560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518415+0000","last_change":"2023-06-10T18:56:19.261716+0000","last_active":"2023-06-10T18:56:20.518415+0000","last_peered":"2023-06-10T18:56:20.518415+0000","last_clean":"2023-06-10T18:56:20.518415+0000","last_became_active":"2023-06-10T18:56:19.261482+0000","last_became_peered":"2023-06-10T18:56:19.261482+0000","last_unstale":"2023-06-10T18:56:20.518415+0000","last_undegraded":"2023-06-10T18:56:20.518415+0000","last_fullsized":"2023-06-10T18:56:20.518415+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:35:41.624038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246612+0000","last_change":"2023-06-10T18:56:19.252081+0000","last_active":"2023-06-10T18:56:20.246612+0000","last_peered":"2023-06-10T18:56:20.246612+0000","last_clean":"2023-06-10T18:56:20.246612+0000","last_became_active":"2023-06-10T18:56:19.251785+0000","last_became_peered":"2023-06-10T18:56:19.251785+0000","last_unstale":"2023-06-10T18:56:20.246612+0000","last_undegraded":"2023-06-10T18:56:20.246612+0000","last_fullsized":"2023-06-10T18:56:20.246612+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:00.942133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518939+0000","last_change":"2023-06-10T18:56:19.265399+0000","last_active":"2023-06-10T18:56:20.518939+0000","last_peered":"2023-06-10T18:56:20.518939+0000","last_clean":"2023-06-10T18:56:20.518939+0000","last_became_active":"2023-06-10T18:56:19.265228+0000","last_became_peered":"2023-06-10T18:56:19.265228+0000","last_unstale":"2023-06-10T18:56:20.518939+0000","last_undegraded":"2023-06-10T18:56:20.518939+0000","last_fullsized":"2023-06-10T18:56:20.518939+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:46:42.880931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518680+0000","last_change":"2023-06-10T18:56:19.258766+0000","last_active":"2023-06-10T18:56:20.518680+0000","last_peered":"2023-06-10T18:56:20.518680+0000","last_clean":"2023-06-10T18:56:20.518680+0000","last_became_active":"2023-06-10T18:56:19.258522+0000","last_became_peered":"2023-06-10T18:56:19.258522+0000","last_unstale":"2023-06-10T18:56:20.518680+0000","last_undegraded":"2023-06-10T18:56:20.518680+0000","last_fullsized":"2023-06-10T18:56:20.518680+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:41:48.275015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520172+0000","last_change":"2023-06-10T18:56:19.262850+0000","last_active":"2023-06-10T18:56:20.520172+0000","last_peered":"2023-06-10T18:56:20.520172+0000","last_clean":"2023-06-10T18:56:20.520172+0000","last_became_active":"2023-06-10T18:56:19.262347+0000","last_became_peered":"2023-06-10T18:56:19.262347+0000","last_unstale":"2023-06-10T18:56:20.520172+0000","last_undegraded":"2023-06-10T18:56:20.520172+0000","last_fullsized":"2023-06-10T18:56:20.520172+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:47:32.514870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520262+0000","last_change":"2023-06-10T18:56:19.250803+0000","last_active":"2023-06-10T18:56:20.520262+0000","last_peered":"2023-06-10T18:56:20.520262+0000","last_clean":"2023-06-10T18:56:20.520262+0000","last_became_active":"2023-06-10T18:56:19.250389+0000","last_became_peered":"2023-06-10T18:56:19.250389+0000","last_unstale":"2023-06-10T18:56:20.520262+0000","last_undegraded":"2023-06-10T18:56:20.520262+0000","last_fullsized":"2023-06-10T18:56:20.520262+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:50:33.873950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245763+0000","last_change":"2023-06-10T18:56:19.252384+0000","last_active":"2023-06-10T18:56:20.245763+0000","last_peered":"2023-06-10T18:56:20.245763+0000","last_clean":"2023-06-10T18:56:20.245763+0000","last_became_active":"2023-06-10T18:56:19.252118+0000","last_became_peered":"2023-06-10T18:56:19.252118+0000","last_unstale":"2023-06-10T18:56:20.245763+0000","last_undegraded":"2023-06-10T18:56:20.245763+0000","last_fullsized":"2023-06-10T18:56:20.245763+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:47:55.643000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"127'4","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519711+0000","last_change":"2023-06-10T18:56:18.853256+0000","last_active":"2023-06-10T18:56:20.519711+0000","last_peered":"2023-06-10T18:56:20.519711+0000","last_clean":"2023-06-10T18:56:20.519711+0000","last_became_active":"2023-06-10T18:56:18.853041+0000","last_became_peered":"2023-06-10T18:56:18.853041+0000","last_unstale":"2023-06-10T18:56:20.519711+0000","last_undegraded":"2023-06-10T18:56:20.519711+0000","last_fullsized":"2023-06-10T18:56:20.519711+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:16.005018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519542+0000","last_change":"2023-06-10T18:56:19.259353+0000","last_active":"2023-06-10T18:56:20.519542+0000","last_peered":"2023-06-10T18:56:20.519542+0000","last_clean":"2023-06-10T18:56:20.519542+0000","last_became_active":"2023-06-10T18:56:19.259030+0000","last_became_peered":"2023-06-10T18:56:19.259030+0000","last_unstale":"2023-06-10T18:56:20.519542+0000","last_undegraded":"2023-06-10T18:56:20.519542+0000","last_fullsized":"2023-06-10T18:56:20.519542+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:03.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246823+0000","last_change":"2023-06-10T18:56:18.848880+0000","last_active":"2023-06-10T18:56:20.246823+0000","last_peered":"2023-06-10T18:56:20.246823+0000","last_clean":"2023-06-10T18:56:20.246823+0000","last_became_active":"2023-06-10T18:56:18.848539+0000","last_became_peered":"2023-06-10T18:56:18.848539+0000","last_unstale":"2023-06-10T18:56:20.246823+0000","last_undegraded":"2023-06-10T18:56:20.246823+0000","last_fullsized":"2023-06-10T18:56:20.246823+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:33:25.762698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519997+0000","last_change":"2023-06-10T18:56:19.263608+0000","last_active":"2023-06-10T18:56:20.519997+0000","last_peered":"2023-06-10T18:56:20.519997+0000","last_clean":"2023-06-10T18:56:20.519997+0000","last_became_active":"2023-06-10T18:56:19.263363+0000","last_became_peered":"2023-06-10T18:56:19.263363+0000","last_unstale":"2023-06-10T18:56:20.519997+0000","last_undegraded":"2023-06-10T18:56:20.519997+0000","last_fullsized":"2023-06-10T18:56:20.519997+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:44.904473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518836+0000","last_change":"2023-06-10T18:56:18.843662+0000","last_active":"2023-06-10T18:56:20.518836+0000","last_peered":"2023-06-10T18:56:20.518836+0000","last_clean":"2023-06-10T18:56:20.518836+0000","last_became_active":"2023-06-10T18:56:18.843387+0000","last_became_peered":"2023-06-10T18:56:18.843387+0000","last_unstale":"2023-06-10T18:56:20.518836+0000","last_undegraded":"2023-06-10T18:56:20.518836+0000","last_fullsized":"2023-06-10T18:56:20.518836+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:07:04.352439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519150+0000","last_change":"2023-06-10T18:56:19.258632+0000","last_active":"2023-06-10T18:56:20.519150+0000","last_peered":"2023-06-10T18:56:20.519150+0000","last_clean":"2023-06-10T18:56:20.519150+0000","last_became_active":"2023-06-10T18:56:19.258265+0000","last_became_peered":"2023-06-10T18:56:19.258265+0000","last_unstale":"2023-06-10T18:56:20.519150+0000","last_undegraded":"2023-06-10T18:56:20.519150+0000","last_fullsized":"2023-06-10T18:56:20.519150+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:44:50.097515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247044+0000","last_change":"2023-06-10T18:56:18.850661+0000","last_active":"2023-06-10T18:56:20.247044+0000","last_peered":"2023-06-10T18:56:20.247044+0000","last_clean":"2023-06-10T18:56:20.247044+0000","last_became_active":"2023-06-10T18:56:18.847270+0000","last_became_peered":"2023-06-10T18:56:18.847270+0000","last_unstale":"2023-06-10T18:56:20.247044+0000","last_undegraded":"2023-06-10T18:56:20.247044+0000","last_fullsized":"2023-06-10T18:56:20.247044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:50:19.654934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245839+0000","last_change":"2023-06-10T18:56:19.261256+0000","last_active":"2023-06-10T18:56:20.245839+0000","last_peered":"2023-06-10T18:56:20.245839+0000","last_clean":"2023-06-10T18:56:20.245839+0000","last_became_active":"2023-06-10T18:56:19.261024+0000","last_became_peered":"2023-06-10T18:56:19.261024+0000","last_unstale":"2023-06-10T18:56:20.245839+0000","last_undegraded":"2023-06-10T18:56:20.245839+0000","last_fullsized":"2023-06-10T18:56:20.245839+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:28:11.857799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246795+0000","last_change":"2023-06-10T18:56:18.847863+0000","last_active":"2023-06-10T18:56:20.246795+0000","last_peered":"2023-06-10T18:56:20.246795+0000","last_clean":"2023-06-10T18:56:20.246795+0000","last_became_active":"2023-06-10T18:56:18.846987+0000","last_became_peered":"2023-06-10T18:56:18.846987+0000","last_unstale":"2023-06-10T18:56:20.246795+0000","last_undegraded":"2023-06-10T18:56:20.246795+0000","last_fullsized":"2023-06-10T18:56:20.246795+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:02.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519180+0000","last_change":"2023-06-10T18:56:19.250165+0000","last_active":"2023-06-10T18:56:20.519180+0000","last_peered":"2023-06-10T18:56:20.519180+0000","last_clean":"2023-06-10T18:56:20.519180+0000","last_became_active":"2023-06-10T18:56:19.249871+0000","last_became_peered":"2023-06-10T18:56:19.249871+0000","last_unstale":"2023-06-10T18:56:20.519180+0000","last_undegraded":"2023-06-10T18:56:20.519180+0000","last_fullsized":"2023-06-10T18:56:20.519180+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:42:24.041414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245892+0000","last_change":"2023-06-10T18:56:18.846176+0000","last_active":"2023-06-10T18:56:20.245892+0000","last_peered":"2023-06-10T18:56:20.245892+0000","last_clean":"2023-06-10T18:56:20.245892+0000","last_became_active":"2023-06-10T18:56:18.845916+0000","last_became_peered":"2023-06-10T18:56:18.845916+0000","last_unstale":"2023-06-10T18:56:20.245892+0000","last_undegraded":"2023-06-10T18:56:20.245892+0000","last_fullsized":"2023-06-10T18:56:20.245892+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:28.537547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519480+0000","last_change":"2023-06-10T18:56:19.256374+0000","last_active":"2023-06-10T18:56:20.519480+0000","last_peered":"2023-06-10T18:56:20.519480+0000","last_clean":"2023-06-10T18:56:20.519480+0000","last_became_active":"2023-06-10T18:56:19.255002+0000","last_became_peered":"2023-06-10T18:56:19.255002+0000","last_unstale":"2023-06-10T18:56:20.519480+0000","last_undegraded":"2023-06-10T18:56:20.519480+0000","last_fullsized":"2023-06-10T18:56:20.519480+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:37:33.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520237+0000","last_change":"2023-06-10T18:56:18.847846+0000","last_active":"2023-06-10T18:56:20.520237+0000","last_peered":"2023-06-10T18:56:20.520237+0000","last_clean":"2023-06-10T18:56:20.520237+0000","last_became_active":"2023-06-10T18:56:18.847574+0000","last_became_peered":"2023-06-10T18:56:18.847574+0000","last_unstale":"2023-06-10T18:56:20.520237+0000","last_undegraded":"2023-06-10T18:56:20.520237+0000","last_fullsized":"2023-06-10T18:56:20.520237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:27:46.670228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246859+0000","last_change":"2023-06-10T18:56:19.245734+0000","last_active":"2023-06-10T18:56:20.246859+0000","last_peered":"2023-06-10T18:56:20.246859+0000","last_clean":"2023-06-10T18:56:20.246859+0000","last_became_active":"2023-06-10T18:56:19.245539+0000","last_became_peered":"2023-06-10T18:56:19.245539+0000","last_unstale":"2023-06-10T18:56:20.246859+0000","last_undegraded":"2023-06-10T18:56:20.246859+0000","last_fullsized":"2023-06-10T18:56:20.246859+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:42:26.350809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246772+0000","last_change":"2023-06-10T18:56:18.847196+0000","last_active":"2023-06-10T18:56:20.246772+0000","last_peered":"2023-06-10T18:56:20.246772+0000","last_clean":"2023-06-10T18:56:20.246772+0000","last_became_active":"2023-06-10T18:56:18.846894+0000","last_became_peered":"2023-06-10T18:56:18.846894+0000","last_unstale":"2023-06-10T18:56:20.246772+0000","last_undegraded":"2023-06-10T18:56:20.246772+0000","last_fullsized":"2023-06-10T18:56:20.246772+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:37:32.116803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519991+0000","last_change":"2023-06-10T18:56:19.258695+0000","last_active":"2023-06-10T18:56:20.519991+0000","last_peered":"2023-06-10T18:56:20.519991+0000","last_clean":"2023-06-10T18:56:20.519991+0000","last_became_active":"2023-06-10T18:56:19.258388+0000","last_became_peered":"2023-06-10T18:56:19.258388+0000","last_unstale":"2023-06-10T18:56:20.519991+0000","last_undegraded":"2023-06-10T18:56:20.519991+0000","last_fullsized":"2023-06-10T18:56:20.519991+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:50:20.661715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247001+0000","last_change":"2023-06-10T18:56:18.850289+0000","last_active":"2023-06-10T18:56:20.247001+0000","last_peered":"2023-06-10T18:56:20.247001+0000","last_clean":"2023-06-10T18:56:20.247001+0000","last_became_active":"2023-06-10T18:56:18.850034+0000","last_became_peered":"2023-06-10T18:56:18.850034+0000","last_unstale":"2023-06-10T18:56:20.247001+0000","last_undegraded":"2023-06-10T18:56:20.247001+0000","last_fullsized":"2023-06-10T18:56:20.247001+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:05.996772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246836+0000","last_change":"2023-06-10T18:56:19.251727+0000","last_active":"2023-06-10T18:56:20.246836+0000","last_peered":"2023-06-10T18:56:20.246836+0000","last_clean":"2023-06-10T18:56:20.246836+0000","last_became_active":"2023-06-10T18:56:19.251400+0000","last_became_peered":"2023-06-10T18:56:19.251400+0000","last_unstale":"2023-06-10T18:56:20.246836+0000","last_undegraded":"2023-06-10T18:56:20.246836+0000","last_fullsized":"2023-06-10T18:56:20.246836+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:03:56.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518418+0000","last_change":"2023-06-10T18:56:18.842107+0000","last_active":"2023-06-10T18:56:20.518418+0000","last_peered":"2023-06-10T18:56:20.518418+0000","last_clean":"2023-06-10T18:56:20.518418+0000","last_became_active":"2023-06-10T18:56:18.841780+0000","last_became_peered":"2023-06-10T18:56:18.841780+0000","last_unstale":"2023-06-10T18:56:20.518418+0000","last_undegraded":"2023-06-10T18:56:20.518418+0000","last_fullsized":"2023-06-10T18:56:20.518418+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:36:33.630870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245865+0000","last_change":"2023-06-10T18:56:19.251710+0000","last_active":"2023-06-10T18:56:20.245865+0000","last_peered":"2023-06-10T18:56:20.245865+0000","last_clean":"2023-06-10T18:56:20.245865+0000","last_became_active":"2023-06-10T18:56:19.251258+0000","last_became_peered":"2023-06-10T18:56:19.251258+0000","last_unstale":"2023-06-10T18:56:20.245865+0000","last_undegraded":"2023-06-10T18:56:20.245865+0000","last_fullsized":"2023-06-10T18:56:20.245865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:13:52.908092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"127'1","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519279+0000","last_change":"2023-06-10T18:56:18.851511+0000","last_active":"2023-06-10T18:56:20.519279+0000","last_peered":"2023-06-10T18:56:20.519279+0000","last_clean":"2023-06-10T18:56:20.519279+0000","last_became_active":"2023-06-10T18:56:18.851186+0000","last_became_peered":"2023-06-10T18:56:18.851186+0000","last_unstale":"2023-06-10T18:56:20.519279+0000","last_undegraded":"2023-06-10T18:56:20.519279+0000","last_fullsized":"2023-06-10T18:56:20.519279+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:01:16.501429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247139+0000","last_change":"2023-06-10T18:56:19.246608+0000","last_active":"2023-06-10T18:56:20.247139+0000","last_peered":"2023-06-10T18:56:20.247139+0000","last_clean":"2023-06-10T18:56:20.247139+0000","last_became_active":"2023-06-10T18:56:19.246369+0000","last_became_peered":"2023-06-10T18:56:19.246369+0000","last_unstale":"2023-06-10T18:56:20.247139+0000","last_undegraded":"2023-06-10T18:56:20.247139+0000","last_fullsized":"2023-06-10T18:56:20.247139+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:32.917468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246701+0000","last_change":"2023-06-10T18:56:18.848967+0000","last_active":"2023-06-10T18:56:20.246701+0000","last_peered":"2023-06-10T18:56:20.246701+0000","last_clean":"2023-06-10T18:56:20.246701+0000","last_became_active":"2023-06-10T18:56:18.848459+0000","last_became_peered":"2023-06-10T18:56:18.848459+0000","last_unstale":"2023-06-10T18:56:20.246701+0000","last_undegraded":"2023-06-10T18:56:20.246701+0000","last_fullsized":"2023-06-10T18:56:20.246701+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:19.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518520+0000","last_change":"2023-06-10T18:56:19.261143+0000","last_active":"2023-06-10T18:56:20.518520+0000","last_peered":"2023-06-10T18:56:20.518520+0000","last_clean":"2023-06-10T18:56:20.518520+0000","last_became_active":"2023-06-10T18:56:19.260888+0000","last_became_peered":"2023-06-10T18:56:19.260888+0000","last_unstale":"2023-06-10T18:56:20.518520+0000","last_undegraded":"2023-06-10T18:56:20.518520+0000","last_fullsized":"2023-06-10T18:56:20.518520+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:12:27.952753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246460+0000","last_change":"2023-06-10T18:56:18.852961+0000","last_active":"2023-06-10T18:56:20.246460+0000","last_peered":"2023-06-10T18:56:20.246460+0000","last_clean":"2023-06-10T18:56:20.246460+0000","last_became_active":"2023-06-10T18:56:18.852501+0000","last_became_peered":"2023-06-10T18:56:18.852501+0000","last_unstale":"2023-06-10T18:56:20.246460+0000","last_undegraded":"2023-06-10T18:56:20.246460+0000","last_fullsized":"2023-06-10T18:56:20.246460+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:16.011497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519213+0000","last_change":"2023-06-10T18:56:19.258451+0000","last_active":"2023-06-10T18:56:20.519213+0000","last_peered":"2023-06-10T18:56:20.519213+0000","last_clean":"2023-06-10T18:56:20.519213+0000","last_became_active":"2023-06-10T18:56:19.258165+0000","last_became_peered":"2023-06-10T18:56:19.258165+0000","last_unstale":"2023-06-10T18:56:20.519213+0000","last_undegraded":"2023-06-10T18:56:20.519213+0000","last_fullsized":"2023-06-10T18:56:20.519213+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:56:20.546763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245961+0000","last_change":"2023-06-10T18:56:18.843168+0000","last_active":"2023-06-10T18:56:20.245961+0000","last_peered":"2023-06-10T18:56:20.245961+0000","last_clean":"2023-06-10T18:56:20.245961+0000","last_became_active":"2023-06-10T18:56:18.842943+0000","last_became_peered":"2023-06-10T18:56:18.842943+0000","last_unstale":"2023-06-10T18:56:20.245961+0000","last_undegraded":"2023-06-10T18:56:20.245961+0000","last_fullsized":"2023-06-10T18:56:20.245961+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:06:53.564832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246689+0000","last_change":"2023-06-10T18:56:19.248252+0000","last_active":"2023-06-10T18:56:20.246689+0000","last_peered":"2023-06-10T18:56:20.246689+0000","last_clean":"2023-06-10T18:56:20.246689+0000","last_became_active":"2023-06-10T18:56:19.247996+0000","last_became_peered":"2023-06-10T18:56:19.247996+0000","last_unstale":"2023-06-10T18:56:20.246689+0000","last_undegraded":"2023-06-10T18:56:20.246689+0000","last_fullsized":"2023-06-10T18:56:20.246689+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:27:29.569098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245558+0000","last_change":"2023-06-10T18:56:18.850692+0000","last_active":"2023-06-10T18:56:20.245558+0000","last_peered":"2023-06-10T18:56:20.245558+0000","last_clean":"2023-06-10T18:56:20.245558+0000","last_became_active":"2023-06-10T18:56:18.850314+0000","last_became_peered":"2023-06-10T18:56:18.850314+0000","last_unstale":"2023-06-10T18:56:20.245558+0000","last_undegraded":"2023-06-10T18:56:20.245558+0000","last_fullsized":"2023-06-10T18:56:20.245558+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:07:41.036551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520262+0000","last_change":"2023-06-10T18:56:19.262645+0000","last_active":"2023-06-10T18:56:20.520262+0000","last_peered":"2023-06-10T18:56:20.520262+0000","last_clean":"2023-06-10T18:56:20.520262+0000","last_became_active":"2023-06-10T18:56:19.261931+0000","last_became_peered":"2023-06-10T18:56:19.261931+0000","last_unstale":"2023-06-10T18:56:20.520262+0000","last_undegraded":"2023-06-10T18:56:20.520262+0000","last_fullsized":"2023-06-10T18:56:20.520262+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:19:10.184151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518910+0000","last_change":"2023-06-10T18:56:18.843402+0000","last_active":"2023-06-10T18:56:20.518910+0000","last_peered":"2023-06-10T18:56:20.518910+0000","last_clean":"2023-06-10T18:56:20.518910+0000","last_became_active":"2023-06-10T18:56:18.843091+0000","last_became_peered":"2023-06-10T18:56:18.843091+0000","last_unstale":"2023-06-10T18:56:20.518910+0000","last_undegraded":"2023-06-10T18:56:20.518910+0000","last_fullsized":"2023-06-10T18:56:20.518910+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:45:32.097648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519729+0000","last_change":"2023-06-10T18:56:19.260813+0000","last_active":"2023-06-10T18:56:20.519729+0000","last_peered":"2023-06-10T18:56:20.519729+0000","last_clean":"2023-06-10T18:56:20.519729+0000","last_became_active":"2023-06-10T18:56:19.260109+0000","last_became_peered":"2023-06-10T18:56:19.260109+0000","last_unstale":"2023-06-10T18:56:20.519729+0000","last_undegraded":"2023-06-10T18:56:20.519729+0000","last_fullsized":"2023-06-10T18:56:20.519729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:03:55.423532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518873+0000","last_change":"2023-06-10T18:56:18.851835+0000","last_active":"2023-06-10T18:56:20.518873+0000","last_peered":"2023-06-10T18:56:20.518873+0000","last_clean":"2023-06-10T18:56:20.518873+0000","last_became_active":"2023-06-10T18:56:18.851500+0000","last_became_peered":"2023-06-10T18:56:18.851500+0000","last_unstale":"2023-06-10T18:56:20.518873+0000","last_undegraded":"2023-06-10T18:56:20.518873+0000","last_fullsized":"2023-06-10T18:56:20.518873+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:42:00.197990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518742+0000","last_change":"2023-06-10T18:56:19.258700+0000","last_active":"2023-06-10T18:56:20.518742+0000","last_peered":"2023-06-10T18:56:20.518742+0000","last_clean":"2023-06-10T18:56:20.518742+0000","last_became_active":"2023-06-10T18:56:19.258444+0000","last_became_peered":"2023-06-10T18:56:19.258444+0000","last_unstale":"2023-06-10T18:56:20.518742+0000","last_undegraded":"2023-06-10T18:56:20.518742+0000","last_fullsized":"2023-06-10T18:56:20.518742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:43:06.568005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519391+0000","last_change":"2023-06-10T18:56:18.847973+0000","last_active":"2023-06-10T18:56:20.519391+0000","last_peered":"2023-06-10T18:56:20.519391+0000","last_clean":"2023-06-10T18:56:20.519391+0000","last_became_active":"2023-06-10T18:56:18.846967+0000","last_became_peered":"2023-06-10T18:56:18.846967+0000","last_unstale":"2023-06-10T18:56:20.519391+0000","last_undegraded":"2023-06-10T18:56:20.519391+0000","last_fullsized":"2023-06-10T18:56:20.519391+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:03:54.133766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247174+0000","last_change":"2023-06-10T18:56:19.256774+0000","last_active":"2023-06-10T18:56:20.247174+0000","last_peered":"2023-06-10T18:56:20.247174+0000","last_clean":"2023-06-10T18:56:20.247174+0000","last_became_active":"2023-06-10T18:56:19.256514+0000","last_became_peered":"2023-06-10T18:56:19.256514+0000","last_unstale":"2023-06-10T18:56:20.247174+0000","last_undegraded":"2023-06-10T18:56:20.247174+0000","last_fullsized":"2023-06-10T18:56:20.247174+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:37.827141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247254+0000","last_change":"2023-06-10T18:56:18.844664+0000","last_active":"2023-06-10T18:56:20.247254+0000","last_peered":"2023-06-10T18:56:20.247254+0000","last_clean":"2023-06-10T18:56:20.247254+0000","last_became_active":"2023-06-10T18:56:18.844378+0000","last_became_peered":"2023-06-10T18:56:18.844378+0000","last_unstale":"2023-06-10T18:56:20.247254+0000","last_undegraded":"2023-06-10T18:56:20.247254+0000","last_fullsized":"2023-06-10T18:56:20.247254+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:14.213391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519137+0000","last_change":"2023-06-10T18:56:19.259157+0000","last_active":"2023-06-10T18:56:20.519137+0000","last_peered":"2023-06-10T18:56:20.519137+0000","last_clean":"2023-06-10T18:56:20.519137+0000","last_became_active":"2023-06-10T18:56:19.258900+0000","last_became_peered":"2023-06-10T18:56:19.258900+0000","last_unstale":"2023-06-10T18:56:20.519137+0000","last_undegraded":"2023-06-10T18:56:20.519137+0000","last_fullsized":"2023-06-10T18:56:20.519137+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:31:06.947960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518861+0000","last_change":"2023-06-10T18:56:18.846358+0000","last_active":"2023-06-10T18:56:20.518861+0000","last_peered":"2023-06-10T18:56:20.518861+0000","last_clean":"2023-06-10T18:56:20.518861+0000","last_became_active":"2023-06-10T18:56:18.846074+0000","last_became_peered":"2023-06-10T18:56:18.846074+0000","last_unstale":"2023-06-10T18:56:20.518861+0000","last_undegraded":"2023-06-10T18:56:20.518861+0000","last_fullsized":"2023-06-10T18:56:20.518861+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:18:14.213391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519552+0000","last_change":"2023-06-10T18:56:19.262755+0000","last_active":"2023-06-10T18:56:20.519552+0000","last_peered":"2023-06-10T18:56:20.519552+0000","last_clean":"2023-06-10T18:56:20.519552+0000","last_became_active":"2023-06-10T18:56:19.262476+0000","last_became_peered":"2023-06-10T18:56:19.262476+0000","last_unstale":"2023-06-10T18:56:20.519552+0000","last_undegraded":"2023-06-10T18:56:20.519552+0000","last_fullsized":"2023-06-10T18:56:20.519552+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:14:21.738968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246862+0000","last_change":"2023-06-10T18:56:18.845783+0000","last_active":"2023-06-10T18:56:20.246862+0000","last_peered":"2023-06-10T18:56:20.246862+0000","last_clean":"2023-06-10T18:56:20.246862+0000","last_became_active":"2023-06-10T18:56:18.845251+0000","last_became_peered":"2023-06-10T18:56:18.845251+0000","last_unstale":"2023-06-10T18:56:20.246862+0000","last_undegraded":"2023-06-10T18:56:20.246862+0000","last_fullsized":"2023-06-10T18:56:20.246862+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:06.641446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246449+0000","last_change":"2023-06-10T18:56:19.256710+0000","last_active":"2023-06-10T18:56:20.246449+0000","last_peered":"2023-06-10T18:56:20.246449+0000","last_clean":"2023-06-10T18:56:20.246449+0000","last_became_active":"2023-06-10T18:56:19.256485+0000","last_became_peered":"2023-06-10T18:56:19.256485+0000","last_unstale":"2023-06-10T18:56:20.246449+0000","last_undegraded":"2023-06-10T18:56:20.246449+0000","last_fullsized":"2023-06-10T18:56:20.246449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:31.814128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520127+0000","last_change":"2023-06-10T18:56:18.847777+0000","last_active":"2023-06-10T18:56:20.520127+0000","last_peered":"2023-06-10T18:56:20.520127+0000","last_clean":"2023-06-10T18:56:20.520127+0000","last_became_active":"2023-06-10T18:56:18.847494+0000","last_became_peered":"2023-06-10T18:56:18.847494+0000","last_unstale":"2023-06-10T18:56:20.520127+0000","last_undegraded":"2023-06-10T18:56:20.520127+0000","last_fullsized":"2023-06-10T18:56:20.520127+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:05:30.131521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245804+0000","last_change":"2023-06-10T18:56:19.253071+0000","last_active":"2023-06-10T18:56:20.245804+0000","last_peered":"2023-06-10T18:56:20.245804+0000","last_clean":"2023-06-10T18:56:20.245804+0000","last_became_active":"2023-06-10T18:56:19.252820+0000","last_became_peered":"2023-06-10T18:56:19.252820+0000","last_unstale":"2023-06-10T18:56:20.245804+0000","last_undegraded":"2023-06-10T18:56:20.245804+0000","last_fullsized":"2023-06-10T18:56:20.245804+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:07:06.433739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245738+0000","last_change":"2023-06-10T18:56:18.851293+0000","last_active":"2023-06-10T18:56:20.245738+0000","last_peered":"2023-06-10T18:56:20.245738+0000","last_clean":"2023-06-10T18:56:20.245738+0000","last_became_active":"2023-06-10T18:56:18.850991+0000","last_became_peered":"2023-06-10T18:56:18.850991+0000","last_unstale":"2023-06-10T18:56:20.245738+0000","last_undegraded":"2023-06-10T18:56:20.245738+0000","last_fullsized":"2023-06-10T18:56:20.245738+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:54:01.094221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246701+0000","last_change":"2023-06-10T18:56:19.256825+0000","last_active":"2023-06-10T18:56:20.246701+0000","last_peered":"2023-06-10T18:56:20.246701+0000","last_clean":"2023-06-10T18:56:20.246701+0000","last_became_active":"2023-06-10T18:56:19.256533+0000","last_became_peered":"2023-06-10T18:56:19.256533+0000","last_unstale":"2023-06-10T18:56:20.246701+0000","last_undegraded":"2023-06-10T18:56:20.246701+0000","last_fullsized":"2023-06-10T18:56:20.246701+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:53:34.855752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518289+0000","last_change":"2023-06-10T18:56:18.845834+0000","last_active":"2023-06-10T18:56:20.518289+0000","last_peered":"2023-06-10T18:56:20.518289+0000","last_clean":"2023-06-10T18:56:20.518289+0000","last_became_active":"2023-06-10T18:56:18.845492+0000","last_became_peered":"2023-06-10T18:56:18.845492+0000","last_unstale":"2023-06-10T18:56:20.518289+0000","last_undegraded":"2023-06-10T18:56:20.518289+0000","last_fullsized":"2023-06-10T18:56:20.518289+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:36.820360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519014+0000","last_change":"2023-06-10T18:56:19.262372+0000","last_active":"2023-06-10T18:56:20.519014+0000","last_peered":"2023-06-10T18:56:20.519014+0000","last_clean":"2023-06-10T18:56:20.519014+0000","last_became_active":"2023-06-10T18:56:19.262061+0000","last_became_peered":"2023-06-10T18:56:19.262061+0000","last_unstale":"2023-06-10T18:56:20.519014+0000","last_undegraded":"2023-06-10T18:56:20.519014+0000","last_fullsized":"2023-06-10T18:56:20.519014+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:11:35.768462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"127'6","reported_seq":19,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519581+0000","last_change":"2023-06-10T18:56:18.845270+0000","last_active":"2023-06-10T18:56:20.519581+0000","last_peered":"2023-06-10T18:56:20.519581+0000","last_clean":"2023-06-10T18:56:20.519581+0000","last_became_active":"2023-06-10T18:56:18.845010+0000","last_became_peered":"2023-06-10T18:56:18.845010+0000","last_unstale":"2023-06-10T18:56:20.519581+0000","last_undegraded":"2023-06-10T18:56:20.519581+0000","last_fullsized":"2023-06-10T18:56:20.519581+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:30:50.870671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":202,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247242+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:56:20.247242+0000","last_peered":"2023-06-10T18:56:20.247242+0000","last_clean":"2023-06-10T18:56:20.247242+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:56:20.247242+0000","last_undegraded":"2023-06-10T18:56:20.247242+0000","last_fullsized":"2023-06-10T18:56:20.247242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:24:18.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518654+0000","last_change":"2023-06-10T18:56:19.255151+0000","last_active":"2023-06-10T18:56:20.518654+0000","last_peered":"2023-06-10T18:56:20.518654+0000","last_clean":"2023-06-10T18:56:20.518654+0000","last_became_active":"2023-06-10T18:56:19.254885+0000","last_became_peered":"2023-06-10T18:56:19.254885+0000","last_unstale":"2023-06-10T18:56:20.518654+0000","last_undegraded":"2023-06-10T18:56:20.518654+0000","last_fullsized":"2023-06-10T18:56:20.518654+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:16:14.135900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518792+0000","last_change":"2023-06-10T18:56:18.853944+0000","last_active":"2023-06-10T18:56:20.518792+0000","last_peered":"2023-06-10T18:56:20.518792+0000","last_clean":"2023-06-10T18:56:20.518792+0000","last_became_active":"2023-06-10T18:56:18.853689+0000","last_became_peered":"2023-06-10T18:56:18.853689+0000","last_unstale":"2023-06-10T18:56:20.518792+0000","last_undegraded":"2023-06-10T18:56:20.518792+0000","last_fullsized":"2023-06-10T18:56:20.518792+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:50.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":228,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246281+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:56:20.246281+0000","last_peered":"2023-06-10T18:56:20.246281+0000","last_clean":"2023-06-10T18:56:20.246281+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:56:20.246281+0000","last_undegraded":"2023-06-10T18:56:20.246281+0000","last_fullsized":"2023-06-10T18:56:20.246281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:16.797895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246654+0000","last_change":"2023-06-10T18:56:19.256478+0000","last_active":"2023-06-10T18:56:20.246654+0000","last_peered":"2023-06-10T18:56:20.246654+0000","last_clean":"2023-06-10T18:56:20.246654+0000","last_became_active":"2023-06-10T18:56:19.256199+0000","last_became_peered":"2023-06-10T18:56:19.256199+0000","last_unstale":"2023-06-10T18:56:20.246654+0000","last_undegraded":"2023-06-10T18:56:20.246654+0000","last_fullsized":"2023-06-10T18:56:20.246654+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:44.558873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246819+0000","last_change":"2023-06-10T18:56:18.845106+0000","last_active":"2023-06-10T18:56:20.246819+0000","last_peered":"2023-06-10T18:56:20.246819+0000","last_clean":"2023-06-10T18:56:20.246819+0000","last_became_active":"2023-06-10T18:56:18.844839+0000","last_became_peered":"2023-06-10T18:56:18.844839+0000","last_unstale":"2023-06-10T18:56:20.246819+0000","last_undegraded":"2023-06-10T18:56:20.246819+0000","last_fullsized":"2023-06-10T18:56:20.246819+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:30:50.870671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":200,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518706+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:56:20.518706+0000","last_peered":"2023-06-10T18:56:20.518706+0000","last_clean":"2023-06-10T18:56:20.518706+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:56:20.518706+0000","last_undegraded":"2023-06-10T18:56:20.518706+0000","last_fullsized":"2023-06-10T18:56:20.518706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:09.166839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519495+0000","last_change":"2023-06-10T18:56:19.257690+0000","last_active":"2023-06-10T18:56:20.519495+0000","last_peered":"2023-06-10T18:56:20.519495+0000","last_clean":"2023-06-10T18:56:20.519495+0000","last_became_active":"2023-06-10T18:56:19.257406+0000","last_became_peered":"2023-06-10T18:56:19.257406+0000","last_unstale":"2023-06-10T18:56:20.519495+0000","last_undegraded":"2023-06-10T18:56:20.519495+0000","last_fullsized":"2023-06-10T18:56:20.519495+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:13.798958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"127'2","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520026+0000","last_change":"2023-06-10T18:56:18.852820+0000","last_active":"2023-06-10T18:56:20.520026+0000","last_peered":"2023-06-10T18:56:20.520026+0000","last_clean":"2023-06-10T18:56:20.520026+0000","last_became_active":"2023-06-10T18:56:18.852540+0000","last_became_peered":"2023-06-10T18:56:18.852540+0000","last_unstale":"2023-06-10T18:56:20.520026+0000","last_undegraded":"2023-06-10T18:56:20.520026+0000","last_fullsized":"2023-06-10T18:56:20.520026+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:30.825236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":228,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246554+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:56:20.246554+0000","last_peered":"2023-06-10T18:56:20.246554+0000","last_clean":"2023-06-10T18:56:20.246554+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:56:20.246554+0000","last_undegraded":"2023-06-10T18:56:20.246554+0000","last_fullsized":"2023-06-10T18:56:20.246554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T18:50:41.734840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518775+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:56:20.518775+0000","last_peered":"2023-06-10T18:56:20.518775+0000","last_clean":"2023-06-10T18:56:20.518775+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:56:20.518775+0000","last_undegraded":"2023-06-10T18:56:20.518775+0000","last_fullsized":"2023-06-10T18:56:20.518775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:00:00.794334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520031+0000","last_change":"2023-06-10T18:56:19.255667+0000","last_active":"2023-06-10T18:56:20.520031+0000","last_peered":"2023-06-10T18:56:20.520031+0000","last_clean":"2023-06-10T18:56:20.520031+0000","last_became_active":"2023-06-10T18:56:19.254988+0000","last_became_peered":"2023-06-10T18:56:19.254988+0000","last_unstale":"2023-06-10T18:56:20.520031+0000","last_undegraded":"2023-06-10T18:56:20.520031+0000","last_fullsized":"2023-06-10T18:56:20.520031+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:55:49.705761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"127'8","reported_seq":21,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519928+0000","last_change":"2023-06-10T18:56:18.846798+0000","last_active":"2023-06-10T18:56:20.519928+0000","last_peered":"2023-06-10T18:56:20.519928+0000","last_clean":"2023-06-10T18:56:20.519928+0000","last_became_active":"2023-06-10T18:56:18.846561+0000","last_became_peered":"2023-06-10T18:56:18.846561+0000","last_unstale":"2023-06-10T18:56:20.519928+0000","last_undegraded":"2023-06-10T18:56:20.519928+0000","last_fullsized":"2023-06-10T18:56:20.519928+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:02.027782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":215,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518572+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:56:20.518572+0000","last_peered":"2023-06-10T18:56:20.518572+0000","last_clean":"2023-06-10T18:56:20.518572+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:56:20.518572+0000","last_undegraded":"2023-06-10T18:56:20.518572+0000","last_fullsized":"2023-06-10T18:56:20.518572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:06.990202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.520137+0000","last_change":"2023-06-10T18:56:19.257337+0000","last_active":"2023-06-10T18:56:20.520137+0000","last_peered":"2023-06-10T18:56:20.520137+0000","last_clean":"2023-06-10T18:56:20.520137+0000","last_became_active":"2023-06-10T18:56:19.256968+0000","last_became_peered":"2023-06-10T18:56:19.256968+0000","last_unstale":"2023-06-10T18:56:20.520137+0000","last_undegraded":"2023-06-10T18:56:20.520137+0000","last_fullsized":"2023-06-10T18:56:20.520137+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:25:21.372024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"127'3","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246534+0000","last_change":"2023-06-10T18:56:18.846399+0000","last_active":"2023-06-10T18:56:20.246534+0000","last_peered":"2023-06-10T18:56:20.246534+0000","last_clean":"2023-06-10T18:56:20.246534+0000","last_became_active":"2023-06-10T18:56:18.846132+0000","last_became_peered":"2023-06-10T18:56:18.846132+0000","last_unstale":"2023-06-10T18:56:20.246534+0000","last_undegraded":"2023-06-10T18:56:20.246534+0000","last_fullsized":"2023-06-10T18:56:20.246534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:42:43.655486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519242+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:56:20.519242+0000","last_peered":"2023-06-10T18:56:20.519242+0000","last_clean":"2023-06-10T18:56:20.519242+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:56:20.519242+0000","last_undegraded":"2023-06-10T18:56:20.519242+0000","last_fullsized":"2023-06-10T18:56:20.519242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:14.316978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246757+0000","last_change":"2023-06-10T18:56:19.253481+0000","last_active":"2023-06-10T18:56:20.246757+0000","last_peered":"2023-06-10T18:56:20.246757+0000","last_clean":"2023-06-10T18:56:20.246757+0000","last_became_active":"2023-06-10T18:56:19.253226+0000","last_became_peered":"2023-06-10T18:56:19.253226+0000","last_unstale":"2023-06-10T18:56:20.246757+0000","last_undegraded":"2023-06-10T18:56:20.246757+0000","last_fullsized":"2023-06-10T18:56:20.246757+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:39.665478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.245912+0000","last_change":"2023-06-10T18:56:18.846585+0000","last_active":"2023-06-10T18:56:20.245912+0000","last_peered":"2023-06-10T18:56:20.245912+0000","last_clean":"2023-06-10T18:56:20.245912+0000","last_became_active":"2023-06-10T18:56:18.846315+0000","last_became_peered":"2023-06-10T18:56:18.846315+0000","last_unstale":"2023-06-10T18:56:20.245912+0000","last_undegraded":"2023-06-10T18:56:20.245912+0000","last_fullsized":"2023-06-10T18:56:20.245912+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:33:42.718180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":202,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.247082+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:56:20.247082+0000","last_peered":"2023-06-10T18:56:20.247082+0000","last_clean":"2023-06-10T18:56:20.247082+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:56:20.247082+0000","last_undegraded":"2023-06-10T18:56:20.247082+0000","last_fullsized":"2023-06-10T18:56:20.247082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:19:53.698784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.518951+0000","last_change":"2023-06-10T18:56:19.254848+0000","last_active":"2023-06-10T18:56:20.518951+0000","last_peered":"2023-06-10T18:56:20.518951+0000","last_clean":"2023-06-10T18:56:20.518951+0000","last_became_active":"2023-06-10T18:56:19.254487+0000","last_became_peered":"2023-06-10T18:56:19.254487+0000","last_unstale":"2023-06-10T18:56:20.518951+0000","last_undegraded":"2023-06-10T18:56:20.518951+0000","last_fullsized":"2023-06-10T18:56:20.518951+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:18.826560+0000","last_clean_scrub_stamp":"2023-06-10T18:56:18.826560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:41:29.585917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.246895+0000","last_change":"2023-06-10T18:56:18.852946+0000","last_active":"2023-06-10T18:56:20.246895+0000","last_peered":"2023-06-10T18:56:20.246895+0000","last_clean":"2023-06-10T18:56:20.246895+0000","last_became_active":"2023-06-10T18:56:18.852523+0000","last_became_peered":"2023-06-10T18:56:18.852523+0000","last_unstale":"2023-06-10T18:56:20.246895+0000","last_undegraded":"2023-06-10T18:56:20.246895+0000","last_fullsized":"2023-06-10T18:56:20.246895+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:56:17.819779+0000","last_clean_scrub_stamp":"2023-06-10T18:56:17.819779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:40:04.391011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":198,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-06-10T18:56:20.519635+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:56:20.519635+0000","last_peered":"2023-06-10T18:56:20.519635+0000","last_clean":"2023-06-10T18:56:20.519635+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:56:20.519635+0000","last_undegraded":"2023-06-10T18:56:20.519635+0000","last_fullsized":"2023-06-10T18:56:20.519635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:30:10.044276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575001,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29112,"kb_used_data":2160,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342728,"statfs":{"total":96636764160,"available":96606953472,"internally_reserved":0,"allocated":2211840,"data_stored":1368719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.442,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.74299999999999999,"15min":0.79000000000000004},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.72099999999999997,"5min":0.78100000000000003,"15min":0.91700000000000004},"last":0.505}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.89700000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70999999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73699999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.76300000000000001,"15min":0.81100000000000005},"last":0.60299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575002,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.51600000000000001},"min":{"1min":0.36099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70799999999999996,"5min":0.77100000000000002,"15min":0.81100000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.78700000000000003},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.51500000000000001},"min":{"1min":0.42599999999999999,"5min":0.316,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.65600000000000003},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.61499999999999999},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66900000000000004,"5min":0.872,"15min":6.6689999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.252},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.81100000000000005,"15min":0.82899999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53300000000000003,"15min":0.54600000000000004},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67900000000000005,"5min":0.77100000000000002,"15min":0.77500000000000002},"last":0.53700000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.498,"15min":0.498},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":0.874,"15min":0.874},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.27600000000000002},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.89600000000000002},"last":0.36599999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.68300000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.46600000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575001,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28504,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343336,"statfs":{"total":96636764160,"available":96607576064,"internally_reserved":0,"allocated":1589248,"data_stored":775130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.33000000000000002},"min":{"1min":0.20499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.498}]},{"osd":2,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54000000000000004,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.40000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59699999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61099999999999999,"15min":0.627},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.81999999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.93000000000000005},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60299999999999998,"15min":0.626},"min":{"1min":0.51500000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.78900000000000003,"5min":0.83899999999999997,"15min":1.008},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.206,"15min":1.206},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":0.873,"15min":0.88700000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.442,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":1.224,"15min":1.224},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59299999999999997,"15min":0.61599999999999999},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.56499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575001,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33180,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338660,"statfs":{"total":96636764160,"available":96602787840,"internally_reserved":0,"allocated":2183168,"data_stored":1365498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.46200000000000002,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34200000000000003,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.437,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48299999999999998,"5min":11.439,"15min":11.439},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.48099999999999998,"15min":0.42199999999999999},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.49099999999999999,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":5.766,"15min":5.766},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40500000000000003,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51400000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.75900000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":2.802,"15min":2.802},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81599999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80500000000000005,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79300000000000004,"5min":2.907,"15min":2.907},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.41299999999999998,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":1.363},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.79200000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.61299999999999999},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.38900000000000001},"max":{"1min":0.82999999999999996,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834575002,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34899999999999998,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41999999999999998,"15min":0.38500000000000001},"min":{"1min":0.27300000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57499999999999996,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52700000000000002,"5min":2.847,"15min":2.847},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.40200000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59499999999999997,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.41099999999999998},"min":{"1min":0.32000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56899999999999995,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.60699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":5.742,"15min":5.742},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.64300000000000002,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.62}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":6.593,"15min":6.593},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.77600000000000002,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.66400000000000003},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.82299999999999995,"5min":6.617,"15min":6.617},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.71899999999999997,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.73099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33000000000000002,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.45000000000000001,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.25},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54400000000000004,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45700000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57199999999999995,"5min":0.68600000000000005,"15min":1.3},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54000000000000004,"5min":0.82999999999999996,"15min":1.109},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.439},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.61299999999999999,"5min":1.5960000000000001,"15min":1.599},"last":0.308}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35899999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89700000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":1.613,"15min":1.613},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.60399999999999998,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":1.4410000000000001,"15min":3.802},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.65300000000000002},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.86499999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.84699999999999998,"5min":1.679,"15min":1.679},"last":0.70699999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.45200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":1.47,"15min":1.47},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":0.91100000000000003,"5min":1.548,"15min":1.8939999999999999},"last":0.67500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575001,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28508,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343332,"statfs":{"total":96636764160,"available":96607571968,"internally_reserved":0,"allocated":1593344,"data_stored":775200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.45000000000000001,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.745,"15min":1.119},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.503}]},{"osd":1,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.63,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.873,"15min":0.96599999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.64800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.746,"15min":0.79800000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.94599999999999995,"15min":1.069},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":0.63800000000000001,"5min":0.755,"15min":0.88300000000000001},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.54300000000000004,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28199999999999997},"max":{"1min":0.60199999999999998,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.70399999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.377}]},{"osd":7,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.68400000000000005,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.36699999999999999,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.48799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575002,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1601536,"data_stored":776261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66200000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55100000000000005,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32300000000000001},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.443}]},{"osd":1,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.69999999999999996,"5min":0.75800000000000001,"15min":0.78500000000000003},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38300000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.46100000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.56200000000000006},"min":{"1min":0.36199999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69899999999999995,"5min":0.76100000000000001,"15min":0.76800000000000002},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53300000000000003,"15min":0.504},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.73099999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.70199999999999996,"5min":0.75800000000000001,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.879},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Jun 10 18:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.754,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72999999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:56:23.718 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:56:23.719 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:56:23.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.873+0000 7eff19154700 1 Processor -- start 2023-06-10T18:56:23.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.873+0000 7eff19154700 1 -- start start 2023-06-10T18:56:23.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff19154700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff14157b30 0x7eff140ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff19154700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff140d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff19154700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff140ad3c0 con 0x7eff140d0830 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff19154700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff140cecf0 con 0x7eff14157b30 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff19154700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff14157f00 con 0x7eff140ce880 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff12d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff14157b30 0x7eff140ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff12d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff14157b30 0x7eff140ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37340/0 (socket says 172.21.15.104:37340) 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff12d9d700 1 -- 172.21.15.104:0/2093376417 learned_addr learned my addr 172.21.15.104:0/2093376417 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.874+0000 7eff1259c700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff140d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff12d9d700 1 -- 172.21.15.104:0/2093376417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 msgr2=0x7eff140d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff12d9d700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff140d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff12d9d700 1 -- 172.21.15.104:0/2093376417 >> v1:172.21.15.114:6789/0 conn(0x7eff140d0830 legacy=0x7eff1415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff12d9d700 1 -- 172.21.15.104:0/2093376417 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140b0170 con 0x7eff14157b30 2023-06-10T18:56:23.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff12d9d700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff14157b30 0x7eff140ce340 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7eff040098f0 tx=0x7eff04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c06de141da7fa6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff11d9b700 1 -- 172.21.15.104:0/2093376417 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04011830 con 0x7eff14157b30 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff11d9b700 1 -- 172.21.15.104:0/2093376417 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff04011990 con 0x7eff14157b30 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff11d9b700 1 -- 172.21.15.104:0/2093376417 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04011d20 con 0x7eff14157b30 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 -- 172.21.15.104:0/2093376417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff14157b30 msgr2=0x7eff140ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff14157b30 0x7eff140ce340 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7eff040098f0 tx=0x7eff04009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 -- 172.21.15.104:0/2093376417 shutdown_connections 2023-06-10T18:56:23.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff14157b30 0x7eff140ce340 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 --2- 172.21.15.104:0/2093376417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff140d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 -- 172.21.15.104:0/2093376417 >> 172.21.15.104:0/2093376417 conn(0x7eff14089fd0 msgr2=0x7eff1408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:23.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 -- 172.21.15.104:0/2093376417 shutdown_connections 2023-06-10T18:56:23.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.875+0000 7eff19154700 1 -- 172.21.15.104:0/2093376417 wait complete. 2023-06-10T18:56:23.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.877+0000 7eff19154700 1 Processor -- start 2023-06-10T18:56:23.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.877+0000 7eff19154700 1 -- start start 2023-06-10T18:56:23.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff14150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff14157b30 0x7eff140c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff140b1b60 con 0x7eff140d0830 2023-06-10T18:56:23.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff14151250 con 0x7eff14157b30 2023-06-10T18:56:23.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff140b0f20 con 0x7eff140ce880 2023-06-10T18:56:23.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37346/0 (socket says 172.21.15.104:37346) 2023-06-10T18:56:23.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 -- 172.21.15.104:0/3144548983 learned_addr learned my addr 172.21.15.104:0/3144548983 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:23.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff12d9d700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff14150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1359e700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff14157b30 0x7eff140c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 -- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 msgr2=0x7eff14150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff14150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 -- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff14157b30 msgr2=0x7eff140c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff14157b30 0x7eff140c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 -- 172.21.15.104:0/3144548983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04009550 con 0x7eff140d0830 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff1259c700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7eff0800e9b0 tx=0x7eff0800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08009710 con 0x7eff140d0830 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0800ee50 con 0x7eff140d0830 2023-06-10T18:56:23.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0800ca50 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.878+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff140b11c0 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.879+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff140b1780 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.880+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff14074050 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.880+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff0800cbb0 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.880+0000 7efefb7fe700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.880+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7eff08089550 con 0x7eff140d0830 2023-06-10T18:56:23.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.880+0000 7eff12d9d700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:23.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.881+0000 7eff12d9d700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7eff04014040 tx=0x7eff04008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:23.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:23.885+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff08057fe0 con 0x7eff140d0830 2023-06-10T18:56:24.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.149+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7eff140afad0 con 0x7eff140d0830 2023-06-10T18:56:24.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.153+0000 7efefb7fe700 1 -- 172.21.15.104:0/3144548983 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7eff0800ce60 con 0x7eff140d0830 2023-06-10T18:56:24.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 msgr2=0x7eff0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7eff04014040 tx=0x7eff04008030 comp rx=0 tx=0).stop 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 msgr2=0x7eff140b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7eff0800e9b0 tx=0x7eff0800ecc0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 shutdown_connections 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff14157b30 0x7eff140c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff140ce880 0x7eff14150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 --2- 172.21.15.104:0/3144548983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff140d0830 0x7eff140b1f80 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 >> 172.21.15.104:0/3144548983 conn(0x7eff14089fd0 msgr2=0x7eff14152f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 shutdown_connections 2023-06-10T18:56:24.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.155+0000 7eff19154700 1 -- 172.21.15.104:0/3144548983 wait complete. 2023-06-10T18:56:24.163 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:56:24.172 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:24.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.289+0000 7f834fbf6700 1 Processor -- start 2023-06-10T18:56:24.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.289+0000 7f834fbf6700 1 -- start start 2023-06-10T18:56:24.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.290+0000 7f834fbf6700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f834815c040 0x7f83480ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.290+0000 7f834fbf6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.290+0000 7f834fbf6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83480ccea0 con 0x7f83480ae280 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.290+0000 7f834fbf6700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83480ae6f0 con 0x7f834815c040 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.290+0000 7f834fbf6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83480ce2c0 con 0x7f8348071130 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834e193700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f834815c040 0x7f83480ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834e193700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:56740/0 (socket says 172.21.15.104:56740) 2023-06-10T18:56:24.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834e193700 1 -- 172.21.15.104:0/284985747 learned_addr learned my addr 172.21.15.104:0/284985747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 -- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 msgr2=0x7f83480cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 -- 172.21.15.104:0/284985747 >> v1:172.21.15.114:6789/0 conn(0x7f83480ae280 legacy=0x7f8348070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834c990700 1 -- 172.21.15.104:0/284985747 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223498549 0 0) 0x7f8344001110 con 0x7f83480ae280 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 -- 172.21.15.104:0/284985747 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83480cf750 con 0x7f834815c040 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834e193700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:24.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834d992700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480ce780 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8338009840 tx=0x7f8338009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55c6e88c14803b0d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834c990700 1 -- 172.21.15.104:0/284985747 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338011830 con 0x7f834815c040 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834c990700 1 -- 172.21.15.104:0/284985747 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338011990 con 0x7f834815c040 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.291+0000 7f834c990700 1 -- 172.21.15.104:0/284985747 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338011d20 con 0x7f834815c040 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 -- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 msgr2=0x7f83480ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480ce780 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8338009840 tx=0x7f8338009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 -- 172.21.15.104:0/284985747 shutdown_connections 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f83480cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 --2- 172.21.15.104:0/284985747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480ce780 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 -- 172.21.15.104:0/284985747 >> 172.21.15.104:0/284985747 conn(0x7f83480b1610 msgr2=0x7f83480ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 -- 172.21.15.104:0/284985747 shutdown_connections 2023-06-10T18:56:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.292+0000 7f834fbf6700 1 -- 172.21.15.104:0/284985747 wait complete. 2023-06-10T18:56:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.293+0000 7f834fbf6700 1 Processor -- start 2023-06-10T18:56:24.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 -- start start 2023-06-10T18:56:24.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f8348158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8348150af0 con 0x7f83480ae280 2023-06-10T18:56:24.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83480c93c0 con 0x7f834815c040 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.294+0000 7f834fbf6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83480ca3d0 con 0x7f8348071130 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37360/0 (socket says 172.21.15.104:37360) 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 -- 172.21.15.104:0/3493685319 learned_addr learned my addr 172.21.15.104:0/3493685319 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834e193700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d992700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f8348158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 -- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 msgr2=0x7f8348158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f8348158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 -- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 msgr2=0x7f83480caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 -- 172.21.15.104:0/3493685319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340009b30 con 0x7f83480ae280 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834d191700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f834400c750 tx=0x7f834400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440054e0 con 0x7f83480ae280 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8344003680 con 0x7f83480ae280 2023-06-10T18:56:24.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344003a30 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83380094a0 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.295+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83480c9610 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.296+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8348074050 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.297+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8344003b90 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.297+0000 7f833e7fc700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 0x7f832c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.297+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f83440887f0 con 0x7f83480ae280 2023-06-10T18:56:24.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.297+0000 7f834d992700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 0x7f832c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:24.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.298+0000 7f834d992700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 0x7f832c06d190 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f83380090b0 tx=0x7f8338008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:24.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.301+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8344057280 con 0x7f83480ae280 2023-06-10T18:56:24.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.529+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83480ccea0 con 0x7f83480ae280 2023-06-10T18:56:24.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.530+0000 7f833e7fc700 1 -- 172.21.15.104:0/3493685319 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f83480ccea0 con 0x7f83480ae280 2023-06-10T18:56:24.535 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:24.535 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 msgr2=0x7f832c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 0x7f832c06d190 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f83380090b0 tx=0x7f8338008030 comp rx=0 tx=0).stop 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 msgr2=0x7f83481586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f834400c750 tx=0x7f834400ca60 comp rx=0 tx=0).stop 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 shutdown_connections 2023-06-10T18:56:24.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f832c06ace0 0x7f832c06d190 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834815c040 0x7f83480caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8348071130 0x7f8348158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 --2- 172.21.15.104:0/3493685319 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83480ae280 0x7f83481586c0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 >> 172.21.15.104:0/3493685319 conn(0x7f83480b1610 msgr2=0x7f83480acae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 shutdown_connections 2023-06-10T18:56:24.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:24.532+0000 7f834fbf6700 1 -- 172.21.15.104:0/3493685319 wait complete. 2023-06-10T18:56:24.539 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:24.828 INFO:tasks.ceph.mds.d.smithi114.stderr:2023-06-10T18:56:24.822+0000 7f4a310a4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:56:24.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:56:19.230688+0000', 'modified': '2023-06-10T18:56:22.252369+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9231}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9231': {'gid': 9231, 'name': 'd', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6836/3783425308', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3783425308}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 3783425308}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:56:24.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:56:24.951 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:56:25.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.071+0000 7f0dd21c3700 1 Processor -- start 2023-06-10T18:56:25.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.071+0000 7f0dd21c3700 1 -- start start 2023-06-10T18:56:25.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dd21c3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0dcc070a20 0x7f0dcc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dd21c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070df0 0x7f0dcc1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dd21c3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc06ae00 con 0x7f0dcc158ce0 2023-06-10T18:56:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dd21c3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc069e70 con 0x7f0dcc070a20 2023-06-10T18:56:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dd21c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc1595f0 con 0x7f0dcc070df0 2023-06-10T18:56:25.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0dcc070a20 0x7f0dcc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0dcc070a20 0x7f0dcc06a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37366/0 (socket says 172.21.15.104:37366) 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/150585075 learned_addr learned my addr 172.21.15.104:0/150585075 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070df0 0x7f0dcc1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/150585075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070df0 msgr2=0x7f0dcc1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070df0 0x7f0dcc1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/150585075 >> v1:172.21.15.114:6789/0 conn(0x7f0dcc158ce0 legacy=0x7f0dcc159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.072+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/150585075 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc1598d0 con 0x7f0dcc070a20 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc070a20 0x7f0dcc06a8c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f0db400bb90 tx=0x7f0db400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dc421b3f8036f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dca7fc700 1 -- 172.21.15.104:0/150585075 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400dd50 con 0x7f0dcc070a20 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dca7fc700 1 -- 172.21.15.104:0/150585075 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4010470 con 0x7f0dcc070a20 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dca7fc700 1 -- 172.21.15.104:0/150585075 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4010800 con 0x7f0dcc070a20 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 -- 172.21.15.104:0/150585075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc070a20 msgr2=0x7f0dcc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc070a20 0x7f0dcc06a8c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f0db400bb90 tx=0x7f0db400bea0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 -- 172.21.15.104:0/150585075 shutdown_connections 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc070a20 0x7f0dcc06a8c0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/150585075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070df0 0x7f0dcc1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 -- 172.21.15.104:0/150585075 >> 172.21.15.104:0/150585075 conn(0x7f0dcc0b1610 msgr2=0x7f0dcc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 -- 172.21.15.104:0/150585075 shutdown_connections 2023-06-10T18:56:25.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.073+0000 7f0dd21c3700 1 -- 172.21.15.104:0/150585075 wait complete. 2023-06-10T18:56:25.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.074+0000 7f0dd21c3700 1 Processor -- start 2023-06-10T18:56:25.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 -- start start 2023-06-10T18:56:25.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070a20 0x7f0dcc0cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dcc070df0 0x7f0dcc0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc13f500 con 0x7f0dcc158ce0 2023-06-10T18:56:25.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0cb5a0 con 0x7f0dcc070df0 2023-06-10T18:56:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dd21c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0b4170 con 0x7f0dcc070a20 2023-06-10T18:56:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.075+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37376/0 (socket says 172.21.15.104:37376) 2023-06-10T18:56:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 -- 172.21.15.104:0/3391963983 learned_addr learned my addr 172.21.15.104:0/3391963983 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dcc070df0 0x7f0dcc0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070a20 0x7f0dcc0cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 -- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070a20 msgr2=0x7f0dcc0cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070a20 0x7f0dcc0cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 -- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dcc070df0 msgr2=0x7f0dcc0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dcc070df0 0x7f0dcc0ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 -- 172.21.15.104:0/3391963983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db400b7f0 con 0x7f0dcc158ce0 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400cb50 tx=0x7f0dc400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dc3fff700 1 -- 172.21.15.104:0/3391963983 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4003c00 con 0x7f0dcc158ce0 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dc3fff700 1 -- 172.21.15.104:0/3391963983 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc4003d60 con 0x7f0dcc158ce0 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc0b4450 con 0x7f0dcc158ce0 2023-06-10T18:56:25.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dc3fff700 1 -- 172.21.15.104:0/3391963983 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400aa20 con 0x7f0dcc158ce0 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.076+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc0b4a10 con 0x7f0dcc158ce0 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.077+0000 7f0dc3fff700 1 -- 172.21.15.104:0/3391963983 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0dc400ab80 con 0x7f0dcc158ce0 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.078+0000 7f0dc3fff700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 0x7f0dac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.078+0000 7f0dc3fff700 1 -- 172.21.15.104:0/3391963983 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0dc40893c0 con 0x7f0dcc158ce0 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.078+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 0x7f0dac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.079+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 0x7f0dac06d190 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0db400bee0 tx=0x7f0db4013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.080+0000 7f0dde348b80 1 Processor -- start 2023-06-10T18:56:25.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.080+0000 7f0dde348b80 1 -- start start 2023-06-10T18:56:25.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 0x55a6515dd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6514b7c60 con 0x55a6515b0740 2023-06-10T18:56:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a65150a9f0 con 0x55a6515d9f00 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a651503e80 con 0x55a6515b3270 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37384/0 (socket says 172.21.15.104:37384) 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 -- 172.21.15.104:0/4119469877 learned_addr learned my addr 172.21.15.104:0/4119469877 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 0x55a6515dd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 -- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 msgr2=0x55a6515b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 -- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 msgr2=0x55a6515dd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 0x55a6515dd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcaffd700 1 -- 172.21.15.104:0/4119469877 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6513855e0 con 0x55a6515b0740 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.081+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 0x55a6515dd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0099d0 tx=0x7f0dbc009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0111fd808416dd1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dc17fa700 1 -- 172.21.15.104:0/4119469877 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00c070 con 0x55a6515b0740 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dc17fa700 1 -- 172.21.15.104:0/4119469877 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc004230 con 0x55a6515b0740 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dc17fa700 1 -- 172.21.15.104:0/4119469877 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc011750 con 0x55a6515b0740 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 -- 172.21.15.104:0/4119469877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 msgr2=0x55a6515d99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0099d0 tx=0x7f0dbc009ce0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 -- 172.21.15.104:0/4119469877 shutdown_connections 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515d9f00 0x55a6515dd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 --2- 172.21.15.104:0/4119469877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515d99c0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 -- 172.21.15.104:0/4119469877 >> 172.21.15.104:0/4119469877 conn(0x55a6515a7fc0 msgr2=0x55a6515f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 -- 172.21.15.104:0/4119469877 shutdown_connections 2023-06-10T18:56:25.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.082+0000 7f0dde348b80 1 -- 172.21.15.104:0/4119469877 wait complete. 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.083+0000 7f0dde348b80 1 -- start start 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.083+0000 7f0dde348b80 1 Processor -- start 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515def60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6514a3a70 con 0x55a6515d9f00 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6514a3770 con 0x55a6515b0740 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6513878d0 con 0x55a6515b3270 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dde348b80 10 client.0 _ll_register_callbacks cb 0x55a6514a9210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37388/0 (socket says 172.21.15.104:37388) 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 -- 172.21.15.104:0/2993767538 learned_addr learned my addr 172.21.15.104:0/2993767538 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515def60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 -- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 msgr2=0x55a6515b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 -- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 msgr2=0x55a6515def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515def60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 -- 172.21.15.104:0/2993767538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc00e040 con 0x55a6515d9f00 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4057ce0 tx=0x7f0dc40603e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c0ce6ac7603c33 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0daa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.084+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc406bde0 con 0x55a6515d9f00 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0dc40574d0 con 0x55a6515d9f00 2023-06-10T18:56:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6515a5780 con 0x55a6515d9f00 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dc4057690 con 0x55a6515d9f00 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 10 client.9366 fetch_fsmap learned FSMap version 80 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x55a651331d90 con 0x55a6515d9f00 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a6515a5780 con 0x55a6515d9f00 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 80) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0dc405fea0 con 0x55a6515d9f00 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 10 client.9366 fetch_fsmap finished waiting for FSMap version 80 2023-06-10T18:56:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 10 client.9366 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 20 client.9366 populate_metadata read hostname 'smithi104' 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a6515e0900 0x55a65160c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 4 client.9366 mds_command: new command op to 9231 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 --> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6514fce60 con 0x55a6515e0900 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.085+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a6515e0900 0x55a65160c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0daa7fc700 10 client.9366 ms_handle_connect on v2:172.21.15.114:6834/3783425308 2023-06-10T18:56:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a6515e0900 0x55a65160c300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c16672d3546280dc server_cookie=c67ec20abdf2d2a1 in_seq=0 out_seq=0 2023-06-10T18:56:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2993767538 <== mds.0 v2:172.21.15.114:6834/3783425308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a6514fce60 con 0x55a6515e0900 2023-06-10T18:56:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0daa7fc700 10 client.9366 handle_command_reply: tid=0 2023-06-10T18:56:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 client.9366 shutdown 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 msgr2=0x55a6515d7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4057ce0 tx=0x7f0dc40603e0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 shutdown_connections 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a6515e0900 0x55a65160c300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515def60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0daa7fc700 0 client.9366 ms_handle_reset on v2:172.21.15.114:6834/3783425308 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515d9f00 0x55a6515d7050 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2993767538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515b3270 0x55a6515b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 >> 172.21.15.104:0/2993767538 conn(0x55a6515a7fc0 msgr2=0x55a6515d9790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 shutdown_connections 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.086+0000 7f0dde348b80 1 -- 172.21.15.104:0/2993767538 wait complete. 2023-06-10T18:56:25.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.087+0000 7f0dde348b80 20 client.9366 trim_cache size 0 max 16384 2023-06-10T18:56:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.104+0000 7f0dde348b80 1 Processor -- start 2023-06-10T18:56:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.104+0000 7f0dde348b80 1 -- start start 2023-06-10T18:56:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b3270 0x55a6515fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a6515fc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6515b46f0 con 0x55a6515b0740 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6515b2c30 con 0x55a6515b3270 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6515fb0a0 con 0x55a6515d9f00 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37398/0 (socket says 172.21.15.104:37398) 2023-06-10T18:56:25.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/3690769382 learned_addr learned my addr 172.21.15.104:0/3690769382 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b3270 0x55a6515fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a6515fc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 msgr2=0x55a6515fc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a6515fc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b3270 msgr2=0x55a6515fbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b3270 0x55a6515fbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 -- 172.21.15.104:0/3690769382 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a65131ac10 con 0x55a6515b0740 2023-06-10T18:56:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06c890 tx=0x7f0db4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c326fd9d5d237c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dc1ffb700 1 -- 172.21.15.104:0/3690769382 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db402c070 con 0x55a6515b0740 2023-06-10T18:56:25.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dc1ffb700 1 -- 172.21.15.104:0/3690769382 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db402d040 con 0x55a6515b0740 2023-06-10T18:56:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.105+0000 7f0dc1ffb700 1 -- 172.21.15.104:0/3690769382 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4033dd0 con 0x55a6515b0740 2023-06-10T18:56:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 -- 172.21.15.104:0/3690769382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 msgr2=0x55a6515f8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc06c890 tx=0x7f0db4014000 comp rx=0 tx=0).stop 2023-06-10T18:56:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 -- 172.21.15.104:0/3690769382 shutdown_connections 2023-06-10T18:56:25.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a6515fc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b0740 0x55a6515f8660 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 --2- 172.21.15.104:0/3690769382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b3270 0x55a6515fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 -- 172.21.15.104:0/3690769382 >> 172.21.15.104:0/3690769382 conn(0x55a6515a7fc0 msgr2=0x55a6515a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 -- 172.21.15.104:0/3690769382 shutdown_connections 2023-06-10T18:56:25.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.106+0000 7f0dde348b80 1 -- 172.21.15.104:0/3690769382 wait complete. 2023-06-10T18:56:25.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 -- start start 2023-06-10T18:56:25.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 Processor -- start 2023-06-10T18:56:25.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a65160e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6515f9c80 con 0x55a6515b3270 2023-06-10T18:56:25.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a651615240 con 0x55a6515b0740 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.107+0000 7f0dde348b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a651662c90 con 0x55a6515d9f00 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dde348b80 10 client.0 _ll_register_callbacks cb 0x55a6514a9210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37402/0 (socket says 172.21.15.104:37402) 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 -- 172.21.15.104:0/2094349246 learned_addr learned my addr 172.21.15.104:0/2094349246 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcbfff700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a65160e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 -- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 msgr2=0x55a65160e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a65160e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 -- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 msgr2=0x55a6515fc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515fc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 -- 172.21.15.104:0/2094349246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc1598d0 con 0x55a6515b3270 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dcaffd700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009d20 tx=0x7f0dbc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b68d38670b1feb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0daa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc011c90 con 0x55a6515b3270 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0dbc011df0 con 0x55a6515b3270 2023-06-10T18:56:25.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0dbc004230 con 0x55a6515b3270 2023-06-10T18:56:25.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dbc0036a0 con 0x55a6515b3270 2023-06-10T18:56:25.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dde348b80 10 client.9372 fetch_fsmap learned FSMap version 80 2023-06-10T18:56:25.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x55a651358200 con 0x55a6515b3270 2023-06-10T18:56:25.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.108+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc004230 con 0x55a6515b3270 2023-06-10T18:56:25.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 80) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0dbc003800 con 0x55a6515b3270 2023-06-10T18:56:25.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 10 client.9372 fetch_fsmap finished waiting for FSMap version 80 2023-06-10T18:56:25.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 10 client.9372 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:56:25.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 20 client.9372 populate_metadata read hostname 'smithi104' 2023-06-10T18:56:25.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a651663370 0x55a651663740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 4 client.9372 mds_command: new command op to 9231 tid=0[{"prefix": "status"}] 2023-06-10T18:56:25.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 --> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6514bf000 con 0x55a651663370 2023-06-10T18:56:25.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a651663370 0x55a651663740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0daa7fc700 10 client.9372 ms_handle_connect on v2:172.21.15.114:6834/3783425308 2023-06-10T18:56:25.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dcb7fe700 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a651663370 0x55a651663740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7ff39acca6d4cb server_cookie=8ae54feffe334564 in_seq=0 out_seq=0 2023-06-10T18:56:25.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0daa7fc700 1 -- 172.21.15.104:0/2094349246 <== mds.0 v2:172.21.15.114:6834/3783425308 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55a6514bf000 con 0x55a651663370 2023-06-10T18:56:25.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0daa7fc700 10 client.9372 handle_command_reply: tid=0 2023-06-10T18:56:25.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 client.9372 shutdown 2023-06-10T18:56:25.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 msgr2=0x55a651662750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009d20 tx=0x7f0dbc0047c0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 shutdown_connections 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:6834/3783425308,v1:172.21.15.114:6836/3783425308] conn(0x55a651663370 0x55a651663740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a6515b0740 0x55a6515fc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0daa7fc700 0 client.9372 ms_handle_reset on v2:172.21.15.114:6834/3783425308 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55a6515d9f00 0x55a65160e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 --2- 172.21.15.104:0/2094349246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a6515b3270 0x55a651662750 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.109+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 >> 172.21.15.104:0/2094349246 conn(0x55a6515a7fc0 msgr2=0x55a6515fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 shutdown_connections 2023-06-10T18:56:25.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dde348b80 1 -- 172.21.15.104:0/2094349246 wait complete. 2023-06-10T18:56:25.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dde348b80 20 client.9372 trim_cache size 0 max 16384 2023-06-10T18:56:25.133 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:56:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:56:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:56:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 9231, 2023-06-10T18:56:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:56:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:56:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:56:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8773301440000001, 2023-06-10T18:56:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 80, 2023-06-10T18:56:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 128, 2023-06-10T18:56:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 128, 2023-06-10T18:56:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.203461663000001 2023-06-10T18:56:25.135 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:56:25.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 msgr2=0x7f0dac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 0x7f0dac06d190 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0db400bee0 tx=0x7f0db4013000 comp rx=0 tx=0).stop 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 msgr2=0x7f0dcc13e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400cb50 tx=0x7f0dc400ce60 comp rx=0 tx=0).stop 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 shutdown_connections 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dcc158ce0 0x7f0dcc13e840 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dcc070a20 0x7f0dcc0cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dac06ace0 0x7f0dac06d190 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 --2- 172.21.15.104:0/3391963983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dcc070df0 0x7f0dcc0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 >> 172.21.15.104:0/3391963983 conn(0x7f0dcc0b1610 msgr2=0x7f0dcc0afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 shutdown_connections 2023-06-10T18:56:25.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.110+0000 7f0dd21c3700 1 -- 172.21.15.104:0/3391963983 wait complete. 2023-06-10T18:56:25.138 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:56:25.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.253+0000 7f30604fa700 1 Processor -- start 2023-06-10T18:56:25.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.253+0000 7f30604fa700 1 -- start start 2023-06-10T18:56:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f30604fa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f30580cb300 0x7f3058163c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f30604fa700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580c9e60 con 0x7f30580cd850 2023-06-10T18:56:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f30604fa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580aee70 con 0x7f30580aeaa0 2023-06-10T18:56:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f30604fa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580c9bc0 con 0x7f30580cb300 2023-06-10T18:56:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f305da95700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f30580cd850 0x7f305806cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35236/0 (socket says 172.21.15.104:35236) 2023-06-10T18:56:25.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f305da95700 1 -- 172.21.15.104:0/3705640475 learned_addr learned my addr 172.21.15.104:0/3705640475 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:25.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f305ea97700 1 --2- 172.21.15.104:0/3705640475 >> v2:172.21.15.114:3300/0 conn(0x7f30580cb300 0x7f3058163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469605810 0 0) 0x7f30580c9e60 con 0x7f30580cd850 2023-06-10T18:56:25.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.254+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30440035b0 con 0x7f30580cd850 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1786390036 0 0) 0x7f30440035b0 con 0x7f30580cd850 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 >> v2:172.21.15.114:3300/0 conn(0x7f30580cb300 msgr2=0x7f3058163c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 --2- 172.21.15.104:0/3705640475 >> v2:172.21.15.114:3300/0 conn(0x7f30580cb300 0x7f3058163c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 >> v1:172.21.15.114:6789/0 conn(0x7f30580aeaa0 legacy=0x7f30580cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305806e540 con 0x7f30580cd850 2023-06-10T18:56:25.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3054001670 con 0x7f30580cd850 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 >> v1:172.21.15.104:6789/0 conn(0x7f30580cd850 legacy=0x7f305806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 0x7f3044003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30440043f0 0x7f30440048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30440050b0 0x7f304400dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580c9bc0 con 0x7f3044003a00 2023-06-10T18:56:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304400e210 con 0x7f30440050b0 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304400e3d0 con 0x7f30440043f0 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 0x7f3044003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305ea97700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30440050b0 0x7f304400dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305da95700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30440043f0 0x7f30440048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 -- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30440043f0 msgr2=0x7f30440048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30440043f0 0x7f30440048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 -- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30440050b0 msgr2=0x7f304400dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30440050b0 0x7f304400dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 -- 172.21.15.104:0/3705640475 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30440035b0 con 0x7f3044003a00 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305ea97700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30440050b0 0x7f304400dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305ea97700 1 -- 172.21.15.104:0/3705640475 reap_dead start 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305da95700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30440043f0 0x7f30440048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305e296700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 0x7f3044003dd0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f30480019e0 tx=0x7f3044005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24b9fd6fcaaa159 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480052d0 con 0x7f3044003a00 2023-06-10T18:56:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3048005430 con 0x7f3044003a00 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.255+0000 7f305d294700 1 -- 172.21.15.104:0/3705640475 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048003910 con 0x7f3044003a00 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 -- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 msgr2=0x7f3044003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 0x7f3044003dd0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f30480019e0 tx=0x7f3044005880 comp rx=0 tx=0).stop 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 -- 172.21.15.104:0/3705640475 shutdown_connections 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 --2- 172.21.15.104:0/3705640475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3044003a00 0x7f3044003dd0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 -- 172.21.15.104:0/3705640475 >> 172.21.15.104:0/3705640475 conn(0x7f305806b8a0 msgr2=0x7f30580cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 -- 172.21.15.104:0/3705640475 shutdown_connections 2023-06-10T18:56:25.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.256+0000 7f30604fa700 1 -- 172.21.15.104:0/3705640475 wait complete. 2023-06-10T18:56:25.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.258+0000 7f30604fa700 1 Processor -- start 2023-06-10T18:56:25.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.258+0000 7f30604fa700 1 -- start start 2023-06-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3044003a00 0x7f30580c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30581f0a10 0x7f30581f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581f13c0 con 0x7f3058157230 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b4a00 con 0x7f30581f0a10 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b4ce0 con 0x7f3044003a00 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37424/0 (socket says 172.21.15.104:37424) 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 -- 172.21.15.104:0/1684733152 learned_addr learned my addr 172.21.15.104:0/1684733152 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305e296700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3044003a00 0x7f30580c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305da95700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30581f0a10 0x7f30581f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 -- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3044003a00 msgr2=0x7f30580c7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3044003a00 0x7f30580c7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 -- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30581f0a10 msgr2=0x7f30581f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30581f0a10 0x7f30581f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 -- 172.21.15.104:0/1684733152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305815a170 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f305ea97700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f30580c9b50 tx=0x7f30500101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305001b070 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000f040 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305815a450 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000e050 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.259+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581ed290 con 0x7f3058157230 2023-06-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.260+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3058074050 con 0x7f3058157230 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.261+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3050010a60 con 0x7f3058157230 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.261+0000 7f304effd700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 0x7f303c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.261+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f30500896f0 con 0x7f3058157230 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.261+0000 7f305e296700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 0x7f303c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.262+0000 7f305e296700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 0x7f303c06d190 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3048015040 tx=0x7f304800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.265+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3050054660 con 0x7f3058157230 2023-06-10T18:56:25.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.495+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3058159030 con 0x7f3058157230 2023-06-10T18:56:25.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.495+0000 7f304effd700 1 -- 172.21.15.104:0/1684733152 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3050057c80 con 0x7f3058157230 2023-06-10T18:56:25.501 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:56:25.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 msgr2=0x7f303c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 0x7f303c06d190 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3048015040 tx=0x7f304800a000 comp rx=0 tx=0).stop 2023-06-10T18:56:25.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 msgr2=0x7f30581576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:25.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f30580c9b50 tx=0x7f30500101c0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 shutdown_connections 2023-06-10T18:56:25.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3058157230 0x7f30581576a0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3044003a00 0x7f30580c7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f303c06ace0 0x7f303c06d190 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 --2- 172.21.15.104:0/1684733152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30581f0a10 0x7f30581f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:25.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 >> 172.21.15.104:0/1684733152 conn(0x7f305806b8a0 msgr2=0x7f30580b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:25.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 shutdown_connections 2023-06-10T18:56:25.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:25.497+0000 7f30604fa700 1 -- 172.21.15.104:0/1684733152 wait complete. 2023-06-10T18:56:25.514 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:56:25.514 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:56:25.515 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:56:25.515 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:56:25.515 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:25.515 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:56:25.516 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:56:25.516 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:56:25.516 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:56:25.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.518 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:56:25.534 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:25.535 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:25.536 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41699sec preferred_lft 41699sec 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.537 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:56:25.538 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:56:25.539 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:56:25.539 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:25.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.539 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:56:25.599 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:56:25.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.600 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:56:25.659 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:56:25.659 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:56:25.659 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:56:25.660 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:56:25.660 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:56:25.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.660 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:56:25.661 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:56:25.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:56:25.661 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:56:25.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:56:25.662 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:56:25.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.763 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:56:25.780 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:56:25.780 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:56:25.780 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:56:25.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:25.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:56:25.846 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:56:25.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:56:25.847 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:25.902 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:56:25.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:56:25.904 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:25.959 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:56:26.025 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:56:26.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:56:26.091 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:56:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:56:26.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:26.094 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:56:26.150 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:56:26.151 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:56:26.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:56:26.218 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:56:26.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:56:26.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:56:26.273+0000 7ff819c7f5c0 -1 init, newargv = 0x563c942a4c20 newargc=15 2023-06-10T18:56:26.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[148724]: starting ceph client 2023-06-10T18:56:26.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[148724]: starting fuse 2023-06-10T18:56:26.287 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:56:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:56:26.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:26.289 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:56:26.345 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:56:26.345 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:56:26.346 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:56:26.346 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:56:26.347 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:56:26.347 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:56:26.347 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:56:26.347 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:26.348 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:56:26.348 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:56:26.348 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:56:26.349 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:56:26.349 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:26.349 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:56:26.350 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:56:26.350 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:56:26.350 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:56:26.350 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:26.351 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:56:26.351 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:56:26.351 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:56:26.352 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:56:26.352 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:56:26.352 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:56:26.353 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:56:26.353 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:56:26.353 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:26.353 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:56:26.354 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:56:26.467 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.148724.asok 2023-06-10T18:56:26.649 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.148724.asok 2023-06-10T18:56:26.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:26.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.148724.asok status 2023-06-10T18:56:26.805 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:56:26.805 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "148724", 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:56:26.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 9387, 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:56:26.807 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 9387 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3796851905 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:26.808 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3796851905 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.9387 192.168.0.1:0/3796851905", 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3796851905", 2023-06-10T18:56:26.809 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 80, 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 128, 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:56:26.810 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:56:26.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:26.815 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:56:26.815 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:26.832 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:56:26.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:26.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:26.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:26.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:56:27.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.056+0000 7fb2a6139700 1 Processor -- start 2023-06-10T18:56:27.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.056+0000 7fb2a6139700 1 -- start start 2023-06-10T18:56:27.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a015c730 0x7fb2a006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2a00cc000 0x7fb2a00cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb2a006cd70 0x7fb2a006d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a00aed80 con 0x7fb2a00cc000 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a00c9c60 con 0x7fb2a006cd70 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb2a6139700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00cc9b0 con 0x7fb2a015c730 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2a00cc000 0x7fb2a00cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb2a00cc000 0x7fb2a00cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37444/0 (socket says 172.21.15.104:37444) 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 -- 172.21.15.104:0/3082555250 learned_addr learned my addr 172.21.15.104:0/3082555250 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:27.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29ffff700 1 --2- 172.21.15.104:0/3082555250 >> v2:172.21.15.114:3300/0 conn(0x7fb2a006cd70 0x7fb2a006d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 -- 172.21.15.104:0/3082555250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a015c730 msgr2=0x7fb2a006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29f7fe700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a015c730 0x7fb2a006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a015c730 0x7fb2a006e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 -- 172.21.15.104:0/3082555250 >> v2:172.21.15.114:3300/0 conn(0x7fb2a006cd70 msgr2=0x7fb2a006d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 --2- 172.21.15.104:0/3082555250 >> v2:172.21.15.114:3300/0 conn(0x7fb2a006cd70 0x7fb2a006d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 -- 172.21.15.104:0/3082555250 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a015cb00 con 0x7fb2a00cc000 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.057+0000 7fb29effd700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a00cc000 0x7fb2a00cc470 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00709d0 tx=0x7fb29000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87ed3ef7b220be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:27.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb29e7fc700 1 -- 172.21.15.104:0/3082555250 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29000dd30 con 0x7fb2a00cc000 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb29e7fc700 1 -- 172.21.15.104:0/3082555250 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29000de90 con 0x7fb2a00cc000 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb29e7fc700 1 -- 172.21.15.104:0/3082555250 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290010690 con 0x7fb2a00cc000 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 -- 172.21.15.104:0/3082555250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a00cc000 msgr2=0x7fb2a00cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a00cc000 0x7fb2a00cc470 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00709d0 tx=0x7fb29000be70 comp rx=0 tx=0).stop 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 -- 172.21.15.104:0/3082555250 shutdown_connections 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3082555250 >> v2:172.21.15.114:3300/0 conn(0x7fb2a006cd70 0x7fb2a006d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a015c730 0x7fb2a006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3082555250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a00cc000 0x7fb2a00cc470 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 -- 172.21.15.104:0/3082555250 >> 172.21.15.104:0/3082555250 conn(0x7fb2a006b8a0 msgr2=0x7fb2a00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:27.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 -- 172.21.15.104:0/3082555250 shutdown_connections 2023-06-10T18:56:27.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.058+0000 7fb2a6139700 1 -- 172.21.15.104:0/3082555250 wait complete. 2023-06-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.059+0000 7fb2a6139700 1 Processor -- start 2023-06-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 -- start start 2023-06-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 0x7fb2a00aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 0x7fb2a0156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0157460 con 0x7fb2a015c730 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0157740 con 0x7fb2a00cc000 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb2a6139700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00b14a0 con 0x7fb2a006cd70 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb29ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb29ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37452/0 (socket says 172.21.15.104:37452) 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.060+0000 7fb29ffff700 1 -- 172.21.15.104:0/3850389512 learned_addr learned my addr 172.21.15.104:0/3850389512 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:27.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29f7fe700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 0x7fb2a00aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29effd700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 0x7fb2a0156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 -- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 msgr2=0x7fb2a00aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 0x7fb2a00aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 -- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 msgr2=0x7fb2a0156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 0x7fb2a0156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 -- 172.21.15.104:0/3850389512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29000b7f0 con 0x7fb2a015c730 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29effd700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 0x7fb2a0156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:27.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29f7fe700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 0x7fb2a00aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb29ffff700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fb29800cd20 tx=0x7fb29800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298003a20 con 0x7fb2a015c730 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb298003b80 con 0x7fb2a015c730 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a00b1780 con 0x7fb2a015c730 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298004020 con 0x7fb2a015c730 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.061+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a00c73d0 con 0x7fb2a015c730 2023-06-10T18:56:27.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.062+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a0074050 con 0x7fb2a015c730 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.062+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb298004180 con 0x7fb2a015c730 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.063+0000 7fb297fff700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 0x7fb28006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.063+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb298012070 con 0x7fb2a015c730 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.063+0000 7fb29f7fe700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 0x7fb28006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.064+0000 7fb29f7fe700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 0x7fb28006d190 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb28800aa10 tx=0x7fb288008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:27.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.067+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb298054d10 con 0x7fb2a015c730 2023-06-10T18:56:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.298+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb2a0157880 con 0x7fb2a015c730 2023-06-10T18:56:27.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.298+0000 7fb297fff700 1 -- 172.21.15.104:0/3850389512 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb298058330 con 0x7fb2a015c730 2023-06-10T18:56:27.303 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:56:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 msgr2=0x7fb28006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 0x7fb28006d190 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb28800aa10 tx=0x7fb288008040 comp rx=0 tx=0).stop 2023-06-10T18:56:27.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 msgr2=0x7fb2a00c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fb29800cd20 tx=0x7fb29800a570 comp rx=0 tx=0).stop 2023-06-10T18:56:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 shutdown_connections 2023-06-10T18:56:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2a015c730 0x7fb2a00c66e0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2a006cd70 0x7fb2a00aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb28006ace0 0x7fb28006d190 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 --2- 172.21.15.104:0/3850389512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2a00cc000 0x7fb2a0156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 >> 172.21.15.104:0/3850389512 conn(0x7fb2a006b8a0 msgr2=0x7fb2a0072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 shutdown_connections 2023-06-10T18:56:27.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:27.300+0000 7fb2a6139700 1 -- 172.21.15.104:0/3850389512 wait complete. 2023-06-10T18:56:27.318 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:56:27.319 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:56:27.320 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:56:27.320 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:56:27.321 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:27.321 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:56:27.321 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:56:27.322 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:56:27.322 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:56:27.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.323 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:56:27.345 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:56:27.345 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:56:27.345 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:56:27.345 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.346 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:56:27.346 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.346 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:27.346 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:27.347 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:27.347 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:27.347 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:56:27.347 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:27.347 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41693sec preferred_lft 41693sec 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.348 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:56:27.349 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:56:27.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.351 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:56:27.415 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:56:27.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.417 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:56:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:56:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:56:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:56:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:56:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:56:27.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.479 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:56:27.479 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:56:27.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:56:27.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:56:27.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:56:27.480 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:56:27.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.590 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:56:27.608 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:56:27.608 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:56:27.608 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:56:27.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:56:27.676 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:56:27.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:56:27.678 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:27.734 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:56:27.734 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:56:27.734 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:27.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:56:27.858 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:56:27.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:56:27.923 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:56:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:56:27.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:27.925 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:56:27.981 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:56:27.982 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:56:28.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:56:28.049 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:56:28.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:56:28.108 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:56:28.103+0000 7f9be13d35c0 -1 init, newargv = 0x55d848963a80 newargc=15 2023-06-10T18:56:28.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114930]: starting ceph client 2023-06-10T18:56:28.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114930]: starting fuse 2023-06-10T18:56:28.117 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:56:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:56:28.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:28.119 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:56:28.175 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:56:28.176 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:56:28.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:56:28.176 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:56:28.177 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:56:28.177 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:56:28.177 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:56:28.178 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:56:28.178 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:56:28.178 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:56:28.179 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:56:28.179 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:56:28.179 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:56:28.179 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:56:28.180 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:56:28.181 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:56:28.181 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:56:28.182 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:56:28.182 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:56:28.182 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:56:28.182 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:56:28.183 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:56:28.183 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:56:28.183 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:56:28.184 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:56:28.184 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:56:28.184 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:56:28.184 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:56:28.185 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:56:28.297 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.114930.asok 2023-06-10T18:56:28.479 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.114930.asok 2023-06-10T18:56:28.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:28.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.114930.asok status 2023-06-10T18:56:28.635 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:56:28.635 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:56:28.635 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:56:28.635 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:56:28.635 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "114930", 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:56:28.636 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 8080, 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 8080 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:56:28.637 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 456025825 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:56:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 456025825 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.8080 192.168.0.1:0/456025825", 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/456025825", 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:56:28.639 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 80, 2023-06-10T18:56:28.640 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 128, 2023-06-10T18:56:28.640 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:56:28.640 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:56:28.640 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:56:28.640 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:56:28.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:28.645 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:56:28.645 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:28.664 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:56:28.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:28.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:28.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:28.730 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:56:28.877 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:56:28.877 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:56:28.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:56:29.011 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:56:29.012 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:56:29.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:29.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.127+0000 7f15c7d3a700 1 Processor -- start 2023-06-10T18:56:29.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.127+0000 7f15c7d3a700 1 -- start start 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c7d3a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15c006e4b0 0x7f15c006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c7d3a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15c0069e70 0x7f15c00ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c7d3a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c006d290 con 0x7f15c006e0e0 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c7d3a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c0071640 con 0x7f15c006e4b0 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c7d3a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c00aef00 con 0x7f15c0069e70 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15c006e4b0 0x7f15c006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15c006e4b0 0x7f15c006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37458/0 (socket says 172.21.15.104:37458) 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 -- 172.21.15.104:0/135994445 learned_addr learned my addr 172.21.15.104:0/135994445 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c62d7700 1 --2- 172.21.15.104:0/135994445 >> v2:172.21.15.114:3300/0 conn(0x7f15c0069e70 0x7f15c00ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 -- 172.21.15.104:0/135994445 >> v2:172.21.15.114:3300/0 conn(0x7f15c0069e70 msgr2=0x7f15c00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 --2- 172.21.15.104:0/135994445 >> v2:172.21.15.114:3300/0 conn(0x7f15c0069e70 0x7f15c00ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 -- 172.21.15.104:0/135994445 >> v1:172.21.15.114:6789/0 conn(0x7f15c006e0e0 legacy=0x7f15c00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.128+0000 7f15c52d5700 1 -- 172.21.15.104:0/135994445 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00ca2d0 con 0x7f15c006e4b0 2023-06-10T18:56:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c52d5700 1 --2- 172.21.15.104:0/135994445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c006e4b0 0x7f15c006cd50 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f15c0071210 tx=0x7f15bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0f947f61808484 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c4ad4700 1 -- 172.21.15.104:0/135994445 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc011830 con 0x7f15c006e4b0 2023-06-10T18:56:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c4ad4700 1 -- 172.21.15.104:0/135994445 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc011990 con 0x7f15c006e4b0 2023-06-10T18:56:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c4ad4700 1 -- 172.21.15.104:0/135994445 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc011d20 con 0x7f15c006e4b0 2023-06-10T18:56:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 -- 172.21.15.104:0/135994445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c006e4b0 msgr2=0x7f15c006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/135994445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c006e4b0 0x7f15c006cd50 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f15c0071210 tx=0x7f15bc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 -- 172.21.15.104:0/135994445 shutdown_connections 2023-06-10T18:56:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/135994445 >> v2:172.21.15.114:3300/0 conn(0x7f15c0069e70 0x7f15c00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/135994445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c006e4b0 0x7f15c006cd50 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 -- 172.21.15.104:0/135994445 >> 172.21.15.104:0/135994445 conn(0x7f15c006b8a0 msgr2=0x7f15c00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:29.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 -- 172.21.15.104:0/135994445 shutdown_connections 2023-06-10T18:56:29.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.129+0000 7f15c7d3a700 1 -- 172.21.15.104:0/135994445 wait complete. 2023-06-10T18:56:29.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 Processor -- start 2023-06-10T18:56:29.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 -- start start 2023-06-10T18:56:29.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15c006e0e0 0x7f15c01f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15c006e4b0 0x7f15c01572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00b1210 con 0x7f15c0069e70 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00b1b70 con 0x7f15c006e0e0 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c7d3a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00b0590 con 0x7f15c006e4b0 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c5ad6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c5ad6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37462/0 (socket says 172.21.15.104:37462) 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.131+0000 7f15c5ad6700 1 -- 172.21.15.104:0/4122629774 learned_addr learned my addr 172.21.15.104:0/4122629774 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c52d5700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15c006e0e0 0x7f15c01f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c62d7700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15c006e4b0 0x7f15c01572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 -- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15c006e4b0 msgr2=0x7f15c01572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15c006e4b0 0x7f15c01572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 -- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15c006e0e0 msgr2=0x7f15c01f4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15c006e0e0 0x7f15c01f4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 -- 172.21.15.104:0/4122629774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc009550 con 0x7f15c0069e70 2023-06-10T18:56:29.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c5ad6700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f15b000c4d0 tx=0x7f15b000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:29.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b00052d0 con 0x7f15c0069e70 2023-06-10T18:56:29.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0005430 con 0x7f15c0069e70 2023-06-10T18:56:29.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c00b0870 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b00038f0 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.132+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c00b0e30 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.133+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c0074050 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.133+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f15b0003a50 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.134+0000 7f15b67fc700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 0x7f15a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.134+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f15b0088890 con 0x7f15c0069e70 2023-06-10T18:56:29.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.134+0000 7f15c52d5700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 0x7f15a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:29.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.134+0000 7f15c52d5700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 0x7f15a406d190 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009c10 tx=0x7f15bc01d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:29.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.138+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f15b0053890 con 0x7f15c0069e70 2023-06-10T18:56:29.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.368+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15c00acea0 con 0x7f15c0069e70 2023-06-10T18:56:29.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.369+0000 7f15b67fc700 1 -- 172.21.15.104:0/4122629774 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f15b0056eb0 con 0x7f15c0069e70 2023-06-10T18:56:29.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:29.374 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 msgr2=0x7f15a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 0x7f15a406d190 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009c10 tx=0x7f15bc01d040 comp rx=0 tx=0).stop 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 msgr2=0x7f15c00b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f15b000c4d0 tx=0x7f15b000c7e0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 shutdown_connections 2023-06-10T18:56:29.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15c006e4b0 0x7f15c01572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15c0069e70 0x7f15c00b1630 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15a406ace0 0x7f15a406d190 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 --2- 172.21.15.104:0/4122629774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15c006e0e0 0x7f15c01f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 >> 172.21.15.104:0/4122629774 conn(0x7f15c006b8a0 msgr2=0x7f15c00ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 shutdown_connections 2023-06-10T18:56:29.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:29.371+0000 7f15c7d3a700 1 -- 172.21.15.104:0/4122629774 wait complete. 2023-06-10T18:56:29.378 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:29.387 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:29.388 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:29.405 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:56:29.406 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:56:29.406 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:29.407 DEBUG:teuthology.orchestra.run.smithi104:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2023-06-10T18:56:29.407 DEBUG:teuthology.orchestra.run.smithi104:> f.write("content") 2023-06-10T18:56:29.407 DEBUG:teuthology.orchestra.run.smithi104:> f.flush() 2023-06-10T18:56:29.408 DEBUG:teuthology.orchestra.run.smithi104:> while True: 2023-06-10T18:56:29.408 DEBUG:teuthology.orchestra.run.smithi104:> time.sleep(1) 2023-06-10T18:56:29.408 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:56:29.447 DEBUG:teuthology.orchestra.run.smithi104:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/background_file 2023-06-10T18:56:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:56:29.464 INFO:teuthology.orchestra.run.smithi104.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2023-06-10T18:56:30.465 DEBUG:teuthology.orchestra.run.smithi104:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/background_file 2023-06-10T18:56:30.487 INFO:teuthology.orchestra.run.smithi104.stdout:7 2023-06-10T18:56:30.488 INFO:tasks.cephfs.mount:File background_file became visible with size 7 from 0 after 1s 2023-06-10T18:56:30.489 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:56:30.490 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:56:30.490 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:56:30.491 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:56:30.491 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:56:30.491 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:30.491 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:56:30.492 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:56:30.492 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:56:30.492 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:56:30.493 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:30.493 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:56:30.493 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:56:30.493 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:56:30.494 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:56:30.494 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:30.495 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:56:30.495 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:56:30.496 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:56:30.496 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:56:30.497 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:56:30.498 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:56:30.498 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:56:30.499 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:56:30.500 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:56:30.501 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:56:30.501 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:56:30.619 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.148724.asok 2023-06-10T18:56:30.800 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.148724.asok 2023-06-10T18:56:30.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:30.801 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.148724.asok mds_sessions 2023-06-10T18:56:30.960 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:56:30.960 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 9387, 2023-06-10T18:56:30.960 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:56:30.960 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 9387 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:56:30.961 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3796851905 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.9387 v1:192.168.0.1:0/3796851905", 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/3796851905", 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:56:30.962 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6834", 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3783425308 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:30.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6836", 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3783425308 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:56:30.964 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:57:27.275110+0000", 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:56:27.275110+0000", 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 2, 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:56:30.965 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:56:30.966 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 80 2023-06-10T18:56:30.966 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:56:30.970 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.1/background_file 2023-06-10T18:56:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.1/background_file 2023-06-10T18:56:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 1 IO Block: 4194304 regular file 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 2fh/47d Inode: 1099511627776 Links: 1 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0664/-rw-rw-r--) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fusefs_t:s0 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:56:29.505945929 +0000 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:56:29.511916194 +0000 2023-06-10T18:56:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:56:29.511916194 +0000 2023-06-10T18:56:31.001 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:56:31.001 INFO:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2023-06-10T18:56:31.001 INFO:tasks.cephfs.mount:Suspending the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:56:31.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:31.002 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 down 2023-06-10T18:56:36.038 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:36.039 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:36.059 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:56:36.060 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:56:36.060 DEBUG:teuthology.orchestra.run.smithi114:> import time 2023-06-10T18:56:36.060 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:56:36.061 DEBUG:teuthology.orchestra.run.smithi114:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2023-06-10T18:56:36.061 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:56:36.061 DEBUG:teuthology.orchestra.run.smithi114:> while True: 2023-06-10T18:56:36.062 DEBUG:teuthology.orchestra.run.smithi114:> os.write(fd, b'"'"'content'"'"') 2023-06-10T18:56:36.062 DEBUG:teuthology.orchestra.run.smithi114:> time.sleep(1) 2023-06-10T18:56:36.062 DEBUG:teuthology.orchestra.run.smithi114:> if not False: 2023-06-10T18:56:36.062 DEBUG:teuthology.orchestra.run.smithi114:> break 2023-06-10T18:56:36.063 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:56:36.063 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:56:36.063 DEBUG:teuthology.orchestra.run.smithi114:> os.close(fd) 2023-06-10T18:56:36.063 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:56:39.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:56:39.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:56:41.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:41.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.331+0000 7fa5d3596700 1 Processor -- start 2023-06-10T18:56:41.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.331+0000 7fa5d3596700 1 -- start start 2023-06-10T18:56:41.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d3596700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa5cc0ada30 0x7fa5cc0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d3596700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0aed20 con 0x7fa5cc0ae930 2023-06-10T18:56:41.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d3596700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0cee70 con 0x7fa5cc0ade00 2023-06-10T18:56:41.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d3596700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cf430 con 0x7fa5cc0ada30 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d1b33700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa5cc0ae930 0x7fa5cc06d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35304/0 (socket says 172.21.15.104:35304) 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d1b33700 1 -- 172.21.15.104:0/2016235320 learned_addr learned my addr 172.21.15.104:0/2016235320 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2016235320 >> v2:172.21.15.114:3300/0 conn(0x7fa5cc0ada30 0x7fa5cc0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658360939 0 0) 0x7fa5cc0aed20 con 0x7fa5cc0ae930 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80035b0 con 0x7fa5cc0ae930 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114091552 0 0) 0x7fa5b80035b0 con 0x7fa5cc0ae930 2023-06-10T18:56:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 >> v2:172.21.15.114:3300/0 conn(0x7fa5cc0ada30 msgr2=0x7fa5cc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 --2- 172.21.15.104:0/2016235320 >> v2:172.21.15.114:3300/0 conn(0x7fa5cc0ada30 0x7fa5cc0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 >> v1:172.21.15.114:6789/0 conn(0x7fa5cc0ade00 legacy=0x7fa5cc0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0cf150 con 0x7fa5cc0ae930 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa5c40019f0 con 0x7fa5cc0ae930 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 >> v1:172.21.15.104:6789/0 conn(0x7fa5cc0ae930 legacy=0x7fa5cc06d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.332+0000 7fa5c3fff700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5b8003a00 0x7fa5b8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 0x7fa5b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8005200 0x7fa5b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cf430 con 0x7fa5b8004850 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b800c110 con 0x7fa5b8005200 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b800c2d0 con 0x7fa5b8003a00 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 0x7fa5b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d1b33700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8005200 0x7fa5b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5b8003a00 0x7fa5b8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5b8003a00 msgr2=0x7fa5b8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5b8003a00 0x7fa5b8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8005200 msgr2=0x7fa5b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8005200 0x7fa5b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2016235320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80035b0 con 0x7fa5b8004850 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d1b33700 1 -- 172.21.15.104:0/2016235320 reap_dead start 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 0x7fa5b8004cc0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc067210 tx=0x7fa5b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91e77ca7a1dfab3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:41.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0052d0 con 0x7fa5b8004850 2023-06-10T18:56:41.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc005450 con 0x7fa5b8004850 2023-06-10T18:56:41.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.333+0000 7fa5c3fff700 1 -- 172.21.15.104:0/2016235320 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0038c0 con 0x7fa5b8004850 2023-06-10T18:56:41.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 -- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 msgr2=0x7fa5b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 0x7fa5b8004cc0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc067210 tx=0x7fa5b8005880 comp rx=0 tx=0).stop 2023-06-10T18:56:41.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 -- 172.21.15.104:0/2016235320 shutdown_connections 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2016235320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5b8004850 0x7fa5b8004cc0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 -- 172.21.15.104:0/2016235320 >> 172.21.15.104:0/2016235320 conn(0x7fa5cc06b8a0 msgr2=0x7fa5cc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 -- 172.21.15.104:0/2016235320 shutdown_connections 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.334+0000 7fa5d3596700 1 -- 172.21.15.104:0/2016235320 wait complete. 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.335+0000 7fa5d3596700 1 Processor -- start 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.335+0000 7fa5d3596700 1 -- start start 2023-06-10T18:56:41.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 0x7fa5cc0c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5cc09c980 0x7fa5cc09ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0c40f0 con 0x7fa5cc0c4510 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc09d340 con 0x7fa5b8004850 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d3596700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1f5d50 con 0x7fa5cc09c980 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37486/0 (socket says 172.21.15.104:37486) 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2840669806 learned_addr learned my addr 172.21.15.104:0/2840669806 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 0x7fa5cc0c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5cc09c980 msgr2=0x7fa5cc09ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5cc09c980 0x7fa5cc09ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 msgr2=0x7fa5cc0c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 0x7fa5cc0c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 -- 172.21.15.104:0/2840669806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0c3470 con 0x7fa5cc0c4510 2023-06-10T18:56:41.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 0x7fa5cc0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5d0b31700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c0e0 tx=0x7fa5bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.336+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc001290 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.337+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc007040 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.337+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc0c3750 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.337+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0073f0 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.337+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc0c3d10 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.338+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc0c4230 con 0x7fa5cc0c4510 2023-06-10T18:56:41.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.338+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa5bc007550 con 0x7fa5cc0c4510 2023-06-10T18:56:41.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.338+0000 7fa5c1ffb700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 0x7fa5b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:41.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.339+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa5bc089de0 con 0x7fa5cc0c4510 2023-06-10T18:56:41.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.339+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 0x7fa5b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:41.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.339+0000 7fa5d1332700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 0x7fa5b006d190 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009b30 tx=0x7fa5c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:41.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.343+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5cc0c4230 con 0x7fa5cc0c4510 2023-06-10T18:56:41.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.570+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5cc0c4230 con 0x7fa5cc0c4510 2023-06-10T18:56:41.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.570+0000 7fa5c1ffb700 1 -- 172.21.15.104:0/2840669806 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fa5cc0c4230 con 0x7fa5cc0c4510 2023-06-10T18:56:41.575 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:41.576 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 msgr2=0x7fa5b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 0x7fa5b006d190 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009b30 tx=0x7fa5c8008040 comp rx=0 tx=0).stop 2023-06-10T18:56:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 msgr2=0x7fa5cc1f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c0e0 tx=0x7fa5bc001120 comp rx=0 tx=0).stop 2023-06-10T18:56:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 shutdown_connections 2023-06-10T18:56:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa5b006ace0 0x7fa5b006d190 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b8004850 0x7fa5cc0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5cc09c980 0x7fa5cc09ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 --2- 172.21.15.104:0/2840669806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5cc0c4510 0x7fa5cc1f5160 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 >> 172.21.15.104:0/2840669806 conn(0x7fa5cc06b8a0 msgr2=0x7fa5cc0ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 shutdown_connections 2023-06-10T18:56:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:41.572+0000 7fa5d3596700 1 -- 172.21.15.104:0/2840669806 wait complete. 2023-06-10T18:56:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:41.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session evict 9387 2023-06-10T18:56:42.742 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2023-06-10T18:56:44.975 INFO:tasks.cephfs.test_client_recovery:cap_waiter waited 2.232088327407837s 2023-06-10T18:56:44.975 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:56:44.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:44.976 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:56:45.064 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:56:45.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:56:45.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:56:45.203 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:56:45.204 INFO:tasks.cephfs.mount:Terminating background process 2023-06-10T18:56:45.204 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:45.205 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:45.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.378+0000 7f6ebad7a700 1 Processor -- start 2023-06-10T18:56:45.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.378+0000 7f6ebad7a700 1 -- start start 2023-06-10T18:56:45.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6ebad7a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6eb406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6ebad7a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb406d290 con 0x7f6eb406e4b0 2023-06-10T18:56:45.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6ebad7a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40ca2d0 con 0x7f6eb406e0e0 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6ebad7a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4071640 con 0x7f6eb40cf100 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6eb3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6eb406e4b0 0x7f6eb40ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35320/0 (socket says 172.21.15.104:35320) 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6eb3fff700 1 -- 172.21.15.104:0/1464780466 learned_addr learned my addr 172.21.15.104:0/1464780466 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6eb9317700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6eb406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764461798 0 0) 0x7f6eb406d290 con 0x7f6eb406e4b0 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.379+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea40035b0 con 0x7f6eb406e4b0 2023-06-10T18:56:45.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088120863 0 0) 0x7f6ea40035b0 con 0x7f6eb406e4b0 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 msgr2=0x7f6eb406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6eb406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 >> v1:172.21.15.114:6789/0 conn(0x7f6eb406e0e0 legacy=0x7f6eb40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb406d4c0 con 0x7f6eb406e4b0 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6ea0001670 con 0x7f6eb406e4b0 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 >> v1:172.21.15.104:6789/0 conn(0x7f6eb406e4b0 legacy=0x7f6eb40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ea4003a30 0x7f6ea4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6ea4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4004d90 0x7f6ea4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4071640 con 0x7f6ea4004d90 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea400bac0 con 0x7f6ea4003a30 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea400bc80 con 0x7f6eb40cf100 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4004d90 0x7f6ea4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb8b16700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ea4003a30 0x7f6ea4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb3fff700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6ea4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 -- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 msgr2=0x7f6ea4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40cf100 0x7f6ea4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 -- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ea4003a30 msgr2=0x7f6ea4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ea4003a30 0x7f6ea4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 -- 172.21.15.104:0/1464780466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40035b0 con 0x7f6ea4004d90 2023-06-10T18:56:45.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb9317700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4004d90 0x7f6ea4005200 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009f90 tx=0x7f6ea4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9eda3e0398fba3e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00ec30 con 0x7f6ea4004d90 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac00f040 con 0x7f6ea4004d90 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.380+0000 7f6eb37fe700 1 -- 172.21.15.104:0/1464780466 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac014480 con 0x7f6ea4004d90 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 -- 172.21.15.104:0/1464780466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4004d90 msgr2=0x7f6ea4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/1464780466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ea4004d90 0x7f6ea4005200 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009f90 tx=0x7f6ea4005880 comp rx=0 tx=0).stop 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6eb9317700 1 -- 172.21.15.104:0/1464780466 reap_dead start 2023-06-10T18:56:45.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 -- 172.21.15.104:0/1464780466 shutdown_connections 2023-06-10T18:56:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 -- 172.21.15.104:0/1464780466 >> 172.21.15.104:0/1464780466 conn(0x7f6eb406b8a0 msgr2=0x7f6eb40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 -- 172.21.15.104:0/1464780466 shutdown_connections 2023-06-10T18:56:45.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.381+0000 7f6ebad7a700 1 -- 172.21.15.104:0/1464780466 wait complete. 2023-06-10T18:56:45.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.382+0000 7f6ebad7a700 1 Processor -- start 2023-06-10T18:56:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.382+0000 7f6ebad7a700 1 -- start start 2023-06-10T18:56:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40aef40 0x7f6eb40b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb40af310 0x7f6eb4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb41f3150 con 0x7f6eb41f2c00 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4159110 con 0x7f6eb40af310 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6ebad7a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40b0770 con 0x7f6eb40aef40 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37506/0 (socket says 172.21.15.104:37506) 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 -- 172.21.15.104:0/3056489812 learned_addr learned my addr 172.21.15.104:0/3056489812 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:45.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb9317700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb40af310 0x7f6eb4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 -- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40aef40 msgr2=0x7f6eb40b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb8b16700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40aef40 0x7f6eb40b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40aef40 0x7f6eb40b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 -- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb40af310 msgr2=0x7f6eb4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb40af310 0x7f6eb4158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.383+0000 7f6eb3fff700 1 -- 172.21.15.104:0/3056489812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40b0a50 con 0x7f6eb41f2c00 2023-06-10T18:56:45.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6eb3fff700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007290 tx=0x7f6ea0007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000f660 con 0x7f6eb41f2c00 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea000f7c0 con 0x7f6eb41f2c00 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb40b0d30 con 0x7f6eb41f2c00 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000fb70 con 0x7f6eb41f2c00 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.384+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb40af950 con 0x7f6eb41f2c00 2023-06-10T18:56:45.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.385+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb40b1250 con 0x7f6eb41f2c00 2023-06-10T18:56:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.385+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6ea000fcd0 con 0x7f6eb41f2c00 2023-06-10T18:56:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.385+0000 7f6eb17fa700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 0x7f6e9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.386+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6ea0089610 con 0x7f6eb41f2c00 2023-06-10T18:56:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.386+0000 7f6eb8b16700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 0x7f6e9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.386+0000 7f6eb8b16700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 0x7f6e9806d190 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40ae800 tx=0x7f6ea8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.390+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6eb40b1250 con 0x7f6eb41f2c00 2023-06-10T18:56:45.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:56:45.538+0000 7ff7f77fe700 -1 client.9387 I was blocklisted at osd epoch 129 2023-06-10T18:56:45.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:56:45.538+0000 7ff7f77fe700 -1 client.9387 remove_session_caps still has dirty data on 0x10000000000.head(faked_ino=0 nref=7 ll_ref=2 cap_refs={4=0,1024=1,4096=0,8192=1} open={2=0} mode=100664 size=7/4194304 nlink=1 btime=2023-06-10T18:56:29.505945+0000 mtime=2023-06-10T18:56:29.511916+0000 ctime=2023-06-10T18:56:29.511916+0000 change_attr=1 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 7] parents=0x1.head["background_file"] 0x7ff7f0009850) 2023-06-10T18:56:45.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.577+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eb40b1250 con 0x7f6eb41f2c00 2023-06-10T18:56:45.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.579+0000 7f6eb17fa700 1 -- 172.21.15.104:0/3056489812 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+18674 (secure 0 0 0) 0x7f6eb40b1250 con 0x7f6eb41f2c00 2023-06-10T18:56:45.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:45.585 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":129,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:42.706404+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3796851905":"2023-06-10T19:56:41.727300+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:45.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 msgr2=0x7f6e9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 0x7f6e9806d190 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40ae800 tx=0x7f6ea8009040 comp rx=0 tx=0).stop 2023-06-10T18:56:45.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 msgr2=0x7f6eb4158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0007290 tx=0x7f6ea0007650 comp rx=0 tx=0).stop 2023-06-10T18:56:45.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 shutdown_connections 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e9806ace0 0x7f6e9806d190 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb40af310 0x7f6eb4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6eb40aef40 0x7f6eb40b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 --2- 172.21.15.104:0/3056489812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6eb41f2c00 0x7f6eb4158690 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 >> 172.21.15.104:0/3056489812 conn(0x7f6eb406b8a0 msgr2=0x7f6eb415bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 shutdown_connections 2023-06-10T18:56:45.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.581+0000 7f6ebad7a700 1 -- 172.21.15.104:0/3056489812 wait complete. 2023-06-10T18:56:45.599 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:56:17.300478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:56:18.192765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:45.600 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:45.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.715+0000 7fd5a0cb6700 1 Processor -- start 2023-06-10T18:56:45.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.715+0000 7fd5a0cb6700 1 -- start start 2023-06-10T18:56:45.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd5a0cb6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd59c0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd5a0cb6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0ce340 con 0x7fd59c0ccff0 2023-06-10T18:56:45.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd5a0cb6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0d00d0 con 0x7fd59c0ce7d0 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd5a0cb6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0cd460 con 0x7fd59c06af80 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd599d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd59c0ccff0 0x7fd59c06b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35342/0 (socket says 172.21.15.104:35342) 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd599d9b700 1 -- 172.21.15.104:0/4019688444 learned_addr learned my addr 172.21.15.104:0/4019688444 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.716+0000 7fd59a59c700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd59c0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641309553 0 0) 0x7fd59c0ce340 con 0x7fd59c0ccff0 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5840035b0 con 0x7fd59c0ccff0 2023-06-10T18:56:45.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3651470148 0 0) 0x7fd5840035b0 con 0x7fd59c0ccff0 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 msgr2=0x7fd59c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd59c0d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 >> v1:172.21.15.114:6789/0 conn(0x7fd59c0ce7d0 legacy=0x7fd59c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0cd680 con 0x7fd59c0ccff0 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd58c001670 con 0x7fd59c0ccff0 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 >> v1:172.21.15.104:6789/0 conn(0x7fd59c0ccff0 legacy=0x7fd59c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd584003a30 0x7fd584003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd584004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584004d90 0x7fd584005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0cd460 con 0x7fd584003a30 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400ba10 con 0x7fd584004d90 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400bbd0 con 0x7fd59c06af80 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd584003a30 0x7fd584003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd599d9b700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd584004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59ad9d700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584004d90 0x7fd584005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 -- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 msgr2=0x7fd584004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c06af80 0x7fd584004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 -- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584004d90 msgr2=0x7fd584005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584004d90 0x7fd584005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 -- 172.21.15.104:0/4019688444 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840035b0 con 0x7fd584003a30 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.717+0000 7fd59a59c700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd584003a30 0x7fd584003e00 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fd590009b60 tx=0x7fd584005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7097c1d4344c890 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000e6c0 con 0x7fd584003a30 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000e820 con 0x7fd584003a30 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd59959a700 1 -- 172.21.15.104:0/4019688444 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000eb90 con 0x7fd584003a30 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/4019688444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd584003a30 msgr2=0x7fd584003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/4019688444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd584003a30 0x7fd584003e00 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fd590009b60 tx=0x7fd584005880 comp rx=0 tx=0).stop 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd59ad9d700 1 -- 172.21.15.104:0/4019688444 reap_dead start 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/4019688444 shutdown_connections 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/4019688444 >> 172.21.15.104:0/4019688444 conn(0x7fd59c089fd0 msgr2=0x7fd59c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:45.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/4019688444 shutdown_connections 2023-06-10T18:56:45.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.718+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/4019688444 wait complete. 2023-06-10T18:56:45.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 Processor -- start 2023-06-10T18:56:45.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 -- start start 2023-06-10T18:56:45.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584003a30 0x7fd59c154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c0d0160 0x7fd59c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0d0600 con 0x7fd59c0b1560 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b0d70 con 0x7fd584003a30 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.720+0000 7fd5a0cb6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b01f0 con 0x7fd59c0d0160 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37528/0 (socket says 172.21.15.104:37528) 2023-06-10T18:56:45.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 -- 172.21.15.104:0/1160958093 learned_addr learned my addr 172.21.15.104:0/1160958093 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd59ad9d700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c0d0160 0x7fd59c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd59a59c700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584003a30 0x7fd59c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 -- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c0d0160 msgr2=0x7fd59c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c0d0160 0x7fd59c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 -- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584003a30 msgr2=0x7fd59c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584003a30 0x7fd59c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 -- 172.21.15.104:0/1160958093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0b04d0 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.721+0000 7fd599d9b700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fd58c007290 tx=0x7fd58c0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.722+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c011580 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.722+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0b07b0 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.722+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c011700 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.722+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c011ab0 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.722+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c0b19d0 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.723+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c074050 con 0x7fd59c0b1560 2023-06-10T18:56:45.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.723+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd58c011c10 con 0x7fd59c0b1560 2023-06-10T18:56:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.723+0000 7fd58affd700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 0x7fd57c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.724+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd58c089560 con 0x7fd59c0b1560 2023-06-10T18:56:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.724+0000 7fd59a59c700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 0x7fd57c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.724+0000 7fd59a59c700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 0x7fd57c06d190 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd590009a80 tx=0x7fd59001f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:45.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.729+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd58c054560 con 0x7fd59c0b1560 2023-06-10T18:56:45.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.915+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd59c06dd50 con 0x7fd59c0b1560 2023-06-10T18:56:45.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.915+0000 7fd58affd700 1 -- 172.21.15.104:0/1160958093 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd58c057b80 con 0x7fd59c0b1560 2023-06-10T18:56:45.921 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:45.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 msgr2=0x7fd57c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 0x7fd57c06d190 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd590009a80 tx=0x7fd59001f040 comp rx=0 tx=0).stop 2023-06-10T18:56:45.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 msgr2=0x7fd59c0cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:45.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fd58c007290 tx=0x7fd58c0075a0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 shutdown_connections 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd57c06ace0 0x7fd57c06d190 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd584003a30 0x7fd59c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd59c0d0160 0x7fd59c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 --2- 172.21.15.104:0/1160958093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd59c0b1560 0x7fd59c0cfc20 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 >> 172.21.15.104:0/1160958093 conn(0x7fd59c089fd0 msgr2=0x7fd59c0ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:45.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 shutdown_connections 2023-06-10T18:56:45.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:45.917+0000 7fd5a0cb6700 1 -- 172.21.15.104:0/1160958093 wait complete. 2023-06-10T18:56:45.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.051+0000 7f409c528700 1 Processor -- start 2023-06-10T18:56:46.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 -- start start 2023-06-10T18:56:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409415c6f0 0x7f40940b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f409406ea70 0x7f40940b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 0x7f40940ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40940ae7e0 con 0x7f409406ea70 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409406df90 con 0x7f40940aec00 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.052+0000 7f409c528700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409406e270 con 0x7f409415c6f0 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f409406ea70 0x7f40940b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f409406ea70 0x7f40940b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37536/0 (socket says 172.21.15.104:37536) 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 -- 172.21.15.104:0/1101433898 learned_addr learned my addr 172.21.15.104:0/1101433898 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f409aac5700 1 --2- 172.21.15.104:0/1101433898 >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 0x7f40940ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f409a2c4700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409415c6f0 0x7f40940b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 -- 172.21.15.104:0/1101433898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409415c6f0 msgr2=0x7f40940b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409415c6f0 0x7f40940b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 -- 172.21.15.104:0/1101433898 >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 msgr2=0x7f40940ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 --2- 172.21.15.104:0/1101433898 >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 0x7f40940ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 -- 172.21.15.104:0/1101433898 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409406e550 con 0x7f409406ea70 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f409aac5700 1 --2- 172.21.15.104:0/1101433898 >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 0x7f40940ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f4099ac3700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409406ea70 0x7f40940b15b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f4094072040 tx=0x7f409000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a27ee269e697ae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f40992c2700 1 -- 172.21.15.104:0/1101433898 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000dd30 con 0x7f409406ea70 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f40992c2700 1 -- 172.21.15.104:0/1101433898 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409000de90 con 0x7f409406ea70 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.053+0000 7f40992c2700 1 -- 172.21.15.104:0/1101433898 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090010690 con 0x7f409406ea70 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 -- 172.21.15.104:0/1101433898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409406ea70 msgr2=0x7f40940b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409406ea70 0x7f40940b15b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f4094072040 tx=0x7f409000be70 comp rx=0 tx=0).stop 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 -- 172.21.15.104:0/1101433898 shutdown_connections 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 --2- 172.21.15.104:0/1101433898 >> v2:172.21.15.114:3300/0 conn(0x7f40940aec00 0x7f40940ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409415c6f0 0x7f40940b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 --2- 172.21.15.104:0/1101433898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409406ea70 0x7f40940b15b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 -- 172.21.15.104:0/1101433898 >> 172.21.15.104:0/1101433898 conn(0x7f409406b8a0 msgr2=0x7f40940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 -- 172.21.15.104:0/1101433898 shutdown_connections 2023-06-10T18:56:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.054+0000 7f409c528700 1 -- 172.21.15.104:0/1101433898 wait complete. 2023-06-10T18:56:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.055+0000 7f409c528700 1 Processor -- start 2023-06-10T18:56:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.055+0000 7f409c528700 1 -- start start 2023-06-10T18:56:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409406ea70 0x7f40941ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 0x7f40941f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940c7430 con 0x7f409415c6f0 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940c67b0 con 0x7f40940aec00 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409c528700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40940c6a90 con 0x7f409406ea70 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37544/0 (socket says 172.21.15.104:37544) 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 -- 172.21.15.104:0/2792171810 learned_addr learned my addr 172.21.15.104:0/2792171810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409a2c4700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409406ea70 0x7f40941ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f4099ac3700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 0x7f40941f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 -- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409406ea70 msgr2=0x7f40941ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409406ea70 0x7f40941ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 -- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 msgr2=0x7f40941f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 0x7f40941f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 -- 172.21.15.104:0/2792171810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409000b7f0 con 0x7f409415c6f0 2023-06-10T18:56:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f4099ac3700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 0x7f40941f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.056+0000 7f409aac5700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f408c007de0 tx=0x7f408c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00ddd0 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c00f040 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c003c60 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940c6d70 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4094157cb0 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.057+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40940c7570 con 0x7f409415c6f0 2023-06-10T18:56:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.058+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f408c004030 con 0x7f409415c6f0 2023-06-10T18:56:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.058+0000 7f408affd700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 0x7f407806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.058+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f408c089670 con 0x7f409415c6f0 2023-06-10T18:56:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.059+0000 7f409a2c4700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 0x7f407806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.059+0000 7f409a2c4700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 0x7f407806d190 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4084006fd0 tx=0x7f4084008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.063+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40940c7570 con 0x7f409415c6f0 2023-06-10T18:56:46.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.248+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f40940c7570 con 0x7f409415c6f0 2023-06-10T18:56:46.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.248+0000 7f408affd700 1 -- 172.21.15.104:0/2792171810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v129) v1 ==== 92+0+10 (secure 0 0 0) 0x7f40940c7570 con 0x7f409415c6f0 2023-06-10T18:56:46.255 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 msgr2=0x7f407806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 0x7f407806d190 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4084006fd0 tx=0x7f4084008040 comp rx=0 tx=0).stop 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 msgr2=0x7f4094157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f408c007de0 tx=0x7f408c00d3b0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 shutdown_connections 2023-06-10T18:56:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f409415c6f0 0x7f4094157770 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f409406ea70 0x7f40941ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f407806ace0 0x7f407806d190 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.250+0000 7f409c528700 1 --2- 172.21.15.104:0/2792171810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40940aec00 0x7f40941f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.251+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 >> 172.21.15.104:0/2792171810 conn(0x7f409406b8a0 msgr2=0x7f40940ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.251+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 shutdown_connections 2023-06-10T18:56:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.251+0000 7f409c528700 1 -- 172.21.15.104:0/2792171810 wait complete. 2023-06-10T18:56:46.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:56:46.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.384+0000 7f03f8f37700 1 Processor -- start 2023-06-10T18:56:46.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.384+0000 7f03f8f37700 1 -- start start 2023-06-10T18:56:46.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.385+0000 7f03f8f37700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03f40cb5e0 0x7f03f40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.385+0000 7f03f8f37700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40cc9f0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.385+0000 7f03f8f37700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f40cc100 con 0x7f03f40cb210 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.385+0000 7f03f8f37700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f40cb9e0 con 0x7f03f40cb5e0 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f1d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f03f40cd8f0 0x7f03f406d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35378/0 (socket says 172.21.15.104:35378) 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f1d9b700 1 -- 172.21.15.104:0/2886967814 learned_addr learned my addr 172.21.15.104:0/2886967814 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2886967814 >> v2:172.21.15.114:3300/0 conn(0x7f03f40cb5e0 0x7f03f40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508417271 0 0) 0x7f03f40cc9f0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc0035b0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255228383 0 0) 0x7f03f40cc100 con 0x7f03f40cb210 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f40cc9f0 con 0x7f03f40cb210 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421240078 0 0) 0x7f03dc0035b0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 >> v2:172.21.15.114:3300/0 conn(0x7f03f40cb5e0 msgr2=0x7f03f40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 --2- 172.21.15.104:0/2886967814 >> v2:172.21.15.114:3300/0 conn(0x7f03f40cb5e0 0x7f03f40cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 >> v1:172.21.15.114:6789/0 conn(0x7f03f40cb210 legacy=0x7f03f40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40cc3a0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f03e40015f0 con 0x7f03f40cd8f0 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 >> v1:172.21.15.104:6789/0 conn(0x7f03f40cd8f0 legacy=0x7f03f406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 0x7f03dc003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc004850 0x7f03dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03dc005200 0x7f03dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc00c0e0 con 0x7f03dc003be0 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc00c220 con 0x7f03dc004850 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc00c3b0 con 0x7f03dc005200 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 0x7f03dc003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f1d9b700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc004850 0x7f03dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03dc005200 0x7f03dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 -- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03dc005200 msgr2=0x7f03dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03dc005200 0x7f03dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 -- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc004850 msgr2=0x7f03dc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc004850 0x7f03dc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f259c700 1 -- 172.21.15.104:0/2886967814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0035b0 con 0x7f03dc003be0 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03dc005200 0x7f03dc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f1d9b700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc004850 0x7f03dc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.386+0000 7f03f2d9d700 1 -- 172.21.15.104:0/2886967814 reap_dead start 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f259c700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 0x7f03dc003fd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f03e8001970 tx=0x7f03dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb92ce77381a572c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8001460 con 0x7f03dc003be0 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e80052d0 con 0x7f03dc003be0 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f159a700 1 -- 172.21.15.104:0/2886967814 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8001da0 con 0x7f03dc003be0 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 -- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 msgr2=0x7f03dc003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 0x7f03dc003fd0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f03e8001970 tx=0x7f03dc005880 comp rx=0 tx=0).stop 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 -- 172.21.15.104:0/2886967814 shutdown_connections 2023-06-10T18:56:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2886967814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03dc003be0 0x7f03dc003fd0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 -- 172.21.15.104:0/2886967814 >> 172.21.15.104:0/2886967814 conn(0x7f03f406b8a0 msgr2=0x7f03f40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 -- 172.21.15.104:0/2886967814 shutdown_connections 2023-06-10T18:56:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.387+0000 7f03f8f37700 1 -- 172.21.15.104:0/2886967814 wait complete. 2023-06-10T18:56:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 Processor -- start 2023-06-10T18:56:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 -- start start 2023-06-10T18:56:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 0x7f03f4158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f40b4280 0x7f03f40b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41e14a0 con 0x7f03f40c50d0 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41e0610 con 0x7f03dc003be0 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.389+0000 7f03f8f37700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40c5a60 con 0x7f03f40b4280 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37564/0 (socket says 172.21.15.104:37564) 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 -- 172.21.15.104:0/2218905802 learned_addr learned my addr 172.21.15.104:0/2218905802 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f259c700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 0x7f03f4158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f1d9b700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f40b4280 0x7f03f40b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 -- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f40b4280 msgr2=0x7f03f40b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f40b4280 0x7f03f40b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 -- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 msgr2=0x7f03f4158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 0x7f03f4158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 -- 172.21.15.104:0/2218905802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40b5190 con 0x7f03f40c50d0 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f259c700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 0x7f03f4158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f2d9d700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f03f40703e0 tx=0x7f03ec00f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00fe60 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec016420 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f41e0820 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0167b0 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.390+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f41e0de0 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.391+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f41e15e0 con 0x7f03f40c50d0 2023-06-10T18:56:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.391+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03ec016910 con 0x7f03f40c50d0 2023-06-10T18:56:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.392+0000 7f03e2ffd700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 0x7f03d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.392+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f03ec08a0e0 con 0x7f03f40c50d0 2023-06-10T18:56:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.392+0000 7f03f259c700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 0x7f03d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.393+0000 7f03f259c700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 0x7f03d406d190 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f03e8015040 tx=0x7f03e800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.396+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f03f41e15e0 con 0x7f03f40c50d0 2023-06-10T18:56:46.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.583+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f03f41e15e0 con 0x7f03f40c50d0 2023-06-10T18:56:46.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.584+0000 7f03e2ffd700 1 -- 172.21.15.104:0/2218905802 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f03f41e15e0 con 0x7f03f40c50d0 2023-06-10T18:56:46.590 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:56:46.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.585+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 msgr2=0x7f03d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.585+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 0x7f03d406d190 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f03e8015040 tx=0x7f03e800a000 comp rx=0 tx=0).stop 2023-06-10T18:56:46.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.585+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 msgr2=0x7f03f40c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.585+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f03f40703e0 tx=0x7f03ec00f430 comp rx=0 tx=0).stop 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 shutdown_connections 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03d406ace0 0x7f03d406d190 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03dc003be0 0x7f03f4158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03f40c50d0 0x7f03f40c5520 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 --2- 172.21.15.104:0/2218905802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f40b4280 0x7f03f40b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 >> 172.21.15.104:0/2218905802 conn(0x7f03f406b8a0 msgr2=0x7f03f406e390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 shutdown_connections 2023-06-10T18:56:46.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.586+0000 7f03f8f37700 1 -- 172.21.15.104:0/2218905802 wait complete. 2023-06-10T18:56:46.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:56:46.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.720+0000 7fbe20501700 1 Processor -- start 2023-06-10T18:56:46.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.720+0000 7fbe20501700 1 -- start start 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe20501700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe180b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe20501700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbe180aec00 0x7fbe180b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe20501700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180ae7e0 con 0x7fbe1806ea70 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe20501700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1806df90 con 0x7fbe180aec00 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe20501700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806e270 con 0x7fbe1815c6f0 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1da9c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbe1806ea70 0x7fbe180ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35396/0 (socket says 172.21.15.104:35396) 2023-06-10T18:56:46.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1da9c700 1 -- 172.21.15.104:0/862116262 learned_addr learned my addr 172.21.15.104:0/862116262 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1e29d700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe180b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766094071 0 0) 0x7fbe180ae7e0 con 0x7fbe1806ea70 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe040035b0 con 0x7fbe1806ea70 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/862116262 >> v2:172.21.15.114:3300/0 conn(0x7fbe180aec00 0x7fbe180b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063255965 0 0) 0x7fbe040035b0 con 0x7fbe1806ea70 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 msgr2=0x7fbe180b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe180b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 >> v2:172.21.15.114:3300/0 conn(0x7fbe180aec00 msgr2=0x7fbe180b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 --2- 172.21.15.104:0/862116262 >> v2:172.21.15.114:3300/0 conn(0x7fbe180aec00 0x7fbe180b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1806e550 con 0x7fbe1806ea70 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fbe08001670 con 0x7fbe1806ea70 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 >> v1:172.21.15.104:6789/0 conn(0x7fbe1806ea70 legacy=0x7fbe180ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.721+0000 7fbe1d29b700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04003a00 0x7fbe04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe04004d90 0x7fbe04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806df90 con 0x7fbe04004d90 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806e270 con 0x7fbe04003a00 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0400ba10 con 0x7fbe1815c6f0 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe04004d90 0x7fbe04005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1da9c700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1e29d700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04003a00 0x7fbe04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 -- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 msgr2=0x7fbe04004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe1815c6f0 0x7fbe04004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 -- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04003a00 msgr2=0x7fbe04003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04003a00 0x7fbe04003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 -- 172.21.15.104:0/862116262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040035b0 con 0x7fbe04004d90 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe04004d90 0x7fbe04005200 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009f90 tx=0x7fbe04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968d4e6dd400b14 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000ec30 con 0x7fbe04004d90 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1000f040 con 0x7fbe04004d90 2023-06-10T18:56:46.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.722+0000 7fbe1d29b700 1 -- 172.21.15.104:0/862116262 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10014480 con 0x7fbe04004d90 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 -- 172.21.15.104:0/862116262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe04004d90 msgr2=0x7fbe04005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 --2- 172.21.15.104:0/862116262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe04004d90 0x7fbe04005200 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009f90 tx=0x7fbe04005880 comp rx=0 tx=0).stop 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe1ea9e700 1 -- 172.21.15.104:0/862116262 reap_dead start 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 -- 172.21.15.104:0/862116262 shutdown_connections 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 -- 172.21.15.104:0/862116262 >> 172.21.15.104:0/862116262 conn(0x7fbe1806b8a0 msgr2=0x7fbe180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 -- 172.21.15.104:0/862116262 shutdown_connections 2023-06-10T18:56:46.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.723+0000 7fbe20501700 1 -- 172.21.15.104:0/862116262 wait complete. 2023-06-10T18:56:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.724+0000 7fbe20501700 1 Processor -- start 2023-06-10T18:56:46.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.724+0000 7fbe20501700 1 -- start start 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04004d90 0x7fbe180c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe180c5b10 0x7fbe180c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180c3a70 con 0x7fbe18146680 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe181f1a60 con 0x7fbe04004d90 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe20501700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe18149620 con 0x7fbe180c5b10 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37582/0 (socket says 172.21.15.104:37582) 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 -- 172.21.15.104:0/825612445 learned_addr learned my addr 172.21.15.104:0/825612445 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1ea9e700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe180c5b10 0x7fbe180c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1e29d700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04004d90 0x7fbe180c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 -- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe180c5b10 msgr2=0x7fbe180c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe180c5b10 0x7fbe180c5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 -- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04004d90 msgr2=0x7fbe180c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04004d90 0x7fbe180c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 -- 172.21.15.104:0/825612445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18149340 con 0x7fbe18146680 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.725+0000 7fbe1da9c700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000fb0 tx=0x7fbe080075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.726+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800f630 con 0x7fbe18146680 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.726+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe180c5030 con 0x7fbe18146680 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.726+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0800f7b0 con 0x7fbe18146680 2023-06-10T18:56:46.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.726+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe180c55f0 con 0x7fbe18146680 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.726+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800fb60 con 0x7fbe18146680 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.727+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe180c4e90 con 0x7fbe18146680 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.727+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbe0800fcc0 con 0x7fbe18146680 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.728+0000 7fbe0effd700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 0x7fbdfc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.728+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fbe080895d0 con 0x7fbe18146680 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.728+0000 7fbe1e29d700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 0x7fbdfc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.728+0000 7fbe1e29d700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 0x7fbdfc06d190 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbe180cb570 tx=0x7fbe140091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.732+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbe180c4e90 con 0x7fbe18146680 2023-06-10T18:56:46.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.916+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe180c4e90 con 0x7fbe18146680 2023-06-10T18:56:46.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.916+0000 7fbe0effd700 1 -- 172.21.15.104:0/825612445 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe180c4e90 con 0x7fbe18146680 2023-06-10T18:56:46.923 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:46.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 msgr2=0x7fbdfc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 0x7fbdfc06d190 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbe180cb570 tx=0x7fbe140091c0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 msgr2=0x7fbe18146af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:46.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000fb0 tx=0x7fbe080075f0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 shutdown_connections 2023-06-10T18:56:46.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdfc06ace0 0x7fbdfc06d190 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe04004d90 0x7fbe180c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe180c5b10 0x7fbe180c5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 --2- 172.21.15.104:0/825612445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe18146680 0x7fbe18146af0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 >> 172.21.15.104:0/825612445 conn(0x7fbe1806b8a0 msgr2=0x7fbe180cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 shutdown_connections 2023-06-10T18:56:46.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:46.918+0000 7fbe20501700 1 -- 172.21.15.104:0/825612445 wait complete. 2023-06-10T18:56:46.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:56:47.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 Processor -- start 2023-06-10T18:56:47.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 -- start start 2023-06-10T18:56:47.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 0x7f91a006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91a006e4b0 0x7f91a00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a006d290 con 0x7f91a00ca100 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a0071640 con 0x7f91a006e4b0 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.052+0000 7f91a83c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00aef00 con 0x7f91a006e0e0 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 0x7f91a006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a595e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91a006e4b0 0x7f91a00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a6960700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f91a00ca100 0x7f91a00ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41264/0 (socket says 172.21.15.104:41264) 2023-06-10T18:56:47.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a6960700 1 -- 172.21.15.104:0/3090355544 learned_addr learned my addr 172.21.15.104:0/3090355544 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 -- 172.21.15.104:0/3090355544 >> v2:172.21.15.114:3300/0 conn(0x7f91a006e4b0 msgr2=0x7f91a00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 --2- 172.21.15.104:0/3090355544 >> v2:172.21.15.114:3300/0 conn(0x7f91a006e4b0 0x7f91a00c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 -- 172.21.15.104:0/3090355544 >> v1:172.21.15.114:6789/0 conn(0x7f91a00ca100 legacy=0x7f91a00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 -- 172.21.15.104:0/3090355544 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a006d4c0 con 0x7f91a006e0e0 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a615f700 1 --2- 172.21.15.104:0/3090355544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 0x7f91a006cd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9190009820 tx=0x7f9190009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b733483432e9173 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.053+0000 7f91a515d700 1 -- 172.21.15.104:0/3090355544 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190005e30 con 0x7f91a006e0e0 2023-06-10T18:56:47.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a515d700 1 -- 172.21.15.104:0/3090355544 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190006030 con 0x7f91a006e0e0 2023-06-10T18:56:47.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a515d700 1 -- 172.21.15.104:0/3090355544 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900063c0 con 0x7f91a006e0e0 2023-06-10T18:56:47.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 -- 172.21.15.104:0/3090355544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 msgr2=0x7f91a006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 --2- 172.21.15.104:0/3090355544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 0x7f91a006cd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9190009820 tx=0x7f9190009b30 comp rx=0 tx=0).stop 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 -- 172.21.15.104:0/3090355544 shutdown_connections 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 --2- 172.21.15.104:0/3090355544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a006e0e0 0x7f91a006cd50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 --2- 172.21.15.104:0/3090355544 >> v2:172.21.15.114:3300/0 conn(0x7f91a006e4b0 0x7f91a00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 -- 172.21.15.104:0/3090355544 >> 172.21.15.104:0/3090355544 conn(0x7f91a006b8a0 msgr2=0x7f91a00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:47.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 -- 172.21.15.104:0/3090355544 shutdown_connections 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.054+0000 7f91a83c3700 1 -- 172.21.15.104:0/3090355544 wait complete. 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.055+0000 7f91a83c3700 1 Processor -- start 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.055+0000 7f91a83c3700 1 -- start start 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a006e0e0 0x7f91a014ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a00ca100 0x7f91a00c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01504c0 con 0x7f91a006e4b0 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00b2fd0 con 0x7f91a006e0e0 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a83c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00c8eb0 con 0x7f91a00ca100 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a595e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a595e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37596/0 (socket says 172.21.15.104:37596) 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a595e700 1 -- 172.21.15.104:0/4220393705 learned_addr learned my addr 172.21.15.104:0/4220393705 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a615f700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a006e0e0 0x7f91a014ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a6960700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a00ca100 0x7f91a00c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a595e700 1 -- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a00ca100 msgr2=0x7f91a00c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.056+0000 7f91a595e700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a00ca100 0x7f91a00c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a595e700 1 -- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a006e0e0 msgr2=0x7f91a014ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a595e700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a006e0e0 0x7f91a014ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a595e700 1 -- 172.21.15.104:0/4220393705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190009430 con 0x7f91a006e4b0 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a595e700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e9b0 tx=0x7f919c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c009710 con 0x7f91a006e4b0 2023-06-10T18:56:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c00ee50 con 0x7f91a006e4b0 2023-06-10T18:56:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a00c7ba0 con 0x7f91a006e4b0 2023-06-10T18:56:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c00ca50 con 0x7f91a006e4b0 2023-06-10T18:56:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.057+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a00c8160 con 0x7f91a006e4b0 2023-06-10T18:56:47.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.058+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a00c8380 con 0x7f91a006e4b0 2023-06-10T18:56:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.058+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f919c00cbb0 con 0x7f91a006e4b0 2023-06-10T18:56:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.059+0000 7f9196ffd700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 0x7f918406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.059+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f919c0895e0 con 0x7f91a006e4b0 2023-06-10T18:56:47.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.059+0000 7f91a615f700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 0x7f918406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.059+0000 7f91a615f700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 0x7f918406d190 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9190009b70 tx=0x7f9190008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.063+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f919c057fb0 con 0x7f91a006e4b0 2023-06-10T18:56:47.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.265+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f91a0074050 con 0x7f91a006e4b0 2023-06-10T18:56:47.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.265+0000 7f9196ffd700 1 -- 172.21.15.104:0/4220393705 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v129) v1 ==== 80+0+57 (secure 0 0 0) 0x7f919c00ce60 con 0x7f91a006e4b0 2023-06-10T18:56:47.272 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/3796851905 2023-06-10T19:56:41.727300+0000 2023-06-10T18:56:47.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 msgr2=0x7f918406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 0x7f918406d190 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9190009b70 tx=0x7f9190008070 comp rx=0 tx=0).stop 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 msgr2=0x7f91a00c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e9b0 tx=0x7f919c00ecc0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 shutdown_connections 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f918406ace0 0x7f918406d190 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a006e0e0 0x7f91a014ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91a00ca100 0x7f91a00c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 --2- 172.21.15.104:0/4220393705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91a006e4b0 0x7f91a00c7020 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 >> 172.21.15.104:0/4220393705 conn(0x7f91a006b8a0 msgr2=0x7f91a00adaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:47.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 shutdown_connections 2023-06-10T18:56:47.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.267+0000 7f91a83c3700 1 -- 172.21.15.104:0/4220393705 wait complete. 2023-06-10T18:56:47.275 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:56:47.285 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T18:56:47.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:47.286 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T18:56:47.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[148724]: fuse finished with error 0 and tester_r 0 2023-06-10T18:56:47.321 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:56:47.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:47.322 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:56:47.487 INFO:tasks.cephfs.mount:Terminating background process 2023-06-10T18:56:47.487 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:47.488 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:47.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:47.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.640+0000 7f921ba85700 1 Processor -- start 2023-06-10T18:56:47.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.640+0000 7f921ba85700 1 -- start start 2023-06-10T18:56:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921ba85700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f92140b4ff0 0x7f92140ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921ba85700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92140ce890 0x7f921415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921ba85700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140b6070 con 0x7f921415a420 2023-06-10T18:56:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921ba85700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92140b1d70 con 0x7f92140ce890 2023-06-10T18:56:47.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921ba85700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92140cecb0 con 0x7f92140b4ff0 2023-06-10T18:56:47.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92140ce890 0x7f921415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f92140ce890 0x7f921415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37606/0 (socket says 172.21.15.104:37606) 2023-06-10T18:56:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 -- 172.21.15.104:0/1908831721 learned_addr learned my addr 172.21.15.104:0/1908831721 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f9219020700 1 --2- 172.21.15.104:0/1908831721 >> v2:172.21.15.114:3300/0 conn(0x7f92140b4ff0 0x7f92140ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 -- 172.21.15.104:0/1908831721 >> v2:172.21.15.114:3300/0 conn(0x7f92140b4ff0 msgr2=0x7f92140ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 --2- 172.21.15.104:0/1908831721 >> v2:172.21.15.114:3300/0 conn(0x7f92140b4ff0 0x7f92140ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 -- 172.21.15.104:0/1908831721 >> v1:172.21.15.114:6789/0 conn(0x7f921415a420 legacy=0x7f92140b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 -- 172.21.15.104:0/1908831721 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140b5460 con 0x7f92140ce890 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.641+0000 7f921a022700 1 --2- 172.21.15.104:0/1908831721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140ce890 0x7f921415a7f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f9214066590 tx=0x7f920c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5193e83644c7587 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921881f700 1 -- 172.21.15.104:0/1908831721 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00e070 con 0x7f92140ce890 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921881f700 1 -- 172.21.15.104:0/1908831721 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c003a20 con 0x7f92140ce890 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921881f700 1 -- 172.21.15.104:0/1908831721 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c003dd0 con 0x7f92140ce890 2023-06-10T18:56:47.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 -- 172.21.15.104:0/1908831721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140ce890 msgr2=0x7f921415a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 --2- 172.21.15.104:0/1908831721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140ce890 0x7f921415a7f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f9214066590 tx=0x7f920c00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 -- 172.21.15.104:0/1908831721 shutdown_connections 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 --2- 172.21.15.104:0/1908831721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140ce890 0x7f921415a7f0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 --2- 172.21.15.104:0/1908831721 >> v2:172.21.15.114:3300/0 conn(0x7f92140b4ff0 0x7f92140ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 -- 172.21.15.104:0/1908831721 >> 172.21.15.104:0/1908831721 conn(0x7f9214089fd0 msgr2=0x7f92140d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 -- 172.21.15.104:0/1908831721 shutdown_connections 2023-06-10T18:56:47.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.642+0000 7f921ba85700 1 -- 172.21.15.104:0/1908831721 wait complete. 2023-06-10T18:56:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.644+0000 7f921ba85700 1 Processor -- start 2023-06-10T18:56:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.644+0000 7f921ba85700 1 -- start start 2023-06-10T18:56:47.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92140ce890 0x7f92141f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f921415a420 0x7f921415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92141f1520 con 0x7f92140b4ff0 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921414d6c0 con 0x7f921415a420 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921414d9a0 con 0x7f92140ce890 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37608/0 (socket says 172.21.15.104:37608) 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 -- 172.21.15.104:0/634557385 learned_addr learned my addr 172.21.15.104:0/634557385 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 -- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92140ce890 msgr2=0x7f92141f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92140ce890 0x7f92141f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 -- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f921415a420 msgr2=0x7f921415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921a022700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f921415a420 0x7f921415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f921415a420 0x7f921415c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 -- 172.21.15.104:0/634557385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c009b30 con 0x7f92140b4ff0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f9219821700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f921000c4d0 tx=0x7f921000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210011790 con 0x7f92140b4ff0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92100118f0 con 0x7f92140b4ff0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921414dc80 con 0x7f92140b4ff0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210011ca0 con 0x7f92140b4ff0 2023-06-10T18:56:47.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.645+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921414e240 con 0x7f92140b4ff0 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.646+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9214074050 con 0x7f92140b4ff0 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.647+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f921001c460 con 0x7f92140b4ff0 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.647+0000 7f920a7fc700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 0x7f91f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.647+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9210089430 con 0x7f92140b4ff0 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.648+0000 7f9219020700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 0x7f91f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.648+0000 7f9219020700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 0x7f91f806d190 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f921415d110 tx=0x7f9204008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.652+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9210054370 con 0x7f92140b4ff0 2023-06-10T18:56:47.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.842+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f921414e400 con 0x7f92140b4ff0 2023-06-10T18:56:47.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.843+0000 7f920a7fc700 1 -- 172.21.15.104:0/634557385 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+18674 (secure 0 0 0) 0x7f9210057990 con 0x7f92140b4ff0 2023-06-10T18:56:47.849 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:47.850 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":129,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:42.706404+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3796851905":"2023-06-10T19:56:41.727300+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 msgr2=0x7f91f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 0x7f91f806d190 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f921415d110 tx=0x7f9204008040 comp rx=0 tx=0).stop 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 msgr2=0x7f92141f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f921000c4d0 tx=0x7f921000c890 comp rx=0 tx=0).stop 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 shutdown_connections 2023-06-10T18:56:47.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f91f806ace0 0x7f91f806d190 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f921415a420 0x7f921415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92140b4ff0 0x7f92141f0fe0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 --2- 172.21.15.104:0/634557385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92140ce890 0x7f92141f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 >> 172.21.15.104:0/634557385 conn(0x7f9214089fd0 msgr2=0x7f9214157c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 shutdown_connections 2023-06-10T18:56:47.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.845+0000 7f921ba85700 1 -- 172.21.15.104:0/634557385 wait complete. 2023-06-10T18:56:47.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:56:17.300478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:56:18.192765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:47.864 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:47.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 Processor -- start 2023-06-10T18:56:47.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 -- start start 2023-06-10T18:56:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907c15c580 0x7f907c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06e6e0 0x7f907c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c06e250 con 0x7f907c06cde0 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.982+0000 7f90823fb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c15c950 con 0x7f907c15c580 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f90823fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0cf0e0 con 0x7f907c06e6e0 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907c15c580 0x7f907c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907c15c580 0x7f907c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37616/0 (socket says 172.21.15.104:37616) 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 -- 172.21.15.104:0/3122512267 learned_addr learned my addr 172.21.15.104:0/3122512267 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907b7fe700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06e6e0 0x7f907c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 -- 172.21.15.104:0/3122512267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06e6e0 msgr2=0x7f907c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06e6e0 0x7f907c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 -- 172.21.15.104:0/3122512267 >> v1:172.21.15.114:6789/0 conn(0x7f907c06cde0 legacy=0x7f907c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 -- 172.21.15.104:0/3122512267 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0b1e80 con 0x7f907c15c580 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907bfff700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c15c580 0x7f907c0b1070 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f906c006fd0 tx=0x7f906c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884c769e06cf7b06 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907affd700 1 -- 172.21.15.104:0/3122512267 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c005e30 con 0x7f907c15c580 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907affd700 1 -- 172.21.15.104:0/3122512267 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c006030 con 0x7f907c15c580 2023-06-10T18:56:47.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.983+0000 7f907affd700 1 -- 172.21.15.104:0/3122512267 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0063c0 con 0x7f907c15c580 2023-06-10T18:56:47.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 -- 172.21.15.104:0/3122512267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c15c580 msgr2=0x7f907c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c15c580 0x7f907c0b1070 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f906c006fd0 tx=0x7f906c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 -- 172.21.15.104:0/3122512267 shutdown_connections 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c15c580 0x7f907c0b1070 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 --2- 172.21.15.104:0/3122512267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06e6e0 0x7f907c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 -- 172.21.15.104:0/3122512267 >> 172.21.15.104:0/3122512267 conn(0x7f907c06b8a0 msgr2=0x7f907c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 -- 172.21.15.104:0/3122512267 shutdown_connections 2023-06-10T18:56:47.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.984+0000 7f90823fb700 1 -- 172.21.15.104:0/3122512267 wait complete. 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.985+0000 7f90823fb700 1 Processor -- start 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 -- start start 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06cde0 0x7f907c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907c15c580 0x7f907c157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c1f2a10 con 0x7f907c06e6e0 2023-06-10T18:56:47.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0ad110 con 0x7f907c15c580 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.986+0000 7f90823fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c1f1e40 con 0x7f907c06cde0 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37624/0 (socket says 172.21.15.104:37624) 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 -- 172.21.15.104:0/168784275 learned_addr learned my addr 172.21.15.104:0/168784275 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907bfff700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06cde0 0x7f907c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f9080998700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907c15c580 0x7f907c157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 -- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06cde0 msgr2=0x7f907c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06cde0 0x7f907c0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 -- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907c15c580 msgr2=0x7f907c157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907c15c580 0x7f907c157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 -- 172.21.15.104:0/168784275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c009500 con 0x7f907c06e6e0 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f907b7fe700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f907000d830 tx=0x7f907000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700125f0 con 0x7f907c06e6e0 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070012750 con 0x7f907c06e6e0 2023-06-10T18:56:47.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070012ae0 con 0x7f907c06e6e0 2023-06-10T18:56:47.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c1f20e0 con 0x7f907c06e6e0 2023-06-10T18:56:47.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.987+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c1f26a0 con 0x7f907c06e6e0 2023-06-10T18:56:47.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.988+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c074050 con 0x7f907c06e6e0 2023-06-10T18:56:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.989+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9070012c40 con 0x7f907c06e6e0 2023-06-10T18:56:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.989+0000 7f9078ff9700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 0x7f906806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.989+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f90700895f0 con 0x7f907c06e6e0 2023-06-10T18:56:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.989+0000 7f907bfff700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 0x7f906806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.990+0000 7f907bfff700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 0x7f906806d190 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f906c013040 tx=0x7f906c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:47.993+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9070054530 con 0x7f907c06e6e0 2023-06-10T18:56:48.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.177+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f907c072f90 con 0x7f907c06e6e0 2023-06-10T18:56:48.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.178+0000 7f9078ff9700 1 -- 172.21.15.104:0/168784275 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9070057b50 con 0x7f907c06e6e0 2023-06-10T18:56:48.185 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:48.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 msgr2=0x7f906806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 0x7f906806d190 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f906c013040 tx=0x7f906c008070 comp rx=0 tx=0).stop 2023-06-10T18:56:48.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 msgr2=0x7f907c0acbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f907000d830 tx=0x7f907000db40 comp rx=0 tx=0).stop 2023-06-10T18:56:48.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 shutdown_connections 2023-06-10T18:56:48.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f906806ace0 0x7f906806d190 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907c15c580 0x7f907c157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907c06cde0 0x7f907c0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 --2- 172.21.15.104:0/168784275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907c06e6e0 0x7f907c0acbd0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 >> 172.21.15.104:0/168784275 conn(0x7f907c06b8a0 msgr2=0x7f907c158f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:48.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 shutdown_connections 2023-06-10T18:56:48.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.180+0000 7f90823fb700 1 -- 172.21.15.104:0/168784275 wait complete. 2023-06-10T18:56:48.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:48.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.315+0000 7f8d62bdc700 1 Processor -- start 2023-06-10T18:56:48.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.315+0000 7f8d62bdc700 1 -- start start 2023-06-10T18:56:48.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d62bdc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d5c069e70 0x7f8d5c06a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d62bdc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d62bdc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c06e690 con 0x7f8d5c15c500 2023-06-10T18:56:48.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d62bdc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c0aef20 con 0x7f8d5c069e70 2023-06-10T18:56:48.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d62bdc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c15c8d0 con 0x7f8d5c06dcd0 2023-06-10T18:56:48.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8d5c069e70 0x7f8d5c06a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d60978700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8d5c15c500 0x7f8d5c0cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35458/0 (socket says 172.21.15.104:35458) 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d60978700 1 -- 172.21.15.104:0/747473658 learned_addr learned my addr 172.21.15.104:0/747473658 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d61179700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 -- 172.21.15.104:0/747473658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 msgr2=0x7f8d5c06e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c06e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 -- 172.21.15.104:0/747473658 >> v1:172.21.15.104:6789/0 conn(0x7f8d5c15c500 legacy=0x7f8d5c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5b7fe700 1 -- 172.21.15.104:0/747473658 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511377794 0 0) 0x7f8d5c06e690 con 0x7f8d5c15c500 2023-06-10T18:56:48.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 -- 172.21.15.104:0/747473658 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c06abd0 con 0x7f8d5c069e70 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5bfff700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c06a310 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f8d48009870 tx=0x7f8d48009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397bb687ec2b1993 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5b7fe700 1 -- 172.21.15.104:0/747473658 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48011830 con 0x7f8d5c069e70 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.316+0000 7f8d5b7fe700 1 -- 172.21.15.104:0/747473658 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d48011990 con 0x7f8d5c069e70 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d5b7fe700 1 -- 172.21.15.104:0/747473658 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48011d20 con 0x7f8d5c069e70 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 -- 172.21.15.104:0/747473658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 msgr2=0x7f8d5c06a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c06a310 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f8d48009870 tx=0x7f8d48009b80 comp rx=0 tx=0).stop 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 -- 172.21.15.104:0/747473658 shutdown_connections 2023-06-10T18:56:48.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c06e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/747473658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c06a310 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 -- 172.21.15.104:0/747473658 >> 172.21.15.104:0/747473658 conn(0x7f8d5c06b8a0 msgr2=0x7f8d5c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:48.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 -- 172.21.15.104:0/747473658 shutdown_connections 2023-06-10T18:56:48.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.317+0000 7f8d62bdc700 1 -- 172.21.15.104:0/747473658 wait complete. 2023-06-10T18:56:48.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.319+0000 7f8d62bdc700 1 Processor -- start 2023-06-10T18:56:48.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.319+0000 7f8d62bdc700 1 -- start start 2023-06-10T18:56:48.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5c15c500 0x7f8d5c0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c8050 con 0x7f8d5c069e70 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c4250 con 0x7f8d5c15c500 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c4530 con 0x7f8d5c06dcd0 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37636/0 (socket says 172.21.15.104:37636) 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 -- 172.21.15.104:0/1351490604 learned_addr learned my addr 172.21.15.104:0/1351490604 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d5bfff700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d61179700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5c15c500 0x7f8d5c0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 -- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 msgr2=0x7f8d5c0c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c0c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 -- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5c15c500 msgr2=0x7f8d5c0c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5c15c500 0x7f8d5c0c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 -- 172.21.15.104:0/1351490604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480094a0 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d60978700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c700 tx=0x7f8d5000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500074e0 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d50007640 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50005a30 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.320+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c0c7a00 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.321+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c0b4880 con 0x7f8d5c069e70 2023-06-10T18:56:48.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.321+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c074050 con 0x7f8d5c069e70 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.322+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d50005b90 con 0x7f8d5c069e70 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.322+0000 7f8d597fa700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 0x7f8d4006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.322+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8d50088a20 con 0x7f8d5c069e70 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.322+0000 7f8d5bfff700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 0x7f8d4006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.323+0000 7f8d5bfff700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 0x7f8d4006d1c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000c00 tx=0x7f8d4801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.327+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d50053a00 con 0x7f8d5c069e70 2023-06-10T18:56:48.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.512+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8d5c0c2e60 con 0x7f8d5c069e70 2023-06-10T18:56:48.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.513+0000 7f8d597fa700 1 -- 172.21.15.104:0/1351490604 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v129) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8d50017020 con 0x7f8d5c069e70 2023-06-10T18:56:48.519 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:48.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 msgr2=0x7f8d4006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 0x7f8d4006d1c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8d48000c00 tx=0x7f8d4801c040 comp rx=0 tx=0).stop 2023-06-10T18:56:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 msgr2=0x7f8d5c14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c700 tx=0x7f8d5000cac0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 shutdown_connections 2023-06-10T18:56:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d4006ad10 0x7f8d4006d1c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5c15c500 0x7f8d5c0c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d5c069e70 0x7f8d5c14b280 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 --2- 172.21.15.104:0/1351490604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d5c06dcd0 0x7f8d5c0c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 >> 172.21.15.104:0/1351490604 conn(0x7f8d5c06b8a0 msgr2=0x7f8d5c0ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 shutdown_connections 2023-06-10T18:56:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.515+0000 7f8d62bdc700 1 -- 172.21.15.104:0/1351490604 wait complete. 2023-06-10T18:56:48.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:56:48.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.650+0000 7f534c7f7700 1 Processor -- start 2023-06-10T18:56:48.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.650+0000 7f534c7f7700 1 -- start start 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.651+0000 7f534c7f7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 0x7f534406e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.651+0000 7f534c7f7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f534406c800 0x7f534406cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.651+0000 7f534c7f7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53440cec40 con 0x7f5344143a40 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.651+0000 7f534c7f7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440cca90 con 0x7f534406c800 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.651+0000 7f534c7f7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53440ce960 con 0x7f5344070a10 2023-06-10T18:56:48.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f534406c800 0x7f534406cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349d92700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5344143a40 0x7f5344070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35472/0 (socket says 172.21.15.104:35472) 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f534406c800 0x7f534406cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37648/0 (socket says 172.21.15.104:37648) 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349d92700 1 -- 172.21.15.104:0/878579418 learned_addr learned my addr 172.21.15.104:0/878579418 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534a593700 1 --2- 172.21.15.104:0/878579418 >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 0x7f534406e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 -- 172.21.15.104:0/878579418 >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 msgr2=0x7f534406e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:48.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 --2- 172.21.15.104:0/878579418 >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 0x7f534406e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 -- 172.21.15.104:0/878579418 >> v1:172.21.15.104:6789/0 conn(0x7f5344143a40 legacy=0x7f5344070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349591700 1 -- 172.21.15.104:0/878579418 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484849377 0 0) 0x7f53440cec40 con 0x7f5344143a40 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 -- 172.21.15.104:0/878579418 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534406a9b0 con 0x7f534406c800 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534a593700 1 --2- 172.21.15.104:0/878579418 >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 0x7f534406e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f534ad94700 1 --2- 172.21.15.104:0/878579418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f534406c800 0x7f534406cc40 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f5344070810 tx=0x7f533c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642eef73fa1b428a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349591700 1 -- 172.21.15.104:0/878579418 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c00e070 con 0x7f534406c800 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349591700 1 -- 172.21.15.104:0/878579418 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c00bbd0 con 0x7f534406c800 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.652+0000 7f5349591700 1 -- 172.21.15.104:0/878579418 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c012430 con 0x7f534406c800 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 -- 172.21.15.104:0/878579418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f534406c800 msgr2=0x7f534406cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 --2- 172.21.15.104:0/878579418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f534406c800 0x7f534406cc40 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f5344070810 tx=0x7f533c00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 -- 172.21.15.104:0/878579418 shutdown_connections 2023-06-10T18:56:48.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 --2- 172.21.15.104:0/878579418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f534406c800 0x7f534406cc40 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 --2- 172.21.15.104:0/878579418 >> v2:172.21.15.114:3300/0 conn(0x7f5344070a10 0x7f534406e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 -- 172.21.15.104:0/878579418 >> 172.21.15.104:0/878579418 conn(0x7f53440ceec0 msgr2=0x7f53440cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:48.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 -- 172.21.15.104:0/878579418 shutdown_connections 2023-06-10T18:56:48.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.653+0000 7f534c7f7700 1 -- 172.21.15.104:0/878579418 wait complete. 2023-06-10T18:56:48.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.654+0000 7f534c7f7700 1 Processor -- start 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.654+0000 7f534c7f7700 1 -- start start 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f534406c800 0x7f534413ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 0x7f53440c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53440c7f40 con 0x7f5344143a40 2023-06-10T18:56:48.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53441596c0 con 0x7f5344070a10 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5344159960 con 0x7f534406c800 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37654/0 (socket says 172.21.15.104:37654) 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 -- 172.21.15.104:0/4176853146 learned_addr learned my addr 172.21.15.104:0/4176853146 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f5349d92700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 0x7f53440c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534a593700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f534406c800 0x7f534413ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 -- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f534406c800 msgr2=0x7f534413ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f534406c800 0x7f534413ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 -- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 msgr2=0x7f53440c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 0x7f53440c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 -- 172.21.15.104:0/4176853146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c009a70 con 0x7f5344143a40 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f5349d92700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 0x7f53440c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534ad94700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f533c00b370 tx=0x7f533c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c012430 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c020070 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53440cd0a0 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c004280 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.655+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53440c7af0 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.656+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440c7d10 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.657+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f533c0043e0 con 0x7f5344143a40 2023-06-10T18:56:48.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.657+0000 7f533affd700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 0x7f532806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.657+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f533c022070 con 0x7f5344143a40 2023-06-10T18:56:48.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.657+0000 7f534a593700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 0x7f532806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.658+0000 7f534a593700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 0x7f532806d190 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5334006fd0 tx=0x7f5334008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.661+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f53440c7d10 con 0x7f5344143a40 2023-06-10T18:56:48.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.848+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53440c7d10 con 0x7f5344143a40 2023-06-10T18:56:48.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.848+0000 7f533affd700 1 -- 172.21.15.104:0/4176853146 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f53440c7d10 con 0x7f5344143a40 2023-06-10T18:56:48.855 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:56:48.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 msgr2=0x7f532806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 0x7f532806d190 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5334006fd0 tx=0x7f5334008040 comp rx=0 tx=0).stop 2023-06-10T18:56:48.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 msgr2=0x7f53440b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f533c00b370 tx=0x7f533c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 shutdown_connections 2023-06-10T18:56:48.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5344143a40 0x7f53440b1df0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f534406c800 0x7f534413ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f532806ace0 0x7f532806d190 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 --2- 172.21.15.104:0/4176853146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5344070a10 0x7f53440c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 >> 172.21.15.104:0/4176853146 conn(0x7f53440ceec0 msgr2=0x7f53440cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 shutdown_connections 2023-06-10T18:56:48.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.850+0000 7f534c7f7700 1 -- 172.21.15.104:0/4176853146 wait complete. 2023-06-10T18:56:48.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:56:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.985+0000 7f20ade05700 1 Processor -- start 2023-06-10T18:56:48.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.985+0000 7f20ade05700 1 -- start start 2023-06-10T18:56:48.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20ade05700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20ade05700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f20a806d4f0 0x7f20a80ca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20ade05700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80d0480 con 0x7f20a806a3b0 2023-06-10T18:56:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20ade05700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a806a800 con 0x7f20a806d4f0 2023-06-10T18:56:48.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20ade05700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a815dd00 con 0x7f20a80aec40 2023-06-10T18:56:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f20a806d4f0 0x7f20a80ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f20a806d4f0 0x7f20a80ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37662/0 (socket says 172.21.15.104:37662) 2023-06-10T18:56:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 -- 172.21.15.104:0/2488071916 learned_addr learned my addr 172.21.15.104:0/2488071916 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:48.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a77fe700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 -- 172.21.15.104:0/2488071916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 msgr2=0x7f20a8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a8069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 -- 172.21.15.104:0/2488071916 >> v1:172.21.15.114:6789/0 conn(0x7f20a806a3b0 legacy=0x7f20a806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:48.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 -- 172.21.15.104:0/2488071916 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a815a610 con 0x7f20a806d4f0 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a7fff700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806d4f0 0x7f20a80ca850 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009e50 tx=0x7f20a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfda172b4f15451 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a67fc700 1 -- 172.21.15.104:0/2488071916 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000e070 con 0x7f20a806d4f0 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a67fc700 1 -- 172.21.15.104:0/2488071916 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0003a20 con 0x7f20a806d4f0 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.986+0000 7f20a67fc700 1 -- 172.21.15.104:0/2488071916 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0003dd0 con 0x7f20a806d4f0 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 -- 172.21.15.104:0/2488071916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806d4f0 msgr2=0x7f20a80ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:48.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806d4f0 0x7f20a80ca850 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009e50 tx=0x7f20a000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:48.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 -- 172.21.15.104:0/2488071916 shutdown_connections 2023-06-10T18:56:48.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806d4f0 0x7f20a80ca850 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:48.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 --2- 172.21.15.104:0/2488071916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a8069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 -- 172.21.15.104:0/2488071916 >> 172.21.15.104:0/2488071916 conn(0x7f20a806b8a0 msgr2=0x7f20a80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:49.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 -- 172.21.15.104:0/2488071916 shutdown_connections 2023-06-10T18:56:49.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.987+0000 7f20ade05700 1 -- 172.21.15.104:0/2488071916 wait complete. 2023-06-10T18:56:49.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.989+0000 7f20ade05700 1 Processor -- start 2023-06-10T18:56:49.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.989+0000 7f20ade05700 1 -- start start 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a806d4f0 0x7f20a80cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a80c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81f2600 con 0x7f20a806a3b0 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81f35d0 con 0x7f20a806d4f0 2023-06-10T18:56:49.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c9310 con 0x7f20a80aec40 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37664/0 (socket says 172.21.15.104:37664) 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 -- 172.21.15.104:0/3149881093 learned_addr learned my addr 172.21.15.104:0/3149881093 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a7fff700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a80c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a6ffd700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a806d4f0 0x7f20a80cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 -- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 msgr2=0x7f20a80c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a80c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 -- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a806d4f0 msgr2=0x7f20a80cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a806d4f0 0x7f20a80cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 -- 172.21.15.104:0/3149881093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0009b30 con 0x7f20a806a3b0 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20a77fe700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c7f0 tx=0x7f209c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c011800 con 0x7f20a806a3b0 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c011960 con 0x7f20a806a3b0 2023-06-10T18:56:49.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a81f2810 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.990+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c011d10 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.991+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a81f2dd0 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.991+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a8074050 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.992+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f209c018360 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.992+0000 7f208ffff700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 0x7f209406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.992+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f209c0895c0 con 0x7f20a806a3b0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.992+0000 7f20a6ffd700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 0x7f209406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.993+0000 7f20a6ffd700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 0x7f209406d190 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f20a80ae120 tx=0x7f2098008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:49.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:48.996+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f209c057fa0 con 0x7f20a806a3b0 2023-06-10T18:56:49.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.182+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f20a80c9890 con 0x7f20a806a3b0 2023-06-10T18:56:49.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.182+0000 7f208ffff700 1 -- 172.21.15.104:0/3149881093 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7f209c057b30 con 0x7f20a806a3b0 2023-06-10T18:56:49.189 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:49.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 msgr2=0x7f209406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 0x7f209406d190 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f20a80ae120 tx=0x7f2098008040 comp rx=0 tx=0).stop 2023-06-10T18:56:49.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 msgr2=0x7f20a80cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c7f0 tx=0x7f209c00cbb0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 shutdown_connections 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20a80aec40 0x7f20a80c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f20a806a3b0 0x7f20a80cbde0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f209406ace0 0x7f209406d190 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 --2- 172.21.15.104:0/3149881093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a806d4f0 0x7f20a80cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.184+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 >> 172.21.15.104:0/3149881093 conn(0x7f20a806b8a0 msgr2=0x7f20a8155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.185+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 shutdown_connections 2023-06-10T18:56:49.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.185+0000 7f20ade05700 1 -- 172.21.15.104:0/3149881093 wait complete. 2023-06-10T18:56:49.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:56:49.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.317+0000 7f8c15fc9700 1 Processor -- start 2023-06-10T18:56:49.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.317+0000 7f8c15fc9700 1 -- start start 2023-06-10T18:56:49.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c15fc9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c1006e0e0 0x7f8c1006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c15fc9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c10069e70 0x7f8c100ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c15fc9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1006d290 con 0x7f8c1006e4b0 2023-06-10T18:56:49.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c15fc9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c10071640 con 0x7f8c1006e0e0 2023-06-10T18:56:49.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c15fc9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c100aef00 con 0x7f8c10069e70 2023-06-10T18:56:49.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c1006e0e0 0x7f8c1006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c1006e0e0 0x7f8c1006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37672/0 (socket says 172.21.15.104:37672) 2023-06-10T18:56:49.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 -- 172.21.15.104:0/179947456 learned_addr learned my addr 172.21.15.104:0/179947456 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:49.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0ffff700 1 --2- 172.21.15.104:0/179947456 >> v2:172.21.15.114:3300/0 conn(0x7f8c10069e70 0x7f8c100ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 -- 172.21.15.104:0/179947456 >> v2:172.21.15.114:3300/0 conn(0x7f8c10069e70 msgr2=0x7f8c100ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/179947456 >> v2:172.21.15.114:3300/0 conn(0x7f8c10069e70 0x7f8c100ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 -- 172.21.15.104:0/179947456 >> v1:172.21.15.114:6789/0 conn(0x7f8c1006e4b0 legacy=0x7f8c100c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.318+0000 7f8c0f7fe700 1 -- 172.21.15.104:0/179947456 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c100ca2d0 con 0x7f8c1006e0e0 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/179947456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c1006cd50 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8c10071210 tx=0x7f8c00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c610f1c7eca29778 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c0e7fc700 1 -- 172.21.15.104:0/179947456 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00005e30 con 0x7f8c1006e0e0 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c0e7fc700 1 -- 172.21.15.104:0/179947456 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00004030 con 0x7f8c1006e0e0 2023-06-10T18:56:49.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c0e7fc700 1 -- 172.21.15.104:0/179947456 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000043e0 con 0x7f8c1006e0e0 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 -- 172.21.15.104:0/179947456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 msgr2=0x7f8c1006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/179947456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c1006cd50 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8c10071210 tx=0x7f8c00009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 -- 172.21.15.104:0/179947456 shutdown_connections 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/179947456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c1006cd50 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/179947456 >> v2:172.21.15.114:3300/0 conn(0x7f8c10069e70 0x7f8c100ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 -- 172.21.15.104:0/179947456 >> 172.21.15.104:0/179947456 conn(0x7f8c1006b8a0 msgr2=0x7f8c100cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:49.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 -- 172.21.15.104:0/179947456 shutdown_connections 2023-06-10T18:56:49.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.319+0000 7f8c15fc9700 1 -- 172.21.15.104:0/179947456 wait complete. 2023-06-10T18:56:49.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.321+0000 7f8c15fc9700 1 Processor -- start 2023-06-10T18:56:49.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.321+0000 7f8c15fc9700 1 -- start start 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 0x7f8c101470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c1006e4b0 0x7f8c1009c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1015b160 con 0x7f8c1006e0e0 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1009c4c0 con 0x7f8c10069e70 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c15fc9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1009b610 con 0x7f8c1006e4b0 2023-06-10T18:56:49.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37678/0 (socket says 172.21.15.104:37678) 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0effd700 1 -- 172.21.15.104:0/4059994049 learned_addr learned my addr 172.21.15.104:0/4059994049 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 0x7f8c101470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0ffff700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c1006e4b0 0x7f8c1009c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.322+0000 7f8c0effd700 1 -- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c1006e4b0 msgr2=0x7f8c1009c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0effd700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c1006e4b0 0x7f8c1009c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0effd700 1 -- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 msgr2=0x7f8c101470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0effd700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 0x7f8c101470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0effd700 1 -- 172.21.15.104:0/4059994049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009500 con 0x7f8c1006e0e0 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 0x7f8c101470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c0effd700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c4e0 tx=0x7f8c0400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040052d0 con 0x7f8c1006e0e0 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04005430 con 0x7f8c1006e0e0 2023-06-10T18:56:49.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040038c0 con 0x7f8c1006e0e0 2023-06-10T18:56:49.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1009b8b0 con 0x7f8c1006e0e0 2023-06-10T18:56:49.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.323+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1009be70 con 0x7f8c1006e0e0 2023-06-10T18:56:49.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.324+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c10074050 con 0x7f8c1006e0e0 2023-06-10T18:56:49.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.324+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c04003a20 con 0x7f8c1006e0e0 2023-06-10T18:56:49.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.325+0000 7f8bf7fff700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 0x7f8bfc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:49.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.325+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8c04088b20 con 0x7f8c1006e0e0 2023-06-10T18:56:49.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.325+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 0x7f8bfc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:49.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.325+0000 7f8c0f7fe700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 0x7f8bfc06d190 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8c00014040 tx=0x7f8c00008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:49.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.329+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c04053a50 con 0x7f8c1006e0e0 2023-06-10T18:56:49.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.529+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8c1009c600 con 0x7f8c1006e0e0 2023-06-10T18:56:49.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.529+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4059994049 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v129) v1 ==== 80+0+57 (secure 0 0 0) 0x7f8c04017050 con 0x7f8c1006e0e0 2023-06-10T18:56:49.536 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/3796851905 2023-06-10T19:56:41.727300+0000 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.531+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 msgr2=0x7f8bfc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.531+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 0x7f8bfc06d190 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8c00014040 tx=0x7f8c00008070 comp rx=0 tx=0).stop 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.531+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 msgr2=0x7f8c10099d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.531+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c4e0 tx=0x7f8c0400c8a0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 shutdown_connections 2023-06-10T18:56:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bfc06ace0 0x7f8bfc06d190 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c10069e70 0x7f8c101470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c1006e4b0 0x7f8c1009c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 --2- 172.21.15.104:0/4059994049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c1006e0e0 0x7f8c10099d50 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 >> 172.21.15.104:0/4059994049 conn(0x7f8c1006b8a0 msgr2=0x7f8c1006a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 shutdown_connections 2023-06-10T18:56:49.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.532+0000 7f8c15fc9700 1 -- 172.21.15.104:0/4059994049 wait complete. 2023-06-10T18:56:49.540 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:56:49.549 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:56:49.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:49.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:56:49.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[114930]: fuse finished with error 0 and tester_r 0 2023-06-10T18:56:49.592 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:56:49.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:49.593 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:56:49.728 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:56:49.729 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:49.746 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:56:49.747 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:56:49.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:49.747 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:56:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:56:49.804 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:56:49.804 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:56:49.805 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:56:49.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:56:49.824 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:56:49.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:56:49.824 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:56:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:56:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:56:49.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 Processor -- start 2023-06-10T18:56:50.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 -- start start 2023-06-10T18:56:50.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 0x7fe92406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92406d290 con 0x7fe92406e0e0 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe924071640 con 0x7fe92406e4b0 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.998+0000 7fe92a0a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240aef00 con 0x7fe924069e70 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe922ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 0x7fe92406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe922ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 0x7fe92406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:53420/0 (socket says 172.21.15.104:53420) 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe9237fe700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe92406e0e0 0x7fe9240c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41358/0 (socket says 172.21.15.104:41358) 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:57072/0 (socket says 172.21.15.104:57072) 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe922ffd700 1 -- 172.21.15.104:0/3532485072 learned_addr learned my addr 172.21.15.104:0/3532485072 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 -- 172.21.15.104:0/3532485072 >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 msgr2=0x7fe92406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 --2- 172.21.15.104:0/3532485072 >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 0x7fe92406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 -- 172.21.15.104:0/3532485072 >> v1:172.21.15.114:6789/0 conn(0x7fe92406e0e0 legacy=0x7fe9240c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 -- 172.21.15.104:0/3532485072 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240ca2d0 con 0x7fe924069e70 2023-06-10T18:56:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe923fff700 1 --2- 172.21.15.104:0/3532485072 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009d90 tx=0x7fe91c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6824fac60705824e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe9227fc700 1 -- 172.21.15.104:0/3532485072 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c00e070 con 0x7fe924069e70 2023-06-10T18:56:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:49.999+0000 7fe9227fc700 1 -- 172.21.15.104:0/3532485072 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c00bb20 con 0x7fe924069e70 2023-06-10T18:56:50.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe9227fc700 1 -- 172.21.15.104:0/3532485072 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c012430 con 0x7fe924069e70 2023-06-10T18:56:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3532485072 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 msgr2=0x7fe9240ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3532485072 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009d90 tx=0x7fe91c00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3532485072 shutdown_connections 2023-06-10T18:56:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3532485072 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe924069e70 0x7fe9240ce9f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3532485072 >> v2:172.21.15.114:3300/0 conn(0x7fe92406e4b0 0x7fe92406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3532485072 >> 172.21.15.104:0/3532485072 conn(0x7fe92406b8a0 msgr2=0x7fe9240cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3532485072 shutdown_connections 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.000+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3532485072 wait complete. 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.001+0000 7fe92a0a4700 1 Processor -- start 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.001+0000 7fe92a0a4700 1 -- start start 2023-06-10T18:56:50.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 0x7fe9240b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 0x7fe9240b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240c7d30 con 0x7fe924069e70 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240b5270 con 0x7fe92406e0e0 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe92a0a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240c7f80 con 0x7fe92406e4b0 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe9237fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe9237fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37690/0 (socket says 172.21.15.104:37690) 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.002+0000 7fe9237fe700 1 -- 172.21.15.104:0/3052974268 learned_addr learned my addr 172.21.15.104:0/3052974268 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe922ffd700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 0x7fe9240b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe923fff700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 0x7fe9240b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 -- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 msgr2=0x7fe9240b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 0x7fe9240b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 -- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 msgr2=0x7fe9240b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 0x7fe9240b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 -- 172.21.15.104:0/3052974268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c009a70 con 0x7fe924069e70 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe922ffd700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 0x7fe9240b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe923fff700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 0x7fe9240b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe9237fe700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c6b0 tx=0x7fe91400ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914005d00 con 0x7fe924069e70 2023-06-10T18:56:50.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914005e60 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914007310 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240c8260 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.003+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9240cb1c0 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.004+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240cb380 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.004+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe914007470 con 0x7fe924069e70 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.005+0000 7fe90bfff700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 0x7fe91006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.005+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fe914088a80 con 0x7fe924069e70 2023-06-10T18:56:50.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.005+0000 7fe922ffd700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 0x7fe91006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.005+0000 7fe922ffd700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 0x7fe91006d190 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe9240cbf70 tx=0x7fe918008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.009+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe914053a50 con 0x7fe924069e70 2023-06-10T18:56:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.238+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe924074050 con 0x7fe924069e70 2023-06-10T18:56:50.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.238+0000 7fe90bfff700 1 -- 172.21.15.104:0/3052974268 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fe914017090 con 0x7fe924069e70 2023-06-10T18:56:50.245 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:50.245 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:50.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 msgr2=0x7fe91006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 0x7fe91006d190 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe9240cbf70 tx=0x7fe918008040 comp rx=0 tx=0).stop 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 msgr2=0x7fe9240cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c6b0 tx=0x7fe91400ca70 comp rx=0 tx=0).stop 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 shutdown_connections 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe92406e4b0 0x7fe9240b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe924069e70 0x7fe9240cac80 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe91006ace0 0x7fe91006d190 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 --2- 172.21.15.104:0/3052974268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe92406e0e0 0x7fe9240b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.240+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 >> 172.21.15.104:0/3052974268 conn(0x7fe92406b8a0 msgr2=0x7fe9240ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.241+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 shutdown_connections 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.241+0000 7fe92a0a4700 1 -- 172.21.15.104:0/3052974268 wait complete. 2023-06-10T18:56:50.248 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:50.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:50.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.374+0000 7fe123b94700 1 Processor -- start 2023-06-10T18:56:50.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.374+0000 7fe123b94700 1 -- start start 2023-06-10T18:56:50.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe123b94700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe11c0aeaa0 0x7fe11c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe123b94700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0cd680 0x7fe11c06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe123b94700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0cdae0 con 0x7fe11c0b1080 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe123b94700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c0c9bc0 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe123b94700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0c9e60 con 0x7fe11c0cd680 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe11c0aeaa0 0x7fe11c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe11c0aeaa0 0x7fe11c0cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37698/0 (socket says 172.21.15.104:37698) 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 -- 172.21.15.104:0/2299891338 learned_addr learned my addr 172.21.15.104:0/2299891338 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe122131700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0cd680 0x7fe11c06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 -- 172.21.15.104:0/2299891338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0cd680 msgr2=0x7fe11c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0cd680 0x7fe11c06dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.375+0000 7fe121930700 1 -- 172.21.15.104:0/2299891338 >> v1:172.21.15.114:6789/0 conn(0x7fe11c0b1080 legacy=0x7fe11c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe121930700 1 -- 172.21.15.104:0/2299891338 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c0aee70 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe121930700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0aeaa0 0x7fe11c0cd140 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe10c006fd0 tx=0x7fe10c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7073d347c69aa668 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe12092e700 1 -- 172.21.15.104:0/2299891338 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c005e30 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe12092e700 1 -- 172.21.15.104:0/2299891338 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c006030 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe12092e700 1 -- 172.21.15.104:0/2299891338 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0063c0 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 -- 172.21.15.104:0/2299891338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0aeaa0 msgr2=0x7fe11c0cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0aeaa0 0x7fe11c0cd140 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe10c006fd0 tx=0x7fe10c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 -- 172.21.15.104:0/2299891338 shutdown_connections 2023-06-10T18:56:50.385 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0aeaa0 0x7fe11c0cd140 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 --2- 172.21.15.104:0/2299891338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0cd680 0x7fe11c06dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 -- 172.21.15.104:0/2299891338 >> 172.21.15.104:0/2299891338 conn(0x7fe11c06b8a0 msgr2=0x7fe11c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 -- 172.21.15.104:0/2299891338 shutdown_connections 2023-06-10T18:56:50.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.376+0000 7fe123b94700 1 -- 172.21.15.104:0/2299891338 wait complete. 2023-06-10T18:56:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 Processor -- start 2023-06-10T18:56:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 -- start start 2023-06-10T18:56:50.388 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0aeaa0 0x7fe11c152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 0x7fe11c0b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c15a660 con 0x7fe11c0cd680 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0b3750 con 0x7fe11c0b1080 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe123b94700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c153410 con 0x7fe11c0aeaa0 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe122131700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe122131700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37708/0 (socket says 172.21.15.104:37708) 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.378+0000 7fe122131700 1 -- 172.21.15.104:0/373273063 learned_addr learned my addr 172.21.15.104:0/373273063 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe12112f700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 0x7fe11c0b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe121930700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0aeaa0 0x7fe11c152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 -- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0aeaa0 msgr2=0x7fe11c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0aeaa0 0x7fe11c152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 -- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 msgr2=0x7fe11c0b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 0x7fe11c0b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 -- 172.21.15.104:0/373273063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c009500 con 0x7fe11c0cd680 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe12112f700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 0x7fe11c0b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe122131700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fe114007e60 tx=0x7fe11400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11400ede0 con 0x7fe11c0cd680 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe114014420 con 0x7fe11c0cd680 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c0b2eb0 con 0x7fe11c0cd680 2023-06-10T18:56:50.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140147d0 con 0x7fe11c0cd680 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.379+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c0b3470 con 0x7fe11c0cd680 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.380+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c074050 con 0x7fe11c0cd680 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.380+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe114014930 con 0x7fe11c0cd680 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.381+0000 7fe1127fc700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 0x7fe10006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.381+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fe114089cc0 con 0x7fe11c0cd680 2023-06-10T18:56:50.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.381+0000 7fe121930700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 0x7fe10006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.382+0000 7fe121930700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 0x7fe10006d190 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe10c013040 tx=0x7fe10c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.386+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe114054c00 con 0x7fe11c0cd680 2023-06-10T18:56:50.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.573+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe11c0ce830 con 0x7fe11c0cd680 2023-06-10T18:56:50.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.574+0000 7fe1127fc700 1 -- 172.21.15.104:0/373273063 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+18674 (secure 0 0 0) 0x7fe114058220 con 0x7fe11c0cd680 2023-06-10T18:56:50.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:50.581 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":129,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:42.706404+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3796851905":"2023-06-10T19:56:41.727300+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:50.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 msgr2=0x7fe10006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 0x7fe10006d190 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe10c013040 tx=0x7fe10c008070 comp rx=0 tx=0).stop 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 msgr2=0x7fe11c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fe114007e60 tx=0x7fe11400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 shutdown_connections 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe11c0cd680 0x7fe11c069e70 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe11c0aeaa0 0x7fe11c152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe10006ace0 0x7fe10006d190 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 --2- 172.21.15.104:0/373273063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe11c0b1080 0x7fe11c0b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 >> 172.21.15.104:0/373273063 conn(0x7fe11c06b8a0 msgr2=0x7fe11c0af400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 shutdown_connections 2023-06-10T18:56:50.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.576+0000 7fe123b94700 1 -- 172.21.15.104:0/373273063 wait complete. 2023-06-10T18:56:50.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:56:17.300478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:56:18.192765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:56:50.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:56:50.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.712+0000 7f4531130700 1 Processor -- start 2023-06-10T18:56:50.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.712+0000 7f4531130700 1 -- start start 2023-06-10T18:56:50.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f4531130700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c15c6f0 0x7f452c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f4531130700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f452c0aec00 0x7f452c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f4531130700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0ae7e0 con 0x7f452c06ea70 2023-06-10T18:56:50.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f4531130700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c06df90 con 0x7f452c0aec00 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f4531130700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c06e270 con 0x7f452c15c6f0 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f452b59e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f452c0aec00 0x7f452c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f452a59c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f452c06ea70 0x7f452c0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41386/0 (socket says 172.21.15.104:41386) 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f452a59c700 1 -- 172.21.15.104:0/3443432911 learned_addr learned my addr 172.21.15.104:0/3443432911 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.713+0000 7f452ad9d700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c15c6f0 0x7f452c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f452b59e700 1 -- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c15c6f0 msgr2=0x7f452c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:50.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f452b59e700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c15c6f0 0x7f452c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f452b59e700 1 -- 172.21.15.104:0/3443432911 >> v1:172.21.15.114:6789/0 conn(0x7f452c06ea70 legacy=0x7f452c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f452b59e700 1 -- 172.21.15.104:0/3443432911 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c06e550 con 0x7f452c0aec00 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4529d9b700 1 -- 172.21.15.104:0/3443432911 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386284646 0 0) 0x7f452c0ae7e0 con 0x7f452c06ea70 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f452b59e700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c0aec00 0x7f452c0b15b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4524009d90 tx=0x7f452400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce56310540031e95 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4529d9b700 1 -- 172.21.15.104:0/3443432911 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452400e070 con 0x7f452c0aec00 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4529d9b700 1 -- 172.21.15.104:0/3443432911 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452400bb20 con 0x7f452c0aec00 2023-06-10T18:56:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4529d9b700 1 -- 172.21.15.104:0/3443432911 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524012430 con 0x7f452c0aec00 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 -- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c0aec00 msgr2=0x7f452c0b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c0aec00 0x7f452c0b15b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4524009d90 tx=0x7f452400b330 comp rx=0 tx=0).stop 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 -- 172.21.15.104:0/3443432911 shutdown_connections 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c0aec00 0x7f452c0b15b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 --2- 172.21.15.104:0/3443432911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c15c6f0 0x7f452c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.714+0000 7f4531130700 1 -- 172.21.15.104:0/3443432911 >> 172.21.15.104:0/3443432911 conn(0x7f452c06b8a0 msgr2=0x7f452c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.715+0000 7f4531130700 1 -- 172.21.15.104:0/3443432911 shutdown_connections 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.715+0000 7f4531130700 1 -- 172.21.15.104:0/3443432911 wait complete. 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.716+0000 7f4531130700 1 Processor -- start 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.716+0000 7f4531130700 1 -- start start 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 0x7f452c152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c15c6f0 0x7f452c0c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0caf40 con 0x7f452c06ea70 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0ca450 con 0x7f452c15c6f0 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0ca730 con 0x7f452c0aec00 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37716/0 (socket says 172.21.15.104:37716) 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 -- 172.21.15.104:0/2171128607 learned_addr learned my addr 172.21.15.104:0/2171128607 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452b59e700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c15c6f0 0x7f452c0c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452a59c700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 0x7f452c152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 -- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 msgr2=0x7f452c152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 0x7f452c152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 -- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c15c6f0 msgr2=0x7f452c0c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c15c6f0 0x7f452c0c39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 -- 172.21.15.104:0/2171128607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524009a70 con 0x7f452c06ea70 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452a59c700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 0x7f452c152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f452ad9d700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f452000e900 tx=0x7f452000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520009650 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452000ee50 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c0caa10 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452000ca50 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.717+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c0ca0d0 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.718+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c0cb420 con 0x7f452c06ea70 2023-06-10T18:56:50.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.719+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f452000cbb0 con 0x7f452c06ea70 2023-06-10T18:56:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.719+0000 7f45137fe700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 0x7f451806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:50.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.719+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f45200898e0 con 0x7f452c06ea70 2023-06-10T18:56:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.719+0000 7f452a59c700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 0x7f451806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.720+0000 7f452a59c700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 0x7f451806d190 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f452c154210 tx=0x7f451c007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:50.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.724+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f452c0cb420 con 0x7f452c06ea70 2023-06-10T18:56:50.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.912+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f452c0cb420 con 0x7f452c06ea70 2023-06-10T18:56:50.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.913+0000 7f45137fe700 1 -- 172.21.15.104:0/2171128607 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4520017020 con 0x7f452c06ea70 2023-06-10T18:56:50.919 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.914+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 msgr2=0x7f451806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 0x7f451806d190 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f452c154210 tx=0x7f451c007ca0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 msgr2=0x7f452c0c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f452000e900 tx=0x7f452000ec10 comp rx=0 tx=0).stop 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 shutdown_connections 2023-06-10T18:56:50.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f451806ace0 0x7f451806d190 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f452c15c6f0 0x7f452c0c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f452c06ea70 0x7f452c0c9b90 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 --2- 172.21.15.104:0/2171128607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f452c0aec00 0x7f452c152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 >> 172.21.15.104:0/2171128607 conn(0x7f452c06b8a0 msgr2=0x7f452c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 shutdown_connections 2023-06-10T18:56:50.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:50.915+0000 7f4531130700 1 -- 172.21.15.104:0/2171128607 wait complete. 2023-06-10T18:56:50.933 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:56:51.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.050+0000 7f9d9e1b9700 1 Processor -- start 2023-06-10T18:56:51.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.050+0000 7f9d9e1b9700 1 -- start start 2023-06-10T18:56:51.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.050+0000 7f9d9e1b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d98158c10 0x7f9d980cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d9e1b9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d98158fe0 0x7f9d9806d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d9e1b9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9d980c4b90 0x7f9d9806d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d9e1b9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980c5d80 con 0x7f9d98158fe0 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d9e1b9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980cec80 con 0x7f9d980c4b90 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d9e1b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980cd920 con 0x7f9d98158c10 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d98158fe0 0x7f9d9806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d98158fe0 0x7f9d9806d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37726/0 (socket says 172.21.15.104:37726) 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 -- 172.21.15.104:0/275341132 learned_addr learned my addr 172.21.15.104:0/275341132 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d977fe700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d98158c10 0x7f9d980cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d97fff700 1 --2- 172.21.15.104:0/275341132 >> v2:172.21.15.114:3300/0 conn(0x7f9d980c4b90 0x7f9d9806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 -- 172.21.15.104:0/275341132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d98158c10 msgr2=0x7f9d980cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d98158c10 0x7f9d980cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 -- 172.21.15.104:0/275341132 >> v2:172.21.15.114:3300/0 conn(0x7f9d980c4b90 msgr2=0x7f9d9806d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:51.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 --2- 172.21.15.104:0/275341132 >> v2:172.21.15.114:3300/0 conn(0x7f9d980c4b90 0x7f9d9806d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 -- 172.21.15.104:0/275341132 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980cdbe0 con 0x7f9d98158fe0 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d96ffd700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d9806d440 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9d980c4a50 tx=0x7f9d8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ad3ab687222d78 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d967fc700 1 -- 172.21.15.104:0/275341132 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00b870 con 0x7f9d98158fe0 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d967fc700 1 -- 172.21.15.104:0/275341132 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00b9d0 con 0x7f9d98158fe0 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.051+0000 7f9d967fc700 1 -- 172.21.15.104:0/275341132 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00bd60 con 0x7f9d98158fe0 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/275341132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 msgr2=0x7f9d9806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d9806d440 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9d980c4a50 tx=0x7f9d8c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/275341132 shutdown_connections 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/275341132 >> v2:172.21.15.114:3300/0 conn(0x7f9d980c4b90 0x7f9d9806d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d98158c10 0x7f9d980cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/275341132 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d9806d440 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/275341132 >> 172.21.15.104:0/275341132 conn(0x7f9d980ceec0 msgr2=0x7f9d980cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/275341132 shutdown_connections 2023-06-10T18:56:51.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.052+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/275341132 wait complete. 2023-06-10T18:56:51.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.053+0000 7f9d9e1b9700 1 Processor -- start 2023-06-10T18:56:51.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.053+0000 7f9d9e1b9700 1 -- start start 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d980c4b90 0x7f9d98141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d98158c10 0x7f9d9814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98141ec0 con 0x7f9d98158fe0 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98142490 con 0x7f9d98158c10 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d9e1b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9814ede0 con 0x7f9d980c4b90 2023-06-10T18:56:51.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37734/0 (socket says 172.21.15.104:37734) 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 -- 172.21.15.104:0/4204681243 learned_addr learned my addr 172.21.15.104:0/4204681243 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 -- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d980c4b90 msgr2=0x7f9d98141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d980c4b90 0x7f9d98141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 -- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d98158c10 msgr2=0x7f9d9814e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d96ffd700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d98158c10 0x7f9d9814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d98158c10 0x7f9d9814e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.054+0000 7f9d97fff700 1 -- 172.21.15.104:0/4204681243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c009550 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d97fff700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9d90007e40 tx=0x7f9d9000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9000eea0 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90014420 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9814f0c0 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900147b0 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.055+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9814f680 con 0x7f9d98158fe0 2023-06-10T18:56:51.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.056+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d98074050 con 0x7f9d98158fe0 2023-06-10T18:56:51.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.056+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9d90014910 con 0x7f9d98158fe0 2023-06-10T18:56:51.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.056+0000 7f9d7ffff700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 0x7f9d8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.056+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9d90089eb0 con 0x7f9d98158fe0 2023-06-10T18:56:51.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.057+0000 7f9d977fe700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 0x7f9d8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.057+0000 7f9d977fe700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 0x7f9d8406d190 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009770 tx=0x7f9d88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.061+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9d90058880 con 0x7f9d98158fe0 2023-06-10T18:56:51.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.246+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9d98142600 con 0x7f9d98158fe0 2023-06-10T18:56:51.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.246+0000 7f9d7ffff700 1 -- 172.21.15.104:0/4204681243 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v129) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9d90058410 con 0x7f9d98158fe0 2023-06-10T18:56:51.253 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.248+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 msgr2=0x7f9d8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.248+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 0x7f9d8406d190 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009770 tx=0x7f9d88008040 comp rx=0 tx=0).stop 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.248+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 msgr2=0x7f9d980ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.248+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9d90007e40 tx=0x7f9d9000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 shutdown_connections 2023-06-10T18:56:51.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d98158fe0 0x7f9d980ca3a0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d980c4b90 0x7f9d98141980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d8406ace0 0x7f9d8406d190 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 --2- 172.21.15.104:0/4204681243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d98158c10 0x7f9d9814e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 >> 172.21.15.104:0/4204681243 conn(0x7f9d980ceec0 msgr2=0x7f9d980cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 shutdown_connections 2023-06-10T18:56:51.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.249+0000 7f9d9e1b9700 1 -- 172.21.15.104:0/4204681243 wait complete. 2023-06-10T18:56:51.268 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:56:51.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.384+0000 7fad33654700 1 Processor -- start 2023-06-10T18:56:51.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.384+0000 7fad33654700 1 -- start start 2023-06-10T18:56:51.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad33654700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 0x7fad2c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad33654700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad33654700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0cc9f0 con 0x7fad2c0cb210 2023-06-10T18:56:51.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad33654700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c0cbbd0 con 0x7fad2c0cf0b0 2023-06-10T18:56:51.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad33654700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0cc100 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad31bf1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad31bf1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:53476/0 (socket says 172.21.15.104:53476) 2023-06-10T18:56:51.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad30bef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 0x7fad2c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad31bf1700 1 -- 172.21.15.104:0/3730743473 learned_addr learned my addr 172.21.15.104:0/3730743473 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.385+0000 7fad313f0700 1 --1- 172.21.15.104:0/3730743473 >> v1:172.21.15.114:6789/0 conn(0x7fad2c0cb210 0x7fad2c0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41412/0 (socket says 172.21.15.104:41412) 2023-06-10T18:56:51.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad30bef700 1 -- 172.21.15.104:0/3730743473 >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 msgr2=0x7fad2c06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:51.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad30bef700 1 --2- 172.21.15.104:0/3730743473 >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad30bef700 1 -- 172.21.15.104:0/3730743473 >> v1:172.21.15.114:6789/0 conn(0x7fad2c0cb210 legacy=0x7fad2c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad30bef700 1 -- 172.21.15.104:0/3730743473 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0cc3a0 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad31bf1700 1 --2- 172.21.15.104:0/3730743473 >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad30bef700 1 --2- 172.21.15.104:0/3730743473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 0x7fad2c0cb5e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad28009840 tx=0x7fad28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd92f70900e7d33f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad23fff700 1 -- 172.21.15.104:0/3730743473 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28011830 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad23fff700 1 -- 172.21.15.104:0/3730743473 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28011990 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.386+0000 7fad23fff700 1 -- 172.21.15.104:0/3730743473 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28011d20 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 -- 172.21.15.104:0/3730743473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 msgr2=0x7fad2c0cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 --2- 172.21.15.104:0/3730743473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 0x7fad2c0cb5e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad28009840 tx=0x7fad28009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 -- 172.21.15.104:0/3730743473 shutdown_connections 2023-06-10T18:56:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 --2- 172.21.15.104:0/3730743473 >> v2:172.21.15.114:3300/0 conn(0x7fad2c0cf0b0 0x7fad2c06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 --2- 172.21.15.104:0/3730743473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cd8f0 0x7fad2c0cb5e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 -- 172.21.15.104:0/3730743473 >> 172.21.15.104:0/3730743473 conn(0x7fad2c06b8a0 msgr2=0x7fad2c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 -- 172.21.15.104:0/3730743473 shutdown_connections 2023-06-10T18:56:51.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.387+0000 7fad33654700 1 -- 172.21.15.104:0/3730743473 wait complete. 2023-06-10T18:56:51.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.388+0000 7fad33654700 1 Processor -- start 2023-06-10T18:56:51.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.388+0000 7fad33654700 1 -- start start 2023-06-10T18:56:51.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 0x7fad2c0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cf0b0 0x7fad2c1e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1f1c10 con 0x7fad2c0cb210 2023-06-10T18:56:51.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1f1eb0 con 0x7fad2c0cd8f0 2023-06-10T18:56:51.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad33654700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1f2190 con 0x7fad2c0cf0b0 2023-06-10T18:56:51.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37744/0 (socket says 172.21.15.104:37744) 2023-06-10T18:56:51.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 -- 172.21.15.104:0/4076653400 learned_addr learned my addr 172.21.15.104:0/4076653400 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad30bef700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 0x7fad2c0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad31bf1700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cf0b0 0x7fad2c1e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 -- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cf0b0 msgr2=0x7fad2c1e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cf0b0 0x7fad2c1e2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 -- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 msgr2=0x7fad2c0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 0x7fad2c0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 -- 172.21.15.104:0/4076653400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24009b30 con 0x7fad2c0cb210 2023-06-10T18:56:51.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad30bef700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 0x7fad2c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:51.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.389+0000 7fad313f0700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c480 tx=0x7fad1c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.390+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c005d00 con 0x7fad2c0cb210 2023-06-10T18:56:51.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.390+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c005e60 con 0x7fad2c0cb210 2023-06-10T18:56:51.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.390+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad280094a0 con 0x7fad2c0cb210 2023-06-10T18:56:51.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.390+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0072f0 con 0x7fad2c0cb210 2023-06-10T18:56:51.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.390+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c1f2750 con 0x7fad2c0cb210 2023-06-10T18:56:51.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.391+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c074050 con 0x7fad2c0cb210 2023-06-10T18:56:51.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.391+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fad1c007450 con 0x7fad2c0cb210 2023-06-10T18:56:51.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.391+0000 7fad21ffb700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 0x7fad1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.392+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fad1c088b30 con 0x7fad2c0cb210 2023-06-10T18:56:51.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.392+0000 7fad30bef700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 0x7fad1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.392+0000 7fad30bef700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 0x7fad1006d190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1f2b70 tx=0x7fad28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.396+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fad1c053b00 con 0x7fad2c0cb210 2023-06-10T18:56:51.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.583+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad2c0c6210 con 0x7fad2c0cb210 2023-06-10T18:56:51.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.583+0000 7fad21ffb700 1 -- 172.21.15.104:0/4076653400 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad1c017020 con 0x7fad2c0cb210 2023-06-10T18:56:51.590 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:56:51.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 msgr2=0x7fad1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 0x7fad1006d190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1f2b70 tx=0x7fad28008030 comp rx=0 tx=0).stop 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 msgr2=0x7fad2c0c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c480 tx=0x7fad1c00c840 comp rx=0 tx=0).stop 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 shutdown_connections 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad2c0cf0b0 0x7fad2c1e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad2c0cb210 0x7fad2c0c8a60 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fad1006ace0 0x7fad1006d190 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 --2- 172.21.15.104:0/4076653400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2c0cd8f0 0x7fad2c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 >> 172.21.15.104:0/4076653400 conn(0x7fad2c06b8a0 msgr2=0x7fad2c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 shutdown_connections 2023-06-10T18:56:51.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.586+0000 7fad33654700 1 -- 172.21.15.104:0/4076653400 wait complete. 2023-06-10T18:56:51.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:56:51.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 Processor -- start 2023-06-10T18:56:51.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 -- start start 2023-06-10T18:56:51.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd62806af80 0x7fd6280d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd6280cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6280ce340 con 0x7fd6280ccff0 2023-06-10T18:56:51.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6280cd490 con 0x7fd62806af80 2023-06-10T18:56:51.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.721+0000 7fd62db07700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6280d00d0 con 0x7fd6280ce7d0 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd62806af80 0x7fd6280d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd62806af80 0x7fd6280d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37752/0 (socket says 172.21.15.104:37752) 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 -- 172.21.15.104:0/822960139 learned_addr learned my addr 172.21.15.104:0/822960139 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd627fff700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd6280cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 -- 172.21.15.104:0/822960139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 msgr2=0x7fd6280cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd6280cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 -- 172.21.15.104:0/822960139 >> v1:172.21.15.114:6789/0 conn(0x7fd6280ccff0 legacy=0x7fd62806b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:51.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 -- 172.21.15.104:0/822960139 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6280cd680 con 0x7fd62806af80 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6277fe700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6280d0d20 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098f0 tx=0x7fd618009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e023f24c2f64a604 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6267fc700 1 -- 172.21.15.104:0/822960139 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd618004230 con 0x7fd62806af80 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6267fc700 1 -- 172.21.15.104:0/822960139 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd618004390 con 0x7fd62806af80 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.722+0000 7fd6267fc700 1 -- 172.21.15.104:0/822960139 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180036a0 con 0x7fd62806af80 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 -- 172.21.15.104:0/822960139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 msgr2=0x7fd6280d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6280d0d20 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098f0 tx=0x7fd618009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:51.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 -- 172.21.15.104:0/822960139 shutdown_connections 2023-06-10T18:56:51.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6280d0d20 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 --2- 172.21.15.104:0/822960139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd6280cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 -- 172.21.15.104:0/822960139 >> 172.21.15.104:0/822960139 conn(0x7fd628089fd0 msgr2=0x7fd62808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 -- 172.21.15.104:0/822960139 shutdown_connections 2023-06-10T18:56:51.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.723+0000 7fd62db07700 1 -- 172.21.15.104:0/822960139 wait complete. 2023-06-10T18:56:51.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.724+0000 7fd62db07700 1 Processor -- start 2023-06-10T18:56:51.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.724+0000 7fd62db07700 1 -- start start 2023-06-10T18:56:51.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6280ccff0 0x7fd62815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd62814fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6280c59b0 con 0x7fd62806af80 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6280b0de0 con 0x7fd6280ccff0 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd62db07700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6280c5c20 con 0x7fd6280ce7d0 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37756/0 (socket says 172.21.15.104:37756) 2023-06-10T18:56:51.735 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 -- 172.21.15.104:0/2784477675 learned_addr learned my addr 172.21.15.104:0/2784477675 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd627fff700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd62814fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd626ffd700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6280ccff0 0x7fd62815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 -- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 msgr2=0x7fd62814fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd62814fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 -- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6280ccff0 msgr2=0x7fd62815b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6280ccff0 0x7fd62815b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 -- 172.21.15.104:0/2784477675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618009550 con 0x7fd62806af80 2023-06-10T18:56:51.736 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.725+0000 7fd6277fe700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fd618009840 tx=0x7fd618004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.726+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd618003cc0 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.726+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61801f070 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.726+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6280c5f00 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.726+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61800b8a0 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.726+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6280b0600 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.727+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62806c600 con 0x7fd62806af80 2023-06-10T18:56:51.737 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.728+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd618021030 con 0x7fd62806af80 2023-06-10T18:56:51.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.728+0000 7fd60ffff700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 0x7fd61406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:51.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.728+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd61808aef0 con 0x7fd62806af80 2023-06-10T18:56:51.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.728+0000 7fd626ffd700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 0x7fd61406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:51.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.729+0000 7fd626ffd700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 0x7fd61406d190 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd6280adb50 tx=0x7fd61c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:51.739 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.732+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd618055fa0 con 0x7fd62806af80 2023-06-10T18:56:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.918+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd62806c600 con 0x7fd62806af80 2023-06-10T18:56:51.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.918+0000 7fd60ffff700 1 -- 172.21.15.104:0/2784477675 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6180595c0 con 0x7fd62806af80 2023-06-10T18:56:51.925 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:56:51.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 msgr2=0x7fd61406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 0x7fd61406d190 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd6280adb50 tx=0x7fd61c009040 comp rx=0 tx=0).stop 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 msgr2=0x7fd6281584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fd618009840 tx=0x7fd618004750 comp rx=0 tx=0).stop 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 shutdown_connections 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6280ce7d0 0x7fd62814fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd62806af80 0x7fd6281584a0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd61406ace0 0x7fd61406d190 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 --2- 172.21.15.104:0/2784477675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6280ccff0 0x7fd62815b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:51.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 >> 172.21.15.104:0/2784477675 conn(0x7fd628089fd0 msgr2=0x7fd62806f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:51.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 shutdown_connections 2023-06-10T18:56:51.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:51.920+0000 7fd62db07700 1 -- 172.21.15.104:0/2784477675 wait complete. 2023-06-10T18:56:51.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:52.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.052+0000 7fcb860de700 1 Processor -- start 2023-06-10T18:56:52.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.052+0000 7fcb860de700 1 -- start start 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb860de700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e0e0 0x7fcb8006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb860de700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb8006e4b0 0x7fcb800c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb860de700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8006d290 con 0x7fcb800ca100 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb860de700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb80071640 con 0x7fcb8006e4b0 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb860de700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800aef00 con 0x7fcb8006e0e0 2023-06-10T18:56:52.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcb800ca100 0x7fcb800ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35596/0 (socket says 172.21.15.104:35596) 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7ffff700 1 -- 172.21.15.104:0/1143783615 learned_addr learned my addr 172.21.15.104:0/1143783615 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 --2- 172.21.15.104:0/1143783615 >> v2:172.21.15.114:3300/0 conn(0x7fcb8006e4b0 0x7fcb800c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7f7fe700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e0e0 0x7fcb8006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319737768 0 0) 0x7fcb8006d290 con 0x7fcb800ca100 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb680035b0 con 0x7fcb800ca100 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908136095 0 0) 0x7fcb680035b0 con 0x7fcb800ca100 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 -- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e0e0 msgr2=0x7fcb8006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:52.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e0e0 0x7fcb8006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 -- 172.21.15.104:0/1143783615 >> v1:172.21.15.104:6789/0 conn(0x7fcb800ca100 legacy=0x7fcb800ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 -- 172.21.15.104:0/1143783615 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8006d4c0 con 0x7fcb8006e4b0 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.053+0000 7fcb7effd700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb8006e4b0 0x7fcb800c9bc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009840 tx=0x7fcb74009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a323ea36893e4bc9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb780019f0 con 0x7fcb8006e4b0 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74004030 con 0x7fcb8006e4b0 2023-06-10T18:56:52.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb7e7fc700 1 -- 172.21.15.104:0/1143783615 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740043c0 con 0x7fcb8006e4b0 2023-06-10T18:56:52.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 -- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb8006e4b0 msgr2=0x7fcb800c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb8006e4b0 0x7fcb800c9bc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009840 tx=0x7fcb74009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 -- 172.21.15.104:0/1143783615 shutdown_connections 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e0e0 0x7fcb8006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 --2- 172.21.15.104:0/1143783615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb8006e4b0 0x7fcb800c9bc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 -- 172.21.15.104:0/1143783615 >> 172.21.15.104:0/1143783615 conn(0x7fcb8006b8a0 msgr2=0x7fcb800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 -- 172.21.15.104:0/1143783615 shutdown_connections 2023-06-10T18:56:52.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.054+0000 7fcb860de700 1 -- 172.21.15.104:0/1143783615 wait complete. 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.055+0000 7fcb860de700 1 Processor -- start 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 -- start start 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e4b0 0x7fcb801478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb800ca100 0x7fcb800c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800aff30 con 0x7fcb8006e0e0 2023-06-10T18:56:52.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b0210 con 0x7fcb800ca100 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb860de700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800b04f0 con 0x7fcb8006e4b0 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37770/0 (socket says 172.21.15.104:37770) 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.056+0000 7fcb7f7fe700 1 -- 172.21.15.104:0/2909841050 learned_addr learned my addr 172.21.15.104:0/2909841050 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7effd700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e4b0 0x7fcb801478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7ffff700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb800ca100 0x7fcb800c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 -- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e4b0 msgr2=0x7fcb801478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e4b0 0x7fcb801478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 -- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb800ca100 msgr2=0x7fcb800c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb800ca100 0x7fcb800c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 -- 172.21.15.104:0/2909841050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740094a0 con 0x7fcb8006e0e0 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb7f7fe700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000e900 tx=0x7fcb7000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70009650 con 0x7fcb8006e0e0 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7000c850 con 0x7fcb8006e0e0 2023-06-10T18:56:52.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb800b07d0 con 0x7fcb8006e0e0 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000cbc0 con 0x7fcb8006e0e0 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.057+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb800b0d90 con 0x7fcb8006e0e0 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.058+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb800b0f20 con 0x7fcb8006e0e0 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.058+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb7000cd20 con 0x7fcb8006e0e0 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.059+0000 7fcb67fff700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 0x7fcb6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.059+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fcb70089980 con 0x7fcb8006e0e0 2023-06-10T18:56:52.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.059+0000 7fcb7effd700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 0x7fcb6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.059+0000 7fcb7effd700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 0x7fcb6c06d190 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcb740090b0 tx=0x7fcb74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.063+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb70054950 con 0x7fcb8006e0e0 2023-06-10T18:56:52.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.289+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb80074050 con 0x7fcb8006e0e0 2023-06-10T18:56:52.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.290+0000 7fcb67fff700 1 -- 172.21.15.104:0/2909841050 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fcb70057f70 con 0x7fcb8006e0e0 2023-06-10T18:56:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:52.296 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 msgr2=0x7fcb6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 0x7fcb6c06d190 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcb740090b0 tx=0x7fcb74008030 comp rx=0 tx=0).stop 2023-06-10T18:56:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 msgr2=0x7fcb800b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000e900 tx=0x7fcb7000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 shutdown_connections 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb6c06ace0 0x7fcb6c06d190 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb800ca100 0x7fcb800c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb8006e0e0 0x7fcb800b2430 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 --2- 172.21.15.104:0/2909841050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb8006e4b0 0x7fcb801478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 >> 172.21.15.104:0/2909841050 conn(0x7fcb8006b8a0 msgr2=0x7fcb800ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 shutdown_connections 2023-06-10T18:56:52.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.292+0000 7fcb860de700 1 -- 172.21.15.104:0/2909841050 wait complete. 2023-06-10T18:56:52.300 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:52.310 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:52.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.423+0000 7f628259d700 1 Processor -- start 2023-06-10T18:56:52.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.423+0000 7f628259d700 1 -- start start 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628259d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f627c0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628259d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0cee30 con 0x7f627c0ae9b0 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628259d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0cf2c0 con 0x7f627c15ca00 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628259d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627c0aed80 con 0x7f627c0ca170 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628159b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f627c0ae9b0 0x7f627c0ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35606/0 (socket says 172.21.15.104:35606) 2023-06-10T18:56:52.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f628159b700 1 -- 172.21.15.104:0/653817084 learned_addr learned my addr 172.21.15.104:0/653817084 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f6281d9c700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f627c0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730513620 0 0) 0x7f627c0cee30 con 0x7f627c0ae9b0 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.424+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62680035b0 con 0x7f627c0ae9b0 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1563438223 0 0) 0x7f62680035b0 con 0x7f627c0ae9b0 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 msgr2=0x7f627c0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f627c0ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 >> v1:172.21.15.114:6789/0 conn(0x7f627c15ca00 legacy=0x7f627c0c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0cd680 con 0x7f627c0ae9b0 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f626c0017f0 con 0x7f627c0ae9b0 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 >> v1:172.21.15.104:6789/0 conn(0x7f627c0ae9b0 legacy=0x7f627c0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f6268003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6268003f70 0x7f6268004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6268004d90 0x7f6268005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c0aed80 con 0x7f6268004d90 2023-06-10T18:56:52.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626800ba40 con 0x7f6268003f70 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626800de10 con 0x7f627c0ca170 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6268004d90 0x7f6268005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f628159b700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f6268003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6280d9a700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6268003f70 0x7f6268004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 -- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 msgr2=0x7f6268003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0ca170 0x7f6268003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 -- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6268003f70 msgr2=0x7f6268004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6268003f70 0x7f6268004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 -- 172.21.15.104:0/653817084 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62680035b0 con 0x7f6268004d90 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.425+0000 7f6281d9c700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6268004d90 0x7f6268005200 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f627c071110 tx=0x7f6268005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9305b8e26d7e260 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627400ebd0 con 0x7f6268004d90 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627400f040 con 0x7f6268004d90 2023-06-10T18:56:52.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f6273fff700 1 -- 172.21.15.104:0/653817084 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274014480 con 0x7f6268004d90 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 -- 172.21.15.104:0/653817084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6268004d90 msgr2=0x7f6268005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 --2- 172.21.15.104:0/653817084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6268004d90 0x7f6268005200 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f627c071110 tx=0x7f6268005880 comp rx=0 tx=0).stop 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f6281d9c700 1 -- 172.21.15.104:0/653817084 reap_dead start 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 -- 172.21.15.104:0/653817084 shutdown_connections 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 -- 172.21.15.104:0/653817084 >> 172.21.15.104:0/653817084 conn(0x7f627c06b8a0 msgr2=0x7f627c0cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:52.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 -- 172.21.15.104:0/653817084 shutdown_connections 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.426+0000 7f628259d700 1 -- 172.21.15.104:0/653817084 wait complete. 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.428+0000 7f628259d700 1 Processor -- start 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.428+0000 7f628259d700 1 -- start start 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f627c158be0 0x7f627c142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 0x7f627c1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c0c50a0 con 0x7f627c158810 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c1487f0 con 0x7f627c158be0 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627c0c3a70 con 0x7f627c0c5a00 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37788/0 (socket says 172.21.15.104:37788) 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 -- 172.21.15.104:0/3644634532 learned_addr learned my addr 172.21.15.104:0/3644634532 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f6280d9a700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f627c158be0 0x7f627c142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f6281d9c700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 0x7f627c1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 -- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 msgr2=0x7f627c1429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 0x7f627c1429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 -- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f627c158be0 msgr2=0x7f627c142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f627c158be0 0x7f627c142460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 -- 172.21.15.104:0/3644634532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0c3d50 con 0x7f627c158810 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f6281d9c700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 0x7f627c1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628159b700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f626c00faf0 tx=0x7f626c00feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c0072e0 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626c007460 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.429+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c0c49e0 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.430+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c005830 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.430+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c0c2f00 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.431+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627c04ea20 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.431+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f626c005990 con 0x7f627c158810 2023-06-10T18:56:52.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.431+0000 7f6271ffb700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 0x7f626006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.432+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f626c089770 con 0x7f627c158810 2023-06-10T18:56:52.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.432+0000 7f6280d9a700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 0x7f626006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.432+0000 7f6280d9a700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 0x7f626006d190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f62780011a0 tx=0x7f6278009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.436+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f626c0546a0 con 0x7f627c158810 2023-06-10T18:56:52.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.626+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f627c06bca0 con 0x7f627c158810 2023-06-10T18:56:52.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.627+0000 7f6271ffb700 1 -- 172.21.15.104:0/3644634532 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+18674 (secure 0 0 0) 0x7f626c057cc0 con 0x7f627c158810 2023-06-10T18:56:52.633 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:52.634 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":129,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:42.706404+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3796851905":"2023-06-10T19:56:41.727300+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 msgr2=0x7f626006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 0x7f626006d190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f62780011a0 tx=0x7f6278009040 comp rx=0 tx=0).stop 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 msgr2=0x7f627c0c54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f626c00faf0 tx=0x7f626c00feb0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 shutdown_connections 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f627c0c5a00 0x7f627c1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f627c158810 0x7f627c0c54c0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f626006ace0 0x7f626006d190 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 --2- 172.21.15.104:0/3644634532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f627c158be0 0x7f627c142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 >> 172.21.15.104:0/3644634532 conn(0x7f627c06b8a0 msgr2=0x7f627c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:52.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 shutdown_connections 2023-06-10T18:56:52.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.629+0000 7f628259d700 1 -- 172.21.15.104:0/3644634532 wait complete. 2023-06-10T18:56:52.647 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:56:52.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:52.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.763+0000 7f4247860700 1 Processor -- start 2023-06-10T18:56:52.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.763+0000 7f4247860700 1 -- start start 2023-06-10T18:56:52.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.763+0000 7f4247860700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400ccfa0 0x7f42400d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.763+0000 7f4247860700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42400d0520 0x7f42400d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.764+0000 7f4247860700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42400cd430 con 0x7f42400d2b20 2023-06-10T18:56:52.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.764+0000 7f4247860700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400d0ee0 con 0x7f42400d0520 2023-06-10T18:56:52.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.764+0000 7f4247860700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400cd710 con 0x7f42400ccfa0 2023-06-10T18:56:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42400d0520 0x7f42400d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f42455fc700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f42400d2b20 0x7f42400b2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41470/0 (socket says 172.21.15.104:41470) 2023-06-10T18:56:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f42455fc700 1 -- 172.21.15.104:0/1216614159 learned_addr learned my addr 172.21.15.104:0/1216614159 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:52.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 -- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400ccfa0 msgr2=0x7f42400d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 --2- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400ccfa0 0x7f42400d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 -- 172.21.15.104:0/1216614159 >> v1:172.21.15.114:6789/0 conn(0x7f42400d2b20 legacy=0x7f42400b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 -- 172.21.15.104:0/1216614159 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400cd9f0 con 0x7f42400d0520 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f4245dfd700 1 --2- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400d0520 0x7f42400d09a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4238009840 tx=0x7f4238009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b15afa04d23f978f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.765+0000 7f423ffff700 1 -- 172.21.15.104:0/1216614159 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238011830 con 0x7f42400d0520 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.766+0000 7f423ffff700 1 -- 172.21.15.104:0/1216614159 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238011990 con 0x7f42400d0520 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.766+0000 7f423ffff700 1 -- 172.21.15.104:0/1216614159 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238011d20 con 0x7f42400d0520 2023-06-10T18:56:52.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.766+0000 7f4247860700 1 -- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400d0520 msgr2=0x7f42400d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.766+0000 7f4247860700 1 --2- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400d0520 0x7f42400d09a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4238009840 tx=0x7f4238009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 -- 172.21.15.104:0/1216614159 shutdown_connections 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 --2- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400d0520 0x7f42400d09a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 --2- 172.21.15.104:0/1216614159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400ccfa0 0x7f42400d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 -- 172.21.15.104:0/1216614159 >> 172.21.15.104:0/1216614159 conn(0x7f4240071ec0 msgr2=0x7f424009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 -- 172.21.15.104:0/1216614159 shutdown_connections 2023-06-10T18:56:52.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.767+0000 7f4247860700 1 -- 172.21.15.104:0/1216614159 wait complete. 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.768+0000 7f4247860700 1 Processor -- start 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.768+0000 7f4247860700 1 -- start start 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 0x7f42400cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 0x7f42400b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424015f860 con 0x7f42400ccfa0 2023-06-10T18:56:52.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400cc8d0 con 0x7f42400cb970 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f4247860700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400b6b00 con 0x7f42400d0520 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f42455fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f42455fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37802/0 (socket says 172.21.15.104:37802) 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.769+0000 7f42455fc700 1 -- 172.21.15.104:0/1809532133 learned_addr learned my addr 172.21.15.104:0/1809532133 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4245dfd700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 0x7f42400b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 -- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 msgr2=0x7f42400cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4244dfb700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 0x7f42400cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 0x7f42400cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 -- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 msgr2=0x7f42400b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 0x7f42400b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 -- 172.21.15.104:0/1809532133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380094a0 con 0x7f42400ccfa0 2023-06-10T18:56:52.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4245dfd700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 0x7f42400b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4244dfb700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 0x7f42400cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f42455fc700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f423000caa0 tx=0x7f423000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000aa40 con 0x7f42400ccfa0 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000aba0 con 0x7f42400ccfa0 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42400b5780 con 0x7f42400ccfa0 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230012400 con 0x7f42400ccfa0 2023-06-10T18:56:52.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.770+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42400b5d40 con 0x7f42400ccfa0 2023-06-10T18:56:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.771+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f423001b3b0 con 0x7f42400ccfa0 2023-06-10T18:56:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.772+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240074050 con 0x7f42400ccfa0 2023-06-10T18:56:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.772+0000 7f423dffb700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 0x7f422c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.772+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f42300893d0 con 0x7f42400ccfa0 2023-06-10T18:56:52.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.772+0000 7f4244dfb700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 0x7f422c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.773+0000 7f4244dfb700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 0x7f422c06d190 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4240159400 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:52.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:52.777+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4230054250 con 0x7f42400ccfa0 2023-06-10T18:56:53.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.004+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42400ccd80 con 0x7f42400ccfa0 2023-06-10T18:56:53.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.005+0000 7f423dffb700 1 -- 172.21.15.104:0/1809532133 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f4230057870 con 0x7f42400ccfa0 2023-06-10T18:56:53.011 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:53.012 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 msgr2=0x7f422c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 0x7f422c06d190 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4240159400 tx=0x7f4234008040 comp rx=0 tx=0).stop 2023-06-10T18:56:53.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 msgr2=0x7f4240158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f423000caa0 tx=0x7f423000ce60 comp rx=0 tx=0).stop 2023-06-10T18:56:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 shutdown_connections 2023-06-10T18:56:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f422c06ace0 0x7f422c06d190 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42400cb970 0x7f42400b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42400ccfa0 0x7f4240158380 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 --2- 172.21.15.104:0/1809532133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42400d0520 0x7f42400cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 >> 172.21.15.104:0/1809532133 conn(0x7f4240071ec0 msgr2=0x7f424006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 shutdown_connections 2023-06-10T18:56:53.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.007+0000 7f4247860700 1 -- 172.21.15.104:0/1809532133 wait complete. 2023-06-10T18:56:53.016 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:53.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:56:53.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.138+0000 7f11aa293700 1 Processor -- start 2023-06-10T18:56:53.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.138+0000 7f11aa293700 1 -- start start 2023-06-10T18:56:53.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11aa293700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 0x7f11a40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11aa293700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a40cdae0 con 0x7f11a40cd680 2023-06-10T18:56:53.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11aa293700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a40c9e60 con 0x7f11a40b1080 2023-06-10T18:56:53.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11aa293700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40c9bc0 con 0x7f11a40aeaa0 2023-06-10T18:56:53.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a8830700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f11a40cd680 0x7f11a406dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35642/0 (socket says 172.21.15.104:35642) 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a8830700 1 -- 172.21.15.104:0/1216157022 learned_addr learned my addr 172.21.15.104:0/1216157022 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a3fff700 1 --2- 172.21.15.104:0/1216157022 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 0x7f11a40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099623434 0 0) 0x7f11a40cdae0 con 0x7f11a40cd680 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c0035b0 con 0x7f11a40cd680 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147700224 0 0) 0x7f118c0035b0 con 0x7f11a40cd680 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a3fff700 1 -- 172.21.15.104:0/1216157022 >> v1:172.21.15.114:6789/0 conn(0x7f11a40b1080 legacy=0x7f11a40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a3fff700 1 -- 172.21.15.104:0/1216157022 >> v1:172.21.15.104:6789/0 conn(0x7f11a40cd680 legacy=0x7f11a406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.139+0000 7f11a3fff700 1 -- 172.21.15.104:0/1216157022 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a40aee70 con 0x7f11a40aeaa0 2023-06-10T18:56:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11a3fff700 1 --2- 172.21.15.104:0/1216157022 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 0x7f11a40cd140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1198009850 tx=0x7f1198009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30ab4d83ba2f67bb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c0019f0 con 0x7f11a40aeaa0 2023-06-10T18:56:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198005c30 con 0x7f11a40aeaa0 2023-06-10T18:56:53.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11a2ffd700 1 -- 172.21.15.104:0/1216157022 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198006030 con 0x7f11a40aeaa0 2023-06-10T18:56:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 -- 172.21.15.104:0/1216157022 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 msgr2=0x7f11a40cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 --2- 172.21.15.104:0/1216157022 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 0x7f11a40cd140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1198009850 tx=0x7f1198009b60 comp rx=0 tx=0).stop 2023-06-10T18:56:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 -- 172.21.15.104:0/1216157022 shutdown_connections 2023-06-10T18:56:53.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 --2- 172.21.15.104:0/1216157022 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40aeaa0 0x7f11a40cd140 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 -- 172.21.15.104:0/1216157022 >> 172.21.15.104:0/1216157022 conn(0x7f11a406b8a0 msgr2=0x7f11a40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 -- 172.21.15.104:0/1216157022 shutdown_connections 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.140+0000 7f11aa293700 1 -- 172.21.15.104:0/1216157022 wait complete. 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 Processor -- start 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 -- start start 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11a40aeaa0 0x7f11a40c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40b1080 0x7f11a415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40b29f0 con 0x7f11a40cd680 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40b2cd0 con 0x7f11a40aeaa0 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.142+0000 7f11aa293700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40b2fb0 con 0x7f11a40b1080 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37820/0 (socket says 172.21.15.104:37820) 2023-06-10T18:56:53.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11a40aeaa0 0x7f11a40c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 -- 172.21.15.104:0/1867425169 learned_addr learned my addr 172.21.15.104:0/1867425169 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a37fe700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40b1080 0x7f11a415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 -- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40b1080 msgr2=0x7f11a415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40b1080 0x7f11a415b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 -- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11a40aeaa0 msgr2=0x7f11a40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11a40aeaa0 0x7f11a40c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 -- 172.21.15.104:0/1867425169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1198009430 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a8830700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f119c002230 tx=0x7f119c00cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c00aaa0 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c00ac20 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c0070d0 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a40b3290 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.143+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a40c98e0 con 0x7f11a40cd680 2023-06-10T18:56:53.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.144+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40b37b0 con 0x7f11a40cd680 2023-06-10T18:56:53.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.145+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f119c007230 con 0x7f11a40cd680 2023-06-10T18:56:53.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.145+0000 7f11a0ff9700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.145+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f119c089160 con 0x7f11a40cd680 2023-06-10T18:56:53.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.145+0000 7f11a3fff700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.146+0000 7f11a3fff700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f11980097a0 tx=0x7f1198007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.149+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11a40b37b0 con 0x7f11a40cd680 2023-06-10T18:56:53.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.374+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a40b37b0 con 0x7f11a40cd680 2023-06-10T18:56:53.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.375+0000 7f11a0ff9700 1 -- 172.21.15.104:0/1867425169 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f11a40b37b0 con 0x7f11a40cd680 2023-06-10T18:56:53.381 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:53.382 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":80,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:56:19.230688+0000","modified":"2023-06-10T18:56:22.252369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9231":{"gid":9231,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6836/3783425308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3783425308},{"type":"v1","addr":"172.21.15.114:6836","nonce":3783425308}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-06-10T18:56:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 msgr2=0x7f119006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f11980097a0 tx=0x7f1198007000 comp rx=0 tx=0).stop 2023-06-10T18:56:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 msgr2=0x7f11a40c6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f119c002230 tx=0x7f119c00cec0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 shutdown_connections 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f119006ace0 0x7f119006d190 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11a40aeaa0 0x7f11a40c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11a40cd680 0x7f11a40c6990 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 --2- 172.21.15.104:0/1867425169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11a40b1080 0x7f11a415b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 >> 172.21.15.104:0/1867425169 conn(0x7f11a406b8a0 msgr2=0x7f11a40af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 shutdown_connections 2023-06-10T18:56:53.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.377+0000 7f11aa293700 1 -- 172.21.15.104:0/1867425169 wait complete. 2023-06-10T18:56:53.385 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 80 2023-06-10T18:56:53.395 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:56:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.509+0000 7f8cefc3e700 1 Processor -- start 2023-06-10T18:56:53.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.509+0000 7f8cefc3e700 1 -- start start 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cefc3e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ce80d0c00 0x7f8ce806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cefc3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce806b820 0x7f8ce80cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cefc3e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce806bfd0 con 0x7f8ce80b1e30 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cefc3e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce806c2b0 con 0x7f8ce80d0c00 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cefc3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce80ccf70 con 0x7f8ce806b820 2023-06-10T18:56:53.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ce80d0c00 0x7f8ce806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ce80d0c00 0x7f8ce806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37826/0 (socket says 172.21.15.104:37826) 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 -- 172.21.15.104:0/3648666489 learned_addr learned my addr 172.21.15.104:0/3648666489 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced9da700 1 --1- 172.21.15.104:0/3648666489 >> v1:172.21.15.104:6789/0 conn(0x7f8ce80b1e30 0x7f8ce80d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35652/0 (socket says 172.21.15.104:35652) 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cee1db700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce806b820 0x7f8ce80cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 -- 172.21.15.104:0/3648666489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce806b820 msgr2=0x7f8ce80cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce806b820 0x7f8ce80cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 -- 172.21.15.104:0/3648666489 >> v1:172.21.15.104:6789/0 conn(0x7f8ce80b1e30 legacy=0x7f8ce80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8ced1d9700 1 -- 172.21.15.104:0/3648666489 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80cd250 con 0x7f8ce80d0c00 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.510+0000 7f8cec9d8700 1 -- 172.21.15.104:0/3648666489 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607461938 0 0) 0x7f8ce806bfd0 con 0x7f8ce80b1e30 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce80d0c00 0x7f8ce806b2e0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098f0 tx=0x7f8ce4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf45ad9211f9474d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cec9d8700 1 -- 172.21.15.104:0/3648666489 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce400b8a0 con 0x7f8ce80d0c00 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cec9d8700 1 -- 172.21.15.104:0/3648666489 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce400ba00 con 0x7f8ce80d0c00 2023-06-10T18:56:53.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cec9d8700 1 -- 172.21.15.104:0/3648666489 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce400bd90 con 0x7f8ce80d0c00 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3648666489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce80d0c00 msgr2=0x7f8ce806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce80d0c00 0x7f8ce806b2e0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098f0 tx=0x7f8ce4009c00 comp rx=0 tx=0).stop 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3648666489 shutdown_connections 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce806b820 0x7f8ce80cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3648666489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce80d0c00 0x7f8ce806b2e0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3648666489 >> 172.21.15.104:0/3648666489 conn(0x7f8ce8071ec0 msgr2=0x7f8ce809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3648666489 shutdown_connections 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.511+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3648666489 wait complete. 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.512+0000 7f8cefc3e700 1 Processor -- start 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 -- start start 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 0x7f8ce814ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ce80d0c00 0x7f8ce815e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce815f830 con 0x7f8ce806b820 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce814b280 con 0x7f8ce80d0c00 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8cefc3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce815ebb0 con 0x7f8ce80b1e30 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8ced9da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8ced9da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37830/0 (socket says 172.21.15.104:37830) 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.513+0000 7f8ced9da700 1 -- 172.21.15.104:0/3584119779 learned_addr learned my addr 172.21.15.104:0/3584119779 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:53.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 0x7f8ce814ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cee1db700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ce80d0c00 0x7f8ce815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 -- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 msgr2=0x7f8ce814ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 0x7f8ce814ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 -- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ce80d0c00 msgr2=0x7f8ce815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ce80d0c00 0x7f8ce815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 -- 172.21.15.104:0/3584119779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4009550 con 0x7f8ce806b820 2023-06-10T18:56:53.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 0x7f8ce814ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8ced9da700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80011e0 tx=0x7f8cd800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8004c80 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd8004e00 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce815ee90 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8007310 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.514+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce815f450 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.515+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce80b5750 con 0x7f8ce806b820 2023-06-10T18:56:53.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.515+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8cd8007470 con 0x7f8ce806b820 2023-06-10T18:56:53.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.516+0000 7f8cde7fc700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 0x7f8ccc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.516+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8cd80893a0 con 0x7f8ce806b820 2023-06-10T18:56:53.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.516+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 0x7f8ccc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.516+0000 7f8ced1d9700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 0x7f8ccc06d190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4014040 tx=0x7f8ce4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.520+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8cd80542b0 con 0x7f8ce806b820 2023-06-10T18:56:53.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.707+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ce80d1860 con 0x7f8ce806b820 2023-06-10T18:56:53.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.709+0000 7f8cde7fc700 1 -- 172.21.15.104:0/3584119779 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+18715 (secure 0 0 0) 0x7f8cd80578d0 con 0x7f8ce806b820 2023-06-10T18:56:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:56:53.715 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":130,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:56:52.751925+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:56:17.300478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-06-10T18:56:18.192765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.e","mappings":[{"from":2,"to":0}]},{"pgid":"19.31","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3796851905":"2023-06-10T19:56:41.727300+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:56:53.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 msgr2=0x7f8ccc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 0x7f8ccc06d190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4014040 tx=0x7f8ce4008030 comp rx=0 tx=0).stop 2023-06-10T18:56:53.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 msgr2=0x7f8ce815d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80011e0 tx=0x7f8cd800cae0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 shutdown_connections 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8ccc06ace0 0x7f8ccc06d190 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ce80d0c00 0x7f8ce815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ce806b820 0x7f8ce815d6d0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 --2- 172.21.15.104:0/3584119779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ce80b1e30 0x7f8ce814ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 >> 172.21.15.104:0/3584119779 conn(0x7f8ce8071ec0 msgr2=0x7f8ce8072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 shutdown_connections 2023-06-10T18:56:53.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.711+0000 7f8cefc3e700 1 -- 172.21.15.104:0/3584119779 wait complete. 2023-06-10T18:56:53.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:56:53.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 Processor -- start 2023-06-10T18:56:53.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 -- start start 2023-06-10T18:56:53.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03c406e4b0 0x7f03c406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03c4069e70 0x7f03c40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c406d290 con 0x7f03c406e0e0 2023-06-10T18:56:53.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c4071640 con 0x7f03c4069e70 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.848+0000 7f03c9110700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c40aef00 con 0x7f03c406e4b0 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03c4069e70 0x7f03c40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f03c4069e70 0x7f03c40ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37838/0 (socket says 172.21.15.104:37838) 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 -- 172.21.15.104:0/1874421799 learned_addr learned my addr 172.21.15.104:0/1874421799 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c259c700 1 --2- 172.21.15.104:0/1874421799 >> v2:172.21.15.114:3300/0 conn(0x7f03c406e4b0 0x7f03c406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 -- 172.21.15.104:0/1874421799 >> v2:172.21.15.114:3300/0 conn(0x7f03c406e4b0 msgr2=0x7f03c406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 --2- 172.21.15.104:0/1874421799 >> v2:172.21.15.114:3300/0 conn(0x7f03c406e4b0 0x7f03c406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 -- 172.21.15.104:0/1874421799 >> v1:172.21.15.114:6789/0 conn(0x7f03c406e0e0 legacy=0x7f03c40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 -- 172.21.15.104:0/1874421799 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40ca2d0 con 0x7f03c4069e70 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c359e700 1 --2- 172.21.15.104:0/1874421799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40ce9f0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f03c4070320 tx=0x7f03bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0c2444f015a2b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c1d9b700 1 -- 172.21.15.104:0/1874421799 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00e070 con 0x7f03c4069e70 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c1d9b700 1 -- 172.21.15.104:0/1874421799 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc003a20 con 0x7f03c4069e70 2023-06-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.849+0000 7f03c1d9b700 1 -- 172.21.15.104:0/1874421799 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc003dd0 con 0x7f03c4069e70 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 -- 172.21.15.104:0/1874421799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 msgr2=0x7f03c40ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 --2- 172.21.15.104:0/1874421799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40ce9f0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f03c4070320 tx=0x7f03bc00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 -- 172.21.15.104:0/1874421799 shutdown_connections 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 --2- 172.21.15.104:0/1874421799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40ce9f0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 --2- 172.21.15.104:0/1874421799 >> v2:172.21.15.114:3300/0 conn(0x7f03c406e4b0 0x7f03c406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 -- 172.21.15.104:0/1874421799 >> 172.21.15.104:0/1874421799 conn(0x7f03c406b8a0 msgr2=0x7f03c40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 -- 172.21.15.104:0/1874421799 shutdown_connections 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.850+0000 7f03c9110700 1 -- 172.21.15.104:0/1874421799 wait complete. 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.851+0000 7f03c9110700 1 Processor -- start 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.851+0000 7f03c9110700 1 -- start start 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03c406e0e0 0x7f03c4146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 0x7f03c41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40b2430 con 0x7f03c4069e70 2023-06-10T18:56:53.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40c81d0 con 0x7f03c406e4b0 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c9110700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40b2640 con 0x7f03c406e0e0 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37840/0 (socket says 172.21.15.104:37840) 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 -- 172.21.15.104:0/157950767 learned_addr learned my addr 172.21.15.104:0/157950767 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c259c700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03c406e0e0 0x7f03c4146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c359e700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 0x7f03c41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 -- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03c406e0e0 msgr2=0x7f03c4146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03c406e0e0 0x7f03c4146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 -- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 msgr2=0x7f03c41474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 0x7f03c41474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c2d9d700 1 -- 172.21.15.104:0/157950767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc009b30 con 0x7f03c4069e70 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.852+0000 7f03c359e700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 0x7f03c41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03c2d9d700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f03b400b450 tx=0x7f03b400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4004020 con 0x7f03c4069e70 2023-06-10T18:56:53.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b4004180 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4004530 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c40b2920 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.853+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c40b2c00 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.854+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c4074050 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.854+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03b40036d0 con 0x7f03c4069e70 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.855+0000 7f03ab7fe700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 0x7f03b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:53.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.855+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f03b4088e00 con 0x7f03c4069e70 2023-06-10T18:56:53.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.855+0000 7f03c259c700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 0x7f03b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:53.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.856+0000 7f03c259c700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 0x7f03b006d190 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f03b800ab60 tx=0x7f03b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:53.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:53.859+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f03b4053d90 con 0x7f03c4069e70 2023-06-10T18:56:54.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.087+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f03c4070b20 con 0x7f03c4069e70 2023-06-10T18:56:54.769 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:56:54.763+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:56:54.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.770+0000 7f03ab7fe700 1 -- 172.21.15.104:0/157950767 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7f03b4015090 con 0x7f03c4069e70 2023-06-10T18:56:54.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 msgr2=0x7f03b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 0x7f03b006d190 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f03b800ab60 tx=0x7f03b8008040 comp rx=0 tx=0).stop 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 msgr2=0x7f03c40c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f03b400b450 tx=0x7f03b400b760 comp rx=0 tx=0).stop 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 shutdown_connections 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03b006ace0 0x7f03b006d190 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03c406e4b0 0x7f03c41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03c4069e70 0x7f03c40c8970 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 --2- 172.21.15.104:0/157950767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03c406e0e0 0x7f03c4146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 >> 172.21.15.104:0/157950767 conn(0x7f03c406b8a0 msgr2=0x7f03c40cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:54.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 shutdown_connections 2023-06-10T18:56:54.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.773+0000 7f03c9110700 1 -- 172.21.15.104:0/157950767 wait complete. 2023-06-10T18:56:54.781 INFO:tasks.ceph.mds.d.smithi114.stderr: -3408> 2023-06-10T18:56:24.822+0000 7f4a310a4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:56:54.782 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:56:54.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:56:54.808 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:56:54.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.914+0000 7f589692b700 1 Processor -- start 2023-06-10T18:56:54.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 -- start start 2023-06-10T18:56:54.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5890069e70 0x7f589006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f589006dcd0 0x7f589006e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589006e650 con 0x7f589015c500 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58900cf4d0 con 0x7f5890069e70 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f589692b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58900ce040 con 0x7f589006dcd0 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f588f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5890069e70 0x7f589006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.915+0000 7f588ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f589015c500 0x7f58900cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35676/0 (socket says 172.21.15.104:35676) 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588ffff700 1 -- 172.21.15.104:0/4151546550 learned_addr learned my addr 172.21.15.104:0/4151546550 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f5894ec8700 1 --2- 172.21.15.104:0/4151546550 >> v2:172.21.15.114:3300/0 conn(0x7f589006dcd0 0x7f589006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588f7fe700 1 -- 172.21.15.104:0/4151546550 >> v2:172.21.15.114:3300/0 conn(0x7f589006dcd0 msgr2=0x7f589006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588f7fe700 1 --2- 172.21.15.104:0/4151546550 >> v2:172.21.15.114:3300/0 conn(0x7f589006dcd0 0x7f589006e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588f7fe700 1 -- 172.21.15.104:0/4151546550 >> v1:172.21.15.104:6789/0 conn(0x7f589015c500 legacy=0x7f58900cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588f7fe700 1 -- 172.21.15.104:0/4151546550 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589015c8d0 con 0x7f5890069e70 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588effd700 1 -- 172.21.15.104:0/4151546550 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041340023 0 0) 0x7f589006e650 con 0x7f589015c500 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588f7fe700 1 --2- 172.21.15.104:0/4151546550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f589006a2e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f58900cd0a0 tx=0x7f5878009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990b6715930e353e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588effd700 1 -- 172.21.15.104:0/4151546550 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878011830 con 0x7f5890069e70 2023-06-10T18:56:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588effd700 1 -- 172.21.15.104:0/4151546550 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5878011990 con 0x7f5890069e70 2023-06-10T18:56:54.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.916+0000 7f588effd700 1 -- 172.21.15.104:0/4151546550 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878011d20 con 0x7f5890069e70 2023-06-10T18:56:54.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 -- 172.21.15.104:0/4151546550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 msgr2=0x7f589006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 --2- 172.21.15.104:0/4151546550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f589006a2e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f58900cd0a0 tx=0x7f5878009b20 comp rx=0 tx=0).stop 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 -- 172.21.15.104:0/4151546550 shutdown_connections 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 --2- 172.21.15.104:0/4151546550 >> v2:172.21.15.114:3300/0 conn(0x7f589006dcd0 0x7f589006e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 --2- 172.21.15.104:0/4151546550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f589006a2e0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 -- 172.21.15.104:0/4151546550 >> 172.21.15.104:0/4151546550 conn(0x7f589006b8a0 msgr2=0x7f58900cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 -- 172.21.15.104:0/4151546550 shutdown_connections 2023-06-10T18:56:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.917+0000 7f589692b700 1 -- 172.21.15.104:0/4151546550 wait complete. 2023-06-10T18:56:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.918+0000 7f589692b700 1 Processor -- start 2023-06-10T18:56:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.918+0000 7f589692b700 1 -- start start 2023-06-10T18:56:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589006dcd0 0x7f58900c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f589015c500 0x7f58901f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890150e40 con 0x7f5890069e70 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58900b4c10 con 0x7f589006dcd0 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f589692b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58900b4ef0 con 0x7f589015c500 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37854/0 (socket says 172.21.15.104:37854) 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 -- 172.21.15.104:0/3072498205 learned_addr learned my addr 172.21.15.104:0/3072498205 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588f7fe700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589006dcd0 0x7f58900c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f5894ec8700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f589015c500 0x7f58901f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 -- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f589015c500 msgr2=0x7f58901f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f589015c500 0x7f58901f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 -- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589006dcd0 msgr2=0x7f58900c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:54.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589006dcd0 0x7f58900c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.919+0000 7f588ffff700 1 -- 172.21.15.104:0/3072498205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780094a0 con 0x7f5890069e70 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f588ffff700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f58800010e0 tx=0x7f588000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58800074e0 con 0x7f5890069e70 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880007640 con 0x7f5890069e70 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58900b51d0 con 0x7f5890069e70 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880005a70 con 0x7f5890069e70 2023-06-10T18:56:54.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.920+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58900b2840 con 0x7f5890069e70 2023-06-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.921+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5890074050 con 0x7f5890069e70 2023-06-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.921+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5880005bd0 con 0x7f5890069e70 2023-06-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.921+0000 7f588cff9700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 0x7f587006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.922+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f5880088a90 con 0x7f5890069e70 2023-06-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.922+0000 7f588f7fe700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 0x7f587006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:54.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.922+0000 7f588f7fe700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 0x7f587006d190 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f58780090b0 tx=0x7f587801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:54.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:54.926+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f58800539d0 con 0x7f5890069e70 2023-06-10T18:56:55.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.155+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f589006c320 con 0x7f5890069e70 2023-06-10T18:56:55.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.774+0000 7f588cff9700 1 -- 172.21.15.104:0/3072498205 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5880007e40 con 0x7f5890069e70 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 msgr2=0x7f587006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 0x7f587006d190 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f58780090b0 tx=0x7f587801c040 comp rx=0 tx=0).stop 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 msgr2=0x7f5890150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f58800010e0 tx=0x7f588000ca30 comp rx=0 tx=0).stop 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 shutdown_connections 2023-06-10T18:56:55.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f589015c500 0x7f58901f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5890069e70 0x7f5890150900 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f587006ace0 0x7f587006d190 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 --2- 172.21.15.104:0/3072498205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589006dcd0 0x7f58900c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.776+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 >> 172.21.15.104:0/3072498205 conn(0x7f589006b8a0 msgr2=0x7f58900b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.777+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 shutdown_connections 2023-06-10T18:56:55.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.777+0000 7f589692b700 1 -- 172.21.15.104:0/3072498205 wait complete. 2023-06-10T18:56:55.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:56:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 Processor -- start 2023-06-10T18:56:55.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 -- start start 2023-06-10T18:56:55.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc3815c580 0x7fbc380b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc3806cde0 0x7fbc380ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3806e250 con 0x7fbc3806e6e0 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3815c950 con 0x7fbc3806cde0 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.920+0000 7fbc3e745700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380b1e80 con 0x7fbc3815c580 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbc3806cde0 0x7fbc380ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc377fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbc3806e6e0 0x7fbc380b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35690/0 (socket says 172.21.15.104:35690) 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc377fe700 1 -- 172.21.15.104:0/1882506832 learned_addr learned my addr 172.21.15.104:0/1882506832 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:55.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 -- 172.21.15.104:0/1882506832 >> v2:172.21.15.114:3300/0 conn(0x7fbc3815c580 msgr2=0x7fbc380b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc37fff700 1 --2- 172.21.15.104:0/1882506832 >> v2:172.21.15.114:3300/0 conn(0x7fbc3815c580 0x7fbc380b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 --2- 172.21.15.104:0/1882506832 >> v2:172.21.15.114:3300/0 conn(0x7fbc3815c580 0x7fbc380b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 -- 172.21.15.104:0/1882506832 >> v1:172.21.15.104:6789/0 conn(0x7fbc3806e6e0 legacy=0x7fbc380b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 -- 172.21.15.104:0/1882506832 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3806d920 con 0x7fbc3806cde0 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1882506832 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750194010 0 0) 0x7fbc3806e250 con 0x7fbc3806e6e0 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc3cce2700 1 --2- 172.21.15.104:0/1882506832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc380ce0d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fbc38070300 tx=0x7fbc3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517d526aaf69781 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:55.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1882506832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000e070 con 0x7fbc3806cde0 2023-06-10T18:56:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1882506832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3000bbd0 con 0x7fbc3806cde0 2023-06-10T18:56:55.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.921+0000 7fbc36ffd700 1 -- 172.21.15.104:0/1882506832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30012430 con 0x7fbc3806cde0 2023-06-10T18:56:55.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 -- 172.21.15.104:0/1882506832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 msgr2=0x7fbc380ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 --2- 172.21.15.104:0/1882506832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc380ce0d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fbc38070300 tx=0x7fbc3000b330 comp rx=0 tx=0).stop 2023-06-10T18:56:55.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 -- 172.21.15.104:0/1882506832 shutdown_connections 2023-06-10T18:56:55.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 --2- 172.21.15.104:0/1882506832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc380ce0d0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 --2- 172.21.15.104:0/1882506832 >> v2:172.21.15.114:3300/0 conn(0x7fbc3815c580 0x7fbc380b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 -- 172.21.15.104:0/1882506832 >> 172.21.15.104:0/1882506832 conn(0x7fbc3806b8a0 msgr2=0x7fbc380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 -- 172.21.15.104:0/1882506832 shutdown_connections 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.922+0000 7fbc3e745700 1 -- 172.21.15.104:0/1882506832 wait complete. 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.923+0000 7fbc3e745700 1 Processor -- start 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.923+0000 7fbc3e745700 1 -- start start 2023-06-10T18:56:55.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806e6e0 0x7fbc381e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc3815c580 0x7fbc380cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b2ad0 con 0x7fbc3806cde0 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc38158ea0 con 0x7fbc3806e6e0 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc3e745700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b24c0 con 0x7fbc3815c580 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:55.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37866/0 (socket says 172.21.15.104:37866) 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 -- 172.21.15.104:0/4281290953 learned_addr learned my addr 172.21.15.104:0/4281290953 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 -- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc3815c580 msgr2=0x7fbc380cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc3815c580 0x7fbc380cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 -- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806e6e0 msgr2=0x7fbc381e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc377fe700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806e6e0 0x7fbc381e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806e6e0 0x7fbc381e4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:55.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 -- 172.21.15.104:0/4281290953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30009a70 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc37fff700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c820 tx=0x7fbc2800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.924+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28011800 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.925+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28011960 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.925+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28011cf0 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.925+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc380b5110 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.925+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc380b53f0 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.926+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc38074050 con 0x7fbc3806cde0 2023-06-10T18:56:55.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.926+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbc28018360 con 0x7fbc3806cde0 2023-06-10T18:56:55.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.926+0000 7fbc34ff9700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:55.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.926+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6569+0+0 (secure 0 0 0) 0x7fbc280895a0 con 0x7fbc3806cde0 2023-06-10T18:56:55.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.927+0000 7fbc377fe700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:55.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.927+0000 7fbc377fe700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbc380ae7e0 tx=0x7fbc2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:55.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:55.931+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbc28054450 con 0x7fbc3806cde0 2023-06-10T18:56:56.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.115+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbc38159090 con 0x7fbc3806cde0 2023-06-10T18:56:56.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.828+0000 7fbc34ff9700 1 -- 172.21.15.104:0/4281290953 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbc28057a70 con 0x7fbc3806cde0 2023-06-10T18:56:56.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 msgr2=0x7fbc2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbc380ae7e0 tx=0x7fbc2c009040 comp rx=0 tx=0).stop 2023-06-10T18:56:56.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 msgr2=0x7fbc38158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c820 tx=0x7fbc2800cbe0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 shutdown_connections 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc3815c580 0x7fbc380cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc3806cde0 0x7fbc38158960 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbc2406ace0 0x7fbc2406d190 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 --2- 172.21.15.104:0/4281290953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc3806e6e0 0x7fbc381e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 >> 172.21.15.104:0/4281290953 conn(0x7fbc3806b8a0 msgr2=0x7fbc380c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 shutdown_connections 2023-06-10T18:56:56.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.831+0000 7fbc3e745700 1 -- 172.21.15.104:0/4281290953 wait complete. 2023-06-10T18:56:56.839 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:56:56.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:56:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.976+0000 7fa6d8858700 1 Processor -- start 2023-06-10T18:56:56.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.976+0000 7fa6d8858700 1 -- start start 2023-06-10T18:56:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d8858700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d8858700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:56.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d8858700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40ce040 con 0x7fa6d4069e70 2023-06-10T18:56:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d8858700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d415c8d0 con 0x7fa6d415c500 2023-06-10T18:56:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d8858700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d406da40 con 0x7fa6d406e3e0 2023-06-10T18:56:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d259c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:56.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:56.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d259c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:53608/0 (socket says 172.21.15.104:53608) 2023-06-10T18:56:56.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:57258/0 (socket says 172.21.15.104:57258) 2023-06-10T18:56:56.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d259c700 1 -- 172.21.15.104:0/3899605940 learned_addr learned my addr 172.21.15.104:0/3899605940 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:56.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d1d9b700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa6d4069e70 0x7fa6d406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41548/0 (socket says 172.21.15.104:41548) 2023-06-10T18:56:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 -- 172.21.15.104:0/3899605940 >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 msgr2=0x7fa6d40cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 --2- 172.21.15.104:0/3899605940 >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 -- 172.21.15.104:0/3899605940 >> v1:172.21.15.114:6789/0 conn(0x7fa6d4069e70 legacy=0x7fa6d406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d2d9d700 1 -- 172.21.15.104:0/3899605940 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d406a9c0 con 0x7fa6d406e3e0 2023-06-10T18:56:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.977+0000 7fa6d259c700 1 --2- 172.21.15.104:0/3899605940 >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:56:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d2d9d700 1 --2- 172.21.15.104:0/3899605940 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009d90 tx=0x7fa6cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf19f1dfa448ae07 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:56.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d159a700 1 -- 172.21.15.104:0/3899605940 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00e070 con 0x7fa6d406e3e0 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d159a700 1 -- 172.21.15.104:0/3899605940 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc00bb20 con 0x7fa6d406e3e0 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d159a700 1 -- 172.21.15.104:0/3899605940 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc012430 con 0x7fa6d406e3e0 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 -- 172.21.15.104:0/3899605940 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 msgr2=0x7fa6d406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 --2- 172.21.15.104:0/3899605940 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009d90 tx=0x7fa6cc00b330 comp rx=0 tx=0).stop 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 -- 172.21.15.104:0/3899605940 shutdown_connections 2023-06-10T18:56:56.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 --2- 172.21.15.104:0/3899605940 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d406e3e0 0x7fa6d406a2e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 --2- 172.21.15.104:0/3899605940 >> v2:172.21.15.114:3300/0 conn(0x7fa6d415c500 0x7fa6d40cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.978+0000 7fa6d8858700 1 -- 172.21.15.104:0/3899605940 >> 172.21.15.104:0/3899605940 conn(0x7fa6d406b8a0 msgr2=0x7fa6d40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.979+0000 7fa6d8858700 1 -- 172.21.15.104:0/3899605940 shutdown_connections 2023-06-10T18:56:56.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.979+0000 7fa6d8858700 1 -- 172.21.15.104:0/3899605940 wait complete. 2023-06-10T18:56:56.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.980+0000 7fa6d8858700 1 Processor -- start 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.980+0000 7fa6d8858700 1 -- start start 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 0x7fa6d40b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d415c500 0x7fa6d41e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d41535e0 con 0x7fa6d406e3e0 2023-06-10T18:56:56.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d41e4c00 con 0x7fa6d4069e70 2023-06-10T18:56:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d8858700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d41e4ea0 con 0x7fa6d415c500 2023-06-10T18:56:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:56.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37880/0 (socket says 172.21.15.104:37880) 2023-06-10T18:56:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.981+0000 7fa6d1d9b700 1 -- 172.21.15.104:0/319740783 learned_addr learned my addr 172.21.15.104:0/319740783 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d259c700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 0x7fa6d40b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d2d9d700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d415c500 0x7fa6d41e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:56.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 -- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d415c500 msgr2=0x7fa6d41e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d415c500 0x7fa6d41e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 -- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 msgr2=0x7fa6d40b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 0x7fa6d40b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:56.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 -- 172.21.15.104:0/319740783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009500 con 0x7fa6d406e3e0 2023-06-10T18:56:56.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d259c700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 0x7fa6d40b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:56.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d1d9b700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c500 tx=0x7fa6c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:56.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8005cb0 con 0x7fa6d406e3e0 2023-06-10T18:56:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8005e10 con 0x7fa6d406e3e0 2023-06-10T18:56:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc009a70 con 0x7fa6d406e3e0 2023-06-10T18:56:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80072d0 con 0x7fa6d406e3e0 2023-06-10T18:56:57.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.982+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d41e5460 con 0x7fa6d406e3e0 2023-06-10T18:56:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.983+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d4074050 con 0x7fa6d406e3e0 2023-06-10T18:56:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.983+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa6c8007430 con 0x7fa6d406e3e0 2023-06-10T18:56:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.984+0000 7fa6c2ffd700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 0x7fa6b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.984+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6131+0+0 (secure 0 0 0) 0x7fa6c80888c0 con 0x7fa6d406e3e0 2023-06-10T18:56:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.984+0000 7fa6d259c700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 0x7fa6b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.985+0000 7fa6d259c700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 0x7fa6b406d190 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4006fd0 tx=0x7fa6c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:57.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:56.988+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa6c8053930 con 0x7fa6d406e3e0 2023-06-10T18:56:57.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.173+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6d40c9de0 con 0x7fa6d406e3e0 2023-06-10T18:56:57.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.836+0000 7fa6c2ffd700 1 -- 172.21.15.104:0/319740783 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa6c8056f50 con 0x7fa6d406e3e0 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 msgr2=0x7fa6b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 0x7fa6b406d190 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4006fd0 tx=0x7fa6c4008040 comp rx=0 tx=0).stop 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 msgr2=0x7fa6d4159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c500 tx=0x7fa6c800c8c0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 shutdown_connections 2023-06-10T18:56:57.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6b406ace0 0x7fa6b406d190 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6d4069e70 0x7fa6d40b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6d415c500 0x7fa6d41e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 --2- 172.21.15.104:0/319740783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6d406e3e0 0x7fa6d4159c50 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.838+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 >> 172.21.15.104:0/319740783 conn(0x7fa6d406b8a0 msgr2=0x7fa6d40b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.839+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 shutdown_connections 2023-06-10T18:56:57.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.839+0000 7fa6d8858700 1 -- 172.21.15.104:0/319740783 wait complete. 2023-06-10T18:56:57.847 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:56:57.860 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:56:57.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.978+0000 7fd83952e700 1 Processor -- start 2023-06-10T18:56:57.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.978+0000 7fd83952e700 1 -- start start 2023-06-10T18:56:57.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd83952e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd834069e70 0x7fd83406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd83952e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd83406e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd83952e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83406e690 con 0x7fd83415c500 2023-06-10T18:56:57.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd83952e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340cf4d0 con 0x7fd834069e70 2023-06-10T18:56:57.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd83952e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340ce040 con 0x7fd83406dcd0 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd832ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd83415c500 0x7fd8340cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35714/0 (socket says 172.21.15.104:35714) 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.979+0000 7fd832ffd700 1 -- 172.21.15.104:0/94883333 learned_addr learned my addr 172.21.15.104:0/94883333 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8327fc700 1 --2- 172.21.15.104:0/94883333 >> v2:172.21.15.114:3300/0 conn(0x7fd834069e70 0x7fd83406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd83406e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079348571 0 0) 0x7fd83406e690 con 0x7fd83415c500 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd81c0035b0 con 0x7fd83415c500 2023-06-10T18:56:57.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388458966 0 0) 0x7fd81c0035b0 con 0x7fd83415c500 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 msgr2=0x7fd83406e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd83406e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 >> v2:172.21.15.114:3300/0 conn(0x7fd834069e70 msgr2=0x7fd83406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 --2- 172.21.15.104:0/94883333 >> v2:172.21.15.114:3300/0 conn(0x7fd834069e70 0x7fd83406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83415c8d0 con 0x7fd83415c500 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd8280017f0 con 0x7fd83415c500 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 >> v1:172.21.15.104:6789/0 conn(0x7fd83415c500 legacy=0x7fd8340cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c003a00 0x7fd81c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd81c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd81c004d90 0x7fd81c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340cf4d0 con 0x7fd81c004d90 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340ce040 con 0x7fd81c003a00 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c00ba10 con 0x7fd83406dcd0 2023-06-10T18:56:57.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd81c004d90 0x7fd81c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd832ffd700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c003a00 0x7fd81c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8327fc700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd81c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 -- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 msgr2=0x7fd81c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd83406dcd0 0x7fd81c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 -- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c003a00 msgr2=0x7fd81c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:57.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c003a00 0x7fd81c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.980+0000 7fd8337fe700 1 -- 172.21.15.104:0/94883333 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0035b0 con 0x7fd81c004d90 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd8337fe700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd81c004d90 0x7fd81c005200 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009fc0 tx=0x7fd81c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8fdee996aa98c1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00ebd0 con 0x7fd81c004d90 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00f040 con 0x7fd81c004d90 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd831ffb700 1 -- 172.21.15.104:0/94883333 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c014480 con 0x7fd81c004d90 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 -- 172.21.15.104:0/94883333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd81c004d90 msgr2=0x7fd81c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 --2- 172.21.15.104:0/94883333 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd81c004d90 0x7fd81c005200 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009fc0 tx=0x7fd81c005880 comp rx=0 tx=0).stop 2023-06-10T18:56:57.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd8337fe700 1 -- 172.21.15.104:0/94883333 reap_dead start 2023-06-10T18:56:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 -- 172.21.15.104:0/94883333 shutdown_connections 2023-06-10T18:56:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 -- 172.21.15.104:0/94883333 >> 172.21.15.104:0/94883333 conn(0x7fd83406b8a0 msgr2=0x7fd8340cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 -- 172.21.15.104:0/94883333 shutdown_connections 2023-06-10T18:56:57.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.981+0000 7fd83952e700 1 -- 172.21.15.104:0/94883333 wait complete. 2023-06-10T18:56:57.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.983+0000 7fd83952e700 1 Processor -- start 2023-06-10T18:56:57.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.983+0000 7fd83952e700 1 -- start start 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c004d90 0x7fd8340c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8340c72d0 0x7fd8340b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340b5220 con 0x7fd8340c53d0 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c88e0 con 0x7fd81c004d90 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd83952e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c8450 con 0x7fd8340c72d0 2023-06-10T18:56:57.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37902/0 (socket says 172.21.15.104:37902) 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 -- 172.21.15.104:0/1913822374 learned_addr learned my addr 172.21.15.104:0/1913822374 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd832ffd700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c004d90 0x7fd8340c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8327fc700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8340c72d0 0x7fd8340b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 -- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8340c72d0 msgr2=0x7fd8340b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8340c72d0 0x7fd8340b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 -- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c004d90 msgr2=0x7fd8340c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:57.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c004d90 0x7fd8340c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.984+0000 7fd8337fe700 1 -- 172.21.15.104:0/1913822374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340c7840 con 0x7fd8340c53d0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd8337fe700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009f60 tx=0x7fd82c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c007000 con 0x7fd8340c53d0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c003e70 con 0x7fd8340c53d0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340c7b20 con 0x7fd8340c53d0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c014ab0 con 0x7fd8340c53d0 2023-06-10T18:56:57.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.985+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340c80e0 con 0x7fd8340c53d0 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.986+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340c8270 con 0x7fd8340c53d0 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.986+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd82c023020 con 0x7fd8340c53d0 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.986+0000 7fd81b7fe700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 0x7fd82006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.987+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd82c08a8b0 con 0x7fd8340c53d0 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.987+0000 7fd832ffd700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 0x7fd82006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.987+0000 7fd832ffd700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 0x7fd82006d190 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd828000c00 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:57.991+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd82c055c00 con 0x7fd8340c53d0 2023-06-10T18:56:58.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.179+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd834074050 con 0x7fd8340c53d0 2023-06-10T18:56:58.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.183+0000 7fd81b7fe700 1 -- 172.21.15.104:0/1913822374 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd82c059220 con 0x7fd8340c53d0 2023-06-10T18:56:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 msgr2=0x7fd82006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 0x7fd82006d190 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd828000c00 tx=0x7fd828009040 comp rx=0 tx=0).stop 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 msgr2=0x7fd8340c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009f60 tx=0x7fd82c009750 comp rx=0 tx=0).stop 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 shutdown_connections 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd82006ace0 0x7fd82006d190 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd81c004d90 0x7fd8340c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8340c53d0 0x7fd8340c5850 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 --2- 172.21.15.104:0/1913822374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8340c72d0 0x7fd8340b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 >> 172.21.15.104:0/1913822374 conn(0x7fd83406b8a0 msgr2=0x7fd83406e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:58.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 shutdown_connections 2023-06-10T18:56:58.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.185+0000 7fd83952e700 1 -- 172.21.15.104:0/1913822374 wait complete. 2023-06-10T18:56:58.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps' 2023-06-10T18:56:58.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.318+0000 7fa823349700 1 Processor -- start 2023-06-10T18:56:58.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.318+0000 7fa823349700 1 -- start start 2023-06-10T18:56:58.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa823349700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa81c070a20 0x7fa81c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa823349700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c06ae00 con 0x7fa81c158eb0 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa823349700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c069e70 con 0x7fa81c070df0 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa823349700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c1595f0 con 0x7fa81c070a20 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8218e6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa81c158eb0 0x7fa81c159c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35738/0 (socket says 172.21.15.104:35738) 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8218e6700 1 -- 172.21.15.104:0/159315211 learned_addr learned my addr 172.21.15.104:0/159315211 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8210e5700 1 --2- 172.21.15.104:0/159315211 >> v2:172.21.15.114:3300/0 conn(0x7fa81c070a20 0x7fa81c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa813fff700 1 -- 172.21.15.104:0/159315211 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331472547 0 0) 0x7fa814001490 con 0x7fa81c158eb0 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa813fff700 1 -- 172.21.15.104:0/159315211 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8080035b0 con 0x7fa81c158eb0 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8210e5700 1 -- 172.21.15.104:0/159315211 >> v1:172.21.15.114:6789/0 conn(0x7fa81c070df0 legacy=0x7fa81c1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8210e5700 1 -- 172.21.15.104:0/159315211 >> v1:172.21.15.104:6789/0 conn(0x7fa81c158eb0 legacy=0x7fa81c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8210e5700 1 -- 172.21.15.104:0/159315211 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1598d0 con 0x7fa81c070a20 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.319+0000 7fa8210e5700 1 --2- 172.21.15.104:0/159315211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070a20 0x7fa81c06a8c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa818009840 tx=0x7fa818009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a0f8e855e6fdc40 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa813fff700 1 -- 172.21.15.104:0/159315211 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818011830 con 0x7fa81c070a20 2023-06-10T18:56:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa813fff700 1 -- 172.21.15.104:0/159315211 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818011990 con 0x7fa81c070a20 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa813fff700 1 -- 172.21.15.104:0/159315211 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818011d20 con 0x7fa81c070a20 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 -- 172.21.15.104:0/159315211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070a20 msgr2=0x7fa81c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 --2- 172.21.15.104:0/159315211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070a20 0x7fa81c06a8c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa818009840 tx=0x7fa818009b50 comp rx=0 tx=0).stop 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 -- 172.21.15.104:0/159315211 shutdown_connections 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 --2- 172.21.15.104:0/159315211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070a20 0x7fa81c06a8c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 -- 172.21.15.104:0/159315211 >> 172.21.15.104:0/159315211 conn(0x7fa81c0b1610 msgr2=0x7fa81c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:58.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 -- 172.21.15.104:0/159315211 shutdown_connections 2023-06-10T18:56:58.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.320+0000 7fa823349700 1 -- 172.21.15.104:0/159315211 wait complete. 2023-06-10T18:56:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.321+0000 7fa823349700 1 Processor -- start 2023-06-10T18:56:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.321+0000 7fa823349700 1 -- start start 2023-06-10T18:56:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070df0 0x7fa81c1ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa81c158eb0 0x7fa81c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0c5670 con 0x7fa81c070a20 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1ee270 con 0x7fa81c070df0 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa823349700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0c50b0 con 0x7fa81c158eb0 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37912/0 (socket says 172.21.15.104:37912) 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 -- 172.21.15.104:0/3350499888 learned_addr learned my addr 172.21.15.104:0/3350499888 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:58.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 -- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa81c158eb0 msgr2=0x7fa81c154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa81c158eb0 0x7fa81c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8210e5700 1 -- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070df0 msgr2=0x7fa81c1ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.322+0000 7fa8208e4700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070df0 0x7fa81c1ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa8210e5700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070df0 0x7fa81c1ee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa8210e5700 1 -- 172.21.15.104:0/3350499888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180094a0 con 0x7fa81c070a20 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa8210e5700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fa818009fd0 tx=0x7fa818004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8180038a0 con 0x7fa81c070a20 2023-06-10T18:56:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81801f070 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c0c5390 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818003c40 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.323+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1eec40 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.324+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c071780 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.324+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa818026480 con 0x7fa81c070a20 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.325+0000 7fa811ffb700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.325+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fa818021070 con 0x7fa81c070a20 2023-06-10T18:56:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.325+0000 7fa8208e4700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.325+0000 7fa8208e4700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0c2550 tx=0x7fa80c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.329+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa81808ce60 con 0x7fa81c070a20 2023-06-10T18:56:58.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.593+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps"]} v 0) v1 -- 0x7fa81c1edd30 con 0x7fa81c070a20 2023-06-10T18:56:58.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.836+0000 7fa811ffb700 1 -- 172.21.15.104:0/3350499888 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_evicted_caps"]}]=0 v607) v1 ==== 149+0+0 (secure 0 0 0) 0x7fa818055e50 con 0x7fa81c070a20 2023-06-10T18:56:58.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.838+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 msgr2=0x7fa80006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.838+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0c2550 tx=0x7fa80c009040 comp rx=0 tx=0).stop 2023-06-10T18:56:58.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.838+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 msgr2=0x7fa81c0c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.838+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fa818009fd0 tx=0x7fa818004750 comp rx=0 tx=0).stop 2023-06-10T18:56:58.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 shutdown_connections 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa81c158eb0 0x7fa81c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa81c070a20 0x7fa81c0c8930 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 --2- 172.21.15.104:0/3350499888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa81c070df0 0x7fa81c1ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 >> 172.21.15.104:0/3350499888 conn(0x7fa81c0b1610 msgr2=0x7fa81c0aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 shutdown_connections 2023-06-10T18:56:58.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.839+0000 7fa823349700 1 -- 172.21.15.104:0/3350499888 wait complete. 2023-06-10T18:56:58.857 INFO:tasks.cephfs_test_runner:test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:56:58.858 INFO:tasks.cephfs_test_runner:Starting test: test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:56:58.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock' 2023-06-10T18:56:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.974+0000 7fdb08635700 1 Processor -- start 2023-06-10T18:56:58.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.974+0000 7fdb08635700 1 -- start start 2023-06-10T18:56:58.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb08635700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdb0015a420 0x7fdb000b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb08635700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb000b4ff0 0x7fdb000ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb08635700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb000b6070 con 0x7fdb000ce890 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb08635700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb000cecb0 con 0x7fdb000b4ff0 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb08635700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb000b5460 con 0x7fdb0015a420 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdb000b4ff0 0x7fdb000ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb06bd2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdb000ce890 0x7fdb000b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35752/0 (socket says 172.21.15.104:35752) 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb06bd2700 1 -- 172.21.15.104:0/481577279 learned_addr learned my addr 172.21.15.104:0/481577279 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb063d1700 1 --2- 172.21.15.104:0/481577279 >> v2:172.21.15.114:3300/0 conn(0x7fdb0015a420 0x7fdb000b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 -- 172.21.15.104:0/481577279 >> v2:172.21.15.114:3300/0 conn(0x7fdb0015a420 msgr2=0x7fdb000b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 --2- 172.21.15.104:0/481577279 >> v2:172.21.15.114:3300/0 conn(0x7fdb0015a420 0x7fdb000b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 -- 172.21.15.104:0/481577279 >> v1:172.21.15.104:6789/0 conn(0x7fdb000ce890 legacy=0x7fdb000b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb053cf700 1 -- 172.21.15.104:0/481577279 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049330533 0 0) 0x7fdb000b6070 con 0x7fdb000ce890 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 -- 172.21.15.104:0/481577279 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000b5740 con 0x7fdb000b4ff0 2023-06-10T18:56:58.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.975+0000 7fdb05bd0700 1 --2- 172.21.15.104:0/481577279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000ce350 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fdb000b32e0 tx=0x7fdaf0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8470204e28f393f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb053cf700 1 -- 172.21.15.104:0/481577279 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0011830 con 0x7fdb000b4ff0 2023-06-10T18:56:58.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb053cf700 1 -- 172.21.15.104:0/481577279 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf0011990 con 0x7fdb000b4ff0 2023-06-10T18:56:58.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb053cf700 1 -- 172.21.15.104:0/481577279 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0011d20 con 0x7fdb000b4ff0 2023-06-10T18:56:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 -- 172.21.15.104:0/481577279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 msgr2=0x7fdb000ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 --2- 172.21.15.104:0/481577279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000ce350 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fdb000b32e0 tx=0x7fdaf0009b20 comp rx=0 tx=0).stop 2023-06-10T18:56:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 -- 172.21.15.104:0/481577279 shutdown_connections 2023-06-10T18:56:58.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 --2- 172.21.15.104:0/481577279 >> v2:172.21.15.114:3300/0 conn(0x7fdb0015a420 0x7fdb000b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 --2- 172.21.15.104:0/481577279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000ce350 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 -- 172.21.15.104:0/481577279 >> 172.21.15.104:0/481577279 conn(0x7fdb00089fd0 msgr2=0x7fdb000d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 -- 172.21.15.104:0/481577279 shutdown_connections 2023-06-10T18:56:58.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.976+0000 7fdb08635700 1 -- 172.21.15.104:0/481577279 wait complete. 2023-06-10T18:56:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.977+0000 7fdb08635700 1 Processor -- start 2023-06-10T18:56:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.977+0000 7fdb08635700 1 -- start start 2023-06-10T18:56:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb000ce890 0x7fdb001f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb0015a420 0x7fdb001f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000c8620 con 0x7fdb000b4ff0 2023-06-10T18:56:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000c8900 con 0x7fdb000ce890 2023-06-10T18:56:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb08635700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000c8be0 con 0x7fdb0015a420 2023-06-10T18:56:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37926/0 (socket says 172.21.15.104:37926) 2023-06-10T18:56:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 -- 172.21.15.104:0/1585689738 learned_addr learned my addr 172.21.15.104:0/1585689738 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:56:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 -- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb0015a420 msgr2=0x7fdb001f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb0015a420 0x7fdb001f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 -- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb000ce890 msgr2=0x7fdb001f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:58.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.978+0000 7fdb063d1700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb000ce890 0x7fdb001f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdb063d1700 1 -- 172.21.15.104:0/1585689738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00094a0 con 0x7fdb000b4ff0 2023-06-10T18:56:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdb063d1700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00d800 tx=0x7fdafc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0125f0 con 0x7fdb000b4ff0 2023-06-10T18:56:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc012750 con 0x7fdb000b4ff0 2023-06-10T18:56:58.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb000c8ec0 con 0x7fdb000b4ff0 2023-06-10T18:56:58.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc012b00 con 0x7fdb000b4ff0 2023-06-10T18:56:58.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.979+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb000c8230 con 0x7fdb000b4ff0 2023-06-10T18:56:58.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.980+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb00074050 con 0x7fdb000b4ff0 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.981+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdafc012c60 con 0x7fdb000b4ff0 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.981+0000 7fdaf6ffd700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 0x7fdae406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.981+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fdafc08a030 con 0x7fdb000b4ff0 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.982+0000 7fdb05bd0700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 0x7fdae406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.982+0000 7fdb05bd0700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 0x7fdae406d190 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00090b0 tx=0x7fdaf0010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:58.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:58.985+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdafc0553e0 con 0x7fdb000b4ff0 2023-06-10T18:56:59.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.254+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock"]} v 0) v1 -- 0x7fdb000b4860 con 0x7fdb000b4ff0 2023-06-10T18:56:59.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.840+0000 7fdaf6ffd700 1 -- 172.21.15.104:0/1585689738 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock"]}]=0 v608) v1 ==== 148+0+0 (secure 0 0 0) 0x7fdafc058a00 con 0x7fdb000b4ff0 2023-06-10T18:56:59.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 msgr2=0x7fdae406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 0x7fdae406d190 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00090b0 tx=0x7fdaf0010000 comp rx=0 tx=0).stop 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 msgr2=0x7fdb000c3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00d800 tx=0x7fdafc00dbc0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 shutdown_connections 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb0015a420 0x7fdb001f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb000b4ff0 0x7fdb000c3260 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae406ace0 0x7fdae406d190 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 --2- 172.21.15.104:0/1585689738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb000ce890 0x7fdb001f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 >> 172.21.15.104:0/1585689738 conn(0x7fdb00089fd0 msgr2=0x7fdb00157d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 shutdown_connections 2023-06-10T18:56:59.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.843+0000 7fdb08635700 1 -- 172.21.15.104:0/1585689738 wait complete. 2023-06-10T18:56:59.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:56:59.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.980+0000 7f405c90d700 1 Processor -- start 2023-06-10T18:56:59.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.980+0000 7f405c90d700 1 -- start start 2023-06-10T18:56:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405c90d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f40580b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405c90d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405806e250 con 0x7f405806cde0 2023-06-10T18:56:59.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405c90d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405815c950 con 0x7f405806e6e0 2023-06-10T18:56:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405c90d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40580b1e80 con 0x7f405815c580 2023-06-10T18:56:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f4056d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f405806cde0 0x7f40580ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35766/0 (socket says 172.21.15.104:35766) 2023-06-10T18:56:59.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f4056d9d700 1 -- 172.21.15.104:0/2617925612 learned_addr learned my addr 172.21.15.104:0/2617925612 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:56:59.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389757009 0 0) 0x7f405806e250 con 0x7f405806cde0 2023-06-10T18:56:59.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405659c700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f40580b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:59.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.981+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40400035b0 con 0x7f405806cde0 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3561879588 0 0) 0x7f40400035b0 con 0x7f405806cde0 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 msgr2=0x7f40580b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f40580b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 >> v1:172.21.15.114:6789/0 conn(0x7f405806e6e0 legacy=0x7f40580b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405806d920 con 0x7f405806cde0 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f40500019f0 con 0x7f405806cde0 2023-06-10T18:56:59.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 >> v1:172.21.15.104:6789/0 conn(0x7f405806cde0 legacy=0x7f40580ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f4040003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4040003fb0 0x7f4040004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4040004d90 0x7f4040005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580b1e80 con 0x7f4040003fb0 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404000baf0 con 0x7f4040004d90 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404000bcb0 con 0x7f405815c580 2023-06-10T18:56:59.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4040003fb0 0x7f4040004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405659c700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f4040003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4056d9d700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4040004d90 0x7f4040005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 -- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 msgr2=0x7f4040003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f405815c580 0x7f4040003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 -- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4040004d90 msgr2=0x7f4040005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4040004d90 0x7f4040005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 -- 172.21.15.104:0/2617925612 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400035b0 con 0x7f4040003fb0 2023-06-10T18:56:59.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4056d9d700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4040004d90 0x7f4040005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f4055d9b700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4040003fb0 0x7f4040004850 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4048007fd0 tx=0x7f4040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2668e65448303afa server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480052d0 con 0x7f4040003fb0 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048005470 con 0x7f4040003fb0 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.982+0000 7f405559a700 1 -- 172.21.15.104:0/2617925612 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480038c0 con 0x7f4040003fb0 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 -- 172.21.15.104:0/2617925612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4040003fb0 msgr2=0x7f4040004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:56:59.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 --2- 172.21.15.104:0/2617925612 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4040003fb0 0x7f4040004850 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4048007fd0 tx=0x7f4040005880 comp rx=0 tx=0).stop 2023-06-10T18:56:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f4056d9d700 1 -- 172.21.15.104:0/2617925612 reap_dead start 2023-06-10T18:56:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 -- 172.21.15.104:0/2617925612 shutdown_connections 2023-06-10T18:56:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 -- 172.21.15.104:0/2617925612 >> 172.21.15.104:0/2617925612 conn(0x7f405806b8a0 msgr2=0x7f40580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:56:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 -- 172.21.15.104:0/2617925612 shutdown_connections 2023-06-10T18:56:59.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.983+0000 7f405c90d700 1 -- 172.21.15.104:0/2617925612 wait complete. 2023-06-10T18:56:59.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.985+0000 7f405c90d700 1 Processor -- start 2023-06-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.985+0000 7f405c90d700 1 -- start start 2023-06-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4040003fb0 0x7f4058157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 0x7f40581e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581555b0 con 0x7f40580c3090 2023-06-10T18:57:00.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581564d0 con 0x7f4058155130 2023-06-10T18:57:00.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40581567b0 con 0x7f4040003fb0 2023-06-10T18:57:00.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37948/0 (socket says 172.21.15.104:37948) 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 -- 172.21.15.104:0/1141444693 learned_addr learned my addr 172.21.15.104:0/1141444693 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405659c700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4040003fb0 0x7f4058157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4056d9d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 0x7f40581e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 -- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4040003fb0 msgr2=0x7f4058157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4040003fb0 0x7f4058157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 -- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 msgr2=0x7f40581e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 0x7f40581e5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 -- 172.21.15.104:0/1141444693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058156a90 con 0x7f40580c3090 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4056d9d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 0x7f40581e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4055d9b700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4048004000 tx=0x7f4048001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048007240 con 0x7f40580c3090 2023-06-10T18:57:00.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40480073a0 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4058156d70 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048001290 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.986+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40580c3630 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.987+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40581570e0 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.988+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4048003680 con 0x7f40580c3090 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.988+0000 7f4046ffd700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 0x7f403806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.988+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4048089de0 con 0x7f40580c3090 2023-06-10T18:57:00.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.989+0000 7f405659c700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 0x7f403806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.989+0000 7f405659c700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 0x7f403806d190 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f40581e6e30 tx=0x7f404c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:56:59.993+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40480550b0 con 0x7f40580c3090 2023-06-10T18:57:00.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.182+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f40580d0500 con 0x7f40580c3090 2023-06-10T18:57:00.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.185+0000 7f4046ffd700 1 -- 172.21.15.104:0/1141444693 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f40480586d0 con 0x7f40580c3090 2023-06-10T18:57:00.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 msgr2=0x7f403806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 0x7f403806d190 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f40581e6e30 tx=0x7f404c008040 comp rx=0 tx=0).stop 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 msgr2=0x7f4058158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4048004000 tx=0x7f4048001120 comp rx=0 tx=0).stop 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 shutdown_connections 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f403806ace0 0x7f403806d190 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4058155130 0x7f40581e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4040003fb0 0x7f4058157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 --2- 172.21.15.104:0/1141444693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40580c3090 0x7f4058158210 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 >> 172.21.15.104:0/1141444693 conn(0x7f405806b8a0 msgr2=0x7f40580ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.187+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 shutdown_connections 2023-06-10T18:57:00.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.188+0000 7f405c90d700 1 -- 172.21.15.104:0/1141444693 wait complete. 2023-06-10T18:57:00.205 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:57:00.206 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:00.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:00.312 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:57:00.313 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:00.313 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:00.330 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:57:00.330 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:00.331 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:00.350 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:57:00.350 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:00.351 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:00.387 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:57:00.387 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:00.388 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:00.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:57:00.409 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:57:00.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.540+0000 7f4b4e905700 1 Processor -- start 2023-06-10T18:57:00.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.540+0000 7f4b4e905700 1 -- start start 2023-06-10T18:57:00.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4e905700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4b48158e50 0x7f4b480cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4e905700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b480cb5b0 0x7f4b480c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4e905700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480c8de0 con 0x7f4b480cdbb0 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4e905700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b480ccc90 con 0x7f4b48158e50 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4e905700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480ccf30 con 0x7f4b480cb5b0 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b477fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4b480cdbb0 0x7f4b480caea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35784/0 (socket says 172.21.15.104:35784) 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b477fe700 1 -- 172.21.15.104:0/1416310161 learned_addr learned my addr 172.21.15.104:0/1416310161 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b480cb5b0 0x7f4b480c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b47fff700 1 --2- 172.21.15.104:0/1416310161 >> v2:172.21.15.114:3300/0 conn(0x7f4b48158e50 0x7f4b480cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003211863 0 0) 0x7f4b480c8de0 con 0x7f4b480cdbb0 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b300035b0 con 0x7f4b480cdbb0 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165962667 0 0) 0x7f4b300035b0 con 0x7f4b480cdbb0 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b480cb5b0 msgr2=0x7f4b480c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b480cb5b0 0x7f4b480c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 >> v2:172.21.15.114:3300/0 conn(0x7f4b48158e50 msgr2=0x7f4b480cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 --2- 172.21.15.104:0/1416310161 >> v2:172.21.15.114:3300/0 conn(0x7f4b48158e50 0x7f4b480cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480cd210 con 0x7f4b480cdbb0 2023-06-10T18:57:00.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4b38001670 con 0x7f4b480cdbb0 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 >> v1:172.21.15.104:6789/0 conn(0x7f4b480cdbb0 legacy=0x7f4b480caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.541+0000 7f4b46ffd700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b30003a00 0x7f4b30003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b300043e0 0x7f4b300048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b30004f20 0x7f4b3000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480ccc90 con 0x7f4b300043e0 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480ccf30 con 0x7f4b30004f20 2023-06-10T18:57:00.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3000bf50 con 0x7f4b30003a00 2023-06-10T18:57:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b480cb5b0 0x7f4b480c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b300043e0 0x7f4b300048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b47fff700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b30003a00 0x7f4b30003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b30004f20 0x7f4b3000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 -- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b30003a00 msgr2=0x7f4b30003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b30003a00 0x7f4b30003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 -- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b30004f20 msgr2=0x7f4b3000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b30004f20 0x7f4b3000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 -- 172.21.15.104:0/1416310161 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300035b0 con 0x7f4b300043e0 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b477fe700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b300043e0 0x7f4b300048b0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c320 tx=0x7f4b30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e280c0f95c0c1bf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800ce60 con 0x7f4b300043e0 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b38005960 con 0x7f4b300043e0 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.542+0000 7f4b46ffd700 1 -- 172.21.15.104:0/1416310161 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38005cf0 con 0x7f4b300043e0 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 -- 172.21.15.104:0/1416310161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b300043e0 msgr2=0x7f4b300048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 --2- 172.21.15.104:0/1416310161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b300043e0 0x7f4b300048b0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c320 tx=0x7f4b30005880 comp rx=0 tx=0).stop 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4cea2700 1 -- 172.21.15.104:0/1416310161 reap_dead start 2023-06-10T18:57:00.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 -- 172.21.15.104:0/1416310161 shutdown_connections 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 -- 172.21.15.104:0/1416310161 >> 172.21.15.104:0/1416310161 conn(0x7f4b4806ab00 msgr2=0x7f4b480b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 -- 172.21.15.104:0/1416310161 shutdown_connections 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.543+0000 7f4b4e905700 1 -- 172.21.15.104:0/1416310161 wait complete. 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.544+0000 7f4b4e905700 1 Processor -- start 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.544+0000 7f4b4e905700 1 -- start start 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 0x7f4b480caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 0x7f4b480c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c6580 con 0x7f4b48153590 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c5fc0 con 0x7f4b481576c0 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4e905700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c62a0 con 0x7f4b300043e0 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:37970/0 (socket says 172.21.15.104:37970) 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 -- 172.21.15.104:0/2713750213 learned_addr learned my addr 172.21.15.104:0/2713750213 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:00.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b47fff700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 0x7f4b480caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b477fe700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 0x7f4b480c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 -- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 msgr2=0x7f4b480caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 0x7f4b480caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 -- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 msgr2=0x7f4b480c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 0x7f4b480c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 -- 172.21.15.104:0/2713750213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480c6b80 con 0x7f4b48153590 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b47fff700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 0x7f4b480caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:00.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b477fe700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 0x7f4b480c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.545+0000 7f4b4cea2700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f4b480b46c0 tx=0x7f4b40011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.546+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000e070 con 0x7f4b48153590 2023-06-10T18:57:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.546+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b40003a20 con 0x7f4b48153590 2023-06-10T18:57:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.546+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b480c6e60 con 0x7f4b48153590 2023-06-10T18:57:00.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.546+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40009ea0 con 0x7f4b48153590 2023-06-10T18:57:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.546+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b480cb0e0 con 0x7f4b48153590 2023-06-10T18:57:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.547+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b48074050 con 0x7f4b48153590 2023-06-10T18:57:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.547+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4b40003bd0 con 0x7f4b48153590 2023-06-10T18:57:00.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.547+0000 7f4b44ff9700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 0x7f4b3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.547+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4b40089b10 con 0x7f4b48153590 2023-06-10T18:57:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.548+0000 7f4b47fff700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 0x7f4b3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.548+0000 7f4b47fff700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 0x7f4b3406d190 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009ae0 tx=0x7f4b3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.552+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4b40054d50 con 0x7f4b48153590 2023-06-10T18:57:00.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.780+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b480b0ab0 con 0x7f4b48153590 2023-06-10T18:57:00.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.780+0000 7f4b44ff9700 1 -- 172.21.15.104:0/2713750213 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f4b40058370 con 0x7f4b48153590 2023-06-10T18:57:00.787 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:00.787 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":82,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3722420040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3722420040},{"type":"v1","addr":"172.21.15.114:6837","nonce":3722420040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":7100,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1729964946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1729964946},{"type":"v1","addr":"172.21.15.114:6836","nonce":1729964946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/3151674726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3151674726},{"type":"v1","addr":"172.21.15.104:6834","nonce":3151674726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9222,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3323203100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3323203100},{"type":"v1","addr":"172.21.15.104:6835","nonce":3323203100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[]} 2023-06-10T18:57:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 msgr2=0x7f4b3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 0x7f4b3406d190 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009ae0 tx=0x7f4b3c008040 comp rx=0 tx=0).stop 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 msgr2=0x7f4b48153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f4b480b46c0 tx=0x7f4b40011790 comp rx=0 tx=0).stop 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 shutdown_connections 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b48153590 0x7f4b48153a30 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b300043e0 0x7f4b480caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b3406ace0 0x7f4b3406d190 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.782+0000 7f4b4e905700 1 --2- 172.21.15.104:0/2713750213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b481576c0 0x7f4b480c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.783+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 >> 172.21.15.104:0/2713750213 conn(0x7f4b4806ab00 msgr2=0x7f4b48070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.783+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 shutdown_connections 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.783+0000 7f4b4e905700 1 -- 172.21.15.104:0/2713750213 wait complete. 2023-06-10T18:57:00.790 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 82 2023-06-10T18:57:00.801 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:57:00.801 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:57:00.801 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:57:00.802 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:00.802 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:57:00.802 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:57:00.802 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:57:00.802 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:00.803 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:57:00.803 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:57:00.803 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:57:00.803 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:00.804 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:57:00.804 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:57:00.804 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:57:00.804 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:00.811 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:57:00.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:57:00.813 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:57:00.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:57:00.814 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:57:00.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:57:00.815 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:57:00.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:57:00.855 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:57:00.855 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:00.856 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:57:00.856 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:57:00.857 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:00.857 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:00.858 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:57:00.858 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:00.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:57:00.932 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:57:00.933 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:57:00.940 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:57:00.941 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:57:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.982+0000 7f876a607700 1 Processor -- start 2023-06-10T18:57:00.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.982+0000 7f876a607700 1 -- start start 2023-06-10T18:57:00.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f876a607700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f876a607700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f876415c7a0 0x7f87640cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f876a607700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87640cb2d0 con 0x7f87640701a0 2023-06-10T18:57:00.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f876a607700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876415cca0 con 0x7f876415c7a0 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f876a607700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876415cf80 con 0x7f87641567f0 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f876415c7a0 0x7f87640cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8768ba4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f87640701a0 0x7f8764160150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35830/0 (socket says 172.21.15.104:35830) 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f876415c7a0 0x7f87640cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38000/0 (socket says 172.21.15.104:38000) 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8768ba4700 1 -- 172.21.15.104:0/522738341 learned_addr learned my addr 172.21.15.104:0/522738341 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 -- 172.21.15.104:0/522738341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 msgr2=0x7f87640cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:00.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8763fff700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8762ffd700 1 -- 172.21.15.104:0/522738341 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304348613 0 0) 0x7f87640cb2d0 con 0x7f87640701a0 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 -- 172.21.15.104:0/522738341 >> v1:172.21.15.104:6789/0 conn(0x7f87640701a0 legacy=0x7f8764160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 -- 172.21.15.104:0/522738341 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876415d260 con 0x7f876415c7a0 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8763fff700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f87637fe700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640cd560 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8758009840 tx=0x7f8758009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b466a406aba15c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:00.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8762ffd700 1 -- 172.21.15.104:0/522738341 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758011830 con 0x7f876415c7a0 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8762ffd700 1 -- 172.21.15.104:0/522738341 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758011990 con 0x7f876415c7a0 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.983+0000 7f8762ffd700 1 -- 172.21.15.104:0/522738341 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758011d20 con 0x7f876415c7a0 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 -- 172.21.15.104:0/522738341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 msgr2=0x7f87640cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640cd560 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f8758009840 tx=0x7f8758009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 -- 172.21.15.104:0/522738341 shutdown_connections 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 --2- 172.21.15.104:0/522738341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640cd560 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 -- 172.21.15.104:0/522738341 >> 172.21.15.104:0/522738341 conn(0x7f876406e370 msgr2=0x7f876406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:00.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 -- 172.21.15.104:0/522738341 shutdown_connections 2023-06-10T18:57:00.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.984+0000 7f876a607700 1 -- 172.21.15.104:0/522738341 wait complete. 2023-06-10T18:57:00.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.985+0000 7f876a607700 1 Processor -- start 2023-06-10T18:57:00.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.985+0000 7f876a607700 1 -- start start 2023-06-10T18:57:00.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87640701a0 0x7f87640b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b6440 con 0x7f876415c7a0 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640c95e0 con 0x7f87640701a0 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f876a607700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640c6c00 con 0x7f87641567f0 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f8768ba4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f8768ba4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38008/0 (socket says 172.21.15.104:38008) 2023-06-10T18:57:00.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f8768ba4700 1 -- 172.21.15.104:0/565526088 learned_addr learned my addr 172.21.15.104:0/565526088 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f87637fe700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.986+0000 7f8763fff700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87640701a0 0x7f87640b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 -- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 msgr2=0x7f87640c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 -- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87640701a0 msgr2=0x7f87640b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87640701a0 0x7f87640b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:00.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 -- 172.21.15.104:0/565526088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87580094a0 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8768ba4700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f875c00cba0 tx=0x7f875c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c00aaa0 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c00ac20 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87640c6ee0 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0070d0 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.987+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87640b3610 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.988+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8764074050 con 0x7f876415c7a0 2023-06-10T18:57:01.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.988+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f875c007230 con 0x7f876415c7a0 2023-06-10T18:57:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.989+0000 7f8760ff9700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 0x7f875006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.989+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f875c088fe0 con 0x7f876415c7a0 2023-06-10T18:57:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.989+0000 7f8763fff700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 0x7f875006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.989+0000 7f8763fff700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 0x7f875006d190 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8754006fd0 tx=0x7f8754008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:01.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:00.993+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f875c0542b0 con 0x7f876415c7a0 2023-06-10T18:57:01.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.195+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f87640b66c0 con 0x7f876415c7a0 2023-06-10T18:57:01.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.990+0000 7f8760ff9700 1 -- 172.21.15.104:0/565526088 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v139) v1 ==== 97+0+0 (secure 0 0 0) 0x7f875c018080 con 0x7f876415c7a0 2023-06-10T18:57:01.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 msgr2=0x7f875006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:01.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 0x7f875006d190 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8754006fd0 tx=0x7f8754008040 comp rx=0 tx=0).stop 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 msgr2=0x7f87640c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f875c00cba0 tx=0x7f875c00ceb0 comp rx=0 tx=0).stop 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 shutdown_connections 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f875006ace0 0x7f875006d190 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87640701a0 0x7f87640b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876415c7a0 0x7f87640c5b60 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:01.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 --2- 172.21.15.104:0/565526088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87641567f0 0x7f87640c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 >> 172.21.15.104:0/565526088 conn(0x7f876406e370 msgr2=0x7f876406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 shutdown_connections 2023-06-10T18:57:02.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:01.992+0000 7f876a607700 1 -- 172.21.15.104:0/565526088 wait complete. 2023-06-10T18:57:02.000 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:57:02.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:57:02.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.126+0000 7f69fa8d5700 1 Processor -- start 2023-06-10T18:57:02.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.126+0000 7f69fa8d5700 1 -- start start 2023-06-10T18:57:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69fa8d5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f69f40cd850 0x7f69f406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69fa8d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cb300 0x7f69f406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69fa8d5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40c9e60 con 0x7f69f40aeaa0 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69fa8d5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40c9bc0 con 0x7f69f40cd850 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69fa8d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40aee70 con 0x7f69f40cb300 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69f37fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f69f40cd850 0x7f69f406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69f3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f69f40aeaa0 0x7f69f40cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35844/0 (socket says 172.21.15.104:35844) 2023-06-10T18:57:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.127+0000 7f69f3fff700 1 -- 172.21.15.104:0/4035301770 learned_addr learned my addr 172.21.15.104:0/4035301770 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f37fe700 1 -- 172.21.15.104:0/4035301770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cb300 msgr2=0x7f69f406d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f37fe700 1 --2- 172.21.15.104:0/4035301770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cb300 0x7f69f406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f37fe700 1 -- 172.21.15.104:0/4035301770 >> v1:172.21.15.104:6789/0 conn(0x7f69f40aeaa0 legacy=0x7f69f40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f37fe700 1 -- 172.21.15.104:0/4035301770 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f406e540 con 0x7f69f40cd850 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f2ffd700 1 -- 172.21.15.104:0/4035301770 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673911549 0 0) 0x7f69f40c9e60 con 0x7f69f40aeaa0 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f37fe700 1 --2- 172.21.15.104:0/4035301770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40cd850 0x7f69f406cd50 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009840 tx=0x7f69e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9292ff6c9734bcc1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f2ffd700 1 -- 172.21.15.104:0/4035301770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011830 con 0x7f69f40cd850 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f2ffd700 1 -- 172.21.15.104:0/4035301770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4011990 con 0x7f69f40cd850 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69f2ffd700 1 -- 172.21.15.104:0/4035301770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011d20 con 0x7f69f40cd850 2023-06-10T18:57:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69fa8d5700 1 -- 172.21.15.104:0/4035301770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40cd850 msgr2=0x7f69f406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.128+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/4035301770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40cd850 0x7f69f406cd50 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009840 tx=0x7f69e4009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:02.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 -- 172.21.15.104:0/4035301770 shutdown_connections 2023-06-10T18:57:02.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/4035301770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cb300 0x7f69f406d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/4035301770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40cd850 0x7f69f406cd50 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 -- 172.21.15.104:0/4035301770 >> 172.21.15.104:0/4035301770 conn(0x7f69f406b8a0 msgr2=0x7f69f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 -- 172.21.15.104:0/4035301770 shutdown_connections 2023-06-10T18:57:02.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.129+0000 7f69fa8d5700 1 -- 172.21.15.104:0/4035301770 wait complete. 2023-06-10T18:57:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.130+0000 7f69fa8d5700 1 Processor -- start 2023-06-10T18:57:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.130+0000 7f69fa8d5700 1 -- start start 2023-06-10T18:57:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f40cb300 0x7f69f40c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cd850 0x7f69f4156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f415a460 con 0x7f69f40aeaa0 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f415a740 con 0x7f69f40cb300 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f415aa20 con 0x7f69f40cd850 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38022/0 (socket says 172.21.15.104:38022) 2023-06-10T18:57:02.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 -- 172.21.15.104:0/2212327552 learned_addr learned my addr 172.21.15.104:0/2212327552 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f37fe700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f40cb300 0x7f69f40c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f8e72700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cd850 0x7f69f4156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 -- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cd850 msgr2=0x7f69f4156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cd850 0x7f69f4156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 -- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f40cb300 msgr2=0x7f69f40c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f40cb300 0x7f69f40c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 -- 172.21.15.104:0/2212327552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40094a0 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f3fff700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e910 tx=0x7f69e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8005a30 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8005b90 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8007070 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f415ad00 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.131+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f40c8800 con 0x7f69f40aeaa0 2023-06-10T18:57:02.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.132+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f415b220 con 0x7f69f40aeaa0 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.133+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f69e80071d0 con 0x7f69f40aeaa0 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.133+0000 7f69f0ff9700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 0x7f69e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.133+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f69e808b590 con 0x7f69f40aeaa0 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.133+0000 7f69f37fe700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 0x7f69e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.134+0000 7f69f37fe700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 0x7f69e006d190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f69f41f1b40 tx=0x7f69e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.138+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f69f415b220 con 0x7f69f40aeaa0 2023-06-10T18:57:02.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.398+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f69f4074050 con 0x7f69f40aeaa0 2023-06-10T18:57:02.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.398+0000 7f69f0ff9700 1 -- 172.21.15.104:0/2212327552 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f69e8056850 con 0x7f69f40aeaa0 2023-06-10T18:57:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:02.405 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:57:02.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:02.406 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.407 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:57:02.408 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.409 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.410 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.413 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.414 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:57:02.415 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.418 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.419 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.420 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:57:02.422 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:02.423 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:57:02.424 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.425 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:02.427 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.428 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:57:02.429 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:57:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:57:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:02.430 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:02.431 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:57:02.431 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:02.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.400+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 msgr2=0x7f69e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.400+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 0x7f69e006d190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f69f41f1b40 tx=0x7f69e4008030 comp rx=0 tx=0).stop 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 msgr2=0x7f69f4159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e910 tx=0x7f69e800ecd0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 shutdown_connections 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69f40cd850 0x7f69f4156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69f40aeaa0 0x7f69f4159e30 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f69e006ace0 0x7f69e006d190 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 --2- 172.21.15.104:0/2212327552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f40cb300 0x7f69f40c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 >> 172.21.15.104:0/2212327552 conn(0x7f69f406b8a0 msgr2=0x7f69f40b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 shutdown_connections 2023-06-10T18:57:02.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.401+0000 7f69fa8d5700 1 -- 172.21.15.104:0/2212327552 wait complete. 2023-06-10T18:57:02.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:02.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 Processor -- start 2023-06-10T18:57:02.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 -- start start 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f31d80ce510 0x7f31d80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d80cee20 con 0x7f31d80705b0 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d8071fe0 con 0x7f31d8070130 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31dd50b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80caa90 con 0x7f31d80ce510 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.549+0000 7f31d6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f31d80ce510 0x7f31d80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d77fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f31d80705b0 0x7f31d80cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35860/0 (socket says 172.21.15.104:35860) 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d77fe700 1 -- 172.21.15.104:0/4102851228 learned_addr learned my addr 172.21.15.104:0/4102851228 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d6ffd700 1 -- 172.21.15.104:0/4102851228 >> v1:172.21.15.114:6789/0 conn(0x7f31d8070130 legacy=0x7f31d80c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d6ffd700 1 -- 172.21.15.104:0/4102851228 >> v1:172.21.15.104:6789/0 conn(0x7f31d80705b0 legacy=0x7f31d80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d5ffb700 1 -- 172.21.15.104:0/4102851228 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39212489 0 0) 0x7f31d80cee20 con 0x7f31d80705b0 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d6ffd700 1 -- 172.21.15.104:0/4102851228 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d80722c0 con 0x7f31d80ce510 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d6ffd700 1 --2- 172.21.15.104:0/4102851228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d80ce510 0x7f31d80ce8e0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009840 tx=0x7f31c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a254962eb54fe8a4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d5ffb700 1 -- 172.21.15.104:0/4102851228 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011830 con 0x7f31d80ce510 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d5ffb700 1 -- 172.21.15.104:0/4102851228 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8011990 con 0x7f31d80ce510 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.550+0000 7f31d5ffb700 1 -- 172.21.15.104:0/4102851228 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8011d20 con 0x7f31d80ce510 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 -- 172.21.15.104:0/4102851228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d80ce510 msgr2=0x7f31d80ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 --2- 172.21.15.104:0/4102851228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d80ce510 0x7f31d80ce8e0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009840 tx=0x7f31c8009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 -- 172.21.15.104:0/4102851228 shutdown_connections 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 --2- 172.21.15.104:0/4102851228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d80ce510 0x7f31d80ce8e0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 -- 172.21.15.104:0/4102851228 >> 172.21.15.104:0/4102851228 conn(0x7f31d806ab00 msgr2=0x7f31d80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 -- 172.21.15.104:0/4102851228 shutdown_connections 2023-06-10T18:57:02.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.551+0000 7f31dd50b700 1 -- 172.21.15.104:0/4102851228 wait complete. 2023-06-10T18:57:02.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.552+0000 7f31dd50b700 1 Processor -- start 2023-06-10T18:57:02.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.552+0000 7f31dd50b700 1 -- start start 2023-06-10T18:57:02.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 0x7f31d80c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31d80ce510 0x7f31d80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80cde70 con 0x7f31d8070130 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80ab8c0 con 0x7f31d80ce510 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d814ffe0 con 0x7f31d80705b0 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38034/0 (socket says 172.21.15.104:38034) 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 -- 172.21.15.104:0/1636362090 learned_addr learned my addr 172.21.15.104:0/1636362090 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d77fe700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31d80ce510 0x7f31d80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d67fc700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 0x7f31d80c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 -- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 msgr2=0x7f31d80c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 0x7f31d80c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 -- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31d80ce510 msgr2=0x7f31d80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31d80ce510 0x7f31d80c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 -- 172.21.15.104:0/1636362090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c80094a0 con 0x7f31d8070130 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d67fc700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 0x7f31d80c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:02.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31d6ffd700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f31d80ce4e0 tx=0x7f31c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c80036a0 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c801f070 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d80c70a0 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8003a20 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.553+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d80c7660 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.554+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d8074050 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.555+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f31c8003b80 con 0x7f31d8070130 2023-06-10T18:57:02.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.555+0000 7f31bf7fe700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 0x7f31c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.555+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f31c8021070 con 0x7f31d8070130 2023-06-10T18:57:02.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.556+0000 7f31d67fc700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 0x7f31c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.556+0000 7f31d67fc700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 0x7f31c406d190 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f31d80c8850 tx=0x7f31cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.560+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f31c8055f80 con 0x7f31d8070130 2023-06-10T18:57:02.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.746+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31d80ae7f0 con 0x7f31d8070130 2023-06-10T18:57:02.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.747+0000 7f31bf7fe700 1 -- 172.21.15.104:0/1636362090 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+14902 (secure 0 0 0) 0x7f31c8015090 con 0x7f31d8070130 2023-06-10T18:57:02.753 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:02.753 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":139,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:01.982678+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 msgr2=0x7f31c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 0x7f31c406d190 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f31d80c8850 tx=0x7f31cc009040 comp rx=0 tx=0).stop 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 msgr2=0x7f31d814faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f31d80ce4e0 tx=0x7f31c8004750 comp rx=0 tx=0).stop 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 shutdown_connections 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f31c406ace0 0x7f31c406d190 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31d80ce510 0x7f31d80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31d8070130 0x7f31d814faa0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 --2- 172.21.15.104:0/1636362090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31d80705b0 0x7f31d80c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 >> 172.21.15.104:0/1636362090 conn(0x7f31d806ab00 msgr2=0x7f31d80cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 shutdown_connections 2023-06-10T18:57:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.749+0000 7f31dd50b700 1 -- 172.21.15.104:0/1636362090 wait complete. 2023-06-10T18:57:02.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:57:02.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:57:02.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.885+0000 7fc2d7ec1700 1 Processor -- start 2023-06-10T18:57:02.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.885+0000 7fc2d7ec1700 1 -- start start 2023-06-10T18:57:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d7ec1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2d006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d7ec1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d006d7d0 con 0x7fc2d0071120 2023-06-10T18:57:02.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d7ec1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d00cd6c0 con 0x7fc2d00ce390 2023-06-10T18:57:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d7ec1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00c95d0 con 0x7fc2d01586a0 2023-06-10T18:57:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d645e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc2d0071120 0x7fc2d00c9a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35872/0 (socket says 172.21.15.104:35872) 2023-06-10T18:57:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.886+0000 7fc2d645e700 1 -- 172.21.15.104:0/2729136049 learned_addr learned my addr 172.21.15.104:0/2729136049 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:02.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038425761 0 0) 0x7fc2d006d7d0 con 0x7fc2d0071120 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2bc0035b0 con 0x7fc2d0071120 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2d006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945206210 0 0) 0x7fc2bc0035b0 con 0x7fc2d0071120 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 msgr2=0x7fc2d006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2d006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 >> v1:172.21.15.114:6789/0 conn(0x7fc2d00ce390 legacy=0x7fc2d0070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d006d530 con 0x7fc2d0071120 2023-06-10T18:57:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc2c80019f0 con 0x7fc2d0071120 2023-06-10T18:57:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 >> v1:172.21.15.104:6789/0 conn(0x7fc2d0071120 legacy=0x7fc2d00c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2bc003a30 0x7fc2bc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc004d90 0x7fc2bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00c95d0 con 0x7fc2bc003a30 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc00bbb0 con 0x7fc2bc004d90 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc00bd70 con 0x7fc2d01586a0 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2bc003a30 0x7fc2bc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d545c700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d645e700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc004d90 0x7fc2bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 -- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 msgr2=0x7fc2bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d01586a0 0x7fc2bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 -- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc004d90 msgr2=0x7fc2bc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2bc004d90 0x7fc2bc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 -- 172.21.15.104:0/2729136049 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0035b0 con 0x7fc2bc003a30 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.887+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2bc003a30 0x7fc2bc003e40 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000aaa0 tx=0x7fc2bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8893aa3f8ad00c7e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000a6b0 con 0x7fc2bc003a30 2023-06-10T18:57:02.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c000f970 con 0x7fc2bc003a30 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d4c5b700 1 -- 172.21.15.104:0/2729136049 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000fd00 con 0x7fc2bc003a30 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/2729136049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2bc003a30 msgr2=0x7fc2bc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/2729136049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2bc003a30 0x7fc2bc003e40 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000aaa0 tx=0x7fc2bc005880 comp rx=0 tx=0).stop 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d645e700 1 -- 172.21.15.104:0/2729136049 reap_dead start 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/2729136049 shutdown_connections 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/2729136049 >> 172.21.15.104:0/2729136049 conn(0x7fc2d00ceec0 msgr2=0x7fc2d00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:02.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/2729136049 shutdown_connections 2023-06-10T18:57:02.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.888+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/2729136049 wait complete. 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.889+0000 7fc2d7ec1700 1 Processor -- start 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.889+0000 7fc2d7ec1700 1 -- start start 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2bc003a30 0x7fc2d014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d00b2800 0x7fc2d00b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00c8960 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d014e680 con 0x7fc2d00b2800 2023-06-10T18:57:02.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d7ec1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00b3250 con 0x7fc2bc003a30 2023-06-10T18:57:02.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d645e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d645e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38058/0 (socket says 172.21.15.104:38058) 2023-06-10T18:57:02.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d645e700 1 -- 172.21.15.104:0/755718882 learned_addr learned my addr 172.21.15.104:0/755718882 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:02.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2bc003a30 0x7fc2d014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.890+0000 7fc2d545c700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d00b2800 0x7fc2d00b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 -- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2bc003a30 msgr2=0x7fc2d014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2bc003a30 0x7fc2d014e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 -- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d00b2800 msgr2=0x7fc2d00b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d00b2800 0x7fc2d00b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:02.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 -- 172.21.15.104:0/755718882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d00b3530 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d645e700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006cc50 tx=0x7fc2c8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c800ece0 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c800ee40 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d00b3810 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8013660 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.891+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d00b3dd0 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.892+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d0152450 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.892+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc2c80137c0 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.893+0000 7fc2c67fc700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 0x7fc2b406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.893+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc2c8089960 con 0x7fc2d00c7fb0 2023-06-10T18:57:02.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.893+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 0x7fc2b406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:02.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.893+0000 7fc2d5c5d700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 0x7fc2b406d1c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0000c00 tx=0x7fc2c000d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:02.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:02.897+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc2c8054c50 con 0x7fc2d00c7fb0 2023-06-10T18:57:03.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.081+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc2d0074050 con 0x7fc2d00c7fb0 2023-06-10T18:57:03.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.081+0000 7fc2c67fc700 1 -- 172.21.15.104:0/755718882 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2c8058270 con 0x7fc2d00c7fb0 2023-06-10T18:57:03.088 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:57:03.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 msgr2=0x7fc2b406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 0x7fc2b406d1c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0000c00 tx=0x7fc2c000d000 comp rx=0 tx=0).stop 2023-06-10T18:57:03.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 msgr2=0x7fc2d00c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006cc50 tx=0x7fc2c8007c90 comp rx=0 tx=0).stop 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 shutdown_connections 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00c7fb0 0x7fc2d00c8420 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2bc003a30 0x7fc2d014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b406ad10 0x7fc2b406d1c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 --2- 172.21.15.104:0/755718882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d00b2800 0x7fc2d00b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 >> 172.21.15.104:0/755718882 conn(0x7fc2d00ceec0 msgr2=0x7fc2d00c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 shutdown_connections 2023-06-10T18:57:03.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.083+0000 7fc2d7ec1700 1 -- 172.21.15.104:0/755718882 wait complete. 2023-06-10T18:57:03.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:57:03.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.217+0000 7f289cf6e700 1 Processor -- start 2023-06-10T18:57:03.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.218+0000 7f289cf6e700 1 -- start start 2023-06-10T18:57:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.218+0000 7f289cf6e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28980cb210 0x7f28980cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.218+0000 7f289cf6e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 0x7f28980cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289cf6e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28980cc9f0 con 0x7f28980cd720 2023-06-10T18:57:03.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289cf6e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28980cbbd0 con 0x7f28980cb210 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289cf6e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28980cdb20 con 0x7f28980cf0b0 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f28980cb210 0x7f28980cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f2896d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f28980cd720 0x7f28980c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35892/0 (socket says 172.21.15.104:35892) 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f2896d9d700 1 -- 172.21.15.104:0/1525721041 learned_addr learned my addr 172.21.15.104:0/1525721041 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f2895d9b700 1 --2- 172.21.15.104:0/1525721041 >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 0x7f28980cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 -- 172.21.15.104:0/1525721041 >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 msgr2=0x7f28980cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 --2- 172.21.15.104:0/1525721041 >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 0x7f28980cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 -- 172.21.15.104:0/1525721041 >> v1:172.21.15.104:6789/0 conn(0x7f28980cd720 legacy=0x7f28980c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289559a700 1 -- 172.21.15.104:0/1525721041 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690266382 0 0) 0x7f28980cc9f0 con 0x7f28980cd720 2023-06-10T18:57:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 -- 172.21.15.104:0/1525721041 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28980cc100 con 0x7f28980cb210 2023-06-10T18:57:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f2895d9b700 1 --2- 172.21.15.104:0/1525721041 >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 0x7f28980cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:03.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289659c700 1 --2- 172.21.15.104:0/1525721041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cb210 0x7f28980cb5e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f28980af020 tx=0x7f288c009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8cd1ef9c259dfc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289559a700 1 -- 172.21.15.104:0/1525721041 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c005e30 con 0x7f28980cb210 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289559a700 1 -- 172.21.15.104:0/1525721041 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c006030 con 0x7f28980cb210 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.219+0000 7f289559a700 1 -- 172.21.15.104:0/1525721041 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c0063c0 con 0x7f28980cb210 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 -- 172.21.15.104:0/1525721041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cb210 msgr2=0x7f28980cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 --2- 172.21.15.104:0/1525721041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cb210 0x7f28980cb5e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f28980af020 tx=0x7f288c009b00 comp rx=0 tx=0).stop 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 -- 172.21.15.104:0/1525721041 shutdown_connections 2023-06-10T18:57:03.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 --2- 172.21.15.104:0/1525721041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cb210 0x7f28980cb5e0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 --2- 172.21.15.104:0/1525721041 >> v2:172.21.15.114:3300/0 conn(0x7f28980cf0b0 0x7f28980cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 -- 172.21.15.104:0/1525721041 >> 172.21.15.104:0/1525721041 conn(0x7f289806b8a0 msgr2=0x7f28980cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 -- 172.21.15.104:0/1525721041 shutdown_connections 2023-06-10T18:57:03.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.220+0000 7f289cf6e700 1 -- 172.21.15.104:0/1525721041 wait complete. 2023-06-10T18:57:03.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.221+0000 7f289cf6e700 1 Processor -- start 2023-06-10T18:57:03.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.221+0000 7f289cf6e700 1 -- start start 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28980cb210 0x7f2898150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 0x7f289814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28980c4b60 con 0x7f28980cf0b0 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28980b5540 con 0x7f28980cb210 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289cf6e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28980c4d90 con 0x7f28980cd720 2023-06-10T18:57:03.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38072/0 (socket says 172.21.15.104:38072) 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 -- 172.21.15.104:0/3071903786 learned_addr learned my addr 172.21.15.104:0/3071903786 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f289659c700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28980cb210 0x7f2898150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2895d9b700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 0x7f289814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 -- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 msgr2=0x7f289814e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 0x7f289814e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 -- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28980cb210 msgr2=0x7f2898150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28980cb210 0x7f2898150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 -- 172.21.15.104:0/3071903786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c009430 con 0x7f28980cf0b0 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2895d9b700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 0x7f289814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.222+0000 7f2896d9d700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f2890002230 tx=0x7f289000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.223+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289000ac20 con 0x7f28980cf0b0 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.223+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289000ada0 con 0x7f28980cf0b0 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.223+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28980c5070 con 0x7f28980cf0b0 2023-06-10T18:57:03.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.223+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890007310 con 0x7f28980cf0b0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.223+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28980c5630 con 0x7f28980cf0b0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.224+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2898074050 con 0x7f28980cf0b0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.224+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2890007470 con 0x7f28980cf0b0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.224+0000 7f2886ffd700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 0x7f287806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.224+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f28900891f0 con 0x7f28980cf0b0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.225+0000 7f289659c700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 0x7f287806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.225+0000 7f289659c700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 0x7f287806d190 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.229+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f28900544b0 con 0x7f28980cf0b0 2023-06-10T18:57:03.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.413+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f28980ce9b0 con 0x7f28980cf0b0 2023-06-10T18:57:03.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.413+0000 7f2886ffd700 1 -- 172.21.15.104:0/3071903786 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v139) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2890018020 con 0x7f28980cf0b0 2023-06-10T18:57:03.421 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:57:03.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 msgr2=0x7f287806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 0x7f287806d190 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c008070 comp rx=0 tx=0).stop 2023-06-10T18:57:03.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 msgr2=0x7f28980c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f2890002230 tx=0x7f289000a060 comp rx=0 tx=0).stop 2023-06-10T18:57:03.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 shutdown_connections 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f287806ace0 0x7f287806d190 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28980cb210 0x7f2898150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28980cf0b0 0x7f28980c4620 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 --2- 172.21.15.104:0/3071903786 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28980cd720 0x7f289814e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 >> 172.21.15.104:0/3071903786 conn(0x7f289806b8a0 msgr2=0x7f28980b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 shutdown_connections 2023-06-10T18:57:03.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.415+0000 7f289cf6e700 1 -- 172.21.15.104:0/3071903786 wait complete. 2023-06-10T18:57:03.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:57:03.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.546+0000 7f9a872d2700 1 Processor -- start 2023-06-10T18:57:03.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.547+0000 7f9a872d2700 1 -- start start 2023-06-10T18:57:03.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.547+0000 7f9a872d2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 0x7f9a800cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.547+0000 7f9a872d2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a8008a3d0 0x7f9a800cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.547+0000 7f9a872d2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a800cadd0 con 0x7f9a800ce350 2023-06-10T18:57:03.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a872d2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800cfeb0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a872d2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800cb0b0 con 0x7f9a8006b1f0 2023-06-10T18:57:03.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9a8008a3d0 0x7f9a800cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8486d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9a800ce350 0x7f9a800cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35904/0 (socket says 172.21.15.104:35904) 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8486d700 1 -- 172.21.15.104:0/3703718110 learned_addr learned my addr 172.21.15.104:0/3703718110 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8506e700 1 --2- 172.21.15.104:0/3703718110 >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 0x7f9a800cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 -- 172.21.15.104:0/3703718110 >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 msgr2=0x7f9a800cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 --2- 172.21.15.104:0/3703718110 >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 0x7f9a800cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 -- 172.21.15.104:0/3703718110 >> v1:172.21.15.104:6789/0 conn(0x7f9a800ce350 legacy=0x7f9a800cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 -- 172.21.15.104:0/3703718110 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800cf550 con 0x7f9a8008a3d0 2023-06-10T18:57:03.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a77fff700 1 -- 172.21.15.104:0/3703718110 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898247543 0 0) 0x7f9a800cadd0 con 0x7f9a800ce350 2023-06-10T18:57:03.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8506e700 1 --2- 172.21.15.104:0/3703718110 >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 0x7f9a800cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:03.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a8586f700 1 --2- 172.21.15.104:0/3703718110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a800cb540 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f9a8006e2d0 tx=0x7f9a7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8726843dc864b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a77fff700 1 -- 172.21.15.104:0/3703718110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800e070 con 0x7f9a8008a3d0 2023-06-10T18:57:03.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a77fff700 1 -- 172.21.15.104:0/3703718110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7800bbd0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.548+0000 7f9a77fff700 1 -- 172.21.15.104:0/3703718110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78012430 con 0x7f9a8008a3d0 2023-06-10T18:57:03.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 -- 172.21.15.104:0/3703718110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 msgr2=0x7f9a800cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 --2- 172.21.15.104:0/3703718110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a800cb540 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f9a8006e2d0 tx=0x7f9a7800b330 comp rx=0 tx=0).stop 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 -- 172.21.15.104:0/3703718110 shutdown_connections 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 --2- 172.21.15.104:0/3703718110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a800cb540 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 --2- 172.21.15.104:0/3703718110 >> v2:172.21.15.114:3300/0 conn(0x7f9a8006b1f0 0x7f9a800cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 -- 172.21.15.104:0/3703718110 >> 172.21.15.104:0/3703718110 conn(0x7f9a80089fd0 msgr2=0x7f9a800d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 -- 172.21.15.104:0/3703718110 shutdown_connections 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.549+0000 7f9a872d2700 1 -- 172.21.15.104:0/3703718110 wait complete. 2023-06-10T18:57:03.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.550+0000 7f9a872d2700 1 Processor -- start 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.550+0000 7f9a872d2700 1 -- start start 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a8006b1f0 0x7f9a800c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a800ce350 0x7f9a800c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800c4830 con 0x7f9a8008a3d0 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800c4b10 con 0x7f9a800ce350 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a872d2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800c4df0 con 0x7f9a8006b1f0 2023-06-10T18:57:03.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a8486d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a8486d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38084/0 (socket says 172.21.15.104:38084) 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.551+0000 7f9a8486d700 1 -- 172.21.15.104:0/2775617432 learned_addr learned my addr 172.21.15.104:0/2775617432 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8506e700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a8006b1f0 0x7f9a800c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 -- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a8006b1f0 msgr2=0x7f9a800c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8586f700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a800ce350 0x7f9a800c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a8006b1f0 0x7f9a800c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 -- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a800ce350 msgr2=0x7f9a800c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a800ce350 0x7f9a800c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 -- 172.21.15.104:0/2775617432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78009a70 con 0x7f9a8008a3d0 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a8486d700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00c540 tx=0x7f9a7c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c004c80 con 0x7f9a8008a3d0 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c004de0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a800c50d0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c007310 con 0x7f9a8008a3d0 2023-06-10T18:57:03.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.552+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a800c9860 con 0x7f9a8008a3d0 2023-06-10T18:57:03.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.553+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a80074050 con 0x7f9a8008a3d0 2023-06-10T18:57:03.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.553+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a7c007470 con 0x7f9a8008a3d0 2023-06-10T18:57:03.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.554+0000 7f9a75ffb700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 0x7f9a6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.554+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f9a7c089090 con 0x7f9a8008a3d0 2023-06-10T18:57:03.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.554+0000 7f9a8506e700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 0x7f9a6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.554+0000 7f9a8506e700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 0x7f9a6406d190 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.558+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a7c0542c0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.747+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9a80071640 con 0x7f9a8008a3d0 2023-06-10T18:57:03.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.748+0000 7f9a75ffb700 1 -- 172.21.15.104:0/2775617432 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v139) v1 ==== 84+0+126 (secure 0 0 0) 0x7f9a7c0578e0 con 0x7f9a8008a3d0 2023-06-10T18:57:03.754 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:03.755 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:57:03.756 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:57:03.756 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:03.756 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 msgr2=0x7f9a6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 0x7f9a6406d190 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006fd0 tx=0x7f9a70008040 comp rx=0 tx=0).stop 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 msgr2=0x7f9a8015f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00c540 tx=0x7f9a7c00c900 comp rx=0 tx=0).stop 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 shutdown_connections 2023-06-10T18:57:03.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a6406ace0 0x7f9a6406d190 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a800ce350 0x7f9a800c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a8006b1f0 0x7f9a800c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 --2- 172.21.15.104:0/2775617432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9a8008a3d0 0x7f9a8015f710 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 >> 172.21.15.104:0/2775617432 conn(0x7f9a80089fd0 msgr2=0x7f9a800ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 shutdown_connections 2023-06-10T18:57:03.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.750+0000 7f9a872d2700 1 -- 172.21.15.104:0/2775617432 wait complete. 2023-06-10T18:57:03.768 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:57:03.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:57:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.888+0000 7f0ccd10c700 1 Processor -- start 2023-06-10T18:57:03.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.888+0000 7f0ccd10c700 1 -- start start 2023-06-10T18:57:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0ccd10c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cc815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0ccd10c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0cc80cd150 0x7f0cc80cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0ccd10c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc80cdab0 con 0x7f0cc80af940 2023-06-10T18:57:03.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0ccd10c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc815dab0 con 0x7f0cc80cd150 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0ccd10c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc815aa00 con 0x7f0cc81545a0 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0cc80af940 0x7f0cc815e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35918/0 (socket says 172.21.15.104:35918) 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc659c700 1 -- 172.21.15.104:0/3051707075 learned_addr learned my addr 172.21.15.104:0/3051707075 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777298043 0 0) 0x7f0cc80cdab0 con 0x7f0cc80af940 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb00035b0 con 0x7f0cc80af940 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cc815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc759e700 1 --2- 172.21.15.104:0/3051707075 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80cd150 0x7f0cc80cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666097868 0 0) 0x7f0cb00035b0 con 0x7f0cc80af940 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 msgr2=0x7f0cc815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cc815a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80cd150 msgr2=0x7f0cc80cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 --2- 172.21.15.104:0/3051707075 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80cd150 0x7f0cc80cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8154970 con 0x7f0cc80af940 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0cb8001670 con 0x7f0cc80af940 2023-06-10T18:57:03.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 >> v1:172.21.15.104:6789/0 conn(0x7f0cc80af940 legacy=0x7f0cc815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.889+0000 7f0cc5d9b700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cb0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb0003f80 0x7f0cb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0004d90 0x7f0cb0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc815dab0 con 0x7f0cb0003f80 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc815aa00 con 0x7f0cb0004d90 2023-06-10T18:57:03.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb000ba10 con 0x7f0cc81545a0 2023-06-10T18:57:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb0003f80 0x7f0cb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc759e700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0004d90 0x7f0cb0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cb0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 -- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 msgr2=0x7f0cb0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cb0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 -- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0004d90 msgr2=0x7f0cb0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0004d90 0x7f0cb0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 -- 172.21.15.104:0/3051707075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00035b0 con 0x7f0cb0003f80 2023-06-10T18:57:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc81545a0 0x7f0cb0003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:03.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc659c700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb0003f80 0x7f0cb0004850 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f0cb800c370 tx=0x7f0cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77ffe0c0a03637a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8007c60 con 0x7f0cb0003f80 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8005960 con 0x7f0cb0003f80 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.890+0000 7f0cc5d9b700 1 -- 172.21.15.104:0/3051707075 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8005d10 con 0x7f0cb0003f80 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 -- 172.21.15.104:0/3051707075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb0003f80 msgr2=0x7f0cb0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/3051707075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cb0003f80 0x7f0cb0004850 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f0cb800c370 tx=0x7f0cb0005880 comp rx=0 tx=0).stop 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0cc759e700 1 -- 172.21.15.104:0/3051707075 reap_dead start 2023-06-10T18:57:03.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 -- 172.21.15.104:0/3051707075 shutdown_connections 2023-06-10T18:57:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 -- 172.21.15.104:0/3051707075 >> 172.21.15.104:0/3051707075 conn(0x7f0cc806e370 msgr2=0x7f0cc806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 -- 172.21.15.104:0/3051707075 shutdown_connections 2023-06-10T18:57:03.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.891+0000 7f0ccd10c700 1 -- 172.21.15.104:0/3051707075 wait complete. 2023-06-10T18:57:03.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.892+0000 7f0ccd10c700 1 Processor -- start 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.892+0000 7f0ccd10c700 1 -- start start 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0003f80 0x7f0cc8154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c5b50 0x7f0cc8144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8155040 con 0x7f0cc8152a30 2023-06-10T18:57:03.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80c56c0 con 0x7f0cb0003f80 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0ccd10c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80c4ab0 con 0x7f0cc80c5b50 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38100/0 (socket says 172.21.15.104:38100) 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 -- 172.21.15.104:0/2301593343 learned_addr learned my addr 172.21.15.104:0/2301593343 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0003f80 0x7f0cc8154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc759e700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c5b50 0x7f0cc8144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 -- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c5b50 msgr2=0x7f0cc8144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c5b50 0x7f0cc8144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 -- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0003f80 msgr2=0x7f0cc8154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0003f80 0x7f0cc8154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.893+0000 7f0cc659c700 1 -- 172.21.15.104:0/2301593343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc80c4d90 con 0x7f0cc8152a30 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0cc659c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00046d0 tx=0x7f0cb8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8004870 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb80049f0 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc80c5070 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8004da0 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.894+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc80c3a70 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.895+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc8074050 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.895+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0cb80070d0 con 0x7f0cc8152a30 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.896+0000 7f0caf7fe700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 0x7f0cb406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:03.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.896+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f0cb808a5e0 con 0x7f0cc8152a30 2023-06-10T18:57:03.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.896+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 0x7f0cb406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:03.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.897+0000 7f0cc6d9d700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 0x7f0cb406d190 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009ae0 tx=0x7f0cbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:03.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:03.900+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0cb8055930 con 0x7f0cc8152a30 2023-06-10T18:57:04.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.088+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0cc80c5800 con 0x7f0cc8152a30 2023-06-10T18:57:04.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.866+0000 7f0caf7fe700 1 -- 172.21.15.104:0/2301593343 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v140) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0cb8007dc0 con 0x7f0cc8152a30 2023-06-10T18:57:04.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.869+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 msgr2=0x7f0cb406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:04.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.869+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 0x7f0cb406d190 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009ae0 tx=0x7f0cbc008040 comp rx=0 tx=0).stop 2023-06-10T18:57:04.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.869+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 msgr2=0x7f0cc80c3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:04.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.869+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00046d0 tx=0x7f0cb8000fe0 comp rx=0 tx=0).stop 2023-06-10T18:57:04.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 shutdown_connections 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0cb406ace0 0x7f0cb406d190 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cb0003f80 0x7f0cc8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c5b50 0x7f0cc8144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 --2- 172.21.15.104:0/2301593343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cc8152a30 0x7f0cc80c3530 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 >> 172.21.15.104:0/2301593343 conn(0x7f0cc806e370 msgr2=0x7f0cc80b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 shutdown_connections 2023-06-10T18:57:04.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:04.870+0000 7f0ccd10c700 1 -- 172.21.15.104:0/2301593343 wait complete. 2023-06-10T18:57:04.878 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:57:04.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:57:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.021+0000 7f1a8206c700 1 Processor -- start 2023-06-10T18:57:05.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.021+0000 7f1a8206c700 1 -- start start 2023-06-10T18:57:05.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.021+0000 7f1a8206c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.021+0000 7f1a8206c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.021+0000 7f1a8206c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c06e250 con 0x7f1a7c06e6e0 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a8206c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7c15c950 con 0x7f1a7c15c580 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a8206c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0b1e80 con 0x7f1a7c06cde0 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7b7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7b7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:53842/0 (socket says 172.21.15.104:53842) 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:57496/0 (socket says 172.21.15.104:57496) 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7affd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1a7c06e6e0 0x7f1a7c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:41782/0 (socket says 172.21.15.104:41782) 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 -- 172.21.15.104:0/1359085159 learned_addr learned my addr 172.21.15.104:0/1359085159 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 -- 172.21.15.104:0/1359085159 >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 msgr2=0x7f1a7c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 --2- 172.21.15.104:0/1359085159 >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 -- 172.21.15.104:0/1359085159 >> v1:172.21.15.114:6789/0 conn(0x7f1a7c06e6e0 legacy=0x7f1a7c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7bfff700 1 -- 172.21.15.104:0/1359085159 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c06d920 con 0x7f1a7c06cde0 2023-06-10T18:57:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.022+0000 7f1a7b7fe700 1 --2- 172.21.15.104:0/1359085159 >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a7bfff700 1 --2- 172.21.15.104:0/1359085159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009e50 tx=0x7f1a7400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f89cf6fa3396f1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a7a7fc700 1 -- 172.21.15.104:0/1359085159 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400e070 con 0x7f1a7c06cde0 2023-06-10T18:57:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a7a7fc700 1 -- 172.21.15.104:0/1359085159 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7400b910 con 0x7f1a7c06cde0 2023-06-10T18:57:05.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a7a7fc700 1 -- 172.21.15.104:0/1359085159 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400bcc0 con 0x7f1a7c06cde0 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 -- 172.21.15.104:0/1359085159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 msgr2=0x7f1a7c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 --2- 172.21.15.104:0/1359085159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009e50 tx=0x7f1a7400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 -- 172.21.15.104:0/1359085159 shutdown_connections 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 --2- 172.21.15.104:0/1359085159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c0ce0d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 --2- 172.21.15.104:0/1359085159 >> v2:172.21.15.114:3300/0 conn(0x7f1a7c15c580 0x7f1a7c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 -- 172.21.15.104:0/1359085159 >> 172.21.15.104:0/1359085159 conn(0x7f1a7c06b8a0 msgr2=0x7f1a7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 -- 172.21.15.104:0/1359085159 shutdown_connections 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.023+0000 7f1a8206c700 1 -- 172.21.15.104:0/1359085159 wait complete. 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.024+0000 7f1a8206c700 1 Processor -- start 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 -- start start 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c14a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a7c06e6e0 0x7f1a7c0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0cef20 con 0x7f1a7c15c580 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0ceb80 con 0x7f1a7c06e6e0 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.025+0000 7f1a8206c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c1e3ad0 con 0x7f1a7c06cde0 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38116/0 (socket says 172.21.15.104:38116) 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 -- 172.21.15.104:0/2030386353 learned_addr learned my addr 172.21.15.104:0/2030386353 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7affd700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a7c06e6e0 0x7f1a7c0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7b7fe700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c14a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 -- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 msgr2=0x7f1a7c14a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c14a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 -- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a7c06e6e0 msgr2=0x7f1a7c0b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a7c06e6e0 0x7f1a7c0b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 -- 172.21.15.104:0/2030386353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a70009430 con 0x7f1a7c15c580 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a7bfff700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0b1fc0 tx=0x7f1a74004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a740092e0 con 0x7f1a7c15c580 2023-06-10T18:57:05.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74004960 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7401c680 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a74009b30 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.026+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c1e4090 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.027+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c0cf060 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.028+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a7401c7e0 con 0x7f1a7c15c580 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.028+0000 7f1a63fff700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 0x7f1a6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:05.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.028+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f1a7408b030 con 0x7f1a7c15c580 2023-06-10T18:57:05.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.028+0000 7f1a7b7fe700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 0x7f1a6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:05.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.029+0000 7f1a7b7fe700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 0x7f1a6806d190 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009c40 tx=0x7f1a70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:05.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.033+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1a7c0cf060 con 0x7f1a7c15c580 2023-06-10T18:57:05.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.218+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1a7c0cecf0 con 0x7f1a7c15c580 2023-06-10T18:57:05.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.874+0000 7f1a63fff700 1 -- 172.21.15.104:0/2030386353 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v141) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1a7c0cecf0 con 0x7f1a7c15c580 2023-06-10T18:57:05.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 msgr2=0x7f1a6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 0x7f1a6806d190 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009c40 tx=0x7f1a70008040 comp rx=0 tx=0).stop 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 msgr2=0x7f1a7c0cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0b1fc0 tx=0x7f1a74004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 shutdown_connections 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a7c15c580 0x7f1a7c0cf340 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a7c06cde0 0x7f1a7c14a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1a6806ace0 0x7f1a6806d190 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.879+0000 7f1a8206c700 1 --2- 172.21.15.104:0/2030386353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a7c06e6e0 0x7f1a7c0b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:05.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.880+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 >> 172.21.15.104:0/2030386353 conn(0x7f1a7c06b8a0 msgr2=0x7f1a7c0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:05.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.880+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 shutdown_connections 2023-06-10T18:57:05.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:05.880+0000 7f1a8206c700 1 -- 172.21.15.104:0/2030386353 wait complete. 2023-06-10T18:57:05.890 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:57:05.911 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:57:06.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.029+0000 7f0ac143e700 1 Processor -- start 2023-06-10T18:57:06.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.029+0000 7f0ac143e700 1 -- start start 2023-06-10T18:57:06.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0abc070a10 0x7f0abc070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0abc15fd40 0x7f0abc160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc0713a0 con 0x7f0abc15fd40 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc0c4770 con 0x7f0abc070a10 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0ac143e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc069e70 con 0x7f0abc145cc0 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0abb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0abc15fd40 0x7f0abc160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0abb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0abc15fd40 0x7f0abc160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38126/0 (socket says 172.21.15.104:38126) 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0abb7fe700 1 -- 172.21.15.104:0/3974978310 learned_addr learned my addr 172.21.15.104:0/3974978310 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0aba7fc700 1 --2- 172.21.15.104:0/3974978310 >> v2:172.21.15.114:3300/0 conn(0x7f0abc070a10 0x7f0abc070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.030+0000 7f0abaffd700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 -- 172.21.15.104:0/3974978310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 msgr2=0x7f0abc06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc06c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 -- 172.21.15.104:0/3974978310 >> v2:172.21.15.114:3300/0 conn(0x7f0abc070a10 msgr2=0x7f0abc070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 --2- 172.21.15.104:0/3974978310 >> v2:172.21.15.114:3300/0 conn(0x7f0abc070a10 0x7f0abc070e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 -- 172.21.15.104:0/3974978310 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc0c49c0 con 0x7f0abc15fd40 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0abb7fe700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc15fd40 0x7f0abc160180 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0cfd40 tx=0x7f0ab400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecd0b15f3f436da server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0ab9ffb700 1 -- 172.21.15.104:0/3974978310 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab400e070 con 0x7f0abc15fd40 2023-06-10T18:57:06.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0ab9ffb700 1 -- 172.21.15.104:0/3974978310 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab400b910 con 0x7f0abc15fd40 2023-06-10T18:57:06.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.031+0000 7f0ab9ffb700 1 -- 172.21.15.104:0/3974978310 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab400bcc0 con 0x7f0abc15fd40 2023-06-10T18:57:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 -- 172.21.15.104:0/3974978310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc15fd40 msgr2=0x7f0abc160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc15fd40 0x7f0abc160180 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0cfd40 tx=0x7f0ab400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 -- 172.21.15.104:0/3974978310 shutdown_connections 2023-06-10T18:57:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc15fd40 0x7f0abc160180 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 --2- 172.21.15.104:0/3974978310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 --2- 172.21.15.104:0/3974978310 >> v2:172.21.15.114:3300/0 conn(0x7f0abc070a10 0x7f0abc070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 -- 172.21.15.104:0/3974978310 >> 172.21.15.104:0/3974978310 conn(0x7f0abc0ceec0 msgr2=0x7f0abc0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 -- 172.21.15.104:0/3974978310 shutdown_connections 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.032+0000 7f0ac143e700 1 -- 172.21.15.104:0/3974978310 wait complete. 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.033+0000 7f0ac143e700 1 Processor -- start 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.033+0000 7f0ac143e700 1 -- start start 2023-06-10T18:57:06.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 0x7f0abc0ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15d220 con 0x7f0abc070a10 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc145260 con 0x7f0abc15fd40 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15d470 con 0x7f0abc145cc0 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38128/0 (socket says 172.21.15.104:38128) 2023-06-10T18:57:06.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 -- 172.21.15.104:0/2816937709 learned_addr learned my addr 172.21.15.104:0/2816937709 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0aba7fc700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abb7fe700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 0x7f0abc0ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 -- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 msgr2=0x7f0abc15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 -- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 msgr2=0x7f0abc0ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 0x7f0abc0ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 -- 172.21.15.104:0/2816937709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab4009b30 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abb7fe700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 0x7f0abc0ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0abaffd700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e870 tx=0x7f0ab000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0009590 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0013680 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.034+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc15d750 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.035+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00139f0 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.035+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc15b830 con 0x7f0abc070a10 2023-06-10T18:57:06.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.035+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc074050 con 0x7f0abc070a10 2023-06-10T18:57:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.036+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0ab0013b50 con 0x7f0abc070a10 2023-06-10T18:57:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.036+0000 7f0aa37fe700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 0x7f0aa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.036+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f0ab00894c0 con 0x7f0abc070a10 2023-06-10T18:57:06.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.037+0000 7f0aba7fc700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 0x7f0aa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.037+0000 7f0aba7fc700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 0x7f0aa806d190 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0abc15cd50 tx=0x7f0aac009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.041+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ab0054490 con 0x7f0abc070a10 2023-06-10T18:57:06.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.269+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0abc0ce1e0 con 0x7f0abc070a10 2023-06-10T18:57:06.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.274+0000 7f0aa37fe700 1 -- 172.21.15.104:0/2816937709 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0ab0057ab0 con 0x7f0abc070a10 2023-06-10T18:57:06.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.281+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 msgr2=0x7f0aa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.281+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 0x7f0aa806d190 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0abc15cd50 tx=0x7f0aac009210 comp rx=0 tx=0).stop 2023-06-10T18:57:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.281+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 msgr2=0x7f0abc15c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.281+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e870 tx=0x7f0ab000eb80 comp rx=0 tx=0).stop 2023-06-10T18:57:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.282+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 shutdown_connections 2023-06-10T18:57:06.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.282+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0aa806ace0 0x7f0aa806d190 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.282+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc15fd40 0x7f0abc0ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.283+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0abc070a10 0x7f0abc15c810 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.283+0000 7f0ac143e700 1 --2- 172.21.15.104:0/2816937709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0abc145cc0 0x7f0abc15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.283+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 >> 172.21.15.104:0/2816937709 conn(0x7f0abc0ceec0 msgr2=0x7f0abc0c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.285+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 shutdown_connections 2023-06-10T18:57:06.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.287+0000 7f0ac143e700 1 -- 172.21.15.104:0/2816937709 wait complete. 2023-06-10T18:57:06.297 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 20 and data pool 21 2023-06-10T18:57:06.311 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:06.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.425+0000 7fa2c76bf700 1 Processor -- start 2023-06-10T18:57:06.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.425+0000 7fa2c76bf700 1 -- start start 2023-06-10T18:57:06.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c76bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa2c00cf030 0x7fa2c00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c76bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c0069e70 0x7fa2c006a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c76bf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c006dcb0 con 0x7fa2c015c500 2023-06-10T18:57:06.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c76bf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c006e850 con 0x7fa2c00cf030 2023-06-10T18:57:06.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c76bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c006a830 con 0x7fa2c0069e70 2023-06-10T18:57:06.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa2c00cf030 0x7fa2c00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa2c00cf030 0x7fa2c00ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38138/0 (socket says 172.21.15.104:38138) 2023-06-10T18:57:06.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c545b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa2c015c500 0x7fa2c006e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35964/0 (socket says 172.21.15.104:35964) 2023-06-10T18:57:06.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 -- 172.21.15.104:0/2056328770 learned_addr learned my addr 172.21.15.104:0/2056328770 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 -- 172.21.15.104:0/2056328770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c0069e70 msgr2=0x7fa2c006a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:06.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 --2- 172.21.15.104:0/2056328770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c0069e70 0x7fa2c006a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 -- 172.21.15.104:0/2056328770 >> v1:172.21.15.104:6789/0 conn(0x7fa2c015c500 legacy=0x7fa2c006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 -- 172.21.15.104:0/2056328770 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00cf460 con 0x7fa2c00cf030 2023-06-10T18:57:06.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2c4c5a700 1 --2- 172.21.15.104:0/2056328770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c00cf030 0x7fa2c00ce040 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00098f0 tx=0x7fa2b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c5e5636396f5c5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.426+0000 7fa2b7fff700 1 -- 172.21.15.104:0/2056328770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000b8a0 con 0x7fa2c00cf030 2023-06-10T18:57:06.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2b7fff700 1 -- 172.21.15.104:0/2056328770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b000ba00 con 0x7fa2c00cf030 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2b7fff700 1 -- 172.21.15.104:0/2056328770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000bd90 con 0x7fa2c00cf030 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 -- 172.21.15.104:0/2056328770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c00cf030 msgr2=0x7fa2c00ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/2056328770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c00cf030 0x7fa2c00ce040 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00098f0 tx=0x7fa2b0009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 -- 172.21.15.104:0/2056328770 shutdown_connections 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/2056328770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c0069e70 0x7fa2c006a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/2056328770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c00cf030 0x7fa2c00ce040 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 -- 172.21.15.104:0/2056328770 >> 172.21.15.104:0/2056328770 conn(0x7fa2c006b8a0 msgr2=0x7fa2c00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 -- 172.21.15.104:0/2056328770 shutdown_connections 2023-06-10T18:57:06.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.427+0000 7fa2c76bf700 1 -- 172.21.15.104:0/2056328770 wait complete. 2023-06-10T18:57:06.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.428+0000 7fa2c76bf700 1 Processor -- start 2023-06-10T18:57:06.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.428+0000 7fa2c76bf700 1 -- start start 2023-06-10T18:57:06.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c00cf030 0x7fa2c00c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 0x7fa2c00c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c4a60 con 0x7fa2c0069e70 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c41c0 con 0x7fa2c015c500 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c76bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c44a0 con 0x7fa2c00cf030 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38142/0 (socket says 172.21.15.104:38142) 2023-06-10T18:57:06.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 -- 172.21.15.104:0/3233099178 learned_addr learned my addr 172.21.15.104:0/3233099178 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 -- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c00cf030 msgr2=0x7fa2c00c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c00cf030 0x7fa2c00c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 -- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 msgr2=0x7fa2c00c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c5c5c700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 0x7fa2c00c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 0x7fa2c00c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.429+0000 7fa2c545b700 1 -- 172.21.15.104:0/3233099178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0009550 con 0x7fa2c0069e70 2023-06-10T18:57:06.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2c5c5c700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 0x7fa2c00c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2c545b700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c480 tx=0x7fa2bc00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0074e0 con 0x7fa2c0069e70 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc007640 con 0x7fa2c0069e70 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c00c4780 con 0x7fa2c0069e70 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc005a50 con 0x7fa2c0069e70 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.430+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c00c3450 con 0x7fa2c0069e70 2023-06-10T18:57:06.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.431+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0074050 con 0x7fa2c0069e70 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.432+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa2bc005bb0 con 0x7fa2c0069e70 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.432+0000 7fa2b5ffb700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 0x7fa2a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.432+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa2bc088f30 con 0x7fa2c0069e70 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.432+0000 7fa2c4c5a700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 0x7fa2a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.433+0000 7fa2c4c5a700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 0x7fa2a406d190 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00c5130 tx=0x7fa2b0010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.436+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2bc053f80 con 0x7fa2c0069e70 2023-06-10T18:57:06.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.622+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2c00c6180 con 0x7fa2c0069e70 2023-06-10T18:57:06.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.624+0000 7fa2b5ffb700 1 -- 172.21.15.104:0/3233099178 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+18627 (secure 0 0 0) 0x7fa2bc017090 con 0x7fa2c0069e70 2023-06-10T18:57:06.630 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:06.630 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":142,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:06.271298+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 msgr2=0x7fa2a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 0x7fa2a406d190 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00c5130 tx=0x7fa2b0010000 comp rx=0 tx=0).stop 2023-06-10T18:57:06.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 msgr2=0x7fa2c0158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c480 tx=0x7fa2bc00c840 comp rx=0 tx=0).stop 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 shutdown_connections 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa2a406ace0 0x7fa2a406d190 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2c015c500 0x7fa2c00c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2c0069e70 0x7fa2c0158dd0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 --2- 172.21.15.104:0/3233099178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2c00cf030 0x7fa2c00c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 >> 172.21.15.104:0/3233099178 conn(0x7fa2c006b8a0 msgr2=0x7fa2c006e140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 shutdown_connections 2023-06-10T18:57:06.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.626+0000 7fa2c76bf700 1 -- 172.21.15.104:0/3233099178 wait complete. 2023-06-10T18:57:06.644 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:06.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.758+0000 7f94621d5700 1 Processor -- start 2023-06-10T18:57:06.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.758+0000 7f94621d5700 1 -- start start 2023-06-10T18:57:06.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 0x7f945c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f945c0ca100 0x7f945c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c071640 con 0x7f945c0ca100 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0aef00 con 0x7f945c06e0e0 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f94621d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c06d290 con 0x7f945c06e4b0 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f945c0ca100 0x7f945c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f945c0ca100 0x7f945c0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38150/0 (socket says 172.21.15.104:38150) 2023-06-10T18:57:06.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 -- 172.21.15.104:0/1068392014 learned_addr learned my addr 172.21.15.104:0/1068392014 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945affd700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945b7fe700 1 --2- 172.21.15.104:0/1068392014 >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 0x7f945c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 -- 172.21.15.104:0/1068392014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 msgr2=0x7f945c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 -- 172.21.15.104:0/1068392014 >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 msgr2=0x7f945c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 --2- 172.21.15.104:0/1068392014 >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 0x7f945c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945bfff700 1 -- 172.21.15.104:0/1068392014 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c06d4c0 con 0x7f945c0ca100 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.759+0000 7f945b7fe700 1 --2- 172.21.15.104:0/1068392014 >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 0x7f945c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f945bfff700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c0ca100 0x7f945c0ce9f0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f945c06af50 tx=0x7f945400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87f93409d3e632d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f945a7fc700 1 -- 172.21.15.104:0/1068392014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945400e070 con 0x7f945c0ca100 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f945a7fc700 1 -- 172.21.15.104:0/1068392014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945400b8d0 con 0x7f945c0ca100 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f945a7fc700 1 -- 172.21.15.104:0/1068392014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945400bc80 con 0x7f945c0ca100 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 -- 172.21.15.104:0/1068392014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c0ca100 msgr2=0x7f945c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c0ca100 0x7f945c0ce9f0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f945c06af50 tx=0x7f945400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:06.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 -- 172.21.15.104:0/1068392014 shutdown_connections 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c0ca100 0x7f945c0ce9f0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 --2- 172.21.15.104:0/1068392014 >> v2:172.21.15.114:3300/0 conn(0x7f945c06e0e0 0x7f945c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 --2- 172.21.15.104:0/1068392014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 -- 172.21.15.104:0/1068392014 >> 172.21.15.104:0/1068392014 conn(0x7f945c06b8a0 msgr2=0x7f945c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 -- 172.21.15.104:0/1068392014 shutdown_connections 2023-06-10T18:57:06.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.760+0000 7f94621d5700 1 -- 172.21.15.104:0/1068392014 wait complete. 2023-06-10T18:57:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.761+0000 7f94621d5700 1 Processor -- start 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 -- start start 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c1f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 0x7f945c0b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0af870 con 0x7f945c06e0e0 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0afb50 con 0x7f945c0ca100 2023-06-10T18:57:06.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.762+0000 7f94621d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0afe30 con 0x7f945c06e4b0 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38156/0 (socket says 172.21.15.104:38156) 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 -- 172.21.15.104:0/2461448472 learned_addr learned my addr 172.21.15.104:0/2461448472 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945affd700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c1f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945bfff700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 0x7f945c0b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 -- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 msgr2=0x7f945c1f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c1f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 -- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 msgr2=0x7f945c0b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 0x7f945c0b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 -- 172.21.15.104:0/2461448472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454009b30 con 0x7f945c06e0e0 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945bfff700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 0x7f945c0b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f945b7fe700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f944400e810 tx=0x7f944400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444005c20 con 0x7f945c06e0e0 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9444005d80 con 0x7f945c06e0e0 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c0b24d0 con 0x7f945c06e0e0 2023-06-10T18:57:06.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444004250 con 0x7f945c06e0e0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.763+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c0af580 con 0x7f945c06e0e0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.764+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c070520 con 0x7f945c06e0e0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.765+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94440043b0 con 0x7f945c06e0e0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.765+0000 7f9453fff700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 0x7f943c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.765+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f94440892e0 con 0x7f945c06e0e0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.765+0000 7f945affd700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 0x7f943c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:06.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.766+0000 7f945affd700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 0x7f943c06d190 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f945c0b2120 tx=0x7f944c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:06.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.769+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9444054250 con 0x7f945c06e0e0 2023-06-10T18:57:06.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.956+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f945c0c60b0 con 0x7f945c06e0e0 2023-06-10T18:57:06.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.957+0000 7f9453fff700 1 -- 172.21.15.104:0/2461448472 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+18627 (secure 0 0 0) 0x7f9444057870 con 0x7f945c06e0e0 2023-06-10T18:57:06.963 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:06.964 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":142,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:06.271298+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:06.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 msgr2=0x7f943c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 0x7f943c06d190 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f945c0b2120 tx=0x7f944c008040 comp rx=0 tx=0).stop 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 msgr2=0x7f945c0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f944400e810 tx=0x7f944400ebd0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 shutdown_connections 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f943c06ace0 0x7f943c06d190 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f945c0ca100 0x7f945c0b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945c06e0e0 0x7f945c0af040 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 --2- 172.21.15.104:0/2461448472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f945c06e4b0 0x7f945c1f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 >> 172.21.15.104:0/2461448472 conn(0x7f945c06b8a0 msgr2=0x7f945c0ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 shutdown_connections 2023-06-10T18:57:06.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:06.959+0000 7f94621d5700 1 -- 172.21.15.104:0/2461448472 wait complete. 2023-06-10T18:57:06.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:57:07.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.093+0000 7f0b22036700 1 Processor -- start 2023-06-10T18:57:07.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.093+0000 7f0b22036700 1 -- start start 2023-06-10T18:57:07.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b22036700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0b1c069e70 0x7f0b1c06a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b22036700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0b1c06dcd0 0x7f0b1c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b22036700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c06e650 con 0x7f0b1c15c500 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b22036700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1c0aef20 con 0x7f0b1c069e70 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b22036700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1c15c8d0 con 0x7f0b1c06dcd0 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b1affd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0b1c069e70 0x7f0b1c06a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b1b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0b1c15c500 0x7f0b1c0cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35992/0 (socket says 172.21.15.104:35992) 2023-06-10T18:57:07.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.094+0000 7f0b1b7fe700 1 -- 172.21.15.104:0/3367250236 learned_addr learned my addr 172.21.15.104:0/3367250236 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1bfff700 1 --2- 172.21.15.104:0/3367250236 >> v2:172.21.15.114:3300/0 conn(0x7f0b1c06dcd0 0x7f0b1c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1affd700 1 -- 172.21.15.104:0/3367250236 >> v2:172.21.15.114:3300/0 conn(0x7f0b1c06dcd0 msgr2=0x7f0b1c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1affd700 1 --2- 172.21.15.104:0/3367250236 >> v2:172.21.15.114:3300/0 conn(0x7f0b1c06dcd0 0x7f0b1c06e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1affd700 1 -- 172.21.15.104:0/3367250236 >> v1:172.21.15.104:6789/0 conn(0x7f0b1c15c500 legacy=0x7f0b1c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1affd700 1 -- 172.21.15.104:0/3367250236 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c06abd0 con 0x7f0b1c069e70 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1a7fc700 1 -- 172.21.15.104:0/3367250236 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917032425 0 0) 0x7f0b1c06e650 con 0x7f0b1c15c500 2023-06-10T18:57:07.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1affd700 1 --2- 172.21.15.104:0/3367250236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c069e70 0x7f0b1c06a310 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009840 tx=0x7f0b0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace4d30c1226e401 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1a7fc700 1 -- 172.21.15.104:0/3367250236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c011830 con 0x7f0b1c069e70 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1a7fc700 1 -- 172.21.15.104:0/3367250236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c011990 con 0x7f0b1c069e70 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b1a7fc700 1 -- 172.21.15.104:0/3367250236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c011d20 con 0x7f0b1c069e70 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b22036700 1 -- 172.21.15.104:0/3367250236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c069e70 msgr2=0x7f0b1c06a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b22036700 1 --2- 172.21.15.104:0/3367250236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c069e70 0x7f0b1c06a310 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009840 tx=0x7f0b0c009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.095+0000 7f0b22036700 1 -- 172.21.15.104:0/3367250236 shutdown_connections 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.096+0000 7f0b22036700 1 --2- 172.21.15.104:0/3367250236 >> v2:172.21.15.114:3300/0 conn(0x7f0b1c06dcd0 0x7f0b1c06e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:07.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.096+0000 7f0b22036700 1 --2- 172.21.15.104:0/3367250236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c069e70 0x7f0b1c06a310 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.096+0000 7f0b22036700 1 -- 172.21.15.104:0/3367250236 >> 172.21.15.104:0/3367250236 conn(0x7f0b1c06b8a0 msgr2=0x7f0b1c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.096+0000 7f0b22036700 1 -- 172.21.15.104:0/3367250236 shutdown_connections 2023-06-10T18:57:07.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.096+0000 7f0b22036700 1 -- 172.21.15.104:0/3367250236 wait complete. 2023-06-10T18:57:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.097+0000 7f0b22036700 1 Processor -- start 2023-06-10T18:57:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.097+0000 7f0b22036700 1 -- start start 2023-06-10T18:57:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 0x7f0b1c151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 0x7f0b1c0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c15b140 con 0x7f0b1c15c500 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c1520a0 con 0x7f0b1c069e70 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c0cccd0 con 0x7f0b1c06dcd0 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38174/0 (socket says 172.21.15.104:38174) 2023-06-10T18:57:07.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 -- 172.21.15.104:0/943258774 learned_addr learned my addr 172.21.15.104:0/943258774 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1b7fe700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 0x7f0b1c151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1affd700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 0x7f0b1c0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 -- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 msgr2=0x7f0b1c0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 0x7f0b1c0cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 -- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 msgr2=0x7f0b1c151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 0x7f0b1c151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:07.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 -- 172.21.15.104:0/943258774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0094a0 con 0x7f0b1c15c500 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1affd700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 0x7f0b1c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1b7fe700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 0x7f0b1c151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b1bfff700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f0b14007e60 tx=0x7f0b1400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400ede0 con 0x7f0b1c15c500 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b14014420 con 0x7f0b1c15c500 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c0c7b80 con 0x7f0b1c15c500 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b140147d0 con 0x7f0b1c15c500 2023-06-10T18:57:07.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.098+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c0c8140 con 0x7f0b1c15c500 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.099+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c074050 con 0x7f0b1c15c500 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.100+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0b14014930 con 0x7f0b1c15c500 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.100+0000 7f0b03fff700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.100+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0b14089d40 con 0x7f0b1c15c500 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.101+0000 7f0b1b7fe700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:07.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.101+0000 7f0b1b7fe700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 0x7f0b0806d190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0b100017b0 tx=0x7f0b10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:07.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.105+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0b14054cb0 con 0x7f0b1c15c500 2023-06-10T18:57:07.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:07.337+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0b1c0c2ef0 con 0x7f0b1c15c500 2023-06-10T18:57:08.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.283+0000 7f0b03fff700 1 -- 172.21.15.104:0/943258774 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0b140582d0 con 0x7f0b1c15c500 2023-06-10T18:57:08.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 msgr2=0x7f0b0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 0x7f0b0806d190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0b100017b0 tx=0x7f0b10009040 comp rx=0 tx=0).stop 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 msgr2=0x7f0b1c0c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f0b14007e60 tx=0x7f0b1400e3c0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 shutdown_connections 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b1c069e70 0x7f0b1c151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0b1c15c500 0x7f0b1c0c3140 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 --2- 172.21.15.104:0/943258774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0b1c06dcd0 0x7f0b1c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 >> 172.21.15.104:0/943258774 conn(0x7f0b1c06b8a0 msgr2=0x7f0b1c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:08.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.286+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 shutdown_connections 2023-06-10T18:57:08.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.287+0000 7f0b22036700 1 -- 172.21.15.104:0/943258774 wait complete. 2023-06-10T18:57:08.307 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:57:08.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:57:08.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.430+0000 7f974520d700 1 Processor -- start 2023-06-10T18:57:08.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.430+0000 7f974520d700 1 -- start start 2023-06-10T18:57:08.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f974520d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400d25c0 0x7f97400d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f974520d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f97400b05d0 0x7f97400b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f974520d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400af530 con 0x7f97400d04f0 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f974520d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97400af810 con 0x7f97400b05d0 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f974520d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97400afaf0 con 0x7f97400d25c0 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f973f59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f97400b05d0 0x7f97400b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f973ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f97400d04f0 0x7f97400d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36006/0 (socket says 172.21.15.104:36006) 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.431+0000 7f973ed9d700 1 -- 172.21.15.104:0/716820310 learned_addr learned my addr 172.21.15.104:0/716820310 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:08.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973e59c700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400d25c0 0x7f97400d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973f59e700 1 -- 172.21.15.104:0/716820310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400d25c0 msgr2=0x7f97400d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973f59e700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400d25c0 0x7f97400d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973f59e700 1 -- 172.21.15.104:0/716820310 >> v1:172.21.15.104:6789/0 conn(0x7f97400d04f0 legacy=0x7f97400d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973f59e700 1 -- 172.21.15.104:0/716820310 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400afdd0 con 0x7f97400b05d0 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973dd9b700 1 -- 172.21.15.104:0/716820310 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739049872 0 0) 0x7f97400af530 con 0x7f97400d04f0 2023-06-10T18:57:08.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973f59e700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400b05d0 0x7f97400b0a50 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f9738009d90 tx=0x7f973800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5b18360dbc06cb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973dd9b700 1 -- 172.21.15.104:0/716820310 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973800e070 con 0x7f97400b05d0 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973dd9b700 1 -- 172.21.15.104:0/716820310 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800bbd0 con 0x7f97400b05d0 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f973dd9b700 1 -- 172.21.15.104:0/716820310 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738012430 con 0x7f97400b05d0 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 -- 172.21.15.104:0/716820310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400b05d0 msgr2=0x7f97400b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400b05d0 0x7f97400b0a50 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f9738009d90 tx=0x7f973800b330 comp rx=0 tx=0).stop 2023-06-10T18:57:08.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 -- 172.21.15.104:0/716820310 shutdown_connections 2023-06-10T18:57:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400b05d0 0x7f97400b0a50 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 --2- 172.21.15.104:0/716820310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400d25c0 0x7f97400d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.432+0000 7f974520d700 1 -- 172.21.15.104:0/716820310 >> 172.21.15.104:0/716820310 conn(0x7f9740071ec0 msgr2=0x7f974009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.433+0000 7f974520d700 1 -- 172.21.15.104:0/716820310 shutdown_connections 2023-06-10T18:57:08.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.433+0000 7f974520d700 1 -- 172.21.15.104:0/716820310 wait complete. 2023-06-10T18:57:08.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.434+0000 7f974520d700 1 Processor -- start 2023-06-10T18:57:08.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.434+0000 7f974520d700 1 -- start start 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400b05d0 0x7f97400cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d04f0 0x7f974014e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974014f3b0 con 0x7f97400d25c0 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400cfbb0 con 0x7f97400d04f0 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97401e3a10 con 0x7f97400b05d0 2023-06-10T18:57:08.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38188/0 (socket says 172.21.15.104:38188) 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 -- 172.21.15.104:0/57289780 learned_addr learned my addr 172.21.15.104:0/57289780 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 -- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400b05d0 msgr2=0x7f97400cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973e59c700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d04f0 0x7f974014e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973ed9d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400b05d0 0x7f97400cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400b05d0 0x7f97400cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 -- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d04f0 msgr2=0x7f974014e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d04f0 0x7f974014e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 -- 172.21.15.104:0/57289780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009a70 con 0x7f97400d25c0 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f973f59e700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f97400b5510 tx=0x7f9738004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738012430 con 0x7f97400d25c0 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738020070 con 0x7f97400d25c0 2023-06-10T18:57:08.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97401e3cf0 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.435+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738004280 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.436+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97401e42b0 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.437+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9740067010 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.437+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f97380043e0 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.437+0000 7f97277fe700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 0x7f972c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.437+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9738022070 con 0x7f97400d25c0 2023-06-10T18:57:08.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.438+0000 7f973ed9d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 0x7f972c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:08.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.438+0000 7f973ed9d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 0x7f972c06d190 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9734001360 tx=0x7f9734009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:08.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.442+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9738056b30 con 0x7f97400d25c0 2023-06-10T18:57:08.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:08.675+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9740074050 con 0x7f97400d25c0 2023-06-10T18:57:09.430 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:57:09.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:57:09.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.671+0000 7f97277fe700 1 -- 172.21.15.104:0/57289780 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f973805a150 con 0x7f97400d25c0 2023-06-10T18:57:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 msgr2=0x7f972c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 0x7f972c06d190 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9734001360 tx=0x7f9734009040 comp rx=0 tx=0).stop 2023-06-10T18:57:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 msgr2=0x7f974014ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:09.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f97400b5510 tx=0x7f9738004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 shutdown_connections 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97400d25c0 0x7f974014ee70 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97400b05d0 0x7f97400cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f972c06ace0 0x7f972c06d190 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 --2- 172.21.15.104:0/57289780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d04f0 0x7f974014e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 >> 172.21.15.104:0/57289780 conn(0x7f9740071ec0 msgr2=0x7f97400ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:09.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 shutdown_connections 2023-06-10T18:57:09.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.674+0000 7f974520d700 1 -- 172.21.15.104:0/57289780 wait complete. 2023-06-10T18:57:09.694 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:57:09.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:09.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.813+0000 7fbda3b74700 1 Processor -- start 2023-06-10T18:57:09.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.813+0000 7fbda3b74700 1 -- start start 2023-06-10T18:57:09.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda3b74700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda3b74700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda3b74700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c0cecf0 con 0x7fbd9c15a420 2023-06-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda3b74700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c0b6070 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda3b74700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0b1d70 con 0x7fbd9c0ce890 2023-06-10T18:57:09.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38194/0 (socket says 172.21.15.104:38194) 2023-06-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 -- 172.21.15.104:0/3546592676 learned_addr learned my addr 172.21.15.104:0/3546592676 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda2111700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 -- 172.21.15.104:0/3546592676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 msgr2=0x7fbd9c15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:09.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 -- 172.21.15.104:0/3546592676 >> v1:172.21.15.114:6789/0 conn(0x7fbd9c15a420 legacy=0x7fbd9c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:09.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 -- 172.21.15.104:0/3546592676 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c0b5460 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda2111700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.814+0000 7fbda110f700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098f0 tx=0x7fbd8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1a4a0e8981a0cd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda090e700 1 -- 172.21.15.104:0/3546592676 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c004230 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda090e700 1 -- 172.21.15.104:0/3546592676 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c004390 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda090e700 1 -- 172.21.15.104:0/3546592676 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0036a0 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 -- 172.21.15.104:0/3546592676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c0b4ff0 msgr2=0x7fbd9c0ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098f0 tx=0x7fbd8c009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:09.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 -- 172.21.15.104:0/3546592676 shutdown_connections 2023-06-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 --2- 172.21.15.104:0/3546592676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c0ce350 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 -- 172.21.15.104:0/3546592676 >> 172.21.15.104:0/3546592676 conn(0x7fbd9c089fd0 msgr2=0x7fbd9c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:09.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 -- 172.21.15.104:0/3546592676 shutdown_connections 2023-06-10T18:57:09.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.815+0000 7fbda3b74700 1 -- 172.21.15.104:0/3546592676 wait complete. 2023-06-10T18:57:09.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.816+0000 7fbda3b74700 1 Processor -- start 2023-06-10T18:57:09.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.816+0000 7fbda3b74700 1 -- start start 2023-06-10T18:57:09.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c15c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0c53a0 con 0x7fbd9c15a420 2023-06-10T18:57:09.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15cb80 con 0x7fbd9c0b4ff0 2023-06-10T18:57:09.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda3b74700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c1f0d90 con 0x7fbd9c0ce890 2023-06-10T18:57:09.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38202/0 (socket says 172.21.15.104:38202) 2023-06-10T18:57:09.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 -- 172.21.15.104:0/2860161627 learned_addr learned my addr 172.21.15.104:0/2860161627 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:09.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda1910700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c15c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda110f700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 -- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 msgr2=0x7fbd9c1f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:09.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c1f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 -- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 msgr2=0x7fbd9c15c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:09.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c15c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:09.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda2111700 1 -- 172.21.15.104:0/2860161627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c009550 con 0x7fbd9c15a420 2023-06-10T18:57:09.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.817+0000 7fbda1910700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:09.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbda2111700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400cef0 tx=0x7fbd9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:09.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400ae50 con 0x7fbd9c15a420 2023-06-10T18:57:09.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd94015440 con 0x7fbd9c15a420 2023-06-10T18:57:09.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c0c4820 con 0x7fbd9c15a420 2023-06-10T18:57:09.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd940157f0 con 0x7fbd9c15a420 2023-06-10T18:57:09.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.818+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c0c4de0 con 0x7fbd9c15a420 2023-06-10T18:57:09.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.819+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c074050 con 0x7fbd9c15a420 2023-06-10T18:57:09.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.819+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbd9400d430 con 0x7fbd9c15a420 2023-06-10T18:57:09.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.819+0000 7fbd927fc700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 0x7fbd8006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:09.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.820+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbd94089c20 con 0x7fbd9c15a420 2023-06-10T18:57:09.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.820+0000 7fbda1910700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 0x7fbd8006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:09.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.820+0000 7fbda1910700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 0x7fbd8006d190 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009930 tx=0x7fbd98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:09.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:09.824+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbd94054b90 con 0x7fbd9c15a420 2023-06-10T18:57:10.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.054+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd9c0b1b30 con 0x7fbd9c15a420 2023-06-10T18:57:10.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.054+0000 7fbd927fc700 1 -- 172.21.15.104:0/2860161627 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fbd9401a090 con 0x7fbd9c15a420 2023-06-10T18:57:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:10.061 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/184392101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":184392101},{"type":"v1","addr":"172.21.15.104:6834","nonce":184392101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9630,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2169449406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2169449406},{"type":"v1","addr":"172.21.15.104:6835","nonce":2169449406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3805828460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3805828460},{"type":"v1","addr":"172.21.15.114:6836","nonce":3805828460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:09.668599+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9642},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9642":{"gid":9642,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2371761194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2371761194},{"type":"v1","addr":"172.21.15.114:6837","nonce":2371761194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 msgr2=0x7fbd8006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 0x7fbd8006d190 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009930 tx=0x7fbd98008040 comp rx=0 tx=0).stop 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 msgr2=0x7fbd9c0c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400cef0 tx=0x7fbd9400a340 comp rx=0 tx=0).stop 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 shutdown_connections 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd8006ace0 0x7fbd8006d190 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd9c0b4ff0 0x7fbd9c15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd9c15a420 0x7fbd9c0c5e70 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 --2- 172.21.15.104:0/2860161627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd9c0ce890 0x7fbd9c1f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 >> 172.21.15.104:0/2860161627 conn(0x7fbd9c089fd0 msgr2=0x7fbd9c0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 shutdown_connections 2023-06-10T18:57:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.056+0000 7fbda3b74700 1 -- 172.21.15.104:0/2860161627 wait complete. 2023-06-10T18:57:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 88 2023-06-10T18:57:10.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:10.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.188+0000 7fe1bfbc5700 1 Processor -- start 2023-06-10T18:57:10.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.188+0000 7fe1bfbc5700 1 -- start start 2023-06-10T18:57:10.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bfbc5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe1b806e0e0 0x7fe1b806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bfbc5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b806e4b0 0x7fe1b80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bfbc5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b806d290 con 0x7fe1b80ca100 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bfbc5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b8071640 con 0x7fe1b806e0e0 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bfbc5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80aef00 con 0x7fe1b806e4b0 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe1b806e0e0 0x7fe1b806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe1b806e0e0 0x7fe1b806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38206/0 (socket says 172.21.15.104:38206) 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 -- 172.21.15.104:0/1704145361 learned_addr learned my addr 172.21.15.104:0/1704145361 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 -- 172.21.15.104:0/1704145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b806e4b0 msgr2=0x7fe1b80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd160700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b806e4b0 0x7fe1b80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b806e4b0 0x7fe1b80c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 -- 172.21.15.104:0/1704145361 >> v1:172.21.15.114:6789/0 conn(0x7fe1b80ca100 legacy=0x7fe1b80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 -- 172.21.15.104:0/1704145361 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b806d4c0 con 0x7fe1b806e0e0 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bd961700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b806cd50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4006fd0 tx=0x7fe1b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2332ebc33e0be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.189+0000 7fe1bc95f700 1 -- 172.21.15.104:0/1704145361 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4005e30 con 0x7fe1b806e0e0 2023-06-10T18:57:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bc95f700 1 -- 172.21.15.104:0/1704145361 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b4006030 con 0x7fe1b806e0e0 2023-06-10T18:57:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bc95f700 1 -- 172.21.15.104:0/1704145361 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40063c0 con 0x7fe1b806e0e0 2023-06-10T18:57:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/1704145361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 msgr2=0x7fe1b806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b806cd50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4006fd0 tx=0x7fe1b4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/1704145361 shutdown_connections 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b806cd50 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/1704145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b806e4b0 0x7fe1b80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/1704145361 >> 172.21.15.104:0/1704145361 conn(0x7fe1b806b8a0 msgr2=0x7fe1b80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/1704145361 shutdown_connections 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.190+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/1704145361 wait complete. 2023-06-10T18:57:10.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.191+0000 7fe1bfbc5700 1 Processor -- start 2023-06-10T18:57:10.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.191+0000 7fe1bfbc5700 1 -- start start 2023-06-10T18:57:10.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1b806e4b0 0x7fe1b814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b80ca100 0x7fe1b80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b814f790 con 0x7fe1b806e0e0 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80b5150 con 0x7fe1b806e4b0 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bfbc5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80b0e90 con 0x7fe1b80ca100 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38210/0 (socket says 172.21.15.104:38210) 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 -- 172.21.15.104:0/2119784771 learned_addr learned my addr 172.21.15.104:0/2119784771 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1be162700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b80ca100 0x7fe1b80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd160700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1b806e4b0 0x7fe1b814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 -- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b80ca100 msgr2=0x7fe1b80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b80ca100 0x7fe1b80b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 -- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1b806e4b0 msgr2=0x7fe1b814f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1b806e4b0 0x7fe1b814f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.192+0000 7fe1bd961700 1 -- 172.21.15.104:0/2119784771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b4009500 con 0x7fe1b806e0e0 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1bd961700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4000c00 tx=0x7fe1b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4025430 con 0x7fe1b806e0e0 2023-06-10T18:57:10.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b4020070 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b80b1170 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40257d0 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.193+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b814f990 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.194+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b80723b0 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.194+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe1b4022030 con 0x7fe1b806e0e0 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.194+0000 7fe1ae7fc700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 0x7fe19c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.195+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe1b408ac10 con 0x7fe1b806e0e0 2023-06-10T18:57:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.195+0000 7fe1bd160700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 0x7fe19c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.195+0000 7fe1bd160700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 0x7fe19c06d190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80af200 tx=0x7fe1a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.199+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe1b4055c10 con 0x7fe1b806e0e0 2023-06-10T18:57:10.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.386+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b8074050 con 0x7fe1b806e0e0 2023-06-10T18:57:10.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.388+0000 7fe1ae7fc700 1 -- 172.21.15.104:0/2119784771 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+18614 (secure 0 0 0) 0x7fe1b4059230 con 0x7fe1b806e0e0 2023-06-10T18:57:10.397 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:10.398 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":143,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:07.273441+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:10.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 msgr2=0x7fe19c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 0x7fe19c06d190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80af200 tx=0x7fe1a8008040 comp rx=0 tx=0).stop 2023-06-10T18:57:10.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 msgr2=0x7fe1b80b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4000c00 tx=0x7fe1b4004750 comp rx=0 tx=0).stop 2023-06-10T18:57:10.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 shutdown_connections 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b80ca100 0x7fe1b80b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1b806e0e0 0x7fe1b80b4b70 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe19c06ace0 0x7fe19c06d190 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 --2- 172.21.15.104:0/2119784771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1b806e4b0 0x7fe1b814f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 >> 172.21.15.104:0/2119784771 conn(0x7fe1b806b8a0 msgr2=0x7fe1b80adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 shutdown_connections 2023-06-10T18:57:10.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.390+0000 7fe1bfbc5700 1 -- 172.21.15.104:0/2119784771 wait complete. 2023-06-10T18:57:10.409 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:57:10.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:57:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.525+0000 7fcc5f5ef700 1 Processor -- start 2023-06-10T18:57:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.525+0000 7fcc5f5ef700 1 -- start start 2023-06-10T18:57:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc580aeaa0 0x7fcc580cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcc580ca2f0 0x7fcc580ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc580c9bc0 con 0x7fcc580ca2f0 2023-06-10T18:57:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc580c9e60 con 0x7fcc580aeaa0 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5f5ef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580aee70 con 0x7fcc580b1080 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcc580ca2f0 0x7fcc580ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcc580ca2f0 0x7fcc580ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38220/0 (socket says 172.21.15.104:38220) 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1974403642 learned_addr learned my addr 172.21.15.104:0/1974403642 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5cb8a700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1974403642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 msgr2=0x7fcc580cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5d38b700 1 --2- 172.21.15.104:0/1974403642 >> v2:172.21.15.114:3300/0 conn(0x7fcc580aeaa0 0x7fcc580cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1974403642 >> v2:172.21.15.114:3300/0 conn(0x7fcc580aeaa0 msgr2=0x7fcc580cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1974403642 >> v2:172.21.15.114:3300/0 conn(0x7fcc580aeaa0 0x7fcc580cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1974403642 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5806e540 con 0x7fcc580ca2f0 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc580ca750 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fcc580cf5d0 tx=0x7fcc5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd1a819d3a043b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc4ffff700 1 -- 172.21.15.104:0/1974403642 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000e070 con 0x7fcc580ca2f0 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc4ffff700 1 -- 172.21.15.104:0/1974403642 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5000b9c0 con 0x7fcc580ca2f0 2023-06-10T18:57:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.526+0000 7fcc4ffff700 1 -- 172.21.15.104:0/1974403642 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000bd70 con 0x7fcc580ca2f0 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1974403642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 msgr2=0x7fcc580ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc580ca750 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fcc580cf5d0 tx=0x7fcc5000b330 comp rx=0 tx=0).stop 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1974403642 shutdown_connections 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc580ca750 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1974403642 >> v2:172.21.15.114:3300/0 conn(0x7fcc580aeaa0 0x7fcc580cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1974403642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1974403642 >> 172.21.15.104:0/1974403642 conn(0x7fcc5806b8a0 msgr2=0x7fcc580cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1974403642 shutdown_connections 2023-06-10T18:57:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.527+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1974403642 wait complete. 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.528+0000 7fcc5f5ef700 1 Processor -- start 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.528+0000 7fcc5f5ef700 1 -- start start 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc581e3b40 con 0x7fcc580ca2f0 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580c7590 con 0x7fcc580aeaa0 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5f5ef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580c77a0 con 0x7fcc580b1080 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38226/0 (socket says 172.21.15.104:38226) 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1551293249 learned_addr learned my addr 172.21.15.104:0/1551293249 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5cb8a700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5d38b700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 msgr2=0x7fcc580c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 msgr2=0x7fcc58156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 -- 172.21.15.104:0/1551293249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc50009b30 con 0x7fcc580ca2f0 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc5db8c700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b370 tx=0x7fcc50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.529+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50004390 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.530+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50004510 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.530+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5815a150 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.530+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500048c0 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.530+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc581e1ca0 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.531+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc58074050 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.531+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcc5001b430 con 0x7fcc580ca2f0 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.531+0000 7fcc4dffb700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 0x7fcc3c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.532+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcc5008ab10 con 0x7fcc580ca2f0 2023-06-10T18:57:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.532+0000 7fcc5d38b700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 0x7fcc3c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.532+0000 7fcc5d38b700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 0x7fcc3c06d1c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcc48006fd0 tx=0x7fcc48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.536+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcc50055b10 con 0x7fcc580ca2f0 2023-06-10T18:57:10.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.718+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc5806eb00 con 0x7fcc3c06ad10 2023-06-10T18:57:10.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.727+0000 7fcc4dffb700 1 -- 172.21.15.104:0/1551293249 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311942 (secure 0 0 0) 0x7fcc4800c040 con 0x7fcc3c06ad10 2023-06-10T18:57:10.739 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:10.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 msgr2=0x7fcc3c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 0x7fcc3c06d1c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcc48006fd0 tx=0x7fcc48008040 comp rx=0 tx=0).stop 2023-06-10T18:57:10.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 msgr2=0x7fcc58159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000b370 tx=0x7fcc50004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 shutdown_connections 2023-06-10T18:57:10.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc3c06ad10 0x7fcc3c06d1c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc580aeaa0 0x7fcc58156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc580ca2f0 0x7fcc58159c10 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 --2- 172.21.15.104:0/1551293249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc580b1080 0x7fcc580c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 >> 172.21.15.104:0/1551293249 conn(0x7fcc5806b8a0 msgr2=0x7fcc58158ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 shutdown_connections 2023-06-10T18:57:10.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.730+0000 7fcc5f5ef700 1 -- 172.21.15.104:0/1551293249 wait complete. 2023-06-10T18:57:10.747 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:57:10.748 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2023-06-10T18:57:09.557653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238664,"kb_used_data":14856,"kb_used_omap":0,"kb_used_meta":223744,"kb_avail":754736056,"statfs":{"total":773094113280,"available":772849721344,"internally_reserved":0,"allocated":15212544,"data_stored":8021564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":229113856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.688968"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301606+0000","last_change":"2023-06-10T18:57:06.297714+0000","last_active":"2023-06-10T18:57:06.301606+0000","last_peered":"2023-06-10T18:57:06.301606+0000","last_clean":"2023-06-10T18:57:06.301606+0000","last_became_active":"2023-06-10T18:57:06.297412+0000","last_became_peered":"2023-06-10T18:57:06.297412+0000","last_unstale":"2023-06-10T18:57:06.301606+0000","last_undegraded":"2023-06-10T18:57:06.301606+0000","last_fullsized":"2023-06-10T18:57:06.301606+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:38.813010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.289519+0000","last_change":"2023-06-10T18:57:06.287731+0000","last_active":"2023-06-10T18:57:06.289519+0000","last_peered":"2023-06-10T18:57:06.289519+0000","last_clean":"2023-06-10T18:57:06.289519+0000","last_became_active":"2023-06-10T18:57:06.287484+0000","last_became_peered":"2023-06-10T18:57:06.287484+0000","last_unstale":"2023-06-10T18:57:06.289519+0000","last_undegraded":"2023-06-10T18:57:06.289519+0000","last_fullsized":"2023-06-10T18:57:06.289519+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:31:26.705650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300144+0000","last_change":"2023-06-10T18:57:06.297444+0000","last_active":"2023-06-10T18:57:06.300144+0000","last_peered":"2023-06-10T18:57:06.300144+0000","last_clean":"2023-06-10T18:57:06.300144+0000","last_became_active":"2023-06-10T18:57:06.297012+0000","last_became_peered":"2023-06-10T18:57:06.297012+0000","last_unstale":"2023-06-10T18:57:06.300144+0000","last_undegraded":"2023-06-10T18:57:06.300144+0000","last_fullsized":"2023-06-10T18:57:06.300144+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:51:55.664753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.310776+0000","last_change":"2023-06-10T18:57:06.305473+0000","last_active":"2023-06-10T18:57:06.310776+0000","last_peered":"2023-06-10T18:57:06.310776+0000","last_clean":"2023-06-10T18:57:06.310776+0000","last_became_active":"2023-06-10T18:57:06.305183+0000","last_became_peered":"2023-06-10T18:57:06.305183+0000","last_unstale":"2023-06-10T18:57:06.310776+0000","last_undegraded":"2023-06-10T18:57:06.310776+0000","last_fullsized":"2023-06-10T18:57:06.310776+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:44:25.994305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301956+0000","last_change":"2023-06-10T18:57:06.296998+0000","last_active":"2023-06-10T18:57:06.301956+0000","last_peered":"2023-06-10T18:57:06.301956+0000","last_clean":"2023-06-10T18:57:06.301956+0000","last_became_active":"2023-06-10T18:57:06.296411+0000","last_became_peered":"2023-06-10T18:57:06.296411+0000","last_unstale":"2023-06-10T18:57:06.301956+0000","last_undegraded":"2023-06-10T18:57:06.301956+0000","last_fullsized":"2023-06-10T18:57:06.301956+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:44.912561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301709+0000","last_change":"2023-06-10T18:57:06.300023+0000","last_active":"2023-06-10T18:57:06.301709+0000","last_peered":"2023-06-10T18:57:06.301709+0000","last_clean":"2023-06-10T18:57:06.301709+0000","last_became_active":"2023-06-10T18:57:06.299748+0000","last_became_peered":"2023-06-10T18:57:06.299748+0000","last_unstale":"2023-06-10T18:57:06.301709+0000","last_undegraded":"2023-06-10T18:57:06.301709+0000","last_fullsized":"2023-06-10T18:57:06.301709+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:24.710048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297340+0000","last_change":"2023-06-10T18:57:06.293985+0000","last_active":"2023-06-10T18:57:06.297340+0000","last_peered":"2023-06-10T18:57:06.297340+0000","last_clean":"2023-06-10T18:57:06.297340+0000","last_became_active":"2023-06-10T18:57:06.293691+0000","last_became_peered":"2023-06-10T18:57:06.293691+0000","last_unstale":"2023-06-10T18:57:06.297340+0000","last_undegraded":"2023-06-10T18:57:06.297340+0000","last_fullsized":"2023-06-10T18:57:06.297340+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:06:57.488806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297815+0000","last_change":"2023-06-10T18:57:06.291070+0000","last_active":"2023-06-10T18:57:06.297815+0000","last_peered":"2023-06-10T18:57:06.297815+0000","last_clean":"2023-06-10T18:57:06.297815+0000","last_became_active":"2023-06-10T18:57:06.290761+0000","last_became_peered":"2023-06-10T18:57:06.290761+0000","last_unstale":"2023-06-10T18:57:06.297815+0000","last_undegraded":"2023-06-10T18:57:06.297815+0000","last_fullsized":"2023-06-10T18:57:06.297815+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:51:41.766689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.293185+0000","last_change":"2023-06-10T18:57:06.292298+0000","last_active":"2023-06-10T18:57:06.293185+0000","last_peered":"2023-06-10T18:57:06.293185+0000","last_clean":"2023-06-10T18:57:06.293185+0000","last_became_active":"2023-06-10T18:57:06.292029+0000","last_became_peered":"2023-06-10T18:57:06.292029+0000","last_unstale":"2023-06-10T18:57:06.293185+0000","last_undegraded":"2023-06-10T18:57:06.293185+0000","last_fullsized":"2023-06-10T18:57:06.293185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:02.889694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303551+0000","last_change":"2023-06-10T18:57:06.291645+0000","last_active":"2023-06-10T18:57:06.303551+0000","last_peered":"2023-06-10T18:57:06.303551+0000","last_clean":"2023-06-10T18:57:06.303551+0000","last_became_active":"2023-06-10T18:57:06.291360+0000","last_became_peered":"2023-06-10T18:57:06.291360+0000","last_unstale":"2023-06-10T18:57:06.303551+0000","last_undegraded":"2023-06-10T18:57:06.303551+0000","last_fullsized":"2023-06-10T18:57:06.303551+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:59:33.952373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291326+0000","last_change":"2023-06-10T18:57:06.289639+0000","last_active":"2023-06-10T18:57:06.291326+0000","last_peered":"2023-06-10T18:57:06.291326+0000","last_clean":"2023-06-10T18:57:06.291326+0000","last_became_active":"2023-06-10T18:57:06.289374+0000","last_became_peered":"2023-06-10T18:57:06.289374+0000","last_unstale":"2023-06-10T18:57:06.291326+0000","last_undegraded":"2023-06-10T18:57:06.291326+0000","last_fullsized":"2023-06-10T18:57:06.291326+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:52.532414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304492+0000","last_change":"2023-06-10T18:57:06.291060+0000","last_active":"2023-06-10T18:57:06.304492+0000","last_peered":"2023-06-10T18:57:06.304492+0000","last_clean":"2023-06-10T18:57:06.304492+0000","last_became_active":"2023-06-10T18:57:06.290761+0000","last_became_peered":"2023-06-10T18:57:06.290761+0000","last_unstale":"2023-06-10T18:57:06.304492+0000","last_undegraded":"2023-06-10T18:57:06.304492+0000","last_fullsized":"2023-06-10T18:57:06.304492+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:40.224886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301976+0000","last_change":"2023-06-10T18:57:06.297178+0000","last_active":"2023-06-10T18:57:06.301976+0000","last_peered":"2023-06-10T18:57:06.301976+0000","last_clean":"2023-06-10T18:57:06.301976+0000","last_became_active":"2023-06-10T18:57:06.296744+0000","last_became_peered":"2023-06-10T18:57:06.296744+0000","last_unstale":"2023-06-10T18:57:06.301976+0000","last_undegraded":"2023-06-10T18:57:06.301976+0000","last_fullsized":"2023-06-10T18:57:06.301976+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:06:33.493990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308525+0000","last_change":"2023-06-10T18:57:06.301659+0000","last_active":"2023-06-10T18:57:06.308525+0000","last_peered":"2023-06-10T18:57:06.308525+0000","last_clean":"2023-06-10T18:57:06.308525+0000","last_became_active":"2023-06-10T18:57:06.301393+0000","last_became_peered":"2023-06-10T18:57:06.301393+0000","last_unstale":"2023-06-10T18:57:06.308525+0000","last_undegraded":"2023-06-10T18:57:06.308525+0000","last_fullsized":"2023-06-10T18:57:06.308525+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:52.061556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303611+0000","last_change":"2023-06-10T18:57:06.290826+0000","last_active":"2023-06-10T18:57:06.303611+0000","last_peered":"2023-06-10T18:57:06.303611+0000","last_clean":"2023-06-10T18:57:06.303611+0000","last_became_active":"2023-06-10T18:57:06.288114+0000","last_became_peered":"2023-06-10T18:57:06.288114+0000","last_unstale":"2023-06-10T18:57:06.303611+0000","last_undegraded":"2023-06-10T18:57:06.303611+0000","last_fullsized":"2023-06-10T18:57:06.303611+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:33.596698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296258+0000","last_change":"2023-06-10T18:57:06.291980+0000","last_active":"2023-06-10T18:57:06.296258+0000","last_peered":"2023-06-10T18:57:06.296258+0000","last_clean":"2023-06-10T18:57:06.296258+0000","last_became_active":"2023-06-10T18:57:06.291725+0000","last_became_peered":"2023-06-10T18:57:06.291725+0000","last_unstale":"2023-06-10T18:57:06.296258+0000","last_undegraded":"2023-06-10T18:57:06.296258+0000","last_fullsized":"2023-06-10T18:57:06.296258+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:54.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303768+0000","last_change":"2023-06-10T18:57:06.295934+0000","last_active":"2023-06-10T18:57:06.303768+0000","last_peered":"2023-06-10T18:57:06.303768+0000","last_clean":"2023-06-10T18:57:06.303768+0000","last_became_active":"2023-06-10T18:57:06.295652+0000","last_became_peered":"2023-06-10T18:57:06.295652+0000","last_unstale":"2023-06-10T18:57:06.303768+0000","last_undegraded":"2023-06-10T18:57:06.303768+0000","last_fullsized":"2023-06-10T18:57:06.303768+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:35.395931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300203+0000","last_change":"2023-06-10T18:57:06.295669+0000","last_active":"2023-06-10T18:57:06.300203+0000","last_peered":"2023-06-10T18:57:06.300203+0000","last_clean":"2023-06-10T18:57:06.300203+0000","last_became_active":"2023-06-10T18:57:06.295202+0000","last_became_peered":"2023-06-10T18:57:06.295202+0000","last_unstale":"2023-06-10T18:57:06.300203+0000","last_undegraded":"2023-06-10T18:57:06.300203+0000","last_fullsized":"2023-06-10T18:57:06.300203+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:35.567178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291307+0000","last_change":"2023-06-10T18:57:06.288875+0000","last_active":"2023-06-10T18:57:06.291307+0000","last_peered":"2023-06-10T18:57:06.291307+0000","last_clean":"2023-06-10T18:57:06.291307+0000","last_became_active":"2023-06-10T18:57:06.288621+0000","last_became_peered":"2023-06-10T18:57:06.288621+0000","last_unstale":"2023-06-10T18:57:06.291307+0000","last_undegraded":"2023-06-10T18:57:06.291307+0000","last_fullsized":"2023-06-10T18:57:06.291307+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:22:32.631362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.292439+0000","last_change":"2023-06-10T18:57:06.285317+0000","last_active":"2023-06-10T18:57:06.292439+0000","last_peered":"2023-06-10T18:57:06.292439+0000","last_clean":"2023-06-10T18:57:06.292439+0000","last_became_active":"2023-06-10T18:57:06.285025+0000","last_became_peered":"2023-06-10T18:57:06.285025+0000","last_unstale":"2023-06-10T18:57:06.292439+0000","last_undegraded":"2023-06-10T18:57:06.292439+0000","last_fullsized":"2023-06-10T18:57:06.292439+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:31:38.917623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308103+0000","last_change":"2023-06-10T18:57:06.303459+0000","last_active":"2023-06-10T18:57:06.308103+0000","last_peered":"2023-06-10T18:57:06.308103+0000","last_clean":"2023-06-10T18:57:06.308103+0000","last_became_active":"2023-06-10T18:57:06.303154+0000","last_became_peered":"2023-06-10T18:57:06.303154+0000","last_unstale":"2023-06-10T18:57:06.308103+0000","last_undegraded":"2023-06-10T18:57:06.308103+0000","last_fullsized":"2023-06-10T18:57:06.308103+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:32.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301219+0000","last_change":"2023-06-10T18:57:06.294969+0000","last_active":"2023-06-10T18:57:06.301219+0000","last_peered":"2023-06-10T18:57:06.301219+0000","last_clean":"2023-06-10T18:57:06.301219+0000","last_became_active":"2023-06-10T18:57:06.294288+0000","last_became_peered":"2023-06-10T18:57:06.294288+0000","last_unstale":"2023-06-10T18:57:06.301219+0000","last_undegraded":"2023-06-10T18:57:06.301219+0000","last_fullsized":"2023-06-10T18:57:06.301219+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:56:48.995690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308438+0000","last_change":"2023-06-10T18:57:06.301269+0000","last_active":"2023-06-10T18:57:06.308438+0000","last_peered":"2023-06-10T18:57:06.308438+0000","last_clean":"2023-06-10T18:57:06.308438+0000","last_became_active":"2023-06-10T18:57:06.300578+0000","last_became_peered":"2023-06-10T18:57:06.300578+0000","last_unstale":"2023-06-10T18:57:06.308438+0000","last_undegraded":"2023-06-10T18:57:06.308438+0000","last_fullsized":"2023-06-10T18:57:06.308438+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:43:01.323340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303659+0000","last_change":"2023-06-10T18:57:06.295930+0000","last_active":"2023-06-10T18:57:06.303659+0000","last_peered":"2023-06-10T18:57:06.303659+0000","last_clean":"2023-06-10T18:57:06.303659+0000","last_became_active":"2023-06-10T18:57:06.295634+0000","last_became_peered":"2023-06-10T18:57:06.295634+0000","last_unstale":"2023-06-10T18:57:06.303659+0000","last_undegraded":"2023-06-10T18:57:06.303659+0000","last_fullsized":"2023-06-10T18:57:06.303659+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:40:36.880638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296230+0000","last_change":"2023-06-10T18:57:06.291768+0000","last_active":"2023-06-10T18:57:06.296230+0000","last_peered":"2023-06-10T18:57:06.296230+0000","last_clean":"2023-06-10T18:57:06.296230+0000","last_became_active":"2023-06-10T18:57:06.291502+0000","last_became_peered":"2023-06-10T18:57:06.291502+0000","last_unstale":"2023-06-10T18:57:06.296230+0000","last_undegraded":"2023-06-10T18:57:06.296230+0000","last_fullsized":"2023-06-10T18:57:06.296230+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:19:02.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291266+0000","last_change":"2023-06-10T18:57:06.288263+0000","last_active":"2023-06-10T18:57:06.291266+0000","last_peered":"2023-06-10T18:57:06.291266+0000","last_clean":"2023-06-10T18:57:06.291266+0000","last_became_active":"2023-06-10T18:57:06.287725+0000","last_became_peered":"2023-06-10T18:57:06.287725+0000","last_unstale":"2023-06-10T18:57:06.291266+0000","last_undegraded":"2023-06-10T18:57:06.291266+0000","last_fullsized":"2023-06-10T18:57:06.291266+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:46.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296567+0000","last_change":"2023-06-10T18:57:06.294527+0000","last_active":"2023-06-10T18:57:06.296567+0000","last_peered":"2023-06-10T18:57:06.296567+0000","last_clean":"2023-06-10T18:57:06.296567+0000","last_became_active":"2023-06-10T18:57:06.294265+0000","last_became_peered":"2023-06-10T18:57:06.294265+0000","last_unstale":"2023-06-10T18:57:06.296567+0000","last_undegraded":"2023-06-10T18:57:06.296567+0000","last_fullsized":"2023-06-10T18:57:06.296567+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:02:21.698563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301661+0000","last_change":"2023-06-10T18:57:06.299820+0000","last_active":"2023-06-10T18:57:06.301661+0000","last_peered":"2023-06-10T18:57:06.301661+0000","last_clean":"2023-06-10T18:57:06.301661+0000","last_became_active":"2023-06-10T18:57:06.299562+0000","last_became_peered":"2023-06-10T18:57:06.299562+0000","last_unstale":"2023-06-10T18:57:06.301661+0000","last_undegraded":"2023-06-10T18:57:06.301661+0000","last_fullsized":"2023-06-10T18:57:06.301661+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:44:14.441609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301725+0000","last_change":"2023-06-10T18:57:06.300147+0000","last_active":"2023-06-10T18:57:06.301725+0000","last_peered":"2023-06-10T18:57:06.301725+0000","last_clean":"2023-06-10T18:57:06.301725+0000","last_became_active":"2023-06-10T18:57:06.299910+0000","last_became_peered":"2023-06-10T18:57:06.299910+0000","last_unstale":"2023-06-10T18:57:06.301725+0000","last_undegraded":"2023-06-10T18:57:06.301725+0000","last_fullsized":"2023-06-10T18:57:06.301725+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:47.638826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308396+0000","last_change":"2023-06-10T18:57:06.300152+0000","last_active":"2023-06-10T18:57:06.308396+0000","last_peered":"2023-06-10T18:57:06.308396+0000","last_clean":"2023-06-10T18:57:06.308396+0000","last_became_active":"2023-06-10T18:57:06.299138+0000","last_became_peered":"2023-06-10T18:57:06.299138+0000","last_unstale":"2023-06-10T18:57:06.308396+0000","last_undegraded":"2023-06-10T18:57:06.308396+0000","last_fullsized":"2023-06-10T18:57:06.308396+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:23:32.937223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296130+0000","last_change":"2023-06-10T18:57:06.292989+0000","last_active":"2023-06-10T18:57:06.296130+0000","last_peered":"2023-06-10T18:57:06.296130+0000","last_clean":"2023-06-10T18:57:06.296130+0000","last_became_active":"2023-06-10T18:57:06.292794+0000","last_became_peered":"2023-06-10T18:57:06.292794+0000","last_unstale":"2023-06-10T18:57:06.296130+0000","last_undegraded":"2023-06-10T18:57:06.296130+0000","last_fullsized":"2023-06-10T18:57:06.296130+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:54.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296189+0000","last_change":"2023-06-10T18:57:06.289366+0000","last_active":"2023-06-10T18:57:06.296189+0000","last_peered":"2023-06-10T18:57:06.296189+0000","last_clean":"2023-06-10T18:57:06.296189+0000","last_became_active":"2023-06-10T18:57:06.289133+0000","last_became_peered":"2023-06-10T18:57:06.289133+0000","last_unstale":"2023-06-10T18:57:06.296189+0000","last_undegraded":"2023-06-10T18:57:06.296189+0000","last_fullsized":"2023-06-10T18:57:06.296189+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:50.245154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303530+0000","last_change":"2023-06-10T18:57:06.291559+0000","last_active":"2023-06-10T18:57:06.303530+0000","last_peered":"2023-06-10T18:57:06.303530+0000","last_clean":"2023-06-10T18:57:06.303530+0000","last_became_active":"2023-06-10T18:57:06.291298+0000","last_became_peered":"2023-06-10T18:57:06.291298+0000","last_unstale":"2023-06-10T18:57:06.303530+0000","last_undegraded":"2023-06-10T18:57:06.303530+0000","last_fullsized":"2023-06-10T18:57:06.303530+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:31:02.419870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"142'4","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303705+0000","last_change":"2023-06-10T18:57:05.890463+0000","last_active":"2023-06-10T18:57:06.303705+0000","last_peered":"2023-06-10T18:57:06.303705+0000","last_clean":"2023-06-10T18:57:06.303705+0000","last_became_active":"2023-06-10T18:57:05.890169+0000","last_became_peered":"2023-06-10T18:57:05.890169+0000","last_unstale":"2023-06-10T18:57:06.303705+0000","last_undegraded":"2023-06-10T18:57:06.303705+0000","last_fullsized":"2023-06-10T18:57:06.303705+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:44:58.842632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288238+0000","last_change":"2023-06-10T18:57:06.287075+0000","last_active":"2023-06-10T18:57:06.288238+0000","last_peered":"2023-06-10T18:57:06.288238+0000","last_clean":"2023-06-10T18:57:06.288238+0000","last_became_active":"2023-06-10T18:57:06.286824+0000","last_became_peered":"2023-06-10T18:57:06.286824+0000","last_unstale":"2023-06-10T18:57:06.288238+0000","last_undegraded":"2023-06-10T18:57:06.288238+0000","last_fullsized":"2023-06-10T18:57:06.288238+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:40:52.437963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294921+0000","last_change":"2023-06-10T18:57:05.891307+0000","last_active":"2023-06-10T18:57:06.294921+0000","last_peered":"2023-06-10T18:57:06.294921+0000","last_clean":"2023-06-10T18:57:06.294921+0000","last_became_active":"2023-06-10T18:57:05.890991+0000","last_became_peered":"2023-06-10T18:57:05.890991+0000","last_unstale":"2023-06-10T18:57:06.294921+0000","last_undegraded":"2023-06-10T18:57:06.294921+0000","last_fullsized":"2023-06-10T18:57:06.294921+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:04:42.293086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298082+0000","last_change":"2023-06-10T18:57:06.295683+0000","last_active":"2023-06-10T18:57:06.298082+0000","last_peered":"2023-06-10T18:57:06.298082+0000","last_clean":"2023-06-10T18:57:06.298082+0000","last_became_active":"2023-06-10T18:57:06.295247+0000","last_became_peered":"2023-06-10T18:57:06.295247+0000","last_unstale":"2023-06-10T18:57:06.298082+0000","last_undegraded":"2023-06-10T18:57:06.298082+0000","last_fullsized":"2023-06-10T18:57:06.298082+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:39.968687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304730+0000","last_change":"2023-06-10T18:57:05.897535+0000","last_active":"2023-06-10T18:57:06.304730+0000","last_peered":"2023-06-10T18:57:06.304730+0000","last_clean":"2023-06-10T18:57:06.304730+0000","last_became_active":"2023-06-10T18:57:05.897201+0000","last_became_peered":"2023-06-10T18:57:05.897201+0000","last_unstale":"2023-06-10T18:57:06.304730+0000","last_undegraded":"2023-06-10T18:57:06.304730+0000","last_fullsized":"2023-06-10T18:57:06.304730+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:36:55.303678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303542+0000","last_change":"2023-06-10T18:57:06.291018+0000","last_active":"2023-06-10T18:57:06.303542+0000","last_peered":"2023-06-10T18:57:06.303542+0000","last_clean":"2023-06-10T18:57:06.303542+0000","last_became_active":"2023-06-10T18:57:06.290720+0000","last_became_peered":"2023-06-10T18:57:06.290720+0000","last_unstale":"2023-06-10T18:57:06.303542+0000","last_undegraded":"2023-06-10T18:57:06.303542+0000","last_fullsized":"2023-06-10T18:57:06.303542+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:55.165083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306054+0000","last_change":"2023-06-10T18:57:05.897501+0000","last_active":"2023-06-10T18:57:06.306054+0000","last_peered":"2023-06-10T18:57:06.306054+0000","last_clean":"2023-06-10T18:57:06.306054+0000","last_became_active":"2023-06-10T18:57:05.897209+0000","last_became_peered":"2023-06-10T18:57:05.897209+0000","last_unstale":"2023-06-10T18:57:06.306054+0000","last_undegraded":"2023-06-10T18:57:06.306054+0000","last_fullsized":"2023-06-10T18:57:06.306054+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:10:55.733983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303346+0000","last_change":"2023-06-10T18:57:06.300013+0000","last_active":"2023-06-10T18:57:06.303346+0000","last_peered":"2023-06-10T18:57:06.303346+0000","last_clean":"2023-06-10T18:57:06.303346+0000","last_became_active":"2023-06-10T18:57:06.299765+0000","last_became_peered":"2023-06-10T18:57:06.299765+0000","last_unstale":"2023-06-10T18:57:06.303346+0000","last_undegraded":"2023-06-10T18:57:06.303346+0000","last_fullsized":"2023-06-10T18:57:06.303346+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:29.838822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295005+0000","last_change":"2023-06-10T18:57:05.891709+0000","last_active":"2023-06-10T18:57:06.295005+0000","last_peered":"2023-06-10T18:57:06.295005+0000","last_clean":"2023-06-10T18:57:06.295005+0000","last_became_active":"2023-06-10T18:57:05.891451+0000","last_became_peered":"2023-06-10T18:57:05.891451+0000","last_unstale":"2023-06-10T18:57:06.295005+0000","last_undegraded":"2023-06-10T18:57:06.295005+0000","last_fullsized":"2023-06-10T18:57:06.295005+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:38:19.155656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308499+0000","last_change":"2023-06-10T18:57:06.301633+0000","last_active":"2023-06-10T18:57:06.308499+0000","last_peered":"2023-06-10T18:57:06.308499+0000","last_clean":"2023-06-10T18:57:06.308499+0000","last_became_active":"2023-06-10T18:57:06.301373+0000","last_became_peered":"2023-06-10T18:57:06.301373+0000","last_unstale":"2023-06-10T18:57:06.308499+0000","last_undegraded":"2023-06-10T18:57:06.308499+0000","last_fullsized":"2023-06-10T18:57:06.308499+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:39:31.292187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296280+0000","last_change":"2023-06-10T18:57:05.891907+0000","last_active":"2023-06-10T18:57:06.296280+0000","last_peered":"2023-06-10T18:57:06.296280+0000","last_clean":"2023-06-10T18:57:06.296280+0000","last_became_active":"2023-06-10T18:57:05.891639+0000","last_became_peered":"2023-06-10T18:57:05.891639+0000","last_unstale":"2023-06-10T18:57:06.296280+0000","last_undegraded":"2023-06-10T18:57:06.296280+0000","last_fullsized":"2023-06-10T18:57:06.296280+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:46.630726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288089+0000","last_change":"2023-06-10T18:57:06.285846+0000","last_active":"2023-06-10T18:57:06.288089+0000","last_peered":"2023-06-10T18:57:06.288089+0000","last_clean":"2023-06-10T18:57:06.288089+0000","last_became_active":"2023-06-10T18:57:06.285594+0000","last_became_peered":"2023-06-10T18:57:06.285594+0000","last_unstale":"2023-06-10T18:57:06.288089+0000","last_undegraded":"2023-06-10T18:57:06.288089+0000","last_fullsized":"2023-06-10T18:57:06.288089+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:47.982305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280621+0000","last_change":"2023-06-10T18:57:05.894080+0000","last_active":"2023-06-10T18:57:06.280621+0000","last_peered":"2023-06-10T18:57:06.280621+0000","last_clean":"2023-06-10T18:57:06.280621+0000","last_became_active":"2023-06-10T18:57:05.893807+0000","last_became_peered":"2023-06-10T18:57:05.893807+0000","last_unstale":"2023-06-10T18:57:06.280621+0000","last_undegraded":"2023-06-10T18:57:06.280621+0000","last_fullsized":"2023-06-10T18:57:06.280621+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:18.949540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302165+0000","last_change":"2023-06-10T18:57:06.293271+0000","last_active":"2023-06-10T18:57:06.302165+0000","last_peered":"2023-06-10T18:57:06.302165+0000","last_clean":"2023-06-10T18:57:06.302165+0000","last_became_active":"2023-06-10T18:57:06.293012+0000","last_became_peered":"2023-06-10T18:57:06.293012+0000","last_unstale":"2023-06-10T18:57:06.302165+0000","last_undegraded":"2023-06-10T18:57:06.302165+0000","last_fullsized":"2023-06-10T18:57:06.302165+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:44:52.255039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291872+0000","last_change":"2023-06-10T18:57:05.894286+0000","last_active":"2023-06-10T18:57:06.291872+0000","last_peered":"2023-06-10T18:57:06.291872+0000","last_clean":"2023-06-10T18:57:06.291872+0000","last_became_active":"2023-06-10T18:57:05.894010+0000","last_became_peered":"2023-06-10T18:57:05.894010+0000","last_unstale":"2023-06-10T18:57:06.291872+0000","last_undegraded":"2023-06-10T18:57:06.291872+0000","last_fullsized":"2023-06-10T18:57:06.291872+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:16.330457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303570+0000","last_change":"2023-06-10T18:57:06.296034+0000","last_active":"2023-06-10T18:57:06.303570+0000","last_peered":"2023-06-10T18:57:06.303570+0000","last_clean":"2023-06-10T18:57:06.303570+0000","last_became_active":"2023-06-10T18:57:06.295133+0000","last_became_peered":"2023-06-10T18:57:06.295133+0000","last_unstale":"2023-06-10T18:57:06.303570+0000","last_undegraded":"2023-06-10T18:57:06.303570+0000","last_fullsized":"2023-06-10T18:57:06.303570+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:40:40.257206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288982+0000","last_change":"2023-06-10T18:57:05.894027+0000","last_active":"2023-06-10T18:57:06.288982+0000","last_peered":"2023-06-10T18:57:06.288982+0000","last_clean":"2023-06-10T18:57:06.288982+0000","last_became_active":"2023-06-10T18:57:05.893740+0000","last_became_peered":"2023-06-10T18:57:05.893740+0000","last_unstale":"2023-06-10T18:57:06.288982+0000","last_undegraded":"2023-06-10T18:57:06.288982+0000","last_fullsized":"2023-06-10T18:57:06.288982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:49:39.686208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301825+0000","last_change":"2023-06-10T18:57:06.298565+0000","last_active":"2023-06-10T18:57:06.301825+0000","last_peered":"2023-06-10T18:57:06.301825+0000","last_clean":"2023-06-10T18:57:06.301825+0000","last_became_active":"2023-06-10T18:57:06.298293+0000","last_became_peered":"2023-06-10T18:57:06.298293+0000","last_unstale":"2023-06-10T18:57:06.301825+0000","last_undegraded":"2023-06-10T18:57:06.301825+0000","last_fullsized":"2023-06-10T18:57:06.301825+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:57.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291743+0000","last_change":"2023-06-10T18:57:05.896585+0000","last_active":"2023-06-10T18:57:06.291743+0000","last_peered":"2023-06-10T18:57:06.291743+0000","last_clean":"2023-06-10T18:57:06.291743+0000","last_became_active":"2023-06-10T18:57:05.896296+0000","last_became_peered":"2023-06-10T18:57:05.896296+0000","last_unstale":"2023-06-10T18:57:06.291743+0000","last_undegraded":"2023-06-10T18:57:06.291743+0000","last_fullsized":"2023-06-10T18:57:06.291743+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:12.801550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.310881+0000","last_change":"2023-06-10T18:57:06.306547+0000","last_active":"2023-06-10T18:57:06.310881+0000","last_peered":"2023-06-10T18:57:06.310881+0000","last_clean":"2023-06-10T18:57:06.310881+0000","last_became_active":"2023-06-10T18:57:06.306251+0000","last_became_peered":"2023-06-10T18:57:06.306251+0000","last_unstale":"2023-06-10T18:57:06.310881+0000","last_undegraded":"2023-06-10T18:57:06.310881+0000","last_fullsized":"2023-06-10T18:57:06.310881+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:36.407925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295152+0000","last_change":"2023-06-10T18:57:05.891658+0000","last_active":"2023-06-10T18:57:06.295152+0000","last_peered":"2023-06-10T18:57:06.295152+0000","last_clean":"2023-06-10T18:57:06.295152+0000","last_became_active":"2023-06-10T18:57:05.891453+0000","last_became_peered":"2023-06-10T18:57:05.891453+0000","last_unstale":"2023-06-10T18:57:06.295152+0000","last_undegraded":"2023-06-10T18:57:06.295152+0000","last_fullsized":"2023-06-10T18:57:06.295152+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:51:06.693787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303943+0000","last_change":"2023-06-10T18:57:06.301440+0000","last_active":"2023-06-10T18:57:06.303943+0000","last_peered":"2023-06-10T18:57:06.303943+0000","last_clean":"2023-06-10T18:57:06.303943+0000","last_became_active":"2023-06-10T18:57:06.300823+0000","last_became_peered":"2023-06-10T18:57:06.300823+0000","last_unstale":"2023-06-10T18:57:06.303943+0000","last_undegraded":"2023-06-10T18:57:06.303943+0000","last_fullsized":"2023-06-10T18:57:06.303943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:44.912561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280989+0000","last_change":"2023-06-10T18:57:05.893739+0000","last_active":"2023-06-10T18:57:06.280989+0000","last_peered":"2023-06-10T18:57:06.280989+0000","last_clean":"2023-06-10T18:57:06.280989+0000","last_became_active":"2023-06-10T18:57:05.893509+0000","last_became_peered":"2023-06-10T18:57:05.893509+0000","last_unstale":"2023-06-10T18:57:06.280989+0000","last_undegraded":"2023-06-10T18:57:06.280989+0000","last_fullsized":"2023-06-10T18:57:06.280989+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:37.414047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301242+0000","last_change":"2023-06-10T18:57:06.295139+0000","last_active":"2023-06-10T18:57:06.301242+0000","last_peered":"2023-06-10T18:57:06.301242+0000","last_clean":"2023-06-10T18:57:06.301242+0000","last_became_active":"2023-06-10T18:57:06.294710+0000","last_became_peered":"2023-06-10T18:57:06.294710+0000","last_unstale":"2023-06-10T18:57:06.301242+0000","last_undegraded":"2023-06-10T18:57:06.301242+0000","last_fullsized":"2023-06-10T18:57:06.301242+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:40.099913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300572+0000","last_change":"2023-06-10T18:57:05.894270+0000","last_active":"2023-06-10T18:57:06.300572+0000","last_peered":"2023-06-10T18:57:06.300572+0000","last_clean":"2023-06-10T18:57:06.300572+0000","last_became_active":"2023-06-10T18:57:05.894007+0000","last_became_peered":"2023-06-10T18:57:05.894007+0000","last_unstale":"2023-06-10T18:57:06.300572+0000","last_undegraded":"2023-06-10T18:57:06.300572+0000","last_fullsized":"2023-06-10T18:57:06.300572+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:32:24.362671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297462+0000","last_change":"2023-06-10T18:57:06.294563+0000","last_active":"2023-06-10T18:57:06.297462+0000","last_peered":"2023-06-10T18:57:06.297462+0000","last_clean":"2023-06-10T18:57:06.297462+0000","last_became_active":"2023-06-10T18:57:06.294315+0000","last_became_peered":"2023-06-10T18:57:06.294315+0000","last_unstale":"2023-06-10T18:57:06.297462+0000","last_undegraded":"2023-06-10T18:57:06.297462+0000","last_fullsized":"2023-06-10T18:57:06.297462+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:57.399990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276667+0000","last_change":"2023-06-10T18:57:05.891287+0000","last_active":"2023-06-10T18:57:06.276667+0000","last_peered":"2023-06-10T18:57:06.276667+0000","last_clean":"2023-06-10T18:57:06.276667+0000","last_became_active":"2023-06-10T18:57:05.890926+0000","last_became_peered":"2023-06-10T18:57:05.890926+0000","last_unstale":"2023-06-10T18:57:06.276667+0000","last_undegraded":"2023-06-10T18:57:06.276667+0000","last_fullsized":"2023-06-10T18:57:06.276667+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:16:51.861690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304778+0000","last_change":"2023-06-10T18:57:06.296990+0000","last_active":"2023-06-10T18:57:06.304778+0000","last_peered":"2023-06-10T18:57:06.304778+0000","last_clean":"2023-06-10T18:57:06.304778+0000","last_became_active":"2023-06-10T18:57:06.296735+0000","last_became_peered":"2023-06-10T18:57:06.296735+0000","last_unstale":"2023-06-10T18:57:06.304778+0000","last_undegraded":"2023-06-10T18:57:06.304778+0000","last_fullsized":"2023-06-10T18:57:06.304778+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:08:53.770582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276831+0000","last_change":"2023-06-10T18:57:05.882543+0000","last_active":"2023-06-10T18:57:06.276831+0000","last_peered":"2023-06-10T18:57:06.276831+0000","last_clean":"2023-06-10T18:57:06.276831+0000","last_became_active":"2023-06-10T18:57:05.882240+0000","last_became_peered":"2023-06-10T18:57:05.882240+0000","last_unstale":"2023-06-10T18:57:06.276831+0000","last_undegraded":"2023-06-10T18:57:06.276831+0000","last_fullsized":"2023-06-10T18:57:06.276831+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:29:23.642745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297318+0000","last_change":"2023-06-10T18:57:06.293974+0000","last_active":"2023-06-10T18:57:06.297318+0000","last_peered":"2023-06-10T18:57:06.297318+0000","last_clean":"2023-06-10T18:57:06.297318+0000","last_became_active":"2023-06-10T18:57:06.293657+0000","last_became_peered":"2023-06-10T18:57:06.293657+0000","last_unstale":"2023-06-10T18:57:06.297318+0000","last_undegraded":"2023-06-10T18:57:06.297318+0000","last_fullsized":"2023-06-10T18:57:06.297318+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:17:47.371118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.292966+0000","last_change":"2023-06-10T18:57:05.899655+0000","last_active":"2023-06-10T18:57:06.292966+0000","last_peered":"2023-06-10T18:57:06.292966+0000","last_clean":"2023-06-10T18:57:06.292966+0000","last_became_active":"2023-06-10T18:57:05.899375+0000","last_became_peered":"2023-06-10T18:57:05.899375+0000","last_unstale":"2023-06-10T18:57:06.292966+0000","last_undegraded":"2023-06-10T18:57:06.292966+0000","last_fullsized":"2023-06-10T18:57:06.292966+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:42.047607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302065+0000","last_change":"2023-06-10T18:57:06.297782+0000","last_active":"2023-06-10T18:57:06.302065+0000","last_peered":"2023-06-10T18:57:06.302065+0000","last_clean":"2023-06-10T18:57:06.302065+0000","last_became_active":"2023-06-10T18:57:06.297528+0000","last_became_peered":"2023-06-10T18:57:06.297528+0000","last_unstale":"2023-06-10T18:57:06.302065+0000","last_undegraded":"2023-06-10T18:57:06.302065+0000","last_fullsized":"2023-06-10T18:57:06.302065+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:05:33.759425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.299006+0000","last_change":"2023-06-10T18:57:05.894777+0000","last_active":"2023-06-10T18:57:06.299006+0000","last_peered":"2023-06-10T18:57:06.299006+0000","last_clean":"2023-06-10T18:57:06.299006+0000","last_became_active":"2023-06-10T18:57:05.894495+0000","last_became_peered":"2023-06-10T18:57:05.894495+0000","last_unstale":"2023-06-10T18:57:06.299006+0000","last_undegraded":"2023-06-10T18:57:06.299006+0000","last_fullsized":"2023-06-10T18:57:06.299006+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:59:16.984055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305112+0000","last_change":"2023-06-10T18:57:06.303500+0000","last_active":"2023-06-10T18:57:06.305112+0000","last_peered":"2023-06-10T18:57:06.305112+0000","last_clean":"2023-06-10T18:57:06.305112+0000","last_became_active":"2023-06-10T18:57:06.302848+0000","last_became_peered":"2023-06-10T18:57:06.302848+0000","last_unstale":"2023-06-10T18:57:06.305112+0000","last_undegraded":"2023-06-10T18:57:06.305112+0000","last_fullsized":"2023-06-10T18:57:06.305112+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:22:44.460864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281122+0000","last_change":"2023-06-10T18:57:05.891356+0000","last_active":"2023-06-10T18:57:06.281122+0000","last_peered":"2023-06-10T18:57:06.281122+0000","last_clean":"2023-06-10T18:57:06.281122+0000","last_became_active":"2023-06-10T18:57:05.891063+0000","last_became_peered":"2023-06-10T18:57:05.891063+0000","last_unstale":"2023-06-10T18:57:06.281122+0000","last_undegraded":"2023-06-10T18:57:06.281122+0000","last_fullsized":"2023-06-10T18:57:06.281122+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:05:49.162586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296977+0000","last_change":"2023-06-10T18:57:06.294500+0000","last_active":"2023-06-10T18:57:06.296977+0000","last_peered":"2023-06-10T18:57:06.296977+0000","last_clean":"2023-06-10T18:57:06.296977+0000","last_became_active":"2023-06-10T18:57:06.294241+0000","last_became_peered":"2023-06-10T18:57:06.294241+0000","last_unstale":"2023-06-10T18:57:06.296977+0000","last_undegraded":"2023-06-10T18:57:06.296977+0000","last_fullsized":"2023-06-10T18:57:06.296977+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:22:06.734988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281031+0000","last_change":"2023-06-10T18:57:05.893984+0000","last_active":"2023-06-10T18:57:06.281031+0000","last_peered":"2023-06-10T18:57:06.281031+0000","last_clean":"2023-06-10T18:57:06.281031+0000","last_became_active":"2023-06-10T18:57:05.893698+0000","last_became_peered":"2023-06-10T18:57:05.893698+0000","last_unstale":"2023-06-10T18:57:06.281031+0000","last_undegraded":"2023-06-10T18:57:06.281031+0000","last_fullsized":"2023-06-10T18:57:06.281031+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:48:10.217395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303961+0000","last_change":"2023-06-10T18:57:06.301590+0000","last_active":"2023-06-10T18:57:06.303961+0000","last_peered":"2023-06-10T18:57:06.303961+0000","last_clean":"2023-06-10T18:57:06.303961+0000","last_became_active":"2023-06-10T18:57:06.301196+0000","last_became_peered":"2023-06-10T18:57:06.301196+0000","last_unstale":"2023-06-10T18:57:06.303961+0000","last_undegraded":"2023-06-10T18:57:06.303961+0000","last_fullsized":"2023-06-10T18:57:06.303961+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:52:42.334283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.282965+0000","last_change":"2023-06-10T18:57:05.893650+0000","last_active":"2023-06-10T18:57:06.282965+0000","last_peered":"2023-06-10T18:57:06.282965+0000","last_clean":"2023-06-10T18:57:06.282965+0000","last_became_active":"2023-06-10T18:57:05.893372+0000","last_became_peered":"2023-06-10T18:57:05.893372+0000","last_unstale":"2023-06-10T18:57:06.282965+0000","last_undegraded":"2023-06-10T18:57:06.282965+0000","last_fullsized":"2023-06-10T18:57:06.282965+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:51.005484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306630+0000","last_change":"2023-06-10T18:57:06.302151+0000","last_active":"2023-06-10T18:57:06.306630+0000","last_peered":"2023-06-10T18:57:06.306630+0000","last_clean":"2023-06-10T18:57:06.306630+0000","last_became_active":"2023-06-10T18:57:06.301909+0000","last_became_peered":"2023-06-10T18:57:06.301909+0000","last_unstale":"2023-06-10T18:57:06.306630+0000","last_undegraded":"2023-06-10T18:57:06.306630+0000","last_fullsized":"2023-06-10T18:57:06.306630+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:36.745933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277634+0000","last_change":"2023-06-10T18:57:05.894244+0000","last_active":"2023-06-10T18:57:06.277634+0000","last_peered":"2023-06-10T18:57:06.277634+0000","last_clean":"2023-06-10T18:57:06.277634+0000","last_became_active":"2023-06-10T18:57:05.893661+0000","last_became_peered":"2023-06-10T18:57:05.893661+0000","last_unstale":"2023-06-10T18:57:06.277634+0000","last_undegraded":"2023-06-10T18:57:06.277634+0000","last_fullsized":"2023-06-10T18:57:06.277634+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:05.887470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305634+0000","last_change":"2023-06-10T18:57:06.301650+0000","last_active":"2023-06-10T18:57:06.305634+0000","last_peered":"2023-06-10T18:57:06.305634+0000","last_clean":"2023-06-10T18:57:06.305634+0000","last_became_active":"2023-06-10T18:57:06.301374+0000","last_became_peered":"2023-06-10T18:57:06.301374+0000","last_unstale":"2023-06-10T18:57:06.305634+0000","last_undegraded":"2023-06-10T18:57:06.305634+0000","last_fullsized":"2023-06-10T18:57:06.305634+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:07:32.465152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276813+0000","last_change":"2023-06-10T18:57:05.895075+0000","last_active":"2023-06-10T18:57:06.276813+0000","last_peered":"2023-06-10T18:57:06.276813+0000","last_clean":"2023-06-10T18:57:06.276813+0000","last_became_active":"2023-06-10T18:57:05.894845+0000","last_became_peered":"2023-06-10T18:57:05.894845+0000","last_unstale":"2023-06-10T18:57:06.276813+0000","last_undegraded":"2023-06-10T18:57:06.276813+0000","last_fullsized":"2023-06-10T18:57:06.276813+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:37.229097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303640+0000","last_change":"2023-06-10T18:57:06.292114+0000","last_active":"2023-06-10T18:57:06.303640+0000","last_peered":"2023-06-10T18:57:06.303640+0000","last_clean":"2023-06-10T18:57:06.303640+0000","last_became_active":"2023-06-10T18:57:06.291836+0000","last_became_peered":"2023-06-10T18:57:06.291836+0000","last_unstale":"2023-06-10T18:57:06.303640+0000","last_undegraded":"2023-06-10T18:57:06.303640+0000","last_fullsized":"2023-06-10T18:57:06.303640+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:40:22.085538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294288+0000","last_change":"2023-06-10T18:57:05.897620+0000","last_active":"2023-06-10T18:57:06.294288+0000","last_peered":"2023-06-10T18:57:06.294288+0000","last_clean":"2023-06-10T18:57:06.294288+0000","last_became_active":"2023-06-10T18:57:05.897328+0000","last_became_peered":"2023-06-10T18:57:05.897328+0000","last_unstale":"2023-06-10T18:57:06.294288+0000","last_undegraded":"2023-06-10T18:57:06.294288+0000","last_fullsized":"2023-06-10T18:57:06.294288+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:37.122966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298587+0000","last_change":"2023-06-10T18:57:06.288646+0000","last_active":"2023-06-10T18:57:06.298587+0000","last_peered":"2023-06-10T18:57:06.298587+0000","last_clean":"2023-06-10T18:57:06.298587+0000","last_became_active":"2023-06-10T18:57:06.288343+0000","last_became_peered":"2023-06-10T18:57:06.288343+0000","last_unstale":"2023-06-10T18:57:06.298587+0000","last_undegraded":"2023-06-10T18:57:06.298587+0000","last_fullsized":"2023-06-10T18:57:06.298587+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:29.568194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277273+0000","last_change":"2023-06-10T18:57:05.891907+0000","last_active":"2023-06-10T18:57:06.277273+0000","last_peered":"2023-06-10T18:57:06.277273+0000","last_clean":"2023-06-10T18:57:06.277273+0000","last_became_active":"2023-06-10T18:57:05.890965+0000","last_became_peered":"2023-06-10T18:57:05.890965+0000","last_unstale":"2023-06-10T18:57:06.277273+0000","last_undegraded":"2023-06-10T18:57:06.277273+0000","last_fullsized":"2023-06-10T18:57:06.277273+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:06.888272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.289485+0000","last_change":"2023-06-10T18:57:06.287323+0000","last_active":"2023-06-10T18:57:06.289485+0000","last_peered":"2023-06-10T18:57:06.289485+0000","last_clean":"2023-06-10T18:57:06.289485+0000","last_became_active":"2023-06-10T18:57:06.287060+0000","last_became_peered":"2023-06-10T18:57:06.287060+0000","last_unstale":"2023-06-10T18:57:06.289485+0000","last_undegraded":"2023-06-10T18:57:06.289485+0000","last_fullsized":"2023-06-10T18:57:06.289485+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:03:45.081545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294565+0000","last_change":"2023-06-10T18:57:05.893836+0000","last_active":"2023-06-10T18:57:06.294565+0000","last_peered":"2023-06-10T18:57:06.294565+0000","last_clean":"2023-06-10T18:57:06.294565+0000","last_became_active":"2023-06-10T18:57:05.893615+0000","last_became_peered":"2023-06-10T18:57:05.893615+0000","last_unstale":"2023-06-10T18:57:06.294565+0000","last_undegraded":"2023-06-10T18:57:06.294565+0000","last_fullsized":"2023-06-10T18:57:06.294565+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:06.605292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":225,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.283049+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:57:06.283049+0000","last_peered":"2023-06-10T18:57:06.283049+0000","last_clean":"2023-06-10T18:57:06.283049+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:57:06.283049+0000","last_undegraded":"2023-06-10T18:57:06.283049+0000","last_fullsized":"2023-06-10T18:57:06.283049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:15:15.608939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296108+0000","last_change":"2023-06-10T18:57:06.286518+0000","last_active":"2023-06-10T18:57:06.296108+0000","last_peered":"2023-06-10T18:57:06.296108+0000","last_clean":"2023-06-10T18:57:06.296108+0000","last_became_active":"2023-06-10T18:57:06.286269+0000","last_became_peered":"2023-06-10T18:57:06.286269+0000","last_unstale":"2023-06-10T18:57:06.296108+0000","last_undegraded":"2023-06-10T18:57:06.296108+0000","last_fullsized":"2023-06-10T18:57:06.296108+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:43:13.390981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"142'6","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306984+0000","last_change":"2023-06-10T18:57:05.893815+0000","last_active":"2023-06-10T18:57:06.306984+0000","last_peered":"2023-06-10T18:57:06.306984+0000","last_clean":"2023-06-10T18:57:06.306984+0000","last_became_active":"2023-06-10T18:57:05.893377+0000","last_became_peered":"2023-06-10T18:57:05.893377+0000","last_unstale":"2023-06-10T18:57:06.306984+0000","last_undegraded":"2023-06-10T18:57:06.306984+0000","last_fullsized":"2023-06-10T18:57:06.306984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:30.713907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281497+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:57:06.281497+0000","last_peered":"2023-06-10T18:57:06.281497+0000","last_clean":"2023-06-10T18:57:06.281497+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:57:06.281497+0000","last_undegraded":"2023-06-10T18:57:06.281497+0000","last_fullsized":"2023-06-10T18:57:06.281497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:39:38.987578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298616+0000","last_change":"2023-06-10T18:57:06.288758+0000","last_active":"2023-06-10T18:57:06.298616+0000","last_peered":"2023-06-10T18:57:06.298616+0000","last_clean":"2023-06-10T18:57:06.298616+0000","last_became_active":"2023-06-10T18:57:06.288430+0000","last_became_peered":"2023-06-10T18:57:06.288430+0000","last_unstale":"2023-06-10T18:57:06.298616+0000","last_undegraded":"2023-06-10T18:57:06.298616+0000","last_fullsized":"2023-06-10T18:57:06.298616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:06.549584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281772+0000","last_change":"2023-06-10T18:57:05.890694+0000","last_active":"2023-06-10T18:57:06.281772+0000","last_peered":"2023-06-10T18:57:06.281772+0000","last_clean":"2023-06-10T18:57:06.281772+0000","last_became_active":"2023-06-10T18:57:05.890263+0000","last_became_peered":"2023-06-10T18:57:05.890263+0000","last_unstale":"2023-06-10T18:57:06.281772+0000","last_undegraded":"2023-06-10T18:57:06.281772+0000","last_fullsized":"2023-06-10T18:57:06.281772+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:08:13.251117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280703+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:57:06.280703+0000","last_peered":"2023-06-10T18:57:06.280703+0000","last_clean":"2023-06-10T18:57:06.280703+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:57:06.280703+0000","last_undegraded":"2023-06-10T18:57:06.280703+0000","last_fullsized":"2023-06-10T18:57:06.280703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:18:21.520608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297623+0000","last_change":"2023-06-10T18:57:06.294753+0000","last_active":"2023-06-10T18:57:06.297623+0000","last_peered":"2023-06-10T18:57:06.297623+0000","last_clean":"2023-06-10T18:57:06.297623+0000","last_became_active":"2023-06-10T18:57:06.294482+0000","last_became_peered":"2023-06-10T18:57:06.294482+0000","last_unstale":"2023-06-10T18:57:06.297623+0000","last_undegraded":"2023-06-10T18:57:06.297623+0000","last_fullsized":"2023-06-10T18:57:06.297623+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:20.011547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"142'3","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305645+0000","last_change":"2023-06-10T18:57:05.899693+0000","last_active":"2023-06-10T18:57:06.305645+0000","last_peered":"2023-06-10T18:57:06.305645+0000","last_clean":"2023-06-10T18:57:06.305645+0000","last_became_active":"2023-06-10T18:57:05.899421+0000","last_became_peered":"2023-06-10T18:57:05.899421+0000","last_unstale":"2023-06-10T18:57:06.305645+0000","last_undegraded":"2023-06-10T18:57:06.305645+0000","last_fullsized":"2023-06-10T18:57:06.305645+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:02.921513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281961+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:57:06.281961+0000","last_peered":"2023-06-10T18:57:06.281961+0000","last_clean":"2023-06-10T18:57:06.281961+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:57:06.281961+0000","last_undegraded":"2023-06-10T18:57:06.281961+0000","last_fullsized":"2023-06-10T18:57:06.281961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:38:35.466305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":395,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277718+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:57:06.277718+0000","last_peered":"2023-06-10T18:57:06.277718+0000","last_clean":"2023-06-10T18:57:06.277718+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:57:06.277718+0000","last_undegraded":"2023-06-10T18:57:06.277718+0000","last_fullsized":"2023-06-10T18:57:06.277718+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:55:45.228337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303572+0000","last_change":"2023-06-10T18:57:06.295895+0000","last_active":"2023-06-10T18:57:06.303572+0000","last_peered":"2023-06-10T18:57:06.303572+0000","last_clean":"2023-06-10T18:57:06.303572+0000","last_became_active":"2023-06-10T18:57:06.295546+0000","last_became_peered":"2023-06-10T18:57:06.295546+0000","last_unstale":"2023-06-10T18:57:06.303572+0000","last_undegraded":"2023-06-10T18:57:06.303572+0000","last_fullsized":"2023-06-10T18:57:06.303572+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:20.267388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277000+0000","last_change":"2023-06-10T18:57:05.891325+0000","last_active":"2023-06-10T18:57:06.277000+0000","last_peered":"2023-06-10T18:57:06.277000+0000","last_clean":"2023-06-10T18:57:06.277000+0000","last_became_active":"2023-06-10T18:57:05.891019+0000","last_became_peered":"2023-06-10T18:57:05.891019+0000","last_unstale":"2023-06-10T18:57:06.277000+0000","last_undegraded":"2023-06-10T18:57:06.277000+0000","last_fullsized":"2023-06-10T18:57:06.277000+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:45:33.524090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":239,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277576+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:57:06.277576+0000","last_peered":"2023-06-10T18:57:06.277576+0000","last_clean":"2023-06-10T18:57:06.277576+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:57:06.277576+0000","last_undegraded":"2023-06-10T18:57:06.277576+0000","last_fullsized":"2023-06-10T18:57:06.277576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:11.090504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308458+0000","last_change":"2023-06-10T18:57:06.305593+0000","last_active":"2023-06-10T18:57:06.308458+0000","last_peered":"2023-06-10T18:57:06.308458+0000","last_clean":"2023-06-10T18:57:06.308458+0000","last_became_active":"2023-06-10T18:57:06.305302+0000","last_became_peered":"2023-06-10T18:57:06.305302+0000","last_unstale":"2023-06-10T18:57:06.308458+0000","last_undegraded":"2023-06-10T18:57:06.308458+0000","last_fullsized":"2023-06-10T18:57:06.308458+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:09.273874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306828+0000","last_change":"2023-06-10T18:57:05.896711+0000","last_active":"2023-06-10T18:57:06.306828+0000","last_peered":"2023-06-10T18:57:06.306828+0000","last_clean":"2023-06-10T18:57:06.306828+0000","last_became_active":"2023-06-10T18:57:05.896396+0000","last_became_peered":"2023-06-10T18:57:05.896396+0000","last_unstale":"2023-06-10T18:57:06.306828+0000","last_undegraded":"2023-06-10T18:57:06.306828+0000","last_fullsized":"2023-06-10T18:57:06.306828+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:30:33.907001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":239,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.278102+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:57:06.278102+0000","last_peered":"2023-06-10T18:57:06.278102+0000","last_clean":"2023-06-10T18:57:06.278102+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:57:06.278102+0000","last_undegraded":"2023-06-10T18:57:06.278102+0000","last_fullsized":"2023-06-10T18:57:06.278102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:07.012040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300106+0000","last_change":"2023-06-10T18:57:06.298183+0000","last_active":"2023-06-10T18:57:06.300106+0000","last_peered":"2023-06-10T18:57:06.300106+0000","last_clean":"2023-06-10T18:57:06.300106+0000","last_became_active":"2023-06-10T18:57:06.297862+0000","last_became_peered":"2023-06-10T18:57:06.297862+0000","last_unstale":"2023-06-10T18:57:06.300106+0000","last_undegraded":"2023-06-10T18:57:06.300106+0000","last_fullsized":"2023-06-10T18:57:06.300106+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:20:03.469821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"142'8","reported_seq":19,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301620+0000","last_change":"2023-06-10T18:57:05.896132+0000","last_active":"2023-06-10T18:57:06.301620+0000","last_peered":"2023-06-10T18:57:06.301620+0000","last_clean":"2023-06-10T18:57:06.301620+0000","last_became_active":"2023-06-10T18:57:05.895841+0000","last_became_peered":"2023-06-10T18:57:05.895841+0000","last_unstale":"2023-06-10T18:57:06.301620+0000","last_undegraded":"2023-06-10T18:57:06.301620+0000","last_fullsized":"2023-06-10T18:57:06.301620+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:49.066635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":225,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281234+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:57:06.281234+0000","last_peered":"2023-06-10T18:57:06.281234+0000","last_clean":"2023-06-10T18:57:06.281234+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:57:06.281234+0000","last_undegraded":"2023-06-10T18:57:06.281234+0000","last_fullsized":"2023-06-10T18:57:06.281234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:22:18.966433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302006+0000","last_change":"2023-06-10T18:57:06.298349+0000","last_active":"2023-06-10T18:57:06.302006+0000","last_peered":"2023-06-10T18:57:06.302006+0000","last_clean":"2023-06-10T18:57:06.302006+0000","last_became_active":"2023-06-10T18:57:06.298084+0000","last_became_peered":"2023-06-10T18:57:06.298084+0000","last_unstale":"2023-06-10T18:57:06.302006+0000","last_undegraded":"2023-06-10T18:57:06.302006+0000","last_fullsized":"2023-06-10T18:57:06.302006+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:49.072234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.279755+0000","last_change":"2023-06-10T18:57:05.897138+0000","last_active":"2023-06-10T18:57:06.279755+0000","last_peered":"2023-06-10T18:57:06.279755+0000","last_clean":"2023-06-10T18:57:06.279755+0000","last_became_active":"2023-06-10T18:57:05.896865+0000","last_became_peered":"2023-06-10T18:57:05.896865+0000","last_unstale":"2023-06-10T18:57:06.279755+0000","last_undegraded":"2023-06-10T18:57:06.279755+0000","last_fullsized":"2023-06-10T18:57:06.279755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:05.541484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":223,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295875+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:57:06.295875+0000","last_peered":"2023-06-10T18:57:06.295875+0000","last_clean":"2023-06-10T18:57:06.295875+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:57:06.295875+0000","last_undegraded":"2023-06-10T18:57:06.295875+0000","last_fullsized":"2023-06-10T18:57:06.295875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:26:53.901290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2342912,"data_stored":1442507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81999999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.442,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.74299999999999999,"15min":0.79000000000000004},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.72099999999999997,"5min":0.78100000000000003,"15min":0.91700000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.89700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70999999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.69699999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73699999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.748},{"interface":"front","average":{"1min":0.496,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.72599999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.76300000000000001,"15min":0.81100000000000005},"last":0.73599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28668,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343172,"statfs":{"total":96636764160,"available":96607408128,"internally_reserved":0,"allocated":1757184,"data_stored":858068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.51600000000000001},"min":{"1min":0.36099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70799999999999996,"5min":0.77100000000000002,"15min":0.81100000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.78700000000000003},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.51500000000000001},"min":{"1min":0.42599999999999999,"5min":0.316,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.65600000000000003},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.70099999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.61499999999999999},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66900000000000004,"5min":0.872,"15min":6.6689999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.252},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.81100000000000005,"15min":0.82899999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53300000000000003,"15min":0.54600000000000004},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67900000000000005,"5min":0.77100000000000002,"15min":0.77500000000000002},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.498,"15min":0.498},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":0.874,"15min":0.874},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.27600000000000002},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.89600000000000002},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.68300000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.432},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.68400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28668,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343172,"statfs":{"total":96636764160,"available":96607408128,"internally_reserved":0,"allocated":1757184,"data_stored":858068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.33000000000000002},"min":{"1min":0.20499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54000000000000004,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.40000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59699999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61099999999999999,"15min":0.627},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.81999999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.93000000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60299999999999998,"15min":0.626},"min":{"1min":0.51500000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.78900000000000003,"5min":0.83899999999999997,"15min":1.008},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.206,"15min":1.206},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":0.873,"15min":0.88700000000000001},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.442,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":1.224,"15min":1.224},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59299999999999997,"15min":0.61599999999999999},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.58299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2351104,"data_stored":1448436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.46200000000000002,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34200000000000003,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.437,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48299999999999998,"5min":11.439,"15min":11.439},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.48099999999999998,"15min":0.42199999999999999},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.49099999999999999,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":5.766,"15min":5.766},"last":0.375},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40500000000000003,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51400000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.75900000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":2.802,"15min":2.802},"last":0.432}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81599999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80500000000000005,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.48499999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79300000000000004,"5min":2.907,"15min":2.907},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.41299999999999998,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":1.363},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.79200000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.61299999999999999},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.38900000000000001},"max":{"1min":0.82999999999999996,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.498}]}]},{"osd":2,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28672,"kb_used_data":1720,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343168,"statfs":{"total":96636764160,"available":96607404032,"internally_reserved":0,"allocated":1761280,"data_stored":859087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34899999999999998,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41999999999999998,"15min":0.38500000000000001},"min":{"1min":0.27300000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57499999999999996,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52700000000000002,"5min":2.847,"15min":2.847},"last":0.37},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59499999999999997,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.41099999999999998},"min":{"1min":0.32000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56899999999999995,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":5.742,"15min":5.742},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.64300000000000002,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":6.593,"15min":6.593},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.77600000000000002,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.745},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.61499999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.66400000000000003},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.82299999999999995,"5min":6.617,"15min":6.617},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.71899999999999997,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.69499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1748992,"data_stored":852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33000000000000002,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.45000000000000001,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.315},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54400000000000004,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45700000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57199999999999995,"5min":0.68600000000000005,"15min":1.3},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54000000000000004,"5min":0.82999999999999996,"15min":1.109},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.439},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.61299999999999999,"5min":1.5960000000000001,"15min":1.599},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35899999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89700000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":1.613,"15min":1.613},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.60399999999999998,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":1.4410000000000001,"15min":3.802},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.65300000000000002},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.86499999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.84699999999999998,"5min":1.679,"15min":1.679},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.45200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":1.47,"15min":1.47},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":0.91100000000000003,"5min":1.548,"15min":1.8939999999999999},"last":0.55400000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28656,"kb_used_data":1704,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343184,"statfs":{"total":96636764160,"available":96607420416,"internally_reserved":0,"allocated":1744896,"data_stored":851120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.45000000000000001,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.745,"15min":1.119},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.63,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.873,"15min":0.96599999999999997},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.64800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.746,"15min":0.79800000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.38}]},{"osd":5,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.94599999999999995,"15min":1.069},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":0.63800000000000001,"5min":0.755,"15min":0.88300000000000001},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.54300000000000004,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28199999999999997},"max":{"1min":0.60199999999999998,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.70399999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.68400000000000005,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.36699999999999999,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.57399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1748992,"data_stored":852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73799999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53800000000000003,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72499999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.749,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.78500000000000003},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.39700000000000002,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71199999999999997,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.496}]},{"osd":3,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55100000000000005,"15min":0.54200000000000004},"min":{"1min":0.32700000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57999999999999996,"15min":0.56599999999999995},"min":{"1min":0.46100000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42999999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.35499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.70799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.34300000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.309,"15min":0.309},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.879},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.748,"5min":0.77400000000000002,"15min":0.88400000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79800000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:57:10.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:57:10.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.905+0000 7f8d42037700 1 Processor -- start 2023-06-10T18:57:10.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.905+0000 7f8d42037700 1 -- start start 2023-06-10T18:57:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d42037700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d3c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d42037700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d3c0b1080 0x7f8d3c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d42037700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0c9e60 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d42037700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c9bc0 con 0x7f8d3c0b1080 2023-06-10T18:57:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d42037700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0aee70 con 0x7f8d3c0aeaa0 2023-06-10T18:57:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d3bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8d3c0ca2f0 0x7f8d3c0ca750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36062/0 (socket says 172.21.15.104:36062) 2023-06-10T18:57:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3992186166 learned_addr learned my addr 172.21.15.104:0/3992186166 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:10.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d3b7fe700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d3c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398506786 0 0) 0x7f8d3c0c9e60 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.906+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d280035b0 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3affd700 1 --2- 172.21.15.104:0/3992186166 >> v2:172.21.15.114:3300/0 conn(0x7f8d3c0b1080 0x7f8d3c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3583680367 0 0) 0x7f8d280035b0 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 msgr2=0x7f8d3c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d3c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 >> v2:172.21.15.114:3300/0 conn(0x7f8d3c0b1080 msgr2=0x7f8d3c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 --2- 172.21.15.104:0/3992186166 >> v2:172.21.15.114:3300/0 conn(0x7f8d3c0b1080 0x7f8d3c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c06e540 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8d340019f0 con 0x7f8d3c0ca2f0 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3affd700 1 --2- 172.21.15.104:0/3992186166 >> v2:172.21.15.114:3300/0 conn(0x7f8d3c0b1080 0x7f8d3c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 >> v1:172.21.15.104:6789/0 conn(0x7f8d3c0ca2f0 legacy=0x7f8d3c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28003a00 0x7f8d28003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d28004d90 0x7f8d28005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c9bc0 con 0x7f8d28004d90 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0aee70 con 0x7f8d28003a00 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2800ba10 con 0x7f8d3c0aeaa0 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d28004d90 0x7f8d28005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3b7fe700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28003a00 0x7f8d28003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3affd700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 msgr2=0x7f8d28004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0aeaa0 0x7f8d28004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28003a00 msgr2=0x7f8d28003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d28003a00 0x7f8d28003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3992186166 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d280035b0 con 0x7f8d28004d90 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d28004d90 0x7f8d28005200 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400a830 tx=0x7f8d28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90b0da0852ed313 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3400a320 con 0x7f8d28004d90 2023-06-10T18:57:10.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34012600 con 0x7f8d28004d90 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.907+0000 7f8d3a7fc700 1 -- 172.21.15.104:0/3992186166 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340129b0 con 0x7f8d28004d90 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 -- 172.21.15.104:0/3992186166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d28004d90 msgr2=0x7f8d28005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 --2- 172.21.15.104:0/3992186166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d28004d90 0x7f8d28005200 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400a830 tx=0x7f8d28005880 comp rx=0 tx=0).stop 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3992186166 reap_dead start 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 -- 172.21.15.104:0/3992186166 shutdown_connections 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 -- 172.21.15.104:0/3992186166 >> 172.21.15.104:0/3992186166 conn(0x7f8d3c06b8a0 msgr2=0x7f8d3c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:10.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 -- 172.21.15.104:0/3992186166 shutdown_connections 2023-06-10T18:57:10.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.908+0000 7f8d42037700 1 -- 172.21.15.104:0/3992186166 wait complete. 2023-06-10T18:57:10.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.909+0000 7f8d42037700 1 Processor -- start 2023-06-10T18:57:10.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 -- start start 2023-06-10T18:57:10.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d3c0c5820 0x7f8d3c1f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0c5bf0 0x7f8d3c145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c149810 con 0x7f8d3c1461f0 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c8da0 con 0x7f8d3c0c5820 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d42037700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c9080 con 0x7f8d3c0c5bf0 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38246/0 (socket says 172.21.15.104:38246) 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.910+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3579707460 learned_addr learned my addr 172.21.15.104:0/3579707460 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0c5bf0 msgr2=0x7f8d3c145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3affd700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0c5bf0 0x7f8d3c145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0c5bf0 0x7f8d3c145cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3b7fe700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d3c0c5820 0x7f8d3c1f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d3c0c5820 msgr2=0x7f8d3c1f13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d3c0c5820 0x7f8d3c1f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 -- 172.21.15.104:0/3579707460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0c9360 con 0x7f8d3c1461f0 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d3bfff700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f8d34001cb0 tx=0x7f8d3c0cf160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3400a320 con 0x7f8d3c1461f0 2023-06-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d340019c0 con 0x7f8d3c1461f0 2023-06-10T18:57:10.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c1454b0 con 0x7f8d3c1461f0 2023-06-10T18:57:10.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34027410 con 0x7f8d3c1461f0 2023-06-10T18:57:10.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.911+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c1f1920 con 0x7f8d3c1461f0 2023-06-10T18:57:10.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.912+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c074050 con 0x7f8d3c1461f0 2023-06-10T18:57:10.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.912+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d34019070 con 0x7f8d3c1461f0 2023-06-10T18:57:10.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.913+0000 7f8d33fff700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 0x7f8d1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:10.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.913+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d3408a9f0 con 0x7f8d3c1461f0 2023-06-10T18:57:10.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.913+0000 7f8d3b7fe700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 0x7f8d1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:10.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.913+0000 7f8d3b7fe700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 0x7f8d1c06d190 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c1f0a30 tx=0x7f8d24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:10.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:10.917+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d340559f0 con 0x7f8d3c1461f0 2023-06-10T18:57:11.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.100+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d3c06eb00 con 0x7f8d1c06ace0 2023-06-10T18:57:11.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.109+0000 7f8d33fff700 1 -- 172.21.15.104:0/3579707460 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311942 (secure 0 0 0) 0x7f8d24009e80 con 0x7f8d1c06ace0 2023-06-10T18:57:11.118 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:11.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 msgr2=0x7f8d1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 0x7f8d1c06d190 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c1f0a30 tx=0x7f8d24008040 comp rx=0 tx=0).stop 2023-06-10T18:57:11.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 msgr2=0x7f8d3c1578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f8d34001cb0 tx=0x7f8d3c0cf160 comp rx=0 tx=0).stop 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 shutdown_connections 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8d1c06ace0 0x7f8d1c06d190 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d3c0c5820 0x7f8d3c1f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d3c1461f0 0x7f8d3c1578c0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 --2- 172.21.15.104:0/3579707460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d3c0c5bf0 0x7f8d3c145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 >> 172.21.15.104:0/3579707460 conn(0x7f8d3c06b8a0 msgr2=0x7f8d3c069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:11.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 shutdown_connections 2023-06-10T18:57:11.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.113+0000 7f8d42037700 1 -- 172.21.15.104:0/3579707460 wait complete. 2023-06-10T18:57:11.122 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:57:11.131 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2023-06-10T18:57:09.557653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238664,"kb_used_data":14856,"kb_used_omap":0,"kb_used_meta":223744,"kb_avail":754736056,"statfs":{"total":773094113280,"available":772849721344,"internally_reserved":0,"allocated":15212544,"data_stored":8021564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":229113856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.688968"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301606+0000","last_change":"2023-06-10T18:57:06.297714+0000","last_active":"2023-06-10T18:57:06.301606+0000","last_peered":"2023-06-10T18:57:06.301606+0000","last_clean":"2023-06-10T18:57:06.301606+0000","last_became_active":"2023-06-10T18:57:06.297412+0000","last_became_peered":"2023-06-10T18:57:06.297412+0000","last_unstale":"2023-06-10T18:57:06.301606+0000","last_undegraded":"2023-06-10T18:57:06.301606+0000","last_fullsized":"2023-06-10T18:57:06.301606+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:38.813010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.289519+0000","last_change":"2023-06-10T18:57:06.287731+0000","last_active":"2023-06-10T18:57:06.289519+0000","last_peered":"2023-06-10T18:57:06.289519+0000","last_clean":"2023-06-10T18:57:06.289519+0000","last_became_active":"2023-06-10T18:57:06.287484+0000","last_became_peered":"2023-06-10T18:57:06.287484+0000","last_unstale":"2023-06-10T18:57:06.289519+0000","last_undegraded":"2023-06-10T18:57:06.289519+0000","last_fullsized":"2023-06-10T18:57:06.289519+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:31:26.705650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300144+0000","last_change":"2023-06-10T18:57:06.297444+0000","last_active":"2023-06-10T18:57:06.300144+0000","last_peered":"2023-06-10T18:57:06.300144+0000","last_clean":"2023-06-10T18:57:06.300144+0000","last_became_active":"2023-06-10T18:57:06.297012+0000","last_became_peered":"2023-06-10T18:57:06.297012+0000","last_unstale":"2023-06-10T18:57:06.300144+0000","last_undegraded":"2023-06-10T18:57:06.300144+0000","last_fullsized":"2023-06-10T18:57:06.300144+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:51:55.664753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.310776+0000","last_change":"2023-06-10T18:57:06.305473+0000","last_active":"2023-06-10T18:57:06.310776+0000","last_peered":"2023-06-10T18:57:06.310776+0000","last_clean":"2023-06-10T18:57:06.310776+0000","last_became_active":"2023-06-10T18:57:06.305183+0000","last_became_peered":"2023-06-10T18:57:06.305183+0000","last_unstale":"2023-06-10T18:57:06.310776+0000","last_undegraded":"2023-06-10T18:57:06.310776+0000","last_fullsized":"2023-06-10T18:57:06.310776+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:44:25.994305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301956+0000","last_change":"2023-06-10T18:57:06.296998+0000","last_active":"2023-06-10T18:57:06.301956+0000","last_peered":"2023-06-10T18:57:06.301956+0000","last_clean":"2023-06-10T18:57:06.301956+0000","last_became_active":"2023-06-10T18:57:06.296411+0000","last_became_peered":"2023-06-10T18:57:06.296411+0000","last_unstale":"2023-06-10T18:57:06.301956+0000","last_undegraded":"2023-06-10T18:57:06.301956+0000","last_fullsized":"2023-06-10T18:57:06.301956+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:44.912561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301709+0000","last_change":"2023-06-10T18:57:06.300023+0000","last_active":"2023-06-10T18:57:06.301709+0000","last_peered":"2023-06-10T18:57:06.301709+0000","last_clean":"2023-06-10T18:57:06.301709+0000","last_became_active":"2023-06-10T18:57:06.299748+0000","last_became_peered":"2023-06-10T18:57:06.299748+0000","last_unstale":"2023-06-10T18:57:06.301709+0000","last_undegraded":"2023-06-10T18:57:06.301709+0000","last_fullsized":"2023-06-10T18:57:06.301709+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:24.710048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297340+0000","last_change":"2023-06-10T18:57:06.293985+0000","last_active":"2023-06-10T18:57:06.297340+0000","last_peered":"2023-06-10T18:57:06.297340+0000","last_clean":"2023-06-10T18:57:06.297340+0000","last_became_active":"2023-06-10T18:57:06.293691+0000","last_became_peered":"2023-06-10T18:57:06.293691+0000","last_unstale":"2023-06-10T18:57:06.297340+0000","last_undegraded":"2023-06-10T18:57:06.297340+0000","last_fullsized":"2023-06-10T18:57:06.297340+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:06:57.488806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297815+0000","last_change":"2023-06-10T18:57:06.291070+0000","last_active":"2023-06-10T18:57:06.297815+0000","last_peered":"2023-06-10T18:57:06.297815+0000","last_clean":"2023-06-10T18:57:06.297815+0000","last_became_active":"2023-06-10T18:57:06.290761+0000","last_became_peered":"2023-06-10T18:57:06.290761+0000","last_unstale":"2023-06-10T18:57:06.297815+0000","last_undegraded":"2023-06-10T18:57:06.297815+0000","last_fullsized":"2023-06-10T18:57:06.297815+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:51:41.766689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.293185+0000","last_change":"2023-06-10T18:57:06.292298+0000","last_active":"2023-06-10T18:57:06.293185+0000","last_peered":"2023-06-10T18:57:06.293185+0000","last_clean":"2023-06-10T18:57:06.293185+0000","last_became_active":"2023-06-10T18:57:06.292029+0000","last_became_peered":"2023-06-10T18:57:06.292029+0000","last_unstale":"2023-06-10T18:57:06.293185+0000","last_undegraded":"2023-06-10T18:57:06.293185+0000","last_fullsized":"2023-06-10T18:57:06.293185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:02.889694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303551+0000","last_change":"2023-06-10T18:57:06.291645+0000","last_active":"2023-06-10T18:57:06.303551+0000","last_peered":"2023-06-10T18:57:06.303551+0000","last_clean":"2023-06-10T18:57:06.303551+0000","last_became_active":"2023-06-10T18:57:06.291360+0000","last_became_peered":"2023-06-10T18:57:06.291360+0000","last_unstale":"2023-06-10T18:57:06.303551+0000","last_undegraded":"2023-06-10T18:57:06.303551+0000","last_fullsized":"2023-06-10T18:57:06.303551+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:59:33.952373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291326+0000","last_change":"2023-06-10T18:57:06.289639+0000","last_active":"2023-06-10T18:57:06.291326+0000","last_peered":"2023-06-10T18:57:06.291326+0000","last_clean":"2023-06-10T18:57:06.291326+0000","last_became_active":"2023-06-10T18:57:06.289374+0000","last_became_peered":"2023-06-10T18:57:06.289374+0000","last_unstale":"2023-06-10T18:57:06.291326+0000","last_undegraded":"2023-06-10T18:57:06.291326+0000","last_fullsized":"2023-06-10T18:57:06.291326+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:31:52.532414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304492+0000","last_change":"2023-06-10T18:57:06.291060+0000","last_active":"2023-06-10T18:57:06.304492+0000","last_peered":"2023-06-10T18:57:06.304492+0000","last_clean":"2023-06-10T18:57:06.304492+0000","last_became_active":"2023-06-10T18:57:06.290761+0000","last_became_peered":"2023-06-10T18:57:06.290761+0000","last_unstale":"2023-06-10T18:57:06.304492+0000","last_undegraded":"2023-06-10T18:57:06.304492+0000","last_fullsized":"2023-06-10T18:57:06.304492+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:40.224886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301976+0000","last_change":"2023-06-10T18:57:06.297178+0000","last_active":"2023-06-10T18:57:06.301976+0000","last_peered":"2023-06-10T18:57:06.301976+0000","last_clean":"2023-06-10T18:57:06.301976+0000","last_became_active":"2023-06-10T18:57:06.296744+0000","last_became_peered":"2023-06-10T18:57:06.296744+0000","last_unstale":"2023-06-10T18:57:06.301976+0000","last_undegraded":"2023-06-10T18:57:06.301976+0000","last_fullsized":"2023-06-10T18:57:06.301976+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:06:33.493990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308525+0000","last_change":"2023-06-10T18:57:06.301659+0000","last_active":"2023-06-10T18:57:06.308525+0000","last_peered":"2023-06-10T18:57:06.308525+0000","last_clean":"2023-06-10T18:57:06.308525+0000","last_became_active":"2023-06-10T18:57:06.301393+0000","last_became_peered":"2023-06-10T18:57:06.301393+0000","last_unstale":"2023-06-10T18:57:06.308525+0000","last_undegraded":"2023-06-10T18:57:06.308525+0000","last_fullsized":"2023-06-10T18:57:06.308525+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:52.061556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303611+0000","last_change":"2023-06-10T18:57:06.290826+0000","last_active":"2023-06-10T18:57:06.303611+0000","last_peered":"2023-06-10T18:57:06.303611+0000","last_clean":"2023-06-10T18:57:06.303611+0000","last_became_active":"2023-06-10T18:57:06.288114+0000","last_became_peered":"2023-06-10T18:57:06.288114+0000","last_unstale":"2023-06-10T18:57:06.303611+0000","last_undegraded":"2023-06-10T18:57:06.303611+0000","last_fullsized":"2023-06-10T18:57:06.303611+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:33.596698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296258+0000","last_change":"2023-06-10T18:57:06.291980+0000","last_active":"2023-06-10T18:57:06.296258+0000","last_peered":"2023-06-10T18:57:06.296258+0000","last_clean":"2023-06-10T18:57:06.296258+0000","last_became_active":"2023-06-10T18:57:06.291725+0000","last_became_peered":"2023-06-10T18:57:06.291725+0000","last_unstale":"2023-06-10T18:57:06.296258+0000","last_undegraded":"2023-06-10T18:57:06.296258+0000","last_fullsized":"2023-06-10T18:57:06.296258+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:41:54.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303768+0000","last_change":"2023-06-10T18:57:06.295934+0000","last_active":"2023-06-10T18:57:06.303768+0000","last_peered":"2023-06-10T18:57:06.303768+0000","last_clean":"2023-06-10T18:57:06.303768+0000","last_became_active":"2023-06-10T18:57:06.295652+0000","last_became_peered":"2023-06-10T18:57:06.295652+0000","last_unstale":"2023-06-10T18:57:06.303768+0000","last_undegraded":"2023-06-10T18:57:06.303768+0000","last_fullsized":"2023-06-10T18:57:06.303768+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:17:35.395931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300203+0000","last_change":"2023-06-10T18:57:06.295669+0000","last_active":"2023-06-10T18:57:06.300203+0000","last_peered":"2023-06-10T18:57:06.300203+0000","last_clean":"2023-06-10T18:57:06.300203+0000","last_became_active":"2023-06-10T18:57:06.295202+0000","last_became_peered":"2023-06-10T18:57:06.295202+0000","last_unstale":"2023-06-10T18:57:06.300203+0000","last_undegraded":"2023-06-10T18:57:06.300203+0000","last_fullsized":"2023-06-10T18:57:06.300203+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:46:35.567178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291307+0000","last_change":"2023-06-10T18:57:06.288875+0000","last_active":"2023-06-10T18:57:06.291307+0000","last_peered":"2023-06-10T18:57:06.291307+0000","last_clean":"2023-06-10T18:57:06.291307+0000","last_became_active":"2023-06-10T18:57:06.288621+0000","last_became_peered":"2023-06-10T18:57:06.288621+0000","last_unstale":"2023-06-10T18:57:06.291307+0000","last_undegraded":"2023-06-10T18:57:06.291307+0000","last_fullsized":"2023-06-10T18:57:06.291307+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:22:32.631362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.292439+0000","last_change":"2023-06-10T18:57:06.285317+0000","last_active":"2023-06-10T18:57:06.292439+0000","last_peered":"2023-06-10T18:57:06.292439+0000","last_clean":"2023-06-10T18:57:06.292439+0000","last_became_active":"2023-06-10T18:57:06.285025+0000","last_became_peered":"2023-06-10T18:57:06.285025+0000","last_unstale":"2023-06-10T18:57:06.292439+0000","last_undegraded":"2023-06-10T18:57:06.292439+0000","last_fullsized":"2023-06-10T18:57:06.292439+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:31:38.917623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308103+0000","last_change":"2023-06-10T18:57:06.303459+0000","last_active":"2023-06-10T18:57:06.308103+0000","last_peered":"2023-06-10T18:57:06.308103+0000","last_clean":"2023-06-10T18:57:06.308103+0000","last_became_active":"2023-06-10T18:57:06.303154+0000","last_became_peered":"2023-06-10T18:57:06.303154+0000","last_unstale":"2023-06-10T18:57:06.308103+0000","last_undegraded":"2023-06-10T18:57:06.308103+0000","last_fullsized":"2023-06-10T18:57:06.308103+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:32.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301219+0000","last_change":"2023-06-10T18:57:06.294969+0000","last_active":"2023-06-10T18:57:06.301219+0000","last_peered":"2023-06-10T18:57:06.301219+0000","last_clean":"2023-06-10T18:57:06.301219+0000","last_became_active":"2023-06-10T18:57:06.294288+0000","last_became_peered":"2023-06-10T18:57:06.294288+0000","last_unstale":"2023-06-10T18:57:06.301219+0000","last_undegraded":"2023-06-10T18:57:06.301219+0000","last_fullsized":"2023-06-10T18:57:06.301219+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:56:48.995690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308438+0000","last_change":"2023-06-10T18:57:06.301269+0000","last_active":"2023-06-10T18:57:06.308438+0000","last_peered":"2023-06-10T18:57:06.308438+0000","last_clean":"2023-06-10T18:57:06.308438+0000","last_became_active":"2023-06-10T18:57:06.300578+0000","last_became_peered":"2023-06-10T18:57:06.300578+0000","last_unstale":"2023-06-10T18:57:06.308438+0000","last_undegraded":"2023-06-10T18:57:06.308438+0000","last_fullsized":"2023-06-10T18:57:06.308438+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:43:01.323340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303659+0000","last_change":"2023-06-10T18:57:06.295930+0000","last_active":"2023-06-10T18:57:06.303659+0000","last_peered":"2023-06-10T18:57:06.303659+0000","last_clean":"2023-06-10T18:57:06.303659+0000","last_became_active":"2023-06-10T18:57:06.295634+0000","last_became_peered":"2023-06-10T18:57:06.295634+0000","last_unstale":"2023-06-10T18:57:06.303659+0000","last_undegraded":"2023-06-10T18:57:06.303659+0000","last_fullsized":"2023-06-10T18:57:06.303659+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:40:36.880638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296230+0000","last_change":"2023-06-10T18:57:06.291768+0000","last_active":"2023-06-10T18:57:06.296230+0000","last_peered":"2023-06-10T18:57:06.296230+0000","last_clean":"2023-06-10T18:57:06.296230+0000","last_became_active":"2023-06-10T18:57:06.291502+0000","last_became_peered":"2023-06-10T18:57:06.291502+0000","last_unstale":"2023-06-10T18:57:06.296230+0000","last_undegraded":"2023-06-10T18:57:06.296230+0000","last_fullsized":"2023-06-10T18:57:06.296230+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:19:02.260343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291266+0000","last_change":"2023-06-10T18:57:06.288263+0000","last_active":"2023-06-10T18:57:06.291266+0000","last_peered":"2023-06-10T18:57:06.291266+0000","last_clean":"2023-06-10T18:57:06.291266+0000","last_became_active":"2023-06-10T18:57:06.287725+0000","last_became_peered":"2023-06-10T18:57:06.287725+0000","last_unstale":"2023-06-10T18:57:06.291266+0000","last_undegraded":"2023-06-10T18:57:06.291266+0000","last_fullsized":"2023-06-10T18:57:06.291266+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:35:46.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296567+0000","last_change":"2023-06-10T18:57:06.294527+0000","last_active":"2023-06-10T18:57:06.296567+0000","last_peered":"2023-06-10T18:57:06.296567+0000","last_clean":"2023-06-10T18:57:06.296567+0000","last_became_active":"2023-06-10T18:57:06.294265+0000","last_became_peered":"2023-06-10T18:57:06.294265+0000","last_unstale":"2023-06-10T18:57:06.296567+0000","last_undegraded":"2023-06-10T18:57:06.296567+0000","last_fullsized":"2023-06-10T18:57:06.296567+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:02:21.698563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301661+0000","last_change":"2023-06-10T18:57:06.299820+0000","last_active":"2023-06-10T18:57:06.301661+0000","last_peered":"2023-06-10T18:57:06.301661+0000","last_clean":"2023-06-10T18:57:06.301661+0000","last_became_active":"2023-06-10T18:57:06.299562+0000","last_became_peered":"2023-06-10T18:57:06.299562+0000","last_unstale":"2023-06-10T18:57:06.301661+0000","last_undegraded":"2023-06-10T18:57:06.301661+0000","last_fullsized":"2023-06-10T18:57:06.301661+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:44:14.441609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301725+0000","last_change":"2023-06-10T18:57:06.300147+0000","last_active":"2023-06-10T18:57:06.301725+0000","last_peered":"2023-06-10T18:57:06.301725+0000","last_clean":"2023-06-10T18:57:06.301725+0000","last_became_active":"2023-06-10T18:57:06.299910+0000","last_became_peered":"2023-06-10T18:57:06.299910+0000","last_unstale":"2023-06-10T18:57:06.301725+0000","last_undegraded":"2023-06-10T18:57:06.301725+0000","last_fullsized":"2023-06-10T18:57:06.301725+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:47.638826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308396+0000","last_change":"2023-06-10T18:57:06.300152+0000","last_active":"2023-06-10T18:57:06.308396+0000","last_peered":"2023-06-10T18:57:06.308396+0000","last_clean":"2023-06-10T18:57:06.308396+0000","last_became_active":"2023-06-10T18:57:06.299138+0000","last_became_peered":"2023-06-10T18:57:06.299138+0000","last_unstale":"2023-06-10T18:57:06.308396+0000","last_undegraded":"2023-06-10T18:57:06.308396+0000","last_fullsized":"2023-06-10T18:57:06.308396+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:23:32.937223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296130+0000","last_change":"2023-06-10T18:57:06.292989+0000","last_active":"2023-06-10T18:57:06.296130+0000","last_peered":"2023-06-10T18:57:06.296130+0000","last_clean":"2023-06-10T18:57:06.296130+0000","last_became_active":"2023-06-10T18:57:06.292794+0000","last_became_peered":"2023-06-10T18:57:06.292794+0000","last_unstale":"2023-06-10T18:57:06.296130+0000","last_undegraded":"2023-06-10T18:57:06.296130+0000","last_fullsized":"2023-06-10T18:57:06.296130+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:35:54.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296189+0000","last_change":"2023-06-10T18:57:06.289366+0000","last_active":"2023-06-10T18:57:06.296189+0000","last_peered":"2023-06-10T18:57:06.296189+0000","last_clean":"2023-06-10T18:57:06.296189+0000","last_became_active":"2023-06-10T18:57:06.289133+0000","last_became_peered":"2023-06-10T18:57:06.289133+0000","last_unstale":"2023-06-10T18:57:06.296189+0000","last_undegraded":"2023-06-10T18:57:06.296189+0000","last_fullsized":"2023-06-10T18:57:06.296189+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:12:50.245154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303530+0000","last_change":"2023-06-10T18:57:06.291559+0000","last_active":"2023-06-10T18:57:06.303530+0000","last_peered":"2023-06-10T18:57:06.303530+0000","last_clean":"2023-06-10T18:57:06.303530+0000","last_became_active":"2023-06-10T18:57:06.291298+0000","last_became_peered":"2023-06-10T18:57:06.291298+0000","last_unstale":"2023-06-10T18:57:06.303530+0000","last_undegraded":"2023-06-10T18:57:06.303530+0000","last_fullsized":"2023-06-10T18:57:06.303530+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:31:02.419870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"142'4","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303705+0000","last_change":"2023-06-10T18:57:05.890463+0000","last_active":"2023-06-10T18:57:06.303705+0000","last_peered":"2023-06-10T18:57:06.303705+0000","last_clean":"2023-06-10T18:57:06.303705+0000","last_became_active":"2023-06-10T18:57:05.890169+0000","last_became_peered":"2023-06-10T18:57:05.890169+0000","last_unstale":"2023-06-10T18:57:06.303705+0000","last_undegraded":"2023-06-10T18:57:06.303705+0000","last_fullsized":"2023-06-10T18:57:06.303705+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:44:58.842632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288238+0000","last_change":"2023-06-10T18:57:06.287075+0000","last_active":"2023-06-10T18:57:06.288238+0000","last_peered":"2023-06-10T18:57:06.288238+0000","last_clean":"2023-06-10T18:57:06.288238+0000","last_became_active":"2023-06-10T18:57:06.286824+0000","last_became_peered":"2023-06-10T18:57:06.286824+0000","last_unstale":"2023-06-10T18:57:06.288238+0000","last_undegraded":"2023-06-10T18:57:06.288238+0000","last_fullsized":"2023-06-10T18:57:06.288238+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:40:52.437963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294921+0000","last_change":"2023-06-10T18:57:05.891307+0000","last_active":"2023-06-10T18:57:06.294921+0000","last_peered":"2023-06-10T18:57:06.294921+0000","last_clean":"2023-06-10T18:57:06.294921+0000","last_became_active":"2023-06-10T18:57:05.890991+0000","last_became_peered":"2023-06-10T18:57:05.890991+0000","last_unstale":"2023-06-10T18:57:06.294921+0000","last_undegraded":"2023-06-10T18:57:06.294921+0000","last_fullsized":"2023-06-10T18:57:06.294921+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:04:42.293086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298082+0000","last_change":"2023-06-10T18:57:06.295683+0000","last_active":"2023-06-10T18:57:06.298082+0000","last_peered":"2023-06-10T18:57:06.298082+0000","last_clean":"2023-06-10T18:57:06.298082+0000","last_became_active":"2023-06-10T18:57:06.295247+0000","last_became_peered":"2023-06-10T18:57:06.295247+0000","last_unstale":"2023-06-10T18:57:06.298082+0000","last_undegraded":"2023-06-10T18:57:06.298082+0000","last_fullsized":"2023-06-10T18:57:06.298082+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:00:39.968687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304730+0000","last_change":"2023-06-10T18:57:05.897535+0000","last_active":"2023-06-10T18:57:06.304730+0000","last_peered":"2023-06-10T18:57:06.304730+0000","last_clean":"2023-06-10T18:57:06.304730+0000","last_became_active":"2023-06-10T18:57:05.897201+0000","last_became_peered":"2023-06-10T18:57:05.897201+0000","last_unstale":"2023-06-10T18:57:06.304730+0000","last_undegraded":"2023-06-10T18:57:06.304730+0000","last_fullsized":"2023-06-10T18:57:06.304730+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:36:55.303678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303542+0000","last_change":"2023-06-10T18:57:06.291018+0000","last_active":"2023-06-10T18:57:06.303542+0000","last_peered":"2023-06-10T18:57:06.303542+0000","last_clean":"2023-06-10T18:57:06.303542+0000","last_became_active":"2023-06-10T18:57:06.290720+0000","last_became_peered":"2023-06-10T18:57:06.290720+0000","last_unstale":"2023-06-10T18:57:06.303542+0000","last_undegraded":"2023-06-10T18:57:06.303542+0000","last_fullsized":"2023-06-10T18:57:06.303542+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:55.165083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306054+0000","last_change":"2023-06-10T18:57:05.897501+0000","last_active":"2023-06-10T18:57:06.306054+0000","last_peered":"2023-06-10T18:57:06.306054+0000","last_clean":"2023-06-10T18:57:06.306054+0000","last_became_active":"2023-06-10T18:57:05.897209+0000","last_became_peered":"2023-06-10T18:57:05.897209+0000","last_unstale":"2023-06-10T18:57:06.306054+0000","last_undegraded":"2023-06-10T18:57:06.306054+0000","last_fullsized":"2023-06-10T18:57:06.306054+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:10:55.733983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303346+0000","last_change":"2023-06-10T18:57:06.300013+0000","last_active":"2023-06-10T18:57:06.303346+0000","last_peered":"2023-06-10T18:57:06.303346+0000","last_clean":"2023-06-10T18:57:06.303346+0000","last_became_active":"2023-06-10T18:57:06.299765+0000","last_became_peered":"2023-06-10T18:57:06.299765+0000","last_unstale":"2023-06-10T18:57:06.303346+0000","last_undegraded":"2023-06-10T18:57:06.303346+0000","last_fullsized":"2023-06-10T18:57:06.303346+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:29.838822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295005+0000","last_change":"2023-06-10T18:57:05.891709+0000","last_active":"2023-06-10T18:57:06.295005+0000","last_peered":"2023-06-10T18:57:06.295005+0000","last_clean":"2023-06-10T18:57:06.295005+0000","last_became_active":"2023-06-10T18:57:05.891451+0000","last_became_peered":"2023-06-10T18:57:05.891451+0000","last_unstale":"2023-06-10T18:57:06.295005+0000","last_undegraded":"2023-06-10T18:57:06.295005+0000","last_fullsized":"2023-06-10T18:57:06.295005+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:38:19.155656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308499+0000","last_change":"2023-06-10T18:57:06.301633+0000","last_active":"2023-06-10T18:57:06.308499+0000","last_peered":"2023-06-10T18:57:06.308499+0000","last_clean":"2023-06-10T18:57:06.308499+0000","last_became_active":"2023-06-10T18:57:06.301373+0000","last_became_peered":"2023-06-10T18:57:06.301373+0000","last_unstale":"2023-06-10T18:57:06.308499+0000","last_undegraded":"2023-06-10T18:57:06.308499+0000","last_fullsized":"2023-06-10T18:57:06.308499+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:39:31.292187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296280+0000","last_change":"2023-06-10T18:57:05.891907+0000","last_active":"2023-06-10T18:57:06.296280+0000","last_peered":"2023-06-10T18:57:06.296280+0000","last_clean":"2023-06-10T18:57:06.296280+0000","last_became_active":"2023-06-10T18:57:05.891639+0000","last_became_peered":"2023-06-10T18:57:05.891639+0000","last_unstale":"2023-06-10T18:57:06.296280+0000","last_undegraded":"2023-06-10T18:57:06.296280+0000","last_fullsized":"2023-06-10T18:57:06.296280+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:46.630726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288089+0000","last_change":"2023-06-10T18:57:06.285846+0000","last_active":"2023-06-10T18:57:06.288089+0000","last_peered":"2023-06-10T18:57:06.288089+0000","last_clean":"2023-06-10T18:57:06.288089+0000","last_became_active":"2023-06-10T18:57:06.285594+0000","last_became_peered":"2023-06-10T18:57:06.285594+0000","last_unstale":"2023-06-10T18:57:06.288089+0000","last_undegraded":"2023-06-10T18:57:06.288089+0000","last_fullsized":"2023-06-10T18:57:06.288089+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:47.982305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280621+0000","last_change":"2023-06-10T18:57:05.894080+0000","last_active":"2023-06-10T18:57:06.280621+0000","last_peered":"2023-06-10T18:57:06.280621+0000","last_clean":"2023-06-10T18:57:06.280621+0000","last_became_active":"2023-06-10T18:57:05.893807+0000","last_became_peered":"2023-06-10T18:57:05.893807+0000","last_unstale":"2023-06-10T18:57:06.280621+0000","last_undegraded":"2023-06-10T18:57:06.280621+0000","last_fullsized":"2023-06-10T18:57:06.280621+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:37:18.949540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302165+0000","last_change":"2023-06-10T18:57:06.293271+0000","last_active":"2023-06-10T18:57:06.302165+0000","last_peered":"2023-06-10T18:57:06.302165+0000","last_clean":"2023-06-10T18:57:06.302165+0000","last_became_active":"2023-06-10T18:57:06.293012+0000","last_became_peered":"2023-06-10T18:57:06.293012+0000","last_unstale":"2023-06-10T18:57:06.302165+0000","last_undegraded":"2023-06-10T18:57:06.302165+0000","last_fullsized":"2023-06-10T18:57:06.302165+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:44:52.255039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291872+0000","last_change":"2023-06-10T18:57:05.894286+0000","last_active":"2023-06-10T18:57:06.291872+0000","last_peered":"2023-06-10T18:57:06.291872+0000","last_clean":"2023-06-10T18:57:06.291872+0000","last_became_active":"2023-06-10T18:57:05.894010+0000","last_became_peered":"2023-06-10T18:57:05.894010+0000","last_unstale":"2023-06-10T18:57:06.291872+0000","last_undegraded":"2023-06-10T18:57:06.291872+0000","last_fullsized":"2023-06-10T18:57:06.291872+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:16.330457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303570+0000","last_change":"2023-06-10T18:57:06.296034+0000","last_active":"2023-06-10T18:57:06.303570+0000","last_peered":"2023-06-10T18:57:06.303570+0000","last_clean":"2023-06-10T18:57:06.303570+0000","last_became_active":"2023-06-10T18:57:06.295133+0000","last_became_peered":"2023-06-10T18:57:06.295133+0000","last_unstale":"2023-06-10T18:57:06.303570+0000","last_undegraded":"2023-06-10T18:57:06.303570+0000","last_fullsized":"2023-06-10T18:57:06.303570+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:40:40.257206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.288982+0000","last_change":"2023-06-10T18:57:05.894027+0000","last_active":"2023-06-10T18:57:06.288982+0000","last_peered":"2023-06-10T18:57:06.288982+0000","last_clean":"2023-06-10T18:57:06.288982+0000","last_became_active":"2023-06-10T18:57:05.893740+0000","last_became_peered":"2023-06-10T18:57:05.893740+0000","last_unstale":"2023-06-10T18:57:06.288982+0000","last_undegraded":"2023-06-10T18:57:06.288982+0000","last_fullsized":"2023-06-10T18:57:06.288982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:49:39.686208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301825+0000","last_change":"2023-06-10T18:57:06.298565+0000","last_active":"2023-06-10T18:57:06.301825+0000","last_peered":"2023-06-10T18:57:06.301825+0000","last_clean":"2023-06-10T18:57:06.301825+0000","last_became_active":"2023-06-10T18:57:06.298293+0000","last_became_peered":"2023-06-10T18:57:06.298293+0000","last_unstale":"2023-06-10T18:57:06.301825+0000","last_undegraded":"2023-06-10T18:57:06.301825+0000","last_fullsized":"2023-06-10T18:57:06.301825+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:57.045489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.291743+0000","last_change":"2023-06-10T18:57:05.896585+0000","last_active":"2023-06-10T18:57:06.291743+0000","last_peered":"2023-06-10T18:57:06.291743+0000","last_clean":"2023-06-10T18:57:06.291743+0000","last_became_active":"2023-06-10T18:57:05.896296+0000","last_became_peered":"2023-06-10T18:57:05.896296+0000","last_unstale":"2023-06-10T18:57:06.291743+0000","last_undegraded":"2023-06-10T18:57:06.291743+0000","last_fullsized":"2023-06-10T18:57:06.291743+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:34:12.801550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.310881+0000","last_change":"2023-06-10T18:57:06.306547+0000","last_active":"2023-06-10T18:57:06.310881+0000","last_peered":"2023-06-10T18:57:06.310881+0000","last_clean":"2023-06-10T18:57:06.310881+0000","last_became_active":"2023-06-10T18:57:06.306251+0000","last_became_peered":"2023-06-10T18:57:06.306251+0000","last_unstale":"2023-06-10T18:57:06.310881+0000","last_undegraded":"2023-06-10T18:57:06.310881+0000","last_fullsized":"2023-06-10T18:57:06.310881+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:36.407925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"142'1","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295152+0000","last_change":"2023-06-10T18:57:05.891658+0000","last_active":"2023-06-10T18:57:06.295152+0000","last_peered":"2023-06-10T18:57:06.295152+0000","last_clean":"2023-06-10T18:57:06.295152+0000","last_became_active":"2023-06-10T18:57:05.891453+0000","last_became_peered":"2023-06-10T18:57:05.891453+0000","last_unstale":"2023-06-10T18:57:06.295152+0000","last_undegraded":"2023-06-10T18:57:06.295152+0000","last_fullsized":"2023-06-10T18:57:06.295152+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:51:06.693787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303943+0000","last_change":"2023-06-10T18:57:06.301440+0000","last_active":"2023-06-10T18:57:06.303943+0000","last_peered":"2023-06-10T18:57:06.303943+0000","last_clean":"2023-06-10T18:57:06.303943+0000","last_became_active":"2023-06-10T18:57:06.300823+0000","last_became_peered":"2023-06-10T18:57:06.300823+0000","last_unstale":"2023-06-10T18:57:06.303943+0000","last_undegraded":"2023-06-10T18:57:06.303943+0000","last_fullsized":"2023-06-10T18:57:06.303943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:10:44.912561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280989+0000","last_change":"2023-06-10T18:57:05.893739+0000","last_active":"2023-06-10T18:57:06.280989+0000","last_peered":"2023-06-10T18:57:06.280989+0000","last_clean":"2023-06-10T18:57:06.280989+0000","last_became_active":"2023-06-10T18:57:05.893509+0000","last_became_peered":"2023-06-10T18:57:05.893509+0000","last_unstale":"2023-06-10T18:57:06.280989+0000","last_undegraded":"2023-06-10T18:57:06.280989+0000","last_fullsized":"2023-06-10T18:57:06.280989+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:51:37.414047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301242+0000","last_change":"2023-06-10T18:57:06.295139+0000","last_active":"2023-06-10T18:57:06.301242+0000","last_peered":"2023-06-10T18:57:06.301242+0000","last_clean":"2023-06-10T18:57:06.301242+0000","last_became_active":"2023-06-10T18:57:06.294710+0000","last_became_peered":"2023-06-10T18:57:06.294710+0000","last_unstale":"2023-06-10T18:57:06.301242+0000","last_undegraded":"2023-06-10T18:57:06.301242+0000","last_fullsized":"2023-06-10T18:57:06.301242+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:14:40.099913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300572+0000","last_change":"2023-06-10T18:57:05.894270+0000","last_active":"2023-06-10T18:57:06.300572+0000","last_peered":"2023-06-10T18:57:06.300572+0000","last_clean":"2023-06-10T18:57:06.300572+0000","last_became_active":"2023-06-10T18:57:05.894007+0000","last_became_peered":"2023-06-10T18:57:05.894007+0000","last_unstale":"2023-06-10T18:57:06.300572+0000","last_undegraded":"2023-06-10T18:57:06.300572+0000","last_fullsized":"2023-06-10T18:57:06.300572+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:32:24.362671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297462+0000","last_change":"2023-06-10T18:57:06.294563+0000","last_active":"2023-06-10T18:57:06.297462+0000","last_peered":"2023-06-10T18:57:06.297462+0000","last_clean":"2023-06-10T18:57:06.297462+0000","last_became_active":"2023-06-10T18:57:06.294315+0000","last_became_peered":"2023-06-10T18:57:06.294315+0000","last_unstale":"2023-06-10T18:57:06.297462+0000","last_undegraded":"2023-06-10T18:57:06.297462+0000","last_fullsized":"2023-06-10T18:57:06.297462+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:22:57.399990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276667+0000","last_change":"2023-06-10T18:57:05.891287+0000","last_active":"2023-06-10T18:57:06.276667+0000","last_peered":"2023-06-10T18:57:06.276667+0000","last_clean":"2023-06-10T18:57:06.276667+0000","last_became_active":"2023-06-10T18:57:05.890926+0000","last_became_peered":"2023-06-10T18:57:05.890926+0000","last_unstale":"2023-06-10T18:57:06.276667+0000","last_undegraded":"2023-06-10T18:57:06.276667+0000","last_fullsized":"2023-06-10T18:57:06.276667+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:16:51.861690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.304778+0000","last_change":"2023-06-10T18:57:06.296990+0000","last_active":"2023-06-10T18:57:06.304778+0000","last_peered":"2023-06-10T18:57:06.304778+0000","last_clean":"2023-06-10T18:57:06.304778+0000","last_became_active":"2023-06-10T18:57:06.296735+0000","last_became_peered":"2023-06-10T18:57:06.296735+0000","last_unstale":"2023-06-10T18:57:06.304778+0000","last_undegraded":"2023-06-10T18:57:06.304778+0000","last_fullsized":"2023-06-10T18:57:06.304778+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:08:53.770582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276831+0000","last_change":"2023-06-10T18:57:05.882543+0000","last_active":"2023-06-10T18:57:06.276831+0000","last_peered":"2023-06-10T18:57:06.276831+0000","last_clean":"2023-06-10T18:57:06.276831+0000","last_became_active":"2023-06-10T18:57:05.882240+0000","last_became_peered":"2023-06-10T18:57:05.882240+0000","last_unstale":"2023-06-10T18:57:06.276831+0000","last_undegraded":"2023-06-10T18:57:06.276831+0000","last_fullsized":"2023-06-10T18:57:06.276831+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:29:23.642745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297318+0000","last_change":"2023-06-10T18:57:06.293974+0000","last_active":"2023-06-10T18:57:06.297318+0000","last_peered":"2023-06-10T18:57:06.297318+0000","last_clean":"2023-06-10T18:57:06.297318+0000","last_became_active":"2023-06-10T18:57:06.293657+0000","last_became_peered":"2023-06-10T18:57:06.293657+0000","last_unstale":"2023-06-10T18:57:06.297318+0000","last_undegraded":"2023-06-10T18:57:06.297318+0000","last_fullsized":"2023-06-10T18:57:06.297318+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:17:47.371118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.292966+0000","last_change":"2023-06-10T18:57:05.899655+0000","last_active":"2023-06-10T18:57:06.292966+0000","last_peered":"2023-06-10T18:57:06.292966+0000","last_clean":"2023-06-10T18:57:06.292966+0000","last_became_active":"2023-06-10T18:57:05.899375+0000","last_became_peered":"2023-06-10T18:57:05.899375+0000","last_unstale":"2023-06-10T18:57:06.292966+0000","last_undegraded":"2023-06-10T18:57:06.292966+0000","last_fullsized":"2023-06-10T18:57:06.292966+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:33:42.047607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302065+0000","last_change":"2023-06-10T18:57:06.297782+0000","last_active":"2023-06-10T18:57:06.302065+0000","last_peered":"2023-06-10T18:57:06.302065+0000","last_clean":"2023-06-10T18:57:06.302065+0000","last_became_active":"2023-06-10T18:57:06.297528+0000","last_became_peered":"2023-06-10T18:57:06.297528+0000","last_unstale":"2023-06-10T18:57:06.302065+0000","last_undegraded":"2023-06-10T18:57:06.302065+0000","last_fullsized":"2023-06-10T18:57:06.302065+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:05:33.759425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.299006+0000","last_change":"2023-06-10T18:57:05.894777+0000","last_active":"2023-06-10T18:57:06.299006+0000","last_peered":"2023-06-10T18:57:06.299006+0000","last_clean":"2023-06-10T18:57:06.299006+0000","last_became_active":"2023-06-10T18:57:05.894495+0000","last_became_peered":"2023-06-10T18:57:05.894495+0000","last_unstale":"2023-06-10T18:57:06.299006+0000","last_undegraded":"2023-06-10T18:57:06.299006+0000","last_fullsized":"2023-06-10T18:57:06.299006+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:59:16.984055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305112+0000","last_change":"2023-06-10T18:57:06.303500+0000","last_active":"2023-06-10T18:57:06.305112+0000","last_peered":"2023-06-10T18:57:06.305112+0000","last_clean":"2023-06-10T18:57:06.305112+0000","last_became_active":"2023-06-10T18:57:06.302848+0000","last_became_peered":"2023-06-10T18:57:06.302848+0000","last_unstale":"2023-06-10T18:57:06.305112+0000","last_undegraded":"2023-06-10T18:57:06.305112+0000","last_fullsized":"2023-06-10T18:57:06.305112+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:22:44.460864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281122+0000","last_change":"2023-06-10T18:57:05.891356+0000","last_active":"2023-06-10T18:57:06.281122+0000","last_peered":"2023-06-10T18:57:06.281122+0000","last_clean":"2023-06-10T18:57:06.281122+0000","last_became_active":"2023-06-10T18:57:05.891063+0000","last_became_peered":"2023-06-10T18:57:05.891063+0000","last_unstale":"2023-06-10T18:57:06.281122+0000","last_undegraded":"2023-06-10T18:57:06.281122+0000","last_fullsized":"2023-06-10T18:57:06.281122+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:05:49.162586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296977+0000","last_change":"2023-06-10T18:57:06.294500+0000","last_active":"2023-06-10T18:57:06.296977+0000","last_peered":"2023-06-10T18:57:06.296977+0000","last_clean":"2023-06-10T18:57:06.296977+0000","last_became_active":"2023-06-10T18:57:06.294241+0000","last_became_peered":"2023-06-10T18:57:06.294241+0000","last_unstale":"2023-06-10T18:57:06.296977+0000","last_undegraded":"2023-06-10T18:57:06.296977+0000","last_fullsized":"2023-06-10T18:57:06.296977+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:22:06.734988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281031+0000","last_change":"2023-06-10T18:57:05.893984+0000","last_active":"2023-06-10T18:57:06.281031+0000","last_peered":"2023-06-10T18:57:06.281031+0000","last_clean":"2023-06-10T18:57:06.281031+0000","last_became_active":"2023-06-10T18:57:05.893698+0000","last_became_peered":"2023-06-10T18:57:05.893698+0000","last_unstale":"2023-06-10T18:57:06.281031+0000","last_undegraded":"2023-06-10T18:57:06.281031+0000","last_fullsized":"2023-06-10T18:57:06.281031+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:48:10.217395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303961+0000","last_change":"2023-06-10T18:57:06.301590+0000","last_active":"2023-06-10T18:57:06.303961+0000","last_peered":"2023-06-10T18:57:06.303961+0000","last_clean":"2023-06-10T18:57:06.303961+0000","last_became_active":"2023-06-10T18:57:06.301196+0000","last_became_peered":"2023-06-10T18:57:06.301196+0000","last_unstale":"2023-06-10T18:57:06.303961+0000","last_undegraded":"2023-06-10T18:57:06.303961+0000","last_fullsized":"2023-06-10T18:57:06.303961+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:52:42.334283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.282965+0000","last_change":"2023-06-10T18:57:05.893650+0000","last_active":"2023-06-10T18:57:06.282965+0000","last_peered":"2023-06-10T18:57:06.282965+0000","last_clean":"2023-06-10T18:57:06.282965+0000","last_became_active":"2023-06-10T18:57:05.893372+0000","last_became_peered":"2023-06-10T18:57:05.893372+0000","last_unstale":"2023-06-10T18:57:06.282965+0000","last_undegraded":"2023-06-10T18:57:06.282965+0000","last_fullsized":"2023-06-10T18:57:06.282965+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:51.005484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306630+0000","last_change":"2023-06-10T18:57:06.302151+0000","last_active":"2023-06-10T18:57:06.306630+0000","last_peered":"2023-06-10T18:57:06.306630+0000","last_clean":"2023-06-10T18:57:06.306630+0000","last_became_active":"2023-06-10T18:57:06.301909+0000","last_became_peered":"2023-06-10T18:57:06.301909+0000","last_unstale":"2023-06-10T18:57:06.306630+0000","last_undegraded":"2023-06-10T18:57:06.306630+0000","last_fullsized":"2023-06-10T18:57:06.306630+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:56:36.745933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277634+0000","last_change":"2023-06-10T18:57:05.894244+0000","last_active":"2023-06-10T18:57:06.277634+0000","last_peered":"2023-06-10T18:57:06.277634+0000","last_clean":"2023-06-10T18:57:06.277634+0000","last_became_active":"2023-06-10T18:57:05.893661+0000","last_became_peered":"2023-06-10T18:57:05.893661+0000","last_unstale":"2023-06-10T18:57:06.277634+0000","last_undegraded":"2023-06-10T18:57:06.277634+0000","last_fullsized":"2023-06-10T18:57:06.277634+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:56:05.887470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305634+0000","last_change":"2023-06-10T18:57:06.301650+0000","last_active":"2023-06-10T18:57:06.305634+0000","last_peered":"2023-06-10T18:57:06.305634+0000","last_clean":"2023-06-10T18:57:06.305634+0000","last_became_active":"2023-06-10T18:57:06.301374+0000","last_became_peered":"2023-06-10T18:57:06.301374+0000","last_unstale":"2023-06-10T18:57:06.305634+0000","last_undegraded":"2023-06-10T18:57:06.305634+0000","last_fullsized":"2023-06-10T18:57:06.305634+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:07:32.465152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.276813+0000","last_change":"2023-06-10T18:57:05.895075+0000","last_active":"2023-06-10T18:57:06.276813+0000","last_peered":"2023-06-10T18:57:06.276813+0000","last_clean":"2023-06-10T18:57:06.276813+0000","last_became_active":"2023-06-10T18:57:05.894845+0000","last_became_peered":"2023-06-10T18:57:05.894845+0000","last_unstale":"2023-06-10T18:57:06.276813+0000","last_undegraded":"2023-06-10T18:57:06.276813+0000","last_fullsized":"2023-06-10T18:57:06.276813+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:37.229097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303640+0000","last_change":"2023-06-10T18:57:06.292114+0000","last_active":"2023-06-10T18:57:06.303640+0000","last_peered":"2023-06-10T18:57:06.303640+0000","last_clean":"2023-06-10T18:57:06.303640+0000","last_became_active":"2023-06-10T18:57:06.291836+0000","last_became_peered":"2023-06-10T18:57:06.291836+0000","last_unstale":"2023-06-10T18:57:06.303640+0000","last_undegraded":"2023-06-10T18:57:06.303640+0000","last_fullsized":"2023-06-10T18:57:06.303640+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:40:22.085538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294288+0000","last_change":"2023-06-10T18:57:05.897620+0000","last_active":"2023-06-10T18:57:06.294288+0000","last_peered":"2023-06-10T18:57:06.294288+0000","last_clean":"2023-06-10T18:57:06.294288+0000","last_became_active":"2023-06-10T18:57:05.897328+0000","last_became_peered":"2023-06-10T18:57:05.897328+0000","last_unstale":"2023-06-10T18:57:06.294288+0000","last_undegraded":"2023-06-10T18:57:06.294288+0000","last_fullsized":"2023-06-10T18:57:06.294288+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:37.122966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298587+0000","last_change":"2023-06-10T18:57:06.288646+0000","last_active":"2023-06-10T18:57:06.298587+0000","last_peered":"2023-06-10T18:57:06.298587+0000","last_clean":"2023-06-10T18:57:06.298587+0000","last_became_active":"2023-06-10T18:57:06.288343+0000","last_became_peered":"2023-06-10T18:57:06.288343+0000","last_unstale":"2023-06-10T18:57:06.298587+0000","last_undegraded":"2023-06-10T18:57:06.298587+0000","last_fullsized":"2023-06-10T18:57:06.298587+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:29.568194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277273+0000","last_change":"2023-06-10T18:57:05.891907+0000","last_active":"2023-06-10T18:57:06.277273+0000","last_peered":"2023-06-10T18:57:06.277273+0000","last_clean":"2023-06-10T18:57:06.277273+0000","last_became_active":"2023-06-10T18:57:05.890965+0000","last_became_peered":"2023-06-10T18:57:05.890965+0000","last_unstale":"2023-06-10T18:57:06.277273+0000","last_undegraded":"2023-06-10T18:57:06.277273+0000","last_fullsized":"2023-06-10T18:57:06.277273+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:06.888272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.289485+0000","last_change":"2023-06-10T18:57:06.287323+0000","last_active":"2023-06-10T18:57:06.289485+0000","last_peered":"2023-06-10T18:57:06.289485+0000","last_clean":"2023-06-10T18:57:06.289485+0000","last_became_active":"2023-06-10T18:57:06.287060+0000","last_became_peered":"2023-06-10T18:57:06.287060+0000","last_unstale":"2023-06-10T18:57:06.289485+0000","last_undegraded":"2023-06-10T18:57:06.289485+0000","last_fullsized":"2023-06-10T18:57:06.289485+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:03:45.081545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.294565+0000","last_change":"2023-06-10T18:57:05.893836+0000","last_active":"2023-06-10T18:57:06.294565+0000","last_peered":"2023-06-10T18:57:06.294565+0000","last_clean":"2023-06-10T18:57:06.294565+0000","last_became_active":"2023-06-10T18:57:05.893615+0000","last_became_peered":"2023-06-10T18:57:05.893615+0000","last_unstale":"2023-06-10T18:57:06.294565+0000","last_undegraded":"2023-06-10T18:57:06.294565+0000","last_fullsized":"2023-06-10T18:57:06.294565+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:06.605292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":225,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.283049+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:57:06.283049+0000","last_peered":"2023-06-10T18:57:06.283049+0000","last_clean":"2023-06-10T18:57:06.283049+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:57:06.283049+0000","last_undegraded":"2023-06-10T18:57:06.283049+0000","last_fullsized":"2023-06-10T18:57:06.283049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:15:15.608939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.296108+0000","last_change":"2023-06-10T18:57:06.286518+0000","last_active":"2023-06-10T18:57:06.296108+0000","last_peered":"2023-06-10T18:57:06.296108+0000","last_clean":"2023-06-10T18:57:06.296108+0000","last_became_active":"2023-06-10T18:57:06.286269+0000","last_became_peered":"2023-06-10T18:57:06.286269+0000","last_unstale":"2023-06-10T18:57:06.296108+0000","last_undegraded":"2023-06-10T18:57:06.296108+0000","last_fullsized":"2023-06-10T18:57:06.296108+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:43:13.390981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"142'6","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306984+0000","last_change":"2023-06-10T18:57:05.893815+0000","last_active":"2023-06-10T18:57:06.306984+0000","last_peered":"2023-06-10T18:57:06.306984+0000","last_clean":"2023-06-10T18:57:06.306984+0000","last_became_active":"2023-06-10T18:57:05.893377+0000","last_became_peered":"2023-06-10T18:57:05.893377+0000","last_unstale":"2023-06-10T18:57:06.306984+0000","last_undegraded":"2023-06-10T18:57:06.306984+0000","last_fullsized":"2023-06-10T18:57:06.306984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:24:30.713907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":255,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281497+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:57:06.281497+0000","last_peered":"2023-06-10T18:57:06.281497+0000","last_clean":"2023-06-10T18:57:06.281497+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:57:06.281497+0000","last_undegraded":"2023-06-10T18:57:06.281497+0000","last_fullsized":"2023-06-10T18:57:06.281497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:39:38.987578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.298616+0000","last_change":"2023-06-10T18:57:06.288758+0000","last_active":"2023-06-10T18:57:06.298616+0000","last_peered":"2023-06-10T18:57:06.298616+0000","last_clean":"2023-06-10T18:57:06.298616+0000","last_became_active":"2023-06-10T18:57:06.288430+0000","last_became_peered":"2023-06-10T18:57:06.288430+0000","last_unstale":"2023-06-10T18:57:06.298616+0000","last_undegraded":"2023-06-10T18:57:06.298616+0000","last_fullsized":"2023-06-10T18:57:06.298616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:06.549584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281772+0000","last_change":"2023-06-10T18:57:05.890694+0000","last_active":"2023-06-10T18:57:06.281772+0000","last_peered":"2023-06-10T18:57:06.281772+0000","last_clean":"2023-06-10T18:57:06.281772+0000","last_became_active":"2023-06-10T18:57:05.890263+0000","last_became_peered":"2023-06-10T18:57:05.890263+0000","last_unstale":"2023-06-10T18:57:06.281772+0000","last_undegraded":"2023-06-10T18:57:06.281772+0000","last_fullsized":"2023-06-10T18:57:06.281772+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:08:13.251117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":227,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.280703+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:57:06.280703+0000","last_peered":"2023-06-10T18:57:06.280703+0000","last_clean":"2023-06-10T18:57:06.280703+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:57:06.280703+0000","last_undegraded":"2023-06-10T18:57:06.280703+0000","last_fullsized":"2023-06-10T18:57:06.280703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:18:21.520608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.297623+0000","last_change":"2023-06-10T18:57:06.294753+0000","last_active":"2023-06-10T18:57:06.297623+0000","last_peered":"2023-06-10T18:57:06.297623+0000","last_clean":"2023-06-10T18:57:06.297623+0000","last_became_active":"2023-06-10T18:57:06.294482+0000","last_became_peered":"2023-06-10T18:57:06.294482+0000","last_unstale":"2023-06-10T18:57:06.297623+0000","last_undegraded":"2023-06-10T18:57:06.297623+0000","last_fullsized":"2023-06-10T18:57:06.297623+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:20.011547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"142'3","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.305645+0000","last_change":"2023-06-10T18:57:05.899693+0000","last_active":"2023-06-10T18:57:06.305645+0000","last_peered":"2023-06-10T18:57:06.305645+0000","last_clean":"2023-06-10T18:57:06.305645+0000","last_became_active":"2023-06-10T18:57:05.899421+0000","last_became_peered":"2023-06-10T18:57:05.899421+0000","last_unstale":"2023-06-10T18:57:06.305645+0000","last_undegraded":"2023-06-10T18:57:06.305645+0000","last_fullsized":"2023-06-10T18:57:06.305645+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:02.921513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281961+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:57:06.281961+0000","last_peered":"2023-06-10T18:57:06.281961+0000","last_clean":"2023-06-10T18:57:06.281961+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:57:06.281961+0000","last_undegraded":"2023-06-10T18:57:06.281961+0000","last_fullsized":"2023-06-10T18:57:06.281961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:38:35.466305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":395,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277718+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:57:06.277718+0000","last_peered":"2023-06-10T18:57:06.277718+0000","last_clean":"2023-06-10T18:57:06.277718+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:57:06.277718+0000","last_undegraded":"2023-06-10T18:57:06.277718+0000","last_fullsized":"2023-06-10T18:57:06.277718+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:55:45.228337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.303572+0000","last_change":"2023-06-10T18:57:06.295895+0000","last_active":"2023-06-10T18:57:06.303572+0000","last_peered":"2023-06-10T18:57:06.303572+0000","last_clean":"2023-06-10T18:57:06.303572+0000","last_became_active":"2023-06-10T18:57:06.295546+0000","last_became_peered":"2023-06-10T18:57:06.295546+0000","last_unstale":"2023-06-10T18:57:06.303572+0000","last_undegraded":"2023-06-10T18:57:06.303572+0000","last_fullsized":"2023-06-10T18:57:06.303572+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:22:20.267388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277000+0000","last_change":"2023-06-10T18:57:05.891325+0000","last_active":"2023-06-10T18:57:06.277000+0000","last_peered":"2023-06-10T18:57:06.277000+0000","last_clean":"2023-06-10T18:57:06.277000+0000","last_became_active":"2023-06-10T18:57:05.891019+0000","last_became_peered":"2023-06-10T18:57:05.891019+0000","last_unstale":"2023-06-10T18:57:06.277000+0000","last_undegraded":"2023-06-10T18:57:06.277000+0000","last_fullsized":"2023-06-10T18:57:06.277000+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:45:33.524090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":239,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.277576+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:57:06.277576+0000","last_peered":"2023-06-10T18:57:06.277576+0000","last_clean":"2023-06-10T18:57:06.277576+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:57:06.277576+0000","last_undegraded":"2023-06-10T18:57:06.277576+0000","last_fullsized":"2023-06-10T18:57:06.277576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:11.090504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.308458+0000","last_change":"2023-06-10T18:57:06.305593+0000","last_active":"2023-06-10T18:57:06.308458+0000","last_peered":"2023-06-10T18:57:06.308458+0000","last_clean":"2023-06-10T18:57:06.308458+0000","last_became_active":"2023-06-10T18:57:06.305302+0000","last_became_peered":"2023-06-10T18:57:06.305302+0000","last_unstale":"2023-06-10T18:57:06.308458+0000","last_undegraded":"2023-06-10T18:57:06.308458+0000","last_fullsized":"2023-06-10T18:57:06.308458+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:09.273874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"142'2","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.306828+0000","last_change":"2023-06-10T18:57:05.896711+0000","last_active":"2023-06-10T18:57:06.306828+0000","last_peered":"2023-06-10T18:57:06.306828+0000","last_clean":"2023-06-10T18:57:06.306828+0000","last_became_active":"2023-06-10T18:57:05.896396+0000","last_became_peered":"2023-06-10T18:57:05.896396+0000","last_unstale":"2023-06-10T18:57:06.306828+0000","last_undegraded":"2023-06-10T18:57:06.306828+0000","last_fullsized":"2023-06-10T18:57:06.306828+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:30:33.907001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":239,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.278102+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:57:06.278102+0000","last_peered":"2023-06-10T18:57:06.278102+0000","last_clean":"2023-06-10T18:57:06.278102+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:57:06.278102+0000","last_undegraded":"2023-06-10T18:57:06.278102+0000","last_fullsized":"2023-06-10T18:57:06.278102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:07.012040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.300106+0000","last_change":"2023-06-10T18:57:06.298183+0000","last_active":"2023-06-10T18:57:06.300106+0000","last_peered":"2023-06-10T18:57:06.300106+0000","last_clean":"2023-06-10T18:57:06.300106+0000","last_became_active":"2023-06-10T18:57:06.297862+0000","last_became_peered":"2023-06-10T18:57:06.297862+0000","last_unstale":"2023-06-10T18:57:06.300106+0000","last_undegraded":"2023-06-10T18:57:06.300106+0000","last_fullsized":"2023-06-10T18:57:06.300106+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:20:03.469821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"142'8","reported_seq":19,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.301620+0000","last_change":"2023-06-10T18:57:05.896132+0000","last_active":"2023-06-10T18:57:06.301620+0000","last_peered":"2023-06-10T18:57:06.301620+0000","last_clean":"2023-06-10T18:57:06.301620+0000","last_became_active":"2023-06-10T18:57:05.895841+0000","last_became_peered":"2023-06-10T18:57:05.895841+0000","last_unstale":"2023-06-10T18:57:06.301620+0000","last_undegraded":"2023-06-10T18:57:06.301620+0000","last_fullsized":"2023-06-10T18:57:06.301620+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:56:49.066635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":225,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.281234+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:57:06.281234+0000","last_peered":"2023-06-10T18:57:06.281234+0000","last_clean":"2023-06-10T18:57:06.281234+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:57:06.281234+0000","last_undegraded":"2023-06-10T18:57:06.281234+0000","last_fullsized":"2023-06-10T18:57:06.281234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:22:18.966433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.302006+0000","last_change":"2023-06-10T18:57:06.298349+0000","last_active":"2023-06-10T18:57:06.302006+0000","last_peered":"2023-06-10T18:57:06.302006+0000","last_clean":"2023-06-10T18:57:06.302006+0000","last_became_active":"2023-06-10T18:57:06.298084+0000","last_became_peered":"2023-06-10T18:57:06.298084+0000","last_unstale":"2023-06-10T18:57:06.302006+0000","last_undegraded":"2023-06-10T18:57:06.302006+0000","last_fullsized":"2023-06-10T18:57:06.302006+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:05.866731+0000","last_clean_scrub_stamp":"2023-06-10T18:57:05.866731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:39:49.072234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.279755+0000","last_change":"2023-06-10T18:57:05.897138+0000","last_active":"2023-06-10T18:57:06.279755+0000","last_peered":"2023-06-10T18:57:06.279755+0000","last_clean":"2023-06-10T18:57:06.279755+0000","last_became_active":"2023-06-10T18:57:05.896865+0000","last_became_peered":"2023-06-10T18:57:05.896865+0000","last_unstale":"2023-06-10T18:57:06.279755+0000","last_undegraded":"2023-06-10T18:57:06.279755+0000","last_fullsized":"2023-06-10T18:57:06.279755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:04.858631+0000","last_clean_scrub_stamp":"2023-06-10T18:57:04.858631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:05.541484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":223,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-06-10T18:57:06.295875+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:57:06.295875+0000","last_peered":"2023-06-10T18:57:06.295875+0000","last_clean":"2023-06-10T18:57:06.295875+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:57:06.295875+0000","last_undegraded":"2023-06-10T18:57:06.295875+0000","last_fullsized":"2023-06-10T18:57:06.295875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:26:53.901290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2342912,"data_stored":1442507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.52500000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.39700000000000002,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73699999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47999999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.80600000000000005,"15min":0.81200000000000006},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.79100000000000004,"15min":0.81999999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.442,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.74299999999999999,"15min":0.79000000000000004},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":0.72099999999999997,"5min":0.78100000000000003,"15min":0.91700000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.89700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70999999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.69699999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73699999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.748},{"interface":"front","average":{"1min":0.496,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.753,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.72599999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.85199999999999998,"15min":0.93899999999999995},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.76300000000000001,"15min":0.81100000000000005},"last":0.73599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28668,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343172,"statfs":{"total":96636764160,"available":96607408128,"internally_reserved":0,"allocated":1757184,"data_stored":858068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.51600000000000001},"min":{"1min":0.36099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70799999999999996,"5min":0.77100000000000002,"15min":0.81100000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.78700000000000003},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.51500000000000001},"min":{"1min":0.42599999999999999,"5min":0.316,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.65600000000000003},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.70099999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.61499999999999999},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66900000000000004,"5min":0.872,"15min":6.6689999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69099999999999995,"5min":0.76900000000000002,"15min":1.252},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.54600000000000004},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.66600000000000004,"5min":0.81100000000000005,"15min":0.82899999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53300000000000003,"15min":0.54600000000000004},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67900000000000005,"5min":0.77100000000000002,"15min":0.77500000000000002},"last":0.72599999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.498,"15min":0.498},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":0.874,"15min":0.874},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.27600000000000002},"max":{"1min":0.69899999999999995,"5min":0.746,"15min":0.89600000000000002},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.28799999999999998},"max":{"1min":0.68300000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.621},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.71099999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:56:15 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.432},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.68400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28668,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343172,"statfs":{"total":96636764160,"available":96607408128,"internally_reserved":0,"allocated":1757184,"data_stored":858068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.33000000000000002},"min":{"1min":0.20499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.54000000000000004,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.40000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.39200000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59699999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80800000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.61099999999999999,"15min":0.627},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.34899999999999998},"max":{"1min":0.81999999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":0.84599999999999997,"15min":0.93000000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60299999999999998,"15min":0.626},"min":{"1min":0.51500000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.78900000000000003,"5min":0.83899999999999997,"15min":1.008},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.206,"15min":1.206},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":0.873,"15min":0.88700000000000001},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.442,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":1.224,"15min":1.224},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59299999999999997,"15min":0.61599999999999999},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.58299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2351104,"data_stored":1448436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.46200000000000002,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34200000000000003,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.49299999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.437,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.48299999999999998,"5min":11.439,"15min":11.439},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.48099999999999998,"15min":0.42199999999999999},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.49099999999999999,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.39100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25900000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47999999999999998,"5min":5.766,"15min":5.766},"last":0.375},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40500000000000003,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.51400000000000001,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.58699999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34200000000000003},"max":{"1min":0.75900000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":2.802,"15min":2.802},"last":0.432}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81599999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80500000000000005,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.48499999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79300000000000004,"5min":2.907,"15min":2.907},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.41299999999999998,"5min":0.377,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97699999999999998,"15min":1.363},"last":0.44500000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.79200000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.61299999999999999},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.38900000000000001},"max":{"1min":0.82999999999999996,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.498}]}]},{"osd":2,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28672,"kb_used_data":1720,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343168,"statfs":{"total":96636764160,"available":96607404032,"internally_reserved":0,"allocated":1761280,"data_stored":859087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34899999999999998,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41999999999999998,"15min":0.38500000000000001},"min":{"1min":0.27300000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57499999999999996,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38600000000000001,"15min":0.373},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52700000000000002,"5min":2.847,"15min":2.847},"last":0.37},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59499999999999997,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.41099999999999998},"min":{"1min":0.32000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56899999999999995,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.46500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":5.742,"15min":5.742},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.64300000000000002,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":6.593,"15min":6.593},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66700000000000004,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":0.77600000000000002,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.745},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.79700000000000004,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.61499999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70999999999999996,"15min":0.66400000000000003},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.82299999999999995,"5min":6.617,"15min":6.617},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.71899999999999997,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.37,"15min":0.37},"max":{"1min":0.82599999999999996,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.69499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1748992,"data_stored":852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33000000000000002,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.45000000000000001,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.315},{"interface":"front","average":{"1min":0.377,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54400000000000004,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.40899999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45700000000000002,"5min":0.60599999999999998,"15min":0.99299999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57199999999999995,"5min":0.68600000000000005,"15min":1.3},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54000000000000004,"5min":0.82999999999999996,"15min":1.109},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.439},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.61299999999999999,"5min":1.5960000000000001,"15min":1.599},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.91100000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35899999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89700000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85099999999999998,"5min":1.613,"15min":1.613},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.60399999999999998,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86099999999999999,"5min":1.4410000000000001,"15min":3.802},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.65300000000000002},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.376},"max":{"1min":0.86499999999999999,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.84699999999999998,"5min":1.679,"15min":1.679},"last":0.56999999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:56:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61599999999999999,"15min":0.627},"min":{"1min":0.45200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":1.47,"15min":1.47},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":0.91100000000000003,"5min":1.548,"15min":1.8939999999999999},"last":0.55400000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28656,"kb_used_data":1704,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343184,"statfs":{"total":96636764160,"available":96607420416,"internally_reserved":0,"allocated":1744896,"data_stored":851120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.45000000000000001,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.66800000000000004,"5min":0.745,"15min":1.119},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.63,"5min":0.75800000000000001,"15min":0.77800000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.625,"5min":0.873,"15min":0.96599999999999997},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.64800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.746,"15min":0.79800000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.38}]},{"osd":5,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.94599999999999995,"15min":1.069},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.50600000000000001},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":0.63800000000000001,"5min":0.755,"15min":0.88300000000000001},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.54300000000000004,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28199999999999997},"max":{"1min":0.60199999999999998,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.32100000000000001},"max":{"1min":0.70399999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:56:21 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.68400000000000005,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.36699999999999999,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.57399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575011,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1748992,"data_stored":852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73799999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53800000000000003,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72499999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.749,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.78500000000000003},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.39700000000000002,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53400000000000003,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71199999999999997,"5min":0.75700000000000001,"15min":0.79700000000000004},"last":0.496}]},{"osd":3,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55100000000000005,"15min":0.54200000000000004},"min":{"1min":0.32700000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57999999999999996,"15min":0.56599999999999995},"min":{"1min":0.46100000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42999999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.504,"15min":0.48699999999999999},"min":{"1min":0.35499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.70799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.34300000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.309,"15min":0.309},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.79200000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.879},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:56:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.748,"5min":0.77400000000000002,"15min":0.88400000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79800000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:57:11.138 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:57:11.139 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:57:11.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.291+0000 7fc12c1a4700 1 Processor -- start 2023-06-10T18:57:11.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 -- start start 2023-06-10T18:57:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc12415c500 0x7fc1240cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12406e3e0 0x7fc12406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1240ce040 con 0x7fc124069e70 2023-06-10T18:57:11.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12415c8d0 con 0x7fc12415c500 2023-06-10T18:57:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.292+0000 7fc12c1a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12406da40 con 0x7fc12406e3e0 2023-06-10T18:57:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc12415c500 0x7fc1240cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc12415c500 0x7fc1240cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38250/0 (socket says 172.21.15.104:38250) 2023-06-10T18:57:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 -- 172.21.15.104:0/732491453 learned_addr learned my addr 172.21.15.104:0/732491453 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:11.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc12a741700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12406e3e0 0x7fc12406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 -- 172.21.15.104:0/732491453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12406e3e0 msgr2=0x7fc12406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12406e3e0 0x7fc12406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 -- 172.21.15.104:0/732491453 >> v1:172.21.15.114:6789/0 conn(0x7fc124069e70 legacy=0x7fc12406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 -- 172.21.15.104:0/732491453 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12406a9c0 con 0x7fc12415c500 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc129f40700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc12415c500 0x7fc1240cedc0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669bf534f790b1d1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc128f3e700 1 -- 172.21.15.104:0/732491453 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114005e30 con 0x7fc12415c500 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc128f3e700 1 -- 172.21.15.104:0/732491453 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114004030 con 0x7fc12415c500 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.293+0000 7fc128f3e700 1 -- 172.21.15.104:0/732491453 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1140043e0 con 0x7fc12415c500 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 -- 172.21.15.104:0/732491453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc12415c500 msgr2=0x7fc1240cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc12415c500 0x7fc1240cedc0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 -- 172.21.15.104:0/732491453 shutdown_connections 2023-06-10T18:57:11.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc12415c500 0x7fc1240cedc0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/732491453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12406e3e0 0x7fc12406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 -- 172.21.15.104:0/732491453 >> 172.21.15.104:0/732491453 conn(0x7fc12406b8a0 msgr2=0x7fc1240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 -- 172.21.15.104:0/732491453 shutdown_connections 2023-06-10T18:57:11.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.294+0000 7fc12c1a4700 1 -- 172.21.15.104:0/732491453 wait complete. 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.295+0000 7fc12c1a4700 1 Processor -- start 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.295+0000 7fc12c1a4700 1 -- start start 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12406e3e0 0x7fc1240c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 0x7fc124155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1240c3b50 con 0x7fc124069e70 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1240c43f0 con 0x7fc12406e3e0 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1240c57c0 con 0x7fc12415c500 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38256/0 (socket says 172.21.15.104:38256) 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 -- 172.21.15.104:0/3802546138 learned_addr learned my addr 172.21.15.104:0/3802546138 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12a741700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 0x7fc124155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12973f700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12406e3e0 0x7fc1240c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 -- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 msgr2=0x7fc124155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 0x7fc124155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 -- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12406e3e0 msgr2=0x7fc1240c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12406e3e0 0x7fc1240c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 -- 172.21.15.104:0/3802546138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114009500 con 0x7fc124069e70 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12a741700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 0x7fc124155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc129f40700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fc114009c10 tx=0x7fc114004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114003dd0 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11401c420 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1240c3da0 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11401c7b0 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.296+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc124155f30 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.297+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc124066f10 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.298+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc11401c910 con 0x7fc124069e70 2023-06-10T18:57:11.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.298+0000 7fc11a7fc700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 0x7fc10806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.298+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc11408a670 con 0x7fc124069e70 2023-06-10T18:57:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.298+0000 7fc12973f700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 0x7fc10806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.299+0000 7fc12973f700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 0x7fc10806d190 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc1241f4fe0 tx=0x7fc120009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.303+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc114055670 con 0x7fc124069e70 2023-06-10T18:57:11.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.568+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc1240c5aa0 con 0x7fc124069e70 2023-06-10T18:57:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.572+0000 7fc11a7fc700 1 -- 172.21.15.104:0/3802546138 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7fc114058c90 con 0x7fc124069e70 2023-06-10T18:57:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 msgr2=0x7fc10806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 0x7fc10806d190 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc1241f4fe0 tx=0x7fc120009040 comp rx=0 tx=0).stop 2023-06-10T18:57:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 msgr2=0x7fc1240c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fc114009c10 tx=0x7fc114004750 comp rx=0 tx=0).stop 2023-06-10T18:57:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 shutdown_connections 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12415c500 0x7fc124155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc124069e70 0x7fc1240c4880 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc10806ace0 0x7fc10806d190 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 --2- 172.21.15.104:0/3802546138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12406e3e0 0x7fc1240c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 >> 172.21.15.104:0/3802546138 conn(0x7fc12406b8a0 msgr2=0x7fc1240ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 shutdown_connections 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.574+0000 7fc12c1a4700 1 -- 172.21.15.104:0/3802546138 wait complete. 2023-06-10T18:57:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:57:11.592 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.707+0000 7fd4a1b88700 1 Processor -- start 2023-06-10T18:57:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.707+0000 7fd4a1b88700 1 -- start start 2023-06-10T18:57:11.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd4a1b88700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd49c06b1f0 0x7fd49c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd4a1b88700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c0ce350 0x7fd49c0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd4a1b88700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0cadd0 con 0x7fd49c08a3d0 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd4a1b88700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c0cfeb0 con 0x7fd49c06b1f0 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd4a1b88700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0cb0b0 con 0x7fd49c0ce350 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd49c06b1f0 0x7fd49c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd49c06b1f0 0x7fd49c0cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38264/0 (socket says 172.21.15.104:38264) 2023-06-10T18:57:11.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 -- 172.21.15.104:0/3660492670 learned_addr learned my addr 172.21.15.104:0/3660492670 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49affd700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c0ce350 0x7fd49c0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 -- 172.21.15.104:0/3660492670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c0ce350 msgr2=0x7fd49c0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c0ce350 0x7fd49c0cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 -- 172.21.15.104:0/3660492670 >> v1:172.21.15.114:6789/0 conn(0x7fd49c08a3d0 legacy=0x7fd49c160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 -- 172.21.15.104:0/3660492670 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c0cf550 con 0x7fd49c06b1f0 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c06b1f0 0x7fd49c0cd090 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0098f0 tx=0x7fd48c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31d86b7427b4a67 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49a7fc700 1 -- 172.21.15.104:0/3660492670 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00b8a0 con 0x7fd49c06b1f0 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49a7fc700 1 -- 172.21.15.104:0/3660492670 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c00ba00 con 0x7fd49c06b1f0 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.708+0000 7fd49a7fc700 1 -- 172.21.15.104:0/3660492670 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00bd90 con 0x7fd49c06b1f0 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 -- 172.21.15.104:0/3660492670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c06b1f0 msgr2=0x7fd49c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c06b1f0 0x7fd49c0cd090 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0098f0 tx=0x7fd48c009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 -- 172.21.15.104:0/3660492670 shutdown_connections 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c06b1f0 0x7fd49c0cd090 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/3660492670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c0ce350 0x7fd49c0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 -- 172.21.15.104:0/3660492670 >> 172.21.15.104:0/3660492670 conn(0x7fd49c089fd0 msgr2=0x7fd49c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 -- 172.21.15.104:0/3660492670 shutdown_connections 2023-06-10T18:57:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.709+0000 7fd4a1b88700 1 -- 172.21.15.104:0/3660492670 wait complete. 2023-06-10T18:57:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.710+0000 7fd4a1b88700 1 Processor -- start 2023-06-10T18:57:11.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.710+0000 7fd4a1b88700 1 -- start start 2023-06-10T18:57:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 0x7fd49c151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c08a3d0 0x7fd49c0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0b11e0 con 0x7fd49c0ce350 2023-06-10T18:57:11.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0b0f00 con 0x7fd49c08a3d0 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd4a1b88700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0c55d0 con 0x7fd49c06b1f0 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38274/0 (socket says 172.21.15.104:38274) 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 -- 172.21.15.104:0/4238762513 learned_addr learned my addr 172.21.15.104:0/4238762513 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 0x7fd49c151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49affd700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c08a3d0 0x7fd49c0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 -- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 msgr2=0x7fd49c151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 0x7fd49c151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 -- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c08a3d0 msgr2=0x7fd49c0b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c08a3d0 0x7fd49c0b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49bfff700 1 -- 172.21.15.104:0/4238762513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c009550 con 0x7fd49c0ce350 2023-06-10T18:57:11.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.711+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 0x7fd49c151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd49bfff700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fd49400cc50 tx=0x7fd49400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400ad80 con 0x7fd49c0ce350 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400ce50 con 0x7fd49c0ce350 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494003c40 con 0x7fd49c0ce350 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c0c58b0 con 0x7fd49c0ce350 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.712+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c152100 con 0x7fd49c0ce350 2023-06-10T18:57:11.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.713+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c0c5dd0 con 0x7fd49c0ce350 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.713+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd4940050d0 con 0x7fd49c0ce350 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.713+0000 7fd483fff700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 0x7fd48806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.714+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd494089800 con 0x7fd49c0ce350 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.714+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 0x7fd48806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.714+0000 7fd49b7fe700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 0x7fd48806d1c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd48c000c00 tx=0x7fd48c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:11.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.718+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd49c0c5dd0 con 0x7fd49c0ce350 2023-06-10T18:57:11.862 INFO:tasks.ceph.mds.d.smithi114.stderr:2023-06-10T18:57:11.856+0000 7f8d19b6a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:57:11.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.945+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd49c0c5dd0 con 0x7fd49c0ce350 2023-06-10T18:57:11.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.946+0000 7fd483fff700 1 -- 172.21.15.104:0/4238762513 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4682 (secure 0 0 0) 0x7fd49c0c5dd0 con 0x7fd49c0ce350 2023-06-10T18:57:11.952 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:11.953 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/184392101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":184392101},{"type":"v1","addr":"172.21.15.104:6834","nonce":184392101}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9630,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2169449406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":2169449406},{"type":"v1","addr":"172.21.15.104:6835","nonce":2169449406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9639,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3805828460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3805828460},{"type":"v1","addr":"172.21.15.114:6836","nonce":3805828460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:09.668599+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9642},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9642":{"gid":9642,"name":"d","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2371761194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2371761194},{"type":"v1","addr":"172.21.15.114:6837","nonce":2371761194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:11.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 msgr2=0x7fd48806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 0x7fd48806d1c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd48c000c00 tx=0x7fd48c008030 comp rx=0 tx=0).stop 2023-06-10T18:57:11.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 msgr2=0x7fd49c0c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fd49400cc50 tx=0x7fd49400a340 comp rx=0 tx=0).stop 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 shutdown_connections 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd49c0ce350 0x7fd49c0c9a20 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd49c06b1f0 0x7fd49c151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd48806ad10 0x7fd48806d1c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 --2- 172.21.15.104:0/4238762513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd49c08a3d0 0x7fd49c0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:11.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 >> 172.21.15.104:0/4238762513 conn(0x7fd49c089fd0 msgr2=0x7fd49c15fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:11.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 shutdown_connections 2023-06-10T18:57:11.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:11.948+0000 7fd4a1b88700 1 -- 172.21.15.104:0/4238762513 wait complete. 2023-06-10T18:57:11.956 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 88 2023-06-10T18:57:12.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:57:06.271602+0000', 'modified': '2023-06-10T18:57:09.668599+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9642}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9642': {'gid': 9642, 'name': 'd', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/2371761194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 2371761194}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 2371761194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:57:12.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:57:12.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-10T18:57:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.488+0000 7fb3a5239700 1 Processor -- start 2023-06-10T18:57:12.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.488+0000 7fb3a5239700 1 -- start start 2023-06-10T18:57:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb3a5239700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb3a00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb3a5239700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3a008a3d0 0x7fb3a00cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb3a5239700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00cadd0 con 0x7fb3a006b1f0 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb3a5239700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a00cfeb0 con 0x7fb3a008a3d0 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb3a5239700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00cb0b0 con 0x7fb3a00ce350 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb39ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb3a006b1f0 0x7fb3a00cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36106/0 (socket says 172.21.15.104:36106) 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb39ed9d700 1 -- 172.21.15.104:0/664637835 learned_addr learned my addr 172.21.15.104:0/664637835 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb39e59c700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb3a00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb39f59e700 1 --2- 172.21.15.104:0/664637835 >> v2:172.21.15.114:3300/0 conn(0x7fb3a008a3d0 0x7fb3a00cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.489+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515713804 0 0) 0x7fb3a00cadd0 con 0x7fb3a006b1f0 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3880035b0 con 0x7fb3a006b1f0 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3616442476 0 0) 0x7fb3880035b0 con 0x7fb3a006b1f0 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 msgr2=0x7fb3a00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb3a00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 >> v2:172.21.15.114:3300/0 conn(0x7fb3a008a3d0 msgr2=0x7fb3a00cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 --2- 172.21.15.104:0/664637835 >> v2:172.21.15.114:3300/0 conn(0x7fb3a008a3d0 0x7fb3a00cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00cf550 con 0x7fb3a006b1f0 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb394001670 con 0x7fb3a006b1f0 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 >> v1:172.21.15.104:6789/0 conn(0x7fb3a006b1f0 legacy=0x7fb3a00cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb388003a00 0x7fb388003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb388004430 0x7fb3880048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb388005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00cfeb0 con 0x7fb388003a00 2023-06-10T18:57:12.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00cb0b0 con 0x7fb388004430 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38800ba10 con 0x7fb3a00ce350 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb388003a00 0x7fb388003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39e59c700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb388004430 0x7fb3880048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39f59e700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb388005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 -- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 msgr2=0x7fb388005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3a00ce350 0x7fb388005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 -- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb388004430 msgr2=0x7fb3880048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb388004430 0x7fb3880048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 -- 172.21.15.104:0/664637835 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880035b0 con 0x7fb388003a00 2023-06-10T18:57:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb388003a00 0x7fb388003e10 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb39400c350 tx=0x7fb388005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd915a7acd5fc47 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39400ce50 con 0x7fb388003a00 2023-06-10T18:57:12.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb394005960 con 0x7fb388003a00 2023-06-10T18:57:12.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.490+0000 7fb39dd9b700 1 -- 172.21.15.104:0/664637835 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394005d10 con 0x7fb388003a00 2023-06-10T18:57:12.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 -- 172.21.15.104:0/664637835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb388003a00 msgr2=0x7fb388003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 --2- 172.21.15.104:0/664637835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb388003a00 0x7fb388003e10 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb39400c350 tx=0x7fb388005880 comp rx=0 tx=0).stop 2023-06-10T18:57:12.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb39f59e700 1 -- 172.21.15.104:0/664637835 reap_dead start 2023-06-10T18:57:12.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 -- 172.21.15.104:0/664637835 shutdown_connections 2023-06-10T18:57:12.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 -- 172.21.15.104:0/664637835 >> 172.21.15.104:0/664637835 conn(0x7fb3a0089fd0 msgr2=0x7fb3a00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 -- 172.21.15.104:0/664637835 shutdown_connections 2023-06-10T18:57:12.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.491+0000 7fb3a5239700 1 -- 172.21.15.104:0/664637835 wait complete. 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 Processor -- start 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 -- start start 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb388003a00 0x7fb3a01e6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 0x7fb3a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a01e6860 con 0x7fb3a015b7c0 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00c8a90 con 0x7fb3a00b4870 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb3a5239700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00c98f0 con 0x7fb388003a00 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb39e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb39e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38292/0 (socket says 172.21.15.104:38292) 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.493+0000 7fb39e59c700 1 -- 172.21.15.104:0/4181094167 learned_addr learned my addr 172.21.15.104:0/4181094167 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39f59e700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 0x7fb3a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb388003a00 0x7fb3a01e6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 -- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb388003a00 msgr2=0x7fb3a01e6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb388003a00 0x7fb3a01e6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 -- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 msgr2=0x7fb3a015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 0x7fb3a015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 -- 172.21.15.104:0/4181094167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00c8ce0 con 0x7fb3a015b7c0 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39f59e700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 0x7fb3a015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb39e59c700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fb390011a00 tx=0x7fb390011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb3877fe700 1 -- 172.21.15.104:0/4181094167 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39000f940 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb3877fe700 1 -- 172.21.15.104:0/4181094167 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39000faa0 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a00c8fc0 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb3877fe700 1 -- 172.21.15.104:0/4181094167 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39000fe30 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.494+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a00c9580 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.495+0000 7fb3877fe700 1 -- 172.21.15.104:0/4181094167 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb390014440 con 0x7fb3a015b7c0 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.496+0000 7fb3877fe700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 0x7fb38c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.496+0000 7fb3877fe700 1 -- 172.21.15.104:0/4181094167 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb390089d00 con 0x7fb3a015b7c0 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.496+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 0x7fb38c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.496+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 0x7fb38c06d190 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb394015040 tx=0x7fb39400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.498+0000 7fb3b13beb80 1 Processor -- start 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.498+0000 7fb3b13beb80 1 -- start start 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c0d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c33820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2b11c60 con 0x5594c2c0a740 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2b649f0 con 0x5594c2c0cf50 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5594c2b5de80 con 0x5594c2c34d80 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38300/0 (socket says 172.21.15.104:38300) 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 -- 172.21.15.104:0/2428929944 learned_addr learned my addr 172.21.15.104:0/2428929944 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39e59c700 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c33820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c0d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 -- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 msgr2=0x5594c2c33820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c33820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 -- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 msgr2=0x5594c2c0d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c0d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 -- 172.21.15.104:0/2428929944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c29df5e0 con 0x5594c2c0a740 2023-06-10T18:57:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb398009380 tx=0x7fb398009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c3766678191665 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb384ff9700 1 -- 172.21.15.104:0/2428929944 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800e040 con 0x5594c2c0a740 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb384ff9700 1 -- 172.21.15.104:0/2428929944 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39800f040 con 0x5594c2c0a740 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.499+0000 7fb384ff9700 1 -- 172.21.15.104:0/2428929944 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398003c40 con 0x5594c2c0a740 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2428929944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 msgr2=0x5594c2c37440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb398009380 tx=0x7fb398009750 comp rx=0 tx=0).stop 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2428929944 shutdown_connections 2023-06-10T18:57:12.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c0d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0a740 0x5594c2c37440 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2428929944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c33820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2428929944 >> 172.21.15.104:0/2428929944 conn(0x5594c2c01fc0 msgr2=0x5594c2c521a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2428929944 shutdown_connections 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.500+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2428929944 wait complete. 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.501+0000 7fb3b13beb80 1 -- start start 2023-06-10T18:57:12.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.501+0000 7fb3b13beb80 1 Processor -- start 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c2d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0cf50 0x5594c2c3ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2afda70 con 0x5594c2c34d80 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2afd770 con 0x5594c2c0a740 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5594c29e18d0 con 0x5594c2c0cf50 2023-06-10T18:57:12.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 10 client.0 _ll_register_callbacks cb 0x5594c2b03210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38306/0 (socket says 172.21.15.104:38306) 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 -- 172.21.15.104:0/2892251394 learned_addr learned my addr 172.21.15.104:0/2892251394 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c2d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39e59c700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0cf50 0x5594c2c3ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 -- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0cf50 msgr2=0x5594c2c3ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0cf50 0x5594c2c3ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 -- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 msgr2=0x5594c2c2d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c2d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 -- 172.21.15.104:0/2892251394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb398009b10 con 0x5594c2c34d80 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb39f59e700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fb398019040 tx=0x7fb398004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4505921bce580c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb37dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800e040 con 0x5594c2c34d80 2023-06-10T18:57:12.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb398026070 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594c2bff780 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb398004240 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 10 client.9786 fetch_fsmap learned FSMap version 88 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.502+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x5594c298bd90 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5594c2bff780 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 88) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb398014040 con 0x5594c2c34d80 2023-06-10T18:57:12.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 10 client.9786 fetch_fsmap finished waiting for FSMap version 88 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 10 client.9786 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 20 client.9786 populate_metadata read hostname 'smithi104' 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c64de0 0x5594c2c651b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 4 client.9786 mds_command: new command op to 9642 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 --> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594c2b56e60 con 0x5594c2c64de0 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c64de0 0x5594c2c651b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb37dffb700 10 client.9786 ms_handle_connect on v2:172.21.15.114:6835/2371761194 2023-06-10T18:57:12.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.503+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c64de0 0x5594c2c651b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=929d0c77bec9649f server_cookie=b86ba17562be96d3 in_seq=0 out_seq=0 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb37dffb700 1 -- 172.21.15.104:0/2892251394 <== mds.0 v2:172.21.15.114:6835/2371761194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5594c2b56e60 con 0x5594c2c64de0 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb37dffb700 10 client.9786 handle_command_reply: tid=0 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 client.9786 shutdown 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 msgr2=0x5594c2c06220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fb398019040 tx=0x7fb398004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 shutdown_connections 2023-06-10T18:57:12.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c64de0 0x5594c2c651b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c2d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb37dffb700 0 client.9786 ms_handle_reset on v2:172.21.15.114:6835/2371761194 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c06220 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/2892251394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0cf50 0x5594c2c3ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 >> 172.21.15.104:0/2892251394 conn(0x5594c2c01fc0 msgr2=0x5594c2c2fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 shutdown_connections 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 1 -- 172.21.15.104:0/2892251394 wait complete. 2023-06-10T18:57:12.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.504+0000 7fb3b13beb80 20 client.9786 trim_cache size 0 max 16384 2023-06-10T18:57:12.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.522+0000 7fb3b13beb80 1 Processor -- start 2023-06-10T18:57:12.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.522+0000 7fb3b13beb80 1 -- start start 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c73db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c08460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2c065e0 con 0x5594c2c0cf50 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2c2d0a0 con 0x5594c2c0a740 2023-06-10T18:57:12.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5594c2c2d250 con 0x5594c2c34d80 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38312/0 (socket says 172.21.15.104:38312) 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 -- 172.21.15.104:0/259298875 learned_addr learned my addr 172.21.15.104:0/259298875 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 -- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 msgr2=0x5594c2c08460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c08460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 -- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 msgr2=0x5594c2c73db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c73db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 -- 172.21.15.104:0/259298875 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c2974c10 con 0x5594c2c0cf50 2023-06-10T18:57:12.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.523+0000 7fb39e59c700 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb3900586e0 tx=0x7fb39005e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248bae838aae654c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3857fa700 1 -- 172.21.15.104:0/259298875 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390056d80 con 0x5594c2c0cf50 2023-06-10T18:57:12.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3857fa700 1 -- 172.21.15.104:0/259298875 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb390070070 con 0x5594c2c0cf50 2023-06-10T18:57:12.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3857fa700 1 -- 172.21.15.104:0/259298875 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900580f0 con 0x5594c2c0cf50 2023-06-10T18:57:12.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 -- 172.21.15.104:0/259298875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 msgr2=0x5594c2c095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb3900586e0 tx=0x7fb39005e040 comp rx=0 tx=0).stop 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 -- 172.21.15.104:0/259298875 shutdown_connections 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0a740 0x5594c2c73db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c34d80 0x5594c2c08460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/259298875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c0cf50 0x5594c2c095b0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 -- 172.21.15.104:0/259298875 >> 172.21.15.104:0/259298875 conn(0x5594c2c01fc0 msgr2=0x5594c2c0c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 -- 172.21.15.104:0/259298875 shutdown_connections 2023-06-10T18:57:12.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.524+0000 7fb3b13beb80 1 -- 172.21.15.104:0/259298875 wait complete. 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.525+0000 7fb3b13beb80 1 -- start start 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.525+0000 7fb3b13beb80 1 Processor -- start 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0a740 0x5594c2c9e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c30250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2c9ebf0 con 0x5594c2c34d80 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c2cac810 con 0x5594c2c0cf50 2023-06-10T18:57:12.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5594c2c30d80 con 0x5594c2c0a740 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb3b13beb80 10 client.0 _ll_register_callbacks cb 0x5594c2b03210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38318/0 (socket says 172.21.15.104:38318) 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 -- 172.21.15.104:0/1026441360 learned_addr learned my addr 172.21.15.104:0/1026441360 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0a740 0x5594c2c9e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39e59c700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c30250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 -- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0a740 msgr2=0x5594c2c9e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0a740 0x5594c2c9e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 -- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 msgr2=0x5594c2c30250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c30250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.526+0000 7fb39f59e700 1 -- 172.21.15.104:0/1026441360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a00c9580 con 0x5594c2c34d80 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb39f59e700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb3980072a0 tx=0x7fb398004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fb72d2e67236ba server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:57:12.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39800e030 con 0x5594c2c34d80 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb398026040 con 0x5594c2c34d80 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb39800f040 con 0x5594c2c34d80 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3980043f0 con 0x5594c2c34d80 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 10 client.9792 fetch_fsmap learned FSMap version 88 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x5594c29b2200 con 0x5594c2c34d80 2023-06-10T18:57:12.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39800f040 con 0x5594c2c34d80 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 88) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb398041070 con 0x5594c2c34d80 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 10 client.9792 fetch_fsmap finished waiting for FSMap version 88 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 10 client.9792 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 20 client.9792 populate_metadata read hostname 'smithi104' 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c2dfb0 0x5594c2c2e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 4 client.9792 mds_command: new command op to 9642 tid=0[{"prefix": "status"}] 2023-06-10T18:57:12.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 --> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5594c2b19000 con 0x5594c2c2dfb0 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.527+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c2dfb0 0x5594c2c2e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb37dffb700 10 client.9792 ms_handle_connect on v2:172.21.15.114:6835/2371761194 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb39ed9d700 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c2dfb0 0x5594c2c2e380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe168edefda5403a server_cookie=eced5dfc73b925a7 in_seq=0 out_seq=0 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb37dffb700 1 -- 172.21.15.104:0/1026441360 <== mds.0 v2:172.21.15.114:6835/2371761194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5594c2b19000 con 0x5594c2c2dfb0 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb37dffb700 10 client.9792 handle_command_reply: tid=0 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 client.9792 shutdown 2023-06-10T18:57:12.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 msgr2=0x5594c2c30790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb3980072a0 tx=0x7fb398004b40 comp rx=0 tx=0).stop 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 shutdown_connections 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:6835/2371761194,v1:172.21.15.114:6837/2371761194] conn(0x5594c2c2dfb0 0x5594c2c2e380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5594c2c34d80 0x5594c2c30790 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb37dffb700 0 client.9792 ms_handle_reset on v2:172.21.15.114:6835/2371761194 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5594c2c0a740 0x5594c2c9e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 --2- 172.21.15.104:0/1026441360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594c2c0cf50 0x5594c2c30250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.528+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 >> 172.21.15.104:0/1026441360 conn(0x5594c2c01fc0 msgr2=0x5594c2c73c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 shutdown_connections 2023-06-10T18:57:12.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3b13beb80 1 -- 172.21.15.104:0/1026441360 wait complete. 2023-06-10T18:57:12.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3b13beb80 20 client.9792 trim_cache size 0 max 16384 2023-06-10T18:57:12.545 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 9642, 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:57:12.546 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 6.2559707060000003, 2023-06-10T18:57:12.547 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 88, 2023-06-10T18:57:12.547 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 143, 2023-06-10T18:57:12.547 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 143, 2023-06-10T18:57:12.547 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.591932590000001 2023-06-10T18:57:12.547 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:12.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 msgr2=0x7fb38c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 0x7fb38c06d190 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb394015040 tx=0x7fb39400a000 comp rx=0 tx=0).stop 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 msgr2=0x7fb3a00c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fb390011a00 tx=0x7fb390011dc0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 shutdown_connections 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb38c06ace0 0x7fb38c06d190 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a00b4870 0x7fb3a015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb388003a00 0x7fb3a01e6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 --2- 172.21.15.104:0/4181094167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a015b7c0 0x7fb3a00c41e0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 >> 172.21.15.104:0/4181094167 conn(0x7fb3a0089fd0 msgr2=0x7fb3a00cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 shutdown_connections 2023-06-10T18:57:12.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.529+0000 7fb3a5239700 1 -- 172.21.15.104:0/4181094167 wait complete. 2023-06-10T18:57:12.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:57:12.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.664+0000 7f6c3154f700 1 Processor -- start 2023-06-10T18:57:12.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.664+0000 7f6c3154f700 1 -- start start 2023-06-10T18:57:12.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c3154f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 0x7f6c2c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c3154f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c0ccca0 con 0x7f6c2c15a250 2023-06-10T18:57:12.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c3154f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c0cee50 con 0x7f6c2c0cfa00 2023-06-10T18:57:12.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c3154f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c0cf130 con 0x7f6c2c0cad90 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c2a7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6c2c15a250 0x7f6c2c0b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36152/0 (socket says 172.21.15.104:36152) 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c2a7fc700 1 -- 172.21.15.104:0/2910353190 learned_addr learned my addr 172.21.15.104:0/2910353190 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.665+0000 7f6c2b7fe700 1 --2- 172.21.15.104:0/2910353190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 0x7f6c2c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440618943 0 0) 0x7f6c2c0ccca0 con 0x7f6c2c15a250 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c140035b0 con 0x7f6c2c15a250 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552233728 0 0) 0x7f6c2c0cee50 con 0x7f6c2c0cfa00 2023-06-10T18:57:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c2b7fe700 1 -- 172.21.15.104:0/2910353190 >> v1:172.21.15.114:6789/0 conn(0x7f6c2c0cfa00 legacy=0x7f6c2c0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c2b7fe700 1 -- 172.21.15.104:0/2910353190 >> v1:172.21.15.104:6789/0 conn(0x7f6c2c15a250 legacy=0x7f6c2c0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c2b7fe700 1 -- 172.21.15.104:0/2910353190 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c0cc930 con 0x7f6c2c0cad90 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c2b7fe700 1 --2- 172.21.15.104:0/2910353190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 0x7f6c2c0701e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009dc0 tx=0x7f6c2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f5bf44a7ecbe21 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400e070 con 0x7f6c2c0cad90 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2400bb20 con 0x7f6c2c0cad90 2023-06-10T18:57:12.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.666+0000 7f6c29ffb700 1 -- 172.21.15.104:0/2910353190 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400be90 con 0x7f6c2c0cad90 2023-06-10T18:57:12.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 -- 172.21.15.104:0/2910353190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 msgr2=0x7f6c2c0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2910353190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 0x7f6c2c0701e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009dc0 tx=0x7f6c2400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:12.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 -- 172.21.15.104:0/2910353190 shutdown_connections 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2910353190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cad90 0x7f6c2c0701e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 -- 172.21.15.104:0/2910353190 >> 172.21.15.104:0/2910353190 conn(0x7f6c2c06e370 msgr2=0x7f6c2c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 -- 172.21.15.104:0/2910353190 shutdown_connections 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.667+0000 7f6c3154f700 1 -- 172.21.15.104:0/2910353190 wait complete. 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.668+0000 7f6c3154f700 1 Processor -- start 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.668+0000 7f6c3154f700 1 -- start start 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cfa00 0x7f6c2c15ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c2c15a250 0x7f6c2c1f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c15d370 con 0x7f6c2c0cad90 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c0c5cd0 con 0x7f6c2c15a250 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c0c5fb0 con 0x7f6c2c0cfa00 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38328/0 (socket says 172.21.15.104:38328) 2023-06-10T18:57:12.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 -- 172.21.15.104:0/2834854752 learned_addr learned my addr 172.21.15.104:0/2834854752 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 -- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cfa00 msgr2=0x7f6c2c15ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2a7fc700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cfa00 0x7f6c2c15ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2b7fe700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c2c15a250 0x7f6c2c1f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cfa00 0x7f6c2c15ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 -- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c2c15a250 msgr2=0x7f6c2c1f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c2c15a250 0x7f6c2c1f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 -- 172.21.15.104:0/2834854752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24009a70 con 0x7f6c2c0cad90 2023-06-10T18:57:12.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c2affd700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c500 tx=0x7f6c2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200054e0 con 0x7f6c2c0cad90 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20003680 con 0x7f6c2c0cad90 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c0c6290 con 0x7f6c2c0cad90 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.669+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20003a30 con 0x7f6c2c0cad90 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.670+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c0c6850 con 0x7f6c2c0cad90 2023-06-10T18:57:12.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.670+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c074050 con 0x7f6c2c0cad90 2023-06-10T18:57:12.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.671+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6c20003b90 con 0x7f6c2c0cad90 2023-06-10T18:57:12.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.671+0000 7f6c137fe700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 0x7f6c1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:12.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.671+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6c200889c0 con 0x7f6c2c0cad90 2023-06-10T18:57:12.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.672+0000 7f6c2a7fc700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 0x7f6c1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:12.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.672+0000 7f6c2a7fc700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 0x7f6c1806d190 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0af860 tx=0x7f6c1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:12.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.676+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6c200539c0 con 0x7f6c2c0cad90 2023-06-10T18:57:12.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.905+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c2c15d4b0 con 0x7f6c2c0cad90 2023-06-10T18:57:12.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.905+0000 7f6c137fe700 1 -- 172.21.15.104:0/2834854752 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v88) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6c2c15d4b0 con 0x7f6c2c0cad90 2023-06-10T18:57:12.912 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.907+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 msgr2=0x7f6c1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.907+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 0x7f6c1806d190 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0af860 tx=0x7f6c1c009040 comp rx=0 tx=0).stop 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.907+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 msgr2=0x7f6c2c0b0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.907+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c500 tx=0x7f6c2000c8c0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 shutdown_connections 2023-06-10T18:57:12.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6c1806ace0 0x7f6c1806d190 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c2c15a250 0x7f6c2c1f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c2c0cad90 0x7f6c2c0b0230 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 --2- 172.21.15.104:0/2834854752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c2c0cfa00 0x7f6c2c15ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 >> 172.21.15.104:0/2834854752 conn(0x7f6c2c06e370 msgr2=0x7f6c2c156530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 shutdown_connections 2023-06-10T18:57:12.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:12.908+0000 7f6c3154f700 1 -- 172.21.15.104:0/2834854752 wait complete. 2023-06-10T18:57:12.925 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:57:12.926 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:57:12.926 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:57:12.926 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:57:12.928 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:12.928 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:57:12.928 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:57:12.929 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:57:12.929 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:12.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:12.930 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:57:12.946 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.947 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:57:12.948 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41651sec preferred_lft 41651sec 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:12.949 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:57:12.950 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.951 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:12.951 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:57:12.951 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:57:12.951 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:12.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:12.952 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:57:13.011 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:57:13.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.013 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:57:13.071 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:57:13.071 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:57:13.071 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:57:13.071 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:57:13.072 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:57:13.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.072 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:57:13.073 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:57:13.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:57:13.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:57:13.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:57:13.074 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:57:13.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.175 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:57:13.191 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:57:13.192 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:57:13.192 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:13.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:57:13.258 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:57:13.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:13.259 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:13.314 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:13.315 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:13.315 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:13.372 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:57:13.439 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:13.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:13.505 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:13.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.507 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:57:13.564 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:57:13.565 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:57:13.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:57:13.634 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:13.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:13.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:57:13.688+0000 7fa5a7d155c0 -1 init, newargv = 0x55d5a3b3fac0 newargc=15 2023-06-10T18:57:13.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[152305]: starting ceph client 2023-06-10T18:57:13.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[152305]: starting fuse 2023-06-10T18:57:13.702 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:13.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:13.705 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:57:13.762 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:57:13.763 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:57:13.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:13.764 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:57:13.764 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:57:13.764 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:57:13.765 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:57:13.765 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:13.765 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:57:13.765 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:57:13.766 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:57:13.766 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:57:13.766 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:13.767 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:57:13.767 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:57:13.767 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:57:13.768 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:57:13.768 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:13.768 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:57:13.769 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:57:13.769 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:57:13.769 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:57:13.770 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:57:13.770 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:57:13.771 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:57:13.771 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:57:13.771 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:13.772 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:57:13.772 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:57:13.885 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.152305.asok 2023-06-10T18:57:14.068 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.152305.asok 2023-06-10T18:57:14.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.069 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.152305.asok status 2023-06-10T18:57:14.223 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:14.223 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:57:14.223 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:57:14.223 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:57:14.223 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "152305", 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:57:14.224 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 9804, 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 9804 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:14.225 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2732798687 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:14.226 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2732798687 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.9804 192.168.0.1:0/2732798687", 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/2732798687", 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 88, 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 143, 2023-06-10T18:57:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:57:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:57:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:57:14.228 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:14.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.233 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:57:14.233 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:14.250 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:57:14.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:14.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.251 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:14.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:57:14.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.471+0000 7f0864869700 1 Processor -- start 2023-06-10T18:57:14.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 -- start start 2023-06-10T18:57:14.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08600b1e30 0x7f08600d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08600d0a30 0x7f086006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086006bfd0 con 0x7f086006b820 2023-06-10T18:57:14.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08600d0ea0 con 0x7f08600b1e30 2023-06-10T18:57:14.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.472+0000 7f0864869700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086006c2b0 con 0x7f08600d0a30 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08600b1e30 0x7f08600d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f08600b1e30 0x7f08600d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38350/0 (socket says 172.21.15.104:38350) 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 -- 172.21.15.104:0/778586552 learned_addr learned my addr 172.21.15.104:0/778586552 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 -- 172.21.15.104:0/778586552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08600d0a30 msgr2=0x7f086006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 --2- 172.21.15.104:0/778586552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08600d0a30 0x7f086006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 -- 172.21.15.104:0/778586552 >> v1:172.21.15.114:6789/0 conn(0x7f086006b820 legacy=0x7f08600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:14.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 -- 172.21.15.104:0/778586552 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600ccf70 con 0x7f08600b1e30 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085e59c700 1 --2- 172.21.15.104:0/778586552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600b1e30 0x7f08600d04f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f08480098f0 tx=0x7f0848009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321dc785b9b5f571 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085d59a700 1 -- 172.21.15.104:0/778586552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848004230 con 0x7f08600b1e30 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085d59a700 1 -- 172.21.15.104:0/778586552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0848004390 con 0x7f08600b1e30 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.473+0000 7f085d59a700 1 -- 172.21.15.104:0/778586552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480036a0 con 0x7f08600b1e30 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 -- 172.21.15.104:0/778586552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600b1e30 msgr2=0x7f08600d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 --2- 172.21.15.104:0/778586552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600b1e30 0x7f08600d04f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f08480098f0 tx=0x7f0848009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:14.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 -- 172.21.15.104:0/778586552 shutdown_connections 2023-06-10T18:57:14.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 --2- 172.21.15.104:0/778586552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600b1e30 0x7f08600d04f0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 --2- 172.21.15.104:0/778586552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08600d0a30 0x7f086006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 -- 172.21.15.104:0/778586552 >> 172.21.15.104:0/778586552 conn(0x7f0860071ec0 msgr2=0x7f086009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:14.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 -- 172.21.15.104:0/778586552 shutdown_connections 2023-06-10T18:57:14.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.474+0000 7f0864869700 1 -- 172.21.15.104:0/778586552 wait complete. 2023-06-10T18:57:14.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.475+0000 7f0864869700 1 Processor -- start 2023-06-10T18:57:14.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.475+0000 7f0864869700 1 -- start start 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 0x7f086015e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 0x7f086015b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086015dd40 con 0x7f08600d0a30 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086015f830 con 0x7f08600b1e30 2023-06-10T18:57:14.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f0864869700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086015f2b0 con 0x7f086006b820 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38360/0 (socket says 172.21.15.104:38360) 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 -- 172.21.15.104:0/2606753147 learned_addr learned my addr 172.21.15.104:0/2606753147 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085dd9b700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 0x7f086015b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085e59c700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 0x7f086015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 -- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 msgr2=0x7f086015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 0x7f086015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 -- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 msgr2=0x7f086015b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 0x7f086015b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 -- 172.21.15.104:0/2606753147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848009550 con 0x7f08600d0a30 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085dd9b700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 0x7f086015b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085e59c700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 0x7f086015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.476+0000 7f085ed9d700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f085800b990 tx=0x7f085800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.477+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800c970 con 0x7f08600d0a30 2023-06-10T18:57:14.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.477+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085800cad0 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.477+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086015f550 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.477+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800ce80 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.477+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601e50f0 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.478+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08600d1860 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.478+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0858006d50 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.478+0000 7f0856ffd700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 0x7f084006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.479+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0858088fb0 con 0x7f08600d0a30 2023-06-10T18:57:14.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.479+0000 7f085e59c700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 0x7f084006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:14.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.479+0000 7f085e59c700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 0x7f084006d190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0848009130 tx=0x7f0848008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:14.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.483+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f08580bb310 con 0x7f08600d0a30 2023-06-10T18:57:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.714+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f086015de80 con 0x7f08600d0a30 2023-06-10T18:57:14.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.714+0000 7f0856ffd700 1 -- 172.21.15.104:0/2606753147 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v88) v1 ==== 53+0+73 (secure 0 0 0) 0x7f086015de80 con 0x7f08600d0a30 2023-06-10T18:57:14.721 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:57:14.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 msgr2=0x7f084006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 0x7f084006d190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0848009130 tx=0x7f0848008030 comp rx=0 tx=0).stop 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 msgr2=0x7f08601e4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f085800b990 tx=0x7f085800bd50 comp rx=0 tx=0).stop 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 shutdown_connections 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08600d0a30 0x7f08601e4bb0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086006b820 0x7f086015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f084006ace0 0x7f084006d190 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.716+0000 7f0864869700 1 --2- 172.21.15.104:0/2606753147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08600b1e30 0x7f086015b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.717+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 >> 172.21.15.104:0/2606753147 conn(0x7f0860071ec0 msgr2=0x7f08600b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.717+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 shutdown_connections 2023-06-10T18:57:14.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:14.717+0000 7f0864869700 1 -- 172.21.15.104:0/2606753147 wait complete. 2023-06-10T18:57:14.734 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:57:14.734 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:57:14.734 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:57:14.735 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:57:14.735 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:14.735 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:57:14.736 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:57:14.736 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:57:14.736 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:14.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.737 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:57:14.758 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:57:14.758 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:14.759 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:57:14.760 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41646sec preferred_lft 41646sec 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:57:14.761 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:57:14.762 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:57:14.763 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:14.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.763 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:57:14.829 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:57:14.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.830 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:57:14.890 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:57:14.891 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:57:14.891 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:57:14.891 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:57:14.891 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:57:14.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:14.892 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:57:14.892 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:57:14.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:57:14.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:57:14.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:57:14.893 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:57:15.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:15.000 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:57:15.018 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:57:15.018 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:57:15.019 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:15.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:15.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:57:15.085 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:57:15.086 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:15.087 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:15.141 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:15.142 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:15.143 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:15.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:57:15.264 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:15.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:15.330 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:15.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:15.333 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:57:15.390 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:57:15.391 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:57:15.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:57:15.457 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:15.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:15.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:57:15.511+0000 7f51222895c0 -1 init, newargv = 0x55b89047fae0 newargc=15 2023-06-10T18:57:15.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[115813]: starting ceph client 2023-06-10T18:57:15.522 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[115813]: starting fuse 2023-06-10T18:57:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:15.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:15.526 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:57:15.583 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:57:15.583 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:57:15.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:15.584 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:57:15.584 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:57:15.585 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:57:15.585 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:57:15.585 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:15.586 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:57:15.586 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:57:15.586 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:57:15.587 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:57:15.587 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:15.588 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:57:15.588 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:57:15.588 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:57:15.589 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:57:15.589 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:15.589 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:57:15.590 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:57:15.590 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:57:15.590 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:57:15.591 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:57:15.591 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:57:15.591 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:57:15.591 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:57:15.592 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:15.592 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:57:15.592 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:15.706 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.115813.asok 2023-06-10T18:57:15.886 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.115813.asok 2023-06-10T18:57:15.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:15.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.115813.asok status 2023-06-10T18:57:16.043 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:57:16.043 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:57:16.043 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:57:16.044 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:57:16.044 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:57:16.044 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:57:16.044 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "115813", 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 8365, 2023-06-10T18:57:16.045 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 8365 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:57:16.046 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1628805912 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:16.047 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1628805912 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.8365 192.168.0.1:0/1628805912", 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1628805912", 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 88, 2023-06-10T18:57:16.048 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 143, 2023-06-10T18:57:16.049 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:57:16.049 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:57:16.049 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:57:16.049 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:57:16.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:16.053 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:57:16.054 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:16.071 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:57:16.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:16.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:16.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:16.137 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:57:16.284 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:57:16.284 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:57:16.285 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:57:16.419 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:57:16.420 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:57:16.421 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q fuse --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:57:16.443 INFO:teuthology.orchestra.run.smithi104.stdout:2.9.7-15.el8 2023-06-10T18:57:16.443 INFO:teuthology.packaging:The installed version of fuse is 2.9.7-15.el8 2023-06-10T18:57:16.444 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q fuse --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:57:16.469 INFO:teuthology.orchestra.run.smithi114.stdout:2.9.7-15.el8 2023-06-10T18:57:16.470 INFO:teuthology.packaging:The installed version of fuse is 2.9.7-15.el8 2023-06-10T18:57:16.470 INFO:tasks.cephfs.test_client_recovery:flock locks are available 2023-06-10T18:57:16.471 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:16.471 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:16.500 INFO:tasks.cephfs.mount:lock_background file background_file 2023-06-10T18:57:16.501 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:16.501 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:57:16.501 DEBUG:teuthology.orchestra.run.smithi104:> import fcntl 2023-06-10T18:57:16.502 DEBUG:teuthology.orchestra.run.smithi104:> import struct 2023-06-10T18:57:16.502 DEBUG:teuthology.orchestra.run.smithi104:> f1 = open("/home/ubuntu/cephtest/mnt.0/background_file-1", '"'"'w'"'"') 2023-06-10T18:57:16.502 DEBUG:teuthology.orchestra.run.smithi104:> fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB) 2023-06-10T18:57:16.503 DEBUG:teuthology.orchestra.run.smithi104:> f2 = open("/home/ubuntu/cephtest/mnt.0/background_file-2", '"'"'w'"'"') 2023-06-10T18:57:16.503 DEBUG:teuthology.orchestra.run.smithi104:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:16.503 DEBUG:teuthology.orchestra.run.smithi104:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:16.503 DEBUG:teuthology.orchestra.run.smithi104:> while True: 2023-06-10T18:57:16.504 DEBUG:teuthology.orchestra.run.smithi104:> time.sleep(1) 2023-06-10T18:57:16.504 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:57:16.543 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:57:16.561 INFO:teuthology.orchestra.run.smithi114.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.1/background_file-2': No such file or directory 2023-06-10T18:57:17.562 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4194304 regular empty file 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 2fh/47d Inode: 1099511627777 Links: 1 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0664/-rw-rw-r--) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fusefs_t:s0 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:57:16.606426616 +0000 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:57:16.606426616 +0000 2023-06-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:57:16.606426616 +0000 2023-06-10T18:57:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:57:17.594 INFO:tasks.cephfs.mount:File background_file-2 became visible from 1 after 1s 2023-06-10T18:57:17.594 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:17.595 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:17.653 INFO:tasks.cephfs.mount:check lock on file background_file 2023-06-10T18:57:17.654 DEBUG:teuthology.orchestra.run.smithi114:> python3 -c ' 2023-06-10T18:57:17.654 DEBUG:teuthology.orchestra.run.smithi114:> import fcntl 2023-06-10T18:57:17.654 DEBUG:teuthology.orchestra.run.smithi114:> import errno 2023-06-10T18:57:17.654 DEBUG:teuthology.orchestra.run.smithi114:> import struct 2023-06-10T18:57:17.655 DEBUG:teuthology.orchestra.run.smithi114:> f1 = open("/home/ubuntu/cephtest/mnt.1/background_file-1", '"'"'r'"'"') 2023-06-10T18:57:17.655 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:17.655 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB) 2023-06-10T18:57:17.655 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:17.656 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:17.656 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:17.656 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:17.656 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("flock on file /home/ubuntu/cephtest/mnt.1/background_file-1 not found") 2023-06-10T18:57:17.657 DEBUG:teuthology.orchestra.run.smithi114:> f2 = open("/home/ubuntu/cephtest/mnt.1/background_file-2", '"'"'r'"'"') 2023-06-10T18:57:17.657 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:17.657 DEBUG:teuthology.orchestra.run.smithi114:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:17.657 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:17.658 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:17.658 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:17.658 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:17.659 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:17.659 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("posix lock on file /home/ubuntu/cephtest/mnt.1/background_file-2 not found") 2023-06-10T18:57:17.659 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:17.748 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:57:17.749 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:17.749 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:57:17.749 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:17.749 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:57:17.750 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:17.750 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:57:17.750 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:17.759 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:57:17.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2023-06-10T18:57:17.761 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:57:17.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2023-06-10T18:57:17.762 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:57:17.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2023-06-10T18:57:17.769 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:57:17.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2023-06-10T18:57:17.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.941+0000 7f1150f85700 1 Processor -- start 2023-06-10T18:57:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.941+0000 7f1150f85700 1 -- start start 2023-06-10T18:57:17.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.942+0000 7fc98b6c7700 1 Processor -- start 2023-06-10T18:57:17.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.942+0000 7fc98b6c7700 1 -- start start 2023-06-10T18:57:17.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f1150f85700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f114c15a230 0x7f114c0ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f1150f85700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f114c06d810 0x7f114c06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f1150f85700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0cc050 con 0x7f114c069e70 2023-06-10T18:57:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f1150f85700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c06e290 con 0x7f114c06d810 2023-06-10T18:57:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f1150f85700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c0cba00 con 0x7f114c15a230 2023-06-10T18:57:17.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114a59c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f114c15a230 0x7f114c0ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114ad9d700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f114c069e70 0x7f114c0cc4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:42044/0 (socket says 172.21.15.104:42044) 2023-06-10T18:57:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114ad9d700 1 -- 172.21.15.104:0/3041364574 learned_addr learned my addr 172.21.15.104:0/3041364574 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:17.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc98b6c7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc98415a0b0 0x7fc9840717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc98b6c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc984071130 0x7fc98415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc98b6c7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98406b2e0 con 0x7fc9840c93e0 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc98b6c7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98415e150 con 0x7fc98415a0b0 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc98b6c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415db40 con 0x7fc984071130 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc988c62700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc9840c93e0 0x7fc984070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36202/0 (socket says 172.21.15.104:36202) 2023-06-10T18:57:17.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc988c62700 1 -- 172.21.15.104:0/1799648120 learned_addr learned my addr 172.21.15.104:0/1799648120 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:17.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114a59c700 1 -- 172.21.15.104:0/3041364574 >> v2:172.21.15.104:3300/0 conn(0x7f114c06d810 msgr2=0x7f114c06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114a59c700 1 --2- 172.21.15.104:0/3041364574 >> v2:172.21.15.104:3300/0 conn(0x7f114c06d810 0x7f114c06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114a59c700 1 -- 172.21.15.104:0/3041364574 >> v1:172.21.15.114:6789/0 conn(0x7f114c069e70 legacy=0x7f114c0cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114959a700 1 -- 172.21.15.104:0/3041364574 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938389627 0 0) 0x7f114c0cc050 con 0x7f114c069e70 2023-06-10T18:57:17.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7f114a59c700 1 -- 172.21.15.104:0/3041364574 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c0cbce0 con 0x7f114c15a230 2023-06-10T18:57:17.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.943+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613217872 0 0) 0x7fc98406b2e0 con 0x7fc9840c93e0 2023-06-10T18:57:17.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9700035b0 con 0x7fc9840c93e0 2023-06-10T18:57:17.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc989463700 1 --2- 172.21.15.104:0/1799648120 >> v2:172.21.15.104:3300/0 conn(0x7fc98415a0b0 0x7fc9840717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7f114a59c700 1 --2- 172.21.15.104:0/3041364574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c15a230 0x7f114c0ac2c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f114c0b21d0 tx=0x7f113c009b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e7f048250067628 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3929382336 0 0) 0x7fc9700035b0 con 0x7fc9840c93e0 2023-06-10T18:57:17.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc984071130 msgr2=0x7fc98415e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-10T18:57:17.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc984071130 0x7fc98415e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7f114959a700 1 -- 172.21.15.104:0/3041364574 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c005e30 con 0x7f114c15a230 2023-06-10T18:57:17.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 >> v2:172.21.15.104:3300/0 conn(0x7fc98415a0b0 msgr2=0x7fc9840717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1799648120 >> v2:172.21.15.104:3300/0 conn(0x7fc98415a0b0 0x7fc9840717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98406b530 con 0x7fc9840c93e0 2023-06-10T18:57:17.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc974001670 con 0x7fc9840c93e0 2023-06-10T18:57:17.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7f114959a700 1 -- 172.21.15.104:0/3041364574 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c006030 con 0x7f114c15a230 2023-06-10T18:57:17.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7f114959a700 1 -- 172.21.15.104:0/3041364574 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c0063c0 con 0x7f114c15a230 2023-06-10T18:57:17.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 >> v1:172.21.15.104:6789/0 conn(0x7fc9840c93e0 legacy=0x7fc984070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc970003a00 0x7fc970003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9700043e0 0x7fc9700048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc970005060 0x7fc97000fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415e150 con 0x7fc970005060 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98415db40 con 0x7fc970003a00 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc970010410 con 0x7fc9700043e0 2023-06-10T18:57:17.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc988c62700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9700043e0 0x7fc9700048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc989463700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc970003a00 0x7fc970003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc988c62700 1 -- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc970003a00 msgr2=0x7fc970003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc988c62700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc970003a00 0x7fc970003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc988c62700 1 -- 172.21.15.104:0/1799648120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc970005060 msgr2=0x7fc97000fe20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.944+0000 7fc988c62700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc970005060 0x7fc97000fe20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7fc988c62700 1 -- 172.21.15.104:0/1799648120 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98406b530 con 0x7fc9700043e0 2023-06-10T18:57:17.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7fc988c62700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9700043e0 0x7fc9700048b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c510 tx=0x7fc970005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=827813cda8da3346 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 -- 172.21.15.104:0/3041364574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c15a230 msgr2=0x7f114c0ac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 --2- 172.21.15.104:0/3041364574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c15a230 0x7f114c0ac2c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f114c0b21d0 tx=0x7f113c009b00 comp rx=0 tx=0).stop 2023-06-10T18:57:17.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 -- 172.21.15.104:0/3041364574 shutdown_connections 2023-06-10T18:57:17.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 --2- 172.21.15.104:0/3041364574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c15a230 0x7f114c0ac2c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974005960 con 0x7fc9700043e0 2023-06-10T18:57:17.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 --2- 172.21.15.104:0/3041364574 >> v2:172.21.15.104:3300/0 conn(0x7f114c06d810 0x7f114c06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 -- 172.21.15.104:0/3041364574 >> 172.21.15.104:0/3041364574 conn(0x7f114c06b8a0 msgr2=0x7f114c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:17.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc974005ac0 con 0x7fc9700043e0 2023-06-10T18:57:17.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7fc97bfff700 1 -- 172.21.15.104:0/1799648120 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974005e70 con 0x7fc9700043e0 2023-06-10T18:57:17.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 -- 172.21.15.104:0/3041364574 shutdown_connections 2023-06-10T18:57:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.945+0000 7f1150f85700 1 -- 172.21.15.104:0/3041364574 wait complete. 2023-06-10T18:57:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 -- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9700043e0 msgr2=0x7fc9700048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/1799648120 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9700043e0 0x7fc9700048b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c510 tx=0x7fc970005880 comp rx=0 tx=0).stop 2023-06-10T18:57:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc989c64700 1 -- 172.21.15.104:0/1799648120 reap_dead start 2023-06-10T18:57:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 -- 172.21.15.104:0/1799648120 shutdown_connections 2023-06-10T18:57:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 -- 172.21.15.104:0/1799648120 >> 172.21.15.104:0/1799648120 conn(0x7fc9840b1610 msgr2=0x7fc9840ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 -- 172.21.15.104:0/1799648120 shutdown_connections 2023-06-10T18:57:17.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7fc98b6c7700 1 -- 172.21.15.104:0/1799648120 wait complete. 2023-06-10T18:57:17.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.946+0000 7f1150f85700 1 Processor -- start 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 -- start start 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c06d810 0x7f114c15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f114c15a230 0x7f114c0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15c1e0 con 0x7f114c069e70 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c156bf0 con 0x7f114c06d810 2023-06-10T18:57:17.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.947+0000 7f1150f85700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15b5d0 con 0x7f114c15a230 2023-06-10T18:57:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38382/0 (socket says 172.21.15.104:38382) 2023-06-10T18:57:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 -- 172.21.15.104:0/3399496701 learned_addr learned my addr 172.21.15.104:0/3399496701 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114ad9d700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f114c15a230 0x7f114c0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7fc98b6c7700 1 Processor -- start 2023-06-10T18:57:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 -- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f114c15a230 msgr2=0x7f114c0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f114c15a230 0x7f114c0b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 -- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c06d810 msgr2=0x7f114c15c670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-10T18:57:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7fc98b6c7700 1 -- start start 2023-06-10T18:57:17.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c06d810 0x7f114c15c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 -- 172.21.15.104:0/3399496701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c009430 con 0x7f114c069e70 2023-06-10T18:57:17.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f3699956700 1 Processor -- start 2023-06-10T18:57:17.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f114a59c700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f113c009b40 tx=0x7f113c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f3699956700 1 -- start start 2023-06-10T18:57:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c004240 con 0x7f114c069e70 2023-06-10T18:57:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c15b8b0 con 0x7f114c069e70 2023-06-10T18:57:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c0043c0 con 0x7f114c069e70 2023-06-10T18:57:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c0036a0 con 0x7f114c069e70 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc984155c60 0x7fc9840af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9840af8e0 0x7fc9840b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840afd50 con 0x7fc9700043e0 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9840ce540 con 0x7fc984155c60 2023-06-10T18:57:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc98b6c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc984154410 con 0x7fc9840af8e0 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36940ae280 0x7f36940cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3694070a20 0x7f3694070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36940713e0 con 0x7f3694070a20 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36940ccea0 con 0x7f36940ae280 2023-06-10T18:57:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3699956700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36940ae6f0 con 0x7f369415c040 2023-06-10T18:57:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.948+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c15be70 con 0x7f114c069e70 2023-06-10T18:57:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc989463700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc989463700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38394/0 (socket says 172.21.15.104:38394) 2023-06-10T18:57:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7fc989463700 1 -- 172.21.15.104:0/560597526 learned_addr learned my addr 172.21.15.104:0/560597526 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:57784/0 (socket says 172.21.15.104:57784) 2023-06-10T18:57:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.949+0000 7f3692ffd700 1 -- 172.21.15.104:0/2930414124 learned_addr learned my addr 172.21.15.104:0/2930414124 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc988c62700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc984155c60 0x7fc9840af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 -- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9840af8e0 msgr2=0x7fc9840b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9840af8e0 0x7fc9840b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 -- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc984155c60 msgr2=0x7fc9840af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc984155c60 0x7fc9840af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 -- 172.21.15.104:0/560597526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9841546f0 con 0x7fc9700043e0 2023-06-10T18:57:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 -- 172.21.15.104:0/2930414124 >> v2:172.21.15.114:3300/0 conn(0x7f36940ae280 msgr2=0x7f36940cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 --2- 172.21.15.104:0/2930414124 >> v2:172.21.15.114:3300/0 conn(0x7f36940ae280 0x7f36940cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 -- 172.21.15.104:0/2930414124 >> v2:172.21.15.104:3300/0 conn(0x7f3694070a20 msgr2=0x7f3694070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 --2- 172.21.15.104:0/2930414124 >> v2:172.21.15.104:3300/0 conn(0x7f3694070a20 0x7f3694070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 -- 172.21.15.104:0/2930414124 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940ce2c0 con 0x7f369415c040 2023-06-10T18:57:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c074050 con 0x7f114c069e70 2023-06-10T18:57:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc989463700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc980011aa0 tx=0x7fc980011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3692ffd700 1 --2- 172.21.15.104:0/2930414124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f36940663f0 tx=0x7f3684009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7ffbd696ddd7d1b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3691ffb700 1 -- 172.21.15.104:0/2930414124 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368400b870 con 0x7f369415c040 2023-06-10T18:57:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000f940 con 0x7fc9700043e0 2023-06-10T18:57:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98000faa0 con 0x7fc9700043e0 2023-06-10T18:57:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9841549d0 con 0x7fc9700043e0 2023-06-10T18:57:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3691ffb700 1 -- 172.21.15.104:0/2930414124 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368400b9d0 con 0x7f369415c040 2023-06-10T18:57:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.950+0000 7f3691ffb700 1 -- 172.21.15.104:0/2930414124 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368400bd40 con 0x7f369415c040 2023-06-10T18:57:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc984154f90 con 0x7fc9700043e0 2023-06-10T18:57:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f113c022070 con 0x7f114c069e70 2023-06-10T18:57:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 -- 172.21.15.104:0/2930414124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 msgr2=0x7f36940ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 --2- 172.21.15.104:0/2930414124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f36940663f0 tx=0x7f3684009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 -- 172.21.15.104:0/2930414124 shutdown_connections 2023-06-10T18:57:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 --2- 172.21.15.104:0/2930414124 >> v2:172.21.15.104:3300/0 conn(0x7f3694070a20 0x7f3694070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 --2- 172.21.15.104:0/2930414124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ce780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 --2- 172.21.15.104:0/2930414124 >> v2:172.21.15.114:3300/0 conn(0x7f36940ae280 0x7f36940cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 -- 172.21.15.104:0/2930414124 >> 172.21.15.104:0/2930414124 conn(0x7f36940b1610 msgr2=0x7f36940ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 -- 172.21.15.104:0/2930414124 shutdown_connections 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f3699956700 1 -- 172.21.15.104:0/2930414124 wait complete. 2023-06-10T18:57:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f113affd700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 0x7f112c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.951+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f113c08aeb0 con 0x7f114c069e70 2023-06-10T18:57:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.952+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000fe50 con 0x7fc9700043e0 2023-06-10T18:57:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.952+0000 7f1149d9b700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 0x7f112c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.952+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9640052f0 con 0x7fc9700043e0 2023-06-10T18:57:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.953+0000 7f1149d9b700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 0x7f112c06d190 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f114c0ae790 tx=0x7f1140006cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.953+0000 7f3699956700 1 Processor -- start 2023-06-10T18:57:17.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.954+0000 7f3699956700 1 -- start start 2023-06-10T18:57:17.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.954+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc980014440 con 0x7fc9700043e0 2023-06-10T18:57:17.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.954+0000 7fc979ffb700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 0x7fc96806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.954+0000 7fc988c62700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 0x7fc96806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694157b00 con 0x7f36940ae280 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941539b0 con 0x7f3694070a20 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3699956700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3694158df0 con 0x7f369415c040 2023-06-10T18:57:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38408/0 (socket says 172.21.15.104:38408) 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:54140/0 (socket says 172.21.15.104:54140) 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 -- 172.21.15.104:0/135463452 learned_addr learned my addr 172.21.15.104:0/135463452 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36937fe700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 -- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 msgr2=0x7f36940ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 -- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 msgr2=0x7f36941586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 -- 172.21.15.104:0/135463452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3684009550 con 0x7f36940ae280 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f3692ffd700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.955+0000 7f36927fc700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f368800b530 tx=0x7f368800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.957+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc98005c150 con 0x7fc9700043e0 2023-06-10T18:57:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.957+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688011830 con 0x7f36940ae280 2023-06-10T18:57:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.957+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36941571d0 con 0x7f36940ae280 2023-06-10T18:57:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.957+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3694157790 con 0x7f36940ae280 2023-06-10T18:57:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.957+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f113c055eb0 con 0x7f114c069e70 2023-06-10T18:57:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.958+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694074050 con 0x7f36940ae280 2023-06-10T18:57:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.959+0000 7fc988c62700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 0x7fc96806d1e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fc9700047d0 tx=0x7fc97400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.962+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3688011990 con 0x7f36940ae280 2023-06-10T18:57:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.962+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688009380 con 0x7f36940ae280 2023-06-10T18:57:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.962+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc98008e060 con 0x7fc9700043e0 2023-06-10T18:57:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.963+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3688011d00 con 0x7f36940ae280 2023-06-10T18:57:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.964+0000 7f367bfff700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 0x7f368006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.964+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3688089b10 con 0x7f36940ae280 2023-06-10T18:57:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.964+0000 7f3692ffd700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 0x7f368006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.964+0000 7f3692ffd700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 0x7f368006d190 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3684000c00 tx=0x7f3684008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.967+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3688054ad0 con 0x7f36940ae280 2023-06-10T18:57:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.987+0000 7f4eb20b9700 1 Processor -- start 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.987+0000 7f4eb20b9700 1 -- start start 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eb20b9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4eac070a20 0x7f4eac06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eb20b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac158eb0 0x7f4eac159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eb20b9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac06ae00 con 0x7f4eac070df0 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eb20b9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac069e70 con 0x7f4eac070a20 2023-06-10T18:57:17.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eb20b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1595f0 con 0x7f4eac158eb0 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eab7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4eac070a20 0x7f4eac06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eab7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4eac070a20 0x7f4eac06a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38414/0 (socket says 172.21.15.104:38414) 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.988+0000 7f4eaaffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4eac070df0 0x7f4eac1587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36246/0 (socket says 172.21.15.104:36246) 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 -- 172.21.15.104:0/4144330960 learned_addr learned my addr 172.21.15.104:0/4144330960 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eabfff700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac158eb0 0x7f4eac159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 -- 172.21.15.104:0/4144330960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac158eb0 msgr2=0x7f4eac159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac158eb0 0x7f4eac159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:17.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 -- 172.21.15.104:0/4144330960 >> v1:172.21.15.104:6789/0 conn(0x7f4eac070df0 legacy=0x7f4eac1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eaa7fc700 1 -- 172.21.15.104:0/4144330960 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387612457 0 0) 0x7f4ea0001110 con 0x7f4eac070df0 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 -- 172.21.15.104:0/4144330960 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac1598d0 con 0x7f4eac070a20 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eab7fe700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070a20 0x7f4eac06a8c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0098f0 tx=0x7f4e9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd429473c73f3c7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eaa7fc700 1 -- 172.21.15.104:0/4144330960 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00b8a0 con 0x7f4eac070a20 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eaa7fc700 1 -- 172.21.15.104:0/4144330960 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c00ba00 con 0x7f4eac070a20 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.989+0000 7f4eaa7fc700 1 -- 172.21.15.104:0/4144330960 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00bd90 con 0x7f4eac070a20 2023-06-10T18:57:18.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 -- 172.21.15.104:0/4144330960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070a20 msgr2=0x7f4eac06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070a20 0x7f4eac06a8c0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0098f0 tx=0x7f4e9c009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 -- 172.21.15.104:0/4144330960 shutdown_connections 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070a20 0x7f4eac06a8c0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/4144330960 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac158eb0 0x7f4eac159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 -- 172.21.15.104:0/4144330960 >> 172.21.15.104:0/4144330960 conn(0x7f4eac0b1610 msgr2=0x7f4eac0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 -- 172.21.15.104:0/4144330960 shutdown_connections 2023-06-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.990+0000 7f4eb20b9700 1 -- 172.21.15.104:0/4144330960 wait complete. 2023-06-10T18:57:18.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.992+0000 7f4eb20b9700 1 Processor -- start 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.992+0000 7f4eb20b9700 1 -- start start 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac070a20 0x7f4eac143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 0x7f4eac0bdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b25b0 con 0x7f4eac070df0 2023-06-10T18:57:18.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0be3e0 con 0x7f4eac158eb0 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eb20b9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b2030 con 0x7f4eac070a20 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eaaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eaaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38422/0 (socket says 172.21.15.104:38422) 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.993+0000 7f4eaaffd700 1 -- 172.21.15.104:0/2335853039 learned_addr learned my addr 172.21.15.104:0/2335853039 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eab7fe700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac070a20 0x7f4eac143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eabfff700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 0x7f4eac0bdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 -- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac070a20 msgr2=0x7f4eac143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac070a20 0x7f4eac143e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 -- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 msgr2=0x7f4eac0bdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 0x7f4eac0bdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 -- 172.21.15.104:0/2335853039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c009550 con 0x7f4eac070df0 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eabfff700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 0x7f4eac0bdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eaaffd700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c520 tx=0x7f4ea000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0004c80 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea0004e00 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac0c2d10 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0007310 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.994+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac0c2a30 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.996+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac071780 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.996+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4ea0007470 con 0x7f4eac070df0 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.997+0000 7f4e93fff700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 0x7f4e9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.997+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4ea0089150 con 0x7f4eac070df0 2023-06-10T18:57:18.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.997+0000 7f4eab7fe700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 0x7f4e9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:17.997+0000 7f4eab7fe700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 0x7f4e9806d190 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009c40 tx=0x7f4e9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.001+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4ea0054040 con 0x7f4eac070df0 2023-06-10T18:57:18.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.254+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f3694157920 con 0x7f36940ae280 2023-06-10T18:57:18.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.271+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f114c0aec60 con 0x7f114c069e70 2023-06-10T18:57:18.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.276+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fc964005870 con 0x7fc9700043e0 2023-06-10T18:57:18.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.285+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f4eac0b1e90 con 0x7f4eac070df0 2023-06-10T18:57:18.700 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:57:18.694+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:57:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.696+0000 7f367bfff700 1 -- 172.21.15.104:0/135463452 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 9639 v89) v1 ==== 95+0+0 (secure 0 0 0) 0x7f36880580f0 con 0x7f36940ae280 2023-06-10T18:57:18.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.697+0000 7f113affd700 1 -- 172.21.15.104:0/3399496701 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 9642 v89) v1 ==== 95+0+0 (secure 0 0 0) 0x7f113c0594d0 con 0x7f114c069e70 2023-06-10T18:57:18.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.697+0000 7fc979ffb700 1 -- 172.21.15.104:0/560597526 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 9627 v89) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc98008e020 con 0x7fc9700043e0 2023-06-10T18:57:18.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.697+0000 7f4e93fff700 1 -- 172.21.15.104:0/2335853039 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 9630 v89) v1 ==== 95+0+0 (secure 0 0 0) 0x7f4eac0b1e90 con 0x7f4eac070df0 2023-06-10T18:57:18.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 msgr2=0x7f4e9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 0x7f4e9806d190 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009c40 tx=0x7f4e9c008030 comp rx=0 tx=0).stop 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 msgr2=0x7f4eac0bd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c520 tx=0x7f4ea000c830 comp rx=0 tx=0).stop 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 shutdown_connections 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e9806ace0 0x7f4e9806d190 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4eac158eb0 0x7f4eac0bdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4eac070a20 0x7f4eac143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 --2- 172.21.15.104:0/2335853039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4eac070df0 0x7f4eac0bd960 secure :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c520 tx=0x7f4ea000c830 comp rx=0 tx=0).stop 2023-06-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.699+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 >> 172.21.15.104:0/2335853039 conn(0x7f4eac0b1610 msgr2=0x7f4eac0aebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 shutdown_connections 2023-06-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f4eb20b9700 1 -- 172.21.15.104:0/2335853039 wait complete. 2023-06-10T18:57:18.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 msgr2=0x7fc96806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 0x7fc96806d1e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fc9700047d0 tx=0x7fc97400a000 comp rx=0 tx=0).stop 2023-06-10T18:57:18.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 msgr2=0x7fc9841501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc980011aa0 tx=0x7fc980011db0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 shutdown_connections 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc9840af8e0 0x7fc9840b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9700043e0 0x7fc9841501d0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc96806ad30 0x7fc96806d1e0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 --2- 172.21.15.104:0/560597526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc984155c60 0x7fc9840af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 >> 172.21.15.104:0/560597526 conn(0x7fc9840b1610 msgr2=0x7fc98415a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 msgr2=0x7f368006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 0x7f368006d190 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3684000c00 tx=0x7f3684008030 comp rx=0 tx=0).stop 2023-06-10T18:57:18.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 msgr2=0x7f36941529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 shutdown_connections 2023-06-10T18:57:18.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f368800b530 tx=0x7f368800b840 comp rx=0 tx=0).stop 2023-06-10T18:57:18.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7fc98b6c7700 1 -- 172.21.15.104:0/560597526 wait complete. 2023-06-10T18:57:18.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 shutdown_connections 2023-06-10T18:57:18.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f368006ace0 0x7f368006d190 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3694070a20 0x7f36941586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f369415c040 0x7f36940ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 --2- 172.21.15.104:0/135463452 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36940ae280 0x7f36941529b0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 >> 172.21.15.104:0/135463452 conn(0x7f36940b1610 msgr2=0x7f36940acbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 msgr2=0x7f112c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 0x7f112c06d190 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f114c0ae790 tx=0x7f1140006cb0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 msgr2=0x7f114c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f113c009b40 tx=0x7f113c004750 comp rx=0 tx=0).stop 2023-06-10T18:57:18.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.701+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 shutdown_connections 2023-06-10T18:57:18.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 shutdown_connections 2023-06-10T18:57:18.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f114c15a230 0x7f114c0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f114c069e70 0x7f114c1566b0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f112c06ace0 0x7f112c06d190 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 --2- 172.21.15.104:0/3399496701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f114c06d810 0x7f114c15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.701+0000 7f3699956700 1 -- 172.21.15.104:0/135463452 wait complete. 2023-06-10T18:57:18.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.700+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 >> 172.21.15.104:0/3399496701 conn(0x7f114c06b8a0 msgr2=0x7f114c0cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.701+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 shutdown_connections 2023-06-10T18:57:18.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.701+0000 7f1150f85700 1 -- 172.21.15.104:0/3399496701 wait complete. 2023-06-10T18:57:18.716 INFO:teuthology.orchestra.run.smithi104.stderr:failed mds gid 9630 2023-06-10T18:57:18.717 INFO:teuthology.orchestra.run.smithi104.stderr:failed mds gid 9627 2023-06-10T18:57:18.717 INFO:teuthology.orchestra.run.smithi104.stderr:failed mds gid 9642 2023-06-10T18:57:18.717 INFO:teuthology.orchestra.run.smithi104.stderr:failed mds gid 9639 2023-06-10T18:57:18.729 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:57:18.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:57:18.731 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:57:18.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:57:18.732 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:57:18.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:57:18.733 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:57:18.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:57:18.736 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:57:18.737 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:18.737 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:57:18.738 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:18.773 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:57:18.773 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:18.774 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:57:18.774 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:18.775 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:18.826 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:57:18.826 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:57:18.853 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:57:18.854 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:57:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 Processor -- start 2023-06-10T18:57:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 -- start start 2023-06-10T18:57:18.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e0c0d2b20 0x7f4e0c0d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0cd430 con 0x7f4e0c0d0520 2023-06-10T18:57:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0cd710 con 0x7f4e0c0ccfa0 2023-06-10T18:57:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.896+0000 7f4e11858700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0c0cd9f0 con 0x7f4e0c0d2b20 2023-06-10T18:57:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4e0c0d0520 0x7f4e0c0ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36284/0 (socket says 172.21.15.104:36284) 2023-06-10T18:57:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0b7fe700 1 -- 172.21.15.104:0/3819374844 learned_addr learned my addr 172.21.15.104:0/3819374844 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:18.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0affd700 1 --2- 172.21.15.104:0/3819374844 >> v2:172.21.15.114:3300/0 conn(0x7f4e0c0d2b20 0x7f4e0c0d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826180607 0 0) 0x7f4e0c0cd430 con 0x7f4e0c0d0520 2023-06-10T18:57:18.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df80035b0 con 0x7f4e0c0d0520 2023-06-10T18:57:18.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0affd700 1 -- 172.21.15.104:0/3819374844 >> v1:172.21.15.114:6789/0 conn(0x7f4e0c0ccfa0 legacy=0x7f4e0c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911296828 0 0) 0x7f4e0c0cd710 con 0x7f4e0c0ccfa0 2023-06-10T18:57:18.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0affd700 1 -- 172.21.15.104:0/3819374844 >> v1:172.21.15.104:6789/0 conn(0x7f4e0c0d0520 legacy=0x7f4e0c0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0affd700 1 -- 172.21.15.104:0/3819374844 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0cdcd0 con 0x7f4e0c0d2b20 2023-06-10T18:57:18.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e0affd700 1 --2- 172.21.15.104:0/3819374844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0d2b20 0x7f4e0c0d2f00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009d90 tx=0x7f4e0400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c6c9a2a4a82e4d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400e070 con 0x7f4e0c0d2b20 2023-06-10T18:57:18.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04003a20 con 0x7f4e0c0d2b20 2023-06-10T18:57:18.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.897+0000 7f4e09ffb700 1 -- 172.21.15.104:0/3819374844 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc001670 con 0x7f4e0c0d2b20 2023-06-10T18:57:18.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 -- 172.21.15.104:0/3819374844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0d2b20 msgr2=0x7f4e0c0d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 --2- 172.21.15.104:0/3819374844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0d2b20 0x7f4e0c0d2f00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009d90 tx=0x7f4e0400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:18.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 -- 172.21.15.104:0/3819374844 shutdown_connections 2023-06-10T18:57:18.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 --2- 172.21.15.104:0/3819374844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0d2b20 0x7f4e0c0d2f00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 -- 172.21.15.104:0/3819374844 >> 172.21.15.104:0/3819374844 conn(0x7f4e0c071ec0 msgr2=0x7f4e0c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:18.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 -- 172.21.15.104:0/3819374844 shutdown_connections 2023-06-10T18:57:18.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.898+0000 7f4e11858700 1 -- 172.21.15.104:0/3819374844 wait complete. 2023-06-10T18:57:18.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.900+0000 7f4e11858700 1 Processor -- start 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.900+0000 7f4e11858700 1 -- start start 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0ccfa0 0x7f4e0c0b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e0c158470 0x7f4e0c15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c15eef0 con 0x7f4e0c0d0520 2023-06-10T18:57:18.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c0b3ff0 con 0x7f4e0c0ccfa0 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c15e830 con 0x7f4e0c158470 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38460/0 (socket says 172.21.15.104:38460) 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 -- 172.21.15.104:0/1158314443 learned_addr learned my addr 172.21.15.104:0/1158314443 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0affd700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0ccfa0 0x7f4e0c0b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0b7fe700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e0c158470 0x7f4e0c15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 -- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e0c158470 msgr2=0x7f4e0c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e0c158470 0x7f4e0c15d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 -- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0ccfa0 msgr2=0x7f4e0c0b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:18.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0ccfa0 0x7f4e0c0b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 -- 172.21.15.104:0/1158314443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04009a70 con 0x7f4e0c0d0520 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e0a7fc700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f4df40011e0 tx=0x7f4df400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40054e0 con 0x7f4e0c0d0520 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4003680 con 0x7f4e0c0d0520 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c15eb10 con 0x7f4e0c0d0520 2023-06-10T18:57:18.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4003a30 con 0x7f4e0c0d0520 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.901+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c15e110 con 0x7f4e0c0d0520 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.902+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c15eca0 con 0x7f4e0c0d0520 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.903+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4df4003b90 con 0x7f4e0c0d0520 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.903+0000 7f4e03fff700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 0x7f4dec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.903+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4df4088b90 con 0x7f4e0c0d0520 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.903+0000 7f4e0affd700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 0x7f4dec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:18.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.904+0000 7f4e0affd700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 0x7f4dec06d190 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f80 tx=0x7f4e04010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:18.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:18.908+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4df4053aa0 con 0x7f4e0c0d0520 2023-06-10T18:57:19.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.136+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0c074050 con 0x7f4e0c0d0520 2023-06-10T18:57:19.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.137+0000 7f4e03fff700 1 -- 172.21.15.104:0/1158314443 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+1639 (secure 0 0 0) 0x7f4df4017020 con 0x7f4e0c0d0520 2023-06-10T18:57:19.144 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:19.144 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:18.695476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:19.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 msgr2=0x7f4dec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:19.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 0x7f4dec06d190 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f80 tx=0x7f4e04010000 comp rx=0 tx=0).stop 2023-06-10T18:57:19.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 msgr2=0x7f4e0c15f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:19.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f4df40011e0 tx=0x7f4df400cae0 comp rx=0 tx=0).stop 2023-06-10T18:57:19.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 shutdown_connections 2023-06-10T18:57:19.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4dec06ace0 0x7f4dec06d190 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:19.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e0c0ccfa0 0x7f4e0c0b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:19.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e0c158470 0x7f4e0c15d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:19.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 --2- 172.21.15.104:0/1158314443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e0c0d0520 0x7f4e0c15f360 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:19.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 >> 172.21.15.104:0/1158314443 conn(0x7f4e0c071ec0 msgr2=0x7f4e0c06b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:19.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 shutdown_connections 2023-06-10T18:57:19.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:19.139+0000 7f4e11858700 1 -- 172.21.15.104:0/1158314443 wait complete. 2023-06-10T18:57:19.149 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 89 2023-06-10T18:57:19.157 DEBUG:tasks.cephfs.filesystem:mapped states [] to None 2023-06-10T18:57:20.158 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.284+0000 7eff24e4e700 1 Processor -- start 2023-06-10T18:57:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.284+0000 7eff24e4e700 1 -- start start 2023-06-10T18:57:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff200b1e30 0x7eff200d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff2006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff2006b820 0x7eff200cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff200d0ea0 con 0x7eff200b1e30 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2006bfd0 con 0x7eff2006b820 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.285+0000 7eff24e4e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2006c2b0 con 0x7eff200d0a30 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff200b1e30 0x7eff200d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff200b1e30 0x7eff200d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38484/0 (socket says 172.21.15.104:38484) 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 -- 172.21.15.104:0/559554946 learned_addr learned my addr 172.21.15.104:0/559554946 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1ed9d700 1 --2- 172.21.15.104:0/559554946 >> v2:172.21.15.114:3300/0 conn(0x7eff2006b820 0x7eff200cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1dd9b700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff2006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 -- 172.21.15.104:0/559554946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 msgr2=0x7eff2006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff2006b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 -- 172.21.15.104:0/559554946 >> v2:172.21.15.114:3300/0 conn(0x7eff2006b820 msgr2=0x7eff200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 --2- 172.21.15.104:0/559554946 >> v2:172.21.15.114:3300/0 conn(0x7eff2006b820 0x7eff200cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 -- 172.21.15.104:0/559554946 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff200ccf70 con 0x7eff200b1e30 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1e59c700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff200d04f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7eff2008b0f0 tx=0x7eff10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157053602d22d30a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1d59a700 1 -- 172.21.15.104:0/559554946 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10011830 con 0x7eff200b1e30 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1d59a700 1 -- 172.21.15.104:0/559554946 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10011990 con 0x7eff200b1e30 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.286+0000 7eff1d59a700 1 -- 172.21.15.104:0/559554946 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10011d20 con 0x7eff200b1e30 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 -- 172.21.15.104:0/559554946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 msgr2=0x7eff200d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff200d04f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7eff2008b0f0 tx=0x7eff10009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 -- 172.21.15.104:0/559554946 shutdown_connections 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff200d04f0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 --2- 172.21.15.104:0/559554946 >> v2:172.21.15.114:3300/0 conn(0x7eff2006b820 0x7eff200cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 --2- 172.21.15.104:0/559554946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff2006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 -- 172.21.15.104:0/559554946 >> 172.21.15.104:0/559554946 conn(0x7eff20071ec0 msgr2=0x7eff2009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:20.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 -- 172.21.15.104:0/559554946 shutdown_connections 2023-06-10T18:57:20.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.287+0000 7eff24e4e700 1 -- 172.21.15.104:0/559554946 wait complete. 2023-06-10T18:57:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.288+0000 7eff24e4e700 1 Processor -- start 2023-06-10T18:57:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.288+0000 7eff24e4e700 1 -- start start 2023-06-10T18:57:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff2006b820 0x7eff2015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff200b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff201e7e30 con 0x7eff200b1e30 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2015b8d0 con 0x7eff2006b820 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff24e4e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff201e7220 con 0x7eff200d0a30 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38492/0 (socket says 172.21.15.104:38492) 2023-06-10T18:57:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 -- 172.21.15.104:0/2355573403 learned_addr learned my addr 172.21.15.104:0/2355573403 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1e59c700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff2006b820 0x7eff2015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1ed9d700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff200b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 -- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 msgr2=0x7eff200b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff200b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 -- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff2006b820 msgr2=0x7eff2015b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff2006b820 0x7eff2015b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.289+0000 7eff1dd9b700 1 -- 172.21.15.104:0/2355573403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10009550 con 0x7eff200b1e30 2023-06-10T18:57:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff1dd9b700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7eff1400d800 tx=0x7eff1400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140125f0 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14012750 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff201e7500 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14012b00 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.290+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff201e7ac0 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.291+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff200d1860 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.291+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff14012c60 con 0x7eff200b1e30 2023-06-10T18:57:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.292+0000 7eff0effd700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.292+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7eff14089650 con 0x7eff200b1e30 2023-06-10T18:57:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.292+0000 7eff1e59c700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.292+0000 7eff1e59c700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7eff10014040 tx=0x7eff10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.296+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff14054560 con 0x7eff200b1e30 2023-06-10T18:57:20.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.524+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff200c9b80 con 0x7eff200b1e30 2023-06-10T18:57:20.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.525+0000 7eff0effd700 1 -- 172.21.15.104:0/2355573403 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4681 (secure 0 0 0) 0x7eff200c9b80 con 0x7eff200b1e30 2023-06-10T18:57:20.531 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:20.532 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:19.707668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 msgr2=0x7eff0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7eff10014040 tx=0x7eff10008030 comp rx=0 tx=0).stop 2023-06-10T18:57:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 msgr2=0x7eff201e4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:20.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7eff1400d800 tx=0x7eff1400db10 comp rx=0 tx=0).stop 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 shutdown_connections 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff0006ace0 0x7eff0006d190 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff2006b820 0x7eff2015b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff200d0a30 0x7eff200b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 --2- 172.21.15.104:0/2355573403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff200b1e30 0x7eff201e4bb0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 >> 172.21.15.104:0/2355573403 conn(0x7eff20071ec0 msgr2=0x7eff200b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:20.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 shutdown_connections 2023-06-10T18:57:20.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:20.527+0000 7eff24e4e700 1 -- 172.21.15.104:0/2355573403 wait complete. 2023-06-10T18:57:20.535 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 91 2023-06-10T18:57:20.545 DEBUG:tasks.cephfs.filesystem:mapped states ['up:replay'] to None 2023-06-10T18:57:21.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:21.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.669+0000 7f0d4b594700 1 Processor -- start 2023-06-10T18:57:21.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.669+0000 7f0d4b594700 1 -- start start 2023-06-10T18:57:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d4b594700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d4b594700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d44069e70 0x7f0d440ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d4b594700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4406d290 con 0x7f0d4406e0e0 2023-06-10T18:57:21.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d4b594700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d44071640 con 0x7f0d44069e70 2023-06-10T18:57:21.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d4b594700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d440aef00 con 0x7f0d4406e4b0 2023-06-10T18:57:21.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d48b2f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d49b31700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d44069e70 0x7f0d440ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d48b2f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:57890/0 (socket says 172.21.15.104:57890) 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d49b31700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d44069e70 0x7f0d440ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:54242/0 (socket says 172.21.15.104:54242) 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d48b2f700 1 -- 172.21.15.104:0/1306685559 learned_addr learned my addr 172.21.15.104:0/1306685559 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.670+0000 7f0d49330700 1 --1- 172.21.15.104:0/1306685559 >> v1:172.21.15.114:6789/0 conn(0x7f0d4406e0e0 0x7f0d440c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:42178/0 (socket says 172.21.15.104:42178) 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d49b31700 1 -- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 msgr2=0x7f0d4406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:21.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d49b31700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d49b31700 1 -- 172.21.15.104:0/1306685559 >> v1:172.21.15.114:6789/0 conn(0x7f0d4406e0e0 legacy=0x7f0d440c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d49b31700 1 -- 172.21.15.104:0/1306685559 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d440ca2d0 con 0x7f0d44069e70 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d3bfff700 1 -- 172.21.15.104:0/1306685559 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688901447 0 0) 0x7f0d4406d290 con 0x7f0d4406e0e0 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d48b2f700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d49b31700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d440ce9f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009d90 tx=0x7f0d3c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e70ad40953bcfac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:21.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d3bfff700 1 -- 172.21.15.104:0/1306685559 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00e070 con 0x7f0d44069e70 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d3bfff700 1 -- 172.21.15.104:0/1306685559 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c00bb20 con 0x7f0d44069e70 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d3bfff700 1 -- 172.21.15.104:0/1306685559 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c012430 con 0x7f0d44069e70 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d4b594700 1 -- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 msgr2=0x7f0d440ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d4b594700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d440ce9f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009d90 tx=0x7f0d3c00b330 comp rx=0 tx=0).stop 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.671+0000 7f0d4b594700 1 -- 172.21.15.104:0/1306685559 shutdown_connections 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.672+0000 7f0d4b594700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d440ce9f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.672+0000 7f0d4b594700 1 --2- 172.21.15.104:0/1306685559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d4406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.672+0000 7f0d4b594700 1 -- 172.21.15.104:0/1306685559 >> 172.21.15.104:0/1306685559 conn(0x7f0d4406b8a0 msgr2=0x7f0d440cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.672+0000 7f0d4b594700 1 -- 172.21.15.104:0/1306685559 shutdown_connections 2023-06-10T18:57:21.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.672+0000 7f0d4b594700 1 -- 172.21.15.104:0/1306685559 wait complete. 2023-06-10T18:57:21.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.673+0000 7f0d4b594700 1 Processor -- start 2023-06-10T18:57:21.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.673+0000 7f0d4b594700 1 -- start start 2023-06-10T18:57:21.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.673+0000 7f0d4b594700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d44152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.673+0000 7f0d4b594700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.673+0000 7f0d4b594700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d44159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d4b594700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d440b09b0 con 0x7f0d4406e0e0 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d4b594700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d44153230 con 0x7f0d44069e70 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d4b594700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d440b0110 con 0x7f0d4406e4b0 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38512/0 (socket says 172.21.15.104:38512) 2023-06-10T18:57:21.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 -- 172.21.15.104:0/4279769365 learned_addr learned my addr 172.21.15.104:0/4279769365 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d49330700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d44152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d49b31700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d44159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 -- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 msgr2=0x7f0d44159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d44159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 -- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 msgr2=0x7f0d44152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d44152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 -- 172.21.15.104:0/4279769365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34009550 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d48b2f700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400ebe0 tx=0x7f0d3400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400c850 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3400c9b0 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c009a70 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.674+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400cd40 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.675+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d440b06d0 con 0x7f0d4406e0e0 2023-06-10T18:57:21.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.676+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d44074050 con 0x7f0d4406e0e0 2023-06-10T18:57:21.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.676+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0d34013470 con 0x7f0d4406e0e0 2023-06-10T18:57:21.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.677+0000 7f0d39ffb700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 0x7f0d2806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:21.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.677+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f0d34089f70 con 0x7f0d4406e0e0 2023-06-10T18:57:21.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.677+0000 7f0d49330700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 0x7f0d2806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:21.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.677+0000 7f0d49330700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 0x7f0d2806d1c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0d40000fb0 tx=0x7f0d40000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:21.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.680+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0d34054ed0 con 0x7f0d4406e0e0 2023-06-10T18:57:21.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.914+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d440adef0 con 0x7f0d4406e0e0 2023-06-10T18:57:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.914+0000 7f0d39ffb700 1 -- 172.21.15.104:0/4279769365 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f0d34017090 con 0x7f0d4406e0e0 2023-06-10T18:57:21.921 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:21.922 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:21.711299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:21.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 msgr2=0x7f0d2806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 0x7f0d2806d1c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0d40000fb0 tx=0x7f0d40000ed0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 msgr2=0x7f0d44153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400ebe0 tx=0x7f0d3400eef0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 shutdown_connections 2023-06-10T18:57:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d2806ad10 0x7f0d2806d1c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d44069e70 0x7f0d44152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4406e4b0 0x7f0d44159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 --2- 172.21.15.104:0/4279769365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d4406e0e0 0x7f0d44153f70 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 >> 172.21.15.104:0/4279769365 conn(0x7f0d4406b8a0 msgr2=0x7f0d440cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 shutdown_connections 2023-06-10T18:57:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:21.916+0000 7f0d4b594700 1 -- 172.21.15.104:0/4279769365 wait complete. 2023-06-10T18:57:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 93 2023-06-10T18:57:21.934 DEBUG:tasks.cephfs.filesystem:mapped states ['up:rejoin'] to None 2023-06-10T18:57:22.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:23.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.059+0000 7f2e17cfb700 1 Processor -- start 2023-06-10T18:57:23.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.059+0000 7f2e17cfb700 1 -- start start 2023-06-10T18:57:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e17cfb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2e1006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e17cfb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1006e5e0 con 0x7f2e100b1070 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e17cfb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e100ae7e0 con 0x7f2e100aec00 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e17cfb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1006df90 con 0x7f2e1006ea70 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e15a97700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2e100b1070 0x7f2e100b1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36348/0 (socket says 172.21.15.104:36348) 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.060+0000 7f2e15a97700 1 -- 172.21.15.104:0/3157266544 learned_addr learned my addr 172.21.15.104:0/3157266544 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942071042 0 0) 0x7f2e1006e5e0 con 0x7f2e100b1070 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dfc0035b0 con 0x7f2e100b1070 2023-06-10T18:57:23.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2e1006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2196438453 0 0) 0x7f2dfc0035b0 con 0x7f2e100b1070 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 msgr2=0x7f2e1006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2e1006cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 >> v1:172.21.15.114:6789/0 conn(0x7f2e100aec00 legacy=0x7f2e1006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1006e270 con 0x7f2e100b1070 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2e000017f0 con 0x7f2e100b1070 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 >> v1:172.21.15.104:6789/0 conn(0x7f2e100b1070 legacy=0x7f2e100b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2dfc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2dfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dfc004d90 0x7f2dfc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1006df90 con 0x7f2dfc003fb0 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc00bba0 con 0x7f2dfc004d90 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc00df70 con 0x7f2e1006ea70 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2dfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15a97700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2dfc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e16298700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dfc004d90 0x7f2dfc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 -- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 msgr2=0x7f2dfc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e1006ea70 0x7f2dfc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 -- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dfc004d90 msgr2=0x7f2dfc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dfc004d90 0x7f2dfc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.061+0000 7f2e15296700 1 -- 172.21.15.104:0/3157266544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0035b0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e15296700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2dfc004850 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2e1006ee60 tx=0x7f2dfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07e230f706a87be server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c00e6c0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c00e820 con 0x7f2dfc003fb0 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e14a95700 1 -- 172.21.15.104:0/3157266544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c00ebb0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 -- 172.21.15.104:0/3157266544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 msgr2=0x7f2dfc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/3157266544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2dfc004850 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2e1006ee60 tx=0x7f2dfc005880 comp rx=0 tx=0).stop 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e16298700 1 -- 172.21.15.104:0/3157266544 reap_dead start 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 -- 172.21.15.104:0/3157266544 shutdown_connections 2023-06-10T18:57:23.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 -- 172.21.15.104:0/3157266544 >> 172.21.15.104:0/3157266544 conn(0x7f2e1006b8a0 msgr2=0x7f2e100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:23.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 -- 172.21.15.104:0/3157266544 shutdown_connections 2023-06-10T18:57:23.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.062+0000 7f2e17cfb700 1 -- 172.21.15.104:0/3157266544 wait complete. 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 Processor -- start 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 -- start start 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e10155b60 0x7f2e101dfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 0x7f2e100b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e101e0f10 con 0x7f2dfc003fb0 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100b4fd0 con 0x7f2e100b45f0 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e17cfb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e101e0390 con 0x7f2e10155b60 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e15a97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e15a97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38532/0 (socket says 172.21.15.104:38532) 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.064+0000 7f2e15a97700 1 -- 172.21.15.104:0/1457423785 learned_addr learned my addr 172.21.15.104:0/1457423785 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e16298700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 0x7f2e100b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15296700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e10155b60 0x7f2e101dfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 -- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e10155b60 msgr2=0x7f2e101dfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e10155b60 0x7f2e101dfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 -- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 msgr2=0x7f2e100b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 0x7f2e100b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 -- 172.21.15.104:0/1457423785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e101e0670 con 0x7f2dfc003fb0 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e16298700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 0x7f2e100b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:23.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e15a97700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f2e00007fd0 tx=0x7f2e0000fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00001760 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e000072e0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e101e0950 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00007690 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.065+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e101e00a0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.066+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e10066f10 con 0x7f2dfc003fb0 2023-06-10T18:57:23.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.066+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2e00005830 con 0x7f2dfc003fb0 2023-06-10T18:57:23.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.067+0000 7f2e067fc700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 0x7f2df406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.067+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f2e000894d0 con 0x7f2dfc003fb0 2023-06-10T18:57:23.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.067+0000 7f2e15296700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 0x7f2df406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.068+0000 7f2e15296700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 0x7f2df406d190 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0090b0 tx=0x7f2e0c01f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.071+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2e00057f00 con 0x7f2dfc003fb0 2023-06-10T18:57:23.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.301+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e10074050 con 0x7f2dfc003fb0 2023-06-10T18:57:23.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.302+0000 7f2e067fc700 1 -- 172.21.15.104:0/1457423785 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f2e00057a90 con 0x7f2dfc003fb0 2023-06-10T18:57:23.308 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:23.309 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:22.712458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":4,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:23.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 msgr2=0x7f2df406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 0x7f2df406d190 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0090b0 tx=0x7f2e0c01f040 comp rx=0 tx=0).stop 2023-06-10T18:57:23.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 msgr2=0x7f2e101e2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f2e00007fd0 tx=0x7f2e0000fda0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 shutdown_connections 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2df406ace0 0x7f2df406d190 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e100b45f0 0x7f2e100b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dfc003fb0 0x7f2e101e2d10 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 --2- 172.21.15.104:0/1457423785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2e10155b60 0x7f2e101dfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 >> 172.21.15.104:0/1457423785 conn(0x7f2e1006b8a0 msgr2=0x7f2e101586e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:23.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 shutdown_connections 2023-06-10T18:57:23.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.304+0000 7f2e17cfb700 1 -- 172.21.15.104:0/1457423785 wait complete. 2023-06-10T18:57:23.312 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 94 2023-06-10T18:57:23.322 DEBUG:tasks.cephfs.filesystem:mapped states ['up:active'] to up:active 2023-06-10T18:57:23.322 DEBUG:tasks.cephfs.filesystem:reached state 'up:active' in 4.547428607940674s 2023-06-10T18:57:23.322 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:23.323 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:23.344 INFO:tasks.cephfs.mount:check lock on file background_file 2023-06-10T18:57:23.345 DEBUG:teuthology.orchestra.run.smithi114:> python3 -c ' 2023-06-10T18:57:23.345 DEBUG:teuthology.orchestra.run.smithi114:> import fcntl 2023-06-10T18:57:23.345 DEBUG:teuthology.orchestra.run.smithi114:> import errno 2023-06-10T18:57:23.346 DEBUG:teuthology.orchestra.run.smithi114:> import struct 2023-06-10T18:57:23.346 DEBUG:teuthology.orchestra.run.smithi114:> f1 = open("/home/ubuntu/cephtest/mnt.1/background_file-1", '"'"'r'"'"') 2023-06-10T18:57:23.346 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:23.346 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB) 2023-06-10T18:57:23.347 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:23.347 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:23.347 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:23.348 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:23.348 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("flock on file /home/ubuntu/cephtest/mnt.1/background_file-1 not found") 2023-06-10T18:57:23.348 DEBUG:teuthology.orchestra.run.smithi114:> f2 = open("/home/ubuntu/cephtest/mnt.1/background_file-2", '"'"'r'"'"') 2023-06-10T18:57:23.348 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:23.349 DEBUG:teuthology.orchestra.run.smithi114:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:23.349 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:23.349 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:23.349 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:23.350 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:23.350 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:23.350 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("posix lock on file /home/ubuntu/cephtest/mnt.1/background_file-2 not found") 2023-06-10T18:57:23.351 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:23.442 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:57:23.443 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:23.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:23.567 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:57:23.568 INFO:tasks.cephfs.mount:Terminating background process 2023-06-10T18:57:23.568 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:23.569 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:23.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:23.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.741+0000 7f0962c57700 1 Processor -- start 2023-06-10T18:57:23.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.741+0000 7f0962c57700 1 -- start start 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f095c0d2b20 0x7f095c0d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f095c0ccfa0 0x7f095c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c0b2780 0x7f095c0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c0cd430 con 0x7f095c0d2b20 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c0cd710 con 0x7f095c0ccfa0 2023-06-10T18:57:23.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f0962c57700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0cd9f0 con 0x7f095c0b2780 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f09609f3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f095c0d2b20 0x7f095c0d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f09609f3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f095c0d2b20 0x7f095c0d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38540/0 (socket says 172.21.15.104:38540) 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.742+0000 7f09609f3700 1 -- 172.21.15.104:0/832074231 learned_addr learned my addr 172.21.15.104:0/832074231 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 -- 172.21.15.104:0/832074231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c0b2780 msgr2=0x7f095c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 --2- 172.21.15.104:0/832074231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c0b2780 0x7f095c0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 -- 172.21.15.104:0/832074231 >> v2:172.21.15.114:3300/0 conn(0x7f095c0ccfa0 msgr2=0x7f095c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:23.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 --2- 172.21.15.104:0/832074231 >> v2:172.21.15.114:3300/0 conn(0x7f095c0ccfa0 0x7f095c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 -- 172.21.15.104:0/832074231 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c0cdcd0 con 0x7f095c0d2b20 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f09609f3700 1 --2- 172.21.15.104:0/832074231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0d2b20 0x7f095c0d2f00 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f095c067b00 tx=0x7f0950009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e43de953abe059 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f095b7fe700 1 -- 172.21.15.104:0/832074231 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000b870 con 0x7f095c0d2b20 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f095b7fe700 1 -- 172.21.15.104:0/832074231 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000b9d0 con 0x7f095c0d2b20 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f095b7fe700 1 -- 172.21.15.104:0/832074231 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000bd60 con 0x7f095c0d2b20 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 -- 172.21.15.104:0/832074231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0d2b20 msgr2=0x7f095c0d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 --2- 172.21.15.104:0/832074231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0d2b20 0x7f095c0d2f00 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f095c067b00 tx=0x7f0950009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 -- 172.21.15.104:0/832074231 shutdown_connections 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 --2- 172.21.15.104:0/832074231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0d2b20 0x7f095c0d2f00 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 --2- 172.21.15.104:0/832074231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c0b2780 0x7f095c0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.743+0000 7f0962c57700 1 --2- 172.21.15.104:0/832074231 >> v2:172.21.15.114:3300/0 conn(0x7f095c0ccfa0 0x7f095c0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.744+0000 7f0962c57700 1 -- 172.21.15.104:0/832074231 >> 172.21.15.104:0/832074231 conn(0x7f095c071ec0 msgr2=0x7f095c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.744+0000 7f0962c57700 1 -- 172.21.15.104:0/832074231 shutdown_connections 2023-06-10T18:57:23.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.744+0000 7f0962c57700 1 -- 172.21.15.104:0/832074231 wait complete. 2023-06-10T18:57:23.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.745+0000 7f0962c57700 1 Processor -- start 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.745+0000 7f0962c57700 1 -- start start 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f095c0ccfa0 0x7f095c0c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 0x7f095c0b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c159000 con 0x7f095c0b2780 2023-06-10T18:57:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c15ed10 con 0x7f095c0ccfa0 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c15eff0 con 0x7f095c158420 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38546/0 (socket says 172.21.15.104:38546) 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 -- 172.21.15.104:0/1822969413 learned_addr learned my addr 172.21.15.104:0/1822969413 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f095bfff700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f095c0ccfa0 0x7f095c0c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09611f4700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 0x7f095c0b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 -- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 msgr2=0x7f095c0b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 0x7f095c0b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 -- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f095c0ccfa0 msgr2=0x7f095c0c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f095c0ccfa0 0x7f095c0c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 -- 172.21.15.104:0/1822969413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950009550 con 0x7f095c0b2780 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09611f4700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 0x7f095c0b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09609f3700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f0950009130 tx=0x7f0950004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950004520 con 0x7f095c0b2780 2023-06-10T18:57:23.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095001e070 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500038e0 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c15f2d0 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.746+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c0c6350 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.747+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c074050 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.748+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0950003a40 con 0x7f095c0b2780 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.748+0000 7f09597fa700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.748+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f0950020070 con 0x7f095c0b2780 2023-06-10T18:57:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.749+0000 7f095bfff700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.749+0000 7f095bfff700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0948006fd0 tx=0x7f0948008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.753+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0950055c10 con 0x7f095c0b2780 2023-06-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.941+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f095c06cbd0 con 0x7f095c0b2780 2023-06-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.942+0000 7f09597fa700 1 -- 172.21.15.104:0/1822969413 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18745 (secure 0 0 0) 0x7f0950015090 con 0x7f095c0b2780 2023-06-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":147,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:18.287462+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2371761194":"2023-06-11T18:57:18.273257+0000","172.21.15.114:6835/2371761194":"2023-06-11T18:57:18.273257+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.944+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 msgr2=0x7f094006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.944+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0948006fd0 tx=0x7f0948008040 comp rx=0 tx=0).stop 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 msgr2=0x7f095c15e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f0950009130 tx=0x7f0950004750 comp rx=0 tx=0).stop 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 shutdown_connections 2023-06-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f095c158420 0x7f095c0b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f095c0b2780 0x7f095c15e6f0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f094006ace0 0x7f094006d190 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 --2- 172.21.15.104:0/1822969413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f095c0ccfa0 0x7f095c0c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 >> 172.21.15.104:0/1822969413 conn(0x7f095c071ec0 msgr2=0x7f095c0d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 shutdown_connections 2023-06-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:23.945+0000 7f0962c57700 1 -- 172.21.15.104:0/1822969413 wait complete. 2023-06-10T18:57:23.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:04.108341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:05.238053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:57:23.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:57:24.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.080+0000 7fc81706d700 1 Processor -- start 2023-06-10T18:57:24.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.080+0000 7fc81706d700 1 -- start start 2023-06-10T18:57:24.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc8101424d0 0x7fc8100c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc81006d3b0 0x7fc81006d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006dd70 con 0x7fc8101424d0 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8100cd7c0 con 0x7fc81006d3b0 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc81706d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100cdaa0 con 0x7fc810142980 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc814e09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc8101424d0 0x7fc8100c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc814e09700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc8101424d0 0x7fc8100c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38554/0 (socket says 172.21.15.104:38554) 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.081+0000 7fc814e09700 1 -- 172.21.15.104:0/3491321732 learned_addr learned my addr 172.21.15.104:0/3491321732 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 -- 172.21.15.104:0/3491321732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 msgr2=0x7fc8100ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:24.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 --2- 172.21.15.104:0/3491321732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 -- 172.21.15.104:0/3491321732 >> v2:172.21.15.114:3300/0 conn(0x7fc81006d3b0 msgr2=0x7fc81006d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 --2- 172.21.15.104:0/3491321732 >> v2:172.21.15.114:3300/0 conn(0x7fc81006d3b0 0x7fc81006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 -- 172.21.15.104:0/3491321732 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8100cdd80 con 0x7fc8101424d0 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc814e09700 1 --2- 172.21.15.104:0/3491321732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc8100c55c0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fc8100cb8f0 tx=0x7fc804009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd989661a313121f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc80f7fe700 1 -- 172.21.15.104:0/3491321732 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80400b870 con 0x7fc8101424d0 2023-06-10T18:57:24.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc80f7fe700 1 -- 172.21.15.104:0/3491321732 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80400b9d0 con 0x7fc8101424d0 2023-06-10T18:57:24.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc80f7fe700 1 -- 172.21.15.104:0/3491321732 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80400bd60 con 0x7fc8101424d0 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc81706d700 1 -- 172.21.15.104:0/3491321732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 msgr2=0x7fc8100c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc81706d700 1 --2- 172.21.15.104:0/3491321732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc8100c55c0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fc8100cb8f0 tx=0x7fc804009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc81706d700 1 -- 172.21.15.104:0/3491321732 shutdown_connections 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.082+0000 7fc81706d700 1 --2- 172.21.15.104:0/3491321732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc8100c55c0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.083+0000 7fc81706d700 1 --2- 172.21.15.104:0/3491321732 >> v2:172.21.15.114:3300/0 conn(0x7fc81006d3b0 0x7fc81006d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.083+0000 7fc81706d700 1 --2- 172.21.15.104:0/3491321732 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.083+0000 7fc81706d700 1 -- 172.21.15.104:0/3491321732 >> 172.21.15.104:0/3491321732 conn(0x7fc8100ceec0 msgr2=0x7fc8100cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.083+0000 7fc81706d700 1 -- 172.21.15.104:0/3491321732 shutdown_connections 2023-06-10T18:57:24.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.083+0000 7fc81706d700 1 -- 172.21.15.104:0/3491321732 wait complete. 2023-06-10T18:57:24.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.084+0000 7fc81706d700 1 Processor -- start 2023-06-10T18:57:24.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.084+0000 7fc81706d700 1 -- start start 2023-06-10T18:57:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc81006d3b0 0x7fc8100b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b27d0 con 0x7fc8101424d0 2023-06-10T18:57:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b2ab0 con 0x7fc81006d3b0 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b2d90 con 0x7fc810142980 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38562/0 (socket says 172.21.15.104:38562) 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 -- 172.21.15.104:0/2372317233 learned_addr learned my addr 172.21.15.104:0/2372317233 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc814e09700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc81006d3b0 0x7fc8100b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81560a700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 -- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 msgr2=0x7fc8100b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 -- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc81006d3b0 msgr2=0x7fc8100b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc81006d3b0 0x7fc8100b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 -- 172.21.15.104:0/2372317233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009550 con 0x7fc8101424d0 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80ffff700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b530 tx=0x7fc7fc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc011830 con 0x7fc8101424d0 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc011990 con 0x7fc8101424d0 2023-06-10T18:57:24.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8100b3070 con 0x7fc8101424d0 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc011d40 con 0x7fc8101424d0 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.085+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc810158160 con 0x7fc8101424d0 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.086+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81006db90 con 0x7fc8101424d0 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.087+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc7fc01a400 con 0x7fc8101424d0 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.087+0000 7fc80d7fa700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 0x7fc7f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.087+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fc7fc089630 con 0x7fc8101424d0 2023-06-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.087+0000 7fc814e09700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 0x7fc7f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.088+0000 7fc814e09700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 0x7fc7f406d1c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc804014040 tx=0x7fc804008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.091+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc7fc054570 con 0x7fc8101424d0 2023-06-10T18:57:24.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.277+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc810074050 con 0x7fc8101424d0 2023-06-10T18:57:24.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.277+0000 7fc80d7fa700 1 -- 172.21.15.104:0/2372317233 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7fc0150a0 con 0x7fc8101424d0 2023-06-10T18:57:24.284 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:57:24.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 msgr2=0x7fc7f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 0x7fc7f406d1c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc804014040 tx=0x7fc804008030 comp rx=0 tx=0).stop 2023-06-10T18:57:24.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 msgr2=0x7fc810157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b530 tx=0x7fc7fc00b840 comp rx=0 tx=0).stop 2023-06-10T18:57:24.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 shutdown_connections 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc7f406ad10 0x7fc7f406d1c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc81006d3b0 0x7fc8100b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc810142980 0x7fc8100b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 --2- 172.21.15.104:0/2372317233 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8101424d0 0x7fc810157c20 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 >> 172.21.15.104:0/2372317233 conn(0x7fc8100ceec0 msgr2=0x7fc8100cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 shutdown_connections 2023-06-10T18:57:24.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.280+0000 7fc81706d700 1 -- 172.21.15.104:0/2372317233 wait complete. 2023-06-10T18:57:24.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:57:24.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 Processor -- start 2023-06-10T18:57:24.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 -- start start 2023-06-10T18:57:24.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4b6c156760 0x7f4b6c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0ce850 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0cb4a0 con 0x7f4b6c15d1a0 2023-06-10T18:57:24.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b72b9e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c0b30a0 con 0x7f4b6c156760 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.415+0000 7f4b7113b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4b6c0b21c0 0x7f4b6c0b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36396/0 (socket says 172.21.15.104:36396) 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7113b700 1 -- 172.21.15.104:0/4143750591 learned_addr learned my addr 172.21.15.104:0/4143750591 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 --2- 172.21.15.104:0/4143750591 >> v2:172.21.15.114:3300/0 conn(0x7f4b6c156760 0x7f4b6c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482013341 0 0) 0x7f4b6c0ce850 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c0035b0 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2514466447 0 0) 0x7f4b5c0035b0 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 >> v2:172.21.15.114:3300/0 conn(0x7f4b6c156760 msgr2=0x7f4b6c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 --2- 172.21.15.104:0/4143750591 >> v2:172.21.15.114:3300/0 conn(0x7f4b6c156760 0x7f4b6c0ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 >> v1:172.21.15.114:6789/0 conn(0x7f4b6c15d1a0 legacy=0x7f4b6c0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0cfca0 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4b640019f0 con 0x7f4b6c0b21c0 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 >> v1:172.21.15.104:6789/0 conn(0x7f4b6c0b21c0 legacy=0x7f4b6c0b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 0x7f4b5c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c004850 0x7f4b5c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b5c005200 0x7f4b5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0b30a0 con 0x7f4b5c003a00 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c00c080 con 0x7f4b5c005200 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c00c240 con 0x7f4b5c004850 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 0x7f4b5c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7113b700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b5c005200 0x7f4b5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c004850 0x7f4b5c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 -- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c004850 msgr2=0x7f4b5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c004850 0x7f4b5c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 -- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b5c005200 msgr2=0x7f4b5c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b5c005200 0x7f4b5c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7093a700 1 -- 172.21.15.104:0/4143750591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0035b0 con 0x7f4b5c003a00 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7113b700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b5c005200 0x7f4b5c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.416+0000 7f4b7113b700 1 -- 172.21.15.104:0/4143750591 reap_dead start 2023-06-10T18:57:24.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b7093a700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 0x7f4b5c003e10 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009b60 tx=0x7f4b5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72ff5da5eae57b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000e6c0 con 0x7f4b5c003a00 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6000e820 con 0x7f4b5c003a00 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b6b7fe700 1 -- 172.21.15.104:0/4143750591 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000eb90 con 0x7f4b5c003a00 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 -- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 msgr2=0x7f4b5c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 0x7f4b5c003e10 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009b60 tx=0x7f4b5c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 -- 172.21.15.104:0/4143750591 shutdown_connections 2023-06-10T18:57:24.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/4143750591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b5c003a00 0x7f4b5c003e10 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 -- 172.21.15.104:0/4143750591 >> 172.21.15.104:0/4143750591 conn(0x7f4b6c06e370 msgr2=0x7f4b6c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 -- 172.21.15.104:0/4143750591 shutdown_connections 2023-06-10T18:57:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.417+0000 7f4b72b9e700 1 -- 172.21.15.104:0/4143750591 wait complete. 2023-06-10T18:57:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.418+0000 7f4b72b9e700 1 Processor -- start 2023-06-10T18:57:24.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.418+0000 7f4b72b9e700 1 -- start start 2023-06-10T18:57:24.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c003a00 0x7f4b6c155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c0c8e60 0x7f4b6c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c155580 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0b5050 con 0x7f4b6c0c8e60 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b72b9e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0b4440 con 0x7f4b5c003a00 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38580/0 (socket says 172.21.15.104:38580) 2023-06-10T18:57:24.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.419+0000 7f4b6bfff700 1 -- 172.21.15.104:0/1071393920 learned_addr learned my addr 172.21.15.104:0/1071393920 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b7113b700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c0c8e60 0x7f4b6c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b7093a700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c003a00 0x7f4b6c155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 -- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c003a00 msgr2=0x7f4b6c155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c003a00 0x7f4b6c155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 -- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c0c8e60 msgr2=0x7f4b6c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c0c8e60 0x7f4b6c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 -- 172.21.15.104:0/1071393920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0b4720 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b6bfff700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e580 tx=0x7f4b5800e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580052e0 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b58005440 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c0b4a00 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580038a0 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.420+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c0b4060 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.421+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c074050 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.421+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4b58003a00 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.422+0000 7f4b697fa700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 0x7f4b5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.422+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4b58088ad0 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.422+0000 7f4b7093a700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 0x7f4b5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.422+0000 7f4b7093a700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 0x7f4b5006d190 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4b600090b0 tx=0x7f4b6001e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.426+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4b58053b20 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.611+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4b6c0b41f0 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.612+0000 7f4b697fa700 1 -- 172.21.15.104:0/1071393920 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b58019070 con 0x7f4b6c0c89f0 2023-06-10T18:57:24.618 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:57:24.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 msgr2=0x7f4b5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 0x7f4b5006d190 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4b600090b0 tx=0x7f4b6001e040 comp rx=0 tx=0).stop 2023-06-10T18:57:24.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 msgr2=0x7f4b6c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e580 tx=0x7f4b5800e890 comp rx=0 tx=0).stop 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 shutdown_connections 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4b5006ace0 0x7f4b5006d190 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b6c0c8e60 0x7f4b6c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b5c003a00 0x7f4b6c155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 --2- 172.21.15.104:0/1071393920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4b6c0c89f0 0x7f4b6c0c9f90 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 >> 172.21.15.104:0/1071393920 conn(0x7f4b6c06e370 msgr2=0x7f4b6c0cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 shutdown_connections 2023-06-10T18:57:24.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.614+0000 7f4b72b9e700 1 -- 172.21.15.104:0/1071393920 wait complete. 2023-06-10T18:57:24.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:57:24.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.749+0000 7fc463eb1700 1 Processor -- start 2023-06-10T18:57:24.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.749+0000 7fc463eb1700 1 -- start start 2023-06-10T18:57:24.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc463eb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc463eb1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b2b20 con 0x7fc45c156760 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc463eb1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0cb4a0 con 0x7fc45c15d1a0 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc463eb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0b30a0 con 0x7fc45c0b21c0 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc461c4d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc45c156760 0x7fc45c0ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36414/0 (socket says 172.21.15.104:36414) 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc461c4d700 1 -- 172.21.15.104:0/1010587916 learned_addr learned my addr 172.21.15.104:0/1010587916 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:24.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc46244e700 1 --2- 172.21.15.104:0/1010587916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529498038 0 0) 0x7fc45c0b2b20 con 0x7fc45c156760 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4480035b0 con 0x7fc45c156760 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610560321 0 0) 0x7fc45c0cb4a0 con 0x7fc45c15d1a0 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c0b2b20 con 0x7fc45c15d1a0 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260558833 0 0) 0x7fc4480035b0 con 0x7fc45c156760 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc46244e700 1 -- 172.21.15.104:0/1010587916 >> v1:172.21.15.114:6789/0 conn(0x7fc45c15d1a0 legacy=0x7fc45c0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc46244e700 1 -- 172.21.15.104:0/1010587916 >> v1:172.21.15.104:6789/0 conn(0x7fc45c156760 legacy=0x7fc45c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc46244e700 1 -- 172.21.15.104:0/1010587916 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0cfca0 con 0x7fc45c0b21c0 2023-06-10T18:57:24.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.750+0000 7fc46244e700 1 --2- 172.21.15.104:0/1010587916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0b25e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc454009dc0 tx=0x7fc45400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8397ee00c911a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45400e070 con 0x7fc45c0b21c0 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc454003a20 con 0x7fc45c0b21c0 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc460c4b700 1 -- 172.21.15.104:0/1010587916 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458001670 con 0x7fc45c0b21c0 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 -- 172.21.15.104:0/1010587916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 msgr2=0x7fc45c0b25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 --2- 172.21.15.104:0/1010587916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0b25e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc454009dc0 tx=0x7fc45400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 -- 172.21.15.104:0/1010587916 shutdown_connections 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 --2- 172.21.15.104:0/1010587916 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0b25e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 -- 172.21.15.104:0/1010587916 >> 172.21.15.104:0/1010587916 conn(0x7fc45c06e370 msgr2=0x7fc45c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 -- 172.21.15.104:0/1010587916 shutdown_connections 2023-06-10T18:57:24.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.751+0000 7fc463eb1700 1 -- 172.21.15.104:0/1010587916 wait complete. 2023-06-10T18:57:24.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.752+0000 7fc463eb1700 1 Processor -- start 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.752+0000 7fc463eb1700 1 -- start start 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45c15d1a0 0x7fc45c0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0c7170 con 0x7fc45c156760 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c152470 con 0x7fc45c15d1a0 2023-06-10T18:57:24.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc463eb1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c152e30 con 0x7fc45c0b21c0 2023-06-10T18:57:24.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc46144c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc46144c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38594/0 (socket says 172.21.15.104:38594) 2023-06-10T18:57:24.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc46144c700 1 -- 172.21.15.104:0/2785528250 learned_addr learned my addr 172.21.15.104:0/2785528250 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:24.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc46244e700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45c15d1a0 0x7fc45c0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.753+0000 7fc461c4d700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 -- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 msgr2=0x7fc45c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 -- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45c15d1a0 msgr2=0x7fc45c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45c15d1a0 0x7fc45c0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 -- 172.21.15.104:0/2785528250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454009a70 con 0x7fc45c156760 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc46144c700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c450 tx=0x7fc44c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c0052d0 con 0x7fc45c156760 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c005430 con 0x7fc45c156760 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c153110 con 0x7fc45c156760 2023-06-10T18:57:24.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c0038c0 con 0x7fc45c156760 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.754+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c1536d0 con 0x7fc45c156760 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.755+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c074050 con 0x7fc45c156760 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.755+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc44c003a20 con 0x7fc45c156760 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.756+0000 7fc4527fc700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 0x7fc44006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.756+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fc44c088b70 con 0x7fc45c156760 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.756+0000 7fc461c4d700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 0x7fc44006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:24.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.756+0000 7fc461c4d700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 0x7fc44006d190 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc458001380 tx=0x7fc4580015f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:24.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.760+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc44c053a00 con 0x7fc45c156760 2023-06-10T18:57:24.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.950+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc45c0cab60 con 0x7fc45c156760 2023-06-10T18:57:24.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.951+0000 7fc4527fc700 1 -- 172.21.15.104:0/2785528250 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc44c057020 con 0x7fc45c156760 2023-06-10T18:57:24.958 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:57:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 msgr2=0x7fc44006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 0x7fc44006d190 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc458001380 tx=0x7fc4580015f0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 msgr2=0x7fc45c0b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c450 tx=0x7fc44c00c810 comp rx=0 tx=0).stop 2023-06-10T18:57:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 shutdown_connections 2023-06-10T18:57:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc44006ace0 0x7fc44006d190 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc45c15d1a0 0x7fc45c0c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc45c0b21c0 0x7fc45c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 --2- 172.21.15.104:0/2785528250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc45c156760 0x7fc45c0b3570 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:24.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 >> 172.21.15.104:0/2785528250 conn(0x7fc45c06e370 msgr2=0x7fc45c06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:24.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 shutdown_connections 2023-06-10T18:57:24.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:24.953+0000 7fc463eb1700 1 -- 172.21.15.104:0/2785528250 wait complete. 2023-06-10T18:57:24.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:57:25.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.087+0000 7ff283609700 1 Processor -- start 2023-06-10T18:57:25.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.087+0000 7ff283609700 1 -- start start 2023-06-10T18:57:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff283609700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff27c06dea0 0x7ff27c160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff283609700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 0x7ff27c0ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff283609700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27c15c820 con 0x7ff27c069e70 2023-06-10T18:57:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff283609700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27c06dd60 con 0x7ff27c06dea0 2023-06-10T18:57:25.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff283609700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27c15cad0 con 0x7ff27c06e270 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff2813a5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff27c06dea0 0x7ff27c160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff281ba6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff27c069e70 0x7ff27c06a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36432/0 (socket says 172.21.15.104:36432) 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff281ba6700 1 -- 172.21.15.104:0/4175483543 learned_addr learned my addr 172.21.15.104:0/4175483543 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff280ba4700 1 --2- 172.21.15.104:0/4175483543 >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 0x7ff27c0ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff2813a5700 1 -- 172.21.15.104:0/4175483543 >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 msgr2=0x7ff27c0ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff2813a5700 1 --2- 172.21.15.104:0/4175483543 >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 0x7ff27c0ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff2813a5700 1 -- 172.21.15.104:0/4175483543 >> v1:172.21.15.104:6789/0 conn(0x7ff27c069e70 legacy=0x7ff27c06a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff2813a5700 1 -- 172.21.15.104:0/4175483543 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c15cdb0 con 0x7ff27c06dea0 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff273fff700 1 -- 172.21.15.104:0/4175483543 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009240115 0 0) 0x7ff27c15c820 con 0x7ff27c069e70 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.088+0000 7ff280ba4700 1 --2- 172.21.15.104:0/4175483543 >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 0x7ff27c0ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff2813a5700 1 --2- 172.21.15.104:0/4175483543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06dea0 0x7ff27c160ad0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7ff27c070320 tx=0x7ff26c009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b710f61c98da82b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff273fff700 1 -- 172.21.15.104:0/4175483543 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c005e30 con 0x7ff27c06dea0 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff273fff700 1 -- 172.21.15.104:0/4175483543 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c006030 con 0x7ff27c06dea0 2023-06-10T18:57:25.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff273fff700 1 -- 172.21.15.104:0/4175483543 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0063c0 con 0x7ff27c06dea0 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 -- 172.21.15.104:0/4175483543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06dea0 msgr2=0x7ff27c160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 --2- 172.21.15.104:0/4175483543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06dea0 0x7ff27c160ad0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7ff27c070320 tx=0x7ff26c009b00 comp rx=0 tx=0).stop 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 -- 172.21.15.104:0/4175483543 shutdown_connections 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 --2- 172.21.15.104:0/4175483543 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06dea0 0x7ff27c160ad0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 --2- 172.21.15.104:0/4175483543 >> v2:172.21.15.114:3300/0 conn(0x7ff27c06e270 0x7ff27c0ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 -- 172.21.15.104:0/4175483543 >> 172.21.15.104:0/4175483543 conn(0x7ff27c06b8a0 msgr2=0x7ff27c0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 -- 172.21.15.104:0/4175483543 shutdown_connections 2023-06-10T18:57:25.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.089+0000 7ff283609700 1 -- 172.21.15.104:0/4175483543 wait complete. 2023-06-10T18:57:25.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 Processor -- start 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 -- start start 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 0x7ff27c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 0x7ff27c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c0c77b0 con 0x7ff27c06e270 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c0cc4e0 con 0x7ff27c06dea0 2023-06-10T18:57:25.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff283609700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c0cc7c0 con 0x7ff27c069e70 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff281ba6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff281ba6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38608/0 (socket says 172.21.15.104:38608) 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.091+0000 7ff281ba6700 1 -- 172.21.15.104:0/3830533810 learned_addr learned my addr 172.21.15.104:0/3830533810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff2813a5700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 0x7ff27c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff280ba4700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 0x7ff27c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 -- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 msgr2=0x7ff27c0b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 0x7ff27c0b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 -- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 msgr2=0x7ff27c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 0x7ff27c0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 -- 172.21.15.104:0/3830533810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c009430 con 0x7ff27c06e270 2023-06-10T18:57:25.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff2813a5700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 0x7ff27c0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:25.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff280ba4700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 0x7ff27c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff281ba6700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7ff27400cba0 tx=0x7ff27400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400ab80 con 0x7ff27c06e270 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27400ad00 con 0x7ff27c06e270 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c0ccaa0 con 0x7ff27c06e270 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274007310 con 0x7ff27c06e270 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.092+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c145450 con 0x7ff27c06e270 2023-06-10T18:57:25.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.093+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c0c7960 con 0x7ff27c06e270 2023-06-10T18:57:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.094+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff274007470 con 0x7ff27c06e270 2023-06-10T18:57:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.094+0000 7ff271ffb700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 0x7ff26006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.094+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff274089480 con 0x7ff27c06e270 2023-06-10T18:57:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.094+0000 7ff2813a5700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 0x7ff26006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.095+0000 7ff2813a5700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 0x7ff26006d190 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff26c013040 tx=0x7ff26c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.098+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff27c0c7960 con 0x7ff27c06e270 2023-06-10T18:57:25.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.283+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff27c0c7960 con 0x7ff27c06e270 2023-06-10T18:57:25.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.284+0000 7ff271ffb700 1 -- 172.21.15.104:0/3830533810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff27c0c7960 con 0x7ff27c06e270 2023-06-10T18:57:25.290 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:57:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 msgr2=0x7ff26006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 0x7ff26006d190 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff26c013040 tx=0x7ff26c008070 comp rx=0 tx=0).stop 2023-06-10T18:57:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 msgr2=0x7ff27c0cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7ff27400cba0 tx=0x7ff27400cf60 comp rx=0 tx=0).stop 2023-06-10T18:57:25.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 shutdown_connections 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff27c06e270 0x7ff27c0cbea0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff27c069e70 0x7ff27c0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff26006ace0 0x7ff26006d190 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 --2- 172.21.15.104:0/3830533810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27c06dea0 0x7ff27c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 >> 172.21.15.104:0/3830533810 conn(0x7ff27c06b8a0 msgr2=0x7ff27c15da90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 shutdown_connections 2023-06-10T18:57:25.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.286+0000 7ff283609700 1 -- 172.21.15.104:0/3830533810 wait complete. 2023-06-10T18:57:25.304 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:57:25.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.418+0000 7f8c78755700 1 Processor -- start 2023-06-10T18:57:25.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.418+0000 7f8c78755700 1 -- start start 2023-06-10T18:57:25.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c78755700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c70070a20 0x7f8c7006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c78755700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 0x7f8c70159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c78755700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7006ae00 con 0x7f8c70070df0 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c78755700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c70069e70 con 0x7f8c70070a20 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c78755700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c701595f0 con 0x7f8c70158eb0 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c70070a20 0x7f8c7006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8c70070a20 0x7f8c7006a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38614/0 (socket says 172.21.15.104:38614) 2023-06-10T18:57:25.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 -- 172.21.15.104:0/2075646730 learned_addr learned my addr 172.21.15.104:0/2075646730 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c76cf2700 1 --2- 172.21.15.104:0/2075646730 >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 0x7f8c70159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 -- 172.21.15.104:0/2075646730 >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 msgr2=0x7f8c70159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 --2- 172.21.15.104:0/2075646730 >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 0x7f8c70159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 -- 172.21.15.104:0/2075646730 >> v1:172.21.15.114:6789/0 conn(0x7f8c70070df0 legacy=0x7f8c701587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 -- 172.21.15.104:0/2075646730 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c701598d0 con 0x7f8c70070a20 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c76cf2700 1 --2- 172.21.15.104:0/2075646730 >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 0x7f8c70159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:25.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.419+0000 7f8c764f1700 1 --2- 172.21.15.104:0/2075646730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70070a20 0x7f8c7006a8c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8c700709b0 tx=0x7f8c6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a615c3492553a0a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c754ef700 1 -- 172.21.15.104:0/2075646730 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c004230 con 0x7f8c70070a20 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c754ef700 1 -- 172.21.15.104:0/2075646730 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c004390 con 0x7f8c70070a20 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c754ef700 1 -- 172.21.15.104:0/2075646730 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0036a0 con 0x7f8c70070a20 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 -- 172.21.15.104:0/2075646730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70070a20 msgr2=0x7f8c7006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 --2- 172.21.15.104:0/2075646730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70070a20 0x7f8c7006a8c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f8c700709b0 tx=0x7f8c6c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 -- 172.21.15.104:0/2075646730 shutdown_connections 2023-06-10T18:57:25.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 --2- 172.21.15.104:0/2075646730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70070a20 0x7f8c7006a8c0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 --2- 172.21.15.104:0/2075646730 >> v2:172.21.15.114:3300/0 conn(0x7f8c70158eb0 0x7f8c70159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 -- 172.21.15.104:0/2075646730 >> 172.21.15.104:0/2075646730 conn(0x7f8c700b1610 msgr2=0x7f8c700ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:25.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 -- 172.21.15.104:0/2075646730 shutdown_connections 2023-06-10T18:57:25.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.420+0000 7f8c78755700 1 -- 172.21.15.104:0/2075646730 wait complete. 2023-06-10T18:57:25.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.421+0000 7f8c78755700 1 Processor -- start 2023-06-10T18:57:25.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.421+0000 7f8c78755700 1 -- start start 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c70070a20 0x7f8c700b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c70070df0 0x7f8c700c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7014a070 con 0x7f8c70158eb0 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700b4610 con 0x7f8c70070df0 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c78755700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700b48b0 con 0x7f8c70070a20 2023-06-10T18:57:25.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c76cf2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c76cf2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38624/0 (socket says 172.21.15.104:38624) 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c76cf2700 1 -- 172.21.15.104:0/1923028411 learned_addr learned my addr 172.21.15.104:0/1923028411 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c764f1700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c70070a20 0x7f8c700b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.422+0000 7f8c75cf0700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c70070df0 0x7f8c700c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 -- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c70070a20 msgr2=0x7f8c700b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c70070a20 0x7f8c700b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 -- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c70070df0 msgr2=0x7f8c700c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c70070df0 0x7f8c700c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 -- 172.21.15.104:0/1923028411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c009550 con 0x7f8c70158eb0 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c76cf2700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800cd50 tx=0x7f8c6800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6800f480 con 0x7f8c70158eb0 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6800f600 con 0x7f8c70158eb0 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6800f9b0 con 0x7f8c70158eb0 2023-06-10T18:57:25.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c700b2230 con 0x7f8c70158eb0 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.423+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c700b4b90 con 0x7f8c70158eb0 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.424+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7014a2e0 con 0x7f8c70158eb0 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.424+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c6800b040 con 0x7f8c70158eb0 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.425+0000 7f8c66ffd700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 0x7f8c5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.425+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8c68089ec0 con 0x7f8c70158eb0 2023-06-10T18:57:25.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.425+0000 7f8c764f1700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 0x7f8c5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.425+0000 7f8c764f1700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 0x7f8c5406d190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009130 tx=0x7f8c6c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.429+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c68054dc0 con 0x7f8c70158eb0 2023-06-10T18:57:25.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.629+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8c70071780 con 0x7f8c70158eb0 2023-06-10T18:57:25.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.629+0000 7f8c66ffd700 1 -- 172.21.15.104:0/1923028411 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v147) v1 ==== 80+0+124 (secure 0 0 0) 0x7f8c6801d080 con 0x7f8c70158eb0 2023-06-10T18:57:25.635 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6837/2371761194 2023-06-11T18:57:18.273257+0000 2023-06-10T18:57:25.635 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6835/2371761194 2023-06-11T18:57:18.273257+0000 2023-06-10T18:57:25.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 msgr2=0x7f8c5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 0x7f8c5406d190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009130 tx=0x7f8c6c008030 comp rx=0 tx=0).stop 2023-06-10T18:57:25.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 msgr2=0x7f8c7014a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800cd50 tx=0x7f8c6800a560 comp rx=0 tx=0).stop 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 shutdown_connections 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c70158eb0 0x7f8c7014a470 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c70070a20 0x7f8c700b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8c5406ace0 0x7f8c5406d190 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 --2- 172.21.15.104:0/1923028411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c70070df0 0x7f8c700c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 >> 172.21.15.104:0/1923028411 conn(0x7f8c700b1610 msgr2=0x7f8c70065c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:25.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 shutdown_connections 2023-06-10T18:57:25.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.631+0000 7f8c78755700 1 -- 172.21.15.104:0/1923028411 wait complete. 2023-06-10T18:57:25.639 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:57:25.649 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:57:25.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:25.649 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:25.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[152305]: fuse finished with error 0 and tester_r 0 2023-06-10T18:57:25.684 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:57:25.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:25.685 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:57:25.716 INFO:tasks.ceph.mds.a.smithi104.stderr:2023-06-10T18:57:25.710+0000 7ff6bbd5a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:57:25.731 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:25.732 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:25.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:25.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.885+0000 7f3b3069b700 1 Processor -- start 2023-06-10T18:57:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.885+0000 7f3b3069b700 1 -- start start 2023-06-10T18:57:25.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b3069b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 0x7f3b280cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b3069b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 0x7f3b280b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b3069b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280b17a0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b3069b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2815a8c0 con 0x7f3b280ceb00 2023-06-10T18:57:25.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b3069b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280b14c0 con 0x7f3b280b0d90 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b2dc36700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3b280cd9e0 0x7f3b280ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36456/0 (socket says 172.21.15.104:36456) 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b2dc36700 1 -- 172.21.15.104:0/2605256792 learned_addr learned my addr 172.21.15.104:0/2605256792 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b2ec38700 1 --2- 172.21.15.104:0/2605256792 >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 0x7f3b280b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.886+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2605256792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 0x7f3b280cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862694323 0 0) 0x7f3b280b17a0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b140035b0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2e437700 1 -- 172.21.15.104:0/2605256792 >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 msgr2=0x7f3b280b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2605256792 >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 0x7f3b280b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2e437700 1 -- 172.21.15.104:0/2605256792 >> v1:172.21.15.104:6789/0 conn(0x7f3b280cd9e0 legacy=0x7f3b280ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587974384 0 0) 0x7f3b140035b0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2e437700 1 -- 172.21.15.104:0/2605256792 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280cc9c0 con 0x7f3b280b0d90 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2ec38700 1 --2- 172.21.15.104:0/2605256792 >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 0x7f3b280b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2605256792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 0x7f3b280cd4a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009840 tx=0x7f3b18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1495f598b0069fff server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24001670 con 0x7f3b280b0d90 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18004030 con 0x7f3b280b0d90 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.887+0000 7f3b2d435700 1 -- 172.21.15.104:0/2605256792 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b180043a0 con 0x7f3b280b0d90 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 -- 172.21.15.104:0/2605256792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 msgr2=0x7f3b280cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2605256792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 0x7f3b280cd4a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009840 tx=0x7f3b18009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 -- 172.21.15.104:0/2605256792 shutdown_connections 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2605256792 >> v2:172.21.15.114:3300/0 conn(0x7f3b280ceb00 0x7f3b280b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2605256792 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280b0d90 0x7f3b280cd4a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 -- 172.21.15.104:0/2605256792 >> 172.21.15.104:0/2605256792 conn(0x7f3b28089fd0 msgr2=0x7f3b280d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:25.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 -- 172.21.15.104:0/2605256792 shutdown_connections 2023-06-10T18:57:25.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.888+0000 7f3b3069b700 1 -- 172.21.15.104:0/2605256792 wait complete. 2023-06-10T18:57:25.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.889+0000 7f3b3069b700 1 Processor -- start 2023-06-10T18:57:25.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.889+0000 7f3b3069b700 1 -- start start 2023-06-10T18:57:25.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b280b0d90 0x7f3b2809dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 0x7f3b280cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2809c300 con 0x7f3b280cd9e0 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2809b6f0 con 0x7f3b280b0d90 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2809b9d0 con 0x7f3b280ceb00 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38636/0 (socket says 172.21.15.104:38636) 2023-06-10T18:57:25.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 -- 172.21.15.104:0/2499467914 learned_addr learned my addr 172.21.15.104:0/2499467914 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b280b0d90 0x7f3b2809dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2ec38700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 0x7f3b280cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 -- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 msgr2=0x7f3b280cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 0x7f3b280cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 -- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b280b0d90 msgr2=0x7f3b2809dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b280b0d90 0x7f3b2809dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 -- 172.21.15.104:0/2499467914 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20009a70 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2ec38700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 0x7f3b280cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b2dc36700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f3b240011e0 tx=0x7f3b2400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24007560 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b240076c0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b180094a0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24004a90 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.890+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2809bf90 con 0x7f3b280cd9e0 2023-06-10T18:57:25.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.891+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b280cffc0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.892+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3b24004bf0 con 0x7f3b280cd9e0 2023-06-10T18:57:25.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.892+0000 7f3b1effd700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 0x7f3b0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:25.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.892+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3b24088a30 con 0x7f3b280cd9e0 2023-06-10T18:57:25.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.893+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 0x7f3b0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:25.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.893+0000 7f3b2e437700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 0x7f3b0c06d190 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3b180090b0 tx=0x7f3b18008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:25.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:25.897+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3b240539d0 con 0x7f3b280cd9e0 2023-06-10T18:57:26.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.083+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b28074050 con 0x7f3b280cd9e0 2023-06-10T18:57:26.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.085+0000 7f3b1effd700 1 -- 172.21.15.104:0/2499467914 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18745 (secure 0 0 0) 0x7f3b24018090 con 0x7f3b280cd9e0 2023-06-10T18:57:26.091 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:26.092 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":147,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:18.287462+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2371761194":"2023-06-11T18:57:18.273257+0000","172.21.15.114:6835/2371761194":"2023-06-11T18:57:18.273257+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:26.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 msgr2=0x7f3b0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 0x7f3b0c06d190 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3b180090b0 tx=0x7f3b18008030 comp rx=0 tx=0).stop 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 msgr2=0x7f3b2809aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f3b240011e0 tx=0x7f3b2400cb00 comp rx=0 tx=0).stop 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 shutdown_connections 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b0c06ace0 0x7f3b0c06d190 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b280b0d90 0x7f3b2809dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b280ceb00 0x7f3b280cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 --2- 172.21.15.104:0/2499467914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b280cd9e0 0x7f3b2809aec0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 >> 172.21.15.104:0/2499467914 conn(0x7f3b28089fd0 msgr2=0x7f3b280cc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 shutdown_connections 2023-06-10T18:57:26.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.087+0000 7f3b3069b700 1 -- 172.21.15.104:0/2499467914 wait complete. 2023-06-10T18:57:26.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:04.108341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:05.238053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:57:26.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:57:26.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.223+0000 7fe0631f1700 1 Processor -- start 2023-06-10T18:57:26.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.223+0000 7fe0631f1700 1 -- start start 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe0631f1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe05c070a10 0x7fe05c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe0631f1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe05c06a5e0 0x7fe05c06a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe0631f1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0c4770 con 0x7fe05c145cc0 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe0631f1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c0713d0 con 0x7fe05c070a10 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe0631f1700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c069e70 con 0x7fe05c06a5e0 2023-06-10T18:57:26.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe05c070a10 0x7fe05c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe05c070a10 0x7fe05c070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38644/0 (socket says 172.21.15.104:38644) 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe060f8d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe05c145cc0 0x7fe05c06c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36470/0 (socket says 172.21.15.104:36470) 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 -- 172.21.15.104:0/3803956745 learned_addr learned my addr 172.21.15.104:0/3803956745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe06178e700 1 --2- 172.21.15.104:0/3803956745 >> v2:172.21.15.114:3300/0 conn(0x7fe05c06a5e0 0x7fe05c06a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 -- 172.21.15.104:0/3803956745 >> v2:172.21.15.114:3300/0 conn(0x7fe05c06a5e0 msgr2=0x7fe05c06a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 --2- 172.21.15.104:0/3803956745 >> v2:172.21.15.114:3300/0 conn(0x7fe05c06a5e0 0x7fe05c06a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 -- 172.21.15.104:0/3803956745 >> v1:172.21.15.104:6789/0 conn(0x7fe05c145cc0 legacy=0x7fe05c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05b7fe700 1 -- 172.21.15.104:0/3803956745 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554719865 0 0) 0x7fe05c0c4770 con 0x7fe05c145cc0 2023-06-10T18:57:26.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.224+0000 7fe05bfff700 1 -- 172.21.15.104:0/3803956745 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c06cf10 con 0x7fe05c070a10 2023-06-10T18:57:26.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe05bfff700 1 --2- 172.21.15.104:0/3803956745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c070a10 0x7fe05c070e90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fe05c06ea70 tx=0x7fe048009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebae2e3a422c0ae server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe05b7fe700 1 -- 172.21.15.104:0/3803956745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04800b870 con 0x7fe05c070a10 2023-06-10T18:57:26.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe05b7fe700 1 -- 172.21.15.104:0/3803956745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04800b9d0 con 0x7fe05c070a10 2023-06-10T18:57:26.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe05b7fe700 1 -- 172.21.15.104:0/3803956745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04800bd60 con 0x7fe05c070a10 2023-06-10T18:57:26.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 -- 172.21.15.104:0/3803956745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c070a10 msgr2=0x7fe05c070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 --2- 172.21.15.104:0/3803956745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c070a10 0x7fe05c070e90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fe05c06ea70 tx=0x7fe048009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 -- 172.21.15.104:0/3803956745 shutdown_connections 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 --2- 172.21.15.104:0/3803956745 >> v2:172.21.15.114:3300/0 conn(0x7fe05c06a5e0 0x7fe05c06a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 --2- 172.21.15.104:0/3803956745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c070a10 0x7fe05c070e90 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 -- 172.21.15.104:0/3803956745 >> 172.21.15.104:0/3803956745 conn(0x7fe05c0ceec0 msgr2=0x7fe05c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 -- 172.21.15.104:0/3803956745 shutdown_connections 2023-06-10T18:57:26.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.225+0000 7fe0631f1700 1 -- 172.21.15.104:0/3803956745 wait complete. 2023-06-10T18:57:26.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 Processor -- start 2023-06-10T18:57:26.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 -- start start 2023-06-10T18:57:26.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 0x7fe05c0abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 0x7fe05c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c7610 con 0x7fe05c06a5e0 2023-06-10T18:57:26.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c78b0 con 0x7fe05c070a10 2023-06-10T18:57:26.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.227+0000 7fe0631f1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0c7b90 con 0x7fe05c145cc0 2023-06-10T18:57:26.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38648/0 (socket says 172.21.15.104:38648) 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 -- 172.21.15.104:0/1931175281 learned_addr learned my addr 172.21.15.104:0/1931175281 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe05bfff700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 0x7fe05c0abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe06178e700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 0x7fe05c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 -- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 msgr2=0x7fe05c0c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 0x7fe05c0c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 -- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 msgr2=0x7fe05c0abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 0x7fe05c0abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 -- 172.21.15.104:0/1931175281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048009550 con 0x7fe05c06a5e0 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe06178e700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 0x7fe05c0c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe05bfff700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 0x7fe05c0abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe060f8d700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c550 tx=0x7fe05000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050004c80 con 0x7fe05c06a5e0 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe050004e00 con 0x7fe05c06a5e0 2023-06-10T18:57:26.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c0c7e70 con 0x7fe05c06a5e0 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050007310 con 0x7fe05c06a5e0 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.228+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c144ab0 con 0x7fe05c06a5e0 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.229+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c144ca0 con 0x7fe05c06a5e0 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.230+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe050007470 con 0x7fe05c06a5e0 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.230+0000 7fe0597fa700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 0x7fe04006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.230+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fe0500892a0 con 0x7fe05c06a5e0 2023-06-10T18:57:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.230+0000 7fe05bfff700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 0x7fe04006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.231+0000 7fe05bfff700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 0x7fe04006d190 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe05c160a00 tx=0x7fe048008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.235+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe050054240 con 0x7fe05c06a5e0 2023-06-10T18:57:26.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.418+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe05c074050 con 0x7fe05c06a5e0 2023-06-10T18:57:26.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.418+0000 7fe0597fa700 1 -- 172.21.15.104:0/1931175281 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe050057860 con 0x7fe05c06a5e0 2023-06-10T18:57:26.425 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.420+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 msgr2=0x7fe04006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.420+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 0x7fe04006d190 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe05c160a00 tx=0x7fe048008030 comp rx=0 tx=0).stop 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.420+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 msgr2=0x7fe05c144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.420+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c550 tx=0x7fe05000c910 comp rx=0 tx=0).stop 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 shutdown_connections 2023-06-10T18:57:26.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c145cc0 0x7fe05c0c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe05c06a5e0 0x7fe05c144570 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe04006ace0 0x7fe04006d190 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 --2- 172.21.15.104:0/1931175281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c070a10 0x7fe05c0abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 >> 172.21.15.104:0/1931175281 conn(0x7fe05c0ceec0 msgr2=0x7fe05c15b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 shutdown_connections 2023-06-10T18:57:26.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.421+0000 7fe0631f1700 1 -- 172.21.15.104:0/1931175281 wait complete. 2023-06-10T18:57:26.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:57:26.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.556+0000 7fcb986b7700 1 Processor -- start 2023-06-10T18:57:26.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.556+0000 7fcb986b7700 1 -- start start 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb986b7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb9006e3e0 0x7fcb9006a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb986b7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9015c8d0 con 0x7fcb9015c500 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb986b7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9006abd0 con 0x7fcb90069e70 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb986b7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb900aef20 con 0x7fcb9006e3e0 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb96453700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcb9015c500 0x7fcb900cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36484/0 (socket says 172.21.15.104:36484) 2023-06-10T18:57:26.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb96453700 1 -- 172.21.15.104:0/3850451373 learned_addr learned my addr 172.21.15.104:0/3850451373 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424668283 0 0) 0x7fcb9015c8d0 con 0x7fcb9015c500 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb96c54700 1 --2- 172.21.15.104:0/3850451373 >> v2:172.21.15.114:3300/0 conn(0x7fcb9006e3e0 0x7fcb9006a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.557+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7c0035b0 con 0x7fcb9015c500 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 823575088 0 0) 0x7fcb7c0035b0 con 0x7fcb9015c500 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 >> v2:172.21.15.114:3300/0 conn(0x7fcb9006e3e0 msgr2=0x7fcb9006a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 --2- 172.21.15.104:0/3850451373 >> v2:172.21.15.114:3300/0 conn(0x7fcb9006e3e0 0x7fcb9006a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 >> v1:172.21.15.114:6789/0 conn(0x7fcb90069e70 legacy=0x7fcb9006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9006a8f0 con 0x7fcb9015c500 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcb800017f0 con 0x7fcb9015c500 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 >> v1:172.21.15.104:6789/0 conn(0x7fcb9015c500 legacy=0x7fcb900cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c003a00 0x7fcb7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 0x7fcb7c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb7c005200 0x7fcb7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c00c050 con 0x7fcb7c004850 2023-06-10T18:57:26.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c00c1c0 con 0x7fcb7c005200 2023-06-10T18:57:26.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c00c380 con 0x7fcb7c003a00 2023-06-10T18:57:26.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 0x7fcb7c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb96453700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c003a00 0x7fcb7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb96c54700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb7c005200 0x7fcb7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 -- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c003a00 msgr2=0x7fcb7c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c003a00 0x7fcb7c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 -- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb7c005200 msgr2=0x7fcb7c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb7c005200 0x7fcb7c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 -- 172.21.15.104:0/3850451373 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0035b0 con 0x7fcb7c004850 2023-06-10T18:57:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb96c54700 1 -- 172.21.15.104:0/3850451373 reap_dead start 2023-06-10T18:57:26.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95c52700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 0x7fcb7c004cc0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0017c0 tx=0x7fcb7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293d01ea239df93d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0052d0 con 0x7fcb7c004850 2023-06-10T18:57:26.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c005450 con 0x7fcb7c004850 2023-06-10T18:57:26.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.558+0000 7fcb95451700 1 -- 172.21.15.104:0/3850451373 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0038c0 con 0x7fcb7c004850 2023-06-10T18:57:26.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 -- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 msgr2=0x7fcb7c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 0x7fcb7c004cc0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0017c0 tx=0x7fcb7c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:26.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 -- 172.21.15.104:0/3850451373 shutdown_connections 2023-06-10T18:57:26.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 --2- 172.21.15.104:0/3850451373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c004850 0x7fcb7c004cc0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 -- 172.21.15.104:0/3850451373 >> 172.21.15.104:0/3850451373 conn(0x7fcb9006b8a0 msgr2=0x7fcb900cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 -- 172.21.15.104:0/3850451373 shutdown_connections 2023-06-10T18:57:26.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.559+0000 7fcb986b7700 1 -- 172.21.15.104:0/3850451373 wait complete. 2023-06-10T18:57:26.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.560+0000 7fcb986b7700 1 Processor -- start 2023-06-10T18:57:26.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.560+0000 7fcb986b7700 1 -- start start 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c004850 0x7fcb900b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901f2fd0 0x7fcb901f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c6c20 con 0x7fcb900c5f80 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb90159480 con 0x7fcb901f2fd0 2023-06-10T18:57:26.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb986b7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c6e10 con 0x7fcb7c004850 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38670/0 (socket says 172.21.15.104:38670) 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 -- 172.21.15.104:0/5875116 learned_addr learned my addr 172.21.15.104:0/5875116 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb96c54700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901f2fd0 0x7fcb901f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb96453700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c004850 0x7fcb900b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 -- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c004850 msgr2=0x7fcb900b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c004850 0x7fcb900b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 -- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901f2fd0 msgr2=0x7fcb901f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901f2fd0 0x7fcb901f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 -- 172.21.15.104:0/5875116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900c70f0 con 0x7fcb900c5f80 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.561+0000 7fcb95c52700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c190 tx=0x7fcb8c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.562+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c001290 con 0x7fcb900c5f80 2023-06-10T18:57:26.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.562+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c007040 con 0x7fcb900c5f80 2023-06-10T18:57:26.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.562+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb900c73d0 con 0x7fcb900c5f80 2023-06-10T18:57:26.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.562+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0073f0 con 0x7fcb900c5f80 2023-06-10T18:57:26.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.562+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb900c7990 con 0x7fcb900c5f80 2023-06-10T18:57:26.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.563+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb900c7b20 con 0x7fcb900c5f80 2023-06-10T18:57:26.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.563+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb8c007550 con 0x7fcb900c5f80 2023-06-10T18:57:26.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.563+0000 7fcb86ffd700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 0x7fcb7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.564+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fcb8c089ed0 con 0x7fcb900c5f80 2023-06-10T18:57:26.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.564+0000 7fcb96453700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 0x7fcb7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.564+0000 7fcb96453700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 0x7fcb7406d190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000c00 tx=0x7fcb80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.568+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb8c054f90 con 0x7fcb900c5f80 2023-06-10T18:57:26.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.755+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcb90074050 con 0x7fcb900c5f80 2023-06-10T18:57:26.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.756+0000 7fcb86ffd700 1 -- 172.21.15.104:0/5875116 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcb8c0170a0 con 0x7fcb900c5f80 2023-06-10T18:57:26.764 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:57:26.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 msgr2=0x7fcb7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 0x7fcb7406d190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000c00 tx=0x7fcb80009040 comp rx=0 tx=0).stop 2023-06-10T18:57:26.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 msgr2=0x7fcb900c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00c190 tx=0x7fcb8c001120 comp rx=0 tx=0).stop 2023-06-10T18:57:26.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 shutdown_connections 2023-06-10T18:57:26.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcb7406ace0 0x7fcb7406d190 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901f2fd0 0x7fcb901f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb7c004850 0x7fcb900b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 --2- 172.21.15.104:0/5875116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb900c5f80 0x7fcb900c66e0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 >> 172.21.15.104:0/5875116 conn(0x7fcb9006b8a0 msgr2=0x7fcb900ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 shutdown_connections 2023-06-10T18:57:26.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.758+0000 7fcb986b7700 1 -- 172.21.15.104:0/5875116 wait complete. 2023-06-10T18:57:26.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:57:26.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.895+0000 7f12cc111700 1 Processor -- start 2023-06-10T18:57:26.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.895+0000 7f12cc111700 1 -- start start 2023-06-10T18:57:26.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12c4156760 0x7f12c40ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12c415d1a0 0x7f12c40cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c40b21c0 0x7f12c40b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c40b30a0 con 0x7f12c415d1a0 2023-06-10T18:57:26.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c415d740 con 0x7f12c4156760 2023-06-10T18:57:26.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12cc111700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40ce850 con 0x7f12c40b21c0 2023-06-10T18:57:26.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12c415d1a0 0x7f12c40cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12c415d1a0 0x7f12c40cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38676/0 (socket says 172.21.15.104:38676) 2023-06-10T18:57:26.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 -- 172.21.15.104:0/156866617 learned_addr learned my addr 172.21.15.104:0/156866617 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:26.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 -- 172.21.15.104:0/156866617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c40b21c0 msgr2=0x7f12c40b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 --2- 172.21.15.104:0/156866617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c40b21c0 0x7f12c40b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 -- 172.21.15.104:0/156866617 >> v2:172.21.15.114:3300/0 conn(0x7f12c4156760 msgr2=0x7f12c40ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 --2- 172.21.15.104:0/156866617 >> v2:172.21.15.114:3300/0 conn(0x7f12c4156760 0x7f12c40ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 -- 172.21.15.104:0/156866617 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40b2b20 con 0x7f12c415d1a0 2023-06-10T18:57:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.896+0000 7f12c96ac700 1 --2- 172.21.15.104:0/156866617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c40cad90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f12c406f090 tx=0x7f12c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c22b1a7c2f1659c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12c8eab700 1 -- 172.21.15.104:0/156866617 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c000b870 con 0x7f12c415d1a0 2023-06-10T18:57:26.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12c8eab700 1 -- 172.21.15.104:0/156866617 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c000b9d0 con 0x7f12c415d1a0 2023-06-10T18:57:26.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12c8eab700 1 -- 172.21.15.104:0/156866617 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c000bd60 con 0x7f12c415d1a0 2023-06-10T18:57:26.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 -- 172.21.15.104:0/156866617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 msgr2=0x7f12c40cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 --2- 172.21.15.104:0/156866617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c40cad90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f12c406f090 tx=0x7f12c0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 -- 172.21.15.104:0/156866617 shutdown_connections 2023-06-10T18:57:26.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 --2- 172.21.15.104:0/156866617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c40b21c0 0x7f12c40b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 --2- 172.21.15.104:0/156866617 >> v2:172.21.15.114:3300/0 conn(0x7f12c4156760 0x7f12c40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 --2- 172.21.15.104:0/156866617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c40cad90 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 -- 172.21.15.104:0/156866617 >> 172.21.15.104:0/156866617 conn(0x7f12c406e370 msgr2=0x7f12c406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:26.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 -- 172.21.15.104:0/156866617 shutdown_connections 2023-06-10T18:57:26.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.897+0000 7f12cc111700 1 -- 172.21.15.104:0/156866617 wait complete. 2023-06-10T18:57:26.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.898+0000 7f12cc111700 1 Processor -- start 2023-06-10T18:57:26.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.898+0000 7f12cc111700 1 -- start start 2023-06-10T18:57:26.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c40b21c0 0x7f12c415e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c4156760 0x7f12c414b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c8df0 con 0x7f12c415d1a0 2023-06-10T18:57:26.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c52b0 con 0x7f12c40b21c0 2023-06-10T18:57:26.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12cc111700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c5590 con 0x7f12c4156760 2023-06-10T18:57:26.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12ca6ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12ca6ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38684/0 (socket says 172.21.15.104:38684) 2023-06-10T18:57:26.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12ca6ae700 1 -- 172.21.15.104:0/2365357397 learned_addr learned my addr 172.21.15.104:0/2365357397 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:26.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12c96ac700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c4156760 0x7f12c414b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12c9ead700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c40b21c0 0x7f12c415e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12ca6ae700 1 -- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c4156760 msgr2=0x7f12c414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.899+0000 7f12ca6ae700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c4156760 0x7f12c414b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ca6ae700 1 -- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c40b21c0 msgr2=0x7f12c415e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:26.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ca6ae700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c40b21c0 0x7f12c415e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ca6ae700 1 -- 172.21.15.104:0/2365357397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0009550 con 0x7f12c415d1a0 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ca6ae700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b970 tx=0x7f12bc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc00c9b0 con 0x7f12c415d1a0 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc00cb30 con 0x7f12c415d1a0 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c40c5870 con 0x7f12c415d1a0 2023-06-10T18:57:26.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc014440 con 0x7f12c415d1a0 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.900+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c415eed0 con 0x7f12c415d1a0 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.901+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c415f060 con 0x7f12c415d1a0 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.901+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12bc01d3b0 con 0x7f12c415d1a0 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.902+0000 7f12ba7fc700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.902+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f12bc006e40 con 0x7f12c415d1a0 2023-06-10T18:57:26.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.902+0000 7f12c9ead700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:26.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.902+0000 7f12c9ead700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 0x7f12a806d190 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f12b4006fd0 tx=0x7f12b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:26.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:26.906+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12bc0542e0 con 0x7f12c415d1a0 2023-06-10T18:57:27.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.091+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12c4074050 con 0x7f12c415d1a0 2023-06-10T18:57:27.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.092+0000 7f12ba7fc700 1 -- 172.21.15.104:0/2365357397 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12bc057900 con 0x7f12c415d1a0 2023-06-10T18:57:27.099 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:57:27.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 msgr2=0x7f12a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 0x7f12a806d190 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f12b4006fd0 tx=0x7f12b4008040 comp rx=0 tx=0).stop 2023-06-10T18:57:27.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 msgr2=0x7f12c414bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b970 tx=0x7f12bc00bd30 comp rx=0 tx=0).stop 2023-06-10T18:57:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 shutdown_connections 2023-06-10T18:57:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12c40b21c0 0x7f12c415e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c415d1a0 0x7f12c414bcc0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 --2- 172.21.15.104:0/2365357397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12c4156760 0x7f12c414b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 >> 172.21.15.104:0/2365357397 conn(0x7f12c406e370 msgr2=0x7f12c415d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:27.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 shutdown_connections 2023-06-10T18:57:27.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.094+0000 7f12cc111700 1 -- 172.21.15.104:0/2365357397 wait complete. 2023-06-10T18:57:27.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:57:27.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.229+0000 7f3e35468700 1 Processor -- start 2023-06-10T18:57:27.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.229+0000 7f3e35468700 1 -- start start 2023-06-10T18:57:27.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.229+0000 7f3e35468700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e300ce510 0x7f3e300ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.229+0000 7f3e35468700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e3015b190 0x7f3e30070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e35468700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3015bcc0 con 0x7f3e3015dad0 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e35468700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e30071c90 con 0x7f3e3015b190 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e35468700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30157e60 con 0x7f3e300ce510 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e3015b190 0x7f3e30070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3e3015b190 0x7f3e30070130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38690/0 (socket says 172.21.15.104:38690) 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 -- 172.21.15.104:0/2241231827 learned_addr learned my addr 172.21.15.104:0/2241231827 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2effd700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e300ce510 0x7f3e300ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 -- 172.21.15.104:0/2241231827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e300ce510 msgr2=0x7f3e300ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e300ce510 0x7f3e300ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 -- 172.21.15.104:0/2241231827 >> v1:172.21.15.114:6789/0 conn(0x7f3e3015dad0 legacy=0x7f3e3015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 -- 172.21.15.104:0/2241231827 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e300cee20 con 0x7f3e3015b190 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2e7fc700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015b190 0x7f3e30070130 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f3e240098f0 tx=0x7f3e24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c55d117fad0146e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2dffb700 1 -- 172.21.15.104:0/2241231827 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004230 con 0x7f3e3015b190 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2dffb700 1 -- 172.21.15.104:0/2241231827 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24004390 con 0x7f3e3015b190 2023-06-10T18:57:27.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.230+0000 7f3e2dffb700 1 -- 172.21.15.104:0/2241231827 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240036a0 con 0x7f3e3015b190 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 -- 172.21.15.104:0/2241231827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015b190 msgr2=0x7f3e30070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015b190 0x7f3e30070130 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f3e240098f0 tx=0x7f3e24009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 -- 172.21.15.104:0/2241231827 shutdown_connections 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e300ce510 0x7f3e300ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 --2- 172.21.15.104:0/2241231827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015b190 0x7f3e30070130 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 -- 172.21.15.104:0/2241231827 >> 172.21.15.104:0/2241231827 conn(0x7f3e3006aac0 msgr2=0x7f3e300b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:27.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 -- 172.21.15.104:0/2241231827 shutdown_connections 2023-06-10T18:57:27.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.231+0000 7f3e35468700 1 -- 172.21.15.104:0/2241231827 wait complete. 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.232+0000 7f3e35468700 1 Processor -- start 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.232+0000 7f3e35468700 1 -- start start 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e300ce510 0x7f3e3014b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e3015b190 0x7f3e30157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3014c520 con 0x7f3e3015dad0 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3014b8a0 con 0x7f3e300ce510 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e35468700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3014bb80 con 0x7f3e3015b190 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38698/0 (socket says 172.21.15.104:38698) 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 -- 172.21.15.104:0/22718589 learned_addr learned my addr 172.21.15.104:0/22718589 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2e7fc700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e3015b190 0x7f3e30157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2effd700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e300ce510 0x7f3e3014b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 -- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e3015b190 msgr2=0x7f3e30157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e3015b190 0x7f3e30157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 -- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e300ce510 msgr2=0x7f3e3014b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e300ce510 0x7f3e3014b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 -- 172.21.15.104:0/22718589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24009550 con 0x7f3e3015dad0 2023-06-10T18:57:27.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.233+0000 7f3e2f7fe700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800cc10 tx=0x7f3e2800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.234+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800aa60 con 0x7f3e3015dad0 2023-06-10T18:57:27.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.234+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800abc0 con 0x7f3e3015dad0 2023-06-10T18:57:27.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.234+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3014be60 con 0x7f3e3015dad0 2023-06-10T18:57:27.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.234+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28012420 con 0x7f3e3015dad0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.234+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3014b590 con 0x7f3e3015dad0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.235+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3014c660 con 0x7f3e3015dad0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.235+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3e28012580 con 0x7f3e3015dad0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.235+0000 7f3e177fe700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 0x7f3e1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.235+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3e28089950 con 0x7f3e3015dad0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.236+0000 7f3e2effd700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 0x7f3e1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.236+0000 7f3e2effd700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 0x7f3e1c06d190 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3e20006b20 tx=0x7f3e20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.240+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3e3014c660 con 0x7f3e3015dad0 2023-06-10T18:57:27.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.425+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e3014c660 con 0x7f3e3015dad0 2023-06-10T18:57:27.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.426+0000 7f3e177fe700 1 -- 172.21.15.104:0/22718589 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3e3014c660 con 0x7f3e3015dad0 2023-06-10T18:57:27.432 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:57:27.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 msgr2=0x7f3e1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 0x7f3e1c06d190 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3e20006b20 tx=0x7f3e20008040 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 msgr2=0x7f3e300ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800cc10 tx=0x7f3e2800cf20 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 shutdown_connections 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e1c06ace0 0x7f3e1c06d190 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e300ce510 0x7f3e3014b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e3015dad0 0x7f3e300ca080 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 --2- 172.21.15.104:0/22718589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e3015b190 0x7f3e30157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 >> 172.21.15.104:0/22718589 conn(0x7f3e3006aac0 msgr2=0x7f3e3015a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:27.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 shutdown_connections 2023-06-10T18:57:27.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.428+0000 7f3e35468700 1 -- 172.21.15.104:0/22718589 wait complete. 2023-06-10T18:57:27.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:57:27.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.561+0000 7f87d16bf700 1 Processor -- start 2023-06-10T18:57:27.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.561+0000 7f87d16bf700 1 -- start start 2023-06-10T18:57:27.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87d16bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87d16bf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc15cb00 con 0x7f87cc0cb0c0 2023-06-10T18:57:27.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87d16bf700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc06ecd0 con 0x7f87cc0cc000 2023-06-10T18:57:27.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87d16bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0aed80 con 0x7f87cc15c730 2023-06-10T18:57:27.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87cb7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f87cc0cb0c0 0x7f87cc06d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36534/0 (socket says 172.21.15.104:36534) 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87cb7fe700 1 -- 172.21.15.104:0/380333790 learned_addr learned my addr 172.21.15.104:0/380333790 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87caffd700 1 --2- 172.21.15.104:0/380333790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.562+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100257577 0 0) 0x7f87c4001490 con 0x7f87cc0cb0c0 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40035b0 con 0x7f87cc0cb0c0 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300969979 0 0) 0x7f87cc06ecd0 con 0x7f87cc0cc000 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87caffd700 1 -- 172.21.15.104:0/380333790 >> v1:172.21.15.114:6789/0 conn(0x7f87cc0cc000 legacy=0x7f87cc06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87caffd700 1 -- 172.21.15.104:0/380333790 >> v1:172.21.15.104:6789/0 conn(0x7f87cc0cb0c0 legacy=0x7f87cc06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87caffd700 1 -- 172.21.15.104:0/380333790 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc06db90 con 0x7f87cc15c730 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87caffd700 1 --2- 172.21.15.104:0/380333790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc06e790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009850 tx=0x7f87c0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae6feeebb3dd196 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0005e30 con 0x7f87cc15c730 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c0006030 con 0x7f87cc15c730 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.563+0000 7f87c9ffb700 1 -- 172.21.15.104:0/380333790 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c00063e0 con 0x7f87cc15c730 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 -- 172.21.15.104:0/380333790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 msgr2=0x7f87cc06e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 --2- 172.21.15.104:0/380333790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc06e790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009850 tx=0x7f87c0009b60 comp rx=0 tx=0).stop 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 -- 172.21.15.104:0/380333790 shutdown_connections 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 --2- 172.21.15.104:0/380333790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc06e790 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 -- 172.21.15.104:0/380333790 >> 172.21.15.104:0/380333790 conn(0x7f87cc06b8a0 msgr2=0x7f87cc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 -- 172.21.15.104:0/380333790 shutdown_connections 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.564+0000 7f87d16bf700 1 -- 172.21.15.104:0/380333790 wait complete. 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.565+0000 7f87d16bf700 1 Processor -- start 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.565+0000 7f87d16bf700 1 -- start start 2023-06-10T18:57:27.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87cc0cb0c0 0x7f87cc0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0b4520 con 0x7f87cc0cc000 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc1512b0 con 0x7f87cc0cb0c0 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87d16bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0b35c0 con 0x7f87cc15c730 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38710/0 (socket says 172.21.15.104:38710) 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 -- 172.21.15.104:0/2531889214 learned_addr learned my addr 172.21.15.104:0/2531889214 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87caffd700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87cc0cb0c0 0x7f87cc0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87cb7fe700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 -- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 msgr2=0x7f87cc0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc0b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 -- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87cc0cb0c0 msgr2=0x7f87cc0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87cc0cb0c0 0x7f87cc0b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 -- 172.21.15.104:0/2531889214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0009430 con 0x7f87cc0cc000 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.566+0000 7f87ca7fc700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c500 tx=0x7f87bc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.567+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0054e0 con 0x7f87cc0cc000 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.567+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc003680 con 0x7f87cc0cc000 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.567+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc0b38a0 con 0x7f87cc0cc000 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.567+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc003a30 con 0x7f87cc0cc000 2023-06-10T18:57:27.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.567+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc0b3e60 con 0x7f87cc0cc000 2023-06-10T18:57:27.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.568+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc074050 con 0x7f87cc0cc000 2023-06-10T18:57:27.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.568+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f87bc003b90 con 0x7f87cc0cc000 2023-06-10T18:57:27.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.569+0000 7f87b37fe700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 0x7f87b806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:27.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.569+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f87bc088970 con 0x7f87cc0cc000 2023-06-10T18:57:27.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.569+0000 7f87caffd700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 0x7f87b806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:27.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.570+0000 7f87caffd700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 0x7f87b806d1c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f87c0000c00 tx=0x7f87c0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:27.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.573+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f87bc053910 con 0x7f87cc0cc000 2023-06-10T18:57:27.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.775+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f87cc070aa0 con 0x7f87cc0cc000 2023-06-10T18:57:27.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.775+0000 7f87b37fe700 1 -- 172.21.15.104:0/2531889214 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v147) v1 ==== 80+0+124 (secure 0 0 0) 0x7f87bc017090 con 0x7f87cc0cc000 2023-06-10T18:57:27.782 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6837/2371761194 2023-06-11T18:57:18.273257+0000 2023-06-10T18:57:27.782 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.114:6835/2371761194 2023-06-11T18:57:18.273257+0000 2023-06-10T18:57:27.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.777+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 msgr2=0x7f87b806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.777+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 0x7f87b806d1c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f87c0000c00 tx=0x7f87c0008070 comp rx=0 tx=0).stop 2023-06-10T18:57:27.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.777+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 msgr2=0x7f87cc150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:27.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.777+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c500 tx=0x7f87bc00c810 comp rx=0 tx=0).stop 2023-06-10T18:57:27.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 shutdown_connections 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f87b806ad10 0x7f87b806d1c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87cc0cb0c0 0x7f87cc0b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87cc15c730 0x7f87cc0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 --2- 172.21.15.104:0/2531889214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc0cc000 0x7f87cc150d70 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 >> 172.21.15.104:0/2531889214 conn(0x7f87cc06b8a0 msgr2=0x7f87cc069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 shutdown_connections 2023-06-10T18:57:27.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:27.778+0000 7f87d16bf700 1 -- 172.21.15.104:0/2531889214 wait complete. 2023-06-10T18:57:27.786 INFO:teuthology.orchestra.run.smithi104.stderr:listed 2 entries 2023-06-10T18:57:27.796 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:57:27.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:27.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:27.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[115813]: fuse finished with error 0 and tester_r 0 2023-06-10T18:57:27.838 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:57:27.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:27.839 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:57:27.923 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:27.923 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:27.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:57:27.941 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:57:27.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:27.942 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:57:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:57:27.998 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:57:27.998 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:27.999 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:28.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:57:28.020 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:57:28.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:28.020 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:57:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:57:28.080 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:57:28.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:28.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.194+0000 7f7575742700 1 Processor -- start 2023-06-10T18:57:28.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.194+0000 7f7575742700 1 -- start start 2023-06-10T18:57:28.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f7575742700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f757006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f7575742700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006bfd0 con 0x7f75700b1e30 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f7575742700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006c2b0 con 0x7f757006b820 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f7575742700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700ccf70 con 0x7f75700d0c00 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f756effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f75700b1e30 0x7f75700d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36544/0 (socket says 172.21.15.104:36544) 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f756effd700 1 -- 172.21.15.104:0/1266431477 learned_addr learned my addr 172.21.15.104:0/1266431477 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f756e7fc700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f757006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837605801 0 0) 0x7f757006bfd0 con 0x7f75700b1e30 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.195+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0035b0 con 0x7f75700b1e30 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702086066 0 0) 0x7f755c0035b0 con 0x7f75700b1e30 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 msgr2=0x7f757006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f757006b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 >> v1:172.21.15.114:6789/0 conn(0x7f757006b820 legacy=0x7f75700cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75700cd250 con 0x7f75700b1e30 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7558001670 con 0x7f75700b1e30 2023-06-10T18:57:28.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 >> v1:172.21.15.104:6789/0 conn(0x7f75700b1e30 legacy=0x7f75700d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f755c003a30 0x7f755c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f755c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c004d90 0x7f755c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700ccf70 con 0x7f755c003a30 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c00bac0 con 0x7f755c004d90 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c00bc50 con 0x7f75700d0c00 2023-06-10T18:57:28.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f755c003a30 0x7f755c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756e7fc700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f755c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756f7fe700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c004d90 0x7f755c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 -- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 msgr2=0x7f755c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700d0c00 0x7f755c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 -- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c004d90 msgr2=0x7f755c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c004d90 0x7f755c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 -- 172.21.15.104:0/1266431477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0035b0 con 0x7f755c003a30 2023-06-10T18:57:28.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756effd700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f755c003a30 0x7f755c003e00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f755800c2f0 tx=0x7f755c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c55ba2eaf3cb9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755800ce30 con 0x7f755c003a30 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558005960 con 0x7f755c003a30 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.196+0000 7f756dffb700 1 -- 172.21.15.104:0/1266431477 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558005cf0 con 0x7f755c003a30 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 -- 172.21.15.104:0/1266431477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f755c003a30 msgr2=0x7f755c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 --2- 172.21.15.104:0/1266431477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f755c003a30 0x7f755c003e00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f755800c2f0 tx=0x7f755c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f756f7fe700 1 -- 172.21.15.104:0/1266431477 reap_dead start 2023-06-10T18:57:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 -- 172.21.15.104:0/1266431477 shutdown_connections 2023-06-10T18:57:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 -- 172.21.15.104:0/1266431477 >> 172.21.15.104:0/1266431477 conn(0x7f7570071ec0 msgr2=0x7f757009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 -- 172.21.15.104:0/1266431477 shutdown_connections 2023-06-10T18:57:28.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.197+0000 7f7575742700 1 -- 172.21.15.104:0/1266431477 wait complete. 2023-06-10T18:57:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.199+0000 7f7575742700 1 Processor -- start 2023-06-10T18:57:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.199+0000 7f7575742700 1 -- start start 2023-06-10T18:57:28.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 0x7f75700c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 0x7f75700b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b2bb0 con 0x7f75700b2200 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700ca7d0 con 0x7f755c003a30 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b4bd0 con 0x7f75700b4110 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38732/0 (socket says 172.21.15.104:38732) 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 -- 172.21.15.104:0/1514633470 learned_addr learned my addr 172.21.15.104:0/1514633470 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756e7fc700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 0x7f75700b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756effd700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 0x7f75700c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 -- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 msgr2=0x7f75700b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 0x7f75700b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 -- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 msgr2=0x7f75700c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 0x7f75700c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 -- 172.21.15.104:0/1514633470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75700b4eb0 con 0x7f75700b2200 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756e7fc700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 0x7f75700b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756effd700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 0x7f75700c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f756f7fe700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f75700d1f00 tx=0x7f756800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568003c10 con 0x7f75700b2200 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7568003d70 con 0x7f75700b2200 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700b5190 con 0x7f75700b2200 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568005310 con 0x7f75700b2200 2023-06-10T18:57:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.200+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757015dd40 con 0x7f75700b2200 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.201+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75700d1860 con 0x7f75700b2200 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.202+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7568005470 con 0x7f75700b2200 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.202+0000 7f75677fe700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 0x7f755006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.202+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f7568089820 con 0x7f75700b2200 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.202+0000 7f756effd700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 0x7f755006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.203+0000 7f756effd700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 0x7f755006d190 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7558015040 tx=0x7f7558007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.206+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f75680546b0 con 0x7f75700b2200 2023-06-10T18:57:28.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.435+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7570074050 con 0x7f75700b2200 2023-06-10T18:57:28.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.436+0000 7f75677fe700 1 -- 172.21.15.104:0/1514633470 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f7568057cd0 con 0x7f75700b2200 2023-06-10T18:57:28.442 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:28.442 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:22.712458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":4,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 msgr2=0x7f755006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 0x7f755006d190 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7558015040 tx=0x7f7558007000 comp rx=0 tx=0).stop 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 msgr2=0x7f75700b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f75700d1f00 tx=0x7f756800ef20 comp rx=0 tx=0).stop 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 shutdown_connections 2023-06-10T18:57:28.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755006ace0 0x7f755006d190 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f755c003a30 0x7f75700c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75700b2200 0x7f75700b2670 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 --2- 172.21.15.104:0/1514633470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75700b4110 0x7f75700b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.437+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 >> 172.21.15.104:0/1514633470 conn(0x7f7570071ec0 msgr2=0x7f75700b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.438+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 shutdown_connections 2023-06-10T18:57:28.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.438+0000 7f7575742700 1 -- 172.21.15.104:0/1514633470 wait complete. 2023-06-10T18:57:28.446 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 94 2023-06-10T18:57:28.456 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:28.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.569+0000 7f73d42d9700 1 Processor -- start 2023-06-10T18:57:28.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.569+0000 7f73d42d9700 1 -- start start 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73cc06e0e0 0x7f73cc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 0x7f73cc0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc071640 con 0x7f73cc06e0e0 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc0aef00 con 0x7f73cc0ca100 2023-06-10T18:57:28.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.570+0000 7f73d42d9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc06d290 con 0x7f73cc06e4b0 2023-06-10T18:57:28.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73cc06e0e0 0x7f73cc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73cc06e0e0 0x7f73cc06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38736/0 (socket says 172.21.15.104:38736) 2023-06-10T18:57:28.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 -- 172.21.15.104:0/2895559631 learned_addr learned my addr 172.21.15.104:0/2895559631 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:28.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2876700 1 --2- 172.21.15.104:0/2895559631 >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 0x7f73cc0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d1874700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 -- 172.21.15.104:0/2895559631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 msgr2=0x7f73cc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:28.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 -- 172.21.15.104:0/2895559631 >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 msgr2=0x7f73cc0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 --2- 172.21.15.104:0/2895559631 >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 0x7f73cc0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 -- 172.21.15.104:0/2895559631 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc06d4c0 con 0x7f73cc06e0e0 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d1874700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2876700 1 --2- 172.21.15.104:0/2895559631 >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 0x7f73cc0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d2075700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc06e0e0 0x7f73cc06cd50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f73cc071210 tx=0x7f73bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9e8e4368a30f9d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d1073700 1 -- 172.21.15.104:0/2895559631 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc005e30 con 0x7f73cc06e0e0 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d1073700 1 -- 172.21.15.104:0/2895559631 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc004030 con 0x7f73cc06e0e0 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.571+0000 7f73d1073700 1 -- 172.21.15.104:0/2895559631 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc0043e0 con 0x7f73cc06e0e0 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 -- 172.21.15.104:0/2895559631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc06e0e0 msgr2=0x7f73cc06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc06e0e0 0x7f73cc06cd50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f73cc071210 tx=0x7f73bc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 -- 172.21.15.104:0/2895559631 shutdown_connections 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc06e0e0 0x7f73cc06cd50 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 --2- 172.21.15.104:0/2895559631 >> v2:172.21.15.114:3300/0 conn(0x7f73cc0ca100 0x7f73cc0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 --2- 172.21.15.104:0/2895559631 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 -- 172.21.15.104:0/2895559631 >> 172.21.15.104:0/2895559631 conn(0x7f73cc06b8a0 msgr2=0x7f73cc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:28.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 -- 172.21.15.104:0/2895559631 shutdown_connections 2023-06-10T18:57:28.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.572+0000 7f73d42d9700 1 -- 172.21.15.104:0/2895559631 wait complete. 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.573+0000 7f73d42d9700 1 Processor -- start 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.573+0000 7f73d42d9700 1 -- start start 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73cc06e0e0 0x7f73cc0afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc1548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b0010 con 0x7f73cc0ca100 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0c81f0 con 0x7f73cc06e0e0 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0c7850 con 0x7f73cc06e4b0 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38746/0 (socket says 172.21.15.104:38746) 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 -- 172.21.15.104:0/870752824 learned_addr learned my addr 172.21.15.104:0/870752824 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2075700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73cc06e0e0 0x7f73cc0afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d1874700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc1548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 -- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 msgr2=0x7f73cc1548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc1548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 -- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73cc06e0e0 msgr2=0x7f73cc0afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73cc06e0e0 0x7f73cc0afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 -- 172.21.15.104:0/870752824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009500 con 0x7f73cc0ca100 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d2876700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f73c400cef0 tx=0x7f73c400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c400ad00 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c400ae60 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc0c7b30 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.574+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4015680 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.575+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc154600 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.575+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc0c8330 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.576+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f73c400d430 con 0x7f73cc0ca100 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.576+0000 7f73c2ffd700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 0x7f73b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.576+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f73c4089ae0 con 0x7f73cc0ca100 2023-06-10T18:57:28.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.577+0000 7f73d2075700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 0x7f73b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.577+0000 7f73d2075700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 0x7f73b006d190 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f73bc000c00 tx=0x7f73bc008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.581+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f73cc0c8330 con 0x7f73cc0ca100 2023-06-10T18:57:28.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.771+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73cc0c8330 con 0x7f73cc0ca100 2023-06-10T18:57:28.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.773+0000 7f73c2ffd700 1 -- 172.21.15.104:0/870752824 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18745 (secure 0 0 0) 0x7f73cc0c8330 con 0x7f73cc0ca100 2023-06-10T18:57:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:28.780 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":147,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:18.287462+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2371761194":"2023-06-11T18:57:18.273257+0000","172.21.15.114:6835/2371761194":"2023-06-11T18:57:18.273257+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:28.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 msgr2=0x7f73b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 0x7f73b006d190 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f73bc000c00 tx=0x7f73bc008070 comp rx=0 tx=0).stop 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 msgr2=0x7f73cc1540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f73c400cef0 tx=0x7f73c400cf20 comp rx=0 tx=0).stop 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 shutdown_connections 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f73b006ace0 0x7f73b006d190 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73cc06e0e0 0x7f73cc0afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73cc0ca100 0x7f73cc1540c0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 --2- 172.21.15.104:0/870752824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73cc06e4b0 0x7f73cc1548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 >> 172.21.15.104:0/870752824 conn(0x7f73cc06b8a0 msgr2=0x7f73cc0adaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 shutdown_connections 2023-06-10T18:57:28.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.775+0000 7f73d42d9700 1 -- 172.21.15.104:0/870752824 wait complete. 2023-06-10T18:57:28.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:04.108341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:05.238053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:57:28.795 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:57:28.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.913+0000 7f6a51e08700 1 Processor -- start 2023-06-10T18:57:28.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.913+0000 7f6a51e08700 1 -- start start 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a51e08700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a4c06af80 0x7f6a4c0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a51e08700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0ce340 con 0x7f6a4c0ccff0 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a51e08700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0d00d0 con 0x7f6a4c0ce7d0 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a51e08700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4c0cd490 con 0x7f6a4c06af80 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6a4c06af80 0x7f6a4c0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6a4c0ccff0 0x7f6a4c06b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36578/0 (socket says 172.21.15.104:36578) 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4affd700 1 -- 172.21.15.104:0/2791748463 learned_addr learned my addr 172.21.15.104:0/2791748463 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/2791748463 >> v1:172.21.15.114:6789/0 conn(0x7f6a4c0ce7d0 legacy=0x7f6a4c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/2791748463 >> v1:172.21.15.104:6789/0 conn(0x7f6a4c0ccff0 legacy=0x7f6a4c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/2791748463 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0cd680 con 0x7f6a4c06af80 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.914+0000 7f6a4a7fc700 1 -- 172.21.15.104:0/2791748463 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279893622 0 0) 0x7f6a4c0ce340 con 0x7f6a4c0ccff0 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a4b7fe700 1 --2- 172.21.15.104:0/2791748463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c0d0d20 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009840 tx=0x7f6a40009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a60ec25d8dd6a59 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a4a7fc700 1 -- 172.21.15.104:0/2791748463 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40011830 con 0x7f6a4c06af80 2023-06-10T18:57:28.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a4a7fc700 1 -- 172.21.15.104:0/2791748463 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40011990 con 0x7f6a4c06af80 2023-06-10T18:57:28.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a4a7fc700 1 -- 172.21.15.104:0/2791748463 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40011d20 con 0x7f6a4c06af80 2023-06-10T18:57:28.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 -- 172.21.15.104:0/2791748463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 msgr2=0x7f6a4c0d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 --2- 172.21.15.104:0/2791748463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c0d0d20 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009840 tx=0x7f6a40009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 -- 172.21.15.104:0/2791748463 shutdown_connections 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 --2- 172.21.15.104:0/2791748463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c0d0d20 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 -- 172.21.15.104:0/2791748463 >> 172.21.15.104:0/2791748463 conn(0x7f6a4c089fd0 msgr2=0x7f6a4c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 -- 172.21.15.104:0/2791748463 shutdown_connections 2023-06-10T18:57:28.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.915+0000 7f6a51e08700 1 -- 172.21.15.104:0/2791748463 wait complete. 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.916+0000 7f6a51e08700 1 Processor -- start 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 -- start start 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4c0ccff0 0x7f6a4c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a4c0ce7d0 0x7f6a4c1e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0b4c20 con 0x7f6a4c06af80 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c1e6080 con 0x7f6a4c0ccff0 2023-06-10T18:57:28.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a51e08700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0b4050 con 0x7f6a4c0ce7d0 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38756/0 (socket says 172.21.15.104:38756) 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.917+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/1252770112 learned_addr learned my addr 172.21.15.104:0/1252770112 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4affd700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4c0ccff0 0x7f6a4c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4bfff700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a4c0ce7d0 0x7f6a4c1e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a4c0ce7d0 msgr2=0x7f6a4c1e86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a4c0ce7d0 0x7f6a4c1e86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4c0ccff0 msgr2=0x7f6a4c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4c0ccff0 0x7f6a4c0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 -- 172.21.15.104:0/1252770112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400094a0 con 0x7f6a4c06af80 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a4b7fe700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009f50 tx=0x7f6a40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4001a8f0 con 0x7f6a4c06af80 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c0b42f0 con 0x7f6a4c06af80 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4001aa50 con 0x7f6a4c06af80 2023-06-10T18:57:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4001ade0 con 0x7f6a4c06af80 2023-06-10T18:57:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.918+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c0b48b0 con 0x7f6a4c06af80 2023-06-10T18:57:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.919+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c074050 con 0x7f6a4c06af80 2023-06-10T18:57:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.919+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a40022070 con 0x7f6a4c06af80 2023-06-10T18:57:28.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.920+0000 7f6a33fff700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 0x7f6a3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:28.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.920+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f6a4008a5c0 con 0x7f6a4c06af80 2023-06-10T18:57:28.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.920+0000 7f6a4affd700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 0x7f6a3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:28.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.920+0000 7f6a4affd700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 0x7f6a3806d190 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c007fd0 tx=0x7f6a3c007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:28.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:28.924+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a400554e0 con 0x7f6a4c06af80 2023-06-10T18:57:29.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.108+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a4c15bc50 con 0x7f6a4c06af80 2023-06-10T18:57:29.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.108+0000 7f6a33fff700 1 -- 172.21.15.104:0/1252770112 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a40058b00 con 0x7f6a4c06af80 2023-06-10T18:57:29.115 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 msgr2=0x7f6a3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 0x7f6a3806d190 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c007fd0 tx=0x7f6a3c007c20 comp rx=0 tx=0).stop 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 msgr2=0x7f6a4c1e5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009f50 tx=0x7f6a40004750 comp rx=0 tx=0).stop 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 shutdown_connections 2023-06-10T18:57:29.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a4c0ce7d0 0x7f6a4c1e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a4c06af80 0x7f6a4c1e5b40 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a3806ace0 0x7f6a3806d190 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 --2- 172.21.15.104:0/1252770112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4c0ccff0 0x7f6a4c0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.110+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 >> 172.21.15.104:0/1252770112 conn(0x7f6a4c089fd0 msgr2=0x7f6a4c0cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.111+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 shutdown_connections 2023-06-10T18:57:29.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.111+0000 7f6a51e08700 1 -- 172.21.15.104:0/1252770112 wait complete. 2023-06-10T18:57:29.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:57:29.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.245+0000 7f3160129700 1 Processor -- start 2023-06-10T18:57:29.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.245+0000 7f3160129700 1 -- start start 2023-06-10T18:57:29.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f3160129700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f315806acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f3160129700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31580c4770 con 0x7f3158070a10 2023-06-10T18:57:29.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f3160129700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3158069e70 con 0x7f3158145cc0 2023-06-10T18:57:29.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f3160129700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315806cf10 con 0x7f3158070e90 2023-06-10T18:57:29.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315d6c4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3158070a10 0x7f315806a5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36592/0 (socket says 172.21.15.104:36592) 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315d6c4700 1 -- 172.21.15.104:0/661448957 learned_addr learned my addr 172.21.15.104:0/661448957 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315e6c6700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f315806acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826067733 0 0) 0x7f31580c4770 con 0x7f3158070a10 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31440035b0 con 0x7f3158070a10 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887208998 0 0) 0x7f3158069e70 con 0x7f3158145cc0 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31580c4770 con 0x7f3158145cc0 2023-06-10T18:57:29.255 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027525250 0 0) 0x7f31440035b0 con 0x7f3158070a10 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 msgr2=0x7f315806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f315806acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 >> v1:172.21.15.114:6789/0 conn(0x7f3158145cc0 legacy=0x7f315806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31580c49c0 con 0x7f3158070a10 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315e6c6700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f315806acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.246+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3148001670 con 0x7f3158070a10 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 >> v1:172.21.15.104:6789/0 conn(0x7f3158070a10 legacy=0x7f315806a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f3144003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f3144004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3144004d90 0x7f3144005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315806cf10 con 0x7f3144004100 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314400bac0 con 0x7f3144004d90 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314400bc80 con 0x7f3158070e90 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f3144004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315dec5700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f3144003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315e6c6700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3144004d90 0x7f3144005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 -- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 msgr2=0x7f3144003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158070e90 0x7f3144003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 -- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3144004d90 msgr2=0x7f3144005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3144004d90 0x7f3144005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 -- 172.21.15.104:0/661448957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3150009a70 con 0x7f3144004100 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315e6c6700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3144004d90 0x7f3144005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:29.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315d6c4700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f3144004850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f3148001eb0 tx=0x7f3144005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34feafc2258fcdbc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314800cd20 con 0x7f3144004100 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148001af0 con 0x7f3144004100 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.247+0000 7f315cec3700 1 -- 172.21.15.104:0/661448957 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314800cea0 con 0x7f3144004100 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 -- 172.21.15.104:0/661448957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 msgr2=0x7f3144004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 --2- 172.21.15.104:0/661448957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f3144004850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f3148001eb0 tx=0x7f3144005880 comp rx=0 tx=0).stop 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f315e6c6700 1 -- 172.21.15.104:0/661448957 reap_dead start 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 -- 172.21.15.104:0/661448957 shutdown_connections 2023-06-10T18:57:29.259 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 -- 172.21.15.104:0/661448957 >> 172.21.15.104:0/661448957 conn(0x7f31580ceec0 msgr2=0x7f31580cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 -- 172.21.15.104:0/661448957 shutdown_connections 2023-06-10T18:57:29.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.248+0000 7f3160129700 1 -- 172.21.15.104:0/661448957 wait complete. 2023-06-10T18:57:29.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.249+0000 7f3160129700 1 Processor -- start 2023-06-10T18:57:29.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.249+0000 7f3160129700 1 -- start start 2023-06-10T18:57:29.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158158d00 0x7f315815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315815cd50 0x7f31580c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315815d660 con 0x7f3144004100 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315815d380 con 0x7f315815cd50 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31580b33f0 con 0x7f3158158d00 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38776/0 (socket says 172.21.15.104:38776) 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 -- 172.21.15.104:0/3964715500 learned_addr learned my addr 172.21.15.104:0/3964715500 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:29.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315e6c6700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315815cd50 0x7f31580c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315d6c4700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158158d00 0x7f315815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 -- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158158d00 msgr2=0x7f315815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158158d00 0x7f315815c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 -- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315815cd50 msgr2=0x7f31580c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315815cd50 0x7f31580c70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 -- 172.21.15.104:0/3964715500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31580b36d0 con 0x7f3144004100 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f315dec5700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f315400f7e0 tx=0x7f315400fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540052e0 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154005440 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540038c0 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31580b39b0 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.250+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3158159170 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.251+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315815d120 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.252+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3154003a20 con 0x7f3144004100 2023-06-10T18:57:29.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.252+0000 7f314e7fc700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 0x7f313c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.252+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f31540894c0 con 0x7f3144004100 2023-06-10T18:57:29.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.252+0000 7f315d6c4700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 0x7f313c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.253+0000 7f315d6c4700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 0x7f313c06d190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f31581f5530 tx=0x7f314800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.256+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3154054460 con 0x7f3144004100 2023-06-10T18:57:29.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.441+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3158074050 con 0x7f3144004100 2023-06-10T18:57:29.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.441+0000 7f314e7fc700 1 -- 172.21.15.104:0/3964715500 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3154057a80 con 0x7f3144004100 2023-06-10T18:57:29.448 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:57:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 msgr2=0x7f313c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 0x7f313c06d190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f31581f5530 tx=0x7f314800a000 comp rx=0 tx=0).stop 2023-06-10T18:57:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 msgr2=0x7f31580b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f315400f7e0 tx=0x7f315400fba0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 shutdown_connections 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f313c06ace0 0x7f313c06d190 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315815cd50 0x7f31580c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3144004100 0x7f31580b3ed0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 --2- 172.21.15.104:0/3964715500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3158158d00 0x7f315815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 >> 172.21.15.104:0/3964715500 conn(0x7f31580ceec0 msgr2=0x7f31580ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 shutdown_connections 2023-06-10T18:57:29.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.444+0000 7f3160129700 1 -- 172.21.15.104:0/3964715500 wait complete. 2023-06-10T18:57:29.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:57:29.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.580+0000 7f30861cf700 1 Processor -- start 2023-06-10T18:57:29.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.580+0000 7f30861cf700 1 -- start start 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f30861cf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3080145cc0 0x7f308006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f30861cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080070a10 0x7f3080070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f30861cf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30800713a0 con 0x7f308015fd40 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f30861cf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30800c4770 con 0x7f3080145cc0 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f30861cf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3080069e70 con 0x7f3080070a10 2023-06-10T18:57:29.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3080145cc0 0x7f308006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f308015fd40 0x7f3080160180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36616/0 (socket says 172.21.15.104:36616) 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307ffff700 1 -- 172.21.15.104:0/1330455954 learned_addr learned my addr 172.21.15.104:0/1330455954 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3080145cc0 0x7f308006c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38784/0 (socket says 172.21.15.104:38784) 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307effd700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080070a10 0x7f3080070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 -- 172.21.15.104:0/1330455954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080070a10 msgr2=0x7f3080070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080070a10 0x7f3080070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 -- 172.21.15.104:0/1330455954 >> v1:172.21.15.104:6789/0 conn(0x7f308015fd40 legacy=0x7f3080160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307f7fe700 1 -- 172.21.15.104:0/1330455954 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30800c49c0 con 0x7f3080145cc0 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.581+0000 7f307e7fc700 1 -- 172.21.15.104:0/1330455954 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180460615 0 0) 0x7f3078001490 con 0x7f308015fd40 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f307f7fe700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080145cc0 0x7f308006c800 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f3068009840 tx=0x7f3068009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17de4ab7a67e1b53 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f307e7fc700 1 -- 172.21.15.104:0/1330455954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011830 con 0x7f3080145cc0 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f307e7fc700 1 -- 172.21.15.104:0/1330455954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3068011990 con 0x7f3080145cc0 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f307e7fc700 1 -- 172.21.15.104:0/1330455954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068011d20 con 0x7f3080145cc0 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 -- 172.21.15.104:0/1330455954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080145cc0 msgr2=0x7f308006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080145cc0 0x7f308006c800 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f3068009840 tx=0x7f3068009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 -- 172.21.15.104:0/1330455954 shutdown_connections 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080145cc0 0x7f308006c800 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 --2- 172.21.15.104:0/1330455954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080070a10 0x7f3080070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 -- 172.21.15.104:0/1330455954 >> 172.21.15.104:0/1330455954 conn(0x7f30800ceec0 msgr2=0x7f30800cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 -- 172.21.15.104:0/1330455954 shutdown_connections 2023-06-10T18:57:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.582+0000 7f30861cf700 1 -- 172.21.15.104:0/1330455954 wait complete. 2023-06-10T18:57:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.583+0000 7f30861cf700 1 Processor -- start 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 -- start start 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080145cc0 0x7f308015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 0x7f30801e6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b3470 con 0x7f3080070a10 2023-06-10T18:57:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b23d0 con 0x7f308015fd40 2023-06-10T18:57:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f30861cf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b26b0 con 0x7f3080145cc0 2023-06-10T18:57:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.584+0000 7f307f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38790/0 (socket says 172.21.15.104:38790) 2023-06-10T18:57:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 -- 172.21.15.104:0/975796113 learned_addr learned my addr 172.21.15.104:0/975796113 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307effd700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080145cc0 0x7f308015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307ffff700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 0x7f30801e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 -- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080145cc0 msgr2=0x7f308015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080145cc0 0x7f308015a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 -- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 msgr2=0x7f30801e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 0x7f30801e6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 -- 172.21.15.104:0/975796113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680094a0 con 0x7f3080070a10 2023-06-10T18:57:29.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307ffff700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 0x7f30801e6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:29.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f307f7fe700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f30680090b0 tx=0x7f3068004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680038a0 con 0x7f3080070a10 2023-06-10T18:57:29.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306801f070 con 0x7f3080070a10 2023-06-10T18:57:29.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068003c60 con 0x7f3080070a10 2023-06-10T18:57:29.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30800b2990 con 0x7f3080070a10 2023-06-10T18:57:29.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.585+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30800b2f50 con 0x7f3080070a10 2023-06-10T18:57:29.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.586+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3080074050 con 0x7f3080070a10 2023-06-10T18:57:29.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.587+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f306801a430 con 0x7f3080070a10 2023-06-10T18:57:29.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.587+0000 7f3077fff700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 0x7f306006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.602 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.587+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f3068021070 con 0x7f3080070a10 2023-06-10T18:57:29.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.587+0000 7f307effd700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 0x7f306006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.588+0000 7f307effd700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 0x7f306006d190 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f308013c480 tx=0x7f3070008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.603 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.591+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f30680560c0 con 0x7f3080070a10 2023-06-10T18:57:29.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.776+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f30800cdb70 con 0x7f3080070a10 2023-06-10T18:57:29.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.776+0000 7f3077fff700 1 -- 172.21.15.104:0/975796113 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30680596e0 con 0x7f3080070a10 2023-06-10T18:57:29.783 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:57:29.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 msgr2=0x7f306006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 0x7f306006d190 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f308013c480 tx=0x7f3070008040 comp rx=0 tx=0).stop 2023-06-10T18:57:29.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 msgr2=0x7f30800b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f30680090b0 tx=0x7f3068004750 comp rx=0 tx=0).stop 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 shutdown_connections 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f306006ace0 0x7f306006d190 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f308015fd40 0x7f30801e6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3080070a10 0x7f30800b1df0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 --2- 172.21.15.104:0/975796113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3080145cc0 0x7f308015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 >> 172.21.15.104:0/975796113 conn(0x7f30800ceec0 msgr2=0x7f30800ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 shutdown_connections 2023-06-10T18:57:29.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.779+0000 7f30861cf700 1 -- 172.21.15.104:0/975796113 wait complete. 2023-06-10T18:57:29.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:57:29.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.917+0000 7fbf9d116700 1 Processor -- start 2023-06-10T18:57:29.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.917+0000 7fbf9d116700 1 -- start start 2023-06-10T18:57:29.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9d116700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf98156760 0x7fbf980ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9d116700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf9815d1a0 0x7fbf980cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9d116700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf980ce850 con 0x7fbf980b21c0 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9d116700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980b30a0 con 0x7fbf9815d1a0 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9d116700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9815d740 con 0x7fbf98156760 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf9815d1a0 0x7fbf980cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9759e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbf980b21c0 0x7fbf98160290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36630/0 (socket says 172.21.15.104:36630) 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9759e700 1 -- 172.21.15.104:0/2663391812 learned_addr learned my addr 172.21.15.104:0/2663391812 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:29.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbf9815d1a0 0x7fbf980cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38800/0 (socket says 172.21.15.104:38800) 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf96d9d700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf98156760 0x7fbf980ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 -- 172.21.15.104:0/2663391812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf98156760 msgr2=0x7fbf980ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf98156760 0x7fbf980ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 -- 172.21.15.104:0/2663391812 >> v1:172.21.15.104:6789/0 conn(0x7fbf980b21c0 legacy=0x7fbf98160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 -- 172.21.15.104:0/2663391812 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980cfca0 con 0x7fbf9815d1a0 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf95d9b700 1 -- 172.21.15.104:0/2663391812 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172178497 0 0) 0x7fbf980ce850 con 0x7fbf980b21c0 2023-06-10T18:57:29.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf9659c700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9815d1a0 0x7fbf980cad90 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009840 tx=0x7fbf88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d849619e922dc1a9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.918+0000 7fbf95d9b700 1 -- 172.21.15.104:0/2663391812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88011830 con 0x7fbf9815d1a0 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf95d9b700 1 -- 172.21.15.104:0/2663391812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf88011990 con 0x7fbf9815d1a0 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf95d9b700 1 -- 172.21.15.104:0/2663391812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88011d20 con 0x7fbf9815d1a0 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 -- 172.21.15.104:0/2663391812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9815d1a0 msgr2=0x7fbf980cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9815d1a0 0x7fbf980cad90 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009840 tx=0x7fbf88009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 -- 172.21.15.104:0/2663391812 shutdown_connections 2023-06-10T18:57:29.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf98156760 0x7fbf980ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 --2- 172.21.15.104:0/2663391812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9815d1a0 0x7fbf980cad90 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 -- 172.21.15.104:0/2663391812 >> 172.21.15.104:0/2663391812 conn(0x7fbf9806e370 msgr2=0x7fbf9806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:29.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 -- 172.21.15.104:0/2663391812 shutdown_connections 2023-06-10T18:57:29.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.919+0000 7fbf9d116700 1 -- 172.21.15.104:0/2663391812 wait complete. 2023-06-10T18:57:29.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 Processor -- start 2023-06-10T18:57:29.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 -- start start 2023-06-10T18:57:29.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 0x7fbf98153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf9815d1a0 0x7fbf980c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf981541c0 con 0x7fbf98156760 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980c5480 con 0x7fbf980b21c0 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9d116700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf981e4f10 con 0x7fbf9815d1a0 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38804/0 (socket says 172.21.15.104:38804) 2023-06-10T18:57:29.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.921+0000 7fbf9659c700 1 -- 172.21.15.104:0/1560496321 learned_addr learned my addr 172.21.15.104:0/1560496321 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9759e700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf9815d1a0 0x7fbf980c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf96d9d700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 0x7fbf98153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 -- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf9815d1a0 msgr2=0x7fbf980c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf9815d1a0 0x7fbf980c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 -- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 msgr2=0x7fbf98153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:29.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 0x7fbf98153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:29.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 -- 172.21.15.104:0/1560496321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf880094a0 con 0x7fbf98156760 2023-06-10T18:57:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf96d9d700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 0x7fbf98153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9659c700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fbf88014040 tx=0x7fbf88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf880036a0 con 0x7fbf98156760 2023-06-10T18:57:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8801f070 con 0x7fbf98156760 2023-06-10T18:57:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf981e51f0 con 0x7fbf98156760 2023-06-10T18:57:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88003a40 con 0x7fbf98156760 2023-06-10T18:57:29.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.922+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf981e57b0 con 0x7fbf98156760 2023-06-10T18:57:29.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.923+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf98074050 con 0x7fbf98156760 2023-06-10T18:57:29.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.923+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbf88003ba0 con 0x7fbf98156760 2023-06-10T18:57:29.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.924+0000 7fbf7f7fe700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:29.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.924+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fbf88021070 con 0x7fbf98156760 2023-06-10T18:57:29.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.924+0000 7fbf96d9d700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:29.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.924+0000 7fbf96d9d700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf980b4710 tx=0x7fbf8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:29.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:29.928+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbf88055f30 con 0x7fbf98156760 2023-06-10T18:57:30.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.115+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf98070340 con 0x7fbf98156760 2023-06-10T18:57:30.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.115+0000 7fbf7f7fe700 1 -- 172.21.15.104:0/1560496321 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf880150a0 con 0x7fbf98156760 2023-06-10T18:57:30.122 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:57:30.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 msgr2=0x7fbf8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf980b4710 tx=0x7fbf8c008040 comp rx=0 tx=0).stop 2023-06-10T18:57:30.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 msgr2=0x7fbf980b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fbf88014040 tx=0x7fbf88004750 comp rx=0 tx=0).stop 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 shutdown_connections 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbf8406ace0 0x7fbf8406d190 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf980b21c0 0x7fbf98153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf9815d1a0 0x7fbf980c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 --2- 172.21.15.104:0/1560496321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf98156760 0x7fbf980b4920 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 >> 172.21.15.104:0/1560496321 conn(0x7fbf9806e370 msgr2=0x7fbf98155de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 shutdown_connections 2023-06-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.117+0000 7fbf9d116700 1 -- 172.21.15.104:0/1560496321 wait complete. 2023-06-10T18:57:30.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:30.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.250+0000 7f0f33b4b700 1 Processor -- start 2023-06-10T18:57:30.256 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.250+0000 7f0f33b4b700 1 -- start start 2023-06-10T18:57:30.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f33b4b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 0x7f0f2c0ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f33b4b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 0x7f0f2c06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.257 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f33b4b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c0cc050 con 0x7f0f2c06d810 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f33b4b700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c06e290 con 0x7f0f2c06a580 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f33b4b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0cba00 con 0x7f0f2c15a230 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f318e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 0x7f0f2c0ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f320e8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 0x7f0f2c06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f310e6700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0f2c06d810 0x7f0f2c069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:42486/0 (socket says 172.21.15.104:42486) 2023-06-10T18:57:30.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f310e6700 1 -- 172.21.15.104:0/1487247317 learned_addr learned my addr 172.21.15.104:0/1487247317 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:30.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f318e7700 1 -- 172.21.15.104:0/1487247317 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 msgr2=0x7f0f2c06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.251+0000 7f0f318e7700 1 --2- 172.21.15.104:0/1487247317 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 0x7f0f2c06dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f318e7700 1 -- 172.21.15.104:0/1487247317 >> v1:172.21.15.114:6789/0 conn(0x7f0f2c06d810 legacy=0x7f0f2c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f318e7700 1 -- 172.21.15.104:0/1487247317 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0cbce0 con 0x7f0f2c15a230 2023-06-10T18:57:30.260 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f320e8700 1 --2- 172.21.15.104:0/1487247317 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 0x7f0f2c06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f318e7700 1 --2- 172.21.15.104:0/1487247317 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 0x7f0f2c0ac2c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009820 tx=0x7f0f28009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af82ceeaabc5bf0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f308e5700 1 -- 172.21.15.104:0/1487247317 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28005e30 con 0x7f0f2c15a230 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f308e5700 1 -- 172.21.15.104:0/1487247317 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28006030 con 0x7f0f2c15a230 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f308e5700 1 -- 172.21.15.104:0/1487247317 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f280063c0 con 0x7f0f2c15a230 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 -- 172.21.15.104:0/1487247317 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 msgr2=0x7f0f2c0ac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/1487247317 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 0x7f0f2c0ac2c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009820 tx=0x7f0f28009b30 comp rx=0 tx=0).stop 2023-06-10T18:57:30.261 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 -- 172.21.15.104:0/1487247317 shutdown_connections 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/1487247317 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c06a580 0x7f0f2c06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/1487247317 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c15a230 0x7f0f2c0ac2c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 -- 172.21.15.104:0/1487247317 >> 172.21.15.104:0/1487247317 conn(0x7f0f2c06b8a0 msgr2=0x7f0f2c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.252+0000 7f0f33b4b700 1 -- 172.21.15.104:0/1487247317 shutdown_connections 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.253+0000 7f0f33b4b700 1 -- 172.21.15.104:0/1487247317 wait complete. 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 Processor -- start 2023-06-10T18:57:30.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 -- start start 2023-06-10T18:57:30.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06a580 0x7f0f2c0c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c06d810 0x7f0f2c156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b2eb0 con 0x7f0f2c15a230 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b3190 con 0x7f0f2c06d810 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f33b4b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0b3470 con 0x7f0f2c06a580 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.254+0000 7f0f320e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38822/0 (socket says 172.21.15.104:38822) 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 -- 172.21.15.104:0/3413276006 learned_addr learned my addr 172.21.15.104:0/3413276006 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f318e7700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06a580 0x7f0f2c0c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f310e6700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c06d810 0x7f0f2c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 -- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06a580 msgr2=0x7f0f2c0c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06a580 0x7f0f2c0c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 -- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c06d810 msgr2=0x7f0f2c156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c06d810 0x7f0f2c156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 -- 172.21.15.104:0/3413276006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24009a70 con 0x7f0f2c15a230 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f320e8700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400efc0 tx=0x7f0f2400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400cc90 con 0x7f0f2c15a230 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f28009430 con 0x7f0f2c15a230 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2400f040 con 0x7f0f2c15a230 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f240146c0 con 0x7f0f2c15a230 2023-06-10T18:57:30.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.255+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c0b3a30 con 0x7f0f2c15a230 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.256+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c0b3c50 con 0x7f0f2c15a230 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.256+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0f24014820 con 0x7f0f2c15a230 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.257+0000 7f0f227fc700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.257+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f0f2408a260 con 0x7f0f2c15a230 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.257+0000 7f0f318e7700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.258+0000 7f0f318e7700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009b70 tx=0x7f0f28008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.261+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0f24055180 con 0x7f0f2c15a230 2023-06-10T18:57:30.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.492+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f2c074050 con 0x7f0f2c15a230 2023-06-10T18:57:30.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.492+0000 7f0f227fc700 1 -- 172.21.15.104:0/3413276006 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f0f240587a0 con 0x7f0f2c15a230 2023-06-10T18:57:30.499 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:30.499 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:22.712458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":4,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:30.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 msgr2=0x7f0f1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009b70 tx=0x7f0f28008070 comp rx=0 tx=0).stop 2023-06-10T18:57:30.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 msgr2=0x7f0f2c156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400efc0 tx=0x7f0f2400c3a0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 shutdown_connections 2023-06-10T18:57:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c15a230 0x7f0f2c156d90 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06a580 0x7f0f2c0c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 --2- 172.21.15.104:0/3413276006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c06d810 0x7f0f2c156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 >> 172.21.15.104:0/3413276006 conn(0x7f0f2c06b8a0 msgr2=0x7f0f2c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 shutdown_connections 2023-06-10T18:57:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.494+0000 7f0f33b4b700 1 -- 172.21.15.104:0/3413276006 wait complete. 2023-06-10T18:57:30.503 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 94 2023-06-10T18:57:30.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:30.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.626+0000 7fa8e54cc700 1 Processor -- start 2023-06-10T18:57:30.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.626+0000 7fa8e54cc700 1 -- start start 2023-06-10T18:57:30.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8e54cc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa8e00ada30 0x7fa8e00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8e54cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8e00ae930 0x7fa8e00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8e54cc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e00aed70 con 0x7fa8e00ade00 2023-06-10T18:57:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8e54cc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00cf430 con 0x7fa8e00ada30 2023-06-10T18:57:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8e54cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cee70 con 0x7fa8e00ae930 2023-06-10T18:57:30.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8de7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa8e00ade00 0x7fa8e00ce0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36656/0 (socket says 172.21.15.104:36656) 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.627+0000 7fa8de7fc700 1 -- 172.21.15.104:0/1378081827 learned_addr learned my addr 172.21.15.104:0/1378081827 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1378081827 >> v2:172.21.15.114:3300/0 conn(0x7fa8e00ada30 0x7fa8e00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809418683 0 0) 0x7fa8e00aed70 con 0x7fa8e00ade00 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c80035b0 con 0x7fa8e00ade00 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8e00ae930 0x7fa8e00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090794764 0 0) 0x7fa8c80035b0 con 0x7fa8e00ade00 2023-06-10T18:57:30.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8e00ae930 msgr2=0x7fa8e00ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8e00ae930 0x7fa8e00ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 >> v2:172.21.15.114:3300/0 conn(0x7fa8e00ada30 msgr2=0x7fa8e00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 --2- 172.21.15.104:0/1378081827 >> v2:172.21.15.114:3300/0 conn(0x7fa8e00ada30 0x7fa8e00ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cf150 con 0x7fa8e00ade00 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa8d0001670 con 0x7fa8e00ade00 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 >> v1:172.21.15.104:6789/0 conn(0x7fa8e00ade00 legacy=0x7fa8e00ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8c8003a00 0x7fa8c8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8004430 0x7fa8c80048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8c8005060 0x7fa8c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cf430 con 0x7fa8c8003a00 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cee70 con 0x7fa8c8005060 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c800c000 con 0x7fa8c8004430 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8c8003a00 0x7fa8c8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8e00ae930 0x7fa8e00ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8de7fc700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8004430 0x7fa8c80048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8c8005060 0x7fa8c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 -- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8004430 msgr2=0x7fa8c80048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8004430 0x7fa8c80048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 -- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8c8005060 msgr2=0x7fa8c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8c8005060 0x7fa8c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 -- 172.21.15.104:0/1378081827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80035b0 con 0x7fa8c8003a00 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8c8005060 0x7fa8c800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.628+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8c8003a00 0x7fa8c8003e10 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009b40 tx=0x7fa8c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40653598e7afd03b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4005a20 con 0x7fa8c8003a00 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4005b80 con 0x7fa8c8003a00 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8ddffb700 1 -- 172.21.15.104:0/1378081827 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d400e6c0 con 0x7fa8c8003a00 2023-06-10T18:57:30.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1378081827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8c8003a00 msgr2=0x7fa8c8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1378081827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8c8003a00 0x7fa8c8003e10 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009b40 tx=0x7fa8c8005880 comp rx=0 tx=0).stop 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8df7fe700 1 -- 172.21.15.104:0/1378081827 reap_dead start 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1378081827 shutdown_connections 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1378081827 >> 172.21.15.104:0/1378081827 conn(0x7fa8e006b8a0 msgr2=0x7fa8e00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1378081827 shutdown_connections 2023-06-10T18:57:30.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.629+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1378081827 wait complete. 2023-06-10T18:57:30.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.630+0000 7fa8e54cc700 1 Processor -- start 2023-06-10T18:57:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.630+0000 7fa8e54cc700 1 -- start start 2023-06-10T18:57:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8003a00 0x7fa8e01f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8e00c59b0 0x7fa8e0153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e01f6440 con 0x7fa8e00b3e90 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00c4c80 con 0x7fa8e00c59b0 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8e54cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00c5520 con 0x7fa8c8003a00 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38842/0 (socket says 172.21.15.104:38842) 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8df7fe700 1 -- 172.21.15.104:0/1218428989 learned_addr learned my addr 172.21.15.104:0/1218428989 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8003a00 0x7fa8e01f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.631+0000 7fa8de7fc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8e00c59b0 0x7fa8e0153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 -- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8003a00 msgr2=0x7fa8e01f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8003a00 0x7fa8e01f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 -- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8e00c59b0 msgr2=0x7fa8e0153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8e00c59b0 0x7fa8e0153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 -- 172.21.15.104:0/1218428989 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00b4430 con 0x7fa8e00b3e90 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8df7fe700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0072060 tx=0x7fa8d8004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8007b20 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d8007c80 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e00c4ed0 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8014400 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.632+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e0154430 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.633+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0074050 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.633+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa8d801d3b0 con 0x7fa8e00b3e90 2023-06-10T18:57:30.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.634+0000 7fa8c77fe700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 0x7fa8cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.634+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fa8d8089270 con 0x7fa8e00b3e90 2023-06-10T18:57:30.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.634+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 0x7fa8cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.634+0000 7fa8deffd700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 0x7fa8cc06d190 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4000c00 tx=0x7fa8d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.638+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa8d8054170 con 0x7fa8e00b3e90 2023-06-10T18:57:30.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.828+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8e00c5660 con 0x7fa8e00b3e90 2023-06-10T18:57:30.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.829+0000 7fa8c77fe700 1 -- 172.21.15.104:0/1218428989 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18745 (secure 0 0 0) 0x7fa8d8057790 con 0x7fa8e00b3e90 2023-06-10T18:57:30.835 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:30.836 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":147,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:18.287462+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2371761194":"2023-06-11T18:57:18.273257+0000","172.21.15.114:6835/2371761194":"2023-06-11T18:57:18.273257+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:30.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 msgr2=0x7fa8cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 0x7fa8cc06d190 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4000c00 tx=0x7fa8d4008030 comp rx=0 tx=0).stop 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 msgr2=0x7fa8e0159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0072060 tx=0x7fa8d8004970 comp rx=0 tx=0).stop 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 shutdown_connections 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e00b3e90 0x7fa8e0159af0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8c8003a00 0x7fa8e01f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa8cc06ace0 0x7fa8cc06d190 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 --2- 172.21.15.104:0/1218428989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8e00c59b0 0x7fa8e0153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 >> 172.21.15.104:0/1218428989 conn(0x7fa8e006b8a0 msgr2=0x7fa8e00aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 shutdown_connections 2023-06-10T18:57:30.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.831+0000 7fa8e54cc700 1 -- 172.21.15.104:0/1218428989 wait complete. 2023-06-10T18:57:30.850 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:57:30.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:30.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.963+0000 7f9ab0562700 1 Processor -- start 2023-06-10T18:57:30.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.963+0000 7f9ab0562700 1 -- start start 2023-06-10T18:57:30.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9aa815a420 0x7f9aa80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa80b4ff0 0x7f9aa80ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9aa80ce890 0x7f9aa80b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa80ced10 con 0x7f9aa815a420 2023-06-10T18:57:30.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa80b6070 con 0x7f9aa80ce890 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9ab0562700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80b5460 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9aa815a420 0x7f9aa80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9aa815a420 0x7f9aa80b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38846/0 (socket says 172.21.15.104:38846) 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 -- 172.21.15.104:0/3456838314 learned_addr learned my addr 172.21.15.104:0/3456838314 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 -- 172.21.15.104:0/3456838314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa80b4ff0 msgr2=0x7f9aa80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/3456838314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa80b4ff0 0x7f9aa80ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 -- 172.21.15.104:0/3456838314 >> v2:172.21.15.114:3300/0 conn(0x7f9aa80ce890 msgr2=0x7f9aa80b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/3456838314 >> v2:172.21.15.114:3300/0 conn(0x7f9aa80ce890 0x7f9aa80b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.964+0000 7f9aae2fe700 1 -- 172.21.15.104:0/3456838314 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80b5740 con 0x7f9aa815a420 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/3456838314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa815a420 0x7f9aa80b1660 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f9aa808abf0 tx=0x7f9a98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2590e535012d0a27 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9aad2fc700 1 -- 172.21.15.104:0/3456838314 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800b870 con 0x7f9aa815a420 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9aad2fc700 1 -- 172.21.15.104:0/3456838314 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9800b9d0 con 0x7f9aa815a420 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9aad2fc700 1 -- 172.21.15.104:0/3456838314 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800bd60 con 0x7f9aa815a420 2023-06-10T18:57:30.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 -- 172.21.15.104:0/3456838314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa815a420 msgr2=0x7f9aa80b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 --2- 172.21.15.104:0/3456838314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa815a420 0x7f9aa80b1660 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f9aa808abf0 tx=0x7f9a98009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 -- 172.21.15.104:0/3456838314 shutdown_connections 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 --2- 172.21.15.104:0/3456838314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa815a420 0x7f9aa80b1660 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 --2- 172.21.15.104:0/3456838314 >> v2:172.21.15.114:3300/0 conn(0x7f9aa80ce890 0x7f9aa80b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 --2- 172.21.15.104:0/3456838314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa80b4ff0 0x7f9aa80ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 -- 172.21.15.104:0/3456838314 >> 172.21.15.104:0/3456838314 conn(0x7f9aa8089fd0 msgr2=0x7f9aa80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:30.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 -- 172.21.15.104:0/3456838314 shutdown_connections 2023-06-10T18:57:30.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.965+0000 7f9ab0562700 1 -- 172.21.15.104:0/3456838314 wait complete. 2023-06-10T18:57:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.966+0000 7f9ab0562700 1 Processor -- start 2023-06-10T18:57:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 -- start start 2023-06-10T18:57:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aa80ce890 0x7f9aa80c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa815a420 0x7f9aa8155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c8920 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c8080 con 0x7f9aa80ce890 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9ab0562700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c8360 con 0x7f9aa815a420 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9aae2fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9aae2fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38852/0 (socket says 172.21.15.104:38852) 2023-06-10T18:57:30.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.967+0000 7f9aae2fe700 1 -- 172.21.15.104:0/2010947255 learned_addr learned my addr 172.21.15.104:0/2010947255 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aaeaff700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa815a420 0x7f9aa8155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 -- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa815a420 msgr2=0x7f9aa8155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aadafd700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aa80ce890 0x7f9aa80c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa815a420 0x7f9aa8155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 -- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aa80ce890 msgr2=0x7f9aa80c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aa80ce890 0x7f9aa80c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 -- 172.21.15.104:0/2010947255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98009550 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9aae2fe700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8073ac0 tx=0x7f9a98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98011860 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9801f070 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa80c8640 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98011bc0 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.968+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa80c7d90 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.969+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa8074050 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.969+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a98021030 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.970+0000 7f9a9effd700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 0x7f9a8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.970+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f9a9808ac40 con 0x7f9aa80b4ff0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.970+0000 7f9aadafd700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 0x7f9a8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:30.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.971+0000 7f9aadafd700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 0x7f9a8c06d190 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80c7800 tx=0x7f9aa4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:30.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:30.974+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a98055be0 con 0x7f9aa80b4ff0 2023-06-10T18:57:31.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.203+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aa8072b10 con 0x7f9aa80b4ff0 2023-06-10T18:57:31.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.203+0000 7f9a9effd700 1 -- 172.21.15.104:0/2010947255 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f9a98059200 con 0x7f9aa80b4ff0 2023-06-10T18:57:31.210 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:31.210 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:22.712458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":4,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:31.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 msgr2=0x7f9a8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 0x7f9a8c06d190 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80c7800 tx=0x7f9aa4008040 comp rx=0 tx=0).stop 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 msgr2=0x7f9aa8154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8073ac0 tx=0x7f9a98004750 comp rx=0 tx=0).stop 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 shutdown_connections 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9aa815a420 0x7f9aa8155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa80b4ff0 0x7f9aa8154bf0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9a8c06ace0 0x7f9a8c06d190 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 --2- 172.21.15.104:0/2010947255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aa80ce890 0x7f9aa80c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 >> 172.21.15.104:0/2010947255 conn(0x7f9aa8089fd0 msgr2=0x7f9aa8157720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:31.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 shutdown_connections 2023-06-10T18:57:31.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.205+0000 7f9ab0562700 1 -- 172.21.15.104:0/2010947255 wait complete. 2023-06-10T18:57:31.214 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 94 2023-06-10T18:57:31.223 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 Processor -- start 2023-06-10T18:57:31.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 -- start start 2023-06-10T18:57:31.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f435006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435006d290 con 0x7f4350069e70 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43500ca2d0 con 0x7f435006e0e0 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.337+0000 7f435714a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4350071640 con 0x7f435006e4b0 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f43556e7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4350069e70 0x7f43500ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36692/0 (socket says 172.21.15.104:36692) 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f43556e7700 1 -- 172.21.15.104:0/1191812782 learned_addr learned my addr 172.21.15.104:0/1191812782 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434ffff700 1 --2- 172.21.15.104:0/1191812782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f435006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009164519 0 0) 0x7f435006d290 con 0x7f4350069e70 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43400035b0 con 0x7f4350069e70 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656110525 0 0) 0x7f43500ca2d0 con 0x7f435006e0e0 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435006d290 con 0x7f435006e0e0 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2071971149 0 0) 0x7f43400035b0 con 0x7f4350069e70 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434ffff700 1 -- 172.21.15.104:0/1191812782 >> v1:172.21.15.114:6789/0 conn(0x7f435006e0e0 legacy=0x7f43500c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434ffff700 1 -- 172.21.15.104:0/1191812782 >> v1:172.21.15.104:6789/0 conn(0x7f4350069e70 legacy=0x7f43500ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434ffff700 1 -- 172.21.15.104:0/1191812782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435006d4c0 con 0x7f435006e4b0 2023-06-10T18:57:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.338+0000 7f434ffff700 1 --2- 172.21.15.104:0/1191812782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f435006cd50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f433c009870 tx=0x7f433c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a54cd516196677f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43480019f0 con 0x7f435006e4b0 2023-06-10T18:57:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c004030 con 0x7f435006e4b0 2023-06-10T18:57:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f434f7fe700 1 -- 172.21.15.104:0/1191812782 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c0043c0 con 0x7f435006e4b0 2023-06-10T18:57:31.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 -- 172.21.15.104:0/1191812782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 msgr2=0x7f435006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 --2- 172.21.15.104:0/1191812782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f435006cd50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f433c009870 tx=0x7f433c009b80 comp rx=0 tx=0).stop 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 -- 172.21.15.104:0/1191812782 shutdown_connections 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 --2- 172.21.15.104:0/1191812782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f435006cd50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 -- 172.21.15.104:0/1191812782 >> 172.21.15.104:0/1191812782 conn(0x7f435006b8a0 msgr2=0x7f43500cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 -- 172.21.15.104:0/1191812782 shutdown_connections 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.339+0000 7f435714a700 1 -- 172.21.15.104:0/1191812782 wait complete. 2023-06-10T18:57:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.340+0000 7f435714a700 1 Processor -- start 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.340+0000 7f435714a700 1 -- start start 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f435006e0e0 0x7f43500b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f43500b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435015a1c0 con 0x7f4350069e70 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43500b5410 con 0x7f435006e0e0 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f435714a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43500b2ee0 con 0x7f435006e4b0 2023-06-10T18:57:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f4354ee6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f4354ee6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38866/0 (socket says 172.21.15.104:38866) 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.341+0000 7f4354ee6700 1 -- 172.21.15.104:0/775761912 learned_addr learned my addr 172.21.15.104:0/775761912 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f434ffff700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f435006e0e0 0x7f43500b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 -- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 msgr2=0x7f43500b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f43556e7700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f43500b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f43500b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 -- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f435006e0e0 msgr2=0x7f43500b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f435006e0e0 0x7f43500b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 -- 172.21.15.104:0/775761912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0094a0 con 0x7f4350069e70 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f4354ee6700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f4344001f00 tx=0x7f434400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43440052d0 con 0x7f4350069e70 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344005430 con 0x7f4350069e70 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43500b31c0 con 0x7f4350069e70 2023-06-10T18:57:31.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43440038f0 con 0x7f4350069e70 2023-06-10T18:57:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.342+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43500b3780 con 0x7f4350069e70 2023-06-10T18:57:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.343+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4350074050 con 0x7f4350069e70 2023-06-10T18:57:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.343+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4344003a50 con 0x7f4350069e70 2023-06-10T18:57:31.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.344+0000 7f434d7fa700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 0x7f433406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.344+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4344088af0 con 0x7f4350069e70 2023-06-10T18:57:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.344+0000 7f434ffff700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 0x7f433406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.345+0000 7f434ffff700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 0x7f433406d190 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f43500cba00 tx=0x7f433c013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.348+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4344053a90 con 0x7f4350069e70 2023-06-10T18:57:31.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.578+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f435006da50 con 0x7f4350069e70 2023-06-10T18:57:31.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.578+0000 7f434d7fa700 1 -- 172.21.15.104:0/775761912 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4681 (secure 0 0 0) 0x7f43440570b0 con 0x7f4350069e70 2023-06-10T18:57:31.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:31.585 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:06.271602+0000","modified":"2023-06-10T18:57:22.712458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":145,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9861},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9861":{"gid":9861,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":4,"addr":"172.21.15.104:6835/1189653810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1189653810},{"type":"v1","addr":"172.21.15.104:6835","nonce":1189653810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-06-10T18:57:31.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.580+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 msgr2=0x7f433406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.580+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 0x7f433406d190 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f43500cba00 tx=0x7f433c013040 comp rx=0 tx=0).stop 2023-06-10T18:57:31.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.580+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 msgr2=0x7f4350159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f4344001f00 tx=0x7f434400c860 comp rx=0 tx=0).stop 2023-06-10T18:57:31.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 shutdown_connections 2023-06-10T18:57:31.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f435006e4b0 0x7f43500b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4350069e70 0x7f4350159920 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f433406ace0 0x7f433406d190 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 --2- 172.21.15.104:0/775761912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f435006e0e0 0x7f43500b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 >> 172.21.15.104:0/775761912 conn(0x7f435006b8a0 msgr2=0x7f43500c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:31.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 shutdown_connections 2023-06-10T18:57:31.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.581+0000 7f435714a700 1 -- 172.21.15.104:0/775761912 wait complete. 2023-06-10T18:57:31.593 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 94 2023-06-10T18:57:31.599 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:31.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.713+0000 7f816cac3700 1 Processor -- start 2023-06-10T18:57:31.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.713+0000 7f816cac3700 1 -- start start 2023-06-10T18:57:31.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816cac3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f816815a110 0x7f81680af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816cac3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816cac3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680d0690 con 0x7f81680b2340 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816cac3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680afb00 con 0x7f816815a110 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816cac3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81680cd2d0 con 0x7f81680cfd20 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f8165d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f8165d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:58258/0 (socket says 172.21.15.104:58258) 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f816659c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f816815a110 0x7f81680af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.714+0000 7f8165d9b700 1 -- 172.21.15.104:0/474510459 learned_addr learned my addr 172.21.15.104:0/474510459 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f8165d9b700 1 -- 172.21.15.104:0/474510459 >> v2:172.21.15.114:3300/0 conn(0x7f816815a110 msgr2=0x7f81680af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f8165d9b700 1 --2- 172.21.15.104:0/474510459 >> v2:172.21.15.114:3300/0 conn(0x7f816815a110 0x7f81680af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f8165d9b700 1 -- 172.21.15.104:0/474510459 >> v1:172.21.15.114:6789/0 conn(0x7f81680b2340 legacy=0x7f81680b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f8165d9b700 1 -- 172.21.15.104:0/474510459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680cccc0 con 0x7f81680cfd20 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f8165d9b700 1 --2- 172.21.15.104:0/474510459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f81500098f0 tx=0x7f8150009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a19f9cc0e55c841e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f816559a700 1 -- 172.21.15.104:0/474510459 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8150011830 con 0x7f81680cfd20 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f816559a700 1 -- 172.21.15.104:0/474510459 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8150011990 con 0x7f81680cfd20 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.715+0000 7f816559a700 1 -- 172.21.15.104:0/474510459 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8150011d20 con 0x7f81680cfd20 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 -- 172.21.15.104:0/474510459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 msgr2=0x7f81680d0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 --2- 172.21.15.104:0/474510459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f81500098f0 tx=0x7f8150009c00 comp rx=0 tx=0).stop 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 -- 172.21.15.104:0/474510459 shutdown_connections 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 --2- 172.21.15.104:0/474510459 >> v2:172.21.15.114:3300/0 conn(0x7f816815a110 0x7f81680af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 --2- 172.21.15.104:0/474510459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680cfd20 0x7f81680d0150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 -- 172.21.15.104:0/474510459 >> 172.21.15.104:0/474510459 conn(0x7f816806e5c0 msgr2=0x7f816806e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:31.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 -- 172.21.15.104:0/474510459 shutdown_connections 2023-06-10T18:57:31.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.716+0000 7f816cac3700 1 -- 172.21.15.104:0/474510459 wait complete. 2023-06-10T18:57:31.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.717+0000 7f816cac3700 1 Processor -- start 2023-06-10T18:57:31.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.717+0000 7f816cac3700 1 -- start start 2023-06-10T18:57:31.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 0x7f81680c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 0x7f816815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816815bf40 con 0x7f81680b2340 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81680c57b0 con 0x7f81680cfd20 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816cac3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816815b0e0 con 0x7f816815a110 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38878/0 (socket says 172.21.15.104:38878) 2023-06-10T18:57:31.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 -- 172.21.15.104:0/4092822659 learned_addr learned my addr 172.21.15.104:0/4092822659 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f8166d9d700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 0x7f816815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f8165d9b700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 0x7f81680c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 -- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 msgr2=0x7f816815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 0x7f816815aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 -- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 msgr2=0x7f81680c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 0x7f81680c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 -- 172.21.15.104:0/4092822659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150009550 con 0x7f81680b2340 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f8166d9d700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 0x7f816815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f8165d9b700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 0x7f81680c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f816659c700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f815800e870 tx=0x7f815800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.718+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158009590 con 0x7f81680b2340 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.719+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816806b100 con 0x7f81680b2340 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.719+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815800ee70 con 0x7f81680b2340 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.719+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815800ca50 con 0x7f81680b2340 2023-06-10T18:57:31.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.719+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81680c5c80 con 0x7f81680b2340 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.719+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816806b320 con 0x7f81680b2340 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.720+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f815800cbb0 con 0x7f81680b2340 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.720+0000 7f815effd700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 0x7f814806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.720+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f81580896f0 con 0x7f81680b2340 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.721+0000 7f8165d9b700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 0x7f814806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:31.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.721+0000 7f8165d9b700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 0x7f814806d190 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f81680c6a00 tx=0x7f8150008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:31.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.725+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8158057f80 con 0x7f81680b2340 2023-06-10T18:57:31.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.913+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8168074050 con 0x7f81680b2340 2023-06-10T18:57:31.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.914+0000 7f815effd700 1 -- 172.21.15.104:0/4092822659 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+18745 (secure 0 0 0) 0x7f815800ce60 con 0x7f81680b2340 2023-06-10T18:57:31.921 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:31.921 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":147,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:18.287462+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:04.108341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:05.238053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2371761194":"2023-06-11T18:57:18.273257+0000","172.21.15.114:6835/2371761194":"2023-06-11T18:57:18.273257+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 msgr2=0x7f814806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 0x7f814806d190 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f81680c6a00 tx=0x7f8150008030 comp rx=0 tx=0).stop 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 msgr2=0x7f81680c6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f815800e870 tx=0x7f815800ec30 comp rx=0 tx=0).stop 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 shutdown_connections 2023-06-10T18:57:31.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f816815a110 0x7f816815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81680b2340 0x7f81680c6a70 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f814806ace0 0x7f814806d190 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 --2- 172.21.15.104:0/4092822659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81680cfd20 0x7f81680c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.916+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 >> 172.21.15.104:0/4092822659 conn(0x7f816806e5c0 msgr2=0x7f81680b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.917+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 shutdown_connections 2023-06-10T18:57:31.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:31.917+0000 7f816cac3700 1 -- 172.21.15.104:0/4092822659 wait complete. 2023-06-10T18:57:31.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:57:32.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.051+0000 7f4e7c8bd700 1 Processor -- start 2023-06-10T18:57:32.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.051+0000 7f4e7c8bd700 1 -- start start 2023-06-10T18:57:32.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.052+0000 7f4e7c8bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e780d0c00 0x7f4e7806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.052+0000 7f4e7c8bd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e7806b820 0x7f4e780cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.052+0000 7f4e7c8bd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7806bfd0 con 0x7f4e780b1e30 2023-06-10T18:57:32.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.052+0000 7f4e7c8bd700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7806c2b0 con 0x7f4e7806b820 2023-06-10T18:57:32.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.052+0000 7f4e7c8bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780ccf70 con 0x7f4e780d0c00 2023-06-10T18:57:32.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e7806b820 0x7f4e780cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.060 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e75d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e780d0c00 0x7f4e7806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e7806b820 0x7f4e780cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:54624/0 (socket says 172.21.15.104:54624) 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 -- 172.21.15.104:0/1155117165 learned_addr learned my addr 172.21.15.104:0/1155117165 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 -- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e780d0c00 msgr2=0x7f4e7806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 --2- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e780d0c00 0x7f4e7806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 -- 172.21.15.104:0/1155117165 >> v1:172.21.15.114:6789/0 conn(0x7f4e780b1e30 legacy=0x7f4e780d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 -- 172.21.15.104:0/1155117165 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780cd250 con 0x7f4e7806b820 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.053+0000 7f4e76d9d700 1 --2- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e7806b820 0x7f4e780cda50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009e50 tx=0x7f4e7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57859109a5d56b44 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7559a700 1 -- 172.21.15.104:0/1155117165 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000e070 con 0x7f4e7806b820 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7559a700 1 -- 172.21.15.104:0/1155117165 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000b910 con 0x7f4e7806b820 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7559a700 1 -- 172.21.15.104:0/1155117165 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000bcc0 con 0x7f4e7806b820 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e7806b820 msgr2=0x7f4e780cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e7806b820 0x7f4e780cda50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009e50 tx=0x7f4e7000b330 comp rx=0 tx=0).stop 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/1155117165 shutdown_connections 2023-06-10T18:57:32.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e7806b820 0x7f4e780cda50 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/1155117165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e780d0c00 0x7f4e7806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/1155117165 >> 172.21.15.104:0/1155117165 conn(0x7f4e78071ec0 msgr2=0x7f4e7809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:32.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/1155117165 shutdown_connections 2023-06-10T18:57:32.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.054+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/1155117165 wait complete. 2023-06-10T18:57:32.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 Processor -- start 2023-06-10T18:57:32.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 -- start start 2023-06-10T18:57:32.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e7806b820 0x7f4e78156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e780b1e30 0x7f4e780b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781576b0 con 0x7f4e780d0c00 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781574d0 con 0x7f4e780b1e30 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e7c8bd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780b61f0 con 0x7f4e7806b820 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e76d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.056+0000 7f4e76d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38896/0 (socket says 172.21.15.104:38896) 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 -- 172.21.15.104:0/689728413 learned_addr learned my addr 172.21.15.104:0/689728413 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e75d9b700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e780b1e30 0x7f4e780b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e7659c700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e7806b820 0x7f4e78156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 -- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e7806b820 msgr2=0x7f4e78156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e7806b820 0x7f4e78156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 -- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e780b1e30 msgr2=0x7f4e780b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e780b1e30 0x7f4e780b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 -- 172.21.15.104:0/689728413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009b30 con 0x7f4e780d0c00 2023-06-10T18:57:32.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e76d9d700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4e780b5510 tx=0x7f4e70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70003ea0 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70004920 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e780b64d0 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7001c680 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.057+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7815c790 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.058+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e780d1860 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.059+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4e7001c7e0 con 0x7f4e780d0c00 2023-06-10T18:57:32.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.059+0000 7f4e66ffd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.059+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f4e7008b210 con 0x7f4e780d0c00 2023-06-10T18:57:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.059+0000 7f4e7659c700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.060+0000 7f4e7659c700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0018a0 tx=0x7f4e6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.063+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4e700561b0 con 0x7f4e780d0c00 2023-06-10T18:57:32.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.294+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4e78074050 con 0x7f4e780d0c00 2023-06-10T18:57:32.757 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:57:32.750+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:57:32.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.758+0000 7f4e66ffd700 1 -- 172.21.15.104:0/689728413 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e700597d0 con 0x7f4e780d0c00 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 msgr2=0x7f4e5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0018a0 tx=0x7f4e6c009040 comp rx=0 tx=0).stop 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 msgr2=0x7f4e780b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4e780b5510 tx=0x7f4e70004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 shutdown_connections 2023-06-10T18:57:32.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e780d0c00 0x7f4e780b3980 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e7806b820 0x7f4e78156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 --2- 172.21.15.104:0/689728413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e780b1e30 0x7f4e780b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 >> 172.21.15.104:0/689728413 conn(0x7f4e78071ec0 msgr2=0x7f4e780b4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 shutdown_connections 2023-06-10T18:57:32.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.760+0000 7f4e7c8bd700 1 -- 172.21.15.104:0/689728413 wait complete. 2023-06-10T18:57:32.769 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:57:32.770 INFO:tasks.ceph.mds.a.smithi104.stderr: -335> 2023-06-10T18:57:25.710+0000 7ff6bbd5a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:57:32.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:57:32.795 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:57:32.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.899+0000 7f6e43014700 1 Processor -- start 2023-06-10T18:57:32.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.899+0000 7f6e43014700 1 -- start start 2023-06-10T18:57:32.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e43014700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e3c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e43014700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6e3c06cde0 0x7f6e3c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e43014700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e3c06e250 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e43014700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3c15c950 con 0x7f6e3c06cde0 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e43014700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0b1e80 con 0x7f6e3c15c580 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6e3c06e6e0 0x7f6e3c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36740/0 (socket says 172.21.15.104:36740) 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3bfff700 1 -- 172.21.15.104:0/658269156 learned_addr learned my addr 172.21.15.104:0/658269156 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:32.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e40db0700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e3c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e415b1700 1 --2- 172.21.15.104:0/658269156 >> v2:172.21.15.114:3300/0 conn(0x7f6e3c06cde0 0x7f6e3c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882262307 0 0) 0x7f6e3c06e250 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c0035b0 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299453899 0 0) 0x7f6e2c0035b0 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 msgr2=0x7f6e3c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e3c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 >> v2:172.21.15.114:3300/0 conn(0x7f6e3c06cde0 msgr2=0x7f6e3c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:32.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.900+0000 7f6e3b7fe700 1 --2- 172.21.15.104:0/658269156 >> v2:172.21.15.114:3300/0 conn(0x7f6e3c06cde0 0x7f6e3c0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c06d920 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6e28001670 con 0x7f6e3c06e6e0 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 >> v1:172.21.15.104:6789/0 conn(0x7f6e3c06e6e0 legacy=0x7f6e3c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e2c003a00 0x7f6e2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c004430 0x7f6e2c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e2c004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c15c950 con 0x7f6e2c003a00 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0b1e80 con 0x7f6e2c004430 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c00ba10 con 0x7f6e3c15c580 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e2c003a00 0x7f6e2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3bfff700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c004430 0x7f6e2c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e415b1700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e2c004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 -- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 msgr2=0x7f6e2c004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:32.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c15c580 0x7f6e2c004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 -- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c004430 msgr2=0x7f6e2c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c004430 0x7f6e2c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 -- 172.21.15.104:0/658269156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0035b0 con 0x7f6e2c003a00 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e40db0700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e2c003a00 0x7f6e2c003e10 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009b40 tx=0x7f6e2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fded8107efb23aea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30005a20 con 0x7f6e2c003a00 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e30005b80 con 0x7f6e2c003a00 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.901+0000 7f6e3b7fe700 1 -- 172.21.15.104:0/658269156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3000e6c0 con 0x7f6e2c003a00 2023-06-10T18:57:32.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 -- 172.21.15.104:0/658269156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e2c003a00 msgr2=0x7f6e2c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 --2- 172.21.15.104:0/658269156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e2c003a00 0x7f6e2c003e10 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009b40 tx=0x7f6e2c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e415b1700 1 -- 172.21.15.104:0/658269156 reap_dead start 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 -- 172.21.15.104:0/658269156 shutdown_connections 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 -- 172.21.15.104:0/658269156 >> 172.21.15.104:0/658269156 conn(0x7f6e3c06b8a0 msgr2=0x7f6e3c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 -- 172.21.15.104:0/658269156 shutdown_connections 2023-06-10T18:57:32.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.902+0000 7f6e43014700 1 -- 172.21.15.104:0/658269156 wait complete. 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.903+0000 7f6e43014700 1 Processor -- start 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.903+0000 7f6e43014700 1 -- start start 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c003a00 0x7f6e3c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c0b3aa0 0x7f6e3c0b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0b3610 con 0x7f6e3c0ad150 2023-06-10T18:57:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c158370 con 0x7f6e2c003a00 2023-06-10T18:57:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e43014700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0b2a00 con 0x7f6e3c0b3aa0 2023-06-10T18:57:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38926/0 (socket says 172.21.15.104:38926) 2023-06-10T18:57:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 -- 172.21.15.104:0/2479010822 learned_addr learned my addr 172.21.15.104:0/2479010822 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e40db0700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c003a00 0x7f6e3c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e415b1700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c0b3aa0 0x7f6e3c0b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 -- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c0b3aa0 msgr2=0x7f6e3c0b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c0b3aa0 0x7f6e3c0b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 -- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c003a00 msgr2=0x7f6e3c0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c003a00 0x7f6e3c0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 -- 172.21.15.104:0/2479010822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c0b2ce0 con 0x7f6e3c0ad150 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e3bfff700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f6e28007340 tx=0x7f6e28007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800f6c0 con 0x7f6e3c0ad150 2023-06-10T18:57:32.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.904+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800f820 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.905+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2800fbd0 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.905+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c0b2fc0 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.905+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3c0ad5c0 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.905+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3c066f10 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.906+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6e2800fd30 con 0x7f6e3c0ad150 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.906+0000 7f6e397fa700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 0x7f6e2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:32.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.906+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6554+0+0 (secure 0 0 0) 0x7f6e28089710 con 0x7f6e3c0ad150 2023-06-10T18:57:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.906+0000 7f6e40db0700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 0x7f6e2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.907+0000 7f6e40db0700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 0x7f6e2006d190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6e30000c00 tx=0x7f6e30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:32.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:32.911+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6e28058060 con 0x7f6e3c0ad150 2023-06-10T18:57:33.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.141+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6e3c074050 con 0x7f6e3c0ad150 2023-06-10T18:57:33.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.761+0000 7f6e397fa700 1 -- 172.21.15.104:0/2479010822 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e28057bf0 con 0x7f6e3c0ad150 2023-06-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 msgr2=0x7f6e2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 0x7f6e2006d190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6e30000c00 tx=0x7f6e30008030 comp rx=0 tx=0).stop 2023-06-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 msgr2=0x7f6e3c157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f6e28007340 tx=0x7f6e28007650 comp rx=0 tx=0).stop 2023-06-10T18:57:33.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 shutdown_connections 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e2006ace0 0x7f6e2006d190 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e2c003a00 0x7f6e3c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e3c0b3aa0 0x7f6e3c0b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 --2- 172.21.15.104:0/2479010822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e3c0ad150 0x7f6e3c157e30 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.764+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 >> 172.21.15.104:0/2479010822 conn(0x7f6e3c06b8a0 msgr2=0x7f6e3c1599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.765+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 shutdown_connections 2023-06-10T18:57:33.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.765+0000 7f6e43014700 1 -- 172.21.15.104:0/2479010822 wait complete. 2023-06-10T18:57:33.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:57:33.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.946+0000 7fe1a2e0c700 1 Processor -- start 2023-06-10T18:57:33.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.946+0000 7fe1a2e0c700 1 -- start start 2023-06-10T18:57:33.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a2e0c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe19c0d04f0 0x7fe19c0d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a2e0c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe19c0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a2e0c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c0af530 con 0x7fe19c0d25c0 2023-06-10T18:57:33.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a2e0c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c0af810 con 0x7fe19c0d04f0 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a2e0c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0afaf0 con 0x7fe19c0b05d0 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe19c0d25c0 0x7fe19c0d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36762/0 (socket says 172.21.15.104:36762) 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19bfff700 1 -- 172.21.15.104:0/3088857694 learned_addr learned my addr 172.21.15.104:0/3088857694 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897405525 0 0) 0x7fe19c0af530 con 0x7fe19c0d25c0 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe18c0035b0 con 0x7fe19c0d25c0 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a13a9700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe19c0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a0ba8700 1 --2- 172.21.15.104:0/3088857694 >> v2:172.21.15.114:3300/0 conn(0x7fe19c0d04f0 0x7fe19c0d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649819907 0 0) 0x7fe18c0035b0 con 0x7fe19c0d25c0 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 msgr2=0x7fe19c0b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe19c0b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 >> v2:172.21.15.114:3300/0 conn(0x7fe19c0d04f0 msgr2=0x7fe19c0d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 --2- 172.21.15.104:0/3088857694 >> v2:172.21.15.114:3300/0 conn(0x7fe19c0d04f0 0x7fe19c0d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0afdd0 con 0x7fe19c0d25c0 2023-06-10T18:57:33.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fe188001670 con 0x7fe19c0d25c0 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe1a13a9700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe19c0b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 >> v1:172.21.15.104:6789/0 conn(0x7fe19c0d25c0 legacy=0x7fe19c0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c003a00 0x7fe18c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe18c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe18c0050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0af810 con 0x7fe18c004430 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.947+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0afaf0 con 0x7fe18c003a00 2023-06-10T18:57:33.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c00ba90 con 0x7fe19c0b05d0 2023-06-10T18:57:33.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe18c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe1a0ba8700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c003a00 0x7fe18c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe1a13a9700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe18c0050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 -- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 msgr2=0x7fe18c0050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:33.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c0b05d0 0x7fe18c0050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 -- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c003a00 msgr2=0x7fe18c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c003a00 0x7fe18c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 -- 172.21.15.104:0/3088857694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0035b0 con 0x7fe18c004430 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19bfff700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe18c0048f0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fe18800c320 tx=0x7fe18c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec4932f1c275fa3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800ce60 con 0x7fe18c004430 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe188005960 con 0x7fe18c004430 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.948+0000 7fe19b7fe700 1 -- 172.21.15.104:0/3088857694 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188005d10 con 0x7fe18c004430 2023-06-10T18:57:33.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/3088857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 msgr2=0x7fe18c0048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/3088857694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe18c0048f0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fe18800c320 tx=0x7fe18c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a13a9700 1 -- 172.21.15.104:0/3088857694 reap_dead start 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/3088857694 shutdown_connections 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/3088857694 >> 172.21.15.104:0/3088857694 conn(0x7fe19c071ec0 msgr2=0x7fe19c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/3088857694 shutdown_connections 2023-06-10T18:57:33.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.949+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/3088857694 wait complete. 2023-06-10T18:57:33.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.950+0000 7fe1a2e0c700 1 Processor -- start 2023-06-10T18:57:33.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.950+0000 7fe1a2e0c700 1 -- start start 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c1f11b0 0x7fe19c1f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe19c1586e0 0x7fe19c158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1e3980 con 0x7fe18c004430 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1f1b60 con 0x7fe19c1586e0 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c159380 con 0x7fe19c1f11b0 2023-06-10T18:57:33.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38946/0 (socket says 172.21.15.104:38946) 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 -- 172.21.15.104:0/859412192 learned_addr learned my addr 172.21.15.104:0/859412192 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe19bfff700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c1f11b0 0x7fe19c1f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a13a9700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe19c1586e0 0x7fe19c158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 -- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c1f11b0 msgr2=0x7fe19c1f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c1f11b0 0x7fe19c1f1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 -- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe19c1586e0 msgr2=0x7fe19c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe19c1586e0 0x7fe19c158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 -- 172.21.15.104:0/859412192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c159660 con 0x7fe18c004430 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a0ba8700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe190011a70 tx=0x7fe190011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190011eb0 con 0x7fe18c004430 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19000f8c0 con 0x7fe18c004430 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.951+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c1e3bf0 con 0x7fe18c004430 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.952+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19000fc90 con 0x7fe18c004430 2023-06-10T18:57:33.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.952+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c159940 con 0x7fe18c004430 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.952+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c067010 con 0x7fe18c004430 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.953+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe190014440 con 0x7fe18c004430 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.953+0000 7fe1997fa700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 0x7fe18006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.953+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6554+0+0 (secure 0 0 0) 0x7fe190089c90 con 0x7fe18c004430 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.953+0000 7fe19bfff700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 0x7fe18006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.954+0000 7fe19bfff700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 0x7fe18006d190 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe188015040 tx=0x7fe18800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:33.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:33.958+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe1900585e0 con 0x7fe18c004430 2023-06-10T18:57:34.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.143+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe19c074050 con 0x7fe18c004430 2023-06-10T18:57:34.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.814+0000 7fe1997fa700 1 -- 172.21.15.104:0/859412192 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v149) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe190058170 con 0x7fe18c004430 2023-06-10T18:57:34.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 msgr2=0x7fe18006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:34.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 0x7fe18006d190 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe188015040 tx=0x7fe18800a000 comp rx=0 tx=0).stop 2023-06-10T18:57:34.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 msgr2=0x7fe19c15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:34.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe190011a70 tx=0x7fe190011d80 comp rx=0 tx=0).stop 2023-06-10T18:57:34.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 shutdown_connections 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe18006ace0 0x7fe18006d190 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe19c1586e0 0x7fe19c158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe18c004430 0x7fe19c15a850 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 --2- 172.21.15.104:0/859412192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe19c1f11b0 0x7fe19c1f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 >> 172.21.15.104:0/859412192 conn(0x7fe19c071ec0 msgr2=0x7fe19c0cba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:34.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 shutdown_connections 2023-06-10T18:57:34.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.818+0000 7fe1a2e0c700 1 -- 172.21.15.104:0/859412192 wait complete. 2023-06-10T18:57:34.826 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:57:34.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:57:34.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.967+0000 7f54059c4700 1 Processor -- start 2023-06-10T18:57:34.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.967+0000 7f54059c4700 1 -- start start 2023-06-10T18:57:34.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f54059c4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 0x7f5400160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f54059c4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54000cae10 0x7f54000cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f54059c4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54000d0340 con 0x7f540006c720 2023-06-10T18:57:34.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f54059c4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54000d1660 con 0x7f54000cae10 2023-06-10T18:57:34.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f54059c4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540015cb60 con 0x7f54000ce390 2023-06-10T18:57:34.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54000cae10 0x7f54000cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f54000cae10 0x7f54000cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38956/0 (socket says 172.21.15.104:38956) 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 -- 172.21.15.104:0/502285538 learned_addr learned my addr 172.21.15.104:0/502285538 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53feffd700 1 --2- 172.21.15.104:0/502285538 >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 0x7f5400160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 -- 172.21.15.104:0/502285538 >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 msgr2=0x7f5400160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 --2- 172.21.15.104:0/502285538 >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 0x7f5400160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 -- 172.21.15.104:0/502285538 >> v1:172.21.15.114:6789/0 conn(0x7f540006c720 legacy=0x7f54000ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:34.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 -- 172.21.15.104:0/502285538 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400161230 con 0x7f54000cae10 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53feffd700 1 --2- 172.21.15.104:0/502285538 >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 0x7f5400160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53ff7fe700 1 --2- 172.21.15.104:0/502285538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000cb210 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f54000b24b0 tx=0x7f53f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9ecd5ca700c5a1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53fdffb700 1 -- 172.21.15.104:0/502285538 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f800e070 con 0x7f54000cae10 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53fdffb700 1 -- 172.21.15.104:0/502285538 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8003a20 con 0x7f54000cae10 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.968+0000 7f53fdffb700 1 -- 172.21.15.104:0/502285538 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8003dd0 con 0x7f54000cae10 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 -- 172.21.15.104:0/502285538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 msgr2=0x7f54000cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:34.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 --2- 172.21.15.104:0/502285538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000cb210 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f54000b24b0 tx=0x7f53f800b330 comp rx=0 tx=0).stop 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 -- 172.21.15.104:0/502285538 shutdown_connections 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 --2- 172.21.15.104:0/502285538 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000cb210 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 --2- 172.21.15.104:0/502285538 >> v2:172.21.15.114:3300/0 conn(0x7f54000ce390 0x7f5400160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 -- 172.21.15.104:0/502285538 >> 172.21.15.104:0/502285538 conn(0x7f5400069e70 msgr2=0x7f540006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 -- 172.21.15.104:0/502285538 shutdown_connections 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.969+0000 7f54059c4700 1 -- 172.21.15.104:0/502285538 wait complete. 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.970+0000 7f54059c4700 1 Processor -- start 2023-06-10T18:57:34.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.971+0000 7f54059c4700 1 -- start start 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 0x7f5400157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54000ce390 0x7f54000c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400156b90 con 0x7f54000cae10 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400155f80 con 0x7f54000ce390 2023-06-10T18:57:34.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5400156260 con 0x7f540006c720 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38960/0 (socket says 172.21.15.104:38960) 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 -- 172.21.15.104:0/4287169772 learned_addr learned my addr 172.21.15.104:0/4287169772 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53ff7fe700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54000ce390 0x7f54000c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53feffd700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 0x7f5400157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 -- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 msgr2=0x7f5400157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 0x7f5400157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 -- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54000ce390 msgr2=0x7f54000c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54000ce390 0x7f54000c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 -- 172.21.15.104:0/4287169772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8009b30 con 0x7f54000cae10 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53feffd700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 0x7f5400157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53fe7fc700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f53f000b450 tx=0x7f53f000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0011830 con 0x7f54000cae10 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f0011990 con 0x7f54000cae10 2023-06-10T18:57:34.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5400156540 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0011d40 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.972+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5400157560 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.973+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5400156d00 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.974+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f53f001a400 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.974+0000 7f53e7fff700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 0x7f53ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.974+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f53f0089080 con 0x7f54000cae10 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.974+0000 7f53feffd700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 0x7f53ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:34.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.975+0000 7f53feffd700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 0x7f53ec06d190 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f53f4006fd0 tx=0x7f53f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:34.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:34.979+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f53f00541b0 con 0x7f54000cae10 2023-06-10T18:57:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.165+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5400074050 con 0x7f54000cae10 2023-06-10T18:57:35.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.819+0000 7f53e7fff700 1 -- 172.21.15.104:0/4287169772 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v150) v1 ==== 169+0+0 (secure 0 0 0) 0x7f53f00577d0 con 0x7f54000cae10 2023-06-10T18:57:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.822+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 msgr2=0x7f53ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.822+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 0x7f53ec06d190 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f53f4006fd0 tx=0x7f53f4008040 comp rx=0 tx=0).stop 2023-06-10T18:57:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.822+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 msgr2=0x7f54000ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.822+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f53f000b450 tx=0x7f53f000b760 comp rx=0 tx=0).stop 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 shutdown_connections 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53ec06ace0 0x7f53ec06d190 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54000ce390 0x7f54000c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f540006c720 0x7f5400157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 --2- 172.21.15.104:0/4287169772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54000cae10 0x7f54000ca080 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 >> 172.21.15.104:0/4287169772 conn(0x7f5400069e70 msgr2=0x7f54000ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:35.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 shutdown_connections 2023-06-10T18:57:35.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.823+0000 7f54059c4700 1 -- 172.21.15.104:0/4287169772 wait complete. 2023-06-10T18:57:35.831 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:57:35.843 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:57:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.963+0000 7ff9b85ba700 1 Processor -- start 2023-06-10T18:57:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.963+0000 7ff9b85ba700 1 -- start start 2023-06-10T18:57:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b85ba700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff9b0158fe0 0x7ff9b006d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b85ba700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00c5d80 con 0x7ff9b0158c10 2023-06-10T18:57:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b85ba700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00cec80 con 0x7ff9b00c4b90 2023-06-10T18:57:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b85ba700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b00cdbe0 con 0x7ff9b0158fe0 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b6356700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff9b0158c10 0x7ff9b00cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36794/0 (socket says 172.21.15.104:36794) 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b6356700 1 -- 172.21.15.104:0/893804531 learned_addr learned my addr 172.21.15.104:0/893804531 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281707892 0 0) 0x7ff9b00c5d80 con 0x7ff9b0158c10 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff99c0035b0 con 0x7ff9b0158c10 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> v2:172.21.15.114:3300/0 conn(0x7ff9b0158fe0 0x7ff9b006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823442950 0 0) 0x7ff99c0035b0 con 0x7ff9b0158c10 2023-06-10T18:57:35.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 >> v2:172.21.15.114:3300/0 conn(0x7ff9b0158fe0 msgr2=0x7ff9b006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 --2- 172.21.15.104:0/893804531 >> v2:172.21.15.114:3300/0 conn(0x7ff9b0158fe0 0x7ff9b006d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 >> v1:172.21.15.114:6789/0 conn(0x7ff9b00c4b90 legacy=0x7ff9b0069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00cdec0 con 0x7ff9b0158c10 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff9ac001670 con 0x7ff9b0158c10 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> v2:172.21.15.114:3300/0 conn(0x7ff9b0158fe0 0x7ff9b006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 >> v1:172.21.15.104:6789/0 conn(0x7ff9b0158c10 legacy=0x7ff9b00cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.964+0000 7ff9b5354700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c003a00 0x7ff99c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 0x7ff99c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c005200 0x7ff99c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00cdbe0 con 0x7ff99c004850 2023-06-10T18:57:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c00c190 con 0x7ff99c003a00 2023-06-10T18:57:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c00c350 con 0x7ff99c005200 2023-06-10T18:57:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 0x7ff99c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b6356700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c003a00 0x7ff99c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b6b57700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c005200 0x7ff99c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 -- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c005200 msgr2=0x7ff99c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c005200 0x7ff99c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 -- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c003a00 msgr2=0x7ff99c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c003a00 0x7ff99c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 -- 172.21.15.104:0/893804531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0035b0 con 0x7ff99c004850 2023-06-10T18:57:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b6b57700 1 -- 172.21.15.104:0/893804531 reap_dead start 2023-06-10T18:57:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 0x7ff99c004cc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00661a0 tx=0x7ff99c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513ab95f87d6ce06 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000c070 con 0x7ff99c004850 2023-06-10T18:57:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a0013610 con 0x7ff99c004850 2023-06-10T18:57:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.965+0000 7ff9b5354700 1 -- 172.21.15.104:0/893804531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0009560 con 0x7ff99c004850 2023-06-10T18:57:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 -- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 msgr2=0x7ff99c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 0x7ff99c004cc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00661a0 tx=0x7ff99c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 -- 172.21.15.104:0/893804531 shutdown_connections 2023-06-10T18:57:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/893804531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff99c004850 0x7ff99c004cc0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 -- 172.21.15.104:0/893804531 >> 172.21.15.104:0/893804531 conn(0x7ff9b00ceec0 msgr2=0x7ff9b00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:35.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 -- 172.21.15.104:0/893804531 shutdown_connections 2023-06-10T18:57:35.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.966+0000 7ff9b85ba700 1 -- 172.21.15.104:0/893804531 wait complete. 2023-06-10T18:57:35.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.967+0000 7ff9b85ba700 1 Processor -- start 2023-06-10T18:57:35.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.967+0000 7ff9b85ba700 1 -- start start 2023-06-10T18:57:35.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c004850 0x7ff9b0143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b00cb740 0x7ff9b00cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00cc0f0 con 0x7ff9b014da70 2023-06-10T18:57:35.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00b1e90 con 0x7ff99c004850 2023-06-10T18:57:35.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b0143d40 con 0x7ff9b00cb740 2023-06-10T18:57:35.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38980/0 (socket says 172.21.15.104:38980) 2023-06-10T18:57:35.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 -- 172.21.15.104:0/1820226371 learned_addr learned my addr 172.21.15.104:0/1820226371 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:35.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b6356700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c004850 0x7ff9b0143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b6b57700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b00cb740 0x7ff9b00cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 -- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b00cb740 msgr2=0x7ff9b00cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b00cb740 0x7ff9b00cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 -- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c004850 msgr2=0x7ff9b0143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c004850 0x7ff9b0143800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 -- 172.21.15.104:0/1820226371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00b2130 con 0x7ff9b014da70 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b5b55700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c160 tx=0x7ff9a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a001c630 con 0x7ff9b014da70 2023-06-10T18:57:35.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a001c7b0 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b00b2410 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a001cb60 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.968+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b00b29d0 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.969+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b00b2b60 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.970+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff9a001ccc0 con 0x7ff9b014da70 2023-06-10T18:57:35.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.970+0000 7ff9a6ffd700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 0x7ff99406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.970+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5702+0+0 (secure 0 0 0) 0x7ff9a0089df0 con 0x7ff9b014da70 2023-06-10T18:57:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.970+0000 7ff9b6356700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 0x7ff99406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.971+0000 7ff9b6356700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 0x7ff99406d190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007db0 tx=0x7ff9ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:35.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:35.974+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff9a0055140 con 0x7ff9b014da70 2023-06-10T18:57:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.160+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff9b00b2d40 con 0x7ff9b014da70 2023-06-10T18:57:36.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.164+0000 7ff9a6ffd700 1 -- 172.21.15.104:0/1820226371 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff9a0058760 con 0x7ff9b014da70 2023-06-10T18:57:36.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 msgr2=0x7ff99406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 0x7ff99406d190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007db0 tx=0x7ff9ac009040 comp rx=0 tx=0).stop 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 msgr2=0x7ff9b015bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c160 tx=0x7ff9a00047c0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 shutdown_connections 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff99406ace0 0x7ff99406d190 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c004850 0x7ff9b0143800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b00cb740 0x7ff9b00cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 --2- 172.21.15.104:0/1820226371 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9b014da70 0x7ff9b015bb30 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 >> 172.21.15.104:0/1820226371 conn(0x7ff9b00ceec0 msgr2=0x7ff9b00cc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 shutdown_connections 2023-06-10T18:57:36.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.166+0000 7ff9b85ba700 1 -- 172.21.15.104:0/1820226371 wait complete. 2023-06-10T18:57:36.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock' 2023-06-10T18:57:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.304+0000 7f3b7c77f700 1 Processor -- start 2023-06-10T18:57:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.304+0000 7f3b7c77f700 1 -- start start 2023-06-10T18:57:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7c77f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b7406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7c77f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7415cb00 con 0x7f3b740ca0b0 2023-06-10T18:57:36.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7c77f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7406ecd0 con 0x7f3b7415c730 2023-06-10T18:57:36.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7c77f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740aed80 con 0x7f3b740cc710 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7ad1c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3b740ca0b0 0x7f3b740ca550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36818/0 (socket says 172.21.15.104:36818) 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.305+0000 7f3b7ad1c700 1 -- 172.21.15.104:0/220211169 learned_addr learned my addr 172.21.15.104:0/220211169 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79d1a700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b7406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513828216 0 0) 0x7f3b6c001490 con 0x7f3b740ca0b0 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b600035b0 con 0x7f3b740ca0b0 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533871644 0 0) 0x7f3b600035b0 con 0x7f3b740ca0b0 2023-06-10T18:57:36.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 msgr2=0x7f3b7406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b7406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 >> v1:172.21.15.114:6789/0 conn(0x7f3b7415c730 legacy=0x7f3b740cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740cd280 con 0x7f3b740ca0b0 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3b6c001b80 con 0x7f3b740ca0b0 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 >> v1:172.21.15.104:6789/0 conn(0x7f3b740ca0b0 legacy=0x7f3b740ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b60003a30 0x7f3b60003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b60004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b60005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740aed80 con 0x7f3b60004d90 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6000bb20 con 0x7f3b60003a30 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6000bce0 con 0x7f3b740cc710 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b60005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7a51b700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b60003a30 0x7f3b60003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79d1a700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b60004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 -- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 msgr2=0x7f3b60004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b60004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 -- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b60003a30 msgr2=0x7f3b60003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b60003a30 0x7f3b60003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 -- 172.21.15.104:0/220211169 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600035b0 con 0x7f3b60004d90 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b79d1a700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740cc710 0x7f3b60004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:36.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.306+0000 7f3b7ad1c700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b60005200 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00a830 tx=0x7f3b60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9708525cea8f8ac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00a490 con 0x7f3b60004d90 2023-06-10T18:57:36.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c012620 con 0x7f3b60004d90 2023-06-10T18:57:36.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b79519700 1 -- 172.21.15.104:0/220211169 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0129d0 con 0x7f3b60004d90 2023-06-10T18:57:36.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 -- 172.21.15.104:0/220211169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 msgr2=0x7f3b60005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/220211169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b60005200 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00a830 tx=0x7f3b60005880 comp rx=0 tx=0).stop 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7ad1c700 1 -- 172.21.15.104:0/220211169 reap_dead start 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 -- 172.21.15.104:0/220211169 shutdown_connections 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 -- 172.21.15.104:0/220211169 >> 172.21.15.104:0/220211169 conn(0x7f3b7406b8a0 msgr2=0x7f3b740cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 -- 172.21.15.104:0/220211169 shutdown_connections 2023-06-10T18:57:36.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.307+0000 7f3b7c77f700 1 -- 172.21.15.104:0/220211169 wait complete. 2023-06-10T18:57:36.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.308+0000 7f3b7c77f700 1 Processor -- start 2023-06-10T18:57:36.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.308+0000 7f3b7c77f700 1 -- start start 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740c6890 0x7f3b7415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b7415adf0 0x7f3b740cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740af0c0 con 0x7f3b60004d90 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740c6ce0 con 0x7f3b7415adf0 2023-06-10T18:57:36.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7c77f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740af360 con 0x7f3b740c6890 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38998/0 (socket says 172.21.15.104:38998) 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 -- 172.21.15.104:0/1127779610 learned_addr learned my addr 172.21.15.104:0/1127779610 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 -- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740c6890 msgr2=0x7f3b7415a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740c6890 0x7f3b7415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 -- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b7415adf0 msgr2=0x7f3b740cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b7415adf0 0x7f3b740cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.309+0000 7f3b7a51b700 1 -- 172.21.15.104:0/1127779610 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740af640 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b7a51b700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400f820 tx=0x7f3b6400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b640052e0 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b64005440 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b740af920 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b640038f0 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.310+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b74151c30 con 0x7f3b60004d90 2023-06-10T18:57:36.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.311+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b74151df0 con 0x7f3b60004d90 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.311+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3b64003a50 con 0x7f3b60004d90 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.311+0000 7f3b6affd700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 0x7f3b5806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.312+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f3b640891e0 con 0x7f3b60004d90 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.312+0000 7f3b79d1a700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 0x7f3b5806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.312+0000 7f3b79d1a700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 0x7f3b5806d1c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3b740ae840 tx=0x7f3b70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.316+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3b64054360 con 0x7f3b60004d90 2023-06-10T18:57:36.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.579+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock"]} v 0) v1 -- 0x7f3b74074050 con 0x7f3b60004d90 2023-06-10T18:57:36.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.822+0000 7f3b6affd700 1 -- 172.21.15.104:0/1127779610 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock"]}]=0 v643) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3b64057980 con 0x7f3b60004d90 2023-06-10T18:57:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 msgr2=0x7f3b5806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 0x7f3b5806d1c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3b740ae840 tx=0x7f3b70008040 comp rx=0 tx=0).stop 2023-06-10T18:57:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 msgr2=0x7f3b741479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400f820 tx=0x7f3b6400fbe0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 shutdown_connections 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3b5806ad10 0x7f3b5806d1c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b7415adf0 0x7f3b740cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b60004d90 0x7f3b741479c0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 --2- 172.21.15.104:0/1127779610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3b740c6890 0x7f3b7415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 >> 172.21.15.104:0/1127779610 conn(0x7f3b7406b8a0 msgr2=0x7f3b7415b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 shutdown_connections 2023-06-10T18:57:36.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.824+0000 7f3b7c77f700 1 -- 172.21.15.104:0/1127779610 wait complete. 2023-06-10T18:57:36.843 INFO:tasks.cephfs_test_runner:test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:57:36.844 INFO:tasks.cephfs_test_runner:Starting test: test_filelock_eviction (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:57:36.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction' 2023-06-10T18:57:36.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.958+0000 7ff088160700 1 Processor -- start 2023-06-10T18:57:36.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.958+0000 7ff088160700 1 -- start start 2023-06-10T18:57:36.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff088160700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff08006e0e0 0x7ff08006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff088160700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff088160700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08006d290 con 0x7ff0800ca100 2023-06-10T18:57:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff088160700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080071640 con 0x7ff08006e0e0 2023-06-10T18:57:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff088160700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800aef00 con 0x7ff08006e4b0 2023-06-10T18:57:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff085efc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff08006e0e0 0x7ff08006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff0866fd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff0800ca100 0x7ff0800ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36838/0 (socket says 172.21.15.104:36838) 2023-06-10T18:57:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.959+0000 7ff0866fd700 1 -- 172.21.15.104:0/2857889007 learned_addr learned my addr 172.21.15.104:0/2857889007 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff0856fb700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff085efc700 1 -- 172.21.15.104:0/2857889007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 msgr2=0x7ff0800c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff085efc700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff085efc700 1 -- 172.21.15.104:0/2857889007 >> v1:172.21.15.104:6789/0 conn(0x7ff0800ca100 legacy=0x7ff0800ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff084efa700 1 -- 172.21.15.104:0/2857889007 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990810559 0 0) 0x7ff08006d290 con 0x7ff0800ca100 2023-06-10T18:57:36.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff085efc700 1 -- 172.21.15.104:0/2857889007 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08006d4c0 con 0x7ff08006e0e0 2023-06-10T18:57:36.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff085efc700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08006e0e0 0x7ff08006cd50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff070009820 tx=0x7ff070009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde0283d57668e5b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff084efa700 1 -- 172.21.15.104:0/2857889007 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070005e30 con 0x7ff08006e0e0 2023-06-10T18:57:36.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff084efa700 1 -- 172.21.15.104:0/2857889007 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070006030 con 0x7ff08006e0e0 2023-06-10T18:57:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.960+0000 7ff084efa700 1 -- 172.21.15.104:0/2857889007 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700063e0 con 0x7ff08006e0e0 2023-06-10T18:57:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 -- 172.21.15.104:0/2857889007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08006e0e0 msgr2=0x7ff08006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08006e0e0 0x7ff08006cd50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff070009820 tx=0x7ff070009b30 comp rx=0 tx=0).stop 2023-06-10T18:57:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 -- 172.21.15.104:0/2857889007 shutdown_connections 2023-06-10T18:57:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff08006e0e0 0x7ff08006cd50 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 --2- 172.21.15.104:0/2857889007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 -- 172.21.15.104:0/2857889007 >> 172.21.15.104:0/2857889007 conn(0x7ff08006b8a0 msgr2=0x7ff0800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 -- 172.21.15.104:0/2857889007 shutdown_connections 2023-06-10T18:57:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.961+0000 7ff088160700 1 -- 172.21.15.104:0/2857889007 wait complete. 2023-06-10T18:57:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.962+0000 7ff088160700 1 Processor -- start 2023-06-10T18:57:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.962+0000 7ff088160700 1 -- start start 2023-06-10T18:57:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 0x7ff080142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080142f80 con 0x7ff0800ca100 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080159710 con 0x7ff08006e0e0 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff088160700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff080158b00 con 0x7ff08006e4b0 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff0866fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff0866fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39016/0 (socket says 172.21.15.104:39016) 2023-06-10T18:57:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.963+0000 7ff0866fd700 1 -- 172.21.15.104:0/492971906 learned_addr learned my addr 172.21.15.104:0/492971906 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff085efc700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 0x7ff080142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0856fb700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 -- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 msgr2=0x7ff0800af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 -- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 msgr2=0x7ff080142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 0x7ff080142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 -- 172.21.15.104:0/492971906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff070009430 con 0x7ff0800ca100 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff085efc700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 0x7ff080142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0866fd700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7ff07800ccd0 tx=0x7ff07800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800ada0 con 0x7ff0800ca100 2023-06-10T18:57:36.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0780070d0 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078007480 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff080158de0 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.964+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801593a0 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.965+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff080074050 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.965+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff0780075e0 con 0x7ff0800ca100 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.966+0000 7ff0767fc700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:36.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.966+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5702+0+0 (secure 0 0 0) 0x7ff078089000 con 0x7ff0800ca100 2023-06-10T18:57:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.966+0000 7ff085efc700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.966+0000 7ff085efc700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff070013040 tx=0x7ff070008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:36.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:36.970+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff078054200 con 0x7ff0800ca100 2023-06-10T18:57:37.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.237+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction"]} v 0) v1 -- 0x7ff080159850 con 0x7ff0800ca100 2023-06-10T18:57:37.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.827+0000 7ff0767fc700 1 -- 172.21.15.104:0/492971906 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction"]}]=0 v644) v1 ==== 157+0+0 (secure 0 0 0) 0x7ff078057820 con 0x7ff0800ca100 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.829+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 msgr2=0x7ff06406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.829+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff070013040 tx=0x7ff070008070 comp rx=0 tx=0).stop 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.829+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 msgr2=0x7ff0800c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.829+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7ff07800ccd0 tx=0x7ff07800a340 comp rx=0 tx=0).stop 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 shutdown_connections 2023-06-10T18:57:37.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff06406ace0 0x7ff06406d190 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff08006e0e0 0x7ff080142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0800ca100 0x7ff0800c8080 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 --2- 172.21.15.104:0/492971906 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff08006e4b0 0x7ff0800af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 >> 172.21.15.104:0/492971906 conn(0x7ff08006b8a0 msgr2=0x7ff0800ca940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 shutdown_connections 2023-06-10T18:57:37.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.830+0000 7ff088160700 1 -- 172.21.15.104:0/492971906 wait complete. 2023-06-10T18:57:37.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:57:37.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.969+0000 7fe05bf3e700 1 Processor -- start 2023-06-10T18:57:37.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.969+0000 7fe05bf3e700 1 -- start start 2023-06-10T18:57:37.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe05bf3e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe054158fe0 0x7fe05406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:37.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe05bf3e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0540c5d80 con 0x7fe054158c10 2023-06-10T18:57:37.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe05bf3e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0540cec80 con 0x7fe0540c4b90 2023-06-10T18:57:37.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe05bf3e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0540cdbe0 con 0x7fe054158fe0 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe0594d9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe054158fe0 0x7fe05406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe059cda700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe054158c10 0x7fe0540cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36848/0 (socket says 172.21.15.104:36848) 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe059cda700 1 -- 172.21.15.104:0/3687338798 learned_addr learned my addr 172.21.15.104:0/3687338798 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe0594d9700 1 -- 172.21.15.104:0/3687338798 >> v1:172.21.15.114:6789/0 conn(0x7fe0540c4b90 legacy=0x7fe054069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe0594d9700 1 -- 172.21.15.104:0/3687338798 >> v1:172.21.15.104:6789/0 conn(0x7fe054158c10 legacy=0x7fe0540cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe0594d9700 1 -- 172.21.15.104:0/3687338798 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540cdec0 con 0x7fe054158fe0 2023-06-10T18:57:37.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe0594d9700 1 --2- 172.21.15.104:0/3687338798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe054158fe0 0x7fe05406d440 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fe044009840 tx=0x7fe044009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55c823523617921 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:37.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe058cd8700 1 -- 172.21.15.104:0/3687338798 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe044011830 con 0x7fe054158fe0 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe058cd8700 1 -- 172.21.15.104:0/3687338798 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe044011990 con 0x7fe054158fe0 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.970+0000 7fe058cd8700 1 -- 172.21.15.104:0/3687338798 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe044011d20 con 0x7fe054158fe0 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3687338798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe054158fe0 msgr2=0x7fe05406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3687338798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe054158fe0 0x7fe05406d440 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fe044009840 tx=0x7fe044009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3687338798 shutdown_connections 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3687338798 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe054158fe0 0x7fe05406d440 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3687338798 >> 172.21.15.104:0/3687338798 conn(0x7fe0540ceec0 msgr2=0x7fe0540cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:37.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3687338798 shutdown_connections 2023-06-10T18:57:37.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.971+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3687338798 wait complete. 2023-06-10T18:57:37.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.972+0000 7fe05bf3e700 1 Processor -- start 2023-06-10T18:57:37.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.972+0000 7fe05bf3e700 1 -- start start 2023-06-10T18:57:37.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:37.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe054158c10 0x7fe054157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:37.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe054158fe0 0x7fe0541f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:37.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054158270 con 0x7fe0540c4b90 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054145300 con 0x7fe054158fe0 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05bf3e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0541455e0 con 0x7fe054158c10 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39026/0 (socket says 172.21.15.104:39026) 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 -- 172.21.15.104:0/3183892017 learned_addr learned my addr 172.21.15.104:0/3183892017 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe0594d9700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe054158c10 0x7fe054157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:37.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe05a4db700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe054158fe0 0x7fe0541f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 -- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe054158c10 msgr2=0x7fe054157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe054158c10 0x7fe054157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 -- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe054158fe0 msgr2=0x7fe0541f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe054158fe0 0x7fe0541f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 -- 172.21.15.104:0/3183892017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0440094a0 con 0x7fe0540c4b90 2023-06-10T18:57:37.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.973+0000 7fe059cda700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c520 tx=0x7fe05000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:37.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.974+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050005d70 con 0x7fe0540c4b90 2023-06-10T18:57:37.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.974+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0500070d0 con 0x7fe0540c4b90 2023-06-10T18:57:37.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.974+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0541458c0 con 0x7fe0540c4b90 2023-06-10T18:57:37.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.974+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050007440 con 0x7fe0540c4b90 2023-06-10T18:57:37.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.974+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0541f2260 con 0x7fe0540c4b90 2023-06-10T18:57:37.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.975+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe054145de0 con 0x7fe0540c4b90 2023-06-10T18:57:37.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.975+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe0500075a0 con 0x7fe0540c4b90 2023-06-10T18:57:37.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.975+0000 7fe04a7fc700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 0x7fe03806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:37.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.975+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5702+0+0 (secure 0 0 0) 0x7fe0500887e0 con 0x7fe0540c4b90 2023-06-10T18:57:37.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.976+0000 7fe0594d9700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 0x7fe03806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:37.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.976+0000 7fe0594d9700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 0x7fe03806d190 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe0540661a0 tx=0x7fe04401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:37.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:37.980+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe054145de0 con 0x7fe0540c4b90 2023-06-10T18:57:38.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.167+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe054145de0 con 0x7fe0540c4b90 2023-06-10T18:57:38.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.171+0000 7fe04a7fc700 1 -- 172.21.15.104:0/3183892017 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe054145de0 con 0x7fe0540c4b90 2023-06-10T18:57:38.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 msgr2=0x7fe03806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 0x7fe03806d190 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe0540661a0 tx=0x7fe04401c040 comp rx=0 tx=0).stop 2023-06-10T18:57:38.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 msgr2=0x7fe054147d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c520 tx=0x7fe05000c8e0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 shutdown_connections 2023-06-10T18:57:38.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe03806ace0 0x7fe03806d190 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe054158fe0 0x7fe0541f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0540c4b90 0x7fe054147d40 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 --2- 172.21.15.104:0/3183892017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe054158c10 0x7fe054157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 >> 172.21.15.104:0/3183892017 conn(0x7fe0540ceec0 msgr2=0x7fe054153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 shutdown_connections 2023-06-10T18:57:38.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.173+0000 7fe05bf3e700 1 -- 172.21.15.104:0/3183892017 wait complete. 2023-06-10T18:57:38.191 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:57:38.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:38.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:57:38.280 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:57:38.281 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:38.281 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:38.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:57:38.299 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:38.299 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:38.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:57:38.319 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:38.320 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:38.354 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:57:38.355 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:38.355 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:38.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:57:38.379 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:57:38.379 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:38.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.508+0000 7f35269f5700 1 Processor -- start 2023-06-10T18:57:38.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.508+0000 7f35269f5700 1 -- start start 2023-06-10T18:57:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f35269f5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f352006a8c0 0x7f35201587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f35269f5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3520069e70 con 0x7f3520070a20 2023-06-10T18:57:38.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f35269f5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35201595f0 con 0x7f3520158ce0 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f35269f5700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35201598d0 con 0x7f352006a8c0 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f352006a8c0 0x7f35201587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3520070a20 0x7f3520070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36862/0 (socket says 172.21.15.104:36862) 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351ffff700 1 -- 172.21.15.104:0/1084128171 learned_addr learned my addr 172.21.15.104:0/1084128171 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351f7fe700 1 -- 172.21.15.104:0/1084128171 >> v1:172.21.15.114:6789/0 conn(0x7f3520158ce0 legacy=0x7f3520159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351f7fe700 1 -- 172.21.15.104:0/1084128171 >> v1:172.21.15.104:6789/0 conn(0x7f3520070a20 legacy=0x7f3520070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351effd700 1 -- 172.21.15.104:0/1084128171 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799875856 0 0) 0x7f3520069e70 con 0x7f3520070a20 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351f7fe700 1 -- 172.21.15.104:0/1084128171 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352006a0c0 con 0x7f352006a8c0 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351f7fe700 1 --2- 172.21.15.104:0/1084128171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201587a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f351000a9e0 tx=0x7f351000acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c07d7eabdc32057 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351effd700 1 -- 172.21.15.104:0/1084128171 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000cb70 con 0x7f352006a8c0 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351effd700 1 -- 172.21.15.104:0/1084128171 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351000ccd0 con 0x7f352006a8c0 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.509+0000 7f351effd700 1 -- 172.21.15.104:0/1084128171 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510018620 con 0x7f352006a8c0 2023-06-10T18:57:38.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 -- 172.21.15.104:0/1084128171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 msgr2=0x7f35201587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 --2- 172.21.15.104:0/1084128171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201587a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f351000a9e0 tx=0x7f351000acf0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 -- 172.21.15.104:0/1084128171 shutdown_connections 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 --2- 172.21.15.104:0/1084128171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201587a0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 -- 172.21.15.104:0/1084128171 >> 172.21.15.104:0/1084128171 conn(0x7f35200b1610 msgr2=0x7f35200ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 -- 172.21.15.104:0/1084128171 shutdown_connections 2023-06-10T18:57:38.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.510+0000 7f35269f5700 1 -- 172.21.15.104:0/1084128171 wait complete. 2023-06-10T18:57:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.511+0000 7f35269f5700 1 Processor -- start 2023-06-10T18:57:38.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.511+0000 7f35269f5700 1 -- start start 2023-06-10T18:57:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3520070a20 0x7f35201e1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3520158ce0 0x7f35201e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200cb420 con 0x7f352006a8c0 2023-06-10T18:57:38.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200c9bb0 con 0x7f3520158ce0 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f35269f5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3520148560 con 0x7f3520070a20 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39040/0 (socket says 172.21.15.104:39040) 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 -- 172.21.15.104:0/2515005015 learned_addr learned my addr 172.21.15.104:0/2515005015 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f3524f92700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3520158ce0 0x7f35201e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 -- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3520070a20 msgr2=0x7f35201e1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3520070a20 0x7f35201e1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 -- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3520158ce0 msgr2=0x7f35201e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3520158ce0 0x7f35201e1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.512+0000 7f351ffff700 1 -- 172.21.15.104:0/2515005015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351000a640 con 0x7f352006a8c0 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f351ffff700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f351400c520 tx=0x7f351400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514004c80 con 0x7f352006a8c0 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514004e00 con 0x7f352006a8c0 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514007310 con 0x7f352006a8c0 2023-06-10T18:57:38.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35200cb140 con 0x7f352006a8c0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.513+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35200c9840 con 0x7f352006a8c0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.514+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3520071780 con 0x7f352006a8c0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.514+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3514007470 con 0x7f352006a8c0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.514+0000 7f351cff9700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.515+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5702+0+0 (secure 0 0 0) 0x7f3514089050 con 0x7f352006a8c0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.515+0000 7f351f7fe700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.515+0000 7f351f7fe700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f352006de10 tx=0x7f3510011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.519+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3514054260 con 0x7f352006a8c0 2023-06-10T18:57:38.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.750+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35200c93c0 con 0x7f352006a8c0 2023-06-10T18:57:38.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.751+0000 7f351cff9700 1 -- 172.21.15.104:0/2515005015 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3536 (secure 0 0 0) 0x7f35200c93c0 con 0x7f352006a8c0 2023-06-10T18:57:38.757 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:38.758 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/536785816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":536785816},{"type":"v1","addr":"172.21.15.114:6836","nonce":536785816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":7259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/656120783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":656120783},{"type":"v1","addr":"172.21.15.114:6837","nonce":656120783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":9858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/2625740841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2625740841},{"type":"v1","addr":"172.21.15.104:6834","nonce":2625740841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10041,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3352797876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3352797876},{"type":"v1","addr":"172.21.15.104:6835","nonce":3352797876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[]} 2023-06-10T18:57:38.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 msgr2=0x7f350c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f352006de10 tx=0x7f3510011000 comp rx=0 tx=0).stop 2023-06-10T18:57:38.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 msgr2=0x7f35201ef170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f351400c520 tx=0x7f351400c8e0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 shutdown_connections 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3520158ce0 0x7f35201e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f352006a8c0 0x7f35201ef170 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 --2- 172.21.15.104:0/2515005015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3520070a20 0x7f35201e1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 >> 172.21.15.104:0/2515005015 conn(0x7f35200b1610 msgr2=0x7f35200cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:38.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 shutdown_connections 2023-06-10T18:57:38.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.753+0000 7f35269f5700 1 -- 172.21.15.104:0/2515005015 wait complete. 2023-06-10T18:57:38.761 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 96 2023-06-10T18:57:38.773 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:57:38.774 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:57:38.775 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:57:38.775 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:38.775 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:57:38.775 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:57:38.776 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:57:38.776 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:38.777 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:57:38.777 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:57:38.778 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:57:38.778 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:38.778 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:57:38.778 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:57:38.779 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:57:38.779 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:57:38.783 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:57:38.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:57:38.785 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:57:38.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:57:38.791 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:57:38.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:57:38.793 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:57:38.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:57:38.826 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:57:38.827 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:38.828 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:57:38.828 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:38.834 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:57:38.835 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:38.835 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:57:38.836 DEBUG:teuthology.parallel:result is None 2023-06-10T18:57:38.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:57:38.905 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:57:38.905 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:57:38.920 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:57:38.921 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:57:38.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.961+0000 7fc761e22700 1 Processor -- start 2023-06-10T18:57:38.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.961+0000 7fc761e22700 1 -- start start 2023-06-10T18:57:38.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc761e22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc75c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc761e22700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc75c0ce890 0x7fc75c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc761e22700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c0b6070 con 0x7fc75c15a420 2023-06-10T18:57:38.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc761e22700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75c0ced10 con 0x7fc75c0ce890 2023-06-10T18:57:38.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc761e22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0b1d70 con 0x7fc75c0b4ff0 2023-06-10T18:57:38.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc75c15a420 0x7fc75c0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36900/0 (socket says 172.21.15.104:36900) 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75b7fe700 1 -- 172.21.15.104:0/948907500 learned_addr learned my addr 172.21.15.104:0/948907500 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972600115 0 0) 0x7fc75c0b6070 con 0x7fc75c15a420 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75bfff700 1 --2- 172.21.15.104:0/948907500 >> v2:172.21.15.114:3300/0 conn(0x7fc75c0ce890 0x7fc75c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75affd700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc75c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7440035b0 con 0x7fc75c15a420 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873711413 0 0) 0x7fc7440035b0 con 0x7fc75c15a420 2023-06-10T18:57:38.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 msgr2=0x7fc75c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc75c0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 >> v2:172.21.15.114:3300/0 conn(0x7fc75c0ce890 msgr2=0x7fc75c15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 --2- 172.21.15.104:0/948907500 >> v2:172.21.15.114:3300/0 conn(0x7fc75c0ce890 0x7fc75c15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c0b5460 con 0x7fc75c15a420 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc750001670 con 0x7fc75c15a420 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 >> v1:172.21.15.104:6789/0 conn(0x7fc75c15a420 legacy=0x7fc75c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc744003a00 0x7fc744003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc744004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.962+0000 7fc75a7fc700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc744004d90 0x7fc744005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0ced10 con 0x7fc744004d90 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0b1d70 con 0x7fc744003a00 2023-06-10T18:57:38.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74400ba10 con 0x7fc75c0b4ff0 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75b7fe700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc744003a00 0x7fc744003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc744004d90 0x7fc744005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75affd700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc744004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 -- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 msgr2=0x7fc744004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c0b4ff0 0x7fc744004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 -- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc744003a00 msgr2=0x7fc744003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc744003a00 0x7fc744003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 -- 172.21.15.104:0/948907500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440035b0 con 0x7fc744004d90 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75bfff700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc744004d90 0x7fc744005200 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fc754009fc0 tx=0x7fc744005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8780a0ea022dc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75400ebd0 con 0x7fc744004d90 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75400f040 con 0x7fc744004d90 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.963+0000 7fc75a7fc700 1 -- 172.21.15.104:0/948907500 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754014480 con 0x7fc744004d90 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 -- 172.21.15.104:0/948907500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc744004d90 msgr2=0x7fc744005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 --2- 172.21.15.104:0/948907500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc744004d90 0x7fc744005200 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fc754009fc0 tx=0x7fc744005880 comp rx=0 tx=0).stop 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc75bfff700 1 -- 172.21.15.104:0/948907500 reap_dead start 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 -- 172.21.15.104:0/948907500 shutdown_connections 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 -- 172.21.15.104:0/948907500 >> 172.21.15.104:0/948907500 conn(0x7fc75c089fd0 msgr2=0x7fc75c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 -- 172.21.15.104:0/948907500 shutdown_connections 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.964+0000 7fc761e22700 1 -- 172.21.15.104:0/948907500 wait complete. 2023-06-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.965+0000 7fc761e22700 1 Processor -- start 2023-06-10T18:57:38.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.965+0000 7fc761e22700 1 -- start start 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc75c14fae0 0x7fc75c0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c06c6e0 0x7fc75c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0cbdb0 con 0x7fc75c153a30 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c06cb80 con 0x7fc75c14fae0 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc761e22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c06ce60 con 0x7fc75c06c6e0 2023-06-10T18:57:38.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39086/0 (socket says 172.21.15.104:39086) 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 -- 172.21.15.104:0/3778653932 learned_addr learned my addr 172.21.15.104:0/3778653932 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75bfff700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c06c6e0 0x7fc75c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75b7fe700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc75c14fae0 0x7fc75c0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 -- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c06c6e0 msgr2=0x7fc75c0c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c06c6e0 0x7fc75c0c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 -- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc75c14fae0 msgr2=0x7fc75c0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:38.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc75c14fae0 0x7fc75c0cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.966+0000 7fc75affd700 1 -- 172.21.15.104:0/3778653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c0cad90 con 0x7fc75c153a30 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc75affd700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fc74c011aa0 tx=0x7fc74c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c011ee0 con 0x7fc75c153a30 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c00f940 con 0x7fc75c153a30 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c0cb070 con 0x7fc75c153a30 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c00fcf0 con 0x7fc75c153a30 2023-06-10T18:57:38.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.967+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c14feb0 con 0x7fc75c153a30 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.968+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c0cb590 con 0x7fc75c153a30 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.968+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc74c014440 con 0x7fc75c153a30 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.968+0000 7fc743fff700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 0x7fc74806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.969+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5702+0+0 (secure 0 0 0) 0x7fc74c089a50 con 0x7fc75c153a30 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.969+0000 7fc75b7fe700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 0x7fc74806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.969+0000 7fc75b7fe700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 0x7fc74806d190 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc750001050 tx=0x7fc750007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:38.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:38.973+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc75c0cb590 con 0x7fc75c153a30 2023-06-10T18:57:39.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.173+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc75c0cb590 con 0x7fc75c153a30 2023-06-10T18:57:39.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:57:39.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:57:39.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.963+0000 7fc743fff700 1 -- 172.21.15.104:0/3778653932 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc75c0cb590 con 0x7fc75c153a30 2023-06-10T18:57:39.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 msgr2=0x7fc74806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 0x7fc74806d190 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc750001050 tx=0x7fc750007980 comp rx=0 tx=0).stop 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 msgr2=0x7fc75c06c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fc74c011aa0 tx=0x7fc74c011db0 comp rx=0 tx=0).stop 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 shutdown_connections 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc74806ace0 0x7fc74806d190 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc75c14fae0 0x7fc75c0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:39.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc75c06c6e0 0x7fc75c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:39.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 --2- 172.21.15.104:0/3778653932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc75c153a30 0x7fc75c06c1a0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:39.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 >> 172.21.15.104:0/3778653932 conn(0x7fc75c089fd0 msgr2=0x7fc75c157410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:39.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 shutdown_connections 2023-06-10T18:57:39.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:39.966+0000 7fc761e22700 1 -- 172.21.15.104:0/3778653932 wait complete. 2023-06-10T18:57:39.974 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:57:39.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:57:40.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 Processor -- start 2023-06-10T18:57:40.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 -- start start 2023-06-10T18:57:40.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc4a4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a40d0340 con 0x7fc4a40cae10 2023-06-10T18:57:40.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a415cb60 con 0x7fc4a406c720 2023-06-10T18:57:40.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.103+0000 7fc4a986a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40d1660 con 0x7fc4a40ce390 2023-06-10T18:57:40.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a37fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc4a40cae10 0x7fc4a40cb250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36926/0 (socket says 172.21.15.104:36926) 2023-06-10T18:57:40.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a37fe700 1 -- 172.21.15.104:0/717121418 learned_addr learned my addr 172.21.15.104:0/717121418 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:40.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a2ffd700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc4a4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868390427 0 0) 0x7fc4a40d0340 con 0x7fc4a40cae10 2023-06-10T18:57:40.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c0035b0 con 0x7fc4a40cae10 2023-06-10T18:57:40.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095321225 0 0) 0x7fc48c0035b0 con 0x7fc4a40cae10 2023-06-10T18:57:40.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 msgr2=0x7fc4a4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:40.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc4a4160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 >> v1:172.21.15.114:6789/0 conn(0x7fc4a406c720 legacy=0x7fc4a40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4161230 con 0x7fc4a40cae10 2023-06-10T18:57:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc49c0019f0 con 0x7fc4a40cae10 2023-06-10T18:57:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 >> v1:172.21.15.104:6789/0 conn(0x7fc4a40cae10 legacy=0x7fc4a40cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc48c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48c003f70 0x7fc48c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48c004d90 0x7fc48c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40d1660 con 0x7fc48c004d90 2023-06-10T18:57:40.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c00ba40 con 0x7fc48c003f70 2023-06-10T18:57:40.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c00bc00 con 0x7fc4a40ce390 2023-06-10T18:57:40.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48c004d90 0x7fc48c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a2ffd700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc48c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.104+0000 7fc4a27fc700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48c003f70 0x7fc48c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 -- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 msgr2=0x7fc48c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40ce390 0x7fc48c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 -- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48c003f70 msgr2=0x7fc48c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48c003f70 0x7fc48c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 -- 172.21.15.104:0/717121418 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0035b0 con 0x7fc48c004d90 2023-06-10T18:57:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a27fc700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc48c003f70 0x7fc48c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:40.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48c004d90 0x7fc48c005200 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00a830 tx=0x7fc48c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94321d747b9184bb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c012600 con 0x7fc48c004d90 2023-06-10T18:57:40.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c012780 con 0x7fc48c004d90 2023-06-10T18:57:40.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a1ffb700 1 -- 172.21.15.104:0/717121418 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c012b30 con 0x7fc48c004d90 2023-06-10T18:57:40.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 -- 172.21.15.104:0/717121418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48c004d90 msgr2=0x7fc48c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 --2- 172.21.15.104:0/717121418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc48c004d90 0x7fc48c005200 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00a830 tx=0x7fc48c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a37fe700 1 -- 172.21.15.104:0/717121418 reap_dead start 2023-06-10T18:57:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 -- 172.21.15.104:0/717121418 shutdown_connections 2023-06-10T18:57:40.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 -- 172.21.15.104:0/717121418 >> 172.21.15.104:0/717121418 conn(0x7fc4a4069e70 msgr2=0x7fc4a406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 -- 172.21.15.104:0/717121418 shutdown_connections 2023-06-10T18:57:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.105+0000 7fc4a986a700 1 -- 172.21.15.104:0/717121418 wait complete. 2023-06-10T18:57:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 Processor -- start 2023-06-10T18:57:40.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 -- start start 2023-06-10T18:57:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a40ca190 0x7fc4a40c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40c8140 0x7fc4a40b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40c61a0 con 0x7fc4a40b5e30 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40b4440 con 0x7fc4a40ca190 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.107+0000 7fc4a986a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40b62b0 con 0x7fc4a40c8140 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39110/0 (socket says 172.21.15.104:39110) 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 -- 172.21.15.104:0/3317423902 learned_addr learned my addr 172.21.15.104:0/3317423902 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:40.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a27fc700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40c8140 0x7fc4a40b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a2ffd700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a40ca190 0x7fc4a40c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 -- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40c8140 msgr2=0x7fc4a40b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40c8140 0x7fc4a40b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 -- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a40ca190 msgr2=0x7fc4a40c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a40ca190 0x7fc4a40c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 -- 172.21.15.104:0/3317423902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a40c5280 con 0x7fc4a40b5e30 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a37fe700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00ba10 tx=0x7fc49c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c012d00 con 0x7fc4a40b5e30 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c012e60 con 0x7fc4a40b5e30 2023-06-10T18:57:40.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a40c5520 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c01e670 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.108+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a40c5ae0 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.109+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a40cac60 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.110+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc49c01e7d0 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.110+0000 7fc48bfff700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 0x7fc49006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.110+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc49c08b000 con 0x7fc4a40b5e30 2023-06-10T18:57:40.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.110+0000 7fc4a2ffd700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 0x7fc49006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.111+0000 7fc4a2ffd700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 0x7fc49006d190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4154ae0 tx=0x7fc498009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.114+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc49c056390 con 0x7fc4a40b5e30 2023-06-10T18:57:40.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.377+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc4a40c62e0 con 0x7fc4a40b5e30 2023-06-10T18:57:40.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.377+0000 7fc48bfff700 1 -- 172.21.15.104:0/3317423902 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc4a40c62e0 con 0x7fc4a40b5e30 2023-06-10T18:57:40.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:40.384 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:40.384 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:57:40.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:40.385 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.386 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.387 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:57:40.387 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:57:40.387 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.388 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.388 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:40.388 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:40.389 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.390 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:57:40.390 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:57:40.391 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.391 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.391 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:40.391 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:40.392 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.393 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:57:40.393 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:57:40.393 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.394 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.394 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:40.394 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:40.394 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.396 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:57:40.396 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:57:40.396 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:57:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:57:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:57:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:57:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:57:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:57:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:57:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:57:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:57:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:57:40.420 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:40.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 msgr2=0x7fc49006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 0x7fc49006d190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4154ae0 tx=0x7fc498009210 comp rx=0 tx=0).stop 2023-06-10T18:57:40.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 msgr2=0x7fc4a41f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00ba10 tx=0x7fc49c001d80 comp rx=0 tx=0).stop 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 shutdown_connections 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc49006ace0 0x7fc49006d190 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a40ca190 0x7fc4a40c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc4a40b5e30 0x7fc4a41f59f0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 --2- 172.21.15.104:0/3317423902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc4a40c8140 0x7fc4a40b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 >> 172.21.15.104:0/3317423902 conn(0x7fc4a4069e70 msgr2=0x7fc4a40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:40.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 shutdown_connections 2023-06-10T18:57:40.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.379+0000 7fc4a986a700 1 -- 172.21.15.104:0/3317423902 wait complete. 2023-06-10T18:57:40.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:40.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.537+0000 7f8abbc11700 1 Processor -- start 2023-06-10T18:57:40.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.537+0000 7f8abbc11700 1 -- start start 2023-06-10T18:57:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8abbc11700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ab40cf100 0x7f8ab406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8abbc11700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab406d290 con 0x7f8ab406e4b0 2023-06-10T18:57:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8abbc11700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab40ca2d0 con 0x7f8ab406e0e0 2023-06-10T18:57:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8abbc11700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab4071640 con 0x7f8ab40cf100 2023-06-10T18:57:40.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8aba1ae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ab40cf100 0x7f8ab406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8ab91ac700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ab406e4b0 0x7f8ab40ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36944/0 (socket says 172.21.15.104:36944) 2023-06-10T18:57:40.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8ab91ac700 1 -- 172.21.15.104:0/1220047215 learned_addr learned my addr 172.21.15.104:0/1220047215 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:40.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8aba1ae700 1 -- 172.21.15.104:0/1220047215 >> v1:172.21.15.114:6789/0 conn(0x7f8ab406e0e0 legacy=0x7f8ab40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8aba1ae700 1 -- 172.21.15.104:0/1220047215 >> v1:172.21.15.104:6789/0 conn(0x7f8ab406e4b0 legacy=0x7f8ab40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8aba1ae700 1 -- 172.21.15.104:0/1220047215 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab406d4c0 con 0x7f8ab40cf100 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8ab89ab700 1 -- 172.21.15.104:0/1220047215 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851546675 0 0) 0x7f8ab406d290 con 0x7f8ab406e4b0 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8aba1ae700 1 --2- 172.21.15.104:0/1220047215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40cf100 0x7f8ab406cd50 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009d90 tx=0x7f8aac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5974a1580a480dcb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8ab89ab700 1 -- 172.21.15.104:0/1220047215 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac00e070 con 0x7f8ab40cf100 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.538+0000 7f8ab89ab700 1 -- 172.21.15.104:0/1220047215 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac00bbd0 con 0x7f8ab40cf100 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8ab89ab700 1 -- 172.21.15.104:0/1220047215 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac012430 con 0x7f8ab40cf100 2023-06-10T18:57:40.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 -- 172.21.15.104:0/1220047215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40cf100 msgr2=0x7f8ab406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 --2- 172.21.15.104:0/1220047215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40cf100 0x7f8ab406cd50 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009d90 tx=0x7f8aac00b330 comp rx=0 tx=0).stop 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 -- 172.21.15.104:0/1220047215 shutdown_connections 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 --2- 172.21.15.104:0/1220047215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40cf100 0x7f8ab406cd50 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 -- 172.21.15.104:0/1220047215 >> 172.21.15.104:0/1220047215 conn(0x7f8ab406b8a0 msgr2=0x7f8ab40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 -- 172.21.15.104:0/1220047215 shutdown_connections 2023-06-10T18:57:40.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.539+0000 7f8abbc11700 1 -- 172.21.15.104:0/1220047215 wait complete. 2023-06-10T18:57:40.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.540+0000 7f8abbc11700 1 Processor -- start 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.540+0000 7f8abbc11700 1 -- start start 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406e0e0 0x7f8ab4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40cf100 0x7f8ab40b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b2fe0 con 0x7f8ab406e4b0 2023-06-10T18:57:40.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab41f59d0 con 0x7f8ab406e0e0 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8abbc11700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b26b0 con 0x7f8ab40cf100 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39122/0 (socket says 172.21.15.104:39122) 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 -- 172.21.15.104:0/344367878 learned_addr learned my addr 172.21.15.104:0/344367878 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8aba1ae700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40cf100 0x7f8ab40b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 -- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40cf100 msgr2=0x7f8ab40b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.553 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40cf100 0x7f8ab40b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab99ad700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406e0e0 0x7f8ab4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.541+0000 7f8ab91ac700 1 -- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406e0e0 msgr2=0x7f8ab4157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8ab91ac700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406e0e0 0x7f8ab4157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8ab91ac700 1 -- 172.21.15.104:0/344367878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009a70 con 0x7f8ab406e4b0 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8ab91ac700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400c520 tx=0x7f8aa400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4004c80 con 0x7f8ab406e4b0 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa4004e00 con 0x7f8ab406e4b0 2023-06-10T18:57:40.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab40b2990 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4007310 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.542+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab4157a60 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.543+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab40b3150 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.543+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8aa4007470 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.544+0000 7f8aaa7fc700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 0x7f8a9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.544+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8aa4089030 con 0x7f8ab406e4b0 2023-06-10T18:57:40.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.544+0000 7f8ab99ad700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 0x7f8a9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.544+0000 7f8ab99ad700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 0x7f8a9806d190 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0000fb0 tx=0x7f8ab0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.548+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8aa40542f0 con 0x7f8ab406e4b0 2023-06-10T18:57:40.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.735+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ab4074050 con 0x7f8ab406e4b0 2023-06-10T18:57:40.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.736+0000 7f8aaa7fc700 1 -- 172.21.15.104:0/344367878 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+14902 (secure 0 0 0) 0x7f8aa4057910 con 0x7f8ab406e4b0 2023-06-10T18:57:40.743 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:40.743 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":155,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:39.958334+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:40.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 msgr2=0x7f8a9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 0x7f8a9806d190 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0000fb0 tx=0x7f8ab0009040 comp rx=0 tx=0).stop 2023-06-10T18:57:40.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 msgr2=0x7f8ab409ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400c520 tx=0x7f8aa400c8e0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 shutdown_connections 2023-06-10T18:57:40.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9806ace0 0x7f8a9806d190 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406e0e0 0x7f8ab4157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40cf100 0x7f8ab40b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 --2- 172.21.15.104:0/344367878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406e4b0 0x7f8ab409ae10 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 >> 172.21.15.104:0/344367878 conn(0x7f8ab406b8a0 msgr2=0x7f8ab415bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:40.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 shutdown_connections 2023-06-10T18:57:40.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.738+0000 7f8abbc11700 1 -- 172.21.15.104:0/344367878 wait complete. 2023-06-10T18:57:40.757 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:57:40.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:57:40.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 Processor -- start 2023-06-10T18:57:40.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 -- start start 2023-06-10T18:57:40.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f47a006c800 0x7f47a006cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00cec40 con 0x7f47a0143a40 2023-06-10T18:57:40.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00cca90 con 0x7f47a006c800 2023-06-10T18:57:40.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.875+0000 7f47a64b3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00ce960 con 0x7f47a0070a10 2023-06-10T18:57:40.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f47a006c800 0x7f47a006cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f47a006c800 0x7f47a006cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39132/0 (socket says 172.21.15.104:39132) 2023-06-10T18:57:40.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 -- 172.21.15.104:0/1120000615 learned_addr learned my addr 172.21.15.104:0/1120000615 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f479ffff700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 -- 172.21.15.104:0/1120000615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 msgr2=0x7f47a006e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:40.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a006e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 -- 172.21.15.104:0/1120000615 >> v1:172.21.15.114:6789/0 conn(0x7f47a0143a40 legacy=0x7f47a0070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 -- 172.21.15.104:0/1120000615 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a006a9b0 con 0x7f47a006c800 2023-06-10T18:57:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f47a4a50700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a006cc80 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e50 tx=0x7f479800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185892085c54ea95 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f479effd700 1 -- 172.21.15.104:0/1120000615 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479800e070 con 0x7f47a006c800 2023-06-10T18:57:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f479effd700 1 -- 172.21.15.104:0/1120000615 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800b9c0 con 0x7f47a006c800 2023-06-10T18:57:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.876+0000 7f479effd700 1 -- 172.21.15.104:0/1120000615 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479800bd70 con 0x7f47a006c800 2023-06-10T18:57:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 -- 172.21.15.104:0/1120000615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 msgr2=0x7f47a006cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a006cc80 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f4798009e50 tx=0x7f479800b330 comp rx=0 tx=0).stop 2023-06-10T18:57:40.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 -- 172.21.15.104:0/1120000615 shutdown_connections 2023-06-10T18:57:40.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a006cc80 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 --2- 172.21.15.104:0/1120000615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a006e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 -- 172.21.15.104:0/1120000615 >> 172.21.15.104:0/1120000615 conn(0x7f47a00ceec0 msgr2=0x7f47a00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 -- 172.21.15.104:0/1120000615 shutdown_connections 2023-06-10T18:57:40.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.877+0000 7f47a64b3700 1 -- 172.21.15.104:0/1120000615 wait complete. 2023-06-10T18:57:40.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.878+0000 7f47a64b3700 1 Processor -- start 2023-06-10T18:57:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.878+0000 7f47a64b3700 1 -- start start 2023-06-10T18:57:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a013fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 0x7f47a0158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b39d0 con 0x7f47a006c800 2023-06-10T18:57:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b3e00 con 0x7f47a0143a40 2023-06-10T18:57:40.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a64b3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b40a0 con 0x7f47a0070a10 2023-06-10T18:57:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39134/0 (socket says 172.21.15.104:39134) 2023-06-10T18:57:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 -- 172.21.15.104:0/44481483 learned_addr learned my addr 172.21.15.104:0/44481483 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:40.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479f7fe700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a013fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a4a50700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 0x7f47a0158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 -- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 msgr2=0x7f47a013fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a013fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 -- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 msgr2=0x7f47a0158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 0x7f47a0158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f479ffff700 1 -- 172.21.15.104:0/44481483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009b30 con 0x7f47a006c800 2023-06-10T18:57:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.879+0000 7f47a4a50700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 0x7f47a0158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:40.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f479ffff700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f479000d800 tx=0x7f479000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47900125f0 con 0x7f47a006c800 2023-06-10T18:57:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790012750 con 0x7f47a006c800 2023-06-10T18:57:40.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a00cd0a0 con 0x7f47a006c800 2023-06-10T18:57:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790012b00 con 0x7f47a006c800 2023-06-10T18:57:40.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.880+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a00b4660 con 0x7f47a006c800 2023-06-10T18:57:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.881+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00b47f0 con 0x7f47a006c800 2023-06-10T18:57:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.881+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4790012c60 con 0x7f47a006c800 2023-06-10T18:57:40.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.881+0000 7f479cff9700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 0x7f478c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.881+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f47900895c0 con 0x7f47a006c800 2023-06-10T18:57:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.882+0000 7f479f7fe700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 0x7f478c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:40.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.882+0000 7f479f7fe700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 0x7f478c06d190 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4794000f60 tx=0x7f4794009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:40.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:40.886+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4790054910 con 0x7f47a006c800 2023-06-10T18:57:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.069+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47a00b49d0 con 0x7f47a006c800 2023-06-10T18:57:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.070+0000 7f479cff9700 1 -- 172.21.15.104:0/44481483 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4790057f30 con 0x7f47a006c800 2023-06-10T18:57:41.077 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:57:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 msgr2=0x7f478c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 0x7f478c06d190 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4794000f60 tx=0x7f4794009040 comp rx=0 tx=0).stop 2023-06-10T18:57:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 msgr2=0x7f47a00b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f479000d800 tx=0x7f479000dbc0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 shutdown_connections 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f478c06ace0 0x7f478c06d190 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a0143a40 0x7f47a0158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47a006c800 0x7f47a00b3340 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 --2- 172.21.15.104:0/44481483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a0070a10 0x7f47a013fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 >> 172.21.15.104:0/44481483 conn(0x7f47a00ceec0 msgr2=0x7f47a013f320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 shutdown_connections 2023-06-10T18:57:41.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.072+0000 7f47a64b3700 1 -- 172.21.15.104:0/44481483 wait complete. 2023-06-10T18:57:41.090 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:57:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.207+0000 7f3ed05da700 1 Processor -- start 2023-06-10T18:57:41.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 -- start start 2023-06-10T18:57:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ec8070a10 0x7f3ec806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3ec806c800 0x7f3ec806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80cec40 con 0x7f3ec8143a40 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80cca90 con 0x7f3ec8070a10 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.208+0000 7f3ed05da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80ce960 con 0x7f3ec806c800 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ec8070a10 0x7f3ec806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecdb75700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3ec8143a40 0x7f3ec8070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36970/0 (socket says 172.21.15.104:36970) 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecdb75700 1 -- 172.21.15.104:0/2911248357 learned_addr learned my addr 172.21.15.104:0/2911248357 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3eceb77700 1 --2- 172.21.15.104:0/2911248357 >> v2:172.21.15.114:3300/0 conn(0x7f3ec806c800 0x7f3ec806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 -- 172.21.15.104:0/2911248357 >> v2:172.21.15.114:3300/0 conn(0x7f3ec806c800 msgr2=0x7f3ec806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:41.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 --2- 172.21.15.104:0/2911248357 >> v2:172.21.15.114:3300/0 conn(0x7f3ec806c800 0x7f3ec806cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 -- 172.21.15.104:0/2911248357 >> v1:172.21.15.104:6789/0 conn(0x7f3ec8143a40 legacy=0x7f3ec8070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecd374700 1 -- 172.21.15.104:0/2911248357 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541917692 0 0) 0x7f3ec80cec40 con 0x7f3ec8143a40 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 -- 172.21.15.104:0/2911248357 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec806a9b0 con 0x7f3ec8070a10 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ece376700 1 --2- 172.21.15.104:0/2911248357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec8070a10 0x7f3ec806e560 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80707c0 tx=0x7f3eb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c714dc7038d86512 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecd374700 1 -- 172.21.15.104:0/2911248357 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8011830 con 0x7f3ec8070a10 2023-06-10T18:57:41.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecd374700 1 -- 172.21.15.104:0/2911248357 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb8011990 con 0x7f3ec8070a10 2023-06-10T18:57:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.209+0000 7f3ecd374700 1 -- 172.21.15.104:0/2911248357 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8011d20 con 0x7f3ec8070a10 2023-06-10T18:57:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 -- 172.21.15.104:0/2911248357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec8070a10 msgr2=0x7f3ec806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2911248357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec8070a10 0x7f3ec806e560 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80707c0 tx=0x7f3eb8009b20 comp rx=0 tx=0).stop 2023-06-10T18:57:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 -- 172.21.15.104:0/2911248357 shutdown_connections 2023-06-10T18:57:41.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2911248357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec8070a10 0x7f3ec806e560 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2911248357 >> v2:172.21.15.114:3300/0 conn(0x7f3ec806c800 0x7f3ec806cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 -- 172.21.15.104:0/2911248357 >> 172.21.15.104:0/2911248357 conn(0x7f3ec80ceec0 msgr2=0x7f3ec80cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 -- 172.21.15.104:0/2911248357 shutdown_connections 2023-06-10T18:57:41.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.210+0000 7f3ed05da700 1 -- 172.21.15.104:0/2911248357 wait complete. 2023-06-10T18:57:41.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.211+0000 7f3ed05da700 1 Processor -- start 2023-06-10T18:57:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.211+0000 7f3ed05da700 1 -- start start 2023-06-10T18:57:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ec8070a10 0x7f3ec80c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 0x7f3ec81589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b3dc0 con 0x7f3ec806c800 2023-06-10T18:57:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81e4fa0 con 0x7f3ec8070a10 2023-06-10T18:57:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ed05da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b31b0 con 0x7f3ec8143a40 2023-06-10T18:57:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ece376700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ece376700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39148/0 (socket says 172.21.15.104:39148) 2023-06-10T18:57:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.212+0000 7f3ece376700 1 -- 172.21.15.104:0/2630007970 learned_addr learned my addr 172.21.15.104:0/2630007970 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ecdb75700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ec8070a10 0x7f3ec80c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3eceb77700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 0x7f3ec81589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 -- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 msgr2=0x7f3ec81589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 0x7f3ec81589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 -- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ec8070a10 msgr2=0x7f3ec80c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ec8070a10 0x7f3ec80c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 -- 172.21.15.104:0/2630007970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80094a0 con 0x7f3ec806c800 2023-06-10T18:57:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3eceb77700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 0x7f3ec81589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ece376700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80045d0 con 0x7f3ec806c800 2023-06-10T18:57:41.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb801e070 con 0x7f3ec806c800 2023-06-10T18:57:41.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80038e0 con 0x7f3ec806c800 2023-06-10T18:57:41.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec80cd0a0 con 0x7f3ec806c800 2023-06-10T18:57:41.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.213+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec80b3770 con 0x7f3ec806c800 2023-06-10T18:57:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.214+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8074050 con 0x7f3ec806c800 2023-06-10T18:57:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.214+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3eb8003a40 con 0x7f3ec806c800 2023-06-10T18:57:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.215+0000 7f3ebeffd700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 0x7f3eac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.215+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f3eb808a1e0 con 0x7f3ec806c800 2023-06-10T18:57:41.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.215+0000 7f3ecdb75700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 0x7f3eac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.215+0000 7f3ecdb75700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 0x7f3eac06d190 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8143980 tx=0x7f3ec4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.219+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3eb8055530 con 0x7f3ec806c800 2023-06-10T18:57:41.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.405+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3ec806b030 con 0x7f3ec806c800 2023-06-10T18:57:41.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.405+0000 7f3ebeffd700 1 -- 172.21.15.104:0/2630007970 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v155) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3eb8058b50 con 0x7f3ec806c800 2023-06-10T18:57:41.411 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:57:41.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 msgr2=0x7f3eac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 0x7f3eac06d190 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8143980 tx=0x7f3ec4007de0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 msgr2=0x7f3ec81e4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8000c00 tx=0x7f3eb8004750 comp rx=0 tx=0).stop 2023-06-10T18:57:41.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 shutdown_connections 2023-06-10T18:57:41.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ec8143a40 0x7f3ec81589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec806c800 0x7f3ec81e4a60 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3eac06ace0 0x7f3eac06d190 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 --2- 172.21.15.104:0/2630007970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ec8070a10 0x7f3ec80c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 >> 172.21.15.104:0/2630007970 conn(0x7f3ec80ceec0 msgr2=0x7f3ec8070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 shutdown_connections 2023-06-10T18:57:41.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.407+0000 7f3ed05da700 1 -- 172.21.15.104:0/2630007970 wait complete. 2023-06-10T18:57:41.425 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:57:41.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.538+0000 7fae08072700 1 Processor -- start 2023-06-10T18:57:41.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 -- start start 2023-06-10T18:57:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae001567f0 0x7fae000cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae000705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae00070b20 con 0x7fae0015c7a0 2023-06-10T18:57:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae000cb2d0 con 0x7fae001567f0 2023-06-10T18:57:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae08072700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015cca0 con 0x7fae000701a0 2023-06-10T18:57:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae05e0e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae001567f0 0x7fae000cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae0560d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fae0015c7a0 0x7fae000cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36984/0 (socket says 172.21.15.104:36984) 2023-06-10T18:57:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.539+0000 7fae0560d700 1 -- 172.21.15.104:0/3617328815 learned_addr learned my addr 172.21.15.104:0/3617328815 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:41.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae0660f700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae000705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae05e0e700 1 -- 172.21.15.104:0/3617328815 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 msgr2=0x7fae000705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae05e0e700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae000705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae05e0e700 1 -- 172.21.15.104:0/3617328815 >> v1:172.21.15.104:6789/0 conn(0x7fae0015c7a0 legacy=0x7fae000cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae04e0c700 1 -- 172.21.15.104:0/3617328815 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260664948 0 0) 0x7fae00070b20 con 0x7fae0015c7a0 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae05e0e700 1 -- 172.21.15.104:0/3617328815 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0015cf80 con 0x7fae001567f0 2023-06-10T18:57:41.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae05e0e700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000cad90 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009840 tx=0x7fadf0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acbf2017bcc0c41 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae04e0c700 1 -- 172.21.15.104:0/3617328815 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0011830 con 0x7fae001567f0 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae04e0c700 1 -- 172.21.15.104:0/3617328815 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0011990 con 0x7fae001567f0 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.540+0000 7fae04e0c700 1 -- 172.21.15.104:0/3617328815 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0011d20 con 0x7fae001567f0 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 -- 172.21.15.104:0/3617328815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 msgr2=0x7fae000cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000cad90 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009840 tx=0x7fadf0009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:41.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 -- 172.21.15.104:0/3617328815 shutdown_connections 2023-06-10T18:57:41.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000cad90 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 --2- 172.21.15.104:0/3617328815 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae000705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 -- 172.21.15.104:0/3617328815 >> 172.21.15.104:0/3617328815 conn(0x7fae0006e370 msgr2=0x7fae0006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 -- 172.21.15.104:0/3617328815 shutdown_connections 2023-06-10T18:57:41.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.541+0000 7fae08072700 1 -- 172.21.15.104:0/3617328815 wait complete. 2023-06-10T18:57:41.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.542+0000 7fae08072700 1 Processor -- start 2023-06-10T18:57:41.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.542+0000 7fae08072700 1 -- start start 2023-06-10T18:57:41.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae001549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0015c7a0 0x7fae0006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae000b48a0 con 0x7fae001567f0 2023-06-10T18:57:41.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae000b4b80 con 0x7fae0015c7a0 2023-06-10T18:57:41.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae000b4e60 con 0x7fae000701a0 2023-06-10T18:57:41.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39164/0 (socket says 172.21.15.104:39164) 2023-06-10T18:57:41.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 -- 172.21.15.104:0/1548403262 learned_addr learned my addr 172.21.15.104:0/1548403262 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:41.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae05e0e700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae001549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0660f700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0015c7a0 0x7fae0006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 -- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 msgr2=0x7fae001549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae001549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 -- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0015c7a0 msgr2=0x7fae0006e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0015c7a0 0x7fae0006e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 -- 172.21.15.104:0/1548403262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf00094a0 con 0x7fae001567f0 2023-06-10T18:57:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae0560d700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0011e0 tx=0x7fadfc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc004c80 con 0x7fae001567f0 2023-06-10T18:57:41.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae000b5140 con 0x7fae001567f0 2023-06-10T18:57:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc004de0 con 0x7fae001567f0 2023-06-10T18:57:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae000b5940 con 0x7fae001567f0 2023-06-10T18:57:41.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.543+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0072d0 con 0x7fae001567f0 2023-06-10T18:57:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.544+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade80052f0 con 0x7fae001567f0 2023-06-10T18:57:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.545+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fadfc007430 con 0x7fae001567f0 2023-06-10T18:57:41.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.545+0000 7fadf67fc700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 0x7fade406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.545+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fadfc088f30 con 0x7fae001567f0 2023-06-10T18:57:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.545+0000 7fae05e0e700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 0x7fade406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.546+0000 7fae05e0e700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 0x7fade406d190 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fadf0000c00 tx=0x7fadf0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.549+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fadfc0540d0 con 0x7fae001567f0 2023-06-10T18:57:41.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.739+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fade80055a0 con 0x7fae001567f0 2023-06-10T18:57:41.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.739+0000 7fadf67fc700 1 -- 172.21.15.104:0/1548403262 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v155) v1 ==== 84+0+126 (secure 0 0 0) 0x7fadfc0576f0 con 0x7fae001567f0 2023-06-10T18:57:41.746 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:41.746 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:57:41.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:41.746 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:57:41.747 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:57:41.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:41.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:41.747 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:57:41.748 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:57:41.748 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:41.748 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:57:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 msgr2=0x7fade406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 0x7fade406d190 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fadf0000c00 tx=0x7fadf0008030 comp rx=0 tx=0).stop 2023-06-10T18:57:41.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 msgr2=0x7fae000c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0011e0 tx=0x7fadfc00cae0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 shutdown_connections 2023-06-10T18:57:41.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fade406ace0 0x7fade406d190 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae0015c7a0 0x7fae0006e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae000701a0 0x7fae001549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 --2- 172.21.15.104:0/1548403262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae001567f0 0x7fae000c6260 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 >> 172.21.15.104:0/1548403262 conn(0x7fae0006e370 msgr2=0x7fae0015f220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 shutdown_connections 2023-06-10T18:57:41.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.741+0000 7fae08072700 1 -- 172.21.15.104:0/1548403262 wait complete. 2023-06-10T18:57:41.759 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:57:41.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:57:41.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.878+0000 7f84a0d5e700 1 Processor -- start 2023-06-10T18:57:41.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.878+0000 7f84a0d5e700 1 -- start start 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.879+0000 7f84a0d5e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f849c15c730 0x7f849c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.879+0000 7f84a0d5e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.879+0000 7f84a0d5e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c15cb00 con 0x7f849c0cd720 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.879+0000 7f84a0d5e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849c0cdbc0 con 0x7f849c15c730 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.879+0000 7f84a0d5e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0acd90 con 0x7f849c0ae230 2023-06-10T18:57:41.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849a59c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f849c15c730 0x7f849c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:58558/0 (socket says 172.21.15.104:58558) 2023-06-10T18:57:41.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 -- 172.21.15.104:0/2970132819 learned_addr learned my addr 172.21.15.104:0/2970132819 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:41.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 -- 172.21.15.104:0/2970132819 >> v2:172.21.15.114:3300/0 conn(0x7f849c15c730 msgr2=0x7f849c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:41.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 --2- 172.21.15.104:0/2970132819 >> v2:172.21.15.114:3300/0 conn(0x7f849c15c730 0x7f849c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 -- 172.21.15.104:0/2970132819 >> v1:172.21.15.114:6789/0 conn(0x7f849c0cd720 legacy=0x7f849c06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 -- 172.21.15.104:0/2970132819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0ac460 con 0x7f849c0ae230 2023-06-10T18:57:41.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.880+0000 7f849ad9d700 1 --2- 172.21.15.104:0/2970132819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8494009e80 tx=0x7f849400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6c4ee863b2972c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f849959a700 1 -- 172.21.15.104:0/2970132819 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849400e070 con 0x7f849c0ae230 2023-06-10T18:57:41.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f849959a700 1 -- 172.21.15.104:0/2970132819 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849400b8d0 con 0x7f849c0ae230 2023-06-10T18:57:41.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f849959a700 1 -- 172.21.15.104:0/2970132819 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849400bc80 con 0x7f849c0ae230 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 -- 172.21.15.104:0/2970132819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 msgr2=0x7f849c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/2970132819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8494009e80 tx=0x7f849400b330 comp rx=0 tx=0).stop 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 -- 172.21.15.104:0/2970132819 shutdown_connections 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/2970132819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0ae230 0x7f849c06d480 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/2970132819 >> v2:172.21.15.114:3300/0 conn(0x7f849c15c730 0x7f849c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 -- 172.21.15.104:0/2970132819 >> 172.21.15.104:0/2970132819 conn(0x7f849c06b8a0 msgr2=0x7f849c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:41.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 -- 172.21.15.104:0/2970132819 shutdown_connections 2023-06-10T18:57:41.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.881+0000 7f84a0d5e700 1 -- 172.21.15.104:0/2970132819 wait complete. 2023-06-10T18:57:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.882+0000 7f84a0d5e700 1 Processor -- start 2023-06-10T18:57:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 -- start start 2023-06-10T18:57:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 0x7f849c0c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0cd720 0x7f849c1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0c8a30 con 0x7f849c15c730 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0c74c0 con 0x7f849c0ae230 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f84a0d5e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0c8c60 con 0x7f849c0cd720 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f849ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f849ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39180/0 (socket says 172.21.15.104:39180) 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f849ad9d700 1 -- 172.21.15.104:0/3522122343 learned_addr learned my addr 172.21.15.104:0/3522122343 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:41.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.883+0000 7f8499d9b700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0cd720 0x7f849c1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849a59c700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 0x7f849c0c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 -- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0cd720 msgr2=0x7f849c1f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0cd720 0x7f849c1f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 -- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 msgr2=0x7f849c0c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 0x7f849c0c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 -- 172.21.15.104:0/3522122343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494009b30 con 0x7f849c15c730 2023-06-10T18:57:41.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849a59c700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 0x7f849c0c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f849ad9d700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f849c0aced0 tx=0x7f8494004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84940099b0 con 0x7f849c15c730 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8494004250 con 0x7f849c15c730 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c0c8f40 con 0x7f849c15c730 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494004600 con 0x7f849c15c730 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.884+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c0c9500 con 0x7f849c15c730 2023-06-10T18:57:41.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.885+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c066f10 con 0x7f849c15c730 2023-06-10T18:57:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.885+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8494004760 con 0x7f849c15c730 2023-06-10T18:57:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.886+0000 7f848affd700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 0x7f847c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.886+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f849408a870 con 0x7f849c15c730 2023-06-10T18:57:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.886+0000 7f849a59c700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 0x7f847c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:41.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.886+0000 7f849a59c700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 0x7f847c06d190 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f849c09d0f0 tx=0x7f8490008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:41.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:41.890+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8494055c10 con 0x7f849c15c730 2023-06-10T18:57:42.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.078+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f849c074050 con 0x7f849c15c730 2023-06-10T18:57:42.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.847+0000 7f848affd700 1 -- 172.21.15.104:0/3522122343 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v156) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8494059230 con 0x7f849c15c730 2023-06-10T18:57:42.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 msgr2=0x7f847c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:42.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 0x7f847c06d190 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f849c09d0f0 tx=0x7f8490008040 comp rx=0 tx=0).stop 2023-06-10T18:57:42.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 msgr2=0x7f849c157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:42.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f849c0aced0 tx=0x7f8494004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:42.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 shutdown_connections 2023-06-10T18:57:42.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f847c06ace0 0x7f847c06d190 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0ae230 0x7f849c0c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f849c15c730 0x7f849c157240 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 --2- 172.21.15.104:0/3522122343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f849c0cd720 0x7f849c1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 >> 172.21.15.104:0/3522122343 conn(0x7f849c06b8a0 msgr2=0x7f849c0c9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 shutdown_connections 2023-06-10T18:57:42.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.849+0000 7f84a0d5e700 1 -- 172.21.15.104:0/3522122343 wait complete. 2023-06-10T18:57:42.858 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:57:42.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:57:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.995+0000 7f046e330700 1 Processor -- start 2023-06-10T18:57:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.995+0000 7f046e330700 1 -- start start 2023-06-10T18:57:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f046e330700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f04680cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f046e330700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046815c8d0 con 0x7f04680cf200 2023-06-10T18:57:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f046e330700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046806abd0 con 0x7f046806dcd0 2023-06-10T18:57:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f046e330700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04680aef20 con 0x7f046815c500 2023-06-10T18:57:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f04677fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f04680cf200 0x7f0468069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37012/0 (socket says 172.21.15.104:37012) 2023-06-10T18:57:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f04677fe700 1 -- 172.21.15.104:0/4284301025 learned_addr learned my addr 172.21.15.104:0/4284301025 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0467fff700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f04680cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668790218 0 0) 0x7f046815c8d0 con 0x7f04680cf200 2023-06-10T18:57:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04500035b0 con 0x7f04680cf200 2023-06-10T18:57:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556674091 0 0) 0x7f046806abd0 con 0x7f046806dcd0 2023-06-10T18:57:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046815c8d0 con 0x7f046806dcd0 2023-06-10T18:57:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381040108 0 0) 0x7f04500035b0 con 0x7f04680cf200 2023-06-10T18:57:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 msgr2=0x7f04680cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f04680cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 >> v1:172.21.15.114:6789/0 conn(0x7f046806dcd0 legacy=0x7f04680c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046806a8f0 con 0x7f04680cf200 2023-06-10T18:57:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0458001670 con 0x7f04680cf200 2023-06-10T18:57:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 >> v1:172.21.15.104:6789/0 conn(0x7f04680cf200 legacy=0x7f0468069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.996+0000 7f0466ffd700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0450003bc0 0x7f0450003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0450004850 0x7f0450004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f0450005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045000bac0 con 0x7f0450004850 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045000bc00 con 0x7f0450003bc0 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045000bd70 con 0x7f046815c500 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0450004850 0x7f0450004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0467fff700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0450003bc0 0x7f0450003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f046c8cd700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f0450005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 -- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 msgr2=0x7f0450005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f046815c500 0x7f0450005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 -- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0450003bc0 msgr2=0x7f0450003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:43.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0450003bc0 0x7f0450003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 -- 172.21.15.104:0/4284301025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04500035b0 con 0x7f0450004850 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f04677fe700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0450004850 0x7f0450004cc0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f045800c210 tx=0x7f0450005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18aa708758da389f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045800cd50 con 0x7f0450004850 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458001af0 con 0x7f0450004850 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.997+0000 7f0466ffd700 1 -- 172.21.15.104:0/4284301025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458005ba0 con 0x7f0450004850 2023-06-10T18:57:43.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 -- 172.21.15.104:0/4284301025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0450004850 msgr2=0x7f0450004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 --2- 172.21.15.104:0/4284301025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0450004850 0x7f0450004cc0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f045800c210 tx=0x7f0450005880 comp rx=0 tx=0).stop 2023-06-10T18:57:43.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046c8cd700 1 -- 172.21.15.104:0/4284301025 reap_dead start 2023-06-10T18:57:43.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 -- 172.21.15.104:0/4284301025 shutdown_connections 2023-06-10T18:57:43.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 -- 172.21.15.104:0/4284301025 >> 172.21.15.104:0/4284301025 conn(0x7f046806b8a0 msgr2=0x7f04680cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:43.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 -- 172.21.15.104:0/4284301025 shutdown_connections 2023-06-10T18:57:43.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.998+0000 7f046e330700 1 -- 172.21.15.104:0/4284301025 wait complete. 2023-06-10T18:57:43.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.999+0000 7f046e330700 1 Processor -- start 2023-06-10T18:57:43.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:42.999+0000 7f046e330700 1 -- start start 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0450004850 0x7f04681f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04681f5cb0 0x7f04680c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680cb530 con 0x7f0468159180 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0468159b30 con 0x7f04681f5cb0 2023-06-10T18:57:43.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04681f2320 con 0x7f0450004850 2023-06-10T18:57:43.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39198/0 (socket says 172.21.15.104:39198) 2023-06-10T18:57:43.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 -- 172.21.15.104:0/233689470 learned_addr learned my addr 172.21.15.104:0/233689470 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:43.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f04677fe700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04681f5cb0 0x7f04680c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f0467fff700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0450004850 0x7f04681f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 -- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0450004850 msgr2=0x7f04681f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0450004850 0x7f04681f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 -- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04681f5cb0 msgr2=0x7f04680c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04681f5cb0 0x7f04680c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 -- 172.21.15.104:0/233689470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04681f2600 con 0x7f0468159180 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046c8cd700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f046806d730 tx=0x7f046000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460003c10 con 0x7f0468159180 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0460003d70 con 0x7f0468159180 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04681f28e0 con 0x7f0468159180 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046000dc00 con 0x7f0468159180 2023-06-10T18:57:43.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04681f6120 con 0x7f0468159180 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.001+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0468074050 con 0x7f0468159180 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.002+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f046000dd60 con 0x7f0468159180 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.002+0000 7f0464ff9700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 0x7f045406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.002+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f0460089820 con 0x7f0468159180 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.004+0000 7f0467fff700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 0x7f045406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.004+0000 7f0467fff700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 0x7f045406d190 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f045c009a40 tx=0x7f045c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:43.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.008+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f04600548e0 con 0x7f0468159180 2023-06-10T18:57:43.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.195+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f04680b3a50 con 0x7f0468159180 2023-06-10T18:57:43.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.857+0000 7f0464ff9700 1 -- 172.21.15.104:0/233689470 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v157) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0460057f00 con 0x7f0468159180 2023-06-10T18:57:43.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 msgr2=0x7f045406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 0x7f045406d190 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f045c009a40 tx=0x7f045c008040 comp rx=0 tx=0).stop 2023-06-10T18:57:43.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 msgr2=0x7f04681595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f046806d730 tx=0x7f046000d430 comp rx=0 tx=0).stop 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 shutdown_connections 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0468159180 0x7f04681595f0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0450004850 0x7f04681f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f045406ace0 0x7f045406d190 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 --2- 172.21.15.104:0/233689470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04681f5cb0 0x7f04680c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:43.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 >> 172.21.15.104:0/233689470 conn(0x7f046806b8a0 msgr2=0x7f04680b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:43.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 shutdown_connections 2023-06-10T18:57:43.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:43.862+0000 7f046e330700 1 -- 172.21.15.104:0/233689470 wait complete. 2023-06-10T18:57:43.873 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:57:43.888 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:57:44.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.007+0000 7fca9c69a700 1 Processor -- start 2023-06-10T18:57:44.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.007+0000 7fca9c69a700 1 -- start start 2023-06-10T18:57:44.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9c69a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca940cea50 0x7fca940d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9c69a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca940ad3c0 con 0x7fca94157b30 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9c69a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca94157f00 con 0x7fca940d0830 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9c69a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940b0170 con 0x7fca940cea50 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9a436700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fca94157b30 0x7fca940ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37030/0 (socket says 172.21.15.104:37030) 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.008+0000 7fca9a436700 1 -- 172.21.15.104:0/3484557476 learned_addr learned my addr 172.21.15.104:0/3484557476 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:44.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99c35700 1 --2- 172.21.15.104:0/3484557476 >> v2:172.21.15.114:3300/0 conn(0x7fca940cea50 0x7fca940d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63081849 0 0) 0x7fca940ad3c0 con 0x7fca94157b30 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca800035b0 con 0x7fca94157b30 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001633579 0 0) 0x7fca800035b0 con 0x7fca94157b30 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 >> v2:172.21.15.114:3300/0 conn(0x7fca940cea50 msgr2=0x7fca940d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 --2- 172.21.15.104:0/3484557476 >> v2:172.21.15.114:3300/0 conn(0x7fca940cea50 0x7fca940d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 >> v1:172.21.15.114:6789/0 conn(0x7fca940d0830 legacy=0x7fca940b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940b0410 con 0x7fca94157b30 2023-06-10T18:57:44.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fca84001670 con 0x7fca94157b30 2023-06-10T18:57:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 >> v1:172.21.15.104:6789/0 conn(0x7fca94157b30 legacy=0x7fca940ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca80003a00 0x7fca80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80004850 0x7fca80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 0x7fca8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940b0170 con 0x7fca80005200 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000c080 con 0x7fca80004850 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000c240 con 0x7fca80003a00 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 0x7fca8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca99c35700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80004850 0x7fca80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9a436700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca80003a00 0x7fca80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 -- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca80003a00 msgr2=0x7fca80003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:44.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca80003a00 0x7fca80003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 -- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80004850 msgr2=0x7fca80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80004850 0x7fca80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 -- 172.21.15.104:0/3484557476 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800035b0 con 0x7fca80005200 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.009+0000 7fca9ac37700 1 -- 172.21.15.104:0/3484557476 reap_dead start 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9ac37700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 0x7fca8000ba10 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00c190 tx=0x7fca80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a918e4209f8d67 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c001db0 con 0x7fca80005200 2023-06-10T18:57:44.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c00ccf0 con 0x7fca80005200 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca99434700 1 -- 172.21.15.104:0/3484557476 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c003c40 con 0x7fca80005200 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 -- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 msgr2=0x7fca8000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 0x7fca8000ba10 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00c190 tx=0x7fca80005880 comp rx=0 tx=0).stop 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 -- 172.21.15.104:0/3484557476 shutdown_connections 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 --2- 172.21.15.104:0/3484557476 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca80005200 0x7fca8000ba10 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 -- 172.21.15.104:0/3484557476 >> 172.21.15.104:0/3484557476 conn(0x7fca94089fd0 msgr2=0x7fca9408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 -- 172.21.15.104:0/3484557476 shutdown_connections 2023-06-10T18:57:44.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.010+0000 7fca9c69a700 1 -- 172.21.15.104:0/3484557476 wait complete. 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.011+0000 7fca9c69a700 1 Processor -- start 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 -- start start 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80005200 0x7fca94156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca940b5210 0x7fca940c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940c5f90 con 0x7fca940b4860 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940b56b0 con 0x7fca80005200 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9c69a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca941e2750 con 0x7fca940b5210 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca99c35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca99c35700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39216/0 (socket says 172.21.15.104:39216) 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca99c35700 1 -- 172.21.15.104:0/104009548 learned_addr learned my addr 172.21.15.104:0/104009548 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9a436700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80005200 0x7fca94156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.012+0000 7fca9ac37700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca940b5210 0x7fca940c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 -- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca940b5210 msgr2=0x7fca940c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca940b5210 0x7fca940c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 -- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80005200 msgr2=0x7fca94156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80005200 0x7fca94156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 -- 172.21.15.104:0/104009548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940c5310 con 0x7fca940b4860 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca99c35700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fca90011930 tx=0x7fca90011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90011e70 con 0x7fca940b4860 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9000f940 con 0x7fca940b4860 2023-06-10T18:57:44.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca940c55f0 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9000fcf0 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.013+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca940c5bb0 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.014+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca94074050 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.015+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fca90014440 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.015+0000 7fca8affd700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 0x7fca7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.015+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fca90089b30 con 0x7fca940b4860 2023-06-10T18:57:44.027 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.015+0000 7fca9a436700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 0x7fca7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.016+0000 7fca9a436700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 0x7fca7806d190 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fca84001c90 tx=0x7fca84001b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.020+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fca90054b90 con 0x7fca940b4860 2023-06-10T18:57:44.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.257+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fca9406dd50 con 0x7fca940b4860 2023-06-10T18:57:44.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.264+0000 7fca8affd700 1 -- 172.21.15.104:0/104009548 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v98) v1 ==== 174+0+0 (secure 0 0 0) 0x7fca900581b0 con 0x7fca940b4860 2023-06-10T18:57:44.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.269+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 msgr2=0x7fca7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.269+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 0x7fca7806d190 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fca84001c90 tx=0x7fca84001b20 comp rx=0 tx=0).stop 2023-06-10T18:57:44.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.269+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 msgr2=0x7fca940b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.269+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fca90011930 tx=0x7fca90011cf0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 shutdown_connections 2023-06-10T18:57:44.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fca7806ace0 0x7fca7806d190 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80005200 0x7fca94156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca940b5210 0x7fca940c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 --2- 172.21.15.104:0/104009548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca940b4860 0x7fca940b4cd0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 >> 172.21.15.104:0/104009548 conn(0x7fca94089fd0 msgr2=0x7fca940b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 shutdown_connections 2023-06-10T18:57:44.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.270+0000 7fca9c69a700 1 -- 172.21.15.104:0/104009548 wait complete. 2023-06-10T18:57:44.280 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 22 and data pool 23 2023-06-10T18:57:44.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:44.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 Processor -- start 2023-06-10T18:57:44.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 -- start start 2023-06-10T18:57:44.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3fb815c500 0x7f3fb80cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3fb8069e70 0x7f3fb806a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb806dcb0 con 0x7f3fb80ce040 2023-06-10T18:57:44.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb806e850 con 0x7f3fb8069e70 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.413+0000 7f3fbdccb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb806a830 con 0x7f3fb815c500 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3fb8069e70 0x7f3fb806a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3faeffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3fb80ce040 0x7f3fb806e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37064/0 (socket says 172.21.15.104:37064) 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3faeffd700 1 -- 172.21.15.104:0/3392540626 learned_addr learned my addr 172.21.15.104:0/3392540626 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb77fe700 1 --2- 172.21.15.104:0/3392540626 >> v2:172.21.15.114:3300/0 conn(0x7f3fb815c500 0x7f3fb80cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 -- 172.21.15.104:0/3392540626 >> v2:172.21.15.114:3300/0 conn(0x7f3fb815c500 msgr2=0x7f3fb80cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 --2- 172.21.15.104:0/3392540626 >> v2:172.21.15.114:3300/0 conn(0x7f3fb815c500 0x7f3fb80cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 -- 172.21.15.104:0/3392540626 >> v1:172.21.15.104:6789/0 conn(0x7f3fb80ce040 legacy=0x7f3fb806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 -- 172.21.15.104:0/3392540626 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80ce470 con 0x7f3fb8069e70 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb6ffd700 1 -- 172.21.15.104:0/3392540626 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131676852 0 0) 0x7f3fb806dcb0 con 0x7f3fb80ce040 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb7fff700 1 --2- 172.21.15.104:0/3392540626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb806a2f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8067130 tx=0x7f3fb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc064972d1a1ea8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb6ffd700 1 -- 172.21.15.104:0/3392540626 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000e070 con 0x7f3fb8069e70 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb6ffd700 1 -- 172.21.15.104:0/3392540626 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb000bbd0 con 0x7f3fb8069e70 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.414+0000 7f3fb6ffd700 1 -- 172.21.15.104:0/3392540626 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0012430 con 0x7f3fb8069e70 2023-06-10T18:57:44.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3392540626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 msgr2=0x7f3fb806a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3392540626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb806a2f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8067130 tx=0x7f3fb000b330 comp rx=0 tx=0).stop 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3392540626 shutdown_connections 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3392540626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb806a2f0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3392540626 >> v2:172.21.15.114:3300/0 conn(0x7f3fb815c500 0x7f3fb80cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3392540626 >> 172.21.15.104:0/3392540626 conn(0x7f3fb806b8a0 msgr2=0x7f3fb80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3392540626 shutdown_connections 2023-06-10T18:57:44.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.415+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3392540626 wait complete. 2023-06-10T18:57:44.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.416+0000 7f3fbdccb700 1 Processor -- start 2023-06-10T18:57:44.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.416+0000 7f3fbdccb700 1 -- start start 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3fb80ce040 0x7f3fb809ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 0x7f3fb80b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb809b080 con 0x7f3fb8069e70 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb81485a0 con 0x7f3fb815c500 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fbdccb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb809d3f0 con 0x7f3fb80ce040 2023-06-10T18:57:44.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39240/0 (socket says 172.21.15.104:39240) 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 -- 172.21.15.104:0/3080328833 learned_addr learned my addr 172.21.15.104:0/3080328833 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 -- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3fb80ce040 msgr2=0x7f3fb809ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3fb80ce040 0x7f3fb809ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 -- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 msgr2=0x7f3fb80b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb7fff700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 0x7f3fb80b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 0x7f3fb80b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 -- 172.21.15.104:0/3080328833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0009a70 con 0x7f3fb8069e70 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb7fff700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 0x7f3fb80b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.417+0000 7f3fb77fe700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800e760 tx=0x7f3fa800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.418+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8005c20 con 0x7f3fb8069e70 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.418+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa8005da0 con 0x7f3fb8069e70 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.418+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb809cb50 con 0x7f3fb8069e70 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.418+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8006260 con 0x7f3fb8069e70 2023-06-10T18:57:44.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.418+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb809d110 con 0x7f3fb8069e70 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.419+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb8074050 con 0x7f3fb8069e70 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.419+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3fa80063c0 con 0x7f3fb8069e70 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.419+0000 7f3fb4ff9700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 0x7f3f9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.419+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3fa8088f80 con 0x7f3fb8069e70 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.420+0000 7f3faeffd700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 0x7f3f9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.420+0000 7f3faeffd700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 0x7f3f9806d190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00011e0 tx=0x7f3fa00079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.424+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3fa8053ef0 con 0x7f3fb8069e70 2023-06-10T18:57:44.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.613+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fb80aea40 con 0x7f3fb8069e70 2023-06-10T18:57:44.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.614+0000 7f3fb4ff9700 1 -- 172.21.15.104:0/3080328833 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+18619 (secure 0 0 0) 0x7f3fa8020090 con 0x7f3fb8069e70 2023-06-10T18:57:44.620 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:44.620 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":158,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:44.258010+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 msgr2=0x7f3f9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 0x7f3f9806d190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00011e0 tx=0x7f3fa00079b0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 msgr2=0x7f3fb809c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800e760 tx=0x7f3fa800eb20 comp rx=0 tx=0).stop 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 shutdown_connections 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3f9806ace0 0x7f3f9806d190 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb815c500 0x7f3fb80b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb8069e70 0x7f3fb809c470 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 --2- 172.21.15.104:0/3080328833 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3fb80ce040 0x7f3fb809ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.616+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 >> 172.21.15.104:0/3080328833 conn(0x7f3fb806b8a0 msgr2=0x7f3fb80b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.617+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 shutdown_connections 2023-06-10T18:57:44.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.617+0000 7f3fbdccb700 1 -- 172.21.15.104:0/3080328833 wait complete. 2023-06-10T18:57:44.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:44.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.749+0000 7f0d67ca1700 1 Processor -- start 2023-06-10T18:57:44.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.749+0000 7f0d67ca1700 1 -- start start 2023-06-10T18:57:44.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d67ca1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e0e0 0x7f0d6006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d67ca1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0d60069e70 0x7f0d600ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d67ca1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6006d290 con 0x7f0d6006e4b0 2023-06-10T18:57:44.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d67ca1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d60071640 con 0x7f0d60069e70 2023-06-10T18:57:44.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d67ca1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600aef00 con 0x7f0d6006e0e0 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6623e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0d60069e70 0x7f0d600ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6523c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0d6006e4b0 0x7f0d600c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37076/0 (socket says 172.21.15.104:37076) 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6523c700 1 -- 172.21.15.104:0/1540252924 learned_addr learned my addr 172.21.15.104:0/1540252924 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d65a3d700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e0e0 0x7f0d6006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6623e700 1 -- 172.21.15.104:0/1540252924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e0e0 msgr2=0x7f0d6006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6623e700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e0e0 0x7f0d6006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6623e700 1 -- 172.21.15.104:0/1540252924 >> v1:172.21.15.104:6789/0 conn(0x7f0d6006e4b0 legacy=0x7f0d600c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d6623e700 1 -- 172.21.15.104:0/1540252924 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d600ca2d0 con 0x7f0d60069e70 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.750+0000 7f0d64a3b700 1 -- 172.21.15.104:0/1540252924 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941290287 0 0) 0x7f0d6006d290 con 0x7f0d6006e4b0 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d6623e700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d60069e70 0x7f0d600ce9f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009d90 tx=0x7f0d5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d108a323d80de19 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d64a3b700 1 -- 172.21.15.104:0/1540252924 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800e070 con 0x7f0d60069e70 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d64a3b700 1 -- 172.21.15.104:0/1540252924 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5800bbd0 con 0x7f0d60069e70 2023-06-10T18:57:44.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d64a3b700 1 -- 172.21.15.104:0/1540252924 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58012430 con 0x7f0d60069e70 2023-06-10T18:57:44.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 -- 172.21.15.104:0/1540252924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d60069e70 msgr2=0x7f0d600ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d60069e70 0x7f0d600ce9f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009d90 tx=0x7f0d5800b330 comp rx=0 tx=0).stop 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 -- 172.21.15.104:0/1540252924 shutdown_connections 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d60069e70 0x7f0d600ce9f0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/1540252924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e0e0 0x7f0d6006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 -- 172.21.15.104:0/1540252924 >> 172.21.15.104:0/1540252924 conn(0x7f0d6006b8a0 msgr2=0x7f0d600cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 -- 172.21.15.104:0/1540252924 shutdown_connections 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.751+0000 7f0d67ca1700 1 -- 172.21.15.104:0/1540252924 wait complete. 2023-06-10T18:57:44.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 Processor -- start 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 -- start start 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d60069e70 0x7f0d60157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e4b0 0x7f0d600c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6009d1c0 con 0x7f0d6006e0e0 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600af740 con 0x7f0d60069e70 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d67ca1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600afa20 con 0x7f0d6006e4b0 2023-06-10T18:57:44.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d6523c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d6523c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39256/0 (socket says 172.21.15.104:39256) 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.753+0000 7f0d6523c700 1 -- 172.21.15.104:0/836769976 learned_addr learned my addr 172.21.15.104:0/836769976 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6623e700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e4b0 0x7f0d600c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 -- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e4b0 msgr2=0x7f0d600c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e4b0 0x7f0d600c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 -- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d60069e70 msgr2=0x7f0d60157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d60069e70 0x7f0d60157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 -- 172.21.15.104:0/836769976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d58009a70 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d6523c700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0011e0 tx=0x7f0d5c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c004c80 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c004e00 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c007310 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d600afd00 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.754+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d600b02c0 con 0x7f0d6006e0e0 2023-06-10T18:57:44.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.755+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60074050 con 0x7f0d6006e0e0 2023-06-10T18:57:44.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.755+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0d5c007470 con 0x7f0d6006e0e0 2023-06-10T18:57:44.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.756+0000 7f0d567fc700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 0x7f0d4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:44.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.756+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0d5c089480 con 0x7f0d6006e0e0 2023-06-10T18:57:44.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.756+0000 7f0d65a3d700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 0x7f0d4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:44.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.756+0000 7f0d65a3d700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 0x7f0d4406d190 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f0d50006eb0 tx=0x7f0d50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:44.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.760+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0d5c054360 con 0x7f0d6006e0e0 2023-06-10T18:57:44.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.949+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d600ae2b0 con 0x7f0d6006e0e0 2023-06-10T18:57:44.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.950+0000 7f0d567fc700 1 -- 172.21.15.104:0/836769976 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+18619 (secure 0 0 0) 0x7f0d5c057980 con 0x7f0d6006e0e0 2023-06-10T18:57:44.956 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:44.956 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":158,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:44.258010+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:44.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 msgr2=0x7f0d4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 0x7f0d4406d190 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f0d50006eb0 tx=0x7f0d50008040 comp rx=0 tx=0).stop 2023-06-10T18:57:44.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 msgr2=0x7f0d600af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:44.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0011e0 tx=0x7f0d5c00cae0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 shutdown_connections 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d4406ace0 0x7f0d4406d190 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d60069e70 0x7f0d60157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d6006e4b0 0x7f0d600c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 --2- 172.21.15.104:0/836769976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d6006e0e0 0x7f0d600af040 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 >> 172.21.15.104:0/836769976 conn(0x7f0d6006b8a0 msgr2=0x7f0d600cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 shutdown_connections 2023-06-10T18:57:44.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:44.952+0000 7f0d67ca1700 1 -- 172.21.15.104:0/836769976 wait complete. 2023-06-10T18:57:44.970 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:57:45.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.087+0000 7ff842f6a700 1 Processor -- start 2023-06-10T18:57:45.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.087+0000 7ff842f6a700 1 -- start start 2023-06-10T18:57:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff83c1586a0 0x7ff83c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff83c0ce390 0x7ff83c070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c070f50 0x7ff83c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0713f0 con 0x7ff83c1586a0 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c06d7d0 con 0x7ff83c0ce390 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff842f6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0cd6c0 con 0x7ff83c070f50 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff83c1586a0 0x7ff83c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff83c1586a0 0x7ff83c06cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39262/0 (socket says 172.21.15.104:39262) 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 -- 172.21.15.104:0/611003564 learned_addr learned my addr 172.21.15.104:0/611003564 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:45.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff83bfff700 1 --2- 172.21.15.104:0/611003564 >> v2:172.21.15.114:3300/0 conn(0x7ff83c0ce390 0x7ff83c070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 -- 172.21.15.104:0/611003564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c070f50 msgr2=0x7ff83c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 --2- 172.21.15.104:0/611003564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c070f50 0x7ff83c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 -- 172.21.15.104:0/611003564 >> v2:172.21.15.114:3300/0 conn(0x7ff83c0ce390 msgr2=0x7ff83c070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 --2- 172.21.15.104:0/611003564 >> v2:172.21.15.114:3300/0 conn(0x7ff83c0ce390 0x7ff83c070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 -- 172.21.15.104:0/611003564 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0c95d0 con 0x7ff83c1586a0 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff840d06700 1 --2- 172.21.15.104:0/611003564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c1586a0 0x7ff83c06cff0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ff83c06e670 tx=0x7ff830009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2755b61c3b2a6a3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:45.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff83b7fe700 1 -- 172.21.15.104:0/611003564 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83000b870 con 0x7ff83c1586a0 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff83b7fe700 1 -- 172.21.15.104:0/611003564 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83000b9d0 con 0x7ff83c1586a0 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.088+0000 7ff83b7fe700 1 -- 172.21.15.104:0/611003564 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83000bd60 con 0x7ff83c1586a0 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 -- 172.21.15.104:0/611003564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c1586a0 msgr2=0x7ff83c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 --2- 172.21.15.104:0/611003564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c1586a0 0x7ff83c06cff0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ff83c06e670 tx=0x7ff830009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 -- 172.21.15.104:0/611003564 shutdown_connections 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 --2- 172.21.15.104:0/611003564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c1586a0 0x7ff83c06cff0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 --2- 172.21.15.104:0/611003564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c070f50 0x7ff83c069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 --2- 172.21.15.104:0/611003564 >> v2:172.21.15.114:3300/0 conn(0x7ff83c0ce390 0x7ff83c070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 -- 172.21.15.104:0/611003564 >> 172.21.15.104:0/611003564 conn(0x7ff83c0ceec0 msgr2=0x7ff83c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 -- 172.21.15.104:0/611003564 shutdown_connections 2023-06-10T18:57:45.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.089+0000 7ff842f6a700 1 -- 172.21.15.104:0/611003564 wait complete. 2023-06-10T18:57:45.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.090+0000 7ff842f6a700 1 Processor -- start 2023-06-10T18:57:45.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.090+0000 7ff842f6a700 1 -- start start 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff83c070f50 0x7ff83c14d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c1586a0 0x7ff83c14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1e5cc0 con 0x7ff83c0ce390 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1e4c20 con 0x7ff83c070f50 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff842f6a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1e4f00 con 0x7ff83c1586a0 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39268/0 (socket says 172.21.15.104:39268) 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 -- 172.21.15.104:0/2181122736 learned_addr learned my addr 172.21.15.104:0/2181122736 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff840d06700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff83c070f50 0x7ff83c14d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff841507700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c1586a0 0x7ff83c14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 -- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c1586a0 msgr2=0x7ff83c14d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c1586a0 0x7ff83c14d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 -- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff83c070f50 msgr2=0x7ff83c14d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:45.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff83c070f50 0x7ff83c14d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.091+0000 7ff83bfff700 1 -- 172.21.15.104:0/2181122736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff830009550 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff83bfff700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c7f0 tx=0x7ff82800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828004220 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff828004380 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c1e51e0 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280036a0 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.092+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c1e57a0 con 0x7ff83c0ce390 2023-06-10T18:57:45.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.093+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c074050 con 0x7ff83c0ce390 2023-06-10T18:57:45.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.093+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff828003800 con 0x7ff83c0ce390 2023-06-10T18:57:45.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.093+0000 7ff8397fa700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 0x7ff82006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:45.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.094+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff8280890b0 con 0x7ff83c0ce390 2023-06-10T18:57:45.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.094+0000 7ff840d06700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 0x7ff82006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:45.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.094+0000 7ff840d06700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 0x7ff82006d190 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff830014040 tx=0x7ff830008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:45.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.098+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff828057ab0 con 0x7ff83c0ce390 2023-06-10T18:57:45.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:45.330+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff83c0b43f0 con 0x7ff83c0ce390 2023-06-10T18:57:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.269+0000 7ff8397fa700 1 -- 172.21.15.104:0/2181122736 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v101) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff828057640 con 0x7ff83c0ce390 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 msgr2=0x7ff82006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 0x7ff82006d190 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff830014040 tx=0x7ff830008030 comp rx=0 tx=0).stop 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 msgr2=0x7ff83c1e45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c7f0 tx=0x7ff82800cb00 comp rx=0 tx=0).stop 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 shutdown_connections 2023-06-10T18:57:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff82006ace0 0x7ff82006d190 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff83c070f50 0x7ff83c14d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff83c1586a0 0x7ff83c14d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 --2- 172.21.15.104:0/2181122736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff83c0ce390 0x7ff83c1e45a0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.272+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 >> 172.21.15.104:0/2181122736 conn(0x7ff83c0ceec0 msgr2=0x7ff83c0ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.273+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 shutdown_connections 2023-06-10T18:57:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.273+0000 7ff842f6a700 1 -- 172.21.15.104:0/2181122736 wait complete. 2023-06-10T18:57:46.292 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:57:46.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:57:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.415+0000 7fef5cb7a700 1 Processor -- start 2023-06-10T18:57:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 -- start start 2023-06-10T18:57:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fef580b1780 0x7fef5806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 0x7fef5806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580ae7e0 con 0x7fef5815c6f0 2023-06-10T18:57:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5806df90 con 0x7fef580b1780 2023-06-10T18:57:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.416+0000 7fef5cb7a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5806e270 con 0x7fef5806ea70 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fef580b1780 0x7fef5806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fef580b1780 0x7fef5806cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39278/0 (socket says 172.21.15.104:39278) 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 -- 172.21.15.104:0/2525652014 learned_addr learned my addr 172.21.15.104:0/2525652014 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef56d9d700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 0x7fef5806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 -- 172.21.15.104:0/2525652014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 msgr2=0x7fef5806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 0x7fef5806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 -- 172.21.15.104:0/2525652014 >> v1:172.21.15.114:6789/0 conn(0x7fef5815c6f0 legacy=0x7fef580b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 -- 172.21.15.104:0/2525652014 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5806e550 con 0x7fef580b1780 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef56d9d700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 0x7fef5806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef55d9b700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef580b1780 0x7fef5806cde0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fef4800bb90 tx=0x7fef4800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b79b00dd644bdf1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef5559a700 1 -- 172.21.15.104:0/2525652014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4800dd50 con 0x7fef580b1780 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef5559a700 1 -- 172.21.15.104:0/2525652014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef48010470 con 0x7fef580b1780 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.417+0000 7fef5559a700 1 -- 172.21.15.104:0/2525652014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48010800 con 0x7fef580b1780 2023-06-10T18:57:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 -- 172.21.15.104:0/2525652014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef580b1780 msgr2=0x7fef5806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef580b1780 0x7fef5806cde0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fef4800bb90 tx=0x7fef4800bea0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 -- 172.21.15.104:0/2525652014 shutdown_connections 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef5806ea70 0x7fef5806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/2525652014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef580b1780 0x7fef5806cde0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 -- 172.21.15.104:0/2525652014 >> 172.21.15.104:0/2525652014 conn(0x7fef5806b8a0 msgr2=0x7fef580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 -- 172.21.15.104:0/2525652014 shutdown_connections 2023-06-10T18:57:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.418+0000 7fef5cb7a700 1 -- 172.21.15.104:0/2525652014 wait complete. 2023-06-10T18:57:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.419+0000 7fef5cb7a700 1 Processor -- start 2023-06-10T18:57:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.419+0000 7fef5cb7a700 1 -- start start 2023-06-10T18:57:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef580b1780 0x7fef58151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef5815c6f0 0x7fef580c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580c43b0 con 0x7fef5806ea70 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580c4650 con 0x7fef5815c6f0 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5cb7a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580c4930 con 0x7fef580b1780 2023-06-10T18:57:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39282/0 (socket says 172.21.15.104:39282) 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 -- 172.21.15.104:0/3360725211 learned_addr learned my addr 172.21.15.104:0/3360725211 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef56d9d700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef5815c6f0 0x7fef580c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef55d9b700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef580b1780 0x7fef58151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 -- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef580b1780 msgr2=0x7fef58151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef580b1780 0x7fef58151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 -- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef5815c6f0 msgr2=0x7fef580c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef5815c6f0 0x7fef580c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 -- 172.21.15.104:0/3360725211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4800b7f0 con 0x7fef5806ea70 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.420+0000 7fef5659c700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c4d0 tx=0x7fef4c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.421+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0052d0 con 0x7fef5806ea70 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.421+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c005430 con 0x7fef5806ea70 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.421+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0038f0 con 0x7fef5806ea70 2023-06-10T18:57:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.421+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef580c4c10 con 0x7fef5806ea70 2023-06-10T18:57:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.421+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef580c5950 con 0x7fef5806ea70 2023-06-10T18:57:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.422+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580c5ae0 con 0x7fef5806ea70 2023-06-10T18:57:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.422+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fef4c003a50 con 0x7fef5806ea70 2023-06-10T18:57:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.422+0000 7fef46ffd700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 0x7fef3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.422+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fef4c088a70 con 0x7fef5806ea70 2023-06-10T18:57:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.423+0000 7fef55d9b700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 0x7fef3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.423+0000 7fef55d9b700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 0x7fef3806d190 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fef580b1b70 tx=0x7fef48013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.427+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fef4c0539e0 con 0x7fef5806ea70 2023-06-10T18:57:46.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:46.658+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fef580acc50 con 0x7fef5806ea70 2023-06-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.278+0000 7fef46ffd700 1 -- 172.21.15.104:0/3360725211 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v102) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef4c057000 con 0x7fef5806ea70 2023-06-10T18:57:47.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 msgr2=0x7fef3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 0x7fef3806d190 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fef580b1b70 tx=0x7fef48013000 comp rx=0 tx=0).stop 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 msgr2=0x7fef58151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c4d0 tx=0x7fef4c00c890 comp rx=0 tx=0).stop 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 shutdown_connections 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fef3806ace0 0x7fef3806d190 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef5815c6f0 0x7fef580c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fef5806ea70 0x7fef58151c60 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 --2- 172.21.15.104:0/3360725211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef580b1780 0x7fef58151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 >> 172.21.15.104:0/3360725211 conn(0x7fef5806b8a0 msgr2=0x7fef580b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 shutdown_connections 2023-06-10T18:57:47.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.281+0000 7fef5cb7a700 1 -- 172.21.15.104:0/3360725211 wait complete. 2023-06-10T18:57:47.300 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:57:47.300 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:47.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.421+0000 7f96cf4ab700 1 Processor -- start 2023-06-10T18:57:47.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.421+0000 7f96cf4ab700 1 -- start start 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cf4ab700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96c806e0e0 0x7f96c806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cf4ab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cf4ab700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c806d290 con 0x7f96c806e4b0 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cf4ab700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c8071640 con 0x7f96c806e0e0 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cf4ab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c80aef00 con 0x7f96c8069e70 2023-06-10T18:57:47.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cd247700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f96c806e0e0 0x7f96c806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cca46700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f96c806e4b0 0x7f96c80c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37118/0 (socket says 172.21.15.104:37118) 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.422+0000 7f96cca46700 1 -- 172.21.15.104:0/3220379399 learned_addr learned my addr 172.21.15.104:0/3220379399 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cda48700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cd247700 1 -- 172.21.15.104:0/3220379399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 msgr2=0x7f96c80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cd247700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c80ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cd247700 1 -- 172.21.15.104:0/3220379399 >> v1:172.21.15.104:6789/0 conn(0x7f96c806e4b0 legacy=0x7f96c80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96bffff700 1 -- 172.21.15.104:0/3220379399 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368024223 0 0) 0x7f96c806d290 con 0x7f96c806e4b0 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cd247700 1 -- 172.21.15.104:0/3220379399 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c80ca2d0 con 0x7f96c806e0e0 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cda48700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c80ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96cd247700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e0e0 0x7f96c806cd50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009820 tx=0x7f96b8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221276e54c2e3371 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96bffff700 1 -- 172.21.15.104:0/3220379399 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8005e30 con 0x7f96c806e0e0 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96bffff700 1 -- 172.21.15.104:0/3220379399 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8006030 con 0x7f96c806e0e0 2023-06-10T18:57:47.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.423+0000 7f96bffff700 1 -- 172.21.15.104:0/3220379399 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b80063e0 con 0x7f96c806e0e0 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 -- 172.21.15.104:0/3220379399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e0e0 msgr2=0x7f96c806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e0e0 0x7f96c806cd50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009820 tx=0x7f96b8009b30 comp rx=0 tx=0).stop 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 -- 172.21.15.104:0/3220379399 shutdown_connections 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e0e0 0x7f96c806cd50 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/3220379399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c80ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 -- 172.21.15.104:0/3220379399 >> 172.21.15.104:0/3220379399 conn(0x7f96c806b8a0 msgr2=0x7f96c80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:47.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 -- 172.21.15.104:0/3220379399 shutdown_connections 2023-06-10T18:57:47.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.424+0000 7f96cf4ab700 1 -- 172.21.15.104:0/3220379399 wait complete. 2023-06-10T18:57:47.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.425+0000 7f96cf4ab700 1 Processor -- start 2023-06-10T18:57:47.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.425+0000 7f96cf4ab700 1 -- start start 2023-06-10T18:57:47.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c814bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96c806e0e0 0x7f96c80ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c814c330 con 0x7f96c806e4b0 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81e5580 con 0x7f96c806e0e0 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c80b3010 con 0x7f96c8069e70 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39300/0 (socket says 172.21.15.104:39300) 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 -- 172.21.15.104:0/2372596849 learned_addr learned my addr 172.21.15.104:0/2372596849 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:47.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 -- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 msgr2=0x7f96c814bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cca46700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96c806e0e0 0x7f96c80ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cd247700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c814bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c814bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 -- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96c806e0e0 msgr2=0x7f96c80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96c806e0e0 0x7f96c80ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 -- 172.21.15.104:0/2372596849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8009430 con 0x7f96c806e4b0 2023-06-10T18:57:47.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cda48700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f96c000cd50 tx=0x7f96c000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c000f480 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c000f600 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c80b32f0 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.426+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c000f9b0 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.427+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c80b38b0 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.427+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c8074050 con 0x7f96c806e4b0 2023-06-10T18:57:47.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.428+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96c000b040 con 0x7f96c806e4b0 2023-06-10T18:57:47.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.428+0000 7f96bdffb700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 0x7f96ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.428+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f96c0089dd0 con 0x7f96c806e4b0 2023-06-10T18:57:47.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.428+0000 7f96cd247700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 0x7f96ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.429+0000 7f96cd247700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 0x7f96ac06d190 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f96c80b2040 tx=0x7f96b8008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.433+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f96c0054d40 con 0x7f96c806e4b0 2023-06-10T18:57:47.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.663+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c80707a0 con 0x7f96c806e4b0 2023-06-10T18:57:47.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.664+0000 7f96bdffb700 1 -- 172.21.15.104:0/2372596849 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f96c0058360 con 0x7f96c806e4b0 2023-06-10T18:57:47.669 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:47.670 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:57:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 msgr2=0x7f96ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 0x7f96ac06d190 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f96c80b2040 tx=0x7f96b8008070 comp rx=0 tx=0).stop 2023-06-10T18:57:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 msgr2=0x7f96c80ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f96c000cd50 tx=0x7f96c000a560 comp rx=0 tx=0).stop 2023-06-10T18:57:47.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 shutdown_connections 2023-06-10T18:57:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96c806e4b0 0x7f96c80ced80 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96c8069e70 0x7f96c814bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f96ac06ace0 0x7f96ac06d190 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 --2- 172.21.15.104:0/2372596849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96c806e0e0 0x7f96c80ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 >> 172.21.15.104:0/2372596849 conn(0x7f96c806b8a0 msgr2=0x7f96c80ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 shutdown_connections 2023-06-10T18:57:47.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.666+0000 7f96cf4ab700 1 -- 172.21.15.104:0/2372596849 wait complete. 2023-06-10T18:57:47.676 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:57:47.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:57:47.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 Processor -- start 2023-06-10T18:57:47.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 -- start start 2023-06-10T18:57:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8aac06e0e0 0x7f8aac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac06d290 con 0x7f8aac069e70 2023-06-10T18:57:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0ca2d0 con 0x7f8aac06e4b0 2023-06-10T18:57:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab2b05700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac071640 con 0x7f8aac06e0e0 2023-06-10T18:57:47.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.799+0000 7f8ab10a2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8aac069e70 0x7f8aac0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37136/0 (socket says 172.21.15.104:37136) 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8ab10a2700 1 -- 172.21.15.104:0/870489267 learned_addr learned my addr 172.21.15.104:0/870489267 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348935533 0 0) 0x7f8aac06d290 con 0x7f8aac069e70 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c0035b0 con 0x7f8aac069e70 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8ab08a1700 1 --2- 172.21.15.104:0/870489267 >> v2:172.21.15.114:3300/0 conn(0x7f8aac06e0e0 0x7f8aac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835129779 0 0) 0x7f8a9c0035b0 con 0x7f8aac069e70 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 >> v2:172.21.15.114:3300/0 conn(0x7f8aac06e0e0 msgr2=0x7f8aac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:47.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 --2- 172.21.15.104:0/870489267 >> v2:172.21.15.114:3300/0 conn(0x7f8aac06e0e0 0x7f8aac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 >> v1:172.21.15.114:6789/0 conn(0x7f8aac06e4b0 legacy=0x7f8aac0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac06d4c0 con 0x7f8aac069e70 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8aa40019f0 con 0x7f8aac069e70 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 >> v1:172.21.15.104:6789/0 conn(0x7f8aac069e70 legacy=0x7f8aac0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a9c003a00 0x7f8a9c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8a9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a9c005320 0x7f8a9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac071640 con 0x7f8a9c004850 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c00c250 con 0x7f8a9c005320 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c00c410 con 0x7f8a9c003a00 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aabfff700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8a9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8ab08a1700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a9c003a00 0x7f8a9c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8ab10a2700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a9c005320 0x7f8a9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aabfff700 1 -- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a9c003a00 msgr2=0x7f8a9c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aabfff700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a9c003a00 0x7f8a9c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aabfff700 1 -- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a9c005320 msgr2=0x7f8a9c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.800+0000 7f8aabfff700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a9c005320 0x7f8a9c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8aabfff700 1 -- 172.21.15.104:0/870489267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0035b0 con 0x7f8a9c004850 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab10a2700 1 -- 172.21.15.104:0/870489267 reap_dead start 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8aabfff700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8a9c004cc0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8a980017c0 tx=0x7f8a9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326462486f503972 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a980052d0 con 0x7f8a9c004850 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98005450 con 0x7f8a9c004850 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8aab7fe700 1 -- 172.21.15.104:0/870489267 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a980038c0 con 0x7f8a9c004850 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 -- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 msgr2=0x7f8a9c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8a9c004cc0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8a980017c0 tx=0x7f8a9c005880 comp rx=0 tx=0).stop 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 -- 172.21.15.104:0/870489267 shutdown_connections 2023-06-10T18:57:47.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/870489267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8a9c004cc0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 -- 172.21.15.104:0/870489267 >> 172.21.15.104:0/870489267 conn(0x7f8aac06b8a0 msgr2=0x7f8aac0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:47.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 -- 172.21.15.104:0/870489267 shutdown_connections 2023-06-10T18:57:47.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.801+0000 7f8ab2b05700 1 -- 172.21.15.104:0/870489267 wait complete. 2023-06-10T18:57:47.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.802+0000 7f8ab2b05700 1 Processor -- start 2023-06-10T18:57:47.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 -- start start 2023-06-10T18:57:47.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8aac159380 0x7f8aac0cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 0x7f8aac1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1582e0 con 0x7f8a9c004850 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1585c0 con 0x7f8aac159380 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab2b05700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1588a0 con 0x7f8aac156f30 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab08a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.803+0000 7f8ab08a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39316/0 (socket says 172.21.15.104:39316) 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 -- 172.21.15.104:0/1127904321 learned_addr learned my addr 172.21.15.104:0/1127904321 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8aabfff700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8aac159380 0x7f8aac0cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab10a2700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 0x7f8aac1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 -- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 msgr2=0x7f8aac1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 0x7f8aac1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 -- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8aac159380 msgr2=0x7f8aac0cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8aac159380 0x7f8aac0cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 -- 172.21.15.104:0/1127904321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac158b80 con 0x7f8a9c004850 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab10a2700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 0x7f8aac1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:47.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab08a1700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0011560 tx=0x7f8aa0011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0005e30 con 0x7f8a9c004850 2023-06-10T18:57:47.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa0006050 con 0x7f8a9c004850 2023-06-10T18:57:47.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac158e60 con 0x7f8a9c004850 2023-06-10T18:57:47.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00063c0 con 0x7f8a9c004850 2023-06-10T18:57:47.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.804+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac1578e0 con 0x7f8a9c004850 2023-06-10T18:57:47.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.805+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac0cdd80 con 0x7f8a9c004850 2023-06-10T18:57:47.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.806+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8aa0006520 con 0x7f8a9c004850 2023-06-10T18:57:47.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.806+0000 7f8aa97fa700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 0x7f8a9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:47.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.806+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8aa0089e20 con 0x7f8a9c004850 2023-06-10T18:57:47.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.806+0000 7f8aabfff700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 0x7f8a9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:47.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.807+0000 7f8aabfff700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 0x7f8a9006d190 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8a98005040 tx=0x7f8a98009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:47.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.810+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8aa0054e40 con 0x7f8a9c004850 2023-06-10T18:57:48.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.996+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aac074050 con 0x7f8a9c004850 2023-06-10T18:57:48.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.997+0000 7f8aa97fa700 1 -- 172.21.15.104:0/1127904321 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+18606 (secure 0 0 0) 0x7f8aa0058460 con 0x7f8a9c004850 2023-06-10T18:57:48.003 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:48.003 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":159,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:45.260324+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:57:48.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 msgr2=0x7f8a9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 0x7f8a9006d190 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8a98005040 tx=0x7f8a98009030 comp rx=0 tx=0).stop 2023-06-10T18:57:48.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 msgr2=0x7f8aac0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0011560 tx=0x7f8aa0011870 comp rx=0 tx=0).stop 2023-06-10T18:57:48.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 shutdown_connections 2023-06-10T18:57:48.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8aac156f30 0x7f8aac1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a9c004850 0x7f8aac0ce9f0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8a9006ace0 0x7f8a9006d190 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 --2- 172.21.15.104:0/1127904321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8aac159380 0x7f8aac0cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 >> 172.21.15.104:0/1127904321 conn(0x7f8aac06b8a0 msgr2=0x7f8aac06a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 shutdown_connections 2023-06-10T18:57:48.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:47.999+0000 7f8ab2b05700 1 -- 172.21.15.104:0/1127904321 wait complete. 2023-06-10T18:57:48.017 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:57:48.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:57:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.131+0000 7f2ad1d29700 1 Processor -- start 2023-06-10T18:57:48.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.131+0000 7f2ad1d29700 1 -- start start 2023-06-10T18:57:48.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2ad1d29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2acc0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2ad1d29700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0b6070 con 0x7f2acc0ce890 2023-06-10T18:57:48.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2ad1d29700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0b1d70 con 0x7f2acc15a420 2023-06-10T18:57:48.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2ad1d29700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0ced10 con 0x7f2acc0b4ff0 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2acbfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2acc0ce890 0x7f2acc15a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37156/0 (socket says 172.21.15.104:37156) 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.132+0000 7f2acbfff700 1 -- 172.21.15.104:0/2821149955 learned_addr learned my addr 172.21.15.104:0/2821149955 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2acc0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797157592 0 0) 0x7f2acc0b6070 con 0x7f2acc0ce890 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab40035b0 con 0x7f2acc0ce890 2023-06-10T18:57:48.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252084224 0 0) 0x7f2ab40035b0 con 0x7f2acc0ce890 2023-06-10T18:57:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 msgr2=0x7f2acc0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2acc0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 >> v1:172.21.15.114:6789/0 conn(0x7f2acc15a420 legacy=0x7f2acc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0b5460 con 0x7f2acc0ce890 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2ac40019f0 con 0x7f2acc0ce890 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 >> v1:172.21.15.104:6789/0 conn(0x7f2acc0ce890 legacy=0x7f2acc15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab4003a30 0x7f2ab4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab4004850 0x7f2ab4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2ab4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0ced10 con 0x7f2ab4004850 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab400bac0 con 0x7f2ab4003a30 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab400bc50 con 0x7f2acc0b4ff0 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab4004850 0x7f2ab4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acb7fe700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab4003a30 0x7f2ab4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acbfff700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2ab4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 -- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 msgr2=0x7f2ab4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:48.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b4ff0 0x7f2ab4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 -- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab4003a30 msgr2=0x7f2ab4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab4003a30 0x7f2ab4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 -- 172.21.15.104:0/2821149955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab40035b0 con 0x7f2ab4004850 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acb7fe700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab4003a30 0x7f2ab4003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.133+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab4004850 0x7f2ab4004cc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009b60 tx=0x7f2ab4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e15dc17420da77 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00e6c0 con 0x7f2ab4004850 2023-06-10T18:57:48.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc00e820 con 0x7f2ab4004850 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2aca7fc700 1 -- 172.21.15.104:0/2821149955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00eb90 con 0x7f2ab4004850 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2821149955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab4004850 msgr2=0x7f2ab4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2821149955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab4004850 0x7f2ab4004cc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009b60 tx=0x7f2ab4005880 comp rx=0 tx=0).stop 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2acbfff700 1 -- 172.21.15.104:0/2821149955 reap_dead start 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2821149955 shutdown_connections 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2821149955 >> 172.21.15.104:0/2821149955 conn(0x7f2acc089fd0 msgr2=0x7f2acc0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2821149955 shutdown_connections 2023-06-10T18:57:48.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.134+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2821149955 wait complete. 2023-06-10T18:57:48.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.135+0000 7f2ad1d29700 1 Processor -- start 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.135+0000 7f2ad1d29700 1 -- start start 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ab4004850 0x7f2acc158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc1567e0 0x7f2acc156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0c8cb0 con 0x7f2acc0c8260 2023-06-10T18:57:48.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0c8f90 con 0x7f2acc1567e0 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2ad1d29700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0c9270 con 0x7f2ab4004850 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2acaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2acaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39338/0 (socket says 172.21.15.104:39338) 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.136+0000 7f2acaffd700 1 -- 172.21.15.104:0/2951069151 learned_addr learned my addr 172.21.15.104:0/2951069151 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 -- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ab4004850 msgr2=0x7f2acc158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acbfff700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc1567e0 0x7f2acc156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ab4004850 0x7f2acc158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 -- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc1567e0 msgr2=0x7f2acc156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc1567e0 0x7f2acc156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 -- 172.21.15.104:0/2951069151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0c9550 con 0x7f2acc0c8260 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2acaffd700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40047a0 tx=0x7f2abc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc021070 con 0x7f2acc0c8260 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc0045d0 con 0x7f2acc0c8260 2023-06-10T18:57:48.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc0c8cb0 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00e6c0 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.137+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc157190 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.138+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc071920 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.138+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2abc025070 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.139+0000 7f2ab3fff700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 0x7f2ab806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.139+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2abc08a720 con 0x7f2acc0c8260 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.139+0000 7f2acb7fe700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 0x7f2ab806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.139+0000 7f2acb7fe700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 0x7f2ab806d190 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0007b30 tx=0x7f2ac0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.143+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2abc055840 con 0x7f2acc0c8260 2023-06-10T18:57:48.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.331+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2acc0b5a20 con 0x7f2ab806ace0 2023-06-10T18:57:48.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.340+0000 7f2ab3fff700 1 -- 172.21.15.104:0/2951069151 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311612 (secure 0 0 0) 0x7f2ac00050d0 con 0x7f2ab806ace0 2023-06-10T18:57:48.349 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:48.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 msgr2=0x7f2ab806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 0x7f2ab806d190 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0007b30 tx=0x7f2ac0009040 comp rx=0 tx=0).stop 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 msgr2=0x7f2acc0c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40047a0 tx=0x7f2abc0047c0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 shutdown_connections 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ab806ace0 0x7f2ab806d190 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc1567e0 0x7f2acc156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ab4004850 0x7f2acc158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 --2- 172.21.15.104:0/2951069151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2acc0c8260 0x7f2acc0c86d0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.343+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 >> 172.21.15.104:0/2951069151 conn(0x7f2acc089fd0 msgr2=0x7f2acc1596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.344+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 shutdown_connections 2023-06-10T18:57:48.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.344+0000 7f2ad1d29700 1 -- 172.21.15.104:0/2951069151 wait complete. 2023-06-10T18:57:48.352 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:57:48.361 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2023-06-10T18:57:47.567597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244664,"kb_used_data":16760,"kb_used_omap":0,"kb_used_meta":227840,"kb_avail":754730056,"statfs":{"total":773094113280,"available":772843577344,"internally_reserved":0,"allocated":17162240,"data_stored":9000928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":233308160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719619"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437402+0000","last_change":"2023-06-10T18:57:44.283604+0000","last_active":"2023-06-10T18:57:45.437402+0000","last_peered":"2023-06-10T18:57:45.437402+0000","last_clean":"2023-06-10T18:57:45.437402+0000","last_became_active":"2023-06-10T18:57:44.283222+0000","last_became_peered":"2023-06-10T18:57:44.283222+0000","last_unstale":"2023-06-10T18:57:45.437402+0000","last_undegraded":"2023-06-10T18:57:45.437402+0000","last_fullsized":"2023-06-10T18:57:45.437402+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:33.572694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436977+0000","last_change":"2023-06-10T18:57:44.286875+0000","last_active":"2023-06-10T18:57:45.436977+0000","last_peered":"2023-06-10T18:57:45.436977+0000","last_clean":"2023-06-10T18:57:45.436977+0000","last_became_active":"2023-06-10T18:57:44.286449+0000","last_became_peered":"2023-06-10T18:57:44.286449+0000","last_unstale":"2023-06-10T18:57:45.436977+0000","last_undegraded":"2023-06-10T18:57:45.436977+0000","last_fullsized":"2023-06-10T18:57:45.436977+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:41:23.785356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270814+0000","last_change":"2023-06-10T18:57:44.293253+0000","last_active":"2023-06-10T18:57:45.270814+0000","last_peered":"2023-06-10T18:57:45.270814+0000","last_clean":"2023-06-10T18:57:45.270814+0000","last_became_active":"2023-06-10T18:57:44.292969+0000","last_became_peered":"2023-06-10T18:57:44.292969+0000","last_unstale":"2023-06-10T18:57:45.270814+0000","last_undegraded":"2023-06-10T18:57:45.270814+0000","last_fullsized":"2023-06-10T18:57:45.270814+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:16.217590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437507+0000","last_change":"2023-06-10T18:57:44.280914+0000","last_active":"2023-06-10T18:57:45.437507+0000","last_peered":"2023-06-10T18:57:45.437507+0000","last_clean":"2023-06-10T18:57:45.437507+0000","last_became_active":"2023-06-10T18:57:44.280484+0000","last_became_peered":"2023-06-10T18:57:44.280484+0000","last_unstale":"2023-06-10T18:57:45.437507+0000","last_undegraded":"2023-06-10T18:57:45.437507+0000","last_fullsized":"2023-06-10T18:57:45.437507+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:11.577091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437089+0000","last_change":"2023-06-10T18:57:44.291192+0000","last_active":"2023-06-10T18:57:45.437089+0000","last_peered":"2023-06-10T18:57:45.437089+0000","last_clean":"2023-06-10T18:57:45.437089+0000","last_became_active":"2023-06-10T18:57:44.290898+0000","last_became_peered":"2023-06-10T18:57:44.290898+0000","last_unstale":"2023-06-10T18:57:45.437089+0000","last_undegraded":"2023-06-10T18:57:45.437089+0000","last_fullsized":"2023-06-10T18:57:45.437089+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:47.708549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437302+0000","last_change":"2023-06-10T18:57:44.287600+0000","last_active":"2023-06-10T18:57:45.437302+0000","last_peered":"2023-06-10T18:57:45.437302+0000","last_clean":"2023-06-10T18:57:45.437302+0000","last_became_active":"2023-06-10T18:57:44.287306+0000","last_became_peered":"2023-06-10T18:57:44.287306+0000","last_unstale":"2023-06-10T18:57:45.437302+0000","last_undegraded":"2023-06-10T18:57:45.437302+0000","last_fullsized":"2023-06-10T18:57:45.437302+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:19:38.702174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268698+0000","last_change":"2023-06-10T18:57:44.290226+0000","last_active":"2023-06-10T18:57:45.268698+0000","last_peered":"2023-06-10T18:57:45.268698+0000","last_clean":"2023-06-10T18:57:45.268698+0000","last_became_active":"2023-06-10T18:57:44.289960+0000","last_became_peered":"2023-06-10T18:57:44.289960+0000","last_unstale":"2023-06-10T18:57:45.268698+0000","last_undegraded":"2023-06-10T18:57:45.268698+0000","last_fullsized":"2023-06-10T18:57:45.268698+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:10:42.872345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436804+0000","last_change":"2023-06-10T18:57:44.285495+0000","last_active":"2023-06-10T18:57:45.436804+0000","last_peered":"2023-06-10T18:57:45.436804+0000","last_clean":"2023-06-10T18:57:45.436804+0000","last_became_active":"2023-06-10T18:57:44.285037+0000","last_became_peered":"2023-06-10T18:57:44.285037+0000","last_unstale":"2023-06-10T18:57:45.436804+0000","last_undegraded":"2023-06-10T18:57:45.436804+0000","last_fullsized":"2023-06-10T18:57:45.436804+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:04.458580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437402+0000","last_change":"2023-06-10T18:57:44.292670+0000","last_active":"2023-06-10T18:57:45.437402+0000","last_peered":"2023-06-10T18:57:45.437402+0000","last_clean":"2023-06-10T18:57:45.437402+0000","last_became_active":"2023-06-10T18:57:44.292385+0000","last_became_peered":"2023-06-10T18:57:44.292385+0000","last_unstale":"2023-06-10T18:57:45.437402+0000","last_undegraded":"2023-06-10T18:57:45.437402+0000","last_fullsized":"2023-06-10T18:57:45.437402+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:07.473584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437588+0000","last_change":"2023-06-10T18:57:44.286703+0000","last_active":"2023-06-10T18:57:45.437588+0000","last_peered":"2023-06-10T18:57:45.437588+0000","last_clean":"2023-06-10T18:57:45.437588+0000","last_became_active":"2023-06-10T18:57:44.286125+0000","last_became_peered":"2023-06-10T18:57:44.286125+0000","last_unstale":"2023-06-10T18:57:45.437588+0000","last_undegraded":"2023-06-10T18:57:45.437588+0000","last_fullsized":"2023-06-10T18:57:45.437588+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:09.492898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437228+0000","last_change":"2023-06-10T18:57:44.291139+0000","last_active":"2023-06-10T18:57:45.437228+0000","last_peered":"2023-06-10T18:57:45.437228+0000","last_clean":"2023-06-10T18:57:45.437228+0000","last_became_active":"2023-06-10T18:57:44.290850+0000","last_became_peered":"2023-06-10T18:57:44.290850+0000","last_unstale":"2023-06-10T18:57:45.437228+0000","last_undegraded":"2023-06-10T18:57:45.437228+0000","last_fullsized":"2023-06-10T18:57:45.437228+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:21:56.384098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437297+0000","last_change":"2023-06-10T18:57:44.289202+0000","last_active":"2023-06-10T18:57:45.437297+0000","last_peered":"2023-06-10T18:57:45.437297+0000","last_clean":"2023-06-10T18:57:45.437297+0000","last_became_active":"2023-06-10T18:57:44.287564+0000","last_became_peered":"2023-06-10T18:57:44.287564+0000","last_unstale":"2023-06-10T18:57:45.437297+0000","last_undegraded":"2023-06-10T18:57:45.437297+0000","last_fullsized":"2023-06-10T18:57:45.437297+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:02.229426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270797+0000","last_change":"2023-06-10T18:57:44.282894+0000","last_active":"2023-06-10T18:57:45.270797+0000","last_peered":"2023-06-10T18:57:45.270797+0000","last_clean":"2023-06-10T18:57:45.270797+0000","last_became_active":"2023-06-10T18:57:44.282552+0000","last_became_peered":"2023-06-10T18:57:44.282552+0000","last_unstale":"2023-06-10T18:57:45.270797+0000","last_undegraded":"2023-06-10T18:57:45.270797+0000","last_fullsized":"2023-06-10T18:57:45.270797+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:25.425413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270063+0000","last_change":"2023-06-10T18:57:44.285585+0000","last_active":"2023-06-10T18:57:45.270063+0000","last_peered":"2023-06-10T18:57:45.270063+0000","last_clean":"2023-06-10T18:57:45.270063+0000","last_became_active":"2023-06-10T18:57:44.283623+0000","last_became_peered":"2023-06-10T18:57:44.283623+0000","last_unstale":"2023-06-10T18:57:45.270063+0000","last_undegraded":"2023-06-10T18:57:45.270063+0000","last_fullsized":"2023-06-10T18:57:45.270063+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:19.758784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436943+0000","last_change":"2023-06-10T18:57:44.273697+0000","last_active":"2023-06-10T18:57:45.436943+0000","last_peered":"2023-06-10T18:57:45.436943+0000","last_clean":"2023-06-10T18:57:45.436943+0000","last_became_active":"2023-06-10T18:57:44.272521+0000","last_became_peered":"2023-06-10T18:57:44.272521+0000","last_unstale":"2023-06-10T18:57:45.436943+0000","last_undegraded":"2023-06-10T18:57:45.436943+0000","last_fullsized":"2023-06-10T18:57:45.436943+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:43.082174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268813+0000","last_change":"2023-06-10T18:57:44.283599+0000","last_active":"2023-06-10T18:57:45.268813+0000","last_peered":"2023-06-10T18:57:45.268813+0000","last_clean":"2023-06-10T18:57:45.268813+0000","last_became_active":"2023-06-10T18:57:44.283351+0000","last_became_peered":"2023-06-10T18:57:44.283351+0000","last_unstale":"2023-06-10T18:57:45.268813+0000","last_undegraded":"2023-06-10T18:57:45.268813+0000","last_fullsized":"2023-06-10T18:57:45.268813+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:32.256088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437270+0000","last_change":"2023-06-10T18:57:44.293823+0000","last_active":"2023-06-10T18:57:45.437270+0000","last_peered":"2023-06-10T18:57:45.437270+0000","last_clean":"2023-06-10T18:57:45.437270+0000","last_became_active":"2023-06-10T18:57:44.293603+0000","last_became_peered":"2023-06-10T18:57:44.293603+0000","last_unstale":"2023-06-10T18:57:45.437270+0000","last_undegraded":"2023-06-10T18:57:45.437270+0000","last_fullsized":"2023-06-10T18:57:45.437270+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:15.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438099+0000","last_change":"2023-06-10T18:57:44.285495+0000","last_active":"2023-06-10T18:57:45.438099+0000","last_peered":"2023-06-10T18:57:45.438099+0000","last_clean":"2023-06-10T18:57:45.438099+0000","last_became_active":"2023-06-10T18:57:44.285094+0000","last_became_peered":"2023-06-10T18:57:44.285094+0000","last_unstale":"2023-06-10T18:57:45.438099+0000","last_undegraded":"2023-06-10T18:57:45.438099+0000","last_fullsized":"2023-06-10T18:57:45.438099+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:39:03.044414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436912+0000","last_change":"2023-06-10T18:57:44.290254+0000","last_active":"2023-06-10T18:57:45.436912+0000","last_peered":"2023-06-10T18:57:45.436912+0000","last_clean":"2023-06-10T18:57:45.436912+0000","last_became_active":"2023-06-10T18:57:44.289962+0000","last_became_peered":"2023-06-10T18:57:44.289962+0000","last_unstale":"2023-06-10T18:57:45.436912+0000","last_undegraded":"2023-06-10T18:57:45.436912+0000","last_fullsized":"2023-06-10T18:57:45.436912+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:13.547571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270327+0000","last_change":"2023-06-10T18:57:44.289241+0000","last_active":"2023-06-10T18:57:45.270327+0000","last_peered":"2023-06-10T18:57:45.270327+0000","last_clean":"2023-06-10T18:57:45.270327+0000","last_became_active":"2023-06-10T18:57:44.288980+0000","last_became_peered":"2023-06-10T18:57:44.288980+0000","last_unstale":"2023-06-10T18:57:45.270327+0000","last_undegraded":"2023-06-10T18:57:45.270327+0000","last_fullsized":"2023-06-10T18:57:45.270327+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:45.835332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437834+0000","last_change":"2023-06-10T18:57:44.283706+0000","last_active":"2023-06-10T18:57:45.437834+0000","last_peered":"2023-06-10T18:57:45.437834+0000","last_clean":"2023-06-10T18:57:45.437834+0000","last_became_active":"2023-06-10T18:57:44.283443+0000","last_became_peered":"2023-06-10T18:57:44.283443+0000","last_unstale":"2023-06-10T18:57:45.437834+0000","last_undegraded":"2023-06-10T18:57:45.437834+0000","last_fullsized":"2023-06-10T18:57:45.437834+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:08:10.445545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437334+0000","last_change":"2023-06-10T18:57:44.285648+0000","last_active":"2023-06-10T18:57:45.437334+0000","last_peered":"2023-06-10T18:57:45.437334+0000","last_clean":"2023-06-10T18:57:45.437334+0000","last_became_active":"2023-06-10T18:57:44.285374+0000","last_became_peered":"2023-06-10T18:57:44.285374+0000","last_unstale":"2023-06-10T18:57:45.437334+0000","last_undegraded":"2023-06-10T18:57:45.437334+0000","last_fullsized":"2023-06-10T18:57:45.437334+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:34:10.143555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436943+0000","last_change":"2023-06-10T18:57:44.286190+0000","last_active":"2023-06-10T18:57:45.436943+0000","last_peered":"2023-06-10T18:57:45.436943+0000","last_clean":"2023-06-10T18:57:45.436943+0000","last_became_active":"2023-06-10T18:57:44.285822+0000","last_became_peered":"2023-06-10T18:57:44.285822+0000","last_unstale":"2023-06-10T18:57:45.436943+0000","last_undegraded":"2023-06-10T18:57:45.436943+0000","last_fullsized":"2023-06-10T18:57:45.436943+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:34:21.036099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436894+0000","last_change":"2023-06-10T18:57:44.285308+0000","last_active":"2023-06-10T18:57:45.436894+0000","last_peered":"2023-06-10T18:57:45.436894+0000","last_clean":"2023-06-10T18:57:45.436894+0000","last_became_active":"2023-06-10T18:57:44.284652+0000","last_became_peered":"2023-06-10T18:57:44.284652+0000","last_unstale":"2023-06-10T18:57:45.436894+0000","last_undegraded":"2023-06-10T18:57:45.436894+0000","last_fullsized":"2023-06-10T18:57:45.436894+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:29.741070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270501+0000","last_change":"2023-06-10T18:57:44.291329+0000","last_active":"2023-06-10T18:57:45.270501+0000","last_peered":"2023-06-10T18:57:45.270501+0000","last_clean":"2023-06-10T18:57:45.270501+0000","last_became_active":"2023-06-10T18:57:44.291071+0000","last_became_peered":"2023-06-10T18:57:44.291071+0000","last_unstale":"2023-06-10T18:57:45.270501+0000","last_undegraded":"2023-06-10T18:57:45.270501+0000","last_fullsized":"2023-06-10T18:57:45.270501+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:00:21.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268768+0000","last_change":"2023-06-10T18:57:44.279087+0000","last_active":"2023-06-10T18:57:45.268768+0000","last_peered":"2023-06-10T18:57:45.268768+0000","last_clean":"2023-06-10T18:57:45.268768+0000","last_became_active":"2023-06-10T18:57:44.278829+0000","last_became_peered":"2023-06-10T18:57:44.278829+0000","last_unstale":"2023-06-10T18:57:45.268768+0000","last_undegraded":"2023-06-10T18:57:45.268768+0000","last_fullsized":"2023-06-10T18:57:45.268768+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:24.570339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270809+0000","last_change":"2023-06-10T18:57:44.286304+0000","last_active":"2023-06-10T18:57:45.270809+0000","last_peered":"2023-06-10T18:57:45.270809+0000","last_clean":"2023-06-10T18:57:45.270809+0000","last_became_active":"2023-06-10T18:57:44.285993+0000","last_became_peered":"2023-06-10T18:57:44.285993+0000","last_unstale":"2023-06-10T18:57:45.270809+0000","last_undegraded":"2023-06-10T18:57:45.270809+0000","last_fullsized":"2023-06-10T18:57:45.270809+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:30.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437639+0000","last_change":"2023-06-10T18:57:44.278817+0000","last_active":"2023-06-10T18:57:45.437639+0000","last_peered":"2023-06-10T18:57:45.437639+0000","last_clean":"2023-06-10T18:57:45.437639+0000","last_became_active":"2023-06-10T18:57:44.278539+0000","last_became_peered":"2023-06-10T18:57:44.278539+0000","last_unstale":"2023-06-10T18:57:45.437639+0000","last_undegraded":"2023-06-10T18:57:45.437639+0000","last_fullsized":"2023-06-10T18:57:45.437639+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:10.604887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270241+0000","last_change":"2023-06-10T18:57:44.283599+0000","last_active":"2023-06-10T18:57:45.270241+0000","last_peered":"2023-06-10T18:57:45.270241+0000","last_clean":"2023-06-10T18:57:45.270241+0000","last_became_active":"2023-06-10T18:57:44.282650+0000","last_became_peered":"2023-06-10T18:57:44.282650+0000","last_unstale":"2023-06-10T18:57:45.270241+0000","last_undegraded":"2023-06-10T18:57:45.270241+0000","last_fullsized":"2023-06-10T18:57:45.270241+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:01.272573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269193+0000","last_change":"2023-06-10T18:57:44.286563+0000","last_active":"2023-06-10T18:57:45.269193+0000","last_peered":"2023-06-10T18:57:45.269193+0000","last_clean":"2023-06-10T18:57:45.269193+0000","last_became_active":"2023-06-10T18:57:44.286238+0000","last_became_peered":"2023-06-10T18:57:44.286238+0000","last_unstale":"2023-06-10T18:57:45.269193+0000","last_undegraded":"2023-06-10T18:57:45.269193+0000","last_fullsized":"2023-06-10T18:57:45.269193+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:12:23.856395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269253+0000","last_change":"2023-06-10T18:57:44.289030+0000","last_active":"2023-06-10T18:57:45.269253+0000","last_peered":"2023-06-10T18:57:45.269253+0000","last_clean":"2023-06-10T18:57:45.269253+0000","last_became_active":"2023-06-10T18:57:44.285903+0000","last_became_peered":"2023-06-10T18:57:44.285903+0000","last_unstale":"2023-06-10T18:57:45.269253+0000","last_undegraded":"2023-06-10T18:57:45.269253+0000","last_fullsized":"2023-06-10T18:57:45.269253+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:29:27.097073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437768+0000","last_change":"2023-06-10T18:57:44.285109+0000","last_active":"2023-06-10T18:57:45.437768+0000","last_peered":"2023-06-10T18:57:45.437768+0000","last_clean":"2023-06-10T18:57:45.437768+0000","last_became_active":"2023-06-10T18:57:44.284836+0000","last_became_peered":"2023-06-10T18:57:44.284836+0000","last_unstale":"2023-06-10T18:57:45.437768+0000","last_undegraded":"2023-06-10T18:57:45.437768+0000","last_fullsized":"2023-06-10T18:57:45.437768+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:19.338867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269061+0000","last_change":"2023-06-10T18:57:44.278516+0000","last_active":"2023-06-10T18:57:45.269061+0000","last_peered":"2023-06-10T18:57:45.269061+0000","last_clean":"2023-06-10T18:57:45.269061+0000","last_became_active":"2023-06-10T18:57:44.278317+0000","last_became_peered":"2023-06-10T18:57:44.278317+0000","last_unstale":"2023-06-10T18:57:45.269061+0000","last_undegraded":"2023-06-10T18:57:45.269061+0000","last_fullsized":"2023-06-10T18:57:45.269061+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:41.651403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"158'4","reported_seq":17,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437274+0000","last_change":"2023-06-10T18:57:43.878349+0000","last_active":"2023-06-10T18:57:45.437274+0000","last_peered":"2023-06-10T18:57:45.437274+0000","last_clean":"2023-06-10T18:57:45.437274+0000","last_became_active":"2023-06-10T18:57:43.878026+0000","last_became_peered":"2023-06-10T18:57:43.878026+0000","last_unstale":"2023-06-10T18:57:45.437274+0000","last_undegraded":"2023-06-10T18:57:45.437274+0000","last_fullsized":"2023-06-10T18:57:45.437274+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:11:06.161680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270382+0000","last_change":"2023-06-10T18:57:44.286549+0000","last_active":"2023-06-10T18:57:45.270382+0000","last_peered":"2023-06-10T18:57:45.270382+0000","last_clean":"2023-06-10T18:57:45.270382+0000","last_became_active":"2023-06-10T18:57:44.286220+0000","last_became_peered":"2023-06-10T18:57:44.286220+0000","last_unstale":"2023-06-10T18:57:45.270382+0000","last_undegraded":"2023-06-10T18:57:45.270382+0000","last_fullsized":"2023-06-10T18:57:45.270382+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:36:53.266200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270576+0000","last_change":"2023-06-10T18:57:43.877624+0000","last_active":"2023-06-10T18:57:45.270576+0000","last_peered":"2023-06-10T18:57:45.270576+0000","last_clean":"2023-06-10T18:57:45.270576+0000","last_became_active":"2023-06-10T18:57:43.877330+0000","last_became_peered":"2023-06-10T18:57:43.877330+0000","last_unstale":"2023-06-10T18:57:45.270576+0000","last_undegraded":"2023-06-10T18:57:45.270576+0000","last_fullsized":"2023-06-10T18:57:45.270576+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:24.192051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437017+0000","last_change":"2023-06-10T18:57:44.287612+0000","last_active":"2023-06-10T18:57:45.437017+0000","last_peered":"2023-06-10T18:57:45.437017+0000","last_clean":"2023-06-10T18:57:45.437017+0000","last_became_active":"2023-06-10T18:57:44.287361+0000","last_became_peered":"2023-06-10T18:57:44.287361+0000","last_unstale":"2023-06-10T18:57:45.437017+0000","last_undegraded":"2023-06-10T18:57:45.437017+0000","last_fullsized":"2023-06-10T18:57:45.437017+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:45.593784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438054+0000","last_change":"2023-06-10T18:57:43.880376+0000","last_active":"2023-06-10T18:57:45.438054+0000","last_peered":"2023-06-10T18:57:45.438054+0000","last_clean":"2023-06-10T18:57:45.438054+0000","last_became_active":"2023-06-10T18:57:43.880095+0000","last_became_peered":"2023-06-10T18:57:43.880095+0000","last_unstale":"2023-06-10T18:57:45.438054+0000","last_undegraded":"2023-06-10T18:57:45.438054+0000","last_fullsized":"2023-06-10T18:57:45.438054+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:45:36.825550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270924+0000","last_change":"2023-06-10T18:57:44.288721+0000","last_active":"2023-06-10T18:57:45.270924+0000","last_peered":"2023-06-10T18:57:45.270924+0000","last_clean":"2023-06-10T18:57:45.270924+0000","last_became_active":"2023-06-10T18:57:44.287119+0000","last_became_peered":"2023-06-10T18:57:44.287119+0000","last_unstale":"2023-06-10T18:57:45.270924+0000","last_undegraded":"2023-06-10T18:57:45.270924+0000","last_fullsized":"2023-06-10T18:57:45.270924+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:02.631238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437768+0000","last_change":"2023-06-10T18:57:43.874389+0000","last_active":"2023-06-10T18:57:45.437768+0000","last_peered":"2023-06-10T18:57:45.437768+0000","last_clean":"2023-06-10T18:57:45.437768+0000","last_became_active":"2023-06-10T18:57:43.874069+0000","last_became_peered":"2023-06-10T18:57:43.874069+0000","last_unstale":"2023-06-10T18:57:45.437768+0000","last_undegraded":"2023-06-10T18:57:45.437768+0000","last_fullsized":"2023-06-10T18:57:45.437768+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:15.212015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268996+0000","last_change":"2023-06-10T18:57:44.278777+0000","last_active":"2023-06-10T18:57:45.268996+0000","last_peered":"2023-06-10T18:57:45.268996+0000","last_clean":"2023-06-10T18:57:45.268996+0000","last_became_active":"2023-06-10T18:57:44.278009+0000","last_became_peered":"2023-06-10T18:57:44.278009+0000","last_unstale":"2023-06-10T18:57:45.268996+0000","last_undegraded":"2023-06-10T18:57:45.268996+0000","last_fullsized":"2023-06-10T18:57:45.268996+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:09:51.453377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270579+0000","last_change":"2023-06-10T18:57:43.878033+0000","last_active":"2023-06-10T18:57:45.270579+0000","last_peered":"2023-06-10T18:57:45.270579+0000","last_clean":"2023-06-10T18:57:45.270579+0000","last_became_active":"2023-06-10T18:57:43.877065+0000","last_became_peered":"2023-06-10T18:57:43.877065+0000","last_unstale":"2023-06-10T18:57:45.270579+0000","last_undegraded":"2023-06-10T18:57:45.270579+0000","last_fullsized":"2023-06-10T18:57:45.270579+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:15.451498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436949+0000","last_change":"2023-06-10T18:57:44.275648+0000","last_active":"2023-06-10T18:57:45.436949+0000","last_peered":"2023-06-10T18:57:45.436949+0000","last_clean":"2023-06-10T18:57:45.436949+0000","last_became_active":"2023-06-10T18:57:44.275470+0000","last_became_peered":"2023-06-10T18:57:44.275470+0000","last_unstale":"2023-06-10T18:57:45.436949+0000","last_undegraded":"2023-06-10T18:57:45.436949+0000","last_fullsized":"2023-06-10T18:57:45.436949+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:01.272573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437048+0000","last_change":"2023-06-10T18:57:43.874147+0000","last_active":"2023-06-10T18:57:45.437048+0000","last_peered":"2023-06-10T18:57:45.437048+0000","last_clean":"2023-06-10T18:57:45.437048+0000","last_became_active":"2023-06-10T18:57:43.873866+0000","last_became_peered":"2023-06-10T18:57:43.873866+0000","last_unstale":"2023-06-10T18:57:45.437048+0000","last_undegraded":"2023-06-10T18:57:45.437048+0000","last_fullsized":"2023-06-10T18:57:45.437048+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:44.829758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438249+0000","last_change":"2023-06-10T18:57:44.284456+0000","last_active":"2023-06-10T18:57:45.438249+0000","last_peered":"2023-06-10T18:57:45.438249+0000","last_clean":"2023-06-10T18:57:45.438249+0000","last_became_active":"2023-06-10T18:57:44.284195+0000","last_became_peered":"2023-06-10T18:57:44.284195+0000","last_unstale":"2023-06-10T18:57:45.438249+0000","last_undegraded":"2023-06-10T18:57:45.438249+0000","last_fullsized":"2023-06-10T18:57:45.438249+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:11.605609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437131+0000","last_change":"2023-06-10T18:57:43.880827+0000","last_active":"2023-06-10T18:57:45.437131+0000","last_peered":"2023-06-10T18:57:45.437131+0000","last_clean":"2023-06-10T18:57:45.437131+0000","last_became_active":"2023-06-10T18:57:43.880424+0000","last_became_peered":"2023-06-10T18:57:43.880424+0000","last_unstale":"2023-06-10T18:57:45.437131+0000","last_undegraded":"2023-06-10T18:57:45.437131+0000","last_fullsized":"2023-06-10T18:57:45.437131+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:47:08.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436983+0000","last_change":"2023-06-10T18:57:44.282545+0000","last_active":"2023-06-10T18:57:45.436983+0000","last_peered":"2023-06-10T18:57:45.436983+0000","last_clean":"2023-06-10T18:57:45.436983+0000","last_became_active":"2023-06-10T18:57:44.282263+0000","last_became_peered":"2023-06-10T18:57:44.282263+0000","last_unstale":"2023-06-10T18:57:45.436983+0000","last_undegraded":"2023-06-10T18:57:45.436983+0000","last_fullsized":"2023-06-10T18:57:45.436983+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:16.111459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268635+0000","last_change":"2023-06-10T18:57:43.877331+0000","last_active":"2023-06-10T18:57:45.268635+0000","last_peered":"2023-06-10T18:57:45.268635+0000","last_clean":"2023-06-10T18:57:45.268635+0000","last_became_active":"2023-06-10T18:57:43.877030+0000","last_became_peered":"2023-06-10T18:57:43.877030+0000","last_unstale":"2023-06-10T18:57:45.268635+0000","last_undegraded":"2023-06-10T18:57:45.268635+0000","last_fullsized":"2023-06-10T18:57:45.268635+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:30.896540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438023+0000","last_change":"2023-06-10T18:57:44.281465+0000","last_active":"2023-06-10T18:57:45.438023+0000","last_peered":"2023-06-10T18:57:45.438023+0000","last_clean":"2023-06-10T18:57:45.438023+0000","last_became_active":"2023-06-10T18:57:44.281150+0000","last_became_peered":"2023-06-10T18:57:44.281150+0000","last_unstale":"2023-06-10T18:57:45.438023+0000","last_undegraded":"2023-06-10T18:57:45.438023+0000","last_fullsized":"2023-06-10T18:57:45.438023+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:46:12.512583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270607+0000","last_change":"2023-06-10T18:57:43.877882+0000","last_active":"2023-06-10T18:57:45.270607+0000","last_peered":"2023-06-10T18:57:45.270607+0000","last_clean":"2023-06-10T18:57:45.270607+0000","last_became_active":"2023-06-10T18:57:43.877609+0000","last_became_peered":"2023-06-10T18:57:43.877609+0000","last_unstale":"2023-06-10T18:57:45.270607+0000","last_undegraded":"2023-06-10T18:57:45.270607+0000","last_fullsized":"2023-06-10T18:57:45.270607+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:40.135041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437510+0000","last_change":"2023-06-10T18:57:44.286368+0000","last_active":"2023-06-10T18:57:45.437510+0000","last_peered":"2023-06-10T18:57:45.437510+0000","last_clean":"2023-06-10T18:57:45.437510+0000","last_became_active":"2023-06-10T18:57:44.279534+0000","last_became_peered":"2023-06-10T18:57:44.279534+0000","last_unstale":"2023-06-10T18:57:45.437510+0000","last_undegraded":"2023-06-10T18:57:45.437510+0000","last_fullsized":"2023-06-10T18:57:45.437510+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:26:46.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270508+0000","last_change":"2023-06-10T18:57:43.877598+0000","last_active":"2023-06-10T18:57:45.270508+0000","last_peered":"2023-06-10T18:57:45.270508+0000","last_clean":"2023-06-10T18:57:45.270508+0000","last_became_active":"2023-06-10T18:57:43.877095+0000","last_became_peered":"2023-06-10T18:57:43.877095+0000","last_unstale":"2023-06-10T18:57:45.270508+0000","last_undegraded":"2023-06-10T18:57:45.270508+0000","last_fullsized":"2023-06-10T18:57:45.270508+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:10.600034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437700+0000","last_change":"2023-06-10T18:57:44.275306+0000","last_active":"2023-06-10T18:57:45.437700+0000","last_peered":"2023-06-10T18:57:45.437700+0000","last_clean":"2023-06-10T18:57:45.437700+0000","last_became_active":"2023-06-10T18:57:44.275037+0000","last_became_peered":"2023-06-10T18:57:44.275037+0000","last_unstale":"2023-06-10T18:57:45.437700+0000","last_undegraded":"2023-06-10T18:57:45.437700+0000","last_fullsized":"2023-06-10T18:57:45.437700+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:30.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437896+0000","last_change":"2023-06-10T18:57:43.873255+0000","last_active":"2023-06-10T18:57:45.437896+0000","last_peered":"2023-06-10T18:57:45.437896+0000","last_clean":"2023-06-10T18:57:45.437896+0000","last_became_active":"2023-06-10T18:57:43.873015+0000","last_became_peered":"2023-06-10T18:57:43.873015+0000","last_unstale":"2023-06-10T18:57:45.437896+0000","last_undegraded":"2023-06-10T18:57:45.437896+0000","last_fullsized":"2023-06-10T18:57:45.437896+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:44:16.432344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268864+0000","last_change":"2023-06-10T18:57:44.277872+0000","last_active":"2023-06-10T18:57:45.268864+0000","last_peered":"2023-06-10T18:57:45.268864+0000","last_clean":"2023-06-10T18:57:45.268864+0000","last_became_active":"2023-06-10T18:57:44.277571+0000","last_became_peered":"2023-06-10T18:57:44.277571+0000","last_unstale":"2023-06-10T18:57:45.268864+0000","last_undegraded":"2023-06-10T18:57:45.268864+0000","last_fullsized":"2023-06-10T18:57:45.268864+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:59:00.994547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437718+0000","last_change":"2023-06-10T18:57:43.874744+0000","last_active":"2023-06-10T18:57:45.437718+0000","last_peered":"2023-06-10T18:57:45.437718+0000","last_clean":"2023-06-10T18:57:45.437718+0000","last_became_active":"2023-06-10T18:57:43.873677+0000","last_became_peered":"2023-06-10T18:57:43.873677+0000","last_unstale":"2023-06-10T18:57:45.437718+0000","last_undegraded":"2023-06-10T18:57:45.437718+0000","last_fullsized":"2023-06-10T18:57:45.437718+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:50:10.187968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270192+0000","last_change":"2023-06-10T18:57:44.290334+0000","last_active":"2023-06-10T18:57:45.270192+0000","last_peered":"2023-06-10T18:57:45.270192+0000","last_clean":"2023-06-10T18:57:45.270192+0000","last_became_active":"2023-06-10T18:57:44.290072+0000","last_became_peered":"2023-06-10T18:57:44.290072+0000","last_unstale":"2023-06-10T18:57:45.270192+0000","last_undegraded":"2023-06-10T18:57:45.270192+0000","last_fullsized":"2023-06-10T18:57:45.270192+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:35.412940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437213+0000","last_change":"2023-06-10T18:57:43.877921+0000","last_active":"2023-06-10T18:57:45.437213+0000","last_peered":"2023-06-10T18:57:45.437213+0000","last_clean":"2023-06-10T18:57:45.437213+0000","last_became_active":"2023-06-10T18:57:43.877688+0000","last_became_peered":"2023-06-10T18:57:43.877688+0000","last_unstale":"2023-06-10T18:57:45.437213+0000","last_undegraded":"2023-06-10T18:57:45.437213+0000","last_fullsized":"2023-06-10T18:57:45.437213+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:40.135041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268915+0000","last_change":"2023-06-10T18:57:44.284304+0000","last_active":"2023-06-10T18:57:45.268915+0000","last_peered":"2023-06-10T18:57:45.268915+0000","last_clean":"2023-06-10T18:57:45.268915+0000","last_became_active":"2023-06-10T18:57:44.284057+0000","last_became_peered":"2023-06-10T18:57:44.284057+0000","last_unstale":"2023-06-10T18:57:45.268915+0000","last_undegraded":"2023-06-10T18:57:45.268915+0000","last_fullsized":"2023-06-10T18:57:45.268915+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:12:40.958894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437280+0000","last_change":"2023-06-10T18:57:43.880681+0000","last_active":"2023-06-10T18:57:45.437280+0000","last_peered":"2023-06-10T18:57:45.437280+0000","last_clean":"2023-06-10T18:57:45.437280+0000","last_became_active":"2023-06-10T18:57:43.880382+0000","last_became_peered":"2023-06-10T18:57:43.880382+0000","last_unstale":"2023-06-10T18:57:45.437280+0000","last_undegraded":"2023-06-10T18:57:45.437280+0000","last_fullsized":"2023-06-10T18:57:45.437280+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:43:40.554724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436815+0000","last_change":"2023-06-10T18:57:44.287502+0000","last_active":"2023-06-10T18:57:45.436815+0000","last_peered":"2023-06-10T18:57:45.436815+0000","last_clean":"2023-06-10T18:57:45.436815+0000","last_became_active":"2023-06-10T18:57:44.287085+0000","last_became_peered":"2023-06-10T18:57:44.287085+0000","last_unstale":"2023-06-10T18:57:45.436815+0000","last_undegraded":"2023-06-10T18:57:45.436815+0000","last_fullsized":"2023-06-10T18:57:45.436815+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:54.450021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438311+0000","last_change":"2023-06-10T18:57:43.878884+0000","last_active":"2023-06-10T18:57:45.438311+0000","last_peered":"2023-06-10T18:57:45.438311+0000","last_clean":"2023-06-10T18:57:45.438311+0000","last_became_active":"2023-06-10T18:57:43.878589+0000","last_became_peered":"2023-06-10T18:57:43.878589+0000","last_unstale":"2023-06-10T18:57:45.438311+0000","last_undegraded":"2023-06-10T18:57:45.438311+0000","last_fullsized":"2023-06-10T18:57:45.438311+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:31:39.394688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437078+0000","last_change":"2023-06-10T18:57:44.286048+0000","last_active":"2023-06-10T18:57:45.437078+0000","last_peered":"2023-06-10T18:57:45.437078+0000","last_clean":"2023-06-10T18:57:45.437078+0000","last_became_active":"2023-06-10T18:57:44.285488+0000","last_became_peered":"2023-06-10T18:57:44.285488+0000","last_unstale":"2023-06-10T18:57:45.437078+0000","last_undegraded":"2023-06-10T18:57:45.437078+0000","last_fullsized":"2023-06-10T18:57:45.437078+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:40:35.228387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436992+0000","last_change":"2023-06-10T18:57:43.874293+0000","last_active":"2023-06-10T18:57:45.436992+0000","last_peered":"2023-06-10T18:57:45.436992+0000","last_clean":"2023-06-10T18:57:45.436992+0000","last_became_active":"2023-06-10T18:57:43.873983+0000","last_became_peered":"2023-06-10T18:57:43.873983+0000","last_unstale":"2023-06-10T18:57:45.436992+0000","last_undegraded":"2023-06-10T18:57:45.436992+0000","last_fullsized":"2023-06-10T18:57:45.436992+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:16:54.682898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438225+0000","last_change":"2023-06-10T18:57:44.285624+0000","last_active":"2023-06-10T18:57:45.438225+0000","last_peered":"2023-06-10T18:57:45.438225+0000","last_clean":"2023-06-10T18:57:45.438225+0000","last_became_active":"2023-06-10T18:57:44.284489+0000","last_became_peered":"2023-06-10T18:57:44.284489+0000","last_unstale":"2023-06-10T18:57:45.438225+0000","last_undegraded":"2023-06-10T18:57:45.438225+0000","last_fullsized":"2023-06-10T18:57:45.438225+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:19.490820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437774+0000","last_change":"2023-06-10T18:57:43.878054+0000","last_active":"2023-06-10T18:57:45.437774+0000","last_peered":"2023-06-10T18:57:45.437774+0000","last_clean":"2023-06-10T18:57:45.437774+0000","last_became_active":"2023-06-10T18:57:43.877764+0000","last_became_peered":"2023-06-10T18:57:43.877764+0000","last_unstale":"2023-06-10T18:57:45.437774+0000","last_undegraded":"2023-06-10T18:57:45.437774+0000","last_fullsized":"2023-06-10T18:57:45.437774+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:43.195522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269085+0000","last_change":"2023-06-10T18:57:44.283811+0000","last_active":"2023-06-10T18:57:45.269085+0000","last_peered":"2023-06-10T18:57:45.269085+0000","last_clean":"2023-06-10T18:57:45.269085+0000","last_became_active":"2023-06-10T18:57:44.283573+0000","last_became_peered":"2023-06-10T18:57:44.283573+0000","last_unstale":"2023-06-10T18:57:45.269085+0000","last_undegraded":"2023-06-10T18:57:45.269085+0000","last_fullsized":"2023-06-10T18:57:45.269085+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:29:27.097073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270802+0000","last_change":"2023-06-10T18:57:43.877466+0000","last_active":"2023-06-10T18:57:45.270802+0000","last_peered":"2023-06-10T18:57:45.270802+0000","last_clean":"2023-06-10T18:57:45.270802+0000","last_became_active":"2023-06-10T18:57:43.877147+0000","last_became_peered":"2023-06-10T18:57:43.877147+0000","last_unstale":"2023-06-10T18:57:45.270802+0000","last_undegraded":"2023-06-10T18:57:45.270802+0000","last_fullsized":"2023-06-10T18:57:45.270802+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:09.599312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437140+0000","last_change":"2023-06-10T18:57:44.281630+0000","last_active":"2023-06-10T18:57:45.437140+0000","last_peered":"2023-06-10T18:57:45.437140+0000","last_clean":"2023-06-10T18:57:45.437140+0000","last_became_active":"2023-06-10T18:57:44.281355+0000","last_became_peered":"2023-06-10T18:57:44.281355+0000","last_unstale":"2023-06-10T18:57:45.437140+0000","last_undegraded":"2023-06-10T18:57:45.437140+0000","last_fullsized":"2023-06-10T18:57:45.437140+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:10.647884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437442+0000","last_change":"2023-06-10T18:57:43.879476+0000","last_active":"2023-06-10T18:57:45.437442+0000","last_peered":"2023-06-10T18:57:45.437442+0000","last_clean":"2023-06-10T18:57:45.437442+0000","last_became_active":"2023-06-10T18:57:43.879101+0000","last_became_peered":"2023-06-10T18:57:43.879101+0000","last_unstale":"2023-06-10T18:57:45.437442+0000","last_undegraded":"2023-06-10T18:57:45.437442+0000","last_fullsized":"2023-06-10T18:57:45.437442+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:46:10.642926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437324+0000","last_change":"2023-06-10T18:57:44.285168+0000","last_active":"2023-06-10T18:57:45.437324+0000","last_peered":"2023-06-10T18:57:45.437324+0000","last_clean":"2023-06-10T18:57:45.437324+0000","last_became_active":"2023-06-10T18:57:44.284899+0000","last_became_peered":"2023-06-10T18:57:44.284899+0000","last_unstale":"2023-06-10T18:57:45.437324+0000","last_undegraded":"2023-06-10T18:57:45.437324+0000","last_fullsized":"2023-06-10T18:57:45.437324+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:56.852581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438148+0000","last_change":"2023-06-10T18:57:43.875119+0000","last_active":"2023-06-10T18:57:45.438148+0000","last_peered":"2023-06-10T18:57:45.438148+0000","last_clean":"2023-06-10T18:57:45.438148+0000","last_became_active":"2023-06-10T18:57:43.874740+0000","last_became_peered":"2023-06-10T18:57:43.874740+0000","last_unstale":"2023-06-10T18:57:45.438148+0000","last_undegraded":"2023-06-10T18:57:45.438148+0000","last_fullsized":"2023-06-10T18:57:45.438148+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:06.322495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437799+0000","last_change":"2023-06-10T18:57:44.286143+0000","last_active":"2023-06-10T18:57:45.437799+0000","last_peered":"2023-06-10T18:57:45.437799+0000","last_clean":"2023-06-10T18:57:45.437799+0000","last_became_active":"2023-06-10T18:57:44.285996+0000","last_became_peered":"2023-06-10T18:57:44.285996+0000","last_unstale":"2023-06-10T18:57:45.437799+0000","last_undegraded":"2023-06-10T18:57:45.437799+0000","last_fullsized":"2023-06-10T18:57:45.437799+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:48:57.535434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269501+0000","last_change":"2023-06-10T18:57:43.873254+0000","last_active":"2023-06-10T18:57:45.269501+0000","last_peered":"2023-06-10T18:57:45.269501+0000","last_clean":"2023-06-10T18:57:45.269501+0000","last_became_active":"2023-06-10T18:57:43.872988+0000","last_became_peered":"2023-06-10T18:57:43.872988+0000","last_unstale":"2023-06-10T18:57:45.269501+0000","last_undegraded":"2023-06-10T18:57:45.269501+0000","last_fullsized":"2023-06-10T18:57:45.269501+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:16:47.882047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269497+0000","last_change":"2023-06-10T18:57:44.290520+0000","last_active":"2023-06-10T18:57:45.269497+0000","last_peered":"2023-06-10T18:57:45.269497+0000","last_clean":"2023-06-10T18:57:45.269497+0000","last_became_active":"2023-06-10T18:57:44.286486+0000","last_became_peered":"2023-06-10T18:57:44.286486+0000","last_unstale":"2023-06-10T18:57:45.269497+0000","last_undegraded":"2023-06-10T18:57:45.269497+0000","last_fullsized":"2023-06-10T18:57:45.269497+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:17:36.326210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437176+0000","last_change":"2023-06-10T18:57:43.875756+0000","last_active":"2023-06-10T18:57:45.437176+0000","last_peered":"2023-06-10T18:57:45.437176+0000","last_clean":"2023-06-10T18:57:45.437176+0000","last_became_active":"2023-06-10T18:57:43.875490+0000","last_became_peered":"2023-06-10T18:57:43.875490+0000","last_unstale":"2023-06-10T18:57:45.437176+0000","last_undegraded":"2023-06-10T18:57:45.437176+0000","last_fullsized":"2023-06-10T18:57:45.437176+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:15.451498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268521+0000","last_change":"2023-06-10T18:57:44.283326+0000","last_active":"2023-06-10T18:57:45.268521+0000","last_peered":"2023-06-10T18:57:45.268521+0000","last_clean":"2023-06-10T18:57:45.268521+0000","last_became_active":"2023-06-10T18:57:44.283060+0000","last_became_peered":"2023-06-10T18:57:44.283060+0000","last_unstale":"2023-06-10T18:57:45.268521+0000","last_undegraded":"2023-06-10T18:57:45.268521+0000","last_fullsized":"2023-06-10T18:57:45.268521+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:46:49.078616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437276+0000","last_change":"2023-06-10T18:57:43.879111+0000","last_active":"2023-06-10T18:57:45.437276+0000","last_peered":"2023-06-10T18:57:45.437276+0000","last_clean":"2023-06-10T18:57:45.437276+0000","last_became_active":"2023-06-10T18:57:43.878821+0000","last_became_peered":"2023-06-10T18:57:43.878821+0000","last_unstale":"2023-06-10T18:57:45.437276+0000","last_undegraded":"2023-06-10T18:57:45.437276+0000","last_fullsized":"2023-06-10T18:57:45.437276+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:22:51.743471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270347+0000","last_change":"2023-06-10T18:57:44.284933+0000","last_active":"2023-06-10T18:57:45.270347+0000","last_peered":"2023-06-10T18:57:45.270347+0000","last_clean":"2023-06-10T18:57:45.270347+0000","last_became_active":"2023-06-10T18:57:44.284659+0000","last_became_peered":"2023-06-10T18:57:44.284659+0000","last_unstale":"2023-06-10T18:57:45.270347+0000","last_undegraded":"2023-06-10T18:57:45.270347+0000","last_fullsized":"2023-06-10T18:57:45.270347+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:05.333291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269303+0000","last_change":"2023-06-10T18:57:43.872121+0000","last_active":"2023-06-10T18:57:45.269303+0000","last_peered":"2023-06-10T18:57:45.269303+0000","last_clean":"2023-06-10T18:57:45.269303+0000","last_became_active":"2023-06-10T18:57:43.871705+0000","last_became_peered":"2023-06-10T18:57:43.871705+0000","last_unstale":"2023-06-10T18:57:45.269303+0000","last_undegraded":"2023-06-10T18:57:45.269303+0000","last_fullsized":"2023-06-10T18:57:45.269303+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:15.440292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270022+0000","last_change":"2023-06-10T18:57:44.285387+0000","last_active":"2023-06-10T18:57:45.270022+0000","last_peered":"2023-06-10T18:57:45.270022+0000","last_clean":"2023-06-10T18:57:45.270022+0000","last_became_active":"2023-06-10T18:57:44.285125+0000","last_became_peered":"2023-06-10T18:57:44.285125+0000","last_unstale":"2023-06-10T18:57:45.270022+0000","last_undegraded":"2023-06-10T18:57:45.270022+0000","last_fullsized":"2023-06-10T18:57:45.270022+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:20:46.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"158'8","reported_seq":21,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269179+0000","last_change":"2023-06-10T18:57:43.869514+0000","last_active":"2023-06-10T18:57:45.269179+0000","last_peered":"2023-06-10T18:57:45.269179+0000","last_clean":"2023-06-10T18:57:45.269179+0000","last_became_active":"2023-06-10T18:57:43.869224+0000","last_became_peered":"2023-06-10T18:57:43.869224+0000","last_unstale":"2023-06-10T18:57:45.269179+0000","last_undegraded":"2023-06-10T18:57:45.269179+0000","last_fullsized":"2023-06-10T18:57:45.269179+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:40.645828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437779+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:57:45.437779+0000","last_peered":"2023-06-10T18:57:45.437779+0000","last_clean":"2023-06-10T18:57:45.437779+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:57:45.437779+0000","last_undegraded":"2023-06-10T18:57:45.437779+0000","last_fullsized":"2023-06-10T18:57:45.437779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:49:51.818379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437640+0000","last_change":"2023-06-10T18:57:44.282760+0000","last_active":"2023-06-10T18:57:45.437640+0000","last_peered":"2023-06-10T18:57:45.437640+0000","last_clean":"2023-06-10T18:57:45.437640+0000","last_became_active":"2023-06-10T18:57:44.279860+0000","last_became_peered":"2023-06-10T18:57:44.279860+0000","last_unstale":"2023-06-10T18:57:45.437640+0000","last_undegraded":"2023-06-10T18:57:45.437640+0000","last_fullsized":"2023-06-10T18:57:45.437640+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:20:35.204715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437485+0000","last_change":"2023-06-10T18:57:43.881757+0000","last_active":"2023-06-10T18:57:45.437485+0000","last_peered":"2023-06-10T18:57:45.437485+0000","last_clean":"2023-06-10T18:57:45.437485+0000","last_became_active":"2023-06-10T18:57:43.881510+0000","last_became_peered":"2023-06-10T18:57:43.881510+0000","last_unstale":"2023-06-10T18:57:45.437485+0000","last_undegraded":"2023-06-10T18:57:45.437485+0000","last_fullsized":"2023-06-10T18:57:45.437485+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:05:06.543012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":289,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270159+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:57:45.270159+0000","last_peered":"2023-06-10T18:57:45.270159+0000","last_clean":"2023-06-10T18:57:45.270159+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:57:45.270159+0000","last_undegraded":"2023-06-10T18:57:45.270159+0000","last_fullsized":"2023-06-10T18:57:45.270159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:27:04.959211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438089+0000","last_change":"2023-06-10T18:57:44.280028+0000","last_active":"2023-06-10T18:57:45.438089+0000","last_peered":"2023-06-10T18:57:45.438089+0000","last_clean":"2023-06-10T18:57:45.438089+0000","last_became_active":"2023-06-10T18:57:44.279780+0000","last_became_peered":"2023-06-10T18:57:44.279780+0000","last_unstale":"2023-06-10T18:57:45.438089+0000","last_undegraded":"2023-06-10T18:57:45.438089+0000","last_fullsized":"2023-06-10T18:57:45.438089+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:00:00.235679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"158'3","reported_seq":16,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437126+0000","last_change":"2023-06-10T18:57:43.877140+0000","last_active":"2023-06-10T18:57:45.437126+0000","last_peered":"2023-06-10T18:57:45.437126+0000","last_clean":"2023-06-10T18:57:45.437126+0000","last_became_active":"2023-06-10T18:57:43.876617+0000","last_became_peered":"2023-06-10T18:57:43.876617+0000","last_unstale":"2023-06-10T18:57:45.437126+0000","last_undegraded":"2023-06-10T18:57:45.437126+0000","last_fullsized":"2023-06-10T18:57:45.437126+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:11.889919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437061+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:57:45.437061+0000","last_peered":"2023-06-10T18:57:45.437061+0000","last_clean":"2023-06-10T18:57:45.437061+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:57:45.437061+0000","last_undegraded":"2023-06-10T18:57:45.437061+0000","last_fullsized":"2023-06-10T18:57:45.437061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:43:00.582909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438230+0000","last_change":"2023-06-10T18:57:44.285385+0000","last_active":"2023-06-10T18:57:45.438230+0000","last_peered":"2023-06-10T18:57:45.438230+0000","last_clean":"2023-06-10T18:57:45.438230+0000","last_became_active":"2023-06-10T18:57:44.284732+0000","last_became_peered":"2023-06-10T18:57:44.284732+0000","last_unstale":"2023-06-10T18:57:45.438230+0000","last_undegraded":"2023-06-10T18:57:45.438230+0000","last_fullsized":"2023-06-10T18:57:45.438230+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:08:44.539633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270411+0000","last_change":"2023-06-10T18:57:43.874498+0000","last_active":"2023-06-10T18:57:45.270411+0000","last_peered":"2023-06-10T18:57:45.270411+0000","last_clean":"2023-06-10T18:57:45.270411+0000","last_became_active":"2023-06-10T18:57:43.874220+0000","last_became_peered":"2023-06-10T18:57:43.874220+0000","last_unstale":"2023-06-10T18:57:45.270411+0000","last_undegraded":"2023-06-10T18:57:45.270411+0000","last_fullsized":"2023-06-10T18:57:45.270411+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:42.190892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270742+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:57:45.270742+0000","last_peered":"2023-06-10T18:57:45.270742+0000","last_clean":"2023-06-10T18:57:45.270742+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:57:45.270742+0000","last_undegraded":"2023-06-10T18:57:45.270742+0000","last_fullsized":"2023-06-10T18:57:45.270742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:07:30.249993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":419,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437089+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:57:45.437089+0000","last_peered":"2023-06-10T18:57:45.437089+0000","last_clean":"2023-06-10T18:57:45.437089+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:57:45.437089+0000","last_undegraded":"2023-06-10T18:57:45.437089+0000","last_fullsized":"2023-06-10T18:57:45.437089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:58.704100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270025+0000","last_change":"2023-06-10T18:57:44.294117+0000","last_active":"2023-06-10T18:57:45.270025+0000","last_peered":"2023-06-10T18:57:45.270025+0000","last_clean":"2023-06-10T18:57:45.270025+0000","last_became_active":"2023-06-10T18:57:44.293811+0000","last_became_peered":"2023-06-10T18:57:44.293811+0000","last_unstale":"2023-06-10T18:57:45.270025+0000","last_undegraded":"2023-06-10T18:57:45.270025+0000","last_fullsized":"2023-06-10T18:57:45.270025+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:45:58.571683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"158'6","reported_seq":19,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269698+0000","last_change":"2023-06-10T18:57:43.870392+0000","last_active":"2023-06-10T18:57:45.269698+0000","last_peered":"2023-06-10T18:57:45.269698+0000","last_clean":"2023-06-10T18:57:45.269698+0000","last_became_active":"2023-06-10T18:57:43.870112+0000","last_became_peered":"2023-06-10T18:57:43.870112+0000","last_unstale":"2023-06-10T18:57:45.269698+0000","last_undegraded":"2023-06-10T18:57:45.269698+0000","last_fullsized":"2023-06-10T18:57:45.269698+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:56.601006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":263,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436995+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:57:45.436995+0000","last_peered":"2023-06-10T18:57:45.436995+0000","last_clean":"2023-06-10T18:57:45.436995+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:57:45.436995+0000","last_undegraded":"2023-06-10T18:57:45.436995+0000","last_fullsized":"2023-06-10T18:57:45.436995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:22:54.945540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437252+0000","last_change":"2023-06-10T18:57:44.280226+0000","last_active":"2023-06-10T18:57:45.437252+0000","last_peered":"2023-06-10T18:57:45.437252+0000","last_clean":"2023-06-10T18:57:45.437252+0000","last_became_active":"2023-06-10T18:57:44.279997+0000","last_became_peered":"2023-06-10T18:57:44.279997+0000","last_unstale":"2023-06-10T18:57:45.437252+0000","last_undegraded":"2023-06-10T18:57:45.437252+0000","last_fullsized":"2023-06-10T18:57:45.437252+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:25.425413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437791+0000","last_change":"2023-06-10T18:57:43.878016+0000","last_active":"2023-06-10T18:57:45.437791+0000","last_peered":"2023-06-10T18:57:45.437791+0000","last_clean":"2023-06-10T18:57:45.437791+0000","last_became_active":"2023-06-10T18:57:43.877732+0000","last_became_peered":"2023-06-10T18:57:43.877732+0000","last_unstale":"2023-06-10T18:57:45.437791+0000","last_undegraded":"2023-06-10T18:57:45.437791+0000","last_fullsized":"2023-06-10T18:57:45.437791+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:01.625663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437944+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:57:45.437944+0000","last_peered":"2023-06-10T18:57:45.437944+0000","last_clean":"2023-06-10T18:57:45.437944+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:57:45.437944+0000","last_undegraded":"2023-06-10T18:57:45.437944+0000","last_fullsized":"2023-06-10T18:57:45.437944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:16:26.239474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269056+0000","last_change":"2023-06-10T18:57:44.286854+0000","last_active":"2023-06-10T18:57:45.269056+0000","last_peered":"2023-06-10T18:57:45.269056+0000","last_clean":"2023-06-10T18:57:45.269056+0000","last_became_active":"2023-06-10T18:57:44.286553+0000","last_became_peered":"2023-06-10T18:57:44.286553+0000","last_unstale":"2023-06-10T18:57:45.269056+0000","last_undegraded":"2023-06-10T18:57:45.269056+0000","last_fullsized":"2023-06-10T18:57:45.269056+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:38:24.475492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437907+0000","last_change":"2023-06-10T18:57:43.877173+0000","last_active":"2023-06-10T18:57:45.437907+0000","last_peered":"2023-06-10T18:57:45.437907+0000","last_clean":"2023-06-10T18:57:45.437907+0000","last_became_active":"2023-06-10T18:57:43.876840+0000","last_became_peered":"2023-06-10T18:57:43.876840+0000","last_unstale":"2023-06-10T18:57:45.437907+0000","last_undegraded":"2023-06-10T18:57:45.437907+0000","last_fullsized":"2023-06-10T18:57:45.437907+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:46:00.963751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438098+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:57:45.438098+0000","last_peered":"2023-06-10T18:57:45.438098+0000","last_clean":"2023-06-10T18:57:45.438098+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:57:45.438098+0000","last_undegraded":"2023-06-10T18:57:45.438098+0000","last_fullsized":"2023-06-10T18:57:45.438098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:07:36.878181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437449+0000","last_change":"2023-06-10T18:57:44.287278+0000","last_active":"2023-06-10T18:57:45.437449+0000","last_peered":"2023-06-10T18:57:45.437449+0000","last_clean":"2023-06-10T18:57:45.437449+0000","last_became_active":"2023-06-10T18:57:44.286171+0000","last_became_peered":"2023-06-10T18:57:44.286171+0000","last_unstale":"2023-06-10T18:57:45.437449+0000","last_undegraded":"2023-06-10T18:57:45.437449+0000","last_fullsized":"2023-06-10T18:57:45.437449+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:06:56.158866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437077+0000","last_change":"2023-06-10T18:57:43.874180+0000","last_active":"2023-06-10T18:57:45.437077+0000","last_peered":"2023-06-10T18:57:45.437077+0000","last_clean":"2023-06-10T18:57:45.437077+0000","last_became_active":"2023-06-10T18:57:43.873903+0000","last_became_peered":"2023-06-10T18:57:43.873903+0000","last_unstale":"2023-06-10T18:57:45.437077+0000","last_undegraded":"2023-06-10T18:57:45.437077+0000","last_fullsized":"2023-06-10T18:57:45.437077+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:00:20.227568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437540+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:57:45.437540+0000","last_peered":"2023-06-10T18:57:45.437540+0000","last_clean":"2023-06-10T18:57:45.437540+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:57:45.437540+0000","last_undegraded":"2023-06-10T18:57:45.437540+0000","last_fullsized":"2023-06-10T18:57:45.437540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:07.496989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":2548,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":2609152,"data_stored":1570117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.81499999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.442}]},{"osd":1,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.752,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.503,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76300000000000001,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.495}]},{"osd":2,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.39000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.69399999999999995,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.79000000000000004},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.52600000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.89700000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45600000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":0.77900000000000003,"15min":0.83799999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.34599999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75900000000000001,"5min":0.80200000000000005,"15min":0.93899999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.56100000000000005,"15min":0.55300000000000005},"min":{"1min":0.42099999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.45900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1990656,"data_stored":976446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":0.51500000000000001},"min":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.628},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.29199999999999998},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.872,"15min":6.6689999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.252},"last":0.495}]},{"osd":3,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.373},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53600000000000003,"15min":0.54500000000000004},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.499,"15min":0.495},"min":{"1min":0.33300000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64300000000000002,"5min":0.874,"15min":0.874},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71999999999999997,"5min":0.746,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55200000000000005,"15min":0.53400000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67700000000000005,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1990656,"data_stored":976458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.308,"15min":0.32700000000000001},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.46000000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34399999999999997,"15min":0.34899999999999998},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.498,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.33500000000000002,"15min":0.34300000000000003},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.48299999999999998,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.35199999999999998,"15min":0.375},"min":{"1min":0.24099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51600000000000001,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.318}]},{"osd":3,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.379,"15min":0.39700000000000002},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54800000000000004,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.378,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53200000000000003,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.34899999999999998},"max":{"1min":0.871,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.372,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.46400000000000002,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.008},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.83399999999999996,"5min":1.206,"15min":1.206},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.873,"15min":0.88700000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45900000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":1.224,"15min":1.224},"last":0.499},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.61399999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.45100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33568,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338272,"statfs":{"total":96636764160,"available":96602390528,"internally_reserved":0,"allocated":2580480,"data_stored":1566792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.20300000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.499,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.309},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.35199999999999998},"min":{"1min":0.24099999999999999,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.55300000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.435,"15min":0.39200000000000002},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.42499999999999999,"5min":11.439,"15min":11.439},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48099999999999998,"15min":0.41999999999999998},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.58699999999999997,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.379},"min":{"1min":0.252,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47899999999999998,"5min":5.766,"15min":5.766},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.52300000000000002,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60599999999999998,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73399999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.71299999999999997,"5min":2.802,"15min":2.802},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81499999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78300000000000003,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.56299999999999994}]},{"osd":6,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82699999999999996,"5min":2.907,"15min":2.907},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.85499999999999998,"5min":0.97699999999999998,"15min":1.363},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.88400000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76000000000000001,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.52100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28900,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342940,"statfs":{"total":96636764160,"available":96607170560,"internally_reserved":0,"allocated":1994752,"data_stored":977533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34999999999999998,"15min":0.34300000000000003},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.65900000000000003,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.42199999999999999,"15min":0.38700000000000001},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.23699999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38500000000000001,"15min":0.376},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57599999999999996,"5min":2.847,"15min":2.847},"last":0.373},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42899999999999999,"15min":0.39900000000000002},"min":{"1min":0.253,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.85099999999999998,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.29099999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.43099999999999999,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.80200000000000005,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.312},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42399999999999999,"15min":0.41399999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.24,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63600000000000001,"15min":0.61599999999999999},"min":{"1min":0.40600000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.208,"5min":5.742,"15min":5.742},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.62},"min":{"1min":0.433,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.39500000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.68300000000000005,"15min":0.64600000000000002},"min":{"1min":0.38600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.86799999999999999,"5min":6.593,"15min":6.593},"last":0.433},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":1.002,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.443}]},{"osd":6,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.66900000000000004,"15min":0.63400000000000001},"min":{"1min":0.45100000000000001,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.171,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71099999999999997,"15min":0.66500000000000004},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.84099999999999997,"5min":6.617,"15min":6.617},"last":0.499},{"interface":"front","average":{"1min":0.622,"5min":0.70599999999999996,"15min":0.65800000000000003},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":0.97699999999999998,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.40600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575018,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1994752,"data_stored":977477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.34799999999999998},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.246,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35099999999999998,"15min":0.36699999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.50700000000000001,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.374},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.38,"15min":0.40200000000000002},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.68600000000000005,"15min":1.3},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.40100000000000002},"min":{"1min":0.29599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.41799999999999998,"15min":0.435},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.623,"5min":1.5960000000000001,"15min":1.599},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78700000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0720000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1100000000000001,"5min":1.613,"15min":1.613},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.095,"5min":1.4410000000000001,"15min":3.802},"last":0.74299999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.63800000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":1.018,"5min":1.679,"15min":1.679},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.625,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0549999999999999,"5min":1.47,"15min":1.47},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.628,"15min":0.628},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80900000000000005,"5min":1.548,"15min":1.8939999999999999},"last":0.623}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28908,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342932,"statfs":{"total":96636764160,"available":96607162368,"internally_reserved":0,"allocated":2002944,"data_stored":978058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":1.119},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.379,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.63,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.77500000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52400000000000002,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.77800000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.68899999999999995,"5min":0.873,"15min":0.96599999999999997},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.57899999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.75}]},{"osd":3,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.52500000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.68500000000000005,"5min":0.70699999999999996,"15min":0.79800000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70899999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.49099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.72099999999999997,"5min":0.94599999999999995,"15min":1.069},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.502,"15min":0.503},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72599999999999998,"5min":0.755,"15min":0.88300000000000001},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71099999999999997,"5min":0.90400000000000003,"15min":2.1579999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.70699999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76200000000000001,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.82399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28904,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342936,"statfs":{"total":96636764160,"available":96607166464,"internally_reserved":0,"allocated":1998848,"data_stored":978047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29199999999999998},"max":{"1min":0.72799999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.70799999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.622}]},{"osd":2,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.79700000000000004},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77600000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68700000000000006,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69499999999999995,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.51300000000000001},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.64800000000000002,"5min":0.75900000000000001,"15min":0.79200000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.879},"last":0.64900000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.54500000000000004},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.88400000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59599999999999997,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:57:48.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:57:48.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.515+0000 7f2239f45700 1 Processor -- start 2023-06-10T18:57:48.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.515+0000 7f2239f45700 1 -- start start 2023-06-10T18:57:48.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2239f45700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f22340ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2239f45700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2239f45700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340b6070 con 0x7f223415a420 2023-06-10T18:57:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2239f45700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340ced10 con 0x7f22340ce890 2023-06-10T18:57:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2239f45700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b1d70 con 0x7f22340b4ff0 2023-06-10T18:57:48.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2233fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f22340ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2233fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:55082/0 (socket says 172.21.15.104:55082) 2023-06-10T18:57:48.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2233fff700 1 -- 172.21.15.104:0/4190747688 learned_addr learned my addr 172.21.15.104:0/4190747688 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:48.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 -- 172.21.15.104:0/4190747688 >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 msgr2=0x7f223415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 --2- 172.21.15.104:0/4190747688 >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 -- 172.21.15.104:0/4190747688 >> v1:172.21.15.114:6789/0 conn(0x7f223415a420 legacy=0x7f22340b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 -- 172.21.15.104:0/4190747688 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340b5460 con 0x7f22340b4ff0 2023-06-10T18:57:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2233fff700 1 --2- 172.21.15.104:0/4190747688 >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:57:48.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.516+0000 7f2232ffd700 1 --2- 172.21.15.104:0/4190747688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f22340ce350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2228009840 tx=0x7f2228009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a03a39bc1c72ff7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f22327fc700 1 -- 172.21.15.104:0/4190747688 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228011830 con 0x7f22340b4ff0 2023-06-10T18:57:48.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f22327fc700 1 -- 172.21.15.104:0/4190747688 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2228011990 con 0x7f22340b4ff0 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f22327fc700 1 -- 172.21.15.104:0/4190747688 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228011d20 con 0x7f22340b4ff0 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 -- 172.21.15.104:0/4190747688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 msgr2=0x7f22340ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 --2- 172.21.15.104:0/4190747688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f22340ce350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2228009840 tx=0x7f2228009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 -- 172.21.15.104:0/4190747688 shutdown_connections 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 --2- 172.21.15.104:0/4190747688 >> v2:172.21.15.114:3300/0 conn(0x7f22340ce890 0x7f223415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 --2- 172.21.15.104:0/4190747688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f22340ce350 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 -- 172.21.15.104:0/4190747688 >> 172.21.15.104:0/4190747688 conn(0x7f2234089fd0 msgr2=0x7f22340d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 -- 172.21.15.104:0/4190747688 shutdown_connections 2023-06-10T18:57:48.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.517+0000 7f2239f45700 1 -- 172.21.15.104:0/4190747688 wait complete. 2023-06-10T18:57:48.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.518+0000 7f2239f45700 1 Processor -- start 2023-06-10T18:57:48.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 -- start start 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f223414eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a420 0x7f22340ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8300 con 0x7f22340ce890 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8570 con 0x7f223415a420 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2239f45700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c8850 con 0x7f22340b4ff0 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2232ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2232ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39352/0 (socket says 172.21.15.104:39352) 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.519+0000 7f2232ffd700 1 -- 172.21.15.104:0/3736821978 learned_addr learned my addr 172.21.15.104:0/3736821978 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2233fff700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a420 0x7f22340ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f22337fe700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f223414eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 -- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 msgr2=0x7f223414eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f223414eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 -- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a420 msgr2=0x7f22340ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a420 0x7f22340ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 -- 172.21.15.104:0/3736821978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c009b30 con 0x7f22340ce890 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f22337fe700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f223414eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2232ffd700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f2228009fd0 tx=0x7f2228004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280036a0 con 0x7f22340ce890 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222801e070 con 0x7f22340ce890 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22280094a0 con 0x7f22340ce890 2023-06-10T18:57:48.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228003a60 con 0x7f22340ce890 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.520+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22340c8e10 con 0x7f22340ce890 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.521+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340c9000 con 0x7f22340ce890 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.522+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2228003bc0 con 0x7f22340ce890 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.523+0000 7f221bfff700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 0x7f222006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.523+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f222808aae0 con 0x7f22340ce890 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.523+0000 7f22337fe700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 0x7f222006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.523+0000 7f22337fe700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 0x7f222006d190 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2224000f60 tx=0x7f2224009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.527+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2228055a40 con 0x7f22340ce890 2023-06-10T18:57:48.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.711+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22340c86e0 con 0x7f222006ace0 2023-06-10T18:57:48.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.719+0000 7f221bfff700 1 -- 172.21.15.104:0/3736821978 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311612 (secure 0 0 0) 0x7f22340c86e0 con 0x7f222006ace0 2023-06-10T18:57:48.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:48.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 msgr2=0x7f222006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 0x7f222006d190 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2224000f60 tx=0x7f2224009040 comp rx=0 tx=0).stop 2023-06-10T18:57:48.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 msgr2=0x7f223414e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f2228009fd0 tx=0x7f2228004750 comp rx=0 tx=0).stop 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 shutdown_connections 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f222006ace0 0x7f222006d190 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f223415a420 0x7f22340ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22340b4ff0 0x7f223414eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 --2- 172.21.15.104:0/3736821978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22340ce890 0x7f223414e700 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 >> 172.21.15.104:0/3736821978 conn(0x7f2234089fd0 msgr2=0x7f22340b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.743 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 shutdown_connections 2023-06-10T18:57:48.744 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.723+0000 7f2239f45700 1 -- 172.21.15.104:0/3736821978 wait complete. 2023-06-10T18:57:48.744 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:57:48.744 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2023-06-10T18:57:47.567597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244664,"kb_used_data":16760,"kb_used_omap":0,"kb_used_meta":227840,"kb_avail":754730056,"statfs":{"total":773094113280,"available":772843577344,"internally_reserved":0,"allocated":17162240,"data_stored":9000928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":233308160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719619"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437402+0000","last_change":"2023-06-10T18:57:44.283604+0000","last_active":"2023-06-10T18:57:45.437402+0000","last_peered":"2023-06-10T18:57:45.437402+0000","last_clean":"2023-06-10T18:57:45.437402+0000","last_became_active":"2023-06-10T18:57:44.283222+0000","last_became_peered":"2023-06-10T18:57:44.283222+0000","last_unstale":"2023-06-10T18:57:45.437402+0000","last_undegraded":"2023-06-10T18:57:45.437402+0000","last_fullsized":"2023-06-10T18:57:45.437402+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:33.572694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436977+0000","last_change":"2023-06-10T18:57:44.286875+0000","last_active":"2023-06-10T18:57:45.436977+0000","last_peered":"2023-06-10T18:57:45.436977+0000","last_clean":"2023-06-10T18:57:45.436977+0000","last_became_active":"2023-06-10T18:57:44.286449+0000","last_became_peered":"2023-06-10T18:57:44.286449+0000","last_unstale":"2023-06-10T18:57:45.436977+0000","last_undegraded":"2023-06-10T18:57:45.436977+0000","last_fullsized":"2023-06-10T18:57:45.436977+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:41:23.785356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270814+0000","last_change":"2023-06-10T18:57:44.293253+0000","last_active":"2023-06-10T18:57:45.270814+0000","last_peered":"2023-06-10T18:57:45.270814+0000","last_clean":"2023-06-10T18:57:45.270814+0000","last_became_active":"2023-06-10T18:57:44.292969+0000","last_became_peered":"2023-06-10T18:57:44.292969+0000","last_unstale":"2023-06-10T18:57:45.270814+0000","last_undegraded":"2023-06-10T18:57:45.270814+0000","last_fullsized":"2023-06-10T18:57:45.270814+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:16.217590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437507+0000","last_change":"2023-06-10T18:57:44.280914+0000","last_active":"2023-06-10T18:57:45.437507+0000","last_peered":"2023-06-10T18:57:45.437507+0000","last_clean":"2023-06-10T18:57:45.437507+0000","last_became_active":"2023-06-10T18:57:44.280484+0000","last_became_peered":"2023-06-10T18:57:44.280484+0000","last_unstale":"2023-06-10T18:57:45.437507+0000","last_undegraded":"2023-06-10T18:57:45.437507+0000","last_fullsized":"2023-06-10T18:57:45.437507+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:11.577091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437089+0000","last_change":"2023-06-10T18:57:44.291192+0000","last_active":"2023-06-10T18:57:45.437089+0000","last_peered":"2023-06-10T18:57:45.437089+0000","last_clean":"2023-06-10T18:57:45.437089+0000","last_became_active":"2023-06-10T18:57:44.290898+0000","last_became_peered":"2023-06-10T18:57:44.290898+0000","last_unstale":"2023-06-10T18:57:45.437089+0000","last_undegraded":"2023-06-10T18:57:45.437089+0000","last_fullsized":"2023-06-10T18:57:45.437089+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:03:47.708549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437302+0000","last_change":"2023-06-10T18:57:44.287600+0000","last_active":"2023-06-10T18:57:45.437302+0000","last_peered":"2023-06-10T18:57:45.437302+0000","last_clean":"2023-06-10T18:57:45.437302+0000","last_became_active":"2023-06-10T18:57:44.287306+0000","last_became_peered":"2023-06-10T18:57:44.287306+0000","last_unstale":"2023-06-10T18:57:45.437302+0000","last_undegraded":"2023-06-10T18:57:45.437302+0000","last_fullsized":"2023-06-10T18:57:45.437302+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:19:38.702174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268698+0000","last_change":"2023-06-10T18:57:44.290226+0000","last_active":"2023-06-10T18:57:45.268698+0000","last_peered":"2023-06-10T18:57:45.268698+0000","last_clean":"2023-06-10T18:57:45.268698+0000","last_became_active":"2023-06-10T18:57:44.289960+0000","last_became_peered":"2023-06-10T18:57:44.289960+0000","last_unstale":"2023-06-10T18:57:45.268698+0000","last_undegraded":"2023-06-10T18:57:45.268698+0000","last_fullsized":"2023-06-10T18:57:45.268698+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:10:42.872345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436804+0000","last_change":"2023-06-10T18:57:44.285495+0000","last_active":"2023-06-10T18:57:45.436804+0000","last_peered":"2023-06-10T18:57:45.436804+0000","last_clean":"2023-06-10T18:57:45.436804+0000","last_became_active":"2023-06-10T18:57:44.285037+0000","last_became_peered":"2023-06-10T18:57:44.285037+0000","last_unstale":"2023-06-10T18:57:45.436804+0000","last_undegraded":"2023-06-10T18:57:45.436804+0000","last_fullsized":"2023-06-10T18:57:45.436804+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:04.458580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437402+0000","last_change":"2023-06-10T18:57:44.292670+0000","last_active":"2023-06-10T18:57:45.437402+0000","last_peered":"2023-06-10T18:57:45.437402+0000","last_clean":"2023-06-10T18:57:45.437402+0000","last_became_active":"2023-06-10T18:57:44.292385+0000","last_became_peered":"2023-06-10T18:57:44.292385+0000","last_unstale":"2023-06-10T18:57:45.437402+0000","last_undegraded":"2023-06-10T18:57:45.437402+0000","last_fullsized":"2023-06-10T18:57:45.437402+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:07.473584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437588+0000","last_change":"2023-06-10T18:57:44.286703+0000","last_active":"2023-06-10T18:57:45.437588+0000","last_peered":"2023-06-10T18:57:45.437588+0000","last_clean":"2023-06-10T18:57:45.437588+0000","last_became_active":"2023-06-10T18:57:44.286125+0000","last_became_peered":"2023-06-10T18:57:44.286125+0000","last_unstale":"2023-06-10T18:57:45.437588+0000","last_undegraded":"2023-06-10T18:57:45.437588+0000","last_fullsized":"2023-06-10T18:57:45.437588+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:09.492898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437228+0000","last_change":"2023-06-10T18:57:44.291139+0000","last_active":"2023-06-10T18:57:45.437228+0000","last_peered":"2023-06-10T18:57:45.437228+0000","last_clean":"2023-06-10T18:57:45.437228+0000","last_became_active":"2023-06-10T18:57:44.290850+0000","last_became_peered":"2023-06-10T18:57:44.290850+0000","last_unstale":"2023-06-10T18:57:45.437228+0000","last_undegraded":"2023-06-10T18:57:45.437228+0000","last_fullsized":"2023-06-10T18:57:45.437228+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:21:56.384098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437297+0000","last_change":"2023-06-10T18:57:44.289202+0000","last_active":"2023-06-10T18:57:45.437297+0000","last_peered":"2023-06-10T18:57:45.437297+0000","last_clean":"2023-06-10T18:57:45.437297+0000","last_became_active":"2023-06-10T18:57:44.287564+0000","last_became_peered":"2023-06-10T18:57:44.287564+0000","last_unstale":"2023-06-10T18:57:45.437297+0000","last_undegraded":"2023-06-10T18:57:45.437297+0000","last_fullsized":"2023-06-10T18:57:45.437297+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:02.229426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270797+0000","last_change":"2023-06-10T18:57:44.282894+0000","last_active":"2023-06-10T18:57:45.270797+0000","last_peered":"2023-06-10T18:57:45.270797+0000","last_clean":"2023-06-10T18:57:45.270797+0000","last_became_active":"2023-06-10T18:57:44.282552+0000","last_became_peered":"2023-06-10T18:57:44.282552+0000","last_unstale":"2023-06-10T18:57:45.270797+0000","last_undegraded":"2023-06-10T18:57:45.270797+0000","last_fullsized":"2023-06-10T18:57:45.270797+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:25.425413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270063+0000","last_change":"2023-06-10T18:57:44.285585+0000","last_active":"2023-06-10T18:57:45.270063+0000","last_peered":"2023-06-10T18:57:45.270063+0000","last_clean":"2023-06-10T18:57:45.270063+0000","last_became_active":"2023-06-10T18:57:44.283623+0000","last_became_peered":"2023-06-10T18:57:44.283623+0000","last_unstale":"2023-06-10T18:57:45.270063+0000","last_undegraded":"2023-06-10T18:57:45.270063+0000","last_fullsized":"2023-06-10T18:57:45.270063+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:20:19.758784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436943+0000","last_change":"2023-06-10T18:57:44.273697+0000","last_active":"2023-06-10T18:57:45.436943+0000","last_peered":"2023-06-10T18:57:45.436943+0000","last_clean":"2023-06-10T18:57:45.436943+0000","last_became_active":"2023-06-10T18:57:44.272521+0000","last_became_peered":"2023-06-10T18:57:44.272521+0000","last_unstale":"2023-06-10T18:57:45.436943+0000","last_undegraded":"2023-06-10T18:57:45.436943+0000","last_fullsized":"2023-06-10T18:57:45.436943+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:00:43.082174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268813+0000","last_change":"2023-06-10T18:57:44.283599+0000","last_active":"2023-06-10T18:57:45.268813+0000","last_peered":"2023-06-10T18:57:45.268813+0000","last_clean":"2023-06-10T18:57:45.268813+0000","last_became_active":"2023-06-10T18:57:44.283351+0000","last_became_peered":"2023-06-10T18:57:44.283351+0000","last_unstale":"2023-06-10T18:57:45.268813+0000","last_undegraded":"2023-06-10T18:57:45.268813+0000","last_fullsized":"2023-06-10T18:57:45.268813+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:28:32.256088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437270+0000","last_change":"2023-06-10T18:57:44.293823+0000","last_active":"2023-06-10T18:57:45.437270+0000","last_peered":"2023-06-10T18:57:45.437270+0000","last_clean":"2023-06-10T18:57:45.437270+0000","last_became_active":"2023-06-10T18:57:44.293603+0000","last_became_peered":"2023-06-10T18:57:44.293603+0000","last_unstale":"2023-06-10T18:57:45.437270+0000","last_undegraded":"2023-06-10T18:57:45.437270+0000","last_fullsized":"2023-06-10T18:57:45.437270+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:49:15.731959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438099+0000","last_change":"2023-06-10T18:57:44.285495+0000","last_active":"2023-06-10T18:57:45.438099+0000","last_peered":"2023-06-10T18:57:45.438099+0000","last_clean":"2023-06-10T18:57:45.438099+0000","last_became_active":"2023-06-10T18:57:44.285094+0000","last_became_peered":"2023-06-10T18:57:44.285094+0000","last_unstale":"2023-06-10T18:57:45.438099+0000","last_undegraded":"2023-06-10T18:57:45.438099+0000","last_fullsized":"2023-06-10T18:57:45.438099+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:39:03.044414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436912+0000","last_change":"2023-06-10T18:57:44.290254+0000","last_active":"2023-06-10T18:57:45.436912+0000","last_peered":"2023-06-10T18:57:45.436912+0000","last_clean":"2023-06-10T18:57:45.436912+0000","last_became_active":"2023-06-10T18:57:44.289962+0000","last_became_peered":"2023-06-10T18:57:44.289962+0000","last_unstale":"2023-06-10T18:57:45.436912+0000","last_undegraded":"2023-06-10T18:57:45.436912+0000","last_fullsized":"2023-06-10T18:57:45.436912+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:47:13.547571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270327+0000","last_change":"2023-06-10T18:57:44.289241+0000","last_active":"2023-06-10T18:57:45.270327+0000","last_peered":"2023-06-10T18:57:45.270327+0000","last_clean":"2023-06-10T18:57:45.270327+0000","last_became_active":"2023-06-10T18:57:44.288980+0000","last_became_peered":"2023-06-10T18:57:44.288980+0000","last_unstale":"2023-06-10T18:57:45.270327+0000","last_undegraded":"2023-06-10T18:57:45.270327+0000","last_fullsized":"2023-06-10T18:57:45.270327+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:45.835332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437834+0000","last_change":"2023-06-10T18:57:44.283706+0000","last_active":"2023-06-10T18:57:45.437834+0000","last_peered":"2023-06-10T18:57:45.437834+0000","last_clean":"2023-06-10T18:57:45.437834+0000","last_became_active":"2023-06-10T18:57:44.283443+0000","last_became_peered":"2023-06-10T18:57:44.283443+0000","last_unstale":"2023-06-10T18:57:45.437834+0000","last_undegraded":"2023-06-10T18:57:45.437834+0000","last_fullsized":"2023-06-10T18:57:45.437834+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:08:10.445545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437334+0000","last_change":"2023-06-10T18:57:44.285648+0000","last_active":"2023-06-10T18:57:45.437334+0000","last_peered":"2023-06-10T18:57:45.437334+0000","last_clean":"2023-06-10T18:57:45.437334+0000","last_became_active":"2023-06-10T18:57:44.285374+0000","last_became_peered":"2023-06-10T18:57:44.285374+0000","last_unstale":"2023-06-10T18:57:45.437334+0000","last_undegraded":"2023-06-10T18:57:45.437334+0000","last_fullsized":"2023-06-10T18:57:45.437334+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:34:10.143555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436943+0000","last_change":"2023-06-10T18:57:44.286190+0000","last_active":"2023-06-10T18:57:45.436943+0000","last_peered":"2023-06-10T18:57:45.436943+0000","last_clean":"2023-06-10T18:57:45.436943+0000","last_became_active":"2023-06-10T18:57:44.285822+0000","last_became_peered":"2023-06-10T18:57:44.285822+0000","last_unstale":"2023-06-10T18:57:45.436943+0000","last_undegraded":"2023-06-10T18:57:45.436943+0000","last_fullsized":"2023-06-10T18:57:45.436943+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:34:21.036099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436894+0000","last_change":"2023-06-10T18:57:44.285308+0000","last_active":"2023-06-10T18:57:45.436894+0000","last_peered":"2023-06-10T18:57:45.436894+0000","last_clean":"2023-06-10T18:57:45.436894+0000","last_became_active":"2023-06-10T18:57:44.284652+0000","last_became_peered":"2023-06-10T18:57:44.284652+0000","last_unstale":"2023-06-10T18:57:45.436894+0000","last_undegraded":"2023-06-10T18:57:45.436894+0000","last_fullsized":"2023-06-10T18:57:45.436894+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:32:29.741070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270501+0000","last_change":"2023-06-10T18:57:44.291329+0000","last_active":"2023-06-10T18:57:45.270501+0000","last_peered":"2023-06-10T18:57:45.270501+0000","last_clean":"2023-06-10T18:57:45.270501+0000","last_became_active":"2023-06-10T18:57:44.291071+0000","last_became_peered":"2023-06-10T18:57:44.291071+0000","last_unstale":"2023-06-10T18:57:45.270501+0000","last_undegraded":"2023-06-10T18:57:45.270501+0000","last_fullsized":"2023-06-10T18:57:45.270501+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:00:21.233143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268768+0000","last_change":"2023-06-10T18:57:44.279087+0000","last_active":"2023-06-10T18:57:45.268768+0000","last_peered":"2023-06-10T18:57:45.268768+0000","last_clean":"2023-06-10T18:57:45.268768+0000","last_became_active":"2023-06-10T18:57:44.278829+0000","last_became_peered":"2023-06-10T18:57:44.278829+0000","last_unstale":"2023-06-10T18:57:45.268768+0000","last_undegraded":"2023-06-10T18:57:45.268768+0000","last_fullsized":"2023-06-10T18:57:45.268768+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:10:24.570339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270809+0000","last_change":"2023-06-10T18:57:44.286304+0000","last_active":"2023-06-10T18:57:45.270809+0000","last_peered":"2023-06-10T18:57:45.270809+0000","last_clean":"2023-06-10T18:57:45.270809+0000","last_became_active":"2023-06-10T18:57:44.285993+0000","last_became_peered":"2023-06-10T18:57:44.285993+0000","last_unstale":"2023-06-10T18:57:45.270809+0000","last_undegraded":"2023-06-10T18:57:45.270809+0000","last_fullsized":"2023-06-10T18:57:45.270809+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:30.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437639+0000","last_change":"2023-06-10T18:57:44.278817+0000","last_active":"2023-06-10T18:57:45.437639+0000","last_peered":"2023-06-10T18:57:45.437639+0000","last_clean":"2023-06-10T18:57:45.437639+0000","last_became_active":"2023-06-10T18:57:44.278539+0000","last_became_peered":"2023-06-10T18:57:44.278539+0000","last_unstale":"2023-06-10T18:57:45.437639+0000","last_undegraded":"2023-06-10T18:57:45.437639+0000","last_fullsized":"2023-06-10T18:57:45.437639+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:10.604887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270241+0000","last_change":"2023-06-10T18:57:44.283599+0000","last_active":"2023-06-10T18:57:45.270241+0000","last_peered":"2023-06-10T18:57:45.270241+0000","last_clean":"2023-06-10T18:57:45.270241+0000","last_became_active":"2023-06-10T18:57:44.282650+0000","last_became_peered":"2023-06-10T18:57:44.282650+0000","last_unstale":"2023-06-10T18:57:45.270241+0000","last_undegraded":"2023-06-10T18:57:45.270241+0000","last_fullsized":"2023-06-10T18:57:45.270241+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:01.272573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269193+0000","last_change":"2023-06-10T18:57:44.286563+0000","last_active":"2023-06-10T18:57:45.269193+0000","last_peered":"2023-06-10T18:57:45.269193+0000","last_clean":"2023-06-10T18:57:45.269193+0000","last_became_active":"2023-06-10T18:57:44.286238+0000","last_became_peered":"2023-06-10T18:57:44.286238+0000","last_unstale":"2023-06-10T18:57:45.269193+0000","last_undegraded":"2023-06-10T18:57:45.269193+0000","last_fullsized":"2023-06-10T18:57:45.269193+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:12:23.856395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269253+0000","last_change":"2023-06-10T18:57:44.289030+0000","last_active":"2023-06-10T18:57:45.269253+0000","last_peered":"2023-06-10T18:57:45.269253+0000","last_clean":"2023-06-10T18:57:45.269253+0000","last_became_active":"2023-06-10T18:57:44.285903+0000","last_became_peered":"2023-06-10T18:57:44.285903+0000","last_unstale":"2023-06-10T18:57:45.269253+0000","last_undegraded":"2023-06-10T18:57:45.269253+0000","last_fullsized":"2023-06-10T18:57:45.269253+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:29:27.097073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437768+0000","last_change":"2023-06-10T18:57:44.285109+0000","last_active":"2023-06-10T18:57:45.437768+0000","last_peered":"2023-06-10T18:57:45.437768+0000","last_clean":"2023-06-10T18:57:45.437768+0000","last_became_active":"2023-06-10T18:57:44.284836+0000","last_became_peered":"2023-06-10T18:57:44.284836+0000","last_unstale":"2023-06-10T18:57:45.437768+0000","last_undegraded":"2023-06-10T18:57:45.437768+0000","last_fullsized":"2023-06-10T18:57:45.437768+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:19.338867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269061+0000","last_change":"2023-06-10T18:57:44.278516+0000","last_active":"2023-06-10T18:57:45.269061+0000","last_peered":"2023-06-10T18:57:45.269061+0000","last_clean":"2023-06-10T18:57:45.269061+0000","last_became_active":"2023-06-10T18:57:44.278317+0000","last_became_peered":"2023-06-10T18:57:44.278317+0000","last_unstale":"2023-06-10T18:57:45.269061+0000","last_undegraded":"2023-06-10T18:57:45.269061+0000","last_fullsized":"2023-06-10T18:57:45.269061+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:41.651403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"158'4","reported_seq":17,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437274+0000","last_change":"2023-06-10T18:57:43.878349+0000","last_active":"2023-06-10T18:57:45.437274+0000","last_peered":"2023-06-10T18:57:45.437274+0000","last_clean":"2023-06-10T18:57:45.437274+0000","last_became_active":"2023-06-10T18:57:43.878026+0000","last_became_peered":"2023-06-10T18:57:43.878026+0000","last_unstale":"2023-06-10T18:57:45.437274+0000","last_undegraded":"2023-06-10T18:57:45.437274+0000","last_fullsized":"2023-06-10T18:57:45.437274+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:11:06.161680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270382+0000","last_change":"2023-06-10T18:57:44.286549+0000","last_active":"2023-06-10T18:57:45.270382+0000","last_peered":"2023-06-10T18:57:45.270382+0000","last_clean":"2023-06-10T18:57:45.270382+0000","last_became_active":"2023-06-10T18:57:44.286220+0000","last_became_peered":"2023-06-10T18:57:44.286220+0000","last_unstale":"2023-06-10T18:57:45.270382+0000","last_undegraded":"2023-06-10T18:57:45.270382+0000","last_fullsized":"2023-06-10T18:57:45.270382+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:36:53.266200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270576+0000","last_change":"2023-06-10T18:57:43.877624+0000","last_active":"2023-06-10T18:57:45.270576+0000","last_peered":"2023-06-10T18:57:45.270576+0000","last_clean":"2023-06-10T18:57:45.270576+0000","last_became_active":"2023-06-10T18:57:43.877330+0000","last_became_peered":"2023-06-10T18:57:43.877330+0000","last_unstale":"2023-06-10T18:57:45.270576+0000","last_undegraded":"2023-06-10T18:57:45.270576+0000","last_fullsized":"2023-06-10T18:57:45.270576+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:24.192051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437017+0000","last_change":"2023-06-10T18:57:44.287612+0000","last_active":"2023-06-10T18:57:45.437017+0000","last_peered":"2023-06-10T18:57:45.437017+0000","last_clean":"2023-06-10T18:57:45.437017+0000","last_became_active":"2023-06-10T18:57:44.287361+0000","last_became_peered":"2023-06-10T18:57:44.287361+0000","last_unstale":"2023-06-10T18:57:45.437017+0000","last_undegraded":"2023-06-10T18:57:45.437017+0000","last_fullsized":"2023-06-10T18:57:45.437017+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:11:45.593784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438054+0000","last_change":"2023-06-10T18:57:43.880376+0000","last_active":"2023-06-10T18:57:45.438054+0000","last_peered":"2023-06-10T18:57:45.438054+0000","last_clean":"2023-06-10T18:57:45.438054+0000","last_became_active":"2023-06-10T18:57:43.880095+0000","last_became_peered":"2023-06-10T18:57:43.880095+0000","last_unstale":"2023-06-10T18:57:45.438054+0000","last_undegraded":"2023-06-10T18:57:45.438054+0000","last_fullsized":"2023-06-10T18:57:45.438054+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:45:36.825550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270924+0000","last_change":"2023-06-10T18:57:44.288721+0000","last_active":"2023-06-10T18:57:45.270924+0000","last_peered":"2023-06-10T18:57:45.270924+0000","last_clean":"2023-06-10T18:57:45.270924+0000","last_became_active":"2023-06-10T18:57:44.287119+0000","last_became_peered":"2023-06-10T18:57:44.287119+0000","last_unstale":"2023-06-10T18:57:45.270924+0000","last_undegraded":"2023-06-10T18:57:45.270924+0000","last_fullsized":"2023-06-10T18:57:45.270924+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:02.631238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437768+0000","last_change":"2023-06-10T18:57:43.874389+0000","last_active":"2023-06-10T18:57:45.437768+0000","last_peered":"2023-06-10T18:57:45.437768+0000","last_clean":"2023-06-10T18:57:45.437768+0000","last_became_active":"2023-06-10T18:57:43.874069+0000","last_became_peered":"2023-06-10T18:57:43.874069+0000","last_unstale":"2023-06-10T18:57:45.437768+0000","last_undegraded":"2023-06-10T18:57:45.437768+0000","last_fullsized":"2023-06-10T18:57:45.437768+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:17:15.212015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268996+0000","last_change":"2023-06-10T18:57:44.278777+0000","last_active":"2023-06-10T18:57:45.268996+0000","last_peered":"2023-06-10T18:57:45.268996+0000","last_clean":"2023-06-10T18:57:45.268996+0000","last_became_active":"2023-06-10T18:57:44.278009+0000","last_became_peered":"2023-06-10T18:57:44.278009+0000","last_unstale":"2023-06-10T18:57:45.268996+0000","last_undegraded":"2023-06-10T18:57:45.268996+0000","last_fullsized":"2023-06-10T18:57:45.268996+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:09:51.453377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270579+0000","last_change":"2023-06-10T18:57:43.878033+0000","last_active":"2023-06-10T18:57:45.270579+0000","last_peered":"2023-06-10T18:57:45.270579+0000","last_clean":"2023-06-10T18:57:45.270579+0000","last_became_active":"2023-06-10T18:57:43.877065+0000","last_became_peered":"2023-06-10T18:57:43.877065+0000","last_unstale":"2023-06-10T18:57:45.270579+0000","last_undegraded":"2023-06-10T18:57:45.270579+0000","last_fullsized":"2023-06-10T18:57:45.270579+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:15.451498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436949+0000","last_change":"2023-06-10T18:57:44.275648+0000","last_active":"2023-06-10T18:57:45.436949+0000","last_peered":"2023-06-10T18:57:45.436949+0000","last_clean":"2023-06-10T18:57:45.436949+0000","last_became_active":"2023-06-10T18:57:44.275470+0000","last_became_peered":"2023-06-10T18:57:44.275470+0000","last_unstale":"2023-06-10T18:57:45.436949+0000","last_undegraded":"2023-06-10T18:57:45.436949+0000","last_fullsized":"2023-06-10T18:57:45.436949+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:01.272573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437048+0000","last_change":"2023-06-10T18:57:43.874147+0000","last_active":"2023-06-10T18:57:45.437048+0000","last_peered":"2023-06-10T18:57:45.437048+0000","last_clean":"2023-06-10T18:57:45.437048+0000","last_became_active":"2023-06-10T18:57:43.873866+0000","last_became_peered":"2023-06-10T18:57:43.873866+0000","last_unstale":"2023-06-10T18:57:45.437048+0000","last_undegraded":"2023-06-10T18:57:45.437048+0000","last_fullsized":"2023-06-10T18:57:45.437048+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:44.829758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438249+0000","last_change":"2023-06-10T18:57:44.284456+0000","last_active":"2023-06-10T18:57:45.438249+0000","last_peered":"2023-06-10T18:57:45.438249+0000","last_clean":"2023-06-10T18:57:45.438249+0000","last_became_active":"2023-06-10T18:57:44.284195+0000","last_became_peered":"2023-06-10T18:57:44.284195+0000","last_unstale":"2023-06-10T18:57:45.438249+0000","last_undegraded":"2023-06-10T18:57:45.438249+0000","last_fullsized":"2023-06-10T18:57:45.438249+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:11.605609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437131+0000","last_change":"2023-06-10T18:57:43.880827+0000","last_active":"2023-06-10T18:57:45.437131+0000","last_peered":"2023-06-10T18:57:45.437131+0000","last_clean":"2023-06-10T18:57:45.437131+0000","last_became_active":"2023-06-10T18:57:43.880424+0000","last_became_peered":"2023-06-10T18:57:43.880424+0000","last_unstale":"2023-06-10T18:57:45.437131+0000","last_undegraded":"2023-06-10T18:57:45.437131+0000","last_fullsized":"2023-06-10T18:57:45.437131+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:47:08.744765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436983+0000","last_change":"2023-06-10T18:57:44.282545+0000","last_active":"2023-06-10T18:57:45.436983+0000","last_peered":"2023-06-10T18:57:45.436983+0000","last_clean":"2023-06-10T18:57:45.436983+0000","last_became_active":"2023-06-10T18:57:44.282263+0000","last_became_peered":"2023-06-10T18:57:44.282263+0000","last_unstale":"2023-06-10T18:57:45.436983+0000","last_undegraded":"2023-06-10T18:57:45.436983+0000","last_fullsized":"2023-06-10T18:57:45.436983+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:11:16.111459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268635+0000","last_change":"2023-06-10T18:57:43.877331+0000","last_active":"2023-06-10T18:57:45.268635+0000","last_peered":"2023-06-10T18:57:45.268635+0000","last_clean":"2023-06-10T18:57:45.268635+0000","last_became_active":"2023-06-10T18:57:43.877030+0000","last_became_peered":"2023-06-10T18:57:43.877030+0000","last_unstale":"2023-06-10T18:57:45.268635+0000","last_undegraded":"2023-06-10T18:57:45.268635+0000","last_fullsized":"2023-06-10T18:57:45.268635+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:02:30.896540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438023+0000","last_change":"2023-06-10T18:57:44.281465+0000","last_active":"2023-06-10T18:57:45.438023+0000","last_peered":"2023-06-10T18:57:45.438023+0000","last_clean":"2023-06-10T18:57:45.438023+0000","last_became_active":"2023-06-10T18:57:44.281150+0000","last_became_peered":"2023-06-10T18:57:44.281150+0000","last_unstale":"2023-06-10T18:57:45.438023+0000","last_undegraded":"2023-06-10T18:57:45.438023+0000","last_fullsized":"2023-06-10T18:57:45.438023+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:46:12.512583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270607+0000","last_change":"2023-06-10T18:57:43.877882+0000","last_active":"2023-06-10T18:57:45.270607+0000","last_peered":"2023-06-10T18:57:45.270607+0000","last_clean":"2023-06-10T18:57:45.270607+0000","last_became_active":"2023-06-10T18:57:43.877609+0000","last_became_peered":"2023-06-10T18:57:43.877609+0000","last_unstale":"2023-06-10T18:57:45.270607+0000","last_undegraded":"2023-06-10T18:57:45.270607+0000","last_fullsized":"2023-06-10T18:57:45.270607+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:40.135041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437510+0000","last_change":"2023-06-10T18:57:44.286368+0000","last_active":"2023-06-10T18:57:45.437510+0000","last_peered":"2023-06-10T18:57:45.437510+0000","last_clean":"2023-06-10T18:57:45.437510+0000","last_became_active":"2023-06-10T18:57:44.279534+0000","last_became_peered":"2023-06-10T18:57:44.279534+0000","last_unstale":"2023-06-10T18:57:45.437510+0000","last_undegraded":"2023-06-10T18:57:45.437510+0000","last_fullsized":"2023-06-10T18:57:45.437510+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:26:46.392589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270508+0000","last_change":"2023-06-10T18:57:43.877598+0000","last_active":"2023-06-10T18:57:45.270508+0000","last_peered":"2023-06-10T18:57:45.270508+0000","last_clean":"2023-06-10T18:57:45.270508+0000","last_became_active":"2023-06-10T18:57:43.877095+0000","last_became_peered":"2023-06-10T18:57:43.877095+0000","last_unstale":"2023-06-10T18:57:45.270508+0000","last_undegraded":"2023-06-10T18:57:45.270508+0000","last_fullsized":"2023-06-10T18:57:45.270508+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:10.600034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437700+0000","last_change":"2023-06-10T18:57:44.275306+0000","last_active":"2023-06-10T18:57:45.437700+0000","last_peered":"2023-06-10T18:57:45.437700+0000","last_clean":"2023-06-10T18:57:45.437700+0000","last_became_active":"2023-06-10T18:57:44.275037+0000","last_became_peered":"2023-06-10T18:57:44.275037+0000","last_unstale":"2023-06-10T18:57:45.437700+0000","last_undegraded":"2023-06-10T18:57:45.437700+0000","last_fullsized":"2023-06-10T18:57:45.437700+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:30.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"158'1","reported_seq":14,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437896+0000","last_change":"2023-06-10T18:57:43.873255+0000","last_active":"2023-06-10T18:57:45.437896+0000","last_peered":"2023-06-10T18:57:45.437896+0000","last_clean":"2023-06-10T18:57:45.437896+0000","last_became_active":"2023-06-10T18:57:43.873015+0000","last_became_peered":"2023-06-10T18:57:43.873015+0000","last_unstale":"2023-06-10T18:57:45.437896+0000","last_undegraded":"2023-06-10T18:57:45.437896+0000","last_fullsized":"2023-06-10T18:57:45.437896+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:44:16.432344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268864+0000","last_change":"2023-06-10T18:57:44.277872+0000","last_active":"2023-06-10T18:57:45.268864+0000","last_peered":"2023-06-10T18:57:45.268864+0000","last_clean":"2023-06-10T18:57:45.268864+0000","last_became_active":"2023-06-10T18:57:44.277571+0000","last_became_peered":"2023-06-10T18:57:44.277571+0000","last_unstale":"2023-06-10T18:57:45.268864+0000","last_undegraded":"2023-06-10T18:57:45.268864+0000","last_fullsized":"2023-06-10T18:57:45.268864+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:59:00.994547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437718+0000","last_change":"2023-06-10T18:57:43.874744+0000","last_active":"2023-06-10T18:57:45.437718+0000","last_peered":"2023-06-10T18:57:45.437718+0000","last_clean":"2023-06-10T18:57:45.437718+0000","last_became_active":"2023-06-10T18:57:43.873677+0000","last_became_peered":"2023-06-10T18:57:43.873677+0000","last_unstale":"2023-06-10T18:57:45.437718+0000","last_undegraded":"2023-06-10T18:57:45.437718+0000","last_fullsized":"2023-06-10T18:57:45.437718+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:50:10.187968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270192+0000","last_change":"2023-06-10T18:57:44.290334+0000","last_active":"2023-06-10T18:57:45.270192+0000","last_peered":"2023-06-10T18:57:45.270192+0000","last_clean":"2023-06-10T18:57:45.270192+0000","last_became_active":"2023-06-10T18:57:44.290072+0000","last_became_peered":"2023-06-10T18:57:44.290072+0000","last_unstale":"2023-06-10T18:57:45.270192+0000","last_undegraded":"2023-06-10T18:57:45.270192+0000","last_fullsized":"2023-06-10T18:57:45.270192+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:05:35.412940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437213+0000","last_change":"2023-06-10T18:57:43.877921+0000","last_active":"2023-06-10T18:57:45.437213+0000","last_peered":"2023-06-10T18:57:45.437213+0000","last_clean":"2023-06-10T18:57:45.437213+0000","last_became_active":"2023-06-10T18:57:43.877688+0000","last_became_peered":"2023-06-10T18:57:43.877688+0000","last_unstale":"2023-06-10T18:57:45.437213+0000","last_undegraded":"2023-06-10T18:57:45.437213+0000","last_fullsized":"2023-06-10T18:57:45.437213+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:05:40.135041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268915+0000","last_change":"2023-06-10T18:57:44.284304+0000","last_active":"2023-06-10T18:57:45.268915+0000","last_peered":"2023-06-10T18:57:45.268915+0000","last_clean":"2023-06-10T18:57:45.268915+0000","last_became_active":"2023-06-10T18:57:44.284057+0000","last_became_peered":"2023-06-10T18:57:44.284057+0000","last_unstale":"2023-06-10T18:57:45.268915+0000","last_undegraded":"2023-06-10T18:57:45.268915+0000","last_fullsized":"2023-06-10T18:57:45.268915+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:12:40.958894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437280+0000","last_change":"2023-06-10T18:57:43.880681+0000","last_active":"2023-06-10T18:57:45.437280+0000","last_peered":"2023-06-10T18:57:45.437280+0000","last_clean":"2023-06-10T18:57:45.437280+0000","last_became_active":"2023-06-10T18:57:43.880382+0000","last_became_peered":"2023-06-10T18:57:43.880382+0000","last_unstale":"2023-06-10T18:57:45.437280+0000","last_undegraded":"2023-06-10T18:57:45.437280+0000","last_fullsized":"2023-06-10T18:57:45.437280+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:43:40.554724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436815+0000","last_change":"2023-06-10T18:57:44.287502+0000","last_active":"2023-06-10T18:57:45.436815+0000","last_peered":"2023-06-10T18:57:45.436815+0000","last_clean":"2023-06-10T18:57:45.436815+0000","last_became_active":"2023-06-10T18:57:44.287085+0000","last_became_peered":"2023-06-10T18:57:44.287085+0000","last_unstale":"2023-06-10T18:57:45.436815+0000","last_undegraded":"2023-06-10T18:57:45.436815+0000","last_fullsized":"2023-06-10T18:57:45.436815+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:54.450021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438311+0000","last_change":"2023-06-10T18:57:43.878884+0000","last_active":"2023-06-10T18:57:45.438311+0000","last_peered":"2023-06-10T18:57:45.438311+0000","last_clean":"2023-06-10T18:57:45.438311+0000","last_became_active":"2023-06-10T18:57:43.878589+0000","last_became_peered":"2023-06-10T18:57:43.878589+0000","last_unstale":"2023-06-10T18:57:45.438311+0000","last_undegraded":"2023-06-10T18:57:45.438311+0000","last_fullsized":"2023-06-10T18:57:45.438311+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:31:39.394688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437078+0000","last_change":"2023-06-10T18:57:44.286048+0000","last_active":"2023-06-10T18:57:45.437078+0000","last_peered":"2023-06-10T18:57:45.437078+0000","last_clean":"2023-06-10T18:57:45.437078+0000","last_became_active":"2023-06-10T18:57:44.285488+0000","last_became_peered":"2023-06-10T18:57:44.285488+0000","last_unstale":"2023-06-10T18:57:45.437078+0000","last_undegraded":"2023-06-10T18:57:45.437078+0000","last_fullsized":"2023-06-10T18:57:45.437078+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:40:35.228387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436992+0000","last_change":"2023-06-10T18:57:43.874293+0000","last_active":"2023-06-10T18:57:45.436992+0000","last_peered":"2023-06-10T18:57:45.436992+0000","last_clean":"2023-06-10T18:57:45.436992+0000","last_became_active":"2023-06-10T18:57:43.873983+0000","last_became_peered":"2023-06-10T18:57:43.873983+0000","last_unstale":"2023-06-10T18:57:45.436992+0000","last_undegraded":"2023-06-10T18:57:45.436992+0000","last_fullsized":"2023-06-10T18:57:45.436992+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:16:54.682898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438225+0000","last_change":"2023-06-10T18:57:44.285624+0000","last_active":"2023-06-10T18:57:45.438225+0000","last_peered":"2023-06-10T18:57:45.438225+0000","last_clean":"2023-06-10T18:57:45.438225+0000","last_became_active":"2023-06-10T18:57:44.284489+0000","last_became_peered":"2023-06-10T18:57:44.284489+0000","last_unstale":"2023-06-10T18:57:45.438225+0000","last_undegraded":"2023-06-10T18:57:45.438225+0000","last_fullsized":"2023-06-10T18:57:45.438225+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:17:19.490820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437774+0000","last_change":"2023-06-10T18:57:43.878054+0000","last_active":"2023-06-10T18:57:45.437774+0000","last_peered":"2023-06-10T18:57:45.437774+0000","last_clean":"2023-06-10T18:57:45.437774+0000","last_became_active":"2023-06-10T18:57:43.877764+0000","last_became_peered":"2023-06-10T18:57:43.877764+0000","last_unstale":"2023-06-10T18:57:45.437774+0000","last_undegraded":"2023-06-10T18:57:45.437774+0000","last_fullsized":"2023-06-10T18:57:45.437774+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:43.195522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269085+0000","last_change":"2023-06-10T18:57:44.283811+0000","last_active":"2023-06-10T18:57:45.269085+0000","last_peered":"2023-06-10T18:57:45.269085+0000","last_clean":"2023-06-10T18:57:45.269085+0000","last_became_active":"2023-06-10T18:57:44.283573+0000","last_became_peered":"2023-06-10T18:57:44.283573+0000","last_unstale":"2023-06-10T18:57:45.269085+0000","last_undegraded":"2023-06-10T18:57:45.269085+0000","last_fullsized":"2023-06-10T18:57:45.269085+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:29:27.097073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270802+0000","last_change":"2023-06-10T18:57:43.877466+0000","last_active":"2023-06-10T18:57:45.270802+0000","last_peered":"2023-06-10T18:57:45.270802+0000","last_clean":"2023-06-10T18:57:45.270802+0000","last_became_active":"2023-06-10T18:57:43.877147+0000","last_became_peered":"2023-06-10T18:57:43.877147+0000","last_unstale":"2023-06-10T18:57:45.270802+0000","last_undegraded":"2023-06-10T18:57:45.270802+0000","last_fullsized":"2023-06-10T18:57:45.270802+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:09.599312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437140+0000","last_change":"2023-06-10T18:57:44.281630+0000","last_active":"2023-06-10T18:57:45.437140+0000","last_peered":"2023-06-10T18:57:45.437140+0000","last_clean":"2023-06-10T18:57:45.437140+0000","last_became_active":"2023-06-10T18:57:44.281355+0000","last_became_peered":"2023-06-10T18:57:44.281355+0000","last_unstale":"2023-06-10T18:57:45.437140+0000","last_undegraded":"2023-06-10T18:57:45.437140+0000","last_fullsized":"2023-06-10T18:57:45.437140+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:10.647884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437442+0000","last_change":"2023-06-10T18:57:43.879476+0000","last_active":"2023-06-10T18:57:45.437442+0000","last_peered":"2023-06-10T18:57:45.437442+0000","last_clean":"2023-06-10T18:57:45.437442+0000","last_became_active":"2023-06-10T18:57:43.879101+0000","last_became_peered":"2023-06-10T18:57:43.879101+0000","last_unstale":"2023-06-10T18:57:45.437442+0000","last_undegraded":"2023-06-10T18:57:45.437442+0000","last_fullsized":"2023-06-10T18:57:45.437442+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:46:10.642926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437324+0000","last_change":"2023-06-10T18:57:44.285168+0000","last_active":"2023-06-10T18:57:45.437324+0000","last_peered":"2023-06-10T18:57:45.437324+0000","last_clean":"2023-06-10T18:57:45.437324+0000","last_became_active":"2023-06-10T18:57:44.284899+0000","last_became_peered":"2023-06-10T18:57:44.284899+0000","last_unstale":"2023-06-10T18:57:45.437324+0000","last_undegraded":"2023-06-10T18:57:45.437324+0000","last_fullsized":"2023-06-10T18:57:45.437324+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:56.852581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438148+0000","last_change":"2023-06-10T18:57:43.875119+0000","last_active":"2023-06-10T18:57:45.438148+0000","last_peered":"2023-06-10T18:57:45.438148+0000","last_clean":"2023-06-10T18:57:45.438148+0000","last_became_active":"2023-06-10T18:57:43.874740+0000","last_became_peered":"2023-06-10T18:57:43.874740+0000","last_unstale":"2023-06-10T18:57:45.438148+0000","last_undegraded":"2023-06-10T18:57:45.438148+0000","last_fullsized":"2023-06-10T18:57:45.438148+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:37:06.322495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437799+0000","last_change":"2023-06-10T18:57:44.286143+0000","last_active":"2023-06-10T18:57:45.437799+0000","last_peered":"2023-06-10T18:57:45.437799+0000","last_clean":"2023-06-10T18:57:45.437799+0000","last_became_active":"2023-06-10T18:57:44.285996+0000","last_became_peered":"2023-06-10T18:57:44.285996+0000","last_unstale":"2023-06-10T18:57:45.437799+0000","last_undegraded":"2023-06-10T18:57:45.437799+0000","last_fullsized":"2023-06-10T18:57:45.437799+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:48:57.535434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269501+0000","last_change":"2023-06-10T18:57:43.873254+0000","last_active":"2023-06-10T18:57:45.269501+0000","last_peered":"2023-06-10T18:57:45.269501+0000","last_clean":"2023-06-10T18:57:45.269501+0000","last_became_active":"2023-06-10T18:57:43.872988+0000","last_became_peered":"2023-06-10T18:57:43.872988+0000","last_unstale":"2023-06-10T18:57:45.269501+0000","last_undegraded":"2023-06-10T18:57:45.269501+0000","last_fullsized":"2023-06-10T18:57:45.269501+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:16:47.882047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269497+0000","last_change":"2023-06-10T18:57:44.290520+0000","last_active":"2023-06-10T18:57:45.269497+0000","last_peered":"2023-06-10T18:57:45.269497+0000","last_clean":"2023-06-10T18:57:45.269497+0000","last_became_active":"2023-06-10T18:57:44.286486+0000","last_became_peered":"2023-06-10T18:57:44.286486+0000","last_unstale":"2023-06-10T18:57:45.269497+0000","last_undegraded":"2023-06-10T18:57:45.269497+0000","last_fullsized":"2023-06-10T18:57:45.269497+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:17:36.326210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437176+0000","last_change":"2023-06-10T18:57:43.875756+0000","last_active":"2023-06-10T18:57:45.437176+0000","last_peered":"2023-06-10T18:57:45.437176+0000","last_clean":"2023-06-10T18:57:45.437176+0000","last_became_active":"2023-06-10T18:57:43.875490+0000","last_became_peered":"2023-06-10T18:57:43.875490+0000","last_unstale":"2023-06-10T18:57:45.437176+0000","last_undegraded":"2023-06-10T18:57:45.437176+0000","last_fullsized":"2023-06-10T18:57:45.437176+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:40:15.451498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.268521+0000","last_change":"2023-06-10T18:57:44.283326+0000","last_active":"2023-06-10T18:57:45.268521+0000","last_peered":"2023-06-10T18:57:45.268521+0000","last_clean":"2023-06-10T18:57:45.268521+0000","last_became_active":"2023-06-10T18:57:44.283060+0000","last_became_peered":"2023-06-10T18:57:44.283060+0000","last_unstale":"2023-06-10T18:57:45.268521+0000","last_undegraded":"2023-06-10T18:57:45.268521+0000","last_fullsized":"2023-06-10T18:57:45.268521+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:46:49.078616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437276+0000","last_change":"2023-06-10T18:57:43.879111+0000","last_active":"2023-06-10T18:57:45.437276+0000","last_peered":"2023-06-10T18:57:45.437276+0000","last_clean":"2023-06-10T18:57:45.437276+0000","last_became_active":"2023-06-10T18:57:43.878821+0000","last_became_peered":"2023-06-10T18:57:43.878821+0000","last_unstale":"2023-06-10T18:57:45.437276+0000","last_undegraded":"2023-06-10T18:57:45.437276+0000","last_fullsized":"2023-06-10T18:57:45.437276+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:22:51.743471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270347+0000","last_change":"2023-06-10T18:57:44.284933+0000","last_active":"2023-06-10T18:57:45.270347+0000","last_peered":"2023-06-10T18:57:45.270347+0000","last_clean":"2023-06-10T18:57:45.270347+0000","last_became_active":"2023-06-10T18:57:44.284659+0000","last_became_peered":"2023-06-10T18:57:44.284659+0000","last_unstale":"2023-06-10T18:57:45.270347+0000","last_undegraded":"2023-06-10T18:57:45.270347+0000","last_fullsized":"2023-06-10T18:57:45.270347+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:05.333291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269303+0000","last_change":"2023-06-10T18:57:43.872121+0000","last_active":"2023-06-10T18:57:45.269303+0000","last_peered":"2023-06-10T18:57:45.269303+0000","last_clean":"2023-06-10T18:57:45.269303+0000","last_became_active":"2023-06-10T18:57:43.871705+0000","last_became_peered":"2023-06-10T18:57:43.871705+0000","last_unstale":"2023-06-10T18:57:45.269303+0000","last_undegraded":"2023-06-10T18:57:45.269303+0000","last_fullsized":"2023-06-10T18:57:45.269303+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:38:15.440292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270022+0000","last_change":"2023-06-10T18:57:44.285387+0000","last_active":"2023-06-10T18:57:45.270022+0000","last_peered":"2023-06-10T18:57:45.270022+0000","last_clean":"2023-06-10T18:57:45.270022+0000","last_became_active":"2023-06-10T18:57:44.285125+0000","last_became_peered":"2023-06-10T18:57:44.285125+0000","last_unstale":"2023-06-10T18:57:45.270022+0000","last_undegraded":"2023-06-10T18:57:45.270022+0000","last_fullsized":"2023-06-10T18:57:45.270022+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:20:46.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"158'8","reported_seq":21,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269179+0000","last_change":"2023-06-10T18:57:43.869514+0000","last_active":"2023-06-10T18:57:45.269179+0000","last_peered":"2023-06-10T18:57:45.269179+0000","last_clean":"2023-06-10T18:57:45.269179+0000","last_became_active":"2023-06-10T18:57:43.869224+0000","last_became_peered":"2023-06-10T18:57:43.869224+0000","last_unstale":"2023-06-10T18:57:45.269179+0000","last_undegraded":"2023-06-10T18:57:45.269179+0000","last_fullsized":"2023-06-10T18:57:45.269179+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:23:40.645828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437779+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:57:45.437779+0000","last_peered":"2023-06-10T18:57:45.437779+0000","last_clean":"2023-06-10T18:57:45.437779+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:57:45.437779+0000","last_undegraded":"2023-06-10T18:57:45.437779+0000","last_fullsized":"2023-06-10T18:57:45.437779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:49:51.818379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437640+0000","last_change":"2023-06-10T18:57:44.282760+0000","last_active":"2023-06-10T18:57:45.437640+0000","last_peered":"2023-06-10T18:57:45.437640+0000","last_clean":"2023-06-10T18:57:45.437640+0000","last_became_active":"2023-06-10T18:57:44.279860+0000","last_became_peered":"2023-06-10T18:57:44.279860+0000","last_unstale":"2023-06-10T18:57:45.437640+0000","last_undegraded":"2023-06-10T18:57:45.437640+0000","last_fullsized":"2023-06-10T18:57:45.437640+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:20:35.204715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437485+0000","last_change":"2023-06-10T18:57:43.881757+0000","last_active":"2023-06-10T18:57:45.437485+0000","last_peered":"2023-06-10T18:57:45.437485+0000","last_clean":"2023-06-10T18:57:45.437485+0000","last_became_active":"2023-06-10T18:57:43.881510+0000","last_became_peered":"2023-06-10T18:57:43.881510+0000","last_unstale":"2023-06-10T18:57:45.437485+0000","last_undegraded":"2023-06-10T18:57:45.437485+0000","last_fullsized":"2023-06-10T18:57:45.437485+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:05:06.543012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":289,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270159+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:57:45.270159+0000","last_peered":"2023-06-10T18:57:45.270159+0000","last_clean":"2023-06-10T18:57:45.270159+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:57:45.270159+0000","last_undegraded":"2023-06-10T18:57:45.270159+0000","last_fullsized":"2023-06-10T18:57:45.270159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:27:04.959211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438089+0000","last_change":"2023-06-10T18:57:44.280028+0000","last_active":"2023-06-10T18:57:45.438089+0000","last_peered":"2023-06-10T18:57:45.438089+0000","last_clean":"2023-06-10T18:57:45.438089+0000","last_became_active":"2023-06-10T18:57:44.279780+0000","last_became_peered":"2023-06-10T18:57:44.279780+0000","last_unstale":"2023-06-10T18:57:45.438089+0000","last_undegraded":"2023-06-10T18:57:45.438089+0000","last_fullsized":"2023-06-10T18:57:45.438089+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:00:00.235679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"158'3","reported_seq":16,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437126+0000","last_change":"2023-06-10T18:57:43.877140+0000","last_active":"2023-06-10T18:57:45.437126+0000","last_peered":"2023-06-10T18:57:45.437126+0000","last_clean":"2023-06-10T18:57:45.437126+0000","last_became_active":"2023-06-10T18:57:43.876617+0000","last_became_peered":"2023-06-10T18:57:43.876617+0000","last_unstale":"2023-06-10T18:57:45.437126+0000","last_undegraded":"2023-06-10T18:57:45.437126+0000","last_fullsized":"2023-06-10T18:57:45.437126+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:11.889919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437061+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:57:45.437061+0000","last_peered":"2023-06-10T18:57:45.437061+0000","last_clean":"2023-06-10T18:57:45.437061+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:57:45.437061+0000","last_undegraded":"2023-06-10T18:57:45.437061+0000","last_fullsized":"2023-06-10T18:57:45.437061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:43:00.582909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438230+0000","last_change":"2023-06-10T18:57:44.285385+0000","last_active":"2023-06-10T18:57:45.438230+0000","last_peered":"2023-06-10T18:57:45.438230+0000","last_clean":"2023-06-10T18:57:45.438230+0000","last_became_active":"2023-06-10T18:57:44.284732+0000","last_became_peered":"2023-06-10T18:57:44.284732+0000","last_unstale":"2023-06-10T18:57:45.438230+0000","last_undegraded":"2023-06-10T18:57:45.438230+0000","last_fullsized":"2023-06-10T18:57:45.438230+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:08:44.539633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270411+0000","last_change":"2023-06-10T18:57:43.874498+0000","last_active":"2023-06-10T18:57:45.270411+0000","last_peered":"2023-06-10T18:57:45.270411+0000","last_clean":"2023-06-10T18:57:45.270411+0000","last_became_active":"2023-06-10T18:57:43.874220+0000","last_became_peered":"2023-06-10T18:57:43.874220+0000","last_unstale":"2023-06-10T18:57:45.270411+0000","last_undegraded":"2023-06-10T18:57:45.270411+0000","last_fullsized":"2023-06-10T18:57:45.270411+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:57:42.190892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270742+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:57:45.270742+0000","last_peered":"2023-06-10T18:57:45.270742+0000","last_clean":"2023-06-10T18:57:45.270742+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:57:45.270742+0000","last_undegraded":"2023-06-10T18:57:45.270742+0000","last_fullsized":"2023-06-10T18:57:45.270742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:07:30.249993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":419,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437089+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:57:45.437089+0000","last_peered":"2023-06-10T18:57:45.437089+0000","last_clean":"2023-06-10T18:57:45.437089+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:57:45.437089+0000","last_undegraded":"2023-06-10T18:57:45.437089+0000","last_fullsized":"2023-06-10T18:57:45.437089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:52:58.704100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.270025+0000","last_change":"2023-06-10T18:57:44.294117+0000","last_active":"2023-06-10T18:57:45.270025+0000","last_peered":"2023-06-10T18:57:45.270025+0000","last_clean":"2023-06-10T18:57:45.270025+0000","last_became_active":"2023-06-10T18:57:44.293811+0000","last_became_peered":"2023-06-10T18:57:44.293811+0000","last_unstale":"2023-06-10T18:57:45.270025+0000","last_undegraded":"2023-06-10T18:57:45.270025+0000","last_fullsized":"2023-06-10T18:57:45.270025+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:45:58.571683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"158'6","reported_seq":19,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269698+0000","last_change":"2023-06-10T18:57:43.870392+0000","last_active":"2023-06-10T18:57:45.269698+0000","last_peered":"2023-06-10T18:57:45.269698+0000","last_clean":"2023-06-10T18:57:45.269698+0000","last_became_active":"2023-06-10T18:57:43.870112+0000","last_became_peered":"2023-06-10T18:57:43.870112+0000","last_unstale":"2023-06-10T18:57:45.269698+0000","last_undegraded":"2023-06-10T18:57:45.269698+0000","last_fullsized":"2023-06-10T18:57:45.269698+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:09:56.601006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":263,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.436995+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:57:45.436995+0000","last_peered":"2023-06-10T18:57:45.436995+0000","last_clean":"2023-06-10T18:57:45.436995+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:57:45.436995+0000","last_undegraded":"2023-06-10T18:57:45.436995+0000","last_fullsized":"2023-06-10T18:57:45.436995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:22:54.945540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437252+0000","last_change":"2023-06-10T18:57:44.280226+0000","last_active":"2023-06-10T18:57:45.437252+0000","last_peered":"2023-06-10T18:57:45.437252+0000","last_clean":"2023-06-10T18:57:45.437252+0000","last_became_active":"2023-06-10T18:57:44.279997+0000","last_became_peered":"2023-06-10T18:57:44.279997+0000","last_unstale":"2023-06-10T18:57:45.437252+0000","last_undegraded":"2023-06-10T18:57:45.437252+0000","last_fullsized":"2023-06-10T18:57:45.437252+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:28:25.425413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437791+0000","last_change":"2023-06-10T18:57:43.878016+0000","last_active":"2023-06-10T18:57:45.437791+0000","last_peered":"2023-06-10T18:57:45.437791+0000","last_clean":"2023-06-10T18:57:45.437791+0000","last_became_active":"2023-06-10T18:57:43.877732+0000","last_became_peered":"2023-06-10T18:57:43.877732+0000","last_unstale":"2023-06-10T18:57:45.437791+0000","last_undegraded":"2023-06-10T18:57:45.437791+0000","last_fullsized":"2023-06-10T18:57:45.437791+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:01.625663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437944+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:57:45.437944+0000","last_peered":"2023-06-10T18:57:45.437944+0000","last_clean":"2023-06-10T18:57:45.437944+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:57:45.437944+0000","last_undegraded":"2023-06-10T18:57:45.437944+0000","last_fullsized":"2023-06-10T18:57:45.437944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:16:26.239474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.269056+0000","last_change":"2023-06-10T18:57:44.286854+0000","last_active":"2023-06-10T18:57:45.269056+0000","last_peered":"2023-06-10T18:57:45.269056+0000","last_clean":"2023-06-10T18:57:45.269056+0000","last_became_active":"2023-06-10T18:57:44.286553+0000","last_became_peered":"2023-06-10T18:57:44.286553+0000","last_unstale":"2023-06-10T18:57:45.269056+0000","last_undegraded":"2023-06-10T18:57:45.269056+0000","last_fullsized":"2023-06-10T18:57:45.269056+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:38:24.475492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":13,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437907+0000","last_change":"2023-06-10T18:57:43.877173+0000","last_active":"2023-06-10T18:57:45.437907+0000","last_peered":"2023-06-10T18:57:45.437907+0000","last_clean":"2023-06-10T18:57:45.437907+0000","last_became_active":"2023-06-10T18:57:43.876840+0000","last_became_peered":"2023-06-10T18:57:43.876840+0000","last_unstale":"2023-06-10T18:57:45.437907+0000","last_undegraded":"2023-06-10T18:57:45.437907+0000","last_fullsized":"2023-06-10T18:57:45.437907+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:46:00.963751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.438098+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:57:45.438098+0000","last_peered":"2023-06-10T18:57:45.438098+0000","last_clean":"2023-06-10T18:57:45.438098+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:57:45.438098+0000","last_undegraded":"2023-06-10T18:57:45.438098+0000","last_fullsized":"2023-06-10T18:57:45.438098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:07:36.878181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437449+0000","last_change":"2023-06-10T18:57:44.287278+0000","last_active":"2023-06-10T18:57:45.437449+0000","last_peered":"2023-06-10T18:57:45.437449+0000","last_clean":"2023-06-10T18:57:45.437449+0000","last_became_active":"2023-06-10T18:57:44.286171+0000","last_became_peered":"2023-06-10T18:57:44.286171+0000","last_unstale":"2023-06-10T18:57:45.437449+0000","last_undegraded":"2023-06-10T18:57:45.437449+0000","last_fullsized":"2023-06-10T18:57:45.437449+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:43.847124+0000","last_clean_scrub_stamp":"2023-06-10T18:57:43.847124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:06:56.158866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"158'2","reported_seq":15,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437077+0000","last_change":"2023-06-10T18:57:43.874180+0000","last_active":"2023-06-10T18:57:45.437077+0000","last_peered":"2023-06-10T18:57:45.437077+0000","last_clean":"2023-06-10T18:57:45.437077+0000","last_became_active":"2023-06-10T18:57:43.873903+0000","last_became_peered":"2023-06-10T18:57:43.873903+0000","last_unstale":"2023-06-10T18:57:45.437077+0000","last_undegraded":"2023-06-10T18:57:45.437077+0000","last_fullsized":"2023-06-10T18:57:45.437077+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":156,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:57:42.841549+0000","last_clean_scrub_stamp":"2023-06-10T18:57:42.841549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:00:20.227568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-06-10T18:57:45.437540+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:57:45.437540+0000","last_peered":"2023-06-10T18:57:45.437540+0000","last_clean":"2023-06-10T18:57:45.437540+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:57:45.437540+0000","last_undegraded":"2023-06-10T18:57:45.437540+0000","last_fullsized":"2023-06-10T18:57:45.437540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:30:07.496989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":2548,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":2609152,"data_stored":1570117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.81499999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.442}]},{"osd":1,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.45200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.752,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.503,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76300000000000001,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.495}]},{"osd":2,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.39000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.69399999999999995,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.79000000000000004},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.52600000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.33300000000000002,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.89700000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.45600000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":0.77900000000000003,"15min":0.83799999999999997},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.34599999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.75900000000000001,"5min":0.80200000000000005,"15min":0.93899999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.56100000000000005,"15min":0.55300000000000005},"min":{"1min":0.42099999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.45900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1990656,"data_stored":976446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":0.51500000000000001},"min":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.628},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.29199999999999998},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.872,"15min":6.6689999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.252},"last":0.495}]},{"osd":3,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33300000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.373},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53600000000000003,"15min":0.54500000000000004},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.499,"15min":0.495},"min":{"1min":0.33300000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64300000000000002,"5min":0.874,"15min":0.874},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71999999999999997,"5min":0.746,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49299999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55200000000000005,"15min":0.53400000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67700000000000005,"5min":0.82299999999999995,"15min":0.92700000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28896,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342944,"statfs":{"total":96636764160,"available":96607174656,"internally_reserved":0,"allocated":1990656,"data_stored":976458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.308,"15min":0.32700000000000001},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.46000000000000002,"5min":0.51800000000000002,"15min":0.753},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34399999999999997,"15min":0.34899999999999998},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.498,"5min":0.51700000000000002,"15min":0.59599999999999997},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.33500000000000002,"15min":0.34300000000000003},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.48299999999999998,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.35199999999999998,"15min":0.375},"min":{"1min":0.24099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51600000000000001,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.318}]},{"osd":3,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.379,"15min":0.39700000000000002},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54800000000000004,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.378,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53200000000000003,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.34899999999999998},"max":{"1min":0.871,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59999999999999998,"15min":0.60399999999999998},"min":{"1min":0.372,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.626},"min":{"1min":0.46400000000000002,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.008},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.83399999999999996,"5min":1.206,"15min":1.206},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.873,"15min":0.88700000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:57:12 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45900000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":1.224,"15min":1.224},"last":0.499},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.61399999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80900000000000005,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.45100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33568,"kb_used_data":2520,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338272,"statfs":{"total":96636764160,"available":96602390528,"internally_reserved":0,"allocated":2580480,"data_stored":1566792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.20300000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.499,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.309},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.35199999999999998},"min":{"1min":0.24099999999999999,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.55300000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.435,"15min":0.39200000000000002},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.42499999999999999,"5min":11.439,"15min":11.439},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48099999999999998,"15min":0.41999999999999998},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.58699999999999997,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.379},"min":{"1min":0.252,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.47899999999999998,"5min":5.766,"15min":5.766},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.52300000000000002,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.37}]},{"osd":4,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60599999999999998,"15min":0.58399999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73399999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.71299999999999997,"5min":2.802,"15min":2.802},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81499999999999995,"5min":0.85399999999999998,"15min":0.999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78300000000000003,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.56299999999999994}]},{"osd":6,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82699999999999996,"5min":2.907,"15min":2.907},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.85499999999999998,"5min":0.97699999999999998,"15min":1.363},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:57:19 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.88400000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.626,"15min":0.61099999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76000000000000001,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.52100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28900,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342940,"statfs":{"total":96636764160,"available":96607170560,"internally_reserved":0,"allocated":1994752,"data_stored":977533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34999999999999998,"15min":0.34300000000000003},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.65900000000000003,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.42199999999999999,"15min":0.38700000000000001},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":6.4450000000000003,"15min":6.4450000000000003},"last":0.23699999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.38500000000000001,"15min":0.376},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57599999999999996,"5min":2.847,"15min":2.847},"last":0.373},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42899999999999999,"15min":0.39900000000000002},"min":{"1min":0.253,"5min":0.251,"15min":0.23799999999999999},"max":{"1min":0.85099999999999998,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.29099999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.43099999999999999,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.80200000000000005,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.312},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42399999999999999,"15min":0.41399999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.24,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63600000000000001,"15min":0.61599999999999999},"min":{"1min":0.40600000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.208,"5min":5.742,"15min":5.742},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.62},"min":{"1min":0.433,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":5.6429999999999998,"15min":5.6429999999999998},"last":0.39500000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.68300000000000005,"15min":0.64600000000000002},"min":{"1min":0.38600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.86799999999999999,"5min":6.593,"15min":6.593},"last":0.433},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.48599999999999999,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":1.002,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.443}]},{"osd":6,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.66900000000000004,"15min":0.63400000000000001},"min":{"1min":0.45100000000000001,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":8.7850000000000001,"15min":8.7850000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64900000000000002,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.171,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71099999999999997,"15min":0.66500000000000004},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.84099999999999997,"5min":6.617,"15min":6.617},"last":0.499},{"interface":"front","average":{"1min":0.622,"5min":0.70599999999999996,"15min":0.65800000000000003},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":0.97699999999999998,"5min":8.7149999999999999,"15min":8.7149999999999999},"last":0.40600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575018,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1994752,"data_stored":977477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.34799999999999998},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.19400000000000001},"max":{"1min":0.53900000000000003,"5min":0.56799999999999995,"15min":1.2170000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.37},"min":{"1min":0.246,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.59599999999999997,"15min":1.8600000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35099999999999998,"15min":0.36699999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.50700000000000001,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.374},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.38,"15min":0.40200000000000002},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.68600000000000005,"15min":1.3},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.40100000000000002},"min":{"1min":0.29599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57599999999999996,"5min":0.82999999999999996,"15min":1.109},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.41799999999999998,"15min":0.435},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.623,"5min":1.5960000000000001,"15min":1.599},"last":0.55400000000000005}]},{"osd":4,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78700000000000003,"5min":1.5049999999999999,"15min":1.925},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0720000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1100000000000001,"5min":1.613,"15min":1.613},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.095,"5min":1.4410000000000001,"15min":3.802},"last":0.74299999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.63800000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.5289999999999999,"15min":3.8180000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":1.018,"5min":1.679,"15min":1.679},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:57:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.625,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0549999999999999,"5min":1.47,"15min":1.47},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.628,"15min":0.628},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80900000000000005,"5min":1.548,"15min":1.8939999999999999},"last":0.623}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28908,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342932,"statfs":{"total":96636764160,"available":96607162368,"internally_reserved":0,"allocated":2002944,"data_stored":978058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.745,"15min":1.119},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.53400000000000003},"min":{"1min":0.379,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.63,"5min":0.71599999999999997,"15min":1.1639999999999999},"last":0.77500000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52400000000000002,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.77800000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.68899999999999995,"5min":0.873,"15min":0.96599999999999997},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.66700000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33200000000000002},"max":{"1min":0.57899999999999996,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.75}]},{"osd":3,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.52500000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.68500000000000005,"5min":0.70699999999999996,"15min":0.79800000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.70899999999999996,"5min":0.85299999999999998,"15min":1.042},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.5,"15min":0.49099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.72099999999999997,"5min":0.94599999999999995,"15min":1.069},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.502,"15min":0.503},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72599999999999998,"5min":0.755,"15min":0.88300000000000001},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71099999999999997,"5min":0.90400000000000003,"15min":2.1579999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.53800000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.70699999999999996,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76200000000000001,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.97699999999999998,"15min":1.1459999999999999},"last":0.82399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28904,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94342936,"statfs":{"total":96636764160,"available":96607166464,"internally_reserved":0,"allocated":1998848,"data_stored":978047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29199999999999998},"max":{"1min":0.72799999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.69599999999999995,"5min":0.72499999999999998,"15min":0.81000000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.70799999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.622}]},{"osd":2,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.71799999999999997,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.79700000000000004},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77600000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68700000000000006,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.34999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69499999999999995,"5min":0.70899999999999996,"15min":0.83999999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.51300000000000001},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.64800000000000002,"5min":0.75900000000000001,"15min":0.79200000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.879},"last":0.64900000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.54500000000000004},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.88400000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59599999999999997,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:57:48.752 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:57:48.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:57:48.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.874+0000 7f55eae4f700 1 Processor -- start 2023-06-10T18:57:48.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.874+0000 7f55eae4f700 1 -- start start 2023-06-10T18:57:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55eae4f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f55e40ccff0 0x7f55e40ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55eae4f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40ce340 con 0x7f55e40d0d20 2023-06-10T18:57:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55eae4f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40d00d0 con 0x7f55e406af80 2023-06-10T18:57:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55eae4f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e40cd470 con 0x7f55e40ccff0 2023-06-10T18:57:48.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55e93ec700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f55e40ccff0 0x7f55e40ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55e3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f55e40d0d20 0x7f55e40ce7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37186/0 (socket says 172.21.15.104:37186) 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.875+0000 7f55e3fff700 1 -- 172.21.15.104:0/345870873 learned_addr learned my addr 172.21.15.104:0/345870873 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e93ec700 1 -- 172.21.15.104:0/345870873 >> v1:172.21.15.114:6789/0 conn(0x7f55e406af80 legacy=0x7f55e406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e93ec700 1 -- 172.21.15.104:0/345870873 >> v1:172.21.15.104:6789/0 conn(0x7f55e40d0d20 legacy=0x7f55e40ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e93ec700 1 -- 172.21.15.104:0/345870873 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e40cd680 con 0x7f55e40ccff0 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e37fe700 1 -- 172.21.15.104:0/345870873 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120281292 0 0) 0x7f55e40ce340 con 0x7f55e40d0d20 2023-06-10T18:57:48.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e93ec700 1 --2- 172.21.15.104:0/345870873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40ccff0 0x7f55e40ceee0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009d90 tx=0x7f55dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8550d674671dadd2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e37fe700 1 -- 172.21.15.104:0/345870873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00e070 con 0x7f55e40ccff0 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e37fe700 1 -- 172.21.15.104:0/345870873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc00bbd0 con 0x7f55e40ccff0 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55e37fe700 1 -- 172.21.15.104:0/345870873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc012430 con 0x7f55e40ccff0 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55eae4f700 1 -- 172.21.15.104:0/345870873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40ccff0 msgr2=0x7f55e40ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55eae4f700 1 --2- 172.21.15.104:0/345870873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40ccff0 0x7f55e40ceee0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009d90 tx=0x7f55dc00b330 comp rx=0 tx=0).stop 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.876+0000 7f55eae4f700 1 -- 172.21.15.104:0/345870873 shutdown_connections 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.877+0000 7f55eae4f700 1 --2- 172.21.15.104:0/345870873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40ccff0 0x7f55e40ceee0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.877+0000 7f55eae4f700 1 -- 172.21.15.104:0/345870873 >> 172.21.15.104:0/345870873 conn(0x7f55e4089fd0 msgr2=0x7f55e408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:48.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.877+0000 7f55eae4f700 1 -- 172.21.15.104:0/345870873 shutdown_connections 2023-06-10T18:57:48.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.877+0000 7f55eae4f700 1 -- 172.21.15.104:0/345870873 wait complete. 2023-06-10T18:57:48.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.878+0000 7f55eae4f700 1 Processor -- start 2023-06-10T18:57:48.886 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.878+0000 7f55eae4f700 1 -- start start 2023-06-10T18:57:48.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55e406af80 0x7f55e40b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55e40ccff0 0x7f55e4157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e4158860 con 0x7f55e40d0d20 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e40b4840 con 0x7f55e406af80 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e40b4b20 con 0x7f55e40ccff0 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39368/0 (socket says 172.21.15.104:39368) 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 -- 172.21.15.104:0/1975323708 learned_addr learned my addr 172.21.15.104:0/1975323708 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e3fff700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55e40ccff0 0x7f55e4157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e8beb700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55e406af80 0x7f55e40b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 -- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55e40ccff0 msgr2=0x7f55e4157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55e40ccff0 0x7f55e4157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 -- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55e406af80 msgr2=0x7f55e40b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:48.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55e406af80 0x7f55e40b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 -- 172.21.15.104:0/1975323708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc009a70 con 0x7f55e40d0d20 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e93ec700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f55dc014040 tx=0x7f55dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc01b810 con 0x7f55e40d0d20 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc020070 con 0x7f55e40d0d20 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e40b4e00 con 0x7f55e40d0d20 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc01bbd0 con 0x7f55e40d0d20 2023-06-10T18:57:48.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.879+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e40b44f0 con 0x7f55e40d0d20 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.880+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e4158cf0 con 0x7f55e40d0d20 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.881+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f55dc01bd30 con 0x7f55e40d0d20 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.881+0000 7f55e17fa700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 0x7f55c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.881+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f55dc022070 con 0x7f55e40d0d20 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.882+0000 7f55e8beb700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 0x7f55c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:48.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.882+0000 7f55e8beb700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 0x7f55c806d190 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f55d8007fd0 tx=0x7f55d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:48.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:48.886+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f55e4158cf0 con 0x7f55e40d0d20 2023-06-10T18:57:49.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.149+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f55e4158cf0 con 0x7f55e40d0d20 2023-06-10T18:57:49.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.154+0000 7f55e17fa700 1 -- 172.21.15.104:0/1975323708 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+0 (secure 0 0 0) 0x7f55e4158cf0 con 0x7f55e40d0d20 2023-06-10T18:57:49.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 msgr2=0x7f55c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 0x7f55c806d190 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f55d8007fd0 tx=0x7f55d8009040 comp rx=0 tx=0).stop 2023-06-10T18:57:49.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 msgr2=0x7f55e40b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f55dc014040 tx=0x7f55dc004ad0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 shutdown_connections 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f55c806ace0 0x7f55c806d190 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55e406af80 0x7f55e40b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55e40d0d20 0x7f55e40b58d0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 --2- 172.21.15.104:0/1975323708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55e40ccff0 0x7f55e4157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 >> 172.21.15.104:0/1975323708 conn(0x7f55e4089fd0 msgr2=0x7f55e40ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:49.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 shutdown_connections 2023-06-10T18:57:49.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.156+0000 7f55eae4f700 1 -- 172.21.15.104:0/1975323708 wait complete. 2023-06-10T18:57:49.163 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:57:49.173 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:49.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.287+0000 7f892cc10700 1 Processor -- start 2023-06-10T18:57:49.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.287+0000 7f892cc10700 1 -- start start 2023-06-10T18:57:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892cc10700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f89280cf100 0x7f892806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892cc10700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892806d290 con 0x7f892806e0e0 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892cc10700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89280ca2d0 con 0x7f892806e4b0 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892cc10700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8928071640 con 0x7f89280cf100 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f892806e0e0 0x7f89280c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37200/0 (socket says 172.21.15.104:37200) 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.288+0000 7f892659c700 1 -- 172.21.15.104:0/2122444860 learned_addr learned my addr 172.21.15.104:0/2122444860 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2122444860 >> v2:172.21.15.114:3300/0 conn(0x7f89280cf100 0x7f892806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945450875 0 0) 0x7f892806d290 con 0x7f892806e0e0 2023-06-10T18:57:49.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89140035b0 con 0x7f892806e0e0 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565387638 0 0) 0x7f89140035b0 con 0x7f892806e0e0 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 >> v2:172.21.15.114:3300/0 conn(0x7f89280cf100 msgr2=0x7f892806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 --2- 172.21.15.104:0/2122444860 >> v2:172.21.15.114:3300/0 conn(0x7f89280cf100 0x7f892806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 >> v1:172.21.15.114:6789/0 conn(0x7f892806e4b0 legacy=0x7f89280ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892806d4c0 con 0x7f892806e0e0 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f89100017f0 con 0x7f892806e0e0 2023-06-10T18:57:49.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 >> v1:172.21.15.104:6789/0 conn(0x7f892806e0e0 legacy=0x7f89280c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914003a00 0x7f8914003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8914004850 0x7f8914004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 0x7f891400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8928071640 con 0x7f8914005200 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891400c110 con 0x7f8914004850 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891400c2d0 con 0x7f8914003a00 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 0x7f891400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8925d9b700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8914004850 0x7f8914004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f892659c700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914003a00 0x7f8914003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 -- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914003a00 msgr2=0x7f8914003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914003a00 0x7f8914003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 -- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8914004850 msgr2=0x7f8914004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8914004850 0x7f8914004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 -- 172.21.15.104:0/2122444860 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89140035b0 con 0x7f8914005200 2023-06-10T18:57:49.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.289+0000 7f8926d9d700 1 -- 172.21.15.104:0/2122444860 reap_dead start 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 0x7f891400ba10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8920009f90 tx=0x7f8914005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b581a4bd4ec95ea6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920003a20 con 0x7f8914005200 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892000f040 con 0x7f8914005200 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892559a700 1 -- 172.21.15.104:0/2122444860 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920003da0 con 0x7f8914005200 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 -- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 msgr2=0x7f891400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 0x7f891400ba10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8920009f90 tx=0x7f8914005880 comp rx=0 tx=0).stop 2023-06-10T18:57:49.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 -- 172.21.15.104:0/2122444860 shutdown_connections 2023-06-10T18:57:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 --2- 172.21.15.104:0/2122444860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8914005200 0x7f891400ba10 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 -- 172.21.15.104:0/2122444860 >> 172.21.15.104:0/2122444860 conn(0x7f892806b8a0 msgr2=0x7f89280cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 -- 172.21.15.104:0/2122444860 shutdown_connections 2023-06-10T18:57:49.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.290+0000 7f892cc10700 1 -- 172.21.15.104:0/2122444860 wait complete. 2023-06-10T18:57:49.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 Processor -- start 2023-06-10T18:57:49.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 -- start start 2023-06-10T18:57:49.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 0x7f89280c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89280b4d40 0x7f8928156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b54b0 con 0x7f8928156b80 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b25d0 con 0x7f89280b4d40 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f892cc10700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b2870 con 0x7f8914005200 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f8926d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.292+0000 7f8926d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39388/0 (socket says 172.21.15.104:39388) 2023-06-10T18:57:49.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 -- 172.21.15.104:0/2082701899 learned_addr learned my addr 172.21.15.104:0/2082701899 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8925d9b700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89280b4d40 0x7f8928156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f892659c700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 0x7f89280c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 -- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 msgr2=0x7f89280c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 0x7f89280c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 -- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89280b4d40 msgr2=0x7f8928156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89280b4d40 0x7f8928156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 -- 172.21.15.104:0/2082701899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280b2b50 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f892659c700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 0x7f89280c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f8926d9d700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8928072080 tx=0x7f8920004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920003e10 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920004230 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89200045e0 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280b2e30 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.293+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8928154b60 con 0x7f8928156b80 2023-06-10T18:57:49.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.294+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89280b3020 con 0x7f8928156b80 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.294+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8920004740 con 0x7f8928156b80 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.295+0000 7f891effd700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 0x7f890806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.295+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f892008aba0 con 0x7f8928156b80 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.295+0000 7f892659c700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 0x7f890806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.296+0000 7f892659c700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 0x7f890806d190 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f89100011d0 tx=0x7f8910009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:49.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.299+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8920055ba0 con 0x7f8928156b80 2023-06-10T18:57:49.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.528+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8928074050 con 0x7f8928156b80 2023-06-10T18:57:49.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.529+0000 7f891effd700 1 -- 172.21.15.104:0/2082701899 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f8920018070 con 0x7f8928156b80 2023-06-10T18:57:49.535 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:49.535 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:57:49.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 msgr2=0x7f890806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 0x7f890806d190 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f89100011d0 tx=0x7f8910009040 comp rx=0 tx=0).stop 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 msgr2=0x7f89280b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f8928072080 tx=0x7f8920004b40 comp rx=0 tx=0).stop 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 shutdown_connections 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8928156b80 0x7f89280b3350 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8914005200 0x7f89280c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f890806ace0 0x7f890806d190 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 --2- 172.21.15.104:0/2082701899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89280b4d40 0x7f8928156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 >> 172.21.15.104:0/2082701899 conn(0x7f892806b8a0 msgr2=0x7f89280cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 shutdown_connections 2023-06-10T18:57:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:49.531+0000 7f892cc10700 1 -- 172.21.15.104:0/2082701899 wait complete. 2023-06-10T18:57:49.538 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:57:49.844 INFO:tasks.ceph.mds.a.smithi104.stderr:2023-06-10T18:57:49.839+0000 7f415be21700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:57:49.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:57:44.258356+0000', 'modified': '2023-06-10T18:57:47.275932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10107}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10107': {'gid': 10107, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6834/2386386628', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 2386386628}, {'type': 'v1', 'addr': '172.21.15.104:6834', 'nonce': 2386386628}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:57:49.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:57:49.956 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-06-10T18:57:50.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.072+0000 7f57268dc700 1 Processor -- start 2023-06-10T18:57:50.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 -- start start 2023-06-10T18:57:50.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f572006cf70 0x7f57200ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f57200ac700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572006e770 con 0x7f572015d820 2023-06-10T18:57:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572006ae40 con 0x7f572006cf70 2023-06-10T18:57:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.073+0000 7f57268dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200ca910 con 0x7f572006d4c0 2023-06-10T18:57:50.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f572015d820 0x7f57200cb690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37220/0 (socket says 172.21.15.104:37220) 2023-06-10T18:57:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 -- 172.21.15.104:0/2122441137 learned_addr learned my addr 172.21.15.104:0/2122441137 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590429846 0 0) 0x7f572006e770 con 0x7f572015d820 2023-06-10T18:57:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57080035b0 con 0x7f572015d820 2023-06-10T18:57:50.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571f7fe700 1 --2- 172.21.15.104:0/2122441137 >> v2:172.21.15.114:3300/0 conn(0x7f572006cf70 0x7f57200ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f5724e79700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f57200ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354591747 0 0) 0x7f57080035b0 con 0x7f572015d820 2023-06-10T18:57:50.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 msgr2=0x7f57200ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f57200ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 >> v2:172.21.15.114:3300/0 conn(0x7f572006cf70 msgr2=0x7f57200ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 --2- 172.21.15.104:0/2122441137 >> v2:172.21.15.114:3300/0 conn(0x7f572006cf70 0x7f57200ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200cbda0 con 0x7f572015d820 2023-06-10T18:57:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f57100017f0 con 0x7f572015d820 2023-06-10T18:57:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571f7fe700 1 --2- 172.21.15.104:0/2122441137 >> v2:172.21.15.114:3300/0 conn(0x7f572006cf70 0x7f57200ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 >> v1:172.21.15.104:6789/0 conn(0x7f572015d820 legacy=0x7f57200cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5708003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f5708004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5708004d90 0x7f5708005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572006ae40 con 0x7f5708003a00 2023-06-10T18:57:50.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200ca910 con 0x7f5708004d90 2023-06-10T18:57:50.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570800baf0 con 0x7f572006d4c0 2023-06-10T18:57:50.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5708003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f5724e79700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5708004d90 0x7f5708005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571f7fe700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f5708004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 -- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 msgr2=0x7f5708004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f572006d4c0 0x7f5708004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 -- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5708004d90 msgr2=0x7f5708005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5708004d90 0x7f5708005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.074+0000 7f571ffff700 1 -- 172.21.15.104:0/2122441137 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57080035b0 con 0x7f5708003a00 2023-06-10T18:57:50.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f571ffff700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5708003e10 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f571000c320 tx=0x7f5708005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da5b2d4dc6910a2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000ce60 con 0x7f5708003a00 2023-06-10T18:57:50.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710007bf0 con 0x7f5708003a00 2023-06-10T18:57:50.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f571effd700 1 -- 172.21.15.104:0/2122441137 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710007310 con 0x7f5708003a00 2023-06-10T18:57:50.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 -- 172.21.15.104:0/2122441137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 msgr2=0x7f5708003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 --2- 172.21.15.104:0/2122441137 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5708003e10 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f571000c320 tx=0x7f5708005880 comp rx=0 tx=0).stop 2023-06-10T18:57:50.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f5724e79700 1 -- 172.21.15.104:0/2122441137 reap_dead start 2023-06-10T18:57:50.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 -- 172.21.15.104:0/2122441137 shutdown_connections 2023-06-10T18:57:50.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 -- 172.21.15.104:0/2122441137 >> 172.21.15.104:0/2122441137 conn(0x7f572006b8a0 msgr2=0x7f57200cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 -- 172.21.15.104:0/2122441137 shutdown_connections 2023-06-10T18:57:50.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.075+0000 7f57268dc700 1 -- 172.21.15.104:0/2122441137 wait complete. 2023-06-10T18:57:50.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.076+0000 7f57268dc700 1 Processor -- start 2023-06-10T18:57:50.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 -- start start 2023-06-10T18:57:50.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57200c8e00 0x7f57200cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 0x7f57200cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57201530f0 con 0x7f5708003a00 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200c44f0 con 0x7f57200c8e00 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f57268dc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5720151df0 con 0x7f57200c9270 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39404/0 (socket says 172.21.15.104:39404) 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f571ffff700 1 -- 172.21.15.104:0/1924277832 learned_addr learned my addr 172.21.15.104:0/1924277832 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.077+0000 7f571f7fe700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57200c8e00 0x7f57200cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f5724e79700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 0x7f57200cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 -- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 msgr2=0x7f57200cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 0x7f57200cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 -- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57200c8e00 msgr2=0x7f57200cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57200c8e00 0x7f57200cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 -- 172.21.15.104:0/1924277832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57201520d0 con 0x7f5708003a00 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f5724e79700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 0x7f57200cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:50.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571ffff700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f5710000ef0 tx=0x7f5710001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571cff9700 1 -- 172.21.15.104:0/1924277832 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100072d0 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571cff9700 1 -- 172.21.15.104:0/1924277832 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710007430 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f571cff9700 1 -- 172.21.15.104:0/1924277832 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100077c0 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57201530f0 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.078+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5720152690 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.079+0000 7f571cff9700 1 -- 172.21.15.104:0/1924277832 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f571001e430 con 0x7f5708003a00 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.080+0000 7f571cff9700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 0x7f570c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.080+0000 7f571cff9700 1 -- 172.21.15.104:0/1924277832 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f571008aa90 con 0x7f5708003a00 2023-06-10T18:57:50.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.080+0000 7f571f7fe700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 0x7f570c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.080+0000 7f571f7fe700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 0x7f570c06d190 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5714006f10 tx=0x7f5714008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.082+0000 7f5732a61b80 1 Processor -- start 2023-06-10T18:57:50.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.082+0000 7f5732a61b80 1 -- start start 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf30f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 0x55cbdf313440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf1edc60 con 0x55cbdf2e8f50 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf2409f0 con 0x55cbdf2e6740 2023-06-10T18:57:50.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf239e80 con 0x55cbdf310d80 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39412/0 (socket says 172.21.15.104:39412) 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 -- 172.21.15.104:0/876369081 learned_addr learned my addr 172.21.15.104:0/876369081 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571f7fe700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf30f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5724e79700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 0x55cbdf313440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 -- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 msgr2=0x55cbdf30f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf30f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 -- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 msgr2=0x55cbdf313440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 0x55cbdf313440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 -- 172.21.15.104:0/876369081 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbdf0bb5e0 con 0x55cbdf2e8f50 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5724e79700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 0x55cbdf313440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f571ffff700 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f5710061570 tx=0x7f57100590b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bff280a693fb360 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5705ffb700 1 -- 172.21.15.104:0/876369081 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710061aa0 con 0x55cbdf2e8f50 2023-06-10T18:57:50.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5705ffb700 1 -- 172.21.15.104:0/876369081 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571008a040 con 0x55cbdf2e8f50 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.083+0000 7f5705ffb700 1 -- 172.21.15.104:0/876369081 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710057af0 con 0x55cbdf2e8f50 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 -- 172.21.15.104:0/876369081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 msgr2=0x55cbdf2e9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f5710061570 tx=0x7f57100590b0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 -- 172.21.15.104:0/876369081 shutdown_connections 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e6740 0x55cbdf313440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e9320 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 --2- 172.21.15.104:0/876369081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf30f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 -- 172.21.15.104:0/876369081 >> 172.21.15.104:0/876369081 conn(0x55cbdf2ddfc0 msgr2=0x55cbdf32e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 -- 172.21.15.104:0/876369081 shutdown_connections 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.084+0000 7f5732a61b80 1 -- 172.21.15.104:0/876369081 wait complete. 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.085+0000 7f5732a61b80 1 -- start start 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.085+0000 7f5732a61b80 1 Processor -- start 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf32a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf310d80 0x55cbdf32bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf1d9a70 con 0x55cbdf2e8f50 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf1d9770 con 0x55cbdf310d80 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf0bd8d0 con 0x55cbdf2e6740 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 10 client.0 _ll_register_callbacks cb 0x55cbdf1df210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39420/0 (socket says 172.21.15.104:39420) 2023-06-10T18:57:50.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 -- 172.21.15.104:0/595149847 learned_addr learned my addr 172.21.15.104:0/595149847 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571ffff700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf32a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5724e79700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf310d80 0x55cbdf32bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 -- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 msgr2=0x55cbdf32a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf32a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 -- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf310d80 msgr2=0x55cbdf32bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf310d80 0x55cbdf32bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 -- 172.21.15.104:0/595149847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5720152690 con 0x55cbdf2e8f50 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571ffff700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf32a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f571f7fe700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x55cbdf2e5010 tx=0x7f57140047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b400f4ac057d838c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f56feffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714016cd0 con 0x55cbdf2e8f50 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5714016e30 con 0x55cbdf2e8f50 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cbdf2db780 con 0x55cbdf2e8f50 2023-06-10T18:57:50.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5714005a20 con 0x55cbdf2e8f50 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 10 client.10266 fetch_fsmap learned FSMap version 102 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.086+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55cbdf067d90 con 0x55cbdf2e8f50 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cbdf2db780 con 0x55cbdf2e8f50 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 102) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5714005b80 con 0x55cbdf2e8f50 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 10 client.10266 fetch_fsmap finished waiting for FSMap version 102 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 10 client.10266 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 20 client.10266 populate_metadata read hostname 'smithi104' 2023-06-10T18:57:50.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf3298d0 0x55cbdf329ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 4 client.10266 mds_command: new command op to 10107 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 --> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cbdf232e60 con 0x55cbdf3298d0 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f571ffff700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf3298d0 0x55cbdf329ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f56feffd700 10 client.10266 ms_handle_connect on v2:172.21.15.104:6832/2386386628 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.087+0000 7f571ffff700 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf3298d0 0x55cbdf329ca0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27bea19709cba6b7 server_cookie=1ee5f1781908d7ae in_seq=0 out_seq=0 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f56feffd700 1 -- 172.21.15.104:0/595149847 <== mds.0 v2:172.21.15.104:6832/2386386628 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cbdf232e60 con 0x55cbdf3298d0 2023-06-10T18:57:50.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f56feffd700 10 client.10266 handle_command_reply: tid=0 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 client.10266 shutdown 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 msgr2=0x55cbdf32f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x55cbdf2e5010 tx=0x7f57140047c0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 shutdown_connections 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf3298d0 0x55cbdf329ca0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf310d80 0x55cbdf32bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f56feffd700 0 client.10266 ms_handle_reset on v2:172.21.15.104:6832/2386386628 2023-06-10T18:57:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf32a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 --2- 172.21.15.104:0/595149847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e8f50 0x55cbdf32f690 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 >> 172.21.15.104:0/595149847 conn(0x55cbdf2ddfc0 msgr2=0x55cbdf30fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 shutdown_connections 2023-06-10T18:57:50.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 1 -- 172.21.15.104:0/595149847 wait complete. 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.088+0000 7f5732a61b80 20 client.10266 trim_cache size 0 max 16384 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.106+0000 7f5732a61b80 1 Processor -- start 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.106+0000 7f5732a61b80 1 -- start start 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf326cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf2ddb60 con 0x55cbdf2e6740 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf2c5440 con 0x55cbdf2e8f50 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf310260 con 0x55cbdf310d80 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39426/0 (socket says 172.21.15.104:39426) 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 -- 172.21.15.104:0/2087409043 learned_addr learned my addr 172.21.15.104:0/2087409043 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5724e79700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf326cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571f7fe700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 -- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 msgr2=0x55cbdf326cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf326cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 -- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 msgr2=0x55cbdf2e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 -- 172.21.15.104:0/2087409043 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbdf2ea750 con 0x55cbdf2e6740 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f5724e79700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf326cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571f7fe700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f571ffff700 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f5710088e00 tx=0x7f5710055bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a70262fa7f70da7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f57067fc700 1 -- 172.21.15.104:0/2087409043 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710087040 con 0x55cbdf2e6740 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f57067fc700 1 -- 172.21.15.104:0/2087409043 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710057af0 con 0x55cbdf2e6740 2023-06-10T18:57:50.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.107+0000 7f57067fc700 1 -- 172.21.15.104:0/2087409043 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710077620 con 0x55cbdf2e6740 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 -- 172.21.15.104:0/2087409043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 msgr2=0x55cbdf2e55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f5710088e00 tx=0x7f5710055bc0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 -- 172.21.15.104:0/2087409043 shutdown_connections 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf310d80 0x55cbdf326cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf2e6740 0x55cbdf2e55b0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 --2- 172.21.15.104:0/2087409043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf2e5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 -- 172.21.15.104:0/2087409043 >> 172.21.15.104:0/2087409043 conn(0x55cbdf2ddfc0 msgr2=0x55cbdf32afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 -- 172.21.15.104:0/2087409043 shutdown_connections 2023-06-10T18:57:50.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.108+0000 7f5732a61b80 1 -- 172.21.15.104:0/2087409043 wait complete. 2023-06-10T18:57:50.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.109+0000 7f5732a61b80 1 -- start start 2023-06-10T18:57:50.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.109+0000 7f5732a61b80 1 Processor -- start 2023-06-10T18:57:50.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf323b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf324070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf34b870 con 0x55cbdf310d80 2023-06-10T18:57:50.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf349dc0 con 0x55cbdf2e8f50 2023-06-10T18:57:50.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cbdf2e5d70 con 0x55cbdf2e6740 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 10 client.0 _ll_register_callbacks cb 0x7f57140137d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39436/0 (socket says 172.21.15.104:39436) 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 -- 172.21.15.104:0/1954997083 learned_addr learned my addr 172.21.15.104:0/1954997083 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f571ffff700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf323b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f571f7fe700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf324070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 -- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 msgr2=0x55cbdf323b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf323b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 -- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 msgr2=0x55cbdf324070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf324070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 -- 172.21.15.104:0/1954997083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cbdf0bb5e0 con 0x55cbdf310d80 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5724e79700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f5720070c70 tx=0x7f5718007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1519b7aac26befba server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f56feffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:57:50.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718017620 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5718017780 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f571008a040 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5718017940 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 10 client.10272 fetch_fsmap learned FSMap version 102 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.110+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55cbdf050c10 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571008a040 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 102) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5718017aa0 con 0x55cbdf310d80 2023-06-10T18:57:50.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 10 client.10272 fetch_fsmap finished waiting for FSMap version 102 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 10 client.10272 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 20 client.10272 populate_metadata read hostname 'smithi104' 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf327200 0x55cbdf3275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 4 client.10272 mds_command: new command op to 10107 tid=0[{"prefix": "status"}] 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 --> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cbdf1f5000 con 0x55cbdf327200 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f571ffff700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf327200 0x55cbdf3275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f56feffd700 10 client.10272 ms_handle_connect on v2:172.21.15.104:6832/2386386628 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f571ffff700 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf327200 0x55cbdf3275d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dd8ebd0a1607daf server_cookie=83fec720078d4230 in_seq=0 out_seq=0 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f56feffd700 1 -- 172.21.15.104:0/1954997083 <== mds.0 v2:172.21.15.104:6832/2386386628 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55cbdf1f5000 con 0x55cbdf327200 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f56feffd700 10 client.10272 handle_command_reply: tid=0 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 client.10272 shutdown 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 msgr2=0x55cbdf3245b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f5720070c70 tx=0x7f5718007bb0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 shutdown_connections 2023-06-10T18:57:50.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.111+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:6832/2386386628,v1:172.21.15.104:6834/2386386628] conn(0x55cbdf327200 0x55cbdf3275d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cbdf310d80 0x55cbdf3245b0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f56feffd700 0 client.10272 ms_handle_reset on v2:172.21.15.104:6832/2386386628 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55cbdf2e6740 0x55cbdf323b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 --2- 172.21.15.104:0/1954997083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cbdf2e8f50 0x55cbdf324070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 >> 172.21.15.104:0/1954997083 conn(0x55cbdf2ddfc0 msgr2=0x55cbdf32afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 shutdown_connections 2023-06-10T18:57:50.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 1 -- 172.21.15.104:0/1954997083 wait complete. 2023-06-10T18:57:50.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f5732a61b80 20 client.10272 trim_cache size 0 max 16384 2023-06-10T18:57:50.142 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:57:50.142 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10107, 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8517674619999998, 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 102, 2023-06-10T18:57:50.143 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 159, 2023-06-10T18:57:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 159, 2023-06-10T18:57:50.144 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.210185295 2023-06-10T18:57:50.144 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:50.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 msgr2=0x7f570c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 0x7f570c06d190 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5714006f10 tx=0x7f5714008040 comp rx=0 tx=0).stop 2023-06-10T18:57:50.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 msgr2=0x7f5720152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f5710000ef0 tx=0x7f5710001bf0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 shutdown_connections 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f57200c9270 0x7f57200cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5708003a00 0x7f5720152bb0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f570c06ace0 0x7f570c06d190 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 --2- 172.21.15.104:0/1924277832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57200c8e00 0x7f57200cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 >> 172.21.15.104:0/1924277832 conn(0x7f572006b8a0 msgr2=0x7f57200b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 shutdown_connections 2023-06-10T18:57:50.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.112+0000 7f57268dc700 1 -- 172.21.15.104:0/1924277832 wait complete. 2023-06-10T18:57:50.147 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:57:50.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.261+0000 7ff66236d700 1 Processor -- start 2023-06-10T18:57:50.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.261+0000 7ff66236d700 1 -- start start 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66236d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff65c06e680 0x7ff65c0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66236d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c06a840 con 0x7ff65c0ae310 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66236d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0aeab0 con 0x7ff65c15c2d0 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66236d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65c069f10 con 0x7ff65c06e680 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff65b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff65c06e680 0x7ff65c0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66090a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff65c0ae310 0x7ff65c0ce210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37272/0 (socket says 172.21.15.104:37272) 2023-06-10T18:57:50.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.262+0000 7ff66090a700 1 -- 172.21.15.104:0/1592975377 learned_addr learned my addr 172.21.15.104:0/1592975377 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65b7fe700 1 -- 172.21.15.104:0/1592975377 >> v1:172.21.15.114:6789/0 conn(0x7ff65c15c2d0 legacy=0x7ff65c06df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65b7fe700 1 -- 172.21.15.104:0/1592975377 >> v1:172.21.15.104:6789/0 conn(0x7ff65c0ae310 legacy=0x7ff65c0ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65b7fe700 1 -- 172.21.15.104:0/1592975377 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c06a1f0 con 0x7ff65c06e680 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65affd700 1 -- 172.21.15.104:0/1592975377 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058021971 0 0) 0x7ff65c06a840 con 0x7ff65c0ae310 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65b7fe700 1 --2- 172.21.15.104:0/1592975377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0b1ad0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7ff650009840 tx=0x7ff650009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f37c6ead0ea8513 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65affd700 1 -- 172.21.15.104:0/1592975377 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650004230 con 0x7ff65c06e680 2023-06-10T18:57:50.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65affd700 1 -- 172.21.15.104:0/1592975377 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff650004390 con 0x7ff65c06e680 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff65affd700 1 -- 172.21.15.104:0/1592975377 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6500036a0 con 0x7ff65c06e680 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 -- 172.21.15.104:0/1592975377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 msgr2=0x7ff65c0b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 --2- 172.21.15.104:0/1592975377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0b1ad0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7ff650009840 tx=0x7ff650009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 -- 172.21.15.104:0/1592975377 shutdown_connections 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 --2- 172.21.15.104:0/1592975377 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0b1ad0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 -- 172.21.15.104:0/1592975377 >> 172.21.15.104:0/1592975377 conn(0x7ff65c06b8a0 msgr2=0x7ff65c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 -- 172.21.15.104:0/1592975377 shutdown_connections 2023-06-10T18:57:50.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.263+0000 7ff66236d700 1 -- 172.21.15.104:0/1592975377 wait complete. 2023-06-10T18:57:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 Processor -- start 2023-06-10T18:57:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 -- start start 2023-06-10T18:57:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 0x7ff65c155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 0x7ff65c15ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c0c75c0 con 0x7ff65c06e680 2023-06-10T18:57:50.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c0c78a0 con 0x7ff65c15c2d0 2023-06-10T18:57:50.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.265+0000 7ff66236d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c0c7b80 con 0x7ff65c0ae310 2023-06-10T18:57:50.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39446/0 (socket says 172.21.15.104:39446) 2023-06-10T18:57:50.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 -- 172.21.15.104:0/233984464 learned_addr learned my addr 172.21.15.104:0/233984464 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:50.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65b7fe700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 0x7ff65c155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff66090a700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 0x7ff65c15ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 -- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 msgr2=0x7ff65c155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 0x7ff65c155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 -- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 msgr2=0x7ff65c15ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 0x7ff65c15ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 -- 172.21.15.104:0/233984464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6500094a0 con 0x7ff65c06e680 2023-06-10T18:57:50.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65b7fe700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 0x7ff65c155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:50.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff66090a700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 0x7ff65c15ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:57:50.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff65bfff700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00c520 tx=0x7ff64c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0054e0 con 0x7ff65c06e680 2023-06-10T18:57:50.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c0c7e60 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c15b040 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c0036d0 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.266+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c003a40 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.267+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c0052f0 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.268+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff64c003ba0 con 0x7ff65c06e680 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.268+0000 7ff658ff9700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 0x7ff64806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:50.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.268+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff64c088ab0 con 0x7ff65c06e680 2023-06-10T18:57:50.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.268+0000 7ff65b7fe700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 0x7ff64806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:50.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.269+0000 7ff65b7fe700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 0x7ff64806d190 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7ff65c06ea90 tx=0x7ff650008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:50.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.272+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff64c053a30 con 0x7ff65c06e680 2023-06-10T18:57:50.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.503+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff63c005870 con 0x7ff65c06e680 2023-06-10T18:57:50.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.504+0000 7ff658ff9700 1 -- 172.21.15.104:0/233984464 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v102) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff64c057050 con 0x7ff65c06e680 2023-06-10T18:57:50.509 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:57:50.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 msgr2=0x7ff64806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 0x7ff64806d190 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7ff65c06ea90 tx=0x7ff650008030 comp rx=0 tx=0).stop 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 msgr2=0x7ff65c0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00c520 tx=0x7ff64c00c8e0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 shutdown_connections 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64806ace0 0x7ff64806d190 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff65c15c2d0 0x7ff65c15ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff65c06e680 0x7ff65c0c6fe0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 --2- 172.21.15.104:0/233984464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff65c0ae310 0x7ff65c155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:50.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 >> 172.21.15.104:0/233984464 conn(0x7ff65c06b8a0 msgr2=0x7ff65c0c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:50.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 shutdown_connections 2023-06-10T18:57:50.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:50.506+0000 7ff66236d700 1 -- 172.21.15.104:0/233984464 wait complete. 2023-06-10T18:57:50.523 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:57:50.523 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:57:50.524 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:57:50.524 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:57:50.524 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:50.525 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:57:50.525 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:57:50.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:57:50.525 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:50.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.526 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:57:50.543 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:57:50.543 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:50.544 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:50.545 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41614sec preferred_lft 41614sec 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.546 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:50.547 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:57:50.548 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:57:50.548 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:50.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.548 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:57:50.609 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:57:50.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.611 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:57:50.669 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:57:50.669 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:57:50.669 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:57:50.669 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:57:50.669 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:57:50.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.670 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:57:50.671 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:57:50.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:57:50.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:57:50.671 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:57:50.672 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:57:50.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.773 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:57:50.790 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:57:50.790 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:57:50.791 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:50.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:50.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:57:50.857 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:57:50.857 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:50.857 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:50.914 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:57:50.915 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:50.915 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:50.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:57:51.037 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:51.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:51.102 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:51.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:51.104 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:57:51.160 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:57:51.161 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:57:51.203 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:57:51.230 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:51.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:51.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:57:51.285+0000 7f5cf880a5c0 -1 init, newargv = 0x556dfe4068e0 newargc=15 2023-06-10T18:57:51.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[156171]: starting ceph client 2023-06-10T18:57:51.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[156171]: starting fuse 2023-06-10T18:57:51.296 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:51.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:51.298 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:57:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:57:51.355 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:57:51.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:51.356 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:57:51.357 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:57:51.357 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:57:51.357 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:57:51.357 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:51.358 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:57:51.358 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:57:51.358 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:57:51.359 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:57:51.359 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:51.359 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:57:51.359 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:57:51.360 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:57:51.360 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:57:51.360 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:51.361 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:57:51.361 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:57:51.361 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:57:51.362 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:57:51.362 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:57:51.362 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:57:51.363 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:57:51.363 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:57:51.363 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:51.363 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:57:51.364 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:57:51.477 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.156171.asok 2023-06-10T18:57:51.660 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.156171.asok 2023-06-10T18:57:51.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:51.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.156171.asok status 2023-06-10T18:57:51.818 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:51.818 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:57:51.818 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:57:51.818 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "156171", 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10290, 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10290 2023-06-10T18:57:51.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3682236906 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:57:51.821 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3682236906 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10290 192.168.0.1:0/3682236906", 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3682236906", 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 102, 2023-06-10T18:57:51.822 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 159, 2023-06-10T18:57:51.823 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:57:51.823 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:57:51.823 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:57:51.823 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:51.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:51.829 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:57:51.829 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:51.846 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:57:51.846 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:51.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:51.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:57:51.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:57:52.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 Processor -- start 2023-06-10T18:57:52.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 -- start start 2023-06-10T18:57:52.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25e80cec20 0x7f25e815d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e806a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e806a810 con 0x7f25e8156740 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e815cdf0 con 0x7f25e80cec20 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.069+0000 7f25ed894700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80cf0a0 con 0x7f25e8069e70 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f25e80cec20 0x7f25e815d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f25e8156740 0x7f25e80cd250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37296/0 (socket says 172.21.15.104:37296) 2023-06-10T18:57:52.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e6ffd700 1 -- 172.21.15.104:0/1491839654 learned_addr learned my addr 172.21.15.104:0/1491839654 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e77fe700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e806a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 -- 172.21.15.104:0/1491839654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 msgr2=0x7f25e806a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e806a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 -- 172.21.15.104:0/1491839654 >> v1:172.21.15.104:6789/0 conn(0x7f25e8156740 legacy=0x7f25e80cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 -- 172.21.15.104:0/1491839654 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e80cf380 con 0x7f25e80cec20 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e5ffb700 1 -- 172.21.15.104:0/1491839654 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504352514 0 0) 0x7f25e806a810 con 0x7f25e8156740 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e67fc700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e80cec20 0x7f25e815d280 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009840 tx=0x7f25d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f48e3317754b49c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:52.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e5ffb700 1 -- 172.21.15.104:0/1491839654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8011830 con 0x7f25e80cec20 2023-06-10T18:57:52.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e5ffb700 1 -- 172.21.15.104:0/1491839654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8011990 con 0x7f25e80cec20 2023-06-10T18:57:52.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.070+0000 7f25e5ffb700 1 -- 172.21.15.104:0/1491839654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8011d20 con 0x7f25e80cec20 2023-06-10T18:57:52.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 -- 172.21.15.104:0/1491839654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e80cec20 msgr2=0x7f25e815d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e80cec20 0x7f25e815d280 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009840 tx=0x7f25d8009b50 comp rx=0 tx=0).stop 2023-06-10T18:57:52.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 -- 172.21.15.104:0/1491839654 shutdown_connections 2023-06-10T18:57:52.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e806a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 --2- 172.21.15.104:0/1491839654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e80cec20 0x7f25e815d280 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 -- 172.21.15.104:0/1491839654 >> 172.21.15.104:0/1491839654 conn(0x7f25e806d240 msgr2=0x7f25e806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 -- 172.21.15.104:0/1491839654 shutdown_connections 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.071+0000 7f25ed894700 1 -- 172.21.15.104:0/1491839654 wait complete. 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.072+0000 7f25ed894700 1 Processor -- start 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.072+0000 7f25ed894700 1 -- start start 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e8159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25e8156740 0x7f25e8143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8155040 con 0x7f25e8143bd0 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80c5f90 con 0x7f25e8156740 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25ed894700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e815a0d0 con 0x7f25e8069e70 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39478/0 (socket says 172.21.15.104:39478) 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 -- 172.21.15.104:0/1912044188 learned_addr learned my addr 172.21.15.104:0/1912044188 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:52.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e6ffd700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e8159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 -- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 msgr2=0x7f25e8159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e67fc700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25e8156740 0x7f25e8143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e8159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 -- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25e8156740 msgr2=0x7f25e8143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25e8156740 0x7f25e8143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 -- 172.21.15.104:0/1912044188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80094a0 con 0x7f25e8143bd0 2023-06-10T18:57:52.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25e77fe700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f25e0007e70 tx=0x7f25e000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000ee50 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.073+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0014420 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.074+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e81417d0 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.074+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e00147d0 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.074+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e80c5cb0 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.074+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e8141630 con 0x7f25e8143bd0 2023-06-10T18:57:52.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.075+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f25e0014930 con 0x7f25e8143bd0 2023-06-10T18:57:52.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.075+0000 7f25cffff700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 0x7f25d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:52.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.075+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f25e0089d80 con 0x7f25e8143bd0 2023-06-10T18:57:52.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.076+0000 7f25e6ffd700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 0x7f25d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:52.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.076+0000 7f25e6ffd700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 0x7f25d406d190 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f25dc000f60 tx=0x7f25dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:52.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.080+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f25e8141630 con 0x7f25e8143bd0 2023-06-10T18:57:52.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.309+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f25e8141630 con 0x7f25e8143bd0 2023-06-10T18:57:52.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.310+0000 7f25cffff700 1 -- 172.21.15.104:0/1912044188 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v102) v1 ==== 53+0+73 (secure 0 0 0) 0x7f25e8141630 con 0x7f25e8143bd0 2023-06-10T18:57:52.315 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:57:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 msgr2=0x7f25d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 0x7f25d406d190 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f25dc000f60 tx=0x7f25dc009040 comp rx=0 tx=0).stop 2023-06-10T18:57:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 msgr2=0x7f25e8144040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:52.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f25e0007e70 tx=0x7f25e000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 shutdown_connections 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25e8143bd0 0x7f25e8144040 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25e8069e70 0x7f25e8159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f25d406ace0 0x7f25d406d190 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 --2- 172.21.15.104:0/1912044188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25e8156740 0x7f25e8143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 >> 172.21.15.104:0/1912044188 conn(0x7f25e806d240 msgr2=0x7f25e815a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:52.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 shutdown_connections 2023-06-10T18:57:52.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:52.312+0000 7f25ed894700 1 -- 172.21.15.104:0/1912044188 wait complete. 2023-06-10T18:57:52.329 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:57:52.329 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:57:52.329 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:57:52.330 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:57:52.330 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:52.330 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:57:52.330 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:57:52.331 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:57:52.331 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:52.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.332 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:57:52.353 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:57:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:57:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:57:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:52.354 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41608sec preferred_lft 41608sec 2023-06-10T18:57:52.355 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:57:52.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:57:52.357 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:57:52.358 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:57:52.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.358 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:57:52.423 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:57:52.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.426 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:57:52.487 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:57:52.487 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:57:52.487 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:57:52.488 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:57:52.488 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:57:52.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.489 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:57:52.489 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:57:52.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:57:52.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:57:52.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:57:52.490 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:57:52.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.597 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:57:52.614 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:57:52.614 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:57:52.615 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:52.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:57:52.682 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:57:52.682 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:52.683 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:52.739 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:57:52.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:57:52.740 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:52.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:57:52.861 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:52.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:52.931 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:52.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:52.932 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:57:52.992 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:57:52.992 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:57:53.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:57:53.059 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:57:53.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:57:53.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:57:53.115+0000 7fd2c39385c0 -1 init, newargv = 0x55f149559b20 newargc=15 2023-06-10T18:57:53.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[116776]: starting ceph client 2023-06-10T18:57:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:57:53.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[116776]: starting fuse 2023-06-10T18:57:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:57:53.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:53.129 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:57:53.187 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:57:53.187 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:57:53.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:53.189 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:57:53.190 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:57:53.191 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:57:53.192 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:57:53.192 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:53.193 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:57:53.194 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:57:53.194 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:57:53.194 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:57:53.195 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:53.196 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:57:53.196 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:57:53.197 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:57:53.197 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:57:53.199 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:53.199 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:57:53.199 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:57:53.200 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:57:53.200 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:57:53.200 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:57:53.200 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:57:53.201 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:57:53.201 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:57:53.201 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:57:53.202 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:57:53.202 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:53.308 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.116776.asok 2023-06-10T18:57:53.491 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.116776.asok 2023-06-10T18:57:53.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:53.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.116776.asok status 2023-06-10T18:57:53.647 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:57:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:57:53.647 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "116776", 2023-06-10T18:57:53.648 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 8701, 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:57:53.649 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 8701 2023-06-10T18:57:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:57:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:57:53.650 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2191687823 2023-06-10T18:57:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:57:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:57:53.651 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2191687823 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.8701 192.168.0.1:0/2191687823", 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/2191687823", 2023-06-10T18:57:53.652 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 102, 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 159, 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:57:53.653 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:57:53.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:53.657 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:57:53.657 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:53.674 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:57:53.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:53.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:53.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:57:53.741 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:57:53.888 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:57:53.888 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:57:53.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:57:54.024 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:57:54.025 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:57:54.026 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q fuse --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:57:54.048 INFO:teuthology.orchestra.run.smithi104.stdout:2.9.7-15.el8 2023-06-10T18:57:54.048 INFO:teuthology.packaging:The installed version of fuse is 2.9.7-15.el8 2023-06-10T18:57:54.049 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q fuse --qf '%{VERSION}-%{RELEASE}' 2023-06-10T18:57:54.070 INFO:teuthology.orchestra.run.smithi114.stdout:2.9.7-15.el8 2023-06-10T18:57:54.071 INFO:teuthology.packaging:The installed version of fuse is 2.9.7-15.el8 2023-06-10T18:57:54.071 INFO:tasks.cephfs.test_client_recovery:flock locks are available 2023-06-10T18:57:54.072 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:57:54.072 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:54.104 INFO:tasks.cephfs.mount:lock_background file background_file 2023-06-10T18:57:54.105 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:54.105 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:57:54.106 DEBUG:teuthology.orchestra.run.smithi104:> import fcntl 2023-06-10T18:57:54.106 DEBUG:teuthology.orchestra.run.smithi104:> import struct 2023-06-10T18:57:54.106 DEBUG:teuthology.orchestra.run.smithi104:> f1 = open("/home/ubuntu/cephtest/mnt.0/background_file-1", '"'"'w'"'"') 2023-06-10T18:57:54.106 DEBUG:teuthology.orchestra.run.smithi104:> fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB) 2023-06-10T18:57:54.107 DEBUG:teuthology.orchestra.run.smithi104:> f2 = open("/home/ubuntu/cephtest/mnt.0/background_file-2", '"'"'w'"'"') 2023-06-10T18:57:54.107 DEBUG:teuthology.orchestra.run.smithi104:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:54.107 DEBUG:teuthology.orchestra.run.smithi104:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:54.107 DEBUG:teuthology.orchestra.run.smithi104:> while True: 2023-06-10T18:57:54.108 DEBUG:teuthology.orchestra.run.smithi104:> time.sleep(1) 2023-06-10T18:57:54.108 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:57:54.147 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:57:54.164 INFO:teuthology.orchestra.run.smithi114.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.1/background_file-2': No such file or directory 2023-06-10T18:57:55.165 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.1/background_file-2 2023-06-10T18:57:55.195 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 4194304 regular empty file 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 2fh/47d Inode: 1099511627777 Links: 1 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0664/-rw-rw-r--) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:fusefs_t:s0 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-06-10 18:57:54.211693623 +0000 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-06-10 18:57:54.211693623 +0000 2023-06-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-06-10 18:57:54.211693623 +0000 2023-06-10T18:57:55.197 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-06-10T18:57:55.197 INFO:tasks.cephfs.mount:File background_file-2 became visible from 1 after 1s 2023-06-10T18:57:55.197 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:55.198 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:55.257 INFO:tasks.cephfs.mount:check lock on file background_file 2023-06-10T18:57:55.258 DEBUG:teuthology.orchestra.run.smithi114:> python3 -c ' 2023-06-10T18:57:55.258 DEBUG:teuthology.orchestra.run.smithi114:> import fcntl 2023-06-10T18:57:55.258 DEBUG:teuthology.orchestra.run.smithi114:> import errno 2023-06-10T18:57:55.258 DEBUG:teuthology.orchestra.run.smithi114:> import struct 2023-06-10T18:57:55.259 DEBUG:teuthology.orchestra.run.smithi114:> f1 = open("/home/ubuntu/cephtest/mnt.1/background_file-1", '"'"'r'"'"') 2023-06-10T18:57:55.259 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:55.259 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB) 2023-06-10T18:57:55.260 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:55.260 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:55.260 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:55.260 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:55.261 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("flock on file /home/ubuntu/cephtest/mnt.1/background_file-1 not found") 2023-06-10T18:57:55.261 DEBUG:teuthology.orchestra.run.smithi114:> f2 = open("/home/ubuntu/cephtest/mnt.1/background_file-2", '"'"'r'"'"') 2023-06-10T18:57:55.261 DEBUG:teuthology.orchestra.run.smithi114:> try: 2023-06-10T18:57:55.262 DEBUG:teuthology.orchestra.run.smithi114:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:55.262 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:55.262 DEBUG:teuthology.orchestra.run.smithi114:> except IOError as e: 2023-06-10T18:57:55.262 DEBUG:teuthology.orchestra.run.smithi114:> if e.errno == errno.EAGAIN: 2023-06-10T18:57:55.263 DEBUG:teuthology.orchestra.run.smithi114:> pass 2023-06-10T18:57:55.263 DEBUG:teuthology.orchestra.run.smithi114:> else: 2023-06-10T18:57:55.263 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("posix lock on file /home/ubuntu/cephtest/mnt.1/background_file-2 not found") 2023-06-10T18:57:55.263 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:55.349 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:57:55.349 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:57:55.368 INFO:tasks.cephfs.mount:lock_and_release file background_file 2023-06-10T18:57:55.369 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:55.369 DEBUG:teuthology.orchestra.run.smithi114:> import time 2023-06-10T18:57:55.369 DEBUG:teuthology.orchestra.run.smithi114:> import fcntl 2023-06-10T18:57:55.370 DEBUG:teuthology.orchestra.run.smithi114:> import struct 2023-06-10T18:57:55.370 DEBUG:teuthology.orchestra.run.smithi114:> f1 = open("/home/ubuntu/cephtest/mnt.1/background_file-1", '"'"'w'"'"') 2023-06-10T18:57:55.370 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.flock(f1, fcntl.LOCK_EX) 2023-06-10T18:57:55.370 DEBUG:teuthology.orchestra.run.smithi114:> f2 = open("/home/ubuntu/cephtest/mnt.1/background_file-2", '"'"'w'"'"') 2023-06-10T18:57:55.371 DEBUG:teuthology.orchestra.run.smithi114:> lockdata = struct.pack('"'"'hhllhh'"'"', fcntl.F_WRLCK, 0, 0, 0, 0, 0) 2023-06-10T18:57:55.371 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.fcntl(f2, fcntl.F_SETLK, lockdata) 2023-06-10T18:57:55.371 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:57:57.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:57:57.615 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:57:57.615 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:57:57.615 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:57:57.616 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:57:57.616 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:57.617 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:57:57.617 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:57:57.618 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:57:57.618 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:57:57.619 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:57.619 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:57:57.620 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:57:57.620 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:57:57.621 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:57:57.621 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:57.622 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:57:57.622 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:57:57.623 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:57:57.623 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:57:57.624 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:57:57.624 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:57:57.625 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:57:57.625 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:57:57.626 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:57:57.626 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:57:57.627 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:57:57.720 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.156171.asok 2023-06-10T18:57:57.902 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.156171.asok 2023-06-10T18:57:57.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:57:57.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.156171.asok mds_sessions 2023-06-10T18:57:58.064 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:57:58.064 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10290, 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10290 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:57:58.065 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3682236906 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10290 v1:192.168.0.1:0/3682236906", 2023-06-10T18:57:58.066 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/3682236906", 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:58.067 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6832", 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2386386628 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6834", 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 2386386628 2023-06-10T18:57:58.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 40, 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:58:52.287082+0000", 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:57:52.287082+0000", 2023-06-10T18:57:58.069 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 3, 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 102 2023-06-10T18:57:58.070 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:57:58.074 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:57:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.188+0000 7f6e73120700 1 Processor -- start 2023-06-10T18:57:58.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.188+0000 7f6e73120700 1 -- start start 2023-06-10T18:57:58.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e73120700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e73120700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e73120700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c0aed70 con 0x7f6e6c0ade00 2023-06-10T18:57:58.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e73120700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0cf430 con 0x7f6e6c0ada30 2023-06-10T18:57:58.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e73120700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0cee70 con 0x7f6e6c0ae930 2023-06-10T18:57:58.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e70ebc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e70ebc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39482/0 (socket says 172.21.15.104:39482) 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.189+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2460686399 learned_addr learned my addr 172.21.15.104:0/2460686399 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e716bd700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2460686399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 msgr2=0x7f6e6c0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e70ebc700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2460686399 >> v1:172.21.15.114:6789/0 conn(0x7f6e6c0ade00 legacy=0x7f6e6c0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2460686399 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0cf150 con 0x7f6e6c0ada30 2023-06-10T18:57:58.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e70ebc700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006fd0 tx=0x7f6e60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ff7fa8b02f6d94 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e6b7fe700 1 -- 172.21.15.104:0/2460686399 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60005e30 con 0x7f6e6c0ada30 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e6b7fe700 1 -- 172.21.15.104:0/2460686399 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60004030 con 0x7f6e6c0ada30 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e6b7fe700 1 -- 172.21.15.104:0/2460686399 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e600043e0 con 0x7f6e6c0ada30 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e73120700 1 -- 172.21.15.104:0/2460686399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 msgr2=0x7f6e6c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.190+0000 7f6e73120700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006fd0 tx=0x7f6e60009bd0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 -- 172.21.15.104:0/2460686399 shutdown_connections 2023-06-10T18:57:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c0ae3f0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 --2- 172.21.15.104:0/2460686399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 -- 172.21.15.104:0/2460686399 >> 172.21.15.104:0/2460686399 conn(0x7f6e6c06b8a0 msgr2=0x7f6e6c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 -- 172.21.15.104:0/2460686399 shutdown_connections 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.191+0000 7f6e73120700 1 -- 172.21.15.104:0/2460686399 wait complete. 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.192+0000 7f6e73120700 1 Processor -- start 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.192+0000 7f6e73120700 1 -- start start 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 0x7f6e6c0b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c14e170 con 0x7f6e6c0ada30 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c099340 con 0x7f6e6c0b1a00 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c099590 con 0x7f6e6c0ae930 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39488/0 (socket says 172.21.15.104:39488) 2023-06-10T18:57:58.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2154465678 learned_addr learned my addr 172.21.15.104:0/2154465678 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e716bd700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 0x7f6e6c0b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 msgr2=0x7f6e6c0aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e6bfff700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 msgr2=0x7f6e6c0b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 0x7f6e6c0b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 -- 172.21.15.104:0/2154465678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60009500 con 0x7f6e6c0ada30 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e716bd700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 0x7f6e6c0b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e70ebc700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009130 tx=0x7f6e60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60003d90 con 0x7f6e6c0ada30 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60011420 con 0x7f6e6c0ada30 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c1e4040 con 0x7f6e6c0ada30 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e600117d0 con 0x7f6e6c0ada30 2023-06-10T18:57:58.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.193+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c1e4600 con 0x7f6e6c0ada30 2023-06-10T18:57:58.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.194+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c074050 con 0x7f6e6c0ada30 2023-06-10T18:57:58.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.195+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6e60011930 con 0x7f6e6c0ada30 2023-06-10T18:57:58.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.195+0000 7f6e697fa700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 0x7f6e5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:57:58.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.195+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6e6008a7b0 con 0x7f6e6c0ada30 2023-06-10T18:57:58.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.195+0000 7f6e6bfff700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 0x7f6e5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:57:58.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.196+0000 7f6e6bfff700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 0x7f6e5006d190 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c06a920 tx=0x7f6e58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:57:58.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.200+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6e600557b0 con 0x7f6e6c0ada30 2023-06-10T18:57:58.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.430+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6c148280 con 0x7f6e6c0ada30 2023-06-10T18:57:58.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.430+0000 7f6e697fa700 1 -- 172.21.15.104:0/2154465678 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f6e60088e60 con 0x7f6e6c0ada30 2023-06-10T18:57:58.435 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:57:58.436 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:57:58.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 msgr2=0x7f6e5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 0x7f6e5006d190 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c06a920 tx=0x7f6e58009040 comp rx=0 tx=0).stop 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 msgr2=0x7f6e6c14dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009130 tx=0x7f6e60004750 comp rx=0 tx=0).stop 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 shutdown_connections 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6e5006ace0 0x7f6e5006d190 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e6c0b1a00 0x7f6e6c0b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e6c0ada30 0x7f6e6c14dc30 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 --2- 172.21.15.104:0/2154465678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e6c0ae930 0x7f6e6c0aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 >> 172.21.15.104:0/2154465678 conn(0x7f6e6c06b8a0 msgr2=0x7f6e6c0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 shutdown_connections 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:57:58.432+0000 7f6e73120700 1 -- 172.21.15.104:0/2154465678 wait complete. 2023-06-10T18:57:58.439 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:57:58.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session evict 10290 2023-06-10T18:57:58.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:57:58.721+0000 7f5cd5ffb700 -1 client.10290 I was blocklisted at osd epoch 160 2023-06-10T18:57:58.749 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2023-06-10T18:57:58.850 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=10 retry_count=0)... 2023-06-10T18:58:03.852 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2023-06-10T18:58:03.863 INFO:tasks.cephfs.mount:Killing connection on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:58:03.864 INFO:tasks.cephfs.mount:Suspending the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:03.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:03.865 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 down 2023-06-10T18:58:03.947 INFO:tasks.cephfs.mount:Cleaning up killed connection on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:58:03.948 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:03.948 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:04.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:04.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.170+0000 7fd92b8b0700 1 Processor -- start 2023-06-10T18:58:04.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd92b8b0700 1 -- start start 2023-06-10T18:58:04.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd92b8b0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd92408a3d0 0x7fd924167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd92b8b0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240cadd0 con 0x7fd9240ce350 2023-06-10T18:58:04.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd92b8b0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240cb0b0 con 0x7fd92406b1f0 2023-06-10T18:58:04.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd92b8b0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9240cfeb0 con 0x7fd92408a3d0 2023-06-10T18:58:04.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd928e4b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd9240ce350 0x7fd9240cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37332/0 (socket says 172.21.15.104:37332) 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.171+0000 7fd928e4b700 1 -- 172.21.15.104:0/279935814 learned_addr learned my addr 172.21.15.104:0/279935814 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd929e4d700 1 --2- 172.21.15.104:0/279935814 >> v2:172.21.15.114:3300/0 conn(0x7fd92408a3d0 0x7fd924167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539972635 0 0) 0x7fd9240cadd0 con 0x7fd9240ce350 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9100035b0 con 0x7fd9240ce350 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119110039 0 0) 0x7fd9100035b0 con 0x7fd9240ce350 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 >> v2:172.21.15.114:3300/0 conn(0x7fd92408a3d0 msgr2=0x7fd924167940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 --2- 172.21.15.104:0/279935814 >> v2:172.21.15.114:3300/0 conn(0x7fd92408a3d0 0x7fd924167940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 >> v1:172.21.15.114:6789/0 conn(0x7fd92406b1f0 legacy=0x7fd9240cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240cdeb0 con 0x7fd9240ce350 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd920001670 con 0x7fd9240ce350 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 >> v1:172.21.15.104:6789/0 conn(0x7fd9240ce350 legacy=0x7fd9240cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd910003a00 0x7fd910003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 0x7fd910004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910005200 0x7fd91000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240cfeb0 con 0x7fd910004850 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91000c110 con 0x7fd910003a00 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91000c2d0 con 0x7fd910005200 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 0x7fd910004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd92964c700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd910003a00 0x7fd910003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd929e4d700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910005200 0x7fd91000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 -- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910005200 msgr2=0x7fd91000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910005200 0x7fd91000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 -- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd910003a00 msgr2=0x7fd910003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd910003a00 0x7fd910003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd928e4b700 1 -- 172.21.15.104:0/279935814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100035b0 con 0x7fd910004850 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.172+0000 7fd929e4d700 1 -- 172.21.15.104:0/279935814 reap_dead start 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd928e4b700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 0x7fd910004cc0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd92000c210 tx=0x7fd910005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a89db0017271f8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920007c10 con 0x7fd910004850 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000cdb0 con 0x7fd910004850 2023-06-10T18:58:04.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd91bfff700 1 -- 172.21.15.104:0/279935814 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920005ba0 con 0x7fd910004850 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 -- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 msgr2=0x7fd910004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 0x7fd910004cc0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd92000c210 tx=0x7fd910005880 comp rx=0 tx=0).stop 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 -- 172.21.15.104:0/279935814 shutdown_connections 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/279935814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd910004850 0x7fd910004cc0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 -- 172.21.15.104:0/279935814 >> 172.21.15.104:0/279935814 conn(0x7fd924089fd0 msgr2=0x7fd9240d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 -- 172.21.15.104:0/279935814 shutdown_connections 2023-06-10T18:58:04.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.173+0000 7fd92b8b0700 1 -- 172.21.15.104:0/279935814 wait complete. 2023-06-10T18:58:04.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 Processor -- start 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 -- start start 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910004850 0x7fd924151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd92409a480 0x7fd9240b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240c6450 con 0x7fd92409a8f0 2023-06-10T18:58:04.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240c6730 con 0x7fd92409a480 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92b8b0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240c6a10 con 0x7fd910004850 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd929e4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd929e4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39518/0 (socket says 172.21.15.104:39518) 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd929e4d700 1 -- 172.21.15.104:0/2524997477 learned_addr learned my addr 172.21.15.104:0/2524997477 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd928e4b700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd92409a480 0x7fd9240b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.175+0000 7fd92964c700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910004850 0x7fd924151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 -- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910004850 msgr2=0x7fd924151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910004850 0x7fd924151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 -- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd92409a480 msgr2=0x7fd9240b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd92409a480 0x7fd9240b4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 -- 172.21.15.104:0/2524997477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240c6cf0 con 0x7fd92409a8f0 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd929e4d700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fd92408ad60 tx=0x7fd91c0111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c00e070 con 0x7fd92409a8f0 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c003cf0 con 0x7fd92409a8f0 2023-06-10T18:58:04.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0117a0 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9240c6fd0 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.176+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9240c74f0 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.177+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd924074050 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.178+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd91c011900 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.178+0000 7fd919ffb700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 0x7fd90806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.178+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd91c089990 con 0x7fd92409a8f0 2023-06-10T18:58:04.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.178+0000 7fd92964c700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 0x7fd90806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.179+0000 7fd92964c700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 0x7fd90806d190 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd9240c53c0 tx=0x7fd914009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.182+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd91c0548d0 con 0x7fd92409a8f0 2023-06-10T18:58:04.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.369+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd924071640 con 0x7fd92409a8f0 2023-06-10T18:58:04.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.370+0000 7fd919ffb700 1 -- 172.21.15.104:0/2524997477 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7fd91c057ef0 con 0x7fd92409a8f0 2023-06-10T18:58:04.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:04.376 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 msgr2=0x7fd90806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 0x7fd90806d190 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd9240c53c0 tx=0x7fd914009040 comp rx=0 tx=0).stop 2023-06-10T18:58:04.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 msgr2=0x7fd9240b54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fd92408ad60 tx=0x7fd91c0111d0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 shutdown_connections 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd92409a8f0 0x7fd9240b54e0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd910004850 0x7fd924151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd90806ace0 0x7fd90806d190 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 --2- 172.21.15.104:0/2524997477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd92409a480 0x7fd9240b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 >> 172.21.15.104:0/2524997477 conn(0x7fd924089fd0 msgr2=0x7fd92415fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 shutdown_connections 2023-06-10T18:58:04.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.372+0000 7fd92b8b0700 1 -- 172.21.15.104:0/2524997477 wait complete. 2023-06-10T18:58:04.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:42.097445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:43.215419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:04.392 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:04.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.510+0000 7f2eb531d700 1 Processor -- start 2023-06-10T18:58:04.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.510+0000 7f2eb531d700 1 -- start start 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eb531d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb00d04f0 0x7f2eb00d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eb531d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2eb00d25c0 0x7f2eb00d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eb531d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb00d0e00 con 0x7f2eb006b510 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eb531d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb00af530 con 0x7f2eb00d25c0 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eb531d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00af810 con 0x7f2eb00d04f0 2023-06-10T18:58:04.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eae7fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2eb00d25c0 0x7f2eb00d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb00d04f0 0x7f2eb00d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eae7fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2eb00d25c0 0x7f2eb00d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:55256/0 (socket says 172.21.15.104:55256) 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.511+0000 7f2eae7fc700 1 -- 172.21.15.104:0/2195629885 learned_addr learned my addr 172.21.15.104:0/2195629885 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eae7fc700 1 -- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb00d04f0 msgr2=0x7f2eb00d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eae7fc700 1 --2- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb00d04f0 0x7f2eb00d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eae7fc700 1 -- 172.21.15.104:0/2195629885 >> v1:172.21.15.114:6789/0 conn(0x7f2eb006b510 legacy=0x7f2eb006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eae7fc700 1 -- 172.21.15.104:0/2195629885 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00afaf0 con 0x7f2eb00d25c0 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eadffb700 1 -- 172.21.15.104:0/2195629885 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199852965 0 0) 0x7f2eb00d0e00 con 0x7f2eb006b510 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eae7fc700 1 --2- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d25c0 0x7f2eb00d08c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098f0 tx=0x7f2ea0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=750e6f05b8fdea0d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eadffb700 1 -- 172.21.15.104:0/2195629885 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011830 con 0x7f2eb00d25c0 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eadffb700 1 -- 172.21.15.104:0/2195629885 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0011990 con 0x7f2eb00d25c0 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eadffb700 1 -- 172.21.15.104:0/2195629885 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011d20 con 0x7f2eb00d25c0 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eb531d700 1 -- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d25c0 msgr2=0x7f2eb00d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d25c0 0x7f2eb00d08c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098f0 tx=0x7f2ea0009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.512+0000 7f2eb531d700 1 -- 172.21.15.104:0/2195629885 shutdown_connections 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.513+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb00d04f0 0x7f2eb00d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.513+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2195629885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d25c0 0x7f2eb00d08c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.513+0000 7f2eb531d700 1 -- 172.21.15.104:0/2195629885 >> 172.21.15.104:0/2195629885 conn(0x7f2eb0071ec0 msgr2=0x7f2eb009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.513+0000 7f2eb531d700 1 -- 172.21.15.104:0/2195629885 shutdown_connections 2023-06-10T18:58:04.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.513+0000 7f2eb531d700 1 -- 172.21.15.104:0/2195629885 wait complete. 2023-06-10T18:58:04.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.514+0000 7f2eb531d700 1 Processor -- start 2023-06-10T18:58:04.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.514+0000 7f2eb531d700 1 -- start start 2023-06-10T18:58:04.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb006b510 0x7f2eb00b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 0x7f2eb01e4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00b2a50 con 0x7f2eb00d25c0 2023-06-10T18:58:04.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb01e6710 con 0x7f2eb00d04f0 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00b3420 con 0x7f2eb006b510 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39530/0 (socket says 172.21.15.104:39530) 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaeffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb006b510 0x7f2eb00b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 -- 172.21.15.104:0/2357924684 learned_addr learned my addr 172.21.15.104:0/2357924684 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eae7fc700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 0x7f2eb01e4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.526 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 -- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb006b510 msgr2=0x7f2eb00b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb006b510 0x7f2eb00b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 -- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 msgr2=0x7f2eb01e4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 0x7f2eb01e4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 -- 172.21.15.104:0/2357924684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0009550 con 0x7f2eb00d25c0 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eae7fc700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 0x7f2eb01e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eaf7fe700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c810 tx=0x7f2ea800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003e30 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8005070 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8005420 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.515+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb01e4e60 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.516+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb01f2c30 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.516+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb0067010 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.517+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2ea8005580 con 0x7f2eb00d25c0 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.517+0000 7f2e977fe700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 0x7f2e9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.517+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2ea80891a0 con 0x7f2eb00d25c0 2023-06-10T18:58:04.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.517+0000 7f2eaeffd700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 0x7f2e9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.518+0000 7f2eaeffd700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 0x7f2e9c06d190 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009bc0 tx=0x7f2ea40091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.522+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2ea8054170 con 0x7f2eb00d25c0 2023-06-10T18:58:04.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.710+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2eb0074050 con 0x7f2eb00d25c0 2023-06-10T18:58:04.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.711+0000 7f2e977fe700 1 -- 172.21.15.104:0/2357924684 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v160) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ea8057790 con 0x7f2eb00d25c0 2023-06-10T18:58:04.716 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 msgr2=0x7f2e9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 0x7f2e9c06d190 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009bc0 tx=0x7f2ea40091c0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 msgr2=0x7f2eb0157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c810 tx=0x7f2ea800cb20 comp rx=0 tx=0).stop 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 shutdown_connections 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb00d25c0 0x7f2eb0157940 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb006b510 0x7f2eb00b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2e9c06ace0 0x7f2e9c06d190 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 --2- 172.21.15.104:0/2357924684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb00d04f0 0x7f2eb01e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 >> 172.21.15.104:0/2357924684 conn(0x7f2eb0071ec0 msgr2=0x7f2eb00b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 shutdown_connections 2023-06-10T18:58:04.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.713+0000 7f2eb531d700 1 -- 172.21.15.104:0/2357924684 wait complete. 2023-06-10T18:58:04.730 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:04.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.848+0000 7fdaf9f7c700 1 Processor -- start 2023-06-10T18:58:04.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.848+0000 7fdaf9f7c700 1 -- start start 2023-06-10T18:58:04.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf9f7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdaf40cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf9f7c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40cda90 con 0x7fdaf40af940 2023-06-10T18:58:04.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf9f7c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf4154970 con 0x7fdaf41545a0 2023-06-10T18:58:04.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf9f7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415dab0 con 0x7fdaf40cd150 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf2ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdaf40af940 0x7fdaf415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37364/0 (socket says 172.21.15.104:37364) 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.849+0000 7fdaf2ffd700 1 -- 172.21.15.104:0/1411125423 learned_addr learned my addr 172.21.15.104:0/1411125423 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373194625 0 0) 0x7fdaf40cda90 con 0x7fdaf40af940 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc0035b0 con 0x7fdaf40af940 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdaf40cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732639629 0 0) 0x7fdadc0035b0 con 0x7fdaf40af940 2023-06-10T18:58:04.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 msgr2=0x7fdaf40cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdaf40cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 >> v1:172.21.15.114:6789/0 conn(0x7fdaf41545a0 legacy=0x7fdaf415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40cb0b0 con 0x7fdaf40af940 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdae4001670 con 0x7fdaf40af940 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdaf40cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:04.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 >> v1:172.21.15.104:6789/0 conn(0x7fdaf40af940 legacy=0x7fdaf415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdadc003a30 0x7fdadc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc004850 0x7fdadc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdadc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415dab0 con 0x7fdadc003a30 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00baf0 con 0x7fdadc004850 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00bcb0 con 0x7fdaf40cd150 2023-06-10T18:58:04.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdadc003a30 0x7fdadc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdadc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf2ffd700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc004850 0x7fdadc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 -- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 msgr2=0x7fdadc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdadc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 -- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc004850 msgr2=0x7fdadc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc004850 0x7fdadc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 -- 172.21.15.104:0/1411125423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0035b0 con 0x7fdadc003a30 2023-06-10T18:58:04.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40cd150 0x7fdadc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf2ffd700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc004850 0x7fdadc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdadc003a30 0x7fdadc003e40 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fdae80017c0 tx=0x7fdadc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1138c9170987d82 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.850+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae80052d0 con 0x7fdadc003a30 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8005430 con 0x7fdadc003a30 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf27fc700 1 -- 172.21.15.104:0/1411125423 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae80038c0 con 0x7fdadc003a30 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1411125423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdadc003a30 msgr2=0x7fdadc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1411125423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdadc003a30 0x7fdadc003e40 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fdae80017c0 tx=0x7fdadc005880 comp rx=0 tx=0).stop 2023-06-10T18:58:04.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf3fff700 1 -- 172.21.15.104:0/1411125423 reap_dead start 2023-06-10T18:58:04.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1411125423 shutdown_connections 2023-06-10T18:58:04.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1411125423 >> 172.21.15.104:0/1411125423 conn(0x7fdaf406e370 msgr2=0x7fdaf406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:04.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1411125423 shutdown_connections 2023-06-10T18:58:04.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.851+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1411125423 wait complete. 2023-06-10T18:58:04.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.852+0000 7fdaf9f7c700 1 Processor -- start 2023-06-10T18:58:04.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 -- start start 2023-06-10T18:58:04.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 0x7fdaf41f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40b18f0 0x7fdaf414a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40b2b10 con 0x7fdaf415ca90 2023-06-10T18:58:04.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415d470 con 0x7fdadc003a30 2023-06-10T18:58:04.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf9f7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40b03f0 con 0x7fdaf40b18f0 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39550/0 (socket says 172.21.15.104:39550) 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.853+0000 7fdaf3fff700 1 -- 172.21.15.104:0/1222858189 learned_addr learned my addr 172.21.15.104:0/1222858189 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 0x7fdaf41f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 -- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40b18f0 msgr2=0x7fdaf414a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf2ffd700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40b18f0 0x7fdaf414a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40b18f0 0x7fdaf414a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 -- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 msgr2=0x7fdaf41f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 0x7fdaf41f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 -- 172.21.15.104:0/1222858189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf41f2a50 con 0x7fdaf415ca90 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 0x7fdaf41f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf3fff700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4072150 tx=0x7fdaec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec007b80 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec00f050 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf414aa40 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec014410 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.854+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40b2170 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.855+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf40b2c50 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.855+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdaec014570 con 0x7fdaf415ca90 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.856+0000 7fdadbfff700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 0x7fdae006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:04.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.856+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fdaec0897b0 con 0x7fdaf415ca90 2023-06-10T18:58:04.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.856+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 0x7fdae006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:04.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.856+0000 7fdaf37fe700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 0x7fdae006d190 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40ca810 tx=0x7fdae8009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:04.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:04.860+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdaf40b2c50 con 0x7fdaf415ca90 2023-06-10T18:58:05.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.045+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdaf40b2c50 con 0x7fdaf415ca90 2023-06-10T18:58:05.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.046+0000 7fdadbfff700 1 -- 172.21.15.104:0/1222858189 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdaf40b2c50 con 0x7fdaf415ca90 2023-06-10T18:58:05.052 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:05.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 msgr2=0x7fdae006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 0x7fdae006d190 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40ca810 tx=0x7fdae8009030 comp rx=0 tx=0).stop 2023-06-10T18:58:05.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 msgr2=0x7fdaf415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4072150 tx=0x7fdaec004740 comp rx=0 tx=0).stop 2023-06-10T18:58:05.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 shutdown_connections 2023-06-10T18:58:05.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdae006ace0 0x7fdae006d190 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdadc003a30 0x7fdaf41f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdaf415ca90 0x7fdaf415cf30 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 --2- 172.21.15.104:0/1222858189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdaf40b18f0 0x7fdaf414a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 >> 172.21.15.104:0/1222858189 conn(0x7fdaf406e370 msgr2=0x7fdaf40b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 shutdown_connections 2023-06-10T18:58:05.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.048+0000 7fdaf9f7c700 1 -- 172.21.15.104:0/1222858189 wait complete. 2023-06-10T18:58:05.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:05.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.183+0000 7fbb99670700 1 Processor -- start 2023-06-10T18:58:05.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.183+0000 7fbb99670700 1 -- start start 2023-06-10T18:58:05.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb99670700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb9406e6e0 0x7fbb940b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb99670700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb99670700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9406e250 con 0x7fbb9415c580 2023-06-10T18:58:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb99670700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9415c950 con 0x7fbb9406e6e0 2023-06-10T18:58:05.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb99670700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940b1e80 con 0x7fbb9406cde0 2023-06-10T18:58:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb937fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb927fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb9406e6e0 0x7fbb940b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb937fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:58942/0 (socket says 172.21.15.104:58942) 2023-06-10T18:58:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb927fc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb9406e6e0 0x7fbb940b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:55290/0 (socket says 172.21.15.104:55290) 2023-06-10T18:58:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb937fe700 1 -- 172.21.15.104:0/222788265 learned_addr learned my addr 172.21.15.104:0/222788265 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.184+0000 7fbb92ffd700 1 --1- 172.21.15.104:0/222788265 >> v1:172.21.15.114:6789/0 conn(0x7fbb9415c580 0x7fbb940b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:43228/0 (socket says 172.21.15.104:43228) 2023-06-10T18:58:05.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb927fc700 1 -- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 msgr2=0x7fbb940ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb927fc700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb927fc700 1 -- 172.21.15.104:0/222788265 >> v1:172.21.15.114:6789/0 conn(0x7fbb9415c580 legacy=0x7fbb940b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb91ffb700 1 -- 172.21.15.104:0/222788265 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532546452 0 0) 0x7fbb9406e250 con 0x7fbb9415c580 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb927fc700 1 -- 172.21.15.104:0/222788265 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9406d920 con 0x7fbb9406e6e0 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb937fe700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:05.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb927fc700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b1780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009840 tx=0x7fbb84009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77cba0c7d427672 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb91ffb700 1 -- 172.21.15.104:0/222788265 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84011830 con 0x7fbb9406e6e0 2023-06-10T18:58:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb91ffb700 1 -- 172.21.15.104:0/222788265 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84011990 con 0x7fbb9406e6e0 2023-06-10T18:58:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.185+0000 7fbb91ffb700 1 -- 172.21.15.104:0/222788265 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84011d20 con 0x7fbb9406e6e0 2023-06-10T18:58:05.203 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 -- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 msgr2=0x7fbb940b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b1780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009840 tx=0x7fbb84009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 -- 172.21.15.104:0/222788265 shutdown_connections 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9406cde0 0x7fbb940ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 --2- 172.21.15.104:0/222788265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b1780 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 -- 172.21.15.104:0/222788265 >> 172.21.15.104:0/222788265 conn(0x7fbb9406b8a0 msgr2=0x7fbb940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 -- 172.21.15.104:0/222788265 shutdown_connections 2023-06-10T18:58:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.186+0000 7fbb99670700 1 -- 172.21.15.104:0/222788265 wait complete. 2023-06-10T18:58:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.187+0000 7fbb99670700 1 Processor -- start 2023-06-10T18:58:05.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.187+0000 7fbb99670700 1 -- start start 2023-06-10T18:58:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 0x7fbb940b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9414b450 con 0x7fbb9406cde0 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb941f1b50 con 0x7fbb9406e6e0 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb99670700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9414a2e0 con 0x7fbb9415c580 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39562/0 (socket says 172.21.15.104:39562) 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 -- 172.21.15.104:0/478814799 learned_addr learned my addr 172.21.15.104:0/478814799 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb937fe700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 0x7fbb940b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb927fc700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 -- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 msgr2=0x7fbb940b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 0x7fbb940b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 -- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 msgr2=0x7fbb940b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 -- 172.21.15.104:0/478814799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c009b30 con 0x7fbb9406cde0 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb937fe700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 0x7fbb940b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb927fc700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.188+0000 7fbb92ffd700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800c4b0 tx=0x7fbb8800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.189+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb88004cd0 con 0x7fbb9406cde0 2023-06-10T18:58:05.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.189+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88004e50 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.189+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb840094a0 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.189+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb88007310 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.189+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9414aab0 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.190+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb94074050 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.190+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbb88007470 con 0x7fbb9406cde0 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.190+0000 7fbb7b7fe700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 0x7fbb8006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.191+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fbb880892c0 con 0x7fbb9406cde0 2023-06-10T18:58:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.191+0000 7fbb927fc700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 0x7fbb8006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.191+0000 7fbb927fc700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 0x7fbb8006d190 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406eb20 tx=0x7fbb84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.195+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbb88054210 con 0x7fbb9406cde0 2023-06-10T18:58:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.382+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb94159c60 con 0x7fbb9406cde0 2023-06-10T18:58:05.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.382+0000 7fbb7b7fe700 1 -- 172.21.15.104:0/478814799 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v160) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb88057830 con 0x7fbb9406cde0 2023-06-10T18:58:05.388 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:05.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 msgr2=0x7fbb8006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 0x7fbb8006d190 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406eb20 tx=0x7fbb84008030 comp rx=0 tx=0).stop 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 msgr2=0x7fbb941f1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800c4b0 tx=0x7fbb8800c870 comp rx=0 tx=0).stop 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 shutdown_connections 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb9415c580 0x7fbb940b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb9406cde0 0x7fbb941f1610 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb8006ace0 0x7fbb8006d190 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 --2- 172.21.15.104:0/478814799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb9406e6e0 0x7fbb940b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 >> 172.21.15.104:0/478814799 conn(0x7fbb9406b8a0 msgr2=0x7fbb940cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 shutdown_connections 2023-06-10T18:58:05.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.384+0000 7fbb99670700 1 -- 172.21.15.104:0/478814799 wait complete. 2023-06-10T18:58:05.402 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:05.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.519+0000 7f7578243700 1 Processor -- start 2023-06-10T18:58:05.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.519+0000 7f7578243700 1 -- start start 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7578243700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f757015c580 0x7f75700b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7578243700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 0x7f75700ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7578243700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006e250 con 0x7f757006e6e0 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7578243700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757015c950 con 0x7f757015c580 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7578243700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b1e80 con 0x7f757006cde0 2023-06-10T18:58:05.527 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7575fdf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f757015c580 0x7f75700b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f75757de700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f757006e6e0 0x7f75700b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37396/0 (socket says 172.21.15.104:37396) 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f75757de700 1 -- 172.21.15.104:0/2229626744 learned_addr learned my addr 172.21.15.104:0/2229626744 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.520+0000 7f7575fdf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f757015c580 0x7f75700b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39568/0 (socket says 172.21.15.104:39568) 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f75767e0700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 0x7f75700ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7575fdf700 1 -- 172.21.15.104:0/2229626744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 msgr2=0x7f75700ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:05.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7575fdf700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 0x7f75700ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7575fdf700 1 -- 172.21.15.104:0/2229626744 >> v1:172.21.15.104:6789/0 conn(0x7f757006e6e0 legacy=0x7f75700b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7575fdf700 1 -- 172.21.15.104:0/2229626744 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757006d920 con 0x7f757015c580 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f75767e0700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 0x7f75700ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7575fdf700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757015c580 0x7f75700b1070 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f7560009820 tx=0x7f7560009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde3ee2b26c63d1c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7574fdd700 1 -- 172.21.15.104:0/2229626744 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7560005e30 con 0x7f757015c580 2023-06-10T18:58:05.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7574fdd700 1 -- 172.21.15.104:0/2229626744 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7560006030 con 0x7f757015c580 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7574fdd700 1 -- 172.21.15.104:0/2229626744 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75600063e0 con 0x7f757015c580 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7578243700 1 -- 172.21.15.104:0/2229626744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757015c580 msgr2=0x7f75700b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.521+0000 7f7578243700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757015c580 0x7f75700b1070 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f7560009820 tx=0x7f7560009b30 comp rx=0 tx=0).stop 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 -- 172.21.15.104:0/2229626744 shutdown_connections 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757015c580 0x7f75700b1070 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 --2- 172.21.15.104:0/2229626744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757006cde0 0x7f75700ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 -- 172.21.15.104:0/2229626744 >> 172.21.15.104:0/2229626744 conn(0x7f757006b8a0 msgr2=0x7f75700cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 -- 172.21.15.104:0/2229626744 shutdown_connections 2023-06-10T18:58:05.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.522+0000 7f7578243700 1 -- 172.21.15.104:0/2229626744 wait complete. 2023-06-10T18:58:05.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.523+0000 7f7578243700 1 Processor -- start 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.523+0000 7f7578243700 1 -- start start 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757006cde0 0x7f7570158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 0x7f75700b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700c4bd0 con 0x7f757006e6e0 2023-06-10T18:58:05.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700c4eb0 con 0x7f757006cde0 2023-06-10T18:58:05.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700c5190 con 0x7f757015c580 2023-06-10T18:58:05.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39576/0 (socket says 172.21.15.104:39576) 2023-06-10T18:58:05.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 -- 172.21.15.104:0/3642794464 learned_addr learned my addr 172.21.15.104:0/3642794464 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:05.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7575fdf700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757006cde0 0x7f7570158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75767e0700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 0x7f75700b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 -- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 msgr2=0x7f75700b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 0x7f75700b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 -- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757006cde0 msgr2=0x7f7570158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757006cde0 0x7f7570158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 -- 172.21.15.104:0/3642794464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7560009430 con 0x7f757006e6e0 2023-06-10T18:58:05.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75767e0700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 0x7f75700b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75757de700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c005d70 con 0x7f757006e6e0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c0070d0 con 0x7f757006e6e0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700c5470 con 0x7f757006e6e0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c007440 con 0x7f757006e6e0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.524+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75700c3090 con 0x7f757006e6e0 2023-06-10T18:58:05.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.525+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7570074050 con 0x7f757006e6e0 2023-06-10T18:58:05.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.526+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f756c0075a0 con 0x7f757006e6e0 2023-06-10T18:58:05.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.526+0000 7f75667fc700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 0x7f755406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.526+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f756c0889b0 con 0x7f757006e6e0 2023-06-10T18:58:05.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.526+0000 7f7575fdf700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 0x7f755406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.527+0000 7f7575fdf700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 0x7f755406d190 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7560013040 tx=0x7f7560008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.530+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f756c0539a0 con 0x7f757006e6e0 2023-06-10T18:58:05.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.715+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75700c32b0 con 0x7f757006e6e0 2023-06-10T18:58:05.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.715+0000 7f75667fc700 1 -- 172.21.15.104:0/3642794464 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v160) v1 ==== 100+0+11 (secure 0 0 0) 0x7f756c056fc0 con 0x7f757006e6e0 2023-06-10T18:58:05.721 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 msgr2=0x7f755406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 0x7f755406d190 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7560013040 tx=0x7f7560008070 comp rx=0 tx=0).stop 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 msgr2=0x7f75700b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c500 tx=0x7f756c00c8c0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 shutdown_connections 2023-06-10T18:58:05.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f755406ace0 0x7f755406d190 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757006cde0 0x7f7570158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f757015c580 0x7f75700b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 --2- 172.21.15.104:0/3642794464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f757006e6e0 0x7f75700b4c20 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 >> 172.21.15.104:0/3642794464 conn(0x7f757006b8a0 msgr2=0x7f75700c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 shutdown_connections 2023-06-10T18:58:05.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.717+0000 7f7578243700 1 -- 172.21.15.104:0/3642794464 wait complete. 2023-06-10T18:58:05.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:05.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.851+0000 7fcd85b58700 1 Processor -- start 2023-06-10T18:58:05.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.851+0000 7fcd85b58700 1 -- start start 2023-06-10T18:58:05.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd800aeaa0 0x7fcd800cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 0x7fcd800cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd800c9bc0 con 0x7fcd800aeaa0 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd800c9e60 con 0x7fcd800b1080 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd85b58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800aee70 con 0x7fcd800ca2f0 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd800aeaa0 0x7fcd800cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fcd800aeaa0 0x7fcd800cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39582/0 (socket says 172.21.15.104:39582) 2023-06-10T18:58:05.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 -- 172.21.15.104:0/706634738 learned_addr learned my addr 172.21.15.104:0/706634738 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7effd700 1 --2- 172.21.15.104:0/706634738 >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 0x7fcd800cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7ffff700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 -- 172.21.15.104:0/706634738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 msgr2=0x7fcd800ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 -- 172.21.15.104:0/706634738 >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 msgr2=0x7fcd800cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:05.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/706634738 >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 0x7fcd800cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 -- 172.21.15.104:0/706634738 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8006e540 con 0x7fcd800aeaa0 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7effd700 1 --2- 172.21.15.104:0/706634738 >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 0x7fcd800cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800aeaa0 0x7fcd800cd140 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fcd80070110 tx=0x7fcd68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7978618db90d58 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7e7fc700 1 -- 172.21.15.104:0/706634738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68005e30 con 0x7fcd800aeaa0 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.852+0000 7fcd7e7fc700 1 -- 172.21.15.104:0/706634738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68004030 con 0x7fcd800aeaa0 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd7e7fc700 1 -- 172.21.15.104:0/706634738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd680043e0 con 0x7fcd800aeaa0 2023-06-10T18:58:05.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 -- 172.21.15.104:0/706634738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800aeaa0 msgr2=0x7fcd800cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800aeaa0 0x7fcd800cd140 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fcd80070110 tx=0x7fcd68009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 -- 172.21.15.104:0/706634738 shutdown_connections 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800aeaa0 0x7fcd800cd140 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 --2- 172.21.15.104:0/706634738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 --2- 172.21.15.104:0/706634738 >> v2:172.21.15.114:3300/0 conn(0x7fcd800b1080 0x7fcd800cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 -- 172.21.15.104:0/706634738 >> 172.21.15.104:0/706634738 conn(0x7fcd8006b8a0 msgr2=0x7fcd800cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:05.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 -- 172.21.15.104:0/706634738 shutdown_connections 2023-06-10T18:58:05.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.853+0000 7fcd85b58700 1 -- 172.21.15.104:0/706634738 wait complete. 2023-06-10T18:58:05.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.854+0000 7fcd85b58700 1 Processor -- start 2023-06-10T18:58:05.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.854+0000 7fcd85b58700 1 -- start start 2023-06-10T18:58:05.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 0x7fcd800b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c4a10 con 0x7fcd800b1080 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801f5c30 con 0x7fcd800aeaa0 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd85b58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c35a0 con 0x7fcd800ca2f0 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd7effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd7effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39590/0 (socket says 172.21.15.104:39590) 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd7effd700 1 -- 172.21.15.104:0/2271659101 learned_addr learned my addr 172.21.15.104:0/2271659101 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.855+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 0x7fcd800b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7ffff700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 -- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 msgr2=0x7fcd800c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 -- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 msgr2=0x7fcd800b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 0x7fcd800b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 -- 172.21.15.104:0/2271659101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68009500 con 0x7fcd800b1080 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 0x7fcd800b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:05.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd7effd700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d830 tx=0x7fcd7000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70004030 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd70004190 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd800b5190 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70004520 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.856+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd800b3470 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.857+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd80074050 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.857+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcd700036a0 con 0x7fcd800b1080 2023-06-10T18:58:05.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.858+0000 7fcd77fff700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:05.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.858+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fcd70089b20 con 0x7fcd800b1080 2023-06-10T18:58:05.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.858+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:05.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.858+0000 7fcd7f7fe700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 0x7fcd6006d190 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fcd80070110 tx=0x7fcd68008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:05.869 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:05.862+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcd70058580 con 0x7fcd800b1080 2023-06-10T18:58:06.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.065+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcd8006c320 con 0x7fcd800b1080 2023-06-10T18:58:06.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.066+0000 7fcd77fff700 1 -- 172.21.15.104:0/2271659101 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v160) v1 ==== 80+0+57 (secure 0 0 0) 0x7fcd70058110 con 0x7fcd800b1080 2023-06-10T18:58:06.071 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/3682236906 2023-06-10T19:57:58.572945+0000 2023-06-10T18:58:06.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 msgr2=0x7fcd6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 0x7fcd6006d190 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fcd80070110 tx=0x7fcd68008070 comp rx=0 tx=0).stop 2023-06-10T18:58:06.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 msgr2=0x7fcd801f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d830 tx=0x7fcd7000dbf0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 shutdown_connections 2023-06-10T18:58:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd800aeaa0 0x7fcd800b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd800ca2f0 0x7fcd800c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 --2- 172.21.15.104:0/2271659101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd800b1080 0x7fcd801f56f0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 >> 172.21.15.104:0/2271659101 conn(0x7fcd8006b8a0 msgr2=0x7fcd80158ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:06.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 shutdown_connections 2023-06-10T18:58:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.068+0000 7fcd85b58700 1 -- 172.21.15.104:0/2271659101 wait complete. 2023-06-10T18:58:06.076 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:58:06.085 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T18:58:06.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.087 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:06.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:58:06.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.128 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:58:06.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:58:06.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.340+0000 7efeb9b7c700 1 Processor -- start 2023-06-10T18:58:06.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.340+0000 7efeb9b7c700 1 -- start start 2023-06-10T18:58:06.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb9b7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb415a2b0 0x7efeb40ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb9b7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efeb40cb860 0x7efeb406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb9b7c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb415a680 con 0x7efeb4069e70 2023-06-10T18:58:06.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb9b7c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb40d0510 con 0x7efeb40cb860 2023-06-10T18:58:06.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb9b7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb406a5e0 con 0x7efeb415a2b0 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efeb40cb860 0x7efeb406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efeb40cb860 0x7efeb406d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39600/0 (socket says 172.21.15.104:39600) 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.341+0000 7efeb3fff700 1 -- 172.21.15.104:0/1377583096 learned_addr learned my addr 172.21.15.104:0/1377583096 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb3fff700 1 -- 172.21.15.104:0/1377583096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb415a2b0 msgr2=0x7efeb40ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb3fff700 1 --2- 172.21.15.104:0/1377583096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb415a2b0 0x7efeb40ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb3fff700 1 -- 172.21.15.104:0/1377583096 >> v1:172.21.15.114:6789/0 conn(0x7efeb4069e70 legacy=0x7efeb40cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb3fff700 1 -- 172.21.15.104:0/1377583096 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb406a8c0 con 0x7efeb40cb860 2023-06-10T18:58:06.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb3fff700 1 --2- 172.21.15.104:0/1377583096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb40cb860 0x7efeb406d860 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7efeac009e50 tx=0x7efeac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76594cb4e8128cf6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb27fc700 1 -- 172.21.15.104:0/1377583096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac00e070 con 0x7efeb40cb860 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb27fc700 1 -- 172.21.15.104:0/1377583096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac003a20 con 0x7efeb40cb860 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb27fc700 1 -- 172.21.15.104:0/1377583096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac003dd0 con 0x7efeb40cb860 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 -- 172.21.15.104:0/1377583096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb40cb860 msgr2=0x7efeb406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/1377583096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb40cb860 0x7efeb406d860 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7efeac009e50 tx=0x7efeac00b330 comp rx=0 tx=0).stop 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 -- 172.21.15.104:0/1377583096 shutdown_connections 2023-06-10T18:58:06.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/1377583096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb40cb860 0x7efeb406d860 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/1377583096 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb415a2b0 0x7efeb40ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 -- 172.21.15.104:0/1377583096 >> 172.21.15.104:0/1377583096 conn(0x7efeb406b8a0 msgr2=0x7efeb40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:06.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 -- 172.21.15.104:0/1377583096 shutdown_connections 2023-06-10T18:58:06.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.342+0000 7efeb9b7c700 1 -- 172.21.15.104:0/1377583096 wait complete. 2023-06-10T18:58:06.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.344+0000 7efeb9b7c700 1 Processor -- start 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.344+0000 7efeb9b7c700 1 -- start start 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.344+0000 7efeb9b7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeb4069e70 0x7efeb414ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.344+0000 7efeb9b7c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb40cb860 0x7efeb40c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.344+0000 7efeb9b7c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb9b7c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c56e0 con 0x7efeb415a2b0 2023-06-10T18:58:06.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb9b7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c4ad0 con 0x7efeb4069e70 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb9b7c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c4db0 con 0x7efeb40cb860 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39606/0 (socket says 172.21.15.104:39606) 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 -- 172.21.15.104:0/2798307544 learned_addr learned my addr 172.21.15.104:0/2798307544 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb37fe700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeb4069e70 0x7efeb414ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb2ffd700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb40cb860 0x7efeb40c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:06.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 -- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb40cb860 msgr2=0x7efeb40c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb40cb860 0x7efeb40c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 -- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeb4069e70 msgr2=0x7efeb414ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeb4069e70 0x7efeb414ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 -- 172.21.15.104:0/2798307544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac009b30 con 0x7efeb415a2b0 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb3fff700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7efeb406cc20 tx=0x7efeac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac0094d0 con 0x7efeb415a2b0 2023-06-10T18:58:06.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac004240 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb40c5090 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac0045f0 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.345+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb414f0f0 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.346+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb40c5890 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.347+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efeac019070 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.347+0000 7efeabfff700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 0x7efe9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.347+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7efeac08ac90 con 0x7efeb415a2b0 2023-06-10T18:58:06.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.347+0000 7efeb37fe700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 0x7efe9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:06.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.348+0000 7efeb37fe700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 0x7efe9406d190 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009960 tx=0x7efe9c0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:06.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.352+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efeb40c5890 con 0x7efeb415a2b0 2023-06-10T18:58:06.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.582+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efeb40c5890 con 0x7efeb415a2b0 2023-06-10T18:58:06.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.583+0000 7efeabfff700 1 -- 172.21.15.104:0/2798307544 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v102) v1 ==== 53+0+73 (secure 0 0 0) 0x7efeb40c5890 con 0x7efeb415a2b0 2023-06-10T18:58:06.588 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:58:06.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 msgr2=0x7efe9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 0x7efe9406d190 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009960 tx=0x7efe9c0090e0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 msgr2=0x7efeb40c2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7efeb406cc20 tx=0x7efeac004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 shutdown_connections 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efe9406ace0 0x7efe9406d190 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeb4069e70 0x7efeb414ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efeb415a2b0 0x7efeb40c2ed0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 --2- 172.21.15.104:0/2798307544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efeb40cb860 0x7efeb40c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:06.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 >> 172.21.15.104:0/2798307544 conn(0x7efeb406b8a0 msgr2=0x7efeb40ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:06.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 shutdown_connections 2023-06-10T18:58:06.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:06.585+0000 7efeb9b7c700 1 -- 172.21.15.104:0/2798307544 wait complete. 2023-06-10T18:58:06.602 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:58:06.603 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:58:06.603 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:58:06.603 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:58:06.604 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:06.604 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:58:06.604 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:58:06.604 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:58:06.605 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:06.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.605 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:58:06.623 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:06.623 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:06.623 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:06.623 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:06.624 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41598sec preferred_lft 41598sec 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:58:06.625 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state DOWN group default qlen 1000 2023-06-10T18:58:06.626 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:58:06.627 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:06.627 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:58:06.627 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:58:06.627 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:06.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.628 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:58:06.688 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:58:06.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.690 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:58:06.749 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:58:06.749 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:58:06.749 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:58:06.749 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:58:06.750 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:58:06.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.750 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:06.751 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:06.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:58:06.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:58:06.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:58:06.752 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:06.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.854 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:58:06.871 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:58:06.871 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:58:06.872 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:06.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:06.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:58:06.945 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:58:06.946 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:06.946 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:07.001 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:07.002 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:07.002 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:07.058 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:07.124 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:07.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:07.191 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:07.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:07.194 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:07.249 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:58:07.250 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:58:07.292 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:07.318 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:07.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:07.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:58:07.371+0000 7f612db855c0 -1 init, newargv = 0x558ec4969950 newargc=15 2023-06-10T18:58:07.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[157294]: starting ceph client 2023-06-10T18:58:07.383 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[157294]: starting fuse 2023-06-10T18:58:07.387 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:07.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:07.389 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:07.445 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:58:07.446 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:58:07.446 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:58:07.447 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:58:07.447 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:58:07.447 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:58:07.448 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:58:07.448 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:07.448 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:58:07.449 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:58:07.449 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:58:07.449 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:58:07.449 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:07.450 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:58:07.450 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:58:07.450 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:58:07.451 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:58:07.451 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:07.451 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:58:07.451 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:58:07.452 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:58:07.452 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:58:07.452 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:58:07.453 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:58:07.453 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:58:07.453 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:58:07.455 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:07.455 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:58:07.456 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:58:07.567 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.157294.asok 2023-06-10T18:58:07.750 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.157294.asok 2023-06-10T18:58:07.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:07.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.157294.asok status 2023-06-10T18:58:07.906 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:07.906 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:58:07.906 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:58:07.906 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:58:07.906 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "157294", 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:58:07.907 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10362, 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10362 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:07.908 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1165519202 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:07.909 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1165519202 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10362 192.168.0.1:0/1165519202", 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/1165519202", 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 102, 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 160, 2023-06-10T18:58:07.910 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:58:07.911 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:58:07.911 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:58:07.911 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:07.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:07.916 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:58:07.917 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:07.935 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:58:07.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:07.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:07.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:08.001 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:58:08.001 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:58:08.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:58:08.138 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:58:08.138 INFO:tasks.cephfs.mount:Terminating background process 2023-06-10T18:58:08.138 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:08.139 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:08.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:08.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.311+0000 7fa825c9c700 1 Processor -- start 2023-06-10T18:58:08.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 -- start start 2023-06-10T18:58:08.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa82015c730 0x7fa82006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 0x7fa8200cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82015cb00 con 0x7fa8200cc000 2023-06-10T18:58:08.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8200aed80 con 0x7fa82015c730 2023-06-10T18:58:08.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa825c9c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200c9c60 con 0x7fa8200cb0c0 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa81f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa82015c730 0x7fa82006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa81f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fa82015c730 0x7fa82006e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39624/0 (socket says 172.21.15.104:39624) 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.312+0000 7fa81f7fe700 1 -- 172.21.15.104:0/4225374838 learned_addr learned my addr 172.21.15.104:0/4225374838 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81f7fe700 1 -- 172.21.15.104:0/4225374838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 msgr2=0x7fa8200cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81ffff700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 0x7fa8200cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81f7fe700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 0x7fa8200cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81f7fe700 1 -- 172.21.15.104:0/4225374838 >> v1:172.21.15.114:6789/0 conn(0x7fa8200cc000 legacy=0x7fa82006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81f7fe700 1 -- 172.21.15.104:0/4225374838 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82006ecd0 con 0x7fa82015c730 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81ffff700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 0x7fa8200cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81f7fe700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa82015c730 0x7fa82006e790 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fa810006fd0 tx=0x7fa810009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2686d278a0ad2b26 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81e7fc700 1 -- 172.21.15.104:0/4225374838 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810005e30 con 0x7fa82015c730 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81e7fc700 1 -- 172.21.15.104:0/4225374838 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810006030 con 0x7fa82015c730 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.313+0000 7fa81e7fc700 1 -- 172.21.15.104:0/4225374838 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8100063c0 con 0x7fa82015c730 2023-06-10T18:58:08.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 -- 172.21.15.104:0/4225374838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa82015c730 msgr2=0x7fa82006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa82015c730 0x7fa82006e790 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fa810006fd0 tx=0x7fa810009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 -- 172.21.15.104:0/4225374838 shutdown_connections 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa82015c730 0x7fa82006e790 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 --2- 172.21.15.104:0/4225374838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cb0c0 0x7fa8200cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 -- 172.21.15.104:0/4225374838 >> 172.21.15.104:0/4225374838 conn(0x7fa82006b8a0 msgr2=0x7fa8200cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 -- 172.21.15.104:0/4225374838 shutdown_connections 2023-06-10T18:58:08.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.314+0000 7fa825c9c700 1 -- 172.21.15.104:0/4225374838 wait complete. 2023-06-10T18:58:08.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.315+0000 7fa825c9c700 1 Processor -- start 2023-06-10T18:58:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.315+0000 7fa825c9c700 1 -- start start 2023-06-10T18:58:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 0x7fa8201509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 0x7fa82015a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820150f20 con 0x7fa8200cb0c0 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8201511c0 con 0x7fa82015c730 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8201514a0 con 0x7fa8200cc000 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39630/0 (socket says 172.21.15.104:39630) 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 -- 172.21.15.104:0/3461126128 learned_addr learned my addr 172.21.15.104:0/3461126128 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81ffff700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 0x7fa82015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81effd700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 0x7fa8201509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 -- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 msgr2=0x7fa8201509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 0x7fa8201509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 -- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 msgr2=0x7fa82015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 0x7fa82015a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 -- 172.21.15.104:0/3461126128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810009500 con 0x7fa8200cb0c0 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81ffff700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 0x7fa82015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81effd700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 0x7fa8201509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa81f7fe700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fa810009c10 tx=0x7fa810004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810006930 con 0x7fa8200cb0c0 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810020070 con 0x7fa8200cb0c0 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa820151780 con 0x7fa8200cb0c0 2023-06-10T18:58:08.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810025650 con 0x7fa8200cb0c0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.316+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82015a5c0 con 0x7fa8200cb0c0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.318+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820074050 con 0x7fa8200cb0c0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.318+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa810022030 con 0x7fa8200cb0c0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.318+0000 7fa817fff700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.318+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fa81008aad0 con 0x7fa8200cb0c0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.319+0000 7fa81effd700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.319+0000 7fa81effd700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa8201e3790 tx=0x7fa808009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.323+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa810055aa0 con 0x7fa8200cb0c0 2023-06-10T18:58:08.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.513+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa82015cb00 con 0x7fa8200cb0c0 2023-06-10T18:58:08.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.514+0000 7fa817fff700 1 -- 172.21.15.104:0/3461126128 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7fa82015cb00 con 0x7fa8200cb0c0 2023-06-10T18:58:08.521 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:08.521 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:08.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 msgr2=0x7fa80006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa8201e3790 tx=0x7fa808009040 comp rx=0 tx=0).stop 2023-06-10T18:58:08.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 msgr2=0x7fa8200b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fa810009c10 tx=0x7fa810004750 comp rx=0 tx=0).stop 2023-06-10T18:58:08.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 shutdown_connections 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa80006ace0 0x7fa80006d190 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa82015c730 0x7fa82015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8200cb0c0 0x7fa8200b1b90 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 --2- 172.21.15.104:0/3461126128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa8200cc000 0x7fa8201509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 >> 172.21.15.104:0/3461126128 conn(0x7fa82006b8a0 msgr2=0x7fa82015a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 shutdown_connections 2023-06-10T18:58:08.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.517+0000 7fa825c9c700 1 -- 172.21.15.104:0/3461126128 wait complete. 2023-06-10T18:58:08.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:42.097445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:43.215419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:08.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:08.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.657+0000 7fba5a05a700 1 Processor -- start 2023-06-10T18:58:08.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.657+0000 7fba5a05a700 1 -- start start 2023-06-10T18:58:08.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba5a05a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 0x7fba540d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba5a05a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba54145ba0 con 0x7fba540cec90 2023-06-10T18:58:08.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba5a05a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540d2890 con 0x7fba540d0a60 2023-06-10T18:58:08.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba5a05a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba540d0370 con 0x7fba540d23f0 2023-06-10T18:58:08.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba537fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba540cec90 0x7fba540cfc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37466/0 (socket says 172.21.15.104:37466) 2023-06-10T18:58:08.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba537fe700 1 -- 172.21.15.104:0/4246484655 learned_addr learned my addr 172.21.15.104:0/4246484655 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:08.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba52ffd700 1 --2- 172.21.15.104:0/4246484655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 0x7fba540d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698897457 0 0) 0x7fba54145ba0 con 0x7fba540cec90 2023-06-10T18:58:08.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c0035b0 con 0x7fba540cec90 2023-06-10T18:58:08.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011009700 0 0) 0x7fba540d2890 con 0x7fba540d0a60 2023-06-10T18:58:08.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba54145ba0 con 0x7fba540d0a60 2023-06-10T18:58:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692736088 0 0) 0x7fba3c0035b0 con 0x7fba540cec90 2023-06-10T18:58:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba52ffd700 1 -- 172.21.15.104:0/4246484655 >> v1:172.21.15.114:6789/0 conn(0x7fba540d0a60 legacy=0x7fba540ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba52ffd700 1 -- 172.21.15.104:0/4246484655 >> v1:172.21.15.104:6789/0 conn(0x7fba540cec90 legacy=0x7fba540cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba52ffd700 1 -- 172.21.15.104:0/4246484655 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba540d2b70 con 0x7fba540d23f0 2023-06-10T18:58:08.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.658+0000 7fba52ffd700 1 --2- 172.21.15.104:0/4246484655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 0x7fba540d0520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fba48009870 tx=0x7fba48009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=814e9dc21b763042 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44001670 con 0x7fba540d23f0 2023-06-10T18:58:08.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba48004030 con 0x7fba540d23f0 2023-06-10T18:58:08.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba527fc700 1 -- 172.21.15.104:0/4246484655 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba480043c0 con 0x7fba540d23f0 2023-06-10T18:58:08.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 -- 172.21.15.104:0/4246484655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 msgr2=0x7fba540d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 --2- 172.21.15.104:0/4246484655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 0x7fba540d0520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fba48009870 tx=0x7fba48009b80 comp rx=0 tx=0).stop 2023-06-10T18:58:08.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 -- 172.21.15.104:0/4246484655 shutdown_connections 2023-06-10T18:58:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 --2- 172.21.15.104:0/4246484655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d23f0 0x7fba540d0520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 -- 172.21.15.104:0/4246484655 >> 172.21.15.104:0/4246484655 conn(0x7fba54071ec0 msgr2=0x7fba5409bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 -- 172.21.15.104:0/4246484655 shutdown_connections 2023-06-10T18:58:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.659+0000 7fba5a05a700 1 -- 172.21.15.104:0/4246484655 wait complete. 2023-06-10T18:58:08.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.660+0000 7fba5a05a700 1 Processor -- start 2023-06-10T18:58:08.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 -- start start 2023-06-10T18:58:08.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 0x7fba541f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d0a60 0x7fba5415a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba540c6dc0 con 0x7fba540d23f0 2023-06-10T18:58:08.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba541f6650 con 0x7fba540cec90 2023-06-10T18:58:08.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba5a05a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba540c61b0 con 0x7fba540d0a60 2023-06-10T18:58:08.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39648/0 (socket says 172.21.15.104:39648) 2023-06-10T18:58:08.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.661+0000 7fba53fff700 1 -- 172.21.15.104:0/1440367866 learned_addr learned my addr 172.21.15.104:0/1440367866 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:08.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba537fe700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 0x7fba541f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba52ffd700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d0a60 0x7fba5415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 -- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d0a60 msgr2=0x7fba5415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d0a60 0x7fba5415a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 -- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 msgr2=0x7fba541f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 0x7fba541f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 -- 172.21.15.104:0/1440367866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480094a0 con 0x7fba540d23f0 2023-06-10T18:58:08.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba537fe700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 0x7fba541f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:08.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba53fff700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00cbf0 tx=0x7fba4c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c003c00 con 0x7fba540d23f0 2023-06-10T18:58:08.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c003d60 con 0x7fba540d23f0 2023-06-10T18:58:08.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba540c6490 con 0x7fba540d23f0 2023-06-10T18:58:08.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00abe0 con 0x7fba540d23f0 2023-06-10T18:58:08.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.662+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba540c6a50 con 0x7fba540d23f0 2023-06-10T18:58:08.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.663+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba540c6f30 con 0x7fba540d23f0 2023-06-10T18:58:08.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.663+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba4c00ad40 con 0x7fba540d23f0 2023-06-10T18:58:08.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.664+0000 7fba3bfff700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 0x7fba4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:08.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.664+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fba4c089680 con 0x7fba540d23f0 2023-06-10T18:58:08.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.664+0000 7fba537fe700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 0x7fba4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:08.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.664+0000 7fba537fe700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 0x7fba4006d190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba44000f60 tx=0x7fba44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:08.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.668+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba4c054650 con 0x7fba540d23f0 2023-06-10T18:58:08.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.857+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba54074050 con 0x7fba540d23f0 2023-06-10T18:58:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.858+0000 7fba3bfff700 1 -- 172.21.15.104:0/1440367866 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v160) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba4c057c70 con 0x7fba540d23f0 2023-06-10T18:58:08.864 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:08.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.859+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 msgr2=0x7fba4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.859+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 0x7fba4006d190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba44000f60 tx=0x7fba44009040 comp rx=0 tx=0).stop 2023-06-10T18:58:08.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.859+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 msgr2=0x7fba5415e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:08.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.859+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00cbf0 tx=0x7fba4c00a060 comp rx=0 tx=0).stop 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 shutdown_connections 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fba4006ace0 0x7fba4006d190 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba540cec90 0x7fba541f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba540d23f0 0x7fba5415e8a0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 --2- 172.21.15.104:0/1440367866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba540d0a60 0x7fba5415a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 >> 172.21.15.104:0/1440367866 conn(0x7fba54071ec0 msgr2=0x7fba5406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 shutdown_connections 2023-06-10T18:58:08.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.860+0000 7fba5a05a700 1 -- 172.21.15.104:0/1440367866 wait complete. 2023-06-10T18:58:08.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:09.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.994+0000 7f7891c15700 1 Processor -- start 2023-06-10T18:58:09.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.994+0000 7f7891c15700 1 -- start start 2023-06-10T18:58:09.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f7891c15700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f788c15c580 0x7f788c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f7891c15700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c06e250 con 0x7f788c06cde0 2023-06-10T18:58:09.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f7891c15700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c15c950 con 0x7f788c06e6e0 2023-06-10T18:58:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f7891c15700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0b1e80 con 0x7f788c15c580 2023-06-10T18:58:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f788b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f788c15c580 0x7f788c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f788bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f788c06cde0 0x7f788c0ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37484/0 (socket says 172.21.15.104:37484) 2023-06-10T18:58:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.995+0000 7f788bfff700 1 -- 172.21.15.104:0/3889874841 learned_addr learned my addr 172.21.15.104:0/3889874841 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:09.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788b7fe700 1 -- 172.21.15.104:0/3889874841 >> v1:172.21.15.114:6789/0 conn(0x7f788c06e6e0 legacy=0x7f788c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788b7fe700 1 -- 172.21.15.104:0/3889874841 >> v1:172.21.15.104:6789/0 conn(0x7f788c06cde0 legacy=0x7f788c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788a7fc700 1 -- 172.21.15.104:0/3889874841 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730719027 0 0) 0x7f788c06e250 con 0x7f788c06cde0 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788b7fe700 1 -- 172.21.15.104:0/3889874841 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c06d920 con 0x7f788c15c580 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788b7fe700 1 --2- 172.21.15.104:0/3889874841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c0b1070 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7880009820 tx=0x7f7880009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac61cf20e146808d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788a7fc700 1 -- 172.21.15.104:0/3889874841 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880005e30 con 0x7f788c15c580 2023-06-10T18:58:09.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788a7fc700 1 -- 172.21.15.104:0/3889874841 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880004030 con 0x7f788c15c580 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f788a7fc700 1 -- 172.21.15.104:0/3889874841 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78800043c0 con 0x7f788c15c580 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f7891c15700 1 -- 172.21.15.104:0/3889874841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 msgr2=0x7f788c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f7891c15700 1 --2- 172.21.15.104:0/3889874841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c0b1070 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7880009820 tx=0x7f7880009b30 comp rx=0 tx=0).stop 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f7891c15700 1 -- 172.21.15.104:0/3889874841 shutdown_connections 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f7891c15700 1 --2- 172.21.15.104:0/3889874841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c0b1070 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.996+0000 7f7891c15700 1 -- 172.21.15.104:0/3889874841 >> 172.21.15.104:0/3889874841 conn(0x7f788c06b8a0 msgr2=0x7f788c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.997+0000 7f7891c15700 1 -- 172.21.15.104:0/3889874841 shutdown_connections 2023-06-10T18:58:09.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.997+0000 7f7891c15700 1 -- 172.21.15.104:0/3889874841 wait complete. 2023-06-10T18:58:09.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.998+0000 7f7891c15700 1 Processor -- start 2023-06-10T18:58:09.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.998+0000 7f7891c15700 1 -- start start 2023-06-10T18:58:09.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c06cde0 0x7f788c149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f788c06e6e0 0x7f788c1e2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c14a100 con 0x7f788c15c580 2023-06-10T18:58:09.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0c8550 con 0x7f788c06cde0 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0c8830 con 0x7f788c06e6e0 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39662/0 (socket says 172.21.15.104:39662) 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 -- 172.21.15.104:0/2621004016 learned_addr learned my addr 172.21.15.104:0/2621004016 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788b7fe700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c06cde0 0x7f788c149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788affd700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f788c06e6e0 0x7f788c1e2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 -- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f788c06e6e0 msgr2=0x7f788c1e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f788c06e6e0 0x7f788c1e2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 -- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c06cde0 msgr2=0x7f788c149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c06cde0 0x7f788c149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 -- 172.21.15.104:0/2621004016 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7880009430 con 0x7f788c15c580 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f788bfff700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f788400cca0 tx=0x7f788400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788400ab70 con 0x7f788c15c580 2023-06-10T18:58:09.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788400acd0 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c0c8b10 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884007310 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:08.999+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c0c90d0 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.000+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c074050 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.001+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7884007470 con 0x7f788c15c580 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.001+0000 7f7873fff700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 0x7f787806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.001+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f78840893c0 con 0x7f788c15c580 2023-06-10T18:58:09.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.001+0000 7f788b7fe700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 0x7f787806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.002+0000 7f788b7fe700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 0x7f787806d190 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.005+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7884054300 con 0x7f788c15c580 2023-06-10T18:58:09.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.190+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f788c1e2610 con 0x7f788c15c580 2023-06-10T18:58:09.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.191+0000 7f7873fff700 1 -- 172.21.15.104:0/2621004016 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7884057920 con 0x7f788c15c580 2023-06-10T18:58:09.197 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:09.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 msgr2=0x7f787806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 0x7f787806d190 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880008070 comp rx=0 tx=0).stop 2023-06-10T18:58:09.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 msgr2=0x7f788c1e3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f788400cca0 tx=0x7f788400a060 comp rx=0 tx=0).stop 2023-06-10T18:58:09.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 shutdown_connections 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f787806ace0 0x7f787806d190 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c06cde0 0x7f788c149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f788c15c580 0x7f788c1e3070 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 --2- 172.21.15.104:0/2621004016 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f788c06e6e0 0x7f788c1e2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.192+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 >> 172.21.15.104:0/2621004016 conn(0x7f788c06b8a0 msgr2=0x7f788c159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.193+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 shutdown_connections 2023-06-10T18:58:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.193+0000 7f7891c15700 1 -- 172.21.15.104:0/2621004016 wait complete. 2023-06-10T18:58:09.211 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:09.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.339+0000 7ff68b109700 1 Processor -- start 2023-06-10T18:58:09.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.339+0000 7ff68b109700 1 -- start start 2023-06-10T18:58:09.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff68b109700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6840ce510 0x7ff6840ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff68b109700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff68415b190 0x7ff684070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff68b109700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68415bcc0 con 0x7ff68415dad0 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff68b109700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff684071c90 con 0x7ff6840ce510 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff68b109700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff684157e60 con 0x7ff68415b190 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6840ce510 0x7ff6840ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6896a6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff68415dad0 0x7ff68415ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37498/0 (socket says 172.21.15.104:37498) 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff6840ce510 0x7ff6840ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39666/0 (socket says 172.21.15.104:39666) 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6896a6700 1 -- 172.21.15.104:0/908583263 learned_addr learned my addr 172.21.15.104:0/908583263 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff683fff700 1 --2- 172.21.15.104:0/908583263 >> v2:172.21.15.114:3300/0 conn(0x7ff68415b190 0x7ff684070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 -- 172.21.15.104:0/908583263 >> v2:172.21.15.114:3300/0 conn(0x7ff68415b190 msgr2=0x7ff684070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 --2- 172.21.15.104:0/908583263 >> v2:172.21.15.114:3300/0 conn(0x7ff68415b190 0x7ff684070130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 -- 172.21.15.104:0/908583263 >> v1:172.21.15.104:6789/0 conn(0x7ff68415dad0 legacy=0x7ff68415ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 -- 172.21.15.104:0/908583263 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840cee20 con 0x7ff6840ce510 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6837fe700 1 -- 172.21.15.104:0/908583263 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487750015 0 0) 0x7ff68415bcc0 con 0x7ff68415dad0 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff688ea5700 1 --2- 172.21.15.104:0/908583263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840ce8e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7ff6840b4650 tx=0x7ff678009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c662761fbaee71c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6837fe700 1 -- 172.21.15.104:0/908583263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678011830 con 0x7ff6840ce510 2023-06-10T18:58:09.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6837fe700 1 -- 172.21.15.104:0/908583263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff678011990 con 0x7ff6840ce510 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.340+0000 7ff6837fe700 1 -- 172.21.15.104:0/908583263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678011d20 con 0x7ff6840ce510 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 -- 172.21.15.104:0/908583263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 msgr2=0x7ff6840ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 --2- 172.21.15.104:0/908583263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840ce8e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7ff6840b4650 tx=0x7ff678009b20 comp rx=0 tx=0).stop 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 -- 172.21.15.104:0/908583263 shutdown_connections 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 --2- 172.21.15.104:0/908583263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840ce8e0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 --2- 172.21.15.104:0/908583263 >> v2:172.21.15.114:3300/0 conn(0x7ff68415b190 0x7ff684070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 -- 172.21.15.104:0/908583263 >> 172.21.15.104:0/908583263 conn(0x7ff68406aac0 msgr2=0x7ff6840b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 -- 172.21.15.104:0/908583263 shutdown_connections 2023-06-10T18:58:09.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.341+0000 7ff68b109700 1 -- 172.21.15.104:0/908583263 wait complete. 2023-06-10T18:58:09.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 Processor -- start 2023-06-10T18:58:09.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 -- start start 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff68415b190 0x7ff684153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff68415dad0 0x7ff684155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840c9c20 con 0x7ff6840ce510 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840c9ed0 con 0x7ff68415dad0 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.343+0000 7ff68b109700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840ca1b0 con 0x7ff68415b190 2023-06-10T18:58:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39672/0 (socket says 172.21.15.104:39672) 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 -- 172.21.15.104:0/1292044493 learned_addr learned my addr 172.21.15.104:0/1292044493 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff6896a6700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff68415dad0 0x7ff684155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff683fff700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff68415b190 0x7ff684153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 -- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff68415b190 msgr2=0x7ff684153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff68415b190 0x7ff684153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 -- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff68415dad0 msgr2=0x7ff684155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff68415dad0 0x7ff684155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 -- 172.21.15.104:0/1292044493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6780094a0 con 0x7ff6840ce510 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff688ea5700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff678000c00 tx=0x7ff678004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678004550 con 0x7ff6840ce510 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67801e070 con 0x7ff6840ce510 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6840ca490 con 0x7ff6840ce510 2023-06-10T18:58:09.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6780038e0 con 0x7ff6840ce510 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.344+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6840caa50 con 0x7ff6840ce510 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.345+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff684074050 con 0x7ff6840ce510 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.346+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff678003a40 con 0x7ff6840ce510 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.346+0000 7ff6817fa700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 0x7ff66806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.346+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff67808a430 con 0x7ff6840ce510 2023-06-10T18:58:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.346+0000 7ff683fff700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 0x7ff66806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.347+0000 7ff683fff700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 0x7ff66806d190 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff6840b4620 tx=0x7ff670009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.350+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff678055400 con 0x7ff6840ce510 2023-06-10T18:58:09.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.536+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff6840b0980 con 0x7ff6840ce510 2023-06-10T18:58:09.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:58:09.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.537+0000 7ff6817fa700 1 -- 172.21.15.104:0/1292044493 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v160) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff678058a20 con 0x7ff6840ce510 2023-06-10T18:58:09.545 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 msgr2=0x7ff66806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 0x7ff66806d190 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff6840b4620 tx=0x7ff670009210 comp rx=0 tx=0).stop 2023-06-10T18:58:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 msgr2=0x7ff6840c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff678000c00 tx=0x7ff678004750 comp rx=0 tx=0).stop 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 shutdown_connections 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff66806ace0 0x7ff66806d190 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff68415dad0 0x7ff684155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6840ce510 0x7ff6840c96e0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 --2- 172.21.15.104:0/1292044493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff68415b190 0x7ff684153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 >> 172.21.15.104:0/1292044493 conn(0x7ff68406aac0 msgr2=0x7ff68415d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 shutdown_connections 2023-06-10T18:58:09.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.539+0000 7ff68b109700 1 -- 172.21.15.104:0/1292044493 wait complete. 2023-06-10T18:58:09.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:58:09.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:09.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.714+0000 7f65e4c75700 1 Processor -- start 2023-06-10T18:58:09.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.714+0000 7f65e4c75700 1 -- start start 2023-06-10T18:58:09.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65e4c75700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65e006e6e0 0x7f65e00b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65e4c75700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e006e250 con 0x7f65e015c580 2023-06-10T18:58:09.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65e4c75700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e015c950 con 0x7f65e006cde0 2023-06-10T18:58:09.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65e4c75700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00b1e80 con 0x7f65e006e6e0 2023-06-10T18:58:09.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65ddd9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f65e006e6e0 0x7f65e00b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65de59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f65e015c580 0x7f65e00b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37508/0 (socket says 172.21.15.104:37508) 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65de59c700 1 -- 172.21.15.104:0/1553903769 learned_addr learned my addr 172.21.15.104:0/1553903769 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65ddd9b700 1 -- 172.21.15.104:0/1553903769 >> v1:172.21.15.114:6789/0 conn(0x7f65e006cde0 legacy=0x7f65e00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65ddd9b700 1 -- 172.21.15.104:0/1553903769 >> v1:172.21.15.104:6789/0 conn(0x7f65e015c580 legacy=0x7f65e00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65ddd9b700 1 -- 172.21.15.104:0/1553903769 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e006d920 con 0x7f65e006e6e0 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.715+0000 7f65dd59a700 1 -- 172.21.15.104:0/1553903769 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162025203 0 0) 0x7f65e006e250 con 0x7f65e015c580 2023-06-10T18:58:09.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65ddd9b700 1 --2- 172.21.15.104:0/1553903769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e006e6e0 0x7f65e00b1780 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009840 tx=0x7f65d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba263e93614e030 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65dd59a700 1 -- 172.21.15.104:0/1553903769 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004230 con 0x7f65e006e6e0 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65dd59a700 1 -- 172.21.15.104:0/1553903769 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0004390 con 0x7f65e006e6e0 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65dd59a700 1 -- 172.21.15.104:0/1553903769 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d00036a0 con 0x7f65e006e6e0 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 -- 172.21.15.104:0/1553903769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e006e6e0 msgr2=0x7f65e00b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 --2- 172.21.15.104:0/1553903769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e006e6e0 0x7f65e00b1780 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009840 tx=0x7f65d0009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 -- 172.21.15.104:0/1553903769 shutdown_connections 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 --2- 172.21.15.104:0/1553903769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e006e6e0 0x7f65e00b1780 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 -- 172.21.15.104:0/1553903769 >> 172.21.15.104:0/1553903769 conn(0x7f65e006b8a0 msgr2=0x7f65e00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 -- 172.21.15.104:0/1553903769 shutdown_connections 2023-06-10T18:58:09.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.716+0000 7f65e4c75700 1 -- 172.21.15.104:0/1553903769 wait complete. 2023-06-10T18:58:09.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.718+0000 7f65e4c75700 1 Processor -- start 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.718+0000 7f65e4c75700 1 -- start start 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.718+0000 7f65e4c75700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65e006cde0 0x7f65e01e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.718+0000 7f65e4c75700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e006e6e0 0x7f65e01f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.718+0000 7f65e4c75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65e4c75700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e014d470 con 0x7f65e015c580 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65e4c75700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01f60f0 con 0x7f65e006e6e0 2023-06-10T18:58:09.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65e4c75700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01e5500 con 0x7f65e006cde0 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39690/0 (socket says 172.21.15.104:39690) 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 -- 172.21.15.104:0/981071415 learned_addr learned my addr 172.21.15.104:0/981071415 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ddd9b700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e006e6e0 0x7f65e01f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65de59c700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65e006cde0 0x7f65e01e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 -- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65e006cde0 msgr2=0x7f65e01e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.729 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65e006cde0 0x7f65e01e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 -- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e006e6e0 msgr2=0x7f65e01f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e006e6e0 0x7f65e01f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 -- 172.21.15.104:0/981071415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00094a0 con 0x7f65e015c580 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.719+0000 7f65ded9d700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c7a0 tx=0x7f65d800cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.721+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8003e10 con 0x7f65e015c580 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.721+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8005070 con 0x7f65e015c580 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.721+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8005420 con 0x7f65e015c580 2023-06-10T18:58:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.721+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e01e57e0 con 0x7f65e015c580 2023-06-10T18:58:09.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.721+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e01e5da0 con 0x7f65e015c580 2023-06-10T18:58:09.731 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.722+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e0074050 con 0x7f65e015c580 2023-06-10T18:58:09.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.723+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f65d8005580 con 0x7f65e015c580 2023-06-10T18:58:09.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.723+0000 7f65ceffd700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 0x7f65c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:09.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.723+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f65d8088700 con 0x7f65e015c580 2023-06-10T18:58:09.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.723+0000 7f65de59c700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 0x7f65c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:09.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.724+0000 7f65de59c700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 0x7f65c006d190 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f65d4007fd0 tx=0x7f65d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:09.733 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.727+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f65d8053640 con 0x7f65e015c580 2023-06-10T18:58:09.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.912+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65e01f1a60 con 0x7f65e015c580 2023-06-10T18:58:09.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.913+0000 7f65ceffd700 1 -- 172.21.15.104:0/981071415 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v160) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65d8056c60 con 0x7f65e015c580 2023-06-10T18:58:09.919 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:09.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.914+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 msgr2=0x7f65c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.914+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 0x7f65c006d190 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f65d4007fd0 tx=0x7f65d4009040 comp rx=0 tx=0).stop 2023-06-10T18:58:09.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.914+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 msgr2=0x7f65e01f1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.914+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c7a0 tx=0x7f65d800cab0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.914+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 shutdown_connections 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65e015c580 0x7f65e01f1520 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65e006cde0 0x7f65e01e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f65c006ace0 0x7f65c006d190 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 --2- 172.21.15.104:0/981071415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e006e6e0 0x7f65e01f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:09.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 >> 172.21.15.104:0/981071415 conn(0x7f65e006b8a0 msgr2=0x7f65e00cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:09.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 shutdown_connections 2023-06-10T18:58:09.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:09.915+0000 7f65e4c75700 1 -- 172.21.15.104:0/981071415 wait complete. 2023-06-10T18:58:09.934 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:10.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.048+0000 7fbd4c497700 1 Processor -- start 2023-06-10T18:58:10.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.048+0000 7fbd4c497700 1 -- start start 2023-06-10T18:58:10.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4c497700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd441567f0 0x7fbd440cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4c497700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd4415c7a0 0x7fbd440cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4c497700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd440cb2d0 con 0x7fbd440701a0 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4c497700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4415cca0 con 0x7fbd441567f0 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4c497700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4415cf80 con 0x7fbd4415c7a0 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd441567f0 0x7fbd440cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbd441567f0 0x7fbd440cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39694/0 (socket says 172.21.15.104:39694) 2023-06-10T18:58:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 -- 172.21.15.104:0/2658773139 learned_addr learned my addr 172.21.15.104:0/2658773139 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd49a32700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd4415c7a0 0x7fbd440cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 -- 172.21.15.104:0/2658773139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd4415c7a0 msgr2=0x7fbd440cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd4415c7a0 0x7fbd440cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 -- 172.21.15.104:0/2658773139 >> v1:172.21.15.114:6789/0 conn(0x7fbd440701a0 legacy=0x7fbd440705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 -- 172.21.15.104:0/2658773139 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4415d260 con 0x7fbd441567f0 2023-06-10T18:58:10.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd4a233700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd441567f0 0x7fbd440cad90 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fbd400098f0 tx=0x7fbd40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef3152af61ad97b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.058 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd49231700 1 -- 172.21.15.104:0/2658773139 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000b8a0 con 0x7fbd441567f0 2023-06-10T18:58:10.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd49231700 1 -- 172.21.15.104:0/2658773139 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4000ba00 con 0x7fbd441567f0 2023-06-10T18:58:10.059 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.049+0000 7fbd49231700 1 -- 172.21.15.104:0/2658773139 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000bd90 con 0x7fbd441567f0 2023-06-10T18:58:10.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 -- 172.21.15.104:0/2658773139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd441567f0 msgr2=0x7fbd440cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd441567f0 0x7fbd440cad90 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fbd400098f0 tx=0x7fbd40009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:10.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 -- 172.21.15.104:0/2658773139 shutdown_connections 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd441567f0 0x7fbd440cad90 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 --2- 172.21.15.104:0/2658773139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd4415c7a0 0x7fbd440cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 -- 172.21.15.104:0/2658773139 >> 172.21.15.104:0/2658773139 conn(0x7fbd4406e370 msgr2=0x7fbd4406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 -- 172.21.15.104:0/2658773139 shutdown_connections 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.050+0000 7fbd4c497700 1 -- 172.21.15.104:0/2658773139 wait complete. 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.051+0000 7fbd4c497700 1 Processor -- start 2023-06-10T18:58:10.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.051+0000 7fbd4c497700 1 -- start start 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd440701a0 0x7fbd441f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd441567f0 0x7fbd441f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd441f1f30 con 0x7fbd4415c7a0 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440c43f0 con 0x7fbd440701a0 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4c497700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440b3730 con 0x7fbd441567f0 2023-06-10T18:58:10.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39704/0 (socket says 172.21.15.104:39704) 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 -- 172.21.15.104:0/55697828 learned_addr learned my addr 172.21.15.104:0/55697828 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd49a32700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd441567f0 0x7fbd441f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4a233700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd440701a0 0x7fbd441f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 -- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd441567f0 msgr2=0x7fbd441f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd441567f0 0x7fbd441f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 -- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd440701a0 msgr2=0x7fbd441f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd440701a0 0x7fbd441f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.052+0000 7fbd4aa34700 1 -- 172.21.15.104:0/55697828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40009550 con 0x7fbd4415c7a0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd4aa34700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00cb80 tx=0x7fbd3c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c003c00 con 0x7fbd4415c7a0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c003d60 con 0x7fbd4415c7a0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c00abb0 con 0x7fbd4415c7a0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd441f64d0 con 0x7fbd4415c7a0 2023-06-10T18:58:10.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.053+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd441f5f10 con 0x7fbd4415c7a0 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.054+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd440b38e0 con 0x7fbd4415c7a0 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.054+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbd3c00ad10 con 0x7fbd4415c7a0 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.054+0000 7fbd3affd700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 0x7fbd2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.054+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fbd3c089630 con 0x7fbd4415c7a0 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.055+0000 7fbd4a233700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 0x7fbd2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.055+0000 7fbd4a233700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 0x7fbd2806d190 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009c40 tx=0x7fbd40008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.059+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbd440b38e0 con 0x7fbd4415c7a0 2023-06-10T18:58:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.258+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd440b38a0 con 0x7fbd4415c7a0 2023-06-10T18:58:10.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.258+0000 7fbd3affd700 1 -- 172.21.15.104:0/55697828 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v160) v1 ==== 80+0+57 (secure 0 0 0) 0x7fbd3c054570 con 0x7fbd4415c7a0 2023-06-10T18:58:10.265 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/3682236906 2023-06-10T19:57:58.572945+0000 2023-06-10T18:58:10.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 msgr2=0x7fbd2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 0x7fbd2806d190 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009c40 tx=0x7fbd40008030 comp rx=0 tx=0).stop 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 msgr2=0x7fbd4415d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00cb80 tx=0x7fbd3c00cf40 comp rx=0 tx=0).stop 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 shutdown_connections 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbd2806ace0 0x7fbd2806d190 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd440701a0 0x7fbd441f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbd4415c7a0 0x7fbd4415d940 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 --2- 172.21.15.104:0/55697828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd441567f0 0x7fbd441f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 >> 172.21.15.104:0/55697828 conn(0x7fbd4406e370 msgr2=0x7fbd4415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:10.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 shutdown_connections 2023-06-10T18:58:10.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.260+0000 7fbd4c497700 1 -- 172.21.15.104:0/55697828 wait complete. 2023-06-10T18:58:10.268 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:58:10.278 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:58:10.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:10.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:10.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[157294]: fuse finished with error 0 and tester_r 0 2023-06-10T18:58:10.317 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:58:10.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:10.318 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:58:10.374 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:10.375 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:10.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.528+0000 7f576e040700 1 Processor -- start 2023-06-10T18:58:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.528+0000 7f576e040700 1 -- start start 2023-06-10T18:58:10.535 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f576e040700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f57680cf100 0x7f576806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f576e040700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576806d290 con 0x7f576806e0e0 2023-06-10T18:58:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f576e040700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57680ca2d0 con 0x7f576806e4b0 2023-06-10T18:58:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f576e040700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5768071640 con 0x7f57680cf100 2023-06-10T18:58:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f57677fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f576806e0e0 0x7f57680c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37536/0 (socket says 172.21.15.104:37536) 2023-06-10T18:58:10.536 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f5767fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f57680cf100 0x7f576806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f57677fe700 1 -- 172.21.15.104:0/1775853820 learned_addr learned my addr 172.21.15.104:0/1775853820 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f5767fff700 1 -- 172.21.15.104:0/1775853820 >> v1:172.21.15.114:6789/0 conn(0x7f576806e4b0 legacy=0x7f57680ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f5767fff700 1 -- 172.21.15.104:0/1775853820 >> v1:172.21.15.104:6789/0 conn(0x7f576806e0e0 legacy=0x7f57680c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f57667fc700 1 -- 172.21.15.104:0/1775853820 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927409487 0 0) 0x7f576806d290 con 0x7f576806e0e0 2023-06-10T18:58:10.537 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f5767fff700 1 -- 172.21.15.104:0/1775853820 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576806d4c0 con 0x7f57680cf100 2023-06-10T18:58:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.529+0000 7f5767fff700 1 --2- 172.21.15.104:0/1775853820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57680cf100 0x7f576806cd50 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f5760009d90 tx=0x7f576000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca33d026f6aa87f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f57667fc700 1 -- 172.21.15.104:0/1775853820 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000e070 con 0x7f57680cf100 2023-06-10T18:58:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f57667fc700 1 -- 172.21.15.104:0/1775853820 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576000bbd0 con 0x7f57680cf100 2023-06-10T18:58:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f57667fc700 1 -- 172.21.15.104:0/1775853820 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760012430 con 0x7f57680cf100 2023-06-10T18:58:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 -- 172.21.15.104:0/1775853820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57680cf100 msgr2=0x7f576806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.540 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 --2- 172.21.15.104:0/1775853820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57680cf100 0x7f576806cd50 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f5760009d90 tx=0x7f576000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 -- 172.21.15.104:0/1775853820 shutdown_connections 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 --2- 172.21.15.104:0/1775853820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57680cf100 0x7f576806cd50 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 -- 172.21.15.104:0/1775853820 >> 172.21.15.104:0/1775853820 conn(0x7f576806b8a0 msgr2=0x7f57680cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 -- 172.21.15.104:0/1775853820 shutdown_connections 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.530+0000 7f576e040700 1 -- 172.21.15.104:0/1775853820 wait complete. 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.531+0000 7f576e040700 1 Processor -- start 2023-06-10T18:58:10.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 -- start start 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f576806e0e0 0x7f57681e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57680cf100 0x7f57681f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5768158050 con 0x7f576806e4b0 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680c71d0 con 0x7f57680cf100 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f576e040700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57680c74b0 con 0x7f576806e0e0 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f5766ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f5766ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39716/0 (socket says 172.21.15.104:39716) 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.532+0000 7f5766ffd700 1 -- 172.21.15.104:0/545259059 learned_addr learned my addr 172.21.15.104:0/545259059 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5767fff700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57680cf100 0x7f57681f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f57677fe700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f576806e0e0 0x7f57681e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 -- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f576806e0e0 msgr2=0x7f57681e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f576806e0e0 0x7f57681e5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 -- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57680cf100 msgr2=0x7f57681f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57680cf100 0x7f57681f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 -- 172.21.15.104:0/545259059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5760009a70 con 0x7f576806e4b0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f5766ffd700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f575800c4e0 tx=0x7f575800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580052d0 con 0x7f576806e4b0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758005430 con 0x7f576806e4b0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57680c7790 con 0x7f576806e4b0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580038c0 con 0x7f576806e4b0 2023-06-10T18:58:10.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.533+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57680c7f90 con 0x7f576806e4b0 2023-06-10T18:58:10.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.534+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57680c7cb0 con 0x7f576806e4b0 2023-06-10T18:58:10.545 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.534+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5758003a20 con 0x7f576806e4b0 2023-06-10T18:58:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.535+0000 7f574ffff700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 0x7f575406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.535+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f5758088a20 con 0x7f576806e4b0 2023-06-10T18:58:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.535+0000 7f57677fe700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 0x7f575406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.535+0000 7f57677fe700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 0x7f575406d190 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f575c007fd0 tx=0x7f575c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.539+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f57680c7cb0 con 0x7f576806e4b0 2023-06-10T18:58:10.734 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.728+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57680c7cb0 con 0x7f576806e4b0 2023-06-10T18:58:10.738 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.732+0000 7f574ffff700 1 -- 172.21.15.104:0/545259059 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7f57680c7cb0 con 0x7f576806e4b0 2023-06-10T18:58:10.738 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:10.739 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 msgr2=0x7f575406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.740 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 0x7f575406d190 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f575c007fd0 tx=0x7f575c009040 comp rx=0 tx=0).stop 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 msgr2=0x7f57681f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f575800c4e0 tx=0x7f575800c7f0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 shutdown_connections 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f575406ace0 0x7f575406d190 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57680cf100 0x7f57681f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f576806e0e0 0x7f57681e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 --2- 172.21.15.104:0/545259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f576806e4b0 0x7f57681f5570 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 >> 172.21.15.104:0/545259059 conn(0x7f576806b8a0 msgr2=0x7f57680cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:10.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 shutdown_connections 2023-06-10T18:58:10.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.734+0000 7f576e040700 1 -- 172.21.15.104:0/545259059 wait complete. 2023-06-10T18:58:10.759 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:42.097445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:43.215419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:10.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:10.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 Processor -- start 2023-06-10T18:58:10.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 -- start start 2023-06-10T18:58:10.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb3680cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3680d23f0 0x7fb3680d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368145ba0 con 0x7fb3680d0a60 2023-06-10T18:58:10.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3680d0ee0 con 0x7fb3680d23f0 2023-06-10T18:58:10.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.878+0000 7fb36ee58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680d2890 con 0x7fb3680cec90 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb36d3f5700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb3680d0a60 0x7fb36815fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37554/0 (socket says 172.21.15.104:37554) 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb36d3f5700 1 -- 172.21.15.104:0/1265527637 learned_addr learned my addr 172.21.15.104:0/1265527637 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb367fff700 1 --2- 172.21.15.104:0/1265527637 >> v2:172.21.15.114:3300/0 conn(0x7fb3680d23f0 0x7fb3680d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb3680cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844635087 0 0) 0x7fb368145ba0 con 0x7fb3680d0a60 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3580035b0 con 0x7fb3680d0a60 2023-06-10T18:58:10.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 351321846 0 0) 0x7fb3580035b0 con 0x7fb3680d0a60 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 msgr2=0x7fb3680cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb3680cfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 >> v2:172.21.15.114:3300/0 conn(0x7fb3680d23f0 msgr2=0x7fb3680d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 --2- 172.21.15.104:0/1265527637 >> v2:172.21.15.114:3300/0 conn(0x7fb3680d23f0 0x7fb3680d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680d2b70 con 0x7fb3680d0a60 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb3600019f0 con 0x7fb3680d0a60 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 >> v1:172.21.15.104:6789/0 conn(0x7fb3680d0a60 legacy=0x7fb36815fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb358003a00 0x7fb358003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358004430 0x7fb3580048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb358005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680d0ee0 con 0x7fb358003a00 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680d2890 con 0x7fb358004430 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35800ba10 con 0x7fb3680cec90 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb358003a00 0x7fb358003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb367fff700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358004430 0x7fb3580048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.879+0000 7fb36d3f5700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb358005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 -- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 msgr2=0x7fb358005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3680cec90 0x7fb358005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 -- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358004430 msgr2=0x7fb3580048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358004430 0x7fb3580048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 -- 172.21.15.104:0/1265527637 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580035b0 con 0x7fb358003a00 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb367fff700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358004430 0x7fb3580048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:10.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb358003a00 0x7fb358003e10 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009b90 tx=0x7fb358005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c9ba20bc8f0784 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00e6c0 con 0x7fb358003a00 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c00e820 con 0x7fb358003a00 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb3677fe700 1 -- 172.21.15.104:0/1265527637 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00eb90 con 0x7fb358003a00 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36ee58700 1 -- 172.21.15.104:0/1265527637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb358003a00 msgr2=0x7fb358003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36ee58700 1 --2- 172.21.15.104:0/1265527637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb358003a00 0x7fb358003e10 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009b90 tx=0x7fb358005880 comp rx=0 tx=0).stop 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36d3f5700 1 -- 172.21.15.104:0/1265527637 reap_dead start 2023-06-10T18:58:10.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36ee58700 1 -- 172.21.15.104:0/1265527637 shutdown_connections 2023-06-10T18:58:10.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36ee58700 1 -- 172.21.15.104:0/1265527637 >> 172.21.15.104:0/1265527637 conn(0x7fb368071ec0 msgr2=0x7fb36809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:10.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.880+0000 7fb36ee58700 1 -- 172.21.15.104:0/1265527637 shutdown_connections 2023-06-10T18:58:10.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.881+0000 7fb36ee58700 1 -- 172.21.15.104:0/1265527637 wait complete. 2023-06-10T18:58:10.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.882+0000 7fb36ee58700 1 Processor -- start 2023-06-10T18:58:10.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.882+0000 7fb36ee58700 1 -- start start 2023-06-10T18:58:10.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.882+0000 7fb36ee58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358003a00 0x7fb3681f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681f2f70 0x7fb3681ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681f2d70 con 0x7fb3680b0db0 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681f34f0 con 0x7fb358003a00 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681f37d0 con 0x7fb3681f2f70 2023-06-10T18:58:10.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39738/0 (socket says 172.21.15.104:39738) 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 -- 172.21.15.104:0/743138514 learned_addr learned my addr 172.21.15.104:0/743138514 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358003a00 0x7fb3681f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb367fff700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681f2f70 0x7fb3681ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 -- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681f2f70 msgr2=0x7fb3681ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681f2f70 0x7fb3681ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 -- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358003a00 msgr2=0x7fb3681f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:10.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358003a00 0x7fb3681f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 -- 172.21.15.104:0/743138514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3681f3ab0 con 0x7fb3680b0db0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36d3f5700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fb360001330 tx=0x7fb360007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36000ebb0 con 0x7fb3680b0db0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36000ed30 con 0x7fb3680b0db0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3681f3d90 con 0x7fb3680b0db0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360013660 con 0x7fb3680b0db0 2023-06-10T18:58:10.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.883+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3681eff30 con 0x7fb3680b0db0 2023-06-10T18:58:10.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.884+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3681f42b0 con 0x7fb3680b0db0 2023-06-10T18:58:10.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.885+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb3600137c0 con 0x7fb3680b0db0 2023-06-10T18:58:10.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.885+0000 7fb3657fa700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:10.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.885+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb360089bc0 con 0x7fb3680b0db0 2023-06-10T18:58:10.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.885+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:10.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.886+0000 7fb36cbf4700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0090b0 tx=0x7fb35c01f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:10.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:10.889+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb3681f42b0 con 0x7fb3680b0db0 2023-06-10T18:58:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.078+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3681f42b0 con 0x7fb3680b0db0 2023-06-10T18:58:11.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.079+0000 7fb3657fa700 1 -- 172.21.15.104:0/743138514 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v160) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb360054a60 con 0x7fb3680b0db0 2023-06-10T18:58:11.086 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 msgr2=0x7fb34c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0090b0 tx=0x7fb35c01f040 comp rx=0 tx=0).stop 2023-06-10T18:58:11.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 msgr2=0x7fb368157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fb360001330 tx=0x7fb360007c90 comp rx=0 tx=0).stop 2023-06-10T18:58:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 shutdown_connections 2023-06-10T18:58:11.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb34c06ace0 0x7fb34c06d190 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb358003a00 0x7fb3681f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3680b0db0 0x7fb368157590 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 --2- 172.21.15.104:0/743138514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681f2f70 0x7fb3681ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 >> 172.21.15.104:0/743138514 conn(0x7fb368071ec0 msgr2=0x7fb36806b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 shutdown_connections 2023-06-10T18:58:11.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.081+0000 7fb36ee58700 1 -- 172.21.15.104:0/743138514 wait complete. 2023-06-10T18:58:11.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:11.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.216+0000 7f5463542700 1 Processor -- start 2023-06-10T18:58:11.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.216+0000 7f5463542700 1 -- start start 2023-06-10T18:58:11.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5463542700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 0x7f545c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5463542700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f545c06cde0 0x7f545c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5463542700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c06e250 con 0x7f545c06e6e0 2023-06-10T18:58:11.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5463542700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545c15c950 con 0x7f545c06cde0 2023-06-10T18:58:11.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5463542700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545c0b1e80 con 0x7f545c15c580 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f545c06cde0 0x7f545c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5460add700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f545c06e6e0 0x7f545c0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37570/0 (socket says 172.21.15.104:37570) 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5460add700 1 -- 172.21.15.104:0/3427275672 learned_addr learned my addr 172.21.15.104:0/3427275672 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f54612de700 1 --2- 172.21.15.104:0/3427275672 >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 0x7f545c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 -- 172.21.15.104:0/3427275672 >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 msgr2=0x7f545c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 --2- 172.21.15.104:0/3427275672 >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 0x7f545c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 -- 172.21.15.104:0/3427275672 >> v1:172.21.15.104:6789/0 conn(0x7f545c06e6e0 legacy=0x7f545c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 -- 172.21.15.104:0/3427275672 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c06d920 con 0x7f545c06cde0 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5453fff700 1 -- 172.21.15.104:0/3427275672 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415398763 0 0) 0x7f545c06e250 con 0x7f545c06e6e0 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f54612de700 1 --2- 172.21.15.104:0/3427275672 >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 0x7f545c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5461adf700 1 --2- 172.21.15.104:0/3427275672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06cde0 0x7f545c0ce0d0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f545c070300 tx=0x7f545400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e3306ebda6a78 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5453fff700 1 -- 172.21.15.104:0/3427275672 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545400e070 con 0x7f545c06cde0 2023-06-10T18:58:11.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5453fff700 1 -- 172.21.15.104:0/3427275672 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545400bbd0 con 0x7f545c06cde0 2023-06-10T18:58:11.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.217+0000 7f5453fff700 1 -- 172.21.15.104:0/3427275672 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454012430 con 0x7f545c06cde0 2023-06-10T18:58:11.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 -- 172.21.15.104:0/3427275672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06cde0 msgr2=0x7f545c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 --2- 172.21.15.104:0/3427275672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06cde0 0x7f545c0ce0d0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f545c070300 tx=0x7f545400b330 comp rx=0 tx=0).stop 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 -- 172.21.15.104:0/3427275672 shutdown_connections 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 --2- 172.21.15.104:0/3427275672 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06cde0 0x7f545c0ce0d0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 --2- 172.21.15.104:0/3427275672 >> v2:172.21.15.114:3300/0 conn(0x7f545c15c580 0x7f545c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 -- 172.21.15.104:0/3427275672 >> 172.21.15.104:0/3427275672 conn(0x7f545c06b8a0 msgr2=0x7f545c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 -- 172.21.15.104:0/3427275672 shutdown_connections 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.218+0000 7f5463542700 1 -- 172.21.15.104:0/3427275672 wait complete. 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.219+0000 7f5463542700 1 Processor -- start 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.219+0000 7f5463542700 1 -- start start 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f545c06cde0 0x7f545c158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f545c15c580 0x7f545c0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c0b2ad0 con 0x7f545c06e6e0 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c158ea0 con 0x7f545c15c580 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5463542700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545c0b24c0 con 0x7f545c06cde0 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39748/0 (socket says 172.21.15.104:39748) 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 -- 172.21.15.104:0/349254894 learned_addr learned my addr 172.21.15.104:0/349254894 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f54612de700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f545c06cde0 0x7f545c158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5461adf700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f545c15c580 0x7f545c0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 -- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f545c06cde0 msgr2=0x7f545c158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f545c06cde0 0x7f545c158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 -- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f545c15c580 msgr2=0x7f545c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f545c15c580 0x7f545c0cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 -- 172.21.15.104:0/349254894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454009a70 con 0x7f545c06e6e0 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5460add700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c570 tx=0x7f544c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.220+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c004c80 con 0x7f545c06e6e0 2023-06-10T18:58:11.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.221+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c004e00 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.221+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c0b5110 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.221+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c007310 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.221+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c0b53f0 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.222+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545c0b2c80 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.222+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f544c007470 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.222+0000 7f5451ffb700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 0x7f544006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.223+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f544c089370 con 0x7f545c06e6e0 2023-06-10T18:58:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.223+0000 7f54612de700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 0x7f544006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.223+0000 7f54612de700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 0x7f544006d190 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f545800c920 tx=0x7f5458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.227+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f545c0b2c80 con 0x7f545c06e6e0 2023-06-10T18:58:11.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.412+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f545c0b2c80 con 0x7f545c06e6e0 2023-06-10T18:58:11.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.413+0000 7f5451ffb700 1 -- 172.21.15.104:0/349254894 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7f545c0b2c80 con 0x7f545c06e6e0 2023-06-10T18:58:11.420 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:11.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 msgr2=0x7f544006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 0x7f544006d190 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f545800c920 tx=0x7f5458008040 comp rx=0 tx=0).stop 2023-06-10T18:58:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 msgr2=0x7f545c1e4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c570 tx=0x7f544c00c930 comp rx=0 tx=0).stop 2023-06-10T18:58:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 shutdown_connections 2023-06-10T18:58:11.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f544006ace0 0x7f544006d190 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f545c15c580 0x7f545c0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f545c06cde0 0x7f545c158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 --2- 172.21.15.104:0/349254894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f545c06e6e0 0x7f545c1e4f40 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 >> 172.21.15.104:0/349254894 conn(0x7f545c06b8a0 msgr2=0x7f545c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 shutdown_connections 2023-06-10T18:58:11.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.415+0000 7f5463542700 1 -- 172.21.15.104:0/349254894 wait complete. 2023-06-10T18:58:11.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:11.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.550+0000 7f738f01d700 1 Processor -- start 2023-06-10T18:58:11.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.550+0000 7f738f01d700 1 -- start start 2023-06-10T18:58:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 0x7f73880b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f73880b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73880aec00 0x7f73880ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73880ae7e0 con 0x7f73880aec00 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738806df90 con 0x7f738815c6f0 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738f01d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738806e270 con 0x7f738806ea70 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73880aec00 0x7f73880ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f73880aec00 0x7f73880ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39758/0 (socket says 172.21.15.104:39758) 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 -- 172.21.15.104:0/2037313507 learned_addr learned my addr 172.21.15.104:0/2037313507 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:11.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f7387fff700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f73880b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738cdb9700 1 --2- 172.21.15.104:0/2037313507 >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 0x7f73880b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 -- 172.21.15.104:0/2037313507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 msgr2=0x7f73880b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f73880b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 -- 172.21.15.104:0/2037313507 >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 msgr2=0x7f73880b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 --2- 172.21.15.104:0/2037313507 >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 0x7f73880b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 -- 172.21.15.104:0/2037313507 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738806e550 con 0x7f73880aec00 2023-06-10T18:58:11.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f7387fff700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f73880b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738cdb9700 1 --2- 172.21.15.104:0/2037313507 >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 0x7f73880b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f738d5ba700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73880aec00 0x7f73880ac1c0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f73880d0660 tx=0x7f738000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207a90731b593edb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f73877fe700 1 -- 172.21.15.104:0/2037313507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738000e070 con 0x7f73880aec00 2023-06-10T18:58:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.551+0000 7f73877fe700 1 -- 172.21.15.104:0/2037313507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738000b9c0 con 0x7f73880aec00 2023-06-10T18:58:11.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f73877fe700 1 -- 172.21.15.104:0/2037313507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738000bd70 con 0x7f73880aec00 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 -- 172.21.15.104:0/2037313507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73880aec00 msgr2=0x7f73880ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73880aec00 0x7f73880ac1c0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f73880d0660 tx=0x7f738000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 -- 172.21.15.104:0/2037313507 shutdown_connections 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73880aec00 0x7f73880ac1c0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 --2- 172.21.15.104:0/2037313507 >> v2:172.21.15.114:3300/0 conn(0x7f738815c6f0 0x7f73880b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 --2- 172.21.15.104:0/2037313507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f73880b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 -- 172.21.15.104:0/2037313507 >> 172.21.15.104:0/2037313507 conn(0x7f738806b8a0 msgr2=0x7f73880cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 -- 172.21.15.104:0/2037313507 shutdown_connections 2023-06-10T18:58:11.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.552+0000 7f738f01d700 1 -- 172.21.15.104:0/2037313507 wait complete. 2023-06-10T18:58:11.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.553+0000 7f738f01d700 1 Processor -- start 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.553+0000 7f738f01d700 1 -- start start 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f7388153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73880aec00 0x7f73880c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73880cad70 con 0x7f738815c6f0 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73880ca9f0 con 0x7f73880aec00 2023-06-10T18:58:11.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738f01d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73880c70b0 con 0x7f738806ea70 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738d5ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738d5ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39764/0 (socket says 172.21.15.104:39764) 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.554+0000 7f738d5ba700 1 -- 172.21.15.104:0/720257747 learned_addr learned my addr 172.21.15.104:0/720257747 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738cdb9700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f7388153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f7387fff700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73880aec00 0x7f73880c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 -- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 msgr2=0x7f7388153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f7388153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 -- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73880aec00 msgr2=0x7f73880c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73880aec00 0x7f73880c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 -- 172.21.15.104:0/720257747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380009b30 con 0x7f738815c6f0 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738d5ba700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f738000b370 tx=0x7f7380004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380004310 con 0x7f738815c6f0 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7380004470 con 0x7f738815c6f0 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73880c7390 con 0x7f738815c6f0 2023-06-10T18:58:11.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380004820 con 0x7f738815c6f0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.555+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73881540a0 con 0x7f738815c6f0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.556+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73880c78b0 con 0x7f738815c6f0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.556+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f738001c440 con 0x7f738815c6f0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.557+0000 7f73857fa700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 0x7f736c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.557+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f738008ac20 con 0x7f738815c6f0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.557+0000 7f738cdb9700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 0x7f736c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.557+0000 7f738cdb9700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 0x7f736c06d190 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f737c00b840 tx=0x7f737c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.561+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f73880c78b0 con 0x7f738815c6f0 2023-06-10T18:58:11.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.750+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73880caeb0 con 0x7f738815c6f0 2023-06-10T18:58:11.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.750+0000 7f73857fa700 1 -- 172.21.15.104:0/720257747 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v160) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7380020020 con 0x7f738815c6f0 2023-06-10T18:58:11.757 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:11.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 msgr2=0x7f736c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 0x7f736c06d190 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f737c00b840 tx=0x7f737c008040 comp rx=0 tx=0).stop 2023-06-10T18:58:11.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 msgr2=0x7f73880b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f738000b370 tx=0x7f7380004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 shutdown_connections 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f738815c6f0 0x7f73880b4630 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f738806ea70 0x7f7388153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f736c06ace0 0x7f736c06d190 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 --2- 172.21.15.104:0/720257747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73880aec00 0x7f73880c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 >> 172.21.15.104:0/720257747 conn(0x7f738806b8a0 msgr2=0x7f738815b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 shutdown_connections 2023-06-10T18:58:11.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.753+0000 7f738f01d700 1 -- 172.21.15.104:0/720257747 wait complete. 2023-06-10T18:58:11.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:11.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.888+0000 7f939c424700 1 Processor -- start 2023-06-10T18:58:11.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.888+0000 7f939c424700 1 -- start start 2023-06-10T18:58:11.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f939415c580 0x7f93940b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f939406e6e0 0x7f93940b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f9394069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939406e250 con 0x7f939415c580 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939415c950 con 0x7f939406e6e0 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939c424700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b1e80 con 0x7f939406cde0 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939a1c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f939415c580 0x7f93940b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939a1c0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f939415c580 0x7f93940b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39768/0 (socket says 172.21.15.104:39768) 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939a1c0700 1 -- 172.21.15.104:0/103293086 learned_addr learned my addr 172.21.15.104:0/103293086 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:11.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.889+0000 7f939a9c1700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f9394069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f93999bf700 1 --2- 172.21.15.104:0/103293086 >> v2:172.21.15.114:3300/0 conn(0x7f939406e6e0 0x7f93940b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 -- 172.21.15.104:0/103293086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 msgr2=0x7f9394069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f9394069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 -- 172.21.15.104:0/103293086 >> v2:172.21.15.114:3300/0 conn(0x7f939406e6e0 msgr2=0x7f93940b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 --2- 172.21.15.104:0/103293086 >> v2:172.21.15.114:3300/0 conn(0x7f939406e6e0 0x7f93940b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 -- 172.21.15.104:0/103293086 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939406d920 con 0x7f939415c580 2023-06-10T18:58:11.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939a1c0700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939415c580 0x7f93940b1070 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f939406af30 tx=0x7f9384009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccff2a29d459944 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f93991be700 1 -- 172.21.15.104:0/103293086 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384005e30 con 0x7f939415c580 2023-06-10T18:58:11.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f93991be700 1 -- 172.21.15.104:0/103293086 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384004030 con 0x7f939415c580 2023-06-10T18:58:11.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f93991be700 1 -- 172.21.15.104:0/103293086 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93840043e0 con 0x7f939415c580 2023-06-10T18:58:11.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939c424700 1 -- 172.21.15.104:0/103293086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939415c580 msgr2=0x7f93940b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939c424700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939415c580 0x7f93940b1070 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f939406af30 tx=0x7f9384009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.890+0000 7f939c424700 1 -- 172.21.15.104:0/103293086 shutdown_connections 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939415c580 0x7f93940b1070 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 --2- 172.21.15.104:0/103293086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f9394069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 --2- 172.21.15.104:0/103293086 >> v2:172.21.15.114:3300/0 conn(0x7f939406e6e0 0x7f93940b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 -- 172.21.15.104:0/103293086 >> 172.21.15.104:0/103293086 conn(0x7f939406b8a0 msgr2=0x7f93940cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:11.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 -- 172.21.15.104:0/103293086 shutdown_connections 2023-06-10T18:58:11.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.891+0000 7f939c424700 1 -- 172.21.15.104:0/103293086 wait complete. 2023-06-10T18:58:11.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 Processor -- start 2023-06-10T18:58:11.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 -- start start 2023-06-10T18:58:11.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f93940ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f939415c580 0x7f93940c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b3bc0 con 0x7f939406e6e0 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b3ea0 con 0x7f939415c580 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.892+0000 7f939c424700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b4180 con 0x7f939406cde0 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39776/0 (socket says 172.21.15.104:39776) 2023-06-10T18:58:11.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 -- 172.21.15.104:0/4185606193 learned_addr learned my addr 172.21.15.104:0/4185606193 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f939a1c0700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f93940ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f939a9c1700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f939415c580 0x7f93940c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 -- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 msgr2=0x7f93940ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f93940ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 -- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f939415c580 msgr2=0x7f93940c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:11.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f939415c580 0x7f93940c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 -- 172.21.15.104:0/4185606193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9384009500 con 0x7f939406e6e0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f93999bf700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f939000e810 tx=0x7f939000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000c070 con 0x7f939406e6e0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390013610 con 0x7f939406e6e0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93940b4460 con 0x7f939406e6e0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93900139a0 con 0x7f939406e6e0 2023-06-10T18:58:11.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.893+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93940ce980 con 0x7f939406e6e0 2023-06-10T18:58:11.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.894+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93940b4980 con 0x7f939406e6e0 2023-06-10T18:58:11.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.895+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9390013b00 con 0x7f939406e6e0 2023-06-10T18:58:11.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.895+0000 7f938affd700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 0x7f937806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:11.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.895+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f9390089000 con 0x7f939406e6e0 2023-06-10T18:58:11.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.895+0000 7f939a1c0700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 0x7f937806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:11.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.896+0000 7f939a1c0700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 0x7f937806d190 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f939406af30 tx=0x7f9384008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:11.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:11.899+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f93940b4980 con 0x7f939406e6e0 2023-06-10T18:58:12.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.085+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93940b4980 con 0x7f939406e6e0 2023-06-10T18:58:12.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.086+0000 7f938affd700 1 -- 172.21.15.104:0/4185606193 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v160) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93940b4980 con 0x7f939406e6e0 2023-06-10T18:58:12.092 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:12.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 msgr2=0x7f937806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 0x7f937806d190 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f939406af30 tx=0x7f9384008070 comp rx=0 tx=0).stop 2023-06-10T18:58:12.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 msgr2=0x7f93940b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f939000e810 tx=0x7f939000eb20 comp rx=0 tx=0).stop 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 shutdown_connections 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f937806ace0 0x7f937806d190 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f939415c580 0x7f93940c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f939406cde0 0x7f93940ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 --2- 172.21.15.104:0/4185606193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f939406e6e0 0x7f93940b4c60 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.087+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 >> 172.21.15.104:0/4185606193 conn(0x7f939406b8a0 msgr2=0x7f93940cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:12.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.088+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 shutdown_connections 2023-06-10T18:58:12.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.088+0000 7f939c424700 1 -- 172.21.15.104:0/4185606193 wait complete. 2023-06-10T18:58:12.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:12.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.220+0000 7f5b230cb700 1 Processor -- start 2023-06-10T18:58:12.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.220+0000 7f5b230cb700 1 -- start start 2023-06-10T18:58:12.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.220+0000 7f5b230cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b1c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.220+0000 7f5b230cb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0cccd0 con 0x7f5b1c06d150 2023-06-10T18:58:12.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b230cb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c06a010 con 0x7f5b1c06b180 2023-06-10T18:58:12.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b230cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0ccfb0 con 0x7f5b1c158c50 2023-06-10T18:58:12.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b21668700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5b1c06d150 0x7f5b1c0c9b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37612/0 (socket says 172.21.15.104:37612) 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b21668700 1 -- 172.21.15.104:0/1163416037 learned_addr learned my addr 172.21.15.104:0/1163416037 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b1c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47554091 0 0) 0x7f5b1c0cccd0 con 0x7f5b1c06d150 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c0035b0 con 0x7f5b1c06d150 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173608097 0 0) 0x7f5b0c0035b0 con 0x7f5b1c06d150 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 msgr2=0x7f5b1c06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b1c06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 >> v1:172.21.15.114:6789/0 conn(0x7f5b1c06b180 legacy=0x7f5b1c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c06a2f0 con 0x7f5b1c06d150 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5b140019f0 con 0x7f5b1c06d150 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b1c06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 >> v1:172.21.15.104:6789/0 conn(0x7f5b1c06d150 legacy=0x7f5b1c0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0c003a30 0x7f5b0c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0c0043d0 0x7f5b0c004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b0c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0ccfb0 con 0x7f5b0c003a30 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00ba40 con 0x7f5b0c0043d0 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00bc00 con 0x7f5b1c158c50 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0c003a30 0x7f5b0c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b1bfff700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0c0043d0 0x7f5b0c004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.221+0000 7f5b21668700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b0c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 -- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 msgr2=0x7f5b0c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:12.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b1c158c50 0x7f5b0c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 -- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0c0043d0 msgr2=0x7f5b0c004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0c0043d0 0x7f5b0c004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 -- 172.21.15.104:0/1163416037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100094a0 con 0x7f5b0c003a30 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b20e67700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0c003a30 0x7f5b0c003e00 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009d30 tx=0x7f5b0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc3909163829db7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10009e80 con 0x7f5b0c003a30 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1000e9a0 con 0x7f5b0c003a30 2023-06-10T18:58:12.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b1b7fe700 1 -- 172.21.15.104:0/1163416037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1000ed10 con 0x7f5b0c003a30 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b230cb700 1 -- 172.21.15.104:0/1163416037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0c003a30 msgr2=0x7f5b0c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b230cb700 1 --2- 172.21.15.104:0/1163416037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0c003a30 0x7f5b0c003e00 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009d30 tx=0x7f5b0c005880 comp rx=0 tx=0).stop 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b21668700 1 -- 172.21.15.104:0/1163416037 reap_dead start 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b230cb700 1 -- 172.21.15.104:0/1163416037 shutdown_connections 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.222+0000 7f5b230cb700 1 -- 172.21.15.104:0/1163416037 >> 172.21.15.104:0/1163416037 conn(0x7f5b1c0b1610 msgr2=0x7f5b1c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.223+0000 7f5b230cb700 1 -- 172.21.15.104:0/1163416037 shutdown_connections 2023-06-10T18:58:12.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.223+0000 7f5b230cb700 1 -- 172.21.15.104:0/1163416037 wait complete. 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.224+0000 7f5b230cb700 1 Processor -- start 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.224+0000 7f5b230cb700 1 -- start start 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.224+0000 7f5b230cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0c003a30 0x7f5b1c0c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b1c0cbb70 0x7f5b1c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c1497d0 con 0x7f5b1c149360 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0cc520 con 0x7f5b1c0cbb70 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0c5520 con 0x7f5b0c003a30 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39794/0 (socket says 172.21.15.104:39794) 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 -- 172.21.15.104:0/2892367095 learned_addr learned my addr 172.21.15.104:0/2892367095 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b20e67700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0c003a30 0x7f5b1c0c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b21668700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b1c0cbb70 0x7f5b1c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 -- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0c003a30 msgr2=0x7f5b1c0c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0c003a30 0x7f5b1c0c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 -- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b1c0cbb70 msgr2=0x7f5b1c0cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b1c0cbb70 0x7f5b1c0cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 -- 172.21.15.104:0/2892367095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0c5800 con 0x7f5b1c149360 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b1bfff700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f5b08001eb0 tx=0x7f5b08011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080054e0 con 0x7f5b1c149360 2023-06-10T18:58:12.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0800f690 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800fa40 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c0b1fb0 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.225+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c0b2570 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.226+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c0ce820 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.227+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5b08018020 con 0x7f5b1c149360 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.227+0000 7f5b197fa700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 0x7f5b0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.227+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f5b0808b180 con 0x7f5b1c149360 2023-06-10T18:58:12.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.227+0000 7f5b20e67700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 0x7f5b0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.228+0000 7f5b20e67700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 0x7f5b0006d190 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5b100090b0 tx=0x7f5b1001f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.231+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5b08056150 con 0x7f5b1c149360 2023-06-10T18:58:12.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.438+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5b1c074050 con 0x7f5b1c149360 2023-06-10T18:58:12.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.438+0000 7f5b197fa700 1 -- 172.21.15.104:0/2892367095 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v160) v1 ==== 80+0+57 (secure 0 0 0) 0x7f5b08020050 con 0x7f5b1c149360 2023-06-10T18:58:12.445 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/3682236906 2023-06-10T19:57:58.572945+0000 2023-06-10T18:58:12.447 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 msgr2=0x7f5b0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 0x7f5b0006d190 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5b100090b0 tx=0x7f5b1001f040 comp rx=0 tx=0).stop 2023-06-10T18:58:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 msgr2=0x7f5b1c0c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f5b08001eb0 tx=0x7f5b08011cb0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 shutdown_connections 2023-06-10T18:58:12.448 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5b0006ace0 0x7f5b0006d190 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b1c0cbb70 0x7f5b1c0cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0c003a30 0x7f5b1c0c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 --2- 172.21.15.104:0/2892367095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1c149360 0x7f5b1c0c2930 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 >> 172.21.15.104:0/2892367095 conn(0x7f5b1c0b1610 msgr2=0x7f5b1c0ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 shutdown_connections 2023-06-10T18:58:12.449 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.440+0000 7f5b230cb700 1 -- 172.21.15.104:0/2892367095 wait complete. 2023-06-10T18:58:12.450 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:58:12.459 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:58:12.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:12.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:12.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[116776]: fuse finished with error 0 and tester_r 0 2023-06-10T18:58:12.501 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:58:12.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:12.502 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:58:12.595 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:12.596 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:12.614 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:58:12.614 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:58:12.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:12.615 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:58:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:58:12.673 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:58:12.674 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:12.675 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:12.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:58:12.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:58:12.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:12.695 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:58:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:58:12.802 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:58:12.803 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:12.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.916+0000 7f233e3bc700 1 Processor -- start 2023-06-10T18:58:12.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.916+0000 7f233e3bc700 1 -- start start 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233e3bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f233806b1f0 0x7f23380cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233e3bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f233808a3d0 0x7f23380cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233e3bc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380cadd0 con 0x7f23380ce350 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233e3bc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23380cfeb0 con 0x7f233808a3d0 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233e3bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cb0b0 con 0x7f233806b1f0 2023-06-10T18:58:12.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233c959700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f233808a3d0 0x7f23380cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f23377fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f23380ce350 0x7f23380cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37630/0 (socket says 172.21.15.104:37630) 2023-06-10T18:58:12.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f23377fe700 1 -- 172.21.15.104:0/1858299677 learned_addr learned my addr 172.21.15.104:0/1858299677 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:12.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233c959700 1 -- 172.21.15.104:0/1858299677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f233806b1f0 msgr2=0x7f23380cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:12.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233c959700 1 --2- 172.21.15.104:0/1858299677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f233806b1f0 0x7f23380cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233c959700 1 -- 172.21.15.104:0/1858299677 >> v1:172.21.15.104:6789/0 conn(0x7f23380ce350 legacy=0x7f23380cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f233c959700 1 -- 172.21.15.104:0/1858299677 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380cf550 con 0x7f233808a3d0 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.917+0000 7f2336ffd700 1 -- 172.21.15.104:0/1858299677 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855055238 0 0) 0x7f23380cadd0 con 0x7f23380ce350 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233c959700 1 --2- 172.21.15.104:0/1858299677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233808a3d0 0x7f23380cb540 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2330009d90 tx=0x7f233000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa249689e7c0b74 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f2336ffd700 1 -- 172.21.15.104:0/1858299677 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233000e070 con 0x7f233808a3d0 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f2336ffd700 1 -- 172.21.15.104:0/1858299677 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233000bbd0 con 0x7f233808a3d0 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f2336ffd700 1 -- 172.21.15.104:0/1858299677 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330012430 con 0x7f233808a3d0 2023-06-10T18:58:12.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 -- 172.21.15.104:0/1858299677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233808a3d0 msgr2=0x7f23380cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 --2- 172.21.15.104:0/1858299677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233808a3d0 0x7f23380cb540 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2330009d90 tx=0x7f233000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 -- 172.21.15.104:0/1858299677 shutdown_connections 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 --2- 172.21.15.104:0/1858299677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233808a3d0 0x7f23380cb540 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 --2- 172.21.15.104:0/1858299677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f233806b1f0 0x7f23380cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 -- 172.21.15.104:0/1858299677 >> 172.21.15.104:0/1858299677 conn(0x7f2338089fd0 msgr2=0x7f23380d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:12.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 -- 172.21.15.104:0/1858299677 shutdown_connections 2023-06-10T18:58:12.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.918+0000 7f233e3bc700 1 -- 172.21.15.104:0/1858299677 wait complete. 2023-06-10T18:58:12.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.920+0000 7f233e3bc700 1 Processor -- start 2023-06-10T18:58:12.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.920+0000 7f233e3bc700 1 -- start start 2023-06-10T18:58:12.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 0x7f23380adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23380ce350 0x7f23380c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380b03c0 con 0x7f233806b1f0 2023-06-10T18:58:12.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380b11c0 con 0x7f233808a3d0 2023-06-10T18:58:12.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233e3bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380b0640 con 0x7f23380ce350 2023-06-10T18:58:12.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39806/0 (socket says 172.21.15.104:39806) 2023-06-10T18:58:12.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 -- 172.21.15.104:0/3156469647 learned_addr learned my addr 172.21.15.104:0/3156469647 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:12.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f23377fe700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 0x7f23380adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f233c959700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23380ce350 0x7f23380c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 -- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23380ce350 msgr2=0x7f23380c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23380ce350 0x7f23380c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 -- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 msgr2=0x7f23380adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 0x7f23380adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:12.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 -- 172.21.15.104:0/3156469647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009a70 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f23377fe700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 0x7f23380adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.921+0000 7f2337fff700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f232c00b530 tx=0x7f232c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011830 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0119b0 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23380b0920 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011d60 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23380b0ee0 con 0x7f233806b1f0 2023-06-10T18:58:12.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.922+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338074050 con 0x7f233806b1f0 2023-06-10T18:58:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.923+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f232c01a400 con 0x7f233806b1f0 2023-06-10T18:58:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.923+0000 7f2334ff9700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 0x7f232406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.923+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f232c089620 con 0x7f233806b1f0 2023-06-10T18:58:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.924+0000 7f23377fe700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 0x7f232406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:12.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.924+0000 7f23377fe700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 0x7f232406d190 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f233806ee10 tx=0x7f2328009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:12.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:12.927+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f232c0545f0 con 0x7f233806b1f0 2023-06-10T18:58:13.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.155+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23380b1aa0 con 0x7f233806b1f0 2023-06-10T18:58:13.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.156+0000 7f2334ff9700 1 -- 172.21.15.104:0/3156469647 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f232c057c10 con 0x7f233806b1f0 2023-06-10T18:58:13.162 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:13.162 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:58:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.157+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 msgr2=0x7f232406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.157+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 0x7f232406d190 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f233806ee10 tx=0x7f2328009040 comp rx=0 tx=0).stop 2023-06-10T18:58:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.157+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 msgr2=0x7f2338151bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.157+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f232c00b530 tx=0x7f232c00b8f0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 shutdown_connections 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23380ce350 0x7f23380c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f233806b1f0 0x7f2338151bc0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f232406ace0 0x7f232406d190 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 --2- 172.21.15.104:0/3156469647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233808a3d0 0x7f23380adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 >> 172.21.15.104:0/3156469647 conn(0x7f2338089fd0 msgr2=0x7f233815fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 shutdown_connections 2023-06-10T18:58:13.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.158+0000 7f233e3bc700 1 -- 172.21.15.104:0/3156469647 wait complete. 2023-06-10T18:58:13.166 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:58:13.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:13.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.290+0000 7f8a08279700 1 Processor -- start 2023-06-10T18:58:13.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.290+0000 7f8a08279700 1 -- start start 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a08279700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a000cea60 0x7f8a00165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a08279700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000b6070 con 0x7f8a000b4ff0 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a08279700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000b1d70 con 0x7f8a0015a420 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a08279700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000b5460 con 0x7f8a000cea60 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a06816700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a000cea60 0x7f8a00165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a05814700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8a000b4ff0 0x7f8a000ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37644/0 (socket says 172.21.15.104:37644) 2023-06-10T18:58:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.291+0000 7f8a05814700 1 -- 172.21.15.104:0/3866599853 learned_addr learned my addr 172.21.15.104:0/3866599853 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a06816700 1 -- 172.21.15.104:0/3866599853 >> v1:172.21.15.114:6789/0 conn(0x7f8a0015a420 legacy=0x7f8a000b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a06816700 1 -- 172.21.15.104:0/3866599853 >> v1:172.21.15.104:6789/0 conn(0x7f8a000b4ff0 legacy=0x7f8a000ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a05013700 1 -- 172.21.15.104:0/3866599853 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210077988 0 0) 0x7f8a000b6070 con 0x7f8a000b4ff0 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a06816700 1 -- 172.21.15.104:0/3866599853 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000b5740 con 0x7f8a000cea60 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a06816700 1 --2- 172.21.15.104:0/3866599853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a00165060 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009d90 tx=0x7f89f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4471cb79040508c8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a05013700 1 -- 172.21.15.104:0/3866599853 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800e070 con 0x7f8a000cea60 2023-06-10T18:58:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a05013700 1 -- 172.21.15.104:0/3866599853 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f800bb90 con 0x7f8a000cea60 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.292+0000 7f8a05013700 1 -- 172.21.15.104:0/3866599853 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8003a20 con 0x7f8a000cea60 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 -- 172.21.15.104:0/3866599853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 msgr2=0x7f8a00165060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 --2- 172.21.15.104:0/3866599853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a00165060 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009d90 tx=0x7f89f800b330 comp rx=0 tx=0).stop 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 -- 172.21.15.104:0/3866599853 shutdown_connections 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 --2- 172.21.15.104:0/3866599853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a00165060 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 -- 172.21.15.104:0/3866599853 >> 172.21.15.104:0/3866599853 conn(0x7f8a00089fd0 msgr2=0x7f8a000d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 -- 172.21.15.104:0/3866599853 shutdown_connections 2023-06-10T18:58:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.293+0000 7f8a08279700 1 -- 172.21.15.104:0/3866599853 wait complete. 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.294+0000 7f8a08279700 1 Processor -- start 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.294+0000 7f8a08279700 1 -- start start 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a000b4ff0 0x7f8a00152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 0x7f8a001f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a00152dc0 con 0x7f8a000cea60 2023-06-10T18:58:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0015b770 con 0x7f8a0015a420 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0015ba50 con 0x7f8a000b4ff0 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39822/0 (socket says 172.21.15.104:39822) 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 -- 172.21.15.104:0/936508738 learned_addr learned my addr 172.21.15.104:0/936508738 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a06816700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 0x7f8a001f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a06015700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a000b4ff0 0x7f8a00152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 -- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a000b4ff0 msgr2=0x7f8a00152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a000b4ff0 0x7f8a00152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 -- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 msgr2=0x7f8a001f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 0x7f8a001f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 -- 172.21.15.104:0/936508738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8009a70 con 0x7f8a000cea60 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a06816700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 0x7f8a001f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a05814700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0011e0 tx=0x7f89fc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc004c80 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc004e00 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0015bd30 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc007310 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.295+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0015d620 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.296+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a00074050 con 0x7f8a000cea60 2023-06-10T18:58:13.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.297+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f89fc007470 con 0x7f8a000cea60 2023-06-10T18:58:13.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.297+0000 7f89f6ffd700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 0x7f89e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.297+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f89fc0892e0 con 0x7f8a000cea60 2023-06-10T18:58:13.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.297+0000 7f8a06015700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 0x7f89e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.298+0000 7f8a06015700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 0x7f89e406d190 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f89f000a7b0 tx=0x7f89f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.301+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f89fc0542b0 con 0x7f8a000cea60 2023-06-10T18:58:13.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.487+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a00156930 con 0x7f8a000cea60 2023-06-10T18:58:13.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.489+0000 7f89f6ffd700 1 -- 172.21.15.104:0/936508738 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7f89fc0578d0 con 0x7f8a000cea60 2023-06-10T18:58:13.495 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:13.495 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 msgr2=0x7f89e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 0x7f89e406d190 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f89f000a7b0 tx=0x7f89f0008040 comp rx=0 tx=0).stop 2023-06-10T18:58:13.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 msgr2=0x7f8a0015d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0011e0 tx=0x7f89fc00cae0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 shutdown_connections 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e406ace0 0x7f89e406d190 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a0015a420 0x7f8a001f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a000b4ff0 0x7f8a00152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 --2- 172.21.15.104:0/936508738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a000cea60 0x7f8a0015d0e0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 >> 172.21.15.104:0/936508738 conn(0x7f8a00089fd0 msgr2=0x7f8a000b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 shutdown_connections 2023-06-10T18:58:13.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.491+0000 7f8a08279700 1 -- 172.21.15.104:0/936508738 wait complete. 2023-06-10T18:58:13.510 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:57:42.097445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '158', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:57:43.215419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:13.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:13.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.627+0000 7ff182157700 1 Processor -- start 2023-06-10T18:58:13.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.627+0000 7ff182157700 1 -- start start 2023-06-10T18:58:13.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff182157700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff17c156760 0x7ff17c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff182157700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 0x7ff17c0b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff182157700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c0ce850 con 0x7ff17c15d1a0 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff182157700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff17c0b30a0 con 0x7ff17c156760 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff182157700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c15d740 con 0x7ff17c0b21c0 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff17b7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff17c156760 0x7ff17c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff17bfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 0x7ff17c0b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff17b7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff17c156760 0x7ff17c0ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:55562/0 (socket says 172.21.15.104:55562) 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.628+0000 7ff17b7fe700 1 -- 172.21.15.104:0/637896936 learned_addr learned my addr 172.21.15.104:0/637896936 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:13.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17b7fe700 1 -- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 msgr2=0x7ff17c0b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17b7fe700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 0x7ff17c0b25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17b7fe700 1 -- 172.21.15.104:0/637896936 >> v1:172.21.15.114:6789/0 conn(0x7ff17c15d1a0 legacy=0x7ff17c0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17b7fe700 1 -- 172.21.15.104:0/637896936 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c0b2b20 con 0x7ff17c156760 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17bfff700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 0x7ff17c0b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17b7fe700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ce310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff1700098f0 tx=0x7ff170009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d638a129e6da805e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17a7fc700 1 -- 172.21.15.104:0/637896936 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170011830 con 0x7ff17c156760 2023-06-10T18:58:13.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17a7fc700 1 -- 172.21.15.104:0/637896936 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff170011990 con 0x7ff17c156760 2023-06-10T18:58:13.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.629+0000 7ff17a7fc700 1 -- 172.21.15.104:0/637896936 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170011d20 con 0x7ff17c156760 2023-06-10T18:58:13.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 -- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 msgr2=0x7ff17c0ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ce310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff1700098f0 tx=0x7ff170009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:13.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 -- 172.21.15.104:0/637896936 shutdown_connections 2023-06-10T18:58:13.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c0b21c0 0x7ff17c0b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 --2- 172.21.15.104:0/637896936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ce310 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 -- 172.21.15.104:0/637896936 >> 172.21.15.104:0/637896936 conn(0x7ff17c06e370 msgr2=0x7ff17c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 -- 172.21.15.104:0/637896936 shutdown_connections 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.630+0000 7ff182157700 1 -- 172.21.15.104:0/637896936 wait complete. 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.631+0000 7ff182157700 1 Processor -- start 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.631+0000 7ff182157700 1 -- start start 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c15d1a0 0x7ff17c0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c0c8500 con 0x7ff17c0b21c0 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c0af660 con 0x7ff17c156760 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff182157700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c0ae5c0 con 0x7ff17c15d1a0 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39834/0 (socket says 172.21.15.104:39834) 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 -- 172.21.15.104:0/686921099 learned_addr learned my addr 172.21.15.104:0/686921099 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:13.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17affd700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17bfff700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c15d1a0 0x7ff17c0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 -- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c15d1a0 msgr2=0x7ff17c0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c15d1a0 0x7ff17c0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 -- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 msgr2=0x7ff17c0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.632+0000 7ff17b7fe700 1 -- 172.21.15.104:0/686921099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff174009a70 con 0x7ff17c0b21c0 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff17b7fe700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ff170000c00 tx=0x7ff170004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1700043c0 con 0x7ff17c0b21c0 2023-06-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17001f070 con 0x7ff17c0b21c0 2023-06-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff170009550 con 0x7ff17c0b21c0 2023-06-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1700038a0 con 0x7ff17c0b21c0 2023-06-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.633+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17c0aeb80 con 0x7ff17c0b21c0 2023-06-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.634+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17c074050 con 0x7ff17c0b21c0 2023-06-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.634+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff170021030 con 0x7ff17c0b21c0 2023-06-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.634+0000 7ff163fff700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 0x7ff16806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.635+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff17008a930 con 0x7ff17c0b21c0 2023-06-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.635+0000 7ff17affd700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 0x7ff16806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.635+0000 7ff17affd700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 0x7ff16806d190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0c7c50 tx=0x7ff16c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.639+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff170055900 con 0x7ff17c0b21c0 2023-06-10T18:58:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.825+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff17c0ccbb0 con 0x7ff17c0b21c0 2023-06-10T18:58:13.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.826+0000 7ff163fff700 1 -- 172.21.15.104:0/686921099 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v160) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff170015020 con 0x7ff17c0b21c0 2023-06-10T18:58:13.832 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 msgr2=0x7ff16806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 0x7ff16806d190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0c7c50 tx=0x7ff16c009040 comp rx=0 tx=0).stop 2023-06-10T18:58:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 msgr2=0x7ff17c1f1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ff170000c00 tx=0x7ff170004750 comp rx=0 tx=0).stop 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 shutdown_connections 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff17c15d1a0 0x7ff17c0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff17c0b21c0 0x7ff17c1f1a50 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff16806ace0 0x7ff16806d190 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 --2- 172.21.15.104:0/686921099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17c156760 0x7ff17c0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 >> 172.21.15.104:0/686921099 conn(0x7ff17c06e370 msgr2=0x7ff17c15bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 shutdown_connections 2023-06-10T18:58:13.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.828+0000 7ff182157700 1 -- 172.21.15.104:0/686921099 wait complete. 2023-06-10T18:58:13.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.965+0000 7f976bcc3700 1 Processor -- start 2023-06-10T18:58:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.965+0000 7f976bcc3700 1 -- start start 2023-06-10T18:58:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976bcc3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f976415c580 0x7f97640b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976bcc3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f976406cde0 0x7f97640ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976bcc3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976406e250 con 0x7f976406e6e0 2023-06-10T18:58:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976bcc3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976415c950 con 0x7f976406cde0 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976bcc3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97640b1e80 con 0x7f976415c580 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f976406cde0 0x7f97640ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f976406cde0 0x7f97640ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39842/0 (socket says 172.21.15.104:39842) 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 -- 172.21.15.104:0/3801590765 learned_addr learned my addr 172.21.15.104:0/3801590765 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 -- 172.21.15.104:0/3801590765 >> v2:172.21.15.114:3300/0 conn(0x7f976415c580 msgr2=0x7f97640b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 --2- 172.21.15.104:0/3801590765 >> v2:172.21.15.114:3300/0 conn(0x7f976415c580 0x7f97640b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 -- 172.21.15.104:0/3801590765 >> v1:172.21.15.114:6789/0 conn(0x7f976406e6e0 legacy=0x7f97640b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.966+0000 7f976a260700 1 -- 172.21.15.104:0/3801590765 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976406d920 con 0x7f976406cde0 2023-06-10T18:58:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976a260700 1 --2- 172.21.15.104:0/3801590765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976406cde0 0x7f97640ce0d0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f9764070300 tx=0x7f975c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f5d85ddccb0983 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f9768a5d700 1 -- 172.21.15.104:0/3801590765 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c00e070 con 0x7f976406cde0 2023-06-10T18:58:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f9768a5d700 1 -- 172.21.15.104:0/3801590765 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c003a20 con 0x7f976406cde0 2023-06-10T18:58:13.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f9768a5d700 1 -- 172.21.15.104:0/3801590765 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c003dd0 con 0x7f976406cde0 2023-06-10T18:58:13.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 -- 172.21.15.104:0/3801590765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976406cde0 msgr2=0x7f97640ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3801590765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976406cde0 0x7f97640ce0d0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f9764070300 tx=0x7f975c00b330 comp rx=0 tx=0).stop 2023-06-10T18:58:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 -- 172.21.15.104:0/3801590765 shutdown_connections 2023-06-10T18:58:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3801590765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976406cde0 0x7f97640ce0d0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3801590765 >> v2:172.21.15.114:3300/0 conn(0x7f976415c580 0x7f97640b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 -- 172.21.15.104:0/3801590765 >> 172.21.15.104:0/3801590765 conn(0x7f976406b8a0 msgr2=0x7f97640cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:13.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 -- 172.21.15.104:0/3801590765 shutdown_connections 2023-06-10T18:58:13.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.967+0000 7f976bcc3700 1 -- 172.21.15.104:0/3801590765 wait complete. 2023-06-10T18:58:13.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.968+0000 7f976bcc3700 1 Processor -- start 2023-06-10T18:58:13.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 -- start start 2023-06-10T18:58:13.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f976406cde0 0x7f97641f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 0x7f97640b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9764158b00 con 0x7f976415c580 2023-06-10T18:58:13.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97640c8640 con 0x7f976406e6e0 2023-06-10T18:58:13.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976bcc3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9764157ef0 con 0x7f976406cde0 2023-06-10T18:58:13.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976a260700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976a260700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39852/0 (socket says 172.21.15.104:39852) 2023-06-10T18:58:13.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976a260700 1 -- 172.21.15.104:0/3858266346 learned_addr learned my addr 172.21.15.104:0/3858266346 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:13.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.969+0000 7f976925e700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 0x7f97640b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f9769a5f700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f976406cde0 0x7f97641f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 -- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f976406cde0 msgr2=0x7f97641f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f976406cde0 0x7f97641f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 -- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 msgr2=0x7f97640b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:13.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 0x7f97640b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:13.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 -- 172.21.15.104:0/3858266346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c009b30 con 0x7f976415c580 2023-06-10T18:58:13.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976925e700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 0x7f97640b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:13.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976a260700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f97640b21e0 tx=0x7f975c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c004440 con 0x7f976415c580 2023-06-10T18:58:13.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c0045a0 con 0x7f976415c580 2023-06-10T18:58:13.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c004950 con 0x7f976415c580 2023-06-10T18:58:13.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97641581d0 con 0x7f976415c580 2023-06-10T18:58:13.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.970+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9764158790 con 0x7f976415c580 2023-06-10T18:58:13.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.971+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764158cb0 con 0x7f976415c580 2023-06-10T18:58:13.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.971+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f975c025450 con 0x7f976415c580 2023-06-10T18:58:13.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.972+0000 7f975a7fc700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:13.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.972+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f975c08af00 con 0x7f976415c580 2023-06-10T18:58:13.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.972+0000 7f9769a5f700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:13.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.972+0000 7f9769a5f700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9754009870 tx=0x7f9754008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:13.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:13.976+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9764158cb0 con 0x7f976415c580 2023-06-10T18:58:14.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.163+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9764158cb0 con 0x7f976415c580 2023-06-10T18:58:14.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.163+0000 7f975a7fc700 1 -- 172.21.15.104:0/3858266346 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9764158cb0 con 0x7f976415c580 2023-06-10T18:58:14.170 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:14.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 msgr2=0x7f974806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9754009870 tx=0x7f9754008040 comp rx=0 tx=0).stop 2023-06-10T18:58:14.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 msgr2=0x7f97640ad6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f97640b21e0 tx=0x7f975c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 shutdown_connections 2023-06-10T18:58:14.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f976415c580 0x7f97640ad6a0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f976406cde0 0x7f97641f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f974806ace0 0x7f974806d190 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 --2- 172.21.15.104:0/3858266346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f976406e6e0 0x7f97640b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 >> 172.21.15.104:0/3858266346 conn(0x7f976406b8a0 msgr2=0x7f9764159460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 shutdown_connections 2023-06-10T18:58:14.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.165+0000 7f976bcc3700 1 -- 172.21.15.104:0/3858266346 wait complete. 2023-06-10T18:58:14.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:14.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 Processor -- start 2023-06-10T18:58:14.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 -- start start 2023-06-10T18:58:14.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0df80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80ad3c0 con 0x7f0df80d0830 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df8157f00 con 0x7f0df8157b30 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.300+0000 7f0dfebd2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df80b0170 con 0x7f0df80cea50 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0dfd16f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0df80d0830 0x7f0df80b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37688/0 (socket says 172.21.15.104:37688) 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0dfd16f700 1 -- 172.21.15.104:0/3773554959 learned_addr learned my addr 172.21.15.104:0/3773554959 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df7fff700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0df80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485666016 0 0) 0x7f0df80ad3c0 con 0x7f0df80d0830 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de80035b0 con 0x7f0df80d0830 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145496700 0 0) 0x7f0de80035b0 con 0x7f0df80d0830 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 msgr2=0x7f0df80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0df80d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 >> v1:172.21.15.114:6789/0 conn(0x7f0df8157b30 legacy=0x7f0df80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80b0410 con 0x7f0df80d0830 2023-06-10T18:58:14.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0df00019f0 con 0x7f0df80d0830 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 >> v1:172.21.15.104:6789/0 conn(0x7f0df80d0830 legacy=0x7f0df80b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de8003a30 0x7f0de8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de80043d0 0x7f0de8004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0de80050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df80b0170 con 0x7f0de8003a30 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de800ba10 con 0x7f0de80043d0 2023-06-10T18:58:14.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de800bbd0 con 0x7f0df80cea50 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de8003a30 0x7f0de8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0df7fff700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de80043d0 0x7f0de8004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.301+0000 7f0dfd16f700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0de80050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 -- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 msgr2=0x7f0de80050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df80cea50 0x7f0de80050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 -- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de80043d0 msgr2=0x7f0de8004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de80043d0 0x7f0de8004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 -- 172.21.15.104:0/3773554959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de80035b0 con 0x7f0de8003a30 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de8003a30 0x7f0de8003e00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0018d0 tx=0x7f0de8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62b138c3377542a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00c690 con 0x7f0de8003a30 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec00c810 con 0x7f0de8003a30 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0df77fe700 1 -- 172.21.15.104:0/3773554959 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00cbc0 con 0x7f0de8003a30 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 -- 172.21.15.104:0/3773554959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de8003a30 msgr2=0x7f0de8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/3773554959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de8003a30 0x7f0de8003e00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0018d0 tx=0x7f0de8005880 comp rx=0 tx=0).stop 2023-06-10T18:58:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfd16f700 1 -- 172.21.15.104:0/3773554959 reap_dead start 2023-06-10T18:58:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 -- 172.21.15.104:0/3773554959 shutdown_connections 2023-06-10T18:58:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 -- 172.21.15.104:0/3773554959 >> 172.21.15.104:0/3773554959 conn(0x7f0df8089fd0 msgr2=0x7f0df808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 -- 172.21.15.104:0/3773554959 shutdown_connections 2023-06-10T18:58:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.302+0000 7f0dfebd2700 1 -- 172.21.15.104:0/3773554959 wait complete. 2023-06-10T18:58:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 Processor -- start 2023-06-10T18:58:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 -- start start 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0df80c69e0 0x7f0df81e5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df8154d30 0x7f0df80b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df81551d0 con 0x7f0df81e58d0 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df81e5e80 con 0x7f0df80c69e0 2023-06-10T18:58:14.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.304+0000 7f0dfebd2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df81e6160 con 0x7f0df8154d30 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39870/0 (socket says 172.21.15.104:39870) 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 -- 172.21.15.104:0/2848901510 learned_addr learned my addr 172.21.15.104:0/2848901510 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0df80c69e0 0x7f0df81e5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0dfd16f700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df8154d30 0x7f0df80b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 -- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df8154d30 msgr2=0x7f0df80b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df8154d30 0x7f0df80b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 -- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0df80c69e0 msgr2=0x7f0df81e5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0df80c69e0 0x7f0df81e5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 -- 172.21.15.104:0/2848901510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df81e6440 con 0x7f0df81e58d0 2023-06-10T18:58:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df7fff700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0de4011940 tx=0x7f0de4011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de4004240 con 0x7f0df81e58d0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de40043a0 con 0x7f0df81e58d0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df81e6720 con 0x7f0df81e58d0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de40038c0 con 0x7f0df81e58d0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.305+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df8153260 con 0x7f0df81e58d0 2023-06-10T18:58:14.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.306+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df80b39b0 con 0x7f0df81e58d0 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.307+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0de4003a20 con 0x7f0df81e58d0 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.307+0000 7f0df57fa700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 0x7f0ddc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.307+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f0de4089910 con 0x7f0df81e58d0 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.307+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 0x7f0ddc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.308+0000 7f0dfc96e700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 0x7f0ddc06d190 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0df808ab20 tx=0x7f0dec005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.312+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0de4054850 con 0x7f0df81e58d0 2023-06-10T18:58:14.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.498+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0df8074050 con 0x7f0df81e58d0 2023-06-10T18:58:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.498+0000 7f0df57fa700 1 -- 172.21.15.104:0/2848901510 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v160) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0de4057e70 con 0x7f0df81e58d0 2023-06-10T18:58:14.505 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 msgr2=0x7f0ddc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 0x7f0ddc06d190 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0df808ab20 tx=0x7f0dec005040 comp rx=0 tx=0).stop 2023-06-10T18:58:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 msgr2=0x7f0df81547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f0de4011940 tx=0x7f0de4011c50 comp rx=0 tx=0).stop 2023-06-10T18:58:14.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 shutdown_connections 2023-06-10T18:58:14.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0ddc06ace0 0x7f0ddc06d190 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0df80c69e0 0x7f0df81e5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0df8154d30 0x7f0df80b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 --2- 172.21.15.104:0/2848901510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0df81e58d0 0x7f0df81547f0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 >> 172.21.15.104:0/2848901510 conn(0x7f0df8089fd0 msgr2=0x7f0df80ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 shutdown_connections 2023-06-10T18:58:14.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.500+0000 7f0dfebd2700 1 -- 172.21.15.104:0/2848901510 wait complete. 2023-06-10T18:58:14.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.636+0000 7fb99ec46700 1 Processor -- start 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.636+0000 7fb99ec46700 1 -- start start 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99ec46700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb99806e6e0 0x7fb9980b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99ec46700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 0x7fb9980ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99ec46700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99806e250 con 0x7fb99815c580 2023-06-10T18:58:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99ec46700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99815c950 con 0x7fb99806e6e0 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99ec46700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9980b1e80 con 0x7fb99806cde0 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb99806e6e0 0x7fb9980b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb99806e6e0 0x7fb9980b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39878/0 (socket says 172.21.15.104:39878) 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 -- 172.21.15.104:0/168906451 learned_addr learned my addr 172.21.15.104:0/168906451 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 -- 172.21.15.104:0/168906451 >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 msgr2=0x7fb9980ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99d1e3700 1 --2- 172.21.15.104:0/168906451 >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 0x7fb9980ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 --2- 172.21.15.104:0/168906451 >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 0x7fb9980ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 -- 172.21.15.104:0/168906451 >> v1:172.21.15.114:6789/0 conn(0x7fb99815c580 legacy=0x7fb9980b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 -- 172.21.15.104:0/168906451 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99806d920 con 0x7fb99806e6e0 2023-06-10T18:58:14.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb99d1e3700 1 --2- 172.21.15.104:0/168906451 >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 0x7fb9980ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:14.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb997fff700 1 --2- 172.21.15.104:0/168906451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9980b1780 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fb9980cbc60 tx=0x7fb984009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b72620437396314 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb9977fe700 1 -- 172.21.15.104:0/168906451 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984004230 con 0x7fb99806e6e0 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb9977fe700 1 -- 172.21.15.104:0/168906451 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984004390 con 0x7fb99806e6e0 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.637+0000 7fb9977fe700 1 -- 172.21.15.104:0/168906451 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840036a0 con 0x7fb99806e6e0 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 -- 172.21.15.104:0/168906451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 msgr2=0x7fb9980b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 --2- 172.21.15.104:0/168906451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9980b1780 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fb9980cbc60 tx=0x7fb984009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 -- 172.21.15.104:0/168906451 shutdown_connections 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 --2- 172.21.15.104:0/168906451 >> v2:172.21.15.114:3300/0 conn(0x7fb99806cde0 0x7fb9980ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 --2- 172.21.15.104:0/168906451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9980b1780 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 -- 172.21.15.104:0/168906451 >> 172.21.15.104:0/168906451 conn(0x7fb99806b8a0 msgr2=0x7fb9980cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 -- 172.21.15.104:0/168906451 shutdown_connections 2023-06-10T18:58:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.638+0000 7fb99ec46700 1 -- 172.21.15.104:0/168906451 wait complete. 2023-06-10T18:58:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.639+0000 7fb99ec46700 1 Processor -- start 2023-06-10T18:58:14.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.639+0000 7fb99ec46700 1 -- start start 2023-06-10T18:58:14.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 0x7fb9981f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 0x7fb9981e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980b3e10 con 0x7fb99806e6e0 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9981f5c00 con 0x7fb99806cde0 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99ec46700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9981f5ee0 con 0x7fb99815c580 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39884/0 (socket says 172.21.15.104:39884) 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 -- 172.21.15.104:0/3281841843 learned_addr learned my addr 172.21.15.104:0/3281841843 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99c9e2700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 0x7fb9981f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99d1e3700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 0x7fb9981e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 -- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 msgr2=0x7fb9981e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 0x7fb9981e1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 -- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 msgr2=0x7fb9981f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 0x7fb9981f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb997fff700 1 -- 172.21.15.104:0/3281841843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb984009550 con 0x7fb99806e6e0 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99c9e2700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 0x7fb9981f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:14.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.640+0000 7fb99d1e3700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 0x7fb9981e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb997fff700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fb984009130 tx=0x7fb984004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984003dc0 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98401f070 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9981f61c0 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984011870 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.641+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9981f5810 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.642+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb998074050 con 0x7fb99806e6e0 2023-06-10T18:58:14.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.642+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb984021030 con 0x7fb99806e6e0 2023-06-10T18:58:14.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.642+0000 7fb9957fa700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 0x7fb97c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.643+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb98408ad30 con 0x7fb99806e6e0 2023-06-10T18:58:14.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.643+0000 7fb99c9e2700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 0x7fb97c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.643+0000 7fb99c9e2700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 0x7fb97c06d190 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006fd0 tx=0x7fb98c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.647+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb984055e20 con 0x7fb99806e6e0 2023-06-10T18:58:14.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.836+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb998066f10 con 0x7fb99806e6e0 2023-06-10T18:58:14.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.836+0000 7fb9957fa700 1 -- 172.21.15.104:0/3281841843 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v160) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb984059440 con 0x7fb99806e6e0 2023-06-10T18:58:14.843 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:14.844 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 msgr2=0x7fb97c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 0x7fb97c06d190 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006fd0 tx=0x7fb98c008040 comp rx=0 tx=0).stop 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 msgr2=0x7fb9981e49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fb984009130 tx=0x7fb984004750 comp rx=0 tx=0).stop 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 shutdown_connections 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb97c06ace0 0x7fb97c06d190 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99806cde0 0x7fb9981f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb99815c580 0x7fb9981e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 --2- 172.21.15.104:0/3281841843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99806e6e0 0x7fb9981e49f0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 >> 172.21.15.104:0/3281841843 conn(0x7fb99806b8a0 msgr2=0x7fb998152720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 shutdown_connections 2023-06-10T18:58:14.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.838+0000 7fb99ec46700 1 -- 172.21.15.104:0/3281841843 wait complete. 2023-06-10T18:58:14.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.971+0000 7fa1c7e22700 1 Processor -- start 2023-06-10T18:58:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.971+0000 7fa1c7e22700 1 -- start start 2023-06-10T18:58:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c7e22700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1c00cd850 0x7fa1c006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c7e22700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c00c9e60 con 0x7fa1c00cb300 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c7e22700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c00aee70 con 0x7fa1c00aeaa0 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c7e22700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c00c9bc0 con 0x7fa1c00cd850 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c63bf700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa1c00cb300 0x7fa1c006d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37722/0 (socket says 172.21.15.104:37722) 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c63bf700 1 -- 172.21.15.104:0/3974170857 learned_addr learned my addr 172.21.15.104:0/3974170857 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c53bd700 1 --2- 172.21.15.104:0/3974170857 >> v2:172.21.15.114:3300/0 conn(0x7fa1c00cd850 0x7fa1c006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316578300 0 0) 0x7fa1c00c9e60 con 0x7fa1c00cb300 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ac0035b0 con 0x7fa1c00cb300 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.972+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2294057497 0 0) 0x7fa1ac0035b0 con 0x7fa1c00cb300 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 >> v2:172.21.15.114:3300/0 conn(0x7fa1c00cd850 msgr2=0x7fa1c006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 --2- 172.21.15.104:0/3974170857 >> v2:172.21.15.114:3300/0 conn(0x7fa1c00cd850 0x7fa1c006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 >> v1:172.21.15.114:6789/0 conn(0x7fa1c00aeaa0 legacy=0x7fa1c00cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c006e540 con 0x7fa1c00cb300 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa1b80019f0 con 0x7fa1c00cb300 2023-06-10T18:58:14.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 >> v1:172.21.15.104:6789/0 conn(0x7fa1c00cb300 legacy=0x7fa1c006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac003a00 0x7fa1ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 0x7fa1ac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1ac005200 0x7fa1ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c00c9bc0 con 0x7fa1ac004850 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac00c060 con 0x7fa1ac005200 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac00c220 con 0x7fa1ac003a00 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 0x7fa1ac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c5bbe700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac003a00 0x7fa1ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c63bf700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1ac005200 0x7fa1ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 -- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac003a00 msgr2=0x7fa1ac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac003a00 0x7fa1ac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 -- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1ac005200 msgr2=0x7fa1ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1ac005200 0x7fa1ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 -- 172.21.15.104:0/3974170857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0035b0 con 0x7fa1ac004850 2023-06-10T18:58:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c63bf700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1ac005200 0x7fa1ac00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c63bf700 1 -- 172.21.15.104:0/3974170857 reap_dead start 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c53bd700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 0x7fa1ac004cc0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc009b60 tx=0x7fa1ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a34319448ae9cf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc00e6c0 con 0x7fa1ac004850 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc00e820 con 0x7fa1ac004850 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.973+0000 7fa1c4bbc700 1 -- 172.21.15.104:0/3974170857 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc00eb90 con 0x7fa1ac004850 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 -- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 msgr2=0x7fa1ac004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 0x7fa1ac004cc0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc009b60 tx=0x7fa1ac005880 comp rx=0 tx=0).stop 2023-06-10T18:58:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 -- 172.21.15.104:0/3974170857 shutdown_connections 2023-06-10T18:58:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/3974170857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1ac004850 0x7fa1ac004cc0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 -- 172.21.15.104:0/3974170857 >> 172.21.15.104:0/3974170857 conn(0x7fa1c006b8a0 msgr2=0x7fa1c00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 -- 172.21.15.104:0/3974170857 shutdown_connections 2023-06-10T18:58:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.974+0000 7fa1c7e22700 1 -- 172.21.15.104:0/3974170857 wait complete. 2023-06-10T18:58:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.975+0000 7fa1c7e22700 1 Processor -- start 2023-06-10T18:58:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.975+0000 7fa1c7e22700 1 -- start start 2023-06-10T18:58:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac004850 0x7fa1c01f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1c00c38a0 0x7fa1c01f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01f0260 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01efca0 con 0x7fa1c00c38a0 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c01eff80 con 0x7fa1ac004850 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39906/0 (socket says 172.21.15.104:39906) 2023-06-10T18:58:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 -- 172.21.15.104:0/324834237 learned_addr learned my addr 172.21.15.104:0/324834237 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 -- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac004850 msgr2=0x7fa1c01f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac004850 0x7fa1c01f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 -- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1c00c38a0 msgr2=0x7fa1c01f2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1c00c38a0 0x7fa1c01f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 -- 172.21.15.104:0/324834237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c015ad30 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c63bf700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00cbf40 tx=0x7fa1b8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800ecb0 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b800ee10 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.976+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c015b010 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.977+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8013660 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.977+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c01ef920 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.977+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c015b200 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.978+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa1b80137c0 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.978+0000 7fa1b67fc700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 0x7fa1a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.978+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fa1b8089c20 con 0x7fa1c01ef2c0 2023-06-10T18:58:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.979+0000 7fa1c5bbe700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 0x7fa1a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:14.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.979+0000 7fa1c5bbe700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 0x7fa1a406d190 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000ef0 tx=0x7fa1b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:14.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:14.983+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa1b8054bf0 con 0x7fa1c01ef2c0 2023-06-10T18:58:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.212+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1c0074050 con 0x7fa1c01ef2c0 2023-06-10T18:58:15.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.213+0000 7fa1b67fc700 1 -- 172.21.15.104:0/324834237 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7fa1b8058210 con 0x7fa1c01ef2c0 2023-06-10T18:58:15.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:15.219 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:58:15.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 msgr2=0x7fa1a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 0x7fa1a406d190 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000ef0 tx=0x7fa1b0009040 comp rx=0 tx=0).stop 2023-06-10T18:58:15.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 msgr2=0x7fa1c006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00cbf40 tx=0x7fa1b8007c90 comp rx=0 tx=0).stop 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 shutdown_connections 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1c01ef2c0 0x7fa1c006d680 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1ac004850 0x7fa1c01f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa1a406ace0 0x7fa1a406d190 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 --2- 172.21.15.104:0/324834237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1c00c38a0 0x7fa1c01f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 >> 172.21.15.104:0/324834237 conn(0x7fa1c006b8a0 msgr2=0x7fa1c006a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:15.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 shutdown_connections 2023-06-10T18:58:15.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.215+0000 7fa1c7e22700 1 -- 172.21.15.104:0/324834237 wait complete. 2023-06-10T18:58:15.223 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:58:15.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:15.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.348+0000 7f59418d1700 1 Processor -- start 2023-06-10T18:58:15.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.348+0000 7f59418d1700 1 -- start start 2023-06-10T18:58:15.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f59418d1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f593c06e4b0 0x7f593c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f59418d1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f59418d1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c06d290 con 0x7f593c06e0e0 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f59418d1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c071640 con 0x7f593c06e4b0 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f59418d1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0aef00 con 0x7f593c069e70 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f593a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f593c06e4b0 0x7f593c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f593affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f593c06e0e0 0x7f593c0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37738/0 (socket says 172.21.15.104:37738) 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.349+0000 7f593affd700 1 -- 172.21.15.104:0/2911041263 learned_addr learned my addr 172.21.15.104:0/2911041263 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593b7fe700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593a7fc700 1 -- 172.21.15.104:0/2911041263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 msgr2=0x7f593c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593a7fc700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593a7fc700 1 -- 172.21.15.104:0/2911041263 >> v1:172.21.15.104:6789/0 conn(0x7f593c06e0e0 legacy=0x7f593c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f5939ffb700 1 -- 172.21.15.104:0/2911041263 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787565610 0 0) 0x7f593c06d290 con 0x7f593c06e0e0 2023-06-10T18:58:15.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593a7fc700 1 -- 172.21.15.104:0/2911041263 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0ca2d0 con 0x7f593c06e4b0 2023-06-10T18:58:15.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593b7fe700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:15.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f593a7fc700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c06cd50 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f592c009840 tx=0x7f592c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c86f1630cc704e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f5939ffb700 1 -- 172.21.15.104:0/2911041263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c011830 con 0x7f593c06e4b0 2023-06-10T18:58:15.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f5939ffb700 1 -- 172.21.15.104:0/2911041263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c011990 con 0x7f593c06e4b0 2023-06-10T18:58:15.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.350+0000 7f5939ffb700 1 -- 172.21.15.104:0/2911041263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c011d20 con 0x7f593c06e4b0 2023-06-10T18:58:15.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 -- 172.21.15.104:0/2911041263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 msgr2=0x7f593c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c06cd50 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f592c009840 tx=0x7f592c009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:15.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 -- 172.21.15.104:0/2911041263 shutdown_connections 2023-06-10T18:58:15.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 --2- 172.21.15.104:0/2911041263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c06cd50 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 -- 172.21.15.104:0/2911041263 >> 172.21.15.104:0/2911041263 conn(0x7f593c06b8a0 msgr2=0x7f593c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 -- 172.21.15.104:0/2911041263 shutdown_connections 2023-06-10T18:58:15.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.351+0000 7f59418d1700 1 -- 172.21.15.104:0/2911041263 wait complete. 2023-06-10T18:58:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.352+0000 7f59418d1700 1 Processor -- start 2023-06-10T18:58:15.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.352+0000 7f59418d1700 1 -- start start 2023-06-10T18:58:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 0x7f593c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c09b900 con 0x7f593c06e4b0 2023-06-10T18:58:15.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c09b2f0 con 0x7f593c06e0e0 2023-06-10T18:58:15.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f59418d1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0b19d0 con 0x7f593c069e70 2023-06-10T18:58:15.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f593b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f593b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39920/0 (socket says 172.21.15.104:39920) 2023-06-10T18:58:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.353+0000 7f593b7fe700 1 -- 172.21.15.104:0/2449189888 learned_addr learned my addr 172.21.15.104:0/2449189888 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593affd700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593a7fc700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 0x7f593c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 -- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 msgr2=0x7f593c0995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 -- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 msgr2=0x7f593c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 0x7f593c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 -- 172.21.15.104:0/2449189888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0094a0 con 0x7f593c06e4b0 2023-06-10T18:58:15.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593a7fc700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 0x7f593c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:15.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f593b7fe700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f593400cd80 tx=0x7f593400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400f480 con 0x7f593c06e4b0 2023-06-10T18:58:15.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593400f600 con 0x7f593c06e4b0 2023-06-10T18:58:15.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c0b1410 con 0x7f593c06e4b0 2023-06-10T18:58:15.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400f9b0 con 0x7f593c06e4b0 2023-06-10T18:58:15.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.354+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c09b520 con 0x7f593c06e4b0 2023-06-10T18:58:15.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.355+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c09ba40 con 0x7f593c06e4b0 2023-06-10T18:58:15.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.355+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f593400b040 con 0x7f593c06e4b0 2023-06-10T18:58:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.356+0000 7f5923fff700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 0x7f592806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.356+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f5934089d80 con 0x7f593c06e4b0 2023-06-10T18:58:15.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.356+0000 7f593affd700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 0x7f592806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.357+0000 7f593affd700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 0x7f592806d190 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f59300010e0 tx=0x7f5930009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.360+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f593c09ba40 con 0x7f593c06e4b0 2023-06-10T18:58:15.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.546+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593c09ba40 con 0x7f593c06e4b0 2023-06-10T18:58:15.554 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.547+0000 7f5923fff700 1 -- 172.21.15.104:0/2449189888 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7f593c09ba40 con 0x7f593c06e4b0 2023-06-10T18:58:15.554 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:15.554 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 msgr2=0x7f592806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 0x7f592806d190 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f59300010e0 tx=0x7f5930009040 comp rx=0 tx=0).stop 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 msgr2=0x7f593c09bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f593400cd80 tx=0x7f593400a560 comp rx=0 tx=0).stop 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 shutdown_connections 2023-06-10T18:58:15.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f593c06e4b0 0x7f593c09bd20 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f593c069e70 0x7f593c0995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f592806ace0 0x7f592806d190 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 --2- 172.21.15.104:0/2449189888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f593c06e0e0 0x7f593c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.549+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 >> 172.21.15.104:0/2449189888 conn(0x7f593c06b8a0 msgr2=0x7f593c0ad010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:15.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.550+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 shutdown_connections 2023-06-10T18:58:15.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.550+0000 7f59418d1700 1 -- 172.21.15.104:0/2449189888 wait complete. 2023-06-10T18:58:15.568 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:58:15.568 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:15.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.682+0000 7f8df4a32700 1 Processor -- start 2023-06-10T18:58:15.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.682+0000 7f8df4a32700 1 -- start start 2023-06-10T18:58:15.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8df4a32700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8df015c040 0x7f8df00ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8df4a32700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8df4a32700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00ccea0 con 0x7f8df00ae280 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8df4a32700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df00ae6f0 con 0x7f8df015c040 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8df4a32700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00ce2c0 con 0x7f8df0071130 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8dee59c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8df015c040 0x7f8df00ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:59312/0 (socket says 172.21.15.104:59312) 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 -- 172.21.15.104:0/4017585453 learned_addr learned my addr 172.21.15.104:0/4017585453 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 -- 172.21.15.104:0/4017585453 >> v2:172.21.15.114:3300/0 conn(0x7f8df015c040 msgr2=0x7f8df00ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 --2- 172.21.15.104:0/4017585453 >> v2:172.21.15.114:3300/0 conn(0x7f8df015c040 0x7f8df00ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 -- 172.21.15.104:0/4017585453 >> v1:172.21.15.114:6789/0 conn(0x7f8df00ae280 legacy=0x7f8df0070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.683+0000 7f8deed9d700 1 -- 172.21.15.104:0/4017585453 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00cf750 con 0x7f8df0071130 2023-06-10T18:58:15.691 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8deed9d700 1 --2- 172.21.15.104:0/4017585453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009e50 tx=0x7f8de800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5037b2c1b0be595 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8ded59a700 1 -- 172.21.15.104:0/4017585453 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de800e070 con 0x7f8df0071130 2023-06-10T18:58:15.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8ded59a700 1 -- 172.21.15.104:0/4017585453 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de800b910 con 0x7f8df0071130 2023-06-10T18:58:15.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8ded59a700 1 -- 172.21.15.104:0/4017585453 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de800bcc0 con 0x7f8df0071130 2023-06-10T18:58:15.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 -- 172.21.15.104:0/4017585453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 msgr2=0x7f8df00cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 --2- 172.21.15.104:0/4017585453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009e50 tx=0x7f8de800b330 comp rx=0 tx=0).stop 2023-06-10T18:58:15.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 -- 172.21.15.104:0/4017585453 shutdown_connections 2023-06-10T18:58:15.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 --2- 172.21.15.104:0/4017585453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df0071130 0x7f8df00cc960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 --2- 172.21.15.104:0/4017585453 >> v2:172.21.15.114:3300/0 conn(0x7f8df015c040 0x7f8df00ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 -- 172.21.15.104:0/4017585453 >> 172.21.15.104:0/4017585453 conn(0x7f8df00b1610 msgr2=0x7f8df00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 -- 172.21.15.104:0/4017585453 shutdown_connections 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.684+0000 7f8df4a32700 1 -- 172.21.15.104:0/4017585453 wait complete. 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 Processor -- start 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 -- start start 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df00ae280 0x7f8df00c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df015c040 0x7f8df00c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0158e50 con 0x7f8df0071130 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00c67d0 con 0x7f8df015c040 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.686+0000 7f8df4a32700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00c6ab0 con 0x7f8df00ae280 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39930/0 (socket says 172.21.15.104:39930) 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 -- 172.21.15.104:0/748365797 learned_addr learned my addr 172.21.15.104:0/748365797 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:15.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dedd9b700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df00ae280 0x7f8df00c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8deed9d700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df015c040 0x7f8df00c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 -- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df00ae280 msgr2=0x7f8df00c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df00ae280 0x7f8df00c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 -- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df015c040 msgr2=0x7f8df00c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df015c040 0x7f8df00c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 -- 172.21.15.104:0/748365797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8009b30 con 0x7f8df0071130 2023-06-10T18:58:15.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8dee59c700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e9b0 tx=0x7f8de000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0009710 con 0x7f8df0071130 2023-06-10T18:58:15.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de000ee50 con 0x7f8df0071130 2023-06-10T18:58:15.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000ca50 con 0x7f8df0071130 2023-06-10T18:58:15.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df00c6d90 con 0x7f8df0071130 2023-06-10T18:58:15.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.687+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df00c7350 con 0x7f8df0071130 2023-06-10T18:58:15.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.688+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df0074050 con 0x7f8df0071130 2023-06-10T18:58:15.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.688+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8de000cbb0 con 0x7f8df0071130 2023-06-10T18:58:15.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.689+0000 7f8ddeffd700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 0x7f8dd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:15.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.689+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8de0089650 con 0x7f8df0071130 2023-06-10T18:58:15.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.689+0000 7f8dedd9b700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 0x7f8dd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:15.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.690+0000 7f8dedd9b700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 0x7f8dd006d190 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8de4007fd0 tx=0x7f8de4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:15.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.693+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8de0057fa0 con 0x7f8df0071130 2023-06-10T18:58:15.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.925+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8df0071750 con 0x7f8df0071130 2023-06-10T18:58:15.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.925+0000 7f8ddeffd700 1 -- 172.21.15.104:0/748365797 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f8de000ce60 con 0x7f8df0071130 2023-06-10T18:58:15.932 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:15.932 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:58:15.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 msgr2=0x7f8dd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 0x7f8dd006d190 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8de4007fd0 tx=0x7f8de4009040 comp rx=0 tx=0).stop 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 msgr2=0x7f8df0158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e9b0 tx=0x7f8de000ecc0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 shutdown_connections 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8dd006ace0 0x7f8dd006d190 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df015c040 0x7f8df00c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df0071130 0x7f8df0158910 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 --2- 172.21.15.104:0/748365797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8df00ae280 0x7f8df00c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 >> 172.21.15.104:0/748365797 conn(0x7f8df00b1610 msgr2=0x7f8df00afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 shutdown_connections 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:15.927+0000 7f8df4a32700 1 -- 172.21.15.104:0/748365797 wait complete. 2023-06-10T18:58:15.935 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:58:15.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:16.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.060+0000 7f31065bb700 1 Processor -- start 2023-06-10T18:58:16.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.060+0000 7f31065bb700 1 -- start start 2023-06-10T18:58:16.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.061+0000 7f31065bb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 0x7f310006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.061+0000 7f31065bb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.061+0000 7f31065bb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310015cb00 con 0x7f31000cc000 2023-06-10T18:58:16.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.061+0000 7f31065bb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31000aed80 con 0x7f310015c730 2023-06-10T18:58:16.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.061+0000 7f31065bb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31000c9c60 con 0x7f31000cb0c0 2023-06-10T18:58:16.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.071 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f30fffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 0x7f310006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:59326/0 (socket says 172.21.15.104:59326) 2023-06-10T18:58:16.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 -- 172.21.15.104:0/2618257969 learned_addr learned my addr 172.21.15.104:0/2618257969 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:16.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f30fffff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 0x7f310006e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:55674/0 (socket says 172.21.15.104:55674) 2023-06-10T18:58:16.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 -- 172.21.15.104:0/2618257969 >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 msgr2=0x7f310006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:16.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 --2- 172.21.15.104:0/2618257969 >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 0x7f310006e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 -- 172.21.15.104:0/2618257969 >> v1:172.21.15.114:6789/0 conn(0x7f31000cc000 legacy=0x7f310006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 -- 172.21.15.104:0/2618257969 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310006ecd0 con 0x7f31000cb0c0 2023-06-10T18:58:16.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f3104b58700 1 --2- 172.21.15.104:0/2618257969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f30f800d150 tx=0x7f30f800d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba371629844670b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f30feffd700 1 -- 172.21.15.104:0/2618257969 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8010070 con 0x7f31000cb0c0 2023-06-10T18:58:16.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.062+0000 7f30feffd700 1 -- 172.21.15.104:0/2618257969 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f800dd10 con 0x7f31000cb0c0 2023-06-10T18:58:16.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f30feffd700 1 -- 172.21.15.104:0/2618257969 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800b310 con 0x7f31000cb0c0 2023-06-10T18:58:16.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 -- 172.21.15.104:0/2618257969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 msgr2=0x7f31000cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 --2- 172.21.15.104:0/2618257969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f30f800d150 tx=0x7f30f800d460 comp rx=0 tx=0).stop 2023-06-10T18:58:16.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 -- 172.21.15.104:0/2618257969 shutdown_connections 2023-06-10T18:58:16.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 --2- 172.21.15.104:0/2618257969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cb0c0 0x7f31000cc470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 --2- 172.21.15.104:0/2618257969 >> v2:172.21.15.114:3300/0 conn(0x7f310015c730 0x7f310006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 -- 172.21.15.104:0/2618257969 >> 172.21.15.104:0/2618257969 conn(0x7f310006b8a0 msgr2=0x7f31000cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:16.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 -- 172.21.15.104:0/2618257969 shutdown_connections 2023-06-10T18:58:16.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.063+0000 7f31065bb700 1 -- 172.21.15.104:0/2618257969 wait complete. 2023-06-10T18:58:16.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.064+0000 7f31065bb700 1 Processor -- start 2023-06-10T18:58:16.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.064+0000 7f31065bb700 1 -- start start 2023-06-10T18:58:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31000cb0c0 0x7f3100154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cc000 0x7f31000af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31001f5730 con 0x7f310015c730 2023-06-10T18:58:16.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31001f5fd0 con 0x7f31000cb0c0 2023-06-10T18:58:16.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f31065bb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31000c5cf0 con 0x7f31000cc000 2023-06-10T18:58:16.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f3104b58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f3104b58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39948/0 (socket says 172.21.15.104:39948) 2023-06-10T18:58:16.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f3104b58700 1 -- 172.21.15.104:0/1979718976 learned_addr learned my addr 172.21.15.104:0/1979718976 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:16.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f30ff7fe700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cc000 0x7f31000af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.065+0000 7f30fffff700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31000cb0c0 0x7f3100154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 -- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cc000 msgr2=0x7f31000af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cc000 0x7f31000af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 -- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31000cb0c0 msgr2=0x7f3100154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31000cb0c0 0x7f3100154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 -- 172.21.15.104:0/1979718976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f800bd40 con 0x7f310015c730 2023-06-10T18:58:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f3104b58700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f31000aeec0 tx=0x7f30f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8009a10 con 0x7f310015c730 2023-06-10T18:58:16.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f8009b70 con 0x7f310015c730 2023-06-10T18:58:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800b4c0 con 0x7f310015c730 2023-06-10T18:58:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31001f5910 con 0x7f310015c730 2023-06-10T18:58:16.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.066+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3100155260 con 0x7f310015c730 2023-06-10T18:58:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.067+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31001f6110 con 0x7f310015c730 2023-06-10T18:58:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.067+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f30f8004040 con 0x7f310015c730 2023-06-10T18:58:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.068+0000 7f30fcff9700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 0x7f30ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.068+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f30f808ac20 con 0x7f310015c730 2023-06-10T18:58:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.068+0000 7f30fffff700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 0x7f30ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.068+0000 7f30fffff700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 0x7f30ec06d190 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009bc0 tx=0x7f30f40091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.072+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f30f8055bf0 con 0x7f310015c730 2023-06-10T18:58:16.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.304+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3100066f10 con 0x7f310015c730 2023-06-10T18:58:16.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.304+0000 7f30fcff9700 1 -- 172.21.15.104:0/1979718976 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4692 (secure 0 0 0) 0x7f30f8059210 con 0x7f310015c730 2023-06-10T18:58:16.311 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:16.311 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:57:44.258356+0000","modified":"2023-06-10T18:57:47.275932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10107":{"gid":10107,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6834/2386386628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":2386386628},{"type":"v1","addr":"172.21.15.104:6834","nonce":2386386628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-06-10T18:58:16.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.306+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 msgr2=0x7f30ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.306+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 0x7f30ec06d190 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009bc0 tx=0x7f30f40091c0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.306+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 msgr2=0x7f31000adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.306+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f31000aeec0 tx=0x7f30f8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 shutdown_connections 2023-06-10T18:58:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f30ec06ace0 0x7f30ec06d190 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31000cb0c0 0x7f3100154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f310015c730 0x7f31000adac0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 --2- 172.21.15.104:0/1979718976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31000cc000 0x7f31000af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 >> 172.21.15.104:0/1979718976 conn(0x7f310006b8a0 msgr2=0x7f310006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:16.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 shutdown_connections 2023-06-10T18:58:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.307+0000 7f31065bb700 1 -- 172.21.15.104:0/1979718976 wait complete. 2023-06-10T18:58:16.317 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 102 2023-06-10T18:58:16.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:16.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.439+0000 7fd168815700 1 Processor -- start 2023-06-10T18:58:16.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.439+0000 7fd168815700 1 -- start start 2023-06-10T18:58:16.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd168815700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd16406b1f0 0x7fd1640cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd168815700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640cadd0 con 0x7fd1640ce350 2023-06-10T18:58:16.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd168815700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640cb0b0 con 0x7fd16408a3d0 2023-06-10T18:58:16.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd168815700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640cfeb0 con 0x7fd16406b1f0 2023-06-10T18:58:16.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd16406b1f0 0x7fd1640cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd161d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd1640ce350 0x7fd1640cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37780/0 (socket says 172.21.15.104:37780) 2023-06-10T18:58:16.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd161d9b700 1 -- 172.21.15.104:0/389364106 learned_addr learned my addr 172.21.15.104:0/389364106 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16259c700 1 -- 172.21.15.104:0/389364106 >> v1:172.21.15.114:6789/0 conn(0x7fd16408a3d0 legacy=0x7fd1640cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16259c700 1 -- 172.21.15.104:0/389364106 >> v1:172.21.15.104:6789/0 conn(0x7fd1640ce350 legacy=0x7fd1640cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16159a700 1 -- 172.21.15.104:0/389364106 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396271681 0 0) 0x7fd1640cadd0 con 0x7fd1640ce350 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16259c700 1 -- 172.21.15.104:0/389364106 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640cf550 con 0x7fd16406b1f0 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16259c700 1 --2- 172.21.15.104:0/389364106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16406b1f0 0x7fd1640cd090 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fd154009840 tx=0x7fd154009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e914f6e31a8efa3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16159a700 1 -- 172.21.15.104:0/389364106 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154004230 con 0x7fd16406b1f0 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16159a700 1 -- 172.21.15.104:0/389364106 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd154004390 con 0x7fd16406b1f0 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.440+0000 7fd16159a700 1 -- 172.21.15.104:0/389364106 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540036a0 con 0x7fd16406b1f0 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 -- 172.21.15.104:0/389364106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16406b1f0 msgr2=0x7fd1640cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 --2- 172.21.15.104:0/389364106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16406b1f0 0x7fd1640cd090 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fd154009840 tx=0x7fd154009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 -- 172.21.15.104:0/389364106 shutdown_connections 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 --2- 172.21.15.104:0/389364106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16406b1f0 0x7fd1640cd090 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 -- 172.21.15.104:0/389364106 >> 172.21.15.104:0/389364106 conn(0x7fd164089fd0 msgr2=0x7fd1640d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 -- 172.21.15.104:0/389364106 shutdown_connections 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.441+0000 7fd168815700 1 -- 172.21.15.104:0/389364106 wait complete. 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.442+0000 7fd168815700 1 Processor -- start 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.442+0000 7fd168815700 1 -- start start 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 0x7fd164156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1640ce350 0x7fd1640ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640ca5c0 con 0x7fd16408a3d0 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641568d0 con 0x7fd16406b1f0 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd168815700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415cf30 con 0x7fd1640ce350 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd161d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd161d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39962/0 (socket says 172.21.15.104:39962) 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd161d9b700 1 -- 172.21.15.104:0/2645010819 learned_addr learned my addr 172.21.15.104:0/2645010819 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.443+0000 7fd162d9d700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1640ce350 0x7fd1640ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd16259c700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 0x7fd164156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 -- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1640ce350 msgr2=0x7fd1640ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1640ce350 0x7fd1640ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 -- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 msgr2=0x7fd164156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 0x7fd164156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 -- 172.21.15.104:0/2645010819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1540094a0 con 0x7fd16408a3d0 2023-06-10T18:58:16.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd16259c700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 0x7fd164156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd161d9b700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fd15800c520 tx=0x7fd15800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158004c80 con 0x7fd16408a3d0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158004de0 con 0x7fd16408a3d0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1641e60a0 con 0x7fd16408a3d0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580072d0 con 0x7fd16408a3d0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.444+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1641e6660 con 0x7fd16408a3d0 2023-06-10T18:58:16.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.445+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd164074050 con 0x7fd16408a3d0 2023-06-10T18:58:16.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.445+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd158007430 con 0x7fd16408a3d0 2023-06-10T18:58:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.446+0000 7fd152ffd700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 0x7fd14406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.446+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd158089240 con 0x7fd16408a3d0 2023-06-10T18:58:16.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.446+0000 7fd16259c700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 0x7fd14406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.447+0000 7fd16259c700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 0x7fd14406d190 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd154000c00 tx=0x7fd15401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.450+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd158054180 con 0x7fd16408a3d0 2023-06-10T18:58:16.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.638+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd164072bf0 con 0x7fd16408a3d0 2023-06-10T18:58:16.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.639+0000 7fd152ffd700 1 -- 172.21.15.104:0/2645010819 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+18666 (secure 0 0 0) 0x7fd1580577a0 con 0x7fd16408a3d0 2023-06-10T18:58:16.646 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:16.646 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":160,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:57:58.687497+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:57:42.097445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-06-10T18:57:43.215419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/3682236906":"2023-06-10T19:57:58.572945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:16.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 msgr2=0x7fd14406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 0x7fd14406d190 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fd154000c00 tx=0x7fd15401c040 comp rx=0 tx=0).stop 2023-06-10T18:58:16.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 msgr2=0x7fd1640c9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fd15800c520 tx=0x7fd15800c830 comp rx=0 tx=0).stop 2023-06-10T18:58:16.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 shutdown_connections 2023-06-10T18:58:16.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd14406ace0 0x7fd14406d190 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd16406b1f0 0x7fd164156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1640ce350 0x7fd1640ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 --2- 172.21.15.104:0/2645010819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd16408a3d0 0x7fd1640c9b40 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 >> 172.21.15.104:0/2645010819 conn(0x7fd164089fd0 msgr2=0x7fd1640cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:16.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 shutdown_connections 2023-06-10T18:58:16.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.641+0000 7fd168815700 1 -- 172.21.15.104:0/2645010819 wait complete. 2023-06-10T18:58:16.660 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:58:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.776+0000 7f3e91c3f700 1 Processor -- start 2023-06-10T18:58:16.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.776+0000 7f3e91c3f700 1 -- start start 2023-06-10T18:58:16.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e91c3f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3e8c158e50 0x7f3e8c0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e91c3f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0c8de0 con 0x7f3e8c0cb5b0 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e91c3f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0ccc90 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e91c3f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c0ccf30 con 0x7f3e8c158e50 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e8bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3e8c0cb5b0 0x7f3e8c0c9270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37794/0 (socket says 172.21.15.104:37794) 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e8bfff700 1 -- 172.21.15.104:0/338941416 learned_addr learned my addr 172.21.15.104:0/338941416 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e8b7fe700 1 --2- 172.21.15.104:0/338941416 >> v2:172.21.15.114:3300/0 conn(0x7f3e8c158e50 0x7f3e8c0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034092774 0 0) 0x7f3e8c0c8de0 con 0x7f3e8c0cb5b0 2023-06-10T18:58:16.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.777+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e780035b0 con 0x7f3e8c0cb5b0 2023-06-10T18:58:16.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885899261 0 0) 0x7f3e780035b0 con 0x7f3e8c0cb5b0 2023-06-10T18:58:16.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8b7fe700 1 -- 172.21.15.104:0/338941416 >> v1:172.21.15.114:6789/0 conn(0x7f3e8c0cdbb0 legacy=0x7f3e8c0caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8b7fe700 1 -- 172.21.15.104:0/338941416 >> v1:172.21.15.104:6789/0 conn(0x7f3e8c0cb5b0 legacy=0x7f3e8c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8b7fe700 1 -- 172.21.15.104:0/338941416 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0cd210 con 0x7f3e8c158e50 2023-06-10T18:58:16.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8b7fe700 1 --2- 172.21.15.104:0/338941416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c158e50 0x7f3e8c0cd670 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009840 tx=0x7f3e74009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b542b6490892d338 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e840019f0 con 0x7f3e8c158e50 2023-06-10T18:58:16.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e74004030 con 0x7f3e8c158e50 2023-06-10T18:58:16.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.778+0000 7f3e8a7fc700 1 -- 172.21.15.104:0/338941416 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740043a0 con 0x7f3e8c158e50 2023-06-10T18:58:16.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 -- 172.21.15.104:0/338941416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c158e50 msgr2=0x7f3e8c0cd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/338941416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c158e50 0x7f3e8c0cd670 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009840 tx=0x7f3e74009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:16.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 -- 172.21.15.104:0/338941416 shutdown_connections 2023-06-10T18:58:16.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/338941416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c158e50 0x7f3e8c0cd670 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 -- 172.21.15.104:0/338941416 >> 172.21.15.104:0/338941416 conn(0x7f3e8c06ab00 msgr2=0x7f3e8c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:16.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 -- 172.21.15.104:0/338941416 shutdown_connections 2023-06-10T18:58:16.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.779+0000 7f3e91c3f700 1 -- 172.21.15.104:0/338941416 wait complete. 2023-06-10T18:58:16.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.780+0000 7f3e91c3f700 1 Processor -- start 2023-06-10T18:58:16.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.780+0000 7f3e91c3f700 1 -- start start 2023-06-10T18:58:16.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c0cb5b0 0x7f3e8c1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e8c158e50 0x7f3e8c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1ede10 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0c84e0 con 0x7f3e8c0cb5b0 2023-06-10T18:58:16.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1ee010 con 0x7f3e8c158e50 2023-06-10T18:58:16.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:39974/0 (socket says 172.21.15.104:39974) 2023-06-10T18:58:16.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 -- 172.21.15.104:0/2236546003 learned_addr learned my addr 172.21.15.104:0/2236546003 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8bfff700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e8c158e50 0x7f3e8c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8b7fe700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c0cb5b0 0x7f3e8c1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 -- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e8c158e50 msgr2=0x7f3e8c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e8c158e50 0x7f3e8c0c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 -- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c0cb5b0 msgr2=0x7f3e8c1f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:16.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c0cb5b0 0x7f3e8c1f1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 -- 172.21.15.104:0/2236546003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e740094a0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e8affd700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c6b0 tx=0x7f3e7c00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0052d0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c005450 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c1ee2f0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.781+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0038c0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.782+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c1ee8b0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.782+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c074050 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.783+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3e7c003a20 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.783+0000 7f3e83fff700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 0x7f3e6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:16.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.783+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f3e7c088ac0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:16.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.783+0000 7f3e8b7fe700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 0x7f3e6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:16.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.784+0000 7f3e8b7fe700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 0x7f3e6c06d190 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009b90 tx=0x7f3e74013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:16.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:16.787+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3e7c053980 con 0x7f3e8c0cdbb0 2023-06-10T18:58:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.017+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3e8c0c89b0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:17.755 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:58:17.748+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:58:17.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.756+0000 7f3e83fff700 1 -- 172.21.15.104:0/2236546003 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v103) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3e7c056fa0 con 0x7f3e8c0cdbb0 2023-06-10T18:58:17.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 msgr2=0x7f3e6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 0x7f3e6c06d190 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009b90 tx=0x7f3e74013040 comp rx=0 tx=0).stop 2023-06-10T18:58:17.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 msgr2=0x7f3e8c1523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c6b0 tx=0x7f3e7c00ca70 comp rx=0 tx=0).stop 2023-06-10T18:58:17.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 shutdown_connections 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3e6c06ace0 0x7f3e6c06d190 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e8c0cb5b0 0x7f3e8c1f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3e8c158e50 0x7f3e8c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 --2- 172.21.15.104:0/2236546003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e8c0cdbb0 0x7f3e8c1523e0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 >> 172.21.15.104:0/2236546003 conn(0x7f3e8c06ab00 msgr2=0x7f3e8c0ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:17.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 shutdown_connections 2023-06-10T18:58:17.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.759+0000 7f3e91c3f700 1 -- 172.21.15.104:0/2236546003 wait complete. 2023-06-10T18:58:17.768 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:58:17.768 INFO:tasks.ceph.mds.a.smithi104.stderr: -7619> 2023-06-10T18:57:49.839+0000 7f415be21700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:58:17.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:58:17.805 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:58:17.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.903+0000 7f0904033700 1 Processor -- start 2023-06-10T18:58:17.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f0904033700 1 -- start start 2023-06-10T18:58:17.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f0904033700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08fc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f0904033700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc0ce310 con 0x7f08fc0cf470 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f0904033700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc0ce8d0 con 0x7f08fc0cad90 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f0904033700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc15e600 con 0x7f08fc0b6030 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f09025d0700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f08fc0cf470 0x7f08fc0cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37824/0 (socket says 172.21.15.104:37824) 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.904+0000 7f09025d0700 1 -- 172.21.15.104:0/3154271011 learned_addr learned my addr 172.21.15.104:0/3154271011 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08fc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008144161 0 0) 0x7f08fc0ce310 con 0x7f08fc0cf470 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e80035b0 con 0x7f08fc0cf470 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107050309 0 0) 0x7f08e80035b0 con 0x7f08fc0cf470 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 msgr2=0x7f08fc0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08fc0cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 >> v1:172.21.15.114:6789/0 conn(0x7f08fc0cad90 legacy=0x7f08fc0ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc15e8e0 con 0x7f08fc0cf470 2023-06-10T18:58:17.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f08f40019f0 con 0x7f08fc0cf470 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 >> v1:172.21.15.104:6789/0 conn(0x7f08fc0cf470 legacy=0x7f08fc0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e8003a30 0x7f08e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8004d90 0x7f08e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc15e600 con 0x7f08e8003a30 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e800bb20 con 0x7f08e8004d90 2023-06-10T18:58:17.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e800bce0 con 0x7f08fc0b6030 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e8003a30 0x7f08e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f09015ce700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f09025d0700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8004d90 0x7f08e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 -- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 msgr2=0x7f08e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc0b6030 0x7f08e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 -- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8004d90 msgr2=0x7f08e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8004d90 0x7f08e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f0901dcf700 1 -- 172.21.15.104:0/3154271011 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e80035b0 con 0x7f08e8003a30 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.905+0000 7f09025d0700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8004d90 0x7f08e8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e8003a30 0x7f08e8003e00 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009b60 tx=0x7f08e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551fd43264ca8c03 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec00e6c0 con 0x7f08e8003a30 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec00e820 con 0x7f08e8003a30 2023-06-10T18:58:17.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0900dcd700 1 -- 172.21.15.104:0/3154271011 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec00eb90 con 0x7f08e8003a30 2023-06-10T18:58:17.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 -- 172.21.15.104:0/3154271011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e8003a30 msgr2=0x7f08e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 --2- 172.21.15.104:0/3154271011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08e8003a30 0x7f08e8003e00 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009b60 tx=0x7f08e8005880 comp rx=0 tx=0).stop 2023-06-10T18:58:17.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f09025d0700 1 -- 172.21.15.104:0/3154271011 reap_dead start 2023-06-10T18:58:17.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 -- 172.21.15.104:0/3154271011 shutdown_connections 2023-06-10T18:58:17.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 -- 172.21.15.104:0/3154271011 >> 172.21.15.104:0/3154271011 conn(0x7f08fc06e370 msgr2=0x7f08fc06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:17.920 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 -- 172.21.15.104:0/3154271011 shutdown_connections 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.906+0000 7f0904033700 1 -- 172.21.15.104:0/3154271011 wait complete. 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 Processor -- start 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 -- start start 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 0x7f08fc0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc15c0d0 0x7f08fc15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c4510 con 0x7f08fc1e49f0 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1e5fc0 con 0x7f08e8003a30 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0904033700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc15ca80 con 0x7f08fc15c0d0 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f09025d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f09025d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40008/0 (socket says 172.21.15.104:40008) 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f09025d0700 1 -- 172.21.15.104:0/3199416339 learned_addr learned my addr 172.21.15.104:0/3199416339 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 0x7f08fc0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.922 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.908+0000 7f09015ce700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc15c0d0 0x7f08fc15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 -- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc15c0d0 msgr2=0x7f08fc15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc15c0d0 0x7f08fc15c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 -- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 msgr2=0x7f08fc0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 0x7f08fc0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 -- 172.21.15.104:0/3199416339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc1e5720 con 0x7f08fc1e49f0 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 0x7f08fc0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f09025d0700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0d0630 tx=0x7f08f400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:17.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4017070 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f400eb90 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc1e5a00 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f400ed40 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.909+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc1e53d0 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.910+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc0c4270 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.910+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f08f4013420 con 0x7f08fc1e49f0 2023-06-10T18:58:17.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.911+0000 7f08f27fc700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 0x7f08e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:17.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.911+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f08f4089150 con 0x7f08fc1e49f0 2023-06-10T18:58:17.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.911+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 0x7f08e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:17.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.912+0000 7f0901dcf700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 0x7f08e006d190 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0090b0 tx=0x7f08ec01f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:17.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:17.915+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f08fc0c4270 con 0x7f08fc1e49f0 2023-06-10T18:58:18.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.144+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08fc0c4650 con 0x7f08fc1e49f0 2023-06-10T18:58:18.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.760+0000 7f08f27fc700 1 -- 172.21.15.104:0/3199416339 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v104) v1 ==== 104+0+0 (secure 0 0 0) 0x7f08fc0c4650 con 0x7f08fc1e49f0 2023-06-10T18:58:18.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 msgr2=0x7f08e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 0x7f08e006d190 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0090b0 tx=0x7f08ec01f040 comp rx=0 tx=0).stop 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 msgr2=0x7f08fc1e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0d0630 tx=0x7f08f400e440 comp rx=0 tx=0).stop 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 shutdown_connections 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08e006ace0 0x7f08e006d190 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e8003a30 0x7f08fc0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08fc1e49f0 0x7f08fc1e4e90 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 --2- 172.21.15.104:0/3199416339 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08fc15c0d0 0x7f08fc15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 >> 172.21.15.104:0/3199416339 conn(0x7f08fc06e370 msgr2=0x7f08fc159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:18.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 shutdown_connections 2023-06-10T18:58:18.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.763+0000 7f0904033700 1 -- 172.21.15.104:0/3199416339 wait complete. 2023-06-10T18:58:18.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:58:18.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.944+0000 7f6d6e7cc700 1 Processor -- start 2023-06-10T18:58:18.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.944+0000 7f6d6e7cc700 1 -- start start 2023-06-10T18:58:18.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6d6815c730 0x7f6d680cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 0x7f6d6806cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d6806d2b0 0x7f6d6806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d680cdb90 con 0x7f6d6806d2b0 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6815cb00 con 0x7f6d6815c730 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6e7cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680acd90 con 0x7f6d680cd720 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6cd69700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d6806d2b0 0x7f6d6806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6cd69700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6d6806d2b0 0x7f6d6806d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40018/0 (socket says 172.21.15.104:40018) 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.945+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1803596449 learned_addr learned my addr 172.21.15.104:0/1803596449 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d677fe700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 0x7f6d6806cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d67fff700 1 --2- 172.21.15.104:0/1803596449 >> v2:172.21.15.114:3300/0 conn(0x7f6d6815c730 0x7f6d680cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1803596449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 msgr2=0x7f6d6806cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 0x7f6d6806cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1803596449 >> v2:172.21.15.114:3300/0 conn(0x7f6d6815c730 msgr2=0x7f6d680cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1803596449 >> v2:172.21.15.114:3300/0 conn(0x7f6d6815c730 0x7f6d680cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1803596449 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680ac460 con 0x7f6d6806d2b0 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d677fe700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 0x7f6d6806cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6806d2b0 0x7f6d6806d730 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f6d680cde10 tx=0x7f6d6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9445606e5ec916 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d66ffd700 1 -- 172.21.15.104:0/1803596449 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000e070 con 0x7f6d6806d2b0 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d66ffd700 1 -- 172.21.15.104:0/1803596449 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6000b9c0 con 0x7f6d6806d2b0 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d66ffd700 1 -- 172.21.15.104:0/1803596449 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000bd70 con 0x7f6d6806d2b0 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1803596449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6806d2b0 msgr2=0x7f6d6806d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:18.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.946+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6806d2b0 0x7f6d6806d730 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f6d680cde10 tx=0x7f6d6000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1803596449 shutdown_connections 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6806d2b0 0x7f6d6806d730 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1803596449 >> v2:172.21.15.114:3300/0 conn(0x7f6d6815c730 0x7f6d680cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1803596449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d680cd720 0x7f6d6806cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1803596449 >> 172.21.15.104:0/1803596449 conn(0x7f6d6806b8a0 msgr2=0x7f6d680cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1803596449 shutdown_connections 2023-06-10T18:58:18.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.947+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1803596449 wait complete. 2023-06-10T18:58:18.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.948+0000 7f6d6e7cc700 1 Processor -- start 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.948+0000 7f6d6e7cc700 1 -- start start 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d6806d2b0 0x7f6d6815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d680cd720 0x7f6d680c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6815a6e0 con 0x7f6d6815c730 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680c5380 con 0x7f6d680cd720 2023-06-10T18:58:18.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680c5620 con 0x7f6d6806d2b0 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40024/0 (socket says 172.21.15.104:40024) 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1336557287 learned_addr learned my addr 172.21.15.104:0/1336557287 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d677fe700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d680cd720 0x7f6d680c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d67fff700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d6806d2b0 0x7f6d6815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d6806d2b0 msgr2=0x7f6d6815a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d6806d2b0 0x7f6d6815a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d680cd720 msgr2=0x7f6d680c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d680cd720 0x7f6d680c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 -- 172.21.15.104:0/1336557287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60009b30 con 0x7f6d6815c730 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6cd69700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b370 tx=0x7f6d60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60004250 con 0x7f6d6815c730 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d600043f0 con 0x7f6d6815c730 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d680c5900 con 0x7f6d6815c730 2023-06-10T18:58:18.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.949+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d600047a0 con 0x7f6d6815c730 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.950+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d680c5ec0 con 0x7f6d6815c730 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.950+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68066f10 con 0x7f6d6815c730 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.951+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6d6001c440 con 0x7f6d6815c730 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.951+0000 7f6d64ff9700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 0x7f6d5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.951+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f6d6008ab10 con 0x7f6d6815c730 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.951+0000 7f6d67fff700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 0x7f6d5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:18.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.952+0000 7f6d67fff700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 0x7f6d5406d190 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800aaa0 tx=0x7f6d58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:18.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:18.956+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6d60055b40 con 0x7f6d6815c730 2023-06-10T18:58:19.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.142+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d68074050 con 0x7f6d6815c730 2023-06-10T18:58:19.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.769+0000 7f6d64ff9700 1 -- 172.21.15.104:0/1336557287 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v162) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6d600200d0 con 0x7f6d6815c730 2023-06-10T18:58:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.771+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 msgr2=0x7f6d5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.771+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 0x7f6d5406d190 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800aaa0 tx=0x7f6d58008040 comp rx=0 tx=0).stop 2023-06-10T18:58:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.771+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 msgr2=0x7f6d68146360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.771+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b370 tx=0x7f6d60004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 shutdown_connections 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d6815c730 0x7f6d68146360 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d6806d2b0 0x7f6d6815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6d5406ace0 0x7f6d5406d190 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 --2- 172.21.15.104:0/1336557287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d680cd720 0x7f6d680c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 >> 172.21.15.104:0/1336557287 conn(0x7f6d6806b8a0 msgr2=0x7f6d681e1950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:19.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 shutdown_connections 2023-06-10T18:58:19.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.772+0000 7f6d6e7cc700 1 -- 172.21.15.104:0/1336557287 wait complete. 2023-06-10T18:58:19.780 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:58:19.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:58:19.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.917+0000 7fda6dcc5700 1 Processor -- start 2023-06-10T18:58:19.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.917+0000 7fda6dcc5700 1 -- start start 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda6dcc5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda6806e6e0 0x7fda680b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda6dcc5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6806e250 con 0x7fda6806cde0 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda6dcc5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6815c950 con 0x7fda6815c580 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda6dcc5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda680b1e80 con 0x7fda6806e6e0 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda67fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fda6806cde0 0x7fda680ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37860/0 (socket says 172.21.15.104:37860) 2023-06-10T18:58:19.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.918+0000 7fda67fff700 1 -- 172.21.15.104:0/980741683 learned_addr learned my addr 172.21.15.104:0/980741683 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047114225 0 0) 0x7fda6806e250 con 0x7fda6806cde0 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda66ffd700 1 --2- 172.21.15.104:0/980741683 >> v2:172.21.15.114:3300/0 conn(0x7fda6806e6e0 0x7fda680b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda540035b0 con 0x7fda6806cde0 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278877610 0 0) 0x7fda540035b0 con 0x7fda6806cde0 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 >> v2:172.21.15.114:3300/0 conn(0x7fda6806e6e0 msgr2=0x7fda680b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 --2- 172.21.15.104:0/980741683 >> v2:172.21.15.114:3300/0 conn(0x7fda6806e6e0 0x7fda680b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 >> v1:172.21.15.114:6789/0 conn(0x7fda6815c580 legacy=0x7fda680b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6806d920 con 0x7fda6806cde0 2023-06-10T18:58:19.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fda600019f0 con 0x7fda6806cde0 2023-06-10T18:58:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 >> v1:172.21.15.104:6789/0 conn(0x7fda6806cde0 legacy=0x7fda680ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 0x7fda54003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54004850 0x7fda54004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda54005200 0x7fda5400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda680b1e80 con 0x7fda54003a00 2023-06-10T18:58:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5400c080 con 0x7fda54004850 2023-06-10T18:58:19.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5400c240 con 0x7fda54005200 2023-06-10T18:58:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 0x7fda54003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda66ffd700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54004850 0x7fda54004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda67fff700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda54005200 0x7fda5400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 -- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda54005200 msgr2=0x7fda5400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda54005200 0x7fda5400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 -- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54004850 msgr2=0x7fda54004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54004850 0x7fda54004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda677fe700 1 -- 172.21.15.104:0/980741683 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda540035b0 con 0x7fda54003a00 2023-06-10T18:58:19.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda67fff700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda54005200 0x7fda5400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.919+0000 7fda67fff700 1 -- 172.21.15.104:0/980741683 reap_dead start 2023-06-10T18:58:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda677fe700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 0x7fda54003e10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fda50001820 tx=0x7fda54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae94b6681056b254 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500012e0 con 0x7fda54003a00 2023-06-10T18:58:19.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda500052d0 con 0x7fda54003a00 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda667fc700 1 -- 172.21.15.104:0/980741683 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500036d0 con 0x7fda54003a00 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 -- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 msgr2=0x7fda54003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 0x7fda54003e10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fda50001820 tx=0x7fda54005880 comp rx=0 tx=0).stop 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 -- 172.21.15.104:0/980741683 shutdown_connections 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/980741683 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda54003a00 0x7fda54003e10 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 -- 172.21.15.104:0/980741683 >> 172.21.15.104:0/980741683 conn(0x7fda6806b8a0 msgr2=0x7fda680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 -- 172.21.15.104:0/980741683 shutdown_connections 2023-06-10T18:58:19.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.920+0000 7fda6dcc5700 1 -- 172.21.15.104:0/980741683 wait complete. 2023-06-10T18:58:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.921+0000 7fda6dcc5700 1 Processor -- start 2023-06-10T18:58:19.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 -- start start 2023-06-10T18:58:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54003a00 0x7fda68153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda680ad8a0 0x7fda680c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda680cf280 con 0x7fda680cfd20 2023-06-10T18:58:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda680ade10 con 0x7fda54003a00 2023-06-10T18:58:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda6dcc5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda680ae0f0 con 0x7fda680ad8a0 2023-06-10T18:58:19.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.922+0000 7fda67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40044/0 (socket says 172.21.15.104:40044) 2023-06-10T18:58:19.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 -- 172.21.15.104:0/3902538113 learned_addr learned my addr 172.21.15.104:0/3902538113 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:19.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda66ffd700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda680ad8a0 0x7fda680c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda677fe700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54003a00 0x7fda68153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 -- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda680ad8a0 msgr2=0x7fda680c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda680ad8a0 0x7fda680c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 -- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54003a00 msgr2=0x7fda68153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:19.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54003a00 0x7fda68153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:19.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 -- 172.21.15.104:0/3902538113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680ae3d0 con 0x7fda680cfd20 2023-06-10T18:58:19.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda67fff700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fda68070390 tx=0x7fda6000e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:19.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6000ed80 con 0x7fda680cfd20 2023-06-10T18:58:19.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda60013420 con 0x7fda680cfd20 2023-06-10T18:58:19.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda680cf520 con 0x7fda680cfd20 2023-06-10T18:58:19.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6000eee0 con 0x7fda680cfd20 2023-06-10T18:58:19.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.923+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda68153bc0 con 0x7fda680cfd20 2023-06-10T18:58:19.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.924+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda68074050 con 0x7fda680cfd20 2023-06-10T18:58:19.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.924+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fda600137d0 con 0x7fda680cfd20 2023-06-10T18:58:19.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.925+0000 7fda5ffff700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 0x7fda4806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:19.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.925+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda60089b50 con 0x7fda680cfd20 2023-06-10T18:58:19.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.925+0000 7fda677fe700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 0x7fda4806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.925+0000 7fda677fe700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 0x7fda4806d190 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fda50005040 tx=0x7fda50009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:19.929+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fda60054bf0 con 0x7fda680cfd20 2023-06-10T18:58:20.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.115+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fda68066f10 con 0x7fda680cfd20 2023-06-10T18:58:20.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.775+0000 7fda5ffff700 1 -- 172.21.15.104:0/3902538113 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v163) v1 ==== 169+0+0 (secure 0 0 0) 0x7fda60058210 con 0x7fda680cfd20 2023-06-10T18:58:20.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 msgr2=0x7fda4806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 0x7fda4806d190 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fda50005040 tx=0x7fda50009030 comp rx=0 tx=0).stop 2023-06-10T18:58:20.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 msgr2=0x7fda680b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fda68070390 tx=0x7fda6000e4f0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 shutdown_connections 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fda4806ace0 0x7fda4806d190 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda54003a00 0x7fda68153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fda680cfd20 0x7fda680b0560 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 --2- 172.21.15.104:0/3902538113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda680ad8a0 0x7fda680c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 >> 172.21.15.104:0/3902538113 conn(0x7fda6806b8a0 msgr2=0x7fda68155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 shutdown_connections 2023-06-10T18:58:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.777+0000 7fda6dcc5700 1 -- 172.21.15.104:0/3902538113 wait complete. 2023-06-10T18:58:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:58:20.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:58:20.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.914+0000 7ff3a68eb700 1 Processor -- start 2023-06-10T18:58:20.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.914+0000 7ff3a68eb700 1 -- start start 2023-06-10T18:58:20.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a68eb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff3a015c6f0 0x7ff3a00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a68eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff3a00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a68eb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00ae7e0 con 0x7ff3a006ea70 2023-06-10T18:58:20.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a68eb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a006df90 con 0x7ff3a015c6f0 2023-06-10T18:58:20.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a68eb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a006e270 con 0x7ff3a00aec00 2023-06-10T18:58:20.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff3a006ea70 0x7ff3a00ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37878/0 (socket says 172.21.15.104:37878) 2023-06-10T18:58:20.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39f7fe700 1 -- 172.21.15.104:0/2650088787 learned_addr learned my addr 172.21.15.104:0/2650088787 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:20.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805953016 0 0) 0x7ff3a00ae7e0 con 0x7ff3a006ea70 2023-06-10T18:58:20.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c0035b0 con 0x7ff3a006ea70 2023-06-10T18:58:20.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39ffff700 1 --2- 172.21.15.104:0/2650088787 >> v2:172.21.15.114:3300/0 conn(0x7ff3a015c6f0 0x7ff3a00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff3a00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836662306 0 0) 0x7ff38c0035b0 con 0x7ff3a006ea70 2023-06-10T18:58:20.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 msgr2=0x7ff3a00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff3a00b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 >> v2:172.21.15.114:3300/0 conn(0x7ff3a015c6f0 msgr2=0x7ff3a00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 --2- 172.21.15.104:0/2650088787 >> v2:172.21.15.114:3300/0 conn(0x7ff3a015c6f0 0x7ff3a00b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a006e550 con 0x7ff3a006ea70 2023-06-10T18:58:20.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.915+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7ff390001670 con 0x7ff3a006ea70 2023-06-10T18:58:20.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 >> v1:172.21.15.104:6789/0 conn(0x7ff3a006ea70 legacy=0x7ff3a00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff38c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff38c003f80 0x7ff38c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff38c004d90 0x7ff38c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a006df90 con 0x7ff38c004d90 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a006e270 con 0x7ff38c003f80 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c00ba40 con 0x7ff3a00aec00 2023-06-10T18:58:20.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff38c004d90 0x7ff38c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39ffff700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff38c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39f7fe700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff38c003f80 0x7ff38c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 -- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 msgr2=0x7ff38c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00aec00 0x7ff38c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 -- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff38c003f80 msgr2=0x7ff38c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff38c003f80 0x7ff38c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 -- 172.21.15.104:0/2650088787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0035b0 con 0x7ff38c004d90 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff38c004d90 0x7ff38c005200 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7ff398009f90 tx=0x7ff38c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86be2ded447be343 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398010de0 con 0x7ff38c004d90 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398012430 con 0x7ff38c004d90 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.916+0000 7ff39effd700 1 -- 172.21.15.104:0/2650088787 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3980127c0 con 0x7ff38c004d90 2023-06-10T18:58:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 -- 172.21.15.104:0/2650088787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff38c004d90 msgr2=0x7ff38c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/2650088787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff38c004d90 0x7ff38c005200 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7ff398009f90 tx=0x7ff38c005880 comp rx=0 tx=0).stop 2023-06-10T18:58:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a4e88700 1 -- 172.21.15.104:0/2650088787 reap_dead start 2023-06-10T18:58:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 -- 172.21.15.104:0/2650088787 shutdown_connections 2023-06-10T18:58:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 -- 172.21.15.104:0/2650088787 >> 172.21.15.104:0/2650088787 conn(0x7ff3a006b8a0 msgr2=0x7ff3a00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 -- 172.21.15.104:0/2650088787 shutdown_connections 2023-06-10T18:58:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.917+0000 7ff3a68eb700 1 -- 172.21.15.104:0/2650088787 wait complete. 2023-06-10T18:58:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.918+0000 7ff3a68eb700 1 Processor -- start 2023-06-10T18:58:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.918+0000 7ff3a68eb700 1 -- start start 2023-06-10T18:58:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 0x7ff3a006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 0x7ff3a00c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a006a820 con 0x7ff3a01f0f60 2023-06-10T18:58:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a00b2420 con 0x7ff3a0069e70 2023-06-10T18:58:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a68eb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01f1330 con 0x7ff3a00c3090 2023-06-10T18:58:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40060/0 (socket says 172.21.15.104:40060) 2023-06-10T18:58:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 -- 172.21.15.104:0/3702602185 learned_addr learned my addr 172.21.15.104:0/3702602185 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39f7fe700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 0x7ff3a006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 0x7ff3a00c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 -- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 msgr2=0x7ff3a00c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 0x7ff3a00c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 -- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 msgr2=0x7ff3a006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 0x7ff3a006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 -- 172.21.15.104:0/3702602185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a00b2670 con 0x7ff3a01f0f60 2023-06-10T18:58:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39f7fe700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 0x7ff3a006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff3a4e88700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 0x7ff3a00c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.919+0000 7ff39ffff700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7ff388011a70 tx=0x7ff388011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.920+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388005a20 con 0x7ff3a01f0f60 2023-06-10T18:58:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.920+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388005b80 con 0x7ff3a01f0f60 2023-06-10T18:58:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.920+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00b2950 con 0x7ff3a01f0f60 2023-06-10T18:58:20.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.920+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38800f9a0 con 0x7ff3a01f0f60 2023-06-10T18:58:20.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.920+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a00b2f10 con 0x7ff3a01f0f60 2023-06-10T18:58:20.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.921+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0074050 con 0x7ff3a01f0f60 2023-06-10T18:58:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.921+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff38800fb00 con 0x7ff3a01f0f60 2023-06-10T18:58:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.921+0000 7ff39cff9700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.921+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5659+0+0 (secure 0 0 0) 0x7ff388089780 con 0x7ff3a01f0f60 2023-06-10T18:58:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.922+0000 7ff39f7fe700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.922+0000 7ff39f7fe700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 0x7ff38006d190 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff390000fb0 tx=0x7ff390009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:20.926+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3880549c0 con 0x7ff3a01f0f60 2023-06-10T18:58:21.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.111+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff3a00d0030 con 0x7ff3a01f0f60 2023-06-10T18:58:21.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.114+0000 7ff39cff9700 1 -- 172.21.15.104:0/3702602185 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff388057fe0 con 0x7ff3a01f0f60 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 msgr2=0x7ff38006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 0x7ff38006d190 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff390000fb0 tx=0x7ff390009040 comp rx=0 tx=0).stop 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 msgr2=0x7ff3a00c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7ff388011a70 tx=0x7ff388011e30 comp rx=0 tx=0).stop 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 shutdown_connections 2023-06-10T18:58:21.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3a00c3090 0x7ff3a00c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3a01f0f60 0x7ff3a00c4a70 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 --2- 172.21.15.104:0/3702602185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3a0069e70 0x7ff3a006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.116+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 >> 172.21.15.104:0/3702602185 conn(0x7ff3a006b8a0 msgr2=0x7ff3a00adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.117+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 shutdown_connections 2023-06-10T18:58:21.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.117+0000 7ff3a68eb700 1 -- 172.21.15.104:0/3702602185 wait complete. 2023-06-10T18:58:21.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction' 2023-06-10T18:58:21.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.257+0000 7fba189d0700 1 Processor -- start 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.257+0000 7fba189d0700 1 -- start start 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.258+0000 7fba189d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba140cadd0 0x7fba140cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.258+0000 7fba189d0700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140b2010 con 0x7fba140b15e0 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.258+0000 7fba189d0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140b1a50 con 0x7fba141609f0 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.258+0000 7fba189d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1408a690 con 0x7fba140cadd0 2023-06-10T18:58:21.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba11d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba140b15e0 0x7fba140b0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37898/0 (socket says 172.21.15.104:37898) 2023-06-10T18:58:21.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba11d9b700 1 -- 172.21.15.104:0/3080355024 learned_addr learned my addr 172.21.15.104:0/3080355024 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:21.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba12d9d700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba140cadd0 0x7fba140cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269224703 0 0) 0x7fba140b2010 con 0x7fba140b15e0 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc0035b0 con 0x7fba140b15e0 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326406132 0 0) 0x7fba140b1a50 con 0x7fba141609f0 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba140b2010 con 0x7fba141609f0 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400896685 0 0) 0x7fb9fc0035b0 con 0x7fba140b15e0 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba140cadd0 msgr2=0x7fba140cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba140cadd0 0x7fba140cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 >> v1:172.21.15.114:6789/0 conn(0x7fba141609f0 legacy=0x7fba140d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140b1d30 con 0x7fba140b15e0 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.259+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fba08001670 con 0x7fba140b15e0 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 >> v1:172.21.15.104:6789/0 conn(0x7fba140b15e0 legacy=0x7fba140b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9fc003bf0 0x7fb9fc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc004850 0x7fb9fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc005200 0x7fb9fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba04001230 con 0x7fb9fc003bf0 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1408a690 con 0x7fb9fc004850 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc00c050 con 0x7fb9fc005200 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9fc003bf0 0x7fb9fc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba11d9b700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc004850 0x7fb9fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba12d9d700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc005200 0x7fb9fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 -- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc005200 msgr2=0x7fb9fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc005200 0x7fb9fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 -- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc004850 msgr2=0x7fb9fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc004850 0x7fb9fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 -- 172.21.15.104:0/3080355024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0035b0 con 0x7fb9fc003bf0 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba11d9b700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9fc004850 0x7fb9fc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.260+0000 7fba1259c700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9fc003bf0 0x7fb9fc003fc0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fba04007f90 tx=0x7fb9fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db62c563f592ac6e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.261+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007a00 con 0x7fb9fc003bf0 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.261+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba040052d0 con 0x7fb9fc003bf0 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.261+0000 7fba1159a700 1 -- 172.21.15.104:0/3080355024 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0400c910 con 0x7fb9fc003bf0 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 -- 172.21.15.104:0/3080355024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9fc003bf0 msgr2=0x7fb9fc003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 --2- 172.21.15.104:0/3080355024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9fc003bf0 0x7fb9fc003fc0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fba04007f90 tx=0x7fb9fc005880 comp rx=0 tx=0).stop 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba12d9d700 1 -- 172.21.15.104:0/3080355024 reap_dead start 2023-06-10T18:58:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 -- 172.21.15.104:0/3080355024 shutdown_connections 2023-06-10T18:58:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 -- 172.21.15.104:0/3080355024 >> 172.21.15.104:0/3080355024 conn(0x7fba14089fd0 msgr2=0x7fba140d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 -- 172.21.15.104:0/3080355024 shutdown_connections 2023-06-10T18:58:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.262+0000 7fba189d0700 1 -- 172.21.15.104:0/3080355024 wait complete. 2023-06-10T18:58:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.264+0000 7fba189d0700 1 Processor -- start 2023-06-10T18:58:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.264+0000 7fba189d0700 1 -- start start 2023-06-10T18:58:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc003bf0 0x7fba140c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba141588e0 0x7fba140b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140c4c90 con 0x7fba140c42b0 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140ca360 con 0x7fba141588e0 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba140b44f0 con 0x7fb9fc003bf0 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40084/0 (socket says 172.21.15.104:40084) 2023-06-10T18:58:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 -- 172.21.15.104:0/1731161308 learned_addr learned my addr 172.21.15.104:0/1731161308 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba1259c700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc003bf0 0x7fba140c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba11d9b700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba141588e0 0x7fba140b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 -- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc003bf0 msgr2=0x7fba140c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc003bf0 0x7fba140c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 -- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba141588e0 msgr2=0x7fba140b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba141588e0 0x7fba140b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 -- 172.21.15.104:0/1731161308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140c9ac0 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba12d9d700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fba1406e120 tx=0x7fba0c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c00e040 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c00f040 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba140c9da0 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.265+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c00f830 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.266+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba140c6180 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.267+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba14074050 con 0x7fba140c42b0 2023-06-10T18:58:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.267+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba0c00f990 con 0x7fba140c42b0 2023-06-10T18:58:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.267+0000 7fba02ffd700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 0x7fb9f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.267+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fba0c089610 con 0x7fba140c42b0 2023-06-10T18:58:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.268+0000 7fba1259c700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 0x7fb9f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.268+0000 7fba1259c700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 0x7fb9f406d190 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fba04008fd0 tx=0x7fba04005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.272+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba0c0548e0 con 0x7fba140c42b0 2023-06-10T18:58:21.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.536+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction"]} v 0) v1 -- 0x7fba140b13c0 con 0x7fba140c42b0 2023-06-10T18:58:21.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.777+0000 7fba02ffd700 1 -- 172.21.15.104:0/1731161308 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock_eviction"]}]=0 v680) v1 ==== 154+0+0 (secure 0 0 0) 0x7fba0c057f00 con 0x7fba140c42b0 2023-06-10T18:58:21.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 msgr2=0x7fb9f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 0x7fb9f406d190 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fba04008fd0 tx=0x7fba04005000 comp rx=0 tx=0).stop 2023-06-10T18:58:21.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 msgr2=0x7fba140c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fba1406e120 tx=0x7fba0c011ed0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 shutdown_connections 2023-06-10T18:58:21.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba140c42b0 0x7fba140c4750 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9fc003bf0 0x7fba140c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb9f406ace0 0x7fb9f406d190 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 --2- 172.21.15.104:0/1731161308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba141588e0 0x7fba140b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 >> 172.21.15.104:0/1731161308 conn(0x7fba14089fd0 msgr2=0x7fba14072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:21.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 shutdown_connections 2023-06-10T18:58:21.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.779+0000 7fba189d0700 1 -- 172.21.15.104:0/1731161308 wait complete. 2023-06-10T18:58:21.798 INFO:tasks.cephfs_test_runner:test_filelock_eviction (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ok 2023-06-10T18:58:21.799 INFO:tasks.cephfs_test_runner:Starting test: test_mount_after_evicted_client (tasks.cephfs.test_client_recovery.TestClientRecovery) Test if a new mount of same fs works after client eviction. 2023-06-10T18:58:21.800 INFO:tasks.cephfs_test_runner:test_mount_after_evicted_client (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:58:21.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client' 2023-06-10T18:58:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.915+0000 7fc1a8d1b700 1 Processor -- start 2023-06-10T18:58:21.921 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.915+0000 7fc1a8d1b700 1 -- start start 2023-06-10T18:58:21.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a8d1b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1a415a2b0 0x7fc1a40ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.923 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a8d1b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a40cb860 0x7fc1a406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a8d1b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a415a680 con 0x7fc1a4069e70 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a8d1b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a40d0510 con 0x7fc1a415a2b0 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a8d1b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a406a5e0 con 0x7fc1a40cb860 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1a415a2b0 0x7fc1a40ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc1a415a2b0 0x7fc1a40ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40088/0 (socket says 172.21.15.104:40088) 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 -- 172.21.15.104:0/1016549220 learned_addr learned my addr 172.21.15.104:0/1016549220 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:21.924 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a2d9d700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a40cb860 0x7fc1a406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 -- 172.21.15.104:0/1016549220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a40cb860 msgr2=0x7fc1a406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a40cb860 0x7fc1a406d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 -- 172.21.15.104:0/1016549220 >> v1:172.21.15.114:6789/0 conn(0x7fc1a4069e70 legacy=0x7fc1a40cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 -- 172.21.15.104:0/1016549220 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a406a8c0 con 0x7fc1a415a2b0 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a259c700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40ac230 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fc194006fd0 tx=0x7fc194009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dde742f77fec99 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a159a700 1 -- 172.21.15.104:0/1016549220 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194005e30 con 0x7fc1a415a2b0 2023-06-10T18:58:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a159a700 1 -- 172.21.15.104:0/1016549220 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc194006030 con 0x7fc1a415a2b0 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.916+0000 7fc1a159a700 1 -- 172.21.15.104:0/1016549220 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940063c0 con 0x7fc1a415a2b0 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/1016549220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 msgr2=0x7fc1a40ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40ac230 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fc194006fd0 tx=0x7fc194009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/1016549220 shutdown_connections 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40ac230 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/1016549220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a40cb860 0x7fc1a406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/1016549220 >> 172.21.15.104:0/1016549220 conn(0x7fc1a406b8a0 msgr2=0x7fc1a40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/1016549220 shutdown_connections 2023-06-10T18:58:21.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.917+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/1016549220 wait complete. 2023-06-10T18:58:21.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.918+0000 7fc1a8d1b700 1 Processor -- start 2023-06-10T18:58:21.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.918+0000 7fc1a8d1b700 1 -- start start 2023-06-10T18:58:21.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a4069e70 0x7fc1a4152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1a40cb860 0x7fc1a40c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a415b8f0 con 0x7fc1a415a2b0 2023-06-10T18:58:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40c9350 con 0x7fc1a40cb860 2023-06-10T18:58:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a8d1b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b0b00 con 0x7fc1a4069e70 2023-06-10T18:58:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40098/0 (socket says 172.21.15.104:40098) 2023-06-10T18:58:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 -- 172.21.15.104:0/846106564 learned_addr learned my addr 172.21.15.104:0/846106564 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a259c700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a4069e70 0x7fc1a4152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a1d9b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1a40cb860 0x7fc1a40c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 -- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a4069e70 msgr2=0x7fc1a4152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a4069e70 0x7fc1a4152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 -- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1a40cb860 msgr2=0x7fc1a40c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:21.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1a40cb860 0x7fc1a40c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:21.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.919+0000 7fc1a2d9d700 1 -- 172.21.15.104:0/846106564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194009500 con 0x7fc1a415a2b0 2023-06-10T18:58:21.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc1a2d9d700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00cda0 tx=0x7fc19c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c003a20 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c003b80 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a40b0da0 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c004020 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.920+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a40c73c0 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.921+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a415ba60 con 0x7fc1a415a2b0 2023-06-10T18:58:21.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.921+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc19c004180 con 0x7fc1a415a2b0 2023-06-10T18:58:21.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.922+0000 7fc192ffd700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 0x7fc18406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:21.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.922+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc19c012070 con 0x7fc1a415a2b0 2023-06-10T18:58:21.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.922+0000 7fc1a259c700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 0x7fc18406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:21.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.922+0000 7fc1a259c700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 0x7fc18406d190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc194013040 tx=0x7fc194008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:21.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:21.926+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc1a415ba60 con 0x7fc1a415a2b0 2023-06-10T18:58:22.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.187+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client"]} v 0) v1 -- 0x7fc1a415ba30 con 0x7fc1a415a2b0 2023-06-10T18:58:22.784 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.778+0000 7fc192ffd700 1 -- 172.21.15.104:0/846106564 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client"]}]=0 v681) v1 ==== 166+0+0 (secure 0 0 0) 0x7fc19c054d00 con 0x7fc1a415a2b0 2023-06-10T18:58:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 msgr2=0x7fc18406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 0x7fc18406d190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc194013040 tx=0x7fc194008070 comp rx=0 tx=0).stop 2023-06-10T18:58:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 msgr2=0x7fc1a40c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00cda0 tx=0x7fc19c00a570 comp rx=0 tx=0).stop 2023-06-10T18:58:22.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 shutdown_connections 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1a415a2b0 0x7fc1a40c8e10 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1a4069e70 0x7fc1a4152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc18406ace0 0x7fc18406d190 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 --2- 172.21.15.104:0/846106564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1a40cb860 0x7fc1a40c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.780+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 >> 172.21.15.104:0/846106564 conn(0x7fc1a406b8a0 msgr2=0x7fc1a40ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.781+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 shutdown_connections 2023-06-10T18:58:22.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.781+0000 7fc1a8d1b700 1 -- 172.21.15.104:0/846106564 wait complete. 2023-06-10T18:58:22.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:58:22.925 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.919+0000 7fae38f5d700 1 Processor -- start 2023-06-10T18:58:22.926 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.919+0000 7fae38f5d700 1 -- start start 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae38f5d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae340c4a60 0x7fae3406d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae38f5d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae34159eb0 0x7fae340cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae38f5d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3406dbc0 con 0x7fae3406b320 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae38f5d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3406dea0 con 0x7fae34159eb0 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae38f5d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3406e180 con 0x7fae340c4a60 2023-06-10T18:58:22.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fae34159eb0 0x7fae340cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae3259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fae3406b320 0x7fae340c4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37930/0 (socket says 172.21.15.104:37930) 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae3259c700 1 -- 172.21.15.104:0/943842038 learned_addr learned my addr 172.21.15.104:0/943842038 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae31d9b700 1 --2- 172.21.15.104:0/943842038 >> v2:172.21.15.114:3300/0 conn(0x7fae340c4a60 0x7fae3406d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 -- 172.21.15.104:0/943842038 >> v2:172.21.15.114:3300/0 conn(0x7fae340c4a60 msgr2=0x7fae3406d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 --2- 172.21.15.104:0/943842038 >> v2:172.21.15.114:3300/0 conn(0x7fae340c4a60 0x7fae3406d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 -- 172.21.15.104:0/943842038 >> v1:172.21.15.104:6789/0 conn(0x7fae3406b320 legacy=0x7fae340c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.930 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 -- 172.21.15.104:0/943842038 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3406e460 con 0x7fae34159eb0 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae3159a700 1 -- 172.21.15.104:0/943842038 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728396937 0 0) 0x7fae3406dbc0 con 0x7fae3406b320 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.920+0000 7fae32d9d700 1 --2- 172.21.15.104:0/943842038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae34159eb0 0x7fae340cd510 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fae3406ba10 tx=0x7fae2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5051a84c00bae2f4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae3159a700 1 -- 172.21.15.104:0/943842038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c00e070 con 0x7fae34159eb0 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae3159a700 1 -- 172.21.15.104:0/943842038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c00bbd0 con 0x7fae34159eb0 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae3159a700 1 -- 172.21.15.104:0/943842038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c012430 con 0x7fae34159eb0 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 -- 172.21.15.104:0/943842038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae34159eb0 msgr2=0x7fae340cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 --2- 172.21.15.104:0/943842038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae34159eb0 0x7fae340cd510 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fae3406ba10 tx=0x7fae2c00b330 comp rx=0 tx=0).stop 2023-06-10T18:58:22.931 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 -- 172.21.15.104:0/943842038 shutdown_connections 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 --2- 172.21.15.104:0/943842038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae34159eb0 0x7fae340cd510 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 --2- 172.21.15.104:0/943842038 >> v2:172.21.15.114:3300/0 conn(0x7fae340c4a60 0x7fae3406d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 -- 172.21.15.104:0/943842038 >> 172.21.15.104:0/943842038 conn(0x7fae340cedc0 msgr2=0x7fae340cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 -- 172.21.15.104:0/943842038 shutdown_connections 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.921+0000 7fae38f5d700 1 -- 172.21.15.104:0/943842038 wait complete. 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 Processor -- start 2023-06-10T18:58:22.932 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 -- start start 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 0x7fae340c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae34159eb0 0x7fae3414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae340c6ea0 con 0x7fae340c4a60 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae340c7180 con 0x7fae3406b320 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae38f5d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae340c7460 con 0x7fae34159eb0 2023-06-10T18:58:22.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae31d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae31d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40110/0 (socket says 172.21.15.104:40110) 2023-06-10T18:58:22.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.923+0000 7fae31d9b700 1 -- 172.21.15.104:0/3630481885 learned_addr learned my addr 172.21.15.104:0/3630481885 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:22.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae32d9d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae34159eb0 0x7fae3414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae3259c700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 0x7fae340c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 -- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae34159eb0 msgr2=0x7fae3414c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae34159eb0 0x7fae3414c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 -- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 msgr2=0x7fae340c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:22.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 0x7fae340c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:22.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 -- 172.21.15.104:0/3630481885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c009a70 con 0x7fae340c4a60 2023-06-10T18:58:22.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae3259c700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 0x7fae340c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:22.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae31d9b700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e7e0 tx=0x7fae2800eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:22.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28013610 con 0x7fae340c4a60 2023-06-10T18:58:22.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae280137b0 con 0x7fae340c4a60 2023-06-10T18:58:22.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae340c7740 con 0x7fae340c4a60 2023-06-10T18:58:22.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28013b60 con 0x7fae340c4a60 2023-06-10T18:58:22.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.924+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae340c8b70 con 0x7fae340c4a60 2023-06-10T18:58:22.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.925+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae34074050 con 0x7fae340c4a60 2023-06-10T18:58:22.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.925+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fae28013cc0 con 0x7fae340c4a60 2023-06-10T18:58:22.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.926+0000 7fae22ffd700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 0x7fae1406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:22.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.926+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fae28088f90 con 0x7fae340c4a60 2023-06-10T18:58:22.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.926+0000 7fae3259c700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 0x7fae1406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:22.941 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.927+0000 7fae3259c700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 0x7fae1406d190 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fae24000f60 tx=0x7fae24007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:22.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:22.930+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fae28057cf0 con 0x7fae340c4a60 2023-06-10T18:58:23.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.119+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fae3406ce70 con 0x7fae340c4a60 2023-06-10T18:58:23.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.123+0000 7fae22ffd700 1 -- 172.21.15.104:0/3630481885 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fae280170c0 con 0x7fae340c4a60 2023-06-10T18:58:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 msgr2=0x7fae1406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 0x7fae1406d190 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fae24000f60 tx=0x7fae24007d30 comp rx=0 tx=0).stop 2023-06-10T18:58:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 msgr2=0x7fae3414c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e7e0 tx=0x7fae2800eba0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 shutdown_connections 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fae1406ace0 0x7fae1406d190 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae3406b320 0x7fae340c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae34159eb0 0x7fae3414c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 --2- 172.21.15.104:0/3630481885 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae340c4a60 0x7fae3414c2e0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 >> 172.21.15.104:0/3630481885 conn(0x7fae340cedc0 msgr2=0x7fae340cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 shutdown_connections 2023-06-10T18:58:23.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.125+0000 7fae38f5d700 1 -- 172.21.15.104:0/3630481885 wait complete. 2023-06-10T18:58:23.143 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:58:23.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:58:23.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:58:23.235 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:58:23.236 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:23.236 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:23.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:58:23.254 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:23.254 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:23.273 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:58:23.274 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:23.274 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:23.310 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-06-10T18:58:23.310 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:23.311 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:23.331 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:58:23.332 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:58:23.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:23.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.462+0000 7fac3e770700 1 Processor -- start 2023-06-10T18:58:23.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.462+0000 7fac3e770700 1 -- start start 2023-06-10T18:58:23.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3e770700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 0x7fac3806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3e770700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac3806e4b0 0x7fac380c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3e770700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3806d290 con 0x7fac380ca100 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3e770700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac38071640 con 0x7fac3806e4b0 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3e770700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380aef00 con 0x7fac3806e0e0 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac3806e4b0 0x7fac380c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3cd0d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fac380ca100 0x7fac380ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37944/0 (socket says 172.21.15.104:37944) 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fac3806e4b0 0x7fac380c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40120/0 (socket says 172.21.15.104:40120) 2023-06-10T18:58:23.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.463+0000 7fac3cd0d700 1 -- 172.21.15.104:0/792862702 learned_addr learned my addr 172.21.15.104:0/792862702 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac37fff700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 0x7fac3806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac377fe700 1 -- 172.21.15.104:0/792862702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 msgr2=0x7fac3806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac377fe700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 0x7fac3806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac377fe700 1 -- 172.21.15.104:0/792862702 >> v1:172.21.15.104:6789/0 conn(0x7fac380ca100 legacy=0x7fac380ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac377fe700 1 -- 172.21.15.104:0/792862702 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3806d4c0 con 0x7fac3806e4b0 2023-06-10T18:58:23.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac37fff700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 0x7fac3806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:23.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac377fe700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac3806e4b0 0x7fac380c9bc0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009840 tx=0x7fac2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0536aaf87b756e0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:23.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac36ffd700 1 -- 172.21.15.104:0/792862702 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c011830 con 0x7fac3806e4b0 2023-06-10T18:58:23.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac36ffd700 1 -- 172.21.15.104:0/792862702 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c011990 con 0x7fac3806e4b0 2023-06-10T18:58:23.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac36ffd700 1 -- 172.21.15.104:0/792862702 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c011d20 con 0x7fac3806e4b0 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 -- 172.21.15.104:0/792862702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac3806e4b0 msgr2=0x7fac380c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac3806e4b0 0x7fac380c9bc0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009840 tx=0x7fac2c009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 -- 172.21.15.104:0/792862702 shutdown_connections 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e0e0 0x7fac3806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 --2- 172.21.15.104:0/792862702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac3806e4b0 0x7fac380c9bc0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 -- 172.21.15.104:0/792862702 >> 172.21.15.104:0/792862702 conn(0x7fac3806b8a0 msgr2=0x7fac380cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:23.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.464+0000 7fac3e770700 1 -- 172.21.15.104:0/792862702 shutdown_connections 2023-06-10T18:58:23.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.465+0000 7fac3e770700 1 -- 172.21.15.104:0/792862702 wait complete. 2023-06-10T18:58:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.466+0000 7fac3e770700 1 Processor -- start 2023-06-10T18:58:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.466+0000 7fac3e770700 1 -- start start 2023-06-10T18:58:23.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.466+0000 7fac3e770700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 0x7fac380b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.466+0000 7fac3e770700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 0x7fac38159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac380b01c0 con 0x7fac380ca100 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac380b04a0 con 0x7fac3806e0e0 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380b0780 con 0x7fac3806e4b0 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40126/0 (socket says 172.21.15.104:40126) 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 -- 172.21.15.104:0/4058580837 learned_addr learned my addr 172.21.15.104:0/4058580837 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac377fe700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 0x7fac38159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac37fff700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 0x7fac380b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 -- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 msgr2=0x7fac38159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 0x7fac38159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 -- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 msgr2=0x7fac380b1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 0x7fac380b1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 -- 172.21.15.104:0/4058580837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0094a0 con 0x7fac380ca100 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac37fff700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 0x7fac380b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac377fe700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 0x7fac38159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3cd0d700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fac3000cbc0 tx=0x7fac3000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3000ab50 con 0x7fac380ca100 2023-06-10T18:58:23.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3000acb0 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac380b0a60 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30012600 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.467+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac380b1020 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.468+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac380b1210 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.469+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac3001b3b0 con 0x7fac380ca100 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.469+0000 7fac34ff9700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 0x7fac2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:23.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.469+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fac30088f80 con 0x7fac380ca100 2023-06-10T18:58:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.469+0000 7fac37fff700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 0x7fac2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.470+0000 7fac37fff700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 0x7fac2406d190 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fac28006fd0 tx=0x7fac28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:23.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.473+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac300bb310 con 0x7fac380ca100 2023-06-10T18:58:23.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.703+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac38074050 con 0x7fac380ca100 2023-06-10T18:58:23.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.703+0000 7fac34ff9700 1 -- 172.21.15.104:0/4058580837 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3544 (secure 0 0 0) 0x7fac30017090 con 0x7fac380ca100 2023-06-10T18:58:23.710 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:23.710 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7415,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1389491471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1389491471},{"type":"v1","addr":"172.21.15.114:6836","nonce":1389491471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1343725493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":1343725493},{"type":"v1","addr":"172.21.15.104:6835","nonce":1343725493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":8590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4203474708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4203474708},{"type":"v1","addr":"172.21.15.114:6837","nonce":4203474708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":10530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1769428145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1769428145},{"type":"v1","addr":"172.21.15.104:6834","nonce":1769428145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":104}],"filesystems":[]} 2023-06-10T18:58:23.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 msgr2=0x7fac2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 0x7fac2406d190 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fac28006fd0 tx=0x7fac28008040 comp rx=0 tx=0).stop 2023-06-10T18:58:23.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 msgr2=0x7fac38099460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:23.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fac3000cbc0 tx=0x7fac3000cf80 comp rx=0 tx=0).stop 2023-06-10T18:58:23.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 shutdown_connections 2023-06-10T18:58:23.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fac2406ace0 0x7fac2406d190 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac3806e0e0 0x7fac380b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fac380ca100 0x7fac38099460 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 --2- 172.21.15.104:0/4058580837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3806e4b0 0x7fac38159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 >> 172.21.15.104:0/4058580837 conn(0x7fac3806b8a0 msgr2=0x7fac380cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 shutdown_connections 2023-06-10T18:58:23.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.706+0000 7fac3e770700 1 -- 172.21.15.104:0/4058580837 wait complete. 2023-06-10T18:58:23.715 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 104 2023-06-10T18:58:23.724 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:58:23.725 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:58:23.725 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:58:23.725 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:58:23.726 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:58:23.726 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:58:23.726 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:58:23.726 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:58:23.727 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:58:23.727 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:58:23.727 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:58:23.727 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:58:23.728 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:58:23.728 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:58:23.728 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:58:23.728 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:58:23.734 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:58:23.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:58:23.735 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:58:23.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:58:23.738 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:58:23.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:58:23.739 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:58:23.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:58:23.781 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:58:23.781 DEBUG:teuthology.parallel:result is None 2023-06-10T18:58:23.782 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:58:23.783 DEBUG:teuthology.parallel:result is None 2023-06-10T18:58:23.784 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:58:23.784 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:58:23.784 DEBUG:teuthology.parallel:result is None 2023-06-10T18:58:23.785 DEBUG:teuthology.parallel:result is None 2023-06-10T18:58:23.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:58:23.860 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:58:24.177 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:58:24.179 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:58:24.180 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.908+0000 7f5d5d23d700 1 Processor -- start 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.908+0000 7f5d5d23d700 1 -- start start 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5d23d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 0x7f5d580701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5d23d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5d580cad90 0x7f5d580b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5d23d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d580ccca0 con 0x7f5d5815a250 2023-06-10T18:58:24.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5d23d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d580cee50 con 0x7f5d580cad90 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5d23d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580cf130 con 0x7f5d580cfa00 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d56d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 0x7f5d580701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5759e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5d580cad90 0x7f5d580b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5659c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5d5815a250 0x7f5d580cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:43828/0 (socket says 172.21.15.104:43828) 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.909+0000 7f5d5659c700 1 -- 172.21.15.104:0/1863815588 learned_addr learned my addr 172.21.15.104:0/1863815588 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5759e700 1 -- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 msgr2=0x7f5d580701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5759e700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 0x7f5d580701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5759e700 1 -- 172.21.15.104:0/1863815588 >> v1:172.21.15.114:6789/0 conn(0x7f5d5815a250 legacy=0x7f5d580cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5759e700 1 -- 172.21.15.104:0/1863815588 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d580cc930 con 0x7f5d580cad90 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d55d9b700 1 -- 172.21.15.104:0/1863815588 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23216521 0 0) 0x7f5d580ccca0 con 0x7f5d5815a250 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d56d9d700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 0x7f5d580701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5759e700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cad90 0x7f5d580b3590 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009840 tx=0x7f5d48009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8abec625bac8d2f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d55d9b700 1 -- 172.21.15.104:0/1863815588 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48011830 con 0x7f5d580cad90 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d55d9b700 1 -- 172.21.15.104:0/1863815588 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d48011990 con 0x7f5d580cad90 2023-06-10T18:58:24.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d55d9b700 1 -- 172.21.15.104:0/1863815588 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48011d20 con 0x7f5d580cad90 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5d23d700 1 -- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cad90 msgr2=0x7f5d580b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.910+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cad90 0x7f5d580b3590 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009840 tx=0x7f5d48009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 -- 172.21.15.104:0/1863815588 shutdown_connections 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cad90 0x7f5d580b3590 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/1863815588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cfa00 0x7f5d580701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 -- 172.21.15.104:0/1863815588 >> 172.21.15.104:0/1863815588 conn(0x7f5d5806e370 msgr2=0x7f5d5806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 -- 172.21.15.104:0/1863815588 shutdown_connections 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.911+0000 7f5d5d23d700 1 -- 172.21.15.104:0/1863815588 wait complete. 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.912+0000 7f5d5d23d700 1 Processor -- start 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.912+0000 7f5d5d23d700 1 -- start start 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cad90 0x7f5d5814bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cfa00 0x7f5d580c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580c5140 con 0x7f5d5815a250 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580c4270 con 0x7f5d580cfa00 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5d23d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5814c1d0 con 0x7f5d580cad90 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40164/0 (socket says 172.21.15.104:40164) 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 -- 172.21.15.104:0/2945605959 learned_addr learned my addr 172.21.15.104:0/2945605959 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:24.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5659c700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cfa00 0x7f5d580c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d56d9d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cad90 0x7f5d5814bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 -- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cad90 msgr2=0x7f5d5814bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cad90 0x7f5d5814bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 -- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cfa00 msgr2=0x7f5d580c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cfa00 0x7f5d580c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 -- 172.21.15.104:0/2945605959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50009a70 con 0x7f5d5815a250 2023-06-10T18:58:24.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.913+0000 7f5d5759e700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009fd0 tx=0x7f5d48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:24.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.914+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d480038a0 con 0x7f5d5815a250 2023-06-10T18:58:24.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.914+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4801f070 con 0x7f5d5815a250 2023-06-10T18:58:24.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.914+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d480094a0 con 0x7f5d5815a250 2023-06-10T18:58:24.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.914+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48003c60 con 0x7f5d5815a250 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.914+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d580c47a0 con 0x7f5d5815a250 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.915+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d580c5280 con 0x7f5d5815a250 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.915+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d4801a430 con 0x7f5d5815a250 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.915+0000 7f5d4f7fe700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 0x7f5d3806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.916+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f5d48021070 con 0x7f5d5815a250 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.916+0000 7f5d56d9d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 0x7f5d3806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.916+0000 7f5d56d9d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 0x7f5d3806d1c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007600 tx=0x7f5d5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:24.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:23.920+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d580c5280 con 0x7f5d5815a250 2023-06-10T18:58:24.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.118+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5d580c5280 con 0x7f5d5815a250 2023-06-10T18:58:24.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.896+0000 7f5d4f7fe700 1 -- 172.21.15.104:0/2945605959 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v168) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5d580c5280 con 0x7f5d5815a250 2023-06-10T18:58:24.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 msgr2=0x7f5d3806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 0x7f5d3806d1c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007600 tx=0x7f5d5000b040 comp rx=0 tx=0).stop 2023-06-10T18:58:24.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 msgr2=0x7f5d5815cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009fd0 tx=0x7f5d48004750 comp rx=0 tx=0).stop 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 shutdown_connections 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d5815a250 0x7f5d5815cb80 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d580cad90 0x7f5d5814bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5d3806ad10 0x7f5d3806d1c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 --2- 172.21.15.104:0/2945605959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d580cfa00 0x7f5d580c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:24.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 >> 172.21.15.104:0/2945605959 conn(0x7f5d5806e370 msgr2=0x7f5d581e3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:24.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 shutdown_connections 2023-06-10T18:58:24.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:24.899+0000 7f5d5d23d700 1 -- 172.21.15.104:0/2945605959 wait complete. 2023-06-10T18:58:24.908 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:58:24.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:58:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.037+0000 7f23ec3ac700 1 Processor -- start 2023-06-10T18:58:25.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 -- start start 2023-06-10T18:58:25.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 0x7f23e406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e406dcb0 con 0x7f23e415c500 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e406e850 con 0x7f23e4069e70 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.038+0000 7f23ec3ac700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e406a830 con 0x7f23e40cf030 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23ea949700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 0x7f23e406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:59558/0 (socket says 172.21.15.104:59558) 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 -- 172.21.15.104:0/1255769796 learned_addr learned my addr 172.21.15.104:0/1255769796 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 -- 172.21.15.104:0/1255769796 >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 msgr2=0x7f23e406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:25.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 --2- 172.21.15.104:0/1255769796 >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 0x7f23e406a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 -- 172.21.15.104:0/1255769796 >> v1:172.21.15.114:6789/0 conn(0x7f23e415c500 legacy=0x7f23e406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 -- 172.21.15.104:0/1255769796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e40cf460 con 0x7f23e40cf030 2023-06-10T18:58:25.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23ea949700 1 --2- 172.21.15.104:0/1255769796 >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 0x7f23e406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:25.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9947700 1 --2- 172.21.15.104:0/1255769796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f23e00098f0 tx=0x7f23e0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc14dd9ecb73df33 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.039+0000 7f23e9146700 1 -- 172.21.15.104:0/1255769796 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0011830 con 0x7f23e40cf030 2023-06-10T18:58:25.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23e9146700 1 -- 172.21.15.104:0/1255769796 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e0011990 con 0x7f23e40cf030 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23e9146700 1 -- 172.21.15.104:0/1255769796 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0011d20 con 0x7f23e40cf030 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 -- 172.21.15.104:0/1255769796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 msgr2=0x7f23e40ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/1255769796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f23e00098f0 tx=0x7f23e0009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 -- 172.21.15.104:0/1255769796 shutdown_connections 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/1255769796 >> v2:172.21.15.114:3300/0 conn(0x7f23e4069e70 0x7f23e406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/1255769796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e40cf030 0x7f23e40ce040 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 -- 172.21.15.104:0/1255769796 >> 172.21.15.104:0/1255769796 conn(0x7f23e406b8a0 msgr2=0x7f23e40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:25.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 -- 172.21.15.104:0/1255769796 shutdown_connections 2023-06-10T18:58:25.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.040+0000 7f23ec3ac700 1 -- 172.21.15.104:0/1255769796 wait complete. 2023-06-10T18:58:25.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.042+0000 7f23ec3ac700 1 Processor -- start 2023-06-10T18:58:25.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.042+0000 7f23ec3ac700 1 -- start start 2023-06-10T18:58:25.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 0x7f23e4146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 0x7f23e409a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4147670 con 0x7f23e40cf030 2023-06-10T18:58:25.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e41478c0 con 0x7f23e4069e70 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4147ba0 con 0x7f23e415c500 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40180/0 (socket says 172.21.15.104:40180) 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 -- 172.21.15.104:0/2094422152 learned_addr learned my addr 172.21.15.104:0/2094422152 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ea148700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 0x7f23e4146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ea949700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 0x7f23e409a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 -- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 msgr2=0x7f23e409a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 0x7f23e409a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 -- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 msgr2=0x7f23e4146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 0x7f23e4146fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 -- 172.21.15.104:0/2094422152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc009a70 con 0x7f23e40cf030 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ea148700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 0x7f23e4146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ea949700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 0x7f23e409a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23e9947700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009c40 tx=0x7f23e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00036a0 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e001f070 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e0009550 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0003a40 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.043+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e4148160 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.044+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e4074050 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.045+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f23e0003ba0 con 0x7f23e40cf030 2023-06-10T18:58:25.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.045+0000 7f23daffd700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 0x7f23c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.045+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f23e0021070 con 0x7f23e40cf030 2023-06-10T18:58:25.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.045+0000 7f23ea148700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 0x7f23c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.046+0000 7f23ea148700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 0x7f23c806d190 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f23e409a820 tx=0x7f23d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.050+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f23e0055f30 con 0x7f23e40cf030 2023-06-10T18:58:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.311+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f23e406d930 con 0x7f23e40cf030 2023-06-10T18:58:25.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.312+0000 7f23daffd700 1 -- 172.21.15.104:0/2094422152 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f23e0015090 con 0x7f23e40cf030 2023-06-10T18:58:25.319 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:25.319 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:25.319 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:58:25.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:58:25.320 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.321 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:58:25.322 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:58:25.324 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.325 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:58:25.326 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:58:25.327 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.331 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:58:25.332 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.333 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:58:25.335 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:58:25.336 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:58:25.337 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.338 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:58:25.340 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.341 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:58:25.342 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:58:25.343 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 msgr2=0x7f23c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 0x7f23c806d190 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f23e409a820 tx=0x7f23d4009040 comp rx=0 tx=0).stop 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 msgr2=0x7f23e40c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009c40 tx=0x7f23e0004750 comp rx=0 tx=0).stop 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 shutdown_connections 2023-06-10T18:58:25.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f23c806ace0 0x7f23c806d190 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23e4069e70 0x7f23e4146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23e415c500 0x7f23e409a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 --2- 172.21.15.104:0/2094422152 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23e40cf030 0x7f23e40c68e0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 >> 172.21.15.104:0/2094422152 conn(0x7f23e406b8a0 msgr2=0x7f23e40b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 shutdown_connections 2023-06-10T18:58:25.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.314+0000 7f23ec3ac700 1 -- 172.21.15.104:0/2094422152 wait complete. 2023-06-10T18:58:25.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:25.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.460+0000 7fc2e1d0f700 1 Processor -- start 2023-06-10T18:58:25.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 -- start start 2023-06-10T18:58:25.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2dc1567f0 0x7fc2dc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc0701a0 0x7fc2dc0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc070b20 con 0x7fc2dc15c7a0 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2dc0cb2d0 con 0x7fc2dc1567f0 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2e1d0f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc15cca0 con 0x7fc2dc0701a0 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2db7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2dc1567f0 0x7fc2dc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2db7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2dc1567f0 0x7fc2dc0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40186/0 (socket says 172.21.15.104:40186) 2023-06-10T18:58:25.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.461+0000 7fc2db7fe700 1 -- 172.21.15.104:0/4134777206 learned_addr learned my addr 172.21.15.104:0/4134777206 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2dbfff700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc0701a0 0x7fc2dc0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2db7fe700 1 -- 172.21.15.104:0/4134777206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc0701a0 msgr2=0x7fc2dc0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2db7fe700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc0701a0 0x7fc2dc0705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2db7fe700 1 -- 172.21.15.104:0/4134777206 >> v1:172.21.15.114:6789/0 conn(0x7fc2dc15c7a0 legacy=0x7fc2dc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2db7fe700 1 -- 172.21.15.104:0/4134777206 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc15cf80 con 0x7fc2dc1567f0 2023-06-10T18:58:25.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2db7fe700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0cad90 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0098f0 tx=0x7fc2cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6796e8512a824cea server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2da7fc700 1 -- 172.21.15.104:0/4134777206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc011830 con 0x7fc2dc1567f0 2023-06-10T18:58:25.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2da7fc700 1 -- 172.21.15.104:0/4134777206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc011990 con 0x7fc2dc1567f0 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.462+0000 7fc2da7fc700 1 -- 172.21.15.104:0/4134777206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc011d20 con 0x7fc2dc1567f0 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/4134777206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc1567f0 msgr2=0x7fc2dc0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0cad90 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0098f0 tx=0x7fc2cc009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/4134777206 shutdown_connections 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0cad90 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/4134777206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc0701a0 0x7fc2dc0705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/4134777206 >> 172.21.15.104:0/4134777206 conn(0x7fc2dc06e370 msgr2=0x7fc2dc06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:25.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/4134777206 shutdown_connections 2023-06-10T18:58:25.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.463+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/4134777206 wait complete. 2023-06-10T18:58:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.464+0000 7fc2e1d0f700 1 Processor -- start 2023-06-10T18:58:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.464+0000 7fc2e1d0f700 1 -- start start 2023-06-10T18:58:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc15c7a0 0x7fc2dc0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1f1c30 con 0x7fc2dc0701a0 2023-06-10T18:58:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc14e4f0 con 0x7fc2dc1567f0 2023-06-10T18:58:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc0cc420 con 0x7fc2dc15c7a0 2023-06-10T18:58:25.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40192/0 (socket says 172.21.15.104:40192) 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 -- 172.21.15.104:0/1358607100 learned_addr learned my addr 172.21.15.104:0/1358607100 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2daffd700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 -- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc15c7a0 msgr2=0x7fc2dc0afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc15c7a0 0x7fc2dc0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 -- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2dc1567f0 msgr2=0x7fc2dc0b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 -- 172.21.15.104:0/1358607100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc009550 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2db7fe700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000c00 tx=0x7fc2cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc004570 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc01f070 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc0b1ad0 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0038e0 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.465+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc0b2090 con 0x7fc2dc0701a0 2023-06-10T18:58:25.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.466+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc074050 con 0x7fc2dc0701a0 2023-06-10T18:58:25.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.467+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc2cc021030 con 0x7fc2dc0701a0 2023-06-10T18:58:25.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.467+0000 7fc2d3fff700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 0x7fc2bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.467+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc2cc08a710 con 0x7fc2dc0701a0 2023-06-10T18:58:25.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.467+0000 7fc2daffd700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 0x7fc2bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.468+0000 7fc2daffd700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 0x7fc2bc06d190 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc06f1a0 tx=0x7fc2c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.472+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc2cc0559e0 con 0x7fc2dc0701a0 2023-06-10T18:58:25.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.661+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2dc0b66c0 con 0x7fc2dc0701a0 2023-06-10T18:58:25.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.662+0000 7fc2d3fff700 1 -- 172.21.15.104:0/1358607100 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+14902 (secure 0 0 0) 0x7fc2cc059000 con 0x7fc2dc0701a0 2023-06-10T18:58:25.669 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:25.669 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":168,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:24.893689+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:25.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 msgr2=0x7fc2bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 0x7fc2bc06d190 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc06f1a0 tx=0x7fc2c4009040 comp rx=0 tx=0).stop 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 msgr2=0x7fc2dc1f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000c00 tx=0x7fc2cc004750 comp rx=0 tx=0).stop 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 shutdown_connections 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2dc15c7a0 0x7fc2dc0afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2dc0701a0 0x7fc2dc1f16f0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2bc06ace0 0x7fc2bc06d190 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 --2- 172.21.15.104:0/1358607100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2dc1567f0 0x7fc2dc0b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 >> 172.21.15.104:0/1358607100 conn(0x7fc2dc06e370 msgr2=0x7fc2dc15a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:25.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 shutdown_connections 2023-06-10T18:58:25.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.664+0000 7fc2e1d0f700 1 -- 172.21.15.104:0/1358607100 wait complete. 2023-06-10T18:58:25.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:25.684 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:25.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.801+0000 7ff31888d700 1 Processor -- start 2023-06-10T18:58:25.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.801+0000 7ff31888d700 1 -- start start 2023-06-10T18:58:25.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31888d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff31415a0a0 0x7ff31415a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31888d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140cc050 con 0x7ff31406e3d0 2023-06-10T18:58:25.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31888d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3140ac9d0 con 0x7ff31406d350 2023-06-10T18:58:25.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31888d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31406d0e0 con 0x7ff31415a0a0 2023-06-10T18:58:25.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff31415a0a0 0x7ff31415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff31415a0a0 0x7ff31415a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40204/0 (socket says 172.21.15.104:40204) 2023-06-10T18:58:25.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff312d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff31406e3d0 0x7ff314069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38028/0 (socket says 172.21.15.104:38028) 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 -- 172.21.15.104:0/1547090775 learned_addr learned my addr 172.21.15.104:0/1547090775 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 -- 172.21.15.104:0/1547090775 >> v1:172.21.15.114:6789/0 conn(0x7ff31406d350 legacy=0x7ff3140ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 -- 172.21.15.104:0/1547090775 >> v1:172.21.15.104:6789/0 conn(0x7ff31406e3d0 legacy=0x7ff314069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31159a700 1 -- 172.21.15.104:0/1547090775 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812662727 0 0) 0x7ff3140cc050 con 0x7ff31406e3d0 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 -- 172.21.15.104:0/1547090775 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31406ce00 con 0x7ff31415a0a0 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31259c700 1 --2- 172.21.15.104:0/1547090775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31415a0a0 0x7ff31415a470 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff304006fd0 tx=0x7ff304009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb3a0a77724f5ab server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.802+0000 7ff31159a700 1 -- 172.21.15.104:0/1547090775 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304005e30 con 0x7ff31415a0a0 2023-06-10T18:58:25.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31159a700 1 -- 172.21.15.104:0/1547090775 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304004030 con 0x7ff31415a0a0 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31159a700 1 -- 172.21.15.104:0/1547090775 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3040043e0 con 0x7ff31415a0a0 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 -- 172.21.15.104:0/1547090775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31415a0a0 msgr2=0x7ff31415a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 --2- 172.21.15.104:0/1547090775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31415a0a0 0x7ff31415a470 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff304006fd0 tx=0x7ff304009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 -- 172.21.15.104:0/1547090775 shutdown_connections 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 --2- 172.21.15.104:0/1547090775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31415a0a0 0x7ff31415a470 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 -- 172.21.15.104:0/1547090775 >> 172.21.15.104:0/1547090775 conn(0x7ff31406b8a0 msgr2=0x7ff3140cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:25.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 -- 172.21.15.104:0/1547090775 shutdown_connections 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.803+0000 7ff31888d700 1 -- 172.21.15.104:0/1547090775 wait complete. 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.804+0000 7ff31888d700 1 Processor -- start 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.804+0000 7ff31888d700 1 -- start start 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff31406e3d0 0x7ff3140c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 0x7ff3140c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140b1d20 con 0x7ff31406d350 2023-06-10T18:58:25.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140b1f10 con 0x7ff31406e3d0 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31888d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3141f07d0 con 0x7ff31415a0a0 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40206/0 (socket says 172.21.15.104:40206) 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.805+0000 7ff31259c700 1 -- 172.21.15.104:0/519739778 learned_addr learned my addr 172.21.15.104:0/519739778 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff311d9b700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff31406e3d0 0x7ff3140c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff312d9d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 0x7ff3140c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 -- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 msgr2=0x7ff3140c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 0x7ff3140c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 -- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff31406e3d0 msgr2=0x7ff3140c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff31406e3d0 0x7ff3140c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 -- 172.21.15.104:0/519739778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff304009500 con 0x7ff31406d350 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff312d9d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 0x7ff3140c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31259c700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7ff304000c00 tx=0x7ff304004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304011620 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304011780 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3141f0c60 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304011b30 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.806+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3141f0180 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.807+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff314074050 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.807+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff304011c90 con 0x7ff31406d350 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.808+0000 7ff302ffd700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 0x7ff2f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:25.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.808+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff30408a570 con 0x7ff31406d350 2023-06-10T18:58:25.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.808+0000 7ff311d9b700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 0x7ff2f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:25.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.809+0000 7ff311d9b700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 0x7ff2f406d190 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff3141f0e70 tx=0x7ff308009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:25.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.812+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3040558c0 con 0x7ff31406d350 2023-06-10T18:58:26.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.998+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3140714d0 con 0x7ff31406d350 2023-06-10T18:58:26.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:25.999+0000 7ff302ffd700 1 -- 172.21.15.104:0/519739778 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff304058ee0 con 0x7ff31406d350 2023-06-10T18:58:26.005 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 msgr2=0x7ff2f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 0x7ff2f406d190 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff3141f0e70 tx=0x7ff308009040 comp rx=0 tx=0).stop 2023-06-10T18:58:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 msgr2=0x7ff3140b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7ff304000c00 tx=0x7ff304004750 comp rx=0 tx=0).stop 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 shutdown_connections 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff31415a0a0 0x7ff3140c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff31406d350 0x7ff3140b17e0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff2f406ace0 0x7ff2f406d190 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 --2- 172.21.15.104:0/519739778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff31406e3d0 0x7ff3140c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 >> 172.21.15.104:0/519739778 conn(0x7ff31406b8a0 msgr2=0x7ff314157890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 shutdown_connections 2023-06-10T18:58:26.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.001+0000 7ff31888d700 1 -- 172.21.15.104:0/519739778 wait complete. 2023-06-10T18:58:26.019 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:26.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.136+0000 7f5d0d971700 1 Processor -- start 2023-06-10T18:58:26.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.136+0000 7f5d0d971700 1 -- start start 2023-06-10T18:58:26.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d0d971700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5d080d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d0d971700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080ce340 con 0x7f5d080ccff0 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d0d971700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080d00d0 con 0x7f5d080ce7d0 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d0d971700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080cd460 con 0x7f5d0806af80 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d067fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5d080ccff0 0x7f5d0806b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38042/0 (socket says 172.21.15.104:38042) 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d067fc700 1 -- 172.21.15.104:0/1554680507 learned_addr learned my addr 172.21.15.104:0/1554680507 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d06ffd700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5d080d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091213484 0 0) 0x7f5d080ce340 con 0x7f5d080ccff0 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.137+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf00035b0 con 0x7f5d080ccff0 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882792222 0 0) 0x7f5d080d00d0 con 0x7f5d080ce7d0 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d080ce340 con 0x7f5d080ce7d0 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9339930 0 0) 0x7f5cf00035b0 con 0x7f5d080ccff0 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 msgr2=0x7f5d080d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5d080d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 >> v1:172.21.15.114:6789/0 conn(0x7f5d080ce7d0 legacy=0x7f5d080cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080cd680 con 0x7f5d080ccff0 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5cf8001670 con 0x7f5d080ccff0 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 >> v1:172.21.15.104:6789/0 conn(0x7f5d080ccff0 legacy=0x7f5d0806b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5cf0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5cf0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cf0004d90 0x7f5cf0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080cd460 con 0x7f5cf0004170 2023-06-10T18:58:26.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf000bb20 con 0x7f5cf0004d90 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf000bce0 con 0x7f5d0806af80 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5cf0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d06ffd700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5cf0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d077fe700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cf0004d90 0x7f5cf0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 -- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 msgr2=0x7f5cf0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d0806af80 0x7f5cf0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 -- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cf0004d90 msgr2=0x7f5cf0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cf0004d90 0x7f5cf0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 -- 172.21.15.104:0/1554680507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf00035b0 con 0x7f5cf0004170 2023-06-10T18:58:26.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5cf0004850 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c110 tx=0x7f5cf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6734a88bc7a6ef1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800cc50 con 0x7f5cf0004170 2023-06-10T18:58:26.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.138+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf800cdb0 con 0x7f5cf0004170 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d05ffb700 1 -- 172.21.15.104:0/1554680507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8005ba0 con 0x7f5cf0004170 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 -- 172.21.15.104:0/1554680507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 msgr2=0x7f5cf0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1554680507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5cf0004850 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c110 tx=0x7f5cf0005880 comp rx=0 tx=0).stop 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d077fe700 1 -- 172.21.15.104:0/1554680507 reap_dead start 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 -- 172.21.15.104:0/1554680507 shutdown_connections 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 -- 172.21.15.104:0/1554680507 >> 172.21.15.104:0/1554680507 conn(0x7f5d08089fd0 msgr2=0x7f5d0808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 -- 172.21.15.104:0/1554680507 shutdown_connections 2023-06-10T18:58:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.139+0000 7f5d0d971700 1 -- 172.21.15.104:0/1554680507 wait complete. 2023-06-10T18:58:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.140+0000 7f5d0d971700 1 Processor -- start 2023-06-10T18:58:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 -- start start 2023-06-10T18:58:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 0x7f5d08153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d080b16a0 0x7f5d080b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080c5130 con 0x7f5cf0004170 2023-06-10T18:58:26.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d081585f0 con 0x7f5d081531a0 2023-06-10T18:58:26.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d0d971700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080c42a0 con 0x7f5d080b16a0 2023-06-10T18:58:26.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d06ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d06ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40224/0 (socket says 172.21.15.104:40224) 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.141+0000 7f5d06ffd700 1 -- 172.21.15.104:0/1796107907 learned_addr learned my addr 172.21.15.104:0/1796107907 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 0x7f5d08153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d077fe700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d080b16a0 0x7f5d080b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 -- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d080b16a0 msgr2=0x7f5d080b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d080b16a0 0x7f5d080b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 -- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 msgr2=0x7f5d08153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 0x7f5d08153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 -- 172.21.15.104:0/1796107907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080b2030 con 0x7f5cf0004170 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 0x7f5d08153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d06ffd700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc011aa0 tx=0x7f5cfc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00f980 con 0x7f5cf0004170 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc00fae0 con 0x7f5cf0004170 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080c44b0 con 0x7f5cf0004170 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00fe90 con 0x7f5cf0004170 2023-06-10T18:58:26.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.142+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d080c4a70 con 0x7f5cf0004170 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.143+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08074050 con 0x7f5cf0004170 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.143+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5cfc014440 con 0x7f5cf0004170 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.144+0000 7f5ceffff700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 0x7f5cf406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.144+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f5cfc089910 con 0x7f5cf0004170 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.144+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 0x7f5cf406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.144+0000 7f5d067fc700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 0x7f5cf406d190 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8015040 tx=0x7f5cf8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.148+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5cfc054d20 con 0x7f5cf0004170 2023-06-10T18:58:26.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.330+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5d080c70b0 con 0x7f5cf0004170 2023-06-10T18:58:26.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.331+0000 7f5ceffff700 1 -- 172.21.15.104:0/1796107907 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v168) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5cfc058340 con 0x7f5cf0004170 2023-06-10T18:58:26.338 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 msgr2=0x7f5cf406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 0x7f5cf406d190 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8015040 tx=0x7f5cf8007000 comp rx=0 tx=0).stop 2023-06-10T18:58:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 msgr2=0x7f5d08155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc011aa0 tx=0x7f5cfc011e60 comp rx=0 tx=0).stop 2023-06-10T18:58:26.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 shutdown_connections 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5d080b16a0 0x7f5d080b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5cf0004170 0x7f5d08155940 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5cf406ace0 0x7f5cf406d190 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 --2- 172.21.15.104:0/1796107907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d081531a0 0x7f5d08153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 >> 172.21.15.104:0/1796107907 conn(0x7f5d08089fd0 msgr2=0x7f5d080b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 shutdown_connections 2023-06-10T18:58:26.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.333+0000 7f5d0d971700 1 -- 172.21.15.104:0/1796107907 wait complete. 2023-06-10T18:58:26.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:58:26.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.465+0000 7f2056d9f700 1 Processor -- start 2023-06-10T18:58:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.465+0000 7f2056d9f700 1 -- start start 2023-06-10T18:58:26.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f2056d9f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f20500701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f2056d9f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f20500cad90 0x7f20500b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f2056d9f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500ccca0 con 0x7f205015a250 2023-06-10T18:58:26.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f2056d9f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20500cee50 con 0x7f20500cad90 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f2056d9f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500cf130 con 0x7f20500cfa00 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f204ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f205015a250 0x7f20500cfdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38060/0 (socket says 172.21.15.104:38060) 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.466+0000 7f204ffff700 1 -- 172.21.15.104:0/1036474207 learned_addr learned my addr 172.21.15.104:0/1036474207 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f2054b3b700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f20500701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f205533c700 1 --2- 172.21.15.104:0/1036474207 >> v2:172.21.15.114:3300/0 conn(0x7f20500cad90 0x7f20500b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764993756 0 0) 0x7f20500ccca0 con 0x7f205015a250 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20400035b0 con 0x7f205015a250 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4193218688 0 0) 0x7f20400035b0 con 0x7f205015a250 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 msgr2=0x7f20500701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f20500701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 >> v2:172.21.15.114:3300/0 conn(0x7f20500cad90 msgr2=0x7f20500b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 --2- 172.21.15.104:0/1036474207 >> v2:172.21.15.114:3300/0 conn(0x7f20500cad90 0x7f20500b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500cc930 con 0x7f205015a250 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f203c001670 con 0x7f205015a250 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 >> v1:172.21.15.104:6789/0 conn(0x7f205015a250 legacy=0x7f20500cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003a00 0x7f2040003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20400048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f2040005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500cee50 con 0x7f2040004430 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500cf130 con 0x7f2040003a00 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204000ba10 con 0x7f20500cfa00 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20400048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f2054b3b700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003a00 0x7f2040003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f205533c700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f2040005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 -- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 msgr2=0x7f2040005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500cfa00 0x7f2040005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 -- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003a00 msgr2=0x7f2040003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003a00 0x7f2040003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 -- 172.21.15.104:0/1036474207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400035b0 con 0x7f2040004430 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.467+0000 7f204ffff700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20400048f0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c320 tx=0x7f2040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd5a85488e89711 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c00ce20 con 0x7f2040004430 2023-06-10T18:58:26.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c005960 con 0x7f2040004430 2023-06-10T18:58:26.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f204f7fe700 1 -- 172.21.15.104:0/1036474207 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c005d10 con 0x7f2040004430 2023-06-10T18:58:26.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 -- 172.21.15.104:0/1036474207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 msgr2=0x7f20400048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1036474207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20400048f0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c320 tx=0x7f2040005880 comp rx=0 tx=0).stop 2023-06-10T18:58:26.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f205533c700 1 -- 172.21.15.104:0/1036474207 reap_dead start 2023-06-10T18:58:26.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 -- 172.21.15.104:0/1036474207 shutdown_connections 2023-06-10T18:58:26.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 -- 172.21.15.104:0/1036474207 >> 172.21.15.104:0/1036474207 conn(0x7f205006e370 msgr2=0x7f205006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 -- 172.21.15.104:0/1036474207 shutdown_connections 2023-06-10T18:58:26.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.468+0000 7f2056d9f700 1 -- 172.21.15.104:0/1036474207 wait complete. 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.469+0000 7f2056d9f700 1 Processor -- start 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.469+0000 7f2056d9f700 1 -- start start 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 0x7f20500c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 0x7f20500b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500c6630 con 0x7f2040004430 2023-06-10T18:58:26.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500c4e20 con 0x7f205014d810 2023-06-10T18:58:26.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2056d9f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500c5100 con 0x7f20500c5be0 2023-06-10T18:58:26.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2054b3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2054b3b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40244/0 (socket says 172.21.15.104:40244) 2023-06-10T18:58:26.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f2054b3b700 1 -- 172.21.15.104:0/1322562000 learned_addr learned my addr 172.21.15.104:0/1322562000 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:26.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.470+0000 7f204ffff700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 0x7f20500c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f205533c700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 0x7f20500b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 -- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 msgr2=0x7f20500c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 0x7f20500c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 -- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 msgr2=0x7f20500b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 0x7f20500b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 -- 172.21.15.104:0/1322562000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500c53e0 con 0x7f2040004430 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f204ffff700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 0x7f20500c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f205533c700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 0x7f20500b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2054b3b700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f20440119a0 tx=0x7f2044011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400f940 con 0x7f2040004430 2023-06-10T18:58:26.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204400faa0 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500c56c0 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400fe30 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.471+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20500b2010 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.472+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20500b2290 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.472+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2044014440 con 0x7f2040004430 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.473+0000 7f204d7fa700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 0x7f203406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.473+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f2044089850 con 0x7f2040004430 2023-06-10T18:58:26.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.473+0000 7f204ffff700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 0x7f203406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.473+0000 7f204ffff700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 0x7f203406d190 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f203c008fd0 tx=0x7f203c00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.477+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2044054ba0 con 0x7f2040004430 2023-06-10T18:58:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.666+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f20500c58b0 con 0x7f2040004430 2023-06-10T18:58:26.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.667+0000 7f204d7fa700 1 -- 172.21.15.104:0/1322562000 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v168) v1 ==== 84+0+126 (secure 0 0 0) 0x7f20440581c0 con 0x7f2040004430 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:58:26.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:26.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:26.675 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:58:26.675 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:58:26.675 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:26.675 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:58:26.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 msgr2=0x7f203406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 0x7f203406d190 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f203c008fd0 tx=0x7f203c00a000 comp rx=0 tx=0).stop 2023-06-10T18:58:26.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 msgr2=0x7f20500ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f20440119a0 tx=0x7f2044011d60 comp rx=0 tx=0).stop 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 shutdown_connections 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f203406ace0 0x7f203406d190 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f205014d810 0x7f20500b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2040004430 0x7f20500ca7c0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 --2- 172.21.15.104:0/1322562000 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f20500c5be0 0x7f20500c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 >> 172.21.15.104:0/1322562000 conn(0x7f205006e370 msgr2=0x7f20500ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 shutdown_connections 2023-06-10T18:58:26.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.669+0000 7f2056d9f700 1 -- 172.21.15.104:0/1322562000 wait complete. 2023-06-10T18:58:26.687 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:58:26.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:58:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.804+0000 7ff9fc21b700 1 Processor -- start 2023-06-10T18:58:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.805+0000 7ff9fc21b700 1 -- start start 2023-06-10T18:58:26.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.805+0000 7ff9fc21b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff9f415a330 0x7ff9f40ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.805+0000 7ff9fc21b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f415a700 con 0x7ff9f40b1780 2023-06-10T18:58:26.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.805+0000 7ff9fc21b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f406d7b0 con 0x7ff9f40cb420 2023-06-10T18:58:26.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.805+0000 7ff9fc21b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f40cbe00 con 0x7ff9f415a330 2023-06-10T18:58:26.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9fa7b8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff9f40b1780 0x7ff9f406dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38084/0 (socket says 172.21.15.104:38084) 2023-06-10T18:58:26.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9fa7b8700 1 -- 172.21.15.104:0/2356769237 learned_addr learned my addr 172.21.15.104:0/2356769237 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2356769237 >> v2:172.21.15.104:3300/0 conn(0x7ff9f415a330 0x7ff9f40ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f9fb7700 1 -- 172.21.15.104:0/2356769237 >> v1:172.21.15.114:6789/0 conn(0x7ff9f40cb420 legacy=0x7ff9f40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f9fb7700 1 -- 172.21.15.104:0/2356769237 >> v1:172.21.15.104:6789/0 conn(0x7ff9f40b1780 legacy=0x7ff9f406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f8fb5700 1 -- 172.21.15.104:0/2356769237 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680984527 0 0) 0x7ff9f415a700 con 0x7ff9f40b1780 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f9fb7700 1 -- 172.21.15.104:0/2356769237 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f406ce80 con 0x7ff9f415a330 2023-06-10T18:58:26.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2356769237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40ac1c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009de0 tx=0x7ff9ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fab15511370ad2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f8fb5700 1 -- 172.21.15.104:0/2356769237 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec00e070 con 0x7ff9f415a330 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f8fb5700 1 -- 172.21.15.104:0/2356769237 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec00bbd0 con 0x7ff9f415a330 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.806+0000 7ff9f8fb5700 1 -- 172.21.15.104:0/2356769237 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec012430 con 0x7ff9f415a330 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2356769237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 msgr2=0x7ff9f40ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2356769237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40ac1c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009de0 tx=0x7ff9ec00b330 comp rx=0 tx=0).stop 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2356769237 shutdown_connections 2023-06-10T18:58:26.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2356769237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40ac1c0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2356769237 >> 172.21.15.104:0/2356769237 conn(0x7ff9f406b8a0 msgr2=0x7ff9f40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:26.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2356769237 shutdown_connections 2023-06-10T18:58:26.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.807+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2356769237 wait complete. 2023-06-10T18:58:26.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.808+0000 7ff9fc21b700 1 Processor -- start 2023-06-10T18:58:26.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.808+0000 7ff9fc21b700 1 -- start start 2023-06-10T18:58:26.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 0x7ff9f41567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 0x7ff9f406a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40c3230 con 0x7ff9f415a330 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40c3510 con 0x7ff9f40b1780 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40c37f0 con 0x7ff9f40cb420 2023-06-10T18:58:26.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40262/0 (socket says 172.21.15.104:40262) 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 -- 172.21.15.104:0/2472863893 learned_addr learned my addr 172.21.15.104:0/2472863893 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 0x7ff9f41567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f97b6700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 0x7ff9f406a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 -- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 msgr2=0x7ff9f406a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 0x7ff9f406a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 -- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 msgr2=0x7ff9f41567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 0x7ff9f41567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 -- 172.21.15.104:0/2472863893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec009a70 con 0x7ff9f415a330 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f97b6700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 0x7ff9f406a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 0x7ff9f41567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fa7b8700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c450 tx=0x7ff9e400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4004c80 con 0x7ff9f415a330 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4004de0 con 0x7ff9f415a330 2023-06-10T18:58:26.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f40c3ad0 con 0x7ff9f415a330 2023-06-10T18:58:26.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40072d0 con 0x7ff9f415a330 2023-06-10T18:58:26.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.809+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f40c5890 con 0x7ff9f415a330 2023-06-10T18:58:26.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.810+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f40c3d10 con 0x7ff9f415a330 2023-06-10T18:58:26.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.811+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff9e4007430 con 0x7ff9f415a330 2023-06-10T18:58:26.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.811+0000 7ff9f27fc700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 0x7ff9e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:26.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.812+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff9e4089280 con 0x7ff9f415a330 2023-06-10T18:58:26.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.812+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 0x7ff9e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:26.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.812+0000 7ff9f9fb7700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 0x7ff9e006d190 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40cf2b0 tx=0x7ff9ec011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:26.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:26.815+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff9e40546b0 con 0x7ff9f415a330 2023-06-10T18:58:27.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.006+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff9f40c3370 con 0x7ff9f415a330 2023-06-10T18:58:27.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.910+0000 7ff9f27fc700 1 -- 172.21.15.104:0/2472863893 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v169) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff9f40c3370 con 0x7ff9f415a330 2023-06-10T18:58:27.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 msgr2=0x7ff9e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 0x7ff9e006d190 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40cf2b0 tx=0x7ff9ec011000 comp rx=0 tx=0).stop 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 msgr2=0x7ff9f40c5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c450 tx=0x7ff9e400c810 comp rx=0 tx=0).stop 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 shutdown_connections 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9e006ace0 0x7ff9e006d190 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:27.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f40b1780 0x7ff9f41567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f415a330 0x7ff9f40c5350 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 --2- 172.21.15.104:0/2472863893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f40cb420 0x7ff9f406a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 >> 172.21.15.104:0/2472863893 conn(0x7ff9f406b8a0 msgr2=0x7ff9f41593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 shutdown_connections 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:27.915+0000 7ff9fc21b700 1 -- 172.21.15.104:0/2472863893 wait complete. 2023-06-10T18:58:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:58:27.965 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:58:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.089+0000 7f51c995c700 1 Processor -- start 2023-06-10T18:58:28.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.089+0000 7f51c995c700 1 -- start start 2023-06-10T18:58:28.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c995c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51c4158790 0x7f51c40ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c995c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c4071580 con 0x7f51c40caf70 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c995c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40c9020 con 0x7f51c40ccfe0 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c995c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c4070f30 con 0x7f51c4158790 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51c4158790 0x7f51c40ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f51c4158790 0x7f51c40ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40266/0 (socket says 172.21.15.104:40266) 2023-06-10T18:58:28.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c37fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f51c40caf70 0x7f51c4071a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38098/0 (socket says 172.21.15.104:38098) 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 -- 172.21.15.104:0/2565584997 learned_addr learned my addr 172.21.15.104:0/2565584997 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 -- 172.21.15.104:0/2565584997 >> v1:172.21.15.114:6789/0 conn(0x7f51c40ccfe0 legacy=0x7f51c40c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 -- 172.21.15.104:0/2565584997 >> v1:172.21.15.104:6789/0 conn(0x7f51c40caf70 legacy=0x7f51c4071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 -- 172.21.15.104:0/2565584997 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c4071210 con 0x7f51c4158790 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c1ffb700 1 -- 172.21.15.104:0/2565584997 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884769507 0 0) 0x7f51c4071580 con 0x7f51c40caf70 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.090+0000 7f51c2ffd700 1 --2- 172.21.15.104:0/2565584997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c4158790 0x7f51c40ccaa0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f51b400aa90 tx=0x7f51b400ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6436c927020715 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c1ffb700 1 -- 172.21.15.104:0/2565584997 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400cb30 con 0x7f51c4158790 2023-06-10T18:58:28.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c1ffb700 1 -- 172.21.15.104:0/2565584997 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b400cc90 con 0x7f51c4158790 2023-06-10T18:58:28.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c1ffb700 1 -- 172.21.15.104:0/2565584997 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4004030 con 0x7f51c4158790 2023-06-10T18:58:28.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 -- 172.21.15.104:0/2565584997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c4158790 msgr2=0x7f51c40ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 --2- 172.21.15.104:0/2565584997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c4158790 0x7f51c40ccaa0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f51b400aa90 tx=0x7f51b400ada0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 -- 172.21.15.104:0/2565584997 shutdown_connections 2023-06-10T18:58:28.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 --2- 172.21.15.104:0/2565584997 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c4158790 0x7f51c40ccaa0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 -- 172.21.15.104:0/2565584997 >> 172.21.15.104:0/2565584997 conn(0x7f51c406aac0 msgr2=0x7f51c40b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 -- 172.21.15.104:0/2565584997 shutdown_connections 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.091+0000 7f51c995c700 1 -- 172.21.15.104:0/2565584997 wait complete. 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.092+0000 7f51c995c700 1 Processor -- start 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 -- start start 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51c40caf70 0x7f51c4151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:28.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4158790 0x7f51c4071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4155ce0 con 0x7f51c40ccfe0 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c415a590 con 0x7f51c4158790 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c995c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4159cf0 con 0x7f51c40caf70 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c27fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40274/0 (socket says 172.21.15.104:40274) 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.093+0000 7f51c27fc700 1 -- 172.21.15.104:0/2431518398 learned_addr learned my addr 172.21.15.104:0/2431518398 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:28.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c37fe700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4158790 0x7f51c4071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c2ffd700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51c40caf70 0x7f51c4151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 -- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51c40caf70 msgr2=0x7f51c4151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51c40caf70 0x7f51c4151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 -- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4158790 msgr2=0x7f51c4071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4158790 0x7f51c4071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 -- 172.21.15.104:0/2431518398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b400a6f0 con 0x7f51c40ccfe0 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c27fc700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c4d0 tx=0x7f51b800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:28.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8011790 con 0x7f51c40ccfe0 2023-06-10T18:58:28.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b80118f0 con 0x7f51c40ccfe0 2023-06-10T18:58:28.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8011ca0 con 0x7f51c40ccfe0 2023-06-10T18:58:28.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4159fd0 con 0x7f51c40ccfe0 2023-06-10T18:58:28.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.094+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c4152190 con 0x7f51c40ccfe0 2023-06-10T18:58:28.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.095+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4155b40 con 0x7f51c40ccfe0 2023-06-10T18:58:28.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.095+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f51b801c460 con 0x7f51c40ccfe0 2023-06-10T18:58:28.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.096+0000 7f51abfff700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 0x7f51b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:28.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.096+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f51b8089190 con 0x7f51c40ccfe0 2023-06-10T18:58:28.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.096+0000 7f51c2ffd700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 0x7f51b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:28.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.096+0000 7f51c2ffd700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 0x7f51b006d190 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f51b400a240 tx=0x7f51b4012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:28.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.100+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f51c4155b40 con 0x7f51c40ccfe0 2023-06-10T18:58:28.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.287+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f51c4155b40 con 0x7f51c40ccfe0 2023-06-10T18:58:28.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.911+0000 7f51abfff700 1 -- 172.21.15.104:0/2431518398 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v170) v1 ==== 170+0+0 (secure 0 0 0) 0x7f51c4155b40 con 0x7f51c40ccfe0 2023-06-10T18:58:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 msgr2=0x7f51b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 0x7f51b006d190 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f51b400a240 tx=0x7f51b4012000 comp rx=0 tx=0).stop 2023-06-10T18:58:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 msgr2=0x7f51c41561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:28.927 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c4d0 tx=0x7f51b800c890 comp rx=0 tx=0).stop 2023-06-10T18:58:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 shutdown_connections 2023-06-10T18:58:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.920+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f51b006ace0 0x7f51b006d190 secure :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f51b400a240 tx=0x7f51b4012000 comp rx=0 tx=0).stop 2023-06-10T18:58:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.921+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4158790 0x7f51c4071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.921+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f51c40caf70 0x7f51c4151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.928 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.921+0000 7f51c995c700 1 --2- 172.21.15.104:0/2431518398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51c40ccfe0 0x7f51c41561c0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.921+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 >> 172.21.15.104:0/2431518398 conn(0x7f51c406aac0 msgr2=0x7f51c415be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.921+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 shutdown_connections 2023-06-10T18:58:28.929 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:28.922+0000 7f51c995c700 1 -- 172.21.15.104:0/2431518398 wait complete. 2023-06-10T18:58:28.934 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:58:28.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:58:29.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.063+0000 7fdc524d5700 1 Processor -- start 2023-06-10T18:58:29.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.063+0000 7fdc524d5700 1 -- start start 2023-06-10T18:58:29.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc524d5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdc4c06ea70 0x7fdc4c06d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc524d5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c0ae7e0 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc524d5700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c06df90 con 0x7fdc4c0b1780 2023-06-10T18:58:29.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc524d5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c06e270 con 0x7fdc4c06ea70 2023-06-10T18:58:29.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc4bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdc4c15c6f0 0x7fdc4c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38108/0 (socket says 172.21.15.104:38108) 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc4bfff700 1 -- 172.21.15.104:0/2269299935 learned_addr learned my addr 172.21.15.104:0/2269299935 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.064+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2269299935 >> v2:172.21.15.114:3300/0 conn(0x7fdc4c06ea70 0x7fdc4c06d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487982826 0 0) 0x7fdc4c0ae7e0 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc380035b0 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201384415 0 0) 0x7fdc380035b0 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 >> v2:172.21.15.114:3300/0 conn(0x7fdc4c06ea70 msgr2=0x7fdc4c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 --2- 172.21.15.104:0/2269299935 >> v2:172.21.15.114:3300/0 conn(0x7fdc4c06ea70 0x7fdc4c06d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 >> v1:172.21.15.114:6789/0 conn(0x7fdc4c0b1780 legacy=0x7fdc4c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c06e550 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdc3c0017f0 con 0x7fdc4c15c6f0 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 >> v1:172.21.15.104:6789/0 conn(0x7fdc4c15c6f0 legacy=0x7fdc4c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc38003a00 0x7fdc38003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38004850 0x7fdc38004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 0x7fdc3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c06e270 con 0x7fdc38005200 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3800c110 con 0x7fdc38004850 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3800c2d0 con 0x7fdc38003a00 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 0x7fdc3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4b7fe700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38004850 0x7fdc38004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4bfff700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc38003a00 0x7fdc38003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 -- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc38003a00 msgr2=0x7fdc38003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc38003a00 0x7fdc38003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 -- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38004850 msgr2=0x7fdc38004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38004850 0x7fdc38004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 -- 172.21.15.104:0/2269299935 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc380035b0 con 0x7fdc38005200 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 -- 172.21.15.104:0/2269299935 reap_dead start 2023-06-10T18:58:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 0x7fdc3800ba10 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c072060 tx=0x7fdc38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaef3d02d562e865 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.065+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44018070 con 0x7fdc38005200 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4400f040 con 0x7fdc38005200 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc4affd700 1 -- 172.21.15.104:0/2269299935 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400ec10 con 0x7fdc38005200 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 -- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 msgr2=0x7fdc3800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 0x7fdc3800ba10 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c072060 tx=0x7fdc38005880 comp rx=0 tx=0).stop 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 -- 172.21.15.104:0/2269299935 shutdown_connections 2023-06-10T18:58:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2269299935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc38005200 0x7fdc3800ba10 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 -- 172.21.15.104:0/2269299935 >> 172.21.15.104:0/2269299935 conn(0x7fdc4c06b8a0 msgr2=0x7fdc4c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 -- 172.21.15.104:0/2269299935 shutdown_connections 2023-06-10T18:58:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.066+0000 7fdc524d5700 1 -- 172.21.15.104:0/2269299935 wait complete. 2023-06-10T18:58:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.067+0000 7fdc524d5700 1 Processor -- start 2023-06-10T18:58:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.067+0000 7fdc524d5700 1 -- start start 2023-06-10T18:58:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38005200 0x7fdc4c0cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 0x7fdc4c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c0b2c80 con 0x7fdc4c14e670 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c14f020 con 0x7fdc38005200 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc524d5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c0c73b0 con 0x7fdc4c0c6630 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc50a72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc50a72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40296/0 (socket says 172.21.15.104:40296) 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc50a72700 1 -- 172.21.15.104:0/2119114831 learned_addr learned my addr 172.21.15.104:0/2119114831 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc4bfff700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38005200 0x7fdc4c0cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.068+0000 7fdc4b7fe700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 0x7fdc4c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 -- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 msgr2=0x7fdc4c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 0x7fdc4c0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 -- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38005200 msgr2=0x7fdc4c0cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38005200 0x7fdc4c0cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 -- 172.21.15.104:0/2119114831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c0c70d0 con 0x7fdc4c14e670 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc4b7fe700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 0x7fdc4c0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc50a72700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400e700 tx=0x7fdc44009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44018040 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc44014cf0 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c0b25c0 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44007000 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.069+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c15b940 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.070+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c0b2420 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.070+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdc440206c0 con 0x7fdc4c14e670 2023-06-10T18:58:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.071+0000 7fdc48ff9700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 0x7fdc2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.071+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fdc4408b280 con 0x7fdc4c14e670 2023-06-10T18:58:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.071+0000 7fdc4bfff700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 0x7fdc2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.071+0000 7fdc4bfff700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 0x7fdc2c06d190 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.075+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdc4c0b2420 con 0x7fdc4c14e670 2023-06-10T18:58:29.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.305+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdc4c0b2dc0 con 0x7fdc4c14e670 2023-06-10T18:58:29.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.310+0000 7fdc48ff9700 1 -- 172.21.15.104:0/2119114831 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v106) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdc4c0b2dc0 con 0x7fdc4c14e670 2023-06-10T18:58:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.314+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 msgr2=0x7fdc2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.314+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 0x7fdc2c06d190 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000c00 tx=0x7fdc3c009040 comp rx=0 tx=0).stop 2023-06-10T18:58:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.314+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 msgr2=0x7fdc4c14eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.314+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400e700 tx=0x7fdc44009750 comp rx=0 tx=0).stop 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 shutdown_connections 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdc2c06ace0 0x7fdc2c06d190 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc38005200 0x7fdc4c0cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc4c14e670 0x7fdc4c14eae0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 --2- 172.21.15.104:0/2119114831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc4c0c6630 0x7fdc4c0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 >> 172.21.15.104:0/2119114831 conn(0x7fdc4c06b8a0 msgr2=0x7fdc4c0b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 shutdown_connections 2023-06-10T18:58:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.315+0000 7fdc524d5700 1 -- 172.21.15.104:0/2119114831 wait complete. 2023-06-10T18:58:29.324 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 24 and data pool 25 2023-06-10T18:58:29.344 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:29.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.457+0000 7f9d7a99d700 1 Processor -- start 2023-06-10T18:58:29.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.457+0000 7f9d7a99d700 1 -- start start 2023-06-10T18:58:29.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d7a99d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d7415c500 0x7f9d740cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d7a99d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9d74069e70 0x7f9d7406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d7a99d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7406dcb0 con 0x7f9d740ce040 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d7a99d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7406e850 con 0x7f9d7415c500 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d7a99d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7406a830 con 0x7f9d74069e70 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d73fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d7415c500 0x7f9d740cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d737fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9d740ce040 0x7f9d7406e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38130/0 (socket says 172.21.15.104:38130) 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d73fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9d7415c500 0x7f9d740cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40300/0 (socket says 172.21.15.104:40300) 2023-06-10T18:58:29.467 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d737fe700 1 -- 172.21.15.104:0/2192059423 learned_addr learned my addr 172.21.15.104:0/2192059423 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:29.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.458+0000 7f9d78f3a700 1 --2- 172.21.15.104:0/2192059423 >> v2:172.21.15.114:3300/0 conn(0x7f9d74069e70 0x7f9d7406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d73fff700 1 -- 172.21.15.104:0/2192059423 >> v2:172.21.15.114:3300/0 conn(0x7f9d74069e70 msgr2=0x7f9d7406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:29.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d73fff700 1 --2- 172.21.15.104:0/2192059423 >> v2:172.21.15.114:3300/0 conn(0x7f9d74069e70 0x7f9d7406a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d73fff700 1 -- 172.21.15.104:0/2192059423 >> v1:172.21.15.104:6789/0 conn(0x7f9d740ce040 legacy=0x7f9d7406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d73fff700 1 -- 172.21.15.104:0/2192059423 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740ce470 con 0x7f9d7415c500 2023-06-10T18:58:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d72ffd700 1 -- 172.21.15.104:0/2192059423 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915069506 0 0) 0x7f9d7406dcb0 con 0x7f9d740ce040 2023-06-10T18:58:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d73fff700 1 --2- 172.21.15.104:0/2192059423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d740cf030 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f9d740cddb0 tx=0x7f9d64009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e1cd95cb798568 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d72ffd700 1 -- 172.21.15.104:0/2192059423 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64005e30 con 0x7f9d7415c500 2023-06-10T18:58:29.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d72ffd700 1 -- 172.21.15.104:0/2192059423 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64006030 con 0x7f9d7415c500 2023-06-10T18:58:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d72ffd700 1 -- 172.21.15.104:0/2192059423 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640063c0 con 0x7f9d7415c500 2023-06-10T18:58:29.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 -- 172.21.15.104:0/2192059423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 msgr2=0x7f9d740cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/2192059423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d740cf030 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f9d740cddb0 tx=0x7f9d64009b00 comp rx=0 tx=0).stop 2023-06-10T18:58:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 -- 172.21.15.104:0/2192059423 shutdown_connections 2023-06-10T18:58:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/2192059423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d740cf030 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/2192059423 >> v2:172.21.15.114:3300/0 conn(0x7f9d74069e70 0x7f9d7406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 -- 172.21.15.104:0/2192059423 >> 172.21.15.104:0/2192059423 conn(0x7f9d7406b8a0 msgr2=0x7f9d740cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:29.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.459+0000 7f9d7a99d700 1 -- 172.21.15.104:0/2192059423 shutdown_connections 2023-06-10T18:58:29.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.460+0000 7f9d7a99d700 1 -- 172.21.15.104:0/2192059423 wait complete. 2023-06-10T18:58:29.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.461+0000 7f9d7a99d700 1 Processor -- start 2023-06-10T18:58:29.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.461+0000 7f9d7a99d700 1 -- start start 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d74069e70 0x7f9d74159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d740ce040 0x7f9d740b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7415a290 con 0x7f9d7415c500 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740ccc20 con 0x7f9d74069e70 2023-06-10T18:58:29.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b35c0 con 0x7f9d740ce040 2023-06-10T18:58:29.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40310/0 (socket says 172.21.15.104:40310) 2023-06-10T18:58:29.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 -- 172.21.15.104:0/161840793 learned_addr learned my addr 172.21.15.104:0/161840793 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:29.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d73fff700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d74069e70 0x7f9d74159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d737fe700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d740ce040 0x7f9d740b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 -- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d740ce040 msgr2=0x7f9d740b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d740ce040 0x7f9d740b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 -- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d74069e70 msgr2=0x7f9d74159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d74069e70 0x7f9d74159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 -- 172.21.15.104:0/161840793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d64009430 con 0x7f9d7415c500 2023-06-10T18:58:29.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d78f3a700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c007e90 tx=0x7f9d6c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00ede0 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c014420 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0147d0 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d74157f50 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.462+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d740b2c90 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.463+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7406fae0 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.464+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9d6c014930 con 0x7f9d7415c500 2023-06-10T18:58:29.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.464+0000 7f9d70ff9700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 0x7f9d6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.464+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9d6c089de0 con 0x7f9d7415c500 2023-06-10T18:58:29.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.464+0000 7f9d73fff700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 0x7f9d6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.465+0000 7f9d73fff700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 0x7f9d6006d190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009b40 tx=0x7f9d64008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.469+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9d6c054cc0 con 0x7f9d7415c500 2023-06-10T18:58:29.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.656+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d74074050 con 0x7f9d7415c500 2023-06-10T18:58:29.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.657+0000 7f9d70ff9700 1 -- 172.21.15.104:0/161840793 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+18627 (secure 0 0 0) 0x7f9d6c0582e0 con 0x7f9d7415c500 2023-06-10T18:58:29.664 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:29.664 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":171,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:29.307384+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:29.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 msgr2=0x7f9d6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 0x7f9d6006d190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009b40 tx=0x7f9d64008070 comp rx=0 tx=0).stop 2023-06-10T18:58:29.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 msgr2=0x7f9d741e5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c007e90 tx=0x7f9d6c00e3c0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 shutdown_connections 2023-06-10T18:58:29.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9d6006ace0 0x7f9d6006d190 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d74069e70 0x7f9d74159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7415c500 0x7f9d741e5eb0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 --2- 172.21.15.104:0/161840793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d740ce040 0x7f9d740b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 >> 172.21.15.104:0/161840793 conn(0x7f9d7406b8a0 msgr2=0x7f9d740ada80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 shutdown_connections 2023-06-10T18:58:29.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.659+0000 7f9d7a99d700 1 -- 172.21.15.104:0/161840793 wait complete. 2023-06-10T18:58:29.677 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:29.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.790+0000 7ff49547f700 1 Processor -- start 2023-06-10T18:58:29.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.790+0000 7ff49547f700 1 -- start start 2023-06-10T18:58:29.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff49547f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff4900d0c00 0x7ff49006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff49547f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49006bfd0 con 0x7ff49006b820 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff49547f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49006c2b0 con 0x7ff4900b1e30 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff49547f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4900ccf70 con 0x7ff4900d0c00 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48e7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff4900d0c00 0x7ff49006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff49006b820 0x7ff4900cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38146/0 (socket says 172.21.15.104:38146) 2023-06-10T18:58:29.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48f7fe700 1 -- 172.21.15.104:0/3514117073 learned_addr learned my addr 172.21.15.104:0/3514117073 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48e7fc700 1 -- 172.21.15.104:0/3514117073 >> v1:172.21.15.114:6789/0 conn(0x7ff4900b1e30 legacy=0x7ff4900d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48e7fc700 1 -- 172.21.15.104:0/3514117073 >> v1:172.21.15.104:6789/0 conn(0x7ff49006b820 legacy=0x7ff4900cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48dffb700 1 -- 172.21.15.104:0/3514117073 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860871802 0 0) 0x7ff49006bfd0 con 0x7ff49006b820 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.791+0000 7ff48e7fc700 1 -- 172.21.15.104:0/3514117073 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4900cd250 con 0x7ff4900d0c00 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff48e7fc700 1 --2- 172.21.15.104:0/3514117073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49006b2e0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7ff480009840 tx=0x7ff480009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c380524b2d5856c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff48dffb700 1 -- 172.21.15.104:0/3514117073 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480011830 con 0x7ff4900d0c00 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff48dffb700 1 -- 172.21.15.104:0/3514117073 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480011990 con 0x7ff4900d0c00 2023-06-10T18:58:29.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff48dffb700 1 -- 172.21.15.104:0/3514117073 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480011d20 con 0x7ff4900d0c00 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 -- 172.21.15.104:0/3514117073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 msgr2=0x7ff49006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 --2- 172.21.15.104:0/3514117073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49006b2e0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7ff480009840 tx=0x7ff480009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 -- 172.21.15.104:0/3514117073 shutdown_connections 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 --2- 172.21.15.104:0/3514117073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49006b2e0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 -- 172.21.15.104:0/3514117073 >> 172.21.15.104:0/3514117073 conn(0x7ff490071ec0 msgr2=0x7ff49009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 -- 172.21.15.104:0/3514117073 shutdown_connections 2023-06-10T18:58:29.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.792+0000 7ff49547f700 1 -- 172.21.15.104:0/3514117073 wait complete. 2023-06-10T18:58:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 Processor -- start 2023-06-10T18:58:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 -- start start 2023-06-10T18:58:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff49006b820 0x7ff4900c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 0x7ff49015df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900b5a80 con 0x7ff4900d0c00 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49015da70 con 0x7ff49006b820 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.794+0000 7ff49547f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49015ce60 con 0x7ff4900b1e30 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40324/0 (socket says 172.21.15.104:40324) 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 -- 172.21.15.104:0/231860020 learned_addr learned my addr 172.21.15.104:0/231860020 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:29.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48effd700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff49006b820 0x7ff4900c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48e7fc700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 0x7ff49015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 -- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 msgr2=0x7ff49015df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 0x7ff49015df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 -- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff49006b820 msgr2=0x7ff4900c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff49006b820 0x7ff4900c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 -- 172.21.15.104:0/231860020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4800094a0 con 0x7ff4900d0c00 2023-06-10T18:58:29.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48e7fc700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 0x7ff49015df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:29.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff48f7fe700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7ff488002230 tx=0x7ff48800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800ab70 con 0x7ff4900d0c00 2023-06-10T18:58:29.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48800acf0 con 0x7ff4900d0c00 2023-06-10T18:58:29.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49015d140 con 0x7ff4900d0c00 2023-06-10T18:58:29.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488007310 con 0x7ff4900d0c00 2023-06-10T18:58:29.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.795+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49015d700 con 0x7ff4900d0c00 2023-06-10T18:58:29.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.796+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4900d1860 con 0x7ff4900d0c00 2023-06-10T18:58:29.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.796+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff488007470 con 0x7ff4900d0c00 2023-06-10T18:58:29.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.797+0000 7ff4777fe700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 0x7ff47c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:29.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.797+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff488089470 con 0x7ff4900d0c00 2023-06-10T18:58:29.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.797+0000 7ff48effd700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 0x7ff47c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:29.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.797+0000 7ff48effd700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 0x7ff47c06d190 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff49015fbc0 tx=0x7ff484009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:29.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.801+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff4880543e0 con 0x7ff4900d0c00 2023-06-10T18:58:29.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.989+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff49015dbe0 con 0x7ff4900d0c00 2023-06-10T18:58:29.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.991+0000 7ff4777fe700 1 -- 172.21.15.104:0/231860020 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+18627 (secure 0 0 0) 0x7ff48800ae60 con 0x7ff4900d0c00 2023-06-10T18:58:29.997 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:29.998 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":171,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:29.307384+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:30.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 msgr2=0x7ff47c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 0x7ff47c06d190 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff49015fbc0 tx=0x7ff484009040 comp rx=0 tx=0).stop 2023-06-10T18:58:30.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 msgr2=0x7ff49015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.000 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7ff488002230 tx=0x7ff48800cf50 comp rx=0 tx=0).stop 2023-06-10T18:58:30.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 shutdown_connections 2023-06-10T18:58:30.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff47c06ace0 0x7ff47c06d190 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff49006b820 0x7ff4900c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4900d0c00 0x7ff49015ef90 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 --2- 172.21.15.104:0/231860020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4900b1e30 0x7ff49015df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 >> 172.21.15.104:0/231860020 conn(0x7ff490071ec0 msgr2=0x7ff49015be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:30.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 shutdown_connections 2023-06-10T18:58:30.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:29.993+0000 7ff49547f700 1 -- 172.21.15.104:0/231860020 wait complete. 2023-06-10T18:58:30.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:58:30.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.129+0000 7fb7e06da700 1 Processor -- start 2023-06-10T18:58:30.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.129+0000 7fb7e06da700 1 -- start start 2023-06-10T18:58:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7e06da700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7d80cd150 0x7fb7d80cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7e06da700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80cda90 con 0x7fb7d80af940 2023-06-10T18:58:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7e06da700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d8154970 con 0x7fb7d81545a0 2023-06-10T18:58:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7e06da700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d815dab0 con 0x7fb7d80cd150 2023-06-10T18:58:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7ddc75700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb7d80af940 0x7fb7d815e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38160/0 (socket says 172.21.15.104:38160) 2023-06-10T18:58:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.130+0000 7fb7ddc75700 1 -- 172.21.15.104:0/2022153803 learned_addr learned my addr 172.21.15.104:0/2022153803 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:30.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dec77700 1 --2- 172.21.15.104:0/2022153803 >> v2:172.21.15.114:3300/0 conn(0x7fb7d80cd150 0x7fb7d80cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999183965 0 0) 0x7fb7d80cda90 con 0x7fb7d80af940 2023-06-10T18:58:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c40035b0 con 0x7fb7d80af940 2023-06-10T18:58:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1059066154 0 0) 0x7fb7c40035b0 con 0x7fb7d80af940 2023-06-10T18:58:30.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 >> v2:172.21.15.114:3300/0 conn(0x7fb7d80cd150 msgr2=0x7fb7d80cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:30.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 --2- 172.21.15.104:0/2022153803 >> v2:172.21.15.114:3300/0 conn(0x7fb7d80cd150 0x7fb7d80cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 >> v1:172.21.15.114:6789/0 conn(0x7fb7d81545a0 legacy=0x7fb7d815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80cb0b0 con 0x7fb7d80af940 2023-06-10T18:58:30.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb7d4001670 con 0x7fb7d80af940 2023-06-10T18:58:30.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 >> v1:172.21.15.104:6789/0 conn(0x7fb7d80af940 legacy=0x7fb7d815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 0x7fb7c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c4004850 0x7fb7c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4005200 0x7fb7c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d815dab0 con 0x7fb7c4003a00 2023-06-10T18:58:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c400c080 con 0x7fb7c4004850 2023-06-10T18:58:30.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c400c240 con 0x7fb7c4005200 2023-06-10T18:58:30.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7de476700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 0x7fb7c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7ddc75700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c4004850 0x7fb7c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.131+0000 7fb7dec77700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4005200 0x7fb7c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 -- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4005200 msgr2=0x7fb7c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4005200 0x7fb7c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 -- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c4004850 msgr2=0x7fb7c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:30.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c4004850 0x7fb7c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 -- 172.21.15.104:0/2022153803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40035b0 con 0x7fb7c4003a00 2023-06-10T18:58:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7dec77700 1 -- 172.21.15.104:0/2022153803 reap_dead start 2023-06-10T18:58:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7de476700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 0x7fb7c4003e10 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80017c0 tx=0x7fb7c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f49eccc3fae4b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:30.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80052d0 con 0x7fb7c4003a00 2023-06-10T18:58:30.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c8005430 con 0x7fb7c4003a00 2023-06-10T18:58:30.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7dd474700 1 -- 172.21.15.104:0/2022153803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80038c0 con 0x7fb7c4003a00 2023-06-10T18:58:30.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 -- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 msgr2=0x7fb7c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 0x7fb7c4003e10 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80017c0 tx=0x7fb7c4005880 comp rx=0 tx=0).stop 2023-06-10T18:58:30.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 -- 172.21.15.104:0/2022153803 shutdown_connections 2023-06-10T18:58:30.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 --2- 172.21.15.104:0/2022153803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7c4003a00 0x7fb7c4003e10 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 -- 172.21.15.104:0/2022153803 >> 172.21.15.104:0/2022153803 conn(0x7fb7d806e370 msgr2=0x7fb7d806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:30.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.132+0000 7fb7e06da700 1 -- 172.21.15.104:0/2022153803 shutdown_connections 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.133+0000 7fb7e06da700 1 -- 172.21.15.104:0/2022153803 wait complete. 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 Processor -- start 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 -- start start 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 0x7fb7d80c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 0x7fb7d80c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c5b50 con 0x7fb7d80c4d40 2023-06-10T18:58:30.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c6520 con 0x7fb7d81e4630 2023-06-10T18:58:30.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7e06da700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d806c130 con 0x7fb7c4003a00 2023-06-10T18:58:30.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.134+0000 7fb7dec77700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40344/0 (socket says 172.21.15.104:40344) 2023-06-10T18:58:30.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 -- 172.21.15.104:0/1190757128 learned_addr learned my addr 172.21.15.104:0/1190757128 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:30.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7ddc75700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 0x7fb7d80c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7de476700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 0x7fb7d80c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 -- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 msgr2=0x7fb7d80c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 0x7fb7d80c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 -- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 msgr2=0x7fb7d80c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 0x7fb7d80c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 -- 172.21.15.104:0/1190757128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d806c410 con 0x7fb7d80c4d40 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7de476700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 0x7fb7d80c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7ddc75700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 0x7fb7d80c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7dec77700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80aff70 tx=0x7fb7d00114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:30.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d000e070 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d0003c30 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0003e20 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d806c6f0 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.135+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d80c56f0 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.136+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d806c930 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.137+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb7d0011a10 con 0x7fb7d80c4d40 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.137+0000 7fb7ceffd700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 0x7fb7bc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:30.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.137+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb7d0089a10 con 0x7fb7d80c4d40 2023-06-10T18:58:30.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.137+0000 7fb7de476700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 0x7fb7bc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:30.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.138+0000 7fb7de476700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 0x7fb7bc06d1c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8015040 tx=0x7fb7c8009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:30.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.141+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb7d00548f0 con 0x7fb7d80c4d40 2023-06-10T18:58:30.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:30.375+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb7d8074050 con 0x7fb7d80c4d40 2023-06-10T18:58:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.332+0000 7fb7ceffd700 1 -- 172.21.15.104:0/1190757128 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v109) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb7d0057f10 con 0x7fb7d80c4d40 2023-06-10T18:58:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 msgr2=0x7fb7bc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 0x7fb7bc06d1c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8015040 tx=0x7fb7c8009030 comp rx=0 tx=0).stop 2023-06-10T18:58:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 msgr2=0x7fb7d80c51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80aff70 tx=0x7fb7d00114b0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 shutdown_connections 2023-06-10T18:58:31.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d80c4d40 0x7fb7d80c51b0 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c4003a00 0x7fb7d80c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb7bc06ad10 0x7fb7bc06d1c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 --2- 172.21.15.104:0/1190757128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7d81e4630 0x7fb7d80c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 >> 172.21.15.104:0/1190757128 conn(0x7fb7d806e370 msgr2=0x7fb7d80b4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:31.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 shutdown_connections 2023-06-10T18:58:31.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.335+0000 7fb7e06da700 1 -- 172.21.15.104:0/1190757128 wait complete. 2023-06-10T18:58:31.357 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:58:31.358 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:58:31.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.489+0000 7f8ba513a700 1 Processor -- start 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.489+0000 7f8ba513a700 1 -- start start 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8ba513a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba015c6f0 0x7f8ba00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8ba513a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ba00aec00 0x7f8ba00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8ba513a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00ae7e0 con 0x7f8ba006ea70 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8ba513a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba006df90 con 0x7f8ba00aec00 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8ba513a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba006e270 con 0x7f8ba015c6f0 2023-06-10T18:58:31.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ba00aec00 0x7f8ba00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:31.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9e59c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ba006ea70 0x7f8ba00ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38178/0 (socket says 172.21.15.104:38178) 2023-06-10T18:58:31.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9e59c700 1 -- 172.21.15.104:0/2590124552 learned_addr learned my addr 172.21.15.104:0/2590124552 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:31.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 -- 172.21.15.104:0/2590124552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba015c6f0 msgr2=0x7f8ba00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:31.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 --2- 172.21.15.104:0/2590124552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba015c6f0 0x7f8ba00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 -- 172.21.15.104:0/2590124552 >> v1:172.21.15.104:6789/0 conn(0x7f8ba006ea70 legacy=0x7f8ba00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 -- 172.21.15.104:0/2590124552 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba006e550 con 0x7f8ba00aec00 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9dd9b700 1 -- 172.21.15.104:0/2590124552 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237088734 0 0) 0x7f8ba00ae7e0 con 0x7f8ba006ea70 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9f59e700 1 --2- 172.21.15.104:0/2590124552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba00aec00 0x7f8ba00b15b0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009d90 tx=0x7f8b9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8db41de02196529 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9dd9b700 1 -- 172.21.15.104:0/2590124552 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9800e070 con 0x7f8ba00aec00 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9dd9b700 1 -- 172.21.15.104:0/2590124552 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9800bbd0 con 0x7f8ba00aec00 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.490+0000 7f8b9dd9b700 1 -- 172.21.15.104:0/2590124552 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98012430 con 0x7f8ba00aec00 2023-06-10T18:58:31.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 -- 172.21.15.104:0/2590124552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba00aec00 msgr2=0x7f8ba00b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 --2- 172.21.15.104:0/2590124552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba00aec00 0x7f8ba00b15b0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009d90 tx=0x7f8b9800b330 comp rx=0 tx=0).stop 2023-06-10T18:58:31.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 -- 172.21.15.104:0/2590124552 shutdown_connections 2023-06-10T18:58:31.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 --2- 172.21.15.104:0/2590124552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba00aec00 0x7f8ba00b15b0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 --2- 172.21.15.104:0/2590124552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba015c6f0 0x7f8ba00b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 -- 172.21.15.104:0/2590124552 >> 172.21.15.104:0/2590124552 conn(0x7f8ba006b8a0 msgr2=0x7f8ba00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 -- 172.21.15.104:0/2590124552 shutdown_connections 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.491+0000 7f8ba513a700 1 -- 172.21.15.104:0/2590124552 wait complete. 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.492+0000 7f8ba513a700 1 Processor -- start 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.492+0000 7f8ba513a700 1 -- start start 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 0x7f8ba00b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 0x7f8ba00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0155b30 con 0x7f8ba006ea70 2023-06-10T18:58:31.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0155e10 con 0x7f8ba015c6f0 2023-06-10T18:58:31.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8ba513a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba00c9ea0 con 0x7f8ba00aec00 2023-06-10T18:58:31.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:31.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40354/0 (socket says 172.21.15.104:40354) 2023-06-10T18:58:31.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9ed9d700 1 -- 172.21.15.104:0/655296616 learned_addr learned my addr 172.21.15.104:0/655296616 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:31.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9f59e700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 0x7f8ba00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9e59c700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 0x7f8ba00b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9ed9d700 1 -- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 msgr2=0x7f8ba00b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.493+0000 7f8b9ed9d700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 0x7f8ba00b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9ed9d700 1 -- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 msgr2=0x7f8ba00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:31.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9ed9d700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 0x7f8ba00b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9ed9d700 1 -- 172.21.15.104:0/655296616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009a70 con 0x7f8ba006ea70 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9e59c700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 0x7f8ba00b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9f59e700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 0x7f8ba00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b9ed9d700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b480 tx=0x7f8b9000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90005a00 con 0x7f8ba006ea70 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90005b60 con 0x7f8ba006ea70 2023-06-10T18:58:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba00ca180 con 0x7f8ba006ea70 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90011630 con 0x7f8ba006ea70 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.494+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba0151930 con 0x7f8ba006ea70 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.495+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8b90011790 con 0x7f8ba006ea70 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.495+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba00cdb20 con 0x7f8ba006ea70 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.496+0000 7f8b877fe700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 0x7f8b8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:31.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.496+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8b90089200 con 0x7f8ba006ea70 2023-06-10T18:58:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.496+0000 7f8b9e59c700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 0x7f8b8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.497+0000 7f8b9e59c700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 0x7f8b8c06d190 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00b4b30 tx=0x7f8b94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:31.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.500+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ba00cdb20 con 0x7f8ba006ea70 2023-06-10T18:58:31.742 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:31.730+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8ba00ca6a0 con 0x7f8ba006ea70 2023-06-10T18:58:32.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.333+0000 7f8b877fe700 1 -- 172.21.15.104:0/655296616 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v110) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8ba00ca6a0 con 0x7f8ba006ea70 2023-06-10T18:58:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 msgr2=0x7f8b8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 0x7f8b8c06d190 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00b4b30 tx=0x7f8b94009040 comp rx=0 tx=0).stop 2023-06-10T18:58:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 msgr2=0x7f8ba01513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b480 tx=0x7f8b9000b840 comp rx=0 tx=0).stop 2023-06-10T18:58:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 shutdown_connections 2023-06-10T18:58:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b8c06ace0 0x7f8b8c06d190 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba015c6f0 0x7f8ba00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ba006ea70 0x7f8ba01513f0 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 --2- 172.21.15.104:0/655296616 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ba00aec00 0x7f8ba00b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 >> 172.21.15.104:0/655296616 conn(0x7f8ba006b8a0 msgr2=0x7f8ba00b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 shutdown_connections 2023-06-10T18:58:32.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.336+0000 7f8ba513a700 1 -- 172.21.15.104:0/655296616 wait complete. 2023-06-10T18:58:32.356 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:58:32.357 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:32.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 Processor -- start 2023-06-10T18:58:32.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 -- start start 2023-06-10T18:58:32.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 0x7f297406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2974069e70 0x7f29740ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297406d290 con 0x7f297406e4b0 2023-06-10T18:58:32.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2974071640 con 0x7f2974069e70 2023-06-10T18:58:32.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.479+0000 7f297a8b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740aef00 con 0x7f297406e0e0 2023-06-10T18:58:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2974069e70 0x7f29740ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2974069e70 0x7f29740ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40366/0 (socket says 172.21.15.104:40366) 2023-06-10T18:58:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 -- 172.21.15.104:0/1903190312 learned_addr learned my addr 172.21.15.104:0/1903190312 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2973fff700 1 --2- 172.21.15.104:0/1903190312 >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 0x7f297406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 -- 172.21.15.104:0/1903190312 >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 msgr2=0x7f297406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 --2- 172.21.15.104:0/1903190312 >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 0x7f297406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 -- 172.21.15.104:0/1903190312 >> v1:172.21.15.114:6789/0 conn(0x7f297406e4b0 legacy=0x7f29740c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 -- 172.21.15.104:0/1903190312 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29740ca2d0 con 0x7f2974069e70 2023-06-10T18:58:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2973fff700 1 --2- 172.21.15.104:0/1903190312 >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 0x7f297406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:32.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2978e4f700 1 --2- 172.21.15.104:0/1903190312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2974069e70 0x7f29740ce9f0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f297406af50 tx=0x7f296c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a765244424ce008d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2972ffd700 1 -- 172.21.15.104:0/1903190312 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c00e070 con 0x7f2974069e70 2023-06-10T18:58:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2972ffd700 1 -- 172.21.15.104:0/1903190312 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c003a20 con 0x7f2974069e70 2023-06-10T18:58:32.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.480+0000 7f2972ffd700 1 -- 172.21.15.104:0/1903190312 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c003dd0 con 0x7f2974069e70 2023-06-10T18:58:32.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 -- 172.21.15.104:0/1903190312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2974069e70 msgr2=0x7f29740ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 --2- 172.21.15.104:0/1903190312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2974069e70 0x7f29740ce9f0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f297406af50 tx=0x7f296c00b330 comp rx=0 tx=0).stop 2023-06-10T18:58:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 -- 172.21.15.104:0/1903190312 shutdown_connections 2023-06-10T18:58:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 --2- 172.21.15.104:0/1903190312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2974069e70 0x7f29740ce9f0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 --2- 172.21.15.104:0/1903190312 >> v2:172.21.15.114:3300/0 conn(0x7f297406e0e0 0x7f297406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 -- 172.21.15.104:0/1903190312 >> 172.21.15.104:0/1903190312 conn(0x7f297406b8a0 msgr2=0x7f29740cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 -- 172.21.15.104:0/1903190312 shutdown_connections 2023-06-10T18:58:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.481+0000 7f297a8b2700 1 -- 172.21.15.104:0/1903190312 wait complete. 2023-06-10T18:58:32.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.482+0000 7f297a8b2700 1 Processor -- start 2023-06-10T18:58:32.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.482+0000 7f297a8b2700 1 -- start start 2023-06-10T18:58:32.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 0x7f29740cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f297406e0e0 0x7f2974148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2974149010 con 0x7f297406e4b0 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29740b1120 con 0x7f2974069e70 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f297a8b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29740b1400 con 0x7f297406e0e0 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40372/0 (socket says 172.21.15.104:40372) 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 -- 172.21.15.104:0/684513210 learned_addr learned my addr 172.21.15.104:0/684513210 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2973fff700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 0x7f29740cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 -- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f297406e0e0 msgr2=0x7f2974148590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f297406e0e0 0x7f2974148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 -- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 msgr2=0x7f29740cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 0x7f29740cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 -- 172.21.15.104:0/684513210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c009b30 con 0x7f297406e4b0 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2973fff700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 0x7f29740cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.483+0000 7f2978e4f700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f296c015040 tx=0x7f296c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c004440 con 0x7f297406e4b0 2023-06-10T18:58:32.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c0045c0 con 0x7f297406e4b0 2023-06-10T18:58:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29740b16e0 con 0x7f297406e4b0 2023-06-10T18:58:32.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c004970 con 0x7f297406e4b0 2023-06-10T18:58:32.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29740b1ca0 con 0x7f297406e4b0 2023-06-10T18:58:32.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.484+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2974074050 con 0x7f297406e4b0 2023-06-10T18:58:32.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.485+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f296c012430 con 0x7f297406e4b0 2023-06-10T18:58:32.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.485+0000 7f2970ff9700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 0x7f296006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.486+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f296c08ab50 con 0x7f297406e4b0 2023-06-10T18:58:32.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.486+0000 7f2973fff700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 0x7f296006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.486+0000 7f2973fff700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 0x7f296006d1c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2964006fd0 tx=0x7f2964008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.490+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f296c055ad0 con 0x7f297406e4b0 2023-06-10T18:58:32.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.717+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29740723b0 con 0x7f297406e4b0 2023-06-10T18:58:32.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.717+0000 7f2970ff9700 1 -- 172.21.15.104:0/684513210 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f296c0590f0 con 0x7f297406e4b0 2023-06-10T18:58:32.724 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:32.724 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:58:32.725 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 msgr2=0x7f296006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 0x7f296006d1c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2964006fd0 tx=0x7f2964008040 comp rx=0 tx=0).stop 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 msgr2=0x7f2974148ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f296c015040 tx=0x7f296c004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 shutdown_connections 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f296006ad10 0x7f296006d1c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.726 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2974069e70 0x7f29740cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f297406e4b0 0x7f2974148ad0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 --2- 172.21.15.104:0/684513210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f297406e0e0 0x7f2974148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.727 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 >> 172.21.15.104:0/684513210 conn(0x7f297406b8a0 msgr2=0x7f2974072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:32.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 shutdown_connections 2023-06-10T18:58:32.728 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.719+0000 7f297a8b2700 1 -- 172.21.15.104:0/684513210 wait complete. 2023-06-10T18:58:32.728 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:58:32.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:32.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.857+0000 7f2908dda700 1 Processor -- start 2023-06-10T18:58:32.868 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.857+0000 7f2908dda700 1 -- start start 2023-06-10T18:58:32.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.858+0000 7f2908dda700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29040b4ff0 0x7f29040ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.858+0000 7f2908dda700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29040ce890 0x7f290415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.858+0000 7f2908dda700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29040cecf0 con 0x7f290415a420 2023-06-10T18:58:32.870 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.858+0000 7f2908dda700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29040b6070 con 0x7f29040b4ff0 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.858+0000 7f2908dda700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040b1d70 con 0x7f29040ce890 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29040b4ff0 0x7f29040ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f290415a420 0x7f29040b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38204/0 (socket says 172.21.15.104:38204) 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290259c700 1 -- 172.21.15.104:0/1541874601 learned_addr learned my addr 172.21.15.104:0/1541874601 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 -- 172.21.15.104:0/1541874601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29040ce890 msgr2=0x7f290415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 --2- 172.21.15.104:0/1541874601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29040ce890 0x7f290415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 -- 172.21.15.104:0/1541874601 >> v1:172.21.15.104:6789/0 conn(0x7f290415a420 legacy=0x7f29040b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 -- 172.21.15.104:0/1541874601 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29040b5460 con 0x7f29040b4ff0 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290159a700 1 -- 172.21.15.104:0/1541874601 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394751225 0 0) 0x7f29040cecf0 con 0x7f290415a420 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f2901d9b700 1 --2- 172.21.15.104:0/1541874601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040b4ff0 0x7f29040ce350 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009840 tx=0x7f28f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d307bc6df81423 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290159a700 1 -- 172.21.15.104:0/1541874601 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4011830 con 0x7f29040b4ff0 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290159a700 1 -- 172.21.15.104:0/1541874601 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4011990 con 0x7f29040b4ff0 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.859+0000 7f290159a700 1 -- 172.21.15.104:0/1541874601 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4011d20 con 0x7f29040b4ff0 2023-06-10T18:58:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 -- 172.21.15.104:0/1541874601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040b4ff0 msgr2=0x7f29040ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 --2- 172.21.15.104:0/1541874601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040b4ff0 0x7f29040ce350 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009840 tx=0x7f28f4009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 -- 172.21.15.104:0/1541874601 shutdown_connections 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 --2- 172.21.15.104:0/1541874601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29040ce890 0x7f290415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 --2- 172.21.15.104:0/1541874601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040b4ff0 0x7f29040ce350 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 -- 172.21.15.104:0/1541874601 >> 172.21.15.104:0/1541874601 conn(0x7f2904089fd0 msgr2=0x7f29040d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 -- 172.21.15.104:0/1541874601 shutdown_connections 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.860+0000 7f2908dda700 1 -- 172.21.15.104:0/1541874601 wait complete. 2023-06-10T18:58:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.861+0000 7f2908dda700 1 Processor -- start 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.861+0000 7f2908dda700 1 -- start start 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 0x7f290415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f290415a420 0x7f29040ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040c5310 con 0x7f29040ce890 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040c4700 con 0x7f29040b4ff0 2023-06-10T18:58:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040c49e0 con 0x7f290415a420 2023-06-10T18:58:32.875 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40384/0 (socket says 172.21.15.104:40384) 2023-06-10T18:58:32.876 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 -- 172.21.15.104:0/2446652546 learned_addr learned my addr 172.21.15.104:0/2446652546 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f290259c700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 0x7f290415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 -- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f290415a420 msgr2=0x7f29040ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f290415a420 0x7f29040ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 -- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 msgr2=0x7f290415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 0x7f290415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 -- 172.21.15.104:0/2446652546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40094a0 con 0x7f29040ce890 2023-06-10T18:58:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f290259c700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 0x7f290415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2901d9b700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f28f40090b0 tx=0x7f28f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40045f0 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f401f070 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29040c4cc0 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4003b00 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.862+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290415cf50 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.863+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29040c5480 con 0x7f29040ce890 2023-06-10T18:58:32.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.864+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f28f4003c60 con 0x7f29040ce890 2023-06-10T18:58:32.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.864+0000 7f28f2ffd700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 0x7f28e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:32.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.864+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f28f4021070 con 0x7f29040ce890 2023-06-10T18:58:32.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.865+0000 7f290259c700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 0x7f28e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:32.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.865+0000 7f290259c700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 0x7f28e406d190 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f28f8007fd0 tx=0x7f28f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:32.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:32.869+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f28f4056090 con 0x7f29040ce890 2023-06-10T18:58:33.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.056+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2904074050 con 0x7f29040ce890 2023-06-10T18:58:33.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.057+0000 7f28f2ffd700 1 -- 172.21.15.104:0/2446652546 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+18614 (secure 0 0 0) 0x7f28f40596b0 con 0x7f29040ce890 2023-06-10T18:58:33.065 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:33.065 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":172,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:30.309579+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:33.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 msgr2=0x7f28e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 0x7f28e406d190 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f28f8007fd0 tx=0x7f28f8009040 comp rx=0 tx=0).stop 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 msgr2=0x7f290415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f28f40090b0 tx=0x7f28f4004750 comp rx=0 tx=0).stop 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 shutdown_connections 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f28e406ace0 0x7f28e406d190 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29040b4ff0 0x7f290415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f290415a420 0x7f29040ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 --2- 172.21.15.104:0/2446652546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29040ce890 0x7f290415acf0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 >> 172.21.15.104:0/2446652546 conn(0x7f2904089fd0 msgr2=0x7f29040b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 shutdown_connections 2023-06-10T18:58:33.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.060+0000 7f2908dda700 1 -- 172.21.15.104:0/2446652546 wait complete. 2023-06-10T18:58:33.079 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:58:33.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:58:33.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.195+0000 7f71450ad700 1 Processor -- start 2023-06-10T18:58:33.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.195+0000 7f71450ad700 1 -- start start 2023-06-10T18:58:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f71450ad700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f714006e0e0 0x7f714006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f71450ad700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f71400c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f71450ad700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714006d290 con 0x7f71400ca100 2023-06-10T18:58:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f71450ad700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7140071640 con 0x7f714006e0e0 2023-06-10T18:58:33.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f71450ad700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400aef00 con 0x7f714006e4b0 2023-06-10T18:58:33.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713f59e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f71400ca100 0x7f71400ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38222/0 (socket says 172.21.15.104:38222) 2023-06-10T18:58:33.207 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713f59e700 1 -- 172.21.15.104:0/79397873 learned_addr learned my addr 172.21.15.104:0/79397873 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698686218 0 0) 0x7f714006d290 con 0x7f71400ca100 2023-06-10T18:58:33.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71280035b0 con 0x7f71400ca100 2023-06-10T18:58:33.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713ed9d700 1 --2- 172.21.15.104:0/79397873 >> v2:172.21.15.114:3300/0 conn(0x7f714006e0e0 0x7f714006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713e59c700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f71400c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412549383 0 0) 0x7f71280035b0 con 0x7f71400ca100 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 msgr2=0x7f71400c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f71400c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 >> v2:172.21.15.114:3300/0 conn(0x7f714006e0e0 msgr2=0x7f714006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 --2- 172.21.15.104:0/79397873 >> v2:172.21.15.114:3300/0 conn(0x7f714006e0e0 0x7f714006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714006d4c0 con 0x7f71400ca100 2023-06-10T18:58:33.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f71380019f0 con 0x7f71400ca100 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 >> v1:172.21.15.104:6789/0 conn(0x7f71400ca100 legacy=0x7f71400ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f7128003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7128004030 0x7f7128004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7128004d90 0x7f7128005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7140071640 con 0x7f7128004d90 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400aef00 con 0x7f7128004030 2023-06-10T18:58:33.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.196+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712800bba0 con 0x7f714006e4b0 2023-06-10T18:58:33.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7128004d90 0x7f7128005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713e59c700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7128004030 0x7f7128004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713ed9d700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f7128003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 -- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 msgr2=0x7f7128003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714006e4b0 0x7f7128003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 -- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7128004030 msgr2=0x7f7128004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7128004030 0x7f7128004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 -- 172.21.15.104:0/79397873 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280035b0 con 0x7f7128004d90 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713f59e700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7128004d90 0x7f7128005200 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f713800a830 tx=0x7f7128005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a6a586eae0a547 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138012600 con 0x7f7128004d90 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138012760 con 0x7f7128004d90 2023-06-10T18:58:33.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.197+0000 7f713dd9b700 1 -- 172.21.15.104:0/79397873 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138012b10 con 0x7f7128004d90 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 -- 172.21.15.104:0/79397873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7128004d90 msgr2=0x7f7128005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 --2- 172.21.15.104:0/79397873 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7128004d90 0x7f7128005200 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f713800a830 tx=0x7f7128005880 comp rx=0 tx=0).stop 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f713f59e700 1 -- 172.21.15.104:0/79397873 reap_dead start 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 -- 172.21.15.104:0/79397873 shutdown_connections 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 -- 172.21.15.104:0/79397873 >> 172.21.15.104:0/79397873 conn(0x7f714006b8a0 msgr2=0x7f71400cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 -- 172.21.15.104:0/79397873 shutdown_connections 2023-06-10T18:58:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.198+0000 7f71450ad700 1 -- 172.21.15.104:0/79397873 wait complete. 2023-06-10T18:58:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.199+0000 7f71450ad700 1 Processor -- start 2023-06-10T18:58:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.199+0000 7f71450ad700 1 -- start start 2023-06-10T18:58:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 0x7f71400c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71400c8e80 0x7f7140158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714014fa20 con 0x7f714014fbc0 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714015a8d0 con 0x7f71400c8e80 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f71450ad700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400c7170 con 0x7f714014f650 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40404/0 (socket says 172.21.15.104:40404) 2023-06-10T18:58:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 -- 172.21.15.104:0/3407833526 learned_addr learned my addr 172.21.15.104:0/3407833526 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713f59e700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71400c8e80 0x7f7140158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713ed9d700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 0x7f71400c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 -- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 msgr2=0x7f71400c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 0x7f71400c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 -- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71400c8e80 msgr2=0x7f7140158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71400c8e80 0x7f7140158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 -- 172.21.15.104:0/3407833526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400afb40 con 0x7f714014fbc0 2023-06-10T18:58:33.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713ed9d700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 0x7f71400c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.200+0000 7f713e59c700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f71300116f0 tx=0x7f7130011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.201+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000c070 con 0x7f714014fbc0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.201+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130015870 con 0x7f714014fbc0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.201+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71400afe20 con 0x7f714014fbc0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.201+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130015c20 con 0x7f714014fbc0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.201+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71400c7450 con 0x7f714014fbc0 2023-06-10T18:58:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.202+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7140067760 con 0x7f714014fbc0 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.202+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7130013400 con 0x7f714014fbc0 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.202+0000 7f71277fe700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 0x7f712c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.203+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f71300897c0 con 0x7f714014fbc0 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.203+0000 7f713ed9d700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 0x7f712c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.203+0000 7f713ed9d700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 0x7f712c06d190 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.207+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7130054730 con 0x7f714014fbc0 2023-06-10T18:58:33.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.392+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f714006da80 con 0x7f712c06ace0 2023-06-10T18:58:33.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.401+0000 7f71277fe700 1 -- 172.21.15.104:0/3407833526 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311839 (secure 0 0 0) 0x7f7134009e20 con 0x7f712c06ace0 2023-06-10T18:58:33.410 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:33.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 msgr2=0x7f712c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 0x7f712c06d190 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134008040 comp rx=0 tx=0).stop 2023-06-10T18:58:33.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 msgr2=0x7f71400c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f71300116f0 tx=0x7f7130011ab0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 shutdown_connections 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f712c06ace0 0x7f712c06d190 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71400c8e80 0x7f7140158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f714014f650 0x7f71400c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 --2- 172.21.15.104:0/3407833526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f714014fbc0 0x7f71400c8940 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 >> 172.21.15.104:0/3407833526 conn(0x7f714006b8a0 msgr2=0x7f71400ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 shutdown_connections 2023-06-10T18:58:33.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.404+0000 7f71450ad700 1 -- 172.21.15.104:0/3407833526 wait complete. 2023-06-10T18:58:33.414 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:58:33.424 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2023-06-10T18:58:31.578840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266432,"kb_used_data":18048,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754708288,"statfs":{"total":773094113280,"available":772821286912,"internally_reserved":0,"allocated":18481152,"data_stored":9655742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000798"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034725+0000","last_change":"2023-06-10T18:58:29.338467+0000","last_active":"2023-06-10T18:58:31.034725+0000","last_peered":"2023-06-10T18:58:31.034725+0000","last_clean":"2023-06-10T18:58:31.034725+0000","last_became_active":"2023-06-10T18:58:29.338200+0000","last_became_peered":"2023-06-10T18:58:29.338200+0000","last_unstale":"2023-06-10T18:58:31.034725+0000","last_undegraded":"2023-06-10T18:58:31.034725+0000","last_fullsized":"2023-06-10T18:58:31.034725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:52:01.917085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034033+0000","last_change":"2023-06-10T18:58:29.337738+0000","last_active":"2023-06-10T18:58:31.034033+0000","last_peered":"2023-06-10T18:58:31.034033+0000","last_clean":"2023-06-10T18:58:31.034033+0000","last_became_active":"2023-06-10T18:58:29.337457+0000","last_became_peered":"2023-06-10T18:58:29.337457+0000","last_unstale":"2023-06-10T18:58:31.034033+0000","last_undegraded":"2023-06-10T18:58:31.034033+0000","last_fullsized":"2023-06-10T18:58:31.034033+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:33.687163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033647+0000","last_change":"2023-06-10T18:58:29.323437+0000","last_active":"2023-06-10T18:58:31.033647+0000","last_peered":"2023-06-10T18:58:31.033647+0000","last_clean":"2023-06-10T18:58:31.033647+0000","last_became_active":"2023-06-10T18:58:29.323202+0000","last_became_peered":"2023-06-10T18:58:29.323202+0000","last_unstale":"2023-06-10T18:58:31.033647+0000","last_undegraded":"2023-06-10T18:58:31.033647+0000","last_fullsized":"2023-06-10T18:58:31.033647+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:32:59.801173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034351+0000","last_change":"2023-06-10T18:58:29.337452+0000","last_active":"2023-06-10T18:58:31.034351+0000","last_peered":"2023-06-10T18:58:31.034351+0000","last_clean":"2023-06-10T18:58:31.034351+0000","last_became_active":"2023-06-10T18:58:29.337181+0000","last_became_peered":"2023-06-10T18:58:29.337181+0000","last_unstale":"2023-06-10T18:58:31.034351+0000","last_undegraded":"2023-06-10T18:58:31.034351+0000","last_fullsized":"2023-06-10T18:58:31.034351+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:48:42.173586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032905+0000","last_change":"2023-06-10T18:58:29.332301+0000","last_active":"2023-06-10T18:58:31.032905+0000","last_peered":"2023-06-10T18:58:31.032905+0000","last_clean":"2023-06-10T18:58:31.032905+0000","last_became_active":"2023-06-10T18:58:29.332064+0000","last_became_peered":"2023-06-10T18:58:29.332064+0000","last_unstale":"2023-06-10T18:58:31.032905+0000","last_undegraded":"2023-06-10T18:58:31.032905+0000","last_fullsized":"2023-06-10T18:58:31.032905+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:52.532995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033041+0000","last_change":"2023-06-10T18:58:29.330081+0000","last_active":"2023-06-10T18:58:31.033041+0000","last_peered":"2023-06-10T18:58:31.033041+0000","last_clean":"2023-06-10T18:58:31.033041+0000","last_became_active":"2023-06-10T18:58:29.329825+0000","last_became_peered":"2023-06-10T18:58:29.329825+0000","last_unstale":"2023-06-10T18:58:31.033041+0000","last_undegraded":"2023-06-10T18:58:31.033041+0000","last_fullsized":"2023-06-10T18:58:31.033041+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:27.154386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326157+0000","last_change":"2023-06-10T18:58:29.338369+0000","last_active":"2023-06-10T18:58:30.326157+0000","last_peered":"2023-06-10T18:58:30.326157+0000","last_clean":"2023-06-10T18:58:30.326157+0000","last_became_active":"2023-06-10T18:58:29.337617+0000","last_became_peered":"2023-06-10T18:58:29.337617+0000","last_unstale":"2023-06-10T18:58:30.326157+0000","last_undegraded":"2023-06-10T18:58:30.326157+0000","last_fullsized":"2023-06-10T18:58:30.326157+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:27.421848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319811+0000","last_change":"2023-06-10T18:58:29.333536+0000","last_active":"2023-06-10T18:58:30.319811+0000","last_peered":"2023-06-10T18:58:30.319811+0000","last_clean":"2023-06-10T18:58:30.319811+0000","last_became_active":"2023-06-10T18:58:29.333303+0000","last_became_peered":"2023-06-10T18:58:29.333303+0000","last_unstale":"2023-06-10T18:58:30.319811+0000","last_undegraded":"2023-06-10T18:58:30.319811+0000","last_fullsized":"2023-06-10T18:58:30.319811+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:03:48.813988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325988+0000","last_change":"2023-06-10T18:58:29.326844+0000","last_active":"2023-06-10T18:58:30.325988+0000","last_peered":"2023-06-10T18:58:30.325988+0000","last_clean":"2023-06-10T18:58:30.325988+0000","last_became_active":"2023-06-10T18:58:29.326564+0000","last_became_peered":"2023-06-10T18:58:29.326564+0000","last_unstale":"2023-06-10T18:58:30.325988+0000","last_undegraded":"2023-06-10T18:58:30.325988+0000","last_fullsized":"2023-06-10T18:58:30.325988+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:39.177180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941459+0000","last_change":"2023-06-10T18:58:29.333335+0000","last_active":"2023-06-10T18:58:30.941459+0000","last_peered":"2023-06-10T18:58:30.941459+0000","last_clean":"2023-06-10T18:58:30.941459+0000","last_became_active":"2023-06-10T18:58:29.333067+0000","last_became_peered":"2023-06-10T18:58:29.333067+0000","last_unstale":"2023-06-10T18:58:30.941459+0000","last_undegraded":"2023-06-10T18:58:30.941459+0000","last_fullsized":"2023-06-10T18:58:30.941459+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:18:15.909594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326749+0000","last_change":"2023-06-10T18:58:29.335540+0000","last_active":"2023-06-10T18:58:30.326749+0000","last_peered":"2023-06-10T18:58:30.326749+0000","last_clean":"2023-06-10T18:58:30.326749+0000","last_became_active":"2023-06-10T18:58:29.335247+0000","last_became_peered":"2023-06-10T18:58:29.335247+0000","last_unstale":"2023-06-10T18:58:30.326749+0000","last_undegraded":"2023-06-10T18:58:30.326749+0000","last_fullsized":"2023-06-10T18:58:30.326749+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:06.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034114+0000","last_change":"2023-06-10T18:58:29.335587+0000","last_active":"2023-06-10T18:58:31.034114+0000","last_peered":"2023-06-10T18:58:31.034114+0000","last_clean":"2023-06-10T18:58:31.034114+0000","last_became_active":"2023-06-10T18:58:29.335235+0000","last_became_peered":"2023-06-10T18:58:29.335235+0000","last_unstale":"2023-06-10T18:58:31.034114+0000","last_undegraded":"2023-06-10T18:58:31.034114+0000","last_fullsized":"2023-06-10T18:58:31.034114+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:17:31.970779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033984+0000","last_change":"2023-06-10T18:58:29.332847+0000","last_active":"2023-06-10T18:58:31.033984+0000","last_peered":"2023-06-10T18:58:31.033984+0000","last_clean":"2023-06-10T18:58:31.033984+0000","last_became_active":"2023-06-10T18:58:29.332600+0000","last_became_peered":"2023-06-10T18:58:29.332600+0000","last_unstale":"2023-06-10T18:58:31.033984+0000","last_undegraded":"2023-06-10T18:58:31.033984+0000","last_fullsized":"2023-06-10T18:58:31.033984+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:45:13.186666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325662+0000","last_change":"2023-06-10T18:58:29.330986+0000","last_active":"2023-06-10T18:58:30.325662+0000","last_peered":"2023-06-10T18:58:30.325662+0000","last_clean":"2023-06-10T18:58:30.325662+0000","last_became_active":"2023-06-10T18:58:29.330604+0000","last_became_peered":"2023-06-10T18:58:29.330604+0000","last_unstale":"2023-06-10T18:58:30.325662+0000","last_undegraded":"2023-06-10T18:58:30.325662+0000","last_fullsized":"2023-06-10T18:58:30.325662+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:19.397055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034404+0000","last_change":"2023-06-10T18:58:29.338141+0000","last_active":"2023-06-10T18:58:31.034404+0000","last_peered":"2023-06-10T18:58:31.034404+0000","last_clean":"2023-06-10T18:58:31.034404+0000","last_became_active":"2023-06-10T18:58:29.337876+0000","last_became_peered":"2023-06-10T18:58:29.337876+0000","last_unstale":"2023-06-10T18:58:31.034404+0000","last_undegraded":"2023-06-10T18:58:31.034404+0000","last_fullsized":"2023-06-10T18:58:31.034404+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:25.429290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320404+0000","last_change":"2023-06-10T18:58:29.322664+0000","last_active":"2023-06-10T18:58:30.320404+0000","last_peered":"2023-06-10T18:58:30.320404+0000","last_clean":"2023-06-10T18:58:30.320404+0000","last_became_active":"2023-06-10T18:58:29.322408+0000","last_became_peered":"2023-06-10T18:58:29.322408+0000","last_unstale":"2023-06-10T18:58:30.320404+0000","last_undegraded":"2023-06-10T18:58:30.320404+0000","last_fullsized":"2023-06-10T18:58:30.320404+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:34.331285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033584+0000","last_change":"2023-06-10T18:58:29.333305+0000","last_active":"2023-06-10T18:58:31.033584+0000","last_peered":"2023-06-10T18:58:31.033584+0000","last_clean":"2023-06-10T18:58:31.033584+0000","last_became_active":"2023-06-10T18:58:29.332773+0000","last_became_peered":"2023-06-10T18:58:29.332773+0000","last_unstale":"2023-06-10T18:58:31.033584+0000","last_undegraded":"2023-06-10T18:58:31.033584+0000","last_fullsized":"2023-06-10T18:58:31.033584+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:57:37.325886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034891+0000","last_change":"2023-06-10T18:58:29.336430+0000","last_active":"2023-06-10T18:58:31.034891+0000","last_peered":"2023-06-10T18:58:31.034891+0000","last_clean":"2023-06-10T18:58:31.034891+0000","last_became_active":"2023-06-10T18:58:29.334374+0000","last_became_peered":"2023-06-10T18:58:29.334374+0000","last_unstale":"2023-06-10T18:58:31.034891+0000","last_undegraded":"2023-06-10T18:58:31.034891+0000","last_fullsized":"2023-06-10T18:58:31.034891+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:13.541013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941270+0000","last_change":"2023-06-10T18:58:29.329251+0000","last_active":"2023-06-10T18:58:30.941270+0000","last_peered":"2023-06-10T18:58:30.941270+0000","last_clean":"2023-06-10T18:58:30.941270+0000","last_became_active":"2023-06-10T18:58:29.328923+0000","last_became_peered":"2023-06-10T18:58:29.328923+0000","last_unstale":"2023-06-10T18:58:30.941270+0000","last_undegraded":"2023-06-10T18:58:30.941270+0000","last_fullsized":"2023-06-10T18:58:30.941270+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:41:01.516484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033090+0000","last_change":"2023-06-10T18:58:29.330085+0000","last_active":"2023-06-10T18:58:31.033090+0000","last_peered":"2023-06-10T18:58:31.033090+0000","last_clean":"2023-06-10T18:58:31.033090+0000","last_became_active":"2023-06-10T18:58:29.329839+0000","last_became_peered":"2023-06-10T18:58:29.329839+0000","last_unstale":"2023-06-10T18:58:31.033090+0000","last_undegraded":"2023-06-10T18:58:31.033090+0000","last_fullsized":"2023-06-10T18:58:31.033090+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:15.805525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033042+0000","last_change":"2023-06-10T18:58:29.331865+0000","last_active":"2023-06-10T18:58:31.033042+0000","last_peered":"2023-06-10T18:58:31.033042+0000","last_clean":"2023-06-10T18:58:31.033042+0000","last_became_active":"2023-06-10T18:58:29.331590+0000","last_became_peered":"2023-06-10T18:58:29.331590+0000","last_unstale":"2023-06-10T18:58:31.033042+0000","last_undegraded":"2023-06-10T18:58:31.033042+0000","last_fullsized":"2023-06-10T18:58:31.033042+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:44:33.883497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941155+0000","last_change":"2023-06-10T18:58:29.330153+0000","last_active":"2023-06-10T18:58:30.941155+0000","last_peered":"2023-06-10T18:58:30.941155+0000","last_clean":"2023-06-10T18:58:30.941155+0000","last_became_active":"2023-06-10T18:58:29.329721+0000","last_became_peered":"2023-06-10T18:58:29.329721+0000","last_unstale":"2023-06-10T18:58:30.941155+0000","last_undegraded":"2023-06-10T18:58:30.941155+0000","last_fullsized":"2023-06-10T18:58:30.941155+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:06.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033306+0000","last_change":"2023-06-10T18:58:29.328205+0000","last_active":"2023-06-10T18:58:31.033306+0000","last_peered":"2023-06-10T18:58:31.033306+0000","last_clean":"2023-06-10T18:58:31.033306+0000","last_became_active":"2023-06-10T18:58:29.327913+0000","last_became_peered":"2023-06-10T18:58:29.327913+0000","last_unstale":"2023-06-10T18:58:31.033306+0000","last_undegraded":"2023-06-10T18:58:31.033306+0000","last_fullsized":"2023-06-10T18:58:31.033306+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:49.210752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320049+0000","last_change":"2023-06-10T18:58:29.321853+0000","last_active":"2023-06-10T18:58:30.320049+0000","last_peered":"2023-06-10T18:58:30.320049+0000","last_clean":"2023-06-10T18:58:30.320049+0000","last_became_active":"2023-06-10T18:58:29.320585+0000","last_became_peered":"2023-06-10T18:58:29.320585+0000","last_unstale":"2023-06-10T18:58:30.320049+0000","last_undegraded":"2023-06-10T18:58:30.320049+0000","last_fullsized":"2023-06-10T18:58:30.320049+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:40:58.370826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034187+0000","last_change":"2023-06-10T18:58:29.338307+0000","last_active":"2023-06-10T18:58:31.034187+0000","last_peered":"2023-06-10T18:58:31.034187+0000","last_clean":"2023-06-10T18:58:31.034187+0000","last_became_active":"2023-06-10T18:58:29.338028+0000","last_became_peered":"2023-06-10T18:58:29.338028+0000","last_unstale":"2023-06-10T18:58:31.034187+0000","last_undegraded":"2023-06-10T18:58:31.034187+0000","last_fullsized":"2023-06-10T18:58:31.034187+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:59:38.236995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326394+0000","last_change":"2023-06-10T18:58:29.335394+0000","last_active":"2023-06-10T18:58:30.326394+0000","last_peered":"2023-06-10T18:58:30.326394+0000","last_clean":"2023-06-10T18:58:30.326394+0000","last_became_active":"2023-06-10T18:58:29.335106+0000","last_became_peered":"2023-06-10T18:58:29.335106+0000","last_unstale":"2023-06-10T18:58:30.326394+0000","last_undegraded":"2023-06-10T18:58:30.326394+0000","last_fullsized":"2023-06-10T18:58:30.326394+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:04.818195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033770+0000","last_change":"2023-06-10T18:58:29.330235+0000","last_active":"2023-06-10T18:58:31.033770+0000","last_peered":"2023-06-10T18:58:31.033770+0000","last_clean":"2023-06-10T18:58:31.033770+0000","last_became_active":"2023-06-10T18:58:29.329818+0000","last_became_peered":"2023-06-10T18:58:29.329818+0000","last_unstale":"2023-06-10T18:58:31.033770+0000","last_undegraded":"2023-06-10T18:58:31.033770+0000","last_fullsized":"2023-06-10T18:58:31.033770+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:36.894140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033223+0000","last_change":"2023-06-10T18:58:29.337668+0000","last_active":"2023-06-10T18:58:31.033223+0000","last_peered":"2023-06-10T18:58:31.033223+0000","last_clean":"2023-06-10T18:58:31.033223+0000","last_became_active":"2023-06-10T18:58:29.337399+0000","last_became_peered":"2023-06-10T18:58:29.337399+0000","last_unstale":"2023-06-10T18:58:31.033223+0000","last_undegraded":"2023-06-10T18:58:31.033223+0000","last_fullsized":"2023-06-10T18:58:31.033223+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:35.151278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034162+0000","last_change":"2023-06-10T18:58:29.335888+0000","last_active":"2023-06-10T18:58:31.034162+0000","last_peered":"2023-06-10T18:58:31.034162+0000","last_clean":"2023-06-10T18:58:31.034162+0000","last_became_active":"2023-06-10T18:58:29.335581+0000","last_became_peered":"2023-06-10T18:58:29.335581+0000","last_unstale":"2023-06-10T18:58:31.034162+0000","last_undegraded":"2023-06-10T18:58:31.034162+0000","last_fullsized":"2023-06-10T18:58:31.034162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:10:22.985514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325450+0000","last_change":"2023-06-10T18:58:29.329772+0000","last_active":"2023-06-10T18:58:30.325450+0000","last_peered":"2023-06-10T18:58:30.325450+0000","last_clean":"2023-06-10T18:58:30.325450+0000","last_became_active":"2023-06-10T18:58:29.329467+0000","last_became_peered":"2023-06-10T18:58:29.329467+0000","last_unstale":"2023-06-10T18:58:30.325450+0000","last_undegraded":"2023-06-10T18:58:30.325450+0000","last_fullsized":"2023-06-10T18:58:30.325450+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:33:36.805448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033422+0000","last_change":"2023-06-10T18:58:29.328763+0000","last_active":"2023-06-10T18:58:31.033422+0000","last_peered":"2023-06-10T18:58:31.033422+0000","last_clean":"2023-06-10T18:58:31.033422+0000","last_became_active":"2023-06-10T18:58:29.328510+0000","last_became_peered":"2023-06-10T18:58:29.328510+0000","last_unstale":"2023-06-10T18:58:31.033422+0000","last_undegraded":"2023-06-10T18:58:31.033422+0000","last_fullsized":"2023-06-10T18:58:31.033422+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:10.729171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032819+0000","last_change":"2023-06-10T18:58:29.332110+0000","last_active":"2023-06-10T18:58:31.032819+0000","last_peered":"2023-06-10T18:58:31.032819+0000","last_clean":"2023-06-10T18:58:31.032819+0000","last_became_active":"2023-06-10T18:58:29.331850+0000","last_became_peered":"2023-06-10T18:58:29.331850+0000","last_unstale":"2023-06-10T18:58:31.032819+0000","last_undegraded":"2023-06-10T18:58:31.032819+0000","last_fullsized":"2023-06-10T18:58:31.032819+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:23:37.808457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034233+0000","last_change":"2023-06-10T18:58:29.320564+0000","last_active":"2023-06-10T18:58:31.034233+0000","last_peered":"2023-06-10T18:58:31.034233+0000","last_clean":"2023-06-10T18:58:31.034233+0000","last_became_active":"2023-06-10T18:58:29.320312+0000","last_became_peered":"2023-06-10T18:58:29.320312+0000","last_unstale":"2023-06-10T18:58:31.034233+0000","last_undegraded":"2023-06-10T18:58:31.034233+0000","last_fullsized":"2023-06-10T18:58:31.034233+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:39.177180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326615+0000","last_change":"2023-06-10T18:58:28.927638+0000","last_active":"2023-06-10T18:58:30.326615+0000","last_peered":"2023-06-10T18:58:30.326615+0000","last_clean":"2023-06-10T18:58:30.326615+0000","last_became_active":"2023-06-10T18:58:28.927421+0000","last_became_peered":"2023-06-10T18:58:28.927421+0000","last_unstale":"2023-06-10T18:58:30.326615+0000","last_undegraded":"2023-06-10T18:58:30.326615+0000","last_fullsized":"2023-06-10T18:58:30.326615+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:29.887872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319939+0000","last_change":"2023-06-10T18:58:29.336412+0000","last_active":"2023-06-10T18:58:30.319939+0000","last_peered":"2023-06-10T18:58:30.319939+0000","last_clean":"2023-06-10T18:58:30.319939+0000","last_became_active":"2023-06-10T18:58:29.336166+0000","last_became_peered":"2023-06-10T18:58:29.336166+0000","last_unstale":"2023-06-10T18:58:30.319939+0000","last_undegraded":"2023-06-10T18:58:30.319939+0000","last_fullsized":"2023-06-10T18:58:30.319939+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:42.726360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033243+0000","last_change":"2023-06-10T18:58:28.932065+0000","last_active":"2023-06-10T18:58:31.033243+0000","last_peered":"2023-06-10T18:58:31.033243+0000","last_clean":"2023-06-10T18:58:31.033243+0000","last_became_active":"2023-06-10T18:58:28.931859+0000","last_became_peered":"2023-06-10T18:58:28.931859+0000","last_unstale":"2023-06-10T18:58:31.033243+0000","last_undegraded":"2023-06-10T18:58:31.033243+0000","last_fullsized":"2023-06-10T18:58:31.033243+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:39:00.868092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326557+0000","last_change":"2023-06-10T18:58:29.333048+0000","last_active":"2023-06-10T18:58:30.326557+0000","last_peered":"2023-06-10T18:58:30.326557+0000","last_clean":"2023-06-10T18:58:30.326557+0000","last_became_active":"2023-06-10T18:58:29.332769+0000","last_became_peered":"2023-06-10T18:58:29.332769+0000","last_unstale":"2023-06-10T18:58:30.326557+0000","last_undegraded":"2023-06-10T18:58:30.326557+0000","last_fullsized":"2023-06-10T18:58:30.326557+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:46.331984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"171'3","reported_seq":16,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032991+0000","last_change":"2023-06-10T18:58:28.929567+0000","last_active":"2023-06-10T18:58:31.032991+0000","last_peered":"2023-06-10T18:58:31.032991+0000","last_clean":"2023-06-10T18:58:31.032991+0000","last_became_active":"2023-06-10T18:58:28.929303+0000","last_became_peered":"2023-06-10T18:58:28.929303+0000","last_unstale":"2023-06-10T18:58:31.032991+0000","last_undegraded":"2023-06-10T18:58:31.032991+0000","last_fullsized":"2023-06-10T18:58:31.032991+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:46:55.701041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941378+0000","last_change":"2023-06-10T18:58:29.332724+0000","last_active":"2023-06-10T18:58:30.941378+0000","last_peered":"2023-06-10T18:58:30.941378+0000","last_clean":"2023-06-10T18:58:30.941378+0000","last_became_active":"2023-06-10T18:58:29.327203+0000","last_became_peered":"2023-06-10T18:58:29.327203+0000","last_unstale":"2023-06-10T18:58:30.941378+0000","last_undegraded":"2023-06-10T18:58:30.941378+0000","last_fullsized":"2023-06-10T18:58:30.941378+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:56.636502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"171'6","reported_seq":19,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033219+0000","last_change":"2023-06-10T18:58:28.925504+0000","last_active":"2023-06-10T18:58:31.033219+0000","last_peered":"2023-06-10T18:58:31.033219+0000","last_clean":"2023-06-10T18:58:31.033219+0000","last_became_active":"2023-06-10T18:58:28.925118+0000","last_became_peered":"2023-06-10T18:58:28.925118+0000","last_unstale":"2023-06-10T18:58:31.033219+0000","last_undegraded":"2023-06-10T18:58:31.033219+0000","last_fullsized":"2023-06-10T18:58:31.033219+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:04.555847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033177+0000","last_change":"2023-06-10T18:58:29.336138+0000","last_active":"2023-06-10T18:58:31.033177+0000","last_peered":"2023-06-10T18:58:31.033177+0000","last_clean":"2023-06-10T18:58:31.033177+0000","last_became_active":"2023-06-10T18:58:29.335874+0000","last_became_peered":"2023-06-10T18:58:29.335874+0000","last_unstale":"2023-06-10T18:58:31.033177+0000","last_undegraded":"2023-06-10T18:58:31.033177+0000","last_fullsized":"2023-06-10T18:58:31.033177+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:57.905042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034523+0000","last_change":"2023-06-10T18:58:28.930171+0000","last_active":"2023-06-10T18:58:31.034523+0000","last_peered":"2023-06-10T18:58:31.034523+0000","last_clean":"2023-06-10T18:58:31.034523+0000","last_became_active":"2023-06-10T18:58:28.929795+0000","last_became_peered":"2023-06-10T18:58:28.929795+0000","last_unstale":"2023-06-10T18:58:31.034523+0000","last_undegraded":"2023-06-10T18:58:31.034523+0000","last_fullsized":"2023-06-10T18:58:31.034523+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:00.845042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326421+0000","last_change":"2023-06-10T18:58:29.338140+0000","last_active":"2023-06-10T18:58:30.326421+0000","last_peered":"2023-06-10T18:58:30.326421+0000","last_clean":"2023-06-10T18:58:30.326421+0000","last_became_active":"2023-06-10T18:58:29.337640+0000","last_became_peered":"2023-06-10T18:58:29.337640+0000","last_unstale":"2023-06-10T18:58:30.326421+0000","last_undegraded":"2023-06-10T18:58:30.326421+0000","last_fullsized":"2023-06-10T18:58:30.326421+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:06:20.472351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941340+0000","last_change":"2023-06-10T18:58:28.925333+0000","last_active":"2023-06-10T18:58:30.941340+0000","last_peered":"2023-06-10T18:58:30.941340+0000","last_clean":"2023-06-10T18:58:30.941340+0000","last_became_active":"2023-06-10T18:58:28.925070+0000","last_became_peered":"2023-06-10T18:58:28.925070+0000","last_unstale":"2023-06-10T18:58:30.941340+0000","last_undegraded":"2023-06-10T18:58:30.941340+0000","last_fullsized":"2023-06-10T18:58:30.941340+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:09.477953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325336+0000","last_change":"2023-06-10T18:58:29.328323+0000","last_active":"2023-06-10T18:58:30.325336+0000","last_peered":"2023-06-10T18:58:30.325336+0000","last_clean":"2023-06-10T18:58:30.325336+0000","last_became_active":"2023-06-10T18:58:29.328043+0000","last_became_peered":"2023-06-10T18:58:29.328043+0000","last_unstale":"2023-06-10T18:58:30.325336+0000","last_undegraded":"2023-06-10T18:58:30.325336+0000","last_fullsized":"2023-06-10T18:58:30.325336+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:57.905042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319978+0000","last_change":"2023-06-10T18:58:28.926858+0000","last_active":"2023-06-10T18:58:30.319978+0000","last_peered":"2023-06-10T18:58:30.319978+0000","last_clean":"2023-06-10T18:58:30.319978+0000","last_became_active":"2023-06-10T18:58:28.926622+0000","last_became_peered":"2023-06-10T18:58:28.926622+0000","last_unstale":"2023-06-10T18:58:30.319978+0000","last_undegraded":"2023-06-10T18:58:30.319978+0000","last_fullsized":"2023-06-10T18:58:30.319978+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:26:58.521384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941119+0000","last_change":"2023-06-10T18:58:29.336086+0000","last_active":"2023-06-10T18:58:30.941119+0000","last_peered":"2023-06-10T18:58:30.941119+0000","last_clean":"2023-06-10T18:58:30.941119+0000","last_became_active":"2023-06-10T18:58:29.335834+0000","last_became_peered":"2023-06-10T18:58:29.335834+0000","last_unstale":"2023-06-10T18:58:30.941119+0000","last_undegraded":"2023-06-10T18:58:30.941119+0000","last_fullsized":"2023-06-10T18:58:30.941119+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:30.894743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"171'8","reported_seq":21,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033901+0000","last_change":"2023-06-10T18:58:28.932267+0000","last_active":"2023-06-10T18:58:31.033901+0000","last_peered":"2023-06-10T18:58:31.033901+0000","last_clean":"2023-06-10T18:58:31.033901+0000","last_became_active":"2023-06-10T18:58:28.931885+0000","last_became_peered":"2023-06-10T18:58:28.931885+0000","last_unstale":"2023-06-10T18:58:31.033901+0000","last_undegraded":"2023-06-10T18:58:31.033901+0000","last_fullsized":"2023-06-10T18:58:31.033901+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:36:36.219963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941379+0000","last_change":"2023-06-10T18:58:29.332794+0000","last_active":"2023-06-10T18:58:30.941379+0000","last_peered":"2023-06-10T18:58:30.941379+0000","last_clean":"2023-06-10T18:58:30.941379+0000","last_became_active":"2023-06-10T18:58:29.332535+0000","last_became_peered":"2023-06-10T18:58:29.332535+0000","last_unstale":"2023-06-10T18:58:30.941379+0000","last_undegraded":"2023-06-10T18:58:30.941379+0000","last_fullsized":"2023-06-10T18:58:30.941379+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:55.664298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033713+0000","last_change":"2023-06-10T18:58:28.928078+0000","last_active":"2023-06-10T18:58:31.033713+0000","last_peered":"2023-06-10T18:58:31.033713+0000","last_clean":"2023-06-10T18:58:31.033713+0000","last_became_active":"2023-06-10T18:58:28.927805+0000","last_became_peered":"2023-06-10T18:58:28.927805+0000","last_unstale":"2023-06-10T18:58:31.033713+0000","last_undegraded":"2023-06-10T18:58:31.033713+0000","last_fullsized":"2023-06-10T18:58:31.033713+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:48.098992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326185+0000","last_change":"2023-06-10T18:58:29.337015+0000","last_active":"2023-06-10T18:58:30.326185+0000","last_peered":"2023-06-10T18:58:30.326185+0000","last_clean":"2023-06-10T18:58:30.326185+0000","last_became_active":"2023-06-10T18:58:29.336720+0000","last_became_peered":"2023-06-10T18:58:29.336720+0000","last_unstale":"2023-06-10T18:58:30.326185+0000","last_undegraded":"2023-06-10T18:58:30.326185+0000","last_fullsized":"2023-06-10T18:58:30.326185+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:54.912807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320235+0000","last_change":"2023-06-10T18:58:28.925424+0000","last_active":"2023-06-10T18:58:30.320235+0000","last_peered":"2023-06-10T18:58:30.320235+0000","last_clean":"2023-06-10T18:58:30.320235+0000","last_became_active":"2023-06-10T18:58:28.925177+0000","last_became_peered":"2023-06-10T18:58:28.925177+0000","last_unstale":"2023-06-10T18:58:30.320235+0000","last_undegraded":"2023-06-10T18:58:30.320235+0000","last_fullsized":"2023-06-10T18:58:30.320235+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:03.099551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325465+0000","last_change":"2023-06-10T18:58:29.329690+0000","last_active":"2023-06-10T18:58:30.325465+0000","last_peered":"2023-06-10T18:58:30.325465+0000","last_clean":"2023-06-10T18:58:30.325465+0000","last_became_active":"2023-06-10T18:58:29.329374+0000","last_became_peered":"2023-06-10T18:58:29.329374+0000","last_unstale":"2023-06-10T18:58:30.325465+0000","last_undegraded":"2023-06-10T18:58:30.325465+0000","last_fullsized":"2023-06-10T18:58:30.325465+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:49.210752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034892+0000","last_change":"2023-06-10T18:58:28.930054+0000","last_active":"2023-06-10T18:58:31.034892+0000","last_peered":"2023-06-10T18:58:31.034892+0000","last_clean":"2023-06-10T18:58:31.034892+0000","last_became_active":"2023-06-10T18:58:28.929768+0000","last_became_peered":"2023-06-10T18:58:28.929768+0000","last_unstale":"2023-06-10T18:58:31.034892+0000","last_undegraded":"2023-06-10T18:58:31.034892+0000","last_fullsized":"2023-06-10T18:58:31.034892+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:56:22.879300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033850+0000","last_change":"2023-06-10T18:58:29.330698+0000","last_active":"2023-06-10T18:58:31.033850+0000","last_peered":"2023-06-10T18:58:31.033850+0000","last_clean":"2023-06-10T18:58:31.033850+0000","last_became_active":"2023-06-10T18:58:29.330393+0000","last_became_peered":"2023-06-10T18:58:29.330393+0000","last_unstale":"2023-06-10T18:58:31.033850+0000","last_undegraded":"2023-06-10T18:58:31.033850+0000","last_fullsized":"2023-06-10T18:58:31.033850+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:37.201110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326798+0000","last_change":"2023-06-10T18:58:28.927198+0000","last_active":"2023-06-10T18:58:30.326798+0000","last_peered":"2023-06-10T18:58:30.326798+0000","last_clean":"2023-06-10T18:58:30.326798+0000","last_became_active":"2023-06-10T18:58:28.926933+0000","last_became_peered":"2023-06-10T18:58:28.926933+0000","last_unstale":"2023-06-10T18:58:30.326798+0000","last_undegraded":"2023-06-10T18:58:30.326798+0000","last_fullsized":"2023-06-10T18:58:30.326798+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:55.658148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941199+0000","last_change":"2023-06-10T18:58:29.335284+0000","last_active":"2023-06-10T18:58:30.941199+0000","last_peered":"2023-06-10T18:58:30.941199+0000","last_clean":"2023-06-10T18:58:30.941199+0000","last_became_active":"2023-06-10T18:58:29.334805+0000","last_became_peered":"2023-06-10T18:58:29.334805+0000","last_unstale":"2023-06-10T18:58:30.941199+0000","last_undegraded":"2023-06-10T18:58:30.941199+0000","last_fullsized":"2023-06-10T18:58:30.941199+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:56.665019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033485+0000","last_change":"2023-06-10T18:58:28.925764+0000","last_active":"2023-06-10T18:58:31.033485+0000","last_peered":"2023-06-10T18:58:31.033485+0000","last_clean":"2023-06-10T18:58:31.033485+0000","last_became_active":"2023-06-10T18:58:28.925498+0000","last_became_peered":"2023-06-10T18:58:28.925498+0000","last_unstale":"2023-06-10T18:58:31.033485+0000","last_undegraded":"2023-06-10T18:58:31.033485+0000","last_fullsized":"2023-06-10T18:58:31.033485+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:18.397233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326516+0000","last_change":"2023-06-10T18:58:29.331972+0000","last_active":"2023-06-10T18:58:30.326516+0000","last_peered":"2023-06-10T18:58:30.326516+0000","last_clean":"2023-06-10T18:58:30.326516+0000","last_became_active":"2023-06-10T18:58:29.331665+0000","last_became_peered":"2023-06-10T18:58:29.331665+0000","last_unstale":"2023-06-10T18:58:30.326516+0000","last_undegraded":"2023-06-10T18:58:30.326516+0000","last_fullsized":"2023-06-10T18:58:30.326516+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:31.345719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033315+0000","last_change":"2023-06-10T18:58:28.931781+0000","last_active":"2023-06-10T18:58:31.033315+0000","last_peered":"2023-06-10T18:58:31.033315+0000","last_clean":"2023-06-10T18:58:31.033315+0000","last_became_active":"2023-06-10T18:58:28.931545+0000","last_became_peered":"2023-06-10T18:58:28.931545+0000","last_unstale":"2023-06-10T18:58:31.033315+0000","last_undegraded":"2023-06-10T18:58:31.033315+0000","last_fullsized":"2023-06-10T18:58:31.033315+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:19:49.031474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033833+0000","last_change":"2023-06-10T18:58:29.337012+0000","last_active":"2023-06-10T18:58:31.033833+0000","last_peered":"2023-06-10T18:58:31.033833+0000","last_clean":"2023-06-10T18:58:31.033833+0000","last_became_active":"2023-06-10T18:58:29.336693+0000","last_became_peered":"2023-06-10T18:58:29.336693+0000","last_unstale":"2023-06-10T18:58:31.033833+0000","last_undegraded":"2023-06-10T18:58:31.033833+0000","last_fullsized":"2023-06-10T18:58:31.033833+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:44:46.684104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033438+0000","last_change":"2023-06-10T18:58:28.925774+0000","last_active":"2023-06-10T18:58:31.033438+0000","last_peered":"2023-06-10T18:58:31.033438+0000","last_clean":"2023-06-10T18:58:31.033438+0000","last_became_active":"2023-06-10T18:58:28.925095+0000","last_became_peered":"2023-06-10T18:58:28.925095+0000","last_unstale":"2023-06-10T18:58:31.033438+0000","last_undegraded":"2023-06-10T18:58:31.033438+0000","last_fullsized":"2023-06-10T18:58:31.033438+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:18.390184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032866+0000","last_change":"2023-06-10T18:58:29.331135+0000","last_active":"2023-06-10T18:58:31.032866+0000","last_peered":"2023-06-10T18:58:31.032866+0000","last_clean":"2023-06-10T18:58:31.032866+0000","last_became_active":"2023-06-10T18:58:29.330894+0000","last_became_peered":"2023-06-10T18:58:29.330894+0000","last_unstale":"2023-06-10T18:58:31.032866+0000","last_undegraded":"2023-06-10T18:58:31.032866+0000","last_fullsized":"2023-06-10T18:58:31.032866+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:47.288837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325384+0000","last_change":"2023-06-10T18:58:28.930099+0000","last_active":"2023-06-10T18:58:30.325384+0000","last_peered":"2023-06-10T18:58:30.325384+0000","last_clean":"2023-06-10T18:58:30.325384+0000","last_became_active":"2023-06-10T18:58:28.929828+0000","last_became_peered":"2023-06-10T18:58:28.929828+0000","last_unstale":"2023-06-10T18:58:30.325384+0000","last_undegraded":"2023-06-10T18:58:30.325384+0000","last_fullsized":"2023-06-10T18:58:30.325384+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:04.555847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033000+0000","last_change":"2023-06-10T18:58:29.331310+0000","last_active":"2023-06-10T18:58:31.033000+0000","last_peered":"2023-06-10T18:58:31.033000+0000","last_clean":"2023-06-10T18:58:31.033000+0000","last_became_active":"2023-06-10T18:58:29.331050+0000","last_became_peered":"2023-06-10T18:58:29.331050+0000","last_unstale":"2023-06-10T18:58:31.033000+0000","last_undegraded":"2023-06-10T18:58:31.033000+0000","last_fullsized":"2023-06-10T18:58:31.033000+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:54.552309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034902+0000","last_change":"2023-06-10T18:58:28.929910+0000","last_active":"2023-06-10T18:58:31.034902+0000","last_peered":"2023-06-10T18:58:31.034902+0000","last_clean":"2023-06-10T18:58:31.034902+0000","last_became_active":"2023-06-10T18:58:28.929616+0000","last_became_peered":"2023-06-10T18:58:28.929616+0000","last_unstale":"2023-06-10T18:58:31.034902+0000","last_undegraded":"2023-06-10T18:58:31.034902+0000","last_fullsized":"2023-06-10T18:58:31.034902+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:54.340547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941602+0000","last_change":"2023-06-10T18:58:29.334526+0000","last_active":"2023-06-10T18:58:30.941602+0000","last_peered":"2023-06-10T18:58:30.941602+0000","last_clean":"2023-06-10T18:58:30.941602+0000","last_became_active":"2023-06-10T18:58:29.334254+0000","last_became_peered":"2023-06-10T18:58:29.334254+0000","last_unstale":"2023-06-10T18:58:30.941602+0000","last_undegraded":"2023-06-10T18:58:30.941602+0000","last_fullsized":"2023-06-10T18:58:30.941602+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:46:57.571994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941513+0000","last_change":"2023-06-10T18:58:28.927876+0000","last_active":"2023-06-10T18:58:30.941513+0000","last_peered":"2023-06-10T18:58:30.941513+0000","last_clean":"2023-06-10T18:58:30.941513+0000","last_became_active":"2023-06-10T18:58:28.927626+0000","last_became_peered":"2023-06-10T18:58:28.927626+0000","last_unstale":"2023-06-10T18:58:30.941513+0000","last_undegraded":"2023-06-10T18:58:30.941513+0000","last_fullsized":"2023-06-10T18:58:30.941513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:46.683778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320230+0000","last_change":"2023-06-10T18:58:29.321139+0000","last_active":"2023-06-10T18:58:30.320230+0000","last_peered":"2023-06-10T18:58:30.320230+0000","last_clean":"2023-06-10T18:58:30.320230+0000","last_became_active":"2023-06-10T18:58:29.320894+0000","last_became_peered":"2023-06-10T18:58:29.320894+0000","last_unstale":"2023-06-10T18:58:30.320230+0000","last_undegraded":"2023-06-10T18:58:30.320230+0000","last_fullsized":"2023-06-10T18:58:30.320230+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:25.201509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941484+0000","last_change":"2023-06-10T18:58:28.925916+0000","last_active":"2023-06-10T18:58:30.941484+0000","last_peered":"2023-06-10T18:58:30.941484+0000","last_clean":"2023-06-10T18:58:30.941484+0000","last_became_active":"2023-06-10T18:58:28.925623+0000","last_became_peered":"2023-06-10T18:58:28.925623+0000","last_unstale":"2023-06-10T18:58:30.941484+0000","last_undegraded":"2023-06-10T18:58:30.941484+0000","last_fullsized":"2023-06-10T18:58:30.941484+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:18:00.270130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320137+0000","last_change":"2023-06-10T18:58:29.325738+0000","last_active":"2023-06-10T18:58:30.320137+0000","last_peered":"2023-06-10T18:58:30.320137+0000","last_clean":"2023-06-10T18:58:30.320137+0000","last_became_active":"2023-06-10T18:58:29.325474+0000","last_became_peered":"2023-06-10T18:58:29.325474+0000","last_unstale":"2023-06-10T18:58:30.320137+0000","last_undegraded":"2023-06-10T18:58:30.320137+0000","last_fullsized":"2023-06-10T18:58:30.320137+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:54:33.547054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320363+0000","last_change":"2023-06-10T18:58:28.925914+0000","last_active":"2023-06-10T18:58:30.320363+0000","last_peered":"2023-06-10T18:58:30.320363+0000","last_clean":"2023-06-10T18:58:30.320363+0000","last_became_active":"2023-06-10T18:58:28.925668+0000","last_became_peered":"2023-06-10T18:58:28.925668+0000","last_unstale":"2023-06-10T18:58:30.320363+0000","last_undegraded":"2023-06-10T18:58:30.320363+0000","last_fullsized":"2023-06-10T18:58:30.320363+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:12.959466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034586+0000","last_change":"2023-06-10T18:58:29.335655+0000","last_active":"2023-06-10T18:58:31.034586+0000","last_peered":"2023-06-10T18:58:31.034586+0000","last_clean":"2023-06-10T18:58:31.034586+0000","last_became_active":"2023-06-10T18:58:29.334341+0000","last_became_peered":"2023-06-10T18:58:29.334341+0000","last_unstale":"2023-06-10T18:58:31.034586+0000","last_undegraded":"2023-06-10T18:58:31.034586+0000","last_fullsized":"2023-06-10T18:58:31.034586+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:00:50.629285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325598+0000","last_change":"2023-06-10T18:58:28.926873+0000","last_active":"2023-06-10T18:58:30.325598+0000","last_peered":"2023-06-10T18:58:30.325598+0000","last_clean":"2023-06-10T18:58:30.325598+0000","last_became_active":"2023-06-10T18:58:28.926574+0000","last_became_peered":"2023-06-10T18:58:28.926574+0000","last_unstale":"2023-06-10T18:58:30.325598+0000","last_undegraded":"2023-06-10T18:58:30.325598+0000","last_fullsized":"2023-06-10T18:58:30.325598+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:09.722300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034682+0000","last_change":"2023-06-10T18:58:29.336989+0000","last_active":"2023-06-10T18:58:31.034682+0000","last_peered":"2023-06-10T18:58:31.034682+0000","last_clean":"2023-06-10T18:58:31.034682+0000","last_became_active":"2023-06-10T18:58:29.336719+0000","last_became_peered":"2023-06-10T18:58:29.336719+0000","last_unstale":"2023-06-10T18:58:31.034682+0000","last_undegraded":"2023-06-10T18:58:31.034682+0000","last_fullsized":"2023-06-10T18:58:31.034682+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:37:37.855621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325595+0000","last_change":"2023-06-10T18:58:28.925738+0000","last_active":"2023-06-10T18:58:30.325595+0000","last_peered":"2023-06-10T18:58:30.325595+0000","last_clean":"2023-06-10T18:58:30.325595+0000","last_became_active":"2023-06-10T18:58:28.925499+0000","last_became_peered":"2023-06-10T18:58:28.925499+0000","last_unstale":"2023-06-10T18:58:30.325595+0000","last_undegraded":"2023-06-10T18:58:30.325595+0000","last_fullsized":"2023-06-10T18:58:30.325595+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:42:47.934788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034074+0000","last_change":"2023-06-10T18:58:29.331690+0000","last_active":"2023-06-10T18:58:31.034074+0000","last_peered":"2023-06-10T18:58:31.034074+0000","last_clean":"2023-06-10T18:58:31.034074+0000","last_became_active":"2023-06-10T18:58:29.331421+0000","last_became_peered":"2023-06-10T18:58:29.331421+0000","last_unstale":"2023-06-10T18:58:31.034074+0000","last_undegraded":"2023-06-10T18:58:31.034074+0000","last_fullsized":"2023-06-10T18:58:31.034074+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:11:09.571556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034441+0000","last_change":"2023-06-10T18:58:28.924203+0000","last_active":"2023-06-10T18:58:31.034441+0000","last_peered":"2023-06-10T18:58:31.034441+0000","last_clean":"2023-06-10T18:58:31.034441+0000","last_became_active":"2023-06-10T18:58:28.923525+0000","last_became_peered":"2023-06-10T18:58:28.923525+0000","last_unstale":"2023-06-10T18:58:31.034441+0000","last_undegraded":"2023-06-10T18:58:31.034441+0000","last_fullsized":"2023-06-10T18:58:31.034441+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:33:56.336777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032871+0000","last_change":"2023-06-10T18:58:29.329662+0000","last_active":"2023-06-10T18:58:31.032871+0000","last_peered":"2023-06-10T18:58:31.032871+0000","last_clean":"2023-06-10T18:58:31.032871+0000","last_became_active":"2023-06-10T18:58:29.329402+0000","last_became_peered":"2023-06-10T18:58:29.329402+0000","last_unstale":"2023-06-10T18:58:31.032871+0000","last_undegraded":"2023-06-10T18:58:31.032871+0000","last_fullsized":"2023-06-10T18:58:31.032871+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:34:55.202966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033264+0000","last_change":"2023-06-10T18:58:28.925396+0000","last_active":"2023-06-10T18:58:31.033264+0000","last_peered":"2023-06-10T18:58:31.033264+0000","last_clean":"2023-06-10T18:58:31.033264+0000","last_became_active":"2023-06-10T18:58:28.925033+0000","last_became_peered":"2023-06-10T18:58:28.925033+0000","last_unstale":"2023-06-10T18:58:31.033264+0000","last_undegraded":"2023-06-10T18:58:31.033264+0000","last_fullsized":"2023-06-10T18:58:31.033264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:33:35.798577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941574+0000","last_change":"2023-06-10T18:58:29.335959+0000","last_active":"2023-06-10T18:58:30.941574+0000","last_peered":"2023-06-10T18:58:30.941574+0000","last_clean":"2023-06-10T18:58:30.941574+0000","last_became_active":"2023-06-10T18:58:29.335703+0000","last_became_peered":"2023-06-10T18:58:29.335703+0000","last_unstale":"2023-06-10T18:58:30.941574+0000","last_undegraded":"2023-06-10T18:58:30.941574+0000","last_fullsized":"2023-06-10T18:58:30.941574+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:04:18.204887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325948+0000","last_change":"2023-06-10T18:58:28.924006+0000","last_active":"2023-06-10T18:58:30.325948+0000","last_peered":"2023-06-10T18:58:30.325948+0000","last_clean":"2023-06-10T18:58:30.325948+0000","last_became_active":"2023-06-10T18:58:28.923748+0000","last_became_peered":"2023-06-10T18:58:28.923748+0000","last_unstale":"2023-06-10T18:58:30.325948+0000","last_undegraded":"2023-06-10T18:58:30.325948+0000","last_fullsized":"2023-06-10T18:58:30.325948+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:52.907184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033315+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:58:31.033315+0000","last_peered":"2023-06-10T18:58:31.033315+0000","last_clean":"2023-06-10T18:58:31.033315+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:58:31.033315+0000","last_undegraded":"2023-06-10T18:58:31.033315+0000","last_fullsized":"2023-06-10T18:58:31.033315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:28:22.555151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326198+0000","last_change":"2023-06-10T18:58:29.338528+0000","last_active":"2023-06-10T18:58:30.326198+0000","last_peered":"2023-06-10T18:58:30.326198+0000","last_clean":"2023-06-10T18:58:30.326198+0000","last_became_active":"2023-06-10T18:58:29.338238+0000","last_became_peered":"2023-06-10T18:58:29.338238+0000","last_unstale":"2023-06-10T18:58:30.326198+0000","last_undegraded":"2023-06-10T18:58:30.326198+0000","last_fullsized":"2023-06-10T18:58:30.326198+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:21:31.181768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033705+0000","last_change":"2023-06-10T18:58:28.932027+0000","last_active":"2023-06-10T18:58:31.033705+0000","last_peered":"2023-06-10T18:58:31.033705+0000","last_clean":"2023-06-10T18:58:31.033705+0000","last_became_active":"2023-06-10T18:58:28.931781+0000","last_became_peered":"2023-06-10T18:58:28.931781+0000","last_unstale":"2023-06-10T18:58:31.033705+0000","last_undegraded":"2023-06-10T18:58:31.033705+0000","last_fullsized":"2023-06-10T18:58:31.033705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:35.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":315,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325704+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:58:30.325704+0000","last_peered":"2023-06-10T18:58:30.325704+0000","last_clean":"2023-06-10T18:58:30.325704+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:58:30.325704+0000","last_undegraded":"2023-06-10T18:58:30.325704+0000","last_fullsized":"2023-06-10T18:58:30.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:33:53.017596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941311+0000","last_change":"2023-06-10T18:58:29.329954+0000","last_active":"2023-06-10T18:58:30.941311+0000","last_peered":"2023-06-10T18:58:30.941311+0000","last_clean":"2023-06-10T18:58:30.941311+0000","last_became_active":"2023-06-10T18:58:29.329420+0000","last_became_peered":"2023-06-10T18:58:29.329420+0000","last_unstale":"2023-06-10T18:58:30.941311+0000","last_undegraded":"2023-06-10T18:58:30.941311+0000","last_fullsized":"2023-06-10T18:58:30.941311+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:26.200027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033188+0000","last_change":"2023-06-10T18:58:28.928009+0000","last_active":"2023-06-10T18:58:31.033188+0000","last_peered":"2023-06-10T18:58:31.033188+0000","last_clean":"2023-06-10T18:58:31.033188+0000","last_became_active":"2023-06-10T18:58:28.927750+0000","last_became_peered":"2023-06-10T18:58:28.927750+0000","last_unstale":"2023-06-10T18:58:31.033188+0000","last_undegraded":"2023-06-10T18:58:31.033188+0000","last_fullsized":"2023-06-10T18:58:31.033188+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:38:52.650003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":269,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033025+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:58:31.033025+0000","last_peered":"2023-06-10T18:58:31.033025+0000","last_clean":"2023-06-10T18:58:31.033025+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:58:31.033025+0000","last_undegraded":"2023-06-10T18:58:31.033025+0000","last_fullsized":"2023-06-10T18:58:31.033025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:19:27.951425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.940999+0000","last_change":"2023-06-10T18:58:29.332300+0000","last_active":"2023-06-10T18:58:30.940999+0000","last_peered":"2023-06-10T18:58:30.940999+0000","last_clean":"2023-06-10T18:58:30.940999+0000","last_became_active":"2023-06-10T18:58:29.327341+0000","last_became_peered":"2023-06-10T18:58:29.327341+0000","last_unstale":"2023-06-10T18:58:30.940999+0000","last_undegraded":"2023-06-10T18:58:30.940999+0000","last_fullsized":"2023-06-10T18:58:30.940999+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:46.331984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034660+0000","last_change":"2023-06-10T18:58:28.934714+0000","last_active":"2023-06-10T18:58:31.034660+0000","last_peered":"2023-06-10T18:58:31.034660+0000","last_clean":"2023-06-10T18:58:31.034660+0000","last_became_active":"2023-06-10T18:58:28.934412+0000","last_became_peered":"2023-06-10T18:58:28.934412+0000","last_unstale":"2023-06-10T18:58:31.034660+0000","last_undegraded":"2023-06-10T18:58:31.034660+0000","last_fullsized":"2023-06-10T18:58:31.034660+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:22:14.017996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":315,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325823+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:58:30.325823+0000","last_peered":"2023-06-10T18:58:30.325823+0000","last_clean":"2023-06-10T18:58:30.325823+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:58:30.325823+0000","last_undegraded":"2023-06-10T18:58:30.325823+0000","last_fullsized":"2023-06-10T18:58:30.325823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:18:10.507632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":442,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034787+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:58:31.034787+0000","last_peered":"2023-06-10T18:58:31.034787+0000","last_clean":"2023-06-10T18:58:31.034787+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:58:31.034787+0000","last_undegraded":"2023-06-10T18:58:31.034787+0000","last_fullsized":"2023-06-10T18:58:31.034787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:04.090060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320151+0000","last_change":"2023-06-10T18:58:29.320040+0000","last_active":"2023-06-10T18:58:30.320151+0000","last_peered":"2023-06-10T18:58:30.320151+0000","last_clean":"2023-06-10T18:58:30.320151+0000","last_became_active":"2023-06-10T18:58:29.319768+0000","last_became_peered":"2023-06-10T18:58:29.319768+0000","last_unstale":"2023-06-10T18:58:30.320151+0000","last_undegraded":"2023-06-10T18:58:30.320151+0000","last_fullsized":"2023-06-10T18:58:30.320151+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:44:31.201848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320167+0000","last_change":"2023-06-10T18:58:28.922699+0000","last_active":"2023-06-10T18:58:30.320167+0000","last_peered":"2023-06-10T18:58:30.320167+0000","last_clean":"2023-06-10T18:58:30.320167+0000","last_became_active":"2023-06-10T18:58:28.922425+0000","last_became_peered":"2023-06-10T18:58:28.922425+0000","last_unstale":"2023-06-10T18:58:30.320167+0000","last_undegraded":"2023-06-10T18:58:30.320167+0000","last_fullsized":"2023-06-10T18:58:30.320167+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:40:19.926035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034668+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:58:31.034668+0000","last_peered":"2023-06-10T18:58:31.034668+0000","last_clean":"2023-06-10T18:58:31.034668+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:58:31.034668+0000","last_undegraded":"2023-06-10T18:58:31.034668+0000","last_fullsized":"2023-06-10T18:58:31.034668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:31:10.616083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320039+0000","last_change":"2023-06-10T18:58:29.326002+0000","last_active":"2023-06-10T18:58:30.320039+0000","last_peered":"2023-06-10T18:58:30.320039+0000","last_clean":"2023-06-10T18:58:30.320039+0000","last_became_active":"2023-06-10T18:58:29.325588+0000","last_became_peered":"2023-06-10T18:58:29.325588+0000","last_unstale":"2023-06-10T18:58:30.320039+0000","last_undegraded":"2023-06-10T18:58:30.320039+0000","last_fullsized":"2023-06-10T18:58:30.320039+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:39:24.026553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"171'4","reported_seq":17,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033649+0000","last_change":"2023-06-10T18:58:28.925351+0000","last_active":"2023-06-10T18:58:31.033649+0000","last_peered":"2023-06-10T18:58:31.033649+0000","last_clean":"2023-06-10T18:58:31.033649+0000","last_became_active":"2023-06-10T18:58:28.925089+0000","last_became_peered":"2023-06-10T18:58:28.925089+0000","last_unstale":"2023-06-10T18:58:31.033649+0000","last_undegraded":"2023-06-10T18:58:31.033649+0000","last_fullsized":"2023-06-10T18:58:31.033649+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:28:42.293438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":285,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034744+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:58:31.034744+0000","last_peered":"2023-06-10T18:58:31.034744+0000","last_clean":"2023-06-10T18:58:31.034744+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:58:31.034744+0000","last_undegraded":"2023-06-10T18:58:31.034744+0000","last_fullsized":"2023-06-10T18:58:31.034744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:28.956704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325869+0000","last_change":"2023-06-10T18:58:29.332169+0000","last_active":"2023-06-10T18:58:30.325869+0000","last_peered":"2023-06-10T18:58:30.325869+0000","last_clean":"2023-06-10T18:58:30.325869+0000","last_became_active":"2023-06-10T18:58:29.331878+0000","last_became_peered":"2023-06-10T18:58:29.331878+0000","last_unstale":"2023-06-10T18:58:30.325869+0000","last_undegraded":"2023-06-10T18:58:30.325869+0000","last_fullsized":"2023-06-10T18:58:30.325869+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:32:59.801173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034294+0000","last_change":"2023-06-10T18:58:28.925704+0000","last_active":"2023-06-10T18:58:31.034294+0000","last_peered":"2023-06-10T18:58:31.034294+0000","last_clean":"2023-06-10T18:58:31.034294+0000","last_became_active":"2023-06-10T18:58:28.925383+0000","last_became_peered":"2023-06-10T18:58:28.925383+0000","last_unstale":"2023-06-10T18:58:31.034294+0000","last_undegraded":"2023-06-10T18:58:31.034294+0000","last_fullsized":"2023-06-10T18:58:31.034294+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:33:33.783242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034310+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:58:31.034310+0000","last_peered":"2023-06-10T18:58:31.034310+0000","last_clean":"2023-06-10T18:58:31.034310+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:58:31.034310+0000","last_undegraded":"2023-06-10T18:58:31.034310+0000","last_fullsized":"2023-06-10T18:58:31.034310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:28.526299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326420+0000","last_change":"2023-06-10T18:58:29.333782+0000","last_active":"2023-06-10T18:58:30.326420+0000","last_peered":"2023-06-10T18:58:30.326420+0000","last_clean":"2023-06-10T18:58:30.326420+0000","last_became_active":"2023-06-10T18:58:29.333508+0000","last_became_peered":"2023-06-10T18:58:29.333508+0000","last_unstale":"2023-06-10T18:58:30.326420+0000","last_undegraded":"2023-06-10T18:58:30.326420+0000","last_fullsized":"2023-06-10T18:58:30.326420+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:50.392702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033893+0000","last_change":"2023-06-10T18:58:28.927435+0000","last_active":"2023-06-10T18:58:31.033893+0000","last_peered":"2023-06-10T18:58:31.033893+0000","last_clean":"2023-06-10T18:58:31.033893+0000","last_became_active":"2023-06-10T18:58:28.926429+0000","last_became_peered":"2023-06-10T18:58:28.926429+0000","last_unstale":"2023-06-10T18:58:31.033893+0000","last_undegraded":"2023-06-10T18:58:31.033893+0000","last_fullsized":"2023-06-10T18:58:31.033893+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:52.907184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034220+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:58:31.034220+0000","last_peered":"2023-06-10T18:58:31.034220+0000","last_clean":"2023-06-10T18:58:31.034220+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:58:31.034220+0000","last_undegraded":"2023-06-10T18:58:31.034220+0000","last_fullsized":"2023-06-10T18:58:31.034220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:50:59.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33744,"kb_used_data":2696,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338096,"statfs":{"total":96636764160,"available":96602210304,"internally_reserved":0,"allocated":2760704,"data_stored":1651510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.54500000000000004},"min":{"1min":0.38100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64900000000000002,"5min":0.749,"15min":0.81499999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73599999999999999,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":0.63600000000000001,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68000000000000005,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.51900000000000002},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.436},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52700000000000002,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.66100000000000003,"5min":0.745,"15min":0.79000000000000004},"last":0.59299999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.32300000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.65200000000000002,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.68600000000000005,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69399999999999995,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.93899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.72399999999999998,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.498}]}]},{"osd":6,"up_from":13,"seq":55834575028,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33140,"kb_used_data":2092,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338700,"statfs":{"total":96636764160,"available":96602828800,"internally_reserved":0,"allocated":2142208,"data_stored":1053046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.504,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51500000000000001,"15min":0.51800000000000002},"min":{"1min":0.38700000000000001,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.309},"max":{"1min":0.60899999999999999,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.498},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.33600000000000002,"15min":0.29199999999999998},"max":{"1min":0.78400000000000003,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.57799999999999996},"min":{"1min":0.40300000000000002,"5min":0.36799999999999999,"15min":0.32200000000000001},"max":{"1min":0.73299999999999998,"5min":0.872,"15min":6.6689999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.78800000000000003,"15min":1.252},"last":0.68200000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53200000000000003,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.747,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83899999999999997,"5min":0.874,"15min":0.874},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":0.746,"15min":0.89600000000000002},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.44500000000000001,"5min":0.312,"15min":0.307},"max":{"1min":0.81299999999999994,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.74099999999999999,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.47499999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2166784,"data_stored":1061583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.32500000000000001},"min":{"1min":0.216,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.42999999999999999,"5min":0.51800000000000002,"15min":0.753},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34599999999999997,"15min":0.35099999999999998},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.219},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.318}]},{"osd":2,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.34200000000000003},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.503,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35199999999999998,"15min":0.373},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20799999999999999},"max":{"1min":0.503,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53800000000000003,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.39100000000000001},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88500000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.62,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.434,"15min":0.34899999999999998},"max":{"1min":0.89400000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82799999999999996,"5min":0.85999999999999999,"15min":1.008},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.78000000000000003,"5min":1.206,"15min":1.206},"last":0.625},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.873,"15min":0.88700000000000001},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.41399999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":1.224,"15min":1.224},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77000000000000002,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.59099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575027,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33736,"kb_used_data":2688,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338104,"statfs":{"total":96636764160,"available":96602218496,"internally_reserved":0,"allocated":2752512,"data_stored":1650409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33900000000000002,"15min":0.33800000000000002},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.47299999999999998,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.34999999999999998},"min":{"1min":0.23400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.505,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.308}]},{"osd":2,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.435,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52600000000000002,"5min":11.439,"15min":11.439},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.48199999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.54000000000000004,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.27300000000000002,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.48999999999999999,"5min":5.766,"15min":5.766},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.27000000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.64800000000000002,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79700000000000004,"5min":2.802,"15min":2.802},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.999},"last":0.502},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.71799999999999997,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77100000000000002,"5min":2.907,"15min":2.907},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58399999999999996,"15min":0.59899999999999998},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.71499999999999997,"5min":0.86099999999999999,"15min":1.363},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":1.0489999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72899999999999998,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.44700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575028,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":2108,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2158592,"data_stored":1054759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.318,"15min":0.34200000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.45300000000000001,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.38500000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.48299999999999998,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.377},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47099999999999997,"5min":0.57599999999999996,"15min":2.847},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36699999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.45900000000000002,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.622}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34699999999999998,"15min":0.38500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.442,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.25800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54100000000000004,"5min":1.24,"15min":2.8999999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.61499999999999999},"min":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":1.208,"15min":5.742},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58499999999999996,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81200000000000006,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61799999999999999,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":6.593},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.78800000000000003,"5min":1.002,"15min":7.0880000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":0.79500000000000004,"5min":1.026,"15min":8.7850000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":1.171,"15min":5.6180000000000003},"last":0.76800000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.64300000000000002,"15min":0.66400000000000003},"min":{"1min":0.47399999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.84399999999999997,"5min":0.87,"15min":6.617},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.65900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.754}]}]},{"osd":3,"up_from":13,"seq":55834575027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":2120,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2170880,"data_stored":1061232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.32200000000000001,"15min":0.34200000000000003},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.48199999999999998,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34499999999999997,"15min":0.36199999999999999},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.20200000000000001},"max":{"1min":0.44400000000000001,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.254}]},{"osd":1,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34300000000000003,"15min":0.36099999999999999},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.38100000000000001,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.372},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.379,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52000000000000002,"5min":0.64900000000000002,"15min":1.3},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.39600000000000002},"min":{"1min":0.28999999999999998,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.57799999999999996,"15min":1.109},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.39900000000000002,"15min":0.42599999999999999},"min":{"1min":0.29899999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55400000000000005,"5min":0.623,"15min":1.599},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57399999999999995,"15min":0.59599999999999997},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71299999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":0.72799999999999998,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.42699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.752,"5min":1.1100000000000001,"15min":1.613},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.63},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76300000000000001,"5min":1.095,"15min":3.802},"last":0.78800000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84999999999999998,"5min":0.86499999999999999,"15min":3.8180000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.39700000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":1.018,"15min":1.679},"last":0.628}]},{"osd":7,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":1.0549999999999999,"15min":1.47},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73599999999999999,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.66500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":2120,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2170880,"data_stored":1062165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.71999999999999997,"5min":0.745,"15min":1.119},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.1639999999999999},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77800000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.873,"15min":0.96599999999999997},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.67100000000000004,"5min":0.72799999999999998,"15min":0.80800000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.33200000000000002},"max":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.624}]},{"osd":3,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.44500000000000001,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.79800000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.042},"last":0.65000000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.67800000000000005,"5min":0.94599999999999995,"15min":1.069},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.501,"15min":0.504},"min":{"1min":0.36299999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.88300000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.501},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.90100000000000002,"15min":1.1459999999999999},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575028,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":2108,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2158592,"data_stored":1061038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.81000000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51400000000000001,"15min":0.49299999999999999},"min":{"1min":0.41899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.83999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.627}]},{"osd":6,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.879},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.54900000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:58:33.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:58:33.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.580+0000 7f27862ec700 1 Processor -- start 2023-06-10T18:58:33.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.580+0000 7f27862ec700 1 -- start start 2023-06-10T18:58:33.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f278015c500 0x7f27800cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800ce040 0x7f278006e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2780069e70 0x7f278006a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278006a830 con 0x7f278015c500 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278006dcb0 con 0x7f2780069e70 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f27862ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27800ce470 con 0x7f27800ce040 2023-06-10T18:58:33.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f278015c500 0x7f27800cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f278015c500 0x7f27800cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40410/0 (socket says 172.21.15.104:40410) 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 -- 172.21.15.104:0/188852337 learned_addr learned my addr 172.21.15.104:0/188852337 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277f7fe700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800ce040 0x7f278006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f2784889700 1 --2- 172.21.15.104:0/188852337 >> v2:172.21.15.114:3300/0 conn(0x7f2780069e70 0x7f278006a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 -- 172.21.15.104:0/188852337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800ce040 msgr2=0x7f278006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800ce040 0x7f278006e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 -- 172.21.15.104:0/188852337 >> v2:172.21.15.114:3300/0 conn(0x7f2780069e70 msgr2=0x7f278006a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 --2- 172.21.15.104:0/188852337 >> v2:172.21.15.114:3300/0 conn(0x7f2780069e70 0x7f278006a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 -- 172.21.15.104:0/188852337 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278006d0a0 con 0x7f278015c500 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277ffff700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f278015c500 0x7f27800cf030 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f2780066e20 tx=0x7f2770009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95940645fc055cb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277effd700 1 -- 172.21.15.104:0/188852337 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770005e30 con 0x7f278015c500 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277effd700 1 -- 172.21.15.104:0/188852337 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770004030 con 0x7f278015c500 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.581+0000 7f277effd700 1 -- 172.21.15.104:0/188852337 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700043e0 con 0x7f278015c500 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 -- 172.21.15.104:0/188852337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f278015c500 msgr2=0x7f27800cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f278015c500 0x7f27800cf030 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f2780066e20 tx=0x7f2770009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 -- 172.21.15.104:0/188852337 shutdown_connections 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f278015c500 0x7f27800cf030 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 --2- 172.21.15.104:0/188852337 >> v2:172.21.15.114:3300/0 conn(0x7f2780069e70 0x7f278006a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 --2- 172.21.15.104:0/188852337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27800ce040 0x7f278006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 -- 172.21.15.104:0/188852337 >> 172.21.15.104:0/188852337 conn(0x7f278006b8a0 msgr2=0x7f27800cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 -- 172.21.15.104:0/188852337 shutdown_connections 2023-06-10T18:58:33.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.582+0000 7f27862ec700 1 -- 172.21.15.104:0/188852337 wait complete. 2023-06-10T18:58:33.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.583+0000 7f27862ec700 1 Processor -- start 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.583+0000 7f27862ec700 1 -- start start 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 0x7f2780145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 0x7f27800cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801482c0 con 0x7f2780069e70 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c7420 con 0x7f27800ce040 2023-06-10T18:58:33.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f27862ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27800c6810 con 0x7f278015c500 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40416/0 (socket says 172.21.15.104:40416) 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 -- 172.21.15.104:0/837769395 learned_addr learned my addr 172.21.15.104:0/837769395 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f2784889700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 0x7f27800cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277f7fe700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 0x7f2780145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 -- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 msgr2=0x7f27800cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 0x7f27800cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 -- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 msgr2=0x7f2780145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 0x7f2780145330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277ffff700 1 -- 172.21.15.104:0/837769395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2770009500 con 0x7f2780069e70 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f2784889700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 0x7f27800cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.584+0000 7f277f7fe700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 0x7f2780145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:33.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f277ffff700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f2770009c10 tx=0x7f2770004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770003ce0 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2770003e60 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27800c6af0 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277001c660 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.585+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27800c70b0 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.586+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27800c7240 con 0x7f2780069e70 2023-06-10T18:58:33.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.586+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f277001c7c0 con 0x7f2780069e70 2023-06-10T18:58:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.587+0000 7f277cff9700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 0x7f276c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.587+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f277008a5a0 con 0x7f2780069e70 2023-06-10T18:58:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.587+0000 7f277f7fe700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 0x7f276c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.587+0000 7f277f7fe700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 0x7f276c06d190 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f27801f5550 tx=0x7f2774008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.591+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2770055500 con 0x7f2780069e70 2023-06-10T18:58:33.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.775+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f278006d660 con 0x7f276c06ace0 2023-06-10T18:58:33.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.784+0000 7f277cff9700 1 -- 172.21.15.104:0/837769395 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311839 (secure 0 0 0) 0x7f2774011720 con 0x7f276c06ace0 2023-06-10T18:58:33.794 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:33.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 msgr2=0x7f276c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 0x7f276c06d190 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f27801f5550 tx=0x7f2774008040 comp rx=0 tx=0).stop 2023-06-10T18:58:33.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 msgr2=0x7f27801f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f2770009c10 tx=0x7f2770004750 comp rx=0 tx=0).stop 2023-06-10T18:58:33.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 shutdown_connections 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f278015c500 0x7f27800cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2780069e70 0x7f27801f18c0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f276c06ace0 0x7f276c06d190 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 --2- 172.21.15.104:0/837769395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27800ce040 0x7f2780145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 >> 172.21.15.104:0/837769395 conn(0x7f278006b8a0 msgr2=0x7f27800b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 shutdown_connections 2023-06-10T18:58:33.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.787+0000 7f27862ec700 1 -- 172.21.15.104:0/837769395 wait complete. 2023-06-10T18:58:33.798 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:58:33.805 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2023-06-10T18:58:33.579459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266432,"kb_used_data":18048,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754708288,"statfs":{"total":773094113280,"available":772821286912,"internally_reserved":0,"allocated":18481152,"data_stored":9655742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.671124"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034725+0000","last_change":"2023-06-10T18:58:29.338467+0000","last_active":"2023-06-10T18:58:31.034725+0000","last_peered":"2023-06-10T18:58:31.034725+0000","last_clean":"2023-06-10T18:58:31.034725+0000","last_became_active":"2023-06-10T18:58:29.338200+0000","last_became_peered":"2023-06-10T18:58:29.338200+0000","last_unstale":"2023-06-10T18:58:31.034725+0000","last_undegraded":"2023-06-10T18:58:31.034725+0000","last_fullsized":"2023-06-10T18:58:31.034725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:52:01.917085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034033+0000","last_change":"2023-06-10T18:58:29.337738+0000","last_active":"2023-06-10T18:58:31.034033+0000","last_peered":"2023-06-10T18:58:31.034033+0000","last_clean":"2023-06-10T18:58:31.034033+0000","last_became_active":"2023-06-10T18:58:29.337457+0000","last_became_peered":"2023-06-10T18:58:29.337457+0000","last_unstale":"2023-06-10T18:58:31.034033+0000","last_undegraded":"2023-06-10T18:58:31.034033+0000","last_fullsized":"2023-06-10T18:58:31.034033+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:04:33.687163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033647+0000","last_change":"2023-06-10T18:58:29.323437+0000","last_active":"2023-06-10T18:58:31.033647+0000","last_peered":"2023-06-10T18:58:31.033647+0000","last_clean":"2023-06-10T18:58:31.033647+0000","last_became_active":"2023-06-10T18:58:29.323202+0000","last_became_peered":"2023-06-10T18:58:29.323202+0000","last_unstale":"2023-06-10T18:58:31.033647+0000","last_undegraded":"2023-06-10T18:58:31.033647+0000","last_fullsized":"2023-06-10T18:58:31.033647+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:32:59.801173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034351+0000","last_change":"2023-06-10T18:58:29.337452+0000","last_active":"2023-06-10T18:58:31.034351+0000","last_peered":"2023-06-10T18:58:31.034351+0000","last_clean":"2023-06-10T18:58:31.034351+0000","last_became_active":"2023-06-10T18:58:29.337181+0000","last_became_peered":"2023-06-10T18:58:29.337181+0000","last_unstale":"2023-06-10T18:58:31.034351+0000","last_undegraded":"2023-06-10T18:58:31.034351+0000","last_fullsized":"2023-06-10T18:58:31.034351+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:48:42.173586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032905+0000","last_change":"2023-06-10T18:58:29.332301+0000","last_active":"2023-06-10T18:58:31.032905+0000","last_peered":"2023-06-10T18:58:31.032905+0000","last_clean":"2023-06-10T18:58:31.032905+0000","last_became_active":"2023-06-10T18:58:29.332064+0000","last_became_peered":"2023-06-10T18:58:29.332064+0000","last_unstale":"2023-06-10T18:58:31.032905+0000","last_undegraded":"2023-06-10T18:58:31.032905+0000","last_fullsized":"2023-06-10T18:58:31.032905+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:11:52.532995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033041+0000","last_change":"2023-06-10T18:58:29.330081+0000","last_active":"2023-06-10T18:58:31.033041+0000","last_peered":"2023-06-10T18:58:31.033041+0000","last_clean":"2023-06-10T18:58:31.033041+0000","last_became_active":"2023-06-10T18:58:29.329825+0000","last_became_peered":"2023-06-10T18:58:29.329825+0000","last_unstale":"2023-06-10T18:58:31.033041+0000","last_undegraded":"2023-06-10T18:58:31.033041+0000","last_fullsized":"2023-06-10T18:58:31.033041+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:40:27.154386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326157+0000","last_change":"2023-06-10T18:58:29.338369+0000","last_active":"2023-06-10T18:58:30.326157+0000","last_peered":"2023-06-10T18:58:30.326157+0000","last_clean":"2023-06-10T18:58:30.326157+0000","last_became_active":"2023-06-10T18:58:29.337617+0000","last_became_peered":"2023-06-10T18:58:29.337617+0000","last_unstale":"2023-06-10T18:58:30.326157+0000","last_undegraded":"2023-06-10T18:58:30.326157+0000","last_fullsized":"2023-06-10T18:58:30.326157+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:27.421848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319811+0000","last_change":"2023-06-10T18:58:29.333536+0000","last_active":"2023-06-10T18:58:30.319811+0000","last_peered":"2023-06-10T18:58:30.319811+0000","last_clean":"2023-06-10T18:58:30.319811+0000","last_became_active":"2023-06-10T18:58:29.333303+0000","last_became_peered":"2023-06-10T18:58:29.333303+0000","last_unstale":"2023-06-10T18:58:30.319811+0000","last_undegraded":"2023-06-10T18:58:30.319811+0000","last_fullsized":"2023-06-10T18:58:30.319811+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:03:48.813988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325988+0000","last_change":"2023-06-10T18:58:29.326844+0000","last_active":"2023-06-10T18:58:30.325988+0000","last_peered":"2023-06-10T18:58:30.325988+0000","last_clean":"2023-06-10T18:58:30.325988+0000","last_became_active":"2023-06-10T18:58:29.326564+0000","last_became_peered":"2023-06-10T18:58:29.326564+0000","last_unstale":"2023-06-10T18:58:30.325988+0000","last_undegraded":"2023-06-10T18:58:30.325988+0000","last_fullsized":"2023-06-10T18:58:30.325988+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:39.177180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941459+0000","last_change":"2023-06-10T18:58:29.333335+0000","last_active":"2023-06-10T18:58:30.941459+0000","last_peered":"2023-06-10T18:58:30.941459+0000","last_clean":"2023-06-10T18:58:30.941459+0000","last_became_active":"2023-06-10T18:58:29.333067+0000","last_became_peered":"2023-06-10T18:58:29.333067+0000","last_unstale":"2023-06-10T18:58:30.941459+0000","last_undegraded":"2023-06-10T18:58:30.941459+0000","last_fullsized":"2023-06-10T18:58:30.941459+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:18:15.909594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326749+0000","last_change":"2023-06-10T18:58:29.335540+0000","last_active":"2023-06-10T18:58:30.326749+0000","last_peered":"2023-06-10T18:58:30.326749+0000","last_clean":"2023-06-10T18:58:30.326749+0000","last_became_active":"2023-06-10T18:58:29.335247+0000","last_became_peered":"2023-06-10T18:58:29.335247+0000","last_unstale":"2023-06-10T18:58:30.326749+0000","last_undegraded":"2023-06-10T18:58:30.326749+0000","last_fullsized":"2023-06-10T18:58:30.326749+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:06.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034114+0000","last_change":"2023-06-10T18:58:29.335587+0000","last_active":"2023-06-10T18:58:31.034114+0000","last_peered":"2023-06-10T18:58:31.034114+0000","last_clean":"2023-06-10T18:58:31.034114+0000","last_became_active":"2023-06-10T18:58:29.335235+0000","last_became_peered":"2023-06-10T18:58:29.335235+0000","last_unstale":"2023-06-10T18:58:31.034114+0000","last_undegraded":"2023-06-10T18:58:31.034114+0000","last_fullsized":"2023-06-10T18:58:31.034114+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:17:31.970779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033984+0000","last_change":"2023-06-10T18:58:29.332847+0000","last_active":"2023-06-10T18:58:31.033984+0000","last_peered":"2023-06-10T18:58:31.033984+0000","last_clean":"2023-06-10T18:58:31.033984+0000","last_became_active":"2023-06-10T18:58:29.332600+0000","last_became_peered":"2023-06-10T18:58:29.332600+0000","last_unstale":"2023-06-10T18:58:31.033984+0000","last_undegraded":"2023-06-10T18:58:31.033984+0000","last_fullsized":"2023-06-10T18:58:31.033984+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:45:13.186666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325662+0000","last_change":"2023-06-10T18:58:29.330986+0000","last_active":"2023-06-10T18:58:30.325662+0000","last_peered":"2023-06-10T18:58:30.325662+0000","last_clean":"2023-06-10T18:58:30.325662+0000","last_became_active":"2023-06-10T18:58:29.330604+0000","last_became_peered":"2023-06-10T18:58:29.330604+0000","last_unstale":"2023-06-10T18:58:30.325662+0000","last_undegraded":"2023-06-10T18:58:30.325662+0000","last_fullsized":"2023-06-10T18:58:30.325662+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:19.397055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034404+0000","last_change":"2023-06-10T18:58:29.338141+0000","last_active":"2023-06-10T18:58:31.034404+0000","last_peered":"2023-06-10T18:58:31.034404+0000","last_clean":"2023-06-10T18:58:31.034404+0000","last_became_active":"2023-06-10T18:58:29.337876+0000","last_became_peered":"2023-06-10T18:58:29.337876+0000","last_unstale":"2023-06-10T18:58:31.034404+0000","last_undegraded":"2023-06-10T18:58:31.034404+0000","last_fullsized":"2023-06-10T18:58:31.034404+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:36:25.429290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320404+0000","last_change":"2023-06-10T18:58:29.322664+0000","last_active":"2023-06-10T18:58:30.320404+0000","last_peered":"2023-06-10T18:58:30.320404+0000","last_clean":"2023-06-10T18:58:30.320404+0000","last_became_active":"2023-06-10T18:58:29.322408+0000","last_became_peered":"2023-06-10T18:58:29.322408+0000","last_unstale":"2023-06-10T18:58:30.320404+0000","last_undegraded":"2023-06-10T18:58:30.320404+0000","last_fullsized":"2023-06-10T18:58:30.320404+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:47:34.331285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033584+0000","last_change":"2023-06-10T18:58:29.333305+0000","last_active":"2023-06-10T18:58:31.033584+0000","last_peered":"2023-06-10T18:58:31.033584+0000","last_clean":"2023-06-10T18:58:31.033584+0000","last_became_active":"2023-06-10T18:58:29.332773+0000","last_became_peered":"2023-06-10T18:58:29.332773+0000","last_unstale":"2023-06-10T18:58:31.033584+0000","last_undegraded":"2023-06-10T18:58:31.033584+0000","last_fullsized":"2023-06-10T18:58:31.033584+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:57:37.325886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034891+0000","last_change":"2023-06-10T18:58:29.336430+0000","last_active":"2023-06-10T18:58:31.034891+0000","last_peered":"2023-06-10T18:58:31.034891+0000","last_clean":"2023-06-10T18:58:31.034891+0000","last_became_active":"2023-06-10T18:58:29.334374+0000","last_became_peered":"2023-06-10T18:58:29.334374+0000","last_unstale":"2023-06-10T18:58:31.034891+0000","last_undegraded":"2023-06-10T18:58:31.034891+0000","last_fullsized":"2023-06-10T18:58:31.034891+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:13.541013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941270+0000","last_change":"2023-06-10T18:58:29.329251+0000","last_active":"2023-06-10T18:58:30.941270+0000","last_peered":"2023-06-10T18:58:30.941270+0000","last_clean":"2023-06-10T18:58:30.941270+0000","last_became_active":"2023-06-10T18:58:29.328923+0000","last_became_peered":"2023-06-10T18:58:29.328923+0000","last_unstale":"2023-06-10T18:58:30.941270+0000","last_undegraded":"2023-06-10T18:58:30.941270+0000","last_fullsized":"2023-06-10T18:58:30.941270+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:41:01.516484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033090+0000","last_change":"2023-06-10T18:58:29.330085+0000","last_active":"2023-06-10T18:58:31.033090+0000","last_peered":"2023-06-10T18:58:31.033090+0000","last_clean":"2023-06-10T18:58:31.033090+0000","last_became_active":"2023-06-10T18:58:29.329839+0000","last_became_peered":"2023-06-10T18:58:29.329839+0000","last_unstale":"2023-06-10T18:58:31.033090+0000","last_undegraded":"2023-06-10T18:58:31.033090+0000","last_fullsized":"2023-06-10T18:58:31.033090+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:15.805525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033042+0000","last_change":"2023-06-10T18:58:29.331865+0000","last_active":"2023-06-10T18:58:31.033042+0000","last_peered":"2023-06-10T18:58:31.033042+0000","last_clean":"2023-06-10T18:58:31.033042+0000","last_became_active":"2023-06-10T18:58:29.331590+0000","last_became_peered":"2023-06-10T18:58:29.331590+0000","last_unstale":"2023-06-10T18:58:31.033042+0000","last_undegraded":"2023-06-10T18:58:31.033042+0000","last_fullsized":"2023-06-10T18:58:31.033042+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:44:33.883497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941155+0000","last_change":"2023-06-10T18:58:29.330153+0000","last_active":"2023-06-10T18:58:30.941155+0000","last_peered":"2023-06-10T18:58:30.941155+0000","last_clean":"2023-06-10T18:58:30.941155+0000","last_became_active":"2023-06-10T18:58:29.329721+0000","last_became_peered":"2023-06-10T18:58:29.329721+0000","last_unstale":"2023-06-10T18:58:30.941155+0000","last_undegraded":"2023-06-10T18:58:30.941155+0000","last_fullsized":"2023-06-10T18:58:30.941155+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:06.292554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033306+0000","last_change":"2023-06-10T18:58:29.328205+0000","last_active":"2023-06-10T18:58:31.033306+0000","last_peered":"2023-06-10T18:58:31.033306+0000","last_clean":"2023-06-10T18:58:31.033306+0000","last_became_active":"2023-06-10T18:58:29.327913+0000","last_became_peered":"2023-06-10T18:58:29.327913+0000","last_unstale":"2023-06-10T18:58:31.033306+0000","last_undegraded":"2023-06-10T18:58:31.033306+0000","last_fullsized":"2023-06-10T18:58:31.033306+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:49.210752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320049+0000","last_change":"2023-06-10T18:58:29.321853+0000","last_active":"2023-06-10T18:58:30.320049+0000","last_peered":"2023-06-10T18:58:30.320049+0000","last_clean":"2023-06-10T18:58:30.320049+0000","last_became_active":"2023-06-10T18:58:29.320585+0000","last_became_peered":"2023-06-10T18:58:29.320585+0000","last_unstale":"2023-06-10T18:58:30.320049+0000","last_undegraded":"2023-06-10T18:58:30.320049+0000","last_fullsized":"2023-06-10T18:58:30.320049+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:40:58.370826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034187+0000","last_change":"2023-06-10T18:58:29.338307+0000","last_active":"2023-06-10T18:58:31.034187+0000","last_peered":"2023-06-10T18:58:31.034187+0000","last_clean":"2023-06-10T18:58:31.034187+0000","last_became_active":"2023-06-10T18:58:29.338028+0000","last_became_peered":"2023-06-10T18:58:29.338028+0000","last_unstale":"2023-06-10T18:58:31.034187+0000","last_undegraded":"2023-06-10T18:58:31.034187+0000","last_fullsized":"2023-06-10T18:58:31.034187+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:59:38.236995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326394+0000","last_change":"2023-06-10T18:58:29.335394+0000","last_active":"2023-06-10T18:58:30.326394+0000","last_peered":"2023-06-10T18:58:30.326394+0000","last_clean":"2023-06-10T18:58:30.326394+0000","last_became_active":"2023-06-10T18:58:29.335106+0000","last_became_peered":"2023-06-10T18:58:29.335106+0000","last_unstale":"2023-06-10T18:58:30.326394+0000","last_undegraded":"2023-06-10T18:58:30.326394+0000","last_fullsized":"2023-06-10T18:58:30.326394+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:04.818195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033770+0000","last_change":"2023-06-10T18:58:29.330235+0000","last_active":"2023-06-10T18:58:31.033770+0000","last_peered":"2023-06-10T18:58:31.033770+0000","last_clean":"2023-06-10T18:58:31.033770+0000","last_became_active":"2023-06-10T18:58:29.329818+0000","last_became_peered":"2023-06-10T18:58:29.329818+0000","last_unstale":"2023-06-10T18:58:31.033770+0000","last_undegraded":"2023-06-10T18:58:31.033770+0000","last_fullsized":"2023-06-10T18:58:31.033770+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:32:36.894140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033223+0000","last_change":"2023-06-10T18:58:29.337668+0000","last_active":"2023-06-10T18:58:31.033223+0000","last_peered":"2023-06-10T18:58:31.033223+0000","last_clean":"2023-06-10T18:58:31.033223+0000","last_became_active":"2023-06-10T18:58:29.337399+0000","last_became_peered":"2023-06-10T18:58:29.337399+0000","last_unstale":"2023-06-10T18:58:31.033223+0000","last_undegraded":"2023-06-10T18:58:31.033223+0000","last_fullsized":"2023-06-10T18:58:31.033223+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:35.151278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034162+0000","last_change":"2023-06-10T18:58:29.335888+0000","last_active":"2023-06-10T18:58:31.034162+0000","last_peered":"2023-06-10T18:58:31.034162+0000","last_clean":"2023-06-10T18:58:31.034162+0000","last_became_active":"2023-06-10T18:58:29.335581+0000","last_became_peered":"2023-06-10T18:58:29.335581+0000","last_unstale":"2023-06-10T18:58:31.034162+0000","last_undegraded":"2023-06-10T18:58:31.034162+0000","last_fullsized":"2023-06-10T18:58:31.034162+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:10:22.985514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325450+0000","last_change":"2023-06-10T18:58:29.329772+0000","last_active":"2023-06-10T18:58:30.325450+0000","last_peered":"2023-06-10T18:58:30.325450+0000","last_clean":"2023-06-10T18:58:30.325450+0000","last_became_active":"2023-06-10T18:58:29.329467+0000","last_became_peered":"2023-06-10T18:58:29.329467+0000","last_unstale":"2023-06-10T18:58:30.325450+0000","last_undegraded":"2023-06-10T18:58:30.325450+0000","last_fullsized":"2023-06-10T18:58:30.325450+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:33:36.805448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033422+0000","last_change":"2023-06-10T18:58:29.328763+0000","last_active":"2023-06-10T18:58:31.033422+0000","last_peered":"2023-06-10T18:58:31.033422+0000","last_clean":"2023-06-10T18:58:31.033422+0000","last_became_active":"2023-06-10T18:58:29.328510+0000","last_became_peered":"2023-06-10T18:58:29.328510+0000","last_unstale":"2023-06-10T18:58:31.033422+0000","last_undegraded":"2023-06-10T18:58:31.033422+0000","last_fullsized":"2023-06-10T18:58:31.033422+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:10.729171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032819+0000","last_change":"2023-06-10T18:58:29.332110+0000","last_active":"2023-06-10T18:58:31.032819+0000","last_peered":"2023-06-10T18:58:31.032819+0000","last_clean":"2023-06-10T18:58:31.032819+0000","last_became_active":"2023-06-10T18:58:29.331850+0000","last_became_peered":"2023-06-10T18:58:29.331850+0000","last_unstale":"2023-06-10T18:58:31.032819+0000","last_undegraded":"2023-06-10T18:58:31.032819+0000","last_fullsized":"2023-06-10T18:58:31.032819+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:23:37.808457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034233+0000","last_change":"2023-06-10T18:58:29.320564+0000","last_active":"2023-06-10T18:58:31.034233+0000","last_peered":"2023-06-10T18:58:31.034233+0000","last_clean":"2023-06-10T18:58:31.034233+0000","last_became_active":"2023-06-10T18:58:29.320312+0000","last_became_peered":"2023-06-10T18:58:29.320312+0000","last_unstale":"2023-06-10T18:58:31.034233+0000","last_undegraded":"2023-06-10T18:58:31.034233+0000","last_fullsized":"2023-06-10T18:58:31.034233+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:31:39.177180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326615+0000","last_change":"2023-06-10T18:58:28.927638+0000","last_active":"2023-06-10T18:58:30.326615+0000","last_peered":"2023-06-10T18:58:30.326615+0000","last_clean":"2023-06-10T18:58:30.326615+0000","last_became_active":"2023-06-10T18:58:28.927421+0000","last_became_peered":"2023-06-10T18:58:28.927421+0000","last_unstale":"2023-06-10T18:58:30.326615+0000","last_undegraded":"2023-06-10T18:58:30.326615+0000","last_fullsized":"2023-06-10T18:58:30.326615+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:29.887872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319939+0000","last_change":"2023-06-10T18:58:29.336412+0000","last_active":"2023-06-10T18:58:30.319939+0000","last_peered":"2023-06-10T18:58:30.319939+0000","last_clean":"2023-06-10T18:58:30.319939+0000","last_became_active":"2023-06-10T18:58:29.336166+0000","last_became_peered":"2023-06-10T18:58:29.336166+0000","last_unstale":"2023-06-10T18:58:30.319939+0000","last_undegraded":"2023-06-10T18:58:30.319939+0000","last_fullsized":"2023-06-10T18:58:30.319939+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:44:42.726360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033243+0000","last_change":"2023-06-10T18:58:28.932065+0000","last_active":"2023-06-10T18:58:31.033243+0000","last_peered":"2023-06-10T18:58:31.033243+0000","last_clean":"2023-06-10T18:58:31.033243+0000","last_became_active":"2023-06-10T18:58:28.931859+0000","last_became_peered":"2023-06-10T18:58:28.931859+0000","last_unstale":"2023-06-10T18:58:31.033243+0000","last_undegraded":"2023-06-10T18:58:31.033243+0000","last_fullsized":"2023-06-10T18:58:31.033243+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:39:00.868092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326557+0000","last_change":"2023-06-10T18:58:29.333048+0000","last_active":"2023-06-10T18:58:30.326557+0000","last_peered":"2023-06-10T18:58:30.326557+0000","last_clean":"2023-06-10T18:58:30.326557+0000","last_became_active":"2023-06-10T18:58:29.332769+0000","last_became_peered":"2023-06-10T18:58:29.332769+0000","last_unstale":"2023-06-10T18:58:30.326557+0000","last_undegraded":"2023-06-10T18:58:30.326557+0000","last_fullsized":"2023-06-10T18:58:30.326557+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:46.331984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"171'3","reported_seq":16,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032991+0000","last_change":"2023-06-10T18:58:28.929567+0000","last_active":"2023-06-10T18:58:31.032991+0000","last_peered":"2023-06-10T18:58:31.032991+0000","last_clean":"2023-06-10T18:58:31.032991+0000","last_became_active":"2023-06-10T18:58:28.929303+0000","last_became_peered":"2023-06-10T18:58:28.929303+0000","last_unstale":"2023-06-10T18:58:31.032991+0000","last_undegraded":"2023-06-10T18:58:31.032991+0000","last_fullsized":"2023-06-10T18:58:31.032991+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:46:55.701041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941378+0000","last_change":"2023-06-10T18:58:29.332724+0000","last_active":"2023-06-10T18:58:30.941378+0000","last_peered":"2023-06-10T18:58:30.941378+0000","last_clean":"2023-06-10T18:58:30.941378+0000","last_became_active":"2023-06-10T18:58:29.327203+0000","last_became_peered":"2023-06-10T18:58:29.327203+0000","last_unstale":"2023-06-10T18:58:30.941378+0000","last_undegraded":"2023-06-10T18:58:30.941378+0000","last_fullsized":"2023-06-10T18:58:30.941378+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:21:56.636502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"171'6","reported_seq":19,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033219+0000","last_change":"2023-06-10T18:58:28.925504+0000","last_active":"2023-06-10T18:58:31.033219+0000","last_peered":"2023-06-10T18:58:31.033219+0000","last_clean":"2023-06-10T18:58:31.033219+0000","last_became_active":"2023-06-10T18:58:28.925118+0000","last_became_peered":"2023-06-10T18:58:28.925118+0000","last_unstale":"2023-06-10T18:58:31.033219+0000","last_undegraded":"2023-06-10T18:58:31.033219+0000","last_fullsized":"2023-06-10T18:58:31.033219+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:04.555847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033177+0000","last_change":"2023-06-10T18:58:29.336138+0000","last_active":"2023-06-10T18:58:31.033177+0000","last_peered":"2023-06-10T18:58:31.033177+0000","last_clean":"2023-06-10T18:58:31.033177+0000","last_became_active":"2023-06-10T18:58:29.335874+0000","last_became_peered":"2023-06-10T18:58:29.335874+0000","last_unstale":"2023-06-10T18:58:31.033177+0000","last_undegraded":"2023-06-10T18:58:31.033177+0000","last_fullsized":"2023-06-10T18:58:31.033177+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:57.905042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034523+0000","last_change":"2023-06-10T18:58:28.930171+0000","last_active":"2023-06-10T18:58:31.034523+0000","last_peered":"2023-06-10T18:58:31.034523+0000","last_clean":"2023-06-10T18:58:31.034523+0000","last_became_active":"2023-06-10T18:58:28.929795+0000","last_became_peered":"2023-06-10T18:58:28.929795+0000","last_unstale":"2023-06-10T18:58:31.034523+0000","last_undegraded":"2023-06-10T18:58:31.034523+0000","last_fullsized":"2023-06-10T18:58:31.034523+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:00.845042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326421+0000","last_change":"2023-06-10T18:58:29.338140+0000","last_active":"2023-06-10T18:58:30.326421+0000","last_peered":"2023-06-10T18:58:30.326421+0000","last_clean":"2023-06-10T18:58:30.326421+0000","last_became_active":"2023-06-10T18:58:29.337640+0000","last_became_peered":"2023-06-10T18:58:29.337640+0000","last_unstale":"2023-06-10T18:58:30.326421+0000","last_undegraded":"2023-06-10T18:58:30.326421+0000","last_fullsized":"2023-06-10T18:58:30.326421+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:06:20.472351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941340+0000","last_change":"2023-06-10T18:58:28.925333+0000","last_active":"2023-06-10T18:58:30.941340+0000","last_peered":"2023-06-10T18:58:30.941340+0000","last_clean":"2023-06-10T18:58:30.941340+0000","last_became_active":"2023-06-10T18:58:28.925070+0000","last_became_peered":"2023-06-10T18:58:28.925070+0000","last_unstale":"2023-06-10T18:58:30.941340+0000","last_undegraded":"2023-06-10T18:58:30.941340+0000","last_fullsized":"2023-06-10T18:58:30.941340+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:29:09.477953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325336+0000","last_change":"2023-06-10T18:58:29.328323+0000","last_active":"2023-06-10T18:58:30.325336+0000","last_peered":"2023-06-10T18:58:30.325336+0000","last_clean":"2023-06-10T18:58:30.325336+0000","last_became_active":"2023-06-10T18:58:29.328043+0000","last_became_peered":"2023-06-10T18:58:29.328043+0000","last_unstale":"2023-06-10T18:58:30.325336+0000","last_undegraded":"2023-06-10T18:58:30.325336+0000","last_fullsized":"2023-06-10T18:58:30.325336+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:57.905042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.319978+0000","last_change":"2023-06-10T18:58:28.926858+0000","last_active":"2023-06-10T18:58:30.319978+0000","last_peered":"2023-06-10T18:58:30.319978+0000","last_clean":"2023-06-10T18:58:30.319978+0000","last_became_active":"2023-06-10T18:58:28.926622+0000","last_became_peered":"2023-06-10T18:58:28.926622+0000","last_unstale":"2023-06-10T18:58:30.319978+0000","last_undegraded":"2023-06-10T18:58:30.319978+0000","last_fullsized":"2023-06-10T18:58:30.319978+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:26:58.521384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941119+0000","last_change":"2023-06-10T18:58:29.336086+0000","last_active":"2023-06-10T18:58:30.941119+0000","last_peered":"2023-06-10T18:58:30.941119+0000","last_clean":"2023-06-10T18:58:30.941119+0000","last_became_active":"2023-06-10T18:58:29.335834+0000","last_became_peered":"2023-06-10T18:58:29.335834+0000","last_unstale":"2023-06-10T18:58:30.941119+0000","last_undegraded":"2023-06-10T18:58:30.941119+0000","last_fullsized":"2023-06-10T18:58:30.941119+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:12:30.894743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"171'8","reported_seq":21,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033901+0000","last_change":"2023-06-10T18:58:28.932267+0000","last_active":"2023-06-10T18:58:31.033901+0000","last_peered":"2023-06-10T18:58:31.033901+0000","last_clean":"2023-06-10T18:58:31.033901+0000","last_became_active":"2023-06-10T18:58:28.931885+0000","last_became_peered":"2023-06-10T18:58:28.931885+0000","last_unstale":"2023-06-10T18:58:31.033901+0000","last_undegraded":"2023-06-10T18:58:31.033901+0000","last_fullsized":"2023-06-10T18:58:31.033901+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:36:36.219963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941379+0000","last_change":"2023-06-10T18:58:29.332794+0000","last_active":"2023-06-10T18:58:30.941379+0000","last_peered":"2023-06-10T18:58:30.941379+0000","last_clean":"2023-06-10T18:58:30.941379+0000","last_became_active":"2023-06-10T18:58:29.332535+0000","last_became_peered":"2023-06-10T18:58:29.332535+0000","last_unstale":"2023-06-10T18:58:30.941379+0000","last_undegraded":"2023-06-10T18:58:30.941379+0000","last_fullsized":"2023-06-10T18:58:30.941379+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:55.664298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033713+0000","last_change":"2023-06-10T18:58:28.928078+0000","last_active":"2023-06-10T18:58:31.033713+0000","last_peered":"2023-06-10T18:58:31.033713+0000","last_clean":"2023-06-10T18:58:31.033713+0000","last_became_active":"2023-06-10T18:58:28.927805+0000","last_became_peered":"2023-06-10T18:58:28.927805+0000","last_unstale":"2023-06-10T18:58:31.033713+0000","last_undegraded":"2023-06-10T18:58:31.033713+0000","last_fullsized":"2023-06-10T18:58:31.033713+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:20:48.098992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326185+0000","last_change":"2023-06-10T18:58:29.337015+0000","last_active":"2023-06-10T18:58:30.326185+0000","last_peered":"2023-06-10T18:58:30.326185+0000","last_clean":"2023-06-10T18:58:30.326185+0000","last_became_active":"2023-06-10T18:58:29.336720+0000","last_became_peered":"2023-06-10T18:58:29.336720+0000","last_unstale":"2023-06-10T18:58:30.326185+0000","last_undegraded":"2023-06-10T18:58:30.326185+0000","last_fullsized":"2023-06-10T18:58:30.326185+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:44:54.912807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320235+0000","last_change":"2023-06-10T18:58:28.925424+0000","last_active":"2023-06-10T18:58:30.320235+0000","last_peered":"2023-06-10T18:58:30.320235+0000","last_clean":"2023-06-10T18:58:30.320235+0000","last_became_active":"2023-06-10T18:58:28.925177+0000","last_became_peered":"2023-06-10T18:58:28.925177+0000","last_unstale":"2023-06-10T18:58:30.320235+0000","last_undegraded":"2023-06-10T18:58:30.320235+0000","last_fullsized":"2023-06-10T18:58:30.320235+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:14:03.099551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325465+0000","last_change":"2023-06-10T18:58:29.329690+0000","last_active":"2023-06-10T18:58:30.325465+0000","last_peered":"2023-06-10T18:58:30.325465+0000","last_clean":"2023-06-10T18:58:30.325465+0000","last_became_active":"2023-06-10T18:58:29.329374+0000","last_became_peered":"2023-06-10T18:58:29.329374+0000","last_unstale":"2023-06-10T18:58:30.325465+0000","last_undegraded":"2023-06-10T18:58:30.325465+0000","last_fullsized":"2023-06-10T18:58:30.325465+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:47:49.210752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034892+0000","last_change":"2023-06-10T18:58:28.930054+0000","last_active":"2023-06-10T18:58:31.034892+0000","last_peered":"2023-06-10T18:58:31.034892+0000","last_clean":"2023-06-10T18:58:31.034892+0000","last_became_active":"2023-06-10T18:58:28.929768+0000","last_became_peered":"2023-06-10T18:58:28.929768+0000","last_unstale":"2023-06-10T18:58:31.034892+0000","last_undegraded":"2023-06-10T18:58:31.034892+0000","last_fullsized":"2023-06-10T18:58:31.034892+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:56:22.879300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033850+0000","last_change":"2023-06-10T18:58:29.330698+0000","last_active":"2023-06-10T18:58:31.033850+0000","last_peered":"2023-06-10T18:58:31.033850+0000","last_clean":"2023-06-10T18:58:31.033850+0000","last_became_active":"2023-06-10T18:58:29.330393+0000","last_became_peered":"2023-06-10T18:58:29.330393+0000","last_unstale":"2023-06-10T18:58:31.033850+0000","last_undegraded":"2023-06-10T18:58:31.033850+0000","last_fullsized":"2023-06-10T18:58:31.033850+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:37.201110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326798+0000","last_change":"2023-06-10T18:58:28.927198+0000","last_active":"2023-06-10T18:58:30.326798+0000","last_peered":"2023-06-10T18:58:30.326798+0000","last_clean":"2023-06-10T18:58:30.326798+0000","last_became_active":"2023-06-10T18:58:28.926933+0000","last_became_peered":"2023-06-10T18:58:28.926933+0000","last_unstale":"2023-06-10T18:58:30.326798+0000","last_undegraded":"2023-06-10T18:58:30.326798+0000","last_fullsized":"2023-06-10T18:58:30.326798+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:55.658148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941199+0000","last_change":"2023-06-10T18:58:29.335284+0000","last_active":"2023-06-10T18:58:30.941199+0000","last_peered":"2023-06-10T18:58:30.941199+0000","last_clean":"2023-06-10T18:58:30.941199+0000","last_became_active":"2023-06-10T18:58:29.334805+0000","last_became_peered":"2023-06-10T18:58:29.334805+0000","last_unstale":"2023-06-10T18:58:30.941199+0000","last_undegraded":"2023-06-10T18:58:30.941199+0000","last_fullsized":"2023-06-10T18:58:30.941199+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:42:56.665019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033485+0000","last_change":"2023-06-10T18:58:28.925764+0000","last_active":"2023-06-10T18:58:31.033485+0000","last_peered":"2023-06-10T18:58:31.033485+0000","last_clean":"2023-06-10T18:58:31.033485+0000","last_became_active":"2023-06-10T18:58:28.925498+0000","last_became_peered":"2023-06-10T18:58:28.925498+0000","last_unstale":"2023-06-10T18:58:31.033485+0000","last_undegraded":"2023-06-10T18:58:31.033485+0000","last_fullsized":"2023-06-10T18:58:31.033485+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:18.397233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326516+0000","last_change":"2023-06-10T18:58:29.331972+0000","last_active":"2023-06-10T18:58:30.326516+0000","last_peered":"2023-06-10T18:58:30.326516+0000","last_clean":"2023-06-10T18:58:30.326516+0000","last_became_active":"2023-06-10T18:58:29.331665+0000","last_became_peered":"2023-06-10T18:58:29.331665+0000","last_unstale":"2023-06-10T18:58:30.326516+0000","last_undegraded":"2023-06-10T18:58:30.326516+0000","last_fullsized":"2023-06-10T18:58:30.326516+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:41:31.345719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033315+0000","last_change":"2023-06-10T18:58:28.931781+0000","last_active":"2023-06-10T18:58:31.033315+0000","last_peered":"2023-06-10T18:58:31.033315+0000","last_clean":"2023-06-10T18:58:31.033315+0000","last_became_active":"2023-06-10T18:58:28.931545+0000","last_became_peered":"2023-06-10T18:58:28.931545+0000","last_unstale":"2023-06-10T18:58:31.033315+0000","last_undegraded":"2023-06-10T18:58:31.033315+0000","last_fullsized":"2023-06-10T18:58:31.033315+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:19:49.031474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033833+0000","last_change":"2023-06-10T18:58:29.337012+0000","last_active":"2023-06-10T18:58:31.033833+0000","last_peered":"2023-06-10T18:58:31.033833+0000","last_clean":"2023-06-10T18:58:31.033833+0000","last_became_active":"2023-06-10T18:58:29.336693+0000","last_became_peered":"2023-06-10T18:58:29.336693+0000","last_unstale":"2023-06-10T18:58:31.033833+0000","last_undegraded":"2023-06-10T18:58:31.033833+0000","last_fullsized":"2023-06-10T18:58:31.033833+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:44:46.684104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033438+0000","last_change":"2023-06-10T18:58:28.925774+0000","last_active":"2023-06-10T18:58:31.033438+0000","last_peered":"2023-06-10T18:58:31.033438+0000","last_clean":"2023-06-10T18:58:31.033438+0000","last_became_active":"2023-06-10T18:58:28.925095+0000","last_became_peered":"2023-06-10T18:58:28.925095+0000","last_unstale":"2023-06-10T18:58:31.033438+0000","last_undegraded":"2023-06-10T18:58:31.033438+0000","last_fullsized":"2023-06-10T18:58:31.033438+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:18.390184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032866+0000","last_change":"2023-06-10T18:58:29.331135+0000","last_active":"2023-06-10T18:58:31.032866+0000","last_peered":"2023-06-10T18:58:31.032866+0000","last_clean":"2023-06-10T18:58:31.032866+0000","last_became_active":"2023-06-10T18:58:29.330894+0000","last_became_peered":"2023-06-10T18:58:29.330894+0000","last_unstale":"2023-06-10T18:58:31.032866+0000","last_undegraded":"2023-06-10T18:58:31.032866+0000","last_fullsized":"2023-06-10T18:58:31.032866+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:07:47.288837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325384+0000","last_change":"2023-06-10T18:58:28.930099+0000","last_active":"2023-06-10T18:58:30.325384+0000","last_peered":"2023-06-10T18:58:30.325384+0000","last_clean":"2023-06-10T18:58:30.325384+0000","last_became_active":"2023-06-10T18:58:28.929828+0000","last_became_peered":"2023-06-10T18:58:28.929828+0000","last_unstale":"2023-06-10T18:58:30.325384+0000","last_undegraded":"2023-06-10T18:58:30.325384+0000","last_fullsized":"2023-06-10T18:58:30.325384+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:04.555847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033000+0000","last_change":"2023-06-10T18:58:29.331310+0000","last_active":"2023-06-10T18:58:31.033000+0000","last_peered":"2023-06-10T18:58:31.033000+0000","last_clean":"2023-06-10T18:58:31.033000+0000","last_became_active":"2023-06-10T18:58:29.331050+0000","last_became_peered":"2023-06-10T18:58:29.331050+0000","last_unstale":"2023-06-10T18:58:31.033000+0000","last_undegraded":"2023-06-10T18:58:31.033000+0000","last_fullsized":"2023-06-10T18:58:31.033000+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:34:54.552309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034902+0000","last_change":"2023-06-10T18:58:28.929910+0000","last_active":"2023-06-10T18:58:31.034902+0000","last_peered":"2023-06-10T18:58:31.034902+0000","last_clean":"2023-06-10T18:58:31.034902+0000","last_became_active":"2023-06-10T18:58:28.929616+0000","last_became_peered":"2023-06-10T18:58:28.929616+0000","last_unstale":"2023-06-10T18:58:31.034902+0000","last_undegraded":"2023-06-10T18:58:31.034902+0000","last_fullsized":"2023-06-10T18:58:31.034902+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:56:54.340547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941602+0000","last_change":"2023-06-10T18:58:29.334526+0000","last_active":"2023-06-10T18:58:30.941602+0000","last_peered":"2023-06-10T18:58:30.941602+0000","last_clean":"2023-06-10T18:58:30.941602+0000","last_became_active":"2023-06-10T18:58:29.334254+0000","last_became_peered":"2023-06-10T18:58:29.334254+0000","last_unstale":"2023-06-10T18:58:30.941602+0000","last_undegraded":"2023-06-10T18:58:30.941602+0000","last_fullsized":"2023-06-10T18:58:30.941602+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:46:57.571994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941513+0000","last_change":"2023-06-10T18:58:28.927876+0000","last_active":"2023-06-10T18:58:30.941513+0000","last_peered":"2023-06-10T18:58:30.941513+0000","last_clean":"2023-06-10T18:58:30.941513+0000","last_became_active":"2023-06-10T18:58:28.927626+0000","last_became_peered":"2023-06-10T18:58:28.927626+0000","last_unstale":"2023-06-10T18:58:30.941513+0000","last_undegraded":"2023-06-10T18:58:30.941513+0000","last_fullsized":"2023-06-10T18:58:30.941513+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:30:46.683778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320230+0000","last_change":"2023-06-10T18:58:29.321139+0000","last_active":"2023-06-10T18:58:30.320230+0000","last_peered":"2023-06-10T18:58:30.320230+0000","last_clean":"2023-06-10T18:58:30.320230+0000","last_became_active":"2023-06-10T18:58:29.320894+0000","last_became_peered":"2023-06-10T18:58:29.320894+0000","last_unstale":"2023-06-10T18:58:30.320230+0000","last_undegraded":"2023-06-10T18:58:30.320230+0000","last_fullsized":"2023-06-10T18:58:30.320230+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:27:25.201509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941484+0000","last_change":"2023-06-10T18:58:28.925916+0000","last_active":"2023-06-10T18:58:30.941484+0000","last_peered":"2023-06-10T18:58:30.941484+0000","last_clean":"2023-06-10T18:58:30.941484+0000","last_became_active":"2023-06-10T18:58:28.925623+0000","last_became_peered":"2023-06-10T18:58:28.925623+0000","last_unstale":"2023-06-10T18:58:30.941484+0000","last_undegraded":"2023-06-10T18:58:30.941484+0000","last_fullsized":"2023-06-10T18:58:30.941484+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:18:00.270130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320137+0000","last_change":"2023-06-10T18:58:29.325738+0000","last_active":"2023-06-10T18:58:30.320137+0000","last_peered":"2023-06-10T18:58:30.320137+0000","last_clean":"2023-06-10T18:58:30.320137+0000","last_became_active":"2023-06-10T18:58:29.325474+0000","last_became_peered":"2023-06-10T18:58:29.325474+0000","last_unstale":"2023-06-10T18:58:30.320137+0000","last_undegraded":"2023-06-10T18:58:30.320137+0000","last_fullsized":"2023-06-10T18:58:30.320137+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:54:33.547054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320363+0000","last_change":"2023-06-10T18:58:28.925914+0000","last_active":"2023-06-10T18:58:30.320363+0000","last_peered":"2023-06-10T18:58:30.320363+0000","last_clean":"2023-06-10T18:58:30.320363+0000","last_became_active":"2023-06-10T18:58:28.925668+0000","last_became_peered":"2023-06-10T18:58:28.925668+0000","last_unstale":"2023-06-10T18:58:30.320363+0000","last_undegraded":"2023-06-10T18:58:30.320363+0000","last_fullsized":"2023-06-10T18:58:30.320363+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:12.959466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034586+0000","last_change":"2023-06-10T18:58:29.335655+0000","last_active":"2023-06-10T18:58:31.034586+0000","last_peered":"2023-06-10T18:58:31.034586+0000","last_clean":"2023-06-10T18:58:31.034586+0000","last_became_active":"2023-06-10T18:58:29.334341+0000","last_became_peered":"2023-06-10T18:58:29.334341+0000","last_unstale":"2023-06-10T18:58:31.034586+0000","last_undegraded":"2023-06-10T18:58:31.034586+0000","last_fullsized":"2023-06-10T18:58:31.034586+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:00:50.629285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325598+0000","last_change":"2023-06-10T18:58:28.926873+0000","last_active":"2023-06-10T18:58:30.325598+0000","last_peered":"2023-06-10T18:58:30.325598+0000","last_clean":"2023-06-10T18:58:30.325598+0000","last_became_active":"2023-06-10T18:58:28.926574+0000","last_became_peered":"2023-06-10T18:58:28.926574+0000","last_unstale":"2023-06-10T18:58:30.325598+0000","last_undegraded":"2023-06-10T18:58:30.325598+0000","last_fullsized":"2023-06-10T18:58:30.325598+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:50:09.722300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034682+0000","last_change":"2023-06-10T18:58:29.336989+0000","last_active":"2023-06-10T18:58:31.034682+0000","last_peered":"2023-06-10T18:58:31.034682+0000","last_clean":"2023-06-10T18:58:31.034682+0000","last_became_active":"2023-06-10T18:58:29.336719+0000","last_became_peered":"2023-06-10T18:58:29.336719+0000","last_unstale":"2023-06-10T18:58:31.034682+0000","last_undegraded":"2023-06-10T18:58:31.034682+0000","last_fullsized":"2023-06-10T18:58:31.034682+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:37:37.855621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325595+0000","last_change":"2023-06-10T18:58:28.925738+0000","last_active":"2023-06-10T18:58:30.325595+0000","last_peered":"2023-06-10T18:58:30.325595+0000","last_clean":"2023-06-10T18:58:30.325595+0000","last_became_active":"2023-06-10T18:58:28.925499+0000","last_became_peered":"2023-06-10T18:58:28.925499+0000","last_unstale":"2023-06-10T18:58:30.325595+0000","last_undegraded":"2023-06-10T18:58:30.325595+0000","last_fullsized":"2023-06-10T18:58:30.325595+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:42:47.934788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034074+0000","last_change":"2023-06-10T18:58:29.331690+0000","last_active":"2023-06-10T18:58:31.034074+0000","last_peered":"2023-06-10T18:58:31.034074+0000","last_clean":"2023-06-10T18:58:31.034074+0000","last_became_active":"2023-06-10T18:58:29.331421+0000","last_became_peered":"2023-06-10T18:58:29.331421+0000","last_unstale":"2023-06-10T18:58:31.034074+0000","last_undegraded":"2023-06-10T18:58:31.034074+0000","last_fullsized":"2023-06-10T18:58:31.034074+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:11:09.571556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034441+0000","last_change":"2023-06-10T18:58:28.924203+0000","last_active":"2023-06-10T18:58:31.034441+0000","last_peered":"2023-06-10T18:58:31.034441+0000","last_clean":"2023-06-10T18:58:31.034441+0000","last_became_active":"2023-06-10T18:58:28.923525+0000","last_became_peered":"2023-06-10T18:58:28.923525+0000","last_unstale":"2023-06-10T18:58:31.034441+0000","last_undegraded":"2023-06-10T18:58:31.034441+0000","last_fullsized":"2023-06-10T18:58:31.034441+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:33:56.336777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.032871+0000","last_change":"2023-06-10T18:58:29.329662+0000","last_active":"2023-06-10T18:58:31.032871+0000","last_peered":"2023-06-10T18:58:31.032871+0000","last_clean":"2023-06-10T18:58:31.032871+0000","last_became_active":"2023-06-10T18:58:29.329402+0000","last_became_peered":"2023-06-10T18:58:29.329402+0000","last_unstale":"2023-06-10T18:58:31.032871+0000","last_undegraded":"2023-06-10T18:58:31.032871+0000","last_fullsized":"2023-06-10T18:58:31.032871+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:34:55.202966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033264+0000","last_change":"2023-06-10T18:58:28.925396+0000","last_active":"2023-06-10T18:58:31.033264+0000","last_peered":"2023-06-10T18:58:31.033264+0000","last_clean":"2023-06-10T18:58:31.033264+0000","last_became_active":"2023-06-10T18:58:28.925033+0000","last_became_peered":"2023-06-10T18:58:28.925033+0000","last_unstale":"2023-06-10T18:58:31.033264+0000","last_undegraded":"2023-06-10T18:58:31.033264+0000","last_fullsized":"2023-06-10T18:58:31.033264+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:33:35.798577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941574+0000","last_change":"2023-06-10T18:58:29.335959+0000","last_active":"2023-06-10T18:58:30.941574+0000","last_peered":"2023-06-10T18:58:30.941574+0000","last_clean":"2023-06-10T18:58:30.941574+0000","last_became_active":"2023-06-10T18:58:29.335703+0000","last_became_peered":"2023-06-10T18:58:29.335703+0000","last_unstale":"2023-06-10T18:58:30.941574+0000","last_undegraded":"2023-06-10T18:58:30.941574+0000","last_fullsized":"2023-06-10T18:58:30.941574+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:04:18.204887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325948+0000","last_change":"2023-06-10T18:58:28.924006+0000","last_active":"2023-06-10T18:58:30.325948+0000","last_peered":"2023-06-10T18:58:30.325948+0000","last_clean":"2023-06-10T18:58:30.325948+0000","last_became_active":"2023-06-10T18:58:28.923748+0000","last_became_peered":"2023-06-10T18:58:28.923748+0000","last_unstale":"2023-06-10T18:58:30.325948+0000","last_undegraded":"2023-06-10T18:58:30.325948+0000","last_fullsized":"2023-06-10T18:58:30.325948+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:52.907184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033315+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:58:31.033315+0000","last_peered":"2023-06-10T18:58:31.033315+0000","last_clean":"2023-06-10T18:58:31.033315+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:58:31.033315+0000","last_undegraded":"2023-06-10T18:58:31.033315+0000","last_fullsized":"2023-06-10T18:58:31.033315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:28:22.555151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326198+0000","last_change":"2023-06-10T18:58:29.338528+0000","last_active":"2023-06-10T18:58:30.326198+0000","last_peered":"2023-06-10T18:58:30.326198+0000","last_clean":"2023-06-10T18:58:30.326198+0000","last_became_active":"2023-06-10T18:58:29.338238+0000","last_became_peered":"2023-06-10T18:58:29.338238+0000","last_unstale":"2023-06-10T18:58:30.326198+0000","last_undegraded":"2023-06-10T18:58:30.326198+0000","last_fullsized":"2023-06-10T18:58:30.326198+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:21:31.181768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033705+0000","last_change":"2023-06-10T18:58:28.932027+0000","last_active":"2023-06-10T18:58:31.033705+0000","last_peered":"2023-06-10T18:58:31.033705+0000","last_clean":"2023-06-10T18:58:31.033705+0000","last_became_active":"2023-06-10T18:58:28.931781+0000","last_became_peered":"2023-06-10T18:58:28.931781+0000","last_unstale":"2023-06-10T18:58:31.033705+0000","last_undegraded":"2023-06-10T18:58:31.033705+0000","last_fullsized":"2023-06-10T18:58:31.033705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:13:35.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":315,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325704+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:58:30.325704+0000","last_peered":"2023-06-10T18:58:30.325704+0000","last_clean":"2023-06-10T18:58:30.325704+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:58:30.325704+0000","last_undegraded":"2023-06-10T18:58:30.325704+0000","last_fullsized":"2023-06-10T18:58:30.325704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:33:53.017596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.941311+0000","last_change":"2023-06-10T18:58:29.329954+0000","last_active":"2023-06-10T18:58:30.941311+0000","last_peered":"2023-06-10T18:58:30.941311+0000","last_clean":"2023-06-10T18:58:30.941311+0000","last_became_active":"2023-06-10T18:58:29.329420+0000","last_became_peered":"2023-06-10T18:58:29.329420+0000","last_unstale":"2023-06-10T18:58:30.941311+0000","last_undegraded":"2023-06-10T18:58:30.941311+0000","last_fullsized":"2023-06-10T18:58:30.941311+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:26.200027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033188+0000","last_change":"2023-06-10T18:58:28.928009+0000","last_active":"2023-06-10T18:58:31.033188+0000","last_peered":"2023-06-10T18:58:31.033188+0000","last_clean":"2023-06-10T18:58:31.033188+0000","last_became_active":"2023-06-10T18:58:28.927750+0000","last_became_peered":"2023-06-10T18:58:28.927750+0000","last_unstale":"2023-06-10T18:58:31.033188+0000","last_undegraded":"2023-06-10T18:58:31.033188+0000","last_fullsized":"2023-06-10T18:58:31.033188+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:38:52.650003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":269,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033025+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:58:31.033025+0000","last_peered":"2023-06-10T18:58:31.033025+0000","last_clean":"2023-06-10T18:58:31.033025+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:58:31.033025+0000","last_undegraded":"2023-06-10T18:58:31.033025+0000","last_fullsized":"2023-06-10T18:58:31.033025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:19:27.951425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.940999+0000","last_change":"2023-06-10T18:58:29.332300+0000","last_active":"2023-06-10T18:58:30.940999+0000","last_peered":"2023-06-10T18:58:30.940999+0000","last_clean":"2023-06-10T18:58:30.940999+0000","last_became_active":"2023-06-10T18:58:29.327341+0000","last_became_peered":"2023-06-10T18:58:29.327341+0000","last_unstale":"2023-06-10T18:58:30.940999+0000","last_undegraded":"2023-06-10T18:58:30.940999+0000","last_fullsized":"2023-06-10T18:58:30.940999+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:06:46.331984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034660+0000","last_change":"2023-06-10T18:58:28.934714+0000","last_active":"2023-06-10T18:58:31.034660+0000","last_peered":"2023-06-10T18:58:31.034660+0000","last_clean":"2023-06-10T18:58:31.034660+0000","last_became_active":"2023-06-10T18:58:28.934412+0000","last_became_peered":"2023-06-10T18:58:28.934412+0000","last_unstale":"2023-06-10T18:58:31.034660+0000","last_undegraded":"2023-06-10T18:58:31.034660+0000","last_fullsized":"2023-06-10T18:58:31.034660+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:22:14.017996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":315,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325823+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:58:30.325823+0000","last_peered":"2023-06-10T18:58:30.325823+0000","last_clean":"2023-06-10T18:58:30.325823+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:58:30.325823+0000","last_undegraded":"2023-06-10T18:58:30.325823+0000","last_fullsized":"2023-06-10T18:58:30.325823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:18:10.507632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":442,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034787+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:58:31.034787+0000","last_peered":"2023-06-10T18:58:31.034787+0000","last_clean":"2023-06-10T18:58:31.034787+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:58:31.034787+0000","last_undegraded":"2023-06-10T18:58:31.034787+0000","last_fullsized":"2023-06-10T18:58:31.034787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:31:04.090060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320151+0000","last_change":"2023-06-10T18:58:29.320040+0000","last_active":"2023-06-10T18:58:30.320151+0000","last_peered":"2023-06-10T18:58:30.320151+0000","last_clean":"2023-06-10T18:58:30.320151+0000","last_became_active":"2023-06-10T18:58:29.319768+0000","last_became_peered":"2023-06-10T18:58:29.319768+0000","last_unstale":"2023-06-10T18:58:30.320151+0000","last_undegraded":"2023-06-10T18:58:30.320151+0000","last_fullsized":"2023-06-10T18:58:30.320151+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:44:31.201848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"171'1","reported_seq":14,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320167+0000","last_change":"2023-06-10T18:58:28.922699+0000","last_active":"2023-06-10T18:58:30.320167+0000","last_peered":"2023-06-10T18:58:30.320167+0000","last_clean":"2023-06-10T18:58:30.320167+0000","last_became_active":"2023-06-10T18:58:28.922425+0000","last_became_peered":"2023-06-10T18:58:28.922425+0000","last_unstale":"2023-06-10T18:58:30.320167+0000","last_undegraded":"2023-06-10T18:58:30.320167+0000","last_fullsized":"2023-06-10T18:58:30.320167+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:40:19.926035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034668+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:58:31.034668+0000","last_peered":"2023-06-10T18:58:31.034668+0000","last_clean":"2023-06-10T18:58:31.034668+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:58:31.034668+0000","last_undegraded":"2023-06-10T18:58:31.034668+0000","last_fullsized":"2023-06-10T18:58:31.034668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:31:10.616083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.320039+0000","last_change":"2023-06-10T18:58:29.326002+0000","last_active":"2023-06-10T18:58:30.320039+0000","last_peered":"2023-06-10T18:58:30.320039+0000","last_clean":"2023-06-10T18:58:30.320039+0000","last_became_active":"2023-06-10T18:58:29.325588+0000","last_became_peered":"2023-06-10T18:58:29.325588+0000","last_unstale":"2023-06-10T18:58:30.320039+0000","last_undegraded":"2023-06-10T18:58:30.320039+0000","last_fullsized":"2023-06-10T18:58:30.320039+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:39:24.026553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"171'4","reported_seq":17,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033649+0000","last_change":"2023-06-10T18:58:28.925351+0000","last_active":"2023-06-10T18:58:31.033649+0000","last_peered":"2023-06-10T18:58:31.033649+0000","last_clean":"2023-06-10T18:58:31.033649+0000","last_became_active":"2023-06-10T18:58:28.925089+0000","last_became_peered":"2023-06-10T18:58:28.925089+0000","last_unstale":"2023-06-10T18:58:31.033649+0000","last_undegraded":"2023-06-10T18:58:31.033649+0000","last_fullsized":"2023-06-10T18:58:31.033649+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:28:42.293438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":285,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034744+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:58:31.034744+0000","last_peered":"2023-06-10T18:58:31.034744+0000","last_clean":"2023-06-10T18:58:31.034744+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:58:31.034744+0000","last_undegraded":"2023-06-10T18:58:31.034744+0000","last_fullsized":"2023-06-10T18:58:31.034744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:35:28.956704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.325869+0000","last_change":"2023-06-10T18:58:29.332169+0000","last_active":"2023-06-10T18:58:30.325869+0000","last_peered":"2023-06-10T18:58:30.325869+0000","last_clean":"2023-06-10T18:58:30.325869+0000","last_became_active":"2023-06-10T18:58:29.331878+0000","last_became_peered":"2023-06-10T18:58:29.331878+0000","last_unstale":"2023-06-10T18:58:30.325869+0000","last_undegraded":"2023-06-10T18:58:30.325869+0000","last_fullsized":"2023-06-10T18:58:30.325869+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:32:59.801173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034294+0000","last_change":"2023-06-10T18:58:28.925704+0000","last_active":"2023-06-10T18:58:31.034294+0000","last_peered":"2023-06-10T18:58:31.034294+0000","last_clean":"2023-06-10T18:58:31.034294+0000","last_became_active":"2023-06-10T18:58:28.925383+0000","last_became_peered":"2023-06-10T18:58:28.925383+0000","last_unstale":"2023-06-10T18:58:31.034294+0000","last_undegraded":"2023-06-10T18:58:31.034294+0000","last_fullsized":"2023-06-10T18:58:31.034294+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:33:33.783242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034310+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:58:31.034310+0000","last_peered":"2023-06-10T18:58:31.034310+0000","last_clean":"2023-06-10T18:58:31.034310+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:58:31.034310+0000","last_undegraded":"2023-06-10T18:58:31.034310+0000","last_fullsized":"2023-06-10T18:58:31.034310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:30:28.526299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:30.326420+0000","last_change":"2023-06-10T18:58:29.333782+0000","last_active":"2023-06-10T18:58:30.326420+0000","last_peered":"2023-06-10T18:58:30.326420+0000","last_clean":"2023-06-10T18:58:30.326420+0000","last_became_active":"2023-06-10T18:58:29.333508+0000","last_became_peered":"2023-06-10T18:58:29.333508+0000","last_unstale":"2023-06-10T18:58:30.326420+0000","last_undegraded":"2023-06-10T18:58:30.326420+0000","last_fullsized":"2023-06-10T18:58:30.326420+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:28.906535+0000","last_clean_scrub_stamp":"2023-06-10T18:58:28.906535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:00:50.392702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"171'2","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.033893+0000","last_change":"2023-06-10T18:58:28.927435+0000","last_active":"2023-06-10T18:58:31.033893+0000","last_peered":"2023-06-10T18:58:31.033893+0000","last_clean":"2023-06-10T18:58:31.033893+0000","last_became_active":"2023-06-10T18:58:28.926429+0000","last_became_peered":"2023-06-10T18:58:28.926429+0000","last_unstale":"2023-06-10T18:58:31.033893+0000","last_undegraded":"2023-06-10T18:58:31.033893+0000","last_fullsized":"2023-06-10T18:58:31.033893+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:58:27.899664+0000","last_clean_scrub_stamp":"2023-06-10T18:58:27.899664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:41:52.907184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-06-10T18:58:31.034220+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:58:31.034220+0000","last_peered":"2023-06-10T18:58:31.034220+0000","last_clean":"2023-06-10T18:58:31.034220+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:58:31.034220+0000","last_undegraded":"2023-06-10T18:58:31.034220+0000","last_fullsized":"2023-06-10T18:58:31.034220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:50:59.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33744,"kb_used_data":2696,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338096,"statfs":{"total":96636764160,"available":96602210304,"internally_reserved":0,"allocated":2760704,"data_stored":1651510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.54500000000000004},"min":{"1min":0.38100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64900000000000002,"5min":0.749,"15min":0.81499999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.73599999999999999,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":0.63600000000000001,"5min":0.76900000000000002,"15min":0.86099999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68000000000000005,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.51900000000000002},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.436},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52700000000000002,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.27300000000000002},"max":{"1min":0.66100000000000003,"5min":0.745,"15min":0.79000000000000004},"last":0.59299999999999997}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.53700000000000003},"min":{"1min":0.32300000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.65200000000000002,"5min":0.78200000000000003,"15min":0.78300000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.68600000000000005,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69399999999999995,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.96499999999999997},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.93899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.72399999999999998,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.498}]}]},{"osd":6,"up_from":13,"seq":55834575028,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33140,"kb_used_data":2092,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338700,"statfs":{"total":96636764160,"available":96602828800,"internally_reserved":0,"allocated":2142208,"data_stored":1053046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.504,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51500000000000001,"15min":0.51800000000000002},"min":{"1min":0.38700000000000001,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.309},"max":{"1min":0.60899999999999999,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.498},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.59999999999999998},"min":{"1min":0.45500000000000002,"5min":0.33600000000000002,"15min":0.29199999999999998},"max":{"1min":0.78400000000000003,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.57799999999999996},"min":{"1min":0.40300000000000002,"5min":0.36799999999999999,"15min":0.32200000000000001},"max":{"1min":0.73299999999999998,"5min":0.872,"15min":6.6689999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.70599999999999996,"5min":0.78800000000000003,"15min":1.252},"last":0.68200000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53200000000000003,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.747,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.83899999999999997,"5min":0.874,"15min":0.874},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.25,"15min":0.25},"max":{"1min":0.69999999999999996,"5min":0.746,"15min":0.89600000000000002},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.44500000000000001,"5min":0.312,"15min":0.307},"max":{"1min":0.81299999999999994,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.74099999999999999,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.47499999999999998,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33164,"kb_used_data":2116,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338676,"statfs":{"total":96636764160,"available":96602804224,"internally_reserved":0,"allocated":2166784,"data_stored":1061583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.32500000000000001},"min":{"1min":0.216,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.42999999999999999,"5min":0.51800000000000002,"15min":0.753},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34599999999999997,"15min":0.35099999999999998},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.219},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.318}]},{"osd":2,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.34200000000000003},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.503,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35199999999999998,"15min":0.373},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20799999999999999},"max":{"1min":0.503,"5min":0.56000000000000005,"15min":0.71899999999999997},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53800000000000003,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.39100000000000001},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88500000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.62,"15min":0.624},"min":{"1min":0.46899999999999997,"5min":0.434,"15min":0.34899999999999998},"max":{"1min":0.89400000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82799999999999996,"5min":0.85999999999999999,"15min":1.008},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.78000000000000003,"5min":1.206,"15min":1.206},"last":0.625},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.873,"15min":0.88700000000000001},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.41399999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84199999999999997,"5min":1.224,"15min":1.224},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77000000000000002,"5min":0.83899999999999997,"15min":0.97099999999999997},"last":0.59099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575027,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33736,"kb_used_data":2688,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338104,"statfs":{"total":96636764160,"available":96602218496,"internally_reserved":0,"allocated":2752512,"data_stored":1650409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33900000000000002,"15min":0.33800000000000002},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.47299999999999998,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.34999999999999998},"min":{"1min":0.23400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.505,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.308}]},{"osd":2,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.435,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52600000000000002,"5min":11.439,"15min":11.439},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.48199999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.54000000000000004,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.27300000000000002,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.48999999999999999,"5min":5.766,"15min":5.766},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.27000000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.64800000000000002,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79700000000000004,"5min":2.802,"15min":2.802},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.999},"last":0.502},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.71799999999999997,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77100000000000002,"5min":2.907,"15min":2.907},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58399999999999996,"15min":0.59899999999999998},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.71499999999999997,"5min":0.86099999999999999,"15min":1.363},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":1.0489999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72899999999999998,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.44700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575028,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":2108,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2158592,"data_stored":1054759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.318,"15min":0.34200000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.45300000000000001,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.38500000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.48299999999999998,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.377},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47099999999999997,"5min":0.57599999999999996,"15min":2.847},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36699999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.45900000000000002,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.622}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34699999999999998,"15min":0.38500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.442,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.25800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54100000000000004,"5min":1.24,"15min":2.8999999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.61499999999999999},"min":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":1.208,"15min":5.742},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58499999999999996,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81200000000000006,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61799999999999999,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":6.593},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.78800000000000003,"5min":1.002,"15min":7.0880000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":0.79500000000000004,"5min":1.026,"15min":8.7850000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":1.171,"15min":5.6180000000000003},"last":0.76800000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.64300000000000002,"15min":0.66400000000000003},"min":{"1min":0.47399999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.84399999999999997,"5min":0.87,"15min":6.617},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.65900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.754}]}]},{"osd":3,"up_from":13,"seq":55834575027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":2120,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2170880,"data_stored":1061232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.32200000000000001,"15min":0.34200000000000003},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.48199999999999998,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34499999999999997,"15min":0.36199999999999999},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.20200000000000001},"max":{"1min":0.44400000000000001,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.254}]},{"osd":1,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34300000000000003,"15min":0.36099999999999999},"min":{"1min":0.23899999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.38100000000000001,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.372},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.379,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52000000000000002,"5min":0.64900000000000002,"15min":1.3},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.39600000000000002},"min":{"1min":0.28999999999999998,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.57799999999999996,"15min":1.109},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.39900000000000002,"15min":0.42599999999999999},"min":{"1min":0.29899999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55400000000000005,"5min":0.623,"15min":1.599},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57399999999999995,"15min":0.59599999999999997},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71299999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":0.72799999999999998,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.42699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.752,"5min":1.1100000000000001,"15min":1.613},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59599999999999997,"15min":0.63},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76300000000000001,"5min":1.095,"15min":3.802},"last":0.78800000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84999999999999998,"5min":0.86499999999999999,"15min":3.8180000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.39700000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":1.018,"15min":1.679},"last":0.628}]},{"osd":7,"last update":"Sat Jun 10 18:58:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":1.0549999999999999,"15min":1.47},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73599999999999999,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.66500000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33168,"kb_used_data":2120,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338672,"statfs":{"total":96636764160,"available":96602800128,"internally_reserved":0,"allocated":2170880,"data_stored":1062165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.71999999999999997,"5min":0.745,"15min":1.119},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.1639999999999999},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77800000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.873,"15min":0.96599999999999997},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.67100000000000004,"5min":0.72799999999999998,"15min":0.80800000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.33200000000000002},"max":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.624}]},{"osd":3,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.44500000000000001,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.79800000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.042},"last":0.65000000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.67800000000000005,"5min":0.94599999999999995,"15min":1.069},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.501,"15min":0.504},"min":{"1min":0.36299999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.88300000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.501},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.90100000000000002,"15min":1.1459999999999999},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575028,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33156,"kb_used_data":2108,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338684,"statfs":{"total":96636764160,"available":96602812416,"internally_reserved":0,"allocated":2158592,"data_stored":1061038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.81000000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51400000000000001,"15min":0.49299999999999999},"min":{"1min":0.41899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.83999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.627}]},{"osd":6,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.879},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.54900000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:58:33.812 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:58:33.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:58:33.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.967+0000 7f95045a9700 1 Processor -- start 2023-06-10T18:58:33.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.967+0000 7f95045a9700 1 -- start start 2023-06-10T18:58:33.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f95045a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94fc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f95045a9700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc06e250 con 0x7f94fc06cde0 2023-06-10T18:58:33.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f95045a9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc15c950 con 0x7f94fc06e6e0 2023-06-10T18:58:33.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f95045a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc0b1e80 con 0x7f94fc15c580 2023-06-10T18:58:33.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9502b46700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f94fc06cde0 0x7f94fc0ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38256/0 (socket says 172.21.15.104:38256) 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9502b46700 1 -- 172.21.15.104:0/2257100520 learned_addr learned my addr 172.21.15.104:0/2257100520 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9502345700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94fc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253073946 0 0) 0x7f94fc06e250 con 0x7f94fc06cde0 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e80035b0 con 0x7f94fc06cde0 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068201598 0 0) 0x7f94fc15c950 con 0x7f94fc06e6e0 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94fc06e250 con 0x7f94fc06e6e0 2023-06-10T18:58:33.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25678639 0 0) 0x7f94e80035b0 con 0x7f94fc06cde0 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 msgr2=0x7f94fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94fc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 >> v1:172.21.15.114:6789/0 conn(0x7f94fc06e6e0 legacy=0x7f94fc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc06d920 con 0x7f94fc06cde0 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f94f40019f0 con 0x7f94fc06cde0 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 >> v1:172.21.15.104:6789/0 conn(0x7f94fc06cde0 legacy=0x7f94fc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8003bc0 0x7f94e8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.968+0000 7f9501343700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8004850 0x7f94e8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc0b1e80 con 0x7f94e8003bc0 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e800bac0 con 0x7f94e8004850 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e800bc30 con 0x7f94fc15c580 2023-06-10T18:58:33.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8003bc0 0x7f94e8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501b44700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8004850 0x7f94e8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502b46700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 -- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 msgr2=0x7f94e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc15c580 0x7f94e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 -- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8004850 msgr2=0x7f94e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8004850 0x7f94e8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 -- 172.21.15.104:0/2257100520 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80035b0 con 0x7f94e8003bc0 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501b44700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8004850 0x7f94e8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9502345700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8003bc0 0x7f94e8003fb0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009b40 tx=0x7f94e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b168e665f8a71e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec005a20 con 0x7f94e8003bc0 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec005b80 con 0x7f94e8003bc0 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.969+0000 7f9501343700 1 -- 172.21.15.104:0/2257100520 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00e6c0 con 0x7f94e8003bc0 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 -- 172.21.15.104:0/2257100520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8003bc0 msgr2=0x7f94e8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 --2- 172.21.15.104:0/2257100520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8003bc0 0x7f94e8003fb0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009b40 tx=0x7f94e8005880 comp rx=0 tx=0).stop 2023-06-10T18:58:33.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f9502b46700 1 -- 172.21.15.104:0/2257100520 reap_dead start 2023-06-10T18:58:33.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 -- 172.21.15.104:0/2257100520 shutdown_connections 2023-06-10T18:58:33.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 -- 172.21.15.104:0/2257100520 >> 172.21.15.104:0/2257100520 conn(0x7f94fc06b8a0 msgr2=0x7f94fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:33.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 -- 172.21.15.104:0/2257100520 shutdown_connections 2023-06-10T18:58:33.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.970+0000 7f95045a9700 1 -- 172.21.15.104:0/2257100520 wait complete. 2023-06-10T18:58:33.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.971+0000 7f95045a9700 1 Processor -- start 2023-06-10T18:58:33.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.971+0000 7f95045a9700 1 -- start start 2023-06-10T18:58:33.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8003bc0 0x7f94fc158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 0x7f94fc151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc152720 con 0x7f94fc1551f0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc14f230 con 0x7f94e8003bc0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc1573e0 con 0x7f94fc1512e0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40438/0 (socket says 172.21.15.104:40438) 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 -- 172.21.15.104:0/2426584355 learned_addr learned my addr 172.21.15.104:0/2426584355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9502345700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8003bc0 0x7f94fc158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9502b46700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 0x7f94fc151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 -- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 msgr2=0x7f94fc151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 0x7f94fc151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 -- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8003bc0 msgr2=0x7f94fc158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8003bc0 0x7f94fc158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 -- 172.21.15.104:0/2426584355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc157960 con 0x7f94fc1551f0 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9502b46700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 0x7f94fc151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f9501b44700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f94f800e7b0 tx=0x7f94f800eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f80052e0 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f8005440 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc1e6850 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f80038a0 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.972+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc152160 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.973+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc074050 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.974+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94f8003a00 con 0x7f94fc1551f0 2023-06-10T18:58:33.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.974+0000 7f94f2ffd700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 0x7f94e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:33.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.974+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f94f8088e90 con 0x7f94fc1551f0 2023-06-10T18:58:33.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.974+0000 7f9502345700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 0x7f94e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:33.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.975+0000 7f9502345700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 0x7f94e006d190 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f94ec000c00 tx=0x7f94ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:33.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:33.978+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94f8053e30 con 0x7f94fc1551f0 2023-06-10T18:58:34.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.240+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f94fc0b5450 con 0x7f94fc1551f0 2023-06-10T18:58:34.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.244+0000 7f94f2ffd700 1 -- 172.21.15.104:0/2426584355 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+0 (secure 0 0 0) 0x7f94f8003cb0 con 0x7f94fc1551f0 2023-06-10T18:58:34.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 msgr2=0x7f94e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 0x7f94e006d190 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f94ec000c00 tx=0x7f94ec008030 comp rx=0 tx=0).stop 2023-06-10T18:58:34.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 msgr2=0x7f94fc1e6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f94f800e7b0 tx=0x7f94f800eac0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 shutdown_connections 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f94e006ace0 0x7f94e006d190 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8003bc0 0x7f94fc158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94fc1512e0 0x7f94fc151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 --2- 172.21.15.104:0/2426584355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94fc1551f0 0x7f94fc1e6310 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 >> 172.21.15.104:0/2426584355 conn(0x7f94fc06b8a0 msgr2=0x7f94fc159990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:34.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 shutdown_connections 2023-06-10T18:58:34.254 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.246+0000 7f95045a9700 1 -- 172.21.15.104:0/2426584355 wait complete. 2023-06-10T18:58:34.254 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:58:34.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.379+0000 7fa6be711700 1 Processor -- start 2023-06-10T18:58:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.379+0000 7fa6be711700 1 -- start start 2023-06-10T18:58:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6be711700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6b806e4b0 0x7fa6b806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.386 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6be711700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6b80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6be711700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b806d290 con 0x7fa6b806e0e0 2023-06-10T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6be711700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b8071640 con 0x7fa6b806e4b0 2023-06-10T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6be711700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80aef00 con 0x7fa6b8069e70 2023-06-10T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa6b806e0e0 0x7fa6b80c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38272/0 (socket says 172.21.15.104:38272) 2023-06-10T18:58:34.387 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b7fff700 1 -- 172.21.15.104:0/3898915481 learned_addr learned my addr 172.21.15.104:0/3898915481 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:34.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6bccae700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6b80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> v2:172.21.15.114:3300/0 conn(0x7fa6b806e4b0 0x7fa6b806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798492386 0 0) 0x7fa6b806d290 con 0x7fa6b806e0e0 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a00035b0 con 0x7fa6b806e0e0 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154720972 0 0) 0x7fa6a00035b0 con 0x7fa6b806e0e0 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 msgr2=0x7fa6b80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6b80ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 >> v2:172.21.15.114:3300/0 conn(0x7fa6b806e4b0 msgr2=0x7fa6b806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 --2- 172.21.15.104:0/3898915481 >> v2:172.21.15.114:3300/0 conn(0x7fa6b806e4b0 0x7fa6b806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80ca2d0 con 0x7fa6b806e0e0 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fa6a80017f0 con 0x7fa6b806e0e0 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> v2:172.21.15.114:3300/0 conn(0x7fa6b806e4b0 0x7fa6b806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 >> v1:172.21.15.104:6789/0 conn(0x7fa6b806e0e0 legacy=0x7fa6b80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6a0003a00 0x7fa6a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6a0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b8071640 con 0x7fa6a0004850 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80aef00 con 0x7fa6a0003a00 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a000bac0 con 0x7fa6b8069e70 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.380+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b7fff700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6a0003a00 0x7fa6a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6bccae700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6a0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 -- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 msgr2=0x7fa6a0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6a0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 -- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6a0003a00 msgr2=0x7fa6a0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6a0003a00 0x7fa6a0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 -- 172.21.15.104:0/3898915481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00035b0 con 0x7fa6a0004850 2023-06-10T18:58:34.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6bccae700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b8069e70 0x7fa6a0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6a0004cc0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009d30 tx=0x7fa6a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8719a0eaaac2d968 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac009e80 con 0x7fa6a0004850 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac00e9a0 con 0x7fa6a0004850 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6b6ffd700 1 -- 172.21.15.104:0/3898915481 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00ed30 con 0x7fa6a0004850 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6be711700 1 -- 172.21.15.104:0/3898915481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 msgr2=0x7fa6a0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6be711700 1 --2- 172.21.15.104:0/3898915481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6a0004cc0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009d30 tx=0x7fa6a0005880 comp rx=0 tx=0).stop 2023-06-10T18:58:34.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.381+0000 7fa6bccae700 1 -- 172.21.15.104:0/3898915481 reap_dead start 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.382+0000 7fa6be711700 1 -- 172.21.15.104:0/3898915481 shutdown_connections 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.382+0000 7fa6be711700 1 -- 172.21.15.104:0/3898915481 >> 172.21.15.104:0/3898915481 conn(0x7fa6b806b8a0 msgr2=0x7fa6b80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.382+0000 7fa6be711700 1 -- 172.21.15.104:0/3898915481 shutdown_connections 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.382+0000 7fa6be711700 1 -- 172.21.15.104:0/3898915481 wait complete. 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.383+0000 7fa6be711700 1 Processor -- start 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.383+0000 7fa6be711700 1 -- start start 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6b81e64d0 0x7fa6b8159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b80af360 0x7fa6b80ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81e4b80 con 0x7fa6a0004850 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80afc80 con 0x7fa6b81e64d0 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81e4530 con 0x7fa6b80af360 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40456/0 (socket says 172.21.15.104:40456) 2023-06-10T18:58:34.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 -- 172.21.15.104:0/3403417990 learned_addr learned my addr 172.21.15.104:0/3403417990 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6b81e64d0 0x7fa6b8159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6bccae700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b80af360 0x7fa6b80ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 -- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b80af360 msgr2=0x7fa6b80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b80af360 0x7fa6b80ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 -- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6b81e64d0 msgr2=0x7fa6b8159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6b81e64d0 0x7fa6b8159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 -- 172.21.15.104:0/3403417990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b81e4810 con 0x7fa6a0004850 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b7fff700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800f860 tx=0x7fa6a800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80072e0 con 0x7fa6a0004850 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8007460 con 0x7fa6a0004850 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b80b2970 con 0x7fa6a0004850 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8005830 con 0x7fa6a0004850 2023-06-10T18:58:34.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.384+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b80af7e0 con 0x7fa6a0004850 2023-06-10T18:58:34.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.385+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b81e4390 con 0x7fa6a0004850 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.386+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa6a8005990 con 0x7fa6a0004850 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.386+0000 7fa6b4ff9700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 0x7fa6a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.386+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fa6a80893a0 con 0x7fa6a0004850 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.387+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 0x7fa6a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.387+0000 7fa6b77fe700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 0x7fa6a406d190 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0090b0 tx=0x7fa6ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:34.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.391+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa6b81e4390 con 0x7fa6a0004850 2023-06-10T18:58:34.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.617+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6b81e4390 con 0x7fa6a0004850 2023-06-10T18:58:34.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.618+0000 7fa6b4ff9700 1 -- 172.21.15.104:0/3403417990 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7fa6b81e4390 con 0x7fa6a0004850 2023-06-10T18:58:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:34.624 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:58:34.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 msgr2=0x7fa6a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 0x7fa6a406d190 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0090b0 tx=0x7fa6ac008030 comp rx=0 tx=0).stop 2023-06-10T18:58:34.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 msgr2=0x7fa6b80b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:34.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800f860 tx=0x7fa6a800fb70 comp rx=0 tx=0).stop 2023-06-10T18:58:34.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 shutdown_connections 2023-06-10T18:58:34.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6b80af360 0x7fa6b80ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6a0004850 0x7fa6b80b2430 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fa6a406ace0 0x7fa6a406d190 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 --2- 172.21.15.104:0/3403417990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6b81e64d0 0x7fa6b8159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:34.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 >> 172.21.15.104:0/3403417990 conn(0x7fa6b806b8a0 msgr2=0x7fa6b80cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:34.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 shutdown_connections 2023-06-10T18:58:34.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:34.620+0000 7fa6be711700 1 -- 172.21.15.104:0/3403417990 wait complete. 2023-06-10T18:58:34.630 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:58:34.800 INFO:tasks.ceph.mds.b.smithi114.stderr:2023-06-10T18:58:34.794+0000 7f1fcdff5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:58:35.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:58:29.307685+0000', 'modified': '2023-06-10T18:58:32.331551+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10605}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10605': {'gid': 10605, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1296534750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1296534750}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1296534750}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:58:35.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:58:35.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-06-10T18:58:35.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.157+0000 7fe9cb057700 1 Processor -- start 2023-06-10T18:58:35.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.157+0000 7fe9cb057700 1 -- start start 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9cb057700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9c40ce390 0x7fe9c4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9cb057700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40cae10 0x7fe9c40cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9cb057700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40d0340 con 0x7fe9c406c720 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9cb057700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c40d1660 con 0x7fe9c40ce390 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9cb057700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c415cb60 con 0x7fe9c40cae10 2023-06-10T18:58:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9c40ce390 0x7fe9c4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe9c406c720 0x7fe9c40ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38292/0 (socket says 172.21.15.104:38292) 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c3fff700 1 -- 172.21.15.104:0/1510706764 learned_addr learned my addr 172.21.15.104:0/1510706764 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40cae10 0x7fe9c40cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 -- 172.21.15.104:0/1510706764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40cae10 msgr2=0x7fe9c40cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40cae10 0x7fe9c40cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 -- 172.21.15.104:0/1510706764 >> v1:172.21.15.104:6789/0 conn(0x7fe9c406c720 legacy=0x7fe9c40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 -- 172.21.15.104:0/1510706764 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4161230 con 0x7fe9c40ce390 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c37fe700 1 -- 172.21.15.104:0/1510706764 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249545322 0 0) 0x7fe9c40d0340 con 0x7fe9c406c720 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.158+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40ce390 0x7fe9c4160cf0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009840 tx=0x7fe9b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d888f232ab94d2e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9c37fe700 1 -- 172.21.15.104:0/1510706764 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8011830 con 0x7fe9c40ce390 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9c37fe700 1 -- 172.21.15.104:0/1510706764 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b8011990 con 0x7fe9c40ce390 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9c37fe700 1 -- 172.21.15.104:0/1510706764 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8011d20 con 0x7fe9c40ce390 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 -- 172.21.15.104:0/1510706764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40ce390 msgr2=0x7fe9c4160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40ce390 0x7fe9c4160cf0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009840 tx=0x7fe9b8009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 -- 172.21.15.104:0/1510706764 shutdown_connections 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40ce390 0x7fe9c4160cf0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 --2- 172.21.15.104:0/1510706764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40cae10 0x7fe9c40cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 -- 172.21.15.104:0/1510706764 >> 172.21.15.104:0/1510706764 conn(0x7fe9c4069e70 msgr2=0x7fe9c406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 -- 172.21.15.104:0/1510706764 shutdown_connections 2023-06-10T18:58:35.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.159+0000 7fe9cb057700 1 -- 172.21.15.104:0/1510706764 wait complete. 2023-06-10T18:58:35.171 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.160+0000 7fe9cb057700 1 Processor -- start 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.160+0000 7fe9cb057700 1 -- start start 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c406c720 0x7fe9c40b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40ce390 0x7fe9c4159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c40b0b90 con 0x7fe9c40cae10 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c40b0e70 con 0x7fe9c406c720 2023-06-10T18:58:35.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9cb057700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c40b1150 con 0x7fe9c40ce390 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40470/0 (socket says 172.21.15.104:40470) 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9c3fff700 1 -- 172.21.15.104:0/3528799645 learned_addr learned my addr 172.21.15.104:0/3528799645 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.161+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c406c720 0x7fe9c40b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40ce390 0x7fe9c4159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 -- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40ce390 msgr2=0x7fe9c4159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40ce390 0x7fe9c4159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 -- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c406c720 msgr2=0x7fe9c40b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c406c720 0x7fe9c40b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 -- 172.21.15.104:0/3528799645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b80094a0 con 0x7fe9c40cae10 2023-06-10T18:58:35.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c520 tx=0x7fe9b000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c17fa700 1 -- 172.21.15.104:0/3528799645 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0004c80 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c17fa700 1 -- 172.21.15.104:0/3528799645 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b0004de0 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c40b1430 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9c17fa700 1 -- 172.21.15.104:0/3528799645 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b00072d0 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.162+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c415a400 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.164+0000 7fe9c17fa700 1 -- 172.21.15.104:0/3528799645 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe9b0007430 con 0x7fe9c40cae10 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.164+0000 7fe9c17fa700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 0x7fe9a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.164+0000 7fe9c17fa700 1 -- 172.21.15.104:0/3528799645 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe9b00892a0 con 0x7fe9c40cae10 2023-06-10T18:58:35.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.164+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 0x7fe9a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.165+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 0x7fe9a806d190 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009790 tx=0x7fe9b8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.166+0000 7fe9d71dcb80 1 Processor -- start 2023-06-10T18:58:35.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.166+0000 7fe9d71dcb80 1 -- start start 2023-06-10T18:58:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542bf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542e6d80 0x5578542e5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578541c3c60 con 0x5578542bc740 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578542169f0 con 0x5578542e6d80 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55785420fe80 con 0x5578542bef50 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c8df3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542bf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40482/0 (socket says 172.21.15.104:40482) 2023-06-10T18:58:35.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 -- 172.21.15.104:0/4225680311 learned_addr learned my addr 172.21.15.104:0/4225680311 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542e6d80 0x5578542e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 -- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 msgr2=0x5578542bf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542bf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 -- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542e6d80 msgr2=0x5578542e5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542e6d80 0x5578542e5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 -- 172.21.15.104:0/4225680311 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578540915e0 con 0x5578542bc740 2023-06-10T18:58:35.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc007360 tx=0x7fe9bc007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e29ff7c961a50f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9ae7fc700 1 -- 172.21.15.104:0/4225680311 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc015470 con 0x5578542bc740 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9ae7fc700 1 -- 172.21.15.104:0/4225680311 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc00f050 con 0x5578542bc740 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.167+0000 7fe9ae7fc700 1 -- 172.21.15.104:0/4225680311 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0157d0 con 0x5578542bc740 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4225680311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 msgr2=0x5578542e9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc007360 tx=0x7fe9bc007670 comp rx=0 tx=0).stop 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4225680311 shutdown_connections 2023-06-10T18:58:35.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bc740 0x5578542e9440 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542bf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4225680311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542e6d80 0x5578542e5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4225680311 >> 172.21.15.104:0/4225680311 conn(0x5578542b3fc0 msgr2=0x5578543041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4225680311 shutdown_connections 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4225680311 wait complete. 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.168+0000 7fe9d71dcb80 1 -- start start 2023-06-10T18:58:35.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 Processor -- start 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578542fee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542e6d80 0x5578542fcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578541afa70 con 0x5578542bef50 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578541af770 con 0x5578542bc740 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.169+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5578540938d0 con 0x5578542e6d80 2023-06-10T18:58:35.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9d71dcb80 10 client.0 _ll_register_callbacks cb 0x5578541b5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:58:35.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40486/0 (socket says 172.21.15.104:40486) 2023-06-10T18:58:35.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 -- 172.21.15.104:0/4002784807 learned_addr learned my addr 172.21.15.104:0/4002784807 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542e6d80 0x5578542fcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578542fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 -- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542e6d80 msgr2=0x5578542fcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542e6d80 0x5578542fcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 -- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 msgr2=0x5578542fee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578542fee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 -- 172.21.15.104:0/4002784807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc009c60 con 0x5578542bef50 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00543d0 tx=0x7fe9b0060220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1616e32c18df52a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:58:35.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b005ff90 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe9b00890d0 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5578542b1780 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9b0087280 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9d71dcb80 10 client.10749 fetch_fsmap learned FSMap version 110 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x55785403dd90 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5578542b1780 con 0x5578542bef50 2023-06-10T18:58:35.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 110) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe9b0053cc0 con 0x5578542bef50 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.170+0000 7fe9d71dcb80 10 client.10749 fetch_fsmap finished waiting for FSMap version 110 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9d71dcb80 10 client.10749 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9d71dcb80 20 client.10749 populate_metadata read hostname 'smithi104' 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542df9d0 0x557854316a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9d71dcb80 4 client.10749 mds_command: new command op to 10605 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:58:35.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 --> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557854208e60 con 0x5578542df9d0 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542df9d0 0x557854316a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542df9d0 0x557854316a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7977522eab8d936d server_cookie=59b589e9d4100b8 in_seq=0 out_seq=0 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.171+0000 7fe9a37fe700 10 client.10749 ms_handle_connect on v2:172.21.15.114:6835/1296534750 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9a37fe700 1 -- 172.21.15.104:0/4002784807 <== mds.0 v2:172.21.15.114:6835/1296534750 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557854208e60 con 0x5578542df9d0 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9a37fe700 10 client.10749 handle_command_reply: tid=0 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 client.10749 shutdown 2023-06-10T18:58:35.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 msgr2=0x5578542fc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00543d0 tx=0x7fe9b0060220 comp rx=0 tx=0).stop 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 shutdown_connections 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542df9d0 0x557854316a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578542fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9a37fe700 0 client.10749 ms_handle_reset on v2:172.21.15.114:6835/1296534750 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542e6d80 0x5578542fcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/4002784807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542bef50 0x5578542fc980 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 >> 172.21.15.104:0/4002784807 conn(0x5578542b3fc0 msgr2=0x5578542e1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 shutdown_connections 2023-06-10T18:58:35.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.172+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/4002784807 wait complete. 2023-06-10T18:58:35.196 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.173+0000 7fe9d71dcb80 20 client.10749 trim_cache size 0 max 16384 2023-06-10T18:58:35.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.190+0000 7fe9d71dcb80 1 Processor -- start 2023-06-10T18:58:35.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.190+0000 7fe9d71dcb80 1 -- start start 2023-06-10T18:58:35.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578543127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.198 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542be0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x557854304f70 con 0x5578542df9d0 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578543017e0 con 0x5578542bc740 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x557854300080 con 0x5578542bef50 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40496/0 (socket says 172.21.15.104:40496) 2023-06-10T18:58:35.199 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 -- 172.21.15.104:0/2550321740 learned_addr learned my addr 172.21.15.104:0/2550321740 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578543127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 -- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 msgr2=0x5578542be0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542be0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 -- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 msgr2=0x5578543127b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578543127b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 -- 172.21.15.104:0/2550321740 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557854026c10 con 0x5578542df9d0 2023-06-10T18:58:35.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578543127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:35.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc016000 tx=0x7fe9bc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80909ee72b23456 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9aeffd700 1 -- 172.21.15.104:0/2550321740 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc023040 con 0x5578542df9d0 2023-06-10T18:58:35.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9aeffd700 1 -- 172.21.15.104:0/2550321740 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc015470 con 0x5578542df9d0 2023-06-10T18:58:35.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.191+0000 7fe9aeffd700 1 -- 172.21.15.104:0/2550321740 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc007ee0 con 0x5578542df9d0 2023-06-10T18:58:35.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/2550321740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 msgr2=0x5578542fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc016000 tx=0x7fe9bc014000 comp rx=0 tx=0).stop 2023-06-10T18:58:35.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/2550321740 shutdown_connections 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bc740 0x5578543127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x5578542fbd40 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/2550321740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bef50 0x5578542be0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/2550321740 >> 172.21.15.104:0/2550321740 conn(0x5578542b3fc0 msgr2=0x557854325e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/2550321740 shutdown_connections 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.192+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/2550321740 wait complete. 2023-06-10T18:58:35.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.193+0000 7fe9d71dcb80 1 -- start start 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.193+0000 7fe9d71dcb80 1 Processor -- start 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bc740 0x5578543065f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bef50 0x557854368710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578542e06e0 con 0x5578542df9d0 2023-06-10T18:58:35.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557854307ab0 con 0x5578542bef50 2023-06-10T18:58:35.211 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5578542de250 con 0x5578542bc740 2023-06-10T18:58:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 10 client.0 _ll_register_callbacks cb 0x5578541b5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:58:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40502/0 (socket says 172.21.15.104:40502) 2023-06-10T18:58:35.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 -- 172.21.15.104:0/1164982223 learned_addr learned my addr 172.21.15.104:0/1164982223 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bc740 0x5578543065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c3fff700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bef50 0x557854368710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 -- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bc740 msgr2=0x5578543065f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bc740 0x5578543065f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 -- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bef50 msgr2=0x557854368710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bef50 0x557854368710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 -- 172.21.15.104:0/1164982223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578540915e0 con 0x5578542df9d0 2023-06-10T18:58:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9c95f4700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000f80 tx=0x7fe9bc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae1b105c2d6aebe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9a37fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:58:35.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc023040 con 0x5578542df9d0 2023-06-10T18:58:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe9bc028440 con 0x5578542df9d0 2023-06-10T18:58:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9bc00f050 con 0x5578542df9d0 2023-06-10T18:58:35.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9bc007ee0 con 0x5578542df9d0 2023-06-10T18:58:35.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 10 client.10755 fetch_fsmap learned FSMap version 110 2023-06-10T18:58:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.194+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x557854064200 con 0x5578542df9d0 2023-06-10T18:58:35.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc00f050 con 0x5578542df9d0 2023-06-10T18:58:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 110) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe9bc03d3f0 con 0x5578542df9d0 2023-06-10T18:58:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 10 client.10755 fetch_fsmap finished waiting for FSMap version 110 2023-06-10T18:58:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 10 client.10755 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-10T18:58:35.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 20 client.10755 populate_metadata read hostname 'smithi104' 2023-06-10T18:58:35.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542e6d80 0x5578542e3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 4 client.10755 mds_command: new command op to 10605 tid=0[{"prefix": "status"}] 2023-06-10T18:58:35.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 --> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5578541cb000 con 0x5578542e6d80 2023-06-10T18:58:35.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542e6d80 0x5578542e3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9a37fe700 10 client.10755 ms_handle_connect on v2:172.21.15.114:6835/1296534750 2023-06-10T18:58:35.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9c8df3700 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542e6d80 0x5578542e3b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1eb123e584c5752 server_cookie=9aa85e4f5cf05baa in_seq=0 out_seq=0 2023-06-10T18:58:35.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9a37fe700 1 -- 172.21.15.104:0/1164982223 <== mds.0 v2:172.21.15.114:6835/1296534750 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5578541cb000 con 0x5578542e6d80 2023-06-10T18:58:35.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9a37fe700 10 client.10755 handle_command_reply: tid=0 2023-06-10T18:58:35.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.195+0000 7fe9d71dcb80 1 client.10755 shutdown 2023-06-10T18:58:35.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 msgr2=0x557854306cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000f80 tx=0x7fe9bc004b40 comp rx=0 tx=0).stop 2023-06-10T18:58:35.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 shutdown_connections 2023-06-10T18:58:35.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:6835/1296534750,v1:172.21.15.114:6837/1296534750] conn(0x5578542e6d80 0x5578542e3b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5578542df9d0 0x557854306cd0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9a37fe700 0 client.10755 ms_handle_reset on v2:172.21.15.114:6835/1296534750 2023-06-10T18:58:35.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578542bc740 0x5578543065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 --2- 172.21.15.104:0/1164982223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5578542bef50 0x557854368710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 >> 172.21.15.104:0/1164982223 conn(0x5578542b3fc0 msgr2=0x5578542b8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 shutdown_connections 2023-06-10T18:58:35.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 1 -- 172.21.15.104:0/1164982223 wait complete. 2023-06-10T18:58:35.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9d71dcb80 20 client.10755 trim_cache size 0 max 16384 2023-06-10T18:58:35.228 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:58:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:58:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10605, 2023-06-10T18:58:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:58:35.229 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.8871881960000003, 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 110, 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 172, 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 172, 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.329339978 2023-06-10T18:58:35.230 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:35.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 msgr2=0x7fe9a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 0x7fe9a806d190 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009790 tx=0x7fe9b8008030 comp rx=0 tx=0).stop 2023-06-10T18:58:35.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 msgr2=0x7fe9c4159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c520 tx=0x7fe9b000c8e0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 shutdown_connections 2023-06-10T18:58:35.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe9a806ace0 0x7fe9a806d190 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c406c720 0x7fe9c40b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9c40ce390 0x7fe9c4159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 --2- 172.21.15.104:0/3528799645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9c40cae10 0x7fe9c4159880 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 >> 172.21.15.104:0/3528799645 conn(0x7fe9c4069e70 msgr2=0x7fe9c40cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 shutdown_connections 2023-06-10T18:58:35.234 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.196+0000 7fe9cb057700 1 -- 172.21.15.104:0/3528799645 wait complete. 2023-06-10T18:58:35.235 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:58:35.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.349+0000 7fbe4ab3d700 1 Processor -- start 2023-06-10T18:58:35.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.349+0000 7fbe4ab3d700 1 -- start start 2023-06-10T18:58:35.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe4ab3d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe44160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe4ab3d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbe440cae10 0x7fbe440cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe4ab3d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe440d0340 con 0x7fbe4406c720 2023-06-10T18:58:35.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe4ab3d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe440d1660 con 0x7fbe440cae10 2023-06-10T18:58:35.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe4ab3d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4415cb60 con 0x7fbe440ce390 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe43fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbe4406c720 0x7fbe440ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38336/0 (socket says 172.21.15.104:38336) 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe43fff700 1 -- 172.21.15.104:0/3591630618 learned_addr learned my addr 172.21.15.104:0/3591630618 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe488d9700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe44160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe490da700 1 --2- 172.21.15.104:0/3591630618 >> v2:172.21.15.114:3300/0 conn(0x7fbe440cae10 0x7fbe440cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84517795 0 0) 0x7fbe440d0340 con 0x7fbe4406c720 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe340035b0 con 0x7fbe4406c720 2023-06-10T18:58:35.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.350+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3177423630 0 0) 0x7fbe340035b0 con 0x7fbe4406c720 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 msgr2=0x7fbe44160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe44160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 >> v2:172.21.15.114:3300/0 conn(0x7fbe440cae10 msgr2=0x7fbe440cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 --2- 172.21.15.104:0/3591630618 >> v2:172.21.15.114:3300/0 conn(0x7fbe440cae10 0x7fbe440cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44161230 con 0x7fbe4406c720 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fbe30001670 con 0x7fbe4406c720 2023-06-10T18:58:35.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe490da700 1 --2- 172.21.15.104:0/3591630618 >> v2:172.21.15.114:3300/0 conn(0x7fbe440cae10 0x7fbe440cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 >> v1:172.21.15.104:6789/0 conn(0x7fbe4406c720 legacy=0x7fbe440ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe34003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe34003f80 0x7fbe34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe34004d90 0x7fbe34005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe440d1660 con 0x7fbe34003f80 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4415cb60 con 0x7fbe34004d90 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3400ba10 con 0x7fbe440ce390 2023-06-10T18:58:35.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe34003f80 0x7fbe34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe488d9700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe34003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe490da700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe34004d90 0x7fbe34005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 -- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 msgr2=0x7fbe34003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe440ce390 0x7fbe34003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 -- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe34004d90 msgr2=0x7fbe34005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe34004d90 0x7fbe34005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 -- 172.21.15.104:0/3591630618 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe340035b0 con 0x7fbe34003f80 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe43fff700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe34003f80 0x7fbe34004850 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000c320 tx=0x7fbe34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f8f07dd50bd60 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000ce60 con 0x7fbe34003f80 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30005960 con 0x7fbe34003f80 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.351+0000 7fbe437fe700 1 -- 172.21.15.104:0/3591630618 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30005cf0 con 0x7fbe34003f80 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/3591630618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe34003f80 msgr2=0x7fbe34004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/3591630618 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe34003f80 0x7fbe34004850 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000c320 tx=0x7fbe34005880 comp rx=0 tx=0).stop 2023-06-10T18:58:35.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe490da700 1 -- 172.21.15.104:0/3591630618 reap_dead start 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/3591630618 shutdown_connections 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/3591630618 >> 172.21.15.104:0/3591630618 conn(0x7fbe44069e70 msgr2=0x7fbe4406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/3591630618 shutdown_connections 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.352+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/3591630618 wait complete. 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.353+0000 7fbe4ab3d700 1 Processor -- start 2023-06-10T18:58:35.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.353+0000 7fbe4ab3d700 1 -- start start 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe34003f80 0x7fbe440b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe441f55e0 0x7fbe440b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe441f2010 con 0x7fbe440b15b0 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe44151c90 con 0x7fbe441f55e0 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe4ab3d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe440b1f60 con 0x7fbe34003f80 2023-06-10T18:58:35.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40522/0 (socket says 172.21.15.104:40522) 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 -- 172.21.15.104:0/1639352696 learned_addr learned my addr 172.21.15.104:0/1639352696 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe488d9700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe34003f80 0x7fbe440b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 -- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe34003f80 msgr2=0x7fbe440b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe43fff700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe441f55e0 0x7fbe440b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe34003f80 0x7fbe440b5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 -- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe441f55e0 msgr2=0x7fbe440b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe441f55e0 0x7fbe440b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 -- 172.21.15.104:0/1639352696 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe440b4380 con 0x7fbe440b15b0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe490da700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fbe4406b1a0 tx=0x7fbe3c00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.354+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c01c070 con 0x7fbe440b15b0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.355+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c00f040 con 0x7fbe440b15b0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.355+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c00e040 con 0x7fbe440b15b0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.355+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe440b4660 con 0x7fbe440b15b0 2023-06-10T18:58:35.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.355+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe440b4c20 con 0x7fbe440b15b0 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.355+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe441f2180 con 0x7fbe440b15b0 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.356+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbe3c015430 con 0x7fbe440b15b0 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.356+0000 7fbe417fa700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 0x7fbe2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.356+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbe3c0894b0 con 0x7fbe440b15b0 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.357+0000 7fbe488d9700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 0x7fbe2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:35.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.357+0000 7fbe488d9700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 0x7fbe2806d190 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fbe380099c0 tx=0x7fbe38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:35.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.361+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbe441f2180 con 0x7fbe440b15b0 2023-06-10T18:58:35.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.588+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbe441f2150 con 0x7fbe440b15b0 2023-06-10T18:58:35.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.589+0000 7fbe417fa700 1 -- 172.21.15.104:0/1639352696 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v110) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbe3c057eb0 con 0x7fbe440b15b0 2023-06-10T18:58:35.596 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:58:35.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 msgr2=0x7fbe2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 0x7fbe2806d190 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fbe380099c0 tx=0x7fbe38008040 comp rx=0 tx=0).stop 2023-06-10T18:58:35.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 msgr2=0x7fbe440b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:35.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fbe4406b1a0 tx=0x7fbe3c00f490 comp rx=0 tx=0).stop 2023-06-10T18:58:35.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 shutdown_connections 2023-06-10T18:58:35.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe440b15b0 0x7fbe440b1a20 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe34003f80 0x7fbe440b5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe2806ace0 0x7fbe2806d190 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 --2- 172.21.15.104:0/1639352696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe441f55e0 0x7fbe440b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 >> 172.21.15.104:0/1639352696 conn(0x7fbe44069e70 msgr2=0x7fbe4406b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 shutdown_connections 2023-06-10T18:58:35.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:35.591+0000 7fbe4ab3d700 1 -- 172.21.15.104:0/1639352696 wait complete. 2023-06-10T18:58:35.609 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:58:35.609 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:58:35.610 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:58:35.610 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:58:35.610 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:35.610 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:58:35.611 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:58:35.611 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:58:35.611 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:35.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.613 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:58:35.631 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:35.632 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.633 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:35.633 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:35.634 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:35.634 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:35.634 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:35.635 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:58:35.635 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:58:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41569sec preferred_lft 41569sec 2023-06-10T18:58:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:58:35.637 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.637 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:58:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.638 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:58:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.639 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:58:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:58:35.639 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.640 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:58:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:58:35.640 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:35.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.642 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:58:35.697 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:58:35.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.699 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:58:35.759 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:58:35.759 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:58:35.760 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:58:35.760 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:58:35.760 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:58:35.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.762 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:35.762 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:35.763 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:58:35.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:58:35.767 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:58:35.768 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:35.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.864 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:58:35.880 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:58:35.881 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:58:35.881 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:35.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:35.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:58:35.947 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:58:35.948 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:35.949 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:36.004 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:58:36.005 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:36.005 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:36.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:36.129 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:36.130 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:36.194 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:36.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:36.196 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:36.252 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:58:36.252 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:58:36.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:36.321 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:36.321 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:36.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:58:36.373+0000 7f6f70fe25c0 -1 init, newargv = 0x55dc686ab910 newargc=15 2023-06-10T18:58:36.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[160529]: starting ceph client 2023-06-10T18:58:36.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[160529]: starting fuse 2023-06-10T18:58:36.389 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:36.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:36.392 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:36.449 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:58:36.450 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:58:36.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:58:36.450 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:58:36.451 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:58:36.451 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:58:36.451 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:58:36.452 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:36.453 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:58:36.454 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:58:36.454 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:58:36.454 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:58:36.455 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:36.455 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:58:36.455 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:58:36.455 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:58:36.456 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:58:36.456 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:36.456 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:58:36.457 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:58:36.457 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:58:36.457 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:58:36.458 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:58:36.458 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:58:36.458 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:58:36.458 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:58:36.460 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:36.460 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:58:36.461 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:58:36.572 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.160529.asok 2023-06-10T18:58:36.755 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.160529.asok 2023-06-10T18:58:36.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:36.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.160529.asok status 2023-06-10T18:58:36.910 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:58:36.911 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "160529", 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10767, 2023-06-10T18:58:36.912 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:58:36.913 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:58:36.913 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:58:36.913 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10767 2023-06-10T18:58:36.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:36.913 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 827471637 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:36.914 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 827471637 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10767 192.168.0.1:0/827471637", 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/827471637", 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 110, 2023-06-10T18:58:36.915 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 172, 2023-06-10T18:58:36.916 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:58:36.916 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:58:36.916 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:58:36.916 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:36.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:36.922 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:58:36.923 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:36.940 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:58:36.941 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:36.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:36.942 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:58:37.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:58:37.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.165+0000 7eff89517700 1 Processor -- start 2023-06-10T18:58:37.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.165+0000 7eff89517700 1 -- start start 2023-06-10T18:58:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff840d0530 0x7eff840cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff8406c260 0x7eff840d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff840b1e80 0x7eff8415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff840b1c40 con 0x7eff8406c260 2023-06-10T18:58:37.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff840d0e40 con 0x7eff840d0530 2023-06-10T18:58:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff89517700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff840b1680 con 0x7eff840b1e80 2023-06-10T18:58:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff8406c260 0x7eff840d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7eff8406c260 0x7eff840d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40540/0 (socket says 172.21.15.104:40540) 2023-06-10T18:58:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 -- 172.21.15.104:0/259956248 learned_addr learned my addr 172.21.15.104:0/259956248 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff82ffd700 1 --2- 172.21.15.104:0/259956248 >> v2:172.21.15.114:3300/0 conn(0x7eff840d0530 0x7eff840cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 -- 172.21.15.104:0/259956248 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff840b1e80 msgr2=0x7eff8415fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff837fe700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff840b1e80 0x7eff8415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff840b1e80 0x7eff8415fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 -- 172.21.15.104:0/259956248 >> v2:172.21.15.114:3300/0 conn(0x7eff840d0530 msgr2=0x7eff840cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 --2- 172.21.15.104:0/259956248 >> v2:172.21.15.114:3300/0 conn(0x7eff840d0530 0x7eff840cfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.166+0000 7eff827fc700 1 -- 172.21.15.104:0/259956248 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff840b1960 con 0x7eff8406c260 2023-06-10T18:58:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff827fc700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff8406c260 0x7eff840d0900 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7eff84067ae0 tx=0x7eff74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734eb0de9942ba56 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff81ffb700 1 -- 172.21.15.104:0/259956248 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7400b870 con 0x7eff8406c260 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff81ffb700 1 -- 172.21.15.104:0/259956248 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7400b9d0 con 0x7eff8406c260 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff81ffb700 1 -- 172.21.15.104:0/259956248 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7400bd60 con 0x7eff8406c260 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 -- 172.21.15.104:0/259956248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff8406c260 msgr2=0x7eff840d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff8406c260 0x7eff840d0900 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7eff84067ae0 tx=0x7eff74009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 -- 172.21.15.104:0/259956248 shutdown_connections 2023-06-10T18:58:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff840b1e80 0x7eff8415fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 --2- 172.21.15.104:0/259956248 >> v2:172.21.15.114:3300/0 conn(0x7eff840d0530 0x7eff840cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 --2- 172.21.15.104:0/259956248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff8406c260 0x7eff840d0900 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 -- 172.21.15.104:0/259956248 >> 172.21.15.104:0/259956248 conn(0x7eff84071ec0 msgr2=0x7eff8409bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 -- 172.21.15.104:0/259956248 shutdown_connections 2023-06-10T18:58:37.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.167+0000 7eff89517700 1 -- 172.21.15.104:0/259956248 wait complete. 2023-06-10T18:58:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 Processor -- start 2023-06-10T18:58:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 -- start start 2023-06-10T18:58:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c260 0x7eff840b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 0x7eff840b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff840b66e0 con 0x7eff840b1e80 2023-06-10T18:58:37.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff840b71b0 con 0x7eff840d0530 2023-06-10T18:58:37.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff89517700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff840b6910 con 0x7eff8406c260 2023-06-10T18:58:37.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff827fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.169+0000 7eff827fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40548/0 (socket says 172.21.15.104:40548) 2023-06-10T18:58:37.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 -- 172.21.15.104:0/2561029159 learned_addr learned my addr 172.21.15.104:0/2561029159 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff82ffd700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c260 0x7eff840b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff837fe700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 0x7eff840b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 -- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c260 msgr2=0x7eff840b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:37.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c260 0x7eff840b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 -- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 msgr2=0x7eff840b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:37.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 0x7eff840b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 -- 172.21.15.104:0/2561029159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff74009550 con 0x7eff840b1e80 2023-06-10T18:58:37.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff837fe700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 0x7eff840b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:37.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff827fc700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7eff74014040 tx=0x7eff74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:37.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74004520 con 0x7eff840b1e80 2023-06-10T18:58:37.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7401e070 con 0x7eff840b1e80 2023-06-10T18:58:37.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff840b6bf0 con 0x7eff840b1e80 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff740038e0 con 0x7eff840b1e80 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.170+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff840c7490 con 0x7eff840b1e80 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.171+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8408cd50 con 0x7eff840b1e80 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.171+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff74003a40 con 0x7eff840b1e80 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.172+0000 7eff6b7fe700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 0x7eff7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:37.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.172+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7eff74020070 con 0x7eff840b1e80 2023-06-10T18:58:37.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.172+0000 7eff82ffd700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 0x7eff7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:37.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.173+0000 7eff82ffd700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 0x7eff7006d190 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7eff840b53c0 tx=0x7eff78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:37.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.176+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff74055b40 con 0x7eff840b1e80 2023-06-10T18:58:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.410+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7eff8406ba60 con 0x7eff840b1e80 2023-06-10T18:58:37.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.410+0000 7eff6b7fe700 1 -- 172.21.15.104:0/2561029159 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v110) v1 ==== 53+0+73 (secure 0 0 0) 0x7eff74059160 con 0x7eff840b1e80 2023-06-10T18:58:37.417 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:58:37.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 msgr2=0x7eff7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 0x7eff7006d190 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7eff840b53c0 tx=0x7eff78008040 comp rx=0 tx=0).stop 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 msgr2=0x7eff840b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7eff74014040 tx=0x7eff74004750 comp rx=0 tx=0).stop 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 shutdown_connections 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7eff7006ace0 0x7eff7006d190 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff840d0530 0x7eff840b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c260 0x7eff840b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 --2- 172.21.15.104:0/2561029159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff840b1e80 0x7eff840b61a0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 >> 172.21.15.104:0/2561029159 conn(0x7eff84071ec0 msgr2=0x7eff840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 shutdown_connections 2023-06-10T18:58:37.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:37.412+0000 7eff89517700 1 -- 172.21.15.104:0/2561029159 wait complete. 2023-06-10T18:58:37.430 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:58:37.431 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:58:37.431 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:58:37.431 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:58:37.431 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:37.432 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:58:37.432 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:58:37.432 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:58:37.432 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:58:37.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.433 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:58:37.454 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:37.455 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:37.456 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:37.456 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:37.456 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:37.456 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:37.456 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41563sec preferred_lft 41563sec 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.457 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:37.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:58:37.459 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:37.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.460 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:58:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:58:37.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.529 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:58:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:58:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:58:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:58:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:58:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:58:37.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.590 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:58:37.591 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:58:37.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:58:37.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:58:37.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:58:37.592 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:58:37.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.701 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:58:37.718 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:58:37.718 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:58:37.719 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:58:37.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:37.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:58:37.785 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:58:37.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:37.786 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:37.842 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:58:37.843 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:37.843 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:37.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:58:37.965 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:37.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:38.031 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:38.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:38.033 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:58:38.089 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:58:38.090 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:58:38.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:58:38.157 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:38.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:38.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:58:38.210+0000 7f6fa686e5c0 -1 init, newargv = 0x562bac179c20 newargc=15 2023-06-10T18:58:38.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[117671]: starting ceph client 2023-06-10T18:58:38.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[117671]: starting fuse 2023-06-10T18:58:38.226 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:38.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:38.229 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:58:38.285 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:58:38.286 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:58:38.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:58:38.287 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:58:38.287 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:58:38.287 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:58:38.288 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:58:38.288 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:58:38.288 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:58:38.289 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:58:38.289 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:58:38.289 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:58:38.290 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:58:38.290 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:58:38.290 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:58:38.291 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:58:38.291 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:58:38.291 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:58:38.292 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:58:38.292 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:58:38.292 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:58:38.293 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:58:38.293 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:58:38.294 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:58:38.294 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:58:38.295 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:58:38.296 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:58:38.296 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:58:38.296 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:58:38.408 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.117671.asok 2023-06-10T18:58:38.590 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.117671.asok 2023-06-10T18:58:38.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:38.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.117671.asok status 2023-06-10T18:58:38.747 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:58:38.747 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:58:38.748 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:58:38.748 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:58:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:58:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:58:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:58:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "117671", 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 9037, 2023-06-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:58:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:58:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:58:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 9037 2023-06-10T18:58:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:58:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:58:38.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:58:38.752 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:38.752 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1866387648 2023-06-10T18:58:38.752 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:58:38.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:58:38.753 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:58:38.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1866387648 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.9037 192.168.0.1:0/1866387648", 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/1866387648", 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 110, 2023-06-10T18:58:38.754 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 172, 2023-06-10T18:58:38.755 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:58:38.755 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:58:38.755 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:58:38.755 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:58:38.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:38.757 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:58:38.757 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:38.774 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:58:38.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:38.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:38.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:38.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:58:38.990 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:58:38.991 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:58:38.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:58:39.124 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:58:39.125 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:58:39.126 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:39.126 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:39.146 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:39.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.260+0000 7ffbf2545700 1 Processor -- start 2023-06-10T18:58:39.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.260+0000 7ffbf2545700 1 -- start start 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf2545700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 0x7ffbec0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf2545700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbec069e70 0x7ffbec06a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf2545700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec06dcb0 con 0x7ffbec15c500 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf2545700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec06e850 con 0x7ffbec069e70 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf2545700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec06a830 con 0x7ffbec0cf030 2023-06-10T18:58:39.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbec069e70 0x7ffbec06a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffbec069e70 0x7ffbec06a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40558/0 (socket says 172.21.15.104:40558) 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 -- 172.21.15.104:0/1929206563 learned_addr learned my addr 172.21.15.104:0/1929206563 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbebfff700 1 --1- 172.21.15.104:0/1929206563 >> v1:172.21.15.104:6789/0 conn(0x7ffbec15c500 0x7ffbec06e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38382/0 (socket says 172.21.15.104:38382) 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbeb7fe700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 0x7ffbec0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 -- 172.21.15.104:0/1929206563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 msgr2=0x7ffbec0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 0x7ffbec0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 -- 172.21.15.104:0/1929206563 >> v1:172.21.15.104:6789/0 conn(0x7ffbec15c500 legacy=0x7ffbec06e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.270 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 -- 172.21.15.104:0/1929206563 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec0cf460 con 0x7ffbec069e70 2023-06-10T18:58:39.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbeaffd700 1 -- 172.21.15.104:0/1929206563 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739238588 0 0) 0x7ffbec06dcb0 con 0x7ffbec15c500 2023-06-10T18:58:39.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbeb7fe700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 0x7ffbec0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:39.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbf0ae2700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec06a2f0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009e50 tx=0x7ffbe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a797f94738066a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.261+0000 7ffbeaffd700 1 -- 172.21.15.104:0/1929206563 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400e070 con 0x7ffbec069e70 2023-06-10T18:58:39.271 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbeaffd700 1 -- 172.21.15.104:0/1929206563 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe400b9c0 con 0x7ffbec069e70 2023-06-10T18:58:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbeaffd700 1 -- 172.21.15.104:0/1929206563 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400bd70 con 0x7ffbec069e70 2023-06-10T18:58:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 -- 172.21.15.104:0/1929206563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 msgr2=0x7ffbec06a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec06a2f0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009e50 tx=0x7ffbe400b330 comp rx=0 tx=0).stop 2023-06-10T18:58:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 -- 172.21.15.104:0/1929206563 shutdown_connections 2023-06-10T18:58:39.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec06a2f0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 --2- 172.21.15.104:0/1929206563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec0cf030 0x7ffbec0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 -- 172.21.15.104:0/1929206563 >> 172.21.15.104:0/1929206563 conn(0x7ffbec06b8a0 msgr2=0x7ffbec0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:39.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 -- 172.21.15.104:0/1929206563 shutdown_connections 2023-06-10T18:58:39.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.262+0000 7ffbf2545700 1 -- 172.21.15.104:0/1929206563 wait complete. 2023-06-10T18:58:39.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.263+0000 7ffbf2545700 1 Processor -- start 2023-06-10T18:58:39.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 -- start start 2023-06-10T18:58:39.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbec0cf030 0x7ffbec0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec15c500 0x7ffbec1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0b4e40 con 0x7ffbec069e70 2023-06-10T18:58:39.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0cae30 con 0x7ffbec0cf030 2023-06-10T18:58:39.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbf2545700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0b4880 con 0x7ffbec15c500 2023-06-10T18:58:39.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbebfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40560/0 (socket says 172.21.15.104:40560) 2023-06-10T18:58:39.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.264+0000 7ffbebfff700 1 -- 172.21.15.104:0/2228901542 learned_addr learned my addr 172.21.15.104:0/2228901542 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbeb7fe700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbec0cf030 0x7ffbec0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 -- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec15c500 msgr2=0x7ffbec1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbf0ae2700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec15c500 0x7ffbec1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec15c500 0x7ffbec1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 -- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbec0cf030 msgr2=0x7ffbec0b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbec0cf030 0x7ffbec0b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 -- 172.21.15.104:0/2228901542 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe4009b30 con 0x7ffbec069e70 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbebfff700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c4a0 tx=0x7ffbd400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40074e0 con 0x7ffbec069e70 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd4007640 con 0x7ffbec069e70 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec0b4b60 con 0x7ffbec069e70 2023-06-10T18:58:39.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4005a50 con 0x7ffbec069e70 2023-06-10T18:58:39.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.265+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec0c4030 con 0x7ffbec069e70 2023-06-10T18:58:39.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.266+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec0ca920 con 0x7ffbec069e70 2023-06-10T18:58:39.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.266+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffbd4005bb0 con 0x7ffbec069e70 2023-06-10T18:58:39.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.267+0000 7ffbe8ff9700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 0x7ffbcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.267+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffbd4088a70 con 0x7ffbec069e70 2023-06-10T18:58:39.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.267+0000 7ffbeb7fe700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 0x7ffbcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.268+0000 7ffbeb7fe700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 0x7ffbcc06d190 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffbec06db00 tx=0x7ffbdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.271+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffbec0ca920 con 0x7ffbec069e70 2023-06-10T18:58:39.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.459+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbec0ca920 con 0x7ffbec069e70 2023-06-10T18:58:39.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.460+0000 7ffbe8ff9700 1 -- 172.21.15.104:0/2228901542 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+18614 (secure 0 0 0) 0x7ffbec0ca920 con 0x7ffbec069e70 2023-06-10T18:58:39.467 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:39.467 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":172,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:30.309579+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:39.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 msgr2=0x7ffbcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 0x7ffbcc06d190 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffbec06db00 tx=0x7ffbdc008040 comp rx=0 tx=0).stop 2023-06-10T18:58:39.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 msgr2=0x7ffbec0c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c4a0 tx=0x7ffbd400c860 comp rx=0 tx=0).stop 2023-06-10T18:58:39.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 shutdown_connections 2023-06-10T18:58:39.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbec15c500 0x7ffbec1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbec069e70 0x7ffbec0c3af0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffbcc06ace0 0x7ffbcc06d190 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 --2- 172.21.15.104:0/2228901542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbec0cf030 0x7ffbec0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 >> 172.21.15.104:0/2228901542 conn(0x7ffbec06b8a0 msgr2=0x7ffbec06e140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:39.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 shutdown_connections 2023-06-10T18:58:39.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.462+0000 7ffbf2545700 1 -- 172.21.15.104:0/2228901542 wait complete. 2023-06-10T18:58:39.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:39.481 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:39.605 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.599+0000 7f0bfde2d700 1 Processor -- start 2023-06-10T18:58:39.606 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.599+0000 7f0bfde2d700 1 -- start start 2023-06-10T18:58:39.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bfde2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf80af940 0x7f0bf815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.609 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bfde2d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0bf815e720 0x7f0bf80cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bfde2d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf8154970 con 0x7f0bf81545a0 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bfde2d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf815dab0 con 0x7f0bf815e720 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bfde2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80cb0b0 con 0x7f0bf80af940 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0bf815e720 0x7f0bf80cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0bf815e720 0x7f0bf80cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40572/0 (socket says 172.21.15.104:40572) 2023-06-10T18:58:39.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 -- 172.21.15.104:0/1851099572 learned_addr learned my addr 172.21.15.104:0/1851099572 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf6ffd700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf80af940 0x7f0bf815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 -- 172.21.15.104:0/1851099572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf80af940 msgr2=0x7f0bf815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf80af940 0x7f0bf815e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 -- 172.21.15.104:0/1851099572 >> v1:172.21.15.114:6789/0 conn(0x7f0bf81545a0 legacy=0x7f0bf815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.600+0000 7f0bf7fff700 1 -- 172.21.15.104:0/1851099572 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf815ac80 con 0x7f0bf815e720 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bf7fff700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf815e720 0x7f0bf80cd150 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009e50 tx=0x7f0bf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c76dd0ecda18a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bf67fc700 1 -- 172.21.15.104:0/1851099572 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000e070 con 0x7f0bf815e720 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bf67fc700 1 -- 172.21.15.104:0/1851099572 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf000b9c0 con 0x7f0bf815e720 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bf67fc700 1 -- 172.21.15.104:0/1851099572 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000bd70 con 0x7f0bf815e720 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1851099572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf815e720 msgr2=0x7f0bf80cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf815e720 0x7f0bf80cd150 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009e50 tx=0x7f0bf000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1851099572 shutdown_connections 2023-06-10T18:58:39.612 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf815e720 0x7f0bf80cd150 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1851099572 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf80af940 0x7f0bf815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1851099572 >> 172.21.15.104:0/1851099572 conn(0x7f0bf806e370 msgr2=0x7f0bf806e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1851099572 shutdown_connections 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.601+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1851099572 wait complete. 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.602+0000 7f0bfde2d700 1 Processor -- start 2023-06-10T18:58:39.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 -- start start 2023-06-10T18:58:39.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bf81545a0 0x7f0bf80b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf815e720 0x7f0bf81455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80c8ee0 con 0x7f0bf80af940 2023-06-10T18:58:39.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80c8900 con 0x7f0bf81545a0 2023-06-10T18:58:39.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bfde2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80ca660 con 0x7f0bf815e720 2023-06-10T18:58:39.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bf77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bf77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40574/0 (socket says 172.21.15.104:40574) 2023-06-10T18:58:39.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.603+0000 7f0bf77fe700 1 -- 172.21.15.104:0/1536954499 learned_addr learned my addr 172.21.15.104:0/1536954499 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf6ffd700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bf81545a0 0x7f0bf80b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf7fff700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf815e720 0x7f0bf81455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 -- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf815e720 msgr2=0x7f0bf81455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf815e720 0x7f0bf81455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 -- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bf81545a0 msgr2=0x7f0bf80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bf81545a0 0x7f0bf80b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 -- 172.21.15.104:0/1536954499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0009b30 con 0x7f0bf80af940 2023-06-10T18:58:39.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bf77fe700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0be000c4b0 tx=0x7f0be000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be00052d0 con 0x7f0bf80af940 2023-06-10T18:58:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0005430 con 0x7f0bf80af940 2023-06-10T18:58:39.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf81461a0 con 0x7f0bf80af940 2023-06-10T18:58:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be00038c0 con 0x7f0bf80af940 2023-06-10T18:58:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.604+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf80b2fe0 con 0x7f0bf80af940 2023-06-10T18:58:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.605+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf8074050 con 0x7f0bf80af940 2023-06-10T18:58:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.606+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0be0003a20 con 0x7f0bf80af940 2023-06-10T18:58:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.606+0000 7f0beffff700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 0x7f0bd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.606+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0be00888f0 con 0x7f0bf80af940 2023-06-10T18:58:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.606+0000 7f0bf6ffd700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 0x7f0bd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.607+0000 7f0bf6ffd700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 0x7f0bd806d190 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0be8006b20 tx=0x7f0be8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.610+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0be0053860 con 0x7f0bf80af940 2023-06-10T18:58:39.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:58:39.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:58:39.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.802+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bf8070340 con 0x7f0bf80af940 2023-06-10T18:58:39.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.802+0000 7f0beffff700 1 -- 172.21.15.104:0/1536954499 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v172) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0be0056e80 con 0x7f0bf80af940 2023-06-10T18:58:39.809 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:39.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 msgr2=0x7f0bd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 0x7f0bd806d190 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0be8006b20 tx=0x7f0be8008040 comp rx=0 tx=0).stop 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 msgr2=0x7f0bf80c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0be000c4b0 tx=0x7f0be000c7c0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 shutdown_connections 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bf815e720 0x7f0bf81455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bf80af940 0x7f0bf80c9370 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0bd806ace0 0x7f0bd806d190 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 --2- 172.21.15.104:0/1536954499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bf81545a0 0x7f0bf80b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 >> 172.21.15.104:0/1536954499 conn(0x7f0bf806e370 msgr2=0x7f0bf8158030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 shutdown_connections 2023-06-10T18:58:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.804+0000 7f0bfde2d700 1 -- 172.21.15.104:0/1536954499 wait complete. 2023-06-10T18:58:39.823 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:39.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.940+0000 7fccb753e700 1 Processor -- start 2023-06-10T18:58:39.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 -- start start 2023-06-10T18:58:39.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccb0142980 0x7fccb00ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fccb006d3b0 0x7fccb006d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb006dd50 con 0x7fccb01424d0 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb00cd7c0 con 0x7fccb006d3b0 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.941+0000 7fccb753e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb00cdaa0 con 0x7fccb0142980 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fccb006d3b0 0x7fccb006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fccb006d3b0 0x7fccb006d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40584/0 (socket says 172.21.15.104:40584) 2023-06-10T18:58:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 -- 172.21.15.104:0/89818951 learned_addr learned my addr 172.21.15.104:0/89818951 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb4ad9700 1 --2- 172.21.15.104:0/89818951 >> v2:172.21.15.114:3300/0 conn(0x7fccb0142980 0x7fccb00ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 -- 172.21.15.104:0/89818951 >> v2:172.21.15.114:3300/0 conn(0x7fccb0142980 msgr2=0x7fccb00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:39.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 --2- 172.21.15.104:0/89818951 >> v2:172.21.15.114:3300/0 conn(0x7fccb0142980 0x7fccb00ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 -- 172.21.15.104:0/89818951 >> v1:172.21.15.114:6789/0 conn(0x7fccb01424d0 legacy=0x7fccb00c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 -- 172.21.15.104:0/89818951 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00cdd80 con 0x7fccb006d3b0 2023-06-10T18:58:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fccb5adb700 1 --2- 172.21.15.104:0/89818951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb006d810 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fcca800d150 tx=0x7fcca800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7185365e7b55cf81 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fcca7fff700 1 -- 172.21.15.104:0/89818951 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8010070 con 0x7fccb006d3b0 2023-06-10T18:58:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fcca7fff700 1 -- 172.21.15.104:0/89818951 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca800dd80 con 0x7fccb006d3b0 2023-06-10T18:58:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.942+0000 7fcca7fff700 1 -- 172.21.15.104:0/89818951 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca800b310 con 0x7fccb006d3b0 2023-06-10T18:58:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 -- 172.21.15.104:0/89818951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 msgr2=0x7fccb006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 --2- 172.21.15.104:0/89818951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb006d810 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fcca800d150 tx=0x7fcca800d460 comp rx=0 tx=0).stop 2023-06-10T18:58:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 -- 172.21.15.104:0/89818951 shutdown_connections 2023-06-10T18:58:39.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 --2- 172.21.15.104:0/89818951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb006d810 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 --2- 172.21.15.104:0/89818951 >> v2:172.21.15.114:3300/0 conn(0x7fccb0142980 0x7fccb00ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 -- 172.21.15.104:0/89818951 >> 172.21.15.104:0/89818951 conn(0x7fccb00ceec0 msgr2=0x7fccb00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:39.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 -- 172.21.15.104:0/89818951 shutdown_connections 2023-06-10T18:58:39.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.943+0000 7fccb753e700 1 -- 172.21.15.104:0/89818951 wait complete. 2023-06-10T18:58:39.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.944+0000 7fccb753e700 1 Processor -- start 2023-06-10T18:58:39.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.944+0000 7fccb753e700 1 -- start start 2023-06-10T18:58:39.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 0x7fccb00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccb0142980 0x7fccb00ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00c6930 con 0x7fccb006d3b0 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00cac80 con 0x7fccb01424d0 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00c5f90 con 0x7fccb0142980 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40586/0 (socket says 172.21.15.104:40586) 2023-06-10T18:58:39.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 -- 172.21.15.104:0/4076706206 learned_addr learned my addr 172.21.15.104:0/4076706206 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:39.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb4ad9700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 0x7fccb00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb5adb700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccb0142980 0x7fccb00ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 -- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccb0142980 msgr2=0x7fccb00ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccb0142980 0x7fccb00ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 -- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 msgr2=0x7fccb00c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 0x7fccb00c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 -- 172.21.15.104:0/4076706206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca800bd40 con 0x7fccb006d3b0 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb4ad9700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 0x7fccb00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:39.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb52da700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fcca000c4e0 tx=0x7fcca000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00052d0 con 0x7fccb006d3b0 2023-06-10T18:58:39.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0005430 con 0x7fccb006d3b0 2023-06-10T18:58:39.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb00c6270 con 0x7fccb006d3b0 2023-06-10T18:58:39.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.945+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00038c0 con 0x7fccb006d3b0 2023-06-10T18:58:39.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.946+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb015d4c0 con 0x7fccb006d3b0 2023-06-10T18:58:39.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.947+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb00c6ba0 con 0x7fccb006d3b0 2023-06-10T18:58:39.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.947+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcca0003a20 con 0x7fccb006d3b0 2023-06-10T18:58:39.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.947+0000 7fcca5ffb700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 0x7fcc9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:39.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.947+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcca0088a10 con 0x7fccb006d3b0 2023-06-10T18:58:39.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.948+0000 7fccb4ad9700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 0x7fcc9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:39.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.948+0000 7fccb4ad9700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 0x7fcc9406d190 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fccb00c4f20 tx=0x7fccac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:39.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:39.952+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcca0053a10 con 0x7fccb006d3b0 2023-06-10T18:58:40.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.136+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fccb0074050 con 0x7fccb006d3b0 2023-06-10T18:58:40.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.136+0000 7fcca5ffb700 1 -- 172.21.15.104:0/4076706206 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcca0017020 con 0x7fccb006d3b0 2023-06-10T18:58:40.144 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 msgr2=0x7fcc9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 0x7fcc9406d190 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fccb00c4f20 tx=0x7fccac008040 comp rx=0 tx=0).stop 2023-06-10T18:58:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 msgr2=0x7fccb01338c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fcca000c4e0 tx=0x7fcca000c8a0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 shutdown_connections 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccb0142980 0x7fccb00ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccb006d3b0 0x7fccb01338c0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc9406ace0 0x7fcc9406d190 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 --2- 172.21.15.104:0/4076706206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccb01424d0 0x7fccb00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 >> 172.21.15.104:0/4076706206 conn(0x7fccb00ceec0 msgr2=0x7fccb00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 shutdown_connections 2023-06-10T18:58:40.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.138+0000 7fccb753e700 1 -- 172.21.15.104:0/4076706206 wait complete. 2023-06-10T18:58:40.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:40.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.273+0000 7ffb0f7d0700 1 Processor -- start 2023-06-10T18:58:40.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.273+0000 7ffb0f7d0700 1 -- start start 2023-06-10T18:58:40.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0815c6f0 0x7ffb080b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0806ea70 0x7ffb080b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffb080aec00 0x7ffb080ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080ae7e0 con 0x7ffb0806ea70 2023-06-10T18:58:40.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0806df90 con 0x7ffb080aec00 2023-06-10T18:58:40.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0f7d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0806e270 con 0x7ffb0815c6f0 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0806ea70 0x7ffb080b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ffb0806ea70 0x7ffb080b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40594/0 (socket says 172.21.15.104:40594) 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 -- 172.21.15.104:0/1819790688 learned_addr learned my addr 172.21.15.104:0/1819790688 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0dd6d700 1 --2- 172.21.15.104:0/1819790688 >> v2:172.21.15.114:3300/0 conn(0x7ffb080aec00 0x7ffb080ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 -- 172.21.15.104:0/1819790688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0815c6f0 msgr2=0x7ffb080b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:40.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0d56c700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0815c6f0 0x7ffb080b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0815c6f0 0x7ffb080b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 -- 172.21.15.104:0/1819790688 >> v2:172.21.15.114:3300/0 conn(0x7ffb080aec00 msgr2=0x7ffb080ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 --2- 172.21.15.104:0/1819790688 >> v2:172.21.15.114:3300/0 conn(0x7ffb080aec00 0x7ffb080ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.274+0000 7ffb0cd6b700 1 -- 172.21.15.104:0/1819790688 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0806e550 con 0x7ffb0806ea70 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0cd6b700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0806ea70 0x7ffb080b15b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ffb08072040 tx=0x7ffb04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fb625d6c0255ba server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffafffff700 1 -- 172.21.15.104:0/1819790688 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400b870 con 0x7ffb0806ea70 2023-06-10T18:58:40.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffafffff700 1 -- 172.21.15.104:0/1819790688 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0400b9d0 con 0x7ffb0806ea70 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffafffff700 1 -- 172.21.15.104:0/1819790688 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400bd60 con 0x7ffb0806ea70 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/1819790688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0806ea70 msgr2=0x7ffb080b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0806ea70 0x7ffb080b15b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ffb08072040 tx=0x7ffb04009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/1819790688 shutdown_connections 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/1819790688 >> v2:172.21.15.114:3300/0 conn(0x7ffb080aec00 0x7ffb080ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb0815c6f0 0x7ffb080b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/1819790688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0806ea70 0x7ffb080b15b0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/1819790688 >> 172.21.15.104:0/1819790688 conn(0x7ffb0806b8a0 msgr2=0x7ffb080cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/1819790688 shutdown_connections 2023-06-10T18:58:40.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.275+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/1819790688 wait complete. 2023-06-10T18:58:40.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.277+0000 7ffb0f7d0700 1 Processor -- start 2023-06-10T18:58:40.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.277+0000 7ffb0f7d0700 1 -- start start 2023-06-10T18:58:40.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0806ea70 0x7ffb080c3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080aec00 0x7ffb080b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080c38e0 con 0x7ffb0815c6f0 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080c4d00 con 0x7ffb0806ea70 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0f7d0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080c4fa0 con 0x7ffb080aec00 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40604/0 (socket says 172.21.15.104:40604) 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 -- 172.21.15.104:0/3110879810 learned_addr learned my addr 172.21.15.104:0/3110879810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0d56c700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0806ea70 0x7ffb080c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0cd6b700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080aec00 0x7ffb080b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 -- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080aec00 msgr2=0x7ffb080b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080aec00 0x7ffb080b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 -- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0806ea70 msgr2=0x7ffb080c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0806ea70 0x7ffb080c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.278+0000 7ffb0dd6d700 1 -- 172.21.15.104:0/3110879810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04009550 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffb0dd6d700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000cd20 tx=0x7ffb0000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00003a20 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00003b80 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb080c5280 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00004020 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.279+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb080c5840 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.280+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb080c5a60 con 0x7ffb0815c6f0 2023-06-10T18:58:40.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.280+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb00004180 con 0x7ffb0815c6f0 2023-06-10T18:58:40.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.280+0000 7ffafdffb700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.280+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffb00012070 con 0x7ffb0815c6f0 2023-06-10T18:58:40.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.281+0000 7ffb0d56c700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.281+0000 7ffb0d56c700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8006fd0 tx=0x7ffaf8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.285+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb00054d50 con 0x7ffb0815c6f0 2023-06-10T18:58:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.471+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb080b1a30 con 0x7ffb0815c6f0 2023-06-10T18:58:40.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.471+0000 7ffafdffb700 1 -- 172.21.15.104:0/3110879810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v172) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb00058370 con 0x7ffb0815c6f0 2023-06-10T18:58:40.480 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:40.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.473+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 msgr2=0x7ffaec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8006fd0 tx=0x7ffaf8008040 comp rx=0 tx=0).stop 2023-06-10T18:58:40.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 msgr2=0x7ffb080c8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000cd20 tx=0x7ffb0000a570 comp rx=0 tx=0).stop 2023-06-10T18:58:40.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 shutdown_connections 2023-06-10T18:58:40.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ffaec06ace0 0x7ffaec06d190 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb0806ea70 0x7ffb080c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb0815c6f0 0x7ffb080c8b70 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 --2- 172.21.15.104:0/3110879810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffb080aec00 0x7ffb080b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 >> 172.21.15.104:0/3110879810 conn(0x7ffb0806b8a0 msgr2=0x7ffb080ac370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 shutdown_connections 2023-06-10T18:58:40.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.474+0000 7ffb0f7d0700 1 -- 172.21.15.104:0/3110879810 wait complete. 2023-06-10T18:58:40.492 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:40.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.609+0000 7f9fb0740700 1 Processor -- start 2023-06-10T18:58:40.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.609+0000 7f9fb0740700 1 -- start start 2023-06-10T18:58:40.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fb0740700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 0x7f9fa80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fb0740700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9fa806e6e0 0x7f9fa80b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fb0740700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa806e250 con 0x7f9fa806cde0 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fb0740700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa815c950 con 0x7f9fa806e6e0 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fb0740700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80cf0e0 con 0x7f9fa815c580 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fadcdb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9fa806e6e0 0x7f9fa80b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fadcdb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f9fa806e6e0 0x7f9fa80b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40610/0 (socket says 172.21.15.104:40610) 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.610+0000 7f9fadcdb700 1 -- 172.21.15.104:0/2272217594 learned_addr learned my addr 172.21.15.104:0/2272217594 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fae4dc700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 0x7f9fa80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fadcdb700 1 -- 172.21.15.104:0/2272217594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 msgr2=0x7f9fa80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:40.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fadcdb700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 0x7f9fa80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fadcdb700 1 -- 172.21.15.104:0/2272217594 >> v1:172.21.15.114:6789/0 conn(0x7f9fa806cde0 legacy=0x7f9fa8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:40.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fadcdb700 1 -- 172.21.15.104:0/2272217594 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80b1e80 con 0x7f9fa806e6e0 2023-06-10T18:58:40.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fae4dc700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 0x7f9fa80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:40.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fadcdb700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa806e6e0 0x7f9fa80b15b0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40098f0 tx=0x7f9fa4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dd6432c4456bed server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fad4da700 1 -- 172.21.15.104:0/2272217594 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4004230 con 0x7f9fa806e6e0 2023-06-10T18:58:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fad4da700 1 -- 172.21.15.104:0/2272217594 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa4004390 con 0x7f9fa806e6e0 2023-06-10T18:58:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fad4da700 1 -- 172.21.15.104:0/2272217594 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40036a0 con 0x7f9fa806e6e0 2023-06-10T18:58:40.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fb0740700 1 -- 172.21.15.104:0/2272217594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa806e6e0 msgr2=0x7f9fa80b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.611+0000 7f9fb0740700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa806e6e0 0x7f9fa80b15b0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40098f0 tx=0x7f9fa4009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:40.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 -- 172.21.15.104:0/2272217594 shutdown_connections 2023-06-10T18:58:40.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa815c580 0x7f9fa80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 --2- 172.21.15.104:0/2272217594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa806e6e0 0x7f9fa80b15b0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 -- 172.21.15.104:0/2272217594 >> 172.21.15.104:0/2272217594 conn(0x7f9fa806b8a0 msgr2=0x7f9fa80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 -- 172.21.15.104:0/2272217594 shutdown_connections 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.612+0000 7f9fb0740700 1 -- 172.21.15.104:0/2272217594 wait complete. 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.613+0000 7f9fb0740700 1 Processor -- start 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.613+0000 7f9fb0740700 1 -- start start 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.613+0000 7f9fb0740700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fa806cde0 0x7f9fa8154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.613+0000 7f9fb0740700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa806e6e0 0x7f9fa80b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.613+0000 7f9fb0740700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fb0740700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80b3750 con 0x7f9fa815c580 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fb0740700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80b3100 con 0x7f9fa806cde0 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fb0740700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80c33e0 con 0x7f9fa806e6e0 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40618/0 (socket says 172.21.15.104:40618) 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 -- 172.21.15.104:0/4128227962 learned_addr learned my addr 172.21.15.104:0/4128227962 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fae4dc700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fa806cde0 0x7f9fa8154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fadcdb700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa806e6e0 0x7f9fa80b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 -- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa806e6e0 msgr2=0x7f9fa80b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa806e6e0 0x7f9fa80b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 -- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fa806cde0 msgr2=0x7f9fa8154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fa806cde0 0x7f9fa8154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 -- 172.21.15.104:0/4128227962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009550 con 0x7f9fa815c580 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9faecdd700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000b970 tx=0x7f9fa000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa000d860 con 0x7f9fa815c580 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa000d9c0 con 0x7f9fa815c580 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa80b3370 con 0x7f9fa815c580 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa000dd70 con 0x7f9fa815c580 2023-06-10T18:58:40.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.614+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa80c36c0 con 0x7f9fa815c580 2023-06-10T18:58:40.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.615+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa80c9a00 con 0x7f9fa815c580 2023-06-10T18:58:40.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.616+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9fa000e430 con 0x7f9fa815c580 2023-06-10T18:58:40.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.616+0000 7f9f9effd700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 0x7f9f8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.616+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9fa0089660 con 0x7f9fa815c580 2023-06-10T18:58:40.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.616+0000 7f9fae4dc700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 0x7f9f8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.617+0000 7f9fae4dc700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 0x7f9f8c06d190 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800abe0 tx=0x7f9f98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.620+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9fa0054660 con 0x7f9fa815c580 2023-06-10T18:58:40.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.806+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9fa80b3890 con 0x7f9fa815c580 2023-06-10T18:58:40.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.806+0000 7f9f9effd700 1 -- 172.21.15.104:0/4128227962 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v172) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9fa80b3890 con 0x7f9fa815c580 2023-06-10T18:58:40.813 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:40.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 msgr2=0x7f9f8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 0x7f9f8c06d190 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800abe0 tx=0x7f9f98008040 comp rx=0 tx=0).stop 2023-06-10T18:58:40.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 msgr2=0x7f9fa80b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000b970 tx=0x7f9fa000bc80 comp rx=0 tx=0).stop 2023-06-10T18:58:40.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 shutdown_connections 2023-06-10T18:58:40.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9f8c06ace0 0x7f9f8c06d190 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fa806cde0 0x7f9fa8154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9fa815c580 0x7f9fa80b4780 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 --2- 172.21.15.104:0/4128227962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9fa806e6e0 0x7f9fa80b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 >> 172.21.15.104:0/4128227962 conn(0x7f9fa806b8a0 msgr2=0x7f9fa815a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 shutdown_connections 2023-06-10T18:58:40.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.808+0000 7f9fb0740700 1 -- 172.21.15.104:0/4128227962 wait complete. 2023-06-10T18:58:40.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.942+0000 7f15d3543700 1 Processor -- start 2023-06-10T18:58:40.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.942+0000 7f15d3543700 1 -- start start 2023-06-10T18:58:40.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d3543700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15cc06e0e0 0x7f15cc06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d3543700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15cc069e70 0x7f15cc0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d3543700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15cc06d290 con 0x7f15cc06e4b0 2023-06-10T18:58:40.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d3543700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc071640 con 0x7f15cc069e70 2023-06-10T18:58:40.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d3543700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc0aef00 con 0x7f15cc06e0e0 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15cc069e70 0x7f15cc0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f15cc069e70 0x7f15cc0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40624/0 (socket says 172.21.15.104:40624) 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 -- 172.21.15.104:0/166643942 learned_addr learned my addr 172.21.15.104:0/166643942 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d12df700 1 --2- 172.21.15.104:0/166643942 >> v2:172.21.15.114:3300/0 conn(0x7f15cc06e0e0 0x7f15cc06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 -- 172.21.15.104:0/166643942 >> v2:172.21.15.114:3300/0 conn(0x7f15cc06e0e0 msgr2=0x7f15cc06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 --2- 172.21.15.104:0/166643942 >> v2:172.21.15.114:3300/0 conn(0x7f15cc06e0e0 0x7f15cc06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 -- 172.21.15.104:0/166643942 >> v1:172.21.15.114:6789/0 conn(0x7f15cc06e4b0 legacy=0x7f15cc0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 -- 172.21.15.104:0/166643942 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc0ca2d0 con 0x7f15cc069e70 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15d1ae0700 1 --2- 172.21.15.104:0/166643942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc0ce9f0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f15cc06af50 tx=0x7f15c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d445dd5b9c7b392d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15c3fff700 1 -- 172.21.15.104:0/166643942 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c400e070 con 0x7f15cc069e70 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15c3fff700 1 -- 172.21.15.104:0/166643942 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c400b9c0 con 0x7f15cc069e70 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.943+0000 7f15c3fff700 1 -- 172.21.15.104:0/166643942 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c400bd70 con 0x7f15cc069e70 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 -- 172.21.15.104:0/166643942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 msgr2=0x7f15cc0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 --2- 172.21.15.104:0/166643942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc0ce9f0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f15cc06af50 tx=0x7f15c400b330 comp rx=0 tx=0).stop 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 -- 172.21.15.104:0/166643942 shutdown_connections 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 --2- 172.21.15.104:0/166643942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc0ce9f0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 --2- 172.21.15.104:0/166643942 >> v2:172.21.15.114:3300/0 conn(0x7f15cc06e0e0 0x7f15cc06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 -- 172.21.15.104:0/166643942 >> 172.21.15.104:0/166643942 conn(0x7f15cc06b8a0 msgr2=0x7f15cc0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 -- 172.21.15.104:0/166643942 shutdown_connections 2023-06-10T18:58:40.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.944+0000 7f15d3543700 1 -- 172.21.15.104:0/166643942 wait complete. 2023-06-10T18:58:40.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.945+0000 7f15d3543700 1 Processor -- start 2023-06-10T18:58:40.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.945+0000 7f15d3543700 1 -- start start 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15cc06e0e0 0x7f15cc1569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15cc06e4b0 0x7f15cc0ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc0b24d0 con 0x7f15cc069e70 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc155680 con 0x7f15cc06e4b0 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc1560e0 con 0x7f15cc06e0e0 2023-06-10T18:58:40.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40626/0 (socket says 172.21.15.104:40626) 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 -- 172.21.15.104:0/1473844945 learned_addr learned my addr 172.21.15.104:0/1473844945 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d1ae0700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15cc06e4b0 0x7f15cc0ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d0ade700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15cc06e0e0 0x7f15cc1569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 -- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15cc06e0e0 msgr2=0x7f15cc1569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15cc06e0e0 0x7f15cc1569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 -- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15cc06e4b0 msgr2=0x7f15cc0ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:40.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15cc06e4b0 0x7f15cc0ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 -- 172.21.15.104:0/1473844945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4009b30 con 0x7f15cc069e70 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d12df700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00d830 tx=0x7f15bc00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc018070 con 0x7f15cc069e70 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc005a20 con 0x7f15cc069e70 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc005bb0 con 0x7f15cc069e70 2023-06-10T18:58:40.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.946+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc0b1bc0 con 0x7f15cc069e70 2023-06-10T18:58:40.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.947+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15cc0b2a20 con 0x7f15cc069e70 2023-06-10T18:58:40.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.948+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc074050 con 0x7f15cc069e70 2023-06-10T18:58:40.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.948+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f15bc0125f0 con 0x7f15cc069e70 2023-06-10T18:58:40.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.948+0000 7f15c1ffb700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 0x7f15b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:40.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.948+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f15bc089240 con 0x7f15cc069e70 2023-06-10T18:58:40.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.948+0000 7f15d0ade700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 0x7f15b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:40.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.949+0000 7f15d0ade700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 0x7f15b006d190 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f15cc072910 tx=0x7f15c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:40.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:40.952+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f15bc057cd0 con 0x7f15cc069e70 2023-06-10T18:58:41.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.154+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f15cc0c6190 con 0x7f15cc069e70 2023-06-10T18:58:41.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.155+0000 7f15c1ffb700 1 -- 172.21.15.104:0/1473844945 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v172) v1 ==== 80+0+0 (secure 0 0 0) 0x7f15bc057860 con 0x7f15cc069e70 2023-06-10T18:58:41.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 msgr2=0x7f15b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 0x7f15b006d190 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f15cc072910 tx=0x7f15c8009040 comp rx=0 tx=0).stop 2023-06-10T18:58:41.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 msgr2=0x7f15cc155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00d830 tx=0x7f15bc00dbf0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 shutdown_connections 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f15b006ace0 0x7f15b006d190 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15cc06e4b0 0x7f15cc0ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15cc069e70 0x7f15cc155140 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 --2- 172.21.15.104:0/1473844945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15cc06e0e0 0x7f15cc1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 >> 172.21.15.104:0/1473844945 conn(0x7f15cc06b8a0 msgr2=0x7f15cc0adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:41.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 shutdown_connections 2023-06-10T18:58:41.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.157+0000 7f15d3543700 1 -- 172.21.15.104:0/1473844945 wait complete. 2023-06-10T18:58:41.166 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:58:41.176 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:58:41.176 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:41.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.312+0000 7f485f548700 1 Processor -- start 2023-06-10T18:58:41.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.313+0000 7f485f548700 1 -- start start 2023-06-10T18:58:41.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.313+0000 7f485f548700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f48580cea60 0x7f4858165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.313+0000 7f485f548700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580b6070 con 0x7f48580b4ff0 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.313+0000 7f485f548700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580b1d70 con 0x7f485815a420 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.313+0000 7f485f548700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48580b5460 con 0x7f48580cea60 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485dae5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f48580cea60 0x7f4858165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485cae3700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f48580b4ff0 0x7f48580ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38464/0 (socket says 172.21.15.104:38464) 2023-06-10T18:58:41.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485cae3700 1 -- 172.21.15.104:0/3839453484 learned_addr learned my addr 172.21.15.104:0/3839453484 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485dae5700 1 -- 172.21.15.104:0/3839453484 >> v1:172.21.15.114:6789/0 conn(0x7f485815a420 legacy=0x7f48580b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485dae5700 1 -- 172.21.15.104:0/3839453484 >> v1:172.21.15.104:6789/0 conn(0x7f48580b4ff0 legacy=0x7f48580ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485dae5700 1 -- 172.21.15.104:0/3839453484 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48580b5740 con 0x7f48580cea60 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f485dae5700 1 --2- 172.21.15.104:0/3839453484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48580cea60 0x7f4858165060 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f4850009d90 tx=0x7f485000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9b3d68494c9ac1 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f484ffff700 1 -- 172.21.15.104:0/3839453484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485000e070 con 0x7f48580cea60 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f484ffff700 1 -- 172.21.15.104:0/3839453484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485000bb90 con 0x7f48580cea60 2023-06-10T18:58:41.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.314+0000 7f484ffff700 1 -- 172.21.15.104:0/3839453484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850003a20 con 0x7f48580cea60 2023-06-10T18:58:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 -- 172.21.15.104:0/3839453484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48580cea60 msgr2=0x7f4858165060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 --2- 172.21.15.104:0/3839453484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48580cea60 0x7f4858165060 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f4850009d90 tx=0x7f485000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 -- 172.21.15.104:0/3839453484 shutdown_connections 2023-06-10T18:58:41.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 --2- 172.21.15.104:0/3839453484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48580cea60 0x7f4858165060 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 -- 172.21.15.104:0/3839453484 >> 172.21.15.104:0/3839453484 conn(0x7f4858089fd0 msgr2=0x7f48580d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 -- 172.21.15.104:0/3839453484 shutdown_connections 2023-06-10T18:58:41.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.315+0000 7f485f548700 1 -- 172.21.15.104:0/3839453484 wait complete. 2023-06-10T18:58:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.316+0000 7f485f548700 1 Processor -- start 2023-06-10T18:58:41.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.316+0000 7f485f548700 1 -- start start 2023-06-10T18:58:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48580b4ff0 0x7f485814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48580cea60 0x7f48580c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485815b6e0 con 0x7f485815a420 2023-06-10T18:58:41.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485815a7f0 con 0x7f48580cea60 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485815aad0 con 0x7f48580b4ff0 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40644/0 (socket says 172.21.15.104:40644) 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 -- 172.21.15.104:0/3262036917 learned_addr learned my addr 172.21.15.104:0/3262036917 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485cae3700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48580cea60 0x7f48580c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485d2e4700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48580b4ff0 0x7f485814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 -- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48580b4ff0 msgr2=0x7f485814cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48580b4ff0 0x7f485814cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 -- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48580cea60 msgr2=0x7f48580c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48580cea60 0x7f48580c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 -- 172.21.15.104:0/3262036917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850009a70 con 0x7f485815a420 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485dae5700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f4858073ac0 tx=0x7f4850004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850012630 con 0x7f485815a420 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48500127b0 con 0x7f485815a420 2023-06-10T18:58:41.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485815adb0 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.317+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4850012b60 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.318+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485815b370 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.318+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48580b64e0 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.319+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4850012cc0 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.319+0000 7f484dffb700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 0x7f483c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.319+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f485008bc80 con 0x7f485815a420 2023-06-10T18:58:41.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.320+0000 7f485d2e4700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 0x7f483c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.320+0000 7f485d2e4700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 0x7f483c06d190 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4848009870 tx=0x7f4848008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.324+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f48580b64e0 con 0x7f485815a420 2023-06-10T18:58:41.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.513+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48580b64e0 con 0x7f485815a420 2023-06-10T18:58:41.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.515+0000 7f484dffb700 1 -- 172.21.15.104:0/3262036917 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+18614 (secure 0 0 0) 0x7f48580b64e0 con 0x7f485815a420 2023-06-10T18:58:41.522 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:41.523 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":172,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:30.309579+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:41.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 msgr2=0x7f483c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 0x7f483c06d190 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4848009870 tx=0x7f4848008040 comp rx=0 tx=0).stop 2023-06-10T18:58:41.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 msgr2=0x7f48581551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f4858073ac0 tx=0x7f4850004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 shutdown_connections 2023-06-10T18:58:41.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f485815a420 0x7f48581551a0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48580b4ff0 0x7f485814cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f483c06ace0 0x7f483c06d190 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 --2- 172.21.15.104:0/3262036917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48580cea60 0x7f48580c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 >> 172.21.15.104:0/3262036917 conn(0x7f4858089fd0 msgr2=0x7f48580b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:41.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 shutdown_connections 2023-06-10T18:58:41.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.517+0000 7f485f548700 1 -- 172.21.15.104:0/3262036917 wait complete. 2023-06-10T18:58:41.536 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:41.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 Processor -- start 2023-06-10T18:58:41.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 -- start start 2023-06-10T18:58:41.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbe80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.663 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdbe80cd8f0 0x7fdbe806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80cc9f0 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80cbbd0 con 0x7fdbe80cd8f0 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.654+0000 7fdbeed7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80cc100 con 0x7fdbe80cb210 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe7fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdbe80cf0b0 0x7fdbe80cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38478/0 (socket says 172.21.15.104:38478) 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe7fff700 1 -- 172.21.15.104:0/2112149151 learned_addr learned my addr 172.21.15.104:0/2112149151 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:41.664 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462554168 0 0) 0x7fdbe80cc9f0 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbed31b700 1 --2- 172.21.15.104:0/2112149151 >> v2:172.21.15.114:3300/0 conn(0x7fdbe80cd8f0 0x7fdbe806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbe80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd80057c0 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796547761 0 0) 0x7fdbd80057c0 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.665 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 msgr2=0x7fdbe80cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.666 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbe80cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 >> v2:172.21.15.114:3300/0 conn(0x7fdbe80cd8f0 msgr2=0x7fdbe806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.667 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 --2- 172.21.15.104:0/2112149151 >> v2:172.21.15.114:3300/0 conn(0x7fdbe80cd8f0 0x7fdbe806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80cc3a0 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fdbd4001670 con 0x7fdbe80cf0b0 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 >> v1:172.21.15.104:6789/0 conn(0x7fdbe80cf0b0 legacy=0x7fdbe80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbd8005c10 0x7fdbd8006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbd8006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8006fa0 0x7fdbd8007410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80cbbd0 con 0x7fdbd8005c10 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80cc100 con 0x7fdbd8006fa0 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd800dc20 con 0x7fdbe80cb210 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbd8005c10 0x7fdbd8006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbed31b700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8006fa0 0x7fdbd8007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.669 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.655+0000 7fdbe7fff700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbd8006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 -- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 msgr2=0x7fdbd8006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80cb210 0x7fdbd8006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 -- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8006fa0 msgr2=0x7fdbd8007410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8006fa0 0x7fdbd8007410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 -- 172.21.15.104:0/2112149151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd80057c0 con 0x7fdbd8005c10 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbed31b700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8006fa0 0x7fdbd8007410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:41.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbd8005c10 0x7fdbd8006020 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc009b40 tx=0x7fdbd8007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfb58c2c914d85e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc005a20 con 0x7fdbd8005c10 2023-06-10T18:58:41.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc005b80 con 0x7fdbd8005c10 2023-06-10T18:58:41.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbe77fe700 1 -- 172.21.15.104:0/2112149151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc00e660 con 0x7fdbd8005c10 2023-06-10T18:58:41.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 -- 172.21.15.104:0/2112149151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbd8005c10 msgr2=0x7fdbd8006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/2112149151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbd8005c10 0x7fdbd8006020 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc009b40 tx=0x7fdbd8007a90 comp rx=0 tx=0).stop 2023-06-10T18:58:41.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbed31b700 1 -- 172.21.15.104:0/2112149151 reap_dead start 2023-06-10T18:58:41.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 -- 172.21.15.104:0/2112149151 shutdown_connections 2023-06-10T18:58:41.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 -- 172.21.15.104:0/2112149151 >> 172.21.15.104:0/2112149151 conn(0x7fdbe806b8a0 msgr2=0x7fdbe80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:41.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 -- 172.21.15.104:0/2112149151 shutdown_connections 2023-06-10T18:58:41.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.656+0000 7fdbeed7e700 1 -- 172.21.15.104:0/2112149151 wait complete. 2023-06-10T18:58:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.658+0000 7fdbeed7e700 1 Processor -- start 2023-06-10T18:58:41.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.658+0000 7fdbeed7e700 1 -- start start 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 0x7fdbe80b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80b3400 0x7fdbe81f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80b3870 con 0x7fdbe814b040 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80c3470 con 0x7fdbd8005c10 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbeed7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80c3750 con 0x7fdbe80b3400 2023-06-10T18:58:41.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40662/0 (socket says 172.21.15.104:40662) 2023-06-10T18:58:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 -- 172.21.15.104:0/1022381816 learned_addr learned my addr 172.21.15.104:0/1022381816 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 0x7fdbe80b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbed31b700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80b3400 0x7fdbe81f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 -- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80b3400 msgr2=0x7fdbe81f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80b3400 0x7fdbe81f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 -- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 msgr2=0x7fdbe80b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 0x7fdbe80b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbe7fff700 1 -- 172.21.15.104:0/1022381816 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80c3a30 con 0x7fdbe814b040 2023-06-10T18:58:41.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.659+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 0x7fdbe80b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:41.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbe7fff700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400e8c0 tx=0x7fdbd400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400eec0 con 0x7fdbe814b040 2023-06-10T18:58:41.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd400c7e0 con 0x7fdbe814b040 2023-06-10T18:58:41.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe80c3d10 con 0x7fdbe814b040 2023-06-10T18:58:41.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400cb70 con 0x7fdbe814b040 2023-06-10T18:58:41.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.660+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe80b5430 con 0x7fdbe814b040 2023-06-10T18:58:41.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.661+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe8074050 con 0x7fdbe814b040 2023-06-10T18:58:41.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.661+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdbd400ccd0 con 0x7fdbe814b040 2023-06-10T18:58:41.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.661+0000 7fdbe57fa700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 0x7fdbcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.662+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdbd40893f0 con 0x7fdbe814b040 2023-06-10T18:58:41.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.662+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 0x7fdbcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.662+0000 7fdbecb1a700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 0x7fdbcc06d190 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc000c00 tx=0x7fdbdc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.666+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdbd40543f0 con 0x7fdbe814b040 2023-06-10T18:58:41.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.850+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbe80caad0 con 0x7fdbe814b040 2023-06-10T18:58:41.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.850+0000 7fdbe57fa700 1 -- 172.21.15.104:0/1022381816 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v172) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbd4057a10 con 0x7fdbe814b040 2023-06-10T18:58:41.857 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:41.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 msgr2=0x7fdbcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 0x7fdbcc06d190 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc000c00 tx=0x7fdbdc008030 comp rx=0 tx=0).stop 2023-06-10T18:58:41.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 msgr2=0x7fdbe80b2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400e8c0 tx=0x7fdbd400ec80 comp rx=0 tx=0).stop 2023-06-10T18:58:41.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 shutdown_connections 2023-06-10T18:58:41.860 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdbcc06ace0 0x7fdbcc06d190 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8005c10 0x7fdbe80b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbe80b3400 0x7fdbe81f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 --2- 172.21.15.104:0/1022381816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdbe814b040 0x7fdbe80b2ec0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.852+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 >> 172.21.15.104:0/1022381816 conn(0x7fdbe806b8a0 msgr2=0x7fdbe8158b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:41.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.853+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 shutdown_connections 2023-06-10T18:58:41.862 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.853+0000 7fdbeed7e700 1 -- 172.21.15.104:0/1022381816 wait complete. 2023-06-10T18:58:41.871 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:41.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.988+0000 7f2ebd7e3700 1 Processor -- start 2023-06-10T18:58:41.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.988+0000 7f2ebd7e3700 1 -- start start 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.988+0000 7f2ebd7e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.988+0000 7f2ebd7e3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2eb80caf70 0x7f2eb80cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2ebd7e3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8071580 con 0x7f2eb80ccfe0 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2ebd7e3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80cb950 con 0x7f2eb80caf70 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2ebd7e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80c9020 con 0x7f2eb8158790 2023-06-10T18:58:41.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb67fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2eb80ccfe0 0x7f2eb80c95c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38498/0 (socket says 172.21.15.104:38498) 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb67fc700 1 -- 172.21.15.104:0/203247107 learned_addr learned my addr 172.21.15.104:0/203247107 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 --2- 172.21.15.104:0/203247107 >> v2:172.21.15.114:3300/0 conn(0x7f2eb80caf70 0x7f2eb80cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb5ffb700 1 -- 172.21.15.104:0/203247107 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445831032 0 0) 0x7f2eb8071580 con 0x7f2eb80ccfe0 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb5ffb700 1 -- 172.21.15.104:0/203247107 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea00035b0 con 0x7f2eb80ccfe0 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 -- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 msgr2=0x7f2eb80ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:41.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 -- 172.21.15.104:0/203247107 >> v1:172.21.15.104:6789/0 conn(0x7f2eb80ccfe0 legacy=0x7f2eb80c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 -- 172.21.15.104:0/203247107 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8070f30 con 0x7f2eb80caf70 2023-06-10T18:58:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb77fe700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80caf70 0x7f2eb80cb410 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009d90 tx=0x7f2eb000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a8ab140c4cbed14 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.989+0000 7f2eb5ffb700 1 -- 172.21.15.104:0/203247107 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000e070 con 0x7f2eb80caf70 2023-06-10T18:58:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2eb5ffb700 1 -- 172.21.15.104:0/203247107 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb000bb20 con 0x7f2eb80caf70 2023-06-10T18:58:41.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2eb5ffb700 1 -- 172.21.15.104:0/203247107 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000be90 con 0x7f2eb80caf70 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80caf70 msgr2=0x7f2eb80cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80caf70 0x7f2eb80cb410 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009d90 tx=0x7f2eb000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/203247107 shutdown_connections 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80caf70 0x7f2eb80cb410 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/203247107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/203247107 >> 172.21.15.104:0/203247107 conn(0x7f2eb806aac0 msgr2=0x7f2eb80b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/203247107 shutdown_connections 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.990+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/203247107 wait complete. 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.991+0000 7f2ebd7e3700 1 Processor -- start 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.991+0000 7f2ebd7e3700 1 -- start start 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 0x7f2eb80c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80c7760 con 0x7f2eb80caf70 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80c6ae0 con 0x7f2eb80ccfe0 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2ebd7e3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80c6dc0 con 0x7f2eb8158790 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2eb6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2eb6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40674/0 (socket says 172.21.15.104:40674) 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2eb6ffd700 1 -- 172.21.15.104:0/3528038797 learned_addr learned my addr 172.21.15.104:0/3528038797 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.992+0000 7f2eb77fe700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb67fc700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 0x7f2eb80c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 -- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 msgr2=0x7f2eb80b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 -- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 msgr2=0x7f2eb80c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 0x7f2eb80c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 -- 172.21.15.104:0/3528038797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0009a70 con 0x7f2eb80caf70 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb77fe700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:42.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb67fc700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 0x7f2eb80c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800e870 tx=0x7f2ea800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8009590 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea800ee70 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb80c70a0 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800ca50 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.993+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80c7b80 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.994+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb80c78a0 con 0x7f2eb80caf70 2023-06-10T18:58:42.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.994+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2ea800cbb0 con 0x7f2eb80caf70 2023-06-10T18:58:42.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.995+0000 7f2e9f7fe700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 0x7f2ea406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.995+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2ea80894d0 con 0x7f2eb80caf70 2023-06-10T18:58:42.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.995+0000 7f2eb67fc700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 0x7f2ea406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.995+0000 7f2eb67fc700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 0x7f2ea406d190 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80b2cd0 tx=0x7f2eac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:41.999+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2eb80c78a0 con 0x7f2eb80caf70 2023-06-10T18:58:42.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.182+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2eb80c78a0 con 0x7f2eb80caf70 2023-06-10T18:58:42.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.182+0000 7f2e9f7fe700 1 -- 172.21.15.104:0/3528038797 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2eb80c78a0 con 0x7f2eb80caf70 2023-06-10T18:58:42.190 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:42.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 msgr2=0x7f2ea406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 0x7f2ea406d190 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80b2cd0 tx=0x7f2eac009040 comp rx=0 tx=0).stop 2023-06-10T18:58:42.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 msgr2=0x7f2eb80c0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800e870 tx=0x7f2ea800ec30 comp rx=0 tx=0).stop 2023-06-10T18:58:42.192 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 shutdown_connections 2023-06-10T18:58:42.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2eb8158790 0x7f2eb80b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2eb80caf70 0x7f2eb80c0e90 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.193 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ea406ace0 0x7f2ea406d190 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 --2- 172.21.15.104:0/3528038797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb80ccfe0 0x7f2eb80c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 >> 172.21.15.104:0/3528038797 conn(0x7f2eb806aac0 msgr2=0x7f2eb8071910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.194 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 shutdown_connections 2023-06-10T18:58:42.195 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.184+0000 7f2ebd7e3700 1 -- 172.21.15.104:0/3528038797 wait complete. 2023-06-10T18:58:42.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:42.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.318+0000 7f1983b97700 1 Processor -- start 2023-06-10T18:58:42.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.318+0000 7f1983b97700 1 -- start start 2023-06-10T18:58:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1983b97700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c06cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1983b97700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c0ae7e0 con 0x7f197c06ea70 2023-06-10T18:58:42.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1983b97700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c06df90 con 0x7f197c15c6f0 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1983b97700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c06e270 con 0x7f197c0b1780 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1982134700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f197c06ea70 0x7f197c06d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38514/0 (socket says 172.21.15.104:38514) 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1982134700 1 -- 172.21.15.104:0/2854660800 learned_addr learned my addr 172.21.15.104:0/2854660800 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1981132700 1 --2- 172.21.15.104:0/2854660800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c06cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119706815 0 0) 0x7f197c0ae7e0 con 0x7f197c06ea70 2023-06-10T18:58:42.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19680035b0 con 0x7f197c06ea70 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.319+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519283957 0 0) 0x7f197c06df90 con 0x7f197c15c6f0 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c0ae7e0 con 0x7f197c15c6f0 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3453896375 0 0) 0x7f19680035b0 con 0x7f197c06ea70 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1981132700 1 -- 172.21.15.104:0/2854660800 >> v1:172.21.15.114:6789/0 conn(0x7f197c15c6f0 legacy=0x7f197c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1981132700 1 -- 172.21.15.104:0/2854660800 >> v1:172.21.15.104:6789/0 conn(0x7f197c06ea70 legacy=0x7f197c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1981132700 1 -- 172.21.15.104:0/2854660800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c06e550 con 0x7f197c0b1780 2023-06-10T18:58:42.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1981132700 1 --2- 172.21.15.104:0/2854660800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c06cde0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f1978009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b139a143ba0906 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19740019f0 con 0x7f197c0b1780 2023-06-10T18:58:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1978004030 con 0x7f197c0b1780 2023-06-10T18:58:42.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1980931700 1 -- 172.21.15.104:0/2854660800 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19780043c0 con 0x7f197c0b1780 2023-06-10T18:58:42.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1983b97700 1 -- 172.21.15.104:0/2854660800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 msgr2=0x7f197c06cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.320+0000 7f1983b97700 1 --2- 172.21.15.104:0/2854660800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c06cde0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f1978009b80 comp rx=0 tx=0).stop 2023-06-10T18:58:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.321+0000 7f1983b97700 1 -- 172.21.15.104:0/2854660800 shutdown_connections 2023-06-10T18:58:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.321+0000 7f1983b97700 1 --2- 172.21.15.104:0/2854660800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c06cde0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.321+0000 7f1983b97700 1 -- 172.21.15.104:0/2854660800 >> 172.21.15.104:0/2854660800 conn(0x7f197c06b8a0 msgr2=0x7f197c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.321+0000 7f1983b97700 1 -- 172.21.15.104:0/2854660800 shutdown_connections 2023-06-10T18:58:42.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.321+0000 7f1983b97700 1 -- 172.21.15.104:0/2854660800 wait complete. 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 Processor -- start 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 -- start start 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c06ea70 0x7f197c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0b3a70 con 0x7f197c15c6f0 2023-06-10T18:58:42.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0b2e30 con 0x7f197c06ea70 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.322+0000 7f1983b97700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c1534c0 con 0x7f197c0b1780 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40692/0 (socket says 172.21.15.104:40692) 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 -- 172.21.15.104:0/4263781730 learned_addr learned my addr 172.21.15.104:0/4263781730 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1981933700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c06ea70 0x7f197c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1981132700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 -- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 msgr2=0x7f197c1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 -- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c06ea70 msgr2=0x7f197c152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c06ea70 0x7f197c152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 -- 172.21.15.104:0/4263781730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19780094a0 con 0x7f197c15c6f0 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1982134700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f1974001830 tx=0x7f197400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400ab30 con 0x7f197c15c6f0 2023-06-10T18:58:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197400ac90 con 0x7f197c15c6f0 2023-06-10T18:58:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c0b30d0 con 0x7f197c15c6f0 2023-06-10T18:58:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19740070d0 con 0x7f197c15c6f0 2023-06-10T18:58:42.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.323+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c0b3690 con 0x7f197c15c6f0 2023-06-10T18:58:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.324+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c0b3bb0 con 0x7f197c15c6f0 2023-06-10T18:58:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.325+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1974007230 con 0x7f197c15c6f0 2023-06-10T18:58:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.325+0000 7f19727fc700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 0x7f196006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.325+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1974089490 con 0x7f197c15c6f0 2023-06-10T18:58:42.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.325+0000 7f1981933700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 0x7f196006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.326+0000 7f1981933700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 0x7f196006d190 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f196c007fd0 tx=0x7f196c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.329+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f197c0b3bb0 con 0x7f197c15c6f0 2023-06-10T18:58:42.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.514+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f197c0b3bb0 con 0x7f197c15c6f0 2023-06-10T18:58:42.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.515+0000 7f19727fc700 1 -- 172.21.15.104:0/4263781730 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v172) v1 ==== 104+0+11 (secure 0 0 0) 0x7f197c0b3bb0 con 0x7f197c15c6f0 2023-06-10T18:58:42.522 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:42.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 msgr2=0x7f196006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 0x7f196006d190 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f196c007fd0 tx=0x7f196c009040 comp rx=0 tx=0).stop 2023-06-10T18:58:42.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 msgr2=0x7f197c157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f1974001830 tx=0x7f197400a060 comp rx=0 tx=0).stop 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 shutdown_connections 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f196006ace0 0x7f196006d190 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c06ea70 0x7f197c152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f197c15c6f0 0x7f197c157c80 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 --2- 172.21.15.104:0/4263781730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c0b1780 0x7f197c1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 >> 172.21.15.104:0/4263781730 conn(0x7f197c06b8a0 msgr2=0x7f197c15b690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 shutdown_connections 2023-06-10T18:58:42.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.517+0000 7f1983b97700 1 -- 172.21.15.104:0/4263781730 wait complete. 2023-06-10T18:58:42.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:42.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.655+0000 7f43c544d700 1 Processor -- start 2023-06-10T18:58:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.655+0000 7f43c544d700 1 -- start start 2023-06-10T18:58:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43c544d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c00cd8f0 0x7f43c00ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43c544d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43c00ad760 0x7f43c00adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43c544d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c015cb00 con 0x7f43c015c730 2023-06-10T18:58:42.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43c544d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c00acd90 con 0x7f43c00ad760 2023-06-10T18:58:42.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43c544d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c00ac460 con 0x7f43c00cd8f0 2023-06-10T18:58:42.673 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43bf7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43c00ad760 0x7f43c00adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.656+0000 7f43beffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f43c015c730 0x7f43c00cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38524/0 (socket says 172.21.15.104:38524) 2023-06-10T18:58:42.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f43c00ad760 0x7f43c00adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40700/0 (socket says 172.21.15.104:40700) 2023-06-10T18:58:42.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43beffd700 1 -- 172.21.15.104:0/3435378748 learned_addr learned my addr 172.21.15.104:0/3435378748 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:42.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43be7fc700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c00cd8f0 0x7f43c00ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 -- 172.21.15.104:0/3435378748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c00cd8f0 msgr2=0x7f43c00ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c00cd8f0 0x7f43c00ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 -- 172.21.15.104:0/3435378748 >> v1:172.21.15.104:6789/0 conn(0x7f43c015c730 legacy=0x7f43c00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 -- 172.21.15.104:0/3435378748 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c00ac740 con 0x7f43c00ad760 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bdffb700 1 -- 172.21.15.104:0/3435378748 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560452808 0 0) 0x7f43b4001290 con 0x7f43c015c730 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bf7fe700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00ad760 0x7f43c00adc00 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f43b800bfa0 tx=0x7f43b800d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8354662226ac866 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bdffb700 1 -- 172.21.15.104:0/3435378748 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8010070 con 0x7f43c00ad760 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bdffb700 1 -- 172.21.15.104:0/3435378748 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b800b2e0 con 0x7f43c00ad760 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.657+0000 7f43bdffb700 1 -- 172.21.15.104:0/3435378748 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b800dcf0 con 0x7f43c00ad760 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 -- 172.21.15.104:0/3435378748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00ad760 msgr2=0x7f43c00adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00ad760 0x7f43c00adc00 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f43b800bfa0 tx=0x7f43b800d330 comp rx=0 tx=0).stop 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 -- 172.21.15.104:0/3435378748 shutdown_connections 2023-06-10T18:58:42.676 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00ad760 0x7f43c00adc00 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 --2- 172.21.15.104:0/3435378748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c00cd8f0 0x7f43c00ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 -- 172.21.15.104:0/3435378748 >> 172.21.15.104:0/3435378748 conn(0x7f43c006b8a0 msgr2=0x7f43c00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 -- 172.21.15.104:0/3435378748 shutdown_connections 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.658+0000 7f43c544d700 1 -- 172.21.15.104:0/3435378748 wait complete. 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.659+0000 7f43c544d700 1 Processor -- start 2023-06-10T18:58:42.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.659+0000 7f43c544d700 1 -- start start 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43c00ad760 0x7f43c014ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c015c730 0x7f43c00b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c01504d0 con 0x7f43c00cd8f0 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c00b4c50 con 0x7f43c00ad760 2023-06-10T18:58:42.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43c544d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c00afd40 con 0x7f43c015c730 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40704/0 (socket says 172.21.15.104:40704) 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 -- 172.21.15.104:0/3553709004 learned_addr learned my addr 172.21.15.104:0/3553709004 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 -- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c015c730 msgr2=0x7f43c00b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c015c730 0x7f43c00b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 -- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43c00ad760 msgr2=0x7f43c014ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43c00ad760 0x7f43c014ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 -- 172.21.15.104:0/3553709004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b800bc80 con 0x7f43c00cd8f0 2023-06-10T18:58:42.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43be7fc700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e810 tx=0x7f43b000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.660+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0013610 con 0x7f43c00cd8f0 2023-06-10T18:58:42.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.661+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0013770 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.661+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c006f150 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.661+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0013b00 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.661+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c00b0300 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.662+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c00b0520 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.662+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f43b0013c60 con 0x7f43c00cd8f0 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.662+0000 7f43a77fe700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 0x7f43ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.662+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f43b00892e0 con 0x7f43c00cd8f0 2023-06-10T18:58:42.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.662+0000 7f43beffd700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 0x7f43ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.663+0000 7f43beffd700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 0x7f43ac06d190 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f43b4007fd0 tx=0x7f43b40079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:42.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.666+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43b0057c50 con 0x7f43c00cd8f0 2023-06-10T18:58:42.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.854+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43c0074050 con 0x7f43c00cd8f0 2023-06-10T18:58:42.861 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.854+0000 7f43a77fe700 1 -- 172.21.15.104:0/3553709004 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v172) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43b0017020 con 0x7f43c00cd8f0 2023-06-10T18:58:42.862 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:42.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 msgr2=0x7f43ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.863 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 0x7f43ac06d190 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f43b4007fd0 tx=0x7f43b40079a0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 msgr2=0x7f43c00b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e810 tx=0x7f43b000ebd0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 shutdown_connections 2023-06-10T18:58:42.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f43ac06ace0 0x7f43ac06d190 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43c00ad760 0x7f43c014ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43c015c730 0x7f43c00b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 --2- 172.21.15.104:0/3553709004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43c00cd8f0 0x7f43c00b2440 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 >> 172.21.15.104:0/3553709004 conn(0x7f43c006b8a0 msgr2=0x7f43c006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:42.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 shutdown_connections 2023-06-10T18:58:42.867 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.856+0000 7f43c544d700 1 -- 172.21.15.104:0/3553709004 wait complete. 2023-06-10T18:58:42.874 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:42.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.988+0000 7f4234616700 1 Processor -- start 2023-06-10T18:58:42.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.988+0000 7f4234616700 1 -- start start 2023-06-10T18:58:42.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.989+0000 7f4234616700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 0x7f422c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.989+0000 7f4234616700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f422c15a250 0x7f422c0cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.989+0000 7f4234616700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0ccca0 con 0x7f422c0cad90 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.989+0000 7f4234616700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0d0310 con 0x7f422c15a250 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.989+0000 7f4234616700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0cee50 con 0x7f422c0cfa00 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f422c15a250 0x7f422c0cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f42323b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 0x7f422c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f422c15a250 0x7f422c0cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:56444/0 (socket says 172.21.15.104:56444) 2023-06-10T18:58:42.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 -- 172.21.15.104:0/2353145440 learned_addr learned my addr 172.21.15.104:0/2353145440 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4232bb3700 1 --1- 172.21.15.104:0/2353145440 >> v1:172.21.15.114:6789/0 conn(0x7f422c0cad90 0x7f422c0b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44388/0 (socket says 172.21.15.104:44388) 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 -- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 msgr2=0x7f422c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 0x7f422c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 -- 172.21.15.104:0/2353145440 >> v1:172.21.15.114:6789/0 conn(0x7f422c0cad90 legacy=0x7f422c0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 -- 172.21.15.104:0/2353145440 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c0cf130 con 0x7f422c15a250 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f42323b2700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 0x7f422c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:42.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f4231bb1700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c15a250 0x7f422c0cfdd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f42280098f0 tx=0x7f4228009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6b8500da280f2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.990+0000 7f42313b0700 1 -- 172.21.15.104:0/2353145440 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228011830 con 0x7f422c15a250 2023-06-10T18:58:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f42313b0700 1 -- 172.21.15.104:0/2353145440 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228011990 con 0x7f422c15a250 2023-06-10T18:58:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f42313b0700 1 -- 172.21.15.104:0/2353145440 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228011d20 con 0x7f422c15a250 2023-06-10T18:58:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 -- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c15a250 msgr2=0x7f422c0cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:43.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c15a250 0x7f422c0cfdd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f42280098f0 tx=0x7f4228009c00 comp rx=0 tx=0).stop 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 -- 172.21.15.104:0/2353145440 shutdown_connections 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c0cfa00 0x7f422c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 --2- 172.21.15.104:0/2353145440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c15a250 0x7f422c0cfdd0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 -- 172.21.15.104:0/2353145440 >> 172.21.15.104:0/2353145440 conn(0x7f422c06e370 msgr2=0x7f422c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 -- 172.21.15.104:0/2353145440 shutdown_connections 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.991+0000 7f4234616700 1 -- 172.21.15.104:0/2353145440 wait complete. 2023-06-10T18:58:43.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.992+0000 7f4234616700 1 Processor -- start 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.992+0000 7f4234616700 1 -- start start 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c0cad90 0x7f422c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c15a250 0x7f422c0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c06c180 con 0x7f422c0cfa00 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c06cfd0 con 0x7f422c0cad90 2023-06-10T18:58:43.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4234616700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c06ccf0 con 0x7f422c15a250 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4231bb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4231bb1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40718/0 (socket says 172.21.15.104:40718) 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4231bb1700 1 -- 172.21.15.104:0/158346803 learned_addr learned my addr 172.21.15.104:0/158346803 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f42323b2700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c0cad90 0x7f422c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.993+0000 7f4232bb3700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c15a250 0x7f422c0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:43.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 -- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c15a250 msgr2=0x7f422c0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c15a250 0x7f422c0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 -- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c0cad90 msgr2=0x7f422c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c0cad90 0x7f422c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 -- 172.21.15.104:0/158346803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0094a0 con 0x7f422c0cfa00 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4231bb1700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f4228000c00 tx=0x7f4228004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228004480 con 0x7f422c0cfa00 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422801f070 con 0x7f422c0cfa00 2023-06-10T18:58:43.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4228009550 con 0x7f422c0cfa00 2023-06-10T18:58:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280038c0 con 0x7f422c0cfa00 2023-06-10T18:58:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.994+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c06bda0 con 0x7f422c0cfa00 2023-06-10T18:58:43.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.995+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c06c2c0 con 0x7f422c0cfa00 2023-06-10T18:58:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.995+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4228021030 con 0x7f422c0cfa00 2023-06-10T18:58:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.995+0000 7f4222ffd700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 0x7f421006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.996+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f422808aa10 con 0x7f422c0cfa00 2023-06-10T18:58:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.996+0000 7f42323b2700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 0x7f421006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:42.996+0000 7f42323b2700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 0x7f421006d190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f421c009cd0 tx=0x7f421c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.000+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f422c06c2c0 con 0x7f422c0cfa00 2023-06-10T18:58:43.205 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.199+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f422c06c2c0 con 0x7f422c0cfa00 2023-06-10T18:58:43.206 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.199+0000 7f4222ffd700 1 -- 172.21.15.104:0/158346803 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v172) v1 ==== 80+0+0 (secure 0 0 0) 0x7f422c06c2c0 con 0x7f422c0cfa00 2023-06-10T18:58:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 msgr2=0x7f421006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:43.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 0x7f421006d190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f421c009cd0 tx=0x7f421c009210 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 msgr2=0x7f422c0c5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f4228000c00 tx=0x7f4228004750 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 shutdown_connections 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f421006ace0 0x7f421006d190 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f422c0cad90 0x7f422c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f422c15a250 0x7f422c0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 --2- 172.21.15.104:0/158346803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f422c0cfa00 0x7f422c0c5a40 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:43.209 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.201+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 >> 172.21.15.104:0/158346803 conn(0x7f422c06e370 msgr2=0x7f422c1545c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:43.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.202+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 shutdown_connections 2023-06-10T18:58:43.210 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:43.202+0000 7f4234616700 1 -- 172.21.15.104:0/158346803 wait complete. 2023-06-10T18:58:43.210 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:58:43.220 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T18:58:43.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:43.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:43.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[117671]: fuse finished with error 0 and tester_r 0 2023-06-10T18:58:43.260 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:58:49.362 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:58:49.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:49.363 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:58:49.384 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-10T18:58:49.385 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:49.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:58:49.386 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:58:49.386 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:58:49.386 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:58:49.387 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:58:49.387 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:49.387 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:58:49.387 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:58:49.388 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:58:49.388 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:58:49.388 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:49.389 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:58:49.389 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:58:49.389 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:58:49.390 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:58:49.390 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:49.390 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:58:49.390 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:58:49.391 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:58:49.391 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:58:49.391 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:58:49.392 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:58:49.392 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:58:49.392 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:58:49.392 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:49.393 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:58:49.393 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:58:49.488 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.160529.asok 2023-06-10T18:58:49.669 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.160529.asok 2023-06-10T18:58:49.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:49.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.160529.asok mds_sessions 2023-06-10T18:58:49.829 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10767, 2023-06-10T18:58:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10767 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 827471637 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10767 v1:192.168.0.1:0/827471637", 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/827471637", 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6835", 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1296534750 2023-06-10T18:58:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.114:6837", 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 1296534750 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T18:59:37.376109+0000", 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:58:37.376109+0000", 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 1, 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:58:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 110 2023-06-10T18:58:49.835 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:49.839 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:58:49.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.952+0000 7f0f33477700 1 Processor -- start 2023-06-10T18:58:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.952+0000 7f0f33477700 1 -- start start 2023-06-10T18:58:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f33477700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f2c0ce350 0x7f0f2c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f33477700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f33477700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c0cadd0 con 0x7f0f2c06b1f0 2023-06-10T18:58:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f33477700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0cfeb0 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f33477700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0cb0b0 con 0x7f0f2c0ce350 2023-06-10T18:58:49.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31213700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0f2c06b1f0 0x7f0f2c0cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38552/0 (socket says 172.21.15.104:38552) 2023-06-10T18:58:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31213700 1 -- 172.21.15.104:0/1002293721 learned_addr learned my addr 172.21.15.104:0/1002293721 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:49.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 --2- 172.21.15.104:0/1002293721 >> v2:172.21.15.104:3300/0 conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40728/0 (socket says 172.21.15.104:40728) 2023-06-10T18:58:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f30a12700 1 --2- 172.21.15.104:0/1002293721 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c0ce350 0x7f0f2c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 -- 172.21.15.104:0/1002293721 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c0ce350 msgr2=0x7f0f2c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 --2- 172.21.15.104:0/1002293721 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c0ce350 0x7f0f2c0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:49.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 -- 172.21.15.104:0/1002293721 >> v1:172.21.15.104:6789/0 conn(0x7f0f2c06b1f0 legacy=0x7f0f2c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:49.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.953+0000 7f0f31a14700 1 -- 172.21.15.104:0/1002293721 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0cf550 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f31a14700 1 --2- 172.21.15.104:0/1002293721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0d00b0 tx=0x7f0f2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5c98b60aed1a94 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:49.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f23fff700 1 -- 172.21.15.104:0/1002293721 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400e070 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f23fff700 1 -- 172.21.15.104:0/1002293721 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2400bbd0 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f23fff700 1 -- 172.21.15.104:0/1002293721 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24012430 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 -- 172.21.15.104:0/1002293721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 msgr2=0x7f0f2c0cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 --2- 172.21.15.104:0/1002293721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0d00b0 tx=0x7f0f2400b330 comp rx=0 tx=0).stop 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 -- 172.21.15.104:0/1002293721 shutdown_connections 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 --2- 172.21.15.104:0/1002293721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c0cb540 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 --2- 172.21.15.104:0/1002293721 >> v2:172.21.15.114:3300/0 conn(0x7f0f2c0ce350 0x7f0f2c0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:49.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 -- 172.21.15.104:0/1002293721 >> 172.21.15.104:0/1002293721 conn(0x7f0f2c089fd0 msgr2=0x7f0f2c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 -- 172.21.15.104:0/1002293721 shutdown_connections 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.954+0000 7f0f33477700 1 -- 172.21.15.104:0/1002293721 wait complete. 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.956+0000 7f0f33477700 1 Processor -- start 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.956+0000 7f0f33477700 1 -- start start 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 0x7f0f2c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 0x7f0f2c154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c09df60 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1f1bc0 con 0x7f0f2c0ce350 2023-06-10T18:58:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f33477700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0c8f90 con 0x7f0f2c06b1f0 2023-06-10T18:58:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40732/0 (socket says 172.21.15.104:40732) 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 -- 172.21.15.104:0/1505442923 learned_addr learned my addr 172.21.15.104:0/1505442923 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f31a14700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 0x7f0f2c154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 -- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 msgr2=0x7f0f2c06f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f31213700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 0x7f0f2c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 0x7f0f2c06f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:49.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 -- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 msgr2=0x7f0f2c154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 0x7f0f2c154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 -- 172.21.15.104:0/1505442923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24009a70 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f31a14700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 0x7f0f2c154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f31213700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 0x7f0f2c06f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.957+0000 7f0f30a12700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800e810 tx=0x7f0f2800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.958+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28013610 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.958+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28013770 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.958+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c09e190 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.958+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28013b00 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.958+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c0c9510 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.959+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c0c96a0 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.959+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0f28013c60 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.960+0000 7f0f21ffb700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.960+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0f280892b0 con 0x7f0f2c08a3d0 2023-06-10T18:58:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.960+0000 7f0f31213700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.960+0000 7f0f31213700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005910 tx=0x7f0f1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:49.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:49.964+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0f28057ca0 con 0x7f0f2c08a3d0 2023-06-10T18:58:50.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.191+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f2c074050 con 0x7f0f2c08a3d0 2023-06-10T18:58:50.197 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.191+0000 7f0f21ffb700 1 -- 172.21.15.104:0/1505442923 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f0f28057830 con 0x7f0f2c08a3d0 2023-06-10T18:58:50.198 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:50.198 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:58:50.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 msgr2=0x7f0f1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005910 tx=0x7f0f1c009040 comp rx=0 tx=0).stop 2023-06-10T18:58:50.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 msgr2=0x7f0f2c09da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800e810 tx=0x7f0f2800ebd0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 shutdown_connections 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0f1006ace0 0x7f0f1006d190 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2c0ce350 0x7f0f2c154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f2c06b1f0 0x7f0f2c06f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 --2- 172.21.15.104:0/1505442923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f2c08a3d0 0x7f0f2c09da20 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 >> 172.21.15.104:0/1505442923 conn(0x7f0f2c089fd0 msgr2=0x7f0f2c15fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:50.201 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.193+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 shutdown_connections 2023-06-10T18:58:50.202 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.194+0000 7f0f33477700 1 -- 172.21.15.104:0/1505442923 wait complete. 2023-06-10T18:58:50.202 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:58:50.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session evict 10767 2023-06-10T18:58:50.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:58:50.740+0000 7f6f4e7fc700 -1 client.10767 I was blocklisted at osd epoch 173 2023-06-10T18:58:50.757 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2023-06-10T18:58:50.758 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:58:50.877 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.871+0000 7f9856b7e700 1 Processor -- start 2023-06-10T18:58:50.878 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.871+0000 7f9856b7e700 1 -- start start 2023-06-10T18:58:50.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f9856b7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.879 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f9856b7e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98500cad90 0x7f98500b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f9856b7e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98500ccca0 con 0x7f985015a250 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f9856b7e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98500cee50 con 0x7f98500cad90 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f9856b7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98500cf130 con 0x7f98500cfa00 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f985491a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f985511b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98500cad90 0x7f98500b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.880 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f985511b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98500cad90 0x7f98500b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:56482/0 (socket says 172.21.15.104:56482) 2023-06-10T18:58:50.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f985491a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60128/0 (socket says 172.21.15.104:60128) 2023-06-10T18:58:50.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.872+0000 7f985511b700 1 -- 172.21.15.104:0/3461294150 learned_addr learned my addr 172.21.15.104:0/3461294150 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:50.881 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985511b700 1 -- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 msgr2=0x7f98500701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:50.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985511b700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985511b700 1 -- 172.21.15.104:0/3461294150 >> v1:172.21.15.114:6789/0 conn(0x7f985015a250 legacy=0x7f98500cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f984f7fe700 1 -- 172.21.15.104:0/3461294150 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279831214 0 0) 0x7f98500ccca0 con 0x7f985015a250 2023-06-10T18:58:50.882 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985511b700 1 -- 172.21.15.104:0/3461294150 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500cc930 con 0x7f98500cad90 2023-06-10T18:58:50.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985491a700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:58:50.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f985511b700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f98500b3590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9848009e50 tx=0x7f984800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68d2d4c52fd403b6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:50.883 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f984f7fe700 1 -- 172.21.15.104:0/3461294150 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800e070 con 0x7f98500cad90 2023-06-10T18:58:50.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f984f7fe700 1 -- 172.21.15.104:0/3461294150 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984800b910 con 0x7f98500cad90 2023-06-10T18:58:50.884 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.873+0000 7f984f7fe700 1 -- 172.21.15.104:0/3461294150 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800bcc0 con 0x7f98500cad90 2023-06-10T18:58:50.887 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 -- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 msgr2=0x7f98500b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f98500b3590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9848009e50 tx=0x7f984800b330 comp rx=0 tx=0).stop 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 -- 172.21.15.104:0/3461294150 shutdown_connections 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f98500b3590 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 --2- 172.21.15.104:0/3461294150 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 -- 172.21.15.104:0/3461294150 >> 172.21.15.104:0/3461294150 conn(0x7f985006e370 msgr2=0x7f985006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:50.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 -- 172.21.15.104:0/3461294150 shutdown_connections 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.874+0000 7f9856b7e700 1 -- 172.21.15.104:0/3461294150 wait complete. 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.875+0000 7f9856b7e700 1 Processor -- start 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.875+0000 7f9856b7e700 1 -- start start 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f9850150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500b2c30 con 0x7f985015a250 2023-06-10T18:58:50.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98501511b0 con 0x7f98500cad90 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98500b1fb0 con 0x7f98500cfa00 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40754/0 (socket says 172.21.15.104:40754) 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 -- 172.21.15.104:0/811085451 learned_addr learned my addr 172.21.15.104:0/811085451 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f984ffff700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985491a700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f9850150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 -- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 msgr2=0x7f98500b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 -- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 msgr2=0x7f9850150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f9850150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 -- 172.21.15.104:0/811085451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98440094a0 con 0x7f985015a250 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985491a700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f9850150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f984ffff700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f985511b700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f984800b680 tx=0x7f9848004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:50.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480099b0 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98480092e0 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.876+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9848009b30 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.877+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984801c660 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.877+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500b2570 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.877+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9850074050 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.878+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f984801c7c0 con 0x7f985015a250 2023-06-10T18:58:50.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.878+0000 7f984d7fa700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 0x7f983406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:50.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.878+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f984808b280 con 0x7f985015a250 2023-06-10T18:58:50.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.879+0000 7f985491a700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 0x7f983406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:50.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.879+0000 7f985491a700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 0x7f983406d190 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9844009cf0 tx=0x7f9844008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:50.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:50.883+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9848056250 con 0x7f985015a250 2023-06-10T18:58:51.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.112+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f98500b3a20 con 0x7f985015a250 2023-06-10T18:58:51.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.113+0000 7f984d7fa700 1 -- 172.21.15.104:0/811085451 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v110) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9848059870 con 0x7f985015a250 2023-06-10T18:58:51.119 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:58:51.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 msgr2=0x7f983406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:51.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 0x7f983406d190 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9844009cf0 tx=0x7f9844008040 comp rx=0 tx=0).stop 2023-06-10T18:58:51.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 msgr2=0x7f98500b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f984800b680 tx=0x7f9848004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 shutdown_connections 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f983406ace0 0x7f983406d190 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98500cad90 0x7f9850150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f985015a250 0x7f98500b53c0 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 --2- 172.21.15.104:0/811085451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98500cfa00 0x7f98500b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:51.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 >> 172.21.15.104:0/811085451 conn(0x7f985006e370 msgr2=0x7f9850157800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:51.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 shutdown_connections 2023-06-10T18:58:51.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:51.115+0000 7f9856b7e700 1 -- 172.21.15.104:0/811085451 wait complete. 2023-06-10T18:58:51.133 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:58:51.133 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:58:51.133 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:58:51.134 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:58:51.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:51.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:58:51.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:58:51.135 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:58:51.135 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:58:51.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.136 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:58:51.153 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.154 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:51.155 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:51.155 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:51.155 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41554sec preferred_lft 41554sec 2023-06-10T18:58:51.156 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.157 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:58:51.158 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.159 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:58:51.218 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:58:51.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.220 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:58:51.278 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:58:51.279 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:58:51.279 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:58:51.279 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:58:51.279 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:58:51.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.280 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:51.281 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:51.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:58:51.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:58:51.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:58:51.282 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:51.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.386 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:58:51.403 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:58:51.403 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:58:51.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.404 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2023-06-10T18:58:51.460 INFO:teuthology.orchestra.run.smithi104.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-06-10T18:58:51.460 INFO:teuthology.orchestra.run.smithi104.stdout:nsid 1 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.2) 2023-06-10T18:58:51.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.462 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:51.462 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:51.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:58:51.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 2 2023-06-10T18:58:51.464 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:51.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.554 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr 2023-06-10T18:58:51.592 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.593 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.593 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.594 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.594 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.594 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.594 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.595 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 7e:1b:3b:1a:32:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:58:51.595 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.2/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:58:51.595 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.595 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::7c1b:3bff:fe1a:322e/64 scope link 2023-06-10T18:58:51.596 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-06-10T18:58:51.695 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.695 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.695 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.695 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 96:1f:bf:1c:a2:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:58:51.696 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.697 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::941f:bfff:fe1c:a2d0/64 scope link 2023-06-10T18:58:51.697 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.709 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr 2023-06-10T18:58:51.745 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.746 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.746 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.746 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.746 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 7e:1b:3b:1a:32:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.2/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.747 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::7c1b:3bff:fe1a:322e/64 scope link 2023-06-10T18:58:51.748 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.760 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr 2023-06-10T18:58:51.836 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.836 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.837 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 7e:1b:3b:1a:32:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:58:51.838 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.2/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:58:51.838 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.838 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::7c1b:3bff:fe1a:322e/64 scope link 2023-06-10T18:58:51.838 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-06-10T18:58:51.921 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:58:51.921 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:58:51.921 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:58:51.921 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.921 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 96:1f:bf:1c:a2:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::941f:bfff:fe1c:a2d0/64 scope link 2023-06-10T18:58:51.922 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:58:51.934 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.3/16 2023-06-10T18:58:51.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:51.935 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:51.935 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:51.935 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.2 2023-06-10T18:58:51.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.3/16 brd 192.168.255.255 dev veth0 2023-06-10T18:58:51.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-06-10T18:58:51.936 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-06-10T18:58:51.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-06-10T18:58:51.937 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:52.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:52.187 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:58:52.187 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:58:52.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.2 up 2023-06-10T18:58:52.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.2 master ceph-brx 2023-06-10T18:58:52.192 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:58:52.306 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:58:52.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:52.307 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:52.327 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:58:52.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:58:52.327 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:52.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:52.452 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:52.453 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:52.518 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:52.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:52.522 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:52.577 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:58:52.578 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2023-06-10T18:58:52.578 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 0) 2023-06-10T18:58:52.621 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:58:52.648 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:58:52.648 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:58:52.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:58:52.701+0000 7f7847b095c0 -1 init, newargv = 0x563979431ce0 newargc=15 2023-06-10T18:58:52.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[162045]: starting ceph client 2023-06-10T18:58:52.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[162045]: starting fuse 2023-06-10T18:58:52.716 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:58:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:58:52.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:52.718 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:58:52.775 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:58:52.775 INFO:teuthology.orchestra.run.smithi104.stdout:48 2023-06-10T18:58:52.775 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2023-06-10T18:58:52.776 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:58:52.776 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:58:52.776 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:58:52.777 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:58:52.777 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:58:52.777 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:52.777 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:58:52.778 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:58:52.778 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:58:52.778 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:58:52.779 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:52.779 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:58:52.779 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:58:52.779 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:58:52.780 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:58:52.780 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:52.780 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:58:52.781 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:58:52.781 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:58:52.781 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:58:52.782 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:58:52.782 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:58:52.782 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:58:52.782 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:58:52.783 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:58:52.922 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:58:52.923 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:58:53.007 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.162045.asok 2023-06-10T18:58:53.191 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.162045.asok 2023-06-10T18:58:53.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:53.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.162045.asok status 2023-06-10T18:58:53.349 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:58:53.350 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:58:53.351 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "162045", 2023-06-10T18:58:53.351 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:58:53.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:53.351 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:58:53.351 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:58:53.352 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 10878, 2023-06-10T18:58:53.352 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:58:53.352 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:58:53.352 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:58:53.352 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 10878 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.3:0", 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3184479321 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:58:53.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.3:0", 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3184479321 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.10878 192.168.0.3:0/3184479321", 2023-06-10T18:58:53.354 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.3:0/3184479321", 2023-06-10T18:58:53.355 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 110, 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 173, 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:58:53.356 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:58:53.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:53.361 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:58:53.362 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:53.379 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:58:53.380 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:53.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:53.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:53.446 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:53.446 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:53.504 INFO:tasks.cephfs.mount:Creating file a 2023-06-10T18:58:53.505 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.1/a 2023-06-10T18:58:53.565 INFO:tasks.cephfs.mount:Creating file b 2023-06-10T18:58:53.566 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.1/b 2023-06-10T18:58:53.625 INFO:tasks.cephfs.mount:Creating file c 2023-06-10T18:58:53.626 DEBUG:teuthology.orchestra.run.smithi104:> touch /home/ubuntu/cephtest/mnt.1/c 2023-06-10T18:58:53.684 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:53.684 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:53.745 INFO:tasks.cephfs.mount:Checking file a 2023-06-10T18:58:53.745 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.1/a 2023-06-10T18:58:53.802 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.1/a 2023-06-10T18:58:53.803 INFO:tasks.cephfs.mount:Checking file b 2023-06-10T18:58:53.803 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.1/b 2023-06-10T18:58:53.860 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.1/b 2023-06-10T18:58:53.861 INFO:tasks.cephfs.mount:Checking file c 2023-06-10T18:58:53.861 DEBUG:teuthology.orchestra.run.smithi104:> ls /home/ubuntu/cephtest/mnt.1/c 2023-06-10T18:58:53.917 INFO:teuthology.orchestra.run.smithi104.stdout:/home/ubuntu/cephtest/mnt.1/c 2023-06-10T18:58:53.918 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:53.918 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:53.975 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:54.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.128+0000 7fedcba73700 1 Processor -- start 2023-06-10T18:58:54.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.129+0000 7fedcba73700 1 -- start start 2023-06-10T18:58:54.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.129+0000 7fedcba73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc415a0b0 0x7fedc40717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.129+0000 7fedcba73700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fedc40c93e0 0x7fedc415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.129+0000 7fedcba73700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc406b2e0 con 0x7fedc4070a20 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedcba73700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc415e150 con 0x7fedc40c93e0 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedcba73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc415db40 con 0x7fedc415a0b0 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedca010700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fedc4070a20 0x7fedc415a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38604/0 (socket says 172.21.15.104:38604) 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedca010700 1 -- 172.21.15.104:0/601777687 learned_addr learned my addr 172.21.15.104:0/601777687 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:54.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fedc40c93e0 0x7fedc415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc980f700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc415a0b0 0x7fedc40717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc880d700 1 -- 172.21.15.104:0/601777687 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100308744 0 0) 0x7fedc406b2e0 con 0x7fedc4070a20 2023-06-10T18:58:54.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc880d700 1 -- 172.21.15.104:0/601777687 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb00035b0 con 0x7fedc4070a20 2023-06-10T18:58:54.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 -- 172.21.15.104:0/601777687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc415a0b0 msgr2=0x7fedc40717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc415a0b0 0x7fedc40717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 -- 172.21.15.104:0/601777687 >> v1:172.21.15.104:6789/0 conn(0x7fedc4070a20 legacy=0x7fedc415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 -- 172.21.15.104:0/601777687 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc406b530 con 0x7fedc40c93e0 2023-06-10T18:58:54.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc900e700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc415e5e0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009840 tx=0x7fedb4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415a729c2ccd456 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.142 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc880d700 1 -- 172.21.15.104:0/601777687 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4011830 con 0x7fedc40c93e0 2023-06-10T18:58:54.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc880d700 1 -- 172.21.15.104:0/601777687 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb4011990 con 0x7fedc40c93e0 2023-06-10T18:58:54.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.130+0000 7fedc880d700 1 -- 172.21.15.104:0/601777687 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4011d20 con 0x7fedc40c93e0 2023-06-10T18:58:54.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 -- 172.21.15.104:0/601777687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 msgr2=0x7fedc415e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc415e5e0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009840 tx=0x7fedb4009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:54.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 -- 172.21.15.104:0/601777687 shutdown_connections 2023-06-10T18:58:54.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc415a0b0 0x7fedc40717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 --2- 172.21.15.104:0/601777687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc415e5e0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 -- 172.21.15.104:0/601777687 >> 172.21.15.104:0/601777687 conn(0x7fedc40b1610 msgr2=0x7fedc40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:54.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 -- 172.21.15.104:0/601777687 shutdown_connections 2023-06-10T18:58:54.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.131+0000 7fedcba73700 1 -- 172.21.15.104:0/601777687 wait complete. 2023-06-10T18:58:54.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.132+0000 7fedcba73700 1 Processor -- start 2023-06-10T18:58:54.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.132+0000 7fedcba73700 1 -- start start 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 0x7fedc4148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedc415a0b0 0x7fedc41f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40c8000 con 0x7fedc40c93e0 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40b4680 con 0x7fedc415a0b0 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedcba73700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc40c73f0 con 0x7fedc4070a20 2023-06-10T18:58:54.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40780/0 (socket says 172.21.15.104:40780) 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 -- 172.21.15.104:0/1211623629 learned_addr learned my addr 172.21.15.104:0/1211623629 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc980f700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 0x7fedc4148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedca010700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedc415a0b0 0x7fedc41f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 -- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 msgr2=0x7fedc4148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 0x7fedc4148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 -- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedc415a0b0 msgr2=0x7fedc41f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedc415a0b0 0x7fedc41f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 -- 172.21.15.104:0/1211623629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40094a0 con 0x7fedc40c93e0 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc980f700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 0x7fedc4148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedc900e700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009fd0 tx=0x7fedb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40038a0 con 0x7fedc40c93e0 2023-06-10T18:58:54.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.133+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb401f070 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.134+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4003c20 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.134+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc40c76d0 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.134+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc40c7c90 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.135+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc40c7e20 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.135+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fedb401a430 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.135+0000 7fedba7fc700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 0x7feda806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.135+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fedb4021070 con 0x7fedc40c93e0 2023-06-10T18:58:54.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.136+0000 7fedc980f700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 0x7feda806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.136+0000 7fedc980f700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 0x7feda806d190 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007a50 tx=0x7fedc0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.140+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fedb4055ee0 con 0x7fedc40c93e0 2023-06-10T18:58:54.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.326+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedc4074050 con 0x7fedc40c93e0 2023-06-10T18:58:54.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.327+0000 7fedba7fc700 1 -- 172.21.15.104:0/1211623629 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+18714 (secure 0 0 0) 0x7fedb402b090 con 0x7fedc40c93e0 2023-06-10T18:58:54.334 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:54.334 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":174,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:53.743551+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:54.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 msgr2=0x7feda806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 0x7feda806d190 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fedc0007a50 tx=0x7fedc0008040 comp rx=0 tx=0).stop 2023-06-10T18:58:54.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 msgr2=0x7fedc40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009fd0 tx=0x7fedb4004750 comp rx=0 tx=0).stop 2023-06-10T18:58:54.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 shutdown_connections 2023-06-10T18:58:54.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7feda806ace0 0x7feda806d190 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedc415a0b0 0x7fedc41f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedc4070a20 0x7fedc4148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 --2- 172.21.15.104:0/1211623629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedc40c93e0 0x7fedc40b4140 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 >> 172.21.15.104:0/1211623629 conn(0x7fedc40b1610 msgr2=0x7fedc415d410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 shutdown_connections 2023-06-10T18:58:54.337 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.329+0000 7fedcba73700 1 -- 172.21.15.104:0/1211623629 wait complete. 2023-06-10T18:58:54.347 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:54.349 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:54.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.466+0000 7f6181cc2700 1 Processor -- start 2023-06-10T18:58:54.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.466+0000 7f6181cc2700 1 -- start start 2023-06-10T18:58:54.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.466+0000 7f6181cc2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f617c06af80 0x7f617c0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f6181cc2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0ce340 con 0x7f617c0ce7d0 2023-06-10T18:58:54.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f6181cc2700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0d00d0 con 0x7f617c0ccff0 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f6181cc2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c0cd490 con 0x7f617c06af80 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f617c0ce7d0 0x7f617c0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38618/0 (socket says 172.21.15.104:38618) 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617bfff700 1 -- 172.21.15.104:0/2695012067 learned_addr learned my addr 172.21.15.104:0/2695012067 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617b7fe700 1 --2- 172.21.15.104:0/2695012067 >> v2:172.21.15.114:3300/0 conn(0x7f617c06af80 0x7f617c0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056789696 0 0) 0x7f617c0ce340 con 0x7f617c0ce7d0 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61640035b0 con 0x7f617c0ce7d0 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 131459675 0 0) 0x7f61640035b0 con 0x7f617c0ce7d0 2023-06-10T18:58:54.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 >> v2:172.21.15.114:3300/0 conn(0x7f617c06af80 msgr2=0x7f617c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 --2- 172.21.15.104:0/2695012067 >> v2:172.21.15.114:3300/0 conn(0x7f617c06af80 0x7f617c0d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 >> v1:172.21.15.114:6789/0 conn(0x7f617c0ccff0 legacy=0x7f617c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0cd680 con 0x7f617c0ce7d0 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f61740019f0 con 0x7f617c0ce7d0 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 >> v1:172.21.15.104:6789/0 conn(0x7f617c0ce7d0 legacy=0x7f617c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f6164003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164004850 0x7f6164004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6164005200 0x7f616400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616400c050 con 0x7f6164003a00 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616400c1c0 con 0x7f6164004850 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616400c380 con 0x7f6164005200 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.467+0000 7f617b7fe700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f6164003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617affd700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164004850 0x7f6164004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617bfff700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6164005200 0x7f616400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 -- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6164005200 msgr2=0x7f616400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6164005200 0x7f616400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 -- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164004850 msgr2=0x7f6164004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164004850 0x7f6164004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 -- 172.21.15.104:0/2695012067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640035b0 con 0x7f6164003a00 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617affd700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164004850 0x7f6164004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617bfff700 1 -- 172.21.15.104:0/2695012067 reap_dead start 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617b7fe700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f6164003e10 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f616c009b60 tx=0x7f6164005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6201bf15b94ae97d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00e6c0 con 0x7f6164003a00 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00e820 con 0x7f6164003a00 2023-06-10T18:58:54.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f617a7fc700 1 -- 172.21.15.104:0/2695012067 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00eb90 con 0x7f6164003a00 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 -- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 msgr2=0x7f6164003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f6164003e10 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f616c009b60 tx=0x7f6164005880 comp rx=0 tx=0).stop 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 -- 172.21.15.104:0/2695012067 shutdown_connections 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 --2- 172.21.15.104:0/2695012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f6164003e10 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 -- 172.21.15.104:0/2695012067 >> 172.21.15.104:0/2695012067 conn(0x7f617c089fd0 msgr2=0x7f617c08a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.468+0000 7f6181cc2700 1 -- 172.21.15.104:0/2695012067 shutdown_connections 2023-06-10T18:58:54.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.469+0000 7f6181cc2700 1 -- 172.21.15.104:0/2695012067 wait complete. 2023-06-10T18:58:54.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.470+0000 7f6181cc2700 1 Processor -- start 2023-06-10T18:58:54.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.470+0000 7f6181cc2700 1 -- start start 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f617c15a2e0 0x7f617c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 0x7f617c0b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0b2840 con 0x7f6164003a00 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c157690 con 0x7f617c15a2e0 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f6181cc2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c15ae40 con 0x7f617c0b1e90 2023-06-10T18:58:54.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40798/0 (socket says 172.21.15.104:40798) 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 -- 172.21.15.104:0/3541166719 learned_addr learned my addr 172.21.15.104:0/3541166719 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617affd700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f617c15a2e0 0x7f617c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617bfff700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 0x7f617c0b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 -- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 msgr2=0x7f617c0b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 0x7f617c0b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 -- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f617c15a2e0 msgr2=0x7f617c157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f617c15a2e0 0x7f617c157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617b7fe700 1 -- 172.21.15.104:0/3541166719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c15a880 con 0x7f6164003a00 2023-06-10T18:58:54.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.471+0000 7f617bfff700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 0x7f617c0b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:54.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f617b7fe700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f616c0090b0 tx=0x7f616c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00e6c0 con 0x7f6164003a00 2023-06-10T18:58:54.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c003ad0 con 0x7f6164003a00 2023-06-10T18:58:54.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c003e80 con 0x7f6164003a00 2023-06-10T18:58:54.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c15ab60 con 0x7f6164003a00 2023-06-10T18:58:54.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.472+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c15c7b0 con 0x7f6164003a00 2023-06-10T18:58:54.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.473+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c15cb20 con 0x7f6164003a00 2023-06-10T18:58:54.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.474+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f616c013470 con 0x7f6164003a00 2023-06-10T18:58:54.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.474+0000 7f6163fff700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 0x7f616806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.474+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f616c08b180 con 0x7f6164003a00 2023-06-10T18:58:54.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.474+0000 7f617affd700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 0x7f616806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.475+0000 7f617affd700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 0x7f616806d190 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6170000c00 tx=0x7f6170009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.478+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f616c056310 con 0x7f6164003a00 2023-06-10T18:58:54.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.661+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f617c15c940 con 0x7f6164003a00 2023-06-10T18:58:54.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.662+0000 7f6163fff700 1 -- 172.21.15.104:0/3541166719 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7f616c059930 con 0x7f6164003a00 2023-06-10T18:58:54.669 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:54.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.663+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 msgr2=0x7f616806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.663+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 0x7f616806d190 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6170000c00 tx=0x7f6170009040 comp rx=0 tx=0).stop 2023-06-10T18:58:54.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.663+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 msgr2=0x7f617c157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.663+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f616c0090b0 tx=0x7f616c0047c0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 shutdown_connections 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f617c0b1e90 0x7f617c0b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6164003a00 0x7f617c157870 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f616806ace0 0x7f616806d190 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 --2- 172.21.15.104:0/3541166719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f617c15a2e0 0x7f617c157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 >> 172.21.15.104:0/3541166719 conn(0x7f617c089fd0 msgr2=0x7f617c06f920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:54.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 shutdown_connections 2023-06-10T18:58:54.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.664+0000 7f6181cc2700 1 -- 172.21.15.104:0/3541166719 wait complete. 2023-06-10T18:58:54.682 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:54.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.799+0000 7f693473f700 1 Processor -- start 2023-06-10T18:58:54.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.799+0000 7f693473f700 1 -- start start 2023-06-10T18:58:54.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.800+0000 7f693473f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f692c0cb5e0 0x7f692c0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.800+0000 7f693473f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0cc9f0 con 0x7f692c0cd8f0 2023-06-10T18:58:54.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.800+0000 7f693473f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0cc100 con 0x7f692c0cb210 2023-06-10T18:58:54.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.800+0000 7f693473f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692c0cb9e0 con 0x7f692c0cb5e0 2023-06-10T18:58:54.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f6931cda700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f692c0cd8f0 0x7f692c06d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38636/0 (socket says 172.21.15.104:38636) 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f6931cda700 1 -- 172.21.15.104:0/2595937025 learned_addr learned my addr 172.21.15.104:0/2595937025 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f6932cdc700 1 --2- 172.21.15.104:0/2595937025 >> v2:172.21.15.114:3300/0 conn(0x7f692c0cb5e0 0x7f692c0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518441604 0 0) 0x7f692c0cc9f0 con 0x7f692c0cd8f0 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69180035b0 con 0x7f692c0cd8f0 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048014302 0 0) 0x7f69180035b0 con 0x7f692c0cd8f0 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 >> v2:172.21.15.114:3300/0 conn(0x7f692c0cb5e0 msgr2=0x7f692c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 --2- 172.21.15.104:0/2595937025 >> v2:172.21.15.114:3300/0 conn(0x7f692c0cb5e0 0x7f692c0cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 >> v1:172.21.15.114:6789/0 conn(0x7f692c0cb210 legacy=0x7f692c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0cc3a0 con 0x7f692c0cd8f0 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f6928001670 con 0x7f692c0cd8f0 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 >> v1:172.21.15.104:6789/0 conn(0x7f692c0cd8f0 legacy=0x7f692c06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f6918003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6918004850 0x7f6918004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6918005200 0x7f691800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691800c050 con 0x7f6918003a00 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691800c1c0 con 0x7f6918004850 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691800c380 con 0x7f6918005200 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f6918003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f6931cda700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6918004850 0x7f6918004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f6932cdc700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6918005200 0x7f691800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 -- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6918005200 msgr2=0x7f691800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6918005200 0x7f691800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 -- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6918004850 msgr2=0x7f6918004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6918004850 0x7f6918004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.801+0000 7f69324db700 1 -- 172.21.15.104:0/2595937025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69180035b0 con 0x7f6918003a00 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f6932cdc700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6918005200 0x7f691800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f6932cdc700 1 -- 172.21.15.104:0/2595937025 reap_dead start 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f69324db700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f6918003e10 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f691c001820 tx=0x7f6918005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0de5a66c8b555b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0012e0 con 0x7f6918003a00 2023-06-10T18:58:54.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c0052d0 con 0x7f6918003a00 2023-06-10T18:58:54.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f69314d9700 1 -- 172.21.15.104:0/2595937025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0036d0 con 0x7f6918003a00 2023-06-10T18:58:54.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 -- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 msgr2=0x7f6918003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f6918003e10 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f691c001820 tx=0x7f6918005880 comp rx=0 tx=0).stop 2023-06-10T18:58:54.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 -- 172.21.15.104:0/2595937025 shutdown_connections 2023-06-10T18:58:54.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 --2- 172.21.15.104:0/2595937025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f6918003e10 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 -- 172.21.15.104:0/2595937025 >> 172.21.15.104:0/2595937025 conn(0x7f692c06b8a0 msgr2=0x7f692c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:54.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 -- 172.21.15.104:0/2595937025 shutdown_connections 2023-06-10T18:58:54.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.802+0000 7f693473f700 1 -- 172.21.15.104:0/2595937025 wait complete. 2023-06-10T18:58:54.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.804+0000 7f693473f700 1 Processor -- start 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.804+0000 7f693473f700 1 -- start start 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 0x7f692c14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f692c09ab40 0x7f692c0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c09a8a0 con 0x7f6918003a00 2023-06-10T18:58:54.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c0c3590 con 0x7f692c0c30b0 2023-06-10T18:58:54.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f693473f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692c0c3870 con 0x7f692c09ab40 2023-06-10T18:58:54.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40818/0 (socket says 172.21.15.104:40818) 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 -- 172.21.15.104:0/3949353844 learned_addr learned my addr 172.21.15.104:0/3949353844 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f6932cdc700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f692c09ab40 0x7f692c0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f6931cda700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 0x7f692c14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 -- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f692c09ab40 msgr2=0x7f692c0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f692c09ab40 0x7f692c0b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 -- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 msgr2=0x7f692c14fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 0x7f692c14fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 -- 172.21.15.104:0/3949353844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0c3b50 con 0x7f6918003a00 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f6931cda700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 0x7f692c14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f69324db700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f691c005040 tx=0x7f691c001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.805+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c001280 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.806+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c0050d0 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.806+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c0c3e30 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.806+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c01d420 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.806+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c09afe0 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.806+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c09b200 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.807+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f691c01d580 con 0x7f6918003a00 2023-06-10T18:58:54.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.807+0000 7f6922ffd700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 0x7f691006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:54.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.807+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f691c089bf0 con 0x7f6918003a00 2023-06-10T18:58:54.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.807+0000 7f6931cda700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 0x7f691006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:54.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.808+0000 7f6931cda700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 0x7f691006d190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f692c0c4970 tx=0x7f6928009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:54.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:54.812+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f691c054b80 con 0x7f6918003a00 2023-06-10T18:58:55.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.001+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f692c074050 con 0x7f6918003a00 2023-06-10T18:58:55.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.001+0000 7f6922ffd700 1 -- 172.21.15.104:0/3949353844 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v175) v1 ==== 92+0+10 (secure 0 0 0) 0x7f691c0210a0 con 0x7f6918003a00 2023-06-10T18:58:55.008 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:55.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 msgr2=0x7f691006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 0x7f691006d190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f692c0c4970 tx=0x7f6928009040 comp rx=0 tx=0).stop 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 msgr2=0x7f692c1edb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f691c005040 tx=0x7f691c001170 comp rx=0 tx=0).stop 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 shutdown_connections 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f692c09ab40 0x7f692c0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6918003a00 0x7f692c1edb90 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f691006ace0 0x7f691006d190 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 --2- 172.21.15.104:0/3949353844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f692c0c30b0 0x7f692c14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 >> 172.21.15.104:0/3949353844 conn(0x7f692c06b8a0 msgr2=0x7f692c0b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 shutdown_connections 2023-06-10T18:58:55.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.003+0000 7f693473f700 1 -- 172.21.15.104:0/3949353844 wait complete. 2023-06-10T18:58:55.022 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:55.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.139+0000 7f89937ec700 1 Processor -- start 2023-06-10T18:58:55.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.139+0000 7f89937ec700 1 -- start start 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c156760 0x7f898c0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f898c15d1a0 0x7f898c0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f898c0b21c0 0x7f898c0b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c0b30a0 con 0x7f898c15d1a0 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c15d740 con 0x7f898c0b21c0 2023-06-10T18:58:55.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f89937ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0b2b40 con 0x7f898c156760 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f898c15d1a0 0x7f898c0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f898c15d1a0 0x7f898c0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40830/0 (socket says 172.21.15.104:40830) 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 -- 172.21.15.104:0/3833924495 learned_addr learned my addr 172.21.15.104:0/3833924495 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8991d89700 1 --2- 172.21.15.104:0/3833924495 >> v2:172.21.15.114:3300/0 conn(0x7f898c0b21c0 0x7f898c0b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8991588700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c156760 0x7f898c0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 -- 172.21.15.104:0/3833924495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c156760 msgr2=0x7f898c0ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c156760 0x7f898c0ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 -- 172.21.15.104:0/3833924495 >> v2:172.21.15.114:3300/0 conn(0x7f898c0b21c0 msgr2=0x7f898c0b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 --2- 172.21.15.104:0/3833924495 >> v2:172.21.15.114:3300/0 conn(0x7f898c0b21c0 0x7f898c0b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 -- 172.21.15.104:0/3833924495 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0ce850 con 0x7f898c15d1a0 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.140+0000 7f8990d87700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c15d1a0 0x7f898c0cad90 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f898c06f040 tx=0x7f8988009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc56294ec80ed056 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f8983fff700 1 -- 172.21.15.104:0/3833924495 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898800b870 con 0x7f898c15d1a0 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f8983fff700 1 -- 172.21.15.104:0/3833924495 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898800b9d0 con 0x7f898c15d1a0 2023-06-10T18:58:55.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f8983fff700 1 -- 172.21.15.104:0/3833924495 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898800bd60 con 0x7f898c15d1a0 2023-06-10T18:58:55.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 -- 172.21.15.104:0/3833924495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c15d1a0 msgr2=0x7f898c0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c15d1a0 0x7f898c0cad90 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f898c06f040 tx=0x7f8988009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 -- 172.21.15.104:0/3833924495 shutdown_connections 2023-06-10T18:58:55.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 --2- 172.21.15.104:0/3833924495 >> v2:172.21.15.114:3300/0 conn(0x7f898c0b21c0 0x7f898c0b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c156760 0x7f898c0ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 --2- 172.21.15.104:0/3833924495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c15d1a0 0x7f898c0cad90 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 -- 172.21.15.104:0/3833924495 >> 172.21.15.104:0/3833924495 conn(0x7f898c06e370 msgr2=0x7f898c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 -- 172.21.15.104:0/3833924495 shutdown_connections 2023-06-10T18:58:55.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.141+0000 7f89937ec700 1 -- 172.21.15.104:0/3833924495 wait complete. 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.142+0000 7f89937ec700 1 Processor -- start 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 -- start start 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 0x7f898c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c15d1a0 0x7f898c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c14b230 con 0x7f898c156760 2023-06-10T18:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c145340 con 0x7f898c0b21c0 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f89937ec700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0c6260 con 0x7f898c15d1a0 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f8990d87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f8990d87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40834/0 (socket says 172.21.15.104:40834) 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.143+0000 7f8990d87700 1 -- 172.21.15.104:0/485803554 learned_addr learned my addr 172.21.15.104:0/485803554 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8991588700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 0x7f898c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8991d89700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c15d1a0 0x7f898c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 -- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c15d1a0 msgr2=0x7f898c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c15d1a0 0x7f898c0cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 -- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 msgr2=0x7f898c14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 0x7f898c14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 -- 172.21.15.104:0/485803554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988009550 con 0x7f898c156760 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8991588700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 0x7f898c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8990d87700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f8988000c00 tx=0x7f8988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988004540 con 0x7f898c156760 2023-06-10T18:58:55.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89880036a0 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c145590 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988003a50 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.144+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c145b50 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.145+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c074050 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.146+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8988021070 con 0x7f898c156760 2023-06-10T18:58:55.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.146+0000 7f8981ffb700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.146+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f898808ac60 con 0x7f898c156760 2023-06-10T18:58:55.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.146+0000 7f8991588700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.147+0000 7f8991588700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f897c009a00 tx=0x7f897c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.150+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8988055bf0 con 0x7f898c156760 2023-06-10T18:58:55.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.338+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f898c070340 con 0x7f898c156760 2023-06-10T18:58:55.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.339+0000 7f8981ffb700 1 -- 172.21.15.104:0/485803554 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v175) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8988059210 con 0x7f898c156760 2023-06-10T18:58:55.345 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:55.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 msgr2=0x7f897006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f897c009a00 tx=0x7f897c008040 comp rx=0 tx=0).stop 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 msgr2=0x7f898c0cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f8988000c00 tx=0x7f8988004750 comp rx=0 tx=0).stop 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 shutdown_connections 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f897006ace0 0x7f897006d190 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f898c0b21c0 0x7f898c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f898c15d1a0 0x7f898c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 --2- 172.21.15.104:0/485803554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f898c156760 0x7f898c0cb280 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 >> 172.21.15.104:0/485803554 conn(0x7f898c06e370 msgr2=0x7f898c15bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 shutdown_connections 2023-06-10T18:58:55.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.341+0000 7f89937ec700 1 -- 172.21.15.104:0/485803554 wait complete. 2023-06-10T18:58:55.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:55.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.481+0000 7f9930e87700 1 Processor -- start 2023-06-10T18:58:55.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.481+0000 7f9930e87700 1 -- start start 2023-06-10T18:58:55.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9930e87700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 0x7f992c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9930e87700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f992c0cf200 0x7f992c069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9930e87700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c15c8d0 con 0x7f992c06a3b0 2023-06-10T18:58:55.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9930e87700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c0aef20 con 0x7f992c0cf200 2023-06-10T18:58:55.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9930e87700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c06abd0 con 0x7f992c15c500 2023-06-10T18:58:55.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f992c0cf200 0x7f992c069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f992c0cf200 0x7f992c069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40844/0 (socket says 172.21.15.104:40844) 2023-06-10T18:58:55.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 -- 172.21.15.104:0/138108753 learned_addr learned my addr 172.21.15.104:0/138108753 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:55.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 -- 172.21.15.104:0/138108753 >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 msgr2=0x7f992c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f992a59c700 1 --2- 172.21.15.104:0/138108753 >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 0x7f992c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 --2- 172.21.15.104:0/138108753 >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 0x7f992c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 -- 172.21.15.104:0/138108753 >> v1:172.21.15.114:6789/0 conn(0x7f992c06a3b0 legacy=0x7f992c06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 -- 172.21.15.104:0/138108753 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c06a8f0 con 0x7f992c0cf200 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f992a59c700 1 --2- 172.21.15.104:0/138108753 >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 0x7f992c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.482+0000 7f9929d9b700 1 --2- 172.21.15.104:0/138108753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c0cf200 0x7f992c069e70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f992c0b2170 tx=0x7f9920009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3781245320ebb539 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f992959a700 1 -- 172.21.15.104:0/138108753 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920004230 con 0x7f992c0cf200 2023-06-10T18:58:55.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f992959a700 1 -- 172.21.15.104:0/138108753 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9920004390 con 0x7f992c0cf200 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f992959a700 1 -- 172.21.15.104:0/138108753 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99200036a0 con 0x7f992c0cf200 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 -- 172.21.15.104:0/138108753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c0cf200 msgr2=0x7f992c069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 --2- 172.21.15.104:0/138108753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c0cf200 0x7f992c069e70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f992c0b2170 tx=0x7f9920009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 -- 172.21.15.104:0/138108753 shutdown_connections 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 --2- 172.21.15.104:0/138108753 >> v2:172.21.15.114:3300/0 conn(0x7f992c15c500 0x7f992c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 --2- 172.21.15.104:0/138108753 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c0cf200 0x7f992c069e70 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 -- 172.21.15.104:0/138108753 >> 172.21.15.104:0/138108753 conn(0x7f992c06b8a0 msgr2=0x7f992c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 -- 172.21.15.104:0/138108753 shutdown_connections 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.483+0000 7f9930e87700 1 -- 172.21.15.104:0/138108753 wait complete. 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 Processor -- start 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 -- start start 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f992c06a3b0 0x7f992c1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 0x7f992c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c146cd0 con 0x7f992c15c500 2023-06-10T18:58:55.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c1f1900 con 0x7f992c06a3b0 2023-06-10T18:58:55.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f9930e87700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992c15f1a0 con 0x7f992c0cf200 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f992ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f992ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40850/0 (socket says 172.21.15.104:40850) 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.485+0000 7f992ad9d700 1 -- 172.21.15.104:0/17333123 learned_addr learned my addr 172.21.15.104:0/17333123 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992a59c700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f992c06a3b0 0x7f992c1465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f9929d9b700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 0x7f992c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 -- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 msgr2=0x7f992c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 0x7f992c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 -- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f992c06a3b0 msgr2=0x7f992c1465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f992c06a3b0 0x7f992c1465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 -- 172.21.15.104:0/17333123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9920009550 con 0x7f992c15c500 2023-06-10T18:58:55.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f9929d9b700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 0x7f992c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:55.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f992ad9d700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f992400b970 tx=0x7f992400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400c930 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992400cad0 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c15f480 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400ce80 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.486+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c15fa40 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.487+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c146e40 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.487+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9924014440 con 0x7f992c15c500 2023-06-10T18:58:55.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.488+0000 7f991affd700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 0x7f990c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.488+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9924089200 con 0x7f992c15c500 2023-06-10T18:58:55.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.488+0000 7f992a59c700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 0x7f990c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.489+0000 7f992a59c700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 0x7f990c06d190 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f991c00abe0 tx=0x7f991c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.507 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.492+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f992c146e40 con 0x7f992c15c500 2023-06-10T18:58:55.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.678+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f992c146e40 con 0x7f992c15c500 2023-06-10T18:58:55.685 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.678+0000 7f991affd700 1 -- 172.21.15.104:0/17333123 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v175) v1 ==== 100+0+11 (secure 0 0 0) 0x7f992c146e40 con 0x7f992c15c500 2023-06-10T18:58:55.685 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:55.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 msgr2=0x7f990c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 0x7f990c06d190 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f991c00abe0 tx=0x7f991c008040 comp rx=0 tx=0).stop 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 msgr2=0x7f992c0cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f992400b970 tx=0x7f992400bc80 comp rx=0 tx=0).stop 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 shutdown_connections 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f990c06ace0 0x7f990c06d190 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f992c06a3b0 0x7f992c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f992c15c500 0x7f992c0cbcf0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 --2- 172.21.15.104:0/17333123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f992c0cf200 0x7f992c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 >> 172.21.15.104:0/17333123 conn(0x7f992c06b8a0 msgr2=0x7f992c15ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 shutdown_connections 2023-06-10T18:58:55.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.681+0000 7f9930e87700 1 -- 172.21.15.104:0/17333123 wait complete. 2023-06-10T18:58:55.699 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:55.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.814+0000 7fd9fbfcc700 1 Processor -- start 2023-06-10T18:58:55.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.814+0000 7fd9fbfcc700 1 -- start start 2023-06-10T18:58:55.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fbfcc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9f40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fbfcc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40cdae0 con 0x7fd9f40cd680 2023-06-10T18:58:55.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fbfcc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40c9e60 con 0x7fd9f40b1080 2023-06-10T18:58:55.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fbfcc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c9bc0 con 0x7fd9f40aeaa0 2023-06-10T18:58:55.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fa569700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd9f40cd680 0x7fd9f406dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38686/0 (socket says 172.21.15.104:38686) 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.815+0000 7fd9fa569700 1 -- 172.21.15.104:0/781947806 learned_addr learned my addr 172.21.15.104:0/781947806 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9f40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574116131 0 0) 0x7fd9f40cdae0 con 0x7fd9f40cd680 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e00035b0 con 0x7fd9f40cd680 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583262379 0 0) 0x7fd9e00035b0 con 0x7fd9f40cd680 2023-06-10T18:58:55.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 msgr2=0x7fd9f40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9f40cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 >> v1:172.21.15.114:6789/0 conn(0x7fd9f40b1080 legacy=0x7fd9f40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40aee70 con 0x7fd9f40cd680 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd9ec0019f0 con 0x7fd9f40cd680 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 >> v1:172.21.15.104:6789/0 conn(0x7fd9f40cd680 legacy=0x7fd9f406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9e0003a30 0x7fd9e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e0004d90 0x7fd9e0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c9bc0 con 0x7fd9e0003a30 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e000ba10 con 0x7fd9e0004d90 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e000bba0 con 0x7fd9f40aeaa0 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9e0003a30 0x7fd9e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9567700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9fa569700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e0004d90 0x7fd9e0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 -- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 msgr2=0x7fd9e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 -- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e0004d90 msgr2=0x7fd9e0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e0004d90 0x7fd9e0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9d68700 1 -- 172.21.15.104:0/781947806 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e00035b0 con 0x7fd9e0003a30 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9fa569700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e0004d90 0x7fd9e0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:55.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.816+0000 7fd9f9567700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9f40aeaa0 0x7fd9e0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9e0003a30 0x7fd9e0003e00 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009b40 tx=0x7fd9e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfeea91d3b9ea31 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4005a20 con 0x7fd9e0003a30 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4005b80 con 0x7fd9e0003a30 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9f8d66700 1 -- 172.21.15.104:0/781947806 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e400e6c0 con 0x7fd9e0003a30 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/781947806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9e0003a30 msgr2=0x7fd9e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/781947806 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9e0003a30 0x7fd9e0003e00 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009b40 tx=0x7fd9e0005880 comp rx=0 tx=0).stop 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fa569700 1 -- 172.21.15.104:0/781947806 reap_dead start 2023-06-10T18:58:55.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/781947806 shutdown_connections 2023-06-10T18:58:55.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/781947806 >> 172.21.15.104:0/781947806 conn(0x7fd9f406b8a0 msgr2=0x7fd9f40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:55.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/781947806 shutdown_connections 2023-06-10T18:58:55.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.817+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/781947806 wait complete. 2023-06-10T18:58:55.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 Processor -- start 2023-06-10T18:58:55.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 -- start start 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9e0003a30 0x7fd9f40c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f41e5f90 0x7fd9f4148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41e6400 con 0x7fd9f40c6c70 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c8f00 con 0x7fd9f41e5f90 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9fbfcc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f40c70e0 con 0x7fd9e0003a30 2023-06-10T18:58:55.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9f9567700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9f9567700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40868/0 (socket says 172.21.15.104:40868) 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.819+0000 7fd9f9567700 1 -- 172.21.15.104:0/245463247 learned_addr learned my addr 172.21.15.104:0/245463247 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9fa569700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f41e5f90 0x7fd9f4148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9e0003a30 0x7fd9f40c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 -- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9e0003a30 msgr2=0x7fd9f40c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9e0003a30 0x7fd9f40c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 -- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f41e5f90 msgr2=0x7fd9f4148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:55.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f41e5f90 0x7fd9f4148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:55.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 -- 172.21.15.104:0/245463247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f41495c0 con 0x7fd9f40c6c70 2023-06-10T18:58:55.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9f9567700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000e800 tx=0x7fd9f000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0017070 con 0x7fd9f40c6c70 2023-06-10T18:58:55.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f00052e0 con 0x7fd9f40c6c70 2023-06-10T18:58:55.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f40c88b0 con 0x7fd9f40c6c70 2023-06-10T18:58:55.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0005470 con 0x7fd9f40c6c70 2023-06-10T18:58:55.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.820+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f40c8510 con 0x7fd9f40c6c70 2023-06-10T18:58:55.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.821+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f40c86d0 con 0x7fd9f40c6c70 2023-06-10T18:58:55.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.821+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd9f0006860 con 0x7fd9f40c6c70 2023-06-10T18:58:55.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.822+0000 7fd9ea7fc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 0x7fd9d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:55.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.822+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fd9f0088db0 con 0x7fd9f40c6c70 2023-06-10T18:58:55.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.822+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 0x7fd9d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:55.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.822+0000 7fd9f9d68700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 0x7fd9d806d190 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4000c00 tx=0x7fd9e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:55.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:55.826+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd9f0053cb0 con 0x7fd9f40c6c70 2023-06-10T18:58:56.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.029+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd9f4074050 con 0x7fd9f40c6c70 2023-06-10T18:58:56.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.029+0000 7fd9ea7fc700 1 -- 172.21.15.104:0/245463247 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v175) v1 ==== 80+0+56 (secure 0 0 0) 0x7fd9f00572d0 con 0x7fd9f40c6c70 2023-06-10T18:58:56.036 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/827471637 2023-06-10T19:58:50.352178+0000 2023-06-10T18:58:56.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 msgr2=0x7fd9d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 0x7fd9d806d190 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4000c00 tx=0x7fd9e4008030 comp rx=0 tx=0).stop 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 msgr2=0x7fd9f41e5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000e800 tx=0x7fd9f000ebc0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 shutdown_connections 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd9d806ace0 0x7fd9d806d190 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f41e5f90 0x7fd9f4148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9e0003a30 0x7fd9f40c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 --2- 172.21.15.104:0/245463247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9f40c6c70 0x7fd9f41e5a50 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 >> 172.21.15.104:0/245463247 conn(0x7fd9f406b8a0 msgr2=0x7fd9f40ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 shutdown_connections 2023-06-10T18:58:56.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.031+0000 7fd9fbfcc700 1 -- 172.21.15.104:0/245463247 wait complete. 2023-06-10T18:58:56.040 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:58:56.049 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:58:56.050 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:58:56.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:58:56.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 Processor -- start 2023-06-10T18:58:56.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 -- start start 2023-06-10T18:58:56.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec0ce350 0x7f7fec0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fec08a3d0 0x7f7fec0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec0cadd0 con 0x7f7fec06b1f0 2023-06-10T18:58:56.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec0cfeb0 con 0x7f7fec08a3d0 2023-06-10T18:58:56.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.220+0000 7f7ff1eaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0cb0b0 con 0x7f7fec0ce350 2023-06-10T18:58:56.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fec08a3d0 0x7f7fec0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7fec08a3d0 0x7f7fec0cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40878/0 (socket says 172.21.15.104:40878) 2023-06-10T18:58:56.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 -- 172.21.15.104:0/2720396668 learned_addr learned my addr 172.21.15.104:0/2720396668 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:56.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7feaffd700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec0ce350 0x7f7fec0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 -- 172.21.15.104:0/2720396668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec0ce350 msgr2=0x7f7fec0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:56.240 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec0ce350 0x7f7fec0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 -- 172.21.15.104:0/2720396668 >> v1:172.21.15.114:6789/0 conn(0x7f7fec06b1f0 legacy=0x7f7fec0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 -- 172.21.15.104:0/2720396668 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0cf550 con 0x7f7fec08a3d0 2023-06-10T18:58:56.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7febfff700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec08a3d0 0x7f7fec0cb540 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009e50 tx=0x7f7fe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97eb9a4ffa13f280 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7fea7fc700 1 -- 172.21.15.104:0/2720396668 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400e070 con 0x7f7fec08a3d0 2023-06-10T18:58:56.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7fea7fc700 1 -- 172.21.15.104:0/2720396668 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe400b9c0 con 0x7f7fec08a3d0 2023-06-10T18:58:56.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.221+0000 7f7fea7fc700 1 -- 172.21.15.104:0/2720396668 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400bd70 con 0x7f7fec08a3d0 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/2720396668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec08a3d0 msgr2=0x7f7fec0cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec08a3d0 0x7f7fec0cb540 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009e50 tx=0x7f7fe400b330 comp rx=0 tx=0).stop 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/2720396668 shutdown_connections 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec08a3d0 0x7f7fec0cb540 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/2720396668 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec0ce350 0x7f7fec0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/2720396668 >> 172.21.15.104:0/2720396668 conn(0x7f7fec089fd0 msgr2=0x7f7fec0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/2720396668 shutdown_connections 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.222+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/2720396668 wait complete. 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.223+0000 7f7ff1eaf700 1 Processor -- start 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.223+0000 7f7ff1eaf700 1 -- start start 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fec06b1f0 0x7f7fec0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 0x7f7fec0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c8eb0 con 0x7f7fec0ce350 2023-06-10T18:58:56.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c8230 con 0x7f7fec06b1f0 2023-06-10T18:58:56.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c8510 con 0x7f7fec08a3d0 2023-06-10T18:58:56.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40884/0 (socket says 172.21.15.104:40884) 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 -- 172.21.15.104:0/1193182449 learned_addr learned my addr 172.21.15.104:0/1193182449 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7feb7fe700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fec06b1f0 0x7f7fec0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7feaffd700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 0x7f7fec0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 -- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 msgr2=0x7f7fec0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 0x7f7fec0c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 -- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fec06b1f0 msgr2=0x7f7fec0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fec06b1f0 0x7f7fec0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 -- 172.21.15.104:0/1193182449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4009b30 con 0x7f7fec0ce350 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7feaffd700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 0x7f7fec0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7febfff700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f7fec073ac0 tx=0x7f7fe4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4004970 con 0x7f7fec0ce350 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe40099b0 con 0x7f7fec0ce350 2023-06-10T18:58:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec0c87f0 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.224+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe401c680 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.225+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec0cd840 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.225+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec074050 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.226+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7fe401c7e0 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.226+0000 7f7fd3fff700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 0x7f7fd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.226+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f7fe408b200 con 0x7f7fec0ce350 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.227+0000 7f7feb7fe700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 0x7f7fd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.227+0000 7f7feb7fe700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 0x7f7fd806d190 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc001800 tx=0x7f7fdc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.231+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7fe4056190 con 0x7f7fec0ce350 2023-06-10T18:58:56.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.420+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fec071920 con 0x7f7fec0ce350 2023-06-10T18:58:56.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.421+0000 7f7fd3fff700 1 -- 172.21.15.104:0/1193182449 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+18714 (secure 0 0 0) 0x7f7fe40597b0 con 0x7f7fec0ce350 2023-06-10T18:58:56.428 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:58:56.428 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":175,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:54.747698+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:58:56.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 msgr2=0x7f7fd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 0x7f7fd806d190 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc001800 tx=0x7f7fdc009040 comp rx=0 tx=0).stop 2023-06-10T18:58:56.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 msgr2=0x7f7fec0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f7fec073ac0 tx=0x7f7fe4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 shutdown_connections 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7fd806ace0 0x7f7fd806d190 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fec06b1f0 0x7f7fec0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fec0ce350 0x7f7fec0c7850 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 --2- 172.21.15.104:0/1193182449 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7fec08a3d0 0x7f7fec0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 >> 172.21.15.104:0/1193182449 conn(0x7f7fec089fd0 msgr2=0x7f7fec0ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 shutdown_connections 2023-06-10T18:58:56.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.424+0000 7f7ff1eaf700 1 -- 172.21.15.104:0/1193182449 wait complete. 2023-06-10T18:58:56.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:58:56.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:58:56.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.560+0000 7f49d42b8700 1 Processor -- start 2023-06-10T18:58:56.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.560+0000 7f49d42b8700 1 -- start start 2023-06-10T18:58:56.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d42b8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49cc15c580 0x7f49cc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d42b8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc06e250 con 0x7f49cc06e6e0 2023-06-10T18:58:56.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d42b8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc15c950 con 0x7f49cc06cde0 2023-06-10T18:58:56.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d42b8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc0b1e80 con 0x7f49cc15c580 2023-06-10T18:58:56.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1853700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f49cc06e6e0 0x7f49cc0b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38722/0 (socket says 172.21.15.104:38722) 2023-06-10T18:58:56.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1853700 1 -- 172.21.15.104:0/1544401534 learned_addr learned my addr 172.21.15.104:0/1544401534 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:56.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d2054700 1 --2- 172.21.15.104:0/1544401534 >> v2:172.21.15.114:3300/0 conn(0x7f49cc15c580 0x7f49cc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099148275 0 0) 0x7f49cc06e250 con 0x7f49cc06e6e0 2023-06-10T18:58:56.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b80035b0 con 0x7f49cc06e6e0 2023-06-10T18:58:56.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251822635 0 0) 0x7f49b80035b0 con 0x7f49cc06e6e0 2023-06-10T18:58:56.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 >> v2:172.21.15.114:3300/0 conn(0x7f49cc15c580 msgr2=0x7f49cc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:56.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 --2- 172.21.15.104:0/1544401534 >> v2:172.21.15.114:3300/0 conn(0x7f49cc15c580 0x7f49cc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 >> v1:172.21.15.114:6789/0 conn(0x7f49cc06cde0 legacy=0x7f49cc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc06d920 con 0x7f49cc06e6e0 2023-06-10T18:58:56.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f49bc001670 con 0x7f49cc06e6e0 2023-06-10T18:58:56.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 >> v1:172.21.15.104:6789/0 conn(0x7f49cc06e6e0 legacy=0x7f49cc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49b8003a00 0x7f49b8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8004850 0x7f49b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 0x7f49b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0b1e80 con 0x7f49b8005200 2023-06-10T18:58:56.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.561+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800c080 con 0x7f49b8004850 2023-06-10T18:58:56.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800c240 con 0x7f49b8003a00 2023-06-10T18:58:56.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d1853700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8004850 0x7f49b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 0x7f49b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2054700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49b8003a00 0x7f49b8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 -- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49b8003a00 msgr2=0x7f49b8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:56.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49b8003a00 0x7f49b8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 -- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8004850 msgr2=0x7f49b8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:56.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8004850 0x7f49b8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 -- 172.21.15.104:0/1544401534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80035b0 con 0x7f49b8005200 2023-06-10T18:58:56.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 -- 172.21.15.104:0/1544401534 reap_dead start 2023-06-10T18:58:56.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d2855700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 0x7f49b800ba10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c190 tx=0x7f49b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8af1fb39ffbf42 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4001db0 con 0x7f49b8005200 2023-06-10T18:58:56.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c400ccf0 con 0x7f49b8005200 2023-06-10T18:58:56.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.562+0000 7f49d1052700 1 -- 172.21.15.104:0/1544401534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4003c40 con 0x7f49b8005200 2023-06-10T18:58:56.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 -- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 msgr2=0x7f49b800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 0x7f49b800ba10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c190 tx=0x7f49b8005880 comp rx=0 tx=0).stop 2023-06-10T18:58:56.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 -- 172.21.15.104:0/1544401534 shutdown_connections 2023-06-10T18:58:56.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 --2- 172.21.15.104:0/1544401534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49b8005200 0x7f49b800ba10 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 -- 172.21.15.104:0/1544401534 >> 172.21.15.104:0/1544401534 conn(0x7f49cc06b8a0 msgr2=0x7f49cc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 -- 172.21.15.104:0/1544401534 shutdown_connections 2023-06-10T18:58:56.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.563+0000 7f49d42b8700 1 -- 172.21.15.104:0/1544401534 wait complete. 2023-06-10T18:58:56.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.564+0000 7f49d42b8700 1 Processor -- start 2023-06-10T18:58:56.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.564+0000 7f49d42b8700 1 -- start start 2023-06-10T18:58:56.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 0x7f49cc0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49cc14fc20 0x7f49cc0afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0c67d0 con 0x7f49cc158f90 2023-06-10T18:58:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0c8b10 con 0x7f49b8005200 2023-06-10T18:58:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d42b8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0c8830 con 0x7f49cc14fc20 2023-06-10T18:58:56.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d1853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d1853700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40906/0 (socket says 172.21.15.104:40906) 2023-06-10T18:58:56.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.565+0000 7f49d1853700 1 -- 172.21.15.104:0/2401099437 learned_addr learned my addr 172.21.15.104:0/2401099437 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:56.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d2054700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 0x7f49cc0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d2855700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49cc14fc20 0x7f49cc0afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 -- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49cc14fc20 msgr2=0x7f49cc0afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49cc14fc20 0x7f49cc0afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 -- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 msgr2=0x7f49cc0c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 0x7f49cc0c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 -- 172.21.15.104:0/2401099437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0c6a70 con 0x7f49cc158f90 2023-06-10T18:58:56.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d2054700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 0x7f49cc0c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:56.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d1853700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0072c0 tx=0x7f49bc007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0077a0 con 0x7f49cc158f90 2023-06-10T18:58:56.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc00f8d0 con 0x7f49cc158f90 2023-06-10T18:58:56.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc0c6d50 con 0x7f49cc158f90 2023-06-10T18:58:56.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc00fc80 con 0x7f49cc158f90 2023-06-10T18:58:56.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.566+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc0c85e0 con 0x7f49cc158f90 2023-06-10T18:58:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.567+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc0c6630 con 0x7f49cc158f90 2023-06-10T18:58:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.567+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f49bc01d460 con 0x7f49cc158f90 2023-06-10T18:58:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.568+0000 7f49c2ffd700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 0x7f49b006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.568+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f49bc089680 con 0x7f49cc158f90 2023-06-10T18:58:56.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.568+0000 7f49d2054700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 0x7f49b006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.568+0000 7f49d2054700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 0x7f49b006d1c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009bc0 tx=0x7f49c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.572+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f49cc0c6630 con 0x7f49cc158f90 2023-06-10T18:58:56.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.758+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49cc0c6630 con 0x7f49cc158f90 2023-06-10T18:58:56.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.759+0000 7f49c2ffd700 1 -- 172.21.15.104:0/2401099437 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49cc0c6630 con 0x7f49cc158f90 2023-06-10T18:58:56.767 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:58:56.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 msgr2=0x7f49b006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 0x7f49b006d1c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009bc0 tx=0x7f49c8008040 comp rx=0 tx=0).stop 2023-06-10T18:58:56.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 msgr2=0x7f49cc0af8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0072c0 tx=0x7f49bc007680 comp rx=0 tx=0).stop 2023-06-10T18:58:56.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 shutdown_connections 2023-06-10T18:58:56.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f49b006ad10 0x7f49b006d1c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49b8005200 0x7f49cc0c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49cc14fc20 0x7f49cc0afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 --2- 172.21.15.104:0/2401099437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49cc158f90 0x7f49cc0af8e0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 >> 172.21.15.104:0/2401099437 conn(0x7f49cc06b8a0 msgr2=0x7f49cc0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 shutdown_connections 2023-06-10T18:58:56.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.761+0000 7f49d42b8700 1 -- 172.21.15.104:0/2401099437 wait complete. 2023-06-10T18:58:56.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:58:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.896+0000 7f33cffb9700 1 Processor -- start 2023-06-10T18:58:56.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 -- start start 2023-06-10T18:58:56.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c815c6f0 0x7f33c80b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f33c80aec00 0x7f33c80b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c80ae7e0 con 0x7f33c806ea70 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c806df90 con 0x7f33c80aec00 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.897+0000 7f33cffb9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c806e270 con 0x7f33c815c6f0 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f33c80aec00 0x7f33c80b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f33c80aec00 0x7f33c80b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40916/0 (socket says 172.21.15.104:40916) 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 -- 172.21.15.104:0/693241354 learned_addr learned my addr 172.21.15.104:0/693241354 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33cdd55700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c815c6f0 0x7f33c80b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 -- 172.21.15.104:0/693241354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c815c6f0 msgr2=0x7f33c80b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c815c6f0 0x7f33c80b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 -- 172.21.15.104:0/693241354 >> v1:172.21.15.114:6789/0 conn(0x7f33c806ea70 legacy=0x7f33c80ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 -- 172.21.15.104:0/693241354 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c806e550 con 0x7f33c80aec00 2023-06-10T18:58:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ce556700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c80aec00 0x7f33c80b15b0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009e50 tx=0x7f33c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4730849267e0651 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ccd53700 1 -- 172.21.15.104:0/693241354 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c000e070 con 0x7f33c80aec00 2023-06-10T18:58:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ccd53700 1 -- 172.21.15.104:0/693241354 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c000b910 con 0x7f33c80aec00 2023-06-10T18:58:56.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.898+0000 7f33ccd53700 1 -- 172.21.15.104:0/693241354 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c000bcc0 con 0x7f33c80aec00 2023-06-10T18:58:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 -- 172.21.15.104:0/693241354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c80aec00 msgr2=0x7f33c80b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c80aec00 0x7f33c80b15b0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009e50 tx=0x7f33c000b330 comp rx=0 tx=0).stop 2023-06-10T18:58:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 -- 172.21.15.104:0/693241354 shutdown_connections 2023-06-10T18:58:56.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c80aec00 0x7f33c80b15b0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 --2- 172.21.15.104:0/693241354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c815c6f0 0x7f33c80b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 -- 172.21.15.104:0/693241354 >> 172.21.15.104:0/693241354 conn(0x7f33c806b8a0 msgr2=0x7f33c80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 -- 172.21.15.104:0/693241354 shutdown_connections 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.899+0000 7f33cffb9700 1 -- 172.21.15.104:0/693241354 wait complete. 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.900+0000 7f33cffb9700 1 Processor -- start 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.900+0000 7f33cffb9700 1 -- start start 2023-06-10T18:58:56.911 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 0x7f33c80b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 0x7f33c80cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b2c10 con 0x7f33c815c6f0 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80c9480 con 0x7f33c80aec00 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cffb9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80b3570 con 0x7f33c806ea70 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40922/0 (socket says 172.21.15.104:40922) 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 -- 172.21.15.104:0/197649810 learned_addr learned my addr 172.21.15.104:0/197649810 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cd554700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 0x7f33c80cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cdd55700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 0x7f33c80b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 -- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 msgr2=0x7f33c80b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 0x7f33c80b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 -- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 msgr2=0x7f33c80cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 0x7f33c80cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 -- 172.21.15.104:0/197649810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009b30 con 0x7f33c815c6f0 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cd554700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 0x7f33c80cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33cdd55700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 0x7f33c80b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33ce556700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f33c0012000 tx=0x7f33c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.901+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0004920 con 0x7f33c815c6f0 2023-06-10T18:58:56.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.902+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c00099b0 con 0x7f33c815c6f0 2023-06-10T18:58:56.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.902+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c80c91a0 con 0x7f33c815c6f0 2023-06-10T18:58:56.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.902+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c001c680 con 0x7f33c815c6f0 2023-06-10T18:58:56.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.902+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c80b2830 con 0x7f33c815c6f0 2023-06-10T18:58:56.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.902+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c80d03f0 con 0x7f33c815c6f0 2023-06-10T18:58:56.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.903+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f33c001c7e0 con 0x7f33c815c6f0 2023-06-10T18:58:56.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.903+0000 7f33be7fc700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 0x7f33ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:56.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.903+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f33c008b2a0 con 0x7f33c815c6f0 2023-06-10T18:58:56.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.904+0000 7f33cdd55700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 0x7f33ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:56.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.904+0000 7f33cdd55700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 0x7f33ac06d190 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f33c81f2570 tx=0x7f33b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:56.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:56.908+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f33c0056230 con 0x7f33c815c6f0 2023-06-10T18:58:57.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.094+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f33c8066f10 con 0x7f33c815c6f0 2023-06-10T18:58:57.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.094+0000 7f33be7fc700 1 -- 172.21.15.104:0/197649810 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v175) v1 ==== 92+0+10 (secure 0 0 0) 0x7f33c0059850 con 0x7f33c815c6f0 2023-06-10T18:58:57.101 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:58:57.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 msgr2=0x7f33ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 0x7f33ac06d190 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f33c81f2570 tx=0x7f33b8008040 comp rx=0 tx=0).stop 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 msgr2=0x7f33c80c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f33c0012000 tx=0x7f33c0004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 shutdown_connections 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33c815c6f0 0x7f33c80c8700 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33c806ea70 0x7f33c80b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f33ac06ace0 0x7f33ac06d190 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 --2- 172.21.15.104:0/197649810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33c80aec00 0x7f33c80cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 >> 172.21.15.104:0/197649810 conn(0x7f33c806b8a0 msgr2=0x7f33c80af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 shutdown_connections 2023-06-10T18:58:57.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.097+0000 7f33cffb9700 1 -- 172.21.15.104:0/197649810 wait complete. 2023-06-10T18:58:57.115 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:58:57.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 Processor -- start 2023-06-10T18:58:57.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 -- start start 2023-06-10T18:58:57.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50a80cd8f0 0x7f50a80ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80ad760 0x7f50a80adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.241 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a815cb00 con 0x7f50a815c730 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a80acd90 con 0x7f50a80cd8f0 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.232+0000 7f50aed3c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80ac460 con 0x7f50a80ad760 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50acad8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f50a815c730 0x7f50a80cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38754/0 (socket says 172.21.15.104:38754) 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50acad8700 1 -- 172.21.15.104:0/3059518417 learned_addr learned my addr 172.21.15.104:0/3059518417 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 --2- 172.21.15.104:0/3059518417 >> v2:172.21.15.114:3300/0 conn(0x7f50a80cd8f0 0x7f50a80ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50ad2d9700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80ad760 0x7f50a80adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.242 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a77fe700 1 -- 172.21.15.104:0/3059518417 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523562300 0 0) 0x7f509c001290 con 0x7f50a815c730 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a77fe700 1 -- 172.21.15.104:0/3059518417 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50980035b0 con 0x7f50a815c730 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 -- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80ad760 msgr2=0x7f50a80adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80ad760 0x7f50a80adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 -- 172.21.15.104:0/3059518417 >> v1:172.21.15.104:6789/0 conn(0x7f50a815c730 legacy=0x7f50a80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 -- 172.21.15.104:0/3059518417 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80ac740 con 0x7f50a80cd8f0 2023-06-10T18:58:57.243 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.233+0000 7f50a7fff700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a80cd8f0 0x7f50a80ad220 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5094009840 tx=0x7f5094009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab0bcd16fb5eca4a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.244 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50a77fe700 1 -- 172.21.15.104:0/3059518417 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094011830 con 0x7f50a80cd8f0 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50a77fe700 1 -- 172.21.15.104:0/3059518417 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094011990 con 0x7f50a80cd8f0 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50a77fe700 1 -- 172.21.15.104:0/3059518417 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094011d20 con 0x7f50a80cd8f0 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 -- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a80cd8f0 msgr2=0x7f50a80ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a80cd8f0 0x7f50a80ad220 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5094009840 tx=0x7f5094009b50 comp rx=0 tx=0).stop 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 -- 172.21.15.104:0/3059518417 shutdown_connections 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80ad760 0x7f50a80adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.245 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 --2- 172.21.15.104:0/3059518417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a80cd8f0 0x7f50a80ad220 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 -- 172.21.15.104:0/3059518417 >> 172.21.15.104:0/3059518417 conn(0x7f50a806b8a0 msgr2=0x7f50a80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 -- 172.21.15.104:0/3059518417 shutdown_connections 2023-06-10T18:58:57.246 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.234+0000 7f50aed3c700 1 -- 172.21.15.104:0/3059518417 wait complete. 2023-06-10T18:58:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.235+0000 7f50aed3c700 1 Processor -- start 2023-06-10T18:58:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.235+0000 7f50aed3c700 1 -- start start 2023-06-10T18:58:57.248 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 0x7f50a80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 0x7f50a80b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80c7ed0 con 0x7f50a80ad760 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80c81b0 con 0x7f50a815c730 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50aed3c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a80c8490 con 0x7f50a80cd8f0 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50acad8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50acad8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40932/0 (socket says 172.21.15.104:40932) 2023-06-10T18:58:57.249 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.236+0000 7f50acad8700 1 -- 172.21.15.104:0/2516118889 learned_addr learned my addr 172.21.15.104:0/2516118889 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50a7fff700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 0x7f50a80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50ad2d9700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 0x7f50a80b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 -- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 msgr2=0x7f50a80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 0x7f50a80c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 -- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 msgr2=0x7f50a80b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 0x7f50a80b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 -- 172.21.15.104:0/2516118889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940094a0 con 0x7f50a80ad760 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50a7fff700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 0x7f50a80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50ad2d9700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 0x7f50a80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50acad8700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c700 tx=0x7f509c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c005c40 con 0x7f50a80ad760 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c005da0 con 0x7f50a80ad760 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a806f150 con 0x7f50a80ad760 2023-06-10T18:58:57.251 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c0072f0 con 0x7f50a80ad760 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.237+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a80c8a50 con 0x7f50a80ad760 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.238+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a8074050 con 0x7f50a80ad760 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.238+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f509c007450 con 0x7f50a80ad760 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.239+0000 7f50a57fa700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 0x7f508c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.239+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f509c088a40 con 0x7f50a80ad760 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.239+0000 7f50a7fff700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 0x7f508c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.252 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.239+0000 7f50a7fff700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 0x7f508c06d190 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f50a81f34a0 tx=0x7f5094008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.253 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.243+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f509c08ae60 con 0x7f50a80ad760 2023-06-10T18:58:57.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.428+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50a815cb00 con 0x7f50a80ad760 2023-06-10T18:58:57.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.428+0000 7f50a57fa700 1 -- 172.21.15.104:0/2516118889 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v175) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50a815cb00 con 0x7f50a80ad760 2023-06-10T18:58:57.435 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:58:57.436 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 msgr2=0x7f508c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 0x7f508c06d190 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f50a81f34a0 tx=0x7f5094008030 comp rx=0 tx=0).stop 2023-06-10T18:58:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 msgr2=0x7f50a814da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f509c00c700 tx=0x7f509c00cac0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 shutdown_connections 2023-06-10T18:58:57.437 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f508c06ace0 0x7f508c06d190 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a815c730 0x7f50a80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50a80ad760 0x7f50a814da20 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 --2- 172.21.15.104:0/2516118889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f50a80cd8f0 0x7f50a80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 >> 172.21.15.104:0/2516118889 conn(0x7f50a806b8a0 msgr2=0x7f50a806e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 shutdown_connections 2023-06-10T18:58:57.438 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.430+0000 7f50aed3c700 1 -- 172.21.15.104:0/2516118889 wait complete. 2023-06-10T18:58:57.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:58:57.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.565+0000 7f1e025e7700 1 Processor -- start 2023-06-10T18:58:57.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.565+0000 7f1e025e7700 1 -- start start 2023-06-10T18:58:57.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1dfc15c500 0x7f1dfc0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1dfc0cf200 0x7f1dfc069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dfc0aef20 con 0x7f1dfc0cf200 2023-06-10T18:58:57.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dfc15c8d0 con 0x7f1dfc15c500 2023-06-10T18:58:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e025e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc06abd0 con 0x7f1dfc06a3b0 2023-06-10T18:58:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1dfc0cf200 0x7f1dfc069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1dfc0cf200 0x7f1dfc069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40940/0 (socket says 172.21.15.104:40940) 2023-06-10T18:58:57.581 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/3495982251 learned_addr learned my addr 172.21.15.104:0/3495982251 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1e00b84700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfbfff700 1 --2- 172.21.15.104:0/3495982251 >> v2:172.21.15.114:3300/0 conn(0x7f1dfc15c500 0x7f1dfc0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.582 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/3495982251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 msgr2=0x7f1dfc06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc06dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/3495982251 >> v2:172.21.15.114:3300/0 conn(0x7f1dfc15c500 msgr2=0x7f1dfc0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:58:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/3495982251 >> v2:172.21.15.114:3300/0 conn(0x7f1dfc15c500 0x7f1dfc0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.566+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/3495982251 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc06a8f0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc069e70 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0b2170 tx=0x7f1de4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9562041a56aeaf9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1dfaffd700 1 -- 172.21.15.104:0/3495982251 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400b870 con 0x7f1dfc0cf200 2023-06-10T18:58:57.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1dfaffd700 1 -- 172.21.15.104:0/3495982251 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400b9d0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1dfaffd700 1 -- 172.21.15.104:0/3495982251 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400bd60 con 0x7f1dfc0cf200 2023-06-10T18:58:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 -- 172.21.15.104:0/3495982251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 msgr2=0x7f1dfc069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc069e70 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0b2170 tx=0x7f1de4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.585 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 -- 172.21.15.104:0/3495982251 shutdown_connections 2023-06-10T18:58:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc06dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 --2- 172.21.15.104:0/3495982251 >> v2:172.21.15.114:3300/0 conn(0x7f1dfc15c500 0x7f1dfc0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 --2- 172.21.15.104:0/3495982251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc069e70 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 -- 172.21.15.104:0/3495982251 >> 172.21.15.104:0/3495982251 conn(0x7f1dfc06b8a0 msgr2=0x7f1dfc0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 -- 172.21.15.104:0/3495982251 shutdown_connections 2023-06-10T18:58:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.567+0000 7f1e025e7700 1 -- 172.21.15.104:0/3495982251 wait complete. 2023-06-10T18:58:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.568+0000 7f1e025e7700 1 Processor -- start 2023-06-10T18:58:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 -- start start 2023-06-10T18:58:57.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc0ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 0x7f1dfc155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc155c80 con 0x7f1dfc0cf200 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc0b25e0 con 0x7f1dfc15c500 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.569+0000 7f1e025e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc0b28c0 con 0x7f1dfc06a3b0 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40948/0 (socket says 172.21.15.104:40948) 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/236502261 learned_addr learned my addr 172.21.15.104:0/236502261 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfbfff700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc0ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1e00b84700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 0x7f1dfc155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 msgr2=0x7f1dfc0ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc0ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.589 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 msgr2=0x7f1dfc155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 0x7f1dfc155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 -- 172.21.15.104:0/236502261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4009550 con 0x7f1dfc0cf200 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1e00b84700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 0x7f1dfc155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1dfb7fe700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f1de4000c00 tx=0x7f1de4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4004540 con 0x7f1dfc0cf200 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de40036a0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc0b2ba0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4003a50 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.570+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc0b3160 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.571+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dfc066f10 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.572+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1de4021070 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.572+0000 7f1df8ff9700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 0x7f1ddc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.572+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f1de408ac60 con 0x7f1dfc0cf200 2023-06-10T18:58:57.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.572+0000 7f1dfbfff700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 0x7f1ddc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.573+0000 7f1dfbfff700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 0x7f1ddc06d190 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc15ee20 tx=0x7f1dec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.576+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1de4055bf0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.765+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1dfc074050 con 0x7f1dfc0cf200 2023-06-10T18:58:57.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.766+0000 7f1df8ff9700 1 -- 172.21.15.104:0/236502261 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v175) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1de40150a0 con 0x7f1dfc0cf200 2023-06-10T18:58:57.772 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:58:57.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.767+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 msgr2=0x7f1ddc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.767+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 0x7f1ddc06d190 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc15ee20 tx=0x7f1dec008040 comp rx=0 tx=0).stop 2023-06-10T18:58:57.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.767+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 msgr2=0x7f1dfc155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.767+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f1de4000c00 tx=0x7f1de4004750 comp rx=0 tx=0).stop 2023-06-10T18:58:57.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 shutdown_connections 2023-06-10T18:58:57.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1ddc06ace0 0x7f1ddc06d190 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1dfc15c500 0x7f1dfc155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1dfc06a3b0 0x7f1dfc0ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 --2- 172.21.15.104:0/236502261 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1dfc0cf200 0x7f1dfc155200 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 >> 172.21.15.104:0/236502261 conn(0x7f1dfc06b8a0 msgr2=0x7f1dfc15ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 shutdown_connections 2023-06-10T18:58:57.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.768+0000 7f1e025e7700 1 -- 172.21.15.104:0/236502261 wait complete. 2023-06-10T18:58:57.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:58:57.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.899+0000 7ff60824a700 1 Processor -- start 2023-06-10T18:58:57.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.899+0000 7ff60824a700 1 -- start start 2023-06-10T18:58:57.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff60824a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff600165060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff60824a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000b6070 con 0x7ff60015a420 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff60824a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000b1d70 con 0x7ff6000b4ff0 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff60824a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000b5460 con 0x7ff6000cea60 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff605fe6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff60015a420 0x7ff6000b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38782/0 (socket says 172.21.15.104:38782) 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.900+0000 7ff605fe6700 1 -- 172.21.15.104:0/3939925122 learned_addr learned my addr 172.21.15.104:0/3939925122 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff6067e7700 1 --2- 172.21.15.104:0/3939925122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff600165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480258808 0 0) 0x7ff6000b6070 con 0x7ff60015a420 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ec0035b0 con 0x7ff60015a420 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807486852 0 0) 0x7ff6000b1d70 con 0x7ff6000b4ff0 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6000b6070 con 0x7ff6000b4ff0 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff6067e7700 1 -- 172.21.15.104:0/3939925122 >> v1:172.21.15.114:6789/0 conn(0x7ff6000b4ff0 legacy=0x7ff6000ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff6067e7700 1 -- 172.21.15.104:0/3939925122 >> v1:172.21.15.104:6789/0 conn(0x7ff60015a420 legacy=0x7ff6000b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1008168196 0 0) 0x7ff5ec0035b0 con 0x7ff60015a420 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff6067e7700 1 -- 172.21.15.104:0/3939925122 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000b5740 con 0x7ff6000cea60 2023-06-10T18:58:57.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff6067e7700 1 --2- 172.21.15.104:0/3939925122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff600165060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009dc0 tx=0x7ff5f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8405904302ad995 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800e070 con 0x7ff6000cea60 2023-06-10T18:58:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8003a20 con 0x7ff6000cea60 2023-06-10T18:58:57.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.901+0000 7ff604fe4700 1 -- 172.21.15.104:0/3939925122 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0001670 con 0x7ff6000cea60 2023-06-10T18:58:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 -- 172.21.15.104:0/3939925122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 msgr2=0x7ff600165060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 --2- 172.21.15.104:0/3939925122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff600165060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009dc0 tx=0x7ff5f800b330 comp rx=0 tx=0).stop 2023-06-10T18:58:57.912 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 -- 172.21.15.104:0/3939925122 shutdown_connections 2023-06-10T18:58:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 --2- 172.21.15.104:0/3939925122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff600165060 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 -- 172.21.15.104:0/3939925122 >> 172.21.15.104:0/3939925122 conn(0x7ff600089fd0 msgr2=0x7ff6000d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 -- 172.21.15.104:0/3939925122 shutdown_connections 2023-06-10T18:58:57.913 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.902+0000 7ff60824a700 1 -- 172.21.15.104:0/3939925122 wait complete. 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.903+0000 7ff60824a700 1 Processor -- start 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.903+0000 7ff60824a700 1 -- start start 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 0x7ff60014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff6001f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.914 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014c850 con 0x7ff60015a420 2023-06-10T18:58:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014bcc0 con 0x7ff6000b4ff0 2023-06-10T18:58:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff60824a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014bf20 con 0x7ff6000cea60 2023-06-10T18:58:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40964/0 (socket says 172.21.15.104:40964) 2023-06-10T18:58:57.915 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 -- 172.21.15.104:0/2026963302 learned_addr learned my addr 172.21.15.104:0/2026963302 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff605fe6700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 0x7ff60014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6057e5700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff6001f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 -- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 msgr2=0x7ff6001f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff6001f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 -- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 msgr2=0x7ff60014cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 0x7ff60014cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:57.916 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.904+0000 7ff6067e7700 1 -- 172.21.15.104:0/2026963302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8009a70 con 0x7ff60015a420 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff6057e5700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff6001f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff605fe6700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 0x7ff60014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff6067e7700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009d90 tx=0x7ff5f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80046f0 con 0x7ff60015a420 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8004850 con 0x7ff60015a420 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60014c200 con 0x7ff60015a420 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f801d670 con 0x7ff60015a420 2023-06-10T18:58:57.917 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.905+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60014f490 con 0x7ff60015a420 2023-06-10T18:58:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.906+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff600153e50 con 0x7ff60015a420 2023-06-10T18:58:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.906+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff5f801d7d0 con 0x7ff60015a420 2023-06-10T18:58:57.918 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.907+0000 7ff5f67fc700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 0x7ff5e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:58:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.907+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7ff5f808b310 con 0x7ff60015a420 2023-06-10T18:58:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.907+0000 7ff605fe6700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 0x7ff5e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:58:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.907+0000 7ff605fe6700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 0x7ff5e406d190 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000f60 tx=0x7ff5f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:58:57.919 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:57.911+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff5f80562a0 con 0x7ff60015a420 2023-06-10T18:58:58.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.111+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff6000b64e0 con 0x7ff60015a420 2023-06-10T18:58:58.119 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.111+0000 7ff5f67fc700 1 -- 172.21.15.104:0/2026963302 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v175) v1 ==== 80+0+56 (secure 0 0 0) 0x7ff6000b64e0 con 0x7ff60015a420 2023-06-10T18:58:58.121 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/827471637 2023-06-10T19:58:50.352178+0000 2023-06-10T18:58:58.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 msgr2=0x7ff5e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 0x7ff5e406d190 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000f60 tx=0x7ff5f0009040 comp rx=0 tx=0).stop 2023-06-10T18:58:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 msgr2=0x7ff60014b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:58:58.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009d90 tx=0x7ff5f8004ad0 comp rx=0 tx=0).stop 2023-06-10T18:58:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 shutdown_connections 2023-06-10T18:58:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff5e406ace0 0x7ff5e406d190 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:58.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6000b4ff0 0x7ff60014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:58.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff60015a420 0x7ff60014b780 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:58.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 --2- 172.21.15.104:0/2026963302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff6000cea60 0x7ff6001f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:58:58.125 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 >> 172.21.15.104:0/2026963302 conn(0x7ff600089fd0 msgr2=0x7ff600158190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:58:58.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 shutdown_connections 2023-06-10T18:58:58.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:58:58.113+0000 7ff60824a700 1 -- 172.21.15.104:0/2026963302 wait complete. 2023-06-10T18:58:58.128 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:58:58.132 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:58:58.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:58:58.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T18:58:58.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[162045]: fuse finished with error 0 and tester_r 0 2023-06-10T18:58:58.170 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:04.271 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:59:04.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:04.272 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:59:04.293 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:59:04.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:59:04.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:59:04.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:59:04.437 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:04.438 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:04.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:04.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.619+0000 7fc35e1ae700 1 Processor -- start 2023-06-10T18:59:04.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.619+0000 7fc35e1ae700 1 -- start start 2023-06-10T18:59:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc35e1ae700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3580cf030 0x7fc3580ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc35e1ae700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35806dcb0 con 0x7fc358069e70 2023-06-10T18:59:04.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc35e1ae700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3580cf460 con 0x7fc35815c500 2023-06-10T18:59:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc35e1ae700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35806e850 con 0x7fc3580cf030 2023-06-10T18:59:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3580cf030 0x7fc3580ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc357fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc358069e70 0x7fc3580c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38800/0 (socket says 172.21.15.104:38800) 2023-06-10T18:59:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc3580cf030 0x7fc3580ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40970/0 (socket says 172.21.15.104:40970) 2023-06-10T18:59:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc357fff700 1 -- 172.21.15.104:0/1876185010 learned_addr learned my addr 172.21.15.104:0/1876185010 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 -- 172.21.15.104:0/1876185010 >> v1:172.21.15.114:6789/0 conn(0x7fc35815c500 legacy=0x7fc35806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 -- 172.21.15.104:0/1876185010 >> v1:172.21.15.104:6789/0 conn(0x7fc358069e70 legacy=0x7fc3580c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 -- 172.21.15.104:0/1876185010 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35806d0a0 con 0x7fc3580cf030 2023-06-10T18:59:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc3567fc700 1 -- 172.21.15.104:0/1876185010 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217140713 0 0) 0x7fc35806dcb0 con 0x7fc358069e70 2023-06-10T18:59:04.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.620+0000 7fc356ffd700 1 --2- 172.21.15.104:0/1876185010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580ce040 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fc348009840 tx=0x7fc348009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1242f86c8c7ba93f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc3567fc700 1 -- 172.21.15.104:0/1876185010 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348011830 con 0x7fc3580cf030 2023-06-10T18:59:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc3567fc700 1 -- 172.21.15.104:0/1876185010 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc348011990 con 0x7fc3580cf030 2023-06-10T18:59:04.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc3567fc700 1 -- 172.21.15.104:0/1876185010 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348011d20 con 0x7fc3580cf030 2023-06-10T18:59:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 -- 172.21.15.104:0/1876185010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 msgr2=0x7fc3580ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/1876185010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580ce040 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fc348009840 tx=0x7fc348009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 -- 172.21.15.104:0/1876185010 shutdown_connections 2023-06-10T18:59:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/1876185010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580ce040 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 -- 172.21.15.104:0/1876185010 >> 172.21.15.104:0/1876185010 conn(0x7fc35806b8a0 msgr2=0x7fc3580cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 -- 172.21.15.104:0/1876185010 shutdown_connections 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.621+0000 7fc35e1ae700 1 -- 172.21.15.104:0/1876185010 wait complete. 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.622+0000 7fc35e1ae700 1 Processor -- start 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 -- start start 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc358069e70 0x7fc3580b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc35815c500 0x7fc35815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580b55b0 con 0x7fc3580cf030 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580b3d70 con 0x7fc358069e70 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc35e1ae700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580b4ff0 con 0x7fc35815c500 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc356ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc356ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40976/0 (socket says 172.21.15.104:40976) 2023-06-10T18:59:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.623+0000 7fc356ffd700 1 -- 172.21.15.104:0/3125060423 learned_addr learned my addr 172.21.15.104:0/3125060423 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc3577fe700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc358069e70 0x7fc3580b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 -- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc35815c500 msgr2=0x7fc35815aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc35815c500 0x7fc35815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 -- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc358069e70 msgr2=0x7fc3580b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc358069e70 0x7fc3580b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 -- 172.21.15.104:0/3125060423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3480094a0 con 0x7fc3580cf030 2023-06-10T18:59:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc356ffd700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fc3480090b0 tx=0x7fc348004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34801a980 con 0x7fc3580cf030 2023-06-10T18:59:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34801ab00 con 0x7fc3580cf030 2023-06-10T18:59:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348024420 con 0x7fc3580cf030 2023-06-10T18:59:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3580b52d0 con 0x7fc3580cf030 2023-06-10T18:59:04.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.624+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3580c8030 con 0x7fc3580cf030 2023-06-10T18:59:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.625+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3580c7bb0 con 0x7fc3580cf030 2023-06-10T18:59:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.625+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc348022070 con 0x7fc3580cf030 2023-06-10T18:59:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.626+0000 7fc34ffff700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 0x7fc33806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.626+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc34808a9c0 con 0x7fc3580cf030 2023-06-10T18:59:04.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.626+0000 7fc3577fe700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 0x7fc33806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.627+0000 7fc3577fe700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 0x7fc33806d190 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc358099820 tx=0x7fc340009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.630+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc3580c7bb0 con 0x7fc3580cf030 2023-06-10T18:59:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.817+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3580c7bb0 con 0x7fc3580cf030 2023-06-10T18:59:04.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.818+0000 7fc34ffff700 1 -- 172.21.15.104:0/3125060423 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+18714 (secure 0 0 0) 0x7fc3580c7bb0 con 0x7fc3580cf030 2023-06-10T18:59:04.826 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:04.826 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":175,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:54.747698+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.820+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 msgr2=0x7fc33806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.820+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 0x7fc33806d190 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc358099820 tx=0x7fc340009040 comp rx=0 tx=0).stop 2023-06-10T18:59:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.820+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 msgr2=0x7fc3580b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.820+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fc3480090b0 tx=0x7fc348004750 comp rx=0 tx=0).stop 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.820+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 shutdown_connections 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc33806ace0 0x7fc33806d190 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc358069e70 0x7fc3580b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc35815c500 0x7fc35815aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 --2- 172.21.15.104:0/3125060423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3580cf030 0x7fc3580b3830 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 >> 172.21.15.104:0/3125060423 conn(0x7fc35806b8a0 msgr2=0x7fc3580cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:04.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 shutdown_connections 2023-06-10T18:59:04.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.821+0000 7fc35e1ae700 1 -- 172.21.15.104:0/3125060423 wait complete. 2023-06-10T18:59:04.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:59:04.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:59:04.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.958+0000 7f8bfe03f700 1 Processor -- start 2023-06-10T18:59:04.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.958+0000 7f8bfe03f700 1 -- start start 2023-06-10T18:59:04.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8bf815c500 0x7f8bf80cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8bf80cf200 0x7f8bf8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf80aef20 con 0x7f8bf80cf200 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf815c8d0 con 0x7f8bf815c500 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bfe03f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf806abd0 con 0x7f8bf806a3b0 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8bf80cf200 0x7f8bf8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8bf80cf200 0x7f8bf8069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40984/0 (socket says 172.21.15.104:40984) 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 -- 172.21.15.104:0/3329027359 learned_addr learned my addr 172.21.15.104:0/3329027359 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf77fe700 1 --2- 172.21.15.104:0/3329027359 >> v2:172.21.15.114:3300/0 conn(0x7f8bf815c500 0x7f8bf80cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf7fff700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 -- 172.21.15.104:0/3329027359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 msgr2=0x7f8bf806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf806dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 -- 172.21.15.104:0/3329027359 >> v2:172.21.15.114:3300/0 conn(0x7f8bf815c500 msgr2=0x7f8bf80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 --2- 172.21.15.104:0/3329027359 >> v2:172.21.15.114:3300/0 conn(0x7f8bf815c500 0x7f8bf80cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.959+0000 7f8bf6ffd700 1 -- 172.21.15.104:0/3329027359 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf806a8f0 con 0x7f8bf80cf200 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bf6ffd700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf80cf200 0x7f8bf8069e70 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80b2170 tx=0x7f8be8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c243820cb329210 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bf67fc700 1 -- 172.21.15.104:0/3329027359 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be800b870 con 0x7f8bf80cf200 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bf67fc700 1 -- 172.21.15.104:0/3329027359 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be800b9d0 con 0x7f8bf80cf200 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bf67fc700 1 -- 172.21.15.104:0/3329027359 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be800bd60 con 0x7f8bf80cf200 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 -- 172.21.15.104:0/3329027359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf80cf200 msgr2=0x7f8bf8069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf80cf200 0x7f8bf8069e70 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80b2170 tx=0x7f8be8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 -- 172.21.15.104:0/3329027359 shutdown_connections 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf806dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/3329027359 >> v2:172.21.15.114:3300/0 conn(0x7f8bf815c500 0x7f8bf80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/3329027359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf80cf200 0x7f8bf8069e70 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 -- 172.21.15.104:0/3329027359 >> 172.21.15.104:0/3329027359 conn(0x7f8bf806b8a0 msgr2=0x7f8bf80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 -- 172.21.15.104:0/3329027359 shutdown_connections 2023-06-10T18:59:04.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.960+0000 7f8bfe03f700 1 -- 172.21.15.104:0/3329027359 wait complete. 2023-06-10T18:59:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 Processor -- start 2023-06-10T18:59:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 -- start start 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf80ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bf80cf200 0x7f8bf8155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf8155c80 con 0x7f8bf815c500 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf80b25e0 con 0x7f8bf80cf200 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.962+0000 7f8bfe03f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf80b28c0 con 0x7f8bf806a3b0 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:40992/0 (socket says 172.21.15.104:40992) 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4088891102 learned_addr learned my addr 172.21.15.104:0/4088891102 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf77fe700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf80ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf6ffd700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bf80cf200 0x7f8bf8155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 msgr2=0x7f8bf80ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf80ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bf80cf200 msgr2=0x7f8bf8155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bf80cf200 0x7f8bf8155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 -- 172.21.15.104:0/4088891102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8009550 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bf7fff700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0007d20 tx=0x7f8bf000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000ed30 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0007e70 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf80b2ba0 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0014640 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.963+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf80b3160 con 0x7f8bf815c500 2023-06-10T18:59:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.964+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf8074050 con 0x7f8bf815c500 2023-06-10T18:59:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.965+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8bf00147a0 con 0x7f8bf815c500 2023-06-10T18:59:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.965+0000 7f8beffff700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 0x7f8bd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.965+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8bf0089b90 con 0x7f8bf815c500 2023-06-10T18:59:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.965+0000 7f8bf77fe700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 0x7f8bd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.966+0000 7f8bf77fe700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 0x7f8bd806d190 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8bf815ee20 tx=0x7f8be0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:04.969+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8bf0054b20 con 0x7f8bf815c500 2023-06-10T18:59:05.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.155+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8bf806f0a0 con 0x7f8bf815c500 2023-06-10T18:59:05.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.155+0000 7f8beffff700 1 -- 172.21.15.104:0/4088891102 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8bf0058140 con 0x7f8bf815c500 2023-06-10T18:59:05.162 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:59:05.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 msgr2=0x7f8bd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 0x7f8bd806d190 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8bf815ee20 tx=0x7f8be0008040 comp rx=0 tx=0).stop 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 msgr2=0x7f8bf8155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0007d20 tx=0x7f8bf000e3c0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 shutdown_connections 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bf815c500 0x7f8bf8155740 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8bf806a3b0 0x7f8bf80ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8bd806ace0 0x7f8bd806d190 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 --2- 172.21.15.104:0/4088891102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bf80cf200 0x7f8bf8155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 >> 172.21.15.104:0/4088891102 conn(0x7f8bf806b8a0 msgr2=0x7f8bf815ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 shutdown_connections 2023-06-10T18:59:05.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.157+0000 7f8bfe03f700 1 -- 172.21.15.104:0/4088891102 wait complete. 2023-06-10T18:59:05.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:59:05.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.292+0000 7f22948c3700 1 Processor -- start 2023-06-10T18:59:05.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.292+0000 7f22948c3700 1 -- start start 2023-06-10T18:59:05.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f22948c3700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22900cea50 0x7f22900d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f22948c3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900ad3c0 con 0x7f22900d0830 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f22948c3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2290157f00 con 0x7f2290157b30 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f22948c3700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22900b0170 con 0x7f22900cea50 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228ed9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f22900d0830 0x7f22900b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38828/0 (socket says 172.21.15.104:38828) 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228ed9d700 1 -- 172.21.15.104:0/3965373340 learned_addr learned my addr 172.21.15.104:0/3965373340 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228dd9b700 1 --2- 172.21.15.104:0/3965373340 >> v2:172.21.15.114:3300/0 conn(0x7f22900cea50 0x7f22900d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228d59a700 1 -- 172.21.15.104:0/3965373340 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555427270 0 0) 0x7f22900ad3c0 con 0x7f22900d0830 2023-06-10T18:59:05.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228d59a700 1 -- 172.21.15.104:0/3965373340 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22780035b0 con 0x7f22900d0830 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228dd9b700 1 -- 172.21.15.104:0/3965373340 >> v1:172.21.15.114:6789/0 conn(0x7f2290157b30 legacy=0x7f22900ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228dd9b700 1 -- 172.21.15.104:0/3965373340 >> v1:172.21.15.104:6789/0 conn(0x7f22900d0830 legacy=0x7f22900b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228dd9b700 1 -- 172.21.15.104:0/3965373340 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900b0410 con 0x7f22900cea50 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.293+0000 7f228dd9b700 1 --2- 172.21.15.104:0/3965373340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22900cea50 0x7f22900d02f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2284009840 tx=0x7f2284009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a0e3a0c8fea06f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f228d59a700 1 -- 172.21.15.104:0/3965373340 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284011830 con 0x7f22900cea50 2023-06-10T18:59:05.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f228d59a700 1 -- 172.21.15.104:0/3965373340 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284011990 con 0x7f22900cea50 2023-06-10T18:59:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f228d59a700 1 -- 172.21.15.104:0/3965373340 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284011d20 con 0x7f22900cea50 2023-06-10T18:59:05.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 -- 172.21.15.104:0/3965373340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22900cea50 msgr2=0x7f22900d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 --2- 172.21.15.104:0/3965373340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22900cea50 0x7f22900d02f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2284009840 tx=0x7f2284009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 -- 172.21.15.104:0/3965373340 shutdown_connections 2023-06-10T18:59:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 --2- 172.21.15.104:0/3965373340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22900cea50 0x7f22900d02f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 -- 172.21.15.104:0/3965373340 >> 172.21.15.104:0/3965373340 conn(0x7f2290089fd0 msgr2=0x7f229008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 -- 172.21.15.104:0/3965373340 shutdown_connections 2023-06-10T18:59:05.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.294+0000 7f22948c3700 1 -- 172.21.15.104:0/3965373340 wait complete. 2023-06-10T18:59:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.295+0000 7f22948c3700 1 Processor -- start 2023-06-10T18:59:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 -- start start 2023-06-10T18:59:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 0x7f22900b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 0x7f22900b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900c48c0 con 0x7f22900d0830 2023-06-10T18:59:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900b1d00 con 0x7f2290157b30 2023-06-10T18:59:05.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f22948c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900b1fe0 con 0x7f22900cea50 2023-06-10T18:59:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f228dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f228dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41004/0 (socket says 172.21.15.104:41004) 2023-06-10T18:59:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.296+0000 7f228dd9b700 1 -- 172.21.15.104:0/2420281025 learned_addr learned my addr 172.21.15.104:0/2420281025 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:05.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228ed9d700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 0x7f22900b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228e59c700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 0x7f22900b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 -- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 msgr2=0x7f22900b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 0x7f22900b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 -- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 msgr2=0x7f22900b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 0x7f22900b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 -- 172.21.15.104:0/2420281025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840094a0 con 0x7f22900d0830 2023-06-10T18:59:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228e59c700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 0x7f22900b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228ed9d700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 0x7f22900b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:05.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f228dd9b700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f2284009fd0 tx=0x7f2284004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840038a0 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228401f070 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284003c40 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900b22c0 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.297+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22900b6310 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.298+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22900c4a30 con 0x7f22900d0830 2023-06-10T18:59:05.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.298+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2284026480 con 0x7f22900d0830 2023-06-10T18:59:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.299+0000 7f227effd700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.299+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f2284021070 con 0x7f22900d0830 2023-06-10T18:59:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.299+0000 7f228e59c700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.299+0000 7f228e59c700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2280000f60 tx=0x7f2280009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.303+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22900c4a30 con 0x7f22900d0830 2023-06-10T18:59:05.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.487+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f22900c4a00 con 0x7f22900d0830 2023-06-10T18:59:05.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.488+0000 7f227effd700 1 -- 172.21.15.104:0/2420281025 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v175) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2284055e70 con 0x7f22900d0830 2023-06-10T18:59:05.495 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:59:05.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 msgr2=0x7f227006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2280000f60 tx=0x7f2280009040 comp rx=0 tx=0).stop 2023-06-10T18:59:05.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 msgr2=0x7f22900b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f2284009fd0 tx=0x7f2284004750 comp rx=0 tx=0).stop 2023-06-10T18:59:05.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 shutdown_connections 2023-06-10T18:59:05.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f227006ace0 0x7f227006d190 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2290157b30 0x7f22900b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22900cea50 0x7f22900b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 --2- 172.21.15.104:0/2420281025 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22900d0830 0x7f22900b35a0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 >> 172.21.15.104:0/2420281025 conn(0x7f2290089fd0 msgr2=0x7f22900ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 shutdown_connections 2023-06-10T18:59:05.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.490+0000 7f22948c3700 1 -- 172.21.15.104:0/2420281025 wait complete. 2023-06-10T18:59:05.509 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:59:05.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.625+0000 7fb13e13b700 1 Processor -- start 2023-06-10T18:59:05.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.625+0000 7fb13e13b700 1 -- start start 2023-06-10T18:59:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb13e13b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb13806c720 0x7fb1380ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb13e13b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380cae10 0x7fb1380cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb13e13b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380d0340 con 0x7fb1380ce390 2023-06-10T18:59:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb13e13b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1380d1660 con 0x7fb13806c720 2023-06-10T18:59:05.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb13e13b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13815cb60 con 0x7fb1380cae10 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb136ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb13806c720 0x7fb1380ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb136ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb13806c720 0x7fb1380ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41010/0 (socket says 172.21.15.104:41010) 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.626+0000 7fb136ffd700 1 -- 172.21.15.104:0/3241468078 learned_addr learned my addr 172.21.15.104:0/3241468078 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb137fff700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380cae10 0x7fb1380cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb136ffd700 1 -- 172.21.15.104:0/3241468078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380cae10 msgr2=0x7fb1380cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb136ffd700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380cae10 0x7fb1380cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb136ffd700 1 -- 172.21.15.104:0/3241468078 >> v1:172.21.15.114:6789/0 conn(0x7fb1380ce390 legacy=0x7fb138160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb136ffd700 1 -- 172.21.15.104:0/3241468078 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380ceca0 con 0x7fb13806c720 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb136ffd700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13806c720 0x7fb1380ce760 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fb1280098f0 tx=0x7fb128009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c59137606b6c9e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb1367fc700 1 -- 172.21.15.104:0/3241468078 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128004230 con 0x7fb13806c720 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb1367fc700 1 -- 172.21.15.104:0/3241468078 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb128004390 con 0x7fb13806c720 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb1367fc700 1 -- 172.21.15.104:0/3241468078 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1280036a0 con 0x7fb13806c720 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 -- 172.21.15.104:0/3241468078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13806c720 msgr2=0x7fb1380ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13806c720 0x7fb1380ce760 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fb1280098f0 tx=0x7fb128009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 -- 172.21.15.104:0/3241468078 shutdown_connections 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380cae10 0x7fb1380cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 --2- 172.21.15.104:0/3241468078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb13806c720 0x7fb1380ce760 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.627+0000 7fb13e13b700 1 -- 172.21.15.104:0/3241468078 >> 172.21.15.104:0/3241468078 conn(0x7fb138069e70 msgr2=0x7fb13806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.628+0000 7fb13e13b700 1 -- 172.21.15.104:0/3241468078 shutdown_connections 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.628+0000 7fb13e13b700 1 -- 172.21.15.104:0/3241468078 wait complete. 2023-06-10T18:59:05.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.629+0000 7fb13e13b700 1 Processor -- start 2023-06-10T18:59:05.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.629+0000 7fb13e13b700 1 -- start start 2023-06-10T18:59:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.629+0000 7fb13e13b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13806c720 0x7fb1380b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 0x7fb1381f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380c49b0 con 0x7fb1380cae10 2023-06-10T18:59:05.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380c4c90 con 0x7fb13806c720 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380c4f70 con 0x7fb1380ce390 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41016/0 (socket says 172.21.15.104:41016) 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 -- 172.21.15.104:0/419380222 learned_addr learned my addr 172.21.15.104:0/419380222 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb1377fe700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13806c720 0x7fb1380b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb137fff700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 0x7fb1381f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 -- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 msgr2=0x7fb1381f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 0x7fb1381f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 -- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13806c720 msgr2=0x7fb1380b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13806c720 0x7fb1380b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 -- 172.21.15.104:0/419380222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb128009550 con 0x7fb1380cae10 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb137fff700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 0x7fb1381f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb136ffd700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fb128009130 tx=0x7fb128004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12800b660 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12800b7c0 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1380c5250 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12800bb70 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.630+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1380c5840 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.631+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1380b6470 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.632+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb12800bcd0 con 0x7fb1380cae10 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.632+0000 7fb11ffff700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 0x7fb12406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.632+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fb12808af10 con 0x7fb1380cae10 2023-06-10T18:59:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.632+0000 7fb1377fe700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 0x7fb12406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.633+0000 7fb1377fe700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 0x7fb12406d190 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb1380ca7b0 tx=0x7fb12c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.637+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb128055ea0 con 0x7fb1380cae10 2023-06-10T18:59:05.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.823+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1380c4af0 con 0x7fb1380cae10 2023-06-10T18:59:05.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.824+0000 7fb11ffff700 1 -- 172.21.15.104:0/419380222 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v175) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1380c4af0 con 0x7fb1380cae10 2023-06-10T18:59:05.830 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:59:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 msgr2=0x7fb12406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 0x7fb12406d190 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fb1380ca7b0 tx=0x7fb12c007ec0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 msgr2=0x7fb1380c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fb128009130 tx=0x7fb128004750 comp rx=0 tx=0).stop 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 shutdown_connections 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb12406ace0 0x7fb12406d190 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13806c720 0x7fb1380b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1380ce390 0x7fb1381f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 --2- 172.21.15.104:0/419380222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1380cae10 0x7fb1380c42f0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 >> 172.21.15.104:0/419380222 conn(0x7fb138069e70 msgr2=0x7fb13806fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 shutdown_connections 2023-06-10T18:59:05.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.827+0000 7fb13e13b700 1 -- 172.21.15.104:0/419380222 wait complete. 2023-06-10T18:59:05.844 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:59:05.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.960+0000 7f1ea8f3d700 1 Processor -- start 2023-06-10T18:59:05.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.960+0000 7f1ea8f3d700 1 -- start start 2023-06-10T18:59:05.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea8f3d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ea415c730 0x7f1ea40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea8f3d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1ea40ae230 0x7f1ea406d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea8f3d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea415cb00 con 0x7f1ea40cd720 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea8f3d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea40cdbc0 con 0x7f1ea40ae230 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea8f3d700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea40acd90 con 0x7f1ea415c730 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea1d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1ea40cd720 0x7f1ea406cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38850/0 (socket says 172.21.15.104:38850) 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1ea40ae230 0x7f1ea406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea1d9b700 1 -- 172.21.15.104:0/1689524878 learned_addr learned my addr 172.21.15.104:0/1689524878 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea259c700 1 --2- 172.21.15.104:0/1689524878 >> v2:172.21.15.114:3300/0 conn(0x7f1ea415c730 0x7f1ea40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114482650 0 0) 0x7f1e94001110 con 0x7f1ea40cd720 2023-06-10T18:59:05.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.961+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8c0035b0 con 0x7f1ea40cd720 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea2d9d700 1 -- 172.21.15.104:0/1689524878 >> v2:172.21.15.114:3300/0 conn(0x7f1ea415c730 msgr2=0x7f1ea40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865446194 0 0) 0x7f1e8c0035b0 con 0x7f1ea40cd720 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea2d9d700 1 --2- 172.21.15.104:0/1689524878 >> v2:172.21.15.114:3300/0 conn(0x7f1ea415c730 0x7f1ea40cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea2d9d700 1 -- 172.21.15.104:0/1689524878 >> v1:172.21.15.104:6789/0 conn(0x7f1ea40cd720 legacy=0x7f1ea406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea2d9d700 1 -- 172.21.15.104:0/1689524878 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea40ac460 con 0x7f1ea40ae230 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea2d9d700 1 --2- 172.21.15.104:0/1689524878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea406d480 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009d90 tx=0x7f1e9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5a05240ca99700 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00e070 con 0x7f1ea40ae230 2023-06-10T18:59:05.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c003a20 con 0x7f1ea40ae230 2023-06-10T18:59:05.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea159a700 1 -- 172.21.15.104:0/1689524878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94001800 con 0x7f1ea40ae230 2023-06-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/1689524878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 msgr2=0x7f1ea406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.962+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/1689524878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea406d480 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009d90 tx=0x7f1e9c00b330 comp rx=0 tx=0).stop 2023-06-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/1689524878 shutdown_connections 2023-06-10T18:59:05.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/1689524878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea406d480 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/1689524878 >> v2:172.21.15.114:3300/0 conn(0x7f1ea415c730 0x7f1ea40cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/1689524878 >> 172.21.15.104:0/1689524878 conn(0x7f1ea406b8a0 msgr2=0x7f1ea40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/1689524878 shutdown_connections 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.963+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/1689524878 wait complete. 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.964+0000 7f1ea8f3d700 1 Processor -- start 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.964+0000 7f1ea8f3d700 1 -- start start 2023-06-10T18:59:05.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ea40cd720 0x7f1ea41f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ea415c730 0x7f1ea41f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea41f2f10 con 0x7f1ea40ae230 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea41f56b0 con 0x7f1ea40cd720 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea40b1ed0 con 0x7f1ea415c730 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41026/0 (socket says 172.21.15.104:41026) 2023-06-10T18:59:05.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 -- 172.21.15.104:0/3239429588 learned_addr learned my addr 172.21.15.104:0/3239429588 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea1d9b700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ea40cd720 0x7f1ea41f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea2d9d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ea415c730 0x7f1ea41f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 -- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ea415c730 msgr2=0x7f1ea41f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ea415c730 0x7f1ea41f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 -- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ea40cd720 msgr2=0x7f1ea41f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ea40cd720 0x7f1ea41f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:05.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 -- 172.21.15.104:0/3239429588 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c009a70 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea259c700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800e9c0 tx=0x7f1e9800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e980096f0 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9800c850 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea40af110 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800cbc0 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.965+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea40af6d0 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.966+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea4074050 con 0x7f1ea40ae230 2023-06-10T18:59:05.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.967+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1e9800cd20 con 0x7f1ea40ae230 2023-06-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.967+0000 7f1e92ffd700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 0x7f1e8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:05.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.967+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f1e980899f0 con 0x7f1ea40ae230 2023-06-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.967+0000 7f1ea1d9b700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 0x7f1e8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.968+0000 7f1ea1d9b700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 0x7f1e8406d190 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40c9a50 tx=0x7f1e940012f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:05.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:05.972+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1e98054980 con 0x7f1ea40ae230 2023-06-10T18:59:06.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.157+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ea40ae6f0 con 0x7f1ea40ae230 2023-06-10T18:59:06.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.157+0000 7f1e92ffd700 1 -- 172.21.15.104:0/3239429588 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v175) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1e98021020 con 0x7f1ea40ae230 2023-06-10T18:59:06.164 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:59:06.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 msgr2=0x7f1e8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 0x7f1e8406d190 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40c9a50 tx=0x7f1e940012f0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 msgr2=0x7f1ea41f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800e9c0 tx=0x7f1e9800ecd0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 shutdown_connections 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ea415c730 0x7f1ea41f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea40ae230 0x7f1ea41f2410 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1e8406ace0 0x7f1e8406d190 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 --2- 172.21.15.104:0/3239429588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ea40cd720 0x7f1ea41f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 >> 172.21.15.104:0/3239429588 conn(0x7f1ea406b8a0 msgr2=0x7f1ea406e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:06.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 shutdown_connections 2023-06-10T18:59:06.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.159+0000 7f1ea8f3d700 1 -- 172.21.15.104:0/3239429588 wait complete. 2023-06-10T18:59:06.177 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:59:06.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.292+0000 7f024d772700 1 Processor -- start 2023-06-10T18:59:06.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.292+0000 7f024d772700 1 -- start start 2023-06-10T18:59:06.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.292+0000 7f024d772700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f02480ce390 0x7f0248160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f024d772700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f024806c720 0x7f02480ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f024d772700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480d0340 con 0x7f024806b410 2023-06-10T18:59:06.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f024d772700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02480d1660 con 0x7f02480ce390 2023-06-10T18:59:06.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f024d772700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024815cb60 con 0x7f024806c720 2023-06-10T18:59:06.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f02480ce390 0x7f0248160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f02480ce390 0x7f0248160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41034/0 (socket says 172.21.15.104:41034) 2023-06-10T18:59:06.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 -- 172.21.15.104:0/2438214403 learned_addr learned my addr 172.21.15.104:0/2438214403 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:06.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f02477fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f024806b410 0x7f024806b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38866/0 (socket says 172.21.15.104:38866) 2023-06-10T18:59:06.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f02467fc700 1 --2- 172.21.15.104:0/2438214403 >> v2:172.21.15.114:3300/0 conn(0x7f024806c720 0x7f02480ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 -- 172.21.15.104:0/2438214403 >> v2:172.21.15.114:3300/0 conn(0x7f024806c720 msgr2=0x7f02480ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:06.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 --2- 172.21.15.104:0/2438214403 >> v2:172.21.15.114:3300/0 conn(0x7f024806c720 0x7f02480ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 -- 172.21.15.104:0/2438214403 >> v1:172.21.15.104:6789/0 conn(0x7f024806b410 legacy=0x7f024806b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 -- 172.21.15.104:0/2438214403 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248161230 con 0x7f02480ce390 2023-06-10T18:59:06.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0245ffb700 1 -- 172.21.15.104:0/2438214403 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200330335 0 0) 0x7f02480d0340 con 0x7f024806b410 2023-06-10T18:59:06.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0246ffd700 1 --2- 172.21.15.104:0/2438214403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02480ce390 0x7f0248160cf0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f024806ab00 tx=0x7f0238009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2ea37d19258cd9 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0245ffb700 1 -- 172.21.15.104:0/2438214403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238011830 con 0x7f02480ce390 2023-06-10T18:59:06.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0245ffb700 1 -- 172.21.15.104:0/2438214403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238011990 con 0x7f02480ce390 2023-06-10T18:59:06.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.293+0000 7f0245ffb700 1 -- 172.21.15.104:0/2438214403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238011d20 con 0x7f02480ce390 2023-06-10T18:59:06.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 -- 172.21.15.104:0/2438214403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02480ce390 msgr2=0x7f0248160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 --2- 172.21.15.104:0/2438214403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02480ce390 0x7f0248160cf0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f024806ab00 tx=0x7f0238009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 -- 172.21.15.104:0/2438214403 shutdown_connections 2023-06-10T18:59:06.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 --2- 172.21.15.104:0/2438214403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02480ce390 0x7f0248160cf0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 --2- 172.21.15.104:0/2438214403 >> v2:172.21.15.114:3300/0 conn(0x7f024806c720 0x7f02480ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 -- 172.21.15.104:0/2438214403 >> 172.21.15.104:0/2438214403 conn(0x7f0248069e70 msgr2=0x7f024806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:06.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 -- 172.21.15.104:0/2438214403 shutdown_connections 2023-06-10T18:59:06.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.294+0000 7f024d772700 1 -- 172.21.15.104:0/2438214403 wait complete. 2023-06-10T18:59:06.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.295+0000 7f024d772700 1 Processor -- start 2023-06-10T18:59:06.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.295+0000 7f024d772700 1 -- start start 2023-06-10T18:59:06.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 0x7f02480b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02480ce390 0x7f02480b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480b3930 con 0x7f024806b410 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480c48d0 con 0x7f024806c720 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f024d772700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480c4bb0 con 0x7f02480ce390 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41040/0 (socket says 172.21.15.104:41040) 2023-06-10T18:59:06.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 -- 172.21.15.104:0/2727622791 learned_addr learned my addr 172.21.15.104:0/2727622791 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f02467fc700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 0x7f02480b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f02477fe700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02480ce390 0x7f02480b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 -- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02480ce390 msgr2=0x7f02480b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02480ce390 0x7f02480b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 -- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 msgr2=0x7f02480b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 0x7f02480b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 -- 172.21.15.104:0/2727622791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238009550 con 0x7f024806b410 2023-06-10T18:59:06.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f02467fc700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 0x7f02480b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:06.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.296+0000 7f0246ffd700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f0238009fd0 tx=0x7f0238004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02380036a0 con 0x7f024806b410 2023-06-10T18:59:06.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023801f070 con 0x7f024806b410 2023-06-10T18:59:06.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02480c4e90 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238003a20 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480c5450 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.297+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0248074050 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.298+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0238003b80 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.298+0000 7f022f7fe700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 0x7f023406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.298+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f0238021070 con 0x7f024806b410 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.298+0000 7f02467fc700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 0x7f023406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.299+0000 7f02467fc700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 0x7f023406d190 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f02480b0f10 tx=0x7f023c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.303+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0238055ee0 con 0x7f024806b410 2023-06-10T18:59:06.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.503+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f024806b1f0 con 0x7f024806b410 2023-06-10T18:59:06.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.503+0000 7f022f7fe700 1 -- 172.21.15.104:0/2727622791 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v175) v1 ==== 80+0+56 (secure 0 0 0) 0x7f0238015020 con 0x7f024806b410 2023-06-10T18:59:06.512 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/827471637 2023-06-10T19:58:50.352178+0000 2023-06-10T18:59:06.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 msgr2=0x7f023406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 0x7f023406d190 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f02480b0f10 tx=0x7f023c008040 comp rx=0 tx=0).stop 2023-06-10T18:59:06.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 msgr2=0x7f02480c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f0238009fd0 tx=0x7f0238004750 comp rx=0 tx=0).stop 2023-06-10T18:59:06.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 shutdown_connections 2023-06-10T18:59:06.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02480ce390 0x7f02480b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f024806b410 0x7f02480c42f0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f023406ace0 0x7f023406d190 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 --2- 172.21.15.104:0/2727622791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f024806c720 0x7f02480b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 >> 172.21.15.104:0/2727622791 conn(0x7f0248069e70 msgr2=0x7f02480b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:06.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 shutdown_connections 2023-06-10T18:59:06.517 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.505+0000 7f024d772700 1 -- 172.21.15.104:0/2727622791 wait complete. 2023-06-10T18:59:06.517 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T18:59:06.523 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T18:59:06.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:06.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:06.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[160529]: fuse finished with error 0 and tester_r 0 2023-06-10T18:59:06.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:59:06.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:06.559 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:59:06.615 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:59:06.616 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:06.637 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:59:06.637 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:59:06.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:06.638 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T18:59:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:59:06.697 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-06-10T18:59:06.697 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:06.698 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:06.714 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T18:59:06.715 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:59:06.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:06.715 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T18:59:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:59:06.772 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T18:59:06.772 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:59:06.774 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:06.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:59:06.795 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T18:59:06.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:06.796 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T18:59:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T18:59:06.856 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T18:59:06.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:06.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 Processor -- start 2023-06-10T18:59:06.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 -- start start 2023-06-10T18:59:06.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 0x7f38b40ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b406dcb0 con 0x7f38b415c500 2023-06-10T18:59:06.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b406e850 con 0x7f38b40cf030 2023-06-10T18:59:06.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.972+0000 7f38bb39a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b406a830 con 0x7f38b4069e70 2023-06-10T18:59:06.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b8935700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 0x7f38b40ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9136700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f38b415c500 0x7f38b406e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44722/0 (socket says 172.21.15.104:44722) 2023-06-10T18:59:06.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60436/0 (socket says 172.21.15.104:60436) 2023-06-10T18:59:06.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9136700 1 -- 172.21.15.104:0/2447758531 learned_addr learned my addr 172.21.15.104:0/2447758531 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 -- 172.21.15.104:0/2447758531 >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 msgr2=0x7f38b40ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 --2- 172.21.15.104:0/2447758531 >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 0x7f38b40ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 -- 172.21.15.104:0/2447758531 >> v1:172.21.15.114:6789/0 conn(0x7f38b415c500 legacy=0x7f38b406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 -- 172.21.15.104:0/2447758531 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b40cf460 con 0x7f38b4069e70 2023-06-10T18:59:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b8935700 1 --2- 172.21.15.104:0/2447758531 >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 0x7f38b40ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.973+0000 7f38b9937700 1 --2- 172.21.15.104:0/2447758531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009d90 tx=0x7f38a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22df47e856572821 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38b3fff700 1 -- 172.21.15.104:0/2447758531 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a400e070 con 0x7f38b4069e70 2023-06-10T18:59:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38b3fff700 1 -- 172.21.15.104:0/2447758531 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a400bb20 con 0x7f38b4069e70 2023-06-10T18:59:06.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38b3fff700 1 -- 172.21.15.104:0/2447758531 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4012430 con 0x7f38b4069e70 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 -- 172.21.15.104:0/2447758531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 msgr2=0x7f38b406a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 --2- 172.21.15.104:0/2447758531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009d90 tx=0x7f38a400b330 comp rx=0 tx=0).stop 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 -- 172.21.15.104:0/2447758531 shutdown_connections 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 --2- 172.21.15.104:0/2447758531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b4069e70 0x7f38b406a2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 --2- 172.21.15.104:0/2447758531 >> v2:172.21.15.114:3300/0 conn(0x7f38b40cf030 0x7f38b40ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 -- 172.21.15.104:0/2447758531 >> 172.21.15.104:0/2447758531 conn(0x7f38b406b8a0 msgr2=0x7f38b40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 -- 172.21.15.104:0/2447758531 shutdown_connections 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.974+0000 7f38bb39a700 1 -- 172.21.15.104:0/2447758531 wait complete. 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.975+0000 7f38bb39a700 1 Processor -- start 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.975+0000 7f38bb39a700 1 -- start start 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 0x7f38b40b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b415c500 0x7f38b409cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40b1110 con 0x7f38b4069e70 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41552f0 con 0x7f38b40cf030 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38bb39a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40b0490 con 0x7f38b415c500 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41054/0 (socket says 172.21.15.104:41054) 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 -- 172.21.15.104:0/868941381 learned_addr learned my addr 172.21.15.104:0/868941381 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b8935700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 0x7f38b40b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9937700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b415c500 0x7f38b409cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 -- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b415c500 msgr2=0x7f38b409cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b415c500 0x7f38b409cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 -- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 msgr2=0x7f38b40b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 0x7f38b40b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b9136700 1 -- 172.21.15.104:0/868941381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80094a0 con 0x7f38b4069e70 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.976+0000 7f38b8935700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 0x7f38b40b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38b9136700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00c490 tx=0x7f38ac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac005dc0 con 0x7f38b4069e70 2023-06-10T18:59:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac0070d0 con 0x7f38b4069e70 2023-06-10T18:59:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a4009a70 con 0x7f38b4069e70 2023-06-10T18:59:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac007460 con 0x7f38b4069e70 2023-06-10T18:59:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.977+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b40b0a50 con 0x7f38b4069e70 2023-06-10T18:59:06.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.978+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b4074050 con 0x7f38b4069e70 2023-06-10T18:59:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.978+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f38ac0075c0 con 0x7f38b4069e70 2023-06-10T18:59:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.979+0000 7f38b1ffb700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 0x7f38a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:06.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.979+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f38ac088af0 con 0x7f38b4069e70 2023-06-10T18:59:06.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.979+0000 7f38b8935700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 0x7f38a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:06.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.979+0000 7f38b8935700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 0x7f38a006d190 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f38b41f5180 tx=0x7f38a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:06.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:06.983+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f38ac053a80 con 0x7f38b4069e70 2023-06-10T18:59:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.214+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38b415c020 con 0x7f38b4069e70 2023-06-10T18:59:07.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.214+0000 7f38b1ffb700 1 -- 172.21.15.104:0/868941381 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f38ac017090 con 0x7f38b4069e70 2023-06-10T18:59:07.221 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:07.221 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 msgr2=0x7f38a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 0x7f38a006d190 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f38b41f5180 tx=0x7f38a8008040 comp rx=0 tx=0).stop 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 msgr2=0x7f38b4154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00c490 tx=0x7f38ac00c850 comp rx=0 tx=0).stop 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 shutdown_connections 2023-06-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38b415c500 0x7f38b409cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38b4069e70 0x7f38b4154db0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f38a006ace0 0x7f38a006d190 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 --2- 172.21.15.104:0/868941381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b40cf030 0x7f38b40b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 >> 172.21.15.104:0/868941381 conn(0x7f38b406b8a0 msgr2=0x7f38b40ae2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:07.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 shutdown_connections 2023-06-10T18:59:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.216+0000 7f38bb39a700 1 -- 172.21.15.104:0/868941381 wait complete. 2023-06-10T18:59:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:59:07.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:07.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 Processor -- start 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 -- start start 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4080ade00 0x7fe4080ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080aeb00 0x7fe4080ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080cee70 con 0x7fe4080ada30 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4080cf430 con 0x7fe4080ade00 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40d196700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080cf150 con 0x7fe4080aeb00 2023-06-10T18:59:07.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4080ade00 0x7fe4080ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe4080ade00 0x7fe4080ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41064/0 (socket says 172.21.15.104:41064) 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.350+0000 7fe40659c700 1 -- 172.21.15.104:0/3561069890 learned_addr learned my addr 172.21.15.104:0/3561069890 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40759e700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080aeb00 0x7fe4080ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40659c700 1 -- 172.21.15.104:0/3561069890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080aeb00 msgr2=0x7fe4080ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40659c700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080aeb00 0x7fe4080ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40659c700 1 -- 172.21.15.104:0/3561069890 >> v1:172.21.15.114:6789/0 conn(0x7fe4080ada30 legacy=0x7fe4080ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40659c700 1 -- 172.21.15.104:0/3561069890 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40806cf10 con 0x7fe4080ade00 2023-06-10T18:59:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe40659c700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080ade00 0x7fe4080ce0b0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80098f0 tx=0x7fe3f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95356a85773289 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe405d9b700 1 -- 172.21.15.104:0/3561069890 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800b8a0 con 0x7fe4080ade00 2023-06-10T18:59:07.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe405d9b700 1 -- 172.21.15.104:0/3561069890 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f800ba00 con 0x7fe4080ade00 2023-06-10T18:59:07.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.351+0000 7fe405d9b700 1 -- 172.21.15.104:0/3561069890 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800bd90 con 0x7fe4080ade00 2023-06-10T18:59:07.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 -- 172.21.15.104:0/3561069890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080ade00 msgr2=0x7fe4080ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080ade00 0x7fe4080ce0b0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80098f0 tx=0x7fe3f8009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 -- 172.21.15.104:0/3561069890 shutdown_connections 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080aeb00 0x7fe4080ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 --2- 172.21.15.104:0/3561069890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080ade00 0x7fe4080ce0b0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 -- 172.21.15.104:0/3561069890 >> 172.21.15.104:0/3561069890 conn(0x7fe40806b8a0 msgr2=0x7fe4080cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 -- 172.21.15.104:0/3561069890 shutdown_connections 2023-06-10T18:59:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.352+0000 7fe40d196700 1 -- 172.21.15.104:0/3561069890 wait complete. 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.353+0000 7fe40d196700 1 Processor -- start 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.353+0000 7fe40d196700 1 -- start start 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080ada30 0x7fe40815a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 0x7fe4080c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c98d0 con 0x7fe4080aeb00 2023-06-10T18:59:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c66d0 con 0x7fe408159ca0 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c69b0 con 0x7fe4080ada30 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41070/0 (socket says 172.21.15.104:41070) 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 -- 172.21.15.104:0/2750546425 learned_addr learned my addr 172.21.15.104:0/2750546425 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40759e700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 0x7fe4080c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 -- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080ada30 msgr2=0x7fe40815a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080ada30 0x7fe40815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 -- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 msgr2=0x7fe4080c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 0x7fe4080c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 -- 172.21.15.104:0/2750546425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8009550 con 0x7fe4080aeb00 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40759e700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 0x7fe4080c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40659c700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8004480 con 0x7fe4080aeb00 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f801f070 con 0x7fe4080aeb00 2023-06-10T18:59:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.354+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4080c6c90 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.355+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f80038c0 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.355+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4080c7250 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.355+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe408066f70 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.356+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe3f8021030 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.356+0000 7fe3ef7fe700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 0x7fe3f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.356+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe3f808ac30 con 0x7fe4080aeb00 2023-06-10T18:59:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.357+0000 7fe406d9d700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 0x7fe3f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.357+0000 7fe406d9d700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 0x7fe3f406d190 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc000fb0 tx=0x7fe3fc007a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.362+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe3f8055aa0 con 0x7fe4080aeb00 2023-06-10T18:59:07.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.551+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe408074050 con 0x7fe4080aeb00 2023-06-10T18:59:07.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.552+0000 7fe3ef7fe700 1 -- 172.21.15.104:0/2750546425 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+18714 (secure 0 0 0) 0x7fe3f80590c0 con 0x7fe4080aeb00 2023-06-10T18:59:07.559 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:07.559 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":175,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:54.747698+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.554+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 msgr2=0x7fe3f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.554+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 0x7fe3f406d190 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc000fb0 tx=0x7fe3fc007a30 comp rx=0 tx=0).stop 2023-06-10T18:59:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.554+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 msgr2=0x7fe4080c9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.554+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f8004750 comp rx=0 tx=0).stop 2023-06-10T18:59:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 shutdown_connections 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe3f406ace0 0x7fe3f406d190 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe408159ca0 0x7fe4080c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4080ada30 0x7fe40815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 --2- 172.21.15.104:0/2750546425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4080aeb00 0x7fe4080c9390 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 >> 172.21.15.104:0/2750546425 conn(0x7fe40806b8a0 msgr2=0x7fe4080b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 shutdown_connections 2023-06-10T18:59:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.555+0000 7fe40d196700 1 -- 172.21.15.104:0/2750546425 wait complete. 2023-06-10T18:59:07.573 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:58:27.026358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:58:28.306697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:59:07.574 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:59:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.690+0000 7fabf6954700 1 Processor -- start 2023-06-10T18:59:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.690+0000 7fabf6954700 1 -- start start 2023-06-10T18:59:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabf6954700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf0143a40 0x7fabf006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabf6954700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 0x7fabf006cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabf6954700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf00cec40 con 0x7fabf0070a10 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabf6954700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf00cca90 con 0x7fabf0143a40 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabf6954700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00ce960 con 0x7fabf006c800 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabef7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf0143a40 0x7fabf006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabef7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fabf0143a40 0x7fabf006e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41078/0 (socket says 172.21.15.104:41078) 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.691+0000 7fabef7fe700 1 -- 172.21.15.104:0/878544850 learned_addr learned my addr 172.21.15.104:0/878544850 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabf4ef1700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 0x7fabf006cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabef7fe700 1 -- 172.21.15.104:0/878544850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 msgr2=0x7fabf006cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabef7fe700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 0x7fabf006cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabef7fe700 1 -- 172.21.15.104:0/878544850 >> v1:172.21.15.114:6789/0 conn(0x7fabf0070a10 legacy=0x7fabf0070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabef7fe700 1 -- 172.21.15.104:0/878544850 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf006a9b0 con 0x7fabf0143a40 2023-06-10T18:59:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabf4ef1700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 0x7fabf006cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:07.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabef7fe700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf0143a40 0x7fabf006e560 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fabe00098f0 tx=0x7fabe0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b9ee81f6cc3303 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabeeffd700 1 -- 172.21.15.104:0/878544850 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe000b8a0 con 0x7fabf0143a40 2023-06-10T18:59:07.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabeeffd700 1 -- 172.21.15.104:0/878544850 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe000ba00 con 0x7fabf0143a40 2023-06-10T18:59:07.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.692+0000 7fabeeffd700 1 -- 172.21.15.104:0/878544850 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe000bd90 con 0x7fabf0143a40 2023-06-10T18:59:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 -- 172.21.15.104:0/878544850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf0143a40 msgr2=0x7fabf006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf0143a40 0x7fabf006e560 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fabe00098f0 tx=0x7fabe0009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 -- 172.21.15.104:0/878544850 shutdown_connections 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf006c800 0x7fabf006cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 --2- 172.21.15.104:0/878544850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf0143a40 0x7fabf006e560 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 -- 172.21.15.104:0/878544850 >> 172.21.15.104:0/878544850 conn(0x7fabf00ceec0 msgr2=0x7fabf00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 -- 172.21.15.104:0/878544850 shutdown_connections 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.693+0000 7fabf6954700 1 -- 172.21.15.104:0/878544850 wait complete. 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.694+0000 7fabf6954700 1 Processor -- start 2023-06-10T18:59:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.694+0000 7fabf6954700 1 -- start start 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf0070a10 0x7fabf0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 0x7fabf00ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00b4d80 con 0x7fabf006c800 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00b4200 con 0x7fabf0070a10 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00b44e0 con 0x7fabf0143a40 2023-06-10T18:59:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41082/0 (socket says 172.21.15.104:41082) 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 -- 172.21.15.104:0/1367600601 learned_addr learned my addr 172.21.15.104:0/1367600601 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabef7fe700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf0070a10 0x7fabf0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf4ef1700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 0x7fabf00ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 -- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 msgr2=0x7fabf00ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 0x7fabf00ac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.707 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 -- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf0070a10 msgr2=0x7fabf0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf0070a10 0x7fabf0159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 -- 172.21.15.104:0/1367600601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0009550 con 0x7fabf006c800 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf4ef1700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 0x7fabf00ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabeffff700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fabd800c4e0 tx=0x7fabd800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd80052d0 con 0x7fabf006c800 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd8005430 con 0x7fabf006c800 2023-06-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf00cd0a0 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.695+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd80038c0 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.696+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf00b4aa0 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.696+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf0074050 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.697+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fabd8003a20 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.697+0000 7fabecff9700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 0x7fabd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.697+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fabd8088a10 con 0x7fabf006c800 2023-06-10T18:59:07.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.698+0000 7fabef7fe700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 0x7fabd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:07.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.698+0000 7fabef7fe700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 0x7fabd006d190 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009c40 tx=0x7fabe001d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:07.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.702+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fabd8053910 con 0x7fabf006c800 2023-06-10T18:59:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.892+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabf006b030 con 0x7fabf006c800 2023-06-10T18:59:07.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.892+0000 7fabecff9700 1 -- 172.21.15.104:0/1367600601 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabd8017020 con 0x7fabf006c800 2023-06-10T18:59:07.899 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:59:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 msgr2=0x7fabd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 0x7fabd006d190 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009c40 tx=0x7fabe001d040 comp rx=0 tx=0).stop 2023-06-10T18:59:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 msgr2=0x7fabf0159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fabd800c4e0 tx=0x7fabd800c7f0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 shutdown_connections 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabf0143a40 0x7fabf00ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fabf006c800 0x7fabf0159710 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fabd006ace0 0x7fabd006d190 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 --2- 172.21.15.104:0/1367600601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf0070a10 0x7fabf0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.894+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 >> 172.21.15.104:0/1367600601 conn(0x7fabf00ceec0 msgr2=0x7fabf00cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.895+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 shutdown_connections 2023-06-10T18:59:07.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:07.895+0000 7fabf6954700 1 -- 172.21.15.104:0/1367600601 wait complete. 2023-06-10T18:59:07.912 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:59:08.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.029+0000 7fe97cefe700 1 Processor -- start 2023-06-10T18:59:08.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.029+0000 7fe97cefe700 1 -- start start 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97cefe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9780b1780 0x7fe97806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97cefe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97806ea70 0x7fe97806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97cefe700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780ae7e0 con 0x7fe97815c6f0 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97cefe700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97806df90 con 0x7fe9780b1780 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97cefe700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97806e270 con 0x7fe97806ea70 2023-06-10T18:59:08.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe9780b1780 0x7fe97806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97659c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe97815c6f0 0x7fe9780b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38918/0 (socket says 172.21.15.104:38918) 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97659c700 1 -- 172.21.15.104:0/2137572975 learned_addr learned my addr 172.21.15.104:0/2137572975 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe976d9d700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97806ea70 0x7fe97806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 -- 172.21.15.104:0/2137572975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97806ea70 msgr2=0x7fe97806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97806ea70 0x7fe97806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 -- 172.21.15.104:0/2137572975 >> v1:172.21.15.104:6789/0 conn(0x7fe97815c6f0 legacy=0x7fe9780b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97559a700 1 -- 172.21.15.104:0/2137572975 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409437868 0 0) 0x7fe9780ae7e0 con 0x7fe97815c6f0 2023-06-10T18:59:08.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 -- 172.21.15.104:0/2137572975 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97806e550 con 0x7fe9780b1780 2023-06-10T18:59:08.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe975d9b700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe97806cde0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7fe96000bae0 tx=0x7fe96000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48390ace1105da3a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97559a700 1 -- 172.21.15.104:0/2137572975 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96000dd30 con 0x7fe9780b1780 2023-06-10T18:59:08.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.031+0000 7fe97559a700 1 -- 172.21.15.104:0/2137572975 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96000de90 con 0x7fe9780b1780 2023-06-10T18:59:08.039 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97559a700 1 -- 172.21.15.104:0/2137572975 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960010690 con 0x7fe9780b1780 2023-06-10T18:59:08.041 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 -- 172.21.15.104:0/2137572975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 msgr2=0x7fe97806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe97806cde0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7fe96000bae0 tx=0x7fe96000bdf0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 -- 172.21.15.104:0/2137572975 shutdown_connections 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97806ea70 0x7fe97806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 --2- 172.21.15.104:0/2137572975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe97806cde0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 -- 172.21.15.104:0/2137572975 >> 172.21.15.104:0/2137572975 conn(0x7fe97806b8a0 msgr2=0x7fe9780cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 -- 172.21.15.104:0/2137572975 shutdown_connections 2023-06-10T18:59:08.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.032+0000 7fe97cefe700 1 -- 172.21.15.104:0/2137572975 wait complete. 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.033+0000 7fe97cefe700 1 Processor -- start 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 -- start start 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe97806ea70 0x7fe978152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97815c6f0 0x7fe9780cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780cc850 con 0x7fe9780b1780 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780c21f0 con 0x7fe97806ea70 2023-06-10T18:59:08.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe97cefe700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780c24d0 con 0x7fe97815c6f0 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe975d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe975d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41098/0 (socket says 172.21.15.104:41098) 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.034+0000 7fe975d9b700 1 -- 172.21.15.104:0/1993490178 learned_addr learned my addr 172.21.15.104:0/1993490178 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe97659c700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe97806ea70 0x7fe978152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe976d9d700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97815c6f0 0x7fe9780cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 -- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97815c6f0 msgr2=0x7fe9780cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97815c6f0 0x7fe9780cf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 -- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe97806ea70 msgr2=0x7fe978152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe97806ea70 0x7fe978152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 -- 172.21.15.104:0/1993490178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96000b740 con 0x7fe9780b1780 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe975d9b700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fe96000b2c0 tx=0x7fe960004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960003da0 con 0x7fe9780b1780 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe960010470 con 0x7fe9780b1780 2023-06-10T18:59:08.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9780c27b0 con 0x7fe9780b1780 2023-06-10T18:59:08.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960010820 con 0x7fe9780b1780 2023-06-10T18:59:08.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.035+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978153320 con 0x7fe9780b1780 2023-06-10T18:59:08.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.036+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9781553a0 con 0x7fe9780b1780 2023-06-10T18:59:08.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.036+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe960023440 con 0x7fe9780b1780 2023-06-10T18:59:08.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.037+0000 7fe96effd700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 0x7fe95806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.037+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe96008ab10 con 0x7fe9780b1780 2023-06-10T18:59:08.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.037+0000 7fe97659c700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 0x7fe95806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.037+0000 7fe97659c700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 0x7fe95806d190 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe968001750 tx=0x7fe968009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.047 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.041+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe960055aa0 con 0x7fe9780b1780 2023-06-10T18:59:08.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.226+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe9780b2700 con 0x7fe9780b1780 2023-06-10T18:59:08.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.227+0000 7fe96effd700 1 -- 172.21.15.104:0/1993490178 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v175) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe9780b2700 con 0x7fe9780b1780 2023-06-10T18:59:08.232 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 msgr2=0x7fe95806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 0x7fe95806d190 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe968001750 tx=0x7fe968009040 comp rx=0 tx=0).stop 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 msgr2=0x7fe9780cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fe96000b2c0 tx=0x7fe960004750 comp rx=0 tx=0).stop 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 shutdown_connections 2023-06-10T18:59:08.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe95806ace0 0x7fe95806d190 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe97806ea70 0x7fe978152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe97815c6f0 0x7fe9780cf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 --2- 172.21.15.104:0/1993490178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9780b1780 0x7fe9780cf230 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 >> 172.21.15.104:0/1993490178 conn(0x7fe97806b8a0 msgr2=0x7fe97815b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 shutdown_connections 2023-06-10T18:59:08.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.229+0000 7fe97cefe700 1 -- 172.21.15.104:0/1993490178 wait complete. 2023-06-10T18:59:08.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:59:08.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.365+0000 7f0dba6a0700 1 Processor -- start 2023-06-10T18:59:08.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.365+0000 7f0dba6a0700 1 -- start start 2023-06-10T18:59:08.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0dba6a0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0dba6a0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0db406cde0 0x7f0db40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0dba6a0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db406e250 con 0x7f0db406e6e0 2023-06-10T18:59:08.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0dba6a0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db415c950 con 0x7f0db406cde0 2023-06-10T18:59:08.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0dba6a0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40b1e80 con 0x7f0db415c580 2023-06-10T18:59:08.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db3fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db8c3d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0db406cde0 0x7f0db40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db3fff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60488/0 (socket says 172.21.15.104:60488) 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db3fff700 1 -- 172.21.15.104:0/4066986698 learned_addr learned my addr 172.21.15.104:0/4066986698 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db37fe700 1 --1- 172.21.15.104:0/4066986698 >> v1:172.21.15.114:6789/0 conn(0x7f0db406e6e0 0x7f0db40b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44778/0 (socket says 172.21.15.104:44778) 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db8c3d700 1 -- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 msgr2=0x7f0db40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db8c3d700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.366+0000 7f0db8c3d700 1 -- 172.21.15.104:0/4066986698 >> v1:172.21.15.114:6789/0 conn(0x7f0db406e6e0 legacy=0x7f0db40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db8c3d700 1 -- 172.21.15.104:0/4066986698 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db406d920 con 0x7f0db406cde0 2023-06-10T18:59:08.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db3fff700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:08.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db8c3d700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db406cde0 0x7f0db40ce0d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009d90 tx=0x7f0dac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6272d9853a839cac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db2ffd700 1 -- 172.21.15.104:0/4066986698 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00e070 con 0x7f0db406cde0 2023-06-10T18:59:08.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db2ffd700 1 -- 172.21.15.104:0/4066986698 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac00bb20 con 0x7f0db406cde0 2023-06-10T18:59:08.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0db2ffd700 1 -- 172.21.15.104:0/4066986698 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac012430 con 0x7f0db406cde0 2023-06-10T18:59:08.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 -- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db406cde0 msgr2=0x7f0db40ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db406cde0 0x7f0db40ce0d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009d90 tx=0x7f0dac00b330 comp rx=0 tx=0).stop 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 -- 172.21.15.104:0/4066986698 shutdown_connections 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db406cde0 0x7f0db40ce0d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/4066986698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db415c580 0x7f0db40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 -- 172.21.15.104:0/4066986698 >> 172.21.15.104:0/4066986698 conn(0x7f0db406b8a0 msgr2=0x7f0db40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 -- 172.21.15.104:0/4066986698 shutdown_connections 2023-06-10T18:59:08.378 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.367+0000 7f0dba6a0700 1 -- 172.21.15.104:0/4066986698 wait complete. 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 Processor -- start 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 -- start start 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db406cde0 0x7f0db414a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 0x7f0db40c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4147750 con 0x7f0db406e6e0 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40ac1c0 con 0x7f0db415c580 2023-06-10T18:59:08.379 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.369+0000 7f0dba6a0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40ac3d0 con 0x7f0db406cde0 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db8c3d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 0x7f0db40c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41112/0 (socket says 172.21.15.104:41112) 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 -- 172.21.15.104:0/559122544 learned_addr learned my addr 172.21.15.104:0/559122544 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db3fff700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db406cde0 0x7f0db414a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 -- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db406cde0 msgr2=0x7f0db414a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.380 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db406cde0 0x7f0db414a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 -- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 msgr2=0x7f0db40c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 0x7f0db40c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 -- 172.21.15.104:0/559122544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4009500 con 0x7f0db406e6e0 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db8c3d700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 0x7f0db40c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db37fe700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c520 tx=0x7f0d9c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c005cb0 con 0x7f0db406e6e0 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c005e10 con 0x7f0db406e6e0 2023-06-10T18:59:08.381 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac009a70 con 0x7f0db406e6e0 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c0072d0 con 0x7f0db406e6e0 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.370+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db40ac990 con 0x7f0db406e6e0 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.371+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4074050 con 0x7f0db406e6e0 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.372+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0d9c007430 con 0x7f0db406e6e0 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.372+0000 7f0db0ff9700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 0x7f0d9406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.382 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.372+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f0d9c0888c0 con 0x7f0db406e6e0 2023-06-10T18:59:08.383 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.372+0000 7f0db3fff700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 0x7f0d9406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.373+0000 7f0db3fff700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 0x7f0d9406d160 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0da4006fd0 tx=0x7f0da4009290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.384 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.376+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0d9c053870 con 0x7f0db406e6e0 2023-06-10T18:59:08.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.562+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0db40cb2c0 con 0x7f0db406e6e0 2023-06-10T18:59:08.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.563+0000 7f0db0ff9700 1 -- 172.21.15.104:0/559122544 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v175) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0d9c056e90 con 0x7f0db406e6e0 2023-06-10T18:59:08.569 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:59:08.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 msgr2=0x7f0d9406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 0x7f0d9406d160 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0da4006fd0 tx=0x7f0da4009290 comp rx=0 tx=0).stop 2023-06-10T18:59:08.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 msgr2=0x7f0db41479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c520 tx=0x7f0d9c00c830 comp rx=0 tx=0).stop 2023-06-10T18:59:08.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 shutdown_connections 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0d9406acb0 0x7f0d9406d160 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db415c580 0x7f0db40c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0db406cde0 0x7f0db414a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 --2- 172.21.15.104:0/559122544 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0db406e6e0 0x7f0db41479c0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 >> 172.21.15.104:0/559122544 conn(0x7f0db406b8a0 msgr2=0x7f0db40b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 shutdown_connections 2023-06-10T18:59:08.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.565+0000 7f0dba6a0700 1 -- 172.21.15.104:0/559122544 wait complete. 2023-06-10T18:59:08.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:59:08.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.700+0000 7f2cea01a700 1 Processor -- start 2023-06-10T18:59:08.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.700+0000 7f2cea01a700 1 -- start start 2023-06-10T18:59:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2ce406ea70 0x7f2ce40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ce40aec00 0x7f2ce40ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40ae7e0 con 0x7f2ce40aec00 2023-06-10T18:59:08.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce406df90 con 0x7f2ce406ea70 2023-06-10T18:59:08.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2cea01a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce406e270 con 0x7f2ce415c6f0 2023-06-10T18:59:08.711 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ce40aec00 0x7f2ce40ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2ce40aec00 0x7f2ce40ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41122/0 (socket says 172.21.15.104:41122) 2023-06-10T18:59:08.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 -- 172.21.15.104:0/423428745 learned_addr learned my addr 172.21.15.104:0/423428745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:08.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce37fe700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.712 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce2ffd700 1 --2- 172.21.15.104:0/423428745 >> v2:172.21.15.114:3300/0 conn(0x7f2ce406ea70 0x7f2ce40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 -- 172.21.15.104:0/423428745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 msgr2=0x7f2ce40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 -- 172.21.15.104:0/423428745 >> v2:172.21.15.114:3300/0 conn(0x7f2ce406ea70 msgr2=0x7f2ce40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 --2- 172.21.15.104:0/423428745 >> v2:172.21.15.114:3300/0 conn(0x7f2ce406ea70 0x7f2ce40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.713 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 -- 172.21.15.104:0/423428745 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce406e550 con 0x7f2ce40aec00 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce3fff700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac1c0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4067240 tx=0x7f2cdc00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f835b1d1b987ad71 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce27fc700 1 -- 172.21.15.104:0/423428745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc010070 con 0x7f2ce40aec00 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce27fc700 1 -- 172.21.15.104:0/423428745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc00dd90 con 0x7f2ce40aec00 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.701+0000 7f2ce27fc700 1 -- 172.21.15.104:0/423428745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc00b310 con 0x7f2ce40aec00 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 -- 172.21.15.104:0/423428745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce40aec00 msgr2=0x7f2ce40ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.714 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac1c0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4067240 tx=0x7f2cdc00d430 comp rx=0 tx=0).stop 2023-06-10T18:59:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 -- 172.21.15.104:0/423428745 shutdown_connections 2023-06-10T18:59:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac1c0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.715 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 --2- 172.21.15.104:0/423428745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 --2- 172.21.15.104:0/423428745 >> v2:172.21.15.114:3300/0 conn(0x7f2ce406ea70 0x7f2ce40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 -- 172.21.15.104:0/423428745 >> 172.21.15.104:0/423428745 conn(0x7f2ce406b8a0 msgr2=0x7f2ce40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 -- 172.21.15.104:0/423428745 shutdown_connections 2023-06-10T18:59:08.716 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.702+0000 7f2cea01a700 1 -- 172.21.15.104:0/423428745 wait complete. 2023-06-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.703+0000 7f2cea01a700 1 Processor -- start 2023-06-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.703+0000 7f2cea01a700 1 -- start start 2023-06-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.717 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41eebf0 con 0x7f2ce406ea70 2023-06-10T18:59:08.718 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41eee90 con 0x7f2ce40aec00 2023-06-10T18:59:08.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2cea01a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41ef170 con 0x7f2ce415c6f0 2023-06-10T18:59:08.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41124/0 (socket says 172.21.15.104:41124) 2023-06-10T18:59:08.719 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 -- 172.21.15.104:0/3252013846 learned_addr learned my addr 172.21.15.104:0/3252013846 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce2ffd700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce3fff700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.720 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 -- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 msgr2=0x7f2ce40ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 -- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 msgr2=0x7f2ce40ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce37fe700 1 -- 172.21.15.104:0/3252013846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc00bd40 con 0x7f2ce406ea70 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.704+0000 7f2ce2ffd700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2ce37fe700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400d7d0 tx=0x7f2cd400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.721 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40125f0 con 0x7f2ce406ea70 2023-06-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4012750 con 0x7f2ce406ea70 2023-06-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce41ef450 con 0x7f2ce406ea70 2023-06-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4012b00 con 0x7f2ce406ea70 2023-06-10T18:59:08.722 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.705+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce40c9b90 con 0x7f2ce406ea70 2023-06-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.706+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce41ef7c0 con 0x7f2ce406ea70 2023-06-10T18:59:08.723 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.706+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2cd4012c60 con 0x7f2ce406ea70 2023-06-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.706+0000 7f2ccbfff700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 0x7f2cd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.706+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f2cd4089710 con 0x7f2ce406ea70 2023-06-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.707+0000 7f2ce2ffd700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 0x7f2cd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.707+0000 7f2ce2ffd700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 0x7f2cd006d190 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40c9ab0 tx=0x7f2cd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:08.724 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.711+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2cd40546a0 con 0x7f2ce406ea70 2023-06-10T18:59:08.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.899+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ce4074050 con 0x7f2ce406ea70 2023-06-10T18:59:08.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.900+0000 7f2ccbfff700 1 -- 172.21.15.104:0/3252013846 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v175) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2cd4057cc0 con 0x7f2ce406ea70 2023-06-10T18:59:08.906 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:59:08.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 msgr2=0x7f2cd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 0x7f2cd006d190 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40c9ab0 tx=0x7f2cd8008040 comp rx=0 tx=0).stop 2023-06-10T18:59:08.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 msgr2=0x7f2ce40cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:08.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400d7d0 tx=0x7f2cd400db90 comp rx=0 tx=0).stop 2023-06-10T18:59:08.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 shutdown_connections 2023-06-10T18:59:08.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2ce415c6f0 0x7f2ce40ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ce406ea70 0x7f2ce40cb590 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2cd006ace0 0x7f2cd006d190 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 --2- 172.21.15.104:0/3252013846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce40aec00 0x7f2ce40ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:08.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.901+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 >> 172.21.15.104:0/3252013846 conn(0x7f2ce406b8a0 msgr2=0x7f2ce4155850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:08.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.902+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 shutdown_connections 2023-06-10T18:59:08.910 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:08.902+0000 7f2cea01a700 1 -- 172.21.15.104:0/3252013846 wait complete. 2023-06-10T18:59:08.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:09.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.034+0000 7f4e48407700 1 Processor -- start 2023-06-10T18:59:09.040 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.034+0000 7f4e48407700 1 -- start start 2023-06-10T18:59:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e48407700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 0x7f4e4006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e48407700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e40070df0 0x7f4e401587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e48407700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4006ae00 con 0x7f4e40158ce0 2023-06-10T18:59:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e48407700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e40069e70 con 0x7f4e40070df0 2023-06-10T18:59:09.042 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e48407700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e401595f0 con 0x7f4e40070a20 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.035+0000 7f4e459a2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e40070df0 0x7f4e401587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e40070df0 0x7f4e401587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41132/0 (socket says 172.21.15.104:41132) 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 -- 172.21.15.104:0/3141311765 learned_addr learned my addr 172.21.15.104:0/3141311765 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e461a3700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 0x7f4e4006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 -- 172.21.15.104:0/3141311765 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 msgr2=0x7f4e4006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 0x7f4e4006a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 -- 172.21.15.104:0/3141311765 >> v1:172.21.15.114:6789/0 conn(0x7f4e40158ce0 legacy=0x7f4e40159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 -- 172.21.15.104:0/3141311765 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e401598d0 con 0x7f4e40070df0 2023-06-10T18:59:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e461a3700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 0x7f4e4006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e459a2700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e401587a0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0098f0 tx=0x7f4e3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15089ad1fbb68dee server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e451a1700 1 -- 172.21.15.104:0/3141311765 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00b8a0 con 0x7f4e40070df0 2023-06-10T18:59:09.044 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e451a1700 1 -- 172.21.15.104:0/3141311765 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c00ba00 con 0x7f4e40070df0 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.036+0000 7f4e451a1700 1 -- 172.21.15.104:0/3141311765 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00bd90 con 0x7f4e40070df0 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 -- 172.21.15.104:0/3141311765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 msgr2=0x7f4e401587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e401587a0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0098f0 tx=0x7f4e3c009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 -- 172.21.15.104:0/3141311765 shutdown_connections 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40070a20 0x7f4e4006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 --2- 172.21.15.104:0/3141311765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e401587a0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 -- 172.21.15.104:0/3141311765 >> 172.21.15.104:0/3141311765 conn(0x7f4e400b1610 msgr2=0x7f4e400ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 -- 172.21.15.104:0/3141311765 shutdown_connections 2023-06-10T18:59:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.037+0000 7f4e48407700 1 -- 172.21.15.104:0/3141311765 wait complete. 2023-06-10T18:59:09.048 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.038+0000 7f4e48407700 1 Processor -- start 2023-06-10T18:59:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.038+0000 7f4e48407700 1 -- start start 2023-06-10T18:59:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40070a20 0x7f4e40149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 0x7f4e400bd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.049 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400c24e0 con 0x7f4e40070df0 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400bde40 con 0x7f4e40070a20 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400be120 con 0x7f4e40158ce0 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41140/0 (socket says 172.21.15.104:41140) 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 -- 172.21.15.104:0/1628493504 learned_addr learned my addr 172.21.15.104:0/1628493504 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e461a3700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40070a20 0x7f4e40149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e469a4700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 0x7f4e400bd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.050 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 -- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 msgr2=0x7f4e400bd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 0x7f4e400bd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 -- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40070a20 msgr2=0x7f4e40149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40070a20 0x7f4e40149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 -- 172.21.15.104:0/1628493504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c009550 con 0x7f4e40070df0 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e469a4700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 0x7f4e400bd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e459a2700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c000c00 tx=0x7f4e3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.051 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c004380 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c01f070 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0036a0 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.039+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e400be400 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.040+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e400c8bd0 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.040+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e40071780 con 0x7f4e40070df0 2023-06-10T18:59:09.052 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.041+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4e3c021030 con 0x7f4e40070df0 2023-06-10T18:59:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.041+0000 7f4e36ffd700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 0x7f4e2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.041+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f4e3c08a900 con 0x7f4e40070df0 2023-06-10T18:59:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.042+0000 7f4e461a3700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 0x7f4e2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.053 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.042+0000 7f4e461a3700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 0x7f4e2406d190 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4e30006f10 tx=0x7f4e30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.045+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4e3c055890 con 0x7f4e40070df0 2023-06-10T18:59:09.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.275+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e400be590 con 0x7f4e40070df0 2023-06-10T18:59:09.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.275+0000 7f4e36ffd700 1 -- 172.21.15.104:0/1628493504 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f4e3c058eb0 con 0x7f4e40070df0 2023-06-10T18:59:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:09.282 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:59:09.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.277+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 msgr2=0x7f4e2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.277+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 0x7f4e2406d190 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4e30006f10 tx=0x7f4e30008040 comp rx=0 tx=0).stop 2023-06-10T18:59:09.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.277+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 msgr2=0x7f4e400bd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.277+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c000c00 tx=0x7f4e3c004750 comp rx=0 tx=0).stop 2023-06-10T18:59:09.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 shutdown_connections 2023-06-10T18:59:09.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e2406ace0 0x7f4e2406d190 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40070a20 0x7f4e40149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e40158ce0 0x7f4e400bd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 --2- 172.21.15.104:0/1628493504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e40070df0 0x7f4e400bd0b0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 >> 172.21.15.104:0/1628493504 conn(0x7f4e400b1610 msgr2=0x7f4e400aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:09.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 shutdown_connections 2023-06-10T18:59:09.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.278+0000 7f4e48407700 1 -- 172.21.15.104:0/1628493504 wait complete. 2023-06-10T18:59:09.287 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:59:09.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 Processor -- start 2023-06-10T18:59:09.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 -- start start 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5aa00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa00aed90 con 0x7f5aa00ae930 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa00cee70 con 0x7f5aa00ade00 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa67e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa00cf430 con 0x7f5aa00ada30 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.413+0000 7f5aa4d84700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5aa00ae930 0x7f5aa006d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38974/0 (socket says 172.21.15.104:38974) 2023-06-10T18:59:09.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5aa4d84700 1 -- 172.21.15.104:0/1007503442 learned_addr learned my addr 172.21.15.104:0/1007503442 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9ffff700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5aa00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961915247 0 0) 0x7f5aa00aed90 con 0x7f5aa00ae930 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c0035b0 con 0x7f5aa00ae930 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055234643 0 0) 0x7f5aa00cee70 con 0x7f5aa00ade00 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa00aed90 con 0x7f5aa00ade00 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1768550291 0 0) 0x7f5a8c0035b0 con 0x7f5aa00ae930 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 msgr2=0x7f5aa00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5aa00ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 >> v1:172.21.15.114:6789/0 conn(0x7f5aa00ade00 legacy=0x7f5aa00ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa00cf150 con 0x7f5aa00ae930 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f5a980019f0 con 0x7f5aa00ae930 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 >> v1:172.21.15.104:6789/0 conn(0x7f5aa00ae930 legacy=0x7f5aa006d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a8c003bc0 0x7f5a8c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5a8c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5a8c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa00cf430 con 0x7f5a8c004d90 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c00bb10 con 0x7f5a8c003bc0 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c00bcd0 con 0x7f5aa00ada30 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5aa4d84700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5a8c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9ffff700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a8c003bc0 0x7f5a8c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5a9f7fe700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5a8c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.414+0000 7f5aa4d84700 1 -- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 msgr2=0x7f5a8c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa00ada30 0x7f5a8c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 -- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a8c003bc0 msgr2=0x7f5a8c003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a8c003bc0 0x7f5a8c003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 -- 172.21.15.104:0/1007503442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0035b0 con 0x7f5a8c004d90 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5a8c005200 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800a7a0 tx=0x7f5a8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64af67bedd07234 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98012600 con 0x7f5a8c004d90 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98012780 con 0x7f5a8c004d90 2023-06-10T18:59:09.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5a9effd700 1 -- 172.21.15.104:0/1007503442 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98012b30 con 0x7f5a8c004d90 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 -- 172.21.15.104:0/1007503442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 msgr2=0x7f5a8c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/1007503442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5a8c005200 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800a7a0 tx=0x7f5a8c005880 comp rx=0 tx=0).stop 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa4d84700 1 -- 172.21.15.104:0/1007503442 reap_dead start 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 -- 172.21.15.104:0/1007503442 shutdown_connections 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 -- 172.21.15.104:0/1007503442 >> 172.21.15.104:0/1007503442 conn(0x7f5aa006b8a0 msgr2=0x7f5aa00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 -- 172.21.15.104:0/1007503442 shutdown_connections 2023-06-10T18:59:09.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.415+0000 7f5aa67e7700 1 -- 172.21.15.104:0/1007503442 wait complete. 2023-06-10T18:59:09.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 Processor -- start 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 -- start start 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 0x7f5aa015cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa0149590 0x7f5aa0149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.417+0000 7f5aa67e7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01565f0 con 0x7f5a8c004d90 2023-06-10T18:59:09.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5aa67e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa0149f40 con 0x7f5aa0149590 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5aa67e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa015d370 con 0x7f5aa015c920 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41158/0 (socket says 172.21.15.104:41158) 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 -- 172.21.15.104:0/2993172002 learned_addr learned my addr 172.21.15.104:0/2993172002 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9f7fe700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 0x7f5aa015cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5aa4d84700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa0149590 0x7f5aa0149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 -- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 msgr2=0x7f5aa015cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 0x7f5aa015cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 -- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa0149590 msgr2=0x7f5aa0149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa0149590 0x7f5aa0149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 -- 172.21.15.104:0/2993172002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa015d650 con 0x7f5a8c004d90 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9f7fe700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 0x7f5aa015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9ffff700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f5a880119f0 tx=0x7f5a88011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88005c30 con 0x7f5a8c004d90 2023-06-10T18:59:09.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a88005d90 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa015d930 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8800f950 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.418+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa00b40f0 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.419+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa0156760 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.419+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a8800fab0 con 0x7f5a8c004d90 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.420+0000 7f5a9cff9700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 0x7f5a8006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.420+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5a88089f90 con 0x7f5a8c004d90 2023-06-10T18:59:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.420+0000 7f5a9f7fe700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 0x7f5a8006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.421+0000 7f5a9f7fe700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 0x7f5a8006d1c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5aa015a990 tx=0x7f5a90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.424+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a880bc310 con 0x7f5a8c004d90 2023-06-10T18:59:09.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.611+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa0074050 con 0x7f5a8c004d90 2023-06-10T18:59:09.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.613+0000 7f5a9cff9700 1 -- 172.21.15.104:0/2993172002 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+18714 (secure 0 0 0) 0x7f5aa0156760 con 0x7f5a8c004d90 2023-06-10T18:59:09.619 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:09.620 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":175,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:54.747698+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 msgr2=0x7f5a8006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 0x7f5a8006d1c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5aa015a990 tx=0x7f5a90009040 comp rx=0 tx=0).stop 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 msgr2=0x7f5aa0159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f5a880119f0 tx=0x7f5a88011db0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 shutdown_connections 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a8006ad10 0x7f5a8006d1c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa0149590 0x7f5aa0149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8c004d90 0x7f5aa0159f90 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 --2- 172.21.15.104:0/2993172002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5aa015c920 0x7f5aa015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 >> 172.21.15.104:0/2993172002 conn(0x7f5aa006b8a0 msgr2=0x7f5aa00aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:09.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 shutdown_connections 2023-06-10T18:59:09.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.615+0000 7f5aa67e7700 1 -- 172.21.15.104:0/2993172002 wait complete. 2023-06-10T18:59:09.633 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T18:59:09.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:09.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:59:09.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:59:09.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.749+0000 7fc6bff71700 1 Processor -- start 2023-06-10T18:59:09.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.749+0000 7fc6bff71700 1 -- start start 2023-06-10T18:59:09.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bff71700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6b80cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bff71700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6b80ce350 0x7fc6b80cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bff71700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b80cadd0 con 0x7fc6b808a3d0 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bff71700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b80cfeb0 con 0x7fc6b80ce350 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bff71700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80cb0b0 con 0x7fc6b806b1f0 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6be50e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc6b808a3d0 0x7fc6b8160680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38998/0 (socket says 172.21.15.104:38998) 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6be50e700 1 -- 172.21.15.104:0/1143951911 learned_addr learned my addr 172.21.15.104:0/1143951911 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:09.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6b80cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bd50c700 1 --2- 172.21.15.104:0/1143951911 >> v2:172.21.15.114:3300/0 conn(0x7fc6b80ce350 0x7fc6b80cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607633802 0 0) 0x7fc6b80cadd0 con 0x7fc6b808a3d0 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.750+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a40035b0 con 0x7fc6b808a3d0 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826176232 0 0) 0x7fc6a40035b0 con 0x7fc6b808a3d0 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 msgr2=0x7fc6b80cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6b80cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 >> v2:172.21.15.114:3300/0 conn(0x7fc6b80ce350 msgr2=0x7fc6b80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 --2- 172.21.15.104:0/1143951911 >> v2:172.21.15.114:3300/0 conn(0x7fc6b80ce350 0x7fc6b80cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80cf550 con 0x7fc6b808a3d0 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc6b00019f0 con 0x7fc6b808a3d0 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 >> v1:172.21.15.104:6789/0 conn(0x7fc6b808a3d0 legacy=0x7fc6b8160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6a4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4003f80 0x7fc6a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6a4004d90 0x7fc6a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80cfeb0 con 0x7fc6a4004d90 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80cb0b0 con 0x7fc6a4003f80 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a400ba40 con 0x7fc6b806b1f0 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6a4004d90 0x7fc6a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bd50c700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4003f80 0x7fc6a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6a4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 -- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 msgr2=0x7fc6a4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b806b1f0 0x7fc6a4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 -- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4003f80 msgr2=0x7fc6a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4003f80 0x7fc6a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 -- 172.21.15.104:0/1143951911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a40035b0 con 0x7fc6a4004d90 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bd50c700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4003f80 0x7fc6a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:09.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6be50e700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6a4004d90 0x7fc6a4005200 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000a8d0 tx=0x7fc6a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257442e5d1d7708e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000a340 con 0x7fc6a4004d90 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b0012600 con 0x7fc6a4004d90 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.751+0000 7fc6bcd0b700 1 -- 172.21.15.104:0/1143951911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b00129b0 con 0x7fc6a4004d90 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 -- 172.21.15.104:0/1143951911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6a4004d90 msgr2=0x7fc6a4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 --2- 172.21.15.104:0/1143951911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6a4004d90 0x7fc6a4005200 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000a8d0 tx=0x7fc6a4005880 comp rx=0 tx=0).stop 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6be50e700 1 -- 172.21.15.104:0/1143951911 reap_dead start 2023-06-10T18:59:09.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 -- 172.21.15.104:0/1143951911 shutdown_connections 2023-06-10T18:59:09.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 -- 172.21.15.104:0/1143951911 >> 172.21.15.104:0/1143951911 conn(0x7fc6b8089fd0 msgr2=0x7fc6b80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:09.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 -- 172.21.15.104:0/1143951911 shutdown_connections 2023-06-10T18:59:09.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.752+0000 7fc6bff71700 1 -- 172.21.15.104:0/1143951911 wait complete. 2023-06-10T18:59:09.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.753+0000 7fc6bff71700 1 Processor -- start 2023-06-10T18:59:09.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.753+0000 7fc6bff71700 1 -- start start 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 0x7fc6b8149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 0x7fc6b80c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b81e5320 con 0x7fc6b80c4b90 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b81e5bc0 con 0x7fc6a4004d90 2023-06-10T18:59:09.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bff71700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80c5000 con 0x7fc6b80c41e0 2023-06-10T18:59:09.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41182/0 (socket says 172.21.15.104:41182) 2023-06-10T18:59:09.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 -- 172.21.15.104:0/3668555911 learned_addr learned my addr 172.21.15.104:0/3668555911 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:09.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bd50c700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 0x7fc6b80c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 0x7fc6b8149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 -- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 msgr2=0x7fc6b80c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 0x7fc6b80c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 -- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 msgr2=0x7fc6b8149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 0x7fc6b8149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6be50e700 1 -- 172.21.15.104:0/3668555911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80ca220 con 0x7fc6b80c4b90 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 0x7fc6b8149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.754+0000 7fc6bd50c700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 0x7fc6b80c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6be50e700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0012000 tx=0x7fc6b0001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000a340 con 0x7fc6b80c4b90 2023-06-10T18:59:09.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b00019c0 con 0x7fc6b80c4b90 2023-06-10T18:59:09.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b81e5570 con 0x7fc6b80c4b90 2023-06-10T18:59:09.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b001e640 con 0x7fc6b80c4b90 2023-06-10T18:59:09.772 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.755+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b8149fc0 con 0x7fc6b80c4b90 2023-06-10T18:59:09.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.756+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8074050 con 0x7fc6b80c4b90 2023-06-10T18:59:09.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.756+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc6b0019030 con 0x7fc6b80c4b90 2023-06-10T18:59:09.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.757+0000 7fc6ae7fc700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 0x7fc69c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:09.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.757+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc6b008afb0 con 0x7fc6b80c4b90 2023-06-10T18:59:09.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.757+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 0x7fc69c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:09.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.757+0000 7fc6bdd0d700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 0x7fc69c06d190 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009c60 tx=0x7fc6a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:09.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.761+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc6b0055fc0 con 0x7fc6b80c4b90 2023-06-10T18:59:09.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.993+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b81e5d00 con 0x7fc6b80c4b90 2023-06-10T18:59:09.999 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.994+0000 7fc6ae7fc700 1 -- 172.21.15.104:0/3668555911 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7fc6b0017020 con 0x7fc6b80c4b90 2023-06-10T18:59:10.000 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:10.000 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:59:10.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 msgr2=0x7fc69c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 0x7fc69c06d190 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009c60 tx=0x7fc6a8008040 comp rx=0 tx=0).stop 2023-06-10T18:59:10.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 msgr2=0x7fc6b81e6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0012000 tx=0x7fc6b0001360 comp rx=0 tx=0).stop 2023-06-10T18:59:10.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 shutdown_connections 2023-06-10T18:59:10.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc69c06ace0 0x7fc69c06d190 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6a4004d90 0x7fc6b8149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6b80c4b90 0x7fc6b81e6050 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 --2- 172.21.15.104:0/3668555911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6b80c41e0 0x7fc6b80c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 >> 172.21.15.104:0/3668555911 conn(0x7fc6b8089fd0 msgr2=0x7fc6b80ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 shutdown_connections 2023-06-10T18:59:10.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:09.996+0000 7fc6bff71700 1 -- 172.21.15.104:0/3668555911 wait complete. 2023-06-10T18:59:10.006 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:59:10.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.127+0000 7fdd32314700 1 Processor -- start 2023-06-10T18:59:10.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.127+0000 7fdd32314700 1 -- start start 2023-06-10T18:59:10.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd32314700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdd2c0ce510 0x7fdd2c0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd32314700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd32314700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c0cee20 con 0x7fdd2c070130 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd32314700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c0caa90 con 0x7fdd2c0705b0 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd32314700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c071fe0 con 0x7fdd2c0ce510 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd308b1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd308b1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41188/0 (socket says 172.21.15.104:41188) 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.128+0000 7fdd308b1700 1 -- 172.21.15.104:0/610060907 learned_addr learned my addr 172.21.15.104:0/610060907 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd2bfff700 1 --2- 172.21.15.104:0/610060907 >> v2:172.21.15.114:3300/0 conn(0x7fdd2c0ce510 0x7fdd2c0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd308b1700 1 -- 172.21.15.104:0/610060907 >> v2:172.21.15.114:3300/0 conn(0x7fdd2c0ce510 msgr2=0x7fdd2c0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd308b1700 1 --2- 172.21.15.104:0/610060907 >> v2:172.21.15.114:3300/0 conn(0x7fdd2c0ce510 0x7fdd2c0ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd308b1700 1 -- 172.21.15.104:0/610060907 >> v1:172.21.15.114:6789/0 conn(0x7fdd2c070130 legacy=0x7fdd2c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd308b1700 1 -- 172.21.15.104:0/610060907 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c0722c0 con 0x7fdd2c0705b0 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd308b1700 1 --2- 172.21.15.104:0/610060907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0cf550 tx=0x7fdd2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ed414eed03c50b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd2affd700 1 -- 172.21.15.104:0/610060907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2400e070 con 0x7fdd2c0705b0 2023-06-10T18:59:10.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd2affd700 1 -- 172.21.15.104:0/610060907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24003a20 con 0x7fdd2c0705b0 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.129+0000 7fdd2affd700 1 -- 172.21.15.104:0/610060907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24003dd0 con 0x7fdd2c0705b0 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 -- 172.21.15.104:0/610060907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c0705b0 msgr2=0x7fdd2c0cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 --2- 172.21.15.104:0/610060907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0cf550 tx=0x7fdd2400b330 comp rx=0 tx=0).stop 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 -- 172.21.15.104:0/610060907 shutdown_connections 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 --2- 172.21.15.104:0/610060907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c0cb4a0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 --2- 172.21.15.104:0/610060907 >> v2:172.21.15.114:3300/0 conn(0x7fdd2c0ce510 0x7fdd2c0ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 -- 172.21.15.104:0/610060907 >> 172.21.15.104:0/610060907 conn(0x7fdd2c06ab00 msgr2=0x7fdd2c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 -- 172.21.15.104:0/610060907 shutdown_connections 2023-06-10T18:59:10.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.130+0000 7fdd32314700 1 -- 172.21.15.104:0/610060907 wait complete. 2023-06-10T18:59:10.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.131+0000 7fdd32314700 1 Processor -- start 2023-06-10T18:59:10.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.131+0000 7fdd32314700 1 -- start start 2023-06-10T18:59:10.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd2c0ce510 0x7fdd2c0b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0c66d0 con 0x7fdd2c070130 2023-06-10T18:59:10.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c153930 con 0x7fdd2c0705b0 2023-06-10T18:59:10.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd32314700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0c8e10 con 0x7fdd2c0ce510 2023-06-10T18:59:10.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41190/0 (socket says 172.21.15.104:41190) 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 -- 172.21.15.104:0/3610174617 learned_addr learned my addr 172.21.15.104:0/3610174617 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2b7fe700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd308b1700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd2c0ce510 0x7fdd2c0b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 -- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd2c0ce510 msgr2=0x7fdd2c0b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd2c0ce510 0x7fdd2c0b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 -- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 msgr2=0x7fdd2c1533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c1533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 -- 172.21.15.104:0/3610174617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009b30 con 0x7fdd2c070130 2023-06-10T18:59:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2b7fe700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd2bfff700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00c820 tx=0x7fdd1c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.132+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c011800 con 0x7fdd2c070130 2023-06-10T18:59:10.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.133+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c011960 con 0x7fdd2c070130 2023-06-10T18:59:10.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.133+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c0b24c0 con 0x7fdd2c070130 2023-06-10T18:59:10.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.133+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c011d10 con 0x7fdd2c070130 2023-06-10T18:59:10.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.133+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c0c6010 con 0x7fdd2c070130 2023-06-10T18:59:10.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.133+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c0c6810 con 0x7fdd2c070130 2023-06-10T18:59:10.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.134+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdd1c018360 con 0x7fdd2c070130 2023-06-10T18:59:10.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.134+0000 7fdd28ff9700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 0x7fdd1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.135+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fdd1c089640 con 0x7fdd2c070130 2023-06-10T18:59:10.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.135+0000 7fdd2b7fe700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 0x7fdd1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.135+0000 7fdd2b7fe700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 0x7fdd1806d190 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009930 tx=0x7fdd20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.139+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdd2c0c6810 con 0x7fdd2c070130 2023-06-10T18:59:10.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.366+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd2c0c6810 con 0x7fdd2c070130 2023-06-10T18:59:10.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.367+0000 7fdd28ff9700 1 -- 172.21.15.104:0/3610174617 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4697 (secure 0 0 0) 0x7fdd2c0c6810 con 0x7fdd2c070130 2023-06-10T18:59:10.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:10.374 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":110,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:58:29.307685+0000","modified":"2023-06-10T18:58:32.331551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10605},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10605":{"gid":10605,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1296534750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1296534750},{"type":"v1","addr":"172.21.15.114:6837","nonce":1296534750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-06-10T18:59:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 msgr2=0x7fdd1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 0x7fdd1806d190 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009930 tx=0x7fdd20008040 comp rx=0 tx=0).stop 2023-06-10T18:59:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 msgr2=0x7fdd2c0c6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.375 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00c820 tx=0x7fdd1c00cbe0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 shutdown_connections 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd2c0ce510 0x7fdd2c0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd2c070130 0x7fdd2c0c6af0 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd1806ace0 0x7fdd1806d190 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 --2- 172.21.15.104:0/3610174617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c0705b0 0x7fdd2c1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 >> 172.21.15.104:0/3610174617 conn(0x7fdd2c06ab00 msgr2=0x7fdd2c15aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.376 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 shutdown_connections 2023-06-10T18:59:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.369+0000 7fdd32314700 1 -- 172.21.15.104:0/3610174617 wait complete. 2023-06-10T18:59:10.377 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 110 2023-06-10T18:59:10.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:10.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.500+0000 7f715f752700 1 Processor -- start 2023-06-10T18:59:10.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.500+0000 7f715f752700 1 -- start start 2023-06-10T18:59:10.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715f752700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7158070a20 0x7f715806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715f752700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715f752700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715806ae00 con 0x7f7158070df0 2023-06-10T18:59:10.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715f752700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7158069e70 con 0x7f7158070a20 2023-06-10T18:59:10.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715f752700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71581595f0 con 0x7f7158158eb0 2023-06-10T18:59:10.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715dcef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715d4ee700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7158070a20 0x7f715806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715dcef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60586/0 (socket says 172.21.15.104:60586) 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715cced700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7158070df0 0x7f71581587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44872/0 (socket says 172.21.15.104:44872) 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.501+0000 7f715dcef700 1 -- 172.21.15.104:0/2394564066 learned_addr learned my addr 172.21.15.104:0/2394564066 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715d4ee700 1 -- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 msgr2=0x7f7158159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:10.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715d4ee700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715d4ee700 1 -- 172.21.15.104:0/2394564066 >> v1:172.21.15.114:6789/0 conn(0x7f7158070df0 legacy=0x7f71581587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715d4ee700 1 -- 172.21.15.104:0/2394564066 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71581598d0 con 0x7f7158070a20 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715dcef700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715d4ee700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070a20 0x7f715806a8c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7154009840 tx=0x7f7154009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4940556e18514d88 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f714ffff700 1 -- 172.21.15.104:0/2394564066 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154011830 con 0x7f7158070a20 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f714ffff700 1 -- 172.21.15.104:0/2394564066 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7154011990 con 0x7f7158070a20 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f714ffff700 1 -- 172.21.15.104:0/2394564066 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154011d20 con 0x7f7158070a20 2023-06-10T18:59:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 -- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070a20 msgr2=0x7f715806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070a20 0x7f715806a8c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7154009840 tx=0x7f7154009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 -- 172.21.15.104:0/2394564066 shutdown_connections 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f7158159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 --2- 172.21.15.104:0/2394564066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070a20 0x7f715806a8c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.502+0000 7f715f752700 1 -- 172.21.15.104:0/2394564066 >> 172.21.15.104:0/2394564066 conn(0x7f71580b1610 msgr2=0x7f71580ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.503+0000 7f715f752700 1 -- 172.21.15.104:0/2394564066 shutdown_connections 2023-06-10T18:59:10.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.503+0000 7f715f752700 1 -- 172.21.15.104:0/2394564066 wait complete. 2023-06-10T18:59:10.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 Processor -- start 2023-06-10T18:59:10.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 -- start start 2023-06-10T18:59:10.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070df0 0x7f71580c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.528 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f71580acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580b2690 con 0x7f7158070a20 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580b2970 con 0x7f7158070df0 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.504+0000 7f715f752700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71580b2c50 con 0x7f7158158eb0 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41206/0 (socket says 172.21.15.104:41206) 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 -- 172.21.15.104:0/2255986523 learned_addr learned my addr 172.21.15.104:0/2255986523 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.529 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715cced700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070df0 0x7f71580c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715dcef700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f71580acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 -- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 msgr2=0x7f71580acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f71580acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 -- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070df0 msgr2=0x7f71580c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070df0 0x7f71580c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.530 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 -- 172.21.15.104:0/2255986523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009b30 con 0x7f7158070a20 2023-06-10T18:59:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715d4ee700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f7154009fd0 tx=0x7f7154004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.531 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71540036a0 con 0x7f7158070a20 2023-06-10T18:59:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715401e070 con 0x7f7158070a20 2023-06-10T18:59:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71540094a0 con 0x7f7158070a20 2023-06-10T18:59:10.532 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154003a40 con 0x7f7158070a20 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.505+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71580b3210 con 0x7f7158070a20 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.506+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158071780 con 0x7f7158070a20 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.507+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7154003ba0 con 0x7f7158070a20 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.507+0000 7f714dffb700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 0x7f713c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.507+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f715408a4c0 con 0x7f7158070a20 2023-06-10T18:59:10.533 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.507+0000 7f715cced700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 0x7f713c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.508+0000 7f715cced700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 0x7f713c06d190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7148000f60 tx=0x7f7148007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.534 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.511+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7154055450 con 0x7f7158070a20 2023-06-10T18:59:10.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.699+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71580cb680 con 0x7f7158070a20 2023-06-10T18:59:10.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.700+0000 7f714dffb700 1 -- 172.21.15.104:0/2255986523 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+18714 (secure 0 0 0) 0x7f7154020090 con 0x7f7158070a20 2023-06-10T18:59:10.706 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:10.706 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":175,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:58:54.747698+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:58:27.026358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-06-10T18:58:28.306697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.a","mappings":[{"from":0,"to":1}]},{"pgid":"25.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/827471637":"2023-06-10T19:58:50.352178+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:10.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 msgr2=0x7f713c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 0x7f713c06d190 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7148000f60 tx=0x7f7148007ed0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.708 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 msgr2=0x7f7158154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f7154009fd0 tx=0x7f7154004750 comp rx=0 tx=0).stop 2023-06-10T18:59:10.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 shutdown_connections 2023-06-10T18:59:10.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7158158eb0 0x7f71580acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7158070a20 0x7f7158154c10 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.709 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f713c06ace0 0x7f713c06d190 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.702+0000 7f715f752700 1 --2- 172.21.15.104:0/2255986523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7158070df0 0x7f71580c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.703+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 >> 172.21.15.104:0/2255986523 conn(0x7f71580b1610 msgr2=0x7f71580cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.703+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 shutdown_connections 2023-06-10T18:59:10.710 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.703+0000 7f715f752700 1 -- 172.21.15.104:0/2255986523 wait complete. 2023-06-10T18:59:10.721 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T18:59:10.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.837+0000 7fbb48122700 1 Processor -- start 2023-06-10T18:59:10.843 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.837+0000 7fbb48122700 1 -- start start 2023-06-10T18:59:10.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb48122700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4006e4b0 0x7fbb4006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb48122700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb40069e70 0x7fbb400ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb48122700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4006d290 con 0x7fbb4006e0e0 2023-06-10T18:59:10.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb48122700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb40071640 con 0x7fbb4006e4b0 2023-06-10T18:59:10.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb48122700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb400aef00 con 0x7fbb40069e70 2023-06-10T18:59:10.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4006e4b0 0x7fbb4006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fbb4006e4b0 0x7fbb4006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41214/0 (socket says 172.21.15.104:41214) 2023-06-10T18:59:10.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 -- 172.21.15.104:0/3110145576 learned_addr learned my addr 172.21.15.104:0/3110145576 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb466bf700 1 --2- 172.21.15.104:0/3110145576 >> v2:172.21.15.114:3300/0 conn(0x7fbb40069e70 0x7fbb400ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 -- 172.21.15.104:0/3110145576 >> v2:172.21.15.114:3300/0 conn(0x7fbb40069e70 msgr2=0x7fbb400ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:10.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 --2- 172.21.15.104:0/3110145576 >> v2:172.21.15.114:3300/0 conn(0x7fbb40069e70 0x7fbb400ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 -- 172.21.15.104:0/3110145576 >> v1:172.21.15.114:6789/0 conn(0x7fbb4006e0e0 legacy=0x7fbb400c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.838+0000 7fbb456bd700 1 -- 172.21.15.104:0/3110145576 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb400ca2d0 con 0x7fbb4006e4b0 2023-06-10T18:59:10.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb456bd700 1 --2- 172.21.15.104:0/3110145576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb4006cd50 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fbb40071210 tx=0x7fbb3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2e4a659577884c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb44ebc700 1 -- 172.21.15.104:0/3110145576 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00b870 con 0x7fbb4006e4b0 2023-06-10T18:59:10.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb44ebc700 1 -- 172.21.15.104:0/3110145576 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c00b9d0 con 0x7fbb4006e4b0 2023-06-10T18:59:10.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb44ebc700 1 -- 172.21.15.104:0/3110145576 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00bd60 con 0x7fbb4006e4b0 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 -- 172.21.15.104:0/3110145576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 msgr2=0x7fbb4006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 --2- 172.21.15.104:0/3110145576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb4006cd50 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fbb40071210 tx=0x7fbb3c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 -- 172.21.15.104:0/3110145576 shutdown_connections 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 --2- 172.21.15.104:0/3110145576 >> v2:172.21.15.114:3300/0 conn(0x7fbb40069e70 0x7fbb400ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 --2- 172.21.15.104:0/3110145576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb4006cd50 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 -- 172.21.15.104:0/3110145576 >> 172.21.15.104:0/3110145576 conn(0x7fbb4006b8a0 msgr2=0x7fbb400cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 -- 172.21.15.104:0/3110145576 shutdown_connections 2023-06-10T18:59:10.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.839+0000 7fbb48122700 1 -- 172.21.15.104:0/3110145576 wait complete. 2023-06-10T18:59:10.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 Processor -- start 2023-06-10T18:59:10.852 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 -- start start 2023-06-10T18:59:10.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb40069e70 0x7fbb400b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb4006e0e0 0x7fbb401f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b1210 con 0x7fbb4006e4b0 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b1b70 con 0x7fbb4006e0e0 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb48122700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b0590 con 0x7fbb40069e70 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb466bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb466bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41222/0 (socket says 172.21.15.104:41222) 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.841+0000 7fbb466bf700 1 -- 172.21.15.104:0/1107514352 learned_addr learned my addr 172.21.15.104:0/1107514352 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb45ebe700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb40069e70 0x7fbb400b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb456bd700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb4006e0e0 0x7fbb401f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 -- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb40069e70 msgr2=0x7fbb400b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb40069e70 0x7fbb400b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 -- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb4006e0e0 msgr2=0x7fbb401f4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:10.856 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb4006e0e0 0x7fbb401f4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 -- 172.21.15.104:0/1107514352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c009550 con 0x7fbb4006e4b0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb466bf700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fbb38007e60 tx=0x7fbb3800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800ed30 con 0x7fbb4006e4b0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb38014420 con 0x7fbb4006e4b0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb400b0870 con 0x7fbb4006e4b0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380147d0 con 0x7fbb4006e4b0 2023-06-10T18:59:10.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.842+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb400b0e30 con 0x7fbb4006e4b0 2023-06-10T18:59:10.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.843+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb40074050 con 0x7fbb4006e4b0 2023-06-10T18:59:10.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.843+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbb38014930 con 0x7fbb4006e4b0 2023-06-10T18:59:10.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.844+0000 7fbb367fc700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 0x7fbb2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:10.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.844+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fbb38089c20 con 0x7fbb4006e4b0 2023-06-10T18:59:10.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.844+0000 7fbb45ebe700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 0x7fbb2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:10.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.845+0000 7fbb45ebe700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 0x7fbb2406d190 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbb30000fb0 tx=0x7fbb30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:10.859 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:10.848+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbb38054c60 con 0x7fbb4006e4b0 2023-06-10T18:59:11.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.076+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbb400acea0 con 0x7fbb4006e4b0 2023-06-10T18:59:11.803 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T18:59:11.798+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T18:59:11.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.804+0000 7fbb367fc700 1 -- 172.21.15.104:0/1107514352 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbb38058280 con 0x7fbb4006e4b0 2023-06-10T18:59:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 msgr2=0x7fbb2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 0x7fbb2406d190 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbb30000fb0 tx=0x7fbb30009040 comp rx=0 tx=0).stop 2023-06-10T18:59:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 msgr2=0x7fbb401572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fbb38007e60 tx=0x7fbb3800e3c0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 shutdown_connections 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbb4006e4b0 0x7fbb401572e0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbb40069e70 0x7fbb400b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbb2406ace0 0x7fbb2406d190 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 --2- 172.21.15.104:0/1107514352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb4006e0e0 0x7fbb401f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 >> 172.21.15.104:0/1107514352 conn(0x7fbb4006b8a0 msgr2=0x7fbb400ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 shutdown_connections 2023-06-10T18:59:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.807+0000 7fbb48122700 1 -- 172.21.15.104:0/1107514352 wait complete. 2023-06-10T18:59:11.814 INFO:tasks.ceph.mds.b.smithi114.stderr: -3956> 2023-06-10T18:58:34.794+0000 7f1fcdff5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:59:11.814 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T18:59:11.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T18:59:11.843 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:59:11.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.946+0000 7f7565e6e700 1 Processor -- start 2023-06-10T18:59:11.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 -- start start 2023-06-10T18:59:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75600d04f0 0x7f75600d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f75600b05d0 0x7f75600b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600af530 con 0x7f75600d25c0 2023-06-10T18:59:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600af810 con 0x7f75600b05d0 2023-06-10T18:59:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f7565e6e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600afaf0 con 0x7f75600d04f0 2023-06-10T18:59:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f755ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f75600b05d0 0x7f75600b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f755ffff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f75600b05d0 0x7f75600b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41230/0 (socket says 172.21.15.104:41230) 2023-06-10T18:59:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f755effd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f75600d25c0 0x7f75600d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39056/0 (socket says 172.21.15.104:39056) 2023-06-10T18:59:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.947+0000 7f755ffff700 1 -- 172.21.15.104:0/1948266466 learned_addr learned my addr 172.21.15.104:0/1948266466 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755f7fe700 1 --2- 172.21.15.104:0/1948266466 >> v2:172.21.15.114:3300/0 conn(0x7f75600d04f0 0x7f75600d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755ffff700 1 -- 172.21.15.104:0/1948266466 >> v2:172.21.15.114:3300/0 conn(0x7f75600d04f0 msgr2=0x7f75600d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755ffff700 1 --2- 172.21.15.104:0/1948266466 >> v2:172.21.15.114:3300/0 conn(0x7f75600d04f0 0x7f75600d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755ffff700 1 -- 172.21.15.104:0/1948266466 >> v1:172.21.15.104:6789/0 conn(0x7f75600d25c0 legacy=0x7f75600d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755ffff700 1 -- 172.21.15.104:0/1948266466 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600afdd0 con 0x7f75600b05d0 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755e7fc700 1 -- 172.21.15.104:0/1948266466 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162861820 0 0) 0x7f75600af530 con 0x7f75600d25c0 2023-06-10T18:59:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755ffff700 1 --2- 172.21.15.104:0/1948266466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75600b0a50 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f7560066690 tx=0x7f755800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa26bc6ca1a2113d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755e7fc700 1 -- 172.21.15.104:0/1948266466 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755800e070 con 0x7f75600b05d0 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755e7fc700 1 -- 172.21.15.104:0/1948266466 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755800b9c0 con 0x7f75600b05d0 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.948+0000 7f755e7fc700 1 -- 172.21.15.104:0/1948266466 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755800bd70 con 0x7f75600b05d0 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 -- 172.21.15.104:0/1948266466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 msgr2=0x7f75600b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 --2- 172.21.15.104:0/1948266466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75600b0a50 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f7560066690 tx=0x7f755800b330 comp rx=0 tx=0).stop 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 -- 172.21.15.104:0/1948266466 shutdown_connections 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 --2- 172.21.15.104:0/1948266466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75600b0a50 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 --2- 172.21.15.104:0/1948266466 >> v2:172.21.15.114:3300/0 conn(0x7f75600d04f0 0x7f75600d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 -- 172.21.15.104:0/1948266466 >> 172.21.15.104:0/1948266466 conn(0x7f7560071ec0 msgr2=0x7f756009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 -- 172.21.15.104:0/1948266466 shutdown_connections 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.949+0000 7f7565e6e700 1 -- 172.21.15.104:0/1948266466 wait complete. 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.950+0000 7f7565e6e700 1 Processor -- start 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.950+0000 7f7565e6e700 1 -- start start 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75600d04f0 0x7f75600cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 0x7f7560154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75601e47e0 con 0x7f75600b05d0 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600cac50 con 0x7f75600d04f0 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75600caf30 con 0x7f75600d25c0 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41232/0 (socket says 172.21.15.104:41232) 2023-06-10T18:59:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 -- 172.21.15.104:0/75998030 learned_addr learned my addr 172.21.15.104:0/75998030 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755effd700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75600d04f0 0x7f75600cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755ffff700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 0x7f7560154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 -- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 msgr2=0x7f7560154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 0x7f7560154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 -- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75600d04f0 msgr2=0x7f75600cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75600d04f0 0x7f75600cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 -- 172.21.15.104:0/75998030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009b30 con 0x7f75600b05d0 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755ffff700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 0x7f7560154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f755f7fe700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f755400e730 tx=0x7f755400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554013610 con 0x7f75600b05d0 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554013770 con 0x7f75600b05d0 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75601e4400 con 0x7f75600b05d0 2023-06-10T18:59:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.951+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554013b20 con 0x7f75600b05d0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.952+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75600cbb50 con 0x7f75600b05d0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.952+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75601e4a50 con 0x7f75600b05d0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.953+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7554013c80 con 0x7f75600b05d0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.953+0000 7f7547fff700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.953+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f7554088f80 con 0x7f75600b05d0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.953+0000 7f755effd700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.954+0000 7f755effd700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f75600b0e70 tx=0x7f7550007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:11.957+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f75540578d0 con 0x7f75600b05d0 2023-06-10T18:59:12.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.185+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7560067010 con 0x7f75600b05d0 2023-06-10T18:59:12.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.807+0000 7f7547fff700 1 -- 172.21.15.104:0/75998030 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7554057460 con 0x7f75600b05d0 2023-06-10T18:59:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 msgr2=0x7f754c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f75600b0e70 tx=0x7f7550007e70 comp rx=0 tx=0).stop 2023-06-10T18:59:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 msgr2=0x7f75601e4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f755400e730 tx=0x7f755400ea40 comp rx=0 tx=0).stop 2023-06-10T18:59:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 shutdown_connections 2023-06-10T18:59:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75600d25c0 0x7f7560154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75600b05d0 0x7f75601e4c00 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f754c06ace0 0x7f754c06d190 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 --2- 172.21.15.104:0/75998030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75600d04f0 0x7f75600cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 >> 172.21.15.104:0/75998030 conn(0x7f7560071ec0 msgr2=0x7f75600b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.810+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 shutdown_connections 2023-06-10T18:59:12.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.811+0000 7f7565e6e700 1 -- 172.21.15.104:0/75998030 wait complete. 2023-06-10T18:59:12.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T18:59:12.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.952+0000 7f5a8d348700 1 Processor -- start 2023-06-10T18:59:12.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.952+0000 7f5a8d348700 1 -- start start 2023-06-10T18:59:12.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a8d348700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a8d348700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 0x7f5a880cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a8d348700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a880d0480 con 0x7f5a880aec40 2023-06-10T18:59:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a8d348700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8806d770 con 0x7f5a880ca850 2023-06-10T18:59:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a8d348700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8815dd00 con 0x7f5a8806cde0 2023-06-10T18:59:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a867fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a877fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 0x7f5a880cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a86ffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5a880aec40 0x7f5a88069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44914/0 (socket says 172.21.15.104:44914) 2023-06-10T18:59:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a867fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60626/0 (socket says 172.21.15.104:60626) 2023-06-10T18:59:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.953+0000 7f5a86ffd700 1 -- 172.21.15.104:0/2239297992 learned_addr learned my addr 172.21.15.104:0/2239297992 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a867fc700 1 -- 172.21.15.104:0/2239297992 >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 msgr2=0x7f5a880cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a867fc700 1 --2- 172.21.15.104:0/2239297992 >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 0x7f5a880cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a867fc700 1 -- 172.21.15.104:0/2239297992 >> v1:172.21.15.114:6789/0 conn(0x7f5a880aec40 legacy=0x7f5a88069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a85ffb700 1 -- 172.21.15.104:0/2239297992 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635638766 0 0) 0x7f5a880d0480 con 0x7f5a880aec40 2023-06-10T18:59:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a867fc700 1 -- 172.21.15.104:0/2239297992 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8815a610 con 0x7f5a8806cde0 2023-06-10T18:59:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a877fe700 1 --2- 172.21.15.104:0/2239297992 >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 0x7f5a880cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a867fc700 1 --2- 172.21.15.104:0/2239297992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009840 tx=0x7f5a78009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f09f1608f65d120 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a85ffb700 1 -- 172.21.15.104:0/2239297992 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78011830 con 0x7f5a8806cde0 2023-06-10T18:59:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a85ffb700 1 -- 172.21.15.104:0/2239297992 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78011990 con 0x7f5a8806cde0 2023-06-10T18:59:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.954+0000 7f5a85ffb700 1 -- 172.21.15.104:0/2239297992 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78011d20 con 0x7f5a8806cde0 2023-06-10T18:59:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 -- 172.21.15.104:0/2239297992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 msgr2=0x7f5a8806d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 --2- 172.21.15.104:0/2239297992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009840 tx=0x7f5a78009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 -- 172.21.15.104:0/2239297992 shutdown_connections 2023-06-10T18:59:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 --2- 172.21.15.104:0/2239297992 >> v2:172.21.15.114:3300/0 conn(0x7f5a880ca850 0x7f5a880cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 --2- 172.21.15.104:0/2239297992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a8806cde0 0x7f5a8806d230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 -- 172.21.15.104:0/2239297992 >> 172.21.15.104:0/2239297992 conn(0x7f5a8806b8a0 msgr2=0x7f5a880cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 -- 172.21.15.104:0/2239297992 shutdown_connections 2023-06-10T18:59:12.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.955+0000 7f5a8d348700 1 -- 172.21.15.104:0/2239297992 wait complete. 2023-06-10T18:59:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.956+0000 7f5a8d348700 1 Processor -- start 2023-06-10T18:59:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.956+0000 7f5a8d348700 1 -- start start 2023-06-10T18:59:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a880aec40 0x7f5a881ef530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a880ca850 0x7f5a881589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a881efa70 con 0x7f5a8806cde0 2023-06-10T18:59:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8815d6c0 con 0x7f5a880ca850 2023-06-10T18:59:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8814d210 con 0x7f5a880aec40 2023-06-10T18:59:12.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41246/0 (socket says 172.21.15.104:41246) 2023-06-10T18:59:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 -- 172.21.15.104:0/337963318 learned_addr learned my addr 172.21.15.104:0/337963318 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:12.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a867fc700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a880aec40 0x7f5a881ef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a877fe700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a880ca850 0x7f5a881589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 -- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a880aec40 msgr2=0x7f5a881ef530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a880aec40 0x7f5a881ef530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 -- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a880ca850 msgr2=0x7f5a881589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a880ca850 0x7f5a881589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:12.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 -- 172.21.15.104:0/337963318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80009a70 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a86ffd700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c430 tx=0x7f5a7c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c004cd0 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c004e30 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.957+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a780094a0 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.958+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c007310 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.958+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8815d070 con 0x7f5a8806cde0 2023-06-10T18:59:12.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.958+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a88074050 con 0x7f5a8806cde0 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.959+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a7c007470 con 0x7f5a8806cde0 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.959+0000 7f5a6f7fe700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 0x7f5a7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.959+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f5a7c089210 con 0x7f5a8806cde0 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.959+0000 7f5a867fc700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 0x7f5a7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.960+0000 7f5a867fc700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 0x7f5a7406d190 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000c00 tx=0x7f5a7801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:12.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:12.964+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a7c0540c0 con 0x7f5a8806cde0 2023-06-10T18:59:13.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.150+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a880cfd00 con 0x7f5a8806cde0 2023-06-10T18:59:13.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.813+0000 7f5a6f7fe700 1 -- 172.21.15.104:0/337963318 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v177) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5a7c0576e0 con 0x7f5a8806cde0 2023-06-10T18:59:13.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 msgr2=0x7f5a7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 0x7f5a7406d190 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000c00 tx=0x7f5a7801c040 comp rx=0 tx=0).stop 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 msgr2=0x7f5a880c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c430 tx=0x7f5a7c00c740 comp rx=0 tx=0).stop 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 shutdown_connections 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f5a7406ace0 0x7f5a7406d190 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a880ca850 0x7f5a881589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a8806cde0 0x7f5a880c8fb0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 --2- 172.21.15.104:0/337963318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a880aec40 0x7f5a881ef530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 >> 172.21.15.104:0/337963318 conn(0x7f5a8806b8a0 msgr2=0x7f5a88069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 shutdown_connections 2023-06-10T18:59:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.817+0000 7f5a8d348700 1 -- 172.21.15.104:0/337963318 wait complete. 2023-06-10T18:59:13.824 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T18:59:13.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T18:59:13.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.956+0000 7feca82b6700 1 Processor -- start 2023-06-10T18:59:13.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.956+0000 7feca82b6700 1 -- start start 2023-06-10T18:59:13.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca82b6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feca015c580 0x7feca00b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:13.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca82b6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca006e250 con 0x7feca006cde0 2023-06-10T18:59:13.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca82b6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca015c950 con 0x7feca006e6e0 2023-06-10T18:59:13.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca82b6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca00b1e80 con 0x7feca015c580 2023-06-10T18:59:13.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca6052700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feca015c580 0x7feca00b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca6052700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7feca015c580 0x7feca00b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41254/0 (socket says 172.21.15.104:41254) 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.957+0000 7feca6052700 1 -- 172.21.15.104:0/953313537 learned_addr learned my addr 172.21.15.104:0/953313537 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca6052700 1 -- 172.21.15.104:0/953313537 >> v1:172.21.15.114:6789/0 conn(0x7feca006e6e0 legacy=0x7feca00b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca6052700 1 -- 172.21.15.104:0/953313537 >> v1:172.21.15.104:6789/0 conn(0x7feca006cde0 legacy=0x7feca00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca6052700 1 -- 172.21.15.104:0/953313537 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca006d920 con 0x7feca015c580 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca6052700 1 --2- 172.21.15.104:0/953313537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca015c580 0x7feca00b1070 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fec9c006fd0 tx=0x7fec9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ef55f1e42a0828 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:13.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca5050700 1 -- 172.21.15.104:0/953313537 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c005e30 con 0x7feca015c580 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca5050700 1 -- 172.21.15.104:0/953313537 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c006030 con 0x7feca015c580 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca5050700 1 -- 172.21.15.104:0/953313537 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0063e0 con 0x7feca015c580 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca82b6700 1 -- 172.21.15.104:0/953313537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca015c580 msgr2=0x7feca00b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.958+0000 7feca82b6700 1 --2- 172.21.15.104:0/953313537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca015c580 0x7feca00b1070 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fec9c006fd0 tx=0x7fec9c009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.959+0000 7feca82b6700 1 -- 172.21.15.104:0/953313537 shutdown_connections 2023-06-10T18:59:13.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.959+0000 7feca82b6700 1 --2- 172.21.15.104:0/953313537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca015c580 0x7feca00b1070 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.959+0000 7feca82b6700 1 -- 172.21.15.104:0/953313537 >> 172.21.15.104:0/953313537 conn(0x7feca006b8a0 msgr2=0x7feca00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:13.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.959+0000 7feca82b6700 1 -- 172.21.15.104:0/953313537 shutdown_connections 2023-06-10T18:59:13.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.959+0000 7feca82b6700 1 -- 172.21.15.104:0/953313537 wait complete. 2023-06-10T18:59:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.960+0000 7feca82b6700 1 Processor -- start 2023-06-10T18:59:13.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.960+0000 7feca82b6700 1 -- start start 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.960+0000 7feca82b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.960+0000 7feca82b6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feca006e6e0 0x7feca00b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feca015c580 0x7feca01556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca01e3f40 con 0x7feca006cde0 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca00b5030 con 0x7feca006e6e0 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca00b5310 con 0x7feca015c580 2023-06-10T18:59:13.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41258/0 (socket says 172.21.15.104:41258) 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 -- 172.21.15.104:0/1785155015 learned_addr learned my addr 172.21.15.104:0/1785155015 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6853700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feca015c580 0x7feca01556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca5851700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feca006e6e0 0x7feca00b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 -- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feca015c580 msgr2=0x7feca01556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feca015c580 0x7feca01556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 -- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feca006e6e0 msgr2=0x7feca00b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feca006e6e0 0x7feca00b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 -- 172.21.15.104:0/1785155015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c009500 con 0x7feca006cde0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca6052700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009c10 tx=0x7fec9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c004250 con 0x7feca006cde0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c0043f0 con 0x7feca006cde0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca01e4190 con 0x7feca006cde0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0038e0 con 0x7feca006cde0 2023-06-10T18:59:13.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.961+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca00cc670 con 0x7feca006cde0 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.962+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca01e46b0 con 0x7feca006cde0 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.962+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fec9c022070 con 0x7feca006cde0 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.963+0000 7fec96ffd700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 0x7fec8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.963+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6131+0+0 (secure 0 0 0) 0x7fec9c08ae00 con 0x7feca006cde0 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.963+0000 7feca5851700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 0x7fec8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:13.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.964+0000 7feca5851700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 0x7fec8406d190 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7feca00c98c0 tx=0x7fec90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:13.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:13.967+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fec9c055ef0 con 0x7feca006cde0 2023-06-10T18:59:14.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.153+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feca0066f10 con 0x7feca006cde0 2023-06-10T18:59:14.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.823+0000 7fec96ffd700 1 -- 172.21.15.104:0/1785155015 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v178) v1 ==== 169+0+0 (secure 0 0 0) 0x7fec9c059510 con 0x7feca006cde0 2023-06-10T18:59:14.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 msgr2=0x7fec8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 0x7fec8406d190 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7feca00c98c0 tx=0x7fec90009040 comp rx=0 tx=0).stop 2023-06-10T18:59:14.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 msgr2=0x7feca00cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009c10 tx=0x7fec9c004750 comp rx=0 tx=0).stop 2023-06-10T18:59:14.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 shutdown_connections 2023-06-10T18:59:14.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feca015c580 0x7feca01556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feca006cde0 0x7feca00cc130 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fec8406ace0 0x7fec8406d190 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 --2- 172.21.15.104:0/1785155015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feca006e6e0 0x7feca00b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 >> 172.21.15.104:0/1785155015 conn(0x7feca006b8a0 msgr2=0x7feca00b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.826+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 shutdown_connections 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.827+0000 7feca82b6700 1 -- 172.21.15.104:0/1785155015 wait complete. 2023-06-10T18:59:14.834 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T18:59:14.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T18:59:14.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.969+0000 7fece8df8700 1 Processor -- start 2023-06-10T18:59:14.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.969+0000 7fece8df8700 1 -- start start 2023-06-10T18:59:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.970+0000 7fece8df8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece415c730 0x7fece40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.970+0000 7fece8df8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece40cd720 0x7fece406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.970+0000 7fece8df8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece415cb00 con 0x7fece406d2b0 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.970+0000 7fece8df8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece40cdb90 con 0x7fece415c730 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.970+0000 7fece8df8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40acd90 con 0x7fece40cd720 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece415c730 0x7fece40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fece415c730 0x7fece40cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41266/0 (socket says 172.21.15.104:41266) 2023-06-10T18:59:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 -- 172.21.15.104:0/3197879328 learned_addr learned my addr 172.21.15.104:0/3197879328 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece1d9b700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece40cd720 0x7fece406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 -- 172.21.15.104:0/3197879328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece40cd720 msgr2=0x7fece406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece40cd720 0x7fece406cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 -- 172.21.15.104:0/3197879328 >> v1:172.21.15.114:6789/0 conn(0x7fece406d2b0 legacy=0x7fece406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 -- 172.21.15.104:0/3197879328 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40ac460 con 0x7fece415c730 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece259c700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece415c730 0x7fece40cd1e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fecd4006fd0 tx=0x7fecd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1f6cfc49dc1da2 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece159a700 1 -- 172.21.15.104:0/3197879328 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4005e30 con 0x7fece415c730 2023-06-10T18:59:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece159a700 1 -- 172.21.15.104:0/3197879328 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd4004030 con 0x7fece415c730 2023-06-10T18:59:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.971+0000 7fece159a700 1 -- 172.21.15.104:0/3197879328 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd40043e0 con 0x7fece415c730 2023-06-10T18:59:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 -- 172.21.15.104:0/3197879328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece415c730 msgr2=0x7fece40cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece415c730 0x7fece40cd1e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fecd4006fd0 tx=0x7fecd4009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 -- 172.21.15.104:0/3197879328 shutdown_connections 2023-06-10T18:59:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece415c730 0x7fece40cd1e0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 --2- 172.21.15.104:0/3197879328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece40cd720 0x7fece406cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 -- 172.21.15.104:0/3197879328 >> 172.21.15.104:0/3197879328 conn(0x7fece406b8a0 msgr2=0x7fece40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 -- 172.21.15.104:0/3197879328 shutdown_connections 2023-06-10T18:59:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.972+0000 7fece8df8700 1 -- 172.21.15.104:0/3197879328 wait complete. 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.973+0000 7fece8df8700 1 Processor -- start 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.973+0000 7fece8df8700 1 -- start start 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece40cd720 0x7fece40b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece415c730 0x7fece40b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece40c6780 con 0x7fece406d2b0 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece40c6a60 con 0x7fece40cd720 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece8df8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40c6d40 con 0x7fece415c730 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41272/0 (socket says 172.21.15.104:41272) 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 -- 172.21.15.104:0/3618593888 learned_addr learned my addr 172.21.15.104:0/3618593888 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece2d9d700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece415c730 0x7fece40b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece1d9b700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece40cd720 0x7fece40b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 -- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece415c730 msgr2=0x7fece40b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece415c730 0x7fece40b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 -- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece40cd720 msgr2=0x7fece40b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece40cd720 0x7fece40b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 -- 172.21.15.104:0/3618593888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4009500 con 0x7fece406d2b0 2023-06-10T18:59:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fece259c700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009c10 tx=0x7fecd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.974+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd4003e10 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.975+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd401d430 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.975+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd401d7e0 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.975+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece40c7020 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.975+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece40c7820 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.975+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece40c7540 con 0x7fece406d2b0 2023-06-10T18:59:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.976+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fecd401d940 con 0x7fece406d2b0 2023-06-10T18:59:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.976+0000 7fecdaffd700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 0x7fecc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.976+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fecd408a970 con 0x7fece406d2b0 2023-06-10T18:59:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.976+0000 7fece1d9b700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 0x7fecc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.977+0000 7fece1d9b700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 0x7fecc406d190 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fece4099b40 tx=0x7feccc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:14.981+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fece40c7540 con 0x7fece406d2b0 2023-06-10T18:59:15.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.169+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fece40c7540 con 0x7fece406d2b0 2023-06-10T18:59:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.172+0000 7fecdaffd700 1 -- 172.21.15.104:0/3618593888 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fece40c7540 con 0x7fece406d2b0 2023-06-10T18:59:15.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.174+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 msgr2=0x7fecc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.174+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 0x7fecc406d190 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fece4099b40 tx=0x7feccc008040 comp rx=0 tx=0).stop 2023-06-10T18:59:15.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.174+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 msgr2=0x7fece4154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.174+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009c10 tx=0x7fecd4004750 comp rx=0 tx=0).stop 2023-06-10T18:59:15.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 shutdown_connections 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fece415c730 0x7fece40b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fece406d2b0 0x7fece4154620 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fecc406ace0 0x7fecc406d190 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 --2- 172.21.15.104:0/3618593888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fece40cd720 0x7fece40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 >> 172.21.15.104:0/3618593888 conn(0x7fece406b8a0 msgr2=0x7fece40ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 shutdown_connections 2023-06-10T18:59:15.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.175+0000 7fece8df8700 1 -- 172.21.15.104:0/3618593888 wait complete. 2023-06-10T18:59:15.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client' 2023-06-10T18:59:15.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.306+0000 7fb159799700 1 Processor -- start 2023-06-10T18:59:15.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.306+0000 7fb159799700 1 -- start start 2023-06-10T18:59:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb159799700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb159799700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 0x7fb1540ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb159799700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540aed90 con 0x7fb1540ade00 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb159799700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1540cf430 con 0x7fb1540ae930 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb159799700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cee70 con 0x7fb1540ada30 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb152ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb1537fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 0x7fb1540ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb152ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60664/0 (socket says 172.21.15.104:60664) 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb1527fc700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb1540ade00 0x7fb1540ce0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:44954/0 (socket says 172.21.15.104:44954) 2023-06-10T18:59:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.307+0000 7fb152ffd700 1 -- 172.21.15.104:0/1506900578 learned_addr learned my addr 172.21.15.104:0/1506900578 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb152ffd700 1 -- 172.21.15.104:0/1506900578 >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 msgr2=0x7fb1540ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb152ffd700 1 --2- 172.21.15.104:0/1506900578 >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 0x7fb1540ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb152ffd700 1 -- 172.21.15.104:0/1506900578 >> v1:172.21.15.114:6789/0 conn(0x7fb1540ade00 legacy=0x7fb1540ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb151ffb700 1 -- 172.21.15.104:0/1506900578 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771046092 0 0) 0x7fb1540aed90 con 0x7fb1540ade00 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb152ffd700 1 -- 172.21.15.104:0/1506900578 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540cf150 con 0x7fb1540ada30 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb1537fe700 1 --2- 172.21.15.104:0/1506900578 >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 0x7fb1540ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb152ffd700 1 --2- 172.21.15.104:0/1506900578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b63eb814c671ebf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb151ffb700 1 -- 172.21.15.104:0/1506900578 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148005e30 con 0x7fb1540ada30 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb151ffb700 1 -- 172.21.15.104:0/1506900578 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148004030 con 0x7fb1540ada30 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.308+0000 7fb151ffb700 1 -- 172.21.15.104:0/1506900578 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1480043e0 con 0x7fb1540ada30 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 -- 172.21.15.104:0/1506900578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 msgr2=0x7fb1540ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 --2- 172.21.15.104:0/1506900578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 -- 172.21.15.104:0/1506900578 shutdown_connections 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 --2- 172.21.15.104:0/1506900578 >> v2:172.21.15.114:3300/0 conn(0x7fb1540ae930 0x7fb1540ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 --2- 172.21.15.104:0/1506900578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1540ae3f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 -- 172.21.15.104:0/1506900578 >> 172.21.15.104:0/1506900578 conn(0x7fb15406b8a0 msgr2=0x7fb1540cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 -- 172.21.15.104:0/1506900578 shutdown_connections 2023-06-10T18:59:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.309+0000 7fb159799700 1 -- 172.21.15.104:0/1506900578 wait complete. 2023-06-10T18:59:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.310+0000 7fb159799700 1 Processor -- start 2023-06-10T18:59:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.310+0000 7fb159799700 1 -- start start 2023-06-10T18:59:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1541e59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb15414b380 0x7fb154147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15409c0f0 con 0x7fb1540ae930 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15409c3d0 con 0x7fb15414b380 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb159799700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15409c6b0 con 0x7fb1540ada30 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41290/0 (socket says 172.21.15.104:41290) 2023-06-10T18:59:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 -- 172.21.15.104:0/3100780113 learned_addr learned my addr 172.21.15.104:0/3100780113 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb152ffd700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1541e59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1537fe700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb15414b380 0x7fb154147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 -- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 msgr2=0x7fb1541e59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1541e59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 -- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb15414b380 msgr2=0x7fb154147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb15414b380 0x7fb154147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 -- 172.21.15.104:0/3100780113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c009a70 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.311+0000 7fb1527fc700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c540 tx=0x7fb14400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.312+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144004c80 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.312+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb144004de0 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.312+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148009500 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.312+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144007310 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.312+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15409cc70 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.313+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15409ce90 con 0x7fb1540ae930 2023-06-10T18:59:15.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.313+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb144007470 con 0x7fb1540ae930 2023-06-10T18:59:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.313+0000 7fb13b7fe700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 0x7fb14006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.313+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb144088f20 con 0x7fb1540ae930 2023-06-10T18:59:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.314+0000 7fb152ffd700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 0x7fb14006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.314+0000 7fb152ffd700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 0x7fb14006d190 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.318+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb144054220 con 0x7fb1540ae930 2023-06-10T18:59:15.583 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.578+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client"]} v 0) v1 -- 0x7fb1540cd450 con 0x7fb1540ae930 2023-06-10T18:59:15.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.824+0000 7fb13b7fe700 1 -- 172.21.15.104:0/3100780113 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_mount_after_evicted_client"]}]=0 v727) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb144057840 con 0x7fb1540ae930 2023-06-10T18:59:15.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 msgr2=0x7fb14006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 0x7fb14006d190 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008070 comp rx=0 tx=0).stop 2023-06-10T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 msgr2=0x7fb15409b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c540 tx=0x7fb14400c850 comp rx=0 tx=0).stop 2023-06-10T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 shutdown_connections 2023-06-10T18:59:15.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb14006ace0 0x7fb14006d190 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb15414b380 0x7fb154147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1540ada30 0x7fb1541e59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 --2- 172.21.15.104:0/3100780113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1540ae930 0x7fb15409b680 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 >> 172.21.15.104:0/3100780113 conn(0x7fb15406b8a0 msgr2=0x7fb1540b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 shutdown_connections 2023-06-10T18:59:15.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.826+0000 7fb159799700 1 -- 172.21.15.104:0/3100780113 wait complete. 2023-06-10T18:59:15.844 INFO:tasks.cephfs_test_runner:Test if a new mount of same fs works after client eviction. ... ok 2023-06-10T18:59:15.844 INFO:tasks.cephfs_test_runner:Starting test: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T18:59:15.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted' 2023-06-10T18:59:15.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.959+0000 7f563e478700 1 Processor -- start 2023-06-10T18:59:15.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.959+0000 7f563e478700 1 -- start start 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f563e478700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f563815a2b0 0x7f56380ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f563e478700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5638069e70 0x7f56380cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f563e478700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563815a680 con 0x7f56380cb690 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f563e478700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56380d0510 con 0x7f5638069e70 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f563e478700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563806a5e0 con 0x7f563815a2b0 2023-06-10T18:59:15.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5638069e70 0x7f56380cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5638069e70 0x7f56380cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41296/0 (socket says 172.21.15.104:41296) 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 -- 172.21.15.104:0/604040112 learned_addr learned my addr 172.21.15.104:0/604040112 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 -- 172.21.15.104:0/604040112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f563815a2b0 msgr2=0x7f56380ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 --2- 172.21.15.104:0/604040112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f563815a2b0 0x7f56380ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 -- 172.21.15.104:0/604040112 >> v1:172.21.15.104:6789/0 conn(0x7f56380cb690 legacy=0x7f56380b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 -- 172.21.15.104:0/604040112 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563806a8c0 con 0x7f5638069e70 2023-06-10T18:59:15.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f56377fe700 1 --2- 172.21.15.104:0/604040112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5638069e70 0x7f56380cb150 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f56280098f0 tx=0x7f5628009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94101b308a9410f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.960+0000 7f5636ffd700 1 -- 172.21.15.104:0/604040112 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800b8a0 con 0x7f5638069e70 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f5636ffd700 1 -- 172.21.15.104:0/604040112 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562800ba00 con 0x7f5638069e70 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f5636ffd700 1 -- 172.21.15.104:0/604040112 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800bd90 con 0x7f5638069e70 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 -- 172.21.15.104:0/604040112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5638069e70 msgr2=0x7f56380cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 --2- 172.21.15.104:0/604040112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5638069e70 0x7f56380cb150 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f56280098f0 tx=0x7f5628009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 -- 172.21.15.104:0/604040112 shutdown_connections 2023-06-10T18:59:15.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 --2- 172.21.15.104:0/604040112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f563815a2b0 0x7f56380ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 --2- 172.21.15.104:0/604040112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5638069e70 0x7f56380cb150 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 -- 172.21.15.104:0/604040112 >> 172.21.15.104:0/604040112 conn(0x7f563806b8a0 msgr2=0x7f56380cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 -- 172.21.15.104:0/604040112 shutdown_connections 2023-06-10T18:59:15.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.961+0000 7f563e478700 1 -- 172.21.15.104:0/604040112 wait complete. 2023-06-10T18:59:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.962+0000 7f563e478700 1 Processor -- start 2023-06-10T18:59:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.962+0000 7f563e478700 1 -- start start 2023-06-10T18:59:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5638069e70 0x7f563814ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56380cb690 0x7f56380b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56380b4490 con 0x7f563815a2b0 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563814b2b0 con 0x7f5638069e70 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563e478700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56380b46e0 con 0x7f56380cb690 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563ca15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563ca15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41304/0 (socket says 172.21.15.104:41304) 2023-06-10T18:59:15.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.963+0000 7f563ca15700 1 -- 172.21.15.104:0/4011519745 learned_addr learned my addr 172.21.15.104:0/4011519745 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f56377fe700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56380cb690 0x7f56380b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f5637fff700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5638069e70 0x7f563814ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 -- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56380cb690 msgr2=0x7f56380b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56380cb690 0x7f56380b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 -- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5638069e70 msgr2=0x7f563814ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5638069e70 0x7f563814ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:15.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 -- 172.21.15.104:0/4011519745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628009550 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563ca15700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f563000cc50 tx=0x7f563000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563000ad80 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000ce50 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630003c40 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56380b49c0 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.964+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563814f920 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.965+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638067760 con 0x7f563815a2b0 2023-06-10T18:59:15.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.965+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f56300050d0 con 0x7f563815a2b0 2023-06-10T18:59:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.966+0000 7f5634ff9700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.966+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f5630089370 con 0x7f563815a2b0 2023-06-10T18:59:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.966+0000 7f5637fff700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.966+0000 7f5637fff700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 0x7f562406d190 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f562c00a7f0 tx=0x7f562c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:15.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:15.970+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5630054640 con 0x7f563815a2b0 2023-06-10T18:59:16.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.232+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted"]} v 0) v1 -- 0x7f5638074050 con 0x7f563815a2b0 2023-06-10T18:59:16.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.827+0000 7f5634ff9700 1 -- 172.21.15.104:0/4011519745 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted"]}]=0 v728) v1 ==== 167+0+0 (secure 0 0 0) 0x7f5630057c60 con 0x7f563815a2b0 2023-06-10T18:59:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 msgr2=0x7f562406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 0x7f562406d190 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f562c00a7f0 tx=0x7f562c008040 comp rx=0 tx=0).stop 2023-06-10T18:59:16.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 msgr2=0x7f56380b3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f563000cc50 tx=0x7f563000a340 comp rx=0 tx=0).stop 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 shutdown_connections 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5638069e70 0x7f563814ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f563815a2b0 0x7f56380b3f50 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 --2- 172.21.15.104:0/4011519745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56380cb690 0x7f56380b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 >> 172.21.15.104:0/4011519745 conn(0x7f563806b8a0 msgr2=0x7f5638159140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 shutdown_connections 2023-06-10T18:59:16.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.830+0000 7f563e478700 1 -- 172.21.15.104:0/4011519745 wait complete. 2023-06-10T18:59:16.849 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-06-10T18:59:16.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.969+0000 7f9792559700 1 Processor -- start 2023-06-10T18:59:16.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9792559700 1 -- start start 2023-06-10T18:59:16.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9792559700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f978c0ce390 0x7f978c160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9792559700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0d0340 con 0x7f978c0cae10 2023-06-10T18:59:16.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9792559700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c15cb60 con 0x7f978c06c720 2023-06-10T18:59:16.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9792559700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0d1660 con 0x7f978c0ce390 2023-06-10T18:59:16.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.970+0000 7f9790af6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f978c0cae10 0x7f978c0cb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39136/0 (socket says 172.21.15.104:39136) 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f9790af6700 1 -- 172.21.15.104:0/3435605029 learned_addr learned my addr 172.21.15.104:0/3435605029 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 --2- 172.21.15.104:0/3435605029 >> v2:172.21.15.114:3300/0 conn(0x7f978c0ce390 0x7f978c160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353834152 0 0) 0x7f978c0d0340 con 0x7f978c0cae10 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97740035b0 con 0x7f978c0cae10 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776882351 0 0) 0x7f978c15cb60 con 0x7f978c06c720 2023-06-10T18:59:16.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978c0d0340 con 0x7f978c06c720 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740378183 0 0) 0x7f97740035b0 con 0x7f978c0cae10 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 >> v2:172.21.15.114:3300/0 conn(0x7f978c0ce390 msgr2=0x7f978c160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 --2- 172.21.15.104:0/3435605029 >> v2:172.21.15.114:3300/0 conn(0x7f978c0ce390 0x7f978c160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 >> v1:172.21.15.114:6789/0 conn(0x7f978c06c720 legacy=0x7f978c0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c161230 con 0x7f978c0cae10 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f97840019f0 con 0x7f978c0cae10 2023-06-10T18:59:16.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 >> v1:172.21.15.104:6789/0 conn(0x7f978c0cae10 legacy=0x7f978c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 0x7f9774003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774004850 0x7f9774004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9774005200 0x7f977400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0d1660 con 0x7f9774003b90 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977400c060 con 0x7f9774005200 2023-06-10T18:59:16.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977400c220 con 0x7f9774004850 2023-06-10T18:59:16.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 0x7f9774003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f9790af6700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9774005200 0x7f977400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978b7fe700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774004850 0x7f9774004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 -- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774004850 msgr2=0x7f9774004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774004850 0x7f9774004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 -- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9774005200 msgr2=0x7f977400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9774005200 0x7f977400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.971+0000 7f978bfff700 1 -- 172.21.15.104:0/3435605029 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740035b0 con 0x7f9774003b90 2023-06-10T18:59:16.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9790af6700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9774005200 0x7f977400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9790af6700 1 -- 172.21.15.104:0/3435605029 reap_dead start 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f978bfff700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 0x7f9774003f80 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f977c009b60 tx=0x7f9774005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975f28e17371267 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c00e6c0 con 0x7f9774003b90 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c00e820 con 0x7f9774003b90 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f978affd700 1 -- 172.21.15.104:0/3435605029 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c00eb90 con 0x7f9774003b90 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 -- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 msgr2=0x7f9774003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 0x7f9774003f80 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f977c009b60 tx=0x7f9774005880 comp rx=0 tx=0).stop 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 -- 172.21.15.104:0/3435605029 shutdown_connections 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 --2- 172.21.15.104:0/3435605029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9774003b90 0x7f9774003f80 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 -- 172.21.15.104:0/3435605029 >> 172.21.15.104:0/3435605029 conn(0x7f978c069e70 msgr2=0x7f978c06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 -- 172.21.15.104:0/3435605029 shutdown_connections 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.972+0000 7f9792559700 1 -- 172.21.15.104:0/3435605029 wait complete. 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.973+0000 7f9792559700 1 Processor -- start 2023-06-10T18:59:16.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 -- start start 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774003b90 0x7f978c0b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c151a60 0x7f978c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b64e0 con 0x7f978c0c4f80 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b44c0 con 0x7f978c151a60 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.974+0000 7f9792559700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c151ed0 con 0x7f9774003b90 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41320/0 (socket says 172.21.15.104:41320) 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 -- 172.21.15.104:0/727545014 learned_addr learned my addr 172.21.15.104:0/727545014 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f978b7fe700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c151a60 0x7f978c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f978bfff700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774003b90 0x7f978c0b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 -- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774003b90 msgr2=0x7f978c0b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774003b90 0x7f978c0b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 -- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c151a60 msgr2=0x7f978c150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:16.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c151a60 0x7f978c150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 -- 172.21.15.104:0/727545014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0c7180 con 0x7f978c0c4f80 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9790af6700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f9784001330 tx=0x7f978400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784017070 con 0x7f978c0c4f80 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978400ebc0 con 0x7f978c0c4f80 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0b6200 con 0x7f978c0c4f80 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400ed50 con 0x7f978c0c4f80 2023-06-10T18:59:16.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.975+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c0c6da0 con 0x7f978c0c4f80 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.976+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0b0b50 con 0x7f978c0c4f80 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.977+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9784013420 con 0x7f978c0c4f80 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.977+0000 7f9788ff9700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 0x7f977806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.977+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9784088cd0 con 0x7f978c0c4f80 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.977+0000 7f978bfff700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 0x7f977806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:16.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.978+0000 7f978bfff700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 0x7f977806d190 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f977c0090b0 tx=0x7f977c01f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:16.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:16.981+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f978c0b0b50 con 0x7f978c0c4f80 2023-06-10T18:59:17.172 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.166+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f978c0c7180 con 0x7f978c0c4f80 2023-06-10T18:59:17.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.173+0000 7f9788ff9700 1 -- 172.21.15.104:0/727545014 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f978c0c7180 con 0x7f978c0c4f80 2023-06-10T18:59:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 msgr2=0x7f977806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 0x7f977806d190 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f977c0090b0 tx=0x7f977c01f040 comp rx=0 tx=0).stop 2023-06-10T18:59:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 msgr2=0x7f978c0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f9784001330 tx=0x7f978400e440 comp rx=0 tx=0).stop 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 shutdown_connections 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f978c0c4f80 0x7f978c0b35f0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9774003b90 0x7f978c0b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f977806ace0 0x7f977806d190 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 --2- 172.21.15.104:0/727545014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c151a60 0x7f978c150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 >> 172.21.15.104:0/727545014 conn(0x7f978c069e70 msgr2=0x7f978c15b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:17.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 shutdown_connections 2023-06-10T18:59:17.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.175+0000 7f9792559700 1 -- 172.21.15.104:0/727545014 wait complete. 2023-06-10T18:59:17.193 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T18:59:17.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:59:17.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T18:59:17.279 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T18:59:17.279 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:17.280 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:17.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T18:59:17.297 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:59:17.297 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:17.318 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:59:17.319 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:17.320 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:17.353 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T18:59:17.353 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T18:59:17.353 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:17.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T18:59:17.378 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-06-10T18:59:17.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:17.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.506+0000 7f4579d30700 1 Processor -- start 2023-06-10T18:59:17.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 -- start start 2023-06-10T18:59:17.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 0x7f45740ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 0x7f45740ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740cec20 con 0x7f45740ce0b0 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45740cf430 con 0x7f45740ada30 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.507+0000 7f4579d30700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740cee70 con 0x7f45740ade00 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f45737fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 0x7f45740ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 0x7f45740ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4573fff700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45740ce0b0 0x7f45740ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:45002/0 (socket says 172.21.15.104:45002) 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4573fff700 1 -- 172.21.15.104:0/839078350 learned_addr learned my addr 172.21.15.104:0/839078350 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 -- 172.21.15.104:0/839078350 >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 msgr2=0x7f45740ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 --2- 172.21.15.104:0/839078350 >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 0x7f45740ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 -- 172.21.15.104:0/839078350 >> v1:172.21.15.114:6789/0 conn(0x7f45740ce0b0 legacy=0x7f45740ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 -- 172.21.15.104:0/839078350 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740cf150 con 0x7f45740ade00 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f45737fe700 1 --2- 172.21.15.104:0/839078350 >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 0x7f45740ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:17.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f4572ffd700 1 --2- 172.21.15.104:0/839078350 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 0x7f45740ae930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f455c009840 tx=0x7f455c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c130efc162d86771 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f45727fc700 1 -- 172.21.15.104:0/839078350 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c011830 con 0x7f45740ade00 2023-06-10T18:59:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f45727fc700 1 -- 172.21.15.104:0/839078350 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c011990 con 0x7f45740ade00 2023-06-10T18:59:17.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.508+0000 7f45727fc700 1 -- 172.21.15.104:0/839078350 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c011d20 con 0x7f45740ade00 2023-06-10T18:59:17.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 -- 172.21.15.104:0/839078350 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 msgr2=0x7f45740ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 --2- 172.21.15.104:0/839078350 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 0x7f45740ae930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f455c009840 tx=0x7f455c009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 -- 172.21.15.104:0/839078350 shutdown_connections 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 --2- 172.21.15.104:0/839078350 >> v2:172.21.15.114:3300/0 conn(0x7f45740ada30 0x7f45740ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 --2- 172.21.15.104:0/839078350 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ade00 0x7f45740ae930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 -- 172.21.15.104:0/839078350 >> 172.21.15.104:0/839078350 conn(0x7f457406b8a0 msgr2=0x7f45740cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 -- 172.21.15.104:0/839078350 shutdown_connections 2023-06-10T18:59:17.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.509+0000 7f4579d30700 1 -- 172.21.15.104:0/839078350 wait complete. 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.510+0000 7f4579d30700 1 Processor -- start 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.510+0000 7f4579d30700 1 -- start start 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 0x7f45740b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ce0b0 0x7f45740cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45740b4460 con 0x7f45740b3470 2023-06-10T18:59:17.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45740b38e0 con 0x7f45740ada30 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4579d30700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740b3bc0 con 0x7f45740ce0b0 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4573fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4573fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41334/0 (socket says 172.21.15.104:41334) 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4573fff700 1 -- 172.21.15.104:0/902270305 learned_addr learned my addr 172.21.15.104:0/902270305 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f4572ffd700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ce0b0 0x7f45740cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.511+0000 7f45737fe700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 0x7f45740b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 -- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ce0b0 msgr2=0x7f45740cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ce0b0 0x7f45740cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 -- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 msgr2=0x7f45740b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 0x7f45740b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 -- 172.21.15.104:0/902270305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4564009430 con 0x7f45740b3470 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f45737fe700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 0x7f45740b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4573fff700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f456c00cbc0 tx=0x7f456c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c00ab50 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c00acb0 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c0094a0 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c012600 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.512+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45740b4180 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.513+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4574099340 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.513+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f456c01b3b0 con 0x7f45740b3470 2023-06-10T18:59:17.524 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.514+0000 7f456bfff700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 0x7f455406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.514+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f456c088f80 con 0x7f45740b3470 2023-06-10T18:59:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.514+0000 7f45737fe700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 0x7f455406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.514+0000 7f45737fe700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 0x7f455406d190 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f4564009c30 tx=0x7f4564008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.525 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.518+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f456c0bb310 con 0x7f45740b3470 2023-06-10T18:59:17.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.747+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f457406cf10 con 0x7f45740b3470 2023-06-10T18:59:17.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.747+0000 7f456bfff700 1 -- 172.21.15.104:0/902270305 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+3544 (secure 0 0 0) 0x7f456c017090 con 0x7f45740b3470 2023-06-10T18:59:17.753 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:17.753 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6834/1516363849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1516363849},{"type":"v1","addr":"172.21.15.104:6834","nonce":1516363849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2732878935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2732878935},{"type":"v1","addr":"172.21.15.114:6836","nonce":2732878935}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/622446954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":622446954},{"type":"v1","addr":"172.21.15.114:6837","nonce":622446954}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112},{"gid":10590,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3061477274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6833","nonce":3061477274},{"type":"v1","addr":"172.21.15.104:6835","nonce":3061477274}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[]} 2023-06-10T18:59:17.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 msgr2=0x7f455406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 0x7f455406d190 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f4564009c30 tx=0x7f4564008040 comp rx=0 tx=0).stop 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 msgr2=0x7f457409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f456c00cbc0 tx=0x7f456c00cf80 comp rx=0 tx=0).stop 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 shutdown_connections 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f455406ace0 0x7f455406d190 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45740ada30 0x7f45740b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45740b3470 0x7f457409b880 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 --2- 172.21.15.104:0/902270305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45740ce0b0 0x7f45740cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 >> 172.21.15.104:0/902270305 conn(0x7f457406b8a0 msgr2=0x7f45740b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:17.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 shutdown_connections 2023-06-10T18:59:17.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.749+0000 7f4579d30700 1 -- 172.21.15.104:0/902270305 wait complete. 2023-06-10T18:59:17.757 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 112 2023-06-10T18:59:17.767 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-10T18:59:17.767 INFO:tasks.ceph.mds.a:Stopping old one... 2023-06-10T18:59:17.767 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T18:59:17.767 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:17.768 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-10T18:59:17.768 INFO:tasks.ceph.mds.c:Stopping old one... 2023-06-10T18:59:17.768 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T18:59:17.768 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:17.769 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-10T18:59:17.769 INFO:tasks.ceph.mds.b:Stopping old one... 2023-06-10T18:59:17.769 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T18:59:17.769 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:17.770 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-10T18:59:17.770 INFO:tasks.ceph.mds.d:Stopping old one... 2023-06-10T18:59:17.770 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T18:59:17.770 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:17.777 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T18:59:17.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-10T18:59:17.778 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T18:59:17.778 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-10T18:59:17.780 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T18:59:17.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-10T18:59:17.781 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T18:59:17.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-10T18:59:17.819 INFO:tasks.ceph.mds.a:Started 2023-06-10T18:59:17.819 DEBUG:teuthology.parallel:result is None 2023-06-10T18:59:17.820 INFO:tasks.ceph.mds.c:Started 2023-06-10T18:59:17.820 DEBUG:teuthology.parallel:result is None 2023-06-10T18:59:17.822 INFO:tasks.ceph.mds.b:Started 2023-06-10T18:59:17.823 DEBUG:teuthology.parallel:result is None 2023-06-10T18:59:17.823 INFO:tasks.ceph.mds.d:Started 2023-06-10T18:59:17.824 DEBUG:teuthology.parallel:result is None 2023-06-10T18:59:17.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-06-10T18:59:17.900 INFO:tasks.ceph.mds.c.smithi104.stdout:starting mds.c at 2023-06-10T18:59:17.912 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2023-06-10T18:59:17.913 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2023-06-10T18:59:17.915 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T18:59:17.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.964+0000 7fb577ef4700 1 Processor -- start 2023-06-10T18:59:17.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.964+0000 7fb577ef4700 1 -- start start 2023-06-10T18:59:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb577ef4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb570070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb577ef4700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb5700c9350 0x7fb5700ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb577ef4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57015c050 con 0x7fb5700ce510 2023-06-10T18:59:17.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb577ef4700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700ceea0 con 0x7fb5700c9350 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb577ef4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c9970 con 0x7fb57015bc80 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb57548f700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb5700ce510 0x7fb5700c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39190/0 (socket says 172.21.15.104:39190) 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.965+0000 7fb57548f700 1 -- 172.21.15.104:0/2918095730 learned_addr learned my addr 172.21.15.104:0/2918095730 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb575c90700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb570070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb576491700 1 --2- 172.21.15.104:0/2918095730 >> v2:172.21.15.114:3300/0 conn(0x7fb5700c9350 0x7fb5700ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107563536 0 0) 0x7fb57015c050 con 0x7fb5700ce510 2023-06-10T18:59:17.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c0034d0 con 0x7fb5700ce510 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787381776 0 0) 0x7fb55c0034d0 con 0x7fb5700ce510 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 msgr2=0x7fb570070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb570070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 >> v2:172.21.15.114:3300/0 conn(0x7fb5700c9350 msgr2=0x7fb5700ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 --2- 172.21.15.104:0/2918095730 >> v2:172.21.15.114:3300/0 conn(0x7fb5700c9350 0x7fb5700ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5700ccb70 con 0x7fb5700ce510 2023-06-10T18:59:17.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fb56c001670 con 0x7fb5700ce510 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb576491700 1 --2- 172.21.15.104:0/2918095730 >> v2:172.21.15.114:3300/0 conn(0x7fb5700c9350 0x7fb5700ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 >> v1:172.21.15.104:6789/0 conn(0x7fb5700ce510 legacy=0x7fb5700c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb55c003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb55c00a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb55c00ae80 0x7fb55c00d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700ceea0 con 0x7fb55c00ae80 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c9970 con 0x7fb55c004070 2023-06-10T18:59:17.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c00dc30 con 0x7fb57015bc80 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb575c90700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb55c003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb55c00a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb576491700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb55c00ae80 0x7fb55c00d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 -- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 msgr2=0x7fb55c003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb55c003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 -- 172.21.15.104:0/2918095730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb55c00ae80 msgr2=0x7fb55c00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:17.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb55c00ae80 0x7fb55c00d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb57548f700 1 -- 172.21.15.104:0/2918095730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0034d0 con 0x7fb55c004070 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb575c90700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb57015bc80 0x7fb55c003af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.966+0000 7fb576491700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb55c00ae80 0x7fb55c00d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb57548f700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb55c00a8f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c320 tx=0x7fb55c003780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b011a969eecef86f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00ce60 con 0x7fb55c004070 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c005960 con 0x7fb55c004070 2023-06-10T18:59:17.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb574c8e700 1 -- 172.21.15.104:0/2918095730 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c005cf0 con 0x7fb55c004070 2023-06-10T18:59:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 -- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 msgr2=0x7fb55c00a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 --2- 172.21.15.104:0/2918095730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb55c00a8f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c320 tx=0x7fb55c003780 comp rx=0 tx=0).stop 2023-06-10T18:59:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb576491700 1 -- 172.21.15.104:0/2918095730 reap_dead start 2023-06-10T18:59:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 -- 172.21.15.104:0/2918095730 shutdown_connections 2023-06-10T18:59:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 -- 172.21.15.104:0/2918095730 >> 172.21.15.104:0/2918095730 conn(0x7fb57006ab00 msgr2=0x7fb5700b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 -- 172.21.15.104:0/2918095730 shutdown_connections 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.967+0000 7fb577ef4700 1 -- 172.21.15.104:0/2918095730 wait complete. 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.968+0000 7fb577ef4700 1 Processor -- start 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 -- start start 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb5700c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 0x7fb5700acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5701f1f10 con 0x7fb5701f1530 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb570149630 con 0x7fb55c004070 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb577ef4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700c6c00 con 0x7fb5701f21b0 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb576491700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb576491700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41376/0 (socket says 172.21.15.104:41376) 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb576491700 1 -- 172.21.15.104:0/534025236 learned_addr learned my addr 172.21.15.104:0/534025236 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb575c90700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb5700c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.969+0000 7fb57548f700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 0x7fb5700acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 -- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 msgr2=0x7fb5700acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 0x7fb5700acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 -- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 msgr2=0x7fb5700c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb5700c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 -- 172.21.15.104:0/534025236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5700c6ee0 con 0x7fb5701f1530 2023-06-10T18:59:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb575c90700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb5700c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb57548f700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 0x7fb5700acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb576491700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fb5700b46a0 tx=0x7fb56800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800e040 con 0x7fb5701f1530 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56800f040 con 0x7fb5701f1530 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5700c71c0 con 0x7fb5701f1530 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568003c20 con 0x7fb5701f1530 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.970+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5700c6840 con 0x7fb5701f1530 2023-06-10T18:59:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.971+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5700c69d0 con 0x7fb5701f1530 2023-06-10T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.971+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb568004030 con 0x7fb5701f1530 2023-06-10T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.972+0000 7fb5667fc700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 0x7fb55406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.972+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb5680898e0 con 0x7fb5701f1530 2023-06-10T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.972+0000 7fb575c90700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 0x7fb55406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.972+0000 7fb575c90700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 0x7fb55406d190 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb5700cec60 tx=0x7fb560008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:17.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:17.976+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb568054b20 con 0x7fb5701f1530 2023-06-10T18:59:18.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:18.177+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb570149ac0 con 0x7fb5701f1530 2023-06-10T18:59:19.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.179+0000 7fb5667fc700 1 -- 172.21.15.104:0/534025236 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v183) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb570149ac0 con 0x7fb5701f1530 2023-06-10T18:59:19.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 msgr2=0x7fb55406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 0x7fb55406d190 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb5700cec60 tx=0x7fb560008040 comp rx=0 tx=0).stop 2023-06-10T18:59:19.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 msgr2=0x7fb5701f19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fb5700b46a0 tx=0x7fb56800f770 comp rx=0 tx=0).stop 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 shutdown_connections 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb55406ace0 0x7fb55406d190 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb55c004070 0x7fb5700c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5701f1530 0x7fb5701f19d0 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 --2- 172.21.15.104:0/534025236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5701f21b0 0x7fb5700acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 >> 172.21.15.104:0/534025236 conn(0x7fb57006ab00 msgr2=0x7fb57015ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:19.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 shutdown_connections 2023-06-10T18:59:19.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.182+0000 7fb577ef4700 1 -- 172.21.15.104:0/534025236 wait complete. 2023-06-10T18:59:19.189 INFO:teuthology.orchestra.run.smithi104.stderr: removed all blocklist entries 2023-06-10T18:59:19.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-06-10T18:59:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.314+0000 7fe010e24700 1 Processor -- start 2023-06-10T18:59:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.314+0000 7fe010e24700 1 -- start start 2023-06-10T18:59:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe010e24700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe00c06b1f0 0x7fe00c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe010e24700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe00c08a3d0 0x7fe00c0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe010e24700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c0cadd0 con 0x7fe00c0ce350 2023-06-10T18:59:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe010e24700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0cfeb0 con 0x7fe00c06b1f0 2023-06-10T18:59:19.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe010e24700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0cb0b0 con 0x7fe00c08a3d0 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe00c06b1f0 0x7fe00c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe00c06b1f0 0x7fe00c0cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41384/0 (socket says 172.21.15.104:41384) 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 -- 172.21.15.104:0/1925966318 learned_addr learned my addr 172.21.15.104:0/1925966318 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe009d9b700 1 --1- 172.21.15.104:0/1925966318 >> v1:172.21.15.104:6789/0 conn(0x7fe00c0ce350 0x7fe00c0cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39212/0 (socket says 172.21.15.104:39212) 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00ad9d700 1 --2- 172.21.15.104:0/1925966318 >> v2:172.21.15.114:3300/0 conn(0x7fe00c08a3d0 0x7fe00c0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 -- 172.21.15.104:0/1925966318 >> v2:172.21.15.114:3300/0 conn(0x7fe00c08a3d0 msgr2=0x7fe00c0cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:19.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 --2- 172.21.15.104:0/1925966318 >> v2:172.21.15.114:3300/0 conn(0x7fe00c08a3d0 0x7fe00c0cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 -- 172.21.15.104:0/1925966318 >> v1:172.21.15.104:6789/0 conn(0x7fe00c0ce350 legacy=0x7fe00c0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 -- 172.21.15.104:0/1925966318 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c0cf550 con 0x7fe00c06b1f0 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00959a700 1 -- 172.21.15.104:0/1925966318 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319868951 0 0) 0x7fe00c0cadd0 con 0x7fe00c0ce350 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00a59c700 1 --2- 172.21.15.104:0/1925966318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c06b1f0 0x7fe00c0cd090 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0cad60 tx=0x7fdffc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2651c11ee9b2e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00959a700 1 -- 172.21.15.104:0/1925966318 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc00b870 con 0x7fe00c06b1f0 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00959a700 1 -- 172.21.15.104:0/1925966318 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc00b9d0 con 0x7fe00c06b1f0 2023-06-10T18:59:19.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.315+0000 7fe00959a700 1 -- 172.21.15.104:0/1925966318 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc00bd60 con 0x7fe00c06b1f0 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 -- 172.21.15.104:0/1925966318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c06b1f0 msgr2=0x7fe00c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 --2- 172.21.15.104:0/1925966318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c06b1f0 0x7fe00c0cd090 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0cad60 tx=0x7fdffc009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 -- 172.21.15.104:0/1925966318 shutdown_connections 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 --2- 172.21.15.104:0/1925966318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c06b1f0 0x7fe00c0cd090 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 --2- 172.21.15.104:0/1925966318 >> v2:172.21.15.114:3300/0 conn(0x7fe00c08a3d0 0x7fe00c0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 -- 172.21.15.104:0/1925966318 >> 172.21.15.104:0/1925966318 conn(0x7fe00c089fd0 msgr2=0x7fe00c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 -- 172.21.15.104:0/1925966318 shutdown_connections 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.316+0000 7fe010e24700 1 -- 172.21.15.104:0/1925966318 wait complete. 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.317+0000 7fe010e24700 1 Processor -- start 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.317+0000 7fe010e24700 1 -- start start 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe00c06b1f0 0x7fe00c0adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe00c08a3d0 0x7fe00c151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0ae460 con 0x7fe00c0ce350 2023-06-10T18:59:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0c7850 con 0x7fe00c08a3d0 2023-06-10T18:59:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe010e24700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0ae230 con 0x7fe00c06b1f0 2023-06-10T18:59:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe00ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe00ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41392/0 (socket says 172.21.15.104:41392) 2023-06-10T18:59:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe00ad9d700 1 -- 172.21.15.104:0/382784056 learned_addr learned my addr 172.21.15.104:0/382784056 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe009d9b700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe00c08a3d0 0x7fe00c151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.318+0000 7fe00a59c700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe00c06b1f0 0x7fe00c0adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 -- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe00c06b1f0 msgr2=0x7fe00c0adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe00c06b1f0 0x7fe00c0adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 -- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe00c08a3d0 msgr2=0x7fe00c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe00c08a3d0 0x7fe00c151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 -- 172.21.15.104:0/382784056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc009550 con 0x7fe00c0ce350 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe00ad9d700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fe00400cc30 tx=0x7fe00400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00400ab70 con 0x7fe00c0ce350 2023-06-10T18:59:19.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00400acd0 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c0c7a60 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004003c40 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.319+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c0c8020 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.320+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c074050 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.320+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe00400f3f0 con 0x7fe00c0ce350 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.321+0000 7fdffaffd700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 0x7fdfec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.321+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fe004089450 con 0x7fe00c0ce350 2023-06-10T18:59:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.321+0000 7fe00a59c700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 0x7fdfec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.321+0000 7fe00a59c700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 0x7fdfec06d190 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fdffc014040 tx=0x7fdffc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.325+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe004054710 con 0x7fe00c0ce350 2023-06-10T18:59:19.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.586+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe00c0b63e0 con 0x7fe00c0ce350 2023-06-10T18:59:19.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.587+0000 7fdffaffd700 1 -- 172.21.15.104:0/382784056 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe004057d30 con 0x7fe00c0ce350 2023-06-10T18:59:19.593 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:19.593 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:59:19.594 INFO:teuthology.orchestra.run.smithi104.stdout: "auth_dump": [ 2023-06-10T18:59:19.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.594 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.a", 2023-06-10T18:59:19.595 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkWZubORAADqXGGsh+Gk3JuEkAP3vlVA==", 2023-06-10T18:59:19.595 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.595 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.596 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:59:19.596 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:59:19.596 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.597 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.b", 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkchw/ChAAtiMWN0vVEyUsERzP+EXc+w==", 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.598 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.c", 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkJFkxBRAACAu3htJ1Vcqz8V6L3UoVKQ==", 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:59:19.599 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mds.d", 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkj9CLERAAPpVrmdyoCFO94Z4rt0jXpw==", 2023-06-10T18:59:19.600 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:59:19.601 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:59:19.601 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.0", 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBBxIRktvlbLxAAkhDmNkEJ9mze5vvardxi5Q==", 2023-06-10T18:59:19.602 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.1", 2023-06-10T18:59:19.603 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBExIRk6UhnLhAAL28lDd/t20Qb0Yh/URgtKw==", 2023-06-10T18:59:19.604 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.604 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.604 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.604 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.2", 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBHxIRkxVfhLRAALMgQeIC3BrvCqNzccjGeJg==", 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.605 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.3", 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBKxIRkY/OLLRAAhWfZSKmt9oFL3NDoxfJn0Q==", 2023-06-10T18:59:19.606 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.607 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.607 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.607 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.608 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.4", 2023-06-10T18:59:19.608 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBQxIRkE+haMhAAagdV7JzNzR0E+F55TCFQjA==", 2023-06-10T18:59:19.609 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.609 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.5", 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBTxIRkgxxcMRAABMoB1IPWTbdRZKDCb6ZdRA==", 2023-06-10T18:59:19.610 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.611 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.6", 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBWxIRk4tSCMBAAaK4Bc8jooFpNADSvFc6Luw==", 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "osd.7", 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQBZxIRk/DnmLxAAfDbNS0fKF9CNJD/OshqfQA==", 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow profile osd", 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile osd", 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.613 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.0", 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk/G/JFRAAcku3NISbCDjgvNkgBvepOA==", 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow r", 2023-06-10T18:59:19.614 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.1", 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRk9svKHhAAj28I9htBtqzrBqLWW+Pchw==", 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.2", 2023-06-10T18:59:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkCsh6GxAAp0eQ+ujksiG8wYwY+AD3yQ==", 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.3", 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA+xIRkQPWBJBAASe/j8yUQWBqWgNgwiCjaIA==", 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow", 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow r", 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow rw", 2023-06-10T18:59:19.618 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow rwx" 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "client.admin", 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRkTtKLHBAAPZYoNE0byTE0s3xjaC0+xQ==", 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.619 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": "allow *", 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow *", 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.x", 2023-06-10T18:59:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk1uZINhAAlPgDl9pZt1MD/O8jN1/ABg==", 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:19.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "entity": "mgr.y", 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "key": "AQA9xIRk+oA0MhAAfcpO/nMo82K3OIPtL+rLRw==", 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": { 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": "allow *", 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": "allow profile mgr", 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": "allow *" 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:19.623 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:59:19.623 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:59:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 msgr2=0x7fdfec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 0x7fdfec06d190 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fdffc014040 tx=0x7fdffc008030 comp rx=0 tx=0).stop 2023-06-10T18:59:19.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 msgr2=0x7fe00c0c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fe00400cc30 tx=0x7fe00400a1f0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 shutdown_connections 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe00c0ce350 0x7fe00c0c63b0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe00c06b1f0 0x7fe00c0adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdfec06ace0 0x7fdfec06d190 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 --2- 172.21.15.104:0/382784056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe00c08a3d0 0x7fe00c151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 >> 172.21.15.104:0/382784056 conn(0x7fe00c089fd0 msgr2=0x7fe00c0ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 shutdown_connections 2023-06-10T18:59:19.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.589+0000 7fe010e24700 1 -- 172.21.15.104:0/382784056 wait complete. 2023-06-10T18:59:19.626 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:19.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.741+0000 7facdcfc6700 1 Processor -- start 2023-06-10T18:59:19.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.741+0000 7facdcfc6700 1 -- start start 2023-06-10T18:59:19.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facdcfc6700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facd80d2b20 0x7facd80d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facdcfc6700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd80cd430 con 0x7facd80d0520 2023-06-10T18:59:19.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facdcfc6700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd80cd710 con 0x7facd80ccfa0 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facdcfc6700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd80cd9f0 con 0x7facd80d2b20 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facd659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facd80d2b20 0x7facd80d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facd659c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7facd80d2b20 0x7facd80d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41396/0 (socket says 172.21.15.104:41396) 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.742+0000 7facd659c700 1 -- 172.21.15.104:0/2794433895 learned_addr learned my addr 172.21.15.104:0/2794433895 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd659c700 1 -- 172.21.15.104:0/2794433895 >> v1:172.21.15.114:6789/0 conn(0x7facd80ccfa0 legacy=0x7facd80b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd659c700 1 -- 172.21.15.104:0/2794433895 >> v1:172.21.15.104:6789/0 conn(0x7facd80d0520 legacy=0x7facd80ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd559a700 1 -- 172.21.15.104:0/2794433895 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505736872 0 0) 0x7facd80cd430 con 0x7facd80d0520 2023-06-10T18:59:19.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd659c700 1 -- 172.21.15.104:0/2794433895 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80cdcd0 con 0x7facd80d2b20 2023-06-10T18:59:19.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd659c700 1 --2- 172.21.15.104:0/2794433895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd80d2b20 0x7facd80d2f00 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7faccc0098f0 tx=0x7faccc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66f8266ebf9c2cf server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd559a700 1 -- 172.21.15.104:0/2794433895 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc004230 con 0x7facd80d2b20 2023-06-10T18:59:19.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd559a700 1 -- 172.21.15.104:0/2794433895 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc004390 con 0x7facd80d2b20 2023-06-10T18:59:19.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facd559a700 1 -- 172.21.15.104:0/2794433895 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0036a0 con 0x7facd80d2b20 2023-06-10T18:59:19.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facdcfc6700 1 -- 172.21.15.104:0/2794433895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd80d2b20 msgr2=0x7facd80d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.743+0000 7facdcfc6700 1 --2- 172.21.15.104:0/2794433895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd80d2b20 0x7facd80d2f00 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7faccc0098f0 tx=0x7faccc009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.744+0000 7facdcfc6700 1 -- 172.21.15.104:0/2794433895 shutdown_connections 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.744+0000 7facdcfc6700 1 --2- 172.21.15.104:0/2794433895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd80d2b20 0x7facd80d2f00 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.744+0000 7facdcfc6700 1 -- 172.21.15.104:0/2794433895 >> 172.21.15.104:0/2794433895 conn(0x7facd8071ec0 msgr2=0x7facd809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.744+0000 7facdcfc6700 1 -- 172.21.15.104:0/2794433895 shutdown_connections 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.744+0000 7facdcfc6700 1 -- 172.21.15.104:0/2794433895 wait complete. 2023-06-10T18:59:19.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.745+0000 7facdcfc6700 1 Processor -- start 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.745+0000 7facdcfc6700 1 -- start start 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facd80ccfa0 0x7facd81590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80d0520 0x7facd81f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b6950 con 0x7facd81f33d0 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b6c30 con 0x7facd80d0520 2023-06-10T18:59:19.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b6f10 con 0x7facd80ccfa0 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41406/0 (socket says 172.21.15.104:41406) 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 -- 172.21.15.104:0/904858082 learned_addr learned my addr 172.21.15.104:0/904858082 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd659c700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facd80ccfa0 0x7facd81590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd5d9b700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80d0520 0x7facd81f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 -- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facd80ccfa0 msgr2=0x7facd81590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facd80ccfa0 0x7facd81590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 -- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80d0520 msgr2=0x7facd81f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80d0520 0x7facd81f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 -- 172.21.15.104:0/904858082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc009550 con 0x7facd81f33d0 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facd6d9d700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7facd000c810 tx=0x7facd000cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0003e30 con 0x7facd81f33d0 2023-06-10T18:59:19.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0005070 con 0x7facd81f33d0 2023-06-10T18:59:19.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd80b71f0 con 0x7facd81f33d0 2023-06-10T18:59:19.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0005420 con 0x7facd81f33d0 2023-06-10T18:59:19.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.746+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd80b0710 con 0x7facd81f33d0 2023-06-10T18:59:19.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.747+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd80b0990 con 0x7facd81f33d0 2023-06-10T18:59:19.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.748+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7facd0005580 con 0x7facd81f33d0 2023-06-10T18:59:19.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.748+0000 7facc6ffd700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 0x7facb806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:19.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.748+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7facd0088e90 con 0x7facd81f33d0 2023-06-10T18:59:19.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.748+0000 7facd659c700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 0x7facb806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:19.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.749+0000 7facd659c700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 0x7facb806d190 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7faccc009130 tx=0x7faccc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:19.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.753+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7facd0054140 con 0x7facd81f33d0 2023-06-10T18:59:19.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.942+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facd80b6a90 con 0x7facd81f33d0 2023-06-10T18:59:19.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.942+0000 7facc6ffd700 1 -- 172.21.15.104:0/904858082 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+14902 (secure 0 0 0) 0x7facd80b6a90 con 0x7facd81f33d0 2023-06-10T18:59:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:19.948 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":183,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:19.173017+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 msgr2=0x7facb806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 0x7facb806d190 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7faccc009130 tx=0x7faccc008030 comp rx=0 tx=0).stop 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 msgr2=0x7facd80afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7facd000c810 tx=0x7facd000cb20 comp rx=0 tx=0).stop 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 shutdown_connections 2023-06-10T18:59:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facd81f33d0 0x7facd80afff0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facd80ccfa0 0x7facd81590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7facb806ace0 0x7facb806d190 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 --2- 172.21.15.104:0/904858082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd80d0520 0x7facd81f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.944+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 >> 172.21.15.104:0/904858082 conn(0x7facd8071ec0 msgr2=0x7facd806b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.945+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 shutdown_connections 2023-06-10T18:59:19.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:19.945+0000 7facdcfc6700 1 -- 172.21.15.104:0/904858082 wait complete. 2023-06-10T18:59:19.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:59:19.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:59:20.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.080+0000 7f4d16405700 1 Processor -- start 2023-06-10T18:59:20.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d16405700 1 -- start start 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d16405700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4d100b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d16405700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1006e250 con 0x7f4d1006cde0 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d16405700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1015c950 con 0x7f4d1006e6e0 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d16405700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100b1e80 con 0x7f4d1015c580 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d149a2700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4d1006cde0 0x7f4d100ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39240/0 (socket says 172.21.15.104:39240) 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.081+0000 7f4d149a2700 1 -- 172.21.15.104:0/3434109971 learned_addr learned my addr 172.21.15.104:0/3434109971 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:20.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559899485 0 0) 0x7f4d1006e250 con 0x7f4d1006cde0 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc0035b0 con 0x7f4d1006cde0 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4d100b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453183527 0 0) 0x7f4cfc0035b0 con 0x7f4d1006cde0 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 msgr2=0x7f4d100b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4d100b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 >> v1:172.21.15.114:6789/0 conn(0x7f4d1006e6e0 legacy=0x7f4d100b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1006d920 con 0x7f4d1006cde0 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4d080019f0 con 0x7f4d1006cde0 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 >> v1:172.21.15.104:6789/0 conn(0x7f4d1006cde0 legacy=0x7f4d100ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4cfc003a30 0x7f4cfc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4cfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc004d90 0x7f4cfc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100b1e80 con 0x7f4cfc003a30 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc00bac0 con 0x7f4cfc004d90 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc00bc50 con 0x7f4d1015c580 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4cfc003a30 0x7f4cfc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0f7fe700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4cfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d149a2700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc004d90 0x7f4cfc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 -- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 msgr2=0x7f4cfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d1015c580 0x7f4cfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 -- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc004d90 msgr2=0x7f4cfc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc004d90 0x7f4cfc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.082+0000 7f4d0ffff700 1 -- 172.21.15.104:0/3434109971 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0035b0 con 0x7f4cfc003a30 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4cfc003a30 0x7f4cfc003e40 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009b40 tx=0x7f4cfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf7d571c7df226c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00005a20 con 0x7f4cfc003a30 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00005b80 con 0x7f4cfc003a30 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d0effd700 1 -- 172.21.15.104:0/3434109971 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000e6c0 con 0x7f4cfc003a30 2023-06-10T18:59:20.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 -- 172.21.15.104:0/3434109971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4cfc003a30 msgr2=0x7f4cfc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 --2- 172.21.15.104:0/3434109971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4cfc003a30 0x7f4cfc003e40 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009b40 tx=0x7f4cfc005880 comp rx=0 tx=0).stop 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d149a2700 1 -- 172.21.15.104:0/3434109971 reap_dead start 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 -- 172.21.15.104:0/3434109971 shutdown_connections 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 -- 172.21.15.104:0/3434109971 >> 172.21.15.104:0/3434109971 conn(0x7f4d1006b8a0 msgr2=0x7f4d100cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 -- 172.21.15.104:0/3434109971 shutdown_connections 2023-06-10T18:59:20.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.083+0000 7f4d16405700 1 -- 172.21.15.104:0/3434109971 wait complete. 2023-06-10T18:59:20.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.084+0000 7f4d16405700 1 Processor -- start 2023-06-10T18:59:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 -- start start 2023-06-10T18:59:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc003a30 0x7f4d10157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d100b46b0 0x7f4d100ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100ae240 con 0x7f4d100ad860 2023-06-10T18:59:20.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100acac0 con 0x7f4d100b46b0 2023-06-10T18:59:20.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.085+0000 7f4d16405700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100acda0 con 0x7f4cfc003a30 2023-06-10T18:59:20.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41426/0 (socket says 172.21.15.104:41426) 2023-06-10T18:59:20.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 -- 172.21.15.104:0/489080804 learned_addr learned my addr 172.21.15.104:0/489080804 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:20.101 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc003a30 0x7f4d10157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d0f7fe700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d100b46b0 0x7f4d100ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 -- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc003a30 msgr2=0x7f4d10157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc003a30 0x7f4d10157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.102 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 -- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d100b46b0 msgr2=0x7f4d100ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d100b46b0 0x7f4d100ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 -- 172.21.15.104:0/489080804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100ad080 con 0x7f4d100ad860 2023-06-10T18:59:20.103 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d149a2700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000f80 tx=0x7f4d0800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800acb0 con 0x7f4d100ad860 2023-06-10T18:59:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0800ae30 con 0x7f4d100ad860 2023-06-10T18:59:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d100ad360 con 0x7f4d100ad860 2023-06-10T18:59:20.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08013630 con 0x7f4d100ad860 2023-06-10T18:59:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.086+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d10158300 con 0x7f4d100ad860 2023-06-10T18:59:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.087+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d100c3ca0 con 0x7f4d100ad860 2023-06-10T18:59:20.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.088+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4d08013790 con 0x7f4d100ad860 2023-06-10T18:59:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.088+0000 7f4d0cff9700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 0x7f4cf006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.088+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4d08089de0 con 0x7f4d100ad860 2023-06-10T18:59:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.088+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 0x7f4cf006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.089+0000 7f4d0ffff700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 0x7f4cf006d190 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.092+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4d100c3ca0 con 0x7f4d100ad860 2023-06-10T18:59:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.278+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d100c3ca0 con 0x7f4d100ad860 2023-06-10T18:59:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.278+0000 7f4d0cff9700 1 -- 172.21.15.104:0/489080804 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v183) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d100c3ca0 con 0x7f4d100ad860 2023-06-10T18:59:20.284 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:59:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 msgr2=0x7f4cf006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 0x7f4cf006d190 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4d00000c00 tx=0x7f4d00008030 comp rx=0 tx=0).stop 2023-06-10T18:59:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 msgr2=0x7f4d100add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000f80 tx=0x7f4d0800a5f0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 shutdown_connections 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d100ad860 0x7f4d100add00 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc003a30 0x7f4d10157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4cf006ace0 0x7f4cf006d190 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 --2- 172.21.15.104:0/489080804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d100b46b0 0x7f4d100ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 >> 172.21.15.104:0/489080804 conn(0x7f4d1006b8a0 msgr2=0x7f4d1015a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 shutdown_connections 2023-06-10T18:59:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.280+0000 7f4d16405700 1 -- 172.21.15.104:0/489080804 wait complete. 2023-06-10T18:59:20.298 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:59:20.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.415+0000 7f8ea919a700 1 Processor -- start 2023-06-10T18:59:20.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.415+0000 7f8ea919a700 1 -- start start 2023-06-10T18:59:20.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea919a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea919a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 0x7f8ea40cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea919a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ce340 con 0x7f8ea40ccff0 2023-06-10T18:59:20.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea919a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea40cd460 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea919a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40d00d0 con 0x7f8ea406af80 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea2d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea359e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 0x7f8ea40cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea2d9d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60814/0 (socket says 172.21.15.104:60814) 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea2d9d700 1 -- 172.21.15.104:0/4085312623 learned_addr learned my addr 172.21.15.104:0/4085312623 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.416+0000 7f8ea259c700 1 --1- 172.21.15.104:0/4085312623 >> v1:172.21.15.114:6789/0 conn(0x7f8ea40ccff0 0x7f8ea406b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:45106/0 (socket says 172.21.15.104:45106) 2023-06-10T18:59:20.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea2d9d700 1 -- 172.21.15.104:0/4085312623 >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 msgr2=0x7f8ea40cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea2d9d700 1 --2- 172.21.15.104:0/4085312623 >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 0x7f8ea40cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea2d9d700 1 -- 172.21.15.104:0/4085312623 >> v1:172.21.15.114:6789/0 conn(0x7f8ea40ccff0 legacy=0x7f8ea406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea2d9d700 1 -- 172.21.15.104:0/4085312623 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40cd680 con 0x7f8ea406af80 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea359e700 1 --2- 172.21.15.104:0/4085312623 >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 0x7f8ea40cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea2d9d700 1 --2- 172.21.15.104:0/4085312623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009920 tx=0x7f8e98009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862f38cd73b9e916 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea1d9b700 1 -- 172.21.15.104:0/4085312623 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98011830 con 0x7f8ea406af80 2023-06-10T18:59:20.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea1d9b700 1 -- 172.21.15.104:0/4085312623 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98011990 con 0x7f8ea406af80 2023-06-10T18:59:20.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea1d9b700 1 -- 172.21.15.104:0/4085312623 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98011d20 con 0x7f8ea406af80 2023-06-10T18:59:20.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea919a700 1 -- 172.21.15.104:0/4085312623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 msgr2=0x7f8ea40d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.417+0000 7f8ea919a700 1 --2- 172.21.15.104:0/4085312623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009920 tx=0x7f8e98009c30 comp rx=0 tx=0).stop 2023-06-10T18:59:20.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 -- 172.21.15.104:0/4085312623 shutdown_connections 2023-06-10T18:59:20.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 --2- 172.21.15.104:0/4085312623 >> v2:172.21.15.114:3300/0 conn(0x7f8ea40ce7d0 0x7f8ea40cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 --2- 172.21.15.104:0/4085312623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea40d0d20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 -- 172.21.15.104:0/4085312623 >> 172.21.15.104:0/4085312623 conn(0x7f8ea4089fd0 msgr2=0x7f8ea408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 -- 172.21.15.104:0/4085312623 shutdown_connections 2023-06-10T18:59:20.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.418+0000 7f8ea919a700 1 -- 172.21.15.104:0/4085312623 wait complete. 2023-06-10T18:59:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.419+0000 7f8ea919a700 1 Processor -- start 2023-06-10T18:59:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.419+0000 7f8ea919a700 1 -- start start 2023-06-10T18:59:20.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea41507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea40ccff0 0x7f8ea41e7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4150d00 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea415aed0 con 0x7f8ea40ccff0 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea919a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea41e7de0 con 0x7f8ea406af80 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41440/0 (socket says 172.21.15.104:41440) 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 -- 172.21.15.104:0/3655204965 learned_addr learned my addr 172.21.15.104:0/3655204965 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea259c700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea40ccff0 0x7f8ea41e7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea2d9d700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea41507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 -- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 msgr2=0x7f8ea41507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea41507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 -- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea40ccff0 msgr2=0x7f8ea41e7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea40ccff0 0x7f8ea41e7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 -- 172.21.15.104:0/3655204965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c009a70 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.420+0000 7f8ea359e700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00efc0 tx=0x7f8e9c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.421+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00ca90 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.421+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00f040 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.421+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e98009550 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.421+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00ce30 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.421+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea415a880 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.422+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea406c600 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.422+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e9c014480 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.422+0000 7f8e8b7fe700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 0x7f8e9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.422+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f8e9c089820 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.423+0000 7f8ea2d9d700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 0x7f8e9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.434 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.423+0000 7f8ea2d9d700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 0x7f8e9006d190 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e98008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.435 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.427+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e9c054b70 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.609+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8ea4074050 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.609+0000 7f8e8b7fe700 1 -- 172.21.15.104:0/3655204965 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v183) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8e9c009cd0 con 0x7f8ea40ce7d0 2023-06-10T18:59:20.615 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:59:20.616 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 msgr2=0x7f8e9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 0x7f8e9006d190 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8e98000c00 tx=0x7f8e98008030 comp rx=0 tx=0).stop 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 msgr2=0x7f8ea40b5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00efc0 tx=0x7f8e9c00c3a0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 shutdown_connections 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ea40ce7d0 0x7f8ea40b5b10 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.617 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ea406af80 0x7f8ea41507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8e9006ace0 0x7f8e9006d190 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 --2- 172.21.15.104:0/3655204965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ea40ccff0 0x7f8ea41e7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 >> 172.21.15.104:0/3655204965 conn(0x7f8ea4089fd0 msgr2=0x7f8ea406fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 shutdown_connections 2023-06-10T18:59:20.618 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.611+0000 7f8ea919a700 1 -- 172.21.15.104:0/3655204965 wait complete. 2023-06-10T18:59:20.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-10T18:59:20.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 Processor -- start 2023-06-10T18:59:20.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 -- start start 2023-06-10T18:59:20.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f49a006e4b0 0x7f49a006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a006d290 con 0x7f49a006e0e0 2023-06-10T18:59:20.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a00ca2d0 con 0x7f49a0069e70 2023-06-10T18:59:20.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.743+0000 7f49a6702700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a0071640 con 0x7f49a006e4b0 2023-06-10T18:59:20.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f49a006e0e0 0x7f49a00c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39272/0 (socket says 172.21.15.104:39272) 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499ffff700 1 -- 172.21.15.104:0/964415279 learned_addr learned my addr 172.21.15.104:0/964415279 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514869043 0 0) 0x7f49a006d290 con 0x7f49a006e0e0 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49880035b0 con 0x7f49a006e0e0 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499f7fe700 1 --2- 172.21.15.104:0/964415279 >> v2:172.21.15.114:3300/0 conn(0x7f49a006e4b0 0x7f49a006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711072000 0 0) 0x7f49880035b0 con 0x7f49a006e0e0 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 >> v2:172.21.15.114:3300/0 conn(0x7f49a006e4b0 msgr2=0x7f49a006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 --2- 172.21.15.104:0/964415279 >> v2:172.21.15.114:3300/0 conn(0x7f49a006e4b0 0x7f49a006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 >> v1:172.21.15.114:6789/0 conn(0x7f49a0069e70 legacy=0x7f49a00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a006d4c0 con 0x7f49a006e0e0 2023-06-10T18:59:20.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f49940017f0 con 0x7f49a006e0e0 2023-06-10T18:59:20.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 >> v1:172.21.15.104:6789/0 conn(0x7f49a006e0e0 legacy=0x7f49a00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988003a00 0x7f4988003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4988004850 0x7f4988004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 0x7f498800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a0071640 con 0x7f4988005400 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498800c280 con 0x7f4988003a00 2023-06-10T18:59:20.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498800c440 con 0x7f4988004850 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499f7fe700 1 --2- 172.21.15.104:0/964415279 >> v2:172.21.15.114:3300/0 conn(0x7f49a006e4b0 0x7f49a006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f49a4c9f700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 0x7f498800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499ffff700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988003a00 0x7f4988003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.744+0000 7f499f7fe700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4988004850 0x7f4988004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 -- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4988004850 msgr2=0x7f4988004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4988004850 0x7f4988004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 -- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988003a00 msgr2=0x7f4988003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:20.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988003a00 0x7f4988003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 -- 172.21.15.104:0/964415279 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49880035b0 con 0x7f4988005400 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 -- 172.21.15.104:0/964415279 reap_dead start 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a4c9f700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 0x7f498800ba10 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f499800c190 tx=0x7f4988005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aa3f7b5fb33936 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998001db0 con 0x7f4988005400 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499800ccf0 con 0x7f4988005400 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f499effd700 1 -- 172.21.15.104:0/964415279 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998003c40 con 0x7f4988005400 2023-06-10T18:59:20.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 -- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 msgr2=0x7f498800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 0x7f498800ba10 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f499800c190 tx=0x7f4988005880 comp rx=0 tx=0).stop 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 -- 172.21.15.104:0/964415279 shutdown_connections 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 --2- 172.21.15.104:0/964415279 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4988005400 0x7f498800ba10 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 -- 172.21.15.104:0/964415279 >> 172.21.15.104:0/964415279 conn(0x7f49a006b8a0 msgr2=0x7f49a00cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 -- 172.21.15.104:0/964415279 shutdown_connections 2023-06-10T18:59:20.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.745+0000 7f49a6702700 1 -- 172.21.15.104:0/964415279 wait complete. 2023-06-10T18:59:20.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.747+0000 7f49a6702700 1 Processor -- start 2023-06-10T18:59:20.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.747+0000 7f49a6702700 1 -- start start 2023-06-10T18:59:20.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.747+0000 7f49a6702700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988005400 0x7f49a00ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.747+0000 7f49a6702700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49a01f5580 0x7f49a00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01f2320 con 0x7f49a01f2740 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01f19b0 con 0x7f4988005400 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01f5d30 con 0x7f49a01f5580 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41458/0 (socket says 172.21.15.104:41458) 2023-06-10T18:59:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988005400 0x7f49a00ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 -- 172.21.15.104:0/3962927558 learned_addr learned my addr 172.21.15.104:0/3962927558 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a4c9f700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49a01f5580 0x7f49a00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 -- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49a01f5580 msgr2=0x7f49a00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49a01f5580 0x7f49a00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 -- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988005400 msgr2=0x7f49a00ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988005400 0x7f49a00ac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 -- 172.21.15.104:0/3962927558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a01f1c60 con 0x7f49a01f2740 2023-06-10T18:59:20.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499f7fe700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f499000f580 tx=0x7f499000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990014870 con 0x7f49a01f2740 2023-06-10T18:59:20.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499000c050 con 0x7f49a01f2740 2023-06-10T18:59:20.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a01f1f40 con 0x7f49a01f2740 2023-06-10T18:59:20.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990014bf0 con 0x7f49a01f2740 2023-06-10T18:59:20.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.748+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a00ac9e0 con 0x7f49a01f2740 2023-06-10T18:59:20.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.749+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0066f70 con 0x7f49a01f2740 2023-06-10T18:59:20.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.750+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4990014d50 con 0x7f49a01f2740 2023-06-10T18:59:20.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.750+0000 7f499cff9700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 0x7f498c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:20.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.750+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4990089800 con 0x7f49a01f2740 2023-06-10T18:59:20.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.750+0000 7f499ffff700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 0x7f498c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:20.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.751+0000 7f499ffff700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 0x7f498c06d190 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f49a01e5aa0 tx=0x7f4994009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:20.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.754+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f49a0066f70 con 0x7f49a01f2740 2023-06-10T18:59:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.944+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49a01f2460 con 0x7f49a01f2740 2023-06-10T18:59:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.944+0000 7f499cff9700 1 -- 172.21.15.104:0/3962927558 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v183) v1 ==== 84+0+126 (secure 0 0 0) 0x7f49a01f2460 con 0x7f49a01f2740 2023-06-10T18:59:20.950 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:20.950 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2023-06-10T18:59:20.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:20.950 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 1, 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": ".mgr" 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: "poolnum": 2, 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: "poolname": "rbd" 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:20.951 INFO:teuthology.orchestra.run.smithi104.stdout:] 2023-06-10T18:59:20.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 msgr2=0x7f498c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 0x7f498c06d190 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f49a01e5aa0 tx=0x7f4994009040 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 msgr2=0x7f49a01f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f499000f580 tx=0x7f499000f890 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 shutdown_connections 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f498c06ace0 0x7f498c06d190 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4988005400 0x7f49a00ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49a01f5580 0x7f49a00b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 --2- 172.21.15.104:0/3962927558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f49a01f2740 0x7f49a01f5040 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:20.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 >> 172.21.15.104:0/3962927558 conn(0x7f49a006b8a0 msgr2=0x7f49a00ad000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 shutdown_connections 2023-06-10T18:59:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:20.946+0000 7f49a6702700 1 -- 172.21.15.104:0/3962927558 wait complete. 2023-06-10T18:59:20.964 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-10T18:59:20.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-10T18:59:21.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.083+0000 7f494c598700 1 Processor -- start 2023-06-10T18:59:21.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.083+0000 7f494c598700 1 -- start start 2023-06-10T18:59:21.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494c598700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f494415c6f0 0x7f49440b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494c598700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494c598700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49440ae7e0 con 0x7f494406ea70 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494c598700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494406df90 con 0x7f494415c6f0 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494c598700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494406e270 con 0x7f49440aec00 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f494415c6f0 0x7f49440b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f494415c6f0 0x7f49440b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41464/0 (socket says 172.21.15.104:41464) 2023-06-10T18:59:21.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 -- 172.21.15.104:0/3119222116 learned_addr learned my addr 172.21.15.104:0/3119222116 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 -- 172.21.15.104:0/3119222116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 msgr2=0x7f49440b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 --2- 172.21.15.104:0/3119222116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 -- 172.21.15.104:0/3119222116 >> v1:172.21.15.114:6789/0 conn(0x7f494406ea70 legacy=0x7f49440ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.084+0000 7f494a334700 1 -- 172.21.15.104:0/3119222116 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494406e550 con 0x7f494415c6f0 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494a334700 1 --2- 172.21.15.104:0/3119222116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494415c6f0 0x7f49440b1070 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f4934006fd0 tx=0x7f493400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e8ee6365543ba7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f4949332700 1 -- 172.21.15.104:0/3119222116 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934005e30 con 0x7f494415c6f0 2023-06-10T18:59:21.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f4949332700 1 -- 172.21.15.104:0/3119222116 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493400d8e0 con 0x7f494415c6f0 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f4949332700 1 -- 172.21.15.104:0/3119222116 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400dc70 con 0x7f494415c6f0 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 -- 172.21.15.104:0/3119222116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494415c6f0 msgr2=0x7f49440b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 --2- 172.21.15.104:0/3119222116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494415c6f0 0x7f49440b1070 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f4934006fd0 tx=0x7f493400be70 comp rx=0 tx=0).stop 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 -- 172.21.15.104:0/3119222116 shutdown_connections 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 --2- 172.21.15.104:0/3119222116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494415c6f0 0x7f49440b1070 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 --2- 172.21.15.104:0/3119222116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 -- 172.21.15.104:0/3119222116 >> 172.21.15.104:0/3119222116 conn(0x7f494406b8a0 msgr2=0x7f49440cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:21.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 -- 172.21.15.104:0/3119222116 shutdown_connections 2023-06-10T18:59:21.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.085+0000 7f494c598700 1 -- 172.21.15.104:0/3119222116 wait complete. 2023-06-10T18:59:21.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.087+0000 7f494c598700 1 Processor -- start 2023-06-10T18:59:21.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.087+0000 7f494c598700 1 -- start start 2023-06-10T18:59:21.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f494415c6f0 0x7f49440acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4944147100 con 0x7f494406ea70 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4944147590 con 0x7f494415c6f0 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49441441a0 con 0x7f49440aec00 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41470/0 (socket says 172.21.15.104:41470) 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 -- 172.21.15.104:0/2159606221 learned_addr learned my addr 172.21.15.104:0/2159606221 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494ab35700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f494415c6f0 0x7f49440acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:21.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f4949b33700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 -- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 msgr2=0x7f49440b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 -- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f494415c6f0 msgr2=0x7f49440acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f494415c6f0 0x7f49440acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 -- 172.21.15.104:0/2159606221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493400b7a0 con 0x7f494406ea70 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f4949b33700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:21.098 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494a334700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f4944070290 tx=0x7f4934004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934020070 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493401a450 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49440c52c0 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493401a7e0 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.088+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49440c5880 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.089+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49440cdb20 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.090+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f493401a940 con 0x7f494406ea70 2023-06-10T18:59:21.099 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.090+0000 7f493affd700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 0x7f492806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:21.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.090+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f493408d130 con 0x7f494406ea70 2023-06-10T18:59:21.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.091+0000 7f4949b33700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 0x7f492806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:21.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.091+0000 7f4949b33700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 0x7f492806d190 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4944159290 tx=0x7f4940009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:21.100 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.095+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f49440cdb20 con 0x7f494406ea70 2023-06-10T18:59:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.282+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f49440cdb20 con 0x7f494406ea70 2023-06-10T18:59:21.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.844+0000 7f493affd700 1 -- 172.21.15.104:0/2159606221 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v184) v1 ==== 138+0+0 (secure 0 0 0) 0x7f49440cdb20 con 0x7f494406ea70 2023-06-10T18:59:21.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.847+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 msgr2=0x7f492806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:21.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.847+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 0x7f492806d190 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4944159290 tx=0x7f4940009040 comp rx=0 tx=0).stop 2023-06-10T18:59:21.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.847+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 msgr2=0x7f4944143c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:21.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.847+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f4944070290 tx=0x7f4934004750 comp rx=0 tx=0).stop 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 shutdown_connections 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f492806ace0 0x7f492806d190 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f494415c6f0 0x7f49440acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f494406ea70 0x7f4944143c60 secure :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f4944070290 tx=0x7f4934004750 comp rx=0 tx=0).stop 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 --2- 172.21.15.104:0/2159606221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f49440aec00 0x7f49440b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:21.854 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 >> 172.21.15.104:0/2159606221 conn(0x7f494406b8a0 msgr2=0x7f49441585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:21.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 shutdown_connections 2023-06-10T18:59:21.855 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.848+0000 7f494c598700 1 -- 172.21.15.104:0/2159606221 wait complete. 2023-06-10T18:59:21.855 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' created 2023-06-10T18:59:21.869 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-10T18:59:21.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 Processor -- start 2023-06-10T18:59:21.998 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 -- start start 2023-06-10T18:59:22.001 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc3c15c6f0 0x7fcc3c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c06ea70 0x7fcc3c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c0ae7e0 con 0x7fcc3c0aec00 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c06df90 con 0x7fcc3c15c6f0 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.993+0000 7fcc42b68700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c06e270 con 0x7fcc3c06ea70 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc41105700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcc3c0aec00 0x7fcc3c0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39310/0 (socket says 172.21.15.104:39310) 2023-06-10T18:59:22.002 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc41105700 1 -- 172.21.15.104:0/4037987943 learned_addr learned my addr 172.21.15.104:0/4037987943 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc40904700 1 --2- 172.21.15.104:0/4037987943 >> v2:172.21.15.114:3300/0 conn(0x7fcc3c15c6f0 0x7fcc3c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3bfff700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c06ea70 0x7fcc3c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740772890 0 0) 0x7fcc3c0ae7e0 con 0x7fcc3c0aec00 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0035b0 con 0x7fcc3c0aec00 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2397486951 0 0) 0x7fcc2c0035b0 con 0x7fcc3c0aec00 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c06ea70 msgr2=0x7fcc3c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:22.003 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c06ea70 0x7fcc3c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 >> v2:172.21.15.114:3300/0 conn(0x7fcc3c15c6f0 msgr2=0x7fcc3c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 --2- 172.21.15.104:0/4037987943 >> v2:172.21.15.114:3300/0 conn(0x7fcc3c15c6f0 0x7fcc3c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c06e550 con 0x7fcc3c0aec00 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fcc340019f0 con 0x7fcc3c0aec00 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 >> v1:172.21.15.104:6789/0 conn(0x7fcc3c0aec00 legacy=0x7fcc3c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.004 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc2c003a00 0x7fcc2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc2c004350 0x7fcc2c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc2c004de0 0x7fcc2c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c06df90 con 0x7fcc2c004de0 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c06e270 con 0x7fcc2c004350 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c010350 con 0x7fcc2c003a00 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3bfff700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c06ea70 0x7fcc3c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc41105700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc2c004de0 0x7fcc2c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.005 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc40904700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc2c003a00 0x7fcc2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.994+0000 7fcc3bfff700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc2c004350 0x7fcc2c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 -- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc2c003a00 msgr2=0x7fcc2c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc2c003a00 0x7fcc2c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 -- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc2c004350 msgr2=0x7fcc2c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc2c004350 0x7fcc2c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 -- 172.21.15.104:0/4037987943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0035b0 con 0x7fcc2c004de0 2023-06-10T18:59:22.006 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc2c004de0 0x7fcc2c005230 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400a830 tx=0x7fcc2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf5c699afafb804 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34012600 con 0x7fcc2c004de0 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc34012780 con 0x7fcc2c004de0 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc3b7fe700 1 -- 172.21.15.104:0/4037987943 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34012b30 con 0x7fcc2c004de0 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc42b68700 1 -- 172.21.15.104:0/4037987943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc2c004de0 msgr2=0x7fcc2c005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc42b68700 1 --2- 172.21.15.104:0/4037987943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc2c004de0 0x7fcc2c005230 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400a830 tx=0x7fcc2c005880 comp rx=0 tx=0).stop 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc41105700 1 -- 172.21.15.104:0/4037987943 reap_dead start 2023-06-10T18:59:22.007 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc42b68700 1 -- 172.21.15.104:0/4037987943 shutdown_connections 2023-06-10T18:59:22.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc42b68700 1 -- 172.21.15.104:0/4037987943 >> 172.21.15.104:0/4037987943 conn(0x7fcc3c06b8a0 msgr2=0x7fcc3c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:22.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.995+0000 7fcc42b68700 1 -- 172.21.15.104:0/4037987943 shutdown_connections 2023-06-10T18:59:22.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.996+0000 7fcc42b68700 1 -- 172.21.15.104:0/4037987943 wait complete. 2023-06-10T18:59:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.997+0000 7fcc42b68700 1 Processor -- start 2023-06-10T18:59:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.997+0000 7fcc42b68700 1 -- start start 2023-06-10T18:59:22.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c0c89e0 0x7fcc3c0c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc3c14e800 0x7fcc3c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0c6e30 con 0x7fcc3c0b2960 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b2e00 con 0x7fcc3c14e800 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b30e0 con 0x7fcc3c0c89e0 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41494/0 (socket says 172.21.15.104:41494) 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 -- 172.21.15.104:0/2960340409 learned_addr learned my addr 172.21.15.104:0/2960340409 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc40904700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c0c89e0 0x7fcc3c0c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc3bfff700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc3c14e800 0x7fcc3c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 -- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c0c89e0 msgr2=0x7fcc3c0c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c0c89e0 0x7fcc3c0c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 -- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc3c14e800 msgr2=0x7fcc3c0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc3c14e800 0x7fcc3c0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 -- 172.21.15.104:0/2960340409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c0b33c0 con 0x7fcc3c0b2960 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc41105700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc34001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34012610 con 0x7fcc3c0b2960 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc340017b0 con 0x7fcc3c0b2960 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c0c49e0 con 0x7fcc3c0b2960 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34027410 con 0x7fcc3c0b2960 2023-06-10T18:59:22.013 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.998+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c15a3d0 con 0x7fcc3c0b2960 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:21.999+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c0c4f00 con 0x7fcc3c0b2960 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.000+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcc34019070 con 0x7fcc3c0b2960 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.000+0000 7fcc397fa700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 0x7fcc2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.001+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fcc3408a6a0 con 0x7fcc3c0b2960 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.001+0000 7fcc40904700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 0x7fcc2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:22.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.001+0000 7fcc40904700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 0x7fcc2006d190 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c09d510 tx=0x7fcc300091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:22.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.005+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcc3c0c4f00 con 0x7fcc3c0b2960 2023-06-10T18:59:22.200 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.194+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcc3c0c4f00 con 0x7fcc3c0b2960 2023-06-10T18:59:22.853 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.847+0000 7fcc397fa700 1 -- 172.21.15.104:0/2960340409 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v185) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcc3c0c4f00 con 0x7fcc3c0b2960 2023-06-10T18:59:22.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.851+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 msgr2=0x7fcc2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.857 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.851+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 0x7fcc2006d190 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c09d510 tx=0x7fcc300091c0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.851+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 msgr2=0x7fcc3c159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:22.858 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.851+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc34001d80 comp rx=0 tx=0).stop 2023-06-10T18:59:22.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.858+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 shutdown_connections 2023-06-10T18:59:22.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.858+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc3c0b2960 0x7fcc3c159e90 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.858+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcc3c0c89e0 0x7fcc3c0c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.864 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.858+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fcc2006ace0 0x7fcc2006d190 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.858+0000 7fcc42b68700 1 --2- 172.21.15.104:0/2960340409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc3c14e800 0x7fcc3c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:22.865 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.859+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 >> 172.21.15.104:0/2960340409 conn(0x7fcc3c06b8a0 msgr2=0x7fcc3c0ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:22.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.861+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 shutdown_connections 2023-06-10T18:59:22.866 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:22.861+0000 7fcc42b68700 1 -- 172.21.15.104:0/2960340409 wait complete. 2023-06-10T18:59:22.873 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' created 2023-06-10T18:59:22.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-10T18:59:23.008 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.003+0000 7fbdf2ca8700 1 Processor -- start 2023-06-10T18:59:23.009 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.003+0000 7fbdf2ca8700 1 -- start start 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf2ca8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf2ca8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf2ca8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec06e250 con 0x7fbdec15c580 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf2ca8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec15c950 con 0x7fbdec06cde0 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf2ca8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0b1e80 con 0x7fbdec06e6e0 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdebfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.010 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf1245700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf1245700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:57236/0 (socket says 172.21.15.104:57236) 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdebfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:60884/0 (socket says 172.21.15.104:60884) 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf1245700 1 -- 172.21.15.104:0/3938543503 learned_addr learned my addr 172.21.15.104:0/3938543503 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdf0a44700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbdec15c580 0x7fbdec0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:45172/0 (socket says 172.21.15.104:45172) 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdebfff700 1 -- 172.21.15.104:0/3938543503 >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 msgr2=0x7fbdec0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdebfff700 1 --2- 172.21.15.104:0/3938543503 >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.011 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdebfff700 1 -- 172.21.15.104:0/3938543503 >> v1:172.21.15.114:6789/0 conn(0x7fbdec15c580 legacy=0x7fbdec0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdebfff700 1 -- 172.21.15.104:0/3938543503 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec06d920 con 0x7fbdec06e6e0 2023-06-10T18:59:23.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.004+0000 7fbdeb7fe700 1 -- 172.21.15.104:0/3938543503 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195565042 0 0) 0x7fbdec06e250 con 0x7fbdec15c580 2023-06-10T18:59:23.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf1245700 1 --2- 172.21.15.104:0/3938543503 >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:23.012 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdebfff700 1 --2- 172.21.15.104:0/3938543503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009840 tx=0x7fbdd8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c1aae44271c261 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdeb7fe700 1 -- 172.21.15.104:0/3938543503 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8011830 con 0x7fbdec06e6e0 2023-06-10T18:59:23.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdeb7fe700 1 -- 172.21.15.104:0/3938543503 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8011990 con 0x7fbdec06e6e0 2023-06-10T18:59:23.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdeb7fe700 1 -- 172.21.15.104:0/3938543503 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8011d20 con 0x7fbdec06e6e0 2023-06-10T18:59:23.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/3938543503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 msgr2=0x7fbdec0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.014 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/3938543503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009840 tx=0x7fbdd8009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/3938543503 shutdown_connections 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/3938543503 >> v2:172.21.15.114:3300/0 conn(0x7fbdec06cde0 0x7fbdec0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/3938543503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec0b1780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/3938543503 >> 172.21.15.104:0/3938543503 conn(0x7fbdec06b8a0 msgr2=0x7fbdec0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/3938543503 shutdown_connections 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.005+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/3938543503 wait complete. 2023-06-10T18:59:23.015 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 Processor -- start 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 -- start start 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 0x7fbdec1f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1502d0 con 0x7fbdec15c580 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec1582e0 con 0x7fbdec06cde0 2023-06-10T18:59:23.016 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.007+0000 7fbdf2ca8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec15d520 con 0x7fbdec06e6e0 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf1245700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 0x7fbdec1f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdebfff700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf1245700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41508/0 (socket says 172.21.15.104:41508) 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf1245700 1 -- 172.21.15.104:0/2908797369 learned_addr learned my addr 172.21.15.104:0/2908797369 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 -- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 msgr2=0x7fbdec157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.017 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 -- 172.21.15.104:0/2908797369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 msgr2=0x7fbdec0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 -- 172.21.15.104:0/2908797369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde4009b30 con 0x7fbdec06cde0 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf1245700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdebfff700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf0a44700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 0x7fbdec1f14d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fbde000c4c0 tx=0x7fbde000c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0004cd0 con 0x7fbdec06cde0 2023-06-10T18:59:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0004e50 con 0x7fbdec06cde0 2023-06-10T18:59:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd80094a0 con 0x7fbdec06cde0 2023-06-10T18:59:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007310 con 0x7fbdec06cde0 2023-06-10T18:59:23.019 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.008+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec1f1a10 con 0x7fbdec06cde0 2023-06-10T18:59:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.009+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec074050 con 0x7fbdec06cde0 2023-06-10T18:59:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.010+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbde0007470 con 0x7fbdec06cde0 2023-06-10T18:59:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.010+0000 7fbde97fa700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 0x7fbdd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.010+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fbde0089490 con 0x7fbdec06cde0 2023-06-10T18:59:23.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.011+0000 7fbdebfff700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 0x7fbdd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.011+0000 7fbdebfff700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 0x7fbdd006d190 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80090b0 tx=0x7fbdd8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.015+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbde0054540 con 0x7fbdec06cde0 2023-06-10T18:59:23.250 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.245+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbdec0cab80 con 0x7fbdec06cde0 2023-06-10T18:59:23.262 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.257+0000 7fbde97fa700 1 -- 172.21.15.104:0/2908797369 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v114) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbde0057b60 con 0x7fbdec06cde0 2023-06-10T18:59:23.265 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 msgr2=0x7fbdd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 0x7fbdd006d190 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80090b0 tx=0x7fbdd8008030 comp rx=0 tx=0).stop 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 msgr2=0x7fbdec1f14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 0x7fbdec1f14d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fbde000c4c0 tx=0x7fbde000c7d0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 shutdown_connections 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbdd006ace0 0x7fbdd006d190 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec06cde0 0x7fbdec1f14d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbdec15c580 0x7fbdec0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 --2- 172.21.15.104:0/2908797369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06e6e0 0x7fbdec157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.260+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 >> 172.21.15.104:0/2908797369 conn(0x7fbdec06b8a0 msgr2=0x7fbdec15a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.271+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 shutdown_connections 2023-06-10T18:59:23.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.274+0000 7fbdf2ca8700 1 -- 172.21.15.104:0/2908797369 wait complete. 2023-06-10T18:59:23.285 INFO:teuthology.orchestra.run.smithi104.stderr:new fs with metadata pool 26 and data pool 27 2023-06-10T18:59:23.297 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:23.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.412+0000 7fc30b7f8700 1 Processor -- start 2023-06-10T18:59:23.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.412+0000 7fc30b7f8700 1 -- start start 2023-06-10T18:59:23.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc30b7f8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc30415c040 0x7fc3040ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc30b7f8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc304071130 0x7fc3040cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc30b7f8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040ccea0 con 0x7fc3040ae280 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc30b7f8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3040ae6f0 con 0x7fc30415c040 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc30b7f8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3040ce2c0 con 0x7fc304071130 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc30415c040 0x7fc3040ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc30415c040 0x7fc3040ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41526/0 (socket says 172.21.15.104:41526) 2023-06-10T18:59:23.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 -- 172.21.15.104:0/193320784 learned_addr learned my addr 172.21.15.104:0/193320784 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 -- 172.21.15.104:0/193320784 >> v2:172.21.15.114:3300/0 conn(0x7fc304071130 msgr2=0x7fc3040cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 --2- 172.21.15.104:0/193320784 >> v2:172.21.15.114:3300/0 conn(0x7fc304071130 0x7fc3040cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 -- 172.21.15.104:0/193320784 >> v1:172.21.15.114:6789/0 conn(0x7fc3040ae280 legacy=0x7fc304070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 -- 172.21.15.104:0/193320784 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040cf750 con 0x7fc30415c040 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc309594700 1 --2- 172.21.15.104:0/193320784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc30415c040 0x7fc3040ce780 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fc304070490 tx=0x7fc300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3c7ebdb259442f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.413+0000 7fc2fbfff700 1 -- 172.21.15.104:0/193320784 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300004230 con 0x7fc30415c040 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc2fbfff700 1 -- 172.21.15.104:0/193320784 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300004390 con 0x7fc30415c040 2023-06-10T18:59:23.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc2fbfff700 1 -- 172.21.15.104:0/193320784 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000036a0 con 0x7fc30415c040 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 -- 172.21.15.104:0/193320784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc30415c040 msgr2=0x7fc3040ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/193320784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc30415c040 0x7fc3040ce780 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fc304070490 tx=0x7fc300009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 -- 172.21.15.104:0/193320784 shutdown_connections 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/193320784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc30415c040 0x7fc3040ce780 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/193320784 >> v2:172.21.15.114:3300/0 conn(0x7fc304071130 0x7fc3040cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 -- 172.21.15.104:0/193320784 >> 172.21.15.104:0/193320784 conn(0x7fc3040b1610 msgr2=0x7fc3040ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 -- 172.21.15.104:0/193320784 shutdown_connections 2023-06-10T18:59:23.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.414+0000 7fc30b7f8700 1 -- 172.21.15.104:0/193320784 wait complete. 2023-06-10T18:59:23.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.415+0000 7fc30b7f8700 1 Processor -- start 2023-06-10T18:59:23.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.415+0000 7fc30b7f8700 1 -- start start 2023-06-10T18:59:23.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3040ae280 0x7fc3040c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc30415c040 0x7fc30415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040c5980 con 0x7fc304071130 2023-06-10T18:59:23.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040c53c0 con 0x7fc3040ae280 2023-06-10T18:59:23.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc30b7f8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040c56a0 con 0x7fc30415c040 2023-06-10T18:59:23.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41532/0 (socket says 172.21.15.104:41532) 2023-06-10T18:59:23.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 -- 172.21.15.104:0/3928673726 learned_addr learned my addr 172.21.15.104:0/3928673726 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc308d93700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3040ae280 0x7fc3040c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309d95700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc30415c040 0x7fc30415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 -- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc30415c040 msgr2=0x7fc30415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc30415c040 0x7fc30415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 -- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3040ae280 msgr2=0x7fc3040c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3040ae280 0x7fc3040c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.416+0000 7fc309594700 1 -- 172.21.15.104:0/3928673726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009550 con 0x7fc304071130 2023-06-10T18:59:23.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc309594700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fc30406c890 tx=0x7fc300004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300003dc0 con 0x7fc304071130 2023-06-10T18:59:23.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30001f070 con 0x7fc304071130 2023-06-10T18:59:23.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30415c520 con 0x7fc304071130 2023-06-10T18:59:23.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300011870 con 0x7fc304071130 2023-06-10T18:59:23.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.417+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc304159810 con 0x7fc304071130 2023-06-10T18:59:23.431 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.418+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc304074050 con 0x7fc304071130 2023-06-10T18:59:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.418+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc300021030 con 0x7fc304071130 2023-06-10T18:59:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.419+0000 7fc2f9ffb700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 0x7fc2e806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.419+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc30008aac0 con 0x7fc304071130 2023-06-10T18:59:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.419+0000 7fc308d93700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 0x7fc2e806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.432 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.419+0000 7fc308d93700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 0x7fc2e806d1c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4001730 tx=0x7fc2f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.433 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.423+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc300055ac0 con 0x7fc304071130 2023-06-10T18:59:23.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.613+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc304071750 con 0x7fc304071130 2023-06-10T18:59:23.620 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.615+0000 7fc2f9ffb700 1 -- 172.21.15.104:0/3928673726 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+18631 (secure 0 0 0) 0x7fc30008a020 con 0x7fc304071130 2023-06-10T18:59:23.620 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:23.620 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":186,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:23.246975+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 msgr2=0x7fc2e806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 0x7fc2e806d1c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4001730 tx=0x7fc2f4009040 comp rx=0 tx=0).stop 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 msgr2=0x7fc3041592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fc30406c890 tx=0x7fc300004750 comp rx=0 tx=0).stop 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 shutdown_connections 2023-06-10T18:59:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc30415c040 0x7fc30415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304071130 0x7fc3041592d0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2e806ad10 0x7fc2e806d1c0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 --2- 172.21.15.104:0/3928673726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3040ae280 0x7fc3040c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 >> 172.21.15.104:0/3928673726 conn(0x7fc3040b1610 msgr2=0x7fc30406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 shutdown_connections 2023-06-10T18:59:23.624 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.617+0000 7fc30b7f8700 1 -- 172.21.15.104:0/3928673726 wait complete. 2023-06-10T18:59:23.634 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.750+0000 7ff9fe882700 1 Processor -- start 2023-06-10T18:59:23.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.750+0000 7ff9fe882700 1 -- start start 2023-06-10T18:59:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff9f80aeaa0 0x7ff9f80cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f80cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff9f80ca2f0 0x7ff9f80ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80c9bc0 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80c9e60 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fe882700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80aee70 con 0x7ff9f80b1080 2023-06-10T18:59:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9f7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff9f80aeaa0 0x7ff9f80cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9f7fff700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff9f80aeaa0 0x7ff9f80cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41540/0 (socket says 172.21.15.104:41540) 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9f7fff700 1 -- 172.21.15.104:0/1936534299 learned_addr learned my addr 172.21.15.104:0/1936534299 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9fce1f700 1 --2- 172.21.15.104:0/1936534299 >> v2:172.21.15.114:3300/0 conn(0x7ff9f80ca2f0 0x7ff9f80ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.751+0000 7ff9f77fe700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f80cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 -- 172.21.15.104:0/1936534299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 msgr2=0x7ff9f80cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f80cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 -- 172.21.15.104:0/1936534299 >> v2:172.21.15.114:3300/0 conn(0x7ff9f80ca2f0 msgr2=0x7ff9f80ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/1936534299 >> v2:172.21.15.114:3300/0 conn(0x7ff9f80ca2f0 0x7ff9f80ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 -- 172.21.15.104:0/1936534299 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f806e540 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80cd140 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806fa30 tx=0x7ff9e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebe8f31065e3799 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f6ffd700 1 -- 172.21.15.104:0/1936534299 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005e30 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f6ffd700 1 -- 172.21.15.104:0/1936534299 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0004030 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9f6ffd700 1 -- 172.21.15.104:0/1936534299 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00043e0 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 -- 172.21.15.104:0/1936534299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80aeaa0 msgr2=0x7ff9f80cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80cd140 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806fa30 tx=0x7ff9e0009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 -- 172.21.15.104:0/1936534299 shutdown_connections 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80cd140 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 --2- 172.21.15.104:0/1936534299 >> v2:172.21.15.114:3300/0 conn(0x7ff9f80ca2f0 0x7ff9f80ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 --2- 172.21.15.104:0/1936534299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f80cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 -- 172.21.15.104:0/1936534299 >> 172.21.15.104:0/1936534299 conn(0x7ff9f806b8a0 msgr2=0x7ff9f80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 -- 172.21.15.104:0/1936534299 shutdown_connections 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.752+0000 7ff9fe882700 1 -- 172.21.15.104:0/1936534299 wait complete. 2023-06-10T18:59:23.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.754+0000 7ff9fe882700 1 Processor -- start 2023-06-10T18:59:23.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.754+0000 7ff9fe882700 1 -- start start 2023-06-10T18:59:23.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f8147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f81573a0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8147cc0 con 0x7ff9f80aeaa0 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f814ac20 con 0x7ff9f80b1080 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41550/0 (socket says 172.21.15.104:41550) 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 -- 172.21.15.104:0/771275848 learned_addr learned my addr 172.21.15.104:0/771275848 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:23.767 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9f77fe700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f8147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 -- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 msgr2=0x7ff9f8147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f8147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 -- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f80aeaa0 msgr2=0x7ff9f80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.768 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 -- 172.21.15.104:0/771275848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0009500 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fce1f700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0007e30 tx=0x7ff9f00105d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0013470 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f00135f0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f80b24d0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00139a0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.755+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f80b2a90 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.769 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.756+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f80b2cb0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.757+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff9f0013b00 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.757+0000 7ff9f4ff9700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 0x7ff9d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:23.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.757+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff9f008c4a0 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.757+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 0x7ff9d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:23.770 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.758+0000 7ff9f7fff700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 0x7ff9d806d190 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806fa30 tx=0x7ff9e0008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:23.771 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.762+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff9f0057370 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.846 INFO:tasks.ceph.mds.a.smithi104.stderr:2023-06-10T18:59:23.840+0000 7f2da3d73700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T18:59:23.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.952+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9f8074050 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.953+0000 7ff9f4ff9700 1 -- 172.21.15.104:0/771275848 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+18631 (secure 0 0 0) 0x7ff9f005a990 con 0x7ff9f80ca2f0 2023-06-10T18:59:23.959 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:23.959 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":186,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:23.246975+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:23.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 msgr2=0x7ff9d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 0x7ff9d806d190 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7ff9f806fa30 tx=0x7ff9e0008070 comp rx=0 tx=0).stop 2023-06-10T18:59:23.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 msgr2=0x7ff9f80c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:23.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0007e30 tx=0x7ff9f00105d0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 shutdown_connections 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff9d806ace0 0x7ff9d806d190 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f80aeaa0 0x7ff9f80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9f80ca2f0 0x7ff9f80c8bf0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 --2- 172.21.15.104:0/771275848 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9f80b1080 0x7ff9f8147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 >> 172.21.15.104:0/771275848 conn(0x7ff9f806b8a0 msgr2=0x7ff9f8158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 shutdown_connections 2023-06-10T18:59:23.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:23.955+0000 7ff9fe882700 1 -- 172.21.15.104:0/771275848 wait complete. 2023-06-10T18:59:23.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-10T18:59:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.091+0000 7ff0d90f9700 1 Processor -- start 2023-06-10T18:59:24.097 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.091+0000 7ff0d90f9700 1 -- start start 2023-06-10T18:59:24.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d90f9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0d4156a10 0x7ff0d415d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d90f9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 0x7ff0d4160840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.105 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d90f9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40cfce0 con 0x7ff0d4070610 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d90f9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d415d7f0 con 0x7ff0d4156a10 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d90f9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d4160d80 con 0x7ff0d4160440 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0d4156a10 0x7ff0d415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d2d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff0d4156a10 0x7ff0d415d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41554/0 (socket says 172.21.15.104:41554) 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.092+0000 7ff0d2d9d700 1 -- 172.21.15.104:0/1331311007 learned_addr learned my addr 172.21.15.104:0/1331311007 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d359e700 1 --2- 172.21.15.104:0/1331311007 >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 0x7ff0d4160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d2d9d700 1 -- 172.21.15.104:0/1331311007 >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 msgr2=0x7ff0d4160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:24.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d2d9d700 1 --2- 172.21.15.104:0/1331311007 >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 0x7ff0d4160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d2d9d700 1 -- 172.21.15.104:0/1331311007 >> v1:172.21.15.114:6789/0 conn(0x7ff0d4070610 legacy=0x7ff0d40cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d2d9d700 1 -- 172.21.15.104:0/1331311007 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40cff80 con 0x7ff0d4156a10 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d359e700 1 --2- 172.21.15.104:0/1331311007 >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 0x7ff0d4160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d2d9d700 1 --2- 172.21.15.104:0/1331311007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d415d210 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40ce2a0 tx=0x7ff0c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54169c9e60afbe2b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d1d9b700 1 -- 172.21.15.104:0/1331311007 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800b860 con 0x7ff0d4156a10 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d1d9b700 1 -- 172.21.15.104:0/1331311007 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c800b9c0 con 0x7ff0d4156a10 2023-06-10T18:59:24.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d1d9b700 1 -- 172.21.15.104:0/1331311007 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800bd50 con 0x7ff0d4156a10 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1331311007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 msgr2=0x7ff0d415d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1331311007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d415d210 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40ce2a0 tx=0x7ff0c8009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1331311007 shutdown_connections 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1331311007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d415d210 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1331311007 >> v2:172.21.15.114:3300/0 conn(0x7ff0d4160440 0x7ff0d4160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.093+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1331311007 >> 172.21.15.104:0/1331311007 conn(0x7ff0d406e370 msgr2=0x7ff0d406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:24.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.094+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1331311007 shutdown_connections 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.094+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1331311007 wait complete. 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.095+0000 7ff0d90f9700 1 Processor -- start 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.095+0000 7ff0d90f9700 1 -- start start 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0d4070610 0x7ff0d4152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0d4160440 0x7ff0d40b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b1c90 con 0x7ff0d4156a10 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b1080 con 0x7ff0d4160440 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d90f9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d40b1360 con 0x7ff0d4070610 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41562/0 (socket says 172.21.15.104:41562) 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 -- 172.21.15.104:0/1161569497 learned_addr learned my addr 172.21.15.104:0/1161569497 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:24.110 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d2d9d700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0d4070610 0x7ff0d4152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d359e700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0d4160440 0x7ff0d40b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 -- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0d4070610 msgr2=0x7ff0d4152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0d4070610 0x7ff0d4152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 -- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0d4160440 msgr2=0x7ff0d40b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0d4160440 0x7ff0d40b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:24.111 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 -- 172.21.15.104:0/1161569497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8009550 con 0x7ff0d4156a10 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0d259c700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c4e0 tx=0x7ff0c400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.096+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40052d0 con 0x7ff0d4156a10 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.097+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c4005430 con 0x7ff0d4156a10 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.097+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d40b1640 con 0x7ff0d4156a10 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.097+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40038c0 con 0x7ff0d4156a10 2023-06-10T18:59:24.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.097+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d41532d0 con 0x7ff0d4156a10 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.098+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d4074050 con 0x7ff0d4156a10 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.098+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff0c4003a20 con 0x7ff0d4156a10 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.098+0000 7ff0bb7fe700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 0x7ff0c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.098+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff0c4088890 con 0x7ff0d4156a10 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.099+0000 7ff0d2d9d700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 0x7ff0c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:24.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.099+0000 7ff0d2d9d700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 0x7ff0c006d190 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000c00 tx=0x7ff0c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:24.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.103+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff0c4053890 con 0x7ff0d4156a10 2023-06-10T18:59:24.338 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:24.333+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff0d40703f0 con 0x7ff0d4156a10 2023-06-10T18:59:25.263 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.258+0000 7ff0bb7fe700 1 -- 172.21.15.104:0/1161569497 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v116) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff0c4056eb0 con 0x7ff0d4156a10 2023-06-10T18:59:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 msgr2=0x7ff0c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 0x7ff0c006d190 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000c00 tx=0x7ff0c8008030 comp rx=0 tx=0).stop 2023-06-10T18:59:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 msgr2=0x7ff0d41e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.266 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c4e0 tx=0x7ff0c400c7f0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 shutdown_connections 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff0c006ace0 0x7ff0c006d190 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0d4160440 0x7ff0d40b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0d4070610 0x7ff0d4152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 --2- 172.21.15.104:0/1161569497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff0d4156a10 0x7ff0d41e6d50 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 >> 172.21.15.104:0/1161569497 conn(0x7ff0d406e370 msgr2=0x7ff0d4154ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 shutdown_connections 2023-06-10T18:59:25.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.260+0000 7ff0d90f9700 1 -- 172.21.15.104:0/1161569497 wait complete. 2023-06-10T18:59:25.280 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-10T18:59:25.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-10T18:59:25.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.403+0000 7f95e86d4700 1 Processor -- start 2023-06-10T18:59:25.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.403+0000 7f95e86d4700 1 -- start start 2023-06-10T18:59:25.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e86d4700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95e006e6e0 0x7f95e00b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e86d4700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e006e250 con 0x7f95e006cde0 2023-06-10T18:59:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e86d4700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e015c950 con 0x7f95e015c580 2023-06-10T18:59:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e86d4700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e00b1e80 con 0x7f95e006e6e0 2023-06-10T18:59:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95e006e6e0 0x7f95e00b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:25.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e6c71700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f95e006cde0 0x7f95e00ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39400/0 (socket says 172.21.15.104:39400) 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f95e006e6e0 0x7f95e00b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41570/0 (socket says 172.21.15.104:41570) 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e6c71700 1 -- 172.21.15.104:0/3303359288 learned_addr learned my addr 172.21.15.104:0/3303359288 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 -- 172.21.15.104:0/3303359288 >> v1:172.21.15.114:6789/0 conn(0x7f95e015c580 legacy=0x7f95e00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 -- 172.21.15.104:0/3303359288 >> v1:172.21.15.104:6789/0 conn(0x7f95e006cde0 legacy=0x7f95e00ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e546e700 1 -- 172.21.15.104:0/3303359288 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028652047 0 0) 0x7f95e006e250 con 0x7f95e006cde0 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 -- 172.21.15.104:0/3303359288 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e006d920 con 0x7f95e006e6e0 2023-06-10T18:59:25.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e5c6f700 1 --2- 172.21.15.104:0/3303359288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e00b1780 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009840 tx=0x7f95d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07a159e3c5d8001 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e546e700 1 -- 172.21.15.104:0/3303359288 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004230 con 0x7f95e006e6e0 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e546e700 1 -- 172.21.15.104:0/3303359288 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d0004390 con 0x7f95e006e6e0 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.404+0000 7f95e546e700 1 -- 172.21.15.104:0/3303359288 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d00036a0 con 0x7f95e006e6e0 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 -- 172.21.15.104:0/3303359288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 msgr2=0x7f95e00b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 --2- 172.21.15.104:0/3303359288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e00b1780 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009840 tx=0x7f95d0009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 -- 172.21.15.104:0/3303359288 shutdown_connections 2023-06-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 --2- 172.21.15.104:0/3303359288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e00b1780 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 -- 172.21.15.104:0/3303359288 >> 172.21.15.104:0/3303359288 conn(0x7f95e006b8a0 msgr2=0x7f95e00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:25.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 -- 172.21.15.104:0/3303359288 shutdown_connections 2023-06-10T18:59:25.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.405+0000 7f95e86d4700 1 -- 172.21.15.104:0/3303359288 wait complete. 2023-06-10T18:59:25.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.406+0000 7f95e86d4700 1 Processor -- start 2023-06-10T18:59:25.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.406+0000 7f95e86d4700 1 -- start start 2023-06-10T18:59:25.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95e006cde0 0x7f95e00b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95e015c580 0x7f95e01556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e00b53c0 con 0x7f95e006e6e0 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e01e4020 con 0x7f95e006cde0 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e86d4700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e01f5d30 con 0x7f95e015c580 2023-06-10T18:59:25.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41576/0 (socket says 172.21.15.104:41576) 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 -- 172.21.15.104:0/1243698711 learned_addr learned my addr 172.21.15.104:0/1243698711 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e6470700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95e006cde0 0x7f95e00b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e6c71700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95e015c580 0x7f95e01556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 -- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95e015c580 msgr2=0x7f95e01556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95e015c580 0x7f95e01556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 -- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95e006cde0 msgr2=0x7f95e00b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95e006cde0 0x7f95e00b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.407+0000 7f95e5c6f700 1 -- 172.21.15.104:0/1243698711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00094a0 con 0x7f95e006e6e0 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95e5c6f700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f95d00090b0 tx=0x7f95d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0011830 con 0x7f95e006e6e0 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d001e070 con 0x7f95e006e6e0 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e0155be0 con 0x7f95e006e6e0 2023-06-10T18:59:25.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0011bf0 con 0x7f95e006e6e0 2023-06-10T18:59:25.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.408+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e01e69e0 con 0x7f95e006e6e0 2023-06-10T18:59:25.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.409+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e0074050 con 0x7f95e006e6e0 2023-06-10T18:59:25.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.409+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f95d0011d50 con 0x7f95e006e6e0 2023-06-10T18:59:25.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.409+0000 7f95d6ffd700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 0x7f95c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:25.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.410+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f95d008a8f0 con 0x7f95e006e6e0 2023-06-10T18:59:25.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.410+0000 7f95e6470700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 0x7f95c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:25.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.410+0000 7f95e6470700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 0x7f95c406d190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006fd0 tx=0x7f95dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:25.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.414+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f95d00558f0 con 0x7f95e006e6e0 2023-06-10T18:59:25.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:25.644+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f95e0066f10 con 0x7f95e006e6e0 2023-06-10T18:59:26.264 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.259+0000 7f95d6ffd700 1 -- 172.21.15.104:0/1243698711 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v117) v1 ==== 122+0+0 (secure 0 0 0) 0x7f95d0058f10 con 0x7f95e006e6e0 2023-06-10T18:59:26.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 msgr2=0x7f95c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 0x7f95c406d190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f95dc006fd0 tx=0x7f95dc008040 comp rx=0 tx=0).stop 2023-06-10T18:59:26.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 msgr2=0x7f95e01e3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.267 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f95d00090b0 tx=0x7f95d0004750 comp rx=0 tx=0).stop 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 shutdown_connections 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f95c406ace0 0x7f95c406d190 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95e006cde0 0x7f95e00b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95e015c580 0x7f95e01556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 --2- 172.21.15.104:0/1243698711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f95e006e6e0 0x7f95e01e3ae0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.261+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 >> 172.21.15.104:0/1243698711 conn(0x7f95e006b8a0 msgr2=0x7f95e00ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:26.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.262+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 shutdown_connections 2023-06-10T18:59:26.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.262+0000 7f95e86d4700 1 -- 172.21.15.104:0/1243698711 wait complete. 2023-06-10T18:59:26.280 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-10T18:59:26.281 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.398+0000 7f772df58700 1 Processor -- start 2023-06-10T18:59:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.398+0000 7f772df58700 1 -- start start 2023-06-10T18:59:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f772df58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f772806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f772df58700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f772806d150 0x7f77280cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f772df58700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280cccd0 con 0x7f772806b180 2023-06-10T18:59:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f772df58700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77280ccfb0 con 0x7f772806d150 2023-06-10T18:59:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f772df58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77280ce0b0 con 0x7f7728158c50 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f7726ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f772806b180 0x7f772806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39412/0 (socket says 172.21.15.104:39412) 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f7726ffd700 1 -- 172.21.15.104:0/1835912432 learned_addr learned my addr 172.21.15.104:0/1835912432 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f7727fff700 1 --2- 172.21.15.104:0/1835912432 >> v2:172.21.15.114:3300/0 conn(0x7f772806d150 0x7f77280cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77277fe700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f772806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848868072 0 0) 0x7f77280cccd0 con 0x7f772806b180 2023-06-10T18:59:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77100035b0 con 0x7f772806b180 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237769048 0 0) 0x7f77100035b0 con 0x7f772806b180 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 msgr2=0x7f772806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f772806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 >> v2:172.21.15.114:3300/0 conn(0x7f772806d150 msgr2=0x7f77280cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 --2- 172.21.15.104:0/1835912432 >> v2:172.21.15.114:3300/0 conn(0x7f772806d150 0x7f77280cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772806a010 con 0x7f772806b180 2023-06-10T18:59:26.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7718001670 con 0x7f772806b180 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 >> v1:172.21.15.104:6789/0 conn(0x7f772806b180 legacy=0x7f772806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710003a30 0x7f7710003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710004850 0x7f7710004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f7710005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280ccfb0 con 0x7f7710004850 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771000dc20 con 0x7f7710003a30 2023-06-10T18:59:26.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.399+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771000dd60 con 0x7f7728158c50 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710004850 0x7f7710004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7727fff700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f7710005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f77277fe700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710003a30 0x7f7710003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 -- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 msgr2=0x7f7710005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7728158c50 0x7f7710005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 -- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710003a30 msgr2=0x7f7710003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7710003a30 0x7f7710003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 -- 172.21.15.104:0/1835912432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100035b0 con 0x7f7710004850 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f7726ffd700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710004850 0x7f7710004cc0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f771800c2d0 tx=0x7f7710005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910506917dd2badb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718007c10 con 0x7f7710004850 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771800cdb0 con 0x7f7710004850 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.400+0000 7f77267fc700 1 -- 172.21.15.104:0/1835912432 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718005ba0 con 0x7f7710004850 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 -- 172.21.15.104:0/1835912432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710004850 msgr2=0x7f7710004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 --2- 172.21.15.104:0/1835912432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7710004850 0x7f7710004cc0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f771800c2d0 tx=0x7f7710005880 comp rx=0 tx=0).stop 2023-06-10T18:59:26.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f7727fff700 1 -- 172.21.15.104:0/1835912432 reap_dead start 2023-06-10T18:59:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 -- 172.21.15.104:0/1835912432 shutdown_connections 2023-06-10T18:59:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 -- 172.21.15.104:0/1835912432 >> 172.21.15.104:0/1835912432 conn(0x7f77280b1610 msgr2=0x7f77280ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 -- 172.21.15.104:0/1835912432 shutdown_connections 2023-06-10T18:59:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.401+0000 7f772df58700 1 -- 172.21.15.104:0/1835912432 wait complete. 2023-06-10T18:59:26.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.402+0000 7f772df58700 1 Processor -- start 2023-06-10T18:59:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.402+0000 7f772df58700 1 -- start start 2023-06-10T18:59:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7710004850 0x7f77281562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77280c64c0 0x7f77280c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280c3c80 con 0x7f77280b2cd0 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280c2c20 con 0x7f77280c64c0 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77280b3680 con 0x7f7710004850 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41596/0 (socket says 172.21.15.104:41596) 2023-06-10T18:59:26.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 -- 172.21.15.104:0/2559835698 learned_addr learned my addr 172.21.15.104:0/2559835698 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7727fff700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77280c64c0 0x7f77280c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f77277fe700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7710004850 0x7f77281562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 -- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7710004850 msgr2=0x7f77281562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7710004850 0x7f77281562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 -- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77280c64c0 msgr2=0x7f77280c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77280c64c0 0x7f77280c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 -- 172.21.15.104:0/2559835698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280c41d0 con 0x7f77280b2cd0 2023-06-10T18:59:26.417 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f7726ffd700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7718001850 tx=0x7f772806b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718001640 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77180070d0 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77280c76d0 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718007460 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.403+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77281567e0 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.404+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77280c4360 con 0x7f77280b2cd0 2023-06-10T18:59:26.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.405+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7718019030 con 0x7f77280b2cd0 2023-06-10T18:59:26.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.405+0000 7f770ffff700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 0x7f771406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.405+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f771808a5b0 con 0x7f77280b2cd0 2023-06-10T18:59:26.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.405+0000 7f77277fe700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 0x7f771406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.406+0000 7f77277fe700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 0x7f771406d190 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f771c009a60 tx=0x7f771c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.410+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f77180555b0 con 0x7f77280b2cd0 2023-06-10T18:59:26.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.638+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7728074050 con 0x7f77280b2cd0 2023-06-10T18:59:26.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.638+0000 7f770ffff700 1 -- 172.21.15.104:0/2559835698 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f7718007d70 con 0x7f77280b2cd0 2023-06-10T18:59:26.645 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:26.645 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T18:59:26.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 msgr2=0x7f771406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 0x7f771406d190 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f771c009a60 tx=0x7f771c008040 comp rx=0 tx=0).stop 2023-06-10T18:59:26.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 msgr2=0x7f77280b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7718001850 tx=0x7f772806b030 comp rx=0 tx=0).stop 2023-06-10T18:59:26.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 shutdown_connections 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f771406ace0 0x7f771406d190 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77280c64c0 0x7f77280c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7710004850 0x7f77281562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 --2- 172.21.15.104:0/2559835698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77280b2cd0 0x7f77280b3140 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 >> 172.21.15.104:0/2559835698 conn(0x7f77280b1610 msgr2=0x7f77280ca8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:26.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 shutdown_connections 2023-06-10T18:59:26.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.640+0000 7f772df58700 1 -- 172.21.15.104:0/2559835698 wait complete. 2023-06-10T18:59:26.649 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T18:59:26.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:26.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.785+0000 7efee0f21700 1 Processor -- start 2023-06-10T18:59:26.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.785+0000 7efee0f21700 1 -- start start 2023-06-10T18:59:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efee0f21700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efedc158eb0 0x7efedc159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efee0f21700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efedc069e70 con 0x7efedc06a8c0 2023-06-10T18:59:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efee0f21700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efedc1595f0 con 0x7efedc070a20 2023-06-10T18:59:26.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efee0f21700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efedc1598d0 con 0x7efedc158eb0 2023-06-10T18:59:26.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efedad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efedc158eb0 0x7efedc159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efed9d9b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efedc06a8c0 0x7efedc1587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39430/0 (socket says 172.21.15.104:39430) 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.786+0000 7efedad9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efedc158eb0 0x7efedc159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41604/0 (socket says 172.21.15.104:41604) 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efed9d9b700 1 -- 172.21.15.104:0/2982283592 learned_addr learned my addr 172.21.15.104:0/2982283592 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efedad9d700 1 -- 172.21.15.104:0/2982283592 >> v1:172.21.15.114:6789/0 conn(0x7efedc070a20 legacy=0x7efedc070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efedad9d700 1 -- 172.21.15.104:0/2982283592 >> v1:172.21.15.104:6789/0 conn(0x7efedc06a8c0 legacy=0x7efedc1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efedad9d700 1 -- 172.21.15.104:0/2982283592 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc06a0c0 con 0x7efedc158eb0 2023-06-10T18:59:26.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efed959a700 1 -- 172.21.15.104:0/2982283592 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770707840 0 0) 0x7efedc069e70 con 0x7efedc06a8c0 2023-06-10T18:59:26.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efedad9d700 1 --2- 172.21.15.104:0/2982283592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc159c40 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7efed4009d90 tx=0x7efed400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b353c68134ce6b12 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efed959a700 1 -- 172.21.15.104:0/2982283592 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed400e070 con 0x7efedc158eb0 2023-06-10T18:59:26.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efed959a700 1 -- 172.21.15.104:0/2982283592 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed400bb90 con 0x7efedc158eb0 2023-06-10T18:59:26.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.787+0000 7efed959a700 1 -- 172.21.15.104:0/2982283592 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4003a20 con 0x7efedc158eb0 2023-06-10T18:59:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 -- 172.21.15.104:0/2982283592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 msgr2=0x7efedc159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 --2- 172.21.15.104:0/2982283592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc159c40 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7efed4009d90 tx=0x7efed400b330 comp rx=0 tx=0).stop 2023-06-10T18:59:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 -- 172.21.15.104:0/2982283592 shutdown_connections 2023-06-10T18:59:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 --2- 172.21.15.104:0/2982283592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc159c40 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 -- 172.21.15.104:0/2982283592 >> 172.21.15.104:0/2982283592 conn(0x7efedc0b1610 msgr2=0x7efedc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 -- 172.21.15.104:0/2982283592 shutdown_connections 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.788+0000 7efee0f21700 1 -- 172.21.15.104:0/2982283592 wait complete. 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.789+0000 7efee0f21700 1 Processor -- start 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.789+0000 7efee0f21700 1 -- start start 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 0x7efedc1ef390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 0x7efedc0c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc1f0af0 con 0x7efedc158eb0 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc1f3b50 con 0x7efedc070a20 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efee0f21700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc1f3fe0 con 0x7efedc06a8c0 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efedad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efedad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41610/0 (socket says 172.21.15.104:41610) 2023-06-10T18:59:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.790+0000 7efedad9d700 1 -- 172.21.15.104:0/4077878976 learned_addr learned my addr 172.21.15.104:0/4077878976 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efed9d9b700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 0x7efedc0c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efeda59c700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 0x7efedc1ef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 -- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 msgr2=0x7efedc1ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 0x7efedc1ef390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 -- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 msgr2=0x7efedc0c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 0x7efedc0c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 -- 172.21.15.104:0/4077878976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4009a70 con 0x7efedc158eb0 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efed9d9b700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 0x7efedc0c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efeda59c700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 0x7efedc1ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efedad9d700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7efed400b730 tx=0x7efed4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4012630 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4012790 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc1f3220 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4012b40 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.791+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc1f37e0 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.792+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efedc071780 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.792+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efed4012ca0 con 0x7efedc158eb0 2023-06-10T18:59:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.793+0000 7efecaffd700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 0x7efebc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.793+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7efed408bbc0 con 0x7efedc158eb0 2023-06-10T18:59:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.793+0000 7efeda59c700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 0x7efebc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.793+0000 7efeda59c700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 0x7efebc06d190 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7efed0003f40 tx=0x7efed0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.797+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efed4056bc0 con 0x7efedc158eb0 2023-06-10T18:59:26.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.985+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efedc1f3970 con 0x7efedc158eb0 2023-06-10T18:59:26.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.986+0000 7efecaffd700 1 -- 172.21.15.104:0/4077878976 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+18618 (secure 0 0 0) 0x7efed402a090 con 0x7efedc158eb0 2023-06-10T18:59:26.992 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:26.992 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":187,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:24.248976+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 msgr2=0x7efebc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 0x7efebc06d190 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7efed0003f40 tx=0x7efed0009040 comp rx=0 tx=0).stop 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 msgr2=0x7efedc0c4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7efed400b730 tx=0x7efed4004ad0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 shutdown_connections 2023-06-10T18:59:26.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efedc158eb0 0x7efedc0c4ea0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efedc06a8c0 0x7efedc1ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efebc06ace0 0x7efebc06d190 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 --2- 172.21.15.104:0/4077878976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efedc070a20 0x7efedc0c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 >> 172.21.15.104:0/4077878976 conn(0x7efedc0b1610 msgr2=0x7efedc0cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 shutdown_connections 2023-06-10T18:59:26.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:26.988+0000 7efee0f21700 1 -- 172.21.15.104:0/4077878976 wait complete. 2023-06-10T18:59:27.007 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-10T18:59:27.007 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:59:27.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.134+0000 7f6af4062700 1 Processor -- start 2023-06-10T18:59:27.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 -- start start 2023-06-10T18:59:27.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6aec156740 0x7f6aec15d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6aec0cec20 0x7f6aec0cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec15cdf0 con 0x7f6aec069e70 2023-06-10T18:59:27.143 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0cd790 con 0x7f6aec0cec20 2023-06-10T18:59:27.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.135+0000 7f6af4062700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0cf0a0 con 0x7f6aec156740 2023-06-10T18:59:27.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6aec0cec20 0x7f6aec0cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f6aec0cec20 0x7f6aec0cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41614/0 (socket says 172.21.15.104:41614) 2023-06-10T18:59:27.144 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 -- 172.21.15.104:0/4010577696 learned_addr learned my addr 172.21.15.104:0/4010577696 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af25ff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6aec069e70 0x7f6aec06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39446/0 (socket says 172.21.15.104:39446) 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af1dfe700 1 --2- 172.21.15.104:0/4010577696 >> v2:172.21.15.114:3300/0 conn(0x7f6aec156740 0x7f6aec15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 -- 172.21.15.104:0/4010577696 >> v2:172.21.15.114:3300/0 conn(0x7f6aec156740 msgr2=0x7f6aec15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 --2- 172.21.15.104:0/4010577696 >> v2:172.21.15.114:3300/0 conn(0x7f6aec156740 0x7f6aec15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 -- 172.21.15.104:0/4010577696 >> v1:172.21.15.104:6789/0 conn(0x7f6aec069e70 legacy=0x7f6aec06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.145 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 -- 172.21.15.104:0/4010577696 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0cf380 con 0x7f6aec0cec20 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af0dfc700 1 -- 172.21.15.104:0/4010577696 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755474974 0 0) 0x7f6aec15cdf0 con 0x7f6aec069e70 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af15fd700 1 --2- 172.21.15.104:0/4010577696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec0cec20 0x7f6aec0cd250 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f6aec06ae70 tx=0x7f6ae8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6779a25f277cfef server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af0dfc700 1 -- 172.21.15.104:0/4010577696 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800b870 con 0x7f6aec0cec20 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af0dfc700 1 -- 172.21.15.104:0/4010577696 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae800b9d0 con 0x7f6aec0cec20 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.136+0000 7f6af0dfc700 1 -- 172.21.15.104:0/4010577696 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800bd60 con 0x7f6aec0cec20 2023-06-10T18:59:27.146 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 -- 172.21.15.104:0/4010577696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec0cec20 msgr2=0x7f6aec0cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 --2- 172.21.15.104:0/4010577696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec0cec20 0x7f6aec0cd250 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f6aec06ae70 tx=0x7f6ae8009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 -- 172.21.15.104:0/4010577696 shutdown_connections 2023-06-10T18:59:27.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 --2- 172.21.15.104:0/4010577696 >> v2:172.21.15.114:3300/0 conn(0x7f6aec156740 0x7f6aec15d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 --2- 172.21.15.104:0/4010577696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec0cec20 0x7f6aec0cd250 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.147 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 -- 172.21.15.104:0/4010577696 >> 172.21.15.104:0/4010577696 conn(0x7f6aec06d240 msgr2=0x7f6aec06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:27.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 -- 172.21.15.104:0/4010577696 shutdown_connections 2023-06-10T18:59:27.148 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.137+0000 7f6af4062700 1 -- 172.21.15.104:0/4010577696 wait complete. 2023-06-10T18:59:27.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.138+0000 7f6af4062700 1 Processor -- start 2023-06-10T18:59:27.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 -- start start 2023-06-10T18:59:27.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 0x7f6aec0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec0b4a70 0x7f6aec147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0b4f10 con 0x7f6aec069e70 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec1f2dc0 con 0x7f6aec0b4a70 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.139+0000 7f6af4062700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec1484f0 con 0x7f6aec156740 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.151 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41620/0 (socket says 172.21.15.104:41620) 2023-06-10T18:59:27.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 -- 172.21.15.104:0/1145197355 learned_addr learned my addr 172.21.15.104:0/1145197355 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af25ff700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec0b4a70 0x7f6aec147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af15fd700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 0x7f6aec0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 -- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 msgr2=0x7f6aec0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 0x7f6aec0b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 -- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec0b4a70 msgr2=0x7f6aec147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec0b4a70 0x7f6aec147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 -- 172.21.15.104:0/1145197355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae8009550 con 0x7f6aec069e70 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af15fd700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 0x7f6aec0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af1dfe700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00d830 tx=0x7f6adc00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0125f0 con 0x7f6aec069e70 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc012750 con 0x7f6aec069e70 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec148210 con 0x7f6aec069e70 2023-06-10T18:59:27.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc012b00 con 0x7f6aec069e70 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.140+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec145940 con 0x7f6aec069e70 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.141+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec1454c0 con 0x7f6aec069e70 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.142+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6adc012c60 con 0x7f6aec069e70 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.142+0000 7f6ae27fc700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 0x7f6ad006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.142+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6adc0899d0 con 0x7f6aec069e70 2023-06-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.142+0000 7f6af15fd700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 0x7f6ad006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.143+0000 7f6af15fd700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 0x7f6ad006d190 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009130 tx=0x7f6ae8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.146+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6aec1454c0 con 0x7f6aec069e70 2023-06-10T18:59:27.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.329+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6aec156b10 con 0x7f6ad006ace0 2023-06-10T18:59:27.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.338+0000 7f6ae27fc700 1 -- 172.21.15.104:0/1145197355 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296879 (secure 0 0 0) 0x7f6ae800b850 con 0x7f6ad006ace0 2023-06-10T18:59:27.346 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:27.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 msgr2=0x7f6ad006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 0x7f6ad006d190 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009130 tx=0x7f6ae8008030 comp rx=0 tx=0).stop 2023-06-10T18:59:27.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 msgr2=0x7f6aec144e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00d830 tx=0x7f6adc00dbf0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 shutdown_connections 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6ad006ace0 0x7f6ad006d190 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec0b4a70 0x7f6aec147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6aec069e70 0x7f6aec144e00 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 --2- 172.21.15.104:0/1145197355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6aec156740 0x7f6aec0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 >> 172.21.15.104:0/1145197355 conn(0x7f6aec06d240 msgr2=0x7f6aec0b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:27.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 shutdown_connections 2023-06-10T18:59:27.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.341+0000 7f6af4062700 1 -- 172.21.15.104:0/1145197355 wait complete. 2023-06-10T18:59:27.350 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:59:27.358 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2023-06-10T18:59:25.593405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":267512,"kb_used_data":19128,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754707208,"statfs":{"total":773094113280,"available":772820180992,"internally_reserved":0,"allocated":19587072,"data_stored":10220964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000866"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.338977+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:59:19.338977+0000","last_peered":"2023-06-10T18:59:19.338977+0000","last_clean":"2023-06-10T18:59:19.338977+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:59:19.338977+0000","last_undegraded":"2023-06-10T18:59:19.338977+0000","last_fullsized":"2023-06-10T18:59:19.338977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:06:11.905499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":336,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.180043+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:59:19.180043+0000","last_peered":"2023-06-10T18:59:19.180043+0000","last_clean":"2023-06-10T18:59:19.180043+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:59:19.180043+0000","last_undegraded":"2023-06-10T18:59:19.180043+0000","last_fullsized":"2023-06-10T18:59:19.180043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:51:02.524233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":288,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.338911+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:59:19.338911+0000","last_peered":"2023-06-10T18:59:19.338911+0000","last_clean":"2023-06-10T18:59:19.338911+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:59:19.338911+0000","last_undegraded":"2023-06-10T18:59:19.338911+0000","last_fullsized":"2023-06-10T18:59:19.338911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:40:20.155544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":336,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.180056+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:59:19.180056+0000","last_peered":"2023-06-10T18:59:19.180056+0000","last_clean":"2023-06-10T18:59:19.180056+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:59:19.180056+0000","last_undegraded":"2023-06-10T18:59:19.180056+0000","last_fullsized":"2023-06-10T18:59:19.180056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:23:25.610448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":462,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.339456+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:59:19.339456+0000","last_peered":"2023-06-10T18:59:19.339456+0000","last_clean":"2023-06-10T18:59:19.339456+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:59:19.339456+0000","last_undegraded":"2023-06-10T18:59:19.339456+0000","last_fullsized":"2023-06-10T18:59:19.339456+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:50:25.554699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":306,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.339432+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:59:19.339432+0000","last_peered":"2023-06-10T18:59:19.339432+0000","last_clean":"2023-06-10T18:59:19.339432+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:59:19.339432+0000","last_undegraded":"2023-06-10T18:59:19.339432+0000","last_fullsized":"2023-06-10T18:59:19.339432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:07:12.904339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":306,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.339464+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:59:19.339464+0000","last_peered":"2023-06-10T18:59:19.339464+0000","last_clean":"2023-06-10T18:59:19.339464+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:59:19.339464+0000","last_undegraded":"2023-06-10T18:59:19.339464+0000","last_fullsized":"2023-06-10T18:59:19.339464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:10:43.531869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":294,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.339046+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:59:19.339046+0000","last_peered":"2023-06-10T18:59:19.339046+0000","last_clean":"2023-06-10T18:59:19.339046+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:59:19.339046+0000","last_undegraded":"2023-06-10T18:59:19.339046+0000","last_fullsized":"2023-06-10T18:59:19.339046+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:54:42.858596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:22.842705+0000","last_change":"2023-06-10T18:59:22.842705+0000","last_active":"2023-06-10T18:59:22.842705+0000","last_peered":"2023-06-10T18:59:22.842705+0000","last_clean":"2023-06-10T18:59:22.842705+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:22.842705+0000","last_undegraded":"2023-06-10T18:59:22.842705+0000","last_fullsized":"2023-06-10T18:59:22.842705+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-06-10T18:59:21.838366+0000","last_change":"2023-06-10T18:59:21.838366+0000","last_active":"2023-06-10T18:59:21.838366+0000","last_peered":"2023-06-10T18:59:21.838366+0000","last_clean":"2023-06-10T18:59:21.838366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-06-10T18:59:21.838366+0000","last_undegraded":"2023-06-10T18:59:21.838366+0000","last_fullsized":"2023-06-10T18:59:21.838366+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":284,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-06-10T18:59:19.339129+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:59:19.339129+0000","last_peered":"2023-06-10T18:59:19.339129+0000","last_clean":"2023-06-10T18:59:19.339129+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:59:19.339129+0000","last_undegraded":"2023-06-10T18:59:19.339129+0000","last_fullsized":"2023-06-10T18:59:19.339129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:57:41.450880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575037,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33876,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337964,"statfs":{"total":96636764160,"available":96602075136,"internally_reserved":0,"allocated":2895872,"data_stored":1720906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.372},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33292,"kb_used_data":2244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338548,"statfs":{"total":96636764160,"available":96602673152,"internally_reserved":0,"allocated":2297856,"data_stored":1129519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.502},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.53500000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33292,"kb_used_data":2244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338548,"statfs":{"total":96636764160,"available":96602673152,"internally_reserved":0,"allocated":2297856,"data_stored":1129519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.32600000000000001},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.753},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.55600000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34200000000000003},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.52400000000000002,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.312},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.71899999999999997},"last":0.32200000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.29699999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81200000000000006,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.622},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61599999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":0.85999999999999999,"15min":1.008},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":1.206,"15min":1.206},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.79500000000000004,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.64600000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.224,"15min":1.224},"last":0.626},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.97099999999999997},"last":0.55000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575037,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":2824,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":2891776,"data_stored":1719887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33900000000000002,"15min":0.33800000000000002},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.47299999999999998,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.34999999999999998},"min":{"1min":0.23400000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.505,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.31}]},{"osd":2,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.435,"15min":0.38900000000000001},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52600000000000002,"5min":11.439,"15min":11.439},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.48199999999999998,"15min":0.41699999999999998},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":0.54000000000000004,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.27300000000000002,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.48999999999999999,"5min":5.766,"15min":5.766},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.27000000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":0.64800000000000002,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.31900000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.34699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79700000000000004,"5min":2.802,"15min":2.802},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.71799999999999997,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.627}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77100000000000002,"5min":2.907,"15min":2.907},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58399999999999996,"15min":0.59899999999999998},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.71499999999999997,"5min":0.86099999999999999,"15min":1.363},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.36099999999999999},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":1.0489999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72899999999999998,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.70399999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2248,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2301952,"data_stored":1130538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.318,"15min":0.34200000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.45300000000000001,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.38500000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.48299999999999998,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.377},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47099999999999997,"5min":0.57599999999999996,"15min":2.847},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36699999999999999,"15min":0.39900000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.45900000000000002,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.28499999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34699999999999998,"15min":0.38500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.442,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.308},{"interface":"front","average":{"1min":0.379,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.25800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54100000000000004,"5min":1.24,"15min":2.8999999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.61499999999999999},"min":{"1min":0.41599999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":1.208,"15min":5.742},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58499999999999996,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81200000000000006,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.73599999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61799999999999999,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":6.593},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.64900000000000002},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.78800000000000003,"5min":1.002,"15min":7.0880000000000001},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":0.79500000000000004,"5min":1.026,"15min":8.7850000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59899999999999998,"15min":0.622},"min":{"1min":0.45300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":1.171,"15min":5.6180000000000003},"last":0.59799999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:58:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.64300000000000002,"15min":0.66400000000000003},"min":{"1min":0.47399999999999998,"5min":0.42199999999999999,"15min":0.40699999999999997},"max":{"1min":0.84399999999999997,"5min":0.87,"15min":6.617},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.65900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.626}]}]},{"osd":3,"up_from":13,"seq":55834575037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2248,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2301952,"data_stored":1130538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.317}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.499}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.39700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33292,"kb_used_data":2244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338548,"statfs":{"total":96636764160,"available":96602673152,"internally_reserved":0,"allocated":2297856,"data_stored":1129519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.71999999999999997,"5min":0.745,"15min":1.119},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.1639999999999999},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77800000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.873,"15min":0.96599999999999997},"last":0.35799999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.67100000000000004,"5min":0.72799999999999998,"15min":0.80800000000000005},"last":0.435},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.33200000000000002},"max":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.44500000000000001,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.79800000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.042},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.67800000000000005,"5min":0.94599999999999995,"15min":1.069},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.501,"15min":0.504},"min":{"1min":0.36299999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.88300000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.90100000000000002,"15min":1.1459999999999999},"last":0.58999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2248,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2301952,"data_stored":1130538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.51800000000000002,"15min":0.53000000000000003},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.81000000000000005},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.35899999999999999,"15min":0.33300000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.35799999999999998,"5min":0.34000000000000002,"15min":0.307},"max":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.85099999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.373,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51400000000000001,"15min":0.49299999999999999},"min":{"1min":0.41899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.83999999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29399999999999998},"max":{"1min":0.72799999999999998,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.38300000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53700000000000003,"15min":0.51600000000000001},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.879},"last":0.50800000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.54900000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:59:27.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T18:59:27.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.533+0000 7f8a0bb7f700 1 Processor -- start 2023-06-10T18:59:27.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.533+0000 7f8a0bb7f700 1 -- start start 2023-06-10T18:59:27.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0bb7f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a040ade00 0x7f8a040ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0bb7f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040aeb00 0x7f8a040ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0bb7f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040cee70 con 0x7f8a040ada30 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0bb7f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040cf430 con 0x7f8a040ade00 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0bb7f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040cf150 con 0x7f8a040aeb00 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0911a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a040ade00 0x7f8a040ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0911a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8a040ade00 0x7f8a040ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41630/0 (socket says 172.21.15.104:41630) 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.534+0000 7f8a0911a700 1 -- 172.21.15.104:0/3118472186 learned_addr learned my addr 172.21.15.104:0/3118472186 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0a11c700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040aeb00 0x7f8a040ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0911a700 1 -- 172.21.15.104:0/3118472186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040aeb00 msgr2=0x7f8a040ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0911a700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040aeb00 0x7f8a040ce5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0911a700 1 -- 172.21.15.104:0/3118472186 >> v1:172.21.15.114:6789/0 conn(0x7f8a040ada30 legacy=0x7f8a040ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0911a700 1 -- 172.21.15.104:0/3118472186 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0406cf10 con 0x7f8a040ade00 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0911a700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a040ade00 0x7f8a040ce0b0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098f0 tx=0x7f8a00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac07b641ac92a2e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a08919700 1 -- 172.21.15.104:0/3118472186 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00004230 con 0x7f8a040ade00 2023-06-10T18:59:27.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a08919700 1 -- 172.21.15.104:0/3118472186 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a00004390 con 0x7f8a040ade00 2023-06-10T18:59:27.544 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a08919700 1 -- 172.21.15.104:0/3118472186 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a000036a0 con 0x7f8a040ade00 2023-06-10T18:59:27.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/3118472186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a040ade00 msgr2=0x7f8a040ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.535+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a040ade00 0x7f8a040ce0b0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098f0 tx=0x7f8a00009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:27.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/3118472186 shutdown_connections 2023-06-10T18:59:27.546 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040aeb00 0x7f8a040ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/3118472186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a040ade00 0x7f8a040ce0b0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/3118472186 >> 172.21.15.104:0/3118472186 conn(0x7f8a0406b8a0 msgr2=0x7f8a040cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/3118472186 shutdown_connections 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.536+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/3118472186 wait complete. 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.537+0000 7f8a0bb7f700 1 Processor -- start 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.537+0000 7f8a0bb7f700 1 -- start start 2023-06-10T18:59:27.547 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040ada30 0x7f8a041595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a040aeb00 0x7f8a04159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041f1a10 con 0x7f8a0409cdf0 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b3bf0 con 0x7f8a040aeb00 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0bb7f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b3ed0 con 0x7f8a040ada30 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0a11c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.548 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0a11c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41638/0 (socket says 172.21.15.104:41638) 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0a11c700 1 -- 172.21.15.104:0/1223082845 learned_addr learned my addr 172.21.15.104:0/1223082845 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0911a700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a040aeb00 0x7f8a04159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.538+0000 7f8a0991b700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040ada30 0x7f8a041595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 -- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040ada30 msgr2=0x7f8a041595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040ada30 0x7f8a041595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 -- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a040aeb00 msgr2=0x7f8a04159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a040aeb00 0x7f8a04159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.549 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 -- 172.21.15.104:0/1223082845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a00009550 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0a11c700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00cd60 tx=0x7f89fc00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc003a20 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc003ba0 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc004020 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a040b41b0 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.539+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a040b33e0 con 0x7f8a0409cdf0 2023-06-10T18:59:27.550 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.540+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a040b46d0 con 0x7f8a0409cdf0 2023-06-10T18:59:27.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.540+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f89fc004180 con 0x7f8a0409cdf0 2023-06-10T18:59:27.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.541+0000 7f89fa7fc700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 0x7f89e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.541+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f89fc012070 con 0x7f8a0409cdf0 2023-06-10T18:59:27.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.541+0000 7f8a0991b700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 0x7f89e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.541+0000 7f8a0991b700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 0x7f89e806d190 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.545+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8a040b46d0 con 0x7f8a0409cdf0 2023-06-10T18:59:27.732 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.727+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a040b4040 con 0x7f89e806ace0 2023-06-10T18:59:27.741 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.736+0000 7f89fa7fc700 1 -- 172.21.15.104:0/1223082845 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312003 (secure 0 0 0) 0x7f8a040b4040 con 0x7f89e806ace0 2023-06-10T18:59:27.743 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 msgr2=0x7f89e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 0x7f89e806d190 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4008040 comp rx=0 tx=0).stop 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 msgr2=0x7f8a040b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00cd60 tx=0x7f89fc00a570 comp rx=0 tx=0).stop 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 shutdown_connections 2023-06-10T18:59:27.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a0409cdf0 0x7f8a040b2ea0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a040ada30 0x7f8a041595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f89e806ace0 0x7f89e806d190 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 --2- 172.21.15.104:0/1223082845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a040aeb00 0x7f8a04159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 >> 172.21.15.104:0/1223082845 conn(0x7f8a0406b8a0 msgr2=0x7f8a040b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 shutdown_connections 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.739+0000 7f8a0bb7f700 1 -- 172.21.15.104:0/1223082845 wait complete. 2023-06-10T18:59:27.748 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T18:59:27.757 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2023-06-10T18:59:27.593986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267960,"kb_used_data":19576,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706760,"statfs":{"total":773094113280,"available":772819722240,"internally_reserved":0,"allocated":20045824,"data_stored":10448560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.744979"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989489+0000","last_change":"2023-06-10T18:59:23.267779+0000","last_active":"2023-06-10T18:59:24.989489+0000","last_peered":"2023-06-10T18:59:24.989489+0000","last_clean":"2023-06-10T18:59:24.989489+0000","last_became_active":"2023-06-10T18:59:23.267504+0000","last_became_peered":"2023-06-10T18:59:23.267504+0000","last_unstale":"2023-06-10T18:59:24.989489+0000","last_undegraded":"2023-06-10T18:59:24.989489+0000","last_fullsized":"2023-06-10T18:59:24.989489+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:38:14.678984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258609+0000","last_change":"2023-06-10T18:59:23.269251+0000","last_active":"2023-06-10T18:59:24.258609+0000","last_peered":"2023-06-10T18:59:24.258609+0000","last_clean":"2023-06-10T18:59:24.258609+0000","last_became_active":"2023-06-10T18:59:23.268969+0000","last_became_peered":"2023-06-10T18:59:23.268969+0000","last_unstale":"2023-06-10T18:59:24.258609+0000","last_undegraded":"2023-06-10T18:59:24.258609+0000","last_fullsized":"2023-06-10T18:59:24.258609+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:25:00.074408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989457+0000","last_change":"2023-06-10T18:59:23.275053+0000","last_active":"2023-06-10T18:59:24.989457+0000","last_peered":"2023-06-10T18:59:24.989457+0000","last_clean":"2023-06-10T18:59:24.989457+0000","last_became_active":"2023-06-10T18:59:23.273850+0000","last_became_peered":"2023-06-10T18:59:23.273850+0000","last_unstale":"2023-06-10T18:59:24.989457+0000","last_undegraded":"2023-06-10T18:59:24.989457+0000","last_fullsized":"2023-06-10T18:59:24.989457+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:06:25.920426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989603+0000","last_change":"2023-06-10T18:59:23.274615+0000","last_active":"2023-06-10T18:59:24.989603+0000","last_peered":"2023-06-10T18:59:24.989603+0000","last_clean":"2023-06-10T18:59:24.989603+0000","last_became_active":"2023-06-10T18:59:23.273942+0000","last_became_peered":"2023-06-10T18:59:23.273942+0000","last_unstale":"2023-06-10T18:59:24.989603+0000","last_undegraded":"2023-06-10T18:59:24.989603+0000","last_fullsized":"2023-06-10T18:59:24.989603+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:52:39.594989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258452+0000","last_change":"2023-06-10T18:59:23.275953+0000","last_active":"2023-06-10T18:59:24.258452+0000","last_peered":"2023-06-10T18:59:24.258452+0000","last_clean":"2023-06-10T18:59:24.258452+0000","last_became_active":"2023-06-10T18:59:23.275698+0000","last_became_peered":"2023-06-10T18:59:23.275698+0000","last_unstale":"2023-06-10T18:59:24.258452+0000","last_undegraded":"2023-06-10T18:59:24.258452+0000","last_fullsized":"2023-06-10T18:59:24.258452+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:44:35.513207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258423+0000","last_change":"2023-06-10T18:59:23.269891+0000","last_active":"2023-06-10T18:59:24.258423+0000","last_peered":"2023-06-10T18:59:24.258423+0000","last_clean":"2023-06-10T18:59:24.258423+0000","last_became_active":"2023-06-10T18:59:23.269626+0000","last_became_peered":"2023-06-10T18:59:23.269626+0000","last_unstale":"2023-06-10T18:59:24.258423+0000","last_undegraded":"2023-06-10T18:59:24.258423+0000","last_fullsized":"2023-06-10T18:59:24.258423+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:50:38.515257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989392+0000","last_change":"2023-06-10T18:59:23.273575+0000","last_active":"2023-06-10T18:59:24.989392+0000","last_peered":"2023-06-10T18:59:24.989392+0000","last_clean":"2023-06-10T18:59:24.989392+0000","last_became_active":"2023-06-10T18:59:23.272868+0000","last_became_peered":"2023-06-10T18:59:23.272868+0000","last_unstale":"2023-06-10T18:59:24.989392+0000","last_undegraded":"2023-06-10T18:59:24.989392+0000","last_fullsized":"2023-06-10T18:59:24.989392+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:55:41.013375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258067+0000","last_change":"2023-06-10T18:59:23.271274+0000","last_active":"2023-06-10T18:59:24.258067+0000","last_peered":"2023-06-10T18:59:24.258067+0000","last_clean":"2023-06-10T18:59:24.258067+0000","last_became_active":"2023-06-10T18:59:23.270995+0000","last_became_peered":"2023-06-10T18:59:23.270995+0000","last_unstale":"2023-06-10T18:59:24.258067+0000","last_undegraded":"2023-06-10T18:59:24.258067+0000","last_fullsized":"2023-06-10T18:59:24.258067+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:05:59.213468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258181+0000","last_change":"2023-06-10T18:59:23.269144+0000","last_active":"2023-06-10T18:59:24.258181+0000","last_peered":"2023-06-10T18:59:24.258181+0000","last_clean":"2023-06-10T18:59:24.258181+0000","last_became_active":"2023-06-10T18:59:23.268856+0000","last_became_peered":"2023-06-10T18:59:23.268856+0000","last_unstale":"2023-06-10T18:59:24.258181+0000","last_undegraded":"2023-06-10T18:59:24.258181+0000","last_fullsized":"2023-06-10T18:59:24.258181+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:59:26.239252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257813+0000","last_change":"2023-06-10T18:59:23.263341+0000","last_active":"2023-06-10T18:59:24.257813+0000","last_peered":"2023-06-10T18:59:24.257813+0000","last_clean":"2023-06-10T18:59:24.257813+0000","last_became_active":"2023-06-10T18:59:23.263071+0000","last_became_peered":"2023-06-10T18:59:23.263071+0000","last_unstale":"2023-06-10T18:59:24.257813+0000","last_undegraded":"2023-06-10T18:59:24.257813+0000","last_fullsized":"2023-06-10T18:59:24.257813+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:29.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989189+0000","last_change":"2023-06-10T18:59:23.279014+0000","last_active":"2023-06-10T18:59:24.989189+0000","last_peered":"2023-06-10T18:59:24.989189+0000","last_clean":"2023-06-10T18:59:24.989189+0000","last_became_active":"2023-06-10T18:59:23.278768+0000","last_became_peered":"2023-06-10T18:59:23.278768+0000","last_unstale":"2023-06-10T18:59:24.989189+0000","last_undegraded":"2023-06-10T18:59:24.989189+0000","last_fullsized":"2023-06-10T18:59:24.989189+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:40:27.338718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990187+0000","last_change":"2023-06-10T18:59:23.277447+0000","last_active":"2023-06-10T18:59:24.990187+0000","last_peered":"2023-06-10T18:59:24.990187+0000","last_clean":"2023-06-10T18:59:24.990187+0000","last_became_active":"2023-06-10T18:59:23.277142+0000","last_became_peered":"2023-06-10T18:59:23.277142+0000","last_unstale":"2023-06-10T18:59:24.990187+0000","last_undegraded":"2023-06-10T18:59:24.990187+0000","last_fullsized":"2023-06-10T18:59:24.990187+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:17:11.053743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989579+0000","last_change":"2023-06-10T18:59:23.278934+0000","last_active":"2023-06-10T18:59:24.989579+0000","last_peered":"2023-06-10T18:59:24.989579+0000","last_clean":"2023-06-10T18:59:24.989579+0000","last_became_active":"2023-06-10T18:59:23.277528+0000","last_became_peered":"2023-06-10T18:59:23.277528+0000","last_unstale":"2023-06-10T18:59:24.989579+0000","last_undegraded":"2023-06-10T18:59:24.989579+0000","last_fullsized":"2023-06-10T18:59:24.989579+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:11:30.903509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989647+0000","last_change":"2023-06-10T18:59:23.277451+0000","last_active":"2023-06-10T18:59:24.989647+0000","last_peered":"2023-06-10T18:59:24.989647+0000","last_clean":"2023-06-10T18:59:24.989647+0000","last_became_active":"2023-06-10T18:59:23.277111+0000","last_became_peered":"2023-06-10T18:59:23.277111+0000","last_unstale":"2023-06-10T18:59:24.989647+0000","last_undegraded":"2023-06-10T18:59:24.989647+0000","last_fullsized":"2023-06-10T18:59:24.989647+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:21:15.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989600+0000","last_change":"2023-06-10T18:59:23.274218+0000","last_active":"2023-06-10T18:59:24.989600+0000","last_peered":"2023-06-10T18:59:24.989600+0000","last_clean":"2023-06-10T18:59:24.989600+0000","last_became_active":"2023-06-10T18:59:23.273967+0000","last_became_peered":"2023-06-10T18:59:23.273967+0000","last_unstale":"2023-06-10T18:59:24.989600+0000","last_undegraded":"2023-06-10T18:59:24.989600+0000","last_fullsized":"2023-06-10T18:59:24.989600+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:39:24.047220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.256974+0000","last_change":"2023-06-10T18:59:23.277458+0000","last_active":"2023-06-10T18:59:24.256974+0000","last_peered":"2023-06-10T18:59:24.256974+0000","last_clean":"2023-06-10T18:59:24.256974+0000","last_became_active":"2023-06-10T18:59:23.277186+0000","last_became_peered":"2023-06-10T18:59:23.277186+0000","last_unstale":"2023-06-10T18:59:24.256974+0000","last_undegraded":"2023-06-10T18:59:24.256974+0000","last_fullsized":"2023-06-10T18:59:24.256974+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:24:40.339791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989671+0000","last_change":"2023-06-10T18:59:23.266704+0000","last_active":"2023-06-10T18:59:24.989671+0000","last_peered":"2023-06-10T18:59:24.989671+0000","last_clean":"2023-06-10T18:59:24.989671+0000","last_became_active":"2023-06-10T18:59:23.266453+0000","last_became_peered":"2023-06-10T18:59:23.266453+0000","last_unstale":"2023-06-10T18:59:24.989671+0000","last_undegraded":"2023-06-10T18:59:24.989671+0000","last_fullsized":"2023-06-10T18:59:24.989671+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:47:56.725351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257877+0000","last_change":"2023-06-10T18:59:23.274015+0000","last_active":"2023-06-10T18:59:24.257877+0000","last_peered":"2023-06-10T18:59:24.257877+0000","last_clean":"2023-06-10T18:59:24.257877+0000","last_became_active":"2023-06-10T18:59:23.273763+0000","last_became_peered":"2023-06-10T18:59:23.273763+0000","last_unstale":"2023-06-10T18:59:24.257877+0000","last_undegraded":"2023-06-10T18:59:24.257877+0000","last_fullsized":"2023-06-10T18:59:24.257877+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:48:26.862443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257402+0000","last_change":"2023-06-10T18:59:23.263363+0000","last_active":"2023-06-10T18:59:24.257402+0000","last_peered":"2023-06-10T18:59:24.257402+0000","last_clean":"2023-06-10T18:59:24.257402+0000","last_became_active":"2023-06-10T18:59:23.263095+0000","last_became_peered":"2023-06-10T18:59:23.263095+0000","last_unstale":"2023-06-10T18:59:24.257402+0000","last_undegraded":"2023-06-10T18:59:24.257402+0000","last_fullsized":"2023-06-10T18:59:24.257402+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:58:02.218733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258654+0000","last_change":"2023-06-10T18:59:23.271228+0000","last_active":"2023-06-10T18:59:24.258654+0000","last_peered":"2023-06-10T18:59:24.258654+0000","last_clean":"2023-06-10T18:59:24.258654+0000","last_became_active":"2023-06-10T18:59:23.270930+0000","last_became_peered":"2023-06-10T18:59:23.270930+0000","last_unstale":"2023-06-10T18:59:24.258654+0000","last_undegraded":"2023-06-10T18:59:24.258654+0000","last_fullsized":"2023-06-10T18:59:24.258654+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:12:48.517349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990146+0000","last_change":"2023-06-10T18:59:23.274708+0000","last_active":"2023-06-10T18:59:24.990146+0000","last_peered":"2023-06-10T18:59:24.990146+0000","last_clean":"2023-06-10T18:59:24.990146+0000","last_became_active":"2023-06-10T18:59:23.274418+0000","last_became_peered":"2023-06-10T18:59:23.274418+0000","last_unstale":"2023-06-10T18:59:24.990146+0000","last_undegraded":"2023-06-10T18:59:24.990146+0000","last_fullsized":"2023-06-10T18:59:24.990146+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:02:28.094212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990513+0000","last_change":"2023-06-10T18:59:23.277288+0000","last_active":"2023-06-10T18:59:24.990513+0000","last_peered":"2023-06-10T18:59:24.990513+0000","last_clean":"2023-06-10T18:59:24.990513+0000","last_became_active":"2023-06-10T18:59:23.277033+0000","last_became_peered":"2023-06-10T18:59:23.277033+0000","last_unstale":"2023-06-10T18:59:24.990513+0000","last_undegraded":"2023-06-10T18:59:24.990513+0000","last_fullsized":"2023-06-10T18:59:24.990513+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:54:59.878729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989389+0000","last_change":"2023-06-10T18:59:23.275671+0000","last_active":"2023-06-10T18:59:24.989389+0000","last_peered":"2023-06-10T18:59:24.989389+0000","last_clean":"2023-06-10T18:59:24.989389+0000","last_became_active":"2023-06-10T18:59:23.275378+0000","last_became_peered":"2023-06-10T18:59:23.275378+0000","last_unstale":"2023-06-10T18:59:24.989389+0000","last_undegraded":"2023-06-10T18:59:24.989389+0000","last_fullsized":"2023-06-10T18:59:24.989389+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:00:32.210942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989551+0000","last_change":"2023-06-10T18:59:23.274609+0000","last_active":"2023-06-10T18:59:24.989551+0000","last_peered":"2023-06-10T18:59:24.989551+0000","last_clean":"2023-06-10T18:59:24.989551+0000","last_became_active":"2023-06-10T18:59:23.274331+0000","last_became_peered":"2023-06-10T18:59:23.274331+0000","last_unstale":"2023-06-10T18:59:24.989551+0000","last_undegraded":"2023-06-10T18:59:24.989551+0000","last_fullsized":"2023-06-10T18:59:24.989551+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:56:23.293350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258246+0000","last_change":"2023-06-10T18:59:23.266216+0000","last_active":"2023-06-10T18:59:24.258246+0000","last_peered":"2023-06-10T18:59:24.258246+0000","last_clean":"2023-06-10T18:59:24.258246+0000","last_became_active":"2023-06-10T18:59:23.265789+0000","last_became_peered":"2023-06-10T18:59:23.265789+0000","last_unstale":"2023-06-10T18:59:24.258246+0000","last_undegraded":"2023-06-10T18:59:24.258246+0000","last_fullsized":"2023-06-10T18:59:24.258246+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:48:45.354038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989939+0000","last_change":"2023-06-10T18:59:23.269926+0000","last_active":"2023-06-10T18:59:24.989939+0000","last_peered":"2023-06-10T18:59:24.989939+0000","last_clean":"2023-06-10T18:59:24.989939+0000","last_became_active":"2023-06-10T18:59:23.269657+0000","last_became_peered":"2023-06-10T18:59:23.269657+0000","last_unstale":"2023-06-10T18:59:24.989939+0000","last_undegraded":"2023-06-10T18:59:24.989939+0000","last_fullsized":"2023-06-10T18:59:24.989939+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:12:32.775072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990189+0000","last_change":"2023-06-10T18:59:23.273312+0000","last_active":"2023-06-10T18:59:24.990189+0000","last_peered":"2023-06-10T18:59:24.990189+0000","last_clean":"2023-06-10T18:59:24.990189+0000","last_became_active":"2023-06-10T18:59:23.273034+0000","last_became_peered":"2023-06-10T18:59:23.273034+0000","last_unstale":"2023-06-10T18:59:24.990189+0000","last_undegraded":"2023-06-10T18:59:24.990189+0000","last_fullsized":"2023-06-10T18:59:24.990189+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:24:23.381120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989093+0000","last_change":"2023-06-10T18:59:23.273202+0000","last_active":"2023-06-10T18:59:24.989093+0000","last_peered":"2023-06-10T18:59:24.989093+0000","last_clean":"2023-06-10T18:59:24.989093+0000","last_became_active":"2023-06-10T18:59:23.272939+0000","last_became_peered":"2023-06-10T18:59:23.272939+0000","last_unstale":"2023-06-10T18:59:24.989093+0000","last_undegraded":"2023-06-10T18:59:24.989093+0000","last_fullsized":"2023-06-10T18:59:24.989093+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:24:51.362977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989713+0000","last_change":"2023-06-10T18:59:23.279139+0000","last_active":"2023-06-10T18:59:24.989713+0000","last_peered":"2023-06-10T18:59:24.989713+0000","last_clean":"2023-06-10T18:59:24.989713+0000","last_became_active":"2023-06-10T18:59:23.277841+0000","last_became_peered":"2023-06-10T18:59:23.277841+0000","last_unstale":"2023-06-10T18:59:24.989713+0000","last_undegraded":"2023-06-10T18:59:24.989713+0000","last_fullsized":"2023-06-10T18:59:24.989713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:04:01.121396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258039+0000","last_change":"2023-06-10T18:59:23.270177+0000","last_active":"2023-06-10T18:59:24.258039+0000","last_peered":"2023-06-10T18:59:24.258039+0000","last_clean":"2023-06-10T18:59:24.258039+0000","last_became_active":"2023-06-10T18:59:23.269900+0000","last_became_peered":"2023-06-10T18:59:23.269900+0000","last_unstale":"2023-06-10T18:59:24.258039+0000","last_undegraded":"2023-06-10T18:59:24.258039+0000","last_fullsized":"2023-06-10T18:59:24.258039+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:57:49.283588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.256935+0000","last_change":"2023-06-10T18:59:23.266548+0000","last_active":"2023-06-10T18:59:24.256935+0000","last_peered":"2023-06-10T18:59:24.256935+0000","last_clean":"2023-06-10T18:59:24.256935+0000","last_became_active":"2023-06-10T18:59:23.265808+0000","last_became_peered":"2023-06-10T18:59:23.265808+0000","last_unstale":"2023-06-10T18:59:24.256935+0000","last_undegraded":"2023-06-10T18:59:24.256935+0000","last_fullsized":"2023-06-10T18:59:24.256935+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:42:11.914707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989579+0000","last_change":"2023-06-10T18:59:23.269572+0000","last_active":"2023-06-10T18:59:24.989579+0000","last_peered":"2023-06-10T18:59:24.989579+0000","last_clean":"2023-06-10T18:59:24.989579+0000","last_became_active":"2023-06-10T18:59:23.269278+0000","last_became_peered":"2023-06-10T18:59:23.269278+0000","last_unstale":"2023-06-10T18:59:24.989579+0000","last_undegraded":"2023-06-10T18:59:24.989579+0000","last_fullsized":"2023-06-10T18:59:24.989579+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:58:52.705633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990088+0000","last_change":"2023-06-10T18:59:23.269059+0000","last_active":"2023-06-10T18:59:24.990088+0000","last_peered":"2023-06-10T18:59:24.990088+0000","last_clean":"2023-06-10T18:59:24.990088+0000","last_became_active":"2023-06-10T18:59:23.268748+0000","last_became_peered":"2023-06-10T18:59:23.268748+0000","last_unstale":"2023-06-10T18:59:24.990088+0000","last_undegraded":"2023-06-10T18:59:24.990088+0000","last_fullsized":"2023-06-10T18:59:24.990088+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:38:29.857169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258564+0000","last_change":"2023-06-10T18:59:22.858075+0000","last_active":"2023-06-10T18:59:24.258564+0000","last_peered":"2023-06-10T18:59:24.258564+0000","last_clean":"2023-06-10T18:59:24.258564+0000","last_became_active":"2023-06-10T18:59:22.857802+0000","last_became_peered":"2023-06-10T18:59:22.857802+0000","last_unstale":"2023-06-10T18:59:24.258564+0000","last_undegraded":"2023-06-10T18:59:24.258564+0000","last_fullsized":"2023-06-10T18:59:24.258564+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:49:01.394698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989460+0000","last_change":"2023-06-10T18:59:23.278772+0000","last_active":"2023-06-10T18:59:24.989460+0000","last_peered":"2023-06-10T18:59:24.989460+0000","last_clean":"2023-06-10T18:59:24.989460+0000","last_became_active":"2023-06-10T18:59:23.278181+0000","last_became_peered":"2023-06-10T18:59:23.278181+0000","last_unstale":"2023-06-10T18:59:24.989460+0000","last_undegraded":"2023-06-10T18:59:24.989460+0000","last_fullsized":"2023-06-10T18:59:24.989460+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:48:27.807486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990047+0000","last_change":"2023-06-10T18:59:22.861736+0000","last_active":"2023-06-10T18:59:24.990047+0000","last_peered":"2023-06-10T18:59:24.990047+0000","last_clean":"2023-06-10T18:59:24.990047+0000","last_became_active":"2023-06-10T18:59:22.861458+0000","last_became_peered":"2023-06-10T18:59:22.861458+0000","last_unstale":"2023-06-10T18:59:24.990047+0000","last_undegraded":"2023-06-10T18:59:24.990047+0000","last_fullsized":"2023-06-10T18:59:24.990047+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:27:20.417772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989760+0000","last_change":"2023-06-10T18:59:23.277898+0000","last_active":"2023-06-10T18:59:24.989760+0000","last_peered":"2023-06-10T18:59:24.989760+0000","last_clean":"2023-06-10T18:59:24.989760+0000","last_became_active":"2023-06-10T18:59:23.277574+0000","last_became_peered":"2023-06-10T18:59:23.277574+0000","last_unstale":"2023-06-10T18:59:24.989760+0000","last_undegraded":"2023-06-10T18:59:24.989760+0000","last_fullsized":"2023-06-10T18:59:24.989760+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:55:16.042714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"186'3","reported_seq":16,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257637+0000","last_change":"2023-06-10T18:59:22.863919+0000","last_active":"2023-06-10T18:59:24.257637+0000","last_peered":"2023-06-10T18:59:24.257637+0000","last_clean":"2023-06-10T18:59:24.257637+0000","last_became_active":"2023-06-10T18:59:22.863293+0000","last_became_peered":"2023-06-10T18:59:22.863293+0000","last_unstale":"2023-06-10T18:59:24.257637+0000","last_undegraded":"2023-06-10T18:59:24.257637+0000","last_fullsized":"2023-06-10T18:59:24.257637+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:02:33.530604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989701+0000","last_change":"2023-06-10T18:59:23.272875+0000","last_active":"2023-06-10T18:59:24.989701+0000","last_peered":"2023-06-10T18:59:24.989701+0000","last_clean":"2023-06-10T18:59:24.989701+0000","last_became_active":"2023-06-10T18:59:23.272560+0000","last_became_peered":"2023-06-10T18:59:23.272560+0000","last_unstale":"2023-06-10T18:59:24.989701+0000","last_undegraded":"2023-06-10T18:59:24.989701+0000","last_fullsized":"2023-06-10T18:59:24.989701+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:02:29.265361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"186'6","reported_seq":19,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990038+0000","last_change":"2023-06-10T18:59:22.865609+0000","last_active":"2023-06-10T18:59:24.990038+0000","last_peered":"2023-06-10T18:59:24.990038+0000","last_clean":"2023-06-10T18:59:24.990038+0000","last_became_active":"2023-06-10T18:59:22.865302+0000","last_became_peered":"2023-06-10T18:59:22.865302+0000","last_unstale":"2023-06-10T18:59:24.990038+0000","last_undegraded":"2023-06-10T18:59:24.990038+0000","last_fullsized":"2023-06-10T18:59:24.990038+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:58:36.578746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258328+0000","last_change":"2023-06-10T18:59:23.275937+0000","last_active":"2023-06-10T18:59:24.258328+0000","last_peered":"2023-06-10T18:59:24.258328+0000","last_clean":"2023-06-10T18:59:24.258328+0000","last_became_active":"2023-06-10T18:59:23.275679+0000","last_became_peered":"2023-06-10T18:59:23.275679+0000","last_unstale":"2023-06-10T18:59:24.258328+0000","last_undegraded":"2023-06-10T18:59:24.258328+0000","last_fullsized":"2023-06-10T18:59:24.258328+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:01:50.059541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989716+0000","last_change":"2023-06-10T18:59:22.862656+0000","last_active":"2023-06-10T18:59:24.989716+0000","last_peered":"2023-06-10T18:59:24.989716+0000","last_clean":"2023-06-10T18:59:24.989716+0000","last_became_active":"2023-06-10T18:59:22.862371+0000","last_became_peered":"2023-06-10T18:59:22.862371+0000","last_unstale":"2023-06-10T18:59:24.989716+0000","last_undegraded":"2023-06-10T18:59:24.989716+0000","last_fullsized":"2023-06-10T18:59:24.989716+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:15:46.994439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257522+0000","last_change":"2023-06-10T18:59:23.269033+0000","last_active":"2023-06-10T18:59:24.257522+0000","last_peered":"2023-06-10T18:59:24.257522+0000","last_clean":"2023-06-10T18:59:24.257522+0000","last_became_active":"2023-06-10T18:59:23.268824+0000","last_became_peered":"2023-06-10T18:59:23.268824+0000","last_unstale":"2023-06-10T18:59:24.257522+0000","last_undegraded":"2023-06-10T18:59:24.257522+0000","last_fullsized":"2023-06-10T18:59:24.257522+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:22:30.742730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990499+0000","last_change":"2023-06-10T18:59:22.864987+0000","last_active":"2023-06-10T18:59:24.990499+0000","last_peered":"2023-06-10T18:59:24.990499+0000","last_clean":"2023-06-10T18:59:24.990499+0000","last_became_active":"2023-06-10T18:59:22.864220+0000","last_became_peered":"2023-06-10T18:59:22.864220+0000","last_unstale":"2023-06-10T18:59:24.990499+0000","last_undegraded":"2023-06-10T18:59:24.990499+0000","last_fullsized":"2023-06-10T18:59:24.990499+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:54:18.456116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.988939+0000","last_change":"2023-06-10T18:59:23.278812+0000","last_active":"2023-06-10T18:59:24.988939+0000","last_peered":"2023-06-10T18:59:24.988939+0000","last_clean":"2023-06-10T18:59:24.988939+0000","last_became_active":"2023-06-10T18:59:23.278535+0000","last_became_peered":"2023-06-10T18:59:23.278535+0000","last_unstale":"2023-06-10T18:59:24.988939+0000","last_undegraded":"2023-06-10T18:59:24.988939+0000","last_fullsized":"2023-06-10T18:59:24.988939+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:14:38.630542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257091+0000","last_change":"2023-06-10T18:59:22.863471+0000","last_active":"2023-06-10T18:59:24.257091+0000","last_peered":"2023-06-10T18:59:24.257091+0000","last_clean":"2023-06-10T18:59:24.257091+0000","last_became_active":"2023-06-10T18:59:22.863158+0000","last_became_peered":"2023-06-10T18:59:22.863158+0000","last_unstale":"2023-06-10T18:59:24.257091+0000","last_undegraded":"2023-06-10T18:59:24.257091+0000","last_fullsized":"2023-06-10T18:59:24.257091+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:56.348297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989531+0000","last_change":"2023-06-10T18:59:23.269077+0000","last_active":"2023-06-10T18:59:24.989531+0000","last_peered":"2023-06-10T18:59:24.989531+0000","last_clean":"2023-06-10T18:59:24.989531+0000","last_became_active":"2023-06-10T18:59:23.268830+0000","last_became_peered":"2023-06-10T18:59:23.268830+0000","last_unstale":"2023-06-10T18:59:24.989531+0000","last_undegraded":"2023-06-10T18:59:24.989531+0000","last_fullsized":"2023-06-10T18:59:24.989531+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:43:08.858710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"186'8","reported_seq":21,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989097+0000","last_change":"2023-06-10T18:59:22.871004+0000","last_active":"2023-06-10T18:59:24.989097+0000","last_peered":"2023-06-10T18:59:24.989097+0000","last_clean":"2023-06-10T18:59:24.989097+0000","last_became_active":"2023-06-10T18:59:22.870744+0000","last_became_peered":"2023-06-10T18:59:22.870744+0000","last_unstale":"2023-06-10T18:59:24.989097+0000","last_undegraded":"2023-06-10T18:59:24.989097+0000","last_fullsized":"2023-06-10T18:59:24.989097+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:51:23.379263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258448+0000","last_change":"2023-06-10T18:59:23.272443+0000","last_active":"2023-06-10T18:59:24.258448+0000","last_peered":"2023-06-10T18:59:24.258448+0000","last_clean":"2023-06-10T18:59:24.258448+0000","last_became_active":"2023-06-10T18:59:23.272176+0000","last_became_peered":"2023-06-10T18:59:23.272176+0000","last_unstale":"2023-06-10T18:59:24.258448+0000","last_undegraded":"2023-06-10T18:59:24.258448+0000","last_fullsized":"2023-06-10T18:59:24.258448+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:36:01.937935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"186'1","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258589+0000","last_change":"2023-06-10T18:59:22.866138+0000","last_active":"2023-06-10T18:59:24.258589+0000","last_peered":"2023-06-10T18:59:24.258589+0000","last_clean":"2023-06-10T18:59:24.258589+0000","last_became_active":"2023-06-10T18:59:22.865891+0000","last_became_peered":"2023-06-10T18:59:22.865891+0000","last_unstale":"2023-06-10T18:59:24.258589+0000","last_undegraded":"2023-06-10T18:59:24.258589+0000","last_fullsized":"2023-06-10T18:59:24.258589+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:59:07.454245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258296+0000","last_change":"2023-06-10T18:59:23.266047+0000","last_active":"2023-06-10T18:59:24.258296+0000","last_peered":"2023-06-10T18:59:24.258296+0000","last_clean":"2023-06-10T18:59:24.258296+0000","last_became_active":"2023-06-10T18:59:23.265502+0000","last_became_peered":"2023-06-10T18:59:23.265502+0000","last_unstale":"2023-06-10T18:59:24.258296+0000","last_undegraded":"2023-06-10T18:59:24.258296+0000","last_fullsized":"2023-06-10T18:59:24.258296+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:45:24.807904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258540+0000","last_change":"2023-06-10T18:59:22.871483+0000","last_active":"2023-06-10T18:59:24.258540+0000","last_peered":"2023-06-10T18:59:24.258540+0000","last_clean":"2023-06-10T18:59:24.258540+0000","last_became_active":"2023-06-10T18:59:22.871203+0000","last_became_peered":"2023-06-10T18:59:22.871203+0000","last_unstale":"2023-06-10T18:59:24.258540+0000","last_undegraded":"2023-06-10T18:59:24.258540+0000","last_fullsized":"2023-06-10T18:59:24.258540+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:17:45.595573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989376+0000","last_change":"2023-06-10T18:59:23.276132+0000","last_active":"2023-06-10T18:59:24.989376+0000","last_peered":"2023-06-10T18:59:24.989376+0000","last_clean":"2023-06-10T18:59:24.989376+0000","last_became_active":"2023-06-10T18:59:23.275931+0000","last_became_peered":"2023-06-10T18:59:23.275931+0000","last_unstale":"2023-06-10T18:59:24.989376+0000","last_undegraded":"2023-06-10T18:59:24.989376+0000","last_fullsized":"2023-06-10T18:59:24.989376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:35:41.574416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"186'1","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257704+0000","last_change":"2023-06-10T18:59:22.864186+0000","last_active":"2023-06-10T18:59:24.257704+0000","last_peered":"2023-06-10T18:59:24.257704+0000","last_clean":"2023-06-10T18:59:24.257704+0000","last_became_active":"2023-06-10T18:59:22.863650+0000","last_became_peered":"2023-06-10T18:59:22.863650+0000","last_unstale":"2023-06-10T18:59:24.257704+0000","last_undegraded":"2023-06-10T18:59:24.257704+0000","last_fullsized":"2023-06-10T18:59:24.257704+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:00:31.206603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257541+0000","last_change":"2023-06-10T18:59:23.266056+0000","last_active":"2023-06-10T18:59:24.257541+0000","last_peered":"2023-06-10T18:59:24.257541+0000","last_clean":"2023-06-10T18:59:24.257541+0000","last_became_active":"2023-06-10T18:59:23.265772+0000","last_became_peered":"2023-06-10T18:59:23.265772+0000","last_unstale":"2023-06-10T18:59:24.257541+0000","last_undegraded":"2023-06-10T18:59:24.257541+0000","last_fullsized":"2023-06-10T18:59:24.257541+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:51:48.672732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989236+0000","last_change":"2023-06-10T18:59:22.870808+0000","last_active":"2023-06-10T18:59:24.989236+0000","last_peered":"2023-06-10T18:59:24.989236+0000","last_clean":"2023-06-10T18:59:24.989236+0000","last_became_active":"2023-06-10T18:59:22.870511+0000","last_became_peered":"2023-06-10T18:59:22.870511+0000","last_unstale":"2023-06-10T18:59:24.989236+0000","last_undegraded":"2023-06-10T18:59:24.989236+0000","last_fullsized":"2023-06-10T18:59:24.989236+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:37:02.775119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257507+0000","last_change":"2023-06-10T18:59:23.267656+0000","last_active":"2023-06-10T18:59:24.257507+0000","last_peered":"2023-06-10T18:59:24.257507+0000","last_clean":"2023-06-10T18:59:24.257507+0000","last_became_active":"2023-06-10T18:59:23.267411+0000","last_became_peered":"2023-06-10T18:59:23.267411+0000","last_unstale":"2023-06-10T18:59:24.257507+0000","last_undegraded":"2023-06-10T18:59:24.257507+0000","last_fullsized":"2023-06-10T18:59:24.257507+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:11:40.861372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257568+0000","last_change":"2023-06-10T18:59:22.867461+0000","last_active":"2023-06-10T18:59:24.257568+0000","last_peered":"2023-06-10T18:59:24.257568+0000","last_clean":"2023-06-10T18:59:24.257568+0000","last_became_active":"2023-06-10T18:59:22.867172+0000","last_became_peered":"2023-06-10T18:59:22.867172+0000","last_unstale":"2023-06-10T18:59:24.257568+0000","last_undegraded":"2023-06-10T18:59:24.257568+0000","last_fullsized":"2023-06-10T18:59:24.257568+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:21:06.660532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989238+0000","last_change":"2023-06-10T18:59:23.273227+0000","last_active":"2023-06-10T18:59:24.989238+0000","last_peered":"2023-06-10T18:59:24.989238+0000","last_clean":"2023-06-10T18:59:24.989238+0000","last_became_active":"2023-06-10T18:59:23.272950+0000","last_became_peered":"2023-06-10T18:59:23.272950+0000","last_unstale":"2023-06-10T18:59:24.989238+0000","last_undegraded":"2023-06-10T18:59:24.989238+0000","last_fullsized":"2023-06-10T18:59:24.989238+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:57:52.145740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989897+0000","last_change":"2023-06-10T18:59:22.858901+0000","last_active":"2023-06-10T18:59:24.989897+0000","last_peered":"2023-06-10T18:59:24.989897+0000","last_clean":"2023-06-10T18:59:24.989897+0000","last_became_active":"2023-06-10T18:59:22.858661+0000","last_became_peered":"2023-06-10T18:59:22.858661+0000","last_unstale":"2023-06-10T18:59:24.989897+0000","last_undegraded":"2023-06-10T18:59:24.989897+0000","last_fullsized":"2023-06-10T18:59:24.989897+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:29:26.294520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257811+0000","last_change":"2023-06-10T18:59:23.268545+0000","last_active":"2023-06-10T18:59:24.257811+0000","last_peered":"2023-06-10T18:59:24.257811+0000","last_clean":"2023-06-10T18:59:24.257811+0000","last_became_active":"2023-06-10T18:59:23.268276+0000","last_became_peered":"2023-06-10T18:59:23.268276+0000","last_unstale":"2023-06-10T18:59:24.257811+0000","last_undegraded":"2023-06-10T18:59:24.257811+0000","last_fullsized":"2023-06-10T18:59:24.257811+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:50:49.279382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989677+0000","last_change":"2023-06-10T18:59:22.858445+0000","last_active":"2023-06-10T18:59:24.989677+0000","last_peered":"2023-06-10T18:59:24.989677+0000","last_clean":"2023-06-10T18:59:24.989677+0000","last_became_active":"2023-06-10T18:59:22.858215+0000","last_became_peered":"2023-06-10T18:59:22.858215+0000","last_unstale":"2023-06-10T18:59:24.989677+0000","last_undegraded":"2023-06-10T18:59:24.989677+0000","last_fullsized":"2023-06-10T18:59:24.989677+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:55:48.198954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989532+0000","last_change":"2023-06-10T18:59:23.278174+0000","last_active":"2023-06-10T18:59:24.989532+0000","last_peered":"2023-06-10T18:59:24.989532+0000","last_clean":"2023-06-10T18:59:24.989532+0000","last_became_active":"2023-06-10T18:59:23.277901+0000","last_became_peered":"2023-06-10T18:59:23.277901+0000","last_unstale":"2023-06-10T18:59:24.989532+0000","last_undegraded":"2023-06-10T18:59:24.989532+0000","last_fullsized":"2023-06-10T18:59:24.989532+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:37:29.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258143+0000","last_change":"2023-06-10T18:59:22.869065+0000","last_active":"2023-06-10T18:59:24.258143+0000","last_peered":"2023-06-10T18:59:24.258143+0000","last_clean":"2023-06-10T18:59:24.258143+0000","last_became_active":"2023-06-10T18:59:22.868792+0000","last_became_peered":"2023-06-10T18:59:22.868792+0000","last_unstale":"2023-06-10T18:59:24.258143+0000","last_undegraded":"2023-06-10T18:59:24.258143+0000","last_fullsized":"2023-06-10T18:59:24.258143+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:32:48.284545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990242+0000","last_change":"2023-06-10T18:59:23.277952+0000","last_active":"2023-06-10T18:59:24.990242+0000","last_peered":"2023-06-10T18:59:24.990242+0000","last_clean":"2023-06-10T18:59:24.990242+0000","last_became_active":"2023-06-10T18:59:23.277678+0000","last_became_peered":"2023-06-10T18:59:23.277678+0000","last_unstale":"2023-06-10T18:59:24.990242+0000","last_undegraded":"2023-06-10T18:59:24.990242+0000","last_fullsized":"2023-06-10T18:59:24.990242+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:53:38.063647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989507+0000","last_change":"2023-06-10T18:59:22.865360+0000","last_active":"2023-06-10T18:59:24.989507+0000","last_peered":"2023-06-10T18:59:24.989507+0000","last_clean":"2023-06-10T18:59:24.989507+0000","last_became_active":"2023-06-10T18:59:22.865075+0000","last_became_peered":"2023-06-10T18:59:22.865075+0000","last_unstale":"2023-06-10T18:59:24.989507+0000","last_undegraded":"2023-06-10T18:59:24.989507+0000","last_fullsized":"2023-06-10T18:59:24.989507+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:52:31.489428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989769+0000","last_change":"2023-06-10T18:59:23.278394+0000","last_active":"2023-06-10T18:59:24.989769+0000","last_peered":"2023-06-10T18:59:24.989769+0000","last_clean":"2023-06-10T18:59:24.989769+0000","last_became_active":"2023-06-10T18:59:23.278108+0000","last_became_peered":"2023-06-10T18:59:23.278108+0000","last_unstale":"2023-06-10T18:59:24.989769+0000","last_undegraded":"2023-06-10T18:59:24.989769+0000","last_fullsized":"2023-06-10T18:59:24.989769+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:23:47.782700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990238+0000","last_change":"2023-06-10T18:59:22.866629+0000","last_active":"2023-06-10T18:59:24.990238+0000","last_peered":"2023-06-10T18:59:24.990238+0000","last_clean":"2023-06-10T18:59:24.990238+0000","last_became_active":"2023-06-10T18:59:22.866369+0000","last_became_peered":"2023-06-10T18:59:22.866369+0000","last_unstale":"2023-06-10T18:59:24.990238+0000","last_undegraded":"2023-06-10T18:59:24.990238+0000","last_fullsized":"2023-06-10T18:59:24.990238+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:05:13.776464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258070+0000","last_change":"2023-06-10T18:59:23.275515+0000","last_active":"2023-06-10T18:59:24.258070+0000","last_peered":"2023-06-10T18:59:24.258070+0000","last_clean":"2023-06-10T18:59:24.258070+0000","last_became_active":"2023-06-10T18:59:23.275189+0000","last_became_peered":"2023-06-10T18:59:23.275189+0000","last_unstale":"2023-06-10T18:59:24.258070+0000","last_undegraded":"2023-06-10T18:59:24.258070+0000","last_fullsized":"2023-06-10T18:59:24.258070+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:46:30.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258499+0000","last_change":"2023-06-10T18:59:22.871346+0000","last_active":"2023-06-10T18:59:24.258499+0000","last_peered":"2023-06-10T18:59:24.258499+0000","last_clean":"2023-06-10T18:59:24.258499+0000","last_became_active":"2023-06-10T18:59:22.871065+0000","last_became_peered":"2023-06-10T18:59:22.871065+0000","last_unstale":"2023-06-10T18:59:24.258499+0000","last_undegraded":"2023-06-10T18:59:24.258499+0000","last_fullsized":"2023-06-10T18:59:24.258499+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:10:53.074379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989889+0000","last_change":"2023-06-10T18:59:23.271619+0000","last_active":"2023-06-10T18:59:24.989889+0000","last_peered":"2023-06-10T18:59:24.989889+0000","last_clean":"2023-06-10T18:59:24.989889+0000","last_became_active":"2023-06-10T18:59:23.271355+0000","last_became_peered":"2023-06-10T18:59:23.271355+0000","last_unstale":"2023-06-10T18:59:24.989889+0000","last_undegraded":"2023-06-10T18:59:24.989889+0000","last_fullsized":"2023-06-10T18:59:24.989889+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:02:17.919579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989367+0000","last_change":"2023-06-10T18:59:22.871144+0000","last_active":"2023-06-10T18:59:24.989367+0000","last_peered":"2023-06-10T18:59:24.989367+0000","last_clean":"2023-06-10T18:59:24.989367+0000","last_became_active":"2023-06-10T18:59:22.870840+0000","last_became_peered":"2023-06-10T18:59:22.870840+0000","last_unstale":"2023-06-10T18:59:24.989367+0000","last_undegraded":"2023-06-10T18:59:24.989367+0000","last_fullsized":"2023-06-10T18:59:24.989367+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:57:16.493724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989782+0000","last_change":"2023-06-10T18:59:23.267937+0000","last_active":"2023-06-10T18:59:24.989782+0000","last_peered":"2023-06-10T18:59:24.989782+0000","last_clean":"2023-06-10T18:59:24.989782+0000","last_became_active":"2023-06-10T18:59:23.267480+0000","last_became_peered":"2023-06-10T18:59:23.267480+0000","last_unstale":"2023-06-10T18:59:24.989782+0000","last_undegraded":"2023-06-10T18:59:24.989782+0000","last_fullsized":"2023-06-10T18:59:24.989782+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:19:06.854306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989902+0000","last_change":"2023-06-10T18:59:22.865914+0000","last_active":"2023-06-10T18:59:24.989902+0000","last_peered":"2023-06-10T18:59:24.989902+0000","last_clean":"2023-06-10T18:59:24.989902+0000","last_became_active":"2023-06-10T18:59:22.865638+0000","last_became_peered":"2023-06-10T18:59:22.865638+0000","last_unstale":"2023-06-10T18:59:24.989902+0000","last_undegraded":"2023-06-10T18:59:24.989902+0000","last_fullsized":"2023-06-10T18:59:24.989902+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:52:03.149189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989848+0000","last_change":"2023-06-10T18:59:23.279354+0000","last_active":"2023-06-10T18:59:24.989848+0000","last_peered":"2023-06-10T18:59:24.989848+0000","last_clean":"2023-06-10T18:59:24.989848+0000","last_became_active":"2023-06-10T18:59:23.277348+0000","last_became_peered":"2023-06-10T18:59:23.277348+0000","last_unstale":"2023-06-10T18:59:24.989848+0000","last_undegraded":"2023-06-10T18:59:24.989848+0000","last_fullsized":"2023-06-10T18:59:24.989848+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:32:08.231038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258520+0000","last_change":"2023-06-10T18:59:22.865554+0000","last_active":"2023-06-10T18:59:24.258520+0000","last_peered":"2023-06-10T18:59:24.258520+0000","last_clean":"2023-06-10T18:59:24.258520+0000","last_became_active":"2023-06-10T18:59:22.865275+0000","last_became_peered":"2023-06-10T18:59:22.865275+0000","last_unstale":"2023-06-10T18:59:24.258520+0000","last_undegraded":"2023-06-10T18:59:24.258520+0000","last_fullsized":"2023-06-10T18:59:24.258520+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:19:56.996692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.988995+0000","last_change":"2023-06-10T18:59:23.278447+0000","last_active":"2023-06-10T18:59:24.988995+0000","last_peered":"2023-06-10T18:59:24.988995+0000","last_clean":"2023-06-10T18:59:24.988995+0000","last_became_active":"2023-06-10T18:59:23.278184+0000","last_became_peered":"2023-06-10T18:59:23.278184+0000","last_unstale":"2023-06-10T18:59:24.988995+0000","last_undegraded":"2023-06-10T18:59:24.988995+0000","last_fullsized":"2023-06-10T18:59:24.988995+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:24:11.618586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989664+0000","last_change":"2023-06-10T18:59:22.866707+0000","last_active":"2023-06-10T18:59:24.989664+0000","last_peered":"2023-06-10T18:59:24.989664+0000","last_clean":"2023-06-10T18:59:24.989664+0000","last_became_active":"2023-06-10T18:59:22.866430+0000","last_became_peered":"2023-06-10T18:59:22.866430+0000","last_unstale":"2023-06-10T18:59:24.989664+0000","last_undegraded":"2023-06-10T18:59:24.989664+0000","last_fullsized":"2023-06-10T18:59:24.989664+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:48:25.858104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257903+0000","last_change":"2023-06-10T18:59:23.279554+0000","last_active":"2023-06-10T18:59:24.257903+0000","last_peered":"2023-06-10T18:59:24.257903+0000","last_clean":"2023-06-10T18:59:24.257903+0000","last_became_active":"2023-06-10T18:59:23.279266+0000","last_became_peered":"2023-06-10T18:59:23.279266+0000","last_unstale":"2023-06-10T18:59:24.257903+0000","last_undegraded":"2023-06-10T18:59:24.257903+0000","last_fullsized":"2023-06-10T18:59:24.257903+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:07:07.477183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989566+0000","last_change":"2023-06-10T18:59:22.865261+0000","last_active":"2023-06-10T18:59:24.989566+0000","last_peered":"2023-06-10T18:59:24.989566+0000","last_clean":"2023-06-10T18:59:24.989566+0000","last_became_active":"2023-06-10T18:59:22.865017+0000","last_became_peered":"2023-06-10T18:59:22.865017+0000","last_unstale":"2023-06-10T18:59:24.989566+0000","last_undegraded":"2023-06-10T18:59:24.989566+0000","last_fullsized":"2023-06-10T18:59:24.989566+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:33:23.768019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257881+0000","last_change":"2023-06-10T18:59:23.274010+0000","last_active":"2023-06-10T18:59:24.257881+0000","last_peered":"2023-06-10T18:59:24.257881+0000","last_clean":"2023-06-10T18:59:24.257881+0000","last_became_active":"2023-06-10T18:59:23.273748+0000","last_became_peered":"2023-06-10T18:59:23.273748+0000","last_unstale":"2023-06-10T18:59:24.257881+0000","last_undegraded":"2023-06-10T18:59:24.257881+0000","last_fullsized":"2023-06-10T18:59:24.257881+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:37:59.029043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989908+0000","last_change":"2023-06-10T18:59:22.865197+0000","last_active":"2023-06-10T18:59:24.989908+0000","last_peered":"2023-06-10T18:59:24.989908+0000","last_clean":"2023-06-10T18:59:24.989908+0000","last_became_active":"2023-06-10T18:59:22.864938+0000","last_became_peered":"2023-06-10T18:59:22.864938+0000","last_unstale":"2023-06-10T18:59:24.989908+0000","last_undegraded":"2023-06-10T18:59:24.989908+0000","last_fullsized":"2023-06-10T18:59:24.989908+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:39:24.704298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258596+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T18:59:24.258596+0000","last_peered":"2023-06-10T18:59:24.258596+0000","last_clean":"2023-06-10T18:59:24.258596+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T18:59:24.258596+0000","last_undegraded":"2023-06-10T18:59:24.258596+0000","last_fullsized":"2023-06-10T18:59:24.258596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:06:59.401103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989754+0000","last_change":"2023-06-10T18:59:23.268796+0000","last_active":"2023-06-10T18:59:24.989754+0000","last_peered":"2023-06-10T18:59:24.989754+0000","last_clean":"2023-06-10T18:59:24.989754+0000","last_became_active":"2023-06-10T18:59:23.268529+0000","last_became_peered":"2023-06-10T18:59:23.268529+0000","last_unstale":"2023-06-10T18:59:24.989754+0000","last_undegraded":"2023-06-10T18:59:24.989754+0000","last_fullsized":"2023-06-10T18:59:24.989754+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:11:57.606512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"186'1","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258688+0000","last_change":"2023-06-10T18:59:22.863561+0000","last_active":"2023-06-10T18:59:24.258688+0000","last_peered":"2023-06-10T18:59:24.258688+0000","last_clean":"2023-06-10T18:59:24.258688+0000","last_became_active":"2023-06-10T18:59:22.863356+0000","last_became_peered":"2023-06-10T18:59:22.863356+0000","last_unstale":"2023-06-10T18:59:24.258688+0000","last_undegraded":"2023-06-10T18:59:24.258688+0000","last_fullsized":"2023-06-10T18:59:24.258688+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:38:16.403311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":344,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258287+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T18:59:24.258287+0000","last_peered":"2023-06-10T18:59:24.258287+0000","last_clean":"2023-06-10T18:59:24.258287+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T18:59:24.258287+0000","last_undegraded":"2023-06-10T18:59:24.258287+0000","last_fullsized":"2023-06-10T18:59:24.258287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:32:12.690760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257243+0000","last_change":"2023-06-10T18:59:23.272622+0000","last_active":"2023-06-10T18:59:24.257243+0000","last_peered":"2023-06-10T18:59:24.257243+0000","last_clean":"2023-06-10T18:59:24.257243+0000","last_became_active":"2023-06-10T18:59:23.272332+0000","last_became_peered":"2023-06-10T18:59:23.272332+0000","last_unstale":"2023-06-10T18:59:24.257243+0000","last_undegraded":"2023-06-10T18:59:24.257243+0000","last_fullsized":"2023-06-10T18:59:24.257243+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:13:02.835730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258112+0000","last_change":"2023-06-10T18:59:22.862873+0000","last_active":"2023-06-10T18:59:24.258112+0000","last_peered":"2023-06-10T18:59:24.258112+0000","last_clean":"2023-06-10T18:59:24.258112+0000","last_became_active":"2023-06-10T18:59:22.862622+0000","last_became_peered":"2023-06-10T18:59:22.862622+0000","last_unstale":"2023-06-10T18:59:24.258112+0000","last_undegraded":"2023-06-10T18:59:24.258112+0000","last_fullsized":"2023-06-10T18:59:24.258112+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:05:12.432288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989051+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T18:59:24.989051+0000","last_peered":"2023-06-10T18:59:24.989051+0000","last_clean":"2023-06-10T18:59:24.989051+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T18:59:24.989051+0000","last_undegraded":"2023-06-10T18:59:24.989051+0000","last_fullsized":"2023-06-10T18:59:24.989051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:55:35.943360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989854+0000","last_change":"2023-06-10T18:59:23.274375+0000","last_active":"2023-06-10T18:59:24.989854+0000","last_peered":"2023-06-10T18:59:24.989854+0000","last_clean":"2023-06-10T18:59:24.989854+0000","last_became_active":"2023-06-10T18:59:23.274087+0000","last_became_peered":"2023-06-10T18:59:23.274087+0000","last_unstale":"2023-06-10T18:59:24.989854+0000","last_undegraded":"2023-06-10T18:59:24.989854+0000","last_fullsized":"2023-06-10T18:59:24.989854+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:58:06.301405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"186'1","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989987+0000","last_change":"2023-06-10T18:59:22.865658+0000","last_active":"2023-06-10T18:59:24.989987+0000","last_peered":"2023-06-10T18:59:24.989987+0000","last_clean":"2023-06-10T18:59:24.989987+0000","last_became_active":"2023-06-10T18:59:22.865188+0000","last_became_peered":"2023-06-10T18:59:22.865188+0000","last_unstale":"2023-06-10T18:59:24.989987+0000","last_undegraded":"2023-06-10T18:59:24.989987+0000","last_fullsized":"2023-06-10T18:59:24.989987+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:41:34.355324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":344,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.257841+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T18:59:24.257841+0000","last_peered":"2023-06-10T18:59:24.257841+0000","last_clean":"2023-06-10T18:59:24.257841+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T18:59:24.257841+0000","last_undegraded":"2023-06-10T18:59:24.257841+0000","last_fullsized":"2023-06-10T18:59:24.257841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:41:57.499663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":470,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990601+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T18:59:24.990601+0000","last_peered":"2023-06-10T18:59:24.990601+0000","last_clean":"2023-06-10T18:59:24.990601+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T18:59:24.990601+0000","last_undegraded":"2023-06-10T18:59:24.990601+0000","last_fullsized":"2023-06-10T18:59:24.990601+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:24:11.662759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989442+0000","last_change":"2023-06-10T18:59:23.268199+0000","last_active":"2023-06-10T18:59:24.989442+0000","last_peered":"2023-06-10T18:59:24.989442+0000","last_clean":"2023-06-10T18:59:24.989442+0000","last_became_active":"2023-06-10T18:59:23.267945+0000","last_became_peered":"2023-06-10T18:59:23.267945+0000","last_unstale":"2023-06-10T18:59:24.989442+0000","last_undegraded":"2023-06-10T18:59:24.989442+0000","last_fullsized":"2023-06-10T18:59:24.989442+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:10:48.460295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989205+0000","last_change":"2023-06-10T18:59:22.869023+0000","last_active":"2023-06-10T18:59:24.989205+0000","last_peered":"2023-06-10T18:59:24.989205+0000","last_clean":"2023-06-10T18:59:24.989205+0000","last_became_active":"2023-06-10T18:59:22.868781+0000","last_became_peered":"2023-06-10T18:59:22.868781+0000","last_unstale":"2023-06-10T18:59:24.989205+0000","last_undegraded":"2023-06-10T18:59:24.989205+0000","last_fullsized":"2023-06-10T18:59:24.989205+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:02:33.530604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":314,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989635+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T18:59:24.989635+0000","last_peered":"2023-06-10T18:59:24.989635+0000","last_clean":"2023-06-10T18:59:24.989635+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T18:59:24.989635+0000","last_undegraded":"2023-06-10T18:59:24.989635+0000","last_fullsized":"2023-06-10T18:59:24.989635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:06:55.288069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989698+0000","last_change":"2023-06-10T18:59:23.268440+0000","last_active":"2023-06-10T18:59:24.989698+0000","last_peered":"2023-06-10T18:59:24.989698+0000","last_clean":"2023-06-10T18:59:24.989698+0000","last_became_active":"2023-06-10T18:59:23.268182+0000","last_became_peered":"2023-06-10T18:59:23.268182+0000","last_unstale":"2023-06-10T18:59:24.989698+0000","last_undegraded":"2023-06-10T18:59:24.989698+0000","last_fullsized":"2023-06-10T18:59:24.989698+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:24:10.236556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"186'2","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989738+0000","last_change":"2023-06-10T18:59:22.871095+0000","last_active":"2023-06-10T18:59:24.989738+0000","last_peered":"2023-06-10T18:59:24.989738+0000","last_clean":"2023-06-10T18:59:24.989738+0000","last_became_active":"2023-06-10T18:59:22.870822+0000","last_became_peered":"2023-06-10T18:59:22.870822+0000","last_unstale":"2023-06-10T18:59:24.989738+0000","last_undegraded":"2023-06-10T18:59:24.989738+0000","last_fullsized":"2023-06-10T18:59:24.989738+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:48:13.144096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":314,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990091+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T18:59:24.990091+0000","last_peered":"2023-06-10T18:59:24.990091+0000","last_clean":"2023-06-10T18:59:24.990091+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T18:59:24.990091+0000","last_undegraded":"2023-06-10T18:59:24.990091+0000","last_fullsized":"2023-06-10T18:59:24.990091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:13:26.493211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.989895+0000","last_change":"2023-06-10T18:59:23.278597+0000","last_active":"2023-06-10T18:59:24.989895+0000","last_peered":"2023-06-10T18:59:24.989895+0000","last_clean":"2023-06-10T18:59:24.989895+0000","last_became_active":"2023-06-10T18:59:23.278305+0000","last_became_peered":"2023-06-10T18:59:23.278305+0000","last_unstale":"2023-06-10T18:59:24.989895+0000","last_undegraded":"2023-06-10T18:59:24.989895+0000","last_fullsized":"2023-06-10T18:59:24.989895+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:13:55.382736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"186'1","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258494+0000","last_change":"2023-06-10T18:59:22.860594+0000","last_active":"2023-06-10T18:59:24.258494+0000","last_peered":"2023-06-10T18:59:24.258494+0000","last_clean":"2023-06-10T18:59:24.258494+0000","last_became_active":"2023-06-10T18:59:22.860379+0000","last_became_peered":"2023-06-10T18:59:22.860379+0000","last_unstale":"2023-06-10T18:59:24.258494+0000","last_undegraded":"2023-06-10T18:59:24.258494+0000","last_fullsized":"2023-06-10T18:59:24.258494+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:09:41.394551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258364+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T18:59:24.258364+0000","last_peered":"2023-06-10T18:59:24.258364+0000","last_clean":"2023-06-10T18:59:24.258364+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T18:59:24.258364+0000","last_undegraded":"2023-06-10T18:59:24.258364+0000","last_fullsized":"2023-06-10T18:59:24.258364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:49:30.271988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.258339+0000","last_change":"2023-06-10T18:59:23.275014+0000","last_active":"2023-06-10T18:59:24.258339+0000","last_peered":"2023-06-10T18:59:24.258339+0000","last_clean":"2023-06-10T18:59:24.258339+0000","last_became_active":"2023-06-10T18:59:23.274677+0000","last_became_peered":"2023-06-10T18:59:23.274677+0000","last_unstale":"2023-06-10T18:59:24.258339+0000","last_undegraded":"2023-06-10T18:59:24.258339+0000","last_fullsized":"2023-06-10T18:59:24.258339+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:22.842705+0000","last_clean_scrub_stamp":"2023-06-10T18:59:22.842705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:07:06.261343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"186'4","reported_seq":17,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990589+0000","last_change":"2023-06-10T18:59:22.864241+0000","last_active":"2023-06-10T18:59:24.990589+0000","last_peered":"2023-06-10T18:59:24.990589+0000","last_clean":"2023-06-10T18:59:24.990589+0000","last_became_active":"2023-06-10T18:59:22.863991+0000","last_became_peered":"2023-06-10T18:59:22.863991+0000","last_unstale":"2023-06-10T18:59:24.990589+0000","last_undegraded":"2023-06-10T18:59:24.990589+0000","last_fullsized":"2023-06-10T18:59:24.990589+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:59:21.838366+0000","last_clean_scrub_stamp":"2023-06-10T18:59:21.838366+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:59:14.183026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-06-10T18:59:24.990518+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T18:59:24.990518+0000","last_peered":"2023-06-10T18:59:24.990518+0000","last_clean":"2023-06-10T18:59:24.990518+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T18:59:24.990518+0000","last_undegraded":"2023-06-10T18:59:24.990518+0000","last_fullsized":"2023-06-10T18:59:24.990518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:15:29.661938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":2957312,"data_stored":1750329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.70199999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.67600000000000005}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.64400000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.51400000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575039,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2351104,"data_stored":1157960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.434},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.50600000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.442}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.495}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2355200,"data_stored":1157434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.32600000000000001},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.753},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.55600000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34200000000000003},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.52400000000000002,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.313},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.71899999999999997},"last":0.377}]},{"osd":3,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81200000000000006,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.622},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":0.85999999999999999,"15min":1.008},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":1.206,"15min":1.206},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.79500000000000004,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.224,"15min":1.224},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.97099999999999997},"last":0.58999999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575038,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33924,"kb_used_data":2876,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337916,"statfs":{"total":96636764160,"available":96602025984,"internally_reserved":0,"allocated":2945024,"data_stored":1747836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.56499999999999995,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34799999999999998,"15min":0.34899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":0.55700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":11.439,"15min":11.439},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.376},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.65000000000000002,"5min":5.766,"15min":5.766},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.25900000000000001},"max":{"1min":0.60499999999999998,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":2.802,"15min":2.802},"last":0.79000000000000004}]},{"osd":5,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":0.999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.91500000000000004,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.58799999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":2.907,"15min":2.907},"last":0.75},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.40699999999999997,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.363},"last":0.86199999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45800000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.77200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2351104,"data_stored":1158397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.35599999999999998,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55800000000000005,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.376}]},{"osd":1,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.847},"last":0.312},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.40200000000000002},"min":{"1min":0.249,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.622,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.69599999999999995,"5min":1.24,"15min":2.8999999999999999},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.208,"15min":5.742},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.46200000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.623,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":6.593},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":7.0880000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58199999999999996,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.83999999999999997,"5min":1.026,"15min":8.7850000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.171,"15min":5.6180000000000003},"last":0.59899999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64700000000000002,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.87,"15min":6.617},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.38,"15min":0.37},"max":{"1min":0.879,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.5}]}]},{"osd":3,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33356,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338484,"statfs":{"total":96636764160,"available":96602607616,"internally_reserved":0,"allocated":2363392,"data_stored":1158579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.317}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.499}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.39700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33360,"kb_used_data":2312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338480,"statfs":{"total":96636764160,"available":96602603520,"internally_reserved":0,"allocated":2367488,"data_stored":1159046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":0.71999999999999997,"5min":0.745,"15min":1.119},"last":0.44},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.1639999999999999},"last":0.497}]},{"osd":1,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.77800000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.36399999999999999,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.873,"15min":0.96599999999999997},"last":0.48499999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":0.67100000000000004,"5min":0.72799999999999998,"15min":0.80800000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.33200000000000002},"max":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":1.0229999999999999},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51200000000000001,"15min":0.52600000000000002},"min":{"1min":0.44500000000000001,"5min":0.34899999999999998,"15min":0.33400000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.79800000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":1.042},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.67800000000000005,"5min":0.94599999999999995,"15min":1.069},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.501,"15min":0.504},"min":{"1min":0.36299999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.88300000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.78100000000000003,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":1.3169999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:58:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.90100000000000002,"15min":1.1459999999999999},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834575039,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":2300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":2355200,"data_stored":1158979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.379,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.749,"15min":0.78400000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.69399999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61599999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.85099999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65100000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.42999999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.628,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.34699999999999998},"max":{"1min":0.755,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.71799999999999997,"5min":0.747,"15min":0.83999999999999997},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53100000000000003,"15min":0.51800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69799999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.72699999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.879},"last":0.38100000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.55100000000000005},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T18:59:27.765 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-10T18:59:27.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-06-10T18:59:27.933 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.928+0000 7f852d62c700 1 Processor -- start 2023-06-10T18:59:27.934 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.928+0000 7f852d62c700 1 -- start start 2023-06-10T18:59:27.935 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.928+0000 7f852d62c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f85280d04f0 0x7f85280d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f852d62c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f852d62c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85280d0e00 con 0x7f852806b510 2023-06-10T18:59:27.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f852d62c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85280af530 con 0x7f85280d04f0 2023-06-10T18:59:27.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f852d62c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85280af810 con 0x7f85280d25c0 2023-06-10T18:59:27.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f8526ffd700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f85280d04f0 0x7f85280d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:32796/0 (socket says 172.21.15.104:32796) 2023-06-10T18:59:27.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 -- 172.21.15.104:0/3368465381 learned_addr learned my addr 172.21.15.104:0/3368465381 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.937 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 -- 172.21.15.104:0/3368465381 >> v2:172.21.15.114:3300/0 conn(0x7f85280d04f0 msgr2=0x7f85280d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:27.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 --2- 172.21.15.104:0/3368465381 >> v2:172.21.15.114:3300/0 conn(0x7f85280d04f0 0x7f85280d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 -- 172.21.15.104:0/3368465381 >> v1:172.21.15.114:6789/0 conn(0x7f852806b510 legacy=0x7f852806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.929+0000 7f85267fc700 1 -- 172.21.15.104:0/3368465381 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280afaf0 con 0x7f85280d25c0 2023-06-10T18:59:27.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f85267fc700 1 --2- 172.21.15.104:0/3368465381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f851c0098f0 tx=0x7f851c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=954e29ba27f5eac server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f8525ffb700 1 -- 172.21.15.104:0/3368465381 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011830 con 0x7f85280d25c0 2023-06-10T18:59:27.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f8525ffb700 1 -- 172.21.15.104:0/3368465381 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c011990 con 0x7f85280d25c0 2023-06-10T18:59:27.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f8525ffb700 1 -- 172.21.15.104:0/3368465381 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011d20 con 0x7f85280d25c0 2023-06-10T18:59:27.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 -- 172.21.15.104:0/3368465381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 msgr2=0x7f85280d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 --2- 172.21.15.104:0/3368465381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f851c0098f0 tx=0x7f851c009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:27.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 -- 172.21.15.104:0/3368465381 shutdown_connections 2023-06-10T18:59:27.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 --2- 172.21.15.104:0/3368465381 >> v2:172.21.15.114:3300/0 conn(0x7f85280d04f0 0x7f85280d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 --2- 172.21.15.104:0/3368465381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280d25c0 0x7f85280d08c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 -- 172.21.15.104:0/3368465381 >> 172.21.15.104:0/3368465381 conn(0x7f8528071ec0 msgr2=0x7f852809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 -- 172.21.15.104:0/3368465381 shutdown_connections 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.930+0000 7f852d62c700 1 -- 172.21.15.104:0/3368465381 wait complete. 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 Processor -- start 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 -- start start 2023-06-10T18:59:27.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f852806b510 0x7f8528157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280d04f0 0x7f852809f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852809d1c0 con 0x7f85280d25c0 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85280cf1c0 con 0x7f85280d04f0 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.932+0000 7f852d62c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852809cee0 con 0x7f852806b510 2023-06-10T18:59:27.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41652/0 (socket says 172.21.15.104:41652) 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 -- 172.21.15.104:0/2007566176 learned_addr learned my addr 172.21.15.104:0/2007566176 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85267fc700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280d04f0 0x7f852809f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f8526ffd700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f852806b510 0x7f8528157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 -- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f852806b510 msgr2=0x7f8528157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f852806b510 0x7f8528157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 -- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280d04f0 msgr2=0x7f852809f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:27.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280d04f0 0x7f852809f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 -- 172.21.15.104:0/2007566176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c009550 con 0x7f85280d25c0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f85277fe700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f852000cb50 tx=0x7f852000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520003c00 con 0x7f85280d25c0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8520003d60 con 0x7f85280d25c0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85280ceb00 con 0x7f85280d25c0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852000aa20 con 0x7f85280d25c0 2023-06-10T18:59:27.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.933+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8528159eb0 con 0x7f85280d25c0 2023-06-10T18:59:27.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.934+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85280ce960 con 0x7f85280d25c0 2023-06-10T18:59:27.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.935+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f852000ab80 con 0x7f85280d25c0 2023-06-10T18:59:27.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.935+0000 7f850f7fe700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 0x7f851406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:27.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.935+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8520089580 con 0x7f85280d25c0 2023-06-10T18:59:27.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.935+0000 7f8526ffd700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 0x7f851406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:27.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.936+0000 7f8526ffd700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 0x7f851406d190 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f8518009bc0 tx=0x7f85180091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:27.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:27.939+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f85280ce960 con 0x7f85280d25c0 2023-06-10T18:59:28.208 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.203+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f85280ce960 con 0x7f85280d25c0 2023-06-10T18:59:28.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.206+0000 7f850f7fe700 1 -- 172.21.15.104:0/2007566176 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+0 (secure 0 0 0) 0x7f85280ce960 con 0x7f85280d25c0 2023-06-10T18:59:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 msgr2=0x7f851406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 0x7f851406d190 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f8518009bc0 tx=0x7f85180091c0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 msgr2=0x7f852809f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f852000cb50 tx=0x7f852000ce60 comp rx=0 tx=0).stop 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 shutdown_connections 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85280d25c0 0x7f852809f5c0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f852806b510 0x7f8528157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f851406ace0 0x7f851406d190 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 --2- 172.21.15.104:0/2007566176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280d04f0 0x7f852809f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 >> 172.21.15.104:0/2007566176 conn(0x7f8528071ec0 msgr2=0x7f85280b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:28.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 shutdown_connections 2023-06-10T18:59:28.216 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.208+0000 7f852d62c700 1 -- 172.21.15.104:0/2007566176 wait complete. 2023-06-10T18:59:28.216 INFO:teuthology.orchestra.run.smithi104.stderr:updated caps for client.0 2023-06-10T18:59:28.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 Processor -- start 2023-06-10T18:59:28.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 -- start start 2023-06-10T18:59:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ab40ce390 0x7f8ab4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab406c720 0x7f8ab40ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab40d0340 con 0x7f8ab406b410 2023-06-10T18:59:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab40d1660 con 0x7f8ab40ce390 2023-06-10T18:59:28.351 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.342+0000 7f8ab9575700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab415cb60 con 0x7f8ab406c720 2023-06-10T18:59:28.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ab40ce390 0x7f8ab4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ab40ce390 0x7f8ab4160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41656/0 (socket says 172.21.15.104:41656) 2023-06-10T18:59:28.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/248730529 learned_addr learned my addr 172.21.15.104:0/248730529 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:28.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab27fc700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab406c720 0x7f8ab40ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/248730529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab406c720 msgr2=0x7f8ab40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:28.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab406c720 0x7f8ab40ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/248730529 >> v1:172.21.15.114:6789/0 conn(0x7f8ab406b410 legacy=0x7f8ab406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/248730529 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab4161230 con 0x7f8ab40ce390 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab2ffd700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40ce390 0x7f8ab4160cf0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40098f0 tx=0x7f8aa4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e539158e788be3f4 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab1ffb700 1 -- 172.21.15.104:0/248730529 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400b8a0 con 0x7f8ab40ce390 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab1ffb700 1 -- 172.21.15.104:0/248730529 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400ba00 con 0x7f8ab40ce390 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.343+0000 7f8ab1ffb700 1 -- 172.21.15.104:0/248730529 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400bd90 con 0x7f8ab40ce390 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 -- 172.21.15.104:0/248730529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40ce390 msgr2=0x7f8ab4160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40ce390 0x7f8ab4160cf0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40098f0 tx=0x7f8aa4009c00 comp rx=0 tx=0).stop 2023-06-10T18:59:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 -- 172.21.15.104:0/248730529 shutdown_connections 2023-06-10T18:59:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab40ce390 0x7f8ab4160cf0 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 --2- 172.21.15.104:0/248730529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab406c720 0x7f8ab40ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 -- 172.21.15.104:0/248730529 >> 172.21.15.104:0/248730529 conn(0x7f8ab4069e70 msgr2=0x7f8ab406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 -- 172.21.15.104:0/248730529 shutdown_connections 2023-06-10T18:59:28.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.344+0000 7f8ab9575700 1 -- 172.21.15.104:0/248730529 wait complete. 2023-06-10T18:59:28.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.345+0000 7f8ab9575700 1 Processor -- start 2023-06-10T18:59:28.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.345+0000 7f8ab9575700 1 -- start start 2023-06-10T18:59:28.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 0x7f8ab40c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40ce390 0x7f8ab40b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b44c0 con 0x7f8ab406b410 2023-06-10T18:59:28.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab41560e0 con 0x7f8ab406c720 2023-06-10T18:59:28.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab9575700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4151f80 con 0x7f8ab40ce390 2023-06-10T18:59:28.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41662/0 (socket says 172.21.15.104:41662) 2023-06-10T18:59:28.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/2085537599 learned_addr learned my addr 172.21.15.104:0/2085537599 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:28.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab27fc700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 0x7f8ab40c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab37fe700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40ce390 0x7f8ab40b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40ce390 msgr2=0x7f8ab40b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40ce390 0x7f8ab40b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 msgr2=0x7f8ab40c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 0x7f8ab40c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 -- 172.21.15.104:0/2085537599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa4009550 con 0x7f8ab406b410 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab27fc700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 0x7f8ab40c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8ab2ffd700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009c40 tx=0x7f8aa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:28.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.346+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4004480 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.347+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa401f070 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.347+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40038c0 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.347+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab415cd30 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.347+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab40c9350 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.348+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab40b0e30 con 0x7f8ab406b410 2023-06-10T18:59:28.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.348+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8aa4021030 con 0x7f8ab406b410 2023-06-10T18:59:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.348+0000 7f8a9b7fe700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 0x7f8aa006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.348+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8aa408a990 con 0x7f8ab406b410 2023-06-10T18:59:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.348+0000 7f8ab27fc700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 0x7f8aa006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.349+0000 7f8ab27fc700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 0x7f8aa006d190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4157ab0 tx=0x7f8aa8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:28.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.352+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8aa4055990 con 0x7f8ab406b410 2023-06-10T18:59:28.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.578+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ab4074050 con 0x7f8ab406b410 2023-06-10T18:59:28.584 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.579+0000 7f8a9b7fe700 1 -- 172.21.15.104:0/2085537599 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f8aa4058fb0 con 0x7f8ab406b410 2023-06-10T18:59:28.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:28.585 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T18:59:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 msgr2=0x7f8aa006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 0x7f8aa006d190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4157ab0 tx=0x7f8aa8008040 comp rx=0 tx=0).stop 2023-06-10T18:59:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 msgr2=0x7f8ab4151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:28.586 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009c40 tx=0x7f8aa4004750 comp rx=0 tx=0).stop 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 shutdown_connections 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ab40ce390 0x7f8ab40b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ab406b410 0x7f8ab4151a40 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8aa006ace0 0x7f8aa006d190 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 --2- 172.21.15.104:0/2085537599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ab406c720 0x7f8ab40c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:28.587 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 >> 172.21.15.104:0/2085537599 conn(0x7f8ab4069e70 msgr2=0x7f8ab40cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 shutdown_connections 2023-06-10T18:59:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:28.581+0000 7f8ab9575700 1 -- 172.21.15.104:0/2085537599 wait complete. 2023-06-10T18:59:28.588 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T18:59:29.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-10T18:59:23.247307+0000', 'modified': '2023-06-10T18:59:26.256259+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11139': {'gid': 11139, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6835/3645224862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 3645224862}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 3645224862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-10T18:59:29.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-10T18:59:29.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-06-10T18:59:29.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.121+0000 7f7d7d3fb700 1 Processor -- start 2023-06-10T18:59:29.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.121+0000 7f7d7d3fb700 1 -- start start 2023-06-10T18:59:29.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d7d3fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780aeaa0 0x7f7d780cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d7d3fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d780cd680 0x7f7d7806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d7d3fb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d780cdae0 con 0x7f7d780b1080 2023-06-10T18:59:29.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d7d3fb700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780c9bc0 con 0x7f7d780cd680 2023-06-10T18:59:29.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d7d3fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780c9e60 con 0x7f7d780aeaa0 2023-06-10T18:59:29.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d780cd680 0x7f7d7806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7d780cd680 0x7f7d7806dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41672/0 (socket says 172.21.15.104:41672) 2023-06-10T18:59:29.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 -- 172.21.15.104:0/1803188924 learned_addr learned my addr 172.21.15.104:0/1803188924 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780aeaa0 0x7f7d780cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 -- 172.21.15.104:0/1803188924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780aeaa0 msgr2=0x7f7d780cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780aeaa0 0x7f7d780cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 -- 172.21.15.104:0/1803188924 >> v1:172.21.15.114:6789/0 conn(0x7f7d780b1080 legacy=0x7f7d780ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 -- 172.21.15.104:0/1803188924 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780aee70 con 0x7f7d780cd680 2023-06-10T18:59:29.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.122+0000 7f7d777fe700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780cd680 0x7f7d7806dd60 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009e50 tx=0x7f7d7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72db376760b9b87a server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d75ffb700 1 -- 172.21.15.104:0/1803188924 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000e070 con 0x7f7d780cd680 2023-06-10T18:59:29.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d75ffb700 1 -- 172.21.15.104:0/1803188924 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70003a20 con 0x7f7d780cd680 2023-06-10T18:59:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d75ffb700 1 -- 172.21.15.104:0/1803188924 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70003dd0 con 0x7f7d780cd680 2023-06-10T18:59:29.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/1803188924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780cd680 msgr2=0x7f7d7806dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780cd680 0x7f7d7806dd60 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009e50 tx=0x7f7d7000b330 comp rx=0 tx=0).stop 2023-06-10T18:59:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/1803188924 shutdown_connections 2023-06-10T18:59:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780cd680 0x7f7d7806dd60 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.137 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/1803188924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780aeaa0 0x7f7d780cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/1803188924 >> 172.21.15.104:0/1803188924 conn(0x7f7d7806b8a0 msgr2=0x7f7d780cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/1803188924 shutdown_connections 2023-06-10T18:59:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.123+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/1803188924 wait complete. 2023-06-10T18:59:29.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.124+0000 7f7d7d3fb700 1 Processor -- start 2023-06-10T18:59:29.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 -- start start 2023-06-10T18:59:29.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780aeaa0 0x7f7d780b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780cd680 0x7f7d780cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780b4c70 con 0x7f7d780b1080 2023-06-10T18:59:29.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781f1bb0 con 0x7f7d780aeaa0 2023-06-10T18:59:29.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d7d3fb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781474a0 con 0x7f7d780cd680 2023-06-10T18:59:29.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41676/0 (socket says 172.21.15.104:41676) 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d767fc700 1 -- 172.21.15.104:0/3501943068 learned_addr learned my addr 172.21.15.104:0/3501943068 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.125+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780aeaa0 0x7f7d780b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d777fe700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780cd680 0x7f7d780cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 -- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780cd680 msgr2=0x7f7d780cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780cd680 0x7f7d780cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 -- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780aeaa0 msgr2=0x7f7d780b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780aeaa0 0x7f7d780b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 -- 172.21.15.104:0/3501943068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70009b30 con 0x7f7d780b1080 2023-06-10T18:59:29.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d767fc700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c500 tx=0x7f7d6800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d5f7fe700 1 -- 172.21.15.104:0/3501943068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68011790 con 0x7f7d780b1080 2023-06-10T18:59:29.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d5f7fe700 1 -- 172.21.15.104:0/3501943068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d680118f0 con 0x7f7d780b1080 2023-06-10T18:59:29.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d5f7fe700 1 -- 172.21.15.104:0/3501943068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68011ca0 con 0x7f7d780b1080 2023-06-10T18:59:29.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d780b45b0 con 0x7f7d780b1080 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.126+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d780b42a0 con 0x7f7d780b1080 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.128+0000 7f7d5f7fe700 1 -- 172.21.15.104:0/3501943068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7d6801c460 con 0x7f7d780b1080 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.128+0000 7f7d5f7fe700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 0x7f7d6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.128+0000 7f7d5f7fe700 1 -- 172.21.15.104:0/3501943068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7d68089300 con 0x7f7d780b1080 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.128+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 0x7f7d6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.129+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 0x7f7d6406d190 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7d781f51e0 tx=0x7f7d6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.130+0000 7f7d89580b80 1 Processor -- start 2023-06-10T18:59:29.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.130+0000 7f7d89580b80 1 -- start start 2023-06-10T18:59:29.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93be320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f92c2c60 con 0x5601f93e5d80 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f93159f0 con 0x5601f93bdf50 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5601f930ee80 con 0x5601f93bb740 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41684/0 (socket says 172.21.15.104:41684) 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 -- 172.21.15.104:0/116885309 learned_addr learned my addr 172.21.15.104:0/116885309 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 -- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 msgr2=0x5601f93e8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 -- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 msgr2=0x5601f93be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93be320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 -- 172.21.15.104:0/116885309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601f91905e0 con 0x5601f93e5d80 2023-06-10T18:59:29.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.131+0000 7f7d767fc700 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f7d68057cc0 tx=0x7f7d68057c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c774e6b279dba5e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d5cff9700 1 -- 172.21.15.104:0/116885309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68056100 con 0x5601f93e5d80 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d5cff9700 1 -- 172.21.15.104:0/116885309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d680890b0 con 0x5601f93e5d80 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d5cff9700 1 -- 172.21.15.104:0/116885309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6805fe80 con 0x5601f93e5d80 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 -- 172.21.15.104:0/116885309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 msgr2=0x5601f93e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f7d68057cc0 tx=0x7f7d68057c30 comp rx=0 tx=0).stop 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 -- 172.21.15.104:0/116885309 shutdown_connections 2023-06-10T18:59:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 --2- 172.21.15.104:0/116885309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e4820 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 -- 172.21.15.104:0/116885309 >> 172.21.15.104:0/116885309 conn(0x5601f93b2fc0 msgr2=0x5601f94031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 -- 172.21.15.104:0/116885309 shutdown_connections 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.132+0000 7f7d89580b80 1 -- 172.21.15.104:0/116885309 wait complete. 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.133+0000 7f7d89580b80 1 -- start start 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.133+0000 7f7d89580b80 1 Processor -- start 2023-06-10T18:59:29.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f9412e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f92aea70 con 0x5601f93e5d80 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f92ae770 con 0x5601f93bdf50 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5601f91928d0 con 0x5601f93bb740 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 10 client.0 _ll_register_callbacks cb 0x5601f92b4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41692/0 (socket says 172.21.15.104:41692) 2023-06-10T18:59:29.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 -- 172.21.15.104:0/1645752265 learned_addr learned my addr 172.21.15.104:0/1645752265 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d767fc700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f9412e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 -- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 msgr2=0x5601f9412e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f9412e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 -- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 msgr2=0x5601f93e6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 -- 172.21.15.104:0/1645752265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d780b42a0 con 0x5601f93e5d80 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d767fc700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d777fe700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f7d70004000 tx=0x7f7d70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2615a9859802fb74 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d55ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700094f0 con 0x5601f93e5d80 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7d7000be10 con 0x5601f93e5d80 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.134+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601f93b0780 con 0x5601f93e5d80 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d70004240 con 0x5601f93e5d80 2023-06-10T18:59:29.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 10 client.11271 fetch_fsmap learned FSMap version 117 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5601f913cd90 con 0x5601f93e5d80 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5601f93b0780 con 0x5601f93e5d80 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 117) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7d700043a0 con 0x5601f93e5d80 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 10 client.11271 fetch_fsmap finished waiting for FSMap version 117 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 10 client.11271 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 20 client.11271 populate_metadata read hostname 'smithi104' 2023-06-10T18:59:29.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f94010e0 0x5601f941fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 4 client.11271 mds_command: new command op to 11139 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 --> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601f9307e60 con 0x5601f94010e0 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f94010e0 0x5601f941fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d55ffb700 10 client.11271 ms_handle_connect on v2:172.21.15.104:6834/3645224862 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.135+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f94010e0 0x5601f941fbd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beffcee01cd17f81 server_cookie=6ba4bfd3ebcaa5f1 in_seq=0 out_seq=0 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d55ffb700 1 -- 172.21.15.104:0/1645752265 <== mds.0 v2:172.21.15.104:6834/3645224862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5601f9307e60 con 0x5601f94010e0 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d55ffb700 10 client.11271 handle_command_reply: tid=0 2023-06-10T18:59:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 client.11271 shutdown 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 msgr2=0x5601f93e6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f7d70004000 tx=0x7f7d70004ad0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 shutdown_connections 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f94010e0 0x5601f941fbd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93e6690 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d55ffb700 0 client.11271 ms_handle_reset on v2:172.21.15.104:6834/3645224862 2023-06-10T18:59:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f9412e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 --2- 172.21.15.104:0/1645752265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 >> 172.21.15.104:0/1645752265 conn(0x5601f93b2fc0 msgr2=0x5601f93e4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 shutdown_connections 2023-06-10T18:59:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 1 -- 172.21.15.104:0/1645752265 wait complete. 2023-06-10T18:59:29.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.136+0000 7f7d89580b80 20 client.11271 trim_cache size 0 max 16384 2023-06-10T18:59:29.174 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 Processor -- start 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 -- start start 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f93b9070 con 0x5601f93e5d80 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f94117d0 con 0x5601f93bdf50 2023-06-10T18:59:29.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5601f94129d0 con 0x5601f93bb740 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.154+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41698/0 (socket says 172.21.15.104:41698) 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 -- 172.21.15.104:0/2537713088 learned_addr learned my addr 172.21.15.104:0/2537713088 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d767fc700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 -- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 msgr2=0x5601f93e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 -- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 msgr2=0x5601f93e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 -- 172.21.15.104:0/2537713088 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601f9125c10 con 0x5601f93e5d80 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d767fc700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d777fe700 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7d70002230 tx=0x7f7d70018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac6c142925a9a7e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d5d7fa700 1 -- 172.21.15.104:0/2537713088 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700070a0 con 0x5601f93e5d80 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d5d7fa700 1 -- 172.21.15.104:0/2537713088 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70004810 con 0x5601f93e5d80 2023-06-10T18:59:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d5d7fa700 1 -- 172.21.15.104:0/2537713088 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700094d0 con 0x5601f93e5d80 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d89580b80 1 -- 172.21.15.104:0/2537713088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 msgr2=0x5601f93ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.155+0000 7f7d89580b80 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7d70002230 tx=0x7f7d70018000 comp rx=0 tx=0).stop 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 -- 172.21.15.104:0/2537713088 shutdown_connections 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93e5d80 0x5601f93ff4c0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93bb740 0x5601f93e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 --2- 172.21.15.104:0/2537713088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bdf50 0x5601f93e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 -- 172.21.15.104:0/2537713088 >> 172.21.15.104:0/2537713088 conn(0x5601f93b2fc0 msgr2=0x5601f93ddf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 -- 172.21.15.104:0/2537713088 shutdown_connections 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 -- 172.21.15.104:0/2537713088 wait complete. 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 -- start start 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.156+0000 7f7d89580b80 1 Processor -- start 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bb740 0x5601f9402c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93e5d80 0x5601f941c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f93dfdc0 con 0x5601f93bdf50 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601f93bbfd0 con 0x5601f93bb740 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5601f941bac0 con 0x5601f93e5d80 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d89580b80 10 client.0 _ll_register_callbacks cb 0x5601f92b4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41704/0 (socket says 172.21.15.104:41704) 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 -- 172.21.15.104:0/397572601 learned_addr learned my addr 172.21.15.104:0/397572601 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 -- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93e5d80 msgr2=0x5601f941c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93e5d80 0x5601f941c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 -- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bb740 msgr2=0x5601f9402c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bb740 0x5601f9402c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 -- 172.21.15.104:0/397572601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601f913cd90 con 0x5601f93bdf50 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d767fc700 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f7d6806e040 tx=0x7f7d680037f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2128842d51e45c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.157+0000 7f7d55ffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6807a070 con 0x5601f93bdf50 2023-06-10T18:59:29.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mon.0 v2:172.21.15.104:3300/0 2 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7d680872a0 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d680890b0 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d68056360 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 10 client.11277 fetch_fsmap learned FSMap version 117 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5601f9163200 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mon.0 v2:172.21.15.104:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d680890b0 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mon.0 v2:172.21.15.104:3300/0 5 ==== fsmap(e 117) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7d6805fe80 con 0x5601f93bdf50 2023-06-10T18:59:29.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 10 client.11277 fetch_fsmap finished waiting for FSMap version 117 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 10 client.11277 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 20 client.11277 populate_metadata read hostname 'smithi104' 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f93e64e0 0x5601f93e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 4 client.11277 mds_command: new command op to 11139 tid=0[{"prefix": "status"}] 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 --> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5601f92ca000 con 0x5601f93e64e0 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f93e64e0 0x5601f93e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d55ffb700 10 client.11277 ms_handle_connect on v2:172.21.15.104:6834/3645224862 2023-06-10T18:59:29.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.158+0000 7f7d76ffd700 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f93e64e0 0x5601f93e68b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b81547f41a1203a server_cookie=5a3395b8f8885599 in_seq=0 out_seq=0 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d55ffb700 1 -- 172.21.15.104:0/397572601 <== mds.0 v2:172.21.15.104:6834/3645224862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5601f92ca000 con 0x5601f93e64e0 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d55ffb700 10 client.11277 handle_command_reply: tid=0 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 client.11277 shutdown 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 msgr2=0x5601f93df880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f7d6806e040 tx=0x7f7d680037f0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 shutdown_connections 2023-06-10T18:59:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:6834/3645224862,v1:172.21.15.104:6835/3645224862] conn(0x5601f93e64e0 0x5601f93e68b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5601f93bb740 0x5601f9402c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d55ffb700 0 client.11277 ms_handle_reset on v2:172.21.15.104:6834/3645224862 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5601f93e5d80 0x5601f941c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 --2- 172.21.15.104:0/397572601 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5601f93bdf50 0x5601f93df880 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 >> 172.21.15.104:0/397572601 conn(0x5601f93b2fc0 msgr2=0x5601f9423f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 shutdown_connections 2023-06-10T18:59:29.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 1 -- 172.21.15.104:0/397572601 wait complete. 2023-06-10T18:59:29.186 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d89580b80 20 client.11277 trim_cache size 0 max 16384 2023-06-10T18:59:29.186 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:59:29.186 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_fsid": "f08e2e30-012f-4260-8ced-2a652c595f11", 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "whoami": 0, 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 11139, 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "want_state": "up:active", 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "up:active", 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs", 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "rank_uptime": 5.9102531320000002, 2023-06-10T18:59:29.187 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 117, 2023-06-10T18:59:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch": 187, 2023-06-10T18:59:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_epoch_barrier": 187, 2023-06-10T18:59:29.188 INFO:teuthology.orchestra.run.smithi104.stdout: "uptime": 11.247141147000001 2023-06-10T18:59:29.188 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:59:29.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 msgr2=0x7f7d6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 0x7f7d6406d190 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7d781f51e0 tx=0x7f7d6c008040 comp rx=0 tx=0).stop 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 msgr2=0x7f7d780cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c500 tx=0x7f7d6800c810 comp rx=0 tx=0).stop 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 shutdown_connections 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7d6406ace0 0x7f7d6406d190 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780aeaa0 0x7f7d780b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.190 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d780cd680 0x7f7d780cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 --2- 172.21.15.104:0/3501943068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d780b1080 0x7f7d780cf0d0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 >> 172.21.15.104:0/3501943068 conn(0x7f7d7806b8a0 msgr2=0x7f7d7806d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 shutdown_connections 2023-06-10T18:59:29.191 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.159+0000 7f7d7d3fb700 1 -- 172.21.15.104:0/3501943068 wait complete. 2023-06-10T18:59:29.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:59:29.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.306+0000 7f907c11d700 1 Processor -- start 2023-06-10T18:59:29.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.306+0000 7f907c11d700 1 -- start start 2023-06-10T18:59:29.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907c11d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907406e0e0 0x7f907406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907c11d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9074069e70 0x7f90740ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907c11d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907406d290 con 0x7f907406e4b0 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907c11d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9074071640 con 0x7f907406e0e0 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907c11d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90740aef00 con 0x7f9074069e70 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f9079eb9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907406e0e0 0x7f907406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f9079eb9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f907406e0e0 0x7f907406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41710/0 (socket says 172.21.15.104:41710) 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f9079eb9700 1 -- 172.21.15.104:0/2465110482 learned_addr learned my addr 172.21.15.104:0/2465110482 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.307+0000 7f907a6ba700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9074069e70 0x7f90740ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9079eb9700 1 -- 172.21.15.104:0/2465110482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9074069e70 msgr2=0x7f90740ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9079eb9700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9074069e70 0x7f90740ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9079eb9700 1 -- 172.21.15.104:0/2465110482 >> v1:172.21.15.114:6789/0 conn(0x7f907406e4b0 legacy=0x7f90740c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9079eb9700 1 -- 172.21.15.104:0/2465110482 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740ca2d0 con 0x7f907406e0e0 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9079eb9700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907406e0e0 0x7f907406cd50 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989bcbed5665e33c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9078eb7700 1 -- 172.21.15.104:0/2465110482 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064005e30 con 0x7f907406e0e0 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9078eb7700 1 -- 172.21.15.104:0/2465110482 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064004030 con 0x7f907406e0e0 2023-06-10T18:59:29.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f9078eb7700 1 -- 172.21.15.104:0/2465110482 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90640043e0 con 0x7f907406e0e0 2023-06-10T18:59:29.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 -- 172.21.15.104:0/2465110482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907406e0e0 msgr2=0x7f907406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907406e0e0 0x7f907406cd50 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 -- 172.21.15.104:0/2465110482 shutdown_connections 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f907406e0e0 0x7f907406cd50 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 --2- 172.21.15.104:0/2465110482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9074069e70 0x7f90740ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.308+0000 7f907c11d700 1 -- 172.21.15.104:0/2465110482 >> 172.21.15.104:0/2465110482 conn(0x7f907406b8a0 msgr2=0x7f90740cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.309+0000 7f907c11d700 1 -- 172.21.15.104:0/2465110482 shutdown_connections 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.309+0000 7f907c11d700 1 -- 172.21.15.104:0/2465110482 wait complete. 2023-06-10T18:59:29.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 Processor -- start 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 -- start start 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907406e0e0 0x7f90740b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 0x7f9074155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740b2970 con 0x7f9074069e70 2023-06-10T18:59:29.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740b2c20 con 0x7f907406e4b0 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.310+0000 7f907c11d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90740b2f00 con 0x7f907406e0e0 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41718/0 (socket says 172.21.15.104:41718) 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 -- 172.21.15.104:0/1570127715 learned_addr learned my addr 172.21.15.104:0/1570127715 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f907a6ba700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 0x7f9074155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f90796b8700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907406e0e0 0x7f90740b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 -- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907406e0e0 msgr2=0x7f90740b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907406e0e0 0x7f90740b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 -- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 msgr2=0x7f9074155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 0x7f9074155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 -- 172.21.15.104:0/1570127715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064009500 con 0x7f9074069e70 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f907a6ba700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 0x7f9074155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f9079eb9700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f9064009c10 tx=0x7f9064004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064011620 con 0x7f9074069e70 2023-06-10T18:59:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90640117c0 con 0x7f9074069e70 2023-06-10T18:59:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90740b31e0 con 0x7f9074069e70 2023-06-10T18:59:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064011b70 con 0x7f9074069e70 2023-06-10T18:59:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.311+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90740b37a0 con 0x7f9074069e70 2023-06-10T18:59:29.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.312+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9074074050 con 0x7f9074069e70 2023-06-10T18:59:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.313+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9064011cd0 con 0x7f9074069e70 2023-06-10T18:59:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.313+0000 7f906a7fc700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 0x7f905806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.313+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f906408a850 con 0x7f9074069e70 2023-06-10T18:59:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.313+0000 7f90796b8700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 0x7f905806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:29.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.314+0000 7f90796b8700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 0x7f905806d190 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f90740cd160 tx=0x7f9070009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:29.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.317+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9064055910 con 0x7f9074069e70 2023-06-10T18:59:29.551 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.546+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f907415c140 con 0x7f9074069e70 2023-06-10T18:59:29.552 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.547+0000 7f906a7fc700 1 -- 172.21.15.104:0/1570127715 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9064058f30 con 0x7f9074069e70 2023-06-10T18:59:29.552 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:59:29.556 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 msgr2=0x7f905806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 0x7f905806d190 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f90740cd160 tx=0x7f9070009040 comp rx=0 tx=0).stop 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 msgr2=0x7f90740aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f9064009c10 tx=0x7f9064004750 comp rx=0 tx=0).stop 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 shutdown_connections 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f905806ace0 0x7f905806d190 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907406e4b0 0x7f9074155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9074069e70 0x7f90740aff30 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 --2- 172.21.15.104:0/1570127715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907406e0e0 0x7f90740b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:29.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 >> 172.21.15.104:0/1570127715 conn(0x7f907406b8a0 msgr2=0x7f9074072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:29.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 shutdown_connections 2023-06-10T18:59:29.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:29.549+0000 7f907c11d700 1 -- 172.21.15.104:0/1570127715 wait complete. 2023-06-10T18:59:29.566 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:59:29.566 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:59:29.566 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:59:29.566 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:59:29.567 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:29.567 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:59:29.567 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:59:29.568 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:59:29.568 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:29.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.569 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:59:29.586 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:59:29.586 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:59:29.586 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:59:29.586 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.586 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:29.587 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:59:29.588 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:29.588 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:59:29.588 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41515sec preferred_lft 41515sec 2023-06-10T18:59:29.588 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0e:1a:b5:23:06:d7 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.589 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:29.590 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout:11: brx.2@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0e:1a:b5:23:06:d7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::c1a:b5ff:fe23:6d7/64 scope link 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:29.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.592 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:59:29.651 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:59:29.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.653 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:59:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:59:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:59:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:59:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:59:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:59:29.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.712 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:59:29.712 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:59:29.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:59:29.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:59:29.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:59:29.713 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:59:29.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.815 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:59:29.832 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 2) 2023-06-10T18:59:29.832 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:59:29.833 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:59:29.833 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:29.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:29.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:59:29.899 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:59:29.899 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:29.900 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:29.955 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:29.956 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:29.956 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:30.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:59:30.079 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:30.079 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:30.145 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:30.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:30.147 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:59:30.203 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:59:30.204 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-06-10T18:59:30.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:59:30.270 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:30.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:30.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:30.325+0000 7fbd7e1a45c0 -1 init, newargv = 0x560eb3e95ab0 newargc=15 2023-06-10T18:59:30.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[165779]: starting ceph client 2023-06-10T18:59:30.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[165779]: starting fuse 2023-06-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:30.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:30.340 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:59:30.395 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:59:30.396 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:59:30.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:59:30.397 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:59:30.397 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:59:30.397 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:59:30.397 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:59:30.398 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:30.398 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:59:30.398 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:59:30.399 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:59:30.399 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:59:30.399 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:30.399 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:59:30.400 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:59:30.400 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:59:30.400 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:59:30.401 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:30.401 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:59:30.401 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:59:30.402 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:59:30.402 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:59:30.402 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:59:30.402 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:59:30.403 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:59:30.403 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:59:30.403 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:30.404 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:59:30.404 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:59:30.517 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.165779.asok 2023-06-10T18:59:30.700 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.165779.asok 2023-06-10T18:59:30.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:30.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.165779.asok status 2023-06-10T18:59:30.854 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:59:30.855 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "165779", 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 11289, 2023-06-10T18:59:30.856 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 11289 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:30.857 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3522109968 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3522109968 2023-06-10T18:59:30.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.11289 192.168.0.1:0/3522109968", 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/3522109968", 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 117, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 187, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:59:30.859 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:59:30.860 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:59:30.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:30.865 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:59:30.865 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:30.881 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:59:30.881 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:30.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:30.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:30.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:59:31.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.104+0000 7f45db49f700 1 Processor -- start 2023-06-10T18:59:31.109 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.104+0000 7f45db49f700 1 -- start start 2023-06-10T18:59:31.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45db49f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45d40cf100 0x7f45d406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45db49f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d406d290 con 0x7f45d406e0e0 2023-06-10T18:59:31.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45db49f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d40ca2d0 con 0x7f45d406e4b0 2023-06-10T18:59:31.112 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45db49f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d4071640 con 0x7f45d40cf100 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45d923b700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f45d406e0e0 0x7f45d40c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39566/0 (socket says 172.21.15.104:39566) 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45d923b700 1 -- 172.21.15.104:0/3427484395 learned_addr learned my addr 172.21.15.104:0/3427484395 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/3427484395 >> v2:172.21.15.114:3300/0 conn(0x7f45d40cf100 0x7f45d406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407402384 0 0) 0x7f45d406d290 con 0x7f45d406e0e0 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c00035b0 con 0x7f45d406e0e0 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910879690 0 0) 0x7f45c00035b0 con 0x7f45d406e0e0 2023-06-10T18:59:31.113 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 >> v2:172.21.15.114:3300/0 conn(0x7f45d40cf100 msgr2=0x7f45d406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 --2- 172.21.15.104:0/3427484395 >> v2:172.21.15.114:3300/0 conn(0x7f45d40cf100 0x7f45d406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 >> v1:172.21.15.114:6789/0 conn(0x7f45d406e4b0 legacy=0x7f45d40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d406d4c0 con 0x7f45d406e0e0 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f45d00017f0 con 0x7f45d406e0e0 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 >> v1:172.21.15.104:6789/0 conn(0x7f45d406e0e0 legacy=0x7f45d40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0003a00 0x7f45c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45c0004850 0x7f45c0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.114 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 0x7f45c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4071640 con 0x7f45c0005200 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.105+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c000c110 con 0x7f45c0003a00 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c000c2d0 con 0x7f45c0004850 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 0x7f45c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d923b700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0003a00 0x7f45c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d8a3a700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45c0004850 0x7f45c0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.115 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 -- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45c0004850 msgr2=0x7f45c0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45c0004850 0x7f45c0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 -- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0003a00 msgr2=0x7f45c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0003a00 0x7f45c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 -- 172.21.15.104:0/3427484395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00035b0 con 0x7f45c0005200 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d8a3a700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45c0004850 0x7f45c0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 -- 172.21.15.104:0/3427484395 reap_dead start 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 0x7f45c000ba10 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009f90 tx=0x7f45c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cb9eb82b16cecd server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:31.116 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003a20 con 0x7f45c0005200 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc00f040 con 0x7f45c0005200 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.106+0000 7f45cbfff700 1 -- 172.21.15.104:0/3427484395 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003da0 con 0x7f45c0005200 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 -- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 msgr2=0x7f45c000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 0x7f45c000ba10 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009f90 tx=0x7f45c0005880 comp rx=0 tx=0).stop 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 -- 172.21.15.104:0/3427484395 shutdown_connections 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 --2- 172.21.15.104:0/3427484395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45c0005200 0x7f45c000ba10 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 -- 172.21.15.104:0/3427484395 >> 172.21.15.104:0/3427484395 conn(0x7f45d406b8a0 msgr2=0x7f45d40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:31.117 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 -- 172.21.15.104:0/3427484395 shutdown_connections 2023-06-10T18:59:31.118 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.107+0000 7f45db49f700 1 -- 172.21.15.104:0/3427484395 wait complete. 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.108+0000 7f45db49f700 1 Processor -- start 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.108+0000 7f45db49f700 1 -- start start 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0005200 0x7f45d40c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45d40b4d40 0x7f45d4156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.120 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40b54b0 con 0x7f45d4156b80 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40b25d0 con 0x7f45c0005200 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45db49f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d40b2870 con 0x7f45d40b4d40 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41754/0 (socket says 172.21.15.104:41754) 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 -- 172.21.15.104:0/2723747369 learned_addr learned my addr 172.21.15.104:0/2723747369 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d923b700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0005200 0x7f45d40c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d8a3a700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45d40b4d40 0x7f45d4156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.121 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 -- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45d40b4d40 msgr2=0x7f45d4156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45d40b4d40 0x7f45d4156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 -- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0005200 msgr2=0x7f45d40c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0005200 0x7f45d40c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 -- 172.21.15.104:0/2723747369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d40b2b50 con 0x7f45d4156b80 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.109+0000 7f45d9a3c700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f45d4072080 tx=0x7f45cc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.110+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003e10 con 0x7f45d4156b80 2023-06-10T18:59:31.122 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.110+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc004230 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.110+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d40b2e30 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.110+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0045e0 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.110+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d4154b60 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.111+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d40b3070 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.111+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f45cc004740 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.112+0000 7f45c9ffb700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 0x7f45b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.112+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f45cc08ab40 con 0x7f45d4156b80 2023-06-10T18:59:31.123 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.112+0000 7f45d923b700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 0x7f45b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.112+0000 7f45d923b700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 0x7f45b806d190 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f45d00011d0 tx=0x7f45d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:31.124 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.116+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f45d40b3070 con 0x7f45d4156b80 2023-06-10T18:59:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.350+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f45d40b3070 con 0x7f45d4156b80 2023-06-10T18:59:31.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.350+0000 7f45c9ffb700 1 -- 172.21.15.104:0/2723747369 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) v1 ==== 53+0+73 (secure 0 0 0) 0x7f45cc018060 con 0x7f45d4156b80 2023-06-10T18:59:31.356 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:59:31.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 msgr2=0x7f45b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 0x7f45b806d190 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f45d00011d0 tx=0x7f45d0009040 comp rx=0 tx=0).stop 2023-06-10T18:59:31.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 msgr2=0x7f45d40b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f45d4072080 tx=0x7f45cc004b40 comp rx=0 tx=0).stop 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 shutdown_connections 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f45b806ace0 0x7f45b806d190 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45c0005200 0x7f45d40c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d4156b80 0x7f45d40b3350 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 --2- 172.21.15.104:0/2723747369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45d40b4d40 0x7f45d4156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:31.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 >> 172.21.15.104:0/2723747369 conn(0x7f45d406b8a0 msgr2=0x7f45d40cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:31.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 shutdown_connections 2023-06-10T18:59:31.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:31.353+0000 7f45db49f700 1 -- 172.21.15.104:0/2723747369 wait complete. 2023-06-10T18:59:31.370 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:59:31.370 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:59:31.370 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2023-06-10T18:59:31.370 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-06-10T18:59:31.371 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:31.371 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:59:31.371 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:59:31.371 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:59:31.372 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:59:31.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.372 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2023-06-10T18:59:31.392 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:31.393 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41509sec preferred_lft 41509sec 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9a3c:54cd:852f:84e9/64 scope link noprefixroute 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::403a:69ff:fe37:32d6/64 scope link 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 92:dc:d6:b8:18:47 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::90dc:d6ff:feb8:1847/64 scope link 2023-06-10T18:59:31.395 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.396 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:31.396 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether e2:76:57:1a:e8:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T18:59:31.396 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e076:57ff:fe1a:e889/64 scope link 2023-06-10T18:59:31.396 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:31.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.397 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:59:31.459 INFO:teuthology.orchestra.run.smithi114.stdout:1 2023-06-10T18:59:31.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.461 DEBUG:teuthology.orchestra.run.smithi114:> route 2023-06-10T18:59:31.520 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2023-06-10T18:59:31.520 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:59:31.520 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:59:31.521 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:59:31.521 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:59:31.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.521 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2023-06-10T18:59:31.522 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2023-06-10T18:59:31.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:59:31.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:59:31.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:59:31.523 DEBUG:teuthology.orchestra.run.smithi114:> ') 2023-06-10T18:59:31.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.626 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T18:59:31.643 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T18:59:31.643 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T18:59:31.644 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:59:31.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2023-06-10T18:59:31.709 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-06-10T18:59:31.709 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:31.710 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:31.765 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-06-10T18:59:31.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:31.766 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:31.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:59:31.888 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:31.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:31.953 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:31.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:31.955 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:59:32.011 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:59:32.012 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-06-10T18:59:32.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2023-06-10T18:59:32.079 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:32.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:32.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:2023-06-10T18:59:32.132+0000 7f16843b25c0 -1 init, newargv = 0x55eb32857bc0 newargc=15 2023-06-10T18:59:32.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[118542]: starting ceph client 2023-06-10T18:59:32.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[118542]: starting fuse 2023-06-10T18:59:32.146 INFO:teuthology.orchestra.run.smithi114.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:32.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:32.148 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2023-06-10T18:59:32.204 INFO:teuthology.orchestra.run.smithi114.stdout:47 2023-06-10T18:59:32.205 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:59:32.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:59:32.205 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2023-06-10T18:59:32.206 DEBUG:teuthology.orchestra.run.smithi114:> import re 2023-06-10T18:59:32.206 DEBUG:teuthology.orchestra.run.smithi114:> import os 2023-06-10T18:59:32.206 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2023-06-10T18:59:32.207 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:59:32.207 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2023-06-10T18:59:32.207 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-06-10T18:59:32.208 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2023-06-10T18:59:32.208 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-06-10T18:59:32.208 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:59:32.208 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2023-06-10T18:59:32.209 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2023-06-10T18:59:32.209 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2023-06-10T18:59:32.209 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2023-06-10T18:59:32.210 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:59:32.210 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2023-06-10T18:59:32.210 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:59:32.211 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:59:32.211 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:59:32.211 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2023-06-10T18:59:32.211 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2023-06-10T18:59:32.212 DEBUG:teuthology.orchestra.run.smithi114:> return f 2023-06-10T18:59:32.212 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:59:32.212 DEBUG:teuthology.orchestra.run.smithi114:> 2023-06-10T18:59:32.213 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.1")) 2023-06-10T18:59:32.213 DEBUG:teuthology.orchestra.run.smithi114:> ' 2023-06-10T18:59:32.326 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.1.118542.asok 2023-06-10T18:59:32.508 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.118542.asok 2023-06-10T18:59:32.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:32.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.118542.asok status 2023-06-10T18:59:32.664 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "1", 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-06-10T18:59:32.665 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "118542", 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 9481, 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2023-06-10T18:59:32.666 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 9481 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2595006869 2023-06-10T18:59:32.667 INFO:teuthology.orchestra.run.smithi114.stdout: } 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2595006869 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.9481 192.168.0.1:0/2595006869", 2023-06-10T18:59:32.668 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/2595006869", 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 117, 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 187, 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2023-06-10T18:59:32.669 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-06-10T18:59:32.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:32.674 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2023-06-10T18:59:32.675 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:32.691 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2023-06-10T18:59:32.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:32.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:32.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-06-10T18:59:32.757 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_reconnect_timeout 2023-06-10T18:59:32.904 INFO:teuthology.orchestra.run.smithi104.stdout:{"mds_reconnect_timeout":"45.000000"} 2023-06-10T18:59:32.905 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_reconnect_timeout": "45.000000" } 2023-06-10T18:59:32.905 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get ms_max_backoff 2023-06-10T18:59:33.039 INFO:teuthology.orchestra.run.smithi104.stdout:{"ms_max_backoff":"15.000000"} 2023-06-10T18:59:33.040 DEBUG:tasks.cephfs.filesystem:_json_asok output { "ms_max_backoff": "15.000000" } 2023-06-10T18:59:33.040 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:33.041 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:33.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.212+0000 7f3a2bfdc700 1 Processor -- start 2023-06-10T18:59:33.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.212+0000 7f3a2bfdc700 1 -- start start 2023-06-10T18:59:33.219 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.212+0000 7f3a2bfdc700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a2415a230 0x7f3a240ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2bfdc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a2406a580 0x7f3a2406dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2bfdc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a240cc050 con 0x7f3a2406d810 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2bfdc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a2406e290 con 0x7f3a2406a580 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2bfdc700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a240cba00 con 0x7f3a2415a230 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a2406a580 0x7f3a2406dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.220 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a29577700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3a2406d810 0x7f3a24069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39588/0 (socket says 172.21.15.104:39588) 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3a2406a580 0x7f3a2406dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41762/0 (socket says 172.21.15.104:41762) 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a29577700 1 -- 172.21.15.104:0/4255864000 learned_addr learned my addr 172.21.15.104:0/4255864000 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a29d78700 1 --2- 172.21.15.104:0/4255864000 >> v2:172.21.15.114:3300/0 conn(0x7f3a2415a230 0x7f3a240ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 -- 172.21.15.104:0/4255864000 >> v2:172.21.15.114:3300/0 conn(0x7f3a2415a230 msgr2=0x7f3a240ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 --2- 172.21.15.104:0/4255864000 >> v2:172.21.15.114:3300/0 conn(0x7f3a2415a230 0x7f3a240ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 -- 172.21.15.104:0/4255864000 >> v1:172.21.15.104:6789/0 conn(0x7f3a2406d810 legacy=0x7f3a24069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 -- 172.21.15.104:0/4255864000 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240cbce0 con 0x7f3a2406a580 2023-06-10T18:59:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a28d76700 1 -- 172.21.15.104:0/4255864000 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675934301 0 0) 0x7f3a240cc050 con 0x7f3a2406d810 2023-06-10T18:59:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a2a579700 1 --2- 172.21.15.104:0/4255864000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406a580 0x7f3a2406dc80 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f3a240b21d0 tx=0x7f3a1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8424e07114ff21 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a28d76700 1 -- 172.21.15.104:0/4255864000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00e070 con 0x7f3a2406a580 2023-06-10T18:59:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a28d76700 1 -- 172.21.15.104:0/4255864000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c00bbd0 con 0x7f3a2406a580 2023-06-10T18:59:33.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.213+0000 7f3a28d76700 1 -- 172.21.15.104:0/4255864000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c012430 con 0x7f3a2406a580 2023-06-10T18:59:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/4255864000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406a580 msgr2=0x7f3a2406dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/4255864000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406a580 0x7f3a2406dc80 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f3a240b21d0 tx=0x7f3a1c00b330 comp rx=0 tx=0).stop 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/4255864000 shutdown_connections 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/4255864000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406a580 0x7f3a2406dc80 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/4255864000 >> v2:172.21.15.114:3300/0 conn(0x7f3a2415a230 0x7f3a240ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/4255864000 >> 172.21.15.104:0/4255864000 conn(0x7f3a2406b8a0 msgr2=0x7f3a240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/4255864000 shutdown_connections 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.214+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/4255864000 wait complete. 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.216+0000 7f3a2bfdc700 1 Processor -- start 2023-06-10T18:59:33.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.216+0000 7f3a2bfdc700 1 -- start start 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.216+0000 7f3a2bfdc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a2406a580 0x7f3a241595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a2415a230 0x7f3a240b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a24159b30 con 0x7f3a2406d810 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2415a8a0 con 0x7f3a2415a230 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a241540e0 con 0x7f3a2406a580 2023-06-10T18:59:33.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41766/0 (socket says 172.21.15.104:41766) 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 -- 172.21.15.104:0/3444913944 learned_addr learned my addr 172.21.15.104:0/3444913944 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2a579700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a2415a230 0x7f3a240b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29d78700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a2406a580 0x7f3a241595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 -- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a2406a580 msgr2=0x7f3a241595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a2406a580 0x7f3a241595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 -- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a2415a230 msgr2=0x7f3a240b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a2415a230 0x7f3a240b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 -- 172.21.15.104:0/3444913944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c009a70 con 0x7f3a2406d810 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a29577700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f3a180011e0 tx=0x7f3a1800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004c80 con 0x7f3a2406d810 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a18004e00 con 0x7f3a2406d810 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a241543c0 con 0x7f3a2406d810 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18007310 con 0x7f3a2406d810 2023-06-10T18:59:33.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.217+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a24154980 con 0x7f3a2406d810 2023-06-10T18:59:33.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.218+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a24074050 con 0x7f3a2406d810 2023-06-10T18:59:33.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.219+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3a18007470 con 0x7f3a2406d810 2023-06-10T18:59:33.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.219+0000 7f3a167fc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 0x7f3a0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.219+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3a180891e0 con 0x7f3a2406d810 2023-06-10T18:59:33.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.219+0000 7f3a29d78700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 0x7f3a0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.220+0000 7f3a29d78700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 0x7f3a0806d190 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.223+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3a18054150 con 0x7f3a2406d810 2023-06-10T18:59:33.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.412+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a240c9e10 con 0x7f3a2406d810 2023-06-10T18:59:33.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.413+0000 7f3a167fc700 1 -- 172.21.15.104:0/3444913944 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+18618 (secure 0 0 0) 0x7f3a18057770 con 0x7f3a2406d810 2023-06-10T18:59:33.420 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:33.420 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":187,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:24.248976+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 msgr2=0x7f3a0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 0x7f3a0806d190 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3a20006fd0 tx=0x7f3a20008040 comp rx=0 tx=0).stop 2023-06-10T18:59:33.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 msgr2=0x7f3a240c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f3a180011e0 tx=0x7f3a1800ca30 comp rx=0 tx=0).stop 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 shutdown_connections 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a0806ace0 0x7f3a0806d190 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a2415a230 0x7f3a240b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a2406a580 0x7f3a241595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 --2- 172.21.15.104:0/3444913944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a2406d810 0x7f3a240c6da0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 >> 172.21.15.104:0/3444913944 conn(0x7f3a2406b8a0 msgr2=0x7f3a24157c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:33.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 shutdown_connections 2023-06-10T18:59:33.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.416+0000 7f3a2bfdc700 1 -- 172.21.15.104:0/3444913944 wait complete. 2023-06-10T18:59:33.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:59:21.300558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:59:22.213068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:59:33.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:59:33.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 Processor -- start 2023-06-10T18:59:33.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 -- start start 2023-06-10T18:59:33.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f887c0ad930 0x7f887c0add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c15cb00 con 0x7f887c15c730 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c0acd90 con 0x7f887c0cd8f0 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.552+0000 7f8881d1e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c0ac460 con 0x7f887c0ad930 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887b7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f887c15c730 0x7f887c0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39600/0 (socket says 172.21.15.104:39600) 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887b7fe700 1 -- 172.21.15.104:0/3452179422 learned_addr learned my addr 172.21.15.104:0/3452179422 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887bfff700 1 --2- 172.21.15.104:0/3452179422 >> v2:172.21.15.114:3300/0 conn(0x7f887c0ad930 0x7f887c0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887a7fc700 1 -- 172.21.15.104:0/3452179422 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704392734 0 0) 0x7f886c001290 con 0x7f887c15c730 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887a7fc700 1 -- 172.21.15.104:0/3452179422 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88640035b0 con 0x7f887c15c730 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887bfff700 1 -- 172.21.15.104:0/3452179422 >> v1:172.21.15.114:6789/0 conn(0x7f887c0cd8f0 legacy=0x7f887c0ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887bfff700 1 -- 172.21.15.104:0/3452179422 >> v1:172.21.15.104:6789/0 conn(0x7f887c15c730 legacy=0x7f887c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887bfff700 1 -- 172.21.15.104:0/3452179422 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0ac740 con 0x7f887c0ad930 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887bfff700 1 --2- 172.21.15.104:0/3452179422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ad930 0x7f887c0add50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8874009d90 tx=0x7f887400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3b8fa58bb970055 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887a7fc700 1 -- 172.21.15.104:0/3452179422 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400e070 con 0x7f887c0ad930 2023-06-10T18:59:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887a7fc700 1 -- 172.21.15.104:0/3452179422 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400bb20 con 0x7f887c0ad930 2023-06-10T18:59:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.553+0000 7f887a7fc700 1 -- 172.21.15.104:0/3452179422 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874012430 con 0x7f887c0ad930 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 -- 172.21.15.104:0/3452179422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ad930 msgr2=0x7f887c0add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 --2- 172.21.15.104:0/3452179422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ad930 0x7f887c0add50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8874009d90 tx=0x7f887400b330 comp rx=0 tx=0).stop 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 -- 172.21.15.104:0/3452179422 shutdown_connections 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 --2- 172.21.15.104:0/3452179422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ad930 0x7f887c0add50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 -- 172.21.15.104:0/3452179422 >> 172.21.15.104:0/3452179422 conn(0x7f887c06b8a0 msgr2=0x7f887c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:33.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 -- 172.21.15.104:0/3452179422 shutdown_connections 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.554+0000 7f8881d1e700 1 -- 172.21.15.104:0/3452179422 wait complete. 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.555+0000 7f8881d1e700 1 Processor -- start 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.555+0000 7f8881d1e700 1 -- start start 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f887c0cd8f0 0x7f887c0c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c15c730 0x7f887c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.565 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0c7050 con 0x7f887c0ad930 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0c7330 con 0x7f887c15c730 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f8881d1e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0c7610 con 0x7f887c0cd8f0 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41780/0 (socket says 172.21.15.104:41780) 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 -- 172.21.15.104:0/361974719 learned_addr learned my addr 172.21.15.104:0/361974719 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887affd700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f887c0cd8f0 0x7f887c0c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 -- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f887c0cd8f0 msgr2=0x7f887c0c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.566 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f887c0cd8f0 0x7f887c0c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887bfff700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c15c730 0x7f887c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 -- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c15c730 msgr2=0x7f887c06e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c15c730 0x7f887c06e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.556+0000 7f887b7fe700 1 -- 172.21.15.104:0/361974719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009a70 con 0x7f887c0ad930 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f887b7fe700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f886c00c700 tx=0x7f886c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c005c40 con 0x7f887c0ad930 2023-06-10T18:59:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c005da0 con 0x7f887c0ad930 2023-06-10T18:59:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0072f0 con 0x7f887c0ad930 2023-06-10T18:59:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c06f150 con 0x7f887c0ad930 2023-06-10T18:59:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.557+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c0c8130 con 0x7f887c0ad930 2023-06-10T18:59:33.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.558+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c0c99e0 con 0x7f887c0ad930 2023-06-10T18:59:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.558+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f886c007450 con 0x7f887c0ad930 2023-06-10T18:59:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.559+0000 7f8863fff700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 0x7f886806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.559+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f886c088970 con 0x7f887c0ad930 2023-06-10T18:59:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.559+0000 7f887affd700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 0x7f886806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.559+0000 7f887affd700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 0x7f886806d190 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cbab0 tx=0x7f8870009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.563+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f886c0538f0 con 0x7f887c0ad930 2023-06-10T18:59:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.748+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f887c15cb00 con 0x7f887c0ad930 2023-06-10T18:59:33.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.749+0000 7f8863fff700 1 -- 172.21.15.104:0/361974719 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7f887c15cb00 con 0x7f887c0ad930 2023-06-10T18:59:33.755 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:59:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 msgr2=0x7f886806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 0x7f886806d190 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cbab0 tx=0x7f8870009040 comp rx=0 tx=0).stop 2023-06-10T18:59:33.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 msgr2=0x7f887c0c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f886c00c700 tx=0x7f886c00cac0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 shutdown_connections 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f886806ace0 0x7f886806d190 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c15c730 0x7f887c06e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f887c0ad930 0x7f887c0c4090 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 --2- 172.21.15.104:0/361974719 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f887c0cd8f0 0x7f887c0c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.758 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 >> 172.21.15.104:0/361974719 conn(0x7f887c06b8a0 msgr2=0x7f887c0cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:33.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 shutdown_connections 2023-06-10T18:59:33.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.751+0000 7f8881d1e700 1 -- 172.21.15.104:0/361974719 wait complete. 2023-06-10T18:59:33.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:59:33.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.885+0000 7f388473c700 1 Processor -- start 2023-06-10T18:59:33.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.885+0000 7f388473c700 1 -- start start 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f388473c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f387c158fe0 0x7f387c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f388473c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f388473c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387c0c5d80 con 0x7f387c158c10 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f388473c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c0cec80 con 0x7f387c158fe0 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f388473c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0cdbe0 con 0x7f387c0c4b90 2023-06-10T18:59:33.893 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f3881cd7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f387c158fe0 0x7f387c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f38824d8700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f387c158c10 0x7f387c0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39614/0 (socket says 172.21.15.104:39614) 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f38824d8700 1 -- 172.21.15.104:0/1515942656 learned_addr learned my addr 172.21.15.104:0/1515942656 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f3881cd7700 1 -- 172.21.15.104:0/1515942656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 msgr2=0x7f387c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f3881cd7700 1 --2- 172.21.15.104:0/1515942656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f3881cd7700 1 -- 172.21.15.104:0/1515942656 >> v1:172.21.15.104:6789/0 conn(0x7f387c158c10 legacy=0x7f387c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f3881cd7700 1 -- 172.21.15.104:0/1515942656 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c0cdec0 con 0x7f387c158fe0 2023-06-10T18:59:33.894 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.886+0000 7f38814d6700 1 -- 172.21.15.104:0/1515942656 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387201404 0 0) 0x7f387c0c5d80 con 0x7f387c158c10 2023-06-10T18:59:33.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f3881cd7700 1 --2- 172.21.15.104:0/1515942656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c06d440 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f3878009840 tx=0x7f3878009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ab1f832297da16 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f38814d6700 1 -- 172.21.15.104:0/1515942656 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878011830 con 0x7f387c158fe0 2023-06-10T18:59:33.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f38814d6700 1 -- 172.21.15.104:0/1515942656 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3878011990 con 0x7f387c158fe0 2023-06-10T18:59:33.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f38814d6700 1 -- 172.21.15.104:0/1515942656 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878011d20 con 0x7f387c158fe0 2023-06-10T18:59:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 -- 172.21.15.104:0/1515942656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 msgr2=0x7f387c06d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 --2- 172.21.15.104:0/1515942656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c06d440 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f3878009840 tx=0x7f3878009b50 comp rx=0 tx=0).stop 2023-06-10T18:59:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 -- 172.21.15.104:0/1515942656 shutdown_connections 2023-06-10T18:59:33.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 --2- 172.21.15.104:0/1515942656 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c06d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 --2- 172.21.15.104:0/1515942656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c06d440 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 -- 172.21.15.104:0/1515942656 >> 172.21.15.104:0/1515942656 conn(0x7f387c0ceec0 msgr2=0x7f387c0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 -- 172.21.15.104:0/1515942656 shutdown_connections 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.887+0000 7f388473c700 1 -- 172.21.15.104:0/1515942656 wait complete. 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 Processor -- start 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 -- start start 2023-06-10T18:59:33.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c0a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f387c158c10 0x7f387c149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c146f90 con 0x7f387c158fe0 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c149550 con 0x7f387c158c10 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f388473c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0a7b30 con 0x7f387c0c4b90 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f3882cd9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.889+0000 7f3882cd9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41796/0 (socket says 172.21.15.104:41796) 2023-06-10T18:59:33.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 -- 172.21.15.104:0/3087222309 learned_addr learned my addr 172.21.15.104:0/3087222309 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3881cd7700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f387c158c10 0x7f387c149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f38824d8700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c0a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 -- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 msgr2=0x7f387c0a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c0a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 -- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f387c158c10 msgr2=0x7f387c149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f387c158c10 0x7f387c149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:33.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 -- 172.21.15.104:0/3087222309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780094a0 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3882cd9700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f387400b970 tx=0x7f387400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387400c9b0 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387400cb30 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874014440 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c1465f0 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.890+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c146bb0 con 0x7f387c158fe0 2023-06-10T18:59:33.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.891+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c0ce460 con 0x7f387c158fe0 2023-06-10T18:59:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.892+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f387401d3b0 con 0x7f387c158fe0 2023-06-10T18:59:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.892+0000 7f3872ffd700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 0x7f386006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.892+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3874006e40 con 0x7f387c158fe0 2023-06-10T18:59:33.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.892+0000 7f38824d8700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 0x7f386006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.893+0000 7f38824d8700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 0x7f386006d190 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f386c000f60 tx=0x7f386c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:33.903 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:33.897+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f387c0ce460 con 0x7f387c158fe0 2023-06-10T18:59:34.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.082+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f387c1470d0 con 0x7f387c158fe0 2023-06-10T18:59:34.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.083+0000 7f3872ffd700 1 -- 172.21.15.104:0/3087222309 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v187) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3874019020 con 0x7f387c158fe0 2023-06-10T18:59:34.089 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:59:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 msgr2=0x7f386006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 0x7f386006d190 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f386c000f60 tx=0x7f386c007be0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 msgr2=0x7f387c145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f387400b970 tx=0x7f387400bd30 comp rx=0 tx=0).stop 2023-06-10T18:59:34.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 shutdown_connections 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f387c158fe0 0x7f387c145c70 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f387c0c4b90 0x7f387c0a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f386006ace0 0x7f386006d190 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 --2- 172.21.15.104:0/3087222309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f387c158c10 0x7f387c149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 >> 172.21.15.104:0/3087222309 conn(0x7f387c0ceec0 msgr2=0x7f387c1557d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 shutdown_connections 2023-06-10T18:59:34.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.084+0000 7f388473c700 1 -- 172.21.15.104:0/3087222309 wait complete. 2023-06-10T18:59:34.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:59:34.224 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.218+0000 7fd02c83e700 1 Processor -- start 2023-06-10T18:59:34.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.218+0000 7fd02c83e700 1 -- start start 2023-06-10T18:59:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02c83e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd0280cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.227 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02c83e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0280cf200 0x7fd028069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02c83e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02815c8d0 con 0x7fd02806a3b0 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02c83e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0280aef20 con 0x7fd0280cf200 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02c83e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02806abd0 con 0x7fd02815c500 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd026d9d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd02806a3b0 0x7fd02806dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39630/0 (socket says 172.21.15.104:39630) 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd026d9d700 1 -- 172.21.15.104:0/2472242847 learned_addr learned my addr 172.21.15.104:0/2472242847 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd025d9b700 1 --2- 172.21.15.104:0/2472242847 >> v2:172.21.15.114:3300/0 conn(0x7fd0280cf200 0x7fd028069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.228 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501962803 0 0) 0x7fd02815c8d0 con 0x7fd02806a3b0 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02659c700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd0280cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.219+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0100035b0 con 0x7fd02806a3b0 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855667385 0 0) 0x7fd0100035b0 con 0x7fd02806a3b0 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 msgr2=0x7fd0280cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd0280cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 >> v2:172.21.15.114:3300/0 conn(0x7fd0280cf200 msgr2=0x7fd028069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.229 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 --2- 172.21.15.104:0/2472242847 >> v2:172.21.15.114:3300/0 conn(0x7fd0280cf200 0x7fd028069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02806a8f0 con 0x7fd02806a3b0 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd0200019f0 con 0x7fd02806a3b0 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 >> v1:172.21.15.104:6789/0 conn(0x7fd02806a3b0 legacy=0x7fd02806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd010003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd010003f80 0x7fd010004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd010005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02806abd0 con 0x7fd010004d90 2023-06-10T18:59:34.230 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01000ba10 con 0x7fd010003f80 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01000bb50 con 0x7fd02815c500 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd010005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd02659c700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd010003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd025d9b700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd010003f80 0x7fd010004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 -- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 msgr2=0x7fd010003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd02815c500 0x7fd010003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.231 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 -- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd010003f80 msgr2=0x7fd010004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd010003f80 0x7fd010004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 -- 172.21.15.104:0/2472242847 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0100035b0 con 0x7fd010004d90 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.220+0000 7fd026d9d700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd010005200 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fd02000a8d0 tx=0x7fd010005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404968052573e8fc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02000a340 con 0x7fd010004d90 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd020012600 con 0x7fd010004d90 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02559a700 1 -- 172.21.15.104:0/2472242847 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0200129b0 con 0x7fd010004d90 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 -- 172.21.15.104:0/2472242847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 msgr2=0x7fd010005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 --2- 172.21.15.104:0/2472242847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd010005200 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fd02000a8d0 tx=0x7fd010005880 comp rx=0 tx=0).stop 2023-06-10T18:59:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd026d9d700 1 -- 172.21.15.104:0/2472242847 reap_dead start 2023-06-10T18:59:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 -- 172.21.15.104:0/2472242847 shutdown_connections 2023-06-10T18:59:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 -- 172.21.15.104:0/2472242847 >> 172.21.15.104:0/2472242847 conn(0x7fd02806b8a0 msgr2=0x7fd0280cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 -- 172.21.15.104:0/2472242847 shutdown_connections 2023-06-10T18:59:34.233 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.221+0000 7fd02c83e700 1 -- 172.21.15.104:0/2472242847 wait complete. 2023-06-10T18:59:34.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 Processor -- start 2023-06-10T18:59:34.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 -- start start 2023-06-10T18:59:34.235 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd028149e20 0x7fd02814a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0280b3490 0x7fd0280b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028148380 con 0x7fd010004d90 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028147d90 con 0x7fd0280b3490 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02c83e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd028147b20 con 0x7fd028149e20 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.236 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41810/0 (socket says 172.21.15.104:41810) 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.223+0000 7fd02659c700 1 -- 172.21.15.104:0/3320113336 learned_addr learned my addr 172.21.15.104:0/3320113336 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 -- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd028149e20 msgr2=0x7fd02814a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd028149e20 0x7fd02814a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd026d9d700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0280b3490 0x7fd0280b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 -- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0280b3490 msgr2=0x7fd0280b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0280b3490 0x7fd0280b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 -- 172.21.15.104:0/3320113336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02814a8c0 con 0x7fd010004d90 2023-06-10T18:59:34.237 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02659c700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0119f0 tx=0x7fd01c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c005c30 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c005d90 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02814aba0 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00f8f0 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.224+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02815a0c0 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.225+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02814adc0 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.225+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd01c00fa50 con 0x7fd010004d90 2023-06-10T18:59:34.238 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.226+0000 7fd016ffd700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 0x7fd00806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.226+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd01c089da0 con 0x7fd010004d90 2023-06-10T18:59:34.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.226+0000 7fd025d9b700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 0x7fd00806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.227+0000 7fd025d9b700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 0x7fd00806d190 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd018006ba0 tx=0x7fd018006b30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.239 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.230+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd01c054d10 con 0x7fd010004d90 2023-06-10T18:59:34.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.415+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0280c9850 con 0x7fd010004d90 2023-06-10T18:59:34.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.415+0000 7fd016ffd700 1 -- 172.21.15.104:0/3320113336 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v187) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd01c058330 con 0x7fd010004d90 2023-06-10T18:59:34.422 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 msgr2=0x7fd00806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 0x7fd00806d190 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd018006ba0 tx=0x7fd018006b30 comp rx=0 tx=0).stop 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 msgr2=0x7fd0281475e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0119f0 tx=0x7fd01c011db0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 shutdown_connections 2023-06-10T18:59:34.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd00806ace0 0x7fd00806d190 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0280b3490 0x7fd0280b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd010004d90 0x7fd0281475e0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 --2- 172.21.15.104:0/3320113336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd028149e20 0x7fd02814a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 >> 172.21.15.104:0/3320113336 conn(0x7fd02806b8a0 msgr2=0x7fd02815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 shutdown_connections 2023-06-10T18:59:34.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.417+0000 7fd02c83e700 1 -- 172.21.15.104:0/3320113336 wait complete. 2023-06-10T18:59:34.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:59:34.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 Processor -- start 2023-06-10T18:59:34.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 -- start start 2023-06-10T18:59:34.567 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05e80ca870 0x7f05e80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05e815ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e80ccea0 con 0x7f05e80ce880 2023-06-10T18:59:34.568 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.552+0000 7f05ed64f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e80cf850 con 0x7f05e80ca870 2023-06-10T18:59:34.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05ed64f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80cfb30 con 0x7f05e806d0c0 2023-06-10T18:59:34.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e67fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f05e80ce880 0x7f05e806c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39646/0 (socket says 172.21.15.104:39646) 2023-06-10T18:59:34.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e67fc700 1 -- 172.21.15.104:0/4219387323 learned_addr learned my addr 172.21.15.104:0/4219387323 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:34.569 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e6ffd700 1 --2- 172.21.15.104:0/4219387323 >> v2:172.21.15.114:3300/0 conn(0x7f05e80ca870 0x7f05e80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e77fe700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05e815ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610422880 0 0) 0x7f05e80ccea0 con 0x7f05e80ce880 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d00035b0 con 0x7f05e80ce880 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861031563 0 0) 0x7f05d00035b0 con 0x7f05e80ce880 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 msgr2=0x7f05e815ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.570 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05e815ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 >> v2:172.21.15.114:3300/0 conn(0x7f05e80ca870 msgr2=0x7f05e80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 --2- 172.21.15.104:0/4219387323 >> v2:172.21.15.114:3300/0 conn(0x7f05e80ca870 0x7f05e80cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e8071460 con 0x7f05e80ce880 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f05dc001670 con 0x7f05e80ce880 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 >> v1:172.21.15.104:6789/0 conn(0x7f05e80ce880 legacy=0x7f05e806c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d0003a00 0x7f05d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.571 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0004d90 0x7f05d0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80cf850 con 0x7f05d0004d90 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80cfb30 con 0x7f05d0003a00 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d000ba10 con 0x7f05e806d0c0 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e77fe700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0004d90 0x7f05d0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e67fc700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.572 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.553+0000 7f05e6ffd700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d0003a00 0x7f05d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 -- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 msgr2=0x7f05d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e806d0c0 0x7f05d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 -- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d0003a00 msgr2=0x7f05d0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05d0003a00 0x7f05d0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 -- 172.21.15.104:0/4219387323 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d00035b0 con 0x7f05d0004d90 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0004d90 0x7f05d0005200 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009fc0 tx=0x7f05d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137b0c1618c115fe server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e000eb70 con 0x7f05d0004d90 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e000f040 con 0x7f05d0004d90 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e5ffb700 1 -- 172.21.15.104:0/4219387323 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0014480 con 0x7f05d0004d90 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05ed64f700 1 -- 172.21.15.104:0/4219387323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0004d90 msgr2=0x7f05d0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05ed64f700 1 --2- 172.21.15.104:0/4219387323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d0004d90 0x7f05d0005200 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009fc0 tx=0x7f05d0005880 comp rx=0 tx=0).stop 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.554+0000 7f05e77fe700 1 -- 172.21.15.104:0/4219387323 reap_dead start 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.555+0000 7f05ed64f700 1 -- 172.21.15.104:0/4219387323 shutdown_connections 2023-06-10T18:59:34.574 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.555+0000 7f05ed64f700 1 -- 172.21.15.104:0/4219387323 >> 172.21.15.104:0/4219387323 conn(0x7f05e80b1570 msgr2=0x7f05e80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.555+0000 7f05ed64f700 1 -- 172.21.15.104:0/4219387323 shutdown_connections 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.555+0000 7f05ed64f700 1 -- 172.21.15.104:0/4219387323 wait complete. 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.556+0000 7f05ed64f700 1 Processor -- start 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.556+0000 7f05ed64f700 1 -- start start 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e8148410 0x7f05e815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e81487e0 0x7f05e8148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.575 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80c4320 con 0x7f05e80c3970 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e815cf70 con 0x7f05e8148410 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e815c0c0 con 0x7f05e81487e0 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41834/0 (socket says 172.21.15.104:41834) 2023-06-10T18:59:34.576 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 -- 172.21.15.104:0/2897804115 learned_addr learned my addr 172.21.15.104:0/2897804115 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e6ffd700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e8148410 0x7f05e815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e67fc700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e81487e0 0x7f05e8148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 -- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e81487e0 msgr2=0x7f05e8148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e81487e0 0x7f05e8148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 -- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e8148410 msgr2=0x7f05e815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e8148410 0x7f05e815bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.577 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 -- 172.21.15.104:0/2897804115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e815c360 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05e77fe700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f05d0004760 tx=0x7f05e0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0007000 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0022070 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e815c640 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0003a20 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.557+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e815cc00 con 0x7f05e80c3970 2023-06-10T18:59:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.558+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e815d120 con 0x7f05e80c3970 2023-06-10T18:59:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.559+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f05e0003b80 con 0x7f05e80c3970 2023-06-10T18:59:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.559+0000 7f05cf7fe700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 0x7f05d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.559+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f05e0024070 con 0x7f05e80c3970 2023-06-10T18:59:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.559+0000 7f05e6ffd700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 0x7f05d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.560+0000 7f05e6ffd700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 0x7f05d406d190 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009ba0 tx=0x7f05d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.580 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.563+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05e815d120 con 0x7f05e80c3970 2023-06-10T18:59:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.750+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05e815d120 con 0x7f05e80c3970 2023-06-10T18:59:34.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.751+0000 7f05cf7fe700 1 -- 172.21.15.104:0/2897804115 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v187) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05e815d120 con 0x7f05e80c3970 2023-06-10T18:59:34.758 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:59:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.752+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 msgr2=0x7f05d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.752+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 0x7f05d406d190 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009ba0 tx=0x7f05d8008040 comp rx=0 tx=0).stop 2023-06-10T18:59:34.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.752+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 msgr2=0x7f05e80c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.752+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f05d0004760 tx=0x7f05e0009750 comp rx=0 tx=0).stop 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 shutdown_connections 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f05d406ace0 0x7f05d406d190 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05e8148410 0x7f05e815bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05e80c3970 0x7f05e80c3de0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 --2- 172.21.15.104:0/2897804115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05e81487e0 0x7f05e8148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 >> 172.21.15.104:0/2897804115 conn(0x7f05e80b1570 msgr2=0x7f05e80ac160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 shutdown_connections 2023-06-10T18:59:34.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.753+0000 7f05ed64f700 1 -- 172.21.15.104:0/2897804115 wait complete. 2023-06-10T18:59:34.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:59:34.890 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.884+0000 7f4a8c2d7700 1 Processor -- start 2023-06-10T18:59:34.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.884+0000 7f4a8c2d7700 1 -- start start 2023-06-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8c2d7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a840cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8c2d7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a840ce040 0x7f4a8406e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.895 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8c2d7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8406dcb0 con 0x7f4a84069e70 2023-06-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8c2d7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a840ce470 con 0x7f4a840ce040 2023-06-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8c2d7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8406d0a0 con 0x7f4a8415c500 2023-06-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8a874700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4a84069e70 0x7f4a840c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39666/0 (socket says 172.21.15.104:39666) 2023-06-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8a874700 1 -- 172.21.15.104:0/2615039704 learned_addr learned my addr 172.21.15.104:0/2615039704 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:34.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620154430 0 0) 0x7f4a8406dcb0 con 0x7f4a84069e70 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a89872700 1 --2- 172.21.15.104:0/2615039704 >> v2:172.21.15.114:3300/0 conn(0x7f4a840ce040 0x7f4a8406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a8a073700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a840cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.885+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a700035b0 con 0x7f4a84069e70 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662402116 0 0) 0x7f4a700035b0 con 0x7f4a84069e70 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 msgr2=0x7f4a840cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a840cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.897 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 >> v2:172.21.15.114:3300/0 conn(0x7f4a840ce040 msgr2=0x7f4a8406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 --2- 172.21.15.104:0/2615039704 >> v2:172.21.15.114:3300/0 conn(0x7f4a840ce040 0x7f4a8406e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8406d380 con 0x7f4a84069e70 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f4a7c0019f0 con 0x7f4a84069e70 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 >> v1:172.21.15.104:6789/0 conn(0x7f4a84069e70 legacy=0x7f4a840c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a70003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a70003f80 0x7f4a70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.898 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a70005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840ce470 con 0x7f4a70004d90 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8406d0a0 con 0x7f4a70003f80 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7000ba40 con 0x7f4a8415c500 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a70005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a073700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a70003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89872700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a70003f80 0x7f4a70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.899 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 -- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 msgr2=0x7f4a70003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8415c500 0x7f4a70003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 -- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a70003f80 msgr2=0x7f4a70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a70003f80 0x7f4a70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 -- 172.21.15.104:0/2615039704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700035b0 con 0x7f4a70004d90 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89872700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a70003f80 0x7f4a70004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a8a874700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a70005200 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00a860 tx=0x7f4a70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf6580c7a479c43 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.900 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c012600 con 0x7f4a70004d90 2023-06-10T18:59:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.886+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c012780 con 0x7f4a70004d90 2023-06-10T18:59:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a89071700 1 -- 172.21.15.104:0/2615039704 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c012b30 con 0x7f4a70004d90 2023-06-10T18:59:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/2615039704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 msgr2=0x7f4a70005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.901 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/2615039704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a70005200 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00a860 tx=0x7f4a70005880 comp rx=0 tx=0).stop 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8a874700 1 -- 172.21.15.104:0/2615039704 reap_dead start 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/2615039704 shutdown_connections 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/2615039704 >> 172.21.15.104:0/2615039704 conn(0x7f4a8406b8a0 msgr2=0x7f4a840cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/2615039704 shutdown_connections 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.887+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/2615039704 wait complete. 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.888+0000 7f4a8c2d7700 1 Processor -- start 2023-06-10T18:59:34.902 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.888+0000 7f4a8c2d7700 1 -- start start 2023-06-10T18:59:34.904 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a840b4230 0x7f4a841f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a840b46a0 0x7f4a841f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a841f1d40 con 0x7f4a70004d90 2023-06-10T18:59:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a841f5e80 con 0x7f4a840b46a0 2023-06-10T18:59:34.905 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8c2d7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a841f5ba0 con 0x7f4a840b4230 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41850/0 (socket says 172.21.15.104:41850) 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 -- 172.21.15.104:0/3493125473 learned_addr learned my addr 172.21.15.104:0/3493125473 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a89872700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a840b4230 0x7f4a841f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a874700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a840b46a0 0x7f4a841f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 -- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a840b4230 msgr2=0x7f4a841f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.906 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a840b4230 0x7f4a841f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 -- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a840b46a0 msgr2=0x7f4a841f5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a840b46a0 0x7f4a841f5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.889+0000 7f4a8a073700 1 -- 172.21.15.104:0/3493125473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a841e5b20 con 0x7f4a70004d90 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a8a073700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f4a740119f0 tx=0x7f4a74011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74005c30 con 0x7f4a70004d90 2023-06-10T18:59:34.907 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a74005d90 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a841e5e00 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7400f950 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.890+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a841e6b40 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.891+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a841e6320 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.891+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4a7400fab0 con 0x7f4a70004d90 2023-06-10T18:59:34.908 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.892+0000 7f4a7affd700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 0x7f4a6806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:34.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.892+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4a74089d40 con 0x7f4a70004d90 2023-06-10T18:59:34.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.892+0000 7f4a89872700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 0x7f4a6806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:34.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.892+0000 7f4a89872700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 0x7f4a6806d1c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4a841e2970 tx=0x7f4a80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:34.909 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:34.896+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4a841e6320 con 0x7f4a70004d90 2023-06-10T18:59:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.097+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4a841e6320 con 0x7f4a70004d90 2023-06-10T18:59:35.104 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.098+0000 7f4a7affd700 1 -- 172.21.15.104:0/3493125473 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v187) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4a841e6320 con 0x7f4a70004d90 2023-06-10T18:59:35.106 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 msgr2=0x7f4a6806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 0x7f4a6806d1c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4a841e2970 tx=0x7f4a80008040 comp rx=0 tx=0).stop 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 msgr2=0x7f4a841e6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f4a740119f0 tx=0x7f4a74011db0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 shutdown_connections 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4a6806ad10 0x7f4a6806d1c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.107 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a840b46a0 0x7f4a841f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a70004d90 0x7f4a841e6600 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 --2- 172.21.15.104:0/3493125473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a840b4230 0x7f4a841f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 >> 172.21.15.104:0/3493125473 conn(0x7f4a8406b8a0 msgr2=0x7f4a841e15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 shutdown_connections 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.100+0000 7f4a8c2d7700 1 -- 172.21.15.104:0/3493125473 wait complete. 2023-06-10T18:59:35.108 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:59:35.118 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T18:59:35.118 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T18:59:35.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T18:59:35.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.290+0000 7f1f7acaa700 1 Processor -- start 2023-06-10T18:59:35.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.290+0000 7f1f7acaa700 1 -- start start 2023-06-10T18:59:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f7acaa700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1f74070a20 0x7f1f7406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f7acaa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 0x7f1f741587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f7acaa700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7406ae00 con 0x7f1f74158ce0 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f7acaa700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f74069e70 con 0x7f1f74070a20 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f7acaa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f741595f0 con 0x7f1f74070df0 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f78a46700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f1f74070a20 0x7f1f7406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f79247700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1f74158ce0 0x7f1f74159c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39690/0 (socket says 172.21.15.104:39690) 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.291+0000 7f1f79247700 1 -- 172.21.15.104:0/3710217746 learned_addr learned my addr 172.21.15.104:0/3710217746 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f73fff700 1 --2- 172.21.15.104:0/3710217746 >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 0x7f1f741587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f78a46700 1 -- 172.21.15.104:0/3710217746 >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 msgr2=0x7f1f741587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f78a46700 1 --2- 172.21.15.104:0/3710217746 >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 0x7f1f741587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f78a46700 1 -- 172.21.15.104:0/3710217746 >> v1:172.21.15.104:6789/0 conn(0x7f1f74158ce0 legacy=0x7f1f74159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f737fe700 1 -- 172.21.15.104:0/3710217746 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680627300 0 0) 0x7f1f6c001490 con 0x7f1f74158ce0 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f78a46700 1 -- 172.21.15.104:0/3710217746 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f741598d0 con 0x7f1f74070a20 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f73fff700 1 --2- 172.21.15.104:0/3710217746 >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 0x7f1f741587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:35.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f78a46700 1 --2- 172.21.15.104:0/3710217746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f7406a8c0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f1f7406df00 tx=0x7f1f68009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d6fa30102e4060 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f737fe700 1 -- 172.21.15.104:0/3710217746 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68011830 con 0x7f1f74070a20 2023-06-10T18:59:35.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f737fe700 1 -- 172.21.15.104:0/3710217746 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68011990 con 0x7f1f74070a20 2023-06-10T18:59:35.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f737fe700 1 -- 172.21.15.104:0/3710217746 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68011d20 con 0x7f1f74070a20 2023-06-10T18:59:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 -- 172.21.15.104:0/3710217746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 msgr2=0x7f1f7406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/3710217746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f7406a8c0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f1f7406df00 tx=0x7f1f68009b20 comp rx=0 tx=0).stop 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 -- 172.21.15.104:0/3710217746 shutdown_connections 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/3710217746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f7406a8c0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/3710217746 >> v2:172.21.15.114:3300/0 conn(0x7f1f74070df0 0x7f1f741587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.292+0000 7f1f7acaa700 1 -- 172.21.15.104:0/3710217746 >> 172.21.15.104:0/3710217746 conn(0x7f1f740b1610 msgr2=0x7f1f740ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.293+0000 7f1f7acaa700 1 -- 172.21.15.104:0/3710217746 shutdown_connections 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.293+0000 7f1f7acaa700 1 -- 172.21.15.104:0/3710217746 wait complete. 2023-06-10T18:59:35.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.294+0000 7f1f7acaa700 1 Processor -- start 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.294+0000 7f1f7acaa700 1 -- start start 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f74070df0 0x7f1f741f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f74158ce0 0x7f1f740b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f741e3c20 con 0x7f1f74070a20 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f741f3fd0 con 0x7f1f74158ce0 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740c38d0 con 0x7f1f74070df0 2023-06-10T18:59:35.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41864/0 (socket says 172.21.15.104:41864) 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 -- 172.21.15.104:0/195040263 learned_addr learned my addr 172.21.15.104:0/195040263 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f79247700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f74158ce0 0x7f1f740b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f73fff700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f74070df0 0x7f1f741f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 -- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f74070df0 msgr2=0x7f1f741f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f74070df0 0x7f1f741f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 -- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f74158ce0 msgr2=0x7f1f740b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f74158ce0 0x7f1f740b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 -- 172.21.15.104:0/195040263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f680094a0 con 0x7f1f74070a20 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f78a46700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f1f68014040 tx=0x7f1f68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680038a0 con 0x7f1f74070a20 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6801e070 con 0x7f1f74070a20 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f740c3bb0 con 0x7f1f74070a20 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68003c40 con 0x7f1f74070a20 2023-06-10T18:59:35.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.295+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f740c4170 con 0x7f1f74070a20 2023-06-10T18:59:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.296+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f74074050 con 0x7f1f74070a20 2023-06-10T18:59:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.297+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1f6802d450 con 0x7f1f74070a20 2023-06-10T18:59:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.297+0000 7f1f717fa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 0x7f1f5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.297+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1f6808a760 con 0x7f1f74070a20 2023-06-10T18:59:35.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.297+0000 7f1f73fff700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 0x7f1f5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.298+0000 7f1f73fff700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 0x7f1f5806d190 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1f74156950 tx=0x7f1f60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.301+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1f68055760 con 0x7f1f74070a20 2023-06-10T18:59:35.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.488+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f74065de0 con 0x7f1f74070a20 2023-06-10T18:59:35.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.489+0000 7f1f717fa700 1 -- 172.21.15.104:0/195040263 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+18618 (secure 0 0 0) 0x7f1f68058d80 con 0x7f1f74070a20 2023-06-10T18:59:35.496 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:35.496 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":187,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:24.248976+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T18:59:35.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 msgr2=0x7f1f5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 0x7f1f5806d190 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1f74156950 tx=0x7f1f60008040 comp rx=0 tx=0).stop 2023-06-10T18:59:35.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 msgr2=0x7f1f741e3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f1f68014040 tx=0x7f1f68004750 comp rx=0 tx=0).stop 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 shutdown_connections 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f1f5806ace0 0x7f1f5806d190 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f74158ce0 0x7f1f740b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f74070a20 0x7f1f741e3590 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 --2- 172.21.15.104:0/195040263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f74070df0 0x7f1f741f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 >> 172.21.15.104:0/195040263 conn(0x7f1f740b1610 msgr2=0x7f1f740cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 shutdown_connections 2023-06-10T18:59:35.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.492+0000 7f1f7acaa700 1 -- 172.21.15.104:0/195040263 wait complete. 2023-06-10T18:59:35.510 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:59:21.300558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:59:22.213068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T18:59:35.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T18:59:35.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.627+0000 7fae1c19a700 1 Processor -- start 2023-06-10T18:59:35.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.627+0000 7fae1c19a700 1 -- start start 2023-06-10T18:59:35.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae1c19a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae140ccff0 0x7fae140ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae1c19a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140ce340 con 0x7fae140d0d20 2023-06-10T18:59:35.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae1c19a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140d00d0 con 0x7fae1406af80 2023-06-10T18:59:35.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae1c19a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae140cd470 con 0x7fae140ccff0 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae19735700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fae140d0d20 0x7fae140ce7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39702/0 (socket says 172.21.15.104:39702) 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.628+0000 7fae19735700 1 -- 172.21.15.104:0/2612301214 learned_addr learned my addr 172.21.15.104:0/2612301214 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229675981 0 0) 0x7fae140ce340 con 0x7fae140d0d20 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae000035b0 con 0x7fae140d0d20 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 --2- 172.21.15.104:0/2612301214 >> v2:172.21.15.114:3300/0 conn(0x7fae140ccff0 0x7fae140ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 180882086 0 0) 0x7fae000035b0 con 0x7fae140d0d20 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 >> v2:172.21.15.114:3300/0 conn(0x7fae140ccff0 msgr2=0x7fae140ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:35.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 --2- 172.21.15.104:0/2612301214 >> v2:172.21.15.114:3300/0 conn(0x7fae140ccff0 0x7fae140ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 >> v1:172.21.15.114:6789/0 conn(0x7fae1406af80 legacy=0x7fae1406b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae140cd680 con 0x7fae140d0d20 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fae10001670 con 0x7fae140d0d20 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 >> v1:172.21.15.104:6789/0 conn(0x7fae140d0d20 legacy=0x7fae140ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae00003a00 0x7fae00003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00004850 0x7fae00004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 0x7fae0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140cd470 con 0x7fae00005200 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000c080 con 0x7fae00004850 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000c240 con 0x7fae00003a00 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 0x7fae0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae19735700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00004850 0x7fae00004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae19f36700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae00003a00 0x7fae00003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 -- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae00003a00 msgr2=0x7fae00003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:35.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae00003a00 0x7fae00003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 -- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00004850 msgr2=0x7fae00004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00004850 0x7fae00004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.629+0000 7fae1a737700 1 -- 172.21.15.104:0/2612301214 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000035b0 con 0x7fae00005200 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1a737700 1 -- 172.21.15.104:0/2612301214 reap_dead start 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1a737700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 0x7fae0000ba10 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009fc0 tx=0x7fae00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e668a614284b0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003a20 con 0x7fae00005200 2023-06-10T18:59:35.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c00f040 con 0x7fae00005200 2023-06-10T18:59:35.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae18f34700 1 -- 172.21.15.104:0/2612301214 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003de0 con 0x7fae00005200 2023-06-10T18:59:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 -- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 msgr2=0x7fae0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 0x7fae0000ba10 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009fc0 tx=0x7fae00005880 comp rx=0 tx=0).stop 2023-06-10T18:59:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 -- 172.21.15.104:0/2612301214 shutdown_connections 2023-06-10T18:59:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 --2- 172.21.15.104:0/2612301214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae00005200 0x7fae0000ba10 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 -- 172.21.15.104:0/2612301214 >> 172.21.15.104:0/2612301214 conn(0x7fae14089fd0 msgr2=0x7fae1408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 -- 172.21.15.104:0/2612301214 shutdown_connections 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.630+0000 7fae1c19a700 1 -- 172.21.15.104:0/2612301214 wait complete. 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 Processor -- start 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 -- start start 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00005200 0x7fae1415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae140b2210 0x7fae14158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140ae550 con 0x7fae140adb90 2023-06-10T18:59:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae14159c50 con 0x7fae00005200 2023-06-10T18:59:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1c19a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae141590d0 con 0x7fae140b2210 2023-06-10T18:59:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1a737700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1a737700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41888/0 (socket says 172.21.15.104:41888) 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae1a737700 1 -- 172.21.15.104:0/3163102034 learned_addr learned my addr 172.21.15.104:0/3163102034 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae19735700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae140b2210 0x7fae14158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.632+0000 7fae19f36700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00005200 0x7fae1415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 -- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae140b2210 msgr2=0x7fae14158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae140b2210 0x7fae14158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 -- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00005200 msgr2=0x7fae1415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00005200 0x7fae1415ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 -- 172.21.15.104:0/3163102034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae141593b0 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1a737700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fae14070bb0 tx=0x7fae0c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003e10 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c004230 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c0045e0 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae14159690 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.633+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae14158dc0 con 0x7fae140adb90 2023-06-10T18:59:35.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.634+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1408a860 con 0x7fae140adb90 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.634+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fae0c004740 con 0x7fae140adb90 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.635+0000 7fae0a7fc700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 0x7fadf806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.635+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fae0c08a9e0 con 0x7fae140adb90 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.635+0000 7fae19f36700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 0x7fadf806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.635+0000 7fae19f36700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 0x7fadf806d190 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fae140aee90 tx=0x7fae04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.639+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fae0c055a80 con 0x7fae140adb90 2023-06-10T18:59:35.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.825+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae140b3060 con 0x7fae140adb90 2023-06-10T18:59:35.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.825+0000 7fae0a7fc700 1 -- 172.21.15.104:0/3163102034 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae0c0180a0 con 0x7fae140adb90 2023-06-10T18:59:35.832 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 msgr2=0x7fadf806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 0x7fadf806d190 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fae140aee90 tx=0x7fae04009040 comp rx=0 tx=0).stop 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 msgr2=0x7fae140ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fae14070bb0 tx=0x7fae0c009750 comp rx=0 tx=0).stop 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 shutdown_connections 2023-06-10T18:59:35.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fadf806ace0 0x7fadf806d190 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae00005200 0x7fae1415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae140adb90 0x7fae140ae010 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 --2- 172.21.15.104:0/3163102034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae140b2210 0x7fae14158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 >> 172.21.15.104:0/3163102034 conn(0x7fae14089fd0 msgr2=0x7fae1406ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 shutdown_connections 2023-06-10T18:59:35.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.827+0000 7fae1c19a700 1 -- 172.21.15.104:0/3163102034 wait complete. 2023-06-10T18:59:35.845 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T18:59:35.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.961+0000 7f6a6fefd700 1 Processor -- start 2023-06-10T18:59:35.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.961+0000 7f6a6fefd700 1 -- start start 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6fefd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 0x7f6a680b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6fefd700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6806e250 con 0x7f6a6806e6e0 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6fefd700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6815c950 con 0x7f6a6815c580 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6fefd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b1e80 con 0x7f6a6806cde0 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6d498700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6a6806e6e0 0x7f6a680ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39722/0 (socket says 172.21.15.104:39722) 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6d498700 1 -- 172.21.15.104:0/1806896713 learned_addr learned my addr 172.21.15.104:0/1806896713 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:35.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.962+0000 7f6a6e49a700 1 --2- 172.21.15.104:0/1806896713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 0x7f6a680b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145432097 0 0) 0x7f6a6806e250 con 0x7f6a6806e6e0 2023-06-10T18:59:35.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a540035b0 con 0x7f6a6806e6e0 2023-06-10T18:59:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619437147 0 0) 0x7f6a540035b0 con 0x7f6a6806e6e0 2023-06-10T18:59:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6e49a700 1 -- 172.21.15.104:0/1806896713 >> v1:172.21.15.114:6789/0 conn(0x7f6a6815c580 legacy=0x7f6a680b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6e49a700 1 -- 172.21.15.104:0/1806896713 >> v1:172.21.15.104:6789/0 conn(0x7f6a6806e6e0 legacy=0x7f6a680ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6e49a700 1 -- 172.21.15.104:0/1806896713 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6806d920 con 0x7f6a6806cde0 2023-06-10T18:59:35.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6e49a700 1 --2- 172.21.15.104:0/1806896713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 0x7f6a680b1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009dc0 tx=0x7f6a6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c4036578684098 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6000e070 con 0x7f6a6806cde0 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60003a20 con 0x7f6a6806cde0 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.963+0000 7f6a6cc97700 1 -- 172.21.15.104:0/1806896713 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58001670 con 0x7f6a6806cde0 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 -- 172.21.15.104:0/1806896713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 msgr2=0x7f6a680b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/1806896713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 0x7f6a680b1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009dc0 tx=0x7f6a6000b330 comp rx=0 tx=0).stop 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 -- 172.21.15.104:0/1806896713 shutdown_connections 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/1806896713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6806cde0 0x7f6a680b1780 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 -- 172.21.15.104:0/1806896713 >> 172.21.15.104:0/1806896713 conn(0x7f6a6806b8a0 msgr2=0x7f6a680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 -- 172.21.15.104:0/1806896713 shutdown_connections 2023-06-10T18:59:35.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.964+0000 7f6a6fefd700 1 -- 172.21.15.104:0/1806896713 wait complete. 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.965+0000 7f6a6fefd700 1 Processor -- start 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.965+0000 7f6a6fefd700 1 -- start start 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a6806e6e0 0x7f6a680b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6815c580 0x7f6a68151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b1020 con 0x7f6a6806cde0 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681e27e0 con 0x7f6a6806e6e0 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681e2ac0 con 0x7f6a6815c580 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41898/0 (socket says 172.21.15.104:41898) 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 -- 172.21.15.104:0/2165962145 learned_addr learned my addr 172.21.15.104:0/2165962145 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6e49a700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6815c580 0x7f6a68151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6d498700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a6806e6e0 0x7f6a680b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 -- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6815c580 msgr2=0x7f6a68151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6815c580 0x7f6a68151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 -- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a6806e6e0 msgr2=0x7f6a680b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a6806e6e0 0x7f6a680b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 -- 172.21.15.104:0/2165962145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60009a70 con 0x7f6a6806cde0 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6dc99700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c500 tx=0x7f6a6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640054e0 con 0x7f6a6806cde0 2023-06-10T18:59:35.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640036d0 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a681e2da0 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.966+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64003a80 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.967+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681557e0 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.968+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a68074050 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.968+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a64003be0 con 0x7f6a6806cde0 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.968+0000 7f6a5e7fc700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 0x7f6a4c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:35.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.968+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6a64088a00 con 0x7f6a6806cde0 2023-06-10T18:59:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.969+0000 7f6a6d498700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 0x7f6a4c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.969+0000 7f6a6d498700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 0x7f6a4c06d190 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6a680cd5e0 tx=0x7f6a58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:35.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:35.973+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a64053a00 con 0x7f6a6806cde0 2023-06-10T18:59:36.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.156+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6a680d0300 con 0x7f6a6806cde0 2023-06-10T18:59:36.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.156+0000 7f6a5e7fc700 1 -- 172.21.15.104:0/2165962145 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v187) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6a640170a0 con 0x7f6a6806cde0 2023-06-10T18:59:36.163 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 msgr2=0x7f6a4c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 0x7f6a4c06d190 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6a680cd5e0 tx=0x7f6a58009040 comp rx=0 tx=0).stop 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 msgr2=0x7f6a681e2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c500 tx=0x7f6a6400c8c0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 shutdown_connections 2023-06-10T18:59:36.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a6815c580 0x7f6a68151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a6806cde0 0x7f6a681e2200 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f6a4c06ace0 0x7f6a4c06d190 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 --2- 172.21.15.104:0/2165962145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a6806e6e0 0x7f6a680b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 >> 172.21.15.104:0/2165962145 conn(0x7f6a6806b8a0 msgr2=0x7f6a6815a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 shutdown_connections 2023-06-10T18:59:36.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.158+0000 7f6a6fefd700 1 -- 172.21.15.104:0/2165962145 wait complete. 2023-06-10T18:59:36.176 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T18:59:36.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.293+0000 7f0392a22700 1 Processor -- start 2023-06-10T18:59:36.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.293+0000 7f0392a22700 1 -- start start 2023-06-10T18:59:36.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f0392a22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f038c06d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f0392a22700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038c0ae7e0 con 0x7f038c15c6f0 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f0392a22700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038c06df90 con 0x7f038c0b1780 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f0392a22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c06e270 con 0x7f038c06ea70 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f038bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f038c15c6f0 0x7f038c0b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39734/0 (socket says 172.21.15.104:39734) 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f038bfff700 1 -- 172.21.15.104:0/806838272 learned_addr learned my addr 172.21.15.104:0/806838272 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.294+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401707873 0 0) 0x7f038c0ae7e0 con 0x7f038c15c6f0 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03780035b0 con 0x7f038c15c6f0 2023-06-10T18:59:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f0390fbf700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f038c06d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298308699 0 0) 0x7f03780035b0 con 0x7f038c15c6f0 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 msgr2=0x7f038c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f038c06d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 >> v1:172.21.15.114:6789/0 conn(0x7f038c0b1780 legacy=0x7f038c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c06e550 con 0x7f038c15c6f0 2023-06-10T18:59:36.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f03740017f0 con 0x7f038c15c6f0 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f0390fbf700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f038c06d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 >> v1:172.21.15.104:6789/0 conn(0x7f038c15c6f0 legacy=0x7f038c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0378003a30 0x7f0378003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f0378004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f0378005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c06e270 con 0x7f0378004850 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037800bbd0 con 0x7f0378003a30 2023-06-10T18:59:36.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037800bd90 con 0x7f038c06ea70 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f0378004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f0390fbf700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f0378005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038bfff700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0378003a30 0x7f0378003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 -- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 msgr2=0x7f0378005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c06ea70 0x7f0378005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 -- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0378003a30 msgr2=0x7f0378003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0378003a30 0x7f0378003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 -- 172.21.15.104:0/806838272 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03780035b0 con 0x7f0378004850 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.295+0000 7f038b7fe700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f0378004cc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f038c0b1b70 tx=0x7f0378005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79eddacff5ef842 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c0012f0 con 0x7f0378004850 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c0052f0 con 0x7f0378004850 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f038affd700 1 -- 172.21.15.104:0/806838272 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c006860 con 0x7f0378004850 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 -- 172.21.15.104:0/806838272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 msgr2=0x7f0378004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 --2- 172.21.15.104:0/806838272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f0378004cc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f038c0b1b70 tx=0x7f0378005880 comp rx=0 tx=0).stop 2023-06-10T18:59:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0390fbf700 1 -- 172.21.15.104:0/806838272 reap_dead start 2023-06-10T18:59:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 -- 172.21.15.104:0/806838272 shutdown_connections 2023-06-10T18:59:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 -- 172.21.15.104:0/806838272 >> 172.21.15.104:0/806838272 conn(0x7f038c06b8a0 msgr2=0x7f038c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 -- 172.21.15.104:0/806838272 shutdown_connections 2023-06-10T18:59:36.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.296+0000 7f0392a22700 1 -- 172.21.15.104:0/806838272 wait complete. 2023-06-10T18:59:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.297+0000 7f0392a22700 1 Processor -- start 2023-06-10T18:59:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 -- start start 2023-06-10T18:59:36.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f038c0c3d40 0x7f038c0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c1eddc0 0x7f038c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b2f40 con 0x7f0378004850 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c153cd0 con 0x7f038c0c3d40 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0392a22700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b25a0 con 0x7f038c1eddc0 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f038bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f038bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41918/0 (socket says 172.21.15.104:41918) 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f038bfff700 1 -- 172.21.15.104:0/4214722743 learned_addr learned my addr 172.21.15.104:0/4214722743 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.298+0000 7f0390fbf700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c1eddc0 0x7f038c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038b7fe700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f038c0c3d40 0x7f038c0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 -- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c1eddc0 msgr2=0x7f038c0b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c1eddc0 0x7f038c0b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 -- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f038c0c3d40 msgr2=0x7f038c0c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f038c0c3d40 0x7f038c0c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 -- 172.21.15.104:0/4214722743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c0b2880 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f038bfff700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f0374007fd0 tx=0x7f0374001760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740074e0 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0374007640 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374005a70 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c0b2b60 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.299+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c1ee230 con 0x7f0378004850 2023-06-10T18:59:36.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.300+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038c0b3080 con 0x7f0378004850 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.301+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0374018020 con 0x7f0378004850 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.301+0000 7f0388ff9700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 0x7f036c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.301+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f037408b4b0 con 0x7f0378004850 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.302+0000 7f038b7fe700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 0x7f036c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.302+0000 7f038b7fe700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 0x7f036c06d190 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f038c0c4b00 tx=0x7f037c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.305+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f037405a020 con 0x7f0378004850 2023-06-10T18:59:36.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.491+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f038c0b3080 con 0x7f0378004850 2023-06-10T18:59:36.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.491+0000 7f0388ff9700 1 -- 172.21.15.104:0/4214722743 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v187) v1 ==== 104+0+11 (secure 0 0 0) 0x7f038c0b3080 con 0x7f0378004850 2023-06-10T18:59:36.498 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 msgr2=0x7f036c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 0x7f036c06d190 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f038c0c4b00 tx=0x7f037c011000 comp rx=0 tx=0).stop 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 msgr2=0x7f038c153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f0374007fd0 tx=0x7f0374001760 comp rx=0 tx=0).stop 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 shutdown_connections 2023-06-10T18:59:36.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f038c1eddc0 0x7f038c0b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0378004850 0x7f038c153790 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f036c06ace0 0x7f036c06d190 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 --2- 172.21.15.104:0/4214722743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f038c0c3d40 0x7f038c0c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 >> 172.21.15.104:0/4214722743 conn(0x7f038c06b8a0 msgr2=0x7f038c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 shutdown_connections 2023-06-10T18:59:36.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.493+0000 7f0392a22700 1 -- 172.21.15.104:0/4214722743 wait complete. 2023-06-10T18:59:36.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T18:59:36.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.627+0000 7f4d8c3aa700 1 Processor -- start 2023-06-10T18:59:36.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.627+0000 7f4d8c3aa700 1 -- start start 2023-06-10T18:59:36.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.628+0000 7f4d8c3aa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.628+0000 7f4d8c3aa700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4d840d0830 0x7f4d840b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.628+0000 7f4d8c3aa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d840ad3c0 con 0x7f4d84157b30 2023-06-10T18:59:36.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.628+0000 7f4d8c3aa700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d84157f00 con 0x7f4d840d0830 2023-06-10T18:59:36.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.628+0000 7f4d8c3aa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840b0170 con 0x7f4d840cea50 2023-06-10T18:59:36.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89945700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4d840d0830 0x7f4d840b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89945700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:33080/0 (socket says 172.21.15.104:33080) 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89945700 1 -- 172.21.15.104:0/572032111 learned_addr learned my addr 172.21.15.104:0/572032111 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 -- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 msgr2=0x7f4d840d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 -- 172.21.15.104:0/572032111 >> v1:172.21.15.114:6789/0 conn(0x7f4d84157b30 legacy=0x7f4d840ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 -- 172.21.15.104:0/572032111 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840b0410 con 0x7f4d840d0830 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89144700 1 -- 172.21.15.104:0/572032111 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306863066 0 0) 0x7f4d840ad3c0 con 0x7f4d84157b30 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89945700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d8a947700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d840b0ef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009d90 tx=0x7f4d7c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d0f11601c41d40b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89144700 1 -- 172.21.15.104:0/572032111 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00e070 con 0x7f4d840d0830 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89144700 1 -- 172.21.15.104:0/572032111 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c00bb20 con 0x7f4d840d0830 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.629+0000 7f4d89144700 1 -- 172.21.15.104:0/572032111 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c012430 con 0x7f4d840d0830 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 msgr2=0x7f4d840b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d840b0ef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009d90 tx=0x7f4d7c00b330 comp rx=0 tx=0).stop 2023-06-10T18:59:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/572032111 shutdown_connections 2023-06-10T18:59:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d840b0ef0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/572032111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d840cea50 0x7f4d840d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/572032111 >> 172.21.15.104:0/572032111 conn(0x7f4d84089fd0 msgr2=0x7f4d8408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/572032111 shutdown_connections 2023-06-10T18:59:36.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.630+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/572032111 wait complete. 2023-06-10T18:59:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.631+0000 7f4d8c3aa700 1 Processor -- start 2023-06-10T18:59:36.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.631+0000 7f4d8c3aa700 1 -- start start 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d8415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d84157b30 0x7f4d8415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840b65b0 con 0x7f4d840cea50 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d841466e0 con 0x7f4d840d0830 2023-06-10T18:59:36.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8c3aa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840b62d0 con 0x7f4d84157b30 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41932/0 (socket says 172.21.15.104:41932) 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 -- 172.21.15.104:0/4208177831 learned_addr learned my addr 172.21.15.104:0/4208177831 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d89945700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d8415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a947700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d84157b30 0x7f4d8415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 -- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d84157b30 msgr2=0x7f4d8415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d84157b30 0x7f4d8415ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 -- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 msgr2=0x7f4d8415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d8415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.632+0000 7f4d8a146700 1 -- 172.21.15.104:0/4208177831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80009550 con 0x7f4d840cea50 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d8a146700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400c580 tx=0x7f4d7400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d740054e0 con 0x7f4d840cea50 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d74003680 con 0x7f4d840cea50 2023-06-10T18:59:36.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7c009a70 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74003a30 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.633+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d84146bc0 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.634+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d84074050 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.634+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4d74003b90 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.635+0000 7f4d7affd700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 0x7f4d6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.635+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4d74088a70 con 0x7f4d840cea50 2023-06-10T18:59:36.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.635+0000 7f4d89945700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 0x7f4d6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.635+0000 7f4d89945700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 0x7f4d6806d190 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4d84154cb0 tx=0x7f4d80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.639+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4d740539e0 con 0x7f4d840cea50 2023-06-10T18:59:36.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.827+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d8415ce40 con 0x7f4d840cea50 2023-06-10T18:59:36.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.827+0000 7f4d7affd700 1 -- 172.21.15.104:0/4208177831 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v187) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d740170a0 con 0x7f4d840cea50 2023-06-10T18:59:36.834 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T18:59:36.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 msgr2=0x7f4d6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 0x7f4d6806d190 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f4d84154cb0 tx=0x7f4d80008040 comp rx=0 tx=0).stop 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 msgr2=0x7f4d840b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400c580 tx=0x7f4d7400c940 comp rx=0 tx=0).stop 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 shutdown_connections 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4d84157b30 0x7f4d8415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d840cea50 0x7f4d840b3a40 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4d6806ace0 0x7f4d6806d190 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 --2- 172.21.15.104:0/4208177831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d840d0830 0x7f4d8415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 >> 172.21.15.104:0/4208177831 conn(0x7f4d84089fd0 msgr2=0x7f4d840ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 shutdown_connections 2023-06-10T18:59:36.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.829+0000 7f4d8c3aa700 1 -- 172.21.15.104:0/4208177831 wait complete. 2023-06-10T18:59:36.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T18:59:36.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.962+0000 7f12551e9700 1 Processor -- start 2023-06-10T18:59:36.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.962+0000 7f12551e9700 1 -- start start 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f12551e9700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12500aeaa0 0x7f12500cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f12551e9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500cd680 0x7f125006dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f12551e9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12500cdae0 con 0x7f12500b1080 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f12551e9700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12500c9bc0 con 0x7f12500aeaa0 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f12551e9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12500c9e60 con 0x7f12500cd680 2023-06-10T18:59:36.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12500aeaa0 0x7f12500cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f12500aeaa0 0x7f12500cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41940/0 (socket says 172.21.15.104:41940) 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 -- 172.21.15.104:0/3002120106 learned_addr learned my addr 172.21.15.104:0/3002120106 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124f59e700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500cd680 0x7f125006dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 -- 172.21.15.104:0/3002120106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500cd680 msgr2=0x7f125006dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500cd680 0x7f125006dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 -- 172.21.15.104:0/3002120106 >> v1:172.21.15.114:6789/0 conn(0x7f12500b1080 legacy=0x7f12500ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 -- 172.21.15.104:0/3002120106 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500aee70 con 0x7f12500aeaa0 2023-06-10T18:59:36.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124ed9d700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500aeaa0 0x7f12500cd140 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f1244006fd0 tx=0x7f1244009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9264187d09796cd6 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124dd9b700 1 -- 172.21.15.104:0/3002120106 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244005e30 con 0x7f12500aeaa0 2023-06-10T18:59:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.963+0000 7f124dd9b700 1 -- 172.21.15.104:0/3002120106 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1244006030 con 0x7f12500aeaa0 2023-06-10T18:59:36.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f124dd9b700 1 -- 172.21.15.104:0/3002120106 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12440063c0 con 0x7f12500aeaa0 2023-06-10T18:59:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 -- 172.21.15.104:0/3002120106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500aeaa0 msgr2=0x7f12500cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500aeaa0 0x7f12500cd140 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f1244006fd0 tx=0x7f1244009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 -- 172.21.15.104:0/3002120106 shutdown_connections 2023-06-10T18:59:36.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500aeaa0 0x7f12500cd140 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 --2- 172.21.15.104:0/3002120106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500cd680 0x7f125006dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 -- 172.21.15.104:0/3002120106 >> 172.21.15.104:0/3002120106 conn(0x7f125006b8a0 msgr2=0x7f12500cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 -- 172.21.15.104:0/3002120106 shutdown_connections 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.964+0000 7f12551e9700 1 -- 172.21.15.104:0/3002120106 wait complete. 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.965+0000 7f12551e9700 1 Processor -- start 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 -- start start 2023-06-10T18:59:36.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 0x7f12501f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 0x7f125015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12500b3830 con 0x7f12500cd680 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501f1cc0 con 0x7f12500aeaa0 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f12551e9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125015be70 con 0x7f12500b1080 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f124f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f124f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41950/0 (socket says 172.21.15.104:41950) 2023-06-10T18:59:36.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.966+0000 7f124f59e700 1 -- 172.21.15.104:0/920594831 learned_addr learned my addr 172.21.15.104:0/920594831 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124ed9d700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 0x7f12501f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124e59c700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 0x7f125015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 -- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 msgr2=0x7f125015b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 0x7f125015b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 -- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 msgr2=0x7f12501f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 0x7f12501f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:36.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 -- 172.21.15.104:0/920594831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009500 con 0x7f12500cd680 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124ed9d700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 0x7f12501f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124e59c700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 0x7f125015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f124f59e700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f124800cc60 tx=0x7f124800a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124800ce60 con 0x7f12500cd680 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124800acf0 con 0x7f12500cd680 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12500b2c20 con 0x7f12500cd680 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248003c40 con 0x7f12500cd680 2023-06-10T18:59:36.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.967+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12500b31e0 con 0x7f12500cd680 2023-06-10T18:59:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.968+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12500c7770 con 0x7f12500cd680 2023-06-10T18:59:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.968+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f124800f3f0 con 0x7f12500cd680 2023-06-10T18:59:36.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.969+0000 7f12377fe700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 0x7f123c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.969+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1248089770 con 0x7f12500cd680 2023-06-10T18:59:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.969+0000 7f124ed9d700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 0x7f123c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.969+0000 7f124ed9d700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 0x7f123c06d190 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1244013040 tx=0x7f1244008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:36.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:36.973+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12500c7770 con 0x7f12500cd680 2023-06-10T18:59:37.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.174+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f12500c7770 con 0x7f12500cd680 2023-06-10T18:59:37.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.174+0000 7f12377fe700 1 -- 172.21.15.104:0/920594831 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v187) v1 ==== 80+0+0 (secure 0 0 0) 0x7f12500c7770 con 0x7f12500cd680 2023-06-10T18:59:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 msgr2=0x7f123c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 0x7f123c06d190 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f1244013040 tx=0x7f1244008070 comp rx=0 tx=0).stop 2023-06-10T18:59:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 msgr2=0x7f12500ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f124800cc60 tx=0x7f124800a4d0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 shutdown_connections 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f123c06ace0 0x7f123c06d190 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12500aeaa0 0x7f12501f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12500cd680 0x7f12500ad280 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 --2- 172.21.15.104:0/920594831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12500b1080 0x7f125015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 >> 172.21.15.104:0/920594831 conn(0x7f125006b8a0 msgr2=0x7f125006a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:37.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.176+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 shutdown_connections 2023-06-10T18:59:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.177+0000 7f12551e9700 1 -- 172.21.15.104:0/920594831 wait complete. 2023-06-10T18:59:37.185 INFO:teuthology.orchestra.run.smithi104.stderr:listed 0 entries 2023-06-10T18:59:37.194 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi104.front.sepia.ceph.com... 2023-06-10T18:59:37.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:37.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[165779]: fuse finished with error 0 and tester_r 0 2023-06-10T18:59:37.235 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T18:59:37.328 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T18:59:37.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.329 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T18:59:37.345 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-10T18:59:37.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.498+0000 7f639ff21700 1 Processor -- start 2023-06-10T18:59:37.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639ff21700 1 -- start start 2023-06-10T18:59:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639ff21700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f63980cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639ff21700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63980cadd0 con 0x7f639808a3d0 2023-06-10T18:59:37.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639ff21700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63980cb0b0 con 0x7f63980ce350 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639ff21700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63980cfeb0 con 0x7f639806b1f0 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.499+0000 7f639e4be700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f639808a3d0 0x7f63980cb540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39784/0 (socket says 172.21.15.104:39784) 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639e4be700 1 -- 172.21.15.104:0/3784686532 learned_addr learned my addr 172.21.15.104:0/3784686532 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639dcbd700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f63980cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407205952 0 0) 0x7f63980cadd0 con 0x7f639808a3d0 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63840035b0 con 0x7f639808a3d0 2023-06-10T18:59:37.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730443630 0 0) 0x7f63980cb0b0 con 0x7f63980ce350 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63980cadd0 con 0x7f63980ce350 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714049175 0 0) 0x7f63840035b0 con 0x7f639808a3d0 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 msgr2=0x7f63980cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f63980cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 >> v1:172.21.15.114:6789/0 conn(0x7f63980ce350 legacy=0x7f63980cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63980cf550 con 0x7f639808a3d0 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f63900019f0 con 0x7f639808a3d0 2023-06-10T18:59:37.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 >> v1:172.21.15.104:6789/0 conn(0x7f639808a3d0 legacy=0x7f63980cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f6384003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f6384004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6384004d90 0x7f6384005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980cfeb0 con 0x7f6384004120 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638400bba0 con 0x7f6384004d90 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638400bd30 con 0x7f639806b1f0 2023-06-10T18:59:37.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639d4bc700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f6384004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639dcbd700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f6384003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639e4be700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6384004d90 0x7f6384005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639d4bc700 1 -- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 msgr2=0x7f6384003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639d4bc700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f639806b1f0 0x7f6384003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639d4bc700 1 -- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6384004d90 msgr2=0x7f6384005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.500+0000 7f639d4bc700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6384004d90 0x7f6384005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639d4bc700 1 -- 172.21.15.104:0/3784686532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63840035b0 con 0x7f6384004120 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639d4bc700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f6384004850 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f6394001980 tx=0x7f6384005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf52820286ac41 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394001470 con 0x7f6384004120 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63940052d0 con 0x7f6384004120 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ccbb700 1 -- 172.21.15.104:0/3784686532 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394003680 con 0x7f6384004120 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 -- 172.21.15.104:0/3784686532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 msgr2=0x7f6384004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 --2- 172.21.15.104:0/3784686532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f6384004850 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f6394001980 tx=0x7f6384005880 comp rx=0 tx=0).stop 2023-06-10T18:59:37.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639e4be700 1 -- 172.21.15.104:0/3784686532 reap_dead start 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 -- 172.21.15.104:0/3784686532 shutdown_connections 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 -- 172.21.15.104:0/3784686532 >> 172.21.15.104:0/3784686532 conn(0x7f6398089fd0 msgr2=0x7f63980d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 -- 172.21.15.104:0/3784686532 shutdown_connections 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.501+0000 7f639ff21700 1 -- 172.21.15.104:0/3784686532 wait complete. 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 Processor -- start 2023-06-10T18:59:37.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 -- start start 2023-06-10T18:59:37.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.518 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 0x7f639814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 0x7f63980cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980cda40 con 0x7f6384004120 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639814ff10 con 0x7f6398151f40 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639ff21700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639814f5e0 con 0x7f63980cd090 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639dcbd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639dcbd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41964/0 (socket says 172.21.15.104:41964) 2023-06-10T18:59:37.519 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.503+0000 7f639dcbd700 1 -- 172.21.15.104:0/1949597949 learned_addr learned my addr 172.21.15.104:0/1949597949 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:37.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639e4be700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 0x7f63980cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639d4bc700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 0x7f639814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 -- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 msgr2=0x7f63980cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.520 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 0x7f63980cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 -- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 msgr2=0x7f639814ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 0x7f639814ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 -- 172.21.15.104:0/1949597949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639814f8c0 con 0x7f6384004120 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639d4bc700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 0x7f639814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639e4be700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 0x7f63980cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T18:59:37.521 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639dcbd700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f6388011aa0 tx=0x7f6388011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800f980 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638800fae0 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639814fba0 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800fe90 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.504+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63981f5210 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.505+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63981501f0 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.505+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6388014440 con 0x7f6384004120 2023-06-10T18:59:37.522 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.506+0000 7f638e7fc700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:37.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.506+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6388089cf0 con 0x7f6384004120 2023-06-10T18:59:37.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.506+0000 7f639d4bc700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:37.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.506+0000 7f639d4bc700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6394005040 tx=0x7f639400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:37.523 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.510+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f638808e070 con 0x7f6384004120 2023-06-10T18:59:37.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.741+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6398074050 con 0x7f6384004120 2023-06-10T18:59:37.747 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.741+0000 7f638e7fc700 1 -- 172.21.15.104:0/1949597949 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6388054d20 con 0x7f6384004120 2023-06-10T18:59:37.748 INFO:teuthology.orchestra.run.smithi104.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-06-10T18:59:37.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 msgr2=0x7f637c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6394005040 tx=0x7f639400a000 comp rx=0 tx=0).stop 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 msgr2=0x7f63980c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f6388011aa0 tx=0x7f6388011e60 comp rx=0 tx=0).stop 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 shutdown_connections 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63980cd090 0x7f63980cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6384004120 0x7f63980c6b30 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f637c06ace0 0x7f637c06d190 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 --2- 172.21.15.104:0/1949597949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6398151f40 0x7f639814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 >> 172.21.15.104:0/1949597949 conn(0x7f6398089fd0 msgr2=0x7f63980cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 shutdown_connections 2023-06-10T18:59:37.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:37.743+0000 7f639ff21700 1 -- 172.21.15.104:0/1949597949 wait complete. 2023-06-10T18:59:37.761 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-10T18:59:37.762 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-10T18:59:37.762 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2023-06-10T18:59:37.762 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-10T18:59:37.762 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:37.763 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-10T18:59:37.763 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-10T18:59:37.764 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-10T18:59:37.764 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:37.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.766 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2023-06-10T18:59:37.782 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-10T18:59:37.783 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-10T18:59:37.783 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2023-06-10T18:59:37.783 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.783 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2023-06-10T18:59:37.783 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-10T18:59:37.784 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 41507sec preferred_lft 41507sec 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::54e2:468c:e138:40e7/64 scope link noprefixroute 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.785 INFO:teuthology.orchestra.run.smithi104.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0e:1a:b5:23:06:d7 brd ff:ff:ff:ff:ff:ff 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4479:ff:fed6:c752/64 scope link 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:37.786 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 6a:c2:a3:e9:62:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::68c2:a3ff:fee9:625d/64 scope link 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 4e:bd:e3:c5:34:5c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::4cbd:e3ff:fec5:345c/64 scope link 2023-06-10T18:59:37.787 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.788 INFO:teuthology.orchestra.run.smithi104.stdout:11: brx.2@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-06-10T18:59:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0e:1a:b5:23:06:d7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T18:59:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::c1a:b5ff:fe23:6d7/64 scope link 2023-06-10T18:59:37.788 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2023-06-10T18:59:37.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.789 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-10T18:59:37.847 INFO:teuthology.orchestra.run.smithi104.stdout:1 2023-06-10T18:59:37.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.849 DEBUG:teuthology.orchestra.run.smithi104:> route 2023-06-10T18:59:37.907 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2023-06-10T18:59:37.907 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-10T18:59:37.908 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-10T18:59:37.908 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-10T18:59:37.908 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-10T18:59:37.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:37.909 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2023-06-10T18:59:37.909 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2023-06-10T18:59:37.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-10T18:59:37.909 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-10T18:59:37.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-10T18:59:37.910 DEBUG:teuthology.orchestra.run.smithi104:> ') 2023-06-10T18:59:38.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:38.011 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T18:59:38.028 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 2) 2023-06-10T18:59:38.028 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T18:59:38.028 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T18:59:38.028 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:38.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:38.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2023-06-10T18:59:38.094 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-06-10T18:59:38.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:38.096 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:38.152 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-10T18:59:38.152 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-10T18:59:38.153 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:38.209 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:59:38.275 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:38.276 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:38.341 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:38.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:38.343 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:59:38.399 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-10T18:59:38.399 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_reconnect_stale=1 --fuse_disable_pagecache=1) 2023-06-10T18:59:38.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo modprobe fuse 2023-06-10T18:59:38.466 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-10T18:59:38.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-10T18:59:38.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:38.521+0000 7f0fa00ec5c0 -1 init, newargv = 0x55ded37c03e0 newargc=15 2023-06-10T18:59:38.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[167001]: starting ceph client 2023-06-10T18:59:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[167001]: starting fuse 2023-06-10T18:59:38.533 INFO:teuthology.orchestra.run.smithi104.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-10T18:59:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-10T18:59:38.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:38.535 DEBUG:teuthology.orchestra.run.smithi104:> ls /sys/fs/fuse/connections 2023-06-10T18:59:38.591 INFO:teuthology.orchestra.run.smithi104.stdout:47 2023-06-10T18:59:38.591 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-10T18:59:38.592 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:59:38.592 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:59:38.593 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:59:38.593 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:59:38.593 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:59:38.594 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:38.594 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:59:38.594 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:59:38.595 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:59:38.595 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:59:38.595 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:38.595 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:59:38.596 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:59:38.596 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:59:38.596 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:59:38.597 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:38.597 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:59:38.597 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:59:38.598 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:59:38.598 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:59:38.598 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:59:38.598 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:59:38.599 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:59:38.599 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:59:38.599 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:38.600 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:59:38.600 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:59:38.713 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.167001.asok 2023-06-10T18:59:38.895 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.167001.asok 2023-06-10T18:59:38.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:38.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.167001.asok status 2023-06-10T18:59:39.049 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "metadata": { 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_sha1": "219d315c3a0b2075e570fb1964be2ad33fe7a9c8", 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4359-g219d315c (219d315c3a0b2075e570fb1964be2ad33fe7a9c8) reef (dev)", 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "entity_id": "0", 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "hostname": "smithi104", 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-06-10T18:59:39.050 INFO:teuthology.orchestra.run.smithi104.stdout: "pid": "167001", 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "root": "/" 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_count": 0, 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "dentry_pinned_count": 0, 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 11391, 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:59:39.051 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 11391 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 675086267 2023-06-10T18:59:39.052 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 675086267 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.053 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.11391 192.168.0.1:0/675086267", 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "192.168.0.1:0/675086267", 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "inode_count": 1, 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_epoch": 117, 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch": 187, 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_epoch_barrier": 0, 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "blocklisted": false, 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "cephfs" 2023-06-10T18:59:39.054 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:59:39.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:39.060 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:59:39.060 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.076 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:59:39.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:39.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:39.143 DEBUG:teuthology.orchestra.run.smithi104:> stat --file-system '--printf=%T 2023-06-10T18:59:39.144 DEBUG:teuthology.orchestra.run.smithi104:> ' -- /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.199 INFO:teuthology.orchestra.run.smithi104.stdout:fuseblk 2023-06-10T18:59:39.200 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:39.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-10T18:59:39.266 DEBUG:teuthology.orchestra.run.smithi104:> python3 -c ' 2023-06-10T18:59:39.266 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:59:39.267 DEBUG:teuthology.orchestra.run.smithi104:> import sys 2023-06-10T18:59:39.267 DEBUG:teuthology.orchestra.run.smithi104:> import fcntl 2023-06-10T18:59:39.267 DEBUG:teuthology.orchestra.run.smithi104:> import errno 2023-06-10T18:59:39.268 DEBUG:teuthology.orchestra.run.smithi104:> import time 2023-06-10T18:59:39.268 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.268 DEBUG:teuthology.orchestra.run.smithi104:> fd1 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDWR | os.O_CREAT, 0O666) 2023-06-10T18:59:39.268 DEBUG:teuthology.orchestra.run.smithi104:> fd2 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDONLY) 2023-06-10T18:59:39.269 DEBUG:teuthology.orchestra.run.smithi104:> fd3 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDWR | os.O_CREAT, 0O666) 2023-06-10T18:59:39.269 DEBUG:teuthology.orchestra.run.smithi104:> fd4 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDONLY) 2023-06-10T18:59:39.269 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.269 DEBUG:teuthology.orchestra.run.smithi104:> os.write(fd1, b'"'"'content'"'"') 2023-06-10T18:59:39.270 DEBUG:teuthology.orchestra.run.smithi104:> os.read(fd2, 1); 2023-06-10T18:59:39.270 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.270 DEBUG:teuthology.orchestra.run.smithi104:> os.write(fd3, b'"'"'content'"'"') 2023-06-10T18:59:39.270 DEBUG:teuthology.orchestra.run.smithi104:> os.read(fd4, 1); 2023-06-10T18:59:39.271 DEBUG:teuthology.orchestra.run.smithi104:> fcntl.flock(fd4, fcntl.LOCK_SH | fcntl.LOCK_NB) 2023-06-10T18:59:39.271 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.271 DEBUG:teuthology.orchestra.run.smithi104:> print("blocklist") 2023-06-10T18:59:39.271 DEBUG:teuthology.orchestra.run.smithi104:> sys.stdout.flush() 2023-06-10T18:59:39.272 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.272 DEBUG:teuthology.orchestra.run.smithi104:> sys.stdin.readline() 2023-06-10T18:59:39.272 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.272 DEBUG:teuthology.orchestra.run.smithi104:> # wait for mds to close session 2023-06-10T18:59:39.273 DEBUG:teuthology.orchestra.run.smithi104:> time.sleep(10); 2023-06-10T18:59:39.273 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.273 DEBUG:teuthology.orchestra.run.smithi104:> # trigger '"'"'open session'"'"' message. kclient relies on '"'"'session reject'"'"' message 2023-06-10T18:59:39.273 DEBUG:teuthology.orchestra.run.smithi104:> # to detect if itself is blocklisted 2023-06-10T18:59:39.274 DEBUG:teuthology.orchestra.run.smithi104:> try: 2023-06-10T18:59:39.274 DEBUG:teuthology.orchestra.run.smithi104:> os.stat("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1") 2023-06-10T18:59:39.274 DEBUG:teuthology.orchestra.run.smithi104:> except: 2023-06-10T18:59:39.274 DEBUG:teuthology.orchestra.run.smithi104:> pass 2023-06-10T18:59:39.275 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.275 DEBUG:teuthology.orchestra.run.smithi104:> # wait for auto reconnect 2023-06-10T18:59:39.275 DEBUG:teuthology.orchestra.run.smithi104:> time.sleep(10); 2023-06-10T18:59:39.275 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.276 DEBUG:teuthology.orchestra.run.smithi104:> try: 2023-06-10T18:59:39.276 DEBUG:teuthology.orchestra.run.smithi104:> os.write(fd1, b'"'"'content'"'"') 2023-06-10T18:59:39.276 DEBUG:teuthology.orchestra.run.smithi104:> except OSError as e: 2023-06-10T18:59:39.276 DEBUG:teuthology.orchestra.run.smithi104:> if e.errno != errno.EBADF: 2023-06-10T18:59:39.277 DEBUG:teuthology.orchestra.run.smithi104:> raise 2023-06-10T18:59:39.277 DEBUG:teuthology.orchestra.run.smithi104:> else: 2023-06-10T18:59:39.277 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("write() failed to raise error") 2023-06-10T18:59:39.277 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.278 DEBUG:teuthology.orchestra.run.smithi104:> os.read(fd2, 1); 2023-06-10T18:59:39.278 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.278 DEBUG:teuthology.orchestra.run.smithi104:> try: 2023-06-10T18:59:39.278 DEBUG:teuthology.orchestra.run.smithi104:> os.read(fd4, 1) 2023-06-10T18:59:39.279 DEBUG:teuthology.orchestra.run.smithi104:> except OSError as e: 2023-06-10T18:59:39.279 DEBUG:teuthology.orchestra.run.smithi104:> if e.errno != errno.EIO: 2023-06-10T18:59:39.279 DEBUG:teuthology.orchestra.run.smithi104:> raise 2023-06-10T18:59:39.279 DEBUG:teuthology.orchestra.run.smithi104:> else: 2023-06-10T18:59:39.280 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("read() failed to raise error") 2023-06-10T18:59:39.280 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:59:39.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-10T18:59:39.349 DEBUG:teuthology.orchestra.run.smithi104:> import glob 2023-06-10T18:59:39.349 DEBUG:teuthology.orchestra.run.smithi104:> import re 2023-06-10T18:59:39.349 DEBUG:teuthology.orchestra.run.smithi104:> import os 2023-06-10T18:59:39.350 DEBUG:teuthology.orchestra.run.smithi104:> import subprocess 2023-06-10T18:59:39.350 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.350 DEBUG:teuthology.orchestra.run.smithi104:> def _find_admin_socket(client_name): 2023-06-10T18:59:39.351 DEBUG:teuthology.orchestra.run.smithi104:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-06-10T18:59:39.351 DEBUG:teuthology.orchestra.run.smithi104:> files = glob.glob(asok_path) 2023-06-10T18:59:39.351 DEBUG:teuthology.orchestra.run.smithi104:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-06-10T18:59:39.351 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.352 DEBUG:teuthology.orchestra.run.smithi104:> # Given a non-glob path, it better be there 2023-06-10T18:59:39.352 DEBUG:teuthology.orchestra.run.smithi104:> if "*" not in asok_path: 2023-06-10T18:59:39.352 DEBUG:teuthology.orchestra.run.smithi104:> assert(len(files) == 1) 2023-06-10T18:59:39.353 DEBUG:teuthology.orchestra.run.smithi104:> return files[0] 2023-06-10T18:59:39.353 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.353 DEBUG:teuthology.orchestra.run.smithi104:> for f in files: 2023-06-10T18:59:39.354 DEBUG:teuthology.orchestra.run.smithi104:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-10T18:59:39.354 DEBUG:teuthology.orchestra.run.smithi104:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-10T18:59:39.354 DEBUG:teuthology.orchestra.run.smithi104:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-10T18:59:39.355 DEBUG:teuthology.orchestra.run.smithi104:> contents = proc_f.read() 2023-06-10T18:59:39.356 DEBUG:teuthology.orchestra.run.smithi104:> if mountpoint in contents: 2023-06-10T18:59:39.356 DEBUG:teuthology.orchestra.run.smithi104:> return f 2023-06-10T18:59:39.357 DEBUG:teuthology.orchestra.run.smithi104:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-10T18:59:39.357 DEBUG:teuthology.orchestra.run.smithi104:> 2023-06-10T18:59:39.357 DEBUG:teuthology.orchestra.run.smithi104:> print(_find_admin_socket("client.0")) 2023-06-10T18:59:39.358 DEBUG:teuthology.orchestra.run.smithi104:> ' 2023-06-10T18:59:39.440 INFO:teuthology.orchestra.run.smithi104.stdout:/var/run/ceph/ceph-client.0.167001.asok 2023-06-10T18:59:39.622 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.167001.asok 2023-06-10T18:59:39.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T18:59:39.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.167001.asok mds_sessions 2023-06-10T18:59:39.776 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "id": 11391, 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "inst": { 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "name": { 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "client", 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "num": 11391 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.777 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": { 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "192.168.0.1:0", 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 675086267 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "inst_str": "client.11391 v1:192.168.0.1:0/675086267", 2023-06-10T18:59:39.778 INFO:teuthology.orchestra.run.smithi104.stdout: "addr_str": "v1:192.168.0.1:0/675086267", 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions": [ 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": 0, 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "addrs": { 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "addrvec": [ 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v2", 2023-06-10T18:59:39.779 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6834", 2023-06-10T18:59:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3645224862 2023-06-10T18:59:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2023-06-10T18:59:39.780 INFO:teuthology.orchestra.run.smithi104.stdout: "type": "v1", 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "addr": "172.21.15.104:6835", 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "nonce": 3645224862 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2023-06-10T18:59:39.781 INFO:teuthology.orchestra.run.smithi104.stdout: "seq": 0, 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_gen": 0, 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_ttl": "2023-06-10T19:00:39.524394+0000", 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "last_cap_renew_request": "2023-06-10T18:59:39.524394+0000", 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_renew_seq": 2, 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "num_caps": 3, 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: "state": "open" 2023-06-10T18:59:39.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2023-06-10T18:59:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2023-06-10T18:59:39.783 INFO:teuthology.orchestra.run.smithi104.stdout: "mdsmap_epoch": 117 2023-06-10T18:59:39.783 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T18:59:39.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:59:39.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T18:59:39.787 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T18:59:39.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.940+0000 7f09152bf700 1 Processor -- start 2023-06-10T18:59:39.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 -- start start 2023-06-10T18:59:39.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09100ce390 0x7f0910160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f091006c720 0x7f09100ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100d0340 con 0x7f091006b410 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09100d1660 con 0x7f09100ce390 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f09152bf700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091015cb60 con 0x7f091006c720 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f090effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09100ce390 0x7f0910160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f090effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f09100ce390 0x7f0910160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41994/0 (socket says 172.21.15.104:41994) 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f090f7fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f091006b410 0x7f091006b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39826/0 (socket says 172.21.15.104:39826) 2023-06-10T18:59:39.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.941+0000 7f090effd700 1 -- 172.21.15.104:0/2555691403 learned_addr learned my addr 172.21.15.104:0/2555691403 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090e7fc700 1 --2- 172.21.15.104:0/2555691403 >> v2:172.21.15.114:3300/0 conn(0x7f091006c720 0x7f09100ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090effd700 1 -- 172.21.15.104:0/2555691403 >> v2:172.21.15.114:3300/0 conn(0x7f091006c720 msgr2=0x7f09100ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090effd700 1 --2- 172.21.15.104:0/2555691403 >> v2:172.21.15.114:3300/0 conn(0x7f091006c720 0x7f09100ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090effd700 1 -- 172.21.15.104:0/2555691403 >> v1:172.21.15.104:6789/0 conn(0x7f091006b410 legacy=0x7f091006b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090effd700 1 -- 172.21.15.104:0/2555691403 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910161230 con 0x7f09100ce390 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090dffb700 1 -- 172.21.15.104:0/2555691403 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759352640 0 0) 0x7f09100d0340 con 0x7f091006b410 2023-06-10T18:59:39.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090effd700 1 --2- 172.21.15.104:0/2555691403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09100ce390 0x7f0910160cf0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f091006ab00 tx=0x7f0904009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78fb9979b87fc5d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:39.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090dffb700 1 -- 172.21.15.104:0/2555691403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400b870 con 0x7f09100ce390 2023-06-10T18:59:39.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090dffb700 1 -- 172.21.15.104:0/2555691403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090400b9d0 con 0x7f09100ce390 2023-06-10T18:59:39.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.942+0000 7f090dffb700 1 -- 172.21.15.104:0/2555691403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400bd60 con 0x7f09100ce390 2023-06-10T18:59:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 -- 172.21.15.104:0/2555691403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09100ce390 msgr2=0x7f0910160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 --2- 172.21.15.104:0/2555691403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09100ce390 0x7f0910160cf0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f091006ab00 tx=0x7f0904009bd0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 -- 172.21.15.104:0/2555691403 shutdown_connections 2023-06-10T18:59:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 --2- 172.21.15.104:0/2555691403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09100ce390 0x7f0910160cf0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.953 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 --2- 172.21.15.104:0/2555691403 >> v2:172.21.15.114:3300/0 conn(0x7f091006c720 0x7f09100ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 -- 172.21.15.104:0/2555691403 >> 172.21.15.104:0/2555691403 conn(0x7f0910069e70 msgr2=0x7f091006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 -- 172.21.15.104:0/2555691403 shutdown_connections 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.943+0000 7f09152bf700 1 -- 172.21.15.104:0/2555691403 wait complete. 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.944+0000 7f09152bf700 1 Processor -- start 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.944+0000 7f09152bf700 1 -- start start 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f091006b410 0x7f09100c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 0x7f09100ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09101e7130 con 0x7f091006c720 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09100c7bc0 con 0x7f09100ce390 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f09152bf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09101e6520 con 0x7f091006b410 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f090e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f090e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42002/0 (socket says 172.21.15.104:42002) 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.945+0000 7f090e7fc700 1 -- 172.21.15.104:0/2088363907 learned_addr learned my addr 172.21.15.104:0/2088363907 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T18:59:39.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090f7fe700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 0x7f09100ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090effd700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f091006b410 0x7f09100c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 -- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f091006b410 msgr2=0x7f09100c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f091006b410 0x7f09100c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 -- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 msgr2=0x7f09100ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 0x7f09100ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 -- 172.21.15.104:0/2088363907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0904009550 con 0x7f091006c720 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090f7fe700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 0x7f09100ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f090e7fc700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f090000d800 tx=0x7f090000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09000125f0 con 0x7f091006c720 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0900012750 con 0x7f091006c720 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09101e6800 con 0x7f091006c720 2023-06-10T18:59:39.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900012ae0 con 0x7f091006c720 2023-06-10T18:59:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.946+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09101e6dc0 con 0x7f091006c720 2023-06-10T18:59:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.947+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09101e6f80 con 0x7f091006c720 2023-06-10T18:59:39.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.947+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0900012c40 con 0x7f091006c720 2023-06-10T18:59:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.948+0000 7f08f77fe700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 0x7f08fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T18:59:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.948+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0900089990 con 0x7f091006c720 2023-06-10T18:59:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.948+0000 7f090effd700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 0x7f08fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T18:59:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.948+0000 7f090effd700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 0x7f08fc06d190 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0904000c00 tx=0x7f0904008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T18:59:39.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:39.952+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0900054900 con 0x7f091006c720 2023-06-10T18:59:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.179+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0910074050 con 0x7f091006c720 2023-06-10T18:59:40.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.179+0000 7f08f77fe700 1 -- 172.21.15.104:0/2088363907 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f0900057f20 con 0x7f091006c720 2023-06-10T18:59:40.186 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T18:59:40.186 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T18:59:40.187 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 msgr2=0x7f08fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 0x7f08fc06d190 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f0904000c00 tx=0x7f0904008030 comp rx=0 tx=0).stop 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 msgr2=0x7f0910152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f090000d800 tx=0x7f090000dbc0 comp rx=0 tx=0).stop 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 shutdown_connections 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f08fc06ace0 0x7f08fc06d190 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09100ce390 0x7f09100ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:40.188 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f091006b410 0x7f09100c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:40.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 --2- 172.21.15.104:0/2088363907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f091006c720 0x7f0910152000 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T18:59:40.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 >> 172.21.15.104:0/2088363907 conn(0x7f0910069e70 msgr2=0x7f09100b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T18:59:40.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 shutdown_connections 2023-06-10T18:59:40.189 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T18:59:40.181+0000 7f09152bf700 1 -- 172.21.15.104:0/2088363907 wait complete. 2023-06-10T18:59:40.189 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T18:59:40.199 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json session evict 11391 2023-06-10T18:59:40.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:40.727+0000 7f0f7dffb700 -1 client.11391 I was blocklisted at osd epoch 188 2023-06-10T18:59:40.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:40.727+0000 7f0f7dffb700 -1 client.11391 remove_session_caps still has dirty|flushing caps on 0x10000000000.head(faked_ino=0 nref=9 ll_ref=2 cap_refs={4=0,1024=1,2048=0,4096=0,8192=1} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2023-06-10T18:59:39.335001+0000 mtime=2023-06-10T18:59:39.343312+0000 ctime=2023-06-10T18:59:39.343312+0000 change_attr=1 caps=- dirty_caps=Fw objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 7] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7f0f780098b0) 2023-06-10T18:59:40.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:40.727+0000 7f0f7dffb700 -1 client.11391 remove_session_caps still has dirty data on 0x10000000000.head(faked_ino=0 nref=9 ll_ref=2 cap_refs={4=0,1024=1,2048=0,4096=0,8192=1} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2023-06-10T18:59:39.335001+0000 mtime=2023-06-10T18:59:39.343312+0000 ctime=2023-06-10T18:59:39.343312+0000 change_attr=1 caps=- objectset[0x10000000000 ts 0/0 objects 1 dirty_or_tx 7] parents=0x1.head["testfile_reconnect_after_blocklisted.1"] 0x7f0f780098b0) 2023-06-10T18:59:40.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:40.727+0000 7f0f7dffb700 -1 client.11391 remove_session_caps still has dirty|flushing caps on 0x10000000001.head(faked_ino=0 nref=10 ll_ref=2 cap_refs={4=0,1024=1,2048=0,4096=0,8192=1} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2023-06-10T18:59:39.340009+0000 mtime=2023-06-10T18:59:39.343467+0000 ctime=2023-06-10T18:59:39.343467+0000 change_attr=1 caps=- dirty_caps=Fw objectset[0x10000000001 ts 0/0 objects 1 dirty_or_tx 7] parents=0x1.head["testfile_reconnect_after_blocklisted.2"] 0x7f0f7800a070) 2023-06-10T18:59:40.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:2023-06-10T18:59:40.727+0000 7f0f7dffb700 -1 client.11391 remove_session_caps still has dirty data on 0x10000000001.head(faked_ino=0 nref=10 ll_ref=2 cap_refs={4=0,1024=1,2048=0,4096=0,8192=1} open={1=1,3=1} mode=100664 size=7/4194304 nlink=1 btime=2023-06-10T18:59:39.340009+0000 mtime=2023-06-10T18:59:39.343467+0000 ctime=2023-06-10T18:59:39.343467+0000 change_attr=1 caps=- objectset[0x10000000001 ts 0/0 objects 1 dirty_or_tx 7] parents=0x1.head["testfile_reconnect_after_blocklisted.2"] 0x7f0f7800a070) 2023-06-10T18:59:40.777 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2023-06-10T19:00:00.792 INFO:teuthology.orchestra.run.smithi104.stderr:Traceback (most recent call last): 2023-06-10T19:00:00.792 INFO:teuthology.orchestra.run.smithi104.stderr: File "", line 39, in 2023-06-10T19:00:00.792 INFO:teuthology.orchestra.run.smithi104.stderr:BrokenPipeError: [Errno 108] Cannot send after transport endpoint shutdown 2023-06-10T19:00:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:00.797 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-06-10T19:00:00.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T19:00:00.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-06-10T19:00:00.890 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-06-10T19:00:00.890 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T19:00:00.891 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:00.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:01.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.070+0000 7f5a1bef3700 1 Processor -- start 2023-06-10T19:00:01.077 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.070+0000 7f5a1bef3700 1 -- start start 2023-06-10T19:00:01.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1bef3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a14070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1bef3700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a140c9350 0x7f5a140ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1bef3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1415c050 con 0x7f5a140ce510 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1bef3700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140ceea0 con 0x7f5a140c9350 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1bef3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140c9970 con 0x7f5a1415bc80 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1a490700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a140c9350 0x7f5a140ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1a490700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f5a140c9350 0x7f5a140ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42014/0 (socket says 172.21.15.104:42014) 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a1a490700 1 -- 172.21.15.104:0/821281260 learned_addr learned my addr 172.21.15.104:0/821281260 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.071+0000 7f5a19c8f700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a14070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1a490700 1 -- 172.21.15.104:0/821281260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 msgr2=0x7f5a14070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1a490700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a14070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1a490700 1 -- 172.21.15.104:0/821281260 >> v1:172.21.15.114:6789/0 conn(0x7f5a140ce510 legacy=0x7f5a140c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1a490700 1 -- 172.21.15.104:0/821281260 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140ccb70 con 0x7f5a140c9350 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1a490700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140c9350 0x7f5a140ce960 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009e50 tx=0x7f5a0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268bebdcbe25f66 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a18c8d700 1 -- 172.21.15.104:0/821281260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00e070 con 0x7f5a140c9350 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a18c8d700 1 -- 172.21.15.104:0/821281260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00b9c0 con 0x7f5a140c9350 2023-06-10T19:00:01.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a18c8d700 1 -- 172.21.15.104:0/821281260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00bd70 con 0x7f5a140c9350 2023-06-10T19:00:01.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 -- 172.21.15.104:0/821281260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140c9350 msgr2=0x7f5a140ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140c9350 0x7f5a140ce960 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009e50 tx=0x7f5a0c00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 -- 172.21.15.104:0/821281260 shutdown_connections 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140c9350 0x7f5a140ce960 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/821281260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a14070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 -- 172.21.15.104:0/821281260 >> 172.21.15.104:0/821281260 conn(0x7f5a1406ab00 msgr2=0x7f5a140b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.072+0000 7f5a1bef3700 1 -- 172.21.15.104:0/821281260 shutdown_connections 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.073+0000 7f5a1bef3700 1 -- 172.21.15.104:0/821281260 wait complete. 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 Processor -- start 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 -- start start 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a140c9350 0x7f5a140c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a140c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a141ef660 con 0x7f5a140ce510 2023-06-10T19:00:01.084 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140c2580 con 0x7f5a140c9350 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1bef3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a14158e20 con 0x7f5a1415bc80 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.074+0000 7f5a1948e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42018/0 (socket says 172.21.15.104:42018) 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 -- 172.21.15.104:0/1687504689 learned_addr learned my addr 172.21.15.104:0/1687504689 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a19c8f700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a140c9350 0x7f5a140c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1a490700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a140c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.085 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 -- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 msgr2=0x7f5a140c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a140c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 -- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a140c9350 msgr2=0x7f5a140c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a140c9350 0x7f5a140c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 -- 172.21.15.104:0/1687504689 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c009b30 con 0x7f5a140ce510 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1948e700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000e940 tx=0x7f5a1000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a100052d0 con 0x7f5a140ce510 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10005430 con 0x7f5a140ce510 2023-06-10T19:00:01.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a140c6da0 con 0x7f5a140ce510 2023-06-10T19:00:01.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a100108e0 con 0x7f5a140ce510 2023-06-10T19:00:01.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.075+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a140c7360 con 0x7f5a140ce510 2023-06-10T19:00:01.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.076+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a14074050 con 0x7f5a140ce510 2023-06-10T19:00:01.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.076+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a10010a40 con 0x7f5a140ce510 2023-06-10T19:00:01.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.077+0000 7f5a0a7fc700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 0x7f59f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.077+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f5a1008b5d0 con 0x7f5a140ce510 2023-06-10T19:00:01.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.077+0000 7f5a19c8f700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 0x7f59f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.078+0000 7f5a19c8f700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 0x7f59f806d190 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f5a04006b20 tx=0x7f5a04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.082+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a10056510 con 0x7f5a140ce510 2023-06-10T19:00:01.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.267+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a140c26c0 con 0x7f5a140ce510 2023-06-10T19:00:01.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.268+0000 7f5a0a7fc700 1 -- 172.21.15.104:0/1687504689 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18677 (secure 0 0 0) 0x7f5a10059b30 con 0x7f5a140ce510 2023-06-10T19:00:01.274 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:01.275 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":188,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:40.713132+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:01.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 msgr2=0x7f59f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 0x7f59f806d190 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f5a04006b20 tx=0x7f5a04008040 comp rx=0 tx=0).stop 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 msgr2=0x7f5a1414fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000e940 tx=0x7f5a1000ec50 comp rx=0 tx=0).stop 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 shutdown_connections 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f59f806ace0 0x7f59f806d190 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a140c9350 0x7f5a140c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a1415bc80 0x7f5a140c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 --2- 172.21.15.104:0/1687504689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a140ce510 0x7f5a1414fbc0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 >> 172.21.15.104:0/1687504689 conn(0x7f5a1406ab00 msgr2=0x7f5a14070310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 shutdown_connections 2023-06-10T19:00:01.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.270+0000 7f5a1bef3700 1 -- 172.21.15.104:0/1687504689 wait complete. 2023-06-10T19:00:01.288 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:59:21.300558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:59:22.213068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T19:00:01.289 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T19:00:01.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.405+0000 7ff36f4ea700 1 Processor -- start 2023-06-10T19:00:01.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.405+0000 7ff36f4ea700 1 -- start start 2023-06-10T19:00:01.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36f4ea700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff368157b30 0x7ff3680ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36f4ea700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 0x7ff3680b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36f4ea700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3680ad3c0 con 0x7ff3680ce880 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36f4ea700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3680ced20 con 0x7ff368157b30 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36f4ea700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680d0e20 con 0x7ff3680d0a00 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36ca85700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff3680ce880 0x7ff3680d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39854/0 (socket says 172.21.15.104:39854) 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36ca85700 1 -- 172.21.15.104:0/246243456 learned_addr learned my addr 172.21.15.104:0/246243456 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36d286700 1 --2- 172.21.15.104:0/246243456 >> v2:172.21.15.114:3300/0 conn(0x7ff368157b30 0x7ff3680ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.406+0000 7ff36da87700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 0x7ff3680b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253552993 0 0) 0x7ff3680ad3c0 con 0x7ff3680ce880 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3540035b0 con 0x7ff3680ce880 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36d286700 1 -- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 msgr2=0x7ff3680b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36d286700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 0x7ff3680b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36d286700 1 -- 172.21.15.104:0/246243456 >> v1:172.21.15.104:6789/0 conn(0x7ff3680ce880 legacy=0x7ff3680d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222677799 0 0) 0x7ff3540035b0 con 0x7ff3680ce880 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36d286700 1 -- 172.21.15.104:0/246243456 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368157f00 con 0x7ff368157b30 2023-06-10T19:00:01.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36da87700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 0x7ff3680b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff36d286700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff368157b30 0x7ff3680ce340 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff358009840 tx=0x7ff358009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da2f60134fdf306e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364001670 con 0x7ff368157b30 2023-06-10T19:00:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358004030 con 0x7ff368157b30 2023-06-10T19:00:01.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.407+0000 7ff35ffff700 1 -- 172.21.15.104:0/246243456 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3580043a0 con 0x7ff368157b30 2023-06-10T19:00:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 -- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff368157b30 msgr2=0x7ff3680ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff368157b30 0x7ff3680ce340 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff358009840 tx=0x7ff358009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 -- 172.21.15.104:0/246243456 shutdown_connections 2023-06-10T19:00:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3680d0a00 0x7ff3680b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/246243456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff368157b30 0x7ff3680ce340 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 -- 172.21.15.104:0/246243456 >> 172.21.15.104:0/246243456 conn(0x7ff368089fd0 msgr2=0x7ff36808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 -- 172.21.15.104:0/246243456 shutdown_connections 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.408+0000 7ff36f4ea700 1 -- 172.21.15.104:0/246243456 wait complete. 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.409+0000 7ff36f4ea700 1 Processor -- start 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.409+0000 7ff36f4ea700 1 -- start start 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3680d0a00 0x7ff3681512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff368157b30 0x7ff368158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680ce170 con 0x7ff3680ce880 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680cde90 con 0x7ff3680d0a00 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36f4ea700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36806bac0 con 0x7ff368157b30 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42030/0 (socket says 172.21.15.104:42030) 2023-06-10T19:00:01.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 -- 172.21.15.104:0/3841331714 learned_addr learned my addr 172.21.15.104:0/3841331714 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36ca85700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3680d0a00 0x7ff3681512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36da87700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff368157b30 0x7ff368158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 -- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff368157b30 msgr2=0x7ff368158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff368157b30 0x7ff368158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 -- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3680d0a00 msgr2=0x7ff3681512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3680d0a00 0x7ff3681512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 -- 172.21.15.104:0/3841331714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360009a70 con 0x7ff3680ce880 2023-06-10T19:00:01.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff36d286700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7ff3580090b0 tx=0x7ff358004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.410+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358003df0 con 0x7ff3680ce880 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.411+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35801f070 con 0x7ff3680ce880 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.411+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3580094a0 con 0x7ff3680ce880 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.411+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358011830 con 0x7ff3680ce880 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.411+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36806c080 con 0x7ff3680ce880 2023-06-10T19:00:01.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.412+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36806c2a0 con 0x7ff3680ce880 2023-06-10T19:00:01.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.412+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff358021030 con 0x7ff3680ce880 2023-06-10T19:00:01.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.412+0000 7ff35dffb700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 0x7ff34c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.413+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff35808a8e0 con 0x7ff3680ce880 2023-06-10T19:00:01.423 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.413+0000 7ff36ca85700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 0x7ff34c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.413+0000 7ff36ca85700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 0x7ff34c06d1e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff364001eb0 tx=0x7ff364001c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.417+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff358055990 con 0x7ff3680ce880 2023-06-10T19:00:01.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.604+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff368074050 con 0x7ff3680ce880 2023-06-10T19:00:01.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.604+0000 7ff35dffb700 1 -- 172.21.15.104:0/3841331714 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff358011990 con 0x7ff3680ce880 2023-06-10T19:00:01.611 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 msgr2=0x7ff34c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 0x7ff34c06d1e0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff364001eb0 tx=0x7ff364001c60 comp rx=0 tx=0).stop 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 msgr2=0x7ff368151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7ff3580090b0 tx=0x7ff358004750 comp rx=0 tx=0).stop 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 shutdown_connections 2023-06-10T19:00:01.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff368157b30 0x7ff368158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3680ce880 0x7ff368151b10 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff34c06ad30 0x7ff34c06d1e0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 --2- 172.21.15.104:0/3841331714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3680d0a00 0x7ff3681512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 >> 172.21.15.104:0/3841331714 conn(0x7ff368089fd0 msgr2=0x7ff36806f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 shutdown_connections 2023-06-10T19:00:01.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.606+0000 7ff36f4ea700 1 -- 172.21.15.104:0/3841331714 wait complete. 2023-06-10T19:00:01.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T19:00:01.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.741+0000 7f2cb9c39700 1 Processor -- start 2023-06-10T19:00:01.748 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.741+0000 7f2cb9c39700 1 -- start start 2023-06-10T19:00:01.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb9c39700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2cb40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb9c39700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb406e250 con 0x7f2cb406cde0 2023-06-10T19:00:01.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb9c39700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb415c950 con 0x7f2cb406e6e0 2023-06-10T19:00:01.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb9c39700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40b1e80 con 0x7f2cb415c580 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2cb406cde0 0x7f2cb40ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39870/0 (socket says 172.21.15.104:39870) 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb3fff700 1 -- 172.21.15.104:0/393243065 learned_addr learned my addr 172.21.15.104:0/393243065 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.742+0000 7f2cb37fe700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2cb40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249622706 0 0) 0x7f2cb406e250 con 0x7f2cb406cde0 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9c0035b0 con 0x7f2cb406cde0 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801119950 0 0) 0x7f2c9c0035b0 con 0x7f2cb406cde0 2023-06-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 msgr2=0x7f2cb40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2cb40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 >> v1:172.21.15.114:6789/0 conn(0x7f2cb406e6e0 legacy=0x7f2cb40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb406d920 con 0x7f2cb406cde0 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f2cac0019f0 con 0x7f2cb406cde0 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 >> v1:172.21.15.104:6789/0 conn(0x7f2cb406cde0 legacy=0x7f2cb40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2c9c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c9c003f70 0x7f2c9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c9c004d90 0x7f2c9c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40b1e80 con 0x7f2c9c004d90 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00ba40 con 0x7f2c9c003f70 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00bc00 con 0x7f2cb415c580 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c9c004d90 0x7f2c9c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb37fe700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2c9c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb2ffd700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c9c003f70 0x7f2c9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 -- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 msgr2=0x7f2c9c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb415c580 0x7f2c9c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 -- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c9c003f70 msgr2=0x7f2c9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c9c003f70 0x7f2c9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 -- 172.21.15.104:0/393243065 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0035b0 con 0x7f2c9c004d90 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.743+0000 7f2cb3fff700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c9c004d90 0x7f2c9c005200 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00a850 tx=0x7f2c9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95800463b974540 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac015c80 con 0x7f2c9c004d90 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac015de0 con 0x7f2c9c004d90 2023-06-10T19:00:01.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb27fc700 1 -- 172.21.15.104:0/393243065 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac01e6b0 con 0x7f2c9c004d90 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 -- 172.21.15.104:0/393243065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c9c004d90 msgr2=0x7f2c9c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/393243065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c9c004d90 0x7f2c9c005200 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00a850 tx=0x7f2c9c005880 comp rx=0 tx=0).stop 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb3fff700 1 -- 172.21.15.104:0/393243065 reap_dead start 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 -- 172.21.15.104:0/393243065 shutdown_connections 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 -- 172.21.15.104:0/393243065 >> 172.21.15.104:0/393243065 conn(0x7f2cb406b8a0 msgr2=0x7f2cb40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 -- 172.21.15.104:0/393243065 shutdown_connections 2023-06-10T19:00:01.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.744+0000 7f2cb9c39700 1 -- 172.21.15.104:0/393243065 wait complete. 2023-06-10T19:00:01.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.745+0000 7f2cb9c39700 1 Processor -- start 2023-06-10T19:00:01.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 -- start start 2023-06-10T19:00:01.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb41537a0 0x7f2cb40c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.759 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cb40b4560 0x7f2cb40cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40b3700 con 0x7f2cb40ae210 2023-06-10T19:00:01.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40b2a80 con 0x7f2cb40b4560 2023-06-10T19:00:01.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb9c39700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40b2d60 con 0x7f2cb41537a0 2023-06-10T19:00:01.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42052/0 (socket says 172.21.15.104:42052) 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.746+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/2736397058 learned_addr learned my addr 172.21.15.104:0/2736397058 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb37fe700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb41537a0 0x7f2cb40c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb3fff700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cb40b4560 0x7f2cb40cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb41537a0 msgr2=0x7f2cb40c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb41537a0 0x7f2cb40c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cb40b4560 msgr2=0x7f2cb40cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cb40b4560 0x7f2cb40cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/2736397058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40b3040 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb2ffd700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400f860 tx=0x7f2ca400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400fe60 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca40052e0 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb40b3320 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4003680 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.747+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb4153d50 con 0x7f2cb40ae210 2023-06-10T19:00:01.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.748+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb40b3840 con 0x7f2cb40ae210 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.748+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2ca4003800 con 0x7f2cb40ae210 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.749+0000 7f2c9bfff700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 0x7f2ca006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.749+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2ca4089380 con 0x7f2cb40ae210 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.749+0000 7f2cb37fe700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 0x7f2ca006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.749+0000 7f2cb37fe700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 0x7f2ca006d190 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009b80 tx=0x7f2ca8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:01.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.753+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2cb40b3840 con 0x7f2cb40ae210 2023-06-10T19:00:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.939+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2cb4153d50 con 0x7f2cb40ae210 2023-06-10T19:00:01.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.940+0000 7f2c9bfff700 1 -- 172.21.15.104:0/2736397058 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v188) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2cb4153d50 con 0x7f2cb40ae210 2023-06-10T19:00:01.946 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T19:00:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 msgr2=0x7f2ca006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 0x7f2ca006d190 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009b80 tx=0x7f2ca8008040 comp rx=0 tx=0).stop 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 msgr2=0x7f2cb40b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400f860 tx=0x7f2ca400fc20 comp rx=0 tx=0).stop 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 shutdown_connections 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca006ace0 0x7f2ca006d190 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cb40b4560 0x7f2cb40cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cb41537a0 0x7f2cb40c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 --2- 172.21.15.104:0/2736397058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb40ae210 0x7f2cb40b4020 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 >> 172.21.15.104:0/2736397058 conn(0x7f2cb406b8a0 msgr2=0x7f2cb4159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 shutdown_connections 2023-06-10T19:00:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:01.942+0000 7f2cb9c39700 1 -- 172.21.15.104:0/2736397058 wait complete. 2023-06-10T19:00:01.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T19:00:02.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.077+0000 7f94b3289700 1 Processor -- start 2023-06-10T19:00:02.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.077+0000 7f94b3289700 1 -- start start 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b3289700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac0cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b3289700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94ac06a880 0x7f94ac0cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b3289700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac0cab60 con 0x7f94ac0ae950 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b3289700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac0d0540 con 0x7f94ac06a880 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b3289700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0cc880 con 0x7f94ac06e0b0 2023-06-10T19:00:02.086 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1826700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f94ac06a880 0x7f94ac0cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1025700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f94ac0ae950 0x7f94ac06d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39886/0 (socket says 172.21.15.104:39886) 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1025700 1 -- 172.21.15.104:0/3745448221 learned_addr learned my addr 172.21.15.104:0/3745448221 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b0824700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac0cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1826700 1 -- 172.21.15.104:0/3745448221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 msgr2=0x7f94ac0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1826700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac0cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1826700 1 -- 172.21.15.104:0/3745448221 >> v1:172.21.15.104:6789/0 conn(0x7f94ac0ae950 legacy=0x7f94ac06d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94b1826700 1 -- 172.21.15.104:0/3745448221 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac06cea0 con 0x7f94ac06a880 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.078+0000 7f94a3fff700 1 -- 172.21.15.104:0/3745448221 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638696189 0 0) 0x7f94ac0cab60 con 0x7f94ac0ae950 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b1826700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac0cbb10 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009d90 tx=0x7f94a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57d2128fe275ecc server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94a3fff700 1 -- 172.21.15.104:0/3745448221 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a400e070 con 0x7f94ac06a880 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94a3fff700 1 -- 172.21.15.104:0/3745448221 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a400bbd0 con 0x7f94ac06a880 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94a3fff700 1 -- 172.21.15.104:0/3745448221 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4012430 con 0x7f94ac06a880 2023-06-10T19:00:02.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 -- 172.21.15.104:0/3745448221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 msgr2=0x7f94ac0cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac0cbb10 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009d90 tx=0x7f94a400b330 comp rx=0 tx=0).stop 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 -- 172.21.15.104:0/3745448221 shutdown_connections 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac0cbb10 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 --2- 172.21.15.104:0/3745448221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 -- 172.21.15.104:0/3745448221 >> 172.21.15.104:0/3745448221 conn(0x7f94ac06b8a0 msgr2=0x7f94ac0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 -- 172.21.15.104:0/3745448221 shutdown_connections 2023-06-10T19:00:02.089 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.079+0000 7f94b3289700 1 -- 172.21.15.104:0/3745448221 wait complete. 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.081+0000 7f94b3289700 1 Processor -- start 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.081+0000 7f94b3289700 1 -- start start 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94ac0ae950 0x7f94ac0ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac15c730 con 0x7f94ac06a880 2023-06-10T19:00:02.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac15baf0 con 0x7f94ac0ae950 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b3289700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac15bd90 con 0x7f94ac06e0b0 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42064/0 (socket says 172.21.15.104:42064) 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 -- 172.21.15.104:0/3230420493 learned_addr learned my addr 172.21.15.104:0/3230420493 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b0824700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1826700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94ac0ae950 0x7f94ac0ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 -- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 msgr2=0x7f94ac155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 -- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94ac0ae950 msgr2=0x7f94ac0ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94ac0ae950 0x7f94ac0ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 -- 172.21.15.104:0/3230420493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4009a70 con 0x7f94ac06a880 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.082+0000 7f94b1025700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f949c0010e0 tx=0x7f949c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.083+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0074e0 con 0x7f94ac06a880 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.083+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c007640 con 0x7f94ac06a880 2023-06-10T19:00:02.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.083+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c005a70 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.083+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac15c070 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.083+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac1561e0 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.084+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac15c9a0 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.084+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f949c005bd0 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.084+0000 7f94a1ffb700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.085+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f949c088b80 con 0x7f94ac06a880 2023-06-10T19:00:02.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.085+0000 7f94b0824700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.085+0000 7f94b0824700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f94ac15b740 tx=0x7f94a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.096 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.089+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f949c053ac0 con 0x7f94ac06a880 2023-06-10T19:00:02.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.273+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94ac074050 con 0x7f94ac06a880 2023-06-10T19:00:02.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.274+0000 7f94a1ffb700 1 -- 172.21.15.104:0/3230420493 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v188) v1 ==== 104+0+11 (secure 0 0 0) 0x7f949c0170a0 con 0x7f94ac06a880 2023-06-10T19:00:02.281 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T19:00:02.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 msgr2=0x7f949006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f94ac15b740 tx=0x7f94a8008040 comp rx=0 tx=0).stop 2023-06-10T19:00:02.282 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 msgr2=0x7f94ac15cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f949c0010e0 tx=0x7f949c00cae0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 shutdown_connections 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f949006ace0 0x7f949006d190 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94ac0ae950 0x7f94ac0ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac06a880 0x7f94ac15cb50 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 --2- 172.21.15.104:0/3230420493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94ac06e0b0 0x7f94ac155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 >> 172.21.15.104:0/3230420493 conn(0x7f94ac06b8a0 msgr2=0x7f94ac0ac8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 shutdown_connections 2023-06-10T19:00:02.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.276+0000 7f94b3289700 1 -- 172.21.15.104:0/3230420493 wait complete. 2023-06-10T19:00:02.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T19:00:02.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.409+0000 7f48bf02f700 1 Processor -- start 2023-06-10T19:00:02.416 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.409+0000 7f48bf02f700 1 -- start start 2023-06-10T19:00:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f48b815c500 0x7f48b80cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b80cf200 0x7f48b8069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f48b806a3b0 0x7f48b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.418 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b80aef20 con 0x7f48b806a3b0 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b815c8d0 con 0x7f48b815c500 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bf02f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b806abd0 con 0x7f48b80cf200 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bd5cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f48b806a3b0 0x7f48b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bd5cc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f48b806a3b0 0x7f48b806dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42076/0 (socket says 172.21.15.104:42076) 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bd5cc700 1 -- 172.21.15.104:0/1674368094 learned_addr learned my addr 172.21.15.104:0/1674368094 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48bcdcb700 1 --2- 172.21.15.104:0/1674368094 >> v2:172.21.15.114:3300/0 conn(0x7f48b815c500 0x7f48b80cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.410+0000 7f48b7fff700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b80cf200 0x7f48b8069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.419 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 -- 172.21.15.104:0/1674368094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b80cf200 msgr2=0x7f48b8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b80cf200 0x7f48b8069e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 -- 172.21.15.104:0/1674368094 >> v2:172.21.15.114:3300/0 conn(0x7f48b815c500 msgr2=0x7f48b80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 --2- 172.21.15.104:0/1674368094 >> v2:172.21.15.114:3300/0 conn(0x7f48b815c500 0x7f48b80cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 -- 172.21.15.104:0/1674368094 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b806a8f0 con 0x7f48b806a3b0 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48bd5cc700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b806dcd0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f48b80b2170 tx=0x7f48b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735a996634b189b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48b77fe700 1 -- 172.21.15.104:0/1674368094 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000e070 con 0x7f48b806a3b0 2023-06-10T19:00:02.420 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48b77fe700 1 -- 172.21.15.104:0/1674368094 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000b910 con 0x7f48b806a3b0 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.411+0000 7f48b77fe700 1 -- 172.21.15.104:0/1674368094 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000bcc0 con 0x7f48b806a3b0 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 -- 172.21.15.104:0/1674368094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 msgr2=0x7f48b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b806dcd0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f48b80b2170 tx=0x7f48b000b330 comp rx=0 tx=0).stop 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 -- 172.21.15.104:0/1674368094 shutdown_connections 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b806dcd0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 --2- 172.21.15.104:0/1674368094 >> v2:172.21.15.114:3300/0 conn(0x7f48b815c500 0x7f48b80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.421 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 --2- 172.21.15.104:0/1674368094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b80cf200 0x7f48b8069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 -- 172.21.15.104:0/1674368094 >> 172.21.15.104:0/1674368094 conn(0x7f48b806b8a0 msgr2=0x7f48b80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 -- 172.21.15.104:0/1674368094 shutdown_connections 2023-06-10T19:00:02.422 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.412+0000 7f48bf02f700 1 -- 172.21.15.104:0/1674368094 wait complete. 2023-06-10T19:00:02.424 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.413+0000 7f48bf02f700 1 Processor -- start 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.413+0000 7f48bf02f700 1 -- start start 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48b80cf200 0x7f48b80c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b815c500 0x7f48b80c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b81f1760 con 0x7f48b806a3b0 2023-06-10T19:00:02.425 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cbf30 con 0x7f48b80cf200 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cc210 con 0x7f48b815c500 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42078/0 (socket says 172.21.15.104:42078) 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 -- 172.21.15.104:0/3449549692 learned_addr learned my addr 172.21.15.104:0/3449549692 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48b7fff700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48b80cf200 0x7f48b80c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bd5cc700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b815c500 0x7f48b80c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.426 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 -- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b815c500 msgr2=0x7f48b80c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b815c500 0x7f48b80c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 -- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48b80cf200 msgr2=0x7f48b80c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48b80cf200 0x7f48b80c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 -- 172.21.15.104:0/3449549692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0009b30 con 0x7f48b806a3b0 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bcdcb700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00e990 tx=0x7f48ac00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.427 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0096f0 con 0x7f48b806a3b0 2023-06-10T19:00:02.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac00c850 con 0x7f48b806a3b0 2023-06-10T19:00:02.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b80cc4f0 con 0x7f48b806a3b0 2023-06-10T19:00:02.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.414+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00cbc0 con 0x7f48b806a3b0 2023-06-10T19:00:02.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.415+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b80c9f20 con 0x7f48b806a3b0 2023-06-10T19:00:02.428 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.415+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8074050 con 0x7f48b806a3b0 2023-06-10T19:00:02.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.416+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f48ac00cd20 con 0x7f48b806a3b0 2023-06-10T19:00:02.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.416+0000 7f48b57fa700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 0x7f489c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.416+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f48ac089b20 con 0x7f48b806a3b0 2023-06-10T19:00:02.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.417+0000 7f48b7fff700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 0x7f489c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.429 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.417+0000 7f48b7fff700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 0x7f489c06d160 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f48b80af970 tx=0x7f48a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.430 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.421+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f48ac054a60 con 0x7f48b806a3b0 2023-06-10T19:00:02.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.604+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48b80c2ef0 con 0x7f48b806a3b0 2023-06-10T19:00:02.611 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.605+0000 7f48b57fa700 1 -- 172.21.15.104:0/3449549692 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v188) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48ac021020 con 0x7f48b806a3b0 2023-06-10T19:00:02.611 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T19:00:02.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.606+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 msgr2=0x7f489c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.606+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 0x7f489c06d160 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f48b80af970 tx=0x7f48a4008040 comp rx=0 tx=0).stop 2023-06-10T19:00:02.613 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 msgr2=0x7f48b81f1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00e990 tx=0x7f48ac00ed50 comp rx=0 tx=0).stop 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 shutdown_connections 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48b815c500 0x7f48b80c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48b806a3b0 0x7f48b81f1220 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f489c06acb0 0x7f489c06d160 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 --2- 172.21.15.104:0/3449549692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48b80cf200 0x7f48b80c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.614 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 >> 172.21.15.104:0/3449549692 conn(0x7f48b806b8a0 msgr2=0x7f48b80b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 shutdown_connections 2023-06-10T19:00:02.615 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.607+0000 7f48bf02f700 1 -- 172.21.15.104:0/3449549692 wait complete. 2023-06-10T19:00:02.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T19:00:02.745 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.739+0000 7fbe81230700 1 Processor -- start 2023-06-10T19:00:02.746 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.739+0000 7fbe81230700 1 -- start start 2023-06-10T19:00:02.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe81230700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbe7c15a250 0x7fbe7c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe81230700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7c0ccca0 con 0x7fbe7c0cad90 2023-06-10T19:00:02.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe81230700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7c0cee50 con 0x7fbe7c0cfa00 2023-06-10T19:00:02.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe81230700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cf130 con 0x7fbe7c15a250 2023-06-10T19:00:02.749 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe7b59e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbe7c0cad90 0x7fbe7c0b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39918/0 (socket says 172.21.15.104:39918) 2023-06-10T19:00:02.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe7b59e700 1 -- 172.21.15.104:0/533715473 learned_addr learned my addr 172.21.15.104:0/533715473 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:02.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384674881 0 0) 0x7fbe7c0ccca0 con 0x7fbe7c0cad90 2023-06-10T19:00:02.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/533715473 >> v2:172.21.15.114:3300/0 conn(0x7fbe7c15a250 0x7fbe7c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.750 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.740+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe640035b0 con 0x7fbe7c0cad90 2023-06-10T19:00:02.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663743492 0 0) 0x7fbe640035b0 con 0x7fbe7c0cad90 2023-06-10T19:00:02.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 >> v2:172.21.15.114:3300/0 conn(0x7fbe7c15a250 msgr2=0x7fbe7c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 --2- 172.21.15.104:0/533715473 >> v2:172.21.15.114:3300/0 conn(0x7fbe7c15a250 0x7fbe7c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.751 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 >> v1:172.21.15.114:6789/0 conn(0x7fbe7c0cfa00 legacy=0x7fbe7c0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0cc930 con 0x7fbe7c0cad90 2023-06-10T19:00:02.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fbe740019f0 con 0x7fbe7c0cad90 2023-06-10T19:00:02.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 >> v1:172.21.15.104:6789/0 conn(0x7fbe7c0cad90 legacy=0x7fbe7c0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.752 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe64003a00 0x7fbe64003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 0x7fbe64004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64005270 0x7fbe6400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.753 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cf130 con 0x7fbe64004850 2023-06-10T19:00:02.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6400c1b0 con 0x7fbe64003a00 2023-06-10T19:00:02.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6400c370 con 0x7fbe64005270 2023-06-10T19:00:02.754 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 0x7fbe64004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7ad9d700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe64003a00 0x7fbe64003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7b59e700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64005270 0x7fbe6400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 -- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64005270 msgr2=0x7fbe6400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64005270 0x7fbe6400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 -- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe64003a00 msgr2=0x7fbe64003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:02.755 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe64003a00 0x7fbe64003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 -- 172.21.15.104:0/533715473 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640035b0 con 0x7fbe64004850 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7b59e700 1 -- 172.21.15.104:0/533715473 reap_dead start 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 0x7fbe64004cc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009860 tx=0x7fbe64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdb6ee23f2dc05e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00c070 con 0x7fbe64004850 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c013610 con 0x7fbe64004850 2023-06-10T19:00:02.756 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.741+0000 7fbe79d9b700 1 -- 172.21.15.104:0/533715473 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0139c0 con 0x7fbe64004850 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 -- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 msgr2=0x7fbe64004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 0x7fbe64004cc0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009860 tx=0x7fbe64005880 comp rx=0 tx=0).stop 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 -- 172.21.15.104:0/533715473 shutdown_connections 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 --2- 172.21.15.104:0/533715473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe64004850 0x7fbe64004cc0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 -- 172.21.15.104:0/533715473 >> 172.21.15.104:0/533715473 conn(0x7fbe7c06e370 msgr2=0x7fbe7c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 -- 172.21.15.104:0/533715473 shutdown_connections 2023-06-10T19:00:02.757 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.742+0000 7fbe81230700 1 -- 172.21.15.104:0/533715473 wait complete. 2023-06-10T19:00:02.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.743+0000 7fbe81230700 1 Processor -- start 2023-06-10T19:00:02.760 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.743+0000 7fbe81230700 1 -- start start 2023-06-10T19:00:02.761 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64004850 0x7fbe7c1f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 0x7fbe7c15c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c1533e0 con 0x7fbe7c15a650 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0c6230 con 0x7fbe7c0c5d90 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe81230700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0c6510 con 0x7fbe64004850 2023-06-10T19:00:02.762 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42100/0 (socket says 172.21.15.104:42100) 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe7a59c700 1 -- 172.21.15.104:0/340618710 learned_addr learned my addr 172.21.15.104:0/340618710 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.744+0000 7fbe7b59e700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 0x7fbe7c15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7ad9d700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64004850 0x7fbe7c1f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 -- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64004850 msgr2=0x7fbe7c1f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.763 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64004850 0x7fbe7c1f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 -- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 msgr2=0x7fbe7c15c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 0x7fbe7c15c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 -- 172.21.15.104:0/340618710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c153070 con 0x7fbe7c15a650 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7b59e700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 0x7fbe7c15c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe7a59c700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c016040 tx=0x7fbe6c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c004030 con 0x7fbe7c15a650 2023-06-10T19:00:02.764 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c0041b0 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c004560 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7c0c67f0 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.745+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7c1f2d00 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.746+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7c074050 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.746+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbe6c0036a0 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.747+0000 7fbe637fe700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 0x7fbe6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.747+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fbe6c08a0e0 con 0x7fbe7c15a650 2023-06-10T19:00:02.765 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.747+0000 7fbe7ad9d700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 0x7fbe6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:02.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.747+0000 7fbe7ad9d700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 0x7fbe6806d190 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbe70007fd0 tx=0x7fbe70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:02.766 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.751+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbe6c0550b0 con 0x7fbe7c15a650 2023-06-10T19:00:02.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.959+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbe7c153520 con 0x7fbe7c15a650 2023-06-10T19:00:02.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.960+0000 7fbe637fe700 1 -- 172.21.15.104:0/340618710 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v188) v1 ==== 80+0+56 (secure 0 0 0) 0x7fbe6c0586d0 con 0x7fbe7c15a650 2023-06-10T19:00:02.966 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/675086267 2023-06-10T19:59:40.323449+0000 2023-06-10T19:00:02.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 msgr2=0x7fbe6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 0x7fbe6806d190 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbe70007fd0 tx=0x7fbe70009040 comp rx=0 tx=0).stop 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 msgr2=0x7fbe7c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c016040 tx=0x7fbe6c0047c0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 shutdown_connections 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fbe6806ace0 0x7fbe6806d190 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe7c0c5d90 0x7fbe7c15c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe64004850 0x7fbe7c1f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 --2- 172.21.15.104:0/340618710 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe7c15a650 0x7fbe7c0c5850 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 >> 172.21.15.104:0/340618710 conn(0x7fbe7c06e370 msgr2=0x7fbe7c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 shutdown_connections 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:02.962+0000 7fbe81230700 1 -- 172.21.15.104:0/340618710 wait complete. 2023-06-10T19:00:02.970 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T19:00:02.980 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-06-10T19:00:02.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:02.981 DEBUG:teuthology.orchestra.run.smithi104:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-06-10T19:00:03.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi104.stderr:ceph-fuse[167001]: fuse finished with error 0 and tester_r 0 2023-06-10T19:00:03.023 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:03.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:03.024 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-10T19:00:03.148 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T19:00:03.149 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:03.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:03.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.284+0000 7f846f5c3700 1 Processor -- start 2023-06-10T19:00:03.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.284+0000 7f846f5c3700 1 -- start start 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846f5c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f84680cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846f5c3700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84680cdae0 con 0x7f84680cd680 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846f5c3700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84680c9e60 con 0x7f84680b1080 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846f5c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c9bc0 con 0x7f84680aeaa0 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846db60700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f84680cd680 0x7f846806dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39936/0 (socket says 172.21.15.104:39936) 2023-06-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846db60700 1 -- 172.21.15.104:0/367173038 learned_addr learned my addr 172.21.15.104:0/367173038 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900305417 0 0) 0x7f84680cdae0 con 0x7f84680cd680 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f846d35f700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f84680cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.285+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84540035b0 con 0x7f84680cd680 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665542273 0 0) 0x7f84540035b0 con 0x7f84680cd680 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 msgr2=0x7f84680cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f84680cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 >> v1:172.21.15.114:6789/0 conn(0x7f84680b1080 legacy=0x7f84680ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680aee70 con 0x7f84680cd680 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f84600019f0 con 0x7f84680cd680 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 >> v1:172.21.15.104:6789/0 conn(0x7f84680cd680 legacy=0x7f846806dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f8454003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8454003fb0 0x7f8454004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8454004d90 0x7f8454005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c9bc0 con 0x7f8454004d90 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845400baf0 con 0x7f8454003fb0 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845400bcb0 con 0x7f84680aeaa0 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8454004d90 0x7f8454005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846d35f700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f8454003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846cb5e700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8454003fb0 0x7f8454004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 -- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 msgr2=0x7f8454003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f8454003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 -- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8454003fb0 msgr2=0x7f8454004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8454003fb0 0x7f8454004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 -- 172.21.15.104:0/367173038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84540035b0 con 0x7f8454004d90 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846d35f700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680aeaa0 0x7f8454003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:03.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f846db60700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8454004d90 0x7f8454005200 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f846806e470 tx=0x7f8454005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abb24d8ddf504fb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846000a2f0 con 0x7f8454004d90 2023-06-10T19:00:03.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460012600 con 0x7f8454004d90 2023-06-10T19:00:03.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.286+0000 7f845ffff700 1 -- 172.21.15.104:0/367173038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460012970 con 0x7f8454004d90 2023-06-10T19:00:03.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 -- 172.21.15.104:0/367173038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8454004d90 msgr2=0x7f8454005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 --2- 172.21.15.104:0/367173038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8454004d90 0x7f8454005200 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f846806e470 tx=0x7f8454005880 comp rx=0 tx=0).stop 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846db60700 1 -- 172.21.15.104:0/367173038 reap_dead start 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 -- 172.21.15.104:0/367173038 shutdown_connections 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 -- 172.21.15.104:0/367173038 >> 172.21.15.104:0/367173038 conn(0x7f846806b8a0 msgr2=0x7f84680cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 -- 172.21.15.104:0/367173038 shutdown_connections 2023-06-10T19:00:03.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.287+0000 7f846f5c3700 1 -- 172.21.15.104:0/367173038 wait complete. 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.288+0000 7f846f5c3700 1 Processor -- start 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.288+0000 7f846f5c3700 1 -- start start 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f846814a390 0x7f846806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680ae230 0x7f84680ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680c92f0 con 0x7f846814a760 2023-06-10T19:00:03.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680ad190 con 0x7f846814a390 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846f5c3700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84680ad470 con 0x7f84680ae230 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42118/0 (socket says 172.21.15.104:42118) 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 -- 172.21.15.104:0/361010329 learned_addr learned my addr 172.21.15.104:0/361010329 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846d35f700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f846814a390 0x7f846806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846db60700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680ae230 0x7f84680ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 -- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680ae230 msgr2=0x7f84680ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680ae230 0x7f84680ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 -- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f846814a390 msgr2=0x7f846806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f846814a390 0x7f846806d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 -- 172.21.15.104:0/361010329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680ad750 con 0x7f846814a760 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.289+0000 7f846cb5e700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f846400f7f0 tx=0x7f846400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.290+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84640052e0 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.290+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8464005440 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.290+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84680ada30 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.290+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84640038c0 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.290+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84680acaf0 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.291+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84680adf50 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.291+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8464003a20 con 0x7f846814a760 2023-06-10T19:00:03.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.292+0000 7f845dffb700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 0x7f844c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.292+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f84640894d0 con 0x7f846814a760 2023-06-10T19:00:03.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.292+0000 7f846d35f700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 0x7f844c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.292+0000 7f846d35f700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 0x7f844c06d1c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8458009d20 tx=0x7f8458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.296+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f84680adf50 con 0x7f846814a760 2023-06-10T19:00:03.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.484+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84680adf50 con 0x7f846814a760 2023-06-10T19:00:03.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.485+0000 7f845dffb700 1 -- 172.21.15.104:0/361010329 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18677 (secure 0 0 0) 0x7f84680adf50 con 0x7f846814a760 2023-06-10T19:00:03.492 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:03.492 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":188,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:40.713132+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:03.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 msgr2=0x7f844c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 0x7f844c06d1c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8458009d20 tx=0x7f8458008040 comp rx=0 tx=0).stop 2023-06-10T19:00:03.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 msgr2=0x7f846806da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f846400f7f0 tx=0x7f846400fbb0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 shutdown_connections 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f844c06ad10 0x7f844c06d1c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f846814a390 0x7f846806d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84680ae230 0x7f84680ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 --2- 172.21.15.104:0/361010329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f846814a760 0x7f846806da30 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 >> 172.21.15.104:0/361010329 conn(0x7f846806b8a0 msgr2=0x7f846814d080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:03.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 shutdown_connections 2023-06-10T19:00:03.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.487+0000 7f846f5c3700 1 -- 172.21.15.104:0/361010329 wait complete. 2023-06-10T19:00:03.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:59:21.300558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:59:22.213068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T19:00:03.507 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T19:00:03.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.624+0000 7f1272de1700 1 Processor -- start 2023-06-10T19:00:03.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.624+0000 7f1272de1700 1 -- start start 2023-06-10T19:00:03.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1272de1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f126c15c580 0x7f126c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1272de1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 0x7f126c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1272de1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c06e250 con 0x7f126c06cde0 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1272de1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126c15c950 con 0x7f126c15c580 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1272de1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0cf0e0 con 0x7f126c06e6e0 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f126c15c580 0x7f126c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.633 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f127137e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f126c06cde0 0x7f126c069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39956/0 (socket says 172.21.15.104:39956) 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f127137e700 1 -- 172.21.15.104:0/107238646 learned_addr learned my addr 172.21.15.104:0/107238646 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126bfff700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 0x7f126c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 -- 172.21.15.104:0/107238646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 msgr2=0x7f126c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 0x7f126c0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 -- 172.21.15.104:0/107238646 >> v1:172.21.15.104:6789/0 conn(0x7f126c06cde0 legacy=0x7f126c069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 -- 172.21.15.104:0/107238646 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c0b1e80 con 0x7f126c15c580 2023-06-10T19:00:03.634 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126b7fe700 1 -- 172.21.15.104:0/107238646 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485104815 0 0) 0x7f126c06e250 con 0x7f126c06cde0 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126bfff700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 0x7f126c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f1270b7d700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c15c580 0x7f126c0b1070 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f1260009820 tx=0x7f1260009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edeeb6d66e9463b3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126b7fe700 1 -- 172.21.15.104:0/107238646 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260005e30 con 0x7f126c15c580 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126b7fe700 1 -- 172.21.15.104:0/107238646 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260006030 con 0x7f126c15c580 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.625+0000 7f126b7fe700 1 -- 172.21.15.104:0/107238646 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12600063e0 con 0x7f126c15c580 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 -- 172.21.15.104:0/107238646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c15c580 msgr2=0x7f126c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c15c580 0x7f126c0b1070 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f1260009820 tx=0x7f1260009b30 comp rx=0 tx=0).stop 2023-06-10T19:00:03.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 -- 172.21.15.104:0/107238646 shutdown_connections 2023-06-10T19:00:03.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c15c580 0x7f126c0b1070 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 --2- 172.21.15.104:0/107238646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c06e6e0 0x7f126c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 -- 172.21.15.104:0/107238646 >> 172.21.15.104:0/107238646 conn(0x7f126c06b8a0 msgr2=0x7f126c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:03.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 -- 172.21.15.104:0/107238646 shutdown_connections 2023-06-10T19:00:03.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.626+0000 7f1272de1700 1 -- 172.21.15.104:0/107238646 wait complete. 2023-06-10T19:00:03.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.627+0000 7f1272de1700 1 Processor -- start 2023-06-10T19:00:03.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.627+0000 7f1272de1700 1 -- start start 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 0x7f126c0b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c15c580 0x7f126c0ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c158150 con 0x7f126c06cde0 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c1f55a0 con 0x7f126c06e6e0 2023-06-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1272de1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c1f5880 con 0x7f126c15c580 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42130/0 (socket says 172.21.15.104:42130) 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 -- 172.21.15.104:0/325082795 learned_addr learned my addr 172.21.15.104:0/325082795 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f127137e700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c15c580 0x7f126c0ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f126bfff700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 0x7f126c0b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 -- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c15c580 msgr2=0x7f126c0ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c15c580 0x7f126c0ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 -- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 msgr2=0x7f126c0b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 0x7f126c0b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f1270b7d700 1 -- 172.21.15.104:0/325082795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1260009430 con 0x7f126c06cde0 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.628+0000 7f126bfff700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 0x7f126c0b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f1270b7d700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f1260013040 tx=0x7f1260004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260025430 con 0x7f126c06cde0 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260020070 con 0x7f126c06cde0 2023-06-10T19:00:03.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c1f5b60 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12600257f0 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.629+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c1f6120 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.630+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c1f62e0 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.630+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1260022030 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.630+0000 7f12697fa700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 0x7f125006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.631+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f126008acf0 con 0x7f126c06cde0 2023-06-10T19:00:03.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.631+0000 7f126bfff700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 0x7f125006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.631+0000 7f126bfff700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 0x7f125006d190 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f126c15b940 tx=0x7f1258008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.635+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f126008f070 con 0x7f126c06cde0 2023-06-10T19:00:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.821+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f126c066f10 con 0x7f126c06cde0 2023-06-10T19:00:03.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.822+0000 7f12697fa700 1 -- 172.21.15.104:0/325082795 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1260055cc0 con 0x7f126c06cde0 2023-06-10T19:00:03.829 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T19:00:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 msgr2=0x7f125006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 0x7f125006d190 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f126c15b940 tx=0x7f1258008040 comp rx=0 tx=0).stop 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 msgr2=0x7f126c157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f1260013040 tx=0x7f1260004750 comp rx=0 tx=0).stop 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 shutdown_connections 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f126c15c580 0x7f126c0ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f126c06cde0 0x7f126c157c10 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f125006ace0 0x7f125006d190 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 --2- 172.21.15.104:0/325082795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f126c06e6e0 0x7f126c0b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 >> 172.21.15.104:0/325082795 conn(0x7f126c06b8a0 msgr2=0x7f126c158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:03.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 shutdown_connections 2023-06-10T19:00:03.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.824+0000 7f1272de1700 1 -- 172.21.15.104:0/325082795 wait complete. 2023-06-10T19:00:03.842 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T19:00:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.959+0000 7f283d299700 1 Processor -- start 2023-06-10T19:00:03.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.959+0000 7f283d299700 1 -- start start 2023-06-10T19:00:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f283d299700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f283d299700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 0x7f28380b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f283d299700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28380af530 con 0x7f28380d25c0 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f283d299700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28380af810 con 0x7f28380b05d0 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f283d299700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28380afaf0 con 0x7f28380d04f0 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f28377fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 0x7f28380b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:33290/0 (socket says 172.21.15.104:33290) 2023-06-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f28377fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 0x7f28380b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:57874/0 (socket says 172.21.15.104:57874) 2023-06-10T19:00:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f2836ffd700 1 -- 172.21.15.104:0/4147597881 learned_addr learned my addr 172.21.15.104:0/4147597881 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:03.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.960+0000 7f28367fc700 1 --1- 172.21.15.104:0/4147597881 >> v1:172.21.15.114:6789/0 conn(0x7f28380d25c0 0x7f28380d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:45812/0 (socket says 172.21.15.104:45812) 2023-06-10T19:00:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2836ffd700 1 -- 172.21.15.104:0/4147597881 >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 msgr2=0x7f28380b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2836ffd700 1 --2- 172.21.15.104:0/4147597881 >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 0x7f28380b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2836ffd700 1 -- 172.21.15.104:0/4147597881 >> v1:172.21.15.114:6789/0 conn(0x7f28380d25c0 legacy=0x7f28380d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2836ffd700 1 -- 172.21.15.104:0/4147597881 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28380afdd0 con 0x7f28380d04f0 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2836ffd700 1 --2- 172.21.15.104:0/4147597881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f282c006fd0 tx=0x7f282c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7376491ae7c3f45 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2835ffb700 1 -- 172.21.15.104:0/4147597881 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c005e30 con 0x7f28380d04f0 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2835ffb700 1 -- 172.21.15.104:0/4147597881 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c004030 con 0x7f28380d04f0 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f2835ffb700 1 -- 172.21.15.104:0/4147597881 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0043e0 con 0x7f28380d04f0 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 -- 172.21.15.104:0/4147597881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 msgr2=0x7f28380d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 --2- 172.21.15.104:0/4147597881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f282c006fd0 tx=0x7f282c009bd0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 -- 172.21.15.104:0/4147597881 shutdown_connections 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 --2- 172.21.15.104:0/4147597881 >> v2:172.21.15.114:3300/0 conn(0x7f28380b05d0 0x7f28380b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 --2- 172.21.15.104:0/4147597881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380d34d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 -- 172.21.15.104:0/4147597881 >> 172.21.15.104:0/4147597881 conn(0x7f2838071ec0 msgr2=0x7f283809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.961+0000 7f283d299700 1 -- 172.21.15.104:0/4147597881 shutdown_connections 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.962+0000 7f283d299700 1 -- 172.21.15.104:0/4147597881 wait complete. 2023-06-10T19:00:03.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.963+0000 7f283d299700 1 Processor -- start 2023-06-10T19:00:03.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.963+0000 7f283d299700 1 -- start start 2023-06-10T19:00:03.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28380b05d0 0x7f28381f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28380b1f50 con 0x7f28380d25c0 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28380b1340 con 0x7f28380b05d0 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28380b1620 con 0x7f28380d04f0 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42146/0 (socket says 172.21.15.104:42146) 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 -- 172.21.15.104:0/2917647038 learned_addr learned my addr 172.21.15.104:0/2917647038 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f2836ffd700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28380b05d0 0x7f28381f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28367fc700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 -- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 msgr2=0x7f28380b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 -- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28380b05d0 msgr2=0x7f28381f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:03.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28380b05d0 0x7f28381f1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 -- 172.21.15.104:0/2917647038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c009500 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f28377fe700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f283000efc0 tx=0x7f283000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830009ea0 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283000f040 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28380b1900 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000ced0 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.964+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28381f1d60 con 0x7f28380d25c0 2023-06-10T19:00:03.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.965+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2838067010 con 0x7f28380d25c0 2023-06-10T19:00:03.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.966+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2830014480 con 0x7f28380d25c0 2023-06-10T19:00:03.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.966+0000 7f281f7fe700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 0x7f282406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:03.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.966+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f283008a0e0 con 0x7f28380d25c0 2023-06-10T19:00:03.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.967+0000 7f2836ffd700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 0x7f282406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:03.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.967+0000 7f2836ffd700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 0x7f282406d160 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f282c006fd0 tx=0x7f282c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:03.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:03.971+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2830055020 con 0x7f28380d25c0 2023-06-10T19:00:04.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.158+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f283808ccb0 con 0x7f28380d25c0 2023-06-10T19:00:04.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.158+0000 7f281f7fe700 1 -- 172.21.15.104:0/2917647038 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v188) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2830058640 con 0x7f28380d25c0 2023-06-10T19:00:04.165 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 msgr2=0x7f282406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 0x7f282406d160 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f282c006fd0 tx=0x7f282c008070 comp rx=0 tx=0).stop 2023-06-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 msgr2=0x7f2838147750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f283000efc0 tx=0x7f283000c3a0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 shutdown_connections 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f282406acb0 0x7f282406d160 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28380b05d0 0x7f28381f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28380d25c0 0x7f2838147750 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 --2- 172.21.15.104:0/2917647038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28380d04f0 0x7f28380b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 >> 172.21.15.104:0/2917647038 conn(0x7f2838071ec0 msgr2=0x7f28380cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 shutdown_connections 2023-06-10T19:00:04.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.160+0000 7f283d299700 1 -- 172.21.15.104:0/2917647038 wait complete. 2023-06-10T19:00:04.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.295+0000 7fb007ee0700 1 Processor -- start 2023-06-10T19:00:04.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.295+0000 7fb007ee0700 1 -- start start 2023-06-10T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb000157b30 0x7fb0000ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 0x7fb0000d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb00015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000ced20 con 0x7fb000157b30 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000ad3c0 con 0x7fb0000ce880 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb007ee0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb000157f00 con 0x7fb0000d0830 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb000157b30 0x7fb0000ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb000157b30 0x7fb0000ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42150/0 (socket says 172.21.15.104:42150) 2023-06-10T19:00:04.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 -- 172.21.15.104:0/4262255444 learned_addr learned my addr 172.21.15.104:0/4262255444 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb00547b700 1 --2- 172.21.15.104:0/4262255444 >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 0x7fb0000d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 -- 172.21.15.104:0/4262255444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 msgr2=0x7fb00015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb00647d700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb00015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb00015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 -- 172.21.15.104:0/4262255444 >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 msgr2=0x7fb0000d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 --2- 172.21.15.104:0/4262255444 >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 0x7fb0000d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 -- 172.21.15.104:0/4262255444 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000b0170 con 0x7fb000157b30 2023-06-10T19:00:04.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb00547b700 1 --2- 172.21.15.104:0/4262255444 >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 0x7fb0000d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb005c7c700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb0000ce340 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fb00015da10 tx=0x7faffc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586a01d24d949fa3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb004c7a700 1 -- 172.21.15.104:0/4262255444 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00b870 con 0x7fb000157b30 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb004c7a700 1 -- 172.21.15.104:0/4262255444 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00b9d0 con 0x7fb000157b30 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.296+0000 7fb004c7a700 1 -- 172.21.15.104:0/4262255444 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00bd60 con 0x7fb000157b30 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 -- 172.21.15.104:0/4262255444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 msgr2=0x7fb0000ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb0000ce340 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fb00015da10 tx=0x7faffc009bd0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 -- 172.21.15.104:0/4262255444 shutdown_connections 2023-06-10T19:00:04.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb0000ce340 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 --2- 172.21.15.104:0/4262255444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb00015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 --2- 172.21.15.104:0/4262255444 >> v2:172.21.15.114:3300/0 conn(0x7fb0000ce880 0x7fb0000d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 -- 172.21.15.104:0/4262255444 >> 172.21.15.104:0/4262255444 conn(0x7fb000089fd0 msgr2=0x7fb00008a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 -- 172.21.15.104:0/4262255444 shutdown_connections 2023-06-10T19:00:04.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.297+0000 7fb007ee0700 1 -- 172.21.15.104:0/4262255444 wait complete. 2023-06-10T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.298+0000 7fb007ee0700 1 Processor -- start 2023-06-10T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.298+0000 7fb007ee0700 1 -- start start 2023-06-10T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 0x7fb000152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb000159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001f41a0 con 0x7fb000157b30 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001f3ec0 con 0x7fb0000ce880 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb007ee0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000c4300 con 0x7fb0000d0830 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42160/0 (socket says 172.21.15.104:42160) 2023-06-10T19:00:04.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 -- 172.21.15.104:0/2043264655 learned_addr learned my addr 172.21.15.104:0/2043264655 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00547b700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb000159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb005c7c700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 0x7fb000152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 -- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 msgr2=0x7fb000159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb000159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 -- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 msgr2=0x7fb000152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 0x7fb000152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 -- 172.21.15.104:0/2043264655 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc009550 con 0x7fb000157b30 2023-06-10T19:00:04.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb005c7c700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 0x7fb000152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7fb00647d700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7faff800cd70 tx=0x7faff800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.299+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8003a20 con 0x7fb000157b30 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.300+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff8003b80 con 0x7fb000157b30 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.300+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8004020 con 0x7fb000157b30 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.300+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0000c45e0 con 0x7fb000157b30 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.300+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb000153390 con 0x7fb000157b30 2023-06-10T19:00:04.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.301+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0000c4160 con 0x7fb000157b30 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.301+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7faff8004180 con 0x7fb000157b30 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.301+0000 7faff67fc700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 0x7fafe406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.302+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7faff8012070 con 0x7fb000157b30 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.302+0000 7fb005c7c700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 0x7fafe406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.302+0000 7fb005c7c700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 0x7fafe406d190 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7faffc014040 tx=0x7faffc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.306+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb0000c4160 con 0x7fb000157b30 2023-06-10T19:00:04.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.493+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0000c4b00 con 0x7fb000157b30 2023-06-10T19:00:04.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.493+0000 7faff67fc700 1 -- 172.21.15.104:0/2043264655 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v188) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0000c4b00 con 0x7fb000157b30 2023-06-10T19:00:04.500 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T19:00:04.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 msgr2=0x7fafe406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 0x7fafe406d190 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7faffc014040 tx=0x7faffc008030 comp rx=0 tx=0).stop 2023-06-10T19:00:04.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 msgr2=0x7fb000159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7faff800cd70 tx=0x7faff800a570 comp rx=0 tx=0).stop 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 shutdown_connections 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fafe406ace0 0x7fafe406d190 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0000ce880 0x7fb000152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb000157b30 0x7fb000159690 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 --2- 172.21.15.104:0/2043264655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0000d0830 0x7fb000159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.495+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 >> 172.21.15.104:0/2043264655 conn(0x7fb000089fd0 msgr2=0x7fb0000ad530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.496+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 shutdown_connections 2023-06-10T19:00:04.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.496+0000 7fb007ee0700 1 -- 172.21.15.104:0/2043264655 wait complete. 2023-06-10T19:00:04.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T19:00:04.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.631+0000 7f02cc72a700 1 Processor -- start 2023-06-10T19:00:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.631+0000 7f02cc72a700 1 -- start start 2023-06-10T19:00:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cc72a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02c40b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cc72a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02c406cde0 0x7f02c40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cc72a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c406e250 con 0x7f02c415c580 2023-06-10T19:00:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cc72a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c415c950 con 0x7f02c406cde0 2023-06-10T19:00:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cc72a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40b1e80 con 0x7f02c406e6e0 2023-06-10T19:00:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02ca4c6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f02c415c580 0x7f02c40b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39992/0 (socket says 172.21.15.104:39992) 2023-06-10T19:00:04.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02ca4c6700 1 -- 172.21.15.104:0/3522919156 learned_addr learned my addr 172.21.15.104:0/3522919156 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617927038 0 0) 0x7f02c406e250 con 0x7f02c415c580 2023-06-10T19:00:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b00035b0 con 0x7f02c415c580 2023-06-10T19:00:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3522919156 >> v2:172.21.15.114:3300/0 conn(0x7f02c406cde0 0x7f02c40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.632+0000 7f02c9cc5700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02c40b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483347100 0 0) 0x7f02b00035b0 con 0x7f02c415c580 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 msgr2=0x7f02c40b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02c40b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 >> v2:172.21.15.114:3300/0 conn(0x7f02c406cde0 msgr2=0x7f02c40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 --2- 172.21.15.104:0/3522919156 >> v2:172.21.15.114:3300/0 conn(0x7f02c406cde0 0x7f02c40ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c406d920 con 0x7f02c415c580 2023-06-10T19:00:04.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f02b40017f0 con 0x7f02c415c580 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 >> v1:172.21.15.104:6789/0 conn(0x7f02c415c580 legacy=0x7f02c40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0003a00 0x7f02b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02b0004d90 0x7f02b0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c415c950 con 0x7f02b0004d90 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40b1e80 con 0x7f02b0003a00 2023-06-10T19:00:04.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b000ba10 con 0x7f02c406e6e0 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02b0004d90 0x7f02b0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c9cc5700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02ca4c6700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0003a00 0x7f02b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 -- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 msgr2=0x7f02b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02c406e6e0 0x7f02b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 -- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0003a00 msgr2=0x7f02b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0003a00 0x7f02b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 -- 172.21.15.104:0/3522919156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b00035b0 con 0x7f02b0004d90 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02b0004d90 0x7f02b0005200 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009fc0 tx=0x7f02b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c8eb09da5c2518 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc00ebd0 con 0x7f02b0004d90 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc00f040 con 0x7f02b0004d90 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.633+0000 7f02c94c4700 1 -- 172.21.15.104:0/3522919156 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc014480 con 0x7f02b0004d90 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 -- 172.21.15.104:0/3522919156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02b0004d90 msgr2=0x7f02b0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3522919156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02b0004d90 0x7f02b0005200 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009fc0 tx=0x7f02b0005880 comp rx=0 tx=0).stop 2023-06-10T19:00:04.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cacc7700 1 -- 172.21.15.104:0/3522919156 reap_dead start 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 -- 172.21.15.104:0/3522919156 shutdown_connections 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 -- 172.21.15.104:0/3522919156 >> 172.21.15.104:0/3522919156 conn(0x7f02c406b8a0 msgr2=0x7f02c40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 -- 172.21.15.104:0/3522919156 shutdown_connections 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.634+0000 7f02cc72a700 1 -- 172.21.15.104:0/3522919156 wait complete. 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.635+0000 7f02cc72a700 1 Processor -- start 2023-06-10T19:00:04.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.635+0000 7f02cc72a700 1 -- start start 2023-06-10T19:00:04.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b0004d90 0x7f02c40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 0x7f02c41498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41e5780 con 0x7f02c41e5c10 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4147270 con 0x7f02c4146df0 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cc72a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41e4e50 con 0x7f02b0004d90 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42180/0 (socket says 172.21.15.104:42180) 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 -- 172.21.15.104:0/3537329200 learned_addr learned my addr 172.21.15.104:0/3537329200 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 0x7f02c41498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02ca4c6700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b0004d90 0x7f02c40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 -- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b0004d90 msgr2=0x7f02c40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b0004d90 0x7f02c40cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 -- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 msgr2=0x7f02c41498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 0x7f02c41498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02c9cc5700 1 -- 172.21.15.104:0/3537329200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c41e5130 con 0x7f02c41e5c10 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.636+0000 7f02cacc7700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 0x7f02c41498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02c9cc5700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f02c0011aa0 tx=0x7f02c0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c000f980 con 0x7f02c41e5c10 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c000fae0 con 0x7f02c41e5c10 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c41e5410 con 0x7f02c41e5c10 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c000fe90 con 0x7f02c41e5c10 2023-06-10T19:00:04.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.637+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c40cb570 con 0x7f02c41e5c10 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.638+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c41e5930 con 0x7f02c41e5c10 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.638+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f02c0014440 con 0x7f02c41e5c10 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.638+0000 7f02baffd700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 0x7f02a806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.639+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f02c0089c10 con 0x7f02c41e5c10 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.639+0000 7f02ca4c6700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 0x7f02a806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.639+0000 7f02ca4c6700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 0x7f02a806d1c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000ef0 tx=0x7f02b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.653 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.643+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f02c41e5930 con 0x7f02c41e5c10 2023-06-10T19:00:04.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.827+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02c40cb570 con 0x7f02c41e5c10 2023-06-10T19:00:04.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.828+0000 7f02baffd700 1 -- 172.21.15.104:0/3537329200 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v188) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02c40cb570 con 0x7f02c41e5c10 2023-06-10T19:00:04.834 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T19:00:04.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 msgr2=0x7f02a806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 0x7f02a806d1c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000ef0 tx=0x7f02b4009040 comp rx=0 tx=0).stop 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 msgr2=0x7f02c41468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f02c0011aa0 tx=0x7f02c0011e60 comp rx=0 tx=0).stop 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 shutdown_connections 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f02a806ad10 0x7f02a806d1c0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c4146df0 0x7f02c41498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b0004d90 0x7f02c40cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 --2- 172.21.15.104:0/3537329200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c41e5c10 0x7f02c41468b0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 >> 172.21.15.104:0/3537329200 conn(0x7f02c406b8a0 msgr2=0x7f02c40cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 shutdown_connections 2023-06-10T19:00:04.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.830+0000 7f02cc72a700 1 -- 172.21.15.104:0/3537329200 wait complete. 2023-06-10T19:00:04.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-10T19:00:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.962+0000 7f7e0b74c700 1 Processor -- start 2023-06-10T19:00:04.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 -- start start 2023-06-10T19:00:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e041567f0 0x7f7e040cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7e040705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e04070b20 con 0x7f7e0415c7a0 2023-06-10T19:00:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e040cb2d0 con 0x7f7e041567f0 2023-06-10T19:00:04.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.963+0000 7f7e0b74c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0415cca0 con 0x7f7e040701a0 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e08ce7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7e0415c7a0 0x7f7e040cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40012/0 (socket says 172.21.15.104:40012) 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e08ce7700 1 -- 172.21.15.104:0/114104443 learned_addr learned my addr 172.21.15.104:0/114104443 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832137326 0 0) 0x7f7e04070b20 con 0x7f7e0415c7a0 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df00035b0 con 0x7f7e0415c7a0 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7e040705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e094e8700 1 --2- 172.21.15.104:0/114104443 >> v2:172.21.15.114:3300/0 conn(0x7f7e041567f0 0x7f7e040cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459444480 0 0) 0x7f7df00035b0 con 0x7f7e0415c7a0 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 msgr2=0x7f7e040705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7e040705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 >> v2:172.21.15.114:3300/0 conn(0x7f7e041567f0 msgr2=0x7f7e040cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 --2- 172.21.15.104:0/114104443 >> v2:172.21.15.114:3300/0 conn(0x7f7e041567f0 0x7f7e040cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0415cf80 con 0x7f7e0415c7a0 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f7e00001670 con 0x7f7e0415c7a0 2023-06-10T19:00:04.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 >> v1:172.21.15.104:6789/0 conn(0x7f7e0415c7a0 legacy=0x7f7e040cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7df0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df0004030 0x7f7df0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7df0004d90 0x7f7df0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040cb2d0 con 0x7f7df0004d90 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0415cca0 con 0x7f7df0004030 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df000bba0 con 0x7f7e040701a0 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e094e8700 1 --2- 172.21.15.104:0/114104443 >> v2:172.21.15.114:3300/0 conn(0x7f7e041567f0 0x7f7e040cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:04.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7df0004d90 0x7f7df0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e08ce7700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df0004030 0x7f7df0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e094e8700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7df0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 -- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 msgr2=0x7f7df0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e040701a0 0x7f7df0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 -- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df0004030 msgr2=0x7f7df0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df0004030 0x7f7df0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.964+0000 7f7e09ce9700 1 -- 172.21.15.104:0/114104443 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00035b0 con 0x7f7df0004d90 2023-06-10T19:00:04.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7df0004d90 0x7f7df0005200 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc007330 tx=0x7f7df0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81455f9dde33666 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc007e50 con 0x7f7df0004d90 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc014400 con 0x7f7df0004d90 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7dfbfff700 1 -- 172.21.15.104:0/114104443 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc014790 con 0x7f7df0004d90 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 -- 172.21.15.104:0/114104443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7df0004d90 msgr2=0x7f7df0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/114104443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7df0004d90 0x7f7df0005200 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc007330 tx=0x7f7df0005880 comp rx=0 tx=0).stop 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e09ce9700 1 -- 172.21.15.104:0/114104443 reap_dead start 2023-06-10T19:00:04.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 -- 172.21.15.104:0/114104443 shutdown_connections 2023-06-10T19:00:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 -- 172.21.15.104:0/114104443 >> 172.21.15.104:0/114104443 conn(0x7f7e0406e370 msgr2=0x7f7e0406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 -- 172.21.15.104:0/114104443 shutdown_connections 2023-06-10T19:00:04.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.965+0000 7f7e0b74c700 1 -- 172.21.15.104:0/114104443 wait complete. 2023-06-10T19:00:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 Processor -- start 2023-06-10T19:00:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 -- start start 2023-06-10T19:00:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e041f16c0 0x7f7e040b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 0x7f7e04151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040b4720 con 0x7f7e040c8000 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040c89e0 con 0x7f7e040b3570 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e0b74c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040b5080 con 0x7f7e041f16c0 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e09ce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e09ce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42200/0 (socket says 172.21.15.104:42200) 2023-06-10T19:00:04.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.967+0000 7f7e09ce9700 1 -- 172.21.15.104:0/1800504613 learned_addr learned my addr 172.21.15.104:0/1800504613 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e094e8700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e041f16c0 0x7f7e040b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e08ce7700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 0x7f7e04151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 -- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e041f16c0 msgr2=0x7f7e040b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e041f16c0 0x7f7e040b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 -- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 msgr2=0x7f7e04151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 0x7f7e04151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:04.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 -- 172.21.15.104:0/1800504613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040b3aa0 con 0x7f7e040c8000 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e08ce7700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 0x7f7e04151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e09ce9700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7df000baf0 tx=0x7f7dfc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc007000 con 0x7f7e040c8000 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc014980 con 0x7f7e040c8000 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040b3d80 con 0x7f7e040c8000 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc014d30 con 0x7f7e040c8000 2023-06-10T19:00:04.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.968+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e040b4340 con 0x7f7e040c8000 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.969+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e040b4860 con 0x7f7e040c8000 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.970+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7dfc003f20 con 0x7f7e040c8000 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.970+0000 7f7df9ffb700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 0x7f7de806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.970+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7dfc08ab60 con 0x7f7e040c8000 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.970+0000 7f7e094e8700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 0x7f7de806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:04.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.971+0000 7f7e094e8700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 0x7f7de806d190 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009a80 tx=0x7f7df4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:04.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:04.974+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7e040b4860 con 0x7f7e040c8000 2023-06-10T19:00:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.174+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7e040b4860 con 0x7f7e040c8000 2023-06-10T19:00:05.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.175+0000 7f7df9ffb700 1 -- 172.21.15.104:0/1800504613 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v188) v1 ==== 80+0+56 (secure 0 0 0) 0x7f7e040b4860 con 0x7f7e040c8000 2023-06-10T19:00:05.181 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.1:0/675086267 2023-06-10T19:59:40.323449+0000 2023-06-10T19:00:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 msgr2=0x7f7de806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 0x7f7de806d190 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009a80 tx=0x7f7df4008040 comp rx=0 tx=0).stop 2023-06-10T19:00:05.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 msgr2=0x7f7e040c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7df000baf0 tx=0x7f7dfc004b40 comp rx=0 tx=0).stop 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 shutdown_connections 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e040c8000 0x7f7e040c84a0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e041f16c0 0x7f7e040b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7de806ace0 0x7f7de806d190 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 --2- 172.21.15.104:0/1800504613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e040b3570 0x7f7e04151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 >> 172.21.15.104:0/1800504613 conn(0x7f7e0406e370 msgr2=0x7f7e0406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 shutdown_connections 2023-06-10T19:00:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.177+0000 7f7e0b74c700 1 -- 172.21.15.104:0/1800504613 wait complete. 2023-06-10T19:00:05.185 INFO:teuthology.orchestra.run.smithi104.stderr:listed 1 entries 2023-06-10T19:00:05.195 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2023-06-10T19:00:05.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:05.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-06-10T19:00:05.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi114.stderr:ceph-fuse[118542]: fuse finished with error 0 and tester_r 0 2023-06-10T19:00:05.242 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:05.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:05.242 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-06-10T19:00:05.371 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T19:00:05.371 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:05.389 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T19:00:05.390 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:05.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:05.390 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-06-10T19:00:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:05.446 INFO:teuthology.orchestra.run.smithi104.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-06-10T19:00:05.446 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T19:00:05.447 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:05.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T19:00:05.466 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:05.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:05.467 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-06-10T19:00:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:05.525 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-06-10T19:00:05.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T19:00:05.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.638+0000 7f8b7fdaa700 1 Processor -- start 2023-06-10T19:00:05.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.638+0000 7f8b7fdaa700 1 -- start start 2023-06-10T19:00:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7fdaa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b780b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7fdaa700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b780b6070 con 0x7f8b780cea60 2023-06-10T19:00:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7fdaa700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b780b5460 con 0x7f8b780b4ff0 2023-06-10T19:00:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7fdaa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780b5740 con 0x7f8b7815a420 2023-06-10T19:00:05.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7e347700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8b780cea60 0x7f8b7815a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40036/0 (socket says 172.21.15.104:40036) 2023-06-10T19:00:05.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.639+0000 7f8b7e347700 1 -- 172.21.15.104:0/1229535302 learned_addr learned my addr 172.21.15.104:0/1229535302 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:05.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7db46700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b780b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348252843 0 0) 0x7f8b780b6070 con 0x7f8b780cea60 2023-06-10T19:00:05.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b640035b0 con 0x7f8b780cea60 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425061939 0 0) 0x7f8b640035b0 con 0x7f8b780cea60 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 msgr2=0x7f8b780b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b780b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 >> v1:172.21.15.114:6789/0 conn(0x7f8b780b4ff0 legacy=0x7f8b780ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780b5a20 con 0x7f8b780cea60 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f8b700019f0 con 0x7f8b780cea60 2023-06-10T19:00:05.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 >> v1:172.21.15.104:6789/0 conn(0x7f8b780cea60 legacy=0x7f8b7815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b64003a30 0x7f8b64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b64004d90 0x7f8b64005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780b5740 con 0x7f8b64004d90 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6400ba10 con 0x7f8b64003a30 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6400bba0 con 0x7f8b7815a420 2023-06-10T19:00:05.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b64004d90 0x7f8b64005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7db46700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b64003a30 0x7f8b64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7d345700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 -- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 msgr2=0x7f8b64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b7815a420 0x7f8b64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 -- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b64003a30 msgr2=0x7f8b64003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b64003a30 0x7f8b64003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 -- 172.21.15.104:0/1229535302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640035b0 con 0x7f8b64004d90 2023-06-10T19:00:05.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.640+0000 7f8b7e347700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b64004d90 0x7f8b64005200 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f8b78071430 tx=0x7f8b64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca0959ab2fa514b server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:05.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70012600 con 0x7f8b64004d90 2023-06-10T19:00:05.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b70012760 con 0x7f8b64004d90 2023-06-10T19:00:05.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7cb44700 1 -- 172.21.15.104:0/1229535302 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70012b10 con 0x7f8b64004d90 2023-06-10T19:00:05.654 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/1229535302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b64004d90 msgr2=0x7f8b64005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/1229535302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b64004d90 0x7f8b64005200 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f8b78071430 tx=0x7f8b64005880 comp rx=0 tx=0).stop 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7e347700 1 -- 172.21.15.104:0/1229535302 reap_dead start 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/1229535302 shutdown_connections 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/1229535302 >> 172.21.15.104:0/1229535302 conn(0x7f8b78089fd0 msgr2=0x7f8b780d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/1229535302 shutdown_connections 2023-06-10T19:00:05.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.641+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/1229535302 wait complete. 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.642+0000 7f8b7fdaa700 1 Processor -- start 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.642+0000 7f8b7fdaa700 1 -- start start 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 0x7f8b780c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b78153ab0 0x7f8b780cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c88a0 con 0x7f8b78152e00 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c6860 con 0x7f8b78153ab0 2023-06-10T19:00:05.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7fdaa700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c6b40 con 0x7f8b78153560 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7db46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7db46700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42216/0 (socket says 172.21.15.104:42216) 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7db46700 1 -- 172.21.15.104:0/3235542401 learned_addr learned my addr 172.21.15.104:0/3235542401 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7e347700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b78153ab0 0x7f8b780cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.643+0000 7f8b7d345700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 0x7f8b780c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 -- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 msgr2=0x7f8b780c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 0x7f8b780c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 -- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b78153ab0 msgr2=0x7f8b780cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b78153ab0 0x7f8b780cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 -- 172.21.15.104:0/3235542401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780c6e20 con 0x7f8b78152e00 2023-06-10T19:00:05.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7d345700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 0x7f8b780c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7db46700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f8b68011aa0 tx=0x7f8b68011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800f940 con 0x7f8b78152e00 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6800faa0 con 0x7f8b78152e00 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b780c7100 con 0x7f8b78152e00 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800fe50 con 0x7f8b78152e00 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.644+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b781531d0 con 0x7f8b78152e00 2023-06-10T19:00:05.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.645+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b780c8a10 con 0x7f8b78152e00 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.645+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8b68014440 con 0x7f8b78152e00 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.645+0000 7f8b6e7fc700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 0x7f8b5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.646+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8b68089cd0 con 0x7f8b78152e00 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.646+0000 7f8b7d345700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 0x7f8b5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.646+0000 7f8b7d345700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 0x7f8b5c06d190 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8b74007fd0 tx=0x7f8b74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:05.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.650+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8b680587f0 con 0x7f8b78152e00 2023-06-10T19:00:05.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.878+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b78074050 con 0x7f8b78152e00 2023-06-10T19:00:05.885 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.879+0000 7f8b6e7fc700 1 -- 172.21.15.104:0/3235542401 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f8b68058380 con 0x7f8b78152e00 2023-06-10T19:00:05.886 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:05.886 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T19:00:05.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 msgr2=0x7f8b5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 0x7f8b5c06d190 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8b74007fd0 tx=0x7f8b74009040 comp rx=0 tx=0).stop 2023-06-10T19:00:05.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 msgr2=0x7f8b780c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:05.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f8b68011aa0 tx=0x7f8b68011db0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 shutdown_connections 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8b5c06ace0 0x7f8b5c06d190 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b78153ab0 0x7f8b780cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b78152e00 0x7f8b780c4400 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 --2- 172.21.15.104:0/3235542401 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8b78153560 0x7f8b780c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 >> 172.21.15.104:0/3235542401 conn(0x7f8b78089fd0 msgr2=0x7f8b78157c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 shutdown_connections 2023-06-10T19:00:05.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:05.881+0000 7f8b7fdaa700 1 -- 172.21.15.104:0/3235542401 wait complete. 2023-06-10T19:00:05.890 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T19:00:05.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.013+0000 7f232baef700 1 Processor -- start 2023-06-10T19:00:06.020 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 -- start start 2023-06-10T19:00:06.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f232415c500 0x7f23240cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2324069e70 0x7f232406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232406dcb0 con 0x7f23240ce040 2023-06-10T19:00:06.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232406e850 con 0x7f2324069e70 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232baef700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232406a830 con 0x7f232415c500 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.014+0000 7f232a08c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2324069e70 0x7f232406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232908a700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f23240ce040 0x7f232406e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40054/0 (socket says 172.21.15.104:40054) 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f2324069e70 0x7f232406a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42228/0 (socket says 172.21.15.104:42228) 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232908a700 1 -- 172.21.15.104:0/975588446 learned_addr learned my addr 172.21.15.104:0/975588446 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232988b700 1 --2- 172.21.15.104:0/975588446 >> v2:172.21.15.114:3300/0 conn(0x7f232415c500 0x7f23240cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.023 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 -- 172.21.15.104:0/975588446 >> v2:172.21.15.114:3300/0 conn(0x7f232415c500 msgr2=0x7f23240cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 --2- 172.21.15.104:0/975588446 >> v2:172.21.15.114:3300/0 conn(0x7f232415c500 0x7f23240cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 -- 172.21.15.104:0/975588446 >> v1:172.21.15.104:6789/0 conn(0x7f23240ce040 legacy=0x7f232406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f2328889700 1 -- 172.21.15.104:0/975588446 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804263319 0 0) 0x7f232406dcb0 con 0x7f23240ce040 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 -- 172.21.15.104:0/975588446 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240ce470 con 0x7f2324069e70 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f232a08c700 1 --2- 172.21.15.104:0/975588446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2324069e70 0x7f232406a2f0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f2324067130 tx=0x7f231c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6abbe3a3000bb92 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f2328889700 1 -- 172.21.15.104:0/975588446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c00e070 con 0x7f2324069e70 2023-06-10T19:00:06.024 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f2328889700 1 -- 172.21.15.104:0/975588446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c00bbd0 con 0x7f2324069e70 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.015+0000 7f2328889700 1 -- 172.21.15.104:0/975588446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c012430 con 0x7f2324069e70 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 -- 172.21.15.104:0/975588446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2324069e70 msgr2=0x7f232406a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 --2- 172.21.15.104:0/975588446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2324069e70 0x7f232406a2f0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f2324067130 tx=0x7f231c00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 -- 172.21.15.104:0/975588446 shutdown_connections 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 --2- 172.21.15.104:0/975588446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2324069e70 0x7f232406a2f0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 --2- 172.21.15.104:0/975588446 >> v2:172.21.15.114:3300/0 conn(0x7f232415c500 0x7f23240cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 -- 172.21.15.104:0/975588446 >> 172.21.15.104:0/975588446 conn(0x7f232406b8a0 msgr2=0x7f23240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.025 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 -- 172.21.15.104:0/975588446 shutdown_connections 2023-06-10T19:00:06.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.016+0000 7f232baef700 1 -- 172.21.15.104:0/975588446 wait complete. 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.017+0000 7f232baef700 1 Processor -- start 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.017+0000 7f232baef700 1 -- start start 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2324069e70 0x7f2324155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 0x7f23240ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324159be0 con 0x7f23240ce040 2023-06-10T19:00:06.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241558b0 con 0x7f2324069e70 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2324159010 con 0x7f232415c500 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42232/0 (socket says 172.21.15.104:42232) 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 -- 172.21.15.104:0/1067901900 learned_addr learned my addr 172.21.15.104:0/1067901900 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232988b700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2324069e70 0x7f2324155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232a08c700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 0x7f23240ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 -- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 msgr2=0x7f23240ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 0x7f23240ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 -- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2324069e70 msgr2=0x7f2324155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2324069e70 0x7f2324155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 -- 172.21.15.104:0/1067901900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c009a70 con 0x7f23240ce040 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232a08c700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 0x7f23240ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232908a700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f23200011e0 tx=0x7f232000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320004c80 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320004de0 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23241592b0 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320007310 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.018+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324159870 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.019+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324074050 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.020+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2320007470 con 0x7f23240ce040 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.020+0000 7f231a7fc700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 0x7f230806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.020+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2320089360 con 0x7f23240ce040 2023-06-10T19:00:06.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.021+0000 7f232988b700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 0x7f230806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.021+0000 7f232988b700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 0x7f230806d190 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f2314006fd0 tx=0x7f2314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.032 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.025+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2320054290 con 0x7f23240ce040 2023-06-10T19:00:06.217 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.211+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23240ace40 con 0x7f23240ce040 2023-06-10T19:00:06.218 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.212+0000 7f231a7fc700 1 -- 172.21.15.104:0/1067901900 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18677 (secure 0 0 0) 0x7f23200578b0 con 0x7f23240ce040 2023-06-10T19:00:06.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:06.219 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":188,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:40.713132+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:06.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.214+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 msgr2=0x7f230806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.214+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 0x7f230806d190 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f2314006fd0 tx=0x7f2314008040 comp rx=0 tx=0).stop 2023-06-10T19:00:06.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.214+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 msgr2=0x7f23240c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.221 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.214+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f23200011e0 tx=0x7f232000ca30 comp rx=0 tx=0).stop 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 shutdown_connections 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f230806ace0 0x7f230806d190 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2324069e70 0x7f2324155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f232415c500 0x7f23240ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 --2- 172.21.15.104:0/1067901900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240ce040 0x7f23240c7480 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 >> 172.21.15.104:0/1067901900 conn(0x7f232406b8a0 msgr2=0x7f232406e040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 shutdown_connections 2023-06-10T19:00:06.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.215+0000 7f232baef700 1 -- 172.21.15.104:0/1067901900 wait complete. 2023-06-10T19:00:06.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-10T18:43:55.391812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-10T18:44:02.212624+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-10T18:59:21.300558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-06-10T18:59:22.213068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-10T19:00:06.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-10T19:00:06.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.351+0000 7fb97335b700 1 Processor -- start 2023-06-10T19:00:06.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.351+0000 7fb97335b700 1 -- start start 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb97335b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 0x7fb96c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb97335b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0ae7e0 con 0x7fb96c06ea70 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb97335b700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c06df90 con 0x7fb96c0aec00 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb97335b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c06e270 con 0x7fb96c15c6f0 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb9708f6700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb96c06ea70 0x7fb96c0ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40066/0 (socket says 172.21.15.104:40066) 2023-06-10T19:00:06.360 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.352+0000 7fb9708f6700 1 -- 172.21.15.104:0/3060252417 learned_addr learned my addr 172.21.15.104:0/3060252417 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb9710f7700 1 --2- 172.21.15.104:0/3060252417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 0x7fb96c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915962128 0 0) 0x7fb96c0ae7e0 con 0x7fb96c06ea70 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9580035b0 con 0x7fb96c06ea70 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133043071 0 0) 0x7fb96c06df90 con 0x7fb96c0aec00 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb9710f7700 1 -- 172.21.15.104:0/3060252417 >> v1:172.21.15.114:6789/0 conn(0x7fb96c0aec00 legacy=0x7fb96c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb9710f7700 1 -- 172.21.15.104:0/3060252417 >> v1:172.21.15.104:6789/0 conn(0x7fb96c06ea70 legacy=0x7fb96c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb9710f7700 1 -- 172.21.15.104:0/3060252417 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c06e550 con 0x7fb96c15c6f0 2023-06-10T19:00:06.361 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb9710f7700 1 --2- 172.21.15.104:0/3060252417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 0x7fb96c0b1070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb95c009850 tx=0x7fb95c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=746f46a419521715 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968001670 con 0x7fb96c15c6f0 2023-06-10T19:00:06.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95c005c30 con 0x7fb96c15c6f0 2023-06-10T19:00:06.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.353+0000 7fb963fff700 1 -- 172.21.15.104:0/3060252417 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c006030 con 0x7fb96c15c6f0 2023-06-10T19:00:06.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 -- 172.21.15.104:0/3060252417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 msgr2=0x7fb96c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 --2- 172.21.15.104:0/3060252417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 0x7fb96c0b1070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb95c009850 tx=0x7fb95c009b60 comp rx=0 tx=0).stop 2023-06-10T19:00:06.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 -- 172.21.15.104:0/3060252417 shutdown_connections 2023-06-10T19:00:06.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 --2- 172.21.15.104:0/3060252417 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c15c6f0 0x7fb96c0b1070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 -- 172.21.15.104:0/3060252417 >> 172.21.15.104:0/3060252417 conn(0x7fb96c06b8a0 msgr2=0x7fb96c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 -- 172.21.15.104:0/3060252417 shutdown_connections 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.354+0000 7fb97335b700 1 -- 172.21.15.104:0/3060252417 wait complete. 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.355+0000 7fb97335b700 1 Processor -- start 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.355+0000 7fb97335b700 1 -- start start 2023-06-10T19:00:06.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c06ea70 0x7fb96c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb96c0aec00 0x7fb96c0c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c1541a0 con 0x7fb96c15c6f0 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0ca960 con 0x7fb96c0aec00 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb97335b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0cac40 con 0x7fb96c06ea70 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42248/0 (socket says 172.21.15.104:42248) 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 -- 172.21.15.104:0/1983135024 learned_addr learned my addr 172.21.15.104:0/1983135024 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9710f7700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c06ea70 0x7fb96c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9708f6700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb96c0aec00 0x7fb96c0c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 -- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c06ea70 msgr2=0x7fb96c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c06ea70 0x7fb96c153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 -- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb96c0aec00 msgr2=0x7fb96c0c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb96c0aec00 0x7fb96c0c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 -- 172.21.15.104:0/1983135024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c009430 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.356+0000 7fb9718f8700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fb96400c820 tx=0x7fb96400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.357+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964003e30 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.357+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640050f0 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.357+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c0caf20 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.357+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640054a0 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.357+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c0b2e00 con 0x7fb96c15c6f0 2023-06-10T19:00:06.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.358+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c074050 con 0x7fb96c15c6f0 2023-06-10T19:00:06.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.358+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb964005600 con 0x7fb96c15c6f0 2023-06-10T19:00:06.369 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.358+0000 7fb961ffb700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 0x7fb95006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.359+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fb964088db0 con 0x7fb96c15c6f0 2023-06-10T19:00:06.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.359+0000 7fb9710f7700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 0x7fb95006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.359+0000 7fb9710f7700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 0x7fb95006d190 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0097a0 tx=0x7fb95c007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.363+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb964053d80 con 0x7fb96c15c6f0 2023-06-10T19:00:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.548+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb96c0d0030 con 0x7fb96c15c6f0 2023-06-10T19:00:06.555 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.549+0000 7fb961ffb700 1 -- 172.21.15.104:0/1983135024 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9640170a0 con 0x7fb96c15c6f0 2023-06-10T19:00:06.555 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2023-06-10T19:00:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.550+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 msgr2=0x7fb95006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.550+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 0x7fb95006d190 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0097a0 tx=0x7fb95c007000 comp rx=0 tx=0).stop 2023-06-10T19:00:06.557 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.550+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 msgr2=0x7fb96c0ad970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.550+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fb96400c820 tx=0x7fb96400cbe0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 shutdown_connections 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb96c15c6f0 0x7fb96c0ad970 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb96c06ea70 0x7fb96c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb95006ace0 0x7fb95006d190 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 --2- 172.21.15.104:0/1983135024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb96c0aec00 0x7fb96c0c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.558 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 >> 172.21.15.104:0/1983135024 conn(0x7fb96c06b8a0 msgr2=0x7fb96c0ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 shutdown_connections 2023-06-10T19:00:06.559 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.551+0000 7fb97335b700 1 -- 172.21.15.104:0/1983135024 wait complete. 2023-06-10T19:00:06.569 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-10T19:00:06.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.686+0000 7f52cfff2700 1 Processor -- start 2023-06-10T19:00:06.692 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.686+0000 7f52cfff2700 1 -- start start 2023-06-10T19:00:06.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cfff2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f52c80cea50 0x7f52c80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.694 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cfff2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80d0830 0x7f52c80b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cfff2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c80ad3c0 con 0x7f52c8157b30 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cfff2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c8157f00 con 0x7f52c80cea50 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cfff2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b0170 con 0x7f52c80d0830 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f52c80cea50 0x7f52c80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cdd8e700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f52c8157b30 0x7f52c80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40080/0 (socket says 172.21.15.104:40080) 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cdd8e700 1 -- 172.21.15.104:0/1843223098 learned_addr learned my addr 172.21.15.104:0/1843223098 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:06.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52ce58f700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80d0830 0x7f52c80b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 -- 172.21.15.104:0/1843223098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80d0830 msgr2=0x7f52c80b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80d0830 0x7f52c80b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 -- 172.21.15.104:0/1843223098 >> v1:172.21.15.104:6789/0 conn(0x7f52c8157b30 legacy=0x7f52c80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52ccd8c700 1 -- 172.21.15.104:0/1843223098 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463805822 0 0) 0x7f52c80ad3c0 con 0x7f52c8157b30 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 -- 172.21.15.104:0/1843223098 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80b0410 con 0x7f52c80cea50 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.687+0000 7f52cd58d700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c80cea50 0x7f52c80d02f0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009840 tx=0x7f52b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b487de9bb64ea58 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52ccd8c700 1 -- 172.21.15.104:0/1843223098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011830 con 0x7f52c80cea50 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52ccd8c700 1 -- 172.21.15.104:0/1843223098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8011990 con 0x7f52c80cea50 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52ccd8c700 1 -- 172.21.15.104:0/1843223098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011d20 con 0x7f52c80cea50 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 -- 172.21.15.104:0/1843223098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c80cea50 msgr2=0x7f52c80d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c80cea50 0x7f52c80d02f0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009840 tx=0x7f52b8009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 -- 172.21.15.104:0/1843223098 shutdown_connections 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80d0830 0x7f52c80b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1843223098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c80cea50 0x7f52c80d02f0 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 -- 172.21.15.104:0/1843223098 >> 172.21.15.104:0/1843223098 conn(0x7f52c8089fd0 msgr2=0x7f52c808a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 -- 172.21.15.104:0/1843223098 shutdown_connections 2023-06-10T19:00:06.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.688+0000 7f52cfff2700 1 -- 172.21.15.104:0/1843223098 wait complete. 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.689+0000 7f52cfff2700 1 Processor -- start 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.689+0000 7f52cfff2700 1 -- start start 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80cea50 0x7f52c8152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52c80d0830 0x7f52c80b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8159840 con 0x7f52c8157b30 2023-06-10T19:00:06.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b63b0 con 0x7f52c80d0830 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cfff2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8158bc0 con 0x7f52c80cea50 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52ce58f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52ce58f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42262/0 (socket says 172.21.15.104:42262) 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52ce58f700 1 -- 172.21.15.104:0/1138077787 learned_addr learned my addr 172.21.15.104:0/1138077787 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cdd8e700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80cea50 0x7f52c8152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.690+0000 7f52cd58d700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52c80d0830 0x7f52c80b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 -- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80cea50 msgr2=0x7f52c8152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80cea50 0x7f52c8152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 -- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52c80d0830 msgr2=0x7f52c80b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52c80d0830 0x7f52c80b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 -- 172.21.15.104:0/1138077787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80094a0 con 0x7f52c8157b30 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52ce58f700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f52c0007e40 tx=0x7f52c000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000eea0 con 0x7f52c8157b30 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0014420 con 0x7f52c8157b30 2023-06-10T19:00:06.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c8158ea0 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00147b0 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.691+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8159460 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.692+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c8159980 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.692+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f52c0014910 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.693+0000 7f52be7fc700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 0x7f52ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.693+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f52c0089d70 con 0x7f52c8157b30 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.693+0000 7f52cdd8e700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 0x7f52ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:06.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.693+0000 7f52cdd8e700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 0x7f52ac06d190 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f52c4000f60 tx=0x7f52c4007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:06.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.697+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f52c8159980 con 0x7f52c8157b30 2023-06-10T19:00:06.888 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.882+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f52c8159980 con 0x7f52c8157b30 2023-06-10T19:00:06.889 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.882+0000 7f52be7fc700 1 -- 172.21.15.104:0/1138077787 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v188) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52c8159980 con 0x7f52c8157b30 2023-06-10T19:00:06.889 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 8 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 msgr2=0x7f52ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 0x7f52ac06d190 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f52c4000f60 tx=0x7f52c4007c90 comp rx=0 tx=0).stop 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 msgr2=0x7f52c80c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f52c0007e40 tx=0x7f52c000e3c0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 shutdown_connections 2023-06-10T19:00:06.891 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52c8157b30 0x7f52c80c4cb0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52c80cea50 0x7f52c8152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f52ac06ace0 0x7f52ac06d190 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 --2- 172.21.15.104:0/1138077787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52c80d0830 0x7f52c80b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 >> 172.21.15.104:0/1138077787 conn(0x7f52c8089fd0 msgr2=0x7f52c808a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 shutdown_connections 2023-06-10T19:00:06.892 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:06.884+0000 7f52cfff2700 1 -- 172.21.15.104:0/1138077787 wait complete. 2023-06-10T19:00:06.903 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-10T19:00:07.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.019+0000 7f90f670d700 1 Processor -- start 2023-06-10T19:00:07.026 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.019+0000 7f90f670d700 1 -- start start 2023-06-10T19:00:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f670d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d04f0 0x7f90f00d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f670d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90f00b05d0 0x7f90f00b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f670d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00af530 con 0x7f90f00d25c0 2023-06-10T19:00:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f670d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f00af810 con 0x7f90f00b05d0 2023-06-10T19:00:07.028 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f670d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00afaf0 con 0x7f90f00d04f0 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90f00b05d0 0x7f90f00b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f90f00b05d0 0x7f90f00b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42270/0 (socket says 172.21.15.104:42270) 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 -- 172.21.15.104:0/747856492 learned_addr learned my addr 172.21.15.104:0/747856492 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90effff700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d04f0 0x7f90f00d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 -- 172.21.15.104:0/747856492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d04f0 msgr2=0x7f90f00d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d04f0 0x7f90f00d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.029 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 -- 172.21.15.104:0/747856492 >> v1:172.21.15.114:6789/0 conn(0x7f90f00d25c0 legacy=0x7f90f00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 -- 172.21.15.104:0/747856492 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00afdd0 con 0x7f90f00b05d0 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.020+0000 7f90f4caa700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00b05d0 0x7f90f00b0a50 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009e50 tx=0x7f90e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2832918b7fc220 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90eeffd700 1 -- 172.21.15.104:0/747856492 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800e070 con 0x7f90f00b05d0 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90eeffd700 1 -- 172.21.15.104:0/747856492 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8003a20 con 0x7f90f00b05d0 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90eeffd700 1 -- 172.21.15.104:0/747856492 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003dd0 con 0x7f90f00b05d0 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 -- 172.21.15.104:0/747856492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00b05d0 msgr2=0x7f90f00b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00b05d0 0x7f90f00b0a50 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009e50 tx=0x7f90e800b330 comp rx=0 tx=0).stop 2023-06-10T19:00:07.030 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 -- 172.21.15.104:0/747856492 shutdown_connections 2023-06-10T19:00:07.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00b05d0 0x7f90f00b0a50 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 --2- 172.21.15.104:0/747856492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d04f0 0x7f90f00d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 -- 172.21.15.104:0/747856492 >> 172.21.15.104:0/747856492 conn(0x7f90f0071ec0 msgr2=0x7f90f009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 -- 172.21.15.104:0/747856492 shutdown_connections 2023-06-10T19:00:07.031 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.021+0000 7f90f670d700 1 -- 172.21.15.104:0/747856492 wait complete. 2023-06-10T19:00:07.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.023+0000 7f90f670d700 1 Processor -- start 2023-06-10T19:00:07.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.023+0000 7f90f670d700 1 -- start start 2023-06-10T19:00:07.033 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 0x7f90f014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 0x7f90f00b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f014fb50 con 0x7f90f00d04f0 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00b5220 con 0x7f90f00b05d0 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f670d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f014e850 con 0x7f90f00d25c0 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.034 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42274/0 (socket says 172.21.15.104:42274) 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 -- 172.21.15.104:0/2179336807 learned_addr learned my addr 172.21.15.104:0/2179336807 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f4caa700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 0x7f90f00b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90effff700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 0x7f90f014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 -- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 msgr2=0x7f90f00b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 0x7f90f00b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 -- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 msgr2=0x7f90f014f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 0x7f90f014f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.035 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 -- 172.21.15.104:0/2179336807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e8009b30 con 0x7f90f00d04f0 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90f4caa700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 0x7f90f00b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90effff700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 0x7f90f014f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.024+0000 7f90ef7fe700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f90e000b4a0 tx=0x7f90e000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.025+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0011830 con 0x7f90f00d04f0 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.025+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e0011990 con 0x7f90f00d04f0 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.025+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f014eb30 con 0x7f90f00d04f0 2023-06-10T19:00:07.036 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.025+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0011d40 con 0x7f90f00d04f0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.025+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f014f0f0 con 0x7f90f00d04f0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.026+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0067010 con 0x7f90f00d04f0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.026+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90e001a400 con 0x7f90f00d04f0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.026+0000 7f90ecff9700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 0x7f90dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.026+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f90e00893c0 con 0x7f90f00d04f0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.027+0000 7f90effff700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 0x7f90dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.037 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.027+0000 7f90effff700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 0x7f90dc06d190 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f90e4008910 tx=0x7f90e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.038 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.031+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90e0054270 con 0x7f90f00d04f0 2023-06-10T19:00:07.222 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.216+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90f0074050 con 0x7f90f00d04f0 2023-06-10T19:00:07.223 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.216+0000 7f90ecff9700 1 -- 172.21.15.104:0/2179336807 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v188) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90e0057890 con 0x7f90f00d04f0 2023-06-10T19:00:07.223 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 32 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 msgr2=0x7f90dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 0x7f90dc06d190 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f90e4008910 tx=0x7f90e4008040 comp rx=0 tx=0).stop 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 msgr2=0x7f90f00b2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f90e000b4a0 tx=0x7f90e000b860 comp rx=0 tx=0).stop 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 shutdown_connections 2023-06-10T19:00:07.225 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f90dc06ace0 0x7f90dc06d190 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90f00b05d0 0x7f90f014f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90f00d25c0 0x7f90f00b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 --2- 172.21.15.104:0/2179336807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90f00d04f0 0x7f90f00b2a30 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.218+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 >> 172.21.15.104:0/2179336807 conn(0x7f90f0071ec0 msgr2=0x7f90f015b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.219+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 shutdown_connections 2023-06-10T19:00:07.226 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.219+0000 7f90f670d700 1 -- 172.21.15.104:0/2179336807 wait complete. 2023-06-10T19:00:07.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-10T19:00:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.353+0000 7fddf9aa8700 1 Processor -- start 2023-06-10T19:00:07.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.353+0000 7fddf9aa8700 1 -- start start 2023-06-10T19:00:07.362 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf9aa8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fddf40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf9aa8700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4071580 con 0x7fddf40caf70 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf9aa8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40c9020 con 0x7fddf4158790 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf9aa8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4070f30 con 0x7fddf40c95c0 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf3fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fddf40caf70 0x7fddf4071a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40112/0 (socket says 172.21.15.104:40112) 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf3fff700 1 -- 172.21.15.104:0/3138836174 learned_addr learned my addr 172.21.15.104:0/3138836174 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848442487 0 0) 0x7fddf4071580 con 0x7fddf40caf70 2023-06-10T19:00:07.363 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdddc0035b0 con 0x7fddf40caf70 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.354+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fddf40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387913202 0 0) 0x7fdddc0035b0 con 0x7fddf40caf70 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 msgr2=0x7fddf40c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fddf40c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 >> v1:172.21.15.114:6789/0 conn(0x7fddf4158790 legacy=0x7fddf40ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4071210 con 0x7fddf40caf70 2023-06-10T19:00:07.364 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fddec0019f0 con 0x7fddf40caf70 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 >> v1:172.21.15.104:6789/0 conn(0x7fddf40caf70 legacy=0x7fddf4071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdddc003a30 0x7fdddc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fdddc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fdddc005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4070f30 con 0x7fdddc004850 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc00bbd0 con 0x7fdddc003a30 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc00dfa0 con 0x7fddf40c95c0 2023-06-10T19:00:07.365 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fdddc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf37fe700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdddc003a30 0x7fdddc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf3fff700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fdddc005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 -- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 msgr2=0x7fdddc005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fdddc005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 -- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdddc003a30 msgr2=0x7fdddc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdddc003a30 0x7fdddc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 -- 172.21.15.104:0/3138836174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0035b0 con 0x7fdddc004850 2023-06-10T19:00:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf37fe700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdddc003a30 0x7fdddc003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf3fff700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf40c95c0 0x7fdddc005240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fdddc004cc0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fdde8007fd0 tx=0x7fdddc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fb92c43ed66a63 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8005a20 con 0x7fdddc004850 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde8005b80 con 0x7fdddc004850 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.355+0000 7fddf27fc700 1 -- 172.21.15.104:0/3138836174 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde800d640 con 0x7fdddc004850 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 -- 172.21.15.104:0/3138836174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 msgr2=0x7fdddc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/3138836174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fdddc004cc0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fdde8007fd0 tx=0x7fdddc005880 comp rx=0 tx=0).stop 2023-06-10T19:00:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf3fff700 1 -- 172.21.15.104:0/3138836174 reap_dead start 2023-06-10T19:00:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 -- 172.21.15.104:0/3138836174 shutdown_connections 2023-06-10T19:00:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 -- 172.21.15.104:0/3138836174 >> 172.21.15.104:0/3138836174 conn(0x7fddf406aac0 msgr2=0x7fddf40b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 -- 172.21.15.104:0/3138836174 shutdown_connections 2023-06-10T19:00:07.368 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.356+0000 7fddf9aa8700 1 -- 172.21.15.104:0/3138836174 wait complete. 2023-06-10T19:00:07.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.357+0000 7fddf9aa8700 1 Processor -- start 2023-06-10T19:00:07.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.357+0000 7fddf9aa8700 1 -- start start 2023-06-10T19:00:07.370 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf415a400 0x7fddf40b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddf40ab8c0 0x7fddf40abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4151fa0 con 0x7fdddc004850 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40b27b0 con 0x7fddf40ab8c0 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf9aa8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf415a8e0 con 0x7fddf415a400 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42294/0 (socket says 172.21.15.104:42294) 2023-06-10T19:00:07.371 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 -- 172.21.15.104:0/560572330 learned_addr learned my addr 172.21.15.104:0/560572330 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf3fff700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddf40ab8c0 0x7fddf40abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf415a400 0x7fddf40b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 -- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf415a400 msgr2=0x7fddf40b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf415a400 0x7fddf40b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 -- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddf40ab8c0 msgr2=0x7fddf40abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddf40ab8c0 0x7fddf40abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.372 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.358+0000 7fddf37fe700 1 -- 172.21.15.104:0/560572330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf415abc0 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fddf37fe700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdde400e7b0 tx=0x7fdde400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde40052e0 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4005440 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf415aea0 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde40038a0 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.359+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf415b460 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.360+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf415b5f0 con 0x7fdddc004850 2023-06-10T19:00:07.373 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.360+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdde4003a00 con 0x7fdddc004850 2023-06-10T19:00:07.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.360+0000 7fdddbfff700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 0x7fdde006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.360+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fdde4088c40 con 0x7fdddc004850 2023-06-10T19:00:07.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.361+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 0x7fdde006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.361+0000 7fddf2ffd700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 0x7fdde006d190 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fdde8014040 tx=0x7fdde801f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.374 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.365+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdde4053c30 con 0x7fdddc004850 2023-06-10T19:00:07.560 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.552+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddf4074050 con 0x7fdddc004850 2023-06-10T19:00:07.561 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.553+0000 7fdddbfff700 1 -- 172.21.15.104:0/560572330 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v188) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdde40190a0 con 0x7fdddc004850 2023-06-10T19:00:07.562 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 64 2023-06-10T19:00:07.562 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 msgr2=0x7fdde006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 0x7fdde006d190 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fdde8014040 tx=0x7fdde801f040 comp rx=0 tx=0).stop 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 msgr2=0x7fddf415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdde400e7b0 tx=0x7fdde400eac0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 shutdown_connections 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdde006ace0 0x7fdde006d190 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddf40ab8c0 0x7fddf40abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.563 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdddc004850 0x7fddf415b980 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 --2- 172.21.15.104:0/560572330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddf415a400 0x7fddf40b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.554+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 >> 172.21.15.104:0/560572330 conn(0x7fddf406aac0 msgr2=0x7fddf4153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.555+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 shutdown_connections 2023-06-10T19:00:07.564 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.555+0000 7fddf9aa8700 1 -- 172.21.15.104:0/560572330 wait complete. 2023-06-10T19:00:07.573 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T19:00:07.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 Processor -- start 2023-06-10T19:00:07.693 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 -- start start 2023-06-10T19:00:07.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.695 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53dc0cae10 0x7f53dc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53dc0d0340 con 0x7f53dc0ce390 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc0d1660 con 0x7f53dc0cae10 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.687+0000 7f53e1972700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc15cb60 con 0x7f53dc06c720 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53da7fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53dc0cae10 0x7f53dc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53daffd700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f53dc0ce390 0x7f53dc160cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:45974/0 (socket says 172.21.15.104:45974) 2023-06-10T19:00:07.696 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53dc0cae10 0x7f53dc0cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:58038/0 (socket says 172.21.15.104:58038) 2023-06-10T19:00:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53da7fc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:33452/0 (socket says 172.21.15.104:33452) 2023-06-10T19:00:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53daffd700 1 -- 172.21.15.104:0/2096102281 learned_addr learned my addr 172.21.15.104:0/2096102281 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 -- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 msgr2=0x7f53dc0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:07.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 -- 172.21.15.104:0/2096102281 >> v1:172.21.15.114:6789/0 conn(0x7f53dc0ce390 legacy=0x7f53dc160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 -- 172.21.15.104:0/2096102281 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0ceca0 con 0x7f53dc0cae10 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53d9ffb700 1 -- 172.21.15.104:0/2096102281 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558031880 0 0) 0x7f53dc0d0340 con 0x7f53dc0ce390 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53da7fc700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.688+0000 7f53db7fe700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0cb270 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009d90 tx=0x7f53d400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2e27813d8ec5a96 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53d9ffb700 1 -- 172.21.15.104:0/2096102281 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d400e070 con 0x7f53dc0cae10 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53d9ffb700 1 -- 172.21.15.104:0/2096102281 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d400bb20 con 0x7f53dc0cae10 2023-06-10T19:00:07.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53d9ffb700 1 -- 172.21.15.104:0/2096102281 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4012430 con 0x7f53dc0cae10 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 -- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 msgr2=0x7f53dc0cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0cb270 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009d90 tx=0x7f53d400b330 comp rx=0 tx=0).stop 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 -- 172.21.15.104:0/2096102281 shutdown_connections 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0cb270 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 --2- 172.21.15.104:0/2096102281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc0ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 -- 172.21.15.104:0/2096102281 >> 172.21.15.104:0/2096102281 conn(0x7f53dc069e70 msgr2=0x7f53dc06a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 -- 172.21.15.104:0/2096102281 shutdown_connections 2023-06-10T19:00:07.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.689+0000 7f53e1972700 1 -- 172.21.15.104:0/2096102281 wait complete. 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.691+0000 7f53e1972700 1 Processor -- start 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.691+0000 7f53e1972700 1 -- start start 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc0b5110 con 0x7f53dc0ce390 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc1576e0 con 0x7f53dc0cae10 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc15de60 con 0x7f53dc06c720 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42310/0 (socket says 172.21.15.104:42310) 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 -- 172.21.15.104:0/4031983079 learned_addr learned my addr 172.21.15.104:0/4031983079 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53daffd700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53da7fc700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 -- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 msgr2=0x7f53dc1571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc1571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 -- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 msgr2=0x7f53dc0b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 -- 172.21.15.104:0/4031983079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00094a0 con 0x7f53dc0ce390 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53db7fe700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f53d4014040 tx=0x7f53d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4012430 con 0x7f53dc0ce390 2023-06-10T19:00:07.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d4020070 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4009a70 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.692+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4004280 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.693+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc0b5350 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.693+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc159ea0 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.694+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f53d40043e0 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.694+0000 7f53c3fff700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 0x7f53c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.694+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f53d4022070 con 0x7f53dc0ce390 2023-06-10T19:00:07.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.694+0000 7f53daffd700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 0x7f53c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.695+0000 7f53daffd700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 0x7f53c806d190 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f53cc007fd0 tx=0x7f53cc007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:07.706 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.699+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f53dc159ea0 con 0x7f53dc0ce390 2023-06-10T19:00:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.929+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53dc159ea0 con 0x7f53dc0ce390 2023-06-10T19:00:07.936 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.930+0000 7f53c3fff700 1 -- 172.21.15.104:0/4031983079 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f53dc159ea0 con 0x7f53dc0ce390 2023-06-10T19:00:07.937 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:07.937 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T19:00:07.938 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 msgr2=0x7f53c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 0x7f53c806d190 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f53cc007fd0 tx=0x7f53cc007de0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 msgr2=0x7f53dc15d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f53d4014040 tx=0x7f53d4004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 shutdown_connections 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53dc0ce390 0x7f53dc15d200 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53dc06c720 0x7f53dc1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f53c806ace0 0x7f53c806d190 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 --2- 172.21.15.104:0/4031983079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53dc0cae10 0x7f53dc0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 >> 172.21.15.104:0/4031983079 conn(0x7f53dc069e70 msgr2=0x7f53dc0cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 shutdown_connections 2023-06-10T19:00:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:07.932+0000 7f53e1972700 1 -- 172.21.15.104:0/4031983079 wait complete. 2023-06-10T19:00:07.940 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T19:00:07.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.064+0000 7f44928e7700 1 Processor -- start 2023-06-10T19:00:08.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.064+0000 7f44928e7700 1 -- start start 2023-06-10T19:00:08.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.064+0000 7f44928e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c15c730 0x7f448c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.072 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.064+0000 7f44928e7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f448c0cc000 0x7f448c0cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f44928e7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f448c06cd70 0x7f448c06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f44928e7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0aed80 con 0x7f448c06cd70 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f44928e7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0c9c60 con 0x7f448c0cc000 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f44928e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0cc9b0 con 0x7f448c15c730 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f448c06cd70 0x7f448c06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f448c06cd70 0x7f448c06d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42318/0 (socket says 172.21.15.104:42318) 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 -- 172.21.15.104:0/1093882098 learned_addr learned my addr 172.21.15.104:0/1093882098 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:08.073 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f448b7fe700 1 --2- 172.21.15.104:0/1093882098 >> v2:172.21.15.114:3300/0 conn(0x7f448c0cc000 0x7f448c0cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f448bfff700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c15c730 0x7f448c06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 -- 172.21.15.104:0/1093882098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c15c730 msgr2=0x7f448c06e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c15c730 0x7f448c06e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 -- 172.21.15.104:0/1093882098 >> v2:172.21.15.114:3300/0 conn(0x7f448c0cc000 msgr2=0x7f448c0cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 --2- 172.21.15.104:0/1093882098 >> v2:172.21.15.114:3300/0 conn(0x7f448c0cc000 0x7f448c0cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 -- 172.21.15.104:0/1093882098 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c15cb00 con 0x7f448c06cd70 2023-06-10T19:00:08.074 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f4490e84700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c06cd70 0x7f448c06d1f0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f448c070cb0 tx=0x7f448400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdecccd7135b053 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f448affd700 1 -- 172.21.15.104:0/1093882098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448400e070 con 0x7f448c06cd70 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f448affd700 1 -- 172.21.15.104:0/1093882098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448400b9c0 con 0x7f448c06cd70 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.065+0000 7f448affd700 1 -- 172.21.15.104:0/1093882098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448400bd70 con 0x7f448c06cd70 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 -- 172.21.15.104:0/1093882098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c06cd70 msgr2=0x7f448c06d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c06cd70 0x7f448c06d1f0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f448c070cb0 tx=0x7f448400b330 comp rx=0 tx=0).stop 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 -- 172.21.15.104:0/1093882098 shutdown_connections 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c06cd70 0x7f448c06d1f0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.075 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 --2- 172.21.15.104:0/1093882098 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c15c730 0x7f448c06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 --2- 172.21.15.104:0/1093882098 >> v2:172.21.15.114:3300/0 conn(0x7f448c0cc000 0x7f448c0cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 -- 172.21.15.104:0/1093882098 >> 172.21.15.104:0/1093882098 conn(0x7f448c06b8a0 msgr2=0x7f448c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 -- 172.21.15.104:0/1093882098 shutdown_connections 2023-06-10T19:00:08.076 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.066+0000 7f44928e7700 1 -- 172.21.15.104:0/1093882098 wait complete. 2023-06-10T19:00:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.067+0000 7f44928e7700 1 Processor -- start 2023-06-10T19:00:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.067+0000 7f44928e7700 1 -- start start 2023-06-10T19:00:08.078 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 0x7f448c0b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 0x7f448c1f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0c7f30 con 0x7f448c0cc000 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0affa0 con 0x7f448c15c730 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f44928e7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0c78e0 con 0x7f448c06cd70 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.079 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42322/0 (socket says 172.21.15.104:42322) 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 -- 172.21.15.104:0/3369481260 learned_addr learned my addr 172.21.15.104:0/3369481260 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448bfff700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 0x7f448c0b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f4490e84700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 0x7f448c1f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 -- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 msgr2=0x7f448c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 0x7f448c0b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 -- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 msgr2=0x7f448c1f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.080 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 0x7f448c1f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 -- 172.21.15.104:0/3369481260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484009b30 con 0x7f448c0cc000 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448bfff700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 0x7f448c0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f4490e84700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 0x7f448c1f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.068+0000 7f448b7fe700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f448000e810 tx=0x7f448000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.069+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480013610 con 0x7f448c0cc000 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.069+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4480013770 con 0x7f448c0cc000 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.069+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c0c7bc0 con 0x7f448c0cc000 2023-06-10T19:00:08.081 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.069+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480013b00 con 0x7f448c0cc000 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.069+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c0b0280 con 0x7f448c0cc000 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.070+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448c0c7740 con 0x7f448c0cc000 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.070+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4480013c60 con 0x7f448c0cc000 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.071+0000 7f4488ff9700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 0x7f447806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.071+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f4480089230 con 0x7f448c0cc000 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.071+0000 7f448bfff700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 0x7f447806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.082 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.071+0000 7f448bfff700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 0x7f447806d190 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f448c1e32c0 tx=0x7f447c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.083 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.075+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f448c0c7740 con 0x7f448c0cc000 2023-06-10T19:00:08.268 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.262+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f448c0c7740 con 0x7f448c0cc000 2023-06-10T19:00:08.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.263+0000 7f4488ff9700 1 -- 172.21.15.104:0/3369481260 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18677 (secure 0 0 0) 0x7f448c0c7740 con 0x7f448c0cc000 2023-06-10T19:00:08.270 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:08.270 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":188,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:40.713132+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:08.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 msgr2=0x7f447806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 0x7f447806d190 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f448c1e32c0 tx=0x7f447c008040 comp rx=0 tx=0).stop 2023-06-10T19:00:08.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 msgr2=0x7f448c154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f448000e810 tx=0x7f448000ebd0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 shutdown_connections 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f447806ace0 0x7f447806d190 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f448c15c730 0x7f448c1f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f448c06cd70 0x7f448c0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 --2- 172.21.15.104:0/3369481260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f448c0cc000 0x7f448c154a70 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.265+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 >> 172.21.15.104:0/3369481260 conn(0x7f448c06b8a0 msgr2=0x7f448c0cb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.266+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 shutdown_connections 2023-06-10T19:00:08.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.266+0000 7f44928e7700 1 -- 172.21.15.104:0/3369481260 wait complete. 2023-06-10T19:00:08.284 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T19:00:08.285 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T19:00:08.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.397+0000 7f3329f2d700 1 Processor -- start 2023-06-10T19:00:08.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.397+0000 7f3329f2d700 1 -- start start 2023-06-10T19:00:08.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f3329f2d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f332406e790 0x7f33240ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f3329f2d700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332415cb00 con 0x7f332415c730 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f3329f2d700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33240cd280 con 0x7f33240cc710 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f3329f2d700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33240aed80 con 0x7f332406e790 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f33237fe700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f332415c730 0x7f33240cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40156/0 (socket says 172.21.15.104:40156) 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f33237fe700 1 -- 172.21.15.104:0/642995484 learned_addr learned my addr 172.21.15.104:0/642995484 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.398+0000 7f3323fff700 1 --2- 172.21.15.104:0/642995484 >> v2:172.21.15.104:3300/0 conn(0x7f332406e790 0x7f33240ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f33227fc700 1 -- 172.21.15.104:0/642995484 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112819068 0 0) 0x7f330c001290 con 0x7f332415c730 2023-06-10T19:00:08.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f33227fc700 1 -- 172.21.15.104:0/642995484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33100035b0 con 0x7f332415c730 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3323fff700 1 -- 172.21.15.104:0/642995484 >> v1:172.21.15.114:6789/0 conn(0x7f33240cc710 legacy=0x7f33240ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3323fff700 1 -- 172.21.15.104:0/642995484 >> v1:172.21.15.104:6789/0 conn(0x7f332415c730 legacy=0x7f33240cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3323fff700 1 -- 172.21.15.104:0/642995484 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240cd560 con 0x7f332406e790 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3323fff700 1 --2- 172.21.15.104:0/642995484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332406e790 0x7f33240ca7c0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f331c009d90 tx=0x7f331c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1137bb72ff085b8e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f33227fc700 1 -- 172.21.15.104:0/642995484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c00e070 con 0x7f332406e790 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f33227fc700 1 -- 172.21.15.104:0/642995484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c00bb20 con 0x7f332406e790 2023-06-10T19:00:08.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f33227fc700 1 -- 172.21.15.104:0/642995484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c012430 con 0x7f332406e790 2023-06-10T19:00:08.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3329f2d700 1 -- 172.21.15.104:0/642995484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332406e790 msgr2=0x7f33240ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3329f2d700 1 --2- 172.21.15.104:0/642995484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332406e790 0x7f33240ca7c0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f331c009d90 tx=0x7f331c00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3329f2d700 1 -- 172.21.15.104:0/642995484 shutdown_connections 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.399+0000 7f3329f2d700 1 --2- 172.21.15.104:0/642995484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332406e790 0x7f33240ca7c0 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.400+0000 7f3329f2d700 1 -- 172.21.15.104:0/642995484 >> 172.21.15.104:0/642995484 conn(0x7f332406b8a0 msgr2=0x7f33240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.400+0000 7f3329f2d700 1 -- 172.21.15.104:0/642995484 shutdown_connections 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.400+0000 7f3329f2d700 1 -- 172.21.15.104:0/642995484 wait complete. 2023-06-10T19:00:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.401+0000 7f3329f2d700 1 Processor -- start 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.401+0000 7f3329f2d700 1 -- start start 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332406e790 0x7f33240c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33240cc710 0x7f33240b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33240b2c30 con 0x7f332415c730 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33241ed500 con 0x7f332406e790 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33240c5250 con 0x7f33240cc710 2023-06-10T19:00:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42338/0 (socket says 172.21.15.104:42338) 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 -- 172.21.15.104:0/696084060 learned_addr learned my addr 172.21.15.104:0/696084060 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f33237fe700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332406e790 0x7f33240c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3322ffd700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33240cc710 0x7f33240b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 -- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33240cc710 msgr2=0x7f33240b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33240cc710 0x7f33240b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 -- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332406e790 msgr2=0x7f33240c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332406e790 0x7f33240c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 -- 172.21.15.104:0/696084060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c009a70 con 0x7f332415c730 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3323fff700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f33240b2200 tx=0x7f331c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c012430 con 0x7f332415c730 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c020070 con 0x7f332415c730 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33240b25e0 con 0x7f332415c730 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.402+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c004280 con 0x7f332415c730 2023-06-10T19:00:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.403+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33240c4e90 con 0x7f332415c730 2023-06-10T19:00:08.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.403+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33240c5020 con 0x7f332415c730 2023-06-10T19:00:08.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.404+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f331c0043e0 con 0x7f332415c730 2023-06-10T19:00:08.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.404+0000 7f331bfff700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 0x7f330406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.404+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f331c022070 con 0x7f332415c730 2023-06-10T19:00:08.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.404+0000 7f33237fe700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 0x7f330406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.405+0000 7f33237fe700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 0x7f330406d190 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f330c0010e0 tx=0x7f330c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.408+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f331c056310 con 0x7f332415c730 2023-06-10T19:00:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.637+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33240b2440 con 0x7f332415c730 2023-06-10T19:00:08.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.637+0000 7f331bfff700 1 -- 172.21.15.104:0/696084060 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7f33240b2440 con 0x7f332415c730 2023-06-10T19:00:08.644 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:08.644 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 msgr2=0x7f330406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 0x7f330406d190 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f330c0010e0 tx=0x7f330c009040 comp rx=0 tx=0).stop 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 msgr2=0x7f33240b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f33240b2200 tx=0x7f331c004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 shutdown_connections 2023-06-10T19:00:08.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f330406ace0 0x7f330406d190 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332406e790 0x7f33240c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332415c730 0x7f33240b1520 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 --2- 172.21.15.104:0/696084060 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33240cc710 0x7f33240b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 >> 172.21.15.104:0/696084060 conn(0x7f332406b8a0 msgr2=0x7f33240ac610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 shutdown_connections 2023-06-10T19:00:08.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.639+0000 7f3329f2d700 1 -- 172.21.15.104:0/696084060 wait complete. 2023-06-10T19:00:08.648 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T19:00:08.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T19:00:08.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.771+0000 7fdd8a4a1700 1 Processor -- start 2023-06-10T19:00:08.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.771+0000 7fdd8a4a1700 1 -- start start 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd8a4a1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 0x7fdd840ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd8a4a1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd840aeb00 0x7fdd840ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd8a4a1700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840cee70 con 0x7fdd840ada30 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd8a4a1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd840cf430 con 0x7fdd840aeb00 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd8a4a1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840cf150 con 0x7fdd840ade00 2023-06-10T19:00:08.780 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd88a3e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fdd840aeb00 0x7fdd840ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd83fff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdd840ada30 0x7fdd840ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40170/0 (socket says 172.21.15.104:40170) 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd83fff700 1 -- 172.21.15.104:0/2779221111 learned_addr learned my addr 172.21.15.104:0/2779221111 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd837fe700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 0x7fdd840ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd88a3e700 1 -- 172.21.15.104:0/2779221111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 msgr2=0x7fdd840ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd88a3e700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 0x7fdd840ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd88a3e700 1 -- 172.21.15.104:0/2779221111 >> v1:172.21.15.104:6789/0 conn(0x7fdd840ada30 legacy=0x7fdd840ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.781 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd82ffd700 1 -- 172.21.15.104:0/2779221111 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074498825 0 0) 0x7fdd840cee70 con 0x7fdd840ada30 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd88a3e700 1 -- 172.21.15.104:0/2779221111 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8406cf10 con 0x7fdd840aeb00 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.772+0000 7fdd837fe700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 0x7fdd840ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd88a3e700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd840aeb00 0x7fdd840ce5f0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009d90 tx=0x7fdd7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e08780623df7cd8 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd82ffd700 1 -- 172.21.15.104:0/2779221111 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c00e070 con 0x7fdd840aeb00 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd82ffd700 1 -- 172.21.15.104:0/2779221111 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c00bbd0 con 0x7fdd840aeb00 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd82ffd700 1 -- 172.21.15.104:0/2779221111 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c012430 con 0x7fdd840aeb00 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/2779221111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd840aeb00 msgr2=0x7fdd840ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd840aeb00 0x7fdd840ce5f0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009d90 tx=0x7fdd7c00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/2779221111 shutdown_connections 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd840aeb00 0x7fdd840ce5f0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/2779221111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840ade00 0x7fdd840ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/2779221111 >> 172.21.15.104:0/2779221111 conn(0x7fdd8406b8a0 msgr2=0x7fdd840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/2779221111 shutdown_connections 2023-06-10T19:00:08.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.773+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/2779221111 wait complete. 2023-06-10T19:00:08.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.775+0000 7fdd8a4a1700 1 Processor -- start 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.775+0000 7fdd8a4a1700 1 -- start start 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd840ada30 0x7fdd8414d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840aeb00 0x7fdd84099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd84099d00 con 0x7fdd84099880 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8414e490 con 0x7fdd840ada30 2023-06-10T19:00:08.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd8a4a1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8409d120 con 0x7fdd840aeb00 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42352/0 (socket says 172.21.15.104:42352) 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 -- 172.21.15.104:0/3113676691 learned_addr learned my addr 172.21.15.104:0/3113676691 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd837fe700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840aeb00 0x7fdd84099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd83fff700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd840ada30 0x7fdd8414d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 -- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840aeb00 msgr2=0x7fdd84099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840aeb00 0x7fdd84099340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 -- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd840ada30 msgr2=0x7fdd8414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd840ada30 0x7fdd8414d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.776+0000 7fdd88a3e700 1 -- 172.21.15.104:0/3113676691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c009a70 con 0x7fdd84099880 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd88a3e700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b21e0 tx=0x7fdd7c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c003ea0 con 0x7fdd84099880 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c020070 con 0x7fdd84099880 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8414df20 con 0x7fdd84099880 2023-06-10T19:00:08.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c004280 con 0x7fdd84099880 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.777+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8409c360 con 0x7fdd84099880 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.778+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840ceab0 con 0x7fdd84099880 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.778+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdd7c0043e0 con 0x7fdd84099880 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.778+0000 7fdd80ff9700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 0x7fdd7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.779+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fdd7c022070 con 0x7fdd84099880 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.779+0000 7fdd83fff700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 0x7fdd7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:08.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.779+0000 7fdd83fff700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 0x7fdd7006d190 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdd78001670 tx=0x7fdd78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:08.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:08.783+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdd7c0567e0 con 0x7fdd84099880 2023-06-10T19:00:09.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.012+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd84066f70 con 0x7fdd84099880 2023-06-10T19:00:09.018 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.012+0000 7fdd80ff9700 1 -- 172.21.15.104:0/3113676691 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4697 (secure 0 0 0) 0x7fdd7c059e00 con 0x7fdd84099880 2023-06-10T19:00:09.019 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:09.019 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":117,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-10T18:59:23.247307+0000","modified":"2023-06-10T18:59:26.256259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11139":{"gid":11139,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6835/3645224862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3645224862},{"type":"v1","addr":"172.21.15.104:6835","nonce":3645224862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 msgr2=0x7fdd7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 0x7fdd7006d190 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdd78001670 tx=0x7fdd78009040 comp rx=0 tx=0).stop 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 msgr2=0x7fdd8409b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b21e0 tx=0x7fdd7c004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 shutdown_connections 2023-06-10T19:00:09.021 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fdd7006ace0 0x7fdd7006d190 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd840ada30 0x7fdd8414d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd84099880 0x7fdd8409b330 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 --2- 172.21.15.104:0/3113676691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd840aeb00 0x7fdd84099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 >> 172.21.15.104:0/3113676691 conn(0x7fdd8406b8a0 msgr2=0x7fdd840b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 shutdown_connections 2023-06-10T19:00:09.022 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.014+0000 7fdd8a4a1700 1 -- 172.21.15.104:0/3113676691 wait complete. 2023-06-10T19:00:09.023 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 117 2023-06-10T19:00:09.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:09.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.146+0000 7f29597f8700 1 Processor -- start 2023-06-10T19:00:09.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.146+0000 7f29597f8700 1 -- start start 2023-06-10T19:00:09.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295406e0e0 0x7f295406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f295406e4b0 0x7f29540c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29540ca100 0x7f29540ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2954071640 con 0x7f29540ca100 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29540aef00 con 0x7f295406e4b0 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29597f8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295406d290 con 0x7f295406e0e0 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29540ca100 0x7f29540ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f29540ca100 0x7f29540ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42358/0 (socket says 172.21.15.104:42358) 2023-06-10T19:00:09.155 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 -- 172.21.15.104:0/1036131608 learned_addr learned my addr 172.21.15.104:0/1036131608 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29527fc700 1 --2- 172.21.15.104:0/1036131608 >> v2:172.21.15.114:3300/0 conn(0x7f295406e4b0 0x7f29540c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f2952ffd700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295406e0e0 0x7f295406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 -- 172.21.15.104:0/1036131608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295406e0e0 msgr2=0x7f295406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295406e0e0 0x7f295406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 -- 172.21.15.104:0/1036131608 >> v2:172.21.15.114:3300/0 conn(0x7f295406e4b0 msgr2=0x7f29540c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 --2- 172.21.15.104:0/1036131608 >> v2:172.21.15.114:3300/0 conn(0x7f295406e4b0 0x7f29540c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 -- 172.21.15.104:0/1036131608 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295406d4c0 con 0x7f29540ca100 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f29537fe700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29540ca100 0x7f29540ce9f0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f29540706d0 tx=0x7f294c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc5a9f2371d066c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f2951ffb700 1 -- 172.21.15.104:0/1036131608 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c00e070 con 0x7f29540ca100 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f2951ffb700 1 -- 172.21.15.104:0/1036131608 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c00b8d0 con 0x7f29540ca100 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.147+0000 7f2951ffb700 1 -- 172.21.15.104:0/1036131608 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c00bc80 con 0x7f29540ca100 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 -- 172.21.15.104:0/1036131608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29540ca100 msgr2=0x7f29540ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29540ca100 0x7f29540ce9f0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f29540706d0 tx=0x7f294c00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 -- 172.21.15.104:0/1036131608 shutdown_connections 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29540ca100 0x7f29540ce9f0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 --2- 172.21.15.104:0/1036131608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295406e0e0 0x7f295406cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 --2- 172.21.15.104:0/1036131608 >> v2:172.21.15.114:3300/0 conn(0x7f295406e4b0 0x7f29540c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 -- 172.21.15.104:0/1036131608 >> 172.21.15.104:0/1036131608 conn(0x7f295406b8a0 msgr2=0x7f29540cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 -- 172.21.15.104:0/1036131608 shutdown_connections 2023-06-10T19:00:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.148+0000 7f29597f8700 1 -- 172.21.15.104:0/1036131608 wait complete. 2023-06-10T19:00:09.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.149+0000 7f29597f8700 1 Processor -- start 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.149+0000 7f29597f8700 1 -- start start 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f295406e0e0 0x7f29540afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 0x7f29540c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29540c7510 con 0x7f295406e4b0 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29540c6640 con 0x7f295406e0e0 2023-06-10T19:00:09.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29597f8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29540c6890 con 0x7f29540ca100 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42364/0 (socket says 172.21.15.104:42364) 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 -- 172.21.15.104:0/1977453102 learned_addr learned my addr 172.21.15.104:0/1977453102 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f2952ffd700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f295406e0e0 0x7f29540afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29537fe700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 0x7f29540c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 -- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 msgr2=0x7f29540c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 0x7f29540c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 -- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f295406e0e0 msgr2=0x7f29540afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f295406e0e0 0x7f29540afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.150+0000 7f29527fc700 1 -- 172.21.15.104:0/1977453102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c009b30 con 0x7f295406e4b0 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f29537fe700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 0x7f29540c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f29527fc700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f294800e9b0 tx=0x7f294800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948009710 con 0x7f295406e4b0 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294800c850 con 0x7f295406e4b0 2023-06-10T19:00:09.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29540c6b70 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294800cbc0 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.151+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29540c7130 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.152+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29540c7650 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.152+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f294800cd20 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.153+0000 7f293b7fe700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 0x7f294006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.153+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f2948089730 con 0x7f295406e4b0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.153+0000 7f2952ffd700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 0x7f294006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.154+0000 7f2952ffd700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 0x7f294006d190 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f295415c330 tx=0x7f2944008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.157+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f29540c7650 con 0x7f295406e4b0 2023-06-10T19:00:09.354 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.348+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29540c7650 con 0x7f295406e4b0 2023-06-10T19:00:09.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.349+0000 7f293b7fe700 1 -- 172.21.15.104:0/1977453102 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+18677 (secure 0 0 0) 0x7f29540c7650 con 0x7f295406e4b0 2023-06-10T19:00:09.356 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:09.356 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":188,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T18:59:40.713132+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-06-10T18:59:21.300558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-06-10T18:59:22.213068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:09.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 msgr2=0x7f294006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 0x7f294006d190 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f295415c330 tx=0x7f2944008040 comp rx=0 tx=0).stop 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 msgr2=0x7f29540b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f294800e9b0 tx=0x7f294800ed70 comp rx=0 tx=0).stop 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 shutdown_connections 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f294006ace0 0x7f294006d190 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.358 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f295406e0e0 0x7f29540afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29540ca100 0x7f29540c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 --2- 172.21.15.104:0/1977453102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f295406e4b0 0x7f29540b0ab0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 >> 172.21.15.104:0/1977453102 conn(0x7f295406b8a0 msgr2=0x7f2954072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 shutdown_connections 2023-06-10T19:00:09.359 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.351+0000 7f29597f8700 1 -- 172.21.15.104:0/1977453102 wait complete. 2023-06-10T19:00:09.369 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-06-10T19:00:09.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.484+0000 7f614c7bc700 1 Processor -- start 2023-06-10T19:00:09.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.484+0000 7f614c7bc700 1 -- start start 2023-06-10T19:00:09.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614c7bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f614415c730 0x7f614406e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614c7bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61440cc000 0x7f61440cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614c7bc700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614415cb00 con 0x7f614406cd70 2023-06-10T19:00:09.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614c7bc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61440aed80 con 0x7f614415c730 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614c7bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440c9c60 con 0x7f61440cc000 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f614415c730 0x7f614406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f614415c730 0x7f614406e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42370/0 (socket says 172.21.15.104:42370) 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 -- 172.21.15.104:0/2960376817 learned_addr learned my addr 172.21.15.104:0/2960376817 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f6149d57700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61440cc000 0x7f61440cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 -- 172.21.15.104:0/2960376817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61440cc000 msgr2=0x7f61440cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:09.493 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61440cc000 0x7f61440cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.485+0000 7f614a558700 1 -- 172.21.15.104:0/2960376817 >> v1:172.21.15.114:6789/0 conn(0x7f614406cd70 legacy=0x7f614406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614a558700 1 -- 172.21.15.104:0/2960376817 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614406ecd0 con 0x7f614415c730 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614a558700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f614415c730 0x7f614406e790 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2549370d3e93c5f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f6149556700 1 -- 172.21.15.104:0/2960376817 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134005e30 con 0x7f614415c730 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f6149556700 1 -- 172.21.15.104:0/2960376817 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6134004030 con 0x7f614415c730 2023-06-10T19:00:09.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f6149556700 1 -- 172.21.15.104:0/2960376817 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61340043e0 con 0x7f614415c730 2023-06-10T19:00:09.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 -- 172.21.15.104:0/2960376817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f614415c730 msgr2=0x7f614406e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f614415c730 0x7f614406e790 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134009bd0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 -- 172.21.15.104:0/2960376817 shutdown_connections 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f614415c730 0x7f614406e790 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2960376817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61440cc000 0x7f61440cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.486+0000 7f614c7bc700 1 -- 172.21.15.104:0/2960376817 >> 172.21.15.104:0/2960376817 conn(0x7f614406b8a0 msgr2=0x7f61440cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.487+0000 7f614c7bc700 1 -- 172.21.15.104:0/2960376817 shutdown_connections 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.487+0000 7f614c7bc700 1 -- 172.21.15.104:0/2960376817 wait complete. 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 Processor -- start 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 -- start start 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 0x7f6144156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 0x7f61441f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441e1eb0 con 0x7f61440cc000 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441e2190 con 0x7f614415c730 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f614c7bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61441e2470 con 0x7f614406cd70 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.488+0000 7f6149d57700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42378/0 (socket says 172.21.15.104:42378) 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 -- 172.21.15.104:0/2923103099 learned_addr learned my addr 172.21.15.104:0/2923103099 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614a558700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 0x7f6144156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614ad59700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 0x7f61441f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 -- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 msgr2=0x7f6144156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 0x7f6144156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 -- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 msgr2=0x7f61441f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 0x7f61441f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 -- 172.21.15.104:0/2923103099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134009500 con 0x7f61440cc000 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614a558700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 0x7f6144156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614ad59700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 0x7f61441f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:09.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f6149d57700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9b0 tx=0x7f614000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140009710 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614000c850 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61441e2750 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614000cbc0 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.489+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61441570f0 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.490+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61440b2b00 con 0x7f61440cc000 2023-06-10T19:00:09.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.490+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f614000cd20 con 0x7f61440cc000 2023-06-10T19:00:09.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.491+0000 7f613affd700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 0x7f612806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.491+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6140089700 con 0x7f61440cc000 2023-06-10T19:00:09.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.491+0000 7f614a558700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 0x7f612806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.502 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.491+0000 7f614a558700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 0x7f612806d190 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.495+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f61400580d0 con 0x7f61440cc000 2023-06-10T19:00:09.730 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.723+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6144074050 con 0x7f61440cc000 2023-06-10T19:00:09.792 INFO:tasks.ceph.mon.a.smithi104.stderr:2023-06-10T19:00:09.785+0000 7ff056891700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-06-10T19:00:09.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.791+0000 7f613affd700 1 -- 172.21.15.104:0/2923103099 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6140057c60 con 0x7f61440cc000 2023-06-10T19:00:09.799 INFO:tasks.ceph.mds.a.smithi104.stderr: -4094> 2023-06-10T18:59:23.840+0000 7f2da3d73700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 msgr2=0x7f612806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 0x7f612806d190 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6134006fd0 tx=0x7f6134008070 comp rx=0 tx=0).stop 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 msgr2=0x7f6144157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9b0 tx=0x7f614000ed70 comp rx=0 tx=0).stop 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 shutdown_connections 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f612806ace0 0x7f612806d190 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f614415c730 0x7f61441f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f614406cd70 0x7f6144156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 --2- 172.21.15.104:0/2923103099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61440cc000 0x7f6144157d80 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 >> 172.21.15.104:0/2923103099 conn(0x7f614406b8a0 msgr2=0x7f61440cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 shutdown_connections 2023-06-10T19:00:09.801 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.793+0000 7f614c7bc700 1 -- 172.21.15.104:0/2923103099 wait complete. 2023-06-10T19:00:09.802 INFO:teuthology.orchestra.run.smithi104.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-06-10T19:00:09.812 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-06-10T19:00:09.826 INFO:tasks.ceph.mds.a.smithi104.stdout:starting mds.a at 2023-06-10T19:00:09.840 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:09.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.932+0000 7f0a345c7700 1 Processor -- start 2023-06-10T19:00:09.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.932+0000 7f0a345c7700 1 -- start start 2023-06-10T19:00:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.933+0000 7f0a345c7700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.933+0000 7f0a345c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.933+0000 7f0a345c7700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2c0cc9f0 con 0x7f0a2c0cd720 2023-06-10T19:00:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.933+0000 7f0a345c7700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2c0cdba0 con 0x7f0a2c0cb210 2023-06-10T19:00:09.940 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.933+0000 7f0a345c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0cbbd0 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a32363700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a32b64700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a2c0cd720 0x7f0a2c0c9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:46074/0 (socket says 172.21.15.104:46074) 2023-06-10T19:00:09.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a32363700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:58130/0 (socket says 172.21.15.104:58130) 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a32b64700 1 -- 172.21.15.104:0/2571505071 learned_addr learned my addr 172.21.15.104:0/2571505071 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:33550/0 (socket says 172.21.15.104:33550) 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 -- 172.21.15.104:0/2571505071 >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 msgr2=0x7f0a2c0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 --2- 172.21.15.104:0/2571505071 >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 -- 172.21.15.104:0/2571505071 >> v1:172.21.15.114:6789/0 conn(0x7f0a2c0cd720 legacy=0x7f0a2c0c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 -- 172.21.15.104:0/2571505071 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c0cc100 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31361700 1 -- 172.21.15.104:0/2571505071 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201306328 0 0) 0x7f0a2c0cc9f0 con 0x7f0a2c0cd720 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a32363700 1 --2- 172.21.15.104:0/2571505071 >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31b62700 1 --2- 172.21.15.104:0/2571505071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009840 tx=0x7f0a28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2015fe2c876ce293 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31361700 1 -- 172.21.15.104:0/2571505071 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28011830 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.934+0000 7f0a31361700 1 -- 172.21.15.104:0/2571505071 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a28011990 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a31361700 1 -- 172.21.15.104:0/2571505071 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28011d20 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 -- 172.21.15.104:0/2571505071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 msgr2=0x7f0a2c0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 --2- 172.21.15.104:0/2571505071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009840 tx=0x7f0a28009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 -- 172.21.15.104:0/2571505071 shutdown_connections 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 --2- 172.21.15.104:0/2571505071 >> v2:172.21.15.114:3300/0 conn(0x7f0a2c0cb210 0x7f0a2c0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 --2- 172.21.15.104:0/2571505071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cf0b0 0x7f0a2c0cd1e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 -- 172.21.15.104:0/2571505071 >> 172.21.15.104:0/2571505071 conn(0x7f0a2c06b8a0 msgr2=0x7f0a2c0cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 -- 172.21.15.104:0/2571505071 shutdown_connections 2023-06-10T19:00:09.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.935+0000 7f0a345c7700 1 -- 172.21.15.104:0/2571505071 wait complete. 2023-06-10T19:00:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.937+0000 7f0a345c7700 1 Processor -- start 2023-06-10T19:00:09.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.937+0000 7f0a345c7700 1 -- start start 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cd720 0x7f0a2c1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a2c0cf0b0 0x7f0a2c1f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0c5900 con 0x7f0a2c0cb210 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0c5620 con 0x7f0a2c0cf0b0 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0b3580 con 0x7f0a2c0cd720 2023-06-10T19:00:09.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42402/0 (socket says 172.21.15.104:42402) 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 -- 172.21.15.104:0/3027693266 learned_addr learned my addr 172.21.15.104:0/3027693266 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a31b62700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cd720 0x7f0a2c1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32b64700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a2c0cf0b0 0x7f0a2c1f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 -- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cd720 msgr2=0x7f0a2c1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cd720 0x7f0a2c1506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 -- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a2c0cf0b0 msgr2=0x7f0a2c1f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:09.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a2c0cf0b0 0x7f0a2c1f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 -- 172.21.15.104:0/3027693266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c009430 con 0x7f0a2c0cb210 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a32363700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00ebe0 tx=0x7f0a1c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c005a20 con 0x7f0a2c0cb210 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1c005b80 con 0x7f0a2c0cb210 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a280094a0 con 0x7f0a2c0cb210 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c00c850 con 0x7f0a2c0cb210 2023-06-10T19:00:09.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.938+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c0b3b40 con 0x7f0a2c0cb210 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.939+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c074050 con 0x7f0a2c0cb210 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.940+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0a1c00c9b0 con 0x7f0a2c0cb210 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.940+0000 7f0a22ffd700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 0x7f0a1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.940+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f0a1c0897c0 con 0x7f0a2c0cb210 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.940+0000 7f0a31b62700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 0x7f0a1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:09.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.941+0000 7f0a31b62700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 0x7f0a1006d190 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f0a280090b0 tx=0x7f0a28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:09.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:09.944+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0a1c054740 con 0x7f0a2c0cb210 2023-06-10T19:00:10.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.174+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a2c152520 con 0x7f0a2c0cb210 2023-06-10T19:00:10.802 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.795+0000 7f0a22ffd700 1 -- 172.21.15.104:0/3027693266 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a1c057d60 con 0x7f0a2c0cb210 2023-06-10T19:00:10.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 msgr2=0x7f0a1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 0x7f0a1006d190 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f0a280090b0 tx=0x7f0a28008030 comp rx=0 tx=0).stop 2023-06-10T19:00:10.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 msgr2=0x7f0a2c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00ebe0 tx=0x7f0a1c00eef0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 shutdown_connections 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0a1006ace0 0x7f0a1006d190 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a2c0cf0b0 0x7f0a2c1f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a2c0cb210 0x7f0a2c0b4ea0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 --2- 172.21.15.104:0/3027693266 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a2c0cd720 0x7f0a2c1506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 >> 172.21.15.104:0/3027693266 conn(0x7f0a2c06b8a0 msgr2=0x7f0a2c06a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:10.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 shutdown_connections 2023-06-10T19:00:10.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.799+0000 7f0a345c7700 1 -- 172.21.15.104:0/3027693266 wait complete. 2023-06-10T19:00:10.820 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-06-10T19:00:10.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.945+0000 7f0dddd0e700 1 Processor -- start 2023-06-10T19:00:10.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.945+0000 7f0dddd0e700 1 -- start start 2023-06-10T19:00:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dddd0e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0dd806e3e0 0x7f0dd806a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dddd0e700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd815c8d0 con 0x7f0dd8069e70 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dddd0e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd806abd0 con 0x7f0dd815c500 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dddd0e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd80aef20 con 0x7f0dd806e3e0 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd6ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0dd8069e70 0x7f0dd806dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40242/0 (socket says 172.21.15.104:40242) 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd6ffd700 1 -- 172.21.15.104:0/2027320770 learned_addr learned my addr 172.21.15.104:0/2027320770 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd7fff700 1 --2- 172.21.15.104:0/2027320770 >> v2:172.21.15.114:3300/0 conn(0x7f0dd806e3e0 0x7f0dd806a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477470008 0 0) 0x7f0dd815c8d0 con 0x7f0dd8069e70 2023-06-10T19:00:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc00035b0 con 0x7f0dd8069e70 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995408804 0 0) 0x7f0dc00035b0 con 0x7f0dd8069e70 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 >> v2:172.21.15.114:3300/0 conn(0x7f0dd806e3e0 msgr2=0x7f0dd806a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 --2- 172.21.15.104:0/2027320770 >> v2:172.21.15.114:3300/0 conn(0x7f0dd806e3e0 0x7f0dd806a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 >> v1:172.21.15.114:6789/0 conn(0x7f0dd815c500 legacy=0x7f0dd80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd806a8f0 con 0x7f0dd8069e70 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0dc8001670 con 0x7f0dd8069e70 2023-06-10T19:00:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 >> v1:172.21.15.104:6789/0 conn(0x7f0dd8069e70 legacy=0x7f0dd806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 0x7f0dc0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.946+0000 7f0dd67fc700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0004850 0x7f0dc0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dc0005200 0x7f0dc000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc000c050 con 0x7f0dc0003a00 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc000c1c0 con 0x7f0dc0004850 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc000c380 con 0x7f0dc0005200 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 0x7f0dc0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd6ffd700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0004850 0x7f0dc0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd7fff700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dc0005200 0x7f0dc000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 -- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dc0005200 msgr2=0x7f0dc000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dc0005200 0x7f0dc000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 -- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0004850 msgr2=0x7f0dc0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0004850 0x7f0dc0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 -- 172.21.15.104:0/2027320770 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00035b0 con 0x7f0dc0003a00 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd7fff700 1 -- 172.21.15.104:0/2027320770 reap_dead start 2023-06-10T19:00:10.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 0x7f0dc0003e10 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc001820 tx=0x7f0dc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9baf7cff4db34b98 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc0012e0 con 0x7f0dc0003a00 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc0052d0 con 0x7f0dc0003a00 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.947+0000 7f0dd67fc700 1 -- 172.21.15.104:0/2027320770 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc0036d0 con 0x7f0dc0003a00 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 msgr2=0x7f0dc0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 0x7f0dc0003e10 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc001820 tx=0x7f0dc0005880 comp rx=0 tx=0).stop 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2027320770 shutdown_connections 2023-06-10T19:00:10.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2027320770 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dc0003a00 0x7f0dc0003e10 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2027320770 >> 172.21.15.104:0/2027320770 conn(0x7f0dd806b8a0 msgr2=0x7f0dd80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2027320770 shutdown_connections 2023-06-10T19:00:10.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.948+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2027320770 wait complete. 2023-06-10T19:00:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.949+0000 7f0dddd0e700 1 Processor -- start 2023-06-10T19:00:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.949+0000 7f0dddd0e700 1 -- start start 2023-06-10T19:00:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 0x7f0dd8159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd80c3b60 0x7f0dd80c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd815a4b0 con 0x7f0dd815edc0 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd80b26b0 con 0x7f0dc0003a00 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dddd0e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd80c4900 con 0x7f0dd80c3b60 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42426/0 (socket says 172.21.15.104:42426) 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 -- 172.21.15.104:0/2212033446 learned_addr learned my addr 172.21.15.104:0/2212033446 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:10.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd7fff700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd80c3b60 0x7f0dd80c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 0x7f0dd8159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 -- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd80c3b60 msgr2=0x7f0dd80c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd80c3b60 0x7f0dd80c3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 -- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 msgr2=0x7f0dd8159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 0x7f0dd8159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 -- 172.21.15.104:0/2212033446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd80c4be0 con 0x7f0dd815edc0 2023-06-10T19:00:10.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 0x7f0dd8159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dd6ffd700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80072c0 tx=0x7f0dc8007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.950+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80077a0 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.951+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800f8d0 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.951+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd80c4ec0 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.951+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800fc80 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.951+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd815f770 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.951+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd8074050 con 0x7f0dd815edc0 2023-06-10T19:00:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.952+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0dc801d460 con 0x7f0dd815edc0 2023-06-10T19:00:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.952+0000 7f0dbffff700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 0x7f0dc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.952+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f0dc8089660 con 0x7f0dd815edc0 2023-06-10T19:00:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.952+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 0x7f0dc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.953+0000 7f0dd77fe700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 0x7f0dc406d190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc015040 tx=0x7f0dcc009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:10.956+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0dc80546c0 con 0x7f0dd815edc0 2023-06-10T19:00:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.145+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0dd8066f10 con 0x7f0dd815edc0 2023-06-10T19:00:11.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.800+0000 7f0dbffff700 1 -- 172.21.15.104:0/2212033446 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v190) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0dc8057ce0 con 0x7f0dd815edc0 2023-06-10T19:00:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 msgr2=0x7f0dc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 0x7f0dc406d190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc015040 tx=0x7f0dcc009030 comp rx=0 tx=0).stop 2023-06-10T19:00:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 msgr2=0x7f0dd815f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80072c0 tx=0x7f0dc8007680 comp rx=0 tx=0).stop 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 shutdown_connections 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0dc406ace0 0x7f0dc406d190 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dc0003a00 0x7f0dd8159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd80c3b60 0x7f0dd80c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 --2- 172.21.15.104:0/2212033446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dd815edc0 0x7f0dd815f230 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.804+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 >> 172.21.15.104:0/2212033446 conn(0x7f0dd806b8a0 msgr2=0x7f0dd80ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:11.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.805+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 shutdown_connections 2023-06-10T19:00:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.805+0000 7f0dddd0e700 1 -- 172.21.15.104:0/2212033446 wait complete. 2023-06-10T19:00:11.813 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_metadata' removed 2023-06-10T19:00:11.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-06-10T19:00:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.954+0000 7fe6912c7700 1 Processor -- start 2023-06-10T19:00:11.961 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.954+0000 7fe6912c7700 1 -- start start 2023-06-10T19:00:11.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe6912c7700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe68c160ad0 0x7fe68c0ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe6912c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe6912c7700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c15c820 con 0x7fe68c06dea0 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe6912c7700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68c15cad0 con 0x7fe68c160ad0 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe6912c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c15cdb0 con 0x7fe68c06cd50 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe68a7fc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe68c160ad0 0x7fe68c0ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:11.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe68affd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe68c06dea0 0x7fe68c06e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40260/0 (socket says 172.21.15.104:40260) 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.955+0000 7fe68affd700 1 -- 172.21.15.104:0/1409153741 learned_addr learned my addr 172.21.15.104:0/1409153741 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe68a7fc700 1 -- 172.21.15.104:0/1409153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 msgr2=0x7fe68c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe68a7fc700 1 --2- 172.21.15.104:0/1409153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe68a7fc700 1 -- 172.21.15.104:0/1409153741 >> v1:172.21.15.104:6789/0 conn(0x7fe68c06dea0 legacy=0x7fe68c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe68a7fc700 1 -- 172.21.15.104:0/1409153741 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c15d090 con 0x7fe68c160ad0 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe689ffb700 1 -- 172.21.15.104:0/1409153741 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263221600 0 0) 0x7fe68c15c820 con 0x7fe68c06dea0 2023-06-10T19:00:11.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe68a7fc700 1 --2- 172.21.15.104:0/1409153741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c160ad0 0x7fe68c0ae4d0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009840 tx=0x7fe67c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf02f701bb443db3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe689ffb700 1 -- 172.21.15.104:0/1409153741 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c011830 con 0x7fe68c160ad0 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe689ffb700 1 -- 172.21.15.104:0/1409153741 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c011990 con 0x7fe68c160ad0 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe689ffb700 1 -- 172.21.15.104:0/1409153741 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c011d20 con 0x7fe68c160ad0 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 -- 172.21.15.104:0/1409153741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c160ad0 msgr2=0x7fe68c0ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 --2- 172.21.15.104:0/1409153741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c160ad0 0x7fe68c0ae4d0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009840 tx=0x7fe67c009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 -- 172.21.15.104:0/1409153741 shutdown_connections 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 --2- 172.21.15.104:0/1409153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 --2- 172.21.15.104:0/1409153741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c160ad0 0x7fe68c0ae4d0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 -- 172.21.15.104:0/1409153741 >> 172.21.15.104:0/1409153741 conn(0x7fe68c06b8a0 msgr2=0x7fe68c0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.956+0000 7fe6912c7700 1 -- 172.21.15.104:0/1409153741 shutdown_connections 2023-06-10T19:00:11.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.957+0000 7fe6912c7700 1 -- 172.21.15.104:0/1409153741 wait complete. 2023-06-10T19:00:11.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.958+0000 7fe6912c7700 1 Processor -- start 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.958+0000 7fe6912c7700 1 -- start start 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe68c160ad0 0x7fe68c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c14f160 con 0x7fe68c06dea0 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0adb80 con 0x7fe68c160ad0 2023-06-10T19:00:11.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe6912c7700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c157010 con 0x7fe68c06cd50 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42440/0 (socket says 172.21.15.104:42440) 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68a7fc700 1 -- 172.21.15.104:0/2103311546 learned_addr learned my addr 172.21.15.104:0/2103311546 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68affd700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68a7fc700 1 -- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 msgr2=0x7fe68c15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.959+0000 7fe68b7fe700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe68c160ad0 0x7fe68c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:11.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe68a7fc700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c15f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe68a7fc700 1 -- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe68c160ad0 msgr2=0x7fe68c0acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe68a7fc700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe68c160ad0 0x7fe68c0acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe68a7fc700 1 -- 172.21.15.104:0/2103311546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0094a0 con 0x7fe68c06dea0 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe68a7fc700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0090b0 tx=0x7fe67c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0036a0 con 0x7fe68c06dea0 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c003800 con 0x7fe68c06dea0 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c15f800 con 0x7fe68c06dea0 2023-06-10T19:00:11.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c003bb0 con 0x7fe68c06dea0 2023-06-10T19:00:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.960+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c156ca0 con 0x7fe68c06dea0 2023-06-10T19:00:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.961+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c0c9b30 con 0x7fe68c06dea0 2023-06-10T19:00:11.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.962+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe67c003d10 con 0x7fe68c06dea0 2023-06-10T19:00:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.962+0000 7fe6737fe700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 0x7fe67806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.962+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe67c08b1d0 con 0x7fe68c06dea0 2023-06-10T19:00:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.962+0000 7fe68affd700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 0x7fe67806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.963+0000 7fe68affd700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 0x7fe67806d190 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe680001360 tx=0x7fe680009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:11.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:11.966+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe67c0563e0 con 0x7fe68c06dea0 2023-06-10T19:00:12.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.151+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe68c158b70 con 0x7fe68c06dea0 2023-06-10T19:00:12.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.804+0000 7fe6737fe700 1 -- 172.21.15.104:0/2103311546 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v191) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe67c059a00 con 0x7fe68c06dea0 2023-06-10T19:00:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 msgr2=0x7fe67806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 0x7fe67806d190 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe680001360 tx=0x7fe680009040 comp rx=0 tx=0).stop 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 msgr2=0x7fe68c14ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0090b0 tx=0x7fe67c004750 comp rx=0 tx=0).stop 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 shutdown_connections 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe67806ace0 0x7fe67806d190 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe68c160ad0 0x7fe68c0acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe68c06cd50 0x7fe68c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.816 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 --2- 172.21.15.104:0/2103311546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe68c06dea0 0x7fe68c14ec20 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 >> 172.21.15.104:0/2103311546 conn(0x7fe68c06b8a0 msgr2=0x7fe68c06d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 shutdown_connections 2023-06-10T19:00:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.809+0000 7fe6912c7700 1 -- 172.21.15.104:0/2103311546 wait complete. 2023-06-10T19:00:12.817 INFO:teuthology.orchestra.run.smithi104.stderr:pool 'cephfs_data' removed 2023-06-10T19:00:12.834 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-06-10T19:00:12.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.952+0000 7f94700cb700 1 Processor -- start 2023-06-10T19:00:12.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.952+0000 7f94700cb700 1 -- start start 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f94700cb700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f94680ae9c0 0x7f946806a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f94700cb700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680d0530 con 0x7f946806e0b0 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f94700cb700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946806cf10 con 0x7f946806d450 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f94700cb700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680cb220 con 0x7f94680ae9c0 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f946e668700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f946806e0b0 0x7f94680cb640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40278/0 (socket says 172.21.15.104:40278) 2023-06-10T19:00:12.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.953+0000 7f946e668700 1 -- 172.21.15.104:0/2434010850 learned_addr learned my addr 172.21.15.104:0/2434010850 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850316406 0 0) 0x7f94680d0530 con 0x7f946806e0b0 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540035b0 con 0x7f946806e0b0 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946de67700 1 --2- 172.21.15.104:0/2434010850 >> v2:172.21.15.114:3300/0 conn(0x7f94680ae9c0 0x7f946806a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587975765 0 0) 0x7f94540035b0 con 0x7f946806e0b0 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 >> v2:172.21.15.114:3300/0 conn(0x7f94680ae9c0 msgr2=0x7f946806a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 --2- 172.21.15.104:0/2434010850 >> v2:172.21.15.114:3300/0 conn(0x7f94680ae9c0 0x7f946806a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 >> v1:172.21.15.114:6789/0 conn(0x7f946806d450 legacy=0x7f946806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946806d1f0 con 0x7f946806e0b0 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f94600019f0 con 0x7f946806e0b0 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 >> v1:172.21.15.104:6789/0 conn(0x7f946806e0b0 legacy=0x7f94680cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454003a00 0x7f9454003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 0x7f9454004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9454005200 0x7f945400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680cb220 con 0x7f9454004850 2023-06-10T19:00:12.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945400c190 con 0x7f9454003a00 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945400c350 con 0x7f9454005200 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 0x7f9454004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946de67700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454003a00 0x7f9454003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946e668700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9454005200 0x7f945400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 -- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9454005200 msgr2=0x7f945400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9454005200 0x7f945400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 -- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454003a00 msgr2=0x7f9454003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454003a00 0x7f9454003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 -- 172.21.15.104:0/2434010850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94540035b0 con 0x7f9454004850 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946e668700 1 -- 172.21.15.104:0/2434010850 reap_dead start 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.954+0000 7f946d666700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 0x7f9454004cc0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f9464007fd0 tx=0x7f9454005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2295020840d1cb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640052d0 con 0x7f9454004850 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464005450 con 0x7f9454004850 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f946ce65700 1 -- 172.21.15.104:0/2434010850 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640038c0 con 0x7f9454004850 2023-06-10T19:00:12.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 -- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 msgr2=0x7f9454004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 0x7f9454004cc0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f9464007fd0 tx=0x7f9454005880 comp rx=0 tx=0).stop 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 -- 172.21.15.104:0/2434010850 shutdown_connections 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 --2- 172.21.15.104:0/2434010850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9454004850 0x7f9454004cc0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 -- 172.21.15.104:0/2434010850 >> 172.21.15.104:0/2434010850 conn(0x7f946806b8a0 msgr2=0x7f94680cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 -- 172.21.15.104:0/2434010850 shutdown_connections 2023-06-10T19:00:12.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.955+0000 7f94700cb700 1 -- 172.21.15.104:0/2434010850 wait complete. 2023-06-10T19:00:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.956+0000 7f94700cb700 1 Processor -- start 2023-06-10T19:00:12.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.956+0000 7f94700cb700 1 -- start start 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 0x7f9468152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94680c50b0 0x7f94680c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680c5a60 con 0x7f94680b0a90 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680c3ae0 con 0x7f9454004850 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f94700cb700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680c3dc0 con 0x7f94680c50b0 2023-06-10T19:00:12.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42462/0 (socket says 172.21.15.104:42462) 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 -- 172.21.15.104:0/2686973283 learned_addr learned my addr 172.21.15.104:0/2686973283 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946de67700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 0x7f9468152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946d666700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94680c50b0 0x7f94680c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 -- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94680c50b0 msgr2=0x7f94680c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94680c50b0 0x7f94680c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.957+0000 7f946e668700 1 -- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 msgr2=0x7f9468152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f946e668700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 0x7f9468152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f946e668700 1 -- 172.21.15.104:0/2686973283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680c40a0 con 0x7f94680b0a90 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f946de67700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 0x7f9468152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f946e668700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f9460000f80 tx=0x7f9460001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946000ae60 con 0x7f94680b0a90 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94600133f0 con 0x7f94680b0a90 2023-06-10T19:00:12.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94680c4380 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460013780 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.958+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94680b1440 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.959+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680c48a0 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.959+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94600138e0 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.960+0000 7f945e7fc700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 0x7f944c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.960+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f946008a0e0 con 0x7f94680b0a90 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.960+0000 7f946de67700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 0x7f944c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:12.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.960+0000 7f946de67700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 0x7f944c06d190 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f9468063ac0 tx=0x7f9458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:12.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:12.964+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94680c48a0 con 0x7f94680b0a90 2023-06-10T19:00:13.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.153+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f94680c48a0 con 0x7f94680b0a90 2023-06-10T19:00:13.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.157+0000 7f945e7fc700 1 -- 172.21.15.104:0/2686973283 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f94680c48a0 con 0x7f94680b0a90 2023-06-10T19:00:13.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 msgr2=0x7f944c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 0x7f944c06d190 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f9468063ac0 tx=0x7f9458008040 comp rx=0 tx=0).stop 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 msgr2=0x7f94680b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f9460000f80 tx=0x7f9460001ae0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 shutdown_connections 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f944c06ace0 0x7f944c06d190 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9454004850 0x7f9468152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94680b0a90 0x7f94680b0f00 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 --2- 172.21.15.104:0/2686973283 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f94680c50b0 0x7f94680c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 >> 172.21.15.104:0/2686973283 conn(0x7f946806b8a0 msgr2=0x7f9468159620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:13.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 shutdown_connections 2023-06-10T19:00:13.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.159+0000 7f94700cb700 1 -- 172.21.15.104:0/2686973283 wait complete. 2023-06-10T19:00:13.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted' 2023-06-10T19:00:13.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.291+0000 7f22632d2700 1 Processor -- start 2023-06-10T19:00:13.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.291+0000 7f22632d2700 1 -- start start 2023-06-10T19:00:13.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f22632d2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 0x7f225c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f22632d2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f225c069e70 0x7f225c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f22632d2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c06d290 con 0x7f225c06e4b0 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f22632d2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c071640 con 0x7f225c069e70 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f22632d2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0aef00 con 0x7f225c06e0e0 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f226186f700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f225c069e70 0x7f225c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f226086d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f225c06e4b0 0x7f225c0c9bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40296/0 (socket says 172.21.15.104:40296) 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.292+0000 7f226086d700 1 -- 172.21.15.104:0/1232478732 learned_addr learned my addr 172.21.15.104:0/1232478732 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:13.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226106e700 1 --2- 172.21.15.104:0/1232478732 >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 0x7f225c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226186f700 1 -- 172.21.15.104:0/1232478732 >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 msgr2=0x7f225c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226186f700 1 --2- 172.21.15.104:0/1232478732 >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 0x7f225c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226186f700 1 -- 172.21.15.104:0/1232478732 >> v1:172.21.15.104:6789/0 conn(0x7f225c06e4b0 legacy=0x7f225c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226186f700 1 -- 172.21.15.104:0/1232478732 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0ca2d0 con 0x7f225c069e70 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226106e700 1 --2- 172.21.15.104:0/1232478732 >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 0x7f225c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f226186f700 1 --2- 172.21.15.104:0/1232478732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c069e70 0x7f225c0ce9f0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f225c06af50 tx=0x7f225400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7489704f0434a39e server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:13.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f2253fff700 1 -- 172.21.15.104:0/1232478732 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225400e070 con 0x7f225c069e70 2023-06-10T19:00:13.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f2253fff700 1 -- 172.21.15.104:0/1232478732 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225400bbd0 con 0x7f225c069e70 2023-06-10T19:00:13.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f2253fff700 1 -- 172.21.15.104:0/1232478732 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012430 con 0x7f225c069e70 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f22632d2700 1 -- 172.21.15.104:0/1232478732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c069e70 msgr2=0x7f225c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.293+0000 7f22632d2700 1 --2- 172.21.15.104:0/1232478732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c069e70 0x7f225c0ce9f0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f225c06af50 tx=0x7f225400b330 comp rx=0 tx=0).stop 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 -- 172.21.15.104:0/1232478732 shutdown_connections 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 --2- 172.21.15.104:0/1232478732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c069e70 0x7f225c0ce9f0 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 --2- 172.21.15.104:0/1232478732 >> v2:172.21.15.114:3300/0 conn(0x7f225c06e0e0 0x7f225c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 -- 172.21.15.104:0/1232478732 >> 172.21.15.104:0/1232478732 conn(0x7f225c06b8a0 msgr2=0x7f225c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 -- 172.21.15.104:0/1232478732 shutdown_connections 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.294+0000 7f22632d2700 1 -- 172.21.15.104:0/1232478732 wait complete. 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.295+0000 7f22632d2700 1 Processor -- start 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.295+0000 7f22632d2700 1 -- start start 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.295+0000 7f22632d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225c069e70 0x7f225c155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.295+0000 7f22632d2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f225c06e0e0 0x7f225c0af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.295+0000 7f22632d2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f22632d2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0b1fa0 con 0x7f225c06e4b0 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f22632d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c155580 con 0x7f225c069e70 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f22632d2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0b1700 con 0x7f225c06e0e0 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42476/0 (socket says 172.21.15.104:42476) 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 -- 172.21.15.104:0/2318697075 learned_addr learned my addr 172.21.15.104:0/2318697075 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226106e700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225c069e70 0x7f225c155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226086d700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f225c06e0e0 0x7f225c0af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 -- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f225c06e0e0 msgr2=0x7f225c0af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f225c06e0e0 0x7f225c0af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 -- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225c069e70 msgr2=0x7f225c155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225c069e70 0x7f225c155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 -- 172.21.15.104:0/2318697075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254009a70 con 0x7f225c06e4b0 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f226186f700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f225400b370 tx=0x7f2254004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012c40 con 0x7f225c06e4b0 2023-06-10T19:00:13.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2254020070 con 0x7f225c06e4b0 2023-06-10T19:00:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254004040 con 0x7f225c06e4b0 2023-06-10T19:00:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c0b19e0 con 0x7f225c06e4b0 2023-06-10T19:00:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.296+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c0af170 con 0x7f225c06e4b0 2023-06-10T19:00:13.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.297+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c074050 con 0x7f225c06e4b0 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.298+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f22540041a0 con 0x7f225c06e4b0 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.298+0000 7f2251ffb700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 0x7f224006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.298+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2254022070 con 0x7f225c06e4b0 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.299+0000 7f226106e700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 0x7f224006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.299+0000 7f226106e700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 0x7f224006d190 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f224c00ab40 tx=0x7f224c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:13.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.302+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22540569d0 con 0x7f225c06e4b0 2023-06-10T19:00:13.573 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.566+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted"]} v 0) v1 -- 0x7f225c06e8b0 con 0x7f225c06e4b0 2023-06-10T19:00:13.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.816+0000 7f2251ffb700 1 -- 172.21.15.104:0/2318697075 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_client_recovery.TestClientRecovery.test_reconnect_after_blocklisted"]}]=0 v774) v1 ==== 164+0+0 (secure 0 0 0) 0x7f225402c0e0 con 0x7f225c06e4b0 2023-06-10T19:00:13.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 msgr2=0x7f224006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 0x7f224006d190 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f224c00ab40 tx=0x7f224c008040 comp rx=0 tx=0).stop 2023-06-10T19:00:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 msgr2=0x7f225c0adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f225400b370 tx=0x7f2254004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 shutdown_connections 2023-06-10T19:00:13.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f224006ace0 0x7f224006d190 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f225c069e70 0x7f225c155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f225c06e4b0 0x7f225c0adb80 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 --2- 172.21.15.104:0/2318697075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f225c06e0e0 0x7f225c0af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:13.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 >> 172.21.15.104:0/2318697075 conn(0x7f225c06b8a0 msgr2=0x7f225c0ac480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:13.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 shutdown_connections 2023-06-10T19:00:13.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:13.818+0000 7f22632d2700 1 -- 172.21.15.104:0/2318697075 wait complete. 2023-06-10T19:00:13.837 INFO:tasks.cephfs_test_runner:test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) ... ERROR 2023-06-10T19:00:13.837 INFO:tasks.cephfs_test_runner: 2023-06-10T19:00:13.837 INFO:tasks.cephfs_test_runner:====================================================================== 2023-06-10T19:00:13.838 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T19:00:13.838 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-06-10T19:00:13.838 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-06-10T19:00:13.838 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa/tasks/cephfs/test_client_recovery.py", line 725, in test_reconnect_after_blocklisted 2023-06-10T19:00:13.838 INFO:tasks.cephfs_test_runner: rproc.wait() 2023-06-10T19:00:13.839 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/run.py", line 161, in wait 2023-06-10T19:00:13.839 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-06-10T19:00:13.839 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-06-10T19:00:13.839 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-06-10T19:00:13.839 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi104 with status 1: 'python3 -c \'\nimport os\nimport sys\nimport fcntl\nimport errno\nimport time\n\nfd1 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDWR | os.O_CREAT, 0O666)\nfd2 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDONLY)\nfd3 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDWR | os.O_CREAT, 0O666)\nfd4 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDONLY)\n\nos.write(fd1, b\'"\'"\'content\'"\'"\')\nos.read(fd2, 1);\n\nos.write(fd3, b\'"\'"\'content\'"\'"\')\nos.read(fd4, 1);\nfcntl.flock(fd4, fcntl.LOCK_SH | fcntl.LOCK_NB)\n\nprint("blocklist")\nsys.stdout.flush()\n\nsys.stdin.readline()\n\n# wait for mds to close session\ntime.sleep(10);\n\n# trigger \'"\'"\'open session\'"\'"\' message. kclient relies on \'"\'"\'session reject\'"\'"\' message\n# to detect if itself is blocklisted\ntry:\n os.stat("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1")\nexcept:\n pass\n\n# wait for auto reconnect\ntime.sleep(10);\n\ntry:\n os.write(fd1, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EBADF:\n raise\nelse:\n raise RuntimeError("write() failed to raise error")\n\nos.read(fd2, 1);\n\ntry:\n os.read(fd4, 1)\nexcept OSError as e:\n if e.errno != errno.EIO:\n raise\nelse:\n raise RuntimeError("read() failed to raise error")\n\'' 2023-06-10T19:00:13.840 INFO:tasks.cephfs_test_runner: 2023-06-10T19:00:13.840 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-06-10T19:00:13.840 INFO:tasks.cephfs_test_runner:Ran 12 tests in 911.931s 2023-06-10T19:00:13.840 INFO:tasks.cephfs_test_runner: 2023-06-10T19:00:13.841 INFO:tasks.cephfs_test_runner:FAILED (errors=1, skipped=1) 2023-06-10T19:00:13.841 INFO:tasks.cephfs_test_runner: 2023-06-10T19:00:13.841 INFO:tasks.cephfs_test_runner:====================================================================== 2023-06-10T19:00:13.841 INFO:tasks.cephfs_test_runner:ERROR: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T19:00:13.841 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-06-10T19:00:13.842 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-06-10T19:00:13.842 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa/tasks/cephfs/test_client_recovery.py", line 725, in test_reconnect_after_blocklisted 2023-06-10T19:00:13.842 INFO:tasks.cephfs_test_runner: rproc.wait() 2023-06-10T19:00:13.842 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/run.py", line 161, in wait 2023-06-10T19:00:13.842 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-06-10T19:00:13.843 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-06-10T19:00:13.843 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-06-10T19:00:13.843 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi104 with status 1: 'python3 -c \'\nimport os\nimport sys\nimport fcntl\nimport errno\nimport time\n\nfd1 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDWR | os.O_CREAT, 0O666)\nfd2 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1", os.O_RDONLY)\nfd3 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDWR | os.O_CREAT, 0O666)\nfd4 = os.open("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.2", os.O_RDONLY)\n\nos.write(fd1, b\'"\'"\'content\'"\'"\')\nos.read(fd2, 1);\n\nos.write(fd3, b\'"\'"\'content\'"\'"\')\nos.read(fd4, 1);\nfcntl.flock(fd4, fcntl.LOCK_SH | fcntl.LOCK_NB)\n\nprint("blocklist")\nsys.stdout.flush()\n\nsys.stdin.readline()\n\n# wait for mds to close session\ntime.sleep(10);\n\n# trigger \'"\'"\'open session\'"\'"\' message. kclient relies on \'"\'"\'session reject\'"\'"\' message\n# to detect if itself is blocklisted\ntry:\n os.stat("/home/ubuntu/cephtest/mnt.0/testfile_reconnect_after_blocklisted.1")\nexcept:\n pass\n\n# wait for auto reconnect\ntime.sleep(10);\n\ntry:\n os.write(fd1, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EBADF:\n raise\nelse:\n raise RuntimeError("write() failed to raise error")\n\nos.read(fd2, 1);\n\ntry:\n os.read(fd4, 1)\nexcept OSError as e:\n if e.errno != errno.EIO:\n raise\nelse:\n raise RuntimeError("read() failed to raise error")\n\'' 2023-06-10T19:00:13.843 INFO:tasks.cephfs_test_runner: 2023-06-10T19:00:13.844 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T19:00:13.908 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b06794b35d2d41dba25baa651e3bc133 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3486dd872ffd89042ca079d680841ce3ef6f4b6f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-06-10T19:00:13.910 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-06-10T19:00:13.920 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-06-10T19:00:13.953 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-06-10T19:00:13.953 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T19:00:13.954 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:13.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T19:00:13.983 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-06-10T19:00:13.983 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:14.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284264k,nr_inodes=4071066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17381 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-06-10T19:00:14.040 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T19:00:14.040 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:14.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T19:00:14.061 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-06-10T19:00:14.061 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-10T19:00:14.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16708 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-06-10T19:00:14.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.122 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2023-06-10T19:00:14.182 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-06-10T19:00:14.182 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-06-10T19:00:14.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-06-10T19:00:14.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T19:00:14.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2023-06-10T19:00:14.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.453 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2023-06-10T19:00:14.471 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 2) 2023-06-10T19:00:14.471 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-06-10T19:00:14.472 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-10T19:00:14.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-06-10T19:00:14.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.553 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-06-10T19:00:14.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-10T19:00:14.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-10T19:00:14.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2023-06-10T19:00:14.839 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-10T19:00:14.852 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-06-10T19:00:14.853 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T19:00:14.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.968+0000 7f2cc8af9700 1 Processor -- start 2023-06-10T19:00:14.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.968+0000 7f2cc8af9700 1 -- start start 2023-06-10T19:00:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.969+0000 7f2cc8af9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc40ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.969+0000 7f2cc8af9700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2cc40c8c40 0x7f2cc40c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.969+0000 7f2cc8af9700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc415c050 con 0x7f2cc415bc80 2023-06-10T19:00:14.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.969+0000 7f2cc8af9700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc40ceea0 con 0x7f2cc40c8c40 2023-06-10T19:00:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.969+0000 7f2cc8af9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40c9970 con 0x7f2cc40ce510 2023-06-10T19:00:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc1d9b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc40ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2cc40c8c40 0x7f2cc40c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc259c700 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2cc415bc80 0x7f2cc4070310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.104:46154/0 (socket says 172.21.15.104:46154) 2023-06-10T19:00:14.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc259c700 1 -- 172.21.15.104:0/624445503 learned_addr learned my addr 172.21.15.104:0/624445503 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 -- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 msgr2=0x7f2cc40ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc40ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 -- 172.21.15.104:0/624445503 >> v1:172.21.15.114:6789/0 conn(0x7f2cc415bc80 legacy=0x7f2cc4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 -- 172.21.15.104:0/624445503 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40ccb70 con 0x7f2cc40c8c40 2023-06-10T19:00:14.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc159a700 1 -- 172.21.15.104:0/624445503 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329158192 0 0) 0x7f2cc415c050 con 0x7f2cc415bc80 2023-06-10T19:00:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc1d9b700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc40ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc2d9d700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c90c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bf7d743a8ccc678 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.970+0000 7f2cc159a700 1 -- 172.21.15.104:0/624445503 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc00e070 con 0x7f2cc40c8c40 2023-06-10T19:00:14.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc159a700 1 -- 172.21.15.104:0/624445503 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc00bb20 con 0x7f2cc40c8c40 2023-06-10T19:00:14.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc159a700 1 -- 172.21.15.104:0/624445503 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc012430 con 0x7f2cc40c8c40 2023-06-10T19:00:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 -- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc40c8c40 msgr2=0x7f2cc40c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c90c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:14.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 -- 172.21.15.104:0/624445503 shutdown_connections 2023-06-10T19:00:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c90c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/624445503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc40ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 -- 172.21.15.104:0/624445503 >> 172.21.15.104:0/624445503 conn(0x7f2cc406ab00 msgr2=0x7f2cc40b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 -- 172.21.15.104:0/624445503 shutdown_connections 2023-06-10T19:00:14.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.971+0000 7f2cc8af9700 1 -- 172.21.15.104:0/624445503 wait complete. 2023-06-10T19:00:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.973+0000 7f2cc8af9700 1 Processor -- start 2023-06-10T19:00:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.973+0000 7f2cc8af9700 1 -- start start 2023-06-10T19:00:14.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.973+0000 7f2cc8af9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.973+0000 7f2cc8af9700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc41ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 0x7f2cc40b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40c20b0 con 0x7f2cc40c8c40 2023-06-10T19:00:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b40a0 con 0x7f2cc415bc80 2023-06-10T19:00:14.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40c22c0 con 0x7f2cc40ce510 2023-06-10T19:00:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42488/0 (socket says 172.21.15.104:42488) 2023-06-10T19:00:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 -- 172.21.15.104:0/191171374 learned_addr learned my addr 172.21.15.104:0/191171374 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc2d9d700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 0x7f2cc40b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.987 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc1d9b700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc41ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 -- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 msgr2=0x7f2cc41ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc41ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 -- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 msgr2=0x7f2cc40b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 0x7f2cc40b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:14.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 -- 172.21.15.104:0/191171374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb400b740 con 0x7f2cc40c8c40 2023-06-10T19:00:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc2d9d700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 0x7f2cc40b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc259c700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800c4d0 tx=0x7f2cb800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8005e20 con 0x7f2cc40c8c40 2023-06-10T19:00:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb80070d0 con 0x7f2cc40c8c40 2023-06-10T19:00:14.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc009a70 con 0x7f2cc40c8c40 2023-06-10T19:00:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8007480 con 0x7f2cc40c8c40 2023-06-10T19:00:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.974+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40c2880 con 0x7f2cc40c8c40 2023-06-10T19:00:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.975+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc40ce960 con 0x7f2cc40c8c40 2023-06-10T19:00:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.976+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2cb80075e0 con 0x7f2cc40c8c40 2023-06-10T19:00:14.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.976+0000 7f2cb2ffd700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 0x7f2ca406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.976+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2cb8088750 con 0x7f2cc40c8c40 2023-06-10T19:00:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.976+0000 7f2cc1d9b700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 0x7f2ca406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.977+0000 7f2cc1d9b700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 0x7f2ca406d190 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40b0480 tx=0x7f2cb400b420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:14.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:14.981+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2cb8053900 con 0x7f2cc40c8c40 2023-06-10T19:00:15.170 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.164+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cc40cd130 con 0x7f2ca406ace0 2023-06-10T19:00:15.173 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.167+0000 7f2cb2ffd700 1 -- 172.21.15.104:0/191171374 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72687 (secure 0 0 0) 0x7f2cb4011d80 con 0x7f2ca406ace0 2023-06-10T19:00:15.174 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:15.175 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 msgr2=0x7f2ca406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 0x7f2ca406d190 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40b0480 tx=0x7f2cb400b420 comp rx=0 tx=0).stop 2023-06-10T19:00:15.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 msgr2=0x7f2cc40c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.176 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800c4d0 tx=0x7f2cb800c7e0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 shutdown_connections 2023-06-10T19:00:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f2ca406ace0 0x7f2ca406d190 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cc415bc80 0x7f2cc40b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cc40c8c40 0x7f2cc40c78d0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 --2- 172.21.15.104:0/191171374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2cc40ce510 0x7f2cc41ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 >> 172.21.15.104:0/191171374 conn(0x7f2cc406ab00 msgr2=0x7f2cc415aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 shutdown_connections 2023-06-10T19:00:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.169+0000 7f2cc8af9700 1 -- 172.21.15.104:0/191171374 wait complete. 2023-06-10T19:00:15.178 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T19:00:15.187 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2023-06-10T19:00:13.606564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268132,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706588,"statfs":{"total":773094113280,"available":772819546112,"internally_reserved":0,"allocated":20221952,"data_stored":10557055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12945,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001679"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729349+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T19:00:09.729349+0000","last_peered":"2023-06-10T19:00:09.729349+0000","last_clean":"2023-06-10T19:00:09.729349+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T19:00:09.729349+0000","last_undegraded":"2023-06-10T19:00:09.729349+0000","last_fullsized":"2023-06-10T19:00:09.729349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:41.936611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731032+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T19:00:09.731032+0000","last_peered":"2023-06-10T19:00:09.731032+0000","last_clean":"2023-06-10T19:00:09.731032+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T19:00:09.731032+0000","last_undegraded":"2023-06-10T19:00:09.731032+0000","last_fullsized":"2023-06-10T19:00:09.731032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:07:44.579162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.905359+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T19:00:10.905359+0000","last_peered":"2023-06-10T19:00:10.905359+0000","last_clean":"2023-06-10T19:00:10.905359+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T19:00:10.905359+0000","last_undegraded":"2023-06-10T19:00:10.905359+0000","last_fullsized":"2023-06-10T19:00:10.905359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:47.908360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731009+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T19:00:09.731009+0000","last_peered":"2023-06-10T19:00:09.731009+0000","last_clean":"2023-06-10T19:00:09.731009+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T19:00:09.731009+0000","last_undegraded":"2023-06-10T19:00:09.731009+0000","last_fullsized":"2023-06-10T19:00:09.731009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:43:46.398137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":475,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729314+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T19:00:09.729314+0000","last_peered":"2023-06-10T19:00:09.729314+0000","last_clean":"2023-06-10T19:00:09.729314+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T19:00:09.729314+0000","last_undegraded":"2023-06-10T19:00:09.729314+0000","last_fullsized":"2023-06-10T19:00:09.729314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:49.946528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":319,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729170+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T19:00:09.729170+0000","last_peered":"2023-06-10T19:00:09.729170+0000","last_clean":"2023-06-10T19:00:09.729170+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T19:00:09.729170+0000","last_undegraded":"2023-06-10T19:00:09.729170+0000","last_fullsized":"2023-06-10T19:00:09.729170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:38:00.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729544+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T19:00:09.729544+0000","last_peered":"2023-06-10T19:00:09.729544+0000","last_clean":"2023-06-10T19:00:09.729544+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T19:00:09.729544+0000","last_undegraded":"2023-06-10T19:00:09.729544+0000","last_fullsized":"2023-06-10T19:00:09.729544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:59.472915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729631+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T19:00:09.729631+0000","last_peered":"2023-06-10T19:00:09.729631+0000","last_clean":"2023-06-10T19:00:09.729631+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T19:00:09.729631+0000","last_undegraded":"2023-06-10T19:00:09.729631+0000","last_fullsized":"2023-06-10T19:00:09.729631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:02:23.912395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.993635+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T19:00:10.993635+0000","last_peered":"2023-06-10T19:00:10.993635+0000","last_clean":"2023-06-10T19:00:10.993635+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T19:00:10.993635+0000","last_undegraded":"2023-06-10T19:00:10.993635+0000","last_fullsized":"2023-06-10T19:00:10.993635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:08:38.523735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33968,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337872,"statfs":{"total":96636764160,"available":96601980928,"internally_reserved":0,"allocated":2990080,"data_stored":1774377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.73199999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.80500000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.874},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2375680,"data_stored":1171425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.37},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39500000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1170899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.32600000000000001},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.753},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.55600000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34200000000000003},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.52400000000000002,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.245},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.71899999999999997},"last":0.34899999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81200000000000006,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.622},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.501}]},{"osd":5,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":0.85999999999999999,"15min":1.008},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":1.206,"15min":1.206},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.79500000000000004,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.224,"15min":1.224},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.97099999999999997},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":2957312,"data_stored":1754504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.56499999999999995,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34799999999999998,"15min":0.34899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":0.55700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.221}]},{"osd":2,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":11.439,"15min":11.439},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.316}]},{"osd":3,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.376},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.65000000000000002,"5min":5.766,"15min":5.766},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.25900000000000001},"max":{"1min":0.60499999999999998,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":2.802,"15min":2.802},"last":0.5}]},{"osd":5,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":0.999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.91500000000000004,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":2.907,"15min":2.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.40699999999999997,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.363},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45800000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.45300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33356,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338484,"statfs":{"total":96636764160,"available":96602607616,"internally_reserved":0,"allocated":2363392,"data_stored":1165065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.315},{"interface":"front","average":{"1min":0.376,"5min":0.35599999999999998,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55800000000000005,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.847},"last":0.373},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.40200000000000002},"min":{"1min":0.249,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.622,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.69599999999999995,"5min":1.24,"15min":2.8999999999999999},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.208,"15min":5.742},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.623,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":6.593},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":7.0880000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58199999999999996,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.83999999999999997,"5min":1.026,"15min":8.7850000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.171,"15min":5.6180000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64700000000000002,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.87,"15min":6.617},"last":0.497},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.38,"15min":0.37},"max":{"1min":0.879,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.63}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1172044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.224},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.318}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.441},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.499}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1176297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.436},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29299999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.61399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71499999999999997,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.3169999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.1459999999999999},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1172444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.379,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.42099999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.749,"15min":0.78400000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.69399999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.85099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65100000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.442}]},{"osd":3,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.628,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.34699999999999998},"max":{"1min":0.755,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.495}]},{"osd":4,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.71799999999999997,"5min":0.747,"15min":0.83999999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53100000000000003,"15min":0.51800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69799999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.72699999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.879},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.55100000000000005},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T19:00:15.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T19:00:15.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.304+0000 7f3d2410f700 1 Processor -- start 2023-06-10T19:00:15.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.305+0000 7f3d2410f700 1 -- start start 2023-06-10T19:00:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.305+0000 7f3d2410f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d1c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.305+0000 7f3d2410f700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0ccca0 con 0x7f3d1c0cad90 2023-06-10T19:00:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d2410f700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0cee50 con 0x7f3d1c15a250 2023-06-10T19:00:15.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d2410f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0cf130 con 0x7f3d1c0cfa00 2023-06-10T19:00:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d226ac700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3d1c0cad90 0x7f3d1c0b3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40324/0 (socket says 172.21.15.104:40324) 2023-06-10T19:00:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d226ac700 1 -- 172.21.15.104:0/521037878 learned_addr learned my addr 172.21.15.104:0/521037878 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640394971 0 0) 0x7f3d1c0ccca0 con 0x7f3d1c0cad90 2023-06-10T19:00:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d21eab700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d1c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d080035b0 con 0x7f3d1c0cad90 2023-06-10T19:00:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61142051 0 0) 0x7f3d080035b0 con 0x7f3d1c0cad90 2023-06-10T19:00:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 msgr2=0x7f3d1c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d1c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 >> v1:172.21.15.114:6789/0 conn(0x7f3d1c15a250 legacy=0x7f3d1c0cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0cc930 con 0x7f3d1c0cad90 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3d140019f0 con 0x7f3d1c0cad90 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 >> v1:172.21.15.104:6789/0 conn(0x7f3d1c0cad90 legacy=0x7f3d1c0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d08003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d08003f70 0x7f3d08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d08004d90 0x7f3d08005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0cf130 con 0x7f3d08004d90 2023-06-10T19:00:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0800baf0 con 0x7f3d08003f70 2023-06-10T19:00:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0800bcb0 con 0x7f3d1c0cfa00 2023-06-10T19:00:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d226ac700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d08004d90 0x7f3d08005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d21eab700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d08003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.306+0000 7f3d216aa700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d08003f70 0x7f3d08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 -- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 msgr2=0x7f3d08003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0cfa00 0x7f3d08003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 -- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d08003f70 msgr2=0x7f3d08004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:15.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d08003f70 0x7f3d08004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 -- 172.21.15.104:0/521037878 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080035b0 con 0x7f3d08004d90 2023-06-10T19:00:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d08004d90 0x7f3d08005200 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f3d14002230 tx=0x7f3d08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfa2bc2eeb6323d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14012600 con 0x7f3d08004d90 2023-06-10T19:00:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d14012780 con 0x7f3d08004d90 2023-06-10T19:00:15.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d20ea9700 1 -- 172.21.15.104:0/521037878 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14012b30 con 0x7f3d08004d90 2023-06-10T19:00:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d2410f700 1 -- 172.21.15.104:0/521037878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d08004d90 msgr2=0x7f3d08005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d2410f700 1 --2- 172.21.15.104:0/521037878 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d08004d90 0x7f3d08005200 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f3d14002230 tx=0x7f3d08005880 comp rx=0 tx=0).stop 2023-06-10T19:00:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d226ac700 1 -- 172.21.15.104:0/521037878 reap_dead start 2023-06-10T19:00:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d2410f700 1 -- 172.21.15.104:0/521037878 shutdown_connections 2023-06-10T19:00:15.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.307+0000 7f3d2410f700 1 -- 172.21.15.104:0/521037878 >> 172.21.15.104:0/521037878 conn(0x7f3d1c06e370 msgr2=0x7f3d1c06e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.308+0000 7f3d2410f700 1 -- 172.21.15.104:0/521037878 shutdown_connections 2023-06-10T19:00:15.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.308+0000 7f3d2410f700 1 -- 172.21.15.104:0/521037878 wait complete. 2023-06-10T19:00:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 Processor -- start 2023-06-10T19:00:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 -- start start 2023-06-10T19:00:15.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d1c0b0090 0x7f3d1c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0b23c0 0x7f3d1c1ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0b2860 con 0x7f3d1c155ca0 2023-06-10T19:00:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0c60e0 con 0x7f3d1c0b0090 2023-06-10T19:00:15.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.309+0000 7f3d2410f700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0b0870 con 0x7f3d1c0b23c0 2023-06-10T19:00:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42504/0 (socket says 172.21.15.104:42504) 2023-06-10T19:00:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 -- 172.21.15.104:0/1989431204 learned_addr learned my addr 172.21.15.104:0/1989431204 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d216aa700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d1c0b0090 0x7f3d1c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.326 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d226ac700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0b23c0 0x7f3d1c1ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 -- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0b23c0 msgr2=0x7f3d1c1ef3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0b23c0 0x7f3d1c1ef3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 -- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d1c0b0090 msgr2=0x7f3d1c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d1c0b0090 0x7f3d1c0b1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 -- 172.21.15.104:0/1989431204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0b0b50 con 0x7f3d1c155ca0 2023-06-10T19:00:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d21eab700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c011aa0 tx=0x7f3d0c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c011ee0 con 0x7f3d1c155ca0 2023-06-10T19:00:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c00f940 con 0x7f3d1c155ca0 2023-06-10T19:00:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c00fcf0 con 0x7f3d1c155ca0 2023-06-10T19:00:15.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c0b0e30 con 0x7f3d1c155ca0 2023-06-10T19:00:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.310+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c0b13f0 con 0x7f3d1c155ca0 2023-06-10T19:00:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.311+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c0b1760 con 0x7f3d1c155ca0 2023-06-10T19:00:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.312+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3d0c014440 con 0x7f3d1c155ca0 2023-06-10T19:00:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.312+0000 7f3d127fc700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 0x7f3d0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.312+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3d0c0898f0 con 0x7f3d1c155ca0 2023-06-10T19:00:15.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.312+0000 7f3d216aa700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 0x7f3d0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.313+0000 7f3d216aa700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 0x7f3d0006d190 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b3240 tx=0x7f3d18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.316+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d0c058710 con 0x7f3d1c155ca0 2023-06-10T19:00:15.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.499+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d1c0b3ca0 con 0x7f3d0006ace0 2023-06-10T19:00:15.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.503+0000 7f3d127fc700 1 -- 172.21.15.104:0/1989431204 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72687 (secure 0 0 0) 0x7f3d180127b0 con 0x7f3d0006ace0 2023-06-10T19:00:15.510 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:15.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 msgr2=0x7f3d0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.512 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 0x7f3d0006d190 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b3240 tx=0x7f3d18009040 comp rx=0 tx=0).stop 2023-06-10T19:00:15.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 msgr2=0x7f3d1c1f34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.513 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c011aa0 tx=0x7f3d0c011db0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 shutdown_connections 2023-06-10T19:00:15.514 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d1c0b23c0 0x7f3d1c1ef3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d1c155ca0 0x7f3d1c1f34f0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3d0006ace0 0x7f3d0006d190 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.515 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 --2- 172.21.15.104:0/1989431204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d1c0b0090 0x7f3d1c0b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.504+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 >> 172.21.15.104:0/1989431204 conn(0x7f3d1c06e370 msgr2=0x7f3d1c1590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.505+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 shutdown_connections 2023-06-10T19:00:15.516 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.505+0000 7f3d2410f700 1 -- 172.21.15.104:0/1989431204 wait complete. 2023-06-10T19:00:15.516 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T19:00:15.523 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2023-06-10T19:00:13.606564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268132,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706588,"statfs":{"total":773094113280,"available":772819546112,"internally_reserved":0,"allocated":20221952,"data_stored":10557055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12945,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001679"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729349+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T19:00:09.729349+0000","last_peered":"2023-06-10T19:00:09.729349+0000","last_clean":"2023-06-10T19:00:09.729349+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T19:00:09.729349+0000","last_undegraded":"2023-06-10T19:00:09.729349+0000","last_fullsized":"2023-06-10T19:00:09.729349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:41.936611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731032+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T19:00:09.731032+0000","last_peered":"2023-06-10T19:00:09.731032+0000","last_clean":"2023-06-10T19:00:09.731032+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T19:00:09.731032+0000","last_undegraded":"2023-06-10T19:00:09.731032+0000","last_fullsized":"2023-06-10T19:00:09.731032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:07:44.579162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.905359+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T19:00:10.905359+0000","last_peered":"2023-06-10T19:00:10.905359+0000","last_clean":"2023-06-10T19:00:10.905359+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T19:00:10.905359+0000","last_undegraded":"2023-06-10T19:00:10.905359+0000","last_fullsized":"2023-06-10T19:00:10.905359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:47.908360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731009+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T19:00:09.731009+0000","last_peered":"2023-06-10T19:00:09.731009+0000","last_clean":"2023-06-10T19:00:09.731009+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T19:00:09.731009+0000","last_undegraded":"2023-06-10T19:00:09.731009+0000","last_fullsized":"2023-06-10T19:00:09.731009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:43:46.398137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":475,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729314+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T19:00:09.729314+0000","last_peered":"2023-06-10T19:00:09.729314+0000","last_clean":"2023-06-10T19:00:09.729314+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T19:00:09.729314+0000","last_undegraded":"2023-06-10T19:00:09.729314+0000","last_fullsized":"2023-06-10T19:00:09.729314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:49.946528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":319,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729170+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T19:00:09.729170+0000","last_peered":"2023-06-10T19:00:09.729170+0000","last_clean":"2023-06-10T19:00:09.729170+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T19:00:09.729170+0000","last_undegraded":"2023-06-10T19:00:09.729170+0000","last_fullsized":"2023-06-10T19:00:09.729170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:38:00.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729544+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T19:00:09.729544+0000","last_peered":"2023-06-10T19:00:09.729544+0000","last_clean":"2023-06-10T19:00:09.729544+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T19:00:09.729544+0000","last_undegraded":"2023-06-10T19:00:09.729544+0000","last_fullsized":"2023-06-10T19:00:09.729544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:59.472915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729631+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T19:00:09.729631+0000","last_peered":"2023-06-10T19:00:09.729631+0000","last_clean":"2023-06-10T19:00:09.729631+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T19:00:09.729631+0000","last_undegraded":"2023-06-10T19:00:09.729631+0000","last_fullsized":"2023-06-10T19:00:09.729631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:02:23.912395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.993635+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T19:00:10.993635+0000","last_peered":"2023-06-10T19:00:10.993635+0000","last_clean":"2023-06-10T19:00:10.993635+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T19:00:10.993635+0000","last_undegraded":"2023-06-10T19:00:10.993635+0000","last_fullsized":"2023-06-10T19:00:10.993635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:08:38.523735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33968,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337872,"statfs":{"total":96636764160,"available":96601980928,"internally_reserved":0,"allocated":2990080,"data_stored":1774377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.73199999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.80500000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.874},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2375680,"data_stored":1171425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.37},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39500000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1170899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.32600000000000001},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.753},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.55600000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34200000000000003},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.52400000000000002,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.245},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.71899999999999997},"last":0.34899999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81200000000000006,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.622},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.501}]},{"osd":5,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":0.85999999999999999,"15min":1.008},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":1.206,"15min":1.206},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.79500000000000004,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.224,"15min":1.224},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.97099999999999997},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":2957312,"data_stored":1754504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.56499999999999995,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34799999999999998,"15min":0.34899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":0.55700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.221}]},{"osd":2,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":11.439,"15min":11.439},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.316}]},{"osd":3,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.376},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.65000000000000002,"5min":5.766,"15min":5.766},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.25900000000000001},"max":{"1min":0.60499999999999998,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":2.802,"15min":2.802},"last":0.5}]},{"osd":5,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":0.999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.91500000000000004,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":2.907,"15min":2.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.40699999999999997,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.363},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45800000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.45300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33356,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338484,"statfs":{"total":96636764160,"available":96602607616,"internally_reserved":0,"allocated":2363392,"data_stored":1165065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.315},{"interface":"front","average":{"1min":0.376,"5min":0.35599999999999998,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55800000000000005,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.847},"last":0.373},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.40200000000000002},"min":{"1min":0.249,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.622,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.69599999999999995,"5min":1.24,"15min":2.8999999999999999},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.208,"15min":5.742},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.623,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":6.593},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":7.0880000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58199999999999996,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.83999999999999997,"5min":1.026,"15min":8.7850000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.171,"15min":5.6180000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64700000000000002,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.87,"15min":6.617},"last":0.497},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.38,"15min":0.37},"max":{"1min":0.879,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.63}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1172044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.224},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.318}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.441},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.499}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1176297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.436},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29299999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.61399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71499999999999997,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.3169999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.1459999999999999},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1172444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.379,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.42099999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.749,"15min":0.78400000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.69399999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.85099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65100000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.442}]},{"osd":3,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.628,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.34699999999999998},"max":{"1min":0.755,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.495}]},{"osd":4,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.71799999999999997,"5min":0.747,"15min":0.83999999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53100000000000003,"15min":0.51800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69799999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.72699999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.879},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.55100000000000005},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T19:00:15.525 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-06-10T19:00:15.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T19:00:15.645 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.639+0000 7f8ffb72a700 1 Processor -- start 2023-06-10T19:00:15.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.639+0000 7f8ffb72a700 1 -- start start 2023-06-10T19:00:15.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ffb72a700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ff40cd850 0x7f8ff406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ffb72a700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40c9e60 con 0x7f8ff40cb300 2023-06-10T19:00:15.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ffb72a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40aee70 con 0x7f8ff40aeaa0 2023-06-10T19:00:15.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ffb72a700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40c9bc0 con 0x7f8ff40cd850 2023-06-10T19:00:15.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff8cc5700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f8ff40cd850 0x7f8ff406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff9cc7700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ff40cb300 0x7f8ff406d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40344/0 (socket says 172.21.15.104:40344) 2023-06-10T19:00:15.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff9cc7700 1 -- 172.21.15.104:0/3793881711 learned_addr learned my addr 172.21.15.104:0/3793881711 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:15.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/3793881711 >> v1:172.21.15.114:6789/0 conn(0x7f8ff40aeaa0 legacy=0x7f8ff40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/3793881711 >> v1:172.21.15.104:6789/0 conn(0x7f8ff40cb300 legacy=0x7f8ff406d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8febfff700 1 -- 172.21.15.104:0/3793881711 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893343320 0 0) 0x7f8ff40c9e60 con 0x7f8ff40cb300 2023-06-10T19:00:15.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/3793881711 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff406e540 con 0x7f8ff40cd850 2023-06-10T19:00:15.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8ff8cc5700 1 --2- 172.21.15.104:0/3793881711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cd850 0x7f8ff406cd50 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009840 tx=0x7f8fe4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c065fc9bce0b87 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.650 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8febfff700 1 -- 172.21.15.104:0/3793881711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4011830 con 0x7f8ff40cd850 2023-06-10T19:00:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8febfff700 1 -- 172.21.15.104:0/3793881711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4011990 con 0x7f8ff40cd850 2023-06-10T19:00:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.640+0000 7f8febfff700 1 -- 172.21.15.104:0/3793881711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4011d20 con 0x7f8ff40cd850 2023-06-10T19:00:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 -- 172.21.15.104:0/3793881711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cd850 msgr2=0x7f8ff406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/3793881711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cd850 0x7f8ff406cd50 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009840 tx=0x7f8fe4009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 -- 172.21.15.104:0/3793881711 shutdown_connections 2023-06-10T19:00:15.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/3793881711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cd850 0x7f8ff406cd50 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 -- 172.21.15.104:0/3793881711 >> 172.21.15.104:0/3793881711 conn(0x7f8ff406b8a0 msgr2=0x7f8ff40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 -- 172.21.15.104:0/3793881711 shutdown_connections 2023-06-10T19:00:15.652 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.641+0000 7f8ffb72a700 1 -- 172.21.15.104:0/3793881711 wait complete. 2023-06-10T19:00:15.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.642+0000 7f8ffb72a700 1 Processor -- start 2023-06-10T19:00:15.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.642+0000 7f8ffb72a700 1 -- start start 2023-06-10T19:00:15.655 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 0x7f8ff4150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 0x7f8ff40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4155b00 con 0x7f8ff40cb300 2023-06-10T19:00:15.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4154ef0 con 0x7f8ff40cd850 2023-06-10T19:00:15.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ffb72a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41551d0 con 0x7f8ff40aeaa0 2023-06-10T19:00:15.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42520/0 (socket says 172.21.15.104:42520) 2023-06-10T19:00:15.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/2740015422 learned_addr learned my addr 172.21.15.104:0/2740015422 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:15.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff9cc7700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 0x7f8ff40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 msgr2=0x7f8ff4150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff94c6700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 0x7f8ff4150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 0x7f8ff4150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.658 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 msgr2=0x7f8ff40b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 0x7f8ff40b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 -- 172.21.15.104:0/2740015422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40094a0 con 0x7f8ff40cb300 2023-06-10T19:00:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff9cc7700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 0x7f8ff40b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:15.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff94c6700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 0x7f8ff4150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.643+0000 7f8ff8cc5700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4014040 tx=0x7f8fe4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.644+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe401a900 con 0x7f8ff40cb300 2023-06-10T19:00:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.644+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe401aa60 con 0x7f8ff40cb300 2023-06-10T19:00:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.644+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff41554b0 con 0x7f8ff40cb300 2023-06-10T19:00:15.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.644+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe401add0 con 0x7f8ff40cb300 2023-06-10T19:00:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.644+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff4150f60 con 0x7f8ff40cb300 2023-06-10T19:00:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.645+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4155cb0 con 0x7f8ff40cb300 2023-06-10T19:00:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.645+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8fe4022070 con 0x7f8ff40cb300 2023-06-10T19:00:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.646+0000 7f8fe9ffb700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 0x7f8fd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.646+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8fe408a540 con 0x7f8ff40cb300 2023-06-10T19:00:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.646+0000 7f8ff94c6700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 0x7f8fd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.646+0000 7f8ff94c6700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 0x7f8fd806d190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0000fb0 tx=0x7f8ff0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.650+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ff4155cb0 con 0x7f8ff40cb300 2023-06-10T19:00:15.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.832+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ff406eb00 con 0x7f8fd806ace0 2023-06-10T19:00:15.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.836+0000 7f8fe9ffb700 1 -- 172.21.15.104:0/2740015422 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72687 (secure 0 0 0) 0x7f8ff0001d00 con 0x7f8fd806ace0 2023-06-10T19:00:15.844 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:15.845 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.837+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 msgr2=0x7f8fd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.837+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 0x7f8fd806d190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0000fb0 tx=0x7f8ff0009040 comp rx=0 tx=0).stop 2023-06-10T19:00:15.846 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.837+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 msgr2=0x7f8ff4155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.837+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4014040 tx=0x7f8fe4004750 comp rx=0 tx=0).stop 2023-06-10T19:00:15.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 shutdown_connections 2023-06-10T19:00:15.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f8fd806ace0 0x7f8fd806d190 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff40cd850 0x7f8ff40b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ff40aeaa0 0x7f8ff4150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 --2- 172.21.15.104:0/2740015422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ff40cb300 0x7f8ff4155f90 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 >> 172.21.15.104:0/2740015422 conn(0x7f8ff406b8a0 msgr2=0x7f8ff406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 shutdown_connections 2023-06-10T19:00:15.848 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.838+0000 7f8ffb72a700 1 -- 172.21.15.104:0/2740015422 wait complete. 2023-06-10T19:00:15.849 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T19:00:15.856 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2023-06-10T19:00:15.606951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268132,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706588,"statfs":{"total":773094113280,"available":772819546112,"internally_reserved":0,"allocated":20221952,"data_stored":10557055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12945,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001412"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729349+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T19:00:09.729349+0000","last_peered":"2023-06-10T19:00:09.729349+0000","last_clean":"2023-06-10T19:00:09.729349+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T19:00:09.729349+0000","last_undegraded":"2023-06-10T19:00:09.729349+0000","last_fullsized":"2023-06-10T19:00:09.729349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:25:41.936611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731032+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T19:00:09.731032+0000","last_peered":"2023-06-10T19:00:09.731032+0000","last_clean":"2023-06-10T19:00:09.731032+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T19:00:09.731032+0000","last_undegraded":"2023-06-10T19:00:09.731032+0000","last_fullsized":"2023-06-10T19:00:09.731032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:07:44.579162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.905359+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T19:00:10.905359+0000","last_peered":"2023-06-10T19:00:10.905359+0000","last_clean":"2023-06-10T19:00:10.905359+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T19:00:10.905359+0000","last_undegraded":"2023-06-10T19:00:10.905359+0000","last_fullsized":"2023-06-10T19:00:10.905359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:16:47.908360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":348,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.731009+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T19:00:09.731009+0000","last_peered":"2023-06-10T19:00:09.731009+0000","last_clean":"2023-06-10T19:00:09.731009+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T19:00:09.731009+0000","last_undegraded":"2023-06-10T19:00:09.731009+0000","last_fullsized":"2023-06-10T19:00:09.731009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T00:43:46.398137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":475,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729314+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T19:00:09.729314+0000","last_peered":"2023-06-10T19:00:09.729314+0000","last_clean":"2023-06-10T19:00:09.729314+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T19:00:09.729314+0000","last_undegraded":"2023-06-10T19:00:09.729314+0000","last_fullsized":"2023-06-10T19:00:09.729314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:41:49.946528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":319,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729170+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T19:00:09.729170+0000","last_peered":"2023-06-10T19:00:09.729170+0000","last_clean":"2023-06-10T19:00:09.729170+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T19:00:09.729170+0000","last_undegraded":"2023-06-10T19:00:09.729170+0000","last_fullsized":"2023-06-10T19:00:09.729170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T23:38:00.998976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729544+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T19:00:09.729544+0000","last_peered":"2023-06-10T19:00:09.729544+0000","last_clean":"2023-06-10T19:00:09.729544+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T19:00:09.729544+0000","last_undegraded":"2023-06-10T19:00:09.729544+0000","last_fullsized":"2023-06-10T19:00:09.729544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:01:59.472915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:09.729631+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T19:00:09.729631+0000","last_peered":"2023-06-10T19:00:09.729631+0000","last_clean":"2023-06-10T19:00:09.729631+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T19:00:09.729631+0000","last_undegraded":"2023-06-10T19:00:09.729631+0000","last_fullsized":"2023-06-10T19:00:09.729631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:02:23.912395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-06-10T19:00:10.993635+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T19:00:10.993635+0000","last_peered":"2023-06-10T19:00:10.993635+0000","last_clean":"2023-06-10T19:00:10.993635+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T19:00:10.993635+0000","last_undegraded":"2023-06-10T19:00:10.993635+0000","last_fullsized":"2023-06-10T19:00:10.993635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:08:38.523735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33968,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337872,"statfs":{"total":96636764160,"available":96601980928,"internally_reserved":0,"allocated":2990080,"data_stored":1774377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.73199999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.80500000000000005}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.874},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2320,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2375680,"data_stored":1171425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.37},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46700000000000003}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39500000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.496},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1170899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.32600000000000001},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.753},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35299999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.55600000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34200000000000003},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.52400000000000002,"5min":0.58899999999999997,"15min":0.64500000000000002},"last":0.245},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.71899999999999997},"last":0.34899999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57599999999999996,"5min":0.61099999999999999,"15min":0.66600000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.81200000000000006,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.622},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83799999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.501}]},{"osd":5,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.85099999999999998,"15min":0.93000000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":0.85999999999999999,"15min":1.008},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":1.206,"15min":1.206},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.79500000000000004,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Jun 10 18:59:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88400000000000001,"5min":1.224,"15min":1.224},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.97099999999999997},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33936,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337904,"statfs":{"total":96636764160,"available":96602013696,"internally_reserved":0,"allocated":2957312,"data_stored":1754504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.56499999999999995,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34799999999999998,"15min":0.34899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":0.55700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.221}]},{"osd":2,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":11.439,"15min":11.439},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.316}]},{"osd":3,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.376},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.65000000000000002,"5min":5.766,"15min":5.766},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.25900000000000001},"max":{"1min":0.60499999999999998,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":2.802,"15min":2.802},"last":0.5}]},{"osd":5,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":0.999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.91500000000000004,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":2.907,"15min":2.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.40699999999999997,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.363},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45800000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.45300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33356,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338484,"statfs":{"total":96636764160,"available":96602607616,"internally_reserved":0,"allocated":2363392,"data_stored":1165065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.315},{"interface":"front","average":{"1min":0.376,"5min":0.35599999999999998,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55800000000000005,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.847},"last":0.373},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.40200000000000002},"min":{"1min":0.249,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.622,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.69599999999999995,"5min":1.24,"15min":2.8999999999999999},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.208,"15min":5.742},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.623,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":6.593},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":7.0880000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58199999999999996,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.83999999999999997,"5min":1.026,"15min":8.7850000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.171,"15min":5.6180000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64700000000000002,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.87,"15min":6.617},"last":0.497},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.38,"15min":0.37},"max":{"1min":0.879,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.63}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1172044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.224},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.318}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.441},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.499}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33380,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338460,"statfs":{"total":96636764160,"available":96602583040,"internally_reserved":0,"allocated":2387968,"data_stored":1176297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.436},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29299999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.61399999999999999}]},{"osd":3,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71499999999999997,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.3169999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.1459999999999999},"last":0.57099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33372,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338468,"statfs":{"total":96636764160,"available":96602591232,"internally_reserved":0,"allocated":2379776,"data_stored":1172444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.379,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.42099999999999999}]},{"osd":1,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.749,"15min":0.78400000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.69399999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.85099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65100000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.442}]},{"osd":3,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.628,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.34699999999999998},"max":{"1min":0.755,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.495}]},{"osd":4,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.71799999999999997,"5min":0.747,"15min":0.83999999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53100000000000003,"15min":0.51800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69799999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.72699999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.879},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.55100000000000005},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T19:00:15.858 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-10T19:00:15.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.972+0000 7f7a9f677700 1 Processor -- start 2023-06-10T19:00:15.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.972+0000 7f7a9f677700 1 -- start start 2023-06-10T19:00:15.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9f677700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9806af80 0x7f7a9806b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9f677700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980cdec0 con 0x7f7a980b1a20 2023-06-10T19:00:15.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9f677700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9806b890 con 0x7f7a980b15b0 2023-06-10T19:00:15.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9f677700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a980b1ea0 con 0x7f7a9806af80 2023-06-10T19:00:15.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9806af80 0x7f7a9806b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9dc14700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f7a980b1a20 0x7f7a980cd7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40358/0 (socket says 172.21.15.104:40358) 2023-06-10T19:00:15.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f7a9806af80 0x7f7a9806b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42526/0 (socket says 172.21.15.104:42526) 2023-06-10T19:00:15.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9dc14700 1 -- 172.21.15.104:0/3933823047 learned_addr learned my addr 172.21.15.104:0/3933823047 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:15.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 -- 172.21.15.104:0/3933823047 >> v1:172.21.15.114:6789/0 conn(0x7f7a980b15b0 legacy=0x7f7a980cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:15.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 -- 172.21.15.104:0/3933823047 >> v1:172.21.15.104:6789/0 conn(0x7f7a980b1a20 legacy=0x7f7a980cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a8ffff700 1 -- 172.21.15.104:0/3933823047 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478236864 0 0) 0x7f7a980cdec0 con 0x7f7a980b1a20 2023-06-10T19:00:15.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 -- 172.21.15.104:0/3933823047 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980cb880 con 0x7f7a9806af80 2023-06-10T19:00:15.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a9d413700 1 --2- 172.21.15.104:0/3933823047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a9806b350 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a94009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce779836e192277 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.973+0000 7f7a8ffff700 1 -- 172.21.15.104:0/3933823047 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004230 con 0x7f7a9806af80 2023-06-10T19:00:15.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a8ffff700 1 -- 172.21.15.104:0/3933823047 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a94004390 con 0x7f7a9806af80 2023-06-10T19:00:15.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a8ffff700 1 -- 172.21.15.104:0/3933823047 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940036a0 con 0x7f7a9806af80 2023-06-10T19:00:15.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 -- 172.21.15.104:0/3933823047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 msgr2=0x7f7a9806b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 --2- 172.21.15.104:0/3933823047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a9806b350 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a94009b80 comp rx=0 tx=0).stop 2023-06-10T19:00:15.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 -- 172.21.15.104:0/3933823047 shutdown_connections 2023-06-10T19:00:15.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 --2- 172.21.15.104:0/3933823047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a9806b350 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 -- 172.21.15.104:0/3933823047 >> 172.21.15.104:0/3933823047 conn(0x7f7a98089fd0 msgr2=0x7f7a980d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:15.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 -- 172.21.15.104:0/3933823047 shutdown_connections 2023-06-10T19:00:15.986 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.974+0000 7f7a9f677700 1 -- 172.21.15.104:0/3933823047 wait complete. 2023-06-10T19:00:15.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.976+0000 7f7a9f677700 1 Processor -- start 2023-06-10T19:00:15.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.976+0000 7f7a9f677700 1 -- start start 2023-06-10T19:00:15.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a980b15b0 0x7f7a9815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 0x7f7a9815b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981587c0 con 0x7f7a9806af80 2023-06-10T19:00:15.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a980b4510 con 0x7f7a980b15b0 2023-06-10T19:00:15.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981591c0 con 0x7f7a980b1a20 2023-06-10T19:00:15.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42532/0 (socket says 172.21.15.104:42532) 2023-06-10T19:00:15.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 -- 172.21.15.104:0/1283003558 learned_addr learned my addr 172.21.15.104:0/1283003558 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:15.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9cc12700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a980b15b0 0x7f7a9815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9dc14700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 0x7f7a9815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 -- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 msgr2=0x7f7a9815b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 0x7f7a9815b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 -- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a980b15b0 msgr2=0x7f7a9815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:15.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a980b15b0 0x7f7a9815ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:15.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 -- 172.21.15.104:0/1283003558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a940094a0 con 0x7f7a9806af80 2023-06-10T19:00:15.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9dc14700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 0x7f7a9815b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:15.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9d413700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009bc0 tx=0x7f7a940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94011830 con 0x7f7a9806af80 2023-06-10T19:00:15.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a94011990 con 0x7f7a9806af80 2023-06-10T19:00:15.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a980b4230 con 0x7f7a9806af80 2023-06-10T19:00:15.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94011d00 con 0x7f7a9806af80 2023-06-10T19:00:15.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.977+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a980b5970 con 0x7f7a9806af80 2023-06-10T19:00:15.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.978+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a980b54f0 con 0x7f7a9806af80 2023-06-10T19:00:15.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.979+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a9401b440 con 0x7f7a9806af80 2023-06-10T19:00:15.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.979+0000 7f7a8dffb700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:15.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.979+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7a9408ad20 con 0x7f7a9806af80 2023-06-10T19:00:15.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.979+0000 7f7a9cc12700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:15.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.980+0000 7f7a9cc12700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009870 tx=0x7f7a88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:15.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:15.983+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a980b54f0 con 0x7f7a9806af80 2023-06-10T19:00:16.177 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.171+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a980b54f0 con 0x7f7a9806af80 2023-06-10T19:00:16.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.172+0000 7f7a8dffb700 1 -- 172.21.15.104:0/1283003558 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+15093 (secure 0 0 0) 0x7f7a980b54f0 con 0x7f7a9806af80 2023-06-10T19:00:16.178 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:16.179 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":191,"fsid":"f08e2e30-012f-4260-8ced-2a652c595f11","created":"2023-06-10T18:43:43.726997+0000","modified":"2023-06-10T19:00:12.800224+0000","last_up_change":"2023-06-10T18:43:55.363797+0000","last_in_change":"2023-06-10T18:43:46.224581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-10T18:43:55.391812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-10T18:44:02.212624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"909ccd07-b679-4767-9aeb-dded1012650a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6825","nonce":1623254903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6827","nonce":1623254903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6831","nonce":1623254903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1623254903},{"type":"v1","addr":"172.21.15.104:6829","nonce":1623254903}]},"public_addr":"172.21.15.104:6825/1623254903","cluster_addr":"172.21.15.104:6827/1623254903","heartbeat_back_addr":"172.21.15.104:6831/1623254903","heartbeat_front_addr":"172.21.15.104:6829/1623254903","state":["exists","up"]},{"osd":1,"uuid":"32d5a91b-dc25-4bb1-97ec-855dc9e01601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6817","nonce":2177374922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6819","nonce":2177374922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6823","nonce":2177374922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2177374922},{"type":"v1","addr":"172.21.15.104:6821","nonce":2177374922}]},"public_addr":"172.21.15.104:6817/2177374922","cluster_addr":"172.21.15.104:6819/2177374922","heartbeat_back_addr":"172.21.15.104:6823/2177374922","heartbeat_front_addr":"172.21.15.104:6821/2177374922","state":["exists","up"]},{"osd":2,"uuid":"f7f5934b-8789-436d-b4d5-95dd9334b240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6801","nonce":128700238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6803","nonce":128700238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6807","nonce":128700238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":128700238},{"type":"v1","addr":"172.21.15.104:6805","nonce":128700238}]},"public_addr":"172.21.15.104:6801/128700238","cluster_addr":"172.21.15.104:6803/128700238","heartbeat_back_addr":"172.21.15.104:6807/128700238","heartbeat_front_addr":"172.21.15.104:6805/128700238","state":["exists","up"]},{"osd":3,"uuid":"45f300ab-4919-4d85-aca4-c1234f7bed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6809","nonce":3217431778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6811","nonce":3217431778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6815","nonce":3217431778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3217431778},{"type":"v1","addr":"172.21.15.104:6813","nonce":3217431778}]},"public_addr":"172.21.15.104:6809/3217431778","cluster_addr":"172.21.15.104:6811/3217431778","heartbeat_back_addr":"172.21.15.104:6815/3217431778","heartbeat_front_addr":"172.21.15.104:6813/3217431778","state":["exists","up"]},{"osd":4,"uuid":"5e64e93e-2f14-433f-b034-016069e85071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6801","nonce":3316008371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6803","nonce":3316008371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6807","nonce":3316008371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3316008371},{"type":"v1","addr":"172.21.15.114:6805","nonce":3316008371}]},"public_addr":"172.21.15.114:6801/3316008371","cluster_addr":"172.21.15.114:6803/3316008371","heartbeat_back_addr":"172.21.15.114:6807/3316008371","heartbeat_front_addr":"172.21.15.114:6805/3316008371","state":["exists","up"]},{"osd":5,"uuid":"5f09f9c5-d82b-4c8c-8fd7-1273dd788097","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6817","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6819","nonce":1721425129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6821","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6823","nonce":1721425129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6829","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6831","nonce":1721425129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6825","nonce":1721425129},{"type":"v1","addr":"172.21.15.114:6827","nonce":1721425129}]},"public_addr":"172.21.15.114:6819/1721425129","cluster_addr":"172.21.15.114:6823/1721425129","heartbeat_back_addr":"172.21.15.114:6831/1721425129","heartbeat_front_addr":"172.21.15.114:6827/1721425129","state":["exists","up"]},{"osd":6,"uuid":"940b7625-3cf0-4851-8942-24160a1cfd7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6818","nonce":4101853965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6822","nonce":4101853965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6830","nonce":4101853965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4101853965},{"type":"v1","addr":"172.21.15.114:6826","nonce":4101853965}]},"public_addr":"172.21.15.114:6818/4101853965","cluster_addr":"172.21.15.114:6822/4101853965","heartbeat_back_addr":"172.21.15.114:6830/4101853965","heartbeat_front_addr":"172.21.15.114:6826/4101853965","state":["exists","up"]},{"osd":7,"uuid":"aa52f88f-f755-4d94-9418-8c557c565c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6809","nonce":3766591503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6811","nonce":3766591503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6815","nonce":3766591503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3766591503},{"type":"v1","addr":"172.21.15.114:6813","nonce":3766591503}]},"public_addr":"172.21.15.114:6809/3766591503","cluster_addr":"172.21.15.114:6811/3766591503","heartbeat_back_addr":"172.21.15.114:6815/3766591503","heartbeat_front_addr":"172.21.15.114:6813/3766591503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.410331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.379629+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.381224+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.407079+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.414660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.491652+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.490652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-10T18:43:51.494545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6835/3645224862":"2023-06-11T19:00:09.725407+0000","172.21.15.104:6834/3645224862":"2023-06-11T19:00:09.725407+0000","192.168.0.1:0/675086267":"2023-06-10T19:59:40.323449+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-10T19:00:16.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.173+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 msgr2=0x7f7a7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:16.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.173+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009870 tx=0x7f7a88008040 comp rx=0 tx=0).stop 2023-06-10T19:00:16.180 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.173+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 msgr2=0x7f7a98158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:16.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.173+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009bc0 tx=0x7f7a940047c0 comp rx=0 tx=0).stop 2023-06-10T19:00:16.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 shutdown_connections 2023-06-10T19:00:16.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a980b1a20 0x7f7a9815b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:16.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a9806af80 0x7f7a98158280 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:16.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f7a7c06ace0 0x7f7a7c06d190 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:16.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 --2- 172.21.15.104:0/1283003558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a980b15b0 0x7f7a9815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:16.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 >> 172.21.15.104:0/1283003558 conn(0x7f7a98089fd0 msgr2=0x7f7a980cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:16.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 shutdown_connections 2023-06-10T19:00:16.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:16.174+0000 7f7a9f677700 1 -- 172.21.15.104:0/1283003558 wait complete. 2023-06-10T19:00:17.193 INFO:tasks.ceph:Scrubbing osd.0 2023-06-10T19:00:17.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:17.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.324+0000 7f0400984700 1 Processor -- start 2023-06-10T19:00:17.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.324+0000 7f0400984700 1 -- start start 2023-06-10T19:00:17.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f0400984700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f0400984700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03fc15b190 0x7f03fc070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f0400984700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc15bcc0 con 0x7f03fc15dad0 2023-06-10T19:00:17.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f0400984700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc071c90 con 0x7f03fc15b190 2023-06-10T19:00:17.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f0400984700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc157e60 con 0x7f03fc0ce510 2023-06-10T19:00:17.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03fa59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03fc15b190 0x7f03fc070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03fa59c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.104:33692/0 (socket says 172.21.15.104:33692) 2023-06-10T19:00:17.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03fc15b190 0x7f03fc070130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:58276/0 (socket says 172.21.15.104:58276) 2023-06-10T19:00:17.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03fa59c700 1 -- 172.21.15.104:0/3636963413 learned_addr learned my addr 172.21.15.104:0/3636963413 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 -- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 msgr2=0x7f03fc0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 --2- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 -- 172.21.15.104:0/3636963413 >> v1:172.21.15.114:6789/0 conn(0x7f03fc15dad0 legacy=0x7f03fc15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f9d9b700 1 -- 172.21.15.104:0/3636963413 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc0cee20 con 0x7f03fc15b190 2023-06-10T19:00:17.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.325+0000 7f03f959a700 1 -- 172.21.15.104:0/3636963413 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161784892 0 0) 0x7f03fc15bcc0 con 0x7f03fc15dad0 2023-06-10T19:00:17.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f03f9d9b700 1 --2- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15b190 0x7f03fc070130 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009840 tx=0x7f03ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a2505c6e0e6169 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f03f959a700 1 -- 172.21.15.104:0/3636963413 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec011830 con 0x7f03fc15b190 2023-06-10T19:00:17.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f03f959a700 1 -- 172.21.15.104:0/3636963413 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec011990 con 0x7f03fc15b190 2023-06-10T19:00:17.336 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f03f959a700 1 -- 172.21.15.104:0/3636963413 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec011d20 con 0x7f03fc15b190 2023-06-10T19:00:17.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 -- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15b190 msgr2=0x7f03fc070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 --2- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15b190 0x7f03fc070130 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009840 tx=0x7f03ec009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:17.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 -- 172.21.15.104:0/3636963413 shutdown_connections 2023-06-10T19:00:17.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 --2- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 --2- 172.21.15.104:0/3636963413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15b190 0x7f03fc070130 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.326+0000 7f0400984700 1 -- 172.21.15.104:0/3636963413 >> 172.21.15.104:0/3636963413 conn(0x7f03fc06aac0 msgr2=0x7f03fc0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.340 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.327+0000 7f0400984700 1 -- 172.21.15.104:0/3636963413 shutdown_connections 2023-06-10T19:00:17.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.327+0000 7f0400984700 1 -- 172.21.15.104:0/3636963413 wait complete. 2023-06-10T19:00:17.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.328+0000 7f0400984700 1 Processor -- start 2023-06-10T19:00:17.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.328+0000 7f0400984700 1 -- start start 2023-06-10T19:00:17.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 0x7f03fc149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15dad0 0x7f03fc152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0c6760 con 0x7f03fc15b190 2023-06-10T19:00:17.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc156ec0 con 0x7f03fc15dad0 2023-06-10T19:00:17.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0c6970 con 0x7f03fc0ce510 2023-06-10T19:00:17.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 0x7f03fc149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 0x7f03fc149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42548/0 (socket says 172.21.15.104:42548) 2023-06-10T19:00:17.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 -- 172.21.15.104:0/1686788037 learned_addr learned my addr 172.21.15.104:0/1686788037 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03fa59c700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03fad9d700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15dad0 0x7f03fc152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 -- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 msgr2=0x7f03fc0c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 -- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15dad0 msgr2=0x7f03fc152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03fc15dad0 0x7f03fc152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.344 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0094a0 con 0x7f03fc15b190 2023-06-10T19:00:17.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03fa59c700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03fc0ce510 0x7f03fc0c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:17.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03f9d9b700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 0x7f03fc149a40 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0090b0 tx=0x7f03ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0045d0 con 0x7f03fc15b190 2023-06-10T19:00:17.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec01e070 con 0x7f03fc15b190 2023-06-10T19:00:17.345 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc0c6c50 con 0x7f03fc15b190 2023-06-10T19:00:17.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0038e0 con 0x7f03fc15b190 2023-06-10T19:00:17.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.329+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc0b2e60 con 0x7f03fc15b190 2023-06-10T19:00:17.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.330+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03fc072160 con 0x7f03fc15b190 2023-06-10T19:00:17.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03ec003a40 con 0x7f03fc15b190 2023-06-10T19:00:17.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03dc06ace0 0x7f03dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f03ec08aa20 con 0x7f03fc15b190 2023-06-10T19:00:17.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7f03dc0715d0 0x7f03dc073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03dc0740d0 con 0x7f03dc0715d0 2023-06-10T19:00:17.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03ec08ad90 con 0x7f03fc15b190 2023-06-10T19:00:17.347 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.331+0000 7f03fa59c700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03dc06ace0 0x7f03dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.332+0000 7f03fad9d700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7f03dc0715d0 0x7f03dc073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.332+0000 7f03fad9d700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7f03dc0715d0 0x7f03dc073a00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.332+0000 7f03fa59c700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03dc06ace0 0x7f03dc06d190 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009b80 tx=0x7f03f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.332+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== osd.0 v2:172.21.15.104:6824/1623254903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f03dc0740d0 con 0x7f03dc0715d0 2023-06-10T19:00:17.352 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.346+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 --> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f03fc071e80 con 0x7f03dc0715d0 2023-06-10T19:00:17.353 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f03eaffd700 1 -- 172.21.15.104:0/1686788037 <== osd.0 v2:172.21.15.104:6824/1623254903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f03f4001b90 con 0x7f03dc0715d0 2023-06-10T19:00:17.353 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:17.354 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:17.354 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:17.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7f03dc0715d0 msgr2=0x7f03dc073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:6824/1623254903,v1:172.21.15.104:6825/1623254903] conn(0x7f03dc0715d0 0x7f03dc073a00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.355 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03dc06ace0 msgr2=0x7f03dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f03dc06ace0 0x7f03dc06d190 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009b80 tx=0x7f03f0008040 comp rx=0 tx=0).stop 2023-06-10T19:00:17.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 msgr2=0x7f03fc149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 --2- 172.21.15.104:0/1686788037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f03fc15b190 0x7f03fc149a40 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0090b0 tx=0x7f03ec004750 comp rx=0 tx=0).stop 2023-06-10T19:00:17.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f03fad9d700 1 -- 172.21.15.104:0/1686788037 reap_dead start 2023-06-10T19:00:17.356 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 shutdown_connections 2023-06-10T19:00:17.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 >> 172.21.15.104:0/1686788037 conn(0x7f03fc06aac0 msgr2=0x7f03fc0afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 shutdown_connections 2023-06-10T19:00:17.357 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.347+0000 7f0400984700 1 -- 172.21.15.104:0/1686788037 wait complete. 2023-06-10T19:00:17.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-06-10T19:00:17.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.479+0000 7f3cea35e700 1 Processor -- start 2023-06-10T19:00:17.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.479+0000 7f3cea35e700 1 -- start start 2023-06-10T19:00:17.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3cea35e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3ce40ac230 0x7f3ce4069e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3cea35e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ce40cb860 0x7f3ce406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3cea35e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce415a680 con 0x7f3ce415a2b0 2023-06-10T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3cea35e700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce40d0510 con 0x7f3ce40cb860 2023-06-10T19:00:17.488 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3cea35e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce40ac680 con 0x7f3ce40ac230 2023-06-10T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ce40cb860 0x7f3ce406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f3ce40cb860 0x7f3ce406d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42560/0 (socket says 172.21.15.104:42560) 2023-06-10T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 -- 172.21.15.104:0/3366832363 learned_addr learned my addr 172.21.15.104:0/3366832363 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce37fe700 1 --2- 172.21.15.104:0/3366832363 >> v2:172.21.15.114:3300/0 conn(0x7f3ce40ac230 0x7f3ce4069e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 -- 172.21.15.104:0/3366832363 >> v2:172.21.15.114:3300/0 conn(0x7f3ce40ac230 msgr2=0x7f3ce4069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 --2- 172.21.15.104:0/3366832363 >> v2:172.21.15.114:3300/0 conn(0x7f3ce40ac230 0x7f3ce4069e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 -- 172.21.15.104:0/3366832363 >> v1:172.21.15.114:6789/0 conn(0x7f3ce415a2b0 legacy=0x7f3ce40cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.480+0000 7f3ce88fb700 1 -- 172.21.15.104:0/3366832363 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce406a3b0 con 0x7f3ce40cb860 2023-06-10T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3ce88fb700 1 --2- 172.21.15.104:0/3366832363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce40cb860 0x7f3ce406d860 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40700a0 tx=0x7f3cdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a608708c5026960 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3ce2ffd700 1 -- 172.21.15.104:0/3366832363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00e070 con 0x7f3ce40cb860 2023-06-10T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3ce2ffd700 1 -- 172.21.15.104:0/3366832363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc003a20 con 0x7f3ce40cb860 2023-06-10T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3ce2ffd700 1 -- 172.21.15.104:0/3366832363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc003dd0 con 0x7f3ce40cb860 2023-06-10T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 -- 172.21.15.104:0/3366832363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce40cb860 msgr2=0x7f3ce406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 --2- 172.21.15.104:0/3366832363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce40cb860 0x7f3ce406d860 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40700a0 tx=0x7f3cdc00b330 comp rx=0 tx=0).stop 2023-06-10T19:00:17.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 -- 172.21.15.104:0/3366832363 shutdown_connections 2023-06-10T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 --2- 172.21.15.104:0/3366832363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce40cb860 0x7f3ce406d860 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 --2- 172.21.15.104:0/3366832363 >> v2:172.21.15.114:3300/0 conn(0x7f3ce40ac230 0x7f3ce4069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 -- 172.21.15.104:0/3366832363 >> 172.21.15.104:0/3366832363 conn(0x7f3ce406b8a0 msgr2=0x7f3ce40cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 -- 172.21.15.104:0/3366832363 shutdown_connections 2023-06-10T19:00:17.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.481+0000 7f3cea35e700 1 -- 172.21.15.104:0/3366832363 wait complete. 2023-06-10T19:00:17.494 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.483+0000 7f3cea35e700 1 Processor -- start 2023-06-10T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.483+0000 7f3cea35e700 1 -- start start 2023-06-10T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ce40ac230 0x7f3ce40b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ce40cb860 0x7f3ce40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40c4ec0 con 0x7f3ce415a2b0 2023-06-10T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40afce0 con 0x7f3ce40ac230 2023-06-10T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3cea35e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40aff80 con 0x7f3ce40cb860 2023-06-10T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42566/0 (socket says 172.21.15.104:42566) 2023-06-10T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 -- 172.21.15.104:0/4005544160 learned_addr learned my addr 172.21.15.104:0/4005544160 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce3fff700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ce40ac230 0x7f3ce40b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce37fe700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ce40cb860 0x7f3ce40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 -- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ce40cb860 msgr2=0x7f3ce40c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ce40cb860 0x7f3ce40c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 -- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ce40ac230 msgr2=0x7f3ce40b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ce40ac230 0x7f3ce40b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 -- 172.21.15.104:0/4005544160 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc009b30 con 0x7f3ce415a2b0 2023-06-10T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.484+0000 7f3ce88fb700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc015040 tx=0x7f3cdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc004440 con 0x7f3ce415a2b0 2023-06-10T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc0092e0 con 0x7f3ce415a2b0 2023-06-10T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0047e0 con 0x7f3ce415a2b0 2023-06-10T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce40c56e0 con 0x7f3ce415a2b0 2023-06-10T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce40c5400 con 0x7f3ce415a2b0 2023-06-10T19:00:17.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.485+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce4070ad0 con 0x7f3ce415a2b0 2023-06-10T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.486+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3cdc012430 con 0x7f3ce415a2b0 2023-06-10T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.486+0000 7f3ce0ff9700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 0x7f3cd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.486+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3cdc08a670 con 0x7f3ce415a2b0 2023-06-10T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.487+0000 7f3ce3fff700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 0x7f3cd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.487+0000 7f3ce3fff700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 0x7f3cd006d190 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006fd0 tx=0x7f3cd40090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.491+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3cdc055940 con 0x7f3ce415a2b0 2023-06-10T19:00:17.683 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.677+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ce406a8c0 con 0x7f3cd006ace0 2023-06-10T19:00:17.684 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.678+0000 7f3ce0ff9700 1 -- 172.21.15.104:0/4005544160 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3cd4009e20 con 0x7f3cd006ace0 2023-06-10T19:00:17.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 msgr2=0x7f3cd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.686 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 0x7f3cd006d190 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006fd0 tx=0x7f3cd40090e0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 msgr2=0x7f3ce40cbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc015040 tx=0x7f3cdc004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.687 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 shutdown_connections 2023-06-10T19:00:17.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cd006ace0 0x7f3cd006d190 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ce40ac230 0x7f3ce40b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.688 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ce415a2b0 0x7f3ce40cbef0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 --2- 172.21.15.104:0/4005544160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ce40cb860 0x7f3ce40c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.679+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 >> 172.21.15.104:0/4005544160 conn(0x7f3ce406b8a0 msgr2=0x7f3ce406e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.689 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.680+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 shutdown_connections 2023-06-10T19:00:17.690 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.680+0000 7f3cea35e700 1 -- 172.21.15.104:0/4005544160 wait complete. 2023-06-10T19:00:17.690 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 0 to deep-scrub 2023-06-10T19:00:17.698 INFO:tasks.ceph:Scrubbing osd.1 2023-06-10T19:00:17.698 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:17.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.815+0000 7f3a4404a700 1 Processor -- start 2023-06-10T19:00:17.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.815+0000 7f3a4404a700 1 -- start start 2023-06-10T19:00:17.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.816+0000 7f3a4404a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c0ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.816+0000 7f3a4404a700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a3c0cb420 0x7f3a3c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a4404a700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c15a700 con 0x7f3a3c0cb890 2023-06-10T19:00:17.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a3c0cb420 0x7f3a3c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a4404a700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c06d7b0 con 0x7f3a3c0cb420 2023-06-10T19:00:17.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a41de6700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c0ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a4404a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0cbe00 con 0x7f3a3c15a330 2023-06-10T19:00:17.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a3c0cb420 0x7f3a3c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:58306/0 (socket says 172.21.15.104:58306) 2023-06-10T19:00:17.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 -- 172.21.15.104:0/3324311263 learned_addr learned my addr 172.21.15.104:0/3324311263 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 -- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 msgr2=0x7f3a3c0ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 --2- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c0ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 -- 172.21.15.104:0/3324311263 >> v1:172.21.15.114:6789/0 conn(0x7f3a3c0cb890 legacy=0x7f3a3c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 -- 172.21.15.104:0/3324311263 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c06ce80 con 0x7f3a3c0cb420 2023-06-10T19:00:17.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.817+0000 7f3a415e5700 1 --2- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0b1070 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3a380098f0 tx=0x7f3a38009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c1b81b0d351c0b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a40de4700 1 -- 172.21.15.104:0/3324311263 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38011830 con 0x7f3a3c0cb420 2023-06-10T19:00:17.828 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a40de4700 1 -- 172.21.15.104:0/3324311263 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a38011990 con 0x7f3a3c0cb420 2023-06-10T19:00:17.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a40de4700 1 -- 172.21.15.104:0/3324311263 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38011d20 con 0x7f3a3c0cb420 2023-06-10T19:00:17.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 -- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 msgr2=0x7f3a3c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0b1070 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3a380098f0 tx=0x7f3a38009c00 comp rx=0 tx=0).stop 2023-06-10T19:00:17.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 -- 172.21.15.104:0/3324311263 shutdown_connections 2023-06-10T19:00:17.829 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c0ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3324311263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0b1070 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 -- 172.21.15.104:0/3324311263 >> 172.21.15.104:0/3324311263 conn(0x7f3a3c06b8a0 msgr2=0x7f3a3c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.820+0000 7f3a4404a700 1 -- 172.21.15.104:0/3324311263 shutdown_connections 2023-06-10T19:00:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.821+0000 7f3a4404a700 1 -- 172.21.15.104:0/3324311263 wait complete. 2023-06-10T19:00:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.822+0000 7f3a4404a700 1 Processor -- start 2023-06-10T19:00:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.822+0000 7f3a4404a700 1 -- start start 2023-06-10T19:00:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 0x7f3a3c0c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0c4d50 con 0x7f3a3c0cb890 2023-06-10T19:00:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0c3a70 con 0x7f3a3c0cb420 2023-06-10T19:00:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a4404a700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0c5f10 con 0x7f3a3c15a330 2023-06-10T19:00:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 0x7f3a3c0c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 0x7f3a3c0c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42578/0 (socket says 172.21.15.104:42578) 2023-06-10T19:00:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 -- 172.21.15.104:0/3053852553 learned_addr learned my addr 172.21.15.104:0/3053852553 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a41de6700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a425e7700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 -- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 msgr2=0x7f3a3c153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a3c15a330 0x7f3a3c153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 -- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 msgr2=0x7f3a3c0994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a3c0cb420 0x7f3a3c0994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.823+0000 7f3a415e5700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009550 con 0x7f3a3c0cb890 2023-06-10T19:00:17.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a415e5700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 0x7f3a3c0c59d0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009c40 tx=0x7f3a38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380044d0 con 0x7f3a3c0cb890 2023-06-10T19:00:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3801e070 con 0x7f3a3c0cb890 2023-06-10T19:00:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c153aa0 con 0x7f3a3c0cb890 2023-06-10T19:00:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380038a0 con 0x7f3a3c0cb890 2023-06-10T19:00:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.824+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c0c3f90 con 0x7f3a3c0cb890 2023-06-10T19:00:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.825+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a3c06d440 con 0x7f3a3c0cb890 2023-06-10T19:00:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.825+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3a38003a00 con 0x7f3a3c0cb890 2023-06-10T19:00:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.825+0000 7f3a327fc700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2006ace0 0x7f3a2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.825+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3a3808a200 con 0x7f3a3c0cb890 2023-06-10T19:00:17.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a327fc700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f3a200715d0 0x7f3a20073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a41de6700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2006ace0 0x7f3a2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a200740d0 con 0x7f3a200715d0 2023-06-10T19:00:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a3808c8c0 con 0x7f3a3c0cb890 2023-06-10T19:00:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a425e7700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f3a200715d0 0x7f3a20073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.839 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a425e7700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f3a200715d0 0x7f3a20073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.826+0000 7f3a41de6700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2006ace0 0x7f3a2006d190 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c1f3100 tx=0x7f3a2c0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.827+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== osd.1 v2:172.21.15.104:6816/2177374922 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3a200740d0 con 0x7f3a200715d0 2023-06-10T19:00:17.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 --> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3a3c06d160 con 0x7f3a200715d0 2023-06-10T19:00:17.847 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a327fc700 1 -- 172.21.15.104:0/3053852553 <== osd.1 v2:172.21.15.104:6816/2177374922 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3a34003a00 con 0x7f3a200715d0 2023-06-10T19:00:17.847 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:17.848 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:17.848 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f3a200715d0 msgr2=0x7f3a20073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:6816/2177374922,v1:172.21.15.104:6817/2177374922] conn(0x7f3a200715d0 0x7f3a20073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.849 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2006ace0 msgr2=0x7f3a2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3a2006ace0 0x7f3a2006d190 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c1f3100 tx=0x7f3a2c0091c0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 msgr2=0x7f3a3c0c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 --2- 172.21.15.104:0/3053852553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a3c0cb890 0x7f3a3c0c59d0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009c40 tx=0x7f3a38004750 comp rx=0 tx=0).stop 2023-06-10T19:00:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a425e7700 1 -- 172.21.15.104:0/3053852553 reap_dead start 2023-06-10T19:00:17.850 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 shutdown_connections 2023-06-10T19:00:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 >> 172.21.15.104:0/3053852553 conn(0x7f3a3c06b8a0 msgr2=0x7f3a3c158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 shutdown_connections 2023-06-10T19:00:17.851 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.841+0000 7f3a4404a700 1 -- 172.21.15.104:0/3053852553 wait complete. 2023-06-10T19:00:17.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-06-10T19:00:17.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.974+0000 7fe3afec8700 1 Processor -- start 2023-06-10T19:00:17.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.974+0000 7fe3afec8700 1 -- start start 2023-06-10T19:00:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3afec8700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe3a806e0e0 0x7fe3a806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3afec8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3afec8700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a806d290 con 0x7fe3a80ca100 2023-06-10T19:00:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3afec8700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a8071640 con 0x7fe3a806e0e0 2023-06-10T19:00:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3afec8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80aef00 con 0x7fe3a806e4b0 2023-06-10T19:00:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3adc64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe3a806e0e0 0x7fe3a806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3adc64700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe3a806e0e0 0x7fe3a806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42586/0 (socket says 172.21.15.104:42586) 2023-06-10T19:00:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.975+0000 7fe3adc64700 1 -- 172.21.15.104:0/1781928309 learned_addr learned my addr 172.21.15.104:0/1781928309 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3ad463700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3adc64700 1 -- 172.21.15.104:0/1781928309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 msgr2=0x7fe3a80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3adc64700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a80c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3adc64700 1 -- 172.21.15.104:0/1781928309 >> v1:172.21.15.114:6789/0 conn(0x7fe3a80ca100 legacy=0x7fe3a80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3adc64700 1 -- 172.21.15.104:0/1781928309 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a806d4c0 con 0x7fe3a806e0e0 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3adc64700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a806cd50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fe398006fd0 tx=0x7fe398009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb46c8db5b51da7 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3acc62700 1 -- 172.21.15.104:0/1781928309 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398005e30 con 0x7fe3a806e0e0 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3acc62700 1 -- 172.21.15.104:0/1781928309 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398006030 con 0x7fe3a806e0e0 2023-06-10T19:00:17.985 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3acc62700 1 -- 172.21.15.104:0/1781928309 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980063c0 con 0x7fe3a806e0e0 2023-06-10T19:00:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 -- 172.21.15.104:0/1781928309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 msgr2=0x7fe3a806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.988 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a806cd50 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fe398006fd0 tx=0x7fe398009bd0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 -- 172.21.15.104:0/1781928309 shutdown_connections 2023-06-10T19:00:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a806cd50 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 --2- 172.21.15.104:0/1781928309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.976+0000 7fe3afec8700 1 -- 172.21.15.104:0/1781928309 >> 172.21.15.104:0/1781928309 conn(0x7fe3a806b8a0 msgr2=0x7fe3a80cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:17.989 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.977+0000 7fe3afec8700 1 -- 172.21.15.104:0/1781928309 shutdown_connections 2023-06-10T19:00:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.977+0000 7fe3afec8700 1 -- 172.21.15.104:0/1781928309 wait complete. 2023-06-10T19:00:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 Processor -- start 2023-06-10T19:00:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 -- start start 2023-06-10T19:00:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.990 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 0x7fe3a80b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a814f790 con 0x7fe3a806e0e0 2023-06-10T19:00:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80b5150 con 0x7fe3a80ca100 2023-06-10T19:00:17.991 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.978+0000 7fe3afec8700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80b0e90 con 0x7fe3a806e4b0 2023-06-10T19:00:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42590/0 (socket says 172.21.15.104:42590) 2023-06-10T19:00:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 -- 172.21.15.104:0/584908235 learned_addr learned my addr 172.21.15.104:0/584908235 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3ad463700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.992 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3ae465700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 0x7fe3a80b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 -- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 msgr2=0x7fe3a814f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a814f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 -- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 msgr2=0x7fe3a80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 0x7fe3a80b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:17.993 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 -- 172.21.15.104:0/584908235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398009500 con 0x7fe3a806e0e0 2023-06-10T19:00:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3ad463700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a814f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3ae465700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 0x7fe3a80b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3adc64700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe398000c00 tx=0x7fe398004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980068b0 con 0x7fe3a806e0e0 2023-06-10T19:00:17.994 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398020070 con 0x7fe3a806e0e0 2023-06-10T19:00:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a80b1170 con 0x7fe3a806e0e0 2023-06-10T19:00:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398025670 con 0x7fe3a806e0e0 2023-06-10T19:00:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.979+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a814f990 con 0x7fe3a806e0e0 2023-06-10T19:00:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.980+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a80723b0 con 0x7fe3a806e0e0 2023-06-10T19:00:17.995 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.981+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe398022030 con 0x7fe3a806e0e0 2023-06-10T19:00:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.981+0000 7fe39e7fc700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 0x7fe38c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.981+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fe39808a890 con 0x7fe3a806e0e0 2023-06-10T19:00:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.981+0000 7fe3ad463700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 0x7fe38c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:17.996 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.981+0000 7fe3ad463700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 0x7fe38c06d190 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80af200 tx=0x7fe3a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:17.997 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:17.985+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe398055b60 con 0x7fe3a806e0e0 2023-06-10T19:00:18.178 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.172+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3a806da80 con 0x7fe38c06ace0 2023-06-10T19:00:18.179 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.173+0000 7fe39e7fc700 1 -- 172.21.15.104:0/584908235 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe3a4009680 con 0x7fe38c06ace0 2023-06-10T19:00:18.181 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 msgr2=0x7fe38c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 0x7fe38c06d190 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80af200 tx=0x7fe3a4008040 comp rx=0 tx=0).stop 2023-06-10T19:00:18.182 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 msgr2=0x7fe3a80b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe398000c00 tx=0x7fe398004750 comp rx=0 tx=0).stop 2023-06-10T19:00:18.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 shutdown_connections 2023-06-10T19:00:18.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe38c06ace0 0x7fe38c06d190 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.183 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3a80ca100 0x7fe3a80b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3a806e0e0 0x7fe3a80b4b70 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.184 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 --2- 172.21.15.104:0/584908235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3a806e4b0 0x7fe3a814f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 >> 172.21.15.104:0/584908235 conn(0x7fe3a806b8a0 msgr2=0x7fe3a80adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 shutdown_connections 2023-06-10T19:00:18.185 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.175+0000 7fe3afec8700 1 -- 172.21.15.104:0/584908235 wait complete. 2023-06-10T19:00:18.186 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 1 to deep-scrub 2023-06-10T19:00:18.193 INFO:tasks.ceph:Scrubbing osd.2 2023-06-10T19:00:18.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:18.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 Processor -- start 2023-06-10T19:00:18.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 -- start start 2023-06-10T19:00:18.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe20415c6f0 0x7fe2040b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040ae7e0 con 0x7fe20406ea70 2023-06-10T19:00:18.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20406df90 con 0x7fe2040aec00 2023-06-10T19:00:18.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.311+0000 7fe209c41700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20406e270 con 0x7fe20415c6f0 2023-06-10T19:00:18.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2037fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fe20415c6f0 0x7fe2040b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe202ffd700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe20406ea70 0x7fe2040ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40428/0 (socket says 172.21.15.104:40428) 2023-06-10T19:00:18.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe202ffd700 1 -- 172.21.15.104:0/4274650063 learned_addr learned my addr 172.21.15.104:0/4274650063 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:18.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2037fe700 1 -- 172.21.15.104:0/4274650063 >> v1:172.21.15.114:6789/0 conn(0x7fe2040aec00 legacy=0x7fe2040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2037fe700 1 -- 172.21.15.104:0/4274650063 >> v1:172.21.15.104:6789/0 conn(0x7fe20406ea70 legacy=0x7fe2040ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2027fc700 1 -- 172.21.15.104:0/4274650063 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488398108 0 0) 0x7fe2040ae7e0 con 0x7fe20406ea70 2023-06-10T19:00:18.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2037fe700 1 -- 172.21.15.104:0/4274650063 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20406e550 con 0x7fe20415c6f0 2023-06-10T19:00:18.321 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2037fe700 1 --2- 172.21.15.104:0/4274650063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe2040b1070 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009820 tx=0x7fe1ec009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429deb6d91206115 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2027fc700 1 -- 172.21.15.104:0/4274650063 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec005e30 con 0x7fe20415c6f0 2023-06-10T19:00:18.322 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2027fc700 1 -- 172.21.15.104:0/4274650063 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec004030 con 0x7fe20415c6f0 2023-06-10T19:00:18.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.312+0000 7fe2027fc700 1 -- 172.21.15.104:0/4274650063 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec0043c0 con 0x7fe20415c6f0 2023-06-10T19:00:18.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 -- 172.21.15.104:0/4274650063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 msgr2=0x7fe2040b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.323 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 --2- 172.21.15.104:0/4274650063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe2040b1070 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009820 tx=0x7fe1ec009b30 comp rx=0 tx=0).stop 2023-06-10T19:00:18.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 -- 172.21.15.104:0/4274650063 shutdown_connections 2023-06-10T19:00:18.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 --2- 172.21.15.104:0/4274650063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe2040b1070 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 -- 172.21.15.104:0/4274650063 >> 172.21.15.104:0/4274650063 conn(0x7fe20406b8a0 msgr2=0x7fe2040cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 -- 172.21.15.104:0/4274650063 shutdown_connections 2023-06-10T19:00:18.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.313+0000 7fe209c41700 1 -- 172.21.15.104:0/4274650063 wait complete. 2023-06-10T19:00:18.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.314+0000 7fe209c41700 1 Processor -- start 2023-06-10T19:00:18.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.314+0000 7fe209c41700 1 -- start start 2023-06-10T19:00:18.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20406ea70 0x7fe2041457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2040aec00 0x7fe204153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe20415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe204156170 con 0x7fe20415c6f0 2023-06-10T19:00:18.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe204155e90 con 0x7fe2040aec00 2023-06-10T19:00:18.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe209c41700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b2550 con 0x7fe20406ea70 2023-06-10T19:00:18.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe20415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe20415a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42608/0 (socket says 172.21.15.104:42608) 2023-06-10T19:00:18.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 -- 172.21.15.104:0/2365910220 learned_addr learned my addr 172.21.15.104:0/2365910220 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:18.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe202ffd700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2040aec00 0x7fe204153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe2037fe700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20406ea70 0x7fe2041457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 -- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20406ea70 msgr2=0x7fe2041457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.330 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe20406ea70 0x7fe2041457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 -- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2040aec00 msgr2=0x7fe204153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2040aec00 0x7fe204153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec009430 con 0x7fe20415c6f0 2023-06-10T19:00:18.331 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.315+0000 7fe203fff700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe20415a2d0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00be60 tx=0x7fe1fc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00cd90 con 0x7fe20415c6f0 2023-06-10T19:00:18.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc003a00 con 0x7fe20415c6f0 2023-06-10T19:00:18.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2040b2830 con 0x7fe20415c6f0 2023-06-10T19:00:18.332 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc003df0 con 0x7fe20415c6f0 2023-06-10T19:00:18.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2041539a0 con 0x7fe20415c6f0 2023-06-10T19:00:18.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.316+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2040acfe0 con 0x7fe20415c6f0 2023-06-10T19:00:18.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.317+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe1fc003f50 con 0x7fe20415c6f0 2023-06-10T19:00:18.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.317+0000 7fe1fbfff700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1e406ace0 0x7fe1e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.333 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.317+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fe1fc089d40 con 0x7fe20415c6f0 2023-06-10T19:00:18.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe1fbfff700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fe1e40715d0 0x7fe1e4073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1e40740d0 con 0x7fe1e40715d0 2023-06-10T19:00:18.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1fc08a190 con 0x7fe20415c6f0 2023-06-10T19:00:18.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe202ffd700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fe1e40715d0 0x7fe1e4073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe2037fe700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1e406ace0 0x7fe1e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe202ffd700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fe1e40715d0 0x7fe1e4073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.318+0000 7fe2037fe700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1e406ace0 0x7fe1e406d190 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.335 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.319+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== osd.2 v2:172.21.15.104:6800/128700238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe1e40740d0 con 0x7fe1e40715d0 2023-06-10T19:00:18.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 --> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe204153bc0 con 0x7fe1e40715d0 2023-06-10T19:00:18.339 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe1fbfff700 1 -- 172.21.15.104:0/2365910220 <== osd.2 v2:172.21.15.104:6800/128700238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe204153bc0 con 0x7fe1e40715d0 2023-06-10T19:00:18.340 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:18.340 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:18.340 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:18.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fe1e40715d0 msgr2=0x7fe1e4073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.341 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:6800/128700238,v1:172.21.15.104:6801/128700238] conn(0x7fe1e40715d0 0x7fe1e4073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1e406ace0 msgr2=0x7fe1e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fe1e406ace0 0x7fe1e406d190 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec008070 comp rx=0 tx=0).stop 2023-06-10T19:00:18.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 msgr2=0x7fe20415a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 --2- 172.21.15.104:0/2365910220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe20415c6f0 0x7fe20415a2d0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00be60 tx=0x7fe1fc00bf40 comp rx=0 tx=0).stop 2023-06-10T19:00:18.342 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe203fff700 1 -- 172.21.15.104:0/2365910220 reap_dead start 2023-06-10T19:00:18.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 shutdown_connections 2023-06-10T19:00:18.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 >> 172.21.15.104:0/2365910220 conn(0x7fe20406b8a0 msgr2=0x7fe2040cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 shutdown_connections 2023-06-10T19:00:18.343 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.333+0000 7fe209c41700 1 -- 172.21.15.104:0/2365910220 wait complete. 2023-06-10T19:00:18.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-06-10T19:00:18.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.465+0000 7efc9c5b2700 1 Processor -- start 2023-06-10T19:00:18.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.465+0000 7efc9c5b2700 1 -- start start 2023-06-10T19:00:18.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9c5b2700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efc940aeaa0 0x7efc940cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9c5b2700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efc940cd680 0x7efc9406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9c5b2700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940c9e60 con 0x7efc940b1080 2023-06-10T19:00:18.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9c5b2700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940c9bc0 con 0x7efc940aeaa0 2023-06-10T19:00:18.479 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9c5b2700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940cdaa0 con 0x7efc940cd680 2023-06-10T19:00:18.480 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc9a34e700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7efc940aeaa0 0x7efc940cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc99b4d700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efc940b1080 0x7efc940ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40442/0 (socket says 172.21.15.104:40442) 2023-06-10T19:00:18.481 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.466+0000 7efc99b4d700 1 -- 172.21.15.104:0/2809474446 learned_addr learned my addr 172.21.15.104:0/2809474446 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:18.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9ab4f700 1 --2- 172.21.15.104:0/2809474446 >> v2:172.21.15.114:3300/0 conn(0x7efc940cd680 0x7efc9406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.482 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9a34e700 1 -- 172.21.15.104:0/2809474446 >> v2:172.21.15.114:3300/0 conn(0x7efc940cd680 msgr2=0x7efc9406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9a34e700 1 --2- 172.21.15.104:0/2809474446 >> v2:172.21.15.114:3300/0 conn(0x7efc940cd680 0x7efc9406dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.483 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9a34e700 1 -- 172.21.15.104:0/2809474446 >> v1:172.21.15.104:6789/0 conn(0x7efc940b1080 legacy=0x7efc940ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9934c700 1 -- 172.21.15.104:0/2809474446 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375445130 0 0) 0x7efc940c9e60 con 0x7efc940b1080 2023-06-10T19:00:18.484 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9a34e700 1 -- 172.21.15.104:0/2809474446 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940aee70 con 0x7efc940aeaa0 2023-06-10T19:00:18.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9a34e700 1 --2- 172.21.15.104:0/2809474446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc940cd140 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7efc94071030 tx=0x7efc84009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbca691b7c4e946 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.485 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9934c700 1 -- 172.21.15.104:0/2809474446 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84005e30 con 0x7efc940aeaa0 2023-06-10T19:00:18.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9934c700 1 -- 172.21.15.104:0/2809474446 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc84006030 con 0x7efc940aeaa0 2023-06-10T19:00:18.486 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.467+0000 7efc9934c700 1 -- 172.21.15.104:0/2809474446 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc840063c0 con 0x7efc940aeaa0 2023-06-10T19:00:18.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 -- 172.21.15.104:0/2809474446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 msgr2=0x7efc940cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.487 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/2809474446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc940cd140 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7efc94071030 tx=0x7efc84009b00 comp rx=0 tx=0).stop 2023-06-10T19:00:18.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 -- 172.21.15.104:0/2809474446 shutdown_connections 2023-06-10T19:00:18.489 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/2809474446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc940cd140 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/2809474446 >> v2:172.21.15.114:3300/0 conn(0x7efc940cd680 0x7efc9406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.490 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 -- 172.21.15.104:0/2809474446 >> 172.21.15.104:0/2809474446 conn(0x7efc9406b8a0 msgr2=0x7efc940cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 -- 172.21.15.104:0/2809474446 shutdown_connections 2023-06-10T19:00:18.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.468+0000 7efc9c5b2700 1 -- 172.21.15.104:0/2809474446 wait complete. 2023-06-10T19:00:18.491 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.469+0000 7efc9c5b2700 1 Processor -- start 2023-06-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.469+0000 7efc9c5b2700 1 -- start start 2023-06-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc940b1080 0x7efc940b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc940cd680 0x7efc941f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94155880 con 0x7efc940aeaa0 2023-06-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940b4b90 con 0x7efc940b1080 2023-06-10T19:00:18.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9c5b2700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc941f2e10 con 0x7efc940cd680 2023-06-10T19:00:18.497 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9a34e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.498 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9a34e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42620/0 (socket says 172.21.15.104:42620) 2023-06-10T19:00:18.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.470+0000 7efc9a34e700 1 -- 172.21.15.104:0/3033297590 learned_addr learned my addr 172.21.15.104:0/3033297590 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:18.499 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 -- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc940cd680 msgr2=0x7efc941f26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc940cd680 0x7efc941f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.500 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 -- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc940b1080 msgr2=0x7efc940b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc940b1080 0x7efc940b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.501 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 -- 172.21.15.104:0/3033297590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84009430 con 0x7efc940aeaa0 2023-06-10T19:00:18.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9a34e700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7efc84009fd0 tx=0x7efc84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.503 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84004040 con 0x7efc940aeaa0 2023-06-10T19:00:18.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc84020070 con 0x7efc940aeaa0 2023-06-10T19:00:18.504 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc941f2c10 con 0x7efc940aeaa0 2023-06-10T19:00:18.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84004400 con 0x7efc940aeaa0 2023-06-10T19:00:18.505 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.471+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc940b5050 con 0x7efc940aeaa0 2023-06-10T19:00:18.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.472+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94074050 con 0x7efc940aeaa0 2023-06-10T19:00:18.506 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.472+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc84022030 con 0x7efc940aeaa0 2023-06-10T19:00:18.508 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.473+0000 7efc8affd700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.509 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.473+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7efc8408a760 con 0x7efc940aeaa0 2023-06-10T19:00:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.473+0000 7efc99b4d700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.510 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.473+0000 7efc99b4d700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7efc900011e0 tx=0x7efc900079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.511 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.477+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc84055a30 con 0x7efc940aeaa0 2023-06-10T19:00:18.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.663+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7efc9406e540 con 0x7efc7806ace0 2023-06-10T19:00:18.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.664+0000 7efc8affd700 1 -- 172.21.15.104:0/3033297590 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efc900070d0 con 0x7efc7806ace0 2023-06-10T19:00:18.674 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 msgr2=0x7efc7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7efc900011e0 tx=0x7efc900079b0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.675 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 msgr2=0x7efc941f1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.677 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7efc84009fd0 tx=0x7efc84004750 comp rx=0 tx=0).stop 2023-06-10T19:00:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 shutdown_connections 2023-06-10T19:00:18.678 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc940cd680 0x7efc941f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc940aeaa0 0x7efc941f1660 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.679 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7efc7806ace0 0x7efc7806d190 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 --2- 172.21.15.104:0/3033297590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc940b1080 0x7efc940b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.680 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.666+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 >> 172.21.15.104:0/3033297590 conn(0x7efc9406b8a0 msgr2=0x7efc940ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.681 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.667+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 shutdown_connections 2023-06-10T19:00:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.667+0000 7efc9c5b2700 1 -- 172.21.15.104:0/3033297590 wait complete. 2023-06-10T19:00:18.682 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 2 to deep-scrub 2023-06-10T19:00:18.685 INFO:tasks.ceph:Scrubbing osd.3 2023-06-10T19:00:18.685 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:18.813 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.806+0000 7f9bd8a20700 1 Processor -- start 2023-06-10T19:00:18.814 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.806+0000 7f9bd8a20700 1 -- start start 2023-06-10T19:00:18.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.807+0000 7f9bd8a20700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd40cea50 0x7f9bd40d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.807+0000 7f9bd8a20700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9bd40d0830 0x7f9bd40b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.807+0000 7f9bd8a20700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd40ad3c0 con 0x7f9bd4157b30 2023-06-10T19:00:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.807+0000 7f9bd8a20700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd4157f00 con 0x7f9bd40d0830 2023-06-10T19:00:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.807+0000 7f9bd8a20700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd40b0170 con 0x7f9bd40cea50 2023-06-10T19:00:18.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd259c700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9bd4157b30 0x7f9bd40ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40456/0 (socket says 172.21.15.104:40456) 2023-06-10T19:00:18.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd259c700 1 -- 172.21.15.104:0/1752145247 learned_addr learned my addr 172.21.15.104:0/1752145247 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:18.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd40cea50 0x7f9bd40d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> v2:172.21.15.114:3300/0 conn(0x7f9bd40d0830 0x7f9bd40b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436649552 0 0) 0x7f9bd40ad3c0 con 0x7f9bd4157b30 2023-06-10T19:00:18.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bbc0035b0 con 0x7f9bd4157b30 2023-06-10T19:00:18.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043573330 0 0) 0x7f9bbc0035b0 con 0x7f9bd4157b30 2023-06-10T19:00:18.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd40cea50 msgr2=0x7f9bd40d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd40cea50 0x7f9bd40d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 >> v2:172.21.15.114:3300/0 conn(0x7f9bd40d0830 msgr2=0x7f9bd40b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 --2- 172.21.15.104:0/1752145247 >> v2:172.21.15.114:3300/0 conn(0x7f9bd40d0830 0x7f9bd40b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd40b0410 con 0x7f9bd4157b30 2023-06-10T19:00:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f9bc8001670 con 0x7f9bd4157b30 2023-06-10T19:00:18.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 >> v1:172.21.15.104:6789/0 conn(0x7f9bd4157b30 legacy=0x7f9bd40ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc003a00 0x7f9bbc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bbc004430 0x7f9bbc0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bbc004f10 0x7f9bbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4157f00 con 0x7f9bbc004f10 2023-06-10T19:00:18.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd40b0170 con 0x7f9bbc003a00 2023-06-10T19:00:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc00bf50 con 0x7f9bbc004430 2023-06-10T19:00:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd40cea50 0x7f9bd40d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> v2:172.21.15.114:3300/0 conn(0x7f9bd40d0830 0x7f9bd40b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bbc004f10 0x7f9bbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd259c700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc003a00 0x7f9bbc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.808+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bbc004430 0x7f9bbc0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bbc004430 msgr2=0x7f9bbc0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bbc004430 0x7f9bbc0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc003a00 msgr2=0x7f9bbc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc003a00 0x7f9bbc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/1752145247 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0035b0 con 0x7f9bbc004f10 2023-06-10T19:00:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bbc004f10 0x7f9bbc00ba10 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00e350 tx=0x7f9bbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3962c6bc227e737 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00ee10 con 0x7f9bbc004f10 2023-06-10T19:00:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc00f040 con 0x7f9bbc004f10 2023-06-10T19:00:18.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd159a700 1 -- 172.21.15.104:0/1752145247 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc003c40 con 0x7f9bbc004f10 2023-06-10T19:00:18.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd8a20700 1 -- 172.21.15.104:0/1752145247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bbc004f10 msgr2=0x7f9bbc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd8a20700 1 --2- 172.21.15.104:0/1752145247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bbc004f10 0x7f9bbc00ba10 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00e350 tx=0x7f9bbc005880 comp rx=0 tx=0).stop 2023-06-10T19:00:18.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.809+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/1752145247 reap_dead start 2023-06-10T19:00:18.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.810+0000 7f9bd8a20700 1 -- 172.21.15.104:0/1752145247 shutdown_connections 2023-06-10T19:00:18.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.810+0000 7f9bd8a20700 1 -- 172.21.15.104:0/1752145247 >> 172.21.15.104:0/1752145247 conn(0x7f9bd4089fd0 msgr2=0x7f9bd408a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.810+0000 7f9bd8a20700 1 -- 172.21.15.104:0/1752145247 shutdown_connections 2023-06-10T19:00:18.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.810+0000 7f9bd8a20700 1 -- 172.21.15.104:0/1752145247 wait complete. 2023-06-10T19:00:18.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.811+0000 7f9bd8a20700 1 Processor -- start 2023-06-10T19:00:18.827 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.811+0000 7f9bd8a20700 1 -- start start 2023-06-10T19:00:18.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc004f10 0x7f9bd406b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd414b900 0x7f9bd4159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.830 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 0x7f9bd4159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd406bed0 con 0x7f9bd414bd70 2023-06-10T19:00:18.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd40b2120 con 0x7f9bbc004f10 2023-06-10T19:00:18.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd40b2400 con 0x7f9bd414b900 2023-06-10T19:00:18.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 0x7f9bd4159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.831 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 0x7f9bd4159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42644/0 (socket says 172.21.15.104:42644) 2023-06-10T19:00:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/2820626498 learned_addr learned my addr 172.21.15.104:0/2820626498 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd259c700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc004f10 0x7f9bd406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd414b900 0x7f9bd4159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd414b900 msgr2=0x7f9bd4159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.832 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd414b900 0x7f9bd4159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc004f10 msgr2=0x7f9bd406b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc004f10 0x7f9bd406b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd40b26e0 con 0x7f9bd414bd70 2023-06-10T19:00:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9bd414b900 0x7f9bd4159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:18.833 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd2d9d700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 0x7f9bd4159660 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc002230 tx=0x7f9bcc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc004440 con 0x7f9bd414bd70 2023-06-10T19:00:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc022070 con 0x7f9bd414bd70 2023-06-10T19:00:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd406bed0 con 0x7f9bd414bd70 2023-06-10T19:00:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc004800 con 0x7f9bd414bd70 2023-06-10T19:00:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.812+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd40b2ca0 con 0x7f9bd414bd70 2023-06-10T19:00:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.813+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9bd40b09d0 con 0x7f9bd414bd70 2023-06-10T19:00:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9bcc014480 con 0x7f9bd414bd70 2023-06-10T19:00:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9bb406ace0 0x7f9bb406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.835 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9bcc024070 con 0x7f9bd414bd70 2023-06-10T19:00:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7f9bb40715d0 0x7f9bb4073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bb40740d0 con 0x7f9bb40715d0 2023-06-10T19:00:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bcc08d5f0 con 0x7f9bd414bd70 2023-06-10T19:00:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bd259c700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9bb406ace0 0x7f9bb406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.836 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.814+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7f9bb40715d0 0x7f9bb4073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.815+0000 7f9bd1d9b700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7f9bb40715d0 0x7f9bb4073a00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.815+0000 7f9bd259c700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9bb406ace0 0x7f9bb406d190 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000c00 tx=0x7f9bc8007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.837 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.815+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== osd.3 v2:172.21.15.104:6808/3217431778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9bb40740d0 con 0x7f9bb40715d0 2023-06-10T19:00:18.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 --> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9bd40b06f0 con 0x7f9bb40715d0 2023-06-10T19:00:18.838 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bc2ffd700 1 -- 172.21.15.104:0/2820626498 <== osd.3 v2:172.21.15.104:6808/3217431778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9bc4009e30 con 0x7f9bb40715d0 2023-06-10T19:00:18.838 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:18.839 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:18.839 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7f9bb40715d0 msgr2=0x7f9bb4073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:6808/3217431778,v1:172.21.15.104:6809/3217431778] conn(0x7f9bb40715d0 0x7f9bb4073a00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.840 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9bb406ace0 msgr2=0x7f9bb406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f9bb406ace0 0x7f9bb406d190 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000c00 tx=0x7f9bc8007980 comp rx=0 tx=0).stop 2023-06-10T19:00:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 msgr2=0x7f9bd4159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 --2- 172.21.15.104:0/2820626498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bd414bd70 0x7f9bd4159660 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc002230 tx=0x7f9bcc009750 comp rx=0 tx=0).stop 2023-06-10T19:00:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd2d9d700 1 -- 172.21.15.104:0/2820626498 reap_dead start 2023-06-10T19:00:18.841 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 shutdown_connections 2023-06-10T19:00:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 >> 172.21.15.104:0/2820626498 conn(0x7f9bd4089fd0 msgr2=0x7f9bd406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 shutdown_connections 2023-06-10T19:00:18.842 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.830+0000 7f9bd8a20700 1 -- 172.21.15.104:0/2820626498 wait complete. 2023-06-10T19:00:18.848 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-06-10T19:00:18.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.962+0000 7fd613534700 1 Processor -- start 2023-06-10T19:00:18.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.962+0000 7fd613534700 1 -- start start 2023-06-10T19:00:18.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd613534700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd60c0aeaa0 0x7fd60c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd613534700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd613534700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0c9e60 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd613534700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c0c9bc0 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd613534700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0aee70 con 0x7fd60c0b1080 2023-06-10T19:00:18.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd60c0aeaa0 0x7fd60c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fd60c0aeaa0 0x7fd60c0cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42650/0 (socket says 172.21.15.104:42650) 2023-06-10T19:00:18.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 -- 172.21.15.104:0/2229032541 learned_addr learned my addr 172.21.15.104:0/2229032541 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:18.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd610acf700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 -- 172.21.15.104:0/2229032541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 msgr2=0x7fd60c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 -- 172.21.15.104:0/2229032541 >> v1:172.21.15.114:6789/0 conn(0x7fd60c0ca2f0 legacy=0x7fd60c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:18.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd6112d0700 1 -- 172.21.15.104:0/2229032541 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c06e540 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.963+0000 7fd610acf700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd6112d0700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c0cd140 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006fd0 tx=0x7fd5fc00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24156d02e4fa7288 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd603fff700 1 -- 172.21.15.104:0/2229032541 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc005e30 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd603fff700 1 -- 172.21.15.104:0/2229032541 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc004030 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd603fff700 1 -- 172.21.15.104:0/2229032541 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0043e0 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 -- 172.21.15.104:0/2229032541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0aeaa0 msgr2=0x7fd60c0cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c0cd140 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006fd0 tx=0x7fd5fc00be70 comp rx=0 tx=0).stop 2023-06-10T19:00:18.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 -- 172.21.15.104:0/2229032541 shutdown_connections 2023-06-10T19:00:18.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c0cd140 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 --2- 172.21.15.104:0/2229032541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 -- 172.21.15.104:0/2229032541 >> 172.21.15.104:0/2229032541 conn(0x7fd60c06b8a0 msgr2=0x7fd60c0cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:18.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 -- 172.21.15.104:0/2229032541 shutdown_connections 2023-06-10T19:00:18.976 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.964+0000 7fd613534700 1 -- 172.21.15.104:0/2229032541 wait complete. 2023-06-10T19:00:18.977 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.965+0000 7fd613534700 1 Processor -- start 2023-06-10T19:00:18.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 -- start start 2023-06-10T19:00:18.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.978 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c159470 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c159750 con 0x7fd60c0aeaa0 2023-06-10T19:00:18.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd613534700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c159a30 con 0x7fd60c0b1080 2023-06-10T19:00:18.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd611ad1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.979 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd611ad1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42660/0 (socket says 172.21.15.104:42660) 2023-06-10T19:00:18.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd611ad1700 1 -- 172.21.15.104:0/2225397274 learned_addr learned my addr 172.21.15.104:0/2225397274 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:18.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd610acf700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.966+0000 7fd6112d0700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 -- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 msgr2=0x7fd60c0c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.980 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0c3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 -- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd60c0aeaa0 msgr2=0x7fd60c151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:18.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:18.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 -- 172.21.15.104:0/2225397274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc00b7a0 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd611ad1700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fd60400cb50 tx=0x7fd60400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.981 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd604003c00 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604003d60 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c159d10 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60400aa20 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.967+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c15c080 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.982 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.968+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c15c240 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.968+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd60400ab80 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.969+0000 7fd601ffb700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 0x7fd5f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:18.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.969+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd604089220 con 0x7fd60c0ca2f0 2023-06-10T19:00:18.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.969+0000 7fd6112d0700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 0x7fd5f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:18.983 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.969+0000 7fd6112d0700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 0x7fd5f006d190 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b340 tx=0x7fd5fc012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:18.984 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:18.973+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd6040544f0 con 0x7fd60c0ca2f0 2023-06-10T19:00:19.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.156+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd60c1598c0 con 0x7fd5f006ace0 2023-06-10T19:00:19.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.157+0000 7fd601ffb700 1 -- 172.21.15.104:0/2225397274 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd60c1598c0 con 0x7fd5f006ace0 2023-06-10T19:00:19.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.159+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 msgr2=0x7fd5f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.159+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 0x7fd5f006d190 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b340 tx=0x7fd5fc012000 comp rx=0 tx=0).stop 2023-06-10T19:00:19.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.159+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 msgr2=0x7fd60c15bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.159+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fd60400cb50 tx=0x7fd60400ce60 comp rx=0 tx=0).stop 2023-06-10T19:00:19.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 shutdown_connections 2023-06-10T19:00:19.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd5f006ace0 0x7fd5f006d190 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd60c0aeaa0 0x7fd60c151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd60c0ca2f0 0x7fd60c15bb40 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 --2- 172.21.15.104:0/2225397274 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd60c0b1080 0x7fd60c0c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 >> 172.21.15.104:0/2225397274 conn(0x7fd60c06b8a0 msgr2=0x7fd60c0cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 shutdown_connections 2023-06-10T19:00:19.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.160+0000 7fd613534700 1 -- 172.21.15.104:0/2225397274 wait complete. 2023-06-10T19:00:19.169 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 3 to deep-scrub 2023-06-10T19:00:19.178 INFO:tasks.ceph:Scrubbing osd.4 2023-06-10T19:00:19.178 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:19.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.296+0000 7fb8bda0b700 1 Processor -- start 2023-06-10T19:00:19.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.296+0000 7fb8bda0b700 1 -- start start 2023-06-10T19:00:19.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8bda0b700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8b8145cc0 0x7fb8b806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8bda0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b8070a10 0x7fb8b8070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8bda0b700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80713a0 con 0x7fb8b815fd40 2023-06-10T19:00:19.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8bda0b700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b80c4770 con 0x7fb8b8145cc0 2023-06-10T19:00:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8bda0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b8069e70 con 0x7fb8b8070a10 2023-06-10T19:00:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8b6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8b8145cc0 0x7fb8b806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8b6ffd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb8b8145cc0 0x7fb8b806c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42664/0 (socket says 172.21.15.104:42664) 2023-06-10T19:00:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1708141969 learned_addr learned my addr 172.21.15.104:0/1708141969 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.297+0000 7fb8b77fe700 1 --1- 172.21.15.104:0/1708141969 >> v1:172.21.15.104:6789/0 conn(0x7fb8b815fd40 0x7fb8b8160180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40494/0 (socket says 172.21.15.104:40494) 2023-06-10T19:00:19.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1708141969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b8070a10 msgr2=0x7fb8b8070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:19.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b67fc700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b8070a10 0x7fb8b8070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b6ffd700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b8070a10 0x7fb8b8070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b5ffb700 1 -- 172.21.15.104:0/1708141969 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865880418 0 0) 0x7fb8b0001490 con 0x7fb8b815fd40 2023-06-10T19:00:19.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1708141969 >> v1:172.21.15.104:6789/0 conn(0x7fb8b815fd40 legacy=0x7fb8b8160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1708141969 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80c49c0 con 0x7fb8b8145cc0 2023-06-10T19:00:19.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b6ffd700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8145cc0 0x7fb8b806c800 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0098f0 tx=0x7fb8ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdad37247767de3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b5ffb700 1 -- 172.21.15.104:0/1708141969 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00b8a0 con 0x7fb8b8145cc0 2023-06-10T19:00:19.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b5ffb700 1 -- 172.21.15.104:0/1708141969 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac00ba00 con 0x7fb8b8145cc0 2023-06-10T19:00:19.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8b5ffb700 1 -- 172.21.15.104:0/1708141969 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00bd90 con 0x7fb8b8145cc0 2023-06-10T19:00:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1708141969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8145cc0 msgr2=0x7fb8b806c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8145cc0 0x7fb8b806c800 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0098f0 tx=0x7fb8ac009c00 comp rx=0 tx=0).stop 2023-06-10T19:00:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1708141969 shutdown_connections 2023-06-10T19:00:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8145cc0 0x7fb8b806c800 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1708141969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b8070a10 0x7fb8b8070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1708141969 >> 172.21.15.104:0/1708141969 conn(0x7fb8b80ceec0 msgr2=0x7fb8b80cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1708141969 shutdown_connections 2023-06-10T19:00:19.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.298+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1708141969 wait complete. 2023-06-10T19:00:19.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 Processor -- start 2023-06-10T19:00:19.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 -- start start 2023-06-10T19:00:19.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 0x7fb8b815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b8145cc0 0x7fb8b80ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b815fd40 0x7fb8b80adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80cb9a0 con 0x7fb8b8070a10 2023-06-10T19:00:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80ca900 con 0x7fb8b8145cc0 2023-06-10T19:00:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.300+0000 7fb8bda0b700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80cabe0 con 0x7fb8b815fd40 2023-06-10T19:00:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 0x7fb8b815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 0x7fb8b815b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42668/0 (socket says 172.21.15.104:42668) 2023-06-10T19:00:19.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1233273004 learned_addr learned my addr 172.21.15.104:0/1233273004 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b77fe700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b815fd40 0x7fb8b80adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b815fd40 msgr2=0x7fb8b80adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b67fc700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b8145cc0 0x7fb8b80ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8b815fd40 0x7fb8b80adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b8145cc0 msgr2=0x7fb8b80ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b8145cc0 0x7fb8b80ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac009550 con 0x7fb8b8070a10 2023-06-10T19:00:19.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b67fc700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8b8145cc0 0x7fb8b80ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:19.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8b6ffd700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 0x7fb8b815b380 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009c40 tx=0x7fb8ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0036a0 con 0x7fb8b8070a10 2023-06-10T19:00:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac003800 con 0x7fb8b8070a10 2023-06-10T19:00:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b80caec0 con 0x7fb8b8070a10 2023-06-10T19:00:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac003b70 con 0x7fb8b8070a10 2023-06-10T19:00:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.301+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b80cb480 con 0x7fb8b8070a10 2023-06-10T19:00:19.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.302+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8b80c4f80 con 0x7fb8b8070a10 2023-06-10T19:00:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.302+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb8ac003cd0 con 0x7fb8b8070a10 2023-06-10T19:00:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb89ffff700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8a406ace0 0x7fb8a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb8ac021030 con 0x7fb8b8070a10 2023-06-10T19:00:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb89ffff700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fb8a40715d0 0x7fb8a4073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.318 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8a40740d0 con 0x7fb8a40715d0 2023-06-10T19:00:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8ac08f2c0 con 0x7fb8b8070a10 2023-06-10T19:00:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb8b67fc700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8a406ace0 0x7fb8a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.303+0000 7fb8b77fe700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fb8a40715d0 0x7fb8a4073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.304+0000 7fb8b67fc700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8a406ace0 0x7fb8a406d190 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.319 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.304+0000 7fb8b77fe700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fb8a40715d0 0x7fb8a4073a00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.320 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.304+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== osd.4 v2:172.21.15.114:6800/3316008371 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb8a40740d0 con 0x7fb8a40715d0 2023-06-10T19:00:19.324 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.318+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 --> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb8b80c4ca0 con 0x7fb8a40715d0 2023-06-10T19:00:19.325 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb89ffff700 1 -- 172.21.15.104:0/1233273004 <== osd.4 v2:172.21.15.114:6800/3316008371 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb8b0001e30 con 0x7fb8a40715d0 2023-06-10T19:00:19.325 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:19.326 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:19.326 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fb8a40715d0 msgr2=0x7fb8a4073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6800/3316008371,v1:172.21.15.114:6801/3316008371] conn(0x7fb8a40715d0 0x7fb8a4073a00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8a406ace0 msgr2=0x7fb8a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.327 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb8a406ace0 0x7fb8a406d190 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8006fd0 tx=0x7fb8a8008040 comp rx=0 tx=0).stop 2023-06-10T19:00:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 msgr2=0x7fb8b815b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 --2- 172.21.15.104:0/1233273004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b8070a10 0x7fb8b815b380 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009c40 tx=0x7fb8ac004750 comp rx=0 tx=0).stop 2023-06-10T19:00:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8b77fe700 1 -- 172.21.15.104:0/1233273004 reap_dead start 2023-06-10T19:00:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 shutdown_connections 2023-06-10T19:00:19.328 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 >> 172.21.15.104:0/1233273004 conn(0x7fb8b80ceec0 msgr2=0x7fb8b806cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.319+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 shutdown_connections 2023-06-10T19:00:19.329 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.320+0000 7fb8bda0b700 1 -- 172.21.15.104:0/1233273004 wait complete. 2023-06-10T19:00:19.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-06-10T19:00:19.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.451+0000 7fc2dc9e0700 1 Processor -- start 2023-06-10T19:00:19.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.451+0000 7fc2dc9e0700 1 -- start start 2023-06-10T19:00:19.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2dc9e0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d80cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2dc9e0700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2d808a3d0 0x7fc2d80cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2dc9e0700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d80cadd0 con 0x7fc2d80ce350 2023-06-10T19:00:19.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2dc9e0700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d80cfeb0 con 0x7fc2d808a3d0 2023-06-10T19:00:19.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2dc9e0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80cb0b0 con 0x7fc2d806b1f0 2023-06-10T19:00:19.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2d808a3d0 0x7fc2d80cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fc2d808a3d0 0x7fc2d80cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42680/0 (socket says 172.21.15.104:42680) 2023-06-10T19:00:19.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/3198117015 learned_addr learned my addr 172.21.15.104:0/3198117015 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d659c700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d80cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/3198117015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 msgr2=0x7fc2d80cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:19.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d80cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/3198117015 >> v1:172.21.15.114:6789/0 conn(0x7fc2d80ce350 legacy=0x7fc2d80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/3198117015 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d80cf550 con 0x7fc2d808a3d0 2023-06-10T19:00:19.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d659c700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d80cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:19.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d6d9d700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d808a3d0 0x7fc2d80cb540 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009e50 tx=0x7fc2d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7766fde6b7b45c7f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.452+0000 7fc2d559a700 1 -- 172.21.15.104:0/3198117015 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d000e070 con 0x7fc2d808a3d0 2023-06-10T19:00:19.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2d559a700 1 -- 172.21.15.104:0/3198117015 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d000b9c0 con 0x7fc2d808a3d0 2023-06-10T19:00:19.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2d559a700 1 -- 172.21.15.104:0/3198117015 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d000bd70 con 0x7fc2d808a3d0 2023-06-10T19:00:19.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/3198117015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d808a3d0 msgr2=0x7fc2d80cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d808a3d0 0x7fc2d80cb540 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009e50 tx=0x7fc2d000b330 comp rx=0 tx=0).stop 2023-06-10T19:00:19.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/3198117015 shutdown_connections 2023-06-10T19:00:19.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d808a3d0 0x7fc2d80cb540 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/3198117015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d80cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/3198117015 >> 172.21.15.104:0/3198117015 conn(0x7fc2d8089fd0 msgr2=0x7fc2d80d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.465 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/3198117015 shutdown_connections 2023-06-10T19:00:19.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.453+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/3198117015 wait complete. 2023-06-10T19:00:19.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.454+0000 7fc2dc9e0700 1 Processor -- start 2023-06-10T19:00:19.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 -- start start 2023-06-10T19:00:19.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d809b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d808a3d0 0x7fc2d809d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d80c9010 con 0x7fc2d80ce350 2023-06-10T19:00:19.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d815f0c0 con 0x7fc2d808a3d0 2023-06-10T19:00:19.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2dc9e0700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d809e510 con 0x7fc2d806b1f0 2023-06-10T19:00:19.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2d6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2d6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42686/0 (socket says 172.21.15.104:42686) 2023-06-10T19:00:19.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.455+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/2726610490 learned_addr learned my addr 172.21.15.104:0/2726610490 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d5d9b700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d808a3d0 0x7fc2d809d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d659c700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d809b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 msgr2=0x7fc2d809b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d809b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.472 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d808a3d0 msgr2=0x7fc2d809d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d808a3d0 0x7fc2d809d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 -- 172.21.15.104:0/2726610490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0009b30 con 0x7fc2d80ce350 2023-06-10T19:00:19.473 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d659c700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d809b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2d6d9d700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8073ac0 tx=0x7fc2d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00048b0 con 0x7fc2d80ce350 2023-06-10T19:00:19.474 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d00099b0 con 0x7fc2d80ce350 2023-06-10T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d809d730 con 0x7fc2d80ce350 2023-06-10T19:00:19.475 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d001c680 con 0x7fc2d80ce350 2023-06-10T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.456+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d809df50 con 0x7fc2d80ce350 2023-06-10T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.457+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d80c9150 con 0x7fc2d80ce350 2023-06-10T19:00:19.476 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.457+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc2d001c7e0 con 0x7fc2d80ce350 2023-06-10T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.458+0000 7fc2c6ffd700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 0x7fc2b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.458+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc2d008af20 con 0x7fc2d80ce350 2023-06-10T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.458+0000 7fc2d659c700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 0x7fc2b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.477 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.458+0000 7fc2d659c700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 0x7fc2b806d190 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80099f0 tx=0x7fc2c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.478 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.462+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc2d80c9150 con 0x7fc2d80ce350 2023-06-10T19:00:19.656 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.650+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2d80cf240 con 0x7fc2b806ace0 2023-06-10T19:00:19.657 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.651+0000 7fc2c6ffd700 1 -- 172.21.15.104:0/2726610490 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc2c8009680 con 0x7fc2b806ace0 2023-06-10T19:00:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 msgr2=0x7fc2b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.659 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 0x7fc2b806d190 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80099f0 tx=0x7fc2c8008040 comp rx=0 tx=0).stop 2023-06-10T19:00:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 msgr2=0x7fc2d8154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8073ac0 tx=0x7fc2d0004ad0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 shutdown_connections 2023-06-10T19:00:19.660 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d80ce350 0x7fc2d8154b30 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2d806b1f0 0x7fc2d809b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc2b806ace0 0x7fc2b806d190 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 --2- 172.21.15.104:0/2726610490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d808a3d0 0x7fc2d809d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 >> 172.21.15.104:0/2726610490 conn(0x7fc2d8089fd0 msgr2=0x7fc2d815fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.661 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 shutdown_connections 2023-06-10T19:00:19.662 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.653+0000 7fc2dc9e0700 1 -- 172.21.15.104:0/2726610490 wait complete. 2023-06-10T19:00:19.662 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 4 to deep-scrub 2023-06-10T19:00:19.671 INFO:tasks.ceph:Scrubbing osd.5 2023-06-10T19:00:19.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:19.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.788+0000 7f68a9b7e700 1 Processor -- start 2023-06-10T19:00:19.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 -- start start 2023-06-10T19:00:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a415c730 0x7f68a40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68a40ae230 0x7f68a406d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a415cb00 con 0x7f68a40cd720 2023-06-10T19:00:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a40cdb90 con 0x7f68a40ae230 2023-06-10T19:00:19.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.789+0000 7f68a9b7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40acd90 con 0x7f68a415c730 2023-06-10T19:00:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68a40ae230 0x7f68a406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a37fe700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a415c730 0x7f68a40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68a40ae230 0x7f68a406d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.104:58428/0 (socket says 172.21.15.104:58428) 2023-06-10T19:00:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 -- 172.21.15.104:0/701124943 learned_addr learned my addr 172.21.15.104:0/701124943 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 -- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a415c730 msgr2=0x7f68a40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 --2- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a415c730 0x7f68a40cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 -- 172.21.15.104:0/701124943 >> v1:172.21.15.114:6789/0 conn(0x7f68a40cd720 legacy=0x7f68a406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 -- 172.21.15.104:0/701124943 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40ac460 con 0x7f68a40ae230 2023-06-10T19:00:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a27fc700 1 -- 172.21.15.104:0/701124943 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143788889 0 0) 0x7f6894001110 con 0x7f68a40cd720 2023-06-10T19:00:19.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a3fff700 1 --2- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a40ae230 0x7f68a406d480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f689c00d150 tx=0x7f689c00d460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=130324a21aa7d1fb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a27fc700 1 -- 172.21.15.104:0/701124943 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c010070 con 0x7f68a40ae230 2023-06-10T19:00:19.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a27fc700 1 -- 172.21.15.104:0/701124943 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00dd10 con 0x7f68a40ae230 2023-06-10T19:00:19.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.790+0000 7f68a27fc700 1 -- 172.21.15.104:0/701124943 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00b310 con 0x7f68a40ae230 2023-06-10T19:00:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 -- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a40ae230 msgr2=0x7f68a406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a40ae230 0x7f68a406d480 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f689c00d150 tx=0x7f689c00d460 comp rx=0 tx=0).stop 2023-06-10T19:00:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 -- 172.21.15.104:0/701124943 shutdown_connections 2023-06-10T19:00:19.803 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a40ae230 0x7f68a406d480 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/701124943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a415c730 0x7f68a40cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 -- 172.21.15.104:0/701124943 >> 172.21.15.104:0/701124943 conn(0x7f68a406b8a0 msgr2=0x7f68a40cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 -- 172.21.15.104:0/701124943 shutdown_connections 2023-06-10T19:00:19.804 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.791+0000 7f68a9b7e700 1 -- 172.21.15.104:0/701124943 wait complete. 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.792+0000 7f68a9b7e700 1 Processor -- start 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.792+0000 7f68a9b7e700 1 -- start start 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a40ae230 0x7f68a40b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 0x7f68a40ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a415c730 0x7f68a40c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40b5370 con 0x7f68a40cd720 2023-06-10T19:00:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a4151c30 con 0x7f68a415c730 2023-06-10T19:00:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a9b7e700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40b0f10 con 0x7f68a40ae230 2023-06-10T19:00:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 0x7f68a40ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 0x7f68a40ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42698/0 (socket says 172.21.15.104:42698) 2023-06-10T19:00:19.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a2ffd700 1 -- 172.21.15.104:0/4173159151 learned_addr learned my addr 172.21.15.104:0/4173159151 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a3fff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a415c730 0x7f68a40c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.793+0000 7f68a37fe700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a40ae230 0x7f68a40b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 -- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a40ae230 msgr2=0x7f68a40b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68a40ae230 0x7f68a40b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.807 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 -- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a415c730 msgr2=0x7f68a40c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a415c730 0x7f68a40c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c00bd40 con 0x7f68a40cd720 2023-06-10T19:00:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a3fff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68a415c730 0x7f68a40c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a2ffd700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 0x7f68a40ca3c0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f689400c750 tx=0x7f689400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68940054e0 con 0x7f68a40cd720 2023-06-10T19:00:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6894003680 con 0x7f68a40cd720 2023-06-10T19:00:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894003a30 con 0x7f68a40cd720 2023-06-10T19:00:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a40b4760 con 0x7f68a40cd720 2023-06-10T19:00:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.794+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a40b4d20 con 0x7f68a40cd720 2023-06-10T19:00:19.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.795+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68a40aca20 con 0x7f68a40cd720 2023-06-10T19:00:19.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.795+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6894003b90 con 0x7f68a40cd720 2023-06-10T19:00:19.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f688bfff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f689006ace0 0x7f689006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6894088610 con 0x7f68a40cd720 2023-06-10T19:00:19.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f688bfff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f68900715d0 0x7f6890073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68900740d0 con 0x7f68900715d0 2023-06-10T19:00:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7f689408acd0 con 0x7f68a40cd720 2023-06-10T19:00:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f68a37fe700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f689006ace0 0x7f689006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f68a3fff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f68900715d0 0x7f6890073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f68a37fe700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f689006ace0 0x7f689006d190 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6898009b50 tx=0x7f6898008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.811 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.796+0000 7f68a3fff700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f68900715d0 0x7f6890073a00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.812 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.797+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== osd.5 v2:172.21.15.114:6817/1721425129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f68900740d0 con 0x7f68900715d0 2023-06-10T19:00:19.817 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.811+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 --> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f68a40ac740 con 0x7f68900715d0 2023-06-10T19:00:19.818 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f688bfff700 1 -- 172.21.15.104:0/4173159151 <== osd.5 v2:172.21.15.114:6817/1721425129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f689c003a20 con 0x7f68900715d0 2023-06-10T19:00:19.818 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:19.818 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:19.819 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:19.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f68900715d0 msgr2=0x7f6890073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6817/1721425129,v1:172.21.15.114:6819/1721425129] conn(0x7f68900715d0 0x7f6890073a00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f689006ace0 msgr2=0x7f689006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f689006ace0 0x7f689006d190 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6898009b50 tx=0x7f6898008040 comp rx=0 tx=0).stop 2023-06-10T19:00:19.820 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 msgr2=0x7f68a40ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 --2- 172.21.15.104:0/4173159151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68a40cd720 0x7f68a40ca3c0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f689400c750 tx=0x7f689400cb10 comp rx=0 tx=0).stop 2023-06-10T19:00:19.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a3fff700 1 -- 172.21.15.104:0/4173159151 reap_dead start 2023-06-10T19:00:19.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 shutdown_connections 2023-06-10T19:00:19.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 >> 172.21.15.104:0/4173159151 conn(0x7f68a406b8a0 msgr2=0x7f68a406d570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 shutdown_connections 2023-06-10T19:00:19.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.812+0000 7f68a9b7e700 1 -- 172.21.15.104:0/4173159151 wait complete. 2023-06-10T19:00:19.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-06-10T19:00:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.944+0000 7ff6665e1700 1 Processor -- start 2023-06-10T19:00:19.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.944+0000 7ff6665e1700 1 -- start start 2023-06-10T19:00:19.952 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.944+0000 7ff6665e1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e0e0 0x7ff66006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.944+0000 7ff6665e1700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff66006e4b0 0x7ff6600c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff6665e1700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66006d290 con 0x7ff6600ca100 2023-06-10T19:00:19.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff6665e1700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff660071640 con 0x7ff66006e4b0 2023-06-10T19:00:19.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff6665e1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600aef00 con 0x7ff66006e0e0 2023-06-10T19:00:19.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff66006e4b0 0x7ff6600c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7ff66006e4b0 0x7ff6600c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42708/0 (socket says 172.21.15.104:42708) 2023-06-10T19:00:19.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 -- 172.21.15.104:0/657060354 learned_addr learned my addr 172.21.15.104:0/657060354 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65ffff700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e0e0 0x7ff66006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 -- 172.21.15.104:0/657060354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e0e0 msgr2=0x7ff66006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:19.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e0e0 0x7ff66006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 -- 172.21.15.104:0/657060354 >> v1:172.21.15.114:6789/0 conn(0x7ff6600ca100 legacy=0x7ff6600ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 -- 172.21.15.104:0/657060354 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66006d4c0 con 0x7ff66006e4b0 2023-06-10T19:00:19.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65f7fe700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e4b0 0x7ff6600c9bc0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7ff6540098f0 tx=0x7ff654009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba88a4b9d7ef71f server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65effd700 1 -- 172.21.15.104:0/657060354 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65400b8a0 con 0x7ff66006e4b0 2023-06-10T19:00:19.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65effd700 1 -- 172.21.15.104:0/657060354 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65400ba00 con 0x7ff66006e4b0 2023-06-10T19:00:19.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.945+0000 7ff65effd700 1 -- 172.21.15.104:0/657060354 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65400bd90 con 0x7ff66006e4b0 2023-06-10T19:00:19.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 -- 172.21.15.104:0/657060354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e4b0 msgr2=0x7ff6600c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e4b0 0x7ff6600c9bc0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7ff6540098f0 tx=0x7ff654009c00 comp rx=0 tx=0).stop 2023-06-10T19:00:19.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 -- 172.21.15.104:0/657060354 shutdown_connections 2023-06-10T19:00:19.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e0e0 0x7ff66006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 --2- 172.21.15.104:0/657060354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e4b0 0x7ff6600c9bc0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 -- 172.21.15.104:0/657060354 >> 172.21.15.104:0/657060354 conn(0x7ff66006b8a0 msgr2=0x7ff6600cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:19.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 -- 172.21.15.104:0/657060354 shutdown_connections 2023-06-10T19:00:19.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.946+0000 7ff6665e1700 1 -- 172.21.15.104:0/657060354 wait complete. 2023-06-10T19:00:19.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.947+0000 7ff6665e1700 1 Processor -- start 2023-06-10T19:00:19.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.947+0000 7ff6665e1700 1 -- start start 2023-06-10T19:00:19.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e4b0 0x7ff6600c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6600ca100 0x7ff6600adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff660154f00 con 0x7ff66006e0e0 2023-06-10T19:00:19.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600c8600 con 0x7ff6600ca100 2023-06-10T19:00:19.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff6665e1700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600c88e0 con 0x7ff66006e4b0 2023-06-10T19:00:19.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42712/0 (socket says 172.21.15.104:42712) 2023-06-10T19:00:19.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 -- 172.21.15.104:0/2092812893 learned_addr learned my addr 172.21.15.104:0/2092812893 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:19.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 -- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e4b0 msgr2=0x7ff6600c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e4b0 0x7ff6600c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 -- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6600ca100 msgr2=0x7ff6600adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:19.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff664b7e700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6600ca100 0x7ff6600adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6600ca100 0x7ff6600adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:19.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.948+0000 7ff65ffff700 1 -- 172.21.15.104:0/2092812893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff654009550 con 0x7ff66006e0e0 2023-06-10T19:00:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff65ffff700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7ff65000d800 tx=0x7ff65000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650018070 con 0x7ff66006e0e0 2023-06-10T19:00:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff650005a20 con 0x7ff66006e0e0 2023-06-10T19:00:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6600c8bc0 con 0x7ff66006e0e0 2023-06-10T19:00:19.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650005bb0 con 0x7ff66006e0e0 2023-06-10T19:00:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.949+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6600c9180 con 0x7ff66006e0e0 2023-06-10T19:00:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.950+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff660070800 con 0x7ff66006e0e0 2023-06-10T19:00:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.950+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff6500125f0 con 0x7ff66006e0e0 2023-06-10T19:00:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.951+0000 7ff65cff9700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 0x7ff64c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:19.967 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.951+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7ff650088f60 con 0x7ff66006e0e0 2023-06-10T19:00:19.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.951+0000 7ff65f7fe700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 0x7ff64c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:19.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.951+0000 7ff65f7fe700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 0x7ff64c06d190 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff6600aeb80 tx=0x7ff654010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:19.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:19.955+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff650057c30 con 0x7ff66006e0e0 2023-06-10T19:00:20.149 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.143+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7ff66006da80 con 0x7ff64c06ace0 2023-06-10T19:00:20.150 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.143+0000 7ff65cff9700 1 -- 172.21.15.104:0/2092812893 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff65401f070 con 0x7ff64c06ace0 2023-06-10T19:00:20.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 msgr2=0x7ff64c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 0x7ff64c06d190 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7ff6600aeb80 tx=0x7ff654010000 comp rx=0 tx=0).stop 2023-06-10T19:00:20.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 msgr2=0x7ff6601549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.152 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7ff65000d800 tx=0x7ff65000dbc0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 shutdown_connections 2023-06-10T19:00:20.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7ff64c06ace0 0x7ff64c06d190 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6600ca100 0x7ff6600adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff66006e0e0 0x7ff6601549c0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.153 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 --2- 172.21.15.104:0/2092812893 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff66006e4b0 0x7ff6600c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 >> 172.21.15.104:0/2092812893 conn(0x7ff66006b8a0 msgr2=0x7ff6600ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 shutdown_connections 2023-06-10T19:00:20.154 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.145+0000 7ff6665e1700 1 -- 172.21.15.104:0/2092812893 wait complete. 2023-06-10T19:00:20.154 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 5 to deep-scrub 2023-06-10T19:00:20.163 INFO:tasks.ceph:Scrubbing osd.6 2023-06-10T19:00:20.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.281+0000 7fb0b17bc700 1 Processor -- start 2023-06-10T19:00:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.281+0000 7fb0b17bc700 1 -- start start 2023-06-10T19:00:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.281+0000 7fb0b17bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.281+0000 7fb0b17bc700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb0ac0c8df0 0x7fb0ac071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0b17bc700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0715e0 con 0x7fb0ac0c9340 2023-06-10T19:00:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0b17bc700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac0cc860 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0b17bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac15c6b0 con 0x7fb0ac0caf90 2023-06-10T19:00:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7fb0ac0c8df0 0x7fb0ac071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0aa7fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb0ac0c9340 0x7fb0ac0cc150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40550/0 (socket says 172.21.15.104:40550) 2023-06-10T19:00:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0aa7fc700 1 -- 172.21.15.104:0/2550455318 learned_addr learned my addr 172.21.15.104:0/2550455318 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0aaffd700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 -- 172.21.15.104:0/2550455318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 msgr2=0x7fb0ac0cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 -- 172.21.15.104:0/2550455318 >> v1:172.21.15.104:6789/0 conn(0x7fb0ac0c9340 legacy=0x7fb0ac0cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 -- 172.21.15.104:0/2550455318 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0cb8a0 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0a9ffb700 1 -- 172.21.15.104:0/2550455318 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837513302 0 0) 0x7fb0ac0715e0 con 0x7fb0ac0c9340 2023-06-10T19:00:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0ab7fe700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac071a70 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009d90 tx=0x7fb0a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe5f798863063cb server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0a9ffb700 1 -- 172.21.15.104:0/2550455318 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400e070 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0a9ffb700 1 -- 172.21.15.104:0/2550455318 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a400bbd0 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.282+0000 7fb0a9ffb700 1 -- 172.21.15.104:0/2550455318 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4012430 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 -- 172.21.15.104:0/2550455318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 msgr2=0x7fb0ac071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac071a70 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009d90 tx=0x7fb0a400b330 comp rx=0 tx=0).stop 2023-06-10T19:00:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 -- 172.21.15.104:0/2550455318 shutdown_connections 2023-06-10T19:00:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac071a70 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/2550455318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 -- 172.21.15.104:0/2550455318 >> 172.21.15.104:0/2550455318 conn(0x7fb0ac06aac0 msgr2=0x7fb0ac0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 -- 172.21.15.104:0/2550455318 shutdown_connections 2023-06-10T19:00:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.283+0000 7fb0b17bc700 1 -- 172.21.15.104:0/2550455318 wait complete. 2023-06-10T19:00:20.297 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.284+0000 7fb0b17bc700 1 Processor -- start 2023-06-10T19:00:20.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.284+0000 7fb0b17bc700 1 -- start start 2023-06-10T19:00:20.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0ac0c9340 0x7fb0ac0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.298 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0cd7a0 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0ced70 con 0x7fb0ac0c9340 2023-06-10T19:00:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0b17bc700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0cd4c0 con 0x7fb0ac0caf90 2023-06-10T19:00:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42726/0 (socket says 172.21.15.104:42726) 2023-06-10T19:00:20.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 -- 172.21.15.104:0/4224843577 learned_addr learned my addr 172.21.15.104:0/4224843577 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aa7fc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0ac0c9340 0x7fb0ac0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 -- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 msgr2=0x7fb0ac0c4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0ab7fe700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0ac0caf90 0x7fb0ac0c4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 -- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0ac0c9340 msgr2=0x7fb0ac0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0ac0c9340 0x7fb0ac0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4009a70 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aa7fc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0ac0c9340 0x7fb0ac0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.285+0000 7fb0aaffd700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac150cf0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d800 tx=0x7fb09c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.301 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0125f0 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c012750 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c012b00 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac0ce720 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac0cf200 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.286+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0ac0cf420 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.287+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb09c012c60 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.287+0000 7fb0937fe700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb09806ace0 0x7fb09806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.287+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb09c08a180 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0937fe700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fb0980715d0 0x7fb098073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0980740d0 con 0x7fb0980715d0 2023-06-10T19:00:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb09c08a4f0 con 0x7fb0ac0c8df0 2023-06-10T19:00:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0aa7fc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb09806ace0 0x7fb09806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0ab7fe700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fb0980715d0 0x7fb098073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0aa7fc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb09806ace0 0x7fb09806d190 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007fd0 tx=0x7fb0a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.288+0000 7fb0ab7fe700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fb0980715d0 0x7fb098073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.289+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== osd.6 v2:172.21.15.114:6816/4101853965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb0980740d0 con 0x7fb0980715d0 2023-06-10T19:00:20.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.303+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 --> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb0ac0add20 con 0x7fb0980715d0 2023-06-10T19:00:20.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.303+0000 7fb0937fe700 1 -- 172.21.15.104:0/4224843577 <== osd.6 v2:172.21.15.114:6816/4101853965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb0a40098f0 con 0x7fb0980715d0 2023-06-10T19:00:20.310 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:20.310 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:20.311 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:20.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fb0980715d0 msgr2=0x7fb098073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6816/4101853965,v1:172.21.15.114:6818/4101853965] conn(0x7fb0980715d0 0x7fb098073a00 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb09806ace0 msgr2=0x7fb09806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fb09806ace0 0x7fb09806d190 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007fd0 tx=0x7fb0a0009040 comp rx=0 tx=0).stop 2023-06-10T19:00:20.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 msgr2=0x7fb0ac150cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 --2- 172.21.15.104:0/4224843577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0ac0c8df0 0x7fb0ac150cf0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d800 tx=0x7fb09c00dbc0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0ab7fe700 1 -- 172.21.15.104:0/4224843577 reap_dead start 2023-06-10T19:00:20.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 shutdown_connections 2023-06-10T19:00:20.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 >> 172.21.15.104:0/4224843577 conn(0x7fb0ac06aac0 msgr2=0x7fb0ac156de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 shutdown_connections 2023-06-10T19:00:20.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.304+0000 7fb0b17bc700 1 -- 172.21.15.104:0/4224843577 wait complete. 2023-06-10T19:00:20.322 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-06-10T19:00:20.442 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.436+0000 7fd908065700 1 Processor -- start 2023-06-10T19:00:20.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.436+0000 7fd908065700 1 -- start start 2023-06-10T19:00:20.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd908065700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd900070a20 0x7fd90006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd908065700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90006ae00 con 0x7fd900070df0 2023-06-10T19:00:20.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd908065700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd900069e70 con 0x7fd900158eb0 2023-06-10T19:00:20.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd908065700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9001595f0 con 0x7fd900070a20 2023-06-10T19:00:20.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd905600700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd900070df0 0x7fd9001587a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40566/0 (socket says 172.21.15.104:40566) 2023-06-10T19:00:20.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd905600700 1 -- 172.21.15.104:0/2607559499 learned_addr learned my addr 172.21.15.104:0/2607559499 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:20.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> v2:172.21.15.114:3300/0 conn(0x7fd900070a20 0x7fd90006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860990297 0 0) 0x7fd8f0001110 con 0x7fd900070df0 2023-06-10T19:00:20.446 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ec0035b0 con 0x7fd900070df0 2023-06-10T19:00:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.437+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077615500 0 0) 0x7fd8ec0035b0 con 0x7fd900070df0 2023-06-10T19:00:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 >> v2:172.21.15.114:3300/0 conn(0x7fd900070a20 msgr2=0x7fd90006a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.450 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 --2- 172.21.15.104:0/2607559499 >> v2:172.21.15.114:3300/0 conn(0x7fd900070a20 0x7fd90006a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 >> v1:172.21.15.114:6789/0 conn(0x7fd900158eb0 legacy=0x7fd900159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9001598d0 con 0x7fd900070df0 2023-06-10T19:00:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fd8f0001800 con 0x7fd900070df0 2023-06-10T19:00:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> v2:172.21.15.114:3300/0 conn(0x7fd900070a20 0x7fd90006a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-10T19:00:20.451 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 >> v1:172.21.15.104:6789/0 conn(0x7fd900070df0 legacy=0x7fd9001587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 0x7fd8ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8ec004850 0x7fd8ec004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec005200 0x7fd8ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.452 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9001595f0 con 0x7fd8ec003a00 2023-06-10T19:00:20.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec00c080 con 0x7fd8ec005200 2023-06-10T19:00:20.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec00c240 con 0x7fd8ec004850 2023-06-10T19:00:20.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 0x7fd8ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905600700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8ec004850 0x7fd8ec004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.453 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd906602700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec005200 0x7fd8ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 -- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8ec004850 msgr2=0x7fd8ec004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8ec004850 0x7fd8ec004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 -- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec005200 msgr2=0x7fd8ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec005200 0x7fd8ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.454 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd906602700 1 -- 172.21.15.104:0/2607559499 reap_dead start 2023-06-10T19:00:20.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 -- 172.21.15.104:0/2607559499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0035b0 con 0x7fd8ec003a00 2023-06-10T19:00:20.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd905e01700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 0x7fd8ec003e10 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009d30 tx=0x7fd8ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc0dedef7cf573d server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc009e80 con 0x7fd8ec003a00 2023-06-10T19:00:20.455 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc00e9a0 con 0x7fd8ec003a00 2023-06-10T19:00:20.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.438+0000 7fd904dff700 1 -- 172.21.15.104:0/2607559499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00ed10 con 0x7fd8ec003a00 2023-06-10T19:00:20.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 -- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 msgr2=0x7fd8ec003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 0x7fd8ec003e10 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009d30 tx=0x7fd8ec005880 comp rx=0 tx=0).stop 2023-06-10T19:00:20.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 -- 172.21.15.104:0/2607559499 shutdown_connections 2023-06-10T19:00:20.456 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 --2- 172.21.15.104:0/2607559499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8ec003a00 0x7fd8ec003e10 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 -- 172.21.15.104:0/2607559499 >> 172.21.15.104:0/2607559499 conn(0x7fd9000b1610 msgr2=0x7fd9000ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 -- 172.21.15.104:0/2607559499 shutdown_connections 2023-06-10T19:00:20.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.439+0000 7fd908065700 1 -- 172.21.15.104:0/2607559499 wait complete. 2023-06-10T19:00:20.457 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.441+0000 7fd908065700 1 Processor -- start 2023-06-10T19:00:20.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.441+0000 7fd908065700 1 -- start start 2023-06-10T19:00:20.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 0x7fd9001652d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9000aed80 0x7fd9001660e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.458 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd900166a90 con 0x7fd900166620 2023-06-10T19:00:20.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000c1110 con 0x7fd8ec003a00 2023-06-10T19:00:20.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd900166cb0 con 0x7fd9000aed80 2023-06-10T19:00:20.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42752/0 (socket says 172.21.15.104:42752) 2023-06-10T19:00:20.459 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 -- 172.21.15.104:0/156671562 learned_addr learned my addr 172.21.15.104:0/156671562 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:20.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd905600700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9000aed80 0x7fd9001660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd905e01700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 0x7fd9001652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 -- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9000aed80 msgr2=0x7fd9001660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9000aed80 0x7fd9001660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.460 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 -- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 msgr2=0x7fd9001652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 0x7fd9001652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 -- 172.21.15.104:0/156671562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000c13d0 con 0x7fd900166620 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd905e01700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 0x7fd9001652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd906602700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fd900072610 tx=0x7fd8f800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8003cd0 con 0x7fd900166620 2023-06-10T19:00:20.461 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f8003e70 con 0x7fd900166620 2023-06-10T19:00:20.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd900166f90 con 0x7fd900166620 2023-06-10T19:00:20.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.442+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800d9f0 con 0x7fd900166620 2023-06-10T19:00:20.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.443+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9000c87b0 con 0x7fd900166620 2023-06-10T19:00:20.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.444+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd900071780 con 0x7fd900166620 2023-06-10T19:00:20.462 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.444+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd8f800db50 con 0x7fd900166620 2023-06-10T19:00:20.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.444+0000 7fd8f67fc700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 0x7fd8e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.444+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd8f80894e0 con 0x7fd900166620 2023-06-10T19:00:20.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.444+0000 7fd905e01700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 0x7fd8e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.445+0000 7fd905e01700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 0x7fd8e406d190 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0090b0 tx=0x7fd8fc01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.464 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.449+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd8f80547b0 con 0x7fd900166620 2023-06-10T19:00:20.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.637+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fd900166e20 con 0x7fd8e406ace0 2023-06-10T19:00:20.644 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.638+0000 7fd8f67fc700 1 -- 172.21.15.104:0/156671562 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd900166e20 con 0x7fd8e406ace0 2023-06-10T19:00:20.646 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 msgr2=0x7fd8e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 0x7fd8e406d190 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0090b0 tx=0x7fd8fc01e040 comp rx=0 tx=0).stop 2023-06-10T19:00:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 msgr2=0x7fd900142c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fd900072610 tx=0x7fd8f800d150 comp rx=0 tx=0).stop 2023-06-10T19:00:20.647 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 shutdown_connections 2023-06-10T19:00:20.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fd8e406ace0 0x7fd8e406d190 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8ec003a00 0x7fd9001652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd900166620 0x7fd900142c80 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 --2- 172.21.15.104:0/156671562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9000aed80 0x7fd9001660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.648 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 >> 172.21.15.104:0/156671562 conn(0x7fd9000b1610 msgr2=0x7fd9000afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 shutdown_connections 2023-06-10T19:00:20.649 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.640+0000 7fd908065700 1 -- 172.21.15.104:0/156671562 wait complete. 2023-06-10T19:00:20.649 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 6 to deep-scrub 2023-06-10T19:00:20.658 INFO:tasks.ceph:Scrubbing osd.7 2023-06-10T19:00:20.659 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-06-10T19:00:20.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.776+0000 7f82d8038700 1 Processor -- start 2023-06-10T19:00:20.783 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.777+0000 7f82d8038700 1 -- start start 2023-06-10T19:00:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.777+0000 7f82d8038700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82d0069e70 0x7f82d006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.785 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.777+0000 7f82d8038700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00ce040 con 0x7f82d015c500 2023-06-10T19:00:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.777+0000 7f82d8038700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d015c8d0 con 0x7f82d006dcd0 2023-06-10T19:00:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.777+0000 7f82d8038700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d00cf4d0 con 0x7f82d0069e70 2023-06-10T19:00:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d5dd4700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f82d015c500 0x7f82d00cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40584/0 (socket says 172.21.15.104:40584) 2023-06-10T19:00:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d5dd4700 1 -- 172.21.15.104:0/3782474861 learned_addr learned my addr 172.21.15.104:0/3782474861 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:20.786 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d55d3700 1 --2- 172.21.15.104:0/3782474861 >> v2:172.21.15.114:3300/0 conn(0x7f82d0069e70 0x7f82d006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822250629 0 0) 0x7f82d00ce040 con 0x7f82d015c500 2023-06-10T19:00:20.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0035b0 con 0x7f82d015c500 2023-06-10T19:00:20.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892468800 0 0) 0x7f82bc0035b0 con 0x7f82d015c500 2023-06-10T19:00:20.787 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d55d3700 1 -- 172.21.15.104:0/3782474861 >> v1:172.21.15.114:6789/0 conn(0x7f82d006dcd0 legacy=0x7f82d006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d55d3700 1 -- 172.21.15.104:0/3782474861 >> v1:172.21.15.104:6789/0 conn(0x7f82d015c500 legacy=0x7f82d00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d55d3700 1 -- 172.21.15.104:0/3782474861 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d006da40 con 0x7f82d0069e70 2023-06-10T19:00:20.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d55d3700 1 --2- 172.21.15.104:0/3782474861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d006a2e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009840 tx=0x7f82cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=642384761f0e5b12 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c00017f0 con 0x7f82d0069e70 2023-06-10T19:00:20.788 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc004030 con 0x7f82d0069e70 2023-06-10T19:00:20.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.778+0000 7f82d4dd2700 1 -- 172.21.15.104:0/3782474861 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0043a0 con 0x7f82d0069e70 2023-06-10T19:00:20.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 -- 172.21.15.104:0/3782474861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 msgr2=0x7f82d006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 --2- 172.21.15.104:0/3782474861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d006a2e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009840 tx=0x7f82cc009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:20.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 -- 172.21.15.104:0/3782474861 shutdown_connections 2023-06-10T19:00:20.789 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 --2- 172.21.15.104:0/3782474861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d006a2e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 -- 172.21.15.104:0/3782474861 >> 172.21.15.104:0/3782474861 conn(0x7f82d006b8a0 msgr2=0x7f82d00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 -- 172.21.15.104:0/3782474861 shutdown_connections 2023-06-10T19:00:20.790 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.779+0000 7f82d8038700 1 -- 172.21.15.104:0/3782474861 wait complete. 2023-06-10T19:00:20.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.780+0000 7f82d8038700 1 Processor -- start 2023-06-10T19:00:20.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.780+0000 7f82d8038700 1 -- start start 2023-06-10T19:00:20.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d00c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.793 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 0x7f82d01f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d015c500 0x7f82d01548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d01f2860 con 0x7f82d006dcd0 2023-06-10T19:00:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00c6ff0 con 0x7f82d0069e70 2023-06-10T19:00:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d8038700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00c72d0 con 0x7f82d015c500 2023-06-10T19:00:20.794 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 0x7f82d01f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 0x7f82d01f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42764/0 (socket says 172.21.15.104:42764) 2023-06-10T19:00:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 -- 172.21.15.104:0/1613364241 learned_addr learned my addr 172.21.15.104:0/1613364241 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d5dd4700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d00c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d65d5700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d015c500 0x7f82d01548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.795 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 -- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d015c500 msgr2=0x7f82d01548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d015c500 0x7f82d01548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 -- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 msgr2=0x7f82d00c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d0069e70 0x7f82d00c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.781+0000 7f82d55d3700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc0094a0 con 0x7f82d006dcd0 2023-06-10T19:00:20.796 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82d55d3700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 0x7f82d01f2320 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f82cc000c00 tx=0x7f82cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc003df0 con 0x7f82d006dcd0 2023-06-10T19:00:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc01f070 con 0x7f82d006dcd0 2023-06-10T19:00:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc011870 con 0x7f82d006dcd0 2023-06-10T19:00:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d00c75b0 con 0x7f82d006dcd0 2023-06-10T19:00:20.797 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.782+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d0154e10 con 0x7f82d006dcd0 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.783+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82d00ce2e0 con 0x7f82d006dcd0 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.783+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f82cc021030 con 0x7f82d006dcd0 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.783+0000 7f82c67fc700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82b406ace0 0x7f82b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f82cc08a700 con 0x7f82d006dcd0 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82d5dd4700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82b406ace0 0x7f82b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.798 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82c67fc700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f82b40715d0 0x7f82b4073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82b40740d0 con 0x7f82b40715d0 2023-06-10T19:00:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=191) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82cc08cd60 con 0x7f82d006dcd0 2023-06-10T19:00:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82d65d5700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f82b40715d0 0x7f82b4073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82d5dd4700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82b406ace0 0x7f82b406d190 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f82c0007fd0 tx=0x7f82c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.799 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.784+0000 7f82d65d5700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f82b40715d0 0x7f82b4073a00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.800 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.785+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== osd.7 v2:172.21.15.114:6808/3766591503 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f82b40740d0 con 0x7f82b40715d0 2023-06-10T19:00:20.805 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.799+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 --> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f82d006a9c0 con 0x7f82b40715d0 2023-06-10T19:00:20.806 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82c67fc700 1 -- 172.21.15.104:0/1613364241 <== osd.7 v2:172.21.15.114:6808/3766591503 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f82c8012380 con 0x7f82b40715d0 2023-06-10T19:00:20.806 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-06-10T19:00:20.806 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-06-10T19:00:20.807 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-06-10T19:00:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f82b40715d0 msgr2=0x7f82b4073a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6808/3766591503,v1:172.21.15.114:6809/3766591503] conn(0x7f82b40715d0 0x7f82b4073a00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82b406ace0 msgr2=0x7f82b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.808 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f82b406ace0 0x7f82b406d190 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f82c0007fd0 tx=0x7f82c0009040 comp rx=0 tx=0).stop 2023-06-10T19:00:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 msgr2=0x7f82d01f2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 --2- 172.21.15.104:0/1613364241 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d006dcd0 0x7f82d01f2320 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f82cc000c00 tx=0x7f82cc004750 comp rx=0 tx=0).stop 2023-06-10T19:00:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d65d5700 1 -- 172.21.15.104:0/1613364241 reap_dead start 2023-06-10T19:00:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 shutdown_connections 2023-06-10T19:00:20.809 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 >> 172.21.15.104:0/1613364241 conn(0x7f82d006b8a0 msgr2=0x7f82d0072840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 shutdown_connections 2023-06-10T19:00:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.800+0000 7f82d8038700 1 -- 172.21.15.104:0/1613364241 wait complete. 2023-06-10T19:00:20.818 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-06-10T19:00:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.932+0000 7f0c669b5700 1 Processor -- start 2023-06-10T19:00:20.939 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 -- start start 2023-06-10T19:00:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c600cd850 0x7f0c6006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c6006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600c9e60 con 0x7f0c600aeaa0 2023-06-10T19:00:20.942 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c600c9bc0 con 0x7f0c600cd850 2023-06-10T19:00:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c669b5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600aee70 con 0x7f0c600cb300 2023-06-10T19:00:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c5ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0c600aeaa0 0x7f0c600cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40600/0 (socket says 172.21.15.104:40600) 2023-06-10T19:00:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.933+0000 7f0c5ffff700 1 -- 172.21.15.104:0/3066138703 learned_addr learned my addr 172.21.15.104:0/3066138703 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:20.943 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5f7fe700 1 --2- 172.21.15.104:0/3066138703 >> v2:172.21.15.114:3300/0 conn(0x7f0c600cd850 0x7f0c6006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c6006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122173108 0 0) 0x7f0c600c9e60 con 0x7f0c600aeaa0 2023-06-10T19:00:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c480035b0 con 0x7f0c600aeaa0 2023-06-10T19:00:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521363339 0 0) 0x7f0c480035b0 con 0x7f0c600aeaa0 2023-06-10T19:00:20.944 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 msgr2=0x7f0c6006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c6006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 >> v2:172.21.15.114:3300/0 conn(0x7f0c600cd850 msgr2=0x7f0c6006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 --2- 172.21.15.104:0/3066138703 >> v2:172.21.15.114:3300/0 conn(0x7f0c600cd850 0x7f0c6006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6006e540 con 0x7f0c600aeaa0 2023-06-10T19:00:20.945 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f0c500017f0 con 0x7f0c600aeaa0 2023-06-10T19:00:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 >> v1:172.21.15.104:6789/0 conn(0x7f0c600aeaa0 legacy=0x7f0c600cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c48003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c48003f80 0x7f0c48004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.946 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c48004d90 0x7f0c48005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600c9bc0 con 0x7f0c48004d90 2023-06-10T19:00:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600aee70 con 0x7f0c48003f80 2023-06-10T19:00:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4800ba10 con 0x7f0c600cb300 2023-06-10T19:00:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c48004d90 0x7f0c48005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.947 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5f7fe700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c48003f80 0x7f0c48004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c5ffff700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c48003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 -- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 msgr2=0x7f0c48003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c600cb300 0x7f0c48003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 -- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c48003f80 msgr2=0x7f0c48004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:20.948 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c48003f80 0x7f0c48004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.934+0000 7f0c64f52700 1 -- 172.21.15.104:0/3066138703 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c480035b0 con 0x7f0c48004d90 2023-06-10T19:00:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c64f52700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c48004d90 0x7f0c48005200 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009fc0 tx=0x7f0c48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a3fd92002bcbc3 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800ec30 con 0x7f0c48004d90 2023-06-10T19:00:20.949 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5800f040 con 0x7f0c48004d90 2023-06-10T19:00:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c5effd700 1 -- 172.21.15.104:0/3066138703 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58014480 con 0x7f0c48004d90 2023-06-10T19:00:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 -- 172.21.15.104:0/3066138703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c48004d90 msgr2=0x7f0c48005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 --2- 172.21.15.104:0/3066138703 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c48004d90 0x7f0c48005200 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009fc0 tx=0x7f0c48005880 comp rx=0 tx=0).stop 2023-06-10T19:00:20.950 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c64f52700 1 -- 172.21.15.104:0/3066138703 reap_dead start 2023-06-10T19:00:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 -- 172.21.15.104:0/3066138703 shutdown_connections 2023-06-10T19:00:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 -- 172.21.15.104:0/3066138703 >> 172.21.15.104:0/3066138703 conn(0x7f0c6006b8a0 msgr2=0x7f0c600cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 -- 172.21.15.104:0/3066138703 shutdown_connections 2023-06-10T19:00:20.951 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.935+0000 7f0c669b5700 1 -- 172.21.15.104:0/3066138703 wait complete. 2023-06-10T19:00:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.936+0000 7f0c669b5700 1 Processor -- start 2023-06-10T19:00:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 -- start start 2023-06-10T19:00:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c48004d90 0x7f0c600c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.954 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60150b00 0x7f0c600ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600c6b80 con 0x7f0c600c9280 2023-06-10T19:00:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c601f6000 con 0x7f0c60150b00 2023-06-10T19:00:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c669b5700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600c8df0 con 0x7f0c48004d90 2023-06-10T19:00:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.937+0000 7f0c5f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.955 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42786/0 (socket says 172.21.15.104:42786) 2023-06-10T19:00:20.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 -- 172.21.15.104:0/40636985 learned_addr learned my addr 172.21.15.104:0/40636985 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:20.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c64f52700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60150b00 0x7f0c600ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5ffff700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c48004d90 0x7f0c600c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 -- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c48004d90 msgr2=0x7f0c600c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.956 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c48004d90 0x7f0c600c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 -- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60150b00 msgr2=0x7f0c600ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:20.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60150b00 0x7f0c600ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:20.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 -- 172.21.15.104:0/40636985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600c6d70 con 0x7f0c600c9280 2023-06-10T19:00:20.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5f7fe700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f0c54011aa0 tx=0x7f0c54011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.957 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54011ee0 con 0x7f0c600c9280 2023-06-10T19:00:20.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5400f8e0 con 0x7f0c600c9280 2023-06-10T19:00:20.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5400fcb0 con 0x7f0c600c9280 2023-06-10T19:00:20.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c600c84c0 con 0x7f0c600c9280 2023-06-10T19:00:20.958 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.938+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c600c8a80 con 0x7f0c600c9280 2023-06-10T19:00:20.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.939+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c60074050 con 0x7f0c600c9280 2023-06-10T19:00:20.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.939+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0c54014440 con 0x7f0c600c9280 2023-06-10T19:00:20.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.940+0000 7f0c5cff9700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 0x7f0c4c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:20.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.940+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f0c54089940 con 0x7f0c600c9280 2023-06-10T19:00:20.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.940+0000 7f0c5ffff700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 0x7f0c4c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:20.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.940+0000 7f0c5ffff700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 0x7f0c4c06d180 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000ef0 tx=0x7f0c50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:20.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:20.944+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0c54058760 con 0x7f0c600c9280 2023-06-10T19:00:21.135 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.129+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c6006eb00 con 0x7f0c4c06acd0 2023-06-10T19:00:21.136 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.129+0000 7f0c5cff9700 1 -- 172.21.15.104:0/40636985 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0c50007e00 con 0x7f0c4c06acd0 2023-06-10T19:00:21.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 msgr2=0x7f0c4c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.138 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 0x7f0c4c06d180 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000ef0 tx=0x7f0c50009040 comp rx=0 tx=0).stop 2023-06-10T19:00:21.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 msgr2=0x7f0c601f5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f0c54011aa0 tx=0x7f0c54011db0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 shutdown_connections 2023-06-10T19:00:21.139 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f0c4c06acd0 0x7f0c4c06d180 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60150b00 0x7f0c600ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c48004d90 0x7f0c600c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 --2- 172.21.15.104:0/40636985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c600c9280 0x7f0c601f5ac0 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 >> 172.21.15.104:0/40636985 conn(0x7f0c6006b8a0 msgr2=0x7f0c600cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:21.140 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 shutdown_connections 2023-06-10T19:00:21.141 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.132+0000 7f0c669b5700 1 -- 172.21.15.104:0/40636985 wait complete. 2023-06-10T19:00:21.141 INFO:teuthology.orchestra.run.smithi104.stderr:instructed osd(s) 7 to deep-scrub 2023-06-10T19:00:21.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T19:00:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 Processor -- start 2023-06-10T19:00:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 -- start start 2023-06-10T19:00:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc5500b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.272 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5500b4ff0 0x7fc5500ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5500b6070 con 0x7fc5500ce890 2023-06-10T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5500ced10 con 0x7fc5500b4ff0 2023-06-10T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.263+0000 7fc555bee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500b5460 con 0x7fc55015a420 2023-06-10T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54ffff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc5500ce890 0x7fc5500b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40624/0 (socket says 172.21.15.104:40624) 2023-06-10T19:00:21.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54ffff700 1 -- 172.21.15.104:0/4184480598 learned_addr learned my addr 172.21.15.104:0/4184480598 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54f7fe700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc5500b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54effd700 1 --2- 172.21.15.104:0/4184480598 >> v2:172.21.15.114:3300/0 conn(0x7fc5500b4ff0 0x7fc5500ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507178152 0 0) 0x7fc5500b6070 con 0x7fc5500ce890 2023-06-10T19:00:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5380035b0 con 0x7fc5500ce890 2023-06-10T19:00:21.274 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2286964241 0 0) 0x7fc5380035b0 con 0x7fc5500ce890 2023-06-10T19:00:21.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 msgr2=0x7fc5500b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:21.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc5500b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 >> v2:172.21.15.114:3300/0 conn(0x7fc5500b4ff0 msgr2=0x7fc5500ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:21.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 --2- 172.21.15.104:0/4184480598 >> v2:172.21.15.114:3300/0 conn(0x7fc5500b4ff0 0x7fc5500ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500b5740 con 0x7fc5500ce890 2023-06-10T19:00:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7fc5480019f0 con 0x7fc5500ce890 2023-06-10T19:00:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 >> v1:172.21.15.104:6789/0 conn(0x7fc5500ce890 legacy=0x7fc5500b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc538003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc538003f80 0x7fc538004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc538004d90 0x7fc538005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500ced10 con 0x7fc538004d90 2023-06-10T19:00:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500b5460 con 0x7fc538003f80 2023-06-10T19:00:21.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53800ba10 con 0x7fc55015a420 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54ffff700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc538004d90 0x7fc538005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54f7fe700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc538003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.264+0000 7fc54effd700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc538003f80 0x7fc538004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 -- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 msgr2=0x7fc538003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55015a420 0x7fc538003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.278 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 -- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc538003f80 msgr2=0x7fc538004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc538003f80 0x7fc538004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 -- 172.21.15.104:0/4184480598 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5380035b0 con 0x7fc538004d90 2023-06-10T19:00:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc538004d90 0x7fc538005200 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc54800a8d0 tx=0x7fc538005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f0540be07dd647 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54800a340 con 0x7fc538004d90 2023-06-10T19:00:21.279 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc548012600 con 0x7fc538004d90 2023-06-10T19:00:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54e7fc700 1 -- 172.21.15.104:0/4184480598 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5480129b0 con 0x7fc538004d90 2023-06-10T19:00:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 -- 172.21.15.104:0/4184480598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc538004d90 msgr2=0x7fc538005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 --2- 172.21.15.104:0/4184480598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc538004d90 0x7fc538005200 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc54800a8d0 tx=0x7fc538005880 comp rx=0 tx=0).stop 2023-06-10T19:00:21.280 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc54ffff700 1 -- 172.21.15.104:0/4184480598 reap_dead start 2023-06-10T19:00:21.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 -- 172.21.15.104:0/4184480598 shutdown_connections 2023-06-10T19:00:21.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 -- 172.21.15.104:0/4184480598 >> 172.21.15.104:0/4184480598 conn(0x7fc550089fd0 msgr2=0x7fc5500d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:21.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 -- 172.21.15.104:0/4184480598 shutdown_connections 2023-06-10T19:00:21.281 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.265+0000 7fc555bee700 1 -- 172.21.15.104:0/4184480598 wait complete. 2023-06-10T19:00:21.283 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 Processor -- start 2023-06-10T19:00:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 -- start start 2023-06-10T19:00:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 0x7fc5501e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5501e1150 0x7fc5500c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.284 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5501e15f0 con 0x7fc550146b80 2023-06-10T19:00:21.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc550148050 con 0x7fc550146f50 2023-06-10T19:00:21.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.267+0000 7fc555bee700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc550147440 con 0x7fc5501e1150 2023-06-10T19:00:21.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.285 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42804/0 (socket says 172.21.15.104:42804) 2023-06-10T19:00:21.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 -- 172.21.15.104:0/3061909532 learned_addr learned my addr 172.21.15.104:0/3061909532 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:21.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54effd700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 0x7fc5501e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54ffff700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5501e1150 0x7fc5500c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 -- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5501e1150 msgr2=0x7fc5500c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.286 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5501e1150 0x7fc5500c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 -- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 msgr2=0x7fc5501e0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 0x7fc5501e0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 -- 172.21.15.104:0/3061909532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550147720 con 0x7fc550146b80 2023-06-10T19:00:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54effd700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 0x7fc5501e0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:21.287 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc54f7fe700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fc540011ad0 tx=0x7fc540011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:21.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54000fa00 con 0x7fc550146b80 2023-06-10T19:00:21.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54000fb60 con 0x7fc550146b80 2023-06-10T19:00:21.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc550147a00 con 0x7fc550146b80 2023-06-10T19:00:21.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540014440 con 0x7fc550146b80 2023-06-10T19:00:21.288 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.268+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55009e2f0 con 0x7fc550146b80 2023-06-10T19:00:21.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.269+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc550074050 con 0x7fc550146b80 2023-06-10T19:00:21.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.270+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc5400145a0 con 0x7fc550146b80 2023-06-10T19:00:21.289 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.270+0000 7fc537fff700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 0x7fc53c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:21.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.270+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc540089b20 con 0x7fc550146b80 2023-06-10T19:00:21.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.270+0000 7fc54effd700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 0x7fc53c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:21.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.271+0000 7fc54effd700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 0x7fc53c06d190 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc55014b990 tx=0x7fc544008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:21.290 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.274+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc540054e20 con 0x7fc550146b80 2023-06-10T19:00:21.463 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.457+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5500b5d00 con 0x7fc53c06ace0 2023-06-10T19:00:21.466 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.461+0000 7fc537fff700 1 -- 172.21.15.104:0/3061909532 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72940 (secure 0 0 0) 0x7fc5440137a0 con 0x7fc53c06ace0 2023-06-10T19:00:21.467 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:21.468 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 msgr2=0x7fc53c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 0x7fc53c06d190 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc55014b990 tx=0x7fc544008040 comp rx=0 tx=0).stop 2023-06-10T19:00:21.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 msgr2=0x7fc55009ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:21.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fc540011ad0 tx=0x7fc540011e90 comp rx=0 tx=0).stop 2023-06-10T19:00:21.469 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 shutdown_connections 2023-06-10T19:00:21.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5501e1150 0x7fc5500c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc550146b80 0x7fc55009ddb0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7fc53c06ace0 0x7fc53c06d190 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 --2- 172.21.15.104:0/3061909532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc550146f50 0x7fc5501e0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:21.470 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 >> 172.21.15.104:0/3061909532 conn(0x7fc550089fd0 msgr2=0x7fc55006ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:21.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 shutdown_connections 2023-06-10T19:00:21.471 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:21.462+0000 7fc555bee700 1 -- 172.21.15.104:0/3061909532 wait complete. 2023-06-10T19:00:21.471 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T19:00:21.480 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2023-06-10T19:00:19.607973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":268344,"kb_used_data":19960,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706376,"statfs":{"total":773094113280,"available":772819329024,"internally_reserved":0,"allocated":20439040,"data_stored":10655556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12945,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002056"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":310,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:12.805802+0000","last_change":"2023-06-10T18:44:03.803537+0000","last_active":"2023-06-10T19:00:12.805802+0000","last_peered":"2023-06-10T19:00:12.805802+0000","last_clean":"2023-06-10T19:00:12.805802+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T19:00:12.805802+0000","last_undegraded":"2023-06-10T19:00:12.805802+0000","last_fullsized":"2023-06-10T19:00:12.805802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:25:47.131230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":352,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:12.804866+0000","last_change":"2023-06-10T18:44:03.538361+0000","last_active":"2023-06-10T19:00:12.804866+0000","last_peered":"2023-06-10T19:00:12.804866+0000","last_clean":"2023-06-10T19:00:12.804866+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T19:00:12.804866+0000","last_undegraded":"2023-06-10T19:00:12.804866+0000","last_fullsized":"2023-06-10T19:00:12.804866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:44:47.612673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":304,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:12.818709+0000","last_change":"2023-06-10T18:44:03.540946+0000","last_active":"2023-06-10T19:00:12.818709+0000","last_peered":"2023-06-10T19:00:12.818709+0000","last_clean":"2023-06-10T19:00:12.818709+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T19:00:12.818709+0000","last_undegraded":"2023-06-10T19:00:12.818709+0000","last_fullsized":"2023-06-10T19:00:12.818709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T22:04:36.391372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":352,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:12.805193+0000","last_change":"2023-06-10T18:44:03.541697+0000","last_active":"2023-06-10T19:00:12.805193+0000","last_peered":"2023-06-10T19:00:12.805193+0000","last_clean":"2023-06-10T19:00:12.805193+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T19:00:12.805193+0000","last_undegraded":"2023-06-10T19:00:12.805193+0000","last_fullsized":"2023-06-10T19:00:12.805193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:00:39.680886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":479,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:13.648928+0000","last_change":"2023-06-10T18:43:57.403480+0000","last_active":"2023-06-10T19:00:13.648928+0000","last_peered":"2023-06-10T19:00:13.648928+0000","last_clean":"2023-06-10T19:00:13.648928+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T19:00:13.648928+0000","last_undegraded":"2023-06-10T19:00:13.648928+0000","last_fullsized":"2023-06-10T19:00:13.648928+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:43:56.365182+0000","last_clean_scrub_stamp":"2023-06-10T18:43:56.365182+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:45:28.025412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":323,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:13.651182+0000","last_change":"2023-06-10T18:44:03.540651+0000","last_active":"2023-06-10T19:00:13.651182+0000","last_peered":"2023-06-10T19:00:13.651182+0000","last_clean":"2023-06-10T19:00:13.651182+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T19:00:13.651182+0000","last_undegraded":"2023-06-10T19:00:13.651182+0000","last_fullsized":"2023-06-10T19:00:13.651182+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:48:23.250736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":322,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:13.653370+0000","last_change":"2023-06-10T18:44:03.803568+0000","last_active":"2023-06-10T19:00:13.653370+0000","last_peered":"2023-06-10T19:00:13.653370+0000","last_clean":"2023-06-10T19:00:13.653370+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T19:00:13.653370+0000","last_undegraded":"2023-06-10T19:00:13.653370+0000","last_fullsized":"2023-06-10T19:00:13.653370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T19:02:27.276578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":310,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:12.804384+0000","last_change":"2023-06-10T18:44:03.541656+0000","last_active":"2023-06-10T19:00:12.804384+0000","last_peered":"2023-06-10T19:00:12.804384+0000","last_clean":"2023-06-10T19:00:12.804384+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T19:00:12.804384+0000","last_undegraded":"2023-06-10T19:00:12.804384+0000","last_fullsized":"2023-06-10T19:00:12.804384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T21:32:07.320350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":300,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:13.653256+0000","last_change":"2023-06-10T18:44:03.542336+0000","last_active":"2023-06-10T19:00:13.653256+0000","last_peered":"2023-06-10T19:00:13.653256+0000","last_clean":"2023-06-10T19:00:13.653256+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T19:00:13.653256+0000","last_undegraded":"2023-06-10T19:00:13.653256+0000","last_fullsized":"2023-06-10T19:00:13.653256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T18:44:02.520027+0000","last_clean_scrub_stamp":"2023-06-10T18:44:02.520027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T02:02:23.427331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33980,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337860,"statfs":{"total":96636764160,"available":96601968640,"internally_reserved":0,"allocated":3002368,"data_stored":1775230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.372,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.749,"15min":0.81499999999999995},"last":0.443},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.47399999999999998,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.69599999999999995,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.86099999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.77300000000000002,"15min":0.81200000000000006},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":0.70799999999999996,"5min":0.72599999999999998,"15min":0.81999999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.27300000000000002},"max":{"1min":0.70599999999999996,"5min":0.745,"15min":0.79000000000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.91700000000000004},"last":0.41399999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":0.71399999999999997,"15min":0.89700000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66400000000000003,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.378}]},{"osd":5,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29399999999999998},"max":{"1min":0.61099999999999999,"5min":0.80000000000000004,"15min":0.91300000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59599999999999997,"5min":0.753,"15min":0.96499999999999997},"last":0.32500000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.40000000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.93899999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.30299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575049,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.42499999999999999,"5min":0.30499999999999999,"15min":0.24399999999999999},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.503}]},{"osd":1,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.80600000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.621,"5min":0.84899999999999998,"15min":9.6850000000000005},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.74099999999999999,"5min":0.872,"15min":6.6689999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.68200000000000005,"5min":0.78800000000000003,"15min":1.252},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34799999999999998},"max":{"1min":0.72399999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64100000000000001,"5min":0.874,"15min":0.874},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.25,"15min":0.25},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.89600000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Jun 10 18:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.36299999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.92700000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.51200000000000001,"5min":0.53900000000000003,"15min":0.54100000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.53400000000000003,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.245,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51600000000000001,"5min":0.54000000000000004,"15min":0.64500000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.374},"min":{"1min":0.28899999999999998,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.54900000000000004,"5min":0.60999999999999999,"15min":0.61099999999999999},"last":0.316}]},{"osd":3,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.505,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.39200000000000002},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.24099999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":1.1040000000000001},"last":0.29899999999999999}]},{"osd":4,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.88500000000000001,"15min":1.1499999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.40500000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77800000000000002,"5min":0.89400000000000002,"15min":0.96699999999999997},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46000000000000002,"5min":0.372,"15min":0.34799999999999998},"max":{"1min":0.80200000000000005,"5min":0.85099999999999998,"15min":0.89900000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.48199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.85999999999999999,"15min":0.871},"last":0.499}]},{"osd":6,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.34999999999999998},"max":{"1min":0.66600000000000004,"5min":0.85199999999999998,"15min":1.206},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.75,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.45000000000000001}]},{"osd":7,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41099999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.88400000000000001,"15min":1.224},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.91100000000000003},"last":0.57899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575048,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33976,"kb_used_data":2928,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337864,"statfs":{"total":96636764160,"available":96601972736,"internally_reserved":0,"allocated":2998272,"data_stored":1774211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34999999999999998,"15min":0.33700000000000002},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.56499999999999995,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.252},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34799999999999998,"15min":0.34899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.186},"max":{"1min":0.55700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.44600000000000001,"15min":0.38800000000000001},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.54300000000000004,"5min":11.439,"15min":11.439},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":15.385999999999999,"15min":15.385999999999999},"last":0.317}]},{"osd":3,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.376},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":0.65000000000000002,"5min":5.766,"15min":5.766},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41499999999999998,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.25900000000000001},"max":{"1min":0.60499999999999998,"5min":3.3580000000000001,"15min":3.3580000000000001},"last":0.33000000000000002}]},{"osd":4,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84799999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.57599999999999996},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":2.802,"15min":2.802},"last":0.42899999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.47599999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.83499999999999996,"15min":0.999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":0.91500000000000004,"5min":5.7380000000000004,"15min":5.7380000000000004},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":2.907,"15min":2.907},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.40699999999999997,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":1.363},"last":0.50600000000000001}]},{"osd":7,"last update":"Sat Jun 10 18:59:22 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45800000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.44},{"interface":"front","average":{"1min":0.622,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":5.7939999999999996,"15min":5.7939999999999996},"last":0.47799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.35599999999999998,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55800000000000005,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":2.847},"last":0.308},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.40200000000000002},"min":{"1min":0.249,"5min":0.247,"15min":0.23799999999999999},"max":{"1min":0.622,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35399999999999998,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.69599999999999995,"5min":1.24,"15min":2.8999999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.208,"15min":5.742},"last":0.438},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":5.6429999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.623,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":6.593},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":7.0880000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58199999999999996,"15min":0.629},"min":{"1min":0.41799999999999998,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.83999999999999997,"5min":1.026,"15min":8.7850000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":1.171,"15min":5.6180000000000003},"last":0.47799999999999998}]},{"osd":7,"last update":"Sat Jun 10 18:59:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64700000000000002,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.87,"15min":6.617},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.5,"5min":0.38,"15min":0.37},"max":{"1min":0.879,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.46600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575048,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":0.46800000000000003,"5min":0.53900000000000003,"15min":1.2170000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34499999999999997,"15min":0.35599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.48599999999999999,"5min":0.54400000000000004,"15min":1.8600000000000001},"last":0.32400000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.35599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.53200000000000003,"5min":0.57299999999999995,"15min":0.99299999999999999},"last":0.309},{"interface":"front","average":{"1min":0.374,"5min":0.38,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.64900000000000002,"15min":1.3},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55000000000000004,"5min":0.57599999999999996,"15min":1.109},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.623,"15min":1.599},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.91100000000000003,"15min":1.925},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":1.1100000000000001,"15min":1.613},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.095,"15min":3.802},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":3.8180000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.375},"max":{"1min":0.77300000000000002,"5min":1.018,"15min":1.679},"last":0.503}]},{"osd":7,"last update":"Sat Jun 10 18:59:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.44400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96799999999999997,"5min":1.0549999999999999,"15min":1.47},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.91100000000000003,"15min":1.8939999999999999},"last":0.44500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.311,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52100000000000002,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.53600000000000003},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.27100000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29299999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.433}]},{"osd":2,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.497}]},{"osd":3,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33400000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.36199999999999999,"15min":0.34300000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.32200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.38600000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51100000000000001,"15min":0.51600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.71499999999999997,"5min":0.81599999999999995,"15min":2.1579999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.3169999999999999},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Jun 10 18:59:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54900000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.505},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.1459999999999999},"last":0.54200000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575049,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.379,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.69099999999999995}]},{"osd":1,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.749,"15min":0.78400000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.69399999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.626}]},{"osd":2,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.71999999999999997,"5min":0.78300000000000003,"15min":0.85099999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65100000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.64000000000000001}]},{"osd":3,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.628,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.34699999999999998},"max":{"1min":0.755,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.71799999999999997,"5min":0.747,"15min":0.83999999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53100000000000003,"15min":0.51800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69799999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.50900000000000001}]},{"osd":6,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.72699999999999998,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.879},"last":0.58799999999999997}]},{"osd":7,"last update":"Sat Jun 10 18:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57099999999999995,"15min":0.55100000000000005},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.30199999999999999},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T19:00:21.485 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.486 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.486 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.486 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.487 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-06-10T18:43:56.365182+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=43, tm_sec=56, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.487 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.487 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.488 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.488 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-06-10T18:44:02.520027+0000 time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=18, tm_min=44, tm_sec=2, tm_wday=5, tm_yday=161, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=6, tm_mday=10, tm_hour=19, tm_min=0, tm_sec=16, tm_wday=5, tm_yday=161, tm_isdst=0) 2023-06-10T19:00:21.488 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-06-10T19:00:39.998 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:40.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:41.490 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-10T19:00:41.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.617+0000 7f3ccd9cf700 1 Processor -- start 2023-06-10T19:00:41.623 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.617+0000 7f3ccd9cf700 1 -- start start 2023-06-10T19:00:41.625 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3ccd9cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cc80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3ccd9cf700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc806dcb0 con 0x7f3cc80cf030 2023-06-10T19:00:41.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3ccd9cf700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc80cf460 con 0x7f3cc815c500 2023-06-10T19:00:41.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3ccd9cf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc806e850 con 0x7f3cc806a580 2023-06-10T19:00:41.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3cc67fc700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3cc80cf030 0x7f3cc8069e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40642/0 (socket says 172.21.15.104:40642) 2023-06-10T19:00:41.626 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.618+0000 7f3cc67fc700 1 -- 172.21.15.104:0/1368635142 learned_addr learned my addr 172.21.15.104:0/1368635142 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:41.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cc80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986472941 0 0) 0x7f3cc806dcb0 con 0x7f3cc80cf030 2023-06-10T19:00:41.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb00035b0 con 0x7f3cc80cf030 2023-06-10T19:00:41.627 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062852578 0 0) 0x7f3cc80cf460 con 0x7f3cc815c500 2023-06-10T19:00:41.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc806dcb0 con 0x7f3cc815c500 2023-06-10T19:00:41.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474580770 0 0) 0x7f3cb00035b0 con 0x7f3cc80cf030 2023-06-10T19:00:41.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 msgr2=0x7f3cc80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:41.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cc80ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.628 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 >> v1:172.21.15.114:6789/0 conn(0x7f3cc815c500 legacy=0x7f3cc806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc806d0a0 con 0x7f3cc80cf030 2023-06-10T19:00:41.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3864580594 0 0) 0x7f3cb8001670 con 0x7f3cc80cf030 2023-06-10T19:00:41.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 >> v1:172.21.15.104:6789/0 conn(0x7f3cc80cf030 legacy=0x7f3cc8069e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cb0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.629 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb0004170 0x7f3cb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cb0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc806e850 con 0x7f3cb0004d90 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb000bb20 con 0x7f3cb0004170 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb000bce0 con 0x7f3cc806a580 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cb0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.630 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc67fc700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb0004170 0x7f3cb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc6ffd700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cb0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 -- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 msgr2=0x7f3cb0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:41.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc806a580 0x7f3cb0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 -- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb0004170 msgr2=0x7f3cb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:41.631 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cb0004170 0x7f3cb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.619+0000 7f3cc77fe700 1 -- 172.21.15.104:0/1368635142 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00035b0 con 0x7f3cb0004d90 2023-06-10T19:00:41.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cb0005200 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009fc0 tx=0x7f3cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e6e51e6400a8ca server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000ec30 con 0x7f3cb0004d90 2023-06-10T19:00:41.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc000f040 con 0x7f3cb0004d90 2023-06-10T19:00:41.632 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3cc5ffb700 1 -- 172.21.15.104:0/1368635142 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0014480 con 0x7f3cb0004d90 2023-06-10T19:00:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/1368635142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 msgr2=0x7f3cb0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/1368635142 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cb0005200 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009fc0 tx=0x7f3cb0005880 comp rx=0 tx=0).stop 2023-06-10T19:00:41.635 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3cc77fe700 1 -- 172.21.15.104:0/1368635142 reap_dead start 2023-06-10T19:00:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/1368635142 shutdown_connections 2023-06-10T19:00:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/1368635142 >> 172.21.15.104:0/1368635142 conn(0x7f3cc806b8a0 msgr2=0x7f3cc80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/1368635142 shutdown_connections 2023-06-10T19:00:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.620+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/1368635142 wait complete. 2023-06-10T19:00:41.636 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.622+0000 7f3ccd9cf700 1 Processor -- start 2023-06-10T19:00:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.622+0000 7f3ccd9cf700 1 -- start start 2023-06-10T19:00:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc80cf400 0x7f3cc80cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cc815dd30 0x7f3cc815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.637 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80cfdb0 con 0x7f3cb0004d90 2023-06-10T19:00:41.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80c3bf0 con 0x7f3cc815dd30 2023-06-10T19:00:41.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3ccd9cf700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc8152080 con 0x7f3cc80cf400 2023-06-10T19:00:41.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42824/0 (socket says 172.21.15.104:42824) 2023-06-10T19:00:41.638 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 -- 172.21.15.104:0/2512149170 learned_addr learned my addr 172.21.15.104:0/2512149170 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc77fe700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cc815dd30 0x7f3cc815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc67fc700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc80cf400 0x7f3cc80cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 -- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc80cf400 msgr2=0x7f3cc80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc80cf400 0x7f3cc80cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.639 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 -- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cc815dd30 msgr2=0x7f3cc815e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cc815dd30 0x7f3cc815e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 -- 172.21.15.104:0/2512149170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc80c32c0 con 0x7f3cb0004d90 2023-06-10T19:00:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.623+0000 7f3cc6ffd700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00f910 tx=0x7f3cbc00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.624+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0052e0 con 0x7f3cb0004d90 2023-06-10T19:00:41.640 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.624+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc005440 con 0x7f3cb0004d90 2023-06-10T19:00:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.624+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc003910 con 0x7f3cb0004d90 2023-06-10T19:00:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.624+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc80c35a0 con 0x7f3cb0004d90 2023-06-10T19:00:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.624+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc815e6e0 con 0x7f3cb0004d90 2023-06-10T19:00:41.641 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.625+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc8074050 con 0x7f3cb0004d90 2023-06-10T19:00:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.625+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3cbc003a70 con 0x7f3cb0004d90 2023-06-10T19:00:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.625+0000 7f3caffff700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 0x7f3cb406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.625+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3cbc0891d0 con 0x7f3cb0004d90 2023-06-10T19:00:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.626+0000 7f3cc67fc700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 0x7f3cb406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.642 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.626+0000 7f3cc67fc700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 0x7f3cb406d190 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8001050 tx=0x7f3cb8000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.643 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.630+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3cbc054420 con 0x7f3cb0004d90 2023-06-10T19:00:41.815 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.810+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 --> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cc806d660 con 0x7f3cb406ace0 2023-06-10T19:00:41.819 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.813+0000 7f3caffff700 1 -- 172.21.15.104:0/2512149170 <== mgr.4102 v2:172.21.15.114:6832/2421772364 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73261 (secure 0 0 0) 0x7f3cb80070c0 con 0x7f3cb406ace0 2023-06-10T19:00:41.820 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:41.821 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 msgr2=0x7f3cb406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 0x7f3cb406d190 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8001050 tx=0x7f3cb8000f80 comp rx=0 tx=0).stop 2023-06-10T19:00:41.822 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 msgr2=0x7f3cc8151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00f910 tx=0x7f3cbc00fc20 comp rx=0 tx=0).stop 2023-06-10T19:00:41.823 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 shutdown_connections 2023-06-10T19:00:41.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f3cb406ace0 0x7f3cb406d190 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cc815dd30 0x7f3cc815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.824 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3cb0004d90 0x7f3cc8151b40 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 --2- 172.21.15.104:0/2512149170 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cc80cf400 0x7f3cc80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.825 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.815+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 >> 172.21.15.104:0/2512149170 conn(0x7f3cc806b8a0 msgr2=0x7f3cc80b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:41.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.816+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 shutdown_connections 2023-06-10T19:00:41.826 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.816+0000 7f3ccd9cf700 1 -- 172.21.15.104:0/2512149170 wait complete. 2023-06-10T19:00:41.827 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2023-06-10T19:00:41.833 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2023-06-10T19:00:41.613045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":268344,"kb_used_data":19960,"kb_used_omap":0,"kb_used_meta":248320,"kb_avail":754706376,"statfs":{"total":773094113280,"available":772819329024,"internally_reserved":0,"allocated":20439040,"data_stored":10655556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":254279680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002521"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":319,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:19.662349+0000","last_change":"2023-06-10T19:00:19.662400+0000","last_active":"2023-06-10T19:00:19.662349+0000","last_peered":"2023-06-10T19:00:19.662349+0000","last_clean":"2023-06-10T19:00:19.662349+0000","last_became_active":"2023-06-10T18:44:03.803236+0000","last_became_peered":"2023-06-10T18:44:03.803236+0000","last_unstale":"2023-06-10T19:00:19.662349+0000","last_undegraded":"2023-06-10T19:00:19.662349+0000","last_fullsized":"2023-06-10T19:00:19.662349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:19.662235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:19.662235+0000","last_clean_scrub_stamp":"2023-06-10T19:00:19.662235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:34:26.383776+0000","scrub_duration":0.0015864029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":361,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:26.812200+0000","last_change":"2023-06-10T19:00:26.812247+0000","last_active":"2023-06-10T19:00:26.812200+0000","last_peered":"2023-06-10T19:00:26.812200+0000","last_clean":"2023-06-10T19:00:26.812200+0000","last_became_active":"2023-06-10T18:44:03.538024+0000","last_became_peered":"2023-06-10T18:44:03.538024+0000","last_unstale":"2023-06-10T19:00:26.812200+0000","last_undegraded":"2023-06-10T19:00:26.812200+0000","last_fullsized":"2023-06-10T19:00:26.812200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:26.812094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:26.812094+0000","last_clean_scrub_stamp":"2023-06-10T19:00:26.812094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:34:33.533634+0000","scrub_duration":0.0022560869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":313,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:23.002458+0000","last_change":"2023-06-10T19:00:23.002511+0000","last_active":"2023-06-10T19:00:23.002458+0000","last_peered":"2023-06-10T19:00:23.002458+0000","last_clean":"2023-06-10T19:00:23.002458+0000","last_became_active":"2023-06-10T18:44:03.540662+0000","last_became_peered":"2023-06-10T18:44:03.540662+0000","last_unstale":"2023-06-10T19:00:23.002458+0000","last_undegraded":"2023-06-10T19:00:23.002458+0000","last_fullsized":"2023-06-10T19:00:23.002458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:23.002352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:23.002352+0000","last_clean_scrub_stamp":"2023-06-10T19:00:23.002352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T03:23:34.373348+0000","scrub_duration":0.001687175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":361,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:21.720671+0000","last_change":"2023-06-10T19:00:21.720718+0000","last_active":"2023-06-10T19:00:21.720671+0000","last_peered":"2023-06-10T19:00:21.720671+0000","last_clean":"2023-06-10T19:00:21.720671+0000","last_became_active":"2023-06-10T18:44:03.541366+0000","last_became_peered":"2023-06-10T18:44:03.541366+0000","last_unstale":"2023-06-10T19:00:21.720671+0000","last_undegraded":"2023-06-10T19:00:21.720671+0000","last_fullsized":"2023-06-10T19:00:21.720671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:21.720565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:21.720565+0000","last_clean_scrub_stamp":"2023-06-10T19:00:21.720565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:11:20.530151+0000","scrub_duration":0.0018257709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"191'119","reported_seq":490,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:29.369455+0000","last_change":"2023-06-10T19:00:29.369539+0000","last_active":"2023-06-10T19:00:29.369455+0000","last_peered":"2023-06-10T19:00:29.369455+0000","last_clean":"2023-06-10T19:00:29.369455+0000","last_became_active":"2023-06-10T18:43:57.402519+0000","last_became_peered":"2023-06-10T18:43:57.402519+0000","last_unstale":"2023-06-10T19:00:29.369455+0000","last_undegraded":"2023-06-10T19:00:29.369455+0000","last_fullsized":"2023-06-10T19:00:29.369455+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'119","last_scrub_stamp":"2023-06-10T19:00:29.369359+0000","last_deep_scrub":"191'119","last_deep_scrub_stamp":"2023-06-10T19:00:29.369359+0000","last_clean_scrub_stamp":"2023-06-10T19:00:29.369359+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T06:13:07.411970+0000","scrub_duration":0.0055712870000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":332,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:33.281745+0000","last_change":"2023-06-10T19:00:33.281788+0000","last_active":"2023-06-10T19:00:33.281745+0000","last_peered":"2023-06-10T19:00:33.281745+0000","last_clean":"2023-06-10T19:00:33.281745+0000","last_became_active":"2023-06-10T18:44:03.540447+0000","last_became_peered":"2023-06-10T18:44:03.540447+0000","last_unstale":"2023-06-10T19:00:33.281745+0000","last_undegraded":"2023-06-10T19:00:33.281745+0000","last_fullsized":"2023-06-10T19:00:33.281745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:33.281636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:33.281636+0000","last_clean_scrub_stamp":"2023-06-10T19:00:33.281636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T01:21:39.813200+0000","scrub_duration":0.0020144070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":331,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:32.318853+0000","last_change":"2023-06-10T19:00:32.318915+0000","last_active":"2023-06-10T19:00:32.318853+0000","last_peered":"2023-06-10T19:00:32.318853+0000","last_clean":"2023-06-10T19:00:32.318853+0000","last_became_active":"2023-06-10T18:44:03.803340+0000","last_became_peered":"2023-06-10T18:44:03.803340+0000","last_unstale":"2023-06-10T19:00:32.318853+0000","last_undegraded":"2023-06-10T19:00:32.318853+0000","last_fullsized":"2023-06-10T19:00:32.318853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:32.318698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:32.318698+0000","last_clean_scrub_stamp":"2023-06-10T19:00:32.318698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T04:00:34.199733+0000","scrub_duration":0.0021323729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":319,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:20.649037+0000","last_change":"2023-06-10T19:00:20.649072+0000","last_active":"2023-06-10T19:00:20.649037+0000","last_peered":"2023-06-10T19:00:20.649037+0000","last_clean":"2023-06-10T19:00:20.649037+0000","last_became_active":"2023-06-10T18:44:03.541404+0000","last_became_peered":"2023-06-10T18:44:03.541404+0000","last_unstale":"2023-06-10T19:00:20.649037+0000","last_undegraded":"2023-06-10T19:00:20.649037+0000","last_fullsized":"2023-06-10T19:00:20.649037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:20.648960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:20.648960+0000","last_clean_scrub_stamp":"2023-06-10T19:00:20.648960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-12T05:15:54.962142+0000","scrub_duration":0.001564668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-06-10T19:00:21.041544+0000","last_change":"2023-06-10T19:00:21.041590+0000","last_active":"2023-06-10T19:00:21.041544+0000","last_peered":"2023-06-10T19:00:21.041544+0000","last_clean":"2023-06-10T19:00:21.041544+0000","last_became_active":"2023-06-10T18:44:03.541867+0000","last_became_peered":"2023-06-10T18:44:03.541867+0000","last_unstale":"2023-06-10T19:00:21.041544+0000","last_undegraded":"2023-06-10T19:00:21.041544+0000","last_fullsized":"2023-06-10T19:00:21.041544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-10T19:00:21.041434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-10T19:00:21.041434+0000","last_clean_scrub_stamp":"2023-06-10T19:00:21.041434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-06-11T20:20:36.604896+0000","scrub_duration":0.0016539619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575053,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33980,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337860,"statfs":{"total":96636764160,"available":96601968640,"internally_reserved":0,"allocated":3002368,"data_stored":1775230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.25800000000000001},"max":{"1min":0.68100000000000005,"5min":0.749,"15min":0.81499999999999995},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.247},"max":{"1min":0.66100000000000003,"5min":0.78800000000000003,"15min":0.88100000000000001},"last":0.747}]},{"osd":1,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.56200000000000006},"min":{"1min":0.39000000000000001,"5min":0.371,"15min":0.28499999999999998},"max":{"1min":0.71399999999999997,"5min":0.752,"15min":0.82199999999999995},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.33700000000000002},"max":{"1min":0.73199999999999998,"5min":0.76300000000000001,"15min":0.80600000000000005},"last":0.72199999999999998}]},{"osd":2,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.317},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.81999999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.53900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.27300000000000002},"max":{"1min":0.71499999999999997,"5min":0.745,"15min":0.79000000000000004},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.68899999999999995,"5min":0.76800000000000002,"15min":0.78300000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.41399999999999998,"5min":0.40400000000000003,"15min":0.32700000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.91700000000000004},"last":0.57999999999999996}]},{"osd":4,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46899999999999997,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.25600000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.89700000000000002},"last":0.374},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.73099999999999998,"15min":0.83799999999999997},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.91300000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29699999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.96499999999999997},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Jun 10 19:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.93899999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70799999999999996,"5min":0.79500000000000004,"15min":0.81100000000000005},"last":0.44900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575054,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.30499999999999999,"15min":0.28299999999999997},"max":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.28000000000000003},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.5}]},{"osd":1,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.626,"5min":0.76700000000000002,"15min":0.80600000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.29199999999999998},"max":{"1min":0.67400000000000004,"5min":0.78400000000000003,"15min":0.84899999999999998},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":0.83599999999999997,"15min":0.872},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52900000000000003,"15min":0.54400000000000004},"min":{"1min":0.38400000000000001,"5min":0.35899999999999999,"15min":0.30499999999999999},"max":{"1min":0.64600000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.66000000000000003}]},{"osd":3,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.33500000000000002,"15min":0.33400000000000002},"max":{"1min":0.61699999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.34799999999999998},"max":{"1min":0.68700000000000006,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67500000000000004}]},{"osd":4,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48599999999999999},"min":{"1min":0.34599999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71399999999999997,"5min":0.83899999999999997,"15min":0.874},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56499999999999995,"15min":0.56200000000000006},"min":{"1min":0.47299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.68200000000000005,"5min":0.76300000000000001,"15min":0.78600000000000003},"last":0.73199999999999998}]},{"osd":5,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.84099999999999997},"last":0.62},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.378,"5min":0.312,"15min":0.307},"max":{"1min":0.75900000000000001,"5min":0.81299999999999994,"15min":0.90600000000000003},"last":0.69199999999999995}]},{"osd":7,"last update":"Sat Jun 10 19:00:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54400000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.82299999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57199999999999995,"15min":0.57099999999999995},"min":{"1min":0.45700000000000002,"5min":0.35399999999999998,"15min":0.33700000000000002},"max":{"1min":0.77700000000000002,"5min":0.78600000000000003,"15min":0.84799999999999998},"last":0.71299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.23400000000000001,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.51200000000000001,"5min":0.53900000000000003,"15min":0.54100000000000004},"last":0.435},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.53400000000000003,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.378}]},{"osd":2,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.245,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51600000000000001,"5min":0.54000000000000004,"15min":0.64500000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.374},"min":{"1min":0.28899999999999998,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":0.54900000000000004,"5min":0.60999999999999999,"15min":0.61099999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.505,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.39200000000000002},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.24099999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":1.1040000000000001},"last":0.42399999999999999}]},{"osd":4,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.88500000000000001,"15min":1.1499999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.40500000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77800000000000002,"5min":0.89400000000000002,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46000000000000002,"5min":0.372,"15min":0.34799999999999998},"max":{"1min":0.80200000000000005,"5min":0.85099999999999998,"15min":0.89900000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.48199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.85999999999999999,"15min":0.871},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.34999999999999998},"max":{"1min":0.66600000000000004,"5min":0.85199999999999998,"15min":1.206},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58699999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.36899999999999999},"max":{"1min":0.75,"5min":0.82899999999999996,"15min":0.88700000000000001},"last":0.501}]},{"osd":7,"last update":"Sat Jun 10 19:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41099999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.88400000000000001,"15min":1.224},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.86299999999999999,"5min":0.878,"15min":0.91100000000000003},"last":0.54400000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575053,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33976,"kb_used_data":2928,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94337864,"statfs":{"total":96636764160,"available":96601972736,"internally_reserved":0,"allocated":2998272,"data_stored":1774211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32800000000000001,"15min":0.33600000000000002},"min":{"1min":0.22500000000000001,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":0.50700000000000001,"5min":0.56499999999999995,"15min":3.2429999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.221,"5min":0.217,"15min":0.186},"max":{"1min":0.442,"5min":0.55700000000000005,"15min":0.92800000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.35799999999999998,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":0.47899999999999998,"5min":0.54300000000000004,"15min":11.439},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.36899999999999999,"15min":0.41399999999999998},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.50700000000000001,"5min":0.58699999999999997,"15min":15.385999999999999},"last":0.41799999999999998}]},{"osd":3,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.375},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":0.51600000000000001,"5min":0.65000000000000002,"15min":5.766},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.40500000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.58199999999999996,"5min":0.64800000000000002,"15min":3.3580000000000001},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.57899999999999996},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75,"5min":0.84799999999999998,"15min":5.6989999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.35099999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":2.802},"last":0.78300000000000003}]},{"osd":5,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":0.76600000000000001,"5min":0.81599999999999995,"15min":0.90800000000000003},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.91500000000000004,"15min":5.7380000000000004},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82199999999999995,"5min":0.82699999999999996,"15min":2.907},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.74099999999999999,"5min":0.86199999999999999,"15min":0.97699999999999998},"last":0.67200000000000004}]},{"osd":7,"last update":"Sat Jun 10 19:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.36099999999999999},"max":{"1min":0.79200000000000004,"5min":0.90100000000000002,"15min":1.0489999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59399999999999997,"15min":0.61099999999999999},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.82999999999999996,"15min":5.7939999999999996},"last":0.753}]}]},{"osd":2,"up_from":13,"seq":55834575054,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32900000000000001,"15min":0.34300000000000003},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":0.65900000000000003,"15min":2.8759999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35699999999999998,"15min":0.38500000000000001},"min":{"1min":0.24199999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.90700000000000003,"15min":6.4450000000000003},"last":0.36599999999999999}]},{"osd":1,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36799999999999999,"15min":0.379},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47099999999999997,"5min":0.59899999999999998,"15min":2.847},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.247,"15min":0.247},"max":{"1min":0.48299999999999998,"5min":0.85099999999999998,"15min":5.6790000000000003},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34999999999999998,"15min":0.38300000000000001},"min":{"1min":0.249,"5min":0.223,"15min":0.223},"max":{"1min":0.436,"5min":0.80200000000000005,"15min":7.4569999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.40200000000000002,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.505,"5min":1.24,"15min":2.8999999999999999},"last":0.496}]},{"osd":4,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56999999999999995,"15min":0.61399999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.208,"15min":5.742},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57699999999999996,"15min":0.61899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":5.6429999999999998},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60599999999999998,"15min":0.63700000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.29399999999999998},"max":{"1min":0.78500000000000003,"5min":0.86799999999999999,"15min":6.593},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.626,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73199999999999998,"5min":1.002,"15min":7.0880000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.627},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.34599999999999997},"max":{"1min":0.73899999999999999,"5min":1.026,"15min":8.7850000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.61899999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72399999999999998,"5min":1.171,"15min":5.6180000000000003},"last":0.55200000000000005}]},{"osd":7,"last update":"Sat Jun 10 19:00:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63500000000000001,"15min":0.66100000000000003},"min":{"1min":0.438,"5min":0.41399999999999998,"15min":0.40699999999999997},"max":{"1min":0.81999999999999995,"5min":0.85699999999999998,"15min":6.617},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.623,"15min":0.66000000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":0.79700000000000004,"5min":0.97699999999999998,"15min":8.7149999999999999},"last":0.80000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575053,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33100000000000002,"15min":0.33400000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.58099999999999996},"last":0.318},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34899999999999998,"15min":0.34999999999999998},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.20200000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34399999999999997,"15min":0.35199999999999998},"min":{"1min":0.217,"5min":0.20200000000000001,"15min":0.20100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.60599999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38700000000000001,"15min":0.38600000000000001},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.24099999999999999},"max":{"1min":0.624,"5min":0.64900000000000002,"15min":0.68600000000000005},"last":0.495}]},{"osd":2,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38200000000000001,"15min":0.38900000000000001},"min":{"1min":0.24199999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.82999999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40999999999999998,"15min":0.41199999999999998},"min":{"1min":0.28799999999999998,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":1.5960000000000001},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.44800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":1.5049999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.40100000000000002,"5min":0.35299999999999998,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.0720000000000001,"15min":1.6539999999999999},"last":0.78400000000000003}]},{"osd":5,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92000000000000004,"5min":1.1100000000000001,"15min":1.613},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0089999999999999,"5min":1.095,"15min":1.4410000000000001},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1.5289999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.375},"max":{"1min":0.997,"5min":1.018,"15min":1.679},"last":0.753}]},{"osd":7,"last update":"Sat Jun 10 19:00:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.625,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.40699999999999997,"15min":0.35999999999999999},"max":{"1min":1.036,"5min":1.0549999999999999,"15min":1.47},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.42499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.548},"last":0.63900000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575053,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33396,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2404352,"data_stored":1183843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.755,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.38300000000000001,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":0.74399999999999999,"5min":1.23,"15min":1.23},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.54000000000000004},"min":{"1min":0.40200000000000002,"5min":0.36399999999999999,"15min":0.32400000000000001},"max":{"1min":0.754,"5min":1.216,"15min":1.216},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.41999999999999998,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":0.69799999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.32400000000000001},"max":{"1min":0.745,"5min":1.157,"15min":1.157},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.56299999999999994},"min":{"1min":0.47699999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"max":{"1min":0.85099999999999998,"5min":1.284,"15min":1.284},"last":0.375}]},{"osd":3,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.45600000000000002,"5min":0.34100000000000003,"15min":0.33400000000000002},"max":{"1min":0.70799999999999996,"5min":1.175,"15min":1.175},"last":0.629},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.371,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":1.131,"15min":1.131},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.504},"min":{"1min":0.32100000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.63300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.28199999999999997},"max":{"1min":0.65200000000000002,"5min":0.78100000000000003,"15min":2.1579999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.32300000000000001,"15min":0.32100000000000001},"max":{"1min":0.73099999999999998,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.53000000000000003}]},{"osd":7,"last update":"Sat Jun 10 19:00:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70899999999999996,"5min":1.089,"15min":1.089},"last":0.61399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575054,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2352,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2408448,"data_stored":1184862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63100000000000001,"5min":0.77200000000000002,"15min":0.77600000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.54000000000000004},"min":{"1min":0.33800000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.69899999999999995,"5min":0.78800000000000003,"15min":0.81000000000000005},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.52800000000000002},"min":{"1min":0.38200000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.67900000000000005,"5min":0.749,"15min":0.78400000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.376,"5min":0.36299999999999999,"15min":0.35899999999999999},"max":{"1min":0.66900000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69099999999999995,"5min":0.78300000000000003,"15min":0.80400000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70999999999999996,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.55100000000000005},"min":{"1min":0.38800000000000001,"5min":0.32700000000000001,"15min":0.29399999999999998},"max":{"1min":0.67500000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.34699999999999998},"max":{"1min":0.72499999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.58799999999999997}]},{"osd":4,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28100000000000003},"max":{"1min":0.61299999999999999,"5min":0.747,"15min":0.83999999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69599999999999995,"5min":0.78400000000000003,"15min":0.80900000000000005},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53200000000000003,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.76100000000000001,"15min":0.79200000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.64500000000000002,"5min":0.74199999999999999,"15min":0.879},"last":0.54300000000000004}]},{"osd":7,"last update":"Sat Jun 10 19:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66600000000000004,"5min":0.79600000000000004,"15min":0.88400000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.30199999999999999},"max":{"1min":0.75900000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-10T19:00:41.838 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-06-10T19:00:41.959 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.954+0000 7f352531c700 1 Processor -- start 2023-06-10T19:00:41.960 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.954+0000 7f352531c700 1 -- start start 2023-06-10T19:00:41.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f352531c700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35200d04f0 0x7f35200d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f352531c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35200d25c0 0x7f35200d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f352531c700 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35200d0e00 con 0x7f352006b510 2023-06-10T19:00:41.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f352531c700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35200af530 con 0x7f35200d04f0 2023-06-10T19:00:41.962 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f352531c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200af810 con 0x7f35200d25c0 2023-06-10T19:00:41.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35200d04f0 0x7f35200d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f35200d04f0 0x7f35200d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42834/0 (socket says 172.21.15.104:42834) 2023-06-10T19:00:41.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 -- 172.21.15.104:0/4032208516 learned_addr learned my addr 172.21.15.104:0/4032208516 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:41.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 -- 172.21.15.104:0/4032208516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35200d25c0 msgr2=0x7f35200d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:41.963 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351e7fc700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35200d25c0 0x7f35200d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35200d25c0 0x7f35200d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 -- 172.21.15.104:0/4032208516 >> v1:172.21.15.114:6789/0 conn(0x7f352006b510 legacy=0x7f352006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:41.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 -- 172.21.15.104:0/4032208516 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200afaf0 con 0x7f35200d04f0 2023-06-10T19:00:41.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351effd700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d04f0 0x7f35200d34d0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f3514006fd0 tx=0x7f3514009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf0ec45fc8fe263 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.964 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.955+0000 7f351dffb700 1 -- 172.21.15.104:0/4032208516 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514005e30 con 0x7f35200d04f0 2023-06-10T19:00:41.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f351dffb700 1 -- 172.21.15.104:0/4032208516 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514006030 con 0x7f35200d04f0 2023-06-10T19:00:41.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f351dffb700 1 -- 172.21.15.104:0/4032208516 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35140063c0 con 0x7f35200d04f0 2023-06-10T19:00:41.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 -- 172.21.15.104:0/4032208516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d04f0 msgr2=0x7f35200d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d04f0 0x7f35200d34d0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f3514006fd0 tx=0x7f3514009bd0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.965 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 -- 172.21.15.104:0/4032208516 shutdown_connections 2023-06-10T19:00:41.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d04f0 0x7f35200d34d0 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 --2- 172.21.15.104:0/4032208516 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35200d25c0 0x7f35200d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 -- 172.21.15.104:0/4032208516 >> 172.21.15.104:0/4032208516 conn(0x7f3520071ec0 msgr2=0x7f352009bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:41.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 -- 172.21.15.104:0/4032208516 shutdown_connections 2023-06-10T19:00:41.966 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.956+0000 7f352531c700 1 -- 172.21.15.104:0/4032208516 wait complete. 2023-06-10T19:00:41.968 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.958+0000 7f352531c700 1 Processor -- start 2023-06-10T19:00:41.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.958+0000 7f352531c700 1 -- start start 2023-06-10T19:00:41.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f352006b510 0x7f35200b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200d04f0 0x7f35200cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.969 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b5c50 con 0x7f35200d25c0 2023-06-10T19:00:41.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b5f30 con 0x7f35200d04f0 2023-06-10T19:00:41.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f352531c700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200b6210 con 0x7f352006b510 2023-06-10T19:00:41.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.970 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42842/0 (socket says 172.21.15.104:42842) 2023-06-10T19:00:41.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 -- 172.21.15.104:0/1337712291 learned_addr learned my addr 172.21.15.104:0/1337712291 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:41.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351effd700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f352006b510 0x7f35200b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351e7fc700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200d04f0 0x7f35200cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 -- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f352006b510 msgr2=0x7f35200b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.971 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f352006b510 0x7f35200b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 -- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200d04f0 msgr2=0x7f35200cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:41.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200d04f0 0x7f35200cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:41.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 -- 172.21.15.104:0/1337712291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514009500 con 0x7f35200d25c0 2023-06-10T19:00:41.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f351f7fe700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f351800b970 tx=0x7f351800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.972 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.959+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351800d860 con 0x7f35200d25c0 2023-06-10T19:00:41.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.960+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351800d9c0 con 0x7f35200d25c0 2023-06-10T19:00:41.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.960+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35200b64f0 con 0x7f35200d25c0 2023-06-10T19:00:41.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.960+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351800dd70 con 0x7f35200d25c0 2023-06-10T19:00:41.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.960+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35200cde50 con 0x7f35200d25c0 2023-06-10T19:00:41.973 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.960+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35200b6b40 con 0x7f35200d25c0 2023-06-10T19:00:41.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.961+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f351800e430 con 0x7f35200d25c0 2023-06-10T19:00:41.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.961+0000 7f35077fe700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:41.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.961+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3518089360 con 0x7f35200d25c0 2023-06-10T19:00:41.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.962+0000 7f351effd700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:41.974 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.962+0000 7f351effd700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3514013040 tx=0x7f3514008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:41.975 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:41.966+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3518054630 con 0x7f35200d25c0 2023-06-10T19:00:42.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.153+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f3520067010 con 0x7f35200d25c0 2023-06-10T19:00:42.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.158+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3518054630 con 0x7f35200d25c0 2023-06-10T19:00:42.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.159+0000 7f35077fe700 1 -- 172.21.15.104:0/1337712291 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v32)=0 v32) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3518057c50 con 0x7f35200d25c0 2023-06-10T19:00:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 msgr2=0x7f350c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3514013040 tx=0x7f3514008070 comp rx=0 tx=0).stop 2023-06-10T19:00:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 msgr2=0x7f35200ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.167 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f351800b970 tx=0x7f351800bc80 comp rx=0 tx=0).stop 2023-06-10T19:00:42.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 shutdown_connections 2023-06-10T19:00:42.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35200d25c0 0x7f35200ce870 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f352006b510 0x7f35200b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f350c06ace0 0x7f350c06d190 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.168 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 --2- 172.21.15.104:0/1337712291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35200d04f0 0x7f35200cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 >> 172.21.15.104:0/1337712291 conn(0x7f3520071ec0 msgr2=0x7f352015b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:42.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 shutdown_connections 2023-06-10T19:00:42.169 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.161+0000 7f352531c700 1 -- 172.21.15.104:0/1337712291 wait complete. 2023-06-10T19:00:42.179 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-06-10T19:00:42.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-10T19:00:42.299 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.294+0000 7f4e71d70700 1 Processor -- start 2023-06-10T19:00:42.300 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.294+0000 7f4e71d70700 1 -- start start 2023-06-10T19:00:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e71d70700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e6c15a420 0x7f4e6c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.302 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e71d70700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e71d70700 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0b6070 con 0x7f4e6c0ce890 2023-06-10T19:00:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e71d70700 1 -- --> v2:172.21.15.104:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0ced10 con 0x7f4e6c15a420 2023-06-10T19:00:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e71d70700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0b5460 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 --2- >> v2:172.21.15.104:3300/0 conn(0x7f4e6c15a420 0x7f4e6c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6bfff700 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f4e6c0ce890 0x7f4e6c0b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40678/0 (socket says 172.21.15.104:40678) 2023-06-10T19:00:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6bfff700 1 -- 172.21.15.104:0/3381468649 learned_addr learned my addr 172.21.15.104:0/3381468649 (peer_addr_for_me v1:172.21.15.104:0/0) 2023-06-10T19:00:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6affd700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/3381468649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 msgr2=0x7f4e6c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-10T19:00:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.304 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/3381468649 >> v1:172.21.15.104:6789/0 conn(0x7f4e6c0ce890 legacy=0x7f4e6c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/3381468649 --> v2:172.21.15.104:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0b5740 con 0x7f4e6c15a420 2023-06-10T19:00:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6a7fc700 1 -- 172.21.15.104:0/3381468649 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662518387 0 0) 0x7f4e6c0b6070 con 0x7f4e6c0ce890 2023-06-10T19:00:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6affd700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:42.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6b7fe700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0b1660 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009840 tx=0x7f4e60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8128eec0c803d52c server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.295+0000 7f4e6a7fc700 1 -- 172.21.15.104:0/3381468649 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60011830 con 0x7f4e6c15a420 2023-06-10T19:00:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e6a7fc700 1 -- 172.21.15.104:0/3381468649 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4e60011990 con 0x7f4e6c15a420 2023-06-10T19:00:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e6a7fc700 1 -- 172.21.15.104:0/3381468649 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60011d40 con 0x7f4e6c15a420 2023-06-10T19:00:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 -- 172.21.15.104:0/3381468649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c15a420 msgr2=0x7f4e6c0b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0b1660 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009840 tx=0x7f4e60009b50 comp rx=0 tx=0).stop 2023-06-10T19:00:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 -- 172.21.15.104:0/3381468649 shutdown_connections 2023-06-10T19:00:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0b1660 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 --2- 172.21.15.104:0/3381468649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 -- 172.21.15.104:0/3381468649 >> 172.21.15.104:0/3381468649 conn(0x7f4e6c089fd0 msgr2=0x7f4e6c0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:42.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 -- 172.21.15.104:0/3381468649 shutdown_connections 2023-06-10T19:00:42.308 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.296+0000 7f4e71d70700 1 -- 172.21.15.104:0/3381468649 wait complete. 2023-06-10T19:00:42.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.298+0000 7f4e71d70700 1 Processor -- start 2023-06-10T19:00:42.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.298+0000 7f4e71d70700 1 -- start start 2023-06-10T19:00:42.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0ce890 0x7f4e6c1f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0cd2c0 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.311 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0cc450 con 0x7f4e6c15a420 2023-06-10T19:00:42.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1f4550 con 0x7f4e6c0ce890 2023-06-10T19:00:42.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42852/0 (socket says 172.21.15.104:42852) 2023-06-10T19:00:42.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/1987041718 learned_addr learned my addr 172.21.15.104:0/1987041718 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-06-10T19:00:42.312 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6affd700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0ce890 0x7f4e6c1f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6bfff700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0ce890 msgr2=0x7f4e6c1f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0ce890 0x7f4e6c1f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 msgr2=0x7f4e6c0ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.313 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0ad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 -- 172.21.15.104:0/1987041718 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600094a0 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6bfff700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-10T19:00:42.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e6b7fe700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f4e600090b0 tx=0x7f4e60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:42.314 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600038a0 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4e6001f070 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c0cc640 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.299+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60003c80 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.300+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c0ccc00 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.315 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.301+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c0cd400 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.301+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4e6001a430 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.301+0000 7f4e53fff700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-10T19:00:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.302+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4e60021070 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.302+0000 7f4e6affd700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-10T19:00:42.316 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.302+0000 7f4e6affd700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c006fd0 tx=0x7f4e5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-10T19:00:42.317 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.306+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4e6c0cd400 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.538 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.533+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e6c0cd400 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.539 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.534+0000 7f4e53fff700 1 -- 172.21.15.104:0/1987041718 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+3547 (secure 0 0 0) 0x7f4e6c0cd400 con 0x7f4e6c0b4ff0 2023-06-10T19:00:42.540 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-06-10T19:00:42.540 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2343122386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2343122386},{"type":"v1","addr":"172.21.15.114:6836","nonce":2343122386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9349,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2238305807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2238305807},{"type":"v1","addr":"172.21.15.114:6837","nonce":2238305807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113},{"gid":9685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3749096198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3749096198},{"type":"v1","addr":"172.21.15.104:6835","nonce":3749096198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119},{"gid":11121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/3927731064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":3927731064},{"type":"v1","addr":"172.21.15.104:6833","nonce":3927731064}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":113}],"filesystems":[]} 2023-06-10T19:00:42.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 msgr2=0x7f4e5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.541 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c006fd0 tx=0x7f4e5c008040 comp rx=0 tx=0).stop 2023-06-10T19:00:42.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 msgr2=0x7f4e6c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-10T19:00:42.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f4e600090b0 tx=0x7f4e60004750 comp rx=0 tx=0).stop 2023-06-10T19:00:42.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 shutdown_connections 2023-06-10T19:00:42.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:6832/2421772364,v1:172.21.15.114:6833/2421772364] conn(0x7f4e5806ace0 0x7f4e5806d190 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.542 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.535+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15a420 0x7f4e6c0ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.536+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e6c0b4ff0 0x7f4e6c0cbf10 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.536+0000 7f4e71d70700 1 --2- 172.21.15.104:0/1987041718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e6c0ce890 0x7f4e6c1f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-10T19:00:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.536+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 >> 172.21.15.104:0/1987041718 conn(0x7f4e6c089fd0 msgr2=0x7f4e6c0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-10T19:00:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.536+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 shutdown_connections 2023-06-10T19:00:42.543 INFO:teuthology.orchestra.run.smithi104.stderr:2023-06-10T19:00:42.536+0000 7f4e71d70700 1 -- 172.21.15.104:0/1987041718 wait complete. 2023-06-10T19:00:42.544 INFO:teuthology.orchestra.run.smithi104.stderr:dumped fsmap epoch 119 2023-06-10T19:00:42.553 DEBUG:tasks.cephfs.filesystem:already dead... 2023-06-10T19:00:42.553 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-10T19:00:42.554 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-10T19:00:42.554 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.562 INFO:tasks.ceph.mds.a:Stopped 2023-06-10T19:00:42.562 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-10T19:00:42.562 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.610 INFO:tasks.ceph.mds.c:Stopped 2023-06-10T19:00:42.611 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-10T19:00:42.611 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.622 INFO:tasks.ceph.mds.b:Stopped 2023-06-10T19:00:42.622 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-10T19:00:42.623 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.673 INFO:tasks.ceph.mds.d:Stopped 2023-06-10T19:00:42.673 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-10T19:00:42.674 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-10T19:00:42.674 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.710 INFO:tasks.ceph.osd.0:Stopped 2023-06-10T19:00:42.711 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-10T19:00:42.711 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.785 INFO:tasks.ceph.osd.1:Stopped 2023-06-10T19:00:42.786 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-10T19:00:42.787 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.864 INFO:tasks.ceph.osd.2:Stopped 2023-06-10T19:00:42.864 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-10T19:00:42.865 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.938 INFO:tasks.ceph.osd.3:Stopped 2023-06-10T19:00:42.939 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-10T19:00:42.939 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:42.980 INFO:tasks.ceph.osd.4:Stopped 2023-06-10T19:00:42.980 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-10T19:00:42.981 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.023 INFO:tasks.ceph.osd.5:Stopped 2023-06-10T19:00:43.024 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-10T19:00:43.024 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.059 INFO:tasks.ceph.osd.6:Stopped 2023-06-10T19:00:43.060 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-10T19:00:43.061 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.137 INFO:tasks.ceph.osd.7:Stopped 2023-06-10T19:00:43.138 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-10T19:00:43.138 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-10T19:00:43.139 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.160 INFO:tasks.ceph.mgr.y:Stopped 2023-06-10T19:00:43.161 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-10T19:00:43.161 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.204 INFO:tasks.ceph.mgr.x:Stopped 2023-06-10T19:00:43.204 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-10T19:00:43.205 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-10T19:00:43.205 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.216 INFO:tasks.ceph.mon.a:Stopped 2023-06-10T19:00:43.216 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-10T19:00:43.217 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.256 INFO:tasks.ceph.mon.b:Stopped 2023-06-10T19:00:43.256 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-10T19:00:43.257 INFO:teuthology.orchestra.run:waiting for 300 2023-06-10T19:00:43.308 INFO:tasks.ceph.mon.c:Stopped 2023-06-10T19:00:43.309 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-10T19:00:43.309 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 't responding to mclientcaps' | egrep -v 'file system flag refuse_client_session is set' | head -n 1 2023-06-10T19:00:43.341 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:43.343 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-10T19:00:43.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:43.592 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-10T19:00:43.705 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:43.705 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-10T19:00:43.774 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:43.775 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-10T19:00:43.858 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:43.859 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-10T19:00:44.071 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:44.071 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-10T19:00:44.168 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:44.169 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-10T19:00:44.233 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:44.233 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-10T19:00:44.312 INFO:tasks.ceph:Archiving mon data... 2023-06-10T19:00:44.313 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/data/mon.a.tgz 2023-06-10T19:00:44.314 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2023-06-10T19:00:44.331 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.cZROoCtcfL 2023-06-10T19:00:44.332 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cZROoCtcfL 2023-06-10T19:00:44.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.cZROoCtcfL 2023-06-10T19:00:44.604 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.cZROoCtcfL is 479KB 2023-06-10T19:00:44.700 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.cZROoCtcfL 2023-06-10T19:00:44.717 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/data/mon.b.tgz 2023-06-10T19:00:44.717 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2023-06-10T19:00:44.735 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.245rcKRqWJ 2023-06-10T19:00:44.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.245rcKRqWJ 2023-06-10T19:00:44.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.245rcKRqWJ 2023-06-10T19:00:44.991 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.245rcKRqWJ is 483KB 2023-06-10T19:00:45.058 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.245rcKRqWJ 2023-06-10T19:00:45.075 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/data/mon.c.tgz 2023-06-10T19:00:45.075 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2023-06-10T19:00:45.130 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.7imARXDaIo 2023-06-10T19:00:45.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7imARXDaIo 2023-06-10T19:00:45.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.7imARXDaIo 2023-06-10T19:00:45.368 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.7imARXDaIo is 483KB 2023-06-10T19:00:45.461 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.7imARXDaIo 2023-06-10T19:00:45.477 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-10T19:00:45.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-10T19:00:45.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-10T19:00:45.542 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-10T19:00:45.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-10T19:00:45.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-10T19:00:45.595 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-10T19:00:45.616 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-10T19:00:45.622 INFO:tasks.ceph:Archiving crash dumps... 2023-06-10T19:00:45.624 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi104/crash 2023-06-10T19:00:45.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-10T19:00:45.654 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi114/crash 2023-06-10T19:00:45.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-10T19:00:45.688 INFO:tasks.ceph:Shutting down logrotate 2023-06-10T19:00:45.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:45.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:45.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:45.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-10T19:00:45.838 INFO:tasks.ceph:Compressing logs... 2023-06-10T19:00:45.839 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-06-10T19:00:45.842 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-06-10T19:00:45.868 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125762.log 2023-06-10T19:00:45.868 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124852.log 2023-06-10T19:00:45.868 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122780.log 2023-06-10T19:00:45.872 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129041.log 2023-06-10T19:00:45.872 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.125762.log.gz 2023-06-10T19:00:45.872 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124852.log.gz 2023-06-10T19:00:45.872 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142088.log 2023-06-10T19:00:45.872 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112962.log 2023-06-10T19:00:45.873 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122780.log.gz 2023-06-10T19:00:45.873 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151918.log 2023-06-10T19:00:45.873 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129041.log.gz 2023-06-10T19:00:45.873 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.112962.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142088.log.gz 2023-06-10T19:00:45.873 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116664.log 2023-06-10T19:00:45.874 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112962.log.gz 2023-06-10T19:00:45.874 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118659.log 2023-06-10T19:00:45.874 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151918.log: /var/log/ceph/ceph-client.admin.116664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140097.log 2023-06-10T19:00:45.874 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8%/var/log/ceph/ceph-client.admin.118659.log: -- replaced with /var/log/ceph/ceph-client.admin.116664.log.gz 2023-06-10T19:00:45.874 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113285.log 2023-06-10T19:00:45.875 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154018.log 2023-06-10T19:00:45.875 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118659.log.gz 2023-06-10T19:00:45.875 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.151918.log.gz 2023-06-10T19:00:45.875 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113285.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.140097.log.gz 2023-06-10T19:00:45.875 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156751.log 2023-06-10T19:00:45.876 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113285.log.gz 2023-06-10T19:00:45.876 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146029.log 2023-06-10T19:00:45.876 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154018.log.gz 2023-06-10T19:00:45.876 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153293.log 2023-06-10T19:00:45.876 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146029.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.156751.log.gz 2023-06-10T19:00:45.877 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117400.log 2023-06-10T19:00:45.877 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146029.log.gz 2023-06-10T19:00:45.877 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168626.log 2023-06-10T19:00:45.877 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153293.log.gz 2023-06-10T19:00:45.877 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126901.log 2023-06-10T19:00:45.922 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100202.log 2023-06-10T19:00:45.922 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-06-10T19:00:45.923 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.108371.log 2023-06-10T19:00:45.923 INFO:teuthology.orchestra.run.smithi114.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100202.log.gz 2023-06-10T19:00:45.923 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100436.log 2023-06-10T19:00:45.923 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.108371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.114128.log 2023-06-10T19:00:45.923 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-06-10T19:00:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100436.log: /var/log/ceph/ceph-client.1.114128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100293.log 2023-06-10T19:00:45.924 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100436.log.gz 2023-06-10T19:00:45.924 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.1.108371.log.gz 2023-06-10T19:00:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100392.log 2023-06-10T19:00:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100110.log 2023-06-10T19:00:45.925 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100392.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100293.log.gz 2023-06-10T19:00:45.925 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100110.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100392.log.gz 2023-06-10T19:00:45.925 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.114930.log 2023-06-10T19:00:45.925 INFO:teuthology.orchestra.run.smithi114.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100110.log.gz 2023-06-10T19:00:45.926 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 92.6%--verbose -- /var/log/ceph/ceph-client.1.106897.log -- replaced with /var/log/ceph/ceph-client.1.114128.log.gz 2023-06-10T19:00:45.926 INFO:teuthology.orchestra.run.smithi114.stderr: 2023-06-10T19:00:45.926 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-06-10T19:00:45.926 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.114930.log: /var/log/ceph/ceph-client.1.106897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.111326.log 2023-06-10T19:00:45.927 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100248.log 2023-06-10T19:00:45.927 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.1.111326.log: 90.8% -- replaced with /var/log/ceph/ceph-client.1.106897.log.gz 2023-06-10T19:00:45.927 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100248.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100248.log.gz 2023-06-10T19:00:45.927 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.115813.log 2023-06-10T19:00:45.927 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.1.111326.log.gz 2023-06-10T19:00:45.928 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.111958.log 2023-06-10T19:00:45.928 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.115813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110297.log 2023-06-10T19:00:45.928 INFO:teuthology.orchestra.run.smithi114.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.1.114930.log.gz 2023-06-10T19:00:45.928 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.111958.log: /var/log/ceph/ceph-client.1.110297.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-06-10T19:00:45.929 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-06-10T19:00:45.929 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mds.d.log: 91.2% -- replaced with /var/log/ceph/ceph-client.1.111958.log.gz 2023-06-10T19:00:45.929 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-06-10T19:00:45.930 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-06-10T19:00:45.930 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.audit.log: 95.7% -- replaced with /var/log/ceph/ceph-client.1.110297.log.gz 2023-06-10T19:00:45.930 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100339.log 2023-06-10T19:00:45.931 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.1.115813.log.gz 2023-06-10T19:00:45.931 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-06-10T19:00:45.931 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100339.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100339.log.gz 2023-06-10T19:00:45.935 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.118542.log 2023-06-10T19:00:45.935 INFO:teuthology.orchestra.run.smithi114.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-06-10T19:00:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-06-10T19:00:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.118542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.116776.log 2023-06-10T19:00:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.1.116776.log: 91.9% -- replaced with /var/log/ceph/ceph-client.1.118542.log.gz 2023-06-10T19:00:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.107494.log 2023-06-10T19:00:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.3.107494.log: 91.9% -- replaced with /var/log/ceph/ceph-client.3.107494.log.gz 2023-06-10T19:00:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-06-10T19:00:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.7.log: 93.1% -- replaced with /var/log/ceph/ceph-client.1.116776.log.gz 2023-06-10T19:00:45.938 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-06-10T19:00:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.109373.log 2023-06-10T19:00:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.109373.log: 92.4% -- replaced with /var/log/ceph/ceph-client.1.109373.log.gz 2023-06-10T19:00:45.976 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.117671.log 2023-06-10T19:00:45.984 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117400.log: /var/log/ceph/ceph-client.admin.168626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124714.log 2023-06-10T19:00:45.984 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.117400.log.gz 2023-06-10T19:00:45.984 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.168626.log.gz 2023-06-10T19:00:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136448.log 2023-06-10T19:00:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124714.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.126901.log.gz 2023-06-10T19:00:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168378.log 2023-06-10T19:00:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136448.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124714.log.gz 2023-06-10T19:00:45.985 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126126.log 2023-06-10T19:00:45.986 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136448.log.gz 2023-06-10T19:00:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111951.log 2023-06-10T19:00:45.986 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6%/var/log/ceph/ceph-client.admin.126126.log: -- replaced with /var/log/ceph/ceph-client.admin.168378.log.gz 2023-06-10T19:00:45.986 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109801.log 2023-06-10T19:00:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.111951.log: gzip -5 89.2%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.126126.log.gz/var/log/ceph/ceph-client.admin.148337.log 2023-06-10T19:00:45.987 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122826.log 2023-06-10T19:00:45.987 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109801.log.gz 2023-06-10T19:00:45.987 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108814.log 2023-06-10T19:00:45.988 INFO:teuthology.orchestra.run.smithi104.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.0.111951.log.gz 2023-06-10T19:00:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123244.log 2023-06-10T19:00:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108814.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.148337.log.gz 2023-06-10T19:00:45.988 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122826.log.gz 2023-06-10T19:00:45.988 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131301.log 2023-06-10T19:00:45.989 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108814.log.gz 2023-06-10T19:00:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134739.log 2023-06-10T19:00:45.989 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123244.log: /var/log/ceph/ceph-client.admin.131301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146804.log 2023-06-10T19:00:45.989 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123244.log.gz 2023-06-10T19:00:45.989 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.131301.log.gz 2023-06-10T19:00:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106809.log 2023-06-10T19:00:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139664.log 2023-06-10T19:00:45.990 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134739.log.gz 2023-06-10T19:00:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106809.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146804.log.gz 2023-06-10T19:00:45.990 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146528.log 2023-06-10T19:00:45.991 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106809.log.gz 2023-06-10T19:00:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141421.log 2023-06-10T19:00:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146528.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.139664.log.gz 2023-06-10T19:00:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169042.log 2023-06-10T19:00:45.991 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141421.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146528.log.gz 2023-06-10T19:00:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121898.log 2023-06-10T19:00:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169042.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141421.log.gz 2023-06-10T19:00:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125988.log 2023-06-10T19:00:45.992 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.169042.log.gz 2023-06-10T19:00:45.992 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106554.log 2023-06-10T19:00:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121898.log: /var/log/ceph/ceph-client.admin.125988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157654.log 2023-06-10T19:00:45.993 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.121898.log.gz 2023-06-10T19:00:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106554.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125988.log.gz 2023-06-10T19:00:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144232.log 2023-06-10T19:00:45.993 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157654.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.106554.log.gz 2023-06-10T19:00:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141144.log 2023-06-10T19:00:45.994 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3%/var/log/ceph/ceph-client.admin.144232.log: -- replaced with /var/log/ceph/ceph-client.admin.157654.log.gz 2023-06-10T19:00:45.994 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110170.log 2023-06-10T19:00:45.994 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.144232.log.gz 2023-06-10T19:00:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139281.log 2023-06-10T19:00:45.995 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146620.log 2023-06-10T19:00:45.996 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141144.log.gz 2023-06-10T19:00:45.996 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139281.log: gzip -5 88.7%--verbose -- /var/log/ceph/ceph-client.admin.143559.log -- replaced with /var/log/ceph/ceph-client.admin.110170.log.gz 2023-06-10T19:00:45.997 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:45.997 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139281.log.gz 2023-06-10T19:00:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134693.log 2023-06-10T19:00:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143559.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.146620.log.gz 2023-06-10T19:00:45.998 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156350.log 2023-06-10T19:00:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134693.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.143559.log.gz 2023-06-10T19:00:45.999 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150551.log 2023-06-10T19:00:46.000 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142881.log 2023-06-10T19:00:46.000 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134693.log.gz 2023-06-10T19:00:46.001 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.156350.log.gz 2023-06-10T19:00:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122036.log 2023-06-10T19:00:46.001 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150551.log.gz 2023-06-10T19:00:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134831.log 2023-06-10T19:00:46.001 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137569.log 2023-06-10T19:00:46.002 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.142881.log.gz 2023-06-10T19:00:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119211.log 2023-06-10T19:00:46.002 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.122036.log.gz 2023-06-10T19:00:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137569.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134831.log.gz 2023-06-10T19:00:46.002 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130987.log 2023-06-10T19:00:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119211.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137569.log.gz 2023-06-10T19:00:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137661.log 2023-06-10T19:00:46.003 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119211.log.gz 2023-06-10T19:00:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116526.log 2023-06-10T19:00:46.003 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130987.log: /var/log/ceph/ceph-client.admin.137661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140768.log 2023-06-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.130987.log.gz 2023-06-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137661.log.gz 2023-06-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110153.log 2023-06-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152690.log 2023-06-10T19:00:46.004 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116526.log.gz 2023-06-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110153.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140768.log.gz 2023-06-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156889.log 2023-06-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6%/var/log/ceph/ceph-client.admin.152690.log: -- replaced with /var/log/ceph/ceph-client.admin.110153.log.gz 2023-06-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150091.log 2023-06-10T19:00:46.005 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.152690.log.gz 2023-06-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113883.log 2023-06-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.156889.log.gz 2023-06-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138890.log 2023-06-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137477.log 2023-06-10T19:00:46.006 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.150091.log.gz 2023-06-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113883.log.gz 2023-06-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161420.log 2023-06-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.138890.log.gz 2023-06-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135128.log 2023-06-10T19:00:46.007 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161420.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137477.log.gz 2023-06-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113607.log 2023-06-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135128.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.161420.log.gz 2023-06-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147166.log 2023-06-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135128.log.gz 2023-06-10T19:00:46.008 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106416.log 2023-06-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113607.log.gz 2023-06-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124557.log 2023-06-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147166.log: /var/log/ceph/ceph-client.admin.106416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156797.log 2023-06-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106416.log.gz 87.7%gzip 2023-06-10T19:00:46.009 INFO:teuthology.orchestra.run.smithi104.stderr:-5 -- replaced with /var/log/ceph/ceph-client.admin.147166.log.gz--verbose 2023-06-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-osd.2.log 2023-06-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124557.log: /var/log/ceph/ceph-client.admin.156797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167879.log 2023-06-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124557.log.gz 2023-06-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.2.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156797.log.gz 2023-06-10T19:00:46.010 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114405.log 2023-06-10T19:00:46.011 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146574.log 2023-06-10T19:00:46.011 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.167879.log.gz 2023-06-10T19:00:46.011 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138217.log 2023-06-10T19:00:46.011 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114405.log.gz 2023-06-10T19:00:46.011 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165392.log 2023-06-10T19:00:46.012 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138217.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.146574.log.gz 2023-06-10T19:00:46.012 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157901.log 2023-06-10T19:00:46.012 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3%/var/log/ceph/ceph-client.admin.165392.log: -- replaced with /var/log/ceph/ceph-client.admin.138217.log.gz 2023-06-10T19:00:46.012 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170122.log 2023-06-10T19:00:46.012 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109786.log 2023-06-10T19:00:46.013 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.170122.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.157901.log.gz 2023-06-10T19:00:46.013 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.165392.log.gz 2023-06-10T19:00:46.013 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113515.log 2023-06-10T19:00:46.013 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.170122.log.gz 2023-06-10T19:00:46.013 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106056.log /var/log/ceph/ceph-client.admin.109786.log: 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157993.log 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109786.log.gz 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106056.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.113515.log.gz 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143645.log 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157993.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106056.log.gz 2023-06-10T19:00:46.014 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163282.log 2023-06-10T19:00:46.015 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.157993.log.gz 2023-06-10T19:00:46.015 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164374.log 2023-06-10T19:00:46.015 INFO:teuthology.orchestra.run.smithi104.stderr:gzip 88.8%-5 -- replaced with /var/log/ceph/ceph-client.admin.143645.log.gz--verbose 2023-06-10T19:00:46.015 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.166504.log 2023-06-10T19:00:46.015 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163282.log: /var/log/ceph/ceph-client.admin.164374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133677.log 2023-06-10T19:00:46.016 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.163282.log.gz 2023-06-10T19:00:46.016 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.164374.log.gz 2023-06-10T19:00:46.016 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117492.log 2023-06-10T19:00:46.016 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133677.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.166504.log.gz-5 2023-06-10T19:00:46.016 INFO:teuthology.orchestra.run.smithi104.stderr:--verbose -- /var/log/ceph/ceph-client.admin.127436.log 2023-06-10T19:00:46.017 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9%/var/log/ceph/ceph-client.admin.117492.log: -- replaced with /var/log/ceph/ceph-client.admin.133677.log.gz 2023-06-10T19:00:46.017 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.93757.log 2023-06-10T19:00:46.017 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117492.log.gz 2023-06-10T19:00:46.017 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118471.log 2023-06-10T19:00:46.017 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.tmp-client.admin.93757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146850.log 2023-06-10T19:00:46.018 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127436.log.gz 2023-06-10T19:00:46.018 INFO:teuthology.orchestra.run.smithi104.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.93757.log.gz 2023-06-10T19:00:46.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140860.log 2023-06-10T19:00:46.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146850.log: 88.5%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.118471.log.gz--verbose 2023-06-10T19:00:46.018 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.168148.log 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140860.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.146850.log.gz 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146914.log 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168148.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.140860.log.gz 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118935.log 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146914.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.168148.log.gz 2023-06-10T19:00:46.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146075.log 2023-06-10T19:00:46.020 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146914.log.gz 2023-06-10T19:00:46.020 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114905.log 2023-06-10T19:00:46.020 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118935.log.gz 2023-06-10T19:00:46.020 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140701.log 2023-06-10T19:00:46.020 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146075.log: /var/log/ceph/ceph-client.admin.114905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108292.log 2023-06-10T19:00:46.021 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.146075.log.gz 2023-06-10T19:00:46.021 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140701.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114905.log.gz 2023-06-10T19:00:46.021 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121214.log 2023-06-10T19:00:46.021 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140701.log.gz 2023-06-10T19:00:46.021 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119165.log 2023-06-10T19:00:46.022 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108292.log.gz 2023-06-10T19:00:46.022 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151157.log 2023-06-10T19:00:46.022 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121214.log: /var/log/ceph/ceph-client.admin.119165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116116.log 2023-06-10T19:00:46.022 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155322.log 2023-06-10T19:00:46.022 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119165.log.gz 2023-06-10T19:00:46.023 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121214.log.gz 2023-06-10T19:00:46.023 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109432.log 2023-06-10T19:00:46.023 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151157.log.gz 2023-06-10T19:00:46.023 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155322.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116116.log.gz 2023-06-10T19:00:46.023 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134509.log 2023-06-10T19:00:46.024 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155322.log.gz 2023-06-10T19:00:46.024 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110813.log 2023-06-10T19:00:46.024 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134509.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109432.log.gz 2023-06-10T19:00:46.024 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153565.log 2023-06-10T19:00:46.024 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110813.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.134509.log.gz 2023-06-10T19:00:46.025 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128130.log 2023-06-10T19:00:46.025 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110813.log.gz 2023-06-10T19:00:46.025 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132839.log 2023-06-10T19:00:46.025 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128130.log: 88.9%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.153565.log.gz--verbose 2023-06-10T19:00:46.025 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.139578.log 2023-06-10T19:00:46.026 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106904.log 2023-06-10T19:00:46.026 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.128130.log.gz 2023-06-10T19:00:46.026 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.132839.log.gz 2023-06-10T19:00:46.026 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128246.log 2023-06-10T19:00:46.026 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106904.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139578.log.gz 2023-06-10T19:00:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122642.log 2023-06-10T19:00:46.027 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106904.log.gz 2023-06-10T19:00:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123014.log 2023-06-10T19:00:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122642.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128246.log.gz 2023-06-10T19:00:46.027 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136870.log 2023-06-10T19:00:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123014.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.122642.log.gz 2023-06-10T19:00:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-06-10T19:00:46.028 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123014.log.gz 2023-06-10T19:00:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114067.log 2023-06-10T19:00:46.028 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.136870.log.gz 2023-06-10T19:00:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136087.log 2023-06-10T19:00:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123544.log 2023-06-10T19:00:46.029 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114067.log.gz 2023-06-10T19:00:46.029 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120212.log 2023-06-10T19:00:46.029 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.136087.log.gz 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124806.log 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123544.log: /var/log/ceph/ceph-client.admin.120212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139189.log 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.123544.log.gz 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124806.log: gzip 88.3%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.120212.log.gz/var/log/ceph/ceph-client.admin.109795.log 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:46.030 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139189.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124806.log.gz 2023-06-10T19:00:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139756.log 2023-06-10T19:00:46.031 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.139189.log.gz 2023-06-10T19:00:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113331.log 2023-06-10T19:00:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130489.log 2023-06-10T19:00:46.031 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139756.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109795.log.gz 2023-06-10T19:00:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113331.log: 87.3%gzip -- replaced with /var/log/ceph/ceph.log.gz 2023-06-10T19:00:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.149504.log 2023-06-10T19:00:46.032 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.139756.log.gz 2023-06-10T19:00:46.032 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113331.log.gz 2023-06-10T19:00:46.032 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142198.log 2023-06-10T19:00:46.033 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.130489.log.gz 2023-06-10T19:00:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105908.log 2023-06-10T19:00:46.033 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149504.log.gz 2023-06-10T19:00:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118287.log 2023-06-10T19:00:46.033 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142198.log: /var/log/ceph/ceph-client.admin.105908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114021.log 2023-06-10T19:00:46.034 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.142198.log.gz 2023-06-10T19:00:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118287.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105908.log.gz 2023-06-10T19:00:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163098.log 2023-06-10T19:00:46.034 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118287.log.gz 2023-06-10T19:00:46.034 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110167.log 2023-06-10T19:00:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109803.log 2023-06-10T19:00:46.035 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114021.log.gz 2023-06-10T19:00:46.035 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.163098.log.gz 2023-06-10T19:00:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116480.log 2023-06-10T19:00:46.035 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115453.log 2023-06-10T19:00:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109803.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110167.log.gz 2023-06-10T19:00:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106369.log 2023-06-10T19:00:46.036 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109803.log.gz 2023-06-10T19:00:46.036 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9%/var/log/ceph/ceph-client.admin.115453.log: -- replaced with /var/log/ceph/ceph-client.admin.116480.log.gz 2023-06-10T19:00:46.036 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135726.log 2023-06-10T19:00:46.037 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115453.log.gz 2023-06-10T19:00:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138333.log 2023-06-10T19:00:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106369.log: /var/log/ceph/ceph-client.admin.135726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160915.log 2023-06-10T19:00:46.037 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106369.log.gz 2023-06-10T19:00:46.037 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135726.log.gz 2023-06-10T19:00:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109228.log 2023-06-10T19:00:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153201.log 2023-06-10T19:00:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160915.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.138333.log.gz 2023-06-10T19:00:46.038 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146666.log 2023-06-10T19:00:46.038 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160915.log.gz 2023-06-10T19:00:46.039 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109228.log.gz 2023-06-10T19:00:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124668.log 2023-06-10T19:00:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.165779.log 2023-06-10T19:00:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146666.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153201.log.gz 2023-06-10T19:00:46.039 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129268.log 2023-06-10T19:00:46.040 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146666.log.gz 2023-06-10T19:00:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.165779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133815.log 87.8% 2023-06-10T19:00:46.040 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.124668.log.gz 2023-06-10T19:00:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158841.log 2023-06-10T19:00:46.040 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133815.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129268.log.gz 2023-06-10T19:00:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131719.log 2023-06-10T19:00:46.041 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.0.165779.log.gz 2023-06-10T19:00:46.041 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.133815.log.gz 2023-06-10T19:00:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155508.log 2023-06-10T19:00:46.041 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122312.log 2023-06-10T19:00:46.042 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.158841.log.gz 2023-06-10T19:00:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155508.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.131719.log.gz 2023-06-10T19:00:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115177.log 2023-06-10T19:00:46.042 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155508.log.gz 2023-06-10T19:00:46.042 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169180.log 2023-06-10T19:00:46.043 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122312.log.gz 2023-06-10T19:00:46.043 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134187.log 2023-06-10T19:00:46.043 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169180.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115177.log.gz 2023-06-10T19:00:46.043 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159224.log 2023-06-10T19:00:46.043 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.169180.log.gz 2023-06-10T19:00:46.044 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135496.log 2023-06-10T19:00:46.044 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.134187.log.gz 2023-06-10T19:00:46.044 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116940.log 2023-06-10T19:00:46.044 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127482.log 2023-06-10T19:00:46.044 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159224.log.gz 2023-06-10T19:00:46.045 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116940.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135496.log.gz 2023-06-10T19:00:46.045 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115729.log 2023-06-10T19:00:46.045 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.116940.log.gz 2023-06-10T19:00:46.045 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118613.log 2023-06-10T19:00:46.045 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168194.log 2023-06-10T19:00:46.046 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127482.log.gz 2023-06-10T19:00:46.046 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118613.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.115729.log.gz 2023-06-10T19:00:46.046 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153926.log 2023-06-10T19:00:46.046 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.118613.log.gz 2023-06-10T19:00:46.046 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165070.log 2023-06-10T19:00:46.047 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168194.log: /var/log/ceph/ceph-client.admin.153926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142605.log 2023-06-10T19:00:46.047 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.168194.log.gz 2023-06-10T19:00:46.047 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165070.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153926.log.gz 2023-06-10T19:00:46.047 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152484.log 2023-06-10T19:00:46.047 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142605.log: /var/log/ceph/ceph-client.admin.147969.log 2023-06-10T19:00:46.048 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.165070.log.gz 2023-06-10T19:00:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133723.log 2023-06-10T19:00:46.048 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142605.log.gz 2023-06-10T19:00:46.048 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.152484.log.gz 2023-06-10T19:00:46.048 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141052.log 2023-06-10T19:00:46.049 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147969.log: /var/log/ceph/ceph-client.admin.133723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131627.log 2023-06-10T19:00:46.049 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147969.log.gz 2023-06-10T19:00:46.049 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141052.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.133723.log.gz 2023-06-10T19:00:46.049 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158546.log 2023-06-10T19:00:46.049 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.141052.log.gz 2023-06-10T19:00:46.050 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126257.log 2023-06-10T19:00:46.050 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133336.log 2023-06-10T19:00:46.050 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.131627.log.gz 2023-06-10T19:00:46.050 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126257.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158546.log.gz 2023-06-10T19:00:46.050 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139327.log 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.126257.log.gz 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140005.log 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133336.log: /var/log/ceph/ceph-client.admin.139327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143019.log 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.133336.log.gz 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140005.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139327.log.gz 2023-06-10T19:00:46.051 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134555.log 2023-06-10T19:00:46.052 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140005.log.gz 2023-06-10T19:00:46.052 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133069.log 2023-06-10T19:00:46.052 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143019.log.gz 2023-06-10T19:00:46.052 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124622.log 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134555.log: /var/log/ceph/ceph-client.admin.133069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124373.log 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134555.log.gz 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124622.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.133069.log.gz 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143923.log 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137849.log 2023-06-10T19:00:46.053 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124373.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124622.log.gz 2023-06-10T19:00:46.054 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143829.log 2023-06-10T19:00:46.054 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124373.log.gz 2023-06-10T19:00:46.054 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143923.log.gz 2023-06-10T19:00:46.054 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158592.log 2023-06-10T19:00:46.054 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137849.log: /var/log/ceph/ceph-client.admin.143829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149458.log 2023-06-10T19:00:46.055 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.137849.log.gz 2023-06-10T19:00:46.055 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158592.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.143829.log.gz 2023-06-10T19:00:46.055 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136640.log 2023-06-10T19:00:46.055 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158592.log.gz 2023-06-10T19:00:46.055 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147735.log 2023-06-10T19:00:46.056 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160708.log 2023-06-10T19:00:46.056 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149458.log.gz 2023-06-10T19:00:46.056 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.136640.log.gz 2023-06-10T19:00:46.056 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126579.log 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.147735.log.gz 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127168.log 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140960.log 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.160708.log.gz 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127168.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126579.log.gz 2023-06-10T19:00:46.057 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128084.log 2023-06-10T19:00:46.058 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127168.log.gz 2023-06-10T19:00:46.058 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167833.log 2023-06-10T19:00:46.058 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168332.log 2023-06-10T19:00:46.058 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.140960.log.gz 2023-06-10T19:00:46.058 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157516.log 2023-06-10T19:00:46.059 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.128084.log.gz 2023-06-10T19:00:46.059 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168332.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.167833.log.gz 2023-06-10T19:00:46.059 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142789.log 2023-06-10T19:00:46.059 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.168332.log.gz 2023-06-10T19:00:46.059 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159728.log 2023-06-10T19:00:46.060 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2%/var/log/ceph/ceph-client.admin.142789.log: -- replaced with /var/log/ceph/ceph-client.admin.157516.log.gz 2023-06-10T19:00:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108998.log 2023-06-10T19:00:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159728.log: gzip 89.5%-5 -- replaced with /var/log/ceph/ceph-client.admin.142789.log.gz--verbose 2023-06-10T19:00:46.060 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.140517.log 2023-06-10T19:00:46.061 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159728.log.gz 2023-06-10T19:00:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126034.log 2023-06-10T19:00:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140517.log: gzip 89.7%-5 -- replaced with /var/log/ceph/ceph-client.admin.108998.log.gz--verbose 2023-06-10T19:00:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.127808.log 2023-06-10T19:00:46.061 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126034.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140517.log.gz 2023-06-10T19:00:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134095.log 2023-06-10T19:00:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127808.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.126034.log.gz 2023-06-10T19:00:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-06-10T19:00:46.062 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127808.log.gz 2023-06-10T19:00:46.062 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119119.log 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.0.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.134095.log.gz 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152691.log 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126395.log 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119119.log.gz 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136824.log 2023-06-10T19:00:46.063 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152691.log.gz 2023-06-10T19:00:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126441.log 2023-06-10T19:00:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126395.log: /var/log/ceph/ceph-client.admin.136824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141948.log 2023-06-10T19:00:46.064 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.126395.log.gz 2023-06-10T19:00:46.064 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126441.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.136824.log.gz 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170032.log 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141948.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126441.log.gz 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141375.log 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.141948.log.gz 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.170032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142697.log 2023-06-10T19:00:46.065 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165208.log 2023-06-10T19:00:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141375.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.170032.log.gz 2023-06-10T19:00:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142697.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.141375.log.gz 2023-06-10T19:00:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165208.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.142697.log.gz 2023-06-10T19:00:46.066 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155854.log 2023-06-10T19:00:46.066 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.165208.log.gz 2023-06-10T19:00:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154294.log 2023-06-10T19:00:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137523.log 2023-06-10T19:00:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154294.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155854.log.gz 2023-06-10T19:00:46.067 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162736.log 2023-06-10T19:00:46.067 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154294.log.gz 2023-06-10T19:00:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122082.log 2023-06-10T19:00:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151343.log 2023-06-10T19:00:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162736.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137523.log.gz 2023-06-10T19:00:46.068 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168470.log 2023-06-10T19:00:46.068 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162736.log.gz 2023-06-10T19:00:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151343.log: 87.7%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.122082.log.gz--verbose 2023-06-10T19:00:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.153519.log 2023-06-10T19:00:46.069 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151343.log.gz 2023-06-10T19:00:46.069 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142927.log 2023-06-10T19:00:46.069 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.168470.log.gz 2023-06-10T19:00:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:gzip /var/log/ceph/ceph-client.admin.153519.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.145757.log 2023-06-10T19:00:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160142.log 2023-06-10T19:00:46.070 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153519.log.gz 2023-06-10T19:00:46.070 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142927.log.gz 2023-06-10T19:00:46.070 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122220.log 2023-06-10T19:00:46.071 INFO:teuthology.orchestra.run.smithi104.stderr:gzip /var/log/ceph/ceph-client.admin.160142.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.153339.log 2023-06-10T19:00:46.071 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145757.log.gz 2023-06-10T19:00:46.071 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149320.log 2023-06-10T19:00:46.071 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153339.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.160142.log.gz 2023-06-10T19:00:46.071 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122220.log.gz 2023-06-10T19:00:46.072 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153339.log.gz 2023-06-10T19:00:46.072 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149320.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.149320.log.gz 2023-06-10T19:00:46.072 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123861.log 2023-06-10T19:00:46.072 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118889.log 2023-06-10T19:00:46.072 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.123861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153065.log 2023-06-10T19:00:46.073 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147505.log 2023-06-10T19:00:46.073 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153065.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118889.log.gz 2023-06-10T19:00:46.073 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147505.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.153065.log.gz 2023-06-10T19:00:46.073 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.147505.log.gz 2023-06-10T19:00:46.073 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120848.log 2023-06-10T19:00:46.074 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148903.log 2023-06-10T19:00:46.074 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107096.log 2023-06-10T19:00:46.074 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148903.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.120848.log.gz 2023-06-10T19:00:46.074 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.148903.log.gz 2023-06-10T19:00:46.074 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.107096.log: 92.4% -- replaced with /var/log/ceph/ceph-client.0.123861.log.gz 2023-06-10T19:00:46.075 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107096.log.gz 2023-06-10T19:00:46.075 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169686.log 2023-06-10T19:00:46.075 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109791.log 2023-06-10T19:00:46.075 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145803.log 2023-06-10T19:00:46.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109791.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.169686.log.gz 2023-06-10T19:00:46.076 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123106.log 2023-06-10T19:00:46.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145803.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109791.log.gz 2023-06-10T19:00:46.076 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110168.log 2023-06-10T19:00:46.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123106.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.145803.log.gz 2023-06-10T19:00:46.077 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166301.log 2023-06-10T19:00:46.077 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123106.log.gz 2023-06-10T19:00:46.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132885.log 2023-06-10T19:00:46.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166301.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110168.log.gz 2023-06-10T19:00:46.077 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148245.log 2023-06-10T19:00:46.078 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.166301.log.gz 2023-06-10T19:00:46.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123428.log 2023-06-10T19:00:46.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148245.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.132885.log.gz 2023-06-10T19:00:46.078 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-06-10T19:00:46.078 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148245.log.gz 2023-06-10T19:00:46.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139465.log 2023-06-10T19:00:46.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mds.a.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123428.log.gz 2023-06-10T19:00:46.079 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142040.log 2023-06-10T19:00:46.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168286.log 2023-06-10T19:00:46.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142040.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.139465.log.gz 2023-06-10T19:00:46.080 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142040.log.gz 2023-06-10T19:00:46.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168286.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.168286.log.gz 2023-06-10T19:00:46.080 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120689.log 2023-06-10T19:00:46.080 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115269.log 2023-06-10T19:00:46.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155738.log 2023-06-10T19:00:46.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115269.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120689.log.gz 2023-06-10T19:00:46.081 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.115269.log.gz 2023-06-10T19:00:46.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155738.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.155738.log.gz 2023-06-10T19:00:46.081 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160096.log 2023-06-10T19:00:46.081 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137709.log 2023-06-10T19:00:46.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118379.log 2023-06-10T19:00:46.082 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.160096.log.gz 2023-06-10T19:00:46.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137709.log: /var/log/ceph/ceph-client.admin.118379.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137709.log.gz 2023-06-10T19:00:46.082 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118379.log.gz 2023-06-10T19:00:46.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140379.log 2023-06-10T19:00:46.083 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165958.log 2023-06-10T19:00:46.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141743.log 2023-06-10T19:00:46.083 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140379.log.gz 2023-06-10T19:00:46.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156705.log 2023-06-10T19:00:46.083 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108587.log 2023-06-10T19:00:46.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141743.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.165958.log.gz 2023-06-10T19:00:46.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107042.log 2023-06-10T19:00:46.084 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141743.log.gz 2023-06-10T19:00:46.084 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156705.log.gz 2023-06-10T19:00:46.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108587.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108587.log.gz 2023-06-10T19:00:46.085 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132701.log 2023-06-10T19:00:46.085 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164876.log 2023-06-10T19:00:46.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.107042.log: /var/log/ceph/ceph-client.admin.132701.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.107042.log.gz 2023-06-10T19:00:46.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164876.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.132701.log.gz 2023-06-10T19:00:46.085 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.164876.log.gz 2023-06-10T19:00:46.086 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120894.log 2023-06-10T19:00:46.086 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113423.log 2023-06-10T19:00:46.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158887.log 2023-06-10T19:00:46.086 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120894.log.gz 2023-06-10T19:00:46.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143783.log 2023-06-10T19:00:46.087 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136732.log 2023-06-10T19:00:46.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158887.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113423.log.gz 2023-06-10T19:00:46.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108403.log 2023-06-10T19:00:46.087 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.158887.log.gz 2023-06-10T19:00:46.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136732.log: gzip -5 --verbose 89.7%-- /var/log/ceph/ceph-client.admin.136494.log 2023-06-10T19:00:46.088 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.143783.log.gz 2023-06-10T19:00:46.088 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.136732.log.gz 2023-06-10T19:00:46.088 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108403.log: /var/log/ceph/ceph-client.admin.136494.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108403.log.gz 2023-06-10T19:00:46.088 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136494.log.gz 2023-06-10T19:00:46.088 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122920.log 2023-06-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140655.log 2023-06-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135220.log 2023-06-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.122920.log.gz 2023-06-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139143.log 2023-06-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156977.log 2023-06-10T19:00:46.090 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135220.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.140655.log.gz 2023-06-10T19:00:46.090 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139143.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.135220.log.gz 2023-06-10T19:00:46.090 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156977.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139143.log.gz 2023-06-10T19:00:46.090 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156977.log.gz 2023-06-10T19:00:46.090 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141697.log 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133474.log 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120028.log 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141697.log.gz 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134003.log 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161099.log 2023-06-10T19:00:46.091 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133474.log.gz 2023-06-10T19:00:46.092 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120028.log: /var/log/ceph/ceph-client.admin.134003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167787.log 2023-06-10T19:00:46.092 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120028.log.gz 2023-06-10T19:00:46.092 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161099.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134003.log.gz 2023-06-10T19:00:46.092 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137146.log 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161099.log.gz 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113054.log 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137146.log: 88.4%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.167787.log.gz--verbose 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr:-- /var/log/ceph/ceph-client.admin.158979.log 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113054.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137146.log.gz 2023-06-10T19:00:46.093 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.113054.log.gz 2023-06-10T19:00:46.094 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158979.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158979.log.gz 2023-06-10T19:00:46.094 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130351.log 2023-06-10T19:00:46.094 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146712.log 2023-06-10T19:00:46.094 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165116.log 2023-06-10T19:00:46.094 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146712.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130351.log.gz 2023-06-10T19:00:46.095 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165116.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146712.log.gz 2023-06-10T19:00:46.095 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.165116.log.gz 2023-06-10T19:00:46.095 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147411.log 2023-06-10T19:00:46.095 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134049.log 2023-06-10T19:00:46.095 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155268.log 2023-06-10T19:00:46.096 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134049.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.147411.log.gz 2023-06-10T19:00:46.096 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155268.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.134049.log.gz 2023-06-10T19:00:46.096 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116254.log 2023-06-10T19:00:46.096 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.155268.log.gz 2023-06-10T19:00:46.096 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131255.log 2023-06-10T19:00:46.097 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128836.log 2023-06-10T19:00:46.097 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131255.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116254.log.gz 2023-06-10T19:00:46.097 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128836.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.131255.log.gz 2023-06-10T19:00:46.097 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.128836.log.gz 2023-06-10T19:00:46.097 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124948.log 2023-06-10T19:00:46.098 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114654.log 2023-06-10T19:00:46.098 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127344.log 2023-06-10T19:00:46.098 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114654.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.124948.log.gz 2023-06-10T19:00:46.098 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127344.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.114654.log.gz 2023-06-10T19:00:46.098 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126349.log 2023-06-10T19:00:46.099 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.127344.log.gz 2023-06-10T19:00:46.099 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159682.log 2023-06-10T19:00:46.099 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113377.log 2023-06-10T19:00:46.099 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159682.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.126349.log.gz 2023-06-10T19:00:46.099 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113377.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159682.log.gz 2023-06-10T19:00:46.100 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113377.log.gz 2023-06-10T19:00:46.100 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117944.log 2023-06-10T19:00:46.100 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122466.log 2023-06-10T19:00:46.100 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156659.log 2023-06-10T19:00:46.100 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122466.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.117944.log.gz 2023-06-10T19:00:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156659.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.122466.log.gz 2023-06-10T19:00:46.101 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.156659.log.gz 2023-06-10T19:00:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121760.log 2023-06-10T19:00:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114113.log 2023-06-10T19:00:46.101 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154248.log 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114113.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121760.log.gz 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154248.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114113.log.gz 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154248.log.gz 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109430.log 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-06-10T19:00:46.102 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146167.log 2023-06-10T19:00:46.103 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.3.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109430.log.gz 2023-06-10T19:00:46.103 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107142.log 2023-06-10T19:00:46.103 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146167.log.gz 2023-06-10T19:00:46.103 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147643.log 2023-06-10T19:00:46.103 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.107142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146961.log 2023-06-10T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107142.log.gz 2023-06-10T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115085.log 2023-06-10T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147643.log.gz 2023-06-10T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114451.log 2023-06-10T19:00:46.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115085.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146961.log.gz 2023-06-10T19:00:46.105 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.115085.log.gz 2023-06-10T19:00:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135995.log 2023-06-10T19:00:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140609.log 2023-06-10T19:00:46.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135995.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114451.log.gz 2023-06-10T19:00:46.105 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135995.log.gz 2023-06-10T19:00:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140609.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140609.log.gz 2023-06-10T19:00:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121032.log 2023-06-10T19:00:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108633.log 2023-06-10T19:00:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121032.log: /var/log/ceph/ceph-client.admin.108633.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121032.log.gz 2023-06-10T19:00:46.106 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108633.log.gz 2023-06-10T19:00:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131535.log 2023-06-10T19:00:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169824.log 2023-06-10T19:00:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151483.log 2023-06-10T19:00:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169824.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.131535.log.gz 2023-06-10T19:00:46.107 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151483.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.169824.log.gz 2023-06-10T19:00:46.108 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.151483.log.gz 2023-06-10T19:00:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151575.log 2023-06-10T19:00:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163547.log 2023-06-10T19:00:46.108 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130581.log 2023-06-10T19:00:46.108 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.151575.log.gz 2023-06-10T19:00:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163547.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163547.log.gz 2023-06-10T19:00:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154497.log 2023-06-10T19:00:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168562.log 2023-06-10T19:00:46.109 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8%/var/log/ceph/ceph-client.admin.154497.log: -- replaced with /var/log/ceph/ceph-client.admin.130581.log.gz 2023-06-10T19:00:46.109 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168562.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.154497.log.gz 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.168562.log.gz 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139959.log 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142559.log 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149684.log 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139959.log.gz 2023-06-10T19:00:46.110 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106102.log 2023-06-10T19:00:46.111 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142559.log.gz 2023-06-10T19:00:46.111 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149684.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149684.log.gz 2023-06-10T19:00:46.111 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106102.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106102.log.gz 2023-06-10T19:00:46.111 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169640.log 2023-06-10T19:00:46.111 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120507.log 2023-06-10T19:00:46.112 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169594.log 2023-06-10T19:00:46.112 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.169640.log.gz 2023-06-10T19:00:46.112 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120507.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120507.log.gz 2023-06-10T19:00:46.112 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137431.log 2023-06-10T19:00:46.112 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164734.log 2023-06-10T19:00:46.113 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137431.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.169594.log.gz 2023-06-10T19:00:46.113 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164734.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137431.log.gz 2023-06-10T19:00:46.113 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.164734.log.gz 2023-06-10T19:00:46.113 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.152305.log 2023-06-10T19:00:46.113 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130305.log 2023-06-10T19:00:46.114 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.152305.log: /var/log/ceph/ceph-client.admin.130305.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.130305.log.gz 2023-06-10T19:00:46.114 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137757.log 2023-06-10T19:00:46.114 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119823.log 2023-06-10T19:00:46.114 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137757.log: /var/log/ceph/ceph-client.admin.119823.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.137757.log.gz 2023-06-10T19:00:46.114 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.119823.log.gz 2023-06-10T19:00:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151529.log 2023-06-10T19:00:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120986.log 2023-06-10T19:00:46.115 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151529.log: /var/log/ceph/ceph-client.admin.120986.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151529.log.gz 2023-06-10T19:00:46.115 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.0.152305.log.gz 2023-06-10T19:00:46.115 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120986.log.gz 2023-06-10T19:00:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154589.log 2023-06-10T19:00:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.138650.log 2023-06-10T19:00:46.116 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154589.log: /var/log/ceph/ceph-client.0.138650.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154589.log.gz 2023-06-10T19:00:46.117 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.117671.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-06-10T19:00:46.118 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.1.117671.log.gz 2023-06-10T19:00:46.118 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.112756.log 2023-06-10T19:00:46.118 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.112756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100156.log 2023-06-10T19:00:46.118 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.1.112756.log.gz 2023-06-10T19:00:46.119 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.100156.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100156.log.gz 2023-06-10T19:00:46.119 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.113355.log 2023-06-10T19:00:46.119 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.113355.log: 91.1% -- replaced with /var/log/ceph/ceph-client.1.113355.log.gz 2023-06-10T19:00:46.119 INFO:teuthology.orchestra.run.smithi114.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-06-10T19:00:46.140 INFO:teuthology.orchestra.run.smithi114.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-06-10T19:00:46.157 INFO:teuthology.orchestra.run.smithi114.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-06-10T19:00:46.180 INFO:teuthology.orchestra.run.smithi104.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.138650.log.gz 2023-06-10T19:00:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117170.log 2023-06-10T19:00:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126855.log 2023-06-10T19:00:46.181 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117170.log: /var/log/ceph/ceph-client.admin.126855.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.117170.log.gz 2023-06-10T19:00:46.181 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126855.log.gz 2023-06-10T19:00:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113101.log 2023-06-10T19:00:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149822.log 2023-06-10T19:00:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118333.log 2023-06-10T19:00:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149822.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113101.log.gz 2023-06-10T19:00:46.182 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106462.log 2023-06-10T19:00:46.183 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149822.log.gz 2023-06-10T19:00:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118333.log: /var/log/ceph/ceph-client.admin.106462.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118333.log.gz 2023-06-10T19:00:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100310.log 2023-06-10T19:00:46.183 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106462.log.gz 2023-06-10T19:00:46.183 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134463.log 2023-06-10T19:00:46.184 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.100310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143737.log 2023-06-10T19:00:46.184 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100310.log.gz 2023-06-10T19:00:46.184 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134463.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134463.log.gz 2023-06-10T19:00:46.184 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155692.log 2023-06-10T19:00:46.184 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106600.log 2023-06-10T19:00:46.185 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155692.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143737.log.gz 2023-06-10T19:00:46.185 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.155692.log.gz 2023-06-10T19:00:46.185 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159866.log 2023-06-10T19:00:46.185 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150367.log 2023-06-10T19:00:46.185 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159866.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106600.log.gz 2023-06-10T19:00:46.186 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.159866.log.gz 2023-06-10T19:00:46.186 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150367.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150367.log.gz 2023-06-10T19:00:46.186 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148407.log 2023-06-10T19:00:46.186 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141902.log 2023-06-10T19:00:46.186 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148407.log: /var/log/ceph/ceph-client.admin.141902.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.148407.log.gz 2023-06-10T19:00:46.187 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.141902.log.gz 2023-06-10T19:00:46.187 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147877.log 2023-06-10T19:00:46.187 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114267.log 2023-06-10T19:00:46.187 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109435.log 2023-06-10T19:00:46.187 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147877.log.gz 2023-06-10T19:00:46.188 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114267.log: /var/log/ceph/ceph-client.admin.109435.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114267.log.gz 2023-06-10T19:00:46.188 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138033.log 2023-06-10T19:00:46.188 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109435.log.gz 2023-06-10T19:00:46.188 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128038.log 2023-06-10T19:00:46.188 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130832.log 2023-06-10T19:00:46.189 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138033.log.gz 2023-06-10T19:00:46.189 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128038.log: /var/log/ceph/ceph-client.admin.130832.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.128038.log.gz 2023-06-10T19:00:46.189 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130832.log.gz 2023-06-10T19:00:46.189 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114497.log 2023-06-10T19:00:46.189 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126809.log 2023-06-10T19:00:46.190 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126671.log 2023-06-10T19:00:46.190 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114497.log.gz 2023-06-10T19:00:46.190 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126809.log: /var/log/ceph/ceph-client.admin.126671.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126809.log.gz 2023-06-10T19:00:46.190 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.126671.log.gz 2023-06-10T19:00:46.190 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130786.log 2023-06-10T19:00:46.191 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159588.log 2023-06-10T19:00:46.191 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130786.log: /var/log/ceph/ceph-client.admin.159588.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130786.log.gz 2023-06-10T19:00:46.191 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159588.log.gz 2023-06-10T19:00:46.191 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133631.log 2023-06-10T19:00:46.191 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144578.log 2023-06-10T19:00:46.192 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133631.log: /var/log/ceph/ceph-client.admin.144578.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.133631.log.gz 2023-06-10T19:00:46.192 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.144578.log.gz 2023-06-10T19:00:46.192 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122556.log 2023-06-10T19:00:46.192 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131673.log 2023-06-10T19:00:46.192 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124327.log 2023-06-10T19:00:46.193 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131673.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122556.log.gz 2023-06-10T19:00:46.193 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124327.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131673.log.gz 2023-06-10T19:00:46.193 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124327.log.gz 2023-06-10T19:00:46.193 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140906.log 2023-06-10T19:00:46.193 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150979.log 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129333.log 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140906.log.gz 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150979.log: /var/log/ceph/ceph-client.admin.129333.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150979.log.gz 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.129333.log.gz 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123198.log 2023-06-10T19:00:46.194 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117308.log 2023-06-10T19:00:46.195 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106508.log 2023-06-10T19:00:46.195 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117308.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.123198.log.gz 2023-06-10T19:00:46.195 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117308.log.gz 2023-06-10T19:00:46.195 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144370.log 2023-06-10T19:00:46.195 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144186.log 2023-06-10T19:00:46.196 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144370.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106508.log.gz 2023-06-10T19:00:46.196 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144370.log.gz 2023-06-10T19:00:46.196 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121257.log 2023-06-10T19:00:46.196 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122734.log 2023-06-10T19:00:46.196 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.144186.log.gz 2023-06-10T19:00:46.197 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168102.log 2023-06-10T19:00:46.197 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122734.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121257.log.gz 2023-06-10T19:00:46.197 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.122734.log.gz 2023-06-10T19:00:46.197 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168102.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.168102.log.gz 2023-06-10T19:00:46.197 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159494.log 2023-06-10T19:00:46.198 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109798.log 2023-06-10T19:00:46.198 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159494.log: /var/log/ceph/ceph-client.admin.109798.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.159494.log.gz 2023-06-10T19:00:46.198 INFO:teuthology.orchestra.run.smithi104.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109798.log.gz 2023-06-10T19:00:46.198 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124760.log 2023-06-10T19:00:46.198 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115913.log 2023-06-10T19:00:46.199 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153972.log 2023-06-10T19:00:46.199 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.124760.log.gz 2023-06-10T19:00:46.199 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115913.log: /var/log/ceph/ceph-client.admin.153972.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115913.log.gz 2023-06-10T19:00:46.199 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130259.log 2023-06-10T19:00:46.199 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153972.log.gz 2023-06-10T19:00:46.200 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107188.log 2023-06-10T19:00:46.200 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110491.log 2023-06-10T19:00:46.200 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1%/var/log/ceph/ceph-client.admin.107188.log: -- replaced with /var/log/ceph/ceph-client.admin.130259.log.gz 2023-06-10T19:00:46.200 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110491.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110491.log.gz 2023-06-10T19:00:46.200 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155414.log 2023-06-10T19:00:46.201 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.107188.log.gz 2023-06-10T19:00:46.201 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117354.log 2023-06-10T19:00:46.201 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156843.log 2023-06-10T19:00:46.201 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.155414.log.gz 2023-06-10T19:00:46.201 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134233.log 2023-06-10T19:00:46.202 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117354.log.gz 2023-06-10T19:00:46.202 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156843.log: /var/log/ceph/ceph-client.admin.134233.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156843.log.gz 2023-06-10T19:00:46.202 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.134233.log.gz 2023-06-10T19:00:46.202 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124419.log 2023-06-10T19:00:46.202 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169456.log 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140189.log 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124419.log.gz 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169456.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.169456.log.gz 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140471.log 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126533.log 2023-06-10T19:00:46.203 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140471.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140189.log.gz 2023-06-10T19:00:46.204 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140471.log.gz 2023-06-10T19:00:46.204 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125942.log 2023-06-10T19:00:46.204 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169548.log 2023-06-10T19:00:46.204 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126533.log.gz 2023-06-10T19:00:46.204 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125942.log: /var/log/ceph/ceph-client.admin.169548.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125942.log.gz 2023-06-10T19:00:46.205 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.169548.log.gz 2023-06-10T19:00:46.205 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132561.log 2023-06-10T19:00:46.205 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130694.log 2023-06-10T19:00:46.205 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163455.log 2023-06-10T19:00:46.205 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.132561.log.gz 2023-06-10T19:00:46.206 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130694.log: /var/log/ceph/ceph-client.admin.163455.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.130694.log.gz 2023-06-10T19:00:46.206 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.163455.log.gz 2023-06-10T19:00:46.206 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-06-10T19:00:46.206 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140333.log 2023-06-10T19:00:46.206 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148997.log 2023-06-10T19:00:46.207 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140333.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140333.log.gz 2023-06-10T19:00:46.207 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148997.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148997.log.gz 2023-06-10T19:00:46.207 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154358.log 2023-06-10T19:00:46.207 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121530.log 2023-06-10T19:00:46.207 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154358.log: /var/log/ceph/ceph-client.admin.121530.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154358.log.gz 2023-06-10T19:00:46.208 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121530.log.gz 2023-06-10T19:00:46.208 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163052.log 2023-06-10T19:00:46.208 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118128.log 2023-06-10T19:00:46.208 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163052.log: /var/log/ceph/ceph-client.admin.118128.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.163052.log.gz 2023-06-10T19:00:46.208 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118128.log.gz 2023-06-10T19:00:46.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158216.log 2023-06-10T19:00:46.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133023.log 2023-06-10T19:00:46.209 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158216.log: /var/log/ceph/ceph-client.admin.133023.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158216.log.gz 2023-06-10T19:00:46.209 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133023.log.gz 2023-06-10T19:00:46.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110629.log 2023-06-10T19:00:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114159.log 2023-06-10T19:00:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110629.log: /var/log/ceph/ceph-client.admin.114159.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110629.log.gz 2023-06-10T19:00:46.210 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114159.log.gz 2023-06-10T19:00:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142290.log 2023-06-10T19:00:46.210 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137008.log 2023-06-10T19:00:46.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142290.log: /var/log/ceph/ceph-client.admin.137008.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.142290.log.gz 2023-06-10T19:00:46.211 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.137008.log.gz 2023-06-10T19:00:46.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106950.log 2023-06-10T19:00:46.211 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158684.log 2023-06-10T19:00:46.211 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106950.log: /var/log/ceph/ceph-client.admin.158684.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106950.log.gz 2023-06-10T19:00:46.212 INFO:teuthology.orchestra.run.smithi104.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158684.log.gz 2023-06-10T19:00:46.212 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153880.log 2023-06-10T19:00:46.212 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133861.log 2023-06-10T19:00:46.212 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153880.log: /var/log/ceph/ceph-client.admin.133861.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.153880.log.gz 2023-06-10T19:00:46.212 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133861.log.gz 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106192.log 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141605.log 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106192.log: /var/log/ceph/ceph-client.admin.141605.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106192.log.gz 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141605.log.gz 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114543.log 2023-06-10T19:00:46.213 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122358.log 2023-06-10T19:00:46.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114543.log: /var/log/ceph/ceph-client.admin.122358.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114543.log.gz 2023-06-10T19:00:46.214 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.122358.log.gz 2023-06-10T19:00:46.214 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157562.log 2023-06-10T19:00:46.214 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110171.log 2023-06-10T19:00:46.214 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157562.log: /var/log/ceph/ceph-client.admin.110171.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157562.log.gz 2023-06-10T19:00:46.215 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110171.log.gz 2023-06-10T19:00:46.215 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153788.log 2023-06-10T19:00:46.215 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125444.log 2023-06-10T19:00:46.215 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153788.log: /var/log/ceph/ceph-client.admin.125444.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.153788.log.gz 2023-06-10T19:00:46.215 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.125444.log.gz 2023-06-10T19:00:46.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118981.log 2023-06-10T19:00:46.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109182.log 2023-06-10T19:00:46.216 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118981.log: /var/log/ceph/ceph-client.admin.109182.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118981.log.gz 2023-06-10T19:00:46.216 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109182.log.gz 2023-06-10T19:00:46.216 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163936.log 2023-06-10T19:00:46.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115499.log 2023-06-10T19:00:46.217 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163936.log: /var/log/ceph/ceph-client.admin.115499.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.163936.log.gz 2023-06-10T19:00:46.217 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.115499.log.gz 2023-06-10T19:00:46.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116894.log 2023-06-10T19:00:46.217 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140143.log 2023-06-10T19:00:46.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116894.log: /var/log/ceph/ceph-client.admin.140143.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.116894.log.gz 2023-06-10T19:00:46.218 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140143.log.gz 2023-06-10T19:00:46.218 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154543.log 2023-06-10T19:00:46.218 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170076.log 2023-06-10T19:00:46.218 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154543.log: /var/log/ceph/ceph-client.admin.170076.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154543.log.gz 2023-06-10T19:00:46.219 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.170076.log.gz 2023-06-10T19:00:46.219 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125896.log 2023-06-10T19:00:46.219 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116756.log 2023-06-10T19:00:46.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125896.log: /var/log/ceph/ceph-client.admin.116756.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.125896.log.gz 2023-06-10T19:00:46.219 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116756.log.gz 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128973.log 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146758.log 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128973.log: /var/log/ceph/ceph-client.admin.146758.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.128973.log.gz 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.146758.log.gz 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120461.log 2023-06-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-06-10T19:00:46.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135588.log 2023-06-10T19:00:46.221 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120461.log: /var/log/ceph/ceph-client.admin.135588.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.120461.log.gz 2023-06-10T19:00:46.221 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135588.log.gz 2023-06-10T19:00:46.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167305.log 2023-06-10T19:00:46.221 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130167.log 2023-06-10T19:00:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154743.log 2023-06-10T19:00:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130167.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.167305.log.gz 2023-06-10T19:00:46.222 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.130167.log.gz 2023-06-10T19:00:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150781.log 2023-06-10T19:00:46.222 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147321.log 2023-06-10T19:00:46.223 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5%/var/log/ceph/ceph-client.admin.150781.log: -- replaced with /var/log/ceph/ceph-client.admin.154743.log.gz 2023-06-10T19:00:46.223 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150781.log.gz 2023-06-10T19:00:46.223 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116300.log 2023-06-10T19:00:46.223 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153385.log 2023-06-10T19:00:46.223 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2%/var/log/ceph/ceph-client.admin.116300.log: -- replaced with /var/log/ceph/ceph-client.admin.147321.log.gz 2023-06-10T19:00:46.224 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116300.log.gz 2023-06-10T19:00:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153385.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153385.log.gz 2023-06-10T19:00:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131903.log 2023-06-10T19:00:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166412.log 2023-06-10T19:00:46.224 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131903.log: /var/log/ceph/ceph-client.admin.166412.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131903.log.gz 2023-06-10T19:00:46.225 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.166412.log.gz 2023-06-10T19:00:46.225 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142382.log 2023-06-10T19:00:46.225 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147120.log 2023-06-10T19:00:46.225 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162625.log 2023-06-10T19:00:46.225 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147120.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142382.log.gz 2023-06-10T19:00:46.226 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162625.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147120.log.gz 2023-06-10T19:00:46.226 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162625.log.gz 2023-06-10T19:00:46.226 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136594.log 2023-06-10T19:00:46.226 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162920.log 2023-06-10T19:00:46.226 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155462.log 2023-06-10T19:00:46.227 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136594.log.gz 2023-06-10T19:00:46.227 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162920.log: /var/log/ceph/ceph-client.admin.155462.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.162920.log.gz 2023-06-10T19:00:46.227 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.155462.log.gz 2023-06-10T19:00:46.227 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117584.log 2023-06-10T19:00:46.227 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155554.log 2023-06-10T19:00:46.228 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127854.log 2023-06-10T19:00:46.228 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155554.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117584.log.gz 2023-06-10T19:00:46.228 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.155554.log.gz 2023-06-10T19:00:46.228 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150597.log 2023-06-10T19:00:46.228 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142651.log 2023-06-10T19:00:46.229 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150597.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127854.log.gz 2023-06-10T19:00:46.229 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.150597.log.gz 2023-06-10T19:00:46.229 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139419.log 2023-06-10T19:00:46.229 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169272.log 2023-06-10T19:00:46.229 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142651.log.gz 2023-06-10T19:00:46.230 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.139419.log.gz 2023-06-10T19:00:46.230 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.169272.log.gz 2023-06-10T19:00:46.230 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121668.log 2023-06-10T19:00:46.230 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161237.log 2023-06-10T19:00:46.230 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121668.log: 92.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161237.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121668.log.gz 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161237.log.gz 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115821.log 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127622.log 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145374.log 2023-06-10T19:00:46.231 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115821.log.gz 2023-06-10T19:00:46.232 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113791.log 2023-06-10T19:00:46.232 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127946.log 2023-06-10T19:00:46.232 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145374.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127622.log.gz 2023-06-10T19:00:46.232 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162690.log 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145374.log.gz 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147689.log 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113791.log.gz 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162690.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127946.log.gz 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161191.log 2023-06-10T19:00:46.233 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159958.log 2023-06-10T19:00:46.234 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.162690.log.gz 2023-06-10T19:00:46.234 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132655.log 2023-06-10T19:00:46.234 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.147689.log.gz 2023-06-10T19:00:46.234 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159958.log: gzip 88.4%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.161191.log.gz/var/log/ceph/ceph-client.admin.129517.log 2023-06-10T19:00:46.234 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:46.235 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132655.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.159958.log.gz 2023-06-10T19:00:46.235 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129517.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.132655.log.gz 2023-06-10T19:00:46.235 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.129517.log.gz 2023-06-10T19:00:46.235 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135266.log 2023-06-10T19:00:46.235 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114859.log 2023-06-10T19:00:46.236 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144462.log 2023-06-10T19:00:46.236 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114859.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135266.log.gz 2023-06-10T19:00:46.236 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114859.log.gz 2023-06-10T19:00:46.236 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144462.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144462.log.gz 2023-06-10T19:00:46.236 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152689.log 2023-06-10T19:00:46.237 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133949.log 2023-06-10T19:00:46.237 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149776.log 2023-06-10T19:00:46.237 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5%/var/log/ceph/ceph-client.admin.133949.log: -- replaced with /var/log/ceph/ceph-client.admin.152689.log.gz 2023-06-10T19:00:46.237 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149776.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.133949.log.gz 2023-06-10T19:00:46.237 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123060.log 2023-06-10T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149776.log.gz 2023-06-10T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.119644.log 2023-06-10T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125716.log 2023-06-10T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123060.log: /var/log/ceph/ceph-client.0.119644.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.123060.log.gz 2023-06-10T19:00:46.238 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125716.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125716.log.gz 2023-06-10T19:00:46.239 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142835.log 2023-06-10T19:00:46.239 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.0.119644.log.gz 2023-06-10T19:00:46.239 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115131.log 2023-06-10T19:00:46.239 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121622.log 2023-06-10T19:00:46.239 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142835.log: /var/log/ceph/ceph-client.admin.115131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155368.log 2023-06-10T19:00:46.240 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142835.log.gz 2023-06-10T19:00:46.240 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139051.log 2023-06-10T19:00:46.240 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115131.log.gz 2023-06-10T19:00:46.240 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121622.log.gz 2023-06-10T19:00:46.240 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155368.log: /var/log/ceph/ceph-client.admin.139051.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155368.log.gz 2023-06-10T19:00:46.241 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139051.log.gz 2023-06-10T19:00:46.241 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110424.log 2023-06-10T19:00:46.241 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150505.log 2023-06-10T19:00:46.241 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165462.log 2023-06-10T19:00:46.241 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110424.log.gz 2023-06-10T19:00:46.242 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150505.log: /var/log/ceph/ceph-client.admin.165462.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150505.log.gz 2023-06-10T19:00:46.242 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.165462.log.gz 2023-06-10T19:00:46.242 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122872.log 2023-06-10T19:00:46.242 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146259.log 2023-06-10T19:00:46.242 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116024.log 2023-06-10T19:00:46.243 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122872.log.gz 2023-06-10T19:00:46.243 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148015.log 2023-06-10T19:00:46.243 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162441.log 2023-06-10T19:00:46.243 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116024.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146259.log.gz 2023-06-10T19:00:46.243 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163639.log 2023-06-10T19:00:46.244 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.116024.log.gz 2023-06-10T19:00:46.244 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162441.log: gzip 88.1%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.148015.log.gz/var/log/ceph/ceph-client.admin.163501.log 2023-06-10T19:00:46.244 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:46.244 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163639.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.162441.log.gz-5 2023-06-10T19:00:46.244 INFO:teuthology.orchestra.run.smithi104.stderr:--verbose -- /var/log/ceph/ceph-client.admin.135818.log 2023-06-10T19:00:46.245 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4%/var/log/ceph/ceph-client.admin.163501.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.163639.log.gz 2023-06-10T19:00:46.245 INFO:teuthology.orchestra.run.smithi104.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.144117.log 2023-06-10T19:00:46.245 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106763.log 2023-06-10T19:00:46.245 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.163501.log.gz 2023-06-10T19:00:46.245 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144117.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135818.log.gz 2023-06-10T19:00:46.246 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137987.log 2023-06-10T19:00:46.246 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144117.log.gz 2023-06-10T19:00:46.246 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129425.log 2023-06-10T19:00:46.246 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137987.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106763.log.gz 2023-06-10T19:00:46.246 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110169.log 2023-06-10T19:00:46.247 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137987.log.gz 2023-06-10T19:00:46.247 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115683.log 2023-06-10T19:00:46.247 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114313.log 2023-06-10T19:00:46.247 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129425.log.gz 2023-06-10T19:00:46.247 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110169.log.gz 2023-06-10T19:00:46.248 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118425.log 2023-06-10T19:00:46.248 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115683.log.gz 2023-06-10T19:00:46.248 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159134.log 2023-06-10T19:00:46.248 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117216.log 2023-06-10T19:00:46.248 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118425.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.114313.log.gz 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143130.log 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.118425.log.gz 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117262.log 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.159134.log.gz 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117216.log.gz 2023-06-10T19:00:46.249 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164220.log 2023-06-10T19:00:46.250 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138171.log 2023-06-10T19:00:46.250 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143130.log.gz 2023-06-10T19:00:46.250 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129806.log 2023-06-10T19:00:46.250 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117262.log.gz 2023-06-10T19:00:46.250 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138171.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.164220.log.gz 2023-06-10T19:00:46.251 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139802.log 2023-06-10T19:00:46.251 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.138171.log.gz 2023-06-10T19:00:46.251 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151734.log 2023-06-10T19:00:46.251 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139802.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129806.log.gz 2023-06-10T19:00:46.251 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142973.log 2023-06-10T19:00:46.252 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.139802.log.gz 2023-06-10T19:00:46.252 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.156171.log 2023-06-10T19:00:46.252 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.151734.log.gz 2023-06-10T19:00:46.252 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110583.log 2023-06-10T19:00:46.252 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.156171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111181.log 2023-06-10T19:00:46.253 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142973.log.gz 2023-06-10T19:00:46.253 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146436.log 2023-06-10T19:00:46.253 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110583.log.gz 2023-06-10T19:00:46.253 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116986.log 2023-06-10T19:00:46.253 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111181.log.gz 2023-06-10T19:00:46.254 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136540.log 2023-06-10T19:00:46.254 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146436.log.gz 2023-06-10T19:00:46.254 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116986.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-06-10T19:00:46.254 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136540.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116986.log.gz 2023-06-10T19:00:46.254 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136540.log.gz 2023-06-10T19:00:46.255 INFO:teuthology.orchestra.run.smithi104.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.156171.log.gz 2023-06-10T19:00:46.255 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135864.log 2023-06-10T19:00:46.255 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-06-10T19:00:46.255 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110675.log 2023-06-10T19:00:46.255 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114993.log 2023-06-10T19:00:46.256 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116162.log 2023-06-10T19:00:46.256 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135864.log.gz 2023-06-10T19:00:46.256 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9%/var/log/ceph/ceph-client.admin.114993.log: -- replaced with /var/log/ceph/ceph-client.admin.110675.log.gz 2023-06-10T19:00:46.256 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151780.log 2023-06-10T19:00:46.256 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114993.log.gz 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167561.log 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151780.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116162.log.gz 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167561.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151780.log.gz 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.160529.log 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.167561.log.gz 2023-06-10T19:00:46.257 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135036.log 2023-06-10T19:00:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153473.log 2023-06-10T19:00:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.160529.log: /var/log/ceph/ceph-client.admin.135036.log: /var/log/ceph/ceph-client.admin.153473.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135036.log.gz 2023-06-10T19:00:46.258 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.153473.log.gz 2023-06-10T19:00:46.258 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139005.log 2023-06-10T19:00:46.258 INFO:teuthology.orchestra.run.smithi104.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.0.160529.log.gz 2023-06-10T19:00:46.259 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131489.log 2023-06-10T19:00:46.259 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107258.log 2023-06-10T19:00:46.259 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.139005.log.gz 2023-06-10T19:00:46.259 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161053.log 2023-06-10T19:00:46.259 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141467.log 2023-06-10T19:00:46.260 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.107258.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.131489.log.gz 2023-06-10T19:00:46.260 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128882.log 2023-06-10T19:00:46.260 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107258.log.gz 2023-06-10T19:00:46.260 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148153.log 88.9% 2023-06-10T19:00:46.260 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161053.log.gz 2023-06-10T19:00:46.261 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128882.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141467.log.gz 2023-06-10T19:00:46.261 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148153.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128882.log.gz 2023-06-10T19:00:46.261 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.148153.log.gz 2023-06-10T19:00:46.261 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148199.log 2023-06-10T19:00:46.261 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121170.log 2023-06-10T19:00:46.262 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166684.log 2023-06-10T19:00:46.262 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.148199.log.gz 2023-06-10T19:00:46.262 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163593.log 2023-06-10T19:00:46.262 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166684.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121170.log.gz 2023-06-10T19:00:46.262 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163593.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.166684.log.gz 2023-06-10T19:00:46.263 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.163593.log.gz 2023-06-10T19:00:46.263 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153155.log 2023-06-10T19:00:46.263 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135450.log 2023-06-10T19:00:46.263 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124511.log 2023-06-10T19:00:46.263 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153155.log.gz 2023-06-10T19:00:46.264 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166366.log 2023-06-10T19:00:46.264 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162533.log 2023-06-10T19:00:46.264 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124511.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135450.log.gz 2023-06-10T19:00:46.264 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143222.log 2023-06-10T19:00:46.264 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124511.log.gz 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141651.log 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166366.log.gz 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.162533.log.gz 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143222.log: /var/log/ceph/ceph-client.admin.141651.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143222.log.gz 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122266.log 2023-06-10T19:00:46.265 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141651.log.gz 2023-06-10T19:00:46.266 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108246.log 2023-06-10T19:00:46.266 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120802.log 2023-06-10T19:00:46.266 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160961.log 2023-06-10T19:00:46.266 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122266.log.gz 2023-06-10T19:00:46.266 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120802.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108246.log.gz 2023-06-10T19:00:46.267 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151688.log 2023-06-10T19:00:46.267 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.120802.log.gz 2023-06-10T19:00:46.267 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147923.log 2023-06-10T19:00:46.267 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151688.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160961.log.gz 2023-06-10T19:00:46.267 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139097.log 2023-06-10T19:00:46.268 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151688.log.gz 2023-06-10T19:00:46.268 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115867.log 2023-06-10T19:00:46.268 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118797.log 2023-06-10T19:00:46.268 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139097.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.147923.log.gz 2023-06-10T19:00:46.268 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160848.log 2023-06-10T19:00:46.269 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.139097.log.gz 2023-06-10T19:00:46.269 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164688.log 2023-06-10T19:00:46.269 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115867.log.gz 2023-06-10T19:00:46.269 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160848.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118797.log.gz 2023-06-10T19:00:46.269 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164688.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160848.log.gz 2023-06-10T19:00:46.270 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.164688.log.gz 2023-06-10T19:00:46.270 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140563.log 2023-06-10T19:00:46.270 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130121.log 2023-06-10T19:00:46.270 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148061.log 2023-06-10T19:00:46.270 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140563.log.gz 2023-06-10T19:00:46.271 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130121.log: /var/log/ceph/ceph-client.admin.148061.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.130121.log.gz 2023-06-10T19:00:46.271 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147783.log 2023-06-10T19:00:46.271 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.148061.log.gz 2023-06-10T19:00:46.271 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.144895.log 2023-06-10T19:00:46.271 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164128.log 2023-06-10T19:00:46.272 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.144895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119963.log 2023-06-10T19:00:46.272 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.147783.log.gz 2023-06-10T19:00:46.272 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164128.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.164128.log.gz 2023-06-10T19:00:46.272 INFO:teuthology.orchestra.run.smithi104.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.0.144895.log.gz 2023-06-10T19:00:46.272 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133249.log 2023-06-10T19:00:46.273 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134417.log 2023-06-10T19:00:46.273 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.119963.log.gz 2023-06-10T19:00:46.273 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165162.log 2023-06-10T19:00:46.273 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159025.log 2023-06-10T19:00:46.273 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134417.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.133249.log.gz 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131765.log 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.134417.log.gz 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115545.log 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.165162.log.gz 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131765.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159025.log.gz 2023-06-10T19:00:46.274 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115545.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.131765.log.gz 2023-06-10T19:00:46.275 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115545.log.gz 2023-06-10T19:00:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167741.log 2023-06-10T19:00:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121990.log 2023-06-10T19:00:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137803.log 2023-06-10T19:00:46.275 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155646.log 2023-06-10T19:00:46.276 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.167741.log.gz 2023-06-10T19:00:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137803.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121990.log.gz 2023-06-10T19:00:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140277.log 2023-06-10T19:00:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155646.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.137803.log.gz 2023-06-10T19:00:46.276 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143268.log 2023-06-10T19:00:46.277 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5%/var/log/ceph/ceph-client.admin.140277.log: -- replaced with /var/log/ceph/ceph-client.admin.155646.log.gz 2023-06-10T19:00:46.277 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149999.log 2023-06-10T19:00:46.277 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147551.log 2023-06-10T19:00:46.277 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140277.log.gz 2023-06-10T19:00:46.277 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149999.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.143268.log.gz 2023-06-10T19:00:46.278 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149999.log.gz 2023-06-10T19:00:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147551.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.147551.log.gz 2023-06-10T19:00:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154156.log 2023-06-10T19:00:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145328.log 2023-06-10T19:00:46.278 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169962.log 2023-06-10T19:00:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145328.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154156.log.gz 2023-06-10T19:00:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169962.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145328.log.gz 2023-06-10T19:00:46.279 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.169962.log.gz 2023-06-10T19:00:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106277.log 2023-06-10T19:00:46.279 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113653.log 2023-06-10T19:00:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126717.log 2023-06-10T19:00:46.280 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106277.log.gz 2023-06-10T19:00:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121852.log 2023-06-10T19:00:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158308.log 2023-06-10T19:00:46.280 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126717.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113653.log.gz 2023-06-10T19:00:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159820.log 2023-06-10T19:00:46.281 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.126717.log.gz 2023-06-10T19:00:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158308.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.121852.log.gz 2023-06-10T19:00:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158262.log 2023-06-10T19:00:46.281 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158308.log.gz 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121576.log 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159820.log.gz 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151988.log 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148107.log 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158262.log.gz 2023-06-10T19:00:46.282 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121576.log: /var/log/ceph/ceph-client.admin.151988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169410.log 2023-06-10T19:00:46.283 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121576.log.gz 2023-06-10T19:00:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.148107.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.151988.log.gzgzip 2023-06-10T19:00:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.158638.log 2023-06-10T19:00:46.283 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148107.log.gz 2023-06-10T19:00:46.283 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169410.log: /var/log/ceph/ceph-client.admin.158638.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.169410.log.gz 2023-06-10T19:00:46.284 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158638.log.gz 2023-06-10T19:00:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.132382.log 2023-06-10T19:00:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126303.log 2023-06-10T19:00:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.132382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134601.log 2023-06-10T19:00:46.284 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155218.log 2023-06-10T19:00:46.285 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.126303.log.gz 2023-06-10T19:00:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134601.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134601.log.gz 2023-06-10T19:00:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113837.log 2023-06-10T19:00:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155218.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.132382.log.gz 2023-06-10T19:00:46.285 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142336.log 2023-06-10T19:00:46.286 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113837.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.155218.log.gz 2023-06-10T19:00:46.286 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113837.log.gz 2023-06-10T19:00:46.286 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142336.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142336.log.gz 2023-06-10T19:00:46.286 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154405.log 2023-06-10T19:00:46.286 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149550.log 2023-06-10T19:00:46.287 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126080.log 2023-06-10T19:00:46.287 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146482.log 2023-06-10T19:00:46.287 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154405.log.gz 2023-06-10T19:00:46.287 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149550.log.gz 2023-06-10T19:00:46.287 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143469.log 2023-06-10T19:00:46.288 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166209.log 2023-06-10T19:00:46.288 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146482.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126080.log.gz 2023-06-10T19:00:46.288 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166255.log 2023-06-10T19:00:46.288 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146482.log.gz 2023-06-10T19:00:46.288 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115959.log 2023-06-10T19:00:46.289 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143469.log.gz 2023-06-10T19:00:46.289 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166255.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.166209.log.gz 2023-06-10T19:00:46.289 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115959.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.166255.log.gz 2023-06-10T19:00:46.289 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115959.log.gz 2023-06-10T19:00:46.289 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144508.log 2023-06-10T19:00:46.290 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168424.log 2023-06-10T19:00:46.290 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168832.log 2023-06-10T19:00:46.290 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125578.log 2023-06-10T19:00:46.290 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.168424.log.gz 2023-06-10T19:00:46.290 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168832.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.144508.log.gz 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.168832.log.gz 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132747.log 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129171.log 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125578.log.gz 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113147.log 2023-06-10T19:00:46.291 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120369.log 2023-06-10T19:00:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129171.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132747.log.gz 2023-06-10T19:00:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120258.log 2023-06-10T19:00:46.292 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129171.log.gz 2023-06-10T19:00:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120369.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113147.log.gz 2023-06-10T19:00:46.292 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158795.log 2023-06-10T19:00:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120258.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.120369.log.gz 2023-06-10T19:00:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147831.log 2023-06-10T19:00:46.293 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120258.log.gz 2023-06-10T19:00:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136041.log 2023-06-10T19:00:46.293 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142428.log 2023-06-10T19:00:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147831.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.158795.log.gz 2023-06-10T19:00:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136041.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147831.log.gz 2023-06-10T19:00:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142428.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136041.log.gz 2023-06-10T19:00:46.294 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142428.log.gz 2023-06-10T19:00:46.294 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109426.log 2023-06-10T19:00:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125670.log 2023-06-10T19:00:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129563.log 2023-06-10T19:00:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136916.log 2023-06-10T19:00:46.295 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109426.log.gz 2023-06-10T19:00:46.295 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129563.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.125670.log.gz 2023-06-10T19:00:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150137.log 2023-06-10T19:00:46.296 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129563.log.gz 2023-06-10T19:00:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127574.log 2023-06-10T19:00:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169318.log 2023-06-10T19:00:46.296 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150137.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.136916.log.gz 2023-06-10T19:00:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157608.log 2023-06-10T19:00:46.297 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150137.log.gz 2023-06-10T19:00:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169318.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127574.log.gz 2023-06-10T19:00:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129714.log 2023-06-10T19:00:46.297 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157608.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.169318.log.gz 2023-06-10T19:00:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117630.log 2023-06-10T19:00:46.298 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157608.log.gz 2023-06-10T19:00:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106717.log 2023-06-10T19:00:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134785.log 2023-06-10T19:00:46.298 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117630.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129714.log.gz 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106717.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.117630.log.gz 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.126947.log 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129944.log 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134785.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.106717.log.gz 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126947.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134785.log.gz 2023-06-10T19:00:46.299 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129944.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.126947.log.gz 2023-06-10T19:00:46.300 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.129944.log.gz 2023-06-10T19:00:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145941.log 2023-06-10T19:00:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145282.log 2023-06-10T19:00:46.300 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116434.log 2023-06-10T19:00:46.300 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145941.log.gz 2023-06-10T19:00:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165346.log 2023-06-10T19:00:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149953.log 2023-06-10T19:00:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116434.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145282.log.gz 2023-06-10T19:00:46.301 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146344.log 2023-06-10T19:00:46.301 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116434.log.gz 2023-06-10T19:00:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149953.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.165346.log.gz 2023-06-10T19:00:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.157294.log 2023-06-10T19:00:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146344.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149953.log.gz 2023-06-10T19:00:46.302 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.157294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146390.log 2023-06-10T19:00:46.302 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146344.log.gz 2023-06-10T19:00:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153016.log 2023-06-10T19:00:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146390.log: 87.9% -- replaced with /var/log/ceph/ceph-client.0.157294.log.gz 2023-06-10T19:00:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153016.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146390.log.gz 2023-06-10T19:00:46.303 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153016.log.gz 2023-06-10T19:00:46.303 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136225.log 2023-06-10T19:00:46.304 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159636.log 2023-06-10T19:00:46.304 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120553.log 2023-06-10T19:00:46.304 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144324.log 2023-06-10T19:00:46.304 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.136225.log.gz 2023-06-10T19:00:46.304 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120553.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.159636.log.gz 2023-06-10T19:00:46.305 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114608.log 2023-06-10T19:00:46.305 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120553.log.gz 2023-06-10T19:00:46.305 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150735.log 2023-06-10T19:00:46.305 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131581.log 2023-06-10T19:00:46.305 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144324.log.gz 2023-06-10T19:00:46.306 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114608.log.gz 2023-06-10T19:00:46.306 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163844.log 2023-06-10T19:00:46.306 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150735.log.gz 2023-06-10T19:00:46.306 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158039.log 2023-06-10T19:00:46.306 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131581.log.gz 2023-06-10T19:00:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129379.log 2023-06-10T19:00:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158039.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.163844.log.gz 2023-06-10T19:00:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129379.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.158039.log.gz 2023-06-10T19:00:46.307 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129379.log.gz 2023-06-10T19:00:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145895.log 2023-06-10T19:00:46.308 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141329.log 2023-06-10T19:00:46.308 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149412.log 2023-06-10T19:00:46.308 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.145895.log.gz 2023-06-10T19:00:46.308 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149638.log 2023-06-10T19:00:46.308 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109433.log 2023-06-10T19:00:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149412.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141329.log.gz 2023-06-10T19:00:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143691.log 2023-06-10T19:00:46.309 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.149412.log.gz 2023-06-10T19:00:46.309 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149638.log.gz 2023-06-10T19:00:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136310.log 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109433.log.gz 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150183.log 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143691.log.gz 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106996.log 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.136310.log.gz 2023-06-10T19:00:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120756.log 2023-06-10T19:00:46.311 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9%/var/log/ceph/ceph-client.admin.106996.log: -- replaced with /var/log/ceph/ceph-client.admin.150183.log.gz 2023-06-10T19:00:46.311 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106996.log.gz 2023-06-10T19:00:46.311 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120756.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120756.log.gz 2023-06-10T19:00:46.311 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158933.log 2023-06-10T19:00:46.311 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120940.log 2023-06-10T19:00:46.312 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120120.log 2023-06-10T19:00:46.312 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.158933.log.gz 2023-06-10T19:00:46.312 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120940.log: /var/log/ceph/ceph-client.admin.120120.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120940.log.gz 2023-06-10T19:00:46.312 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120120.log.gz 2023-06-10T19:00:46.312 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150689.log 2023-06-10T19:00:46.313 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163685.log 2023-06-10T19:00:46.313 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169088.log 2023-06-10T19:00:46.313 INFO:teuthology.orchestra.run.smithi104.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.150689.log.gz 2023-06-10T19:00:46.313 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120415.log 2023-06-10T19:00:46.313 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169732.log 2023-06-10T19:00:46.314 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169088.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.163685.log.gz 2023-06-10T19:00:46.314 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141559.log 2023-06-10T19:00:46.314 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.169088.log.gz 2023-06-10T19:00:46.314 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120166.log 2023-06-10T19:00:46.314 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120415.log.gz 2023-06-10T19:00:46.315 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141559.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.169732.log.gz 2023-06-10T19:00:46.315 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.141559.log.gz 2023-06-10T19:00:46.315 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153657.log 2023-06-10T19:00:46.315 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126211.log 2023-06-10T19:00:46.315 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153657.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.120166.log.gz 2023-06-10T19:00:46.316 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133161.log 2023-06-10T19:00:46.316 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153657.log.gz 2023-06-10T19:00:46.316 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126211.log: /var/log/ceph/ceph-client.admin.133161.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126211.log.gz 2023-06-10T19:00:46.316 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133161.log.gz 2023-06-10T19:00:46.316 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116618.log 2023-06-10T19:00:46.317 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134141.log 2023-06-10T19:00:46.317 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157855.log 2023-06-10T19:00:46.317 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116618.log.gz 2023-06-10T19:00:46.317 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146213.log 2023-06-10T19:00:46.317 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134141.log.gz 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157855.log: /var/log/ceph/ceph-client.admin.146213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147074.log 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157855.log.gz 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146213.log.gz 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134371.log 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143083.log 2023-06-10T19:00:46.318 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147074.log: /var/log/ceph/ceph-client.admin.134371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110951.log 2023-06-10T19:00:46.319 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.147074.log.gz 2023-06-10T19:00:46.319 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.134371.log.gz 2023-06-10T19:00:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143083.log: /var/log/ceph/ceph-client.admin.110951.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143083.log.gz 2023-06-10T19:00:46.319 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110951.log.gz 2023-06-10T19:00:46.319 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150889.log 2023-06-10T19:00:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135404.log 2023-06-10T19:00:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150889.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-06-10T19:00:46.320 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.150889.log.gz 2023-06-10T19:00:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135404.log: /var/log/ceph/ceph-osd.1.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135404.log.gz 2023-06-10T19:00:46.320 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169364.log 2023-06-10T19:00:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128742.log 2023-06-10T19:00:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139848.log 2023-06-10T19:00:46.321 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.169364.log.gz 2023-06-10T19:00:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128742.log: /var/log/ceph/ceph-client.admin.139848.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128742.log.gz 2023-06-10T19:00:46.321 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151642.log 2023-06-10T19:00:46.322 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.139848.log.gz 2023-06-10T19:00:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122688.log 2023-06-10T19:00:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116802.log 2023-06-10T19:00:46.322 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.151642.log.gz 2023-06-10T19:00:46.322 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155170.log 2023-06-10T19:00:46.323 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122688.log.gz 2023-06-10T19:00:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116802.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116802.log.gz 2023-06-10T19:00:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169226.log 2023-06-10T19:00:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113975.log 2023-06-10T19:00:46.323 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169226.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155170.log.gz 2023-06-10T19:00:46.324 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113975.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.169226.log.gz 2023-06-10T19:00:46.324 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121806.log 2023-06-10T19:00:46.324 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.113975.log.gz 2023-06-10T19:00:46.324 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118036.log 2023-06-10T19:00:46.324 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141190.log 2023-06-10T19:00:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118036.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121806.log.gz 2023-06-10T19:00:46.325 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118036.log.gz 2023-06-10T19:00:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147597.log 2023-06-10T19:00:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121382.log 2023-06-10T19:00:46.325 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141190.log: /var/log/ceph/ceph-client.admin.147597.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141190.log.gz 2023-06-10T19:00:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121382.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147597.log.gz 2023-06-10T19:00:46.326 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121382.log.gz 2023-06-10T19:00:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118567.log 2023-06-10T19:00:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152970.log 2023-06-10T19:00:46.326 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163409.log 2023-06-10T19:00:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152970.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118567.log.gz 2023-06-10T19:00:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163409.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.152970.log.gz 2023-06-10T19:00:46.327 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163409.log.gz 2023-06-10T19:00:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165024.log 2023-06-10T19:00:46.327 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115775.log 2023-06-10T19:00:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163190.log 2023-06-10T19:00:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.165024.log.gz 2023-06-10T19:00:46.328 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115775.log.gz 2023-06-10T19:00:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159448.log 2023-06-10T19:00:46.328 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110537.log 2023-06-10T19:00:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159448.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.163190.log.gz 2023-06-10T19:00:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131949.log 2023-06-10T19:00:46.329 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.159448.log.gz 2023-06-10T19:00:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168694.log 2023-06-10T19:00:46.329 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131949.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110537.log.gz 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126763.log 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.131949.log.gz 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164828.log 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.168694.log.gz 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126763.log: /var/log/ceph/ceph-client.admin.164828.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126763.log.gz 2023-06-10T19:00:46.330 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154451.log 2023-06-10T19:00:46.331 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.164828.log.gz 2023-06-10T19:00:46.331 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108952.log 2023-06-10T19:00:46.331 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111043.log 2023-06-10T19:00:46.331 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154451.log: /var/log/ceph/ceph-client.admin.108952.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154451.log.gz 2023-06-10T19:00:46.331 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111043.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108952.log.gz 2023-06-10T19:00:46.332 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111043.log.gz 2023-06-10T19:00:46.332 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164976.log 2023-06-10T19:00:46.332 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117722.log 2023-06-10T19:00:46.332 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156549.log 2023-06-10T19:00:46.332 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117722.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.164976.log.gz 2023-06-10T19:00:46.333 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.156549.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117722.log.gz 2023-06-10T19:00:46.333 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.156549.log.gz 2023-06-10T19:00:46.333 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127992.log 2023-06-10T19:00:46.333 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116710.log 2023-06-10T19:00:46.333 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109044.log 2023-06-10T19:00:46.334 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127992.log.gz 2023-06-10T19:00:46.334 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116710.log: /var/log/ceph/ceph-client.admin.109044.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116710.log.gz 2023-06-10T19:00:46.334 INFO:teuthology.orchestra.run.smithi104.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109044.log.gz 2023-06-10T19:00:46.334 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118751.log 2023-06-10T19:00:46.334 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155124.log 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125624.log 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155124.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118751.log.gz 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125624.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155124.log.gz 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125624.log.gz 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110767.log 2023-06-10T19:00:46.335 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.112605.log 2023-06-10T19:00:46.336 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108108.log 2023-06-10T19:00:46.336 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.2.112605.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110767.log.gz 2023-06-10T19:00:46.336 INFO:teuthology.orchestra.run.smithi104.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.2.112605.log.gz 2023-06-10T19:00:46.336 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127716.log 2023-06-10T19:00:46.336 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108108.log: /var/log/ceph/ceph-client.admin.127716.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108108.log.gz 2023-06-10T19:00:46.337 INFO:teuthology.orchestra.run.smithi104.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.127716.log.gz 2023-06-10T19:00:46.337 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114813.log 2023-06-10T19:00:46.390 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135174.log 2023-06-10T19:00:46.390 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109434.log 2023-06-10T19:00:46.390 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135174.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114813.log.gz 2023-06-10T19:00:46.391 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162487.log 2023-06-10T19:00:46.391 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135174.log.gz 2023-06-10T19:00:46.391 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109434.log: /var/log/ceph/ceph-client.admin.162487.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109434.log.gz 2023-06-10T19:00:46.391 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162487.log.gz 2023-06-10T19:00:46.391 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135312.log 2023-06-10T19:00:46.392 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145507.log 2023-06-10T19:00:46.392 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137255.log 2023-06-10T19:00:46.392 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145507.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135312.log.gz 2023-06-10T19:00:46.392 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.145507.log.gz 2023-06-10T19:00:46.392 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147212.log 2023-06-10T19:00:46.393 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109431.log 2023-06-10T19:00:46.393 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147212.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.137255.log.gz 2023-06-10T19:00:46.393 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109431.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.147212.log.gz 2023-06-10T19:00:46.393 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109431.log.gz 2023-06-10T19:00:46.393 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159540.log 2023-06-10T19:00:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136686.log 2023-06-10T19:00:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131077.log 2023-06-10T19:00:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136686.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159540.log.gz 2023-06-10T19:00:46.394 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136686.log.gz 2023-06-10T19:00:46.394 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113745.log 2023-06-10T19:00:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162782.log 2023-06-10T19:00:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113745.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131077.log.gz 2023-06-10T19:00:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162782.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113745.log.gz 2023-06-10T19:00:46.395 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.162782.log.gz 2023-06-10T19:00:46.395 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144071.log 2023-06-10T19:00:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111089.log 2023-06-10T19:00:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155078.log 2023-06-10T19:00:46.396 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9%/var/log/ceph/ceph-client.admin.111089.log: -- replaced with /var/log/ceph/ceph-client.admin.144071.log.gz 2023-06-10T19:00:46.396 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155078.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111089.log.gz 2023-06-10T19:00:46.396 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155078.log.gz 2023-06-10T19:00:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100425.log 2023-06-10T19:00:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113239.log 2023-06-10T19:00:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.100425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100004.log 2023-06-10T19:00:46.397 INFO:teuthology.orchestra.run.smithi104.stderr: 87.3%/var/log/ceph/ceph-client.admin.113239.log: -- replaced with /var/log/ceph/ceph-client.admin.100425.log.gz 2023-06-10T19:00:46.397 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.113239.log.gz 2023-06-10T19:00:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149868.log 2023-06-10T19:00:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.100004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112090.log 2023-06-10T19:00:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149868.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100004.log.gz 2023-06-10T19:00:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165254.log 2023-06-10T19:00:46.398 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149868.log.gz 2023-06-10T19:00:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.112090.log: /var/log/ceph/ceph-client.admin.165254.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112090.log.gz 2023-06-10T19:00:46.399 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165254.log.gz 2023-06-10T19:00:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133520.log 2023-06-10T19:00:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150229.log 2023-06-10T19:00:46.399 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133520.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133520.log.gz 2023-06-10T19:00:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150229.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150229.log.gz 2023-06-10T19:00:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109800.log 2023-06-10T19:00:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164596.log 2023-06-10T19:00:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144025.log 2023-06-10T19:00:46.400 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164596.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109800.log.gz 2023-06-10T19:00:46.401 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119073.log 2023-06-10T19:00:46.401 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.164596.log.gz 2023-06-10T19:00:46.401 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144025.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.144025.log.gz 2023-06-10T19:00:46.401 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135082.log 2023-06-10T19:00:46.401 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119073.log.gz 2023-06-10T19:00:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160802.log 2023-06-10T19:00:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140814.log 2023-06-10T19:00:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135082.log: /var/log/ceph/ceph-client.admin.160802.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135082.log.gz 2023-06-10T19:00:46.402 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140814.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160802.log.gz 2023-06-10T19:00:46.402 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140814.log.gz 2023-06-10T19:00:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143314.log 2023-06-10T19:00:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142513.log 2023-06-10T19:00:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168786.log 2023-06-10T19:00:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142513.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143314.log.gz 2023-06-10T19:00:46.403 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168786.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142513.log.gz 2023-06-10T19:00:46.404 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.168786.log.gz 2023-06-10T19:00:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134325.log 2023-06-10T19:00:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113699.log 2023-06-10T19:00:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120074.log 2023-06-10T19:00:46.404 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113699.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134325.log.gz 2023-06-10T19:00:46.405 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113699.log.gz 2023-06-10T19:00:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140051.log 2023-06-10T19:00:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131857.log 2023-06-10T19:00:46.405 INFO:teuthology.orchestra.run.smithi104.stderr: 88.6%/var/log/ceph/ceph-client.admin.140051.log: -- replaced with /var/log/ceph/ceph-client.admin.120074.log.gz 2023-06-10T19:00:46.405 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131857.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.140051.log.gz 2023-06-10T19:00:46.406 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.131857.log.gz 2023-06-10T19:00:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123382.log 2023-06-10T19:00:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130397.log 2023-06-10T19:00:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154202.log 2023-06-10T19:00:46.406 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130397.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.123382.log.gz 2023-06-10T19:00:46.407 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154202.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130397.log.gz 2023-06-10T19:00:46.407 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.154202.log.gz 2023-06-10T19:00:46.407 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155600.log 2023-06-10T19:00:46.407 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110721.log 2023-06-10T19:00:46.407 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117078.log 2023-06-10T19:00:46.408 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110721.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155600.log.gz 2023-06-10T19:00:46.408 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117078.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110721.log.gz 2023-06-10T19:00:46.408 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117078.log.gz 2023-06-10T19:00:46.408 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163777.log 2023-06-10T19:00:46.408 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127900.log 2023-06-10T19:00:46.409 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137345.log 2023-06-10T19:00:46.409 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127900.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.163777.log.gz 2023-06-10T19:00:46.409 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127900.log.gz 2023-06-10T19:00:46.409 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110997.log 2023-06-10T19:00:46.409 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137345.log: /var/log/ceph/ceph-client.admin.110997.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137345.log.gz 2023-06-10T19:00:46.410 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110997.log.gz 2023-06-10T19:00:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140425.log 2023-06-10T19:00:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159912.log 2023-06-10T19:00:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.140425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129471.log 2023-06-10T19:00:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159912.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.140425.log.gz 2023-06-10T19:00:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129471.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159912.log.gz 2023-06-10T19:00:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143875.log 2023-06-10T19:00:46.411 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129471.log.gz 2023-06-10T19:00:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167469.log 2023-06-10T19:00:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133382.log 2023-06-10T19:00:46.412 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143875.log.gz 2023-06-10T19:00:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167469.log: /var/log/ceph/ceph-client.admin.133382.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.167469.log.gz 2023-06-10T19:00:46.412 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.133382.log.gz 2023-06-10T19:00:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153834.log 2023-06-10T19:00:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139511.log 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163982.log 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.153834.log.gz 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139511.log: /var/log/ceph/ceph-client.admin.163982.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139511.log.gz 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163982.log.gz 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110859.log 2023-06-10T19:00:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164074.log 2023-06-10T19:00:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108062.log 2023-06-10T19:00:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164074.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110859.log.gz 2023-06-10T19:00:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108062.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.164074.log.gz 2023-06-10T19:00:46.414 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108062.log.gz 2023-06-10T19:00:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159310.log 2023-06-10T19:00:46.415 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136402.log 2023-06-10T19:00:46.415 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116572.log 2023-06-10T19:00:46.415 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.159310.log.gz 2023-06-10T19:00:46.415 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136402.log: /var/log/ceph/ceph-client.admin.116572.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.136402.log.gz 2023-06-10T19:00:46.415 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116572.log.gz 2023-06-10T19:00:46.416 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108338.log 2023-06-10T19:00:46.416 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121714.log 2023-06-10T19:00:46.416 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122968.log 2023-06-10T19:00:46.416 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121714.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108338.log.gz 2023-06-10T19:00:46.416 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121714.log.gz 2023-06-10T19:00:46.417 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161329.log 2023-06-10T19:00:46.417 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144278.log 2023-06-10T19:00:46.417 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122968.log.gz 2023-06-10T19:00:46.417 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161329.log: /var/log/ceph/ceph-client.admin.144278.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.161329.log.gz 2023-06-10T19:00:46.417 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144278.log.gz 2023-06-10T19:00:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109136.log 2023-06-10T19:00:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141513.log 2023-06-10T19:00:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167607.log 2023-06-10T19:00:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141513.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109136.log.gz 2023-06-10T19:00:46.418 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167607.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141513.log.gz 2023-06-10T19:00:46.419 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.167607.log.gz 2023-06-10T19:00:46.419 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108200.log 2023-06-10T19:00:46.419 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134944.log 2023-06-10T19:00:46.419 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145849.log 2023-06-10T19:00:46.419 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108200.log: /var/log/ceph/ceph-client.admin.134944.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108200.log.gz 2023-06-10T19:00:46.420 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145849.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134944.log.gz 2023-06-10T19:00:46.420 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145849.log.gz 2023-06-10T19:00:46.420 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160050.log 2023-06-10T19:00:46.420 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149730.log 2023-06-10T19:00:46.420 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143176.log 2023-06-10T19:00:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149730.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.160050.log.gz 2023-06-10T19:00:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143176.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149730.log.gz 2023-06-10T19:00:46.421 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143176.log.gz 2023-06-10T19:00:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157721.log 2023-06-10T19:00:46.421 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144416.log 2023-06-10T19:00:46.422 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.157721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150275.log 2023-06-10T19:00:46.422 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.144416.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157721.log.gz 2023-06-10T19:00:46.422 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150275.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144416.log.gz 2023-06-10T19:00:46.422 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150275.log.gz 2023-06-10T19:00:46.422 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159402.log 2023-06-10T19:00:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141237.log 2023-06-10T19:00:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151437.log 2023-06-10T19:00:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141237.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159402.log.gz 2023-06-10T19:00:46.423 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.141237.log.gz 2023-06-10T19:00:46.423 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126625.log 2023-06-10T19:00:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106857.log 2023-06-10T19:00:46.424 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151437.log.gz 2023-06-10T19:00:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126625.log: /var/log/ceph/ceph-client.admin.106857.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126625.log.gz 2023-06-10T19:00:46.424 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106857.log.gz 2023-06-10T19:00:46.424 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163890.log 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109090.log 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131209.log 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109090.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.163890.log.gz 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109090.log.gz 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154986.log 2023-06-10T19:00:46.425 INFO:teuthology.orchestra.run.smithi104.stderr:gzip /var/log/ceph/ceph-client.admin.131209.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.105993.log 2023-06-10T19:00:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154986.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131209.log.gz 2023-06-10T19:00:46.426 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.154986.log.gz 2023-06-10T19:00:46.426 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115591.log 2023-06-10T19:00:46.428 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.105993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164266.log 2023-06-10T19:00:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115591.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.105993.log.gz 2023-06-10T19:00:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164266.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.115591.log.gz 2023-06-10T19:00:46.429 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.164266.log.gz 2023-06-10T19:00:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164550.log 2023-06-10T19:00:46.429 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119917.log 2023-06-10T19:00:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158170.log 2023-06-10T19:00:46.430 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.164550.log.gz 2023-06-10T19:00:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119917.log: /var/log/ceph/ceph-client.admin.158170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133115.log 2023-06-10T19:00:46.430 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119917.log.gz 2023-06-10T19:00:46.430 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115315.log 2023-06-10T19:00:46.431 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158170.log.gz 2023-06-10T19:00:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106669.log 2023-06-10T19:00:46.431 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133115.log.gz 2023-06-10T19:00:46.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115315.log: /var/log/ceph/ceph-client.admin.106669.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115315.log.gz 2023-06-10T19:00:46.431 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106669.log.gz 2023-06-10T19:00:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164780.log 2023-06-10T19:00:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151203.log 2023-06-10T19:00:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141994.log 2023-06-10T19:00:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151203.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.164780.log.gz 2023-06-10T19:00:46.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141994.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151203.log.gz 2023-06-10T19:00:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155032.log 2023-06-10T19:00:46.433 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141994.log.gz 2023-06-10T19:00:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142244.log 2023-06-10T19:00:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132065.log 2023-06-10T19:00:46.433 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155032.log: /var/log/ceph/ceph-client.admin.142244.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.155032.log.gz 2023-06-10T19:00:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132065.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.142244.log.gz 2023-06-10T19:00:46.434 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.132065.log.gz 2023-06-10T19:00:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112790.log 2023-06-10T19:00:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119327.log 2023-06-10T19:00:46.434 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.112790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153703.log 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112790.log.gz 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117768.log 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119327.log.gz 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.153703.log: /var/log/ceph/ceph-client.admin.117768.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153703.log.gz 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161466.log 2023-06-10T19:00:46.435 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117768.log.gz 2023-06-10T19:00:46.436 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141856.log 2023-06-10T19:00:46.436 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147028.log 2023-06-10T19:00:46.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161466.log: /var/log/ceph/ceph-client.admin.141856.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.161466.log.gz 2023-06-10T19:00:46.436 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.147028.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141856.log.gz 2023-06-10T19:00:46.436 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147028.log.gz 2023-06-10T19:00:46.437 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152714.log 2023-06-10T19:00:46.437 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130029.log 2023-06-10T19:00:46.437 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169870.log 2023-06-10T19:00:46.437 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152714.log.gz/var/log/ceph/ceph-client.admin.130029.log: 2023-06-10T19:00:46.437 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169870.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130029.log.gz 2023-06-10T19:00:46.438 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.169870.log.gz 2023-06-10T19:00:46.438 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164642.log 2023-06-10T19:00:46.438 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141283.log 2023-06-10T19:00:46.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.162045.log 2023-06-10T19:00:46.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141283.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.164642.log.gz 2023-06-10T19:00:46.439 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141283.log.gz 2023-06-10T19:00:46.439 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117676.log 2023-06-10T19:00:46.439 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.0.162045.log: -- /var/log/ceph/ceph-client.admin.168516.log 2023-06-10T19:00:46.439 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117676.log: /var/log/ceph/ceph-client.admin.168516.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117676.log.gz 2023-06-10T19:00:46.439 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.168516.log.gz 2023-06-10T19:00:46.440 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135634.log 2023-06-10T19:00:46.440 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132977.log 2023-06-10T19:00:46.440 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135634.log: /var/log/ceph/ceph-client.admin.132977.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135634.log.gz 2023-06-10T19:00:46.440 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132977.log.gz 2023-06-10T19:00:46.440 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137895.log 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.0.162045.log.gz 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138263.log 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160004.log 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.138263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137895.log.gz 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158454.log 2023-06-10T19:00:46.441 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.160004.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.138263.log.gz 2023-06-10T19:00:46.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158454.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160004.log.gz 2023-06-10T19:00:46.442 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158454.log.gz 2023-06-10T19:00:46.442 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168740.log 2023-06-10T19:00:46.442 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150413.log 2023-06-10T19:00:46.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151249.log 2023-06-10T19:00:46.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150413.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.168740.log.gz 2023-06-10T19:00:46.443 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150413.log.gz 2023-06-10T19:00:46.443 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151111.log 2023-06-10T19:00:46.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106323.log 2023-06-10T19:00:46.443 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151111.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151249.log.gz 2023-06-10T19:00:46.444 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.106323.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151111.log.gz 2023-06-10T19:00:46.444 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164028.log 2023-06-10T19:00:46.444 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106323.log.gz 2023-06-10T19:00:46.444 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167653.log 2023-06-10T19:00:46.444 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142743.log 2023-06-10T19:00:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164028.log: /var/log/ceph/ceph-client.admin.167653.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.164028.log.gz 2023-06-10T19:00:46.445 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.167653.log.gz 2023-06-10T19:00:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134898.log 2023-06-10T19:00:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.142743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137941.log 2023-06-10T19:00:46.445 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134898.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142743.log.gz 2023-06-10T19:00:46.446 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137941.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.134898.log.gz 2023-06-10T19:00:46.446 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137941.log.gz 2023-06-10T19:00:46.446 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115407.log 2023-06-10T19:00:46.446 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169134.log 2023-06-10T19:00:46.446 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118082.log 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169134.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115407.log.gz 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118082.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.169134.log.gz 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118082.log.gz 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166550.log 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118843.log 2023-06-10T19:00:46.447 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151391.log 2023-06-10T19:00:46.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118843.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.166550.log.gz 2023-06-10T19:00:46.448 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118843.log.gz 2023-06-10T19:00:46.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113561.log 2023-06-10T19:00:46.448 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151391.log.gz 2023-06-10T19:00:46.448 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148291.log 2023-06-10T19:00:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129852.log 2023-06-10T19:00:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113561.log: /var/log/ceph/ceph-client.admin.148291.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113561.log.gz 2023-06-10T19:00:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129852.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148291.log.gz 2023-06-10T19:00:46.449 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129852.log.gz 2023-06-10T19:00:46.449 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143971.log 2023-06-10T19:00:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127390.log 2023-06-10T19:00:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124040.log 2023-06-10T19:00:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127390.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143971.log.gz 2023-06-10T19:00:46.450 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124040.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.127390.log.gz 2023-06-10T19:00:46.450 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.124040.log.gz 2023-06-10T19:00:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129898.log 2023-06-10T19:00:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146121.log 2023-06-10T19:00:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.129898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150643.log 2023-06-10T19:00:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.146121.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129898.log.gz 2023-06-10T19:00:46.451 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146121.log.gz 2023-06-10T19:00:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131995.log 2023-06-10T19:00:46.452 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150643.log.gz 2023-06-10T19:00:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134647.log 2023-06-10T19:00:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135542.log 2023-06-10T19:00:46.452 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131995.log: /var/log/ceph/ceph-client.admin.134647.log: /var/log/ceph/ceph-client.admin.135542.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134647.log.gz 2023-06-10T19:00:46.453 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.131995.log.gz 2023-06-10T19:00:46.453 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135542.log.gz 2023-06-10T19:00:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121944.log 2023-06-10T19:00:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154940.log 2023-06-10T19:00:46.453 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120642.log 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154940.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121944.log.gz 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108679.log 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154940.log.gz 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120642.log: /var/log/ceph/ceph-client.admin.108679.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120642.log.gz 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108679.log.gz 2023-06-10T19:00:46.454 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159356.log 2023-06-10T19:00:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137054.log 2023-06-10T19:00:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162395.log 2023-06-10T19:00:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137054.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.159356.log.gz 2023-06-10T19:00:46.455 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162395.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137054.log.gz 2023-06-10T19:00:46.455 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.162395.log.gz 2023-06-10T19:00:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111273.log 2023-06-10T19:00:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137100.log 2023-06-10T19:00:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169778.log 2023-06-10T19:00:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137100.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137100.log.gz 2023-06-10T19:00:46.456 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169778.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.111273.log.gz 2023-06-10T19:00:46.457 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.169778.log.gz 2023-06-10T19:00:46.457 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120323.log 2023-06-10T19:00:46.457 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157767.log 2023-06-10T19:00:46.457 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.120323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116388.log 2023-06-10T19:00:46.457 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120323.log.gz/var/log/ceph/ceph-client.admin.157767.log: 2023-06-10T19:00:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116388.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.157767.log.gz 2023-06-10T19:00:46.458 INFO:teuthology.orchestra.run.smithi104.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.116388.log.gz 2023-06-10T19:00:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168056.log 2023-06-10T19:00:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111415.log 2023-06-10T19:00:46.458 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108541.log 2023-06-10T19:00:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111415.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.168056.log.gz 2023-06-10T19:00:46.459 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111415.log.gz 2023-06-10T19:00:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136356.log 2023-06-10T19:00:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127762.log 2023-06-10T19:00:46.459 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108541.log: /var/log/ceph/ceph-client.admin.136356.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108541.log.gz 2023-06-10T19:00:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127762.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.136356.log.gz 2023-06-10T19:00:46.460 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127762.log.gz 2023-06-10T19:00:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145711.log 2023-06-10T19:00:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127258.log 2023-06-10T19:00:46.460 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149366.log 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127258.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.145711.log.gz 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.149366.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.127258.log.gz 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149366.log.gz 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116848.log 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110010.log 2023-06-10T19:00:46.461 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139235.log 2023-06-10T19:00:46.462 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110010.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116848.log.gz 2023-06-10T19:00:46.462 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110010.log.gz 2023-06-10T19:00:46.462 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129760.log 2023-06-10T19:00:46.462 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114700.log 2023-06-10T19:00:46.462 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8%/var/log/ceph/ceph-client.admin.129760.log: -- replaced with /var/log/ceph/ceph-client.admin.139235.log.gz 2023-06-10T19:00:46.463 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114700.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.129760.log.gz 2023-06-10T19:00:46.463 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114700.log.gz 2023-06-10T19:00:46.463 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169502.log 2023-06-10T19:00:46.463 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136778.log 2023-06-10T19:00:46.463 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108495.log 2023-06-10T19:00:46.464 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136778.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.169502.log.gz 2023-06-10T19:00:46.464 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.136778.log.gz 2023-06-10T19:00:46.464 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132793.log 2023-06-10T19:00:46.464 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143360.log /var/log/ceph/ceph-client.admin.108495.log: 2023-06-10T19:00:46.464 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.132793.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108495.log.gz 2023-06-10T19:00:46.465 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.132793.log.gz 2023-06-10T19:00:46.465 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113008.log 2023-06-10T19:00:46.465 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126993.log 2023-06-10T19:00:46.465 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.143360.log: /var/log/ceph/ceph-client.admin.113008.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143360.log.gz 2023-06-10T19:00:46.465 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126993.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113008.log.gz 2023-06-10T19:00:46.466 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.126993.log.gz 2023-06-10T19:00:46.466 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151826.log 2023-06-10T19:00:46.466 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158748.log 2023-06-10T19:00:46.466 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154833.log 2023-06-10T19:00:46.466 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151826.log.gz 2023-06-10T19:00:46.467 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158748.log: /var/log/ceph/ceph-client.admin.154833.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.158748.log.gz 2023-06-10T19:00:46.467 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154833.log.gz 2023-06-10T19:00:46.467 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133428.log 2023-06-10T19:00:46.467 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167001.log 2023-06-10T19:00:46.467 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118174.log 2023-06-10T19:00:46.468 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.167001.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133428.log.gz 2023-06-10T19:00:46.468 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119027.log 2023-06-10T19:00:46.468 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118174.log: /var/log/ceph/ceph-client.admin.119027.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.118174.log.gz 2023-06-10T19:00:46.468 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119027.log.gz 2023-06-10T19:00:46.468 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115039.log 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.0.167001.log.gz 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153247.log 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108768.log 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115039.log: /var/log/ceph/ceph-client.admin.153247.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115039.log.gz 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108768.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.153247.log.gz 2023-06-10T19:00:46.469 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108768.log.gz 2023-06-10T19:00:46.470 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122128.log 2023-06-10T19:00:46.470 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115361.log 2023-06-10T19:00:46.470 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154110.log 2023-06-10T19:00:46.470 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115361.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.122128.log.gz 2023-06-10T19:00:46.470 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154110.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115361.log.gz 2023-06-10T19:00:46.471 INFO:teuthology.orchestra.run.smithi104.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.154110.log.gz 2023-06-10T19:00:46.471 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115223.log 2023-06-10T19:00:46.471 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152873.log 2023-06-10T19:00:46.471 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.115223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130213.log 2023-06-10T19:00:46.471 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.152873.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115223.log.gz 2023-06-10T19:00:46.472 INFO:teuthology.orchestra.run.smithi104.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152873.log.gz 2023-06-10T19:00:46.472 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135680.log 2023-06-10T19:00:46.472 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163236.log 2023-06-10T19:00:46.472 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135680.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130213.log.gz 2023-06-10T19:00:46.472 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163236.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135680.log.gz 2023-06-10T19:00:46.473 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.163236.log.gz 2023-06-10T19:00:46.473 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165300.log 2023-06-10T19:00:46.473 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131347.log 2023-06-10T19:00:46.473 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.165300.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165300.log.gz 2023-06-10T19:00:46.473 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131347.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.131347.log.gz 2023-06-10T19:00:46.474 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130740.log 2023-06-10T19:00:46.474 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113929.log 2023-06-10T19:00:46.474 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130740.log: /var/log/ceph/ceph-client.admin.113929.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130740.log.gz 2023-06-10T19:00:46.474 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124465.log 2023-06-10T19:00:46.474 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113929.log.gz 2023-06-10T19:00:46.475 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130075.log 2023-06-10T19:00:46.475 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134990.log 2023-06-10T19:00:46.475 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.124465.log: /var/log/ceph/ceph-client.admin.130075.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124465.log.gz 2023-06-10T19:00:46.475 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.134990.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130075.log.gz 2023-06-10T19:00:46.475 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.134990.log.gz 2023-06-10T19:00:46.476 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161145.log 2023-06-10T19:00:46.476 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150321.log 2023-06-10T19:00:46.476 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158085.log 2023-06-10T19:00:46.476 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150321.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.161145.log.gz 2023-06-10T19:00:46.476 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158085.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150321.log.gz 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.158085.log.gz 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123336.log 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110905.log 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145577.log 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.110905.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123336.log.gz 2023-06-10T19:00:46.477 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.145577.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110905.log.gz 2023-06-10T19:00:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127670.log 2023-06-10T19:00:46.478 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145577.log.gz 2023-06-10T19:00:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131163.log 2023-06-10T19:00:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119257.log 2023-06-10T19:00:46.478 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.127670.log: /var/log/ceph/ceph-client.admin.131163.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.127670.log.gz 2023-06-10T19:00:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.119257.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131163.log.gz 2023-06-10T19:00:46.479 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.119257.log.gz 2023-06-10T19:00:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113193.log 2023-06-10T19:00:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111135.log 2023-06-10T19:00:46.479 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.113193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167971.log 2023-06-10T19:00:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111135.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113193.log.gz 2023-06-10T19:00:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167971.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111135.log.gz 2023-06-10T19:00:46.480 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.167971.log.gz 2023-06-10T19:00:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154064.log 2023-06-10T19:00:46.480 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168240.log 2023-06-10T19:00:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161007.log 2023-06-10T19:00:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.168240.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154064.log.gz 2023-06-10T19:00:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.161007.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.168240.log.gz 2023-06-10T19:00:46.481 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.161007.log.gz 2023-06-10T19:00:46.481 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130535.log 2023-06-10T19:00:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141098.log 2023-06-10T19:00:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169916.log 2023-06-10T19:00:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141098.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.130535.log.gz 2023-06-10T19:00:46.482 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.169916.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141098.log.gz 2023-06-10T19:00:46.482 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.169916.log.gz 2023-06-10T19:00:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162874.log 2023-06-10T19:00:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109429.log 2023-06-10T19:00:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151065.log 2023-06-10T19:00:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109429.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.162874.log.gz 2023-06-10T19:00:46.483 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151065.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109429.log.gz 2023-06-10T19:00:46.484 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151065.log.gz 2023-06-10T19:00:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131393.log 2023-06-10T19:00:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150459.log 2023-06-10T19:00:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109799.log 2023-06-10T19:00:46.484 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150459.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131393.log.gz 2023-06-10T19:00:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.109799.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150459.log.gz 2023-06-10T19:00:46.485 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109799.log.gz 2023-06-10T19:00:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150045.log 2023-06-10T19:00:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154635.log 2023-06-10T19:00:46.485 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.150045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162579.log 2023-06-10T19:00:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.154635.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150045.log.gz 2023-06-10T19:00:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164464.log 2023-06-10T19:00:46.486 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.154635.log.gz 2023-06-10T19:00:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162579.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.162579.log.gz 2023-06-10T19:00:46.486 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107831.log 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.164464.log.gz 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158354.log 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.107831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136179.log 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158354.log: /var/log/ceph/ceph-client.admin.136179.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.158354.log.gz 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136179.log.gz 2023-06-10T19:00:46.487 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131441.log 2023-06-10T19:00:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157947.log 2023-06-10T19:00:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131441.log: /var/log/ceph/ceph-client.admin.157947.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.131441.log.gz 2023-06-10T19:00:46.488 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157947.log.gz 2023-06-10T19:00:46.488 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117032.log 2023-06-10T19:00:46.488 INFO:teuthology.orchestra.run.smithi104.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.107831.log.gz 2023-06-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128176.log 2023-06-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166071.log 2023-06-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.128176.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117032.log.gz 2023-06-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139373.log 2023-06-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166071.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.128176.log.gz 2023-06-10T19:00:46.490 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139373.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.166071.log.gz 2023-06-10T19:00:46.490 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139373.log.gz 2023-06-10T19:00:46.490 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-06-10T19:00:46.490 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135358.log 2023-06-10T19:00:46.490 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112744.log 2023-06-10T19:00:46.491 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135358.log: /var/log/ceph/ceph-client.admin.112744.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.135358.log.gz 2023-06-10T19:00:46.491 INFO:teuthology.orchestra.run.smithi104.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112744.log.gz 2023-06-10T19:00:46.491 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158500.log 2023-06-10T19:00:46.491 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166458.log 2023-06-10T19:00:46.491 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.158500.log: /var/log/ceph/ceph-client.admin.166458.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158500.log.gz 2023-06-10T19:00:46.492 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.166458.log.gz 2023-06-10T19:00:46.492 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117990.log 2023-06-10T19:00:46.492 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145143.log 2023-06-10T19:00:46.492 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117990.log: /var/log/ceph/ceph-client.admin.145143.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117990.log.gz 2023-06-10T19:00:46.492 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145143.log.gz 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155784.log 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123474.log 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.155784.log: /var/log/ceph/ceph-client.admin.123474.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.155784.log.gz 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.123474.log.gz 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126487.log 2023-06-10T19:00:46.493 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134279.log 2023-06-10T19:00:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.126487.log: /var/log/ceph/ceph-client.admin.134279.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.126487.log.gz 2023-06-10T19:00:46.494 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134279.log.gz 2023-06-10T19:00:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117538.log 2023-06-10T19:00:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164930.log 2023-06-10T19:00:46.494 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117538.log: /var/log/ceph/ceph-client.admin.164930.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117538.log.gz 2023-06-10T19:00:46.495 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.164930.log.gz 2023-06-10T19:00:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111227.log 2023-06-10T19:00:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108906.log 2023-06-10T19:00:46.495 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.111227.log: /var/log/ceph/ceph-client.admin.108906.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111227.log.gz 2023-06-10T19:00:46.495 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108906.log.gz 2023-06-10T19:00:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136962.log 2023-06-10T19:00:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130443.log 2023-06-10T19:00:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136962.log: /var/log/ceph/ceph-client.admin.130443.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136962.log.gz 2023-06-10T19:00:46.496 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.130443.log.gz 2023-06-10T19:00:46.496 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108860.log 2023-06-10T19:00:46.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113469.log 2023-06-10T19:00:46.497 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108860.log: /var/log/ceph/ceph-client.admin.113469.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108860.log.gz 2023-06-10T19:00:46.497 INFO:teuthology.orchestra.run.smithi104.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113469.log.gz 2023-06-10T19:00:46.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166596.log 2023-06-10T19:00:46.497 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141810.log 2023-06-10T19:00:46.498 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166596.log: /var/log/ceph/ceph-client.admin.141810.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.166596.log.gz 2023-06-10T19:00:46.498 INFO:teuthology.orchestra.run.smithi104.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.141810.log.gz 2023-06-10T19:00:46.498 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151872.log 2023-06-10T19:00:46.498 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112916.log 2023-06-10T19:00:46.498 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151872.log: /var/log/ceph/ceph-client.admin.112916.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.151872.log.gz 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112916.log.gz 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139710.log 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138079.log 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.139710.log: /var/log/ceph/ceph-client.admin.138079.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139710.log.gz 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138079.log.gz 2023-06-10T19:00:46.499 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133769.log 2023-06-10T19:00:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163144.log 2023-06-10T19:00:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133769.log: /var/log/ceph/ceph-client.admin.163144.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133769.log.gz 2023-06-10T19:00:46.500 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.163144.log.gz 2023-06-10T19:00:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164174.log 2023-06-10T19:00:46.500 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153611.log 2023-06-10T19:00:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.164174.log: /var/log/ceph/ceph-client.admin.153611.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.164174.log.gz 2023-06-10T19:00:46.501 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153611.log.gz 2023-06-10T19:00:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162828.log 2023-06-10T19:00:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139894.log 2023-06-10T19:00:46.501 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.162828.log: /var/log/ceph/ceph-client.admin.139894.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.162828.log.gz 2023-06-10T19:00:46.502 INFO:teuthology.orchestra.run.smithi104.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139894.log.gz 2023-06-10T19:00:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151295.log 2023-06-10T19:00:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161283.log 2023-06-10T19:00:46.502 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.151295.log: /var/log/ceph/ceph-client.admin.161283.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.151295.log.gz 2023-06-10T19:00:46.502 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161283.log.gz 2023-06-10T19:00:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159774.log 2023-06-10T19:00:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116208.log 2023-06-10T19:00:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.159774.log: /var/log/ceph/ceph-client.admin.116208.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.159774.log.gz 2023-06-10T19:00:46.503 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116208.log.gz 2023-06-10T19:00:46.503 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117446.log 2023-06-10T19:00:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127039.log 2023-06-10T19:00:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117446.log: /var/log/ceph/ceph-client.admin.127039.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117446.log.gz 2023-06-10T19:00:46.504 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127039.log.gz 2023-06-10T19:00:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167925.log 2023-06-10T19:00:46.504 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.148724.log 2023-06-10T19:00:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167925.log: /var/log/ceph/ceph-client.0.148724.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.167925.log.gz 2023-06-10T19:00:46.505 INFO:teuthology.orchestra.run.smithi104.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.148724.log.gz 2023-06-10T19:00:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114767.log 2023-06-10T19:00:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123152.log 2023-06-10T19:00:46.505 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114767.log: /var/log/ceph/ceph-client.admin.123152.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114767.log.gz 2023-06-10T19:00:46.506 INFO:teuthology.orchestra.run.smithi104.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123152.log.gz 2023-06-10T19:00:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166163.log 2023-06-10T19:00:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127528.log 2023-06-10T19:00:46.506 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.166163.log: /var/log/ceph/ceph-client.admin.127528.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.166163.log.gz 2023-06-10T19:00:46.506 INFO:teuthology.orchestra.run.smithi104.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.127528.log.gz 2023-06-10T19:00:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125265.log 2023-06-10T19:00:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118705.log 2023-06-10T19:00:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.125265.log: /var/log/ceph/ceph-client.admin.118705.log: 88.9% -- replaced with /var/log/ceph/ceph-client.0.125265.log.gz 2023-06-10T19:00:46.507 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.118705.log.gz 2023-06-10T19:00:46.507 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133566.log 2023-06-10T19:00:46.508 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121078.log 2023-06-10T19:00:46.508 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.133566.log: /var/log/ceph/ceph-client.admin.121078.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133566.log.gz 2023-06-10T19:00:46.508 INFO:teuthology.orchestra.run.smithi104.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.121078.log.gz 2023-06-10T19:00:46.508 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137615.log 2023-06-10T19:00:46.508 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.137615.log: /var/log/ceph/ceph-mgr.y.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137615.log.gz 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118220.log 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128563.log 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.118220.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118220.log.gz 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.0.128563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130878.log 2023-06-10T19:00:46.509 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130878.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130878.log.gz 2023-06-10T19:00:46.510 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167515.log 2023-06-10T19:00:46.510 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.167515.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.167515.log.gz 2023-06-10T19:00:46.510 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-06-10T19:00:46.510 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122174.log 2023-06-10T19:00:46.510 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.122174.log: 95.6% -- replaced with /var/log/ceph/ceph-client.0.128563.log.gz 2023-06-10T19:00:46.511 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122174.log.gz 2023-06-10T19:00:46.511 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130627.log 2023-06-10T19:00:46.511 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118519.log 2023-06-10T19:00:46.511 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.130627.log: /var/log/ceph/ceph-client.admin.118519.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.130627.log.gz 2023-06-10T19:00:46.511 INFO:teuthology.orchestra.run.smithi104.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118519.log.gz 2023-06-10T19:00:46.512 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121124.log 2023-06-10T19:00:46.512 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149209.log 2023-06-10T19:00:46.512 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.121124.log: /var/log/ceph/ceph-client.admin.149209.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121124.log.gz 2023-06-10T19:00:46.512 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149209.log.gz 2023-06-10T19:00:46.512 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125808.log 2023-06-10T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160212.log 2023-06-10T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.125808.log: /var/log/ceph/ceph-client.admin.160212.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125808.log.gz 2023-06-10T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160212.log.gz 2023-06-10T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108154.log 2023-06-10T19:00:46.513 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117858.log 2023-06-10T19:00:46.514 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.108154.log: /var/log/ceph/ceph-client.admin.117858.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108154.log.gz 2023-06-10T19:00:46.514 INFO:teuthology.orchestra.run.smithi104.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117858.log.gz 2023-06-10T19:00:46.514 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135772.log 2023-06-10T19:00:46.514 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167423.log 2023-06-10T19:00:46.514 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.135772.log: /var/log/ceph/ceph-client.admin.167423.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135772.log.gz 2023-06-10T19:00:46.515 INFO:teuthology.orchestra.run.smithi104.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.167423.log.gz 2023-06-10T19:00:46.515 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117124.log 2023-06-10T19:00:46.515 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108449.log 2023-06-10T19:00:46.515 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.117124.log: /var/log/ceph/ceph-client.admin.108449.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117124.log.gz 2023-06-10T19:00:46.515 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108449.log.gz 2023-06-10T19:00:46.516 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114359.log 2023-06-10T19:00:46.516 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138125.log 2023-06-10T19:00:46.516 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.114359.log: /var/log/ceph/ceph-client.admin.138125.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.114359.log.gz 2023-06-10T19:00:46.516 INFO:teuthology.orchestra.run.smithi104.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.138125.log.gz 2023-06-10T19:00:46.516 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131811.log 2023-06-10T19:00:46.517 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110163.log 2023-06-10T19:00:46.517 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.131811.log: /var/log/ceph/ceph-client.admin.110163.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.131811.log.gz 2023-06-10T19:00:46.517 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110163.log.gz 2023-06-10T19:00:46.517 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116070.log 2023-06-10T19:00:46.517 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158408.log 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.116070.log: /var/log/ceph/ceph-client.admin.158408.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116070.log.gz 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.158408.log.gz 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136133.log 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166117.log 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.136133.log: /var/log/ceph/ceph-client.admin.166117.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136133.log.gz 2023-06-10T19:00:46.518 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.166117.log.gz 2023-06-10T19:00:46.519 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163731.log 2023-06-10T19:00:46.519 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132931.log 2023-06-10T19:00:46.519 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.163731.log: /var/log/ceph/ceph-client.admin.132931.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.163731.log.gz 2023-06-10T19:00:46.519 INFO:teuthology.orchestra.run.smithi104.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.132931.log.gz 2023-06-10T19:00:46.519 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123290.log 2023-06-10T19:00:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110378.log 2023-06-10T19:00:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.123290.log: /var/log/ceph/ceph-client.admin.110378.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123290.log.gz 2023-06-10T19:00:46.520 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110378.log.gz 2023-06-10T19:00:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141006.log 2023-06-10T19:00:46.520 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.141006.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.141006.log.gz 2023-06-10T19:00:46.521 INFO:teuthology.orchestra.run.smithi114.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-06-10T19:00:46.521 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-06-10T19:00:47.253 INFO:teuthology.orchestra.run.smithi104.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-06-10T19:00:47.425 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-06-10T19:00:47.435 INFO:teuthology.orchestra.run.smithi114.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-06-10T19:00:47.467 INFO:teuthology.orchestra.run.smithi114.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-06-10T19:00:47.504 INFO:teuthology.orchestra.run.smithi114.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-06-10T19:00:47.512 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-06-10T19:00:47.530 INFO:teuthology.orchestra.run.smithi114.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-06-10T19:00:47.532 INFO:teuthology.orchestra.run.smithi114.stderr: 2023-06-10T19:00:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m1.633s 2023-06-10T19:00:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m7.608s 2023-06-10T19:00:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.284s 2023-06-10T19:00:47.823 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-06-10T19:00:47.978 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-06-10T19:00:47.979 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-06-10T19:00:47.980 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m2.124s 2023-06-10T19:00:47.980 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m8.180s 2023-06-10T19:00:47.980 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.746s 2023-06-10T19:00:47.981 INFO:tasks.ceph:Archiving logs... 2023-06-10T19:00:47.982 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi104/log 2023-06-10T19:00:47.983 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-10T19:00:49.728 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi114/log 2023-06-10T19:00:49.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-10T19:00:50.601 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-10T19:00:50.610 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-10T19:00:50.610 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-10T19:00:50.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-10T19:00:50.710 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-10T19:00:50.721 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-10T19:00:50.721 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-10T19:00:50.724 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-10T19:00:50.740 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2023-06-10T19:00:50.741 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2023-06-10T19:00:50.745 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-10T19:00:50.745 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2023-06-10T19:00:50.745 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -1209us[-1159us] +/- 97ms 2023-06-10T19:00:50.746 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -1484us[-1435us] +/- 70ms 2023-06-10T19:00:50.746 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +2707us[+2756us] +/- 81ms 2023-06-10T19:00:50.746 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T19:00:50.747 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-10T19:00:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2023-06-10T19:00:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 -1201us[-1201us] +/- 97ms 2023-06-10T19:00:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -1476us[-1470us] +/- 70ms 2023-06-10T19:00:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +2716us[+2716us] +/- 81ms 2023-06-10T19:00:50.749 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-06-10T19:00:50.749 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-10T19:00:50.760 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-10T19:00:50.761 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-10T19:00:50.780 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-10T19:00:50.846 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-10T19:00:50.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-10T19:00:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:50.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-10T19:00:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:51.016 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-10T19:00:51.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-10T19:00:51.044 INFO:teuthology.task.internal:Duration was 1715.382688 seconds 2023-06-10T19:00:51.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-10T19:00:51.059 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-10T19:00:51.060 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-10T19:00:51.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-10T19:00:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-10T19:00:51.110 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-10T19:00:51.404 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-10T19:00:51.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2023-06-10T19:00:51.405 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-10T19:00:51.428 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2023-06-10T19:00:51.428 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-10T19:00:51.450 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-10T19:00:51.450 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-10T19:00:51.470 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-10T19:00:51.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-10T19:00:51.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-10T19:00:51.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-10T19:00:51.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-10T19:00:51.662 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-10T19:00:51.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-10T19:00:51.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-10T19:00:51.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-10T19:00:51.727 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-10T19:00:51.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-10T19:00:51.755 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2023-06-10T19:00:51.780 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2023-06-10T19:00:51.794 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-10T19:00:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:51.824 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-10T19:00:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-10T19:00:51.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-10T19:00:51.861 INFO:teuthology.task.internal:Transferring archived files... 2023-06-10T19:00:51.861 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi104 2023-06-10T19:00:51.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-10T19:00:51.899 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167/remote/smithi114 2023-06-10T19:00:51.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-10T19:00:51.935 INFO:teuthology.task.internal:Removing archive directory... 2023-06-10T19:00:51.935 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-10T19:00:51.937 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-10T19:00:51.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-10T19:00:52.000 INFO:teuthology.task.internal:Not uploading archives. 2023-06-10T19:00:52.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-10T19:00:52.013 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-10T19:00:52.014 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-10T19:00:52.016 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-10T19:00:52.030 INFO:teuthology.orchestra.run.smithi104.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 10 19:00 /home/ubuntu/cephtest 2023-06-10T19:00:52.042 INFO:teuthology.orchestra.run.smithi114.stdout: 262319 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 10 19:00 /home/ubuntu/cephtest 2023-06-10T19:00:52.044 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-10T19:00:52.058 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-10T19:00:52.145 INFO:teuthology.nuke:Checking targets against current locks 2023-06-10T19:00:52.175 DEBUG:teuthology.nuke:shortname: smithi104 2023-06-10T19:00:52.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-10T19:00:52.203 DEBUG:teuthology.nuke:shortname: smithi114 2023-06-10T19:00:52.204 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-10T19:00:52.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-10 18:26:56.703761', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-10T19:00:52.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-06-10_18:20:34-fs:functional-main-distro-default-smithi/7301167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-10 18:26:56.701891', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-10T19:00:52.265 INFO:teuthology.orchestra.console:Power off smithi104 2023-06-10T19:00:52.266 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-10T19:00:52.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:00:52.314 INFO:teuthology.orchestra.console:Power off smithi114 2023-06-10T19:00:52.315 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-10T19:00:52.337 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:00:56.327 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:00:56.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:01:00.448 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:01:00.478 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:01:00.661 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2023-06-10T19:01:04.595 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-10T19:01:04.810 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2023-06-10T19:01:04.916 INFO:teuthology.run:Summary data: description: fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} duration: 1715.3826878070831 failure_reason: 'Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b06794b35d2d41dba25baa651e3bc133 status: fail success: false 2023-06-10T19:01:04.917 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-10T19:01:04.996 INFO:teuthology.run:FAIL